It is true that the unit cannot use the Ld of characters who has refused a challenge, but why would that effect the Crown of Command? It's an item than confers a special rule to the wearer and his unit, not a Ld bonus.
So I would say yes, the unit would still be stubborn.