I never gave this scene much thought before, but can we talk about it for a minute? Before Killian speaks, Rumple is visibly conflicted. He wants Milah back, he doesn’t want Bae to lose his mother; but his fear, his complete lack of self-worth and self-confidence, and the label COWARD echoing in his mind are all holding him back, paralyzing him. Then Killian says, “Go on, pick it up,” and he doesn’t say it menacingly. His voice has this warmer tone to it than the colder, biting one he uses when later stating, “A man unwilling to fight for what he wants deserves what he gets.” He is actually imploring Rumple to pick up the sword and fight for Milah. You can even see it in his eyes.
I’d be willing to say that at this point, Killian hasn’t fallen in love with Milah yet—least of all, he was simply willing to give the woman a second chance at a happier life by letting her travel on his ship with his crew (which, btw, it is worth noting that allowing women at sea was actually something heard of because of nautical superstition). But when Rumple shows at least enough bravery to board his ship, he sees a different chance for Milah: her husband could prove he is more than just a coward by fighting for her. Killian throwing down the sword and telling Rumple to fight was not an act meant to mock the village coward. It was a test. Because in his own way, he is encouraging Rumple. He wants Rumple to fight for Milah, so he gives him a chance to prove himself for her. I would also be willing to say that if Rumple picked up the sword and tried to fight, Killian wouldn’t have just sent him away for naught in the end. Killian has old-fashioned values that hold to the belief that a woman should be fought for (“I do consider myself to be an honorable man, a man with a code”). When Rumple failed the test, Killian saw that he was unwilling to fight for Milah, and believed she deserved better than a life with such a man.
Even at the start, Killian never was a cold-hearted villain that would just steal another man’s wife as if she was no more than an object. He respected the woman’s agency, which was to get away from the husband whom she’d never believe would even fight for her; and he even gave her husband the chance to prove otherwise.