You licensed the right to live in your house to him, and that's the fucking problem.
Imagine you could make arbitrary copies of your house at zero cost, and there were a bunch of people seeking shelter and prepared to pay. Some of them also like to wear shoes at home. You could easily sell them each a copied house and let them do whatever, but instead you decide to rent your houses out (despite being able to create an unlimited amount of them), and then - because you technically still own all those houses - you impose any conditions you want on your new tenants, such as "you can't wear shoes indoors".
The problem here is that most normies decide to be happy with this renting arrangement despite there being no actual reason for it, and then comply with your conditions. The minority that doesn't like your conditions - the ones that usually wear shoes at home, for instance - are understandably angry that you did this; however, they now have no leverage because all the idiot normies happily rent out your non-libre houses, so you have no incentive to cater to the few unhappy ones.
THAT'S what software freedom is about. RMS also believes all houseowners should be able to make and sell copies of the houses too, but others disagree, and it's not the most important part.