Especially when those 2nd, 3rd, + properties are being used as passive short term rentals. Observing the state of the housing situation “Hmm there aren’t enough homes for normal families to each have a chance, I should turn this extra property of mine into a vacation rental.” does this make said person a POS?
If it’s legally habitable, someone could be living there imo. Just price the rent adequately low for the value. I’m not saying it’s morally evil for you to have it, but it’s definitely a luxury.