Especially when those 2nd, 3rd, + properties are being used as passive short term rentals. Observing the state of the housing situation “Hmm there aren’t enough homes for normal families to each have a chance, I should turn this extra property of mine into a vacation rental.” does this make said person a POS?
Are you denying another family a house they would’ve otherwise been able to buy? Then yes.
In healthy system they should be able to afford to build it.