Taxes are not just money taken from people in the abstract. They are, quite concretely, used for specific purposes. In the US (because the OP is about the US),

Taxes are not just money taken from people in the abstract. They are, quite concretely, used for specific purposes. In the US (because the OP is about the US), property taxes in particular are used to fund local services like schools. So if you eliminate taxes on primary homes, you will cripple the public schools.