Why are smaller, cheaper cars more fun to drive?

I love SUVs, but I’ve recently realized that I enjoy driving smaller, underpowered cars more than powerful SUVs. For example, if I had to choose between a Swift/Polo and a Seltos/Fortuner, I’d go with the smaller ones. There’s something about small cars that feels more relatable, almost like they’re ‘one of us.’ Bigger cars, on the other hand, don’t feel like.. I don't know how to put this.. like ‘mine.’ This got me thinking.. do most people feel the same way, or is it because driving a bigger, more expensive car means being extra cautious about the surroundings and worrying about scratches or dents, which can make the experience less enjoyable?