How do Muslims justify the destruction of the places of worship of polytheists?

Since India has lots of Muslims, I am asking this here to understand what their point of view on this is. Muhammad famously destroyed the place of worship of polytheists. There is long history of Muslims destroying places of worship (Somnath fo example) or taking it over and turning into a mosque (Hagia Sophia for example). I have even seen some Pakistani leaders take pride in these. So how do Muslims of India or in general come to terms with this?