Visiting India as a person of Indian decent

I've always wanted to visit India as I'm of Indian descent however some of my friends who are also of Indian descent who had visited India have cautioned me against it. I always get told of stories about beggars who constantly harass you, there's filth everywhere, people trying to scam you especially rickshaw drivers, men groping and constantly staring at you which would make you very uncomfortable. I really want to visit the land of my ancestors as 180 years ago my ancestors left India. I love experiencing different cultures and learning about history. I would like to visit the Taj mahal, red fort , temples the jama mosque, kashmir, Rajasthan, kerala, kashmir, the wagah border ceremony, North East India. I currently work in korea as an English teacher and I would like to visit during my vacation. When I mentioned this to my korean friends and colleagues they all told me it would be a mistake to visit as India is not safe at all. I want to ask people from India if what foreigners say is true or is it exaggerated. Obviously I'm not going to book some cheap hostel to stay in and I will book tour packages with an agency, I'm a Muslim though so would I face any prejudice.