Am i overthinking?? Is this too trivial to think about?

This is my first Reddit post. So, I am a teacher in a school, and today was annual day there. As I was wearing a saree for the event, I thought to take a manual rickshaw instead of an e-rickshaw to reach back home, for which I pay 10 rupees daily. I asked this old man the fare, and he said 100 Rs. I immediately said no and started walking away. Just then he called me again and started insisting that I take it for Rs 50. I agreed and took the ride. When we reached at my address, I handed him a 100 rs note as I didn't have the change with me, and then he said, "Bus ho gya pura," to which I was amazed and gobsmacked. I argued that that's not what we agreed on, and he started blaming me that I didn't tell him the address correctly. It was way too far, blablah ... Firstly, money isn't the main concern here. The issue is dishonesty, greediness, and misbehaving. Secondly, I felt he treated me like that because I am a girl, and he overpowered me with his voice and actions. I should have done something at that time, and that's what's still bothering me.