Why didn't Bhagwan Vishnu ever take birth in a poor labourer',s family?

After Narsingha, Vishnu always took a birth but never in some poor family. Its always some royal family or otherwise influential family. Why?