Website always ranking lower in search despite doing everything right

I run a speed dating service and I have a website for it both as HTTP and HTTPS. Nearly everytime I check rankings for it, most terms never higher than 6 for the HTTPS version, but occasionally for the HTTP version, only the main page ranks 1. I have some banklinks to my site and I have tuned my website everyday ensuring that the pages follow google's pagespeed insights guidelines. I even go above and beyond to make the pages fast and lean. Each page's payload is 50KB average. Meanwhile, Eventbrite, whose site is slow, and tends to receive alot of complaints from people always ranks #1 for almost any event related search query. I'm afraid if I make every page as HTTP that eventually no device will be able to access it, however HTTP pages on my server load faster (by about 20mS each time) than HTTPS which is another google ranking signal. Is google just being selective now and displaying what it feels like as the top results even if they're bad? or is there another SEO factor I'm missing? because quite frankly I've put in quite alot of effort and I'm getting nowhere. And please don't get me started on google ads because I cannot afford that now and that might also be a waste of money particularly if bots click the ads.