Difficult to wake two M1 Macs

Hi all, I have two M1 Macs - my personal Mac Mini (16gb/1tb/Monterey) and a work-provided Macbook Air (8gb/256gb/Sequoia) that is almost always docked. Both of them have difficulty waking from sleep with keyboard or mouse activity - basically I can press a few keys on the keyboard or click the mouse and see my display start to wake up (power light stops blinking, backlight activates briefly), but then go back to sleep. If I try again I usually get the same result. I usually need to continually hammer on the keyboard and mouse before I actually see anything on screen. What gives? I'm using the same keyboard and mouse for both, connected by USB (and have tried other keyboards and mice). Both systems connect to the peripherals through a KVM, and I don't have any problems waking up my Windows machine or my 2012 Mac Mini (i7/16gb/1tb/Mojave). I've also bypassed the KVM and connected straight to each Mac, no difference. Waking up each machine using remote desktop works every time. Anyone else having the same issues?