SMB Activity just seems to die randomly every few minutes

Hello everyone! This year I have set up my own home server/NAS for saving my documents, photos and media to. It's a Lenovo M93P Tiny with a Xeon and 8GB of RAM with a 2.5 Gigabit NIC. My machine is a 2020 M1 MacBook Air on the latest OS. When transferring files over Ethernet/Wi-Fi, things will be progressing as normal and I get about 150MBs - 200MBs and then bang - maybe 1 or 2 kilobytes in or out. This lasts for about 30 seconds and then MacOS decides its ready to start transferring files again. Is this a problem with MacOS or could it be my server? ahhhhh \- Thanks