Thanks for the suggestions.

I see no CPU spike on the VPS, and no CPU spikes on the clients.
I use WG started by root using wg-quick via systemctl on all devices.
I tried setting the MTU to 1280, with no significant changes, apart from slight slowdown compared to MTU of 1420 or 1440.
Smaller packet size also resulted in slightly lower speeds.

I used tcpdump on both client and server to find the negotiated MSS, and it shows an MSS of 1460 outside wg tunnel, so by following the calculations shown in this article https://www.procustodibus.com/blog/2022/12/wireguard-performance-tuning/, 1440 is the correct MTU for the wireguard interface when using IPv4 inside the tunnel.

I had to borrow a phone to set up a mobile hotspot.
It has the same speeds inside the wireguard tunnel as when I tested from my wired connection (250 kbps TCP, 170 kbps UDP).
The loss reported by iperf is dependent on the bandwidth that i test with. But as I increase bandwidth from the client the loss grows towards 100%.

I tried testing in reverse (sending from VPS to devices on different networks) with surprising results:

• TCP, wireguard: 5-10 mbps
• UDP, wireguard: 50 mbps
• TCP, no wireguard: 45 mbps
• UDP, no wireguard: 250 mbps (saturates download speed on client when compared to speedtest.net)

Thanks! I had not read the manpage close enough, I guess.
When specifying the bandwidth, I can saturate the connection using UDP to around 15 Mbits/s. (That this speed is much much lower than the 300-500 Mbits my connection and the VPS is capable of is a problem for a different time, I think).

What I also realized is, that I had not put the iperf server in UDP-mode, so my results reported in another comment are wrong.
I read the results from the client, but the server did not respond.
When running the iperf server in UDP-mode, I get 15 Mbits/s outside the wireguard tunnel and 180 Kbits/s inside the tunnel. With TCP-mode I get 10-15 Mbits/s outside the thunnel and 250 Kbits/s inside the tunnel.

I am currently running mtr from multiple devices:

• in wireguard tunnel: single hop 10.1.1.1, loss% 0,1, avg ping 27.4 ms
• outside wg between same devices:
  • ISP supplied modem/router 55% loss, avg ping 1.6 ms
  • multiple hops without loss, avg ping 16-20 ms
  • random intermediary 30 % loss, avg ping 20 ms
  • endpoint, 5% loss, avg ping 25 ms

It looks like across the board my ISP modem / router is dropping 50-80 % of packets, and that packet loss is ramping up from 4% to 80 percent after a few minutes of running mtr.
It also looks like my VPS endpoint climbs to 20% packet loss over time (5-15 minutes of testing).

Can I use this information to probe further into the devices I have access to (ISP modem and VPS)?

MTU is 1440.

I’m on linux on all devices. I tested packet fragmentation using:
ping -M do -s [packet size] [ip]
No fragmentation using packet size 1472 and below (before wireguard overhead).

I just tested again:

• no wireguard, tcp: 9.75 Mbits/s
• no wireguard, udp: 1.05 Mbits/s
• wireguard, tcp: 248 Kbits/s
• wireguard, udp: 1.05 Mbits/s

So, I get the “full” udp speed, but I get some errors / warnings about ‘connection refused’ and ‘did not receive ack’. Obviously not correctly configured, when it is 10 times slower than tcp.