Configure Proxmox's pveproxy to listen on localhost only
TL;DR: Read the man page
Put the following in
/etc/default/pveproxy (it needs to be created if not already existing).
Oh it's also mentioned in the manpage but it's worth reiterating, tweaking this setting after setting up a cluster is asking for an issue, the cluster normally communicates directly on 8006.
This could break something pretty major so uh, not my fault.