I have 3 proxmox nodes, each with a 500g ssd for boot and another 1tb which I want to use for ceph.

Its recommended to have a separate 10g network for this, which is where the problem starts. Right now i have CRS Mikrotik switch with two sfp+ ports, and i have already ordered two single port connectx-2 cards which I was going to use to have a faster connection between one node containing my NAS, and another which is the one that reads off it the most (plex, etc.).

Now I’m debating changing this. I have 3 options because I dont want to spend too much money on this:

• Don’t have a separate network for ceph, and get a third sfp+ card for my third server + a cheap-ish 8 port sfp+ switch from tp-link for 140euros, and connect it to my main Mikrotik switch. Both ceph and the nas communication would happen over one single port/nic.
• Alternatively, I could do the same as above but get another two sfp+ nics (or just 2 dual ports) and have a separate network for ceph, and also have a fast nas connection. This would be the most expensive.
• Cheapest option: there is this 40euro switch on aliexpress which has two sfp+ ports and 4 2.5G rj45 ones. Could I have two nodes use 10g links and one have a 2.5G on the same ceph network? or would that cause problems? This would be a separate network. Servethehome reviewed what is basically that switch and came away positive.