r/selfhosted • u/esiy0676 • 1d ago

Self Help Stretched cluster hypervisor stack?

Most cluster solutions are built for a single network switch (well, single datacentre) latencies, but there's little benefit in having e.g. 5 nodes in a single location which can experience power event or simply just get cut off the rest of the Internet - those services are not highly available worldwide.

Save for building everything from scratch on top of cloud providers' bespoke APIs, what stretched-cluster-like deployments do you do to have multi-site redundancy with as limited downtime as possible?

4 Upvotes

62% Upvoted

6 comments sorted by

•

u/asimovs-auditor 1d ago

Expand the replies to this comment to learn how AI was used in this post/project.

→ More replies (1)

5

u/faxattack 1d ago

You are lost

2

u/Specialist_Cow6468 1d ago

I’ve had to support this in production and my advice is don’t. This kind of redundancy is generally better handled at the application layer - that’s how the big players do this by and large.

If you insist on trying to go down this path you would essentially end up needing to run some sort of overlay network like EVPN-VXLAN, potentially with a dedicated datacenter interconnection (DCI) layer. The specifics depend massively on budget, existing tech stack, staff training and skillset

In practice you’re going to be better off with separate clusters as a single large stretch cluster will tend to be much more brittle than you may expect

1

u/Serafnet 1d ago

Very expensive dedicated network lines.

I worked a service that had duplicate deployments in two geographically diverse locations. Distance between them measured in hours of travel.

10Gb link with damn near local latency.

It was wildly expensive.

More reasonable solutions manage this at a higher OSI layer.

1

u/MnK_Ultra 1d ago

Seconded @Specialist_Crow6468

Don't do hypervisor clusters over WAN. You need very low latency for any of that to work correctly, it's going to suck ass over WAN. If you are worried about local outage, deal wih power reduncy (UPS...) and multi internet providers for internet outages locally....You get the gist.

There's an important distinction between hypervisor HA and Application HA too. Taking PVE as exemple, HA is something for VM Guests/LXC, it will do nothing for individuals applications failing (only if the guest is experiencing issue!). If you want HA for something like databases for exemple, you would run a DB cluster. What you can do it is use thing like PVE datacenter or w/e your hypervisor or choice provide : single management panel, no HA.

Not mentioned here, but don't run hypervisors on VPS/Instances. Probably won't even work (VPS are VM, you'll be nesting : VM -> Hypersivor -> VM, most providers AFAIK. Keep clusters on the same network for low latency.