When we started designing CubePath's network infrastructure, we had a choice. We could do what most providers do: give each region its own isolated network with standard MTU 1500 and move on. It's cheaper, it's simpler, and honestly, most customers won't notice the difference on day one.

We decided to go a different route. We invested in building a global private network with VLANs spanning multiple regions and MTU 9000 (jumbo frames) across the board. It cost more. It took longer. But it opens up a category of architectures that simply aren't possible on a standard network, and we think that matters.

This post explains why we made that investment and what it enables for the teams building on CubePath.

Why MTU 9000? Because 1500 Bytes Per Packet Is a Bottleneck

Let's start with the technical reality. The default MTU on most networks is 1500 bytes. That's been the standard since Ethernet was designed in the 1980s. Every packet your servers send carries a maximum of 1500 bytes of actual data, plus headers on top.

For casual web traffic, that's fine. But the moment you start moving serious amounts of data between servers, those small packets become a real constraint.

Take a 1 GB database backup. At MTU 1500, that's roughly 700,000 packets that need to be assembled, sent, received, verified, and acknowledged. Every single one of those packets costs CPU cycles. Headers need processing, checksums need calculating, interrupts need handling. Multiply that by the hundreds of transfers happening every hour on a busy infrastructure, and you're burning real compute on network overhead.

Now take the same transfer at MTU 9000. Each packet carries 6 times more data. That 1 GB backup drops to around 120,000 packets. Less CPU overhead, fewer interrupts, higher throughput. In real benchmarks, jumbo frames improve bulk transfer performance by 15-30%. On workloads that constantly shuffle data between nodes, like database replication, distributed storage, or container east-west traffic, that difference compounds throughout the day.

We looked at what our customers are actually building. Database clusters with streaming replication. Distributed storage with Ceph and GlusterFS. Kubernetes clusters with hundreds of pods talking to each other. Big data pipelines moving intermediate results between nodes. All of these are workloads where network efficiency directly translates to performance and cost. So we made the call: MTU 9000 everywhere, not as a premium add-on, but as the foundation.

Why Multi-Region VLANs? Because Real Architectures Don't Live in One Datacenter

Most providers give you a private network within a single region. Your servers in Spain can talk to each other privately, and your servers in Amsterdam can talk to each other privately. But if Spain needs to talk to Amsterdam? You're going over the public internet, setting up VPN tunnels, or paying for some kind of interconnect service.

That creates a friction that pushes teams toward simpler (and more fragile) architectures. If connecting two regions is a project in itself, people avoid it. They run everything in one region and hope nothing goes wrong. Or they build half-baked disaster recovery setups that have never actually been tested because the network layer was too painful to set up.

We wanted to remove that friction entirely. CubePath's VLANs span across regions. Your server in Spain and your server in Amsterdam can sit on the same private network as if they were in the same rack. No VPNs, no tunnels, no special configuration. Just servers that can see each other, with MTU 9000, over a private network that never touches the public internet.

This isn't just a convenience feature. It fundamentally changes what architectures are practical to build.

What This Network Makes Possible

Here's what becomes realistic when your private network is fast, global, and properly sized.

Database Replication That Actually Keeps Up

PostgreSQL streaming replication, MySQL group replication, MongoDB replica sets. All of these depend on shipping data from a primary to one or more replicas over the network. The faster and more efficient that network is, the lower your replication lag.

With MTU 9000 on a private network between regions, your replica in another datacenter stays close to the primary even under heavy write loads. That means your read replicas are serving fresh data, your failover target is actually usable, and your point-in-time recovery doesn't have a multi-second gap.

On a standard MTU 1500 network going through the public internet? Replication lag spikes under load, replicas fall behind, and when you actually need to failover, you find out your standby is minutes behind the primary.

Disaster Recovery That You Can Actually Trust

Disaster recovery is one of those things everyone says they have but few have actually tested. A big reason for that is network complexity. If shipping backups to another region involves setting up VPN tunnels, configuring encryption, dealing with bandwidth costs, and hoping the transfer finishes before the next backup window starts, teams just... don't do it properly.

When your regions are connected on the same VLAN with MTU 9000, disaster recovery becomes much simpler. WAL files ship over the private network in real time. Nightly backups transfer faster because jumbo frames reduce overhead. The network cost is zero because private traffic is free. And because it's easy to set up, teams actually test their recovery procedures instead of assuming they work.

Distributed Storage Without the Network Being the Bottleneck

Ceph, GlusterFS, NFS, iSCSI. All storage protocols that are incredibly sensitive to network performance. A storage cluster is only as fast as the network connecting its nodes.

MTU 9000 reduces packet fragmentation and lets storage traffic flow at near wire speed. If you're running a Ceph cluster across multiple servers, the difference between MTU 1500 and MTU 9000 can be the difference between acceptable performance and constant bottlenecks. We've seen it firsthand with customers running distributed storage, and it's one of the reasons we committed to jumbo frames network-wide.

HA Clusters Where Failover Is Measured in Seconds

High availability clusters depend on fast, reliable communication between nodes. Heartbeats need to arrive on time. State synchronization needs to be fast. When a failover happens, the cluster needs to detect the failure and reorganize in seconds, not minutes.

On CubePath, cluster communication travels over the private network. Heartbeats between Patroni nodes, Redis Sentinel checks, etcd consensus in Kubernetes, HAProxy health checks. All of it runs on a fast, isolated, MTU 9000 network. The result is that failover detection is faster, state transfer during promotion is quicker, and your cluster recovers before your users notice anything happened.

Combine that with CubePath Load Balancers and Floating IPs and you have the building blocks for HA architectures that genuinely work in production. Not just on paper, not just in a runbook nobody has tested, but in real failure scenarios where a node actually goes down at 3 AM and the system handles it on its own.

Kubernetes with Fast East-West Traffic

Kubernetes generates an enormous amount of internal network traffic. Pods talking to other pods, services resolving through cluster DNS, ingress controllers routing to backends, all of it is east-west traffic flowing between nodes.

When that traffic runs on a network with MTU 9000, every inter-pod communication is more efficient. Service mesh sidecars add less overhead. Large payloads between microservices transfer faster. And if you're running a CNI like Cilium or Calico, the underlying network performance directly impacts how fast your pods can communicate.

For CubePath Managed Kubernetes, this means the cluster network performs better out of the box. For self-managed setups, it means you're building on a network that doesn't fight you when your cluster grows.

Big Data Pipelines That Don't Choke on Shuffle

If you've ever run Hadoop, Spark, or any distributed processing framework, you know that the shuffle phase is where everything slows down. Nodes exchange intermediate results with each other, and the speed of that exchange determines how fast your job completes.

MTU 9000 reduces the per-packet overhead during shuffle, which means data moves between nodes faster and your jobs finish sooner. For teams running regular analytics workloads or ETL pipelines across multiple servers, this isn't a marginal improvement. It's the difference between a pipeline that finishes in your batch window and one that doesn't.

The Decision Behind the Investment

Building this network wasn't the easy path. MTU 9000 requires end-to-end support across every switch, router, and NIC in the path. Multi-region VLANs require interconnects between datacenters that most providers avoid because of cost and complexity. It would have been simpler and cheaper to ship standard networking and focus on other features.

But we kept coming back to the same conclusion: the network is the foundation of everything else. You can have the fastest CPUs and the most NVMe storage in the world, but if your servers can't communicate efficiently, your architecture hits a ceiling.

Database replication, distributed storage, HA clusters, container networking, disaster recovery. All of these are limited by how good the network is between your servers. We wanted CubePath to be the kind of infrastructure where you design your architecture based on what's best for your application, not based on what your provider's network can handle.

That's why we built it this way. And it's why we continue investing in expanding this network as we add new regions and capacity.

What's Next

We're continuing to expand our network footprint with new regions and increasing capacity on existing interconnects. As CubePath Managed Kubernetes grows and GPU clusters come online, the demands on the internal network only increase, and the investment in MTU 9000 and multi-region VLANs pays off even more.

If you're designing an architecture that needs fast internal networking, cross-region replication, or real high availability, CubePath's network was built for exactly that.