Why Spine-Leaf Broke the Three-Tier Model

I still have the Visio from a three-tier design review in 2010: core, aggregation, access, a spanning tree, a VLAN plan, and an oversubscription ratio that someone had negotiated with a budget. That design was correct for the traffic it was built for. Storage replication and backups ran north to south at night, the data centre was a destination rather than a mesh, and the aggregation layer was a sensible place to put policy. Then the applications changed, the traffic turned sideways, and the model that had been an asset became a tax.

What the three-tier model assumed

The hierarchy assumed that most traffic leaves the data centre or at least crosses the aggregation layer on its way somewhere else. It assumed that access switches are cheap and their uplinks can be shared, that a VLAN spanning a few access switches is a natural unit of segmentation, and that spanning tree is an acceptable way to keep the L2 domain loop-free, because the L2 domain is small enough that the blocked links are a minor cost.

Those assumptions were not stupid. A tree gives you a predictable failure domain, a clear place to aggregate policy, and a capacity plan that maps to a budget line per tier. The oversubscription ratio at each tier is a deliberate economic choice: you accept a multiplier because not every port transmits at line rate at the same moment. Where that reasoning holds, three-tier is still efficient.

What broke the reasoning was east-west traffic. Once applications were split across many servers, once storage and compute were separated, once replication, batch processing and machine learning moved large flows between racks, the majority of bytes started crossing leaf to leaf. In a tree, a leaf-to-leaf flow goes up to the aggregation layer, through it, and back down, and the aggregation layer is exactly where the oversubscription ratio lives. The hierarchy was optimising for a traffic pattern that no longer dominated.

Why the fabric moved to two tiers

Spine-leaf inverts the assumption. Every leaf connects to every spine, no leaf connects to another leaf, and the path between any two hosts is a fixed two hops, or three with a super-spine layer. All links between leaves and spines carry equal-cost paths, and the fabric uses ECMP rather than a spanning tree. Capacity scales horizontally: add a spine to add bandwidth across the fabric, add a leaf to add ports, and the oversubscription ratio is a property of the uplink count per leaf rather than a per-tier budget decision.

The important move is not physical, though. It is that the L2/L3 boundary moved to the leaf, and tenant L2 segments became an overlay instead of a physical VLAN spanning access switches. The underlay is an IP fabric running BGP or an IGP; the overlay is VXLAN with EVPN as the control plane. That split is what makes the design work at scale and it is also what makes operations different, because the network you troubleshoot is no longer the network you can see on a cabling diagram.

Why did the overlay become necessary rather than optional? Because workloads need L2 adjacency with mobility and a shared address space, and stretching VLANs across a spine-leaf underlay with STP would reintroduce exactly the problem the fabric was built to remove. EVPN gives you MAC and IP reachability distributed through BGP, with route types that describe MAC/IP bindings, inclusive multicast for broadcast and unknown unicast, and IP prefixes for routing. The consequence is that the control plane now carries host state, and the failure modes move from "the spanning tree blocked a link" to "the EVPN route is missing, stale or pointing at the wrong leaf".

What the overlay changes in operations

The day-to-day tasks barely change their names and change completely in how they are done. The table below is the comparison I use when training a team that is moving from a three-tier estate.

MTU deserves its own sentence because it is the most common self-inflicted wound after an overlay deployment. VXLAN adds 50 bytes: 14 for the outer Ethernet header, 20 for the outer IP header, 8 for UDP and 8 for the VXLAN header. An underlay provisioned for 1500-byte frames will silently break any packet that needs a full 1500 bytes inside the tunnel, and the failure appears as an application timeout, not as a network alarm. The fix is a documented underlay MTU, jumbo frames on the fabric links, and a test that sends full-size packets through the overlay rather than pings.

The second wound is ECMP hashing. ECMP distributes flows, not packets, and a small number of very large flows can land on the same spine uplink while the other links sit idle. The aggregate bandwidth is there; the distribution is not. This is why carriers and large data centres care about hash entropy and why some newer platforms offer adaptive routing or packet spraying. The boundaries matter: for many small flows, standard hashing is fine; for a handful of elephant flows, the fabric can be simultaneously congested and under-utilised, and no amount of adding spines will fix a hash that keeps choosing the same path.

The third is operational split. In three-tier, one team could own the whole stack and reason about it physically. In an overlay fabric, the underlay is a routed IP network with its own convergence and its own change windows, and the overlay is a control plane with route types, tunnel state and MAC/IP scale. If the same small team owns both, the cognitive load is real, and the most common production failure I have seen is a change that was correct in one layer and wrong in the other.

Where three-tier is still right, and the post-migration checklist

Three-tier has not been abolished. Campus networks are still trees, because campus traffic is mostly north to south and the users are attached to a physical building. Small data centres with a handful of racks and predictable traffic do not need a fabric. Environments with strict legacy L2 requirements, or where the budget for a full redesign does not exist, are often better served by a collapsed core than by a half-built spine-leaf. The failure mode is adopting the topology without adopting the operational model: a spine-leaf that is really a tree with extra links, managed with spanning tree and per-tier capacity planning, gets the cost and none of the benefit.

Equally, spine-leaf does not fix a bad design. A two-tier fabric with a high oversubscription ratio and a poor hash is still a bottleneck with a fashionable name. The topology buys you a path budget and a segmentation model; it does not buy you the discipline to monitor the underlay and the overlay as two systems.

What I would verify after a move to an overlay fabric:

• Underlay MTU is documented, consistent, and tested with full-size packets through the tunnel.
• ECMP hash configuration is consistent across leaves and spines, and elephant-flow behaviour is known.
• Every leaf has a defined role for anycast gateway, ARP and ND suppression, and it matches the design.
• EVPN route types in use are documented, and missing-route cases are part of the runbooks.
• BUM traffic has an explicit design: multicast underlay or ingress replication, with a scale number.
• Failure isolation is drilled as a two-layer exercise, not a single physical path trace.
• MAC and IP route scale is measured against platform limits, including the headroom for a site failure.
• Underlay and overlay change windows are separate, and nobody changes both at once.
• A leaf failure has been tested for its effect on attached dual-homed and single-homed hosts.
• The team can explain, without notes, why this topology is the right one for the traffic it carries.