Five SD-WAN Migration Failure Modes in the Real Network

Every SD-WAN demo ends with the same slide: three coloured arrows, a green tick on every application, and a claim that the branch is now policy-driven. The migration is the part the slide skips. In production, a branch that had one MPLS circuit and a static default route suddenly has three transports, a controller, and a policy that someone pushed from a central template at 2 a.m. Most of the failures I have seen in that transition come from five places, and they recur across vendors because they are architectural, not product-specific.

Why migrations fail at the seams

An SD-WAN migration changes two things at once. It moves control from distributed edge configuration to a central policy plane, and it moves the underlay from one predictable path to several unpredictable ones. Either change alone is manageable. Together, they create a class of failure where the old mental model and the new tool disagree, and the seam between them is where the incident happens.

The demos are run on a clean topology, a single transport, and a handful of applications with well-known signatures. Production is a branch with legacy L2 devices, an internet circuit that was sized for guest Wi-Fi, a data centre behind a stateful firewall, and an application owner who cannot tell you whether their traffic is TCP or UDP. The failures below all show up in that gap.

Policy, paths and application identity

Centralised policy and its precedence

Moving policy to a controller is the core promise and the first failure point. In a routed network, precedence is written down in configuration order: route maps, ACL sequence numbers, QoS policy nesting. In a central policy plane, precedence is a template model, and the order in which a per-route policy, a per-tunnel policy, a security policy and a QoS policy apply is defined by the vendor's model rather than by the order the operator wrote them.

The failure shows up when a new policy is pushed to a site cohort and the traffic does not do what the text says. It usually happens because the old configuration had an implicit behaviour that was never written down, and the new model makes a different implicit choice. The reason this is dangerous is blast radius: a single template error can reach every site at once, and the rollback is another push through the same controller. What makes it workable is treating policy as code, staging by cohort, previewing the effective policy per site, pinning template versions, and rehearsing the rollback before the first push. Where it does not matter: a ten-site estate with one engineer and no template reuse can safely configure each site and skip most of this.

Asymmetric paths and stateful devices

Multiple transports mean forward and return paths can differ. A branch with local internet breakout can send a flow out of the local circuit and receive the reply through the data centre. If a stateful device sits anywhere in that path, a firewall, a NAT function or a load balancer, the return traffic arrives without its session state and is dropped or reset.

NAT is the classic case. If the outbound translation happens on one transport and the return arrives on another, the external address differs and the state lookup fails. Dynamic routing makes it worse when unequal-cost paths cause a prefix to flap between transports, taking the return path with it. The fix is flow affinity: pin flows to a transport or tunnel for their lifetime, apply stateful policy and NAT on the same tunnel, and prefer symmetric return for flows that cross stateful devices. Where it does not matter: short-lived DNS, QUIC and most SaaS traffic tolerates asymmetry, so the problem is traffic-class dependent rather than universal. Test with a real TCP session through a stateful device, not with a ping.

Application identification

Application-aware routing depends on classification. The platform can classify by deep packet inspection, by IP and port, by domain name, or through a cloud connector. Encrypted traffic limits DPI, SaaS providers rotate address ranges, and first-packet classification can misparse a connection that later turns out to be something else. The result is an "unknown" bucket that grows quietly and falls into the default policy, which is usually the wrong path or the wrong queue.

The failure is rarely total. It is a percentage of sessions classified wrongly, which means the voice or video flow that the design promised to protect gets treated as generic traffic on a lossy internet path. The reason is that application policy is only as good as the classifier, and the classifier is a moving target. What works: a small number of application classes with explicit policy instead of hundreds of signatures, validation with a packet capture at the branch, monitoring the unknown bucket as a first-class metric, and treating signature updates as a change that can reclassify the estate. Where it does not matter: if you steer by subnet and port for a handful of known applications, sophisticated identification is unnecessary.

Failure	First sign in production	Who usually notices
Policy precedence	The same policy behaves differently at two sites	Service desk, after users complain
Asymmetric path	Long-lived sessions reset; short ones are fine	Firewall team, from session logs
Application identity	Protected traffic shares the default path	Application owner, from quality reports
DNS	Internal names resolve to the wrong site or time out	Branch users, immediately
Rollback	Nobody can reach the device to undo the change	Everyone, at the worst possible moment

DNS and rollback

DNS is not a side quest

DNS is the first thing almost every flow does, and it is often the last thing migrated. Split-horizon DNS, internal zones, resolver reachability per transport, proxy settings and encrypted DNS all interact with path selection. If the policy steers DNS to a central resolver over a transport that has degraded, the branch experiences "the internet is down" while every link is technically up.

The specific traps are internal names resolving to a data centre address that is no longer reachable over the chosen path, a branch resolver that caches the wrong answer for the duration of the cutover, and encrypted DNS bypassing the resolver that the design depends on. The reason this belongs in the migration plan rather than the operations guide is that the wrong answer is cached and the symptom outlives the cause. Test split-horizon resolution from the branch, keep TTLs short during cutover, have a fallback resolver, and monitor resolution latency per site. Where it does not matter: a cloud-first estate with no internal zones and no legacy applications will barely notice DNS at all.

The rollback that was never tested

The most expensive failure is the one where the migration works and the rollback does not. A cutover plan is not a change plan. The rollback has to be rehearsed, and it has to work without the thing you just changed, which is the hard part: if the overlay is how you reach the device, you cannot use the overlay to fix the overlay.

That is why out-of-band access at critical sites, a controller-independent management path, exported configurations before the change, and a pre-agreed fallback route are not optional extras. It is also why the old circuit should not be decommissioned until a defined stability window has passed, and why the stop/go gate needs a measurable criterion rather than a feeling. Where a simpler approach is acceptable: small sites can often accept "replace the appliance and rejoin the old circuit" as a rollback, because the site is simple enough to rebuild. Large sites and hubs cannot, and those are the ones where a live fallback has to exist.

Before a site is cut over, I want these checks to run on the day, not the week before. The commands below are illustrative shapes, not vendor-verified output, and the exact syntax depends on the platform.

show sdwan control connections
show sdwan bfd sessions
show ip route 10.10.10.10
show sdwan policy from-vsmart
show logging | include DHCP|DNS

What I would put in the per-site cutover runbook:

• Record the pre-change state: routes, tunnels, DNS answers, firewall sessions and the current configuration export.
• Confirm all transports are healthy before the template is applied, not after.
• Verify controller reachability from an independent path, not through the device being changed.
• Stage the policy change by cohort and compare behaviour between cohorts before widening.
• Test classification with a real capture for the applications that the design claims to protect.
• Test split-horizon DNS resolution from the branch and confirm the fallback resolver works.
• Prove symmetry with a real TCP session through every stateful device on the path.
• Rehearse the rollback on a pilot site, including the case where the controller is unreachable.
• Keep the old circuit live until the stability window has passed and the metrics agree.
• Define the stop/go criterion in numbers before the window opens, and give one person the authority to call it.