There's a genre of network engineering war story that always reads the same way: the change was designed carefully, reviewed, even lab-tested — and the two-hour maintenance window still ran past 2 AM. One well-known write-up of a 10-hour cutover hits the recurring beats: the lab didn't match the field (everything racked together instead of distributed), a version mismatch surfaced mid-window with poor error messaging, and a software bug appeared only under production conditions. The design was fine. The execution path was untested.
That's the distinction this post is about. Testing the end state ("the new config works") is what most teams call lab testing. Rehearsing the cutover — executing the actual runbook, in order, under time, with rollback — is a different exercise, and it's the one that saves maintenance windows.
Bottom line: don't just validate the destination; rehearse the journey. A cutover rehearsal executes every runbook step on a mirror of the affected segment, times each one, injects the failures you fear, and proves the rollback — before the window opens. In NetPilot, the mirror lab deploys from a plain-English description or sanitized configs in ~2 minutes, and the agent executes and times the runbook with you.
Why cutovers fail when the design was right
Post-mortems on blown windows cluster into five failure modes — none of which end-state testing catches:
- Step-ordering mistakes. The route swing happened before the new link was verified; the ACL landed before the session drained. Order is invisible in a config diff — it only exists in execution.
- Missed dependencies. The monitoring VRF, the NTP source, the management path that quietly rode the old circuit — discovered at 1 AM when it disappears.
- Convergence nobody timed. The design converges. But in how long, with how much in-flight loss? "A few seconds" and "ninety seconds" are the difference between a blip and a bridge call.
- The rollback that was never executed. Every change record has a rollback section; almost none have a rollback that was ever run. Rollback under pressure, untested, is just a second untested change.
- Version and platform surprises. Software mismatches and platform quirks that only show up when the exact steps run on the exact OS versions — the strongest argument for rehearsing on real network OS code rather than an approximation.
What a real rehearsal looks like
A cutover rehearsal has four properties that distinguish it from ordinary lab testing:
- It mirrors the affected segment, current state first. You build where you are, not where you're going — old circuit, old routing — because the rehearsal starts from step zero.
- It executes the runbook literally, in order, timed. Every step as written. Where the runbook says "verify," the verification actually runs. Each step gets a measured duration, which is how the "2–3 hour estimate" gets an evidence-based correction before the CAB approves the window.
- It breaks things on purpose. Run it clean once, then run it ugly: fail the new circuit at step 6, drop a BGP session mid-swing, make the worst step slow. The DMVPN cutovers and dual-hub migrations that go smoothly are the ones where someone rehearsed the failure branch, not just the happy path.
- It ends by rolling back. The last rehearsal run executes the rollback section completely and times it. Now the rollback is a tested procedure, and you know your true abort deadline: window end minus measured rollback time.
The rehearsal loop, agent-run
Here's the workflow in a NetPilot testing lab, using the classic case — cutting over to a new circuit at a primary data center on a DMVPN network:
1. Mirror the current state.
"Build a mirror of this segment: dual DMVPN hubs, eight spokes, OSPF inside the tunnels, primary circuit on hub-1. Here are the sanitized configs."
The lab deploys on real NOS images in ~2 minutes — real routing stacks (FRR neighbors fill in for the parts of the network you don't need to model device-by-device), so convergence behavior is executed, not estimated.
2. Baseline it. The agent captures routing tables, adjacency state, and a reachability matrix, and starts continuous probes — the in-flight traffic that will tell you what users would have felt at every step.
3. Execute the runbook, step by step.
"Run cutover step 4: bring up the new 10G circuit on hub-1 and verify the tunnel forms. Don't proceed — show me state first."
The agent executes the step across the affected devices (translating per-vendor syntax where the lab is mixed), reports the resulting state, and records the elapsed time. You approve each gate exactly as you would in the real window. Direct CLI is always available — SSH into a hub mid-rehearsal and eyeball show dmvpn or show ip ospf neighbor yourself; the rehearsal is only as trustworthy as your ability to verify it by hand.
4. Run the failure branches. Re-run from the snapshot, and this time: "fail the new circuit immediately after step 6 and show me what the spokes do." Measure the convergence, the loss, and whether the runbook's contingency actually works.
5. Execute the rollback, for real. The final pass runs the rollback section end to end and times it. The output of the whole exercise is the evidence bundle a change board wants: per-step timings, observed convergence, failure-branch behavior, and a proven rollback — the same pre/post snapshot discipline as network change validation, applied to the procedure instead of the config.
"But the lab isn't the field" — the fidelity objection
The 10-hour-cutover write-up makes the honest counterpoint: their change was lab-tested, and the field still bit them. Three things follow. First, rehearse on real network OS code — version-specific bugs and platform quirks only reproduce where the actual software runs; that's the difference between emulation and simulation. Second, mirror the topology shape that matters (the distributed layout, the management path), not just the configs — cheap to do when the lab is described in plain English rather than racked by hand. Third, treat the residual gap honestly: a rehearsal doesn't guarantee the window, it removes the failure modes you can remove — ordering, dependencies, untimed convergence, unexecuted rollback — which is most of them.
FAQ
What is a network cutover rehearsal?
A cutover rehearsal (also called a mock cutover or dry run) is executing the full maintenance-window runbook — every step, in order, with verifications — against a mirror of the affected network segment before the real window. Each step is timed, failure branches are exercised, and the rollback is actually executed, so the runbook enters the window already proven once.
How do I test a cutover plan without production hardware?
Build the affected segment as a virtual mirror lab on real network OS images and run the runbook against it. In NetPilot, describe the segment in plain English or paste sanitized configs and the lab deploys in ~2 minutes; the agent executes each runbook step, reports state at every gate, and times the whole procedure — with SSH access to verify any step by hand.
What should a cutover rollback plan include?
A tested execution path, not just intent: the exact command sequence to return to the pre-change state, the verification checks proving you're back, the measured time it takes, and the abort deadline it implies (window end minus rollback duration). If the rollback has never been executed anywhere, it's an untested change wearing a safety label — rehearse it in the mirror lab like any other step.
How long should a cutover rehearsal take?
Typically one to three runs of the runbook: a clean pass, a failure-branch pass, and a rollback pass. With an agent-built mirror lab the environment cost drops to minutes, so the rehearsal cost is essentially the runbook execution time itself — which is exactly the number you need to know before the window anyway.
Rehearse before the window opens
Paste your runbook and sanitized configs at app.netpilot.io and run the whole window — steps, failures, rollback — before it's real.
Copy-paste ready: the vendor-migration cutover prompt rehearses a parallel-run migration with equivalence checks before cutover — paste it into app.netpilot.io and adapt the steps to your runbook.
Related: AI-native network testing lab · Test plans & NRFU, generated and executed by AI · Network change validation