From Measurement to Trustworthy Control

Last lecture, Jaber walked you through bufferbloat. A home router’s buffer absorbed 100 packets silently, added 120 ms of pure queueing delay, and destroyed every interactive application on the network. TCP could not see the queue because TCP measures loss, not delay. The queue lied to TCP. FQ-CoDel, CAKE, and L4S were three ways to make the queue stop lying. They each made a microsecond-scale decision — mark this packet, drop that packet, signal early on this one — and they made it without a human in the loop.

That last phrase will keep coming back today. But before we use it, we need to answer a more basic question.

A question I should have asked sooner

If TCP picks its own congestion window, if BGP picks its own best path, if AQM picks its own packets to drop — why does the network need a human operator at all?

The honest answer is that protocols decide within their scope; humans decide about the scope. TCP’s scope is “what cwnd should I set right now on this flow?” It does not decide which flows are allowed, which queue policy the router runs, when to add a new uplink, what to do when half the campus suddenly cannot reach YouTube, whether the new firewall rule is legal under the campus AUP, or how to peer with the eyeball ISP. Those decisions live outside any single protocol. They are operations decisions.

A network operator is the person (or team) responsible for keeping a network running and meeting its service obligations — provisioning capacity, configuring protocols, responding to incidents, attributing failures, satisfying SLAs, defending against attacks, and changing the network when the world it serves changes. Sometimes that operator is one person at a campus IT department. Sometimes it is a 200-person team at a tier-1 carrier. Sometimes — increasingly — it is a coordinated set of teams across multiple companies that share parts of the same data path.

Protocols automate the decisions inside their scope. Everything else falls to the operator. The story of this lecture, and of the chapter behind it, is the story of how operators sense their network, form a belief about it, and decide what to do — and how much of that loop has itself been progressively automated over four decades. The question we will keep asking is what stays in human hands, and what doesn’t, at each step.

Today’s argument is one sentence:

Most of the design decisions in network operations — how telemetry is sampled, how dashboards aggregate, how alerts get thresholded, how often we poll, what gets compressed away before it reaches the operator — only make sense once you treat the human operator as the binding constraint at the end of the loop.

That observation is the chapter’s synthesis (the corpus supports each component individually). It is also the setup for what comes after. If the human is the binding constraint, what happens if the consumer at the end of the loop is not a human? That is Thursday’s lecture. Today we have to establish the constraint first, on its own terms.

This lecture (L15) gets us from the operator’s loop, through the four invariants, to the moment in 2016 when SDN created centralized control. L16 picks up at the data-plane revolution and what comes next.

Act 1: The operator’s loop

Imagine you are the on-call engineer at a campus IT department. It is 8 pm. A student messages the help desk: “YouTube is broken in my dorm room.”

What do you do?

You look at your monitoring dashboard. You see whether any links are saturated. You look at the campus WiFi access point’s signal in that dorm. You check whether the campus backbone is dropping packets. You make a guess about what is wrong. You change something — restart a service, push a configuration, escalate a ticket. You watch the dashboard again to see if anything got better.

You just ran a loop:

Sense — read the dashboard, the logs, the alerts.
Form a belief — guess what is happening based on what you saw.
Decide what to do.
Act — change a configuration, restart, escalate.
Sense again. Update your belief. Decide again. Act again.

This loop is the unit of analysis for network operations. Everything else — dashboards, alerts, SNMP, SDN controllers, modern automation systems — exists to make some part of this loop faster, cheaper, more accurate, or more accountable. Hold that in mind.

The four invariants, in this loop

Chapter 1 introduced four invariants that every networked system must answer. Recall them quickly:

State: what information does the system track?
Time: at what timescales does the system act?
Coordination: who decides what?
Interface: what does the system expose to whoever uses it?

We applied these to TCP, to routing, to scheduling, to AQM. Now we apply them to the operations loop. The four invariants are four constraints on this one loop, not four independent lenses. The loop has to track some state, act at some timescale, decide via some coordination, expose belief and action through some interface. Today we will see how each of those four answers has changed over four decades.

A running example

Let me give you a scenario to anchor everything. Keep it in your head.

It is 8 pm on a Tuesday at a residential college campus. A student in their dorm room is streaming an HD recording of a guest lecture. The playback freezes for three seconds, then resumes at lower quality.

That is the same kind of event L14 ended with — the bufferbloat scenario, but worse, because today we are going to look at it from five different angles. What just happened? Hold that question. We will come back to it at the end of every Act.

Act 2: There is no single “operator” — the Internet is a network of networks

I have been writing “the operator” singular. That is wrong, and the reason it is wrong is structural.

The Internet is not one network. It is a network of networks. Roughly 75,000 distinct administrative domains — autonomous systems (ASes) — interconnect via BGP. Each AS is operated independently by a different organization. There is no central authority for the Internet. There is no one place where all the operational state lives. Anything the campus IT department does ends at the boundary where the campus hands packets to the eyeball ISP. Anything the eyeball ISP does ends at the boundary where it hands packets to a transit provider or to the content provider’s CDN. The packets cross administrative boundaries multiple times on a single web page load, and each boundary is a place where one operator’s authority ends and another’s begins.

This architectural choice — the network-of-networks model — is why the question “who is the operator?” has more than one answer. Every (vantage, stakeholder) pair we are about to enumerate exists because some independent organization stood up its own piece of infrastructure with its own people running its own operations loop. Multi-stakeholder operations is not a side effect of regulation or accident. It is the architecture.

Attribution under partial visibility

Now define the word attribution, because the rest of the act needs it:

Attribution = figuring out which component caused a user-visible failure.

When the student’s YouTube freezes, attribution is the work of figuring out where the failure happened: in the student’s laptop, the dorm’s WiFi access point, the campus backbone, the eyeball ISP, the long-haul carrier, the CDN, the video server. Each one of those crossings is a place where some operator could be responsible — and where some other operator has no visibility. The on-call engineer at the campus does attribution. The content provider’s customer-experience team does attribution. The regulator does attribution. They all do attribution, but they are each looking at completely different data because they each own a different piece of the path.

Two axes that define what data each operator has

The first axis is vantage point — where in the network you can observe from.

An endpoint (a laptop, a phone, a server’s network card) sees its own RTT, packet loss reported by its own TCP stack, application-level Quality of Experience (QoE — how well the application feels to the user). It cannot see anyone else’s traffic.
A first-hop or edge device — the campus border router, an ISP’s customer-facing equipment, a CDN cache — sees traffic crossing one administrative boundary. “Eyeball ISP” is industry slang for the ISP that connects to consumers.
A transit or core router sees aggregated traffic at backbone scale. It can see how much traffic, but not whose traffic in detail, and certainly not what application it is for.
An out-of-band collector is a server that pre-aggregates summaries — bytes per second per link, top source/destination pairs over the last hour. NetFlow exporter and syslog server are common kinds.
A third-party measurement platform — CAIDA, RIPE Atlas, M-Lab — runs probes from many vantage points across the Internet to see things no single operator can see.

The second axis is stakeholder — who you are, and what action a belief has to support.

A carrier or enterprise network operator owns the infrastructure within their domain. They get rich heterogeneous telemetry. They worry about capacity, availability, security, and Service Level Agreements to their customers.
A content provider — YouTube, Netflix, Cloudflare — owns server infrastructure and a CDN. They get player telemetry from the client side. They see their own traffic at every hop, but the underlying network is mostly opaque to them. They worry about end-user QoE.
An application owner owns applications and user accounts. They have very little visibility into the network underneath.
An end user has only a client-side perspective.
A regulator gets whatever data operators are required to share.

The rebuffer through the lattice

Now run the 8-pm rebuffer through every (vantage, stakeholder) pair:

The student sees a spinning wheel.
The content provider’s customer-experience team sees one rebuffer event for one client; the player’s adaptive bitrate algorithm dropped from 1080p to 480p; no other clients on the same CDN edge cache had problems.
The campus NOC (Network Operations Center — the team running the campus network) sees green dashboards. No SNMP alert fired.
The eyeball ISP sees nothing unusual at the border router.
A third-party measurement probe running on campus — if one were deployed — saw a brief latency spike at exactly that moment.

Five different beliefs about the same event. Each one is true within its vantage. No two of them describe the same picture of what happened.

That gap between the network-wide event and any single observer’s belief is not a temporary engineering inconvenience that better tooling will fix. It is the structural consequence of the network-of-networks model. Independent operators deliberately do not share full telemetry with each other — for commercial, regulatory, security, and contractual reasons. Partial observability is the default, not the exception. Attribution always requires composing partial views across stakeholders and vantage points. No single party can do it alone. This will matter every time we discuss operations decisions for the rest of the lecture.

Pause — let me ask you something

Which of those five parties is responsible for the student’s rebuffer? Which one has authority to fix it? Which has the telemetry needed to diagnose it? These are not the same questions, and they may have different answers. Hold that observation. We will come back to it when we discuss the Coordination invariant.

Act 3: What is the loop for?

Once every stakeholder sees only part of the failure, the next question is: what is the belief supposed to optimize? Operations is not observation for its own sake. The loop exists to preserve performance and security. These are the two evaluation criteria for everything we do.

Let me ground them with the rebuffer.

Performant

When the student’s video froze and then dropped from 1080p to 480p, the network failed to be performant. But “performant” means different things to different observers.

For the campus NOC, performance is capacity on the residential VLAN — how many bits per second can the wire carry?
For the content provider’s server team, performance is Quality of Service (QoS) — how quickly does each video chunk reach the player, with how much variation in delay?
For the student, performance is Quality of Experience (QoE) — did the video play smoothly at high resolution?
For an AI training cluster elsewhere on the same campus, performance is Job Completion Time — how long until the GPU job finishes, given the synchronized communication pattern AI training imposes [20].

The same network can be capacity-rich (the link has bandwidth to spare) and still deliver bad QoE (the bitrate keeps switching). Recall from L11 (video streaming): a 100 ms delay is fine for a buffered video but ruinous for an interactive video conference. Recall from L12: 1% packet loss is invisible to a web download (TCP retransmits) but intolerable in a voice call (the listener hears clicks). Performance is a composite, not a scalar.

One qualifier the working definition has to make explicit: performance is always relative to a workload — the traffic demand pattern the network is being asked to serve.

The same residential VLAN is performant at 8 am (a few students checking email) and fails at 8 pm (forty students streaming HD video). The same datacenter fabric is performant for batch web traffic and fails for synchronized AI all-reduce traffic. Operators consistently call this out. Vahdat at Google argues AI traffic has “unique traffic patterns” that demand a different network design [21]. Meta’s 2024 AI-training paper points to low entropy, high burstiness, and synchronized collective operations as workload characteristics that make the workload itself a first-class variable [23]. The Open Networking Foundation frames performance directly as the “convergence of resource supply and fluctuating demand” [24].

Without naming the workload, “performant” has no operational meaning. The working definition, with the qualifier folded in:

A network is performant when it delivers the maximum achievable capacity AND meets the relevant Service Level Objectives at every layer, for the workload it is being asked to serve, subject to the binding constraints of topology, devices, link capacities, and policy.

Secure

The rebuffer was probably not a security event. But suppose a campus backup job had been overwhelmed by a worm that was scanning every machine in the residential block, and the network state behind the rebuffer was a flood of malicious probe packets. Now the operator’s loop has to keep the network secure, not just performant. Same loop, different objective.

Operational security in the network-management literature is a composite with four well-attested components. The chapter’s working definition pulls them together:

A network is secure when (a) its hardware and software are trustworthy through their lifecycle [25], (b) its declared policies (reachability, isolation, access control) are continuously verified against runtime behavior [26], and (c) anomalies — DDoS, data exfiltration, lateral movement, route hijacks — are detected and mitigated within bounded time [8, 9, 10].

Each component is anchored in operational practice:

Trustworthy substrate is the Kandula (NANOG 89, 2023) three-stage lifecycle: establish trust in hardware, establish a trustworthy network OS, maintain trust at runtime via attestation servers that verify signed evidence against known-good values [25]. This is the foundation; if you cannot trust your switches, none of the upper layers help you.
Continuous policy verification appears in Juniper’s Apstra (“real-time precondition validation: is this new request going to violate intent?”) [26] and in the SDN verification literature (HSA, Batfish, Minesweeper) we will get to in Act 8.
Behavioral anomaly detection is Arista’s AVA framing (“learning what is normal and routine on the network”) [27], plus the long anomaly-detection lineage from BotFinder onward.
Bounded-time mitigation is the operational discipline. Microsoft Azure’s RADAR system explicitly “detects and mitigates Microsoft route hijacks on the Internet” at runtime [28]; Vyakaranam (ONUG 2019) frames closed-loop automation as the architectural pattern that makes bounded-time mitigation feasible [29].

Security in this operational sense is a closed-loop property, not the cryptographic kind. The closed loop is the same sense → believe → decide → act loop from Act 1.

Performance and security are not competing. They are the two binding criteria the belief model must serve simultaneously. Every measurement architecture, every controller, every verification regime in the rest of this lecture is in service of delivering them together, under the operator’s budget for time, money, and attention.

Act 4: The four invariants, applied to the loop

The loop has to preserve performance and security. What constrains it? Four invariants from Chapter 1, now in the operations-loop context. I will walk through each one, ground it in the rebuffer, and point forward to where the chronology will move it.

STATE — three layers

State in operations has three layers. Memorize them; we will reuse this everywhere.

Environment state — what is actually happening on the network. Every packet, every queue depth, every routing decision at every microsecond. Unobservable in full.
Measurement signal — what telemetry actually captures. Counters, packet samples, flow records, log entries. A lossy projection of environment state.
Internal belief — the model the operator (or agent) holds about what is happening. Derived from measurement, shaped by the consumer’s representation capacity.

The belief model is the load-bearing object in the loop. Raw telemetry is not belief. Thresholded alerts are not belief. Belief is a structured representation that supports decisions — including the prediction “if I do X, what will happen to the network?”

Pause — apply to the rebuffer. The environment state was: a burst of traffic on the residential VLAN at 8 pm overwhelmed a queue, packets were dropped, the student’s video player saw losses, the adaptive bitrate algorithm reacted. The campus NOC’s measurement signal was the SNMP counter for that link, averaged over 5 minutes — which showed nothing unusual. The NOC’s belief was “the network is healthy.” Reality and belief diverged because the measurement signal compressed the burst out of existence. This is the same E-M-B (Environment-Measurement-Belief) decomposition L14 used for bufferbloat, applied to a different signal.

Why is this hard? Because of a property of network traffic that the field discovered the hard way in 1994.

Start with one web download. The browser fetches a 1 MB page in about 100 ms of intense activity, then sits idle for several seconds before the user clicks again. Bursty. Now combine millions of such users. Statistical intuition from your probability class — and from decades of telephony engineering — says these independent bursts should average out, like flipping a million coins and getting close to 50% heads. The aggregate should be smooth.

Real network traffic does not behave that way. Leland, Taqqu, Willinger, and Wilson at Bellcore ran a year-long trace of Ethernet traffic at a real research lab and found that at every timescale they could observe — milliseconds, seconds, minutes, hours — the traffic looked equally bursty. Zoom in to a 10 ms window and you see bursts and idle gaps. Zoom out to a 10 second window and you see bursts and idle gaps. Zoom out further; same picture. They named this property self-similarity: the statistical shape of the traffic is the same regardless of the zoom level.

This was an empirical refutation of the Poisson process model the field had inherited from telephony, where independent arrivals do smooth out at aggregate scales. Real Internet traffic does not have independent arrivals. Web pages load in bursts. Streaming chunks arrive in bursts. Backup jobs run in bursts. The bursts at one timescale are themselves made of bursts at finer timescales, all the way down. Paxson and Floyd generalized the result to wide-area Internet traffic in 1995.

The consequence for measurement is direct and brutal:

Averaging at any timescale loses information about bursts at every smaller timescale. A 5-minute SNMP counter average can completely hide a 200 ms burst that saturated the link and dropped half its packets. The campus NOC sees 60% link utilization; the student gets a rebuffer. That gap between aggregate and burst is exactly what bit the operator in the running example.
Uniform sampling under-represents the most operationally interesting packets. A typical traffic mix has a heavy tail: a few enormous flows carry most of the bytes; millions of tiny flows carry almost none. If you keep one packet in every 1,000, you get good statistics on the tiny flows you do not care about and poor statistics on the few large flows you do.
Periodic polling can systematically miss periodic events. If your collector polls every 60 seconds and there is a microburst pattern with a 60-second period, you will see either the worst moment or the best moment every cycle, never the truth. This is why Vern Paxson’s 1997 Poisson-modulated probing — randomize the inter-probe intervals so they do not lock onto any rhythm — became the methodological default for active measurement.

So preserving the signal costs more bandwidth (keep more packets), more storage (keep them longer), more compute (process them faster), and — the constraint that dominated the next forty years — more demand on the human operator’s attention to make sense of it all. Every measurement architecture we will discuss in the chronology is one of these trade-offs made one way or another.

TIME — loop bandwidth

Network operations spans an enormous range of timescales:

Timescale	Decision	Example
Microseconds	Per-packet	AQM mark/drop (L14)
Milliseconds	Per-RTT	TCP cwnd adjustment
Seconds	Per-flow	Bitrate adaptation in the video player
Minutes to hours	Per-incident	The on-call engineer triages the student’s ticket
Days to months	Policy / planning	Procure new switches; redesign the topology

A human’s reaction time — see the dashboard, decide, type — is on the order of seconds. Below the seconds timescale, the human cannot be in the loop. This is a physical fact about Homo sapiens, not a policy choice.

The decisions you saw automated in L13 (scheduling) and L14 (AQM) were not automated because operators got lazy. They were automated because there was no other choice. TCP congestion control automated itself in 1988 for the same reason — RTT timescales are tens of milliseconds, too fast for a human.

Pause — apply to the rebuffer. The actual congestion event lasted maybe 200 ms. The operator’s incident-response loop runs at minutes-to-hours. Even if the operator looked, the event would be over by the time they decided what to do. The only way to react to a 200 ms event is for some machine to react — the queue’s AQM logic, the video player’s bitrate algorithm, a controller. The Time invariant is what determines who gets to be in the loop.

COORDINATION — authority and scope

§Act 2 named who can observe what. Coordination is about who can decide and act on what. The two are different. The content provider can observe its own client rebuffers, but it cannot directly act on the eyeball ISP’s routing. The regulator can require operators to report outages, but it cannot directly fix them. Operations decisions must respect administrative-domain boundaries.

The authority spectrum runs from local to federated:

Per-device CLI — log into each router individually and type commands.
SDN controller within one domain — one central program tells every device what to do.
Multi-domain coordination — Google’s WAN coordinates across many sites under one operator.
Multi-stakeholder attribution — recent academic work (MoCE, NSDI 2026) composes experts that are specialized for different (stakeholder, vantage, telemetry) regimes.
Multi-agent fabric — a population of agents replaces the single controller.

We will see this spectrum unfold in the chronology.

Pause — apply to the rebuffer. Who can act? The campus NOC can rebalance the residential VLAN’s traffic allocation. The content provider can warm a cache closer to campus. The eyeball ISP can change peering policy. None of them, acting alone, can guarantee the student does not see a rebuffer at 8 pm tomorrow. The fix requires coordinated action across multiple authorities.

INTERFACE — the consumer contract

When someone says “the interface,” you probably think of an API or a screen. In network operations, the interface is something more specific: the contract for how belief and action cross the boundary between the network and whoever consumes it.

Two distinct consumer contracts to hold in your head:

Human-facing — the belief reaches the consumer as something a human can read: a colored bar on a dashboard, a flashing alert, a paragraph in a runbook. The action leaves the consumer as something a human can do: click a button, type a CLI command, send a Slack message.
Agent-facing — the belief reaches the consumer as something a machine can ingest: a fixed-dimensional vector, a structured query response, a sampled trajectory from a model. The action leaves as something a machine can emit: a function call, a verified policy, a gated decision.

This is where today’s headline thesis lives. Changing the consumer changes the contract. If we change who is at the receiving end of the operations loop, every other invariant has to move with it.

For forty years, the consumer has been a single human operator. Every dashboard, every alert, every runbook, every CLI was designed around the human’s cognitive bandwidth — about seven items in working memory, about ten visual updates per second, about one second from observation to action. The networks could always emit more. The human could never absorb more. The tooling lived in the gap.

(Author note: the strong claim that human cognitive architecture has been the binding constraint for forty years is the chapter’s synthesis, not a single corpus citation. The literature documents the components individually — Sonata’s workload-reduction framing, alert-fatigue critiques, dashboard-design research — but the unified forty-year framing is ours.)

Act 5: 1988–1995 — Counters and the first measurement reality

The invariants describe constraints on actions in a running network. The rest of this lecture follows the tools operators built — over four decades — to satisfy those constraints. The first widely-deployed version of the operator’s loop was brutally simple: poll device counters with SNMP.

One quick boundary before the chronology

This lecture is about network operations — what the loop does on a running network. There is a separate activity called network planning — choosing what the network is: topology, vendors, capacity. Planning runs on month-to-year horizons. Operations runs on microsecond-to-day horizons. A useful rule: planning changes the menu of possible actions; operations picks from the current menu. AQM marking is operations. Buying new spine switches is planning. We will return to planning only when an operations capability could not have emerged without a planning decision that made it possible.

Binding constraint of the era

In 1988, backbone networks ran at megabit speeds. Topologies were small enough to draw on a whiteboard. The Simple Network Management Protocol (SNMP) was standardized that year. SNMP let a management station periodically ask a device — “what’s your byte count? error count? interface status?” — and a human read the answer.

The protocol’s authors knew the polling interval would limit what could be observed. They picked five minutes as the default because a faster interval would have produced more data than the human in the loop could absorb. There was no point emitting state faster than the consumer could integrate it.

The unit of State: per-link device counters

The canonical thing SNMP polled was a device-link counter — a single number incrementing as packets crossed an interface. The Management Information Base (MIB) defined the standard set: bytes-in, bytes-out, packets-in, packets-out, errors-in, discards-in, per interface, per device. An operator’s belief about the entire network was assembled by polling each device every five minutes and watching the numbers trend.

This was enough when networks were small. It would not survive the next decade.

Two events in 1993–1994 that mattered for decades

1993: BPF (the Berkeley Packet Filter). Steve McCanne and Van Jacobson at LBL wrote a way to express a packet filter in user space, compile it to a small bytecode, and run it inside the operating-system kernel without copying every packet across the user/kernel boundary. Result: tcpdump, Wireshark, the entire host-side packet-capture toolchain. Per-packet visibility was now feasible on any host with reasonable CPU.

1994: Self-similarity discovered. Leland, Taqqu, Willinger, and Wilson published a now-canonical paper showing that real Ethernet traffic was fractal: bursts at every observable timescale. The Poisson process — the basis of decades of queueing theory inherited from telephony — was an empirical mismatch. The data was not what we had assumed. The implications for measurement design are still unfolding thirty years later.

Apply to the rebuffer (1993 version)

In 1993, the campus rebuffer would have shown up as a graph blip on the campus SNMP dashboard, averaged over 5 minutes, probably under-resolved. The NOC analyst would have made a judgment call. There was no NetFlow record. There was no programmable controller. There was only the counter graph and the analyst’s experience.

The State in this era was per-link counters; the Time was 5 minutes; the Coordination was per-device (each router its own administrative island); the Interface was a CLI command and a graph.

The mismatch between the network’s speed (rising past 100 Mbps in commercial backbones) and the loop’s speed (5-minute polls) was about to become the binding constraint of the next era.

Pause — let me ask you something

If a 5-minute SNMP average can hide a 200 ms burst, can it also hide a 30-second outage? Walk through what the operator’s belief looks like in each case. The answer reveals which kinds of failure the era’s tooling could and could not see.

Act 6: 1995–2003 — Flow records and probabilistic methodology

By the late 1990s, backbones reached 100 Mbps and rapidly climbed past 1 Gbps. SNMP at 5-minute intervals missed everything interesting. DDoS attacks rose and fell within seconds. Microbursts dropped packets between polls. The per-link counter — the State unit of the prior era — could no longer represent what operators needed to see.

The flow as the new unit of State

In 1996, Cisco shipped NetFlow v5. NetFlow proposed the flow as the operational unit. A flow is a sequence of packets sharing a 5-tuple (source IP, destination IP, source port, destination port, protocol), aggregated over a timeout window. The router emitted one record per flow — not one record per packet — compressing the per-packet stream by orders of magnitude while preserving what operators actually used: who talked to whom, how much, over which protocol.

By the early 2000s the flow record was the operational lingua franca for traffic engineering, security forensics, and billing. NetFlow v9 (2003) made the record format template-driven. sFlow added explicit packet sampling — keep one packet in every $N$ — to make per-flow tractable at higher link rates. IPFIX standardized the protocol at the IETF.

Probabilistic methodology grows up

Meanwhile, Vern Paxson at LBL was doing the field’s foundational work on measurement methodology. The 1997 Network Probe Daemon (NPD) deployments ran 20,000 active TCP transfers between 35 sites worldwide. The 1999 paper articulated a stance the field still operates by:

“The nature of the medium means the phenomenon under study is not stationary, the measurement instruments interact with the phenomenon, and what we see is always a sample rather than the truth.”

Paxson’s Poisson-modulated probing — randomize inter-probe intervals so that the long-run sample distribution converges to the true distribution — was a direct response to the §Act 5 lesson that periodic polling aliases with periodic phenomena. If you probe at 60-second intervals and a microburst pattern repeats every 60 seconds, you systematically observe either the worst or the best, never the truth.

A second observation surface: routing

While flow records gave you per-flow visibility, routing operations needed a different unit. BGP MIBs made the prefix — a group of IP addresses announced as one route — a first-class observable. Researchers built RouteViews and RIPE RIS as third-party platforms collecting per-prefix BGP feeds from hundreds of vantage points. They existed because no single operator could see what they could see — the lattice from §Act 2 making itself felt for the first time, even in 1997.

Apply to the rebuffer (2000 version)

In 2000, the campus rebuffer would leave a trace in three places: a NetFlow record showing a sudden surge of UDP traffic from the streaming server, an SNMP counter spike on the campus uplink, and (if the campus had bothered) an active ICMP probe showing increased RTT. The NOC operator would mentally correlate all three. No tool yet did the correlation automatically. The State invariant got richer; the Time invariant compressed a little (per-flow records arrive per flow, not per poll); the Interface was still dashboards and CLI.

What stayed broken

Control was still distributed by design. A typical backbone operator in 2002 ran thousands of routers, and even with config-management automation (RANCID, vendor management tools) every router still ran its own BGP, OSPF, and scheduling decisions independently. Operators could tune the inputs to those distributed algorithms, but they could not centrally express what the network as a whole should do. Sharp symptoms were now visible — but the operator had no central place to state network-wide intent and have it propagate, let alone verify that the configurations they had pushed actually implemented that intent. Measurement improved State and Time. The next bottleneck was Coordination.

Act 7: 2003–2010 — From vision to centralized control

Distributed vs centralized control — a definition we have been using without one

Before SDN, network control was distributed. Distributed control means each device runs its own algorithm using only local information and what its immediate neighbors tell it; the network’s behavior emerges from those independent local computations. BGP is distributed: each router computes its own best path from a routing table built up from neighbor announcements. OSPF is distributed: each router floods link-state messages and independently runs Dijkstra’s algorithm. TCP CC is distributed: each endpoint runs its own congestion control. No router holds the whole picture; no single program decides.

The opposite is centralized control: one program runs one algorithm over a global view of network state, and pushes decisions down to the devices. The devices become forwarding plumbing; the program is where the thinking happens.

The trade-offs are not trivial. Distributed control survives partial failures by design (each router keeps going if its neighbors die). Centralized control concentrates intelligence and observability in one place, but the controller is now a single point of failure and a place where bugs can break the whole network. Different applications need different points on this trade-off, which is why both models still exist.

The story of Act 7 is the move from purely distributed control — the only model the Internet had at scale until the early 2000s — to logically centralized control within an administrative domain. Operators had sharper symptoms from §Act 6; they still could not centrally express what the network should do because there was no central place to express it from. The fix took seven years and involved both a vision paper and a wave of engineering systems.

The vision: a Knowledge Plane for the Internet

In 2003, David Clark and colleagues at MIT, ISI, and Berkeley published a position paper that named the missing layer [@clark2003knowledge]:

“We propose a new construct, the Knowledge Plane, a pervasive system within the network that builds and maintains high-level models of what the network is supposed to do, in order to provide services and advice to other elements of the network.”

The Knowledge Plane did not propose a controller, an API, or a protocol. It proposed the existence of a layer above the data plane and control plane whose four jobs would be:

Maintain an explicit belief about what the network is supposed to do.
Distinguish raw observations (telemetry, counters, log events) from interpreted beliefs (what those observations mean operationally).
Reason about uncertainty — missing data, conflicting evidence, hidden state.
Provide services and advice to the data and control planes that consume the belief.

The Clark 2003 paper proposed using cognitive and machine-learning techniques as the mechanism by which a Knowledge Plane would form beliefs. The lasting contribution, twenty-three years later, is the architectural abstraction — the named layer — not the specific mechanism. Operations is what you do when sensing has produced a belief and a decision must follow. The Knowledge Plane named the belief.

A complementary paper in 2005, 4D by Greenberg and colleagues, proposed splitting the network architecturally into four planes — Decision, Dissemination, Discovery, Data. The split made explicit that centralized intent and distributed packet forwarding were two different concerns that the existing architecture conflated by accident of history.

The engineering: SDN becomes concrete

The vision papers were not enough; someone had to build the centralized controller. The lineage:

2004: Routing Control Platform (RCP) — Feamster, Balakrishnan, Rexford. Take BGP path-selection out of individual routers; compute it on one server.
2007: Ethane — Casado et al. Every flow has to be authorized by a central controller before it can traverse the network.
2008: NOX — Gude et al. The first formally named network operating system — a programming interface to the controller, treating the network as one programmable system. NOX talked to switches via a new protocol called OpenFlow.
2010: Onix — Koponen et al. Addressed the distributed-systems realities of controller deployment that NOX had elided — replication, failure recovery, consistency.

By 2010, the field had a name for the whole enterprise: Software Defined Networking (SDN).

SDN’s key abstraction

What did all of these systems share? Treat the entire network as one programmable system, with a logically-centralized view of state. Three concrete pieces:

Match-action tables in every switch. Packet-handling is expressed as (match this header pattern → take this action). The match patterns and actions are configurable per switch.
OpenFlow protocol. A controller installs match-action rules into switches and queries them. The protocol is standard and vendor-neutral.
Centralized controller. Runs one program over a global picture of the network — the topology, the rules currently installed, the per-rule counters reporting how many packets matched.

Before SDN: each router ran its own algorithm with a local view. After SDN: one program runs once with a global view.

How this enables observability — not just control

This is the part the chronology has been building toward. The match-action abstraction is what makes both control and observability possible from the same primitive.

Every match-action rule has a counter built in. Every time a packet matches the rule, the counter increments. The controller can query any rule’s counter at any time. Multiply that across the whole network and you get:

Topology is centrally known (the controller maintains it).
Per-flow rules → per-flow counters across every switch.
The question “what flows are active right now?” becomes a controller API call rather than a debugging odyssey.

Compare against per-device CLI from §Act 5: in that world, observing “what flows are active right now” meant logging into each router individually, running a vendor-specific show command, parsing free-form text output, and correlating across devices by hand. Hours. Under SDN, the same query is seconds because the abstraction itself centralizes the view.

The Coordination invariant relaxed — and as a side effect, so did the State and Interface invariants. SDN was not pitched as an observability win, but it turned out to be one.

What does SDN actually let operators do?

The abstraction is intellectually clean. But abstractions are only useful if they enable actions you could not take before. By the mid-2010s, several systems demonstrated concrete control actions that SDN made tractable. A few representative examples:

SDX — Software-Defined Internet Exchange Points (Gupta et al., SIGCOMM 2014). Internet Exchange Points (IXPs) are the physical locations where many ASes peer with each other. Traditionally, traffic between two peering ASes was governed by BGP, which expresses policy at the granularity of an entire AS or prefix — nothing finer. SDX put an SDN controller at the IXP and let operators express application-specific peering policies: send all DNS traffic to AS X, send all video traffic to AS Y, redirect DDoS traffic to a scrubber, perform inbound traffic engineering by influencing how upstream ASes choose return paths. These are decisions BGP alone could not express. SDX showed that centralized control could give operators fine-grained inter-AS authority that the existing routing protocol structurally denied them.
B4 — Google’s software-defined WAN (Jain et al., SIGCOMM 2013). Google had to move enormous traffic between datacenters and discovered that traditional WAN traffic engineering drove links to roughly 30–40% utilization, with the rest kept idle as headroom for failures. B4 took TE off the routers and onto a centralized scheduler that could route traffic with global knowledge of link state, application priorities, and traffic patterns. Result: links routinely run above 90% utilization with bounded loss. The control action — re-route this traffic class onto this path right now — required centralized knowledge that distributed routing could not provide. A 2018 follow-up paper (B4 and After) traced the architectural lessons learned across five years of running it.
Espresso — Google’s programmable peering edge (Yap et al., SIGCOMM 2017). Where SDX did inter-AS control at third-party IXPs, Espresso did it at Google’s own peering edge with the rest of the Internet. Centralized control let Google select egress routes based on per-application latency measurements rather than only BGP best-path; programmable hardware-software disaggregation let new policies deploy in software without waiting for ASIC vendors.
Ethane — per-flow authorization at the enterprise (Casado et al., SIGCOMM 2007). Already mentioned in the SDN lineage above; included here as the earliest concrete SDN control action — every flow has to be authorized by a central controller before it traverses the network. The control action — allow or deny this flow — was the prototypical SDN demonstration.

The pattern across these systems is consistent. SDN’s centralized abstraction paid off because it enabled concrete control actions that distributed protocols could not express: fine-grained inter-AS policy (SDX, Espresso), centralized TE with global optimization (B4), per-flow authorization (Ethane). The Coordination invariant relaxation was operationally meaningful precisely because operators got new things they could do, not just new things they could see.

But — where is SDN actually deployed?

A necessary caveat before we go further. SDN is not universally deployed. The vision was universal; the deployment is concentrated where the operational and cost case was overwhelming.

Heavily deployed:

Hyperscaler datacenters — Google’s Jupiter fabric, Meta’s networks, Microsoft Azure, Amazon. SDN-based since roughly 2013.
Hyperscaler wide-area networks — Google B4 (2013), Microsoft SWAN (2013). More than a decade in production.
SD-WAN for enterprise branch-office connectivity — substantial adoption (Velocloud, Versa, Silver Peak).

Limited or absent:

Tier-1 ISP backbones — partial; mostly used for specific traffic-engineering use cases, not as the foundational architecture.
Public Internet routing — almost none. BGP is still per-AS, per-router; the §1.2 lattice of administrative domains makes SDN-across-the-Internet structurally difficult.
Most enterprise core networks — overlay SDN (Cisco ACI, VMware NSX) is common, but pure OpenFlow-style SDN is rare.
P4 / programmable switches outside hyperscalers — rare; merchant silicon (Broadcom Trident) has some programmability, but most enterprise switches are still fixed-function.

That is a recurring pattern in networking innovations: hyperscalers adopt first because their scale makes the operational case overwhelming; the rest of the Internet trails by years to decades. The chapter takes this seriously. When L16 introduces frontier work on representation learning and agentic operations, the same caveat applies — these are research frontiers, sometimes deployed at one hyperscaler, almost never deployed across the broader Internet.

Apply to the rebuffer (2008 version)

At a hyperscaler-scale campus running SDN in 2010, the controller could push a flow rule prioritizing the student’s video traffic — but only if it could recognize the rebuffer condition. The recognition step was still manual. The infrastructure for centralized action existed. The infrastructure for centralized belief did not yet.

At a typical university campus in 2008, none of this existed. The campus still ran per-device CLI on Cisco gear. The on-call engineer logged into each router, parsed text, correlated by hand.

What stayed broken

Two things. First, where SDN was deployed, it created centralized authority — which created a new failure mode: a bug in the controller could blackhole the entire network at the speed of the control plane. Verification became the gating concern at the hyperscaler scale. Second, where SDN was not deployed (most of the Internet), the operator’s loop still ran on per-device CLI and 5-minute SNMP polls. The chronology that follows assumes SDN is in place, because that is where the next-generation tools were built. But the deployment gap is real and recurring.

Act 8: 2010–2016 — Verification becomes the gate

SDN gave operators centralized programmable control. A misconfigured controller could now break the network at controller speed. The verification regime that emerged in 2012–2017 is what made SDN safe enough to deploy in production.

Header Space Analysis (HSA)

In 2012, Kazemian, Varghese, and McKeown published a paper that treated networks as mathematical objects. A packet header was a point in ${0,1}^L$ space. A switch was a function on that space (drop, forward to port A, modify field, forward to port B). Given a network configuration, HSA could symbolically compute reachability:

Can a packet from $A$ reach $B$?
Are there forwarding loops anywhere in the network?
Are there packets that get dropped on every path (black holes)?

All of this answered without sending a single packet. The framing was a step-change: networks became objects you could prove things about, not just operate.

VeriFlow — verification in real time

HSA worked offline. VeriFlow (Khurshid et al., 2013) made verification real-time: as each rule update arrived at the SDN controller, VeriFlow computed which equivalence classes of packets would be affected, re-evaluated invariants, and rejected the update sub-millisecond if a violation was detected. Verification moved from offline tool to inline gate on the control plane.

Batfish — verification across vendor heterogeneity

In 2015, Fogel and colleagues at UCLA built Batfish. Batfish parsed configuration files from real heterogeneous backbones (Cisco IOS, Juniper JunOS, Arista EOS), simulated the control plane (BGP, OSPF, static routes), computed the resulting routing tables, and answered reachability queries against the simulated network. Operators could now ask: “If I push this config change tonight, what reachability changes?” — and get an answer before the change went live.

“Most network outages are caused by configuration errors, not hardware failures or software bugs. We need to analyze configurations before they are deployed.” — Fogel et al. 2015

A handful of related projects matured in the same window — NetKAT (an algebraic specification language for SDN), Propane (high-level BGP policy synthesis), Minesweeper (symbolic model-checking) — and ONOS launched in 2014 as the first SDN OS designed specifically for carrier-grade deployment.

Apply to the rebuffer (2015 version)

By 2015, the campus could push a policy “prioritize streaming traffic from the dorm VLAN” into its SDN controller and verify, before pushing, that the policy did not break reachability for non-streaming traffic. But the reaction was still hours-scale. A human still had to recognize the rebuffer pattern, decide on the policy change, and push it through the verification pipeline. Faster automation would require a richer measurement substrate — and that, in turn, would require the data plane itself to become programmable. That is where L16 picks up.

Where this lecture lands

The L15 chronology has traced one specific arc. Before autonomy, networks needed:

Observability (Act 5–6 — counters, flow records, probes)
Belief (Act 7 — Knowledge Plane, SDN)
Programmable authority (Act 7 — SDN controllers)
Verification (Act 8 — HSA, VeriFlow, Batfish, NetKAT)

Each step relaxed a different invariant for a different reason. By 2016, the operator’s loop ran on programmable centralized authority with continuously-verifiable policies — but the consumer was still a human, and the data plane was still mostly fixed-function. The Interface contract had not yet started to shift.

Thursday’s lecture (L16) picks up at the data plane. Once switches became programmable, observation moved into the data plane (INT, Sonata). Once observation became rich, ML decision-makers entered specific inner loops (Remy, CS2P, Pensieve). Once ML decisions became opaque, the credibility crisis exposed the trust gap. And from there — representation learning, agentic operations, safety certification, multi-stakeholder attribution — the frontier opens.

The thesis to hold across both lectures: AI-powered network operations is the relaxation of the Interface constraint that has shaped four decades of design.

Optional pre-read for L16

Clark, Partridge, Ramming, Wroclawski. “A Knowledge Plane for the Internet.” SIGCOMM 2003. Twenty-three years old; still the cleanest articulation of what this chapter is about. If you have an hour, read it.

References

[1] Ch12: Network Operations — From Telemetry to Belief to Decision. CS176C textbook chapter draft.

[2] Harsh, V., et al. “MoCE: A Mixture-of-Context-Aware Experts Framework for Troubleshooting Internet-scale Services.” NSDI 2026.

[3] Case, J., et al. RFC 1157: A Simple Network Management Protocol (SNMP). IETF, 1990.

[4] McCanne, S., Jacobson, V. “The BSD Packet Filter.” USENIX Winter 1993.

[5] Leland, W., Taqqu, M., Willinger, W., Wilson, D. “On the Self-Similar Nature of Ethernet Traffic.” IEEE/ACM ToN, 1994.

[6] Paxson, V. “End-to-End Internet Packet Dynamics.” IEEE/ACM ToN, 1999.

[7] Paxson, V., Floyd, S. “Wide Area Traffic: The Failure of Poisson Modeling.” IEEE/ACM ToN, 1995.

[8] Cisco Systems. NetFlow Services Solutions Guide. 1996.

[9] Phaal, P., Panchen, S., McKee, N. RFC 3176: sFlow. IETF, 2001.

[10] Clark, D., Partridge, C., Ramming, C., Wroclawski, J. “A Knowledge Plane for the Internet.” SIGCOMM 2003.

[11] Greenberg, A., et al. “A Clean Slate 4D Approach to Network Control and Management.” SIGCOMM CCR 2005.

[12] Feamster, N., Balakrishnan, H., Rexford, J., Shaikh, A., van der Merwe, J. “The Case for Separating Routing from Routers.” FDNA 2004.

[13] Casado, M., et al. “Ethane: Taking Control of the Enterprise.” SIGCOMM 2007.

[14] Gude, N., et al. “NOX: Towards an Operating System for Networks.” SIGCOMM CCR 2008.

[15] Koponen, T., et al. “Onix: A Distributed Control Platform for Large-scale Production Networks.” OSDI 2010.

[16] Kazemian, P., Varghese, G., McKeown, N. “Header Space Analysis: Static Checking for Networks.” NSDI 2012.

[17] Khurshid, A., Zou, X., Zhou, W., Caesar, M., Godfrey, P. B. “VeriFlow: Verifying Network-Wide Invariants in Real Time.” NSDI 2013.

[18] Fogel, A., et al. “A General Approach to Network Configuration Analysis.” NSDI 2015. (Batfish.)

[19] Berde, P., et al. “ONOS: Towards an Open, Distributed SDN OS.” HotSDN 2014.

[20] Arun, S. “Building Lossless AI Fabrics.” APNIC blog, 2026.

[21] Vahdat, A. “Networking for AI: Why Traffic Patterns Demand New Designs.” Google Cloud Blog, 2024.

[22] Stjernholm, P., et al. “Connectivity Services for Application Service Flows.” Ericsson white paper, 2024.

[23] Meta networking team. “Networks for AI Training Workloads.” SIGCOMM 2024 / Engineering at Meta blog.

[24] Open Networking Foundation. TR-521: SDN Architecture for Transport Networks. ONF, 2014.

[25] Kandula, S. “Building Trustworthy Networks.” NANOG 89 talk, 2023.

[26] Juniper Networks. “Apstra Architecture: Real-time Precondition Validation.” Juniper white paper, 2023.

[27] Arista Networks. “AVA: Autonomous Virtual Assist for Network Operations.” Arista white paper, 2024.

[28] Russinovich, M. “RADAR: Detecting and Mitigating Route Hijacks on the Internet.” Microsoft Azure blog, 2020.

[29] Vyakaranam, V. “Low Code and Closed Loop Automation.” ONUG, 2019.

[30] Gupta, A., Vanbever, L., Shahbaz, M., Donovan, S. P., Schlinker, B., Feamster, N., Rexford, J., Shenker, S., Clark, R., Katz-Bassett, E. “SDX: A Software Defined Internet Exchange.” SIGCOMM 2014.

[31] Jain, S., et al. “B4: Experience with a Globally-Deployed Software Defined WAN.” SIGCOMM 2013.

[32] Yap, K., et al. “Taking the Edge off with Espresso: Scale, Reliability and Programmability for Global Internet Peering.” SIGCOMM 2017.

[33] Hong, C.-Y., et al. “B4 and After: Managing Hierarchy, Partitioning, and Asymmetry for Availability and Scale in Google’s Software-Defined WAN.” SIGCOMM 2018.