Every control room runs on numbers it cannot fully trust. Meters drift. Telemetry drops out. A current transformer reads three percent high for a year before anyone notices. A value arrives late, or not at all. Yet operators need one coherent picture of the grid — every bus voltage, every line flow — accurate enough to make decisions worth millions. Those decisions are concrete: whether to redispatch generation, paying one plant to increase output and another to reduce it; whether a corridor is close enough to its thermal limit to require action; whether a transformer is being loaded beyond its rating. Each is expensive to get wrong, in redispatch costs, in equipment life, or in load disconnected that need not have been.
State estimation is the algorithm that produces that picture. It takes everything the grid’s sensors report — noisy, overlapping, occasionally wrong — and reduces it to a single physically-consistent answer, while flagging the readings that don’t fit.
This article builds the method up from scratch on a network you can open and run yourself. Everything below happens on one small grid in BambooGrid, an open-source power systems simulation UI built on top of pandapower.
The example network
Open BambooGrid and load File ▸ Open example ▸ State estimation demo. You get a small mixed-voltage grid that the rest of the article uses:
- Four 110 kV buses (Bus 1–4) in a mesh of four lines, with Bus 1 tied to the external grid (the voltage reference).
- A 110/20 kV two-winding transformer feeding a 20 kV bus (Bus 5).
- A 110/20/10 kV three-winding transformer feeding a 20 kV and a 10 kV bus (Bus 6 and Bus 7).
- Loads on the HV mesh and on every lower-voltage bus.
The State estimation demo: seven buses across 110 kV, 20 kV and 10 kV, tied to the external grid at Bus 1.
Every bus carries a voltage measurement; the load buses carry power measurements; the lines and transformers carry flow measurements. In total, 35 measurements cover a grid whose complete state is just 15 numbers — a count derived exactly further down. That mismatch, far more readings than unknowns, is what makes the method work.
Press Study ▸ State estimation ▸ Run. The buses update to their estimated voltages and a "converged" notification appears; on a network this size the solve completes in a few iterations. The rest of this article explains what happened.
What state estimation solves for
Describing a grid does not require tracking thousands of quantities. One fact underlies the whole method:
If you know the complex voltage at every bus — its magnitude |V| and its angle ∠θ — then every other quantity in the network can be computed from the grid’s physics.
Line flows, currents and injections all follow from the voltages. So the only thing state estimation solves for — the state — is two numbers per bus.
In this demo that is the voltage magnitude and angle at each of the 7 buses. The reference bus (Bus 1, the external grid) has its angle fixed at zero, the reference every other angle is measured against, so it is not unknown. This is the same role the slack bus plays in a load flow, covered earlier in this series: bus angles are meaningful only relative to a chosen reference, so one bus is fixed at zero and every other angle is expressed with respect to it. That leaves 6 angles + 7 magnitudes = 13 unknowns, plus two more described below — 15 in total.
Everything the estimator does serves the goal of finding those numbers.
The problem with raw measurements
If the voltage at every bus is what we want, why not meter each one directly? Four reasons, each visible in the demo.
- Every meter is noisy. A reading is true value + error, and the size of that error is characterized by the meter's standard deviation, σ — the typical difference between what it reports and the true value, expressed in the meter's own units. A small σ denotes a precise meter, a large σ an imprecise one. The demo's voltage meters carry σ = 0.004 p.u., about ±0.4%. Its power meters carry σ = 2 MW/Mvar flat, regardless of the size of the flow — which means 2 MW on Bus 3's 60 MW load is a relatively precise meter, while the same 2 MW on Bus 7's 12 MW load is a relatively imprecise one. Take any single reading at face value and you inherit its error.
- You cannot measure everything. Voltage angles are rarely metered directly in the field; they are usually inferred from the flows around them. The exception is the phasor measurement unit (PMU), which measures voltage phase angle directly against a GPS time reference and so supplies a state variable rather than a quantity the estimator must deduce. PMU deployment is growing but remains limited, particularly in Europe and particularly below the transmission level, so for most networks much of the state is still never observed directly. This demo contains none, which is the common case.
- The readings disagree. Because of noise, the meters contradict each other. The power one meter reports leaving Bus 2 toward Bus 3 will not exactly equal the power another meter reports arriving at Bus 3, even though physics requires them to match minus losses. Which reading should be preferred?
- Some readings are simply wrong. A frozen sensor, a swapped wire, a units mistake. Feed that into the model and every downstream decision is affected.
State estimation therefore does not rely on any single meter. It asks one question:
“What single, physically-consistent state of the grid best explains all of these noisy readings at once?”
What makes that question answerable is the mismatch noted above: redundancy. With far more measurements than unknowns, the extra readings let the estimator average out noise, infer the quantities nobody measured, and identify readings inconsistent with the rest. That last property is the basis of bad-data detection, covered at the end.
💡 The inverse of a load flow. A load flow assumes the injections are known everywhere exactly and computes the one state that satisfies them; it requires complete, exact input. State estimation takes redundant, noisy, incomplete input and finds the most likely state. Same physics, inverted information problem. (BambooGrid runs both — same network, different study.)
How measurements attach to the grid
Select any bus, line or transformer in the demo and open the inspector. Each attached measurement has a value and a σ, which sets how much weight it carries in the fit and, later, provides the scale its residual is judged against.
The Measurements panel on Bus 3
Bus 3 after a clean solve. Each meter has a value, a σ and an on/off switch. The demo ships one voltage meter per bus; a second has been added here by hand — 1.007 at σ 0.004 alongside 1.01 at σ 0.01 — so two meters disagree about the same quantity and the estimator weights them accordingly. The Rₙ badges are the normalized residuals; the Rₙ = 0.01 is revisited at the end.
The demo uses three kinds of meter:
- Bus voltage V — the magnitude at a bus. This is almost a state directly.
- Bus power injection P, Q — the net power exchanged with the grid at a bus. The value is signed, and the convention matters: pandapower, and therefore BambooGrid, reports bus power in the load convention, positive for power drawn from the network. Bus 3 carries a 60 MW load and its injection measurement reads +60; a bus with net generation reads negative. Net is also meant literally — a bus carrying both load and generation yields one measurement reflecting the balance of the two, not either separately.
- Line / transformer flow P, Q, I — measured at one end of a branch (from/to, or hv/mv/lv).
The important distinction: a measurement is usually not the same as a state. A voltage meter reads one state almost directly. A line-flow meter on Line 2–3 depends on the voltages and angles at both Bus 2 and Bus 3, coupled through the line impedance. A power-injection meter at Bus 3 depends on Bus 3 and every bus it connects to. Most meters are therefore functions of several states at once, and the map of which meter depends on which states is central to solving the problem. That map is made explicit further down.
🔎 Why Bus 1 has no power measurement. The reference bus injection is whatever balances the network, so metering it adds no information.
Two standard terms, both observable in this demo:
- A measurement set is observable when it is sufficient to determine every state. If some part of the grid had nothing measuring it, its state would be indeterminate and the estimation could not run.
- A set is redundant when it contains more than the minimum needed. This demo is heavily redundant by design, which is what lets it reconstruct the state cleanly and, later, identify an erroneous reading.
The objective function J
How does the estimator decide which state is best? It scores each candidate state by how well it explains the meters.
For a given state, physics predicts what each meter should read. The difference between that prediction and the actual reading is the meter’s residual. A good state produces small residuals; a poor state produces large ones. The estimator sums the residuals — squared, and weighted by σ — into a single score:
Term by term:
- measured − predicted is one meter’s residual, the distance between its reading and what the state implies.
- Squaring makes overshoot and undershoot count equally and penalizes large misses disproportionately.
- Dividing by σ² weights each meter by its own noise level. A low-σ meter carries a large weight, so the fit is constrained more tightly to satisfy it; a high-σ meter carries little. This weighting is why state estimation outperforms simple averaging.
- Summing combines every meter’s contribution.
J is a single number: the total, σ-weighted disagreement between the model and the meters. State estimation reduces to finding the state that minimizes J.
Note that even at the optimum, J is small but not zero — noise means no state can satisfy every meter exactly. In the demo the seeded measurements are rounded to a realistic precision, so residuals are small but nonzero. Those residuals carry information of their own, used later for bad-data detection.
Solving it iteratively
We need the state that minimizes J. If the relationship between state and readings were linear, the minimum could be computed in one step. It is not. Grid physics is nonlinear, and the source of that nonlinearity is worth stating precisely, because it determines the structure of the solution.
The real power flowing from bus i to bus j across a branch of reactance X is approximately:
Two properties of that expression make it nonlinear in the state. The voltage magnitudes appear as a product, so the sensitivity of the flow to |Vᵢ| is not a constant — it depends on the current value of |Vⱼ|. The angle difference appears inside a sine, so a one-degree change in θᵢ moves the flow by an amount that depends on the prevailing angle difference. Every flow and injection measurement in the network is built from terms of this form, so the measured-vs-predicted relationship is curved throughout.
One property of real networks limits the practical consequences. Voltages operate close to 1.0 p.u., angle differences between adjacent buses are typically a few degrees, and for small angles sin(θ) ≈ θ. The relationship is therefore close to linear in the region where grids actually operate, which is the assumption the DC approximation adopts outright. State estimation does not make that approximation, but it benefits from the same property: a method that treats the surface as locally linear converges quickly.
That nonlinearity determines how the problem is solved. With a curved objective there is no closed form for the minimum, so the estimator approaches it iteratively:
- Start from an initial guess. A “flat start”: every voltage 1.0, every angle 0. (BambooGrid can also start from a load-flow result.)
- Predict. Compute what every meter should read at the current estimate.
- Compare. Evaluate the residuals, meter by meter.
- Determine the descent direction. Moving toward a lower J requires the local slope of the objective at the current point.
- Step. Move the state along that direction.
- Repeat from step 2 at the new point, until the steps fall below the convergence tolerance.
Descending to the minimum of J
Weighted least squares in one figure: from a flat start, each step follows the local slope — the Jacobian H — toward the minimum. The step size shrinks rapidly, and J settles just above zero at the noise floor.
That "evaluate the slope, step, re-evaluate, repeat" loop is the standard Newton–Raphson structure, the same iterative machinery a load flow uses. The difference is the objective: a load flow solves exact equations, while state estimation minimizes J over redundant, noisy data. (The least-squares variant is often called Gauss–Newton; the two are closely related.)
📜 Where the names come from. The names attached to this machinery correspond to three separate contributions, none of them concerned with power systems. Isaac Newton described the root-finding procedure around 1669, in a form restricted to polynomials and not expressed in terms of derivatives. Joseph Raphson published the simpler iterative formulation in 1690, and it is closer to Raphson's version than Newton's that modern texts reproduce; the joint name is conventional. The least-squares component came from Carl Friedrich Gauss a century later, and he developed it on a problem structurally identical to state estimation. In 1801 the asteroid Ceres was observed briefly and then lost behind the sun. From a small set of noisy, mutually inconsistent telescope observations, Gauss computed the orbit that best fitted all of them and predicted where it would reappear; the prediction was correct. Power systems adopted the method in the late 1960s, when Fred Schweppe formulated static state estimation for grids.
Convergence is fast on this demo — a handful of iterations — because the network is observable, heavily redundant, and its measurements are mutually consistent. Each step re-evaluates the slope at a better point, so the local approximation improves as the estimate converges.
Step 4, the local slope, is where most of the work happens. It has a name, and in BambooGrid it has a view.
The Jacobian: which meter depends on which state
The local slope of the objective is the Jacobian, written H. Open it via Study ▸ Measurement Jacobian (H). It is available immediately after a successful estimation, since the estimator produces it as a by-product of the solve.
The Measurement Jacobian (H) heatmap
35 measurements × 15 states. Strongly colored cells are meters highly sensitive to a state; black cells are structural zeros — the grid’s topology expressed in terms of the measurements. Hovering any cell reads out the derivative behind it.
The layout:
- Each row is a measurement (V Bus 3, P Line 1-2 (from), and so on).
- Each column is a state — first every bus angle ∠, then every bus magnitude |V|.
- Each cell is one partial derivative: if this state increased slightly, how much would this measurement change?
H is a matrix of sensitivities linking measurements to states, and its structure describes the network:
- A strongly colored cell means that measurement is highly sensitive to that state.
- A blank (zero) cell means the measurement is structurally independent of that state. A flow on Line 1–2 carries no direct information about the voltage at Bus 7 on the far side of the network, since they share no electrical connection, so that entry is exactly zero.
Why the zero pattern is the topology. Each cell is a partial derivative, and a partial derivative is zero whenever the state does not appear in that measurement's equation. The question is therefore which states each equation contains. The flow on Line 1–2 is computed from |V₁|, ∠θ₁, |V₂|, ∠θ₂ and the line impedance; buses 3 to 7 do not appear in it, so changing them alters the predicted flow by exactly zero rather than by a small amount. The injection at Bus 3 is the sum of the flows leaving Bus 3, so its equation contains Bus 3 and the buses directly connected to it, and nothing beyond.
The non-zero cells in a row are therefore an enumeration of the buses that measurement is electrically connected to. Row by row, H records what each meter is attached to; taken together, the rows reproduce the network's connectivity in matrix form. The zero pattern alone, with the labels removed, would be sufficient to reconstruct which bus connects to which. The estimator exploits the same property computationally: H is sparse, and sparse solvers are built to take advantage of that.
The Focus field reduces H to the measurements that involve a chosen bus and the states they depend on. It is also where the state columns can be counted directly, including two that are not obvious: the three-winding transformer contributes a star-point angle column and a star-point magnitude column.
The reason lies in how the component is modelled. pandapower does not represent a three-winding transformer as a single element; it represents it as three two-winding transformers in a Y connection, one per winding, joined at a single auxiliary bus known as the star point. That bus exists only inside the model — it has no physical terminal and cannot be metered — but the equations treat it as an ordinary bus with its own voltage angle and magnitude. Both are genuine unknowns the estimator has to determine, inferred entirely from the measurements on the three windings around it. Those two columns take the state count from 13 to 15.
The Jacobian focused on Bus 3
Focus set to Bus 3: 17 measurements × 11 states out of the full 35 × 15, with everything structurally independent of Bus 3 removed.
One point to carry into the next section: H serves two purposes. It is the slope the solver follows to minimize J, and it also determines how an erroneous reading appears in the residuals.
Detecting bad data
This is where the redundancy described earlier becomes useful.
Because there are far more measurements than unknowns, no single reading determines the result on its own. If one reading is wrong, the surrounding measurements constrain the fit: the estimate moves part-way toward the incorrect value, but the other readings prevent it from following completely. The difference that remains is a large residual — a gap between the reported value and what the converged state predicts, a gap that would not exist if the reading were correct. Bad data therefore shows up in the residuals.
How far the estimate moves depends on how much corroboration that particular measurement has, which is precisely what the normalized residual quantifies below.
To reproduce it: run state estimation once on the clean demo, then select a bus and change one measured value — for example, raise Bus 3’s power reading from 60 MW to 80. Re-run. That measurement’s residual increases sharply and BambooGrid raises a bad-data flag.
Re-running after changing one measurement — the bad-data flag trips
Run clean, edit Bus 3’s P measurement from 60 MW to 80, re-run: “Estimation converged, but a measurement looks bad — it’s flagged red in the inspector.” The estimate lands at 74.75 MW and the measurement reaches an Rₙ of 5.12.
There are two questions here — is anything wrong at all, and which measurement is it — and the classical treatment uses two separate tests.
The classical first test
The textbook first step is a chi-square test, and the name describes how J is constructed rather than being arbitrary. Every term in J is a residual divided by that measurement's σ, then squared. Dividing a normally distributed error by its own standard deviation produces a standard normal variable, so each term is approximately a standard normal squared — and a sum of squared standard normal variables is, by definition, chi-square distributed. J is therefore a chi-square variable by construction, with degrees of freedom equal to the number of measurements minus the number of states, which is the redundancy of the measurement set.
That gives a quantitative expectation to test against. Even on a clean solve J is small but nonzero from ordinary noise, and the distribution specifies how large it should plausibly be for a given redundancy. If J comes out far larger than that expectation, something beyond noise is present. The test detects the presence of bad data without identifying which measurement is responsible.
BambooGrid does not run it. It goes directly to the second question and uses that result to answer both.
Which measurement is it?
Blaming the measurement with the largest raw residual is incorrect, for two reasons. A single gross error inflates other measurements’ residuals as well, because all of them are coupled through the shared state. And a residual of “5 MW off” does not mean the same thing on a high-σ meter as on a low-σ one. Raw residuals are not comparable across measurements.
The fix is to put every residual on a common scale. That is the normalized residual, rₙ, and it is the quantity BambooGrid displays and flags on.
What is rₙ? It is a measurement’s raw residual divided by the expected size of that residual, written Ω. Ω combines two factors: the noise level of that particular measurement, and how much corroboration it has from the others. A measurement surrounded by corroborating readings is tightly constrained, so an error in it produces a large, visible residual and its Ω is large. A measurement with little corroboration is weakly constrained, so even an incorrect value produces only a small residual, and its Ω is small. Dividing by Ω answers the comparable question: is this residual large relative to how large it would normally be for this specific measurement? The result is expressed in standard deviations, which are comparable across the network.
The connection back to the previous section: Ω is computed from the same Jacobian H. How much corroboration a measurement has is a question about how measurements and states are coupled, which is exactly what H describes. The matrix that drives the solve also governs how errors surface as residuals.
⚠️ measured − estimated is not rₙ. In the results you might see a voltage row reading: measured 1.0070, estimated 1.0074, rₙ 0.01. The subtraction 1.0070 − 1.0074 = −0.0004 is the raw residual, and it is correct — but the rₙ column reports the normalized residual, that raw difference rescaled by Ω into standard deviations. A raw difference of 0.0004 p.u. on a measurement whose expected spread is around 0.004 p.u. gives an rₙ of roughly 0.01: an essentially exact fit, well below the threshold. The division cannot be reproduced by hand from the table, because the measured and estimated columns are rounded for display and Ω is computed internally rather than shown. In short: measured − estimated gives the magnitude of the discrepancy; rₙ gives its statistical significance.
Together these give the standard Largest Normalized Residual test: compute rₙ for every measurement, take the largest, and if it exceeds 3 (three standard deviations), flag that measurement as bad. This is the whole of BambooGrid’s bad-data detection; there is no separate test upstream of it. If the largest rₙ stays below 3, nothing is flagged and the solve is reported clean. If it does not, the measurement that exceeded the threshold is the result. One test answers both questions, which is why a single measurement is identified rather than a group. In practice you would correct or remove that measurement and re-run, since eliminating the actual error relaxes the inflated residuals it caused elsewhere; the next-largest can then be assessed against a clean solve.
The flagged measurement in the inspector
The same P meter on Bus 3, changed from 60 MW to 80 — a plausible value rather than an extreme one. The estimator settles at 74.8 MW, the normalized residual reaches Rₙ = 5.12, and the measurement is marked LIKELY BAD. Exceeding 3σ is sufficient; a gross error is not required for detection.
The limit: undetectable errors
There is a real limit, and the demo can demonstrate it. Some measurements have no corroboration: remove them and the network is no longer observable. For these, Ω is effectively zero, their residual is forced to zero regardless of what they read, and their error can never appear as a residual. It is undetectable. BambooGrid marks these as critical rather than reporting a meaningless rₙ, since no other measurement is available to check them against.
Switch measurements off in the demo — each has its own toggle — until one becomes critical, and its status changes accordingly. This is the clearest demonstration of the principle underlying the whole method: redundancy is what makes errors detectable. The more corroboration a measurement has, the more reliably an error in it can be found.
Summary
- Raw measurements are noisy, incomplete, mutually inconsistent and occasionally wrong, so none is taken at face value on its own.
- The grid has a single underlying state — voltage magnitude and angle at every bus — from which every other quantity follows.
- State estimation finds the state that best explains all measurements at once, weighting each by its σ.
- It scores a candidate state with J, the total σ-weighted disagreement, and searches for the state that minimizes J.
- Because grid physics is nonlinear, J has no closed-form minimum; the estimator descends iteratively, using the Jacobian H as the local slope, in the Newton–Raphson pattern.
- Residuals expose bad data: the largest normalized residual exceeding 3σ both signals that an error is present and identifies the measurement responsible.
- The same H that drives the solve also underpins the detection, which is why one view in BambooGrid covers both halves of the method.
Open the demo, run the estimation, change a measurement, and the whole sequence is reproducible in a few clicks.
Built on pandapower’s weighted-least-squares estimator, visualized in BambooGrid.
