chc.misspecification¶
Stability tier: experimental.
Whether a plan pays for its model class being wrong: two fits of one class on one log, their difference priced in the plan's own regret. Experimental.
Under a class that holds the truth, every weight on the state estimates the same parameters, and
an instrument estimates what an adjustment set does; only the variance moves. Under a class that
misses the truth, each weight estimates its own projection (see weights in
chc.dynamics_id.fit_causal_residual), so how far two fits land apart is a symptom of the
miss. Whether it matters is the plan's question: misspecification_cost prices the
difference in the regret of planning on one fit when the other's model holds, and tests it against
the law it has when the class holds -- a Hausman test in the plan's metric, whose power lies where
the plan's cost does.
The price is over every parameter, the drift's included. The drift is fitted on what the
channel leaves of the rate, so two fits whose channels differ by dB have drifts that differ by
the log's projection of dB u on the drift's features: their models differ by
dB (u - uhat(x)), with uhat the log's action projected on those features, and not by
dB u. chc.plan.CausalPlan.decision_weight prices the channel alone. On logs whose
actions averaged -1, 0 and +1 it read 0.14, 0.44 and 2.8 of the regret between the two fits'
plans, where this module's quadratic read 0.49, 0.54 and 0.49
(scripts/bench_misspecification.py, section metric).
planned_cost_bias reads the same envelope with one fit's own covariance: how far a plan's
cost on the fit reads from what it pays the plant, in expectation over the fit's error.
MisspecificationCost
dataclass
¶
What the choice between two fits of one class costs a plan. Experimental.
See misspecification_cost.
misspecification_cost(plan, reference, alternative, *, tolerance=None)
¶
What choosing between two fits of one class of one log costs plan, made on
reference's model: estimated, with a standard error, and tested against the class holding
the truth. Experimental.
With d the fits' difference in their parameters, the channel's and the drift's,
alternative's less reference's, S its covariance and W the regret's curvature in
the parameters at the plan, planning on reference when alternative's model holds loses
d' W d / 2 to second order. W comes from chc.plan.CausalPlan.decision_weight's
envelope, with the model perturbed by a residual of the fits' class inside its field, rolled
out by RK4 as the plan is. Since E[dh' W dh] = d' W d + tr(W S), the cost is read as
(dh' W dh - tr(W S)) / 2, with chc.gate.ChannelMove.price's standard error. When
the class holds the truth, d = 0 and dh' W dh is a sum of chi-squares weighted by the
eigenvalues of W S, whose tail Imhof's inversion gives: that is p_value.
S sums, row by row and state by state, the outer products of the difference between the
two fits' influences, each with its own score, so it holds whether the class does or not, and
whichever folds each fit drew; for fits given clusters, it sums each cluster's difference
first, and two-way it is whichever of the three sums the channel's error reads off has the
largest tr(W S). That needs the fits to read one log's rows, which the function cannot
check. Over 200 logs of 4000 rows each (section calibration), when the class held, the
test rejected 1.0%, 6.0% and 11.5% of them at 1, 5 and 10%, and cost_error was twice
the cost's spread. When it missed, the cost fell within 1.96 cost_error of its value on
400 000 rows in 0.915 of them: it read low by a quarter of cost_error on average and spread
1.13 times it, and noise matched the difference's realised spread to 1.1 of its standard
errors. The power grows with the miss: at 5% it caught 13%, 30%, 78% and 97% of the logs as the
channel's slope in the state went 0.005, 0.01, 0.02 and 0.03, where the logs above had 1
(section power). For fits given periods, one unit's or the units' of one declared
cluster, S reads the difference across them by the cosines the channel's error is read
with, each period's transitions summed first (ADR 0075). Read so, S is an estimate the
mixture takes as known; on one unit whose noise was drawn afresh or AR(0.7), at 5% the test
rejected 1.5% to 5.6% of 1000 logs over 50 to 1000 periods, 5 to 40 cosines, below the level
where the cosines are few, since tr(W S) there read the difference high.
On two zones of chc.zones' market, fitted under rk4 unweighted and weighted by the
first zone's supply (section zones), it rejected 0.5%, 3.25% and 7.5% of 400 logs at 1, 5
and 10% where the class held. An incentive that recruits less the more drivers are idle moves
the channel with the state: at 5% the test caught 5%, 5%, 9% and 32% of the logs as that
saturation went 0.1, 0.2, 0.4 and 0.8, while the regret between the two fits' plans went from
0.0047 to 0.0145. Trips bent as far as the harmonic law, which the class misses too, left that
regret and the test's rate where they were.
How far it reaches: as far as W does, second order in the difference. On the logs above,
where the two channels differed by two thirds of the channel's size, d' W d / 2 read half of
the regret between the two fits' plans, three quarters of it at three tenths of that difference
and nine tenths at a tenth (section reach). At a large difference the cost is a floor, not
an estimate. On the zone plant, where the difference is small, cost + noise matched the
regret between the fits' plans within 1% on average at every setting.
What it cannot see: a direction of the channel the log's actions never moved
(chc.dynamics_id.CausalDynamicsFit.unmoved) has no data in either fit's moment: both
hold the channel at zero along it, or read it off the log's rates as least squares does, which
no effect enters, and a miss along it is not theirs to see. unseen counts the directions the
plan's regret weighs, each parameter read at its scale on the log, so the count is the same in
any units of the state and of the actions. A log whose actions the covariates' features
determine, as an undithered deployed plan's own may be, leaves every direction unmoved. On a log
whose second action was always twice the first, the gate read p = 0.39 and two unseen
directions while the plan lost 0.0014 against the truth; held to the log's ratio, the plan had
none unseen and lost 5e-6 (section unseen). Nor can it see the error both fits share: on
the zone plant, the plan lost 0.38 to 0.51 against the truth at every setting, including where
the class held, 32 to 106 times the regret between the fits' plans.
Raises:
| Type | Description |
|---|---|
ValueError
|
on a fit whose channel is not identified, one made without
|
planned_cost_bias(plan, fit, *, tolerance=None)
¶
chc.plan.CausalPlan.planned_cost_bias with S the fit's own covariance: how far
plan, made on fit's model, reads its cost from what it pays the plant, in expectation
over the fit's error, and the cost corrected for it. Experimental.
The error is over every parameter the fit holds, the channel's and the drift's, in the order of
chc.dynamics_id.CausalDynamicsFit.influence: the drift is fitted on what the channel
leaves of the rate, so it carries the channel's error, and the plan pays for both. W,
J_ee and J_e are read with the model moved by a residual of the fit's class inside its
field, rolled out by RK4 as the plan is (misspecification_cost's envelope). S sums
the influence's outer products row by row and state by state, within each cluster for a fit
given clusters, two-way the read of the three with the largest tr(W S), and across the
periods' cosines for one given periods, as the channel's error is read. The fit's own
first-order bias is not known, so first_order is 0.
S holds no error along a direction of the channel the log's actions never moved
(chc.dynamics_id.CausalDynamicsFit.unmoved): the fit holds the channel there, it does
not estimate it, and the bias leaves those directions out. unseen counts the ones the
plan's regret weighs, as misspecification_cost counts them; read the bias as covering
the rest only.
Raises:
| Type | Description |
|---|---|
ValueError
|
on a fit whose channel is not identified; one made without
|