chc.evaluation¶
Stability tier: evolving.
Evaluating a plan from logs before it is deployed, certified before any cost is read.
chc.offpolicy.off_policy_value weights one step at a time, so it estimates a candidate's
value on the logger's own states. A feedback plan on a plant with memory moves the states, and the
value of deploying it is a property of the loop it closes. This module estimates that value, on a
linear-Gaussian loop, and first says whether the logs can:
certify_evaluationreads the model and the logger, never the logs' costs, and returns anEvaluationCertificate: the exact second moment of the weights a method needs, the effective sample size it predicts, and the margin of each condition that keeps it finite.evaluate_plancertifies, raisesInfeasibleEvaluationnaming the failing condition when the certificate refuses, and otherwise returns aPlanEvaluation.
Four methods:
"pdis"-- per-decision importance sampling over episodes, each step's weights normalised to mean one, for the expected cost over the logged horizon from the logs' initial states. The second moment of its weight obeys an exact risk-sensitive Riccati recursion along a "starred" loop (Jacobson 1973), started from the logs' own initial-state law. The certifiable horizon is the longest at whichn / E_b[W_h^2]stays abovemin_effective. A small-gain test in closed form is no substitute: it omits the initial-state law, and a loop that passes it has an infinite second moment fromh = 6when it starts from the logger's stationary law."mis"-- marginalised importance sampling on the stationary state-action laws, for the average cost per step. Its weight has a finite second moment iff2 Sigma_b^z - Sigma_pi^z > 0, which splits into a state part and an action part. The one-step gate is neither necessary nor sufficient for it: a loop for each direction is in the tests."dr"--"mis"with the model's relative value as a control variate, so that its error is the product of the weights' error and the model's."fqe"-- fitted Q evaluation with quadratic features, which is exact on this class. It is certified by the restricted chi-square over those features, evaluates a deterministic plan directly, and works where every weight has infinite variance.
A plan is an AffinePolicy, which holds for ever, or an AffineSchedule, whose gain
and offset change with the step and which ends at its horizon; a prescription's schedule is one,
open loop. Only "pdis" evaluates a schedule: "mis" and "dr" read stationary laws and
"fqe" here is average-cost, so each refuses one. The starred loop then changes with the step,
and the recursion carries the law of the state forward, which serves every horizon in one pass
whether the plan changes or not. "pdis" also scores each step against its own target when the
cost has one per state.
A deterministic plan has no density, so the weights evaluate it smoothed, N(K x + k, tau^2 I),
and smoothing moves its average cost by exactly tau^2 beta, beta = tr(R + B' P B) (over a
horizon, sum_t tr(R + B' P_{t+1} B)): up for a positive semidefinite cost, either way for one
that is not, such as a reward written as a cost. The value reported is the deployed plan's: the
model's tau^2 beta_hat is subtracted, and because its error tau^2 (beta - beta_hat) is
first order in the model's, its size is also added to the interval, times model_error. The
report names it, and its size as a share of the weighted estimate. A randomised plan -- one that
carries its dither into deployment -- is evaluated as it is, and nothing is subtracted. The
smoothing is chosen against the cost by predicted mean squared error; to design a logger, sweep its
covariance through certify_evaluation, which is cheap and reads no data.
Scope: every number here is exact for x' = a x + b u + offset + w and an affine plan, and no
more. A fitted CHC plant enters linearised; a receding-horizon LQ controller whose constraints do
not bind is an affine plan. A clipped or saturating logger, a logger whose randomisation is
confounded with the plant's noise, and a plan whose constraints bind are outside the class, and the
certificate does not see them. Stationary effective sample sizes count transitions as if they were
independent, and a slowly mixing logger has fewer; the stationary interval reads its degrees of
freedom off the batches the weights actually used, so it widens where the certificate cannot see.
The intervals are fixed-n, not anytime-valid.
What packages already do: SCOPE-RL has the sequential estimators, doubly robust and DICE among them, for gym environments, and Open Bandit Pipeline the bandit ones. Neither is maintained: no release since SCOPE-RL 0.2.1 (2023-07) and OBP 0.5.7 (2023-04). What none of them computes is the certificate, which is why this module exists.
LinearGaussianPlant
dataclass
¶
One step of the loop, x' = a x + b u + offset + w with w ~ N(0, noise).
Raises:
| Type | Description |
|---|---|
ValueError
|
on shapes that do not agree, a non-finite entry, or a |
AffinePolicy
dataclass
¶
u | x ~ N(gain x + offset, covariance); a zero covariance is a deterministic plan.
Raises:
| Type | Description |
|---|---|
ValueError
|
on shapes that do not agree, a non-finite entry, or a |
AffineSchedule
dataclass
¶
A plan that changes with the step and ends: u_t | x_t ~ N(gains[t] x_t + offsets[t],
covariance) for t < horizon; a zero covariance is a deterministic schedule. A
prescription's schedule is open loop, every gain zero, and open_loop builds one.
Raises:
| Type | Description |
|---|---|
ValueError
|
on shapes that do not agree, a non-finite entry, or a |
step(t)
¶
The plan at step t.
open_loop(actions, states, covariance=None)
classmethod
¶
u_t = actions[t] whatever the state, over states states: deterministic unless a
covariance is given.
Raises:
| Type | Description |
|---|---|
ValueError
|
on |
InitialLaw
dataclass
¶
x_0 ~ N(mean, covariance), the law the episodes start from; the covariance may be
singular, and is zero for a fixed start.
Raises:
| Type | Description |
|---|---|
ValueError
|
on shapes that do not agree, a non-finite entry, or a |
EvaluationCertificate
dataclass
¶
What the model and the logger say about evaluating a plan, before any cost is read.
log_second_moment is log(1 + chi^2) of the weights the method uses -- for "pdis"
the trajectory weight's at the logged horizon, for "fqe" the restricted chi-square over its
features -- and effective_samples is samples divided by 1 + chi^2. A margin is a
smallest eigenvalue, and its condition holds when it is positive. one_step_margin is the
one-step gate's, averaged over the logger's stationary states, and for a schedule the smallest
over its steps. It is reported for comparison only: it is neither necessary nor sufficient for
any method here, and is None when the logger's loop has no stationary law.
PlanEvaluation
dataclass
¶
A plan's estimated cost, with its certificate and what the model supplied.
value is the average cost per step (stationary methods) or the expected cost over the
logged horizon from the logs' initial states ("pdis") of the plan as given. For a plan the
weights smoothed, model_correction is the model's tau^2 beta_hat, subtracted from the
weighted estimate, with its size added to the interval's half-width times model_error, and
model_share is its size as a share of the weighted estimate's; both are 0 otherwise.
effective_samples is the weights' own, (sum w)^2 / sum w^2, beside the certificate's
prediction, and None for "fqe". degrees_of_freedom is the stationary interval's
Student t: well below 39 when a few of the logger's excursions carry the weights, and
None for "fqe" and "pdis", whose intervals are normal. From
chc.decision.Prescription.evaluate the interval is normal too, its spread read from a
bootstrap over the units the episodes came from (bootstrap) rather than across episodes;
on one unit's episodes it is Student's t with degrees_of_freedom the cosines the
episodes' influences were read across their time by, and bootstrap is None.
"pdis" normalises each step's weights to mean one, which makes its estimate a ratio, biased
at order 1 / n. normalisation_bias is that order's bias under the model, for
independent episodes, -sum_t E_b[W_t^2] (E*_t[c_t] - J_t) / n (Cochran, Sampling
Techniques, 1977, section 6.8): E*_t reads step t's state and action under the
logger's law tilted by W_t^2, J_t is the plan's expected cost at the step. A warning
is logged where it exceeds 0.05 of the interval's half-width. The estimate's actual bias can
be several times it: on one unit's windows, whose state and noise persist, it was 2 to 9
times, and 0.27 to 0.59 of the estimate's standard deviation at a horizon of 20 steps, where
the interval covered 0.63 (ADR 0095).
LoggerComparison
dataclass
¶
The plan against the policy that logged the episodes, on the same episodes.
logger_value is the logs' own mean cost over the horizon: the logger's value from the
episodes' initial states, what deploying the plan would replace. difference is the plan's
value less it. interval is read off the difference within each resample of the units, both
values computed on the same draw, so what the two share --- which units were drawn, the states
they started from --- cancels in it; two marginal intervals combined would count it twice. On
one unit's episodes it is read off the difference's influences, the plan's less each episode's
logged cost, across their time. The plan's model correction widens it as it widens the plan's
own interval.
A plan's value alone does not say that switching to it helps: the policy in place can do better than the plan, and better than any other fixed alternative a trial would compare it with (Hernán and Robins, Causal Inference: What If, Fine Points 22.7 and 22.8). The difference is the switch's own estimate.
UnitBootstrap
dataclass
¶
How the intervals of a PlanEvaluation were read: from resamples draws of the
units with replacement, every episode of a drawn unit kept, the whole analysis run again on
each draw.
refused counts the draws whose certificate refused; the intervals read the others,
accepted of them, accepted_share of the draws. A draw is refused where its units start
wider or weigh heavier than the panel's, so the draws left out are the spread's far side, and
the draws read are conditional on certification. reading says what the intervals are:
"complete"-- no draw was refused;"conditional"-- some were, at mostmax_refusedof them: the intervals read the certified draws alone, conditional on certification;"inconclusive"-- more were: the evaluation sits at the certificate's edge, the draws read are too few and too narrow, and the intervals cover less than they state.
Near the edge of panels of 60 units, the half-width was at most 3.4 % narrower than the same panel's with no draw refused while a tenth of the draws or fewer were, and 5 % to 13 % narrower past that; on panels of 30 units the coverage fell from 0.925 to 0.908 at a tenth refused and to 0.858 at two fifths (ADR 0095).
LoggerCheck
dataclass
¶
Whether the logs' actions read given and a randomisation of their own alone, tested
against columns by chc.independence.gcm_test. Experimental.
The weights read the logger's propensity given the plan's state. A logger that also read
another column before it acted, or the state's past, has another propensity, which the weights
do not see. test.p_value is the chance of a dependence this large if it read nothing else.
A pass is not a confirmation: test.detectable is the partial correlation each pair could
have shown. Nor is a rejection a measured cost. Over 400 units and 12 periods, a logger that
chased demand was flagged on every panel while the estimate's mean error stayed within 0.56 of
its standard error; one that kept half its last incentive was flagged too, and its estimate sat
1.13 standard errors high (ADR 0028).
InfeasibleEvaluation
¶
Bases: ValueError
The certificate refused: a weight has infinite variance, or too little mass is left.
fit_logger(x, u)
¶
Least-squares fit of u ~ N(K x + k, S), with a full covariance S.
Where the propensity was logged at decision time, pass that policy instead: a fitted one is a
model of the logger, and chc.offpolicy.fit_behavior_policy says what its misfit costs.
Raises:
| Type | Description |
|---|---|
ValueError
|
on |
certify_evaluation(plant, logger, plan, method, samples, *, smoothing=None, horizon=None, initial=None, min_effective=100.0)
¶
Whether samples logs of logger on plant can evaluate plan by method.
Reads the model and the two policies, and nothing the plan will be scored on. samples
counts transitions for the stationary methods and episodes for "pdis", which also needs
the logged horizon, a schedule's own by default, and the episodes' initial law. A plan
without a density is evaluated by the weights smoothed by smoothing, and by default by the
tau at which the logs support the most effective samples; evaluate_plan chooses its
own against the cost. The certificate refuses when a weight the method needs has infinite
variance, or leaves fewer than min_effective effective samples ("pdis": at the logged
horizon).
Raises:
| Type | Description |
|---|---|
ValueError
|
on an unknown method, a policy whose shape does not fit the plant, a schedule
for a method other than |
evaluate_plan(logs, plan, method, *, plant, cost, logger=None, smoothing=None, model_error=1.0, min_effective=100.0)
¶
Estimate the cost of deploying plan, from logs of another policy, if the logs can.
logs has "x" and "u": for the stationary methods one logged run, x of shape
(T + 1, n) and u of shape (T, m); for "pdis" E episodes, (E, H + 1, n)
and (E, H, m), whose first states give the initial law. plan is a policy, or for
"pdis" a schedule over the logged horizon. The cost is cost.running: one target for
every state, or for "pdis" one for each of the episodes' H + 1 states, the last of
which no step reads. plant is the model: of the loop's law for every method but
"fqe", of the relative value for "dr", and of the smoothing correction. logger is
the policy that logged, fitted by fit_logger when omitted.
For a plan without a density, smoothing is the tau the weights use, by default the one
that minimises the predicted mean squared error of the reported value, and model_error the
share of the model's correction tau^2 beta_hat counted as unverified, which the interval
carries: 1, the default, counts all of it.
The interval is a nominal 95 % one: from 40 contiguous batch means and Student's t with
Satterthwaite's degrees of freedom for "mis" and "dr", a sandwich for "fqe", and
the delta method over episodes for "pdis". For "pdis" a warning is logged where the
normalisation's first-order bias exceeds 0.05 of the half-width
(PlanEvaluation.normalisation_bias).
Raises:
| Type | Description |
|---|---|
InfeasibleEvaluation
|
when the certificate refuses; it carries the certificate. |
KeyError
|
on logs without |
ValueError
|
on logs of the wrong shape, non-finite logs, a negative |