chc.benchmark

Stability tier: experimental.

Benchmark v0: confounded, constrained control tasks with oracle regret.

Each task ships a confounded offline dataset, a true plant with a computable oracle controller, and an evaluation reporting regret vs oracle, constraint violations, and out-of-support action rate. The point is to measure where causal control beats predictive control — and to be honest where it does not. v0 has pricing (steering), inventory (newsvendor), support-shift (pessimism), model-uncertainty (calibrated ensemble) and confounding-robust (sensitivity radius under a hidden confounder) tasks, plus causal-dynamics (the confounding sits in the plant's own control channel), all in the same TaskResult / leaderboard shape.

TaskResult dataclass

One controller's score on one task.

MultiSeedResult dataclass

One controller's regret aggregated across seeds, with a bootstrap confidence interval.

PricingTask dataclass

Confounded linear steering: drive x to a target; effect of u is confounded in the logs.

The oracle is the controllers' own rule, the one-step certainty-equivalent action, at the true effect and on the same noise path, so a regret is what the estimate costs under that rule. It is not the best policy: the rule ignores control_weight, which the finite-horizon LQR policy weighs, and over 4 000 noise paths that policy costs 0.00098 less on average (standard error 0.0001), 0.0095 less on the path the task draws, in float64. A rule tuned to the drawn path, an effect of 1.03, costs 0.029 less there and 0.018 more on average, and a plan that reads the path's noise 0.29 less; neither is open to a controller. A controller can therefore land a little below the oracle on one path.

run(seed_data=0, seed_eval=1, estimator=None)

Fit the effect (oracle / causal / predictive) from logs and score each controller.

The CHC controller uses the pluggable estimator (default BackdoorOLS, the linear adjustment); pass DoubleML() / EconMLDoubleML() to swap the causal backend. The predictive baseline stays the fixed naive (unadjusted) fit.

InventoryTask dataclass

Newsvendor ordering under a confounded demand-response model (holding / stockout costs).

A fixed-intensity promo lifts demand; in the logs the promo was correlated with a demand driver z (a confounder), so the promo effect is biased. The retailer orders to a newsvendor level from its estimated demand model, so a wrong estimate systematically over- or under-orders.

The oracle orders the critical fractile of the true demand, the order of least expected cost, and is scored on the same n_eval draws as every controller. On those 5 000 draws the order at their own fractile costs 5.7e-5 less in float64, an order that reads the draws it is scored on.

run(seed_data=0, seed_eval=1, estimator=None)

Estimate demand response (oracle / causal / predictive) and score the induced order.

estimator is the pluggable causal backend for the CHC order (default BackdoorOLS).

SupportShiftTask dataclass

Model exploitation under support shift — where pessimism, not causality, is the safeguard.

A linear model matches the true plant on the offline action support, but the plant's control effectiveness collapses for large actions. The greedy controller extrapolates off-support to chase gains the model promises and stalls; pessimism keeps actions in-support and stays safe.

The oracle is the planner's descent from zero on the true plant, and it is the best plan. Past u_sat / sqrt(2) an action buys an effect a smaller one gives for less, and inside it the effect rises with the action and the action's cost is convex in the effect, while the plant is linear in the effect: there the problem is convex. The oracle's actions all lie inside, so where it stops on its rule, its stationarity 8.4e-7 in float64, is the problem's minimum. Descents from 73 starts -- zero, each side of the box throughout, at each step, and for the first 1, 2, 3, 5 or 10 steps, the other two plans and 8 seeded draws -- end on 39 plans, none below the oracle by more than 2.2e-12.

run(seed_data=0)

Optimise on the model (greedy/pessimistic) and the plant (oracle); score on the plant.

ModelUncertaintyTask dataclass

Model exploitation where the safeguard is calibrated model uncertainty, not support distance.

A residual is fit from offline data on a narrow action support; its deep-ensemble members agree there and disagree off it. The greedy controller trusts the ensemble mean and pushes into that high-uncertainty region, where the true plant's cubic drag backfires; calibrated pessimism penalises the ensemble disagreement and stays where the learned model is trustworthy -- the chc.uncertainty U term, complementing the density-distance D of SupportShift.

The oracle is the cheapest of seven descents on the true plant, from zero and from each side of the box held for the first one, two or three steps (_best_descent), searched once per task and shared by its seeds. The plant acts through u - drag u^3, which turns over at |u| = 2.58 and reaches -68.8 on the box's edge, so the best plan pushes through the reversed effect: at 3.773697 in float64 its first four actions are 8, 6.63, -8 and -6.81, and the rest stay inside the turn. The descent from zero alone stops at the effect's peak, at 15.520831, and regrets read against it were 11.75 too low. No plan found costs less: descents from 152 starts, each run to 210 000 steps and polished by L-BFGS-B, and a dynamic programme over the state, end on many plans, none below 3.773697. Every plan costs at least 3.76084, the minimum of the problem relaxed to the effects, in which the plant is linear and the action's cost is replaced by its convex envelope (tests/test_benchmark.py), so the oracle is within 0.0129 of the best plan there is. Against it the calibrated plan's regret is 12.40 and greedy's 1360.23: the row measures what planning on the learned model costs against a plan that knows the plant's turn, which no model fit on the logged support can see.

run(seed_data=0)

Fit an ensemble on the support, then score greedy/calibrated/oracle on the plant.

ConfoundingRobustTask dataclass

Control under HIDDEN confounding: the safeguard is a sensitivity radius, not adjustment.

The actuator gain is calibrated from an observational log whose action was driven by a latent disturbance that also moved the transition. It is never recorded and there is no instrument, so the gain is only partially identified -- the adjustment fix of PricingTask is unavailable by construction, which is why this task takes no estimator argument. Here the confounding attenuates the gain, so the greedy plan believes a weak actuator, over-commands, overshoots the target on the true plant, and demands actions far outside the logged range. The safeguard is chc.uncertainty.ConfoundingRobustPenalty in the lam_unc channel: the partial-identification radius times the action magnitude, bounding the transition error a confounded gain can inject per step.

HONEST TRAPS. (1) The penalty is ONE-SIDED -- it can only shrink actions, so it rescues an understated gain and strictly hurts when the confounding inflates it instead (a test pins this). (2) gamma and the cvar_gap := b_hat calibration are the analyst's inputs, not learned; the row scores the controller at the assumed sensitivity, not an oracle one. (3) With no confounding the robust controller pays a strict premium, and it only wins beyond a problem-dependent confounding threshold -- at half the default confounding it still loses. (4) This is an OPEN-LOOP plan scored on the plant; the closed-loop counterpart is chc.sensitivity.confounding_robust_tracking_benchmark.

The oracle is the planner's descent from zero on the true plant. The plant is linear, so the cost is a convex quadratic in the plan and the descent's plan is the best one: the exact minimiser over the box, by bounded-variable least squares (Stark and Parker 1995), is 6.9e-11 below it in float64, and descents from 73 starts end on the same plan.

run(seed_data=0)

Calibrate the gain on a confounded log, then score greedy/robust/oracle on the plant.

CausalDynamicsTask dataclass

Confounding inside the dynamics model, not beside it -- the chc.dynamics_id consumer.

Every other task here estimates a scalar effect and hands it to a controller. In this one the confounded object is the plant's own control channel: the log's action was chosen from a covariate that also moved the state rate, so a residual fitted by prediction error learns the observational response and the planner inherits it. Three ways in, scored on the same plant:

• mse-id -- unadjusted, which is where chc.train's prediction-error fit lands.
• causal-id -- the confounder is logged and adjusted for (the orthogonal moment).
• causal-iv -- the confounder is latent; only an exogenous action shifter is available.

HONEST TRAPS. (1) The failure is silent on the safety channels. The confounding attenuates the channel to ~0.02 of its true 1.0, so mse-id prices the actuator as useless against the control_weight penalty and gives up: measured max|u| 0.235 against the oracle's 2.516, so it reaches x = 0.135 instead of 0.712. It never approaches the box, never leaves the logged action support (99th percentile of |u| is 4.4), and reports viol = ood = 0 while conceding most of the achievable improvement. Regret is the only column that sees it -- a reminder that constraint and support diagnostics do not detect a mis-scaled channel. Shrinking control_weight flips the same bias into over-commanding, where those columns do fire. (2) The two identified rows are not interchangeable. Adjusting for a logged confounder gives regret 0.014; the instrument gives 0.131, because it explains only ~18% of the action's variance and identification rides on that share alone. Both beat the 6.20 of not identifying at all, but an instrument is a weaker substitute for the confounder than the word "identified" suggests. (3) Only the channel is identified; a_θ stays an observational-conditional drift, so this row scores planning, not forecasting. (4) The plant is control-affine by construction, which is the class the estimator and chc.plan.certify_safety share -- a general nonlinear residual gets no orthogonality guarantee and no row here.

The oracle is the planner's descent from zero on the true plant. The plant is linear in the state and the action, so the cost is a convex quadratic in the plan and the descent's plan is the best one: the exact minimiser over the box, by bounded-variable least squares, is 3.4e-13 below it in float64, and descents from 74 starts end on the same plan.

run(seed_data=0)

Fit the channel three ways from one confounded log, then score each plan on the plant.

DelayOscillationTask dataclass

A delay-blind controller walks into a Hopf bifurcation -- chc.delay's consumer.

Generic marketplace shape: an incentive moves supply, but participants respond tau later, so the plant is x' = channel * u(t - tau) -- the delay sits on the actuation path. Under proportional feedback u = -K x that closes to x' = -channel*K*x(t - tau), whose exact stability boundary is channel*K*tau = pi/2 (chc.delay.delay_margin at pole 0). Three controllers, all minimising the same quadratic cost by the same grid search, differing only in the delay they assume:

• delay-blind -- assumes tau = 0. Its optimum is the memoryless sqrt(q/r), which for the shipped weights is 3.10 against an analytic sqrt(q/r) = 3.162 -- the 2% gap is explicit Euler, whose effective decay -ln(1 - dt K)/dt exceeds K and so shifts the optimum down. The arm fails on its own terms, not because it was handed a bad gain: 3.10 * channel * tau is 1.97x past pi/2, so the loop rings up instead of down.
• delay-aware -- estimates the delay from the log with chc.irf.delay_estimate, turns the interval into a design with chc.delay.robust_delay_design, then runs the same grid search at that delay.
• oracle -- the same grid search at the true tau.

HONEST NOTES. (1) This is a row where the safety columns fire. CausalDynamicsTask documents the opposite trap -- a mis-scaled channel that concedes regret while reporting viol = ood = 0. Here the failure is loud: the blind arm leaves the safe set and the logged action support on most steps. Both traps are real; neither column is a general detector. (2) The blind arm's cost magnitude is an artefact of state_cap, which clips the diverging trajectory so the number stays finite and precision-independent. Read it as a verdict, not a quantity; the ordering and the constraint columns carry the content. (3) The delay is estimated from the rate, not the level. This plant is an integrator, so the level's impulse response is a step -- a plateau with no peak, on which delay_estimate correctly returns an interval spanning it rather than inventing a mode. Differencing turns the step back into a spike. That is a modelling obligation on the caller, not a fallback. (4) The log is sampled every observe_every plant steps, putting tau at 3.33 samples -- deliberately off the observation grid, because an on-grid delay makes the estimate exact and the row uninformative. That is also the case peak_lag(refine=True) exists for, so this task turns it on; it recovers 0.947 of a true 1.0 where the integer argmax can only say 0.900. The residual 5.3% is not noise -- it is the parabola's documented shrinkage, which for a fraction f = 1/3 predicts 0.950. (5) The estimate lands below the truth, which by chc.delay.delay_ball is the destabilising direction. It is harmless here only because that ball has 76% of relative slack; on a plant with a tighter one the same 5.3% would matter.

The oracle's gain is the best of the grid: on its 400 gains the cost has one minimum, at 0.7175, and 20 001 gains between that one's neighbours find 0.7199, 1.6e-5 cheaper, less than the 8.3e-5 the cost rises to the nearer neighbour. It is the best proportional gain to the grid's resolution; no other feedback law is searched.

run(seed_data=0)

Estimate the delay from one log, then score three gains on the true delayed plant.

BenchmarkTask

Bases: Protocol

A benchmark task: one data seed in, one TaskResult per controller out.

leaderboard(results)

Format task results as a table sorted by regret (best first).

run_multiseed(task, seeds)

Aggregate task across seeds into per-controller regret with a bootstrap CI.

task.run(seed_data=s) is called once per seed and the results grouped by controller. The interval is a percentile bootstrap over the seeds -- honest error bars on "does this controller actually win", replacing a single-seed point regret that could be luck of the draw.

leaderboard_multiseed(results)

Format multi-seed results sorted by mean regret (best first), with 95% bootstrap CIs.