Benchmarks¶
Two leaderboards ship with the library as code: every task, every controller and every reference it
is scored against live in chc.benchmark and
chc.causal_bench. The tables on this page were printed by the command above
each one while this site was built, at JAX's default precision, as the same command runs in a
default environment (see the dtype policy).
Control: regret against an oracle¶
Each task ships a confounded offline dataset, a true plant with a computable oracle controller, and an evaluation that scores every controller on the true plant. The point is to measure where causal control beats predictive control, and to be honest where it does not.
| column | meaning |
|---|---|
cost |
the controller's task cost on the true plant |
regret |
the controller's cost minus the oracle's, the best plan found on the true system; each task's docstring says how far from the best it is known to be |
95% CI |
percentile bootstrap over the seeds, for the mean regret |
ood |
fraction of actions outside the logged action support |
viol |
fraction of steps outside the safe state set |
seeds |
how many data seeds the row aggregates |
The delay-oscillation task runs on a single seed on purpose: its closed loop is deterministic given
the gain, and the gain grid quantises away the one seed-dependent piece, the estimated delay. The
LaLonde-DW rows are external data, scored against the randomised experiment rather than an oracle;
the files come from the Rdatasets mirror on first use and are cached under ~/.cache/chc, and the
script prints skipped in their place when it is offline.
uv run python scripts/run_benchmark.py
== pricing (confounded linear steering) ==
controller regret 95% CI ood seeds
causal-CHC -0.00 [-0.00, 0.00] 0.00 12
oracle 0.00 [0.00, 0.00] 0.00 12
predictive 13734.15 [13732.55, 13735.31] 1.00 12
== inventory (confounded demand, newsvendor ordering) ==
controller regret 95% CI ood seeds
oracle 0.00 [0.00, 0.00] 0.00 12
causal-CHC 0.00 [0.00, 0.00] 0.00 12
predictive 1.10 [1.09, 1.10] 0.00 12
== support-shift (model exploitation; pessimism vs greedy) ==
controller regret 95% CI ood seeds
oracle 0.00 [0.00, 0.00] 0.00 12
pessimistic 2.41 [2.35, 2.46] 0.00 12
greedy 7.42 [7.42, 7.42] 0.40 12
== model-uncertainty (calibrated pessimism vs greedy) ==
controller regret 95% CI ood seeds
oracle 0.00 [0.00, 0.00] 0.16 6
calibrated 13.39 [12.81, 14.06] 0.07 6
greedy 1296.93 [1232.45, 1381.23] 0.48 6
== confounding-robust (HIDDEN confounder; sensitivity radius vs certainty-equiv) ==
controller regret 95% CI ood seeds
oracle 0.00 [0.00, 0.00] 0.36 12
robust 3.10 [2.95, 3.23] 0.31 12
greedy 5.20 [4.91, 5.48] 0.48 12
== causal-dynamics (the confounding is in the plant's own control channel) ==
controller regret 95% CI ood seeds
oracle 0.00 [0.00, 0.00] 0.00 12
causal-id 0.01 [0.00, 0.01] 0.00 12
causal-iv 0.02 [0.01, 0.03] 0.00 12
mse-id 3.87 [3.52, 4.21] 0.00 12
== delay-oscillation (a delay-blind gain walks into a Hopf; single seed) ==
controller cost regret viol ood
oracle 0.57 0.00 0.00 0.00
delay-aware 0.57 0.01 0.00 0.00
delay-blind 18954.07 18953.50 0.90 0.97
== LaLonde-DW (external: recover the randomized ATE from CPS-confounded data) ==
estimator ATE ($) bias vs experiment
experimental 1794 0
naive -8498 -10292
backdoor-OLS 699 -1095
double-ML 1031 -763
Causal methods: bias against a known effect¶
Every method in the causal frontier — chc.did, chc.scm,
chc.estimators, chc.gmethods — is run on a
self-contained synthetic data-generating process with a ground-truth effect, next to the naive
built-in it is meant to beat. truth is that effect, est the method's estimate, and the last two
columns are the absolute bias of the method and of its baseline.
uv run python scripts/run_causal_bench.py
== causal frontier vs naive built-ins (bias on a known-effect synthetic DGP) ==
method baseline truth est |bias| base|bias|
Callaway-Sant'Anna TWFE 1.31 1.35 0.04 0.32
de Chaisemartin DID_M TWFE 0.50 0.53 0.03 0.49
augmented SCM plain SCM 2.00 1.92 0.08 1.03
R-learner naive OLS 1.00 1.00 0.00 1.21
g-formula naive pooled 3.30 3.31 0.01 0.80
Head to head: DCBO's dynamic SCMs¶
The two leaderboards above are this library's own. The comparison with the nearest academic method,
DCBO on its own three dynamic SCMs, is Track L of
causaldyn-bench, because running DCBO takes its
reference implementation and GPy, neither of which this library depends on:
results/track_l.md.
When not to use it says what it means for choosing a tool.
As a notebook¶
Tutorial 5 (the scoreboard) runs the pricing, inventory and support-shift tasks on one seed each and plots the regret of each controller against the oracle.