chc.experiment

Stability tier: experimental.

Which experiment, if any, to run before a decision: units per zone, the regret they remove, and a verdict to deploy now, experiment first, or abstain.

The decision (ZoneDecision) is one-shot, over K zones. Zone k's lever u_k moves the zones' states through its own channel b_k, and a known coupling spreads the effect::

x = x0 + coupling diag(b) u,   J(u; b) = (x - x*)' Q (x - x*) / 2 + u' R u / 2,   lo <= u <= hi.

The channels are known up to a Gaussian posterior (ChannelPrior), and acting now on its mean costs regret: what the action loses against the one that knew b. An experiment (ZoneExperiment) sets zone k's lever to +probe_k or -probe_k at random on n_k units, each costing unit_cost_k and each adding probe_k^2 / noise_sd_k^2 to the precision of b_k. design_experiment chooses the n_k that minimise the regret left after the experiment, times the number of times the decision will be used, plus what the experiment spends, within a budget. Then it measures the regret before and after by Monte Carlo, re-solving the decision on every draw, and every number it reports is that measurement.

The local model it optimises (the lab's value-of-information thread, independently verified):

• The decision weight W = J_bu M^-1 J_ub over the free levers, M the Hessian of J in u. It is the Hessian of the certainty-equivalent regret in the channel estimate, so a posterior with covariance S leaves an expected regret of tr(W S) / 2 to leading order. It is averaged over the prior's sigma points, because at a knife edge -- q b^2 = r in one zone -- the plug-in W is zero and the regret is fourth order, not nil.
• A pinned lever, held at its bound by a positive multiplier, has no local weight. It costs regret only when the channel lies past its activation point: s G(z) / (2 S) for a multiplier z standard deviations from zero, with G(z) = E (Z - z)_+^2. What an experiment can buy there is at most delta^2 phi(z) / (S z^3) for every z > 0, delta^2 the multiplier's preposterior variance: a zone far from activation is not worth testing, and one near it is.
• The allocation starts from reverse water-filling at the price of information, unit_cost_k noise_sd_k^2 / probe_k^2, and minimises the full objective from there. With a flat prior that start is decision-Neyman, n_k proportional to noise_sd_k sqrt(W_kk / unit_cost_k) / probe_k; on the lab's four-zone market classical Neyman, which ignores W, left 1.5 times its regret.

What the local model gets wrong, and what is done about it:

• A wide prior. At a prior sd of 37-50% of b the lab's local model put the regret after the experiment 32% low. The numbers reported are Monte Carlo's whatever the model says; model_gap is how far the model was from them, and a gap above 25% is logged as a warning, because the allocation was optimised on the model.
• A channel whose sign is unknown. When a zone that matters stays within two posterior standard deviations of zero after the experiment the budget buys, and the experiment does not pay, the verdict is "abstain": the sign decides which way to act.
• Outside the model, and not checked: a response nonlinear in the lever, interference the coupling does not state, a probe that moves the state it measures, heavy-tailed noise, and a pinned lever's far bound, which the tail law ignores and so overstates what testing it buys.

A channel the logs do not identify -- chc.decision.NotIdentifiedError -- enters with the prior its owner is prepared to state, a wide variance about the best guess; the design then says how many units in which zones identifying it is worth, or "abstain" when nothing affordable is.

ZoneDecision dataclass

A one-shot decision over K zones, x = baseline + coupling diag(b) u and J(u; b) = (x - target)' state_weight (x - target) / 2 + u' action_weight u / 2 on the box lo <= u <= hi.

coupling[i, k] is how much of zone k's channel reaches zone i's state: the identity when zones do not interact, I - P when a lever pulls a share P[i, k] of its effect from its neighbours. baseline is the state with every lever at zero. action_weight must be positive definite, so every channel vector has one best action; a bound may be infinite.

Raises:

Type	Description
ValueError	on shapes that do not agree, a non-finite entry other than a bound, a state_weight that is not symmetric positive semidefinite, an action_weight that is not symmetric positive definite, or a box with lo >= hi.

cost(u, b)

J(u; b).

solve(b)

The best action for channels b, on the box.

regret(u, b)

What u loses against the best action when the channels are b.

ChannelPrior dataclass

What is known about the channels now, b ~ N(mean, covariance): a fit's estimate and its covariance, say. A zero variance is a channel known exactly, and no experiment is spent on it.

Raises:

Type	Description
ValueError	on shapes that do not agree, a non-finite entry, or a covariance that is not symmetric positive semidefinite.

ZoneExperiment dataclass

What a unit of experiment costs and tells, per zone.

A unit in zone k sets the lever to +probe_k or -probe_k at random and observes the zone's response with noise sd noise_sd_k, so it adds probe_k^2 / noise_sd_k^2 to the precision of b_k. It costs unit_cost_k, in the units of the decision's cost.

Raises:

Type	Description
ValueError	on vectors of different lengths, or an entry that is not positive and finite.

information property

The precision one unit adds to each channel.

PinnedZone dataclass

A lever held at its bound at the prior mean: how far its multiplier is from releasing it, in prior standard deviations, and the expected regret the pin costs now, times uses.

ExperimentDesign dataclass

What design_experiment recommends, and the numbers behind it.

The regrets are expectations over the prior, times uses: regret_now of acting on the prior mean, regret_after of acting on the posterior mean once sample_sizes units have run. Both are Monte Carlo's, over draws of the channels with the decision re-solved on every draw, and error is the 95% half-width of their difference. value is what the experiment removes less what it spends. regret_bound is the level quantile of the regret of acting now: with that probability under the prior, deploying now loses at most this much. model_gap is the local model's largest relative miss against the two Monte Carlo regrets, and the allocation was optimised on that model. weights is the prior-averaged decision weight W.

design_experiment(decision, prior, experiment, budget, *, uses=1.0, level=0.95, draws=10000, seed=0)

How many units to run in each zone before deciding, and whether to run any.

uses is how many times the decision will be taken on what is known after the experiment, so the regret it leaves counts uses times and the spend once. The verdict:

• "experiment": the regret the experiment removes exceeds its spend by more than the Monte Carlo error;
• "abstain": it does not, and a zone that matters stays within two posterior standard deviations of zero, so the sign of its channel -- which way to act there -- is still unknown;
• "deploy": it does not, and acting now costs regret_now on average and at most regret_bound with probability level.

Raises:

Type	Description
ValueError	on a prior or an experiment whose length is not the decision's zones, a budget that is negative or not finite, uses that is not positive and finite, a level outside (0, 1), or fewer than 100 draws.