Measuring what a test establishes

Recovery: count non-completion

Suppose 1,000 trials run until a fixed deadline and only 100 heal. A 95th percentile calculated from those 100 completed trials says nothing like “95% of all trials healed by that time.” It describes a selected subset.

completion fraction = completed trials / all eligible trials
F(t) = trials that completed by time t / all eligible trials

This is an illustrative example, not an Auris performance measurement. For a fixed observation deadline, report completion fraction and both directional distributions. Unfinished trials are right-censored when modeling time to an eventual event; for a deadline-based release criterion they have not met that criterion. Do not silently remove them.

An earlier C4 gate calculated percentiles from completing trials alone. The September 5 working-source revision now records the full denominator, non-completion and conditional latency separately, and requires at least 95% of all eligible trials to heal by the chosen deadline. Focused regression checks reject mostly incomplete and entirely incomplete runs. This corrects the gate's accounting; it does not establish a universal protocol recovery bound.

A clean run is not an impossibility proof

With zero observed failures in n independent trials, a one-sided 95% binomial upper confidence limit for the failure probability is 1 − 0.05^(1/n), approximately 3/n for large n. The result assumes independent trials from the population of interest. Correlated schedules, selected seeds or a changing implementation do not automatically meet that model.

A finite randomized protocol run therefore supports behavior under its sampled distribution. It cannot prove an invariant over all adversarial schedules. A counterexample, in contrast, can refute a universal statement once the trace and model are valid.

Media scoring must reject missing evidence

A dropout detector reporting no events may mean clean audio, missing audio, failed decoding or an unsuccessful detector process. A valid score must first verify successful execution, a usable stream, a positive finite duration and the expected reference conditions.

An earlier media scorer could treat missing detector output as “100% audio present.” The September 5 working-source revision checks process success, stream metadata, decode completion and frame coverage, and distinguishes absent streams from invalid analysis. Focused regression checks exercise these failure paths. These are measurement-tool improvements, not a new phone-call benchmark or a cross-app quality ranking.

Silence detection also needs a known continuous-speech reference; normal pauses are not network dropouts. Compare the same devices, codecs, paths, impairment schedule and observation windows, and retain failed calls and missing recordings in the report.

Metadata classifiers

For an observer trying to distinguish two traffic classes, specify the classes, priors, observation window, features and train/test separation. In the balanced guessing convention:

advantage = |Pr[correct guess] − 1/2|

Some definitions multiply this by two. State the convention before comparing bounds. A tested classifier near chance does not establish that all possible classifiers fail. Reusing the test set for tuning or choosing the best-looking run understates uncertainty.

Likewise “one of k recipients” gives a success probability of 1/k only under uniform priors and indistinguishable observations. Real activity, churn, social information and repeated observations can change the posterior sharply. Cohort size alone is not an anonymity proof.

Cost accounting

Compare complete costs under the same workload: upstream and downstream bytes, framing, padding, manifests, bootstrapping, retransmissions, storage, compute and mobile energy. Distinguish decimal MB from binary MiB. A board’s record capacity is not its number of people.

The private-continuity page keeps a measured prototype wire ledger separate from a raw full-download calculation. They answer different questions until their assumptions and omitted costs are aligned.