Post hoc comparison of the best rows of two finished studies.
``holdout_compare`` takes the finished rows of two studies that already exist - the
SELECTION source (the study the user tuned on) and the HOLDOUT source (a second study
the user ran, outside this module, on data the caller says was set aside). It reads
both with the shared reader ``_rows``, joins their rows by the exact parameter vector
and reports, for the ``k`` best eligible vectors of the selection source, what the
holdout source recorded for the same vectors. Nothing here splits data, runs a trial,
writes a file, keeps state or touches a native result: the only side effect is reading
the two sources named. It is a library function; there is no command-line flag and no
field in a native result.
v1 analyses finished studies; native in-loop walk-forward comes later.
Contract ``pineforge-hpo-holdout/v1``, algorithm ``exact_join_midrank_v1``:
* Every row of both sources is keyed by ``_rows.vector_key(parameters, space)``, the
canonical key of the typed parameters of one space, which the rows and results of
both sources must share. Equality is exact: no tolerance and no lattice snapping, so
a near-equal vector is a different vector.
* A row is eligible when its status is ``ok`` and its value for the named metric is a
finite number. Every other row is excluded with one reason: ``status_<status>`` for
a status other than ``ok``, else ``metric_not_recorded`` when the metric is a name
and the row records no such key, else ``metric_unavailable`` when the recorded cell
is ``null``. A metric cell that is a non-finite number, or an ``ok`` row's cell of
another type, or a cell that cannot be written as JSON, or an integer too large for
a binary64, refuses the call.
* Eligible rows of one source that carry one vector are one vector. Their values must
all be equal; a disagreement refuses ``holdout_duplicate_vector``, and the vector is
otherwise represented by its eligible row of lowest ``trial_id`` and counted once.
When several vectors conflict, every eligible row of every conflicting vector of the
first conflicting source (selection first) is named; the other source is not examined.
* The eligible selection vectors are ranked by their value in the caller's direction,
ties broken by ascending representative trial id, so every rank is a distinct
1-based integer. The followed vectors are the best ``min(k, n_sel)`` of them. A
followed vector with no eligible row in the holdout source is reported as unmatched,
with the reason ``absent_in_holdout`` or ``holdout_row_not_eligible`` and the ids of
the holdout rows that hold it; it is never given a value of zero.
* The POOL is the eligible holdout vectors whose key matches the key of at least one
eligible selection vector, of any rank. Equal values in the pool share a mid-rank,
the exact integer or half-integer ``(p + q) / 2`` of the positions the tie group
occupies.
* For every matched followed vector the document reports the difference
``holdout - selection`` (one subtraction) and whether the holdout value is strictly
better in the direction. Over the matched vectors two statistics are reported: the
rank correlation of the matched pairs, whose two rank vectors mid-rank equal values
on each side within the matched set alone, and the share of matched vectors whose
pool mid-rank is at most ``(M + 1) / 2`` with ``M`` the pool size. Every rank,
mid-rank, term and cut is computed in exact rationals and reported as a
``[numerator, denominator]`` pair of integers in lowest terms.
``difference`` and the share are single correctly rounded binary64 operations;
``spearman`` is one correctly rounded conversion of the exact ratio followed by
one correctly rounded square root.
* ``spearman`` is ``null``, with a reason in ``spearman_undefined``, when fewer than
three vectors are matched or when one of the two rank vectors is constant. No
significance test, no p-value, no threshold and no verdict accompanies it.
* ``input_sha256`` is the SHA-256 of the sorted per-row SHA-256 digests of the compact,
key-sorted, ASCII JSON of ``[source_index, trial_id, status, value, vector_key]`` for
every row read of both sources; the metric name, the direction and ``k`` are not
covered. The document depends only on the rows, the space and the arguments.
* ``interpretation`` is the fixed text of ``INTERPRETATION``: what the comparison is
not.
* The admission figure is ``rows_read + n_sel(n_sel - 1)/2 + n_hol(n_hol - 1)/2`` over
the eligible VECTOR counts. The call is refused as ``holdout_resource_exceeded`` only
against a ``max_work`` the caller gave; a figure equal to it is admitted.
* A complete result may carry the ancestor rows of an earlier study it continued from.
The module counts them for each source and refuses ``holdout_ancestor_rows`` when the
holdout source's ancestor rows repeat a trial id of the selection source, because
such a source may be reporting the selection study's own rows; the trial id is the
only witness available, so an id that repeats by a coincidence of numbering is
refused like a real overlap.
A path is read twice to count ancestors. If the second read fails or yields a
different number of rows, the call refuses ``holdout_input_invalid`` for that
source rather than reporting zero ancestors. Parsed sources need no second read.
A refusal is raised as ``HoldoutError``; ``HoldoutError.document()`` renders it as the
refused result document (no comparison). The checks run in this order, so the first
defect decides the code: the arguments (``holdout_parameter_missing`` before
``holdout_parameter_invalid``); the two sources (``holdout_input_invalid``,
``holdout_rows_incomplete``); the space of both sources (``holdout_space_missing``,
``holdout_space_mismatch``); the direction rule; the context of each source on its own
(``holdout_context_mismatch``); the second read of each path source, selection first
(``holdout_input_invalid``); the ancestor rows of the holdout source
(``holdout_ancestor_rows``); the key of every row (``holdout_input_invalid``); the
metric name (``holdout_metric_unknown``); every metric value
(``holdout_value_invalid``); the eligible rows of the selection source
(``holdout_no_eligible_rows``); the repeated vectors (``holdout_duplicate_vector``);
the admission figure (``holdout_resource_exceeded``); and a difference that is not
finite (``holdout_value_invalid``).
1.9.8