- Generated by
1.9.8
|
PineForge HPO 1.11.0
Native hyperparameter optimization for PineForge strategies
|
StudySpec is the JSON configuration consumed by pineforge-hpo run. The implemented executable subset is intentionally narrow: single_strategy, one strategy, one referenced OHLCV dataset, an expression objective, and grid, seeded-random, adaptive dlib global, or native TPE sampling.
All strategy, artifact, and OHLCV paths are resolved relative to the StudySpec file, not the caller's current working directory.
Run it with:
| Field | Executable requirement |
|---|---|
schema_version | Must be 1. |
mode | Must be "single_strategy". |
strategies | Must contain exactly one strategy object. |
datasets | The executable path requires exactly one referenced dataset. |
objective | Must use kind="expression". |
sampler | grid, random, dlib_global, tpe, candidates, or sobol, plus its candidate policy. |
execution | Sequential or threaded native execution. |
symbol_feeds | Optional index path or inline index; fixed other-symbol reads only. |
Unknown fields, duplicate JSON keys, and JSON NaN/Infinity constants are rejected.
Since 0.8.0, the root field accepts a path ("symbol_feeds": "data/feeds/index.json") or an inline index:
Resolve index paths against work.json, CSVs against the index directory, and inline CSVs against work.json. CLI --symbol-feeds FILE overrides either form (FILE relative to cwd). Keys are exact script request strings, including exchange prefixes/suffixes. No alias resolution, aggregation or network fetching occurs. D/W/M/S fold to 1D/1W/1M/1S; other timeframes are 1–5 digits of whole minutes or 1–4 digits followed by D/W/M/S, without leading zeros. Caps: 256 symbols and 256 feeds globally.
Entries contain only syminfo and/or feeds. Installed facts: tickerid (as canonical), type, timezone, session, currency, mintick; unknown fact keys are ignored. Absent/null/empty facts set nothing. Text must be nonempty valid UTF-8, at most 256 Unicode characters, without U+0000–U+001F controls. mintick is a positive finite JSON number, not a boolean. Nested syminfo catalog objects are supported.
CSV requires timestamp,open,high,low,close; optional volume and time_close; other columns are ignored. UTF-8/BOM, Unicode decimal digits and whitespace follow the release harness; a CSV field is at most 131072 characters. Prices are finite; volume is nonnegative finite or empty/NaN. Opens strictly increase; times are Unix milliseconds within ±9007199254740991. Closes are after open and at/before the next open. Missing closes default to open plus timeframe (UTC calendar months clamp to the month's last day). Session-bound feeds should provide closes. Header-only empty feeds are allowed.
Native initialization checks/hashes once, including engine setter validation. Every fresh strategy receives identical immutable facts/bars/closes before running. Index/CSV/setter errors use --symbol-feeds: and identify the symbol/timeframe before terminal trial output. Use chart-native bars (input_tf == script_tf, with bare unit aliases accepted). input.symbol dimensions remain refused (D7); fixed symbol inputs are supported. When a future manifest supplies symbol options, the HPO-side preflight contract requires a non-empty list of unique strings and preserves its declaration order. This validation does not admit symbol search while D7 is in force. Engine/codegen >= 1.1.0 input-kind metadata is required (the optional pins are v1.11.0). input.source and input.enum search dimensions are supported and cannot be symbols. String dimensions require the manifest-level "input_kind_schema": 1 marker. HPO's builder requires a valid modern codegen requests list and a separate cached one-line input.symbol canary whose input metadata actually carries kind: "symbol". The list is present even for symbol-free scripts; its presence alone is no longer sufficient. Individual input kind fields are not a capability signal. An external builder may stamp the marker only after making the same capability check, preserving all original input kinds, and binding the manifest to the exact plugin SHA-256, artifact key and codegen request identity. It must repeat those values in adjacent provenance.json, when supplied. The marker is trusted: matching forged manifest/provenance files are not authenticated. Never stamp an old artifact, infer missing kinds, or transfer the marker between artifacts. Known recorded codegen versions below 1.1.0 are refused even with a marker. unknown is not a version contradiction, but cannot replace the builder's capability check. Python precompiled artifacts cross-check the full request identity, marker, artifact key and plugin hash against adjacent provenance. Native cheaply cross-checks marker, key, hash and codegen version. Unstamped string dimensions fail closed and instruct rebuilding with pineforge-hpo >= 0.8.0 and codegen >= 1.1.0. The artifact-cache identity includes the builder's input-metadata revision so pre-stamp caches rebuild once. Unrelated duplicate titles are ignored; duplicates for searched inputs (or Python fixed inputs) remain refused. Direct native users need the adjacent manifest.json for string-valued search inputs; unreadable optional manifests do not change numeric-only native runs. Required manifest errors name the file; manifests up to 256 MiB are supported. Native warm-encode and space-info accept --symbol-feeds <index.json> as an override of the work file's path/inline index, using the same relative CSV resolution rules.
Results expose applied_runtime.symbol_feeds and runtime_sha256; terminal rows retain the record in space.symbol_feeds. Its canonicalization is pf-symbol-feed-barc-close-le-v1; the symbols object records each symbol's facts and feeds. Each feed records bars, first_ts, last_ts and source_values_sha256 (empty feeds omit bounds). Value hash: SHA-256 of pineforge:symbol-feed:barc-close-le:v1 plus NUL, then rows packed little-endian as five binary64 OHLCV values and two int64 open/close times (<5dqq). Paths do not affect identity. Warm continuation refuses changed/added/removed feeds/facts before any new trial. space_hash remains parameters/objective-only; feed identity is separate. No feeds add no feed-specific fields; TPE's numeric-identity provenance is independent of symbol feeds.
The internal selected-chart routing seam carries two identities on a bound trial row: data_context_sha256 is the study-wide canonical catalogue digest, while values_sha256 is the SHA-256 of that row's selected chart tape. The latter uses the chart-bars domain and little-endian binary64 OHLCV plus int64 open-time bytes; it is never substituted with the study digest or with a request-feed hash. This internal seam remains behind the D7 refusal until the public instrument registry and engine C ABI inputs are reviewed.
A strategy defines exactly one of:
or:
source uses the direct PineScript path:
artifact skips transpilation and compilation and is useful for an already validated plugin. It must remain beside its ArtifactBuilder manifest.json; the runner verifies the plugin hash and uses the manifest to validate every input name, type, bound, and option before execution. The Python distribution is called pineforge-codegen; the repository is pineforge-codegen-oss; the module is pineforge_codegen.
Other strategy fields:
id: non-empty identifier included in provenance;datasets: non-empty list of dataset IDs; the executable currently requires one;fixed_inputs: input titles and finite JSON scalar values applied on every trial;strategy_overrides: fixed runtime overrides applied on every trial;search_space: non-empty map of tunable Pine input titles.An input cannot appear in both fixed_inputs and search_space. Inputs with an empty title are ignored. A study that names an empty title is refused.
Candidate values and fixed_inputs are serialized and passed to strategy_set_input(). strategy_overrides are passed to strategy_set_override() before execution. This allows supported engine fields such as initial capital, commission, pyramiding, slippage, and quantity settings to change per study without retranspiling the Pine source.
The engine validates the final string representation. Invalid values produce a trial error; they must not silently fall back to Pine defaults.
low and high are inclusive integers. step must be positive.
Log-scaled integer, usable by every implemented sampler:
Log bounds must be strictly positive and step must be 1 (its default). Random, dlib global, and TPE use the transformed sampling interval [ln(low - 0.5), ln(high + 0.5)], retain observations at ln(value), and decode by rounding exp(z) to the nearest in-range integer. Grid still enumerates every integer in declaration order; log changes model geometry, not grid cardinality.
Stepped real, usable by all implemented samplers:
Continuous real, usable by random, dlib global, and TPE search:
sampler_default preserves this continuous form for random, dlib global, and TPE. Grid and both finite candidate policies require step whenever low != high. A fixed real with low == high contributes one value and does not require step.
Log-scaled real, usable by random, dlib global, and TPE search:
Both bounds must be strictly positive. A log real cannot declare step, so a non-constant log real is not a finite grid dimension and is rejected by without_replacement and exhaustive. A fixed log real remains a valid one-value finite dimension. Sampled values and native output remain in the original units even though adaptive models use z = ln(value) internally.
Grid order is false, then true.
choices must contain at least one finite JSON scalar. Choice order is significant for deterministic grid enumeration. String, integer, real, and boolean choice types are preserved in native trial JSON and must match the compiled Pine input type. Color-typed manifest inputs accept only packed 0xAARRGGBB integers in 0..4294967295 (excluding booleans), while legacy string-typed color inputs retain string values. Integer dimensions over color inputs support log: true with the same positive bounds and step of 1 required for ordinary integer inputs; this samples the packed numeric word.
Conditional search spaces are not implemented. log is valid only on integer and real dimensions under the contracts above.
without_replacement and exhaustive require an exact finite Cartesian domain. The legal values of an integer or varying stepped-real dimension are
high is an inclusive upper bound but belongs to the lattice only when some integer k reaches it. For example, low=0, high=1, and step=0.3 yield 0, 0.3, 0.6, and 0.9, not an extra off-step endpoint. Fixed real dimensions contribute one value, booleans contribute two, and categorical dimensions contribute their declared number of choices. The domain cardinality is the checked product of these per-dimension counts; overflow or an undefined count is an initialization error for a finite policy.
The binary64 codec evaluates a real grid point with one fused multiply-add. It snaps only a final decoded value no more than one representable value above high; this makes decimal 0..0.3 step 0.1 contain the declared 0.3 while leaving 0..1 step 0.3 off-lattice. A stepped real may contain at most 2^53 ordinals. Construction rejects any real or categorical grid whose distinct ordinals serialize to the same strategy-ABI value.
Uniqueness applies to the complete parameter vector, not to individual fields. The coordinator reserves the vector before returning it to a worker, so pending requests participate in duplicate prevention.
The current run command accepts one dataset. The CSV loader expects normalized OHLCV rows with timestamps compatible with PineForge's public bar contract.
id: unique non-empty ID referenced by the strategy;ohlcv: CSV path relative to the StudySpec;ohlcv_bin: alternative binary chart path relative to the StudySpec; exactly one of ohlcv and ohlcv_bin is required;verify_bin: full (default) or sidecar; requires ohlcv_bin and is forwarded as --verify-bin;input_tf: chart/input timeframe passed to run_backtest_full();script_tf: script timeframe; defaults to input_tf in the loader;chart_timezone: chart timezone; defaults to UTC.ohlcv_bin is forwarded as --ohlcv-bin to the native runner. It maps a regular file read-only (PROT_READ, MAP_PRIVATE) and passes the mapped records directly to the engine. The file has no header: each 48-byte record is little-endian <5dq (open, high, low, close, volume as binary64, then signed int64 bar-open Unix milliseconds). A big-endian host is refused. The file must contain 1 through 2147483647 records and its appended <file>.json sidecar must be a regular file of at most 1 MiB containing a JSON object with:
format: "pf_bar_t/v1";record_size: integer 48;symbol: a string, including an empty string;tf: a string matching the whole canonical value [1-9][0-9]{0,4}|[1-9][0-9]{0,3}[DWMS];count: integer equal to file size divided by 48;first_ts and last_ts: integers equal to the mapped signed int64 timestamps at byte 40 and 48*(count-1)+40;sha256: exactly 64 lowercase hexadecimal characters.Booleans are not integers. Symbol and timeframe are descriptive and are not compared to the run configuration; extra metadata keys are allowed. Metadata or digest mismatches are refused as sidecar (hpo_dataset_invalid, with the engine's cause text sidecar does not match the bar/close files) before a Dataset is returned. full is the default and compares SHA-256 of the entire mapped file, without a prefix. sidecar checks all metadata and digest shapes but skips the digest comparison. Both modes scan every record: OHLC prices must be finite, volume nonnegative or NaN, and timestamps strictly increasing. In-place rewriting or truncation of a mapped file is unsupported. Publish a new file by write-then-rename/os.replace, keeping each mapped inode immutable for the dataset's lifetime. The native runner keeps the original descriptor open and compares st_dev, st_ino, st_size, st_mtim and st_ctim against their load-time values at the end of the run, including the timeout path, before rendering results. The mtime comparison uses the filesystem-stored tv_sec/tv_nsec fields, so its detection resolution is the filesystem's timestamp granularity: the mtime comparison alone cannot see a rewrite that leaves the stored mtime unchanged. A change to device, inode, size or mtime is refused as hpo_dataset_invalid with a changed while mapped diagnostic. When only ctime moves (chmod, chown, xattr, an extra hard link, or rename-over), the bytes in the held mapping are re-hashed against the sidecar digest parsed at load and accepted only if equal; without a digest a ctime change is refused. Rename-over hashes the original mapped inode, without reopening the replacement path. In sidecar mode the bytes were not compared to the digest at load, so even a metadata-only change can refuse if the re-hash finds that the bytes never matched it. The run's results are computed only from the bytes verified at load: a change that the mapped bytes show is refused (the held mapping is re-hashed against the sidecar digest when ctime moved). On platforms where a private read-only mapping does not show later in-place writes (observed on macOS), the run keeps using the original verified bytes and is accepted. This also applies to a same-size rewrite followed by exact mtime restoration (touch -r, os.utime with ns=): POSIX writes and mtime updates advance ctime. Residual: a rewrite that is fully restored (content and mtime) before the end-of-run re-hash can go undetected if trials read the changed pages in between; the guard covers changes still present at the end of the run, and the scenario needs a deliberate local writer. Detection resolution is the filesystem's timestamp granularity: a filesystem with coarse ctime can miss a rewrite within one tick that keeps size, mtime and ctime unchanged. Truncation can cause an unrecoverable SIGBUS before this end-of-run check, terminating the process without failure JSON. The CSV loader is unchanged. The native runner refuses --trials-file, --output or --scheduler-stats paths that alias the binary chart or its sidecar with hpo_cli_usage. --verify-bin requires --ohlcv-bin. Optional symbol_feeds remain CSV and can accompany a binary chart.
Multi-dataset train/validation/test roles and walk-forward folds are not executable yet. A post hoc analysis of finished per-window studies exists as a library function (pineforge_hpo.walkforward); v1 analyses finished studies; native in-loop walk-forward comes later.
The executable objective is an expression:
direction is maximize or minimize.
Supported expression syntax:
+, -, *, /, unary +, unary -;min(a, b), max(a, b), abs(a);<, <=, >, >=, ==, !=.Comparisons evaluate to 1.0 or 0.0. Constraint expressions are satisfied when non-zero.
Metric lookup supports:
metrics.all.<trade-stat>;metrics.longs.<trade-stat>;metrics.shorts.<trade-stat>;metrics.equity.<equity-stat>;returns.{bar,monthly}.* names of Return statistics;report.total_trades, report.net_profit, report.input_bars_processed, report.script_bars_processed, report.magnifier_sample_ticks_total, and their short aliases (without the report. prefix).<trade-stat> and <equity-stat> are the field names of the engine's pf_trade_stats_t and pf_equity_stats_t. Engine 1.0 renamed two equity fields, and both spellings are accepted:
| Name | Pre-1.0 alias | Value |
|---|---|---|
metrics.equity.sharpe_monthly | metrics.equity.sharpe_tv | Sharpe ratio of month-end equity returns (chart timezone), 2%/yr risk-free rate, sample standard deviation, annualized by √12 |
metrics.equity.sortino_monthly | metrics.equity.sortino_tv | Sortino ratio of the same monthly returns, population downside deviation against the same risk-free rate, annualized by √12 |
Both are NaN with fewer than two monthly returns or zero deviation. For Sortino, zero downside deviation means no monthly return fell below the risk-free rate, so the strongest trials can be NaN. Under the default nan_policy (fail_trial) a NaN metric fails the trial.
Use the 1.0 names in new studies. An alias resolves to the same value, so a StudySpec written for an earlier engine runs unchanged; the aliases are also the keys of these two ratios in the engine's JSON report. A result lists each metric under the name the expression used.
Missing/non-finite metrics and division by zero follow the configured reject policy and leave the trial auditable as an objective or constraint error. An unknown metric fails before the first backtest. Constraints always reject division by zero and non-finite operands; a non-finite final objective is never feasible or rankable even when non-finite intermediate objective evaluation is enabled.
The JSON loader recognizes a future kind="registered" shape, and the C++ library exposes ObjectiveFn<Observation> for custom functions. The executable CLI does not resolve registered objective names yet.
Eighteen metric names, returns.{bar,monthly}.{count,skipped,periods_per_year,mean,std, sharpe_per_period,skew,kurt_raw,status}, can be used in an objective or constraint expression or with --record-metric. They are computed from the engine's own ordered (time_ms, equity) curve while the report is still alive, and only for the series an expression names: with no returns.* name nothing is reduced, no return_stats object is written and every byte of the result is what it was before. Contract: pineforge-hpo-return-stats/v1, legacy mode only (the curve is used as the engine produced it; no anchor is added).
| Series | Returns | Period basis P |
|---|---|---|
bar | simple returns between adjacent curve points | (n - 1) / span_years for n >= 3 and a positive span, span_years = (last_time - first_time) / (365.25 * 86400 * 1000) |
monthly | simple returns between the last points of consecutive UTC calendar-month buckets | 12 whenever the series can be formed |
A return is current / prior - 1 in binary64 when the prior equity is positive; a prior at or below zero increments skipped. count is the number of valid returns and does not depend on P. The risk-free rate is fixed at 0.02 per year (0.02 / P per period), std is the sample deviation, skew and kurt_raw are population-moment values (a normal distribution gives 3), mean is raw and sharpe_per_period is the excess ratio, not annualized. Multiply by sqrt(periods_per_year) to compare it with the engine's sharpe_bar or sharpe_monthly; the two agree on the common valid domain (finite equities, positive prior equity, n >= 3 with a positive span for bars, at least two returns and a positive deviation), within floating-point summation order.
An undefined field is JSON null. status is one number per series: 4 an input equity is not finite, 7 monthly for a chart timezone other than UTC (bar is unaffected; every monthly field is null), 8 a derived return, sum, moment or ratio is not finite, 6 the period or time basis is unavailable, 1 fewer than four valid returns, 2 zero variance, 0 otherwise. **status is 0 if and only if every other field of the series is finite**, so a fully finite series with skipped returns or a tiny variance has status 0 and a consumer decides what to do with it. A trial without a report (an engine error) has every requested metric null, status included. The chart timezone is the request's own --chart-timezone; nothing changes process timezone state.
When any series is requested the result carries one object:
series lists the requested series. numeric_build_identity names the build that computed the statistics (reducer source digest, compiler identity and version, translation-unit compile flags and the contract string). It is the same value on the grid, random, TPE and candidates paths, is unrelated to the TPE checkpoint identity, and names a build rather than attesting its arithmetic: equal arithmetic across builds or architectures is claimed only for the pairs a proof tested. There is no new per-trial key; the values are entries of each trial's metrics object under the spelled names. A part reports only its own trials: warm-started ancestors keep their rows exactly as they were, and binary parents carry no statistics. This feature does not decide which trials a later selection or deflated-Sharpe step may count, how constraints or pruning treat these metrics, or how parts merge.
A build whose statistics identity cannot be bound exactly, because it used a multi-configuration generator, a compiler launcher or wrapper, response files, a disabled compilation database, a CMake older than 3.19 or another listed situation, still builds and runs every ordinary study with unchanged results. A run that names a returns.* metric is refused before the plugin, the dataset, any output file or any trial, with the registered hpo_toolchain_unavailable failure (args.reason is native_runner); its diagnostic text names the exact unbound reason. An empty identity is never published. If a build fails the reducer's own contraction probe, the run stops at start with hpo_invariant.
Every sampler requires an explicit non-negative seed and positive trials. The optional candidate_policy defaults to sampler_default. Only tpe (typed model options), candidates (candidates_file) and sobol (scramble) accept a non-empty config; other sampler kinds reject it instead of silently ignoring configuration.
| Value | Supported samplers | Validation and behavior |
|---|---|---|
sampler_default | All | Preserve native sampler behavior. Continuous real dimensions and repeated adaptive proposals remain possible. |
without_replacement | tpe, grid | Require a finite domain, reserve every complete vector at most once, and require trials <= domain cardinality. |
exhaustive | tpe, grid | Require a finite domain and trials == domain cardinality; every declared complete vector is proposed exactly once. |
candidate_policy is a sibling of kind, seed, trials, and config:
The finite policies reject random and dlib_global; they never silently change those samplers into grid or TPE. sampler_default is backward compatible with existing continuous TPE studies.
Grid search enumerates the Cartesian product in stable order. trials caps the number executed; a sufficiently large value exhausts the grid. seed is retained for a uniform reproducibility/provenance shape but does not alter grid order. without_replacement makes the already-unique grid contract explicit and requires the cap not to exceed cardinality. exhaustive requires it to equal cardinality.
Random search requires an explicit non-negative seed and a positive trial count. The native sampler uses a deterministic mt19937_64 sequence and avoids implementation-specific standard distributions. It supports only candidate_policy="sampler_default".
dlib_global wraps dlib's global_function_search through an adaptive ask/tell interface. The seed must be between 0 and 2147483647, inclusive. A fixed logical batch is proposed and reported in trial-id order with the selected fixed feedback lag. The exact deterministic sequence is scoped to the same seed, batch size/lag, dlib version, and native toolchain, independently of worker count.
dlib_global supports only candidate_policy="sampler_default".
Integer and stepped-real dimensions use integer indices; booleans use 0/1; categorical choices use their declaration-order index. A one-value dimension is removed from dlib's vector and restored in the candidate. Categorical indices impose an artificial ordinal geometry, so this is not equivalent to categorical TPE. Duplicate parameter sets may be proposed and are evaluated as separate trials.
dlib always maximizes internally; minimize objectives are sign-inverted. Failed and infeasible requests are abandoned because dlib has no native hard- constraint contract. Only genuine feasible objective values train the model or participate in reported best-trial selection. A future constraint-aware sampler may add explicit violation magnitudes instead of fabricated objective scores.
candidates evaluates every occurrence of an ordered list of complete parameter vectors, once each, through the ordinary trial path. It is the native --sampler candidates --candidates FILE. Trial ID i is the zero-based occurrence position i. Duplicates are kept: the same vector on two lines runs twice and yields two rows with two IDs; nothing is deduplicated, cached or fanned out.
pineforge_candidates_v1: UTF-8 JSON Lines, one flat object per line whose keys are exactly the search-dimension names and whose values are scalars. No header, comment or blank line; one final LF is optional and CRLF is accepted. A number is read by the warm-start importer's rules (an integer dimension needs an integer token, a real dimension accepts any JSON number, a number may match a real categorical choice, a finite space requires the exact lattice value). Duplicate JSON keys, NaN/Infinity, invalid UTF-8 and nesting over 32 are refused. A fixed input may not share a name with a search dimension.sampler.config.candidates_file is required and is the only config key. A relative path resolves against the StudySpec file's directory, like every other study path. Python only checks that the file exists; the native runner reads and validates it.sampler.trials stays a required positive integer and must equal the number of lines N the native runner admits; a different value is refused before any trial. The native CLI accepts --max-trials absent (N is used) or exactly N; an explicit --max-trials 0 is refused like any other mismatch (N is at least 1), whereas zero keeps its "no cap" meaning for the other samplers. seed is required and recorded; it never changes a value.hpo_input_file_invalid (path, file type, size, line length, UTF-8, JSON), hpo_study_spec_invalid with reason sampler (empty list, count, budget), search_space (a vector against the space) or input (fixed-input overlap), and hpo_cli_usage. A diagnostic names a 1-based line number and a declared dimension name at most; it never contains the path or a list value.sampler_default, a pruner other than none, a positive --no-improvement-trials (zero is accepted), and --warm-start (exit 4, hpo_warm_start_rejected). There is no resume for a candidate list in this release and no Python-side rewriting or merging of histories.stop_reason: "trial_budget_reached" at N, even when it happens to cover a whole finite space.candidate_list** is a top-level object present only for this sampler: {"format","source_sha256","list_sha256","count","evaluated","scored","complete", "unevaluated_ranges"}. source_sha256 hashes the exact file bytes; list_sha256 hashes pineforge_candidates_v1\n followed by the canonical key of each vector plus \n in order, so whitespace, key order and number spelling do not change it (and a zero's sign does not either: the original bytes are what source_sha256 distinguishes) while duplicates and order do. evaluated counts every position with a terminal row, scored only ok and constraint_violation, and unevaluated_ranges lists the missing positions as ascending inclusive [first, last] ranges. **complete means every position has a terminal row, not that every score is valid.** Coverage is kept for trials_out all, best-k and none and after a timeout, so consumers must test candidate_list.complete, never stop_reason alone.trials_out=all, its --trials-file, or a binary history made from them is an ordinary parent for the existing loader: the child reports warm_start_model: "rebuilt_history", never an uninterrupted checkpoint, and the usual space-hash, status, capacity and size refusals apply. The space hash does not certify the dataset, window or fixed inputs, so the caller must use the same scoring context; native cannot refuse a wrong-window parent until the window is recorded in result provenance.sobol evaluates a low-discrepancy sequence: the 64-bit Gray-code Sobol sequence over the varying dimensions, with direction numbers from the Joe and Kuo criterion-6 table (rows for dimensions 2 to 1024, dimension 1 implicit; see NOTICE). It is native --sampler sobol [--sobol-scramble digital_shift|none], and sampler.config.scramble is the only config key (default digital_shift). No quality or speed advantage over another sampler is claimed.
low < high, every Boolean, a categorical with two or more choices) sorted by byte-wise UTF-8 name are the Sobol columns; at most 1024. A constant takes no column, and declaration order is irrelevant.n counts from 0 and includes index 0; nothing is skipped or thinned. With digital_shift every column's 64-bit word is XOR-ed with a SplitMix64 word keyed by seed (an unsigned 64-bit integer, kept exact as a decimal string in results); none ignores the seed. A discrete dimension takes the high word of the exact 128-bit product of the word and its count, on the same lattice as the finite-space candidate decoder; a continuous one takes the 53 high bits as a unit. Linear, log and stepped reals and log integers follow the documented binary64 rules; a log integer needs 1 <= low <= 2^52 and high - low + 1 <= 2^53.stop_reason: "trial_budget_reached". Normal budget, wall-limit, patience, pruner, cancel and timeout semantics apply.hpo_study_spec_invalid, reason sampler, exit 1): more than 1024 varying dimensions, an integer range with 2^64 values, a log-integer range outside the limits above, an invalid scramble. A candidate policy other than sampler_default, neither trials nor a wall limit, and a scramble given with another sampler are usage errors. The existing uint64 trial-ID rules stay explicit and no new ceiling is added: the largest ID issued is 2^64 - 2.portable-sobol-v1, its own component, distinct from the TPE checkpoint and the return-statistics identities) is bound, in a portable floating-point environment. Otherwise the run is refused before any plugin, dataset or trial work with hpo_toolchain_unavailable (reason native_runner, the exact unbound reason in the text) or hpo_portable_math_unavailable. A space of integers, Booleans and categoricals has numeric_build_identity: null and is always available.sobol object, present only for this sampler: {contract, table{name, subset_sha256, upstream_sha256}, word_bits, scramble, seed, columns, numeric_build_identity, mapper{contract, revision}, identity, first_index, next_index, exact_stream}. seed, first_index and next_index are decimal strings (seed is null for none). identity is the SHA-256 of the canonical JSON of the descriptor (including the space hash, the numeric identity and the mapper revision); next_index is one more than this part's largest terminal trial ID and counts every terminal trial whether or not its row is retained (trials_out best-k and none included, and a timed-out or fatal result too).--warm-start accepts only a complete Sobol result (trials_out all, at most 256 MiB, earlier parts in warm_start_trials) whose identity equals the new run's. Rows-only files, arrays, JSONL, binary history, results of other samplers, summaries and any identity mismatch are refused with exit 4 (hpo_warm_start_rejected), with no fallback mode and no new checkpoint format. Every parent row, ancestors included, must equal the generator at its trial ID, and the claimed indices and exact_stream are recomputed from the rows: a claim that contradicts them is refused in either direction. An unsorted row array is accepted (row order has no meaning), a repeated ID is not, a parent with no row at all is refused, and a parent whose largest ID is 2^64 - 2 or whose budget would pass 2^64 - 2 is refused. The child starts at the parent's largest ID plus one and never refills a hole. The generic loader's own limits (document size, subnormal parameters, capacity) still apply and are not relaxed; this is not a claim that arbitrary history continues exactly.t holds the point fixed by the run identity and t alone, whatever the workers, batch size, lag, failures or other rows. (E) If the parent rows are exactly IDs 0 to m - 1 and the identities match, the continuation's rows equal rows m onward of an uninterrupted run (parameters only; exact_stream is true). A gapped, cancelled or timed-out parent keeps only point-by-ID meaning (exact_stream false). Whole result objects, counters, the best trial, stop_reason and early_stop are not claimed to concatenate. With a fixed batch size and lag and no external stop, repeated runs and any worker count give identical result bytes; with patience enabled the terminal frontier follows the configured drain, so the stream claims are qualified by those fixed replay inputs. Equality across architectures is claimed only for host pairs a proof tested.tpe is a native, single-objective Tree-structured Parzen Estimator sampler. It fits independent per-dimension marginals to good (l) and bad (g) completed observations. Numeric dimensions use bounded Gaussian mixtures; integer and stepped-real values use quantized bin-mass likelihoods. Boolean and categorical dimensions use prior-smoothed categorical probabilities. It draws ei_candidates joint candidates from l and selects the largest summed log(l / g) score.
The good-set target is:
and is clamped to leave at least one bad observation when the history permits. Before startup_trials usable observations, proposals use the seeded random path. Failed and constraint-violating trials are abandoned and train neither estimator.
sampler.config.history_switch is absent or null by default, meaning never switch: TPE uses exact 0.3.0 full history throughout the study. A positive uint64 maps to native --tpe-history-switch N and explicitly opts into bounded TPE. Below that completed usable observation count, TPE uses the exact 0.3.0 full-history estimator. At the count, it switches to bounded density tables with 25 elites, 64 recent non-elites, and 448 seeded older non-elites by default. This count is independent of worker count, issued/abandoned candidates, and wall time. N=1,000 is a reasonable opt-in for 10k–1M studies, trading refinement quality for history-flat acquisition. Measured median-regret-ratio geomeans are 1.036492 at 3k versus 0.3.0 and 1.031570 at 10k versus the full-history benchmark variant. A larger override has higher memory and acquisition cost before the switch: legacy 16D snapshot asks measured 27,652 microseconds at 1,024 observations and 112,201 at 4,096. Results and TPE terminal records persist the configured value as tpe_history_switch; it is not a trial or wall-time cap.
Under sampler_default, TPE may propose the same complete vector more than once. Under either finite policy, every vector is atomically reserved before it is returned by ask(). Pending, completed, infeasible, and failed vectors stay in the seen set. An abandoned point remains ineligible for a later proposal even though it does not enter either density estimator.
Because expression constraints currently return only pass/fail, TPE does not learn the direction or magnitude of a violation. Studies with a tiny feasible region should encode validity into the search space where possible; otherwise the sampler may revisit infeasible regions. No fabricated penalty objective is inserted automatically.
TPE config fields are strict; unknown fields are rejected:
| Field | Default | Requirement |
|---|---|---|
startup_trials | 10 | Positive integer. |
ei_candidates | 24 | Integer in [1, 1000000]; the upper bound prevents an accidental near-infinite ask(). |
history_switch | null | Never switch when absent/null; positive uint64 opts into bounded models via native --tpe-history-switch N. |
max_threads | 0 | 0 uses min(8, available CPUs); 1..1024 sets a fit/score worker cap, also constrained by affinity/cgroup quotas. Native --tpe-max-threads N. Never changes suggestion bits. |
gamma_fraction | 0.1 | Finite number in (0, 1]. |
gamma_cap | 25 | Positive integer. |
prior_weight | 1.0 | Finite number greater than zero. |
constant_liar | true | Boolean. |
candidate_policy is not a TPE estimator setting and therefore belongs beside config, not inside it.
When constant_liar=true, parameters from outstanding requests are inserted only into the bad estimator. This discourages duplicate concurrent proposals without fabricating an objective value. With the default batches schedule the scheduler still reports completed results in trial-id order, so reproducibility is scoped to the same seed, batch size/lag, pruning policy, sampler config, and native build. A rolling study is not reproducible by running it again; see Rolling schedule.
TPE retains and refits the complete usable history. It is intended for bounded adaptive refinement, not 10^6-10^8 trial enumeration; use an external partitioned stateless search to reduce a mega-scale domain before starting a TPE study.
CMA-ES, evolutionary sampling, durable study storage, and multi-objective Pareto optimization are not currently available.
Threaded execution:
Sequential execution:
Every worker shares the immutable dataset and loaded plugin function table, but every trial creates and destroys its own strategy handle and report. A processes value is reserved in the schema for future cross-plugin portfolio execution; the current single-strategy CLI does not launch process workers. timeout_seconds and fail_fast are also reserved and currently rejected so they cannot appear to take effect without real cancellation semantics.
| Execution field | Default | Contract |
|---|---|---|
schedule | "batches" | "batches" (the default, byte-identical to 0.12.0) or "rolling" (a free worker slot is refilled at once from whatever has finished; the result is NOT byte-replayable and carries replay_contract: "rolling_completion_v1"). See Rolling schedule. |
batch_size | workers | Positive integer, at most 1,000,000. Proposal size, not thread count. Used by batches only; setting it with schedule: "rolling" is refused. |
batch_lag | 0 | 0 commits b before proposing b+1; 1 proposes b+1 from history through b-1. Used by batches only; setting it with schedule: "rolling" is refused. |
pruner | "none" | none, median, or halving. |
pruner_rungs | [0.25, 0.5] | Strictly increasing fractions in (0, 1); the full window is implicit. |
pruner_eta | 2 | Integer >= 2; minimum finite history at each rung. Halving retains the best ceil(n/eta); eta=2 uses the median. |
The native equivalents are --schedule, --batch-size, --batch-lag, --pruner, --pruner-rungs 0.25,0.5, and --pruner-eta. Median always uses the median; its eta only selects history warmup. --scheduler-stats FILE writes timing diagnostics outside the fingerprinted result and has no StudySpec equivalent.
With the default batches schedule, for fixed provenance and explicit batch size, completed trials[] reproduce byte-for-byte with 1, 2, 4, or 8 workers. Without an explicit size, changing workers intentionally changes the default logical size, as in 0.1.x. Lag-one TPE requires constant liar enabled. Cancellation/deadline/timeout truncation and progress-line arrival order are not deterministic replay inputs.
Pruning reruns the exact engine batch path on ceil(fraction * input bars), deduplicating prefixes that round to the same count. Cuts are frozen from committed earlier batches at that rung; ties survive. Under the rolling schedule a trial's cuts are taken from the rung scores of exactly the trials in its logged prefix, and the pruner is fed once per fed-back trial in completion_order. Rung scoring uses the study objective, and constraints apply only on the full window. A pruned trial keeps partial metrics and objective, is not feasible or a best-trial candidate, and consumes one trial from the requested budget. It is abandoned rather than used to train the adaptive sampler.
Pruning adds a pruning trial object with method="prefix_rerun", rungs_completed (prefix rungs only), cut, bars_processed_total, script_bars_processed_total, and magnifier_sample_ticks_total. These totals include all reruns, whereas backtest describes only the last executed window. Use the cumulative totals for compute metering, including on surviving trials. Pruned trials remain billable trials unless the consuming application explicitly chooses another policy. Progress lines use exactly the same terminal object.
The native --trial-timeout-seconds watchdog spans all prefix reruns in a trial, not each rung separately. Surviving the default quarter/half/full schedule requires roughly 1.75 times the full-window bar work; account for that when choosing a timeout for a pruning-enabled study.
Pruning and pipelining stay opt-in: changing feedback or cutting prefixes can change the best found result. See measurements and limitations.
Since 0.14.0, execution.schedule chooses how trials are scheduled. It takes "batches" or "rolling", and the default is "batches".
"batches" is the default and stays byte-identical to 0.12.0. A study that 0.12.0 accepts, and that leaves schedule unset or sets it to "batches", produces the same trials[] bytes as 0.12.0 and the same batch_size, batch_lag, replay_contract (ordered_batches_v1) and continuation_contract values, and its result gains no rolling field. The result also records the version of the build that produced it in pineforge_hpo_version; that value names the running release and is not part of this comparison."rolling" keeps every worker busy. The runner first proposes up to one trial per worker, with nothing finished yet. Whenever a trial finishes, the coordinator feeds its result back to the sampler, if the sampler takes feedback (a tell for a feasible trial with a finite objective, an abandon for any other trial), and at once proposes a new trial into the freed worker slot, from whatever has finished at that moment. It does not wait for a batch to end and it does not wait for earlier trial ids to finish. Proposals continue until the trial budget, the search space or a stop condition ends the study.A rolling study's result is NOT byte-replayable. Running the same study again can give different trials[], a different completion_order, and, when a patience rule or a time limit stops the study, a different number of trials. The order in which trials finish depends on the worker count, the load on the machine and how long each trial takes, and that order decides what each new proposal sees. The result says so itself: its replay_contract is rolling_completion_v1, never ordered_batches_v1. A result that names ordered_batches_v1 may be relied on for the byte-replay rules in this document; a result that names rolling_completion_v1 may not, and nothing promised for ordered_batches_v1 applies to it.
A rolling result does carry a log from which every proposal can be audited and re-derived:
completion_order (study level) lists the ids of the finished trials, each once, in the exact order the coordinator fed them back.completed_before (study level) is an array indexed by trial id: completed_before[i] is the number of entries of completion_order that had been fed back when trial i was proposed. The proposal saw exactly the first completed_before[i] entries of completion_order, in that order, and nothing else; a trial that was proposed earlier and was still running is not in that prefix. The completion log – completion_order and completed_before – is complete whatever trials_out says, and a per-trial record carries neither.For example, with two workers a run might finish trials 1, 0, 3, 2, 5 and 4, in that order. Its completion_order is [1, 0, 3, 2, 5, 4], and completed_before is [0, 0, 1, 2, 3, 4] (indexed by trial id, trials 0 to 5). Trial 3 was proposed after trials 1 and 0 had finished, in that order; trial 2 was still running, so trial 3 did not see it.
The acceptance gate for rolling reads only the result file. It does not compare bytes between runs; it audits the log and replays the proposals from it:
completion_order appears once and is a terminal trial of this run; every id in a trial's prefix is smaller than that trial's own id; and completed_before never decreases as the trial id increases.completion_order that its prefix adds, as a tell or an abandon exactly as above, then ask for the next candidate. The candidate must reproduce the parameters recorded for that trial. Replay needs every trial's recorded outcome, so run it on a result written with trials_out set to all, which is the default.Replaying the log re-derives the recorded proposals; it does not make a second run of the study produce the same bytes.
A study that uses the tpe sampler with constant_liar set to false is refused when schedule is "rolling", as it is with batch_lag: 1, because every rolling proposal is made while other trials are still running.
The statement above that a rolling study does not wait for earlier trial ids to finish has one qualification: with --progress-fd or --trials-file, finished trials are written in trial-id order, and a trial that finishes while twice the worker count of finished trials are already waiting to be written is held back, and so is not fed back, until the earliest unfinished trial finishes and the backlog is written (see batching).
Pruning, patience and checkpoint continuation under rolling follow the rules in the rolling-schedule section of batching.
The native executable accepts additional flags without extending StudySpec v1:
--syminfo FILE: optional positive finite mincontract (since 0.3.1), mintick and pointvalue, optional integer series_origin_ms (for engines that have the run option of that name), plus NUL-free string timezone and session, in flat JSON or {"syminfo": {...}}. Apply after inputs/overrides, in mincontract/series_origin_ms/mintick/pointvalue/timezone/session order. mincontract is the instrument lot-size grid (TradingView syminfo.mincontract): a number reaches the engine as the metadata keys qty_step (order quantities are floored to the grid) and mincontract; null or absent means no grid. In the object that is read, any other value (zero, negative, string, boolean, array, object, or a number outside the double range such as 1e999) is an initialization error (exit 1, syminfo.mincontract must be a positive finite number); bare NaN and Infinity are not JSON and fail earlier (exit 1, invalid JSON at byte N, which does not name the key). series_origin_ms is the epoch milliseconds of the first bar of the TradingView reference series, enabling range-start EMA warm-up and starting request.security at the first whole bucket at or after the origin. It must be a finite JSON number that is an integer from 1 through 9007199254740991 and is applied through strategy_set_syminfo_metadata as series_origin_ms, after the lot grid (qty_step, then mincontract) and before mintick. Absent or null means no call and unchanged runs; any other value fails initialization (exit 1, failure code hpo_study_spec_invalid, arguments {"reason":"study"}, message syminfo.series_origin_ms must be a positive integer). A plugin without strategy_set_syminfo_metadata keeps working when neither metadata key is declared; with mincontract or series_origin_ms, each trial fails with a trial_error naming the key, and the run exits 2 because no trial is feasible. Missing fields and empty strings keep engine defaults; other catalog keys are ignored. Symbol and chart timezones are distinct.--progress-fd N: inherited writable descriptor for one flushed terminal-trial JSON object per line. One writer prevents interleaving. Each object equals its final trials[] entry; drain pipes concurrently and do not assume completion order matches trial-ID order.--max-wall-seconds S: positive finite wall seconds, including native initialization. Workers check before claiming a candidate and adaptive samplers check before a batch. At S, new work stops with stop_reason: "deadline". Existing in-flight trials can finish during a bounded grace. SIGTERM/SIGINT likewise stop new work, and unstarted candidates never become terminal records.--wall-grace-seconds G: non-negative finite grace for draining the study after S; default 30, with 0 allowed. A watchdog runs whenever --max-wall-seconds is set, including without streaming or a per-trial timeout. At S + G, if any trial remains active, it records every in-flight trial as trial_timeout with typed failure_code: "hpo_trial_timeout", failure_origin: "hpo", and empty failure_args. It keeps stop_reason: "deadline" or an earlier "cancelled", publishes completed trials plus these unfinished records, then _exit(3) without worker joins. With no active trials the deadline keeps its cooperative exit 0/2 behavior. This grace has no effect without --max-wall-seconds. Behavior change: a study with a wall cap whose in-flight trials outlast the default grace of 30 seconds now exits 3 with those trials recorded as timeouts; users with long trials must pass a larger --wall-grace-seconds.--record-metric PATH: repeatable extra report metric, validated before execution and recorded under the exact path spelling in every trial's metrics. Unavailable values are null; extra metrics do not affect scoring or feasibility.--trial-timeout-seconds T: positive finite per-trial wall seconds from worker claim. A watchdog emits one trial_timeout record, writes final JSON from already-terminal trials plus that record, then _exit(3) without worker joins. The entire study aborts; other still-running/unstarted trials are excluded.All terminal trials additionally expose failure_code, failure_args, and failure_origin after error. These are null on success; failures retain typed, component-owned metadata independently of the human diagnostic. See Stable failure codes for the scalar/cap rules, initialization failure document, optional engine getters, and checked-setting fallback. A strategy stopped by an empty runtime.error() is an engine_error, never a scored partial success.
All terminal trials expose backtest.magnifier_sample_ticks_total (zero when unavailable). The status set is ok, constraint_violation, objective_error, constraint_error, engine_error, trial_error, and trial_timeout. Stop reasons are trial_budget_reached, search_space_exhausted, sampler_stopped, no_improvement, cancelled, deadline, and trial_timeout; cooperative stops and a latched patience stop override the usual budget/coverage reason without fabricating pending records. When a progress pipe is full, the writer can observe cancellation or a deadline during the final drain with patience enabled or disabled, and the result reports that observed stop.
Exit codes are 0 if a feasible best trial exists, 1 for initialization/I/O failure, 2 if no feasible trial exists, and 3 on a trial timeout or study wall grace expiry even if an earlier trial was feasible. Cooperative stops can write an empty trial table and exit 2. Final JSON is emitted to stdout and to native --output FILE, including stop paths. trials_completed includes timeout terminal records; coverage and best selection use only the terminal table. Timeout records cannot be feasible and make exhaustive_equivalent false.
The default --trials-out all requires a writable directory for two anonymous files. Terminal rows append in completion order to a data file, with offsets in an index file. The program creates and immediately unlinks both files in --output's parent directory when it exists and is a writable directory (. when the parent path is empty). Otherwise it uses non-empty TMPDIR, or /tmp when TMPDIR is unset or empty. The directory is chosen once for both files; creation failures do not trigger a retry in another directory. The index is addressed relative to the first trial ID of the run, so high continuation IDs and gaps are supported. The spool holds the whole trial array on disk until the process exits. On read-only root filesystems, native run and the jailed Python run wrapper can use a writable --output parent for the spool; --trials-out best-k or none avoids opening it, and the Python run wrapper forwards --trials-out, --best-k, --row-space and --max-recorded-space-bytes. The wrapper adds two anonymous spools: raw stdout and the indented rendering published as the result, formatted without loading trial arrays into Python objects. In the jailed profile (PF_SCRATCH_ARENA_FD is set), an explicit --output selects its resolved parent for both wrapper spools, even if inherited TMPDIR is unusable. Jailed runs without --output use two Linux memfd_create(MFD_CLOEXEC) spools without selecting or creating any directory, independently of TMPDIR. An unavailable backend or failed allocation or wrapping reports hpo_output_io_failed without filesystem fallback. Unjailed runs still use non-empty TMPDIR, or /tmp when it is unset or empty, including runs with --output. Allocation failures in a jailed output parent or an unjailed non-empty TMPDIR remain I/O errors without fallback. Directory-based spools require a writable directory even with reduced trial retention. The formatted spool is larger than the raw stdout. Wrapper reads and copies use the chunk size (--result-chunk-bytes, default 64 KiB, minimum 2 bytes), including stderr drains. Captured stderr, each extracted failure field, decoded JSON member names and JSON scalar tokens use --diagnostic-bytes (default 64 KiB, minimum 1 byte). Smaller diagnostic bounds can refuse longer tokens or member names. Both options take decimal integers and are checked before starting the child or creating a wrapper spool. For both options, the largest accepted value is the platform's read-size limit, sys.maxsize; larger values are refused before the child starts. Oversized failure args become null while code and origin are retained when they fit; truncated stderr has \n[native stderr truncated] appended. A very small --diagnostic-bytes can make a failure's code or origin oversized, in which case the failure is reported as hpo_unclassified_error and the stdout fallback is shortened to the bound. Both wrapper spools are closed on success or failure, with no retry in another directory or backend. Anonymous memory backing holds the complete raw and formatted results and can grow as O(output bytes). The chunk and diagnostic bounds do not cap total spool storage or system RAM. Directory-based spools also consume system RAM on tmpfs; process RSS alone does not measure their backing storage. The final trials array copies those rendered rows in trial-ID order without retaining every trial record or the full document as Python objects. This spool is independent of --trials-file FILE and --progress-fd, which retain their ordered NDJSON stream and backpressure only when requested; --trials-file may still be a pipe. Spool creation, writing, finalization or reading failures report hpo_output_io_failed; an incomplete spool is refused instead of being published as a complete trial array. best-k and none continue to retain only the bounded best-k heap.
The wall and per-trial timeout controls are not forwarded by the Python CLI. The Python run wrapper does forward --no-improvement-trials when nonzero. The public Python preparation API prepare_run(study_path, engine_root, cache_dir) returns native argv and artifact JSON without launching, so callers may append the controls and execute the native process directly. Optional native, compiler, and eigen_include keywords preserve CLI overrides. timeout_seconds remains a rejected reserved StudySpec field; it is not an alias for either native wall cap.
Candidate-policy and coverage fields are part of replay provenance alongside sampler implementation, sampler configuration, seed, batch size/lag, pruning policy, artifact key, and search-space definition:
| Field | Contract |
|---|---|
candidate_policy | Requested sampler.candidate_policy. |
candidate_policy_implementation | Stable implementation identity; compare it when replaying a proposal sequence. |
search_space_finite | Whether the declared space has an exact finite cardinality. |
search_space_cardinality | Exact Cartesian cardinality for a finite space; absent/null when no finite value exists. |
trials_requested | Positive trial budget from StudySpec. |
trials_completed | Number of terminal trial records emitted. |
unique_candidates_attempted | Distinct complete parameter vectors represented by terminal trials. |
duplicate_proposals_skipped | Internal duplicate proposals rejected before trial execution; they do not consume trial budget. |
remaining_candidates | Finite cardinality minus unique attempted vectors; absent/null for a non-finite space. |
search_space_exhausted | Whether no unattempted vector remains in a finite space. |
stop_reason | Machine-readable reason that candidate generation stopped. |
replay_contract | ordered_batches_v1 for the default batches schedule. rolling_completion_v1 for rolling, whose result is NOT byte-replayable. |
completion_order | Present only when execution.schedule is "rolling": the finished trial ids, each once, in the order they were fed back. See Rolling schedule. |
completed_before | Present only when execution.schedule is "rolling": a STUDY-LEVEL array indexed by trial id; completed_before[i] is the number of completion_order entries the proposal for trial i had seen. No per-trial record carries either key. |
early_stop | Present only when --no-improvement-trials is enabled; see the patience and drain fields. |
sobol | Present only for sampler: "sobol"; descriptor, identity and this part's index range, see Sobol. |
candidate_list | Present only for sampler: "candidates"; list identity and terminal-position coverage, see Candidate list. |
full_parameter_coverage | Every vector in the declared finite domain has a terminal trial record. |
exhaustive_equivalent | Full parameter coverage and every trial status is ok or constraint_violation. |
Full parameter coverage means that every declared complete vector has a terminal trial record. It is deliberately weaker than objective coverage. A constraint violation or failed evaluation remains seen and can coexist with full parameter coverage, but a failure supplies no comparable objective.
Consequently, candidate_policy="exhaustive" does not alone certify equality with the best result of a successful grid run. exhaustive_equivalent remains false after any pruning, engine, objective, constraint-evaluation, serialization, or other trial error. When it is true, grid-best equivalence additionally assumes the same deterministic artifact, dataset, runtime settings, objective, and constraint policy. Adaptive TPE and grid may visit the same set in different orders, so intermediate best-so-far results need not match even when their final best result does.
Continuation is a CLI/job input, not a new StudySpec field. Pass --warm-start FILE to run or space-info --spec STUDY. The new study must have the same parameter names, kinds, numeric bounds/steps, categorical choice order/types, log flags, objective expression, direction and constraints. Changing the trial budget, seed, workers, batch size, HPO version or engine/codegen pins does not change its space hash. This identity does not certify unchanged market data, fixed inputs or strategy behavior: the caller is responsible for keeping observations semantically comparable.
The input is auto-detected as binary v2 blocks (0.6.0), terminal-trial JSONL, a standalone trial array, or a complete result object. Result objects combine warm_start_trials with new trials. By default, space metadata is repeated on each new trial so a standalone trial array remains self-describing. Native run and Python pineforge-hpo run accept --row-space full|hash|none as a CLI output control, independent of retention; it is not a StudySpec field. full preserves the existing bytes, including each row's declarative space, space_hash_version and space_hash. hash omits row space and keeps only space_hash_version and space_hash for space identity. none omits all three row fields. Every mode retains the full declarative space and its versioned hash at the result's top level.
Both run commands also accept --max-recorded-space-bytes N|unlimited as a CLI output control, not a StudySpec field. Omission and 0 select an automatic inclusive bound of min(UINT64_MAX, 2 * B). B is the exact UTF-8 byte length of the complete emitted space JSON descriptor, already constructed and serialized during admission before the comparison and before trials. It includes dimensions, objective/constraints, escaped strings and symbol-feed metadata. Positive uint64 N sets an operator bound; explicit unlimited bypasses the byte cap. The automatic bound never refuses the descriptor it sizes. Only an operator-set positive bound can cause the byte-size refusal, reporting hpo_recorded_space_too_large with integer actual_bytes and max_bytes arguments and exit 1 before trials or output destinations are opened. The bound applies independently of --row-space and --trials-out. This comparison does not protect allocations used to construct or serialize the descriptor. The Python wrapper forwards explicit values and leaves native's default in effect when the option is omitted.
The mode applies to terminal rows in trials, summary.best_k, --trials-file and --progress-fd, and to ancestor rows in warm_start_trials. In full mode, rows inherited from a compact (hash or none) parent are copied as recorded, so the continuation can contain compact ancestor rows. Compact streams and extracted arrays require the containing result's space alongside them. Hash rows can be joined to it by space_hash; none rows have no space join key. To import compact rows alone as a continuation, attach that declarative space to each row; complete trials_out=all results remain valid parents in every mode. Budget for per-line space overhead when using full; the 256 MiB input cap is a byte cap, not a constant-memory import guarantee. Trial IDs must be unique uint64 integers leaving room for a continuation ID, parameters must lie in the declared space, objectives/rung scores must be finite or null, and statuses must be ok, constraint_violation, engine_error, objective_error, constraint_error, trial_error, trial_timeout, pruned or partial. Feasibility must agree with status. Unknown statuses fail closed at initialization.
Legacy complete results can supply an embedded study_spec or a study JSON path (relative to the parent file) when declarative space was not recorded. Legacy bare JSONL without recorded space cannot prove bounds/objective compatibility and is refused; use the complete parent result or attach its normalized recorded space to each trial. Older hash versions are accepted by recomputing from that recorded space, not by requiring a digest from the current package version.
space_hash is lowercase SHA-256 of UTF-8 JSON, without trailing newline, with sorted object keys, no insignificant whitespace, unescaped non-ASCII characters and standard JSON string escaping. The root contains exactly space_hash_version: 1, parameters and objective:
parameters maps sorted names to kind: integer, real, boolean or categorical.low, high, step as decimal strings, and a Boolean log; omitted step/log normalize to 1/false.low, high and optional step as 16 lowercase hex digits of IEEE-754 binary64 bits (step: null for continuous), and Boolean log. Omitted log normalizes to false; signed zero normalizes to positive zero.kind.choices: each is [type, value] with type integer, real, boolean or string. Integer/real values use the same decimal string/binary64-hex encodings; Boolean/string values remain JSON Boolean/string.objective contains exact expression, direction and lexicographically sorted expression-string constraints. Constraint order does not affect identity; all other order-sensitive values, including choices, are preserved.The recorded declarative space retains normal numeric JSON values, not hash-encoded strings, so future versions can recompute a different canonical form. Changing the definition requires bumping space_hash_version; package/pin bumps do not. The golden digest for examples/single_strategy/study.json is 1188c07588652f2b365a1fc233815d5c51d350e92106051309d9e84ad586071a. Python and native golden/parity tests pin this definition.
Grid scans its original Cartesian ordinal order and skips every parent vector, even failed/pruned ones; sparse histories therefore resume at the first untried ordinal. without_replacement reserves every attempted vector across parent plus new trials. If the combined budget reaches cardinality, these finite policies produce the same final set as a single exhaustive run. Neither rule depends on parent trial IDs being contiguous. IDs always resume after the maximum ID. Budgets that exceed the remaining uint64 trial-ID capacity fail preflight with exit 4. Native wall-only continuations stop at that capacity rather than wrapping IDs, and refuse before loading plugins/data when no additional trial ID is available. An exhaustive continuation's additional budget must equal the remaining cardinality; without_replacement cannot request more than remains. Incompatible warm budgets are refused with exit 4 before compilation or billing.
Random uses continuation_seed(seed, warm_trials): for nonzero warm count, unsigned 64-bit SplitMix-style mixing of seed XOR (warm_trials + 0x9e3779b97f4a7c15), using the 0xbf58476d1ce4e5b9 and 0x94d049bb133111eb multipliers and shifts 30/27/31. Zero warm trials preserves the original seed. Parent vectors are rejected; after 64 unsuccessful finite draws a deterministic untried-ordinal fallback ensures progress. New random points can repeat one another under sampler_default. The finite ordinal fallback can stop a random continuation early once it has visited every non-parent ordinal; its combined coverage is then complete.
TPE trains only on finite completed feasible objectives, as in live tell(); infeasible/failed/pruned/partial trials follow live abandon() semantics. Imported observations count toward startup and the history-switch threshold. Available pruning.rung_scores restore the bounded prefix-pruner histories, not final-objective observations. Changing pruning rungs invalidates their meaning; keep the parent's prefix schedule for pruning equivalence.
Since 0.7.0, TPE imports the ordered parent without proposing any historical trial. A complete result's tpe_sampler_state, or a warm-v2 PFHSTATE extension, restores the exact suggestion RNG, reservoir RNG, finite fallback cursor, retained observations and bounded cache membership when configuration, seed, complete typed history and numerical-build identity match. Since 0.9.0, that identity records TPE algorithm revision 2, the pinned CORE-MATH revision, binary64 arithmetic, nearest rounding, gradual underflow, the implicit-contraction canary and a self-probe of the portable kernels. No host-libm, CPU-vendor, compiler-brand, libc or long-double fingerprint participates. Explicit IEEE FMA is used for correctly-rounded kernels and compensated arithmetic; implicit contraction and fast math remain disabled. Unsupported FP modes and unavailable numeric-build instrumentation fail closed. See the cross-vendor proof.
v0.8.0 checkpoints are rebuilt, not refused: ordered typed objective history is compatible, but revision-1 arithmetic/RNG state is not imported into revision 2. warm_start_model is "rebuilt_history", with warm_start_reason explaining the numerical algorithm/build mismatch and that this is not a bitwise v0.8.0 continuation. Old-state bytes are never mixed with the new model. Identical revision-2 checkpoints can restore across supported Intel, AMD, aarch64 Linux and macOS arm64 hosts.
TPE result JSON records numeric_build_identity (the child computed identity) and parent_numeric_build_identity (the identity extracted from the checkpoint actually compared). Warm runs repeat both fields in warm_start provenance. Parent identity is null for a fresh run, history without a checkpoint, an unknown checkpoint version, or lag-one continuation (which deliberately ignores checkpoints). It is recorded even when a valid foreign checkpoint rebuilds; it is not copied from untrusted parent JSON fields. The app may assert that warm_start_model == "restored_sampler_state" implies a non-null parent identity exactly equal to the child identity. Grid/random results omit these fields.
The checkpoint is checksummed and contains a typed-history fingerprint. Full-history models rebuild from the same chronological rows; bounded caches rebuild from their recorded fitting IDs. Matching future lag-zero ask/tell schedules reproduce the uninterrupted stream bit-for-bit, including a partial final parent batch. The checkpoint requires zero outstanding candidates and the complete parent history, not just best-K rows.
Row-only v0.5/v0.6 parents, mismatching checkpoints and lag-one continuations instead reconstruct trainable history in ID order with the derived continuation RNG and the same history switch and reservations. This intentionally replaces the former complete matching-batch replay contract for legacy TPE parents. Numeric rejection sampling and finite retries consume a model-dependent RNG stream; winning terminal rows do not record its cursor. Advancing a fixed number of RNG draws per trial cannot be exact. The weaker invariant is deterministic, bounded history reconstruction with correct startup progress and exclusion of all reserved vectors. Exact byte replay holds for the same parent bytes, seed, sampler configuration, batch size and worker count (excluding explicitly requested timing sidecars).
The app may choose TPE batch = clamp(floor((trials - exploration) / 20), 2, workers) for worker counts 2..8, and random/grid batch = workers. Batch size is not part of space identity; matching it matters for TPE proposal equivalence, not compatibility.
This is a sampler observation format, not a backtest report or resumable execution checkpoint. Store every earlier attempted trial, including unsuccessful and duplicate-parameter trials. Each chunk is one independently self-describing block. A file is one or more complete blocks concatenated byte-for-byte, with no outer header, separators, alignment padding, compression or footer. Blocks can arrive in any order; IDs determine observation order. Every block must match the current study.
An independent extension block starts with the eight ASCII bytes PFHSTATE, followed by a little-endian uint64 payload length and that many UTF-8 checkpoint bytes. There is no padding. Valid other checkpoint versions rebuild history without proposal replay; they do not restore or fail merely for a different version. Malformed/checksum-invalid envelopes remain errors. The current payload is PFHTPE2\n, a 64-character lowercase SHA-256 digest, \n, and the versioned state. The digest covers the state after the second newline. It checks integrity, not authenticity: warm files are trusted input. The payload contains a configuration/build signature, explicit numerical-build token, counters/history fingerprint, and two canonical engines (MT64 312, 312 uint64 state words, position 0..312). It never uses libc++/libstdc++ stream serialization. All-zero/degenerate states are rejected. Full-history state is independent of trial count; bounded state carries retained and cached fitting IDs only. The switch-transition window never serializes a stale full-history bad-model list. The payload must be nonempty and at most 16 MiB; at least one ordinary trial block is required. No-state warm-v2 golden bytes are unchanged. v0.6 and older binary readers reject this extension; omit sampler_state when producing a row-only export for them.
State blocks can appear anywhere among row blocks. The checkpoint with the greatest attempted-row count wins, independently of block order; identical duplicates are allowed, conflicting checkpoints for the same attempted-row count are rejected. The sampler still verifies the selected checkpoint against the complete imported history and current configuration. A stale or incompatible checkpoint causes history reconstruction, never proposal replay. Native warm-encode preserves the state from complete result JSON. Python writers accept it through the keyword-only sampler_state argument.
All multibyte integers and IEEE-754 binary64 values are little-endian. Offsets are relative to the start of each block. The fixed header is exactly 80 bytes.
| Offset | Bytes | Type/value |
|---|---|---|
| 0 | 8 | Magic: hex 50 46 48 57 41 52 4d 00 (PFHWARM\0) |
| 8 | 2 | uint16 format version: 2 |
| 10 | 2 | uint16 flags: 0, or bit 0 (1) for symbol-feed metadata (0.8.0) |
| 12 | 4 | uint32 header bytes: 80 + 4 * P, plus optional extension bytes |
| 16 | 8 | uint64 total block bytes, including header/descriptors/payload |
| 24 | 8 | uint64 trial count N, strictly positive |
| 32 | 4 | uint32 parameter column count P, equal to study input count |
| 36 | 4 | uint32 objective column count O: exactly 1 in 0.6.0 |
| 40 | 4 | uint32 constraint column count C, equal to study constraint count |
| 44 | 4 | uint32 reserved: 0 |
| 48 | 32 | Raw SHA-256 bytes of canonical space hash v1, not ASCII hex |
| 80 | 4 * P | Parameter descriptors, in lexicographic UTF-8 parameter-name order |
When flags bit 0 is set, descriptors are followed by a uint32 little-endian UTF-8 JSON byte count and the complete applied_runtime.symbol_feeds record (at most 8 MiB), with no padding. header_bytes includes this extension; every block must agree with the current feed identity. Readers before 0.8.0 refuse it. Flags-zero header bytes remain unchanged.
Each descriptor is uint8 kind, uint8 encoding, uint16 reserved=0:
| Kind | Domain | Encoding |
|---|---|---|
| 1 | Integer (including log integer) | 1: int32 grid index |
| 2 | Real with a step | 1: int32 grid index |
| 2 | Real without a step (including log real and fixed bounds) | 2: float64 original value |
| 3 | Boolean | 1: int32 choice index, false=0 and true=1 |
| 4 | Categorical | 1: int32 zero-based index in declared typed choice order |
Integer grid index i decodes to low + i * step. Stepped reals decode with binary64 fused multiply-add fma(double(i), step, low); a last-grid value above the upper bound by permitted endpoint rounding is clamped to high, exactly as the native search space does. Indices must be nonnegative and less than the dimension cardinality; at most 2^31 values are supported per indexed dimension. Log parameters store original units/indices, never logarithmic model coordinates. The converter refuses noncanonical stepped-real values rather than losing bits. Continuous values must be finite and in range; signed zero is preserved and is distinct in the exact tried-vector index, as in the legacy canonical key.
Starting at header_bytes, write these complete columns consecutively:
N uint64 trial IDs. IDs are intentionally widened from the requested int32: the existing API permits IDs above 2^31 and 2^53. Values must be unique across all blocks and less than UINT64_MAX, leaving a continuation ID.N uint8 trial states.P parameter columns, using its descriptor, in descriptor order.O=1 objective column: N float64 values in the original objective direction.C constraint columns: N float64 expression values, in stable lexicographic UTF-8 constraint-expression order (duplicates retain their study order). This matches the order-insensitive constraint identity in hash v1.No column has a name string, JSON, metrics, error text, timestamp, pruning rung or trial envelope. Names and categorical values are resolved from the matched study. Each null objective/constraint is the only accepted nonfinite pattern: canonical quiet NaN bits 0x7ff8000000000000 (bytes 00 00 00 00 00 00 f8 7f). Finite values retain their exact bits, including negative zero. Infinities and other NaN patterns are refused. Missing legacy constraint arrays become null, not inferred numeric evaluations. New constraint_values arrays are stored in study order in JSON, then permuted into canonical binary order. State determines feasibility; numeric constraint columns do not retroactively change v0.5 TPE behavior.
| State | Legacy status | Feasible | Trains TPE |
|---|---|---|---|
| 0 | ok | yes | only if the objective is finite |
| 1 | constraint_violation | no | no |
| 2 | engine_error | no | no |
| 3 | objective_error | no | no |
| 4 | constraint_error | no | no |
| 5 | trial_error | no | no |
| 6 | trial_timeout | no | no |
| 7 | pruned | no | no |
| 8 | partial | no | no |
block_bytes = header_bytes + N * (8 + 1 + sum(parameter_widths) + 8*O + 8*C). Reject unsupported versions/flags/reserved fields, inconsistent counts/descriptors, overflowing lengths, partial/trailing bytes, duplicate IDs, invalid states/scalars and any block hash mismatch with exit 4, warm-start incompatible: ..., before compilation/plugin/data loading or billable output. Finite exhaustion remains exit 5. space-info succeeds and reports zero remaining for exhausted histories.
The native loader uses a read-only mmap and validates fixed-width columns directly; it does not parse or retain JSON. TPE holds shared row references, not one candidate map per trial. Preserve the mapped file unchanged until the run exits; upload/finalize objects before mapping, never append/truncate/rewrite an active input. Total rows are limited to UINT32_MAX; IDs are still uint64. Sorted uint32 row references provide collision-free exact tried-vector lookup, retaining all duplicate observations while counting each vector once. This costs at most 4 bytes/row, plus another 4 bytes/row only when ID sorting is needed. A Cartesian bitmap is unsuitable for continuous or huge spaces and cannot represent all those vectors; no separate tried bitmap is stored.
The app keeps SHA-256 per immutable object and verifies objects before concatenation. There is no redundant per-block checksum. The result's source_sha256 hashes the exact concatenated input bytes; native --warm-digest also requests this digest. space-info (including --warm-details preflight) performs load/validation/tried-index construction without payload hashing. At 1M rows, five float64 inputs need 57,000,100 bytes plus a 4-MB tried index; 32 inputs with four indexed and 28 float64 columns need 257,000,208 bytes plus that index. All-float64 32-input blocks need 273,000,208 bytes. These are format sizes, not promises about OS RSS, TPE density work, replay, result rendering or provenance hashing.
Golden vectors are tests/fixtures/warm_v2_spec.json, warm_v2_trials.json and warm_v2_golden.json: one 247-byte block and the same rows in two blocks, exact hex and SHA-256. They exercise every encoding, non-int32/non-exact-double IDs, nulls and signed zero. Native and Python writers must match these byte-for-byte. The release additionally compares actual v0.5 JSON suggestions with v0.6 JSON, single-block and reordered multi-block inputs, using packed binary64/uint64 suggestion hashes. See benchmarks/warm/README.md for reproduction and boundaries.
JSON/JSONL v0.5 inputs stay accepted for this release with the existing 256-MiB cap. Binary v2 currently supports grid/random/TPE with no active prefix pruner; use JSON when restoring pruning rungs. dlib_global warm start remains unsupported. Warm rows never go to fd 3. A final trials_out=all result reconstructs minimal ancestor records only at rendering time; summary/none users should retain the original binary plus independently encoded new-trial chunks for subsequent continuation.
study.json.transpile_full() supplies input title, type, default, and any literal bounds/step/options retained in the artifact manifest. Numeric options compare by value, so a float input whose options are written as integers accepts an equal float value.search_space explicitly selects the tunable inputs.fixed_inputs are never sampled and cannot overlap a tunable input.candidates list (its results can seed TPE history only); 1.9.8