- Generated by
1.9.8
|
PineForge HPO 1.11.0
Native hyperparameter optimization for PineForge strategies
|
The hardened single-strategy native path is implemented. PineScript initialization is Python; sampling, plugin execution, objective evaluation, and trial scheduling are C++. The executable path currently accepts one strategy and one dataset.
Portfolio observation and custom-objective types exist as C++ contracts, but no account aggregator or portfolio CLI has been implemented.
pineforge-hpo composes two separately distributed products:
pineforge-codegen-oss provides the pineforge-codegen Python distribution and its pineforge_codegen module. transpile_full() emits generated C++, an input manifest, and literal strategy(...) parameters.pineforge-engine executes a compiled strategy through its public C ABI.The HPO repository does not subclass or link against BacktestEngine internals. The native adapter loads a generated strategy plugin with local symbol scope and resolves only its C ABI.
The Python pineforge-hpo run wrapper resolves a StudySpec and translates it into a pineforge-hpo-native run invocation. Python is not in the trial hot loop.
python/pineforge_hpo/transpile.py imports pineforge_codegen.transpile_full and normalizes its response:
Pine lexer/parser/analyzer errors remain distinct from native compiler errors. The codegen identity records both the installed distribution version and a hash of its Python implementation, so an editable or dirty codegen checkout cannot silently reuse a stale strategy artifact.
ArtifactBuilder supports Linux and macOS and writes:
The cache has two identities:
generated_cpp_sha256. This preserves the generated translation unit in the content identity without forcing a repeat transpilation merely to discover the key.Per-key advisory locks serialize the first build. Artifact directories and request indexes are published atomically. Cache hits re-hash the plugin and generated C++ and verify their manifest/provenance identities.
New builds with an engine planner companion record an additive top-level window_plan block in both version-1 metadata files: its absolute build path (audit only), expected SHA-256, planner ABI, engine version/ABI/header/archive hashes, platform and frozen self-test ID. This runtime companion stays outside request_identity, so its presence does not change either compile key. Ordinary builds and cache hits require no helper; selected callers use ArtifactBuilder.enrich_window_plan to verify an existing block or enrich a validated source cache under the same request lock without recompiling. Each metadata file is replaced atomically; a partial or disagreeing pair refuses selected admission. An interrupted two-file enrichment leaves selected admission refused until enrichment is re-run. Ordinary builds and cache hits ignore the window_plan block. Precompiled selected admission requires a recorded block, even with an explicit helper path. Runtime lookup accepts an explicit absolute path or derives the sibling of the selected build/lib/ or lib/ archive from an explicit local engine root with matching content; build/libpineforge.a has no derived companion. The recorded build path never instructs lookup. Python checks only the file hash; native snapshot loading, ABI and behavioral checks remain the native consumer's responsibility.
When codegen provides transpile_with_request_inventory, the bridge uses that single generation and the builder binds its inventory to the generated UTF-8 and linked plugin through codegen's binder. The optional top-level request_feed_inventory object is identical in manifest/provenance and the adjacent request_feed_inventory.json. The object stays outside the compile identity; request_identity.request_feed_inventory records only the producer's schema string or null, forcing one cache rebuild for the new key. Ordinary loading ignores the block and sidecar. Selected callers explicitly use the Python inventory admission API to validate their bindings, inputs and timeframe policy.
The compile contract is:
When the compiler identifies as Clang, the builder adds -fbracket-depth=1024: engine 1.0 requires it of Clang builds outside CMake, because generated C++ for long Pine expression chains nests deeper than Clang's default limit of 256. GCC has no such limit. The flags used are part of the cache identity.
The engine static library is linked with GNU whole-archive on Linux or force_load on macOS. By default (plugin_load="build"), before publication the builder loads the plugin, requires pf_abi_version, strategy_create, strategy_free, strategy_set_input, strategy_set_override, strategy_get_last_error, run_backtest_full, and report_free, then checks that pf_abi_version() matches the selected engine ABI.
For callers that separate compilation from execution, construct ArtifactBuilder(..., plugin_load="deferred") or pass the same option to build_strategy_artifact. A fresh deferred build compiles and publishes without loading the plugin. Both metadata files record "plugin_validation":{"status":"deferred"}, with no confirmed abi_version or required_symbols. The option stays outside both cache identities. A deferred build can reuse an already validated cache entry without loading it or changing its validation metadata.
Pass the returned StrategyArtifact to a fresh process and call ArtifactBuilder.validate_built_plugin(artifact) on a builder configured for the same cache. This method loads the published plugin in the calling process, checks its required symbols and recorded engine ABI, and replaces the pending metadata with the usual abi_version and required_symbols result only after the content checks also succeed. It starts no process and probes neither the compiler nor codegen; the recorded engine ABI suffices without a local engine installation. The method returns the checked artifact. Calling it on an already validated entry checks the cache contents without loading or rewriting it.
The plugin load runs user static initializers while the request lock is held. A hanging initializer makes other builders that need the same request lock time out after lock_timeout_seconds (120 seconds by default).
A later default build that finds a pending cache entry validates it under the request lock before returning, without compiling again. If the request index is missing or invalid, it transpiles again to find the artifact key. Validation failures propagate without recording success. Each metadata replacement is atomic; explicit validation refuses an interrupted, disagreeing pair. A later default build discards that cache entry and transpiles and compiles it again. Recovery when the manifest records validation but provenance remains pending requires a rebuild in a process that permits forking to run the compiler. Window-plan enrichment continues to require a validated artifact.
Deferred loading is not a jail: it moves the first plugin load, including user code in static initializers, to a process the caller chooses. The caller creates that process and installs any restrictions before calling the validation entry. The option does not restrict the compiler or install process restrictions. The ordinary CLI retains its default build-and-run behavior.
With the default option, the builder's load happens while a fresh artifact is published. During study preparation, _plugin_option_values performs a second in-process plugin load when a study's enum input carries an options list and its search gives that dimension ordinal choices. Cached and precompiled artifacts also take this path, with the same user rights as the native runner that later loads and runs the plugin: it reads the strategy's settings receipt to learn the declaration index of each enum member, because the manifest lists an enum input's options as their tags only. A tag choice needs no load; it is matched exactly as the manifest lists it (a bare member name or a tag of another case is refused), checked against the manifest options alone, and reaches the runner unchanged. A plugin whose receipt cannot be read is refused rather than checked against the positions in options, which are the declaration indices only for a fully listed enum. A launcher using deferred artifacts must call validate_built_plugin before study preparation in its chosen restricted process, or avoid the enum-receipt path that loads the plugin.
The native search-space model implements:
step = 1);Candidates are type- and range-checked before serialization to the strategy ABI. Fixed inputs are rejected if they overlap a search dimension.
The optional candidate-policy layer is separate from sampler scoring and ordering:
sampler_default delegates reuse and exhaustion semantics to the sampler and preserves continuous real dimensions;without_replacement requires a finite domain and atomically reserves every complete parameter vector at most once;exhaustive adds the requirement that the requested trial count exactly equals the finite domain cardinality.Only TPE and grid accept the finite policies. Random and dlib reject them rather than silently changing optimizer. This keeps existing continuous TPE behavior backward compatible while making finite coverage an explicit contract.
Finite integer and varying-real dimensions are indexed by non-negative lattice ordinal k, with legal values low + k * step <= high. A varying real must declare a positive step; a fixed real where low == high contributes one value without a step. A non-constant log real remains continuous in StudySpec v1 and is not eligible for a finite policy, while a fixed log real is eligible. Boolean and categorical cardinalities are two and the declared choice count. Checked Cartesian multiplication resolves the full domain cardinality before execution. The stepped-real codec uses fused multiply-add decoding, permits a one-ULP snap only at the declared upper endpoint, caps real lattices at 2^53 ordinals, and rejects any binary64 or strategy-ABI collision before execution.
Four native samplers are implemented:
GridSampler exhausts the Cartesian product in stable dimension/value order;RandomSampler uses std::mt19937_64, an explicit seed, rejection sampling for bounded integers, and a stable 53-bit conversion for real values;DlibGlobalSampler wraps dlib global_function_search with move-only outstanding requests and explicit ask(), tell(), and abandon() calls;TpeSampler fits independent per-dimension Parzen marginals and exposes the same explicit adaptive request lifecycle.Grid and random candidates are generated lazily; dlib and TPE candidates are interleaved with feedback. The scheduler proposes a fixed logical batch (--batch-size, defaulting to workers), runs its backtests on a persistent FIFO thread pool, and commits results in trial-id order. Only the coordinator touches samplers and pruning history. An explicit batch size makes the same seed repeatable across worker counts without letting completion timing alter either proposals or pruning cuts. This describes the default batches schedule. The opt-in rolling schedule (--schedule rolling, since 0.14.0) instead refills a free worker slot at once from whatever has finished; the coordinator is still the only thread that touches samplers and pruning history, and the result records what each proposal saw. See batching.
The default --batch-lag 0 commits batch b before proposing b+1, preserving the 0.1.x adaptive sequence at the default batch size. Opt-in lag one keeps two batches submitted: b+1 uses history through b-1, with b's parameters still pending. TPE includes those parameters in its constant-liar bad estimator; lag-one TPE rejects constant_liar=false. Even an already-finished future is not visible to the coordinator until its whole batch is committed. Pruning cuts use the same frozen history. The lag is persisted replay provenance, not a timing-dependent optimization. Batch size and lag belong to the batches schedule; --schedule rolling takes neither. See batching.
For a finite candidate policy, the coordinator reserves a full vector in the seen set before returning it from ask(). The reservation is therefore visible to every later request in the same batch. Pending, completed, infeasible, and failed vectors remain seen. This uniqueness state is independent of TPE's constant-liar model: constant liar changes acquisition density, while the seen set provides the exact no-repeat guarantee.
Stepped/discrete dlib dimensions are integer indices; linear continuous real dimensions retain their bounds. Log integer and real dimensions use continuous ln(value) coordinates. Log integer coordinate bounds expand by half a step before transformation, and values decode by nearest-integer exp(z). Constant dimensions are omitted and injected on decode. dlib always maximizes, so minimize objectives are negated internally. The dlib dependency is pinned to 20.0.1 by default and remains behind the sampler PIMPL.
TPE separates completed observations into good and bad groups using a bounded gamma fraction. Numeric marginals use bounded Gaussian mixtures with quantized bin-mass likelihoods for discrete values; categorical and boolean marginals use prior-smoothed probabilities. Candidate ranking maximizes the summed per-field log likelihood ratio. Constant-liar batching adds outstanding parameters only to the bad estimator. Failed, infeasible, and pruned requests are abandoned and do not train either model. Under a finite policy, abandonment does not release the parameter vector from the seen set.
TPE finite sampling retains adaptive proposal order while selecting only unseen vectors. without_replacement validates trials <= cardinality; exhaustive validates equality. Grid covers the same declared set in its stable order. The two orders need not have the same intermediate best-so-far sequence.
Version 0.4.0 keeps exact 0.3.0 full-history TPE by default (history_switch=null, meaning never switch). Explicit history_switch=N opts into bounded models at N completed usable observations; below N the estimator remains exact. The retained set is the global best gamma_cap observations plus 64 recent non-elites plus a separately seeded reservoir of older non-elites (default 448). Each observation enters the older pool once when it leaves the recent window; uniform reservoir replacement preserves global bad-model coverage without growing history. An explicit larger switch increases the full-history prefix, not the bounded window. Fitted numeric density tables use 513 points with Gaussian recurrence and an eight-sigma cutoff, interpolating densities and using midpoint mass for tiny discrete bins. Good models refit whenever elite IDs change; bad models refit every 32 completions, with unchanged elites and cached epochs bypassing split construction. Pending constant-liar overlays update on every ask. Eight acquisition draws replace 24 after the switch (configurable). Failed/pruned requests, outstanding proposals, worker count, and wall time do not advance the switch. Only successful finite tell() calls increment the completed observation count; reset() restores exact-prefix mode.
In opted-in mode, no SIMD or candidate-scoring threads are required: bounding observations, model refits, and acquisition draws makes history cost flat while leaving execution cores available for backtests. See scaling measurements. Exact sparse finite reservation/coverage sets use temporary disk indexes with bounded resident workspace; finite spaces up to 100,000,000 use capped bitsets.
Log dimensions share one transform contract across adaptive samplers. TPE encodes observations as z = ln(value) and fits numeric kernels in normalized log space. A real proposal decodes with exp(z). A log integer proposal rounds exp(z) and its likelihood is the Gaussian mass over that integer's transformed rounding bin. Its latent bounds are ln(low - 0.5) and ln(high + 0.5), while observation means remain ln(k). The l/g likelihood ratio can be evaluated in log coordinates because the real-valued Jacobian is common to both models and cancels. Priors are uniform in the transformed coordinate, not in the original linear units.
The implementation computes logarithmic coordinates relative to the lower boundary with log1p and reconstructs them with expm1. This is mathematically equivalent to subtracting absolute logarithms but avoids catastrophic cancellation in narrow, high-magnitude ranges. Log-integer construction also checks representative lower, middle, and upper quantization bins and rejects a domain if normalized double precision cannot keep every bin distinct.
One StrategyPlugin loads the dynamic library and resolves its function table. One immutable Dataset loads the OHLCV CSV or maps a binary .bin chart. Each trial independently performs:
The plugin and dataset may be shared by worker threads, but strategy handles and reports are never shared. Cleanup order is structural: report-owned buffers are freed before the handle.
ReportSnapshot retains report counters and all/long/short/equity metrics independently of the plugin handle lifetime. It optionally copies the equity curve for portfolio/custom-objective consumers; the scalar expression CLI disables that copy.
MetricExpression compiles an expression once. It supports:
Comparisons yield 1.0 or 0.0. The executable CLI evaluates constraints as metric expressions and treats a finite non-zero result as satisfied. Metric identifiers resolve before any backtest. Constraints always reject division by zero and non-finite operands. Objective policies are explicit, and a non-finite final score is never ranked or treated as feasible.
The generic library boundary remains independent of PineForge reports:
Applications can therefore define a custom C++ objective over any observation. The executable StudySpec path currently resolves expression objectives only.
The intended user path accepts Pine source and StudySpec JSON:
compile owns transpilation, compilation, caching, and artifact provenance. run resolves paths relative to the StudySpec, reuses the artifact when valid, and launches the native executable.
pineforge-hpo-native consumes a compiled strategy plugin, one OHLCV CSV, search dimensions, fixed inputs, strategy overrides, objective/constraint expressions, sampler settings, and worker count. It produces a JSON study summary with the best feasible trial and selectable trial retention. all is the compatibility default; best-k and none avoid per-trial payload retention. Full terminal records may stream to fd progress and an optional NDJSON file.
The native CLI intentionally knows nothing about Pine parsing or codegen. Its result JSON includes schema_version, pineforge_hpo_version, and a stable sampler_implementation identifier in addition to the sampler configuration, seed, retained trial table, and artifact key. Proposal-sequence comparisons must match all of those fields plus logical batch size, lag, pruning, and search space. The policy layer also persists candidate_policy, candidate_policy_implementation, search_space_finite, search_space_cardinality, trials_requested, trials_completed, unique_candidates_attempted, duplicate_proposals_skipped, remaining_candidates, search_space_exhausted, stop_reason, full_parameter_coverage, and exhaustive_equivalent.
full_parameter_coverage is a parameter-set statement and may remain true when a terminal trial failed. exhaustive_equivalent additionally requires every trial status to be ok or constraint_violation. It is false after any engine, objective, constraint-evaluation, serialization, or other trial error. The flag still relies on the study's artifact, data, runtime, objective, and constraints being deterministic; it cannot detect external nondeterminism.
In bounded output modes a scalar archive tracks whole-study status/coverage, while a heap retains only best-k feasible records. One ID-ordered writer drains a bounded queue and flushes complete terminal lines. Final summary exposes counts, best-k, and coverage even when trials is empty. The progress stream is the billing authority; final retention never suppresses a terminal progress line.
The executable MVP supports:
Independent trials share the artifact and immutable bars and use fresh handles. Runtime input.*() and strategy(...) values are applied without rebuilding the plugin.
The C++ API defines PortfolioObservation, AccountEquityPoint, Allocation, SleeveSummary, and PortfolioObjectiveFn. A portfolio objective may request only account return, drawdown, allocations, turnover, concentration, or other application-defined features; it does not need every constituent report field.
The following are not implemented:
Completed independent reports also cannot reconstruct a true shared account. Shared cash, margin, order admission, and cross-strategy order sequencing need a future strategy signal/order-intent ABI and a shared broker ledger.
Initialization failures stop before any trial:
Trial-local failures remain visible in the trial table:
Finite-policy coverage treats every terminal trial vector as seen, including a constraint violation or trial-local failure. Consequently, full parameter coverage means every declared vector received a terminal trial record. It does not mean every vector produced a usable objective. Equality with the best value from a successful deterministic grid search additionally requires exhaustive_equivalent=true under identical artifact, dataset, runtime, objective, and constraint inputs.
Starting with v0.11.0, original pineforge-hpo code is licensed under the PineForge Source License 1.2. Releases up to and including v0.10.0 were released under Apache-2.0 and remain available under that license. pineforge-engine is a separate Apache-2.0 runtime dependency. pineforge-codegen, from pineforge-codegen-oss, is a separate source-available dependency with its own terms. The direct-PineScript convenience path does not relicense codegen under Apache-2.0. Precompiled plugin users can run the native HPO layer without installing codegen.
dlib 20.0.1 is the native global-search dependency and is distributed under the Boost Software License 1.0. CMake verifies the pinned source archive hash unless the build explicitly selects an exact-version system package.
1.9.8