- Generated by
1.9.8
|
PineForge HPO 1.11.0
Native hyperparameter optimization for PineForge strategies
|
Historical; superseded by the completed-count switch report. The approximately 1.6% 3k tradeoff below is not the final switch's quality evidence.
BLOCKED for release. The required 3,000-trial geomean is at most 1.01; the selected implementation measures 1.015738. Do not merge/tag this branch as satisfying the quality gate. All eight per-problem medians are within 1.10, but that does not waive the geomean requirement.
All builds, tests, and measurements run on an AWS 8-vCPU Linux spot machine, not a workstation. Timing probes share the machine with gate work and are contended measurements, not isolated speed claims. The engine/codegen pins are unchanged. The pinned HPO-BENCH replica uses COCO revision dd4bd1f0cc7699a2b612448d85aafb94636fe947.
Keep 25 global elites, 64 recent non-elites, and an independently seeded uniform reservoir of 448 older non-elites: at most 537 completed observations after warm-up. Each non-elite enters the older pool once; Algorithm R preserves sampling across the older history without consuming proposal RNG draws. Good models refresh on elite-ID changes instead of staying stale within an epoch. Bad models refresh every 32 completions. Cached epochs skip ranking and model rebuilding. Keep 513-point density tables, live pending-liar overlays, and eight EI draws. Work and retained memory remain independent of history.
Ratio means the ratio of the two per-problem median normalized residual regrets, not the median of paired-seed ratios. Lower is better. Instance 1, batch8/lag0; 3k seeds are 17, 48, 79, 110, 141, 172, 203, 234, 265, 296.
| Problem | 0.3.0 regret | Review regret | Ratio at 3k |
|---|---|---|---|
| rotated_rastrigin20 | 0.33157165 | 0.31782914 | 0.958553 |
| ackley20 | 0.35100715 | 0.33755090 | 0.961664 |
| bbob_f15_d20 | 0.08796020 | 0.09059543 | 1.029959 |
| bbob_f21_d20 | 0.05062354 | 0.05046445 | 0.996858 |
| bbob_f24_d20 | 0.20154199 | 0.20114024 | 0.998007 |
| mixed_irrelevant15 | 0.00007221 | 0.00007273 | 1.007135 |
| rosenbrock20 | 0.00419200 | 0.00461089 | 1.099926 |
| rotated_ellipsoid20 | 0.00387524 | 0.00419637 | 1.082868 |
| Geomean | 1.015738 — FAIL |
10k seeds are 17, 48, 79, 110, 141. Reference is a benchmark-only copy with retention compaction replaced by unbounded retention; density tables, EI draws, and refresh rules are unchanged. It is not a 0.3.0 run at 10k. Both binaries use the same frozen 0.3.0 coordinator extraction so sampler differences are not confounded by the production stop-path changes. Production worker replay is tested separately through 3,000 trials with lag/pruner combinations.
| Problem | Full-history regret | Review regret | Ratio at 10k |
|---|---|---|---|
| rosenbrock10 | 0.00225862 | 0.00235524 | 1.042781 |
| rosenbrock20 | 0.00370510 | 0.00380403 | 1.026699 |
| rotated_ellipsoid20 | 0.00265592 | 0.00246750 | 0.929055 |
| bbob_f21_d10 | 0.02833295 | 0.03092532 | 1.091497 |
| Geomean | 1.020763 |
Measured alternatives were not silently discarded:
| Retention / refresh variant | 3k geomean | Worst problem | Gate |
|---|---|---|---|
| 448 reservoir, good/bad 32/32 | 0.992234 | 1.262748 | Fail |
| 448 reservoir, good/bad 32/128 | 1.018646 | 1.473384 | Fail |
| 448 reservoir with population weights, good/bad 32/128 | 0.935523 | 1.282197 | Fail |
| 448 reservoir, elite-change/32 (selected) | 1.015738 | 1.099926 | Fail |
| Population-weighted reservoir, elite-change/32 | 0.965998 | 1.190807 | Fail |
A 960-observation reservoir, good/bad 32/32, also fails the two smooth-problem probe: ellipsoid 1.224114, Rosenbrock 1.146172. Merely increasing retention, draws, or global mixture mass is not a demonstrated solution. Warm-up parity through 1,000 proposals is explicitly an identity test, not quality evidence.
Random sphere prefill, then 256 timed asks with ordered batch-eight feedback, including model refits and pending-liar work. Microseconds per ask:
| Dims | n=1k | n=10k | n=100k | n=1M |
|---|---|---|---|---|
| 8 | 74.95 | 74.04 | 70.69 | 69.81 |
| 16 | 150.01 | 151.27 | 142.59 | 143.24 |
| 32 | 308.52 | 323.80 | 314.29 | 301.17 |
| 64 | 697.74 | 724.72 | 687.92 | 669.33 |
Every row retains 537 observations. At 64D/1M the acquisition probe peaks at 14,208 KiB. There is no history-growth penalty, but the larger bad mixture raises the constant cost. The original PR's isolated 245-us 64D estimate was 65% of the 375-us W=8/3-ms budget. The current loaded estimate is 178% of that budget: 64D sampling remains a throughput bottleneck. Do not advertise the 65% figure as applying to this sampler; lag1 overlap does not erase serialized sampling cost.
The selected sampler and final stop-path binary complete 1,000,000 trials at 64 continuous dimensions, W=8, batch8/lag1, and a 3-ms synthetic objective in 768.90 s. Peak RSS is 21,292 KiB (20.79 MiB). Proposal time is 690.708 s and direct barrier time 58.915 s. All 1,000,000 received billing lines strict-parse as complete, unique, monotonic terminal records; the stream is 2,145,834,929 bytes with SHA-256 52c216e7bac8a3ebdce349488f0cfd79eeffc84cdfe6c7ffa1ce363dae17d92d. Serialization costs 55.992 us/line and writing 4.094 us/line. Final JSON is 22,887 bytes, rendering in 0.492 ms and writing in 0.108 ms. The volume and bounded-memory gate passes; the long-budget quality gate does not.
Exact finite coverage uses a lazy dense bitset through cardinality 100,000,000, at most 12.5 MB/index, instead of disk insertion inside the run-state mutex. Larger/unknown spaces retain the disk fallback. Boundary/reset/duplicate and out-of-range behavior are tested. In a 100k-trial four-input finite native run, the pre-review binary takes 3.18 s and the bitset binary 3.07 s; billing bytes are identical (42,756,665). The speedup on this machine is modest, not the review's loaded 137-us/trial extrapolation. Dense coverage raises that run's RSS from 4,676 to 16,636 KiB by allocating the 100-million-bit index.
The one-million-insert microprobe measures 0.014506 us/insert for the dense index versus 3.01059 us/insert for the disk fallback: 0.0145 s versus 3.0106 s total. These direct insertion measurements do not include run-state contention.
Timeout insertion into the archive and enqueue into the terminal stream now share one atomic critical section after queue capacity is available. Progress readiness polls are bounded at 50 ms; stopped/stalled readers receive a two-second grace, then an explicit I/O failure. The coordinator wakes every 50 ms while waiting on workers so SIGTERM is observed even under a race detector. Pipes require each serialized record to fit PIPE_BUF, rejecting larger records before writing any fragment; regular output files have no such restriction. Billing uses delivered complete lines, never final-result counts after I/O failure. The cached trials-file flag is immutable. Quality ctest timeout is 300 seconds; 0.3.0 is correctly marked released.
quality_long.py runs/compares the pinned replica, and rejects stale cached studies if a binary changes. full_history_reference.py materializes the benchmark-only unbounded-retention copy and records the exact replacement and source hashes. Use the same frozen coordinator extraction for both quality builds. profile.cpp, ordinal_profile.cpp, and run_native.py --finite cover the acquisition, insertion, and native finite probes. Native before/after finite probes both use bounded output and drain the billing stream.
Compact per-seed quality and timing evidence with verified SHA-256 sidecars is published under ../optuna/evidence/2026-10-03-review-*.csv. Raw study traces, full test logs, and smoke metadata remain review artifacts, not public paths.
1.9.8