# IRSTD verification and risk-control prototype

**Status: research dossier and partially executed prototype, 2026-10-31.** Revised Project B is the provisional priority. None of A/B/C has been established as a novel publishable result. No real dataset has been trained on or evaluated in this environment.

Read in order: `research_decision.md`, `novelty_matrix.md`, `mathematical_analysis.md`, `literature_review.md`, `experiment_plan.md`. Full bibliographic verification, dataset rights/downloads and several upstream implementations remain open, individually marked in the review. There are 10 A, 15 B and 12 C relevant reference placements with overlap, plus a pending NP-loss reference. General loss/calibration papers are explicitly distinguished from IRSTD papers.

## Contents

- `irstd/core.py`: fixed-component candidates, maximum matching, metrics, exact tiny MWIS, layered DP, CP and fixed-sequence risk calibration.
- `irstd/models.py`: original compact U-Net, basic segmentation losses, MLP candidate gate and constrained surrogate.
- `scripts/`: train, predict, train gate, calibrate/evaluate, audit data, benchmark and mathematical checks.
- `tests/`: mathematical counterexamples, exhaustive tiny matching checks, CPU CLI integration and optional PyTorch gradients.
- `configs/ablations.csv`: prepared versus unimplemented comparisons.
- `evidence/papers.json`: bibliographic/status records; no copied paper text.
- `results/`: actual test logs and analytic calculations only.
- `manuscript_outline.md`, `research_log.md`: writing and milestone records.

## Actual execution status

The execution runtime is Python 3.12.14 with NumPy 2.3.5, SciPy 1.17.0 and Pillow 12.3.0. PyTorch is absent; `nvidia-smi` was not found. The user's RTX 3090 is not attached to this runtime. **26 CPU tests pass; 2 PyTorch tests are skipped.** Compilation checks pass. Synthetic fixture tests verify plumbing, not accuracy. Detector/gate training, GPU profiling, upstream baseline reproduction and real dataset metrics have not run. No performance table is populated.

## Setup

The following is a proposed, pinned training environment, not an environment already validated on the 3090. Python 3.12 is recommended. Core package versions match the executed CPU checks. PyTorch 2.5.1 is a conservative explicit pin; CUDA 12.1 wheels are listed in the official [previous versions instructions](https://pytorch.org/get-started/previous-versions/). Confirm driver compatibility on the actual workstation. Do not install MIST's older NATTEN dependencies into this environment.

```bash
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-core.txt
python -m pip install torch==2.5.1 --index-url https://download.pytorch.org/whl/cu121
python -m pip install --no-deps -e .
python -m unittest discover -s tests -v
python scripts/check_math.py
```

For CPU-only evaluation, omit PyTorch; `evaluate.py` imports it only when a learned gate is requested. In the supplied execution environment, tests used `PYTHONPATH=. "$CODEX_PRIMARY_RUNTIME_PYTHON" -m unittest discover -s tests -v` rather than installing packages. Future successful training must save `pip freeze`, Python/torch/CUDA/driver versions and GPU model. Determinism is scoped to a recorded stack, not guaranteed across different hardware/software: [PyTorch reproducibility guidance](https://docs.pytorch.org/docs/stable/notes/randomness.html).

## Data preparation

No data, checkpoints or third-party code are bundled. Follow `data/README.md` and `experiment_plan.md`. Create `data/manifest.csv` with native uint8 image/mask paths, exact split membership and independent scene groups. Original test sets stay untouched; dev and cal come from training groups or separately authorized data. Binary masks cannot resolve overlapping target identities. License and annotation audits are required before claiming a valid benchmark.

```bash
python scripts/audit_data.py --manifest data/manifest.csv --out data/audit.json
```

This detects exact-byte cross-split image leakage, group leakage, malformed data and measures size statistics. It does not detect all near-duplicates or establish independent sampling.

## Minimal experiment sequence

Commands below require real input files and (for training/prediction) PyTorch. They have not been executed here. Output run directories must be new; scripts refuse accidental overwrite.

```bash
python scripts/train.py --manifest data/manifest.csv --out runs/bce_dice_seed0 --loss bce_dice --epochs 100 --seed 0 --device cuda
python scripts/predict.py --manifest data/manifest.csv --checkpoint runs/bce_dice_seed0/best.pt --out runs/prob_seed0 --device cuda
python scripts/train_gate.py --manifest data/manifest.csv --predictions runs/prob_seed0 --out runs/gate_seed0 --mode constrained --alpha-train .1 --epochs 100 --seed 0
python scripts/evaluate.py --manifest data/manifest.csv --predictions runs/prob_seed0 --gate runs/gate_seed0/gate.pt --split cal --calibrate --risk count --alpha .1 --out runs/count_certificate.json
python scripts/evaluate.py --manifest data/manifest.csv --predictions runs/prob_seed0 --gate runs/gate_seed0/gate.pt --split test --certificate runs/count_certificate.json --out runs/count_test.json
```

Calibration always uses the predeclared 101 gate thresholds. The certificate hashes source, model/probability provenance, input files, manifest and parameters; changed artifacts invalidate reuse. Its statistical statement is **conditional on iid calibration units from the deployment distribution**. Filenames/group checks do not prove that assumption. A `reject_all` result means no useful detector was certified at the selected budget. Do not report that as strong detection performance. Do not substitute `--risk any` for `--risk count` without changing the claimed objective.

For the crucial no-gate baseline, omit `--gate` in both calibration and evaluation. For the BCE gate, use `--mode bce`; for the size ablation use `--no-size`. All must use identical candidate pools and calibration access. Gate training currently uses in-sample detector predictions; an out-of-fold or held-out gate-training split is a planned robustness check against detector/gate overfitting. The prototype uses a fixed final gate epoch count; tune it using dev data only.

Uncalibrated segmentation check:

```bash
python scripts/evaluate.py --manifest data/manifest.csv --predictions runs/prob_seed0 --split test --pixel-baseline --threshold .5 --out runs/pixel_test.json
python scripts/benchmark.py --device cuda --size 256 --out runs/runtime_256.json
```

The benchmark uses random images solely for timing. It excludes disk IO and learned-gate costs; the output states FLOP exclusions. Final manuscript latency must include the actual gate and all requested preprocessing, measured separately. Batch-one native-size training deliberately avoids interpolation and augmentation for debugging. It is not a competitive published training recipe.

## Reproducibility and limitations

- Three seeds: 0,1,2; record manifest/archive SHA256 and upstream source revisions before real runs.
- Preserve test/cal separation, exact 8-connectivity, centroid ≤3 pixels, maximum-cardinality one-to-one matching, and all metric denominators.
- Keep native pixel and object metrics separate. `null` means an undefined denominator, not zero accuracy.
- No SLS, SD, AC-SLSIoU, CG-IRNet, graph GNN or full temporal tracker is claimed implemented. Their validation remains in the experiment plan.
- The included CP/union-bound and fixed-sequence procedures are known statistical machinery; proposed novelty and empirical utility remain unverified.
- The MLP is a staged candidate selector, not end-to-end differentiable connected-component detection.
- This package is a local repository scaffold. No GitHub repository has been published or external compute job started.

## Alternative calibration procedure

Add `--calibration-method fixed_sequence` to the calibration command to test the predeclared thresholds from highest to lowest and stop at the first failure. The default remains `bonferroni` for reference. Count risk uses the Hoeffding KL bound on FP/cap; any-alarm risk uses an exact binomial lower-tail test. The output reports a certified risk budget, not a simultaneous table of confidence bounds. Choose the procedure before calibration; selecting whichever procedure works on the same labels requires its own error allocation.

At alpha=.01 and delta=.05, zero observed errors need 299 independent images for the fixed-sequence any-alarm test, or 19,172 for count risk with cap=64. These are procedure-specific calculations, not universal lower bounds. See mathematical_analysis.md.