tiny_object_detection/docs/experiment_plan.md

Experimental plan — no benchmark runs completed

Dataset audit and access status

All counts below are source-reported, not locally measured. Download links were located; dataset payloads were not downloaded. No target-size histogram, annotation accuracy, or license entitlement is inferred from a paper's description.

Dataset / source	Verified metadata and protocol evidence	Rights/download status	Required audit / use
SIRST, often called NUAA-SIRST in other work	Original author requests the name SIRST. Repository contains idx_320, idx_427, images.zip, masks.zip; multiple protocols exist	Public files visible; no dataset license established in inspected page	Use exact original split filenames/hashes; 427-image count widely reported but verify actual payload. Real images; inspect close pairs and omitted dim targets
NUDT-SIRST / DNANet	Author reports 1,327 synthetic images, 256×256; training command uses 50_50	MIT repository license visible; separate dataset scope not established; Baidu download listed	Check synthesis/background reuse across splits; compute mask component areas; synthetic annotation precision is not real sensor validity
IRSTD-1k / ISNet	CVF paper says 1,000, repository 1,001 manually labeled images; no silent reconciliation	Repository advertises dataset MIT agreement, Drive/cloud links; archive agreement not inspected	Use manifest count and published baseline split; document 800/200 versus 800/201 variants, never mix scores. Audit boundary ambiguity
MIST	Synthetic targets in real backgrounds; 78 training/42 testing sequences; 11-sequence hard subset. Masks, boxes and points listed	MIT code repository; dataset license scope unresolved; Drive link present	Preserve sequence splits; val_all/val_hard are test naming in release, not freely tunable dev data. Test rapid/irregular motion
NUDT-MIRSDT via DeepPro	README reports 120 sequences but specifies 80 train +20 test	Drive/Baidu links; license unresolved	Resolve remaining 20 sequences and exact split files. Do not invent an 80/40 split
IRDST	142,727 frames: 40,650 real in 85 scenes; 102,077 synthetic in 317 scenes. Masks, boxes, center labels; boxes padded two pixels beyond mask boundary	Author page with real/synthetic Drive/Baidu links and sequence-version link; license not stated in inspected page	Keep scenes disjoint; use real subset for external robustness. Mask-derived boxes differ from tight boxes
DAUB	Dataset landing page resolves; HyperTea links it	Page body unreadable; license, counts, split and payload unverified	Do not treat as ready-to-use; verify record and annotations first
CSIST-100K	DISTA-Net provides close-target unmixing ecosystem and subpixel metrics	Repository indexed; dataset license/download not tested	Essential adversarial test for A; optional specialized extension for C, not interchangeable with mask-segmentation benchmarks

The first three are single-frame benchmarks and cannot validate temporal/subpixel claims. Source occupancy below a pixel does not imply a one-pixel binary mask; report optical PSF and source location separately. MIST is useful but not a substitute for real video validation.

Data validation before any training

Create manifest.csv with id,image,mask,split,group. Preserve the published test split. From the original training data, allocate independent scene groups to train/dev/cal, with a documented fixed seed and counts. Label comparisons under this reduced-training protocol separately from full-training published scores; retrain every baseline using the same split. Do not use test images for calibration. Calibration-size requirements may force an additional authorized exposure dataset.

Run scripts/audit_data.py. It measures native area bins 1–4, 5–9, 10–25, ≥26 pixels, area quantiles, object counts, empty-image proportion, resolutions, nearest-centroid separations, file hashes and group leakage. It does not establish missing annotations or near-duplicates. Manually inspect all very small components and a stratified sample of at least 50 images, including large clutter, empty frames, close pairs and saturation. Record mask edits as a separate version; report official-label results separately. Check decoded-image hashes/perceptual near-duplicates and source sequence overlap across datasets before transfer evaluation. No such inspection has yet run on real data.

Baseline ladder and implementation status

Baseline / method	Current implementation	Required role
Compact one-pool U-Net + BCE, Dice, BCE+Dice, focal or soft IoU	Written from scratch; PyTorch runtime not available, gradient tests skipped	Sanity/falsification baseline, not published SOTA reproduction
Frozen candidate max-probability score + calibration	Implemented and CPU pipeline tested with synthetic fixtures	Measures benefit of calibration without learned gate
Same candidates + BCE MLP gate + same calibration	Training code prepared, not executed	Controls for capacity/extra stage
Same candidates + constrained matching-aware MLP, with/without area weight	Prototype prepared; pointwise bound CPU tests pass	Tests proposed mechanism
MSHNet + SLS, SD Loss, AC-SLSIoU, CG-IRNet	Sources located; not ported/reproduced	Required close prior comparators, under identical train/calibration splits
NWD and GIoU/DIoU/CIoU	Not implemented in mask prototype	Box-level comparison only with common box head and assignment; do not substitute a mask formula and call it NWD
A: MLP / graph gate / NMS / exact MWIS on same candidates	Exact tiny MWIS oracle only	Test edge correctness, isolated false positives and recall retention
C: single-frame, temporal mean, DP/TBD, DeepPro, MISTNet, HyperTea	Tiny complete-path DP oracle only	All require identical window/causality and explicit release/protocol audit

BCE averages pixel log loss; Focal uses α=.25 and γ=2; soft Dice and IoU use epsilon 1e-6. These definitions are explicit in models.py; focal variants must not be silently changed. Use known baseline source code for SLS/SD after checking licenses and upstream revisions. No third-party implementation is bundled or relabeled.

Evaluation specification

Use native-resolution masks. Main segmentation operating point is probability ≥0.5; diagnostic curves use a predetermined 101-threshold grid 0,.01,…,1. Never choose the final operating point using test labels. For calibrated object selection, proposals are frozen at base threshold .1 with 8-connectivity, retain at most K=64 by component peak score (raster-order tie break), then delete entire components by gate score. The pixel and object thresholds are separate. Default numbers are development hypotheses, not validated settings.

Ground-truth objects are 8-connected mask components. Eligibility: centroid Euclidean distance ≤3 native pixels. Maximum-cardinality bipartite matching enforces one-to-one assignment; the prototype's deterministic augmenting-path ordering resolves ties. A merged prediction can match at most one target. Centroid gating alone can credit a poor mask, hence always report overlap metrics and an overlap-gated sensitivity analysis. Official author metric implementations must also be run for paper comparability; do not label this maximum-matching implementation as identical to a legacy greedy matcher.

Metric	Exact definition / convention
IoU	Sum intersection pixels / sum union pixels over images; all-empty denominator → null
nIoU	Arithmetic mean per-image IoU; both masks empty → 1. Also report nonempty-target subset separately in final study
Pixel F1	2Σ intersection/(Σ predicted pixels+Σ GT pixels); zero denominator → null
Pd	Matched GT objects / total GT objects; no GT → null
Object F1	2TP/(2TP+FP+FN), from one-to-one matching; zero denominator → null
FP/image	Unmatched predicted component count / number of images; primary budget metric
P(any FA)	Fraction of images with at least one unmatched predicted object; secondary, distinct risk
Fa_pixel	Predicted background pixels / total image pixels; report raw fraction and ×10^-6, not FP object count
Fa_unmatched_area	Total pixels in unmatched predicted components / total image pixels; deterministic matching tie rule disclosed
Latency	Batch 1 native 256 and 512 dimensions; 20 warm-up +100 timed iterations; CUDA synchronization; FP32 reference; separately FP16 only if tested
Cost	Trainable parameters, convolution MACs (one multiply-accumulate), FLOPs=2×MACs with excluded operators listed; peak allocated VRAM

Do not call all “Fa” measures equivalent. Matched-component overflow pixels contribute to Fa_pixel but not Fa_unmatched_area. The NeurIPS 2025 PyIRSTDMetrics toolkit provides additional matching/hybrid analyses; cross-check once its source revision is pinned. Threshold sweeps are descriptive test curves, not a source for selecting published operating thresholds.

Report scene-bootstrap uncertainty for descriptive test metrics, while keeping the iid finite-sample calibration theorem separate. Repeat training seeds 0,1,2. Unit tests are not benchmark evidence. For extremely low rates report exposure, observed counts and one-sided intervals. Include all-background real images; positive-only benchmarks miss important operating conditions.

Single RTX 3090 plan

For B, begin with native 256/512 inputs, width 16, batch 1, FP32; the compact baseline is expected to fit, but no VRAM measurement has run. Profile a batch before scaling. Gate training uses five cached component features and is inexpensive on CPU. Record wall time and memory; do not estimate completed GPU-hours. Use larger batches/AMP only as a separately logged change. The prototype intentionally has no augmentation, resizing or AMP to simplify initial verification; stronger baselines must use a matched documented augmentation regime.

For C, begin with 5 frames, ≤64 candidates/frame, bounded-degree motion edges and batch 1. Test windows 3/5/9 only after profiling. MIST upstream uses two 24 GB GPUs; microbatching and accumulation do not exactly preserve BatchNorm behavior. Its older PyTorch/NATTEN stack should live in a separate environment. Window changes and online vs offline access must be disclosed. Do not claim a faithful one-GPU reproduction solely because the model executes.

Ablations and stop conditions

Run configs/ablations.csv in the stated order. Primary comparisons share detector probabilities, proposal masks, K, calibration data and matching rule. Mandatory factors: no gate, BCE gate, constrained gate, no size weights; fixed penalty vs dual updates; actual-match risk vs canonical bound tightness; raw pixel negative penalty; proposal recall/cap/base-threshold changes; one-to-one vs legacy matching; separate calibration vs empirical-only threshold; K=16/32/64; close-pair distance; background-only scenes; distribution shift without recalibration.

Development grid: α_count=.1,.5,1 FP/image, α_any=.01,.05,.1; δ=.05. Each certificate is per declared risk/budget/pipeline. If claiming simultaneous validity across multiple trained models, risk types or budgets, include them in multiplicity accounting or reserve an independent final audit set. The prototype certificate covers its single frozen model and threshold family only. Models and budgets cannot be repeatedly selected by reusing calibration outcomes without adjustment.

Continue only if a useful certificate exists and constrained training improves recall over equally calibrated baselines. If zero-output selection dominates, first fix sample adequacy or target risk scope. Never relax α after seeing the test set and present it as the original budget.

8–12-week work packages

• Weeks 1–2: close-prior full texts, code rights/revisions, data audit, sample-size decision, exact split manifests.
• Weeks 3–4: compact baselines and gate comparisons, three seeds, constraint feasibility and calibration coverage on controlled simulation.
• Weeks 5–6: verified SLS/SD and recent-loss comparisons; second backbone; add hard real negatives.
• Weeks 7–8: cross-dataset and close-pair stress tests; independent metric audit; make go/no-go decision.
• Weeks 9–12: robust repeats where justified, full limitation analysis, manuscript and reproduction package.

Calibration and upstream audit update

Predeclare either Bonferroni tail calibration or fixed-sequence testing; compare training methods under the same choice. For a formal comparison that selects the best procedure using calibration labels, allocate the total delta across procedures or use a fresh independent calibration set. Record procedure, risk type, cap, order, alpha, delta and accepted threshold in each run. Do not treat any-alarm probability as expected object count.

The SLS/SD loss-file audit is recorded in evidence/upstream_audit.md. Implement parity checks for native resolution, empty labels, multi-target images, batch reduction and warmup before labeling a baseline faithful. Their source availability is verified, execution is not. EDGRU's repository revision is now recorded; its license is not established by source visibility.