Auto Router

Tabnetics 1.1.0 ships a packaged V25 calibrated score-router as an explicit opt-in in DFFSConfig. The router is a small scikit-learn/joblib artifact bundled inside the Python package under tabnetics.auto_router; it does not need Hugging Face, network access, or a separate model download. It remains disabled by default until a frozen registered-holdout campaign supports promotion.

The router chooses among supported pipeline candidates before feature selection starts. It predicts balanced accuracy and macro-F1 for each candidate from descriptors computed directly from the user's training data, then applies a calibrated conservative policy. The descriptor intentionally excludes validation-only fields such as historical hard/easy labels, holdout membership, or any dataset identity signal.

Opt-in Usage

Enable the auto-router explicitly:

from tabnetics.pipeline import DFFSConfig, DistributionFeatureSelectionPipeline

config = DFFSConfig(random_seed=42, n_jobs=4, auto_router_enabled=True)
pipeline = DistributionFeatureSelectionPipeline(config)

result = pipeline.run(X, y, dataset_name="my_dataset", seed=42)

During run() or run_pre_split(), tabnetics computes the router descriptor on the training split only, selects a candidate profile, disables the router on the delegated inner run to avoid recursion, and records the decision in the result metadata.

The default uses explicit flags and fixed defaults:

config = DFFSConfig(auto_router_enabled=False)

Cross-Fit Uncertainty Guard

auto_router_crossfit_uncertainty_enabled=False is the default. When enabled, the router requires an embedded, identity-bound cross-fit uncertainty artifact; the packaged V25 artifact does not include one and therefore defaults without changing the pipeline. A non-default candidate must pass a positive utility lower bound, an independently calibrated beats-default threshold, minimum support, and an enabled descriptor-OOD gate. Missing, malformed, mismatched, or unsupported artifacts always select the fixed pipeline default.

This guard is an opt-in safety contract, not a promotion claim. It may be used only with an artifact trained on source-group OOF outcomes and evaluated on a separate frozen holdout.

To inspect the router directly:

from tabnetics.auto_router import predict_auto_router

decision = predict_auto_router(X_train, y_train)
print(decision.metadata["selected_candidate_id"])
print(decision.enabled_methods)

What It Can Change

V25 selects among 12 supported candidates trained from finite, observed validation profiles. The candidate surface covers:

• Method-set breadth: 5-method compact profiles, 16-method full profiles, and one 35-method broad profile.
• Distribution-fitting order: df_stage_position="after_fs" and selected before_fs candidates.
• Classifier selection: sklearn legacy, sklearn MNPO-hybrid, and FLAML/tune-first variants.
• Classifier oracle depth: classifier_oracle_k values 1, 2, and 3.

It does not freely synthesize arbitrary flags. If the router is uncertain, the calibrated policy can fall back to the current default-like candidate.

Evidence Summary

The packaged model is the V25 calibrated MLP score-router trained with 10-fold dataset-level CV. Training excluded the frozen holdout dataset IDs and used only dataset-computable descriptors plus candidate action encodings.

Evidence slice	Result
Training policy groups	513
Training datasets	57
Candidate profiles	12
Mean balanced-accuracy delta vs current default	+0.0038
Mean macro-F1 delta vs current default	+0.0053
Non-default selections	124 / 513
Policy-defaulted selections	264 / 513
Harm > 0.01 BA vs default	31 / 513
Severe harm > 0.03 BA vs default	24 / 513

The latest available frozen-router holdout evidence predates V25 and should be treated as context, not as completed V25 holdout validation: the Val-22 frozen-router predecessor was negative on the primary-decision holdout slice (mean BA delta -0.0139 over 45 dataset-seed groups) and neutral on replay. V25 therefore remains opt-in despite its conservative calibration and internal default fallback.

Source-lineage training contract

The trainer accepts --source-manifest /path/to/attested-sources.json as an opt-in source-grouping contract. It closes held-out IDs over transitive same-source alias/derivative components before candidate construction, target aggregation or descriptor fitting. Model CV and descriptor-OOD leave-out diagnostics then exclude complete source groups. Unknown IDs, unanchored/conflicting components, duplicate JSON assignments, and fewer than two retained sources fail closed. Known registry base_dataset edges and the audited NCI60 strict/regular loader alias are mandatory consistency checks.

A minimal synthetic example, not a biological attestation:

{
  "schema": "tabnetics_router_source_lineage_v1",
  "attestation": {
    "authority": "synthetic-example",
    "evidence_refs": ["Replace with reviewed study/loader/manifest evidence"],
    "scope": "declared_source_lineage_not_sample_identity"
  },
  "dataset_sources": {
    "study_a_table": "study_a",
    "study_b_table": "study_b",
    "study_c_table": "study_c"
  },
  "aliases": [["study_a_table", "study_a_derived_table"]]
}

Use public study tokens, not participant identifiers. Every connected component needs a source anchor; all supplied evidence references and the attesting authority are retained. The manifest byte digest, canonical closure, training/excluded membership and CV fold-plan digests bind generated reports and artifact metadata. Archive the original attested manifest bytes with each actual run; a digest alone is not a recoverable copy. Source IDs are split/provenance metadata, never model descriptor inputs. Declarations do not certify sample-level disjointness: ambiguous panel overlap needs additional provenance and conservative grouping. The fixture above is intentionally incomplete for any real corpus.

This increment does not fix model-family/action-set/policy selection reuse of development outcomes. Its output is labeled source_group_oof_development_not_nested_or_frozen_confirmation. Nested meta-selection and a genuinely fresh source-disjoint frozen panel remain required, as does the separate runtime uncertainty artifact contract. Without the flag, the old literal-ID behavior remains unchanged. The packaged V25 weights are not regenerated or certified by these checks; their historical holdout exclusions were literal IDs and do not establish source closure.

Short CPU tests on a lab host: PYTHONPATH=core/src:. python -m pytest -q core/tests/test_router_source_identity.py meta/tests/test_train_score_expanded_router_ood.py.

Source-nested retrospective score-router training

--nested-source-evaluation --source-manifest ... selects a separate, default-off training path in meta/nested_score_router.py. --nested-outer-splits controls outer source folds; --n-splits controls inner source folds. Exact requested fold counts must be possible; there is no silent literal-ID fallback or fold-count reduction. All aliases/derivatives declared by the source contract stay together. Source declarations still do not certify sample-level disjointness.

Each outer-training partition independently rebuilds the candidate library, recorded default configuration, action encodings, support counts, descriptor scaling/clipping, models and OOD envelope. Each inner-training partition also rebuilds its library. Inner OOF residuals calibrate the existing heuristic offsets; they are not confidence intervals for paired utility. Policy thresholds, beats-default thresholds, support penalties and family selection use only inner evidence. Every grid is evaluated through ScoreExpandedRouter.predict, including its minimum-gain, cost tie-breaking, probability and OOD decisions. New nested artifacts enable the fitted OOD gate unless explicitly disabled. Candidate order is canonical so runtime tie behavior is reproducible.

The query roster precedes outcome filtering. Prediction uses the entire fitted candidate library before looking up selected/default outcomes. An eligible inner grid/family requires finite paired evidence for every requested query; it cannot improve its mean by dropping an unobserved selected action. There is an explicit all-default candidate policy, eligible only with complete default evidence. An infeasible best grid is never substituted. Single-class beats-default targets use their fitted constant prior, not implicit probability one. Outer missing pairs and failed folds remain visible, with requested, predicted/available, paired, source and dataset denominators. Available-pair statistics are labeled incomplete and cannot establish completed evaluation. Consumers must check top-level complete_evaluation; the aggregate summary also retains all requested queries when folds fail, with unpredicted counts explicit. Oracle regret requires all fitted candidate outcomes for that query.

Candidate IDs hash methods plus every nonmissing recorded configuration field, not just the legacy short signature. Conflicting default configurations, classification aliases, unrecognized fields, active omitted meta-selector behavior and inconsistent duplicate outcomes are rejected or unavailable; scores are never reconstructed from accuracy/F1 aliases or missing zeros. This is an exact recorded-configuration identity only: absent values do not attest historical execution defaults. Supplied descriptors currently come from the historical full-dataset cache, not a bound support-only split. Accordingly, all new artifacts/reports are explicitly retrospective and conditional on these inputs, with promotion_eligible=false. They do not qualify a complete fitted pipeline, effective historical configuration, sample-disjoint confirmation, runtime uncertainty guarantee, or SOTA result.

The final artifact repeats the same inner fit/selection procedure on all development data; it does not use outer scores to choose the final family or policy. Outer reports identify the evaluated procedure, not an independent test of those final fitted weights. Reports bind input/config/source-lineage and fold-plan hashes, candidate/fit identities and source-file/environment records. The final artifact is written under <artifact-dir>/nested_source; the report is <report-dir>/<prefix>_NESTED_SOURCE.json. Existing outputs cause rejection, so use fresh output roots. Failed outer folds are persisted even if final fitting fails. The original trainer path and shipped V25 artifacts remain unchanged.

A runnable opt-in command, after real source attestation, descriptor/configuration review and a resource-approved issue run specification, is:

PYTHONPATH=core/src:. OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 \
python -m meta.train_score_expanded_router \
  --source-manifest /path/to/router-inputs/attested-sources.json \
  --nested-source-evaluation --nested-outer-splits 5 --n-splits 5 \
  --n-jobs 1 \
  --artifact-dir /path/to/router-nested-run/artifacts \
  --report-dir /path/to/router-nested-run/reports

This command may load/download the declared run corpus and build dataset descriptors; it is not a claim that a real nested campaign has run. Run CPU checks on an approved CPU host and GPU work on an approved GPU host. Run PYTHONPATH=core/src:. python -m pytest -q meta/tests/test_nested_score_router.py on the CPU lab for tiny synthetic source/row/config/missingness and actual runtime/reload tests. Runtime/reload parity here means meta.ScoreExpandedRouter.predict and its own save/load surface, not automatic equivalence of the packaged core wrapper, apply_router_output, or the complete fitted pipeline. Fresh source-attested router weights, support-only descriptor provenance and frozen full-profile/external evidence remain separate deliverables.

Rationale

The validation campaigns showed that a single static default is serviceable but leaves value on the table: some datasets prefer compact feature-selection stacks, some prefer broader portfolios, and a smaller number prefer alternative distribution-stage or classifier-oracle settings. Manual flag selection is not a good user interface for that evidence.

The auto-router moves those decisions into a reproducible model:

• It uses features available on any new dataset.
• It chooses only from profiles that have actually been run.
• It optimizes both balanced accuracy and macro-F1.
• It applies calibrated lower-confidence behavior instead of chasing raw predicted gains.
• It keeps explicit opt-in support for controlled evaluation and ablations.

The current recommendation is to keep fixed defaults for ordinary library usage and set auto_router_enabled=True only for controlled V25 evaluation. Promotion requires a passing frozen registered-holdout campaign with paired portfolio and per-tier harm analysis.

---

Documentation and webpages on this site are generated from authoritative internal sources using a combination of deterministic rules and generative AI. Errors are possible. Please report issues via GitHub Discussions or email [email protected].