Validation

Validation planning, sharding, suite execution, and campaign-specific job builders.

Package source: tabnetics.validation

Package overview

Validation planning, suite, and shard execution surface behind tabnetics-validation-plan, tabnetics-validation-shard, and tabnetics-validation-suite; evidence-bearing runs use the HuggingFace bundle as the authoritative operational mirror of the public upstream datasets and default to dataset_integrity_policy="error".

Stable exports

CORE_PROJECT_ROOT (constant) - Source. Module-level constant exported by the package surface.
REPO_ROOT (constant) - Source. Module-level constant exported by the package surface.
DEFAULT_MAX_PODS (constant) - Source. Module-level constant exported by the package surface.
VALIDATION_SEEDS (constant) - Source. Module-level constant exported by the package surface.
DEPRECATED_VALIDATION_EXCLUSIONS (constant) - Source. Module-level constant exported by the package surface.
class Job (class) - Source.
class BenchmarkProfile (class) - Source.
class TabArenaPlanProfile (class) - Source.
def build_jobs_legacy(*, include_feature_smoke: bool = True, include_validation_suite: bool = True) -> List[Job] (function) - Source.
def build_jobs_validation1(*, dataset_shards: int = 8) -> List[Job] (function) - Source. Validation-1: full-catalog 5-seed runs for baseline + single-feature toggles.
def build_jobs_validation4(*, dataset_shards: int = 6) -> List[Job] (function) - Source. Validation-4: 3-profile decomposition + PLS-DA guardrail on 6 pods.
def build_jobs_validation5(*, dataset_shards: int = 6, val4_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-5: baseline anchor vs complete B+C+D candidate on 6 pods.
def build_jobs_validation6(*, dataset_shards: int = 6, val5_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-6: two-profile signal check on the 67-dataset Val-5 catalog.
def build_jobs_validation7(*, dataset_shards: int = 6, val6_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-7: classifier upgrade + continuous shrinkage on the 67-dataset catalog.
def build_jobs_validation8(*, dataset_shards: int = 6, val7_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-8: all approved fixes + Stage-2 classifier support on 67 datasets.
def build_jobs_validation9(*, dataset_shards: int = 6, val8_root: Optional[Path] = None, runtime_profile: str = 'full') -> List[Job] (function) - Source. Validation-9: comprehensive full-stack run with classifier selection A/B.
def build_jobs_validation10(*, dataset_shards: int = 6, val9_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-10: simple-vs-MNPO usage across all pipeline stages.
def build_jobs_validation11(*, dataset_shards: int = 4, val10_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-11 v2: 2×2 factorial stage decomposition + extensions.
def build_jobs_validation12(*, dataset_shards: int = 6, val11_root: Optional[Path] = None, val10_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-12: regime-conditional gating comparison.
def build_jobs_validation13(*, dataset_shards: int = 8, val12_root: Optional[Path] = None, val11_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-13: gate fixes + profile expansion + ablation + FS-fraction sweep.
VALIDATION14_DATASETS (constant) - Source. Module-level constant exported by the package surface.
VALIDATION14_ACTIVATION_DATASETS (constant) - Source. Module-level constant exported by the package surface.
VALIDATION14_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation14(*, dataset_shards: int = 6, val13_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-14 full matrix (20 profiles x 35 datasets x 9 seeds).
def build_jobs_validation14_activation_smoke(*, dataset_shards: int = 6, val13_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-14 activation smoke (same profiles, 6 datasets, single seed).
VALIDATION15_DATASETS (constant) - Source. Module-level constant exported by the package surface.
VALIDATION15_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation15(*, dataset_shards: int = 6, val14_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-15 focused matrix (9 profiles x 35 datasets x 9 seeds).
VALIDATION16_DATASETS (constant) - Source. Module-level constant exported by the package surface.
VALIDATION16_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION17_DATASETS (constant) - Source. Module-level constant exported by the package surface.
VALIDATION17_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation16(*, dataset_shards: int = 9, val15_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-16 focused matrix (11 profiles x 64 datasets x 9 seeds).
def build_jobs_validation17(*, dataset_shards: int = 9, val15_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-17 focused matrix: Val-16 profile rerun on the packaged after-FS default.
VAL18_FULL64 (constant) - Source. Module-level constant exported by the package surface.
VAL18_DIAG24 (constant) - Source. Module-level constant exported by the package surface.
VAL18_SINGLETON_METHODS (constant) - Source. Module-level constant exported by the package surface.
VAL18_CLASSIFIER_UNIVERSE (constant) - Source. Module-level constant exported by the package surface.
VAL19_ADDED_CLASSIFIERS (constant) - Source. Module-level constant exported by the package surface.
VAL19_HDLSS_EXTREME_OLD (constant) - Source. Module-level constant exported by the package surface.
VAL19_HDLSS_EXTREME_NEW (constant) - Source. Module-level constant exported by the package surface.
VAL19_HDLSS_MODERATE_OLD (constant) - Source. Module-level constant exported by the package surface.
VAL19_HDLSS_MODERATE_NEW (constant) - Source. Module-level constant exported by the package surface.
VAL19_HDLSS_MODERATE_CPU_OLD (constant) - Source. Module-level constant exported by the package surface.
VAL19_HDLSS_MODERATE_CPU_NEW (constant) - Source. Module-level constant exported by the package surface.
VAL18_CPU_HOSTS (constant) - Source. Module-level constant exported by the package surface.
VAL18_CPU_HOST_CAPACITY_CORES (constant) - Source. Module-level constant exported by the package surface.
VAL18_GPU_HOSTS (constant) - Source. Module-level constant exported by the package surface.
VAL18_HOST_WORKER_TARGETS (constant) - Source. Module-level constant exported by the package surface.
VAL19_HOST_WORKER_TARGETS (constant) - Source. Module-level constant exported by the package surface.
VAL20_CPU_HOSTS (constant) - Source. Module-level constant exported by the package surface.
VAL20_GPU_HOSTS (constant) - Source. Module-level constant exported by the package surface.
VAL20_TABARENA_HOSTS (constant) - Source. Module-level constant exported by the package surface.
VAL20_HOST_CAPACITY_CORES (constant) - Source. Module-level constant exported by the package surface.
VAL20_HOST_WORKER_TARGETS (constant) - Source. Module-level constant exported by the package surface.
TABARENA_DEFAULT_OFFICIAL_FOLDS (constant) - Source. Module-level constant exported by the package surface.
VALIDATION18_ANCHORS_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation18_anchors(*, dataset_shards: int = 9, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-18 Family A: 8 anchor/control profiles x 64 datasets x 9 seeds.
VALIDATION18_SINGLETONS_PROFILE_IDS (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation18_singletons(*, dataset_shards: int = 9, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-18 Family M: 39 methods x 2 variants (raw/scaffold) x 64 datasets x 5 seeds.
VALIDATION18_MNPO_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation18_mnpo(*, dataset_shards: int = 9, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-18 Family N: 25 MNPO/oracle profiles x 64 datasets x 5 seeds.
VALIDATION18_STAGE_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation18_stage(*, dataset_shards: int = 7, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-18 Families D+P+S(D/P): stage/scaffold matrix x 24 diagnostic datasets x 5 seeds.
VALIDATION18_CLASSIFIERS_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation18_classifiers(*, dataset_shards: int = 7, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-18 Family C + S04: CPU classifier sweep plus TabPFN-only reruns.
VALIDATION19_CLASSIFIERS_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation19_classifiers(*, dataset_shards: int = 7, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-19 required surface: 7 singleton diagnostics + 4 pool reruns + 2 oracle-compat controls.
VALIDATION20_ENSEMBLE_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_ORACLE_DIAGNOSTIC_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_EXTENSION_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation20_ensemble(*, dataset_shards: int = 7, val17_root: Optional[Path] = None, include_oracle_diagnostics: bool = True) -> List[Job] (function) - Source. Val-20 extension lane: Family E + classifier-MNPO diagnostics on DIAG24.
VAL20_CORE_CLASSIFIER_POOL (constant) - Source. Module-level constant exported by the package surface.
VAL20_EXPANDED_CLASSIFIER_POOL (constant) - Source. Module-level constant exported by the package surface.
VAL20_EXPANDED_CLASSIFIER_POOL_NO_TABPFN (constant) - Source. Module-level constant exported by the package surface.
VAL20_CUSTOM_FLAML_FAMILIES (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_BRIDGE_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_FLAML_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_TUNE_FIRST_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_TABPFN_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_PIPELINE_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_LR_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_WAVE1_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_WAVE2_RESERVED_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_WAVE3_RESERVED_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_TABARENA_W1_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_TABARENA_W2_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_TABARENA_W3_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_TABARENA_PLAN_KINDS (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation20_tabarena_w1(*, dataset_shards: int = 8) -> List[Job] (function) - Source. Val-20 TabArena Wave 1: probe refresh vs competitive challenger.
def build_jobs_validation20_tabarena_w2(*, dataset_shards: int = 8) -> List[Job] (function) - Source. Val-20 TabArena Wave 2: full-catalog refresh and challengers.
def build_jobs_validation20_tabarena_w3(*, dataset_shards: int = 8) -> List[Job] (function) - Source. Val-20 TabArena Wave 3: final chase between the top candidates and baseline.
def build_jobs_validation20_wave1(*, dataset_shards: int = 8, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-20 Wave 1: the currently runnable main-validation Val-20 lane.
VALIDATION18_CLS_ORACLE_WT_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation18_cls_oracle_wt(*, dataset_shards: int = 7, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-18 Family W: classifier-oracle weighting sweep on DIAG24.
def build_jobs_validation13_clf_oracle(*, dataset_shards: int = 4, val12_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-13 classifier oracle comparison — standalone 2×2 factorial.
VAL21_CPU_HOSTS (constant) - Source. Module-level constant exported by the package surface.
VAL21_GPU_HOSTS (constant) - Source. Module-level constant exported by the package surface.
VAL21_HOST_CAPACITY_CORES (constant) - Source. Module-level constant exported by the package surface.
VAL21_HOST_WORKER_TARGETS (constant) - Source. Module-level constant exported by the package surface.
VAL21_RV_HOLDOUT (constant) - Source. Module-level constant exported by the package surface.
VAL21_SPEC_TEMPLATE_PATH (constant) - Source. Module-level constant exported by the package surface.
VALIDATION21_PHASE1_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION21_PLAN_KINDS (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation21_phase1(*, dataset_shards: int = 8, val17_root: Optional[Path] = None, spec_path: Optional[Path] = None) -> List[Job] (function) - Source.
def build_jobs_validation21_phase2(*, dataset_shards: int = 4, val17_root: Optional[Path] = None, spec_path: Optional[Path] = None) -> List[Job] (function) - Source.
def main() -> None (function) - Source.
FS_BASE_METHODS (constant) - Source. Module-level constant exported by the package surface.
class LoadedTabularDataset (class) - Source.
class DistributionCase (class) - Source.
class AblationConfig (class) - Source.
def extract_meta_features(X: np.ndarray, y: np.ndarray) -> Dict[str, float] (function) - Source. Extract dataset meta-features useful for tier assignment / analysis.
CATALOG (constant) - Source. Module-level constant exported by the package surface.
DATASET_SETS (constant) - Source. Module-level constant exported by the package surface.
COMPONENT_DEFAULTS (constant) - Source. Module-level constant exported by the package surface.
NOOP_COMPONENTS (constant) - Source. Module-level constant exported by the package surface.
def load_feature_selection_dataset(spec: ValidationDatasetSpec, seed: int, allow_synthetic_fallback: bool, sample_cap: int, feature_cap: int, source_policy: Optional[str] = None, class_integrity_policy: str = 'error', class_min_classes: int = 2, class_min_class_count: int = 1, require_hf_source: bool = False) -> LoadedTabularDataset (function) - Source.
def resolve_dataset_ids(catalog: Dict[str, ValidationDatasetSpec], dataset_sets: Sequence[str], explicit_ids: Sequence[str], exclude_ids: Sequence[str], pipelines: Sequence[str], max_datasets: Optional[int]) -> List[str] (function) - Source.
def build_ablation_configs(profile: str, base_components: Dict[str, bool], pipelines: Sequence[str], constrained_components: Optional[Sequence[str]] = None) -> List[AblationConfig] (function) - Source.
def run_validation_suite(args: argparse.Namespace) -> Path (function) - Source.
def build_arg_parser() -> argparse.ArgumentParser (function) - Source.

tabnetics.validation.core.shard_runner - Source. Run a single shard from a generated validation plan, backing tabnetics-validation-shard and consuming the plan_*.json / shards_*.json artifacts emitted by tabnetics-validation-plan.

Module details

`tabnetics.validation.init`

Source file

No top-level public symbols are exported directly from this module.

`tabnetics.validation.generate_plan`

Source file

Generate a sharded validation plan (up to 20 pods) for tabnetics-validation-plan, aligned with the promoted df_stage_position="after_fs" runtime and the stricter HuggingFace public-source mirror policy.

CORE_PROJECT_ROOT (constant) - Source. Module-level constant exported by the package surface.
REPO_ROOT (constant) - Source. Module-level constant exported by the package surface.
DEFAULT_MAX_PODS (constant) - Source. Module-level constant exported by the package surface.
VALIDATION_SEEDS (constant) - Source. Module-level constant exported by the package surface.
DEPRECATED_VALIDATION_EXCLUSIONS (constant) - Source. Module-level constant exported by the package surface.
class Job (class) - Source.
class BenchmarkProfile (class) - Source.
class TabArenaPlanProfile (class) - Source.
def build_jobs_legacy(*, include_feature_smoke: bool = True, include_validation_suite: bool = True) -> List[Job] (function) - Source.
def build_jobs_validation1(*, dataset_shards: int = 8) -> List[Job] (function) - Source. Validation-1: full-catalog 5-seed runs for baseline + single-feature toggles.
def build_jobs_validation4(*, dataset_shards: int = 6) -> List[Job] (function) - Source. Validation-4: 3-profile decomposition + PLS-DA guardrail on 6 pods.
def build_jobs_validation5(*, dataset_shards: int = 6, val4_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-5: baseline anchor vs complete B+C+D candidate on 6 pods.
def build_jobs_validation6(*, dataset_shards: int = 6, val5_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-6: two-profile signal check on the 67-dataset Val-5 catalog.
def build_jobs_validation7(*, dataset_shards: int = 6, val6_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-7: classifier upgrade + continuous shrinkage on the 67-dataset catalog.
def build_jobs_validation8(*, dataset_shards: int = 6, val7_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-8: all approved fixes + Stage-2 classifier support on 67 datasets.
def build_jobs_validation9(*, dataset_shards: int = 6, val8_root: Optional[Path] = None, runtime_profile: str = 'full') -> List[Job] (function) - Source. Validation-9: comprehensive full-stack run with classifier selection A/B.
def build_jobs_validation10(*, dataset_shards: int = 6, val9_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-10: simple-vs-MNPO usage across all pipeline stages.
def build_jobs_validation11(*, dataset_shards: int = 4, val10_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-11 v2: 2×2 factorial stage decomposition + extensions.
def build_jobs_validation12(*, dataset_shards: int = 6, val11_root: Optional[Path] = None, val10_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-12: regime-conditional gating comparison.
def build_jobs_validation13(*, dataset_shards: int = 8, val12_root: Optional[Path] = None, val11_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-13: gate fixes + profile expansion + ablation + FS-fraction sweep.
VALIDATION14_DATASETS (constant) - Source. Module-level constant exported by the package surface.
VALIDATION14_ACTIVATION_DATASETS (constant) - Source. Module-level constant exported by the package surface.
VALIDATION14_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation14(*, dataset_shards: int = 6, val13_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-14 full matrix (20 profiles x 35 datasets x 9 seeds).
def build_jobs_validation14_activation_smoke(*, dataset_shards: int = 6, val13_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-14 activation smoke (same profiles, 6 datasets, single seed).
VALIDATION15_DATASETS (constant) - Source. Module-level constant exported by the package surface.
VALIDATION15_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation15(*, dataset_shards: int = 6, val14_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-15 focused matrix (9 profiles x 35 datasets x 9 seeds).
VALIDATION16_DATASETS (constant) - Source. Module-level constant exported by the package surface.
VALIDATION16_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION17_DATASETS (constant) - Source. Module-level constant exported by the package surface.
VALIDATION17_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation16(*, dataset_shards: int = 9, val15_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-16 focused matrix (11 profiles x 64 datasets x 9 seeds).
def build_jobs_validation17(*, dataset_shards: int = 9, val15_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-17 focused matrix: Val-16 profile rerun on the packaged after-FS default.
VAL18_FULL64 (constant) - Source. Module-level constant exported by the package surface.
VAL18_DIAG24 (constant) - Source. Module-level constant exported by the package surface.
VAL18_SINGLETON_METHODS (constant) - Source. Module-level constant exported by the package surface.
VAL18_CLASSIFIER_UNIVERSE (constant) - Source. Module-level constant exported by the package surface.
VAL19_ADDED_CLASSIFIERS (constant) - Source. Module-level constant exported by the package surface.
VAL19_HDLSS_EXTREME_OLD (constant) - Source. Module-level constant exported by the package surface.
VAL19_HDLSS_EXTREME_NEW (constant) - Source. Module-level constant exported by the package surface.
VAL19_HDLSS_MODERATE_OLD (constant) - Source. Module-level constant exported by the package surface.
VAL19_HDLSS_MODERATE_NEW (constant) - Source. Module-level constant exported by the package surface.
VAL19_HDLSS_MODERATE_CPU_OLD (constant) - Source. Module-level constant exported by the package surface.
VAL19_HDLSS_MODERATE_CPU_NEW (constant) - Source. Module-level constant exported by the package surface.
VAL18_CPU_HOSTS (constant) - Source. Module-level constant exported by the package surface.
VAL18_CPU_HOST_CAPACITY_CORES (constant) - Source. Module-level constant exported by the package surface.
VAL18_GPU_HOSTS (constant) - Source. Module-level constant exported by the package surface.
VAL18_HOST_WORKER_TARGETS (constant) - Source. Module-level constant exported by the package surface.
VAL19_HOST_WORKER_TARGETS (constant) - Source. Module-level constant exported by the package surface.
VAL20_CPU_HOSTS (constant) - Source. Module-level constant exported by the package surface.
VAL20_GPU_HOSTS (constant) - Source. Module-level constant exported by the package surface.
VAL20_TABARENA_HOSTS (constant) - Source. Module-level constant exported by the package surface.
VAL20_HOST_CAPACITY_CORES (constant) - Source. Module-level constant exported by the package surface.
VAL20_HOST_WORKER_TARGETS (constant) - Source. Module-level constant exported by the package surface.
TABARENA_DEFAULT_OFFICIAL_FOLDS (constant) - Source. Module-level constant exported by the package surface.
VALIDATION18_ANCHORS_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation18_anchors(*, dataset_shards: int = 9, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-18 Family A: 8 anchor/control profiles x 64 datasets x 9 seeds.
VALIDATION18_SINGLETONS_PROFILE_IDS (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation18_singletons(*, dataset_shards: int = 9, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-18 Family M: 39 methods x 2 variants (raw/scaffold) x 64 datasets x 5 seeds.
VALIDATION18_MNPO_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation18_mnpo(*, dataset_shards: int = 9, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-18 Family N: 25 MNPO/oracle profiles x 64 datasets x 5 seeds.
VALIDATION18_STAGE_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation18_stage(*, dataset_shards: int = 7, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-18 Families D+P+S(D/P): stage/scaffold matrix x 24 diagnostic datasets x 5 seeds.
VALIDATION18_CLASSIFIERS_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation18_classifiers(*, dataset_shards: int = 7, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-18 Family C + S04: CPU classifier sweep plus TabPFN-only reruns.
VALIDATION19_CLASSIFIERS_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation19_classifiers(*, dataset_shards: int = 7, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-19 required surface: 7 singleton diagnostics + 4 pool reruns + 2 oracle-compat controls.
VALIDATION20_ENSEMBLE_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_ORACLE_DIAGNOSTIC_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_EXTENSION_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation20_ensemble(*, dataset_shards: int = 7, val17_root: Optional[Path] = None, include_oracle_diagnostics: bool = True) -> List[Job] (function) - Source. Val-20 extension lane: Family E + classifier-MNPO diagnostics on DIAG24.
VAL20_CORE_CLASSIFIER_POOL (constant) - Source. Module-level constant exported by the package surface.
VAL20_EXPANDED_CLASSIFIER_POOL (constant) - Source. Module-level constant exported by the package surface.
VAL20_EXPANDED_CLASSIFIER_POOL_NO_TABPFN (constant) - Source. Module-level constant exported by the package surface.
VAL20_CUSTOM_FLAML_FAMILIES (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_BRIDGE_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_FLAML_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_TUNE_FIRST_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_TABPFN_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_PIPELINE_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_LR_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_WAVE1_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_WAVE2_RESERVED_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_WAVE3_RESERVED_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_TABARENA_W1_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_TABARENA_W2_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_TABARENA_W3_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION20_TABARENA_PLAN_KINDS (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation20_tabarena_w1(*, dataset_shards: int = 8) -> List[Job] (function) - Source. Val-20 TabArena Wave 1: probe refresh vs competitive challenger.
def build_jobs_validation20_tabarena_w2(*, dataset_shards: int = 8) -> List[Job] (function) - Source. Val-20 TabArena Wave 2: full-catalog refresh and challengers.
def build_jobs_validation20_tabarena_w3(*, dataset_shards: int = 8) -> List[Job] (function) - Source. Val-20 TabArena Wave 3: final chase between the top candidates and baseline.
def build_jobs_validation20_wave1(*, dataset_shards: int = 8, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-20 Wave 1: the currently runnable main-validation Val-20 lane.
VALIDATION18_CLS_ORACLE_WT_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation18_cls_oracle_wt(*, dataset_shards: int = 7, val17_root: Optional[Path] = None) -> List[Job] (function) - Source. Val-18 Family W: classifier-oracle weighting sweep on DIAG24.
def build_jobs_validation13_clf_oracle(*, dataset_shards: int = 4, val12_root: Optional[Path] = None) -> List[Job] (function) - Source. Validation-13 classifier oracle comparison — standalone 2×2 factorial.
VAL21_CPU_HOSTS (constant) - Source. Module-level constant exported by the package surface.
VAL21_GPU_HOSTS (constant) - Source. Module-level constant exported by the package surface.
VAL21_HOST_CAPACITY_CORES (constant) - Source. Module-level constant exported by the package surface.
VAL21_HOST_WORKER_TARGETS (constant) - Source. Module-level constant exported by the package surface.
VAL21_RV_HOLDOUT (constant) - Source. Module-level constant exported by the package surface.
VAL21_SPEC_TEMPLATE_PATH (constant) - Source. Module-level constant exported by the package surface.
VALIDATION21_PHASE1_PROFILE_MANIFEST (constant) - Source. Module-level constant exported by the package surface.
VALIDATION21_PLAN_KINDS (constant) - Source. Module-level constant exported by the package surface.
def build_jobs_validation21_phase1(*, dataset_shards: int = 8, val17_root: Optional[Path] = None, spec_path: Optional[Path] = None) -> List[Job] (function) - Source.
def build_jobs_validation21_phase2(*, dataset_shards: int = 4, val17_root: Optional[Path] = None, spec_path: Optional[Path] = None) -> List[Job] (function) - Source.
def main() -> None (function) - Source.

`tabnetics.validation.suite`

Source file

Unified validation suite for FS, DF, and integrated pipeline benchmarks, backing tabnetics-validation-suite while enforcing the packaged dataset-integrity defaults and the HuggingFace public-source mirror workflow for evidence-bearing validation datasets.

REPO_ROOT (constant) - Source. Module-level constant exported by the package surface.
FS_BASE_METHODS (constant) - Source. Module-level constant exported by the package surface.
class LoadedTabularDataset (class) - Source.
class DistributionCase (class) - Source.
class AblationConfig (class) - Source.
def extract_meta_features(X: np.ndarray, y: np.ndarray) -> Dict[str, float] (function) - Source. Extract dataset meta-features useful for tier assignment / analysis.
CATALOG (constant) - Source. Module-level constant exported by the package surface.
DATASET_SETS (constant) - Source. Module-level constant exported by the package surface.
COMPONENT_DEFAULTS (constant) - Source. Module-level constant exported by the package surface.
NOOP_COMPONENTS (constant) - Source. Module-level constant exported by the package surface.
def load_feature_selection_dataset(spec: ValidationDatasetSpec, seed: int, allow_synthetic_fallback: bool, sample_cap: int, feature_cap: int, source_policy: Optional[str] = None, class_integrity_policy: str = 'error', class_min_classes: int = 2, class_min_class_count: int = 1, require_hf_source: bool = False) -> LoadedTabularDataset (function) - Source.
def resolve_dataset_ids(catalog: Dict[str, ValidationDatasetSpec], dataset_sets: Sequence[str], explicit_ids: Sequence[str], exclude_ids: Sequence[str], pipelines: Sequence[str], max_datasets: Optional[int]) -> List[str] (function) - Source.
def build_ablation_configs(profile: str, base_components: Dict[str, bool], pipelines: Sequence[str], constrained_components: Optional[Sequence[str]] = None) -> List[AblationConfig] (function) - Source.
def run_validation_suite(args: argparse.Namespace) -> Path (function) - Source.
def build_arg_parser() -> argparse.ArgumentParser (function) - Source.
def main() -> None (function) - Source.

Documentation and webpages on this site are generated from authoritative internal sources using a combination of deterministic rules and generative AI. Errors are possible. Please report issues via GitHub Discussions or email [email protected].

Package overview

Stable exports

Related modules

Module details

tabnetics.validation.__init__

tabnetics.validation.generate_plan

tabnetics.validation.suite

`tabnetics.validation.init`

`tabnetics.validation.generate_plan`

`tabnetics.validation.suite`