Description
Prediction (The Statistical Trap, Probability, Generalization, Repeatability). The Holdout Method (Stability, Internal Validity). Machine Learning Benchmarks (History of Benchmark Design, Evaluation Failures, Dynamic Benchmarks, Multi-Task Benchmarking, Sensitivity). External Validity, Reproducibility, Robustness (Distribution Shift). Adaptivity and Overfitting (Test Set Reuse). Construct Validity (Context, Scope, Lakatosian Defense). Evaluating Uncertainty (Calibration Error, Prediction Sets, Conformal Inference). Forecast Evaluation (Scoring Rules, Accounting for Uncertainty, Confidence Sequences). Games as Benchmarks (History of Benchmark Games). Evaluating Interventions (Temporal Validity, Compliance, Adjustment). Regulation and Deployment (Program Evaluation, Policy Evaluation, Auditing via Sequential Hypothesis Testing).