flood-lab.github.io/HydroTuring · English, Español, 中文

A benchmark that asks one question of any AI hydrologic model: does it conserve what physics says it must conserve?

Not whether it fits a hydrograph. Whether its water budget closes, its energy budget closes, and its routing conserves momentum. A model passes HydroTuring only when every criterion of every probe passes.

Bring a probe or a model, join the paper. The suite is only as good as the physics people bring to it, and only as interesting as what has been put through it. Anyone may propose either — you do not need to be invited, affiliated, or known to us.

One merged probe earns co-authorship on the HydroTuring paper. So do five accepted model proposals. You do not have to be able to package a model to propose it; say so on the form and the work is assigned to someone who can, and it still counts as yours.

Start with a probe or a model, or come talk it over on Discord first. Terms in CONTRIBUTING.md.

$ ht run --model reference_bucket --probe mass/catchment-closure
reference_bucket v1.1.0  ->  ✅ PASS (OK)  [1/1 probes passed]
  ✅ PASS  mass/catchment-closure
        ✅  closure            cumulative residual 0.0000% of sum_pr (limit 5.0%)
        ✅  state_bounds       all storages stay physical
        ✅  et_plausible       cumulative ET is 0.664 of potential ET (limit 1)
        ✅  non_degenerate     partition and variability are non-trivial
        ✅  forcing_fidelity   reported forcing matches the input

The idea

A benchmark that only checks closure is trivially gamed. A set of deliberately broken models lives in this repository to prove it, and no probe is merged until it passes an exactly conservative model and catches the broken ones it names.

reference_cheater is the one worth dwelling on. Its closure residual is exactly zero on every seed, forever. Randomising the forcing cannot touch it, because the cheat is in its internal wiring rather than its memory. It is caught only because the model contract requires absolute storage states rather than tendencies, so the storage it invents has to stay physical, and it does not.

The probes

Thirty-seven: twenty-four under mass, seven under energy and six under momentum. Each was merged only after the acceptance gate saw it pass its declared exact reference and fail a purpose-built broken one on the named criterion. Four physical models, a bucket that conserves water exactly, two hand-written FLEX models and the NWS's SAC-SMA with Snow-17, must pass every probe that can ask them anything; six of the seven energy probes and the three groundwater-exchange probes need outputs they do not report and are not scored for them; mass/snowpack-mass-closure scores only sacsma_snow17, the one physical model that reports snm. A probe that fails a physical model is examined before the model is; that is the first thing done with any probe pull request. Thirteen of the thirty-seven require no model output beyond runoff. That output-only count includes momentum/routing-lag-consistency, which is eligible only when the model also declares that it consumes pr and the three geometry inputs area_km2, main_channel_length_km and centroid_channel_length_km. momentum/wave-celerity-bounds instead enters directly at the routing control volume through prescribed q_in and uses only outlet dis to identify the local transient propagation speed from a paired reach-length counterfactual. ht list prints the probes; ROADMAP.md has three further directions and records their claim status. New probe proposals are welcome too.

Probe	Law	What it asks	The broken model it catches
mass/ungauged-basin-closure	mass	Does the full budget close for independently sampled soil/canopy capacities?	reference_spatial_loss, reference_spatial_gain, reference_spatial_capacity, four further attribute-dependent fixtures, reference_cheater and reference_degenerate
mass/catchment-closure	mass	Does the water budget close over ten generated years?	reference_leaky, reference_cheater, reference_degenerate
mass/resolution-invariance	mass	The same month at the minute, the hour and the day: do the integrated volumes agree?	reference_fixed_step, reference_degenerate
mass/warming-response	mass	The same rain with the air 3 °C warmer and 3 °C cooler: does runoff move the way physics says, in both directions?	reference_degenerate, reference_streamflow_only
mass/causality	mass	One storm added mid-record: nothing may change before it, and runoff must answer after it.	reference_anticipating
mass/dry-down	mass	Two years without rain: runoff can only fall, and no more may drain than the catchment held.	reference_climatology
mass/steady-state	mass	Three years of the same day: does everything settle, and does the budget balance once it has?	reference_restless
mass/spinup-cycle-invariance	mass	A repeated annual climate must produce the same evaluation cycle after different amounts of prior spin-up.	reference_restless, reference_leaky, reference_cheater, reference_degenerate
mass/multi-decadal-drift	mass	Fifty years of repeated warm weather: does total reported storage keep drifting?	reference_slow_drift, reference_gw_slow_drift
mass/extreme-rain	mass	The largest storm scaled up to ten times: runoff may not fall, nor exceed the rain that was added.	reference_saturating
mass/extreme-event-closure	mass	Overlap rainfall events toward fitted 100-year depths in one median-wet year of twenty; check each event's water budget so long-record averaging cannot hide a loss.	reference_in_sample
mass/runoff-bounds	mass	Over ten years, is the runoff possible at all: at least rain minus demand minus storage, at most rain plus storage? The mass question a runoff-only model has to answer.	reference_degenerate, reference_overflowing
mass/area-invariance	mass	The same weather on the same catchment told as ten times larger: every depth must be identical.	reference_area_leak
mass/response-nonnegativity	mass	One 120 mm storm added: from that day on, runoff may never be lower than without it.	reference_overshooting
mass/antecedent-monotonicity	mass	The same storm after a dry month and a wet one: the wetter catchment runs off more, and no more than the extra water.	reference_cheater
mass/phase-counterfactual	mass	The same water falling as rain instead of snow: timing moves, the integrated volumes may not.	reference_sublimating
mass/time-origin-invariance	mass	The same weather under a 28-year calendar shift that preserves seasons and leap days: evaporation, runoff and water stores must agree.	reference_calendar, reference_degenerate, reference_leaky
mass/precipitation-counterfactual	mass	The same seed 20% wetter, 10% wetter and 20% drier: the water added or removed must be partitioned among evaporation, runoff and storage, and runoff must rise from drier to wetter.	reference_cheater, reference_leaky, reference_degenerate
mass/human-abstraction	mass	A prescribed net irrigation withdrawal must leave the budget: the same weather run with and without it, and the difference must account for exactly the abstracted volume.	reference_abstraction_blind, reference_leaky
mass/snowpack-mass-closure	mass	Does the internal snowpack water balance close across accumulation, storage and melt without stage cancellation?	reference_snow_bypass, reference_snowless
mass/aquifer-recharge-ordering	mass	A direct recharge pulse into a passive aquifer can never lower its storage or turn its exchange away from the river; the paired budget arm largely restates groundwater_balance, so the independent content is the ordering.	reference_recharge_blind, reference_recharge_overshoot
mass/groundwater-datum-invariance	mass	A common vertical datum shift leaves the groundwater exchange trajectory unchanged.	reference_datum_dependent, reference_zero_exchange
mass/gw-sw-exchange-consistency	mass	A model reporting groundwater-river exchange: do recharge, the net exchange (gw_sw_exchange, distinct from gwex) and the two signed directional components, and aquifer storage all agree?	reference_exchange_sign_error
energy/pet-consistency	energy	Evaporation reaches demand when the model's own soil is wettest, stays below it, and falls when the soil is driest.	reference_thirsty
energy/latent-heat-et-consistency	energy	The evaporation a model reports as water and the evaporation implied by the latent heat it reports: are they the same evaporation?	reference_two_head, reference_constant_lambda, reference_sublimation_blind, reference_energy_leak
energy/evaporative-partition	energy	One summer without rain under net radiation that did not change: the latent heat a drying surface gives up has to warm the air.	reference_two_head, reference_ground_dodge
energy/surface-energy-closure	energy	Does each day and night close its hourly surface energy budget, without opposite errors cancelling?	reference_diurnal_bias
energy/radiation-consistency	energy	The surface temperature a model reports and the upward longwave it reports: do they describe one surface, hour by hour, at the emissivity it was given?	reference_air_emitter, reference_no_reflection
energy/soil-heat-storage-consistency	energy	Does heat retained in a soil layer agree with its temperature change during heating and recovery?	reference_frozen_soil, reference_half_soil
energy/snowmelt-energy-water	energy	A pack built over a cold winter and then melted down over a dry block that opens below -14 C and warms through zero: does the surface energy budget pay both the fusion for the ice that changed phase and the cold content of warming the pack?	reference_degree_day, reference_warming_free
mass/exchange-response	mass	A declared head-driven exchange must respond to its external head: the same weather with the prescribed head as given, raised and lowered from the first scored step — a model that declares it consumes the head must answer with more inflow when it is raised and less when lowered, by at least a share of how much its own exchange moves that any head-driven boundary clears.	reference_noise_sink, reference_token_exchange
momentum/routing-conservation	momentum	The channel store is never negative and never holds more than its hydrograph can.	reference_stuck_router, reference_leaky_router
momentum/routing-lag-consistency	momentum	The same isolated storm crosses four synthetic catchment geometries: does the runoff peak lie on a broad Snyder travel-time scale and grow across the geometry ladder?	reference_instant_router, reference_inverse_router
momentum/stage-discharge-monotonic	momentum	Does the stage a model reports rise with its discharge, and does the rating loop the way a flood wave does, the rising limb sitting lower than the falling one at the same discharge?	reference_rating_drift, reference_rating_inverted, reference_flat_stage
momentum/uniform-flow-friction-consistency	momentum	At steady uniform flow, do reported discharge and stage satisfy one Manning balance with the declared reach geometry and roughness?	reference_wrong_roughness, reference_wrong_slope
momentum/froude-regime	momentum	A stage and a discharge are two readings of one cross-section: does the pair a model reports stay subcritical, or does it imply a velocity the reach could not deliver?	reference_shallow_rating
momentum/wave-celerity-bounds	momentum	Across low, medium and high hydraulic states, does a short/long reach pair imply positive flood-wave celerity near the declared Manning dQ/dA expectation, with speed increasing as flow rises?	reference_fixed_celerity

Models

models/ holds three kinds. A submitted model is there to be evaluated, and every run of one is appended to models/result.csv; the physical models' runs are archived there too. A physical model is there to test the probes: the exact bucket, two hand-written conceptual models from chrimerss/HydrologicModels and the NWS's SAC-SMA with Snow-17 must pass every probe that can ask them anything, so a probe that fails one is wrong until shown otherwise. All four report water and no energy, so all four are N/A, with reason INCOMPLETE, on the six probes that need an energy output: the four that need the latent, sensible and ground heat fluxes, energy/latent-heat-et-consistency, energy/evaporative-partition, energy/surface-energy-closure and energy/snowmelt-energy-water, and energy/radiation-consistency, which needs a surface temperature and its upward longwave, and energy/soil-heat-storage-consistency, which needs layer boundary heat fluxes and soil temperature. Not scored rather than passing: they have not violated conservation of energy, they have declined to be falsifiable about it, exactly as reference_streamflow_only does on the budget probes. A broken model is broken in one specific way, so that no criterion goes untested.

A standing counts the probes a model passed out of the probes that could score it. A probe that needs a variable the model does not report, or that the model cannot consume, is N/A for it and in neither number, so the totals differ between models.

Most evaluated models are N/A (INCOMPATIBLE) on momentum/routing-lag-consistency, because they do not declare consumption of either required channel-length field. Several adapters do read area, sometimes only to convert runoff depth to discharge, but area alone does not expose the geomorphic travel path this probe changes. Three consume the complete geometry: flex_lumped maps it to its native triangular routing lag and passes, and topmodel and hype map it to their own routing and fail. modflow6, rvic and mizuroute do not report what it reads and are N/A (INCOMPLETE). N/A is neither a pass nor a failure; it means the probe cannot ask the declared model interface this question.

Model	Kind	What it does	Standing
google_flood_forecast	submitted	The mean-embedding forecast LSTM behind Google Flood Hub, at the published weights. Predicts discharge and nothing else.	FAIL (VIOLATION), 5 of 11 probes passed. It passes the runoff bounds, memory, area, causality and steady state, and fails step, extreme rain, phase, warming, dry-down, and a 0.18 mm/day dip after an added storm
dhbv2	submitted	δHBV 2.0, the MHPI group's differentiable HBV: neural networks write the parameters of a bucket model that reports its stores and its evaporation.	FAIL (VIOLATION), 15 of 21 probes passed. Its learned regional-groundwater term, declared as gwex, closes the budget, and the case's stated soil capacity is N/A for a store whose capacity is learned. What remains is a runoff depth that changes with the area it is told, a third less runoff when snow falls as rain, runoff 12.1% of the rain apart between hourly and daily steps, flow that falls below the control after rain is added, a response to added rain above the limit, and on mass/human-abstraction it never reads the prescribed withdrawal, so it accounts for none of the 380 mm.
wflow_sbm	submitted	Deltares' Wflow.jl SBM: a soil column with unsaturated and saturated stores, interception, snow and kinematic-wave routing, adapted in Julia and run on one representative cell at the resolution of Wflow's Moselle model.	FAIL (VIOLATION), 19 of 22 probes passed. Its water budget closes to 2e-4 of the rain and it passes the area, routing, step, calendar, causality, extreme-rain, phase, precipitation-counterfactual, warming and human-abstraction probes, the last by taking the prescribed withdrawal through Wflow's own water demand and allocation. That 2e-4 is water its river kinematic wave creates when open-water evaporation exceeds the rain on the river and the wave floors its discharge rather than drying the channel; on mass/extreme-event-closure it is 0.0016 mm on one 0.006 mm drizzle day on one seed, past that probe's 0.001 mm floor, the only one of the probe's events to fail. Its other two failures both come from its soil column: a storm that does not fill the column makes almost no runoff, and under the shipped mapping a wet month's extra water has already been evaporated down to the rooting depth when the storm arrives, so the wetter catchment runs off barely more than the dry one, about a thousandth of the storm where the probe asks for two hundredths; and once the root zone dries in a rainless spell, drainage from the unsaturated store raises a water table that sits below the roots, and lateral flow rises with no rain. Both move with how the stated soil capacity is mapped onto soil thickness and roots.
summa	submitted	SUMMA 4.0.0, a land model that solves the water and energy balances of canopy, snow, soil and aquifer with one implicit solver, run as one lumped HRU from its shipped test-case setup, with the radiation and humidity it needs mocked from each forcing row.	FAIL (VIOLATION), 9 of 23 probes passed. Its water budget closes to rounding and its surface energy budget to 0.2% of its own net radiation; what fails is a latent heat of vaporisation held at its 0 C value, a canopy that, as the shipped setup configures SUMMA, intercepts all rain and keeps what freezes near 0 C, holding up to 27 mm of ice against a 2 mm capacity on the probes that score it (also the only failure on the precipitation counterfactual), leaf area that keeps its seasons under constant weather, runoff that follows the melt rather than the rain on one seed, a 5.2% runoff response to turning snow into rain on another, Green-Ampt runoff that appears only at the hourly step, and no human water use, so the prescribed withdrawal on the human-abstraction probe is never taken. SUMMA computes its own net radiation from a sky built from rn, so the hourly surface-energy-closure probe, which requires prescribed rn, is N/A; on the two daily energy probes its rn-driven closure passes and drought partition_shift failure are archived scores whose applicability is not yet settled.
cwatm	submitted	CWatM 1.11, IIASA's Community Water Model and an ISIMIP global hydrological model, run on one grid cell with every store it carries reported.	FAIL (VIOLATION), 17 of 21 probes passed. Its budget closes to 0.03%, and its own water-demand module pumps a prescribed withdrawal out of groundwater to within 0.004%. That 0.03% is water its capillary rise creates, a few thousandths of a millimetre a day, and on mass/extreme-event-closure it fails 3 to 8 one-day drizzle events of under 0.07 mm per seed, where 0.001 to 0.005 mm more leaves or is stored than fell, against that probe's allowance of 5% of the rain or 0.001 mm, whichever is larger. It also fails on a groundwater reservoir with no dt that drains 24 times too fast at an hourly step (52% of the rain); on a 0.29 mm/day dip after an added storm, where preferential flow turns surface runoff into interflow that leaves through a slower runoff-concentration lag; and on evaporation on wet soil at 0.70 of demand, near the cap its crop coefficients set. The last two are packaging choices as much as results: this package follows the CWatM-Earth-30min template its parameters come from, and with preferentialFlow = False, the setting of the pinned model repository's own 30′ templates, both pass.
lisflood	submitted	LISFLOOD 5.0.0, the EC Joint Research Centre's distributed model behind EFAS and GloFAS, stepped through its own Python framework on one representative 5 km cell and reporting every store its own water balance module counts.	FAIL (VIOLATION), 19 of 21 probes passed. Its budget closes to 1e-13 mm per step; it reports no heat fluxes and no surface temperature, so the five energy probes that need an energy output cannot ask it anything and are N/A. It fails the step probe because its potential infiltration is a pore-space storage multiplied by the step length, so rain falling within hours runs off at the hourly step (13.0% of the rain between PT1H and PT1D). The two ten-year probes took about 90 s per run under amd64 emulation against a 60 s budget and were archived as ERROR from the host's speed; run natively under Apptainer they finish, and mass/precipitation-counterfactual passes. On mass/human-abstraction, LISFLOOD's own water-use rule takes the groundwater share in full and the rest only from channel water above an environmental-flow reserve, recording what the channel cannot give as shortage. On this one-cell water region it withdraws 17 to 33% of the prescription, from the sourced reserve to none, and leaves 67 to 83% where the probe allows 5%.
modflow6	submitted	MODFLOW 6.7.0, the USGS's modular groundwater model: a transient one-river-cell aquifer run through flopy, reporting the RIV and STO budget terms as signed exchange and groundwater storage. Archived as evidential, not gated on.	PASS, 3 of 3 probes scored
hymod	submitted	HYMOD, a five-parameter conceptual model: a probability-distributed soil store, three quick reservoirs and a slow one, from hydromodel v0.4.0, run unmodified.	FAIL (VIOLATION), 20 of 21 probes passed. Its one failure: it has no withdrawal term, so all 380 mm of the prescribed withdrawal on mass/human-abstraction goes unaccounted for.
xaj	submitted	The Xin'anjiang model, a saturation-excess conceptual model used for flood forecasting in China since the 1980s, from hydromodel v0.4.0, run unmodified.	FAIL (VIOLATION), 19 of 21 probes passed. The impervious fraction is booked inconsistently, so the budget stays open by 2.5% of the rain: inside mass/catchment-closure's 5%, but 133 of 1316 small events fail mass/extreme-event-closure. It has no withdrawal term, so all 380 mm of the prescribed withdrawal on mass/human-abstraction goes unaccounted for.
gr4j	submitted	GR4J, the four-parameter daily rainfall-runoff model, through INRAE's airGR 1.7.9, with its compiled kernel unchanged.	FAIL (VIOLATION), 20 of 21 probes passed. Its exchange term is declared as gwex and the budget closes. It has no withdrawal term, so all 380 mm of the prescribed withdrawal on mass/human-abstraction goes unaccounted for.
topmodel	submitted	TOPMODEL, in Wouter Buytaert's R package at the 0.7.5 commit: runoff from a topographic index and a root zone, with the catchment's delay function.	FAIL (VIOLATION), 19 of 22 probes passed. Rain fills the root-zone deficit before anything runs off, so a wetter catchment's extra water stays in the root zone and the storm runs off no more, which fails mass/antecedent-monotonicity. With no hillslope delay, only the channel's, it fails momentum/routing-lag-consistency; and it has no withdrawal term, so all 380 mm of the prescribed withdrawal on mass/human-abstraction goes unaccounted for.
tank	submitted	The four-tank conceptual model of Sugawara and Funiyuki, as implemented in nzahasan/tank-model 3.0.0b3, vendored unmodified.	FAIL (VIOLATION), 19 of 20 probes passed. Its one failure: it has no withdrawal term, so all 380 mm of the prescribed withdrawal on mass/human-abstraction goes unaccounted for.
topkapi	submitted	TOPKAPI, kinematic waves integrated into soil, overland and channel reservoirs on each cell, from PyTOPKAPI v0.4.0, vendored unmodified.	FAIL (VIOLATION), 20 of 21 probes passed. Overland water is reported in channel. It has no withdrawal term, so all 380 mm of the prescribed withdrawal on mass/human-abstraction goes unaccounted for.
cfe	submitted	CFE, the Conceptual Functional Equivalent of the National Water Model, NOAA-OWP's shared library built unmodified. Hourly, as upstream runs it, so almost every daily probe cannot feed it.	PASS, 1 of 1 probes scored. The one probe that can score it at its hourly step passes. A diagnostic run at PT1D, not archived, scored 11 of 13 and showed the surface cascade's retention depth in channel; see its README.
simhyd	submitted	SIMHYD in the form MARRMoT v2.1.2 gives it, m_18_simhyd_7p_3s. A result here is about that formulation, which its authors distinguish from the original code.	FAIL (VIOLATION), 11 of 20 probes passed. The budget closes. Interception and transpiration are each capped at demand, so together they reach up to twice potential evaporation (et_plausible, energy/pet-consistency); the solver's smoother and tolerance let the canopy exceed its capacity (state_bounds); a rate cap on infiltration moves runoff with the step; and it has no withdrawal term, so all 380 mm of the prescribed withdrawal on mass/human-abstraction goes unaccounted for.
mcp1	submitted	MCPModel1, the single-node Mass-Conserving Perceptron of Wang and Gupta: one store and two gated outflows, trained on the Leaf River, at the published checkpoint.	FAIL (VIOLATION), 18 of 20 probes passed. The budget closes to under 1e-6 of the rain. There is no capacity parameter anywhere in the model, so the case's stated capacity is N/A for its store, which passes that number without running away. What remains is mass/antecedent-monotonicity and mass/human-abstraction.
mcp2	submitted	MCPModel2, MCPModel1 with a learned term that adds water below a learned store level and removes it above, declared as gwex, at the published checkpoint.	FAIL (VIOLATION), 15 of 20 probes passed. The budget closes to under 1e-6 of the rain. There is no capacity parameter anywhere in the model, so the case's stated capacity is N/A for its store, which passes that number without running away. It fails mass/dry-down, energy/pet-consistency, mass/resolution-invariance, mass/antecedent-monotonicity and mass/human-abstraction.
mcp3	submitted	MCPModel3, the three-node MA_5 Mass-Conserving Perceptron: a primary store whose outflows pass through a surface node, reported as channel, and a groundwater node, at the published checkpoint.	FAIL (VIOLATION), 17 of 21 probes passed. The budget closes to under 1e-6 of the rain. There is no capacity parameter anywhere in the model, so the case's stated capacity is N/A for its store, which passes that number without running away. Its surface node fails momentum/routing-conservation (62.4 mm against a 1.9 mm bound), and it also fails mass/antecedent-monotonicity, mass/dry-down and mass/human-abstraction.
mcp4	submitted	MCPModel4, a five-node distributed-state Mass-Conserving Perceptron network whose gates read all five stores through learned matrices, at the published checkpoint.	FAIL (VIOLATION), 14 of 20 probes passed. The budget closes to under 1e-6 of the rain. There is no capacity parameter anywhere in the model, so the case's stated capacity is N/A for its store, which passes that number without running away. Its learned matrices give a wetter catchment that runs off less, a dip after added rain, a 1.17 response to added rain and a rise in a rainless week; it also fails mass/steady-state and mass/human-abstraction.
mcp5	submitted	MCPModel5, a three-node distributed-input Mass-Conserving Perceptron network whose stores each take a learned share of the rain, at the published checkpoint.	FAIL (VIOLATION), 14 of 20 probes passed. The budget closes to under 1e-6 of the rain. There is no capacity parameter anywhere in the model, so the case's stated capacity is N/A for its store, which passes that number without running away. Its learned matrices give a dip after added rain and a response to added rain above the rain added; it also fails mass/dry-down, energy/pet-consistency, mass/human-abstraction, and mass/multi-decadal-drift on non_degenerate.
raven	submitted	Raven v4.12's emulation of HBV-EC, the configuration the proposer named, with the case's soil and canopy capacities mapped in.	FAIL (VIOLATION), 7 of 22 probes passed. Most of the count is one quantity crossing one threshold: at the case's soil capacity the runoff ratio falls under non_degenerate, and two changes that keep the rest of the package score 8 and 7 failures. Six fail in every configuration tried, among them LAKE_EVAP_BASIC taking the full demand. See its README before reading the number.
hype	submitted	HYPE 5.36.0, the model SMHI runs operationally, built unmodified from the release archive, as one subbasin.	FAIL (VIOLATION), 15 of 23 probes passed. The budget closes. The top soil layer is not capped, so mrso passes the case's capacity on the budget probes (state_bounds). Local river length is the square root of area, which fails mass/area-invariance, and all water crossing both reaches in series fails momentum/routing-lag-consistency. This configuration has no withdrawal, so all 380 mm on mass/human-abstraction goes unaccounted for.
prms	submitted	PRMS 6.0.0, the USGS Precipitation-Runoff Modeling System: the release's own Linux executable, unmodified, as one HRU draining to one segment.	FAIL (VIOLATION), 20 of 22 probes passed. Withdrawal stops when the reservoir it comes from runs dry, leaving 28.7% of the prescription on mass/human-abstraction, and Muskingum coefficients in single precision take channel to -0.0005 mm on mass/ungauged-basin-closure. The adapter snaps a canopy within the print quantum to capacity, documented in its README.
vic	submitted	VIC 5.1.0's classic driver. It refuses a daily row, shown on the unmodified executable, so a daily row is held flat over 24 hourly solver steps.	FAIL (VIOLATION), 16 of 23 probes passed. Its water budget closes, and mass/catchment-closure passes. ET exceeds a pet VIC never reads, and its rlus leaves out reflected longwave, (1 - eps) * rlds, on all 720 steps of energy/radiation-consistency.
mhm	submitted	mHM v5.13.4 with its routing module mRM, compiled unmodified from upstream's sources, on one cell.	FAIL (VIOLATION), 21 of 22 probes passed. The budget closes. Its one failure: it has no withdrawal term, so all 380 mm of the prescribed withdrawal on mass/human-abstraction goes unaccounted for. mass/area-invariance does not exercise its multiscale parameter regionalisation here.
shud	submitted	SHUD, the Simulator for Hydrologic Unstructured Domains, at upstream's master, 851fb8c, past the canopy fix 0958682c78da that no tag carries, on a minimal mesh.	FAIL (VIOLATION), 9 of 22 probes passed. What remains is a double count in ET and solver noise at upstream's tolerance. The tag the proposal names, cpu-accel-v1.1.1, lost 30 to 60% of the rain to one line of ET() and passed 3 of 22.
noahmp	submitted	Noah-MP v5.2.1 through HRLDAS, the land scheme of the National Water Model and WRF. One 86400 s step stops upstream's own checks, so a daily row is held flat over 24 hourly solver steps.	FAIL (VIOLATION), 17 of 25 probes passed. Its water budget closes, and mass/catchment-closure passes. ET exceeds a pet Noah-MP never reads as demand, and monthly LAI keeps it moving under constant weather.
rvic	submitted	RVIC 1.1.1, the Lohmann routing model: a linearised Saint-Venant impulse response, built once and convolved with the inflow.	FAIL (VIOLATION), 0 of 1 probes passed. One probe can score a router with its outputs. The impulse response has one velocity for the whole record, so the wave travels at about the same speed at every state, which is what momentum/wave-celerity-bounds exists to catch and is the model's design.
mizuroute	submitted	mizuRoute v3.1.1, ESCOMP's stand-alone river routing model, built with its own Makefile, unmodified, on its diffusive-wave scheme.	PASS, 1 of 1 probes scored. One probe can score a router with its outputs. The scheme decides the verdict: the diffusive wave passes; the default impulse-response scheme shows no delay at all in these cases.
reference_bucket	exact	conserves water exactly by construction	must pass every probe that can ask it anything; N/A on the six energy probes and the three groundwater-exchange probes that need outputs it does not report
reference_exchange_exact	exact	a lagged-head bookkeeping reference that closes its own groundwater balance exactly and reports genuinely bidirectional exchange	must pass mass/gw-sw-exchange-consistency, the only probe that can ask it anything
reference_recharge_order_exact	exact	the exact daily solution of one aquifer cell draining to a river, so added recharge is stored, then released	must pass mass/aquifer-recharge-ordering
reference_recharge_blind	broken	ignores gw_recharge, so its own budget no longer closes	caught by aquifer_recharge_ordering
reference_recharge_overshoot	broken	sends 120% of a recharge day to the river, so its storage falls below the control's	caught by aquifer_recharge_ordering
reference_datum_exact	exact	linear single-aquifer reference with all absolute heads and elevations shifted consistently	must pass mass/groundwater-datum-invariance
reference_datum_dependent	broken	positive exchange whose conductance depends on the numerical initial head	caught by datum_flux_invariance
reference_zero_exchange	broken	frozen aquifer reporting no exchange	caught by datum_flux_invariance
flex_lumped	physical	lumped FLEX/HBV: interception, beta-partitioned unsaturated store, fast and slow reservoirs, geometry-aware triangular lag	must pass every probe that can ask it anything; PASS, 25 of 25
flex_topo	physical	FLEX-Topo: plateau, hillslope and wetland units on real Wark fractions sharing one groundwater store	must pass every probe that can ask it anything; PASS, 24 of 24
sacsma_snow17	physical	the NWS's SAC-SMA with Snow-17 and a gamma unit hydrograph, ported from the legacy Fortran and checked against it	must pass every probe that can ask it anything; PASS, 25 of 25
reference_coupled	exact	the bucket with snow sublimation and a surface energy budget: every kilogram converted at the latent heat of the phase it actually underwent	must pass every criterion of the three energy-flux probes; supports daily and hourly steps
reference_snow_energy	exact	an energy-balance snowpack: melt bought with max(0, Rn - H - LE - G) / lambda_f, liquid held in the pore space and reported as lwsnl, cold content carried as an energy deficit and reported as csnow	must pass every criterion of energy/snowmelt-energy-water; supports daily and hourly steps
reference_degree_day	broken	the same snowpack melted on air temperature through a degree-day factor, with a surface energy budget that closes around its evaporation alone, so nothing charges the fusion	caught by melt_energy
reference_warming_free	broken	melts on the energy actually available and reports its cold content honestly, but charges its budget for the fusion alone, so the pack warms towards 0 °C for nothing	caught by melt_energy
reference_soil_heat	exact	a synthetic fixed-layer fixture driven by incoming radiation, with prescribed depth, heat capacity and initial temperature	must pass soil_heat_storage; checks budget consistency, not temperature accuracy
reference_frozen_soil	broken	retains the conductive fluxes but reports a frozen soil temperature	caught by soil_heat_storage
reference_half_soil	broken	retains the conductive fluxes but halves the reported soil-temperature change	caught by soil_heat_storage
reference_spatial_loss	broken	Attribute-dependent loss fault; exact bucket in the declared reference domain	caught by closure
reference_spatial_gain	broken	Attribute-dependent gain fault; exact bucket in the declared reference domain	caught by closure
reference_spatial_capacity	broken	Fixed storage-output offset outside the declared reference domain	caught by state_bounds
reference_spatial_forcing	broken	Attribute-dependent forcing fault; exact bucket in the declared reference domain	caught by forcing_fidelity
reference_spatial_et	broken	Attribute-dependent et fault; exact bucket in the declared reference domain	caught by et_plausible
reference_spatial_negative	broken	Attribute-dependent negative fault; exact bucket in the declared reference domain	caught by non_degenerate
reference_spatial_frozen	broken	Attribute-dependent frozen fault; exact bucket in the declared reference domain	caught by non_degenerate
reference_abstraction_blind	broken	the same bucket, blind to the prescribed withdrawal, so the two variants come out identical	caught by human_abstraction
reference_two_head	broken	a water head and an energy head that never meet: both budgets close to 1e-15 and the latent heat implies an evaporation it never reported	caught by flux_identity
reference_constant_lambda	broken	coherent, but converts every kilogram at the same latent heat of vaporisation	caught by flux_identity
reference_sublimation_blind	broken	coherent for liquid water, but converts snow sublimation at the latent heat of vaporisation instead of sublimation	caught by flux_identity
reference_ground_dodge	broken	coherent, both budgets close, and the sensible flux never reads the soil: when the soil dries the ground flux absorbs the whole shift	caught by partition_shift
reference_diurnal_bias	broken	shifts sensible heat so the energy residual is +20 W/m2 by day and -20 W/m2 by night; the full-record residual cancels	caught by energy_closure_by_phase
reference_radiative	exact	the coupled reference with a skin: temperature diagnosed from the sensible heat flux through a fixed conductance, upward longwave that skin's emission plus the reflected sky at the emissivity it was given	must pass every criterion of energy/radiation-consistency
reference_air_emitter	broken	reports the skin's temperature but emits at the air's, reflected sky unchanged	caught by radiative_identity
reference_no_reflection	broken	reports emission alone as the total upward longwave, the reflected sky left out: 3 to 23 W/m2 that a 5% tolerance would not reliably see	caught by radiative_identity
reference_energy_leak	broken	discards 15% of net radiation; the energy counterpart of reference_leaky	caught by energy_closure
reference_leaky	broken	hides a silent 15% sink	caught by closure
reference_cheater	broken	solves for storage as whatever balances the budget	caught by state_bounds
reference_degenerate	broken	evaporates all precipitation, produces no runoff	caught by non_degenerate, response_sign
reference_fixed_step	broken	treats every row as a day whatever the step is	caught by resolution_invariance
reference_anticipating	broken	smooths runoff over a centred window, so three days of the future are in every value	caught by causality
reference_climatology	broken	emits the seasonal mean whatever falls, and keeps flowing without rain	caught by dry_down
reference_saturating	broken	caps its daily runoff, so an extreme storm adds rain and no runoff	caught by monotone_response
reference_restless	broken	a recession on an internal thirty-day clock, so it never settles	caught by steady_state
reference_overflowing	broken	reports its runoff plus 80% of the rain again, from nowhere	caught by runoff_bounds
reference_area_leak	broken	loses a share of runoff that grows with the area it is told	caught by invariance (area)
reference_overshooting	broken	a derivative term sharpens its hydrograph, so an added storm lowers later flow	caught by response_nonnegativity
reference_sublimating	broken	loses 40% of every snowfall to an unreported sublimation	caught by phase_invariance
reference_snow_bypass	broken	sends 15% of snowfall directly to the soil around the snowpack, preserving the catchment balance while breaking the internal snowpack balance; the 85% it keeps clears the probe's peak-fraction floor, so closure is the only criterion it trips	caught by closure
reference_snowless	broken	stores no snow and passes all precipitation through the snow module	caught by snowpack_response
reference_thirsty	broken	evaporates a fixed share of its soil store, never reading demand; conserves water exactly	caught by demand_consistency
reference_driven_exchange	exact	a boundary exchange driven by the prescribed external head against a constant catchment head, so it answers the head raised or lowered with a sustained flux; a positive control for mass/exchange-response	must pass mass/exchange-response
reference_evolving_exchange	exact	the same boundary against a catchment head that moves, S dh/dt = C (H − h) with a five-day time constant, so it is in equilibrium when scoring begins and answers a shift with the water that lifts its own head and little more; the positive control that fails if the shift begins during spinup, and the one the response floor is calibrated against	must pass mass/exchange-response
reference_recharge_exchange	exact	the evolving boundary on a losing catchment that drains half its runoff through it, storativity 1 mm/m: a 0.5 mm response against 1700 mm of throughput, exact and monotone in the head; the control that pins the criterion's normalisation, since a share of the gross exchange fails it and a share of the exchange's variation does not	must pass mass/exchange-response
reference_noise_sink	broken	conserves water internally, misreports its evaporation by a seeded 30% and declares the difference as a groundwater exchange; declares the prescribed external head and never reads it, so its budget closes to floating point and its exchange is identical whatever the head is	caught by exchange_response
reference_token_exchange	broken	reference_noise_sink with a head-driven trickle a million times weaker than the controls added, so its exchange answers the head with the right sign and a millionth of its own gross movement	caught by exchange_response
reference_steady_sink	broken	reference_token_exchange with its noise taken out: evaporation overstated by a constant 0.3 mm/day, 13% of the rain, declared as a steady regional inflow, with the same trickle on top. A constant sink adds nothing to the exchange's variation, so the share of the variation cannot see it and the much smaller share of the gross is what does	caught by exchange_response
reference_stuck_router	broken	a routing kernel summing to 0.9, so a tenth of every day's runoff never leaves the channel	caught by routing_conservation
reference_leaky_router	broken	the same kernel summing to 0.999, so a tenth of a percent stays behind: the smallest leak that probe exists to catch, and the one its allowance is calibrated against	caught by routing_conservation
reference_snyder_router	exact	consumes the declared geometry and routes a fixed rain share through a causal, conservative triangular unit hydrograph whose peak follows the duration-corrected Snyder lag from the excess-rainfall centroid	must pass both criteria of routing-lag-consistency
reference_instant_router	broken	accepts the geometry but returns the storm runoff in the rainfall row at every catchment scale	caught by lag_time_bounds
reference_inverse_router	broken	uses in-bounds lags of 1, 2, 1 and 3 days in ascending area order	caught by scaling_monotonicity
reference_rating	exact	the bucket with a real rating curve: yield enters a shallow floodplain and a deep channel reservoir with separated time constants, and the stage is the depth the channel's volume makes in a fixed bed, reported above the case's declared datum, so the gauge rises with the flow and a falling recession sits above where it sat on the way up	must pass every criterion of momentum/stage-discharge-monotonic
reference_rating_drift	broken	derives its stage from a slowly decaying running maximum of discharge (peak = max(q, 0.997 * peak) per day), so the gauge ratchets up with each flood far faster than it relaxes, stepping down only a fraction of a percent a day	caught by rating_monotonic
reference_rating_inverted	broken	reads the loop backwards, high while the flood is arriving and low once it is leaving: monotone in discharge, so only the loop sees it	caught by rating_loop
reference_flat_stage	broken	reports a constant stage, so there is no rating and no loop, only a number that does not vary	caught by non_degenerate
reference_uniform_flow	exact	converts constant effective rainfall to discharge and solves exact rectangular Manning normal depth above the declared bed	must pass uniform_flow_friction
reference_saint_venant	exact	advances one-dimensional continuity and momentum with a finite-volume solver from a non-equilibrium state; no normal-depth lookup	must pass uniform_flow_friction
reference_wrong_roughness	broken	computes stage with Manning roughness 3% above the declared value, a 5.74% near-boundary residual	caught by uniform_flow_friction
reference_wrong_slope	broken	computes stage with bed slope 6% above the declared value, a 6% near-boundary residual	caught by uniform_flow_friction
reference_shallow_rating	broken	draws its gauge from a section five times wider than the one the case declares, so the depth is right for a different reach and too shallow for this one	caught by froude_subcritical, and by uniform_flow_friction at steady flow
reference_streamflow_only	honest limit	reports discharge only, from a store that never reads the temperature	N/A (INCOMPLETE) on budget probes; caught by response_sign
reference_in_sample	broken	removes surface runoff above a fixed 55 mm daily precipitation cutoff	caught by event_water_closure
reference_calendar	broken	a recession that drifts with the calendar year	caught by invariance (time origin)
reference_exchange_sign_error	broken	a genuinely bidirectional aquifer-river exchange that closes its own groundwater balance exactly, but reports the aquifer-to-river component's sign flipped on every other step	caught by exchange_components

How a case is generated

Forcing is generated fresh at run time from a recorded seed. Nothing is committed as data, so there is nothing to memorise, and a reviewer reviews a short deterministic script instead of a binary blob. Each probe runs several seeds and all of them must pass, so no model gets through on a lucky draw. Any run reproduces exactly with ht run --seed <n>.

This is an unseen-sample guarantee, not a claim that the public generator's distribution is secret. For evaluation against generators or data unavailable during training, keep a second probe tree outside the repository and add it with --probe-root /secure/hidden-probes. The model container receives only opaque case metadata and the inputs needed for inference; the host retains the probe identity, generator seed, annotations and scoring code. See docs/adapting-a-model.md.

Verdicts

A probe that can ask the model something passes or fails it, and the reason is recorded separately from the verdict, because these mean different things:

• VIOLATION the model reported its budget and the budget did not close.
• ERROR the adapter or benchmark machinery failed. This is operational, not a scientific verdict, and is the one outcome that makes ht run exit 2.

A probe that cannot ask the model anything is N/A: not scored, and neither a pass nor a fail. That happens two ways:

• INCOMPLETE the model never reported enough to be checked. Every streamflow-only model lands here on the budget probes. It has not violated conservation; it has declined to be falsifiable.
• INCOMPATIBLE the model and probe disagree on timestep, on a forcing or static input one needs and the other does not supply or declare, or on paired-perturbation support, so running them would not be meaningful.

A model passes when at least one probe could be put to it and every probe that could be put to it passes. Otherwise it fails, with the worst reason among the scored probes that did not pass; an unscored probe never supplies that reason. A model that no probe could be put to at all is N/A too, and ht run exits 1 for it as it does for a FAIL.

A scored verdict is one bit. Everything under it stays quantitative, so a paper can show that one model leaks 6% and another 40% long before anyone crosses the line.

Quick start

pip install -e '.[dev]'

ht init-probe --list-templates            # probe shapes to start from
ht init-probe --template invariance      # start a new probe
ht list                                  # probes and models
ht validate                              # schema-check everything
ht gate                                  # the probe acceptance gate
ht run --model reference_bucket          # evaluate one model
ht run --model my-model --json out.json  # machine-readable report

A shared cluster with no Docker daemon runs the same images under Apptainer, with --runner apptainer; see Running on a cluster.

Without installing, ./ht (or ht.cmd in a Windows shell) runs the CLI straight from src/, as does python -m hydroturing with src on PYTHONPATH. Set HT_ASCII=1 for reports with the words and no marks, which is also what you get automatically wherever the output stream cannot carry them.

Submitting a model

Your model may be written in any language. It ships as a container plus a thin adapter that reads /io/request.json and writes /io/output/result.csv. The adapter is usually thirty lines. See docs/adapting-a-model.md, and AGENTS.md if you are having a coding agent build the sandbox for you. Every model reports in one vocabulary: schemas/lexicon.yaml gives each variable's name, units and sign, and what each packaged model's native outputs become.

We want more models, and we want the ones you think matter. Every published rainfall-runoff model, every LSTM, every foundation model with a hydrologic claim is in scope.

Start with a model proposal. The form accepts AI-based, AI+physics and physics models, with separate links for the model code and optional pretrained weights, and asks whether there is a time window you want the test to run over.

You do not need to be able to package it yourself. The packaging status field decides who the issue is assigned to and nothing else: ready means it is yours to finish, help needed means the maintainer takes it. Either way the proposal counts as yours, and five accepted proposals earn co-authorship on the benchmark paper the same as one merged probe. Knowing which models are worth putting through the benchmark is a judgement about the field, and it is not one the maintainer can make alone.

Once a proposal is accepted, fork and build; the pull request comes from the fork and closes the issue.

A submitted model is scored on the largest flood event of the generated record rather than on all ten years of it: by default a month for a daily model and a week for an hourly one, with the full spinup in front, located by the probe's own reference model. That is what keeps a model that takes seconds per forecast inside the probe's time budget. window_days in model.yaml changes it; full asks for the whole record.

The container runs with no network and never sees the probe code, so a model cannot read the tolerance it is being judged against. Every evaluation is appended to models/result.csv.

A model that fails is worth proposing, and so is one most probes cannot score. INCOMPLETE is the current state of nearly every published rainfall-runoff model on the budget probes — it has not violated conservation, it has declined to be falsifiable — and recording that honestly is a large part of what this is for.

Contributing a probe

Please submit one. A benchmark with four probes tests four things; the reason this repository is open is that the physics worth testing is wider than any one group knows. If you have spent time with a conservation law that AI models get wrong, that law is a probe, and we would rather have it from you than approximate it ourselves.

Contributors of merged probes are co-authors on the benchmark paper. The threshold is one probe, merged and passing the acceptance gate, or five accepted model proposals. This is how model intercomparison projects have always worked in this field: you contribute an experiment, you are an author on the paper that reports it. CONTRIBUTING.md has the full terms — author order, the right to decline, and what happens before anything is submitted.

ROADMAP.md lists the probes we want, each with a difficulty and a note on what it discriminates. Claiming one is easier than inventing one, but inventing one is welcome too; propose it first so nobody builds it twice. The two we most want are the cross-budget consistency probes: a model can close its water budget and its energy budget while being incoherent between them, and nothing in the suite currently notices.

Propose first, then fork, then open a pull request — the same three steps for a probe or a model. The proposal issue is where we find out whether a probe discriminates, which is cheaper to learn in a paragraph than in three hundred lines. Once it is labelled accepted it is assigned to you and nobody else will build it.

Then do not start from a blank page:

ht init-probe --list-templates           # the shapes available
ht init-probe --template <kind>          # writes probe-draft.yaml
#   ... fill in the fields ...
ht init-probe --from probe-draft.yaml    # creates probes/<law>/<slug>/
ht gate --probe <law>/<slug>             # prove it discriminates

The templates cover conservation over one case, extrapolation in space and in time, counterfactual response, and invariance. Each scaffolds into a probe that already passes the gate against a placeholder case, so you can watch it separate the reference models before writing any physics, then replace the case with yours. docs/writing-a-probe.md has the details.

The gate is the only bar that matters, and it is a technical one: your probe must pass an exact physical model and catch the deliberately broken ones. See GOVERNANCE.md for how disagreements about tolerances get settled.

Community

Questions, probe ideas, a model you would like evaluated, a tolerance you think is wrong: the HydroTuring Discord is where that conversation happens, before and alongside the issues.

Status

Suite 0.1.0. Thirty-seven probes, twenty-four mass, seven energy, six momentum, synthetic track only. More energy and momentum probes, and the real-data track, are next. The harness runs paired cases and scores labelled regimes; spatial and temporal closure, counterfactual response and invariance are represented in the suite. The roadmap lists the remaining directions and their claim status. Scores are only comparable within a suite version.

Layout

src/hydroturing/   harness: protocol, runners, criteria, scoring
probes/<law>/<id>/ probe.yaml, generate.py
models/<name>/     model.yaml, Dockerfile, adapter
schemas/           JSON schemas for both spec files, and lexicon.yaml, the variables