FiMI Banking Uses Verifiable Rewards to Beat a 12B Baseline on Its Own Tasks

The NPCI AI Research Team's September 3 paper turns the small-versus-large model argument into a banking workflow test. FiMI Banking starts with an open 4.5-billion-effective-parameter Gemma 4 model and a controlled environment covering account and KYC work, deposits and loan installments, government-scheme eligibility, insurance claims, and tax deducted at source. The agent sees synthetic customers, bank-authored product material, callable tools and a seeded database. No real customer records or conversations entered the training pipeline. That design matters because the agent is meant to run on bank-controlled hardware, including fully air-gapped installations, where privacy, cost and reproducibility can outweigh a frontier model's broader reach.

The reinforcement-learning route used 48,245 generated tasks across 100 families, sampled 10,000 for training and held out 1,000 for evaluation. Every task carried a prewritten chain of correct tool calls. Four checks scored the completed dialog: tool sequence, final database state, whether the customer received the required information, and a judged assertion. The first three are exact checks and account for most of the reward; the paper says an independent reference scorer agreed with the implemented reward on 97.2% of reward mass. This is the central contribution. A balance inquiry before a transfer is not treated as equivalent to the same inquiry afterward, and a fluent answer cannot conceal a wrong account state.

On that narrow test, the selected checkpoint increased average held-out reward from 0.610 to 0.697. The 12-billion-parameter reference scored 0.690, so the trained 4.5B model crossed its target with 2.7 times fewer active weights. An order-strict rescore rose from 0.590 to 0.679, while the share of tool calls in the correct order increased from 0.861 to 0.919. The model also shortened its own output from 852 to 602 tokens per dialog, a 29% decline, even as average tool calls increased from 3.9 to 4.3. Under the authors' uncached compute estimate, it used 0.58 PFLOP per dialog against 2.00 for the 12B baseline. These are modeled inference costs inside one harness, not a bank invoice or a latency service-level result.

The failure distribution prevents a triumphalist reading. After training, 21.8% of held-out tasks failed in both attempts, 17.7% passed only one of two, and 60.5% passed both. The simple happy-path control slipped from 0.821 to 0.812. Public benchmark changes were mixed: MMLU-Pro and GPQA-Diamond improved, while IFEval, LiveCodeBench and BBEH declined slightly. Held-out reward peaked at training step 180 and then fell while training reward kept rising, so the authors selected the earlier checkpoint. This is evidence that a compact model can be specialized efficiently inside the task distribution; it is not evidence that it can safely operate an unconstrained bank or replace production controls.

The paper's preference-training study draws an even sharper line between conduct and capability. On a separate roughly 800-case benchmark with three attempts per case, out-of-scope refusal rose from 52% to 80%, credential handling reached 100%, and asking instead of guessing when information was missing rose from 38% to 46%. But multi-tool chains barely changed, from 20% to 21%. The authors also disclose that the published tables do not identify which preferred-response construction produced the reported checkpoint, and that reasoning quality was graded by one language-model judge in one run without confidence intervals. Preference tuning improved behaviors the model already knew; it did not manufacture missing multi-step skill.

A second September 3 paper explains why that judging caveat is operational, not academic. In two preregistered campaigns covering 52,988 request attempts, researchers found same-window repeat rankings at Spearman 0.400 against a required 0.90 and byte-identical next-day replay agreement of 0.78 against a required 0.99. Four shared-endpoint providers showed median agreement from 0.74 to 0.88, and exposed metadata did not predict the instability. The study is deliberately narrow: it examined structured ranking readouts on shared inference services, not model internals or every judging task. Still, it supports a sensible hierarchy for banking evaluation—prefer deterministic account-state, authorization and sequence checks; pilot any language-model judge as a measurement instrument before letting it approve a release.

GitHub's HydraFusion preview shows the same systems-level economics in a different domain. Its Copilot CLI router chooses a single model, a cascade, or a cross-family critique workflow. In GitHub's best tuned offline configuration, verified TerminalBench 2.1 quality exceeded Claude Opus 5 by 4.9 percentage points at 67% lower estimated cost. On DeepSWE and GitHub's internal CheckpointBench, quality was respectively 1.5 and 0.1 points lower while estimated cost fell 36% and 65%. The inputs and conditions were controlled, but the figures are benchmark estimates, one benchmark is internal, and GitHub says real-workload latency, reliability, caching and safety still need validation. The shared lesson is not that one router wins; it is that the economic unit has shifted from a model call to a measured workflow.

Capital is following that possibility. Gimlet Labs announced a $300 million Series B led by Andreessen Horowitz to build an inference cloud that divides work among GPUs, CPUs, near-memory compute and dataflow accelerators. Gimlet claims 5–10 times faster performance for the same power footprint, billions in contracted revenue added since March, a gigawatt-scale data-center pipeline and movement toward hundreds of megawatts of managed capacity. None of those operating claims was independently audited for this edition; contracted revenue is not recognized revenue, pipeline is not delivered capacity, and the company disclosed neither valuation nor customer names. The round nevertheless makes the financial bet legible: investors expect routing, specialization and power efficiency—not only larger accelerators—to carry economic value.

The macro backdrop raises the hurdle for that bet. The U.S. Bureau of Labor Statistics reported 162,000 additional nonfarm payrolls in August, well above the prior 12-month average of 31,000, with unemployment unchanged at 4.1% and average hourly earnings up 3.1% from a year earlier. Yet information employment fell 23,000, including 8,000 jobs at computing-infrastructure, data-processing and web-hosting providers. The two surveys are preliminary and revised over time, and nothing in the release establishes that AI caused those losses. Federal Reserve Governor Christopher Waller separately called AI and data-center plans a source of rapid business investment, while stressing that the sector is capital-intensive and that a hot August inflation reading could justify a September rate increase. Those are one policymaker's conditional views, not an FOMC decision.

For a bank or regulated fintech, the implementation sequence is now clearer. Start with a small set of actions whose correct outcome can be expressed as authorization rules, permitted tool order and final ledger state. Build replayable cases from synthetic identities and versioned product rules. Keep the evaluator outside the agent's control, report repeated-trial failures rather than only average reward, and price prompt rereads, tool calls, latency and exception handling together. Human approval must remain at state-changing steps until the stubborn failure bucket is understood. A sovereign or on-premise model can reduce data exposure, but it does not by itself solve access control, audit logging, model updates or recovery from a wrong transaction.

The next evidence should be harder than another benchmark average. NPCI needs to publish artifacts or access sufficient for independent reproduction, identify the preference checkpoint, run multiple judges or human agreement studies, and test on policy drift and longer chains. HydraFusion needs production distributions for latency, retries and total billed cost. Gimlet needs delivered capacity and customer-verified efficiency. U.S. inflation data and the September 15–16 Fed meeting will set the near-term cost of capital around these deployments. The defensible conclusion is specific: FiMI Banking shows that verifiable workflow design can let a smaller model clear a larger baseline at lower modeled compute. Its 21.8% persistent-failure share shows why the verification layer, not the headline score, is the product.