Evaluation results
Evaluation results

99.6% accuracy against 99.7% for Gemini 3.8 Flash, at 6.3× lower cost

Three card workflows evaluated on sample cases, every response scored programmatically across 3 runs. The same record serves as model validation evidence.

  • 99.6%
    Quality · RBL model
    of card service responses fully correct
    Gemini 99.7% · Claude 99.7%
  • ₹15,000
    Cost · RBL model
    per 10 lakh card service responses
    Gemini ₹94,000 · Claude ₹3,00,000
  • ₹1.5 lakh
    At RBL scale · RBL model
    for 1 crore responses a year
    Gemini ₹9.4 lakh · Claude ₹30 lakh

Card service

324 queries: 27 query types in 12 languages
ModelAll checks passedQuery understoodCorrect next actionCorrect amountCustomer's languageConciseLanguages at 100%Cost per 1,000Median time
RBL model · by Dvar99.6% (968 of 972)100%100%100%99.6%100%10 of 12₹152.6 s
Gemini 3.8 Flash99.7% (969 of 972)100%100%100%99.7%100%10 of 12₹94 · 6.3×2.8 s
Claude Sonnet 5.599.7% (969 of 972)99.8%99.8%99.8%99.8%99.7%9 of 12₹305 · 21×3.0 s

Card disputes

44 disputes across five reasons
ModelAll checks passedCost per 1,000Median time
RBL model · by Dvar100% (132 of 132)₹102.1 s
Gemini 3.8 Flash100% (132 of 132)₹81 · 7.9×3.1 s
Claude Sonnet 5.5100% (132 of 132)₹233 · 23×2.9 s

Card collections · new use case

48 overdue accounts in 8 languages
ModelAll checks passedCost per 1,000Median time
RBL model · by Dvar100% (144 of 144)₹223.1 s
Gemini 3.8 Flash99.3% (143 of 144)₹110 · 5.0×3.2 s
Claude Sonnet 5.5100% (144 of 144)₹257 · 12×3.3 s

Methodology

  • Sample cardholders, accounts, disputes and rules; no RBL Bank data. Rules are written for this demonstration.
  • RBL model · by Dvar: the highest-scoring open model per workflow, at Dvar list price; no custom training applied.
  • The language check verifies script only. Annual figures assume 1 crore responses at the measured cost. Evaluated 22 Nov 2026.

Impact of custom training

A trained 0.5B model: 88.1% accuracy. Claude Sonnet 4.5: 81.2%.

Hinglish banking messages, at 64× lower cost and 65 ms. Public research by Dvar, not a bank deployment: dvarlabs.ai/banking77