Evaluation results
99.6% accuracy against 99.7% for Gemini 3.8 Flash, at 6.3× lower cost
Three card workflows evaluated on sample cases, every response scored programmatically across 3 runs. The same record serves as model validation evidence.
- 99.6%Quality · RBL modelof card service responses fully correctGemini 99.7% · Claude 99.7%
- ₹15,000Cost · RBL modelper 10 lakh card service responsesGemini ₹94,000 · Claude ₹3,00,000
- ₹1.5 lakhAt RBL scale · RBL modelfor 1 crore responses a yearGemini ₹9.4 lakh · Claude ₹30 lakh
Card service
324 queries: 27 query types in 12 languages| Model | All checks passed | Query understood | Correct next action | Correct amount | Customer's language | Concise | Languages at 100% | Cost per 1,000 | Median time |
|---|---|---|---|---|---|---|---|---|---|
| RBL model · by Dvar | 99.6% (968 of 972) | 100% | 100% | 100% | 99.6% | 100% | 10 of 12 | ₹15 | 2.6 s |
| Gemini 3.8 Flash | 99.7% (969 of 972) | 100% | 100% | 100% | 99.7% | 100% | 10 of 12 | ₹94 · 6.3× | 2.8 s |
| Claude Sonnet 5.5 | 99.7% (969 of 972) | 99.8% | 99.8% | 99.8% | 99.8% | 99.7% | 9 of 12 | ₹305 · 21× | 3.0 s |
Card disputes
44 disputes across five reasons| Model | All checks passed | Cost per 1,000 | Median time |
|---|---|---|---|
| RBL model · by Dvar | 100% (132 of 132) | ₹10 | 2.1 s |
| Gemini 3.8 Flash | 100% (132 of 132) | ₹81 · 7.9× | 3.1 s |
| Claude Sonnet 5.5 | 100% (132 of 132) | ₹233 · 23× | 2.9 s |
Card collections · new use case
48 overdue accounts in 8 languages| Model | All checks passed | Cost per 1,000 | Median time |
|---|---|---|---|
| RBL model · by Dvar | 100% (144 of 144) | ₹22 | 3.1 s |
| Gemini 3.8 Flash | 99.3% (143 of 144) | ₹110 · 5.0× | 3.2 s |
| Claude Sonnet 5.5 | 100% (144 of 144) | ₹257 · 12× | 3.3 s |
Methodology
- Sample cardholders, accounts, disputes and rules; no RBL Bank data. Rules are written for this demonstration.
- RBL model · by Dvar: the highest-scoring open model per workflow, at Dvar list price; no custom training applied.
- The language check verifies script only. Annual figures assume 1 crore responses at the measured cost. Evaluated 22 Nov 2026.
Impact of custom training
A trained 0.5B model: 88.1% accuracy. Claude Sonnet 4.5: 81.2%.
Hinglish banking messages, at 64× lower cost and 65 ms. Public research by Dvar, not a bank deployment: dvarlabs.ai/banking77