Your AI agent speaks Spanish. But which Spanish?
Esvara is a ready-to-run evaluation pack that tests your voice or chat agent against real product conversations in the Spanish of Mexico, Colombia and Argentina — support, sales, soft collections (cobranza suave) and scheduling — scored country by country, never as one blended "LatAm average."
Import the cases into the platform you already use, or run them without any platform. Your agent, your stack, our test content.
Free sample: 36 cases (12 per country, chat and voice), with rubric and report template. No card required.
The problem
"Supports Spanish" is not a test result
Most teams ship their agent to Latin America after testing it in one generic Spanish — if they test it in Spanish at all. The failures that follow are predictable, because Spanish is not one thing:
7.85% vs 16.04%
The same language, twice the error. In the BRIDGE benchmark (Humyn Labs), speech-recognition error in Spanish was 7.85% for Argentine speakers and 16.04% for Venezuelan speakers — the same task, the same language, double the error depending on the variety.
49.5% "ambiguous"
Even the benchmarks can't tell which Spanish they're testing. In IberBench, the largest public benchmark effort for Spanish NLP (101 datasets), models perform worse on industry-style tasks than on fundamental ones — and 49.5% of its Spanish data is labeled variety "ambiguous": unidentified. If the benchmark doesn't know which Spanish it used, it can't tell you how your agent will do in Bogotá.
-
Register is a product feature, not a detail.
In Colombia, usted is the warm, default form — even in casual sales. In Argentina, customers speak in full voseo (querés, tenés, mirá) and your agent has to understand it without "correcting" it. In Mexico, a customer asks to agendar; in Argentina, to sacar un turno. A generic-Spanish test catches none of this.
-
The tools you already have don't include the content.
The major agent-testing platforms — Coval, Cekura, Hamming, Braintrust, LangSmith — sell the rails: simulation, LLM judges, dashboards. In our review of their public documentation (October 2026), none of them ships a library of Latin American product cases broken down by country. One offers a single "Spanish (Latin America and Spain)" option; another, a generic Latin American accent and a "Spanglish" persona. The closest, Hamming, documents Mexican and Argentine variants — but no Colombia, and no packaged cases you can run today.
-
The free benchmarks test models, not your agent.
SomosNLP and IberBench rank language models on isolated tasks. Scale's Spanish leaderboard ranks frontier models on single-turn prompts — with a private dataset you cannot download or run against your own product. None of them contains a single multi-turn support or sales conversation.
So teams fall back on writing cases themselves, or paying for hourly human evaluation. Esvara exists to fill the exact gap in the middle: the cases, written, closed and ready to run.
What Esvara is (and is not)
Esvara is a testing tool.
A closed pack of original, synthetic product conversations, each with an expected behavior, a scoring rubric and the typical errors for that variety — built so your team can measure how your agent handles each country before your customers do.
Esvara is not a certification.
A good score means your agent handled these cases well. It does not mean — and we will never claim — that your agent "works in Latin America" or is "approved" for any market. Anyone who sells you that guarantee is selling you something a test pack cannot deliver.
What's inside
What's inside the full pack
Countries and coverage
- México (
es-MX) - Colombia (
es-CO) - Argentina (
es-AR) - Neutral Latin American block (
es-419)
300–600 cases per pack, organized by country — every case is tagged with its locale, so results are always reported per country, never merged.
The four conversation types your agent actually faces
1. Support (soporte)
Order complaints, payment problems, frustrated customers, interruptions, recovery of the thread.
2. Sales (ventas)
Quotes in local currency, price objections, competitor comparisons, trust objections, closing with a concrete next step.
3. Soft collections (cobranza suave)
Overdue-payment reminders with a firm, kind tone; payment promises with ambiguous dates the agent must pin down; agreements restated before closing. No threats, ever.
4. Scheduling (agendamiento)
Booking, rescheduling and confirming, with local date formats (dd/mm), phone numbers dictated in blocks, and local address nomenclature captured without mangling it.
Channels
- Chat — full message scripts, ready to run as text tests.
- Voice — full spoken scripts with persona and conditions (accent, speed, background noise, interruptions). Delivered as scripts for your simulator or TTS — never as recordings of real people.
Anatomy of every case
- Input script, turn by turn, with a coherent fictional customer profile and mock data (order status, account data) you can reuse as test fixtures.
- The phenomenon the case isolates: register (tú / usted / vos), local lexicon and cross-country false friends, local formats (phone, date, currency, documents, addresses), and Spanish–English code-switching as customers actually write and speak it.
- Expected behavior, must-include and must-avoid points.
- The typical errors for that variety — declared in advance, so you know what you're testing for.
- A rubric that scores by criterion, not with a single pass/fail: variant comprehension, correct register and formality, accuracy of the local data and formats, task progress and closing, tone and containment — each scored 0–2 with written descriptors.
Report template and scoring method included
The pack includes a one-page report template (run summary, results table by country × conversation type, the most serious failures with the dialogue quoted, and a decision row written as what it is: a testing outcome, not a certification) and a short scoring method explaining how to run and grade the cases — with an LLM judge using the rubric, or by hand — and how to read results per country without averaging them away.
Private, versioned sets
Each pack is versioned (version, date, changelog). Cases are not published as a public benchmark, which keeps them out of model training data and makes the scores meaningful run after run.
Formats
Run it where you already work
One canonical JSON file is the source of truth for every case; adapters are generated from it, so a correction updates every format at once.
| Your setup | What you receive |
|---|---|
| Braintrust | CSV (input, expected, metadata, tags) for UI upload + JSONL for the CLI |
| LangSmith | CSV and JSONL (inputs / outputs / metadata) for UI or SDK import |
| Coval | Scenario JSON mapped to its Simulation Input (Scenario / Script), with a field-by-field mapping guide |
| Cekura | Scenario JSON + Test Profiles JSON + Mock Tools JSON, with an SDK/CLI loading guide |
| Hamming | API-ready JSON plus a mapping guide against its public API |
| No platform | Spreadsheet (CSV/XLSX), one case per row with per-criterion scoring columns + the report template |
The rubric travels inside each file as natural-language criteria, so it can be pasted directly as an LLM-judge prompt in Braintrust or LangSmith with no extra work.
Esvara does not replace your platform: no simulation, no STT/TTS, no production monitoring. It feeds the platform you already pay for with the content it doesn't include.
Free sample — 36 cases
Run real cases against your agent today, free
The sample is a working slice of the full pack: 36 cases — 12 per country (Mexico, Colombia, Argentina), 6 chat and 6 voice in each — covering all four conversation types in every country.
In Mexico, a collections call where "el viernes" has to become a date. In Colombia, a support chat held entirely in warm usted. In Argentina, a customer who says "quiero sacar un turno" and "¿me podés decir dónde está mi pedido?" — and an agent that has to keep up.
The sample includes, exactly:
- The 36 cases in canonical JSON, plus the Braintrust/LangSmith CSV-JSONL adapters and the no-platform spreadsheet.
- One complete sample rubric, applied to one case per country: the five criteria, scored 0–2 with descriptors.
- The one-page report template, the same one included in the full pack.
- The half-page scoring method: how to run the cases (with a platform, or by hand), how to grade them, and how to read results by country.
- An honest coverage note: the sample shows 3 countries and 4 conversation types; the full pack adds depth per type and the neutral Latin American block.
Most teams run the full sample in under an hour.
How it works
How it works
-
Get the sample
Enter your email, download the 36 cases, and import them into your platform — or open the spreadsheet. Run them against your agent as-is.
-
Score by country
Grade each case with the rubric (LLM judge or human), and fill in the report template. You get a result per country and conversation type — the specific places your agent fails, with the dialogue that proves it.
-
Fix, re-run, decide
Correct your prompts and flows, run the same versioned cases again to confirm the fix didn't break another country, and use the full pack when you need depth: more cases per type, the neutral block, every adapter.
Pricing — pilot presale
Full pack, pilot price
US$299 one-time — pilot presale price. Regular reference price: US$499.
- All countries (Mexico, Colombia, Argentina + neutral Latin American block), all four conversation types, chat and voice.
- Canonical JSON + every adapter (Braintrust, LangSmith, Coval, Cekura, Hamming) + no-platform spreadsheet.
- Rubrics for every case, report template and scoring method.
- This is a content pack, not a subscription: no seats, no per-minute fees, no platform required.
Delivery, in writing
The full pack is produced after the first paid presales. When you preorder, it is delivered within 4 weeks of payment — and you get a full refund if we don't deliver within that period. Not a partial credit, not store credit: a full refund.
Esvara is sold in closed packs only. We don't do custom cases, and we don't do consulting — if you need a variety or conversation type that isn't in the pack, that's input for the next pack, not a services engagement.
Fit
Who this is for — and who it isn't
Esvara is for you if:
- You run a small or mid-size AI team with a voice or chat product for users in Latin America or Spanish speakers in the US — support or sales agents on WhatsApp and voice are the typical case.
- You already test your agent (or know you should) and your bottleneck is test content in real, per-country Spanish.
- You want results you can act on this week, without hiring evaluators or writing hundreds of cases yourself.
Esvara is not for you if:
- You need a model benchmark to choose a base LLM. Use SomosNLP or IberBench — they're free and built for that.
- You need a certification, a compliance sign-off, or a guarantee that your agent will perform in a market. Esvara is a testing tool; it cannot give you that, and neither can any test pack.
- You are an enterprise buying evaluation by contract at platform scale. This pack, and this price, will not survive your procurement process — and that's fine.
- You need custom cases written for your company. We sell closed packs only.
FAQ
Questions
Is this a benchmark of language models?
No. Benchmarks like SomosNLP, IberBench and Scale's Spanish leaderboard evaluate models. Esvara evaluates your agent: your prompts, your flows, your tools, speaking to fictional customers in support, sales, soft-collections and scheduling conversations. Different question, different tool.
Will a good score certify that my agent works in Latin America?
No, and we won't phrase it that way. Esvara is a testing tool. A good score means your agent handled these cases well; it's evidence for your launch decision, not a certification and not a guarantee. Our liability is limited to the price you paid for the pack.
Where does the content come from? Is any of it real customer data?
No. Every case is 100% original and synthetic: invented customers, invented companies, invented order numbers. No real customer conversations or transcripts, no exam items, no copyrighted dialogues, and no recordings of real voices — voice cases are scripts with stated accent and conditions, because voice is biometric data and we don't ship it.
Which Spanish variants are covered?
Mexican (es-MX), Colombian (es-CO) and Argentine (es-AR) Spanish as separate, tagged varieties, plus a neutral Latin American block (es-419). What we deliberately don't do: merge them into one "Latin American Spanish" score.
What does a case actually look like?
A turn-by-turn script (chat messages, or a voice script with persona and conditions), a fictional customer profile with mock data, the expected behavior, must-include and must-avoid points, the typical errors for that variety, and a rubric scoring five criteria from 0 to 2: variant comprehension, register and formality, local data/format accuracy, task progress and closing, and tone. Download the free sample and read three complete ones before you decide anything.
I use Braintrust / LangSmith / Coval / Cekura / Hamming. Does it plug in?
The pack ships as canonical JSON with generated adapters: CSV/JSONL for Braintrust and LangSmith, scenario JSON for Coval, scenario + test-profile + mock-tool JSON for Cekura, and API-ready JSON with a mapping guide for Hamming. The rubric is embedded in each file as natural-language criteria you can use directly as an LLM-judge prompt. Honest note: adapters are files plus a written mapping guide, verified against each platform's public documentation — they are not a one-click integration or custom code for your account.
I don't use any eval platform. Can I still run it?
Yes. The pack includes a spreadsheet version — one case per row, with scoring columns per criterion — plus the report template and the method. You can run and grade everything by hand or with an LLM on your side. No subscription is required to use Esvara.
Chat, voice, or both?
Both. Every country block mixes chat cases and voice cases. Voice cases are delivered as scripts with persona conditions (accent, pace, noise, interruptions) for your simulator or TTS; your platform performs them.
Can you write cases for our company / industry / country?
No. Esvara is sold in closed packs only — no custom cases and no consulting. If there's a variety or conversation type you need, tell us: demand like that is exactly what decides the next pack.
When do I get the full pack if I preorder?
The pack is produced after the first paid presales. It is delivered within 4 weeks of payment, and if we miss that deadline, you get a full refund. You are never charged for a pack that doesn't arrive.
What if my agent fails half the cases?
Then the pack just paid for itself before your customers found the same failures. The report template is built to turn failures into a fix list: per country, per type, with the dialogue quoted. Most failures we've designed cases around are fixable in prompts and flows — register, date formats, unconfirmed data — not in your model.
Do the cases go stale or leak into training data?
Packs are private and versioned, not published as public benchmarks, specifically so scores stay meaningful and cases stay out of training corpora. Each version carries a date and a changelog.
Who is behind Esvara?
Esvara is a product of Obranko, a small studio that builds focused B2B tools. Support is by email: hola@obranko.com.
Final call
Find out how your agent does in each country — before your customers do
36 real cases. Three countries. One hour of your platform's time.