Model Evals
Image2Cal turns calendar images into structured events and manages them through a chat agent. These are the AI models behind it, benchmarked on how accurately they extract events from real calendar screenshots and how correctly they perform calendar actions. All rates are measured against hand-verified ground truth.
Last run Nov 1, 2026, 4:00 AM · 2 calendar images · 3 samples/image · 20 tool-call scenarios
| # | Model | Avg duration | Field extraction | Tool calls |
|---|---|---|---|---|
| 1 | GPT-4o | 1.56s | 100% | 100% |
| 2 | GPT-4.1 Mini | 1.70s | 100% | 100% |
| 3 | Claude Haiku 4.5 | 2.16s | 100% | 100% |
| 4 | Gemini 2.5 Flash | 2.87s | 100% | 100% |
| 5 | Mistral Large | 3.46s | 100% | 100% |
| 6 | Image2Caldefault | 1.69s | 95% | 100% |
| 7 | Claude Sonnet 4.6 | 4.03s | 85% | 100% |
| 8 | Gemini 2.5 Flash Lite | 0.93s | 84% | 83% |
How we test
Field extraction & year
Each model extracts events from 2 real calendar screenshots (red-sox-schedule.json; award-season.json) through the same pipeline the product uses. Field extraction is the percentage of all data fields — year, month, day, title, start/end time, location — that match ground truth. The year is part of this: it usually appears only once in the image (e.g. a season header), so the model must read it from the image rather than default to the current year. Results are averaged over 3 samples per image.
Tool calls
The chat agent performs calendar actions by calling tools. Across 20 scenarios, Tool callsis the percentage where the model picks the right tool (add / update / delete / share / export / list, or staying conversational) with the correct arguments, including timezone-resolved dates (e.g. "tomorrow at 4pm" → the right YYYY-MM-DD and 16:00).
The eval suite is open in our codebase and runs automatically on the 1st of each month. Want to add a model or a harder test case? Get in touch.