May 20, 2026 / Exploring
Consulting AI Benchmark
A dashboard I used to compare AI tools on consulting tasks during an internal evaluation.
Demo
See it in action
Why I built it
I wanted to compare AI tools on the kinds of tasks consultants actually do, without using client data.
The task-library datasets are AI-generated versions of logistics cases. They do not contain proprietary client information.
What I built
Five to ten colleagues used the dashboard during the evaluation.
I built it in November and December 2025. The dashboard compared Copilot, ChatGPT, and Gemini. It helped identify OpenAI as the front-runner for our use cases. As part of the same evaluation, Anthropic was ruled out because its token-based costs were too unpredictable and high.
The app runs on Google Cloud Run, stores results in Firestore, and uses the Google GenAI SDK for inference.
What I learned
Building the dashboard was easier than designing the benchmark. Easy test cases made every model look good and did not help with the decision.
The comparison needed to show task quality and cost predictability, not just one overall score.
What I’d change
I would automate the tests with Promptfoo and make the cases harder.
I would also show where the test data is too thin to support a firm conclusion.
How I built it
I created public test datasets based on consulting tasks, then built the dashboard to compare results by provider, task type, weighted score, and overall score.
Tech stack
- Google Cloud Run
- Firestore
- Google GenAI SDK
- AI-generated public test data
- Promptfoo planned for future automation