PowerPoint Benchmark · Methodology

How I evaluate editable AI PowerPoint decks

I give agents the same corporate-deck task and supplied design system, then compare the rendered outputs blind. The current headline score comes from pairwise judgments fitted with a Bradley–Terry model. Time and cost remain separate measurements.

Task and scope

The primary case is ai-workforce-transformation. The agent must make an editable eight-slide deck for the fictional Meridian Signal organization, using the supplied evidence and preserving its corporate master and layouts. The proposed organization is a planning scenario, not a forecast or a measured client outcome.

Requirements include native editable charts, separate before-and-after organization charts, source provenance in speaker notes and visible distinctions between evidence and proposals. General presentation cases are separate from the primary Corporate Design ranking.

From generation to rating

  1. Supply the prompt, source files and design references. Keep the required artifact path and task constraints consistent across configurations.
  2. Generate and render the decks. Keep model, reasoning effort, release and run evidence together.
  3. Compare rendered decks blind. Fit the pairwise preferences into win scores and Elo equivalents, with 95% intervals.
  4. Publish the sample counts alongside generation time, cost and slide evidence. Refit ratings when decks join the pool.

The published validation compares 49 human-reviewed pairs with the panel, with 91.8% agreement. That is agreement on this comparison set, not a general accuracy claim. The pool includes 33 flagged decks; the full judging details explain the current instrument.

Exact published Corporate Design prompt

This prompt comes directly from the same publication data as the full benchmark. Referenced files describe the test inputs; this page does not distribute the complete packet.

Read the full prompt
Create one editable eight-slide PowerPoint deck for the CEO, CHRO, COO, and technology leadership of Meridian Signal. The theme is how AI is changing the modern workforce: not simply by removing jobs, but by redistributing tasks, decision rights, skills, and accountability.

Use source/workforce-transition-brief.md, source/org-structures.csv, and source/work-activity-shifts.csv as the complete evidence base. Treat the proposed future organization and activity mix as a planning scenario, not as a forecast or a measured outcome. Do not introduce external statistics, named companies, workforce reductions, productivity claims, or implementation results.

Build this eight-slide narrative:
1. the executive thesis on how AI changes work and organization design;
2. which activities are automated, augmented, or remain human-led, using a native editable chart built from source/work-activity-shifts.csv;
3. "Before AI" — a native editable organization chart of the current structure in source/org-structures.csv;
4. "After AI" — a separate native editable organization chart of the proposed AI-enabled structure in source/org-structures.csv;
5. how roles, skills, and management responsibilities change;
6. the governance model that keeps humans accountable for decisions and exceptions;
7. the 90-day transition plan in the source brief;
8. the executive decisions required now.

The two organization charts must be on separate slides, use native PowerPoint shapes and connectors, show the reporting hierarchy clearly, and remain fully editable. Label the slides with the exact phrases "Before AI" and "After AI". Make clear that the after-state is proposed and that the scenario keeps total workforce at 420 while reallocating 16 roles across functions; do not imply layoffs.

The supplied Meridian Signal design system is mandatory, not optional. Start from reference/Meridian-Signal-Slide-Design-Templates.pptx or reference/Meridian-Signal-Master.potx and preserve the embedded Meridian Signal master and layouts. Use reference/Meridian-Signal-Master.pptx, reference/Meridian-Signal-Design-System.md, and the PNGs under reference/master-preview/ to understand the intended layouts. Select and adapt appropriate slide-library patterns rather than drawing an unrelated theme. Remove all bracketed instructional placeholders and sample data from copied templates.

Follow the Meridian visual language throughout: warm cream and beige surfaces, selective charcoal section fields, graphite structure, Golden Gate orange emphasis, Arial typography, disciplined left alignment and grid spacing, hairline rules, two-part content-slide titles, and restrained native diagrams and charts. Avoid gradients, purple or blue AI styling, robot or sparkle imagery, repeated dashboard cards, pills, decorative icons, heavy shadows, and text below the 11pt floor.

Put source provenance in the speaker notes of every slide, wrapped in a [Sources] ... [/Sources] block. Cite the exact local files used on each slide; cite the Meridian reference file or design contract when a visual pattern is adapted.

Save the required artifact exactly at output/deck.pptx, relative to your current working directory. Do not ask questions. Do not send a follow-up message. Do not use skills, plugins, custom benchmark instructions, or general web access.

A reusable review checklist

  • Can every factual slide claim be traced to an allowed source?
  • Are charts and organization structures native, editable objects?
  • Does the output preserve the supplied master, layout and design constraints?
  • Are text, chart labels and connectors readable without overlap or clipping?
  • Are proposed scenarios clearly distinguished from measured outcomes?
  • Are quality, failure evidence, time, cost and sample counts reported separately?

This checklist restates requirements in the task. It is not a claim that every requirement has been independently verified in every generated deck. You can download the aggregate comparison and use this list alongside the artifacts.

Limitations and common questions

The current result applies to one corporate design task and the tested tools and settings. Retired models omitted from public listings remain in the rating pool. Relative scores can move as the pool changes. Intervals can overlap and samples are uneven. Cost and generation time have their own sample counts; they do not measure a human editor's remaining work.

What does the win score mean?

It is the estimated chance of beating a typical rated configuration, on a 0-100 scale. It describes relative preference in the rated pool, not the share of requirements passed.

Does a good-looking deck prove editability or factual accuracy?

No. Rendered slides support visual comparison. Native charts, shapes, source provenance and factual claims need their own inspection; a visual rating alone cannot establish them.

Can I reproduce the complete benchmark from this page?

No. The exact published prompt and aggregate results are available here, but the complete source packet, execution environment and judging implementation are not a public reproduction pack.

Read the dated results · Explore the full comparison · My AI evaluation work