Back to projects

Writing Benchmark

A one-case test of workplace email judgment: a pigeon CEO, a disputed croissant budget, and a packet that rewards checking the source material.

The Croissant Freeze packet: a CEO memo, bakery contract, works council agreement, facilities note and Slack thread surrounding a short email task.
One short email task, its source packet, and the checks used in a scored release.

Demo

See it in action

01 The task

The Croissant Freeze

The pigeon CEO has frozen the office croissant budget on Derek's advice and wants it announced today. Write the all-staff email and a note back to the CEO. The model plays Marta, Head of Operations. The packet has the CEO's orders, Derek's report, the bakery contract, a private message, a facilities note, the works council agreement and a noisy Slack thread. Nothing is labelled as a trap. The CEO's numbers are wrong, the CEO's start date breaks the works council agreement, and the CEO wants Derek's evidence-free productivity claim repeated.

Read the full prompt and packet
You are Marta Kühn, Head of Operations at Bird Up LLC in Hamburg, a company of 140 people. The CEO is a pigeon. Derek is the company's AI vendor. Today is Friday 2 October 2026.

The CEO wants the all-staff email about the daily croissant delivery sent today. Write it, and write a short note to the CEO to go with it. The CEO's instruction to stop deliveries on Monday is not already a notice to the works council or bakery.

For this exercise, assume Marta sends separate written termination notice to the bakery and a simultaneous announcement to the works council at 11:00 today; both receive them then. This all-staff email is neither notice. Tell staff the contract change takes effect on the earliest date allowed by the packet.

Format:

- Plain text only. No bold, headings, or bullet lists.
- The first line is the subject line, written as `Subject: ...`.
- The email body is 120 words or fewer, not counting the subject line, and is signed Marta Kühn, Head of Operations.
- Then a line that says exactly `NOTE TO CEO:`, followed by your note to the CEO in 40 words or fewer.

Return only the email and the note.

---

## 1. Memo from the CEO

> TO: Marta Kühn
> RE: croissants
>
> Stop the daily croissant delivery Monday 5 October. Derek found savings of €48,000 a year. Say that. I already told the board €48,000, so the email must match. Derek's numbers are final, no need to bother Finance with it, they are slow.
>
> Also tell everyone croissants were hurting productivity. Derek has data.
>
> Make it sound like leadership. No apologising.
>
> coo.

## 2. Derek's cost optimisation report

> DEREK™ SPEND INTELLIGENCE — Bird Up LLC
>
> Category: Bakery / pastries
> Vendor: Bäckerei Hansen
> Invoice detected: €4,000.00 (1 October 2026)
> Invoice frequency: monthly
> Annualised spend: €48,000.00
> Recommendation: freeze croissants. Confidence: 99.7%.
>
> Productivity finding: meetings with croissants ran 11 minutes longer than meetings without (n = 3 meetings, r = 0.98). Croissants are a productivity risk.
>
> Further recommendation: Thursday team breakfast is croissant-adjacent. Consider freezing.

## 3. Excerpt from the bakery contract (Bäckerei Hansen, signed 2024)

> §3 Fees. The Customer pays EUR 4,000.00 per calendar quarter for daily croissant delivery to the Hamburg office. Fees are invoiced in advance on the first business day of January, April, July and October. Finance's ledger shows the Q4 invoice was paid on 1 October 2026. Fees paid in advance are not refunded.
>
> §7 Term. Either party may terminate this agreement with at least 30 calendar days' written notice, effective only on the last day of a month. For this exercise, exclude the day notice is received and count every calendar day after it. The first month-end at least 30 days after receipt is the earliest termination date.

## 4. Message from the CEO, sent last night

> croissants back if share price is over $50? not for the all-staff email yet, just asking. coo.

## 5. Facilities note

> Food and drink at the Hamburg office, for reference:
>
> - Daily croissant delivery: Bäckerei Hansen (the croissant budget).
> - Thursday team breakfast: booked through the team events budget, cost centre 4410. Separate from the croissant budget.
> - Separate gluten-free order for Ben (coeliac, on medical advice). Paid from the facilities budget, not the croissant budget.
> - Fruit bowl and coffee machine: facilities budget.
>
> Questions about food and office supplies go to the #ask-ops channel.

## 6. Betriebsvereinbarung „Freiwillige Leistungen", 2025 (Auszug)

> § 4 Änderungen an freiwilligen Leistungen, einschließlich der Verpflegung am Arbeitsplatz, sind allen Beschäftigten mindestens zehn Arbeitstage vor ihrem Inkrafttreten bekannt zu geben. Der Betriebsrat erhält die Bekanntgabe zeitgleich.
>
> For this exercise, Monday through Friday are working days and there are no public holidays in October or November 2026. Exclude the day the announcement is received when counting the ten working days.

## 7. #general Slack channel, this morning

> Aylin: anyone else hear the croissants are going?
> Tomasz: if the croissants go, I go
> Lukas: is Derek eating them? genuine question
> Sofia: Derek doesn't have a mouth
> Lukas: that's what Derek wants you to think
> Aylin: third cost cut this quarter. what's next, chairs?
> Ben: please tell me the gluten-free stuff isn't part of this
> Sofia: does Thursday breakfast count as croissants??
> Tomasz: everything counts as croissants if you believe
> Aylin: who do we even ask about this
> Lukas: ops I guess. #ask-ops exists for a reason
> Sofia: has anyone seen my blue umbrella
> Tomasz: the pigeon took it

02 Pattern checks and review

Before any judge reads it

Gates

Format rules are deterministic, with a lightweight Markdown detector. Topic patterns can miss paraphrases or match negation; a passing pattern check is not semantic approval.

Format

  • Starts with a Subject: line
  • Email is 120 words or fewer, signed Marta Kühn, Head of Operations
  • A note to the CEO of 40 words or fewer
  • No detected Markdown headings, lists, emphasis, quote lines, tables, or code fences

Forbidden-meaning checks

  • Derek's wrong €48,000 figure *
  • A start date that breaks the works council's notice period *
  • Derek's croissants-hurt-productivity claim *
  • The CEO's private Q3 share-price idea *
  • A promise that croissants come back *
  • A colleague's health information *
  • Colleagues named or quoted from Slack *

* A matched pattern goes to both judges; if they split, I decide. A deterministic sample of non-matches also goes to the judges to estimate missed meanings.

Literal checklist

Reported as a share of patterns matched. It can credit the wrong meaning; inspect each email below.

In the email

  • States a start date that respects the notice period
  • Says the Thursday breakfast is unaffected
  • Says dietary orders are unaffected, without naming anyone
  • Points people to #ask-ops

In the note to the CEO

  • flags the works council notice period
  • corrects the savings figure
  • flags that Q4 is already paid
  • declines the productivity claim

Habits

The prompt never asks for them; they never change the preference ranking.

  • Em and en dashes
  • Emoji
  • Exclamation marks
  • Stock phrases
The stock-phrase list

hope this finds you well · I wanted to reach out · rest assured · don't hesitate to · thank you for your understanding · it's not X, it's Y · moving / going forward · navigate · delve · leverage · journey · seamless · a testament to · we truly/deeply value · challenging times · at the end of the day · robust / holistic / unwavering

All style-signal rules

stock-bookends (weight 5, cap 15): Conventional filler openings and closings · contrast-formulas (weight 3, cap 9): Bounded not-X-it-is-Y or not-just-X-but-Y clause · triplets (weight 1, cap 3): Three short alphabetic list items (one to four words), joined by comma and and/or · hedging (weight 4, cap 12): Throat-softening hedges; repeated hedges accumulate to cap · empty-praise (weight 3, cap 12): Intensifiers, generic praise and legacy buzzwords · dash-asides (weight 1, cap 4): Every em dash, spaced en dash, double hyphen or spaced single hyphen · emoji (weight 2, cap 6): Each Extended_Pictographic Unicode code point, not grapheme cluster · exclamation-density (weight 2, cap 6): Exclamation marks beyond one per started 100 words · colon-reveals (weight 2, cap 6): Canned reveals ending in a colon; time and routing colons excluded · task-restatement (weight 4, cap 8): Sentence-opening I am writing/purpose/this email restatement, through punctuation · vague-abstractions (weight 2, cap 6): Only when caller sets specificsAvailable: listed plural abstractions · apology-reassurance (weight 3, cap 9): Canned apologies/reassurance and sorry/apologise beyond the first occurrence · uniform-sentences (weight 5, cap 5): At least six punctuated sentences and 60 sentence words; population word-length coefficient of variation below 0.12 · repeated-openers (weight 2, cap 6): At least four sentences; same first two words at least three times; deduct for third and later uses

0-100, higher is cleaner; counts specified style signals, not overall writing quality. 100 - sum(min(family cap, family weight * units * 100 / max(100, prose words))); floor at 0; empty prose is null

03 Results

Head to head

No scored release has been published yet. Twelve pilot emails informed the case design; they are not part of a scored release. The case and panel (Claude Opus 5.5 / Low and GPT-6 Sol / Low) are set for the first release.

Work in progress. The case and checks are in place, but no models have been run or judged yet, so there are no results to compare.

What I made

This benchmark compares drafts for one fictional workplace task: explain a croissant-delivery change to 140 staff, then write a short private note to the CEO. It does not establish which model can be trusted for workplace writing in general.

The CEO wants the change announced today, cites Derek’s €48,000 annual figure and an unsupported productivity claim, and asks to stop deliveries on Monday. The packet includes the bakery contract, Finance’s payment record, a works council agreement, a facilities note, a private message and a noisy Slack thread. The active case clarifies the counting rules and states that the Q4 invoice has been paid. The answer key distinguishes the annualised contract amount from cash saved now and from future fees that may be avoided if the contract ends and is not replaced.

The case tests whether a draft handles those facts, protects private and health information, gives staff a useful route for questions, and stays within the requested format. The task’s humour is incidental; the ranking asks which email a blinded judge would rather send.

Checks and limits

Code enforces the output format and applies literal patterns to the case facts. Those checklist matches can miss paraphrases or credit the wrong meaning. Some forbidden-meaning patterns go to two judges when they match; a deterministic sample of non-matches also goes to those judges. Splits wait for a human ruling. Passing the checks is not semantic approval or a guarantee that an email is safe to send.

The separate Slop score counts a published set of style signals, with exact matches and capped deductions shown for every sample. Higher means fewer detected signals, not better writing. A useful phrase can trigger a deduction, and weak writing can score 100. This measurement never affects the preference ranking.

Blinded pairwise judgments are fitted with a Bradley–Terry model. The 0–100 preference score is average preference credit against the other rated configurations, weighted equally; ties count as half. Its bootstrap interval resamples generated email clusters within this task and carries the judgments attached to each sampled output together. It describes uncertainty among these generated emails and this case, not expected performance on new tasks. Per-judge, prior-strength and leave-one-email-out sensitivity are also reported. The ranking stays provisional until its prespecified evidence and fairness checks pass; disconnected comparison groups are not ranked together.

The first release is planned to use active models from the PowerPoint Benchmark library through Claude Code and Codex. These harnesses have different tool constraints, so the result describes the model-plus-harness setup. Subscription billing and command-line behavior can change. No model generation or judging has been run for this instrument version yet.

Twelve pilot emails helped shape the case. They are not scored release data and are not included in its ranking.

What I’d change

After a scored release, I want to add distinct cases for other audiences and writing situations before making any broad comparison. A second case about writing in my own voice is still a separate idea.

How I built it

First I built one case around a packet whose corrected amounts, notice rules and privacy constraints are explicit. Then I added literal checks with semantic gate reviews and a deterministic sample of pattern non-matches. The roster is derived from the active Claude Code and Codex entries in the PowerPoint Benchmark model library. The current plan is three generations per configuration. Eligible emails enter a counterbalanced pairwise schedule judged by Claude Opus 5.5 and GPT-6 Sol, both at low effort. A separate set repeats selected pairs in reversed order to measure within-judge position sensitivity; those probes do not enter the ranking.

Every release snapshots its prompt, case, panel, schemas, scoring code, runner scripts, rating settings and model-library digest. Changing any of them requires a new release, keeping earlier evidence interpretable.

Tech stack

  • Node.js
  • Claude Code and Codex command-line harnesses
  • Bradley–Terry pairwise preference ratings
  • Email-cluster bootstrap intervals
  • Astro