Standup notes that don't suck: a 90-day experiment

We ran a controlled experiment last quarter. Two engineering teams of similar size, similar tenure, similar work. Team A kept doing what they'd always done: a rotating note-taker writes a recap each day in Notion. Team B switched to AI-generated standup notes from their meeting copilot, auto-posted to Slack.

We tracked five metrics across 90 days: note completeness, note consistency, readership, follow-up question count, and engineers' subjective satisfaction. The results were uncomfortable.

Team B won on every metric. Not by a small margin. By margins large enough that we ran the numbers three times to make sure we hadn't made a measurement error.

This post is what we learned, what it cost us to learn it, and why the result was more about how humans behave in meetings than about how good the AI was.

The setup

Both teams ran a 15-minute daily standup at 10 AM. Both teams were composed of 6-8 engineers plus an EM and a PM. Both teams were working on roughly comparable surface area (one was a billing-adjacent backend team, the other an experimentation infra team).

Team A's protocol:

• Rotating note-taker, one engineer per week
• Notes were taken live during the meeting in a Notion page
• The note-taker posted a link to the Notion page in the team Slack after standup
• Anyone who missed standup was expected to read the Notion page

Team B's protocol:

• Meeting copilot ran during standup
• Auto-generated summary was posted directly to the team Slack channel within 90 seconds of the meeting ending
• Action items were extracted with explicit owners
• The summary included a section on blockers and a section on decisions made
• The full transcript was available via a link, but the summary was self-contained

We measured for 90 days. Both teams knew they were being measured but didn't see each other's metrics during the experiment.

The five metrics

Completeness

We defined completeness as: of the items discussed in standup, what fraction made it into the notes? We sampled 12 random meetings per team and compared the meeting recording against the notes.

• Team A: 64% completeness on average. Range: 38% to 89%.
• Team B: 91% completeness on average. Range: 78% to 96%.

The AI didn't catch everything — there were always small mentions it missed. But the variance was much lower. Team A's worst notes (38% completeness) corresponded to the weeks where the rotating note-taker was Frank, who is a brilliant engineer and a notoriously distracted writer. Team B's worst notes (78%) corresponded to a week when the audio quality was bad because two engineers were on bad WiFi.

The takeaway: hand-written notes have a high variance because they depend on who's writing them. Auto-generated notes have lower variance because the model is consistent in a way humans on a rotating schedule aren't.

Consistency

We defined consistency as: across the 90 days, how often were the notes posted on time, and how often did they follow the same format?

• Team A: Notes were posted within 30 minutes of standup 71% of the time. Format consistency (same sections, similar structure) was 58%.
• Team B: Notes were posted within 30 minutes of standup 100% of the time. Format consistency was 100%.

The 29% of Team A's days where notes weren't posted on time included three days where they weren't posted at all (the note-taker forgot, was sick, or got pulled into an incident). On those days, anyone who missed standup had to ask someone what had happened.

The takeaway: a system that runs without human attention is, by definition, more consistent than a system that depends on human attention. This was the most predictable finding of the experiment and still managed to surprise us with the size of the gap.

Readership

We instrumented both teams' note destinations with read tracking.

• Team A: Notion page average reads per day: 4.2 (out of 8 team members, plus stakeholders)
• Team B: Slack message average reads per day: 11.8 (out of 8 team members, plus stakeholders — meaning some non-team-members were reading)

Team B had nearly 3x the readership. This was the metric we were most surprised by, because Team B's notes weren't dramatically better in absolute quality — they were just more reliably present, in a place where people already were.

The Slack channel mattered enormously. Team A's notes lived in Notion, which required clicking a link, switching context, and waiting for a page to load. Team B's notes lived directly in the conversation channel that the team was already in. Eight extra readers per day, just from removing two clicks.

We also noticed that adjacent teams started reading Team B's standup notes — the EM of the platform team, the PM on a sibling product, an SRE who liked to keep tabs on infra changes. The notes had become a casual broadcast to the wider org, not just an internal artifact. Team A's notes never got this kind of cross-team readership.

Follow-up questions

We counted, in each team's Slack channel, how many messages contained the pattern "did we decide X?" or "what happened in standup about Y?" or "who's working on Z?"

• Team A: 8.4 such questions per week
• Team B: 2.1 such questions per week

A 4x reduction. The notes were doing more of the work of synchronizing the team, so people had to ask fewer recovery questions. This translates directly to time saved: each follow-up question costs about 6-10 minutes of total attention (asker, responder, possible third-party chime-in). Team B was saving the equivalent of ~40 minutes per week of pure recovery work.

Subjective satisfaction

We surveyed both teams at the end of the 90 days. We asked: "Are the standup notes useful to you? Rate 1-5."

• Team A average rating: 2.8
• Team B average rating: 4.1

This was the result we were most curious about, because it's the most subjective. Team A's lower rating was driven primarily by engineers who said the notes were "incomplete" or "I don't read them anyway." Team B's higher rating was driven by engineers who said the notes were "a useful place to catch up after I missed a meeting" or "I reference them later in the week."

Two engineers on Team B initially said they didn't like the AI version because "it felt impersonal." By the end of the 90 days, both of them had reversed. The reason for the reversal, in both cases, was that the AI version was actually present every day — and the impersonal-but-present beat the personal-but-missing.

What we didn't expect: the change in meeting behavior

Here's the result that surprised us most. Team A's standup quality, as a meeting, was measurably worse than Team B's.

We watched 10 random standups from each team. On Team A, the rotating note-taker — whoever it was that week — was visibly less engaged in the meeting. They were typing notes. When their own turn came, their update was rushed because they were trying to type and talk at the same time. They asked fewer follow-up questions of teammates because they were focused on capturing the previous update.

On Team B, the AI was handling the note-taking. Every engineer participated equally. The conversation was a bit faster — by maybe a minute — because nobody was waiting for someone to finish typing.

We did not predict this. We assumed the meeting itself would be unchanged and only the artifacts would differ. We were wrong. The act of taking notes manually was, on its own, degrading the quality of the meeting. By removing the note-taking burden, Team B made the meeting better as well as the notes.

This is the most underrated argument for AI note-taking. People talk about it as a productivity tool for after the meeting. The bigger value is what it changes during the meeting.

The team A objection (and why it didn't hold up)

Throughout the experiment, Team A had a recurring objection: "But our human notes are more nuanced. The AI summarizes; a human captures intent."

We took this seriously. When we sampled the notes, we did find cases where Team A's note-taker added a useful editorial sentence — "the team seemed especially frustrated with the design review delay" or "we should probably revisit this decision in two weeks."

These editorial moments were real and valuable. They were also rare. About 1 in 8 meetings had an editorial flourish like this. The other 7 in 8 were mechanical recaps that the AI did as well or better.

For the 1 in 8 case, we proposed a hybrid: let the AI handle the mechanical recap, and let any engineer optionally add an editorial comment to the Slack thread. This worked. Team B started doing this naturally around day 40 of the experiment — the EM would often add a sentence or two of editorial commentary in a thread under the auto-generated summary. The thread became a place for human nuance to live on top of the AI's structural capture.

The hybrid is better than either pure approach. Pure human is too unreliable. Pure AI lacks the editorial layer. The hybrid puts the AI in charge of consistency and the human in charge of meaning.

What it cost to find this out

The experiment cost a real amount of EM attention. Three months of running it, setting up the measurement framework, weekly check-ins, the final survey, the analysis. Plus the cost of Team B's tool subscription.

But the cost of not finding this out is the thing we'd estimate as larger. The org was running 14 engineering teams. Most of them had Team A's protocol (rotating human note-taker). Most of them were having the same quality degradation we measured. We estimate the org was losing roughly 90 minutes per engineer per week to bad standup hygiene — across 80 engineers, that's 120 hours per week.

The experiment cost maybe 20 hours of total attention. The result has now propagated to 12 of the 14 teams. The ROI was orders of magnitude positive within a quarter.

What we'd tell another team

If you're considering whether to move standup notes from human to AI: do the experiment. Don't decide in the abstract. Run one team for 60 days, measure the five metrics, and let the data tell you.

A few specific tips if you do:

1. Post the notes in Slack, not in Notion. The destination matters as much as the source. Notes that require a click don't get read.

2. Include explicit owners on action items. Without this, the notes are descriptive but not actionable.

3. Don't try to perfect the AI before launching. The first version will miss things. That's fine. Ship it imperfect, let the team correct it in threads, and iterate.

4. Survey the team about how it changed the meeting itself. This was our most surprising finding and the one most likely to be missed if you only measure the artifacts.

5. Don't ban editorial commentary. The hybrid model — AI structure + human editorial — is the strongest. Let people add color.

We were unprepared for how clean the result was. We expected a marginal improvement, the kind of thing where you have to squint to see the difference. We got something else. Auto-generated standup notes are, in our experience, just better — and the reasons are mostly about human behavior, not AI capability.

Your mileage may vary. But the experiment is cheap, the answer is fast, and the upside if it works for your team is real.