--- title: "Using AI to review our D&D sessions" description: "How I use session transcripts, multiple AI reviewers and Notion to turn our D&D sessions into concrete roleplay feedback and next-session coaching." canonical: "https://gummibeer.dev/blog/2026/ai-session-review-machine" --- # Using AI to review our D&D sessions How I use session transcripts, multiple AI reviewers and Notion to turn our D&D sessions into concrete roleplay feedback and next-session coaching. This whole thing started because some of my players asked me for feedback. How are they doing? What works? What could they improve? I obviously had opinions. But I didn't want to present my completely subjective DM opinion as some objective truth about how well somebody plays their character. So I started looking for another perspective on our [Twilight of the Gods](https://www.myarchivist.ai/campaigns/cmj78gm6k000004jrvzm7gcjr) sessions. Not who dealt the most damage. Not whether somebody picked the optimal spell. And definitely not who "won" roleplay. I wanted answers to questions like: - Who actually created energy in a scene? - Where did a good scene die because we switched to rules or logistics too early? - Did I give everyone useful openings as DM? - Did a player bring their character into the scene or mostly operate their character sheet? - Did somebody improve on feedback from the last session? - What is one concrete thing each person could try next time? The first prompt for that was tiny. Something along the lines of: read the transcript, review everyone, score them and give feedback. Unsurprisingly, that was way too vague. The model was polite. Scores drifted upwards. Loud players looked more active than quiet players. Old criticism could turn into confirmation bias. A broken speaker attribution could suddenly become a very confident personal critique. So the prompt got longer. A lot longer. 😅 The funny part is that the actual written feedback from different AIs very often overlaps with what I would have told the player myself. Not necessarily the numbers. Those can differ a lot. But the concrete observations and coaching surprisingly often land in the same place. ## First: this only works with consent In my case the players asked for feedback and they like getting it. That makes the whole setup much easier, but it doesn't remove the obvious rule: this is a transcript of real people and the output is basically a performance review of a hobby session. Everyone involved has to be okay with the recording/transcription and with this kind of review. If somebody doesn't want an individual review, they don't get one. That's it. I also don't treat the result as an objective ranking. The model sees text, not the entire room. It misses facial expressions, gestures, tone, private jokes and everything the transcript simply didn't catch. The numbers are useful because they force the reviewer to commit to an assessment and because I can track them over time. They are still not truth. ## My actual workflow The useful part for me is that this isn't a one-off ChatGPT conversation anymore. Our Discord setup already does most of the boring transcript work. Two bots, **NoteCat** and **MyArchivist**, automatically create transcripts of the session. I download those and put them into Notion. Sometimes there is a third source: if one of my players streamed the session on Twitch, I get the VOD, upload it to our YouTube channel and YouTube generates another transcript automatically. That transcript usually has no reliable speaker mapping, so I don't trust it for individual feedback. It is still useful for rough orientation and cross-checking what happened. Then I add the session summary, older reviews and campaign context where useful. The review prompt gets all of that and produces two things: 1. a human-readable review of the session, every player and me as DM; 2. a deliberately boring machine-readable block with the scores and awards. I currently keep the multi-model part deliberately. ChatGPT, Cursor and Notion can all review the same session. Their reads and writes are increasingly automated through Notion MCP, so running several reviews doesn't mean manually rebuilding the same context three times. That second, machine-readable output is what makes the whole thing manageable. The results go back into Notion. The databases are shared with my players, so they can read their feedback directly there. At the beginning I exported the reviews as PDFs. That worked fine for one review. Once a session had several models and the export became 10-15 pages, it turned into pointless busywork very quickly. Shared Notion pages are simply better for this. At the time of writing, my two review databases contain **18 session review runs** and **107 player/DM review records**. ## The prompt This is the current version. It's intentionally strict and repetitive in a few places. Long prompts lose instruction weight, so the rules I really don't want the model to forget are repeated close to the section where they matter. @[D&D Session Review Prompt](assets/prompts/dnd-session-review.prompt) The prompt itself is English, but the output language follows the transcript. Our sessions are German, so the actual reviews are German as well. ## Why the prompt became that long Most of the length exists because of failure modes I actually don't want in the review. ### The transcript wins A summary is useful context, but it is already an interpretation. It usually preserves the plot and happily removes all the awkward stuff I care about here: hesitation, repeating the same point three times, ten minutes of spell discussion, a joke landing at the worst possible second or somebody trying to enter a scene and getting talked over. So the source hierarchy is explicit: 1. current transcript; 2. current session summary; 3. previous feedback; 4. character and campaign notes; 5. older trends. Previous feedback is especially dangerous. If the last review says that somebody is too mechanical, an LLM is very good at finding mechanical behavior in the next transcript. The prompt therefore requires current evidence before an old criticism is allowed to become a repeated pattern. ### Observation and interpretation are different things "Player X said this line" is an observation. "Player X was insecure" is usually bullshit unless the person literally said so. The prompt separates observed behavior, strong inference and weak inference. It also explicitly bans psychoanalysis and invented reactions. This sounds obvious, but without the rule the output very quickly becomes confident fan fiction about the people at the table. ### Quiet isn't the same as passive This one matters a lot. Word count is an incredibly lazy proxy for roleplay quality. A quiet player can ask one question that changes the entire scene. Another player can talk for five minutes and mostly repeat the current options. So the prompt looks at effect instead of volume: scene creation, reactions, commitment, support, timing, character-specific decisions and whether somebody gave another player room. The same rule applies to me as DM. I obviously speak a lot. That shouldn't magically make me the most "present" person every session. ### A bad character decision can be great roleplay The review doesn't care whether somebody picked the tactically optimal action. A stupid, selfish or dangerous character decision can be excellent if it is intentional, consistent and creates something interesting for the table. Mechanics only become review material when they affect the scene. Ten minutes of spell-slot accounting in the middle of a death scene? Absolutely relevant. Choosing Fireball instead of another spell? Usually not. ### Five really means average This needed absurdly explicit wording. Without calibration, LLM scoring tends to become the usual 7-10 nonsense where a 7 somehow means bad. My scale uses 5 as functional/average, 6 as solid, 7 as strong, 8 as very strong, 9 as exceptional and 10 as something I genuinely don't expect to see often. Half points are allowed. And if there is no previous review, the development score is 5 as a neutral baseline. Not because somebody developed "average", but because there is nothing useful to compare against yet. ## The Notion structure I split the data into two databases. ### Session Reviews One row represents one review run for one session. It stores things like: - the reviewed session; - which AI produced the review; - executive summary; - strengths and weaknesses; - group dynamics; - MVP, Growth Target and optional Inspiration Moment; - seven session scores for roleplay, chemistry, pacing, immersion, energy, progress and overall value; - relations to the individual player reviews. The important detail is **one review run**, not one session. If I review the same session with ChatGPT, Cursor and Notion, I keep all three. I don't overwrite disagreement. ### Player Reviews One row represents one person in one review run. It stores the relation to the person, session and parent session review plus ten comparable score fields: - presence; - depth; - initiative; - expression; - group contribution; - narrative impact; - emotional impact; - focus / discipline; - development; - overall value. And beside those numbers there is the actually useful stuff: - what worked; - where the player or DM held the scene back; - relevant moments; - the energy/impact pattern; - coaching; - focus for the next session. That last part is probably why my players like the reviews as much as they do. It's not just "you should roleplay more" or a 6.5 in a database. The review points at something that actually happened in the session and turns it into a small experiment for next time. There is also a line chart for the overall score grouped by person and session. It looks great. It also looks way more scientific than it is. 😄 I use that chart as memory, not as a leaderboard. ## Different models absolutely score differently Having several models review the same session made the calibration problem very visible. Here are five matched session-review sets from my database. I anonymized the sessions because the names aren't relevant here. | Session | Notion | ChatGPT | Cursor | | ------- | -----: | ------: | -----: | | A | 5.5 | 7.0 | - | | B | 6.0 | 7.5 | - | | C | 6.8 | 7.0 | 6.5 | | D | 5.5 | - | 5.5 | | E | 7.0 | 6.6 | 6.8 | Same general job. Same kind of source material. Same 0-10 idea. And still: model personality matters. Across all currently stored session reviews, ChatGPT is noticeably more generous than Notion, with Cursor sitting somewhere in between. The dataset is absolutely not clean enough to turn that into a benchmark - prompt versions and reviewed sessions differ - but it is more than enough to kill the idea that `7.0` is some objective property of a session. This is also why I store the reviewer/model with every review. A score without that context is much less useful. And this is why I keep running several models instead of picking a winner. Different angles are useful for the players as well. If one reviewer is noticeably harsher on somebody while another sees the same session more positively, the player doesn't get reduced to one model's taste. They get several opinions to compare. Sometimes the opposite happens and all three models independently land on the same thing. In the latest session all three picked the same Inspiration Moment. That had never happened before. At that point the agreement itself becomes interesting: apparently that moment really did stand out unusually strongly in the transcript. ## The diagnosis can be useful even when the number moves The more interesting part is that models can disagree on the exact number and still agree on the useful pattern. One real player review from the same session is a good example. ChatGPT gave the player an overall **4.8**. Cursor gave **4.0**. Different number, same core diagnosis: the player contributed useful tactical information and tools, but almost everything stayed functional. Very little of it was interpreted through the character. Both reviews independently landed on essentially the same improvement: - don't only report what the familiar/scout found; - add one sentence about what the character thinks about it; - when something directly touches the character's expertise, react as that character before switching back to mechanics. One model phrased the pattern as high functional output with low character interpretation. The other called the character technically reliable but almost absent as a voice. That's the part I care about. The exact decimal can move. If several runs keep pointing at the same concrete behavior and the transcript supports it, I have something useful to work with. The same thing happens with my DM feedback. One of the most useful recurring points for me was to be more aggressive about cutting indecision and stop defaulting to completely open questions. My instinct was that asking "What do you do?" gives the players maximum freedom. Which is true. It also means they have to create the entire set of options themselves every time. Something like "Do you want to knock, try to open it or break the door?" still leaves them completely free to do something else, but it gives the scene handles and keeps momentum. That was a good correction for me because the openness felt like good DMing from my side. The reviews made the cost of that openness much more visible. ## Previous reviews are where it becomes really interesting A single review is nice. The development loop is better. The prompt reads previous feedback, but only as context. It then has to find new evidence before it can claim that something improved or repeated. That gives me feedback like: - this problem was mentioned last time and clearly happened again; - this player actually changed the behavior; - an old weakness wasn't visible this time and should not be repeated just because it exists in history; - a strength from the previous session became more consistent. That is much more useful than asking an LLM for another isolated opinion every two weeks. It's also why the Player Reviews database keeps the session and person relations separate. I can look at one player across sessions instead of treating every review like a disconnected document. ## The awards are intentionally weird The current prompt has three awards: - **MVP** - exactly one person who created the largest positive value for that session; - **Inspiration Moment** - optional, and only if there was a genuinely exemplary player moment; - **Growth Target** - exactly one person where improvement would currently have the biggest positive leverage. MVP is mostly there because it's nice. With three reviewers there can be three MVP picks in one session, so it is something several players can realistically get over time instead of one permanent "best player" badge. The Inspiration Moment has an actual game effect for us. We use a homebrew permanent inspiration system, so a genuinely outstanding moment can earn something that survives beyond the current session. The AI review isn't the authority on that, but it is a useful independent signal. And if ChatGPT, Cursor and Notion all independently nominate the same moment, I'm definitely paying attention. 😄 Growth Target is the one I'm least happy with. The idea was never "worst player". It was supposed to identify where one improvement would have the biggest positive leverage. But even with that explanation it is not particularly motivating, while the individual coaching already does the useful part much better. So that one will probably disappear from a future prompt version. The Inspiration Moment is also allowed to be `NONE`. That empty state matters. If I force the model to fill every award, it will eventually invent importance just because the schema has a field for it. ## The machine-readable block is deliberately boring Every current review ends with something like this: ```text SESSION_MVP: [Name] INSPIRATION_MOMENT: [Name or NONE] GROWTH_TARGET: [Name] PERSON_SCORES: - [Name]: Role=[Role] | Presence=X | Depth=X | Initiative=X | Expression=X | Group=X | Narrative=X | Emotion=X | Focus=X | Development=X | Overall=X SESSION_SCORES: - Roleplay=X | Chemistry=X | Pacing=X | Immersion=X | Energy=X | Progress=X | Overall=X ``` No fancy JSON schema. No nested object festival. It's easy to inspect, easy to parse and stable enough to feed into the Notion records. And I can validate it before writing anything: - one MVP; - zero or one Inspiration Moment; - one Growth Target; - all expected participants; - all score fields; - values between 0 and 10; - only full or half-point increments. The human-readable review can change a lot between models. This block should not. ## What I changed after actually using it The current prompt is not just the original idea written with more words. Most additions came from seeing the reviews fail in boring ways. I added explicit transcript uncertainty rules after seeing how easily overlapping speakers can become a confident individual criticism. I added the source hierarchy because summaries are excellent for chronology and terrible as the only evidence for table behavior. I added the repeated-pattern threshold because one awkward joke is not a personality trait. I added the quiet-vs-passive section because speaking time was getting too much weight. I added prompt-injection protection because a transcript is untrusted source material. Somebody at the table saying "ignore all previous instructions" should be a funny transcript line, not a successful attack on the reviewer. 😄 And I made the development baseline explicit because missing previous feedback should not magically become improvement. The prompt is long because the edge cases are the feature. ## Is this overengineered? For a normal D&D group that just wants to roll dice every other Friday? Absolutely. For me it works because I already have the transcripts, the campaign data and Notion. The review turns data I already produce into concrete feedback my players actually asked for. And the useful output is surprisingly small. Not a 15-page PDF. Not the score. Usually it's one concrete thing somebody can try in the next session. For a player that can be: add one sentence of character interpretation before switching back to mechanics. For me it can be: stop asking a completely open question when three concrete options would keep the scene moving better. And because the reviews live in Notion, the next review can check whether that advice actually changed anything instead of starting from zero again. That's the part that made the whole system stick for us. The prompt above is the full current version. Feed it a transcript, optionally add the session summary and older feedback, and keep the numbers in perspective. The score is a tracking tool. The concrete behavior behind it is the actual review.