Client: Dark Matter Labs, a social research and design lab. Type: AI qualitative analysis. When: February to September 2025, as a contract product engineer.

The problem

Most measures of whether a place is doing well, like GDP, unemployment or house prices, are chosen top-down, by experts far from the people who live there. Dark Matter Labs' Cornerstone Indicators turn that round. Communities run workshops on the question "What does thriving mean for you?", and what they say becomes the indicators they track.

The bottleneck was analysis. An expert had to read every comment from the workshops, sort them into themes, work out what was driving each one and find a way to measure it. That kept the process to a handful of communities.

What I built

My task was to prototype automating that analysis with AI, so communities could run the whole process themselves without bringing Dark Matter Labs in as consultants.

I started by interviewing the team to understand the process, then looked for tools that already solved part of the problem. Google's Jigsaw team had open-sourced Sensemaker, a set of tools for making sense of large public consultations with AI. I adapted it to smaller, more varied workshop data, and built the prototype up in small steps with the team's feedback.

The prototype:

• reads every comment from a community's workshops;
• groups them into the issues people care about;
• works out what drives each issue in that community;
• suggests ways to measure progress, using data that already exists.

I built the AI pipeline and much of the back end, alongside Dark Matter Labs' developer, who built much of the interface.

How I kept it accurate

You can't stop a model inventing things by asking it not to, so I built checks around it.

• Constrained. The AI could only sort comments into a fixed list of themes, and anything that didn't fit went to "Other" instead of being forced into the nearest one.
• Checked in code. Every comment had to come back with a valid theme. Anything missing or off the list was sent back, so nothing was dropped and nothing was made up.
• Traceable. People's own words stayed attached to every result, so anyone could check a theme against what was actually said.

There were too few past examples to build a proper test set, so quality was judged by the team's expert over several rounds of testing.

The result

The team were very happy with the prototype. I documented it and handed it to a senior engineer, worked closely with him at first, and he took it into production.

Source code

The Problem

For too long, education has focussed on the hard skills, the skills that are easy to quantify and measure. No one teaches us how to relate and communicate. It's expected that we already know. That we should have learnt what we needed in the playground, that this should be enough. Little is taught on it, but it's tested... a LOT. Both in the everyday life, and at key turning points. Both implicitly and explicitly.

There's no field in which communication matters more than healthcare. A miscommunication could lead to misdiagnosis, reduced life spans, unimaginable pain and suffering, or even not realising you are critically ill.

There is some work being done to help this. Guidance to doctors has only recently been updated to insist that they explicitly say the words 'die' and 'death', to prevent beating around the bush and issues that can be caused by a very classic British approach to communication.

Guidance to doctors has only recently been updated to insist that they explicitly say the words 'die' and 'death'.

The Solution

To help address this issue, I teamed up with some medical friends from uni to develop a training platform 'OpthalmoPrep'. Our target entry market is Ophthalmology trainees (eye doctors) who need to practice patient communication skills for specialty training interviews. We are looking to scale this to other disciplines.

The key features are:

• Scheduling between users to meet
• Meeting during which users role play patient interaction
• Meetings are recorded and assessed by an LLM to highlight strengths and weaknesses

https://ophthalmo-ai-practice-hub-frontend.vercel.app/

Scheduling

Users can post meetings at chosen time and date, other users can book these meetings

Users receive a calendar invite once they have a confirmed meeting

Meeting

Users meet in video meeting and select a case study

The candidate gets a minimal history on the patient, and has to role play the doctor and diagnose the patient

The actor is playing the patient and given a more lengthy brief

Assessment

Users get an assessment for each session

Each assessment has an overview, highlighting key strengths and weaknesses across the 5 categories

Each category gives detailed evidence in the form of quotes from the interaction

Source Code

I want my AI agents to be more autonomous. Not just faster or cheaper, but more hands off. I've come across the idea of a self-learning loop before and it seemed like a good place to start — the basic concept being that we build up a knowledge base from mistakes the agent has made in the past so it doesn't make them again in the future.

So I went looking for how to do it well. Most of what I found was about the how: which library to use, how to structure a memory store, what vector database to reach for. Useful, but it skips the harder question: what should the agent actually be learning? The most useful thing I found was a post by @itsreallyvivek called how to be good at research. It's aimed at ML researchers, but the practices translate surprisingly well to anyone doing iterative work with AI agents. And buried inside it was an idea that was 70 years old - from Claude Shannon, a founding father of information theory - that turned out to be the most essential piece of all.

How I approached this

Before doing anything else, I ran a baseline: run an agent to produce a set of learnings across a bunch of sessions with no extra guidance, then I reviewed the output. Vivek's post has a move that paid off immediately here: pull your failures, read all of them, sort them into piles, attack the biggest pile. So that's what I did: read through the learnings in detail, listed what was right and wrong, prioritised the list, then worked through the top items to understand why those mistakes were being made. To be clear, the focus here was on what the learning agent was correctly and incorrectly identifying, not where the original agent went wrong in the first place.

That left me with a clear picture of the problems. What it didn't give me was a plan. That's where Shannon came in. His opening move in a 1952 talk on creative thinking was to shrink a problem until it's nearly trivial, crack the small version, then reintroduce the difficulty one piece at a time. When I looked at the pile of problems I was trying to solve for (autonomous, cross-session, handling assumptions, correcting for overfitting) it was obviously the hard version. Shannon's advice was to stop trying to build that and find the version I could actually solve first.

What the baseline revealed

Direct feedback gets captured well. When I explicitly corrected something, or when the agent tried an approach that failed, it picked those moments up reliably. "Verify every URL" came from a session where broken links caused real problems. These learnings are grounded and useful.

Small sample sizes produce over-fit rules. When the agent extracted a learning from a single example, it tended to encode the specifics of that example rather than the underlying principle. The rule looked generalisable but was really just a description of one case. What you actually need is for the learning to capture the family of issues — the underlying principle — so a future agent can recognise the next instance even when it looks different on the surface.

Assumptions are the hardest problem. Where there was no direct feedback, the agent filled the gap — silently, confidently, wrongly. For instance, it assumed that because one source was trusted for topic X, a different source was trusted for topic Y — without being told either of those things. The agent didn't know it had made an assumption.

Learnings retained residue of the errors. When I asked the agent to revise something, it would often include the wrong version in its explanation of the fix: "Unlike the previous solution, the revised version..." The error was still in the output, just reframed. Small thing, but exactly the kind of noise that pollutes a learning file.

The approach: make it trivially small

Shannon's method was to scope down until the problem was solvable, then build from there. So I decided for version 1, I would focus on the learnings that were most easily captured and most impactful — learnings about human intent. These are directly captured in my conversations with the agent: "you shouldn't have done that, you should have done this instead..." and over time would help teach the agent what I am actually trying to achieve with this project, which should help it make smarter decisions in the future.

In-scope: capturing direct human feedback within a single coding session that showed a misalignment of intent.

Out of scope (for now):

• Capturing learning across Claude sessions
• Capturing learning where an agent spotted its own technical mistakes
• Fixing overfitting at scale.

That left me with something concrete. A self-learning loop has three phases: data capture, data processing, and integration. I focused on making each one work at the smallest possible scale before worrying about anything else.

Data capture means having the full loop - plan, implementation, validation - visible in one session. If the loop spans multiple sessions, stitching it together is hard. That's why I opted for a single session.

Data processing is where I had to resist the temptation to capture everything. There were plenty of expensive mistakes in my initial analysis, but one category stood out: cases where the agent diverged from what I actually wanted. This is where mistakes get most expensive, and the learnings that come out of it (what the app is actually for, how it should behave, closer to user requirements than implementation detail) seem to be the most valuable thing a self-learning loop can capture. Capturing that felt more valuable than anything else, so that's where I focused.

Integration is about how to feed learnings back into the development workflow in a way that would actually benefit future tasks. Not everything learned from one task is relevant to the next, so the distinction between task-level and project-level matters. Task-level learnings (what worked, what didn't, specifics of this piece of work) stay scoped to that task. Project-level learnings, the ones that speak to what the product is, how it should behave, what good looks like, get distilled into the main agent configuration so every future session starts with that context already in place.

What I actually built

Firstly, I realised my coding agent needs to ask a lot more questions during planning and implementation, rather than making assumptions. Secondly, I needed my coding agent to extract relevant learnings at the end of a session and export these to a file in the project. Finally, I needed a separate agent to review these learnings and extract a summary of generalisable learnings to the project level CLAUDE.md, to be injected into every coding session for this project. I won't go into the detail here, but these instructions have been incorporated into my relevant skill files.

It's not the autonomous self-improving agent I originally imagined. But the Shannon instinct turns out to be right: getting the small version working properly is harder than it looks, and it teaches you things that would have been invisible in the complex version. The more ambitious version is still on the list. But I'll get there one piece at a time.

I've been building a pipeline that generates medically accurate cases for a simulated consultation platform. Doctors use these cases to practise for the Simulated Consultation Assessment, a key exam to get qualified as a GP. During the exam, actors are brought in to play the role of patients and the doctors are assessed on how well they conduct the consultation. The platform simulates this in a cost and time effective way, allowing the doctors to have a voice conversation with an AI patient and their performance is assessed by another AI. The cases need to be clinically sound, pedagogically useful, and structurally compatible with the web application.

During the prototyping phase, work was spread across Claude chat sessions, Python scripts, Google Drive, and multiple team members. Some steps were automated, others were done by hand, others were done by an AI in a chat window and copy-pasted into JSON files. It worked for proving the concept, but formalising it into a repeatable data transformation pipeline requires a bit more finesse than a simple 'Claude, turn this into a pipeline'.

That's what this article is about. Not the pipeline itself, but the context engineering that had to happen before Claude could build anything useful.

Why you can't just throw everything at the model

The lazy approach is to dump all your documents, code, and data into a conversation and say "work it out." I've tried variations of this and it doesn't work! Sad times :(

Context isn't just information. It's knowing which parts of the research doc were aspirational versus which parts got implemented. It's knowing that a field was renamed in your head three weeks ago but not in the codebase. It's knowing that when your colleague generated some data, a field wasn't strictly controlled and can contain data that doesn't directly map to the field name. It's knowing that another field was being used incorrectly and that this is a known issue you haven't fixed yet.

None of that is in the code. None of it is in the documents. It's the accumulated understanding of the person who did the work. And if you don't find a way to transfer it, the model will fill in the gaps with its own well-meaning yet completely incorrect assumptions.

Step One: Give the Agent some Background

My first step was to write a single markdown file background.md that described the overall process, the key definitions, and crucially, all the quirks. What work had been done, by whom, using what tools. Where the data lived. What the known issues were. What the working assumptions were. The discipline of writing this document was valuable on its own. It forced me to articulate things I'd been carrying around implicitly.

The goal was to give Claude the honest picture, that this is a first draft built on first drafts, that there's uncertainty baked in at every stage, and that the naming isn't consistent. I also included all the links it would need: links to research documents, code repos, data files, commit hashes for specific decisions. Including the links in this document allows Claude to fetch these in the later stages as and when it needs to - a process called "just in time" context.

AI can only amplify the thinking that you have done. One bad line of research compounds into a bad plan, which compounds into a thousand bad lines of code. Getting the context right at this stage is crucial.

Step Two: Create a Target Model

With the goal being to create a data transformation pipeline, I first need a detailed plan of how this should be implemented. This meant creating an accurate target state data transformation model showing how every attribute flows end-to-end through a multi-stage transformation pipeline where the data incrementally gets transformed into the final desired case. From this model I would build my data transformation pipeline. If the task were simpler I'd usually skip to implementing the pipeline itself, but the data transformation model really adds value as it is much easier to reason about than a python pipeline and helps discuss key domain logic decisions with my team mates.

My first attempt was to try and model the manual workflow, every step that had actually been taken, including the scrappy ones. This proved to be the wrong approach. It was too complex, and it made more sense to just try and capture "what we want the pipeline to do." So I pivoted to modelling the target state instead, accepting that there could be some manual steps that are missed. Nothing is perfect, and I could always reconcile the drift later.

Planning the data transformation model alone took 5+ hours and the plan went through 7 updates, starting with some complete rewrites due to incorrect assumptions made by Claude and finishing with minor tweaks to get it just right. The plan grew from a relatively simple schema spec to a 700+ line document that captures every stage of the pipeline, the prompt section structures, and flags every gap or fuzzy relationship where the mapping isn't clean.

The whole purpose of this step was alignment. Not just "here's what I want you to build" but "here's how I think about this domain, here's what's certain and what's uncertain, here's where the gaps are." Anthropic describe context engineering as finding the smallest possible set of high-signal tokens that maximise the likelihood of the desired outcome. For my case, I'd describe it as building a shared mental model between me and the machine.

Step Three: Generate the deliverable

After the planning, the actual generation was the easy part. Claude implemented the plan in about 20 minutes to give me a fully functioning data transformation model. The data model had two layers: a machine-readable JSON bundle that serves as the single source of truth, and a self-contained HTML viewer you can open in the browser with colour-coded categories, interactive lineage tracing, hover tooltips, and all the other bells and whistles that make for a good read.

This is the part that feels like magic to people watching from the outside. But the magic isn't in the generation, it's in the 5+ hours of context work that preceded it. The model wasn't being creative. It was faithfully expanding on a plan that had been meticulously aligned to my intent. What I've taken away from this

What I've Taken Away from This

Context pollution is real: It's the data engineering principle of garbage in, garbage out, applied to AI. If you give the model dirty context, incorrect assumptions, stale names, missing relationships, you get dirty output. And unlike a database where you can query to find the bad records, bad context produces bad output that looks right. The model will confidently generate a data model with the wrong field lineage and it'll be internally consistent. You'll only catch it if you already know what right looks like.

Don't outsource the thinking: I've seen people describe AI as a replacement for understanding. In my experience, it's the opposite. You need to understand your domain more deeply than before, because you need to compress that understanding into a form the model can work with. The research, the planning, the context curation, that's the real work. The generation is the payoff.

The right altitude matters: Anthropic talks about finding the Goldilocks zone between overly rigid instructions and vague high-level guidance. I found this applied not just to prompts but to every piece of context. Sometimes you need a full schema definition. Sometimes you need a simple note that says "this field is fuzzy." Sometimes you need a worked example. The skill is knowing which altitude each piece of information needs to live at so the model can actually use it.

Planning is leverage of intent: The ratio of planning time to generation time in this project was roughly 15:1. That sounds extreme, but it meant the generation step required almost no iteration. Claude didn't go off in the wrong direction because there was no ambiguity about what the right direction was. Every hour spent on the plan saved multiples in rework.

Clear sessions force clean context: One practice I've adopted from reading other people's workflows is starting implementation in a clean session. It sounds minor, but it forces you to think about what context needs to be carried forward and what can be left behind. If you can't summarise it for a fresh session, it probably isn't clean enough. And if the context isn't clean, neither will the output be.

I'm still early in this pipeline build. The data transformation model is step one of a longer process. But the approach, invest heavily in context, align before generating, surface uncertainty explicitly, has already changed how I think about working with AI on complex tasks. The bottleneck isn't the model's capability. It's the quality of context you give it.

I recently started teaching my 12-year-old student to make games using AI (“vibe coding”). We’ve made a few projects together - a platform game and an endless runner/obstacle course.

At first, he was very excited, but that excitement quickly faded. I was miffed. This should be a really fun activity - building and playing his own games. We switched from math lessons to this… I mean, come on!

It feels like he found the learning of math more engaging, which is a surprise as he didn’t always look forward to our lessons. Given we’d switched to what is essentially play, he should be having a blast.

Freedom isn’t fun

In the spirit of play, I thought it’d be fun to let him take the reins and create whatever he wants. With boundless possibility, he chose to build a game he was familiar with.

This was fun initially, but replicating a game becomes tedious once you’ve gotten over the initial buzz of ‘look at what I made’. There’s no creative tension - it’s essentially copying, not inventing. And copying isn’t much fun.

Why did he choose to replicate a game? When possibilities are limitless, the brain naturally seeks safety. Boundless freedom feels exciting to us conceptually. But to the brain, it’s actually anxiety-provoking. There’s no structure to hold on to. When you tell a 12-year-old “make any game you want,” he suddenly faces infinite choices with no clear criteria for what’s good or achievable. To reduce that uncertainty, the mind automatically reached for something familiar. A pattern it already understood. I.e. A game he’s played before.

I made the mistake of equating fun with freedom, but psychologically, fun is about progress toward mastery within a framework. When there’s no friction, there’s nothing to master. Which explains why he was actually finding the math more engaging. A subtle irony for this to be missing in a lesson on building games, when this is a core principle of good game design.

What I did find however, is that he seemed to re-engage when he could hack or subvert the game once it had been created. He started skipping levels and only playing levels that scored higher points so he could earn more points than if he worked through them sequentially. He also made levels impossible for me by making the platforms spaced apart by 999999999999 pixels. Suddenly he had a constraint - the game architecture that we had replicated. Now the fun can begin.

Friction Fuels Engagement

Even without the creative element, I thought he’d be engaged through problem solving. Tweaking the prompt and refining the output. But the problem was, AI is pretty damn good at making games. You’ve got a working game with a single prompt.

Can you please make a about jumping on platforms

Note that he didn't even include the word 'game'.

Well, that’s not quite true. Creating an initial, playable version is easy. Making sure it works with all the bells and whistles is not. That last 20% of perfect is very difficult to solve for, too difficult for a 12 year old. So I settled on accepting good enough solutions, which means the only thing he had to do is prompt AI to implement a checklist without worrying too much about all of them working.

With AI instantly producing results, there’s no struggle. There’s no friction. The trial and error cycle is shallow. He’s not being pushed to think or strategise - just to describe changes. There’s little sense of mastery or discovery. He’s not facing interesting dilemmas or receiving emotionally satisfying feedback. So his brain isn’t getting the dopamine hit that comes from overcoming a challenge - it’s just issuing commands and getting results, like a passive God mode.

Conclusion

He’s not bored because he dislikes games or AI - he’s bored because the creative stakes are too low and the problems too shallow. Im not pushing him hard enough. To either create a novel game, or to create a game really well.

By making games with him, I thought my job had just became much easier and more fun, but as it turns out, I overlooked some crucial elements of engaging learning.

I often wonder: if Socrates had Anki, would he have bothered with the whole “know thyself” thing? Or would he have been too busy reviewing “Front: What is virtue? / Back: A form of knowledge.”

Perhaps both. Because, in the end, learning is less about cramming facts and more about weaving connections between them. Memory is not a filing cabinet; it’s a living web. And flashcards, surprisingly, can either help us build that web - or reduce knowledge to a box-ticking exercise.

Building my own flashcard generation workflow for personalized learning has been a crash course not just in coding, but in pedagogy, cognitive psychology, and the messy business of what it really means to know something. And out of the experiments, frustrations, and “why on earth can’t I remember that?” moments, three key lessons have emerged.

Lesson 1: Comparison is the Thief of Joy, but the Mother of Understanding

A simple card might ask: What was the French Revolution?A better card might ask: How did the American and French Revolutions differ?

The difference matters. Comparison forces the brain to connect dots, not just light them up individually. And it turns out this is exactly how memory strengthens: through relationships, analogies, contrasts. In cognitive science, this is sometimes called elaborative encoding - the act of tying new information to old information in as many ways as possible.

When flashcards evolve from isolated definitions to comparative questions, they stop being rote and start being relational. And relational knowledge is the kind that sticks.

Lesson 2: Depth Doesn’t Sacrifice Breadth - It Powers It

A recurring dilemma in learning: should I rush to cover all the ground (breadth) or pause to deepen my understanding of a single moment (depth)?

Take a simple fact: The American Revolution achieved independence. That’s fine. But ask why - why did the American Revolution stabilise into a functioning republic while the French Revolution spiralled into terror? - and suddenly one card of history opens a door into economics, social structures, leadership, and ideology.

That depth doesn’t slow you down; it cements the shallow fact so firmly that you can now sprint faster across other events. Once you grasp why the American path differed from the French, you can make sense of revolutions elsewhere with much less effort.

This is the paradox of learning: depth actually accelerates breadth. A well-connected idea becomes a mental landmark, making the rest of the territory easier to navigate.

Lesson 3: Not Every Detail Deserves Equal Worship

The final lesson was harder to swallow: not everything is worth remembering.

Some details - like the name of every faction in the French Revolution or every skirmish in the American - slip away, no matter how many times you drill them. At first, this feels like failure. But then the question becomes: do I need to recall every pamphlet, speech, and decree? Or is it enough to know the shape, the intent, and where to look when I need the detail?

Flashcards aren’t about becoming a human historian. They’re about training intuition, fluency, and recall of the concepts that matter most. Sometimes that means reframing a question so it points you to the turning point (say, Saratoga or the Reign of Terror). Sometimes it means letting go of the trivial, trusting that “open book” recall - or simply knowing where to check - is enough.

The real goal of learning, after all, is not flawless regurgitation. It’s flexible application. It’s the ability to ask better questions about history - even when you don’t yet know all the answers.

Closing Thought

In building this flashcard workflow, I didn’t just discover better ways to remember - I discovered better ways to forget. Forgetting the inessential frees up space for the connective tissue of knowledge: comparisons, reasons, meanings.

So perhaps Socrates didn’t need Anki after all. Because the real flashcard he lived by was a single, endless prompt: What is the difference?