The field is the lab

Two projects I am working on have almost nothing in common methodologically, and that has been sitting with me.

One is with a medical consortium. Randomised allocation, a control group, fixed outcomes, a defined measurement window. The tool stays still while the research happens around it. That is the design. It is rigorous and it is right for what they want to know.

The other is in education. Schools, teachers, school leaders. The tool does not stay still. It is used, observed, adjusted, and used again. The prompts that practitioners type into the system are not just interactions. They are data about what people actually need, what they cannot yet articulate, where professional knowledge runs out. The research is partly about what use reveals.

I have started calling this the difference between static and reveal. Not as a technical term. Just as a way of holding the distinction in my head.

What reveal actually means

In a static design, the tool is an intervention. You apply it, you measure the effect, you report. The tool needs to be frozen for the logic to hold. Any change during the study is a confound.

In a reveal design, the tool is more like a field instrument. You send it into a real environment and watch what comes back. More use is not a complication. More use is more signal. The value accumulates precisely because the tool is live and the context is real.

That accumulation is worth naming. Every prompt a teacher types tells you something you could not have learned from an interview or a survey. Not because the prompt is more honest, but because it is situated. It comes from a real moment of need, not a retrospective account of one. Over time, those prompts build a picture of what the practice actually looks like from the inside.

This is what happens when AI tools warm up slowly in real organisations: the signal only becomes visible through use. You cannot pre-specify it.

Not just methodology

I used to think this was a question about research design. Increasingly I think it is a question about what kind of thing a product is.

A product that lives in a static paradigm can be finished. You build it, test it, ship it. The relationship between product and knowledge is sequential: first you learn, then you build, then you deploy.

A product in a reveal paradigm is never finished in that sense. It is viable before it is complete. Deployment is not the end of learning, it is the condition for it. The field is the lab.

This also reframes something I notice when I use tools like Claude. The thumbs up, the thumbs down, the occasional evaluation prompt. Those are not just quality checks. They are a mechanism for pulling the field back in. Every interaction where someone signals that something did or did not work is data that would not exist without deployment. AI literacy in organisations accumulates the same way: not through training programmes, but through use that generates friction, and friction that generates understanding.

What I am still figuring out

I do not think static and reveal are two types of project you choose between. I think most serious implementations need both, at different moments. You reveal first, to understand what the right questions are. Then you fix and measure, to know whether you answered them.

What I keep seeing is projects that skip the reveal phase because it feels unscientific, and then design an effect study around outcomes that do not quite match what the tool actually does in practice. The measurement is rigorous. The thing being measured is slightly wrong.

That is the pattern I am trying to get better at noticing. Not as a critic. As someone who builds the tool that ends up in the middle of that design.