The Unfinished Notebook

We presented a benchmark at AI Wales this week: 57 open-weight language models, 380 prompts each, power and memory telemetry captured per request with Marigold. The dataset and notebook are both available on github . Both were built to be simple rather than complete, to help develop a narrative and direction, rather than a goal.

What the notebook does

The notebook follows the four classic steps of a data science task: Load the data; Check some values and ranges; Perform Exploratory Data Analysis via distributions and a correlation matrix; and, build one basic model: coefficient of variation. Computing this per model and ranking the models by the consistency of power draw.

The final outcome produces a result which is also a baseline for improvement. The most consistent model in the set is a clear number anyone can check against the raw CSV. It is also, on inspection, incomplete answer due to zero values in the data. So now, we have identified our first task: data cleaning. Which can then be performed, new results gathered, inspected; a new improvement identified and so on. This is the iterative loop which we can access once the project is established.

This is a deliberate scaffold. It is too easy and tempting to continue the analysis and improvement, and delaying delivery of the work.

It is important to separate the setup of a project, and the goals of setup, with the delivery of the project. A key goal of setup is to encourage collaboration within a team. A notebook with obvious places for improvement encourages engagement.

A notebook as a narrative, not a pipeline

A notebook is a particular kind of artefact which creates a narrative through a problem. Data collection, parsing, cleaning, exploration, model selection, measurement and metrics and so on, can all be identified as steps in the notebook. In exploratory tasks, this sequence is important and clear. Missing a step is an obvious omission.

That narrative is not required in production where deployment requires integration, testing, and monitoring. A notebook is not a rough draft of a production system. It is the right form for presenting analysis so that a team, or a room of strangers at a meetup, can understand the process, ask questions, engage, and pick the thread up themselves.

What people actually asked

Two questions came up in the room: Had we measured response accuracy? and, had we accounted for batch size and parallel decoding in the timing figures?

Accuracy evaluation and batching are the two things anyone with production LLM experience ask. Both questions can be seen as a way of demonstrating knowledge and suggesting next steps, and “feeling out” what space is left in the project. The useful answer was the same: no, but feel free to contribute!

This turned the demonstration into an invitation. People could leave the room comfortable that they could contribute to the project.

Engagement as a property of the work

The usual approach to team culture treats engagement as a separate activity: an away day, a team lunch, a Friday demo. These are fine on their own terms, but sit outside the actual work, which means the habits they build – showing up, being sociable, sitting through a presentation – do not transfer back into work getting done.

Build the invitation into the work. Design for it to be continued and make it accessible enough that collaboration is the next natural step.

(If you’re building a data team’s engagement into its actual output rather than around it, get in touch.)