A transcript of an agent that completed a task and one that completed it correctly can look the same, in the way a fluent RAG answer can be ungrounded (Evaluating RAG). An agent’s output is a sequence of actions, and each can be right or wrong independently of the final answer. Evaluation therefore follows the loop: each stage measured on its own, in the order the agent runs them, on tasks taken from your own work, and repeated.
We can set out the method in five parts: the task set, the trace, three measurements, repetition, and when to run it again.
Public benchmarks measure general capability on someone else’s tools. The evaluation that predicts behaviour in production uses your tasks, your tools and your failure cases. A few dozen tasks will find the large faults, and the set grows as production supplies new cases.
Each task needs three things written down before the agent runs it: the request as a user would phrase it, the end state that counts as success, and any rules the agent must respect on the way. The end state carries the most weight. tau-bench, a benchmark for customer-service agents, judges a run by comparing the database after the conversation with an annotated goal state, which tests what the agent did and ignores how well it described it. The same check works on a private set: the order is cancelled, the ticket carries the right label, the patch fixes the failing test and the existing tests still pass.
Include tasks the agent should refuse or escalate (a request that breaks a stated policy, a change it has no permission to make) and tasks it has got wrong before. These test the governance problems in Agent Risks, Costs, and Governance, which a set of ordinary requests never reaches.
Every measurement reads from a record of the run: each tool call with its arguments and result, each model call with its token count, the order they happened in, and the final state. Instrumentation Is the Deliverable argues that this record is worth building for its own sake. A durable execution layer produces most of it as a side effect. DBOS, for example, stores each workflow’s inputs and every step’s output as rows in Postgres (What Is DBOS?), so the evaluation queries the same tables the recovery mechanism uses. (If you are building this record for an agent already in production, get in touch.)
Tool-call correctness. For each call, check that the agent chose the right tool and passed valid arguments. Most of this is mechanical: arguments validate against the tool’s schema, identifiers refer to records that exist, and values fall within documented ranges. Where a task has one correct sequence of calls, compare the trace with it; where several sequences are acceptable, check each call against rules instead. The Berkeley Function-Calling Leaderboard measures this stage across models, which helps choose a model before building and says nothing about your own tools.
Planning efficiency. For completed tasks, count steps and cost, and compare each with a budget set per task. A task that should take three steps and takes thirty shows up here as a number before it shows up on an invoice. Read these beside the success rate, since a higher success rate bought with ten times the steps is a different result.
Outcome. Compare the final state with the expected end state, and check the rules separately: an agent that completes a refund by skipping a required approval has failed the task. Use code to check anything code can check. Use a model as judge only for outputs it cannot (the quality of a written summary, for instance), and test the judge against a sample of human labels before relying on it (How to Evaluate an AI Pipeline).
The order serves diagnosis. A fall in outcome while tool calls stay correct points at planning or at the model’s final step. A fall in tool-call correctness most often means something changed in a tool (its schema, its error codes, its data), and the later measurements inherit the fault.
An agent given the same task twice can take different paths to different results, so one run per task overstates reliability. Run each task several times and report two numbers: the share of individual runs that succeed, and the share of tasks that succeed on every run. tau-bench introduced the second as pass^k, for k runs. The gap between the two numbers measures inconsistency, and a production system lives with that gap, because every real request is another draw. (If runs were independent, a 90 per cent success rate per run would give 0.9 to the power 4, about 66 per cent, for four successes in a row.)
Repetition also separates change from noise. A difference between two model versions smaller than the spread between runs of one version is not yet a finding.
Run the full set whenever anything in the loop changes: the model, a tool’s API, the harness or framework version, the system prompt, or the set of tools exposed to the agent. Tool and harness changes cause drops in reliability without any change to the model, and they arrive more often than model releases. Between changes, a scheduled run on a fixed sample catches external tools that changed without notice. The task set, the trace and the three measurements stay the same each time, which is what makes one run comparable with the last.
]]>Workflow tools are usually compared feature by feature: language support, user interface, pricing, the length of the integrations page. Those lists grow with every release and say little about which tool suits a given job. We can sort the tools more usefully with two questions. When is the shape of the work fixed? And where, and how, is the record of progress kept? The answers decide the rules your code must follow, how a failed run recovers, how you repair one, and which problems the tool fits.
We apply both questions to eight tools: Apache Airflow, Dagster, Prefect, AWS Step Functions, Temporal, DBOS, LangGraph and runfox. (runfox is our own library. It is included because it sits at an unusual point on both axes). From the answers we build a procedure for choosing between them.
We can trace the two terms to different communities, and they overlap heavily. Workflow orchestration grew up around data pipelines: a scheduler runs a graph of tasks on a timetable, retries the ones that fail, and shows the history in a user interface. Durable execution grew up around application code: a function records the result of each step as it runs, so a crash or a deployment interrupts it without losing its place. (Our article on DBOS covers that mechanism). Every tool here does some of both, and the two questions locate the real differences more precisely than either label.
Some tools need the whole graph before anything runs. Airflow parses a DAG file into tasks and dependencies; Dagster assembles a job from its ops or assets; Step Functions takes a state machine written in the Amazon States Language; runfox takes a YAML document. Each allows some runtime variation inside the declared frame. Airflow’s dynamic task mapping creates a variable number of task instances from an upstream task’s output, and Dagster’s dynamic outputs do the same. Step Functions has Choice and Map states. runfox evaluates branch conditions against step outputs and expresses loops by resetting a step. None of them lets the running work add a kind of step the definition did not declare, and an Airflow DAG cannot contain a cycle at all.
Other tools discover the shape by running ordinary code. A Temporal workflow, a DBOS workflow and a Prefect flow are functions, and whatever steps they call, in whatever order and however many times, is the workflow. Inngest, Restate and Hatchet’s durable tasks follow the same model. Hatchet offers both models and gives a plain rule for choosing: if the structure is not known before the workflow starts, or it contains loops, a DAG cannot express it.
LangGraph sits between the two. A graph’s nodes and edges are declared before it runs, but conditional edges choose the next node at runtime and cycles are allowed, which is what an agent loop needs. Its functional API drops the declared graph in favour of ordinary functions.
A declared shape is information held before execution. You can review a definition without running it, list the models or services it will call, estimate its cost, and store and version it as data. A discovered shape is expressiveness: anything the language can express, including loops whose length depends on what a model says. Agents fall almost entirely on the discovered side. This is the distinction between static and dynamic planning, seen from the infrastructure end.
The second question has three parts: who owns the store, what a record contains, and how much the tool asks of the store.
A service owns the record. Temporal’s history service writes an ordered event history for every workflow to its own database (Cassandra, MySQL, PostgreSQL or SQLite), and workers run code on its instruction. Step Functions keeps execution history inside AWS for Standard workflows; Express workflows keep none. Airflow’s scheduler records task state in its metadata database. Dagster records runs in its run storage, which can be Postgres, and hands step outputs between steps through IO managers. Prefect reports run state to its API and keeps task results only when result persistence is switched on, which by default it is not. In each case there is a service to operate or rent, and the record is reached through it.
A library writes the record into your database. DBOS writes workflow inputs, every step output and the final result into tables in a Postgres or SQLite database you own, from inside your own process. LangGraph’s checkpointers save snapshots of the graph’s state to Postgres, SQLite or Redis, at a frequency set by its durability mode: on exit only, asynchronously while the next step runs, or synchronously before it starts. runfox keeps one record per run through a Store, with in-memory, SQLite and DynamoDB implementations (the DynamoDB store is built on our dynawrap library, which offers the same interface over Postgres).
Within the second group, the richness of the persistence interface varies most. DBOS asks a great deal of its database: SQL, transactions, and a schema it migrates itself. That buys exactly-once database writes (a step’s outcome is recorded in the same transaction as the application’s write) and a run history you can query and join directly, and it limits the database to Postgres or SQLite. runfox asks almost nothing. Its Store has two operations, load a run’s record and write it back, so anything that can store a document by key can back it, and the same workflow definition runs in-process, against SQLite, or across DynamoDB and SQS without changes. Features that need more from the store (queries across runs, transactions spanning a step and the application’s own data) are absent from the core. The design leaves room for them all the same. An application that commits to one store can build them over the same record format with that store’s native queries. Capability grows by constraining the choice of store for one deployment, and the core stays portable for every other.
Recovery is where the two answers combine.
Tools that discover the shape recover by replay. Temporal re-executes the workflow function against its event history, returning recorded results for completed steps and raising a non-determinism error if the code takes a different path. DBOS does the same against its tables, and Inngest re-invokes the function with the memoized results of completed steps. Replay is the reason these tools require deterministic workflow code: the record only means something if re-running the code makes the same calls in the same order.
Tools that declare the shape recover by resuming from recorded state. There is no code to replay, so there is no determinism rule. Step Functions can redrive a failed Standard execution from the failed state within fourteen days, keeping the results of the steps that succeeded. Dagster re-executes a run from failure, skipping completed steps and loading their outputs through the IO manager. Airflow retries failed tasks and lets an operator clear a task to run it again. runfox rehydrates the run’s record and advances from it, so a restarted process carries on from the last write.
Two tools fall between. Prefect re-runs a flow and skips the tasks whose results it can load from its cache, which works only with persistence switched on. LangGraph resumes from the start of the node that was interrupted, so the node’s work is repeated unless its side effects sit inside tasks, and its documentation asks for deterministic, idempotent workflows for that reason.
Granularity matters as much as mechanism. Replay and per-step state protect each step. A per-node checkpoint protects each node, and a node that makes three model calls and fails on the third repeats all three.
Every tool also needs something to notice that a worker has died. The services handle this centrally. DBOS recovers a process’s own workflows when the process restarts, and across several machines it relies on its Conductor service or on recovery calls you make yourself. runfox, as of version 0.0.7, leaves this to the runner: a worker that dies without writing a result leaves its run waiting, so executors are expected to catch their own errors and return them as data.
When the fault is a bug in the workflow itself, the two answers decide the repair.
In a code-defined workflow the bug can sit anywhere in the code, including the glue between steps. The fix is a deployment followed by a re-run from the faulty step on the new version (Temporal’s reset, DBOS’s fork and rewind). Recorded steps before that point are kept, so expensive work is not repeated, but every in-flight run that reaches the bug fails until the deployment lands.
In a declared workflow, a mistake in wiring or branching is a mistake in data, fixed by editing the definition without redeploying the executor. The executor is still code, and a bug there needs a deployment like any other. Declared definitions can also fail more quietly. In expression languages such as JSON Logic, a reference to a missing field yields null, and a mistyped path can carry that null through to a wrong result with nothing raised along the way.
| Tool | When the shape is fixed | Where progress is recorded | Recovery |
|---|---|---|---|
| Airflow | Parse time; runtime fan-out via task mapping; no cycles | Scheduler’s metadata database | Task retries; clear and re-run |
| Dagster | Definition time; runtime fan-out via dynamic outputs | Run storage; outputs via IO managers | Re-execute from failure |
| Prefect | Runtime, as Python | Prefect API; task results only if persistence is on | Re-run, skipping cached tasks |
| Step Functions | State machine definition | AWS-managed history (Standard workflows) | Redrive from the failed state |
| Temporal | Runtime, as code | Event history in Temporal’s database | Replay |
| DBOS | Runtime, as code | Tables in your Postgres or SQLite | Replay |
| LangGraph | Declared nodes; runtime routing and cycles | Checkpoints in your Postgres, SQLite or Redis | Resume at the start of the interrupted node |
| runfox | YAML definition; branches and loops | One record per run via a Store (memory, SQLite, DynamoDB) | Rehydrate the record and advance |
We can build a selection procedure from the two questions.
If the shape of the work is known before it starts and the work runs on a schedule over data, the declared orchestrators fit: Airflow for task pipelines with its large operator ecosystem, Dagster where the unit of thought is a data asset. Step Functions fits the same declared shape when the steps are AWS services and someone else should run the orchestrator. runfox fits it when the definition should be a portable document, executed by a library inside your own process against whichever store the deployment already has.
If the shape is decided while the work runs (agents, business processes that wait on people, anything with loops of unknown length), the replay family fits, and the second question chooses within it. DBOS suits teams that already run Postgres and want no further service. Temporal suits teams willing to operate or rent a service in return for multi-language workers and a long production record. LangGraph’s checkpointer suits agents already written in LangGraph, provided side effects are wrapped in tasks so that node-level resumption does not repeat them. Prefect sits across the line: flows are ordinary code with a service alongside, and its durability is a result cache that has to be switched on.
The tools differ in many further ways, and those differences matter at the margins. The important ones (determinism rules, recovery granularity, the shape of a repair, what must be operated) follow from where a tool stands on the two questions.
If you are making this choice for an AI system, we can help.
]]>DBOS is an open-source library for durable workflows. You annotate ordinary functions in a Python application (TypeScript, Go and Java are also supported), and DBOS records their progress in a database as they run. If the process dies, the work resumes from the last completed step when the process starts again. DBOS needs a Postgres database and nothing else.
That makes it unusually easy to describe. The state of a running program is a set of ordinary rows in Postgres, and recovery, queues, schedules and human-in-the-loop waits are all reads and writes against those rows. What follows covers what a durable workflow is, what DBOS is, how to use it in Python, how an AI agent fits the model, and what it costs.
A durable workflow is a function whose progress is recorded as it runs, so that a crash, a restart or a deployment interrupts it without losing its place. The function is divided into steps. Each completed step’s result is saved, and after a failure the workflow resumes from the first step with no saved result. Finished work is never repeated; interrupted work is retried.
The problem it addresses is ordinary. Any job that spans several calls to other systems (take a payment, reserve stock, send an email; or ask a model, call a tool, ask the model again) can fail between two of them and leave the job half done. Without a workflow runtime, the usual remedies are hand-written status columns, retry queues and clean-up scripts, rebuilt for each job. A durable workflow moves that bookkeeping into the runtime. Two further properties follow from the record. A workflow can wait for hours or days (on a timer, or for a human decision) without holding a process open, and its history is an audit trail of what ran and what each step returned.
Implementations differ mainly in where the record lives and what drives execution. Temporal and AWS Step Functions run a separate service that holds the history and dispatches work to your code. Azure Durable Functions and DBOS keep the history in storage the application writes to directly (Azure Storage by default in one case, Postgres in the other) and replay the function against it.
DBOS has two primitives.
A workflow is a function decorated with @DBOS.workflow().
A step is a function decorated with @DBOS.step() and called from inside a workflow.
When a workflow starts, DBOS writes its inputs to the database; each time a step completes, DBOS writes the step’s output; when the workflow returns, DBOS writes the result.
from dbos import DBOS
@DBOS.step()
def charge_card(order_id: str) -> str:
... # call the payment provider, return a charge ID
@DBOS.step()
def send_receipt(order_id: str, charge_id: str) -> None:
... # send an email
@DBOS.workflow()
def checkout(order_id: str) -> str:
charge_id = charge_card(order_id)
send_receipt(order_id, charge_id)
return charge_id
Those writes land in a schema called dbos in what DBOS calls the system database.
Three tables carry most of the weight.
workflow_status holds one row per execution: its ID, the function name, a status (PENDING, ENQUEUED, SUCCESS, ERROR and a few others), the executor running it, its queue and its timings.
workflow_input holds the serialised arguments.
operation_outputs holds one row per completed step, keyed by the workflow ID and a function_id that numbers steps in the order they ran (from 1 in Python), with the step’s serialised output or error.
The full schema is in the system database reference.
A checkout that dies after charging the card leaves a PENDING row in workflow_status, its argument in workflow_input, and one row in operation_outputs for charge_card.
Nothing records a receipt.
That is everything DBOS needs to finish the job.
Recovery reads those rows.
On startup DBOS finds workflows still marked PENDING and calls each one again with its stored inputs.
As the function re-executes, each step call first looks for its row in operation_outputs; if the row exists, the step returns the stored output without running.
The first step with no row is where the original run stopped, and execution carries on normally from there.
So charge_card returns the recorded charge ID, the customer is not charged twice, and send_receipt runs for the first time.
The other features use the same tables.
A queued workflow is a workflow_status row with status ENQUEUED, which worker processes poll for under configurable concurrency and rate limits.
A message sent to a running workflow is a row in dbos.notifications, and a value a workflow publishes is a row in dbos.workflow_events.
Schedules have a table of their own.
Since it is all SQL, a workflow’s history is one query away:
SELECT function_id,
function_name,
error IS NOT NULL AS failed,
completed_at_epoch_ms - started_at_epoch_ms AS duration_ms
FROM dbos.operation_outputs
WHERE workflow_uuid = 'checkout-order-1042'
ORDER BY function_id;
(The output column holds Python values pickled by default, so it is readable from Python rather than from SQL. A portable JSON format exists for applications that mix languages.)
Install it with pip install dbos, then configure and launch it when the application starts:
import os
from dbos import DBOS, DBOSConfig
config: DBOSConfig = {
"name": "orders",
"system_database_url": os.environ.get("DBOS_SYSTEM_DATABASE_URL"),
}
DBOS(config=config)
if __name__ == "__main__":
DBOS.launch()
checkout("order-1042")
With no database URL DBOS uses a local SQLite file, which is enough for development.
Anything with more than one process needs Postgres, and the URL is an ordinary postgresql:// connection string.
Replay puts two obligations on your code.
The first is that a workflow function must be deterministic: given the same inputs and the same step outputs, it must call the same steps in the same order.
Anything that could differ between runs (the time, a random number, a network call, a database read) belongs in a step, where its result is recorded once and returned from the record on every replay.
Branching on a step’s output is safe for the same reason.
DBOS raises DBOSStepNondeterminismError when it detects a replay that diverges.
The second is that steps should be safe to repeat.
A step that completes is never run again, but a step interrupted part-way through will be retried, so DBOS guarantees each step runs at least once.
Steps that call unreliable services can retry automatically with @DBOS.step(retries_allowed=True, max_attempts=5).
Replay also means the workflow function itself runs from the top on recovery. Code between steps (logging, building a prompt, assembling a list) executes again, which is harmless as long as it only depends on inputs and step outputs.
Every execution has a workflow ID, which is the key for all of its rows.
By default DBOS generates a random UUID per execution (the ID is not derived from the inputs, so two calls with the same arguments are two executions).
You can supply your own with SetWorkflowID, or with the workflow_id argument when enqueueing, as long as it is unique within the application.
A supplied ID doubles as an idempotency key: run a workflow under an ID that already exists and DBOS returns the existing execution’s result instead of starting a second one.
The useful IDs are built from something the job already has, such as an order number or an inbound request ID, so a retried request or a duplicated message lands on the same workflow:
from dbos import SetWorkflowID
with SetWorkflowID("checkout-order-1042"):
checkout("order-1042")
(Pass workflow_id_reuse_policy="reject" to SetWorkflowID to raise an error on a duplicate instead.)
To re-run a recorded step, you re-execute the workflow from that step onward: the steps before it return their recorded outputs, and that step and everything after it run afresh.
Re-running a single step in the middle while keeping the later results would hand those later steps inputs they never saw, so the rule follows from the replay model.
There are two ways to do it.
DBOS.fork_workflow(workflow_id, start_step) copies the inputs and the earlier steps into a new workflow with a new ID and runs it from start_step.
DBOS.rewind_workflow(workflow_id, start_step) deletes the original workflow’s records from start_step onward and re-runs it under the same ID, which keeps messages, events and child workflows attached (it only works on a workflow that has finished, failed or been cancelled).
Step numbers come from DBOS.list_workflow_steps(workflow_id) or dbos workflow steps on the command line.
Both can target a newer application version, so a workflow that failed on a bug can be re-run from the faulty step once the fix is deployed.
For an agent, the same mechanism re-runs a conversation from a given model call after a prompt change, without repeating the tool calls that came before it.
For work that should run in the background, DBOS.start_workflow returns a handle you can poll or wait on, and a Queue adds flow control:
from dbos import Queue
refunds = Queue("refunds", worker_concurrency=2)
handle = refunds.enqueue(checkout, "order-1042")
result = handle.get_result()
DBOS.sleep pauses a workflow durably: the wake-up time is stored, so a workflow told to sleep for a week still wakes on schedule through any number of restarts.
An agent maps onto DBOS directly.
The agent loop is a workflow; each model call and each tool call is a step.
The loop branches on whatever the model said, which looks like the kind of non-determinism the rules forbid.
It is safe, because on replay the model’s reply comes back from operation_outputs and the model is not asked again.
A crashed agent therefore resumes without paying twice for the calls it had already made.
Here is a refund agent with two tools, one of which waits for a human to approve it:
import json
from dbos import DBOS
@DBOS.step(retries_allowed=True, max_attempts=3)
def think(history: list[dict]) -> dict:
reply = llm.chat.completions.create(model=MODEL, messages=history)
return parse_action(reply.choices[0].message.content) # raises on malformed output
@DBOS.step()
def lookup_order(order_id: str) -> dict:
...
@DBOS.step()
def issue_refund(order_id: str, amount: float) -> dict:
... # keyed on order_id, so a retry overwrites the same record
@DBOS.workflow()
def refund_agent(request: str) -> str:
history = [SYSTEM_PROMPT, {"role": "user", "content": request}]
for _ in range(10):
action = think(history)
if action["action"] == "final_answer":
return action["args"]["text"]
if action["action"] == "issue_refund":
DBOS.set_event("status", {"awaiting_approval": action["args"]})
decision = DBOS.recv("approval", timeout_seconds=24 * 3600)
if decision == "approve":
result = issue_refund(**action["args"])
else:
result = {"error": "refund rejected by reviewer"}
else:
result = lookup_order(**action["args"])
history.append({"role": "assistant", "content": json.dumps(action)})
history.append({"role": "user", "content": json.dumps(result)})
return "step limit reached"
Parsing and validating the model’s reply inside think matters.
A malformed reply raises, DBOS retries the step, and only a valid action is ever recorded.
A bad reply that reached the workflow would be replayed faithfully on every recovery.
The approval is where durability earns its keep.
DBOS.set_event publishes the agent’s state for a dashboard or an inbox to read.
DBOS.recv blocks until a message arrives on the approval topic or the deadline passes, in which case it returns None and the refund is refused.
The deadline and the eventual message are both stored in Postgres, so the process can be restarted or redeployed while a reviewer takes a day to decide, and the workflow carries on waiting.
The reviewer’s decision arrives from any other process with DBOS.send(workflow_id, "approve", topic="approval"), or through DBOSClient from a service that is not itself a DBOS application.
It becomes a row in dbos.notifications.
The same pattern is packaged for several agent frameworks: DBOS has integrations for Pydantic AI, the OpenAI Agents SDK, LlamaIndex and Google ADK in Python, and the Vercel AI SDK in TypeScript, each of which turns model requests and tool calls into steps.
(If durable agents on your own infrastructure are a live problem for you, get in touch.)
Writes. DBOS adds one database write per step and two per workflow. DBOS’s own benchmarks put a single Postgres instance at over 40,000 workflows or steps per second, so for most applications the write count is a non-issue. Write size is the variable to watch, because it follows the size of what your steps return. A step that returns a generated image stores the image; the documented advice is to put large artefacts in object storage and return a reference.
Code changes. Recovery replays new code against old records. A deployment that changes which steps a workflow calls, or their order, can strand workflows that were in flight under the old code. DBOS handles this with patch markers in the code or with application versioning, so that old workflows only recover on processes running compatible code. Either way, deploying long-running workflows needs some forethought.
SQLAlchemy.
DBOS is Postgres-native but not driver-neutral.
Its own tables are reached through SQLAlchemy, and it migrates them itself on launch (or ahead of time with dbos migrate, for database roles without DDL rights).
The same holds for application data, with larger consequences: exactly-once writes are only available through DBOS datasources, which hand you a SQLAlchemy session and add a tracking table to your database.
Any other client, including a plain psycopg2 connection, runs as an ordinary step.
It is retried on failure, so its idempotency is your problem.
Recovery across machines. A single process recovers its own workflows when it starts. With several workers, something has to notice that one has died and hand its workflows to another. DBOS provides this through Conductor, its control plane, or through recovery calls you make yourself. Conductor stays off the execution path (applications keep running if it is unreachable), but it is a separate service to account for.
What DBOS asks, then, is a small set of rules: deterministic workflow functions, steps that tolerate a retry, and some care when changing workflow code. In return, the state of every running program lives in a database you probably already operate, and you can inspect it, join it and repair it with SQL.
]]>Machine learning work is data work, so the legal position on storing, moving and processing that data belongs in the engineering rather than in a review at the end. This piece covers one narrow category: request data of the kind that already sits in server logs and analytics events. It is not legal advice, and other categories raise different questions. (If your situation involves those, get in touch or ask your legal representative.)
Two regimes apply to that data and they test different things. Most discussion collapses them into one, which is why so many teams reach the wrong answer confidently. One regime asks whether you put anything on the user’s device. The other asks whether what you hold can be traced to a person. A system can pass the second and fail the first, and failing the first is the common case.
In the UK, Regulation 6 of the Privacy and Electronic Communications Regulations governs storing information on, or reading information from, a user’s terminal equipment. The EU equivalent is Article 5(3) of the ePrivacy Directive. Consent is required unless the storage is strictly necessary to provide a service the user has explicitly requested.
Three properties of that rule are worth stating plainly, because each of them catches somebody.
It is technology-neutral. The word “cookie” appears in the popular name for the rule and not in the mechanism it describes. The ICO applies it to local storage, session storage, and device fingerprinting on the same basis. Replacing a cookie with localStorage changes nothing.
It does not ask what the stored value means. A random identifier containing no personal data at all still engages the rule, because the regulated act is the storage rather than the content. This is where the intuition fails: teams reason that their identifier is meaningless, conclude they hold no personal data, and never notice they are being tested on a different question.
And analytics does not qualify as strictly necessary. The ICO’s position is that the exemption covers what the user asked for, and nobody visits a website in order to be measured. First-party analytics is still analytics.
So a system that writes anything to the device needs consent, and the GDPR analysis never gets started. The only way past is to store nothing there.
That is achievable, and it constrains the design in one specific way. A session identifier generated fresh on each page load, transmitted with the event and never written back, touches nothing. What you give up is continuity: the server sees a sequence of page loads rather than a visit, and reconstructing a journey across pages requires either device storage or server-side stitching from other signals. That is a real cost and it is the cost of the exemption.
Assume the device question is settled. The remaining question is whether the data you hold is personal data, and the answer to that changed on 4 September 2025.
In EDPS v SRB (C-413/23 P), the Court of Justice ruled that pseudonymised data is not automatically personal data in every case and for every party. What matters is whether the party in question can realistically re-identify the individual. It was the first judgment to state explicitly that the same data can be personal data for the original controller, which holds the key, and not for a recipient who cannot reverse the pseudonymisation and has no other route to identification. The Court confirmed the interpretation applies under the GDPR and not only under the regulation governing EU institutions.
The practical effect is that identifiability is a property of a relationship rather than of a field. The same string can be personal data in one organisation’s hands and anonymous in another’s.
The European Data Protection Board has moved to match. Its draft Guidelines 02/2026 on Anonymisation, published on 7 July 2026, adopt the relative approach the Court endorsed. That is a reversal: the EDPB’s January 2025 draft on pseudonymisation had stated that pseudonymised data counts as personal data even where the recipient lacks the additional information needed to re-identify anyone.
Which is the reason to treat this as current rather than settled. Two drafts in eighteen months took opposite positions, the anonymisation guidelines remain in draft, and the judgment has been referred back to the General Court on the remaining procedural points. The substantive findings are authoritative. The guidance built on them is not finished.
The clearest application is an identifier that is meaningful only to the party that issued it.
A Google click identifier arrives in the landing page URL when someone clicks an advert. It resolves to a specific click, and through that to a user, in Google’s systems and nowhere else. An intermediary that receives it, stores it alongside a page path and a timestamp, and later sends it back to Google with a conversion, has at no point held anything it could resolve to a person. Google learns that a click it already knew about produced a conversion. It learns nothing new about who.
Under the relative test that intermediary is not processing personal data, while Google plainly is. The judgment is what makes this statable rather than merely arguable.
Two qualifications, both of which matter more than the headline.
The test looks at the recipient’s total means rather than at the field in isolation. An identifier alongside a user agent, a timestamp, a request sequence and an IP address is a more distinctive combination than any of them alone, and a regulator assesses the combination. Storing the autonomous system number rather than the address removes the field with the clearest route to a subscriber, and the honest claim is about what you have no lawful means to do rather than about what you happen not to hold.
And the obligation to inform does not move. The Court was explicit that the duty to tell people what happens to their data is assessed prospectively, at the point of collection, from the controller’s perspective, and cannot rest on whether a downstream recipient might identify anyone later. The site operator remains the controller. Their privacy notice still has to say that a click identifier is collected and forwarded. A vendor being outside scope is not the customer being outside scope, and any vendor implying otherwise is selling you a problem.
There is also a contractual layer underneath the statutory one. Google’s own EU user consent policy requires advertisers to obtain consent for ad measurement from users in the EEA. Satisfying the GDPR does not satisfy that, and it is enforced by contract rather than by a regulator.
The rules that fall out are all design decisions rather than compliance decisions, and each is cheaper at build time than afterwards.
Write nothing to the device. Generate the session identifier per request and let it live in the payload. Hold no field you could resolve to a person by any lawful route, which in practice means the autonomous system number rather than the address, and no free-text field that a user could type a name into. Make forwarding opt-in per destination and per field, so the decision to send an identifier somewhere is an explicit act with a record. Carry the consent state as a field, so a customer who does need consent can pass their signal through rather than working around you.
The last one is the point most easily missed. A system that cannot represent consent forces every customer into the same posture, and the customer who wants advert measurement and the customer who wants no third-party processing are asking for incompatible things from the same product.
None of this is difficult. It is difficult to retrofit, because the identifier scheme and the storage model are decided early and everything downstream assumes them. The regimes above are best read as constraints on the first week of a project rather than as a review in the last.
Clientlog is built to these constraints: no device storage, no address retention, forwarding opt-in per destination. The implementation notes are on its own site.
(If you are working out where your analytics or model pipeline sits against this, get in touch.)
]]>Benchmark work through 2025 and 2026 keeps returning the same awkward result: GraphRAG underperforms ordinary vector RAG on a large share of real tasks, while clearly beating it on a narrow set. That pattern is more useful than either the enthusiasm or the dismissal, because it turns the question from whether graphs help retrieval into which questions they help with, and whether your corpus contains enough of them to justify the cost.
This piece covers what the graph adds, how it gets built, the two query shapes where it pays, what it costs, and how to decide without building one first.
A graph is a set of nodes (things) connected by edges (the stated relations between them). The same structure sits behind a family tree or an org chart; here it is built from text rather than drawn by hand.
Standard RAG treats a corpus as independent chunks. Embed each chunk, embed the query, return the nearest few. The retrieval unit is a passage, and the relationships between passages are invisible to the system.
GraphRAG, in the form Microsoft Research published in 2024, runs an extraction pass over the corpus, pulling out entities and the relations between them, and assembles the result into a knowledge graph. It clusters the graph into communities and generates a summary of each one ahead of any query arriving. At query time it retrieves graph fragments (nodes, triples, paths, subgraphs) or community summaries rather than raw passages, and generates from those.
Two things change. The retrieval unit becomes structured, so the model receives “A supplies B, B acquired C in 2019” instead of three paragraphs that each mention one of those facts. And the system holds a representation of the whole corpus, in the community summaries, which plain RAG has no equivalent of.
Microsoft’s original pipeline extracts entities and relations with an LLM prompt, and most write-ups treat that as the method. It is one option among several, and the choice affects cost and coverage more than it affects the resulting graph’s shape.
Classical named-entity recognition paired with a relation-extraction model (or an OpenIE-style triple extractor) produces the same nodes and edges at a fraction of the token cost, without an LLM call per chunk. The trade-off is the one that affects any NER pipeline: a model trained on general text misses the relation types specific to a narrow domain, and a contracts corpus or a clinical record set will need extraction tuned to it rather than the off-the-shelf kind.
Entailment models offer a different edge type again. Instead of “supplies” or “acquired”, the edge becomes “confirms” or “contradicts” between two claims, which suits a corpus built from assertions that agree or disagree with each other (audit findings, conflicting witness statements, versions of a policy) rather than one built from entities acting on each other.
Where the relations already exist in structured form (an ownership register, a citation index, a CRM’s account hierarchy) no extraction step is needed at all. The graph is built directly from data already held, which is the case covered from the embedding side in Embedding-Based Relationship Discovery.
Multi-hop questions. “Which of our suppliers are exposed to the sanctions affecting our second-tier vendors” requires joining facts that appear in different documents and are never stated together. Vector retrieval returns chunks similar to the question, and no chunk resembles a question whose answer is a chain. The graph traverses the chain because the chain is what it stores.
Whole-corpus questions. “What are the recurring themes in three years of incident reports” has no answer in any single passage. Top-k retrieval returns five reports and the model summarises those five. Community summaries were built for exactly this case: the system answers from a hierarchical view of the entire corpus rather than from whatever the nearest-neighbour search surfaced.
Everything else is the majority case. When the answer sits in one passage, and the question resembles that passage, plain vector retrieval finds it, and the graph adds cost without adding accuracy. Ordinary document question-answering, policy lookup, support deflection, and most internal search fall here.
The graph does not replace vector search; it usually sits downstream of it. In Microsoft’s design, entities and community summaries are themselves embedded, and a query’s nearest neighbours among those embeddings decide where in the graph to start: which node, which community. Traversal happens after that, across explicit edges, to pull in connected facts the query embedding alone would not have surfaced. Vector search finds the foothold; graph structure walks outward from it.
Three costs, and the third is the one that catches deployments after they ship.
Indexing. Extraction runs over every chunk to pull entities and relations, then again to generate community summaries. On a large corpus this is a substantial one-off token bill and a slow build, and none of it answers a user’s question directly.
Query latency. Traversing the graph and summarising what comes back adds work at inference time. Reported figures put end-to-end latency at roughly two to three times a comparable vector pipeline, which matters for anything user-facing and matters less for batch analysis.
Incremental updates. The graph index and its summaries grow super-linearly with corpus size, and adding documents can shift community boundaries, invalidating the summaries built from them. A corpus that changes daily either gets rebuilt repeatedly or drifts out of date. This is the cost that decides most deployments, and the one least visible in a proof of concept run against a frozen document set.
The research response to the traversal cost has been to keep the relational signal and drop the walk. LightRAG folds entities and relations directly into a standard dense index and retrieves in two rounds, a fine-grained pass over specific entities and a coarser pass over thematic keys. There is no explicit path-walking step; “ROCm” and the frameworks it supports sit near each other in the same embedding space, found by nearest-neighbour search rather than by following an edge. This removes the traversal cost, allows the index to be patched incrementally, cuts indexing token cost by roughly 60% and roughly halves median query latency against full GraphRAG, without a clear quality penalty on multi-hop benchmarks.
If the reason for wanting a graph is multi-hop retrieval rather than whole-corpus summarisation, the folded-relations approach is the sensible starting point. Community summaries are the expensive component, and they exist to serve the global-question case specifically.
Take a sample of fifty real questions from whoever will use the system, before any implementation exists. Classify each one: does the answer sit in a single passage, does it require joining facts across documents, or does it require a view of the whole corpus. If the first category holds forty of the fifty, build vector RAG and spend the saved effort on chunking and reranking, which will move accuracy further than a graph would.
If the second and third categories carry real weight, the follow-up question is how often the corpus changes. A static corpus (regulations, published research, completed case files) absorbs the indexing cost once. A corpus with daily additions pays it repeatedly, and that recurring cost is what the decision turns on.
The general point holds beyond this choice: the retrieval architecture is a consequence of the question distribution, and the question distribution is knowable before anything is built. Most RAG projects skip that step and discover the distribution afterwards, from complaints.
Using our article on GPU selection: a paragraph stating that the RX 7900 XTX runs on ROCm, and a separate table, further down the document, listing which frameworks ROCm supports. “Which frameworks run on the RX 7900 XTX” has no single-passage answer; a flat vector index retrieves whichever chunk is closest to “RX 7900 XTX” and stops there.
The script below extracts a handful of triples from that article by hand (standing in for an LLM or NER pass), builds the graph, picks an entry node with a toy embedding, and walks the graph to answer the question.
import re
from collections import defaultdict
from math import sqrt
import networkx as nx
# --- 1. Extraction --------------------------------------------------------
# Normally an LLM pass or a NER + relation-extraction model over the
# source article. Written out by hand here so the graph stays legible.
triples = [
("RTX 5090", "uses_driver", "CUDA"),
("RTX 4090", "uses_driver", "CUDA"),
("RTX 3090", "uses_driver", "CUDA"),
("Jetson AGX Thor", "uses_driver", "CUDA"),
("DGX Spark", "uses_driver", "CUDA"),
("RX 7900 XTX", "uses_driver", "ROCm"),
("W7900", "uses_driver", "ROCm"),
("MI300X", "uses_driver", "ROCm"),
("M3 Ultra", "uses_driver", "Metal"),
("M4 Max", "uses_driver", "Metal"),
("CUDA", "supports", "PyTorch"),
("CUDA", "supports", "vLLM"),
("CUDA", "supports", "Ollama"),
("CUDA", "supports", "llama.cpp"),
("CUDA", "supports", "ComfyUI"),
("ROCm", "supports", "PyTorch"),
("ROCm", "supports", "vLLM"),
("ROCm", "supports", "Ollama"),
("ROCm", "supports", "llama.cpp"),
("ROCm", "supports", "ComfyUI"),
("Metal", "supports", "Ollama"),
("Metal", "supports", "llama.cpp"),
("Metal", "supports", "ComfyUI"),
]
G = nx.DiGraph()
for subject, relation, obj in triples:
G.add_edge(subject, obj, relation=relation)
# --- 2. A toy embedding, standing in for a real one ------------------------
# Bag-of-words cosine similarity over each node's name plus its immediate
# neighbours: enough signal to pick an entry point, no LLM call required.
def bow(text):
return re.findall(r"[a-z0-9.]+", text.lower())
def node_text(node):
neighbours = list(G.successors(node)) + list(G.predecessors(node))
return " ".join([node] + neighbours)
def cosine(a, b):
va, vb = defaultdict(int), defaultdict(int)
for w in a: va[w] += 1
for w in b: vb[w] += 1
dot = sum(va[w] * vb.get(w, 0) for w in va)
norm_a = sqrt(sum(v * v for v in va.values()))
norm_b = sqrt(sum(v * v for v in vb.values()))
return dot / (norm_a * norm_b) if norm_a and norm_b else 0.0
node_vectors = {n: bow(node_text(n)) for n in G.nodes}
def entry_point(query):
query_vec = bow(query)
scored = [(cosine(query_vec, vec), n) for n, vec in node_vectors.items()]
return max(scored)[1]
# --- 3. Traversal: the step a flat vector index cannot do ------------------
def frameworks_for(device):
driver = next(iter(G.successors(device)))
frameworks = [n for n in G.successors(driver) if G[driver][n]["relation"] == "supports"]
return driver, frameworks
query = "Which frameworks run on the RX 7900 XTX?"
device = entry_point(query)
driver, frameworks = frameworks_for(device)
print(f"Entry point: {device}")
print(f"Driver stack: {driver}")
print(f"Supported frameworks: {frameworks}")
This prints:
Entry point: RX 7900 XTX
Driver stack: ROCm
Supported frameworks: ['PyTorch', 'vLLM', 'Ollama', 'llama.cpp', 'ComfyUI']
The query names only the card, yet the answer joins a fact stated in one paragraph (device to driver) with a fact stated in a separate table (driver to frameworks), which is what “multi-hop” means in practice. Replace the hand-written triples with an LLM or NER extraction pass, replace the bag-of-words vector with a real sentence embedding, and the shape of the pipeline does not change. Fold the supports edges directly into node_vectors instead of walking them at query time, and this becomes the LightRAG.
(If you are working out whether your question set justifies a graph, get in touch.)
]]>PopStory writes illustrated short stories in several languages, each graded to a reading level. Ten or so model calls stand between a chosen theme and a finished story, and when one of them goes wrong the story arrives wrong without an error. So every stage writes a record: which model ran, under what instruction, on what input, how long it took, what it cost, and whether it failed.
Rendered, those records became a page. Each story now carries public production notes showing the cast, the setting, the five-part plan behind the text, the reference drawings, the models that ran and the bill for the run. The standing objection to generated writing is that it shows no working. The page shows the working.
The instruments that make a model pipeline debuggable are the same artefacts a business process needs and rarely has.
To make a chain of model calls work you have to answer a short list of questions: What are the steps? What does each one consume and produce? What does good look like? Can we check? What does each step cost? What happens when a step fails? and how do you know it failed rather than “looked plausible”?
Ask those five questions of a process in your own organisation. Onboarding, a compliance review, the route from a raised ticket to production code. Most organisations cannot answer them.
This is because human processes degrade gracefully. Where there is a gap, somebody fills it: they ask a colleague, they remember the exception, they find a document is out of date and work round it and so on. The gap rarely gets written down because it doesn’t cause enough of a problem. Or at least, the cost of absorbing it is less than the cost of writing it down. (Humans are very good at that type of judgement) Automation removes the absorber, and the gap surfaces immediately as a defect. This is why so many automation projects are a major discovery process.
Provenance on every output, meaning the model, the exact instruction, the inputs and the timestamp, is an audit trail. In regulated work it decides whether a system can be procured at all, independently of how good the output is.
Cost per call, attributed to the step that incurred it, is unit economics at the resolution of the work rather than the department. We know what a story costs, that image generation dominates the bill, and what text generation failure costs.
Stage records across many runs are process mining. Duration by stage across a batch shows where the time goes, and it is the analysis you would run over a workflow tool’s event log if anyone had kept one.
Evaluations are acceptance criteria that someone who did not do the work can check. Ours is level compliance: a story graded to JLPT N4 either stays inside N4 and N5 kanji or it does not, and a script settles it. “Well written” hands the standard back to the reviewer to supply.
Content addressing, where an output is keyed by a hash of the input that produced it, is impact analysis. Change one clause of a source document and the keys tell you which downstream outputs are now stale and which are untouched. Very few organisations can provide that answer.
Our stage records wrote a cost on success only. A call that spent money and then returned something unusable was recorded as free.
We found it when a degenerate loop cost roughly a hundred times a normal call and the records showed nothing at all. The fix was small: the error carries the cost.
The same bug runs through most reported numbers. Rework, abandoned drafts, the review that bounced three times, the ticket reopened twice, the document nobody could use so they rewrote it themselves. All of it is spend, and none of it reaches the figures that get reported upwards. The organisation knows the cost of time, but not the cost of inefficient work.
No AI is needed to find that. Instrumentation is enough.
Build an instrumentation habit. Ask who else would want to see it. The data already exists, rendering it costs close to nothing, and the debug view is often the thing a customer, an auditor or a reviewer will ask about.
Before automating a process, instrument the human version of it using a machine schema. Record the steps, the inputs and outputs, the duration, the cost, and the failures, including the ones somebody absorbed. People will be reluctant to do this because it is bureaucratic and tedious; exactly what machines love!
Automate, and you hold a specification of what and a baseline to measure against.
At this point, you hold a written description of a process, itemised costs, a specification for automation and a description of the risks and costs.
The failed AI pilots we have observed mostly fail because there was no coherent goal. Instrumenting first moves that finding forward, from a write-off to the deliverable.
]]>At a recent AI Wales meetup, several developers said some version of: “if a service doesn’t offer MCP, I will continue looking.”.
MCP, the Model Context Protocol, is a standard interface for connecting an AI agent to a tool or a data source. An agent using MCP discovers what a service can do and calls it directly, without a developer writing bespoke integration code for each new agent.
A REST API used to be a differentiator. A product that exposed one signalled other systems could be built against it. It’s now assumed, and a product without a REST API cannot be taken seriously as infrastructure.
What MCP does in practice is let an agent find a service and use it, the way a person finds and uses a website: by name, by search, by discovery, without builing the specific connection in advance. A product behind an MCP interface becomes reachable by AI agents, not just developers.
And none of this required much building. FastMCP and FastAPI turn the protocol and the REST layer beneath it into a few decorators around existing functions, not a system designed from scratch. The agentic workers calling the service, the actual AI systems, were built by someone else entirely: Anthropic, OpenAI, whoever ships the agent a person happens to be using that day. In many cases the service itself already existed as a product before any of this started. MCP is bolted onto something already finished.
The return on the work is not proportional. A product that speaks MCP reaches a new audience: agents and their users, discovering and using a service the same way they discover anything else in an agentic workflow. It surfaces use cases the original team never designed for, because the interface is generic enough that someone else’s agent can compose it into a task nobody at the company thought to build. And building the interface forces a clearer answer to what the product actually does, since MCP’s tool definitions have to describe the service in terms an agent can act on directly.
The developers in that room are the case in miniature. Each of them is now a potential audience for a product built this way, and so is every agent they set running on their own behalf afterwards. One integration reaching a person, and everything that person automates through it, is the exponential, network-shaped growth an integration is meant to buy.
]]>The major AI providers bundle two things that should be separate: the model and the infrastructure. When you call the OpenAI or Anthropic API, your data travels to their servers, runs against their model, and returns a result. The provider sits between you and the model, whatever their policy says about logging, retention, or training.
Open-weight models remove that party. Llama, Mistral, Qwen, and others publish their weights publicly, and the model can run anywhere. Marigold is that anywhere: an OpenAI-compatible API, a model-serving worker, and a chat interface, all running on hardware you control. No request reaches OpenAI, Anthropic, or any other external provider. Once your models are cached, it runs fully airgapped.
pip install bayis-marigold
git clone https://github.com/bayinfosys/marigold-examples
marigold deployment start marigold-examples/chat
The first command installs the marigold CLI. The second pulls a set of
pre-written application packages, each one a models.yaml (which models
to load) and a marigold.toml (how to run it). The third brings up the
full stack for that package, Postgres, a cache-init step that downloads
the declared models, the worker, the API, and (for the chat package)
Open WebUI. The model download is the only point in the process that
needs an internet connection; everything after that runs offline.
The full walkthrough, including how to confirm it’s working and where the model cache lives, is in the setup guide.
Marigold exposes the same interface as OpenAI and Anthropic. Existing application code that calls those APIs can point at Marigold instead without changes. The models underneath are open-weight equivalents, running on your own hardware rather than a shared cloud server.
(For the broader architectural case, see Private Inference: Running AI Inside Your Own Infrastructure.)
The model cache is shared across every package on the machine, independent of which one is currently running. Download a model once and every application package that needs it reuses the same copy, rather than each package pulling its own.
Application packages sit on top as the thin, disposable layer: a
models.yaml, a marigold.toml, and application code. Marigold runs
the package; it never runs the package’s own code directly, by design.
This is the same split described in
Models as a Fat Protocol –
the model as the durable, shared layer, applications as thin clients
built on top of it, the shape that made early internet protocols last.
Single model calls handle straightforward tasks. More complex automation requires composing multiple steps: embed a document, classify its content, generate a structured summary, evaluate the output. Marigold supports this through a typed workflow layer, where each step conditions on the outputs of the previous one and the whole pipeline is declared rather than hand-coded.
This is the same pattern described in the runfox workflow engine, which Marigold uses directly as its execution substrate.
The model types cover more than chat: text and image embedding, instruction-following, image generation, image-to-text, image segmentation, depth estimation, text-to-speech, and evaluation models for scoring text, image, and text-image alignment. In combination, these support workflows like quality-gated image generation (generate, score, regenerate until a threshold is met), visual conformance checking (compare an observation against a reference, diff the discrepancy, describe it in natural language), and batch tabular classification against a small labelled set, with low-confidence predictions routed to an instruct model for explanation.
Marigold is open source, on GitHub and PyPI. The tutorials cover setup, local document search, and adding a new model to an existing package; the examples are pre-built application packages to clone and run directly. Sizing your own hardware for this is its own decision: see our GPU options guide for what VRAM and which vendors actually support self-hosted inference in practice.
Marigold publishes benchmark data and implementation notes on its own Build Log.
]]>We presented a benchmark at AI Wales this week: 57 open-weight language models, 380 prompts each, power and memory telemetry captured per request with Marigold. The dataset and notebook are both available on github . Both were built to be simple rather than complete, to help develop a narrative and direction, rather than a goal.
The notebook follows the four classic steps of a data science task: Load the data; Check some values and ranges; Perform Exploratory Data Analysis via distributions and a correlation matrix; and, build one basic model: coefficient of variation. Computing this per model and ranking the models by the consistency of power draw.
The final outcome produces a result which is also a baseline for improvement. The most consistent model in the set is a clear number anyone can check against the raw CSV. It is also, on inspection, incomplete answer due to zero values in the data. So now, we have identified our first task: data cleaning. Which can then be performed, new results gathered, inspected; a new improvement identified and so on. This is the iterative loop which we can access once the project is established.
This is a deliberate scaffold. It is too easy and tempting to continue the analysis and improvement, and delaying delivery of the work.
It is important to separate the setup of a project, and the goals of setup, with the delivery of the project. A key goal of setup is to encourage collaboration within a team. A notebook with obvious places for improvement encourages engagement.
A notebook is a particular kind of artefact which creates a narrative through a problem. Data collection, parsing, cleaning, exploration, model selection, measurement and metrics and so on, can all be identified as steps in the notebook. In exploratory tasks, this sequence is important and clear. Missing a step is an obvious omission.
That narrative is not required in production where deployment requires integration, testing, and monitoring. A notebook is not a rough draft of a production system. It is the right form for presenting analysis so that a team, or a room of strangers at a meetup, can understand the process, ask questions, engage, and pick the thread up themselves.
Two questions came up in the room: Had we measured response accuracy? and, had we accounted for batch size and parallel decoding in the timing figures?
Accuracy evaluation and batching are the two things anyone with production LLM experience ask. Both questions can be seen as a way of demonstrating knowledge and suggesting next steps, and “feeling out” what space is left in the project. The useful answer was the same: no, but feel free to contribute!
This turned the demonstration into an invitation. People could leave the room comfortable that they could contribute to the project.
The usual approach to team culture treats engagement as a separate activity: an away day, a team lunch, a Friday demo. These are fine on their own terms, but sit outside the actual work, which means the habits they build – showing up, being sociable, sitting through a presentation – do not transfer back into work getting done.
Build the invitation into the work. Design for it to be continued and make it accessible enough that collaboration is the next natural step.
(If you’re building a data team’s engagement into its actual output rather than around it, get in touch.)
]]>A fluent answer and a correct one look identical from the outside. A model handed the wrong passage writes just as confidently as one handed the right one. So evaluation has to check each stage of the pipeline in order, since a fault at any stage produces the same symptom downstream: a well-formed answer with nothing underneath it.
Before measuring retrieval, check that the embedding model separates your concepts at all. Build a small set of query-passage pairs from your own domain, ones you already know should sit close together and ones that shouldn’t, and measure cosine similarity between them directly. A general-purpose embedding model trained on web text can genuinely fail to distinguish domain-specific concepts that look similar on the surface (two clauses that use the same words but mean opposite things contractually, say). If the embedding model can’t separate these in a raw similarity test, nothing downstream will fix it. This is a one-off check, not something you re-run continuously, and it’s worth doing before any of the following steps.
Hold the embedding model fixed and vary chunk size or strategy against the same corpus. Context precision and context recall are the relevant pair here: precision asks whether the chunk returned actually contains the answer, recall asks whether the answer exists somewhere in the corpus but got split across a boundary and lost. A chunking strategy that scores well on precision but poorly on recall is cutting documents in the wrong places. This is measured by holding retrieval and generation constant and only changing chunk boundaries, so any change in score is attributable to chunking alone.
With embedding and chunking settled, measure whether the right passages actually surface for a query. Precision at K: of the top K results, how many were relevant. Recall at K: of all relevant passages in the corpus, how many made it into the top K. Mean reciprocal rank: how high the first relevant result ranked, since a relevant passage at position 40 does little good if the model only reads the top ten. These three numbers are what most people mean when they say “RAG evaluation”, but they only mean anything once the two steps above are already accounted for.
Finally, check what the model did with what it was given. Faithfulness measures whether each claim in the answer traces back to the retrieved context, rather than being filled in from the model’s general training. Answer relevancy measures whether the response addresses the question asked, a separate failure from faithfulness: an answer can be fully grounded and still not answer the question. Hallucination, in this specific sense, is a claim with no support in the retrieved context at all, and it is the failure a user is worst placed to catch, since it reads exactly like a correct answer.
RAGAS is a framework, not a library of formulas or a single algorithm: it doesn’t compute a fixed statistic so much as orchestrate a series of judge calls, each one a separate LLM prompt with a narrow task, and aggregate the results per metric. Two stages sit behind that.
Building the dataset. RAGAS generates synthetic question-and-answer pairs directly from your own source documents, using an LLM to write plausible questions and their correct answers from the actual content. This is what makes evaluation practical at scale; without it, someone has to hand-write hundreds of question-answer pairs before any of the above can be measured. Each row in the resulting dataset carries four fields: the question, the ground-truth answer, the contexts your pipeline actually retrieved, and the answer your pipeline actually generated.
Scoring, metric by metric. Each metric is computed with a separate judge-model call (a different LLM from the one that generated the answer), and the mechanics differ per metric rather than reducing to a single rating question.
Faithfulness is computed by decomposing the generated answer into individual claims, then checking each claim separately against the retrieved context for support. The score is the proportion of claims that pass. This claim-by-claim decomposition is why faithfulness catches partial hallucination: four true claims and one fabricated one scores 0.8, not a binary pass or fail.
Context precision looks at the ranked list of retrieved chunks and checks, position by position, whether each is relevant, weighting relevant chunks that appear earlier in the ranking more heavily than ones further down.
Context recall works in the other direction. It takes the ground-truth answer, breaks it into its constituent claims, and checks whether each claim can be attributed to something in the retrieved context. Missing context drops recall even when precision looks fine.
Answer relevancy takes the generated answer and asks the judge model to reverse-engineer plausible questions that answer would be responding to, then compares those reconstructed questions against the original question by embedding similarity. A relevant answer produces reconstructed questions close to the real one; an off-topic answer doesn’t.
Each metric returns a score per row, aggregated across the dataset into a number per metric rather than one blended score, which is the design choice that matters most: a drop in one number tells you which stage of the pipeline broke, rather than telling you only that something, somewhere, is wrong.
LLM-as-judge is not a full substitute for human review. It can miss domain-specific errors a specialist would catch immediately (a subtly wrong dosage, a misstated clause, a transposed number that reads as grammatically fine), so treat RAGAS as the tool for continuous, high-volume measurement, with human review sampled in wherever a missed error is expensive.
None of the four measurements above is a one-time gate. The corpus changes, documents get added, edited, retired, and a chunking or retrieval score measured at launch decays as the source material drifts from what was originally indexed, with no change to the model at all. This is drift specific to the retrieval side of the system, distinct from the model drift a standard ML deployment watches for, and it needs its own detection loop rather than an assumption that generation-side monitoring will catch it.
Re-run the four stages on a fixed schedule against the current corpus, and track retrieval metrics against a rolling sample of real production queries rather than only the original eval set, since real usage drifts away from whatever questions the system was tested against at launch. When a metric drops, it points at the stage responsible: falling context recall means revisit chunking, falling precision at K means revisit retrieval, falling faithfulness means revisit generation. The measurement tells you which part of the pipeline to touch, which is the entire point of measuring each stage separately rather than scoring the system as one number.
(If you’re setting up drift detection for a RAG system already in production, get in touch.)
]]>