Machine Learning Engineer vs Software Engineer

Most teams that struggle with ML projects are not struggling with the ML. They are struggling with expectations imported from software engineering that do not transfer. The failure modes are different, the definition of done is different, and the structure of the work is different. None of this is obvious until something goes wrong.

Four differences are worth understanding before that happens.

Success is statistical, not binary

A software feature either works or it does not. An ML model occupies a position on a distribution. The question is not whether it works but how well it works relative to a baseline, and whether that is good enough for this context.

This changes the language of delivery. Acceptance criteria become performance thresholds. Sign-off requires a negotiated definition of acceptable, not a passing test suite. Teams that skip this negotiation early spend it later, under worse conditions, when the model is already in production.

The product is behaviour, not code

In a conventional system, the logic is in the code. In an ML system, most of the logic is in the weights – learned from training data rather than explicitly written. This means that changing the training data changes the product, even if no code changes. A model retrained on fresh data may behave differently from its predecessor in ways that are not visible from the source.

Debugging follows the same inversion. When an ML system misbehaves, the problem is usually in the data – labelling errors, feature drift, class imbalance – not in a code path. Engineers who reach for a debugger first will spend a long time looking in the wrong place.

Delivery is the start, not the end

A deployed model is the beginning of an optimisation loop, not its conclusion. Without monitoring for performance decay, mechanisms for retraining, and instrumentation to capture feedback, a model that works at launch will silently degrade as the world it was trained on recedes into the past.

This is not unique to ML, but the degradation is less visible than in software. A broken API returns an error. A drifted model returns confident wrong answers. The distinction matters for how teams plan post-deployment work, and for how clients understand what they are buying.

ML projects begin with a waterfall phase

Despite the iterative character of model development, ML projects typically open with a structured sequential phase that has to complete before experimentation can start. Data pipelines need to be built. Data availability and quality need to be confirmed. Labelling strategies and annotation tools need to be in place. Training and evaluation infrastructure needs to exist.

This phase does not compress well. Attempting to start model iteration before it is complete produces experiments whose results cannot be trusted, because the data and infrastructure they depend on are still in flux. The waterfall phase is not a failure of agile discipline; it is a prerequisite for the iteration that follows. See how to structure an ML project for more details.

A Note on Tooling

Software’s toolchain assumes the artefact under version control is text. Git tracks source files, and a debugger (runtime) or static analysis (build time) checks against specification. ML disturbs both assumptions: Training data is also a versioned artefact, usually far larger; tools like git-lfs, DVC, and dataset registries exist to serve this need.

Correctness checking also shifts. A model’s logic is learned, not written, so there is no line to step through on error. The equivalent work happens in logging, observability, and data validation – checking that inputs and outputs still sit within expected bounds. This does much the same job a debugger or linter does in software: verifying that the system behaves as expected in known conditions.

The practical consequence of all four differences is the same: ML projects reward investment in definition and infrastructure early, and punish the assumption that a working model at delivery means the work is complete. Teams that understand this going set better milestones, make better hiring decisions, and have more productive conversations.

(If you are assessing whether your organisation is ready to start an ML project, the data maturity article covers the infrastructure side of that question.)