5 min

When not to use machine learning

The most valuable thing an AI consultancy can do, sometimes, is tell a client they don't need AI.

Most failed engagements fail at the scoping stage rather than the modelling stage. The client identified something they wanted automated, assumed 'automated' meant 'machine learning', and proceeded to spend six to twelve months building a system that a two-hundred-line SQL query would have solved in a week. This happens more often than anyone in the industry admits. The tell is always the same: the proof of concept works on a clean pilot dataset and the production system fails as soon as the distribution of inputs is slightly different from the pilot. The root cause is that the problem never needed ML in the first place. It needed rules, and it got a model.

When ML is the wrong answer

Three situations where machine learning is consistently the wrong answer.

The first is small, well-understood, low-frequency decisions. Consider: determining which of four loan products to recommend to a new customer, based on their age, income, and stated purpose. There are, at most, a few dozen realistic combinations. The bank's product team already has an opinion about each one, written down somewhere. What they want is not an ML model; they want that opinion encoded as rules and applied consistently. A decision tree with eight leaves, hand-specified, beats a gradient-boosted model here on accuracy (because the rules are actually what the business wants), maintainability (rules are readable), and auditability (every decision is reproducible without a model version). The bank's real problem was that their existing rules lived in a spreadsheet nobody trusted. That is a data engineering problem, not a machine learning problem.

The second is decisions that require judgement under unfamiliar context. Approving a non-standard invoice that doesn't match any known template. Waiving a fee for a long-standing customer. Deciding whether a complaint is a serious incident or a misunderstanding. These are tasks humans do well specifically because they can bring outside context: customer history, business judgement, knowledge of current promotions, tone of voice in the original email. Machine learning models don't do this. They average across the training data. Trying to replace judgement work with an ML model produces a system that handles the easy cases (which the humans handle in thirty seconds anyway) and fails confidently on the hard cases (which is the only part anyone was actually paying for). The efficiency gain is negative.

The third is deterministic decisions that happen to be expensive. A regulation says that if a customer is under eighteen, they cannot take out a credit product. That is not a machine-learning problem. It is an IF statement. And yet proposals regularly arrive to 'build a compliance engine' that uses ML to classify customer eligibility. The reason this happens is usually that the data about customer age is spread across three systems that don't agree, and instead of fixing that, someone proposed an ML system that tolerates the disagreement. The ML system doesn't fix the disagreement; it just obscures it. The correct intervention is the unglamorous one: reconcile the data sources. The ML proposal is a way to avoid doing that.

A useful set of questions to ask before committing to an ML solution follows.

Is the decision rule, if it existed, expressible in under fifty lines of SQL? If yes, the SQL is probably the right answer.

Is the decision expected to be made with fewer than a few hundred examples per month? If yes, a human probably scales fine, and automating with ML will fail on edge cases for longer than the human would take to handle them manually.

Does the decision require information that isn't in the data you're training on? If yes, you're not modelling the decision, you're modelling a proxy for it that will drift away from the actual decision over time.

Is there an existing rule-based system that's unpopular because it's rigid? If yes, building an ML system on top doesn't make it less rigid; it just makes the rigidity harder to see. Fix the rule system.

This is not a case against machine learning. There are vast classes of problem (pattern recognition over high-dimensional inputs, probabilistic decisions over millions of examples, optimisation under genuine uncertainty) where ML is the right and only answer. Those problems are real. But they are a smaller share of the problems that get scoped as ML projects than the industry pretends. The honest assessment at the start of an engagement (is this actually an ML problem?) saves more money than any subsequent modelling decision.

If a consultancy won't give you a straight answer to that question, you're probably about to pay them to build something that shouldn't exist.

// The artefact

-- compliance/eligibility.sql: a regulation, not a model
SELECT customer_id,
       CASE
         WHEN age < 18                   THEN 'ineligible: minor'
         WHEN annual_income < 25000      THEN 'ineligible: income'
         WHEN credit_score < 600         THEN 'ineligible: credit'
         ELSE 'eligible'
       END AS decision
FROM customers;

If the rule fits in fifty lines of SQL, the SQL is the system. No model, no drift, no audit gap.

← Back to Insights