Reliable AI Systems: A Practical Way to Control Risky Outputs

In this article9 min read

Run the same calculation through a conventional software service twice and you expect the same answer. If the second answer changes, something is probably broken.

Run the same request through a large language model twice and you may receive two different answers that are both plausible. Neither variation automatically means the model has failed. It does mean the surrounding system needs a more demanding definition of success.

That is the engineering problem hiding beneath much of the current AI conversation. A capable model can produce an impressive demonstration. Building reliable AI systems needs context controls, bounded authority, evaluation evidence, production telemetry and people who know when the system should stop.

The model produces possibilities. Engineering decides which possibilities are safe enough to use.

The useful thing the skills map gets right

In the 22 August 2026 edition of DeepLearning.AI’s The Batch, Andrew Ng expanded the first part of an AI Engineering Skills Map. He said the map was informed by job postings, structured interviews with experts and survey responses.

The top-level map contains four areas: building and deploying AI applications, software-engineering fundamentals, using coding agents and shaping the build. Ng’s latest letter breaks the first area into six capabilities:

• LLM foundations
• Grounding models with data
• Building agentic systems
• Evaluation-driven development
• Operating in production
• Machine-learning foundations

The important insight is not the list itself. It is the reason those capabilities belong together.

Ng describes AI development as more iterative because the output of an AI component is less predictable than the output of conventional code. The engineer’s job is to create “reliable software systems based on unreliable AI components.”

That is a better way to read the map. It is not simply a curriculum for learning six technical topics. It is a control system for managing uncertainty.

The engineering object is the whole system

Teams still talk about AI quality as though it lives inside the model. They compare benchmark results, debate model sizes and switch providers when an output disappoints them.

Model capability matters. It is only one variable.

The same model can behave very differently depending on the instructions it receives, the information available in its context, the tools it can call, the state retained across a session and the checks applied before an answer reaches a user.

This changes what must be designed and reviewed. The unit of engineering is no longer just the model invocation. It is the complete route from user intent to data retrieval, model reasoning, tool execution, verification, human decision and operational feedback.

A strong model can reduce one source of error while the surrounding system quietly introduces five more.

A team that improves prompts but ignores stale data has not solved reliability. A team that builds a sophisticated agent without limiting its permissions has increased capability and exposure together. A team that runs an evaluation once but cannot see failures in production has produced a test result, not an operating model.

Grounding is a data responsibility

Retrieval-augmented generation is often presented as the answer to model limitations: connect the model to company knowledge and it will become useful.

The hard part begins after that sentence.

Someone has to decide which sources are authoritative, how documents are converted into usable inputs, how quickly an update becomes available and what happens when two sources disagree. Permissions need to survive the journey from the source system to the model context. Provenance needs to remain visible enough that a person can understand why an answer was produced.

A vector database is one retrieval mechanism. It is not a substitute for information ownership.

The correct design may combine documents, structured records, a semantic layer and tools that retrieve information only when it is needed. The choice depends on the question, the shape of the data and the consequence of getting the answer wrong.

This is the same boundary explored in Generative AI Customer Insights: What AI Can and Can’t Do. AI can organise and synthesise available evidence. It cannot repair missing evidence or make an unreliable source trustworthy.

Agents need an authority architecture

Agentic systems make the control problem more visible because the model is no longer producing only an answer. It may choose a tool, modify a file, query a system, send a request or decide what to do next.

Anthropic distinguishes workflows from agents. Workflows follow predefined code paths. Agents decide dynamically how to pursue an objective. Its practical advice is to begin with the simplest architecture that works and add autonomy only when the value justifies the extra cost and uncertainty.

That distinction should change the design review. The first question is not whether a team can build an agent. It is whether the task needs model-directed autonomy.

If it does, the design needs explicit answers to operational questions:

• Which tools can the agent call?
• Which data can each tool expose?
• What actions require human approval?
• What is logged for later review?
• How many iterations or how much spend can one task consume?
• What conditions make the agent stop, escalate or roll back?
• Who owns the outcome when the agent’s action is technically valid but operationally wrong?

These are not additions to the architecture. They are the architecture of authority.

Evals turn uncertainty into delivery discipline

Conventional testing asks whether known inputs produce expected outputs. AI evaluation often has to ask a more difficult question: whether a variable output is acceptable for the intended task.

That requires judgement before automation.

A team needs examples that represent real work, including ordinary cases, awkward edge cases and failures that would matter. It needs a taxonomy that separates retrieval failure from instruction failure, tool failure, unsafe action and a fluent answer unsupported by evidence.

Only then can the team choose the right evaluation method:

• Deterministic checks for facts such as schema validity, required fields, forbidden actions and exact calculations.
• Model-based graders for qualities such as relevance or completeness where a rubric can be stated clearly.
• Human review where context, consequence or professional judgement cannot be reduced safely to a score.

OpenAI’s evaluation guidance treats an eval as testing criteria plus a representative data source. The mechanics matter, but the deeper discipline is the loop: inspect errors, decide which failure is worth addressing, change one part of the system and test again.

This is where AI engineering meets product judgement. A system can improve its average score while becoming worse at the cases that matter most. A customer-support assistant that becomes more fluent but less willing to escalate has not necessarily improved.

Release evidence therefore needs to be calibrated to risk. A drafting assistant and an agent authorised to alter a customer record do not need the same threshold, review depth or rollback mechanism.

If the team cannot explain which failure improved, the latest model change is movement—not progress.

Production changes the truth

A prototype is usually tested with cooperative users, known examples and close attention from its builders. Production introduces vague requests, adversarial inputs, incomplete data, unusual tool states, cost pressure and users who interpret the system in ways the team did not predict.

That means deployment is not the end of evaluation. It is the beginning of a different evidence stream.

The NIST AI Risk Management Framework describes AI risk work through Govern, Map, Measure and Manage. Its production guidance includes monitoring system behaviour, gathering user feedback, maintaining incident-response and recovery mechanisms, and managing change throughout the system lifecycle.

For a production AI service, useful telemetry should make several kinds of failure visible:

• Quality falls for a particular user group or task type.
• Retrieval returns stale, irrelevant or unauthorised information.
• An agent repeatedly chooses an expensive or ineffective path.
• Latency rises because the workflow contains too many model calls.
• Costs increase without a corresponding improvement in successful outcomes.
• Users repeatedly override, correct or abandon the system.
• A model or prompt update regresses a previously stable behaviour.

These signals need an owner and a response path. Observability without decision rights becomes a dashboard of known problems.

This is also why measuring AI only through employee sentiment is too weak. As argued in Four Practical Truths About the Hidden AI Productivity Gap, activity and enthusiasm do not establish that end-to-end work has improved.

Machine-learning foundations still matter

The current application layer can make traditional machine-learning knowledge look optional. An API call hides the training process. A framework hides the orchestration. A managed service hides the infrastructure.

The abstraction is useful until something behaves unexpectedly.

Concepts such as bias and variance, representative data, error analysis and the difference between training and inference remain practical tools for reasoning about uncertain output. They help an engineer distinguish a local prompt defect from a broader capability limit, and a noisy test result from a real improvement.

The point is not that every application engineer must train a foundation model. It is that people responsible for system behaviour need enough statistical understanding to avoid treating one convincing example as evidence of reliability.

A five-part release contract

The skills map becomes operational when it changes the release conversation. Before an AI change moves from prototype to production, five questions should have explicit answers.

1. Outcome: what useful result must improve?

State the user or operational outcome, its baseline and the consequence of error. “Use AI to summarise cases” is a feature description. “Reduce preparation time without omitting mandatory evidence” is closer to a release contract.

2. Context: which data may the system use?

Name the authoritative sources, freshness expectation, permission boundary and provenance requirement. Do not allow “the knowledge base” to conceal several sources with different owners and reliability.

3. Action: what can it do without approval?

Define tools, transaction limits, human approval points and stopping conditions. Read access, draft preparation and irreversible external action belong to different authority classes.

4. Evaluation: how will failure be detected?

Maintain a representative test set, explicit failure categories and release thresholds. Record which failures a change fixes and which new risks it may introduce.

5. Operations: how will the team see and recover?

Specify logging, service and cost indicators, incident ownership, rollback and the route for user correction or appeal. A release without a recovery path transfers uncertainty to the user.

What leaders should change now

The organisational implication is uncomfortable. AI engineering cannot sit entirely inside an isolated technical team because the controls depend on product, data, security, operations and subject-matter judgement.

Leaders do not need to dictate model settings. They do need to create the conditions for disciplined iteration:

• Fund evaluation work as product development rather than post-build assurance.
• Give somebody clear ownership of the test set and failure taxonomy.
• Keep data ownership and access decisions visible in the delivery plan.
• Review agent permissions with the same care as human and service-account permissions.
• Require production feedback to shape the next development cycle.
• Measure successful outcomes, overrides and recovery—not only adoption and model usage.
• Preserve a safe path back when a model, prompt or retrieval change behaves differently in production.

Small tests still matter. A Minimum Idea State can expose whether a proposed AI capability changes behaviour before the team commits to a complete production architecture. The evidence requirement should grow with the authority and consequence of the system.

Models will improve. Context techniques, agent frameworks and evaluation tools will keep changing. The durable capability is the ability to observe uncertainty, place boundaries around it and learn from what happens next.

Reliable AI is not a property you buy with a better model. It is a system you keep earning through engineering.