The Autonomy Question: What Aviation Teaches About Accountable AI

By Mark Bold | emerging-tech | 7 min read

Aviation offers useful lessons about automation, monitoring and intervention. Applying those lessons to AI starts with defining what a system may do and whether people can actually take control.

A business can automate an action without resolving who is responsible for it. A system may recommend a price, respond to a customer or move a request through an approval process. The operational question is whether it performs that task well. The leadership question is what happens when its output is wrong, its permissions expand or the person expected to supervise it cannot intervene.

Aviation offers useful ways to examine that second question. Its automation practices should not be treated as proof that every industry needs an aircraft certification regime. They do, however, make the relationship between a machine, an operator and an operating procedure unusually concrete. That relationship matters wherever a business gives software the ability to affect other people.

Scope

This article draws operating lessons from aviation guidance, not aviation legal requirements for other sectors. The FAA's Flightpath Management advisory circular addresses particular aviation operators and training providers. It describes acceptable practices and expressly states that its contents do not themselves have the force and effect of law. NIST's AI Risk Management Framework is voluntary guidance. Neither document creates a universal certification requirement for business AI. Applicable legal duties require separate analysis of the organization, jurisdiction and use. [1] [2]

Automation does not settle responsibility

The FAA guidance describes flightpath management as a responsibility of the crew, including when automated systems are used. It emphasizes awareness of the intended flightpath, monitoring and the ability to intervene. Its discussion of automation places the equipment within a broader operating task rather than treating the operation of the equipment as the entire task. [1]

For a business leader, the transferable question is not whether software is impressive. It is whether the operating arrangement remains understandable when the software takes a consequential step.

Consider a hypothetical supplier review process. An AI tool sorts applications and recommends which suppliers should proceed. That is one arrangement. Allowing it to reject suppliers, send commitments or change purchasing records is another. A successful recommendation pilot does not, by itself, establish that the wider arrangement is appropriate.

The company should describe the proposed action, the permission that enables it and the person who can authorize the resulting commitment. These are business design questions before they become configuration questions. Responsibility should be assigned to someone with enough authority and information to influence the operation.

An approval is not a universal certificate

The original temptation is to borrow aviation's vocabulary and promise that an AI system can be certified safe before it is used. That promise is too broad. Evidence must relate to a particular use, environment and set of limits. A useful internal approval is not a guarantee against every possible failure, and it should not be described as regulatory certification.

NIST organizes its framework around Govern, Map, Measure and Manage. Its approach includes examining intended uses and impacts, evaluating risks and managing systems over their lifecycle. It does not prescribe one risk tolerance for every organization. [2]

The practical implication is to make the approval narrower than the sales presentation. Identify the work the system will perform, the data it will receive, the decisions it may influence and the actions it may take. Record the conditions under which that approval expires or needs reconsideration.

EASA's Artificial Intelligence Roadmap 2.0 describes a human centric approach and a programme of guidance and anticipated rulemaking activities in aviation. It is a roadmap, not evidence that ordinary corporate AI deployments have already been certified under a common aviation standard. [3]

Oversight has to fit the work

Putting a human name into a workflow does not establish meaningful supervision. The person needs relevant information, time to examine it and a practical means to stop or redirect the action. If the process makes those things unavailable, describing it as human oversight can conceal the weakness rather than address it.

The FAA guidance includes training, manual proficiency and intervention strategies. These are operational arrangements, not simply statements that a pilot remains present. NIST likewise identifies human oversight roles and the ability to supersede, disengage or deactivate systems whose performance is inconsistent with intended use. [1] [2]

In the supplier example, a reviewer might see the supporting documents and the reasons for a recommendation before any rejection is sent. Exceptions might be routed to someone with purchasing authority. A pause control might stop outgoing communications without destroying the records needed to examine the problem.

Those are proposed controls, not a prescribed design for every company. Their adequacy depends on the consequences, volume and speed of the work. A low consequence drafting tool and a system that changes contractual records should not automatically receive the same supervision model.

Make a failure understandable

An investigation is harder when the organization cannot reconstruct which version of a system was running, what information was available and which person approved an action. A business should decide what evidence it needs before it becomes dependent on the system.

That does not mean collecting every possible input indefinitely. Records can create privacy, confidentiality, security and retention concerns of their own. Legal and technical teams should determine what information is appropriate to preserve and who may access it.

A proportionate record might identify the relevant system version, approved use, permissions, exception and human response. It should help the company distinguish a poor model output from an incorrect instruction, missing information or an inappropriate business rule.

NIST treats documentation, monitoring and response as parts of risk management. The business recommendation is to translate those ideas into evidence that can support an actual operational decision. A dashboard showing that a tool remains available is not the same as a record showing that its consequential decisions remain within the approved limits. [2]

Rehearse the alternative

Before expanding automation, ask what the business would do if it needed to stop using the system. A fallback can be reduced functionality, a manual process, a different supplier or a temporary suspension of the affected activity. The appropriate choice depends on the operation.

The fallback should be tested at a realistic level. Can staff find the necessary records? Do they know who may pause the process? Can they perform the essential work without relying on the same component that has failed? A written contingency has limited value if nobody can execute it.

NIST includes contingency processes for risks involving outside providers and mechanisms for disengaging or deactivating unsuitable systems. Those recommendations support planning for an alternative, but they do not prove that a particular supplier's export or recovery features will meet the company's needs. Those capabilities need to be checked. [2]

Bring the business question to counsel

Counsel can contribute before the deployment becomes an incident. Useful questions concern the intended transaction, the information involved, the company's promises, relevant approvals and the consequences of a mistaken action. The objective is not to attach a generic disclaimer to an unchanged workflow.

Management should explain what it wants to achieve and what tradeoffs it is considering. Counsel can then help examine the actual obligations and constraints, alongside the technical and operating teams. The result may be a narrower permission, a different contract term, a clearer escalation route or a decision not to automate a particular step.

The aviation lesson is therefore one of discipline rather than regulatory imitation. Define the task, preserve competent intervention and prepare for conditions outside the normal demonstration. The business remains responsible for deciding what authority it gives the system.

Executive Imperatives

Define the permission. Separate a recommendation from an action that changes rights, records or commitments.

Test real intervention. Confirm that the designated person has information, time and a working way to pause the process.

Preserve proportionate evidence. Agree on records, access and retention before a failure makes reconstruction urgent.

Rehearse an alternative. Check how essential work continues if the approved automation must be withdrawn.

Sources

1. [FAA Advisory Circular 120-123, Flightpath Management, especially sections 1.1, 2.1, 2.2 and 3.3.5](https://www.faa.gov/documentLibrary/media/Advisory_Circular/AC_120-123.pdf)

2. [NIST AI 100-1, AI Risk Management Framework 1.0, especially Govern, Map and Manage](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf)

3. [EASA Artificial Intelligence Roadmap 2.0, published May 2023](https://www.easa.europa.eu/en/document-library/general-publications/easa-artificial-intelligence-roadmap-20)