AI Governance Belongs in the Build: What ISO/IEC 42001 Asks of Teams Shipping AI Features

Most software companies never set out to become AI companies. They shipped a summarization feature, then a support chatbot, then wired a large language model into search because a competitor had. A year later a third-party model sits in the data path and nobody can say with confidence what customer data went where.

That is usually when someone asks about governance, and by then it is expensive. AI governance belongs inside engineering, run the way mature teams run security, instead of as a policy that legal writes after launch. ISO/IEC 42001 shows what that involves.

The Retrofit Problem Nobody Budgets For

Retrofitting governance means reconstructing decisions after the fact. Which data set was used to fine-tune the model? Who approved it for production, and against what test results? If nobody wrote it down, someone digs through old tickets and chat threads, and some answers are simply gone.

Security went through this already. Companies that treated each audit as a scramble paid every year, while those that built access reviews and change control into normal work found audits became a matter of pulling existing evidence. AI moves faster, since models get swapped and retrained far more often than servers get patched. Logging prompts and outputs is a small design decision before launch, and a project of its own afterwards.

What the Standard Requires in Practice

ISO/IEC 42001:2023, published by ISO and IEC in December 2023, sets requirements for running and continually improving an AI management system in any organization that provides or uses AI-based products or services. It was the first international management system standard written specifically for AI, and accredited certification bodies now audit against it, so customers can ask to see a certificate.

Teams holding ISO 27001 will recognize the skeleton. Both follow ISO’s harmonized structure for management system standards, so clauses 4 through 10 line up and one program of internal audits and management reviews can serve both.

The AI-specific substance sits in a few clauses. Clause 6.1.2 requires a repeatable AI risk assessment covering consequences for the organization, individuals and society. Clause 6.1.4 adds something ISO 27001 has no equivalent for, an AI system impact assessment that considers intended use and foreseeable misuse. Both must be repeated at planned intervals and when significant changes are proposed, which matters for teams changing models every quarter. ISO/IEC 42005, published in May 2025, gives guidance on running the impact assessment.

Annex A lists 38 reference controls under nine objectives, from AI policy through the system life cycle, data, information for interested parties and third-party relationships. You choose which apply, justify each exclusion in a statement of applicability, and design your own controls where the annex falls short.

Data, Suppliers and the People Watching the Model

Engineering teams feel the life cycle and data controls most. Annex A expects documented requirements for each AI system, verification and validation before deployment, monitoring and event logs once live, and records of where data came from and what quality it must meet. Good machine learning teams already do most of this. The standard asks them to keep the evidence.

Supplier management deserves more attention than it gets, since plenty of AI features are thin layers over a model somebody else trains and hosts. The standard asks for clear allocation of responsibilities between suppliers, your organization and customers. In practice that means knowing what the provider does with prompts, where inference runs and what happens when the model changes underneath you.

Then come human oversight and transparency. Users need to know what a system is for, where it tends to fail and how to report a problem, and someone inside the company needs the authority and tooling to intervene. Planned at design time, that is a disclosure line, a feedback control and a feature flag that switches the model off. Added after a public incident, it arrives in a hurry, with lawyers in the room.

The Questionnaire Is Already in Your Inbox

For most software vendors the pressure arrives through procurement first. Shared Assessments added artificial intelligence as a risk domain in the 2024 edition of its SIG vendor questionnaire. The Cloud Security Alliance released its AI Controls Matrix in July 2025 with a companion questionnaire, the AI-CAIQ, which can be used to evaluate vendors. Where does AI sit in the product? What customer data does it touch, and does any of it train a model? Who reviews the output?

A vendor with a working AI management system answers from records it already keeps. One without answers from memory, slowly, with the hedging procurement teams have learned to spot. For many teams weighing ISO 42001 certification, the strongest argument shows up here, in shorter security reviews and fewer follow-up calls, well before a regulator asks anything.

Where the EU AI Act Fits

The EU AI Act entered into force on 1 August 2024. Bans on certain practices applied from February 2025, obligations for general-purpose AI models from August 2025 and transparency rules from August 2026. The AI Omnibus amendments, in force since 27 July 2026, moved high-risk rules for Annex III areas such as employment to 2 December 2027, and those for AI in regulated products to 2 August 2028. The Act also reaches providers outside the EU that place systems on its market, Canadian vendors included.

ISO/IEC 42001 is not a harmonized standard under the Act, so a certificate creates no presumption of conformity; CEN-CENELEC JTC 21 is drafting those standards separately. Even so, risk management, data quality, logging, documentation and human oversight all feature in the high-risk obligations, and a company already running them starts well ahead of one inventing them against a deadline.

Already shipping AI features, or about to? Start with an inventory of every place a model touches the product or internal operations, who owns it, which supplier sits behind it and what data it sees. Run an impact assessment on the riskiest entry and record what you find, uncomfortable parts included. Then add both documents to the release checklist so the next AI feature cannot ship without them.