A while ago, I was chatting with a few friends about our experiences with vibe coding, or AI-assisted programming. I mentioned that I felt uneasy when AI suddenly generated a large amount of code, but I could not fully understand what it had written.
My friends jokingly replied, “That’s because you haven’t written enough code. We’ve already reached the point where we don’t want to review everything line by line. As long as the tests pass and the results are verified, it’s good enough.”

“Ding!” The word risk immediately flashed through my mind.

To be fair, when we use formulas in Excel to calculate data, we also trust that Excel is correct. Few people would manually recalculate everything to verify the numbers.
Modern software systems have become mature enough that this way of thinking is usually reasonable.

But can we think the same way when facing AI?
My answer is: not entirely.

Traditional software is mostly driven by explicit rules. As long as the logic is correct, test cases are sufficiently covered, and input conditions remain stable, system behavior is generally predictable.
But AI, especially generative AI, does not simply execute predefined rules. It produces outputs through a combination of probability, context, data, and model behavior.
This is why, in AI projects, Human-in-the-Loop (HITL) should not be treated as just a slogan. It should be designed as an explicit control point within the process.

The question is not simply whether humans should review AI outputs. The real questions are: which situations require human review, who has the authority to make the judgment, and under what conditions the system must be paused or escalated.
So we should not only ask:
  • Did the result pass the test?
We also need to ask:
  • In what situations might AI fail?
  • Who would be affected if it fails?
  • Who would detect the failure?
  • Who can intervene?
  • Who has the authority to stop it?
That is what AI Risk Assessment is really about.

AI Risk Assessment Is Not Just a Renamed Project Risk Register

When people hear the term “risk assessment,” they often think of a project Risk Register or RAID Log.
For example, unclear requirements, schedule delays, insufficient resources, budget overruns, or vendor delivery delays.
These risks are still important.
But the risks of an AI project do not only come from project execution. They also come from the behavioral characteristics of AI itself.
AI risk assessment needs to consider several dimensions together:
  • Data
  • Model
  • Use context
  • Governance responsibility
  • Legal and ethical considerations
  • Human-AI interaction
In other words, AI projects should not only manage delivery risk. They also need to manage decision risk, data risk, model risk, and governance risk.
This is why an AI project that succeeds in a PoC does not necessarily mean it is ready for production.
A PoC proves whether something is feasible.
Risk assessment answers whether it can be used responsibly.

Question 1: What Decision Is This AI Helping Someone Make?

The first question in AI risk assessment is not technical. It is a contextual question.
That is:
  • What decision is this AI helping someone make?
  • What would happen if it got the decision wrong?
If AI is only helping employees summarize meeting notes, the risk may be relatively low.

But if AI is used to support customer complaint classification, credit assessment, pricing recommendations, resume screening, medical judgment, compliance review, public-sector case handling, or cybersecurity incident classification, then we should not look at it only from the perspective of efficiency improvement.

In these scenarios, AI outputs may affect people’s rights, resource allocation, financial outcomes, compliance responsibilities, or even public safety.

Therefore, the first criterion for AI risk classification should be its decision impact.

If AI only provides a reference, that is one level of risk. If it influences decisions, the risk level should be higher. If it can automatically execute decisions, it must be subject to stricter governance.

Question 2: What Data Does the AI Use?

Many AI risks may appear to be model problems on the surface, but they are actually data problems.

For example:
Is the data source lawful? Does the data contain personal information, customer data, trade secrets, or sensitive information? Has proper consent been obtained? Is the data quality complete? Is the data biased? Is the data outdated? Will the data become distorted over time?

In traditional systems, data errors usually lead to calculation or reporting errors.
But in AI systems, data problems may be packaged into an answer that looks highly convincing.
This is one of the most dangerous characteristics of generative AI: it may not only be wrong, but wrong in a way that looks true.

Therefore, AI risk assessment must ask:
  • Is the AI’s input data trustworthy?
  • Are the knowledge sources it cites traceable?
  • Could it carry sensitive data into places where that data should not appear?
  • Could data bias lead to unfair outcomes?
Data governance is not a supporting role in AI projects. It is at the core of AI risk management.

Question 3: What Risks Exist in the Model Itself?

When evaluating AI, many people first think about accuracy.
Accuracy is important, but it is not everything.
AI models have many risks that are less common in traditional systems, such as:
  • Hallucination
  • Bias
  • Unstable outputs
  • Lack of explainability
  • Model drift
  • Vulnerability to prompt injection
  • Failure in edge cases
  • Inappropriate content generation
  • Incorrect source citation
  • Sensitive information leakage
These problems cannot be fully solved simply by saying, “The tests passed.”
AI outputs are affected by prompts, context, data sources, model versions, parameter settings, and user inputs.
In other words, passing today’s tests does not guarantee that the system will remain safe tomorrow under a different context, a different dataset, or a different model version.
This is why AI systems require continuous monitoring, not one-time acceptance testing.

Question 4: Who Is Responsible for Monitoring? Who Has the Authority to Pause It?

The most easily overlooked part of AI governance is not technical control. It is accountability.
When companies adopt AI, many people may say, “This is the IT department’s responsibility.”
But the business team may say, “This is part of the business process.”
The security team may say, “We are only responsible for security requirements.”
The compliance team may say, “We need to understand the use context first.”

Eventually, this creates a dangerous situation:
  • Everyone has partial responsibility, but no one owns complete responsibility.
Therefore, AI risk assessment should not only list risks. It should also establish a responsibility matrix.
At minimum, it should answer:
  • Who is the AI Use Case Owner?
  • Who is the Data Owner?
  • Who is the Model or System Owner?
  • Who is responsible for monitoring and human review?
  • Who has the authority to pause the AI system?
  • Who is responsible for reporting and handling incidents?
If these questions are not defined in advance, the problem is usually not that nobody does anything when an incident happens. The problem is that everyone assumes someone else will handle it.

Question 5: Which Decisions Can Be Given to AI, and Which Cannot?

What AI adoption needs is not only a risk list. It also needs a decision authority matrix.
It should define:
  • What can AI complete automatically?
  • What can AI only provide as a recommendation?
  • What must be approved by a human?
  • What situations must be escalated?
  • What outputs are considered unacceptable?
For example, a customer service AI may automatically answer general questions. But if the issue involves refunds, legal liability, or escalated complaints, it must be transferred to a human.
An AI coding assistant may generate code. But if the code involves security validation, access control, database operations, or payment logic, it must go through stricter code review.
An AI document assistant may summarize internal documents. But if the documents involve personal information, contracts, compliance issues, or board materials, data classification and access control must be applied.
This is where AI governance becomes real. It does not end with a disclaimer that “AI-generated outputs are for reference only.”
Organizations need to clearly define AI boundaries, authority, and escalation mechanisms.

Question 6: What Risk Indicators Should Be Monitored After AI Goes Live?

AI systems are not like traditional systems, where things are generally stable as long as the functions are not broken.
AI requires continuous monitoring of risk indicators, such as:
  • Error rate
  • Hallucination rate
  • Human review rejection rate
  • Number of user complaints
  • Abnormal output ratio
  • Sensitive data incident count
  • Model drift indicators
  • Error differences across different groups
  • Percentage of AI recommendations overridden by humans
These indicators are not just KPIs. They are early warning signals for AI governance.
If an organization focuses only on “increased usage” or “reduced processing time,” it may overestimate the value of AI while underestimating the accumulating risks.

How to Design a Simplified AI Risk Assessment Form

To get started quickly, I would suggest using a simple form first rather than creating a thick governance document from the start.
Each AI use case should answer questions from at least five dimensions:
5 dimensions of questions that each AI should answer

The most practical way to use AI risk assessment is to integrate it into existing project governance processes.

If a company already uses a Risk Register or RAID Log, there is no need to create a completely isolated AI risk document.

By adding AI-specific risk items to the existing risk management process, AI risks are more likely to be tracked alongside other project risks rather than appearing only once during a review meeting.

This will become an important new capability for project managers in the AI era.

A Simple Rule for Identifying High-Risk AI

How can we determine whether an AI function is high risk?

I like to use the following equation:

High-risk AI = High-impact decisions + Highly sensitive data + Low explainability + Low human intervention capability.

When these four factors are all high, the system should not be treated as an ordinary IT project.

For example, if AI uses large amounts of personal data to support financial credit decisions, while the model’s reasoning is opaque and humans cannot intervene in time, then this is not simply “using AI to improve efficiency.”

It is a high-risk AI system that requires strict governance.

On the other hand, if AI only helps employees summarize non-sensitive meeting notes and humans decide whether to use the output, the risk level is relatively low.

The purpose of AI governance is not to control every AI system so tightly that nothing can move.

The point is to apply different levels of governance to different levels of AI risk.

From AI Risk Assessment to AI Governance SOP

After risk assessment is completed, the next question is: how should the organization manage these risks?

This is where AI Governance SOPs become valuable.

Companies should have at least a few basic mechanisms.

First, an AI Usage Policy: clearly define which AI tools can be used, what data must not be entered, which scenarios require approval, and which behaviors are prohibited.

Second, an AI Adoption Assessment Process: before any AI use case is adopted, it should complete risk classification, data assessment, security assessment, compliance assessment, and vendor assessment.

Third, an AI Incident Response Process: when AI causes hallucinations, incorrect recommendations, personal data leakage, prompt injection, RAG document leakage, or uncontrolled automated actions, the organization should know how to report, pause, correct, and recover.

Fourth, an AI Change Management Process: model version updates, prompt template changes, RAG knowledge base updates, and data source changes may all affect AI behavior. These changes should be tested, validated, recorded, and approved.

Fifth, an AI Audit Trail: the organization should be able to prove when the AI system was assessed, who approved it, what model and data were used, what outputs were generated, who reviewed them, and how problems were handled.

Without an audit trail, true AI accountability is difficult to achieve.

The Real Question Is Not Whether AI Will Make Mistakes, but Whether We Know When It Does

The value of AI lies in improving efficiency, expanding capability, and accelerating decisions.

But the risk of AI is that it may allow errors to happen faster, at a larger scale, and in a more convincing form.

So when facing AI, we should not only ask:

  • Can it do this?

We also need to ask:

  • If it makes a mistake, can we know?
  • Can we intervene?
  • Can we stop it?
  • Can we be accountable?
  • That is the core of AI Risk Assessment.

It is not meant to block AI adoption. It is meant to help AI adoption go further, become more stable, and earn greater trust.