Context

As with many of the other solutions ultimately expressed in the Artificial Intelligence Act, (Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence), the call for human oversight of AI systems must first be framed in the proper context of principles and philosophy.

Contrary to the expectation stated in the AI Act that AI systems will be overseen by humans, this isn’t at all obvious. AI systems, at least in certain processes, display the capacity for exceeding human capabilities. From this perspective, the expectation that humans will be in a position to properly oversee a machine that exceeds their own intellectual abilities is dubious. This is an argument regularly raised by critics of AI regulations.

We must first recognise the axiological significance of the decision by European lawmakers to introduce mandatory oversight by humans. They chose the direction of vesting primacy in humans, despite the many doubts articulated in this area.

The requirement of human oversight derives from one of the fundamental ethical principles on which the AI Act is grounded—respect for human autonomy. This principle was most fully expressed in the “Ethics Guidelines for Trustworthy Artificial Intelligence” from 2019, drafted by the High-Level Expert Group on Artificial Intelligence set up by the European Commission.

This principle emerges from the assumption that AI systems should be built in an anthropocentric context, i.e. in a model continuing to assume that humans exercise a leading role in shaping reality. AI systems should thus help augment, complement and empower humans’ cognitive abilities. Such systems should not lead to the subordination of the person, or to controlling or shaping humans—in other words, excluding humans from the process of impacting reality. That in certain area the capacities of AI systems may exceed human capacities does not change the notion that it is still only humans who are in a position to understand the broader context of the operation of these systems and to identify that risks that may be posed by their operation. If humans wish to avoid the negative impacts of these risks, they must maintain their own agency, which in this instance is manifested in exercising oversight of AI systems, i.e. making humans the final arbiter on how these systems are used.

The work of the high-level expert group is expressly recognised in recital 7 to the AI Act: “In order to ensure a consistent and high level of protection of public interests as regards health, safety and fundamental rights, common rules for high-risk AI systems should be established. Those rules … should also take into account the European Declaration on Digital Rights and Principles for the Digital Decade and the Ethics guidelines for trustworthy AI of the High-Level Expert Group on Artificial Intelligence (AI HLEG).”

Recital 27 further states: “According to the guidelines of the AI HLEG, human agency and oversight means that AI systems are developed and used as a tool that serves people, respects human dignity and personal autonomy, and that is functioning in a way that can be appropriately controlled and overseen by humans.” Moreover, “those principles should be translated, when possible, in the design and use of AI models.”

It should thus be recognised that the demand for human oversight is one of the foundational principles of the EU regulation on AI systems. At a high level of abstraction, it may be stated that the essence of this demand is to ensure that humans retain intellectual and factual control over the process which an AI system is engaged in carrying out, to guarantee protection of human dignity, rights and safety.

Oversight models

In practice, human oversight of AI systems can take various forms. In this respect, the Ethics Guidelines identify three basic models:

• Human-in-the-loop (HITL), providing for active human involvement in the process carried out using an AI system
• Human-on-the-loop (HOTL), where human interaction is limited to ongoing monitoring of the activity of the AI system in the given process
• Human-in-command (HIC), providing for high-level human supervision of the AI system and the possibility of deciding when and how to use the system.

The basic difference between these models boils down to the degree of engagement of the human factor in the operation of the AI system and generation of outputs by these systems. The HITL model is the most invasive, and involves active engagement of a person in the process for which the AI system is used. The AI system only contributes to the process, while completion of the process requires conscious, active involvement of a person. In this case, the results of the process (such as a decision to deny services) is the effect of a conscious human decision. The HITL model therefore assumes that the process is not fully automated, and cannot be executed entirely by a machine.

The other models are less radical and do not require human engagement in executing the specific process (i.e. they allow full automation of the process). The HOTL model provides for human monitoring of a specific automated process executed by the AI. The HIC model, in turn, provides for maintaining general high-level human control over the system as a whole, which is manifested in the possibility of deciding whether to use or not use the AI system, but not necessarily ongoing monitoring of specific processes executed by the system.

For the purposes of this article, I assume that the HIC model will not lead to supervision of the specific process for which the AI system is deployed. Only the aggregate manner of operation of the system will be subject to human control. For example, in the case of an AI system for taking decisions on employees’ salary increases, the oversight will not analyse individual decisions, but only ensure that the operation of the system does not violate some defined level of parity.

Analysis of the AI Act

The call for human oversight of AI systems is implemented in the AI Act at several levels. These can be broken down into:

• Duties imposed on providers of AI systems
• Duties imposed on deployers of AI systems
• Right and duties of supervisory authorities.

Below I will discuss each of these elements briefly.

Preamble

Under recital 73 of the AI Act, “High-risk AI systems should be designed and developed in such a way that natural persons can oversee their functioning….” Based on this principle, we can identify certain specific functionalities of AI systems furthering the demand for human oversight.

First, where appropriate, AI systems must have in-built constraints which the system cannot override and which guarantee that the system responds to human intervention.

Second, where appropriate, AI systems should contain mechanisms to guide and inform the individuals entrusted with human oversight, so that they can take informed decisions on whether, when and how to intervene to avoid negative consequences or risks, or to stop the system if it is not performing as intended.

These functionalities should exist “where appropriate,” which suggest that it was not the lawmakers’ intention to force providers of every high-risk AI system to introduce these functionalities.

Third, in the case of remote biometric identification systems, there is an enhanced human oversight requirement prohibiting actions or decisions to be taken on the basis of identification resulting from the system unless the identification is separately verified and confirmed by at least two natural persons.

Obligations of AI system providers

The key provision of the AI Act on supervision of AI systems is undoubtedly Art. 14, establishing the aims, basic elements, and requirements for oversight.

Addressee of obligations: It is clear from Art. 16(a) that it is the duty of providers of high-risk AI systems to ensure that their systems are compliant with the oversight requirements in Art. 14. Thus, practical implementation of the oversight requirement will fall mainly on the system provider, as its decisions on the system’s architecture and functionalities will determine how deployers of the system oversee its operation. It should be pointed out in this context that Art. 25(1) provides for the possibility of reclassifying another entity as a provider of an AI system.

It should be stressed that the obligations under Art. 14 apply only to high-risk AI systems, and thus only to certain types of AI systems. Nor do these obligations apply to AI models.

Person conducting oversight: It is clear from Art. 14(1) that oversight of high-risk AI systems must be conducted by natural persons.

Purpose of oversight: Under Art. 14(2), the aim of human oversight is to prevent or minimise risks to health, safety or fundamental rights.

Commensurate and proportionate: Art. 14(3) appears to state the principle of the proportionality of oversight: “The oversight measures shall be commensurate with the risks, level of autonomy and context of use of the high-risk AI system.” Similarly, Art. 14(4), in listing the elements comprising oversight, expressly states that oversight shall be “appropriate and proportionate.”

Method of oversight: Under Art. 14(3), oversight must employ (i) measures identified and built into the system by the provider, and/or (ii) measures identified by the provider, but to be implemented by the deployer of the system.

Elements of oversight: Art. 14(4) sets forth a list of the elements making up human oversight, which will be crucial for the rest of the commentary below:

1. Properly understanding the relevant capacities and limitations of the AI system
2. Duly monitoring the AI system’s operation
3. Detecting and addressing anomalies, dysfunctions and unexpected performance
4. Awareness of the possible tendency of automatically relying or over-relying on the output produced by the AI system (known as “automation bias”)
5. Correctly interpreting the AI system’s output
6. Deciding in any particular situation not to use the AI system, or otherwise disregarding, overriding or reversing the output of the AI system
7. Intervening in the operation of the AI system or interrupting the system.

Special case: Under Art. 14(5), for remote biometric identification systems, there is an express requirement that before an action or decision is taken on the basis of an identification from the system, the identification must be verified by at least two natural persons with the necessary competence, training and authority.

Oversight in the narrow sense under Art. 14 of the AI Act is closely connected with other obligations on the part of providers, involving oversight in a broader sense. The most important of these include:

• Establishing a risk management system with respect to the AI system
• Introducing a quality management system
• Maintaining documentation concerning the AI system
• Keeping the logs of events automatically generated by the AI system
• Subjecting the AI system to the relevant conformity assessment procedure prior to placing it on the market
• Monitoring the AI system after placement on the market
• Obligations involving the transparency of AI systems.

These broader oversight obligations also basically apply only to high-risk AI systems.

Obligations of AI system deployers

The way that the duties of entities using AI systems are framed in the AI Act is far from ideal. On one hand, there is a clear obligation for these entities to ensure that the people entrusted with oversight of AI systems have the relevant competencies. But on the other hand, there is no provision imposing a duty to apply human oversight. There is only a duty for deployers to use AI systems in compliance with the provider’s instructions (which under the AI Act must also contain instructions on exercising oversight). It should be underlined in this respect that system providers are only required to construct AI systems so that they ensure the possibility of exercising oversight. The act does not expressly require systems to be designed in a manner disabling their operation without supervision.

So at first glance it may seem that although many provisions of the act discuss human oversight, the AI Act lacks an express obligation to deploy AI systems under human supervision. It does impose on providers an obligation to construct AI systems so that human oversight is possible during the time when the systems are used. But a duty to apply oversight when deploying AI systems is not directly imposed on deployers. This should be regarded as an essential defect of the AI Act.

This raises the fundamental question whether deployers of AI systems are formally bound at all by the duty to ensure human supervision over these systems. Oversight at the stage of use of AI systems seems vital for achieving the aims of oversight. Lack of an obligation in this respect would basically defeat the purpose of the calls for human oversight.

As indicated, in literal terms, deployers are only required to use AI systems “in accordance with the instructions for use accompanying the systems” (Art. 26(1)) and to “assign human oversight to natural persons who have the necessary competence, training and authority, as well as the necessary support” (Art. 26(2)).

The doubt may arise whether a duty to use the AI system under human supervision can be inferred from the duty to use the system according to the instructions. As the instructions are created by the provider, they might (e.g. through the provider’s neglect) not contain provisions on human oversight. In that case, should it be found that the deployer of the AI system has no duty to exercise human oversight?

In turn, the duty to assign oversight to appropriately qualified individuals, if read literally, may not determine whether or when human oversight must be ensured—it only says whom such oversight should be entrusted to, whenever oversight is exercised. If the drafters’ intention was really to impose on AI system deployers a duty of human oversight, then, in addition to the provisions cited above, the AI Act should also contain an additional provision such as “Deployers shall ensure that during the period of deployment of high-risk AI systems, such systems are effectively overseen by natural persons.”

The lack of such a provision may suggest that deployers of AI systems have far-reaching discretion in deciding whether and when to deploy AI systems under human supervision, including the option of not exercising such oversight. This interpretation could gain additional support from Art. 26(3), which refers to “the deployer’s freedom to organise its own resources and activities for the purpose of implementing the human oversight measures indicated by the provider,” and Art. 14(5), which, as an exception, requires human verification by deployers of certain types of AI systems (remote biometric identification). Under that interpretation, the human oversight provisions in the AI Act would be triggered (apart from remote biometric identification systems) only when the deployer takes a voluntary decision to apply human oversight. The absence of such a decision would not expose the deployer to the allegation of violating the AI Act.

In effect, the issue of human oversight, which is so important for systemic and axiological reasons, would be left to the arbitrary decision of the deployers of AI systems. Thus implementation of the demand for human oversight would end at the stage of introduction of the system onto the market (the provider will still need to ensure the possibility of exercising oversight). At further stages in the lifecycle of the system, implementation of this demand would depend on discretionary decisions by deployers.

But in my own view this interpretation would be erroneous. Despite the evident imperfections of the act in framing the oversight obligations of AI system deployers, it should be recognised that deployers of high-risk AI systems do have a duty to ensure human oversight of the system. The deployer could not arbitrarily decide not to exercise such oversight.

This interpretation is driven by the context of the human oversight provisions of the AI Act. The drafters’ aim for human oversight is quite clearly revealed in the preamble. Under recital 66, “Requirements should apply to high-risk AI systems as regards … human oversight…. Those requirements are necessary to effectively mitigate the risks for health, safety and fundamental rights.”

Similarly, under recital 73, already mentioned, “High-risk AI systems should be designed and developed in such a way that natural persons can oversee their functioning, ensure that they are used as intended and that their impacts are addressed over the system’s lifecycle.”

The provisions within the body of the AI Act should be interpreted in light of these stated intentions, according to which, in the view of the EU lawmakers, ensuring human oversight is a condition for an appropriate level of safety of AI systems. This approach cannot be reconciled with the view that whether or not human oversight is applied could depend on an arbitrary decision of the deployer.

In the specific provisions of the AI Act, we should point first and foremost to Art. 26(1), under which “Deployers of high-risk AI systems shall take appropriate technical and organisational measures to ensure they use such systems in accordance with the instructions for use accompanying the systems….” According to Art. 13(3)(d), such instructions must set forth the human oversight measures put in place by the provider. Thus, a duty to use the oversight measures foreseen by the provider when deploying the system may be inferred indirectly from Art. 26(1). This is because the requirement to deploy AI systems according to the instructions includes the duty to apply the oversight measures indicated in the instructions. Thus Art. 26(3) should be read in this context not as a provision giving the deployer the discretion to abandon human oversight, but as giving the deployer flexibility in the manner of implementing the measures foreseen by the provider of the AI system.

Similarly, Art. 26(2), although poorly worded, should be read in this context as a provision imposing on deployers an obligation to implement human oversight measures, by assigning such oversight to natural person holding the qualifications referred to in that provision.

An additional argument for this interpretation is provided by Art. 27, which imposes on certain deployers a duty to perform an assessment of the impact on fundamental rights which the use of such system may produce. A description of implementation of the oversight measures is a mandatory element of the assessment. This provision seems to assume from the outset that applying human oversight measures is a duty of the deployer, while preparing a description of the implementation of human oversight measures is an additional obligation under Art. 27 addressed solely to selected deployers.

A natural person assigned by the deployer to exercise oversight of an AI system must meet certain requirements under Art. 26(2) of the AI Act, i.e. must have:

• Necessary competence (under recital 91, “in particular an adequate level of AI literacy”)
• Training
• Authority
• Necessary support.

Closely connected with human oversight are the other obligations of deployers involving oversight in the broader sense, the most important of which are:

• Duties involving ensuring adequate quality of the input data
• The duty to maintain event logs generated by the AI system
• Duties to monitor AI systems after placement on the market.

Finally, it should be clearly indicated that the duties described above do not apply to individuals using AI systems for purely personal, non-professional activity. This follows directly from the exclusion in Art. 2(10). Consequently, natural persons using high-risk AI systems for private purposes will not need to ensure any oversight of such systems. But when the same systems are used in commercial activity, they will have to have adequate human oversight in place.

Oversight by supervisory authorities

The obligations of providers and deployers of AI systems involving human oversight in the broad sense are supplemented by the powers vested in the administrative authorities. Most relevant in this context is Art. 79 of the AI Act, which empowers the national market surveillance authority to order withdrawal of an AI system from the market if its assessment finds that a system presenting risks to the health, safety, or fundamental rights of persons does not comply with the requirements of the AI Act. It should also be pointed that, at least literally, this authority applies to any AI system, not just high-risk AI systems.

What does human oversight mean?

I propose to begin a more detailed analysis of the requirements of the AI Act with the fundamental question of what the requirement of human oversight actually means. The act clearly indicates that oversight of AI systems must be exercised by natural persons. This means that the oversight process must be administered so that the human has an understanding and awareness of how the AI system operates, and the results of its operations. Moreover, the oversight system must enable a natural person to apply certain measures to the AI system the person is overseeing.

An adequate level of awareness on the part of the individual exercising oversight should guarantee that the oversight is real, not illusory. The primary aim of entrusting oversight to a human is for the processes executed by AI systems to be filtered through the prism of human awareness, as only then is there a chance of properly grounding these processes in a humanist context (i.e. taking into account the nuances of human dignity, fundamental rights and freedoms, and the broader cultural and social context).

Undoubtedly, enforcing this requirement will mean that the effective operation of many AI systems may be reduced, or perhaps even grind to a halt, as ensuring an adequate level of human awareness on the part of the person exercising oversight may slow the operating speed of AI systems.

But this does not mean that humans overseeing AI systems cannot themselves use other machine-based systems. Machines can support humans in the oversight process (e.g. by delivering relevant information on the processes executed by the systems they are overseeing), but cannot replace humans entirely. The limit to the use of machine systems in oversight is the existence of awareness of the operation of the AI system on the part of the person overseeing the AI system. Machine solutions supporting human oversight cannot replace or exclude this human awareness.

Does the AI Act impose a specific oversight model?

Another issue that deserves some attention is whether the AI Act imposes a specific model of oversight on deployers of AI systems. The provisions seem to expressly force the use of a specific oversight model (in this case the HITL model) only with respect to oversight of remote biometric identification systems (Art. 14(5)). In the case of such systems, active verification by individuals in the decision-making process must be ensured (in other words, in this case the decision-making process cannot be fully automated). To answer the question of whether other provisions of the act also impose a specific oversight model, we should first analyse the oversight elements indicated in Art. 14(4). In this context, Art. 14(4)(e) and Art. 14(4)(d) appear particularly relevant.

Art. 14(4)(e) requires AI system providers to enable the individuals overseeing the system to intervene in the operation of the system or interrupt the system. Ensuring these functionalities need not entail implementation of the HITL model (active human participation in the process executed by the AI system is not necessary to intervene in or interrupt the system’s operation). It appears the HOTL model would suffice in this case, but it is unclear whether an exclusively HIC model would suffice.

In other words, this provision does not expressly determine whether oversight of specific processes executed by the AI system is a condition for fulfilment of these oversight requirements. Adoption of the HOTL model is certainly the safest solution, but I would not necessarily rule out the ability to meet this requirement via the HIC model—particularly if we assume that the aim is not to prevent the occurrence of results of operation of the system at the level of a specific process, but only to ensure the possibility of correcting defective operation of the system for the future. With this approach, oversight of specific processes is generally not necessary. The key is knowledge of the general method of operation of the system, its parameters, trends, etc. For this, the HIC model may prove sufficient.

But recognising that the HOTL model will generally enable fulfilment of the requirements under Art. 14(4)(e) does not resolve all of the relevant issues that may arise in practice in defining the specific oversight functionalities. In practice, supervision of the ongoing operation of the system, which is the essence of the HOTL model, can be conducted in various ways. In particular, it should also be resolved at what moment the oversight should occur, and at what level of detail.

With respect to the timing of oversight, it seems that we should distinguish between three basic possibilities: ex ante oversight, oversight in real time, and ex post oversight.

With ex ante oversight, the natural person exercising oversight becomes aware of the process executed by the AI system (and thus can understand its meaning and context) before the effects of operation of the AI system occur. But the requirement that the oversight occur before the effects occur does not mean that the oversight is to be performed before the AI system generates its output. To the contrary, to fully understand the significance of a process executed by an AI, it may also be necessary to examine the output generated by the system. What is crucial is that the oversight is performed before the output is translated into concrete action impacting reality (e.g. before a decision is taken based on the generated output). In theory, this can prevent the occurrence of certain effects of operation of AI systems.

I would use the notion of real-time oversight to identify a model of HOTL oversight that creates the possibility of developing an awareness of the operation of the AI system at the time that the output generated by the system is translated into a concrete action impacting reality, but without the option of preventing occurrence of the system’s impact before it occurs.

By contrast, ex post oversight refers to an HOTL model where an awareness of the impacts of the AI system’s operations is developed only after the effects have already occurred.

I would break down the level of detail in the oversight into two basic methods of exercising oversight in the HOTL model: oversight of all specific processes executed by the AI system, and oversight of selected processes. In the first model, the person exercising oversight is aware of every single process carried out by the system, and thus for example grasps the substance, significance and context of each automated credit decision taken by the AI system. In the second model, the oversight person is aware only of processes chosen for detailed analysis (for example, where the rules introduced into the system for classification of alarm notifications single out for detailed human analysis only notifications falling into defined parameters). This obviously poses the open question of the rules for picking specific processes for detailed analysis.

In this light, it should be clear that the requirements in Art. 14(4)(e) will be achieved by ex ante oversight and by oversight of all individual processes executed by the system. In that case, the possibility of intervening in the system’s operation and interrupting its operation should, as a rule, always be ensured. This approach also enables the fullest achievement of the aims of this requirement. This is because ex ante oversight and oversight over each process makes it possible to avoid the harms which the AI system could potential exert through the operation of each of its processes.

But in my view, this does not mean that ex ante HOTL oversight and oversight of each individual process is the only model that can comply with Art. 14(4)(e). Nor can ex post oversight or oversight of only selected processes be automatically rejected. The regulation does not directly state that this approach would be contrary to the aim. As I indicated earlier, the wording of this regulation appears to allow the interpretation that the aim is not necessarily to ensure the possibility of intervening in every single process carried out using the AI system (in particular, intervening before the effects of operation of the AI system occur), but only to ensure the possibility of correcting faulty operation of the system for the future (including halting the system). Under this interpretation, we can imagine that this requirement could be met even using oversight in the HIC model—not to mention implementation through the HOTL model in the variant of ex post oversight or oversight of selected processes.

Implementing the requirement under Art. 14(4)(d) should be approached somewhat differently. Under this provision, high-risk AI systems must be designed to allow the oversight individuals “to decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output of the high-risk AI system.” This wording may raise the doubt whether the requirement to ensure the possibility of not using the AI system in a particular situation, and the requirement to “disregard, override or reverse the output of the high-risk AI system,” are two different requirements, or one and the same requirement, which may be achieved in one of the manners mentioned in this provision.

A finding that ensuring the ability to “disregard, override or reverse the output” is a standalone requirement (i.e. that it applies regardless of whether the system allows the system not to be used in a particular situation) would have far-reaching implications for the entire AI sector in Europe. This is because in practice, it means a ban on creating AI systems whose outputs are irreversible. Meanwhile, many high-risk AI systems can generate such irreversible outputs (for example, systems involving critical infrastructure, systems for classification of alarm notifications, and systems that also constitute medical devices).

It seems that the correct understanding of this obligation should primarily take into account the aim of this aspect of oversight, which is to create effective mechanisms for avoiding undesirable impacts from the operation of AI systems. This can be achieved either by ensuring that the AI system is not used in certain situations (particularly when the potential impacts of the system are irreversible), or by ensuring the ability to “disregard, override or reverse the output” of the AI system (if the outputs can be disregarded or reversed).

This implies that if the nature of the outputs of the AI system prevents them from be reversed, overridden or disregarded in some other way, the only solution enabling compliance with the requirements of this provision is to ensure that before impacts occur, a decision can be taken not to use the system. Proper compliance with these requirements therefore calls for at least the HOTL model, in the most rigorous version (i.e. with ex ante oversight as well as oversight of all individual processes). Without such oversight, the person performing the oversight will not be in a position to identify an instance that would require a decision not to use the AI system. An HIC model, involving only high-level oversight of the AI system, cannot ensure that this requirement is met. Similarly, a looser version of the HOTL model (with real-time or ex post oversight, or only oversight of selected processes) will also not prevent the occurrence of undesirable, irreversible effects of the AI system operation, with respect to any specific process executed using the AI system.

We could visualise an example where an AI system monitors students in real time as they take an exam, looking for prohibited behaviours by the students. Assume that there are several thousand students taking the exam at the same time. The system could take a fully automated decision to remove a student from the exam because the system discovers misbehaviour by the student. The student removed from the exam loses the chance to take the exam, and in this sense the decision by the system is irreversible.

Proper implementation of oversight of this AI system would involve the person overseeing the system being aware of every action by the AI system leading to a decision to remove a student from the exam, before that decision is taken. The oversight should be arranged so that the person performing the oversight has enough time to examine the specific instance and then decide whether to use or not use the AI system, or confirm the decision to remove the student from the exam by allowing the AI system to perform its function in this instance.

Exercising oversight in this manner in the example given appears realistic. But it should be acknowledged that in the case of other AI systems, ensuring ex ante oversight at the level of a specific process causing irreversible impacts may prove very difficult (e.g. with respect to systems managing critical infrastructure).

In a case where the AI system provides the ability to disregard, override or reverse the effects of its operation, it seems that it would be permissible to use an ex post variety of the HOTL oversight model, limited to selected processes (e.g. limited to results challenged by the addressee of the decision). It also seems that best practice in compliance with respect to such systems would also involve periodic testing of selected results, whether or not they were challenged.

It should also be pointed out that the requirements for a given model of human oversight need not be based solely on Art. 14 of the AI Act. They can also be inferred for example from Art. 5(1)(d), prohibiting certain practices for assessing risks of commission of criminal offences based on profiling. This provision also indicates the conditions that will make it permissible to use such systems. One condition is where AI systems are used to support human assessments. Therefore, the permissibility of using AI systems to assess the risk of involvement in criminal activity will, in practice, require implementation of the HITL model, where a human is actively involved in the risk assessment process.

To summarise, it appears that the AI Act does not impose a general duty to apply the HITL model of oversight. This model is reserved solely for specific instances (e.g. systems of remote biometric identification and systems for assessing the risk of criminal behaviour). But this does not exclude the possible existence of cases where, in light of the risks generated by the system, HITL will be the only model which, in the system provider’s view, will ensure performance of the desired human oversight. In other cases, applying the HOTL model appears to be a safe approach. In this respect, Art. 14(4)(d) of the act should be borne in mind, suggesting that at least with respect to systems generating irreversible outputs, oversight should be robust enough to allow the person performing the oversight to assess ex ante each individual process executed using the AI system.

It should also be explained that the determination of whether achieving a given requirement forces the use of the HITL or HOTL model does not necessarily mean that the system should not also be monitored in an aggregated or high-level sense. As a rule, oversight in the HIC model should always be present. The differences between AI systems in this respect boil down to whether HIC oversight must also be supplemented by most robust oversight in the HITL or HOTL model.

How to interpret the principle of proportionality?

These observations lead to the conclusion that the AI Act can be interpreted as setting a standard for oversight of high-risk AI systems—a standard that includes the need for oversight to be performed by a natural person and the need to ensure that such person will monitor the ongoing work of the system, in certain instances via the model of ex ante oversight and at the level of the individual processes executed by the AI system.

Apart from the requirements discussed above, the AI Act also includes express provisions establishing the principle of appropriate and proportionate application. This raises the question of whether these principles can in any way modify the standards for oversight of AI systems.

I assume that the use of both “appropriate” and “proportionate” in Art. 14(4) is intentional. I understand “proportionality” to mean the degree of intensity of any oversight measure, tailored to the risk, the level of autonomy, and the context of the AI system in question, while “appropriateness” means adjusting the solutions within the oversight system to the nature of the AI system in question and the processes it executes. Inclusion of these principles in the AI Act should be taken as a clear signal that the elements of oversight indicated in Art. 14 can be modified. The modification might consist of different intensity in applying these measures (for example, systems posing small risks might be monitored only within certain time intervals, while systems with far-reaching autonomy whose outputs could significantly impact reality could require constant monitoring in real time). It might also involve modification of the indicated means in connection with the peculiarities of the given AI system (which might for example not make it necessary to ensure the possibility of deciding not to use the AI system because the nature of the system makes it easy to undo the effects of its operation).

At the same time, I would warn against too broadly interpreting the principle of “appropriateness,” particularly relying on this principle as a justification for failing to apply any of the oversight measures set forth in Art. 14 of the act. It is obvious that the ambitious requirements of Art. 14 will collide with the architecture and functionalities of many high-risk AI systems. But if the demand for human oversight is to have any real significance, this should rather lead to adjusting the architecture of the system to meet the requirements of the AI Act, than to relaxing the requirements of the act to suit the architecture of the system. This applies in particular to systems aimed at increasing efficiency by executing a huge number of processes within a short time. I wouldn’t rule out that at least in the case of certain of these systems, a decrease in the efficiency of the processes will be the price that must be paid to ensure real (and not fictitious) human oversight. I recognise that the hard line in modifications of oversight measures under the “appropriateness” principle would have to be drawn at the need to maintain the objectives of the specific oversight measures. This must mean, first and foremost, the ability to ensure actual human oversight, the ability for a human to halt the work of the system, and the possibility of preventing or reversing undesirable effects of the system’s operations.

How does this affect compliance?

To summarise the foregoing deliberations, while also framing practical conclusions for those charged with implementing oversight of AI in their organisations, I would stress the following aspects:

• As a rule, human oversight must be ensured only in the use of high-risk AI systems. In other cases, human oversight is voluntary.
• When oversight is mandatory, it must be assigned to an appropriately qualified individual. AI systems may be overseen by other machine-based systems only as an aid to oversight performed by a human.
• The standard should always be to ensure oversight via the HIC model, supplemented by the HOTL model, but in the case of systems generating irreversible outputs, HOTL oversight should be performed ex ante and should address each process executed using the AI system.
• Depending on the specifics of the AI system (the context of its operations, the risks it poses, and the level of autonomy), under the principles of “proportionality” and “appropriateness” it is possible to modify the standard for oversight by changing the intensity of the oversight and adjusting the specific model to the nature of the system and the outputs it generates.
• Remote biometric identification systems and systems for assessing the risk of commission of crimes by individuals require special attention. In the case of these systems, oversight in the HITL model is mandatory.
• AI systems that can generate irreversible outputs also require special attention. In the case of these systems, the most robust variants of the HOTL model are applicable.
• Before launching the use of an AI system, the risks from use of the system should be analysed, and it should be determined which oversight model will be optimal for that AI system.

Obviously, these conclusions are only one of the possible ways of interpreting the Artificial Intelligence Act. It is early days, and over time many of points raised here will need to be re-examined. This is also a good time to signal all of the ambiguities and interpretive doubts in the act. This is an opportunity for supervisory authorities to issue interpretations and guidance facilitating practical application of the provisions setting key oversight obligations, before they go into effect.

Krzysztof Wojdyło