8 min read
2026/08/24
In this article, Carles Vidal, Business Director of the Avallain Lab, explores the importance of human agency and oversight in AI-driven digital environments. He considers what these principles mean for educators, learners and edtech providers and how they are reflected in the current AI regulation. He also examines why human judgement, choice and control should remain essential to designing technology as AI takes on a greater role in education.
Preserving Human Agency in AI-Driven Digital Education Environments
From regulation to product design: keeping users in control
Author: Carles Vidal, MSc in Digital Education, Business Director of the Avallain Lab
St. Gallen, August 24, 2026 – One direct consequence of the current integration of artificial intelligence (AI) in educational software is the surge of AI-driven features designed to assist teachers, students and administrators across a range of key educational tasks. These tools are used to generate personalised content, automate feedback and assessment, provide guidance and report on students’ performance, to name a few, all aimed at reducing workload and augmenting users’ productivity in digital education environments.
Educational institutions demand these kinds of capabilities, and edtech companies are delivering them in multiple ways. From an AI ethics perspective, however, implementing AI-based automation features may compromise core tenets of trustworthy AI: human agency and human oversight. In education, these principles mean keeping educators as the final pedagogical authority on the platform while ensuring students can make informed decisions throughout their learning journey. The challenge for the sector is to design tools that fully preserve human agency while meeting the demand for greater AI-driven system autonomy.
Human Agency and Human Oversight
To understand what these principles mean, it is worth examining two relevant sources: the Council of Europe Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law (signed in 2024 and not yet ratified), which aims to set an international standard to protect fundamental rights throughout the AI lifecycle, and the European Union Artificial Intelligence Act (2024), which addresses these principles in the context of a binding legal and technical regulation within the EU market.
From a human rights perspective, the Council of Europe calls on states to protect human dignity, individual autonomy and transparency throughout the AI lifecycle: from its design to its implementation (articles 7 and 8). The framework establishes that individuals must retain self-determination, requiring adequate oversight so users can understand, oversee and challenge automated systems.
In turn, from a regulatory perspective, the EU AI Act recognises human agency and oversight in Recital 27 and Article 14 as key requirements for trustworthy AI. Drawing on the High-Level Expert Group’s Ethics Guidelines (2019), human agency implies that users should be able to make informed, autonomous decisions when interacting with AI systems, and critical to this concept is the right of the individuals not to be subject to a decision based solely on automated processing, when this produces legal effects on users or similarly significantly affects them. Article 14 translates human oversight into actionable regulation by establishing that high-risk AI systems must feature human-machine interface tools that allow persons to oversee, interpret and override automated outputs in real time.
Together, these frameworks define human agency as a key requirement to ensure the users’ right to be autonomous decision makers, and human oversight as a range of technical safeguards to achieve this end.
User Agency in Digital Educational Environments
In the context of educational software, it is important to keep in mind that the EU AI Act regulates AI-driven features, not traditional automation tools that have been on the market for years. Specifically, the Act focuses on ‘machine-based systems’ able to generate ‘predictions, content, recommendations, or decisions that can influence physical or virtual environments’ (Article 3.1). Therefore, common features like pre-defined adaptive pathways or assessment based on standard scoring fall outside the regulation, as they do not rely on AI.
For those edtech companies whose software, or a particular feature of it, falls under the scope of the EU AI Act, these principles have direct regulatory consequences. In this sense, Annex III of the EU AI Act establishes that AI systems that evaluate learning outcomes, assess the appropriate level of education or determine access to educational institutions are classified as high-risk, (except if they ‘do not pose significant risk of harm to the health, safety or fundamental rights of natural persons, including by not materially influencing the outcome of decision-making’ (Article 6.3)). When the exception does not apply, Article 14 requires software providers to build technical oversight controls directly into the user interface, such as review dashboards, score override options, output explainability tools and system stop controls so educators can validate or override AI suggestions.
At a practical level, this means that, for example, an AI feature able to decide if students progress to the next academic year would be classified as high-risk, since it directly determines a critical educational outcome for users. In this case, the software provider must implement oversight measures that allow educators/administrators to review, override or cancel the system’s decisions.
In contrast, an AI chatbot that suggests optional study resources to students, in accordance with the student’s performance, and that students can ignore, should not be considered high-risk, as it does not dictate any decision affecting those users. Therefore, no oversight measures would be required in this case, although even in lower-risk scenarios, user agency can also be supported by making AI interactions transparent or allowing users to disable recommendations.
These two examples illustrate clear risk classification cases and their corresponding mitigating strategies. In practice, following the same logic, software providers need to assess their AI-based features, determine their risk category and implement the appropriate oversight controls or transparency safeguards.
User Agency in Practice: From Principles to Product Design
How can these principles be promoted throughout the lifecycle of digital educational environments? In our research-based report, From the Ground Up (2025), a set of 12 core controls was identified as critical to ensure the ethical use of AI in education. Among these controls, the document recommends User Agency Preservation, which involves designing AI systems that empower teachers and students with choice and control, ensuring that AI in education does not undermine student autonomy or teachers’ professional judgement.
This principle operates at two levels:
- For educators: Ensuring AI acts as a supportive assistant rather than an autocratic system, preserving the educator’s professional decision-making authority.
- For students: Providing meaningful options to shape their learning process, such as setting personal goals, choosing between learning modalities or overriding AI recommendations when appropriate.
The degree to which students exercise their agency will depend on age and context; for younger learners, educators and guardians may carry out this responsibility by overseeing AI interactions on their behalf.
Avallain’s Approach
At Avallain, these principles form the foundation of Avallain Intelligence, our strategy for the responsible use of AI in education, and inform the design of AI features and tools across the product suite. AI-powered capabilities in Avallain Author and Avallain Magnet, alongside the TeacherMatic AI toolkit, are built to support publishers, institutions, content authors and educators with tasks such as drafting content, generating ideas, creating practice activities and improving productivity in content creation and planning. In all cases, AI outputs are positioned as a starting point rather than a finished product. Professional review, adaptation and validation remain essential steps before content reaches learners.
This design philosophy has direct consequences for risk classification. Avallain’s AI features are deliberately designed to preserve human agency and oversight, an approach that avoids high-risk categorisation under the EU AI Act. AI does not determine final grades, progression, certification or learner outcomes. It does not make autonomous educational decisions or assess learners without human involvement. By ensuring that AI outputs are always mediated or supervised by qualified professionals, the software supports assistance and safeguards decision-making.
Human agency is also fostered through transparency and user control. AI features are clearly presented as assistive tools, not as authoritative sources. Users are informed when AI is being used and are encouraged to understand that AI-generated output is probabilistic, may contain errors and requires review.
Conclusion
The integration of AI in educational software presents real opportunities to reduce workload and support productivity across publishers and educational institutions. However, these benefits must not come at the expense of human agency and oversight: the authority of educators to make final pedagogical decisions and the right of learners to remain autonomous actors in their own education.
International frameworks and European regulation recognise and implement these principles, and responsible product design must respond accordingly. The challenge for edtech providers is to harness the potential that AI systems can deliver while ensuring that educators and learners remain at the centre of every decision. At Avallain, this is the approach we have chosen: AI as a transparent, controllable support layer, always subject to human judgement.
About Avallain
For more than two decades, Avallain has enabled publishers, institutions and educators to create and deliver world-class digital education products and programmes. Our award-winning solutions include Avallain Author, an AI-powered authoring tool, Avallain Magnet, a peerless LMS with integrated AI, and TeacherMatic, a ready-to-use AI toolkit created for and refined by educators.
Our technology meets the highest standards with accessibility and human-centred design at its core. Through Avallain Intelligence, our framework for the responsible use of AI in education, we empower our clients to unlock AI’s full potential, applied ethically and safely. Avallain is ISO/IEC 27001:2022 and SOC 2 Type 2 certified and a participant in the United Nations Global Compact.
Find out more at avallain.com
_
Contact:
Daniel Seuling
VP Client Relations & Marketing
