AI-powered language generators in vocational final examinations – information and discussion urgently needed!

Artificial intelligence (AI) for text generation

AI-based language generators such as ChatGPT act as virtual conversation partners. Based on their training with extensive text databases, they generate answers and dialogues in natural language. By using AI and machine learning, they recognise patterns in large amounts of data and use probabilities to generate the best possible answer based on context and learned knowledge (cf. Ouyang et al. 2022; Heaven 2023¹).

Chatbots such as ChatGPT (Generative Pre-Trained Transformer) can not only efficiently create, edit, summarise and evaluate texts, but also write program codes, for example. With the upgrade of the underlying language model GPT 3.5 to 4.0 in March 2023, performance and accuracy have been improved and, in addition to text, handwritten documents and images can now also be used as input (cf. Hahn 2023²).

However, language generators also have weaknesses: problems with longer conversations, logic, factuality and timeliness, as well as risks due to a lack of transparency, security and confidentiality of data (cf. Albrecht 2023; BSI 2023).

According to a survey commissioned by the digital association Bitkom, 70 percent of companies surveyed expect that AI for text generation will be part of everyday working life in the future and that more knowledge about AI must therefore be taught in schools and training.³ In April 2023, the Education Committee of the German Bundestag also called for the acquisition of appropriate skills for dealing with AI.⁴ In the consultations on the EU’s Artificial Intelligence Act (AI Act), AI systems used in general or vocational education are expected to be classified as high-risk, among other things (cf. European Commission 2021). It requires AI-generated texts to be labelled as such. Digital watermarks to identify these texts are under development (cf. Glauner 2023⁵; Weßels 2023⁶).

AI-based language generators in education and examinations

According to experts, the impact of AI-based language generators on the education system will be immense, but difficult to estimate. AI offers great opportunities to support text creation and editing. Texts can be structured, summarised, simplified or corrected, saving time and energy. In addition to linguistic tasks, AI can perform other creative tasks such as generating IT programs, spatial concepts or pieces of music. Thus, on the one hand, AI can support teachers and examiners in creating teaching materials and tasks as well as in evaluating and documenting examinations. On the other hand, it can be used as a personal learning companion, for exam preparation, for first drafts and for overcoming writer’s block. It can also facilitate access to exams for people with impairments or learning difficulties.

However, learners should avoid “delegating” their own assignments to AI and limiting their own learning process. Instead, they should be sensitised to the critical use of AI-generated texts. The educational mandate for responsible and safe use of digital media also applies to text-generating AI applications (cf. Weßels 2023; MSB NRW 2023; Schepers 2023; Spannagel 2023).

In examinations, there is a danger that examinees may pass off AI-based texts as their own. It is reported from the USA that ChatGPT has already mastered exams in law and medicine (cf. Choi et al. 2023; Kung et al. 2023). In Hamburg, the first high school graduate has already been caught cheating with ChatGPT.⁷ Lively discussions are taking place in the general education and higher education sectors. While some want to completely ban ChatGPT and the like in exams and assignments, others consider such a ban unrealistic and uncontrollable. They advocate teaching a critical approach and the targeted use of AI in homework and exams (cf. Pinkwart/Paaßen/Burchardt 2023⁸).

In order to obtain fair and bias-free results in examinations with AI support, it is recommended to design tasks that are as complex as possible and individually tailored to the examination group. Ideally, the entire work and examination process should be accompanied and not only the result evaluated. For privacy reasons, examinees should not use their own devices and accounts. It is also advised to combine different examination formats and instruments and to increase the use of examination interviews and presentations. Some universities have already revised their examination regulations accordingly and issued recommendations to teaching and examination staff, as have some federal states (cf. Weßels 2023⁶; MSB NRW 2023; Spannagel 2023).

Vocational final examinations

One advantage of final examinations in recognised training occupations is the variety of examination instruments used (cf. BIBB Board 2013). Written assignments in final examinations are partly conducted digitally, but without internet access and not on personal devices (cf. Hollmann et al. 2023). Therefore, with the exception of wilful attempts at deception, the use of AI-assisted language generators by examinees is just as impossible here as in oral and practical tasks that are carried out under the supervision of examiners. In practical tasks carried out in the presence of examiners, they can prevent the misuse of AI. The situation is different for work tasks carried out in the company, such as company assignments or company project work in the IT occupations. Here, the training companies must certify to the examination board that the examinees have carried out the tasks independently. Whether a training company can reliably monitor the independence of the task preparation and certify it to the examination board must be discussed. A closer look at this issue and legal clarification seem necessary.

It should be noted that company tasks must be documented with practice-related documents that are intended to support the assessment but do not receive their own weighting for the examination result. For example, industrial clerks, office management clerks and wholesale and foreign trade management clerks prepare a three- to five-page report on the performance of a specialist task. This report itself does not contribute to the evaluation of the examination. However, based on the report, a case-related professional discussion is conducted during the examination, which is assessed. The task of the examination board is to assess whether the examinees have mastered the topic and meet the examination requirements (cf. BIBB Board 2013).

Outlook

The use of AI-controlled language generators is still in its early stages, but development is progressing fast. It is therefore important to have a timely and comprehensive discussion about their use in final VET examinations. In what form can and may AI support the creation and evaluation of examination tasks in the future? What strategies can be used to avoid misuse in examination situations? Do examination instruments, examination regulations and/or examination requirements need to be adapted? VET stakeholders should proactively address these questions and find common answers. If this technological development – like calculators and computers years ago – finds its way into everyday working life, then its responsible use should be taught in training and thus also become part of the examination content. Trainers and examiners must be informed and trained in time, and social partners, policymakers and competent bodies should create the necessary framework conditions.

Barbara Schürger
Academic researcher at BIBB

Translation from the German original (published in BWP 3/2023): Martin Kelsey, GlobalSprachTeam, Berlin