(PDF) A Framework for Automatic Exam Generation based on Intended Learning Outcomes

A Framework for Automatic Exam Generation based on Intended Learning Outcomes Ashraf Amria1, Ahmed Ewais2,3 and Rami Hodrob2 1Deanship of Admission, Registration & Examinations, Al- Quds Open University, Jenin, Palestine 2 Department of Computer Science, Arab American University, Jenin, Palestine 3Web & Information Systems Engineering (WISE) Laboratory, Vrije Universiteit Brussel, Brussels, Belgium Keywords: Bloom’s Taxonomy, Learning Outcomes, Automatic Exam Generator, Assessment. Abstract: Assessment plays important role in learning process in higher education institutions. However, poorly designed exams can fail to achieve the intended learning outcomes of a specific course, which can also have a bad impact on the programs and educational institutes. One of the possible solutions is to standardize the exams based on educational taxonomies. However, this is not an easy process for educators. With the recent technologies, the assessment approaches have been improved by automatically generating exams based on educational taxonomies. This paper presents a framework that allow educators to map questions to intended learning outcomes based on Bloom’s taxonomy. Furthermore, it elaborates on the principles and requirements for generating exams automatically. It also report on a prototype implementation of an authoring tool for generating exams to evaluate the achievements of intended learning outcomes. 1 INTRODUCTION to evaluate the effectiveness of the learning process (Manuel Azevedo et al., 2017). One of the means to measure the impact and the One way to guarantee a correct measurement of output of learning process in schools is the use of the intended learning outcome (ILO) of a specific assessment techniques. In general, assessment plays course module in higher education institutions is to an important role in supporting the learning process provide proper questions that effectively measure the of the students. This support is achieved by evaluating intended learning outcome in the conducted exams, the students’ results and answers using some exercises, quizzes, etc. An approach for realizing automatic tools. This will provide stakeholders good such an effective assessment tools is to relate learning vision and overview of the learning process. outcomes with both learning topics and questions Recently, the era of education is complemented by related to each learning topic. For instance, in effective utilization of technology For instance, (Blumberg, 2009) an approach for maximizing the developing learning materials using different learning process by aligning learning objectives, applications, and using Virtual Reality and learning materials, activities, and course assessment Augmented Reality is used in many different with Blooms’ taxonomy is proposed. The alignment domains. Furthermore, distance learning and e- is done using action verbs of the different levels of learning are also good examples of the use of recent cognitive process. Other researchers (Tofade et al., technology. In many domains, learners can get 2013) proposed some best practices for using certificates from higher education institution using questions in course modules. Among the proposed Massive Open Online Courses (MOOCs) without practices of using educational taxonomies is that of being limited to the place and time. Providing the use of Bloom’s taxonomy to define different certificates can be based on evaluating student’s levels of questions. achievements after following the online course. In general, an educational taxonomy is used to Therefore, electronic exams have been used in a wide describe the learning outcomes using the courses range of domains to measure the effectiveness of syllabi. Furthermore, educational taxonomies can be learning process. For this purpose, researchers used to provide an overview about the different level proposed different approaches for generating exams of understanding about specific learning concepts and topics. Another important aspect for the use of 474 Amria, A., Ewais, A. and Hodrob, R. A Framework for Automatic Exam Generation based on Intended Learning Outcomes. DOI: 10.5220/0006795104740480 In Proceedings of the 10th International Conference on Computer Supported Education (CSEDU 2018), pages 474-480 ISBN: 978-989-758-291-2 Copyright c 2019 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved A Framework for Automatic Exam Generation based on Intended Learning Outcomes educational taxonomies in the learning process is to quizzes, exercises, and homework using Bloom’s identify the level of exams’ questions depending on taxonomy. Furthermore, the proposed approach cognitive levels. For instance, course exams should divides the total mark of the exam over the selected include questions that asses different level of learning questions in the exam based on predefined criteria. effectively. This paper is structured as follow. The next Based on educational and pedagogical theories, section presents a number of existing tools that are researchers proposed different taxonomies to help proposed to generate exams automatically. Then, the educators in developing learning resources, assess- proposed approach to generate examination ments, and learning outcomes. Among the proposed automatically is discussed. Furthermore, a list of taxonomies is the Bloom’s taxonomy (Bloom, 1956) requirements and the conceptual framework are also and its revised version (Krathwohl, 2002). It is mainly presented. Next, the implementation and the based on six levels of the cognitive learning process: developed prototype are discussed. Finally, the paper Remember, Understand, Apply, Analyze, Evaluate, is concluded and future directions are presented. and Create. Furthermore, a list of different action verbs has been identified to describe the intended learning outcomes of a course. The revised version of the 2 LITERATURE REVIEW Bloom’s taxonomy is mainly mapping cognitive dimensions with the knowledge dimensions. Another This section reviews related work dealing with taxonomy is the so-called SOLO taxonomy (Biggs & generating exams out of question bank automatically. Collis, 1982) which has the levels: Prestructural, There are different attempts conducted to Unistructural, Multistructural, Relational, and consider the Bloom’s taxonomy for generating exams Extended Abstract, and which are not only restricted to automatically. For instance, the work presented in cognitive aspects but also deal with knowledge and (Kale & Kiwelekar, 2013) considers four constraints skills. More educational taxonomies that are used in the to generate the exams. The constraints are proper assessment and evaluation are reviewed in (Fuller et coverage of units from course’s syllabus, coverage of al., 2007; Ala-Mutka, 2005). difficulty levels of the questions, coverage of cognitive In general, there are three types of exam levels of Bloom's taxonomy and the distribution of the generation approaches (Cen et al., 2010). The first marks across questions. Such constraints are type is related to offering a question repository that considered for developed algorithm to generate the can be explored by educators to select the questions final paper exam. Another interesting work for for a specific exam. This type is almost similar to the classifying questions according to Bloom’s taxonomy manual creation of the exam. However, the educators is presented in (Omar et al., 2012). The proposed work can inspect the stored questions in the database by is a rule-based approach. However, the generation means of a user interface. The second type is related process of exams is not considered in this work. to generating the exam based on random selection of Other approaches are related to the use of Natural the questions. The third type is related to generating Language Processing (NLP) to classify questions and the exams by means of AI algorithms for realizing assign weight for each question. For instance, authors predefined rules to provide the exam. in (Jayakodi et al., 2016) shows promising results in Normally, identifying simple or difficult using NLP techniques to weight questions according questions is mainly depending on the educators’ to Bloom’s taxonomy cognitive levels. Other intuition and experience. Furthermore, similar researchers (Mohandas et al., 2015) propose the use questions or repetition of questions can happen in of Fuzzy logic algorithm for the selection process of manual created exams. Another possible drawback is the questions depending on difficulty level. related to careless division of the total mark of the Different tools were developed to validate the exam over the composed questions. Finally, manual proposed approaches in the context of automatic preparation of exams with the alignment of questions exam generation. For instance, (Cen et al., 2010) and learning outcomes requires a high mental presented a tool using J2EE tools to support educators demand. Given the previous drawbacks, there is a by identifying the subject, questions types, and possibility of having poorly designed assessments difficulty level. Accordingly, the proposed prototype which can lead to unsatisfactory competing rate of the will generate the exam in MS document format. The intended learning outcomes of the course. For the proposed work does not map questions to the course previous obstacles, we propose a systematic approach syllabus and Bloom’s taxonomy. Other researchers to diminish such drawbacks. The proposed approach (Gangar et al., 2017) proposes a tool which is used for generating automatically course exams, categorizes questions as knowledge-based, memory- 475 CSEDU 2018 - 10th International Conference on Computer Supported Education based, logic-based, or application-based. The work developing of an automatic exam generator. The uses a randomization algorithm for selecting requirements are as follow: questions from the question bank database. Question Variety: this requirement is mainly Furthermore, exams can be generated only for unit considered to provide different types of questions exams or final semester exams. mapped to an ILO. This is achievable by providing More comprehensive review of proposed different types of questions, both subjective and approaches and tools for generating exams objective questions, e.g., essay questions, multiple automatically are presented in (Joshi et al., 2016; choice, true/false, match column, multimedia Tofade et al., 2013; Taqi & Ali, 2016). questions, fill in blank, etc. that are related to a specific learning concept or topic. Randomization: this requirement is used to guaran- 3 AUTOMATIC EXAM tee that the generated exam does not have repeated or GENERATION APPROACH biased questions. It can be realized by means of random algorithms (Marzukhi & Sulaiman, 2014). Considering the different obstacles and challenges Educational Taxonomy Mapping: this requirement related to the assessments in a course module, the is considered to map a learning outcome to both a proposed approach provides a platform for selecting question and a learning topic. This will enable the questions depending on ILOs and distributing marks educators to know the covered ILOs in each exam. based on specific criteria. The proposed approach for Theretofore, the revised version of Bloom’s taxonomy generating the exam is mainly based on Bloom’s is considered in this research work. taxonomy. This enables the system to standardize the Marks Distribution: there is a need to consider a assessment of any course to a great extent. This is fair distribution of the exam total mark over the achieved by assigning the learning topic (contents), composed questions. One way to achieve this is to use which can be a section of a chapter in a specific educators’ experience to give score for each question textbook, a video, or audio to corresponding ILO. manually. Other approach uses algorithms that Furthermore, also the questions related to each consider the ratio of required time to solve the learning topic are assigned to the corresponding ILO. question (defined by the educator) and the specified In the proposed approach, the educator is time for the exam in general (defined by the responsible for defining a question and map it to a educators). This is a simple approach for marks predefined ILO explicitly. The advantage of this distribution for different questions in the exam. approach is that it gives control to the educator. On ILO Validation: this requirement is mainly used the other hand, this can be a disadvantage in the way for validating the defined ILOs according to Bloom’s that it can take quite some time for the educator to do taxonomy. This is done by considering some the mapping process between the learning topics, keywords from a specific level (Remember, questions and the ILOs. However, supporting Understand, Apply, Analyze, Evaluate, Create). In educators with an appropriate and usable tool can other words, matching algorithms can be used to find overcome this issue. Also, the manual approach can the keyword from ILO and match it with a be complemented with classification algorithms to corresponding cognitive level from Bloom’s map topics and questions to related ILO automatically taxonomy. For instance, a defined ILO can be (Jayakodi et al., 2016). “explain the concept of object oriented The next sections presents the requirements for programming”. This ILO is related to the second generating exams based on ILOs. Then, a conceptual level of the revised Bloom’s taxonomy (Krathwohl, framework (models and principles) is presented. 2002), which is the understanding level. As a result, Finally, the algorithm for generating the exams and the validation algorithm starts searching for the action distributing grades is explained. verbs inside the statement of the defined ILO (“explain” in the given example) and map it to the 3.1 Requirements corresponding level of the Bloom’ taxonomy. Other requirements such as the security issues, Based on the reviewed literature (Mohandas et al., usability aspects like ease to use, and ease to 2015; Tofade et al., 2013; Alade & Omoruyi, 2014; understand, are also considered in this work partly. Joshi et al., 2016; Omar et al., 2012), a number of However, there is still a need for evaluating the requirements are derived to be considered in proposed prototype. 476 A Framework for Automatic Exam Generation based on Intended Learning Outcomes 3.2 Conceptual Framework It is important to mention that the generated exam can be an electronic or paper-based exam. Electronic In general, to be able to generate an exam by exams can be used for e-learning applications such as considering a number of parameters such as a number MOOCs where the questions can have multimedia of selected topics, selected ILOs, exam time, etc., contents such as animation, 3D models, simulation there is a need to maintain all required information in model, video, audio, etc. Therefore, the generated file different models. In this approach, generating an is an XML format attached with different multimedia exam depends on Course Model, User Profile, ILO resources. On the other hand, paper exams can be Model, Question bank, Generated Exam repository generated in two formats MS-Word document or PDF and the Generator Engine (see Figure 1). files for use in classical courses. The Generated Exam Repository: a repository of generated questions is used to store historical information of used questions in different exams, semesters, years, etc. Such information can be used by the educators to explore the previous generated exams. In general, generated exams types, which are considered in the generation process, are quizzes, exercises, first exam, second exam, midterm exam, and final exam. Such assessments are used in different universities for different programs such as Engineering, Science, Business, Medical, etc. The kernel of the framework is the Generator Engine which is responsible of realizing the creation Figure 1: Conceptual Model for Generating Exams of the different exams based on IF-ACTION rules. Automatically. The IF-part of the hard coded rules contains three parameters: learning topic, Bloom’s taxonomy level, The Course Model is used to describe the different and required time for solving the question. The topics that will be covered in the course. Each topic is ACTION-part of the rule uses a random algorithm for mapped to the related ILO (from the ILO Model). A selecting a question out of the filtered questions based topic can be related to only one ILO. However, an on the IF-part of the rule. Moreover, the generator ILO can be related to different topics in the same engine is responsible for calculating the marks of course with a specific percentage. each question in the exam depending on timing The User Profile is used to maintain the criteria. More details about the process of filtering, educator’s information such as his user name and selecting, grading questions are presented in the next password, taught courses, and created questions. section. The ILO Model, as mentioned earlier, allows to associate questions to the different predefined ILOs 3.3 Exam Generating Algorithm of the course and this is an important step to assess learning process depending on familiar standards To determine a question in an exam, the list of such as Bloom’s taxonomy. Therefore, a repository of learning topics, which will be evaluated, need to be ILOs is required to hold the information about each defined. This will narrow the possible questions that ILO such as Bloom’s taxonomy level, related course will be used for the generation process of the exam. name, related learning topic, covered percentage of The second level of categorization is related to the the ILO in the learning topic, and related questions. ILO that will be examined in the selected exam. This The Question Bank is required to map each will narrow the sample of the possible questions from question to learning topic. It is important to mention the previous step. Accordingly, the algorithm will that each ILO should be mapped to at least one start the selection of the question in a specific question since ILO can be evaluated by different sequence from the selected topics till the final topics questions types. This mapping is important to help the that are included in the exam. However, selecting a educators in knowing the covered percentage of question related to specific topic and ILO is done as a specific ILO in the exam. Furthermore, this will random selection of the questions. enable educators to keep track of covered ILOs in the The mark for a selected question is dependent on course at a specific moment. the exam itself such as first, second, midterm, final, 477 CSEDU 2018 - 10th International Conference on Computer Supported Education quiz, exercise, etc. For instance, a question can have As a first step, the educator needs to enter the 10 marks in a first exam which has relatively long details about the course so that he can enter the course time to be finished, but it can also have 5 marks in a name, topics to be covered in the course, ILOs and quiz which has only a short time to be completed. their corresponding cognitive level in Blooms’ Depending on a number of studies, there is a taxonomy. Validation of the ILO and the correlation between the time spent to complete the corresponding ILO is done at runtime. As a result a exam and the final grade that the student get at the end notification message will be displayed to the educator of the exam (Beaulieu & Frost, 1994; Landrum & if there is misleading information. Carlson, 2009; Kale & Kiwelekar, 2013). Similarly, After that, question entry is the next step. Each our approach is considering the time specified by the question is added manually using the developed educator to complete a specific question as an prototype. As depicted in Figure 2, a question can be indicator for the score of the question. In other words, added to the database by specifying the course name, Question Mark = (ETQ / ETE) X (EM) where ETQ is related ILO, corresponding dimension of Bloom’s the estimated time for the question, the ETE is the Taxonomy, an expected time for solving the question estimated time for the exam in total and EM is the in minutes, and the question type. After specifying the exam grade in total. question type, the educator will be shown a GUI to enter the question, the options, URL for multimedia contents (for electronic exams) and the correct answer. As mentioned earlier, there are a number of question types such as True/False, Essay, etc. Equation 1: Calculate the mark of the question. Accordingly, the GUI will depend on the option that the educator select for the type of the exam. Following the previous steps in the algorithm, our proposed algorithm satisfy the idea of generating a balanced and sequenced questions approach (Tofade et al., 2013; Susanti et al., 2015) as it sorts the selected questions depending on Bloom’s taxonomy. Therefore, the questions that are mapped to a lower level of the cognitive level in Bloom’s taxonomy such as remembering, understanding, and applying are placed in the first part of the exam. On the other hand, questions that are mapped to advanced level of the cognitive level from Bloom’s taxonomy such as analyze, evaluate, and create can appear later in the Figure 2: Exam Details Screen. exam. According to psychologists, this will create a safe environment as first the students are asked a Other type of questions is related to providing couple of simple questions and then the students are images along with a textual question and the correct involved in the more analytical questions. answer. Obviously, generated exams with multimedia contents such as animation, 3D models, etc. can be used only in electronic exams rather than paper-based 4 IMPLEMENTATION exams. After entering the required information about the To validate our proposed solution for automatic question, the educator will create an exam by generation of exams, we have developed a web-based specifying the following data: name (such as First, prototype using PHP1 and MySQL2 running on Second, etc.), max grade, time to complete the exam, Apache Tomcat3. To be able to handle the question the semester and the exact date and time for bank, a server stores questions and related data such conducting the exam. After filling in the required as ILOs and corresponding learning topics in the information about the exam, the educator will be database. directed to a new screen which asks him to enter the 1 https://secure.php.net 2 https://www.mysql.com 3 http://tomcat.apache.org 478 A Framework for Automatic Exam Generation based on Intended Learning Outcomes content of the exam such as the sections and ILOs to or combine these algorithms (Al-smadi et al., 2016; be included in the exam. See Figure 3. Abduljabbar & Omar, 2015). Finally, the educator can move to the generation Usability of any automatic exam generator system screen which will display a list of selected questions could be a problem as the graphical user interface and based on the proposed algorithm. The educator needs different considered aspects such as ILO, learning to specify the type of the generated exam such as topic, exams time, etc. could become relatively XML (for electronic exams), PDF, or Document. complex. We are planning to conduct an experiment on improved version of the prototype to validate the issue of usability and acceptability of the system. The evaluation will be mainly based on ISONORM 9241/110-S Evaluation Questionnaire (ISONORM), Subjective Impression Questionnaire (SIQ), Qualitative Feedback (QF), and Workload Perception (WP) to validate different aspects such as easy to use, easy to understand, mental demand, etc. Another important improvement, that will be conducted, is to support educators with a Figure 3: Adding a question to the database screen. visualization tool for viewing easily the covered ILO in all generated exams for a specific course. This will There are a number of limitations in the current be helpful in monitoring and tracking the covered proposed prototype. One of the limitations is that the ILOs so that missing ILOs can be included in the educators are not able to modify or update any future exams of the course. generated exam. This functionality can be important Considering standards for assessments such as to allow the educator to change a specific question or IEEE Learning Object Metadata, IMS Question and select manually other alternative questions for a Test Interoperability, etc. will be investigated in next specific ILO. Another limitation is related to the few stage of this research work to enhance the work from number of the questions stored so far in the database. two points of view. First, it will facilitate the automatic mapping process between Bloom’s Taxonomy and questions. Such standards can be used 5 CONCLUSION AND FUTURE to import questions from existing question bank that are part of many Learning Management Systems WORK (LMS) such as Moodle, Blackboard, Canvas, etc. Traditional preparation of exams is considered as a tedious process, difficult to track all topics according to the syllabus, requiring a high mental demand to REFERENCES avoid question repetition and to avoid questions that are too easy or too tough. The proposed prototype Abduljabbar, D. A. & Omar, N. (2015). Exam questions classification based on Bloom’s taxonomy cognitive addresses the above-mentioned obstacles in an level using classifiers combination. Journal of effective way by generating exams automatically Theoretical and Applied Information Technology. 78 based on Bloom’s taxonomy. The proposed work (3). pp. 447–455. facilitates generating exams automatically depending Al-smadi, M., Margit, H. & Christian, G. (2016). An on the intended learning outcomes of a course Enhanced Automated Test Item Creation Based on module. Learners Preferred Concept Space. 7 (3). pp. 397–405. As this paper presents the general goal of our Ala-Mutka, K. M. (2005). A Survey of Automated research, there are a couple of research extensions to Assessment Approaches for Programming Assign- be considered in the future. To improve the alignment ments. Computer Science Education. 15(2), pp.83–102. Alade, O. M. & Omoruyi, I. V. (2014). Table Of Specifica- of assessment with learning outcomes, the next step tion And Its Relevance In Educational Development is to classify different questions, that can be retrieved Assessment. European Journal of Educational and from a Learning Management System (LMS), Development Psychology. 2 (1). pp. 1–17. automatically using some sort of classification Beaulieu, R. P. & Frost, B. (1994). Another Look at the algorithms such as Support Vector Machine (SVM), Time-Score Relationship. Perceptual and Motor Skills. Naïve Bayes (NB), and k-Nearest Neighbour (k-NN) 78 (1). pp. 40–42. 479 CSEDU 2018 - 10th International Conference on Computer Supported Education Biggs, J. B. & Collis, K. F. (1982). Evaluating the Quality Mohandas, M., Chavan, A., Manjarekar, R. & Karekar, D. of Learning: The SOLO Taxonomy (Structure of the (2015). Automated Question Paper Generator System. Observed Learning Outcome). International Journal of Advanced Research in Bloom, B.S. (1956). Taxonomy of Educational Objectives: Computer and Communication Engineering. 4(12), The Classification of Educational Goals. Taxonomy of 676-678. educational objectives: the classification of educational Omar, N., Haris, S. S., Hassan, R., Arshad, H., Rahmat, M., goals. B. S. Bloom (ed.). Longman Group. Zainal, N.F.A. & Zulkifli, R. (2012). Automated Blumberg, P. (2009). Maximizing learning through course Analysis of Exam Questions According to Bloom’s alignment and experience with different types of Taxonomy. Procedia - Social and Behavioral knowledge. Innovative Higher Education. 34 (2), pp Sciences.59 pp. 297 – 303. 93–103. Susanti, Y.., Iida, R.. & Tokunaga, T.. (2015). Automatic Cen, G., Dong, Y., Gao, W., Yu, L., See, S., Wang, Q., generation of english vocabulary tests. CSEDU 2015 - Yang, Y. & Jiang, H. (2010). A implementation of an 7th International Conference on Computer Supported automatic examination paper generation system. Education, Proceedings.pp.77-87. Mathematical and Computer Modelling.51(11), pp. Taqi, M. K. & Ali, R. (2016). Automatic question 1339-1342. classification models for computer programming Fuller, U., Riedesel, C., Thompson, E., Johnson, C.G., examination: A systematic literature review. Journal of Ahoniemi, T., Cukierman, D., Hernán-Losada, I., Theoretical and Applied Information Technology. pp. Jackova, J., Lahtinen, E., Lewis, T. L. & Thompson, 360–374. D.M. (2007). Developing a computer science-specific Tofade, T., Elsner, J. & Haines, S. T. (2013). Best practice learning taxonomy. ACM SIGCSE Bulletin. 39(4), pp. strategies for effective use of questions as a teaching 152-170. tool. American Journal of Pharmaceutical Education. Gangar, F. K., Gori, H.G. & Dalvi, A. (2017). Automatic 77(7).pp. 55. Question Paper Generator System. International Journal of Computer Applications. 66 (10). pp. 42–47. Jayakodi, K., Bandara, M. & Perera, I. (2016). An automatic classifier for exam questions in Engineering: A process for Bloom’s taxonomy. In: Proceedings of 2015 IEEE International Conference on Teaching, Assessment and Learning for Engineering, TALE 2015. 2016. Joshi, A., Kudnekar, P., Joshi, M. & Doiphode, S. (2016). A Survey on Question Paper Generation System. In: IJCA Proceedings on National Conference on Role of Engineers in National Building NCRENB. 2016, pp. 1– 4. Kale, V. M. & Kiwelekar, A.W. (2013). An algorithm for question paper template generation in question paper generation system. In: 2013 The International Conference on Technological Advances in Electrical, Electronics and Computer Engineering, TAEECE 2013. 2013, Konya, p. 256–261. Krathwohl, D.R. (2002). A Revision of Bloom’s Taxonomy: An Overview. Theory Into Practice. 41(4), pp.212-218. Landrum, R.E. & Carlson, H. (2009). The Relationship Between Time to Complete a Test and Test Performance. Psychology Learning & Teaching. Manuel Azevedo, J., Patrícia Oliveira, E. & Damas Beites, P. (2017). How Do Mathematics Teachers in Higher Education Look at E-assessment with Multiple-Choice Questions. In: Proceedings of the 9th International Conference on Computer Supported Education. 2017. pp. 137-145. Marzukhi, S. & Sulaiman, M.F. (2014). Automatic generate examination questions system to enhance preparation of learning assessment. In: 2014 4th World Congress on Information and Communication Technologies, WICT 2014. 2014. pp. 6-9. 480