(PDF) Course Evaluation in Distance Learning: A Qualitative Analysis of Students' Perceptions of Phonology Course offered by All

RESEARCH JOURNAL TAṢDĪQ Vol 04 No 01, 2022 Course Evaluation in Distance Learning: A Qualitative Analysis of Students’ Perceptions of Phonology Course offered by Allama Iqbal Open University eISSN: 2707-6229 pISSN: 2707-6210 DOI: https://doi.org/10.56276/tasdiq.v4i01.81 Dr Ubaidullah Khan Email:

Lecturer, Department of English, Allama Iqbal Open University, Islamabad ABSTRACT This article is part of a course evaluation project at Allama Iqbal Open University, one of the largest universities in Pakistan and the main university offering education through the Distance Learning System. The first part of this project was a quantitative analysis of the responses of students to the course evaluation survey whereas this article presents a qualitative analysis of open-ended questions asked to students studying Phonology course and teachers teaching this course at Islamabad and Rawalpindi campuses of the university. The students presented their point of view regarding the three components under inquiry and suggested improvements in the course by upgrading the course contents and improving the quality of tutorial support. KEYWORDS Distance Learning System, Distance Education, Course Evaluation, Perceptions, Feedback Received: 02-May-22 Accepted:05-Jul-22 Online: 05-Jul-22 Introduction Educational institutions need to evaluate their courses through the stakeholders’ feedback to improve their quality. In today’s outcome-driven educational environment, evaluation plays an important role in assessing the outcomes of the courses. The urge behind evaluation is either to look for observable outcomes or to be able to take decisions regarding improving the course. Evaluation provides a solid foundation to make such decisions for improvement. As Diamond and Sudweek (1980) remark, “without the information base that formal, systematic evaluation provides, decisions about content, structure, and revision must be based on hunches, personal preferences, or tradition” (p. 28). Evaluation becomes extraordinarily important when the courses are being taught in a unique system such as Distance Learning System (hereinafter referred to as DLS). This system of education is different from traditional, face-to-face system as teaching in this system happens remotely, and it is the prime responsibility of the https://doi.org/10.56276/tasdiq.v4i01 Riphah International University, Faisalabad Campus 1 RESEARCH JOURNAL TAṢDĪQ Vol 04 No 01, 2022 educational administrators to ensure effective delivery of the instructional material keeping in mind the absence of a regular classroom atmosphere and a regular teacher, as is the case in traditional education systems. While explaining the DLS, Willis (1995) suggests that in this system “the instructor and students may share a limited common background and typically have minimal face-to-face contact” (as cited in Murphy, 2000, p. 103). The purpose of distance education is to provide opportunities for getting education to as many people as possible, especially those who cannot attend a regular institution for one reason or the other. Allama Iqbal Open University, Islamabad (hereinafter referred to as AIOU) is the institution that provides distance education (DE) to masses throughout Pakistan. It offers a wide range of courses ranging from elementary to PhD levels and caters to millions of students from across the country. The students enrolled in these programmes either cannot afford to study in the regular system due to various social, economic or other constraints or want to improve their qualifications alongside their jobs. DLS is unique in the sense that the students study from the comfort of their homes and do not have to attend regular classes. They do take admission and are bona fide students of the institution, but due to the nature of this system, they are not required to attend daily classes on a given campus. They are rather provided with study material and are required to study this material at home, prepare assignments after studying the course contents and submit them to the tutor allocated for their respective courses of the semester. The classes are held weekly at an educational institute in the locality of the students which is assigned to them by the university as their study centre. By the end of the course work, they are sometimes required to attend a face-to-face workshop, or are simply required to appear in a final exam which is conducted in their vicinity rather than on the main campus. In DLS, the courses offered must be communicative to make up for the absence of a regular teacher, and supportive so that they facilitate the learning process. Educational Approach to the Programme and the Course The Phonology course offered by the Department of English Language and Applied Linguistics, AIOU is part of MA TEFL which is a teacher training programme consisting of various courses related to English language teaching. It is a programme for aspiring and in-service teachers of English who teach at levels ranging from primary to university. The MA TEFL programme is split into two halves: the first part is a postgraduate Diploma TEFL, of which a Phonology course is a part, and the second part is named MA TEFL which includes a thesis component, apart from other theoretical courses. Each of the theoretical courses of the two halves of the programme includes two assignments; one is theory-based and the second one is research-based and also involves a practicum component in which the students are required to present their research while the teacher observes and provides feedback. Phonology is the subject that deals with the pronunciation of the English language, requires greater teacherstudent/ trainer-trainee interaction, hence 18 classes are allocated for this course as compared to 10 allotted tregular/theory-based courses. https://doi.org/10.56276/tasdiq.v4i01 Riphah International University, Faisalabad Campus 2 RESEARCH JOURNAL TAṢDĪQ Vol 04 No 01, 2022 The classes are held on the weekends, during which the students discuss the course contents which includes the course books and allied material related to the contents, which is referred to as ‘supplementary material’. They also discuss their assignment questions and prepare a presentation which is held in the last of the 18 classes. Course Evaluation in DLS The uniqueness of DLS demands that the decisions regarding improvement of the course must be based on a rigorous process of course evaluation. In a regular system, the teachers, as well as other entities such as the curriculum committees or departments, are aware of the feedback about their courses from day one, as the students and the administration are co-located at the campus and their interaction with the students who are end-users of these courses are happening on daily basis. On the other hand, the scenario is different in DLS as here the students are remotely located and their interaction with the teachers and the administration is minimal, and they cannot be aware of the students’ feedback naturally. The students are the most important and most immediate stakeholders in teaching-learning context and their perceptions regarding the course are of immense value. They can point out whether the intended outcomes match their needs or not, and whether the course is meeting their expectations in terms of its outcomes. This article intends to gauge students’ perceptions about one of the most important courses of Diploma TEFL offered by AIOU. It is part of a dissertation written for a Master degree and it undertook to seek students’ responses regarding the course contents, Supplementary Reading Material and the tutorial support provided by the university. The first part of this research (Khan, U., Zeb, S., Uzair, M. (2019) quantitatively analyzed students’ responses to the course evaluation survey whereas this one analyses their responses to open-ended questions in a qualitative way. Research Objectives As also stated in the first article based on this project (Khan, U. et al., 2019) the research was carried out with the following objectives in mind: 1. To investigate the effectiveness of the course contents/material & tutorial support of Phonology offered by AIOU. 2. To suggest improvements in the course contents/material and tutorial support system in the light of the respondents’ feedback Research Question This research explored the following research questions:  What are the students’ and teachers’ perceptions about course contents, supplementary material and tutorial support for Phonology course of Diploma TEFL? Aim of the Research This research is different from the first article of the project as it deals with opinions the of the students regarding the three components of the course about https://doi.org/10.56276/tasdiq.v4i01 Riphah International University, Faisalabad Campus 3 RESEARCH JOURNAL TAṢDĪQ Vol 04 No 01, 2022 which they were asked to comment. Their opinion is intended to highlight the areas that need improvement, and this feedback will likely raise the level of the course and enhance the quality of its content as well as the tutorial support system which is not only about the weekly system, but also about issues such as tutors’ help in the preparation of assignments, and guidance related to the preparation of final presentations/workshops. Significance of Research The purpose of seeking the opinion of the sample is to obtain data that is detailed, thorough and exhaustive, and is not only about the close-ended questions prepared by the researcher. In the qualitative survey, the respondents are free to speak up their minds and comment on issues other than the ones perceived by the researcher as worthy of discussion. They can bring up other issues which may be concealed from the eyes of the administration, of which the researcher was an integral part at the time of conduction of this study. Hence, this study is likely to inform the course planners about weak areas of the course and will equip them with the ideas to overcome the possible issues related to course contents and tutorial support system faced by the students in taking the course. Literature Review Distance Education Distance education means offering education to the students away from the institution. Distance Learning Manuel (2007) defines DE as “the practice of offering educational services – either instruction or support services – to students who are not physically co-located with the individuals providing the service” (p.3). Another definition of DE is provided by The Distance Education and Training Council which defines it as “study, at a distance, with an educational provider that conducts organized, formal learning opportunities for students. The instruction is offered wholly or primarily by distance study, through virtually any media”. Guidelines for Distance Education (2011, p. 6). This media might include web-based education, television or correspondence-based education or education through any kind of other media. Curriculum in DLS As discussed in Distance Learning Manuel (2007), the ACCJC has laid out certain principles for the development of curriculum when teaching is happening through electronic mode. These principles are as much relevant to DE as they are to the electronic mode of DE. According to them, firstly, the courses offered through DE must ‘result in learning outcomes appropriate to the rigour and breadth of the course credit, degree, or certificate awarded’. Secondly, it should be “coherent and complete and result in learning outcomes comparable to those delivered through other means”. Thirdly, courses delivered through DE should adhere to the same principles of academic quality and integrity as courses delivered traditionally. Lastly, the institution should have an effective means of ensuring the integrity of the educational process. The institutions intending to maintain quality can ask https://doi.org/10.56276/tasdiq.v4i01 Riphah International University, Faisalabad Campus 4 RESEARCH JOURNAL TAṢDĪQ Vol 04 No 01, 2022 themselves questions on these principles to ensure whether these considerations are kept in mind while devising the curriculum of the DE courses or not. Role of Tutor in DLS The Distance Learning system provides the students with independence which is not possible in a traditional classroom. It is more student-centred as they are in charge of their learning. Their engagement with the course content is tutorguided which facilitates students’ effective learning. In the context of AIOU, the tutor is a facilitator for the students and plays role of a mediator between the student and the course contents. He explains the course contents with the help of supplementary material and enables them to learn new knowledge, and skills and helps them improve their learning that resulting in better students performance. Course Evaluation and DLS DLS is supposed to be a flexible system and support career development and lifelong learning due to which consumer’s expectations of quality instruction, effective educational outcome, and satisfaction for learning and demand of courses under this system is rising (Debourgh, 1999 as cited by Ali & Ahmed 2011. P. 119). Ali & Ahmed further cite Allen et al. (2002) and Wang (2003) who argue that “the satisfaction of a student can be determined from his level of pleasure as well as the effectiveness of the education that the student experiences” And the more satisfied students are reported by Frederick Sen (2000) to show considerably higher levels of learning than students with low level of satisfaction (p. 119). The focus of the course evaluation should be whether it has provided the students real learning or not. Palloff & Pratt (2008) remark that in learner-centred classroom, course evaluation, rather than just asking whether they like the instructor or not and what they like or dislike in the course, should focus on whether the course provided any real learning opportunity or not. They cite Angelo and Cross (1993) that the instructors should ask themselves three questions: what are essential skills and knowledge they are trying to teach, how can they find out whether the students are learning these skills and knowledge, and how can they help students learn better. (p. 1). The students in DLS “do just or even better than face to face classroom students” (Brownson and Harriman, 2000, as cited by Ali & Ahmed, 2011. P. 120) Distance education, then, is a unique context which provides unique opportunity to the students to be the master of their own learning. The students are dealing with knowledge, skills and techniques which are new to them, and to handle any difficulties that they may encounter in learning the knowledge the course offers, they can always enjoy the facilitation provided to them by the tutor. Effective course evaluation only measures students’ perceptions of the effectiveness of the course in imparting real learning and ensures how much the tutor support is helpful in this learning journey. Research Methodology This research is qualitative as it deals with the students and teachers’ opinions about improvement of the three components of the course, i.e. contents, https://doi.org/10.56276/tasdiq.v4i01 Riphah International University, Faisalabad Campus 5 RESEARCH JOURNAL TAṢDĪQ Vol 04 No 01, 2022 supplementary material, and the tutorial support. The data was collected through a survey which consisted of a questionnaire which carried both closed-ended and open ended questions. The quantitative part has been discussed in detail in Khan, U. et al.(2019) whereas the current article deals with students’ opinions expressed by them in response to open ended questions. Theoretical Framework The purpose of course evaluation is obviously to observe the intended outcomes, effectiveness and impact of the courses and to improve their present shape for the better outcome. As explained in figure 1 below, Kirkpatrick (1998) mentions four levels of evaluation: “Evaluation at the first-level attempts to determine student reactions. Second-level evaluation seeks to determine the amount of learning that has occurred. Evaluation at the third-level focuses on a change in student behavior. Fourth-level evaluation examines holistic results” (as cited by Roberts, Irani, Lundy, Telg (2004). Roberts et al (2004) further elaborate that evaluating a course at the levels beyond the first one can be problematic especially when the course evaluation data serves as the basis of promotion of the teachers teaching the given course. They further remark that establishing base-line data about the amount of student learning and behavior development would be impossible ‘given the broad range of courses offered at most institutions and the variance in academic rigor of these courses” (p. 1-2). They cite example of Ehrmann and Zuniga (1997) who likened the problem of carrying out a holistic course evaluation to finding an elephant in the dark cave with a flashlight, with the object being searched being too large and complex and the flashlight of evaluation being too weak. As elucidated by the example of elephant and the search light, the present study also does not try to discover elephant in the dark care and focuses on one aspect of the course, that is the students reactions to the components of course contents and tutorial support, and leave the ret to the researchers to whom that pursuit would suit better. Pallof & Pratt (2008) cite Brookfield (1995) who states that traditional courses rarely evaluate what they want to and are more of a popularity contest since they ask the students to rate what they liked or disliked. And this research is different from traditional researches as it also seeks students opinion in an open ended way rather than merely doing what Brookfield referred to. https://doi.org/10.56276/tasdiq.v4i01 Riphah International University, Faisalabad Campus 6 RESEARCH JOURNAL TAṢDĪQ Vol 04 No 01, 2022 Research Tool The students’ responses were obtained through a questionnaire. The questions comprised three following main areas of the course. • Course Content/material • Supplementary Material • Tutorial Support Population & Sample The population of the study was Diploma TFEL students enrolled with the Department of English, AIOU, Islamabad. They are normally aspiring /in-service English language teachers of different levels with different educational qualification and experience. The sample comprised of a group of students studying at Islamabad study centre where the researcher taught the course as their tutor. Analysis In the fourth section of the questionnaire, the respondents were asked to give their opinions regarding the areas of the course about which they were asked questions in the first three parts of the questionnaire. The responses of the students are provided below in form of chart, followed by their analysis: Resp. Course Contents/Supplementary Reading Material Tutorial Support A More information and latest information must be added Some relevant and elaborated material should be included in the reading material because we as a working people don’t have time to visit libraries or surf on the internet Nil The material provided for teaching should be more interactive and there should be more involvement of the students than the teachers. During tutorials, certain worksheets for pronunciation and symbols should be used. Nil Should take help of a native speaker, because it is said that the language is what a native speakers speaks. Nil B C D E F G https://doi.org/10.56276/tasdiq.v4i01 Audio visual aids should be provided so that the learners get a better understanding of the contents of the subject Nil The teachers should take responsibility and should be more serious than the students. Nil Nil The audio cassette should also be played in the tutorial meetings. This can enhance the probability of learning as some students does not bother to listen to those even. The Riphah International University, Faisalabad Campus 7 RESEARCH JOURNAL TAṢDĪQ H I J K L Vol 04 No 01, 2022 The book should be revised. The old phonetic symbols should be replaced by new ones. It is to be made interesting. The latest courses of phonology should be introduced and it should be updated frequently. Study material should be in the form of planners along with some reference books from or by different sources. Research report which is compulsory and getting 40 % marks in each component. I need to know the difference between research report and thesis. Our phonology teacher wants us to come up with whole preparation of thesis which is absolutely impossible within short span of ten days where (as) for MA thesis and M Phil thesis three years time is given. The study material (used to be) too much lengthy, it should be comprehensive and concise and to some extent to the point. The material should be more precise and should not be lengthy. M The book demands revision of old phonetic symbols. N O P It should be increased and some material should be added especially the problem of stress is not clear in the book. The study material should be increased in order to make pronunciation in a wide range. It should be up-to-date so that the students’ interest can be maintained. https://doi.org/10.56276/tasdiq.v4i01 tutor can also comment on the audio aid. However, video aid can also be used. Nil There should be lists of activities designed by tutors for mutual and healthy interaction b/w tutors and students. The material of socio (linguistics) and psycholinguistics as well as Phonology need to be updated as the book which I am using now, my father has used some 15 years ago. When I compare the two books (mine and his) material is the same. The students should be facilitated with language lab. It would be better if the tutors were more experienced so as to provide more comprehensive help. Oral practice should be compulsory for the students for more speaking skills. There is less practice in the tutorial classes. The tutorial classes should be increased. Class room lessons should be directed through AV Aids. Nil Riphah International University, Faisalabad Campus 8 RESEARCH JOURNAL TAṢDĪQ Q R S Vol 04 No 01, 2022 The study material must be revised with full of headings. It must be presentable and must appeal to the reader. CDs and more visual aids may be provided. Teacher should help to prepare assignments specially project reports. It creates problems for the students. More practice, more listening T It would be more comprehendible if teach the subject through audio visual aid. U The authentic material of reading should be provided. Table: 1 Nil Class may be commenced in the language laboratory like NUML classrooms and language laboratories. Latest live materials should be used. Movies, slang etc be told. Nil There is no discussion about the syllabus in the classroom. Let us now discuss the important areas highlighted by the respondents in their responses to the open-ended questions Most of the students stressed that the course (i.e. Phonology) needs to be revised; latest information which is more relevant and elaborated should be included. They were of the opinion that the textbook needs to be revised specially in terms of phonetic symbols which are old and confusing. One respondent opined that the book she has is the same which her father studied about fifteen years back, and no revision has been done. She suggested that the course books need to be revised to introduce the students with the latest trends in the discipline. One of the respondents (E) stressed the importance of making the book more interactive, meaning thereby that less focus should be on information and more on activities. The same point was endorsed by another respondent (F) who stressed the need of introducing pronunciation worksheets in the class. In view of E, there should be more student involvement in the course, suggesting that he/she must have experienced a more teacher centered class which may not serve the purpose of teaching pronunciation properly since it requires more of student practice than instruction alone. Regarding the Supplementary Reading Material, the issues that have come to fore in the respondents’ opinions are many. First of these is length of the contents that is too much in view of certain respondents (L & M share this view). One of the respondents indicated the need of including more material regarding ‘stress’ as it was not enough. One of the respondents suggested that native speakers may be hired to present the model of language, but it was not clear whether he / she wanted the audio aid to be recorded in native speaker’s voice or wanted the administration to hire them as tutor (which may be quite idealistic in the given circumstances). Another respondent asserted that the course material needs to be supported with audio visual aids in order to make the course contents more effective. https://doi.org/10.56276/tasdiq.v4i01 Riphah International University, Faisalabad Campus 9 RESEARCH JOURNAL TAṢDĪQ Vol 04 No 01, 2022 The chief concern of the respondents regarding the third area of evaluation was the use of AV aids in the classroom. They felt that tutorial support could be more effective if AV aids are used in the class. One respondent said that the cassette should be used in the class and the teacher’s commentary on the recorded text would benefit the students immensely. The next thing they pointed out was the need of language lab (L & S). They suggested that just like other universities have language laboratories, AIOU must also set up a language lab for its students so as to better train them in Phonology. Respondent E suggested that teachers need to be more proactive and make the class more interactive than it normally is. In view of M, more experienced tutors need to be hired. J advocated for activity based classes in order to increase student teacher interaction. Respondent N supported the idea of increased oral practice in the class so that the students learn phonology better. Respondent O advocated for increase in the number of classes, as according to her the existing number of classes is not enough for practice of sounds. Respondents U & P advocated the use of videos and slang in order to introduce the students to real life language. So we can see that the open ended questions have contributed significantly to enlighten the evaluator about the areas of the course that need improvement. Issued that were not / could not have been made a part of the discussion by the researcher have also been highlighted as the students were given a free hand to give their opinions regarding the areas in which they felt the course lacked. Summary of the Students’ Feedback This section sums up the feedback provided by the students which could be considered take-home message of this research. The important points are listed below: i. ii. iii. iv. v. vi. The students strongly asserted that the book needs to be revised and contents needs to be updated and the old phonetic symbols need to be replaced with latest / standard ones. The students felt that the course contents need to be made more interactive with increased level of students’ involvement. There is not enough material included in the book on the issue of ‘stress’. Supplementary reading material is thought to be too much length by the students. They felt that quality of the material rather than length should be given importance. The students felt that it should be made more organized and presentable. Some students felt that clearer guidelines should be given to the tutors as some of them demand as much preparation from them in presentation as it to be done for a complete research-thesis. The students wanted effective use of AV Aids and asked for arrangement of use of cassette in the class for the purpose of clarification of pronunciation related issues. https://doi.org/10.56276/tasdiq.v4i01 Riphah International University, Faisalabad Campus 10 RESEARCH JOURNAL TAṢDĪQ vii. viii. Vol 04 No 01, 2022 Teachers need to give more attention to oral and listening practice in the classroom. The students demanded for establishment of a language lab as available in other language institutions. Conclusion This research highlighted students’ perceptions of the effectiveness of the course material, supplementary material and the tutor support. Through open-ended questions, it encouraged the students of the course to present the real issues they are facing with the three components of the course being investigated. They provided useful feedback which is likely to be beneficial for the course developing / revision team to update the course content and improve the quality of tutor-support. References Ali, A & Ahmed, I. (2011). Key Factors for Determining Students’ Satisfaction in Distance Learning Courses: A Study of Allama Iqbal Open University. Contemporary Educational Technology, 2(2), pp. 118-134 Diamond, R. M., & Sudweeks, R. R. (1980). A Comprehensive Approach to Course Evaluation. Journal of Instructional Development, Vol. 4, No. 1, pp. 28. Springer. Retrieved from: https://www.jstor.org/stable/30220636 Accessed: 13-01-2022 14:57 UTC Distance Learning Manual . (2007). A Publication of the Accrediting Commission for Community and Junior Colleges Western Association of Schools and Colleges. Pp. 1-21. Retrieved from https://files.eric.ed.gov/fulltext/ED495650.pdf Guidelines for Distance Education (2011).Texas Commission on Law Enforcement Officer Standards and Education. pp. 1-16. Retrieved from https://scholarworks.moreheadstate.edu/cgi/viewcontent.cgi?article=1074& context=msu_theses_dissertations Khan, U., Zeb, S., Uzair, M. (2019). Students Perceptions of Effectiveness of Course Contents,Supplementary Material and Tutorial Support in Phonology Course offered By Allama Iqbal OpenUniversity Islamabad. Global Regional Review (GRR), Vol. IV, (II). Page: 546 ‒ 563 | DOI: 10.31703/grr.2019(IV-II).57 Murphy, T. H. (2000). An evaluation of a distance education course design for general soils. Journal of Agricultural Education Vol. 41 Issue 3 p. 103-113 Palloff & Pratt (2008). Effective Course, faculty and programmed evaluation. 24th Annual conference on Distance learning & Education. Retrieved from http://www.uwex.edu/disted/conference Roberts, T. G., Irani, T., Lundy, L. K., Telg, R. (2004).Practices in student evaluation of distance education courses among land grant institutions, Journal of Agricultural Education. Volume 45, Number 3 https://doi.org/10.56276/tasdiq.v4i01 Riphah International University, Faisalabad Campus 11