(PDF) Developing Self-Regulated Learning through Reflection on Learning Analytics in Online Learning Environments

ascilite2015 Australasian Society for Computers in Learning and Tertiary Education Curtin University, Perth, Australia Conference Proceedings 1 Full Papers Page Number The Conceived, the Perceived and the Lived: Issues with 21 st Century Learning and 13 Teaching Barac, Karin Learning design for science teacher training and educational development 21 Bjælde, Ole E; Caspersen, Michael E; Godsk, Mikkel; Hougaard, Rikke F; Lindberg, Annika Tensions and turning points: exploring teacher decision-making in a complex 31 eLearning environment Bradey, Scott Navigate Me: maximising student potential via online support 43 Clark, Colin; Andreacchio, Jessica; Kusevskis-Hayes, Rita; Lui, Jessie; Perry, Shauna; Taylor, Ethan Designing an authentic professional development cMOOC 53 Cochrane, Thomas; Narayan, Vickel; Burcio-Martin, Victorio; Lees, Amanda; Diesfeld, Kate Investigating the effectiveness of an ecological approach to learning design in a 65 first year mathematics for engineering unit Czaplinski, Iwona Community volunteers in collaborative OER development 77 DeVries, Irwin J A ‘participant first’ approach to designing for collaborative group work in MOOCs 89 Dona, Kulari Lokuge; Gregory, Janet Building graduate attributes using student-generated screencasts 100 Frawley, Jessica Katherine; Dyson, Laurel Evelyn; Tyler, Jonathan; Wakefield, James Self-organising maps and student retention: Understanding multi-faceted drivers 112 Gibson, David Carroll; Ambrose, Matthew; Gardner, Matthew New applications, new global audiences: Educators repurposing and reusing 3D 121 virtual and immersive learning resources Gregory, Sue; Gregory, Brent; Wood, Denise; O’Connell, Judy; Grant, Scott; Hillier, Mathew; Butler, Des; Masters, Yvonne; Stokes-Thompson, Frederick; McDonald, Marcus; Nikolic, Sasha; Ellis, David; Kerr, Tom; de Freitas, Sarah; Farley, Helen; Schutt, Stefan; Sim, Jenny; Gaukrodger, Belma; Jacka, Lisa; Doyle, Jo; Blyth, Phil; Corder, Deborah; Reiners, Torsten; Linegar, Dale; Hearns, Merle; Cox, Robert; Jegathesan, Jay Jay; Sukunesan, Suku; Flintoff, Kim; Irving, Leah Conditions for successful technology enabled learning 134 Henderson, Michael; Finger, Glenn; Larkin, Kevin; Smart, Vicky; Aston, Rachel; Chao, Shu-Hua Full Papers 2 To type or handwrite: student's experience across six e-Exam trials 143 Hillier, Mathew Predictors of students’ perceived course outcomes in e-learning using a Learning 155 Management System Kwok, David Digital leap of teachers: two Finnish examples of rethinking teacher professional 168 development for the digital age Leppisaari, Irja; Vainio, Leena An enhanced learning analytics plugin for Moodle: student engagement and 180 personalised intervention Liu, Danny Yen-Ting; Froissard, Jean-Christophe; Richards, Deborah; Atif, Amara Prior knowledge, confidence and understanding in interactive tutorials and 190 simulations Lodge, Jason M; Kennedy, Gregor Higher education students' use of technologies for assessment within Personal 202 Learning Environments (PLEs) Lounsbury, Lynnette; Mildenhall, Paula; Bolton, David; Northcote, Maria; Anderson, Alan Strong and increasing student demand for lecture capture in the changing 216 Australian university classroom: results of a national and institutional survey Miles, Carol A Analysis of MOOC Forum Participation 224 Poquet, Oleksandra; Dawson, Shane Designing for relatedness: learning design at the virtual cultural interface 235 Reedy, Alison; Sankey, Michael Open and Interactive Publishing as a Catalyst for Educational Innovations 248 Ren, Xiang Learning Design for digital environments: agile, team based and student driven 258 Soulis, Spiros; Nicolettou, Angela Interdisciplinary opportunities and challenges in creating m-learning apps: two case 265 studies Southgate, Erica; Smith, Shamus P; Stephens, Liz; Hickmott, Dan; Billie, Ross Paving the way for institution wide integration of Tablet PC Technologies: 275 supporting early adopters in Science and Engineering Taylor, Diana; Kelly, Jacqui; Schrape, Judy MyCourseMap: an interactive visual map to increase curriculum transparency for 285 university students and staff Tee, Lisa B G; Hattingh, Laetitia; Rodgers, Kate; Ferns, Sonia; Chang, Vanessa; Fyfe, Sue Standing on the shoulders of others: creating sharable learning designs 297 Weaver, Debbi; Duque, Samantha Full Papers 3 Higher Education Teachers’ Experiences with Learning Analytics in Relation to 308 Student Retention West, Deborah; Huijser, Henk; Heath, David; Lizzio, Alf; Toohey, Danny; Miles, Carol Exploratory and Collaborative Learning Scenarios in Virtual World using Unity- 320 based Technology Wilding, Karin; Chang, Vanessa; Gütl, Christian Remote Access Laboratories for Preparing STEM Teachers: A Mixed Methods Study 331 Wu, Ting; Albion, Peter R; Orwin, Lindy; Kist, Alexander; Maxwell, Andrew; Maiti, Ananda A Mobile App in the 1st Year Uni-Life: A Pilot Study 342 Zhao, Yu; Pardo, Abelardo Full Papers 4 Concise papers Page Number Learning maps: A design-based approach for capacity building in tertiary online 353 learning and teaching Adachi, Chie; O'Rourke, Mark Using Learning Design to Unleash the Power of Learning Analytics 358 Atkinson, Simon Paul The future of practice-based research in educational technology: Small steps to 363 improve generalisability of research Alhadad, Sakinah S. J. Features of an online English language testing interface 369 Al Nadabi, Zakiya Fostering deep understanding in geography by inducing and managing confusion: 374 an online learning approach Arguel, Amaël; Lane, Rod Using expectation confirmation theory to understand the learning outcomes of 379 online business simulations Benckendorff, Pierre; Gibbons, Belina; Pratt, Marlene Towards a Pedagogy of Comparative Visualization in 3D Design Disciplines 384 Birt, James R; Nelson, Jonathan; Hovorka, Dirk Implementing blended learning at faculty level: Supporting staff, and the ‘ripple 389 effect’ Borland, Rosy; Loch, Birgit; McManus, Liam The ethical considerations of using social media in educational environments 394 Cameron, Leanne; Tanti, Miriam; Mahoney, Kim Teachers Cloud-based Content Creation in light of the TPACK Framework: 399 Implications for Teacher Education Campbell, Chris; Al Harthi, Aisha; Karimi, Arafeh The Next Wave of Learning with Humanoid Robot: Learning Innovation Design 404 starts with “Hello NAO” Chua, Xin Ni; Chew, Esyin Loop: A learning analytics tool to provide teachers with useful data visualisations 409 Corrin, Linda; Kennedy, Gregor; de Barba, Paula; Bakharia, Aneesha; Lockyer, Lori; Gasevic, Dragan; Williams, David; Dawson, Shane; Copeland, Scott Teaching Complex Theoretical Multi-Step Problems in ICT Networking through 3D 414 Printing and Augmented Reality Cowling, Michael; Birt, James An investigation of blended learning experiences of first-year Chinese transnational 419 program students at an Australian university Dai, Kun Concise Papers 5 A comparison of undergraduate student experiences of assessed versus non- 424 assessed participation in online asynchronous discussion groups: Lessons from a cross disciplinary study in health and sociology Douglas, Tracy; Mather, Carey; Murray, Sandra; Earwaker, Louise; James, Allison; Pittaway, Jane; Robards, Brady; Salter, Susan Digital Futures research and society: action, awareness and accountability 429 Doyle, Joanne; McDonald, Lisa; Cuthill, Michael; Keppell, Mike Making the Connection: Allowing access to digital higher education in a 434 correctional environment Farley, Helen; Dove, Sharron; Seymour, Stephen; Macdonald, John; Abraham, Catherine; Lee, Chris; Hopkins, Susan; Cox, Jacinta; Patching, Louise Badging digital pathways of learning 440 Gibson, David; Coleman, Kathryn; Irving, Leah The Agile Learning Model: Using big data to personalise the acquisition of 445 accounting skills Gregory, Brent; Wysel, Matthew; Gregory, Sue PST Online: Preparing pre-service teachers for teaching in virtual schools 450 Grono, Steve; Masters, Yvonne; Gregory, Sue Occupational Medicine Simulation Project 455 Griffiths, Aaron Can learning analytics provide useful insights? An exploration on course level 460 Heinrich, Eva A pedagogical end game for exams: a look 10 years into the future of high stakes 465 assessment Hillier, Mathew; Gibbons, Belina Are Higher Education Institutions Prepared for Learning Analytics? 471 Ifenthaler, Dirk A blended learning ecosystem: What are the motivational issues for students? 476 Hartnett, Maggie; Kearney, Alison; Mentis, Mandia Measuring creativity in collaborative design projects in pre-service teacher 481 education Kennedy-Clark, Shannon; Kearney, Sean; Eddles-Hirsch, Katrina; De La Hoz, Rod; Galstaun, Vilma; Wheeler, Penny How to develop an online community for pre-service and early career teachers? 486 Kelly, Nick; Clarà, Marc; Pratt, Marlene Collaboration between Primary Students and the Use of an Online Learning 491 Environment: The Previous Collaborative Work Experiences Factor Kokkinaki, Aikaterini A digital what? Creating a playspace to increase the quality of technology-enhanced 497 teaching and learning Lamond, Heather; Rowatt, Andrew John Concise Papers 6 The three pillars to building staff capability to create digital learning experiences 502 Manning, Catherine; Macdonald, Hero Developing Self-Regulated Learning through Reflection on Learning Analytics in 507 Online Learning Environments Mikroyannidis, Alexander; Farrell Frey, Tracie Marie Personalising professional learning mobility in Higher Education 512 Mitchell, Maxine; Cottman, Caroline Connecting fun and learning - an activity-theoretical approach to competency based 517 game development O'Rourke, Mark Learners’ confusion: faulty prior knowledge or a metacognitive monitoring error? 522 Pachman, Mariya; Arguel, Amael; Lockyer, Lori Exploring my university students’ online learning activities in Wikis 527 Quek Choon Lang, Gwendoline; Liu, Cong Learning to swim in an ocean of student data 532 Russel, Carol Benchmarking for technology enhanced learning: Longer term benefits 537 Sankey, Michael Building a framework for improved workplace assessment practice and better 542 outcomes through online platforms Schier, Mark A; Dunn, Louise Promoting Critical Thinking in a Large Class through Outcomes-Based Approach by 545 Means of an Audience Response System Keong Seow, Teck; Swee Kit Soong, Alan Digital andragogy: A 21st century approach to tertiary education 552 Sheffield, Rachel; Blackley, Susan Ellen Blended Learning Adoption Monitoring 557 Smith, Simon Douglas The value of digital critical reflection to global citizenship and global health 562 Stoner, Lee Authentic context as a foundation for gamification and game-based learning 566 Teräs, Hanna; Teräs, Marko; Viteli, Jarmo A gamified eLearning approach to teaching food regulation 571 Teychenne, Danielle Pre-service teachers’ reflections on their participation in 1:1 laptop programs 577 Walker, Rebecca Maria; Blackley, Susan Ellen Concise Papers 7 Mind the Gap: Exploring knowledge decay in online sequential mathematics 582 courses Webby, Brian; Quinn, Diana; Albrecht, Amie; White, Kevin Clearing the Fog: A Learning Analytics Code of Practice 588 Welsh, Simon; McKinney, Stewart Dreaming of Electric Sheep: CSU’s Vision for Analytics-Driven Adaptive Learning 593 and Teaching Welsh, Simon; Uys, Philip SkillBox: a pilot study 599 Whitsed, Rachel Anne; Parker, Joanne Digital equity: A social justice issue for staff, not just students 604 Willems, Julie Concise Papers 8 Posters Page Number Metacognitive Development in Professional Educators: NZ teacher experiences 608 using mobile technologies in a tertiary education environment Abu Askar, Reem Digitise Your Dreams the Indigenous Way 612 Matthews, Aaron; Aggarwal, Rachna; Lim, Siew Leng Introducing StatHand: A Mobile Application Supporting Students’ Statistical 614 Decision Making Allan, Peter; Roberts, Lynne; Baughman, Frank E-learning, resilience and change in higher education: A case study of a College of 616 Business Ayebi-Arthur, Kofi; Davis, Niki; Cunningham, Una Enhancing Student Learning Outcomes with Simulation-based Pedagogies 618 Benckendorff, Pierre; Lohmann, Gui; Pratt, Marlene; Reynolds, Paul; Strickland, Paul; Whitelaw, Paul Creating concept vignettes as a module supplement for active and authentic 621 learning Chatterjee, Chandrima Preparing Students for Future Learning 623 Cheng, Jasmine; Payne, Sally; Banks, Jennifer The use of rubrics for the assessment of digital products in language learning 626 Cowie, Neil Developing an online challenge-based learning platform 629 Gibson, David; Scott, Katy; Irving, Leah Let’s Talk Learning Analytics and Student Retention 631 Heath, David; West, Deborah; Huijser, Henk Experiential Learning in Accounting: Engaging a diverse student cohort through the 633 use of role-plays Kerr, Rosemary; Taplin, Ross; Lee, Alina; Singh, Abhi The CSU Online Learning model 635 Klapdor, Tim MOOCs as spaces to innovate 637 Lockley, Alison Mobile devices in an Interprofessional Community of Practice #NPF14LMD 638 Mentis, Mandia; Holley-Boen, Wendy Posters 9 Technology for Learning: How Do Medical Students Use Technology for Education? 640 Moscova, Michelle; Porter, David Bruce; Schreiber, Kate The Flipped Teacher and the Flipped Learner Framework 642 Reyna Zeballos, Jorge Luis Enhancing Workplace Learning through Mobile Technology: Designing the GPS for 645 WPL Trede, Franziska; Markauskaite, Lina; Goodyear, Peter; Macfarlane, Susie; Tayebjee, Freny; McEwen, Celina Refocussing support on locally connected, digitally enabled communities of 648 practice Tull, Susan Enhancing Queensland Pre-service Teachers’ Self-efficacy to Teach STEM By the 650 Use of Remote Access Laboratories: A Mixed Methods Study Wu, Ting Posters 10 Discussion Papers Page Number Decisions and designs for building enterprise learning systems within an enabled 652 learning paradigm: The case of third party technologies Allan, Garry; Pawlaczek, Zosh 656 Designing for “Flexibility”: Exploring the Complexities of Dual-Mode Teaching Barac, Karin; Davies, Lynda; Boorer, Lenka Connecting or constructing academic literacies on Facebook 659 Bassett, M Technology issues in blended synchronous learning 664 Dalgarno, Barney; Bower, Matt; Lee, Mark J W; Kennedy, Gregor On the Evaluation of OLEs Using the HEART Framework 668 Flaounas, Ilias; Kokkinaki, Aikaterini A practitioner’s guide to learning analytics 672 Gunn, Cathy; McDonald, Jenny; Donald, Claire; Milne, John; Nichols, Mark; Heinrich, Eva STEMming the flow: content delivery through digital media for enhancing students’ 676 construction of knowledge Huber, Elaine The “I”s have it: Development of a framework for implementing Learning Analytics 680 Jones, Hazel Learning analytics - are we at risk of missing the point? 684 Liu, Danny Yen-Ting; Rogers, Tim; Pardo, Abelardo The impact of digital technology on postgraduate supervision 688 Maor, Dorit Is Student Transition to Blended Learning as easy as we think (and what do they 692 think)? Miles, Carol A Learning through doing: Creating a makerspace in the academic library 696 Miller, Karen Engaged and connected: embedding, modelling and practising what works 700 Woodley, Carolyn J; Kehrwald, Benjamin A Discussion Papers 11 Sharing Practice Papers Page Number Developing the Scholarship of Technology Enhanced Learning (SOTEL) 710 Cochrane, Thomas; Narayan, Vickel #NPF14LMD Learners and Mobile Devices: Sharing Practice 713 Cochrane, Thomas; Frielick, Stanley; Narayan, Vickel; Dee Sciascia, Acushla Learning Analytics Special Interest Group: Recognising Outstanding Practice in 716 Learning Analytics Lynch, Grace; Pardo, Abelardo; Welsh, Simon Easing into mobile learning 717 Murphy, Angela; Farley, Helen Sharing Practice Abstracts Clinical Logs: Best Practices in the Design and Implementation 719 Porter, David Bruce; Moscova, Michelle Institution wide information privacy frameworks to support academics in the use of 720 learning analytics Dobozy, Eva; Heath, Jennifer; Reynolds, Pat; Leinonen, Eeva Digitally enabled learning through Bb+ 721 Greenaway, Ruth; Mitchell, Maxine Using interactive multimedia for “flipped lecture” preparation: does it make a 723 difference to student learning? Moscova, Michelle; Kuit, Tracey; Fildes, Karen; Schreiber, Kate; Treweek, Teresa Attention as skill: Contemplation in online learning environments 724 Selvaratnam, Ratna Malar Applying Adaptive Comparative Judgement to videos as an indicator of ‘At Risk’ 726 teaching performance Geer, Ruth Elizabeth Vertical learning in Agricultural Science: It’s all fun and games until… 728 Yench, Emma; Grommen, Sylvia ascilite 2015 Reviewers 729 Sharing Practice 12 The Conceived, the Perceived and the Lived: Issues with 21st Century Learning and Teaching Karin Barac Griffith University, Australia A bespoke course design framework was implemented in an Australian university to help academics convert face-to-face courses to blended or online offerings in response to st increasing demand for universities to offer 21 century learning environments. While the design framework was grounded in evidence-based approaches that exemplify quality delivery, these course designs have had variable reactions from students in their implementation. As such, a student dimension to the evaluation of the framework was added and the findings from the initial pilot are reported here. It has been found that st students may not be as ready for 21 century learning and teaching practices as current rhetoric implies. This paper begins to formulate a theory to help resolve this through an exploration of ideas through the lens of Lefebvre’s production of space (1991). Keywords: Course Design, Student Expectations, Blended Learning, Higher Education Introduction Nationally and internationally universities are striving to attract and retain students through offering flexibility in study options as a response to the ever-increasing competitive environment. This idea of flexibility centres on the idea of study occurring at “any time, any place” allowing students to “balance” study with work and other life commitments. The increasing demand for flexibility in study options has seen a growth in online and blended learning offerings of courses (or units) within university programs. In the 21st century, one defined by rapidly advancing and ubiquitous digital technologies, it is now assumed that academics should be able to naturally incorporate these technologies into their teaching and learning practices (Koehler & Mishra, 2005). However, it has been found that the development of quality blended and online courses represents for many academics the need to not only acquire technical expertise but new pedagogical expertise (Caplan & Graham, 2004) as these learning models and frameworks have yet to be widely adopted by the academic community (Roby, Ashe, Singh, & Clark, 2012). Therefore the challenge facing many universities now, and in the future, is how to provide academics with the professional learning necessary to acquire these new skills so that the quality of course design is not adversely affected and rapid development can be achieved with little specialist support. As blended and online learning designs proliferate the success of these learning environments rely more and more on students accepting responsibility for their role in the learning environment. Research has shown, unfortunately, that as course designs move towards a blended approach students equate less time on campus with less time on task (Vaughan, 2007). We have found a dissonance between student expectations of their learning experience and their demand for flexibility. These divergent student perceptions are problematic given that, in design terms, flexibility relies on a move to student-centred approaches that use technologies to facilitate successful learning. “Designing Online Courses” Framework In 2012-13, the professional learning module “Designing Online Courses” was developed to provide a just-in-time support resource that encompasses both the pedagogical and technological perspectives of the course design process as it is argued that the process of design is the best environment for academics to learn new pedagogies because it allows them to adapt ideas to their own contexts (Bennett, Thomas, Agostinho, Lockyer, Jones, & Harper, 2011). This module serves to support academics in the process of converting a face-to-face delivery mode to an online one by giving them a strong pedagogical perspective on the curriculum design process thereby enabling them to make appropriate technological decisions when implementing the design. While this was originally conceived to apply to online courses we have found that the design framework is equally useful to FP:1 13 those employing blended designs. The first step in developing the module was to ground it in the theoretical frameworks that encompass quality online course design. The two frameworks selected were Community of Inquiry (COI) (Garrison, Anderson, & Archer, 2000), and Technological, Pedagogical, Content Knowledge (TPACK) (Mishra & Koehler, 2006) as they are well documented in educational research on quality online course design (Anderson, 2008; Garrison & Kanuka, 2004; Koehler & Mishra, 2005; Rubin, Fernandes, & Avgerinou, 2012; Wiesenmayer, Kupczynski, & Ice, 2008). It was also important that the content of the module was consumable for academics by providing practical examples that illustrate the theory in practice. This was a deliberate design choice as it has been acknowledged that academics generally do not have the time to take advantage of educational research (Price & Kirkwood, 2013) instead they rely on personal experiences or their conversations with colleagues (Dondi, Mancinelli, & Moretti, 2006; Macdonald & Poniatowska, 2011; Price & Kirkwood, 2013; Spratt, Weaver, Maskill, & Kish, 2003) to improve their practices. The primary objective in the module development was to break down the design process that is required to build courses into achievable steps. As such we defined five distinct, but ultimately interlinked, areas to stage the framework: Getting Started, Curriculum Design, Interaction Design, Assessment Design and Site Design (Barac, Davies, Duffy, Aitkin, & Lodge, 2013). These stages are designed and articulated purposefully to help academics see how content, interactions, activities, sense of community, assessments and teacher presence work together to ensure quality and effectiveness in online courses (Finch & Jacobs, 2012; Roby et al., 2012). The framework would therefore produce courses that would provide students “the time to think deeply and not speed over enormous amounts of content” (Vaughan, Cleveland-Innes, & Garrison, 2014, p. 20). Figure 1: Design Framework Once the module was designed and the content developed it was initially tested and piloted with a number of small groups of academics and it has now been deployed within the large faculty group at an Australian university. In 2014 the first courses designed under this framework were released to students with varying results particularly in those courses employing a fully blended approach. One academic reported to the project team that even though during the semester students were responding favorably to the teaching directions (that the staff had been encouraged to employ to make the environment successful) they nevertheless exhibited very strong negative reactions in the University’s end-of-course evaluation. It is for this reason that a student dimension was added to the evaluation plan for the module and framework that would evaluate the extent students were responding to the quality design factors employed in these courses in addition to the University’s process. Methodology Amundsen and Wilson (2012) found in their meta-analysis that the evaluation of academic development activities in higher education is still a developing field. Perhaps, because it is still a developing field there appears to be some gaps in the current literature: firstly, there seems to be a FP:2 14 concentration of evaluations being centred on participant satisfaction with the activities (Pierson & Borthwick, 2010) rather than investigating the content or application of the activities on their academic practice after completion (Desimone, 2009) and secondly, many of the studies lack rigor of research design (Lawless & Pellegrino, 2007). Consequently, the module evaluation uses a design-based research methodology to address these concerns as this paradigm is increasingly gaining acceptance in evaluating “learning in context” (The Design-Based Research Collective, 2003, p. 5). As a methodology Design-Based Research aims to refine educational theory and practice (Collins, Joseph, & Bielaczyc, 2004) by studying learning designs in action to connect “intended and unintended outcomes” (The Design-Based Research Collective, 2003, p. 7). As such the evaluation is multi-faceted and is being conducted as an iterative cycle of design, evaluation and re-design to align with this paradigm (Wang & Hannafin, 2005). It employs mixed- method approaches that involve both the academics participating in the professional learning module and the students that are enrolled in the courses that have been designed and delivered under the framework. The academic phase of the evaluation involves an online survey, an interview and an analysis of the comprehensive course plan that they complete as part of moving through the framework and module contents. The student phase involves a pre-course and mid-course online survey that largely consists of close-ended questions. The pre-course poll consists of four questions intended to gather students’ study goals for the course. (This poll also serves as a teaching activity that helps orientate the students to their role in the learning environment and gives the teaching team information they can feed into learning activities.) The mid-course poll has seven questions that deal directly with the online and blended components of the course design. This paper describes the student phase of the evaluation. Pilot Study A pilot study was conducted with a large first year undergraduate Law course in semester one of 2015 to test the mid-course survey instrument that will be used to gather data on student expectations and experiences within all courses designed under this framework. The pilot course was designed as a blended learning offering that had significant online content (videos, readings and quizzes) to be completed before the weekly workshop while some on-campus lectures were retained at key points in the semester to check-in with students. An online survey was deployed within the Blackboard course site in the last four weeks of semester. The total number of respondents was 123 students, which represented a 24% response rate from that cohort. Simple descriptive analysis was used on the quantitative questions while the qualitative comments where coded and analysed for themes and frequency using NVIVO. Findings The quantitative questions resulted in 123 responses while the open-ended comments question yielded 63 comments for analysis. In Table 1, the quantitative questions range of scores is reported. The majority of student responses show that students seemed to be largely satisfied with most components of the course. But there was also an alarming level of neutrality when answering the questions related to the blended and online components of the course. The use of the weekly formative quizzes that allowed students to test their knowledge of the content received 76% in the agree and strongly agree range. This is in line with the literature on online course design, which encourages the use of formative checkpoints with instant feedback loops to keep students on track. In an attempt to explore current students study goals in their courses the survey included a question on the number of hours a week they studied in the course. It was found that only 9% of respondents were studying 8-10 hours a week on this course. In fact, 68% of the students sat in the 3-8 hour range per week range, which is well below the university standard of 10 hours per week for a 10-credit point course (Griffith University, 2015). This is interesting, in light of the first result in Table 1 where the students reported high agreement on the guidance on their role in the course. A key component of this guidance was to embed messages on the study-time requirements of this course. This suggests that students may have a fundamental misunderstanding of the time commitment a university degree requires even when direct reference is made to the fact. FP:3 15 Table 1: Quantitative Results Agree – Neutra Disagree – Unanswere Question Strongly Agree l Strongly d Disagree There was clear guidance about my role 74% 16% 8% 2% as the learner, in the learning process in this course. The blend of face-to-face and online 50% 31% 18% 1% learning and teaching is effective for my learning in this course. The use of online technologies helps me 53% 28% 18% 1% learn in this course. This course effectively uses online 72% 16% 10% 2% assessment (e.g. quizzes) to help me learn. This course engages me in learning. 62% 25% 13% - There was clear guidance about the role 74% 16% 8% 2% of the L@G site for learning in this course. The teaching team members effectively 76% 16% 8% - communicate and connect with students. Analysis of the quantitative questions in comparison to the short answer comments reveals that students may hold conflicting ideas about the nature of learning and teaching in higher education. It was found that while 62% of respondents agreed or strongly agreed that the blend of face-to-face and online learning is effective for learning in this course, the qualitative comments contained more references to traditional forms of learning than those about flexibility or the blend of the learning environment. In fact, of the 63 comments supplied by the respondents there were 35 mentions of lectures, with nearly all centered on their reinstatement: - “I think I would have preferred to have a lecture every week, because I like the traditional mode of learning – i.e. face-to-face.” “I really enjoyed the workshops each week, but would have preferred a weekly lecture too!” “I believe that more lectures would have assisted my learning Maybe have lectures once a fortnight” In fact one student even went as far to request the reintroduction of “weekly lectures & do away with the online video [even if it was to] show the videos during weekly lectures so students can gain a grip on the material”. While the students were largely calling for the return of the traditional model there were some positive comments around the nature of blended learning and in particular where they felt it was better suited in the program structure. It was felt that the “independent learning structure … would be better suited for integration in second or third years.” This is something for universities and program design teams to take note of, as it suggests that blended learning can be well received if the students are properly scaffolded through the experience by gradually implementing these strategies. Following with the theme of lectures it was also extremely interesting to find that the mention of lectures was rarely connected to the online videos or vice versa. Comments such as the following show a disconnect between the ideas of “lecture”, “content” and “teaching” in today’s students: - "As a foundational subject, I think it is a wrong decision to only have sporadic lectures when this subject should be laying a solid, in depth foundation of law" FP:4 16 “I just felt like we skimmed over topics because of the lack of lectures.” “I would like to see more lectures as i [sic] feel the workshops were not enough. I didn't like the workshops or the online videos. I often thought the workshops were ineffective. I would prefer a lecture every week where the content and information taught was clear.” This failure to connect the online videos and activities with “lecture” material, (or even teacher presence), is particularly concerning and could severely limit the successful implementation of blended learning with today’s students. Discussion In an effort to explain this dissonance between the academic-driven ideas of “quality” 21st century learning and the reality of current student expectations let us explore Lefebvre ideas of space – space as a construct of the conceived, perceived and lived (Lefebvre, 1991). These ideas were first posed in terms of urban design but have been appropriated by educational researchers as conceptual tools (Middleton, 2014) it appears that this paper is one of the first to apply Lefebvre’s model as a concept to help explain the issues surrounding the application of technology-enabled pedagogies in higher education. Lefebvre expanded the idea of space from its geometric definition as an ‘empty area” to that of a mental construct linked to the physical. This model of space is one into which we bring our own ideas; or others define the meaning for us; or is a reality that we construct by participating together as members of a society. In particular he sought to code and explain the “interaction between ‘subjects’ and their space and surroundings” (Lefebvre, 1991, pp. 17-18). He saw this as being an interaction of the conceived space, perceived space and the lived space or the theoretical, the mental and the social. Specifically, the conceived space is the mental and abstract enclosures constructed by “professionals and technocrats” (Middleton, 2014, p. 11). In our context of learning and teaching space, our subjects are the academics and students, where academics operate and control the conceived realm through their course designs and delivery. The perceived realm incorporates the pre-conceptions and expectations the different subjects have within the environment and the lived is the reality of the subjects operating within that space. Ideally, the three are interconnected states that allow subjects to move from one to the other without confusion. The three domains are seen to constitute a whole “when a common language, a consensus and a code can be established” (Lefebvre, 1991, p. 40). Figure 2 attempts to conceptualise the different pathways (positive and negative) that subjects can take through these realms and where breakdowns might happen. FP:5 17 Figure 2: Conceptualising Academic and Student Paths through Lefebvre’s Realms Optimally both academic and student pathways will be positive if there is a shared understanding between the conceived and the perceived. However, from our current exploration of the data we can see that academics and students are not in this state of the interconnected whole within the learning and teaching environment. It would seem a schism could occur when the pathways cross the conceived into the perceived that can result in a negative experience for the students where academics believe positive outcomes should be occurring. In particular, at this point in time it does not seem that academics and students share a common language or consensus in what the optimum learning environment should be. Future Directions Based on this analysis and exploration through Lefebvre’s lens it would seem more work is needed to st close the gap between the conceived and the perceived for academics and students in 21 century learning and teaching spaces. We need to foster a common understanding through language, symbols and signs. One such way we believe we can help foster this is through the incorporation of infographics into our course designs that help to break down student (and academic) preconceptions of the higher education learning environment and orientate them to the new design frameworks. These infographics will serve to highlight student and staff responsibilities in the learning and teaching space and to raise the awareness of how contact and independent study has been transformed from the traditional lecture/tutorial model. The following image is a prototype we are developing to help orientate students to the nature of teacher-student contact in a blended learning space and that the online content (i.e. videos) is in fact a form of teacher presence. FP:6 18 Figure 3: Student Infographic Prototype (Student-Teacher Contact in a Blended Learning Course) There are currently 89 academics actively using the “Designing Online Courses” framework as a professional development activity. There are currently 19 courses that are specifically being designed under this framework with our specific guidance (and evaluation procedures) that will be implementing these infographics for 2016. Data collection will continue within these courses to provide more data to validate these ideas. Excitingly, the university will be implementing a learning analytics system in 2016 that we have identified as an opportunity to explore the lived experience of the course sites that may provide additional context to university student experience surveys. References Amundsen, C., & Wilson, M. (2012). Are we asking the right questions? A conceptual review of the educational development literature in higher education. Review of educational research, 82(1), 90- 126. Anderson, T. (2008). Towards a theory of online learning. Theory and practice of online learning, 45- 74. Barac, K., Davies, L., Duffy, S., Aitkin, N., & Lodge, J. (2013). Five stages of online course design: Taking the grief out of converting courses for online delivery. Paper presented at the Ascilite, Sydney. Bennett, S., Thomas, L., Agostinho, S., Lockyer, L., Jones, J., & Harper, B. (2011). Understanding the design context for Australian university teachers: Implications for the future of learning design. Learning, Media and Technology, 36(2), 151-167. Caplan, D., & Graham, R. (2004). The development of online courses. Theory and practice of online learning, 175. Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. The Journal of the learning sciences, 13(1), 15-42. Desimone, L. M. (2009). Improving impact studies of teachers’ professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181-199. Dondi, C., Mancinelli, E., & Moretti, M. (2006). Adapting existing competence frameworks to higher education environments. In I. Mac Labhrainn, C. McDonald Legg, D. Schneckenberg, J. Wildt (Eds.), The Challenge of eCompetence in Academic Staff Development, 19-28. FP:7 19 Finch, D., & Jacobs, K. (2012). Online Education: Best Practices to Promote Learning. Paper presented at the Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2), 87-105. Garrison, D. R., & Kanuka, H. (2004). Blended Learning : Uncovering its transformative potential in higher education. Internet and Higher Education, 7, 95-105. Griffith University. (2015). Student Administration Policy. Koehler, M. J., & Mishra, P. (2005). Teachers Learning Technology by Design. Journal of computing in teacher education, 21(3), 94-102. Lawless, K. A., & Pellegrino, J. W. (2007). Professional development in integrating technology into teaching and learning: Knowns, unknowns, and ways to pursue better questions and answers. Review of educational research, 77(4), 575-614. Lefebvre, H. (1991). The Production of Space (D. Nicholson-Smith, Trans.): Oxford: Blackwell. Macdonald, J., & Poniatowska, B. (2011). Designing the professional development of staff for teaching online: an OU (UK) case study. Distance Education, 32(1), 119-134. Middleton, S. (2014). Henri Lefebvre and Education. Oxford: Routledge. Mishra, P., & Koehler, M. J. (2006). Technological Pedagogical Content Knowledge: A Framework for Teacher Knowledge. Teachers College Record, 108(6), 1017 - 1054. Pierson, M., & Borthwick, A. (2010). Framing the Assessment of Educational Technology Professional Development in a Culture of Learning. Journal of Digital Learning in Teacher Education, 26(4). Price, L., & Kirkwood, A. (2013). Using technology for teaching and learning in higher education: a critical review of the role of evidence in informing practice. Higher Education Research and Development, (in press). Roby, T., Ashe, S., Singh, N., & Clark, C. (2012). Shaping the online experience: How administrators can influence student and instructor perceptions through policy and practice. The Internet and Higher Education. Rubin, B., Fernandes, R., & Avgerinou, M. D. (2012). The effects of technology on the Community of Inquiry and satisfaction with online courses. The Internet and Higher Education. Spratt, C., Weaver, D., Maskill, L., & Kish, K. (2003). Online pedagogy and the challenges for academic staff development. Paper presented at the Exploring Educational Technologies Conference - From Strategy to Implementation, Monash University, Melbourne, Australia, July 16- 17, 2003. The Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 5-8. Vaughan, N. (2007). Perspectives on blended learning in higher education. International Journal on E- Learning, 6(1), 81-94. Vaughan, N., Cleveland-Innes, M., & Garrison, D. R. (2014). Teaching in Blended Learning Environments. Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational technology research and development, 53(4), 5-23. Wiesenmayer, R., Kupczynski, L., & Ice, P. (2008). The Role of Technical Support and Pedagogical Guidance provided to Faculty in Online Programs: Considerations for Higher Education Administrators. Online Journal of Distance Learning Administration, 11(4). Barac, K. (2015). The Conceived, the Perceived and the Lived: Issues with 21st Century Learning and Teaching. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:1-FP:8). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:8 20 Learning design for science teacher training and educational development Ole E. Bjælde Michael E. Caspersen Mikkel Godsk Centre for Science Education Centre for Science Education Centre for Science Education Aarhus University, Denmark Aarhus University, Denmark Aarhus University, Denmark Rikke F. Hougaard Annika B. Lindberg Centre for Science Education Centre for Science Education Aarhus University, Denmark Aarhus University, Denmark This paper presents the impact and perception of two initiatives at the Faculty of Science and Technology, Aarhus University: the teacher training module ‘Digital Learning Design’ (DiLD) for assistant professors and postdocs, and the STREAM learning design model and toolkit for enhancing and transforming modules. Both DiLD and the STREAM model have proven to be effective and scalable approaches to encourage educators across all career steps to embrace the potentials of educational technology in science higher education. Moreover, the transformed modules have resulted in higher student satisfaction, increased flexibility in time, pace, and place, and in some cases also improved grades, pass rates and/or feedback. Keywords: learning design, science education, teacher training, educational development Introduction Since the early 00s learning design has gained momentum as an approach to educational development in higher education. The learning design approach provides tools and models that can help educators pedagogically inform and share teaching practices and, when used for educational technology, help qualify the transformation of traditional teaching into blended and online learning. In addition, learning design also helps defeating well-known barriers in more conventional ad hoc approaches to educational development such as missing sustainability of initiatives and the missing link between educational research and practice (Conole, 2013; Cross et al., 2008; Godsk, 2015; Koper & Tattersall, 2010; Laurillard, 2012; Nicol & Draper, 2009). Centre for Science Education (CSE), the pedagogical development unit at Faculty of Science and Technology (ST), Aarhus University, has adopted a strategic approach with a focus on (1) development issues that resonate with educators and (2) solutions that are effective, efficient, and supported by solid research (Vicens & Caspersen, 2014). In order to facilitate this approach and optimise its impact and scalability, a framework-based learning design approach has been adopted. With this approach the educators are active developers of their own practice, and potentially producing reusable and sharable materials and practices (Conole, 2013; Cross et al., 2008; Godsk, 2015; Koper & Tattersall, 2010; Laurillard, 2012). The STREAM model as learning design Faculty of Science and Technology (ST) is one of the four faculties as Aarhus University and has approx. 7,000 students and 1,650 full time academic staff (full-time equivalent) (Aarhus University, 2015). At CSE the aim for educational development is to provide educators with an open-ended learning design, where essential pedagogy-informed aspects of the learning designs are fixed while other aspects are open for variability. The open-ended learning design approach is carefully developed and conveyed particularly regarding efforts in technology-based educational development. In practice this is actualised by means of a learning design framework designed for this and similar settings: ‘the STREAM model’ (Godsk, 2013; Figure 1). ‘STREAM’ is an acronym for ‘Science and Technology Rethinking education through Educational IT towards Augmentation and Modification’, where the terms ‘augmentation’ and ‘modification’ refer to two different levels of blended learning (Godsk, 2014a; Puentedura, 2010). The STREAM model is based on well-tested and acknowledged FP:9 21 teaching strategies for science higher education such as just-in-time teaching (Novak et al., 1999), active learning (Bonwell & Eison, 1991), flipped classroom, peer instruction (Mazur & Hilborn, 1997), and socio-cultural theories used particularly to inform and qualify the apprenticeship between learners (apprentice) and more experienced peers (co-learners and educators) (Fjuk et al., 2004). The model provides an outline of how a module may be transformed into blended and online learning using feedback loops, online out-of-class activities, in-class and online follow-up, and suggests tools and technologies that support the design. In addition to the STREAM model, a toolkit is provided for the educators consisting of a webcast recording facility and a media lab providing easy production of the materials needed for the transformation of modules and technical support, respectively. Figure 1: The STREAM model The STREAM model is currently being used for the transformation of modules, and it is being disseminated through individual meetings with educators, workshops, websites, the teacher training programme, and department meetings. Thus, the STREAM model functions as both a pedagogical framework and an organisational change agent. This is reflected in two major initiatives targeting two different groups of educators: • The teacher training programme, ‘Digital Learning Design’, for assistant professors and postdocs. The programme introduces educational technology and learning design including the STREAM model. • STREAM as a stand-alone learning design model and toolkit for ad hoc assistance to professors and associate professors and their transformation of modules with educational technology. Learning Design in Teacher Training Teaching at Aarhus University is predominated by face-to-face activities including lectures, small class teaching, laboratory teaching, etc. However, it is a specific aim in the university policy to rethink FP:10 22 existing teaching practice with technology (Aarhus University, 2011). To pursue this aim a module on educational technology was included in the mandatory teacher training programme in 2012. The Teacher Training programme is offered primarily to assistant professors and postdocs and counts for 5 ECTS (European Credit Transfer and Accumulation System, 1 ECTS credit corresponds to 25-30 hours of work) (European Union, 2015). The programme includes four mandatory modules of which three are common to participants throughout the university, while the module on educational technology is organised differently for each individual faculty. At ST this module is DiLD and has a workload of 30 hours (1 ECTS credit equivalent to approximately 1.5 hours of participation per weekday during the module). The objective is outlined in the overall module description: The objective of the [DiLD module] is to give an introduction to Educational IT and Educational Technology at Faculty of Science and Technology (ST), Aarhus University. During the module participants will be introduced to the potentials of using different technologies in teaching and it will be demonstrated how technology supported teaching can be designed. The participants will be introduced to the services provided within educational IT at ST and they will develop a digital learning design to be used in their own teaching. (Godsk et al., 2014; p. 1) The DiLD module is designed according to the STREAM model and implemented in the institutional learning management system (LMS), Blackboard Learn (Figure 2). The module consists of four weeks of flexible, entirely online learning (except for a concluding session) and introduces a range of educational technologies and learning design models. By demonstrating how educational technology has a potential to increase the learner flexibility, the module gives the participants a first-hand experience with online learning and serves as inspiration for the participants’ own teaching (Godsk et al., 2013). Each week consists of a learning path of 6-12 steps with 4-6 activities. The activities aim to build upon participants’ existing teaching experience and support the development of their own teaching practice and materials in order to make the module directly applicable (Godsk et al., 2013). Though most participants are not currently teaching online modules; both the institutional strategy for technology in education (Aarhus University, 2011) and the fact that educators are including an increasing number of online elements such as video, online discussion forums, and online assignments in their teaching practice highlight the importance of being proactive by also pedagogically informing their future uptake of technology. As such the DiLD module format serves two purposes: to give as much flexibility as possible to the participants and to illustrate the design of an online module. As prescribed by the STREAM model, DiLD is designed with a continuous interplay between readings, articles, videos, etc. and active learning through participation in moderated discussions and wikis. By mixing individual exploration of online materials and participatory learning, such as asynchronous discussions and peer-feedback, the module design ensures a balance between acquisition of new knowledge, and collaboration and participation (Brown et al., 1989; Lave & Wenger, 1991; Sfard, 1998). The readings and activities are interlinked with a narrative about the topic of the relevant week to bring the reading and activities into a cohesive whole (Weller, 2002) and at the end of each week the activities and readings are wrapped up by the e-moderators through an e-mail send to the participants via the LMS. The subsequent week is then adjusted according to the needs and interests of the participants. The basic idea is to support a progressive learner role where participants progress from being a learner to a designer of digital learning activities through active participation during the module (Lave & Wenger, 2003; Salmon, 2011). FP:11 23 Figure 2: Week 1’s learning path of ‘Digital Learning Design’ as implemented in Blackboard Learn. The module culminates with each participant developing an individual learning design for their own teaching practice describing both concept and materials. The design is then presented at a concluding poster session where peer-feedback is received. In developing the learning designs, the participants are encouraged to adopt an existing learning design approach, such as the STREAM model, the Five-stage Model (Salmon, 2011), or a model for structured discussions for their own teaching development (Sorensen, 2005), or develop their own according to the presented theory. In the individual learning design, module participants identify components of their current teaching practice that need to be transformed or enhanced with educational technology, a suitable learning design model, and relevant technology such as webcasts, lecture captures, learning paths, online discussions, and online exercises. In addition, the participants set the level of the transformation in terms of the revised SAMR model which operates with four levels of transformation of traditional teaching ranging from ‘substitution’, where the technology merely substitutes existing teaching practices, to ‘augmentation’ referring to settings where ‘educational technology is used for enhancing activities or transforming components’ (Godsk, 2014a; p. 184), ‘modification’ referring to where the technology is ‘used for transforming entire activities’ (Godsk, 2014a; p. 184), to ‘redefinition’ where technology is used to completely transform or reinvent the teaching practice (Godsk, 2014a). The efforts associated with running the module, consist of on-going update of the content, moderation and summing up of online discussions, communication with the participants, individual supervision and feedback, organising the poster-presentation, and various administrative tasks and evaluation. This workload is shared between a handful of e-moderators and the module chair and estimated to 504 hours annually (two DiLD modules per year). In addition, the media lab assists the facilitation by organising an online workshop in video conferencing and supporting the participants with technical issues. This assistance is estimated to 75 hours annually. The costs for handling the enrolment, providing a LMS, and providing basic IT support are defrayed by the Educational Development Network and the IT department. The Participants’ Perception of Learning Design The participants were primarily employed as postdocs (40%) or assistant professors (30%) and their teaching experience ranged from experienced lecturers responsible for modules with more than 100 students to postdocs or researchers giving occasional lectures and being involved in project supervision of students. According to a pre-survey carried out in connection with the last two runs of FP:12 24 the module, 7% said they had heard, read about or had first-hand experience with learning design, 5% had used educational technology to transform parts of their teaching to online teaching and 0% had used educational technology to teach entire modules online. At this point it is still not possible to measure the impact of the DiLD module on teaching and learning or the success of using learning design for teacher training. However, indications on how the participants perceived the module is provided by evaluation data collected after the last four repetitions of the module (Autumn 2013, Spring 2014, Autumn 2014, and Spring 2015). The collected data represents 20, 16, 31, and 9 module participants, respectively. In total the data basis is 76 module participants. The module evaluation addresses the participants’ prior experiences with educational technology and learning design, the evaluation of the module, the participants’ perceived learning outcomes, their perception of educational technology and learning design, and a survey of their future plans for adoption. When asked about perceived skills acquisition during the module a majority of participants expressed that the module had enabled them to design and develop blended learning (83%) and transform traditional teaching into blended or online teaching (73%). Most participants agreed or strongly agreed that they gained insight into relevant educational technologies and pedagogical methods and theories (80%) and were able to evaluate the potential of using educational technology in their own teaching (88%). 82% agreed or strongly agreed with the statement: ‘the content of this module is relevant for my own teaching’ and 70% of the participants expressed that their perceived learning outcome during the module was high. In addition, the intended transformational level according to the revised SAMR model provided an indication of an ambitious use of technology. Scrutinising the individual learning designs revealed that 84% aimed at augmenting, 7% modifying, 7% redefining, and 2% substituting their teaching practice with technology. Bearing in mind that Aarhus University is a traditional, campus-based university with an insignificant amount of distance learning, the transformational levels witness a general high level of ambition for educational technology. The individual learning designs also revealed a highly diverse but generally very ambitious and intense use of educational technologies such as videos, discussion forums, learning paths, and peer instruction tools. Various kinds of video formats (30% of individual learning designs) such as webcasts, lecture captures, screencasts, and pencasts, peer instruction tools (15%) such as PeerWise (Denny et al., 2008) and curriculearn (Brodersen, 2014), and the use of learning pathways (14%) were particularly prevailing. The individual learning designs indicated a pronounced uptake of the presented learning design models and in particular the STREAM model. In practice, this meant that more than 80% adopted the STREAM model for their learning design with the remaining 20% split evenly between a completely new learning design model and other existing learning design models such as the Five-stage Model (Salmon, 2011) or a model for structured discussions (Sorensen, 2005) which they found relevant to their own teaching practice (Figure 3). Prospectively, 80% of the participants in the last two runs of the module (i.e. Autumn 2014 and Spring 2015) expressed in the evaluation that they had plans to adopt learning design in their teaching practice within the next year or more, and 45% within the next 6 months. Figure 3: Perceived relevance of the three Figure 4: Potential of educational technology presented learning design models. and learning design in science education. FP:13 25 In spite of the participants’ limited prior experiences with educational technology and learning design, the module led to a highly positive attitude. According to the module evaluations, the participants spent an average of 34 hours on the module (median 35 hours) ranging from 10-87 hours, a bit more than the estimated 30 hours (~1 ECTS) and what was required. Furthermore, most module participants saw a potential for both educational technology (93%) and learning design (88%) in science education (Figure 4). Transforming Modules with Learning Design Besides the DiLD module for assistant professors, the STREAM model and its toolkit are used, presented, and referred to through various channels aiming at all educators. It serves as a reference at meetings with educators, the locally held Frontiers in Science Education 2014 conference, invited talks and workshops on educational technology, development meetings with the educational committees at the faculty, freely available online resources on STREAM (e.g. Godsk, 2015b), and published papers on the topic (cf. Godsk, 2013; 2014a). Furthermore, STREAM has also been a prominent part of educational development meetings with all twelve educational committees at ST in the spring of 2015. Most associate professors and professors are highly self-governed with regards to their teaching practice and uptake of technology and STREAM may be used without CSE’s knowledge inspired by a conference, a workshop, the website, etc. Hence, the full extent of the impact of the STREAM model and toolkit is unknown. For transformations where the educator has been in direct dialogue with CSE, however, the impact on teaching and learning has been assessed. An overview of the completed transformations and their institutional impact in ECTS credits and full-time equivalents (FTEs) as well as impact on students’ learning is provided in Table 1. Institutional impact is expressed in ECTS credits and calculated as (the number of students) x (the number of ECTS credits associated with the module). One FTE corresponds to 60 ECTS. Table 1: The STREAM transformations’ institutional impact and impact on learning. Module Learning Design Institutional Impact on students’ learning impact Calculus 2, The module was Approx. 60% of the The evaluation of the module and examination 2013 modified by replacing 1,184 students results showed that the online students (undergraduat all lectures with followed the obtained significantly better examination e, 5 ECTS) learning paths transformed results, better pass rates, and were significantly containing webcasts, module. I.e. more satisfied with the learning compared to MCQs, reflection approx. 710 the face-to-face students (cf. Godsk, 2014b). exercises, and online students, 3,550 follow-up in Dokeos ECTS/59.2 FTEs LMS. Astrophysics, The module was 123 students, The module evaluation indicated a high 2013 augmented by 615 ECTS/10.3 satisfaction with the format (70 % of the (undergraduat supplementing lectures FTEs students responded that they referred the e, 5 ECTS) with webcasts, learning transformed format to traditional lectures) and paths, online activities, provided evidence of an increased degree of and online feedback in flexibility in time and place, support for Blackboard Learn. repetition and examination preparation, and more time for discussion during lectures (Godsk, 2014a). Microbial The module was 25 students, The end-of-module evaluation indicated a high Physiology modified by replacing 250 ECTS/4.2 student satisfaction (76% preferred the and all lectures with FTEs transformed format to traditional lectures) and a Identification, webcasts structured in higher degree of flexibility in time, place, and 2014 learning paths in pace. 87% most frequently watched the (undergraduat Dokeos. webcasts outside regular teaching hours. e, 10 ECTS) Evolution and The module was 123 students, N/a. Diversity, augmented by 615 ECTS/10.3 2014 transforming parts of FTEs (undergraduat the lectures into e, 5 ECTS) webcasts. FP:14 26 Calculus 1, The module was 1,048 students, The end-of-module evaluation indicated a high 2014 modified by replacing 5,240 ECTS/87.3 student satisfaction (51% preferred the (undergraduat all lectures with FTEs transformed format to traditional lectures), a e, 5 ECTS) learning paths higher degree of flexibility in time, place, and containing webcasts, pace, and a wide utility of the learning paths. MCQs, reflection 81% found that the online activities supported exercises, and online their understanding. follow-up in Blackboard Learn. Calculus 2, Modified as described 821 students, The end-of-module evaluation indicated high 2014 for Calculus 2, 2013. 4,105 ECTS/68.4 student satisfaction (50% preferred the (undergraduat FTEs transformed format to 31% preferring traditional e, 5 ECTS) lectures), a higher degree of flexibility in time, place, and pace, and a wide utility of the learning paths. Astrophysics, The module was 125 students, The examination results and the module 2014 augmented by 625 ECTS/10.4 evaluation provided evidence of a high student (undergraduat replacing lectures and FTEs work rate and satisfaction (85% very satisfied or e, 5 ECTS) 25% of the final satisfied with learning outcome, 76% preferred assessment with the new assessment format), lower fail rates webcasts, learning (50% lower than the previous year) and a wide paths, assessed online use of the flexibility offered. activities, and online feedback in Blackboard Learn. Microbial The module was 12 students, The end-of-module evaluation indicated high Physiology modified by replacing 120 ECTS/2 FTEs degree of flexibility in time, place, and pace. and all lectures with 50% used the webcasts for assignment work Identification, webcasts in and examination preparation. However, only 2015 Blackboard Learn. 25% preferred the transformed format to (undergraduat traditional lectures. e, 10 ECTS) Evolution and The module was 117 students, N/a. Diversity, augmented by 585 ECTS/9.8 2015 transforming parts of FTEs (undergraduat the lectures into e, 5 ECTS) webcasts. In total 9 modules were Approx. 15,705 An overall positive impact on students’ learning, delivered augmented ECTS (261.75 including an increased student satisfaction, a or modified using FTEs) were higher degree of flexibility in time, place, and STREAM. impacted by pace, and in some cases also improved grades learning design. and/or pass rates. To promote the STREAM model and toolkit and help the educators with the adoption, a number of resources have been developed. This includes a website (Godsk, 2015b) with a short introduction to the model, its potential for improving teaching and learning, its practical benefits, a list of already transformed modules and their incentives, and a 6 minutes long video introducing the model and how it is applied. The website and video were launched 6 January 2014 and until now (25 June 2015), the website has been accessed 659 times and the video played 110 times, which is equivalent to an average of 37 views of the website and 6-7 plays of the video per month. In addition, a short learning path has been developed and provided to the 46 educators signed up to the resource page in the LMS. Finally, the educational results and information about the STREAM model and transformations were disseminated to the 213 subscribers of quarterly newsletters of which approximately 30 were educators at the faculty. A press release was issued on the transformation of Calculus, which resulted in news coverage in two media (Loiborg, 2014; Stiften, 2014) and publication of three academic papers, two conference papers (Godsk, 2013; 2014a) and one journal paper (Godsk, 2014b). In total, the initiatives have reached a large portion of the educators at ST through one channel or another and the vast majority of all undergraduate students. Using Learning Design for Educational Development with Technology Using a framework-based learning design approach, exemplified by the STREAM model and toolkit, has demonstrated a number of advantages: FP:15 27 1. STREAM provides a uniform and common language to articulate educational development in the initial phase of implementation as well as later phases of refinements and exchange of experience; 2. STREAM provides the opportunity to more uniformly facilitate technology-based educational development through standard templates and guidelines; 3. the overall learning design (the fixed/invariant parts) is developed by educational experts who can prioritise, integrate and balance the various aspects in an optimal overall design; 4. the specific learning design (refinement of the variant parts) is left to the educators to accommodate specific needs. These can be subject-specific needs or individual preferences or beliefs (still maintaining a common denominator among the learning designs). In addition, the STREAM model has at least two build-in potential advantages: 5. STREAM provides a common structure that addresses analytical and management issues (quality assurance, accreditation, etc.); 6. STREAM ensures a common and recognisable overall LMS structure for students while still providing opportunities for detailed variation to accommodate individual needs and preferences. Some of these advantages are common to many learning design practices in general. This includes the potential to provide a common language for sharing teaching and learning practices, the ability to operationalise the pedagogical knowhow of the educational experts and accommodation of the development of individual learning design according to and by the educators themselves (Agostinho, 2006; Cross & Conole, 2009; Godsk, 2015a; Koper & Tattersall, 2010; Laurillard, 2012; Mor & Winters, 2007). Though the STREAM model is designed with a specific context in mind, the fact that the model is build on well-tested approaches to educational development and a strong research base within the area of learning design, the experiences and findings should apply in other teaching contexts as well. Hence, the authors strongly recommend a learning design approach to educational development with technology, including the STREAM model as the concrete learning design model. Conclusions The educational development effort at Faculty of Science and Technology, Aarhus University, revolves around a learning design approach and in particular the STREAM learning design model. This has proven an effective way of getting educators at the faculty to embrace the potentials of educational efforts, as, for instance, reflected in the fact that 93% of assistant professors and postdocs participating in the Digital Learning Design module see a potential for educational technology in science education, 88% see a potential for learning design, and that 80% expect to adopt learning design within the next year or more. 68% find STREAM relevant to their own teaching practice and the majority feel that the Digital Learning Design module has enabled them to transform, design, and teach with educational technology. The associate professors and professors are exposed to the topic of educational technology and learning design through a string of activities ranging from small meetings to conferences. The process of sharing practices and ideas, including the STREAM learning design model, through many different initiatives has made it possible to reach a large portion of the educators. Furthermore, the process has resulted in a series of transformations, which, judging from the institutional impact and impact on students’ learning, have been highly successful resulting in increased student satisfaction, a higher degree of flexibility in time, place, and pace, and in some cases also improved grades and/or pass rates for a large number of students/FTEs. As an added bonus, the results have led to a persistent inflow of new educators interested in transforming their teaching practice with educational technology and the STREAM model. At this point, the experiences with learning design in terms of the DiLD module and the STREAM model are positive and suggest that learning design is a suitable, scalable, sustainable, and effective approach to educational development for implementing educational technology in science higher education. The approach has demonstrated its practicality and effectiveness for engaging educators in the transformation of traditional teaching practice into blended and online learning, and that a relatively limited institutional effort has the potential to stimulate a highly positive attitude and high FP:16 28 ambitions towards educational technology among science educators. Now, the mission is to measure the actual uptake of learning design among the assistant professors and ensure the continued inflow of professors interested in transforming their teaching practice with technology. Acknowledgements The authors wish to thank Klaus Thomsen (Department of Mathematics), Kai Finster (Department of Bioscience), Tove Hedegaard Jørgensen (Department of Bioscience), and Science Media Lab at Faculty of Science and Technology, Aarhus University. References Agostinho, S. (2006). The use of a visual learning design representation to document and communicate teaching ideas. Proceedings of the 23rd annual ascilite conference: Who’s learning? Whose technology? Retrieved from http://www.ascilite.org.au/conferences/sydney06/proceeding/pdf_papers/p173.pdf. Biggs, J. & Tang, C. (2011). Teaching for Quality Learning at University. Society for Research into Higher Education and Open University Press, 4th edition. Bonwell, C. C., & Eison, J. A. (1991). Active learning: Creating excitement in the classroom. Washington, DC: School of Education and Human Development, George Washington University. Brodersen, D.E. (2014). curriculearn.dk. Retrieved from http://curriculearn.dk/src/index.php. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18 (1), 32–42. Denny, P., Luxton-Reilly, A., Hamer, J. (2008). The PeerWise System of Student Contributed Assessment Questions. Proceedings of the Tenth Conference on Australasian Computing Education (ACE 2008), 78, 69-74, (Wollongong, NSW, Australia). Conole, G. (2013). Designing for learning in an open world. New York: Springer. Cross, S., & Conole, G. (2009). Learn about learning design. Part of the OU Learn about series of guides, The Open University: Milton Keynes. Retrieved November 21, 2013, from http://www.open.ac.uk/blogs/OULDI/wp-content/uploads/2010/11/Learn-about-learning- design_v7.doc. Cross, S., Conole, G., Clark, P., Brasher, A., & Weller, M. (2008). Mapping a landscape of Learning Design: identifying key trends in current practice at the Open University. In: 2008 European LAMS Conference, 25-27 June 2008, Cadiz, Spain. Retrieved from http://oro.open.ac.uk/18640/5/CAD08_022_Final.pdf. European Union (2015). ECTS Users’ Guide. Luxembourg: Publications Office of the European Union. ISBN 978-92-79-43559-1 doi:10.2766/87192 Fjuk, A., Berge, O., Bennedsen, J. B., & Caspersen, M.E. (2004). Learning Object-Orientation through ICT-mediated Apprenticeship, Proceedings of the fourth IEEE International Conference on Advanced Learning Technologies, ICALT 2004, Joensuu, Finland, pp. 380-384. Godsk, M. (2013). STREAM: a Flexible Model for Transforming Higher Science Education into Blended and Online Learning. In T. Bastiaens & G. Marks (Eds.), Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2013 (pp. 722-728). Chesapeake, VA: AACE. Godsk, M., Hougaard, R. F., & Lindberg, A. B. (2013). Teaching Online Teaching Online: Seven Pedagogical Principles for Teacher Training. In World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, 2013, (1), 95-104 Godsk, M. (2014a). Efficient learning design - concept, catalyst, and cases. In B. Hegarty, J. McDonald, & S.-K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014 (pp. 182-189). Godsk, M. (2014b). Improving Learning in a Traditional, Large-Scale Science Module with a Simple and Efficient Learning Design. European Journal of Open, Distance and E-Learning, 17 (2), 143- 159. Godsk, M., Bjælde, O. E., Hougaard, R. F., Lindberg, A. B., & Vicens, Q. (2014). Digital Learning Design 2014, Retrieved from https://aarhus.blackboard.com/webapps/blackboard/content/listContent.jsp?course_id=_29594_1& content_id=_178473_1 FP:17 29 Godsk, M. (2015a). Understanding and Defining ‘Learning Design’: Seven Common Characteristics and Two Continua. Manuscript submitted for publication. Godsk, M. (2015b, September 09). Undervisningsudvikling vha. Educational IT (STREAM). Retrieved from http://cse.au.dk/forskning/projekter/stream/. Koper, R., & Tattersall, C. (Eds.) (2010). Learning Design. A Handbook on Modelling and Delivering Networked Education and Training. Berlin: Springer. Laurillard, D. (2012). Teaching as a design science: Building pedagogical patterns for learning and technology. New York and London: Routledge. Lave, J. & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press. Loiborg, C. (2014). Aarhus Universitet klar til videoundervisning i differentialligninger. Ingeniøren. Retrieved from http://ing.dk/artikel/aarhus-universitet-klar-til-videoundervisning-i- differentialligninger-169643 Mazur, E., & Hilborn, R. C. (1997). Peer instruction: A user's manual. Physics Today. 50 (4), 68-69. Mor, Y., & Winters, N. (2007). Design approaches in technology-enhanced learning. Interactive Learning Environments, 15 (1), 61-75. Nicol, D., & Draper, S. (2009). A blueprint for transformational organisational change in higher education: REAP as case study. Transforming higher education through technology-enhanced learning. Heslington: Higher Education Academy. Novak, G. M., Patterson, E. T., Gavrin, A. D., & Christian, W. (1999). Just-in-Time Teaching: Blending Active Learning with Web Technology. Upper Saddle River, NJ: Prentice Hall. Price, L. & Kirkwood, A. (2011). Enhancing professional learning and teaching through technology: a synthesis of evidence-based practice among teachers in higher education. York, UK: Higher Education Academy. Puentedura, R. (2010). SAMR and TPCK: Intro to Advanced Practice. http://hippasus.com/resources/sweden2010/SAMR_TPCK_IntroToAdvancedPractice.pdf (last accessed 22 September 2015). Salmon, G. (2011). E-moderating. The Key to Teaching and Learning Online. 3rd Edition. New York and London: Routledge. Sfard, A. (1998). On two metaphors for learning and the dangers of choosing just one. Educational Researcher, 27 (2), 4–13. Sorensen, E. K. (2005). Networked eLearning and collaborative knowledge building: Design and facilitation. Contemporary Issues in Technology and Teacher Education, 4(4), 446-455. http://www.citejournal.org/vol4/iss4/general/article3.cfm Stiften (2014, July 14). Video afløser forelæsninger. Århus Stiftstidende. Retrieved from http://stiften.dk/aarhus/video-afloeser-forelaesninger Vicens, Q. & Caspersen, M.E. (2014). Getting more scientists to revamp teaching. Journal of College Science Teaching, 43 (5), 22-27. Weller, M. (2002). Delivering learning on the Net: The why, what & how of online education. London: Psychology Press. Aarhus University (2011). Den faglige udviklingsprocess. Retrieved from http://www.au.dk/fileadmin/res/fau/dok/fau_rapport_090311.pdf. Aarhus University (2015). Key figures and ranking. Retrieved from http://scitech.au.dk/en/about- science-and-technology/key-figures/. Bjælde, O.E., Caspersen, M.E., Godsk, M., Hougaard, R.F., & Lindberg, A.B. (2015). Learning design for science teacher training and educational development. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp.FP: 9-FP:18). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:18 30 Tensions and turning points: exploring teacher decision-making in a complex eLearning environment Scott Bradey James Cook University Understanding how university teachers experience and respond to imperatives to integrate digital technologies into their curricula and teaching practice is essential for addressing the gap between the potential of such technologies to articulate with institutional objectives and their uptake by university teachers. This article reports on a study in a regional Australian university focused on capturing the complex ways that individual and contextual factors can interact to support or impede the integration of technology into teaching practice. The lens of cultural-historical activity theory is used to describe and interpret the complex activity of designing and teaching a blended-mode course from the perspective of an experienced lecturer. An analytical focus on emergent tensions and the identification of turning points as markers of critical encounters requiring the lecturer to make decisions and take action provides an insight into potential transformations in their thinking and practice. Keywords: activity theory, university teaching, blended learning, technology integration Introduction The integration of digital technologies into university curricula is a multi-faceted phenomenon shaped by a complex array of political, cultural, technical and pedagogical factors (Selander, 2008). From the lecturer’s perspective, the task of designing and teaching a blended-mode course is active, intentional, value-laden work with many matters often vying simultaneously for their attention, decision-making and action-taking (Sanders & McCutcheon, 1986). The work of university teachers is far from simple, however a recent literature review of the ways in in which teacher participation has been conceptualised in eLearning research reveals a relatively dispersed and under-theorised account of the relationship between technology, context, human cognition, and action (Bradey, 2015). Some of these interrelationships have been considered from the systems design perspective in the field of human-computer interaction (HCI) (e.g., Kaptelinin, 1996; Nardi, 1996); however, few of these are well represented within educational technology or eLearning. Oliver (2012) argues that the paucity of theorisation has resulted in the prevalence of simplistic accounts of the role of technology in various kinds of teaching and learning, usually involving some kind of causal or determining mechanism. The experience of universities internationally showing that digital technologies have often failed to meet expectations for transforming teaching and learning (Kirkwood & Price, 2011) would seem to suggest a much more complex interplay of factors may be at work, and that more critical and rigorous research is required. As noted by Sam (2012, p. 84) “part of the challenge of conducting research in digital realms is determining how to understand online life holistically and within context”. Finding a research framework that incorporates these various elements is a challenge, as most conceptual frameworks usually separate individuals, contexts, technology, and such, or only combine a few (Kuutti, 1996; Nardi, 1996; Roth & Lee, 2007). This paper demonstrates how the theoretical and interpretive framework of cultural-historical activity theory (CHAT) (Engeström, 1987, 2001) can be used to describe the highly mediated yet dynamic nature of lecturers’ participation in planning and teaching a blended-mode course, and capture the social, cultural and historical factors influencing their decision-making in their local context. In particular the paper shows how the CHAT principle of contradictions can be used to indentify interactions and tensions within and between components of lecturers’ activity systems as potential sources of development and innovation. Kärkkäinen's (1999) concept of ‘turning points’ is employed as an integral component of the interpretive framework to explain how lecturers’ responses to systemic tensions can influence the transformation of established practices. FP:19 31 Research context This paper is based on one of the four case studies within a doctoral research project conducted at a regional Australian university. The research sought to better understand how lecturers, who are experienced university teachers and disciplinary professionals, make decisions about teaching with digital technology in a contemporary blended learning environment. This qualitative study focused on capturing the complex ways that individual and contextual factors can interact to support or impede the integration of technology into teaching practice. The subject of the case study interpreted in this paper is Lisa, an experienced professional journalist who had been teaching in Higher Education for eight years and had been using digital technologies to supplement her courses for the previous two years. However, Lisa had no formal training in teaching or technology. The course in this case study was a second year unit of study in the professional discipline of journalism and was initially structured in a format comprising 13 hours of lectures and 20 hours of tutorials. Tutorial readings were prescribed in the form of textbook chapters. Lisa frequently used stories of real-world experiences as a bridge between the theory found in the course textbook, and the vocational skills students would be expected to demonstrate. Methodology To allow the nature of lecturers’ participation in a complex activity to emerge over time, this exploratory research adopted a qualitative design and a multiple case study approach. Data were gathered over the course of a study period by way of individual and group semi-structured interviews, stimulated recall interviews, online observations and digital artifacts. Data interpretation was undertaken in two phases and employed Rogoff’s (1995) notion of the three planes of sociocultural analysis to focus on the activity taking place on the personal, interpersonal and institutional-community levels. Locating the study within the theoretical and interpretive framework of cultural-historical activity theory provided a means to to study the actions of people on both an individual and societal level simultaneously. A distinctive feature of CHAT is that its unit of analysis is an activity, that is, a conscious action directed at a goal in a particular context over time. Activities in this sense are not one-time brief actions, described by Roth and Lee (2007, p. 98) as “evolving complex structure[s] of mediated and collective human agency.” Each activity consists of interacting components and their relationships to one another: subject, object (motive), community, tools, rules, division of labour, and outcomes. The relationship is often visualised as an activity triangle, with connecting lines indicating a possible interaction between and among all the components. Engeström referred to this as an activity system. In this study, the basic elements common to all participants in the activity are represented in Figure 1. FP:20 32 Virtual Learning Environment Teaching strategies Planning and teaching a Lecturer blended-mode course Student learning Curriculum requirements, Lecturers/Colleagues Lecturer roles Discipline and institutional Learners Students policies/expectations, The profession/Practitioners Support staff Technical standards/conventions Figure 1: A generic activity system in the current study adapted from Engeström (1987) If tensions arise within or between the elements of an activity system then the flow of interactions can become disrupted or discoordinated. These tensions, referred to as contradictions in activity theory are the underlying causes of visible problems and conflicts. While contradictions generate disturbances in an activity system, they are also seen as important drivers for innovation and change. The current analysis drew on Kärkkäinen’s (1999) notion of ‘turning points’ as a way of identifying possible contradictions within participants’ activity systems. Turning points have been used extensively by Russell and Schneiderheinze (Russell, 2004; Russell & Schneiderheinze, 2005; Schneiderheinze, 2003) as indicators of object transformation, that is, ways in which the lecturer delineated the activity of teaching in a new way. Kärkkäinen (1999) defines three indicators of turning points: disturbance clusters (including dilemmas, disturbances and innovation attempts), questions, and interaction of voices. In the current analysis, turning points were operationalised through the interpretation of reflective dialogue with the researcher (Individual interview; Stimulated recall interview) and with other participants (Group interview), guided by the decision indicators illustrated in Table 1. Table 1: Kärkkäinen’s (1999) indicators of turning point events Turning point Decision indicator indicator Disturbance clusters • The participant expresses hesitations, reservations, being "in two minds" things, inconsistent opinions, characterised by clusters of “buts” and negatives (Dilemmas) • The participant expresses difficulty in understanding, disagreement with, or rejection of a situation (Disturbances) • The participant consciously seeks to introduce a new idea or solution (Innovation attempts) Questioning • The participant questions accepted practices, such as ideas presented, present pedagogy and work practices • The participant expresses doubt about whether former ideas and ideologies are worthwhile or workable in practice FP:21 33 Interaction of different • The participants in a collaborative setting present different voices viewpoints on an issue According to Kärkkäinen (1999), transformation can occur in four ways: widening, narrowing, switching and disintegrating. When a disturbance manifesting an underlying contradiction is acknowledged and successfully resolved, a widened or expanded way of thinking and practising becomes possible. However, if the disturbance manifesting an underlying contradiction is not acknowledged and resolved the object may be narrowed. A narrowing of the object could mean that the teacher's concept of the object becomes less broad, for example, more traditionally focused. A switching of the object means that tensions inherent in the implementation of the object caused the lecturer to change her response to the object. The disintegration of the object means that the lecturer’s response in relation to the object will be fragmented. The following section presents an interpretive commentary of Lisa’s case study for the purposes of situating the data within a CHAT framework; describing the trajectory of this participant’s activity as it changed over time; providing additional information to help contextualise the data; identifying systemic tensions underlying the conflicts experienced by the participant; serving as a device for zooming between the personal, interpersonal and institutional-community plane of analysis, and focusing attention on the meaning interpretations of the researcher. Findings and discussion A summary representation of Lisa’s activity system is illustrated in Figure 2. The Subject node of Lisa’s activity system, encapsulates her individual attributes such as beliefs about teaching, learning and technology; personal qualities, attitudes and past experiences. The Mediating tools node represents the cognitive, virtual and physical tools employed in the activity of teaching a blended-mode course. The Object node establishes the purpose of the activity, and the Outcomes node indicates the intended outcomes of the activity. Contextual elements influencing the activity are informed by elements contained in the Division of Labour, Community and Rules nodes. FP:22 34 • Virtual Learning Environment • Blog • Reflective journal • Textbook • Daily newspapers • Teaching strategies: pedagogical, organisational, learning support, assessment Lecturer Planning and teaching a • Beliefs (teaching, blended-mode course learning, technology) • Personal qualities • Attitudes • Stated course • Past experiences outcomes • Students able apply theory to practice • Students able to demonstrate professional skills • Sense of responsibility and privilege of the profession • Institutional culture and policies • Academic colleagues • Lecturer roles: Technologist, • Professional ethical standards • VLE Support staff Designer, Facilitator, • Lecturer’s rules and expectations • Students Administrator, Evaluator for students • Student roles Figure 2: CHAT model of Lisa’s work activity system Lisa experienced tensions in her work activity system in both the planning and teaching phases of her blended-mode course. She experienced these tensions as disturbances, dilemmas, questioning and innovation attempts which were clustered into one turning point event in the planning phase and three turning point events in the teaching phase. Lisa acknowledged and responded to the tensions in her activity system through expanding the scope of her thinking and practice (widening) or by adjusting her expectations and the implementation of the intended task (switching) in order to achieve her intended outcomes. Lisa’s experience of the tensions in her activity system, her responses, and transformations of practice are summarised in Table 2 and interpreted in detail below. In the planning phase of her course, Lisa experienced a turning point event that impacted on her intent to improve both the flexibility and authenticity of her second-year journalism course. Lisa was enthusiastic about experimenting with new technologies in her teaching. Although she lacked experience with both the functional aspects of digital technologies and the process of integrating them into her curriculum she did not perceive this as a problem, preferring instead to take a trial and error approach and let the design emerge. Lisa’s seemingly laissez-faire attitude and her desire to innovate were at odds with the existing school culture that discouraged change and attempts at innovation. The hegemony in Lisa’s school was manifested as non- participation in institutional initiatives such as the development of blended-mode courses and effectively impeded Lisa’s attempts to seek in-house advice and assistance with improving her course design. This socio-cultural barrier represented a significant turning point for Lisa by compelling her to look beyond her own School for support (Table 2, turning point 1). Through initiating a dialogue with a more experienced academic mentor from another discipline, Lisa was able to transcend the barrier imposed by her own School culture, engage in self- directed professional development, and apply her new understandings to the design of the course. Lisa’s planned integration of Blog and Discussion Board tools to articulate with her desired pedagogical objectives represents a significant widening of the object in comparison FP:23 35 with her initial ‘trial and error’ approach.. Although Lisa’s efforts were not well supported in her own School, she was able to sufficiently reduce the tension between the existing culture in the School (Rules) and her own expectations and beliefs (Subject) to allow her intended innovations to proceed. This is represented as a dashed arrow between the Rules and Subject nodes of Lisa’s work activity system (Figure 3) Table 2: Systemic tensions and turning point events influencing Lisa’s object transformation Turning point Indicators of Activity system Practice event turning point tensions transformation PLANNING PHASE 1. Introducing Disturbance: disagreement with Rules (School Widened: Decided flexibility and conservative school culture acting culture) vs Subject to incorporate blog authenticity to discourage innovation (Intention to introduce to enable a new course design publication of Dilemma: how to use technology to and expectations of articles and improve flexibility and authenticity support) Discussion board to facilitate reflective Innovation attempt: connecting with practice a mentor; online publication (Blog), reflective journal and peer support (Discussion Board) TEACHING PHASE 2. Scaffolding Dilemma: how to engage students Community Widened: the blogging in a task/genre/technology with (Students’ Incorporated activity which they have limited experience experience/skills) vs additional Object (Publishing an guidelines, Innovation attempt: attempt to online new story) template, physical integrate support resources into demonstration, VLE expanded role of the Editor 3. Using the Innovation attempt: participants Rules (Lecturers’ Widened: Discussion attempt to initiate peer support rules for reflective Parameters of Board for peer using the Discussion Board journal task) vs reflective journal support Community task extended to (Students’ need for allow personal peer support) feedback; Future intention to Community integrate peer (Students’ need for support peer support) vs Division of labour (Established lecturer and student roles) 4. Capturing and Dilemma: how to track story Mediating tools Switched: Story tracking the versions throughout the process; (Cognitive tool – writing workflow story writing how to efficiently provide individual teaching strategy) vs redesigned to process feedback Object (Timely incorporate VLE completion of the File Exchange and Questioning: whether current time story writing task) Assignment tools intensive feedback strategy is sustainable Widened: Extended reflection activity to Innovation attempt: worked with incorporate student VLE support team to redesign story generated artifacts writing workflow Lisa’s approach to designing her course was shaped by her desire to emulate the professional practice of journalism through active participation in authentic activity mediated by contemporary digital technologies. Through independently seeking the assistance of an academic mentor, Lisa was able to undertake self-directed professional development to explore the capabilities and affordances of the available technology and deepen her understanding of how technology could be integrated into her teaching. Lisa subsequently designed an extended FP:24 36 newsroom role-playing scenario requiring students to undertake researching, writing, editing and production tasks using a public blog to publish real news stories. In effect, Lisa used digital technologies to enable and support a more flexible and authentic course design though their application as a publication medium, reflective journal, submission and feedback tool and peer support mechanism. Lisa’s response acted to reduce the perceived organisational tension within the school by establishing productive relationships outside the school boundaries, and in turn she was able to enact her espoused pedagogogical beliefs. Lecturer • Beliefs (teaching, learning, technology) • Personal qualities • Attitudes • Past experiences Institutional culture and policies Professional ethical standards Lecturer’s rules and expectations for students Figure 3: Tensions in Lisa’s work activity system in the planning phase Lisa’s participation in the teaching phase of the course could be characterised as reflexive and dynamic. She valued student feedback and was always seeking to improve her own teaching strategies and students’ learning experiences. As the course progressed Lisa encountered several dilemmas, but viewed the course organisation and activities as a ‘work in progress’ that could be adapted to suit the current circumstances. In seeking to sustain a realistic and immersive role-play experience, Lisa formed students into teams and structured all interaction around a newsroom scenario. Early in the semester, Lisa perceived the prescribed academic lecture/tutorial format as a disruption to the flow of news as it would occur in the real word of journalism. She soon abandoned the formal lecture structure in favour of regular Announcements in the VLE and tutorials organised as a news conference where students would be expected to research, develop and discuss their ideas for stories. An unanticipated contextual tension arose early in the story production process with the realisation that the majority of students possessed a very limited conception of blogs as an online medium and were not aware of the process of writing for online publication. For Lisa, this introduced the dilemma of how to engage students in a task where they were relatively unfamiliar with both the genre and the tools (Table 2, Turning point 2). From an activity theory perspective, this dilemma represented a tension between the Community node (students’ experience/skills) and the Object node (publishing an online news story) of Lisa’s work activity system (Figure 4). With the intention to remediate the difficulty posed by students’ variation in knowledge and experience, Lisa attempted to scaffold the online story writing process. She sourced supplementary background information about the blog genre including guidelines for authors covering the legal and ethical responsibilities of writing for public online media. Lisa also found a suitable example of current affairs blogs online which was subsequently used as a template to guide students’ contributions. Further, a member of the VLE support team was invited to demonstrate the functionality of the ‘Tropic Zone’ blog being used in the activity. After students had gained some familiarity with their assigned roles and the online story writing process, Lisa adjusted the role of the Editor to introduce a greater degree of authenticity into the role-play. FP:25 37 Lisa’s multi-layered approach to scaffolding student performance effectively mitigated the issue of students’ lack of experience by providing the ‘building blocks’ that students could draw together to complete the task. Lisa’s response resulted in a widening of the blogging activity by initially providing more specific guidance and later by adjusting the role responsibilities. Her actions effectively reduced the tension between the Community and Object nodes of her activity system as illustrated by the dashed arrow (Figure 4). As students progressed through the researching, interviewing, writing, illustrating, editing and publication stages of the blogging activity, they were expected to contribute to a dedicated Discussion Board to evaluate and reflect on their experiences. Lisa interacted with students on the forum to make explicit connections between disciplinary frameworks and students’ developing practice and subsequently used the contributions as material for further discussion during tutorials. Lisa had positioned students as active co-constructors of the course with a view to enhancing their sense of involvement and ownership. Within a few weeks, Lisa noticed that students had begun using the reflective journal forum as a place to share personal experiences with other students effectively extending the use of the Discussion Board to function as a peer support forum. • Virtual Learning Environment • Blog • Reflective journal • Textbook • Daily newspapers • Teaching strategies: pedagogical, organisational, learning support, assessment Planning and teaching a blended- • Institutional culture and policies • Academic • Professional ethical standards colleagues • Lecturer’s rules and expectations for • VLE Support students staff Figure 4: Tensions in Lisa’s work activity system in the teaching phase For Lisa, the spontaneous student-driven evolution of their online activity suggested she had initially underestimated students’ need to connect with each other and share their experiences on a personal level. She had also insufficiently considered the value and utility of the Discussion Board for this purpose (Table 2, Turning point 3). In effect, students ‘broke the rules’ Lisa had set specifying the structure and recommended content of contributions to the reflective journal. This behaviour represented a tension between Lisa’s rules for the reflective journal task (Rules) and students’ need for peer support (Community). Lisa recognised the need for peer support as crucial to students’ formative development as journalists and consequently extended the parameters of the reflective journal task to allow personal reflection and feedback. She also expressed the intention to create a dedicated peer support forum for the following year. Lisa’s response to support the student-initiated innovation attempt immediately resolved the tension by adapting the ‘rules’ to suit the evolving context (Figure 4). FP:26 38 Lisa’s fourth turning point event revolved around the need to track students’ storywriting progress and provide feedback in a timely way. The tension underlying this event was borne from her emphasis on flexibility and authenticity which was intended to emulate the flow of activity in a real newsroom. In an attempt to immerse students in the story writing process, she had relaxed the more rigid academic structures of set lecture times and due dates for assignments in favour of allowing students to pursue news stories in real time. Deadlines were determined on an individual basis. From a student perspective, such an approach was extremely flexible. However, Lisa found it difficult to keep track of the most recently edited version of articles and soon experienced a significant workload issue due to the need to provide frequent feedback. For Lisa, the dual pressures of monitoring student performance and providing timely feedback presented a significant logistical dilemma leading her to question the sustainability of her current practice (Table 2, Turning point 4). In effect, her initial teaching strategy (Cognitive mediating tool) was impeding her own and students’ timely participation in the learning task (Object). In an effort to identify a more efficient workflow, Lisa collaborated with the central VLE support team to design a technology-mediated solution that enabled her to electronically capture stories at different stages of development and return individual feedback to the author. Lisa’s actions did not change the parameters of story writing process per se but represented a switching of the article submission and feedback procedure to a technology- mediated method using the VLE File Exchange and Assignment tool. Later in the study period, Lisa saw an opportunity to capitalise on the VLE’s capacity to capture work in progress by having students submit artifacts, such as emails, generated during unsuccessful or problematic encounters with potential interviewees. For Lisa, these digital artifacts were a way to capture a perspective on student activity that was not always evident in their reflective journal entries. She subsequently widened the reflective journal task to incorporate evaluation of student-generated artifacts as stimuli for discussion. Lisa’s purposeful integration of the appropriate VLE tools into her pedagogical repertoire enabled her to continue her planned monitoring and feedback strategy but using a more efficient and manageable technology-mediated workflow. This solution effectively reduced the tension between the teaching strategy itself (Cognitive mediating tool) and her timely participation in the online learning activity (Object) as illustrated by the dashed arrow between these nodes (Figure 4). Indeed, her early success with technology integration prompted Lisa to later extend the reflective journal task to similarly take advantage of capabilities of the VLE. Overall, Lisa’s decision-making was characterised by self-confidence in her repertoire of pedagogical skills, a deep belief in the importance of good teaching, a concern for the wellbeing of her students, a strong sense of professional identity, a willingness to experiment with new technology, a willingness to take risks, and a positive regard for reflective practice. Lisa’s decisions about using technology in particular ways were strongly influenced by her personal theory of teaching but were also historically mediated by her previous experiences with digital technologies, and her own personal history as a disciplinary professional and university teacher. The design of learning tasks in the planning phase consistently demonstrated Lisa’s purposeful selection of technological tools to facilitate activities aligned with her espoused pedagogical disposition. Significantly, her case reflected the broader finding that the mere presence of functional affordances perceived in a mediating technology did not guarantee its consistent application in a given teaching and learning scenario. Affordance theories offered a useful insight into how Lisa and the other participants perceived the possible uses of digital technologies for teaching and learning in relation to the actualising circumstances in their work activity systems. The analysis presented in this paper reflected the broader finding that participants’ teaching approaches as socially constructed through their interactions with academic colleagues in their schools and students in their courses. Lisa’s case typified the circumstances of many lecturers in the study who frequently found themselves in a regime with a dominant ideology that was at odds with their own personal practical theories of teaching a blended-mode course suggesting that, in a collaborative activity, a group can share one object, but members of the group can relate to the object through differing motivations. Lisa’s case exemplified how socio-cultural tensions can be manifested progressively as a lecturer moves through the planning and teaching phases of their course in the form of questioning, disturbances, dilemmas and FP:27 39 innovation attempts. The case further demonstrated how implementation of the object (planning and teaching a blended-mode course) was achieved through dialogic negotiation with the community (stakeholders) and through exercising individual agency. Like the other experienced experienced lecturers in this study, Lisa demonstrated a strong sense of self-efficacy, was readily able to identify and acknowledge a range of barriers in her activity system, and could assess the elements in her pedagogical context over which she had some influence. When Lisa felt she could control the events in her local context, she responded by widening her thinking and practice, effectively introducing new forms of activity. In other situations where she perceived less control, Lisa tended to change her response to the object by adjusting her expectations and seeking alternative ways to actualise her pedagogical vision. Conclusion Using one case study as an example, this paper demonstrated how cultural-historical activity theory can be successfully applied as descriptive and interpretive framework to gain an insider’s perspective on how university teachers make decisions about teaching with technology in a contemporary blended learning context. A focus on interpreting systemic tensions and critical ‘turning points’ provided a means to indentify markers of object transformation, that is, ways in which the lecturer delineated the activity of teaching in a new way. A key benefit of selecting CHAT as an appropriate framework for eLearning research is that it reframes the traditional notion of participation as an individual’s actions and mental processes and considers the minimal meaningful unit of analysis as an activity system. CHAT is, therefore, capable of providing a more expansive and holistic conception of participation that can take account of individual and social factors, and recognise the socially-situated and culturally- mediated nature of learning (Barab, Evans, & Baek, 2004). An expanded conception of participation that encompasses contextual factors has significant value for eLearning research by enhancing access to many aspects of participation that have been relatively under-explored, including non-visible activity such as navigating through a course website or reading student contributions to a discussion forum. A wider view of participation can also access non-visible activities that occur away from the computer such as reflecting upon ideas; developing personal theories of student engagement; and shifting of pedagogical orientation. Importantly, the conceptual framework of activity theory illuminates the internal dynamics of an activity rather than studying the components in isolation. This interconnectedness makes it possible to describe relationships between members of the community (such as teachers, students, and colleagues) as well as roles adopted; tools shared by the participants, and explicit and implicit rules for collaboration. A central tenet of activity theory is that tools or artifacts mediate all human action and these tools may be physical (e.g., a smartphone), or symbolic (e.g., teaching strategies, arithmetic, language) (Cole & Engeström, 1993). Cultural tools, such as technology, contain both affordances and constraints that mediate the actions of the agent, in this case, the university teacher (Wertsch, del Rio, & Alvarez, 1995). In other words, digital technologies have particular properties that “allow certain actions to be readily performed with them, and which therefore push behaviour in certain directions” (Tolmie & Boyle, 2000, p. 120). Rather than focusing on technology as the agent, CHAT accommodates a consideration of the types of activity afforded or constrained by the technology and acknowledges how the attributes of the technology interact with the surrounding social and cultural context. CHAT also offers insights into the role of cognitive mediating tools by considering the potential interrelationship between lecturers’ pedagogical beliefs, perceptions of the technology, and the teaching strategies employed in a blended-mode setting. Significantly, CHAT’s capacity to examine the manner in which teachers, as agents, have purposefully used tools to achieve the intended outcomes of the course challenges traditional approaches to learning which have tended to ignore mediated activity (Säljö, 1999). The ability of CHAT to represent the “multivoicedness” of complex social situations is particularly useful as it provides a means to capture the dynamic interplay between the vertical and horizontal divisions of labour. For example, tasks may be distributed among community members such as students and academic colleagues (horizontal) and may also be distributed FP:28 40 vertically in that the lecturer may hold multiple roles as technologist, designer, facilitator, administrator and evaluator of the learning activity. Additionally, the concept of multivoicedness can also include the historical beliefs, expectations, and values of different community members, which are imported into current activities, and shape what transpires. CHAT also facilitates the analysis of change over time in an activity system. This affordance is pertinent to eLearning research that seeks to understand why digital technologies have often failed to meet institutional expectations for transforming teaching and learning. Instead of assuming a goodness of fit between lecturers’ peadagogical vision and the institutional expectations for integrating digital technologies, CHAT has the capacity to view an enterprise such as planning and teaching a blended-mode course as an emergent activity that unfolds over time and considers how actualising circumstances can influence the subject’s response to a disturbance such as institutional eLearning imperative. Rather than simply focusing on "what went wrong," the interpretive lens of CHAT affords insight into turning point events as moments when something new was learned and when the participants in an activity conceptualised it in a new way. References Barab, S. A., Evans, M. A., & Baek, E.-O. (2004). Activity theory as a lens for characterizing the participatory unit. In D. H. Jonassen (Ed.), Handbook of research on educational communities and technology (pp. 199-214). Mahwah, NJ: Lawrence Erlbaum Associates. Bradey, S. (2015). How experienced university teachers make decisions about teaching with technology in a complex learning environment: An activity theory analysis. (Doctor of Philosophy), James Cook University, Townsville. Cole, M., & Engeström, Y. (1993). A cultural-historical approach to distributed cognition. In G. Salomon (Ed.), Distributed cognitions: Psychological and educational considerations (pp. 1- 46). Cambridge:: Cambridge University Press. Engeström, Y. (1987). Learning by expanding: An activity-theoretical approach to developmental research. Helsinki: Orienta-Konsultit. Engeström, Y. (2001). Expansive learning at work: Toward an activity-theoretical conceptualization. Journal of Education and Work, 14, 133-156. Kaptelinin, V. (1996). Activity theory: Implications for human-computer interaction. In B. A. Nardi (Ed.), Context and consciousness: Activity theory and human-computer interaction (pp. 103- 116). Cambridge, MA: MIT Press. Kärkkäinen, M. (1999). Teams as Breakers of Traditional work Practices: A Longitudinal Study of Planning and Implementing Curriculum Units in Elementary School Teacher Teams. University of Helsinki, Helsinki. Kirkwood, A., & Price, L. (2011). The influence upon design of differing conceptions of teaching and learning with technology. In A. D. Olofsso & J. O. Lindberg (Eds.), Informed Design of Educational Technologies in Higher Education: Enhanced Learning and Teaching (pp. 1-20): IGI Global. Kuutti, K. (1996). Activity theory as a potential framework for human-computer interaction research. In B. A. Nardi (Ed.), Context and consciousness: Activity theory and human- computer interaction (pp. 17-44). Cambridge, MA.: The MIT Press. Nardi, B. A. (Ed.). (1996). Context and consciousness: Activity theory and human-computer interaction. Cambridge, MA.: MIT Press. Oliver, M. (2012). Learning technology: Theorising the tools we study. British Journal of Educational Technology. Rogoff, B. (1995). Observing sociocultural activity on three planes: Participatory appropriation, guided participation, and apprenticeship. In J. V. Wertsch, P. D. Rio, & A. Alvarez (Eds.), Sociocultural studies of mind (pp. 139-164). New York: Cambridge University Press. Roth, M., & Lee, Y. (2007). Vygotsky’s neglected legacy: Cultural–historical activity theory. Review of Educational Research, 77, 186-232. Russell, D. L. (2004). Paradigm Shift: A Case Study of Innovation in an Educational Setting. International Journal of Instructional Technology and Distance Learning, 1(12), 19-36. Russell, D. L., & Schneiderheinze, A. (2005). Understanding innovation in education using activity theory. Educational Technology & Society, 8(1), 38-53. FP:29 41 Säljö, R. (1999). Learning as the use of tools: A sociocultural perspective on the humantechnology link. In K. Littleton & P. Light (Eds.), Learning with computers: Analysing productive interaction (pp. 144-161). London: Routledge. Sam, C. (2012). Activity Theory and Qualitative Research in Digital Domains. Theory Into Practice, 51(2), 83-90. Sanders, D. P., & McCutcheon, G. (1986). The development of practical theories of teaching. Journal of Curriculum and Supervision, 2(1), 50-67. Schneiderheinze, A. (2003). Adoption as mediated action: How four teachers implemented an innovation cluster. (Doctor of Philosophy), University of Missouri, Columbia. Selander, S. (2008). Designs of learning and the formation and transformation of knowledge in an era of globalization. Studies in Philosophy and Education, 27(4), 267-281. doi: 10.1007/s11217-007-9068-9 Tolmie, A., & Boyle, J. (2000). Factors influencing the success of computer mediated communication (CMC) environments in university teaching: A review and case study. Computers & Education, 34(2), 119-140. Wertsch, J. V., del Rio, P., & Alvarez, A. (1995). Sociocultural studies: History, action, and mediation. In J. V. Wertsch, P. d. Rio, & A. Alvarez (Eds.), Sociocultural studies of mind (pp. 1-34). Cambridge: Cambridge University Press. Bradey, S. (2015). Tensions and turning points: exploring teacher decision-making in a complex eLearning environment. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:19-FP:30). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:30 42 Navigate Me: maximising student potential via online support Colin Clark Jessica Andreacchio Rita Kusevskis-Hayes Student Life and Learning Student Life and Learning Student Life and Learning UNSW Australia UNSW Australia UNSW Australia Jessie Lui Shauna Perry Ethan Taylor Student Life and Learning Student Life and Learning Student Life and Learning UNSW Australia UNSW Australia UNSW Australia This paper reports on the development of NavigateMe, an online tool currently being trialled at the University of New South Wales. The tool is a student-centred initiative designed to support students in accessing university-wide, faculty-based and external information and support services to improve and enhance their learning and university life. Based on responses provided, an action plan is produced that allows students to reflect on their current situation and be directed to specific services and information according to their individual needs and interest at any point in their student life. The tool was developed through a collaborative and iterative process in consultation with staff, students and faculties. The tool is in the strategic plan approved by the DVC(A) and it has received significant funding from the university. Keywords: Online tool; student support; student engagement; technology; enabling; reflection Introduction Despite an increased focus on student support, there remains a concern that services remain underutilised. For example, Reavley, McCann, & Jorm (2012) found that only 10% of students with mental health problems consulted a student counsellor, and that students born overseas were three times more likely to seek such help than their Australian-born counterparts. Brown, Keppell, Hughes, Hard, & Smith (2013) call this reluctance to admit a need for support a “lone wolf” approach to learning. To some extent, this approach may be symptomatic of the lack of effective pathways to assistance for students with emotional or support needs (Laws & Fiedler, 2013). Universities offer a range of services in the areas of academic support, career and employment advice, counselling and psychological services, and offer targeted assistance and programs for students with disabilities or those who have experienced disadvantage. There may also be peer support programs and student-led initiatives. However, research suggests that the effectiveness of these services in providing assistance depends to some extent on students’ personalities and coping styles (Connor-Smith & Flachsbart, 2007). Moreover, the willingness of students to access services may depend on their attitude toward seeking help or the practices of the support service, such as session time limits (Uffelman & Hardin, 2002). Coping may be classified into three styles, which have implications for psychological wellbeing (Heppner, Cook, Wright, & Johnson, 1995). These are: 1. The reactive style, where emotional and cognitive responses tend to impede more positive methods of coping 2. The reflective style, which is characterised as a problem-solving approach 3. The suppressive style, which is a tendency to avoid addressing problems or denying them According to Julal (2012), those who take a reflective style are more likely to seek support from services. Those with the reactive style are less likely to seek help because of their emotional FP:31 43 responses to a perceived difficulty, and those with the suppressive style are prone to denial that support is needed. The problem for universities, then, is how to engage those students who would benefit from support but are reluctant to seek it. Although university services cannot change students’ basic dispositions and increase their willingness to seek support, it may be possible to lower the threshold in terms of the first step—the acknowledgement that a problem exists and that help is available. While investigation of psychological dispositions is beyond the scope of this project, it was postulated that the first step to encouraging help-seeking was to encourage reflection. This assumption is based on the view that task involvement, whereby students retain responsibility for solving their problems, is more likely to encourage help-seeking than a system that simply proposed solutions (Magnusson & Perry, 1992). In summary, reflection is known to improve academic performance (Morisano, Hirsh, Peterson, Pihl, & Shore, 2010; Potter & Bye, 2014). This approach also provides students with information upon which to act, thus encouraging self-management (Robbins, Oh, Le, & Button, 2009). Thus, Student Life and Learning at UNSW decided to construct an online tool by which students could take an easy first step towards reflecting on their progress and identifying any concerns. The tool would then present them with a list of actions, and they could decide whether to proceed on that basis. The use of online tools for support services and resources is a logical extension of the modern campus. Online tools are used for teaching (e.g. Lawrence, 2013) or for monitoring student success (e.g. Kokaua, Sopoaga, Zaharic, & Van der Meer, 2014). Many young people use the Internet to request support from peers as well as a source of information (Piper & MacDonald, 2008). While some students are less familiar with the use of online tools, the university where this project is held uses Internet technology for many of its administrative and academic functions, and students soon develop at least basic competence, and this is sufficient to use the NavigateMe tool. A similar tool has been reported by Smyth & Lodge (2012) for orientation. However, other than in distance education (Brown, et al., 2013; Clark et al., 2015) to the authors’ knowledge few web-based tools are available for student engagement with the university community and student support services. Purpose of the NavigateMe project The NavigateMe project is intended to provide an online tool to encourage reflection on personal goals and alignment with university study. This paper reports on the development of this tool, which was piloted in 2014 and rolled out in July 2015, with a redesign and change of platform planned for December 2015. Accessing an online tool is a less threatening step for students than making an appointment with an advisor, counsellor or student service provider. Students are not asked to make a commitment or admit to failings that may be a source of embarrassment. Thus, NavigateMe is intended to be the first step in a journey to support and improved independent learning. In addition to administrative, personal, academic program and social needs, a new release of NavigateMe will include short tests of mathematical knowledge and English language proficiency, as well as a self-assessment of academic literacy skills. Students can complete these tests and are referred to online resources, university services or other sources of support, or they are provided with suggestions to improve their own knowledge. For example, those concerned about their English proficiency may be referred to the UNSW Learning Centre, to conversation groups, the language exchange program or a variety of online resources with advice on academic writing and grammar. This provides an objective way for students to assess their support needs, given that self-evaluations of academic proficiency are subject to inaccuracy (Pike, 1995). The tool is made available to students at orientation events, on Facebook pages and in newsletters—pitched at all students rather than just those at risk of attrition. It intended to FP:32 44 improve academic outcomes, rather than necessarily to remedy problems. At UNSW, NavigateMe was originally available to all students studying with the Faculties of Art & Design and the Faculty of Science with extension of the tool to all faculties listed as a priority in the Deputy Vice Chancellor’s (Academic) Strategic plan 2014 to 2018, and has received significant funding for development as a result. The NavigateMe tool is now available to all UNSW students, with further revisions scheduled for completion by the end of 2015. Background The University of New South Wales launched this online initiative in response to a need to engage students who may be non-traditional in terms of social, cultural and economic factors (Nelson, 2014; White, 2014; Zepke, 2013). The use of an online tool to augment existing services recognises the need for alternative pathways to support. While there is variation in the technological experience and skill of first-year students, university students generally have sufficient access to and familiarity with online technology to access such an online tool (Kregor, Breslin, & Fountain, 2012), and at UNSW many administrative and teaching functions are performed online, so the online environment is familiar to students. Therefore, such a tool is a useful addition to existing services as first step in engaging students in need of support and encouraging them to reflect upon their needs. Student service staff and faculty advisors report that students usually do minimal initial independent preparation in reflecting on their circumstances prior to face-to-face consultations. Moreover, some students—particularly those from low socioeconomic status (low SES) backgrounds—may lack knowledge of available support or be reluctant to ask for it (White, 2014). Whilst it was originally planned that the NavigateMe tool would assist students on non-good academic standing, it soon became apparent that such students were already far along in the process of disengagement. What was needed was a preventative approach rather than a remedial measure for students in difficulties. NavigateMe has been piloted with two faculties across two Sydney campuses, with content tailored to available resources and student/staff feedback. The tool was extended to all faculties in 2015, and a revised and improved version on a new platform will be completed by the end of 2015 for launch in early 2016. It must be emphasised that the purpose of NavigateMe is not to replace traditional face-to-face services such as general advice, personal counselling, disability services or learning support. Rather, it encourages reflection on and analysis of a student’s needs and empowers users by offering a mix of assisted and self-accessed resources for support. Respondents and service providers are strongly encouraged to use the action plan as the basis of discussion in face-to- face support. The plan can also be used in discussions with students as a guide or framework with advisors during interviews, especially if new to the role or university. Description The NavigateMe tool is accessed as a stand-alone website or via a link on the UNSW website. When students log in they see a menu from which they can select the areas that most concern them. There are five general areas: “admin”, “personal life”, “course”, “program” and/or “uni life” (they can choose any number of these). There is another option of “I would like to talk with someone”, which has an email link to student advisors and information about 24/7 services (Figure 1). Figure 1: Part of the NavigateMe landing page FP:33 45 A list of statements is then displayed in each of the five categories selected by the student. An example item under “Admin” is “I need to withdraw from my course/s”. On each category page, a short explanatory video with an animation is presented for clarification. Once selected items are submitted, there is a screen to check selections and the student can then click to generate an action plan. The action plan appears on the screen, and can be printed or emailed to the student. It consists of advice and links to other sites offering advice. The action plan is organised under four headings: “to read” (links to explanations), “to know” (information to find out about) “to see” (people to consult with, such as administration staff or counsellors depending on the question) and “to do” (advice on practical steps such as “meet other students” followed by links to the web sites of clubs and societies on campus). These categories are shown in Figure 2. Figure 3 shows the items under one of the categories—in this case “Personal Life”. The student selects the items that concern her/him. There is a video that outlines some of the issues listed. Figure 4 shows part of an action plan, which lists actions for the student in terms of people to see or information to read. Figure 2: The NavigateMe “categories” screen Figure 3: Some of the “Personal Life Category” items FP:34 46 Figure 4: Part of an action plan From the outset, it was important to engage students in the development process and ensure that the finished product was inclusive for all students with regards to imagery and practicality of use. With a prototype developed, the tool was trialled with students in the Faculty of Science and the Faculty of Art and Design, and some changes were made to the presentation of the menus and appearance. Following a trial by students with vision impairments, some changes were also made to accommodate students with disabilities so the web page could be used with a screen reader. Changes were also made to the graphics to give the narrator more broad ranging appeal and to alter any images that might appear too depressing or ‘dark’. The tool incorporates icons from the UNSW campus so students will have a sense of familiarity in the online environment. Some changes were also made to the software to make NavigateMe accessible on mobile devices. Subsequently an online survey of users provided feedback on useability and ease of use. This survey showed that approximately a quarter of completions of the tool were on tablets and smartphones. Laptop computers alone accounted for nearly 60% of completions. Trials of NavigateMe and user feedback In March 2014, the tool was made available to students in the Faculty of Science and in July to those in the Faculty of Art & Design. There were over 200 completions in the first month, with student action plans generated. Staff, services and academics were consulted during November. Some students returned to use the tool more than once. Faculty involvement was overwhelmingly positive. FP:35 47 During 2014, NavigateMe was offered to students on non-good standing in the Faculty of Science. In 2014, over 1500 students across two faculties completed the tool and generated action plans. Given that there are approximately 12,000 students in the Faculty of Science and 2,500 in the Faculty of Art and Design, this was considered a reasonable response rate, although it remains an open question whether the students most in need of support were reached. Focus groups were held across both faculties and as a result of student and staff feedback, 36 recommendations were made for changes and additions to items, layout and content. Overall, student reactions were positive. The following comments were typical. You go to ask somebody at administration or student services or something like that and say, "Okay, I need help." I knew the first thing they ask you is, "What do you need help with?" And there's very rarely an easy answer for that; and I think this app is going to be very useful in that sense; to help someone to break down what is it that they actually need help with… (Art and Design student) And a few weeks ago when I saw the NavigateMe, I was like, "This is useful", because I was really stuck, "[What] should I do?" I can only do one commerce major, and I was like, "Which one shall I pick?" I knew all this time, since I started uni, that I was going to do accounting or finance, but I had no idea which one. And so I used that program. (Science/commerce student) In orientation week (O-Week) in July 2015, a revised version of NavigateMe was opened to all students, and promoted to students who attended the Student Life and Learning stall. There were 248 sessions with 206 action plans generated. Of these students, 86% reported that they found it helpful and would recommend it to friends. The numbers of sessions and users since March 2014 are shown in Table 1. These show an increase in the number of users between 2014 and 2015, reflecting the extension of the tool from two faculties to all eight of the UNSW faculties. However, there was only a “soft launch” of the modified 2015 tool in semester 2: the revised tool, “Release 2” is planned for semester 1 2016. Table 1: Website data from Google Analytics 30/09/2015 6/3/–31/12/2014 1/1– 30/9/2015 Change Total website visits 2,037 2,266 + 11% All multi-session 1,108 1,244 + 12% Users (those that logged in and engaged) All pages viewed by 11,553 14,791 + 28% all users New unique users 54.3% 54% + 12% Returning 47.5% 46% (multi-session) users Avg. Time on Site 05:39 06:32 + 15% Action plans created 328 732 + 123% Table 2: Proportions of NavigateMe users by academic career (January–September, 2015) Academic career Percentage Non-Award 12% Postgraduate 29% Research 2% Undergraduate 57% FP:36 48 Most common issues One useful product of the NavigateMe tool is data on the numbers of issues that are most commonly selected by people that use the tool. From March 2014 to September 2015, the ten items most commonly clicked are shown in Table 3. Unfortunately this is a crude measure because it is not currently possible to distinguish between action plans generated by staff and those done individually. Moreover, students can return to the tool and may be counted twice. However, the new release of NavigateMe in early 2016 will permit more precise statistics. Table 3: Most common issues (since March 2014) Rank QUESTION 1 I need advice on my career path 2 Who can I talk to about my progress in the course? 3 I procrastinate and struggle to meet deadlines 4 I don't know if I am doing the right courses/subjects 5 I would like to learn how to study for university I feel low and a bit overwhelmed and don't know 6 what to do 7 I don't know if I am in the right program/degree Depression and/or anxiety is impacting my study 8 and my life 9 I often feel lonely I would like more information on scholarships I may 10 be eligible for Evaluation The NavigateMe tool is evaluated on a regular basis and in relation to the academic calendar using several methods. The tool is revised and updated in response to feedback from students. • From early in the process, student reactions were gauged through focus groups with open questions, and all users were invited to complete a feedback form two weeks after generating an action plan. • Use of the online component is tracked using web analytics of hits, number of action plans generated, and numbers of new and returning users. • Use of the tool in face-to-face service encounters is assessed through surveys of faculty and service staff • Impact on students is assessed using de-identified analyses of subsequent progress Focus groups There have been three focus groups, chosen from respondents to an advertisement for participants. The students were offered a $20 fast food voucher as an inducement. Two focus groups were held in 2014; one with Faculty of Science students (nine students) and one with students studying Art and Design (10 students). The purpose of these groups was to gauge reactions to site content and obtain feedback on common problems that students may wish to include in the tool. From the 2014 groups, 39 changes were made, for example wording of items, personalisation of action plan and modifications for tablet and smartphone access. Another group in 2015 (six students) considered the extensive redesign and the mock-ups proposed for Release 2 in December 2015. Surveys Students who complete NavigateMe and generate an action plan receive an automated email with a link to a survey (on surveymonkey.net). There was also a survey of selected staff members during the pilot phase in early 2014. To date, two versions of the student survey have been used, the first in semester 2 2014 for the pilot version (49 respondents) and the second from July 2015 (30 respondents). FP:37 49 When the pilot version of NavigateMe was created, staff members in support roles (in faculties or administration) were asked to comment on it. Feedback from 18 staff members who had not previously seen the tool was requested on the style of animations, functionality, ease of use, and suggested improvements. As a result of this feedback, some changes to animations and wording were made, and several additions to the actions recommended in action plans. Student survey 1 focused on the use of the tool—which devices it was used on and its helpfulness as a point of referral. Most students had accessed the tool through a laptop (51%) or desktop (27%) computer with tablets (19%) and smartphones (8%) making up the remainder (note that some students had accessed the tool more than once, on different devices). Of the 36 students that responded to the question “Did the tool allow you to identify issues that were relevant to you?” 29 (81%) reported that it was “useful” or “very useful”. Further comments on the website layout, wording of questions and layout of the site and action plan have been considered in the 2015 redevelopment. Table 3 shows responses to the question “Did the tool allow you to learn about services on campus that you were previously unaware of? If so, which? These responses indicate that the tool fulfilled its function as a source of information and referral. Table 3: Services that respondents to survey 1 learned about via NavigateMe Service % of No. respondents Science Faculty Student Centre 14.81% 4 School Student Centre 11.11% 3 Academic Advisors (faculty-based advisors) 37.04% 10 Educational Support Advisors (part of Student Life and Learning) 44.44% 12 Student Central (administrative services) 11.11% 3 Careers and Employment 37.04% 10 Student Development International (services for international 14.81% 4 students) Counselling and Psychological Services 14.81% 4 Student Equity and Disability Unit 11.11% 3 The Learning Centre (academic support) 33.33% 9 Student Conduct and Appeals Officer 14.81% 4 Total Respondents: 27 Student survey 2, created in July 2015 for use in the orientation week (O-Week), received 30 responses. Of these 30, 29 students found the tool easy to use, and 29 reported that the tool had identified issues that were very or somewhat relevant to them. The action plan was useful to 83% of students (on a yes/no scale). Moreover, 28 students found the information clear and very/somewhat concise. Twenty six students (87%) reported that they would recommend the site to others. All but one of the 30 respondents found it easy to use. Overall, the surveys indicate that NavigateMe provides information on support services and achieves its purpose of encouraging reflection on goals and need for support. Future directions NavigateMe is a useful gateway to support services at UNSW. Nonetheless, there are areas where the tool may be further developed and its use extended. The tool was recently accepted as part of the UNSW Advantage program, whereby students who volunteer to manage or market the program for 20 hours can gain credit for their work on their Australian Higher Education Graduate Statement (AHEGS). This is an important step in reducing ongoing costs, improving stability and maintaining relevance to the intended student audience. For the volunteers, this will be an important opportunity to learn about digital marketing, project management, coding and social media. Moreover, there should be regular updating of existing content for the sake of the 44% of users who return. As for the tool itself, further extensions to the range of self-test materials are planned, with content specifically tailored for individual faculties, in terms of subjects covered, genres of FP:38 50 communication/assessment and assistance offered. It has been proposed that the testing component be extended to include aptitude tests and adapt aspects of the tool for prospective students and their parents, to guide their choices of course and career. Conclusion NavigateMe blends student services and faculty information with questions to guide students towards the outcome, a comprehensive action plan able to be used as an online service mixed with key face-to-face contacts. We argue that for millennial students enrolled in a university that uses online technology extensively for administrative and educational purposes it is appropriate to offer an online tool as a first step in seeking support. This online tool encourages reflection on personal goals and offers practical suggestions for students to improve their own university experience either by accessing available services or simply by positive making changes to their lives outside official student services. Moreover, the tool can be adapted for specific campuses and faculties to provide program as well as personal advice. Rather than a response to failure or poor grades, NavigateMe is a proactive and pre-emptive approach to addressing student needs in an accessible format that encourages students to consider their lifestyle and approach to study while seeking further support in a timely manner and leading them towards better informed choices. References Brown, M., Keppell, M., Hughes, H., Hard, N., & Smith, L. (2013). Exploring the disconnections: Student interaction with support services upon commencement of distance education. The International Journal of the First Year in Higher Education, 4(2), 63–74. Clark, J., Gurney, L., Lawrence, S., Leece, R., Malouff, J., Masters, Y., et al. (2015). Embedding an institution-wide capacity building opportunity around transition pedagogy: First Year Teaching and Learning Network Coordinators. The International Journal of the First Year in Higher Education, 6(1), 107. Connor-Smith, J. K., & Flachsbart, C. (2007). Relations Between Personality and Coping: A Meta-Analysis. Journal of Personality & Social Psychology. , 93(6), 1080–1107. Heppner, P. P., Cook, S. W., Wright, D. M., & Johnson, W. C. (1995). Progress in resolving problems: A problem-focused style of coping. . Journal of Counselling Psychology, 42 279– 293. Julal, F. S. (2012). Use of student support services among university students: associations with problem-focused coping, experience of personal difficulty and psychological distress. British Journal of Guidance & Counselling, 41(4), 414–425. Kokaua, J., Sopoaga, F., Zaharic, T., & Van der Meer, J. (2014). The development of a pre- enrolment screening tool to inform targeted support services in the first year in health sciences. The International Journal of the First Year in Higher Education, 5(1), 55–66. Kregor, G., Breslin, M., & Fountain, W. (2012). Experience and beliefs of technology users at an Australian university: Keys to maximising e-learning potential. Australasian Journal of Educational Technology, 28(8), 1382–1404. Lawrence, J. (2013). Designing and evaluating an empowering online pedagogy for commencing students: a case study. The International Journal of the First Year in Higher Education, 4(2), 49–61. Laws, T. A., & Fiedler, B. A. (2013). Students seeking help for mental health problems: Do Australian university websites provide clear pathways? Australian Universities' Review, The,, 55 (2), 35–43. Magnusson, J.-L., & Perry, R. (1992). Academic help-seeking in the university setting: The effects of motivational set, attributional style, and help source characteristics. Research in Higher Education, 33(2), 227–245. Morisano, D., Hirsh, J. B., Peterson, J. B., Pihl, R. O., & Shore, B. M. (2010). Setting, elaborating, and reflecting on personal goals improves academic performance. Journal of Applied Psychology, 95(2), 255–264. Nelson, K. (2014). The first year in higher education—where to from here? The International Journal of the First Year in Higher Education, 5(2), 1–21. FP:39 51 Pike, G. R. (1995). The relationship between self reports of college experiences and achievement test scores. Research in Higher Education, 36(1), 1–21. Piper, F., & MacDonald, B. (2008). ‘Sometimes it’s harder to speak out things’: How first year New Zealand tertiary students use informal online communication to help solve significant problems. Australian e-Journal for the Advancement of Mental Health, 7(3), 135–142. Potter, A., & Bye, L. (2014). “It might have worked for you but …” Evaluating the efficacy of a first year support strategy in multiple units and disciplines. The International Journal of the First Year in Higher Education, 5(2), 57–68. Reavley, N. J., McCann, T. V., & Jorm, A. F. (2012). Actions taken to deal with mental health problems in Australian higher education students. Early Intervention in Psychiatry, 6(2), 159– 165. Robbins, S. B., Oh, I.-S., Le, H., & Button, C. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated Meta-Analytic Path Analyses. Journal of Applied Psychology 94(5), 1163– 1184. Smyth, E., & Lodge, J. (2012). Orientation online: Introducing commencing students to university study. A Practice Report. The International Journal of the First Year in Higher Education, 3(1), 83–90. Uffelman, R. A., & Hardin, S. (2002). Session limits at university counseling centers: Effects on help-seeking attitudes.[Report]. Journal of Counseling Psychology, 49(1), 127–132. White, C. (2014). Using principles of trust to engage support with students from low socioeconomic backgrounds. A Practice Report. The International Journal of the First Year in Higher Education, 5(2), 81–86. Zepke, N. (2013). Student engagement: A complex business supporting the first year experience in tertiary education. The International Journal of the First Year in Higher Education, 4(2), 1–14. Clark, C., Andreacchio, J., Kusevskis-Hayes, R., Lui, J., Perry, S., & Taylor, E. (2015). Navigate Me: maximising student potential via online support. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:31-FP:40). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:40 52 Designing an authentic professional development cMOOC Thomas Cochrane Vickel Narayan Victorio Burcio-Martin Centre for Learning and Teaching Centre for Learning and Teaching Centre for Learning and Teaching, Auckland University of Technology Auckland University of Technology Auckland University of Technology Amanda Lees Kate Diesfeld Health and Society Health and Society Auckland University of Technology Auckland University of Technology While there has been a lot of hype surrounding the potential of MOOCs to transform access to education, the reality of completion rates and participant profiles has tempered this hype such that within the hype cycle MOOCs have already hit the trough of disillusionment. However we argue that embedding cMOOC design within an educational design research methodology can enable the design of authentic professional development model that can indeed demonstrate transformation in pedagogical practice. Our design model links mobile learning theory, practice, and critical reflection within an EDR methodology to create an authentic experience for participating lecturers. Keywords: Educational design research, cMOOC, CMALT, professional development, mlearning Introduction Within their roles as academic advisors and web developer at two different higher education institutions the authors of this paper have explored new forms of lecturer professional development based around the development, nurturing, and brokering of communities of practice (COP) (Cochrane & Narayan, 2014). Based upon principles established by Wenger et al., (Wenger, 1998; Wenger, White, & Smith, 2009), these COPs have generally been comprised of lecturers from a single department of the institution. Typically they have formed a peer support group alongside academic advisors as participants taking on the role of technology stewards. The domain or focus of these COPs has been the exploration of mobile social media as a catalyst for new pedagogical practice (Cochrane, Narayan, & Oldfield, 2013, 2015). The impact of these COPs has been critically evaluated and reported to the wider educational community through the explicit embedding of critical reflection as the scholarship of technology enhanced learning or SOTEL (Wickens, 2006). This has resulted in a wide body of research within a variety of educational contexts that encompasses a network of over 37 co-authors, and over 100 peer reviewed publications. While this approach has demonstrated pedagogical transformation within a range of educational contexts it is inherently a time and resource intense model. With the hype surrounding MOOCs (Massive Online Open Courses) garnering the attention of educators and policy makers world wide, the authors decided to explore how a MOOC could be explicitly designed to upscale our COP professional development model. The goal is to model best practice within the MOOC itself as an extended COP, and to enable the participants to become part of a potentially national and global network of practitioners interested in pedagogical innovation. Therefore we designed the Mosomelt (Mobile social media learning technologies) cMOOC. A variety of lecturer COPs were invited to participate in the inaugural mosomelt cMOOC, with participants joining throughout New Zealand and Australia, and as far afield as France. In this paper we explore the design of the mosomelt cMOOC based around an educational design research methodology that embeds: a framework for linking the theory and practice of mobile learning, the development of an ecology of resources and triggering events, critical reflection via SOTEL, and accreditation of participant eportfolios via CMALT - the certified member of the association of learning technologists (https://www.alt.ac.uk/get-involved/certified-membership). FP:41 53 MOOCs There are broadly two distinct types of MOOCs that have developed: cMOOCs or connectivist MOOC, and xMOOCs that are defined by a more traditional course structure and transmission model of information. Bates makes a clear distinction between the two types of MOOCs: xMOOCs primarily use a teaching model focused on the transmission of information, with high quality content delivery, computer-marked assessment (mainly for student feedback purposes), and automation of all key transactions between participants and the learning platform. There is almost no direct interaction between an individual participant and the instructor responsible for the course… cMOOCs have a very different educational philosophy from xMOOCs, in that cMOOCs place heavy emphasis on networking and in particular on strong content contributions from the participants themselves. (Bates, 2014, p. np) We are interested in the exploration of transformative new pedagogies that focus upon learner- generated content and learner-generated contexts, and therefore the cMOOC fits our goal better than an xMOOC. Connectivism and rhizomatic learning Connectivism (Siemens, 2004) and rhizomatic learning (Cormier, 2008) are the two theoretical foundations behind the development of cMOOCs. Both connectivism and rhizomatic learning decentralise the locus of control of the learning process, focusing upon developing a network of learners that co create the curriculum. Cormier’s version of cMOOC design involves the development of an ecology of resources (EOR) to support participant interaction and community, and the development of triggering events designed to ignite participant discussion and investigation leading to the sharing of participant-generated content. Examples of recent cMOOCs include Rhizo14 (Cormier, 2014), developed by Cormier as a six week series of topics to explore. The major downfall of cMOOCs is that the limited guidance offered to learners results in high dropouts and disillusionment (Mackness & Bell, 2015). While the authors have not been enamored by the hype surrounding MOOCs, we have been inspired by examples of open online courses that are not strictly cMOOCs but demonstrate many of their attributes, for example DS106 (Digital Storytelling 106). Based upon connectivism and connective knowledge DS106 is described as “more community than course” (Levine, 2013, p. 54). These examples highlight the critical role of the teacher as the designer and facilitator of the learning experience. Credentialing MOOCs Various approaches have been taken towards assessing or credentialing MOOCs (Friesen & Wihak, 2013), including: open badges, and certification of completion via enrolment in a delivering platform such as Cousera and EdX. We were concerned with modeling a cMOOC around a network of COPs, rather than creating a formal course as such, with the focus upon participant-generated content rather than the delivery of prescribed content. Using a cMOOC format allowed us to design mosomelt as a generic framework to scaffold a network of COPs exploring mobile social media in a variety of higher education contexts. Typically the course approval timeframe for developing and formally accrediting a new course is around one year. Instead of credentialing the mosomelt cMOOC itself, we decided to design mosomelt as a participant-driven experience that provides participants with a basis for generating an eportfolio of evidence and reflection upon integrating mobile social media within their own teaching practice. This eportfolio is then curated and submitted towards CMALT accreditation at the end of the mosomelt cMOOC. Assessment of the mosomelt cMOOC is via participation and peer review, with formal accreditation of participant eportfolios via the CMALT process. Thus mosomelt provides a catalyst for participants to gain an external independent credential that already exists, and one that embodies participation within a global community of educational experts. Without the need to credential mosomelt as a formal course we were able to design and begin implementing the mosomelt cMOOC within a period of six weeks – creating a fast curriculum design and development model. Authentic mobile learning Burden and Kearney (2015) argue that there is a paradox around the conceptualisation of authentic mobile leaning and its practice when it is often based around classroom activity in formal learning environments. We have argued that mobile learning provides a powerful catalyst FP:42 54 for designing authentic learning environments that bridge formal and informal learning experiences. The key to designing authentic mobile learning is being able to link the unique affordances of mobile devices to the authentic experiences that will broker participation within professional communities. Bannan, Cook and Pachler (2015) argue that “The nature of learning is being augmented and accelerated by new digital tools and media, particularly by mobile devices and the networks and structures to which they connect people (Bannan, et al., 2015, p. 1).” Bannan et al., (2015) identify a range of mobile device affordances, to which we suggest example implementations: • Collaborative and communicative potential; e.g. Twitter • Interactivity and nonlinearity; e.g. Google Now • Distributed knowledge construction; e.g. Google Plus • Multimodal knowledge representation; e.g. YouTube, Jumpcam, Vyclone • Authentic/contextualized/situated material, interaction, tasks and settings; e.g. Augmented Reality • Multi-functionality and convergence; e.g. Siri • Portability, ubiquity, and personal ownership: e.g. Smartphones • User-generated content and contexts: e.g. ePortfolios (Behance) Designing an appropriate ecology of resources for mobile learning will leverage the unique affordances of mobile devices that are relevant to a particular educational context. In particular the crossover between mobile connectivity and social media provides a rich source of resources for social constructivist learning environments. Mobile Social Media With the ubiquity of mobile smart devices that offer constant Internet connectivity, Social Media is now driven by a mobile ecosystem consisting of mobile Apps and connected social media platforms. The ubiquity of mobile device ownership provides an opportunity for exploring the design of authentic learning experiences that focus upon student-generated content and student-generated contexts. These learning experiences create explicit links between formal and informal learning. Thus, mobile learning fosters authentic learning that is not defined by the limits of a walled classroom environment (Cochrane, et al., 2015). We have developed a framework for mobile social media enabling creative pedagogies that can be used to link social constructivist learning theory and collaborative practice in the design of an ecology of resources to support authentic mobile learning scenarios. Similar to Bannan et al., (2015) the framework leverages the unique possibilities of mobile learning to move beyond substitution of current pedagogical strategies towards redefining new pedagogical strategies that were previously difficult or impossible to implement within a traditional classroom setting. The framework maps mobile learning practice to supporting theoretical constructs of creativity (Sternberg, Kaufman, & Pretz, 2002), cognition (Danvers, 2003), educational technology adoption (the SAMR framework (Puentedura, 2006)) and resulting ontological shifts across a pedagogical continuum from teacher-directed pedagogy towards student-determined learning (heutagogy), defined by Luckin et al., (Luckin et al., 2010) as the pedagogy-andragogy-heutagogy continuum (PAH). We have detailed this framework in a variety of contexts (Cochrane & Antonczak, 2014; Cochrane & Rhodes, 2013; Cochrane, Sissons, Mulrennan, & Pamatatau, 2013; Cochrane & Withell, 2013), and provide a summary of the latest version of this framework here in table 1. Table 1: A mobile social media framework for creative pedagogies (modified from Luckin et al., 2010). Pedagogy (P) Andragogy (A) Heutagogy (H) Locus of Control Teacher Learner Learner Early to mid-course: Mid to end of course: Initial establishment of Course Student appropriation Students actively the course and timeframe and of mobile social media participate within an induction into the wider goal and initial active authentic community of learning community participation practice Cognition Level Cognitive Meta-cognitive Epistemic (Danvers, 2003) Knowledge Subject understanding: Process negotiation: Context shaping: production lecturers introduce and students negotiate a students create project FP:43 55 context model the use of a choice of mobile social teams that investigate range of mobile social media tools to and critique media tools appropriate establish an ePortfolio user-generated content. to the learning context based upon These are then shared, user-generated curated, and peer- content reviewed in an authentic COP Substitution & Augmentation: Modification: Redefinition: Portfolio to ePortfolio New forms of SAMR Authentic Community Focus on productivity collaboration (Puentedura, building Mobile device as Mobile device as 2006) Mobile device as personal digital content creation and collaborative tool assistant and curation tool consumption tool Enabling user- Enabling collaboration Supporting Enabling induction into generated content and across user-generated mobile social a supportive learning active participation contexts, and active media community within an authentic participation within a affordances design COP global COP Creativity (Sternberg, et Reproduction Incrementation Reinitiation al., 2002) Reconceptualising mobile social media: Reconceptualising the Reconceptualising the Ontological shift from a social to an role of the teacher role of the learner educational domain This framework creates the foundation for the first stage of an educational design research methodology for curriculum redesign. Educational design research (EDR) Laurillard (2012) makes the case for curriculum design to become a collaborative and design- based activity. In a similar way we are interested in connecting research approaches/methods and design processes. Educational design research (EDR) provides a suitable methodology for innovative curriculum redesign. Design research… integrates rigorous, long-term cycles of applied and empirical research as part of a complex, evolving design process attempting to positively influence and effect change in a learning context through the building of a design intervention through which we uncover pedagogical principles that may be applicable and researchable in similar situations. This is often conducted through identifying and investigating a learning problem, the design and development of an educational innovation and its trial, and iteration in multiple contexts over time. (Bannan, et al., 2015, p. 3) Mor (Emin-Martinez et al., 2014) defines a cycle of steps for enacting EDR within curriculum design that he calls the design inquiry of learning: Imagine: Define an educational challenge that you would like top address. Investigate: Analyse the context, refine the challenge, and identify a suitable pedagogical approach. Inspire: Review examples of past innovations and apply the insights from those to your project. Ideate: Conceptualise a solution. Prototype: A rapid crude implementation to test your ideas. Evaluate: Assess the extent to which your design meets its objectives, identify areas for improvement. Reflect: Produce an account of your design process, the learning experiences you derived from it, and their outcomes. FP:44 56 Bannan (2010) proposes a simpler four stage Integrated Learning Design Framework (ILDF) that encapsulates the design enquiry process: informed exploration, enactment, evaluation of the local impact, evaluation of the broader impact. The intersection of EDR and mobile learning Bannan et al., (2015) argue that the intersection of mobile learning and educational design research provides an approach to deal with the inherent ‘messiness’ of mobile learning. We agree, and propose a curriculum design methodology that is encompassed by an EDR methodology, informed by our mobile social media (MSM) framework, implemented through the design of a mobile social media EOR and a series of triggering events, and evaluated through participant feedback and embedded within a SOTEL research-informed practice approach. Table 2 outlines our simplification of this methodology that links theory, practice, and critical reflection within an EDR methodology. Table 2: The intersection between mobile learning and EDR Methodology Educational Design Research 4 stages of ILDF Informed Enactment Evaluation: Evaluation: Exploration Local Impact Broader Impact Intersection MSM Rhizomatic Learning: SOTEL with mobile Framework Developing an EOR learning informing Designing Triggering Events curriculum Participant Feedback redesign Connecting Theory Practice Critical theory and Reflection practice We used this methodology to guide the development of the mosomelt cMOOC, as outlined in the following section. Our research questions 1. What will an appropriate EOR for sustaining and accrediting an authentic professional development cMOOC look like? 2. How can we design cMOOC-triggering events that focus upon authentic participant- generated mobile learning content? Case study: The mosomelt cMOOC We have found that reconceptualising teaching and learning around new pedagogies requires a significant timeframe to allow for multiple cycles of course redesign, implementation, and critical reflection. In general our professional development COPs have a life cycle that span from one to several years and involve multiple iterations of pedagogical redesign, implementation, and reflection based upon a SOTEL approach. Therefore we decided to implement the mosomelt cMOOC around a full academic year calendar of two twelve-week semesters, rather than the short six-week timeframe typical of many cMOOCs. Our second design parameter was the embedding of the CMALT accreditation process, which allows six months for portfolio curation and submission. The mosomelt cMOOC was therefore designed in two halves: twelve weeks of triggering events exploring the potential of mobile social media in education, followed by twelve weeks of guided participant eportfolio creation for CMALT submission. The mosomelt cMOOC was designed primarily as a framework to link our own professional development COPs, but also to open this to participation from a potentially global community. Hence while mosomelt is designed as a cMOOC the ‘massive’ characteristic is the least important design parameter. Designing a mosomelt cMOOC EOR The ecology of resources supporting the Mosomelt cMOOC was based around an online community discussion forum using Google Plus (G+), participant personal journals using Wordpress, and wider community communication using Twitter. A hashtag (#mosomelt) is used to curate the range of mobile social media platforms explored throughout the mosomelt cMOOC via curation tools such as TAGSExplorer (Hawksey, 2011) and TAGBoard (http://tagboard.com). The mosomelt EOR provides participants with a structure for curating an FP:45 57 eportfolio of evidence and critical reflection for submission towards CMALT accreditation. The mosomelt EOR includes: 3. A G+ community provides a group forum for discussion and sharing of ideas related to the #mosomelt cMOOC. G+ also creates a hub for linking the core social media platforms explored throughout the cMOOC. Wordpress.com is used to provide an outline of each week’s triggering event for the mosomelt cMOOC. Wordpress.com is also the recommended platform for participants to create their own reflective blogs and eportfolios, although any blog host with an RSS feed can be used. A self-hosted installation of Wordpress (http://mosomelt.org) is used to create a participant generated project bank where participants can upload project ideas and comment and rate other participants projects. The project bank utilizes a custom version of a theme developed for the DS106 course (Levine, 2014). Mosomelt.org also hosts a signup form for the participants to enter their contact details to become active participants within the mosomelt cMOOC, including: their G+ profile, Twitter username, and blog address. Participant blogs are then syndicated on Mosomelt.org to enable peer feedback and commenting on one another’s blog posts. Twitter provides a link between participants and their social media activities via the #mosomelt hashtag. Twitter provides an avenue for participation within a global network of like-minded lecturers as well as a broadcast and communication channel for #mosomelt. Designing a series of triggering events The 24 weeks of the mosomelt cMOOC were conceptualised as a series of 24 triggering events, beginning with activities designed to create community, followed by an exploration of the affordances of mobile social media, and then a series of participant generated projects shared through a project bank. The second 12 weeks of the mosomelt cMOOC are designed to guide participants through the requirements of developing a CMALT portfolio based upon the implementation of chosen aspects of their initial 12 week experience within their own teaching practice. The 24 weeks of triggering events are mediated through the mosomelt EOR. The structure of the mosomelt cMOOC in relation to our mobile social media framework is outlined in table 3. Table 3: Overview of the MOSOMELT cMOOC design Timeframe Triggering Activity design Conceptual SAMR events shift Weeks 1-6 Participants Participants create a Teacher Redefinition Introduction explore a series of mobile social media modeled of course to mobile introductory eportfolio from a range educational LMS as a social media mobile social of mobile social media use of mobile collection of and the media platforms tools: G+, Google social media student Mosomelt and short Hangouts, Google and G+ owned community production Drive, YouTube, Vimeo, Community mobile social activities, sharing Twitter, Storify, participation media – their experiences Wordpress, building a via an online Researchgate, and are learning community. invited to join a G+ community community for the course Weeks 7-12 Team based Participants explore Beyond Redefinition Participant collaborative mobile collaboration content of social generated projects over six and co production, delivery to media as a projects weeks, with forming project teams exploration of new several using Google Maps, contextual & pedagogical international guest Vine, Vyclone, and collaborative enabler experts sharing Wikitude. Projects are affordances of their experiences shared for peer mobile via G+ Hangouts. feedback via a “project bank”. Weeks 13-18 CMALT Participants choose a Collaborative Modification FP:46 58 An overview accreditation social learning theory to curriculum of curriculum of CMALT process begins. inform the redesign of a redesign requirements Participants course project for their design a mobile own students. social media Feedback is given activity for their through the G+ own curriculum community. Followed by a week of workshops to embed and strengthen collaboration Week 19-24 Participants Participants use SOTEL Explicit Redefinition Implementing implement and as a framework to reflection on of research and reflecting evaluate their evaluate their course pedagogical as reflective on TEL pedagogical redesign practice practice innovation End of the Portfolio Participants prepare Participation Redefinition cMOOC submission for and submit their within a global of CMALT eportfolios for CMALT educator professional accreditation accreditation network development In the following sections we detail the design of three example triggering events Exploring geolocation During week 3 of mosomelt, participants were invited to co create a collaborative Google Map. The outline of the triggering event was: This week we will explore mobile video production and augmentation via geolocation. You will be invited to collaboratively edit an interactive Google Map, and add a point of interest (POI) with a link to an embedded mobile video. You will receive a link to the collaborative Map through the #mosomelt G+ Community • Slideshow of how to edit a custom Google Map • Example custom Google Map To create and share your own interactive Google Map, login at http://mymaps.google.com. This exercise explores the affordance of smart mobile devices to use their built in GPS and content creation tools (camera, audio and text) to geotag user generated content and create user- generated contexts. User-generated contexts add a contextual layer of information that locates events and experiences within their specific geographic location. Reflect on how can this add value and context to learning activities and experiences. Suggested readings: (Bruns, 2007; Cook, 2007). Exploring collaborative video The week 10 triggering event explored collaborative video production: One of the affordances of the merging of mobile Apps and cloud-based social media platforms is the ability for users to not only generate and share their own content but to also collaborate on it’s production. Explore and create a collaborative video project using an App such as: • Vyclone http://vyclone.com • Jumpcam http://jumpcam.com • Mixbit http://mixbit.com • Frame.io http://frame.io Design an educational scenario that could use collaborative video then upload and share your project outline and any examples via the Project Bank. Reflect on this process on your Wordpress blog. Suggested readings: (Keegan & Bell, 2011; Smith & Byrum, 2013). Exploring augmented reality Week 11 built upon the experiences of co creating a Google Map to create an augmented reality layer for the Wikitude App: This week we are exploring the potential of mobile Augmented Reality (AR) – for example Wikitude, or Junaio, download either of these AR Apps to your mobile device, explore some AR content, then create and share a mobile AR project description to the Project Bank for feedback. Rate another participants mobile AR project. Mobile Augmented Reality utilises a smart device’s built-in camera and geolocation sensors FP:47 59 (GPS, compass, and gyroscope) to overlay the real world environment with digital information, thus augmenting a real-world environment. While mobile AR has predominantly been used for marketing, Museum visits, enhancing Magazines, and other forms of content delivery, there is a range of freely available mobile AR content creation and sharing platforms that can be used for student-generated projects. Start by downloading an AR App to your device – for example Wikitude, and search the available content for project inspiration. For Aucklanders you can search Wikitude for several examples of Architecture student projects: Archifail, Archiwonder, exploreauckland, and the Wynyard Quarter. Hints on using Google Maps and Wikitude to create an AR layer: • Slideshow of creating an interactive Google Map & publishing in Wikitude • https://picasaweb.google.com/104071444159890894025/InteractiveGoogleMaps?feat= directlink#slideshow/5812319248909539778 • Creating an interactive Google Map for geolocating content • https://plus.google.com/+ThomCochrane/posts/SAe1pnLvZfu Reflect on this process on your Wordpress Blog. Suggested readings: (Butchart, 2011; FitzGerald et al., 2013). Results In its first iteration the mosomelt cMOOC has attracted over 40 active participants from six institutions across New Zealand, three institutions in Australia (from Melbourne to Darwin), and as far afield as France. In this section we illustrate the impact of the first half of the mosomelt cMOOC with participant feedback from the development of a new professional development COP within the context of public health education. Three lecturers and one of the authors established the Public Health COP using the mosomelt cMOOC as a framework. The lecturers were equipped with iPad minis and iPhones for use throughout the COP. For one lecturer this was her first experience of using a smartphone, while all three lecturers had limited social media experience and no experience of integrating mobile social media into their teaching practice. The first hurdle was the mosomelt signup process that required participants to create and share a G+ profile, a twitter username, and a Wordpress blog address. Creating and remembering usernames and secure passwords took some time, however the lecturers felt empowered when they succeeded and were then able to join the mosomelt G+ Community, Tweet, and blog from the mobile Apps on their iPads and iPhones. Initial reflections expressed a mix of fear and excitement at what they were experiencing: My very first blog post- eek not really sure what I am doing…but hoping this will change. If technology was a person I don’t think I’d make a very good first impression! I just find instructions really hard to follow and invariably find myself in dark corners of Apps where there seems to be no way out and nowhere to get help… Once I’d mastered creating and loading my Vine video it was almost impossible to understand how I’d got into such a tangle. It all seems so simple now! (Lecturer 1 blog posts, March 2015) Within moments, two colleagues accessed the blog. THEY think I can…so I CAN. Leaping into exciting territory with inspiring and expert colleagues, week by week. (Lecturer 2 blog posts, April 2015) Throughout the first 12 weeks of mosomelt interaction the Public Health lecturers became some of the most active participants and their blog posts illustrated a shift towards conceptualising how they could integrate the use of mobile social media into their own teaching practice, including the use of collaborative video and augmented reality: Week 10 on Vyclone inspired some notions on how it could be applied to invigorate teaching. 1. Four students could video one patient (a student acting as a patient). Each could demonstrate how their video demonstrates their disciplinary perspective, for collaborative discussion and reflection. This promotes and demonstrates the Faculty’s commitment to interdiscplinarity. 2. One mock disaster event could be viewed from the perspective of four FP:48 60 students. From the roof, the overall perspective on how well the strands were managed. From the ground level, the view of the patient, paramedic and other interveners. Again, this contributes to interdisciplinary teaching and understanding. Lecturer 1 and I explored Wikitude and Vyclone. We just created our own independent vyclone on the potential of wikitude for teaching health ethics and law. Admittedly amateur but humorous and we hope inspiring. (Lecturer 2 blog posts, May-June 2015) Back in the office, Lecturer 2 and I decided to practice our new filmmaking skills by creating a brief clip about the ways in which we thought we could use Vyclone: http://www.vyclone.com/movie/556e670f4a384a0306000012 I managed to forget to start to record and then had my finger over the lens for most of the time! But life is for learning! I think students would enjoy using this App. It is straightforward to use and its co- creative nature reflects some of the values that we try to instil in our teaching – working together and recognising different perspectives. (Lecturer 1, blog post June 2015) Overall mosomelt participant feedback thus far has been very positive, and participation levels are high. Table 4 provides an outline of participant activity within the first 12 weeks of the mosomelt cMOOC. Table 5: Mosomelt cMOOC participant first 12 weeks of activity. Mobile social media Activity #mosomelt Tweets 167 conversations involving 69 users Google Plus Community activity 150 posts and 244 comments TAGBoard https://tagboard.com/mosomelt 145 posts Introductory video 31 Vine videos production http://vinebox.co/tag/mosmomelt 10 Instagram videos Collaborative Google Map participants 25 participants Curated social media posts using #mosomelt via 390 Posts Twinesocial http://apps.twinesocial.com/mosomelt Participant blogs 36 Wordpress blogs with an average of 4 pages each. Discussion In this section we discuss the four stages of ILDF within an EDR methodology in the design of the mosomelt cMOOC. Informed Exploration While we have used our mobile social media framework to inform the design of a variety of pedagogical interventions this is the first time we have used the framework to inform the design of a cMOOC. The framework guided the choice of an appropriate EOR and triggering events that leverage the affordances of mobile social media for enabling collaborative learner- generated content and contexts. This methodology links both mobile learning theory and practice, and extends to critical reflection by updating the scholarship of teaching and learning for the mobile social media age by inviting participants to become part of a growing global network of educational researchers via collaborative online research communities such as Researchgate.net, Academia.edu, and Mendeley.com. Enactment In the first iteration of the mosomelt cMOOC Participants enrolled in Mosomelt by creating their own accounts within the EOR social media platforms and then sharing their G+, Twitter, and blog contact details via signing up using a web form at http://mosomelt.org/signup/. They were then invited to become members of the Mosomelt G+ community, which is public but contributions are only allowed by invited members. Participants were then welcomed into the FP:49 61 Mosomelt community via a mention on the #mosomelt Twitter hashtag, and their blogs were curated into a syndicated page at http://mosomelt.org/participants-blogs/. This EOR provided an open public face to the #mosomelt cMOOC, which not all participants were initially comfortable with. Weekly triggering events were outlined on https://mosomelt.wordpress.com before the start of the cMOOC giving participants a structured outline of the 24 weeks. Each weekly triggering event was then detailed further as both a blog post on https://mosomelt.wordpress.com and as a weekly-pinned post on the Mosomelt G+ community. These were both announced via the Twitter hashtag and the same hashtag on G+. So far participants have been far more active in discussion and conversations around #mosomelt on the G+ community than on Twitter. Evaluation: local impact The impact of mosomelt upon the Public Health COP provides an example of transformation of practice. However, not all mosomelt participants are comfortable with publically sharing their journeys, with some COPs preferring to keep their reflections private via collaborating on a Mahara eportfolio. As we head towards the second half of the mosomelt cMOOC and begin focusing upon eportfolios for CMALT accreditation some participants are in catch-up mode. To facilitate this we will run a “winter camp” during the 6 week gap between the end of teaching of the first semester and the beginning of teaching in the second semester 2015. The mosomelt winter camp will consist of four days of workshops that combine both face-to-face modes and online via G+ Hangout covering the 6 project bank project activities. Realistically, some 2015 participants will not be ready for CMALT accreditation this year, while some more experienced practitioners are expected to join the mosomelt cMOOC for the second half to help prepare portfolios for CMALT submission. Thus far we have found the prototype mosomelt cMOOC to be a successful framework for up scaling authentic professional development based around a network of lecturer COPs. Evaluation: broader impact At this stage we are halfway through the first iteration of the mosomelt cMOOC, having just completed the first 12 weeks of triggering events. SOTEL is embedded within the mosomelt cMOOC design explicitly during the second 12 weeks as part of the requirements for CMALT accreditation. As participants begin to publish in peer reviewed conference proceedings, book chapters and journal papers this will create a vehicle for transferring the impact of mosomelt to the wider global education community. In the meantime we are beginning to see the wider impact of the mosomelt cMOOC through the analysis of the open mobile social media EOR behind mosomelt. For example, a TAGSExplorer analysis of the #mosomelt Twitter hashtag shows 69 nodes and 167 edges, indicating the growth in peripheral participation in the #mosomelt community beyond the 44 enrolled participants. At this point we have not explicitly advertised the existence of mosomelt, as we are effectively in the prototyping phase of our EDR, hence the modest growth of the community is to be expected. Conclusions We have demonstrated the use of an EDR methodology for designing an authentic cMOOC for professional development. Our design model links mobile learning theory via a mobile social media framework, practice via the design of a collaborative community engaged by a common EOR and triggering events, and critical reflection via SOTEL within an EDR methodology to create an authentic experience for participating lecturers. By aligning the mosomelt cMOOC with a pre existing accreditation process we have created a fast development model that is validated via active participation and participant-generated personal eportfolios. The CMALT accreditation process and results will be the subject of further evaluation at the end of the first complete iteration of the mosomelt cMOOC. References Bannan, B., Cook, J., & Pachler, N. (2015). Reconceptualizing design research in the age of mobile learning. Interactive Learning Environments, 1-16. Bates, T. (2014). Comparing xMOOCs and cMOOCs: philosophy and practice. http://www.tonybates.ca/2014/10/13/comparing-xmoocs-and-cmoocs-philosophy- and-practice/ FP:50 62 Bruns, A. (2007, March 21-23). Beyond difference: reconfiguring education for the user-led age. Paper presented at the ICE3: Ideas in cyberspace education: digital difference, Ross Priory, Loch Lomond, Scotland. Burden, K., & Kearney, M. (2015). Conceptualising authentic mobile learning. In D. Churchill, T. Chiu & N. Gu (Eds.), Proceedings of the International Mobile Learning Festival: Mobile Learning, MOOCs and 21st Century Learning (pp. 373 - 398). Hong Kong SAR China. Butchart, B. (2011). TechWatch Report: Augmented Reality for Smartphones (PDF). Bristol, UK: JISC. Cochrane, T., & Antonczak, L. (2014). Implementing a Mobile Social Media Framework for Designing Creative Pedagogies. Social Sciences, 3(3), 359-377. Cochrane, T., & Narayan, V. (2014). Cultivating Creative Approaches to Learning. In L. M. Blaschke, C. Kenyon & S. Hase (Eds.), Experiences in Self-Determined Learning (Vol. Paperback and Kindle editions, pp. 149-170): CreateSpace Independent Publishing Platform. Cochrane, T., Narayan, V., & Oldfield, J. (2013). iPadagogy: Appropriating the ipad within pedagogical contexts. International Journal of Mobile Learning and Organisation, 7(1), 48- 65. Cochrane, T., Narayan, V., & Oldfield, J. (2015). Emerging technologies in New Zealand: A pedagogical framework for mobile social media. In V. Bozalek, D. Ngambi, A. Amory, J. Hardman, D. Wood & J. Herrington (Eds.), Activity theory, authentic learning, and emerging technologies: Southern perspectives (pp. 126-143). New York: Routledge. Cochrane, T., & Rhodes, D. (2013). iArchi[tech]ture: Developing a mobile social media framework for pedagogical transformation. Australasian Journal of Educational Technology, 29(3), 372-386. Cochrane, T., Sissons, H., Mulrennan, D., & Pamatatau, R. (2013). Journalism 2.0: Exploring the impact of Mobile and Social Media on Journalism Education. [Journal]. International Journal of Mobile and Blended Learning, 5(2), 22-38. Cochrane, T., & Withell, A. (2013). Augmenting design education with mobile social media: A transferable framework. Journal of the NUS teaching academy (JNUSTA), 3(4), 150-168. Cook, J. (2007). Generating New Learning Contexts: Novel Forms of Reuse and Learning on the Move. Paper presented at the ED-MEDIA 2007 - World Conference on Educational Multimedia, Hypermedia & Telecommunications. Retrieved from http://www.editlib.org/p/25762/ Cormier, D. (2008). Rhizomatic education: Community as curriculum. Innovate, 4(5), np. available http://davecormier.com/edblog/2008/2006/2003/rhizomatic-education-community- as-curriculum/. Cormier, D. (2014). The MOOC that community built. International Journal for Innovation and Quality in Learning, 2(3 Special Issue on Quality in Massive Open Online Courses), 108 - 110. Danvers, J. (2003). Towards a radical pedagogy: Provisional notes on learning and teaching in art & design. International Journal of Art & Design Education, 22(1), 47-57. Emin-Martinez, V., Hansen, C., Rodríguez-Triana, M. J. s., Wasson, B., Mor, Y., Dascalu, M., et al. (2014). TowardsTeacher-led Design Inquiry of Learning. eLearning Papers, 36. FitzGerald, E., Ferguson, R., Adams, A., Gaved, M., Mor, Y., & Thomas, R. (2013). Augmented reality and mobile learning: the state of the art. International Journal of Mobile and Blended Learning, 5(4), 43-58. Friesen, N., & Wihak, C. (2013). From OER to PLAR: Credentialing for Open Education (Vol. 5). Hawksey, M. (2011). Twitter: How to archive event hashtags and create an interactive visualization of the conversation. Blog posted to http://mashe.hawksey.info/2011/11/twitter- how-to-archive-event-hashtags-and-visualize-conversation/ Keegan, H., & Bell, F. (2011). YouTube as a Repository: The Creative Practice of Students as Producers of Open Educational Resources. European Journal of Open, Distance and E- Learning (EURODL)(Special Issue: Creativity and Open Educational Resources), 149-168. Laurillard, D. (2012). Teaching as a design science: Building pedagogical patterns for learning and technology. New York: Routledge. Levine, A. (2013). ds106: Not a Course, Not Like Any MOOC. Educause Review, 48(1), 54-55. Levine, A. (2014, 9 November). Retrieved 8 June, 2015, from https://github.com/cogdog/ds106bank Luckin, R., Clark, W., Garnett, F., Whitworth, A., Akass, J., Cook, J., et al. (2010). Learner- Generated Contexts: A Framework to Support the Effective Use of Technology for Learning. FP:51 63 In M. Lee & C. McLoughlin (Eds.), Web 2.0-Based E-Learning: Applying Social Informatics for Tertiary Teaching (pp. 70-84). Hershey, PA: IGI Global. Mackness, J., & Bell, F. (2015). Rhizo14: A Rhizomatic Learning cMOOC in Sunlight and in Shade. [Massive open online course (MOOC), rhizomatic learning, ethics, learner experience, teaching, Rhizo14]. Open Praxis, 7(1), 25-38. Priem, J., Taraborelli, D., Goth, P., & Neylon, C. (2010, 26 October). Altmetrics: A manifesto. Retrieved 19 June, 2015, from http://altmetrics.org/manifesto/ Puentedura, R. (2006). Transformation, Technology, and Education. Retrieved 18 February, 2013, from http://hippasus.com/resources/tte/puentedura_tte.pdf Siemens, G. (2004). Connectivism: A Learning Theory for the Digital Age. eLearnspace, (12 December). Retrieved from http://www.elearnspace.org/Articles/connectivism.htm Smith, S., & Byrum, D. (2013). Using a BYOD Model to Teach a Graduate Level Video Production Course to In-service Teachers. Paper presented at the Society for Information Technology & Teacher Education International Conference. Sternberg, R. J., Kaufman, J. C., & Pretz, J. E. (2002). The creativity conundrum: A propulsion model of kinds of creative contributions. Philadelphia: Psychology Press. Wenger, E. (1998). Communities of Practice: Learning, Meaning, and Identity. Cambridge: Cambridge University Press. Wenger, E., White, N., & Smith, J. (2009). Digital Habitats: stewarding technology for communities. Portland, Oregon: CPsquare. Wickens, R. (2006). SoTEL: Toward a Scholarship of Technology Enhanced Learning. Canadian Journal of University Continuing Education 32(2), 21-41. Cochrane, T., Narayan, V., Burcio-Martin, V., Lees, A., & Diesfeld, K. (2015). Designing an authentic professional development cMOOC. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:41-FP:52). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:52 64 Investigating the effectiveness of an ecological approach to learning design in a first year mathematics for engineering unit Iwona Czaplinski Science and Engineering Faculty Queensland University of Technology, Australia This paper reports on the results of a project aimed at creating a research- informed, pedagogically reliable, technology-enhanced learning and teaching environment that would foster engagement with learning. A first-year mathematics for engineering unit offered at a large, metropolitan Australian university provides the context for this research. As part of the project, the unit was redesigned using a framework that employed flexible, modular, connected e-learning and teaching experiences. The researchers, interested in an ecological perspective on educational processes, grounded the redesign principles in probabilistic learning design (Kirschner et al., 2004). The effectiveness of the redesigned environment was assessed through the lens of the notion of affordance (Gibson, 1977,1979, Greeno, 1994, Good, 2007). A qualitative analysis of the questionnaire distributed to students at the end of the teaching period provided insight into factors impacting on the successful creation of an environment that encourages complex, multidimensional and multilayered interactions conducive to learning. Keywords: ecology of learning, affordances, blended learning, probabilistic learning design Introduction Modern higher education is facing the challenge of assisting university students to develop 21st century-specific skills such as transmedia navigation, critical thinking, problem solving and creativity. This challenge necessitates an innovative approach to learning and teaching, one that combines recent advances in research on human cognition, perception, acquisition, learning and teaching with the institutional requirements of preparing graduates for the rapidly changing modern world. What would be the best way of describing this modern, dynamic and complex environment? Within the context of higher education, the term “knowledge-based economy” (Powell and Snellman, 2004) emphasises the role of humans’ cognitive skills and capabilities in advancing technological and scientific progress on unprecedented scale. However, the rapidity of these changes makes them equally quickly obsolete, which in its turn, creates a need for more discoveries and progress. This constantly changing nature of knowledge-relying professional environment requires constant upskilling, therefore learning. George Siemens described this phenomenon in terms of “perpetual learning” (Siemens, 2015). According to the researcher, current students are facing 40 years of learning (rather than 4), at different levels and focused on developing/ mastering different skills. So this raises the questions: how are we to assist learners with the development of skills allowing them to perpetually learn? How are we to prepare them for the challenges of this new type of economy – a learning economy? To successfully face the above-mentioned challenges, modern higher education institutions need to take a more holistic approach to designing, developing, implementing and evaluating students’ learning experiences. Technology-enhanced learning (Laurillard et al., 2009) offers a research paradigm able to inform the “design for learning” (Goodyear and Carvalho, 2013 p. 49), the pedagogical approach applied by people to facilitate other people’s learning by “working with networks of interacting digital and non-digital entities” (Goodyear and Carvalho, 2013, p. 49). Such an ontological position implies an ecological worldview on learning and teaching processes, one that is interested in studying a complex network of multilayered interactions and FP:53 65 resulting interdependencies between all constituents of the environment occurring at all levels of interaction: physical, social and cognitive. Mindful of the above-mentioned critical considerations, the researchers adopted a probabilistic (as opposed to the classical, causal) approach to learning design (Kirschner et al., 2004). More precisely, the researchers undertook the task of creating a “world of learning” (Kirschner et al., 2004, p.25), a specific, technology-enhanced learning and teaching environment that provides opportunities for complex, multilayered and multidirectional interactions between all constituents of the environment (i.e. virtual networks and social agents). This type of environment encourages learning processes by providing various opportunities for action. In short, the researchers’ intent was to create an environment that would be cohesive and coherent on one hand and would foster the complexity of interactions on the other. This study investigated if a cohesive, coherent and engaging technology-enhanced learning and teaching environment created by the researchers was successful in promoting learning. A first- year mathematics for engineering unit offered at a large, metropolitan Australian university was chosen as the context for the research. The researchers redesigned the unit to embed flexible, e-learning and teaching experiences within formal and informal settings. The research design focused on investigating the effectiveness of the technological, social and educational opportunities for action, or affordances, (Laurillard et al. 2000; Kirschner, 2002; Kirschner et.al, 2002, 2004; Good, 2007; Czaplinski, 2012; Czaplinski et al. 2015) offered by the created environment. Data were collected through a paper-based questionnaire distributed to students at the end of the teaching period. The questionnaire evaluated the effectiveness of the redesign by looking at students’ perceptions of achieving learning outcomes, satisfaction with the unit’s organisation (cohesive and coherent environment) and teaching approaches, and finally, student engagement with the unit content. In their initial hypothesis formulated at the beginning of the project, the researchers assumed that by creating cohesive and coherent environment that provides multiple and various opportunities for action (including deep engagement with knowledge), the learners will engage in complex and meaningful relationships with both human and non-human constituents of the environment, and in this way will adopt a deep approach to learning. The specific research questions were: 4. What were students’ perceptions of achieving unit learning outcomes? 5. To what extent were students satisfied with the unit organisation? 6. To what extent were students satisfied with the unit delivery? 7. To what extent were students engaged with the unit content? The data analysed through the theoretical lens of the notion of affordance (Gibson, 1977, 1979, Greeno, 1994, Good, 2007; Czaplinski, 2012; Czaplinski et al., 2015), allowed the researchers to shed light on the ways the learning process was mediated by the specifically designed technology-enhanced environment within formal and informal settings. Technology-enhanced learning Relationships, context, emergent patterns, quality, value, critical perspective, diversity and agency are major characteristics of an ecological approach to learning (van Lier, 2010). Together, they pose three important challenges to technology-enhanced learning. First, they require the creation of networks, both human and virtual. Second, the virtual networks need to become a platform for interaction between digital entities, i.e. electronic systems, and non- digital entities, i.e. social agents taking part in the learning and teaching processes. Third, in order to foster learning, they require active engagement happening at various levels, the highest being meaningful and deep engagement with knowledge, (Marton & Säljö, 1976; Entwistle, 1981, 2000, 2009; Ramsden, 1992; Biggs &Tang, 2007), the sine qua non of understanding. All these challenges emphasise the interplay between non-digital and digital constituents of technology-enhanced learning. They both form an entity, they interact with each other, their relationship is bidirectional, hence they both need to be investigated in parallel, since “there is no person without environment and no environment without a person (or organism) dwelling in it” (Goodyear and Carvalho, 2013, p. 50). Such an ecological perspective on human cognition sees acquisition of knowledge as a process taking place outside of the individual (van Lier, 2000; Fettes, 2003, Czaplinski, 2012). It can be described in terms of a constant, dynamic, FP:54 66 labile, and diachronic interaction, a type of discovery of an individual’s world through his/her cognitive tools (Reed, 1996; Fettes, 2003; Czaplinski, 2012). This mutualist point of view, one in which “mind, body and environment cannot be understood in isolation, but are constructions from the flow of purposive activity in the world” (Good 2007, p. 269), has important consequences for theory of learning, learning design and development, especially within technology-enhanced learning and teaching environments. The environment shapes learner’s knowledge as much as the learner shapes his/her environment. Therefore, the provision of opportunities for learning, their quality, learners’ capability and readiness of perceiving them, the decision of taking or not taking them up and the capacity of adapting them to learners’ individual needs become crucial, interdependent constituents of ecological contexts. The acts of cognition, acquisition and learning are based on complex learners’ interactions with the environment, constant discovery and (re-)negotiation of meanings embedded in the environment (van Lier, 2000). Such duality necessitates flexibility of the learning design. On one hand, the ecological worldview requires the learning design to consider learners’ identities and to encourage their agency with the purpose of enhancing their motivation. On the other hand, the learning design should also assess technology for its capability of providing rich and (good) quality learning experiences. For TEL to be effective, educators, developers and designers need to shift attention from individual aspects of the environment and adopt an all-inclusive approach, one that encompasses the characteristics, particularities (and preferably even idiosyncrasies) of both, digital and human constituents allowing all social agents of the educational process (e.g. students, lecturers, tutors, developers, designers, visiting lecturers, etc.) to adapt to the environment. The important question is “how?” How to identify the above- mentioned opportunities, how to make sure they will be perceived by social agents and how to ensure their effective (educationally beneficial) use. Laurillard explained these challenges in the following way: “our perspective is […] oriented towards the role of technology to enable new types of learning experiences and to enrich existing learning scenarios. To do this successfully, we have to understand not just teaching and learning, but also, the context in which the implementation of technology-enhanced learning (TEL) has to take place” (Laurillard et al., 2009 pp. 289-290). Research context and methods The current paper reports on the final stage of a three-semester long project, focusing on successive deliveries of the same, first-year mathematics for engineering unit. This unit is a foundational subject that provides the mathematical knowledge and skills that underpin later engineering studies. The mathematical content includes topics such as functions, complex numbers, calculus, matrices and vectors. The unit has faced several challenges, such as high teaching team turnover, and a diverse range of teaching and pedagogical styles. These included teacher-centred methodologies, characterised by transition-focused lecturing, allowing for limited collaborative learning, drill- focused workshops, and basic use of online tools. The diverse student cohort has posed a double challenge to teaching staff. First, significant discrepancies with mathematical knowledge and skills between students enrolled in the unit have caused some students to experience a sense of being “out of place” and feelings of frustration with unsatisfactory learning progression. Some students reported a sense of confusion as it appeared they lacked a clear understanding of the relevance of the unit to their particular engineering degree. The resultant unit evaluation completed by students indicated a low satisfaction rate and low progression with a reasonably high failure rate forcing many students to repeat the unit. Table 1 summarises the diversity of student cohort based on the degree-type. Table 1: Diversity of student cohort based on field of degree (N=130) FP:55 67 Degree Number of students enrolled Engineering, including: mechanical, civil, electrical, power, 92 telecommunications, aerospace/avionics, medical, mechatronics. Science, including: physics, astrophysics, biology, public health, 13 environmental sciences, chemistry, mathematics, geology. Double degrees, including: engineering/information technology; 14 business/engineering; Information systems 1 Visiting students (High School students) 9 Visiting students (international exchange) 1 In the initial phase of the project, the researchers defined three design principles which constituted the basis for scoping research questions. First, technology needed to be used to create an overarching environment, one that would be easily accessible and would provide all involved with opportunities to connect, regardless their status (learners, educators, learning support), physical location or technological savviness. Second, technology should serve as a catalyst for learning. By interacting with other social agents, and with the technological tools, through and within the technology-enhanced environment, social agents’ attention should be diverted towards the opportunities for learning. That is, while educators’ attention should focus on making the opportunities for learning salient to students, students’ attention should be diverted to perceiving and taking up (or consciously rejecting) multiple affordances for learning. Third, the environment should foster student engagement by providing a platform for blending different educational approaches (e.g. individual learning, collaborative learning, flipped learning) and in this way support the acquisition of knowledge and skills. The above-mentioned principles were enacted in different ways. These new ways included changes made to the online platform, teaching methods, and the inclusion of a learning support team in the unit delivery. Building on the principles of probabilistic learning design (Kirschner et al., 2004), the researchers redesigned the unit with the intention to create a truly blended educational experience. The authors designed technology-enhanced, modular learning and teaching environments that blended physical and virtual spaces into a cohesive and coherent entity. The physical modules included lectures, and workshops and were complemented with the virtual components encompassing pre-lecture videos, WeBWorK (an online testing tool capable of appropriately representing mathematical problems and analysing algebraic responses for correctness), and additional learning resources in the form of contextualised, applied and motivational problems to be used during face-to-face contact hours (named “challenge questions”). In addition, a series of learning support activities, delivered by the university’s mathematics learning support team, was included in the design. The (re-)design principles, were anchored in research within learning design (Kirschner et al., 2004), blended learning (Partridge et al., 2011; Saliba, et al. 2013) in the context of mathematics courses (Stevenson and Zweier, 2011; Calderon, et al., 2012; Carbonell, et al., 2013; Czaplinski et al., 2015) and also tested the effectiveness of an emerging instructional approach of flipped learning (Abeysekera and Dawson, 2015; Estes, Ingram and Liu, 2014; Hamdan et al., 2013; Herreid and Schiller, 2013; Jamaludin and Osman, 2014; Willey and Gardner, 2013). The changes were introduced sequentially over three semesters, starting from summer semester of 2013 until semester 2, 2014. One of the important elements in creating the “world of learning” was to design a learning platform that would reflect the underpinning philosophy of an ecological approach emphasising cohesiveness and coherence of the environment. A platform that would provide a logical, smooth, and straightforward connection between particular virtual modules and, at the same time, would graphically represent the connection between the virtual and physical modules. To this end, the authors analysed technological affordances offered by Blackboard, the standard Learning Management System used at the university. The intention was to identify the affordances offered by the system to identify multiple and varied options to facilitate learners’ perceptions. By providing multiple means of representation, the environment would cater for different types of learners, maximise the opportunities for perceiving the overall organisation of the unit and in this way optimise the opportunities for FP:56 68 learning. It was thought that this would result in higher student satisfaction with the unit design and delivery and better engagement with knowledge. Figures 1 and 2 below show the final design of the platform. Figure 1: Blackboard site screenshot top of the page Figure 2: Blackboard site screenshot bottom of the page Three alternative and complementary visual display means were utilised, providing rich stimuli for perceiving opportunities for different types of actions, namely: 1. Clickable images representing interconnected balls forming a cycle, 2. An interactive unit map, and 3. Clickable tabs. The clickable image emphasised the nature of the activities. The researchers intended to present to students the image of an all-encompassing structure, composed of virtual (“How am I travelling?”, “Online consultations”), in-class (“Workshop/tutorial”, “Lecture”, “Problem-based activities”) and out-of-class opportunities for learning (“STIMulate session” – university sponsored, co-curricular learning support initiative featuring weekly academic-led workshops as well peer-led support for mathematics). This visual representation also reinforced the student- centred approach, harmoniously encompassing interconnected (and interdependent) modules. The tabs, located to the left, played a functional role. Associated with the standard design of the Blackboard site, the tabs were there for those students who would feel lost facing an unexpected design of the site. The tabs also provided additional opportunities for action, such as communication (“Announcements”), as well as emphasising important unit elements (“Assessment”). In this way, information about this part of the unit’s content was displayed using a variety of visual supports, optimising the opportunities for being perceived and accessed. Finally, the clickable unit map not only represented a chronological ordering, assuring students of the orderly, well-planned organisation of the unit, but most importantly clearly provided unit contents (pre-lecture videos presented sequentially, broken down into “steps” within each FP:57 69 weekly module, complemented by additional resources and, again assessment details). The design of the Blackboard, Learning Management System encapsulated pedagogical principles underpinning the redesign of the unit. It provided learners with multiple and diverse occasions for perceiving opportunities for learning taking them up and enacting them through meaningful engagement with content (educational affordances), technological tools (technological affordances), and co-construction of knowledge in collaboration with other students and academics (social affordances). Data collecting involved using mixed methods (quantitative and qualitative) (Hopkins, 2002; McNiff and Whitehead, 2002) administered to students in the form of a paper-based questionnaire at the end of the semester. The questionnaire was designed to provide answers to the above-mentioned research questions focusing on the effectiveness of the design. The questionnaire used a combination of structured (i.e. Likert-scale, open/closed), and unstructured questions (i.e. open comments). The responses were evaluated through the theoretical lenses of the notion of affordance (Laurillard et al. 2000, Kirschner et al., 2002, 2004; Good, 2007; Czaplinski, 2012; Czaplinski et al., 2015), allowing discovery of learners’ patterns of behaviour, hence testing the effectiveness of the created “world of learning”. More precisely, once the survey responses collected, the data were organised in tables, showing numerical representation of students’ responses. Additionally, students’ comments were consulted to clarify/ provide insight into the conclusions drawn from numerical data. To assure the accuracy of the conclusions drawn from qualitative data, quantitative data on student satisfaction and student engagement with Blackboard Learning Management System were collected. The quality of the learning experience is a condition of achieving educational excellence. It depends on the ways within learners’ unique environments, which means the characteristics of a particular cohort, are established, their learning needs identified and catered for, using tailored approaches. This requires educators not only to be aware of their own and learners’ attributes as well educational environment characteristics, but also to be able to analyse them from the perspective of their effectiveness in fostering excellence (Czaplinski, 2012). The concept of affordance offers a theoretical lens for such investigation. Findings and discussion The researchers used a psychological perspective on the notion of affordance (Gibson, 1977, 1979; Good, 2007), which can be explained in terms of a unit of analysis composed of an opportunity for action “nested” (Good, 2007, p. 277) within a functional context. Functional context, or nested in the frame of reference, triggers the act of perceiving an opportunity for action. Frame of reference influences the way how the environment is perceived (including the opportunity) and impacts on the decision to take it up (or not). In its entirety, the three layers form an affordance. Such an interpretation stresses the importance of all constituent ‘layers’ of the concept and emphasises their interdependencies. However, not all affordances are of the same nature. Following from the work of Kirschner et al., (2002, 2004), the researchers investigated the perception and uptake of three different types of affordances: 1. technological, understood as properties of the object that make it easy to use, 2. social, defined as the properties of the environment that encourage social interaction, and 3. educational, understood as the properties of a particular pedagogy applied to a particular cohort of learners within a particular environment (2004, p.28). The first research question investigated students’ perceptions of achieving learning outcomes. The researchers’ objective was to reveal , within the created environment, if students perceived their learning as successful in terms of academic achievement. Or, seen from the theoretical background, if the stimuli embedded in the functional context successfully encouraged the uptake of educational affordances. The researchers assumed that the positive perception of academic achievement results from the uptake of affordances offered by the environment. This might suggest that the interaction between the learner and the environment was conducive to construction of new knowledge. Although this does not directly imply deep learning, there are premises (student satisfaction, perception of achievement) indicating that deep learning might have taken place. The table below summarises the responses of participating students. FP:58 70 Table 2: Summary of questionnaire responses to research question 1 (N=39) I believe that I am better at: Agree Neutral Disagree … understanding and interpreting mathematical notation 30 9 0 … recognizing, manipulating and solving mathematical 33 6 0 expressions … understanding and applying elementary functions, their 29 9 1 derivatives and integrals, complex numbers, matrices and vectors … employing mathematical techniques to solve elementary 26 10 3 problems provided in an engineering context The responses clearly indicate high level of students’ positive perception of their academic achievement . This observation is confirmed by the results included in Student Evaluation Reports, standard evaluation tools, namely “Pulse” collected in the first half of the semester and “In Sight”, focusing on overall student satisfaction and conducted at the end of the semester. The two figures below present the summary of students’ satisfaction. The scale ranges from 0 to 5, with units scoring below 3 considered low performing and those scoring above 4 seen as highly performing. Figure 3: Pulse Student Feedback Results Figure 4: In Sight Student Feedback Results Based on these comments, the researchers concluded that the unit successfully engaged students into learning by creating an appropriate environment. Furthermore, this means that the researchers’ and students’ perceptions of the educational affordances offered by the created environment coincided. Analysed from the notion of affordance, this signifies that the frames of reference of both types of social agents strongly overlapped with regards to understanding what technology-enhanced learning environment should look like in order to be successful in fostering learning. Moreover, it seems that the functional context (actual activities triggering action) appeared to be effective in fostering students’ learning. From the questionnaire responses the researchers conclude that students’ perception of achievement was influenced not only by an appropriate environment; it was also triggered by teaching methods applied during the semester. Responses to the second question confirm the above conclusion. As already mentioned, the unit adopted a modular structure that blended physical and virtual spaces into a cohesive and coherent entity. One of the crucial, and most difficult, parts of the design was the assurance of connection between both types of modules. The most significant challenge was making sure all components were appropriately “blended”. The responses indicated that this objective has been achieved. Table 3 summarises the responses. Table 3: Summary of questionnaire responses to research question 2 (N=39) FP:59 71 Question Agree Neutral Disagree The unit was well organized. 33 5 1 I could see clear connections between pre-lecture videos, 33 6 0 lectures, workshops, STIMulate sessions and online practice quizzes. It is important to note that the results imply students’ high satisfaction with the coherent and cohesive nature of the environment. Students’ comments confirm this conclusion. One student wrote: “ This was one of the best organized units that I have done, which really helped my learning”. In addition, the data on user activity provided by Blackboard Learning Management System shows patterns of behavior suggesting high activity rate maintained almost throughout the whole week, and this for the duration of the semester. The figure 5 below illustrates this observation. Figure 5: User activity by day throughout the semester The pattern of daily activity correlates with the timetable of the unit, with Lectures scheduled for Tuesday (2 hours) and Thursday (1 hour), and Workshops being run on Tuesdays after the Lectures and Wednesdays. It seems that students took the opportunity for engaging with the content through the LMS on a fairly constant basis, with activity happening not only on days of the contact with lecturer/ tutors (Tuesday, Wednesday and Thursday), but also on days when there was no direct contact with the teaching team, including Sunday. While the first two questions primarily focused on the role frames of reference played in making the environment successful, the third research question explored the role functional context played in triggering action, i.e. uptake of the three identified types of affordances (educational, social and technological) . To gain insight into this question, the researchers asked two types of questions investigating: 1. the delivery and 2. the ways students used the online tools. Table 4 summarises the responses. “E” signifies educational affordance and “S” stands for social affordance. Table 4: Summary of questionnaire responses to research question 3, focus on delivery (N=39) Question Agree Neutral Disagree No response Pre-lecture videos helped me with understanding 38 1 0 0 the lecture content.(E) During lectures I could apply the information from 35 3 1 0 the pre-lecture videos to understand the theory being presented. (E) The lectures were taught in the way that allowed me to engage with: 36 2 1 0 FP:60 72 the material (E) 22 17 0 0 my colleagues (S) 31 8 0 0 my lecturer/ tutor (S) 16 20 3 0 STIMulate support (S) The workshops helped me to see how the 25 10 2 2 mathematics relates to my other studies. (E) I found the contextualised, applied and 25 12 1 1 motivational problems used to link lectures and workshops useful for my engagement. (E) These contextualised, applied and motivational 32 4 2 1 problems allowed me to apply the theory covered in the lectures to practical uses in workshops.(E) The workshops were taught in a way that allowed me to engage with: 32 5 2 0 the material (E) 28 10 1 0 my colleagues (S) 29 8 2 0 my lecturer/ tutors (S) 15 21 3 0 STIMulate support (S) The questionnaire responses demonstrate high satisfaction rate with the delivery methods. Engaging, providing strong connections between theoretical (pre-lecture videos, lectures) and practical modules (workshops) and expansions (contextualised, applied and motivational problems), they proved excellent trigger for assisting student in perceiving two types of affordances (educational, social) and successfully taking them up. In other words, not only it provided appropriate, complex stimuli, but it also successfully made them salient to learners in a way that majority of respondents perceived and took the affordances up. This is confirmed by the following comment made by a student in an open-ended section of the survey: “I found workshops were really beneficial as we got to work on a number of examples and developed a deeper understanding of the subject matter”. As for the remaining technological affordance, it was investigated closely with the next question, focusing on the ways the tools were used in the unit. Table 5 summarises the responses. “E” signifies educational affordance, “S” stands for social affordance and “T” relates to technological affordance. Table 5: Summary of questionnaire responses to research question 3, focus on online tools (N=39) Question Agree Neutral Disagree No respon se Online diagnostic: was easy to use (T) 13 20 4 2 helped me with practicing the theory (E) 22 11 4 2 results motivated me to seek external help (such as tutor, 15 17 5 2 peer STIMulate) (S) quizzes were beneficial for my learning (E). 18 16 3 2 Pre-lecture videos were technologically easy to use (T). 34 3 0 2 I watched pre-lecture videos prior to attending lectures 33 3 0 3 (E). The content of the pre-lecture videos was easy to follow 36 0 1 2 (E). The content of the pre-lecture videos helped me with 35 1 1 0 practicing what was presented during lectures & workshops (E). The content of the pre-lecture videos allowed me to 31 5 1 2 discuss some mathematical questions with my peers, tutors, lecturer (S). Overall, pre-lecture videos were beneficial for my learning 36 1 0 2 (E). FP:61 73 In summary, the responses to the third research question “were students satisfied with the unit delivery?” are overwhelmingly positive. Students were satisfied with the ways the unit was delivered and, as a result, they perceived and took up technological, social and educational affordances of the “world of learning”. Finally, as mentioned above, the re-design of the unit also included out-of-class opportunities for learning provided by a university’s learning support program (“STIMulate session”). The researchers made a conscious effort of embedding this module in the structure of the unit to the extent of making it “invisible”, that is completely blending, non-compulsory, supportive and out- of-class activities with the remaining modules of the unit. There were multiple reasons behind the inclusion of the STIMulate sessions, the most important being providing students with as many opportunities of co-constructing knowledge as possible. Based on works by Vygotsky (1978) and his views on Zone of Proximal Development, the researchers believed that this particular module, if appropriately presented to learners as an opportunity of making learning progress, will successfully assist students with learning. Table 6 below summarises students’ responses. Table 6: Summary of questionnaire responses to research question 4 (N=39) Question Agree Neutral Disagree No response I was familiar with the STIMulate section on 14 17 6 2 the unit BB site. I knew where the STIMulate tutors were 24 12 1 2 located. I used STIMulate support for this unit. 13 14 10 2 STIMulate sessions were beneficial for my 15 17 5 2 learning. Based on the responses from students, the researchers conclude that this part of the re-design was the most challenging. Although made salient to students (most students did indicate knowing the location of the STIMulate tutors), it seems that the uptake of this affordance was not fully successful. Respondents’ comments to this question might provide explanation why. Many students reported on not having the additional time to take advantage of this opportunity. For instance, one student wrote: “ Unable to attend STIMulate due to work commitments”, while another student stated: “ Never went, had work on Wednesday”. It seems that students ‘frame of reference (student but at the same time, an employee), prevented students from taking up these educational and social affordances. In summary, the researchers conclude that their ecological approach to learning, based on probabilistic learning design proved successful in promoting students’ engagement with learning not only through unit content but also effective delivery fostering engagement. Conclusion Modern education is facing a challenge on unprecedented scale – how to prepare students to the requirements of the “learning economy”, knowing that the world is only going to become more complex. Complex does not equal complicated. Gardner Campbell (2015), explained the important difference between complexity and complication. While complicated systems can be organised, planned, structured and controlled, complex systems escape such classification (and characterisation). Unpredictable, complex systems are at the forefront of the new order which, with time, could be theorised into a framework or a model. The researchers undertook the task of addressing the complexity of technology-enhanced learning and teaching environments, by adopting an ecological perspective on learning resulting in creation of “world of learning”. The results clearly shows that, overall, the adopted direction proved appropriate and beneficial to student learning. In response to four research questions, the researchers conclude that their attempt in creating a coherent and cohesive technology- enhanced learning and teaching environment was mostly successful. Students’ perceived their academic achievement very positively as they engaged with learning through three modular pillars of the unit’s environment: online, face-to-face delivery by teaching staff and collaborative FP:62 74 co-construction of knowledge by support sessions. The results indicated the importance of a careful observation of the ever-changing environment, analysis of its constituents and reflection on the best ways of making opportunities for learning salient to all social agents. Such holistic understanding of the learning environment, seen as a learning ecosystem encompassing all constituents has the potential of assisting learners with the development of a very important skill – meaningfully engaging with learning. References Abeysekera L., & Dawson, P. (2015) Motivation and cognitive load in the flipped classroom: definition, rationale and a call for research. Higher Education Research & Development, 34(1), 1-14. Biggs, J. & Tang, C. (2007). Teaching for Quality Learning at University. Third Edition. Open University Press. Carbonell, K., et al. (2013). Unleashing the creative potential of faculty to create blended learning. The Internet and Higher Education, 18, 29-37. Calderon, O., et al. (2012). Multidimensional assessment of pilot blended learning programs: Maximizing program effectiveness based on student and faculty feedback. Journal of Asynchronous Learning Networks, 16, pp. 23-37. Campbell, G. (2015). Education for uncertainty. Key note given at HERDSA 2015 Conference: Learning for Life and Work in a Complex World, 6-9 July 2015, Melbourne, Vic. Czaplinski, I. (2012). Affordances of ICTs: an environmental study of a French language unit offered at university level. Master of Philosophy thesis, University of Queensland. http://espace.library.uq.edu.au/view/UQ:289156 Czaplinski, I., Mallet, D., Burrage, P. & Psaltis, S. (2015). Preparing engineering graduates for the knowledge economy through blended delivery of mathematics. In HERDSA 2015 Conference: Learning for Life and Work in a Complex World, 6-9 July 2015, Melbourne, Vic. Retrieved July 10, 2015 from http://eprints.qut.edu.au/84517/ Entwistle, N. (1981). Styles of Learning and Teaching: An Integrated Outline of Educational Psychology for Students, Teachers and Lecturers. Chichester: Wiley. Entwistle, N. (2000). Promoting deep learning through teaching and assessment: conceptual frameworks and educational contexts. TLRP Conference, Leicester, November 2000. Retrieved January 14, 2015, from http://www.tlrp.org/acadpub/Entwistle2000.pdf Entwistle, N. (2009). Teaching for Understanding at University. Deep Approaches and Distinctive Ways of Thinking. Basingstoke: Palgrave Macmillan. Estes, M.D., Ingram, R., & Liu, J.C. (2014). A review of flipped classroom research, practice, and technologies. International HETL Review, 4(6) Fettes, M. (2003). Critical realism, ecological psychology, and imagined communities: foundations for a naturalist theory of language acquisition. In j. Leather, & J.van Dam (Eds.), Ecology of Language Acquisition (pp.31-47). Boston, MA: Kluwer Academic Publishers. Gibson, J.J. (1977). The theory of affordances. In R.E. Shaw & J. Bransford (Eds.), Perceiving, Acting and Knowing. Hillsdale, NJ: Lawrence Erlbaum Associates. Gibson, J.J. (1979). The ecological approach to visual perception. Mahwah, NJ: Erlbaum. Good, J.M.M. (2007). The affordances for social psychology of the ecological approach to social knowing. Theory & Psychology, 17(2), 256-295. Goodyear, P. & Carvalho, L. (2013). The analysis of complex learning environments. In H.Beetham, & R. Sharpe, (Eds.). Rethinking Pedagogy for a Digital Age. Routledge, Second Edition. Greeno, J.G. (1994). Gibson’s Affordances. Psychological Review, 101(2), 336-342. Hamdan, N., McKnight, P., McKnight, K. & Arfstrom, K. (2013). A review of flipped learning. Retrieved February 16, 2015, from http://www.flippedlearning.org/review Herreid C.F. & Schiller, N.A. (2013). Case studies and the flipped classroom. Journal of College Science Teaching, 42(5), 62-66. Hopkins, D. (2002). A teacher’s guide to classroom research. (3rd ed.). Buckingham- Philadelphia: Open University Press. Jamaludin, R. & Osman, Z. (2014). The use of a flipped classroom to enhance student engagement and promote active learning. Journal of Education and Practice, 5(2), pp. 124- 131. FP:63 75 Jochems, W. M. G., van Merriënboer, J. J. G., & Koper R. (Eds.) (2004). Integrated e-learning: implications for pedagogy, technology and organization. London, England: Routledge Falmer. Kirschner, P. (2002). Can we support CSCL? Educational, social and technological affordances for learning. In P. Kirschner (Ed.), Three worlds of CSCL: Can we support CSCL. Inaugural address, Open University of the Netherlands. Retrieved February 10, 2015, from https://www.ou.nl/documents/Promoties-en- oraties/Oraties/Oraties2002/oratieboek_PKI_DEF_Klein_ZO.pdf Kirschner, P., Strijbos, J.-W., & Kreijns, K. (2004). Designing integrated collaborative e-learning. In Jochems, et al. (Eds.) (2004). Integrated e-learning: implications for pedagogy, technology and organization. London, England: Routledge Falmer. Laurillard, D., Stratfold, M., Luckin, R., Plowman, L., & Taylor, J. (2000). Affordances for learning in a non-linear narrative medium. Journal of Interactive Media in Education, 2, 1-19. Laurillard, D., Oliver, M., Wasson, B., & Hoppe, U. (2009). Implementing technology-enhanced learning. In N.Balacheff et al., (Eds.) Technology-Enhanced Learning (pp. 289-306). Springer Science + Business Media. Marton, F., & Säljö, R. (1976). On qualitative differences in learning: 1. Outcome and process. British Journal of Educational Psychology, 46(1), 4-11. McNiff, J. & Whitehead, J. (2002). Action research: principles and practice. (2nd ed.) SAGE. Partridge, H., Ponting, D., & McCay, M. (2011). Good practice report: blended learning. Australian Learning and Teaching Council. Powell, W.W., Snellman, K. (2004). The knowledge economy. Annual Review of Sociology. 30, 199-220. Ramsden, P. (1992). Learning to Teach in Higher Education. London: Routledge. Reed, E.S. (1996). Encouraging the world: towards an ecological psychology. New York :Oxford University Press. Saliba, G., Rankine, L., Cortez, H. (2013). Fundamentals of blended learning. University of Western Sydney. Siemens, G. (2015). How we live and work is an important concern in our complex society. Key note given at HERDSA 2015 Conference: Learning for Life and Work in a Complex World, 6- 9 July 2015, Melbourne, Vic. Stevenson, K. & Zweier, L. (2011). Creating a Learning Flow: A Hybrid course model for High- Failure Maths Classes. EDUCAUSE Quarterly, 34(4). van Lier, L. (2000). From input to affordance: Social-interactive learning from an ecological perspective. In J. Lantolf (Ed.), Sociocultural theory and second language learning (pp. 245- 60). Oxford, England: Oxford University Press. van Lier, L. (2010). The ecology of language learning: practice to theory, theory to practice. Procedia Social and Behavioral Sciences 3, 2–6. DOI:10.1016/j.sbspro.2010.07.005. Retrieved July 10, 2015 from http://www.sciencedirect.com/science/article/pii/S1877042810013790 Vygotsky, L. S. (1978). Interaction between learning and development (M. Lopez-Morillas, Trans.). In M. Cole, V. John-Steiner, S. Scribner, & E. Souberman (Eds.), Mind in society: The development of higher psychological processes (pp. 79-91). Cambridge, MA: Harvard University Press. Willey, K & Gardner, A. (2013). Flipping your classroom without flipping out. Proceedings of the st 41 SEFI Conference, 16-20 September 2013, Leuven, Belgium. Retrieved January 26, 2015 from http://www.academia.edu/5153284/Flipping_your_classroom_without_flipping_out Czaplinski, I. (2015). Investigating the effectiveness of an ecological approach to learning design in a first year mathematics for engineering unit. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:52-FP:64). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:64 76 Community volunteers in collaborative OER development Irwin J. DeVries Open Learning Thompson Rivers University, Canada The purpose of this comparative case study is to explore and examine the practices of open course design and development community volunteers undertaken in the Open Education Resource universitas (OERu) network, an international partnership of member post-secondary institutions. With a focus on the design and development of an OER-based university-level course, the study identifies and describes features of an OERu open design and development volunteer community and compares and contrasts it to a similar community in the free and open source software (FOSS) development field. Keywords: OER, free and open source software, open course design and development, OERu Introduction The purpose of this study is to explore the formation and development of a small community of volunteers who undertook the work of designing and developing an open course in the Open Educational Resource universities (OERu) using an open design and development process. The OERu is an expanding network of over 30 post-secondary institutions and organizations worldwide committed to building OER-based courses and programs, and to providing formal recognition for course completion. Collaborative open course design and development such as that taking place in the OERu is a relatively new phenomenon in higher education. I therefore chose to employ a comparative case study research design (Cresswell, 2013; Stake, 2006) that would enable insights to be gained from a comparison with an open design and development process in a similar field. After an extensive search I located a suitable comparator case in the field of free and open source software (FOSS), where communities of volunteers have for many years collaborated in the open to product open source products. The comparator case study (von Krogh, Paeth & Lakhani, 2003) was similar in many ways in scope, size and structure with the OERu course development project under study. Data were gathered from developer communications, artifacts and developer contribution histories within the OERu’s wiki-based development environment, and from semi-structured interviews with developers. A process of thematic coding and analysis led to the emergence of four themes: ethos and motivation for participating in OERu course development; induction and persistence of volunteers; division of labour; and coordination and communication. Each of these themes is now described, followed by a discussion of findings and conclusion. Motivation and ethos What motivates volunteers to engage in the difficult work of open design and development? Developers interviewed were all highly educated and experienced educators, with busy careers outside their volunteer work in the OERu. In both open design and development and free and open source software (FOSS), developers expressed strong motivation to participate. All OERu volunteers interviewed shared freely their strong personal philosophies concerning reducing barriers to education and credentials, and support for the growth of open educational resources and practices. They saw benefits to their and their institutions’ participation in open design and development projects, particularly where their institutions viewed such engagements as potential catalysts for innovation and transformation. Those in FOSS also wanted to make a contribution to the public good as well as gain skills and participate in the development of software that might be of use to them personally or organizationally as well (Choi & Pruett, 2015; Baytiyeh & Pfaffman, 2010; von Hippel & von Krogh, 2003). FP:65 77 The ethos among developers in the FOSS culture was quite similar to open design and development in the OERu in both respects described by Oberg (2003): open processes and philosophies. OER were rooted in an ideology of sharing content in a free cultural works environment, and FOSS similarly was fostered in the ethos of the GNU General Public License (GPL) and other “open” licenses, which then served as the basis for Creative Commons. Developers in OERu unanimously expressed deep commitment to the philosophies of openness and sharing. For example, My passion [is] to share knowledge. I believe education is a fundamental right, and OER is a vehicle to realizing that mission of widening access… This developer wanted to enable “more affordable access to post secondary education” and was attracted to the OERu because of the fact that …it’s open in all material respects — in terms of its licensing and in terms of its philosophy, in terms of the mission of what the OERu is trying to achieve. All knowledge should be free. It’s part of being, and my philosophy is knowledge is there to be shared. All participants expressed similar commitments to a philosophy of sharing educational resources and opportunities that they reported affirming at a deep personal level. In the words of another developer, Well, I am just a big proponent for the philosophy of open. I just think education is meant to be shared … it makes no sense to me that someone would create something that is useful for students learning and then you put it away, lock it away in your own desktop or, I just can’t compute that. So, I have my own philosophy, all my years, the minute I find something that looks interesting, whether it’s an article, whether it’s a media piece, I immediately take the time to find out who might find it useful. So I totally 100% believe in open. Sharing knowledge, sharing and reaching out … not just to give but to have that community where you can collaborate, where you can ask of the people for help. And in the words of another developer, I was never hiding whatever resources or things I’ve developed...It’s not a treasure that I have to hide and lock in my desk. So I guess it is in a way a personal philosophy.... I didn’t need much of persuasion or conviction to say this is a good thing. I kind of knew it is. Similarly in free and open source software (FOSS), many volunteer development communities are formed to contribute to the “greater good” (Baytiyeh & Pfaffman (2010, p. 1348). Other rewards such as participation in a community, social engagement, recognition and identity construction are expressed as motivators by FOSS developers (Fang & Neufeld, 2009), elements also highlighted by OERu developers in their interviews. For example, one of the main reasons for one developer’s joining was stated as his personal commitment not only to professional development as a university faculty member; but also, I have a personal interest in all open initiatives because personally I’m very committed to bringing education to developing countries, bringing education to those who need it. In a somewhat similar vein, as reported by Dahlander and Wallin (2006), some developers in FOSS also participate as salaried employees “volunteered” by corporations or universities to gain “access and legitimacy” (p. 1256) as well as access to the code. This was also the case with some developers whose time was donated to the OERu by their institution, which saw a strategic advantage in making such a contribution. FP:66 78 Induction and persistence How are volunteers introduced to their project and its community, and how does their participation persist over time? Responding to an open invitation sent to the open OERu email list, a large number of volunteers initially signed up to contribute their time and expertise to the OERu project. This number declined to a smaller fraction who provided substantial contributions or even comments and feedback in the course over time. For instance, 148 virtual participants signed up to participate in initial planning discussions at an early OERu meeting in 2011 in Otego, New Zealand. In the first few weeks following a little more than 30 actually signed up to continue to volunteer to work on the project, and 24 made contributions to the wiki. In the first stage of the project, approximately one third of this number was devoted to developing two courses to completion, and not all of them were original members of the volunteers who originally signed up. A core of these course developers was designated by their institutions to work on their respective courses. Similarly, the Freenet study (Krogh, Spaeth & Lakhani, 2003) found that only four developers contributed 53% of the accepted versions of code in that project. In comparison, in the OERu course, three developers contributed an estimated 95% of the content additions and revisions in the course; in both cases a small number of developers was doing a large proportion of work needed to complete course design and development. In the Freenet case study success in the FOSS community of volunteers, typical of FOSS development more widely, was found to be related to growth in size of the community of developers, “people who contribute to the public good of open source software by writing software code for the project” (Krogh, Spaeth & Lakhani, 2003, p. 1217). Joining behaviours of coders was a major part of the focus of the Freenet study, where it was found that there was a large discrepancy between those who announced initial interest in participating compared with those who ended up making meaningful contributions. “Joining behaviour” was defined as the pathways or “scripts” that volunteer coders would follow, from initial lurking on the project email list to making useful code contributions. One initial barrier to full participation was the difficulty of the Java programming language that was used in coding the project. Also in the OERu, there was a need to learn the wiki mark-up language and conventions as documented in shared artifacts in order to work effectively in design and development. Seemingly obvious indicators of early interest from volunteers in FOSS, such as expressing an interest to contribute, making suggestions for improvements, proposing solutions but with no actual code contributions, asking for a task to work on, engaging in philosophical discussions and such activities did not typically indicate a progression to subsequent code contributions. On the other hand, those who offered contributions of code to fix bugs, engaged in general technical discussions, and offered repeatedly to contribute, along with other such activities tended to go on to become active code contributors. Further, the match between their specialization and the work needed was an important element in joining: An important element of the feature gift giving was that the cost of creating and giving the gift was relatively low to the newcomers. Our interviews with the developers revealed that those that had contributed feature gifts did so on the basis of prior knowledge and experience they had refined in other circumstances (Krogh, Spaeth & Lakhani, 2003, p. 1234). In the setting of the OERu it became evident that more developers with a wider array of skills would be necessary to increase the pace and number of courses developed. One developer observed, It’s a pilot project of how open is going to work.... we definitely have to open it up to many, many, many more people. That to me is how open is supposed to work. I should have been able to immediately feel that I could ask a fellow ID a question, or ask a production person a question, you know when I was stuck with all those questions. There was a later perception by an OEru developer who was initially involved that the primary role given to partner institutions in the OERu overshadowed other developers’ individual FP:67 79 interests. For instance, I was a very enthusiastic WikiEducator, but lost my way when the OER university initiative began as it opened doors for universities, but closed doors for me as an independent educator. I will be lurking if that’s acceptable as I don’t represent a university. While there was no overt restriction on participation by the wider body of those who were volunteers in other parts of WikiEducator, there was also not a notable effort on the part of the community to aggressively recruit those who had initially expressed interest as the focus did indeed fall mainly upon the partner institutions to develop their courses. Nevertheless there were also many communications and invitations to the wider community to comment and provide feedback on developments. In both OERu and FOSS, a high degree of involvement by volunteers is seen as important to the quality and quantity of contributions (Xu, Jones & Shao, 2009). In the Freenet study (Krogh, Spaeth & Lakhani, 2003), because growth of numbers increased with participation, there was interest in the perceived benefits that would draw newcomers to the project. Within the OERu wiki, participation of developers showed a small number (three) who were involved at the very outset in terms of producing actual page edits or comments and remaining similarly involved through the initial OERu planning stage, through the planning and completion stages of the course, indicating a relatively low level of continuity or contributors across the project, constituting only 11% of the initial group of contributors. This finding is not necessarily unexpected, as many initial contributors may understandably have had an interest only in the bigger OERu picture. However, it does reinforce the concern expressed by OERu collaborators that the lack of continuity from end to end made it difficult for later developers to complete the project with a sound understanding of original intentions of early developers. Prior to and alongside the development of OERu courses, overall planning for the OERu was documented in the wiki. A small number of contributors made the largest number of contributions, and one contributor in particular documented most of the discussions and emails in the wiki (Figure 1). A spike in contributions took place early in the project and diminished after that time. The patterns of persistence that emerged in the analysis were of particular interest. They showed both the patterns of continuity of contributors throughout various stages of the project, and the relative amounts of work provided by each. In both cases the patterns provide clues to some of the challenges faced by developers involved in the project. FP:68 80 Figure 1: Contributions made in general OERu planning stage It is helpful at this point to look to another field of collaborative design, architecture. In collaborative design in architecture, developers working together on a design do not typically engage in an ongoing process of negotiation but rather in “…parallel expert actions, each of short duration, bracketed by joint activity of negotiation and evaluation” (Kvan, 2000, p. 412). Similarly, in the OERu course, the most progress in collaboration occurred in occasional conference calls where issues would be settled and tasks negotiated. Developers entering the process later in a project would not have the depth of shared history and understanding as those who had been part of the discussions and negotiations from the very start. They would then need to rely more upon various artifacts in the wiki such as records of previous decisions and notes or revision histories in discussion and history pages. Clearly the process would have benefited from having in place a prescriptive framework for communication roles and strategies among collaborative design teams (e.g., as described by Sonnenwald, 1996), along with effective information retrieval technology. The existence and maintenance of a robust body of volunteers is identified as vital to the ongoing health of an FOSS project, including the growth of established rules and a group culture that fosters commitment and constructive behaviour patterns (Gallego et al., 2015; Hendry, 2008). A difference noted between induction into the OERu and FOSS was described by a developer: … in an open source community if you ask a newbie question and you haven’t even gone through the previous discussion forums, you will be castigated. So in open source there’s this culture of, you go out and read what has been done, and then if you don’t know what’s happening, then you engage with the community. I’ve noticed there’s a lot more tolerance with education folk. However, comparing FOSS development with similar practices in the OER, a developer noted: …the nature of the development [in FOSS] is such that you’ve got objective measures for seniority. You know, if you proved yourself, the code must work and those are the things that it must and this is an objective measure. The developer further noted that educational development is more forgiving in comparison and FP:69 81 thus any challenges that might be faced by late-joining developers would not necessarily be immediately evident, given in particular that there was, by consensus, no common pedagogical approach to learning design. In traditional instructional design, typically all participants in the project are either involved in the project from the very beginning, or if brought in later then are thoroughly debriefed on the project’s history and status. Collaboration in planning is essential to the success of collaborative development teams (Hixon, 2008) and ongoing communication throughout the process is equally important, along with orientation for all participants to the processes and tools used in the development project (Chiu, 2002). However, a developer in this OERu case was left feeling disadvantaged from the outset: … the next person down the road might want to do something with the course but they don’t have all the same philosophy and all the same agreements that [others] had in the beginning. You know, all those conversations … on why you were doing what you were doing in the way you were doing it. How do we share that with the rest of the world? So I know the lessons are there in this pilot project but it’s there in a messy, messy way. We kind of got it in the way of just documenting the process that you would have to clean up because not everybody wants to read through every messy meeting we had. At the end, a different kind of help guide has to come out for the open public …. A really well put together manual would be something useful for the future folks after we’ve learned all our lessons. It should be a little more well organized and concise for the people who come after us. Interestingly, documentation had been developed in the wiki that could have been used by developers, but they were confused by the complexity of the wiki and its flat file structure. Over time another developer pulled these documents together more tightly in one section. To address the challenge for “newbies” beginning later in the project, a starting point for them would then be, it was suggested in the planning node, a place where some work had already been conducted. The expectation would be to make contributions and even improve others’ content, while remaining consistent with the overall direction of the course design. Within this context, however, it was important to have opportunities for developers to gain an understanding of what design thinking had preceded them beyond what was evident in the designed content artifacts or other forms of distributed intelligence. As noted by one developer, there was a need to be able to provide background and context for others just beginning on the course at a later stage. The main way for doing this, apart from abstracting the design from the in-progress artifacts of content and activities, was to review design debates and decisions occurring through and across the OERu wiki and email discussions, and comments provided by developers on talk pages in the relevant section of the course under development. However, this would take a good understanding of the wiki structure and the layout of the OERu, which is complex to a newcomer and takes time to learn. Beyond these elements, a critical factor in working within the open design and development that did not appear prominently in the Freenet study or in FOSS literature in general was mentoring. Throughout the OERu project the more experienced developers were available to provide support and assistance to the newer participants in development. This was seen by several developers as vital to its success. In the experience of one developer, [Originally] I didn’t even have my own WikiEducator page. [A mentor] kind of talked me through how to set up my page, how to bring the images in. She was an email away. She was very, very willing to help. So that made me feel good. [It] was really important because I would have given up and not taken part in the project after week 1. Week 2, if [mentors] weren’t there to help me in that first steep learning curve, then after … just an email away. Very important because as I said the whole project was difficult for me. If [a mentor wasn’t] 11 o’clock also online and answering my questions, I think I would … not [be] doing this. Another viewed membership as a distinctive element that defined open design and development models, based on two key principles of meritocracy and consideration for others in FP:70 82 such acts as mentorship: One is the principle of meritocracy, where one’s seniority — in inverted commas — or respect within a community is actually developed by the expertise you’ve demonstrated within a community and have built up over the years. So there is this key element of meritocracy. You know is it sitting in these open communities, which is a differentiator. I think it’s part of this sort of reward mechanism that’s kudos that takes place in these open communities. So I think that is incredibly important. [Second is] the principle of paying forward. And that helps fuel this ecosystem of mentorship. It’s this whole notion of...someone helped me when I was struggling. Once I’ve acquired the skill it’s now my turn to help somebody else. A further challenge encountered was the effort involved in locating, converting, remixing and formatting the content of the original OER into the wiki. Access to a mentor in the form of a highly experienced WikiEducator developer was seen as a crucial support to the developer. This loomed large in the mind of some developers. Thus for those who had not started from the beginning, and hadn’t arrived with prior appropriate specializations or training, there was a significant barrier to joining. At the same time, by joining at the periphery and learning and being mentored, in the manner of a community of practice (Wenger, 1999), a developer who completed a project found it a substantial learning experience and a good basis from which to move forward with many lessons learned, even as part of a larger philosophy about learning: … it’s been a learning experience and I’m looking at everything really that I do as a learning experience because learning is life and life is learning. I’m not sure who said that but that’s definitely my point of view. So it’s been a great learning experience and I’m continuing to learn and If I’m passionate about others and education, I’ve got to be committed to keep learning. While principles of self organization are largely intended to drive the design and development processes in the OERu, the demands of the environment, the potential challenges with conversion of OERs and the need for various levels and types of expertise appear to suggest the potential advantages of some initial recruitment and negotiation of roles among volunteers and the wider community rather than a more informal processes. In the Freenet study it appeared that while there could be potential within a large enough community for a body of developers to flow in and out of projects, but this would not work well in a startup setting. Division of Labor A vital component in the success of the community in the Freenet study (Krogh, Spaeth & Lakhani, 2003) was identified as specialization of volunteers, i.e., deployment of volunteer talent according to their specialization for “efficient use of knowledge” (p. 1218). In other words, coders were best utilized by working in their areas of greatest expertise, with the implication that a wider variety of types of expertise was required to supply the specific skills needed for particular aspects of the project. With high turnover as found in the Freenet community, this would become even more important, in order to maintain a “critical mass” (p. 1226) of expertise in each of the areas required to complete the project. FOSS projects typically leave it mainly to new volunteers to “work their way in” based on the quantity and quality of their code contributions, and volunteers typically contribute according to their areas of specialization. In the OERu developers with their characteristic instructional design skill set spent much time working well outside their areas of specialization, owing to the fact that few others either were available to take on the various aspects of the course development work and detailed technical implementation, or developers were not aware of them. This was seen as a barrier to overcome as a developer became more acquainted with the new role of learning design in an open wiki environment. For example: I didn’t really plan to be the technology know-how person in the project because that was not my forte. I really was thinking I’d just bring my design expertise and my educational expertise. FP:71 83 The need for developers to venture outside their initial areas of specialization was evident. As described by a developer whose contribution to the project was initially intended to be based on expertise and interest in open education and online learning pedagogy, large amounts of time were spent on such labour intensive work as converting and correcting OER content files, fixing links, tracking down resources, reassembling content from a confusing set of original course files, and so forth. This was described by a developer as “factory work,” and as somewhat distracting from the design goals that were at front of mind in approaching the project: One of the challenges we got in our open design communities, is the extent that our technology people actually engaged in the process. We don’t have a high number of coders or people at that level of technical skill engaging this development process which is kind of odd because if we purporting in sort of open distance learning, professional team approaches, it would be nice to see that sort of skill engaging as well. The lack of sufficient expertise in the technical area was noted by another developer, who felt an inordinate amount of time was spent undertaking repetitive, manual tasks in converting and formatting content when the expertise this individual brought to the project was of a different nature, including design expertise and a particular interest in equity and provision of free learning opportunities to those who are disadvantaged: One of the challenges we have in our open design communities is the extent that our technology people actually engaged in the process. We don’t have a high number of coders or people at that level of technical skill engaging this development process. Yet also there was another OERu developer who didn’t seem to mind applying a mixture of skills to course development: I did find not it too difficult to get used to the wiki mark-up, in particular; it was quite easy, and to be honest I didn’t really follow the tutorials either. But they were useful at the beginning, but I just [applied] the same learning strategy I did when I had to learn HTML… once I got the basic grasp of tags. When I find a good feature I like in the wiki page I just go to the mark-up and copy that, and replace the text or the image with my own. It could be said then that each team will have its unique makeup of skills and interest in performing a broad or narrow array of tasks based on interest, background, time and expertise. Nevertheless, a broader set of skills recruited from the outset will permit more developers to work from their respective strengths and thus avoid unnecessary frustration and discouragement. Coordination and communication Another important factor to be addressed is how coordination and communication occur in the OERu and FOSS environments. In the initial months of the OERu project, the ambitious cross- OERu project management process that was started could not be sustained by developers, as the main developer heading it up moved on to another institution and no others expressed an inclination to continue this role. It did not appear that a comprehensive project management process was feasible for the OERu project, owing to the breadth and complexity of the various course development projects, and the time developers would need to contribute to their own projects let alone step up to take on larger responsibilities. Further, it appeared that quasi- regular synchronous virtual meetings among developers were particularly valuable in discussing challenges, reviewing progress, planning next steps and dividing work. These meetings and the subsequent notes kept by one or multiple participants placed in an appropriate page in the wiki were of ongoing value to developers. In the Freenet study (Krogh, Spaeth & Lakhani, 2003), commitments to code versions were approved by a small group of senior administrators, with increased trust placed in coders who FP:72 84 established a record of high quality contributions. Similarly in the OERu, a meritocracy of developers was seen as a part of an ecosystem where credibility of contributions built up over time would give them increased stature and responsibility in the community. FOSS projects typically display decentralized decision-making and representation, although there are occasions where a formal leadership role or representative body in a not-for-profit foundation is established “to protect the community’s interests” (O’Mahoney, 2007, p. 2). The OERu also is governed by a not-for-profit organization, the Open Education Resource Foundation, with an Executive Director who coordinates the efforts of the OERu and provides much impetus and expertise in moving the OERu community forward. Each of the partner institutions involved in developing OERu courses had a great deal of autonomy as to how the courses were developed, subject to working with the guidelines that had been reached across the partnership by means of polls and rough consensus. Another area for comparison between FOSS and open design and delivery is communication methods. In support of this emphasis, several of those interviewed noted that it would be helpful for the community to review and further organize many valuable but distributed resources across the wiki into a more structured guide to improve sharing of information. Given the nature of developers and the amount of time that they may be involved in a project such as the OERu, this would of course need to be revisited on an ongoing basis, and it would also need to be recognized that no such system would be perfect given the decentralized nature of the community. The practice of maintaining notes on discussion pages both to communicate asynchronously in situ with other developers and to leave a record for others who joined later in the process was viewed as a valuable asset. Development teams would need to become more alert to the importance of maintaining understandings at the outset that as much communication as possible should either occur within the wiki or, if external, documented in the wiki as well. For instance, virtual synchronous meetings would have notes taken and placed in the wiki in a designated page for maintaining meeting records. Also in this area a set of links to the key pages that track ongoing OERu-wide discussions within the wiki on common elements of concern to all developers would need to be maintained in order for those who join projects midstream can quickly be oriented to the essential elements of the project. Discussion The way that volunteer communities function in the OERu and in FOSS settings including the comparator case showed many similarities throughout the study. In terms of motivation, developers in the OERu expressed a very high level of commitment to the underlying principles and ethos of open education and worked beyond usual hours and/or without pay to complete their project, in a manner similar to FOSS developers (Baytiyeh & Pfaffman, 2010; Oberg, 2003). Also, in FOSS, organizations may donate developer time in order to benefit directly or indirectly from the code under development (Dhalander & Wallin, 2006), and in the same way multiple partner institutions sponsored developers to work on the OERu project. Since such arrangements are organizationally encouraged or even required, such work should become part of a regular workload where possible. Successful FOSS projects have relatively well-developed processes for orienting new developers to the communication tools and practices proven to be successful in such environments (Chiu, 2002). This includes not only email lists, discussion boards, wikis and versioning tools, but also system-wide views and visible design rules or artifacts that promote the sharing of knowledge and intelligence. Similar tools and practices were present in the OERu but communication habits of developers tended to spread information across the wiki and in scattered emails in a manner that made it difficult to retrace where key information could be found. Course development teams will benefit from establishing and maintaining clear guidelines for communication and documentation methods. These protocols were well documented in the wiki, and an orientation for new members would be beneficial, along with continuing reminders from more experienced developers. Effective maintenance of FOSS over time improves the quality of the project (Koponen & Hotti, 2005) but requires planning and organization. Above all, new developers who join the project FP:73 85 later in its lifecycle need to be able to gain a sense of the project’s history and organization quickly with the help, for example, of such factors as systematic naming conventions of files and logs (Stewart, Darcy & Daniel, 2005). Developers in the OERu prototype project similarly found it necessary but also difficult to become oriented to the project in a short period of time, which would suggest the need for practices similar to those in FOSS that maintain a system for the support of new joiners in a course development project (Chiu, 2002). As noted by O’Mahoney (2007), “when code and community do not develop in parallel, the learning curve can be steep, which can affect external developers’ ability and motivation to contribute” (2007, p. 142). Recruiting, properly inducting and maintaining a robust community of volunteers have proven to be critical components in the success of FOSS projects. Because there was a high attrition among the initial OERu developer recruits, there were fewer developers and other volunteers involved in completion of prototype courses by the final stages of the prototype course than desirable, increasing stress on the remaining volunteers. In FOSS some attrition occurs because of skills barriers; e.g., a programming language that is out of the skill range of potentially interested contributors (Krogh, Spaeth & Lakhani, 2003). However, volunteers who aren’t meaningfully engaged don’t stay around for a long time in both FOSS and in the OERu (Xu, Jones & Shao, 2009). Successful FOSS projects attract sufficient developers with an appropriate array of skills or specializations to cover off the variety of design and technical needs in a course development project (Krishna Raj & Srinivasa, 2012), and over the longer term bring their experience to the project as mentors or administrators (von Krogh, Spaeth & Lakhani, 2003). The evidence gathered from the OERu wiki and communications emphasize this point. Developers reported that having to take on multiple roles, particularly those that would ordinarily be considered technical in nature such as page design, mark-up and production, diverted their efforts toward focusing on their design strengths. Further, they reported a concern that they had overextended the time they had available to work on the course. While a certain degree of familiarity with the wiki environment is necessary for any wiki developer, engaging in more extensive course development was seen as somewhat onerous. Partner institutions of the OERu could consider an increased effort to recruit both internally and elsewhere a rounded team of developers to complete each course. Collaboration and communication are fundamental to the practice of open design and development. Not only content but also design knowledge need to be shareable in a wider open education ecosystem such as the OERu network and among volunteer development teams. However, research in the sharing not only of content but also of learning designs, design patterns (Alexander, Ishikawa & Silverstein, 1977) or learning design “know-how” (Dalziel, 2008) indicates that translating learning designs from one setting to another is a complex matter. As noted earlier, one pathway for further investigation is the use of visible design rules that guide a high-level view of the design process, while making knowledge of deeper levels of detail unnecessary at certain points (Hossain & Zhu, 2009). These may be further shared and discussed in discussion spaces as has been seen in FOSS development (Björgvinsson & Thorbergsson, 2007). Research into distributed intelligence (Perkins, 1992) as well as mediating artifacts (Conole & Culver, 2009) points to ways in which design knowledge can become more visible and thus shared in a communal work setting where collaboration is centred on representations open for discussion within the community. While an “artifact appears to be a self-contained object, it is in fact a nexus of perspectives” (Zitter et al., 2009), a resource most important in a setting such as the OERu where the community is distributed globally. Mediating artifacts are both available for access by all and able to be negotiated and changed. Mediating artifacts include discourses and processes supporting coordination and negotiation or brokering between different domains within a community of practice (Wenger, 1999). As noted by Dimitriadis et al. (2009), “making design more explicit will facilitate repurposing of the OER” (p. 201). Similarly, Conole et al. (2013) emphasize the importance of social networking spaces where designers can discuss and share ideas on learning designs. Such spaces were in fact available in the planning sections of the OERu wiki. However, because development of learning designs was intended to remain the province of each institution and its developers rather than something shared across the partnership, a robust learning design discussion space did not fully emerge. Rather than become lost in individual exchanges scattered across emails and wiki “talk” pages, a concerted effort to concentrate this discussion could have the potential to create a shared body of knowledge on effective learning designs for FP:74 86 the OERu project or similar open design and development contexts. In the OERu the course prototypes developed for stimulating discussions and negotiations toward consensus exemplified the concept of nexus of perspectives. They perform this function by serving first to generate, and then to record, discussions and decisions in brief summaries, similar to what Scacchi (2007) identifies in open source software projects as “lean descriptions” or “documentary artifacts” (p. 473). Similarly, brief descriptions of decisions may have a similar function and are seen as critical to sharing an understanding of the learning design and other issues faced by the developers. Conclusion As the work of the OERu progresses and the body of developers grows, an increased effort toward sharing of learning designs ideas and experiences may help create a strong community with established practices, tools and shared understanding. New and creative design approaches must grow from the developer body working across the OERu to face the many challenges and opportunities documented in this study. A balance of dynamic design decision- making and intentional collaboration among developers in learning design and related skill areas will help to support such innovation. Along with this work, the community would be wise to observe and learn from the methods used in the many successful free and open source software projects that have emerged over the past decades. References Alexander, C., Ishikawa, S., & Silverstein, M. (1977). A Pattern Language: Towns, Buildings, Construction. Oxford, UK: Oxford University Press. Baytiyeh, H., & Pfaffman, J. (2010). Open source software: A community of altruists. Computers in Human Behavior, 26, 1345–1354. Elsevier Ltd. doi:10.1016/j.chb.2010.04.008. Chiu, M. (2002). An organizational view of design communication in design collaboration. Design Studies, 23, 187–210. Choi, N., & Pruett, J. (2015). The characteristics and motivations of library open source software developers: An empirical study. Library & Information Science Research, 37(2), 109–117. Conole, G., & Culver, J. (2009). Cloudworks: Social networking for learning design. Australasian Journal of Educational Technology, 25(5), 763–782. Creswell, J. (2013). Research design: Qualitative, quantitative, and mixed methods approachs, th 4 ed. Los Angelos: Sage. Dahlander, L., & Wallin, M. (2006). A man on the inside: Unlocking communities as complementary assets. Research Policy, 35(8), 1243–1259. doi:10.1016/j.respol.2006.09.011. Dalziel, J. (2008). Learning design: sharing pedagogical know-how. In T. Liyoshi & V. Kumar (Eds.), Opening up education: The collective advancement of education through open technology, open content, and open knowledge (pp. 375–388). Cambridge, MA: MIT Press. Dijkstra, S. (2001). The design space for solving instructional-design problems. Instructional Science, 29, 275–290. Dimitriadis, Y., McAndrew, P., Conole, G., & Makriyannis, E. (2009). New design approaches to repurposing open educational resources for collaborative learning using mediating artefacts. In Same places, different spaces: Proceedings Ascilite (pp. 200–207). Aukland. Retrieved from http://www.ascilite.org.au/conferences/auckland09/procs/ dimitriadis.pdf Fang, Y., & Neufeld, D. (2009). Understanding Sustained Participation in Open Source Software Projects. Journal of Management Information Systems, 25(4), 9–50. doi:10.2753/MIS0742- 1222250401. Gallego, M., Bueno, S., Racero, F. & Noyes, J. (2015). Open Source Software. Computers in Human Behavior, 66(49), 390-399. Hendry, D. (2008). Public participation in proprietary software development through user roles and discourse. International Journal of Human-Computer Studies, 66, 545–557. doi:10.1016/j.ijhcs.2007.12.002. Hixon, E. (2008). Team-based Online Course Development : A Case Study of Collaboration Models. Online Journal of Distance Learning Administration, 11(4), 1–8. Retrieved from http://www.westga.edu/~distance/ojdla/winter114/hixon114.html. FP:75 87 Hossain, L., & Zhu, D. (2009). Social networks and coordination performance of distributed software development teams. The Journal of High Technology Management Research, 20, 52–61. Elsevier Inc. doi:10.1016/ j.hitech.2009.02.007. Khan, S., & Samuel VanWysberghe. (2008). Cultivating the Under-Mined: Cross-Case Analysis as Knowledge Mobilization. Qualitative Social Research, 9(1). Retrieved from http://www.qualitative-research.net/index.php/ fqs/article/view/334/729. Kirschner, P., van Merriënboer, J., Sloep, P., & Carr, C. (2002). How expert designers design. Performance Improvement Quarterly, 15(4), 86–104. Kvan, T. (2000). Collaborative design: what is it? Automation in Construction, 9(4), 409–415. doi:10.1016/S0926-5805(99)00025-4. Le Maistre, K., & Weston, C. (1996). The priorities established among data sources when instructional designers revise written materials. Educational Technology Research and Development, 44(1), 61–70. Oberg, S. (2003). Bits and Bytes: Serials System Insights: Open Source Software: An Introduction from a Serialist’s Perspective. Serials Review, 29, 36–39. Retrieved from linkinghub.elsevier.com/retrieve/pii/ S0098791303000029. O’Mahony, S. (2007). The governance of open source initiatives: What does it mean to be community managed? Journal of Management & Governance, 11, 139–150. Scacchi, W. (2007). Understanding Requirements for Open Source Software. In K. Lyytinen, P. Loucopoulos, J. Mylopoulos, & B. Robinson (Eds.), Design Requirements Engineering: A Ten-Year Perspective (pp. 467–494). Cleveland, OH: Springer. Sonnenwald, D. (1996). Communication roles that support collaboration during the design process. Design Studies, 17(3), 277–301. Stake, R. E. (2006). Multiple case study analysis. New York: The Guilford Press. von Hippel, E., & von Krogh, G. (2003). Open Source Software and the “Private-Collective” Innovation Model: Issues for Organization Science. Organization Science, 14(2), 209–223. von Krogh, G., Spaeth, S., & Lakhani, K. R. (2003). Community, joining, and specialization in open source software innovation: a case study. Research Policy, 32, 1217–1241. doi:10.1016/S0048-7333(03)00050-7. Wenger, E. (1999). Communities of practice: Learning, meaning, and identity. Cambridge: Cambridge University Press. Xu, B., Jones, D., & Shao, B. (2009). Volunteers’ involvement in online community based software development. Information & Management, 46(3), 151–158. doi:10.1016/j.im.2008.12.005. Yin, R. (2009). Case study research (4th ed.). Thousand Oaks, CA: SAGE Publications, Inc. Zitter, I., Kinkhorst, G., Simons, R., & ten Cate, O. (2009). In search of common ground: A task conceptualization to facilitate the design of (e)learning environments with design patterns. Computers in Human Behavior, 25(5), 999–1009. Elsevier Ltd. doi:10.1016/j.chb.2009.01.001. DeVries, I.J. (2015). Community volunteers in collaborative OER development. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP: 65-FP:76). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:76 88 A ‘participant first’ approach to designing for collaborative group work in MOOCs Kulari Lokuge Dona Janet Gregory Swinburne University of Technology Swinburne University of Technology This paper discusses the learning design of two Massive Open Online Courses (MOOCs), the Carpe Diem MOOC and the Autism MOOC, both of which were designed and delivered by Swinburne University of Technology in Melbourne, Australia. The authors propose a set of principles to guide the design and development of MOOCs where the intent is to facilitate interaction and peer support between participants. They present details of how these principles were enacted in the design of the Carpe Diem MOOC and the Autism MOOC, particularly in the design of groups, and suggest that these principles can be viewed as a ‘participant first’ approach to design. Key elements of this approach include accessibility, navigation, clarity and consistency, purposeful use of tools and resources and effective support to enable participants to engage easily in collaborative work in MOOC environments. Keywords: Massive Open Online Course, MOOC, learning design, Carpe Diem, design principles, online learning, MOOC design MOOCs and learning design approaches Collaborative work and learning in groups is not a new phenomenon in educational institutions (Juwah, 2006), with the importance of collaborative learning well recognised for its ability to lead to higher levels of learning if managed effectively (Frey, Fisher, & Everlove, 2009). However, Khosa and Volet (2013) suggest that whilst there are benefits to collaborative learning, students may need “instruction in the use of learning-enhancing strategies” (p. 871) in order to benefit from the opportunities afforded by collaboration. This is particularly interesting given that group work and collaboration are relatively new phenomena in online courses (Brindley, Walti, & Blaschke, 2009), particularly in Massive Open Online Courses (MOOCs). This paper discusses one approach to the design principles applied to group interaction in two different MOOCs, and argues that a clear set of design principles are needed to enable groups to work effectively in the MOOC environment. The term MOOC was coined by Dave Cormier in 2008 to describe a course – Connectivism and Connective Knowledge – which was offered free to the public, as well as to fee paying university students, and attracted 2,300 participants (Yuan & Powell, 2013). The principle behind MOOCs aligns with concepts of universal access and openness in education as anyone can participate and there is no cost. MOOCs are frequently referred to as a “disruptive force” in higher education (Bates, 2013; Shirky, 2012) as they not only present potentially new business models (Yuan, Powell, & Bill, 2014) but they “disrupt the notion that learning should be controlled by educators and educational institutions …” (Kop, Fournier, & Mak, 2011, p. 75). Their openness can lead to massive enrolments, but there is also a tendency for high drop-out rates. The majority of MOOCs achieve completion rates of up to 13%, with only a few achieving more than 40% (Jordan, 2015), raising interesting questions about how to design for collaboration when numbers of participants are unknown and continuously reducing throughout the course. Consequently, many of the assumptions held about the design for courses in higher education may require rethinking to be transferable into this new context. As Kop et al. (2011) suggest, “a change in the thinking, philosophy, design, and pedagogies of institution-based online courses may be necessary if the affordances of emerging technologies are embraced and adopted within formal educational institutions” (p. 89). Weller (2011) suggests that we now need to design for a “pedagogy of abundance”. He argues that the traditional university model is predicated on the idea of a scarcity of experts, resources and facilities. In a digital, networked environment however, we have access to content as well FP:77 89 as access to peers, experts and other learners, and the opportunity to discuss ideas through social networks (Weller, 2011). Weller presents a number of pedagogical approaches that are better equipped for abundance, including resource-based learning, problem-based learning, constructivism, communities of practice, and connectivism. The “pedagogy of abundance” concept fits well within the MOOC model, and has significant implications for the design of MOOC activities that enable social networks to flourish. Yuan and Powell (2013) note that MOOCs have developed in two distinctly different pedagogical directions based on different ideologies. xMOOCs are designed as online versions of traditional higher education learning and teaching formats using Learning Management Systems such as edX, Udacity, Coursera, OpenEducation and FutureLearn. cMOOCs are based on connectivist theory, espoused by George Siemens and Stephen Downes (Milligan, 2013), and tend to run on open source learning platforms with a pedagogical model of peer learning. Yuan and Powell (2013) argue that: cMOOCs emphasise connected, collaborative learning and the courses are built around a group of like-minded ‘individuals’ who are relatively free from institutional constraints. cMOOCs provide a platform to explore new pedagogies beyond traditional classroom settings and, as such, tend to exist on the radical fringe of HE. On the other hand, the instructional model (xMOOCs) is essentially an extension of the pedagogical models practised within the institutions themselves, which is arguably dominated by the “drill and grill” instructional methods with video presentations, short quizzes and testing (p. 7). Gillani (2014) notes that, irrespective of the type of MOOC, participants are able to interact and collaborate in online discussion forums. However, as MOOCs are open and free, participants will come from a wide range of backgrounds, experience and skill levels (Milligan, 2013), and the challenge is to create a pedagogy and design that accommodates this diversity and enables learning through social connections (Kop et al., 2011). In addition to diversity of background, experience and skills, there are different levels of interaction to be accommodated. Hill (2013) identifies four types of MOOC participants: Lurkers, who enrol but only observe; Drop-Ins, who partially participate; Passive Participants, who view and use course content but do not participate in activities; and Active Participants, who actively participate in activities. Interaction also tends to change over the life-time of the MOOC, with a risk of early information overload as discussion forums are overloaded with small-talk, followed by the sharp decline rate as participants drop-out (Brinton, 2014). Critical literacy skills emerge as one of the key areas needed to learn effectively in connectivist environments. Specifically, Kop (2011) argues that to learn effectively in these environments, participants need to have an open mind, be able to learn cooperatively, have critical analysis skills, and be confident and competent in the use of the tools available to enable learning. (Milligan, 2013). Those with the critical and digital skills are more likely to become the active participants, thereby providing the group with “a high set of resources available in the form of people with varied experiences and expertise” (Gillani et al., 2014, p. 2). However, large groups with high attrition reduces the likelihood that participants will form strong relationships, raising the question of whether smaller groups can be more effective in engaging participants in MOOCs. Gillani (2014) highlights the importance of designing for group interaction, stating: While theoretical perspectives and emphases differ in studies of online learning, it is recognised that understanding the learning process in online forums requires consideration of interactions at the individual and group level. The interactions at the group level within these forums can be viewed as a kind of scaffold through which learning can occur, and therefore, is of significant practical concern when considering the future design and development of courses (p. 1). A number of authors have written extensively about design for online learning, and have developed approaches to encourage interaction and learning through collaboration. Laurillard’s Conversational framework supports the establishment of collaborative learning environments for groups of learners to participate in conversations (Hickey, 2014), and emphasises tutor-student dialogue and actions based on dialogue and reflection (Laurillard, 2012). The framework offers FP:78 90 five ways in which learning resources can be designed and used – as narrative, interactive, adaptive, communicative and productive. The scaffolded learning model, or 5 stage model (Salmon, 2002, 2011), and the structure of online activities or e-tivities (Salmon, 2002, 2013), are designed to encourage and enable collaborative learning (Salmon, Gregory, Lokuge-Dona & Ross 2015) in online environments. Tom (2015) discusses how the use of technology to enhance learning and teaching depends on effective design of the resources. Tom (2015) integrates constructivist and collaborative learning theories in establishing the Five C framework for student centred learning: Consistency – in learning and teaching practices; Collaboration – in problem solving and knowledge construction; Cognition – developing higher order thinking; Conception – understanding concepts; and Creativity – creating solutions by applying concepts learnt. Design principles applicable to learning and teaching online emerge from a variety of discipline areas, including multimedia. For example, Mayer (2001, 2005, 2009) highlights how the cognitive theory of multimedia learning provides ideas for designing online learning resources and environments. Mayer (2009) describes learning as a sense-making process where students build understandings based on coherent representations from the presented learning resources that consists of text, images and audio. He highlights three types of cognitive processing during learning – Extraneous, Essential and Generative – and discusses how learning can be maximised by reducing non-related instructions, presenting essential material in a simple manner to reduce complexities, and creating engaging activities to foster generative processing. Churchill (2011) then presents a number of key principles related to multimedia design that offer key points for consideration in online learning design. These principles can be summarised as follows: • Multimedia – the use of both visual and verbal information • Managing essential processing through segmenting (student paced segments); pre- training (key concepts need to be familiar); and modality (words are spoken rather than written) • Reducing extraneous processing through coherence (excluding extraneous material); signalling (highlighting the organisation of essential material); redundancy (no repeating of material): spatial contiguity (words and pictures are physically integrated): and temporal contiguity (words and pictures are temporally integrated) • Social cues including personalisation (words presented in conversational style); voice (narration in human voice); and image (no need for speaker’s image on screen). What is clear is that the online learning environment, particularly MOOCs, requires new ways of thinking about how we design and deliver learning activities. As Kop et al. (2011) state: The type of support structure that would engage learners in critical learning on an open network should be based on the creation of a place or community where people feel comfortable, trusted, and valued, and where people can access and interact with resources and each other. The new roles that the teacher as facilitator needs to adopt in networked learning environments include aggregating, curating, amplifying, modelling, and persistently being present in coaching or mentoring (pp. 88-89). Designing for MOOCs is a complex task if the variation in participation levels, intentions, capabilities and expectations within any given cohort of participants is to be effectively addressed. A key question is how to design to accommodate the diversity of participants, enabling those who want to actively participate, whilst also providing resources for those who want to observe and learn. In addition, how can the design cater for participants who do not have the critical or digital literacies required to successfully navigate MOOCs, and draw on the learning from related fields such as multimedia to create consistent and coherent experiences for participants. We argue that a ‘participant first’ approach can increase the likelihood of more participants developing the required literacies and potentially therefore feeling more able to actively contribute, and we demonstrate how we attempted to apply this approach in two MOOCs with very different groups of participants. FP:79 91 The Swinburne MOOCs Swinburne University of Technology recently designed and delivered two interactive MOOCs: the Carpe Diem MOOC (CD MOOC) in 2014, and the Autism MOOC in 2015. The CD MOOC, based on the work of Gilly Salmon (2011, 2013), was designed to offer educators the opportunity to learn about the Carpe Diem learning design process through relevant, authentic and experiential academic development (Salmon, Gregory, Lokuge-Dona, & Ross, 2015). The CD MOOC was designed to enable participants to work in groups to learn about, and apply, the Carpe Diem learning design process. The Autism MOOC was designed for a different audience, aiming primarily for participants who are carers and supporters of people with Autism Spectrum Disorder, while it also included some participants diagnosed with Autism Spectrum Disorder. The Autism MOOC was designed to engage participants by offering resources and activities in which participants could share experiences and support each other. Participants in both the CD MOOC and the Autism MOOC were allocated to groups in which they would interact. In the CD MOOC, with enrolments of 1,426, participants were randomly allocated into groups with 30 members. Each group had its own area in the discussion forum in Blackboard Coursesites, and was allocated one facilitator whose role was to provide pedagogical support and enable discussions (Salmon, Gregory, Lokuge Dona, & Ross, 2015; Lokuge, Salmon, Gregory, & Pechenkina, 2014). The Autism MOOC was designed for a bigger cohort, with 15,596 registering for the course and 11,297 actually commencing. The Autism MOOC was set up so that the participants allocated themselves to a group with its own discussion forum, with each group designed to accept a maximum of 300 members. The Autism MOOC also allocated group moderators, however their role was not designed to be as active as the CD MOOC facilitators, but was primarily focussed on ensuring there were no problems in any of the discussion forums. The design for each MOOC focussed on engagement, and established structures and activities to enable high levels of interaction among participants in order to foster support and collaboration. The structure of each MOOC was designed around a key principle relevant to the topic. The CD MOOC structure built on concepts of scaffolded learning (Salmon, 2011) and activities designed for interaction (Salmon, 2002, 2013). Learnings from the CD MOOC were applied to the design of the Autism MOOC, and the concepts of scaffolding and interactive activities were also aligned with the Autism MOOC’s focus on a “person first” (Tobin, 2011) approach to supporting people with Autism Spectrum Disorder. The experience of designing with the “person first” model in mind highlighted the need to be explicit about how we design for all users, and the importance of thinking of the participant first when designing and delivering MOOCs. ‘Participant first’ design principles The ‘participant first’ approach discussed in this paper considers design from the perspective of the participant, and highlights the key design principles for engaging participants and enabling them to work effectively with others to gain the most from their MOOC experience. The ‘participant first’ design principles draw on the existing knowledge within many disciplines, including multimedia (Mayer, 2009; Churchill, 2011), education (Conradie, 2014), and online learning (Brindley, Walti, & Blaschke, 2009). The initial design question for both MOOCs considered what the participants were likely wanting to get out of the MOOC. We considered that participant expectations would include access to resources, opportunity for interactions with others interested in the topic, establishing connections with like-minded people, and exploration of issues and ideas. As designers, we hoped to accommodate different needs and expectations as much as possible. For example, in the CD MOOC we expected participants would want to learn about the Carpe Diem learning design process, and how to use it in practice. As a result, we provided resources, examples, tools and techniques, and opportunities to use these within the CD MOOC, to experience the learning design process as well as discuss it with others. Table 1: The proposed ‘participant first’ design principles for interactive MOOCs FP:80 92 Participant Consider your target participant group – a difficult task in Perspective MOOCs as participants can be very diverse. Consider how to introduce people to each other, the online environment and the material Accessibility Consider issues such as technical requirements and knowledge, technical assistance access for participants with disabilities, accessible language rather than technical jargon, etc Resources Consider types and availability of resources, and if they are easy to access, engaging, relevant and if they going to be openly available to people outside the course Task Value and Consider value and clarity of task if participants are asked to Clarity do something Information and Consider appropriateness, relevance and amount of Support information provided and the level of support provided Consistency Consider consistency of design, language, navigation Interaction Consider what level of interaction is desired in the groups, and what structures/activities/tools are in place to encourage interaction Purpose Consider clearly articulated purpose for the overall MOOC and for the component parts/activities Acknowledgement Consider how to provide acknowledgement of participant involvement Navigation Consider ease of navigation, including sign posting for resources and activities Tools Consider which tools will work best to enhance interaction, including discussion forums, social media tools, etc. Participant perspective As in any design process it is imperative to consider the intended user. We were designing for different participants for the two Swinburne MOOCs – the CD MOOC was aimed at educators interested in learning design, and the Autism MOOC was aimed at carers and supporters of people diagnosed with Autism Spectrum Disorder. Whilst many of the design principles discussed apply to both, we did assume that most educators would have some experience of Learning Management Systems and be confident in working in the MOOC environment. We did not assume any level of technical experience for the participants in the Autism MOOC, so we developed additional resources to assist in navigation and understanding requirements. In both MOOCs, we wanted to establish a sense of community and trust early on, so the first activities were designed in line with the 5 stage model (Salmon, 2011) to provide a comfortable forum in which participants could get to know each other, and explore the learning environment, before focussing on the key content material. The completion rates for both MOOCs were 23 to 24%, compared with a common MOOC completion rate of 10 to 13%. Nevertheless, whilst our completion rates were higher than many MOOCs, it was still a significant drop out rate. Table 2: MOOC engagement summary Autism MOOC engagement summary CD MOOC MOOC Number of registrants 1,426 15,670 Registrants who started the course 71.6% 72.0% FP:81 93 Participants accessing MOOC in the 23 % 24 % last week of the course Accessibility The CD MOOC and Autism MOOC environments were designed to enable any participants unfamiliar with online learning, and/or with any difficulties in using the technology, to find it accessible. We referred to the World Wide Web Consortium (W3C) accessibility guidelines (http://www.w3.org/standards/) and also conformed to Swinburne web style guides. For example, we developed a short video resource that explained how to best access all the resources in the MOOC; we used simple, everyday language and avoided technical and educational jargon; and we created a range of resources to cater for different learning styles, including videos, audio and print materials. All print material was made available as word documents to enable higher accessibility. We also created transcripts for all videos and captions for the Autism MOOC videos. In designing the content pages we ensured plenty of white space and visuals to break the page and make it more appealing to read. In the CD MOOC, where we conducted synchronous webinars, we considered the impact of geography, as we had participants from around the world. Consequently, we ran the synchronous sessions twice a week in two different time zones, as well as providing recordings of all sessions for those who could not attend. Resources Yuan and Powell (2013) suggest that most participants who join MOOCs look for resources, therefore, providing resources that can be easily accessed and that present relevant information is particularly important. In the CD MOOC, all resources (videos, booklets, guidelines) were offered as Open Educational Resources (OERs) and could be downloaded and re-used by participants. The Autism MOOC resources were made available as OERs through Swinburne Commons at the conclusion of the MOOC. Video resources appear to be particularly popular as evidenced by the number of views of videos in both MOOCs. The CD MOOC had a weekly video to introduce each week’s topic, and the Autism MOOC had an introductory video each week, including the Orientation Week (Week 0), and approximately two to three videos presenting additional information and ideas. Table 3: Video resource usage: CD and Autism MOOCs CD MOOC Autism MOOC Resources Viewed Downloaded Viewed Downloaded Week 0 27,908 236 Week 1 1,217 31 29,345 622 Week 2 2,225 108 15,022 388 Week 3 1,204 36 11,031 329 Week 4 513 21 7,511 249 Week 5 244 11 5,309 163 Week 6 220 22 2,934 93 Additional 1,552 65 5,841 446 videos Total views 7,175 294 104,901 2,526 Task Value and Clarity The activities within both the CD MOOC and the Autism MOOC were designed to provide opportunities for social interaction, recognising the value of discussion focussed on real life issues (Marra, Jonassen, Palmer, & Luft, 2014), and to motivate participants to assist each other to solve the issues raised. The MOOCs aimed to engage participants by providing resources along with opportunities to share experiences and develop knowledge and skills. A FP:82 94 key focus was on designing tasks that were clear and engaging to encourage people to participate and thereby set up the condition for valuable interaction – a core element of an interactive MOOC. With both MOOCs, we based the design of tasks on the e-tivity structure (Salmon, 2002, 2013) to make the tasks as clear as possible, and facilitate interaction and discussion to enhance the value of the task for participants. Information and Support The coherence effect suggested by Mayer (2009) suggests that participants learn more deeply when extraneous material is excluded rather than included, so only necessary information should be presented. In designing our MOOCs, we focussed closely on the specific information required for participants to learn about the topic. Within the CD MOOC, information and links to resources were normally contained with the structure of the group activities, and were specific to the purpose of that activity, with the exception of introductory videos. We developed a different structure for the Autism MOOC, where the resources were provided separately to the activities, as they were not specifically linked to the activity tasks and therefore could be read and/or viewed separately. The Autism MOOC structure did add an additional step in navigation, however, as it enabled participants to easily re-visit the resources at any time. For example, in the Autism MOOC there was an introductory video for each week, as well as videos of people talking about their experience and/or strategies, and these could be viewed before moving to the activities. We provided several support mechanisms for MOOC participants, specifically a generic email address for enquiries and support that was open throughout the MOOC, including weekends; an FAQ section with help guides and answers to commonly known issues; and help discussion forums that were monitored by technologists to support MOOC participants with technical issues. We found it particularly important to provide support to participants in the first two weeks of the course whilst they became used to the MOOC environment and learnt how to navigate the MOOC Learning Management System effectively. Consistency A consistent “look and feel”, and particularly consistency of language, was an important aspect of our design as we wanted to establish an environment that participants could easily navigate. This consistency means that as participants progress through the course, they become comfortable in that environment, knowing what they can expect in terms of structure, navigation, tools and language (Churchill, 2011), thereby leaving them free to focus on content and participation (Mayer, 2009). Consistency was also built into the design of the MOOCs by sequencing content with clear sign posts and symbols. For example, in the Autism MOOC we used jigsaw pieces to represent each week and demonstrate progress through the MOOC, and in the CD MOOC we used the e-tivity structure to provide a consistent layout for the activities and location of resources. Consistency of language is particularly important, and our experience demonstrated the importance of checking carefully to ensure that language and instructions presented in one week were aligned and replicated in later weeks to avoid confusion. Interaction The CD MOOC and the Autism MOOC were both designed with interaction in mind. We established a group structure with group sizes of up to 30 members in the CD MOOC, and up to 300 in the Autism MOOC. The activities within the groups were designed to encourage social learning (Conradie, 2014) and allow participants to provide support to each other and assist with solving issues or developing knowledge. The CD MOOC was designed for participants to discuss tasks in their small groups, as well as providing a community area in which they could interact with all members of the MOOC. This appeared to work effectively as participants worked on tasks within their small groups, but also accessed the larger group. It was particularly beneficial in the case where small groups had high attrition rates and became too small, as the remaining members could interact with the broader MOOC community. The Autism MOOC groups were designed to be much larger (up to 300) due to the higher enrolment numbers. Despite anticipated attrition rates, the groups of 300 FP:83 95 were expected to remain large enough to provide participates with a large community to interact with. Given this, a decision was made that an additional MOOC community forum was not needed. One of the difficulties of these large groups was the number of posts in the first two weeks. It possible that some participants may have withdrawn due to difficulty in navigating so many posts. The ideal group size within a MOOC is still unknown, mainly due to dynamic participation and enrolment/withdrawal patterns. The types of MOOC participants mentioned by Hill (2013) make identifying a suitable number of members for groups even more complex. It is interesting to note that there were more posts in the smaller groups in the CD MOOC, raising the question of whether the smaller groups encouraged greater interaction or whether there were differences in the type of participant. Table 4 shows of the number of discussion posts in the CD MOOC and the Autism MOOC. Table 4: Number of Discussion Posts: CD and Autism MOOCs Discussion Number of Average posts MOOC Name posts participants per participant CD MOOC 10,791 1,029 10.4 Autism MOOC 42,011 12,467 3.4 Purpose As with any learning experience, clarity of purpose and learning activities are important in MOOCs. For the CD MOOC and the Autism MOOC, their overall purpose of the MOOC was decided in advance and clearly stated to potential participants. The purpose of each week – the stages in the Carpe Diem learning design process and the steps in the “person first” approach to Autism – was clearly written with details of the aims of the week. The activities, again based on the e-tivity structure (Salmon, 2002, 2013), also had a clearly stated purpose for each activity so that participants understood the value of the tasks. Acknowledgement and reinforcement An interesting finding in the research conducted on the CD MOOC was the expectation by participants that the MOOC facilitators would be actively involved (Salmon et al., 2015), thereby highlighting the value of acknowledgement and recognition of participation. This is not easy in a MOOC environment given the large participant numbers, however it guided our view that at least a ‘light touch’ facilitation would be important in the Autism MOOC. Whilst regular facilitation may assist in acknowledgement and reinforcement, other tools are also available, including badging. In the CD MOOC, badging was used very effectively, with participants commenting that the badges added to their overall motivation to complete the MOOC (Lokuge- Dona, Gregory, Salmon, & Pechenkina, 2014; Salmon et al., 2015). Navigation As discussed previously in this paper, ease of navigation is important to enable participants to easily find and access resources and activities, and interact with others in the MOOC. We used the concepts of signalling and sign posting (Mayer, 2009) to improve navigation and accessibility. In the CD MOOC, we included a link to each activity to indicate how to navigate to other sections of the MOOC, and we used regular announcements to guide participants. The Autism MOOC design was kept very clean, with only two key areas for participants to access – the content section and the activities section. This kept navigation to a minimum and allowed participants to access resources and discussion forums very easily. One of the lessons learnt from the Carpe Diem MOOC was that introducing additional tools required additional navigation requirements that confused participants, so in the Autism MOOC we decided not to use additional tools and to keep navigation as simple as possible. Tools There are many tools available to facilitate interaction in online environments, however in our design we kept to the principle that ‘less is more’ and aimed to use key tools that would achieve our purpose without confusing participants. As both MOOCs were run through an open Learning FP:84 96 Management System, the primary tool used for interaction was the discussion forum. In addition, we used Blackboard Collaborate (virtual classroom) in the CD MOOC for synchronous discussions. In both the CD MOOC and the Autism MOOC, Facebook and Twitter streams were also active, providing a social media presence for participants who already used and liked these tools. Interestingly, participants within the CD MOOC requested Google + as an additional tool for effectively sharing materials, so whilst we were actively designing for simplicity, participants also had their preferred tools for sharing and interacting. Discussion and Conclusion The design principles discussed demonstrate some of the elements for consideration when developing MOOCs where interaction and collaboration is a key focus. The CD MOOC and the Autism MOOC had very different enrolment numbers, hence different group sizes were established (30 and 300 respectively). An interesting issue for future MOOCs is finding a group size that can accommodate significant drop out without groups becoming too small to be viable, but not so large that it is overwhelming in the beginning. The completion rates for the CD MOOC and the Autism MOOC were very similar, however the number of posts per person was much higher in the CD MOOC. Whilst smaller groups in the CD MOOC offered greater opportunity for dialogue, some groups became so small that the remaining participants had less opportunity to collaborate with others. In the larger groups in the Autism MOOC, the number of posts in the first few weeks may have overwhelmed some participants, and may also have reduced opportunity for meaningful discussion leading to the lower overall posts. Designing to ensure the experience is valuable for all participants – whether they complete the MOOC or not – is clearly important, and requires consideration of many of the elements discussed in this paper. We suggest that support through guides and resources, and access to email for technical support, is important particularly for participants who are unfamiliar with the learning tools and techniques used in MOOCs. Accessibility, clarity of task and structure, ease of navigation, and effective use of purposeful tools and resources improves the user experience, and enables participants to focus on the content and the interaction rather than struggling with the environment. The experience of designing two different MOOCs, with the intent of facilitating as much interaction as possible between participants, has highlighted the importance of careful consideration in applying design principles. In particular, we suggest that taking a ‘participant first’ approach focuses the attention of MOOC designers on the needs, aspirations and attributes of the intended MOOC participants, and may help in increasing the completion rate within MOOCs and particularly enable participants to interact with ease. References Bates, T. (2013). Harvard’s current thinking on MOOCs [web log post]. http://tinyurl.com/a2uh86z Brinton, C. G., Chiang, M., Jain, S., Lam, H. K., Liu, Z., & Wong, F. M. F. (2014). “Learning about social learning in MOOCs: From statistical analysis to generative model. IEEE Transactions on Learning Technologies, 7(4), 346-359. Brindley J. E., Walti, C., & Blaschke L. M., & Walti, C. (2009). Creating effective collaborative learning groups in an online environment. The International Review of Research in Open and Distributed Learning, 10(3). http://www.irrodl.org/index.php/irrodl/article/viewArticle/675 Conradie, P. W. (2014). Supporting self-directed learning by connectivism and personal learning environments. International Journal of Information and Education Technology, 4(3), 254-259. doi:10.7763/IJIET.2014.V4.408 Churchill, D. (2011). Conceptual model learning objects and design recommendations for small screens. Journal of Educational Technology & Society, 14(1), 203-216. Frey, N., Fisher, D., & Everlove, S. (2009) Productive group work: How to engage students, build teamwork, and promote understanding. Alexandria VA: ASCD. Gillani, N., Yasseri, T., Eynon, R., & Hjorth, I. (2014). Structural limitations of learning in a crowd: Communication vulnerability and information diffusion in MOOCs. Scientific Reports, 4. doi:10.1038/srep06447 FP:85 97 Hickey, G. (2014). The importance of learning philosophies on technology selection in education. Journal of Learning Design, 7(3), 16-22. https://www.jld.edu.au/article/view/184 Hill, P. (2013). Emerging student patterns in MOOCs: A graphical view [web blog post]. http://mfeldstein.com/emerging_student_patterns_in_moocs_graphical_view/ Juwah, C. (2006). Interactions in online peer learning. In R. C. Sharma & C. Juwah (Eds.), Interactions in online education. Implications for theory and practice (pp. 171-190). New York, NY: Lawrence Erlbaum. Johnson, D. W., Johnson, R. T., & Smith, K. A. (1991). Cooperative learning: Increasing college faculty instructional productivity. ASHE-ERIC Higher Education Report No. 4. Washington, D.C. http://eric.ed.gov/?id=ED343465 Khosa, D. K., & Volet, S. E. (2013). Promoting effective collaborative case-based learning at university: A metacognitive intervention. Studies in Higher Education, 38(6), 870–889. doi:10.1080/03075079.2011.604409 Kop, R., Fournier, H., & Mak, J. S. F. (2011). A pedagogy of abundance or a pedagogy for human beings: Participant support on massive open online courses. The International Review of Research in Open and Distance Learning, 12(7), 74- 93. http://www.irrodl.org/index.php/irrodl/article/view/1041/2025 Kop, R., & Hill, A. (2008). Connectivism: Learning theory of the future or vestige of the past? The International Review of Research in Open and Distance Learning, 9(3). http://www.irrodl.org/index.php/irrodl/article/viewArticle/523 Laurillard, D. (2012). Teaching as a design science: Building pedagogical patterns for learning and technology. New York, NY: Routledge. Lokuge Dona, K., Gregory, J., Salmon, G., & Pechenkina, E. (2014). Badges in the Carpe Diem MOOC. In Rhetoric and Reality: Critical perspectives on educational technology. Paper presented at the ASCILITE Conference, Dunedin, New Zealand. Milligan, C., Littlejohn, A. & Margaryan, A. (2013). Patterns of Engagement in Connectivist MOOCs. MERLOT Journal of Online Learning and Teaching, 9(2). http://jolt.merlot.org/vol9no2/milligan_0613.htm Marra, R. M., Jonassen, D. H., Palmer, B., & Luft, S. (2014). Why problem-based learning works: Theoretical foundations. Journal on Excellence in College Teaching, 25(3/4), 221- 238. Mayer, R. E. (2005). The Cambridge handbook of multimedia learning, New York, NY: Cambridge University Press. Mayer, R. E. (2009). Multimedia learning (2nd ed.). New York, NY: Cambridge University Press. Mayer, R. E., Heiser, J., & Lonn, S. (2001). Cognitive constraints on multimedia learning: When presenting more material results in less understanding. Journal of Educational Psychology, 93(1), 187–198. http://dx.doi.org/10.1037/0022-0663.93.1.187 Salmon, G. (2002). E-tivities: The key to online learning. London, UK: Kogan Page. Salmon, G. (2011). E-moderating: The key to teaching and learning online (3rd ed.). New York, NY: Routledge. Salmon, G. (2013). E-tivities: The key to active online learning (2nd ed.). New York, NY: Routledge. Salmon, G. (2014). Carpe Diem - a team based approach to learning design. http://www.gillysalmon.com/carpe-diem.html Salmon, G., Gregory, J., Lokuge Dona, K., & Ross, B. (2015). Experiential online development for educators: The example of the Carpe Diem MOOC. British Journal of Educational Technology, 46(3), 542-556. doi:10.1111/bjet.12256 Shirky, C. (2012). Napster, udacity, and the academy [web log post]. http://www.case.edu/strategicplan/downloads/Napster-Udacity-and-the-Academy- Clay_Shirky.pdf Smith, K. A., Sheppard, S. D., Johnson, D. W., & Johnson, R. T. (2005). Pedagogies of engagement: Classroom‐based practices. Journal of Engineering Education, 94(1), 87-101. doi:10.1002/j.2168-9830.2005.tb00831.x Tobin, M. (2011). Put me first: The importance of person first language. Innovations and Perspectives, Virginia Department of Education’s Training and Technical Assistance Centre. http://www.ttacnews.vcu.edu/2011/05/put-me-first-the-importance-of-person-first- language/ Tom, M. (2015). Five Cs Framework: A student-centered approach for teaching programming courses to students with diverse disciplinary. Journal of Learning Design, 8(1), 21-37. FP:86 98 Jordan, K. (2015). MOOC completion rates: The data. http://www.katyjordan.com/MOOCproject.html Weller, M. (2011). A pedagogy of abundance. Spanish Journal of Pedagogy, 69(249), 223–235. Yuan, L., Powell, S., & Bill, O. (2014). Beyond MOOCs: Sustainable online learning in institutions. Centre for Educational Technology, Interoperability and Standards (Cetis). http://publications.cetis.org.uk/2014/898 Yuan, L., & Powell, S. (2013). MOOCs and open education: Implications for higher education. A White Paper for JISC CETIS Centre for Educational Technology & Interoperability Standards. http://publications.cetis.org.uk/2013/667 Dona, K.L. & Gregory, J. (2015). A ‘participant first’ approach to designing for collaborative group work in MOOCs. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:77-FP:87). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:87 99 Building graduate attributes using student-generated screencasts Jessica Katherine Frawley Laurel Evelyn Dyson Jonathan Tyler Faculty of Engineering & IT Faculty of Engineering & IT School of Business University of Technology, Sydney University of Technology, Sydney University of Technology, Sydney James Wakefield School of Business University of Technology, Sydney There has been an increasing emphasis in recent years on developing the “soft” skills, or graduate attributes, that students need once they finish their university studies in addition to the specific domain knowledge of their discipline. This paper describes an innovative approach to developing graduate attributes through the introduction of an optional assignment in which first-year accounting students designed and developed screencasts explaining key concepts to their peers. Screencasts have been used in recent years for teaching but the approach of students, rather than teachers, making screencasts is far less common. Quantitative and qualitative analysis of student surveys showed that, in addition to improving their accounting knowledge and providing a fun and different way of learning accounting, the assignment contributed to the development and expression of a number of graduate attributes. These included the students’ ability to communicate ideas to others and skills in multimedia, creativity, teamwork and self-directed learning. Keywords: Graduate Attributes, Student-Generated Content, Peer Learning, Accounting Students Introduction In recent years there has been a growing recognition of the importance of “soft” or generic skills in the workplace over and above the domain-specific knowledge and expertise that are required to effectively exercise a profession (Litchfield, Frawley, & Nettleton, 2010). This has been accompanied by concerns that university education in many fields is in danger of degenerating into “a technical training camp for business and industry rather than fulfilling its mission to educate and empower the individual” (Scott, 2010; p. 381). Universities have responded by mapping graduate attributes across their degree programs and embedding into learning activities the development of skills such as teamwork, interpersonal communication, problem solving, critical thinking, creativity, ethical decision making, time management and lifelong learning. However, there remains some debate about the best method of developing graduate attributes in university courses. Barrie (2005; p.3) calls for a systematic, evidence-based approach to address the development of generic attributes, and notes that many universities have adopted mere “policy statements and relatively surface mapping strategies”, which do not constitute evidence of attainment of generic skills by their graduates. In this paper we describe the implementation of a new screencast assignment aimed at building graduate attributes in undergraduate students enrolled in an introductory accounting subject while also giving them the opportunity to learn accounting in a new and interesting way. A screencast consists of the digital recording or screen capture of any actions taking place on a computer screen, accompanied by a voice narration (Educause, 2006). They have great explanatory power, combining as they do both images and audio explanation of what is being viewed on the screen. Thus they have frequently been employed in instructional software guides and increasingly in education, the best known user being the online Kahn Academy (n.d.). For the most part, however, the trend has been for teachers and experts to produce screencasts, rather than students. Having students create them instead places the students at the centre of learning and moves away from passive instructional methods. Furthermore, it recognizes that students who have been exposed to technology for most of their lives require FP:88 100 new pedagogical methods to engage them (Tapscott, 1998). Having students make screencasts provides many learning benefits. These include the development of generic skills in university students, such as technology skills, creativity, the ability to communicate knowledge and work collaboratively (Mohorovičič, 2012; Shafer, 2010). Screencasts have also been shown to be highly motivating and enjoyable for primary school children to make, providing a different way of learning, aiding their understanding through the need to make repeated attempts at the task, helping them become autonomous learners, and being adaptive to different learning styles and individual speeds of learning (Rocha & Coutinho, 2011). To date there has been little research on the use of student generated screencasts within higher education contexts. Given the evidence from these few studies and other research that demonstrates that learner-centred approaches to education consistently aid in the development of graduate attributes (Barrie, 2005), extending research into student-generated screencasts within higher education is important. This paper commences with an overview of the literature that positions this research in relation to the need for graduates with new skills and capabilities to cope with the modern work environment, as well as higher education and the role of learning technologies. We then provide details of the screencast assignment and how it was first trialled and evaluated before its sustained adoption within the introductory accounting subject at our university. The findings of the evaluation are presented, including results of pre- and post-assignment surveys, and an evaluation of the screencasts by the accounting academics conducting the course. Generally, the screencast assignment provided an avenue for students to learn accounting while improving their ability to communicate accounting knowledge to their peers and to learn new multimedia skills, while also developing other graduate attributes such as creativity, teamwork and independent learning. Issues still remain about how to measure improvements in some of the graduate attributes, and the authors highlight this as an area for future research. This paper’s contributions are two-fold. Firstly, for researchers, this paper aims to deepen our understanding of an innovative application of educational technology to an area of increasing importance. Secondly, for practitioners, the implementation of this assessment could easily be adapted to any other field in which there is a core body of knowledge and principles or concepts that can form the content for students to generate their own screencast. Building graduate attributes in university students Graduate attributes have been defined as the “qualities, skills, and understandings a university community agrees its students should develop during their time with the institution” (Bowden, Hart, King, Trigwell, & Watts, 2000). These skills go beyond mastery of the body of knowledge and emphasise skills and qualities that are applicable to a range of contexts (Barrie, 2004). Whilst descriptors and categories vary, generic graduate attributes typically include transferrable skills such as: critical and creative thinking, communication, teamwork, leadership, ability to apply knowledge, and ethics. Though the emergence of a graduate attributes literature is relatively new, the expectation that graduates acquire skills and qualities supplementary to their disciplinary education is not. Assumptions about the qualities and generic abilities of university graduates can be traced back as far as 1862 (Barrie, 2004) and the learning of generic skills has been described as an existing but hidden curriculum, one that is often incidental and implicit to students’ learning of the body of knowledge (Frawley & Litchfield, 2009). However, it is perhaps only recently that universities and higher education have been called upon to explicitly address soft skill or graduate attribute development. There has been a surge of academic recognition and discussion as to the importance of such skills (e.g. Barrie, 2004, 2005, 2006; Chalmers & Partridge, 2012; Clanchy & Ballard, 1995; de la Harpe & David, 2012). Government, professional societies, accrediting bodies, and employers, have called repeatedly for universities to produce graduates that have the skills necessary to meet the needs of the contemporary workplace (AC Nielsen Research Services, 2000; Australian Chamber of Commerce and Industry & the Business Council of Australia, 2002; Department of Education Science and Training, 2004; Mayer, 1992). In general, there is a perception of the workplace as an increasingly complex and rapidly changing environment operating according to many unpredictable factors. The European University Association (2007; p. 6) points to the shift from a reliance on a body of knowledge to a greater emphasis on dynamic processes: “The complex FP:89 101 questions of the future will not be solved “by the book”, but by creative forward-looking individuals and groups who are not afraid to question established ideas and are able to cope with the insecurity and uncertainty that this entails.” Within the literature, discussions of graduate attributes are routinely tied to stakeholder pressure from employers and industry bodies. The term graduate attributes is sometimes used synonymously with employability skills (e.g. Chalmers & Partridge, 2012, p. 57). This has raised questions as to the nature of knowledge and the role of the university (Barrie & Prosser, 2004, p. 244). However, it would be reductive to think that the embedding of graduate attributes within the curriculum solely served the needs of industry. As Hager and Holland (2006) point out, advantages of the inclusion of graduate attributes within education not only serves industry, but improves course development, course delivery and assessment and quality assurance. Furthermore, definitions of graduate attributes, at least within the Australian literature, constitute more than employability skills. There is recognition that generic skills form a wider role within a student’s life. These include preparing students to be members of society and “agents of social good” (Bowden et al., 2000; Hager, Holland, & Beckett, 2002). They are the skills which form the foundation for the lifelong learning process (Cummings, 1998; Hager & Holland, 2006). Whilst the importance of graduate attributes is acknowledged within the literature, methods for fostering these within university education remain a contentious issue. Focused approaches typically embed graduate attribute learning activities into the context of the discipline, for example in creativity training and brainstorming exercises (Ogilvie, & Simms 2009), or computer simulations that promote the generation of creative solutions (Wynder, 2004). Other academics advocate courses in literature, history or religion for non-humanities students (e.g. Lister, 2010). Fogarty (2010) criticizes this approach as being not scalable to the large numbers of students enrolled in subjects such as accounting, and too indirect a method, given the distance of the humanities from the accounting discipline. Current research suggests that graduate attributes are best developed through learning and teaching that is: integrated into the curriculum (e.g. Litchfield, Frawley, & Nettleton, 2010), employs active approaches (Moy, 1999) and adopts “sophisticated, student-centered and process-focused” pedagogies (de la Harpe & David, 2012; p. 494). Increasingly, researchers and practitioners have enacted these principles and pedagogies with the support of educational technologies. e-Portfolios are a way in which students collect evidence of learning over the course of their degree in a wide range of media formats and reflect on this portfolio in order to develop graduate attributes and provide evidence to both the educational institution and prospective employers of their meeting expected professional standards (Allen & Coleman, 2011; von Konsky & Oliver, 2012). Online Web 2.0 tools, such as blogs and wikis, have been shown to promote communication and collaborative problem solving, and enhance student engagement with and reflection on learning tasks (Douglas, & Ruyter, 2011). Such approaches provide active, student-centred learning where “the learning activity and assessment task are one and the same” (Allen & Coleman, 2011; p. 59). In summary, whilst the literature on learning and teaching of graduate attributes advocates for embedded, active, collaborative and learner-centred approaches, scalability continues to pose a major challenge, particularly in disciplines typified by large enrolments. Technologies offer approaches that have the potential to scale-up and accommodate large subjects, such as the one that is the focus of this paper. Within this challenging educational context, the screencast assignment offers a complex, student-centred task that calls on students to develop a range of graduate attributes to complete it effectively. Implementation and evaluation of the screencast assignment The screencast assignment is the result of a collaboration between Business School academics teaching introductory accounting and Information Technology (IT) academics. Introductory accounting, it should be noted, is one of the largest subjects in the university with enrolments of approximately 1,500 students in the first semester and about 500 students in the mid-year intake. Students include those who willingly take the subject, either as an elective or as the first step in an accounting career, and those who only take the subject because it is a core requirement of their degree; the latter are often poorly motivated. The subject has historically FP:90 102 been perceived to be boring, with low student engagement and high failure rates. The screencast assignment aimed to develop graduate attributes while also improving student engagement by offering of a learning experience that would be different from and more creative than the norm. It further sought to promote the learning of accounting both through students’ creation of screencasts and through peer-learning from the screencasts of others. Graduate attributes that were the focus of the assignment were the ability to communicate accounting knowledge to others and the development of multimedia communication skills. The inclusion of the latter recognised that, in the twenty-first century, communication practices have changed and now include a wide range of media and multimedia (Davies, 2003). Students come to university equipped with existing skills and take part in multimedia practices outside the classroom, uploading their own user-generated content, such as photographs and videos, to file-sharing websites like YouTube and Facebook (Dyson, 2012). The screencast assignment accepted the current practices of the students while incorporating them into the assignment in order to build their multimedia communication skills further. In addition to these two areas of focus, it was hoped that other graduate attribute development would emerge, even though these would be recognized only after the evaluation of the trial. The trial of the assignment required students, working mostly in small teams (2-3 students), to create a short (3-5 minute), standalone screencast explaining an accounting concept to their fellow students. Though the assignment was designed as a team activity, a minority of students expressed a desire to work by themselves and this was allowed. Students attempted the assessment on an optional basis for a bonus 10 marks, in addition to their other assessments. All students were provided with a short instructional brochure on how to make a screencast using free Jing software (www.techsmith.com/jing), and were given an example of a screencast prepared by the teaching team. Headset microphones and access to quiet computer rooms at the university were available. Another resource was provided in the form of one of the IT researchers, who could provide technical help and advice if they needed it. The screencasts were then marked by accounting academics in the Business School and the best of them used in the final revision lecture. The assignment was trialed in the second semester of our academic year as this has smaller numbers of students and so is more manageable for introducing new learning and teaching innovations. Following an evaluation of the trial, some modifications were made to the procedure and the assignment offered as a permanent part of the course. These changes will be detailed after the results of the evaluation have been discussed. Evaluation The aim of the evaluation of the trial was to assess whether the screencast assignment was a success in terms of achieving its objectives and, if necessary, to suggest modifications to improve the assignment for subsequent delivery. Only students who had chosen to make a screencast were invited to provide evaluations. Two surveys were conducted of all those who had elected to undertake the screencast assignment. The response rate was 100% as students were required to register first and could not register or submit the assignment without completing the surveys. Confidentiality was ensured by having the surveys administered and anonymised by one of the IT researchers, rather than the accounting lecturers. The Pre- Assignment Survey was completed when students first registered to do the assignment (n = 124), while the Post-Assignment Survey was conducted once students had completed and uploaded their screencasts (n = 119). The difference between the numbers of students submitting surveys can be attributed to students dropping out of the subject or no longer wishing to undertake the screencast assignment. The Pre- and Post-Assignment Surveys were designed to gauge student perceptions of the following: 8. Students’ knowledge of accounting and their ability to explain it to their peers: pre- and post- assignment (5-point Likert-scale questions). 9. Students’ multimedia and screencasting experience: pre- and post-assignment (5-point Likert-scale questions). 10. Students’ motivation for undertaking the screencast assignment and what they hoped to learn: pre-assignment (open-ended questions). FP:91 103 11. Students’ likes and dislikes of the assignment: post-assignment (open-ended questions). Survey questions were designed using concepts from the literature and previous survey questions on engagement trialed within the accounting subject. The Likert-scale questions focused on the prime graduate attributes that the assignment was expected to develop (students’ ability to communicate accounting knowledge and multimedia communication skills), as well as students’ learning about accounting. The open-ended questions hoped to uncover the development or expression of other graduate attributes, in addition to gaining an indication of student engagement with the activity and any areas for improvement in its delivery. The answers to the open questions were analysed by grouping responses into common themes. In addition, the accounting lecturers and tutors reviewed all the screencasts produced and evaluated the accuracy of the accounting knowledge contained in the screencasts and the level of multimedia skills demonstrated. While the academics were highly experienced in assessing accounting, assessments of the multimedia products and their effectiveness, visual appeal and creativity emerged from discussions with the wider research team. Findings Of the 539 students enrolled in the subject, 124 or 23% elected to undertake the assignment. The total number of screencasts produced was 58. Despite offers of help, few students contacted the technical support person on the research team for assistance. Most students preferred to work it out for themselves from a combination of: the instructional brochure, the example screencast provided, and by “playing” with the technology. No students borrowed the microphones provided and most used their own computers to record the screencasts. Accounting knowledge and ability to communicate it Two Likert-scale questions about students’ accounting knowledge were repeated before and after they had attempted the screencast assignment. One focused on their knowledge of accounting and the other on their confidence in explaining basic accounting concepts to their peers. A 2-tailed (paired-samples) t-test was applied and showed that students saw themselves as significantly better informed about basic accounting concepts after producing a screencast (significant at the 10% level). Furthermore, they rated themselves as better at explaining accounting to their peers after the assignment and this was statistically significant (at the 1% level) (Table 1). Though this demonstrates improvement, students do not, at this stage in the course, feel highly confident or well informed of accounting concepts. Multimedia and screencasting experience The focus of the Pre-Assignment Survey was students’ prior experience of producing multimedia content and, more specifically, whether they had ever made a screencast before. The Post-Assignment Survey, on the other hand, probed their experience of making the screencast for this assignment: whether they had enjoyed learning the multimedia skills necessary, and their degree of satisfaction with the finished product. The results (Table 2) showed that while almost half of students (47%) stated that they had produced some kind of multimedia content previously, an overwhelming 90% of students said that they had never made a screencast before. The Post-Assignment Survey revealed a high degree of satisfaction with the experience offered by the assignment: 80% of students agreed or strongly agreed that they had enjoyed learning the multimedia skills needed to produce the screencast, and three- quarters (76%) were satisfied with their product. Table 1: Students’ accounting knowledge and ability to explain it Questions Mean Probability Value Score (out of 5) I am well informed about basic accounting Pre 2.69 p = 0.092 concepts. Post 2.82 I feel confident about explaining these Pre 2.35 p = 0.002 FP:92 104 accounting concepts to others. Post 2.60 Table 2: Students’ multimedia and screencast experience (% of responses) Pre-Survey Questions Never Once Neutral Sometime Often s I have produced multimedia before 18 13 22 39 8 using cameras, microphones, video Mean Score (out of 5) 3.06 editors, etc. I have previously made screencasts. 90 4 3 2 0 Mean Score (out of 5) 1.48 Post-Survey Questions Strongly Disagree Neutral Agree Strongly Disagree Agree I enjoyed learning multimedia skills. 0 2 18 47 33 Mean Score (out of 5) 4.11 I am satisfied with the final 2 4 18 61 15 screencast. Mean Score (out of 5) 3.83 Students’ motivations and learning objectives The two Pre-Assignment Survey questions “Why did you choose to do a screencast?” and “What do you hope to learn?” were open-ended and thus students could state more than one reason in their answer. Responses were qualitatively and thematically coded to look for dominant themes across the data. For the first question, “Why did you choose to do a screencast?”, the most common reason was bonus marks: 89% of students cited bonus marks as one of their reasons for doing the screencast (see Figure 1). Equal to this, 89% were students’ aiming to better understand accounting, or understand the accounting concept that was the focus of their screencast. Students’ responses typically included more than one reason, for example, “We like the idea of optional, so rather than being forced to complete the assignment – we are enthusiastic to complete the assignment at our own will with an extra incentive of 10 bonus marks”. Numerous other reasons were given, for example, “I believe it is a good way to learn thoroughly a specific concept within this course, whilst also expressing my understanding in a fun, interactive and different medium”. Students also expressed interest in learning how to do a screencast: the reasons for this were divided between wanting to learn how to use the technology (5%) and believing that screencasting would be useful for work (4%): for example, “I like to have the opportunity to gain bonus points and to gain experience in different medium that I have never used, this may be useful in future work”. In response to the second question “What do you hope to learn?” students expressed three dominant themes. There was a very specific accounting content focus for 31% of students: typical responses in this category included “I hope to learn the concept of GST clearing a lot better as I found it difficult to understand before”, or the general statement “I hope to learn more about the accounting concept we chose”. On the other hand, 26% of students were focused on graduate attributes with respect to their intended learning outcomes. Throughout the responses there is reference to creative thinking, teamwork, multimedia and screencasting skills as well as learning how to communicate difficult content to others. Examples of responses are “Teamwork, multimedia skills”, and “Team Work, how we can be creative in explaining concepts on a dry subject”. Excluding four miscellaneous comments, the remaining 40% of students had mixed graduate attribute and content-learning objectives (Figure 1.). FP:93 105 89% 89% 15% 5% 4% 4% 3% 2% Figure 1: Students’ motivations in undertaking the screencast assignment (students could mention more than one) Students’ likes and dislikes Data from the Post-Assignment Survey about students’ likes and dislikes (“What did you like about this assignment? Why? ” and “What did you dislike about this assignment? Why?”) demonstrated that students’ response to the screencast assignment was largely positive. Negative feedback, or dislikes, accounted for a smaller proportion of the total feedback given. As in the open-ended questions from the Pre-Assignment Survey, students could state more than one reason in their answer. The answers to the question about what students liked about the screencast assignment revealed many interesting themes, summarized in Figure 2. It can be seen that students appreciated that it gave them an improved understanding of accounting (29% of students mentioned this, including the opportunity for exploratory research and revision); they liked earning bonus marks and the opportunity to improve their overall grade in the subject (25%); it allowed them to develop their practical multimedia skills and learn new software (25%); they saw it as a different way of learning accounting and presenting information in a different way (25%); the assignment was interesting or fun (23%); and it allowed them to be creative or innovative (22%). Smaller proportions of students liked the fact that the assignment offered choice, either in terms of topic or that the assignment was optional (13%); the teamwork aspect (12%); and teaching other students accounting (10%). The answers to the question about what students disliked about the screencast assignment are summarized in Figure 3. This information was useful for revising and improving the assignment. It should be noted that 29% of students liked everything about the assignment. Of those students who expressed a dislike, the greatest number (31%) focused on technical issues, including the problem that the Jing software used does not allow editing and so students often had to make more than one recording before getting their screencast right. Some students disliked the time restriction on the length of the screencast (15%); some found the instructions about how to make a screencast or the marking criteria inadequate (9% and 8% respectively), and a few had team problems (4%). Accounting academics’ evaluation of the screencasts The accounting lecturers and tutors who marked the screencasts found that the majority demonstrated good multimedia skills. However, the majority of screencasts produced followed FP:94 106 the one model provided by the research team, that is, a slideshow screencast. Students failed to explore other technological approaches. The accounting academics also found that the majority of screencasts demonstrated a reasonably good grasp of the accounting concept being explained. However, many contained minor accounting errors. As a result, only 12% (7 screencasts) were deemed suitable for use as teaching and learning resources. Discussion The evaluation of the new screencast assignment showed that, even though 90% of students had never made a screencast before and sought minimal help in producing their screencasts (aside from one example and a short brochure on how to make them), 23% of students enrolled in the subject elected to make a screencast. 29% 25% 25% 25% 23% 22% 13% 12% 10% 2% of accounting Improved understanding Bonus marks and grades Developing multimedia Different way of Interesting or fun Being creative or Being able to choose Teamwork Teaching other students Other Figure 2: What students liked about the screencast assignment (students could mention more than one) Students accounting knowledge and ability to communicate it The objective of promoting the learning of accounting through the screencast assignment was achieved, even if most screencasts, like many other assignments, had small accounting errors. Students saw themselves as better informed about basic accounting concepts after producing a screencast and this change was statistically significant at the 10% level of significance (Table 1). Was this an effect of undertaking the screencast assignment or the result of learning about accounting in lectures, tutorials and through students’ study for other assignments and exams over the course of the semester? Certainly, students linked it to the assignment: an improved understanding of accounting and the opportunity for researching and revising accounting was the most commonly listed aspect of the assignment that students liked (29% of students – Figure 1). Students’ learning how to communicate accounting to their peers was evident as 10% of students said they liked teaching other students (Figure 1) and students rated themselves as better at explaining accounting to others after the assignment. The latter was statistically significant (Table 1). The seven screencasts that were completely accurate were used in the final revision lecture of the course and so contributed in some way to peer learning although this was not measured. The number of accurate screencasts for peer learning is expected to FP:95 107 increase as the assignment continues to be offered in subsequent semesters. In time it is hoped that a library of resources will be available for use in both lectures and for students’ private study. 31% 29% 15% 9% 8% 4% 2% recording issues Technical problems and Liked everything screencast Time restriction of Instructions provided Assignment criteria Group work Other Figure 3: What students disliked about the screencast assignment (students could mention more than one) Students’ multimedia skills The majority of students (80%) enjoyed learning the multimedia skills required to produce the screencast (Table 2); and many liked the assignment because it allowed them to develop their multimedia skills (25%) (Figure 2). The fact that 90% of students had never produced a screencast before shows that the assignment truly extended their multimedia communication skills and was not merely an exercise in allowing them to practice already acquired user- generated content skills, although it may have built on these. Moreover, the majority of students were satisfied with what they had produced and the accounting academics believed the majority of the screencasts demonstrated good multimedia skills. The fact that 24% of students were either neutral or dissatisfied about the quality of their screencasts (Table 2) was probably due to the lack of editing functions in the Jing software that students were using. Though much more sophisticated software, such as Camtasia, is available on the market, it was too expensive to purchase a license for the large number of students enrolled in the subject and, furthermore, its greater editing sophistication was deemed to create too big a learning curve for students who had little prior experience of making screencasts. With such large numbers of students enrolled in introductory accounting, a simple software package that students could learn and use with minimal support was essential for practical reasons. Other graduate attributes Students’ answers to the pre-assignment question about what they hoped to learn and the post- assignment question about what they liked about the assignment provide evidence that the screencast assignment offered them an avenue for developing additional generic skills. The fact that 66% of students undertaking the assignment wanted to acquire soft skills and not just learn more about accounting shows that students realize that studying a course is not merely a matter of acquiring the body of knowledge, but that graduate attributes are also a necessary component. The range of attributes mentioned included creative thinking and teamwork. Again, being creative or innovative (22% of students) and teamwork (12%) were two of the things students liked most about the assignment (Figure 1). These are important skills in the modern FP:96 108 workplace. A further graduate attribute that can be deduced from the conduct of the assignment is that of learner autonomy and students preparation for lifelong learning. The fact that students did not require the support of the nominated contact on the research team and used their own recording and computer equipment shows that they were prepared to figure things out themselves and use their own resources, despite the technical problems and recording issues that almost a third of students encountered (Figure 3). Improving student engagement In addition to the building of graduate attributes and students’ subject knowledge, a positive aspect of the screencast assignment was that it provided an engaging way of studying accounting. The high level of student motivation demonstrated by the many aspects students liked about the assignment (Figure 2) and the fact that 29% of students could cite nothing they disliked about the assignment (Figure 3) show that our objective of improving student engagement with the subject has been realized, at least for those students (23%) who chose to undertake this optional assignment. For many, it was a different way of learning accounting (25%) and was interesting or fun (23%) (Figure 2). Revising the screencast assignment Following student feedback the screencast assignment has been modified and is now a permanent component of the introductory accounting course. Based on student feedback we decided to keep the assignment optional. Students complaints about needing clearer instructions and marking criteria were acted on by revising the instructional brochure, providing more examples of screencasts and giving more precise criteria. The new exemplar screencasts include different technological approaches in order to stimulate students to expand beyond the slideshow approach and be more creative in this aspect. However, student complaints about the short allowable length of the screencasts have not been followed: in fact, the permissible length was reduced to 3 minutes, instead of 3-5 minutes in the trial semester. The accounting academics felt that the shorter screencasts were more successful in conveying the core message. Conclusions As far as we are aware, the use of student-generated screencasts for building graduate attributes is a unique approach in the accounting discipline, and represents an innovative approach in university education as a whole. The assignment engages with students’ everyday practices and interests in multimedia, while extending their skills to a new and powerful learning and teaching medium which few have prior experience of producing, namely screencasts. The screencast assignment offers students the opportunity for acquiring discipline-specific knowledge while becoming more confident in communicating the concepts they are learning, doing this using newly acquired multimedia skills. There is evidence from our study of the development of other graduate attributes, such as creativity, teamwork and independent learning. Furthermore, the assignment is scalable to the large numbers of students enrolled in accounting and requires little in the way of support once the example screencasts and “how to” notes have been developed. However, issues remain about how to accurately measure the impact of the activity on improving students’ graduate attributes. In this trial, we relied on students’ perceptions of the activity and the accounting academics’ lay evaluation of the effectiveness of multimedia expression in the completed screencasts. More thought will be given to these issues in the future while we continue to pursue this innovative approach to building the graduate attributes our students will require in the workplace. References ACNielsen Research Services (2000). Employer Satisfaction with Graduate Skills. Research Report. Canberra: Department of Education, Training and Youth Affairs, Australian Government Publishing Service. Allen, B., & Coleman, K. (2011). The creative graduate: Cultivating and assessing creativity with eportfolios. In G. Williams, P. Statham, N. Brown & B. Cleland (Eds.), Changing Demands, Changing Directions. Proceedings ascilite Hobart 2011, (pp. 59-69). FP:97 109 Australian Chamber of Commerce and Industry & the Business Council of Australia (2002). Employability Skills for the Future. Canberra: Australian Government Publishing Service. Barrie, S. (2004). A research-based approach to generic graduate attributes policy. Higher Education Research & Development, 23(3), 261-275. Barrie, S. (2005). Rethinking generic graduate attributes. HERDSA News, 27(1), 1-6. Barrie, S. (2006). Understanding what we mean by the generic attributes of graduates. Higher Education, 51(2), 215-241. Barrie, S., & Prosser, M. (2004). Editorial. Higher Education Research & Development, 23(3), 243-246. Bedford, N., Bartholomew, E. E., Bowsher, C. A., Brown, A. L., Davidson, S., Horngren, C. T., Knortz, H. C., Piser, M. M., Shenkir, W. G., Simmons, J. K., Summers, E. L. & Wheeler, J. T. (1986). Future accounting education: Preparing for the expanding profession, Special Report, Executive Summary. Issues in Accounting Education, 1(1), 168-193. Bowden, J., Hart, G., King, B., Trigwell, K., & Watts, O. (2000). Generic capabilities of ATN university graduates. Adelaide: Teaching and Learning Committee Australian Technology Network. Chalmers, D., & Partridge, L. (2012). Teaching graduate attributes and academic skills. In L. Hunt & L. Chalmers (Eds.), University teaching in focus: a learning-centred approach (pp. 56 - 73). Oxon: Routledge. Clanchy, J., & Ballard, B. (1995). Generic skills in the context of higher education. Higher Education Research & Development, 14, 155-166. Cummings, R. (1998). How should we assess and report student generic attributes? Paper presented at the Teaching and Learning in Changing Times, Proceedings of the 7th Annual Teaching Learning Forum, The University of Western Australia, February 1998, Perth. de la Harpe, B., & David, C. (2012). Major Influences on the Teaching and Assessment of Graduate Attributes. Higher Education Research & Development, 31(4), 493 - 510. Davies, J. P. (2003). DOA: Education in the Electronic Culture. Lanham, Maryland: Scarecrow Press. Department of Education, Science and Training (2004). Employability Skills. Canberra: Australian Government Publishing Services. Douglas, K., & Ruyters, M. (2011). Developing graduate attributes through role-plays and online tools: Use of wikis and blogs for preparation and reflection. In S. Baron et al. (Eds.), Proceedings of Global Learning (pp. 316-323), 28 March, Melbourne: AACE. Dyson, L. E. (2012). Student-generated mobile learning: A shift in the educational paradigm for the 21st century. anzMlearn Transactions on Mobile Learning, 1, 15-19. Educause (2006). 7 things you should know about screencasting. Educause Learning Initiative. Retrieved from www.educause.edu/eli. European University Association (2007). Creativity in Higher Education: Report on the EUA Creativity Project 2006-2007. Brussels: European University Association. Fogarty, T. J. (2010). Revitalizing accounting educaiton: A highly applied liberal arts approach. Accounting Education, 19(4), 403-419. Frawley, J., & Litchfield, A. (2009). Engaging students and academics in work-ready learning contextualised for each profession in the curriculum. Paper presented at the The Student Experience, Proceedings of the 32nd HERDSA Annual Conference, 6 - 9 July 2009, Darwin. Hager, P., & Holland, S. (2006). Introduction. In P. Hager & S. Holland (Eds.), Graduate Attributes: Learning and Employability (pp. 1-15). Dorecht, The Netherlands: Springer. Hager, P., Holland, S., & Beckett, D. (2002). Enhancing the learning of and employability of graduates: The role of generic skills. Melbourne: Business/Higher Education Round Table Position Paper No. 9. Kahn Academy (n.d.). www.kahnacademy.org. Lister, R. J. (2010). A role for the compulsory study of literature in accounting education. Accounting Education, 19(4), 329-343. Litchfield, A., Frawley, J., & Nettleton, S. (2010). Contextualising and integrating into the curriculum the learning and teaching of work-ready professional graduate attributes. Higher Education Research and Development, 29(5), 519-534. Mayer, E. (1992). Key competencies: Report of the committee to advise the ACE and MOVET on employment-related key competencies for post compulsory education and training. . Canberra: AGPS. Mohorovičič, S. (2012).Creation and use of screencasts in higher education. MIPRO, May 21- 25, Opatija, Croatia, pp. 1293-1298. FP:98 110 Moy, J. (1999). The Impact of generic competencies on workplace performance. Review of Research Monograph Series. Adelaide: National Centre for Vocational Education Research. ogilvie, d.t. & Simms, S. (2009). The impact of creativity training on an accounting negotiation. Group Decision & Negotiation, 18, 75-87. Rocha, A. M. M., & Coutinho, C. P. (2011). Web 2.0 tools in high school in Portugal: Creating screencasts and vodcasts for learning. US-China Education Review A, 1: 54-62. Shafer, K. G. (2010). The proof is in the screencast. Contemporary Issues in Technology and Mathematics Teacher Education, 10(4). Scott, P. (2010). A commentary on “A role for the compulsory study of literature in accounting education”. Accounting Education, 19(4), 381-384. Tapscott, D. (1998). Growing up digital: The rise of the net generation. New York: McGraw-Hill. von Konsky, B. R., & Oliver, B. (2012). The iPortfolio: Measuring uptake and effective use of an institutional electronic portfolio in higher education. Australian Journal of Educational Technology, 28(1), 67-90. Wynder, M. (2004). Facilitating creativity in management accounting: A computerized business simulation. Accounting Education, 13(2), 231-250. Frawley, J.K., Dyson, L.E., Tyler, J., & Wakefield, J. (2015). Building graduate attributes using student- generated screencasts. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:88-FP:99). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:99 111 Self-organising maps and student retention: Understanding multi-faceted drivers David Carroll Gibson Matthew Ambrose Matthew Gardner Curtin University Curtin University Curtin University Abstract: Student retention is an increasingly important yet complex issue facing universities. Improving retention performance is part of a multidimensional and deeply nested system of relationships with multiple hypothesised drivers of attrition at various sample sizes, population clusters and timescales. This paper reports on the use of a self- organising data technique, Kohonen’s Self Organising Map, to explore the potential retention drivers in a large undergraduate student population in Western Australia over a six-year period. The study applied the self-organizing method to two point-in-time data sets separated by 18 months and was able to identify a number of distinct attrition behaviour profiles appropriate for creating new tailored intervention. Keywords: Attrition, retention, predictive models, machine learning, educational data mining, learning analytics. Introduction The student retention rate is a broadly accepted and important measure of university performance, and is often considered as a proxy for the quality of education and support services provided (Crosling, Heagney, & Thomas, 2009; Olsen, 2007). Poor or declining retention is of concern for universities as it significantly affects financial performance and university reputation (Jensen, 2011), it is of little surprise that there has been significant research focused on understanding drivers of student retention and the development of models to predict student attrition (de Freitas et al., 2014). In the experience of the authors there are number of challenges in the development and use of predictive models of student attrition. • The rigorous experimental conditions that are desirable for the development of predictive models are difficult to achieve (many of the proposed drivers of attrition change simultaneously). • There is a complex time consideration, it can be difficult to assess the exact time of attrition, and indeed a typical attrition scenario is identified only when students fail to re-enrol. • The drivers of the attrition are broad and varied as are the demographic backgrounds and aspirations of students, consequently the functional dependencies of models on gathering and handling of data can be complex. • Even when predictive models are available the outputs are not easily understood by support staff and planning staff, due to the applicability of predictions within a given timeframe, current institutional processes, and the role of increasing information in evolving the predictability characteristics of the modelling approach Here we report on the use of the self-organising map technique, both its predictive ability and its utility in communicating potentially complex information about a student population to non-technical staff responsible for support and intervention planning services. Problem Definition In their interactions with the majority of higher education institutions, students typically access two types of services; academic (e.g. lectures, library materials and journals, tutorials, examinations, grading etc.) and supporting services (e. g. administration, counselling /advisory services, facilities, social services etc.). Additionally, each learner brings a number of demographic attributes (e. g. age, social economic status, prior aptitude for the subjects selected etc.). It is the goal of the education FP:100 112 provider to understand the dependencies between demographic attributes and the academic and support services they offer (or could potentially offer) and design interventions, actions and policy to optimise a desired outcome such as retention. One obstacle to optimising outcomes is a holistic understanding of the broad student population – also known as high dimensionality in the data – consisting of factors such as the variety of their sociocultural, psychological and historical characteristics and how these interact with their current intentions, daily patterns of private and social behaviour and academic performance. A well-established approach to understanding large high dimensional data sets is Kohonen’s Self Organising Map (SOM) (Kohonen, 1990). This section reviews the SOM technique before providing the specifics of our programme. A Kohonen model consists of input vectors 𝑉 = {𝑣1 , 𝑣2, … , 𝑣𝑖 , … , 𝑣𝑚 } with 𝑣𝑖 ∈ ℝ𝑛 and a Self-Organised Map 𝑀 ; a lattice of vectors 𝑀 = �𝑚𝑖,𝑗 � 𝑤𝑤𝑤ℎ 𝑚𝑖,𝑗 ∈ ℝ𝑛 . 𝑀 defines a mapping 𝑓: 𝑉 → 𝑀 ∶ 𝑓(𝑣) = 𝑚𝑖,𝑗 𝑖𝑖𝑖 𝑑�𝑣, 𝑚𝑖,𝑗 � = 𝑚𝑚𝑚{𝑑(𝑣, 𝑚), 𝑚 ∈ 𝑀} with 𝑑 a metric function on ℝ𝑛 , taken to be the Euclidean metric for our purposes here. 𝑀 is calculated according to the algorithm below: 1. Randomise map 𝑀 (a common heuristic is to evenly spread lattice vectors across the plan spanned by the first two principle components of 𝑉) 2. Randomly select input vector 𝑣𝑖 and compare to each 𝑚 to find the lattice point most similar to the input vector (i.e. 𝑚𝑖,𝑗 such that 𝑑�𝑣, 𝑚𝑖,𝑗 � = 𝑚𝑚𝑚{𝑑(𝑣, 𝑚), 𝑚 ∈ 𝑀}). 3. Update lattice points in a neighbourhood of 𝑚𝑖,𝑗 such to increase the similarity of the lattice 2 points to 𝑣𝑖 according to ∆𝑚𝑘,𝑙 = 𝑛0 𝑒𝑒𝑒(−𝑡�𝜏) 𝑒𝑒𝑒(−𝑆 �2𝜎(𝑡)2 ) where 𝑆 is the distance between lattice sites and 𝜎 is a monotonically decreasing function usually taken to be 𝜎(𝑡) = 𝜎0 𝑒𝑒𝑒(−𝑡�𝜏) 4. If 𝑡 is less than the maximum number of iterations increase 𝑡 and return to step 2. Applying the mapping 𝑓 to the input vectors produces a 2 dimensional representation of the higher dimensional data set where similarity of vectors relates to lattice separation (with the most similar input vectors mapped to the same node). Colouring nodes according to a component 𝑚𝑖,𝑗 produces a visually intuitive way to explore data. The goal of the study was to generate profiles of students likely to attrite by combining a large amount of known data from a number of university systems and to engage the stakeholder community in exploring the data, understanding the systems of the university and apply their creativity to generating new interventions, actions and policy to improve retention. Model Parameters The selection of 200+ fields from ten data systems in the university was prioritised based on the ease of data access and the perceived importance determined by interviewing a number of subject matter experts at the university. A consultation and engagement process with students, instructors and leaders from all areas of the university was undertaken to broaden the base of understanding of attrition and retention, surface the mental models of a wide range of stakeholders concerning their concepts and assumptions about potential drivers and leverage points in the system, and to ensure that the results of the project were visible to as wide as possible a group of concerned and active participants. Details of this process have been published in internal reports as well as briefly described in (de Freitas et al., 2014). Based on the consultation process, over 200 hypotheses were created and evaluated (Gibson & de Freitas, 2015) which shaped the choice of factors based on fields in the data systems (Table 1) through a hybrid approach of human shaped machine learning in a series of cycles of consultation and data mining. Prior to applying the self-organizing map technique, the research team followed the typical processes of data mining to collect, clean, transform, and conduct exploratory analysis in an iterative process that resulted in the refinement of data models and algorithms before, during and after the SOM technique is applied and re-applied. We can think of the exploratory process as a series of mappings, refinements and re-mappings, from raw data to meaningful indicators for use in creating M as defined above. M is then optimized for stakeholder consumption, via visualizations, and FP:101 113 interpretive communications of findings and musings concerning a relevant subset of 50 hypotheses from the original 200+. Some hypotheses do not have indicators (yet) in the data systems and cannot be addressed by data mining, and some were superseded by a result from an earlier finding making further analysis pointless. The SOM stage of the process is an example of unsupervised machine learning that is, once the data is made ready, computational resources explore and organize the data without human intervention until a data model ‘settles’ (converges to a solution in the form of a map representation). The map can then be further queried, manipulated and explored by stakeholders working alongside the data science team. Table 1. Data sources Data Source Domains covered Student demographic information including: • Age • Country of birth • Gender Student University Performance Student • Unit and course enrolment, changes and cancellations Enrolment System • Unit performance • Graduation status Pre-university measures • Previous institutions attended • Admissions method (direct applicants, school leaving examinations, existing tertiary qualifications etc.) Learning While the learning management system potentially contains a variety of pertinent management domains, due limitations on time and complexities associated with extracting systems data, only log information (time of day) was included. Library Library web logs revealed indicate when a student accesses the library computer Computer system and whether the access is from a university owned computer Weblogs Students take a number of surveys during their time at the university results from the following surveys are included †: Survey Data • Unit satisfaction • University Facility Satisfaction • Course satisfaction. High school students in the universities geography apply through a third party High School entity owned by public universities. Each university has visibility of all student Leavers applications in a given year and so it was possible to identify whether a student Applications had a higher preference for a competing institution. Students carry electronic cards which they can use to access facilities outside of Card Access normal hours. Logs of these cards can be used to track student usage of these System facilities Australian Bureau of Statistics After sourcing raw data from the above systems the authors combined the data into a single data set to take advantage of the SOM method to explore for trends in the high dimensional data set. For each domain it is not known a priori which features of a given domain are correlated with attrition and retention (e.g. no hypothesis is put in a privileged position) and so for each domain, multiple possible features are created by grouping, transformations, and other methods that combine business intelligence from the expert consultations with data and information expertise. For example from the learning management system weblogs, multiple features are possible based on which semester, the time of day of access and comparisons to the student’s cohort (i.e. students in the same course with a similar proportion of the course completed). Examples include: FP:102 114 • In the first semester of their final course what was the most times in a day the student logged into blackboard • In the first semester of their final course what was the average times in a day the student logged into blackboard • In the first semester of their final course how many times did the student log into blackboard • In the final semester of their final course what percentage of login attempts were made in the morning (7am – 12pm) • In the second last semester of their final course, compared to their cohort, how does this students usage compare, on a directional scale, for login attempts • In the second last semester of their final course, compared to their cohort, how does this students usage compare, on a directional scale, for login attempts in the afternoon 1pm – 6pm Continuing in this manner 95 middle level features were generated from the learning management weblog data. Applying a similar approach the data from the 10 systems that were sourced for the single dataset, 1,273 attributes per student were derived. These features have been called n-grams and motifs when derived from dynamic, highly interactive digital learning experiences, and meso-level (the raw data are called micro-level features and the systems that encompass and act as exogenous influences on these features are call macro-level features or factors). See (Gibson & Jakl, 2013; Gibson & Webb, 2015; Shum, 2011). Status Definition Since there are multiple possibilities for defining when attrition occurs it worth commenting on the definitions used in the model presented here. In an ideal scenario, students wishing to leave a course would inform student services, formally withdraw and complete an exit survey. Practically few students at this university follow such a procedure, many simply stop interacting with university (i.e. stop attending classes or services). We opted to assign a status based on students with active units. A student is considered to attrite if they fail to take any units at the university for two semesters after they were last enrolled in a unit, excepting of students who graduate after their last semester. At any point in time then students can be assigned a status based on the last semester in which they were enrolled in units • Current: the student has taken units in the most recent semester • Graduated: The student has completed their course in the last semester that they interacted with the university. Students enrolled in two courses that complete one course in the last semester they interacted with the university are considered to have graduated for our purposes • Attrition: The student is not current or graduated and two or more semesters have elapsed since they last interacted with the university. • Probable Attrition: The student is not current or graduated and one semester has elapsed since they last interacted with the university. When developing a SOM for exploratory analysis it is often useful to consider modify the definition of the metric function 𝑑 so that the distance is invariant to certain parameters (so that the resulting map does not cluster on these parameters.). In this instance we do not cluster on the statuses above, to avoid having different behaviour profiles collapsed together because they result in attrition, a desirable outcome is to determine if there are different profiles associated with attrition. Scope Students analysed were undergraduate students that studied at least one unit on-site at the universities main campus between 2009 and 2014. Two data sourcing activities took place between one post semester 2014 and post semester 1 2013, in order to understand what movements across the map frequently occur. Results Map Overview An underlying behavioral demographic map was generated using the commercial package Viscovery FP:103 115 to perform the SOM analyses, the resulting hexagonally packed map contains 1200 nodes (approximately square at 33x35 nodes). A modified Ward clustering algorithm (Batagelj, 1988; Murtagh & Legendre, 2011) takes into account the values of each input vector point as well as their positioning on the map and sets the distance between non-adjacent nodes to infinity (ensuring the clusters are connected regions in the lattice). We have broken the resulting map into 8 clusters (Figure 1) which can be thought of as representing 8 profiles of students. The Ward algorithm can be used to divide the map into an arbitrary number of regions; eight regions were chosen to assist in socialising the map with users. With over a thousand parameters that can be viewed against the map, limiting the visualization to eight clusters assisted stakeholders in accessing information, creating meaning and developing insights from the map by generating an underlying easily-understood demographic profiles for non-technical users. Figure 1. Eight clusters determined the Ward algorithm When describing the clusters (or any subset of nodes) the mean value of parameters can be calculated and compared to the mean of the total map (or any other cluster) using a standard t-test. Categorical parameters such as country of birth are transformed into binary (0 or 1), in which case the mean on those parameters for any node or cluster is the proportion of students in that category; proportions are compared by considering the whether the Wilson intervals (Yan & Su, 2010) of the two values overlap within a given confidence. In this way regions can be described by parameters that make them ‘most different’ from the rest of the map. By way of an example some of the key demographic information for regions C1, C4 and C6 are given respectively in Tables 2, 3 and 4, along with examples of descriptions that were used in familiarising users with the map. Table 2. Domestic near-graduation student cluster Cluster Description C1 (n=14,995) Predominantly domestic students that have either graduated or are close to the end of their course in the most recent enrolled semester, slightly higher than average performance than other demographics. Parameter Mean / Cluster mean Confidence Proportion difference from input (mean is different from mean (%) mean of entire set) Citizenship is Australian 83.0% 15.0 >99.9% Percentage of units taken in first semester at 14.7% -16.7 >99.9% university are level 2 Percentage of units taken in first semester at 5.7% 27.4 >99.9% university are level 3 Percentage of course complete in final semester 66.6% 23.9 >99.9% Curtin Students Graduated 46.9% 43.5 >99.9% FP:104 116 Age at Course Start 22.98 4.3 >99.9% Course Weighted Average 64.06 9.0 >99.9% Table 3. International near-graduation student cluster Cluster Description C4 (n=8,434) International students that have either graduated or are close to the end of their course. They are distinct from C1 students in that they are typically taken a high number of level and level 3 units in their first semester of their course. Parameter Mean / Cluster mean Confidence Proportion difference from input (mean is different from mean (%) mean of entire set) Citizenship is Australian 5.4% -92.5% >99.9% Percentage of units taken in first semester at 50.4% 185.7% >99.9% university are level 2 Percentage of units taken in first semester at 11.1% 149.7% >99.9% university are level 3 Percentage of course complete in final semester 62.9% 14.5 >99.9% Curtin Students Graduated 56.8% 73.8% >99.9% Age at Course Start 21.8 -0.9% >99.9% Course Weighted Average 59.52 1.2% >99.9% Attendance mode External 0.02% -90.3% >99.9% Table 4. Domestic external study mode student cluster Cluster Description C6 (n=2,006) Domestic students that are significantly more likely to be taking an external study mode (to be in scope a student has to have taken at least one unit on campus, however the majority of external mode course have a small number of on campus components). On average students are older when commencing their course. Parameter Mean / Cluster mean Confidence Proportion difference from input (mean is different from mean (%) mean of entire set) Citizenship is Australian 94.9% 31.5 >99.9% Percentage of units taken in first semester at 19.9% 12.7% 99.5 university are level 2 Percentage of units taken in first semester at 2.5% -44.5% >99.9% university are level 3 Percentage of course complete in final semester 42.3% 21.3 >99.9% Curtin Students Graduated 21.2% -35.2 >99.9% Age at Course Start 30.39 37.9 >99.9% Course Weighted Average 52.27 -11.1% >99.9% Course: Attendance mode: 50.5% 1,950.3% >99.9% External For clarity we have compared only three of the eight clusters and selected a small number of parameters. In practice stakeholders are engaged in a series of workshops where considerable time is spent providing granular descriptions of each cluster, including areas of study, unit loads, past educational attempts, method of application and acceptance into courses, method of payment, and FP:105 117 other factors, in order to query the data model, test assumptions and understandings, and uncover or discover new relationships worthy of additional investigation or re-entering into the iterative model- building process. Risk Profiles: Typical vs A-typical Risk The SOM is not inherently a binary predictor (i.e. it doesn’t assign likelihood of a particular outcome). Instead, in order to define an ‘at risk’ profile we consider areas of the map where there are a large proportion of students with the status ‘attrition’. It is important to note that since a student can also either have the status ‘current’ or ‘probable attrition’ there are areas on the map where few students have status ‘attrition’ or ‘graduation’. In the SOM these areas are largely concentrated in the top left of the map and overlap segment C2 and C5 (see Fig. 2 and Fig.3 ). Fig 2. Current students: Colors represent the proportion of current students (blue represents 0% and red 100% of students) mapped to a node. Figure 3. Semesters into course: Colors represent the proportion of current students (blue represents 0% and red 100% of students) mapped to a node. Considering nodes where attrition is >40% identifies five connected regions larger than a single node, which we label R1 – R5, (Figure 4). It is reasonable to question whether occupying the same node as previous attrition students is indicative of likelihood of future attrition since by definition students that attrite are separated by two semesters from those that are current. To address this question we have taken two point-in-time data extracts (data slices or snapshots). We found that after 18 months the proportion of attrition for current students from these nodes is [32.01, 36.22] (99.9% CI) compared with [8.18, 8.81] (99.9% CI) for the entire map. FP:106 118 3 1 5 2 4 Figure 4. Attrition Rate: (Top) Colors represent the proportion of current students (blue represents 0% and red 100% of students) mapped to a node (Bottom) Five regions of the map with >= 40% attrition Of the five regions we consider region 1 to be associated with what might be classed “typical attrition” as it aligns with common hypotheses of many subject mater experts. The students in this region are domestic students; males slightly over represented) studying full-time in on-campus courses, and generally taking between 3 and 4 units a semester, which is typical for the entire population. They live slightly further from the university than average and access library and learning management systems less often. They are significantly more likely to have failed units in their first and last semesters. Interestingly, while unit evaluation surveys response rates are lower than average, those students that do respond generally do so positively. When we compared region to 2 to region 1 we found those students to be generally older, more likely to be female and studying part time either externally or online. They access library systems almost exclusively outside of Curtin. Despite similar risk profiles; (Attrition Proportion: R1: [65.9, 70.0] (99.9% C.I.) and R2: [55.6, 68.3] (99.9% C.I.)) the proportion of units failed differs significantly in students first semester. (R1: 42.1% and R2:27.6% T = 8.19). This suggests that resilience to poor performance in part time students is potentially lower, this insight is important for designing targeted interventions; for example, the threshold for reaching out to a such a student will need to be lower. Conclusions and Comments We have demonstrated the use of the Kohonen self-organizing map (SOM) technique for approaching the multifaceted retention and attrition challenges in higher education. The approach outlined here is innovative for two reasons; the first is the utility of the visual element in communicating results to stakeholders and decisions makers. In this hybrid approach, an exhaustive set of hypotheses are collected from stakeholders, exploratory analysis takes place with appropriately sourced big data and the results are iterated with stakeholders as well as data scientists. The iterative exploratory analysis process investigates a large number of hypotheses by supplying evidence that clearly supports or challenges the stakeholder’s assumptions and understandings, making easier the often difficult process of translating untested qualitative and heuristic knowledge into testable quantitative models, and onward to the creation of interventions, actions and policy. Secondly the approach is as broad as the sensor net of incoming and available data affords. Multiple FP:107 119 and varied domains of student behaviour can be analysed in a holistic manner. These behavioural domains range from a student’s engagement with university systems, attitude towards the quality of the pedagogy received, academic engagement and performance and a number of external factors. . The SOM approach has been shown to successfully identify multiple profiles of student attrition, creating new more nuanced risk profiles by separating behaviours originally thought to belong to a single profile as well as creating whole new classes of profiles SOM is not inherently a predictive technique in contrast with logistic models analysis and binary classifiers; but is effective for understanding the characteristics of a total population, identifying complex atypical clusters of behaviour and supplying other modelling approaches (e.g. linear regression, machine learning predictive techniques) with cohorts that have a high coherence among factors suitable further investigation. We have shown that SOM has potential to be combined with statistical and predictive analyses to form a complementary set of techniques for understanding the factors of retention and attrition for the purpose of developing new highly targeted interventions, actions and policy. Future research is planned to test the impact of the definition of attrition to see if the historic at-risk status based on the 2 semesters missing (we waited three semesters to analyse the data) is truly at- risk and whether the factors can lead to predictive estimations before students leave. References Batagelj, V. (1988). Generalized Ward and Related Clustering Problems Ward clustering problem. In Classification and Related Methods of Data Analysis (pp. 67–74). Crosling, G., Heagney, M., & Thomas, L. (2009). Improving student retention in higher education Improving Teaching and Learning. Australian Universities Review, 51(2), 9–18. De Freitas, S., Gibson, D., Du Plessis, C., Halloran, P., Williams, E., Ambrose, M., … Arnab, S. (2014). Foundations of dynamic learning analytics: Using university student data to increase retention. British Journal of Educational Technology. doi:10.1111/bjet.12212 Gibson, D., & de Freitas, S. (2015). Exploratory Analysis in Learning Analytics. Technology, Knowledge and Learning, (March), 1–15. doi:10.1007/s10758-015-9249-5 Gibson, D., & Jakl, P. (2013). Data challenges of leveraging a simulation to assess learning. West Lake Village, CA. Retrieved from http://www.curveshift.com/images/Gibson_Jakl_data_challenges.pdf Gibson, D., & Webb, M. E. (2015). Data science in educational assessment. Education and Information Technologies, June. doi:10.1007/s10639-015-9411-7 Jensen, U. (2011). Factors influencing student retention in higher education. Research & Evaluation. Retrieved from http://www.ksbe.edu/spi/pdfs/retention_brief.pdf Kohonen, T. (1990). The self-organizing map. Proceedings of the IEEE, 78. doi:10.1109/5.58325 Murtagh, F., & Legendre, P. (2011). Ward’s Hierarchical Clustering Method: Clustering Criterion and Agglomerative Algorithm. arXiv Preprint arXiv:1111.6285, (June), 20. Retrieved from http://arxiv.org/abs/1111.6285 Olsen, P. (2007). Staying the course : Retention and attrition in Australian universities Findings. Sydney. Retrieved from http://www.spre.com.au/download/AUIDFRetentionResultsFindings.pdf Shum, S. B. (2011). Learning Analytics. Yan, X., & Su, X. G. (2010). Stratified Wilson and Newcombe Confidence Intervals for Multiple Binomial Proportions. Statistics in Biopharmaceutical Research. doi:10.1198/sbr.2009.0049 Gibson, D.C., Ambrose, M., & Gardner, M. (2015). Self-organising maps and student retention: Understanding multi-faceted drivers. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:100Error! Bookmark not defined.- FP:108). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:108 120 New applications, new global audiences: Educators repurposing and reusing 3D virtual and immersive learning resources Sue Gregory Brent Gregory Denise Wood School of Education UNE Business School Learning and Teaching Services University of New England University of New England Central Queensland University Judy O’Connell Scott Grant Mathew Hillier School of Information Studies School of Languages, Literatures, Cultures Institute for Teaching and Learning Charles Sturt University and Linguistic, Monash University Innovation The University of Queensland Des Butler Yvonne Masters Frederick Stokes- Faculty of Law School of Education Thompson Queensland University of University of New England Learning & Teaching Unit Technology University of South Australia Marcus McDonald Sasha Nikolic David Ellis School of Health Sciences Engineering & Information Sciences School of Education RMIT University University of Wollongong Southern Cross University Tom Kerr Sarah de Freitas Helen Farley Faculty of Business & Economics Learning and Teaching Australian Digital Futures Institute Macquarie University Murdoch University University of Southern Queensland Stefan Schutt Jenny Sim Belma Gaukrodger Centre for Cultural Diversity & Medical Radiations Flexible Learning Wellbeing University of Auckland Nelson Marlborough Institute of Victoria University Technology Lisa Jacka Jo Doyle Phil Blyth School of Education Australian Digital Futures Institute Faculty of Medicine Southern Cross University University of Southern Queensland University of Otago Deborah Corder Torsten Reiners Dale Linegar School of Languages and Social Curtin Business School Oztron Science Curtin University Auckland University of Technology Merle Hearns Robert Cox Jay Jay Jegathesan School of Foundation Studies Faculty of Education, Science, Technology, School of Physics Manukau Institute of Technology and Mathematics The University of Western University of Canberra Australia Suku Sukunesan Kim Flintoff Leah Irving Business Systems and Design Curtin Teaching and Learning Curtin Teaching and Learning Swinburne University of Technology Curtin University Curtin University There continues to be strong interest among established, experienced academic users of 3D virtual environments for their sustained educational use. Consistent with global trends, they plan to further develop and optimise existing applications, reuse skills and experiences gained to develop new applications, and to share and reuse existing virtual resources. This is against a background of varied support from institutions, colleagues, students, funding bodies and also changing understanding and awareness of virtual environments and virtual reality by the general community as a result of consumer developments such as the popularity of multi-user online role playing amongst both children and adults, and the acquisition of technologies by companies with deeply entrenched technologies. At the same time, the ongoing development and availability of FP:109 121 new multiuser virtual environment platforms, associated peripherals and virtual reality technologies promise new and exciting opportunities for educators to collaborate with researchers on a global scale, while also exploring the affordances of these technologies for enhancing the learning outcomes for an increasingly diverse and distributed student population. Keywords: 3D virtual worlds, immersive learning, repurposing, reusing, virtual environments Introduction and background The Australian and New Zealand Virtual Worlds Working Group (VWWG) was established in 2009. Since then, members of the VWWG have written papers for the ascilite conference providing an update on the educational use of virtual worlds across the two countries. This year, following similar interest globally, and in keeping with the New Media Consortium (NMC)’s (Johnson et al., 2015) anticipated growth in the use of flipped classroom approaches and the educational applications of wearable computers, ‘Makerspaces’ and the ‘Internet of Things’, Australian educators are beginning to explore the potential of repurposing and reusing 3D virtual and immersive learning resources to harness augmented spaces. A survey was sent to group members and 30 members, from 24 different institutions across Australia and New Zealand, provided feedback in relation to their current use of 3D virtual and immersive learning environments and, in particular, how they are repurposing and reusing learning resources, including objects, environments and pedagogical approaches. Members of the VWWG provided several standout points to consider. A wide variety of applications were reported as being used through 3D virtual immersive environments across a range of disciplines. There is also a broadened definition of virtual worlds to now encompass 3D virtual environments that include some platforms not traditionally seen to fit the virtual world category such as SketchUp and Google Earth. The reduction in cost of additive technologies and use of other technologies such as 3D printers has broadened the applications of virtual environments through a combination and convergence of these technologies. There is also increasing focus on finding ways, formats and platforms that allow greater sharing of resources. The limitations of some platforms (e.g. hard to use/develop technically, too costly, closed systems, etc.) are pushing academics to explore alternative platforms. In the past, there has been a lack of easily transferable virtual resources, limiting sharing of pedagogical designs and virtual resource development skills across platforms. With the anticipated continued growth in the open education resource movement, finding ways to collaborate and share resources and knowledge globally will be an important goal if educators are to more effectively engage learners in the use of these environments in ways that enhance learning, teaching and assessment outcomes in a sustainable manner. Literature Review Immersive environments have provided instructional, autonomous and collaborative capabilities to support the creation of educational materials and are best grounded in pedagogy rather than being solely driven by the latest technology (Price, 2011). The pedagogical principles underpinning adoption have applied equally to virtual and immersive worlds, single and multi-player environments and related virtual technologies. Identifying the desired learning outcomes is fundamental in shaping effective learning designs for virtual spaces, whether they utilise autonomous learning activities, teacher led activities or participatory group experiences. Since the mid 1990s, virtual worlds have supported a diverse range of activities, including: experiential learning (Jarmon 2008; De Mers, 2012); student perceptions of learning in virtual worlds (Lowe & Clarke, 2008; Huber & Blount, 2014); engagement with specific disciplinary material (Herold, 2009; Lee, 2009; Pereira et al., 2009; Beebe, 2010; Teoh, 2012); supported training and role-play (Gregory et al., 2011; Gregory & Masters, 2012a, 2012b; Neuendorf & Simpson, 2010; Slator & Chaput, 1996) or introduced multi- player 3D games used to stimulate debates and discussion between peers on authentic or complex topics (Brom, Sisler & Slavik, 2009). Drawing on an extensive review of research and field notes from virtual learning environments, Jarmon (2012) found that 3-D virtual environments, in whatever form, would be increasingly used as knowledge and social interaction management tools in the foreseeable future. FP:110 122 The modality of game-based learning is an emerging area of influence with approaches available to create dynamic pedagogical agents of intrinsic motivation, mediated communication, supported self- representation, sensory abilities or situational context responses (Leung, Virwaney, Lin, Armstrong & Dubbelboer, 2013). The use of virtual worlds and mixed reality, coupled with game-based mechanics, is bringing new opportunities to 3D immersive environments (Callaghan et al., 2013; Charles et al., 2011) with game-based learning activities able to drive experiential, diagnostic and role-play learning activities (Toro-Troconis, et al., 2012). Virtual worlds provide opportunities for grounded experiences situated in understanding both practices and content as learners experience the consequences of actions based on inquiry and/or gaming contexts (Vrasidas & Solomou, 2013). Virtual environments can bring geographically distant students and staff together to provide a connection with the main campus. Universities around the world have created thousands of satellite campuses, both domestically and internationally, with the promise that distance is no barrier in obtaining a high quality education (Leung & Waters, 2013; Waters & Leung, 2013). Eaton et al. (2011), provide one such example, linking 16 campuses with 200,000 students and 7,500 staff using Second Life. Despite continued optimism by educators and researchers across disciplines who see value in virtual worlds due to their immersive nature and global reach, a range of challenges continue to hamper their wider use. These challenges include the complexity of technology development, forced updates by vendors, ongoing costs, and a reliance on grant fixed term funding. Vendor and client- side system functionality and structures are still plagued by high levels of uncertainty in development cycles, as well as being complex and difficult to operate for non-technical users (Gupta et al., 2014). Educators need to reuse skills and experiences and share strategies and resources in order to remain responsive to the still emerging nature of 3D immersive virtual environments. It has been argued that the community of practice around virtual worlds in education had done much along this rd path and that now is an opportune time to work toward the 3 generation of virtual world tools (McDonald, Gregory, Farley, Harlim, Sim, & Newman, 2014). McDonald et al. demonstrated that mitigating many of the issues stated above would allow virtual worlds to continue up Gartner’s Slope of Enlightenment. This has indeed been the case in moving from the ‘Trough of Disillusionment’ in 2013 (Lowendahl, 2013) to the ‘Slope of Enlightenment’ in 2014 (Lowendahl, 2014) and then towards the ‘Plateau of Productivity’ in 2015 (Lowendahl, 2015). Rapid growth in consumer technologies, wearable computing and the use of technologies to facilitate creativity and innovation through the collaborative development of digital artefacts (‘makerspaces’), combined with the on-going rapid expansion of game types, platforms, experiences and media-convergence, compels educators to address the challenges, opportunities and potential of 3D virtual environments for more effective use of blended learning approaches to facilitate flexible learning in augmented spaces (Johnson et al., 2015). Method Members of the VWWG participated in an online survey focussed on changing audiences and applications as well as the repurposing and reuse of 3D virtual and immersive learning resources. Of the 183 members invited, a small sample of 30 (16%) completed the survey. The small sample size of respondents is due to the specialised nature of this group. Demographics, including discipline and audiences taught (student, staff or other) were also collected. The survey data was manually coded into themes and then the NMC Report (Johnson et al., 2015) themes provided a lens through which member responses, relating to how they are repurposing and reusing using 3D virtual and immersive learning resources, could be analysed. These themes include: important developments in educational technology in higher education; significant challenges impeding technology adoption in higher education; and key trends in accelerating technology adoption in higher education. The findings from the study are reported in the following section. Findings To provide an overview of how the members of the VWWG are using 3D virtual and immersive learning resources, respondents were asked to provide information on the ways in which they have been using these spaces (see Figure 1), and the disciplines of use (see Figure 2). Members were able to nominate more than one way in which they were using 3D virtual and immersive technologies FP:111 123 (see Figure 1). Research activities undertaken by educators were the main ways in which these spaces were reported to be used by members of the VWWG, closely followed by simulations, machinima, role-plays and presentations. 3% 2% 2% 1% Research Simulations 5% 15% Machinima Role-plays 5% 5% Presentations Virtual tours 14% Discussions Game design 6% Virtual lectures Virtual guest lectures 8% 13% Career planning Laboratory experiments 10% 11% Other Creative arts Figure 1: Discipline and/or non-teaching areas being used Ways in which 3D virtual and immersive environments are being used To provide context, members were asked ways in which 3D and immersive environments were being used at their institutions with respondents reporting a variety of ways. These responses are clustered into four main themes including: the different types of learning and teaching pedagogies incorporated into their learning, teaching and/or research spaces; the various types of learning and teaching activities undertaken; the types of spaces created; and how they were used to interact with others. Table 1 provides an outline of activities within each theme. Table 1: Overview of ways VWWG members use 3D immersive environments Pedagogical approaches used Types of learning teaching activities Creation of spaces/ teaching resources Interaction with others Transformative, Teaching, training, discussion of Designing, Meetings, remote experiential and learning materials, presentations, demonstration of tutorials, community of contextual learning, assessment, role play, scenario business models, practice, orientation, problem solving, practice, treasure hunts, web creating elements of resource centre, game-based quests, building, scripting, authentic learning that advertising, learning, task- simulations, laboratory procedures, enhances situated international events, based learning, combining histories with actual site learning, collaborating presentations, teaching integration of reproductions, self and peer review to create machinima, across campuses, gamification of performance, rapid prototyping, developing resources career development, phobia modelling and physiological and interactive conferences, response tracking activities socialising, research In relation to the discipline (Figure 2), members of the VWWG reported that they were using 3D virtual and immersive learning spaces (more than one discipline could be nominated) in education (most often reported), health and business. Other responses included medicine, statistics, climate change, health and safety training, multimedia, film, information systems, orientation and engineering. The disciplines in which members reported that they least use these spaces, including “other”, were history, law, visual and performing arts, information technology, tourism and pharmacy, with no responses from hospitality, indicating that it was not being used by any of the current members of the VWWG who completed the survey. FP:112 124 2% Education Other 2% 2% 2% 3% 2% Health Business 25% Science Art 3% 3% Engineering Languages 3% 5% Sociology Architecture 5% Construction Social and Behavioural Studies History Law 5% 16% Visual and Performing Arts Tourism 6% 6% 11% Pharmacy Hospitality Figure 2: Disciplines in which VWWG members are teaching using 3D immersive virtual environments Teaching audiences Respondents were also asked to indicate the number of staff, students or other (which included users outside their institution) who were their teaching audience/s. Table 2 provides an overview, indicating that the largest audience was their students. Members were also asked to indicate if their teaching audiences had changed from the past, with 31% indicating that they had. The majority, 69%, stated that they were still using 3D virtual and immersive spaces the same as they had in the past. Table 2: Teaching audience and type of variation Type of teaching Percentage Teaching audience different from the Percentag audience past e Staff 15% Yes 31% Students 59% No 69% Other 26% As indicated in Table 2, the audience reported by the majority of respondents is students, followed by colleagues, then professional staff through collegiate and global connections facilitated by specific projects. Students enrolled in courses utilising 3D and immersive technologies include a mixture of undergraduate and postgraduates, including PhD candidates, as well as those studying at TAFE, or pathway students who are undertaking enabling courses. There has been a focus in some institutions on offering training for workers within industry groups (for example in the mining and construction sector for health and safety training). Change of audience VWWG members who indicated that their teaching audiences had changed in the past year were asked to explain why this change had occurred. Respondents stated that they were now doing things differently, with little work with students directly, their research had been completed, or the uptake from other staff had not occurred. However, others felt their audiences had expanded because the use of 3D virtual worlds was no longer limited to communication or visiting places. These virtual environments now offer enhanced interactivity and authenticity. Consistent with NMC report findings (Johnson et al., 2015), flipped classrooms and blended learning are being used more extensively enabling a more flexible approach to learning and teaching. Several other members stated that their audiences had extended in reach beyond their normal disciplinary field. Others reported the use of these environments to facilitate community engagement, such as projects involving students with disabilities and those with chronic illnesses, seeking to enhance the social and communication skills of these groups. Repurposing or reusing 3D virtual and immersive learning objects and environments VWWG members were also asked to indicate how they were repurposing or reusing 3D virtual and immersive environments. Their responses were able to be categorised using NMC 2015 themes (Johnson et al., 2015). ‘Makerspaces’ FP:113 125 In a design and technology education context, the use of SketchUp as a virtual environment has not only enabled the visualisation of designs in a 3D form, but also in combination with other geographical technologies such as Google Earth, to develop and model designs. A virtual 3D modelling capability is cost effective as certain design problems can be modeled virtually with no resources being used. In recent years, the reduction in the cost of 3D additive and subtractive manufacturing technologies has enabled designers to take that next step in the design process and realise their design prototypes and has made these technologies, such as 3D printers, very accessible. This growing area of interest is again consistent with the NMC report’s predictions that the use of technologies to facilitate innovation and creative skills through ‘Makerspace’ environments are likely to gain greater traction within the coming year (Johnson et al., 2015). Cross-institutional collaboration and open education resources Collaboratively, Australian and New Zealand universities’ colleagues are exploring ways in which to share resources. As the textbooks and curriculum of the New Zealand students are slightly different from those in Australia, members are looking to re-purpose existing virtual resources for use with other institutions’ materials, as well as make their pedagogical materials available for use. Resources have been developed for creating, sharing and storing ‘learning objects’. This is in line with NMCs long-term trend of increasing cross-institutional collaboration (Johnson et al., 2015, p. 2). 3D models off the rack are often purchased when possible. For construction, this is possible, but much more difficult in specialised fields such as pharmaceutical science. Many members access material in Second Life that has been created by other colleagues around the world. There is a vast resource pool which is easy to find and use rather than resorting to continually creating new artefacts. Using these tools makes it easier for students to understand the systems when they see them in operation. Other members have created their own resources to share across various virtual worlds. Often, the resources/objects/environments are completely self-contained, sometimes including the use of Heads Up Display (HUD). Many objects purchased from other creators have come with limited IP rights that are manifest in restrictive permissions assigned to 3D objects, raising barriers to sharing. An alternative is to recreate each object from scratch to ensure that there are no IP right issues, however this is labour intensive and inefficient. But, at the same time, this is the only alternative in some cases. Many members report that they are not sharing their simulation work even though general 3D virtual spaces have been created from existing resources and many are utilising open and free objects within Second Life to construct larger builds. Assets created within Second Life for clinical education and role-playing spaces have, to some degree, been packed up and then reused for projects of similar need. However this has proved difficult and inefficient. This is especially so when virtual land has been unfunded or closed. Builds using open platforms (such as OpenSim) rather than in closed eco systems (such as Second Life) allow packing of objects in inventory archive (IAR) files or whole sims in OpenSim archive (OAR) files, which are then are placed online for others to download and use. Increasing cross-institutional collaboration and extending sharing of resources and pedagogical practices are similarly identified in the NMC report (Johnson et al., 2015) as global trends, which pose significant challenges, hence the report’s prediction that achievement of such goals may still be five or more years away. 3D scanned objects can be created for reuse; for example, authentic spaces can recreate the shape and surface markings of an Egyptian tomb so that scanned objects can be placed within it, providing further context for excavation techniques and object descriptions. Teaching complex thinking and creative problem solving The NMC 2015 report (Johnson et al., 2015) suggests that the teaching of complex thinking will become increasingly important in the next two-three years. Although the NMC report describes complex thinking as beyond creative problem solving and decision making, suggesting complex thinking will require graduates who are able to manage ‘big data’ and be able to take advantage of the latest tools and techniques to solve complex problems and influence systemic change, several VWWG members report using 3D virtual and immersive environments to foster critical thinking, creative problem solving and clinical decision making. Multiple sources of information such as patient case history, blood test results, ECG, radiology information (such as MRI, CT or ultrasound images, etc) are being used for clinical decision-making. Students make informed decisions by selecting the correct objects in the right sequence. The clinical tutor is available to assess/challenge student knowledge and understanding. Students are located all across the continent so the virtual meeting space is ideal. FP:114 126 Machinima is being utilised to support learning in areas as diverse as law, accounting, pre-service teacher education and climate-related decision making. Machinima, using techniques akin to film or television shows (including detailed set dressing, multiple camera angles and post production sound effects), can be utilised to depict complex and engaging narratives for learning. When combined with simulated documents they are capable of creating immersive environments which is an important success factor in online and technology-based learning. Students are inspired to learn by such environments because they are involved in authentic tasks such as negotiation, interpretation of documents and evaluation of evidence, and can appreciate the relevance of what they are studying to their future careers. Moreover, unlike clinical programs, such learning environments are scalable and can offer the same realistic learning experiences for large cohorts of students, regardless of mode of study. It is a cost effective alternative to real world video for educators in the context of limited financial support for development of multimedia resources. Machinima produced by students as evidence of learning can be curated and used as exemplars or resources. Machinima tasks have a real world focus with activities that closely replicate those undertaken by professionals in practice. Existing resources are also being reused for language learning and teaching purposes in Second Life. Objects can be adapted for language practice. Second Life still has the largest community of language learners and volunteers. The use of VWWGs for language learning provides students with the opportunity to communicate and collaborate with peers globally while also fostering their ability to use language in ways that support critical thinking in authentic contexts. Convergence of wearable computers and consumer technologies The NMC report (Johnson et al., 2015) predicts that wearable technology will see significant growth in the coming year and will increasingly be applied in higher education. Several VWWG respondents reported that they already utilise wearable technology in their teaching and research. In particular, the use of the Oculus Rift has been used to immerse students and/or staff during training and professional development sessions. The current trend in teaching in 3D immersive virtual environments has been through the integration of gamification; i.e. the distinction of gamification and serious gaming and how this can be represented in virtual 3D environments. Serious gaming enables the modeling of complex bodily functions and for players to explore within the confines of game mechanics. Students appreciate a well-designed simulation that is both fun and also assists them to build knowledge in an assessable area. Game design is important when gamifying online interactions; however, finding the best solution to encourage site exploration and deep learning is difficult. By using game engines, many assets created outside of those environments can be easily shared. The languages used to drive most 3D engines are similar if not the same. 3D immersive virtual environments have been used for refinement via the introduction of a few new mechanisms for engagement. Consideration of how the spaces are revitalised to allow more independent engagement whilst still providing meaningful scaffolding and feedback via automated mechanisms has been explored. Many existing virtual worlds have the potential to be converted to be more game-like as a simulation. NMC reported the relevance of gamifying learning for students (Johnson, et al., 2015). Teacher education – transference of skills across platforms The virtual world of Twinity has been used to ascertain whether skills that are learned in Second Life and activities that had been used there could be transferred to another virtual world. In terms of the social presence of virtual worlds that helps to support first year transition, Twinity was very successful. Part of what has been tested was the difference between synchronous meetings in a virtual world and those held via webinar software with students. As both were done via typed chat rather than voice, there was a distinct similarity in method of learning and teaching, but the webinar did not have the same visual impact as the virtual world. Students commented both in chat and evaluations about the positive interactivity of Twinity. Sim-on-a-Stick has been used in primary schools to demonstrate to pre-service teachers that it was possible to use the technology in the school environment. In so doing, sharing of objects and environments between primary school students and schools takes place. Primary school builds were taken into the virtual world to create a learning space for pre-service teachers so they could see what was possible for children to produce and learn to build. Research FP:115 127 Much research has been undertaken in 3D immersive environments and here we provide just some examples of what members of the VWWG have used them for. One research study relates to the use of virtual environments by young people who have Autism Spectrum Disorder, particularly in terms of developing their socialisation skills. The Virtual Lab is premised on developing both social skills and personal interests in technology, so the platforms used vary considerably. The most common 3D immersive environment used is Minecraft, especially by the younger groups, with older groups using Unity 3D, Unreal or other 3D game engines, as well as specialist game creation tools such as Sploder, Game Maker and RPG maker. Lab mentors (who are programmers and designers) help participants create their own games and develop both social and coding skills. 3D virtual worlds are used as learning tools for improving socialisation and IT skills rather than for their own sake as teaching environments. Some of the software being used, such as iSee, does not provide sharable objects with the exception of maps, which can be shared. This is the concept of combining entrenched technology (e.g. webcam conferencing) with more recent technology (e.g. 3D virtual environments). This allows users to obtain a greater sensory experience by feeling more engaged with other participants (Safaei et al., 2014). Research in the area of intercultural competence and study abroad suggests that students benefit more if they have prior experiential learning to raise awareness of their world-views and identities. Second Life is proving to be a very useful tool for this as it challenges assumptions and stereotypes, highlighting ways of communicating and developing resilience, critical reflection and deep learning. Research is the backbone of the NMC report (Johnson et al., 2015) and the VWWG community continue researching to ensure that they have the evidence to support their findings. Challenges and how they have been overcome The NMC report (Johnson et al., 2015) documents several challenges facing educators over the coming five years and beyond. These challenges include blending formal and informal learning and adapting to the convergence of a range of technologies, digital literacy, teaching complex thinking and competing models of education. Several of these challenges are evident in the responses from the VWWG community documented in this section. One of the major challenges reported by members has been the cost of purchasing and developing the 3D immersive virtual environments and keeping up with the shifting landscape. These challenges have not yet been overcome in all institutions. With some institutions, central support and technical problems remain the most significant problem and without grant money, development is almost impossible. The level of digital literacy of students remains a significant problem also, making off- campus use of 3D immersive virtual environments more work, as different pedagogical approaches require exploration. Access for students remains a key issue where not all students have quality Internet access. At this stage it is not possible to make virtual world engagement compulsory in courses for that reason. However, some participation is compulsory where computer technology can be guaranteed, such as for on-campus students or students outside the campus who have the required technology. Software based on a Cube 2 engine, and developed by an independent group of educators has also been used, though development and support for this software has been haphazard at best. The limitations of the program often remain unaddressed, despite a large user community. These limitations include the lack of a truly web-based platform for delivery. Other platforms have been explored as a means of achieving the same outcome, such as Minecraft, but the compromises required, including sacrificing authentic surface-mapping for game-play, seem difficult to overcome. General recognition that virtual worlds have a place in higher education has been a challenge for members of the VWWG. Virtual world affordances and advantages have not been well articulated. There is also a general impression that virtual worlds (as associated with Second Life) are ‘done’ and ‘last year’s news’. This may not have been helped by the extreme over hyping of virtual worlds. There is still a perception that virtual worlds are in the ‘Trough of Disillusionment’ to the point they are a ‘dirty word’ in some areas. Last year’s move up the Slope of Enlightenment (Lowendahl, 2014) does not seem to have filtered through and bolstered popular perception of virtual worlds in education. There is currently a lack of recognition by university management in wanting to fund any work in this area. One of the initial challenges was skepticism about the value of using virtual worlds. However, once used for a while, people were able to see why they were beneficial. There has been FP:116 128 a lack of support from many institutions and pre-conceived ideas from students and staff about the value of virtual worlds in relation to teaching and learning. Sometimes this has included constant restructuring and downsizing, which made it difficult to build alliances and partnerships with colleagues in the area of education technology innovation. This has been overcome by working largely outside institutions. External scripters and modellers have been hired to do a lot of work to develop some virtual environments. The costs involved are often high and have limited what can be done. To overcome this, members have undertaken to learn as much about these areas as possible so that there is flexibility to continually develop new ideas, new projects and to optimise current virtual resources. Some items that could be used as part of learning and teaching needs can be purchased ready- made from the Second Life market, but they are often only able to fulfill part of specific needs and therefore need to be modified. Sometimes these objects can lack the permissions necessary to carry out modifications. These types of items also cannot be transferred to other virtual world platforms such as OpenSim. More often than not, members have developed these items themselves, or where funding is available, people have been hired to develop them. Barriers and/or enablers for sharing and/or reuse of 3D virtual world objects/environments Familiarity with a virtual environment can be both an enabler and barrier for object sharing. Those who use the same 3D engines are more likely to do more sharing than developers using a different platform. Object formats, such as those used in 3D animation programs, need to be standardised in the same way as audio, graphic and video files. The following list identifies enablers and barriers for sharing or reusing 3D virtual world objects and/or environments. The list of barriers is much more substantive than the enablers. Enablers Members valued that free objects are available in virtual worlds such as Second Life and OpenSim and that creators of these objects are willing to share. Many objects purchased in these environments, either for free or for a small fee, are provided with permission to enable these objects to be reused or modified. Members also value world-editing software that enables cut-and-paste operations or 3D volumetric object creation between worlds. These digital assets can also be exported easily and saved as single files, including entire worlds. The availability of more open systems providing mechanisms for sharing objects within and beyond given grids or networks is valued. The virtual world community collaborates and shares common teaching and learning tools, often due to being open source. Mailing lists alert educators as to who may have objects available for reuse. Being part of the virtual world community helps educators with regards to sharing virtual world objects, within networks such as the VWWG. Communities of practice have been established and connected outside virtual worlds, such as via blogs and social media, and even attending conferences in person is highly valued. An increase in the quantity and quality of research completed and reported by virtual world educators is an enabler, and finding someone who is willing to mentor has always been valued by VWWG members. One of the biggest enablers is the increasing power of mobile technologies in making virtual worlds accessible to more people than ever before. This makes virtual world education highly mobile/portable and accessible. Barriers Unfortunately, many barriers remain to repurposing and reusing 3D virtual objects and environments. The reasons are myriad and many are presented here. Potential users are often unaware of what is available to modify and reuse. Some users are still unwilling to share their objects and/or environments. Many objects are of poor quality or are unable to be modified. While many ready-made items may be suitable for use in educational scenarios, they often lack the rights to be transferred to other platforms or even shared with other educators on the same platform. Often creators who offer their items for sale in, for example, Second Life, are not willing to customise their items for more focused educational use or to allow transfer to other platforms. This rigidity means that items cannot be used and have to be created from scratch. Sometimes, when a world/space disappears, the assets go with it because the user was unable to save a copy from the designer. FP:117 129 Time was reported by many members as a major barrier, such as a lack of time to search available resources in virtual worlds, a lack of time to train staff in the practice of using the virtual world. Also, there is still a lack of common infrastructure, language and repositories for sharing. Some members also felt that being able to ‘sell’ things in Second Life for ‘real money’ may actually provide a barrier to sharing. Facilitating cross-institutional sharing of resources are considered more challenging barriers to overcome in the longer term, anticipating this process may take more than five years to resolve (Johnson et al., 2015). Integration of scripts from different objects has been seen as a barrier. Scripts on objects function well within specific objects, but shared communication between objects relies on overall similar communication strategies. The major issue with the virtual world of Second Life is it is a closed system, i.e. objects are not likely to be exported to other systems. Therefore, more developments have a single purpose and functionality. Scripts could be used in other objects, however it was not straightforward and management is very limited. Without an established user-base or support community, development of a 3D immersive world can easily get bogged down in the need to solve multiple small problems. Having an easy way to distribute the world online can quickly indicate whether it was truly viable as a means of doing effective online learning. However, it was felt that both closed (Second Life) and open (OpenSim) virtual worlds still require considerable technical skill to use/build and so are beyond the practical reach of many academics without investing considerable time in learning the technical details. This is a medium-term priority consistent with the NMC report’s anticipated more widespread adoption and acceptance of the sharing of open resources within the next three to four years. There is general public perception that virtual words are predominantly for gaming rather than education. Some members felt that students should be encouraged to develop virtual worlds using gaming techniques. Getting talented developers has always been seen as a barrier and users need to identify others with sufficient levels of skill to undertake the various tasks individuals have in mind. This has been difficult, both from the perspective of availability and interest, and also cost. Institutional barriers have been discussed for many years. Members are still frustrated that many of the barriers have not been removed over time. These continuing barriers include the cost to the average consumer in terms of time and money; inappropriate infrastructure by having only one lab in the whole institution set up to run virtual worlds; security/firewall issues; locked down hardware/systems on campus; an ‘off the shelf’ policy from the management of IT support services who just want to ‘buy the license’ to solve pedagogical/technical/ procedural issues; centralised training, knowledge and financial support; lack of funding and foresight; and an inability to think outside of the box. One major institutional barrier reported by many respondents was that it was difficult to get virtual worlds accepted alongside other online learning environments within their institutions. Institution level understanding and support to develop ‘mainstream’ approaches was required. It was also difficult to get other faculty members involved and obtaining the continuing support of management. Recognition and support for the specific values/affordances of virtual worlds were required. The NMC report describes the challenge of providing appropriate reward and recognition for educators undertaking innovative learning and teaching as one of the ‘wicked problems’ on the horizon to be addressed in the longer term. Some respondents felt that promoting machinima as an alternative to traditional videos for presenting messages and aiding decision-making was a way of overcoming many of the barriers to using virtual worlds within their institutions. Many academics could use machinima as an alternate method to using a virtual world with their students yet still provide the immersive experience that these 3D environments offer. By the use of machinima, convincing some colleagues of the value of such learning environments, when they have personal ideologies that do not embrace such methods, may be easier. Conclusions A concerted national push to raise the profile of the 3D immersive virtual world use in tertiary education is needed - it appears that knowledge and awareness of the potential is not yet being realised despite the recognition by Gartner and a move to the Slope of Enlightenment (Lowendahl, FP:118 130 2015). There are new hardware and software platforms being developed constantly that provide new and potentially more flexible environments in which educators can create even richer and more streamlined educational experiences. With the popularity of 3D virtual environment platforms for younger users, and more importantly, the growing recognition by their parents of the potential uses of 3D virtual environments, the future should see growing numbers of tertiary students who have literally grown up using virtual worlds of one kind or another. As existing platforms are refined and new ones developed based on the experience of developing and using existing platforms, it will become easier and easier for non-expert educators to develop the kinds of environments and activities suitable for their specific teaching needs. The reputation of virtual worlds in general appears to be improving over time as a diverse range of platforms and uses are being developed that are attracting a more mainstream audience. Despite the ups and downs of virtual worlds in education over the last few years, they continue to be used in a variety of ways across a range of disciplines and research into their use for a whole range of end purposes has continued unabated until now. The results of the survey indicate there are many changes in the ways in which members are now using virtual worlds for learning and teaching. Within the context of higher education, the use of virtual worlds is still a relatively new and emerging area and the results of the survey indicate a continually shifting and settling within pedagogical practices, institutional support, academic and student attitude, perceived effort versus result and the affordances of specific platforms. Virtual worlds are part of the technology in education continuum, however there remains an ongoing persistence and resilience by educators integrating virtual worlds in teaching practices, despite the challenges. In keeping with the NMC 2015 reported themes, members of the VWWG felt that development/reuse/repurpose of virtual environments in higher education are important, there are still significant challenges impeding technology adoption and have outlined key trends in accelerating technology adoption in higher education. Further data needs to be collected internationally to expand on and confirm these results. References Beebe, M. (2010). Virtual learning. Creative Nursing, (3), 99. Brom, C., Šisler, V., & Slavík, R. (2010). Implementing digital game-based learning in schools: augmented learning environment of “Europe 2045.” Multimedia Systems, 16(1), 23–41. Callaghan, M. J., McCusker, K., Losada, J. L., Harkin, J., & Wilson, S. (2013). Using game-based learning in virtual worlds to teach electronic and electrical engineering. Industrial Informatics, IEEE Transactions on, 9(1), 575–584. Charles, D., Charles, T., McNeill, M., Bustard, D., Black, M. (2011). Game-based feedback for educational multi-user virtual environments. British Journal Of Educational Technology, 42(4), 638–654. De Mers, M. N. (2012). Second Life as a surrogate for experiential learning. In M. Thomas (Ed.), Design, implementation, and evaluation of virtual learning environments, 77–92. Eaton, L. J., Guerra, M., Corliss, S., & Jarmon, L., (2011). A statewide university system (16 campuses) creates collaborative learning communities in Second Life, Educational Media International, 48, 43–53. Gregory, S., Dalgarno, B., Campbell, M., Reiners, T., Knox, V., & Masters, Y. (2011). Changing directions through VirtualPREX: Engaging pre-service teachers in virtual professional experience. In G. Williams, N. Brown, M. Pittard, & B. Cleland (Eds.), Proceedings of the 28th ASCILITE Conference. Hobart, Australia: University of Tasmania. Retrieved from http://www.ascilite.org.au/conferences/hobart11/procs/GregoryS-full.pdf. Gregory, S., & Masters, Y. (2012a). Comparison of role-plays in a virtual world. In P. Jerry, Y. Masters, & N. Tavares-Jones (Eds.), Utopia and a garden party: Experiential learning in virtual worlds, 45–56. Oxford, United Kingdom: Inter-Disciplinary Press. Gregory, S., & Masters, Y. (2012b). Real thinking with virtual hats: A role-playing activity for pre- service teachers in Second Life. Australasian Journal of Educational Technology, 28(Special issue, 3), 420–440. Gupta, M., Jin, S., Sanders, G. L., Sherman, B. A., & Simha, A. (2014). Virtual Worlds: A Review. In M. Khosrow-Pour (Ed.), Inventive approaches for technology integration and information resources management, 162–212. Hershey, PA. Herold, D. K. (2009). Virtual Education: Teaching media studies in Second Life. Journal of Virtual Worlds Research, 2(1). https://journals.tdl.org/jvwr/article/view/380/454. FP:119 131 Huber, E., & Blount, Y. (2014). Second life for assessing postgraduate learning: Student Perspectives. In S. Kennedy-Clark, K. Everett, & P. Wheeler (Eds.) Cases on the assessment of scenario and game-based virtual worlds in higher education, 36–73. Hershey, PA. Jarmon L., Traphagan T. & Mayrath M. (2008). Understanding project-based learning in Second Life with a pedagogy, training, and assessment trio. Educational Media International, 45(3), 157–176. Jarmon, L. (2012). Homo Virtualis: Virtual worlds, learning, and an ecology of embodied interaction. In M. Thomas (Ed.), Design, implementation, and evaluation of virtual learning environments, 58–76. Hershey, PA. Johnson, L., Adams Becker, S., Estrada, V., & Freeman, A. (2015). The NMC horizon report: 2015 higher education edition. Austin, Texas: The New Media Consortium. Lee, P. D. (2009). Using Second Life to teach operations management. Journal of Virtual Worlds Research, 2(1), https://journals.tdl.org/jvwr/article/view/431/464. Leung, S., Virwaney, S., Lin, F., Armstrong, A., & Dubbelboer, A. (2013). TSI-enhanced pedagogical agents to engage learners in virtual worlds. International Journal of Distance Education Technologies (IJDET), 11(1), 1–13. Leung, M. H., & Waters, J. (2013). British degrees made in Hong Kong: an enquiry into the role of space and place in transnational education. Asia Pacific Education Review, 14, 43–53. Lowe, C., & Clark, M. A. (2008). Student perceptions of learning in a virtual world. Proceedings of the 24th Annual Conference on Distance Teaching and Learning. Retrieved from http://www.uwex.edu/disted/conference/Resource_library/proceedings/08_13442.pdf. Lowendahl, J-M. (2013). Hype Cycle for Education, 2013. Report G00251104, 25 July. Gartner Inc, http://www.frankiejackson.net/uploads/2/1/1/6/21168820/hype_cycle_for_education_201_251104 .pdf. Lowendahl, J-M. (2014). Hype Cycle for Education, 2014. Report G00263196, 23 July. Gartner Inc, http://www85.homepage.villanova.edu/timothy.ay/TOP3090/hype_cycle_for_education.pdf. Lowendahl, J-M. (2015). Hype Cycle for Education, 2015 Report G00277499, 8 July, Gartner Inc McDonald, M., Gregory, S., Farley, H., Harlim, J., Sim, J., & Newman, C. (2014). Coming of the third wave: A move toward best practice, user defined tools and mainstream integration for virtual worlds in education. In B. Hegarty, J. McDonald, & S.-K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014 (pp. 161– 170). Neuendorf, P., & Simpson, C. (2010). Redesigning role-plays for a virtual world in health education. In Z. A. et Al (Ed.), Proceedings of Global Learn Asia Pacific (pp. 3546–3550). AACE. Pereira, A. M., Martins, P., Morgado, L., & Fonseca, B. (2009). A virtual environment study in entrepreneurship education of young children. Journal of Virtual Worlds Research, 2(1), https://journals.tdl.org/jvwr/article/view/406/459. Price, C. (2011). The path between pedagogy and technology: Establishing a theoretical basis for the development of educational game environments. In Gaming and Simulations: Concepts, Methodologies, Tools and Applications (pp. 273–295). Hershey, PA: doi:10.4018/978-1-60960- 195-9.ch117. Safaei, F., Pourashraf, P., & Franklin, D. (2014). Large-scale immersive video conferencing by altering video quality and distribution based on the virtual context, Communications Magazine, IEEE, 52, 66–72. Slator, B., M., & Chaput, H. (1996). Learning by learning roles: A virtual role-playing environment for tutoring. In C. Frasson, G. Gauthier, & A. Lesgold (Eds.), Third international conference on intelligent tutoring systems (Vol. 1086, pp. 668–676). Berlin, German: Springer-Verlag. Teoh, J. (2012). Pre-service teachers in second life: Potentials of simulations. Journal of Educational Technology Systems, (4), 415. Toro-Troconis, M., Roberts, N., Smith, S., & Partridge, M. (2012). Students’ perceptions about delivery of game-based learning for virtual patients in Second Life. In N. Zagalo, L. Morgado, & A. Boa-Ventura (Eds.) Virtual worlds and metaverse platforms: New communication and identity paradigms, 138–148 (2012). Hershey, PA. Vrasidas, C., & Solomou, M. (2013). Using educational design research methods to examine the affordances of online games for teacher learning. Educational Media International, 50(3), 192– 205. Waters, J., & Leung, M. (2013). A colourful university life? Transnational higher education and the spatial dimensions of institutional social capital in Hong Kong, Population, Space and Place. 19, 155–167. FP:120 132 Gregory, S., Gregory, B., Wood, D., O’Connell, J., Grant, S., …, & Irving, L. (2015). New applications, new global audiences: Educators repurposing and reusing 3D virtual and immersive learning resources. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:109- FP:121). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:121 133 Conditions for successful technology enabled learning Michael Henderson Glenn Finger Kevin Larkin Monash University Griffith University Griffith University Vicky Smart Rachel Aston Shu-Hua Chao Griffith University Monash University Monash University This paper reports on the findings of a 16 month project funded by the Australian Government Office for Learning and Teaching. The project utilized an iterative mixed method design to investigate (a) what digital technologies are used and valued by students and educators for learning, and (b) the different factors within the ‘ecology’ of the university that contribute to these successful uses of digital technology. In total 2838 students and staff across two Australian universities and a further 114 leaders from all 39 Australian universities participated in the project. Through large scale surveys and in- depth case studies thirteen ‘conditions for success’ were identified that appeared to stimulate, support, and/or sustain specific success stories. These conditions relate to different aspects of the ‘ecology’ of higher education – from individual skills and attitudes through to institutional policymaking. This paper describes the conditions for success, and concludes with challenges to the higher education sector. Keywords: Technology enabled learning Introduction The nature of technological innovation and change in educational institutions is highly complex and contingent on multiple and often-contradictory influences over time (Fullan 2007). Consequently we should be cautious of overly deterministic or simplistic rhetoric of technology-related ‘impact’ and ‘effect’ on universities. This project addresses the long-standing gap between the rhetoric and the realities of technology enabled learning (TEL). For example, it examines the disparities between the educational potential of technology in comparison to what takes place in practice. This is a tension that recurs throughout much of the research and practitioner literature on technology use within higher education. On the one hand, there is evidence for the potential of digital technology to support and sustain meaningful and effective forms of learning. Networked digital technologies have undoubtedly transformed the generation and communication of knowledge and, it follows, that this has influenced the ways in which learning takes place (DeSchryver, 2015). Consequently, the potential to ‘support’, ‘enable’, or even ‘enhance’ learning has therefore been associated with every significant development in digital technology over the past twenty years or so. Recently, this has involved discussions over the educational benefits of podcasting; blogs and micro- blogs; social networking sites; and other forms of social media (Brady, Holcomb & Smith, 2010; Dale & Pymm 2009; Ebner, Lienhardt, Rohs & Meyer, 2010; Veletsianos & Navarrete, 2012). There has been much written about the ways in which digital technology can support creative, connected and collective forms of learning and study (see Buzzetto-More, 2012). New technologies are widely seen to support students in the co-creation of knowledge with peers, engagement in interest-driven informal learning practices, and the personalised engagement with education on an ‘anytime, anyplace, any pace’ basis. On the other hand, concerns remain over the less spectacular realities of digital technology use within university teaching and learning (see Losh, 2014). While many commentators talk of collaborative communities of content creators, in reality many students engage with technology in far more passive, sporadic and solitary ways; both for educational and non-educational purposes (Kennedy, Judd, Dalgarno & Waycott, 2010; Yılmaz, Yilmaz, Öztürk, Sezer & Karademir, 2015). For instance, recent FP:122 134 studies have found that university students often are ineffective in their use of the Internet and other digital research tools. As the recent ‘Net Generation’ study of UK universities concluded, students report varying levels of digital confidence and skills often resulting in “surprise or confusion at the array of [educational] technologies that were available” (Jones, 2012). Similar shortfalls in engagement have been reported with many of the applications and devices presumed to be integral to the lives of current cohorts of students. As another recent study of university students’ use of social networking sites concluded, educators need to “proceed with caution when using technology-enhanced learning, to avoid over-generalising the needs of the so-called Gen Y students” (Lichy, 2012, p.101). This project starts from the premise that any study of technology-related change and innovation needs to recognize the systemic nature of educational activity, and strive to develop understandings of the dynamics of how new technologies and techniques become embedded in the broader ‘ecology’ of local practice. Such an ecological approach also serves to clarify the institutional policies, practices, cultures and routines that shape that appropriation. As Zhao and Frank (2003, p.807) describe, the ecological metaphor offers “a powerful analytical framework for understanding technology use” in education. Understanding the university ‘ecology’ therefore highlights the varied influences at the level of the individual student and teacher, alongside the layered ‘context’ of the classroom, department, faculty, university, local community, state and nation, as well as the presence of many different competing innovations at any one time. Research design The project was conducted from January 2014 through until April 2015 and was designed as an iterative mixed method investigation conducted over three phases as shown in Figure 1; namely Phase One - focusing on how TEL was taking place in two large universities; Phase Two - identifying examples of ‘promising practice’ within the two universities; and Phase Three - exploring how these uses might be sustained across 39 Australian universities in the Australian higher education sector. Figure 1. Project data collection phases In Phase One, large-scale online surveys were administered to students and staff in both universities. The surveys were designed to elicit details about what digital technologies students used in relation to their studies, and their experiences of TEL. The surveys also helped to identify successful instances of TEL. The follow-up group interviews were subsequently carried out with students and staff who responded to the surveys. The focus-group interviews were designed to explore in depth issues and themes arising from the large-scale surveys as well as to validate our interpretation of the large-scale data and to provide an opportunity for new lines of inquiry to emerge. In Phase Two, the project then explored different examples across the two universities where students and teachers identified successful instances of TEL. From the findings arising from Phase One of the project, ten diverse examples of ‘promising practice’ were identified across the two FP:123 135 universities, and examined in detail as stand-alone case studies. ‘Promising practice’ are understood to be programs, activities or strategies that have “worked within one organization and shows promise … for becoming a best practice with long term sustainable impact [and] potential for replication among other organizations” (OACF 2013, n.p). The case studies can be found at: https://bitly.com/whatworksandwhy. The cases were not chosen according to the most ‘interesting’, ‘innovative’ or ‘cutting-edge’ examples of technology use, but rather were chosen to demonstrate sustainable examples of TEL. The Phase One survey data identified patterns of successful TEL, such as the use of supplementary media themes and, coupled with the rich descriptions and examples provided by the focus groups, identified specific instances of successful TEL. Each case study of ‘promising practice’ was drawn from: • Examination of the pedagogic/instructional design elements of these technology-based practices; • Interviews with 45 students: relating to the impact of the technology on their learning outcomes and learning experiences; • In-depth interviews with 12 educators / instructional designers relating to the course design and implementation; • Observation (in-person and online) of the TEL in practice. The purpose of the case studies was two-fold. First, to provide a record of ‘promising practice’ that other educators and institutions may choose to adopt. Second, to provide a rich source of data for analysis, in conjunction with Phase One data, to develop a series of propositions regarding the ‘ecology’ of the TEL, which we have termed ‘conditions for success’. Phase Three then considered ways that current ‘promising practice’ examples of TEL might be leveraged on a widespread and sustained basis across Australian universities. This involved two activities: Expert-group consultations were held within each of the case study universities, whereby 14 teaching and learning university leaders were presented with each of the ten ‘promising practice’ examples, and asked to critically engage with the proposed ‘conditions for success’ required for this technology use to be adopted on a more widespread basis in their institution. This process resulted in a refinement of the phrasing of the ‘conditions for success’ and informed the design of the survey in the next step. A ‘feed-forward’ consultation exercise was then conducted across the 39 universities in Australia. Teaching and learning experts and leaders in each university were contacted and informed of the ‘promising practice’ case studies, and asked to complete brief responses to the ‘conditions for success’ required for the types of TEL identified in this project being adopted on a wide-scale basis. This process was highly successful with responses from 85 senior leaders from all 39 universities, along with 29 other leaders. This process then led to a further refinement of the ‘conditions for success’, and the development of conclusions for ‘moving forward’. Findings – conditions for success This paper focusses on the proposed conditions for success arising from an analysis of the three phases of data collection. Other findings, and more detailed analysis of each phase is provided elsewhere (Henderson, Selwyn & Aston, 2015a; Henderson, Selwyn, Finger & Aston, 2015b). Similarly the 10 case studies are described on the project website [https://bitly.com/whatworksandwhy]. In developing the proposed conditions for success the histories, practices, enablers and challenges highlighted by the rich data of the Phase Two case studies were triangulated with the Phase One survey and focus group data until the project team felt there was theoretical saturation. This resulted in the identification of 16 initial ‘conditions for success’. These were then presented, in Phase Three, to teaching and learning leadership teams from both universities. Out of this process the ‘conditions for success’ were refined to better communicate the key messages. This refined version was then used in the subsequent feed-forward process with all 39 universities. Their feedback led to further refinement and re-organisation to better convey the key messages. The final 13 ‘conditions for success’ are illustrated in Figure 2 and outlined below. FP:124 136 Importantly, TEL is a broad term and cannot usefully be understood as a single practice, process or outcome. Therefore the ‘conditions for success’ revealed by this project are not necessarily applicable to all instances of TEL, nor are they an exhaustive list. However, they do describe a series of significant contributing factors to the ‘success’ of TEL. Conceptually, they have been organised according to those conditions attributable to institutions, educators and the learners themselves. Institutions: resource and culture Clearly, the access to, and reliability of, the technology resourcing was a key issue in leading to successful instances of TEL. In particular, it was observed in this project that successful TEL occurred when: 1. technical infrastructure is reliable and high capacity. University systems require sufficient bandwidth and generous capacity for streaming videos and storing large files. This also includes teaching spaces being able to support large numbers of simultaneous wireless connections. Teaching spaces are technologically flexible and technology friendly. Providing confidence to educators and students that TEL could occur wherever teaching is scheduled to take place. Our data highlight the need for lecture theatres and seminar rooms that are flexible and reliable; set up for lecturers to simply walk up, plug-in and play; had intuitive interfaces and control technologies; appropriate display and recording technologies; and supported ‘bring your own device’. Figure 2. Technology enabled learning: Conditions for success FP:125 137 The data from Phase One and Phase Two also highlighted the issue of how successful TEL is influenced by wider cultures within the university. This includes officially sanctioned TEL activities that have evolved from institutional histories, policies, and practice, but also the use of technologies and activities that are seen as working around the perceived constraints of the institution. The following propositions are key ‘conditions for success’ in relation to institutional culture. Successful TEL occurred when: Digital technology is part of common understandings of teaching and learning. Many of the successful TEL examples were built into the dominant structures of a course (e.g., curriculum and assessment), and presented as an expected mode of teaching and learning. These were not presented as non-standard and/or exceptional ‘innovations’. There are permissive approaches to configuring systems and choosing software . Successful instances of TEL all depended upon the university technical and support systems being configured in ways that allowed (either actively supported or at least did not exclude) staff and students to pursue what were often non-standard uses of technology. Often staff were using a number of ad hoc ‘work arounds’. there is a legacy of innovation that staff can build upon. Many of the successful TEL examples were the legacy of institutional seed-funding and pilot projects. Some of the ‘successes’ from our case studies were the ‘Nth generation’ results of previous university funded projects that were considered to have failed at the time, or simply were discontinued. These projects seeded ideas that were being later realized in local iterations. Evidently, the success of TEL initiatives should not be measured in the short term, suggesting the value of a culture of seed funding and grass roots innovations and acceptance of ‘failure’ as a legitimate process of innovating practice. Educators Successful instances of TEL were largely mediated by the educators themselves. In some instances, these individuals were clearly some of the ‘usual suspects’ when it comes to technology use, in other words, those with personal interests, skills, passions, confidence and/or curiosity when it comes to using technology in their teaching. Yet not all the case studies were being driven by ‘early adopters’. In this project it was observed that successful TEL occurred when: Educators actively design their use of digital technology to support learning, not just teaching. Technologies are often celebrated for the ways they can enhance the ‘delivery’ of the curriculum such as videos, content management systems, and visually appealing presentations. However, such focus on technology enabled teaching should not distract attention from the purposeful use of technologies to support learning. Importantly, this involves educators having a clearly articulated understanding of how students learn so that they can design appropriate technology enabled situations. The uses of digital technology fit with familiar ways of teaching (and learning). Many of the examples of technology ‘working well’ were interventions that had obvious continuations with well-established practices and products. These were forms of technology that worked with, rather than worked against, well-established cultures, traditions and routines of teaching. Digital technologies are used to engage with students. Many of our case studies involved staff making explicit efforts to ‘connect’ and meaningfully interact with their students. For instance, polling, annotation, and flipped classroom strategies were a part of lecturers’ attempts to be reflexive to student learning needs. Such approaches signify a changing understanding of the teacher in higher education, recognizing the value and need to identify-with, engage and respond to students who are no longer understood as passive recipients of knowledge, but rather as people who need to actively assimilate or accommodate new ideas into their individual mental models. Digital technologies and teaching are deliberately orchestrated. Obviously, staff and students need some degree of technical skills to use the digital technologies. However, it was clear from an analysis of the data collected that successful application of TEL required the ability for educators to not only perform with technologies, but also to orchestrate the technologies (often multiple technologies simultaneously such as PowerPoint, video and polling) in meaningful conjunction with teaching (including delivery, student activities, responding to student FP:126 138 needs, etc.). Educators create digital content fit for different modes of consumption. There is an increasing awareness of teaching as performance ‘in the moment’, as well as producing oneself for on-line consumption. Teachers were mindful that teaching is no longer a temporary condition. For instance, synchronous face-to-face teaching is often recorded and has an asynchronous ‘after life’ with students wanting to revise and rewind. Similarly, posting videos, engaging in webcasts, replying to forums, and making broadcast announcements can all be consumed by students in non-linear and asynchronous ways to meet students’ needs. Staff were planning and producing teaching events, activities and resources that support both the immediate goals and these different modes of consumption. Learners In the case studies of successful TEL, students were highly engaged with the digital technology practices. As indicated in the Phase One survey and focus groups, and confirmed in the Phase Two case studies, simply embedding digital technology into the curriculum does guarantee student engagement. In this project, it was observed that successful TEL occurred when: Learners recognize and value the benefits of the technology based practices. These successful instances of TEL were all accepted by students as part of the mainstream course culture. Students saw these technologies as having clear, practical use in terms of understanding content, and of the longer-term benefit in producing assignments and gaining better grades. University technologies mirror students’ everyday technology practices. TEL seems to ‘click’ with students when it fits with their wider digital media practices, that is, when the technologies and their uses are familiar and intuitive. Viewing short videos is a familiar use of digital technology that translates easily over into academic study. However, while the technology may seem familiar, the learning purpose and context can make it new or strange. Assumptions of digital natives valuing, seeking and being expert at new media practices in the context of formal learning needs to be questioned. Consuming short videos for leisure or informal learning can involve significantly different processes to engaging with, for instance, lecture recordings. The issue here is that TEL should be considered in terms of whether or not it involves familiar technologies and practices that can be intuitively applied to the learning context. However, this needs to be critically balanced against making assumptions of learner affinities for, and expertise with, technologies. Technology enabled activities fit with learning preferences. This was particularly evident in recurring themes of visual learning. There is clearly a shift in the minds of many students that they are ‘visual learners’. A number of these examples of promising practice related to this mode of encountering content and engaging with learning. These were uses of technology that framed teaching and learning as an image-based - as well as a text and speech- based - event. Challenges to the conditions for success Phase Three offered a useful opportunity to refine the conditions for success as well as to consider them in terms of institutional strategic priorities. In total, 114 survey responses were received from university leaders and managers. This included 85 senior leaders (ranging from Pro Vice-Chancellors through to Faculty Deans) across all 39 universities in Australia. Our survey of senior leaders from across the 39 Australian universities indicated that, in their institutions, most of the conditions for success are at least two or more years away from being achieved. In addition, the leaders reported a number of challenges to ‘successful’ technology enabled learning being sustained on a mainstream basis. The dominant institutional concerns were: • Financial prudence particularly in relation to limited budgets; • Working with a large and costly infrastructure, including technology and services; • A highly diverse workforce that is difficult to change in terms of attitudes and skills; • The need for managing risks, and ensuring standards and quality of service across the large institution; and • Satisfying a perceived need for innovation that precludes more obvious or familiar ways of engaging in TEL. FP:127 139 There is clearly a tension between the need to balance the diverse needs, requirements and demands of different sections of a ‘university’. Moreover, a one-size-fits-all approach to TEL is also inappropriate. Therefore, any response to the ‘conditions for success’ might be different according to ‘ecological’ variations within and across universities, including disciplines, locations and other contexts. However, the data from all three phases does suggest a number of areas that universities need to actively investigate when working towards sustaining effective use of technology to support student learning. Aligned with the ‘conditions for success’, these areas are presented in relation to institutions, educators and learners. Laying the foundations within institutions: 1. Establishing TEL expectations as an integral part of the university culture: Many of these examples of ‘what works and why’ are currently ‘exceptions to the rule’ rather than mainstream practices. If the university believes in principles such as ‘flipped classroom’ then this needs to be built into dominant structures (e.g., curriculum, assessment, resourcing), and presented to teachers and staff as an accepted and/or expected mode of teaching and learning. Considering TEL strategies such as polling, 3D printing, or social networking as “innovations” signals them as non-standard or exceptions. Providing teaching spaces that are technologically flexible and technology friendly: Lecture theatres and seminar rooms remain key places where TEL takes place. They need to be flexible and reliable – set up for lecturers to simply walk up, plug-in and play. This is now the era of lecturers and students ‘bringing their own devices’. Spaces need to be designed with less emphasis on the lecture-based PC in the corner and, instead, expectations of wireless connectivity and high specification display technology. The aim here is to give confidence that TEL can occur wherever teaching is scheduled to take place. Good resourcing: This is clearly essential to supporting technology use. These are issues that universities are clearly aware of, but should not be forgotten about and requires an understanding of the institution provision and the student provision of these digital resources which constitute the digital ‘ecosystem’ for staff and students. The primary area for attention is sustaining reliable and high capacity technical infrastructures - including sufficient bandwidth and capacity for streaming videos, storing large files, and large numbers of simultaneous wireless connections. Seeding successful forms of TEL: There is a clear tension between universities wanting TEL to be a process of change and innovation, and wanting to retain control over how technologies are used. Many of the successful forms of technology use in this project were organic and ‘bottom up’ in nature – the result of gradual changes and evolutions, rather than imposed change. Evidently, the success of TEL initiatives should not be measured in the short term, suggesting the value of a culture of seed funding and grass roots development and acceptance of ‘failure’ as a legitimate process of changing practice. Working with educators Moving beyond the ‘usual suspects’ to promote TEL principles and practices to staff: There is clearly a role for central university agencies to better establish TEL principles and practices in the collective consciousness of students and staff, not just the ‘usual suspects’, ‘early adopters’ and the ‘already converted’. Educators who engage with teaching and learning initiatives and events are likely to be willing converts or early adopters and do not necessarily further disseminate practices to others. Developing forms of TEL that are relevant to current ways of teaching: TEL works best where there is continuity with familiar ways of teaching and using technology. TEL also works best where there is obvious relevance to the ‘job’ of being a student. Doing the simple things well is likely to build confidence and eventually encourage more radical uses and changes. Working with staff to develop their own understanding of how students learn: Successful instances of TEL in this project were founded on purposeful implementation of digital technologies to support specific learner needs. This often included the educators having a clearly developed sense of the need to engage with students, rather than simply produce content (or oneself) for consumption. Finding ways to cede control to educators who want to try something different: This might include taking a permissive approach to allowing staff to install applications and programs of their choice, or at least being able to choose to use non-enterprise services. This FP:128 140 could take the form of authorities “looking the other way”, but also providing limited funding and technical support for non-enterprise services (e.g. polling systems, blogging, etc.). Working with learners Working directly with learners to develop appropriate and effective forms of TEL. Many of the TEL activities of universities focus on staff. Closer attention should be paid to students. Students are perhaps the best source of identifying and championing best practice of TEL – and could be a key source for creating demand for the spread of better TEL practices. Students also need to be better informed of TEL planning and proposals. TEL should not be something that is ‘done to’ students – rather it should be ‘developed with’ students. This is likely to result in effective and readily accessible forms of TEL. It may also facilitate student recognition of the benefits and purpose of the TEL practices that are implemented. Working directly with learners to help them ‘learn how to learn’ with technology. Students need to be aware of the practices, implications and expectations related to TEL as much, if not more than educators. They need support to use the technology but, more importantly, how to learn with the technology. Conclusion This project began with the assumption that TEL cannot, and should not, be explained as simple interventions with inevitable (positive) outcomes. Analysis of the data in this project confirm Fullan’s (2007) claim that innovation and change in educational institutions is highly complex and contingent on multiple and often-contradictory influences over time. The rhetoric of digital natives, elearning, digital revolution, can lead some to conclusion that the combination of students, digital technologies and education is not only expected but also ultimately successful and largely unproblematic strategy. In contrast this project found the actual usage of technologies for learning is rather low-level and low- key in comparison to the enthusiasms that often surround TEL. For instance students most valued those digital technologies that helped them to managing the logistics of university study (e.g., online access to the library) and when specifically directed to consider learning with technologies they most commonly described forms of consumption of information and content rather than any of the much celebrated forms of active learning with technologies such as collaboration via social media (for discussion see: Selwyn & Gorard, 2015). The ‘reality’ of student experience is also punctured by a number of frequently cited problems including instances where technology: • has failed to function, preventing them from working • distracts them from the task at hand (this includes their own technologies and those around them) • might not be the most suitable tool despite being proscribed by the learning task • is detrimental to their learning, such as “death by PowerPoint” in lectures and poor quality digital learning materials. (for discussion see: Selwyn & Gorard, 2015) Nevertheless, the project did identify patterns and cases where TEL was successful and was sustained over time. This resulted in proposing 13 conditions that support ‘successful’ instances of TEL. These include conditions at different levels: institutions, educators and learners. Obviously, these ‘conditions for success’ are not necessarily applicable to all instances of TEL, nor are they an exhaustive list. In addition, the conditions are difficult to achieve. This was particularly highlighted by the 85 senior leaders from the 39 Australian Universities who clearly revealed a tension in managing these concerns while also balancing the diverse needs, requirements and demands of different sections of a ‘university’ where a one-size-fits-all approach is inappropriate. It seems reasonable therefore to suggest that any response to the ‘conditions for success’ may be different according to ‘ecological’ variations within and across universities, including discipline, location and other contexts. In this vein, we propose that the notion of ‘ecology’ can be usefully employed to drive a more localised and strategically focused approach to TEL. We also propose that the conditions and challenges arising from this project are useful starting points for each institution. Acknowledgement Support for this project has been provided by the Australian Government Office for Learning and Teaching. The views in this project do not necessarily reflect the views of the Australian Government Office for Learning and Teaching. FP:129 141 References Brady, K., Holcomb, L. & Smith, B. (2010). The use of alternative social networking sites in higher educational settings: A case study of the e-learning benefits of Ning in education. Journal of Interactive Online Learning, 9(2), 151-170. Buzzetto-More, N. (2012). Social networking in undergraduate education. Interdisciplinary Journal of Information, Knowledge, and Management, 7(1), 63-90. Dale, C. & Pymm, J. (2009). Podagogy: The iPod as a learning technology. Active Learning in Higher Education, 10(1), 84-96. DeSchryver, M. (2015). Higher-order thinking in an online world. Teachers College Record, 117(3), 1- 44. Ebner, M., Lienhardt, C., Rohs, M. & Meyer, I. (2010). Microblogs in Higher Education – a chance to facilitate informal and process-oriented learning. Computers and Education, 55(1), 92–100. Fullan, M. (2007). The new meaning of educational change [4th edition]. New York: Teachers College Press. Henderson, M., Selwyn, N. & Aston, R. (2015a). What works and why? Student perceptions of ‘useful’ digital technology in university teaching and learning. Studies in Higher Education. Advance online publication. http://dx.doi.org/10.1080/03075079.2015.1007946 Henderson, M., Selwyn, N., Finger, G., & Aston, R. (2015b). Students’ everyday engagement with digital technology in university: exploring patterns of use and ‘usefulness’. Journal of Higher Education Policy and Management, 37(3), 308-319. doi. 10.1080/1360080X.2015.1034424 Jones, C. (2012). Networked learning, stepping beyond the net generation and digital natives. In Dirckinck-Holmfeld, L., Hodgson, V., McConnell, D. (Eds.), Exploring the Theory, Pedagogy and Practice of Networked Learning. New York: Springer. Kennedy, G., Judd, T., Dalgarno, B. & Waycott, J. (2010). Beyond Natives and Immigrants: Exploring the Characteristics of Net Generation Students. Journal of Computer Assisted Learning, 26, 332- 343. Lichy J. (2012). Towards an international culture: Gen Y students and SNS? Active Learning in Higher Education, 13(2), 101-116. Losh, E. (2014). The war on learning. Cambridge, MA: MIT Press. Office of the Administration of Children and Families. (2013). What are best practices? Retrieved April 22, 2015, from http://eclkc.ohs.acf.hhs.gov/hslc/tta-system/operations/mang-sys/fiscal- mang/tAreBestPracti.htm Selwyn, N. & Gorard, S. (2015). Students' use of Wikipedia as an academic resource - Patterns of use and perceptions of usefulness. The Internet and Higher Education, 28(January 2016), 28-34. Veletsianos, G. & Navarrete, C. (2012). Online social networks as formal learning environments: Learner experiences and activities. The International Review of Research in Open and Distance Learning, 13(1), 144-166. Yılmaz, F., Yilmaz, R. Öztürk, H., Sezer, B. & Karademir, T. (2015). Cyberloafing as a barrier to the successful integration of information and communication technologies into teaching and learning environments. Computers in Human Behavior, 45, 290-298. Zhao, Y & Frank, K. (2003). Factors affecting technology uses: an ecological perspective. American Educational Research Journal, 40(4), 807-840. Henderson, M., Finger, G., Larkin, K., Smart, V., Aston, R., & Chao, S.H. (2015). Conditions for successful technology enabled learning (2015). In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:122- FP:130). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:130 142 To type or handwrite: student's experience across six e- Exam trials Mathew Hillier Monash University University of Queensland This paper reports on student's experience of e-Exams as collected via surveys undertaken in conjunction with a series of optional live trials of an open source, bring- your-own-device (BYOD) based e-Exam system in six mid-semester undergraduate examinations during 2014 at The University of Queensland, Australia. A set of surveys were conducted prior and following each exam that covered ease of use, technical issues, comfort, confidence, time, typing versus handwriting prowess. Responses to Likert items were compared between those students who elected to type and those that handwrote their exam. Insights as to which issues proved significant for students will prove useful to institutions looking to implement computerised exams. Keywords: e-exams, computer-assisted assessment, high-stakes testing, bring-your- own-device (BYOD). Introduction A range of drivers, issues and a rationale for the introduction of e-exams have been previously articulated by Hillier & Fluck (2013). Drivers include the increased use of computers in study, work and private life, near ubiquitous ownership of laptops by students reported as high as 94% by (2015), and the societal need for institutions to produce ICT literate graduates equipped with skills for the twenty first century (Binkley, Erstad, Herman, Raizen, Ripley, Miller-Ricci & Rumble, 2012). Issues include the provision of equipment for large, infrequent exam events, and if student owned devices are to be used, the diversity of student owned equipment and the high investment of students in their equipment. An e-exam system also needs to be easy to use when students are under stress, reliable and robust against attempts of misconduct. There is also a need to provide an equivalent exam environment for all candidates, while being reliable, sustainable, scalable for institutions to implement. Problems such as equipment supply, exam integrity, technical support, scalability and location need to be addressed with multiple possible combinations. For example, the dimensions of location and connectivity are mapped in Figure 1 to demonstrate that there is no perfect solution. Online • Space issues for institutions. • No space issue for institutions. • Improved exam management efficiency. • More efficient exam management. • Equipment: need computer labs for 2000 at • Students supply equipment. once. • Less secure: students at home. • More secure: it is supervised. • Needs reliable network. • Needs reliable network. • Tech support more problematic. • Tech support more straightforward (if in labs). • Space issues for institutions. • Less efficient exam management. • No space issue for institutions. • Equipment: need computer labs for 2000 at • Less efficient exam management once. • Students supply equipment. • More secure: it is supervised. • Less secure: students at home. Offline • Network reliability not an issue. • Network reliability not an issue. On Campus Distance Figure 1: The location and connectivity dimensions of the e-Exams problem Authors such as Ripley (2007) and Fluck and Hillier (2014) also argue that a significant untapped potential exists in e-exams to remove a 'block' to curriculum transformation given that existing paper- based mode of assessment can be a significant driver of both learning focus by students (Ramsden, 1992, Gibbs, 1999) and a disincentive for teachers to reform curriculum. The potential of a comprehensive yet open architectural approach to computerised exams would greatly expand the 'pedagogical landscape' in the exam room. A computer enhanced exam platform capable of FP:131 143 sophisticated constructed responses and able to provide the 'tools of the trade' used in professional practice will allow for much more authentic assessment tasks characteristic of a twenty first century problem environment (Binkley et al., 2012) to be set. Such tasks could include working through a complex financial simulation; using a medical diagnostic tool to work up a diagnosis; using computer aided design software to respond to a design problem by producing three dimensional engineering schematic; production of an example of contemporary digital art; carrying out a virtual experiment and analysing the results; and so forth. This approach contrasts to the commonly used paper-based exams that limit the range of assessment activities that can be undertaken in an exam room. Similarly, current approaches to the automation of exam marking rely heavily on selected response, multiple choice style questions or provide an 'armoured word processor' that does little to move pedagogy forward into a twenty first century and instead largely replicate current paper-based questioning in a digital form (Fluck 2015). An Approach to e-Exams We have briefly outlined multiple dimensions that exist in developing an e-Exam solution. Looking at the issue of equipment supply, we argue that we should be making use of the large number of computers owned by students (Dahlstrom & diFilipo, 2013). The current high ownership rate of laptops by students at around 90% in the US (Dahlstrom & Bichsel, 2014) and a little higher at the author's own University at 94% (Hillier, 2015). An e-exam solution that uses bring-your-own devices (BYOD) for exams has been outlined by Hillier & Fluck (2013). Approaches to using BYOD also exist or are under development in Austria (Frankl, Schartner & Zebedin, 2011), Canada (Peregoodoff, 2014), Denmark (Nielsen, 2014), Finland (Lattu, 2014), Germany (Schulz & Apostolopoulos, 2014), Iceland (Alfreosson 2014), Norway (Melve 2014) and Singapore (Keong & Tay 2014). As we transition from pen-on-paper to keyboard-based exams decisions made about the format, processes and technology to be used for e-exams will directly impact students the most as will strategies used to address change management, technology literacy and equity. Work by Dermo (2009), Frankl, Schartner and Zebedin (2012), Terzis and Economides (2011), Mogey and Fluck (2014) identified a range of student's concerns that include integrity (minimising 'cheating'); reliability (stability of the equipment and software to perform error free); familiarity (as to minimise the distraction the computerised environment so that candidates focus on the exam); efficiency (particularly when compared to hand-written exams); and psychology (the impact of stress and anxiety). This range of issues was used to develop a pre-project institution-wide survey reported by Hillier (2014, 2015) that looked at student concerns in the study context. The findings from the survey showed that the main concern related to fear of technology failure, potential for cheating and the resistance by significant proportion of students in moving away from familiar pen-on-paper exams despite issues such as messy handwriting and physical discomfort in longer exams. Overall, a majority of students claimed interest in being able to type responses to an exam with a mean of 3.3 on a 5 point agreement scale. Stronger interest was shown by students in Information Technology, Software Engineering, Education, Law, Commerce, Business and Arts. Those in pure Mathematics, Physics and Engineering programs such as Mechatronics, Civil, Electrical and Chemical thought that the assessments in their discipline would not suit computerisation given their use of long-form formulae and/or extensive use of diagramming in responding to assessments. E-Exam Trial Design The study reported in this paper was undertaken at the University of Queensland, a multi-disciplinary university in Brisbane, Australia serving 50,000 students. The institutional ethics committee approved all data collection processes and instruments used in the study. This paper focuses on the second phase of the study in which live mid-semester exam trials were conducted in six courses. A pre-exam survey was conducted with students in set-up/practice sessions and post-exam surveys were conducted immediately following the exam session. The overall study design is depicted in Figure 2. FP:132 144 Phase 1 Institution wide online survey (see Hillier 2014, 2015) Phase 2, Step 1 e-Exam Trial Expression of interest Typists Handwriters Phase 2, Step 2 Pre-exam preparation survey Phase 2, Step 3 Type the exam Handwrite the exam Phase 2, Step 4 Post-exam survey Figure 2: Study design The set of six e-Exam trials ran across six courses in 2014. Each trial was broken down into four steps. Students undertaking mid-semester examinations worth between 15% to 25% of the course grade were given the choice of typing or handwriting the exam. Despite a desire to more fully utilise the capabilities of the computerised exam system, the choice offered to students directly impacted the nature of questions that could be used in the exam because questions had to work on both paper and electronic formats. The rationale for this choice was that of pragmatism. The findings from an earlier survey we conducted in the study context showed students were 'cautiously optimistic' towards e- exams (Hillier, 2014). Thus, we allowed a gentle introduction of a new approach to doing exams given the diversity of stakeholders involved and overall complexity of running exams (see Hillier & Fluck 2013). A mix of essay, short answer, table filling, diagram labelling and selected- response questions were used with suitable format adjustments made to cater for both paper and screen. See Figure 3 for a mock-up of typical questions. Figure 3: Example question formats Typists used their own laptop. Power sockets and spare laptops were provided in case of equipment incompatibility or failure. The final fall-back was pen-on-paper. The exam trial, depicted in Figure 4, required students to boot their laptop using an e-Exam 'Linux Live' USB storage device (Transforming Exams, 2014). The e-Exam USB contained a modified version of Ubuntu to prevent internet, bluetooth or local drive access along with LibreOffice (word processor) and a custom 'exam starter' that guided students to begin the exam. FP:133 145 Figure 4: e-Exam Trial workflow. Study Method The first step for student involvement in the exam trials was for students to complete an online 'expression of interest' (and consent form) indicating their choice of exam mode. Students were advised that they could change their mind at any time. The default for a non-response was handwriting. Those who expressed interest in typing were then asked to attend a set-up / practice session to provide an opportunity to become familiar with the e-Exam system and to ensure that the e-Exam system was compatible with their laptop. Those that attended the session were asked to complete a survey to collect data about their laptop and their first impressions of the e-Exam system. Finally, all students (both typists and hand-writers) undertook the exam and were asked to complete a post-exam survey. The two surveys used in the exam trial included a number of selected-response and several open text questions that provided an opportunity for students to report their impression and experience of the e- Exam trial. The focus in this paper is on reporting the outcomes of the selected-response questions while the emergent themes from the open response questions are reported elsewhere in Hillier (2015). Note that the responses from the selected-response items in the pre-project survey (phase 1) are reported in Hillier (2014). E-Exam Trial Participation The participant numbers at each step of the trial were monitored with the expectation that there would be attrition given the voluntary nature of the study. The number of students at each stage is displayed in Table 1. Table 1: Number of typists at each stage of the e-exam trial Yes Maybe Total No Steps of trial will type type typists Attrition (hand-write)* 1 Expression of Interest 201 201 361 2.1 Pre - before try 94 16 110 91 10 2.2 Pre - after try 86 15 101 9 23 4 Exam (after) 71 71 30 450 Note: not all respondents completed every question. A number of students electing to hand-write did not fill in the expression of interest and the post-exam survey so are slightly under represented. Similarly not all attendees at the pre-exam set-up session returned a survey. FP:134 146 There were just over 200 students (36%) out of approximately 560 students in the six courses who expressed interest in typing. Of these, 124 attended a set-up/practice session with 115 surveys returned. During the set-up/practice session, 94 said they intended on typing the exam before they had tried the e-Exam system. After trying the e-Exam system with their laptops, 86 said they still intended on typing their exam. Several students were offered the chance to book a university owned laptop due to their own being unsuitable. On exam day, 71 students typed their exam and 450 defaulted to hand-writing their exam. Participation for each of the six courses ranged from 5% to 34% with an overall 16% of students electing to type. The mid-semester exams ranged in duration and structure from 15 minutes of writing prior a practical clinical exam to 100 minutes of writing that involved short answer, essay and selected-response items. All e-Exams utilised word processing documents to facilitate typing. However, some exams used optical mark recognition sheets to collect larger groups’ multiple choice question responses. In cases where there were only a couple of selected response items in an exam, these were included in the word processor document with a response recorded by typing an 'x' into an appropriate box. The details of each course exam and the participation counts are listed in Table 2. Table 2 Number of typists in each course e-exam trial Course and Exam Type Typed Handwrote Animal Biology: 45 min mixed short answer and MCQ (type 'x') 5 109 Zoology (BIOL): 50 min short answer (MCQ section done pen on OMR 10 81 sheet) Criminology: 70 minutes. Single long essay response section (MCQ section 17 50 done pen on OMR sheet) Occupational Therapy: 100 min mixed short answer and MCQ (type 'x') 3 24 Physiotherapy: 15 min (watch video and write into a table) before clinical 25 108 exam Veterinary technology: 90 min theory, mostly short answer 11 78 Totals 71 450 Findings Analysis of selected-response items, in particular Likert scales followed advice from Dermo (2009). The Likert scale data were considered to be non-parametric (Jamieson, 2004) and so Mann & Whitney’s (1947) U test on the variance of two groups and Kruskal & Wallace’s (1952) test in instances of more than two groups were used in SPSS v22. The results of the pre and post exam phases of the data collection are presented in the following sections. Like Dermo (2009) we are interpreting statistical results as an indication of the body of opinion from students rather than a search for a single truth. Means are also given where applicable to assist the reader in understanding responses to five point scales. Pre-exam First Impressions During the set-up/practice session, student's initial impressions and intentions were surveyed prior to tying the e-exam system with their laptop and immediately following their first try of the e-exam system. Students were asked to rate the e-exam system using Likert items including the ease of following set-up instructions, the ease of undertaking the start-up steps, the ease of starting their computer with the USB stick and the ease of using the exam system software. They were also asked about their confidence in their ability to perform the necessary steps in a real exam and if they were 'relaxed' about the idea of using the e-exam system in their upcoming exam. These questions comprised the five point Likert items listed in Table 3, with 5 being 'strongly agree'. Table 3. Selected pre-exam session survey questions (typists only) Question N Mean SD The written instructions were easy to follow. 108 3.9 1.0 It was easy to learn the necessary technical steps. 105 4.0 1.1 FP:135 147 It was easy to start my computer using the e-Exam USB. 108 4.1 1.2 I feel confident I will be able to do these steps in a real exam. 106 4.0 1.1 The software within the e-Exam system was easy to use. 105 4.1 1.1 I now feel relaxed about the idea of using the e-Exam system for my upcoming 106 3.8 1.0 exam. At the end of the session, 115 surveys were returned. A graphical representation of the spread of responses on each item is displayed in Figure 5. Most were rated as 4 on the 5 point scale (5 being strongly agree/positive). Figure 5: Ratings of the BYOD based e-exam system (5 = strongly agree) Technical information was also collected relating to each student's laptop. This included brand/make, model/serial number, operating system used, estimated battery life, any technical adjustments required (e.g. secure boot settings and BIOS/EFI mode) and compatibility with the e-exam system including boot, graphics and performance of touch pads. A wide range of equipment was presented for testing with the single most common brand and operating system being Apple OSX with close to 70% of machines. The remainder of computers utilised versions of Microsoft windows 8 and 7 on nine different brands of hardware. The results of technical testing of student's laptops showed that around 20% were found to be incompatible with the e-exam system due to graphics card or other indeterminate issues related to EFI or BIOS limitations. A planned upgrade to the e-Exam operating system is expected to reduce this issue in the future. Students were offered the opportunity to reserve a backup laptop in the event theirs was not suitable. Several non-critical issues were identified that lead to contingencies being put in place, such as provision of power or additional instructions to adjust screen resolutions where retina screens were used. Figure 6 provides numerical details of student hardware and test results. Laptops Tested Laptop Pass Rate and Issues Encountered All figures are counts Figure 6: Laptop testing results Post-exam Findings The post-exam survey was conducted following the collection of exam responses. The survey contained a number of selected response items covering students experience of the exam session, stress or comfort levels, adequacy of exam timing, ease of use of the exam system, suitability of the FP:136 148 exam questions for computerisation, writing strategies and general use of computers for study related writing tasks. Table 4. Selected post-exam session survey questions Question Typists Hand-writers N Mean SD N Mean SD I typed (or handwrote) this exam. 71 - - 450 - - I felt the e-exam system was easy to use. 69 4.4 0.8 - - - I felt the e-exam system was reliable against technical failures. 69 4.1 1.0 - - - I felt the e-exam system was secure against cheating. 69 4.3 0.9 - - - I liked the fact I could use my own computer. 61 4.5 0.8 - - - I would recommend the e-exam system to others. 68 4.3 0.9 - - - Overall my experience of this exam was positive. 71 4.0 1.0 439 3.8 1.0 I ran out of time. 70 2.6 1.5 437 2.6 1.5 I felt more stressed in this exam than I normally do in other exams. 70 2.6 1.3 439 2.7 1.3 I went back over my responses before submitting. 71 3.5 1.5 439 3.5 1.4 I would like to use a computer for exams in the future. 13 4.2 0.9 99 1.8 1.0 I felt this particular exam suited the use of computers. 70 4.2 0.9 - - - I think my handwriting was neat and legible. - - - 453 3.4 1.2 I experienced discomfort in my writing hand. - - - 389 2.4 1.3 I type faster than I handwrite. 67 4.5 0.9 368 3.7 1.5 I type accurately. 66 4.2 1.0 369 3.5 1.1 When I make errors, I can quickly correct them as part of typing. 67 4.5 0.8 368 3.9 1.1 I often rely on spell check to detect errors. 67 3.4 1.3 368 3.6 1.3 I work more efficiently when I type on a familiar keyboard. 67 4.5 0.9 368 4.3 0.9 My hand-writing is normally neat and legible. 67 3.3 1.4 368 3.5 1.1 Questions relating to student's impressions of using the e-Exam system are shown in Figure 7. The feedback was generally positive with ratings of 4 or above on a 5 point scale across multiple items. 5 4.4 4.0 4.2 4.5 4.3 Boxplots: responses from typists. 4 Bars represent medians. 3 Means shown for clarity. 2 Y-axis Likert scale: 1 5 = Strongly Agree 1 = Strongly Disagree Figure 7: Student impressions of using the exam system Those that typed were also asked if they felt the exam they had just done suited the use of computers. The majority of students agreed or strongly agreed with the statement with a mean of 4.2 (see Figure 8). It is worth noting two issues at play here. First, students who typed are self-selecting and are thus predisposed to agreement. However, the exams were designed such that paper or computer could be used and therefore elements such as multimedia or interactive tools that would have added value were not possible in these exams making the 'value add' of computerisation much more limited. FP:137 149 I felt this particular exam suited the use of computers. X-axis Likert scale: 5 = Strongly Agree 1 = Strongly Disagree Mean agreement 4.2. Figure 8: Student reported suitability of each exam for computerisation. All students were then asked about their direct experience of the exam session conditions. An aggregated analysis across the six courses was performed to compare responses from typists and hand-writers on questions that related to their overall experience of the session, time availability, stress and whether they re-checked their responses prior to submission of responses. Students were also asked if they would consider using a computer in a future exam. Results are graphically presented in Figure 9 and Mann-Whitney U test results in Table 5 that shows the only significant difference in their 'overall experience'. Visual inspection also reveals that typists were slightly less stressed than hand-writers. A question relating to future intended use of computers for an exam was introduced for the final two courses. The differences by exam mode were significant and while this was expected given the self-selected nature of the two groups, there were some hand-writers who had interest in using a computer for exams in the future. 4.04, 3.76 2.61, 2.61 2.56, 2.69 3.48, 3.49 4.23 1.81 Key: Purple (right) = typists Orange (left) = hand-writers Bars represent medians. Means shown for clarity. Likert scales: 5 = Strongly Agree 1 = Strongly Disagree Figure 9: Student reported experience of exam conditions and future intentions. Table 5: Test Statistics for Student reported experience of exam conditions and future intentions Grouping Variable: I went back and I would like to I typed this exam Overall my I felt more stressed read over my use a computer (Yes / No) experience of in this exam than I responses for exams in this exam I ran out of normally do in other before the future was positive time exams submitting Mann-Whitney U 13242.5 15203 14527.5 15145.5 74 Z -2.132 -.083 -.751 -.394 -5.532 Asymp. Sig. (2-tailed) >.05 n/s n/s n/s >.001 Issues identified by students in their post session comments (Hillier, 2015) and in the phase 1 survey (Hillier, 2014) indicated that the neatness of handwriting and discomfort such as cramps experienced in longer exams was a recognised issue. Anecdotal comments from teachers involved in the trial also indicated a perceived decrease in the readability of student's handwriting in exams. To explore these two issues, hand-writers were asked if they thought their handwriting was neat (N 453) and if they had experienced any discomfort in their writing hand (N 389). Figure 10 displays the responses of FP:138 150 students on these two issues by course exam. There were moderate levels of self-reported agreement (mean 3.4) in respect to the neatness of handwriting under exam conditions that was reasonably consistent across the different courses. This would appear to contradict the anecdotal reports from teachers. Significant differences >.001 were reported from a Kruskal-Wallis Test in the level of discomfort experienced when taking into consideration the length of the exam. The 70 minute mark was the transition point where a majority of students felt discomfort. Exams shorter than 70 minutes did not present undue issues for hand-writers although a minority were reporting discomfort in the 45 and 50 minute exams. In the longer exams of 90 and 100 minutes, while higher levels of discomfort were reported had a mixed response rate (VETS 18%, CRIM 73% and OCTY 92%) and lower numbers indicate that results still need to be interpreted with some caution. 107, 3.2 107, 1.8 109, 3.5 107, 2.4 85, 3.5 85, 2.4 49, 3.6 49, 2.9 78, 3.4 16, 3.9 25, 3.7 25, 2.7 N, Mean N, Mean Likert scales: 5 = Strongly Agree, 1 = Strongly Disagree. N = respondents per question. Means shown for clarity. Figure 10: Student reported neatness of handwriting and discomfort by exam duration Students reported in the phase 1 survey (Hillier, 2014) that their typing ability was likely to play a big part in them choosing a computerised exam. We asked trial participants to report on their abilities with respect to typing in general (outside of the exam context) including speed, accuracy, error recovery, spelling and error detection. They were also asked if they felt they were more efficient on a familiar keyboard given a strong response exhibited in the phase 1 survey in relation to using familiar keyboards. We also asked if they felt their handwriting was neat and legible in general. Results comparing those who elected to type the exam with those that hand-wrote are shown in Figure 11 as Boxplots with means also shown for clarity. 4.52, 3.67 4.23, 3.49 4.49, 3.88 3.37, 3.61 4.46, 4.31 3.28, 3.48 Key: Purple (right) = typists Orange (left) = hand- writers Bars represent medians. Means shown for clarity. Y-axis Likert scale: 5 = Strongly Agree 1 = Strongly Disagree Figure 11: Student reported use of typing and writing in general Significant differences were in favour of typists on matters of perceived typing speed, typing accuracy and being able to quickly correct errors when typing. However the degree of reliance on spell check, perceptions of efficiency on a familiar keyboard and self-reported general neatness of handwriting did not appear to be major factors in choosing to type the exam. These results are displayed in Table 6. FP:139 151 Table 6: Test Statistics for student reported typing and writing in general When I make My hand- Grouping Variable: errors, I… I often rely I work more writing is I typed this exam I type I type quickly correct on spell efficiently when I normally (Yes / No) faster than accuratel them as part of check to type on a familiar neat and I handwrite y typing detect errorskeyboard legible Mann-Whitney U 8213 7551.5 8523 11097 10917.5 11621.5 Z -4.637 -5.089 -4.248 -1.342 -1.656 -.770 Asymp. Sig. (2- >.001 >.001 >.001 n/s n/s n/s tailed) Conclusion The above results, in conjunction with findings published elsewhere (Hillier, 2014; 2015) raise awareness of relevant issues for institutions setting out to trial and implement computerised examinations. This paper looked at a range of student self reported impressions of their experience in undertaking a trial e-Exam in their course via 'selected-response' questions to pre and post exam surveys. Students were provided a choice as to typing or handwriting and so we were able to compare responses from these two groups. Self- reported speed of typing over handwriting, typing accuracy and an ability to correct errors when typing were found to be significant factors in students’ choice of exam mode. Students who chose to type reported positively on their experience with the e-Exam system, giving ratings of 4 or above on a 5 point scale. Similarly, typists’ impressions of the experience were positive overall and were slightly less stressed than those that handwrote. Findings show that those that hand- wrote their exam experienced discomfort in their writing hand as the duration of the exam increased. It was found that the 70 minute mark was the point at which the majority of students were impacted. However, some students were still impacted during 45 and 50 minute exams. Acknowledgements Acknowledgement is given to Dr Andrew Fluck at University of Tasmania, as partner on the e-Exams grant project, Ms Marissa Emerson who was the technical developer on the project, Ms Lan Tran, a summer research scholar who assisted with analysis of the first general student survey and Ms Karen Sheppard who assisted with part analysis of the post-session survey. Thanks are also given to the academics and students of the trial courses for their cooperation and willingness to 'give it a go'. References Alfreosson, F. (2014). Bring-your-own-device Exam System for Campuses. Presented at the 28th NORDUnet Conference, Uppsala University, Sweden. Retrieved from https://events.nordu.net/display/NORDU2014/Bring-your-own- device+Exam+System+for+Campuses Binkley, M., Erstad, O., Hermna, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining Twenty-First Century Skills. In Griffin, P., Care, E., & McGaw, B. Assessment and Teaching of 21st Century Skills, Dordrecht, Springer. http://link.springer.com/chapter/10.1007%2F978-94-007-2324-5_2 Dahlstrom, E., & Bichsel, J. (2014). ECAR Study of Undergraduate Students and Information Technology 2014. EDUCAUSE Center for Applied Research. Retrieved from http://net.educause.edu/ir/library/pdf/ss14/ERS1406.pdf Dahlstrom, E., & diFilipo, S. (2013). The Consumerization of Technology and the Bring-Your-Own- Everything (BYOE) Era of Higher Education (Research Report). Louisville, CO, USA: EDUCAUSE Center for Applied Research. Retrieved from http://www.educause.edu/library/resources/byod-and- consumerization-it-higher-education-research-2013 Dermo, J. (2009). E-Assessment and the student learning experience: A survey of student perceptions of e-assessment. British Journal of Educational Technology, 40(2), 203– 214. http://doi.org/10.1111/j.1467-8535.2008.00915.x Fluck, A. (2015). Reporting Progress With Post-paper Digital Exams. Education Technology Solutions Magazine. 12 May. Retrieved FP:140 152 from http://educationtechnologysolutions.com.au/2015/05/12/reporting-progress-with-post-paper- digital-exams/ Fluck, A., & Hillier, M. (2014). eExams Transforming Curriculum. In Now IT’s Personal (pp. 151–158). Adelaide, Australia: ACEC. Retrieved from http://acec2014.acce.edu.au/sites/2014/files/attachments/eExams%20paperd%20- %20REV2b.docx Frankl, G., Schartner, P., & Zebedin, G. (2011). The ‘Secure Exam Environment’ for Online Testing (Vol. 2011, pp. 1201–1211). Presented at the World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education, Honolulu, Hawaii, USA. Retrieved from http://www.editlib.org/p/38879/ Frankl, G., Schartner, P., & Zebedin, G. (2012). Secure online exams using students’ devices (pp. 1– 7). Presented at the IEEE Global Engineering Education Conference (EDUCON), Marrakech. http://doi.org/10.1109/EDUCON.2012.6201111 Gibbs, G. (1999), Using Assessment Strategically to Change the Way Students Learn, in S. Brown S. & Glasner A. (eds), Assessment Matters in Higher Education, Society for Research into Higher Education and Open University Press, Buckingham, UK Hillier, M. (2014). The Very Idea of e-Exams: Student (Pre)conceptions. Australasian Society for Computers in Learning in Tertiary Education conference, Dunedin, New Zealand. Retrieved from http://ascilite.org/conferences/dunedin2014/files/fullpapers/91-Hillier.pdf Hillier, M. (2015). e-Exams with student owned devices: Student voices. Presented at the International Mobile Learning Festival Conference (pp. 582-608), Hong Kong. 22-23 May. Retrieved from http://transformingexams.com/files/Hillier_IMLF2015_full_paper_formatting_fixed.pdf Hillier, M. & Fluck, A. (2013). Arguing again for e-exams in high stakes examinations. In H. Carter, M. Gosper, & J. Hedberg (Eds.), Electric Dreams (pp. 385–396). Macquarie University. Retrieved from http://www.ascilite.org.au/conferences/sydney13/program/papers/Hillier.pdf Jamieson, S. (2004). Likert scales: how to (ab)use them. Medical Education, 38(12), 1217–1218. DOI:10.1111/j.1365-2929.2004.02012.x Keong, S. T., & Tay, J. (2014, September). Bring-your-own-laptop e-exam for a large class at NUS. Presented at the eAssessment Scotland 2014 Online Conference, Dundee, Scotland, UK & Brisbane, Australia. Retrieved from http://transformingassessment.com/eAS_2014/events_10_september_2014.php Kruskal, W. H., & Wallis, W. A. (1952). Use of Ranks in One-Criterion Variance Analysis. Journal of the American Statistical Association, 47(260), 583–621. DOI:10.1080/01621459.1952.10483441 Lattu, M. (2014). Digitalisation of the Finnish Matriculation Examination - geography on the first wave in 2016. Invited talk presented at the Open Source Geospatial Research and Education Symposium, Otaniemi, Espoo, Finland. 10-13 June. Retrieved from http://2014.ogrs- community.org/2014_papers/Lattu_OGRS2014.pdf Mann, H. B., & Whitney, D. R. (1947). On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics, 18(1), 50–60. DOI:10.1214/aoms/1177730491 Melve, I. (2014). Digital Assessments, on Campus and Networks. Presented at the 28th NORDUnet Conference, Uppsala University, Sweden. Retrieved from https://events.nordu.net/display/NORDU2014/Digital+Assessments%2C+on+Campus+and+N etworks Mogey, N., & Fluck, A. (2014). Factors influencing student preference when comparing handwriting and typing for essay style examinations: Essay exams on computer. British Journal of Educational Technology. http://doi.org/10.1111/bjet.12171 Nielsen, K. G. (2014). Digital Assessment with Students’ Own Device: Challenges and Solutions. Presented at the 28th NORDUnet Conference, Uppsala University, Sweden. Retrieved from https://events.nordu.net/display/NORDU2014/Digital+Assessment+with+Students%27+Own+ Device%3A+Challenges+and+Solutions+-+2 Peregoodoff, R. (2014). Large Scale-Fully Online BYOD Final Exams: Not Your Parents Multiple Choice. Presented at the eAssessment Scotland and Transforming Assessment joint online conference. 11 September. Retrieved from http://transformingassessment.com/eAS_2014/events_11_september_2014.php Ramsden, P. (1992), Learning to Teach in Higher Education, Routledge, New York. Ripley, M. (2007). E-assessment: an update on research, policy and practice. UK: Future Lab. Retrieved FP:141 153 from http://archive.futurelab.org.uk/resources/documents/lit_reviews/Assessment_Review_update. pdf Schulz, A., & Apostolopoulos, N. (2014). Ten Years of e-Exams at Freie Universitat Berlin: an Overview. Presented at the eAssessment Scotland and Transforming Assessment joint online conference. Retrieved from http://transformingassessment.com/eAS_2014/events_19_september_2014.php Terzis, V., & Economides, A. A. (2011). The acceptance and use of computer based assessment. Computers & Education, 56(4), 1032–1044. http://doi.org/10.1016/j.compedu.2010.11.017 Transforming Exams (2014). 'e-Exam System' project, http://transformingexams.com Hillier, M. (2015). To type or handwrite: student's experience across six e-Exam trials. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:131- FP:142) Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:142 154 Predictors of students’ perceived course outcomes in e-learning using a Learning Management System David Kwok Republic Polytechnic, Singapore This study examined the factors that influence students’ perceived course outcomes in e- learning using the Learning Management System (LMS), and the extent to which the factors significantly predict course outcomes. A total of 255 polytechnic students completed an online questionnaire measuring their responses to 5 constructs (lecturer support, interaction with peers, perceived ease of use, perceived usefulness and course outcomes). Data analysis was conducted using structural equation modeling. Results showed that perceived usefulness and interaction with peers were significant predictors of course outcomes, whereas perceived ease of use and lecturer support did not. However, perceived ease of use had an indirect relationship with course outcomes through perceived usefulness. Lecturer support also had an indirect relationship with course outcome through interactions with peers. Overall, the four antecedent variables contributed to 77.0% of the total variance in course outcomes. Based on the study findings, implications for educators and researchers are discussed. Keywords: Course outcomes; e-learning; Learning Management System Introduction Electronic learning (E-learning) is becoming prevalent in tertiary education, with many universities increasing their provision and higher number of students signing up for online learning (Liaw, 2008). The growth in e-learning is attributed to the inherent advantages in terms of manpower, cost, flexibility, and convenience (Ozkan and Koseler, 2009). As Sun, Tsai, Finger, Chen, and Yeh (2008) described, e-learning has ‘liberated’ interactions between learners and educators from the limitations of time and space through the asynchronous and synchronous learning possibilities. The rapid development of information communication technologies (ICT) provides tools to expand and support e-learning in education (Findik Coskuncay & Ozkan, 2013). Higher educational institutions are now reviewing their teaching and learning strategies to adapt new e-learning technologies such as knowledge discovery system, e-collaboration tools, and enterprise information portal to help in achieving their pedagogical goals (Cigdem & Topcu, 2015). However, tapping on the e-learning benefits require an effective and efficient delivery mechanism or Learning Management Systems (LMS) to prepare, operate and manage the e-learning process (Kim & Lee, 2007). The e-learning system can be viewed as having several human and non-human entities interacting together in a LMS environment to achieve the intended course outcomes (Eom, Wen, & Ashill, 2006). As enrolments in e-learning courses continue to increase in higher education, it is pertinent for educators to be aware of the factors that contribute to student success in e-learning. Despite the numerous studies on the various factors that predict successful e-learning (e.g. Johnson, Hornick, & Salas,2008; Sun et al., 2008; Lee & Wong, 2013), few of these studies were conducted in the LMS environment. There is also a plethora of studies that employed student achievement, perceived learning and student satisfaction independently to measure success in e-learning (e.g. Alshare, Freeze, Lane, & Wen, 2011; Eom, Wen, & Ashill, 2006; Lim, Morris, & Yoon, 2006). However, few studies have employed the combined measures of perceived learning and student satisfaction as course outcomes in evaluating successful e-learning. Thus, the major goal of this study is to investigate the factors contributing to the perceived course outcomes in e-learning, as measured by perceived learning and student satisfaction, in a LMS environment. FP:143 155 The reminder of the paper is organised as follows. First, I introduce the background of LMS and the relevant literature related to e-learning success. Second, I present the research model and hypotheses. Next, I describe the research methods and present the results. Finally, I discuss the implications of the findings, along with limitations of the study and future research agenda. Review of Related Literature Background of LMS LMS can be broadly defined as an IT platform used by educators to administer, document, track, report and deliver curriculum to students (Naveh, Tubin, & Pliskin, 2010). While LMS varies in specific functionalities, Coates, James, and Baldwin (2005) described the LMS as an institutional-wide and internet-based systems that typically provides an array of pedagogical and course administrative tools of differing complexities and potentials. A variety of e-tools is typically found in LMS including discussion boards, forum, chat, online grading, online assessment, file sharing, management of assignments, syllabi, schedules, announcements and course plans (Findik Coskuncay & Ozkan, 2013). LMS can be implemented to strengthen e-learning programs that blend in-class teaching and online teaching within the learning process (Cigdem & Topcu, 2015). Despite the increased adoption of LMS by higher educational institutions, there has not been a widespread change in pedagogical practices to take advantage of the functionalities afforded by the LMS (McGill & Klobas, 2009). Consistent with this observation, there is also very little understanding of how the LMS impacts teaching and learning (Coates, James, & Baldwin, 2005). In the recent survey conducted by Educause Center for Analysis and Research (ECAR ) on higher education technology employing 75,000 students and 17,000 faculty from 151 tertiary institutions in USA, it was found that while majority of faculty and students valued the LMS as an enhancement to their teaching and learning, student satisfaction is highest for basic LMS features and lowest for advanced features to foster collaborations and engagement in learning (Dahlstrom, Brooks, & Bichsel, 2014). The study also indicated that one reason why the faculty was not taking advantages of the advanced LMS capabilities was because of no clear evidence to show that technology has a positive impact on student learning outcomes. Despite the numerous studies on LMS that have been conducted in terms of its technology acceptance (De Smet, Bourgonjon, De Wever, Schellens, & Valcke, 2012; Sanchez & Hueros, 2010), and how the use of the LMS is related to teaching and learning (Liaw, 2008; Mijatovic, Cudanov, Jednak, & Kadijevich, 2013), little is known how the LMS could benefit learning and influence student success of e-learning in achieving course outcomes. The following section discusses the literature on e-learning success in a collaborative online learning environment using the LMS. E-learning success research There is a corpus of literature that focuses on the range of factors that influence the use and satisfaction of e-learning systems, and most of these studies were conducted in the context of online collaborative learning (e.g. Arbaugh & Benhunan-Fich, 2007; Kang & Im, 2005; Liaw & Huang, 2007; Marks, Sibley, & Arbugh, 2005). Swan (2001) examined the factors that affect student satisfaction and perceived learning in an asynchronous online learning and found that clarity of design, interaction with instructors, and active discussion among participants significantly influenced student satisfaction and perceived learning. Sun et al. (2008) found that learner computer anxiety, instructor attitude toward e- learning, e-learning course flexibility, e-learning course quality, perceived usefulness, perceived ease of use, and diversity in assessment are critical factors that affect learners’ satisfaction. Arbaugh and Benbunan-Fich (2007) investigated the role of interactions in e-learning, and found that while collaborative environments were associated with higher levels of learner-learner and learner-system interaction, only learner-instructor and learner-system interactions were significantly associated with higher perceived learning. Based on two studies conducted for a sample involving 2196 students using LMSs from 29 Austrian universities, it was found that course content that facilitated self-regulated learning led to higher student satisfaction (Paechter & Maier, 2010), and students’ assessment of the instructors’ e-learning expertise and their counselling and support to the students were the best predictors for student learning achievement and course satisfaction (Paechter, Maier, & Macher, 2010). FP:144 156 Lim, Morris, and Yoon (2006) suggested that course outcomes can be an index for evaluating the quality of an e-learning course. Course outcomes comprise of both cognitive (e.g. learning gains and perceived learning application) and affective (e.g. satisfaction) variables (Lim et al., 2006; Paechter, Maier, & Macher, 2010). User satisfaction is one of the most important factors in determining the success of a system implementation in Information System research (Delone & McLean, 1992). Previous research indicated that student satisfaction is an important outcome that influenced the students’ decision to continue or drop-out of an e-learning course (Levy, 2007). In this study, perceived course outcomes consisting of perceived learning and satisfaction will be employed as the dependent variable, while perceived usefulness, perceived ease of use, lecturer support, and interaction with peers are considered as independent variables. For the purpose of this study, e-learning contents and online learning activities were delivered using the LMS. Hence, the research questions are as follow: 1. What are the factors that significantly influence perceived course outcomes among polytechnic students? 2. To what extent do the factors predict the perceived course outcomes among polytechnic students? Research Model and Hypotheses Perceived Ease of Use Perceived ease of use is “the degree to which a person believes that using a system would be free of effort” (Davis, 1989, p.320). In the case of e-learning system, perceived ease of use was found to directly influence perceived usefulness (e.g. Sanchez & Hueros, 2010; Sumark, Hericko, Pusnik, & Polancic, 2011; De Smet, Bourgonjon, Wever, Schellens, & Valcke, 2012; Lee, Hsieh & Chen, 2013). When learners perceived the e-learning to be easy to use, it is likely that they will be satisfied with the system (Sun et al., 2008; Teo & Wong, 2013). In another study, it was found that when learners perceived an e-learning system is easy to use, they tend to devote more time to learning the contents, thus leading to higher satisfaction (Lee, 2010). The following hypotheses were formulated: H 1 : Students’ perceived ease of use will significantly influence their perceived usefulness of e- learning. H 2 : Students’ perceived ease of use will significantly influence their perceived course outcomes in e- learning. Perceived Usefulness Perceived usefulness is defined by Davis (1989) as “the degree to which a person believes that using a particular system will enhance job performance” (p.320). An e-learning system is perceived to be useful if the learners believe that the system will help them acquire the desired knowledge and skills to perform well in their studies (Teo & Wong, 2013). Studies have found that perceived usefulness has a positive relationship with learners’ satisfaction with the e-learning system (Sun et al, 2008; Teo & Wong, 2013). Therefore, it is hypothesised: H 3 : Students’ perceived usefulness will significantly influence their perceived course outcomes in e- learning. Lecturer Support In e-learning, the lecturer plays a critical role as a facilitator in providing support to troubleshoot and resolve both hardware and software issues (Yuksel, 2009). When learners face problems with e- learning, timely assistance to resolve the problems would encourage the learners to continue with the learning, which include interacting with the peer students and lecturers. Past research had shown that lecturer’s timely response to learners’ needs and problems had significantly influence learners’ satisfaction (Arbaugh, 2002; Thurmomd, Wambach, Connors & Frey, 2002). Hence, the following hypotheses were proposed: FP:145 157 H 4 : Students’ perceived lecturer support will significantly influence their perceived ease of use of e- learning. H 5 : Students’ perceived lecturer support will significantly influence their perceived interaction with peer students in e-learning. H 6 : Students’ perceived lecturer support will significantly influence their perceived course outcomes in e-learning. Interaction with Peers In e-learning, interaction with peers allows learners to share information, receive feedback and evaluate their own learning progress (Piccoli, Ahmad, & Ives, 2001). For instance, when using asynchronous learning tool such as discussion forum, students could post comments, review other students’ comments, and respond to these comments. Over a period of time, such student to student interactions should lead to deeper and broader information processing, more knowledge transfer and deeper learning than if learning is done in isolation (Johnson, Hornik, & Salas, 2008). Marks, Sibley and Arbaugh (2005) found that online student-to-student activities had a positive influence on perceived learning, suggesting that learning is facilitated by communications among the students themselves. Other studies indicated that students’ role in interaction most significantly predict student learning and /or satisfaction (Arbaugh, 2002; Borthick & Jones, 2000; Poole, 2000; Arbaugh & Rau, 2007). Hence, the following hypotheses were proposed: H 7 : Students’ interaction with peers will significantly influence their perceived ease of use with e- learning. H 8 : Students’ interaction with peers will significantly influence their perceived course outcomes with e- learning. H 9 : Students’ interaction with peers will significantly influence their perceived usefulness with e- learning. Method Participants Participants were 255 third-year students of a particular polytechnic taking a blended learning module on Laboratory Management. Among the participants, 160 (62.7%) were females and 95 (37.3%) males. A majority of 154 (60.4%) students were Chinese, 51(20.0%) Malay, 32 (12.5%) Indian and 17(7.1%) other races. The mean age of the participants was 19.88 years (SD = 1.68). All of the participants owned and used laptops in school, and they have access to the LMS to support their e- learning or face-to-face lessons. The e-learning portion of the module included participants taking part in the lecturer-led online forum discussion and completing online quizzes. An LMS was employed to these e-learning activities in this study. Procedures All third-year students who took the Laboratory Management module were invited to participate in the study. For those students who agree to take part in the study, they were given a link to access a website to complete the online questionnaire. All participants were briefed on the purpose of the study, and were informed that their participations were strictly voluntary and anonymity safeguarded. The participants have the rights not to participate or withdraw from the study any time. Participants were also informed that no module credit will be given for participating in the study and their responses do not affect their assessment grades. On average, the respondents took not more than 20 minutes to complete the questionnaire. This research study was approved by the Ethics Review Committee at the institution where the research was undertaken. Measures A questionnaire employed in this study comprised of items adapted from several empirical studies using the e-learning systems or LMS (e.g. Naveh, Tubin, & Pliskin, 2010; Paechter, Maier, & Macher, 2010; Sun et al., 2008; Teo & Wong, 2013). FP:146 158 The questionnaire was pilot tested with a group of students and reviewed by a panel of lecturers for face and content validity. It comprises 15 statements on perceived ease of use (3 items), perceived usefulness (3 items), interaction with peers (3 items), lecturer support (3 items) and perceived course outcomes (3 items). Participants were asked to give their responses to each of the statement on a 5- point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree). When answering the questions in the questionnaire, the respondents were asked to relate their experience using the LMS for the e-learning lessons which they had completed. Demographic data such as gender and age were also collected in the questionnaire. Statistical Analysis The analysis of the study was carried out in two stages using a measurement model and structural model (Anderson & Gerbing, 1998). The first stage involved building a measurement model based on a confirmatory factor analysis (CFA), and examining the descriptive statistics, and assessing the validity and reliability. The second stage involved building a structural equation model of the latent constructs, and testing the hypothesised relationships among the constructs. Results Descriptive Statistics The mean ratings of all the five constructs were between 3.54 and 4.16, and above the mid-point of 3.00 of the scale (see Table 1). This indicated an overall favourable response to the constructs measured in the study. The standard deviations ranged from .09 to 1.17, which revealed a wide spread around the mean. The skewness ranged from - .69 to - .05 and kurtosis ranged from - .40 to .65 were all within Kline’s (2005) suggested cut-offs of absolute values greater than 3 and 10 respectively, indicating univariate normality. The Mardia’s coefficient in this study was found to be 91.95, below the recommended value of 255 (p(p+2) = 15(17) = 255 where p is the number of observed variables in the study) by Raykov and Marcoulides (2012). Hence, multivariate normality is met. Therefore, the data is suitable for the purpose of structural equation modeling. Table 1: Descriptive statistics of the constructs Construct Item Mean SD Skewness Kurtosis Perceived Ease of Use (PE) 3 4.16 1.07 - .45 - .27 Perceived Usefulness (PU) 3 3.81 1.14 - .50 - .08 Lecturer Support (LS) 3 4.61 .97 - .69 .65 Interaction with Peers (IP) 3 3.54 1.17 - .05 - .40 Perceived Course Outcomes 3 4.04 1.06 - .69 .32 (CO) Exploratory Factor Analysis The items were subjected to the principle component factor (PCF) analysis with an oblique (promax) rotation. The Kaiser-Mayer-Olkin (KMO) measure of sampling adequacy was found to be .91, exceeding the recommended threshold for factor analysis of .6 (Tabachnik & Fidell, 2012). Results from the Barlett’s test of sphericity provided further support for performing the EFA: Chi-square, 2 χ (105) = 3147.76, p < .001. The number of resultant five factors was extracted, in line with the specific variables intended to be measured in the proposed research model. The total variance explained by the five factors is 84.06%. All the items had standardised factor loadings of over .60, and the present study accepted this threshold as practical significant (Hair, Black, Babin, Anderson, & Tatham, 2006). FP:147 159 Test of the Measurement Model The measurement model was tested using structural equation modeling (SEM), a multivariate technique that combines factor analysis and multiple regressions to simultaneously examine a series of interrelated dependence relationships among measured variables and latent variables as well as several latent constructs (Hair et al., 2006). Maximum likelihood estimation is used in SEM to generate a full-fledged measurement model and it is a robust estimation method, capable of handling large sample size and distribution that deviates from normality (Arbuckle, 2009). The standardised factor loading of each item on the construct in the measurement model is shown in Table 2. All parameter estimates are significant at the p < .001 level, as indicated by the t-values. The 2 R values for all items are above .50, indicating that the each item explained more than half of the variance of the latent variable (construct) that they belong to. As a measure of internal consistency, the Cronbach alpha values of the constructs, which ranged from .86 to .91 are high, and above the .70 threshold recommended by Nunnally and Bernstein (1994). The fit indices for the measurement model were computed using structural equation modeling with AMOS 18.0 (Arbuckle, 2009). Six fit indices were used to assess the goodness of fit for the 2 measurement model, and these comprise of χ /df ratio; goodness-of-fit index, GFI; comparative fit index, CFI; Tucker-Lewis index, TLI, standardised root mean residual, SRMR and root mean square 2 error of approximation, RMSEA. In order to have an acceptable fit for the measurement model, χ /df is expected to be less than 3.0; GFI, TLI and CFI are expected to exceed .9, and RMSEA and SRMR should be less than .08 (Kline, 2005; Hair et al., 2006). The result showed that there was adequate 2 model fit in the measurement model (χ /df = 2.39; TLI = .95; CFI = .97; GFI = .91; RMSEA = .07; SRMR = .08), which provided support to proceed with testing the structural model. Table 2: Results of the measurement model Latent Item SFL SE t-value R2 AVE Cronbach’s Variable (>.70)* (>.50)* alphas (>.70)* Perceived Ease of .71 .89 Use PE1 .789 .054 15.857** .789 PE2 .889 -a -a .889 PE3 .902 .063 19.632** .902 Perceived Usefulness .72 .91 PU1 .845 .042 22.446** .845 a a PU2 .839 - - .839 PU3 .873 .061 15.803** .873 Lecturer Support .83 .91 LS1 .868 .044 21.091** .868 LS2 .949 -a -a .949 LS3 .835 .048 18.834** .835 Interaction with Peers .64 .86 IP1 .775 .063 15.345** .775 IP2 .894 -a -a .894 IP3 .796 .063 13.887** .796 Perceived Course .72 .90 Outcomes CO1 .825 .049 16.435** .825 CO2 .802 .048 15.264** .802 a a CO3 .903 - - .903 Note: SFL = Standardised Factor Loading; AVE = Average Variance Extracted Average Variance Extracted was computed using (Σλ)2/ (Σλ)2+(Σδ). * Indicate an acceptable level of reliability and validity FP:148 160 ** p < .001 -a This parameter was fixed at 1.00 for specification purposes. Convergent and Discriminant Validities Convergent validity examines whether the respective items are measuring the construct that they purported to measure. The item reliability assessed by its factor loadings of the individual items into the underlying construct was between .78 and .90 (see Table 2). This exceeded the threshold of .70 set by Hair et al. (2006), indicating convergent validity at the item level. The average variance extracted (AVE) is the amount of variance captured by the construct in relation to the variance attributable to measurement error. As recommended by Fornell and Larcker (1981), the AVE is deemed adequate if it is equal or exceeds .50. As shown in Table 2, the AVEs ranged between .64 and .83 for all constructs. These exceeded the threshold value of .50, and hence convergent validity of the constructs is adequate. Overall, convergent validity for all measurement items in this study is adequate. Discriminant validity is the extent to which a construct is absolutely distinct from other constructs (Hair et al., 2006). Discriminant validity was assessed by comparing the square root of the AVE for the given construct with the correlations between that construct and all other constructs. As shown in Table 3, the square root of the AVEs were greater than the off-diagonal numbers in the rows and columns in the matrix, and suggested that the construct is more strongly correlated with its items than with other constructs in the model. Hence, discriminant validity of all constructs is acceptable, and deemed adequate for further analyses. Table 3: Discriminant validity for the measurement model Construct PE PU LS IP CO PE (.84) PU .66** (.85) LS .44** .42** (.91) IP .57** .66** .36** (.80) CO .61** .74** .45** .65** (.85) * p < .01; diagonal numbers in parenthesis indicate the square root of the average extracted variance. Test of the Structural Model 2 Based on the result, the fit indices (χ /df = 2.16; TLI = 0.96; CFI = .97; GFI = .92; RMSEA = .07; SRMR = .07) indicated a good fit with the structural model. Figure 2 shows the resulting path coefficients of the research model. The hypotheses in this study were examined by testing the significant relationships of the variables in the predicted direction. Perceived course outcomes were significantly predicted by perceived usefulness (β = .46, p < .001) and interaction with peers (β = .41, p < .001), but not for lecturer support (β = .11, n.s.) and perceived ease of use (β = .01, n.s.). As for perceived ease of use, interaction with peers (β = .64, p < .001) and lecturer support (β = .20, p < .001) were identified to be significant predictors. In terms of perceived usefulness, perceived ease of use (β = .31, p < .001) and interaction with peers (β = .54, p < .001) were significant predictors. Interaction with peers was significantly influenced by lecturer support (β = .41, p < .001). The results of the hypothesis testing showing the standardised path coefficients and t-values were summarised in Table 5. Out of the total 9 hypotheses, 7 were supported. The explanatory power of 2 the model for individual variables was examined using the resulting R for each dependent variable. Perceived course outcomes are found to be significantly determined by the antecedents, resulting in 2 an R of .765. In other words, perceived ease of use, perceived usefulness, interaction with peers and lecturer support explained 76.5% of the variance in perceived course outcomes. Three other endogenous variables, i.e. perceived usefulness, perceived ease of use and interaction with peers had their variances explained by their determinants in magnitude of 62.9%, 56.7% and 21.2%. FP:149 161 Perceived .54** Usefulness (R2=.63) Interaction .46** with Peers .41** (R2=.21) .64** Perceived .31** Course .46** Outcomes .11(ns) (R2=.77) Lecturer Support .20* .01 (ns) Perceived Ease of Use (R2=.57) Figure 1: Standardised path coefficients in the research model (**p <.001, *p < .01, ns: non-significant) Table 5: Results of hypothesis testing Hypotheses Path Path Standardised t-value Results Coefficient Estimate H1 PE  PU .311 .086 3.602** Supported H2 PE  CO .006 .068 .096 Not Supported (n.s.) H3 PU  CO .395 .075 5.299** Supported H4 LS  PE .201 .061 3.276* Supported H5 LS  IP .475 .071 6.659** Supported H6 LS  CO .095 .044 2.167 Not Supported (n.s.) H7 IP  PE .622 .071 8.715** Supported H8 IP  CO .339 .083 4.102** Supported H9 IP  PU .521 .090 5.786** Supported **p <.001, *p < .01, n.s. refers to non-significant Assessment of Direct, Indirect and Total Effects There are multiple interactions that exist among the four factors that have an influence on perceived course outcomes directly or indirectly. Table 6 shows the direct, indirect and total effects of the exogenous and endogenous variables associated with each of the 5 variables in the study. The total effect on a variable is the sum of the respective direct and indirect effects. Based on Cohen’s (2013) guidelines, standardised estimates (or path coefficients ) with values of less than .1are considered small, less than .3 are medium, and more than .5 are large. Interaction with peers is the determinant of perceived course outcomes with a large total effect of .749, followed by lecturer support, perceived usefulness, and perceived ease of use with total effect sizes of .485, .460 and .151 respectively. As for perceived usefulness, a large total effect of .736 was contributed by interaction with peers, whereas lecturer support and perceived ease of use contributed moderate total effects of .401 and .312 respectively. For perceived ease of use, interaction with peers was a strong determinant with total effect of .639 followed by lecturer support with total effect of .495. FP:150 162 Among the four exogenous variables, perceived course outcomes had the largest amount of variance attributed to the four determinants at approximately 77%. This is largely attributed to the total effects contributed by interaction with peers, lecturer support and perceived usefulness. Table 6: Direct, Indirect and Total Effects of the Research Model Standardised Estimates Outcome Determinant Direct Indirect Total Perceived Course Outcomes (CO) PU .460 - .460 2 (R = .77) PE .008 .143 .151 LS .110 .375 .485 IP .406 .343 .749 Perceived Usefulness (PU) PE .312 - .312 2 (R = .63) IP .536 .199 .736 LS - .401 .401 Perceived Ease of Use (PE) LS .201 .294 .495 (R2 = .57) IP .639 - .639 Interaction with Peers (IP) LS .460 - .460 2 (R = .21 ) Discussion The aims of this study were to investigate the factors that influence students’ perceived course outcomes, and to determine the extent to which the factors significantly predict perceived course outcomes. LMS was employed as a platform to deliver the e-learning in this study. It was hypothesised that perceived course outcomes (CO) as a dependent variable, is predicted by four independent variables on perceived ease of use (PE), perceived usefulness (PU), lecturer support (LS) and interaction with peers (IP). Using structural equation modeling, the research model was tested and the results showed a good model fit with the data. Among the 9 hypotheses tested in the research model, 7 were supported and 2 not supported. The four independent variables accounted for 77% of the total variance in the students’ perceived course outcomes. It is noteworthy that 13% of the variance was not explained and accounted for by the model which suggested a limitation of this study and potential for future research. Except for PE and LS, PU and IP were significant predictors of perceived course outcomes. Except for PU, all the 3 other variables (i.e. LS, PE and IP) had indirect effects on CO. In this study, perceived usefulness had a positive and significant influence on perceived course outcomes. On closer examination, perceived usefulness items had higher and significant correlations with satisfaction item (.63 ≤ r ≤ 0.71, p < .01) than with perceived learning achievements (.57 ≤ r ≤ .63, p < .01) in the perceived course outcomes. One possible explanation for this is that when students perceived the e-learning contents and online activities to be useful in helping them to perform well in their studies, their levels of satisfaction with e-learning would increase and perceived learning achievements higher. The positive and significant influence of students’ perceived usefulness on the satisfaction can be found in a few studies related to the use and adoption of e-learning (Sun et al., 2008; Yuan & Ma, 2008; Teo & Wong, 2013). Interaction with peers had a significant influence on perceived course outcomes. Interaction with peers also had the largest total effect on perceived course outcomes (β = .749, p < .01), compared with 3 other variables. Due to the limited literature on perceived course outcomes, this result is somewhat consistent with previous studies which found that active discussion among students significantly influenced students’ satisfaction and perceived learning (Swan, 2001); learner-learner interactions positively predicted perceived learning (Arbaugh & Rau, 2007), and significantly affect students’ satisfaction (Eom, Wen, & Ashill, 2006). In this study, the results showed that the students perceived that participating in the online discussion forum is critical to learning, and they derived satisfaction through participating in the online collaborative learning activities. FP:151 163 Although perceived ease of use did not have a significant influence on perceived course outcomes, the result suggested that it has an indirect effect on perceived course outcomes through perceived usefulness. Employing the steps used in the mediation analysis recommended by Sobel (1982), the result showed that perceived usefulness is a significant mediator between perceived ease of use and perceived course outcomes (z = 8.64, p < .01), reducing the effect of PE  CO by 94.7%. Hence, the finding indicated that perceived course outcomes are not affected by perceived ease of use alone, however when students perceived e-learning to be useful, the perceived ease of use becomes an important consideration in influencing perceived course outcomes. The results showed that lecturer support is not a significant predictor of perceived course outcomes. Applying the mediation analysis (Sobel, 1982) again, interaction with peers is found to be a significant mediator between lecturer support and perceived course outcomes (z = 5.45, p < .01), reducing the effect of LS  CO by 77.3%. Therefore, lecturer support alone may not exert a significant influence on perceived course outcomes. The instructional roles of the lecturers in supporting students’ learning by providing feedback to the students’ work could be extended through encouraging more students to interact with each other in the online activities, as these could have significant influence on the perceived course outcomes. Limitations and Future Research This study validated a model by testing the factors that significantly predict students’ perceived course outcomes in e-learning using a LMS. However, several limitations of this study should be considered for future research to improve the generalizability of the results. First, the participants of this study were predominantly polytechnic students taking a Laboratory Management module with the School of Applied Science; therefore the results of the study may be only applicable to the population represented. Future studies should extend to multiple modules and student representations. Second, this study employed a particular learning management system to deliver the e-learning, and hence further testing with different LMS using different functionalities could be conducted in future. Third, the study employed a self-reported questionnaire which may be subjected to social desirability bias, where respondents have the tendency to over-report or under-report their responses. Finally, 77% of the total variance in the perceived course outcomes can be explained by the four factors in the study, leaving 13% unexplained. Therefore, there is a need to examine other variables (e.g. learner characteristics, course delivery, facilitating conditions etc.) to improve the predictive power of the model. Implications of the Study Despite the limitations, there are a number of potential implications that this study raises for researchers, educators offering e-learning courses using the LMS. First, the study showed that perceived ease of use, lecturer support and interaction with peers explained 63% of the variance in perceived usefulness. Based on this finding, course developers and lecturers should take these three determinants of perceived usefulness into consideration for the design of e-learning contents and online learning activities. The LMS should be easy to navigate and online contents easily accessible. Course lecturers could play important roles to support student learning by giving them timely feedback on their work, encouraging students to participate more actively during the online discussion, and giving assignment to students where they could work collaboratively online. Second, this study found that interaction with peers had the largest total effect on the perceived course outcomes. Hence, course lecturers could formulate strategies to promote more student- student interactions such as employing peer feedback as an instructional tool for students to evaluate students’ work, exploring the use of leader board functionality in the LMS to give virtual points rewards to motivate students for participating in the online discussion forum, and designing an assessment rubric on the number of postings made by the individual students. Finally, although interaction with peers and perceived ease of use do not have a direct and significant influence on perceived course outcomes, these variables should not be dismissed completed. Through the mediators, interactions with peers and perceived ease of use were found to exert indirect FP:152 164 influence on perceived course outcomes, thus explaining the inter-relationships among the variables in the research model that influence perceived course outcomes. For instance, perceived ease of use had an indirect influence on perceived course outcomes through perceived usefulness. Students’ perception of the ease of use with the e-learning system could enhance perceived course outcomes when they also find that the e-learning is useful to them. Course lecturers could help students to be more effective and productive in their learning by exploring the use of LMS functionalities to design more collaborative and engaging online learning activities for them. Conclusion Based on a theoretical framework, this study proposed and tested a research model that examined the impact of the four factors (i.e. perceived ease of use, perceived usefulness, instructor support, interaction with peers) on perceived course outcomes in e-learning using the LMS among polytechnic students. The study showed that perceived usefulness and interaction with peers were significant predictors of perceived course outcomes, whereas perceived ease of use and lecturer support were not significant. The findings of this study have important implications for educators and researchers to be cognisant of the four key factors, and how these interact with each other, in the instructional design of e-learning courses using the LMS to ensure success in students’ e-learning. Appendix Items Used in the Study Lecturer Support LS1 My lecturer gave me adequate feedback about my comments. LS2 My lecturer supported my learning when the lesson was conducted on LMS. LS3 My lecturer conducted the lesson smoothly using LMS. Interaction with Peers IP1 I used the LMS to communicate with my team members. IP2 LMS helped me to work well with my team members. IP3 I could share information with my team members easily through LMS. Perceived Ease of Use PE1 LMS was easy to use. PE2 LMS was easy to navigate. PE3 I found it easy to get LMS to do what I wanted it to do. Perceived Usefulness PU1 Using LMS would improve my learning in this module. PU2 Using LMS made my learning more productive. PU3 I find LMS useful in my learning. Course Outcomes CO1 I gain new knowledge from the e-learning lessons using LMS. CO2 I have increased my knowledge of the subject using LMS. CO3 Overall, I am satisfied with the e-learning lessons using LMS. References Alshare, K. A., Freeze, R. D., Lane, P. L., & Wen, H. J. (2011). The impacts of system and human factors on online learning systems use and learner satisfaction. Decision Sciences Journal of Innovative Education, 9(3), 437-461. Anderson, J. C., & Gerbing, D. W. (1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 103(3), 411. Arbaugh, J., & Rau, B. L. (2007). A study of disciplinary, structural, and behavioral effects on course outcomes in online MBA courses. Decision Sciences Journal of Innovative Education, 5(1), 65-95. Arbaugh, J. B. (2002). Managing the on-line classroom: A study of technological and behavioral characteristics of web-based MBA courses. The Journal of High Technology Management FP:153 165 Research, 13(2), 203-223. Arbaugh, J. B., & Benbunan-Fich, R. (2007). The importance of participant interaction in online environments. Decision Support Systems, 43(3), 853-865. Arbuckle, J. (2009). Amos 18 user's guide: SPSS Incorporated Armonk, New York, USA. Borthick, A. F., & Jones, D. R. (2000). The motivation for collaborative discovery learning online and its application in an information systems assurance course. Issues in Accounting Education, 15(2), 181-210. Cigdem, H., & Topcu, A. (2015). Predictors of instructors’ behavioral intention to use learning management system: A Turkish vocational college example. Computers in Human Behavior, 52, 22-28. Coates, H., James, R., & Baldwin, G. (2005). A critical examination of the effects of learning management systems on university teaching and learning. Tertiary Education and Management, 11, 19-36. Cohen, J. (2013). Statistical power analysis for the behavioral sciences: Academic press. Dahlstrom, E., Brooks, D. C., & Bichsel, J. (2014). The Current Ecosystem of Learning Management Systems in Higher Education: Student, Faculty, and IT Perspectives: Research report. Louisville, CO: ECAR, September 2014. Available from https://net.educause.edu/ir/library/pdf/ers1414.pdf. Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319-340. De Smet, C., Bourgonjon, J., De Wever, B., Schellens, T., & Valcke, M. (2012). Researching instructional use and the technology acceptation of learning management systems by secondary school teachers. Computers & Education, 58(2), 688-696. DeLone, W. H., & McLean, E. R. (1992). Information systems success: The quest for the dependent variable. Information Systems Research, 3(1), 60-95. Eom, S. B., Wen, H. J., & Ashill, N. (2006). The determinants of students' perceived learning outcomes and satisfaction in university online education: An empirical investigation. Decision Sciences Journal of Innovative Education, 4(2), 215-235. Findik Coskuncay, D., & Ozkan, S. (2013). A Model for Instructors' Adoption of Learning Management Systems: Empirical Validation in Higher Education Context. Turkish Online Journal of Educational Technology-TOJET, 12(2), 13-25. Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 39-50. Green, L. S., & Denton, B. (2012). Examination of factors impacting student satisfaction with a new learning management system. Turkish Online Journal of Distance Education, 13(3), 189-197. Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2006). Multivariate data analysis (6 ed.): New Jersey: Prentice-Hall International. Johnson, R. D., Hornik, S., & Salas, E. (2008). An empirical examination of factors contributing to the creation of successful e-learning environments. International Journal of Human-Computer Studies, 66(5), 356-369. Kang, M., & Im, T. (2013). Factors of learner–instructor interaction which predict perceived learning outcomes in online learning environment. Journal of Computer Assisted Learning, 29(3), 292-301. Kim, S.-W., & Lee, M.-G. (2008). Validation of an evaluation model for learning management systems. Journal of Computer Assisted Learning, 24(4), 284-294. Kline, R. B. (2005). Principles and practice of structural equation modeling (2nd Ed.). New York: Guilford Press. Lee, Y.-H., Hsieh, Y.-C., & Chen, Y.-H. (2013). An investigation of employees' use of e-learning systems: applying the technology acceptance model. Behaviour & Information Technology, 32(2), 173-189. Levy, Y. (2007). Comparing dropouts and persistence in e-learning courses. Computers & Education, 48(2), 185-204. Liaw, S.-S. (2008). Investigating students’ perceived satisfaction, behavioral intention, and effectiveness of e-learning: A case study of the Blackboard system. Computers & Education, 51(2), 864-873. Liaw, S. S., & Huang, H. M. (2007). Developing a collaborative e-learning system based on users’ perceptions. In Computer Supported Cooperative Work in Design III (pp. 751-759). Springer Berlin Heidelberg. Liaw, S.-S., Huang, H.-M., & Chen, G.-D. (2007). Surveying instructor and learner attitudes toward e- learning. Computers & Education, 49(4), 1066-1080. Lim, D. H., Morris, M. L., & Yoon, S.-W. (2006). Combined effect of instructional and learner variables on course outcomes within an online learning environment. Journal of Interactive Online Learning, FP:154 166 5(3), 255-269. Marks, R. B., Sibley, S. D., & Arbaugh, J. (2005). A structural equation model of predictors for effective online learning. Journal of Management Education, 29(4), 531-563. McGill, T. J., & Klobas, J. E. (2009). A task–technology fit view of learning management system impact. Computers & Education, 52(2), 496-508. Mijatovic, I., Cudanov, M., Jednak, S., & Kadijevich, D. M. (2013). How the usage of learning management systems influences student achievement. Teaching in Higher Education, 18(5), 506- 517. Naveh, G., Tubin, D., & Pliskin, N. (2010). Student LMS use and satisfaction in academic institutions: The organizational perspective. The Internet and Higher Education, 13(3), 127-133. Nunnally, J., & Bernstein, I. H. (1994). Psychometric theory: New York, NY:McGraw-Hill. Ozkan, S., & Koseler, R. (2009). Multi-dimensional students’ evaluation of e-learning systems in the higher education context: An empirical investigation. Computers & Education, 53(4), 1285-1296. Paechter, M., & Maier, B. (2010). Online or face-to-face? Students' experiences and preferences in e- learning. The Internet and Higher Education, 13(4), 292-297. Paechter, M., Maier, B., & Macher, D. (2010). Students' expectations of, and experiences in e- learning: The relation to learning achievements and course satisfaction. Computers & Education, 54, 222-229. Piccoli, G., Ahmad, R., & Ives, B. (2001). Web-based virtual learning environments: A research framework and a preliminary assessment of effectiveness in basic IT skills training. MIS Quarterly, 401-426. Poole, D. M. (2000). Student participation in a discussion-oriented online course: A case study. Journal of Research on Computing in Education, 33(2), 162-177. Raykov, T., & Marcoulides, G. A. (2012). An introduction to applied multivariate analysis: Routledge. Sánchez, R. A., & Hueros, A. D. (2010). Motivational factors that influence the acceptance of Moodle using TAM. Computers in Human Behavior, 26(6), 1632-1640. Šumak, B., Heričko, M., Pušnik, M., & Polančič, G. (2011). Factors affecting acceptance and use of Moodle: An empirical study based on TAM. Informatica, 35(1). Sun, P.-C., Tsai, R. J., Finger, G., Chen, Y.-Y., & Yeh, D. (2008). What drives a successful e- Learning? An empirical investigation of the critical factors influencing learner satisfaction. Computers & Education, 50(4), 1183-1202. Swan, K. (2001). Virtual interaction: Design factors affecting student satisfaction and perceived learning in asynchronous online courses. Distance Education, 22(2), 306-331. Tabachnick, B. G., & Fidell, L. S. (2012). Using multivariate analysis (6 ed.): Pearson. Teo, T. (2010). Development and validation of the E-learning Acceptance Measure (ElAM). The Internet and Higher Education, 13(3), 148-152. Teo, T., & Wong, S. L. (2013). Modeling key drivers of e-learning satisfaction among student teachers. Journal of Educational Computing Research, 48(1), 71-95. Thurmond, V. A., Wambach, K., Connors, H. R., & Frey, B. B. (2002). Evaluation of student satisfaction: Determining the impact of a web-based environment by controlling for student characteristics. The American Journal of Distance Education, 16(3), 169-190. Yuen, A. H., & Ma, W. W. (2008). Exploring teacher acceptance of e-learning technology. Asia-Pacific Journal of Teacher Education, 36(3), 229-243. Yuksel, I. (2009). Instructor competencies for online courses. Procedia-Social and Behavioral Sciences, 1(1), 1726-1729. Kwok, D. (2015). Predictors of students’ perceived course outcomes in e-learning using a Learning Management System. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:143- FP:155). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:155 167 Digital leap of teachers: two Finnish examples of rethinking teacher professional development for the digital age Irja Leppisaari Leena Vainio Centria University of Applied Sciences, Omnia, Finland Finland Digitisation and modernisation of education are central objectives in educational policy. This challenges to rethink teaching methods and update teacher pedagogic expertise. This article examines how two Finnish vocational education institutions are supporting transition of teacher professional development to the digital age. The comparison identified similar elements of success and areas for development. Strategic planning and leading of development for a digital leap is the starting point for success. Wireless connections must be universally available to enable use of one's own devices (BYOD). However, the key change factor is teacher transformation. Digital technology has led to professional development models being in a state of transition. Traditional face-to-face methods are not enough to modernise teacher competences. Peer learning, teacher- initiated collaborative development, online training, and use of learning badges will be key methods in teachers taking a digital leap. A promising practice is student-teacher partnerships to change practices for the digital age. Keywords: digitisation of education; teacher professional development; digi-pedagogical competences; pedagogical and technical support; trial culture; peer learning; learning badges Introduction: Digitisation of education Digitisation of education is again a focal point of education development. Faster modernisation of teaching and learning methods is a key recommendation for developing the Finnish higher education system in the report of the International evaluation group (23.3.2015) commissioned by the Ministry of Education and Culture (MOEC, 2015). The significance of digital technology is also emphasised in Finland’s new government programme. Finland aims to be a country characterised by a continuous desire to learn something new, with modern learning environments, and full deployment of digital education and new pedagogy affordances in learning (Ratkaisujen Suomi, 2015). The reason for this investment in digitisation is that the use of ICT in teaching and learning has not expanded as expected. Sitra’s report (2015) shows that in Finland we continue to educate for a bygone world, while an EU study (ICT in Education 2013) reveals that Finnish schools have the lowest information technology utilisation rate in Europe. Finland is an underperformer in the uptake of new digital solutions in Europe (Sitra, 2015). The significance of digital technology in education has not been understood profoundly enough and there is little time left to react. What does digital mean in an education context? Digitisation of education as a term and trend is seen as something that meets contemporary needs, but the term is cumbersome and often understood too narrowly. Therefore, it is necessary to consider how the term digital should be understood in the context of meeting an educational organisation’s goals. Ryymin (2015) observes that digital in an educational context can be defined broadly or narrowly. She argues that defining digital as broadly as possible helps an educational organisation understand how it changes the world. Defining the term narrowly as part of an organisation’s everyday operations helps substantialise what it means in practice (Ryymin, 2015). Ryymin (2015) analyses digital to refer to pedagogically meaningful tools and applications, or in the broader educational ecosystem, an experimental culture that enables open knowledge and the sharing of knowledge. She argues that the most important starting point for digital services is client-orientation and correspondingly in digital education learner- centeredness. Häll and From (2014) in their examination of digital education employ a pedagogic digital competence concept, by which they mean the teacher’s approach and ability to design, deliver FP:156 168 and continuously evaluate the delivered education using digital technology. This is informed by theory, contemporary research and experience, and its purpose is to create an effective learning environment in the best possible way. The teacher needs to be able to manage content, and pedagogic and digital competence. Enhanced and improved educational practice through digital technologies is one of the main features of pedagogic digital competence (Häll & From, 2014). What does digital pedagogic competence look like in the light of current knowledge of Finnish vocational basic and higher education? According to Lampelto (2015), digitisation is already very evident in vocational basic and adult education strategies. However not every education provider has a digital education strategy. Digitisation of education has high status in development work and its benefits include achieving more flexible operational methods, cost effectiveness and increased learning motivation. Challenges include attitudinal factors among the staff and high initial investment costs. Teachers’ attitudes towards digitisation are generally considered fairly neutral. Lampelto’s research indicates that digital competence of senior management, teaching staff and support personnel is at a good level, but greater investment into staff training continues to be necessary. Digital learning material, e-courses and social media are already utilised fairly extensively in vocational basic education. The objective is to increase online education, use of cloud services, develop learning environments, and increase use of mobile devices and the number of development personnel (Lampelto, 2015). Kullaslahti, Karento and Töytäri (2015) studied self-evaluations of teachers’ digital pedagogic competence at three universities of applied sciences. Teachers primarily used digital technology in instructions, delivery of material and as a support in contact teaching. The lowest use of digital technology was in delivery of completely online courses or in RDI ventures implemented together with students. Some teachers worked in networks and had adopted this as an everyday practice. Kullaslahti et al. stress that the cornerstone of pedagogic competence is a comprehensive picture of the solutions and operational methods of digital pedagogy. Their research indicates that teachers feel they lack sufficient competence to produce quality pedagogic digital learning material and online solutions. Teachers continue to require digital pedagogy competence development for them to utilise diverse pedagogic approaches and develop competence based curricula (Kullaslahti et al., 2015). To develop competence both at basic and higher levels, it is increasingly necessary to focus also on developing working-life oriented learning solutions in authentic learning environments, in which learning occurs collaboratively between students, teachers and representatives of working-life (cf. Leppisaari, Kleimola, Maunula & Hothenthal, 2012). Ryymin (2015) concludes that all in all digitisation of education requires complex factors and economic investment, for example, technological infrastructure, user-friendly services and new competences. The examination in this article focuses on new competences by considering the effective factors of and especially the operational models that support transformation of teacher competence, the ’digital leap’, as a whole. Theoretical views in rethinking transformation of teacher professional development for the digital age The digital age requires new models of teacher professional development. Below previous studies will be used to analyse from a professional development viewpoint the factors that affect education digitisation. Digitisation of education refers to changes in culture, operational practice and engagement (OPH 2014). This requires firstly strategic leadership of pedagogic competence in an educational organisation. In examining institutional factors which impact adoption of new technologies in education, Phillips (2005) argues that an educational organisation needs to focus development efforts on three key areas: policy (strategic processes), culture (collaboration, motivation) and support (professional development, IT support) in order to attain results in educational innovation. He emphasises that major factors affecting adoption are, however, human and these can only be addressed through effective leadership and change management (Phillips, 2005). Correspondingly Lampelto (2015) stresses the importance of the teaching staff’s commitment to the design and delivery of new operational models that utilise education technology. In addition to technical skills, commitment to digital education requires changes in ways of thinking and understanding, and operational practices (see Kullaslahti et al., 2015). Teaching and learning methods must be modernised to meet 21st century skills (ATC21S, 2011) and requirements, with particular attention paid to innovative educational practices. Digital competence of students already, on average, exceeds that of a school’s operational practices, setting greater demands on teacher competence requirements and pedagogy. This challenges teachers to update pedagogic expertise, their way of thinking (innovation, problem-solving, learning to learn), ways of working (cooperation, team work) FP:157 169 and management of digital tools (cf. ATC21S, 2011). Research indicates that the digipedagogic competence of teachers can be considered the key question in a successful digital leap. Digitisation irrevocably changes teaching. Merely bringing technology into a school is not enough; rather technology must be used to change practices and learning. This is a question of pedagogy, not devices (Sitra 2015, 12). What operational practices best support attainment of the new competence raised above? Traditional continuous education models are not considered viable solutions in bringing teachers’ competence into the digital age (Leppisaari, Vainio & Herrington, 2009; Kronqvist-Hakola et al., 2015; Teräs, 2014). Brooks and Gibson (2012) conclude that the greatest challenge in teacher professional development has been determining what professional development experiences are most effective for improving teaching and learning. The catalyst for the transformation of education may lie in reimaging professional development as professional learning in a digital age (Brooks & Gibson, 2012). Without changes to the fundamental pedagogical models by which teachers teach and learners learn, technology investments have too often focused on the reproduction of existing content knowledge (Fullan & Langworthy, 2014, 30). The digital leap is promoted if professional development provides teachers an opportunity to experience a new kind of learning partnership both among themselves and with students, and the creation of new knowledge and its purposeful use in authentic contexts is central in their learning processes (cf. Fullan & Langworthy, 2014, pp. 310-311; McLoughlin, 2013). Murray and Zoul (2015) found that personalised, 21st Century professional learning strategies empower teachers to take ownership of their professional learning. Via these kinds of strategies education providers confidently learn to build a values-driven school culture, personalised professional roadmaps, and a collaboration-minded staff (Murray & Zoul, 2015). Teacher professional development for the digital age must be integrated into everyday tasks in authentic learning environments (Leppisaari et al., 2009). Ingvarson et al. (2005) concluded in their study that it is not enough to provide well-designed professional development programmes from outside the school. According to Teräs (2014), earlier research has indicated that successful and transformative professional development is not isolated one-time workshops but collaborative and reflective long-term developmental endeavours that are seamlessly integrated into teaching practice. A good professional development programme engages teachers actively in reflecting on their practice, in identifying specific areas for development, and provides opportunities to test new teaching practices (Ingvarson et al., 2005). The relative success of programmes also depends on the extent to which they are extended in time, and planned so that they include activities that strengthen interaction and collaboration in the school (Brooks & Gibson, 2012). In order to change practice, professional development must also be ongoing, sustained, intensive and supported by modeling and coaching, it must allow educators to see and share their own and student work reflectively and collaboratively, and foster a supportive and inspiring environment for testing new teaching and learning ideas (Ingvarson et al., 2005). When teachers are able to experience a more personalised approach to learning that incorporates contemporary technologies and makes authentic connections to their practice they are more likely to take up a similar approach with their students (Brooks & Gibson, 2012). Collective peer learning and development among colleagues has in fact been seen as a way in which permanent changes are effected in an organisation’s learning and operational cultures. Le Cornu (2005) defined peer mentoring as a collegial, interactive and ongoing sharing of knowledge, experiences and support. This allows individuals to function flexibly, situation-specifically in both the role of learner and teacher. In an organisation, peer learning requires a new kind of operational culture and leadership practices (Leppisaari, Meriläinen, Piispanen & Pulkkinen, 2015; Rongas et al., 2013). McLoughlin (2013) argued that while expectations about digital education have run high, the impact of social media and digital tools in teacher professional learning has been rather limited. Digital methods do, however, enable peer learning and learning to be made visible in professional development. A new kind of learning partnership between teachers, students and working-life representatives is also seen as a pedagogic starting point of digital learning. (Fullan & Langworthy 2014, pp. 310-311). Healey (2015) called for student inclusion in teaching and learning partnerships and their development, which is often forgotten in the peer learning dimension and affordances. The fundamental issue in digitisation of education is the change process of pedagogic operational culture. Change needs to occur simultaneously and be process-based in leadership, technology, teaching and learning. From the above review it can be concluded that in creating opportunities and FP:158 170 supporting teachers in the transition of teaching into the digital age, four interlinked dimensions need to be taken into consideration. These are strategic leadership in the transformation of learning culture, use of digital technology and learning spaces (infrastructure, devices, facilities), supporting teaching transformation, and encouraging and providing time for peer learning. Comparison of two cases - four factors in transforming teacher competence for the digital age This article is a comparison of approaches taken by two institutions to digitise education in Finland. We examine from the viewpoint of teacher professional development how a school is taken into the digital era and what kinds of actions can support transformation of teacher competence. The foci of examination are the actions taken in two Finnish vocational education institutions: one institution represents vocational basic and adult education and the other vocational higher education. In this paper we describe the solutions and operational models these institutions have implemented to support professional development for the digital age, and consider the associated challenges and affordances. Concurrently we analyse the factors which impact education digitisation and compare these from a professional development perspective. The aim is to use two cases to increase understanding of ”teachers’ digital leap” as a phenomena by highlighting and identifying related factors and processes which promote or impede the leap (cf. Denscombe, 2010). Below we briefly introduce the educational institutions in our comparison and present a concise history of ICT use for teaching purposes at our case schools. The focus will however be on describing the actions taken in recent years in digital competence development. 1. Omnia (https://www.omnia.fi/international-omnia) was established to serve the VET needs of people of all ages in three neighbouring cities: Espoo, Kirkkonummi and Kauniainen. Espoo is part of the capital region with a population of over 265,000, most of whom live in the inner urban core of the Helsinki metropolitan area. Omnia has become a pioneer and a catalyst for aligning teaching, learning and digital and other technological solutions to changing classrooms and what goes on in them. In its vision, learning can happen anywhere, be personalised and linked with social learning, cooperative learning, problem-solving and development. Omnia is a regional education development centre with five campuses and 860 staff serving around 50,000 students and learners (10,000 of whom are VET students). Omnia’s services include e.g. the following: 1) An upper secondary vocational school, 2) Vocational adult education and training, 3) Apprenticeship training, 4) A liberal adult education centre for open studies, and 5) A general upper secondary school for adults. Omnia challenges its own staff, and its students to step outside their comfort zones and embrace 21st century learning solutions. Omnia’s vision of the future is that it will be digitised and continually require new knowledge and skills and new forms of teaching and learning anywhere and everywhere in both formal and non-formal settings. 2. Centria University of Applied Sciences (http://web.centria.fi/Default.aspx, further Centria) is a multidisciplinary, dynamic and inter national higher education institution, offering its students and staff an environ ment that is innovative, caring and multicultural. Centria is a small higher education institution in Western Finland, with 3 000 students and 250 staff members. It provides student-centred teaching and learning with plenty of practical experience. Centria offers degree programmes in five different fields: Technology, Business, Social Services and Health Care, Culture and Humanities, and Education. With over 500 international students from around 40 different countries, internationalisation is one of Centria’s core values. Centria profiles as a working-life oriented school supporting development of the region’s business and working-life in accordance with their needs (Centria’s Strategy 2020). Averko eLearning Centre (http://www.averko.fi/eng) began in 1997 as a collaborative network and is today a part of Centria, and its operation supports the objectives of Centria’s Strategy 2020 to develop innovative learning environments. Averko’s 18 years of experience in both producing and conducting online education and R&D is of national significance. Averko offers nearly 60 online courses from different fields with over 200 credits, and over 60 teachers act as tutors on these courses. The main foci of Averko have been the following: coordinating online education at Centria with our degree programmes, staff pedagogical development, and active online pedagogical R&D. Authentic learning which meets the challenges of future working- life, utilises digitisation and crosses boundaries is developed at Centria. FP:159 171 Below we will compare the solutions and operational practices of Centria and Omnia in transforming teaching for the digital age based on the perspectives introduced in the theoretical examination above. The transformation landscape will be shaped on the basis of four dimensions arising from the theoretical literature on teacher professional development. These are: 1) strategic leadership, 2) technology, 3) teaching and 4) peer learning (Fullan & Langworthy, 2014; Phillips, 2005; Brooks & Gibson, 2012; Murray & Zoul, 2015; ACTS21; Ryymin, 2015). Case 1: Omnia 1. Strategic leadership Omnia has purposefully invested initially in developing online education and ICT skills and later in the development of mobile learning. Digital education has been rigorously developed at Omnia through further education for teachers and pilot ventures. The underpinning principle has been the learning- by-doing method, in which digital technology supports learning and helps construction of an authentic learning process. In the initial stages the focus centered on developing basic skills in online education and ICT, but gradually shifted more towards utilisation of social media and mobile devices in teaching and learning. The starting point has been activating students as producers of knowledge and creators of new solutions, which has also changed the role of the teacher into guide and activator. Work based learning methods have been developed in development projects, in which cooperation between working-life and school have been integrated and new technology utilised. In addition to educational institutions, working-life initiated studies are delivered at the workplace or genuine problems derived from working-life are resolved, thereby learning not only vocational competences, but also how to utilise technology and develop 21st century skills important for working-life. Work based learning motivates students. Omnia is endeavouring to move from pilots to a comprehensive change in its operational practices and the digital plan drawn up by the entire staff during 2014 will be rooted into the organisation’s activity with systematic, pedagogic and technical support. Digitisation is strongly present in all Omnia’s strategies and in addition to actual digital developers, the ICT unit, HR unit and pedagogic support staff are engaged in development work. Digital technology is not a discrete area of development, but part of everyday activity. Additionally Omnia’s strategic actions involve including students as digital-support for students and teachers, and constructing a learning material bank and a library to support teachers in the development of digital education. 2. Technology At Omnia wireless connection capacity has been strengthened and cloud services have been taken up (Office 365 and Google Edu) to support the use of modern learning environments and devices in teaching and learning. Teachers and administration staff use Office 365 software in their daily tasks, but teachers are able to incorporate any tools they wish in their teaching, for instance Google apps. Online degrees and blended learning use the Moodle online environment, and Adobe Connect online conference system. Various social media networking tools are also used. Omnia provides all teachers with a laptop and smart phone. Teacher in-service training has improved teachers’ abilities to use various apps. Teachers are not provided tablets, but a limited number is available for class use. These can also be borrowed for teaching purposes through the library. 3. Teaching Omnia develops digital skills through continuous education. Omnia’s digital support organises digital skill workshops every Tuesday afternoon. Half of each session is spent introducing the selected topic and half in practice. Teachers can come to the Tuesday workshops to ask for advice, even though their questions may not relate to the topic-of-the-day. Several trainers are present at each session, ensuring adequate guidance. The first Friday of every month is digital skills day. The digital team is available for nonstop support and degree programmes or teacher teams can invite a support team to their own unit to help in practical teaching problems. Furthermore, training in various development ventures related to topical themes such as digital learning environments, the use of game thinking and game mechanics in solving problems, mobile learning, entrepreneurial teaching and learning, and 3D printing are organised. The design thinking approach, which starts from a teacher’s everyday needs, and not the views of pedagogic support staff or instructors, is used in the gathering of pedagogic support material accumulated through pilots. Teachers can also gain competence required for the digital leap through various externally funded and internal development ventures. These externally funded ventures are often collaborative projects FP:160 172 between other VET schools and the corporate world reflecting the school’s current development strategic needs. Participating fields of study and teachers are agreed on with degree programme heads in the project planning stage. These ventures are 2-3 years in duration. Internal development ventures are systematically called for twice a year and involve teacher teams submitting proposals on how to develop their teaching. Ventures are selected from the applications applying the following selection criteria: the venture involves cooperation, develops new innovative teaching methods and results are disseminated within one’s organisation and beyond. In 2014, 12 ventures were executed at Omnia. Ten new trials were initiated in spring 2015 and a new round of applications will be called for in autumn 2015. Not only have these teacher-initiated pilots motivated teachers to develop skills, they have also served to develop new pedagogic and technical support models, and forms of education. Critical issues and areas of development in digital skills have been identified through the pilots. 4. Peer learning Peer learning methods at Omnia have been integrated as a practice and requirement of education and development ventures. Access to technical and pedagogic support requires supporting colleagues and providing support – the together-we-are-more principle. Peer learning is firmly written into the pedagogic strategy and is a solid educational method in everyday teaching work. Peer learning is also the starting point in Omnia’s internal pilots in which teacher teams develop new innovative teaching methods. A further aim is that teachers and students support each other in employing peer learning methods. The greater the transition to a BYOD environment, the greater the need for reciprocal support; one app is no longer taught, but rather the best tool for different needs is identified and there is collaborative learning to use tools and make learning visible. The value of peer learning emerges from authentic learning, voluntary sharing and also from valuing one’s ability. An expert does not always recognise and appreciate his/her ability. Busyness impedes a critical and reform-oriented examination of working methods. Things are done in the accustomed way. When peers at different stages of their career meet, the result can be new insights. Successful peer work demands attunement, a climate of trust and determination. These are practiced in various contexts together with teachers and students. Students have been included in the development of new working methods and are motivating and guiding the uptake of digital methods. Case 2: Centria 1. Strategic leadership Centria has purposefully invested in developing online education for 18 years. Centria’s open university of applied sciences operational model, Averko, has together with degree programmes produced multidisciplinary courses delivered completely online for its degree students and for the open university of applied sciences. In the new strategic policies, development work increasingly focuses on a wider development of learning environments, blended learning and entire degrees studied on the internet. Centria’s pedagogical strategy (2013) outlines three areas of development: integrated learning environments, working-life oriented pedagogic practices and social learning solutions. Transforming teaching for the digital age is correspondingly examined through three windows of development: authentic learning, community, and digital technology (Leppisaari et al., 2015). Averko’s R&D work into authentic learning has informed pedagogic development work (e.g. Leppisaari et al., 2009; Leppisaari et al., 2012), and fields of study, working-life and development networks cooperatively work in projects designed to enable teachers to take learning into the digital age. In early 2014 an extensive education development venture was initiated. Its guiding principle is to take Centria in its entirety into the digital age. Changes in working-life and digitisation are powerful background drivers. The venture aims to make Centria an environment that values and facilitates new digipedagogic approaches (Learning process, 2014). This strategic activity supports renewing ways of teaching, learning, and study. Online education development is integrated into multidisciplinary pedagogic development. What digital education means in practice at Centria as part of its everyday activity will be demonstrated more clearly through the ongoing pedagogic strategy update and action plan of the soon to be initiated digital team (cf. Ryymin, 2015). A strategic step forward in education digitisation will be taken in autumn 2015 when an online Bachelor of Business Administration programme will be offered. FP:161 173 2. Technology Infrastructure at Centria has been updated, wireless connection capacity has been strengthened and cloud services have been taken up (Office 365) to support the use of modern learning environments and devices in teaching and learning. Online degrees and blended learning use the Optima online environment and Adobe Connect online conference system. Various social media networking tools are also used. Centria has three smart classrooms available as Adobe Connect online conference system physical-virtual learning spaces: each campus has a furnished classroom which is equipped with a video conference system and smart tools. The classrooms are connected to each other, so that in a teaching situation the teacher is present in one classroom and participation in the learning event at other campuses is through the video conference system. Video conference systems enable participation through mobile client: the student or teacher can flexibly participate in the learning situation in real-time irrespective of place. Sessions can be recorded and shared e.g. through the learning environment. Centria provides all teachers with a laptop and smart phone. Teacher in-service training has improved teachers’ abilities to use various apps. Centria doesn’t provide teachers tablets, but a limited number is available for class use. Implementation of the Office 365 learning application is a timely issue and Centria offers this possibility to both teachers and students. 3. Teaching Centria has responded to the challenge of raising the teaching staff’s current level of skills to the level demanded by the digital age by initiating in cooperation with Kokkola University Consortium Chydenius a POD training programme (Update Teaching to the Digital Age) for Centria’s teachers. The 4 credit learning path is spread over three semesters. The teaching staff participates in ten days of social and practical-oriented education and produces a development task in groups of 2-4. The development task is a teaching trial which updates work practices. In total, 102 teachers have participated, about 70 people per training day. The POD further education landscape and pedagogic operational models support teachers in taking a digital leap by modeling key operational forms of the updated pedagogy. The learning path concentrates on clarifying a joint vision of transformed teaching and contemporary education challenges (changing learning environments, digitisation, multiculturalism, authenticity, individual and collective learning, co-teaching). A key objective is to initiate discussion and mirror one’s teaching in relation to these factors from a shared expertise (Leppisaari, Meriläinen, Piispanen & Pulkkinen, 2015). In this way the need-specific solutions and contemporary practices for digipedagogy as defined by Kullaslahti et al. (2015) are created, at whose educational digitisation core is learner-centeredness (cf. Ryymin, 2015). Teacher support activities at Centria have been enhanced by the development of a Service Path and Pedagogic Cards in early 2015. The pedagogic Service Path (1-6 consultative meetings according to the pedagogic process’ progress) and the pedagogic ideas and development cards collected in the virtual learning environment offer teachers support to redesign teaching in online degrees and blended learning to build students’ 21st century skills. A digital team provides various trainings and consultations in pedagogically high quality course design and delivery - from setting competence goals to evaluation and feedback (cf. Kullaslahti et al., 2015). Teachers can also gain competence required for the digital leap through various development ventures. Current ongoing externally funded ventures include for instance MOOC-type further education in the field of renewable energy and e-mentoring at the interface of education and working- life. These are collaborative projects between several universities of applied sciences and the corporate world for creating new kinds of pedagogical practices. 4. Peer learning Supporting peer learning among teachers and sharing good online teaching practices have been part of Averko’s activities since 1997. Furthermore, in 2012-2013 pedagogic afternoons were organised. Their aim was the pedagogic peer mentoring and coaching of staff members. As a collegial and social operational culture strengthens among teachers, it is naturally reflected in the teaching operational culture also and supports the establishment of social learning solutions into everyday teaching as stressed by Brooks & Gibson (2012), for example. Teachers need their own experiences of peer learning and community in order to internalise the importance of these central dimensions of digital pedagogy in their teaching and guidance work. Peer learning was integrally linked to the development task in Centria’s POD training in which teacher groups adopted a work method that reformed teaching and utilised digital technology. Collaborative working methods to complete the development task, the FP:162 174 sharing of the tasks, and their peer evaluation applying authentic learning evaluation criteria (Herrington, Reeves & Oliver, 2010) have, according to feedback, supported teachers in updating digipedagogical skills (Leppisaari et al., 2015). Peer learning and sharing have also been supported by research articles in which teacher groups reflect together on education trials. Comparison of the two cases Professional development of teachers is considered in this article as the key angle of approach and factor in digitisation of education for the digital age. Due to its scale as an area of development, organisations need to engage in development in multiple sectors, and these cannot be examined as discrete or isolated factors (cf. Fullan & Langworthy, 2014; Phillips, 2005; Ryymin, 2015). Taking education into the digital age requires changes in the strategic leadership culture and challenges an organisation to new kinds of structural solutions, decision-making and implementations (cf. Ryymin, 2015). Change needs to happen in leadership, technologies and learning spaces, and teaching and learning. Table 1 describes the digital leap phenomenon as a whole and the link between transformation of teaching and the four key development actions. Dimensions 1 and 2 create the requisites for 3 and 4. Table 1: Centria’s and Omnia’s solutions for taking education into the digital age from the view of transforming teaching for the digital age. Teaching for CENTRIA OMNIA the digital age 1. Averko eLearning Centre since 1997: Since 2000 various pilots and pedagogic STRATEGIC online courses and teaching, strategies stress information and LEADERSHIP production teams and online pedagogy communications technology skills and development work significance of online education Since 2014 Centria’s digitisation Since 2010 Learning solutions development strategy, online degrees and blended team – concentrates especially on learning courses developing use of mobile devices Reform of Averko’s operation and 2014 an organisation-wide digital strategy, initiation of digital team: development focusing on four sub-areas: strategic of innovative, authentic and leadership in learning culture reform, use of multidisciplinary learning environments digital technology (infrastructure, devices and agile production of educational and competence), training and support in content cooperatively with fields of transformation of teachers’ pedagogic study, support services for production competence and encouraging peer learning and delivery of online implementations, and providing time for this. quality assurance and pedagogic quality work, RDI ventures on education and working life interface, peer development 2. Wireless access, cloud services, BYOD, learning environments, cloud TECHNOLOG learning environments, BYOD, smart services, learning material bank, technical Y classrooms, technical support support, tablet hire, wireless access 3. TEACHING POD staff training model and teaching Pilots, in-service teacher training, Support and trials, piloting, development ventures, development ventures, pedagogic support, training in agile content production of online digital support given by students transforming implementations and tutoring support: teaching Service Path and Peda-Cards 4. PEER Pedagogic peer and collective Dissemination of good practices, joint LEARNING development, sharing of teaching trials, competence markets, online support and Encouraging publications cooperation network, blogs, presentations and allowing by experts, students engaged in guidance FP:163 175 adequate time Promoting openness and sharing of and motivation, expert badges for peer good practices learning The comparison demonstrates that both organisations have very similar procedures for developing digitisation. Both have centralised ICT support for teachers and the foci of development are convergent: more extensive technological infrastructure, resources for developing teachers' digipedagogic skills and expansion of digitisation as emphasised by management. Both organisations have observed that traditional methods of further education are no longer adequate; rather new approaches are needed, with peer learning bringing about the best results – the starting point for professional competence is an ability to deal with everyday acute problems. Examined from a strategic leadership perspective, the objective of both Centria and Omnia is to reform practice by doing things in a new and more efficient way. The underpinning values at both schools are reform, competence, flexibility and digitisation (Centria’s Learning process 2014; Omnia’s digital plan 2014). An examination of Omnia’s and Centria’s strategic solutions shows that digital pedagogic competences of teachers have been systematically developed at both institutions (cf. Häll & From, 2014; Kullaslahti et al., 2015). Strategic weighting supports the reform of practice – reform at an entire organisational level requires time and new forms of development. Leadership of change at Centria and Omnia is evident through similar choices of strategies (Centria’s learning process 2014, Omnia’s digital strategy 2014): 1. Update of ICT infrastructure to support digitisation, 2. Updating teacher and other staff skills for the digital age: training and pedagogic support provides views into digital technology affordances for education, and 3. Establishment of digital teams to support digitisation of courses and modules. Several degree programmes and programme sections and MOOC studies delivered either completely or almost entirely online are being developed at Centria and Omnia. The second dimension in transforming teaching into the digital age is technical factors. Both organisations recognise that well functioning wireless connections are the starting point for digitisation, supporting the BYOD operational model. Digitisation can best be implemented in environments in which participants can use their own devices effortlessly. In fact, the digital leap fosters opportunities for using one’s own devices as the current economic situation prevents schools from providing all students with the latest technological equipment. Centria and Omnia see the role of technology as a facilitator of a new kind of pedagogy and new operational methods. Technology is not a question of devices, but people (Sitra 2015, 12). For this reason availability of support is a key factor of success in the digitisation of education. It must be guaranteed in a climate of rapid educational change. Implementation of technology should be systematic and planned. Large organisations must ensure that everyone has access to viable systems cost-effectively and sustainably. Pioneers can try and test new devices and programmes, but user-friendly solutions must be available for basic users, solutions which genuinely support the learning process and ease the teaching work. In vocational education, technology should however be at the forefront. Each course should provide an example of genuine working-life by giving an accurate picture not only of the skills required for an occupation, but also of the digital technology employed in a specific field. Today digital technology includes, for example, 3D-printing, augmented reality, big data, mobile services and the like. Every teacher must be current on field-specific digital technologic trends so that students are provided 21st century skills. Professional development of teachers has been targeted at both schools and developed to include methods which are innovative and utilise peer learning and trial culture, as introduced above. Currently both schools are considering how to deliver professional development in the future. There is an endeavour to involve teachers more rigorously in the planning of their own development (cf. Sitra, 2015). In-service training examined in this paper has primarily been executed as contact teaching with the exception of a few online sessions. Now, however, there is a need to consider if blended models or even entirely web-mediated further education courses would most effectively support teachers’ digital leap (cf. Teräs, 2014). Averko has positive experiences from previous years from its Online tutor e-course. Omnia is working with vocational teacher training institutions to deliver the programme Learning Online, in which digital skills of teachers are developed using online course methods and the teacher receives a learning badge on presenting evidence of competence at each completed level (Oppiminen online, http://www.oppiminenonline.com/in-english/). It would appear that e-courses and learning badges as evidence of competence motivate teachers. These models also support peer FP:164 176 learning. Digital education has led to professional development models being in a state of flux. Even digital education developers are slow to apply in their practice what they teach others. Therefore teachers must be given sufficient time to learn to apply new methods. There is in fact a need to continuously ask how students can be partners and change agents (cf. Healey, 2015) in the development of learning culture and seek ways in which they are more strongly employed as joint- developers of digital education. Discussion Our comparison of how two different educational institutions of different level and size are implementing four dimensions of education that take teaching and learning into the digital age revealed similar solutions, and a convergent direction in education technology and methods. Likewise, the problems and challenges were similar. The comparison between the two schools helps us better understand the notion ”teachers’ digital leap”, and the factors and processes essential for its promotion. The four views we introduced also indicate how reforming teacher competence as the key factor in digitisation of education is a complex phenomenon linked to multiple sub-areas. Both institutions have invested financially in the digitisation of education through updating technological infrastructure and providing user-friendly services and pedagogically meaningful tools (cf. Ryymin, 2015). The financial investment in the new competences is also seen in the scope of Centria’s POD training for teaching staff. Convergent with studies by Teräs (2014) and Eskola- Kronqvist et al. (2015) our study indicated that isolated and discrete training for developing teacher competence do not serve teachers, but rather should be linked to a chain of in-service professional experience. The POD training is one example of a long-term development process linked to a teacher’s work. A learning badge model has been implemented at Omnia, demonstrating that a collective learning process can be formed of parts (cf. Oppiminen online). Professional development discussions in which teachers with their supervisors agree on what education, development ventures and methods are needed to acquire competence form a meaningful path that supports ownership of skill development. Pedagogic and technical support must be organised in such a way that all the different parties are aware of what is available. In addition to a teacher–initiated professional development approach arising from everyday needs, peer learning emerges in our examination as a central method through which educational organisations are brought into the digital age and through which teachers can make a digital leap. The endeavour to do together can be observed as the common factor in both examined cases. A digital leap is promoted through an open working culture, the sharing of good practices and peer learning. Convergent with Eskola-Kronqvist et al. (2015) attitude and a desire for change are pivotal factors in the new competence requirements. Particularly in times of transition they need to be present and positive in order to achieve the desired outcomes and objectives. It is difficult to change attitudinal factors, but it has been observed that the best results are achieved through doing and in collaboration with others. Changes in learning culture at an organisational level are slow, but can occur gradually by changing operational methods (Leppisaari et al., 2015). Common to our cases was support of teacher-initiated trials and a preference for collaborative development. Trialing new things should be made easy (cf. Ryymin, 2015). For example, in Centria’s POD training 23 development tasks can be seen as activators of change, as can Omnia’s 22 teacher-initiated pilots. With their help teachers practice smaller and larger digital leaps that affect an organisation’s learning landscape. Learning culture is reformed by supporting teachers to make changes in their work. Teachers should also have access to support as soon as a problem emerges - peer support from colleagues and students is the most effective and quickest. Including students in the development of digital working methods as support for teachers is seen as a good practice at Omnia. New forms of professional development for teachers require the creation of opportunities. A school facilitates reform of teacher competence by simultaneous attention to the four dimensions presented in this article. A digital leap can be taken with the help of strategic pedagogic leadership, technological support, updating of teaching and peer learning methods. Phillips (2005) argues that the support role of an educational institute can be proactive and change can be led from the middle-out, through operational planning and project management, solving problems and facilitating a connection between FP:165 177 strategic vision and the day-to-day work of teaching in a school. Our examples also demonstrate that every teacher can affect change in his/her situation: ”We cannot wait for change to begin from above or below. It must be everywhere at the same time” (Sitra 2015, 14). Increasingly, in a digitising environment we can utilise the affordances of our digitally connected world as we engage in change. Our purpose in the next stage is to broaden our examination of how conditions for a digital leap and support practices are created to three countries in our researcher network, namely Finland, Australia and Korea. This will allow observation of the effect of cultural factors in this phenomenon. Concurrently an opportunity to benchmark and refine best digital leap practices will be created. References ATC21S (2011). Assessment and teaching of 21st century skills. http://atc21s.org/. Brooks, C. & Gibson, S. (2012). Professional Learning in a Digital Age. Canadian Journal of Learning and Technology 38(2), 1-17. Centria’s Learning Process (2014). Unpublished. Centria University of Applied Sciences. Centria’s pedagogical strategy (2013). Pedagogisen toiminnan strateginen toimintaohjelma 2013- 2016. Unpublished. Centria University of Applied Sciences. Centria’s Strategy 2020. http://web.centria.fi/data/liitteet/db71b0ad03514be9ae70c07dd926793d.pdf. Denscombe, M. (2010). The Good Research Guide. Open University Press. Berkshire: McGraw-Hill Education. Eskola-Kronqvist, A., Mäki-Hakola, H., Mäntylä, R. & Nikander, L. (Eds.) (2015). Opettajat rakennemuutoksessa – muutosta luvassa (Teachers in structural change – change in sight). HAMKin e-julkaisuja 22/2015. Hämeenlinna. Fullan, M. & Langworthy, M. (2014). Rich Seam. How New Pedagogies Find Deep Learning. Pearson. http://www.michaelfullan.ca/wp-content/uploads/2014/01/3897.Rich_Seam_web.pdf. Healey, M. (2015). Students as Change Agents in Learning and Teaching in Higher Education. http://www.mickhealey.co.uk/resources Herrington, J., Reeves, T. C. & Oliver, R. (2010). A Guide to authentic learning. New York and London: Routledge. Häll, L. & From, J. (2014.) Pedagogisk digital kompetens – mellan värden, vetande och kunnande. http://www.epedagogik.eu/pdk-mellan-varden-vetande-och-kunnande/ ICT in Education (2013). Survey of Schools: ICT in Education. Benchmarking access, use and attitudes to technology in Europe’s schools. European Commission. Final Study Report. https://ec.europa.eu/digital-agenda/sites/digital-agenda/files/KK-31-13-401-EN-N.pdf Ingvarson, L., Meiers, M. & Beavis, A. (2005). Factors affecting the impact of professional development programs on teachers' knowledge, practice, student outcomes & efficacy. Australian Council for Educational Research. Education Policy Analysis Archives, 13(10). http://epaa.asu.edu/epaa/v13n10/ Kullaslahti, J., Karento, H. & Töytäri, A. (2015, forthcoming). Ammattikorkeakouluopettajan digiosaamisen vahvistaminen (Strengthening University of Applied Sciences teachers’ digital competence). Lampelto, P. (2015). Ammatillisen koulutuksen digitalisaation nykytilanne koulutuspalveluissa (Current situation of educational services in vocational education digitisation). AMKE ry:n jäsenkyselyn yhteenvetoraportti. Helsinki. Le Cornu, R. (2005). Peer mentoring: Engaging pre-service teachers in mentoring one another. Mentoring and Tutoring, 13(3), 355–366. Leppisaari, I., Kleimola, R., Maunula, M. & Hohenthal, T. (2012). Constructing a Working Life-oriented Model for Online Course Modernization. In Yang, H. & Wang, S. (Eds.) Cases on online learning communities and beyond: Investigations and applications (pp. 267–291). The University of Southern Mississippi. Hershey: IGI Global. Leppisaari, I., Meriläinen, M. Piispanen, M. & Pulkkinen, A. (2015). Päivitä opettajuus digiaikaan - POD-kehittämistehtävät tukemassa oppimiskulttuurin uudistumista (Update teaching to the digitial age – POD development tasks supporting learning culture reform). In J. Viteli & A. Östman (Eds.) Tuovi 13: Interaktiivinen tekniikka koulutuksessa 2015-konferenssin tutkijatapaamisen artikkelit. Research Reports 15. Tampereen yliopisto, Informaatiotieteiden yksikkö, 30-46. Leppisaari, I., Vainio, L. & Herrington, J. (2009). Virtual benchmarking as professional development: Peer learning in authentic learning environments. In Same places, different spaces. Proceedings ascilite Auckland 2009. http://www.ascilite.org.au/conferences/auckland09/procs/leppisaari.pdf FP:166 178 McLoughlin, C. (2013). Teacher professional learning in digital age environments. In M. A. Flores et al. (Eds.) Back to Future: Legacies, Continuities and Changes in Educational Policy, Practice and Research (pp. 189-206). Sense Publishers. MOEC (2015). Towards a future proof system for higher education and research in Finland. Ministry of Education and Culture. http://www.minedu.fi/export/sites/default/OPM/Julkaisut/2015/liitteet/okm11.pdf?lang=en Murray, T. C. & Zoul, J. (2015). Leading Professional Learning: Tools to Connect and Empower Teachers. Corwin. USA: SAGE Company. Omnia’s Digi Plan (2014). Omnian digisuunnitelma. Unpublished. Omnia. OPH 2014. Opetushallituksen asettaman koulutuspilvijaoston loppuraportti (Final report of educational cloud unit commissioned by the Board of Education 20.3.2014). http://bit.ly/1GJ6ZjH Oppiminen online (2015) http://www.oppiminenonline.com/ Phillips, R. (2005). Pedagogical, institutional and human factors influencing the widespread adoption of educational technology in higher education. Mobility: maintaining the momentum? Proceedings ascilite Brisbane 2005. http://www.ascilite.org.au/conferences/brisbane05/blogs/proceedings/62_Phillips.pdf. Ratkaisujen Suomi 2015. Pääministeri Juha Sipilän hallituksen ohjelma 29.5.2015 (Government programme of Prime Minister Juha Sipilä) Hallituksen julkaisusarja 10/2015. Valtioneuvoston kanslia. Rongas, A., Vainio, L. & Honkonen, K. (2013). Peer learning supporting collaboration and creativity in organisations. SeOppi 3, 6-8. Ryymin, E. (2015). Koulutuksen digitalisaatio tapahtuu. Tai sitten sitä johdetaan (Education digitisation happens. Or it is led). Blogikirjoitus 15.5.2015. https://digilearnit.wordpress.com/2015/05/15/koulutuksen-digitalisaatio-tapahtuu-tai- sitten-sita-johdetaan/ Sitra (2015). A land of people who love to learn. New Education forum report. Sitra. Helsinki. Teräs, H. (2014). Collaborative online professional development for teachers in higher education. Professional Development in Education. Routledge. DOI: 10.1080/19415257.2014.961094 Vainio, L. & Petäsnoro, M. (2014). Digital methods form a natural part of everyday activities. SeOppi 3, 8-9. Leppisaari, I. & Vainio, L. (2015). Digital leap of teachers: two Finnish examples of rethinking teacher professional development for the digital age. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:156- FP:167). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:167 179 An enhanced learning analytics plugin for Moodle: student engagement and personalised intervention Danny Yen-Ting Liu Jean-Christophe Froissard Deborah Richards Macquarie University Macquarie University Macquarie University Amara Atif Macquarie University Moodle, an open source Learning Management System (LMS), collects a large amount of data on student interactions within it, including content, assessments, and communication. Some of these data can be used as proxy indicators of student engagement, as well as predictors for performance. However, these data are difficult to interrogate and even more difficult to action from within Moodle. We therefore describe a design-based research narrative to develop an enhanced version of an open source Moodle Engagement Analytics Plugin (MEAP). Working with the needs of unit convenors and student support staff, we sought to improve the available information, the way it is represented, and create affordances for action based on this. The enhanced MEAP (MEAP+) allows analyses of gradebook data, assessment submissions, login metrics, and forum interactions, as well as direct action through personalised emails to students based on these analyses. Keywords: Moodle, learning analytics, students at risk, engagement, indicators, intervention. Introduction Higher education institutions are increasingly offering units in online and blended delivery modes. However, the typical heuristics that staff rely upon to detect disengagement are not readily transferrable to, or available in, the online context. The reduced contact and immediacy makes it more difficult for them to be aware of how their students are engaging (Swan, 2003). At the same time, the ubiquity of learning management systems (LMSs) means that many interactions between students, peers, instructors, and content are captured in databases. The relatively young field of learning analytics (and the closely aligned field of educational data mining) seeks make sense of these and other data to better understand and optimise student learning (Siemens & Baker, 2012). For example, participation in online discussion forums, LMS login frequency, and assessment completion have some predictive value for a student’s final grade (Dawson, McWilliam, & Tan, 2008; Falakmasir & Habibi, 2010; Macfadyen & Dawson, 2010; Smith, Lange, & Huston, 2012; Romero & Ventura, 2013) or engagement (Black, Dawson, & Priem, 2008). Indeed, the majority of work in learning analytics to date has focussed on improving student performance and retention (Arnold & Pistilli, 2012; Romero & Ventura, 2013; Jayaprakash, Moody, Lauría, Regan, & Baron, 2014) by determining variables that are indicative of issues in these areas. To close the analytics loop and enact change, student data need to be appropriately understood and acted upon (Clow, 2012). To this end, a number of staff-facing dashboards that graphically represent student data have been conceptualised and developed (Arnold, 2010; Duval, 2011; Verbert, Duval, Klerkx, Govaerts, & Santos, 2013; Pardo, 2014). These typically seek to assist in deciphering complex student interactions and provide information for decision making processes about learning and teaching (Siemens et al., 2011). Such decisions may involve triggering and sending interventions, facilitated by systems that allow staff to contact students and provide timely advice and feedback (Tanes, Arnold, King, & Remnet, 2011; Mattingly, Rice, & Berge, 2012; Jayaprakash et al., 2014). The learning analytics landscape in Australasian higher education In the Australasian context, a number of higher education institutions are starting to use learning analytics to help students and staff understand and optimise learning. A number of recent Office of Learning and Teaching projects have focussed on constructing institutional frameworks around FP:168 180 advancing learning analytics (Dawson, n.d.; West, n.d.), analysing data from social media interactions (Kitto, Cross, Waters, & Lupton, 2015), and understanding how data can be used by teachers (Kennedy et al., 2014). A recent project supported by Ako Aoteorea involves examining how data from LMSs can be used to answer common learning and teaching design questions (Gunn, Donald, McDonald, Milne, & Nichols, n.d.). A number of institutions have also developed bespoke systems for learning analytics (Atif, Richards, Bilgin, & Marrone, 2013; Siemens, Dawson, & Lynch, 2013). For example, the University of South Australia has staff-facing dashboards reflecting LMS and other online activities (T. Rogers, pers. comm.), while Western Sydney University leverages a commercial business intelligence tool to predict students at risk based on indicator variables (Barwick, 2014). Analysis, identification, and referral systems exist at Edith Cowan University (Jackson & Read, 2012) and the University of New England (Leece & Hale, 2009). Systems that combine analysis and identification with direct student intervention have been developed at Central Queensland University (Beer, Tickner, & Jones, 2014; Jones & Clark, 2014), the University of Sydney (Liu, Bridgeman, & Taylor, 2014), and the University of New South Wales (Siemens et al., 2013). These typically combine data from various sources and allow instructors to contact students through electronic and other means. In addition to these bespoke systems, an alternative approach is to leverage the capability of an institution’s existing LMS to support learning analytics (Sclater, 2014). The two main LMSs in the Australian higher educational sector are Moodle and Blackboard Learn, which together command between 78-90% of the market share (Kroner, 2014). Blackboard Inc. markets the proprietary Blackboard Analytics for Learn, which some institutions such as the University of Sydney, the Western Sydney University, and James Cook University are investigating. Moodle, an open-source LMS used in 222 countries with 1442 installations in Australia (Moodle, n.d.), has a small collection of learning analytics plugins made by its developer community. GISMO is an interactive graphical monitoring tool that helps staff understand how students are interacting with unit resources (Mazza & Milani, 2005). From the same team is MOCLog, which analyses and visually represents log data (Mazza, Bettoni, Faré, & Mazzola, 2012). Similarly, Analytics Graphs graphically summarises students’ access in a Moodle unit (Singh, 2015), while SmartKlass is a nascent staff and student dashboard that tracks online interactions (SmartKlass, 2014). Finally, there is an engagement analytics plugin (Dawson & Apperley, 2012), which is the focus of this paper. The Moodle Engagement Analytics Plugin The Moodle Engagement Analytics Plugin (MEAP; https://moodle.org/plugins/view/report_engagement), originally developed by Phillip Dawson, Adam Olley, and Ashley Holman and released under the GNU General Public Licence, provides staff such as unit convenors (who are academically responsible for a unit of study (or course), also referred to as course coordinators, unit coordinators, or similar) and student support staff with information about how students are engaging with a Moodle unit site based on a range of indicators (Dawson & Apperley, 2012). The original MEAP uses three indicators, which analyse students’ login activity, assessment submission activity, and forum viewing and posting activity to produce a total risk rating (Figure 1). Although some authors have queried the ability of such traces of online activity to fully reflect student learning (Lodge & Lewis, 2012; Gašević, Dawson, & Siemens, 2015), these readily measurable and accessible data from an LMS can provide insight into student engagement (e.g. Black et al., 2008; Lonn, Krumm, Waddington, & Teasley, 2012; Fritz, 2013) and predict performance (e.g. Macfadyen & Dawson, 2010). However, because MEAP can only access Moodle LMS data, users need to be aware of the limitations when configuring and interpreting proxy measures of engagement as represented in the MEAP indicators. To allow customisation of the MEAP analysis for each Moodle unit, the three indicators can be weighted relative to each other according to the perceived relative importance of each activity type to students’ engagement in a particular unit. In addition, each indicator has parameters that allow further customisation. For example, the calculated risk rating for the forum indicator can be set to include parameters around number of posts read, posts created, and replies. Even though the reported total risk rating has predictive value for students’ final grade (Liu, Froissard, Richards, & Atif, 2015), currently MEAP does not offer the same level of functionality as other learning analytics tools such as those with complex visualisations and/or in-built intervention systems (e.g. Beer et al., 2014; Jayaprakash et al., 2014; Liu et al., 2014). FP:169 181 Figure 1: Screenshot of existing MEAP user interface. Aims and research questions There have been a number of frameworks suggested for assessing the functionality and quality of learning analytics approaches. Scheffel, Drachsler, Stoyanov, and Specht (2014) proposed a quality indicator framework around the objectives, learning support, learning measures and output, data aspects, and organisational aspects of learning analytics. Jones, Beer, and Clark (2013) proposed a framework which examined the relevancy of information, meaningfulness of the represented information, the affordances for action based on this information, and the scope for change. We selected this IRAC (information, representation, affordances for action, change) framework to assess and enhance MEAP using a design-based research approach. Initial evaluation suggested that the representation of data as percentage risk ratings lacked direct meaning, and there were no affordances for action. Therefore, working in collaboration with staff who were the intended users of this system, our overall aim was to improve the utility and impact of MEAP for staff and students through applying the dimensions of the IRAC framework. Specifically, the questions we wanted to answer were: (1) what additional information would be meaningful to include in MEAP, (2) how might information be better represented, and (3) how can affordances for action be implemented to allow staff to enact necessary interventions? Methods As our research necessitated working closely with unit convenors and student support staff to design, test, and refine MEAP, we followed a design-based research (DBR) methodology. DBR “integrates the development of solutions to practical problems in learning environments with the identification of reusable design principles” (Reeves, 2006, p. 52) in collaboration with practitioners. Here, we describe research that was situated in practitioner contexts (identification of potentially disengaged students within units), integrating design principles with technology to create solutions (application of the IRAC framework to MEAP), and iterative processes to test and refine the innovations (user testing and evaluation of the enhanced MEAP, MEAP+) (Reeves, 2006). Context We worked together with unit convenors and student support staff at a large metropolitan public university on the east coast of Australia with just under 40,000 students and 3,000 staff. The units investigated were at the undergraduate level with between 59 and 1455 students, delivered through either an online or blended mode. These were selected because their Moodle unit sites consisted of a range of activities which students needed to complete (such as online forums, quizzes, and assignments) and they had a relatively high number of at-risk students (at least 10% non-completion and fail rate in the last study period). Design, development, and testing process To better understand the needs of unit convenors (n = 9) and student support staff (n = 3), they were individually interviewed and asked about how they would measure performance and determine if students were engaged. MEAP was then demonstrated, and staff were asked how they might use it, what the challenges may be, how and when it would be useful, and their needs in a system that could help them contact students. Interview transcripts were coded in NVivo 10 (QSR International) using an inductive approach (Thomas, 2006). Initial codes were identified through review of the terms and concepts found in each of the interviewee’s responses to each question. The interview questions sought to elicit the motivations for using an early alert system, the variables and triggers for identifying students at risk, and how best to FP:170 182 contact students. Additionally, we sought to identify concerns and barriers to using an alert system such as MEAP. Given the focused nature of each question, responses to each question tended to represent a code family, which grouped codes that were related (a process considered to be selective coding). To create the codes and code families, three of the authors independently reviewed the transcripts and for each question proposed a set of codes. The remaining author combined the three sets of codes into the final code families which involved renaming of synonyms, removal of duplication, and some restructuring to clarify relationships (such as “is-a”, “has-a”). After review by the team as a whole, the coding scheme was finalised. Based on the needs analyses from these data, and informed by the IRAC framework, we conceptualised any additional information that staff needed, as well as the interfaces that would allow them to identify and contact students. Simple mockups of the screens that staff would use to do these were produced, and the interview data were used to evaluate these in terms of the information and actions that staff wanted to take. This iterative process refined the mockups, from which functional software prototypes of MEAP+ were developed. We undertook usability testing of MEAP+ prototypes by asking staff to work through typical use case scenarios, a widely used approach in user interface design (Constantine & Lockwood, 2001). Findings from usability testing were used to further refine the prototypes. We present here the results of the user needs analyses, the enhancements to MEAP, and an evaluation of MEAP+ based on user needs and the IRAC framework. Results and discussion User needs analyses Three top-level code families were created: (dis)engagement triggers and indicators, the learning analytics system itself, and actions and responses arising from use of such a system. The themes identified as main (dis)engagement triggers and indicators were class attendance, assessment submissions, forum usage, LMS logins, interim grades, the final exam, access to resources, and interactions with the academic staff. The themes relating to the system itself were frequency and timing of usage, motivations for usage (e.g. improving first year retention), features (e.g. automated notifications to students), and concerns/challenges (e.g. increased workload and selecting benchmarks). For actions and responses, the themes identified were the content of intervention messages (e.g. reason for contact and suggested support), and the mode of delivery (e.g. email or phone). As a results of our analyses, we identified one minor and two major enhancements to MEAP, discussed next. A full analysis will be presented in a future publication. Enhancements to MEAP Minor enhancement to identify students: addition to assessment indicator Like many others, our institution predominantly uses Turnitin submissions instead of native Moodle assignments for receiving student work, which were not detected by the existing MEAP. This enhancement therefore targeted the assessment indicator, augmenting it so that it could additionally identify Turnitin submissions along with quizzes and native Moodle assignments to calculate a risk rating based on whether submissions were absent or late. Major enhancement to identify students: gradebook indicator Needs analyses and consideration of the information dimension of the IRAC framework revealed that MEAP was also unable to analyse the data recorded in the Moodle gradebook, the place where students’ marks for the unit are stored. While interim assessment data are commonly neglected in learning analytics (Clow, 2012), these data can yield valuable information in determining a student’s current academic status. Therefore to address this requirement, we developed an indicator which allowed comparison of gradebook item data against customisable parameters (e.g. quiz 1 mark less than 5/10). Each comparison is associated with a user-defined weighting, which together are used to calculate a risk rating by the gradebook indicator based on which comparisons are triggered (Figure 2). FP:171 183 Figure 2: Screenshot of additional, gradebook indicator allowing items from the gradebook to be queried and compared. Major enhancement to improve information representation and afford contacting students Other questions raised by the IRAC framework, namely the abstracted representation of information and affordances for action, were also supported through the needs analyses. Therefore, to provide a clearer picture of student engagement and address the representation challenges around information abstraction, MEAP+ was developed to display some of the raw information that was otherwise just shown as percentage risk ratings (Figure 3). MEAP+ was also designed to afford action based on provided information, in the form of a student contact system that could deliver customisable and personalisable intervention emails, addressing a key component of the learning analytics cycle (Arnold & Pistilli, 2012; Clow, 2012; Jayaprakash et al., 2014). Emails could be composed from suggested snippets that provided short, specific, formative advice (Croton, Willis III, & Fish, 2014) (Figure 4), and all sent emails were logged to maintain a record of student contact. Figure 3: Screenshot of the information representation in MEAP+. FP:172 184 Figure 4: Screenshot of part of the embedded student contact system. Evaluating MEAP+ from staff perspectives As part of the evaluation process, a project reference group provided feedback on the user experience for MEAP+. This group was constituted of associate deans and directors of learning and teaching from faculties, the head of learning and teaching infrastructure, unit convenors, online teaching coordinators, and student support staff. This group endorsed the developments in MEAP+ and recognised that it was a positive step in providing staff with relevant information that was also directly actionable through the interface. The group requested further rollout within the university to interested staff, who will be contacted through faculty and departmental meetings, ad hoc workshops, and other channels. Based on more widespread usage, we will further investigate the uptake and impact of MEAP+ on students and staff. Evaluating MEAP+ using the IRAC framework Information Currently, MEAP+ is able to consume and display available information on grades and measures of online discussion, assessment submission, and accesses to the unit site. Posts to discussion forums, assessments submitted, and LMS sessions have been correlated with student performance (Macfadyen & Dawson, 2010; Jayaprakash et al., 2014) and are commonly used in learning analytics and educational data mining (Romero & Ventura, 2013). Since performance, often measured as final grade, is calculated from interim (or partial) grades collected during the unit, using these as intermediate variables can potentially provide valuable insights and predictive power (Clow, 2012; Jayaprakash et al., 2014). MEAP+ can access these data as long as they are available within Moodle, but other data that are important in many learning analytics applications such as grade point average, prior academic history, current academic standing, or demographic information (Arnold & Pistilli, 2012; Jayaprakash et al., 2014) are inaccessible. However, the design of the new gradebook indicator within MEAP+ is customisable to the extent that one could conceivably upload these data to the gradebook as manual data points and take advantage of the ability of the gradebook indicator to perform basic comparison analyses (Figure 2). This could also be applied to attendance data, which was identified through the needs analyses and is closely related to student performance (Massingham & Herrington, 2006). Although not developed as part of MEAP+, an attendance indicator that plugs into MEAP is available (https://github.com/danmarsden/moodle-engagementindicator_attendance), drawing data from another Moodle plugin for attendance capture. It is important to recognise that the information available in MEAP+, as well as in most other learning analytics tools, are essentially static counts or averages of user data such as average online session FP:173 185 time, number of forum posts contributed, and delays in assignment submission. These may fail to take into consideration the full complexity of learner activity, paint a limited picture of student engagement and learning, and be difficult to derive relevant interventions and recommendations from (Gašević et al., 2015). An alternate approach to counts and averages of these data involved aggregating and classifying them as a number of interactions between agents, such as student- student, student-content, or student-teacher (Agudo-Peregrina, Iglesias-Pradas, Conde-González, & Hernández-García, 2014). These measures were significantly correlated with final unit grade, and this approach presents another perspective on information that can be made available through learning analytics. Interestingly, this study and others (e.g. Jayaprakash et al., 2014) highlight the importance of unit-independent models, even though differences between learners in different units (Wolff, Zdrahal, Nikolov, & Pantucek, 2013) or the pedagogical design of units (Gašević et al., 2015) may have substantial impact on the accuracy of learning analytics. Further comparative research is therefore needed to determine the value of unit-independent and unit-dependent systems and models, and MEAP+ contributes to evidence of the efficacy of the latter. Representation Representations of information in learning analytics systems are also important to aid analyses and decision making - in particular, being able to understand and use the information are crucial (Jones et al., 2013). Highly abstracted representations such as traffic lights can provide students and staff with a quick indication of progress or predicted risk (Arnold & Pistilli, 2012). More elaborate dashboards can provide visual representations that offer quantified insight into student interactions with resources (Duval, 2011; Pardo, 2014). MEAP also has a traffic light interface, but this may not be as informative for reflecting student disengagement compared to the calculated risk ratings that are used to derive the light colour (Liu et al., 2015). Although the MEAP parameters are presumably determined by an instructor before viewing the risk ratings, this abstraction fails to provide a nuanced representation of student interactions. This is especially important if action will be taken based on an instructor’s understanding and application of these representations. In fact, confusion around percentage risk ratings and the need for less abstraction was seen in the staff interviews. Since feedback with explicit suggestions for improvement are more impactful (Tanes et al., 2011), a more nuanced understanding of information will allow more targeted and valuable feedback to be provided to students. As such, the alternative representation in MEAP+ gives instructors deeper and human-readable visibility of variables that have an existing evidence base around student performance and engagement. Since the aim of representation is to allow a learning analytics user to intuitively understand information in a few seconds (Pardo, 2014), the descriptive summary in MEAP+ is more intelligible than percentage risk ratings, and easier to understand than graphical visualisations. However, these representations are currently not customisable (for example, the instructor cannot choose to show number of replies instead of number of posts), so the importance and impact of this would be an area of future investigation. Affordances for action Action based on available information is a critical and often neglected aspect of the learning analytics loop (Clow, 2012). Specifically, affordances for integrated intervention are needed so that the efficiency and workload barriers to adoption are adequately addressed (Macfadyen & Dawson, 2012; Jayaprakash et al., 2014). For example, the Early Alert Student Indicators project at Central Queensland University integrates the sending of ‘nudges’ directly into the informational interface which helps to encourage engagement between staff and students (Beer et al., 2014). In a similar way, MEAP+ integrates information delivery and affordances of action into one coherent touch point, lowering this barrier for adoption. The composition of the messages themselves is also an important consideration, since their summative or formative nature and motivational or instructional focus impact upon the success of interventions (Tanes et al., 2011). In MEAP+, message composition is supported by ‘message snippets’ which appear as suggestions based on the indicator(s) that is/are flagged as triggering the intervention. We derived some of these snippets from PassNote, a repository of short comments based on research-supported good practice which staff can readily select and use (Croton et al., 2014), and composed a number of snippets ourselves. We are conducting further research on the use and customisation of messages delivered through this system, especially in terms of the content and nature of these interventions and their impact on students. This last point not only reflects the efficacy of MEAP+, but also the ethical implications of intervention-based learning analytics, such as ensuring only positive outcomes for students, recognising student agency and autonomy, and appreciating that student success is complex and unlikely to be causally linked to any one intervention (Slade & Prinsloo, 2013; Sclater, 2015). FP:174 186 Change The IRAC framework allowed us to critically evaluate MEAP in the context of blended or fully online units at our institution to perform the task of assisting staff to identify and contact potentially disengaged students. Based on this, we took advantage of the open source nature of MEAP to undertake one cycle of development (Jones et al., 2013), and have released the resultant MEAP+ back to the open source community to encourage further change informed by wider implementation and development. The source code for the beta MEAP+ is available upon request. Conclusions and future directions Using a design-based research approach, we report the design and development of enhancements to MEAP based on needs analyses involving unit convenors and student support staff, supported through the IRAC framework for learning analytics functionality and quality. We extended the informational reach, improved the representation of data, and provided affordances for action directly within MEAP. Our next goal is to implement and evaluate the impact of MEAP+ in a range of units at our institution, and seek to address wider learning analytics quality indicators such as efficiency, helpfulness, availability, and effectiveness (Scheffel et al., 2014). We will explore how best to support staff to interact with the system, how it may be further modified to optimise the task of identifying and contacting students, and how it should be used to meet the needs and expectations of students. Through this more widespread usage, we will investigate the nature of feedback provided by staff, as well as the impact of these interventions on student success. Acknowledgements The authors would like to thank the unit convenors and student support staff who participated in this research. We would also like to acknowledge the technical support staff who provided advice and assistance. This project was funded by a Macquarie University Teaching Development Grant. References Agudo-Peregrina, Á. F., Iglesias-Pradas, S., Conde-González, M. Á., & Hernández-García, Á. (2014). Can we predict success from log data in VLEs? Classification of interactions for learning analytics and their relation with performance in VLE-supported F2F and online learning. Computers in Human Behavior, 31, 542-550. Arnold, K. E. (2010). Signals: Applying Academic Analytics. Educause Quarterly, 33(1), n1. Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: using learning analytics to increase student success. Paper presented at the 2nd International Conference on Learning Analytics and Knowledge, Vancouver. Atif, A., Richards, D., Bilgin, A., & Marrone, M. (2013). Learning analytics in higher education: a summary of tools and approaches. Paper presented at the 30th Australasian Society for Computers in Learning in Tertiary Education Conference (ascilite 2013), Sydney. Barwick, H. (2014). UWS uses data analysis to track student performance. Retrieved from http://www.computerworld.com.au/article/555606/uws-uses-data-analysis-track-student- performance/ Beer, C., Tickner, R., & Jones, D. (2014). Three paths for learning analytics and beyond: moving from rhetoric to reality. Paper presented at the 31st Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ascilite 2014), Dunedin. Black, E. W., Dawson, K., & Priem, J. (2008). Data for free: Using LMS activity logs to measure community in online courses. The Internet and Higher Education, 11(2), 65-70. Clow, D. (2012). The learning analytics cycle: closing the loop effectively. Paper presented at the 2nd International Conference on Learning Analytics and Knowledge, Vancouver. Constantine, L. L., & Lockwood, L. A. (2001). Structure and style in use cases for user interface design. In M. Harmelen (Ed.), Object modeling and user interface design (pp. 245-280): Addison- Wesley. Croton, B. A., Willis III, J. E., & Fish, J. D. (2014). PassNote: A Feedback Tool for Improving Student Success Outcomes. Retrieved from http://www.educause.edu/ero/article/passnote-feedback-tool- improving-student-success Dawson, P., & Apperley, T. (2012). Workshop 1: Open-Source Learning Analytics and “what the student does”. Paper presented at the SoLAR Southern Flare Conference. Dawson, S. P. (n.d.). Learning Analytics in Australia. Retrieved from http://he-analytics.com/ FP:175 187 Dawson, S. P., McWilliam, E., & Tan, J. P.-L. (2008). Teaching smarter: How mining ICT data can inform and improve learning and teaching practice. Paper presented at the Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education, Melbourne. Duval, E. (2011). Attention please!: learning analytics for visualization and recommendation. Paper presented at the 1st International Conference on Learning Analytics and Knowledge. Falakmasir, M. H., & Habibi, J. (2010). Using educational data mining methods to study the impact of virtual classroom in e-learning. Paper presented at the 3rd International Conference on Educational Data Mining, Pittsburgh. Fritz, J. (2013). Using Analytics at UMBC: Encouraging Student Responsibility and Identifying Effective Course Designs. Retrieved from http://www.educause.edu/library/resources/using- analytics-umbc-encouraging-student-responsibility-and-identifying-effective-course-designs Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64-71. Gunn, C., Donald, D., McDonald, J., Milne, J., & Nichols, M. (n.d.). Building an Evidence-Base for Learning Design Using Learning Analytics Data. Retrieved from https://sites.google.com/site/ldlanz2015/project-team Jackson, G., & Read, M. (2012). Connect 4 Success: A Proactive Student Identification and Support Program. Retrieved from http://fyhe.com.au/past_papers/papers12/Papers/9B.pdf Jayaprakash, S. M., Moody, E. W., Lauría, E. J., Regan, J. R., & Baron, J. D. (2014). Early alert of academically at-risk students: An open source analytics initiative. Journal of Learning Analytics, 1(1), 6-47. Jones, D., Beer, C., & Clark, D. (2013). The IRAC framework: Locating the performance zone for learning analytics. Paper presented at the 30th Australasian Society for Computers in Learning in Tertiary Education Conference (ascilite 2013), Sydney. Jones, D., & Clark, D. (2014). Breaking BAD to bridge the reality/rhetoric chasm. Paper presented at the Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ascilite 2014), Dunedin. Kennedy, G., Corrin, L., Lockyer, L., Dawson, S., Williams, D., Mulder, R., et al. (2014). Completing the loop: returning learning analytics to teachers. Paper presented at the Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ascilite 2014), Dunedin. Kitto, K., Cross, S., Waters, Z., & Lupton, M. (2015). Learning analytics beyond the LMS: the connected learning analytics toolkit. Paper presented at the Fifth International Conference on Learning Analytics And Knowledge. Kroner, G. (2014). LMS Data - The first year update: edutechnica. Retrieved from http://edutechnica.com/2014/09/23/lms-data-the-first-year-update/ Leece, R., & Hale, R. (2009). Student Engagement and Retention through e-Motional Intelligence. UNE: Australia. Retrieved from http://www.educationalpolicy.org/events/R09/PDF/Leece_E- Motion.pdf Liu, D. Y. T., Bridgeman, A. J., & Taylor, C. E. (2014). A simple web-based analytics system to efficiently capture, analyse, and respond to student engagement and achievement. The University of Technology, Sydney: Australian Learning Analytics Summer Institute. Liu, D. Y. T., Froissard, J.-C., Richards, D., & Atif, A. (2015). Validating the Effectiveness of the Moodle Engagement Analytics Plugin to Predict Student Academic Performance. Paper presented at the 2015 Americas Conference on Information Systems, Puerto Rico. Lodge, J., & Lewis, M. (2012). Pigeon pecks and mouse clicks: Putting the learning back into learning analytics. Paper presented at the 29th Australasian Society for Computers in Learning in Tertiary Education Conference (ascilite 2012), Wellington. Lonn, S., Krumm, A. E., Waddington, R. J., & Teasley, S. D. (2012). Bridging the gap from knowledge to action: Putting analytics in the hands of academic advisors. Paper presented at the 2nd International Conference on Learning Analytics and Knowledge, Vancouver. Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588-599. Macfadyen, L. P., & Dawson, S. (2012). Numbers are not enough. Why e-learning analytics failed to inform an institutional strategic plan. Journal of Educational Technology & Society, 15(3), 149-163. Massingham, P., & Herrington, T. (2006). Does attendance matter? An examination of student attitudes, participation, performance and attendance. Journal of university teaching & learning practice, 3(2), 3. Mattingly, K. D., Rice, M. C., & Berge, Z. L. (2012). Learning analytics as a tool for closing the assessment loop in higher education. Knowledge Management & E-Learning: An International Journal (KM&EL), 4(3), 236-247. FP:176 188 Mazza, R., Bettoni, M., Faré, M., & Mazzola, L. (2012). Moclog–monitoring online courses with log data. Paper presented at the 1st Moodle Research Conference. Mazza, R., & Milani, C. (2005). Exploring usage analysis in learning systems: Gaining insights from visualisations. Paper presented at the Communication dans: the Workshop on Usage analysis in learning systems, the twelfth International Conference on Artificial Intelligence in Education. Amsterdam, The Netherlands. Moodle. (n.d.). Moodle Statistics. Retrieved from https://moodle.net/stats/ Pardo, A. (2014). Designing Learning Analytics Experiences. In J. A. Larusson & B. White (Eds.), Learning Analytics (pp. 15-38): Springer. Reeves, T. C. (2006). Design research from a technology perspective. In J. v. d. Akker, K. Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational design research (pp. 52-66). Romero, C., & Ventura, S. (2013). Data mining in education. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 3(1), 12-27. Scheffel, M., Drachsler, H., Stoyanov, S., & Specht, M. (2014). Quality indicators for learning analytics. Journal of Educational Technology & Society, 17(4), 117-132. Sclater, N. (2014). Engagement reporting tools for Blackboard and Moodle. Retrieved from http://sclater.com/blog/engagement-reporting-tools-for-blackboard-and-moodle/ Sclater, N. (2015). Code of practice for learning analytics. Retrieved from https://www.jisc.ac.uk/guides/code-of-practice-for-learning-analytics Siemens, G., & Baker, R. S. (2012). Learning analytics and educational data mining: towards communication and collaboration. Paper presented at the 2nd International Conference on Learning Analytics and Knowledge, Vancouver. Siemens, G., Dawson, S., & Lynch, G. (2013). Improving the quality and productivity of the higher education sector: Society for Learning Analytics Research. Siemens, G., Gasevic, D., Haythornthwaite, C., Dawson, S., Shum, S. B., Ferguson, R., et al. (2011). Open Learning Analytics: an integrated & modularized platform: Society for Learning Analytics Research. Singh, J. (2015). New Block: Analytics graphs. Retrieved from http://www.moodleworld.com/2015/03/31/new-block-analytics-graphs Slade, S., & Prinsloo, P. (2013). Learning analytics ethical issues and dilemmas. American Behavioral Scientist, 57(10), 1510-1529. SmartKlass. (2014). SmartKlass | The Learning Analytics Plugin. Retrieved from http://klassdata.com/smartklass-learning-analytics-plugin/ Smith, V. C., Lange, A., & Huston, D. R. (2012). Predictive Modeling to Forecast Student Outcomes and Drive Effective Interventions in Online Community College Courses. Journal of Asynchronous Learning Networks, 16(3), 51-61. Swan, K. (2003). Learning effectiveness: what the research tells us. In J. Bourne & J. C. Moore (Eds.), Elements of Quality Online Education, Practice and Direction (pp. 13-45). Needham, MA: Sloan Center for Online Education. Tanes, Z., Arnold, K. E., King, A. S., & Remnet, M. A. (2011). Using Signals for appropriate feedback: Perceptions and practices. Computers & Education, 57(4), 2414-2422. Thomas, D. R. (2006). A general inductive approach for analyzing qualitative evaluation data. American journal of evaluation, 27(2), 237-246. Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning analytics dashboard applications. American Behavioral Scientist, 57(10), 1500-1509. West, D. (n.d.). Let’s Talk Learning Analytics and Student Retention. Retrieved from http://www.letstalklearninganalytics.edu.au/ Wolff, A., Zdrahal, Z., Nikolov, A., & Pantucek, M. (2013). Improving retention: predicting at-risk students by analysing clicking behaviour in a virtual learning environment. Paper presented at the 3rd international conference on learning analytics and knowledge. Liu, D.-Y.T., Froissard, J.-C., Richards, D., & Atif, A. (2015). An enhanced learning analytics plugin for Moodle: student engagement and personalised intervention. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:168- FP:177). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:177 189 Prior knowledge, confidence and understanding in interactive tutorials and simulations Jason M. Lodge Gregor Kennedy ARC Science of Learning Research ARC Science of Learning Research Centre Centre Melbourne Centre for the Study of Melbourne Centre for the Study of Higher Education, University of Higher Education, University of Melbourne Melbourne The balance between confidence and understanding can be difficult for students to manage, particularly in digital learning environments where they start with different levels of prior knowledge. The level of prior knowledge and perception of how well understood this prior knowledge is will drive the level of engagement and integration of new knowledge as students are exposed to it. Exploring the relationship between these factors is therefore important for the design of digital learning environments. In this paper we describe two studies examining the levels of confidence and understanding reported by students completing interactive and non-interactive exercises in a digital learning environment. The reported levels of confidence and understanding are then contrasted against pre- and post-test performance and self-reports of the experience completed at the conclusion of the session. The results suggest that students’ prior knowledge influences their confidence and perceived difficulty of the material but does not necessarily influence performance. Keywords: prior knowledge, confidence, simulations The importance of not being too confident Confidence is generally seen as an important trait for individuals in many facets of life. Being confident in work and in social settings has been shown to have significant benefits (Bénabou & Tirole, 2002). Despite this, the evidence for the benefits of high levels of confidence in the learning process is uncertain (e.g. Lester, Garofalo & Kroll, 1989). Research related to judgements of learning, for example, indicates that it is common for novices in many knowledge domains to overestimate their level of understanding (Dunlosky & Rawson, 2012). This is most evident in the Dunning-Kruger Effect (Kruger & Dunning, 1999); the observation that the unskilled are often unaware of being unskilled. What these observations suggest is that it might be more productive to be less confident during learning. These observations allude to a broader need for greater understanding of the role of subjective experiences during the learning process so that more effective digital learning environments can be developed. The aspect of subjective experience that has perhaps been most difficult to research is the role of emotions. Emotion in learning has received renewed attention in recent times (Pekrun & Linnenbrink- Garcia, 2014). Among the many emotional states being investigated, confusion, in particular, seems to play an important role in the process of acquiring new conceptual knowledge (D’Mello, Lehman, Pekrun & Graesser, 2014). Confusion has been a particularly difficult state to examine historically as there has been conjecture about whether it is a purely emotional state, a side effect of cognitive processing or a mixture of both (Rozin, & Cohen, 2003). Researchers have recently settled on the notion of an ‘epistemic emotion’ as an operational description of confusion (D’Mello & Graesser, 2014). In other words, confusion is an affective state directly related to knowledge and knowledge acquisition that provides important cues to the learner in relation to their learning (D’Mello, Lehman et al., 2014). This definition recognises the important role that confusion can play in the process of conceptual change. The normalisation of confusion as part of the learning process could help overcome the problem of overconfidence. Confusion can be seen as a standard part of the conceptual change process in several ways. For example, confusion is particularly beneficial when students need to overcome FP:178 190 misconceptions (e.g. Lehman, D'Mello & Graesser, 2012). Misconceptions about various content areas can occur for several different reasons. New content can be counterintuitive, complex, systemic or novel (D’Mello, & Graesser, 2012). In each of these cases, students need to be able to monitor the strategies they draw on to learn the material and adapt the strategy accordingly. Confusion thus serves as a cue that the strategy they are employing is not effective at acquiring the new knowledge and assimilate it with what they already know (D’Mello & Graesser, 2014). Without recognising this confusion, an overconfident learner will attempt to assimilate new information into existing mental representations that remain misconceived (Cordova, Sinatra, Jones, Taasoobshirazi & Lombardi, 2014). As such, it is evident that overcoming both overconfidence and achieving conceptual change could be contingent on the recognition that there is a mismatch between the new information and the existing mental model, a process most often accompanied by the subjective experience of confusion (D’Mello & Graesser, 2014). Much of the research on confusion in digital learning environments to date has focused on creating adaptive intelligent tutoring systems (e.g. D’Mello, Lehman et al., 2014) that build on recent work in affective computing (e.g. Calvo, D'Mello, Gratch, & Kappas, 2014). This line of enquiry has been useful in helping to better understand how systems can be developed that can provide a more nuanced response to learner progress in digital learning environments than would be possible through modeling based on behavior alone. This research, however, has only begun to uncover the complex relationship between confusion, conceptual change and the mental models learners already have in place, i.e. students’ prior knowledge. The research reported in this paper attempts to address this gap in the research literature with emphasis on learning in digital environments. Confidence, confusion and prior knowledge Confusion is important in the context of the studies described here as it is directly related to the process of conceptual change, particularly in situations where the to be learned knowledge is conceptually complex, counterintuitive or commonly misconceived (see also Lodge, 2015). Previous research has found that misconceptions in certain knowledge domains can be particularly difficult for students to overcome. For example Hughes, Lyddy and Lambe (2013), conducted a thorough overview of the misconceptions in psychology. They argue that some notions, such as schizophrenia being characterised by multiple personalities and the myth that we only use 10% of our brains, are particularly persistent. The existence of persistent misconceptions is evident in many disciplines (Hughes et al., 2013). Of equal importance for overcoming misconceptions is the relationship between confusion and prior knowledge. If confusion is not adequately resolved (i.e. students reach an impasse), it often results in either boredom or frustration (D’Mello & Graesser, 2014). These are the negative side effects of confusion. The implications of these side effects are that students either need to be guided beyond the impasse using effective and timely feedback or scaffolding or need to self-regulate their own learning. If any of these processes break down, it is likely that students will rely on their prior knowledge to make sense of the new information. This, in turn can lead to misconceptions being reinforced rather than updated. Therefore understanding this prior knowledge and how it impacts on the conceptual change process is vital if digital learning environments are to be developed to provide the required interventions needed to help students overcome impasses and confusion. There are numerous ways of creating digital learning environments that can adapt to students’ responses. Digital learning environments provide affordances such as the possibility of providing real- time feedback based on student interaction with the environment (e.g. Kennedy, Ioannou, Zhou, Bailey & O’Leary, 2013; Roll, Aleven, McLaren & Koedinger, 2011). However, the sequencing and timing of the task and the feedback has been traditionally linear and built on the assumption that all students start from the same point. In most disciplines in higher education, there is great diversity in the knowledge students have when they first begin a degree program or subject. Better understanding how this prior knowledge influences the strategies students use, their ability to incorporate new knowledge and the interaction between these factors and their level of perceived confidence and understanding will help to better determine how to do so. To progress previous literature on the emotions and judgements of learning in digital learning environments, this paper focuses on the relationship between these factors. Our aim was to determine whether self-reported confidence and understanding collected while students complete FP:179 191 tutorial and simulation sessions in digital learning environments relates to their post-hoc self-reported experience and performance. Understanding the relationships between these variables is important if we are to provide more nuanced and timely scaffolding and feedback during the learning process in digital learning environments. Study 1 The purpose of study one was to build on the limited research to date examining the roles of confusion and confidence in relation to judgements of learning in a digital learning environment. As the first attempt to do so within a broader program of research, this initial study went about examining these factors in an interactive tutorial that would be perceived as highly difficult for learners unfamiliar with the content (see also Lodge & Kennedy, 2015). This was a deliberate decision in order to ensure that there was a maximum likelihood that participants would find the material confusing. The interactive session used in this first study was based on a session that is used in an undergraduate degree program in biomedical science. In this case however, the study was conducted in a computer laboratory rather than ‘in the wild’. Our reasoning for doing so is that we intend to build on this work to later incorporate multiple measures and indicators for confusion including facial electromyography, electroencephalogram and eye-tracking. Combining the laboratory-style methodology commonly utilised in psychological science with authentic educational material can be a difficult proposition given the different paradigms of research in educational technology and psychological science. As the studies reported here are somewhat novel in this regard, there was an exploratory element to the process described here. Methods Participants Volunteers for this study were drawn from the population of students at The University of Melbourne. An advertisement was placed on the careers website. Students from any disciplinary background were invited to participate. Thirty participants were recruited for this study. Twenty of the participants were female. The mean age of the participants was 23.3 (SD = 4.6) years. Students were studying a range of degree programs. Most commonly, students were admitted into Bachelor of Arts, Bachelor of Commerce or Bachelor of Science degrees. No students reported having significant experience with biomedical science. Participants were compensated with a $20 retail voucher for participating in this study. Materials The experimental sessions were conducted in computer laboratories in the Melbourne Graduate School of Education. The computer-based material was presented on a 21.5 inch iMac computer. All other instruments were printed out for ease of use during the experimental sessions. The tutorial material used for this first study is a module on pharmacodynamics developed for use by students in second year biomedical science. The content is complex in nature and is difficult for novice learners to comprehend given the extensive use of technical terms and assumption that users have one or more full time years experience with concepts and processes in biomedical science. This module was used as we wanted to ensure the maximum likelihood that participants would find the material difficult and potentially confusing. Given the nature of the material and the participants, there should also be low levels of prior knowledge, hence providing a basis from which to understand how prior knowledge (or in this case, a lack thereof) interacts with the other factors of interest. Doing so gives us a solid foundation upon which to explore the relationships between variables in this study. Pre and post-tests were developed with the assistance of a content matter expert in The Department of Medical Education at The University of Melbourne. The pre-test consisted of a series of multiple choice questions covering the full range of material included in the pharmacodynamics module. While participants completed the module, they were asked to fill out a series of questions about their experience during the session. An instrument was developed asking students to respond to each new screen in the module. Three questions were asked in relation to each screen. The first question asked the participants to report their level of confidence that they understood the material. The second question was set out in the same way but asked participants to report their perceived level of FP:180 192 understanding of the material on the screen. They were provided with a visual analogue scale from 0 to 9 with the anchor points at 0 ‘Not confident at all’ / ‘Not challenging at all’, at 5 ‘neutral’ (for both) and at 10 ‘Very confident’ / ‘Very challenging’. A final question asking participants to report their overall experience in a few words was also included for each screen. A questionnaire was developed to both collect demographic details and post-hoc self-reported experiences of the module. Standard age and gender questions were incorporated into the instrument as were a series of questions specifically asking for the emotional reaction participants had to the session. This set of questions was adapted from the retrospective affect judgement protocol developed by Graesser et al. (2006) for their studies on emotion in intelligent tutoring systems. All instruments were given to participants in pencil and paper form. Procedure Participants were told of the nature of the study and completed informed consent paperwork before completing a pre-test of their knowledge about pharmacodynamics. After completing the pre-test, participants were then given access to the pharmacodynamics module. They were instructed to complete the paper and pencil instrument at the conclusion of each screen in the module. Participants were given unlimited time to complete the module. Once complete, they were then asked to fill out the questionnaire and lastly to complete the post-test. At the conclusion of the session, participants were debriefed and informally asked about their experiences using the tutorial and participating this the study. After the data for this study were collected, each set of responses was scored and entered into spreadsheet software for further analysis. Figure 1: Sample screen from pharmacodynamics tutorial Results Participants performed marginally worse than chance on the pre-test (M = 7.63, SD = 2.68). After exposure to the module, the mean score across all participants improved to above chance (M = 10.9, SD = 2.83). The difference between pre- and post-tests was significant, t (30) = 6.97, p < 0.001. FP:181 193 12 10 8 MCQ Score 6 4 2 0 Pre Post Figure 2: Mean pre and post-test scores (SE) with chance performance level emphasised Examining the responses to the questions asked throughout the session, it is apparent that there was some variation in reported levels of confidence and understanding. While there is no direct benchmark to compare these mean responses to, it is apparent that different screen designs led to different response patterns. For example, screen 24 included several interactive elements that relied on consolidation of material presented earlier in the module. This can be compared to screen five, for example, where participants reported being more confident in their understanding and found the screen less challenging. This screen was far less interactive and was predominantly informational in nature. The pattern of responses to these questions can be seen in figure 3. 9 8 7 6 5 4 3 2 Confidence 1 Challenge 0 Screen10 Screen11 Screen12 Screen13 Screen14 Screen15 Screen16 Screen17 Screen18 Screen19 Screen20 Screen21 Screen22 Screen23 Screen24 Screen25 Screen26 Screen27 Screen28 Screen29 Screen1 Screen2 Screen3 Screen4 Screen5 Screen6 Screen7 Screen8 Screen9 Figure 3: Mean confidence and challenge ratings/10 during session (SE) The responses to the post-session questionnaire revealed that participants found the session exciting confusing and enjoyable but relatively less interesting, boring or frustrating. This pattern of responses is presented in figure 4. FP:182 194 6 5 4 3 2 1 0 Excited Interested Confused Enjoyed Bored Frustrated Figure 4: Mean (SE) emotion ratings/10 post-session Discussion The results of this first study suggest several interactions between the variables of interest in this program of research. Despite not being particularly confident and finding the material difficult in the pharmacodynamics tutorial, participants significantly improved their overall performance between pre- and post-test. This improvement was also independent of the fact that students reported little to no previous experience with the content of the module. The results of this first study support previous studies suggesting that student levels of confidence are not necessarily a clear indicator of improved performance. While participants did not feel particularly confident in their learning during the session and reported that the material was relatively difficult, their performance between the pre-test and post-test still improved significantly. It is of course recognised that both the ratings made by participants and their performance are relative. The sample size was also comparatively small for this first study. Our aim with this first study was to induce confusion in a laboratory environment whilst attempting to control for previous knowledge. On that count, the results of this study have been successful. Participants indeed appeared to be confused but their confusion did not appear to impair their capacity for learning, independent of prior knowledge. From here we need to develop a better understanding of how these results apply in diverse environments where prior knowledge is a factor. Study 2 The purpose of study two was to expand on the findings of study one by using content that students are much more likely to have prior knowledge of and to expand the range of environments the research program is interested in. The overall design was similar to that used in study one. There were two main modifications. Firstly, the stimulus material was changed to allow for prior knowledge to have some impact. The tutorial module on pharmacodynamics was replaced with a session on blood alcohol concentration (as per Dalgarno, Kennedy & Bennett, 2014). This module has been effectively used in laboratory-based studies as a proxy for realistic educational material. The module also has two distinct versions; a tutorial version and a simulation version. Participants in the tutorial condition were led through the material in a similar manner to the linear progression available in the pharmacodynamics tutorial used in study one. The simulation condition allowed participants to manipulate variables within the simulation to see how various factors impact on blood alcohol concentration. For a full description of how the module operates, please refer to Dalgarno et al. (2014). Beyond the benefit provided by using established material, the blood alcohol concentration FP:183 195 module afforded the added benefit of tracking the methods used by participants in the simulation condition hence giving insight into how the factors of interest in this research impact on student behavior. Audit trails were collected for this purpose and add further richness to the results of these early forays into the role of confusion, confidence and prior knowledge on student learning. Methods Participants Participants were recruited via the same methods as study one. Fifty participants volunteered for the study. Twenty of the participants were male. The mean age of the participants was 23.1 (SD = 4.6) years. As per study one, participants were most commonly students admitted into Bachelor of Arts, Bachelor of Commerce or Bachelor of Science degrees. Participants were again compensated with a $20 retail voucher. Materials The materials used in the second experiment were broadly the same as those used in the first. The main differences in this second study are that a content area that should be more familiar was used. In this instance, the module to be completed was on blood alcohol concentration. An example screen is displayed in figure 5. Figure 5: Sample screen from blood alcohol concentration simulation A further additional manipulation was added. For the second study, two versions of the module were tested; one, a tutorial version, the second a simulation version. The manipulation was simply that participants in the simulation condition were able to alter the factors associated with blood alcohol concentration (as seen on the left of figure 5.) but participants in the tutorial version were not. In this condition variables were altered between screens and participants watched rather than interacted with the module. Procedure The procedure was broadly the same as that for study one. Participants were given content relevant pre and post-tests on the material, asked to rate confidence and understanding during the session and completed a post-session questionnaire on their experience. Participants in the tutorial condition were instructed to work through the entire tutorial whereas the simulation group was asked to complete a corresponding number of runs through the simulation. This approach corresponded with the procedure used by Dalgarno et al. (2014) in that the strategies used by participants formed part of the analysis. They found that participants using a systematic, as compared to a non-systematic approach, to work through the simulation outperformed others in the simulation and tutorial conditions. To ensure the approach taken by students did not influence performance in the current study, results were analysed in a manner consistent with that of Dalgano et al. FP:184 196 Results The participants in the simulation condition were split on the basis of the strategy they used to work through the simulation. As reported, Dalgarno et al. (2014) found a significant difference between participants who used a systematic approach (varying one factor at a time and seeing the effect) and those who did not use a systematic approach (all other approaches, mostly manipulating the variables haphazardly). Of the 25 participants in the simulation condition, only five could be considered to have used a systematic approach. We have conducted an analysis on these groups with some caution given the sample size and differences between the numbers of participants in each condition. When separating the participants out into the three groups (tutorial condition, simulation condition with systematic approach and simulation condition with non-systematic approach), it is apparent that the participants in each group tended to improve their scores between pre and post-test. The scores for each are presented in figure 6. While it is evident that the mean score in each of the three groups improved significantly from pre-test to post-test, F (1, 47) = 19.99, p < 0.001, there was no main effect for the overall differences between the groups, F (2, 47) = 3.136, p = .053 and no interaction effect, F (2, 47) = .331, p = .720. This means that there was no difference between the groups in terms of their increase in performance between pre-test and post-test. 10 9 8 7 6 5 Pre-test Post-test 4 3 2 1 0 Tutorial Sim Systematic Sim Non-systematic Figure 6: Mean (SE) pre and post-test scores by condition with chance performance level emphasised While there is no statistically significant difference between the conditions, there is a trend towards the enhanced performance in the simulation group using a systematic approach over the other two conditions. Given the difficulty in predicting in advance whether participants will adopt a systematic or non-systematic approach, the failure to obtain a significant difference in scores in this case could be due to insufficient statistical power. As we did not find a significant difference between the participants using a systematic and non-systematic approach in the simulation condition, all further analyses were conducted on the basis of a comparison between tutorial and simulation conditions. Ratings of perceived challenge and confidence in understanding the material followed a different pattern than was evident in study one. Participants were highly confident that they understood the material and reported that is was not particularly challenging. The mean responses to these questions are presented in figure 7. Further analysis of this data interestingly showed no difference in this pattern between the tutorial and simulation conditions. FP:185 197 10 9 8 7 6 Confidence 5 Challenge 4 3 2 1 0 Figure 7: Mean confidence and challenge ratings/10 during session (SE) When examining the post-session responses, tutorial and simulation groups were again considered separately and compared. The mean response scores for each are presented in figure 8. As can be seen in the figure, participants in the simulation condition reported being slightly more interested and slightly less confused, bored or frustrated. Again, these differences were not statistically significant, which may again be an artifact of the size of the sample and a lack of statistical power. 7 6 Tutorial Simulation 5 4 Rating 3 2 1 0 Excited Interested Confused Enjoyed Bored Frustrated Figure 8: Mean (SE) emotion ratings/10 post-session by condition Discussion The results from this second study differ to an extent those of Dalgarno (2014). In this case, we did not find a significant difference in performance between the tutorial and simulation conditions, which also did not extend to a deeper analysis on the differences between participants who used a systematic as opposed to a non-systematic approach. These were not the main areas of focus for the current study so a failure to replicate this previous work is not of concern in this instance. Overall, there were some differences in post-hoc reports of experienced emotions during the session but these also proved not to be statistically significant. What is of interest in this study is that, despite there being negligible differences between the conditions in performance, there was a marked difference in the pattern of responses during the session compared to study one. This is a finding we will delve into further in the general discussion. General Discussion FP:186 198 The two studies presented here were attempts to investigate the interplay between emotion, confidence, perceived understanding and prior knowledge in digital learning environments. The first study used a module that included content that was broadly unfamiliar to the participants who volunteered. Participants reported being simultaneously confused, excited and interested in the study but reported relatively low levels of confidence and a high degree of challenge. While performance improvements in study two followed a similar pattern to those in study one (i.e. the mean scores improved from pre-test to post-test but did not approach ceiling), the responses to perceived challenge and confidence were vastly different between the two studies. As the performance improvements did not appear to differ markedly between the studies, it suggests that prior knowledge influences confidence and perceived difficulty of the learning but may have little impact on student capacity to learn new material. This has implications for the role of confusion and confidence in learning. Prior knowledge could seemingly mediate whether students find the material challenging and feel confident in dealing with it but this judgement could be false. Given that participants in study one felt less confident and reported that they found the module challenging in comparison to the participants in study two but still significantly improved their performance between pre-test and post- test, perhaps they underestimate their capacity for absorbing the new material. Perhaps this feeling is related to them finding the material confusing and attaching a negative value on that experience. Further work is required to determine how these factors contribute to the judgements students make while engaged in the learning process. Across the two studies reported, it is also evident that the combination of emotional reactions to the modules participants worked through are varied and complex. This is perhaps not surprising given that emotional aspects of the learning process are difficult to investigate (Immordino‐Yang & Damasio, 2007). Further studies in this program of research will focus on a wider range of digital learning environments and different methods that will give a fuller picture of the interaction between subjective experience and prior knowledge and the effect of this interaction on learning. For example, in addition to the audit trail data relied upon in the current study to examine behaviour, psychophysiological measures such as facial electromyography (EMG; e.g. Hussain, AlZoubi, Calvo & D’Mello, 2011) and electroencephalography (EEG) can be used as more objective measures of emotional arousal than are available through self report. Conclusions The results we obtained across the two studies presented here could be so for many reasons. As we discussed in the introduction, there is a renewed emphasis on the role of emotion and subjective experience in education. One of the reasons why these factors had previously not received as much attention as they are now is because emotion is complex and varies greatly between individuals. Studying emotions like confusion in relation to confidence, understanding and prior knowledge in digital learning environments is thus a difficult exercise. Our aim with these studies was to make an initial foray into the area by attempting to employ mixed methodologies gleaned from the disparate paradigms of psychological science and educational technology. While this research perhaps raises as many questions as answers, the studies described here provide a solid foundation for further work on the role of prior knowledge, confidence and understanding in learning. What is most evident from these studies is that the interplay between these factors is complex and will require a multidimensional approach to reach conclusive findings that will provide categorical principles for guiding the design of digital learning environments. If digital learning environments are to become truly adaptive and able to provide targeted and personalised scaffolding and feedback, a more complete understanding of these factors will be vital. References Bénabou, R. & Tirole, J. (2002). Self-confidence and personal motivation. Quarterly Journal of Economics, 117 (3), 871-915. doi: 10.1162/003355302760193913 Calvo, R. A., D'Mello, S., Gratch, J., & Kappas, A. (Eds.). (2014). The Oxford handbook of affective computing. Oxford, UK: Oxford University Press. Cordova, J. R., Sinatra, G. M., Jones, S. H., Taasoobshirazi, G., & Lombardi, D. (2014). Confidence in prior knowledge, self-efficacy, interest and prior knowledge: Influences on conceptual change. Contemporary Educational Psychology, 39(2), 164-174. FP:187 199 Dalgarno, B., Kennedy, G., & Bennett, S. (2014). The impact of students’ exploration strategies on discovery learning using computer-based simulations. Educational Media International, 51 (4) 310- 329. doi: 10.1080/09523987.2014.977009 D’Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145-157. D’Mello, S. K. & Graesser, A. C. (2014). Confusion. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education. New York: Routledge. D’Mello, S. K., Lehman, B., Pekrun, R., & Graesser, A. (2014). Confusion can be beneficial for learning. Learning and Instruction, 29, 153-170. Dunlosky, J., & Rawson, K. A. (2012). Overconfidence produces underachievement: Inaccurate self evaluations undermine students’ learning and retention. Learning and Instruction, 22(4), 271-280. Goswami, U. (2006). Neuroscience and education: from research to practice? Nature Reviews Neuroscience, 7(5), 406-413. Graesser, A. C., Witherspoon, A., McDaniel, B., D’Mello, S. K., Chipman P., & Gholson, B. (2006). Detection of emotions during learning with AutoTutor. In R. Son (Ed.), Proceedings of the 28th Annual Meeting of the Cognitive Science Society. Mahwah, NJ: Erlbaum. Hughes, S., Lyddy, F., & Lambe, S. (2013). Misconceptions about psychological science: A review. Psychology Learning & Teaching, 12(1), 20-31. Hussain, M. S., AlZoubi, O., Calvo, R. A., & D’Mello, S. K. (2011). Affect detection from multichannel physiology during learning sessions with AutoTutor. In Proceedings of the The 15th International Conference on Artificial Intelligence in Education (AIED), Berlin: Springer. Immordino‐Yang, M. H., & Damasio, A. (2007). We feel, therefore we learn: The relevance of affective and social neuroscience to education. Mind, Brain, and Education, 1(1), 3-10. Kennedy, G., Ioannou, I., Zhou, Y., Bailey, J., & O’Leary, S. (2013). Mining interactions in immersive learning environments for real-time student feedback. Australasian Journal of Educational Technology, 29(2). doi:10.1234/ajet.v29i2.700 Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121. Lehman, B., D'Mello, S., & Graesser, A. (2012). Confusion and complex learning during interactions with computer learning environments. The Internet and Higher Education, 15(3), 184-194. Lester, F. K., Garofalo, J., & Kroll, D. L. (1989). Self-confidence, interest, beliefs, and metacognition: Key influences on problem-solving behavior. In D. B. McLeod & V. M. Adams (Eds.) Affect and mathematical problem solving. New York: Springer. Lodge, J. M. (2015). Conceptual change in psychology: Why being confused is beneficial for learning. th 50 Australian Psychological Society Conference, Gold Coast, Australia. Lodge, J. M. & Kennedy, G. (2015). Performance and emotional responses in simulation versus tutorial style learning. Australasian Experimental Psychology Conference, 2015, University of Sydney, Sydney, Australia. Pekrun, R., & Linnenbrink-Garcia, L. (2014). International handbook of emotions in education. London: Routledge. Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2011). Improving students’ help-seeking skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruction, 21(2), 267-280. Rozin, P., & Cohen, A. B. (2003). High frequency of facial expressions corresponding to confusion, concentration, and worry in an analysis of naturally occurring facial expressions of Americans. Emotion, 3, 68–75. Notes: The authors wish the thank Michael Lew and Terry Judd from The Department of Medical Education at The University of Melbourne for their invaluable assistance in setting up the experimental protocols used in these studies. The Australian Research Council provided funding for this work as part of a Special Research Initiative for the ARC-SRI Science of Learning Research Centre (project number SRI20300015). Lodge, J. M. & Kennedy, G. (2015). Prior knowledge, confidence and understanding in interactive tutorials and simulations. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:178-FP:188). Note: All published papers are refereed, having undergone a double-blind peer-review process. FP:188 200 The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:189 201 Higher education students' use of technologies for assessment within Personal Learning Environments (PLEs) Lynnette Lounsbury Paula Mildenhall David Bolton Avondale College of Higher Education Edith Cowan University West Chester University Lake Macquarie, New South Wales Western Australia Philadelphia, USA Maria Northcote Alan Anderson Avondale College of Higher Education University of New England Lake Macquarie, New South Wales Higher education students' use of technologies has been documented over the years but their specific use of technologies for assessment-related tasks has yet to be fully investigated. Researchers at two higher education institutions recently conducted a study which sought to discover the technologies most commonly used by students within their Personal Learning Environments (PLEs). A specific aim of the study was to determine which of these technologies the students used when they complete and submit assessment tasks such as assignments and examinations. Results from questionnaires, focus groups and mapping exercises are reported and the implications of the findings for developing institutional infrastructure to engage students and support their learning are highlighted. Keywords: assessment, student use of technologies, Personal Learning Environments (PLEs) Introduction Students enrolled in tertiary courses typically use a range of technologies in their personal lives and for study purposes including social media, hand-held and mobile devices, software applications and online technologies; and these technologies have been documented over some years (for example, Conradie, 2014; Gosper, Malfroy, & McKenzie, 2013; Gosper, McKenzie, Pizzica, Malfroy, & Ashford- Rowe, 2014; Johnson & Sherlock, 2014). As a collection, the interplay of these technologies make up a student's Personal Learning Environment (PLE). For the purposes of this paper, the authors have used previous definitions of a PLE by various researchers (Dabbagh & Kitsantas, 2012; Fiedler & Väljataga, 2010; Goldstein & Miller, 1976) to construct the following definition. A PLE is a system, usually self-constructed, that enables learners to manage their own learning and may include technological tools, services, online resources and communities. Higher education students' use of technologies within their PLEs influence how they engage in their university studies. Analysis of students' PLEs is useful as they are situated within and reflective of the specific contexts in which the students' learning takes place. Learning within a PLE is often informal (Attwell, 2007b); that is “unstructured learning within a structured learning environment” (Harvey, 2015). Cross (2006) describes this type of learning as “taking part in meaningful conversations, listening to and telling stories, building personal trust networks that yield advice quickly”. This is in contrast to formal learning which has traditionally being the focus in higher education contexts and is described as “planned learning that derives from activities within a structured learning setting” (Harvey, 2015). When investigating PLEs, informal learning becomes important as well as the more formal environments offered by an institution's Learning Management System (LMS) (Taraghi, Ebner, Till, & Mühlburger 2009). Because many tertiary students' study practices are associated with assessment tasks (for example, assignments, presentations, examinations), their use of specific technologies for assessment purposes within their PLEs needs investigation. Each student's PLE generally comprises diverse and changing technologies, that are reliant upon their varied activities and purposes. As such, a research FP:190 202 approach which focuses on documenting the technologies used within tertiary students' PLEs may provide insight into how university educators could design relevant, contextualised courses and assessment processes that utilise students' current use of technology (Jenkins, Walker, & Voce, 2014). Curriculum design that reflects students' use of technology has been reported as being an important issue by Könings, Brand‐Gruwel and van Merriënboer (2005). Use of PLEs has also been associated with supporting self-regulated learning practices (Dabbagh & Kitsantas, 2012), learner empowerment (Drexler, 2010) and students' participation in learning and teaching processes (Attwell, 2007a). Since many processes involved in preparing assessment tasks require students to work independently, this study sought to investigate the technologies used within students' PLEs during assessment preparation, completion and submission processes. The research reported in this paper particularly focused on two groups of undergraduate students in two higher education institutions in Australia. While much research has been conducted on the technologies students use during their leisure time and during their university studies in general (for example, Castaneda & Soto, 2010; Gosper et al., 2013; Gosper et al., 2014; Hight, Khoo, Cowie, & Torrens, 2014; Wang, Niiya, Mark, Reich, & Warschauer, 2015), less is known about the technologies used by higher education students during the specific processes of preparing, completing and submitting assessment tasks as required components of their university degrees. Background The definition of a PLE has evolved since the first notions emerged of students using technology to learn (Goldstein & Miller, 1976). Whilst there is not necessarily only one way to describe a PLE at present (Fiedler & Väljataga, 2010), researchers are beginning to develop various ways of defining this emerging concept. Attwell (2007b), for example, describes a PLE as being "comprised of all the different tools we use in our everyday life for learning" (p. 4). Dabbagh and Kitsantas (2012) describe a PLE as a "potentially promising pedagogical approach for both integrating formal and informal learning using social media and supporting student self-regulated learning in higher education contexts" (p. 3). Due to the multiplicity of understandings about learning, it is important to acknowledge that social constructivist learning theory clearly describes the type of learning that takes place within a PLE (van Harmelen, 2008; Wild, Mdritscher, & Sigurdarson, 2008). One reason is that the learning environment offered by a PLE provides scaffolding for the learner which is an important component of this theory. The interactive aspect of working in the social media environment allows students a level of personalisation to their learning that frames their overall learning experience. The shared environment promotes levels of engagement and management, from content sharing, to collaborating, through to aggregation and finally to synthesis (Dabbagh & Kitsantas, 2012). Additionally, participating in a social network is at the heart of a PLE and social constructivism theory indicates that learning takes place within a community of practice (Vygotsky, 1933/1978). There is some consensus around the emerging understanding of a PLE. One view is that a PLE encompasses the concept of a learner that is not restricted to the institutional community and formal learning networks but instead able to access a much broader community of practice (Dabbagh & Kitsantas, 2012; Fiedler & Väljataga, 2010; Wild et al., 2008). A PLE is underpinned by the idea of an independent learner who is actively involved in their own learning (van Harmelen, 2008). Whilst previously the LMS was at the centre of student learning experiences (Gosper et al., 2013), this research explored how multiple technologies may work together to form students' PLEs. A PLE, then, is clearly broader than the LMS and has the potential to cater for today’s learner who needs flexibility to utilise all available components of their learning environment (Taraghi et al., 2009). A PLE is defined as an approach to learning in which an individual uses tools of technology to acquire new knowledge and skills within dedicated and non-dedicated settings (Attwell, 2007b). The environment is personal in that each individual may use different tools to learn. The terms "dedicated" and "non-dedicated" are used in place of "formal" and "informal" to acknowledge that formal and informal learning can occur within dedicated settings, as well as non-dedicated settings (Smith, 1988). For example, when taking a course, a student can learn what the teacher is teaching, that is, the objectives or learning outcomes of the course. But within this dedicated setting, a student can also learn other information about the topic being taught which is not necessarily part of the formal FP:191 203 structure of the course. Conversely, even outside of a structured learning environment, there may be more formal learning happening, as when a person uses a language app to study a foreign language. While many learners have traditionally used an LMS it is important to consider how the shift to a PLE occurs (Wild et al., 2008). Taraghi et al. (2009) defined crucial aspects for the shift from a LMS to a PLE as including: personalisation, content, social involvement, ownership, educational and organisational culture and technological aspects (p. 2). If these support networks are to be created it is vital that curriculum designers are aware that learners need digital literacy skills to establish a PLE; they also must be aware how learners interact with tools, artefacts and their social network (Wild et al., 2008). The technologies that underpin the PLE typically comprise informal learning environments and networks that encompass unstructured learning, as defined earlier in the paper. Because the technologies used by college and university students are constantly changing, more contemporary research is required in this field. As technology has become more complex, the technology encompassed has increased from the simple computer program (Goldstein & Miller, 1976) to including new, flexible technologies. Examples of these technologies are tablets, smart phones, laptops and web services (van Harmelen, 2008). Integral to PLEs are Web 2.0 technologies denoting a new generation of web-based tools, environments, and services that enable new forms of collaboration and knowledge sharing between users (Margaryan, Littlejohn, & Vojt, 2011). Web 2.0 technologies are as much a concept as they are a technology. As a concept they characterise the ideas of openness, personalisation, customisation, collaboration, social networking, social presence and user-generated content. As a technology, they represent the second generation of technology available on the internet. The qualitative shift represented by this change allows anyone with an internet connection to access and edit a website, to be involved in a wiki or a blog, and to connect with other users. Such technology also provides opportunities to extend and enhance human communication capabilities. Dabbagh and Kitsantas (2012) suggest that social media can facilitate the creation of PLEs to help learners aggregate and share the results of learning achievements, participate in collective knowledge generation and manage their own meaning making. They also describe a pedagogical framework that lecturers can employ to demonstrate how social media can be used to create these PLEs while also promoting learner-centred pedagogy and facilitating self- regulated learning. As well as the benefits of social media and Web 2.0 technologies, students’ PLES will be shaped by their need to fulfill assessments task requirements in higher education. The focus in our study on students’ use of their PLEs in assessment is important as assessment in higher education drives learning: For most students, assessment requirements literally define the curriculum. Assessment is a potent strategic tool for educators with which to spell out the learning that will be rewarded and to guide students into effective approaches to study. Equally, however, poorly designed assessment has the potential to hinder learning or stifle curriculum innovation (James, McInnis, & Devlin, 2002, p. 7). Overall there has been a lack of theoretical perspectives of assessment in higher education (Yorke, 2003) and this trend appears to extend to considering the role of assessment in students' PLEs. Some research has been conducted into the PLEs used by school students (Clark, Logan, Luckin, Mee, & Oliver, 2009) and higher education students (Valjataga & Laanpere, 2010) but more work is required to determine the types of technologies used by college and university students when preparing their assessment tasks. Atwell (2007) proposes that the development of a PLE has the potential to actually broaden and change the nature of assessment. The research study This reported study focused on university students' use of specific technologies within their Personal Learning Environments (PLEs) by attempting to offer new insights into how to help students integrate their informal use of technologies with their institution's technologies. Specifically, the focus of the research was to determine the technologies and devices used by students for their assessment tasks including studying for tests and examinations, as well as preparing projects and assignments for evaluation as components of their degrees. The technologies these students use define their PLEs FP:192 204 within the context of their assessment tasks. Because there is still some doubt about how much guidance students need to use these technologies for learning, specifically in university learning contexts for assessment purposes, this project aimed to extend our knowledge of students' PLEs which would allow a framework to be developed. The framework will guide the purposeful use of technologies that are typically used as part of their informal PLEs. This framework will synthesise the findings from this first stage of the study and is currently under development for publication at a future date. Research setting: Institutions, courses and students Participants were recruited from two Australian Higher Education Institutions: Edith Cowan University and Avondale College of Higher Education. Edith Cowan University (ECU) is a multi-campus institution located in Perth, Western Australia. ECU is a young university and is an institution that promotes multiple entry pathways. The students who responded to the survey and participated in the focus groups and mapping exercises from ECU were drawn from two metropolitan campuses with about 100 students on each campus. They were second year students comprising a mixture of mature age students and school leavers, and they were predominantly female. All were studying to be generalist primary school teachers. These students chose to undertake the unit MAE2240: Foundations of Primary Mathematics Education in a face-to-face, on campus mode of delivery rather than in online/distance mode. Avondale College of Higher Education is located in Cooranbong, New South Wales, between Sydney and Newcastle. The students who responded to the survey and participated in the focus groups from Avondale were comprised of students enrolled in either a Bachelor of Arts/Bachelor of Teaching or a Bachelor of Arts degree, majoring in areas such as Ancient History, Visual Arts or Communications. More than half of these students were female and most were in the second or third year of their degree. The majority of these students were studying to be secondary teachers with a smaller group involved in the visual arts and writing strands of a Communications bachelor-level degree. All of the Avondale students who completed the survey were studying as on-campus students rather than in online or distance mode. For further details about the number of students enrolled in each of the institutions and in each of the units, see Table 1 below. Table 1: Enrolment numbers in each institution and in each unit Institution Unit Specialisation/ Profile Number % ECU MAE2240 Second Year B.Ed and B/Teach 24 63 (Primary) Avondale CCCR15000 First Year BA COMMs students 4 11 Avondale HIST21000 Most Yr 1, 2 B.A. B/Teach & B.A. 8 21 Avondale ARTS34300 Third year Visual Arts students 2 5 TOTAL 38* 100 *One of 39 survey participants who did not indicate in which unit he or she was enrolled. The following information describes the students who completed the survey. • Of the 39 students who completed the survey, 24, or 62%, were from Edith Cowan University, while 15, or 38%, were from Avondale College of Higher Education. • Of the 39 students who responded to the survey, all were enrolled as on-campus students. • Most of the students were below 20 years of age or between 20 and 24 years of age. • The majority of the students who contributed to the surveys, 27, or 69%, were second-year students. • The majority of the students who contributed to the surveys, 31, or 78%, were female. The following information describes the students who participated in the focus groups and mapping exercises. • Of the 9 students who participated in the focus groups and mapping exercises, 5, or 56%, came from Edith Cowan University, while 4, or 44%, were from Avondale College of Higher Education. • Of the 9 students who participated in the focus groups and mapping exercises, all were enrolled as on-campus students. • Most of the students were below 20 years of age or between 20 and 24 years of age. • The majority of the students who participated in the focus groups and mapping exercises, 9, or 78%, were second-year students. FP:193 205 • The majority of the students, 6, or 67%, were female. Research methodology A mixed methods approach was adopted to determine how students used varied types of technologies, involving both an online survey and focus groups which incorporating a mapping exercise. This mixed methods approach was based upon the work of Clark et al. (2009), with their permission, who followed a similar procedure. The purpose of the questionnaire was to reveal the technologies most commonly used by students for assessment purposes that formed their PLEs. The data from the focus groups, including a mapping exercises, were intended to supplement the survey results and to determine specifically how students use various technologies for assessment purposes within their PLEs. As well as answering questions during the focus groups, the students completed a mapping activity in which they drew their PLEs. Data collection In the online survey, after being asked some demographic information, students were requested to identify the five most common types of technologies or online sites they used to prepare their college or university assessment tasks. Students were asked to list the technologies or online sites they used, as well as the other technologies or online sites which they did not use, but which they thought could be useful. In addition, they were asked about technologies or online sites that detracted or distracted them from their studies and from completing their assessment tasks. For the remainder of the survey, students were presented with names of websites, methods of communicating online, searching sites or search engines, online resources, online gaming sites, and digital devices, and were asked to rate how frequently they used them to prepare their assessment tasks. The surveys were administered to students from Avondale College of Higher Education and Edith Cowan University. During the focus group sessions, students were asked about how they used technologies for assessment purposes. Specifically, they were asked about the technologies and devices they personally used, the technologies and devices they saw being used by others, the mobile nature of technologies and devices, and they also predicted uses of technologies and devices. Students were also asked to draw a representation of their PLE. These drawings included labels and phrases to describe the technologies, drawings of technologies, annotations and visual representations of how the technologies relate to one another or are clustered (see Figure 1 later in the paper). Data analysis The survey data were analysed by calculating frequencies and descriptive statistics. An analysis was done of the demographic data to determine the participants' backgrounds. This analysis included calculating the number of participants, the number of students from each institution, the degrees students were enrolled in, the year of course/degree they were enrolled in, the unit/subject they were enrolled in, the enrolment mode, the number of students of each age, and the number of males and females. To determine the most common technologies or sites used for assessment tasks, the responses to the open-ended questions were classified into one of eight categories: 1) Library, journal databases and academic resources; 2) Devices (laptop, computer in library, smartphone, etc.); 3) Software (Word, PowerPoint, etc.); 4) Learning Management System (for example, Moodle, Blackboard); 5) Content-specific websites (curriculum, professional, etc.); 6) Reference resources (encyclopedias, dictionaries, thesauruses, etc.); 7) Social media and popular online sites (Facebook, YouTube, etc.); and 8) Apps. Frequencies were obtained for each category and the specific responses under each of these categories were grouped. Furthermore, frequencies were determined for each question, and conclusions were drawn regarding whether there were any other technologies or sites that the students did not use but thought could be useful when preparing their assessment tasks, as well as the technologies or sites that detracted or distracted them from working on their assessment tasks. The overall responses for these questions were then summarised. For the ratings of the specific resources, frequencies were tabulated and means were calculated under each category. The individual resources were then rank ordered within the categories to determine which were used most frequently. Transcripts were made of the focus group discussions. The transcripts were reviewed to determine trends in the current and future use of technologies and devices by the students, as well as their FP:194 206 perceptions of the use of technologies and devices by their peers. One of the foci of the discussion was mobile technology. The transcripts were analysed using NVivo, obtaining frequencies of technologies or devices mentioned. Categories of each of the technologies or devices were then determined which enabled the identification of themes and common phrases. The students’ drawings of their PLEs, constructed as a mapping exercise during the focus group discussions, were analysed to determine the technologies and devices used by students for assessment, as well as the connections between the technologies and their uses. Specifically, the analysis identified and summarised the spatial layout of nodes and the relations between them in order to identify and evaluate 1) the main technologies used; 2) connections between the technologies; 3) clusters or types of technologies; and 4) any technologies that appeared to be missing. The results of the data analysis of the survey data were compared with the results of the data analysis from the focus groups and mapping exercises to establish credibility and trustworthiness of findings. This triangulation of the data established links between the two sets of data and allowed for a clearer picture of how students are using technology to complete their assessment tasks. Findings In the first component of the survey students were asked to list the five most common types of technologies or online sites they used in conjunction with their assessment tasks such as completing assignments and preparing for tests or assignments. The students provided 53 different responses and these ranged from highly specific information sites such as ACARA: The Australian Curriculum, Assessment and Reporting Authority, a website which deals with curriculum and assessment issues in Australian education, to pop culture sites like YouTube. The entries were classified according to the type of resources. Table 2, following, shows the frequencies of the resources listed by the students, broken down by category. The most popular resources used in relation to assessment preparation were academic digital sources such as library and journal databases, though this was closely followed by the physical devices used by students to access the internet – encompassing everything from laptops to smartphones. Other categories regularly mentioned included online reference resources, software and social media sites. Mentioned only occasionally were Learning Management Systems, content specific websites and downloadable applications. Table 2: Technologies or online sites used to prepare assessment tasks Library, journal databases and academic resources 36 Devices (e.g., laptop, computer in library, smartphone) 23 Software (e.g., Word, PPT) 18 Learning Management System (e.g., Moodle, Blackboard) 9 Content-specific websites (e.g., curriculum) 8 Reference resources (e.g., encyclopedia, dictionary, 27 thesaurus) Social media and popular online sites (e.g., Facebook, 14 YouTube) Apps 2 TOTAL TECHNOLOGIES/ SITES MENTIONED 137 Each of these categories were then explored with more detailed questions and the responses were broken down into more specific categories, with similar responses being grouped together. Table 3 shows the list of responses and their frequencies, as provided by the students in each area. In the largest category of library, journal databases and academic resources there was a wide variety of sites mentioned, many of which were mentioned only once. Those used more often were Google Scholar, journal databases such as JSTOR, Primosearch and books available online (e-books). Interestingly only one student mentioned readings prescribed by the lecturer. Table 3: Library, journal databases and academic resources College Library (online) 1 Library Sources/Searches 1 e-Books/Books 5 LibraryOne 2 FP:195 207 Library Searches 2 Prescribed Readings from the 1 Lecturers Ecu Student Portal 1 PrimoSearch 5 Google Books 1 Proquest 1 Google Scholar 7 Referencing Guides 1 Journal Articles/Internet 6 University of England's online library 1 Journals/Journals/ Online Databases for journal articles/JSTOR Library databases 1 TOTAL ACADEMIC RESOURCES 3 6 The second most frequently reported resources used in the preparation of assessment tasks were categorised as Reference Resources and, probably unsurprisingly, Google was listed as the most frequently used resource. Others on the list had only minor numbers but included Endnote, the internet as a whole, online dictionaries and Citefast. When the category of Devices was broken down into detail it became clear that of the 23 responses, more than half (13) were using laptops for their assessments. Only four students claimed to be using tablets or convertible tablet/laptops and even less were using phones (3), desktop computers (2) or hardware calculators (1). In terms of Software mentioned in the survey, there were only 18 responses and 8 of these mentioned Microsoft Word as their software of choice. Other Microsoft Office programs such as Excel and PowerPoint were mention 6 times, while all other software had negligible mentions: Adobe PDF Reader (1), OneNote (1) and Pages (1). Of the 18 cited software products, 14 were Microsoft products. Social Media and popular online sites was the fifth most frequently reported resource category used by students when they prepare for, and write their assessments. Of the 14 students who mentioned these social media sites, 10 of them cited YouTube. Other sites mentioned were CiteMe, Facebook, One Drive and Sparknotes. It would seem that in general students are not using traditional social media sites as part of their assessment tasks and are using only a few popular online sites. Less frequently reported were Learning Management Systems, with only nine students mentioning these and only two mentioned by name – Moodle (4) and Blackboard (5). These numbers seem unusually low given that many students are expected to find assessment information and submit assessments via these sites. After the more general introductory questions, students were presented via the online survey with specific resources and asked to indicate how frequently they used them to prepare assessment tasks. These included websites, online communication, search programs, online media, online gaming, and digital devices. Table 4 shows the most commonly ranked responses mean responses. Facebook was the highest ranked website, followed by YouTube, Instagram, Pinterest and Dropbox. Students were given the option to list other websites. The unique responses to this question included Pandora, banking sites, iTunes and the App store, Quizlet, Tumblr, Behance, Kidstube and Kids Britannica Encyclopaedia. These last two were likely influenced by the fact that many of the students were preservice teachers. Students declared their most commonly used methods of online communication to be email, Messenger, online chat and discussion forums – in that order, though others mentioned included Facebook, texting, iMessage, Blackboard, TES (an online forum for educators) and Scoodle. Table 4: Frequency of use for specific resources # Question Never Rarely Occasionall Frequently Very Total Mea y frequentl response n y s 6 Facebook 3 1 4 4 18 30 4.10 18 YouTube 0 3 12 8 7 30 3.63 9 Instagram 16 2 1 2 10 31 2.61 13 Pinterest 10 6 5 7 3 31 2.58 4 Dropbox 12 10 4 3 2 31 2.13 When students were asked about the ways in which they search online for information to prepare for an assessment task, their responses (30 in total), revealed they use search engines far more FP:196 208 frequently than library databases. They were asked on a scale of ‘never, rarely, occasionally, frequently and very frequently’, how often they used these searching technologies and Google was by far the most frequently used with a mean score of 4.8, while library databases scored 3.67 and Wikipedia, 2.57. Other sites mentioned less than three times each included Google Scholar, Chrome, Safari, Bing, online libraries, Google Books and YouTube. Most students claimed that online gaming sites were not used when preparing assessments. The only item that rated highly enough to be worthy of mention was ‘Casual games such as Candy Crush, Farmville, Angry Birds, PVZ etc’. The students were not clear on how these helped them prepare for the assessments other than to help them relax during study periods, an issue also mentioned during the focus groups. When it came to the devices students used to prepare for an assessment task, the internet was most often mentioned. However, the devices being used to access the internet varied. Interestingly, the more portable devices (laptops and phones) were the most popular by far (4.84), with a mean a full point above the next most popular device – the desktop computer (3.27 mean). One student explained this in more detail: Regarding the use of desktop computers, most students I know only resort to these if a laptop is unavailable. Personally, I prefer to use my mobile phone and laptop for study/assessments. I rely heavily on my laptop to complete assignments and prefer online resources to hard cover ones as it is easier to use. During the focus group discussions, students were encouraged to explain and expand on the ideas they offered in the surveys. The first question that students were asked in the focus groups was simply, “What are the most common types of technology or devices that you use, or that you see other students using?” The overwhelming response to this was that students use their laptops with tablets and phones as a secondary source of information. One student mentioned the use of the interactive whiteboard and a couple of students from both groups mentioned taking photos in class of presentations or notes. When describing the advantages of having technology available in class, one student suggested that, “What's good with that is they can have up the slides at the same time and then you can double click to go to your notes at the same time so you can be looking at the modified lecture slides at the same time as taking notes on them.” Both student groups were in agreement that the most commonly used technologies in class were tablets, phones and laptops – for example, accessing the Blackboard app to see lecture notes, looking at the module requirements and collecting information for later study. At home – most students were using their laptops in the final preparation of their assessments. Some students were storing online books on their laptops and others were accessing the digital books through their University iLibrary. One student described a Facebook group they regularly used for their study called Perth WA Teachers, which signposts textbooks for sale, shares program and lesson plans and allows people to ask each other about educational issues. Others used online groups to co-ordinate assignments and one student stated that “almost for every single one of my group or partner assignments, we've made a page or a group chat for it”. Education students were using Facebook pages to communicate whilst on teaching practicums to keep up-to-date with how their other classmates were faring in the classroom. Students also discussed using GoogleDrive to pass documents back and forth that they were editing and working on as a group, particularly for larger files that might not fit applications like Facebook. The file-sharing sites, GoogleDocs and Dropbox, were also mentioned by both focus groups. When it came to their word processing software most students used Microsoft Word, but they mentioned the fact that a lot of students used free software instead – Open Office, Publisher and Pages were mentioned. OneNote was also described as “really wonderful software,” though the students laughingly admitted they still usually chose to use Word. Some students suggested they did not like to experiment with new software when they were busy with assessments – they stuck with things they knew and understood. Students were asked why they were using certain technologies over and above others and ease-of-use was the defining factor of choice. Words and phrases such as “familiarity”, “short-learning curve” and “convenience” were used and students claimed to be more likely to try new technologies if they were recommended and explained by their peers rather than lecturers, tutors or librarians. Another point discussed during the focus groups was the actual differences that using technologies FP:197 209 and the internet made to the ways students completed their assessment tasks. Most of them had never been without these technologies so the discussion was not comparative to a time when an assessment task was completed without using such technologies. Students found that laptops and online software and resources meant that they were portable and could work anywhere, but were limited by the availability of the internet (and in particular free access to the internet). A student described this flexibility as ‘multi-tasking’: “You can be at home doing the washing and reading a book the same time online.” Another described his dependence on the internet in absolute terms: “I never do any assessments unless I've got the internet. When we had the floods, our internet access was cut for a week and I moved to my grandma's because she had internet access so I could do all my assignments. I just can't do them … I think because all my sources tend to be online.” The biggest problems students faced with technologies were associated with the availability of power sources and free internet access. On the rural campus at Avondale, students also struggled with phone reception which they reported using to co-ordinate meetings with other students and, if Wi-Fi was not available, to tether their computers and phones together. A student from ECU mentioned that ergonomically students are always under physical pressure from carrying tablets and laptops. However, despite any obstacle to their use, students were united in their belief that possessing or having access to a tablet or laptop was essential to being a modern student – going so far as to suggest that they should be provided by the University and paid back via HECS debt. Overall, the students who participated in this study across two institutions demonstrated a strong preference for technologies and devices that were portable. Their concerns with the use of technologies for assessment purposes were largely focused on internet connectivity, phone coverage and the availability of Wi-Fi. Discussion and recommendations The students who participated in this study reported Google Scholar as one of the most used technologies for completing assessment tasks. However, they appeared to use very few technologies that were recommended by their lecturers (such as library databases or the institution's LMS); a similar finding was noted by Gosper et al. (2014) who reported an interest in "the number of popular technologies that students use at their own volition" (p. 299). In the study reported in this paper, the most commonly used physical devices, perhaps predictably, were portable devices such as laptops, tablets and smartphones. Students tended to view these physical technologies as central to their PLEs which was evident in their PLE drawings, as shown in Figure 1. Less emphasis was placed on the use of social media than other studies have reported (Mbati, 2013; Wang et al., 2015) but students did appreciate technologies that allowed them to share resources, ideas and support during assessment preparation processes. They especially appreciated the informality and interactivity offered by Facebook but did not show any preference or consistent demand for traditional desktop technologies such as printers or desktop computers, a trend also evident in the 2015 NMC Technology Outlook for Australian Tertiary Education: A Horizon Project Regional Report (The New Media Consortium, 2015). FP:198 210 Figure 1: Sample of technology mapping exercise Based on the findings of this project, students' use of technologies when preparing assessment tasks could be considered far less formal, as noted by Attwell (2007b), than the prescribed use of the institution's LMS or library resources. The variety of technologies used by the students in this study was not wide, a finding which echoes the work of Margaryan, Littlejohn and Vojt (2011): "students use a limited range of mainly established technologies" (p. 429). Even so, this finding conflicts somewhat with the outcomes of Gosper et al.'s research (2014) which found "wider access to freely available open resources and new technologies such as Smartphones and iPads" (p. 290). Because the completion of assessment tasks may be viewed as a high stakes activity by students and lecturers alike, the narrower than expected range and the less than adventurous use of technologies evident in this project may have been due to students' concerns about straying too far from the assessment task specifications. Furthermore, the typical approach of completing assessment tasks just before their due date may have also been a reason that students tended to choose less innovative technologies that required a "higher learning curve" when completing assessment tasks. Despite the narrow range noted in some aspects of the students' technology use, the locations in which the technologies were used by the students in this project incorporated a range of both formal and informal contexts which may have been attributed to the increased use of mobile technologies, also a factor in the changing use of technology reported by other researchers (Gosper et al., 2014; The New Media Consortium, 2015). Even so, the affordances of mobile technologies were reported by the students in this project mainly in terms of their flexibility and portability rather than their capacity to enable social networking with others, which has been reported elsewhere (Cochrane & Withell, 2013). Perhaps the flexibility and convenience of technologies were emphasised above and beyond their social capacities because some aspects of assessment tasks typically require students to be less social, requiring more independent activity, than the generalised use of technologies for study and learning. The increased trend for flexible and mobile use of technologies for learning purposes aligns closely with the "bring your own device" approach and the increasing role of mobile apps, recently reported in reports such as the 2015 NMC Technology Outlook for Australian Tertiary Education (The New Media Consortium, 2015, p. 4). Learning management systems (LMSs) provide faculty members and students access to a wide range of learning applications and services (Conde, García, Rodríguez-Conde, Alier, & García- Holgado, 2014). In their first phase of a qualitative research study, Hustad and Arntzen (2013) reported that participants appreciated the benefit of having all the information in one place, which allowed students to access information anytime and anyplace, while allowing faculty to communicate with students very easily. Hustad and Arntzen (2013) also reported challenges which faculty and students had with the LMS. Participants expressed concerns with organisation and structure, as well as ease-of-use and ease of sharing knowledge. Further, they expressed concerns about the limited time that the information on the LMS was available. The limited available of the information is not conducive for life-long learning. Participants also talked about the challenge of sharing information FP:199 211 from one course to another. In LMSs, each class is typically independent from one another; as Hustad and Arntzen (2013) expressed it, each is its “information silo". This further inhibits the development of personal learning, where the insights from different classes may not be easily integrated to create personal learning environments (PLEs). The central role of the LMS in an institutional context may be at odds with students' views about the LMS, as indicated to an extent by Taraghi, Ebner, Till and Mühlburger's (2009) work: "Nowadays a shift from an institution-centred approach to a learner-centred one becomes necessary to allow individuality through the learning process and to think about learning strategies in general" (p. 1:10). This finding also aligns with the increasing role of adaptive learning technologies that "refer to software and online platforms that adjust to individual students’ needs as they learn" (The New Media Consortium, 2015, p. 17). Personal learning technologies allow for “instruction to be personalized to users’ actions and interests, to provide assistance when needed and present instruction that is understandable, engaging, and situated in relevant and meaningful contexts (Walkington, 2013, p. 932). Because of perceived inflexibility of some LMSs, there is very little room for personalization. If, on the other hand, more control were to be given to students to integrate their own personal learning systems into the LMS, this may would result in a more personalised learning management system. As a solution to the limitations of the learning management system, Stantchev, Colomo-Palacios, Soto-Acosta, and Misra (2014) advocate the integration of cloud-based applications into the LMS. While this may solve some problems, it does not address the limitations of access to information. Conde et al. (2014) therefore, advocate that the LMS be made open to allow for the seamless integration of information from the LMS to a student’s PLE. In a study of such an arrangement, Conde et al. found that this seamless integration personalized the learning environment and positively contributed to students' learning. Hustad and Arntzen (2013) reported a limited use of some of the more interactive technologies, such as discussion board. Although the findings from our study did not indicate a strong use of the LMS in either institution included in the study, Gosper et al. (2014) recent report, Student use of technologies for learning: What has changed since 2010?, reported an increased use in some LMS functions. These variations in the findings across studies about the popularity or oversight of the LMS may simply be accountable to the varied ways in which the LMS is used at each institution. A number of recommendations emerged from this study and they are presented here for consideration by other higher education institutions with students similar to those described throughout this paper. This research indicates that students tend to be more independent, device- wise, compared to previous eras which may have seen students depend on class sets of laptops or tablets. Such resource sets no longer appear necessary. Also, when institutions maintain tight control over institutional devices, this may prevent innovative use of technology by lecturers and students, particularly in the preparation for, and completion of, assessments. Instead, it may be more worthwhile for the institution to contribute infrastructure towards Wi-Fi technology which extends affordances such as device portability, mobility and flexibility. Just as Dabbagh and Kitsantas (2012) acknowledged the potential role of PLEs to support the development of students' self-regulated learning practices, the recognition and promotion of students' use of varied technologies in association with assessment tasks may facilitate student independence. However, there is some tension between the extent to which students are willing to innovate using technology and the extent they are willing to take risks in the assessment arena, although they did show some tendency towards initiating, contributing to and accessing technologies which facilitated sharing of ideas and resources. Modelling the use of innovative technology by lecturers may also serve to encourage students to extend their use of technologies. Furthermore, incorporating students' use of technologies is an important curriculum design consideration (Könings et al., 2005) but the current use of the LMS may require some modification to meet contemporary students' expectations in terms of the its capacity to offer responsive and personalised learning experiences. While the findings of this study suggest that students, on the whole, did not perceive the LMS being used in a way that was clearly relevant to their learning or their assessment needs, there were opportunities to use the LMS as a launching pad from which to link to other available technologies such as relevant search engines, collaborative social media software and innovative apps. FP:200 212 Conclusion In contrast with other studies on PLEs, this study focused on technologies used by undergraduate students for assessment-related tasks. Two cohorts from different institutions were surveyed and participated in focus groups during which they also drew representations of their PLEs. When accessing academic resources, these students used a variety of websites, especially Google Scholar, journal databases and e-books, but the LMS used at each institution did not dominate their thinking. The most commonly used physical devices were portable, including laptops, tablets and smartphones, which students tended to view as central to their PLEs. Students placed high importance on being connected to the internet, especially via Wi-Fi technology, and having phone coverage. However, their use of social media in association with assessment use, although valued as a sharing mechanism, was not as widespread as has been reported in other studies about the use of technology in general by higher education students. Definite preferences were shown for software and tools which were easy to use, convenient to access and quick to learn, especially when recommended by their peers. Although the students' use of technology was considered narrower than expected, they did not feel restricted by their institution's formal technological networks, suggesting their PLEs were broader that the collection of technologies offered by an LMS. More research is required to investigate the contexts in which these main technologies are used by students in association with assessment and the connections between these technologies. Methods used by university students to collate technologies within a single, unifying technology cluster may also be investigated and discovered. From this study, there is some indication that social technologies may be used less during assessment tasks than for general learning purposes. Investigation into how technologies are used by postgraduate students for assessment tasks is also warranted. These areas of research are planned for the following stages of the study. References Attwell, G. (2007a). E-portfolio: The DNA of the Personal Learning Environment? Journal of e- Learning and Knowledge Society, 3(2). Retrieved from http://www.pontydysgu.org/wp- content/uploads/2008/02/eportolioDNAofPLEjournal.pdf Attwell, G. (2007b). The personal learning environments: The future of eLearning? eLearning Papers, 2(1), 1-8. Castaneda, L., & Soto, J. (2010). Building personal learning environments by using and mixing ICT tools in a professional way. Digital Education Review, 18, 9-25. Clark, W., Logan, K., Luckin, R., Mee, A., & Oliver, M. (2009). Beyond Web 2.0: Mapping the technology landscapes of young learners. Journal of Computer Assisted Learning, 25(1), 56-69. Cochrane, T., & Withell, A. (2013). Do 21st Century students dream of electric sheep? A mobile social media framework for creative pedagogies. In H. Carter, M. Gosper & J. Hedberg (Eds.), Electric Dreams. Proceedings ascilite 2013 Sydney (pp. 151-161). Macquarie University: Australasian Society for Computers in Learning in Tertiary Education (ASCILITE). Conde, M. A., García, F., Rodríguez-Conde, M. J., Alier, M., & García-Holgado, A. (2014). Perceived openness of Learning Management Systems by students and teachers in education and technology courses. Computers in Human Behavior, 31(517-526). Conradie, P. W. (2014). Supporting self-directed learning by connectivism and Personal Learning Environments. International Journal of Information and Education Technology, 4(3), 254-259. Cross, J. (2006). The low-hanging fruit Is tasty. Internet Time Blog. Retrieved from http://www.internettime.com/2006/04/informal-learning-clo-april-06/ Dabbagh, N., & Kitsantas, A. (2012). Personal Learning Environments, social media, and self- regulated learning: A natural formula for connecting formal and informal learning. The Internet and Higher Education, 15(1), 3-8. Drexler, W. (2010). The networked student model for construction of personal learning environments: Balancing teacher control and student autonomy. Australasian Journal of Educational Technology, 36(3), 369-385. Fiedler, S., & Väljataga, T. (2010). Interventions for second-order change in higher education: Challenges and barriers. Electronic Journal of e-Learning, 8(2), 85 - 92. Goldstein, I. P., & Miller, M. L. (1976). AI Based Personal Learning Environments: Directions for long term research. Retrieved 06 May, 2015, from http://files.eric.ed.gov/fulltext/ED207580.pdf FP:201 213 Gosper, M., Malfroy, J., & McKenzie, J. (2013). Students' experiences and expectations of technologies: An Australian study designed to inform planning and development decisions. Australasian Journal of Educational Technology, 29(2), 268-282. Gosper, M., McKenzie, J., Pizzica, J., Malfroy, J., & Ashford-Rowe, K. (2014). Student use of technologies for learning: What has changed since 2010? In B. Hegarty, J. McDonald & S. K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014 (pp. 290-301). Dunedin, New Zealand: Australasian Society for Computers in Learning in Tertiary Education. Harvey, L. (2015). Analytic Quality Glossary, Quality Research International. Retrieved 14 October, 2015, from http://www.qualityresearchinternational.com/glossary/ Hight, C., Khoo, E., Cowie, B., & Torrens, R. (2014). Software literacies in the tertiary environment. In B. Hegarty, J. McDonald & S. K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ASCILITE Dunedin 2014 (pp. 410-415). Dunedin, New Zealand: Australasian Society for Computers in Learning in Tertiary Education (ASCILITE). Hustad, E., & Arntzen, A. A. (2013). Facilitating teaching and learning capabilities in social learning management systems: Challenges, issues, and implications for design. Journal of Integrated Design & Process Science, 17(1), 17-35. James, R., McInnis, C., & Devlin, M. (2002). Assessing learning in Australian universities : Ideas, strategies and resources for quality in student assessment. Melbourne, Vic: Centre for the Study of Higher Education for the Australian Universities Teaching Committee. Jenkins, M., Walker, R., & Voce, J. (2014). Achieving flexibility? The rhetoric and reality of the role of learning technologies in UK higher education. In B. Hegarty, J. McDonald & S. K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ASCILITE Dunedin 2014 (pp. 545-548). Dunedin, New Zealand: Australasian Society for Computers in Learning in Tertiary Education (ASCILITE). Johnson, M. W., & Sherlock, D. (2014). Beyond the Personal Learning Environment: Attachment and control in the classroom of the future. Interactive Learning Environments. Special Issue: Learning management system: evolving from silos to structures, 22(2), 146-164. Könings, K. D., Brand‐Gruwel, S., & van Merriënboer, J. J. J. G. (2005). Towards more powerful learning environments through combining the perspectives of designers, teachers, and students. British Journal of Educational Psychology, 75(4), 654-660. Margaryan, A., Littlejohn, A., & Vojt, G. (2011). Are digital natives a myth or reality? University students’ use of digital technologies. Computers & Education, 56(2), 429-440. Mbati, L. (2013). Online social media applications for constructivism and observational learning. International Review of Research in Open and Distance Learning, 14(5). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1579/2709 Smith, M. (1988). Developing youth work. Informal education, mutual aid and popular practice. Milton Keynes: Open University Press. Stantchev, V., Colomo-Palacios, R., Soto-Acosta, P., & Misra, S. (2014). Learning management systems and cloud file hosting services: A study on students’ acceptance. Computers in Human Behavior, 31, 612-619. Taraghi, B., Ebner, M., Till, G., & Mühlburger , H. (2009). Personal Learning Environment: A conceptual study. Paper presented at the ICL2009 The New Media Consortium. (2015). 2015 NMC Technology Outlook for Australian Tertiary Education: A Horizon Project Regional Report. Austin, TX: The New Media Consortium. Valjataga, T., & Laanpere, M. (2010). Learner control and personal learning environment: A challenge for instructional design. Interactive Learning Environments, 18(3), 277-291. van Harmelen, M. (2008). Design trajectories: Four experiments in PLE implementation. Interactive Learning Environments, 16(1), 35-46. Vygotsky, L. (1933/1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press. Walkington, C. A. (2013). Using adaptive learning technologies to personalize instruction to student interests: The impact of relevant contexts on performance and learning outcomes. Journal of Educational Psychology, 105(4), 932-945. Wang, Y., Niiya, M., Mark, G., Reich, S., & Warschauer, M. (2015). Coming of age (digitally): An ecological view of social media use among college students Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 571-582). Vancouver, BC, Canada. Wild, F., Mdritscher, F., & Sigurdarson, S. (2008). Designing for change: Mash-up personal learning environments. eLearning Papers, 9. Retrieved from http://oro.open.ac.uk/25253/1/media15972.pdf FP:202 214 Yorke, M. (2003). Formative assessment in higher education: Moves towards theory and the enhancement of pedogoic practice. Higher Education, 45(4), 477-501. Lounsbury, L., Mildenhall, P., Bolton, D., Northcote, M., & Anderson, A. (2015). Higher education students' use of technologies for assessment within Personal Learning Environments (PLEs). In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. 190-203). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:203 215 Strong and increasing student demand for lecture capture in the changing Australian university classroom: results of a national and institutional survey Carol A. Miles School of Education The University of Newcastle As the use of classroom lecture capture gains wide acceptance and application around the world, this technology is quickly moving into the mainstream for university teaching. The paper reports preliminary findings of a student survey conducted by Echo360 across seven Australian universities to gain student feedback and perspective on the use of lecture capture technology, focusing on the use of the technology and student results at the University of Newcastle, Australia. Specific focus is applied to the use of lecture capture to enhance the flipped and blended styles of teaching and learning that are currently being implemented. Keywords: lecture capture, Echo360, flipped classroom, classroom capture, blended learning Introduction The global practice of capturing lectures for student personal viewing and study has been gaining acceptance for almost two decades (Danielson, Preast, Bender, & Hassall, 2014). For the purposes of this paper, lecture capture refers to the recording of all content displayed on a classroom computer and the voice of the lecturer while presenting to the class (but generally no video of the lecturer). A number of other tools designed to enhance and extend the lecture experience for students (both inside and outside the classroom) are currently in use at the University of Newcastle and elsewhere, through Echo360’s student engagement software and other technologies, but these are not formally discussed here. At the University of Newcastle, Australia (UON), lecture capture technology has been employed for recording lectures for 10 years, initially using the University of Western Australia’s Lectopia product in a few large theatres, and moving to Echo360’s EchoSystem in 2011. A major capital project in 2013 saw all 133 classrooms and lecture theatres of 35 seats and over equipped with lecture capture technology. Recorded lectures are made available through the UONline Virtual Learning Environment (via Blackboard Learn). In 2013, over 10,000 lectures were captured using this automated system, and approximately 14,000 lectures were captured in 2014. In addition to this, currently has 156 active personal capture users (using their own computers to capture content for their students), and captures using this tool are predominantly supporting the flipped classroom learning and teaching model. Availability of lecture capture technology is expanding to smaller rooms in most universities, as is the case at the University of Newcastle, and this would indicate the potential to offer far more lecture recordings in the future. The ultimate aim of this strategy is capturing and making available to students all structured lectures offered in equipped rooms. A considerable amount of research has been conducted worldwide regarding student use of lecture capture technologies, with these technologies having been in common use in higher education institutions globally for many years. Smith and Volker (2013) detail the growth of the technology, and the primary motivations for academic institutions to make recorded lectures available – as well as students’ interest in accessing these lectures to improve their educational experience. A preponderance of research indicates that merely viewing lectures captured in this fashion is not an adequate substitute for a well-planned and executed classroom experience (Williams & Hancock, 2012), but that when provided as part of a wider suite of in- and out-of-classroom experiences and technologies, lecture capture can enhance study strategies of modern, tech-savvy students (Brooks, FP:204 216 Erickson, Greer, & Gutwin, 2011), expand modes of learning to online and mobile platforms and offer flexible learning opportunities (Larkin, 2010). UON’s philosophy regarding captured lectures is that they are primarily for the purpose of enhancing student learning and enabling review of content. The intention has not been to replace attendance at lectures with passive viewing of lecture recordings or otherwise offer these recordings as the primary source of learning for online students. Whilst students continue to come to campus, attend lectures and participate in classroom activities, the demand from students for more content to be captured and put online is clear and growing. Prior to making the decision to commit to a pervasive approach to lecture capture, UON sought to assure that this mode of learning enhancement was desired and that student demand for the service would continue to grow. Therefore, in 2013, the University joined six other Australian universities in collaboration with Echo360 to survey students regarding their use of lecture capture. UON participated in the survey that took place over a two-week period in May 2014 and included seven 1 Echo360 enabled Australian institutions . The primary goal of the survey was to assist institutions in assessing their students’ use of Echo360. The secondary goal was to provide the Echo360 community valuable insight into regional practices, standards and expectations to the benefit of all concerned. 4,206 responses were received from students in the seven institutions. The survey, containing a mix of Likert-type and open-ended questions, was reviewed by UON’s Strategy, Planning and Performance Unit prior to being made available to all UON students within the UONline Virtual Learning Environment. During the 15 day survey period at UON, 1,162 lectures were captured (1,368 hours of content recorded), with 17,353 unique student users accessing the system (approximately half of all UON students) completing 33,364 total views. A total of 458 UON students completed the survey. This is acknowledged as a very low response rate (2.6% of students who accessed the system during the survey period), but findings were directly in line with the overall student responses nationally, so seem to indicate that findings of the national study can be generalised to the UON student population. Analysis of the responses from the UON cohort of students indicated that they were statistically similar (and in some cases identical) to the total respondent pool of 4,206 from across the seven universities. The findings from these 4,206 responses are described below, noting differences in the findings for UON students where they exist. Findings of the Echo360 Student Survey – Student feedback regarding lecture capture technology Analysis indicated that first-year students (See Figure 1 below) viewed captured lectures most frequently (35% of respondents) followed by second-year students (26%). These indicated more active viewing behaviours than third and fourth year students (19% and 8% respectively). However, for many universities, lectures are only captured in large lecture theatres, and these large lectures tend to be first and second year courses. Also, for UON and some other institutions involved in the survey, we have only recently expanded the service offering to take in more classrooms and capture more content; and as such, first and second year students are more accustomed to having this technology available than those in the later stages of their studies. It can be predicted that as the current first- and second-year students progress through their programs, they will expect the same learning supports (i.e. lecture capture) as have been available early in their programs. Male students reported accessing lecture recordings with much greater frequency (73%) than female students (37%). There was no indication from the current study to explain this behaviour, but this may relate to general engagement with technology as well as the potential for impact of the types of programs in which male students are enrolled. Gender differences in student attendance have been reported previously by Arulampalam, Naylor, and Smith (2012) who reported that significantly more male students are absent from classes generally than are female students. This may indicate the possibility that males more than females take advantage of lecture capture technology to make up for missed classes. 1 University of Newcastle, Murdoch University, University of Canberra, University of South Australia, University of Tasmania, Victoria University, Monash University. FP:205 217 Figure 1. Demographic results from student responses in Echo360 national survey No significant difference in viewing habits was reported between full-time and part-time students, which is somewhat surprising given the greater number of activities in which part-time students engage, and which compete for their time and attention. The national survey indicated the following breakdown of hours that students worked while studying: Table 1. Number of weekly hours of work outside of study from Echo360 national survey If you are employed, how many hours per week do you work? Response Response Answer Options Percent Count Not employed 34.80% 1446 1-5 hours 8.90% 368 6-10 hours 13.40% 556 11-20 hours 20.20% 838 21-35 hours 9.90% 411 Full time 12.80% 533 answered question 4152 skipped question 16 This data (see Figure 2) indicates that of all students responding, approximately 22% worked more than 21 hours per week, with almost 13% working full-time. It is an interesting finding that almost 35% of students responding to the survey did not work outside of university at all, but were still taking advantage of the use of captured lectures for study. Results of the UON survey showed significantly more international students (13%) than was reported in the national study overall (4%). This suggests that more UON international students are taking advantage of the technology, potentially to address English language deficiencies. This is most likely a reflection of the general UON demographic, although the percentage of UON international students responding to the survey was just slightly higher than the overall international student population at UON in 2013 (11.5%). Recorded lectures hold clear advantages for students who struggle with understanding spoken English, as they are able to review the lecture, along with slides and other resources, as often as is necessary in order to master content. There is evidence that disabled students may take more advantage of available captured lectures. In the Australian survey, over 6% of students self-identified as having a disability, which is just slightly FP:206 218 higher than the Australian university national average of 5.2% of students self-identifying as disabled 2. Karnad (2013) reported that lecture capture technology can positively assist disabled students in their study and academic success and further focus should be put on the specific ways that this can be accomplished. As additional focus is placed on the support of students with a variety of disabilities, lecture capture technology may have a particular functionality, especially if augmented with closed caption text. The majority of students (66%) indicated that they wished to have classroom-based recording offered for all of their classes. Also, 75% percent indicated that the ability to study both in a face-to-face mode in conjunction with an online resource helps them better understand the concepts. Students indicated qualitatively that the two most common reasons for viewing recordings were that they used the classroom-based recordings to revisit and clarify complicated and confusing topics, and that access to these recordings helped to balance their schedules which included university, family and work responsibilities. A slightly lower percentage (69%) of UON students, when compared with the total national sample (75%) reported that they used recording to balance their schedules. This is a somewhat surprising result considering that UON has a larger than average population of students from low SES backgrounds and with other demographics (such as being first in family to attend university, of mature age, and single parents) that may see them as less prepared for university study on entry. These students could be considered to have more complex and demanding schedules than the ‘average’ Australian university students, and should therefore be in a position to take better advantage of the flexibility of study offered by lecture capture technology. It is possible that this lower percentage is a result of the lack of lecture capture offerings in certain programs, and this may be mediated once all lectures are captured and made available. Figure 3 shows the full responses from students regarding their preference for use of classroom-based lecture recordings: Table 2. Reasons students use classroom-based recordings from Echo360 national survey Why do you use classroom-based recordings? Tick all boxes that apply. Response Response Answer Options Percent Count Helps me stay in a class I would otherwise have dropped 21.7% 684 Helps me balance my schedule/responsibilities 71.2% 2249 Reviewing classroom-based recordings as a substitute for attending class 48.5% 1530 I rely on classroom based recordings because English is not my first language 6.8% 216 Help me better understand instructors with strong accents 29.1% 918 Help me use my time more efficiently 57.1% 1802 To learn at my own pace 62.6% 1976 To revisit and clarify complicated or confusing topics 74.5% 2351 To help prepare for exams 63.9% 2018 I believe classroom-based recordings help me to achieve higher grades 48.9% 1545 Classroom-based recordings improve my overall learning experience 11.3% 358 answered question 3157 skipped question 1011 Student qualitative comments indicated that the four most frequently noted aspects of lecture capture that students liked best were: ‘flexible learning’, the ability to set the pace of their own learning; ‘convenience’, allowing them to view content in line with their own schedule irrespective of their location; ‘reviewing content’, the ability to review difficult concepts multiple times and pause and rewind as necessary; and ‘negotiating commitments’, the ability to manage distractions relating to work, family and medical issues and view lectures as suited their life circumstances. This is very 2 Source: Australian Government Selected Higher Education Statistics – 2013 Student Data: Table 2.6: All Domestic Undergraduate Students by State, Institution and Equity Group, 2013. FP:207 219 much in concert with the quantitative results reported from the survey. Linked to these student perceived benefits of lecture capture were comments relating to the two most favoured features within the system: the ‘variable speed playback’ feature, which allows students to speed up and slow down passages of the recording; and the ‘scenes’ feature, which provides students with visual thumbnails from the whole lecture and allows them to go directly to the area of the lecture that they wish to review. Several negative comments from students were also noted from the qualitative comments. Students were not always satisfied with the audio capture, and were frustrated that they were unable to hear student comments and questions that comprised part of the lecture. Students also indicated dissatisfaction that they were unable to see the lecturers’ expressions and actions (unable to see writing on a white-board or pointing to things off screen). This was particularly noted when lecturers used any form of non-digital presentation tool. Incomplete capture was a source of frustration for students, arising from circumstances where the beginning and/or end of the lecture were cut off on the recording. Some students indicated that distractions when viewing lectures would require them to rewind and re-view several times, making it more time-efficient to attend the actual lecture. Comments were also made regarding the poor audio or video quality of some lectures, indicating that students are becoming discerning consumers of the technology. Several students also commented that a disadvantage of viewing lectures at home is the inability to ask questions as you think of them. All of these disadvantages arise from students using lecture capture as a sole means of ‘attending’ a lecture. Attendance at the face-to-face class would mitigate all of these issues, and students utilising the technology strictly as a review tool, or one used sparingly for those circumstances when classroom attendance is truly not possible, would not experience these problems to such a great extent. At the University of Newcastle, captured lectures are intended strictly as an addition study/review tool, and are not presented as an alternative means of study, or a legitimate/sole means of online learning. It should be noted that many of these concerns voiced by students in relation to their captured lectures are now being actively addressed through the implementation of Echo360’s student engagement functionality. When the next survey is conducted within the next two years, it will be interesting to note changes in student feedback as this functionality allowing much more synchronous student engagement is brought into universities’ virtual learning environments. Generally, the national survey, whose results were reflected by the participating UON students, provides a clear indication that there is general acceptance and appreciation of the technology by the majority of students. There is reason to believe, therefore, that student demand for this resource will remain on an upward trajectory. Echo360 in the Flipped Classroom As many universities are moving into more engaged modes of learning, incorporating blended and fully-online environments, the survey yielded results relating to student opinion on the use of this technology for modern learning environments. • 48% of respondents reported that their instructor published digital content via Echo360, in addition to (or instead of) lecture captures. • Of those respondents, 59% said that their instructor encouraged them to watch this digital content prior to attempting an on-campus or online synchronous teaching activity. • The open-text responses regarding the availability of digital content are predominantly positive, recognising value in the in-depth content focus of this material and the opportunities it provides for more interactivity and discussion in synchronous (face-to-face or online) activities. • The negative comments made in the survey about digital content relate to concerns about synchronous (face-to-face or online) activities being replaced by this content; also, a number of respondents felt that the lack of personal interaction with the digital content (particularly, the inability to ask questions) made it a difficult medium for widespread use. The above indicates that lecture capture technology is taking a significant place in new blended learning environments, but there are still some significant impediments to engagement for students. It can be interpreted that while captured lectures can extend and enhance learning opportunities, FP:208 220 educators must be careful to offer enough enriched activities to ensure students have a variety of options when expected to construct their own learning. Lecture capture technology clearly offers numerous options for assisting students with content mastery. As we move to more complex learning outcomes, such as analysis and synthesis, other opportunities for personal engagement (whether face-to-face or online) must still be considered to provide students with a balanced range of learning opportunities that address a complex variety of learning outcomes and student learning styles. Limitations When considering the above findings, it must be acknowledged that all results are by means of self- report. Gorissen, van Bruggen, and Jochems (2013) discussed the limitations of student self-report regarding the use of lecture capture, and the potential that student motivation for using lecture capture, as well as behaviours that are self-reported are not definitive without triangulating data such as actual usage reports and correlation of that data with grades and other performance indices. As we decide that students are using captured lectures for the right reasons (i.e. not just to enable non- attendance at classes), we must be careful to assure that this is the case. These authors suggest that additional data collection is necessary prior to acceptance of students’ reported activities and motivations. As this learning technology becomes more pervasive, and with the current introduction of complex learning analytics (Bichsel, 2012) to enable much richer analysis of student study behaviours, more definitive determinations will be possible relating to student motivation for accessing recorded lectures. University teachers’ acceptance of lecture capture as a teaching tool University teachers’ acknowledgement and acceptance of lecture capture benefits for students has been mixed and is not as consistent as the ringing endorsement reported by students. It is true that many university teachers have embraced the technology for a decade now, and have voluntarily and even enthusiastically provided captured recordings of their lectures. Germany (2012) reported that many of these academics are actively seeking technological enhancements that will make the lecture viewing experience more interactive for students and see the technology as an integral part of their teaching. As with other learning technologies, it is important for these early adopters to lead the way when entrenching any teaching method into a university’s learning culture. Other university teachers, however, are not so anxious to adopt the technology. Many teaching academics react with suspicion, caution, and even consternation when confronted with the concept that by policy, all of their lectures will be recorded and made available to their students (Larkin, 2010). A primary objection raised by many university teachers is the impact of offering captured lectures on student attendance. Von Konsky, et. al (2009) refuted this concern, reporting similar attendance patterns for students whether or not captured lectures were available for them to view. The current study results indicate that less than half of the respondents (48%) reported that viewing captured lectures was a substitute for attending class. An argument can be made by those who are skeptical, however, that this is almost half of their students admitting to reducing class attendance if the technology is available. Holbrook and Dupont (2009) reported that available captured lectures are more likely to cause early-year students to miss class than those in upper years. This may be related to a maturity of study and learning strategies, but also reflects the more personal nature of smaller upper-year classes, adding more value to face-to-face attendance. This certainly places a greater onus on the university teacher to present a lecture experience that is compelling enough to make students see the benefits of attending. Conclusion Recording lectures and making them available to students online has become standard practice in most Australian universities as well as those around the world. For some, the decision regarding whether or not to capture individual lectures is left up to the university teacher responsible for the course (or School, Faculty, etc.). Others have decided to entrench the facility in an ‘all-in’ or ‘opt-out’ fashion, as has been the decision of the University of Newcastle and several other Australian universities including the University of Melbourne, La Trobe University, and the University of Western Sydney. A primary challenge with this model will be to convince some of those teaching academics who are still suspicious of the impact of the technology to embrace and leverage the potential of lecture capture as a critical learning tool for their students. This will require a well-defined and FP:209 221 resourced communication and professional development strategy on the part of teaching and learning centres. Considering the rapid rate of conversion of teaching methodologies to blended or ‘flipped’ modes of learning, we may look back on lecture capture as a transitional technology, ‘filling in’ for those classes that have not yet been redesigned to include the additional in-class engagement and online content mastery that is becoming increasingly accepted and expected as the preferred method of teaching and learning. We have a long way to go, however, before all (or even most) traditional lectures have been ‘flipped’ in this fashion. For now, increasing student demand and expectation for the availability of captured lectures that can be viewed independently will assure a sound future for traditional lecture capture, and a bright future for the increased interactivity within the platform that is currently being introduced. References Arulampalam, W., Naylor, R., & Smith, J., 2012. Am I missing something? The effects of absence from class on school performance. Economics of Education Review, 31 (4), August 2012, 363- 375. Bichsel, J. (2012) Analytics in Higher Education: Benefits, Barriers, Progress and Recommendations (Research Report). Louisville, CO: EDUCAUSE Centre for Applied Research. Available online at: http://www.educause.edu/ecar Brooks, C., Erickson, G., Greer, J., & Gutwin C. (2011). Is the effectiveness of lecture capture related to teaching approach or content type? Computers and Education, 72, 121-131. doi: 10.1016/j.compedu.2013.10.016 Cameron, D. & Miles, C. (2013). Lecture Capture in the Flipped Classroom: Addressing the Oxymoron. In T. Bastiaens & G. Marks (Eds.), Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2013 (pp. 55-59). Chesapeake, VA: AACE. Danielson, J., Preast, V., Bender, H., & Hassall L. (2014). Modelling and quantifying the behaviours of students in lecture capture environments. Computers and Education, 75, 282-292. doi: 10.1016/j.compedu.2014.03.002 Germany, L., 2012. Beyond lecture capture: What teaching staff want from web-based lecture technologies. Australasian Journal of Educational Technology, 28(7), pp.1208–1220. Gorissen, J. van Bruggen, & Jochems, 2013. Methodological Triangulation of Students’ use of recorded lectures. Holbrook, J. & Dupont, C., 2009. Profcasts and Class Attendance – Does Year in Program Matter? Bioscience Education, 13(June). Available at: www.bioscience.heacademy.ac.uk/journal/vol13/beej-13-c2.pdf. [Accessed 2 December 2014]. Karnad, A., 2013. Student use of recorded lectures. Available at: http://eprints.lse.ac.uk/50929/1/Karnad_Student_use_recorded_2013_author.pdf. [Accessed 3 December, 2014]. Larkin, H. E. (2010). ‘But they won't come to lectures...’ The impact of audio recorded lectures on student experience and attendance. Australasian Journal of Educational Technology, 26(2), 238- 249. http://www.ascilite.org.au/ajet/ajet26/larkin.html Massingham, P. & Herrington, T., 2006. Does Attendance Matter? An Examination of Student Attitudes, Participation, Performance and Attendance. Journal of University Teaching & Learning Practise, 3(2), pp.83–103. Nashash, H., & Gunn, C. (2013). Lecture capture in Engineering classes: Bridging gaps and enhancing learning. Journal of Educational Technology & Society, 16(1), 69- 78. http://www.ifets.info/journals/16_1/7.pdf Smith, E. & Volker, J., 2013. Enhancing Instruction with Visual Media: Using Video and Lecture Capture, IGI Global, Hershey, Pennsylvania. Von Konsky, B.R., Ivins, J. & Gribble, S.J., 2009. Lecture attendance and web based lecture technologies : A comparison of student perceptions and usage patterns. Australasian Journal of Educational Technology, 25(4), pp.581–595. Williams, A., Birch, E., & Hancock, P. (2012). The impact of online lecture recordings on student performance. Australasian Journal of Educational Technology, 28(2), 199- 213. http://www.ascilite.org.au/ajet/ajet28/williams.html FP:210 222 Miles, C.A. (2015). Strong and increasing student demand for lecture capture in the changing Australian university classroom: results of a national and institutional survey. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:204-FP:211). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:211 223 Analysis of MOOC Forum Participation Oleksandra Poquet Shane Dawson School of Education Learning and Teaching Unit University of South Australia University of South Australia The integration of social learning practices into massive open online courses (MOOCs) raises numerous learning and teaching challenges. While research into formal online education has provided some insight into the strategies for facilitating online learner-to- learner and learner-to-teacher interactions, the differences between MOOCs and more mainstream online courses impede any direct adoption and application. This paper reports a study linking the analysis of MOOC learner and teacher interactions to those in formal online education. The study compares MOOC forum activity of the individuals occasionally posting on the forum, and the ones contributing to the forum regularly. Through the social network analysis of forum posting and voting, we highlight the similarities and differences in how the networks of regular and occasional participants develop and interact. The findings provide some insight into how social learning practices can be promoted regardless of the course population size. Keywords: social learning, MOOCs, social network analysis, forum interactions Introduction The rapid push for scaling online learning among universities has in part manifested through the emergence of massive open online courses or MOOCs (Altbach, Reisberg, & Rumbley, 2009). A MOOC is an online non-accredited course, with flexible registration, and offered free of charge. Given the volume of students undertaking massive open courses, this model of education attracted much media attention for its perceived capacity to disrupt formalised tertiary education structures (Bulfin, Pangrazio, & Selwyn, 2014; Hollands & Tirthali, 2014). Despite MOOCs’ potential for widening user access to education, there remain numerous ‘for’ and ‘against’ arguments related to its quality and methods of instruction. The primary narrative so far has centered on the challenges of teaching and learning in the MOOC context and the development of a sustainable business model. The integration of social learning practices into open online courses remains a contested issue for MOOCs. Contemporary education and learning theory support the implementation of pedagogies that enable learning with others. However, the effectiveness of facilitating social learning activities becomes problematic when students reach well into the thousands (Miyazoe & Anderson, 2013; Stewart, 2013). Prior research in formal online education has identified some practices and processes that lead towards active learner-to-learner and learner-to-teacher interactions. For example, the use of meaningful tasks to prompt the exchange of ideas, teachers’ timely feedback and checks for understanding, as well as fostering a sense of community, has been noted to increase the quality and quantity of interactions (Darabi, Liang, Suryavanshi, & Yurekli, 2013; Ravenna, Foster, & Bishop, 2012). Yet, even if the suggested techniques were implemented at scale, empirical studies of MOOCs do not offer sufficient evidence to justify a simple transfer and adoption of formal online education practices. The direct application of effective practices from formal online learning to the MOOC context is prevented by the specific idiosyncrasies of MOOCs. In a formal online course, students enrol to receive credit and formal recognition of mastery, largely providing the student cohort with a shared goal. Conversely, in a MOOC students are driven by diverse goals, from sampling course content, to being interested in a subject, or in peer interactions (Eynon, 2014). Furthermore, MOOCs are much more asynchronous than conventional online education (Mullaney, 2014). In stark contrast to the compulsory start and finish times in formal online courses, individuals can join most MOOCs at any point of time in the course duration. This paper reports on a study linking the analysis of MOOC learner and teacher interactions to those FP:212 224 in formal online education. To do so, we first identify a group of MOOC participants with a certain level of regularity in their forum participation. This sub-group of MOOC participants is comparable with students in more conventional online offerings where instructors expect learners to post on the forum with a certain repeated frequency. A social network perspective is applied to analyse how the network of regular forum participants develops overtime, in relation to that of the entire MOOC cohort. The paper discusses the similarities and differences between the dynamics of regular and occasional MOOC participants, in light of the current research in social learning in MOOCs. Literature Review Much recent empirical research has been dedicated to MOOCs offered through a centralized platform such as edX or Coursera (Gasevic, Kovanovic, Joksimovic, & Siemens, 2014; Veletsianos & Shepherdson, 2015). The availability and access to student interaction data collected during the course offering has enabled institution-based research groups to rapidly investigate MOOCs from many alternate research perspectives. As students interact with course content and with each other on the discussion forum, MOOC platforms record their clicks and logs, as well as associated information, such as time of the logs, or content of the posts. MOOC researchers then extrapolate the trace data to signify student learning and engagement. For example, early efforts to understand MOOCs resulted in analyses of how the entire cohort of enrolled students interacted with the course resources, and which typologies of participants could be observed (Coffrin, Corrin, de Barba, & Kennedy, 2014; Ferguson & Clow, 2015; Kizilcec, Piech, & Schneider, 2013). These analytics suggest that individuals exhibit diverse preferences as to when, how and in which combination they watch lectures, use the forum, or complete assignments, if at all. It has also been observed that MOOCs experience a sharp decrease in participation within the first week(s) of the course before a gradual stabilization of participant activity occurs (Dawson, Joksimović, Kovanović, Gašević, & Siemens, 2015). In short, the numbers of MOOC participants decrease overtime and at different points of the course offering diverse clusters of participants present alternate activity patterns. While investigations of participant activity counts remains the dominant strand of MOOC research there is emerging work exploring student social engagements in forum discussions. In the online education context, discussion participation represents learner-to-learner and learner-to-teacher interactions that are instrumental to shortening spatial, temporal and psychological distance separating the learners (Moore, 1993; Thompson, 2007). The studies of social interactions in MOOCs target the relationship between students social positioning and the quality of posted text with students’ overall course performance, perseverance, and learning. For example, Jiang et al. (2014) demonstrated that for some courses a student’s network centrality measure derived from their discussions with peers was associated with a higher academic performance. In relation to course persistence, Rose et al. (2014) found that students’ inability to become a part of the forum conversation was associated with a high level of course disengagement. Similarly, Yang et al. (2015) investigated posted messages expressing confusion, and observed that the authors of such posts are more likely to disengage from the course, unless their confusion was resolved. In relation to learning, insights from forum analysis tend to conclude that social learning in MOOCs resembles ‘learning in a crowd’ with its fragmented groups and weak relationships (Gillani, 2013; Gillani, Yasseri, Eynon, & Hjorth, 2014; Milligan, 2015). Gillani has suggested that such fragmentation may not be detrimental to learning, as it could foster deeper conversations in smaller groups. However, Kellogg and colleagues (2014) counter this argument noting that forum conversations are typically at a low-level in terms of co-construction of knowledge. The authors demonstrated through the content analysis of the forum discussions that only 7% of all conversations go beyond the negotiation and co-construction of knowledge phases. Structural and content analysis of social interactions in MOOC forums provide valuable insights into learning at scale. However, studies taking on these methods commonly analyse the entire MOOC cohort, and do not overtly integrate the findings from prior research on participation patterns. In this study we suggest that connecting learner typologies with the inquiries into the structure and content of forum discussions will allow a more fine-tuned analysis of MOOC interactions. In their work on learner sub-populations, Ferguson and Clow (2015) distinguished various groups, among them so-called Returners—individuals comprising around 6-8% of the entire MOOC cohort, and characterized by a more regular participation. In alignment with Ferguson and Clow’s work, we delineated a sub- population of learners consistently present on the forum and applied social network analysis to investigate the structure of the entire cohort’s network and the structure of the regular participants’ FP:213 225 network. The study’s research question was: “How did the network of a set of regular forum participants develop compared to the network of the entire cohort in a MOOC?” Methods To address the posed research question, we analysed student interpersonal platform-based interactions of a Solar Energy MOOC offered by the Delft University of Technology via the edX platform. The duration of the course offering was for eight weeks over September – December 12, 2013. It enrolled 57091 students, and 2730 students received a certificate of completion. This MOOC was designed as a bachelor level foundation course, and required basic knowledge of physics and such mathematical skills as integration and differentiation. The course included over 9 hours of video lectures, as well as physics animations, numerous convergent quizzes, four homework assignments and three exams, with estimated 8 hours of workload weekly. Several staff members were appointed to look over the forum. After the first few days of the course active students were selected as community assistants. No special activities were offered to prompt interactions on the forum. Yet, the forum discussions were distributed within the course, as they were embedded next to the videos and assignments. Such strategy made it easier to locate specific discussions, while still within the platform. The course participants did not extensively use social media for course interactions. Facebook group set up by the participants comprised 171 people, including the staff, but was not as vibrant as the edX forum, and was mostly used for sharing links. Furthermore, although the course offered a Twitter hashtag for connecting outside of the platform, we have not detected much activity on Twitter in regards to this MOOC. In this study, we will refer to the main population of the course as the all learners group. This group comprised some 2343 forum participants who created 4727 posts reciprocated by others by a reply or a vote within the eight weeks of MOOC’s duration. Overall, 3820 students participated in the course forum. The group of all learners excluded some 1477 individuals whose forum contributions were never reciprocated by either reply or a vote. The overall pattern of participation on the MOOC forum by the entire cohort followed a typical engagement curve (Figure 1). Since the engagement curve does not capture the regularity of participation, but simply the volume of weekly activity, we also analysed the frequency of student returns to the forum for the entire course cohort (Figure 2). Based on these analyses, participation in at least three weeks of the course was chosen as the criterion for the inclusion in the group of regular participants. As the result, a group of students who returned to the forum to post or vote for (any) three weeks or more weeks of the course comprised 196 individuals. We will refer to this sub- population as regular participants group. Figure 1. Volume of forum participation of the Figure 2. Frequency of forum participation of entire MOOC cohort the entire MOOC cohort Additionally, we considered the MOOC as lasting three thematic modules, in lieu of a simple week-by- FP:214 226 week analysis for the 8-week 3 course duration. Each thematic module was based on the course design (topic modules) and lasted from 2-3 weeks. An assessment task marked the completion of each theme. Figure 3 summarises the presence of students during each thematic module of the course. Almost 75% of all learners engaged in social interactions for the first 1/3 of the course. However, the vast majority of the regular participants sought interactions with peers or instructors during 2/3 of the entire course content. These observations validate the assumed comparability between the regular participants and students enrolling in more formal online learning courses. Fifty percent (50%) of the regular participants were active on the forum during all three thematic modules; and 40% of the regular participants were active during the first two themes. 4 The course contained teaching staff and community assistants (CAs) (11 people). Since these individuals were active and present on the forum, they were all a part of the regular participants network. Their role in the forum was to actively engage with participants, to address questions and promote discussion of course concepts. To capture and distinguish these committed individuals from other regular participants in the structure of the network, we have constructed two versions of the regular participants networks: one that included staff and community assistants, and one that excluded them. To analyse the evolving network configurations we undertook a series of undirected weighted networks for all learners and regular participants (both with and without staff and CAs). These networks constituted participants’ co-occurrence in forum conversations. A conversation was defined as taking turns and contributing answers to one specific question or problem. That is, a co-occurrence of participants in the same forum thread did not result in a connection within the constructed network, if the individuals did not actively engage with one another in relation to the specific question. To illustrate, if A posted a question, and B and C replied to it, then A, B and C would all be linked by undirected edges in a graph. If in the same conversation D up-voted A’s post, the graph would also connect A and D by an undirected edge. Figure 3. Students active on the forum per thematic module To compare the development of regular participants network against all learner network, we calculated the network centralization measures (i.e., betweenness, closeness, degree) and the density for the networks of all learners, regular participants, and regular participants without staff and CAs. Density is considered as a measure of network cohesiveness, and indicates the ratio of all present connections between participants in relation to all possible connections (Carrington, Scott, & Wasserman, 2005). Centralization is a network-level measure that encapsulates the variation of 3 During the first three weeks of the MOOC students were learning introductory concepts that belonged to Theme 1. The second theme lasted next three weeks of the course, and Theme 3 was on offer for the last two weeks. 4 Community assistants (CAs) - students highly active in the first weeks selected by staff to help with the forum FP:215 227 individual centrality measures in a given network for: i) degree—the number of people one has co- occurred with; ii) betweenness—the measure indicating whether the individual has co-participated with other students who are otherwise unconnected; iii) closeness—the number of connections that exist between participants to link them directly, also denoting the “compactness” of the network (Wasserman & Faust, 1994). Centralization and density of the regular participants network in relation to the all learner network were plotted overtime at four different time points representing the identified course thematics: • Stage 1 for the first week of the course; • Stage 2 for thematic module 1 in weeks 2 and 3; • Stage 3 for the thematic module 2 in weeks 4, 5 and 6; • Stage 4 for thematic module 3 in weeks 7 and 8. Data manipulation and analysis was undertaken using the igraph package in R (Csardi & Nepusz, 2006). Results This paper analysed how the network of regular participants developed over the course offering compared with the network of the entire MOOC cohort. Given that regular participants returned to the forum, and were a relatively smaller group, we expected that this sub-population would have a much more cohesive network structure, compared to a loose network representing the entire cohort. The analyses indicate that both the all learners and regular participants networks had a similar structure. While the networks contained a small group of highly interconnected individuals that interacted with many people multiple times, the majority of participants interacted infrequently and only with a few people. We observed that 75% of regular participants most commonly interacted with 1 to 24 people in the course, and communicated with the same person once, on average. Yet, the remaining 25% of regular participants interacted with 24 to 179 people during the course, with the frequency of interaction with the same person ranging from 3 to 147 times. These inferences are derived from Table 1, illustrating the degree distribution, which here represents the number of people a participant co-occurred in a conversation with; while the mean edge weight denotes the frequency of co- occurrence with the same individual. A power-law distribution was observed in the all learners network. The majority (75%) of all learners interacted during the course with 1 to 32 people with an average frequency of once. The remaining 25% interacted with 32 to 593 people, with a frequency ranging from 1 to 147 times. It can be concluded that the most frequent interactions between the same individuals took place within the regular participants, since the maximum values for the edge weight are shared across the two networks. Table 1. Description of All Learner and Regular Participants Networks All Learners Regular Participants Nodes 2434 196 Edges 27559 2016 Density 0.009 0.1 Centralization Degree 0.23 0.81 Betweenness 0.18 0.23 Closeness 0.0009 0.62 Degree Distribution First Quartile 2 7 Median 8 15 Mean 22.6 20.5 Third Quartile 32 24 Maximum 593 179 Edge Weight Distribution FP:216 228 First Quartile 1 1 Median 1 1 Mean 1.5 3.5 Third Quartile 1 3 Maximum 147 147 Average Edge Weight Distribution First Quartile 1 1.4 Median 1 2.1 Mean 1 2.6 Third Quartile 1.32 3 Maximum 12.3 14 The network representing the entire cohort is loosely coupled, highly decentralized, and characterized by most individuals located in smaller interconnected parts of the network that are linked by several individuals who co-participate in these otherwise disparate clusters. The regular participants network is similar in structure. However, the network is much more centralized than the all learners network with a degree value of 0.81 and 0.23 respectively. The closer the centralization measure is to ‘1’, the more highly centralized the network is, and marked by the clear boundary between its core and periphery (Scott, 2013). Given network’s power-law distribution and low betweenness value, one can assume that those who participated in most conversations, i.e. having high brokering power, would also be connected to the highest number of people (degree). Besides being brokers between conversations, they would also shorten the distance between the regular participants, as well as control much of the information within the network thereby explaining the differences in the observed network centrality measures. The pattern of development for the networks representing the entire cohort and more regular participants (both student-only and students-and-staff interactions) has been consistent up until Stage 3 (Figure 4). The network of the entire cohort retained its low density and low closeness centrality indicating that individuals were only connecting with a few people, and the network members were sparsely distributed. The period between Stage 1 and Stage 2 is marked by a growth in the degree and betweenness centrality measures. Such a dynamic signifies an increase in the network centralization post week 1 due to the emergence of more active participants. In contrast, regular participants network shows a steady change in structure from a loose to a more inter-connected network. Although its density remained low at all time points, its value is much higher for regular participants without staff and CAs suggesting that the regular participants activated more possible connections between each other, than those students participating infrequently. The much higher centralization and closeness of the regular participants network as compared to the same network without the staff and CAs points to the role played by this dedicated group of people. The staff and CAs provided a bridge between the structural holes in the network, thereby significantly reducing the distances among the actors. In conclusion, while still maintaining a loosely coupled structure, the regular participants formed stronger relational ties among its central actors and established their wide outreach to loosely interconnected periphery. Such a structure is reflective of the developmental stages of a community of practice. FP:217 229 Figure 4. Development of all learner network (red), regular participants network including staff and CAs (green) and regular participants without staff and CAs (blue). Networks are described by their centralization (betweenness, closeness, degree) and density at four different stages of the MOOC The overall pattern of network development for this MOOC suggests an increase in activity among the regular participants network up until Stage 3. However, in Stage 4 the dynamics of development between the network of occasional and regular participants is reversed. The regular participants network in the last stage becomes less clustered; its members did not extensively interact with each other, as compared to the previous stages. On the other hand, we observe an increase of all group measures for the network representing the entire MOOC cohort. This divergence of the patterns is symptomatic of the increased activity among the occasional participants at the end of the course. The catalyst for the change in user behaviour may be contingent upon a contextual factor. During Stage 4 the staff announced the criteria for awarding a free educational trip for two MOOC participants. These criteria included a perfect score on the exam, as well as a certain level of forum participation. The motivation to fulfil such criteria led to a sudden increase in forum activity during the last two weeks of the course. The flurry of activity extended to individuals that had previously not engaged in discussion activity. For example, one learner who never participated in the forum, created over 300 posts during the final week (week 8), placing a posting into various conversations dating back to a time as early as week one. This person also numerously up voted their own posts to imply a greater level of prestige. Other students exhibited similar behaviour generating larger activity within the all learner network. This type of activity was not well received among the regular participants network with many individuals posting messages indicating their disapproval of such behaviour. It is plausible that this impacted negatively on regular participants’ motivation thereby resulting in the observed diminished discussion activity. Discussion This paper analysed the development of two learner networks (entire cohort and regular participants) evolving from participation in a MOOC discussion forum. The analyses suggest that the network structures observed between the 2 groups are not dis-similar. Regardless of whether forum participation has been sustained or occasional, the networks representing intperpersonal interactions are loosely connected, clustered, with hubs of activity linked by the individuals with higher degree of FP:218 230 participation. Similar to Gilliani’s et al.’s (2014) prior findings we observed that structurally limited conversations occurred in fragmented groups, and a small group of people participated across them. In these disparate conversations around 75% of the participants of the entire cohort, including those who posted regularly, were likely to have one-time encounters with the same person. While it is evident that the vast majority of connections made in the forum could be classified as weak and infrequent, a quarter of the interactions in the regular participants network were recurrent. In fact, there were pairs of individuals who interacted with each other in over a hundred of instances. This suggests that among this diverse and disparate network strong relationships can still be established. The networks of all learners and regular participants resemble each other structurally. Even so, we presume that these two networks may be characterized by different modes of peer production processes. In open online environments individual commitment to collaborative knowledge production ranges from lightweight to heavyweight (Haythornthwaite, 2009). Within this continuum, a heavyweight mode represents strong-tie affiliation with community members, its purpose and peer- negotiated norms. From such a standpoint, infrequent ties formed through forum activity signal lightweight participation made up of interest-based contributions with low-level commitment to maintaining or creating relationships. Given that most ties between participants seem to be one-time occurrences, learning in a MOOC forum can be described as ‘learning in the crowd’. While for most participants in centralized MOOC forums the commitment to social interactions can be regarded as impersonal and lightweight, there are frequent interactions among the same individuals. A quarter of interactions in the regular participants network may be indicative of strong ties typical for heavyweight commitment to forum participation. Heavyweight peer production refers to the sustained contributions to the perceived community, as well as monitoring its viability (Budhathoki & Haythornthwaite, 2012). This study demonstrates that active students appointed to maintain the forum community are active contributors and broker information between conversations. These more active and engaged participants are central to the regular participants network, and are thus more likely to have frequent encounters with their fellow participants. Given that up until Stage 3, the network of regular participants was gradually becoming more interconnected, it can be assumed that students as well as community assistants co-occurred with each other time and again. Such co-occurrences may have resulted in shared history, and may have shifted from impersonal contributions to the one where participants identified each other. It is also plausible that norms of behaviour, interaction and participation were negotiated through this shared history. In this case, regular participants attitude to the sudden raise of ‘random’ contributions by the end of the course is a manifestation of ‘them’ vs. ‘us’ reaction. Such reaction would indicate their developed sense of membership in the group with perceived boundaries. Processes of repeated interaction, norm negotiation, commitments to quality of collective products, are atypical to crowds, but characteristic of the communities. In this paper we can only hypothesize that these two networks represent overlapping social entities defined by different social processes. Characterizing the content of the more frequent ties was beyond the scope of this paper. Current research also did not offer straightforward insights into the nature or quality of the stronger dyadic relationships developed in MOOC forums. We can surmise that stronger ties would be sites for higher percentage of knowledge construction incidents than the low 7% observed by Kellogg et al. (2014) in the entire cohort. It is also reasonable to say that the individual active students are the hyperactive individuals keeping the spirit of the forum (Huang, Dasgupta, Ghosh, Manning, & Sanders, 2014; Papadopoulos, Sritanyaratana, & Klemmer, 2014). They are also probably proficient in learning from many people, which would then define them as experts in crowd-sourced learning, according to the research by Milligan (2015). Alternatively, in reference to the research by Yang et al. (2015), we can expect that the threads expressing confusion and left unresolved would be more typical to the ‘occasional’ participants, while the unresolved threads expressing confusion by regular participants are less likely to result in the disengagement with the course. Yet, these are mere extrapolations of the findings pertaining to research spanning an entire MOOC cohort. Further inquires are required to identify how the strength of a relationship between individual actors in a MOOC influences the quality of discussion and depth of knowledge construction. Understanding the qualitative differences between ties of higher frequency in both all learners and regular participants networks, as well as learning about the attributes of individuals who share strong ties may aid current efforts to devise technology for matching learners for a synchronous conversations in a MOOC forum (Ferschke et al., 2015). The findings from this study also provide some practical conclusions. By establishing the FP:219 231 comparability of the group of regular participants as similar to the formal online learning student groups opens up opportunities for transferring “best practice” and innovating teaching techniques within MOOCs. For example, we observe the importance of forum facilitators and highly active students in the development of the network. Prior research suggests that students in such social positions carry a higher sense of belonging than their less well-connected peers (Dawson, 2008). Our analysis indicates that highly active students and facilitators develop numerous ties of higher frequency. These individuals could potentially take on more of an instructional role by scaling feedback approaches and instilling a sense of belonging, in a manner that is reflective of a teacher in a more formal and bounded groups. However, such activities and roles can also lead to student dominance. In order to avoid such an event, as well as any potential for inadvertently exploiting volunteer efforts, an instructor may consider a rotation of community assistantship, thereby delegating the dedicated role to a number of active students. The present study raises questions related to the types of methodologies and approaches that are effective for researching social learning at scale. To better comprehend the complexities of social learning in MOOCs, researchers’ need to apply appropriate and diverse theoretical lenses to alternate units of analysis – from an entire cohort to individual actors. Networked learning (NL) may provide a sound theoretical framework for describing the various overlapping relationships that co-exist in the complex social organizations that manifest in educational settings (Jones, 2004a, 2004b; Jones, Ferreday, & Hodgson, 2008). Analysing a social entity from the NL perspective does not bias or privilege the strong relationships that imply closeness and unity of purpose within a group of actors (Jones, 2004a, 2004b; Jones & Esnault, 2004). Prior research has well demonstrated the benefits of using NL as an interdisciplinary framework for the analysis of learning due to its capacity to address “multiple scales of groups at multiple granularities of analysis, with multiple methods and theoretical foundations” (Suthers, Hoppe, De Laat, & Shum, 2012). Integral to this work has been the development of methodologies to investigate NL. For instance, de Laat et al. (2007) have outlined the potential of social network analysis (SNA) to inquire about the nature of NL, while Jones (2004b) suggested that network analysis of the links and relationships in NL environments needs to be supplemented with a qualitative analysis of their nature. Consequently, there exist a number of methodological frameworks that utilize social network analysis and complementary methods and diverse techniques to contextualize it (De Laat, Lally, Lipponen, & Simons, 2007; Suthers, 2015). In conclusion, it appears that analysing social interactions in MOOCs from NL perspective and through NL methodologies could further enrich our understanding of learning at scale. Such an approach would allow for the capture of social learning ties of differing strength, as well as defining their role and meaning through the qualitative analysis of such tie types, strength, as well as the socio-cultural dimensions underpinning the network structure and formation. Acknowledgements The authors would like to thank TU Delft Online Learning, in particular Thieme Hennis and Pieter de Vries, for the assistance with obtaining the data, and Srecko Joksimovic for the preparation of the data for this analysis. References Altbach, P. G., Reisberg, L., & Rumbley, L. E. (2009). Trends in global higher education: Tracking an academic revolution. UNESCO Pub.; Sense. Retrieved from http://atepie.cep.edu.rs/public/Altbach,_Reisberg,_Rumbley_Tracking_an_Academic_Revolut ion,_UNESCO_2009.pdf Budhathoki, N. R., & Haythornthwaite, C. (2012). Motivation for Open Collaboration: Crowd and Community Models and the Case of OpenStreetMap. American Behavioral Scientist, 57(5), 548– 575. Bulfin, S., Pangrazio, L., & Selwyn, N. (2014). Making “MOOCs”: The construction of a new digital higher education within news media discourse. The International Review of Research in Open and Distributed Learning, 15(5). Retrieved from http://www.irrodl.org.ezlibproxy.unisa.edu.au/index.php/irrodl/article/view/1856 Carrington, P. J., Scott, J., & Wasserman, S. (2005). Models and methods in social network analysis (Vol. 28). Cambridge university press. FP:220 232 Coffrin, C., Corrin, L., de Barba, P., & Kennedy, G. (2014). Visualizing patterns of student engagement and performance in MOOCs. Proceedings of the Fourth International Conference on Learning Analytics And Knowledge - LAK ’14, 83–92. Csardi, G., & Nepusz, T. (2006). The igraph software package for complex network research. InterJournal, Complex Systems 1695. Retrieved from http://igraph.org Darabi, A., Liang, X., Suryavanshi, R., & Yurekli, H. (2013). Effectiveness of Online Discussion Strategies: A Meta-Analysis. American Journal of Distance Education, 27(4), 228–241. Dawson, S. (2008). A Study of the Relationship between Student Social Networks and Sense of Community. Educational Technology & Society, 11(3), 224–238. Dawson, S., Joksimović, S., Kovanović, V., Gašević, D., & Siemens, G. (2015). Recognising learner autonomy: Lessons and reflections from a joint x/c MOOC. Presented at the The Higher Education Research and Development Society of Australasia, Melbourne, Australia. Retrieved from https://asnevents.s3.amazonaws.com/Abstrakt-FullPaper/22639- Dawson_22639_version03.pdf De Laat, M., Lally, V., Lipponen, L., & Simons, R.-J. (2007). Investigating patterns of interaction in networked learning and computer-supported collaborative learning: A role for Social Network Analysis. International Journal of Computer-Supported Collaborative Learning, 2(1), 87–103. Eynon, R. (2014). Conceptualising interaction and learning in MOOCs. MOOC Research Initiative Report Ferguson, R., & Clow, D. (2015). Examining Engagement: Analysing Learner Subpopulations in Massive Open Online Courses (MOOCs). In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 51–58). Poughkeepsie, New York: ACM. Ferschke, O., Howley, I., Tomar, G., Yang, D., Liu, Y., & Rosé, C. P. (2015). Fostering Discussion across Communication Media in Massive Open Online Courses. In Proceedings of Computer Supported Collaborative Learning. Retrieved from http://www.isls.org/cscl2015/papers/MC-0404- FullPaper-Rose.pdf Gasevic, D., Kovanovic, V., Joksimovic, S., & Siemens, G. (2014). Where is research on massive open online courses headed? A data analysis of the MOOC Research Initiative. The International Review of Research in Open and Distributed Learning, 15(5). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1954 Gillani, N. (2013). Learner Communications in Massively Open Online Courses. Master's Thesis. Retrieved from http://iamsoalive.com/doc/dissertation_learner_communications.pdf Gillani, N., Yasseri, T., Eynon, R., & Hjorth, I. (2014). Structural limitations of learning in a crowd: communication vulnerability and information diffusion in MOOCs. Scientific Reports, 4. Retrieved from http://www.nature.com/articles/doi:10.1038%2Fsrep06447 Haythornthwaite, C. (2009). Crowds and Communities : Light and Heavyweight Models of Peer Production. Big Island, Hawaii, Hawaii International Conference on System Sciences. Hollands, F., & Tirthali, D. (2014). MOOCs : Expectations and Reality. Retrieved from http://www.academicpartnerships.com/sites/default/files/MOOCs_Expectations_and_Reality.p df Huang, J., Dasgupta, A., Ghosh, A., Manning, J., & Sanders, M. (2014). Superposter behavior in MOOC forums. In Proceedings of the first ACM conference on Learning@ scale conference (pp. 117–126). ACM. Jiang, S., Fitzhugh, S. M., & Warschauer, M. (2014). Social Positioning and Performance in MOOCs. In Proceedings of the Workshops held at Educational Data Mining 2014, co-located with 7th International Conference on Educational Data Mining (EDM 2014) (Vol. 1183, p. 14). London, United Kingdom. Jones, C. (2004a). Networks and learning: communities, practices and the metaphor of networks. Research in Learning Technology, 12(1). Retrieved from http://journals.co- action.net/index.php/rlt/article/view/11227 Jones, C. (2004b). The conditions of learning in networks. In Kaleidoscope CSCL SIG First Symposium, October 7th-9th 2004 (p. 13–pages). Retrieved from https://telearn.archives- ouvertes.fr/hal-00190649/ Jones, C., & Esnault, L. (2004). The metaphor of networks in learning: Communities, collaboration and practice. In Networked Learning 2004 Conference (p. 8). file://localhost/Retrieved from https/::telearn.archives-ouvertes.fr:hal-00190280: Jones, C., Ferreday, D., & Hodgson, V. (2008). Networked learning a relational approach: weak and strong ties. Journal of Computer Assisted Learning, 24(2), 90–102. FP:221 233 Kellogg, S., Booth, S., & Oliver, K. (2014). A social network perspective on peer supported learning in MOOCs for educators. The International Review of Research in Open and Distributed Learning, 15(5). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1852 Kizilcec, R. F., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: Analyzing Learner Subpopulations in Massive Open Online Courses. New York, New York, USA: ACM Press. Laat, M. D., Lally, V., Lipponen, L., & Simons, R.-J. (2007). Online teaching in networked learning communities: A multi-method approach to studying the role of the teacher. Instructional Science, 35(3), 257–286. Milligan, S. (2015). Crowd-sourced Learning in MOOCs: Learning Analytics Meets Measurement Theory. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 151–155). Poughkeepsie, New York: ACM. Miyazoe, T., & Anderson, T. (2013). Interaction Equivalency in an OER , MOOCS and Informal Learning Era, 1–15. Moore, M. G. (1993). Theory of transactional distance. Theoretical Principles of Distance Education, 22. Mullaney, T. (2014). Making Sense of Moocs: A Reconceptualization of HarvardX Courses and Their Students. Available at SSRN 2463736. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2463736 Papadopoulos, K., Sritanyaratana, L., & Klemmer, S. R. (2014). Community TAs scale high-touch learning, provide student-staff brokering, and build esprit de corps. In Proceedings of the first ACM conference on Learning@ scale conference (pp. 163–164). ACM. Retrieved from Ravenna, G., Foster, C., & Bishop, C. (2012). Increasing Student Interaction Online : A Review of the Literature. Journal of Technology and Teacher Education, 20(2), 177–203. Rosé, C. P., Carlson, R., Yang, D., Wen, M., Resnick, L., Goldman, P., & Sherer, J. (2014). Social factors that contribute to attrition in moocs. In Proceedings of the first ACM conference on Learning@ scale conference (pp. 197–198). ACM. Scott, J. (2013). Social Network Analysis (Third Edition). SAGE Publications Ltd. Stewart, B. (2013). Massiveness+ openness= new literacies of participation. MERLOT Journal of Online Learning and Teaching, 9(2), 228–238. Suthers, D. (2015). From Contingencies to Network-level Phenomena: Multilevel Analysis of Activity and Actors in Heterogeneous Networked Learning Environments. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 368–377). Poughkeepsie, New York: ACM. Suthers, D., Hoppe, H. U., De Laat, M., & Shum, S. B. (2012). Connecting levels and methods of analysis in networked learning communities. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 11–13). ACM. Thompson, M. (2007). From Distance Education to E-Learning. In R. Andrews & C. Haythornthwaite (Eds.), The SAGE Handbook of E-learning Research (pp. 159–179). London: SAGE Publications Ltd. Veletsianos, G., & Shepherdson, P. (2015). Who studies MOOCs? Interdisciplinarity in MOOC research and its changes over time. The International Review of Research in Open and Distributed Learning, 16(3). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2202 Wasserman, S., & Faust, K. (1994). Social network analysis: Methods and applications. Cambridge: Cambridge University Press. Yang, D., Wen, M., Howley, I., Kraut, R., & Rose, C. (2015). Exploring the effect of confusion in discussion forums of massive open online courses. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale (pp. 121–130). ACM. Poquet, O. & Dawson, S. (2015). Analysis of MOOC Forum Participation. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:212-FP:222). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:222 234 Designing for relatedness: learning design at the virtual cultural interface Alison Reedy Michael Sankey Office of Learning and Teaching Learning Environments and Media Charles Darwin University, Australia University of Southern Queensland, Australia This paper draws on the initial analysis of data from an education design research study that investigated the experience of Indigenous higher education students in online learning. The interrelated themes of racial identity and relatedness were found to be significant to the experiences of these students. The paper examines a number of widely used learning design models and online facilitation approaches to determine the extent to which identity and relatedness are considered in the design of online environments and in the facilitation of learning. It concludes with a series of recommendations as to how an institution may mediate a level of relatedness for its students in online learning environments. Keywords: Relatedness, design models, e-learning, Aboriginal and Torres Strait Islander. Setting the scene This paper explores emergent concepts and practices associated with identity and relatedness as they apply to learning and teaching (L&T) and the way in which Aboriginal and Torres Strait Islander higher education (HE) students experience this in online learning. Relatedness is understood in various ways across cultures: for example, in relation to kinship and country in Indigenous contexts (Martin, 2003), through social capital theory (Coleman, 1988), and in online environments through networked learning (Goodyear, Jones, Asensio, Hodgson, & Steeples, 2005) and connectivism (Siemens, 2004). Relatedness in the context of this study refers to the trust and reciprocity in bonding, binding and linking relationships (Woolcock & Narayan, 2000) that is mediated inside and outside of virtual learning environments and which affirms cultural and racial identity and practices. The conceptual location where these ideas and practices are negotiated and reframed is at the virtual cultural interface, extending Nakata’s (2007) concept of the cultural interface as a space where collaborative meaning making takes place and where worldviews can be renegotiated through cross- cultural interactions. This paper draws from an education design research (EDR) study conducted at Charles Darwin University (CDU), a regional university in the Northern Territory (NT), Australia. Due to its isolated geographic location and the NT’s relatively small and dispersed population of 243,800 people (Australian Bureau of Statistics, 2015) CDU has a strong focus on external delivery. In 2014 62% of its students were enrolled externally (Reedy, Boitshwarelo, Barnes, & Billany, 2015), with almost all its units being offered online through CDU’s learning management system (LMS). Aboriginal and/or Torres Strait Islanders comprise 30% of the population of the NT, the highest of any Australian state or territory (Australian Bureau of Statistics, 2015), and 5.5% of CDU’s HE enrolments (CDU 2015). This compares well to the overall percentage of Indigenous students in Australian universities of 1.4% (Universities Australia, 2014). However, Indigenous student retention at CDU is 20% lower than for non-Indigenous students: 79% against 59% (CDU, 2015). This disparity illustrates there is significant change required in the learning environment to achieve equitable outcomes for Indigenous students. The study was, therefore, undertaken in order to better understand the lived experience of Indigenous higher education (HE) students participating in learning environments where online study is increasingly the norm. The themes of identity and relatedness, described in this paper, are well researched in the fields of Indigenous health, wellbeing and education (Dudgeon, Milroy, & Walker, 2014), however, little is known of their impact on Indigenous HE students in online learning. The study draws on previous work that indicates that cultural difference impacts on the experience of learners in online learning environments (Hall, 2009; Russell, Kinuthia, Lokey-Vega, Tsang-Kosma, & Madathany, 2013) and needs to be considered in online learning design (McLoughlin & Oliver, 2000). The paper explores the extent to which cultural difference, around the notion of connectedness, can lead to a sense of relatedness in the online environment. This is initially seen through the eyes of FP:223 235 Indigenous students and then by examining different online design models, comparing those that focus on online interaction against those that promote online presence, and how this may be applied when designing learning environments. It considers the notion of online presence and the development of an online identity, and how this may be applied to these environments where representation of cultural identity is important. The paper considers some of the issues faced by students when navigating online interactions in what is fast becoming the default way to communicate within courses. And lastly it considers how education designers and academics may design online learning experiences to moderate this phenomenon for Aboriginal and Torres Strait Islander students. Research Design Design This paper is based on a qualitative study in which Indigenous research approaches (Martin, 2003; Rigney, 2006) guided the conduct of a non-Indigenous researcher within an EDR framework (Anderson & Shattuck, 2012; Reeves, McKenney, & Herrington, 2010). The voices of the participants were privileged through a ‘participant-oriented evaluation phase’ within the EDR framework and data were collected through yarning, an approach to gathering rich narratives about a participant’s lived experience regarded as culturally appropriate for Indigenous peoples (Bessarab & Ng'andu, 2010). Participants Participants were drawn from students enrolled at CDU in 2014-15 and students who had withdrawn from online study up to two years prior to the study. To facilitate the identification of participants, the Office of Indigenous Academic Support (OIAS) contacted all Indigenous students in under- and post- graduate coursework units by email to endorse and promote their participation. A relatively low number of students (11) responded, with nine going on to participate. Purposive sampling (Babbie, 2007) was used to identify additional participants from existing university and social networks. In total, sixteen students (11 female, 5 male) participated, ranging in age from 22 to 66 years. This sampling method also ensured the study was inclusive of perspectives across gender, age, discipline areas and geographic location and represented multiple disciplines of study. Research Methods In depth interviews were conducted with participants using the technique of yarning (Bessarab & Ng'andu, 2010; Kickett, 2011). This is a familiar and informal conversational style that can help participants feel comfortable and relaxed. It provided participants with the space to represent themselves and their journeys through education and online learning in their own voices. It also allowed the researcher to build relationships with the participants, to become ‘known’ to them. Interactions commenced with “social yarning” (Bessarab & Ng'andu, 2010, p. 40) to establish relationship prior to commencing the “research topic yarn” (p. 40). The yarning sessions were conducted face to face where geographically possible, or by telephone where participant were located remotely. The length of the sessions varied between 35 minutes to just under 2 hours. In some instances the yarning was conducted over a number of sessions. All sessions were conducted by one researcher, recorded and transcribed (verbatim), and sent back to the participants for verification. The researcher also engaged in “collaborative yarning” (Bessarab and Ng’andu, 2010, p. 40) with the project’s Indigenous reference group. Collaborative yarning is the process of talking about research and is one of four types of yarning that Bessarab & Ng’andu observe takes place within a research setting. Data Analysis The verified transcripts were coded in NVivo (QSR International, 2015), based on key ideas, interesting points, notable examples or incidents, strong positive or negative incidents and/or reactions. Emergent themes were then discussed with the Reference Group in collaborative yarning sessions. The reference group comprised of an academic staff member, a member of the Office of Indigenous Academic Support and a student representative. An additional non-Indigenous academic with extensive experience in Indigenous research in New Zealand also participated in these sessions. This allowed for discussion and clarification of the themes, as well as prompting further lines of FP:224 236 inquiry. This collaborative yarning process was powerful and contributed to collective meaning making of the data. Results: The yarns The interrelated themes of identity and relatedness were extracted from the analysis. These themes are well established in Indigenous research (Dudgeon, Milroy, & Walker, 2014) and are linked to concepts such as resilience (Kickett, 2011). The following section lets us hear these emergent themes in the words of the Indigenous participants. Stories of identity and relatedness were integral to the participants’ understandings of themselves as learners in online environments and it is notable that while these stories tell of very different experiences in terms of the strength of the connections made in these environments, concerns about relatedness and how it is mediated was a common theme. Just as the process of yarning led to the development of relationships and connections between researcher and participants, so too did the participants’ express a fundamental need to connect with other students and establish networks within the learning environment to support their learning. The participants described varying degrees to which they were able to build these connections and relationships online. For some the online learning environment was a foreign space that offered little possibility for connection: “So I guess for the first time I was looking, within a mainstream environment, I was, I felt like I was the outsider” (F37). When the sense of connection within the online learning environment was weak, the participants’ implemented compensatory strategies to leverage networks outside the online learning environment to support their learning intentions. For example, one participant who transitioned from studying internally at one university to externally at the site of this research made the heartfelt comment “You know, I need my husband because I miss my uni” (F29). Another participant also identified the lack of connections she made in the online environment and stated: F46: If I didn’t have had my sister doing the same unit as me I pretty much thought I would have dropped out at that first semester level. AR: What sort of support did you get within the course and from fellow students? F46: Nothing. Only my sister. The extent to which participants drew on existing networks, including family networks, to support their learning varied and was influenced by a range of factors, including the absence of connections being formed within their online learning environments. For some participants family connections were strong, as seen above, where the presence of a sister studying in the same online unit was a factor in stopping the participant dropping out of the course. Some participants, on the other hand, had weak family ties as a result of family breakdown, family separation through stolen generation, and the on going intergenerational impacts of racism that resulted in families disengaging from each other. Regardless of the strength of family ties, each of the participants had existing networks that enabled critical on going support for their participation in HE. Some of these bonds were unexpected, as in the case of the on going friendship and support from an ex-boyfriend’s father who one participant advised “If I need school books and I don’t have the money he’ll loan me the money” (F26). The participants drew on existing networks and close relationships for the support needed to enter HE and remain in it. Networks outside of online learning environments also contributed to participants’ experiences in online learning. The positioning of learning as part of a broader network, not just isolated within online environments, is recognised in the following comment: To be a successful learner, and to be a successful learner as a woman, as a mother, as a partner, as a community member, you’ve got to be able to turn what you’re learning into what you are doing (F37). Given the participants’ desire to extend their existing networks and to build connections with other students and lecturers within their units and course, it was notable that they were overwhelmingly dissatisfied at the extent to which they were able to do this. Some participants explained this as being a result of the lack of opportunity for culturally safe interactions in meaningful learning contexts within designed learning environments. Additionally, most but not all participants wanted to disclose their Indigenous identity as part of their online persona but felt constrained to do so. The vast majority FP:225 237 wanted to connect with other Indigenous students in the first instance. However, the formation of relationships with other Indigenous students in the online environment was not an easy to achieve. In many instances there were few or no other Indigenous students in their online classes and if there were, they were not easily identifiable: “There were no other Aboriginal students online that I knew of” (F37). The frustration felt at this by many participants is expressed in the following comment: So I was like, well you know, what am I supposed to do? Stand up in my online lecture and say, “Hey, I’m a blackfella. Is there anyone else out there? You know? (F29) This inability to locate other Indigenous students was regarded by many students as a lost opportunity for connection and relatedness within the online environment. The extract below indicates the common bonds and relationships that can be established when that opportunity to identify occurs. AR: Would it have made a difference to you in your studies if there had been [other Aboriginal students in the online unit] and if you had known that? F37: Absolutely! Well you know that you can connect with people. There is a connection. There’s this unwritten rule of ‘Yeah! You’re from there! I’m from here! Yeah, yeah! What’s going on over there? You know, there’s instantly, you’ve got something to talk about. There’s always food, relations. In some way you’re always bloody connected. You always find that you know, you’re one person removed from who you’re talking to. While identifying as Indigenous was important for many participants, there were others who felt that this was either not important or they were wary of disclosing their racial positioning in the online environment. It is important to recognise that the participants’ held multi-faceted contemporary Indigenous identities and accordingly wanted to represent themselves in different ways. In the example below, one participant was wary of what others might think if she disclosed her racial identity explicitly online. F39: For Indigenous students. I just think it’s really hard, like I think that if I put on there ‘Hi, I’m [name] and I’m Aboriginal’ I think everyone would be like ‘why?’ AR: Ah. OK. F39: ‘Why? Why do you need to tell me that?’ Do you know what I mean? I know they physically can’t see me, but they’d probably be wondering well ‘why? What do you…? Why are you telling us that?’ The structure of an online learning environment in HE provides a virtual space where people from diverse backgrounds and groups can gather and make connections. In some instances previous experiences of racism was a factor that discouraged students from identifying as Indigenous in these environments: “And then there’s also that risk of being pre-judged” (F39). However, this guardedness made it more difficult to find connections and build relationships. Overall, there was limited evidence of Indigenous students building relationships with other students or their lecturers in the LMS. I just couldn’t make the connections to anybody. I couldn’t make it to the lecturer. I couldn’t make the connection to the other students. I couldn’t even connect with the Indigenous unit (F30). While the participants universally disliked ‘group work’, it did make it possible to build relationships between students. This contradiction speaks to some of the ever present issues in group work such as the logistical difficulties of finding mutually convenient times to meet online, the lack of guidance seen in many group work activities, and experiences of unequal contributions. However, when group work took place it provided opportunities for the participants to make connections with other students. We had to do group sessions and we had people from around the world or around Australia and you had to try and lock in a time. But when we did we worked really well together. But it was up to the students, you know, to really work that out (F28). [We had] random grouping. It was cool in a random grouping. And then you’d introduce yourself, and it was nice that you thought all these other people was just here in the Northern Territory, but not really. We had them all around Australia. And you get to meet people. And we had the CDU student come out to Maningrida so I got to meet a couple of them (F40). These statements highlight the tension in the participants’ desires to connect with others, the value of this when it occurs in meaningful learning contexts, yet their overwhelming reluctance to interact FP:226 238 online in the LMS until connections had been made and the ‘other’ was known. Some of this dilemma may be a result of the perception that the LMS is a formal academic space without a place for social presence, as distinct from social media spaces where connections are inherent in their design. I think that yeah, definitely social media is more of an attraction I guess, because [the LMS] is strictly an academic study setting which I can completely understand, whereas Facebook you can be playing with a Facebook app while you’re following what’s being said. And on top of that you can have your own personal discussions (F26). On the other hand, while there was great variation in the levels of interest in and use of social media tools by the participants in those units where social media was used and the participants opted into it they experienced an increased sense of connectedness. This is illustrated in the conversations below. F28: In one of the units I was studying … the lecturer would put newspaper articles or media stories on Facebook that linked with the unit. The lecturer would also give reminders about when assessments were due. When announcements were made in [the LMS] it would come up on your newsfeed or something or it would be sent as an email as well saying * has pasted on your Facebook. It really helped you connect ‘cause it really went and grabbed the student’s attention. It grabbed my attention. Reminder about study, you know, a reminder about study. So it was really helpful. AR: Did that also connect you with your lecturers or other students more? F28: It did. I felt more connected with the lecturers in those units, I did. And I was able to contact them straight away. Like I could just reply to posts and they would see it straight away. And everyone could see it. And I know discussion boards are like that, but it just felt more open to use and easier to use. AR: Easier probably because it’s something you are familiar with in another part of your life I guess. F28: That’s right. Yeah. It shouldn’t be, but yeah. This engagement and connection between social media tools and academic spaces does not happen without planning and design of learning environment. While some participants could not see, or had not experienced a link between social media and learning, others were more than ready for it. [My phone] it’s connected at my hip! Yeah. Facebook, Twitter, Hotmail, Yahoo, a lot of different things. Skype, Skype friends… (F37) AR: Where does the formal learning and the social live? Do they intersect? F37: Absolutely! On every level. Yeah, because, unless learning is relevant to your life and you can apply it and it’s part of your everyday life, you’re not going to get as much out of learning as what you could if it was formalised. So high, and it’s so disconnected. Everything’s connected. I think for Aboriginal people everything has to be connected to your real life, everything’s got to be… if you’re learning you’ve got to be able to apply that learning in your own environment. I mean we all live and breathe social media and that’s the way our life, that’s the way we function now. So the universities aren’t moving quickly enough. They’re not even in with the realities… (F37) The participants overwhelmingly experienced a sense of disconnection and isolation within the online learning environment, in contrast to the connections and linkages they anticipated. Moreover, this disconnection contrasted with the connections they experience through the use of social media. It is reasonable to suggest that ‘relatedness’ is a factor that influenced their experiences in the learning environment and hence, opportunities to build meaningful connections in these environments may enhance that experience. Additionally, recognising and linking into students’ existing networks and the tools they already use for networking may facilitate linkages between their learning environments and their ‘realities’. It is therefore conceivable and possible to create an environment for students where social presence can be fostered: one where a sense of relatedness can begin to be mediated by combining meaningful online interactions with a students’ identity within an institutional system. FP:227 239 Unpacking Relatedness Social connection is a fundamental human need (Chen et al., 2015) although the way ‘relatedness’ is experienced and understood varies across cultures and worldviews (Dudgeon et al., 2014; Kickett, 2011). The data in this study reveal that the concepts and practices aligned with identity, representation, connection and relatedness that emerged through the yarning process are integral to the way Indigenous students experience online learning. Karen Martin (2003) for example describes Indigenous subjectivities and worldviews as essential for the survival of Aboriginal peoples. The Aboriginal worldview of relatedness has at its core the interconnection between all things land, the self, and people (Martin, 2008). It is through relatedness that the self is understood. The self, or identity, is experienced and recognized in relationship with others as well as in connection to country through kinship ties (Martin, 2009). Similarly, relationship, connection and belonging are linked to resilience in Indigenous peoples (Kickett, 2011). Exploring Aboriginal ways of knowing the world through relatedness and kinship provide a window for non-Indigenous educators to consider the similarities and differences between Indigenous and western concepts of relatedness. In the western paradigm, relatedness is an integral component in the concepts of networked learning (Goodyear, Jones, Asensio, Hodgson, & Steeples, 2005), connectivism (Siemens, 2004), and in theories such as Social Capital Theory (Coleman, 1988), and Self Determination Theory (SDT). SDT, for example, describes intrinsic motivation as being comprised of three components: autonomy, competence and relatedness (Ryan & Deci, 2000). In this theory, relatedness is considered as an essential psychological characteristic of well-being. In SDT relatedness is a factor in determining motivation, a concept of relevance to engagement and persistence in learning. Social capital theory (SCT) similarly situates relatedness as a central component through which ‘resources’ such as trust and reciprocity are generated through the strength of networks in and across groups (Schaefer-McDaniel, 2004). SCT describes three qualitatively different, yet overlapping types of relationships that lead to social capital formation (Torche & Valenzuela, 2011); these are ‘bonding’, ‘binding’ and ‘linking’ relationships. Within this framework, bonding is the formation of networks within homogenous groups. Bonding links are the strongest form of social capital and usually occurs within family and friendship groups and are marked by high levels of trust and reciprocity, as well as shared norms and values. Binding social capital describes the resources, or benefits, derived from networks developed across heterogeneous groups, while linking social capital refers to networks formed across groups of different status, and between individuals and organisations including governments and their agencies. SCT clearly defines the benefits that derive from relatedness at an individual level and in terms of social cohesion. Importantly, the benefits of establishing networks and relationships apply just as much in learning environments as they do more widely in society. The overlapping concepts of relatedness in Indigenous and western theories provide an opportunity for developing shared understandings of Indigenous and non-Indigenous ways of knowing. Relatedness and e-learning The concept of relatedness in online learning is not new. Indeed, in virtual environments the concept of networked learning focuses on the potential of information and communication technology to support connections and collaboration (McConnell, Hodgson, & Dirckinck-Holmfeld, 2012). The very narrative of the Internet is one of connection and the promise, of linking people regardless of race, creed, colour, gender or social status. In this paper knowledge formation in the online environment is considered in terms of opportunities to develop relatedness through networks, through bonding relationships between people from similar backgrounds and also through binding relationships between heterogeneous people and groups. The technologies that enable online learning make access to HE both possible and attractive to ‘non- traditional’ students. As a consequence, online learning has been an important driver in broadening the base of HE, as recommended in the Review of Australian Higher Education (Bradley et al., 2008). With this broadening base, including increasing numbers of Indigenous students entering HE (Thomas & Heath, 2014), building processes for developing networks is increasingly necessary in the design of virtual spaces where interaction, exploration and negotiated knowledge construction can thrive. The distributed nature of online learning has created endless possibilities for connecting people to FP:228 240 each other in educational environments, yet this possibility is juxtaposed by stories of isolation and a lack of connectedness by many online learners (Bolliger & Inan, 2012). The history of technology- enhanced education has focused on the affordances of technology as a means of content distribution, with content being regarded synonymously with knowledge creation/learning. There has been a much lesser focus on teaching practices that are about connecting people with each other as opposed to connecting people with learning (Watters, 2015) and even less of a focus around the identity of the student in the online space and the capacity for this space to be an enabler for recreating that identity (Seitzinger, 2014) Based on the centrality of relatedness, this paper proposes that a future oriented approach to online learning requires a relational stance, not one that replaces the need for knowledge creation, or distribution, but one that first contextualizes the learner within the online space to increase a sense of relatedness. This stance towards learning is one that takes place both (one could argue ‘first’) in relation to others and to the resources. Social interaction and the development of networks are essential in moving towards a state of relatedness. However, the virtual or social presence of an individual can take various forms when interacting with others, largely depending on their context. Social media research indicates that identity and the way we choose to represent ourselves; what we reveal and what remains concealed, changes depending on context and intent (Kietzmann, Hermkens, McCarthy, & Silvestre, 2011). Cultural norms and protocols also influence how engagement occurs in social interactions, but these protocols are confused in the online space where markers such as age, social standing, race and gender may be obscured (Voiskounsky, 1998). Representation, therefore, is an important part of developing connections and relationships in online learning environments. Figure 1, illustrates that the online profile (identity) of a person working within a study context will differ from their personal, or private profile, which again may be different from their professional profile. This difference is largely mediated by the systems or services they represent themselves in and through. These representations influence the networks and connections that a student builds in online learning environments and on the extent to which they achieve a sense of belonging in those spaces. For example, it is now quite common for an institution’s virtual learning environment (VLE) to have a range of tools that would allow students to create on online identity for themselves; one that they would use to represent themselves to other students and staff, but one they may not want publicly accessible. Typically this would happen in the ‘Profile’ section in the learning management system (LMS), or/and in an ePortfolio. It may also incorporate other more nuanced uses of popular social media tools, and the syndication of the outputs from these tools, but in a more guarded way. Figure 1. Social presence mediated for different online contexts In practical term this means a student may choose to represent themselves in a certain way and identify themselves with certain attributes in a study context, where it may be less necessary, and sometime unadvisable, to represent themselves in the same way within their professional context. FP:229 241 This profile, in part, then defines how a student may relate to others within their educational context, which may well then extend to their cultural context and how they choose to represent themselves within that paradigm. The opportunity for self-representation in online environments has resonance for Indigenous learners. There is much written on the colonizing effects of research and of the misinterpretation and misrepresentation on Indigenous peoples (Martin, 2008; Nakata, 2007). The consequent development of Aboriginal research frameworks and methodologies provide approaches for Indigenous researchers to operate in ways that challenge western research and seeks to redefine the way research is conducted to align with Indigenous world views. Similarly, in online environments there is the potential for Indigenous students to represent themselves as they choose to create online identities that suit their purposes, promote their own agendas and represent their own worldviews. Learning design models and relatedness Teaching and learning models are an attempt to simplify inherently complex environments that contain multiple variables. When this takes place online there is an increase in the number of variables. In such environments the promise of a networked and connected world too often doesn’t eventuate. Theories of learning such as Sociocultural Theory (Vygotsky, 1978), Social Constructivism (Vygotsky, 1978), Social Learning Theory (Bandura, 1971) and Connectivism (Siemens, 2004) embrace the social and connected nature of learning. Yet the potential for relationships is never guaranteed by technological design or in their translation online. However, without designing for relatedness it is unlikely that the potential of technologies to create connected learning can be realised. But to what extent is relatedness designed for within established and emerging learning design models? The design of flexible online environments that support a diversity of learners within a western higher educational system, including Indigenous learners at the “cultural interface” (Nakata, 2007) needs more unpacking. The experiences of the Indigenous students in this study indicate that relatedness should be considered at multiple levels within online learning: at the institutional level; at the course design level; and at the unit design level of online interactions. At the institutional level an environment designed for relatedness would allow Indigenous students to represent their multifaceted contemporary identities through an online identity, or profile. It would allow students to connect with other Indigenous students in a culturally safe space. This model would also take into account the continuing influence and flow between personal, social and learning networks, with the learning environment existing not as an isolated ecosystem but one that sits within a wider reality that is connected to all aspects of their world. At a unit level, designing for a related environment would include meaningful and relevant activities that promote student interactions and the sharing of diverse perspectives. The integration of synchronous and/or asynchronous communication tools would be crucial to enable those interactions. Figure 2 represents components of relatedness from the research as evident in a sample of established design models. It indicates that for these design models, the focus is on unit level design and there is little evidence of design models that incorporate institutional level design principles that support the development of relatedness. Figure 2 also shows that while there is no one model that satisfies all of the components of relatedness, that existing learning design models all include some components of relatedness. That no one existing model includes all the components of relatedness is not surprising. Each of the models was developed for a specific purpose and context different to that of the research. FP:230 242 Figure 2: Evaluation of Components of Relatedness in Learning Design models Relatedness at an institution level None of the models represented in Figure 2 factor in identity representation at either institutional or unit levels. For example, none promote the establishment of online profiles that students can use to represent themselves across all their units. Nor do the models incorporate the establishment of institutional physical, or virtual spaces to provide students with the opportunity to extend learning networks with students outside their units. Or in this case, provide opportunities for Indigenous students to engage with other indigenous students enrolled at the institution. In terms of models that demonstrate a relationship between the outside world and the institutional environment, Goodyear’s Problem Space (Goodyear, 2005; Goodyear & Ellis, 2007) acknowledges the “social and physical/digital contexts for learning, as well as the activity itself, are co-produced by students, teachers and others” (p. 341). Also, the fifth Stage of Salmon’s Five Stage Model model considers linkages between student learning activities with existing networks and their intersection with online environments. While the majority of the models refer to the learners background and attributes as influencing learning, Goodyear’s Problem Space and Salmon’s Five Stage Model explicitly link the social nature of learning with others who may be outside of the formal learning environment, positioning the formal learning environment as part of a wider, linked network. Relatedness in unit level learning design By recognising the background and attributes students bring to their learning environments we acknowledge also the networks they leverage to support their learning. At the unit level some of the models take into consideration the background and diversity of the learner. For example Biggs 3-P model (Biggs, 1989), regarded as a classic model of teaching and learning, recognises the characteristics students bring to the learning environment and the diverse factors that influence the development of their worldview. The term ‘presage’ in this model suggests that student characteristics, combined with the context of the environment provides some for-shadowing of the learning experience. The LEPO and Goodyear models also make the link between student background and their learning. Goodyear and Ellis acknowledge that while teachers are not able to “manufacture community” (Goodyear & Ellis, 2007, p. 341), they have a duty to “help set up the social fabric” (p. 341) to support these connections. Goodyear’s model provides a strong framework for understanding online learning spaces as situated locations of networked learning that draw on students’ backgrounds. The Laurillard and Salmon models, on the other hand, do not reference the student background. All the models have a focus on activity as the process through which learning takes place, and as the means through which interactions occur. The design of the activities includes consideration of the tools through which the activity and interactions can take place. Relatedness in unit level interaction design Activities in a learning environment can be designed as interactions between student and content, student and lecturer, and between students. In terms of moving towards relatedness, interactions between students and between students and lecturers are of most significance. All of the models included student to lecturer interaction as integral components of their design. However, only Salmon’s 5 Stage Model and Goodyear’s problem space of educational design are explicit about the interaction between students. This is not to say that student-to-student interactions are precluded in the other models, however, the LEPO model includes the teacher and the student as the main actors, but does not show student-to-student interactions as inherent features. Indeed, Biggs’ 3-P model also may well include peer-to-peer interactions within the context of learning activities, but this is not an explicit. Gilly Salmon’s Five Stage Model of E-learning, on the other hand, focuses on group interaction and group activity and is based on knowledge construction through interaction in staged learning activities and is essentially about group formation and social capital building in the context of learning. Of the models reviewed, Goodyear’s problem space of educational design is the one that positions the concept of relatedness most highly and additionally situates it as a characteristic of well-designed FP:231 243 online learning environments and as a product of skilled facilitation. Elements of each model reviewed contribute in some way to an understanding of how online learning is constituted, but with respect to relatedness, Goodyear’s model is unambiguous about the centrality of connectedness to learning. Recommendations for trial and limitations It is seen from the discussion above that the issues related to traditional design models for online learning are predominantly related to situating students in a unit of study and looking to engage them at that level, where, in a sense, they have to reestablish their identity each time they go into a new unit. This is problematic, particularly where there may not be any collaboration between those teaching these units, and where there is little or no focus on building an online community that is wider than at the unit level. However, what this paper proposes is to link students into an online network greater than just studying a single, or group of single units. That is, the student may create for himself or herself an identity that transcends the single unit in order to represent, or position themselves within their learning in a more holistic manner, as represented in Figure 3. Creating the opportunity for identity representation at a university level will involve institutional commitment and disposition to providing a place for this to occur. This could be as simple as re- conceptualising or extending the Goodyear model to incorporate concepts by which relatedness is achieved, particularly for Indigenous students, or it could be extended to incorporate other systems within the VLE that align with the LMS, such as an ePortfolio, an internal social networking tool, or allowing for the syndication of information from certain social media sites, as seen in Figure 3. Regardless of how it is conceived the following recommendations for trial stem from this work: At an Institutional level: • Engagement with a suite of technologies to facilitate the development of comprehensive student and staff profiles (identity), along with a openness to receive external social networking feeds. • Train staff in the notion of digital and social networking literacy and on how to facilitate student engagement, based on a centralised profile (identity). • Establish and support specific online spaces for Indigenous students (and other defined groups), within the institutional community site, to facilitate the networking across the institution. Figure 3. Institutional community focused model At a Unit level: • Ensure the design of unit environments can facilitate the use of, and align with, student profiles. • Train and support students early in their engagement with the institution on how to represent themselves in a university based profile (identity). This requires a level of sophistication and may address ways to encode Indigeneity not visible to non-Indigenous students or staff, if this is desired. The recommendations are based in the analysis and findings of a design based research project that was located at Charles Darwin University. The findings are in relation to the experience of Indigenous students studying online at CDU and are not presented as generalisable for other contexts. Indeed the recommendations have not yet been tested and are based on deconstruction and analysis of early findings from the research study discussed in the body of this paper. Despite this, based on the evidence provided we recommend that this model be trialed. FP:232 244 Conclusion The experience of Aboriginal and Torres Strait Islander participants indicates that online learning spaces at the site of the study are not conducive to the sorts of relatedness upon which their lives depend. Additionally, representation of identity within the online environment is an essential prerequisite for establishing connections. However, opportunities to develop an online identity within units are ad hoc and the extent to which students are prepared to reveal their identities depends on a range of factors including the extent to which others in the environment are ‘known’. The yearning for connection in online learning environments contrasts with the sense some students have of the online environment as a formal environment rather than a social one, and where the mechanisms for making connections (such as online discussion forums) are not often designed in ways that draw Indigenous students into the learning environment, or seen to be connected with other parts of their lives. Furthermore, well known learning design models that guide the development of online learning spaces and learning interactions have very little focus on ‘relationship’ and ‘connections’, and where they do exist, it is mainly in the context of teacher-student processes and interactions around learning. However, if we take as our starting point the stories that our students tell of their lives and education, we can discern some emerging design principles that may help us establish better online learning ecologies, designed to support their learning journey. The recommendations for trial presented here highlight the social aspects of learning and the need for an institutional level approach to support holistic learning environments. These recommendations provide a means to integrate concepts and practices aligned with relatedness into HE institutions, to create friendlier and safer online spaces for Indigenous, and indeed for all students, in order to enhance the experience of online learning. References Anderson, T., & Shattuck, J. (2012). Design-Based Research: A Decade of Progress in Educational Research? Educational Researcher, 41(1), 10. Australian Bureau of Statistics. (2015). 3101.0 Australian Demographic Statistics, Mar 2015. Canberra: Australian Bureau of Statistics. Babbie, E. (2007). The Practice of Social Research (11 ed.). Belmont CA: Thomson Wadsworth. Bandura, A. (1971). Social Learning Theory. New York: General Learning Press. Bessarab, D., & Ng'andu, B. (2010). Yarning About Yarning as a Legitimate Method in Indigenous Research. International Journal of Critical Indigenous Studies, 3(1), 37-50. Biggs, J. B. (1989). Approaches to the enhancement of tertiary teaching. Higher Education Research and Development, 8(1), 7-25. Bolliger, D., & Inan, F. (2012). Development and Validation of the Online Student Connectedness Survey (OSCS). The International Review of Research in Open and Distributed Learning, 13(3), 41-65. Bradley, D., Noonan, P., Nugent, H., & Scales, B. (2008). Review of Australian Higher Education Final Report. Canberra: Department of Education, Employment and Workplace Relations. Charles Darwin University. (2015). Charles Darwin University Annual Report 2014. Darwin. Chen, B., Vansteenkiste, M., Beyers, W., Boone, L., Deci, E. L., Van der Kaap-Deeder, J., Verstuyf, J. (2015). Basic psychological need satisfaction, need frustration, and need strength across four cultures. Motivation and Emotion, 39(2). Coleman, J. S. (1988). Social Capital in the Creation of Human Capital. The American Journal of Sociology, 94 (Supplement: Organizations and Institutions: Sociological and Economic Approaches to the Analysis of Social Structure ), S95-S120. Dudgeon, P., Milroy, H., & Walker, R. (Eds.). (2014). Working Together: Aboriginal and Torres Strait Islander Mental Health and Wellbeing Principles and Practice (Second ed.). Barton, ACT: Australian Government Department of the Prime Minister and Cabinet. Ellis, R., & Goodyear, P. (2010). Student experiences of e-learning in higher education: the ecology of sustainable innovation. London: RoutledgeFalmer. Goodyear, P. (2005). Educational design and networked learning : patterns, pattern languages and design practice. Australasian Journal of Educational Technology, 21(1), 82-101. Goodyear, P., & Ellis, R. (2007). Students' interpretations of learning tasks: Implications for educational design. Paper presented at the ascilite conference 2007, ICT: Providing choices for learners and learning, Singapore. FP:233 245 Goodyear, P., Jones, C., Asensio, M., Hodgson, V., & Steeples, C. (2005). Networked learning in higher education: Students' expectations and experiences. Higher Education: the international journal of higher education and educational planning, 50(3), 473-508. Hall, A. (2009). Designing Online Learning Environments for Local Contexts, as exemplified in the Sultanate of Oman. (Doctorate in Education), University of Wollongong, Wollongong. Kickett, M. (2011). Examination of How a Culturally Appropriate Definition of Resilience Affects the Physical and Mental Health of Aboriginal People. (PhD), Curtin University. Kietzmann, J. H., Hermkens, K., McCarthy, I. P., & Silvestre, B. S. (2011). Social media? Get serious! Understanding the functional building blocks of social media. Business Horizons, 54(3), 241-251. doi: http://dx.doi.org/10.1016/j.bushor.2011.01.005 Laurillard, D. (2002). Rethinking university teaching: A conversational framework for the effective use of learning technologies (2nd ed.). London: Routledge. Martin, K. (2003). Ways of Knowing, Ways of Being and Ways of Doing: a theoretical framework and methods for Indigenous re-search and Indigenist research. Journal of Australian Studies, 76(Voicing Dissent, New Talents 21C: Next Generation Australian Studies ), 203-214. Martin, K. (2008). Please knock before you enter: Aboriginal regulation of outsiders and the implications for researchers. Teneriffe, Qld: Post Pressed. Martin, K. (2009). Thinkin' you know...getting it wrong...and us missin' out - Aboriginal worldview and knowledge, English literacy - great expectations or grating obsessions? Dare to Lead: National Indigneous Education Conference Melbourne McConnell, D., Hodgson, V., & Dirckinck-Holmfeld, L. (2012). Networked Learning: A Brief History and New Trends. In L. Dirckinck-Holmfeld, V. Hodgson & D. McConnell (Eds.), Exploring the Theory, Pedagogy and Practice of Networked Learning. New York: Springer-Verlag McLoughlin, C., & Oliver, R. (2000). Designing learning environments for cultural inclusivity: A case study of indigenous online learning at tertiary level. Australian Journal of Educational Technology, 16(1), 58-72. Nakata, M. (2007). Disciplining the Savages: Savaging the Disciplines. Canberra: Aboriginal Studies Press. Phillips, R., McNaught, C., & Kennedy, G. (2012). Evaluating e-learning: guiding research and practice. New York Routledge. QSR International. (2015). NVIVO10 for Windows from http://www.qsrinternational.com/products_nvivo.aspx Reedy, A., Boitshwarelo, B., Barnes, J., & Billany, T. (2015). Swimming with crocs: Professional development in a Northern context. Paper presented at the Australian Teacher Education Association (ATEA) conference, 'Strengthening partnerships in teacher education: Building community, connections and creativity', Darwin Reeves, T. C., McKenney, S., & Herrington, J. (2010). Publishing and perishing: The critical importance of educational design research. Australasian Journal of Educational Technology, 27(1). Rigney, L. I. (2006). Indigenist Research and Aboriginal Australia. In J. E. Kunnie & N. I. Goduka (Eds.), Indigenous Peoples Wisdom and Power. affirming our Knowledge Through Narratives. Hampshire: Ashgate. Russell, L. R., Kinuthia, W., Lokey-Vega, A., Tsang-Kosma, W., & Madathany, R. (2013). Identifying complex cultural interactions in the instructional design process: a case study of a cross-border, cross-sector training for innovation program. Educational Technology Research and Development, 61(4), 707-732. doi: 10.1007/s11423-013-9291-8 Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68-78. Salmon, G. (2003). E-moderating: The key to teaching and learning online (2nd ed.). London, New York: RoutledgeFalmer. Schaefer-McDaniel, Nicole J. (2004). Conceptualizing Social Capital among Young People: Toward a New Theory. Children, Youth and Environments 14(1): 140-150. Seitzinger, J. (2014). Curate Me! Exploring online identity through social curation in networked learning. Paper presented at the 9th International Conference on Networked Learning 2014, University of Edinburgh. Siemens, G. (2004). Connectivism: A Learning Theory for the Digital Age. International Journal of Instructional Technology and Distance Learning, 2(1), 3-10, 2(1), 4-10. Thomas, L., & Heath, J. (2014). Institutional wide implementation of key advice for socially inclusive teaching in higher education. A Practice Report. The International Journal of the First Year in Higher Education, 5(1), 125-133. FP:234 246 Torche, F., & Valenzuela, E. (2011). Trust and reciprocity: A theoretical distinction of the sources of social capital. European Journal of Social Theory, 14(2), 181-198. Universities Australia. (2014). Indigenous Higher Education Retrieved 27 September 2015, 2015, from https://http://www.universitiesaustralia.edu.au/uni-participation-quality/Indigenous-Higher- Education Voiskounsky, A. (1998). Internet: Culture Diversity And Unification. Paper presented at the Cultural Attitudes Towards Communication and Technology ’98, University of Sydney, Australia. Vygotsky, L. S. (1978). Mind in Society: The Development of Higher Psychological Processes. Cambridge, MA: Harvard University Press. Watters, A. (2015). Learning Networks, Not Teaching Machines. Paper presented at the EDEN 2015, Barcelona. http://hackeducation.com/2015/06/10/eden2015/ Woolcock, M., & Narayan, D. (2000). Social capital: Implications for development theory, research, and policy. The World Bank Research Observer, 15(2), 225-249. Reedy, A. & Sankey, M. (2015). Designing for relatedness: learning design at the virtual cultural interface. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:223-FP:235). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:235 247 Open and Interactive Publishing as a Catalyst for Educational Innovations Xiang Ren University of Southern Queensland This paper reviews the educational value and innovative uses of open and interactive publishing (OIP) in learning design. OIP is defined in its broadest sense including all the emerging practices brought about by using open approaches and networked technologies to publish and engage with content. It explores two aspects of educational values and uses: (1) Open publications and scholarship provide new forms of open educational resources that stimulate innovations in learning designs and pedagogies beyond textbooks. (2) OIP is by nature a digital learning space whereby creative learners are able to learn from peers and communities through self- and social publishing activities. It also discusses the impact and challenges of OIP inspired innovations, from which practical recommendations are derived. Keywords: open publishing, interactive publishing, OERs, learning design, learning space A Review of Open and Interactive Publishing Open and interactive publishing (OIP) primarily refers to open access to digital content and open licences that allow users to reuse and remix. It also means that an ‘open’ approach is adopted to the creation, distribution, and consumption of content based on creative end-users, democratic participation, and social networks, in which the boundaries between authors and readers are blurred (Ren, 2013). Moreover, crowdsourcing and users’ collective intelligence play an essential role in filtering, assessing, and remixing content. Overall, OIP ensures ‘that there is little or no barrier to 5 access for anyone who can, or wants to, contribute to a particular development or use its output.’ In the academic contexts, OIP could be an umbrella of many emerging publishing practices: open access scholarly publishing, OERs, self-publishing, academic blogging, scholarly social media, social referencing, open data, self-archiving, and crowdsourced publishing. Overall these open practices are creating new value propositions and driving genuine innovations through an emerging publishing ecosystem based on individual users’ creativity and networked collaboration and transforming the landscape of scholarly publishing. OIP is an essential intermediary and enabling technology for open scholarship. Boyer’s classic model of scholarship (discovery, integration, application, and teaching) is being reconceptualised in the context of “open” (Ren, 2015). Veletsianos (2012) lists three specific forms of open scholarship in practice: (1) open access and open publishing; (2) open education; and (3) networked participation. Other researchers also try to redefine scholarship in the post-Web 2.0 environments, emphasising the increasingly essential role of co-creation, social networking and collaboration, for example, ‘co- creating open scholarship’ (Garnett & Ecclesfield, 2012), ‘networked participatory scholarship’ (Veletsianos & Kimmons, 2012) and ‘social scholarship’ (Greenhow & Gleason, 2014). Burton (2009) and Weller (2009) use the term “open scholar” to refer to the changing role and duties of individual scholars in the emerging open knowledge environment. Likewise, open access advocates argue that, academic maxims are shifting from “publish or perish” to “be visible or vanish”. OIP could be a catalyst for genuine innovations in teaching and learning. The full value OIP can yield is more than opening up the ‘access’ of content; rather, it opens up the whole process of knowledge creation and communication. It has significant potential to drive open educational innovations by new 5 The definition is based on the one developed by JISC CETIS, Wilbert Kraan, CETIS Assistant Director, http://jisc.cetis.ac.uk/topic/open FP:236 248 types of content and new models of knowledge production. Open education community needs to broaden visions beyond ‘access’ (or free content) and reinvent practices by harnessing the dynamics of OIP, which echoes the transformation from open educational resources to open educational practices (Ehlers, 2011). This paper aims to systematically review and synthesise the role of OIP as a catalyst for innovations in open education, particularly inspired by the paradigm shift of publishing and scholarship. In the following sections, it focuses on two major aspects: (1) Open scholarship and open publications enabled by OIP provide new forms of open educational resources and stimulate new pedagogies and learning designs beyond traditional textbook teaching. (2) OIP is by nature a digital learning space whereby creative learners can learn from peers and networks through self- and social publishing activities. The dynamics, innovations, examples, challenges, and recommendations will be discussed. The paper ends by a critical reconsideration of the interplay between open Internet and institutional constraints in higher education, which shapes the adoption of OIP as well as other open praxis. Open Publications and the Move beyond Textbook Teaching Blyth (2009) criticizes commercial textbook publishing for inhibiting innovations and failing to create learner-centric and user-friendly (both learners and educators are users here) experiences and address their real needs. Likewise, Saravanan (2013) critiques the limitations of textbook teaching and pedagogies. They are just part of the increasingly strong voice of moving beyond traditional textbook teaching (Loewen, 2013). Open textbook plays a significant role in widening access and reducing students’ cost. However, most open textbook projects have not transformed textbooks- based learning and teaching despite of licencing digital materials openly. The huge scale of new types of scholarship created by OIP has not been fully harnessed, including open access research publications, open data, user-generated-content, and so forth. There are significant opportunities to remix and repurpose open publications and open scholarship into new forms of textbooks that enable and inspire innovative pedagogies and learning designs. As such, the open education community might need to shift their priority from “big OERs” (Weller 2010) created by institutional projects with explicit educational purposes to broad open content in the Internet and explore its educational value innovatively. Open Access Publications and Open Data in STEM Education Open access has become a mandate in major public-funded research systems and most leading universities in the world. As a result, 27 million academic publications have been made openly accessible online (Khabsa & Giles, 2014). This open tAd is influencing research data management as well, making the original lab data openly accessible to the public, in contrast to the traditional academic publishing system that only publishes the final results of research and often only positive results. More than that, driven by the open ethos of science, a growing number of scientists and researchers use blog, slide sharing, preprints, and social media to communicate research and engage the public. Just as Quirós (2009:63) argues, open and interactive initiatives are reinventing academic publishing into ‘a dialogue between scientists [and the public] without mediation or obstacles’. All these are making science more transparent and inclusive than ever. Open research scholarship provides opportunities for educational reuse and repurposing as well. Traditional forms of textbooks are only secondary knowledge rewritten by educators, as a result of which learners access restricted and possibly biased representation of knowledge. Open publications have widened public access to the original representation of knowledge by its creators (as publications) as well as the process of creating and developing knowledge. Technically the process of the social and academic construction of scientific knowledge is accessible to learners who can thus understand how knowledge is originated, developed, revisited, and debated. This is fundamental difference brought about by the OIP inspired new ‘textbooks’. The constructive first step would be harnessing open research publications to reform textbooks. Compared with textbook content, research publications critically represent the latest knowledge developments and written by researchers themselves, which also include a critical review of existing literature and insightful recommendations on future research directions. This will inspire new scientific pedagogies not only in tertiary education, but also possibly at lower levels of STEM education. A FP:237 249 further next step is to harness the dynamics of open science. Mediated by OIP, almost every stage of the research life-circle ranging from proposal, research design, data collection, data analysis, draft, preprints, peer review, and post-publication debates is publicly accessible. This has greatly enriched the knowledge resources that could be reused and remixed for educational purposes, moving far beyond traditional textbooks. A growing number of individual educators have begun to embed open scholarship and open data in STEM teaching, which would otherwise be costly to obtain through commercial sources or doing experiments by themselves. There are also institutional initiatives as well. For example, Connected Curriculum, developed by University College London, is “an institution-wide initiative which aims to ensure that all UCL students are able to learn through participating in research and enquiry at all 6 levels of their programme of study” . It further calls for closing “the divide between teaching and research” through integrating “research into every stage of an undergraduate degree, moving from 7 research-led to research-based teaching” School of Data is another example, focusing on 8 empowering people “with the skills they need to use data effectively” , which is defining a new literacy in the open data age. Open Content Resources in Arts, Social Sciences, and Humanities We are living in a culture and media environment of abundance instead of scarcity (Ren, 2014). This is fundamentally changing our views and ways of using content and knowledge, not only for entertainment and everyday life, but also for teaching and learning. Weller (2010) categorises OERs into “big OERs” created by institutional projects with explicit educational purposes and “little OERs” created by individuals “from a variety of motivations, but can have an educational intention ascribed to them by someone else”. Weller (2010) further points out that, the Web 2.0 enabled little OER “represents a more dynamic model that encourages participation, and may be more sustainable. For learners, a mixture of both [big and little OERs] may also create a varied, engaging experience.” With the rise of user-generated-content (UGC) and born digital publications, like in STEM areas, there is much more materials educators and learners can use than just open ‘educational’ resources (or big OERs) in social sciences and humanities as well. The born digital content and UGC have direct benefits for courses like foreign languages where learners can easily access real language environments through social media. Another direct implication is self-published literary content for the courses like publishing, editing, and creative writing, which provides much more diverse sources of literature with different styles and levels, also at different working stages. This is sharply contrast to the traditional publishing system the mainstream educators depend on, which only publishes the final edited versions of editor-selected literature. Like science, the development of journalistic and creative content is being more transparent and inclusive than ever. Innovative educators are aware of the educational value of born digital content and user-generated-content, i.e. little OERs, and the dynamics of an increasingly open landscape for media, arts, and humanities. The OpenLIVES project at University of Leeds aims to “digitise and publish materials documenting the experiences of Spanish migrants to the UK and returning migrants 9 to Spain, repurposing this data as open educational resources” . It also involves students in the creation and evaluation of these OERs; students in a final-year course were asked to conduct own research using open data and assessed innovatively. It is reported that students valued original research and creative control over their education (Martínez-Arboleda 2013). This example demonstrate the value of open data for learning and teaching in humanities and social science disciplines. Similarly, Beijing Normal University has led a project of online training system for editors 6 http://www.ucl.ac.uk/teaching-learning/strategic_priorities/connected-curriculum 7 http://www.researchresearch.com/index.php?option=com_news&template=rr_2col&v iew=article&articleId=1343435 8 http://schoolofdata.org/ 9 http://www.webarchive.org.uk/wayback/archive/20140614060648/http://www.jisc.ac. uk/whatwedo/programmes/digitisation/content2011_2013/openlives.aspx FP:238 250 based on real journalism content10. The project aims to provide a system whereby users can train and test their sensitivity to valuable news sources and learn editorial selection criteria based on a large- scale database of news reports and readers’ preferences. As such, students are working as editors and gatekeepers in the virtual system, doing multiple choice questions and selecting what they believe the readers are most likely to read. The students’ choice will then be compared with the real world data. This content-rich system is imitating the future working of journalism students by including real-world data so that students can apply the theories into editorial practices. Though there are long way to go to translate open resources into innovations of pedagogies, these initiatives have shown inspiring and convincing examples. Interactive Publishing and Open Learning Space An EU open publishing initiative uses the term “liquid publications” (Cuel, Ponte, & Rossi, 2009) to define the new approach to publishing scientific knowledge: (a) content is updatable and knowledge is continuously evolving; and (b) knowledge is built in a constructivist way based on collective intelligence and social collaboration. In OIP, Internet users are empowered to actively co-create, share, edit, remix, and assess digital content, either individually or collectively. This makes OIP potentially a digital/open learning space, enabling interest-driven, social, and interactive learning. Literat (2012) frames the different levels of artistic participation (receptive, executory, and structural) in online crowdsourced art platforms and suggests that participants can play very different roles ranging from passive audiences of finished artistic product, engaged participant in redesigned projects, to co-designers and co-authors. This framework applies to a wide variety of OIP areas where learners can participate in knowledge developments at different levels and as different roles. Significant opportunities exist in using the OIP platforms as an interactive online learning space, which exist beyond the institutional Learning Management Systems. There are mainly four aspects of innovative learning designs: publications as assignments student publishing publishing selected essays open learning through OIP peer rating (marking) peer supports peer editing and coments crowd-oriented co-develoment learning communities group chat and community knowlege sharing public readership makes a difference leaner-public interaction engagement with 'real world' Figure 1: Major aspects and learning activities in the digital learning space enabled by OIP Student Publishing It is increasingly popular that educators take the advantages of self-publishing to publish the educational content they create. How about students? A large number of courses have writing assignments but students’ works are normally read only by examiners. This lags behind the development of Internet and open publishing. It is now viable and reasonable for students’ writing to have wider readership and educators should encourage it. The leading scientific journal Nature once published a research paper written by a group of pupils based on an experiment they conducted, which implies potentially significant value of student-made innovations. Students’ works like assignments, are part of overall knowledge commons of human beings, which should be accessible to everyone in the digital age. More importantly, publishing students’ works is beneficial for learning itself. Just as Jim Moulton argues, ‘Publishing was important. It gave me the opportunity to take the 10 http://nsts001.com/index.php FP:239 251 moral high ground and ask the [learners] to do their very best because their writing was headed for publication. We all know that real audiences make a difference’.11 Practically there are two major ways to use student publishing to improve learning. One is publications as assignments, in which teachers or educators give clear instruction that students need to publish their assignments online. Educators might give learners more freedom in choosing topics and encourage them to find the topics they are most passionate about. Sometimes educators might even adopt social assessments provided by OIP as part of the overall marking. This will stimulate learners to set a higher level of goals when doing assignments and they will learn how to write for engaging readers rather than pleasing markers. Another way is to publish selected essays and assignments written by students after formal assessments. Educators can encourage students to adapt their assignments into publications if needed. OIP is an enabling technology for such innovations in higher education. There are a wide variety of self-publishing platforms like Lulu and Amazon’s create space whereby students can publish their essays, creative fictions, and other feature articles generated from learning. Student publishing as an open learning activity is not a privilege of subjects like creative writing; there are also opportunities and OIP platforms for students in STEM and other disciplines to publish their works. Undergraduates and postgraduates can publish their original research with various student-run academic journals like Student Pulse which is “an open-access academic journal that highlights the work of students at the undergraduate level and above.” The open access publishing platforms including both online journals and online preprints also welcome high quality submission from university students and some even set a special section for student essays because they regard students’ work as valuable emerging voice in the academia. Peer support OIP encourages and depends on peer editing to improve the quality of content. For leaners, peer editing provides a good opportunity to learn how to write and improve their writing skills. In addition to the direct contribution to content improvement, learners could also benefit from comments and feedback provided by peers. In contrast to peer support within formal online learning environments, learners in open learning space benefit from a wide range of expertise beyond textbooks and classrooms, perhaps including experts and senior level peers in their fields. The feedback and comments might be more insightful and helpful. It is believed that “online writing communities offer students who are gifted a chance to explore and create a supportive peer group.”(Olthouse & Miller, 2012). Such benefits and dynamics apply to other subjects as long as learners are able to find their peers in the OIP platforms. For example, physics students might enjoy high level peer supports if they publish their work with initiatives like arXiv; chemical students might benefit from engagement with their disciplinary blog-sphere ChemBark. Learning Communities A defining feature of OIP is crowd-oriented knowledge development and mass collaboration, illustrated by platforms like Wikipedia. Focusing on educational values and uses, there are many possibilities for OIP to be used as a learning space in this regard. A large number of Wikipedia contributors are students in Higher Education institutions and their creative work in crowdsourcing knowledge is valuable learning experience as well, which should even be recognized by formal assessments and credentialing in some ways. Another important example of mass knowledge development is citizen science, in which students could make substantial contributions associated with their learning process and in a collaborative environment. It is believe that science today is not only for the public, but also from the public. As the participation of wiki-models within institutional eLearning systems is comparatively low, open platforms outside educational institutions might provide better social learning experience, encouraging students to contribute to the mass collaboration of knowledge advancements in broad real world associated with own interests and passion. The value of OIP communities also lies in the consumption of content. Social reference management tools like Mendeley and Zotero are equally valuable for collaborative learning (Estelles, Del Moral, & González, 2010). By looking at other peers’ libraries and the references they stored, learners can 11 http://www.edutopia.org/self-publishing-student-writing FP:240 252 efficiently access the key literature and references in a discipline or a course and their own contribution matters to others as well. The folksonomy built by learners’ collaborative selections might provide different synthesis of knowledge than textbooks and the process of selecting references collaboratively benefits learners in various aspects as well. A step forward in educational innovations is needed in order to harness the affordances of OIP platforms as a supportive and collaborative learning community. It is not just about learning knowledge, it is more about cultivating collaborative skills. The P2PU (peer to peer university) provides a good example of the power of open community in transforming learning and even disrupt the traditional teacher-student paradigm. Learner-public interaction It is believed that science today should be built upon citizen inquiries (Williams, 2010). Likewise, arts and humanities “are now connected to contemporary ideas about citizenship, caring and public engagement.” (Delacruz, 2009). Education should go beyond the academic ivory tower and shifting the priorities from delivering abstract knowledge (fact) to encouraging civic participation. OIP provides valuable enabling technologies and platforms. Through activities like self-publishing and collaborative knowledge developments, students and learners could have their voices heard widely as knowledge creators, commenters, and collaborators in the public sphere of science, literature, arts, and so forth; they can create knowledge, publish content, and interact with the public and the real world. This is not only novel learning experience, but also, an essential part of educating capable citizens in the 21st century. Discussion While OIP is instrumental to education it represents open culture and values as well. OIP is built upon the belief that knowledge is commons and knowledge production is collective, participative and inclusive. Educational innovations is driven and inspired by the open transformation of publication from one-way information flow like traditional textbooks to networked flow based on collaborative models. This echoes the shift of learning theories and paradigms towards connectivism (Siemens, 2005). All these suggests great potential of the educational uses of OIP for reforming learning design and pedagogies. As discussed above, the primary impact of OIP upon education lies in the potential of moving beyond ‘textbook-fact’ model in teaching and learning. Given half of scientific knowledge is proved to be incorrect within 45 years (Arbesman, 2012), it questions the pedagogies based on transferring “fact” to students. By widening learners’ access to research publications, open scholarship, and knowledge production and communication, the adoption of OIP is a constructive first step to reform the traditional paradigms. This provides significant opportunities for further educational innovations through combining the OIP ‘tools’ with various paradigms, cultures, and values. OIP-inspired pedagogies focus more on literacies. There is a steady growth in the emphasis on teaching about the nature of science in STEM education. It is argued that, students need to be taught about the methods of scientific investigation and the role and status of scientific knowledge in the societies at large (Wong & Hodson, 2009). OIP enables learners to participate in real scientific communication and even the whole research life circle by either accessing open scholarship or interacting with research teams. This is valuable in nurturing literacy. Similarly new literacies could be cultivated through participating in creative works or knowledge production in social sciences and humanities. Digital literacy is another essential literacy for students today. As a substantial part of open Internet, there is no doubt that OIP helps with cultivating students’ digital literacy, not only the skills of seeking, reusing, and remixing content, but also the literacy as a connected creative citizen, expressing themselves and engaging audiences creatively. The uses of open content and interactive publishing space in education will foster students’ critical and creative thinking. Rather than just transferring and discussing authoritarian ‘fact’ in textbooks, OIP as a learning space with evolving knowledge and democratic environments provides learners with opportunities to participate in knowledge development as well as directly question and challenge the FP:241 253 authorities. This helps to increase learners’ "21st century skills", in particular, thinking critically, analytically, and creatively (Silva, 2009; Tytler, 2007). OIP enables inquiry-based, interest-driven, and personalised learning outside the walled garden of digital learning based on institutional Learning Management Systems. Learning in open publishing activities is not a process designed by educators and instructors in advance, but an ongoing learner- driven and self-directed process based on learners’ own interests and passion as well as inquiries of knowledge. Moreover, OIP broadens the scale and scope of knowledge access by learners and thus increase the possibilities of more diverse and personalised learning. It will be especially beneficial for the talent students to expand their vision and learning beyond the restriction of textbooks and classrooms. Personalisation comes from both the abundant diverse content provided by open publications and the networked and collaborative dynamics of interactive publishing models. On the other hand, learning is increasingly social and collaborative. Open environment enables collaboration with strangers and much more diverse Internet users globally, which is not available in closed institutional eLearning system. Undeniably the use of OIP is challenging traditional learning and teaching. As some educators argue in the context of creative writing education, ‘The changes created an ideological struggle as new writing practices were adapted from broader societal fields to meet the instructional and regulative discourses of a conventional writing curriculum’ (Mills & Exley, 2014). This applies to broader educational contexts. Generally, it is less challenging to embed OERs into traditional pedagogies and curriculums than broadening the scope of OERs and further facilitating students-led creation and collaboration in OIP platforms. Like any emerging practices, there are obvious technical difficulties to be sort out. For example, the reliability of OIP platforms in terms of the access to content, the archiving and security of usage data, and so forth; the interoperability between OIP platforms outside campus and the institutional Learning Management Systems. Other concerns exist in students’ privacy, ethical issues in student research and other academic activities. New methods for assessment and credentialing are also urgently needed as their current absence creates obvious barriers against to OIP adoption. More than that, the barriers from educators’ mindset, institutional policies, and educational culture are crucial. The perception of educational values and transformational potential of OIP remains limited and biased. Educational innovations associated with OIP require tremulous input of time, creativity, expertise, and workload, which is, however, luxurious resources in current institutional contexts of higher education. The educators generally lack initiative and passion of leading pedagogical innovations. Further, every academic is fighting against busy schedule and competing demands on time and resources. Last but not least, the overall educational culture is built upon formal (traditional) credentialing and accreditation which is structurally incompatible with the informal learning inspired and enabled by OIP as well as open Internet. Despite of the challenges, there are still opportunities for moving forward practically in reforming learning designs and pedagogies through adopting OIP. OIP as initiatives outside the traditional education domain has developed very rapidly, with thousands of mature and large-scale platforms. A growing number of educators within tertiary education system have already taken advantages of various OIP models and resources in educational practices. Deriving from the above discussion on both the dynamics and challenges, the following recommendations are proposed for effectively exploring the value of OIP as a catalyst for educational innovations. • Taming “wild” OERs: Through widening learners’ access to the process of scientific research, OIP is of value in developing new pedagogies that focus on students’ critical thinking and scientific literacy. The term ‘free range’ is sometimes used to describe the openly licenced OERs that could be freely remixed and reused. Defining OERs beyond being “educational”, open publications and content resources provided by OIP are even more ‘free range’. Rather than ‘little OERs’, open and interactive publications are ‘wild’ OERs. Once ‘tamed’ by careful indexing, purposeful learning design, instrumental instruction, open publications could be valuable alternatives to the traditional textbooks and OERs. • Moving beyond institutional LMS: Educational technologists believe that the Web 2.0 inspired platforms could be facilitators and enablers of social and interactive learning and have invested heavily in building such social connections within closed institutional LMS FP:242 254 systems. These initiatives are valuable and functional as they are closely related to educators, learners, and formal learning resources. However, learners’ participation social activities hosted by formal LMS is usually low and during short term only. OIP provides a large amount of digital, open, and informal learning space outside the institutional online learning systems. Using these third party public platforms not only saves money for educational institutions, but also might lead to more interactive and engaging learning, enabling students to interact with the real world. • Redefining “open” textbooks: Open textbooks should not be just openly licenced traditional textbooks. Instead, the deluge of open information and resources are driving reinvention of ‘textbook’. It is not appropriate any more to ‘feed’ learners with ‘manufactured’ learning materials given abundant open and original materials in the OIP systems. Of course there is much to do in tailoring open publications for education, including indexing, filtering, assessing, remixing, and repurposing content. But redefining open textbooks beyond packaging OERs into traditional formats is a realistic and constructive first step in linking OIP with educational innovations. • Open learning design: There is considerable potential to reform pedagogies through open learning design, integrating learning activities with OIP and possibly outside institutional LMS and the controlled traditional domain of education. In the highly self-directed and self- organized knowledge open environment, the roles of educators, institutional supervision, assessment, and credentialing need to be redefined. The challenge lies in the formalisation of OIP-inspired or –enabled learning activities and embedding them into curriculums. It also demands new methods for assessment and credentialing in order to evaluate and recognize open learning activities, for which open badges, micro-credentials and learning analytics might be practically helpful. Open learning design might be easier in the subjects that directly benefit from open publications and OIP, for example, practice-led courses like design and visual arts, lab-based courses like biology, medical sciences. In these subjects, open resources provide valuable references, examples, lab data, which would otherwise cost a fortune to produce by educators. • Collaborating with OIP platforms: Educational technologists and learning designers might need to improve the awareness and capability of collaborating with OIP platforms. Many OIP platforms are built upon open culture and have APIs for educational developers; they also welcome collaboration that could expand their uses for learners and learning purposes. The collaboration, particularly in technological aspects, is necessary to provide a user-friendly, reliable, and efficient interface for educators to conduct innovations in teaching and pedagogies. • Using OIP as a bridge to the real world: It is important for students to learn how to survive in the real world with their knowledge and skills and thus urgent for our education going outside the ‘campus’ (either physical or mindful). OIP is an enabling technology for cultivating ‘free range’ students in an open knowledge environment. Moving beyond textbooks and closed institutional learning environment will also improve students’ employability in future, which is increasingly a priority in Higher Education policy today. Conclusion The fast growth and evolution of digital publishing is somewhat neglected by educational technologists, at least not being considered as a systematic dynamic. It is thus necessary to systematically examine and discuss OIP as a catalyst for open education innovations and differentiate it from other similar or relevant dynamics. It is worth mentioning that OIP itself is no longer an experimental beta, but a mature paradigm with a large number of established platforms and billions of active users. In other words, OIP provides more ready-to-use platforms than other emerging ideas or FP:243 255 eLearning initiatives. In order to explore the potential for education, learning designs need to integrate OIP with pedagogies and course developments innovatively and develop practical instruction and guidelines for educators and learners to engage with various emerging publishing practices. It is equally important for institutional policies changes in assessments and credentialing to recognize open learning activities and creative achievements associated with OIP. Thus, this paper is calling for a deep understanding of the transformative potential and evolutionary value of OIP beyond simple applications like electronic or open textbooks. It calls for initiatives based on the OIP platforms and practices to function as a catalyst for educational innovations. In his book ‘The Battle for Open’, Martin Weller (2014) points out that though open has achieved triumphs in education, there is still much to do. As discussed above, the full educational value OIP can yield is being restricted due to a narrow lens of ‘open’ focusing on free access to content and the reduction of textbook cost. The limited understanding and adoption results from a paradox about OERs: open educational resources are developed and used in a closed institutional system of education. The dynamics and constraints of OIP are just a snapshot of the broad tension between open Internet and closed educational institutions. This highlights the significance and necessity of shifting priorities from open educational resources to open educational innovations and the transformation of pedagogies, mindsets, and policies accordingly. Reference Arbesman, S. (2012). The half-life of facts: Why everything we know has an expiration date: Penguin. Blyth, C. (2009). From textbook to online materials: the changing ecology of foreign language publishing in the era of ICT. In M. Evans (Ed.), Foreign Language Learning with Digital Technology (pp. 174-202). London and New York: Continuum. Burton, G. (2009). The open scholar. Retrieved on 26.11.14 from http://www.academicevolution.com/2009/08/the-open-scholar.html Cuel, R., Ponte, D., & Rossi, A. (2009). Towards an Open/Web 2.0 Scientific Publishing Industry? Preliminary Findings and Open Issues. Liquid Publications Project. Retrieved from http://wiki.liquidpub.org/mediawiki/upload/b/b3/CuelPonteRossi09.pdf Delacruz, E. (2009). Old World Teaching Meets the New Digital Cultural Creatives. International Journal of Art & Design Education, 28(3), 261-268. doi: 10.1111/j.1476-8070.2009.01620.x Ehlers, U.-D. (2011). Extending the territory: From open educational resources to open educational practices [online]. Journal of Open, Flexible and Distance Learning, 15(2), 1-10. Estelles, E., Del Moral, E., & González, F. (2010). Social Bookmarking Tools as Facilitators of Learning and Research Collaborative Processes: The Diigo Case. Interdisciplinary Journal of E- Learning and Learning Objects, 6(1), 175-191. Garnett, F., & Ecclesfield, N. (2012). Towards a framework for co-creating Open Scholarship. 2012, 19. doi: 10.3402/rlt.v19i3.7795 Greenhow, C., & Gleason, B. (2014). Social scholarship: Reconsidering scholarly practices in the age of social media. British Journal of Educational Technology, 45(3), 392-402. doi: 10.1111/bjet.12150 Khabsa, M., & Giles, C. L. (2014). The Number of Scholarly Documents on the Public Web. PLoS ONE, 9(5), e93949. doi: 10.1371/journal.pone.0093949 Literat, I. (2012). The Work of Art in the Age of Mediated Participation: Crowdsourced Art and Collective Creativity (Vol. 6). Loewen, J. W. (2013). Teaching what really happened: How to avoid the tyranny of textbooks and get students excited about doing history: Teachers College Press. Martínez-Arboleda, A. (2013). Liberation in OpenLIVES Critical Pedagogy: “empowerability” and critical action. Caracteres. Estudios culturales y críticos de la esfera digital, 2 (1). Retrieved from http://revistacaracteres.net/revista/vol2n1mayo2013/liberation-in-openlives-critical-pedagogy- empowerability-and-critical-action/ Mills, K. A., & Exley, B. (2014). Time, Space, and Text in the Elementary School Digital Writing Classroom. Written Communication, 31(4), 434-469. doi: 10.1177/0741088314542757 Olthouse, J. M., & Miller, M. T. (2012). Teaching talented writers with Web 2.0 tools. Teaching Exceptional Children, 45(2), 6-14. Quirós, J. L. G., & Gherab, K. (2009). Arguments for an open model of e-science. In B. Cope & A. Phillips (Eds.), The future of the academic journal. Oxford: Chandos. FP:244 256 Ren, X. (2013). Beyond open access: open publishing and the future of digital scholarship. Paper presented at the Proceedings of the 30th Australasian Society for Computers in Learning in Tertiary Education Conference (ASCILITE 2013). Ren, X. (2014). Creative users, social networking, and new models of publishing. Cultural Science, 7(1), 58-67. Ren, X. (2015). Quandary between Communication and Certification: Individual Academics’ Views on Open Access and Open Scholarship. Online Information Review, 39 (5), 682 – 697. Saravanan, V., & Al Najjar, M. (2013). Educational Initiatives in Teacher Education Reform: Curriculum Intervention and Curriculum Innovation. Journal of Teaching and Teacher Education, 1(2), 103-115. Siemens, G. (2005). Connectivism: A learning theory for the digital age. Retrieved from http://er.dut.ac.za/handle/123456789/69 Silva, E. (2009). Measuring skills for 21st-century learning. Phi Delta Kappan, 630-634. Tytler, R. (2007). Re-imagining science education: Engaging students in science for Australia's future. Veletsianos, G., & Kimmons, R. (2012). Networked Participatory Scholarship: Emergent techno- cultural pressures toward open and digital scholarship in online networks. Computers & Education, 58(2), 766-774. doi: http://dx.doi.org/10.1016/j.compedu.2011.10.001 Weller, M. (2009). Thoughts on digital scholarship. Weller, M. (2010). Big and little OER. Paper presented at the Open Ed 2010 Proceedings, Barcelona: UOC, OU, BYU. Weller, M. (2014). Battle for Open:How openness won and why it doesn’t feel like victory. London: Ubiquity Press Limited. Williams, S. N. (2010). A twenty-first century Citizens’ POLIS: introducing a democratic experiment in electronic citizen participation in science and technology decision-making. Public Understanding of Science, 19(5), 528-544. Wong, S. L., & Hodson, D. (2009). More from the Horse’s Mouth: What scientists say about science as a social practice. International Journal of Science Education, 32(11), 1431-1463. doi: 10.1080/09500690903104465 Ren, X. (2015). Open and Interactive Publishing as a Catalyst for Educational Innovations. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:236-FP:245). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:245 257 Learning Design for digital environments: agile, team based and student driven Spiros Soulis Angela Nicolettou Office of Dean, Learning and Teaching Office of Dean, Learning and Teaching RMIT University, Australia RMIT University, Australia Digital learning environments are a catalyst for change and development in Higher Education. One way to respond to this is by going to the foundation of the environment – the learning design process. Using an Australian university’s major project in learning design as an example, this paper will look at how students need to be active members of Curriculum Design Teams to ensure that responsive, relevant and engaging digital learning ecosystems are created. Strategies based in design thinking, socio technical systems, learners as designers, and agile methodologies for project management, will be shown to be central to the effectiveness of the project. Challenges emerging from the projects’ implementation are identified as key directions to be addressed in the evolution of the process. Keywords: learning design, agile, digital learning, design thinking, elearning Digital learning environments The digital age brings with it many challenges and opportunities. For higher education this means change, development, uncertainty, and innovation and in many instances a rethink of how we engage in the core business of education (Beetham & Sharpe, 2013). Such a rethink involves a closer look at the pedagogy and the digital learning ecosystems (Reyna, 2011) that these new environments create. One such change can be seen in the terminology used, and the evolution of the term elearning to digital learning. To support this, Mason & Pillay (2015) argue that the development of digital tools has also enabled the possibility for learners to engage in enquiry that is critical and more in keeping with the demands of a learner for a global society. As the digital learning environment changes for learners, what does it mean for teaching and the development of teachers in this space? The 2015 New Media Consortium Report for Australian Tertiary Education discuss what some of the implications of the digital learning ecosystem are for teachers and suggest: Resetting expectations for the roles of professors and other faculty is also chief among the concerns of the 2015 Australian panel. Integrating more personalised learning opportunities and student-led approaches challenge traditional perceptions of teachers. The goal is for professors and instructors to act as coaches and mentors, rather than lecturers. (Johnson, Adams Becker & Hall, 2015 NMC Technology Outlook for Australian Tertiary education, 2015, p. 3) Such suggestions require a shift in thinking about the relationship between learner and teacher and more importantly the roles that each has in the education process. This paper will explore these issues from the perspective of the learning design process and how design thinking can be used to address some of these challenges while placing students at the center of this process by engaging with learners as designers. It will identify how engaging in design thinking can promote learning, using the example of an Australian University major project Global Learning by Design (Nicolettou & Soulis, 2014) and demonstrating how educators need support in viewing themselves as a facilitator of learning (Kolodner et al., 2003). The idea of lifelong learning and equipping students to meet the demands associated with a challenging 21st century, requires that students are equipped with “meta competences”. Education needs to transition from that of transferring knowledge to fostering individual skills in creative thinking FP:246 258 within a constructivist framework (Scheer, Noweski & Meinel, 2012). Such 21st century skills include design thinking that develop students’ ability to solve problems, allowing for opportunities to experiment. An example of a collaborative digital learning community is evident in Kolodner et al., (2003) work. They investigated notions of cognitive apprenticeship that sees the teacher taking on the role of coach or facilitator in the learning process where learners engaged in meaningful design challenges, creating physical artifacts and sharing insights into their designs. This resulted in the development of communities of learners that foster collaboration (Fisher & Herrmann, 2011) and allow students and teachers an opportunity to work together. Design thinking Constructivist theory identifies learning as being accomplished through experience, with the teacher as a facilitator of learning being able to drive a learning design experience for students. Engagement of learners within this process is a critical element of constructivist learning. As Scheer et al (2012, p. 9) state: Design Thinking realizes what is recommended theoretically in constructivist theory. Especially learning through experience and complex problem solving among other aspects are met in Design Thinking. Siemens (2014) argues that traditional learning theories such as constructivism are limited when applied to digital learning environments. He uses the term connectivism, which encompasses elements of constructivism such as learning from experience, but takes it further to incorporate elements such as social connectedness, managing changing technologies, currency of knowledge and decision-making, to name a few. This view supports the idea that for learning and teaching to be effective digital learning ecosystems, the way universities approach learning design needs to adapt. Owing to the complexity of modern problems, design is not characterized as standard problem solving where the problem and solution are seen as separate, the approach is very much a non-linear one (Cassim, 2013). Design thinking is well suited to educational approaches particularly in digital learning, where solutions are non-linear, as it is thinking that works on “creative hunches” based on incomplete information and abstract forms of thinking (Burdick & Wills, 2011). Further to this, Razzouk & Shute, (2012, p. 14) state: We believe that design thinking is more than just a skill to be acquired and used in limited contexts. Rather, we view it as a way of thinking and being that can potentially enhance the epistemological and ontological nature of schooling. Taking design thinking to a larger scale, the Hasso Plattner Institute at Stanford University usually referred to as the //d.school// has effectively incorporated it as a ‘foundational component’ of its approach to undergraduate programs. In terms of delivery, d.school classes are team-taught with instructors and students coming from a range of disciplines and backgrounds (Miller, 2015). Larry Leifer, professor of mechanical engineering and director of the University Center for Design Research, in an interview for the Chronicle of Higher Education, stated: …the d.school is a kind of anti-university. Universities and their academic disciplines, he says, provide ‘context-independent knowledge’. The world and its problems are not, however, organized by discipline. (Miller, 2015) Education however is built around disciplines and isolated subjects, which ultimately result in breaking down the complexities that are found in real life (Scheer et al., 2012). The d.school is certainly an interesting model; showcasing the potential for design thinking within a university context, that is attempting to tackle the complexity of modern problems. Teacher as learner What then is the impact of such changes for the teacher? Here, we see changes to the role of the FP:247 259 teacher from ‘knowledge expert’ who structures curriculum and learning activities to ‘facilitator or coach’ in an at times unknown learning path. Kolodner et al. (2003) address the practical issues in the Learning by Design approach in identifying that teachers were not totally comfortable with making inquiry happen in the classroom. They also talk about supporting teachers to learn facilitation skills, as a way of introducing a ‘collaborative culture to the classroom’. Further to this, Kapur & Bielaczyc (2011) in their study Designing for Productive Failure outline the need to provide teachers with professional development training on facilitation skills and strategies. In their study the role of the teacher (in the productive failure control group) was not to provide any direct instruction or content related support, but manage the classroom and provide an environment for problem solving. We worked with the teachers to not provide assistance when asked for but rather to constantly assure students that it was okay not to be able to solve the complex problems as long as they tried various ways of solving them, especially highlighting to them the fact that there were multiple representations and solution methods for the problems. (Kapur & Bielaczyc, 2011, p. 52). Their study concluded that compared to direct instruction, the student cohort engaged in productive failure seemed to engage in greater conceptual thinking without compromising performance on well- structured problems. Further, students’ solution methods better correlated with the learning outcomes. What Kapur & Bielaczyc (2011) inadvertently identified was that facilitation skills are critical; teachers need to move away from the role of teachers to that of coaches and facilitators, allowing for more inquiry and problem based learning (Kolodner et al., 2003). Teachers as facilitators of learning, require current skills and a toolkit to actually practice on the key competencies of learning (Scheer et al., 2012). Within an academic environment the role of the teacher as the sage on the stage needs to be challenged. It is not as Kolodner et al., (2003, p. 541) indicate ‘for teachers to be better teachers’ but for teachers to rethink their role as designers of learning, incorporating design thinking into their curriculum and teaching. Learners as designers How do you engage learners, and make them a part of the design thinking approach? Owen (2007) highlights a number of design characteristics, such as being centered on a concern for people and the environment, the ability to visualize, use of language as a tool, the importance of teamwork and the important trait of avoiding the necessity of choice; all critical skills for a 21st century learner. One way to engage learners in the design is by incorporating cycles of redesign or even under design (Fisher, 2011) into the process. Trying to find solutions by exploring, then coming together with peers to present their artifact and receive feedback, which ultimately leads to self-reflection, and then further iteration on the design. Can the skills be learnt? According to Razzouk & Shute (2012) with sufficient practice in meaningful environments as well as adequate feedback and scaffolding, students can learn design thinking. Approaches that involve problem-based learning and inquiry-based learning can ultimately all enhance the students’ design thinking skills (Dym et al., 2005). Encouraging students to think like designers will enable them to better prepare for complex problems not only within their careers but life in general (Razzouk & Shute, 2012). As Fischer (2011) highlights, students are viewed as consumers rather than inheritors of problems, if we don’t engage them in activities that are problem-based and inquiry-driven, how will they develop such skills? How will they problem-solve? Perhaps through design thinking students can bridge across to a connectivist learning framework. Within that framework, we need to embrace elements of productive failure (Kapur, 2008), as students need to be encouraged to engage in activities that foster collaboration. Educational institutions often treat learners as consumers. As a result, learners feel disconnected from the decisions made on their behalf by teachers, and are denied from actively contributing to what will ultimately affect them and their learning. Higher education very much models this approach from how it delivers its curriculum, to how students are supported within various services such as the FP:248 260 library, learning centers and counseling. Students are effectively passive participants. With the advent of social computing a shift has occurred away from a culture of consumers to that of a culture of participation. We have moved away from a world where a small number of individuals define the rules and laws to one where most people are able to actively participate. Within socio- technical environments cultures of participation not only encourage and support users’ participation but also judge it as critical (Fischer, 2011). Socio technical systems (STS) are now everywhere, a part of our personal and professional lives, with some of these overlapping between both domains. Organic in nature successful socio-technical systems rely on the affordances offered by meta-design and cultures of participation. It explores the user as the critical element in the design in order to have systems that are functional and sustainable. Fischer & Herrmann (2014) discuss how due STS’s organic nature co-design is critical not only for their inception but also how they will be ultimately used. STSs can best be described as taking on two different stages in their development the design time and use time. In the design time, system developers anticipate possible needs of users (who may or may not be involved) and create systems on their imagined needs. At use time, users will use the system, however because developers could only perceive what their needs or contexts could be at design time the system often falls short of meeting the user's requirements which then means modifications need to be made. This leads to the critical point within the Fischer and Herrmann (2014) paper that the need to ‘empower users as designers is not a luxury but a necessity’. Due to this complexity, STSs require what Fischer and Giaccardi (2006) have described as ‘meta design’ or ‘designing design’. This framework is emerging as an opportunity to view socio-technical environments as ‘living entities’. It is built on the premise that systems need users at design time to act as co-designers at use time. It requires a sense of pliability and not a fixed premise during the design stage. What we have here is a rationale for greater student involvement in learning design. ‘Global Learning by Design’: one university’s approach to learning design An example of meta-design and user involvement is our work at RMIT University on a major project - Global Learning by Design (GLbD) - which illustrates the importance of users as active contributors from the outset. The work centres on elements of Agile design methodologies as ‘going beyond’ the meta-design and fostering cultures of participation within Curriculum Design Teams (CDT). As its foundation the project establishes CDTs which include academic and support staff that work together using Agile methodology to create learning objects that are captured as learning design patterns for reuse by other discipline areas (Nicolettou & Soulis, 2014). To interpret the concepts identified by Fisher & Herrmann (2014) the example used, will be our own experience managing this project. The vision of GLbD is to provide students with choice in relation to their learning material and use of educational technologies that are innovative and practical. The idea of creating CDTs was premised on the context that all stakeholders must be involved in the learning design from the outset, in order to foster a culture of participation. If users or user representatives, in this case teaching staff and support staff, where bought into the process at a later stage (as has occurred previously in curriculum design) they would feel “misused” and would not foster a sense of ownership with the project (Fischer, 2011). The approach with CDTs in 2014, as part of the GLbD project, fostered a more meta-design approach. However what was missing, was the users themselves - the students. In 2015 CDTs have included students from the outset. The inclusion of students within the CDTs has not only changed the dynamics of the group but has provided valuable input into the design of what in most cases will be a socio-technical system. Students are now informing the design and commenting on modifications, in some instances students are being employed on a casual basis to work on projects or are having their efforts acknowledged as a part of their assessment. The process is also allowing students to work in an authentic workplace context focusing and refining their professional skills. Agile approach Through the GLbD project we have incorporated elements of agile thinking to the learning design FP:249 261 process. We have attempted to foster an approach that is nimble and agile; being able to respond quickly to changes and user requirements. The idea of agile development was born in 2001 from a group of methodologists coming together to pinpoint some broad principles of developing software, culminating in the Agile Manifesto (Chookittikul, Kourik and Maher, 2011). The manifesto recognized that the main elements of agile principles should be adaptive, iterative, straightforward and promoting communication. We identified and incorporated some of those key principles into our work with CDTs that include (Nicolettou & Soulis, 2014): • face to face meetings • identifying motivated individuals • building trust • technical excellence • good design In attempting to foster these principles, CDTs were only part of the answer. What was required was an agile approach to getting the work done, and this is where the software package Trello (www.trello.com) has become affective. Working with Trello has allowed transparency and a collaborative approach to being able to do good design. It has dramatically reduced the amount of emails, making it the venue for communication and completion of tasks. The project coordinator acts as moderator and reminds staff of pressing items that need to be completed. A spreadsheet can be easily exported to identify at what stage tasks are at: To Do, In Progress and Completed. It has allowed for projects to be designed and delivered within a very short time frame. The affordances of the software has allowed for us to draw in our colleagues from offshore campuses. In 2014 Global Learning by Design delivered 12 projects, as of July 2015 we are on schedule to deliver 60 projects by the end of the year. The only variable that has changed from 2014 is the team has employed one extra Educational Developer. In an environment where institutions are rapidly attempting to embrace technologies that are innovative and sustainable these outcomes have been welcomed. GLbD has now been able to build trust and ownership amongst the staff as a good model for learning design. The next evolutionary phase of GLbD is to be able to seed projects to allow for evolutionary growth and reflection allowing users (students) to bring back their evolved system to the curriculum. The other critical element within this approach is the need to continue to have students as active participants and not as Fischer (2011) terms consumers. It is fundamental that through GLbD we are able to foster a culture that allows students an opportunity to design their own learning, and move from that of consumer to an ‘owner of the problem’ (Fischer & Herrmann, 2014). Challenges There are a number of tensions that can be drawn from meta-design and cultures of participation. Meta-design just by its nature creates tension, for example between standardization and improvisation (Fischer & Herrmann, 2014). In at least one GLbD project we have witnessed an approach of too much improvisation as staff wanted to continually keep adding functionality to the socio-technical system, in this case an online e-studio platform. In order to find the right balance, a solution was to end at iteration i08 and send that back for user testing. The developer also welcomed this approach after working solidly on the project for 3 months. Here the challenge presented is that being able to foresee uses at use time cannot be completely anticipated at design time, hence the need to stop and test. Participation overload is a potential drawback within meta-design; participants within these cultures of participation may be forced to contribute to personally irrelevant activities (Fischer & Herrmann, 2014). Within GLbD we incorporate a number of support services (library, study and learning center & employment services) in the CDTs. However during the early scoping stages of the project, it may become apparent these services are not required, and if this is the case it needs to be quickly addressed and resolved. Quality and reliability are challenges highlighted by Fischer (2011) that will require further research, as a greater volume of people are involved and can contribute. Questions such as: how are we able to assess for quality and reliability of systems? As systems are being built and implemented what FP:250 262 testing occurs beyond that? What is the life-cycle of the re-seeding process? How many iterations can a system have? All questions that need to be raised if we are to evolve this learning design process. Measurement will be a major contributor and indicator in future decisions of designing socio-technical systems. A pressure on GLbD this year is how are we measuring the results? How are we improving the student experience? In most instances it can be as simple as measuring how many times students visit a site, its usability, and of course the student surveys where questions need to be linked to specific elements of what has been designed, but is this sufficient? We believe not. Evaluation, not just measurement will be a key focus of GLbD 2016. Capturing the responsiveness, engagement, collaboration and sharing of practice is challenging; it is here where cultures of participation need to ‘go beyond’. What environments like Google+ are now able to do is support these cultures through a virtual community where stakeholders across all projects in GLbD are able to come together and share artifacts such as images, videos, blog posts, papers, patterns and upcoming events. This identifies elements that Fischer (2011) describes as mutual benefit, selflessness in sharing, and empathy in realising that peers are experiencing similar challenges and concerns. This is where: …the rise in social computing has facilitated a shift from consumer cultures to cultures of participation (in which all people are provided the means to participate and to contribute actively in personally meaningful problems). (Fischer, 2011 p.42) It is in such communities that expert knowledge is blurred as participants become experts and experts become participants. Once projects are delivered and implemented it allows for participants to continue to connect and reconnect. Motivation remains high as participants may discover new ways of working or producing learning resources. Community sites also allow for feedback, goal setting and specifically relevant information, all of which are important in motivating people to change their behavior. Conclusion STSs cannot be designed to envisage all future demands and that users being involved as designers is critical. This case study illustrated how a major project on learning design can and should incorporate major elements of meta-design as a framework as well as use agile methodology to facilitate trust, collaboration and good design. Students as end users are critical if we want to move away from a culture of consumerism to one of ownership and participation. Meta-design is about changing and challenging human behavior, motivating and not leaving the decisions in the hands of the ‘experts’ (Fisher & Herrmann, 2014). In using this framework GLbD has had a major impact across the university and is now seen as the model for good learning design, as one academic commented, ‘it changed my life’. References Beetham, H. & Sharpe, R. (2013). Rethinking pedagogy for a digital age: Designing for 21st century learning. Routledge. Burdick, A., & Willis, H. (2011). Digital learning, digital scholarship and design thinking. Design Studies. 32(6), 546–556. Cassim, F. (2013). Hands on, hearts on, minds on: design thinking within an education context. International Journal of Art & Design Education. 32(2), 190-202. Chookittikul, W., Kourik, J. L. & Maher, P. E. (2011). Reducing the Gap between Academia and Industry: The Case for Agile Methods in Thailand. 2011 Eighth International Conference on Information Technology: New Generations, 239–244. doi:10.1109/ITNG.2011.49 Dym, C. L., Agogino, A. M., Eris, O., Frey, D. D. & Leifer, L. J. (2005). Engineering design thinking, teaching, and learning. Journal of Engineering Education. 94(1), 103-120. Fischer, G. (2011). Understanding, fostering, and supporting cultures of participation. Interactions. 18(3), 42-53. Fischer, G. & Giaccardi, E. (2006). Meta-design: A framework for the future of end-user development. In End user development 427-457. Netherlands: Springer. FP:251 263 Fischer, G. & Herrmann, T. (2011). Socio-technical systems: a meta-design perspective. In Abdelnour-Nocera, J. (Ed.) Knowledge and Technological Development Effects on Organizational and Social Structures. Hershey PA: IGI Global. Fischer, G. & Herrmann, T. (2014) Meta-Design: Transforming and Enriching the Design and Use of Socio-Technical Systems. In Randall, D., Schmidt, K. & Wulf, V. (Eds.) Designing Socially Embedded Technologies: A European Challenge. http://l3d.cs.colorado.edu/~gerhard/papers/2014/EUSSET.pdf Johnson, L., Adams Becker, S. & Hall, C. (2015). 2015 NMC Technology Outlook for Australian Tertiary Education: A Horizon Project Regional Report. Austin, Texas: The New Media Consortium. http://cdn.nmc.org/media/2015-nmc-technology-outlook-australian-tertiary- education.pdf Kapur, M. & Bielaczyc, K. (2012). Designing for productive failure. Journal of the Learning Sciences. 21(1), 45-83. Kapur, M. (2008). Productive failure. Cognition and Instruction. 26(3), 379-424. Kolodner, J. L., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J. & Ryan, M. (2003). Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting learning by design (tm) into practice. The journal of the learning sciences. 12(4), 495-547. Kolodner, J. L., Crismond, D., Gray, J., Holbrook, J. & Puntambekar, S. (1998). Learning by design from theory to practice. In Proceedings of the International Conference of the Learning Sciences Vol. 98, 16-22. Mason, J., & Pillay, H. (2015). Opening Digital Learning for Deeper Inquiry. International Handbook of E-Learning Volume 2: Implementation and Case Studies, 1. Miller, P. N. (2015). Is 'design thinking' the new liberal arts? The Chronicle of Higher Education, Retrieved from http://search.proquest.com/docview/1668025893?accountid=13552 Nicolettou, A., & Soulis, S. (2014). Creating Agile Curriculum Design Patterns: the Global Learning by Design Project. In B. Hegarty, J. McDonald, & S.-K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014, 726-729. Owen, C. (2007). Design thinking: Notes on its nature and use. Design Research Quarterly. 2(1), 16- 27. Razzouk, R. & Shute, V. (2012). What Is Design Thinking and Why Is It Important? Review of Educational Research. 82, 330–348 Reyna, J. (2011). Digital teaching and learning ecosystem (DTLE): a theoretical approach for online learning environments. Changing Demands, Changing Directions. Proceedings ascilite Hobart 2011, 1083-1088. Scheer, A., Noweski, C. & Meinel, C. (2012). Transforming constructivist learning into action: Design thinking in education. Design and Technology Education: an International Journal. 17(3). Siemens, G. (2014). Connectivism: A learning theory for the digital age. Retrieved from http://er.dut.ac.za/handle/123456789/69 Soulis, S. & Nicolettou, A. (2015). Learning Design for digital environments: agile, team based and student driven. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:246-FP:252). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:252 264 Interdisciplinary opportunities and challenges in creating m-learning apps: two case studies Erica Southgate Shamus P. Smith Liz Stephens School of Education School of Electrical Engineering and School of Education The University of Newcastle Computer Science The University of Newcastle The University of Newcastle Dan Hickmott Ross Billie School of Electrical Engineering and School of Electrical Engineering and Computer Science Computer Science The University of Newcastle The University of Newcastle Mobile digital devices such as smart phones and tablets support mobile learning (m- learning) and this is reinventing pedagogical and curriculum approaches in education. The unprecedented growth in digital technologies, and the educational apps they support, provides a unique opportunity to increase engagement in learning anywhere and at any time. However, the development of m-learning apps requires collaboration between learning and content experts and technology specialists. Such interdisciplinary collaboration presents both opportunities and challenges. This paper describes two case studies related to m-learning app development with the aim of highlighting the range of educational and technical issues that arose in the collaborative process, and the solutions devised by the interdisciplinary team. Keywords: m-learning, app development, interdisciplinary teams, literacy, academic literacy, higher education, digital learning. Introduction Mobile learning (m-learning) is upon us (Murphy, Farley, Lane, Hafeez-Baig & Carter, 2014; Nicholas, Fletcher & Davis, 2012; Paulins, Balina & Arhipova, 2015). The considerable uptake of mobile devices such as smart phones and tablets has provided students with the opportunity to grasp learning in the palm of their hands. Mobile devices mean that students can facilitate their own learning anywhere and at any time (Gikas & Grant, 2013). The most recent NMC Horizon Report (Johnson, Adams Becker, Estrada & Freeman, 2015) indicates that a proliferation of open educational resources is likely to occur within a mid-term time frame. This includes the development, widespread dissemination, and uptake of free or inexpensive educational apps. The development of such apps will require timely, intensive and creative collaboration between education experts and technology specialists. The educational fruits of such interdisciplinary collaboration will be immense, yet relatively little has been formally documented regarding the productive processes and potential pitfalls of such collaboration (Druin, Stewart, Proft, Bederson & Hollan, 1997; Herrington, Herrington & Mentei, 2009; Shankar, McAfee, Harris & Behara, 2013). The purpose of this paper is to detail two case studies related to m-learning app development with the aim of highlighting both the range of educational and technical issues that arose in the collaborative process, and the solutions devised by the interdisciplinary team. One case study involves the ‘repackaging’ of an existing set of educational videos, targeting undergraduate students, into an app (Uni Tune In app), while the other describes the development of a serious game to improve literacy (Apostrophe Power app). The paper suggests that more scholarly attention needs to be paid to understanding the interdisciplinary experience of educational app development so that teams can harness the most appropriate expertise and skills to improve both the process and products of m- learning collaboration. Some characteristics of interdisciplinary learning FP:253 265 Interdisciplinary learning refers to the bringing together of knowledge and skills from more than one discipline so that these influence each other’s perspectives (Ivanitskaya, Clark, Montgomery, & Primeau, 2002). In contrast to the additive nature of knowledge in multidisciplinary learning, interdisciplinary learning is integrative (Spelt, Harm, Tobi, Luning & Mulde, 2009). The integrative dynamic of interdisciplinary learning requires connections to be made between technical and basic knowledge, concepts, theory, methods of inquiry and, on occasion, paradigms (Ivanitskaya et al., 2002). Interdisciplinarity often involves ‘solving problems and answering questions that cannot be satisfactorily addressed using single methods or approaches’ (Klein, 1990, p.196). Hence, interdisciplinary collaboration involves approaching complex problems by bridging epistemological positions and the cultural attributes of specific disciplines (Woods, 2007). Combined, these aspects of interdisciplinarity can generate significant challenges for research teams. One of these challenges involves communication and the building of common ground (Repko, 2008). Oberg (2009) suggests that, ‘(j)oint construction of common ground can be an especially taxing form of interaction’ for interdisciplinary teams (p.158). Furthermore, effective learning within an interdisciplinary environment is often associated with attributes such as curiosity, respect, openness, patience, diligence and self-regulation (Spelt et al., 2009). Opportunities for individual and group reflection, over extended periods of time, are also key to identifying successes and acting upon opportunities in interdisciplinary teams (Woods, 2007). Interestingly, despite its often interdisciplinary nature, there is limited understanding of how these aspects of interdisciplinarity ‘play out’ in collaborations between educators and software engineers particularly in developing m-learning tools, and in agile design (Matthews, Lomas, Armoutis & Maropoulos, 2006). This paper explores such dynamics through two case studies of m-learning app development. Context for the case studies and the interdisciplinary team The setting for the interdisciplinary collaboration is the University of Newcastle (UON), Australia. UON is a relatively young institution (50 years old) with a strong history of engagement with its local community in regional Australia. This engagement has led to the development of an ethos of equity at UON, particularly with regard to providing access to higher education for ‘non-traditional’ students or groups of people that are underrepresented in Australian universities. Non-traditional students include those from lower socioeconomic and first-in-family backgrounds, Indigenous people, those with a disability, and mature age students (Schuetze & Slowey, 2002). The impetus for the development of the m-learning apps discussed in this paper came from an identified need to assist in the academic preparation of undergraduate students and, in particular, students from non-traditional backgrounds. Specifically, research conducted at UON indicated that many undergraduate students were underprepared for the transition into university study and that their academic literacy needed to be improved to ensure academic success (Southgate, 2012; Southgate, Douglas, Scevak, MacQueen, Rubin & Lindell, 2014). Academic literacy refers to the ability of students to use the English language to make and communicate meaning through speech and writing in academic contexts (Department of Education, Employment and Workplace Relations, 2009). Its core elements are: grammar; sentence structure; comprehension; academic writing; oral communication style; and analytical and critical thinking (Rolls & Wignell, 2009). Research indicates that there is a clear association between academic literacy skill level and success in tertiary studies (Kirkness, 2006; Rolls & Wignell, 2009). The rapid uptake of mobile devices by undergraduate students provided a new opportunity to deliver targeted educational resources to assist students to independently develop study and academic literacy skills. The team that developed the Uni Tune In app comprised an education specialist (Southgate) and a computer scientist (Smith). The team that developed the serious game, Apostrophe Power, included an education specialist (Southgate), an educational designer (Stephens) and computer scientists (Smith, Billie and Hickmott). Case study 1: Uni Tune In app Background and educational issue the app addressed In 2012-13, Southgate led a team of 25 UON academics and university student support staff on an FP:254 266 interdisciplinary project that aimed to produce resources to improve the transition experience of undergraduate students from non-traditional groups (MacQueen, Southgate, Scevak, Clement, 2012). Principles of transition pedagogy (Kift, Nelson & Clarke, 2010) underpinned the production of text and video resources for students and academic staff. One set of 17 short videos, called Tune in to Uni, focused on developing study skills and academic literacy. The intention was for these videos to be integrated into first year courses through the university’s online learning platform. Examples of the videos produced include: active listening; reading like a university student; understanding the assessment task; how to fix ‘run-on’ sentences; and writing in paragraphs. Videos were deliberately short (2-4 minutes), in plain English, and provided worked examples on the topic. To facilitate learning, content in each video topic was ‘chunked’ (Woolfolk & Margetts, 2010) with information broken down into small components that linked to form a larger principle or skill. Opportunities Bringing together academics from various disciplines (education, psychology, linguistics, social work and business) with student support staff (Indigenous engagement, counselling and student learning development) created a ‘hot bed’ for creative ideas. The group decided to tap into an observed ‘YouTube generation’ effect by producing brief learning videos. The specific inclusion of literacy experts, educational psychologists and pedagogical specialists in higher education allowed for the translation of complex ideas into fun and accessible academic literacy and study skills videos. The Tune in to Uni videos communicated study skills and academic literacy information that was unlikely to date. This made the videos ideal for ‘repackaging’ into an app format. Students could download the free app containing embedded videos onto their devices and use them as an academic ‘starter’ guide, anywhere and at any time, without the need for internet access to stream the videos. For the sake of brevity, the app was called Uni Tune In (see Figure 1), and was made available free of charge through the iTunes App Store (March, 2015) and Google Play store (May, 2015). Figure 1: Uni Tune In app screenshot The iOS version of the app was produced first because it made use of an existing app template and the software expertise of the app developer (Smith). However, as soon as the iOS version was released through the iTunes App Store, requests were received from academics and learning advisors at local and international institutions for an Android version. Thus, the development of the Android version was driven by demand. Issues The primary problem with repackaging the videos into an app was shrinking the video content, in MP4 format, to a suitable size. The 17 Tune in to Uni videos were an average of 17.3 megabytes each and the total size of the videos was 294.2 megabytes. The videos were resampled for an iPad screen using Handbrake, an open source video transcoder (see www.handbrake.fr). This reduced the average video size to 4.1 megabytes and the videos’ total size to 69.8 megabytes. The final iOS app was 94.3 megabytes, including the iPhone and iPad user interface components for multiple screen FP:255 267 resolutions. As the target download environment was via wi-fi connections, this final size was deemed acceptable. However, when the Android version was being developed, it was found that the maximum size on Google Play for a standalone app was 50 megabytes. Thus the videos for the Android version were further reduced in resolution until the final app size was 40 megabytes. Each of these changes required additional specialist rework beyond the app and content development. One key decision was whether to develop the app for a single platform (e.g. iOS only) or to build apps for multiple platforms (e.g. iOS, Android or Windows). Building for a single platform can simplify development and testing, and allows easy access to native device capabilities (Paulins et al., 2015), e.g. specific user interface elements, built in cameras, GPS sensors and accelerometers. However, this comes at the cost of limiting potential distribution avenues and accessibility to users with the supported platform only. Creating for multiple deployment platforms has resource implications, as multiple apps need to be built and maintained, with overheads in the technologies required e.g. Apple hardware and the Xcode independent development environment (IDE) for iOS and a Java IDE for Android, and developers with an extended skillset. An alternative is to use a more general development environment, such as HTML5, or an IDE that supports wrapping apps for multi-platform deployment, e.g. Xamarin (www.xamarin.com). However, wrapped apps may: (i) limit access to native device features (Paulins et al., 2015); (ii) add complexity, e.g. the use of third-party technologies, to the app development process; and (iii) require additional technology skills from the developers. For the Uni Tune In app, the choice of a single platform was driven by the desire to quickly generate a prototype, and by the nature of the app development team, in this case a single developer (Smith) with significant iOS app development expertise. The move to an Android version, as noted above, was demand-oriented after the iOS version was deployed. A further issue when developing apps with a small team is the required skillset for content development. In addition to the learning resources, development of the app itself is required, i.e. the underlying coding, and the app user interface such as app graphics, sound elements and interface components. For the Uni Tune In app, the learning resources came from the existing videos, and the app coding from the project’s software engineer. However, in order to provide a professional look and feel for the user interface, an app template was purchased (from www.appdesignvault.com). This significantly reduced the app development time by removing the need to generate user interface graphics. The key decision here was to use a general template with its associated time and cost savings instead of employing a graphic designer (or similar) to develop customized interface content. However, customized interface content would be necessary should an app’s look and feel be required to meet specialized criteria. Lessons learned The interdisciplinary work in creating the content for the Uni Tune In videos was complete by the time the idea for app development occurred (see Figure 2). Although the videos appear to be simplified explanations of study skills or aspects of academic literacy, the process of creating the content was intellectually difficult because it brought together disciplinary perspectives and specialist knowledge. It was also time-consuming, taking twelve months to complete. In contrast, the app development was relatively quick, although there were technical issues to work through. It is worth considering repackaging existing educational resources into apps if they have a reasonable ‘shelf life’ (like academic literacy knowledge and study skills). The mobile-learning format of apps provides educators with an opportunity ‘to enhance their educational toolkit’ (Arnab et al., 2014), expand the uptake of educational resources, and allow students real time access to academic literacy knowledge and skills at the point of need. Uni Tune In is available for free download for iOS devices from the iTunes App Store (https://itunes.apple.com/us/app/uni-tune-in/id971888771) and for Android devices from the Google Play store (https://play.google.com/store/apps/details?id=reptiliaware.com.unitunein) FP:256 268 J F M A M J J A S O N D 2014 Academic literacy content and video development Throwaway prototype app with dummy videos Uni Tune In series videos provided to developer 2015 First full version of Uni Tune In app Uni Tune In (iOS) in iTunes App Store Android Uni Tune In development started Uni Tune In (Android) in Google Play Store Figure 2: Uni Tune In app development timeline (2014-2015) Case study 2: Apostrophe Power app Background and educational issue the app addressed A large proportion of the Australian adult population has poor literacy. The Australian Bureau of Statistics (2006) reports that approximately seven million Australians have literacy below the minimum level needed to fully function in life and work. Poor grammatical literacy has been documented in some of the Australian university student population (Hendricks, Andrew & Fowler, 2014; Scouller, Bonanno, Smith & Krass, 2008; Southgate, 2012). Without adequate literacy, undergraduate students are unlikely to succeed academically or want to continue with their studies. The Apostrophe Power app is a serious game (Connolly, Boyle, MacArthur, Hainey, & Boyle, 2012) designed to assist students to improve their use of the different functions of apostrophes, including ownership, contractions of words, and irregular uses of apostrophes (or ‘misfits’ as we have termed these). In Apostrophe Power, the learner must drag the apostrophe into the correct position in a sentence under a time constraint – this being before the mouse avatar drops into the water as the island it is standing on slowly sinks (see Figure 3). The goal is to place the apostrophe correctly in ten sentences so that the mouse leaps from island to island until it reaches the cheese at the end of the level. There are three levels of difficulty for each apostrophe function and a combination level that combines the uses of apostrophes to test the learner’s skill. Opportunities The advent of the serious games movement has created an opportunity for educators and instructional and software designers to collaborate in the creation of learning games that incorporate the characteristics of leisure games such as fun, flexibility, competition (including self-competition) and goal mastery (Charsky, 2010). People of all ages now play app based games and the Apostrophe Power collaboration capitalized on this trend to develop a fun way to learn about the function of a component of language (apostrophes) to improve the literacy of students. An app based serious game is particularly relevant to the area of FP:257 269 literacy improvement as students can learn in a flexible and fun way, in private. This was important because it was reasoned that serious games played in private could help alleviate feelings of shame or embarrassment felt by students who exhibit poor literacy (Nicholas et al., 2012). This makes app based serious games on sensitive topics an ideal tool for promoting learning and equity in schools and universities. The collaboration in developing Apostrophe Power was a creative dialogue that melded the following: (i) equity issues in higher education and the need to produce a free literacy resource that would be attractive to a wide range of students, including non-traditional students; (ii) instructional design for literacy acquisition; and (iii) the incorporation of game characteristics such as challenge, level of difficulty, rewards, enjoyment and usability. Figure 3: Apostrophe Power app screenshot Issues One of the major challenges in creating the Apostrophe Power app was the time it took to develop the scope and sequence of the exercises in relation to aspects of gamification. For example, there were decisions to be made about grouping or separating the functions of apostrophes into game categories such as contractions, ownership (single and plural possession), and one common example of misuse (its and it’s) that we categorized as misfits. A fourth category containing exercises which combined the various functions of apostrophes was also developed. Within each category we designed three levels of difficulty (easy, medium, hard) and developed a bank of 20 exercises for each level, for the learner to cycle through as they attempted to achieve 10 correct answers. The exercise development was a lengthy process of rewriting and reworking to take into consideration a number of factors. For example, for the ownership category, especially plural possessives, the exercises were crafted to ensure that context was provided, otherwise there could have been more than one correct answer, e.g. “the boys lunches” could mean either one boy who had lots of lunches, or more than one boy, each of whom had one or more lunch. Ensuring context that provided clarity within a 100 character limit (including spaces) was challenging. Repetition of key phrases and concepts needed to be minimized or eliminated. This was an issue not only within each category but across the game as a whole, so that users would not gain the impression that the exercises were boring and repetitive, which could have led to learner disengagement. We were also careful to eliminate or avoid mentioning certain jobs or fields of study, popular culture or Australian cultural references and colloquialisms, and to present exercises in plain English. This ensured that the concepts could be understood by students for whom English is an additional language, and as part of a more common frame of reference. The combination category, in particular, took the most time to develop due to its complicated exercises. Gamifying these exercises involved an almost constant process of dialogue and iteration between education specialists and computer scientists, with considerations of cognitive load and the exercise length, complexity and structure paramount. Much consideration was given to the issue of cognitive load or short term memory and its influence on learning (Woolfolk & Margetts, 2010). Consideration of cognitive load was important in judging the optimal time required to undertake the exercises to build positive excitement rather than negative anxiety in play. Getting the timing right for each level was FP:258 270 also vital. Achieving the correct balance of excitement and time for processing information and undertaking the exercises ensured that users would experience ongoing improvement in their skills, which is a key factor in positive engagement (Whitton, 2011). Similarly, the issue of the length of the training module and placement of hints in the game, both key ‘scaffolds’ (Woolfolk & Margetts, 2010) to assist students towards mastery, took considerable time to resolve and involved experimentation and iterative refinement. Sometimes it was decided not to gamify an aspect of apostrophe usage because it required a different approach in the game. For example, it was recognised that one of the main problems that students had with contractions was not necessarily which words formed a contraction, but where the apostrophe should be positioned to indicate the contraction. So, positioning the apostrophe correctly became the focus of these exercises. In addition, the on-screen functionality required to either form a contraction, or expand an existing contraction, as well as place or remove the apostrophe, was significantly different to the ‘drag and drop’ functionality used in every other game category. Rather than risk frustrating or disengaging users with such a large shift in functionality, it was decided to continue to feature the drag/drop function and to only ask users to correctly place the apostrophe within an existing contraction. One interesting point of tension within the team was the issue of ‘gold-plating’. The education specialists were concerned with both producing content and developing an engaging, aesthetically pleasing user interface. The computer scientists were more reluctant to talk about the latter, preferring to explore it towards the end of the project. This tension was apparent throughout the project as the education specialists expressed their continual desire to imagine the look and feel of the game from the perspective of the learner. Another team challenge was the relative lack of a shared expert knowledge base and specialist language to talk through and resolve issues. Each team member needed to acquire some of the specialist language of the others, and this was more of a tacit rather than an intentional practice. Translating ideas and concepts and their implications between disciplinary fields was important and part of an ongoing experience in building ‘common ground’. A simple but illustrative example of the difficulties in building common ground came with the use of the term ‘place-holder’. This term was used by the computer scientists to refer to parts of the game that were earmarked for development but was not understood by the education specialist. In fact, it was misunderstood as a lack of progress in developing key elements of the game. The inclusion of an educational designer in the team did assist with some of this translational communication but it was (and continues to be) a steep learning curve for all involved. Scholarly investigations into the dynamics of interdisciplinary collaboration in education apps and serious game design are required as a matter of urgency so that pitfalls can be avoided. A number of technical challenges also existed including the app development approach and the development of app content. Similar to the Uni Tune In app, a native app approach was taken in this project. It was felt that a wrapped approach might unduly constrain the project while scoping design issues with initial prototypes. For example, touch-based interaction and the logging of app analytics (Smith, Blackmore & Nesbitt, 2015) were considered desirable features and a native approach would more readily facilitate these features. The project team selected the Android platform for initial development work as this platform supports extremely easy distribution of app prototypes. In comparison to iOS app builds that can be difficult to share directly, the early versions of the Android app could be uploaded to a shared online folder (e.g. www.dropbox.com) and team members could then install and test versions of the app directly on their own devices. This allowed for a very fast review cycle of working prototypes and helped reduce interdisciplinary communication barriers between the software engineers and the education specialists. After the Android version of the app was completed, a specialist developer was then employed to develop an equivalent iOS version. To aid the development of the app itself, specialist developers with Java experience were employed by the project to support the initial Android app. Given the graphical nature of computer games, which is shared with many serious games, there was a need to also generate or obtain suitable graphical components for the app. For the Apostrophe Power app, a combination of in-house graphics and affordable online graphics (from www.gameartguppy.com) were used. The project’s software FP:259 271 engineers developed the in-house graphics. This had the advantage of a fast review cycle for new graphical elements and local customization of graphics for the app, but the disadvantage of diverting resources from app coding and testing. Thus the purchase of some online graphics was a compromise to balance project resources. The Apostrophe Power app will be available for free download for iOS devices from the iTunes App Store and for Android devices from the Google Play Store in December 2015. Lessons learned Gamifying learning for literacy, even for the seemingly straightforward functions of an apostrophe, proved to be thought-provoking and time consuming (see Figure 4). It involved sometimes daily communication about content development between the education specialists, and between education specialists and computer scientists. Melding the learning elements with the gaming elements was challenging, with experimentation and multiple iterations required. Balancing the learning with the gamification led, in one instance, to the decision not to include an important function of the apostrophe in the game (forming contractions). In some cases there were misunderstandings concerning discipline specific terminology and there was a constant tension between the desire to understand from a pedagogical point of view what the learner would see and feel and the issue of gold plating as an end stage process. J F M A M J J A S O N D 2014 Develop apostrophe content/exercises Develop Android app v1 Refine app v2 and apostrophe content 2015 Refine app v2 and apostrophe content (cont.) Develop evaluation (v3) and release (v4) versions Android apps complete iOS app developed Apostrophe Power user study (using v3) Android and iOS versions on respective app stores Figure 4: Apostrophe Power app development timeline (2014-2015) Some general observations The collective experience of the team in developing two educational apps has highlighted a range of issues that both enable and constrain collaboration and the production of high quality m-learning tools. Constraints often relate to technical aspects of the project, time, and the need to be patient, intellectually open and willing to learn with colleagues from other disciplines. Perhaps the greatest overall challenge facing the interdisciplinary team was ‘selling the idea’ to funding bodies who appeared to lack insight into the myriad educational possibilities that m-learning tools and serious games can offer. A key area for further exploration is the area of ‘hybrid’ projects, those that are interdisciplinary in scope and comprise both applied research and product development in the field of higher education, and how hybrid projects can better capture the imagination of traditional funding FP:260 272 bodies. Moreover, attracting enough funding for rigorous evaluation of usability and impact on learning is a further challenge. Another area for exploration is the enhancement of the limited skills that academics have in knowing how to effectively market and promote m-learning tools both within the national higher education sector and globally. Conclusion Interdisciplinary collaboration offers exciting opportunities to repackage existing learning resources into apps and the ability to tap into popular trends in leisure gaming to engage students in independent learning. Interdisciplinary collaboration is not always easy, particularly when adopting a more agile design approach, but it can generate deep expertise and creative synergies. These can be harnessed to develop m-learning tools that respond to complex social problems, including the need to provide all students with the opportunity to develop good academic skills and literacy. Acknowledgments The Tune in to Uni videos were produced by the Swimming with Seahorses project, at the University of Newcastle, Australia. The project was funded through the Commonwealth Government’s Higher Education Participation and Partnership Program (HEPPP), with aim to improve participation and success rates of students from communities under-represented in higher education. The Uni Tune In app development was supported by the Centre of Excellence for Equity in Higher Education (UON). Concept and content was by Erica Southgate, Suzanne MacQueen, Jill Scevak and Deborah Bradbery. Content development was supported by Carol Lindell, Brooke Rosser, Oriel Rose and Monica Gendi. The Apostrophe Power app development was supported in part by the Centre of Excellence for Equity in Higher Education and the Literacy App project (HEPP National Priority Pool Grant 188626554). Thanks also to Georga Knight and Jeffrey Julian for editorial assistance (UON) and Samsung for equipment support. References Arnab, S., Lim, T., Carvalho, M., Bellotti, F., de Freitas, S., Louchart, S., Suttie, N., Berta, R. & De Gloria, A. (2014). Mapping learning and game mechanics for serious games analysis. British Journal of Educational Technology, 46(2), 391-411 Australian Bureau of Statistics. (2006). Summary Results. Adult literacy and life skills survey. Canberra, Australia. Charsky, D. (2010). From edutainment to serious games. Games and Culture, 5(2), 177-98. Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T. & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59(2), 661-68. Department of Education, Employment and Workplace Relations (DEEWR). (2009). Good practice principles for English language proficiency for international students in Australian universities. Canberra: DEEWR. Druin, A., Stewart, J., Proft, D., Bederson, B. & Hollan, J. (1997). KidPad: a design collaboration between children, technologists, and educators. In ACM SIGCHI Conference on Human Factors in Computing Systems, ACM, 463-470. Gikas, J. & Grant, M. M. (2013). Mobile computing devices in higher education: Student perspectives on learning with cellphones, smartphones & social media. The Internet and Higher Education, 19, 18-26. Hendricks, J., Andrew, L. & Fowler, A. C. (2014). The piloting of an Academic Literacy Education Course (ALEC) to improve academic literacy of first semester undergraduate students in a Western Australian University. Journal of Nursing Education and Practice, 4(4), 19-27. Herrington, A., Herrington, J. & Mantei, J. (2009). Design principles for mobile learning. (pp. 95-111). In J. Herrington, A. Herrington, J. Mantei, I. Olney & B. Ferry (Eds.), New technologies, new pedagogies: Mobile learning in higher education, Wollongong: University of Wollongong. Ivanitskaya, L., Clark, D., Montgomery, G., & Primeau, R. (2002). Interdisciplinary learning: Process and outcomes. Innovative Higher Education, 27(2), 95-111. Johnson, L., Adams Becker, S., Estrada, V. & Freeman, A. (2015). NMC Horizon Report: 2015 Higher Education Edition, Austin, Texas: The New Media Consortium. FP:261 273 Kift, S. M., Nelson, K. J. & Clarke J. A. (2010). Transition pedagogy: a third generation approach to FYE: A case study of policy and practice for the higher education sector. The International Journal of the First Year in Higher Education, 1(1), 1-20. Kirkness, A. (2006). Critical reflections on an academic literacies policy five years on. Perth, Higher Education Research and Development Society of Australia. Klein, J. T. (1990). Interdisciplinarity: History, theory, and practice. Detroit, MI: Wayne State University Press. Lapkin, S., Levett-Jones, T. & Gilligan, C. (2013). A systematic review of the effectiveness of interprofessional education in health professional programs. Nurse Education Today, 33(2), 90- 102. MacQueen, S., Southgate, E., Scevak, J.J. & Clement, N.D. (2012). 'It's HEPPP to be fair! Approaches to supporting the learning and well-being of low SES students in higher education'. AARE 2012 Conference Proceedings & Program, Sydney, NSW. Matthews, P., Lomas, C., Armoutis, N. & Maropoulos, P. (2006). Foundations of an agile design methodology. International Journal of Agile Manufacturing, 9(1), 29-38. Murphy, A., Farley, H., Lane, M., Hafeez-Baig, A. & Carter B. (2014). Mobile learning anytime, anywhere: What are our students doing? Australasian Journal of Information Systems, 18(3). Nicholas, K., Fletcher, J. & Davis N. (2012). Raising numeracy and literacy to improve vocational opportunities. Journal of Vocational Education and Training, 64(4), 433-451. Oberg, G. (2009). Facilitating interdisciplinary work: Using quality assessment to create common ground. Higher Education. 57(4), 405-415. Paulins, N., Balina, S. & Arhipova, I. (2015). Learning content development methodology for mobile devices. Procedia Computer Science, 43, 147-153. Repko, A. F. (2008). Interdisciplinary research. Thousand Oaks, CA: Sage. Rolls, N. & Wignell, P. (2009). Communicating at university: Skills for success. Darwin: Charles Darwin University Press. Schuetze, H. G. & Slowey, M. (2002). Participation and exclusion: A comparative analysis of non- traditional students and lifelong learners in higher education. Higher Education, 44(3-4), 309-325. Scouller, K., Bonanno, H., Smith, L. & Krass I. (2008). Student experience and tertiary expectations: Factors predicting academic literacy amongst first-year pharmacy students. Studies in Higher Education, 33(2), 167-178. Shankar, R., McAfee, F., Harris, M. & Behara, R. (2013). Smart phone app development: A multi- college approach. ASEE Annual Conference, Atlanta, GA. Smith S, Blackmore K. & Nesbitt K. (2015). 'A Meta-analysis of Data Collection in Serious Games Research'. Serious Games Analytics: Methodologies for Performance Measurement, Assessment, and Improvement, Springer, New York, 31-55. Southgate E. (2012). 'Academic literacy for first year pre service teachers: Directions for integrated learning support'. 2012 International Teacher Education Dialogue Conference, Coffs Harbour, NSW. Southgate E., Douglas H., Scevak J., MacQueen S., Rubin J. & Lindell C. (2014). 'The academic outcomes of first-in-family in an Australian university: An exploratory study.'. International Studies in Widening Participation, 1, 31-45. Spelt, E., Harm, J. Tobi, H., Luning, P. & Mulder, M. (2009). Teaching and learning in interdisciplinary higher education: A systematic review. Educational Psychology Review, 21(4), 9113-9118. Whitton, N. (2011). Game engagement theory and adult learning. Simulation & Gaming, 42(5), 596- 609. Woods, C. (2007). Researching and developing interdisciplinary teaching: Towards a conceptual framework for classroom communication. Higher Education, 54(6), 853-866. Woolfolk, A. & Margetts, K. (2010). Educational Psychology (2nd ed.). Frenchs Forrest, Sydney: Pearson. Southgate, E., Smith, S.P., Stephans, L., Hickmott, D., & Billie, R. (2015). Interdisciplinary opportunities and challenges in creating m-learning apps: two case studies. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:253-FP:262). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:262 274 Paving the way for institution wide integration of Tablet PC Technologies: supporting early adopters in Science and Engineering Diana Taylor Jacqui Kelly Judy Schrape Faculty Learning Engagement Team Faculty Learning Engagement Team Faculty Learning Engagement Team Science & Engineering, Curtin Science & Engineering, Curtin Science & Engineering, Curtin University University University The implementation of a new technology into an institution can be challenging when faced with limited support and restricted procurement procedures. Academics in the Faculty of Science and Engineering at Curtin University have been using tablet PC technology for several years to transform passive presentations into media rich, collaborative and engaging learning experiences. Recent advancements in tablet PC technology have stimulated new interest in tablet technology but also raises the question of how a university responds to the support and procurement of such new technology. In addition, what professional development is required to ensure that staff are comfortable and competent when teaching effectively with these devices. This paper presents the experiences and findings from a Community of Practice at Curtin University that embarked on evaluating and implementing three models of tablet PC at the university. The Community also engaged in a number of different professional workshops that demonstrated various strategies and fostered communication around current practice. The outcomes presented in this paper indicate the need to support academics using tablet PC’s in a responsive way rather, rather than being prescriptive on tools available through service agreements. The collaborative approach to investigating an educational technology situation used in this project could be seen as a model applicable to other contexts that involve many stakeholders across an institution. Keywords: Tablet PC, , Technology Integration, Science and Engineering, STEM, Tablet Technology Introduction and context Tablet PC’s have been used in science and mathematics education for more than ten years with academics utilising a stylus to annotate lecture slides and tutorial questions to illustrate progressive problem solving through the unique digital inking capability (Mock, 2004). The method of real time problem solving and worked solutions, are said to be an integral part of learning and understanding mathematical concepts (Loch & Donovan, 2006) and therefore practiced by many educators. Academics from Science and Engineering at Curtin University have also used tablet technology to solve mathematical concepts in a virtual classroom (Dong, Lucey, & Leadbeater, 2012) and to create screencasts of worked examples. However, despite the positive effects that tablet PCs have had on student learning (Choate, Kotsanas, & Dawson, 2014; Graves & Plant, 2010) it was found that internal support (Garrick & Koon, 2010), institutional infrastructure and quality of tablet PCs are all factors that may influence the success of implementing desired strategies (Stewart, 2013). A new generation of tablet PCs has generated an increased demand for this technology. Between 2012 and 2013, sales of tablet devices increased by 68% (Rivera & Meulen, 2014) which was likely to have been fuelled by marketing the device as a replacement for laptop computers (Jones, 2014). Improved processing power, coupled with an operating system and productivity software optimised for a touchscreen interface has increased functionality and suitability for teaching. Consequently, academics are looking to these new generation tablet PCs to address teaching and learning needs. In response to this it becomes important to investigate how an institution responds to the procurement and support of the technology as well as providing professional development for effective teaching. Within the Faculty of Science and Engineering academic staff were keen to adopt tablet PC technology in their teaching but faced institutional hurdles when dealing with Information and FP:263 275 Communications Technology (ICT) Procurement or support from Curtin Information Technology Services (CITS); a common challenge faced (Weaver, 2006). In order to address the requirements of academics and alleviate any concerns from these stakeholders, a collaborative project was established that brought together staff from CITS, Curtin Teaching and Learning (CTL) and academics from Science and Engineering into a Tablet PC Community of Practice (CoP). Communication and professional learning methodology The project utilised a CoP framework (Wenger, 2006) in response to an identified need within the Faculty. This framework was established to bring like-minded academic and professional staff together in order to facilitate discourse around the use of tablet PC technology to address teaching and learning needs (McDonald & Star, 2008); as well as to provide a supportive environment to trial and problem solve tablet PC enabled teaching and learning strategies. CoP participants were strategically selected based on their experience or interest in trialling tablet PC technology. Staff involved in this project (n=20) were predominantly academics teaching in Science and Engineering (14), with additional participants drawn from Curtin IT Services (CITS) (two) and Curtin Teaching and Learning (CTL) (four). It was believed that an inter-departmental CoP would allow for the contribution of a variety of perspectives. The Tablet PC CoP was facilitated through a number of communication and professional learning strategies (Table 1) including an online community hub (Blackboard), structured workshops, and email correspondence. Selected articles were posted on the public Curtin Teaching and Learning blog [ http://blogs.curtin.edu.au/cel/?s=tablet+PC ]. The online hub was set up as a Blackboard Community in which CoP participants could self-enrol. The hub contained PnTT project documentation and resources as well as a discussion board and group blog. Participants were encouraged to access the hub regularly to share their experiences and discuss any issues or successes that they encountered while trialling the devices. Workshops were designed and facilitated by CTL staff to encourage discussion on the use of tablet PC’s to address teaching and learning needs and to explore the capabilities and performance of the different models. During the project, academics engaged in five activities described below. Table 1: PnTT CoP Activities Activity Description and Objectives 1 – Out of the Box In the first workshop participants were invited to unpack, examine (Workshop) and briefly trial all four tablets and associated peripherals. At the end of the workshop tablets were distributed to CoP participants. 2 – Technology The performance of each tablet was evaluated and documented evaluation throughout the duration of the project. The evaluation criteria was collaboratively derived by CoP members based on the tablet PC features they considered important when completing daily tasks and teaching activities. 3 – Tablet PC Use cases In the second workshop participants were invited to respond to the in academic practice question “What are the ways tablet PC’s could be used in academic (Workshop) practice?” The GroupMap tool was used to facilitate the documenting of ideas and participants use their tablets to engage in the activity. 4 – Blackboard In the third workshop participants explored Blackboard Collaborate Collaborate (virtual classroom) using their tablet PCs. (Workshop) The aim of the workshop was to: • Trial Blackboard Collaborate (virtual classroom) and its features on the range of tablet PCs • Specifically test the stylus capabilities of the tablet PCs when using the interactive whiteboard in Blackboard Collaborate 5 – Transforming In the fourth workshop participants discussed Ruben Puentedura’s teaching with tablet PC’s Substitution, Augmentation, Modification and Redefinition (SAMR) FP:264 276 Activity Description and Objectives (Workshop) Model (Educational Technology and Mobile Learning, 2014) in relation to tablet PC enhanced teaching strategies. The aim of the workshop was to: • Present the concepts of the SAMR model and example transformations • Capture how academics have or would like to transform their teaching with tablet PC technology • Showcase OneNote features and applications Following each workshop, a set of summary notes were distributed to all participants. The notes captured topical discussions, including tablet PC performance, and were published via the Curtin Teaching and Learning blog [ http://blogs.curtin.edu.au/cel/?s=tablet+PC ]. Outcomes of the CoP activities are referenced in the remainder of the paper. Technology procurement At the start of the project, a range of tablets and associated peripherals were identified in order to evaluate the affordances of each. Initial advice and recommendations on which devices to purchase was sought from both CITS and academic CoP participants who were tablet PC enthusiasts and already possessed some level of expertise in using tablet PC’s in their teaching. Although one of the recently released tablets had received poor reviews from early adopters it was agreed that they be trialled as part of the project because they aligned with the University’s procurement agreements. The tablet specifications from the four vendors are detailed in Table 2 below. Table 2: Tablet specifications Tablets Specifications Tablets purchased Dell Venue 11 Pro 2 x Win8.1Pro/i5 Processor/ 8GB RAM/ 5 256GB SSD 3 x Win8.1Pro /i5 Processor/ 4GB RAM/ 128GB SSD Microsoft Surface Pro Win8.1Pro/i5 Processor/ 4GB RAM/ 128GB 5 2 SSD ASUS Taichi 31 Win8/i5 Processor/ 4GB RAM/ 128GB SSD 2 Sony Vaio Duo 13 Win8/i5 Processor/ 4GB RAM/ 128GB SSD 2 Total 14 Ultimately, after initial receipt of four tablet models, only three formed part of the full evaluation and CoP process. The two ASUS Taichi 31 tablets were returned to the vendor three weeks into the trial due to hardware malfunctions. Technology evaluation Proven performance and stability of any given technology is integral for the successful uptake by academic staff. In addition, the technology must also integrate with the existing university infrastructure and IT systems. Project participants contributed to a technology evaluation activity that spanned the duration of the project and served as an avenue for documenting the performance of their tablet PC. The evaluation criteria were collaboratively derived from CoP members based on what was considered important for daily work tasks and teaching activities. These included pen interaction, monitor display, battery life, integration with teaching spaces and work tasks. An “at a glance” quantitative summary of the technology evaluation activity is presented in Table 3. Based on participant feedback, the tablet PCs were given a score out of five. Please note only three of the four models initially purchased were evaluated and the number of models purchased varied hence it is important to note the number of responses for each device shown in the key. FP:265 277 Table 3: Tablet PC performance ranking 6 5 4 3 2 1 0 Pen interaction Display Battery life Keyboard Integration in Integration teaching spaces with work Sony Vaio Pro 13 (2 resp) Surface Pro 2 (5 resp) Dell Venue 11 Pro (5 resp) Pen interaction Academic staff considered pen interaction to be the most important criteria in the evaluation because of the value of handwritten annotation. The Surface Pro 2 was the highest performing model in terms of responsiveness, latency (lag), calibration, pressure sensitivity and palm detection. The only concern was the weak magnetic attachment for the stylus, which caused it to detach from the tablet when not in use. This issue has been addressed with the Surface 3 series. The Sony Vaio Duo 13 underperformed due to poor palm detection leaving residual marks on work. The Dell Venue 11 Pro was said to be slippery, like writing on glass; furthermore at times the stylus was unresponsive due to poor battery connection in the stylus. Display Overall the performance of all displays was said to be satisfactorily clear and sharp. The Sony Vaio Duo 13 performed better due to the larger screen dimensions and the Dell Venue 11 performed lower due to malfunctioning adaptive brightness settings, a bug carried over from the Dell Venue Pro 8 (Tablet PC Review, 2014). General feedback regarding the display mainly focused on screen resolution and zooming capabilities rather than clarity. This was more an issue of app optimisation rather than the quality of the screen. For complex desktop applications it was recommended that a secondary display be used via the HDMI adaptor. Alternative suggestions were to adjust the resolution of the screen or, where possible, use the app version of the product that has a touchscreen optimised user interface. Battery life Battery life impacts the amount of time an academic can facilitate learning and be mobile without relying on charging cables and power supply. The Surface Pro 2 performed satisfactorily with feedback indicating minimal battery drain whilst in sleep mode, longer performance when in power saving mode, average seven hours battery life, and two to three hours to become fully charged. Sony Vaio Duo 13 users were also generally satisfied with battery performance. The Dell Venue 11 Pro experienced considerable battery issues including battery drain whilst in sleep mode, inability to hold charge and frequent crashing. These faults were also identified by external purchasers and this contributed to the vendors’ decision to recall and replace the product. FP:266 278 Keyboard The physical keyboard is a popular peripheral tool used to facilitate text and numeric input, access quick functions via shortcut keys and inbuilt mouse track pad. A physical keyboard is often used as the onscreen keyboard occupies half the screen, which significantly reduces the amount of screen allocated for viewing applications. Feedback indicated that users had the physical keyboard attached to the device most of the time. Each model of tablet PC approached the physical keyboard differently. The Dell Venue 11 Pro had two keyboards on offer. One came with a built-in battery to extend the duration of mobility however this doubled the overall weight of the device. The Dell slim keyboard exhibited poor tactile response and frustrating typing delays. The Sony Vaio Duo 13 is an all in one unit where the keyboard is permanently attached. Users of this device were satisfied with the design and weight of the device and responsiveness of the keys. It was recommended that an additional cover would protect the device from wear and tear but was not available for that model. Users of the Surface Pro 2 were satisfied with the physical keyboard. The big keys, tactile feedback, backlit keys and doubling up as a screen cover were features that made the keyboard enjoyable to use. Some users did however experience a conflict with the onscreen pop-up keyboard such that it did not automatically popup or hide. This issue was resolved. Integration in teaching spaces Performance in existing teaching spaces is integral to ensuring a seamless student learning experience. Existing infrastructure in many teaching spaces includes access to wifi, power supply, and presentation via VGA, HDMI and AirMedia. The Surface Pro 2 was used in a variety of settings resulting in successful presentation via VGA, HDMI and AirMedia, however in one case the clarity of the display via AirMedia was a little distorted and could detract from the accuracy and legibility of the content. The Sony Vaio Duo 13 also integrated well into teaching spaces. None of the Dell Venue 11 Pro users responded to this criteria. Users stated that the device was not stable enough to use with students. Integration with work The integration with work criteria is fundamental in determining how the devices performed in carrying out desired tasks. The Sony Vaio Duo 13 performed well however one academic chose not to modify the device’s default set up meaning the Windows 8 OS was not upgraded, it was not connected to the university’s network and it could not connect to site wide licensed software. The rationale was that the academic wanted to use applications that would be accessible to students. Despite this academic noted that: The nature of interaction with students has changed and annotating student graphics is possible as are conceptual diagrams and line images. Daily tasks that are completed on other devices are finding their way onto the tablet device, as touch screen interface is a major plus! The second academic stated that carrying out work tasks was limited until the device was upgraded from 8 to 8.1 OS, and access established to site wide Microsoft applications. Feedback for the Dell Venue 11 Pro was limited as the devices were regularly faulty or out of commission.. It was noted that the device efficiently manages native windows productivity tools but only intermittently accesses files on shared network drives. The Surface Pro 2 was used for a variety of tasks including resource development, office productivity, research tasks, meetings, conference presentations and off-campus access. The feedback was also positive highlighting the benefit of accessing network drives, ability to run native windows applications, the ability to play Flash (including iLectures) and valuable annotation applications. Email from CoP member the surface has been terrific and I find so, so much greater flexibility and usefulness than iPad (which I have used for years and thought I loved more than my children!) the windows platform is more functional and the recording of mini lectures using Camtasia has transformed how I give feedback as well as lectures e.g. Mini solution tutorial or feedback video rather than uploading a 'solution'. But of course, journal and OneNote then PDF the section gives the written solution of anything I scribe onto the tablet. FP:267 279 On the down side one academic did experience issues installing MATLAB (windows based discipline specific software) however “overall I am very happy with it”. What was noted is that logging in from home takes a few minutes and that the networked devices are slow to reconfigure to a wireless configuration after being connected to the domain via an Ethernet cable. This may be related to University device authentication protocols. Technology support It was agreed that CITS would provide limited support. Support included initial set up for all devices (i.e. upgrade OS if required, connection to the network drives and access to site wide licenced software) and full support for the Dell Venue 11 Pro (i.e. service jobs for hardware malfunctions). CITS support for the MS Surface Pro 2, Sony Vaio Duo 13 and Asus Taichi 31 was not required beyond initial setup and initial troubleshooting was resolved by phone. An exception to this was the Dell Venue 11 Pro that generated over 25 X-ITS job requests far exceeding the basic jobs logged to connect any of the other tablet PCs to the network. The poor experience with the Dell Venue 11 Pro spanned the length of the project. It took three months to receive the Dell Venue 11 Pro’s due to a recall of the stylus, difficulty in deploying initial setup, followed by a complete engineering hold prior to deployment and further delay to obtain the docking stations. The Dell Venue 11 Pro was deployed in March 2014 and by May (two months into the project) numerous faults were experienced as reported by users. The issues were wide spread resulting in Dell placing a hold on shipments to Australian and New Zealand and recalling all affected devices. Based on their recall parameters only three out of the five Dells were affected, however those that were not recalled still experienced various faults. One month later (June) new tablets were issued however two of the replacements still experienced various faults. Two weeks later (July) all five Dell docking stations were recalled and replaced. The string of events was extremely onerous for everyone involved, particularly for CTL staff as they were unexpectedly coordinating significant technical issues for each device and its accessories. It was evident that the stability of the device was not satisfactory and therefore in July a refund was requested. A detailed report outlining reasons for the refund was submitted which required ongoing follow up correspondence. In September (two months later) ICT procurement approved the refund of the Dell Venue 11 Pro’s and all accessories. After what seemed to be a very complicated process, the refund was eventually received in November. The two Asus Taichi 31 devices experienced significant issues regarding the primary methods of interaction, the stylus, mouse and touchscreen interface. They were deemed unusable three weeks into the trial; were returned to the retail outlet and a full refund was issued. Institutions negotiate service level agreements with technology vendors with the aim of streamlining the procurement process, building in extended support and insurance and getting the best value for money through negotiated fee schedules. Comparing the cases previously described, the devices supplied via the service agreement required considerably more time and red tape compared to the externally purchased devices. Curtin University has acknowledged the need to be more flexible with regards to vendor and contract management and it is hoped that this study can contribute to the discussion and inform the decision making process. Integration with teaching The following discussion regarding the integration of tablet PCs to enhance teaching strategies refers to CoP activities three, four and five as detailed in Table 1. Activity three was a brainstorming workshop entitled Tablet PC in Academic Practice [http://bit.ly/1GQp2Uy ] that facilitated the documentation of how the tablet PC technologies are currently and potentially used in academic practice. Seventeen Tablet PC CoP members participated in the GroupMap activity generating 71 ideas. Details of all ideas can be viewed at http://bit.ly/NRHGGShttp://tinyurl.com/TabletPC4Teaching. A broad range of use cases were identified including: • Synchronous problem solving with annotated explanation • Annotating lecture notes FP:268 280 • Annotating diagrams • Ability to present concepts in a highly visual and progressive manner • Reviewing student work and providing annotated feedback • Collaborative student problem solving • Capability to retain digital record or generate videos • Recording from both front and rear camera (e.g. peer-client consultation activities) • Recording videos to support flipped classroom approaches • Facilitate research activities (electronic record keeping) • Facilitate fieldwork activities • Mobility in the classroom enabling small group facilitation • Academic administration Of these applications academics were primarily interested in using tablet PCs for resource development and classroom facilitation with stylus input and portability of the devices as enablers of this. It is important to note that integration with university systems such as student enrolment and management systems or finance systems do not feature in this study, as academics did not wish to use the tablet PC for this purpose. A selection of specific University supported applications were explored using the tablet PCs including Blackboard Collaborate, AirMedia, Echo360 Personal Capture 5.4 and MS OneNote (not supported). Blackboard Collaborate Blackboard Collaborate facilitates synchronous distributed learning opportunities enabled by functions such as audiovisual presentation, video conferencing, interactive whiteboard, polling and application sharing. A workshop was held to evaluate the effectiveness of Collaborate using tablet PCs. View workshop notes [http://bit.ly/1zNUXIm]. In summary both audio and video output was loud and clear, however pen output on the interactive whiteboard was very jagged. The low resolution of the tablets made it difficult to interact with user interface, particularly those features that require precise stylus/mouse interaction with smaller icons (e.g. raising a hand or selecting a polling option). For this reason some academics preferred using the mouse interaction over the stylus. This application has not been optimised for the tablet PCs and hence would benefit from using an additional display. AirMedia AirMedia enables wireless presentation in teaching spaces. Mobility in the classroom is an aspect that academics have not had much exposure to. Traditionally presentation occurs from a lectern distanced from the students. During the Out of the Box workshop connection to AirMedia was straightforward. As soon as they were connected one academic promptly made themselves comfortable in the middle of the room and commenced live problem solving (via Windows Journal) that was clearly displayed via the projector. During the Transforming Teaching with Tablet PCs workshop [http://bit.ly/1cpmTYA ] AirMedia was used to wirelessly display activities conducted in MS OneNote application. Connection to AirMedia was straightforward however projection was not accurate such that table lines and text were broken as though it was a dotted line. Further investigation regarding screen resolution and AirMedia transmission needs to be carried out. Echo360 Personal Capture Echo360 Personal Capture enables academics and students to record camera and screencast videos that are published directly into the iLecture hosting system. There are a variety of applications for video based learning resources including enabling flipped learning approaches or student created videos. Version 5.4 of the application had only just been released but it was known that it would be rolled out after platform testing. Echo360 Personal Capture performed well across the board in terms of installing the software, recording and processing the video; however the small resolution of the user interface at times required some precision to interact with. Adjusting the screen resolution resolved this. FP:269 281 OneNote OneNote is a productivity tool that has been optimised for tablet devices. OneNote files can be accessed from any platform however functionality will vary with maximum productivity features available on the Windows desktop version. This tool was used to facilitate the Transforming Teaching with Tablet PCs workshop [http://bit.ly/1cpmTYA]. Each participant contributed to the OneNote document synchronously using the stylus or text input with changes appearing without delay on participants’ devices. The document was easy to set up through self-created templates and a variety of sharing permission options. One observation was that annotations appeared out of alignment when working in the desktop version and then viewing in the web version of the product. OneNote was considered a sound productivity tool that caters for resource organisation and collaborative activities. Current projects The demand for Tablet PCs within Curtin University has increased through the current projects described below. Tablet PCs on field trips In 2014 a team in the School of Environment and Agriculture received a grant that funded the purchase of 16 Lenovo tablet PCs to facilitate student engagement and an interactive learning experience whilst on field trips. This project feeds into four Bachelor of Science course majors including Environmental Science, Coastal and Marine Science, Environmental Biology and Agriculture. Tablet PCs in laboratory’s The Department of Chemistry are planning to use tablet PC’s to facilitate new curriculum initiatives in response to industry standards and graduate expectations. The goal is to embed the use of Electronic Laboratory Notebooks (ELN) across the Bachelor of Science (Chemistry Major) course. The approach is to be trialled in a core first year unit with an intake of over 300 students. Outcomes of the project will inform the way ahead for establishing it in other units. The tablet PC technology will be used holistically throughout the unit to increase the student learning experience. The Unit Coordinator is currently exploring strategies to facilitate lecture and laboratory classes including live annotation and recording worked examples. Students will use the technology to conduct the electronic record keeping aspect of their laboratory experiments. In order to achieve this, a laboratory will be equipped with approximately 12 tablet PCs. A variety of tablets will need to be evaluated to ensure their fitness for purpose in terms of cost and performance. The outcomes of the project will inform the technical requirements for rolling out the use of tablet PCs across two additional laboratories. Touch screens in collaborative learning spaces The university has redeveloped a number of teaching and learning spaces into collaborative teaching spaces equipped with a variety of technologies. Live annotation is a recognised teaching strategy reflected in the facilities deployed in these new teaching spaces including whiteboards, interactive whiteboards, document cameras, and a few interactive tablets. There is an opportunity to review these technologies and explore how tablet PCs could further enhance the classroom experience including: • Classroom mobility – Facilitating problem solving at student tables increasing personalised support and teacher student interaction. • Resource development – Capacity to record class videos that include discipline specific software mixed with handwritten workings. • Enhanced annotations – Choice of pen colour, thickness, opacity and an output that is smooth and clear at a high resolution. FP:270 282 Conclusion This project demonstrates how a collaborative effort between stakeholders has resulted in the successful evaluation and implementation of a range of tablet PCs across Curtin University. The product evaluations revealed a wide disparity between the quality and functionality of tablet PCs on the market and highlighted the importance of engaging with the end user (customer) when deciding what product to procure. Furthermore, the documented outcomes and relationships formed through CoP activities have provided a stepping-stone for new users wishing to adopt such technology. The university needs to be confident in the technology they are procuring to address academic demand. Vendors are always improving and introducing new products based on market demand, resulting in an ever-changing landscape of available options. Therefore the channels of communication that have been opened between stakeholders including procurement, IT services, the academic and teaching and learning support should continue in order to foster innovation and meet the needs of the end user. References Choate, J., Kotsanas, G., & Dawson, P. (2014). Exploring tablet PC lectures: Lecturer experiences and student perceptions in biomedicine. Australasian Journal of Educational Technology, 30(2), 167-183. Dong, Y., Lucey, A., & Leadbeater, G. (2012). e-Review Program: An alternative online interaction for a first-year unit of Engineering Mechanics using a virtual classroom. In A. Herrington, J. Schrape & K. Singh (Eds.), Engaging students with learning technologies (pp. 111-127). Perth, Western Australia: Curtin University. Retrieved from http://espace.library.curtin.edu.au/R?func=dbin-jump- full&local_base=gen01-era02&object_id=187380. Garrick, R. D., & Koon, M. (2010). Using Tablet PCs and Pen-based Technology to Address Poor Student Performance in an Engineering Technology Class. In R. H. Reed, D. A. Berque & J. Prey (Eds.), The Impact of Tablet PCs and Pen-based Technology on Education, Going Mainstream (pp. 31-38): Purdue University Press. Graves, L., & Plant, S. (2010). An Attitudinal Study of Pen Technology And Web-based Recordings to Accommodate Students with Disabilities in Post-Secondary Science, Technology, Engineering and Math Courses. In R. H. Reed, D. A. Berque & J. Prey (Eds.), The Impact of Tablet PCs and Pen- based Technology on Education, Going Mainstream (pp. 39-46): Purdue University Press. Jones, D. (2014, May 24). Tablet Computer Sales Increasing Due to New Market Demand. Guardian Liberty Voice. Retrieved from http://guardianlv.com/2014/05/tablet-computer-sales-increasing-due- to-new-market-demand/ Loch, B., & Donovan, D. (2006). Progressive teaching of mathematics with tablet technology e- Journal of Instructional Science and Technology, 9(2). McDonald, J., & Star, C. (2008). The challenges of building an academic community of practice: An Australian case study. Paper presented at the 31st HERDSA Annual Conference, Rotorua, New Zealand. Mock, K. (2004). Teaching with tablet PC's. Journal of Computing Sciences in Colleges, Volume 20(Issue 2), Pages 17-27. Review, T. P. (2014). Dell Venue Pro 8 -- can't change screen brightness. Retrieved June, 2014, from http://forum.tabletpcreview.com/threads/dell-venue-pro-8-cant-change-screen- brightness.61280/ Rivera, J., & Meulen, R. v. d. (2014). Gartner Says Worldwide Tablet Sales Grew 68 Percent in 2013, With Android Capturing 62 Percent of the Market Retrieved 16 June, 2014, from http://www.gartner.com/newsroom/id/2674215 Stewart, A. (2013). Tablet PC use in teaching and learning: a case study. (Master of Education), Edith Cowan University. Weaver, B. (2006). The Case of the Missing Ink. In R. H. Reed, D. A. Berque & J. Prey (Eds.), The Impact of Tablet PCs and Pen-based Technology on Education, Vignettes, Evaluations, and Future Directions. Wenger, E. (2006). Communities of practice: A brief introduction. Retrieved 18 December, 2013, from http://www.ewenger.com/theory FP:271 283 Taylor, D., Kelly, J., & Schrape, J. (2015). Paving the way for institution wide integration of Tablet PC Technologies: supporting early adopters in Science and Engineering. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:263-FP:272). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:272 284 MyCourseMap: an interactive visual map to increase curriculum transparency for university students and staff Lisa B.G. Tee Laetitia Hattingh Kate Rodgers School of Pharmacy School of Pharmacy Curtin Teaching and Learning Curtin University Curtin University Curtin University Sonia Ferns Vanessa Chang Sue Fyfe Curtin Teaching and Learning Curtin Teaching and Learning Office of the Pro Vice Chancellor Curtin University Curtin University Health Sciences, Curtin University MyCourseMap is an interactive curriculum map created to increase curriculum transparency for both students and staff. It provides access to the entire curriculum at a glance, displays alignment of unit learning outcomes, assessments, course learning outcomes, and graduate attributes and links video from employers, graduates and students to help students reflect on the curriculum and its relevance. A prototype developed for the Bachelor of Pharmacy course at Curtin University as a proof-of- concept was tested and evaluated in 2014 and 2015. This evaluation utilised a mixed- methods approach using a blend of quantitative and qualitative data through online survey and structured focus group discussions. From the evaluation, the perceived benefits of the MyCourseMap include students’ increased understanding of their degree structure and its relevance to their chosen profession. From a staff perspective, the MyCourseMap helps with review and development of curriculum and professional accreditation. Barriers and challenges have led to prototype refinements. Keywords: Interactive curriculum map, mobile application, transparency, staff and student evaluation Introduction The increasing pressure on universities to attract and retain students from a wide range of backgrounds demands that more effective modes of engaging students with the curriculum and their course of study are needed (Hagel et al, 2014). Challenges associated with understanding complex course structures and appreciating the relevance of course content is enhanced when students are unfamiliar with academic discourse. Whether students enter university directly from secondary education or return to study as mature age students, academic discourse is frequently unfamiliar and difficult to navigate. Degree structures are often difficult to comprehend for commencing students due to the complexity of the course information, the unfamiliar discourse, the abundance of information and the non-interactive nature in which subject matter is presented. Imbued with academic culture and language, programs of study – including details on individual subjects – are often unfamiliar to the majority of first year students and those considering tertiary education. Typically, brochures are created to complement university handbooks to provide degree information. These include lists and descriptions of units categorized under progressive years. Prospective and current students are also able to access this information online, but the information frequently lacks detail and is presented such that there is minimal student perception of relevance. With increasing government requirements that Australian universities are to adopt a more inclusive approach and be accessible to diverse student cohorts, including those from low socio-economic backgrounds and Indigenous Australians (Department of Education 2008; Lenz, 2007), universities must be more innovative in communicating the structure, relevance and content of curricula as well as the graduate capabilities and outcomes resulting from successful completion of students (Australian Government 2011). It is imperative that universities motivate, engage and inspire students about their course of study in a clear, concise and compelling manner. MyCourseMap is an interactive visual curriculum map that supports students in understanding the structure and integration of units in their chosen or prospective degree and assist them in appreciating FP:273 285 the relevance of individual units of study to the profession or discipline. Furthermore, this dynamic and interactive tool, available through mobile and touch digital technologies, has the capacity to inform, inspire and engage students by enhancing their professional identity development. A comment from a secondary school student upon presentation of the Bachelor of Pharmacy visual degree map as an example of a curriculum map was: “This map will show me the entire degree that I will embark on. With references from videos of students and professionals telling me about the units that I will be taking in the entire degree, I will be more confident and with increased confidence, I will perform better”. For academic staff, curriculum review and renewal may be supported through a variety of mapping exercises undertaken to support a whole-of-program-approach in the incremental and progressive development of students’ achievements that align with program goals, graduate attributes and professional competencies (Ewan, 2009; Hagar & Holland, 2006). The purpose of any program mapping process is to allow cross referencing and support the integrity of the curriculum intent, thus ensuring that students achieve the intended learning outcomes with neither omissions of essential materials nor unnecessary duplication of student and staff effort. With a conceptual framework and supportive database, a coherent curriculum structure will be easy to assemble, manage and update. Furthermore, the potential of generating reports showing the curriculum elements across a course provides valuable ‘intelligence’ for informing curriculum renewal. MyCourseMap will facilitate curriculum mapping through assisting staff with examining the intended, taught, and assessed curriculum. It is important to recognise that this relationship is most frequently examined at a “subject” level, however use of an online tool will enable mapping to the level of students’ individual learning opportunities. This paper provides a strong rationale for the need of an interactive and dynamic tool such as the MyCourseMap. The MyCourseMap prototype and rationale for the entire curriculum map with alignment of unit learning outcomes (ULOs), assessment, course learning outcomes (CLOs) and graduate attributes are described. This is followed by an evaluation of the prototype by staff and students. The paper concludes with the results of the evaluation highlighting perceived benefits, barriers and future plans of the MyCourseMap. Approach to development of MyCourseMap tool Development of the MyCourseMap application is based on previous work undertaken in the development of curriculum mapping tools (Oliver, 2008, 2010; Jones 2009; Lawson, 2010). The curriculum map provides information on the key elements in a curriculum and their relationship (Prideaux, 2003). Although students are key stakeholders of programs of study, curriculum maps are typically created for academic purposes but are not necessarily used by academics on a day to day basis. In addition, the role of the curriculum in communication with students has been relatively neglected (Harden, 2001). This project combines the curriculum map, students as key stakeholders and communication in an innovative approach to engaging students and motivating their learning. Its delivery via the internet and through mobile technologies means it can be accessed from anywhere at any time. Figure 1 provides an image of the visual approach taken, providing a single page shot of all units contributing to the Pharmacy degree and their position over time and across themes. FP:274 286 Figure 1: MyCourseMap visual overview. Specifically, the MyCourseMap structure: • provides immediate relevance for degree content and its organisation • provides a visual picture of the horizontal and vertical integration across the curriculum • identifies the desired graduate capabilities with links to the degree content • links learning outcomes with unit/courses and assessment tasks • allows linking of content and assessment tasks to CLOs • embeds peer, graduate and employer stories (through text, audio and video) to demonstrate the relevance of the course/program structure and content • clearly identifies graduate employment prospects for degrees and degree streams • provides a tool which can also be used for curriculum review and renewal by embedding horizontal and vertical integration within a program of study. For staff, manual construction of curriculum maps is a cumbersome and time-intensive process. The MyCourseMap tool provides a readily accessible curriculum map which enables the curriculum to be interrogated with ease. With the development of the visual curriculum map, information linked to the map will be easily accessible to students and staff. For each degree there will be information on: • the course/program learning outcomes • the graduate attributes • professional competencies • themes which add to the matrix of meaning within the course. The MyCourseMap tool will have the flexibility to allow the curriculum map to be built according to themes (Figure 1) for the Bachelor of Pharmacy at Curtin University, or according to CLOs or professional competency standards. MyCourseMap prototype The MyCourseMap tool was first developed as an iPad App with a series of interactive ‘buttons’, representing units which are linked to the student’s course plan and assessments for the unit (Figure 1 and 2). For academic staff MyCourseMap provides a holistic view and is a readily accessible resource to enable them to view other units, their learning outcomes, teaching approaches and assessments. FP:275 287 Figure 2: Interactive ‘buttons’ representing units linked to detailed unit information including tuition pattern, syllabus, ULOs and assessments In particular, the MyCourseMap prototype allows students to clearly visualise where each assessment links to the ULOs and CLOs, highlighting the relevance of the units and their assessments for students. The application allows for unit buttons to be linked to videos of students enrolled in the course, teaching team, alumni and practitioners (Figure 3). This is a valuable approach as peer learning has been identified as a powerful and relevant learning tool which is particularly relevant to the millennial or Gen Y generation (Nimon, 2006). Figure 3: MyCourseMap homepage and unit buttons linked to short video clips of students, teaching team and professionals MyCourseMap also provides students and staff with explicit and transparent information about the entire curriculum. For example, at a touch of a button, a specific Graduate Attribute associated with particular units in the degree will be highlighted (Figure 4). Similarly CLOs associated with the degree will be illustrated. Students may not always see the relevance of units and their collective contribution. For example, first year students at Curtin are introduced to Interprofessional Education (IPE) and Indigenous Cultures and Health, but may not recognise the relevance this early on in their studies. At a click, all units with learning outcomes associated with IPE, for example, will be highlighted. With MyCourseMap, student engagement will be enhanced with the realisation that there is a continuum of learning throughout the course. Figure 4: All units linked to a specific graduate attribute are highlighted FP:276 288 The tool has particular applicability for year coordinators and staff advising and counselling students on degree progression. It also provides “one stop” information for professional staff who may be the first point of contact for students and provide advice on study plans. Methodology This investigation employed a mixed methods approach using a blend of quantitative and qualitative data to evaluate the robustness of the proof-of-concept of MyCourseMap developed for the Bachelor of Pharmacy curriculum. Through using a mixed methods design, the qualitative data provided a deeper understanding of the findings ascertained through the quantitative data collection and analysis (Creswell & Plano Clark, 2011). The quantitative data were collected through the adminstration of an online survey which gathered the perceptions of both students and staff in relation to the features of the MyCourseMap prototype. The qualitative data was collected through a series of focus group discussions. This study received ethics approval from the Curtin University Human Research Ethics Committee. Potential participants were provided with a participant information sheet with details of the study and signed a consent form. Participants were informed that participation in the study was completely voluntary and that they could withdraw at any time from the study without prejudice. Workshop and data analysis During the workshop participants were provided with an iPad to interrogate and test the robustness of the MyCourseMap App and provide feedback on its features. Each workshop was presented in three parts. 1. Background information of the design and concept of MyCourseMap was provided 2. Participants “played” with the MyCourseMap prototype using the iPad provided to test the functionality of the application including features such as video, interactive capability, entire course view at a glance. 3. Quantitative and qualitative data were collected. Quantitative data were collected via an online survey and focus group discussions evaluated the visual map features and determined further revisions and refinement of features to strengthen the MyCourseMap App. The online survey covered a range of strategies including the use of Likert scales and open-ended questions. Specifically the online survey was designed to gather the perception of students and staff on the features of the MyCourseMap tool and comprised of a suite of three questions: • To what extent do you think the following features have provided relevance or are useful to the course you are enrolled in? • How important do you think each of the following features are for inclusion in MyCourseMap application? • To what extent do you think these features will help enhance learning and teaching experiences in your course? A four point Likert scale was used (1. Very little; 2. Some; 3. Quite a bit; 4. Very relevant, useful or important). Semi-structured focus groups consisting of 8-12 participants and approximately 30 minutes in duration were conducted at the end of workshops. Facilitators used a focus group interview guide with open-ended questions to facilitate the discussion and to ensure some consistency between the various focus group discussions. Discussion focused on the best aspects of the prototype and those aspects that needed improvement and other features which participants thought might be useful. All interviews were audio-recorded and manually transcribed verbatim. Participants were de-identified and codes used in the analysis to indicate whether the participant was a staff member (SF) or student (ST) i.e. SF1 was staff focus group 1.Data were entered into NVivo and analysed using thematic analysis. FP:277 289 Results and discussion Demographic data In total 134 participants completed the online survey. Sixty-one students from all four years of the Bachelor of Pharmacy course and 73 staff members from various schools across Curtin University participated in the workshop (Table 1). The staff members who participated included 35 academics with different roles (teaching academics, course coordinators, and unit coordinators), 32 professional staff (from Curtin Teaching and Learning, Curtin Learning Institute, Curtin Information Technology Services, Student Services), and six administrators. Table 1: Demographic details of students and staff Participants Groups n % responses BPharm Students Year 1 17 12.6 Year 2 2 1.5 Year 3 7 5.2 Year 4 35 26.1 Staff Academic 35 26.1 Professional 32 23.9 Administrator 6 4.5 Total Respondents 134 Area of enrolment or work n % responses Centre for Aboriginal Studies 1 0.8 Curtin Business School 2 1.5 Curtin English Centre 1 0.8 Faculty of Science and Engineering 6 4.6 Faculty of Health Sciences 101 77.1 Faculty of Humanities 5 3.8 Vice-Chancellery 8 6.1 Others 7 5.3 Survey responses Survey responses provided evidence about the MyCourseMap features that were perceived to be relevant, useful and important for inclusion in the mobile application (Figure 5). In order of preference participants valued : (1) the ability to view the entire course map, (2) interactive unit buttons which provide unit details including tuition pattern, ULOs, assessment at a touch, (3) home page which shows the overview of the course, (4) pre-requisite map, (5) key word search functionality, (6) the “contact us” functionality to provide easy access to submission of queries, (7) interactive CLOs, (8) testimonial videos of students, staff and industry, (9) student showcase, (10) interactive graduate attributes, (11) media gallery, (12) staff showcase and (13) breaking news blog. FP:278 290 1. Entire course map view 13. Breaking 4 2. Interactive news 3.5 "unit" buttons 3 12. Staff 2.5 3. Home page showcase 2 1.5 1 4. Pre-requisite 11. Media gallery 0.5 Extent relevant and useful 0 map How important are features 10. Graduate 5. Keyword attributes search 9. Student 6. Contact us showcase 8. Testimonial 7. Interactive videos CLOs Figure 5: Students and staff perception on the relative relevance, usefulness and importance of MyCourseMap features. A four point Likert scale was used, 1 for minimum and 4 for maximum value in terms of relevance, usefulness and importance of features in the MyCourseMap tool. Focus group discussions Between September 2014 and February 2015 a total of eight focus groups were conducted: five with staff members and three with students. Each focus group consisted of between eight and 12 participants and on average took 36 + 13 minutes. Staff and students identified various advantages using MyCourseMap and provided some insights into some of the challenges and barriers to utilising the application. Advantages The four main themes about the advantages of using MyCourseMap that emerged from staff focus group data were: • Incorporation of modern technology, • User-friendly and easy to navigate, • Providing a holistic picture of a degree, and • Multiple applications and uses. Table 2 provides a summary of the four themes with selected quotations to support the themes. Three main themes about the advantages of using MyCourseMap emerged from student focus group data namely: • User-friendly and easy to navigate, • Providing a holistic picture of a degree, and • Useful for planning. Two of the themes overlapped with the staff themes (User-friendly and easy to navigate and Providing a holistic picture of a degree). Table 3 provides a summary of the three themes with selected student quotations to support these themes. FP:279 291 Table 2: Summary of staff themes about MyCourseMap advantages Theme Quote Incorporation of “It looks like most Apps, you know it works like most Apps work, which means modern everybody will know how to use it” SF4 technology “It is exactly what my teenagers would want” SF3 User-friendly “I think this is a great concept and I like the idea that you can click on first year and easy to or second year, or third year and see specific units within each year” SF1 navigate “I am not very technical and all the testers, so I managed to find my way around it very easily so that was good” SF1 Providing a “But I think it is just going to help students to see where they are going, holistic picture literally.” SF4 of a degree “... they can go in and say ...we have to learn how to present to an audience. We’re going to do it in this unit, this unit. I think that sort of mapping will give them much more connection with where they’re going to end up, because we do teach unit by unit” SF4 “... it was great to be able to see an overview, because whenever, as a designer, I would come in to work on something, it would take a while to assemble that information.” SF5 Multiple “I think it would be really useful, particularly just having done our curriculum applications and re-accreditation that we have spent a lot of time doing exactly this.” SF1 uses “You can make sure that they are scaffolder across the course. So you can take one competency and say we teach it to this lower level in the first year and in the second year we will ramp it up a bit. So you can map competencies and you can make sure that it is being done properly, rather than a bit hit-and-miss.” SF1 “I can’t really emphasise how great it is for curriculum builders, unit coordinators, Head of Schools, within Faculty. Even for the admin staff at University.” SF1 “Visually I think it is so much easier. I mean we’ve gone through accreditation. You got all your units and you’re counting down how many do I have and how many on excel spreadsheets and bits of paper. But you can just press this and go “okay my graduate attribute .... Communication, oh I’ve got that many, beautiful.” SF3 Table 3: Summary of student themes about MyCourseMap advantages Theme Quote User-friendly and “It’s quite visually simple, easy to work your way around it, work your way easy to navigate around it.” ST1 “I think it is great because when you are studying and if you need to look at a course outline, or if you need to look at the assessments it’s quick, you can just go to the one spot instead of looking through Blackboard.” ST3 Providing a holistic “When I enrolled I wasn’t sure actually what I was going to be going into. picture of a degree And I had no idea about any of the units that I would be having to enrol in but having this App it shows you all the units.” ST3 “I think especially for the first year which again is, I keep saying this, but it’s a very general year just being able to see where it fits in a greater context and that it is important because it leads into a few other units, is probably quite helpful.” ST3 Useful for planning “Yeah it will help me get prepared ... it helps me prepare for what I am FP:280 292 actually going to do in like the course.” ST3 “... so that was all in one place, it would help me get organised so that I can plan ahead and stuff like that. So yeah in the long run it would help me in my learning experience.” ST3 Barriers or challenges. Staff participants identified barriers and challenges that may impact on academics in MyCourseMap implementation, as summarised below. Complicated for complex courses These concerns involved complex degrees as reflected by the following: “... but imagine if you come from a Bachelor of Science…” SF3 “…all the options that you have in the world, it would be a mess developing them…” SF3 Keeping information up-to-date There were some concerns about the work involved to keep information current and overall the participants agreed that regular updates, i.e. every three months, would be required: “I think I am in agreement with you ... I think they want to know probably what the work load is going to be and what it’s going to lead to. But it’s changing so fast that’s the only thing.” SF2 Participants also identified the need to link MyCourseMap to Student One (Curtin University student data base) to enable changes to be automatically updated: “I think as long as it is linked to the original source of information that it would be a great central area for it. But if you haven’t got that link then it would be nightmare.” SF3 Easy to include too much detail There was agreement that the initial scope of MyCourseMap should not be very broad as this might overwhelm academics: “You don’t want it so big that then people go -Well it’s too much.” SF2 “We need to go back and say what do we want this to be? Is it just for marketing, to get them in and then all those other things that we’ve talked about then is linked further down and it goes to like Blackboard or the website or whatever. So I think we have to maybe make it smaller rather than bigger.” SF3 Student use Some staff participants indicated that MyCourseMap may have limited use for high school students. Other concerns involved the need to make students aware of using the application and there were concerns about students using MyCourseMap instead of Blackboard (Curtin’s learning management system) for important information pertaining to course requirements. Refinement following evaluation of proof-of-concept Evaluation of feedback collected aided in defining software requirements and features to improve, enhance and refine the product. As a result MyCourseMap will in future be delivered online, as a web application rather than iPad App, available any time and on any device. Significant improvements to the user experience include offering an intuitive, easy to navigate holistic picture of a degree with unit information pages that continue the information design aesthetic (Figures FP:281 293 6 to 8). For staff members, the administration interface is easy to use, and unlike complex enterprise systems, new users can get started within minutes. Addressing barriers through good software design Themes emerged in the evaluation of barriers, including the suitability of the map for complicated courses, ensuring up to date information, and the risk that too much detail could see the software become all things to all people. Furthermore there were concerns about efficacy for high school student needs, and conflict with current students as a source of accuracy for unit information. These were strong considerations in the redesign and redevelopment of the MyCourseMap application. The first principle established was to keep the solution simple – but powerful. User experience for both students and staff was considered paramount, and the design ensured that the features were universal and meaningful. Figure 6: Easy to use at-a-glance portal for course information. Tool is designed to include filters for CLOs, graduate attributes, year of study and discipline areas. MyCourseMap is fully configurable but comes pre-loaded with standard information such as teaching periods and course duration. A straightforward three year, semester driven course can be set up within minutes. The ability to add filters and streams enables more complex courses to be configured, such as those with common first year streams that then diverge into specialisations. In addressing the need to ensure up to date information whilst keeping features universal, a solution was developed to allow for manual entry. Technical staff are in the process of developing an Application Programming Interface (API) which will provide a universal language for enterprise systems to connect and share data with MyCourseMap. For small organisations or single schools, this means content can be managed directly in the MyCourseMap system, becoming a single portal for all course information. Enterprise deployments can integrate using the API, Figure 7: MyCourseMap as a web portal with aggregating information from sources such as a capacity to publish blogs, galleries and rich Student Information System (Student One), media. Course Handbook, and the Learning Management System (Blackboard). As such, the refined design ensures that students receive accurate and on-time information, and are directed to the appropriate sources of information (eg. Blackboard). Furthermore, linking with enterprise systems enables greater transparency of their course and the curriculum, which in turn increases engagement from prospective students by getting them excited about the content they are going to learn. FP:282 294 Figure 8: The unit information page can aggregate information Student One, Blackboard and the Course Handbook. Robust software design delivers increased usability and accessibility on any device at any time, and delivers the opportunity to engage current and prospective students with course curricula in a format that is modern, intuitive, and relevant for the audience. Furthermore, the software offers flexibility, allowing it to scale from small training providers to large higher education institutions. The software is applicable to the diverse needs of training providers, not only offering organisations a better planning tool for their students, but providing a marketing tool that motivates students about the topics they are going to learn through the inclusion of rich media. Finally, by its very design, it delivers a transparent map of the curriculum, in a language accessible to all and understandable by specialists including those who evaluate and conduct accreditation of training courses. Conclusion MyCourseMap will provide students and staff with explicit and transparent information about an entire course curriculum. For example, using the touch technology, a Graduate Attribute aligned to specific units in the degree will be highlighted. Similarly CLOs associated with the degree will be illustrated and made clear to both students and staff. Students may not always see the relevance of units and their contribution to the whole course. With MyCourseMap, students will understand that there is a continuum of learning throughout the course. MyCourseMap provides substantial benefits for both staff and students. Awareness of the impending course content and learning experiences can heighten students’ expectations and prepare them for the learning journey and optimise outcomes. Providing staff with an easily accessible tool which enables them to interrogate curriculum and monitor alignment between learning outcomes and assessments facilitates quality curriculum and consistent review and reflection on integrity of the student experience. The MyCourseMap offers scope for future resources to be linked to the curriculum map including simulations and virtual learning tools. Case studies could be developed which may be shared across different disciplines allowing engagement in a discipline specific manner. There is potential for the MyCourseMap to be built into the University’s course review process but operational matters such as ongoing maintenance, defining a business owner, and sourcing funds for continued enhancements will need to be considered. References Australian Government, D. O. I., Innovation, Science, Research and Tertiary Education (2011). Higher Education Standards Framework (Threshold Standards) 2011. 1. C. o. Australia, Commonwealth of Australia. Creswell, J., & Plano Clark, V. (2011). Designing and conducting mixed methods research (2nd ed.). California: Sage Publications. Department of Education, E. a. W. R. (2008). Review of Australian Higher Education Final Report. E. a. W. R. Department of Education. Canberra, Commonwealth of Australia. Ewan, C. (2009). Learning and teaching in Australian universities: A thematic analysis of Cycle 1 AUQA audits. Australian Universities Quality Agency and the Australian Learning and Teaching Council. Retrieved August 18, 2009, from http://www.auqa.edu.au/files/publications/learning_and_teaching.pdf FP:283 295 Hagel, J., Brown, J.S., Mathew, R., Wooll, M., & Tsu, W. (2014). The lifetime learner: A journey through the future of postsecondary education. Deloitte University Press. Hager, P. J., & Holland, S. (2006). Graduate attributes, learning and employability. Dordrecht: Springer. Harden, R. M. (2001). "Curriculum mapping: a tool for transparenty and authentic teaching and learning." Medical Teacher 23(2): 123-137. Jones, S. S. J., B. Oliver, B.Ladyshewsky,R.Flavell,H. (2009). Building academic leadership capability at the course level: developing course coordinators as academic leaders, Curtin university of Technology. Lawson, R., Taylor,T.,Fallshaw,E.,Summers,J.,Kinash,S., French, E., Angus-Leppan,T. (2010). Hunters and gatherers: strategies for curriculum mapping and data collection for assuring learning, Office of Learning and Teaching, Australian Government. $220,000. Lenz, K. B., Adams, J., Bulgren, A., Pouliot, N., & Laraux, M (2007). "Effects of curriculum maps and guiding questions on the test performance of adolescents with learning disabilities. ." learning Disability Quarterly 30(4): 235-244. Nimon, S. (2006). "Generation Y and Higher Education: the other Y2K. ." Journal of Institutional Research 13(1): 24-41. Oliver, B., Jones,S., Whelan,B. (2008). Building course team capacity to enhance graduate employability, ALTC. $220,000. Oliver, B. (2010). Assuring Graduate capabilities, ALTC Teaching Fellowship. Prideaux, D. (2003). "ABC of learning and teaching in medicine: Curriculum design." British Medical Journal 326: 268-270. Tee, L.B.G., Hattingh, L., Rodgers, K., Ferns, S., Chang, V., & Fyfe, S. (2015). MyCourseMap: an interactive visual map to increase curriculum transparency for university students and staff. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:273-FP:284). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:284 296 Standing on the shoulders of others: creating sharable learning designs Debbi Weaver Samantha Duque Learning and Teaching Academic Services International, La Trobe University, Australia Academic Partnerships, USA As online and blended learning becomes the norm in higher education practice, academic developers and learning designers are increasingly required to work as part of curriculum development teams to facilitate the design of engaging and interactive online courses and activities. A range of highly-effective models of workshops and programs focused on curriculum design have been developed and widely reported, each with the primary aim of developing a ‘learning design’. But what form does this learning design take? How is it prepared, shared and edited amongst the curriculum team members? And how is it then translated into a functioning online site or activity for students to access? This paper focuses on the output of curriculum design workshops, and presents a highly simplified and accessible solution for time-poor curriculum teams. Keywords: Learning design; online learning; rapid curriculum design, backwards mapping, Introduction Online education is rapidly becoming a global phenomenon, improving access to education for non- traditional students varying in location and socioeconomic status. International Consultants for Education and Fairs (ICEF, 2012) ranked Australia among the eight countries leading the way in online education. According to ICEF, Over the past five years, the online education market in Australia has grown by almost 20% and is expected to be worth an estimated US$4.68 billion this year. (Australia section, para. 2). This growing demand has caused a shift in many higher education institutions, particularly in the way programs and subjects are designed and developed. More specifically, a shift in the role of the academic developer / curriculum designer has occurred in recent years, moving from the primary role of presenting staff development workshops and consultations on demand, to a more hands-on role as part of curriculum design teams. This move is partly driven by recognition that short-term centrally- offered professional development has (with a few notable exceptions) at best minimal ongoing impact (for a brief review, see Weaver et al 2013), and partly by the increasing emphasis on blended and online learning, requiring curriculum redesigns. Concurrently, the higher education community is recognising that teaching and curriculum design is no longer a solo activity, but one which draws on the expertise of many. The focus has shifted in recent years from the individual teacher designing a module or session to include teams designing whole courses.” (Laurrillard, 2013, p26) A range of models for facilitating team-based curriculum design have been developed, trialed and evaluated, each with a primary objective of developing a shared learning design. Academics are now seeing evidence of hybrid versions of these, where institutions or even individual curriculum designers select preferred components of different models, and compile into their own version – much in the same way that teaching academics are encouraged to select preferred learning objects from open educational resource repositories, and repackage them for their own purposes. This paper focuses on just the output of a curriculum design workshop – the ‘learning design’, and selecting different aspects from a range of models, to propose a simplified yet accessible format. FP:285 297 Terminology Before proceeding, the use of the term ‘learning design’ must be defined for the purposes of this paper. A distinction has been made between learning design (no capitalisation) and Learning Design (capitalised) (Britain, 2004; Dalziel, 2003), with the former term used to “describe a representation of the learning experience to which students are exposed” (Oliver et al, 2013, p228), and the latter to refer to the concept implemented in the IMS specification, with the ultimate aim of this Learning Design being an output which is ‘runnable’ within a software package. Much work has been done in the Learning Design sphere, particularly by James Dalziel with the LAMS package (http://lamsfoundation.org/), but uptake has been poor, likely due to the complexity of working with such technical products: “… attempts to engage practitioners in the Learning Design approach have met with only partial success. This may reflect the poorly established nature of learning design within mainstream educational thinking, but could also indicate more fundamental difficulties with the transfer of standardized vocabularies and methods from an expert group to wider use.” (McAndrew & Goodyear, 2013, p.285-6) For curriculum designers working with faculty members with time constraints, requiring team members to adopt new technologies, even those without complex coding specifications, is untenable. For our purposes, we use the term ‘learning design’ to refer to the design of a complete subject (or unit of study within a program or course), including the subject intended learning outcomes, assessment, learning activities and resources. This paper focuses on the development of a simple method of capturing a learning design, to produce an accessible artefact that is intuitive and embedded with elements of a quality online subject. Curriculum design intensive (CDI) workshops Learning designs are usually the main output of Curriculum Design workshops. Many institutions have developed their own versions of Curriculum Design Intensives (CDIs), or implemented customized versions of those developed elsewhere. The origin of many of these is Gilly Salmon’s Carpe Diem model (Salmon et al, 2008; Armellini & Aiyegbayo, 2010), a two-day facilitated workshop for curriculum teams to design programs or subjects. The team at Oxford Brookes University (Benfield, 2008; also see https://wiki.brookes.ac.uk/display/CDIs/Home) modelled their CDI on Carpe Diem, and took this a step further by involving multiple project teams working on an entire program in the same workshop. Both models are similar, in that they involve cross-disciplinary teams (subject matter experts, e-learning experts, learning technologists, librarians et cetera) designing and building programs and/or subjects in a highly-productive and rapid development environment. La Trobe University adopted the Oxford Brookes CDI model as part of its FOLD (Flexible and Online Learning Development) project in 2012 (Lyons et al, 2013). The Partnership As one arm of its strategy to increase its online offerings, La Trobe University entered into a partnership with Academic Partnerships (AP) to develop postgraduate programs. These programs are designed around six-week subjects, equivalent to the more traditional twelve-week semester-long subjects, on a program rotation model, referred to as a ‘carousel’, providing students with multiple entry points into the program. Through this agreement, AP worked collaboratively with the university to deliver marketing campaigns, enroll students, and retain students, as well as provide support and quality assurance during the program and subject design phase. The Academic Services Team within AP worked in close alignment with the university’s Learning and Teaching Centre on planning the cadence of the partnership, and setting milestones, timelines, and objectives for a successful transition to online instruction. In addition, (and relevant to this paper), they provided hands-on and ongoing assistance to curriculum design teams in program planning, design, and subject development process, and in providing feedback at agreed points during the process. The authors of this paper were a curriculum FP:286 298 designer employed by La Trobe University, and a curriculum quality control consultant employed by Academic Partnerships. Subject Development Process The subject development begins with an initial meeting to explain the processes and agree on expectations and subject development timelines. A team is formed to work through designing the subject, usually including a curriculum designer, subject convener or subject matter expert (SME), and an educational technologist. The primary objective of this stage is the development of a subject learning design, or subject blueprint, detailing the key components of the subject (ILOs, assessment tasks and due dates, weekly topics and key learning activities). This initial session is followed by ongoing communication between the curriculum designer and SME, ideally by face to face meeting, but often via web-conferencing or other means, to complete all remaining aspects of the learning design. Once the learning design has been completed and reviewed, the subject development team begins to build the subject site in the Learning Management System (LMS). The subject undergoes two quality revisions by AP and final edits before going live to students. As the final stage, once the subject has been taught to students, a quality assessment is conducted, including feedback on student interaction, assessments, as well as facilitator presence. This feedback is the foundation for any changes or modifications made prior to the subject being offered for the second time. Capturing the learning design This current project began with an inherited Word document template for a subject learning design from AP, to which minor adaptations were made, primarily to wording to reflect more appropriate Australian language. Subject development teams worked on this document to develop a blueprint for the entire subject, meaning that over time, the document grew to be very large and complex. During online discussions between the authors, it quickly became apparent that the existing template was cumbersome for subject conveners to navigate, and its format was hindering the design of good ‘flow’ for students to work through their materials and activities in an efficient sequence. In an attempt to create a more workable and intuitive template for the curriculum design, acceptable to both institutions, the authors undertook a redesign of the blueprint template, drawing on what they considered the best features of the existing template, and of other models they had experienced, to develop a visually informative design, with intuitive layout, while still encouraging good practice in online teaching. Evaluation of the Inherited Template The structure of the original template was effective at ensuring constructive alignment, but its layout meant it was less effective at capturing the preferred sequence for students to work through the various readings and activities, or the context around these resources and activities. It was effective at capturing the key elements of the design, and for encouraging strong alignment of the subject learning outcomes, weekly (module) learning outcomes, weekly activities, and the assessment. By the use of targeted questions and instructions, the template was also effective in prompting subject conveners to consider facets of their subject design which they may not have previously thought through for their face-to-face teaching (for example – “List the videos or multimedia resources that you would like students to view, with a short description. Include an estimate of the time it will take students to watch this.”). The template was a large table in a Word document, meaning ubiquitous and easily-sharable technology was used, which everyone was familiar with, and could add to, comment upon, and use the review features to track changes. Table rows represented separate components (for example, required readings, optional readings, multimedia etc), which encouraged a focus on content before learning activities, and the set order of these rows did not allow for design teams to indicate which was the preferred or optimal order in which students should complete these activities. In addition, a major issue was the location of the assessment tasks at the bottom of the table, implying that assessment was to be considered after the identification of all readings and resources. FP:287 299 In many cases, this structure encouraged a ‘content dump’ of readings and resources from the existing on-campus subject design, without the necessary rethinking required to redesign for online learning. This often resulted in a lack of, or poorly thought-through, student activities to drive the engagement with these materials. Furthermore, the size and layout of the template (multiple columns on a large A3 size document) made navigation through the document increasingly difficult as more and more details were added to the design and for many, the document quickly became unworkable, especially for time-poor academic staff struggling to find relevant sections in a structure that was not intuitive to use. Criteria of a learning design artefact Regardless of the format in which learning designs are documented, essential elements include identifying the key actors involved (teachers and students), what they are expected to do (teaching and learning tasks), what educational resources are used to support the activities, and the sequence in which the activities unfold. (Lockyer et al, 2013, p1442-1443) Before discussing how the template could be restructured, it is important to define what was required from this artefact. A key artefact from all curriculum design workshops is a draft learning design, or subject roadmap. In some sessions, this is developed on whiteboards and then photographed, which captures ideas, but does not allow easy redevelopment of those ideas. In other sessions, the design is captured on large sheets of butcher’s paper, using coloured Post-it notes® to represent the sequence of different aspects (learning activities, resources, assessment, support etc.), following the method used by Ale Armellini when conducting Carpe Diem workshops (Armellini & Aiyegbayo, 2010). This is an engaging and effective method to collaboratively develop a design, especially when mapping how long activities will take and looking at due dates etc., as it is easy to move the post-it notes around, and visually understand how the subject would roll out for students. However, this model results in a physical artefact which is not then easy to share or digitally edit. Usability criteria The first step in our redesign was identifying the usability criteria for the artefact: • Easy to use- essential to allow time-poor academic staff to begin their subject development process without any training or induction into the use of the template. • Shareable and editable- as the blueprint document evolves through different stages in the subject development process, it must be accessed and edited by different individuals. Pedagogical Criteria When transitioning a subject that was first taught on campus, it is often common for facilitators to want to use the same curriculum and activities. However, a key strategy to discourage a simple translation of face-to-face teaching into the online environment is to encourage subject conveners to take a ‘backwards mapping’ approach, looking at the learning outcomes and assessment tasks before considering learning activities, and leaving decisions on content and other resources until last (Great Schools Partnership, 2013). This often takes subject conveners out of their comfort zone, but is effective in designing the necessary alignment. A learning design template needs to address the following considerations in order to encourage a backwards mapping approach to curriculum. Alignment Constructive alignment among ILOs, assessments, readings, and activities is crucial in any subject design (Biggs & Tang, 2007). The most common pitfalls in a subject are unclear learning objectives as well as assessments that are not aligned with the learning objectives (Jones et al, 2011) Therefore, the learning outcomes and assessments were deemed to be crucial parts of the template; all other activities must revolve around these two main components. Interaction Opportunities for student interaction are considered indicators of quality of online subjects, as identified by the internationally recognized Online Learning Consortium scorecard Standard 1 of Teaching and Learning (OLC, 2015). This Standard states that “Student-to-student and faculty-to- FP:288 300 student interaction are essential characteristics and are encouraged and facilitated”. Also related to this standard, these types of interactions promote the formation of a community of inquiry, and improve student motivation, student engagement, and student satisfaction (Drouin, 2008, Arbaugh et al, 2008). Moreover, Ke (2010) found that high online presence were positively correlated with learning satisfaction. Accordingly, a learning design template should encourage interaction wherever possible and appropriate, both in assessment tasks and weekly learning activities. Flow of activities The template should encourage designers to consider the order in which students are likely to move through the activities and resources, and provide the necessary contextual linkages between these. Since different team members at La Trobe University utilize the template to build the subject in the LMS, the document should be sufficiently detailed for an educational designer to build the LMS site, without continually checking with other members of the design team. Time Providing estimates of the time required for activities and readings not only helps students plan their study, but also helps the design team monitor student workload. In addition, estimates of time required for facilitators to monitor discussions and give feedback on assessment tasks is useful during the planning stage to ensure these tasks can be met effectively. Throughout the six-week subjects, feedback on assessment submissions is required within five days, so the design team must ensure this is achievable for facilitators when planning due dates and other activities requiring moderation. Consistency The elements and key subject characteristics outlined in the template had to promote consistent subject structure throughout the program. Expecting the same subject structure regardless of the content of the subjecthelps build students’ confidence, allowing them to focus on their learning instead of subject navigation. Designing a new template It was decided early on that tables in a Word document was the preferred file type, utilizing ubiquitous technology in a format that is familiar to all staff, and easily editable by all, and to then redevelop the existing template to incorporate additional features, according to the identified criteria (above). The method of planning learning designs used by Ale Armellini in his Carpe Diem workshops, using colour Post-it notes® on large sheets of butcher’s paper, was very attractive. A co-author of this paper had adopted this method in similar workshops, and devised a simple Word table, with shaded columns colour-matched to the Post-it notes® to represent the same components, as a method of digitally capturing the output from a Carpe Diem session. This proved popular with curriculum teams, who were able to instantly recognize the colour-code as one which they had worked with in their workshop, and appreciated the easily sharable and editable format. However, this model comprised a very basic description of the learning design, and did not attempt to record key aspects such as ILOs, constructive alignment, or other necessary components. The authors were also attracted to the task swimlane model developed by Conole (2013a), and built on the earlier concepts of Oliver and Herrington (2001). This model provides a clear and easily understood visualization of the learning pathway that learners were expected to take, including any resources or tools required to undertake that pathway. Conole’s swimlane view was developed in CompendiumLD software, which allows users to create additional swim lanes (to represent other aspects of the design such as estimated time, teacher activity etc.), and also to attach key documents (for example, resources such as assessment instructions, rubrics etc.) to key activities. On the downside, this model requires all curriculum design team members to have access to the software (albeit freely available), and familiarity with the interface and terminology used, which is not always intuitive and takes significant time and effort to master (Conole, 2013b). Additionally, sharing the output can be difficult. Finished designs can be exported as jpeg images, which do not allow for other members of curriculum design teams or external collaborators to edit or add comments. Nevertheless, the concept of displaying learner activity, resources and other aspects (for example, teacher activity, detailed instructions etc.) is a powerful one. FP:289 301 After consulting these previous models, more recent literature and with colleagues, it was decided that the redesigned template should consist of three key sections: 1. Subject overview: Subject name and code, Subject intended learning outcomes (ILOs), and a brief subject description 2. Assessment: Each assessment task is detailed, with a brief description, instructions for students, together with the total marks available, due dates, format of submission, and key assessment criteria 3. Weekly modules: a brief description of the week’s activities, then a detailed plan including all student instructions, links to readings and resources, details on activities (including links to any tools required), and all contextual statements to link these items in a logical and engaging sequence. The structure of the template for the weekly modules was designed in line with Conole’s swimlane concept (Conole, 2013a), including columns for what students do, what resources they need, what facilitators do, and key due dates. These columns were then colour-coded to match the colour of the Post-it notes® used in the curriculum design workshops, similar to the coloured visual representation proposed by Agostinho (2006). After the initial redesign, a period of consultation about the template with colleagues was undertaken. It proved impossible to gather a large group of colleagues together at the same time, so enlarged copies of the draft template (as well as associated documents about the entire CDI process) were taped to the centre of a whiteboard in a shared meeting/lunch room, with instructions written around this to provide minimal explanation and context. Colleagues were asked to leave comments, either on the whiteboard, or using Post-it notes® to add to the template document (so comments could be attached to particular areas). Over the course of a week, colleagues cheerfully engaged with this process, and some dozens of comments were left, providing a wealth of constructive suggestions to further improve both the entire CDI process, and particularly, the learning design template. Learning design template The images below illustrate the structure of the template, including brief descriptions or explanations to help subject conveners understand what is required for each section. Figure 1: Subject overview section of the template. FP:290 302 Figure 2: Weekly learning design section of the template. This section is copied for each module or week of the study period or semester. Implementation and modification The new template design was implemented immediately into the curriculum design process. It proved significantly easier to use than the previous design, and subject development teams reported it quickly became an integral part of the development process, often replacing the previous methods used to capture ideas. In some cases, where the subject convener was located at regional campuses or otherwise not available for face-to-face curriculum design workshops, the template could be used during a Skype meeting to discuss the key alignment and assessment components of the subject, and has been used successfully as a substitute for these face-to-face sessions. FP:291 303 Figure 3: Excerpt from a module learning design Subject conveners appeared to find the layout of the module design section easy to use, and were encouraged to provide quality contextual statements and guiding questions for students, in a format ready to enter directly into the LMS (see Figure 3 for a short excerpt from a module learning design, demonstrating the detail provided for a single activity). Some even used the Teaching Activities column to add notes for tutors and facilitators, which could be added to the LMS (hidden from student view) to assist the teaching team (Figure 3). Due to the short timelines of the subject development process involving multiple subjects at different stages of development, no formal evaluation of the template design could be implemented. However, observations from colleagues attest to the improved focus on student learning: FP:292 304 The current [template] speaks clearly to the academic and designer – a crucial issue. It also shifts emphasis in design from descriptions of content to sequences for teaching and learning activities. Hence [it] focuses on student paths to learning. (John Hannon, Senior Educational Developer, La Trobe University) One of the original objectives was to produce a comprehensive artefact, with sufficient detail to enable a learning technologist to build the entire LMS site. It was quickly discovered that the pilot template included all the elements of the assessment and learning design, but missed a few vital elements usually required for the home page of a subject LMS site – for example, a welcome message (preferably via video) from the subject convener, contact details for the teaching team, a learning schedule to help students plan their study, and links to key university support services. Subsequently, an additional page was added to the template to include a checklist of these items (Figure 4). Figure 4: Checklist of subject home page components Conclusions This paper describes the development of a simple Word document template to capture the learning design of fully online subjects, improving the processes involved in a rapid curriculum development project. At this stage, the learning design template works well, but does not accommodate complex learning designs (for example, multiple pathways or branching of pathways). It was always the intention that this template would evolve as required, and it is anticipated that more complex designs will drive further development of this document. Acknowledgements The authors wish to thank Robyn Yugel for insightful suggestions on improving our paper, Dr. Jennifer Spoor for permission to use screenshots of her subject learning design, and our colleagues for constructive suggestions during our template design phase. We also wish to thank the anonymous reviewers who provided thoughtful feedback on our earlier submission. References Agostinho, S. (2006). The use of a visual learning design representation to document and communicate teaching ideas [online], Who’s Learning? Whose Technology? Proceedings of the 23rd Annual Ascilite Conference, Sydney 2006, http://www.ascilite.org.au/ conferences/ sydney06/ proceeding/ pdf_papers/ p173.pdf. [viewed 18 Jun 2015]. Agostinho, S., Oliver, R., Harper, B., Hedberg, H., & Wills, S. (2002). A tool to evaluate the potential for an ICT-based learning design to foster "high-quality learning". In A. Williamson, C. Gunn, A. Young., & T. Clear (eds.), Winds of change in the sea of learning. Proceedings of the 19th Annual Ascilite conference. (pp. 29-38). Auckland, New Zealand: UNITEC Institute of Technology. http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1128&context=edupapers [viewed 18 Jun 2015]. FP:293 305 Arbaugh, J. B., Cleveland-Innes, M. Diaz, S. R., Garrison, D. R., Ice, P., Richardson, J. C., & Swan, K. P., (2008). Developing a community of inquiry instrument: Testing a measure of the Community of Inquiry framework using a multi-institutional sample. The Internet and Higher Education, 11 (3- 4), 133-136. Armellini, A., & Aiyegbayo, O. (2010). Learning design and assessment with e-tivities. British Journal of Educational Technology, 41 (6), 922–935. Benfield, G. (2008). E-Learning course design intensives: disrupting the norms of curriculum design. Educational Developments, 9 (4). 20-22. rd Biggs, J., & Tang, C. (2007). Teaching for Quality Learning at University (3 ed), Buckingham: SRHE and Open University Press. Britain, S. (2004). A Review of Learning Design: Concept, specifications and tools, JISC Project Report. Online.: www.edtechpost.ca/wordpress/files/2004/07/ACF83C.doc [viewed 18 Jun 2015]. Conole, G. (2013a). Designing for Learning in an Open World, New York: Springer. Conole, G. (2013b). ‘Tools and Resources to Guide Practice’ in Beetham, H. and Sharpe, R. (eds) nd Rethinking Pedagogy for a Digital Age 2 Edition, pp.113–130, Oxford, Routledge Falmer Dalziel, J. (2003). Implementing Learning Design: the Learning Activity Management System (LAMS). In G.Crisp, D.Thiele, I.Scholten, S.Barker and J.Baron (Eds), Interact, Integrate, Impact: Proceedings of the 20th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education. Adelaide, 7-10 December 2003. http://www.ascilite.org.au/conferences/adelaide03/docs/pdf/593.pdf [viewed 18 Jun 2015]. Drouin, M. (2008). The relationship between students’ perceived sense of community and satisfaction, achievement, and retention in an online course. Quarterly Review of Distance Education, 9 (3), 267-284. Great Schools Partnership (2013) Glossary of Education Reform. http://edglossary.org/backward- design/. [viewed 18 Jun 2015]. International Consultants for Education and Fairs (ICEF) (2012). 8 countries leading the way in online education. (2012, June 28). http://monitor.icef.com/2012/06/8-countries-leading-the-way-in-online- education/ [viewed 18 Jun 2015]. Jones, K.A., Jones, J., Vermette, P. J. (2011). Six Common Lesson Planning Pitfalls- Recommendations for Novice Educators, Education, 131 (4), 845-864. Ke, F. (2010). Examining online teaching, cognitive, and social presence of adult students. Computers & Education, 55, 808-820. LAMS Foundation. http://lamsfoundation.org/ [viewed 18 Jun 2015]. Laurillard, D. (2013). “Forward’ in Beetham, H. and Sharpe, R. (eds) Rethinking Pedagogy for a nd Digital Age 2 Edition, 26–30, Oxford, Routledge Falmer Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57(10), 1439-1459. Lyons, J., Hannon, J., & Macken, C. (2013). Sustainable Practice In Embedding Learning Technologies: Curriculum renewal through course design intensives. In Maree Gosper & Dirk Ifenthaler (eds.), Curriculum Models for the 21st Century: Using Learning Technologies in Higher Education (pp. 423-442). Springer, New York McAndrew, P. & Goodyear, P. (2013). ‘Representing Practitioner Experiences through Learning Design and Patterns’ in Beetham, H. and Sharpe, R. (eds) Rethinking Pedagogy for a Digital Age nd 2 Edition, pp.226–255, Oxford, Routledge Falmer Oliver, R., & Herrington, J. (2001). Teaching and Learning Online: A beginner’s guide to e-learning and e-teaching in higher education, Perth: Edith Cowan. Oliver, R., Harper, B., Wills, S., Agostinho, S., & Hedberg, J. G. (2013). ‘Describing ICT-Based Learning Designs that Promote Quality Learning Outcomes’ in Beetham, H. and Sharpe, R. (eds) nd Rethinking Pedagogy for a Digital Age 2 Edition, pp.226–255, Oxford, Routledge Falmer Online Learning Consortium (2015) OLC Quality Scorecard. http://onlinelearningconsortium.org/consult/quality-scorecard/ [viewed 18 Jun 2015]. Oxford Centre for Staff and Learning Development (OCSLD). (2009). Course design intensives. Oxford, UK: Oxford-Brookes University. https://wiki.brookes.ac.uk/display/CDIs/Home [viewed 18 Jun 2015]. Salmon, G., Jones, S. & Armellini, A. (2008). Building institutional capability in e-learning design. Research in Learning Technology, 16, 2, 95– 109. http://www.researchinlearningtechnology.net/index.php/rlt/article/view/10889 [viewed 18 Jun 2015]. FP:294 306 Weaver, D, Robbie, D, Kokonis, S & Miceli, L (2013) Collaborative scholarship as a means of improving both university teaching practice and research capability, International Journal for Academic Development, 18(3), 237-250, DOI:10.1080/1360144X.2012.718993. Weaver, D. & Duque, S. (2015). Standing on the shoulders of others: creating sharable learning designs. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:285-FP:295). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:295 307 Higher Education Teachers’ Experiences with Learning Analytics in Relation to Student Retention Deborah West Henk Huijser David Heath Learning and Teaching Flexible Learning and Innovation Charles Darwin University Charles Darwin University Batchelor Institute of Indigenous Tertiary Education Alf Lizzio Danny Toohey Carol Miles Learning Futures School of Engineering and Information Teaching and Learning Griffith University Technology The University of Newcastle Murdoch University This paper presents findings from a study of Australian and New Zealand academics (n = 276) that teach tertiary education students. The study aimed to explore participants’ early experiences of learning analytics in a higher education milieu in which data analytics is gaining increasing prominence. Broadly speaking participants were asked about: (1) Their teaching context, (2) Their current student retention activities, (3) Their involvement in, and aspirations for, learning analytics use, (4) Their relationship with their institution around learning analytics. The sampled teaching staff broadly indicated a high level of interest but limited level of substantive involvement in learning analytics projects and capacity building activities. Overall, the intention is to present a critical set of voices that assist in identifying and understanding key issues and draw connections to the broader work being done in the field. Keywords: Learning Analytics, Student Retention, Higher Education Introduction This paper reports on one component of an Australian Government Office for Learning and Teaching funded project entitled Learning Analytics: Assisting Universities with Student Retention. Carried out over the past eighteen months, this mixed-method study has been investigating the factors that impact on the implementation of learning analytics for student retention purposes. At the commencement of the project, a survey of Higher Education institutions (n = 24) was carried out that found that typically institutions were, in July and August, 2014, focused on exploring, planning and piloting different tools and applications designed to improve their analytics capacity (West, 2015). Though analytics was the subject of much attention in institutions, what was less clear was the extent to which the focus of analytics would be ‘business’ dimensions like human resources, marketing, performance management, and workload allocation or whether the analytics focus would be more on ‘educational’ dimensions like learning environments, curriculum design, pedagogical intent, and student experience, for example. Although these two broad dimensions are not necessarily dichotomous, the initial institution level survey suggested that integrating human resources, finance, research, and marketing systems into some kind of data warehouse tended to be one of the more advanced strategic priorities within surveyed institutions at the time (West, 2015). The institution level survey provided some useful baseline data around institutional decision making and progress with learning analytics, but the ways that teaching staff were influencing the direction for learning analytics or participating in learning analytics pilots and projects remained unclear. The next phase of the project involved the deployment of an academic level survey, which aimed to further knowledge about the experiences of teaching staff and other academics with learning analytics, explore their aspirations, and elicit their views on key issues identified in the literature. Data from the academic level survey is the primary focus of this paper. FP:296 308 Background Learning analytics The rise of big data, growth in online learning, and changing politics around Higher Education are driving interest in learning analytics (Ferguson, 2012). Ochoa, Suthers, Verbert & Duval (2014: 5) observe that “learning analytics is a new, expanding field that grows at the confluence of learning technologies, educational research, and data science”, before indicating that learning analytics has potential to solve two simple but challenging questions: 1. How do we measure the important characteristics of the learning process? 2. And how do we use those measurements to improve it? Given the breadth of the above description it is perhaps unsurprising that previous research (Corrin, Kennedy & Mulder, 2013) found that understandings of learning analytics vary amongst academic staff. Further, the questions listed by Ochoa and colleagues do not seem too different to those that have existed in Higher Education for many years. However, Ferguson (2012) makes the point that learning analytics typically includes a pair of assumptions around the utilisation of machine readable data and a focus on big data systems and techniques. Student retention The academic and non-academic factors that can influence retention are complex and varied (Nelson, Clarke, Stoodley & Creagh, 2014). Complicating matters are the relationships between retention, success, and engagement. Helpfully though, there are numerous relevant studies, including recent OLT and ALTC projects, on student retention (see Nelson et al, 2014; Willcoxson et al., 2011), alongside studies on learning analytics with some connection to student retention, with Signals at Purdue University (Arnold & Pistilli, 2012) a noted example, though more are emerging (e.g. Harrison, Villano, Lynch & Chen, 2015). Thinking more holistically, Tinto (2009) suggests that to be serious about student retention, universities need to recognise that the roots of student attrition lie not only in students and the situations they face, but also in the character of the educational settings in which students are asked to learn. If one goes back to the definition adopted by the Society for Learning Analytics Research (SoLAR), which articulates learning analytics as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs”, it becomes clear that student retention (and success and engagement) have a natural affinity with learning analytics. Tinto (2009: 3) has articulated four conditions of student success: expectations, support, feedback, and involvement (or engagement) and Nelson et al. (2014: 96) take this idea further and add more detail in their Student Engagement Success and Retention Maturity Model (SESR-MM) that includes the following categories: 1. Learning – assessment, curricula, teaching practices, pedagogical styles 2. Supporting - information, services, resources, ‘people rich’ advice, advocacy and peer support 3. Belonging – interaction, inclusive activities, identity development/formation opportunities 4. Integrating – academic literacies, personal literacies 5. Resourcing – staff development, evidence base, communication, learning environments Both Tinto’s four conditions, and especially Nelson et al.’s categories are potentially measurable, which is where learning analytics becomes particularly relevant. Linking teaching staff to learning analytics and retention Corrin et al (2013) reported on findings from a focus group study featuring 29 staff associated with teaching and learning at one Australian institution. A variety of educational problems, situations, and potential ideas were raised by the participants in their study and these fell into five broad categories: 1. Student performance FP:297 309 2. Student engagement 3. The learning experience 4. Quality of teaching and the curriculum 5. Administrative functions associated with teaching These few studies alone illustrate that sizeable variation exists with respect to how learning analytics might be applied to issues like student retention. With this in mind, the intention of this study was to both incorporate the concepts in these studies into the research instruments and also consider how participant responses to open-ended questions fit or did not fit with these typologies. Aim As learning analytics is multi-disciplinary, multi-method and multi-level in application, and this study was conducted at a time when participant knowledge was difficult to predict, the research questions are necessarily broad in scope. They are: 1. What variety exists in the online environments where teaching takes place? 2. What involvement do teaching staff currently have in using data to respond to retention issues? 3. In which learning analytics related activities have teaching staff been involved? 4. In which retention applications of learning analytics are participants most interested in? 5. How are institutions supporting learning analytics use amongst teaching staff? Method Sampling Procedure The survey employed a purposive, snowball sampling strategy to recruit self-selecting individuals. Given the sizeable pool of potential participants, voluntary nature of the research, and presence of other Higher Education focused projects also seeking participants, obtaining a high response rate was expected to be a significant challenge and this was reflected in the data collection phase. The research team did take a number of steps to try and minimize sample bias and information about participant recruitment and sample demographics will be presented to support evaluation of the representativeness of the sample. Participant Recruitment Invitations were circulated via three main avenues: 1. Project team networks: The project team, reference group and evaluation team were comprised mainly of senior academics so the decision was taken to use their networks with other institutions leaders to facilitate as broad a distribution as the voluntary nature of the project would allow. Although the ideal scenario would have been universal distribution by institutions, in reality the approaches to senior institutional contacts resulted in varied forms of distribution: • Distribution of invitations via a specific learning and teaching mailing list; • Placement of information in a broader newsletter; • Forwarding to department heads for discretionary distribution; • Distribution of the survey invitation throughout the institution; and, • Declining to distribute information about the project. Follow up confirmed that the invitation was circulated to staff in some capacity in at least 25 institutions. In most cases distribution was partial and in three cases it was institution-wide. 2. Professional interest groups: Information about the project was distributed through either meetings or the newsletters of the Higher Education Research and Development Society of Australasia, Universities Australia, Council of Australian Directors of Academic Development, Australasian Council on Open and Distance Education, and Council of Australian University Librarians. 3. Conferences and workshops: As is fairly typical, project team members attended conferences FP:298 310 and conducted workshops as the project progressed, but to avoid a disproportionate number of learning analytics ‘enthusiasts’, participant recruitment via this avenue was intentionally incidental rather than proactive. Table 1 presents data that shows how this overall approach led to response patterns indicative of wide distribution (i.e. one that is not stacked with many participants just from partner institutions, for example). Table 1: Survey completion information Minimum number of institutions with at least one participant 21 Separate days where at least one survey was commenced 47 First survey commenced 2/9/2014 Last survey commenced 13/11/2014 Demographics In total 401 people viewed the survey’s first question. Forty-eight people (12%) who answered no questions or only demographic questions were excluded. Of the remaining 353 participants, 276 (78%) answered yes to the question “do you teach students?” This paper is concerned with those 276 respondents. Using this parameter allows issues specific to teaching staff to be identified and explored. Table 2 presents a summary of the sample demographics. Table 2: Frequency distribution of selected demographic data Variable Category Absolute Relative (n varies due to missing data) Frequenc Frequenc y y Location Australia 269 98% (n = 274) New Zealand 5 2% Primary Work Role Teaching Students 185 67% (n = 276) Learning Support 25 9% Other 24 9% Research 19 7% Management/Administration 12 4% Academic Development 7 3% Student Support 4 1% LMS at Institution Blackboard 175 63% (n = 276) Moodle 89 32% Other 12 4% Employment Basis Full Time 223 81% (n = 275) Part Time 35 13% Casual 15 5% Other 2 1% Academic Level Lecturer 115 42% (n = 276) Senior Lecturer 79 29% Associate Professor 28 10% Associate Lecturer/Tutor 24 9% Professor 18 7% Other 12 4% Length of employment in current Less than 1.5 years 18 7% institution 1.5 – 5 years 57 22% (n = 251) 5 – 10 years 77 31% 10- 20 years 72 29% More than 20 years 27 11% Length of employment in Higher Less than 1.5 years 4 1% Education Sector 1.5 – 5 years 35 13% (n = 269) 5 – 10 years 61 23% 10- 20 years 105 39% More than 20 years 64 24% FP:299 311 Enrolment modes of students Internally enrolled students only 144 52% taught A mix of internal and external 105 38% (n = 276) students Externally enrolled students only 14 5% Other 12 4% In relation to primary work role the ‘other’ response was 9%. This group of responses did not expand much on the categories listed, rather most participants who put ‘other’ did so to express a reluctance to identify a single role as ‘primary’, with eighteen people nominating a split between teaching and research. Materials and Procedure The survey was a purpose designed online questionnaire built and hosted at Qualtrics. It was accessed via a link which made responses anonymous. To alleviate risk of multiple completions by individuals the software allowed one survey attempt per computer/IP address. Participants could save and return to an incomplete survey. After two weeks with no access an in-progress attempt was automatically closed and placed with completed surveys. Ethics approval for the study was granted by the Charles Darwin University Human Research Ethics Committee. Results Question 1. What variety exists in the online environments where teaching takes place? The teaching environments chosen by academics have a range of implications for what is possible with learning analytics. Table 3 details results when participants were asked about their use of tools or utilities outside the LMS for teaching. Table 3: Frequency distribution of online teaching activities of participants Variable Category Absolute Relative Frequenc Frequenc y y Tools or utilities Does not use tools or utilities to teach outside the 120 44% outside the LMS LMS* used for teaching Website hosted externally 57 21% (n = 272) Website hosted by their institution 54 20% Others 53 20% Social media applications 51 19% Mobile apps 22 8% Teaching Provision of access to learning materials 89 63% activities Assessment submission and feedback 75 52% conducted Learning focused interactions between lecturers and 59 41% outside the LMS students (n = 156)** Learning focused interactions between students 48 34% *denotes mutually exclusive response **Not asked of 120 participants not teaching outside the LMS For the first variable in Table 3 there was space to list the applications for the ‘social media’, ‘mobile apps’ and ‘other’ options. Social media applications included: Facebook (28), Twitter (14), YouTube (13), Yammer (3), Instagram (3), Pinterest (2), WordPress (2), Blackboard (2) and 10 singularly mentioned applications. In relation to mobile apps it was apparent that a distinction between social media and mobile apps was not necessarily mutually exclusive or clear. There were 20 different mobile apps mentioned. Finally, in the open ‘other’ category there was a wide mix of responses. Coding into categories found these could be grouped into functions, which are, with examples from participant responses: • Productivity and content creation (e.g. multimedia software, Creative Cloud, iMovie); FP:300 312 • Communication (email, Facebook, Skype); • Discipline dedicated learning resources and tools (many e.g. MathLab; Skritter); • General content repositories housing learning materials (YouTube, Vimeo, Lynda); • Polling and Quizzing (Respondus; PollEverywhere); • Document storage, sharing and portfolio creation (e.g. Google Docs, Mahara); • Virtual and Simulated Learning Environments ( e.g. Smart Sparrow); and, • Shared content creation spaces (e.g. Wikis and Blogs). Question 2: What retention related data are participants accessing and using Methods of identifying at risk students One way of exploring the uptake of learning analytics was to explore the types of data that participants were using to determine risk. Table 4 presents the frequency distribution of selected data sources used by participants. Please note that in the design and pilot phase it was unclear how certain participants would be so there a couple of different types of other categories to reflect this. The responses in the table are presented as they were in the survey instrument. In terms of the other option, class attendance and colleagues were the strongest responses. Table 4: Data sources considered when identifying at-risk students (n = 246) Data source Absolute Relative Frequency Frequency Students self-reporting that they are having issues that might affect their 146 59% retention LMS 140 57% Directly asking students if they are having any issues that might impact their 123 50% retention Student Information System 59 24% Advised by specialist team that has their own retention monitoring 45 18% processes Learning Support 38 15% Student Support 33 13% Does not take action to identify students with retention related risks* 24 10% Consults data from other sources 23 9% Teaching tools or utilities outside the LMS 15 6% Consults data from other source/s but is not sure what they are called 9 4% Library 5 2% * denotes mutually exclusive response Figure 1 shows a frequency distribution of participants’ use of selected indicators to identify at-risk students in relation to retention. Notable here is a trend toward indicators that relate to actual performance than more predictive indicators often collected as part of student enrolment. As with the previous table, class attendance was also the most common response where participants were provided with space to put ‘others’. FP:301 313 Task completion Attainment of certain grades LMS access patterns (e.g. logging in) Use of communication tools (e.g. discussion board) Repeat student status English proficiency Materials or resources access patterns (e.g. opening readings) Time spent in LMS International student status Academic pathway – basis for entry Home language Aboriginal and Torres Strait Islander Status Socio-economic status First in family to enrol in Higher Education Birth country Not sure* 0 40 80 120 160 200 Figure 1: Indicators used to identify at-risk student in relation to retention (n = 276) Involvement in responding to at-risk students Participants were asked whether they had a systematic response when students met identified risk thresholds. 103 participants had a systematic response, of which 24 (23%) indicated it applied to all thresholds and 79 (77%) indicated it applied to some thresholds only. The 103 participants that did have a systematic response were asked about the elements that comprised that response. Overwhelmingly, the most common responses were manually conducted (e.g. manual emails, telephone calls, offers of consultation, manual referrals to support services). The primary automated methods (emails, automated referrals, or those in the ‘other’ category) all had a frequency where n = <15. Question 3: In which learning analytics related activities have teaching staff been involved? The study also investigated participation in learning analytics activities. Results focus on the frequency of learning analytics discussions that teaching staff are involved in and the involvement of teaching staff in a more diverse selection of analytics activities. Learning analytics discussion involvement Figure 2 explores how often the teachers sampled discussed learning analytics with colleagues in different roles. For example, the series on the right hand side represents how often the teaching staff sampled discussed learning analytics with institutional management. Higher bars on the left of each series indicate more frequent discussion. FP:302 314 Daily Weekly Fortnightly Monthly < Monthly Never 180 150 120 90 60 30 0 Teaching Program or School or Learning Students Colleagues in Central L&T Student Institutional Staff Course Co- Faculty support staff (n = 250) Communities Group Staff Support Staff Management (n = 253) ordinator Management (n = 241) of Practice (n = 240) (n = 234) (n = 237) (n = 242) (n = 243) (n = 243) Figure 2: Frequency of learning analytics discussion with selected groups (n varies by variable) Learning analytics activity involvement Participants were also asked about whether they had been involved in a selection of learning analytics related activities. Table 5 presents the results. Table 5: Frequency distribution of involvement in selected learning analytics activities (n = 276) Learning analytics related activity Absolute Relative Frequenc Frequenc None of the listed choices* 108 40% Using learning analytics to help with analysis and decision making 101 37% Reading about learning analytics for their own professional development 100 37% Advocating for the use of learning analytics to colleagues (informal or formal) 70 26% Attending conferences/ training specifically to learn about learning analytics 56 21% Conducting formal research and/or publishing work on the topic of learning 26 10% analytics Being part of the group that is leading learning analytics at their institution 24 9% Delivering training on the use of learning analytics 9 3% *denotes mutually exclusive response Question 4: In which retention applications of learning analytics are participants interested? The survey sought to explore which broad retention related applications of learning analytics participants were most interested in. Participants were asked about their level of interest in nine selected applications with their responses displayed in Figure 3. Longer bars at the top of each series indicate higher interest levels. FP:303 315 A lot of interest A little interest No Interest Identification of at-risk students with a view to staff responding to address the risk Teaching staff evaluating and improving their own teaching practice Students monitoring their own progress and identifying actions they could take Development of broad knowledge base about how effective learning can occur Program teams evaluating and improving their program curriculum Informing potential initiatives to promote student retention (e.g. mentoring) Informing design and layout of online learning sites and environments Identification of student success with a view to providing an affirmation/reward type of response Institutional management evaluating and improving teaching practice across the institution Assistance with decision making about student admissions to the institution 0 40 80 120 160 200 Figure 3: Participant levels of interest in selected potential applications of learning analytics (n varies) In interpreting the results readers need to be mindful of two things. First, participants were able to select ‘not sure’ but this is not displayed to avoid disrupting the visual flow of the chart. The ‘not sure’ option accounted for between 4% and 10% of responses for each application. Additionally, due to missing data, n varied between 247 and 252 across the applications. Question 5: How are institutions supporting learning analytics use amongst teaching staff? Subjective perceptions of needs being met Participants were asked about the extent to which they felt the institution met their needs in relation to selected institutional provisions around learning analytics. Figure 4 shows participant responses when asked to rate their institution on seven indicators. FP:304 316 Good or Very Good Fair Poor or Very Poor Not Sure Ease of learning analytics data access (n = 231) Relevance and comprehensiveness of data that I can access (n = 232) Ease of visualisation and interpretation of data (n = 231) Provision of professional development about learning analytics (n = 232) Opportunities to provide feedback about learning analytics implementation (n = 231) Provision of information about how learning analytics is being used (n = 230) Provision of information about how learning analytics use will affect me (n = 232) 0 20 40 60 80 100 120 Figure 4: Rating of institution at meeting participant needs and expectations in selected areas Professional development and training Participants were also asked whether they have attended or would attend training on five different topics: (1) Introduction to learning analytics concepts and applications; (2) Overview of institutional plan around learning analytics; (3) Accessing data; (4) Interpreting data; and, (5) Responding to data. Results can be summarised into two key points. Firstly, none of the five types of training had been attended by more than 15% of participants. Secondly, participants were interested in training. Each of the five training topics had somewhere between 83% and 86% of participants indicating they have attended, or (more commonly) would attend, training on that topic. Discussion and conclusions Prior to delving into some of the key discussion points to emerge from the data, some limitations will be considered. Firstly, the sample size means that the external validity of the data is quite limited, though the authors have taken steps to carefully describe the sampling process and demographics. Secondly, the sample size also impacted on the statistical power and the end result is a largely descriptive and exploratory survey. However, some contentions are made about the value of the study in the context of these limitations: 1. The study illustrates some issues that are important even if they are not universal; 2. The mixed-method design means that academic level survey data can be connected to other project data (e.g. West, 2015) in very specific ways (e.g. contradictions and tensions between institutional direction and teachers priorities can be considered using the different data sets) 3. Whilst this project was taking place other work was occurring (e.g. a UniSA led OLT project - Dawson et al, in press) which can help expand the breadth and range of understanding; and, 4. At this point in time, sector level research is likely to generate further questions rather than solutions to specific problems because to some extent the key challenges and issues are still being delineated. A key message, consistently reinforced, was that participants generally expressed a high level of interest in learning analytics, but their participation in learning analytics activities was limited, particularly in a collaborative way. Although 37% of participants reported using analytics to help with making decisions, very few participants engaged in frequent (e.g. weekly or fortnightly) discussion with colleagues, especially outside of other teaching staff. As learning analytics is seen as a field where collaborative use of different expertise (e.g. data science, pedagogy, discipline knowledge) is important (Ochoa et al, 2014), a lack of communication represents a barrier to progress. A similar FP:305 317 conclusion can be drawn from the data about training, in which roughly 85% of participants reported interest in attending analytics training, but very few had attended. What might be stopping teaching staff pursuing learning analytics in line with their reported interests? In addition to the data reported in the results section, the survey featured a number of open ended qualitative questions discussed elsewhere (West, 2015), however one of these questions bears mentioning here. When participants were asked what they needed to pursue their learning analytics goals there were four dominant responses - clarity, access, training and time, concisely connected by one participant who suggested: “Tell me what data is available, give me access to it, give me the time to use it and give me guidance in using it”. Even extensive discussion would likely be insufficient to explore all the ways in which these needs might be met, however such views do highlight two related tensions in the learning analytics space: 1. Distributed vs centralised locations as the source analytics of initiation and innovation 2. Homogeneity vs heterogeneity of analytics focus (i.e. how universal are the problems within an institutional context that analytics might be used to address?) Learning analytics has been a hot topic over the past couple of years and significant discussion, particularly at the institutional level, has been about integrating major data systems, with a view to large projects applying predictive analytic and other Business Intelligence techniques for example (West, 2015). Given the often centralised management and focus of these projects teaching staff might be aware of their presence, but not been provided with enough information to form a coherent understanding of what their role might be, or, how the analytic tools, techniques and problem questions of interest to them might differ from those being used centrally by institutional, managers leaders and central departments. There are potentially a number of reasons for this: • Institutional leaders see their analytics as largely about tools for institutional leaders and managers; • Institutional leaders do see a role for teaching staff but are not yet sure what that might be; • Institutional leaders envision a role for teaching staff, but promoting this a future priority; • Institutional leaders are not necessarily aware that teachers are interested in learning analytics; and/or, • Institutional leaders view the individual problems or questions that teachers might address with analytics as distinctly heterogeneous and see analytics initiatives as best driven at distributed or localized contexts. Perhaps lending weight to the final suggestion is that when participants were asked about their teaching activities outside the LMS, there was a wide array of tools and applications utilised. Whilst academics may have originally selected these tools based on their fit to identified learning requirements, many of these tools have embedded analytics functionality (e.g. Iconosquare for Instagram, Google Analytics) that can be used out of the box, or, as learning analytics researchers (e.g. Kitto, Cross, Waters & Lupton, 2015) are increasingly demonstrating, customised to higher education learning settings using open source tools. The key question about who makes decisions about what to pursue with regards to learning analytics is an important one and one that is liable to vary significantly from institution to institution. Some institutions have a clear preference (often supported by policy) that academic staff use the LMS, whereas others allow much more discretion (West, 2015). Similarly, some institutions may be focused on developing learning analytics reports and dashboards for use across the institution, whereas others may see the role of teaching staff, program coordinators and/or educational developers as working together to select and use teaching tools and technologies that meet their unique data needs. Ultimately, one of the overriding themes across the entire project was about the challenge of dealing with the variety of choices that exists in the new era of analytics. Clearly the choices about what to explore and adopt can be at-once dizzyingly complicated and numerous, yet full of possibility. This study represents an initial contribution in the context of a broader community where much is being done to collaboratively build capacity around learning analytics and support people across all levels of the sector to better understand potential uses. FP:306 318 Further Information More information about the project, including presentations and resources (e.g. framework of factors impacting on institution level learning analytics implementation and accompanying set of discussion questions) from the National Forum are available on the project website at www.letstalklearninganalytics.edu.au Acknowledgements Support for this publication has been provided by the Australian Government Office for Learning and Teaching. The views expressed in this paper do not necessarily reflect the views of the Australian Government Office for Learning and Teaching. References Arnold, K. & Pistilli, M. (2012). Course signals at Purdue: Using learning analytics to increase student success, Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ’12), Vancouver, British Columbia, Canada. Corrin, L., Kennedy, G. & Mulder, R. (2013). Enhancing learning analytics through understanding the needs of teachers. In Electric Dreams. Proceedings ascilite Sydney 2013 (pp. 201-205). Ferguson, R., (2012). Learning analytics: drivers, developments and challenges. IJTEL, 4(5/6), 304– 317. Kitto, K., Cross, S., Waters, Z. & Lupton, M. (2015). Learning Analytics beyond the LMS: The Connected Learning Analytics Toolkit. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ’15), Poughkeepsie, NY, USA. Harrison, S., Villano, R., Lynch, G., & Chen, G. (2015). Likelihood analysis of student enrollment outcomes using learning environment variables: A case study approach. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ’15), Poughkeepsie, NY, USA. Nelson, K., Clarke, J., Stoodley, I., & Creagh, T. (2014). Establishing a framework for transforming student engagement, success and retention in higher education institutions. Final Report 2014, Canberra, Australia: Australian Government Office for Learning & Teaching. Ochoa, X., Suthers, D., Verbert, K., & Duval, E. (2014). Analysis and reflections on the third Learning Analytics and Knowledge Conference (LAK 2013). Journal of Learning Analytics, 1(2), 5‐22. Tinto, V. (2009) Taking student retention seriously: rethinking the first year of university. ALTC FYE Curriculum Design Symposium, Queensland University of Technology, Brisbane, Australia. Retrieved May 8, 2015 from http://www.fyecd2009.qut.edu.au/resources/SPE_VincentTinto_5Feb09.pdf West, D. (2015). Method and findings. Proceedings Let’s Talk Learning Analytics. The National Forum for the Learning Analytics: Assisting Universities with Student Retention project. Griffith University, Queensland, Australia. Retrieved from http://www.letstalklearninganalytics.edu.au/wp- content/uploads/2015/05/Findings-Presentation-Final.pdf Willcoxson, L., Manning, M., Hibbins, R., Joy, S., Thomas, J., Girardi, A., Leask, B., Sidoryn, T., Cotter, J., Kavanagh, M., Troedson, D., & Lynch, B. (2011). The whole of university experience: Retention, attrition, learning and personal support interventions during undergraduate business studies. Surrey Hills, NSW: Australian Learning and Teaching Council. Retrieved May 10, 2015 from http://eprints.usq.edu.au/20138/ West, D., Huijser, H., Health, D., Lizzio, A., Toohey, D., & Miles, C. (2015). Higher Education Teachers’ Experiences with Learning Analytics in Relation to Student Retention. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:296-FP307). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:307 319 Exploratory and Collaborative Learning Scenarios in Virtual World using Unity-based Technology Karin Wilding Vanessa Chang Christian Gütl Graz University of Technology, Austria Curtin University, Australia Graz University of Technology, Austria This paper focuses on learning tools developed for the integration in virtual learning worlds that enable instructors to create in-world scenarios more easily. The tools were implemented in consideration of several learning concepts on exploratory, collaborative and challenge-based approaches. It elaborates on the design and development of a virtual world project on two platforms, namely Unity and Open Wonderland which is based on an Egyptian learning world. Users explore the world to find, explore and discard information. Through the process of identification and elimination a story is formed. Users can share information and collaborate with other users in- world and the tasks are supported by tools embedded in the virtual world, such as Textchat, Itemboard and Chatbot. The virtual world in Unity has addressed some of the issues raised in Open Wonderland such as the graphics enhancements, level of interactivities and lessons learned from the first prototype. Keywords: Games-based Learning, Challenge, Virtual Worlds, Exploratory, Collaboration Background There exist countless teaching methods based on various pedagogical concepts. Nonetheless, there are concepts to formulate teaching; for example, the declaration of a few core principles of how teaching can be designed by Strauss (2013). He showed that different amounts of information can be retained by students after a certain time, depending on the activity and teaching method used. This is influenced by factors such as the learning materials and activities, the age of the subjects, or the assessments used for learning. Strauss showed that active learning and completion of tasks when collaborating with one another correlate with increased retention rates. Johnson (1991) affirmed that learning activities that are designed with collaborative work is an effective way to engage students. It is well argued that active learning increased students’ knowledge, understanding, and comprehension of the subject matter (Prince, 2004). Current emphasis of learning approaches lies on technology enhanced learning (TEL), including learning management systems (LMS), personal learning environments (PLEs) and massive open online courses (MOOCs) (Taraghi, Ebner, & Schön, 2013). Computer-supported learning refers to connecting remote students as well as using technologies to improve face-to-face interactions (Balacheff, et al, 2009). It, moreover, allows students to be completely independent while highly connected with others synchronously and also able to communicate asynchronously at any time (Garrison, 2011). The interest and use of virtual worlds for educational purposes has increased in recent years (Berger, 2012; Pirker , 2013). A virtual world differs from traditional course management systems where it includes a three-dimensional graphical setting, the use of avatars to represent participants and the sense of presence with learners in the scene (Calongne, 2008). Virtual Worlds According to Kuznik (2009), virtual worlds are also known as immersive environments. According to the definition of OECD (2011, p. 184), virtual worlds are “persistent virtual environments allowing large numbers of users, who are represented by avatars, to interact in real-time over a computer network such as the Internet”. Corbit, Wofford and Kolodziej (2011, p. 159) define virtual worlds as “online 3-D multi-user, avatar-based systems that support the creation of user-generated content”. Bell (2008) takes into account several definitions that describe the basic characteristics of virtual worlds with a networked of computers and technology needed to create such worlds and the ideas of persistence and synchronous communication with people represented as avatars. FP:308 320 Virtual worlds have great potential for learning and teaching practices (Kuznik, 2009; Duncan, Miller, & Jiang, 2012). Berger (2012) stated that virtual worlds can be used as a tool for group-based learning and collaborative problem solving. Moschini’s (2010) explained that the virtual worlds can be effectively used as a communication and social tool. Virtual worlds also offer opportunities for visualisation, simulation, enhanced social networks, and shared learning experiences. The success of educational scenarios in virtual worlds depends on effective learning design, delivery, and assessment (Moschini, 2010). Logging and recording of the user’s activities can be built in for analytics. The assessed information can be analysed and used to support the users (Corbit, Wofford, & Kolodziej, 2011). The developed artefacts of learning materials can also be re-used and easily accessible to teachers (Corbit, Wofford and Kolodziej, 2011). To ensure a stable virtual world learning environment, Calongne (2008) acknowledged that designers, instructors, and IT professionals are challenged to create stimulating content and to be able to deliver virtual worlds reliably. The user interface and navigation are important, as well as the graphics that are chosen to enhance the learning environment. Gigliotti (1995) confirmed that interface, content, perception, and performance are the key factors to create an aesthetic and motivational virtual world. Immersive learning A great advantage of virtual worlds over traditional learning environments is the increased perception of immersion (Wasko, Teigland, Leidner, & Jarvenpaa, 2011) and presence, which describes the users’ feeling of being in the real setting (Gibson, 2010; Slater, 2009). Although the two concepts are closely related there are some differences, for instance Dalgarno and Lee (2010) define immersion as a measurable characteristic of the world, dependent on technical capabilities to render sensory stimuli, whereas they argue, presence is the subjective reaction of an individual to immersion. Hence, different people can experience a different level of presence but the property of immersion is the same. The level of immersion influences the acceptance of and increased motivation and commitment in a virtual world (Chen, Warden, Wen-Shung Tai, Chen, & Chao, 2011). The more immersed a user is, according to Reiners, Wood and Gregory (2014) the more the user may respond and adapt accordingly. The ability to focus in the world and the feeling of being there are important for successful engagement in virtual learning worlds (McDonald, et al., 2014). Collaborative and social learning Closely related to the feeling of presence is the individual perception of social presence (Kreijns, 2003), awareness (De Lucia, 2009; Gütl C. , 2011), or co-presence (Dalgarno & Lee, 2010) in a virtual world. All three terms refer to the feeling of “being there together with others”. Collaborative learning refers to a group of students working together in small groups to achieve a common goal. The main focus is on student interaction as opposed to solitary student work (Prince, 2004). This feeling of belonging to a social group is supported by, firstly, the use of avatars as graphical representation of the user, secondly, by providing various communication tools, such as visual channel with text and voice chat (De Lucia, 2009; Gütl, 2011). In world, collaborative learning include starting and ending a conversation, responding to prompts, sharing information, asking for help, asking questions and listening (Herrmann, 2015). The active exchange of ideas, moreover, promotes critical thinking (Gokhale, 1995). The students can have varying levels of knowledge and experience and they are responsible not only for their own learning success but also for one another. Group forming and relationship building occur through active engagement among peers, either in a face-to-face or online environment. Active, exploratory and problem-based learning According to Prince’s (2004), active learning requires students to engage in meaningful learning activities. The key factors of active learning are student activity and engagement in the learning process. Active learning refers to engaging students with different learning materials and methods, such as reading, listening, discussing concepts with peers or applying the concepts. The learning success lies within the students’ responsibility (Bonwell & Eison, 1991). If students are not actively involved in the learning process they will most likely become disengaged and distracted (McDonald, et al., 2014). Bonwell (1991) suggests different ways of promoting active learning, such as using discussions, collaborative group learning or games. Collaborative virtual worlds follow the same line of thinking by actively engaging their participants in learning activities and providing numerous FP:309 321 possibilities to collaborate and socialise (Bonwell & Eison, 1991). Moreover, they enable users to explore the world “hands-on” even if it would be too difficult or dangerous in real life (Kuznik, 2009). Thus, virtual worlds are ideally suited to explore a subject of interest. The exploratory learning concept urges learners to explore and experiment to find a path of learning that feels natural to the learner. Only then he or she can come to conclusions (Rieber, 2005). According to de Freitas (2008) virtual worlds can support many scenarios incorporating games or challenge-based learning where students can control their progress through exploratory learning experiences. Problem-based learning is an instructional method that introduces problems in the beginning to provide a motivation and context for the learning cycle (Prince, 2004). As the user is able to make choices on his or her own, and achieve personal learning goals within the environment, virtual worlds lead to greater motivation (De Lucia A. F., 2009; Gütl, 2011). In addition to active participation, game-based approaches can be used to increase the intrinsic motivation of a participant (Garris, Ahlers, & Driskell, 2002). According to Miller (n.d.) it is important to learn through a process of experimentation, trial and error, without fear of failure. Students can explore a scenario that they would not be able to in real life due to geographic, political or content-related boundaries. McDonald et al. (2014) summarise several other learning theories under the terms constructivism, social constructivism, authentic learning and reflective thinking. Constructivism places the learner at the centre of learning and allows him/her to construct and develop the knowledge, whereas social constructivism also takes the collaborative nature of learning into account. Authentic learning and reflective thinking involve problem solving and consider the complexity of the real world, as well as promote group reflection and collaborative construction of learning (McDonald, et al., 2014). These approaches show related properties as active learning, collaborative learning and problem-based learning which are, well suited for education in virtual worlds, where learning in-world is immersive and socially oriented. McDonald et al. (2014, p. 163) summarises that “when learning activities are appropriately designed, students assume an active role in learning by constructing, exploring, negotiating and reflecting on their learning within a virtual community of practice”. These articulations of how theoretical frameworks work with virtual learning worlds were considered during the development of this project. Related Work There was a big hype about virtual worlds platforms from 2003 to 2008 (de Freitas, 2008) but interest has stagnated since then (OECD, 2011). The literature agrees that the interest in virtual worlds have decreased between 2010 and 2012 and as shown in the Trough of Disillusionment of Gartner’s Hype Cycle (Steinert & Leifer, n.d.). There is, however, an increase in the use of virtual worlds as learning environments in recent years (Dawley & Dede, 2014; Duncan, Miller, & Jiang, 2012). For example, virtual worlds were used to facilitate group work as virtual class rooms; for various kinds of assessment or for bringing geographically dispersed students and educators together (McDonald, Gregory, Farley, Harlim, Sim, & & Newman, 2014). Another example is the work conducted by Ibanez et al. (2011) where situated and collaborative leaning were used in an immersion setting which resembled Madrid for foreign language learning. The 3D virtual environment is also used to teach physics (Pirker, 2013). Other showcases include the historic “Giza 3D” project from Harvard University which aims at combining Giza archives, with numerous data of the Giza pyramids near Cairo, with a realistic 3D visualisations of the site (Manuelian, 2013) or the Egyptian Oracle, a project using a 3D replica of an Egypt temple on screen and actors on and off screen (Jacobson, n.d.), or the Shrine Educational Experience that allows users to learn about the Israeli “Shrine of the Book” in a virtual world environment (Di Blas & Paolini, 2003). Development of a virtual learning world The project described in this paper is an extension of a previous prototype that was developed in Open Wonderland (OWL). The goal of this was to create virtual world environments for teachers and use the concept of exploratory and social learning in 3D virtual worlds (Tomes, 2015) to improve student learning. Objectives Included in this project is the requirement to develop a set of universally applicable learning tools that can be re-used in any virtual learning environment (VLE) to enhance the learning activities and tasks. It is intended for these tools to be applicable to various learning scenarios. Three main pedagogical objectives were determined for this learning game: (1) knowledge acquisition, (2) enhancement of the FP:310 322 conceptual understanding, and (3) measurement of the learning progress. These objectives should be facilitated by the use of certain teaching methods implemented in the VLE, and in the case of this project, the following concepts were used: (1) collaborative learning, (2) exploratory learning, and (3) games-based and challenge-based learning. Several in-world learning modules and activities were implemented based on these pedagogical concepts. The software Unity was used as the game engine. The decision to use Unity is based on the feedback received from the evaluation of Tomes’ (2015) OWL learning environment. The evaluation in OWL revealed general approval of virtual worlds for learning purposes but a number of flaws in graphics, controls and interactivities were highlighted in the game. This led to the decision to adapt the learning environment from OWL and improve the game and learning experience in Unity. The following sections will describe and compare the OWL and Unity game engines, and the new improved learning tools will be presented. Selecting a virtual world platform Although this project is an extension of the work of an earlier prototype, replicating the exact world was not possible given the different game engines. This was attributed to the fact that OWL and Unity offer different pre-installed or add-on tools that facilitate the implementation of key features. OWL provides ready to use solutions for text chat, voice chat, different kinds of panels and menus (property panel, error panel, context menu), user list, sticky note, as well as, adaptable features, such as, a whiteboard and avatar creation that were used (Tomes, 2015). OWL is built for educational and business contexts to relay key messages, and features such as collaborative tools were limited. OWL has its own advantages such as the modular style that creates extensibility and the easy drag and drop functionality makes it easy to use for non-experienced computer users. There are other useful tools that OWL offers but were not used in the scope of the game development. These include the built-in high-fidelity immersive audio capability that can be used for playback of audio tracks or communication between users, as well as, the functionality of shared applications which allows shared editing of text documents and runs Linux applications, such as Firefox or Open Office, directly in-world (Tomes, 2015). Unity, on the other hand, has a robust graphics engine platform that allows Unity to detect the best variant for the current video hardware. Unity also provides sharper 3D-objects. Unity can be used across various platform development that includes PC (Windows, Mac, Linux/ Steam OS), consoles (PlayStation, Xbox, Nintendo, Wii), mobile devices (iOS, Android, Windows Phone, Blackberry) and websites (Maratou & Michalis, 2014). Unity does not have built-in tools to support the achievement of this project’s objectives, which was the reason why OWL was chosen in the first instance for Tomes’ study (2015). However, Unity is an intuitive and has an easier to grasp game engine for beginners as compared to Unreal Engine 4 that requires programming C# and JavaScript coding skills (Masters, 2015). Unity also offers a huge asset repository with free 3D models with great graphics support for both visual and audio effect. Unity has efficient rendering and physics engine that included detailed documentation (Marsh, 2014). The next section will briefly describe the modules and features of the game and how teaching and learning methods were integrated using Unity. Story overview The Egyptian learning world as shown in Figure 1 is based around a game area (see Figure 2) where Egyptian artefacts are located. These items have pieces of information attached that form a story. The first step for students is to explore the world and find the items (Figures 3 and 4) throughout the desert area and pyramids, and this constitutes the exploratory concept of the game. It was developed as a first-person game, which refers to the student’s graphical perspective rendered from the viewpoint of the player character (as shown in Figure 3). This facilitates students’ immersion into the game. FP:311 323 Figure 1: Egyptian Virtual World Figure 2: VW Game area Figure 3: Character with item Figure 4: Items with information To make the learning more challenging, items were not just randomly placed in the world but some of the artefacts were hidden. Students were also assigned different roles which, on the one hand, gave hints about the whereabouts of the items but, on the other hand, restricted them from picking up certain items. This meant students had to gather all the information hidden with the items but depending on the role that they were assigned, they might not be able to collect all the items. Moreover, not all items in the world were part of the story; therefore, it is important to identify the artefacts through a process of investigation and elimination. These steps bring forth the important aspect of collaboration. Students had to work together or negotiate by exchanging of information and artefacts in order to master the learning tasks to finish the game. Sharing and discussing the hints provided in the role description may give them some clues with the artefact items and information that they are able to collect. The students must have the knowledge about the items that they require and are able to collect as this is an essential part of the game. The learning activities were designed for students to gain knowledge to unfold a story and be able to practise their communication and negotiation skills. The overall goal for students was to find or enquire all information necessary to understand the whole story. Revision of the acquired knowledge was assessed through a series of quiz. Thus, a vital aspect of solving a problem is for students to choose a path in order to overcome the challenge and be able to watch the story unfold slowly. The aim of the game was for students to find pieces of information and have the ability to link all the details to form a bigger picture. Exploratory module: Storyline, hints, map Games usually feature these four characteristics: they have a goal, rules, restrictions, and require acceptance of the rules by the players (Hastie, 2010). As pointed out by Hastie, the goal does not have to be winning but it relates more to a situation where players use their individual skills to reach a certain end point. Rules include the setup of the game and include definitions of what are required of and permitted to players, whereas the restrictions define what are not allowed. This definition can also be applied to this virtual learning world, as it has game-like characteristics. In this learning world the skills of each player consist of the role-specific information and pick-up restrictions. The game aims to tell students a story about a certain topic. The goal, therefore, is to gather all information necessary to understand the whole storyline. For introduction purposes, there is a beginning statement at the start of the game that teases what the story is about. It, moreover, gives the player a general idea of what he/ she is supposed to do and where to find further information. This should be enough instruction to play the game but there are several helping tools during the game. A menu in the top left corner offers settings, which include user information - referring to the role description, the player’s inventory, and buttons to access the chat, a map of the player’s environment or hints of what to do (Figures 5 and 6). FP:312 324 Figure 5: Map and Hints of the VW Figure 6: Map and Hints of inside of the Environment Pyramid Equipped with these skills and knowledge users can explore the desert area and a maze inside a pyramid to find the items. The Egyptian world gives students the opportunity to explore in a safe environment, as proposed by de Freitas (2008). The student can make choices of his/ her own (De Lucia, 2009; Gütl, 2011) and explore and experiment to find a path of learning that feels natural to the learner (Rieber, 2005). Challenge-based modules: Items, inventory and roles The main focus of the learning world lies on finding items that are hidden throughout the game environment. Attached to these items are pieces of information that form a part of the story which the game tries to tell. By providing role-dependent hints and pick-up restrictions, players have to work together and negotiate deals to gather all parts of the story. This way, students are engage in learning activities. Not exposing students to vast amounts of new information at once but letting them discover small parts supports the steady evolvement of the students’ knowledge. This way of constructing, creating and developing their knowledge and make meaning for their own learning is seen as an important pedagogical theory to engage learners in-world (McDonald, Gregory, Farley, Harlim, Sim, & & Newman, 2014). The teacher’s role as administrator of the game allows adding new items to the game, which can easily be done via an interface in the game. This facilitates easier creation and maintenance of the learning environment for teachers with little technical skills. The inventory is a feature supporting the development of each student’s story base. Each individual inventory lists all items of the game but highlights them according to the categories “picked-up already”, “not yet picked-up” and “not able to pick-up”. This distinction demonstrates students’ progress in the game. Roles were invented to create a distinction between players. The administrator can assign roles to the students. A role consists of some information, usually hints on how or where to find the items or how many there are. As each player starts with different knowledge, the game is highly dependent on the players’ ability to collaborate, share and discuss the items. Hence, they are challenged to make a decision about working together and about how much information they are willing to share. It will force them to use their communication skills to get new information in exchange for their own knowledge. Roles, moreover, restrict players from picking up any item. This is indented as another incentive to collaborate with other students. The roles and restrictions are what dDe Freitas (2008, p. 4) calls “potential for problem – or challenge-based learning” which then leads to different kinds of collaboration as suggested by Bonwell (1991). Challenging students to collaborate to master the learning goals, moreover, “promotes group reflection, multiple perspectives and collaborative construction of learning which can be enhanced by using reflection to assist students in framing and reframing the problems”, according to McDonald et al. (2014, p. 163). Collaborative modules: Chat, Chatbot, Itemboard The structure of the game encourages collaboration between the players to a point that they can only finish the game if they have worked and communicated with one another. These interactions between the students can either take place in the Textchat (see Figure 7) or with help of the Itemboard (see Figure 8). FP:313 325 The Textchat is a tool that allows for multiple students to communicate over the server in real-time. It can be accessed via a button at the top left of the screen at any point during the game. All students currently in the game can discuss their findings and questions in one chat. Unity does not provide out- of-the-box chat-modules, therefore, the conversation tool had to be programmed. The Itemboard is loosely based on the concept of a whiteboard. To prevent the exchange of off-topic information or an overcrowded board full of text, it is not possible to write random text messages on the board. Instead players can simply pin item information to the board in slots, arrange the information slots or delete them. This easy structure provides clarity and a quick overview of the information. The control is very straightforward – there are four slots with add-buttons which, when clicked, draw up a list of items in the player’s inventory to choose from. Once an item is selected it is pinned to the Itemboard. Students can rearrange the information pieces by dragging an information box to another slot. Deleting information is done by clicking the delete-button found in the top right of the each information box. If the Itemboard is full, additional board with four spots can easily be added by clicking on the extend-button found on the right side of the board. Another way to gather information, either of general or item-specific nature, is to use the Chatbot (see Figure 9). Again there are no preconfigured Chatbots offered in Unity’s feature set which is why a very simple decision-tree Chatbot was implemented. It offers several possible questions to choose from and gives the answer and a choice of follow-up questions. A help menu is also provided to further guide the user (see Figure 10). Figure 7: Textchat Figure 8: Itemboard Figure 9: Chatbot Figure 10: Help menu These tools are particularly important to the learning game as “learning is a social activity and learning cannot be uncoupled from the social and cultural context of the learner” as McDonald et al. (2014, p. 163). Due to collaboration learners are exposed to multiple perspectives and opinions (McDonald, Gregory, Farley, Harlim, Sim, & & Newman, 2014). Learning module: Quiz Once students gathered all the information necessary to form the story they can take a quiz, revising all the facts learned. On passing the quiz, the game ends but further development might be use as a starting point for another level. Passing of the quiz gives the student a sense of achievement and the teacher can assess of the knowledge base of each student. FP:314 326 All quiz questions are saved in a XML-document, a software- and hardware-independent document 1 format used for data storage. Teachers can add questions either by editing the XML-file in a standard editor or during gameplay by using the quiz button in the settings menu. There is a pool of questions for each item. When a question is needed for a quiz, the question is randomly picked from a bank of questions. Analytics Module: Logging For analytic reasons all user interactions are documented into a log file. The information would include the duration of the game, collaboration tools chosen for communication, interaction with the items, and so on. This can be useful for analytics purposes to see how students gather their information, the means of communication they prefer, how long it takes them to find items and much more. Improvements over the first prototype Based on an evaluation of the OWL virtual world (first prototype) implemented by Tomes’ (2015), the following issues were raised: • Use of dated and old-fashioned graphics • Poor controls and navigation (especially in the pyramid maze) • Lack of interaction with picked up items • Limited engagement and reward system • Itemboard has no intuitive controls, limited space and is not working as expected The re-design and development of the new world in Unity, moreover, made the following improvements concerning the collaborative tools and challenging nature of the game: In order to oblige to the third and fourth entry in the list, the challenge-based picking-up of items was introduced in order for students to revise the knowledge that they have gained as a result of completing the task of collecting a series of items. In order to pick up certain items and access new information, the players had to answer questions about an item. Thereby, students are forced to learn about the item that they have collected. They also have to consider and figure out if the items collected are part of the story, as not all items in the game area are relevant to the story. Given the negative feedback of the itemboard and the lack of space, this feature was completely revised. An evaluation of the adaptations made to the tool and the improved learning environment in Unity has yet to be done. A comparison between OWL and Unity as game-platforms is useful along with the lessons learned for the implementation of virtual learning worlds. Conclusion The virtual worlds described in this paper provided an example of how immersive learning and activity, challenge or game-based learning can be developed. These tools offer many advantages compared to conventional teaching techniques, such as exploring an environment regardless of geographic or content-related constraints, collaborating with people from around the world and offering a more immersive way of learning than ever before. The importance of adapting to new learning technologies and tools is recognised by educational researchers, practitioners, and software designers. The goal of the project was to revise and redesign educational activities and processes in an immersive, virtual learning environment that incorporates the implementation of a set of learning tools in Unity. The objectives of using the virtual world as an immersive platform is to (1) acquire knowledge (2) enhance the conceptual understanding, (3) assess student learning. In the Egyptian environment, this is done through a series of game-like elements with challenges for students to collect items and their information, assembling the story and gaining an understanding of the subject. The learning was assessed by taking a quiz at the end of the game. The objectives were facilitated by the use collaborative learning, exploratory learning, and challenge-based or game-based learning. 1 http://www.w3schools.com/xml/ 29-06-2015 FP:315 327 The collection of items in the Egyptian game area emphasized the exploratory nature of the game, while the roles, restrictions and pick-up questions presented challenges for students. As discussed, several learning concepts and skills such as exploratory, collaborative, negotiation, problem solving, and decision making have been integrated in this world. Through the process of identification, collaboration, decision making and elimination, users are able to use the collected information to form a story. This design enables each user to learn at his or her own pace and ability. On the other hand, users can also collaborate and seek assistance in-world with the use of Textchat, Itemboard and Chatbot. This virtual world design can be used to exhibit scenarios as students are actually able to explore the environment they are learning about instead of just reading or hearing about it passively. The Unity virtual world will be tested by students in Graz University of Technology and an evaluation of the environment will be available following the trial. Acknowledgements The authors gratefully acknowledge Curtin University and Graz University of Technology for hosting this research. Moreover, we would like to thank Lisa Tomes for her contribution and Johanna Pirker for her advice and technical support. References Balacheff, N., Ludvigsen, S., De Jong, T., Lazonder, A., & Barnes, S. (2009). State of the art in TEL- research. Technology-Enhanced Learning: Principles and Products. Springer Science & Business Media. Beer, M., Fasli, M., & Richards, D. (2011). Multi-Agent Systems for Education and Interactive Entertainment: Design, Use and Experience. New York: Information Science Reference (IGI Global). Bell, M. (2008). Toward a Definition of "Virtual Worlds". Journal of Virual Worlds Research: Past, Present and Future , 1 (1), 2-5. Berger, S. (2012). Virtual 3d world for physics experiments in higher education. Masterthesis. Graz University of Technology. Bonwell, C., & Eison, J. (1991). Active learning: creating excitement in the classroom. aehe-eric higher education report no. 1. ERIC Clearinghouse on Higher Education, The George Washington University. Calongne, C. M. (2008). Educational Frontiers: Learning in a Virtual World. EDUCAUSE Review , 43 (5), 36-48. Chen, J. F., Warden, C. A., Wen-Shung Tai, D., Chen, F.-S., & Chao, C.-Y. (2011). Level of abstraction and feelings of presence in virtual space: business englishnegotiation in open wonderland. 57 (3), 2126-2134. Corbit, M., Wofford, J., & Kolodziej, S. (2011). Learning in Virtual Worlds - Assessment Challenges and Opportunities. In M. W. Corbit, Serious Educational Game Assessment (pp. 159-174). SensePublishers. Dalgarno, & Lee. (2010). What are the learning affordances of 3-d virtual environments? British Journal of Educational Technology , 41 (1), 10-32. Dawley, L., & Dede, C. (2014). Situated Learning in Virtual Worlds and Immersive Simulations. In M. D. J. Michael Spector, Handbook of Research on Educational Communications and Technology (pp. 723-734). Springer. De Freitas, S. (2008). Serious virtual worlds. A scoping guide. JISC e-Learning Programme, The Joint Information Systems Committee (JISC) . De Lucia, A. F. (2009). Development and evaluation of a virtual campus on second life: the case of seconddmi. Computers & Education , 52 (1), 220-233. De Lucia, A., Francese, R., Passero, I., & Tortora, G. (2009). Development and evaluation of a virtual campus on Second Life: The case of SecondDMI. Computers & Education , 52 (1), 220–233. Di Blas, N., & Paolini, P. (2003). Museums and the Web 2003. Retrieved from The see experience: edutainment in 3d virtual worlds.: http://www.archimuse.com/mw2003/papers/diblas/diblas.html Duncan, I., Miller, A., & Jiang, S. (2012). A taxonomy of virtual worlds usage in education. British Journal of Educational Technology , 43 (6), 949-964. FP:316 328 Gütl, C. (2011). The Support of Virtual 3d Worlds for Enhancing Collaboration in Learning Settings. In F. P. Pozzi, Techniques for Fostering Collaboration in Online Learning Communities: Theoretical and Practical Perspectives. (pp. 278–299). IGI Global, Hershey. Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: a research and practice model. Simulation & Gaming , 33 (4), 441-467. Garrison, D. R. (2011). E-learning in the 21st century: a framework for research and practice. (Second ed.). London: Routledge/ Taylor & Francis. Gibson, D. (2010). Living Virtually: Researching New Worlds. International Journal of Gaming and Computer-Mediated Simulations (IJGCMS) , 2 (1), 59-61. Gigliotti, C. (1995). Aesthetics of a Virtual World. Leonardo , 28 (4), 289-295. Gokhale, A. (1995). Collaborative Learning Enhances Critical Thinking. Journal on Technology Education , 7 (1), pp. 22-30. Gregory, S., Scutter, S., Jacka, L., McDonald, M., Farley, H., & Newman, C. (2015). Barriers and Enablers to the Use of Virtual Worlds in Higher Education: An Exploration of Educator Perceptions, Attitudes and Experiences. Educational Technology & Society , 18 (1), 3-12. Hastie, P. (2010). humankinetics. Retrieved from What Characteristics define a Game: http://www.humankinetics.com/excerpts/excerpts/what-characteristics-define-a-game Herrmann, E. (2015). The 4 C’s of 21st century learning for ELLs: Collaboration. Retrieved from http://exclusive.multibriefs.com/content/the-4-cs-of-21st-century-learning-for-ells- collaboration/education Ibanez, M. B. (2011). Design and implementation of a 3d multi-user virtual world for language learning. Educational Technology & Society , 14 (4), 2-10. Jacobson, J. &. The egyptian oracle; religious, recreating egyptian ceremony in mixed reality. unpublished [online]. Johnson, D. W., Johnson, R. T., & Smith, K. A. (1991). Active learning: cooperation in the college classroom. Minnesota: Interaction Book Company. Kreijns, K. K. (2003). Identifying the pitfalls for social interaction in computer-supported collaborative learning environments: a review of the research. Computers in Human Behavior , 335-353. Kuznik, L. (2009). Learning in Virtual Worlds. US-China Education Review , 6 (9), 19-25. Manuelian, P. D. (2013). Giza 3d: digital archaeology and scholarly access to the giza pyramids: the giza project at harvard university. 2013 digital heritage international congress (digitalheritage 2013). IEEE. Maratou, V., & Michalis, X. (2014). V-ALERT - Virtual World for Awareness and Learning on Information Security. Report on 3D virtual worlds platforms and technologies. Lifelong Learning Programme, European Commission. Marsh, J. (2014). Advantages of the Unity Game Engine – The Ultimate Tool for Game Development. Retrieved from udemy: https://blog.udemy.com/unity-game-engine/ Masters, M. (2015). digital-tutors. Retrieved from Unity, Source 2, Unreal Engine 4, or CryENGINE - Which Game Engine Should I Choose?: http://blog.digitaltutors.com/unity-udk-cryengine-game- engine-choose/ McDonald, M., Gregory, S., Farley, H., Harlim, J., Sim, J., & & Newman, C. (2014). Coming of the third wave: a move toward best practice, user defined tools and mainstream integration for virtual worlds in education. In J. M.-K. B. Hegarty, Rhetoric and Reality: Critical perspectives on educational technology. (pp. 161–170). Proceedings ascilite Dunedin 2014. McDonald, M., Gregory, S., Farley, H., Harlim, J., Sim, J., & & Newman, C. (2014). Coming of the third wave: a move toward best practice, user defined tools and mainstream integration for virtual worlds in education. In J. M.-K. B. Hegarty, Rhetoric and Reality: Critical perspectives on educational technology. (pp. 161–170). Proceedings ascilite Dunedin 2014. Miller, K. (n.d.). Game based learning at curtin. Retrieved from http://aliawestbiblia.blogspot.com.au/2015/06/game-based-learning-at-curtin.html Moschini, E. (2010). The second life researcher toolkit–an exploration of inworld tools, methods and approaches for researching educational projects in second life. In Researching learning in virtual worlds (pp. 31 -51). Springer. OECD. (2011). Virtual worlds. immersive online platforms for collaboration, creativity and learning. OECD Digital Economy Papers , 184. Pirker, J. (2013). The Virtual TEAL World - An Interactive and Collaborative Virtual World Environment for Physics Education. MIT, TU Graz. Prince, M. (2004). Does active learning work? A Review of the Research. Journal of Engineering Education , 3, 223-231. FP:317 329 Reiners, T., Wood, L., & Gregory, S. (2014). Experimental study on consumer-technology supported authentic immersion in virtual environments for education and vocational training. In J. M.-K. B. Hegarty, Rhetoric and Reality: Critical perspectives on educational technology. (pp. 171–181). Proceedings ascilite Dunedin 2014. Rieber, L. P. (2005). Multimedia learning in games, simulations, and microworlds. The Cambridge handbook of multimedia learning , 549-567. Slater, M. (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. The Royal Society Publishing , 364 (1535), 3549-3557. Steinert, M., & Leifer, L. Scrutinizing Gartner's hype cycle approach. In Technology Management for Global Economic Growth (PICMET) (pp. 1-13). 2010 Proceedings of PICMET'10. IEEE. Strauss, V. (2013). Why the ’learning pyramid’ is wrong. Retrieved from Washington Post: http://www.washingtonpost.com/blogs/answer-sheet/wp/2013/03/0 Taraghi, B., Ebner, M., & Schön, S. (2013). Systeme im einsatz. wbt, lms, eportfolio-systeme, ple und andere. In L3t. lehrbuch für lernen und lehren mit technologien. Tomes, L. (2015). Exploratory and Social Learning in 3D Virtual Worlds. Graz University of Technology. Wasko, M., Teigland, R., Leidner, D., & Jarvenpaa, S. (2011). Stepping into the internet: new ventures in virtual worlds. MIS Quarterly , 35 (3), pp. 645-652. Wilding, K., Chang, V., & Gütl, C. (2015). Exploratory and Collaborative Learning Scenarios in Virtual World using Unity-based Technology. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:308- FP:318). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:318 330 Remote Access Laboratories for Preparing STEM Teachers: A Mixed Methods Study Wu Ting Peter R Albion Lindy Orwin Australian Digital Futures Institute School of Teacher Education and Early Australian Digital Futures Institute University of Southern Queensland Childhood University of Southern Queensland University of Southern Queensland Alexander Kist Andrew Maxwell Ananda Maiti School of Mechanical and Electrical School of Mechanical and Electrical Australian Digital Futures Institute Engineering Engineering University of Southern Queensland University of Southern Queensland University of Southern Queensland Bandura’s self-efficacy theory provided the conceptual framework for this mixed methods investigation of pre-service teachers’ (PSTs) self-efficacy to teach Science, Technology, Engineering and Mathematics (STEM) subjects. The Science Teaching Efficacy Belief Instrument-B (STEBI-B) was modified to create the Technology Teaching Efficacy Belief Instrument (T-TEBI). Pre-test and post-test T-TEBI scores were measured to investigate changes in PSTs’ self-efficacy to teach technology. Interviews and reflections were used to explore the reasons for changes in pre-service teachers’ self-efficacy. This paper reports results from a pilot study using an innovative Remote Access Laboratory system with PSTs. Keywords: Self-efficacy, STEM, Remote Access Laboratories (RAL) National requirement for STEM teacher preparation Achieving a productive and progressive future for Australia will require a workforce with high levels of scientific and digital literacy developed through studies of Science, Technology, Engineering and Mathematics (STEM) subjects. Among other indicators, a Queensland government report has recognized that “innovation is key to economic growth and STEM is a key driver of innovation” (DETA, 2007, p. v). The Office of the Chief Scientist (2013) has noted the importance of STEM capability as a driver for innovation and prosperity. STEM capability shortages in the national workforce have been linked to declining enrolments in STEM subjects at university; the consequence of declining interest through secondary schooling driven by too little time spent on STEM in primary school (Office of the Chief Scientist, 2013). Primary teachers’ reasons for lack of attention to science include limited exposure to science in their own education (Westerlund, Radcliffe, Smith, Lemke, & West, 2011), limited access to relevant teaching resources, and low confidence in their ability to teach science and technology effectively (Ping, Bradley, Gunderson, Ramirez, Beilock, & Levine, 2011). Australian Curriculum: Technologies The Australian Curriculum: Technologies comprises two subjects, Design and Technologies and Digital Technologies, proposed for study by all students from Foundation to Year 10 (ACARA: Australian Curriculum Assessment and Reporting Authority, 2013). Each is configured as dual strands addressing knowledge and understanding and processes and production skills within which are embedded key ideas of creating preferred futures, project management, systems thinking, design thinking, and computational thinking. For most in-service and pre-service teachers many of the elements in the technologies curriculum were not part of their own schooling or teacher preparation. They will be unsure about the relevant knowledge and skills and will lack the repertoire of teaching ideas that they have for traditional subjects. They will require time and support for preparation. Thus, successful implementation of the Australian Curriculum: Technologies will require the provision of relevant resources and attention to FP:319 331 relevant teacher development. Remote Access Laboratories (RALs) Remote Access Laboratory (RAL) are well established in universities for providing students with more flexible access to experiments, especially in electrical and computer control engineering (Maiti, Maxwell, & Kist, 2013). They have been used effectively in secondary schools (Lowe, Newcombe, & Stumpers, 2013) and may also offer benefits for primary schools through sharing of equipment that is expensive to acquire and maintain. There has been little research on RAL in teacher education (Kist, Maxwell, & Gibbings, 2012). This research is investigating the effects of RAL on the preparedness of pre-service teachers (PSTs) for STEM teaching. RALfie The Remote Access Laboratories for fun, innovation and education (RALfie) project represents a new approach to RAL. Where most RAL systems offer remote access to experiments at a central location such as a university campus, RALfie supports peer-to-peer sharing of experiments. It is creating a learning environment and the associated technical systems to allow children to create low cost RAL, using tools such as the Lego Mindstorms EV3 Programmable Brick, and share them with other learners online (Maxwell, Orwin, Kist, Maiti, Midgley, & Ting, 2013). Others can use the RAL, thus creating two types of participants: Makers and Users of RAL. In this study, PSTs participated as RAL Makers in a two hour face-to-face workshop using Lego to build an experiment and then connect it to the RALfie environment using the interface called a RALfie Box. They also connected IP cameras to the RALfie Box, allowing remote viewing of the experiment in action. A web-based interface enabled remote control. PSTs were then able to view the experiment and control it remotely. Other PSTs were later recruited to participate as Users, accessing established RALfie experiments remotely. RALfie and Teacher Preparation for STEM education Teachers may have low confidence for teaching STEM because they lack STEM experience in their own education. Although self-efficacy, the belief in personal capability to achieve specific goals (Bandura, 1977), is not identical to confidence, it is related. It affects behaviour and persistence in the face of challenges and is informed by successful experience, seeing others succeed, persuasive influences, and emotional responses. Self-efficacy for science teaching has been studied using the Science Teaching Efficacy Belief Instrument (STEBI) (Riggs & Enochs, 1990) which includes subscales to measure self-efficacy (SE) and outcome expectancy (OE). SE refers to teachers’ personal beliefs that they can teach science successfully and OE refers to teachers’ beliefs that their teaching can influence students’ achievement (Bandura, 1997). Professional training has been found to increase primary teachers’ self-efficacy for teaching science (Albion & Spence, 2013). RALfie has the potential to provide hands-on and online opportunities for PSTs to develop capability and confidence for implementing the Australian Curriculum: Technologies. Moreover, it offers resources to support related teaching after they graduate. Research approach and method This study is being conducted by a doctoral student (first author) in conjunction with the broader RALfie project (ralfie.org). The focus of this mixed methods study is on using RALfie to develop PSTs’ preparedness for teaching the technologies curriculum and other STEM subjects. Self-efficacy is related to their inner voice and their internal beliefs. A quantitative approach is insufficient to investigate the nuances of pre-service teachers’ self-efficacy and a qualitative approach is inadequate to explore the relationship between pre-service teachers’ self-efficacy for teaching STEM and engagement with RAL. Therefore, it is important to use a mixed methods approach to understand pre- service teachers’ self-efficacy to teach STEM by the use of RAL. Quantitative methods alone are unable to provide specific reasons why their self-efficacy changed. The limitations of a quantitative approach can be offset by the strengths of qualitative methods (Creswell, 2011). PSTs’ self-efficacy for teaching technologies was measured using the T-TEBI instrument which has FP:320 332 been derived from the STEBI-B (Enochs & Riggs, 1990) by adjusting the wording of items to reflect technology rather than science and, for some items, to better suit the Australian context. The 23 items comprise two subscales for efficacy expectations or self-efficacy (SE, 13 items) and outcome expectancy (OE, 10 items). T-TEBI items are presented for response on a 5-point scale, Strongly Disagree (1) to Strongly Agree (5). The T-TEBI was administered twice, before and after the PSTs have worked with RALfie activities. Quantitative data was analysed using SPSS to detect differences in pre-test and post-test measures. Efficacy expectations measure belief that a person can perform a behaviour necessary for some result and outcome expectancy is the estimation that the behaviour will produce the desired outcomes (Bandura, 1997). Selected PSTs were also interviewed to explore aspects of their experience with RALfie activities. Qualitative data was analysed thematically using the sources of self-efficacy information as a guide (Bandura, 1997). This paper reports the data from a pilot study which was conducted in 2014 to inform the main study which is being conducted in 2015. For the purposes of this paper the qualitative data have also been analysed using the conference themes, globally connected and digitally enabled. Quantitative results There were 15 participants who completed both the pre-test and post-test T-TEBI surveys. Of those, 8 students had participated in RALfie activities, including 6 students who participated in both Maker and User events and 2 who participated only in a User event by remote access. There were 7 students who had not participated in any RALfie activities. All participants were USQ pre-service teachers enrolled in a final year technology curriculum and pedagogy course. The survey was administered online twice, at the beginning and end of semester one in 2014 using LimeSurvey. The URLs were broadcast in the Learning Management System for all students enrolled in the course. Once the survey had closed, data were transferred to SPSS for analysis. The responses for each participant on each subscale (self-efficacy and outcome expectancy) were summed and divided by the number of items to yield a normalized score from 1 to 5. Figure 1 displays the differences in scores on the subscales in a bubble plot format. Filled circles represent 8 PSTs who participated in RALfie activities while the open circles represent 7 PSTs who did not participate. For both subscales there is an unanticipated decrease in scores for the RALfie user group and an increase for the non-users. Figure 1: Differences in self-efficacy and outcome expectancy scores Because the small numbers of respondents to the questionnaires did not generate sufficient data to support full statistical analysis, the responses to individual items were examined for trends that might inform the larger study. Tables 1 and 2 display the distributions of responses for both pre-test and post-test on the SE and OE subscales from the T-TEBI. The first number in each pair represents the scores for the group who participated in RALfie activities. For example, item 2 shows 8/7, meaning that 8 people from the RALfie users and 7 people from non-users group agreed or strongly agreed with the item 2. Reverse scored items are indicated by *. Table 1: T-TEBI Self-Efficacy Scores (SE) N=15 FP:321 333 *reverse score Pre-test Post-test SD/D A/SA SD/D A/SA U U 2 I will continue to find better ways to teach technology. 8/7 8/ 7 * 3 Even if I try very hard, I will not teach technology as well as I will 4/6 1/1 3/0 4/ 3/ 1/ most subjects 4 1 1 5 I am going to know the steps necessary to teach technology 1/1 7/6 3/ 5/ concepts effectively. 1 6 * 6 I am not going to be very effective in monitoring technology 3/5 4/1 1/1 4/ 1/ 3/ learning activities. 6 1 0 * 8 8/7 6/ 2/ I am going to generally teach technology ineffectively. 6 1 1 I am going to understand technology concepts well enough to be 1/1 7/6 2/ 6/ 2 effective in teaching primary technology. 1 6 * 1 I am going to find it difficult to explain to students why technology 5/3 2/4 1/0 4/ 1/ 3/ 7 learning activities work. 4 1 1 1 I am going to typically be able to answer students’ technology 2/0 2/2 4/5 1/ 3/ 3/ 8 questions 0 1 6 * 1 I wonder if I am going to have the necessary skills to teach 3/2 3/2 2/3 3/ 4/ 1/ 9 technology 4 0 3 * 2 Given a choice, I am not going to invite the principal to evaluate 6/5 1/1 0/1 5/ 0/ 2/ 1 my technology teaching. 3 3 1 * 2 When a student has difficulty understanding a technology 7/4 1/3 5/ 3/ 2 concept, I am going to be at a loss as to how to help the student 5 2 understand 2 When teaching technology, I am going to welcome student 8/7 8/ 3 questions 7 * 2 4/6 2/1 2/0 5/ 2/ 1/ I do not know what to do to turn students on to technology 4 6 1 0 The most notable changes in responses for self-efficacy items as shown in Table 1 was for items 5, 12 and 22. Those items all refer to ‘technology concepts’ and the PSTs who participated in RALfie activities recorded decreases in self-efficacy as measured by those items. Perhaps the most likely explanation is that the RALfie activities involved unfamiliar concepts and their limited exposure was not sufficient to develop confidence. On the other hand, they recorded increases for item 24, suggesting that their experience with the RALfie activities was engaging and they see the value of such activities in their own classrooms. At the same time the students who had not participated in RALfie activities recorded increases in their self-efficacy as indicated by items 17, 21 and 22, most likely resulting from their experience in the course they were studying. Table 2: T-TEBI Outcome Expectancy Scores (OE) N=15 *reverse score Pre-test Post-test SD/D SD/D A/SA A/SA U U 1 When a student does better than usual in technology, it is often 3/4 5/3 3/2 5/5 because the teacher exerted a little extra effort. 4 When the technology grades of students improve, it is often due to 2/2 6/5 1/1 7/6 their teacher having found a more effective teaching approach 7 If students are underachieving in technology, it is most likely due to 0/4 3/2 5/1 0/1 2/1 6/5 ineffective technology teaching 9 The inadequacy of a student’s technology background can be 0/3 8/4 0/1 8/6 overcome by good teaching. * 10 The low technology achievement of students can not generally be 2/3 5/3 1/1 2/2 5/3 1/2 blamed on their teachers 11 When a low-achieving child progresses in technology, it is usually 2/2 6/5 2/2 6/5 due to extra attention given by the teacher * 13 Increased effort in technology teaching produces little change in 7/7 1/0 6/7 2/0 students’ technology achievement. 14 The teacher is generally responsible for the achievement of students 2/4 6/3 0/1 2/4 6/2 in technology FP:322 334 15 Students’ achievement in technology is directly related to their 0/1 2/1 6/5 0/1 1/2 7/4 teacher’s effectiveness in technology teaching 16 If parents comment that their child is showing more interest in 0/1 3/6 5/0 0/1 4/3 4/3 technology, it is probably due to the child’s teacher Table 2 presents the patterns of responses on the outcome expectancy items. The students who participated in RALfie events mostly recorded positive values on the pre-test, leaving little scope for increases though there were some on items 4, 7 and 16. Those who did not participate in RALfie activities recorded increases on those items and also on items 1 and 9. Overall the data indicate belief that teachers can make a positive difference to learning in technology through effective teaching. Qualitative results There were two sources of qualitative data, interviews and PSTs’ reflections in their assignment work. Six participants were interviewed. Aby, Shasha, Jo and Bek participated in both Maker and User Events for 5 hours. They were mature aged PSTs in their final year of preparation as primary school teachers. Daniel and George, who participated in only the User Event for 1 hour, were mature aged PSTs in their first year of preparation as primary school teachers. Both of them had one year of study for an Engineering degree before switching to Education. Lilian was one of three PSTs who voluntarily wrote reflections about RALfie as part of their assignment. Lilian participated in the Maker Event for 2 hours. Four themes that were evident in the qualitative data are reported in this paper. Thematic analysis was used to analyse interview data (Clarke, 2013). Theme 1. RALfie broadened PSTs’ view of enacting the Technology Curriculum RALfie as an innovative technology provides both hands-on and online modes for participants to access STEM experiments. By engaging with RALfie, PSTs have a new experience of working with technology. RALfie activities are matched the requirements of the Technologies Curriculum such as “explore and use a range of digital systems with peripheral devices for different purposes, and transmit different types of data (ACTDIK007)”. RALfie broadened PSTs’ understanding of how to implement the new Technologies Curriculum in classrooms. RALfie provides opportunities for PSTs to learn to teach the Australian Curriculum: Technologies in an innovative way rather than the old-fashioned ICT style. Aby said, “Had I not known about RALfie and had the access to it in my previous courses, I would still have no idea about any other technologies that could be put into classrooms. I have never seen a Mindstorms kit ever in my prac. I probably would keep doing the same old-fashioned ICT that kind of stuff like technology in the curriculum.” RALfie is an innovative system which provides hands-on and online opportunities for teachers to integrate RALfie into their teaching practice in the classroom. RALfie broadened PST’s understanding of the new Technologies Curriculum. Instead of keeping on doing the old-fashioned ICT, RALfie offered creative ways to teach the Technologies Curriculum. Aby also commented, “Now I would be keener to use them because I have access to it before. I have experience with it…Seen that pendulum idea, wow it is pretty cool.” This response is consistent with enactive mastery experiences which is the principal source of self-efficacy information (Bandura, 1997). PST’s past successful experience working with RALfie will increase their self-efficacy to use RALfie or similar systems to implement the Technologies Curriculum in their classroom. Therefore, it is important to engage more PSTs in activities like RALfie that provide more opportunities for them to integrate technology activities for teaching the new curriculum. RALfie will make a difference for lesson planning and technology teaching in the classroom. Aby commented that “It [RALfie] will make a very big difference in regards to our lesson planning for the technology curriculum. Probably make us more innovative in how we are going to teach things.” Aby realized the significant educational purpose that RALfie can fulfill in assisting them to teach the new Curriculum. That is consistent with Daniel’s comment “It [the Pendulum activity] is just something appeal to them [school students]. It is different to learn about physics from how they normally would in the classroom”. By participating in both the Maker and User Events, Aby had more exposure to RALfie activities in both hands-on and online modes. Spending more time with RALfie is more likely to increase PSTs’ understanding of the RALfie concept, self-efficacy and improve capacity and capability to teach the Technologies Curriculum in an innovative and creative way. Although Daniel FP:323 335 participated in the User Event only for 1 hour, his engineering background probably assisted him to appreciate that RAlfie is different from the traditional way of teaching STEM. Theme 2: Globally connected With emerging technologies, the Internet has become commonplace in developed countries and is booming in developing countries. It is possible for anyone with digital skills to share knowledge and ideas and be connected online (Bell, 2010). Connectivism is focused on technology-enabled learning for the digital age (Siemens, 2005). One of the principles of connectivism is that “learning is a process of connecting specialized modes or information sources” (Siemens, 2005, p. 4). Learning is focused on connections and specialized information sets or databases that enable us to learn more are far more important than our current state of knowing. RALfie activity demonstrated that the world is globally connected. Shasha said “I think it [RALfie] can make differences in a lot of areas. I think the most important one would be just for students to learn how connected the world is. If they are in another part of the country they can control it”. The connectivity of RALfie is the most important element which makes it different in Shasha’s opinion. In the digital age, people acquire competence from forming connections (Siemens, 2005). RALfie is a new learning tool which changes the learning environment and has a great potential to impact on learning. According to Connectivism, the ability to see connections among fields, ideas, and concepts is a core skill (Siemens, 2005). Shasha has learned through perceiving connections that RALfie enabled. RALfie offers both hands-on and online environments which represent innovation and change in technology-enabled learning (Bell, 2010). Jo shared the same view with Shasha about the connectivity of RALfie and stated that “Users can go online and it can be done anywhere in Australia as long as you’ve got internet”. Furthermore, Jo explained the benefit of the connectivity that RALfie enabled. “So it is probably easier and cheaper alternatives for schools who cannot buy them. But it provides all students with the same chance for building knowledge and learning”. Jo’s vision of RALfie resonates with a major goal of RALfie: to use the potential to be globally connected to provide an equal learning opportunity for students who are disadvantaged due to their isolated location (Wu, Albion, Maxwell, Kist, Orwin, & Maiti, 2015). Students can access RALfie and learn anytime and anywhere, which could engage more children to learn STEM in the future. RALfie is globally connected which shows the future of the world. Importantly, RALfie links the real hands-on experiments to remote access. Jo stated that “the traditional ways were not that interesting whereas this User event and the Maker event are very interesting. It shows where the world is moving. The world is moving to remote access…so it connects the real world and where the world is heading into. It provides a small snapshot of what could be in the future”. Connectivity is the future which links people and resources globally regardless of location. However, if networked activities are to be substituted for hands-on activities, it is important to retain the sense of reality. The connection between hands-on experiments and remote access makes RALfie real and engaging for PSTs. Learning by making, tinkering and inventing is consistent with Piagetian Theory because hands-on activities are concrete (Martinez, 2013). Learning starts with concrete learning and proceeds to more abstract learning. The physicality of the Maker Event is concrete and the User Event is more abstract, involving aspects of computational thinking and conceptual thinking. The integration of Maker and User events is in line with people’s learning stages as described by Piaget (Piaget, 1973). Theme 3: Digitally enabled Digital skills and digital confidence are important for citizens in 21st century because they underpin the digital economy. Digital skills are fundamental to the growth and competitiveness of the economy. It is of great importance to enhance Australians’ digital skills to participate in the digital world. Digital technology also changes the way we communicate with one another, gain knowledge and discover new ideas. It also shapes teaching and learning in school subjects. Therefore, it is urgent to empower Australian students to use and access computer technology effectively to participate in Digital Economy (Office of the Chief Scientist, 2013). Correspondingly, teachers need to make the most use of technology in classroom teaching and learning to fully prepare students to participate and embrace the digital economy. FP:324 336 RALfie offers both hands-on and online opportunities for PSTs to embrace digital skills and digital confidence. Aby stated that “because the world is kind of focusing more and more on technology. It [RALfie] is heading in that direction and fostering innovation. As the technology progresses, it can progress with it”. RALfie is digital technology which addresses key skills related to computational thinking and associated concepts, such as design thinking and conceptual thinking. Computational thinking is a key concept in the new Australian Curriculum: Technologies (ACARA, 2014). Computational thinking will empower children to change the future of the world (Catlin & Woollard, 2014). The purpose and relevance of learning digital skills is very important and needs to be understood by PSTs and can be developed through the use of RAlfie. Shasha stated that “I definitely like the interactive part. As you command it, you can physically see what is happening when you are doing it”. Maker Events offer opportunities for PSTs to physically manipulate equipment and design STEM experiments which enables them to understand the relevance of the experiment. PSTs had a sense of ownership of the RALfie experiment because they designed and built them (Martinez, 2013). “I like the pendulum idea as an online activity...it was also based on science concepts of gravity. I think it is very helpful in the classroom”. Moving from hands-on RALfie to online RALfie is in line with Piaget’s theory that learning should move from concrete learning to more abstract learning (Piaget, 1973). RALfie is a vehicle to teach science and technology together, which is consistent with the integration across learning areas favoured by the Australian Curriculum: Technologies (ACARA, 2014). RALfie enables students to “Investigate the main components of common digital systems, their basic functions and interactions, and how such digital systems may connect together to form networks to transmit data” (ACTDIK014). Multi-level engagement through the use of RALfie has been identified. George commented on the online system by stating that “I like how it has experiment, system, levels and quests. Good fun stuff like that. I like interacting with a lab remotely. I feel that is exciting”. It is clear that George is engaged with RALfie cognitively by applying computational thinking skills and emotionally by feeling excited about the online system (Munns & Martin, 2005). Daniel said “You play with Lego Mindstorm kits and you are doing it remotely. For high school students it would be amazing experience just to be able to set it up and get it working and play with it.” Daniel’s view that using the online system is highly engaging to youth is in line with the experience being engaging. Both Daniel and George were engaged cognitively and emotionally with RALfie, which is helpful for them to be self-efficacious about working with technologies. Even though they participated as Users only, they can still foresee the potential excitement of hands-on RALfie activities. That is consistent with their year of Engineering study that built up their self-efficacy in use the online RALfie system and be able to understand the hands-on activities. Both Daniel and George demonstrated their agency and power to learn by the use of RALfie (Bandura, 1997). The multidimensional construct of behavioural, cognitive, agentic and emotional engagement is substantive engagement (Munns & Martin, 2005). High levels of engagement with technologies are beneficial for these PSTs to develop their capacity and capability to work effectively with digital technologies in their future classrooms. However, Bek had a different opinion on the online system. She participated in both Maker and User Event and said, “I like working with the hands-on part. I did not like the programming so much. I found that quite complicated. Physically move it and handle the stuff are lots of fun. I found it is pretty engaging.” Although she participated in both modes of RALfie, she did not enjoy the programming side of the User Event and was more engaged at the behavioural level. The cognitive activities were beyond her current abilities and she would require more time and support to learn the relevant skills. However, being able to express her preference of RALfie activities demonstrated her agentic engagement by communicating likes and dislikes (Reeve & Tseng, 2011). Additionally, she might need to be matched with Daniel and George who are more comfortable with programming and computational thinking. In that way, there will be opportunities for Bek to learn vicariously from her peers (Bandura, 1997). Theme 4. Frustration when using RALfie Frustration is defined as an emotional state “which arises when the progress a user is making towards achieving a given goal is impeded” (Gilleade & Dix, 2004, p. 228). Frustration is a negative emotion. However, frustration is sometimes deemed necessary to heighten the overall experience which can be monitored to indicate when a user is in need of support (Gilleade & Dix, 2004). Frustration can be FP:325 337 used as an indicator for change and professional training, which could assist the users to tackle situations they deem too hard to handle by themselves. Therefore, PSTs’ frustration offers a chance for the RALfie research team to provide professional training for the future users to avoid the negative emotion, which will help future users to persist through obstacles. Frustration caused by RALfie has been identified in PST’s reflection. Lilian wrote her reflection in her assignment: “At first this activity [RALfie Maker activity] was daunting and I felt overwhelmed, as I had never used this software [Lego Mindstorms] before. Although after collaboratively working through the explicit instructions with my group we were able to successfully create the car to move around its assigned network”. The lack of previous successful experience working with RALfie resulted in frustration and low self-efficacy for using it which is consistent with self-efficacy theory (Bandura, 1997). The lecturer’s clear instructions were used to scaffold and facilitate the PSTs’ learning. In order to increase PTS’s self-efficacy to use RALfie to gain knowledge and skills to teach the Technologies curriculum, it is important to provide professional training to scaffold them. By working with a group, PSTs learn from each other in a collaborative way. That is consistent with Vygotsky’s theory that interaction with peers is an effective way of developing skills (Vygotsky, 1978). Frustration, caused by the complexity or unfamiliarity with the programming aspects of RALfie, has been identified in PSTs’ interviews. Shasha commented that “I know a lot of people when we started courses related to RALfie, they were so worried that they had no experiences in ICT and technology. Something like that might really intimidate them and put them off”. RALfie as an innovative technology is foreign and new to many PSTs. The lack of previous experience of working with the tasks found within the RALfie system leads to worries and frustration which is consistent with self-efficacy theory (Bandura, 1997). From Vygotsky’s view, RALfie was perceived to be beyond the PSTs’ Zone of Proximal Development (ZPD). The ZPD has been defined as “the distance between the actual developmental level as determined by independent problem solving and the level of potential development as determined through problem solving under adult guidance, or in collaboration with more capable peers”(Vygotsky, 1978, p. 86). Frustration of PSTs can be monitored for change and used for professional training. PSTs need to have more time exposure to RALfie Maker and User Events. The RALfie activities need to be monitored to a reasonable level of difficulty to meet individual learning needs. A sequence of progressively more difficult tasks and the use of levels in the gamification system are planned for the production version of RALfie and these may mitigate some of the frustration experienced with the tasks used in the research events. This research has confirmed the need for appropriate professional training resources in RALfie that will develop PSTs digital skills. PSTs who used RALfie may have had an over-inflated view of what they knew about technology before they started the RALfie activities. Encountering RALfie in a one-off, high level Maker Event, instead of in the structured progression of activities from simple to more complex, might be far from the zone of proximal development (Vygotsky, 1978). They may have had too little contact with the system to learn enough background information for the task they encountered. One or two experiences may be inadequate to move from novice to competent Maker. The chosen activity may not be suitable for novices so choosing a simpler activity might not have the same effect. From a technical point of view, RALfie is a prototype system which will keep developing and progressing. It is important to reassure PSTs that the technology will keep improving. Future versions RALfie will be provide more support for novices and the platform will be more stable. Ensuring the participants are aware they are working with a prototype of the RALfie system and explaining how RALfie has evolved from a concept to reality could be reassuring to the PSTs. It is important so that they can believe that future RALfie activities will be more user-friendly. In this way PSTs may persist through current setbacks and use RALfie in the future. Lessons learned from pilot study The pilot study allowed for simple trials with the T-TEBI instrument derived from the STEBI-B (Enochs & Riggs, 1990) as described above. Although the small number of participants precluded statistical analysis of the data including standard checks of reliability, the pilot study provided an opportunity to test operation and usefulness of the online questionnaire and to confirm that participants were able to interpret the questions. The major study will require larger numbers of participants in both RALfie user FP:326 338 and non-user groups and efforts have been made to encourage responses by offering entry to a prize draw for those who complete both pre-test and post-test. The pilot study also provided opportunity to test interview questions and practise the techniques to be employed in conducting interviews and analysing the data. During the interviews it became apparent that the previous histories of participants as learners of science and technology in schools or beyond were significant. Those experiences influenced participants’ initial attitudes to the activities as well as the knowledge and skills they were able to bring to the activities. Interviews in the main study will be adapted to ensure that relevant information about previous experiences is collected. Techniques for managing and analysing transcribed data in Nvivo were tested and adapted. That experience will inform the processes to be used in the main study. Discussion In reviewing the results from the T-TEBI pre-test and post-test, it was evident that the scores for SE and OE were slightly decreased for PSTs who had engaged with the RALfie activities but not for those who had not engaged with RALfie. From the interviews it was found that the RALfie concept was difficult to understand for PSTs without prior background, resulting in varying degrees of frustration as they tried to complete the activities. That frustration offers at least a partial explanation for the apparent decrease in self-efficacy following participation in the RALfie activities. The primary source of self-efficacy information is success with an activity and lack of success is prone to have the inverse effect. No doubt that accounts for at least part of the reason why their self-efficacy of teaching technology concepts dropped after the involvement of RALfie. However, there was a positive shift on the item about turning learners on to technology, suggesting that despite the difficulties they found the RALfie activities engaging and saw the value of similar activities for their own future classes. That was supported by several comments in the interview data. The change in outcome expectancy was less marked and the comparatively high pre-test scores suggests that there may have been some ceiling effect in play. That is, there was little scope for increased scores. It is possible that higher levels of outcome expectancy, coupled with desire to be the sort of teacher who could make a difference, contributed to the willingness of PSTs to volunteer for the RALfie activities so that the volunteers were somewhat self-selected for higher levels of OE. The non-volunteers did show some increase in OE between pre-test and post-test. In part that will have been because their lower initial scores allowed more scope for increase. PSTs who had engineering background had more positive responses to the programming activity whereas PSTs who did not have engineering background preferred the hands-on activity rather than programming activity. PSTs’ background knowledge and experience has an impact on their self- efficacy for using the abstract programming system. In order to build up PSTs’ self-efficacy to teach STEM there would be benefit in seeking to include a wider range of activities in which they could gain positive experiences of working with STEM. Interview data also indicated that the lecturer’s explicit instructions helped PSTs to understand the RALfie concepts. That will have contributed to increased confidence for working with the RALfie activities and is consistent with the third source of self-efficacy information, verbal persuasion (Bandura, 1977). There would be value in offering PSTs additional instruction relevant to RALfie and other technology activities as a means of enhancing their self-efficacy for engaging with STEM subjects as learners and teachers. Negative changes of PSTs’ self-efficacy to teach science and technology have been identified from the pre-test and post-test T-TEBI measures. However, from interviews there are plenty of positive comments that PSTs made about RALfie. The quantitative data did not quite match with the qualitative data in the sense that PSTs feel more self-efficacious to teach the Curriculum and RALfie enabled them to learn digital skills. Self-efficacy is a specific construct (Bandura, 1997). It is important to understand PSTs’ self-efficacy to construct an experiment, connect the experiment to a server to test networks, program the interface, and remote control the experiment. PSTs’ self-efficacy of using RALfie should be directly and specifically linked to key skills used in the RALfie. However, the T-TEBI instrument alone is not good enough to show the whole picture of PSTs’ self-efficacy. Therefore, it is important to expand and enrich the T-TEBI instrument by adding specific RAL-related questions such as “I will be able to control an experiment remotely”. FP:327 339 Even though frustration is a negative emotion, there are possible benefits for PSTs experiencing some frustration in the process of learning new technologies. When PSTs have a whole and full experience of using RALfie in the future, they may shift from being frustrated to being more self- efficacious by attending more professional training and having more time exposure to RALfie. The process of developing self-efficacy is important for PSTs to be more persistent when encountering setbacks in the future as they have had an experience of moving from being frustrated to being self- efficacious. Those PSTs who have been through the process of developing self-efficacy should be more determined and resilient to tackle more difficult technologies in the future and should understand more deeply about how to develop their students’ self-efficacy for using RALfie in the classroom. Limitations of this study need to be addressed as well. The small sample size constrains the analysis of the quantitative data and does not permit generalization. In the interview process, there is the possibility of researcher bias in the process of data collection or analysis. To minimize bias, the researcher will cross check themes over time. This research has progressed to a larger trial after modification of the instruments and methods based on the findings of this pilot. Results of that study will be shared in future publications. Acknowledgement: This research was supported through the Australian Government's Collaborative Research Networks (CRN) program. References ACARA: Australian Curriculum Assessment and Reporting Authority. (2013). Draft Australian curriculum: Technologies foundation to Year 10. Sydney. Retrieved 8th August 2014, from http://www.acara.edu.au/curriculum_1/learning_areas/technologies.html Albion, P. R., & Spence, K. G. (2013). Primary Connections in a provincial Queensland school system: Relationships to science teaching self-efficacy and practices. International Journal of Environmental & Science Education, 8(3). Bandura, A. (1997). Self-efficacy: The exercise of control. New York: WH Freeman. Bell, F. (2010). Connectivism: Its place in theory-informed research and innovation in technology- enabled learning. The International Review Of Research In Open And Distributed Learning, 12(3), 98-118. Catlin, D., & Woollard, J. (2014). Educational robots and computational thinking. Paper presented at the Proceedings of 4th International Workshop Teaching Robotics, Teaching with Robotics & 5th International Conference Robotics in Education. Clarke, V. V. (2013). Teaching thematic analysis. Psychologist, 26(2), 120-123. Creswell, J. W. (2011). Designing and conducting mixed methods research (2nd ed. ed.). DETA, D. o. E. T. a. t. A. (2007). Towards a 10-year plan for science, teachnology, engineering and mathematics (STEM) educaiton and skills in Queensland. Department of Education Training and the Arts. Queensland. from http://education.qld.gov.au/projects/stemplan/docs/stem-discussion- paper.pdf Enochs, L. G., & Riggs, I. M. (1990). Further development of an elementary science teaching efficacy belief instrument: A preservice elementary scale. School Science and Mathematics, 90(8), 694- 706. doi: 10.1111/j.1949-8594.1990.tb12048.x Gilleade, K. M., & Dix, A. (2004). Using frustration in the design of adaptive videogames. Paper presented at the Proceedings of the 2004 ACM SIGCHI International Conference on Advances in computer entertainment technology. Kist, A. A., Maxwell, A. D., & Gibbings, P. D. (2012). Expanding the concept of remote access laboratories. Paper presented at the Proceedings of the 2012 ASEE Annual Conference and Exposition (ASEE 2012). Lowe, D., Newcombe, P., & Stumpers, B. (2013). Evaluation of the Use of Remote Laboratories for Secondary School Science Education. Research in Science Education, 43(3), 1197-1219. Maiti, A., Maxwell, A. D., & Kist, A. (2013). An overview of system architectures for remote laboratories. Paper presented at the Teaching, Assessment and Learning for Engineering (TALE), 2013 IEEE International Conference on. Martinez, S. L. (2013). Invent to learn : making, tinkering, and engineering in the classroom. Torrance Calif: Constructing Modern Knowledge Press. FP:328 340 Maxwell, A. D., Orwin, L., Kist, A. A., Maiti, A., Midgley, W., & Ting, W. (2013). An inverted remote laboratory-makers and gamers. Paper presented at the Proceedings of the 24th Annual Conference of the Australasian Association for Engineering Education (AaeE 2013). MCEETYA. (1999). The Adelaide declaration on national goals for schooling in the twenty-first century. Canberra: M inisterial Council on. Munns, G., & Martin, A. (2005). It’s all about MeE: A motivation and engagement framework. Paper presented at the Australian Association for Research in Education Conference, University of Western Sydney. Office of the Chief Scientist. (2013). Science, technology, engineering and mathematics in the national interest: A strategic approach. Canberra. Retrieved 8th August 2014, from http://www.chiefscientist.gov.au/2013/07/science-technology-engineering-and-mathematics- in-the-national-interest-a-strategic-approach/ Piaget, J. (1973). To Understand is to Invent: The Future of Education: Penguin Books. Ping, R. M., Bradley, C., Gunderson, E. A., Ramirez, G., Beilock, S. L., & Levine, S. C. (2011). Alleviating anxiety about spatial ability in elementary school teachers. Paper presented at the Annual Meeting of the Cognitive Science Society, Boston, MA. Reeve, J., & Tseng, C.-M. (2011). Agency as a fourth aspect of students’ engagement during learning activities. Contemporary Educational Psychology, 36(4), 257-267. Riggs, I. M., & Enochs, L. G. (1990). Toward the development of an elementary teacher's science teaching efficacy belief instrument. Science Education, 74(6), 625-637. Siemens, G. (2005). Connectivism: A learning theory for the digital age. International Journal of Instructional Technology and Distance Learning, 2(1), 3-10. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes: Harvard University Press. Westerlund, J. F., Radcliffe, R. A., Smith, D. A., Lemke, M. R., & West, S. S. (2011). Profiles of US science teacher candidates: Safeguards amidst the gathering storm. International Journal of Environmental & Science Education, 6(3). Wu, T., Albion, P., Maxwell, A., Kist, A., Orwin, L., & Maiti, A. (2015). Remote Access Laboratories for Preparing STEM Teachers: Preliminary Exploration. Paper presented at the Society for Information Technology & Teacher Education International Conference. Ting, W., Albion, P.R., Orwin, L., Kist, A., Maxwell, A., & Maiti. A. (2015). Remote Access Laboratories for Preparing STEM Teachers: A Mixed Methods Study. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:319-FP:329). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:329 341 A Mobile App in the 1st Year Uni-Life: A Pilot Study Yu Zhao Abelardo Pardo The University of Sydney School of Electrical and Information Engineering The University of Sydney The transition process that students undergo from high school to university, especially during the first year has a significant impact on their academic success. Higher education institutions try to cater for the needs of these students with a variety of initiatives. Although there are numerous resources made available in university websites, in most cases, they are underutilized. With the high adoption rate of smart phones among university students, mobile apps can be used to provide personalised support during the transition from high school to university. But, questions such as what is the truly relevant information that should be given to students, how should the information be delivered, and how should such a mobile application be designed remain unanswered. To explore these issues, we have developed a prototype mobile application called “myUniMate”. We conducted a pilot study in which 13 first year engineering students used the app for 6 weeks during a normal semester. Both qualitative and quantitative data was gathered to analyse the usability and feasibility of the app and to identify the features that were more useful. The obtained results have provided clear guidelines for the evolution of the application. Keywords: transition, mobile learning. Introduction Enhancing universal higher education provision is one of the goals for Australia in recent years (Gale and Parker 2014), and supporting students to have a smooth transition from high school to university is one of the aspects that require special attention to reach this goal (Briggs et al., 2012; Krause, 2001). Various research projects have been conducted to explore this transition process in order to support and encourage students entering higher education, (Richardson et al, 2012; Tinto, 1998; Wilcox, Winn, and Fyvie-Gauld, 2005). Results of previous studies indicate that the first year in university is one of the most important transitions in a student’s life (Richardson et al. 2012). A high- quality academic experience, adequate academic support, social involvement and peer support are factors that contribute to a successful transition (Tinto 1998; Wilcox, Winn, and Fyvie-Gauld 2005). Various approaches have been developed by researchers to support the students during the first year in university. Student peer coaches were adopted in Pitkethly & Prosser (2001), Pitkethly & Prosser (2001) and Huon & Sankey (2002) suggested that course coordinator should be assigned to work with small groups of new students particularly during the induction week. A “Transition to University” workshop was described by Peat et. al., (2000) and The University of Tasmania has deployed a “UniStart” program to nurture critical thinking and independent study skills in commencing students (Adam, Hartigan, & Brown, 2010). However, these interventions have two drawbacks. The first one is that they are not tailored to student’s needs. They are initiatives offered to all the students, but it is difficult to estimate how many of them truly need them. The second drawback is that these approaches do not scale when the number of commencing students is large due to its resource- intensive nature. With the advancements of mobile technology and the increasing adoption rate of mobile phones, these devices have become an integral part of people’s lives. Specially designed mobile applications have been created to improve students’ learning. For example, in Steel, C. (2012), mMobile apps were used to supportallow students learn to make better use of their time to learn languages (Steel, C. 2012).., while, Cheong et.al. (2012) created a framework designed based on collaborative learning theories to support collaborative learning within groups of students. Kinash et. al. (2012) also designed a study to analyse if mobile learning really “does the job” of supporting students’ learning activities. However, not much research has been done. on the analysis using mobile applications to support first year university students in their transition. FP:330 342 To evaluate the feasibility of using a mobile application in supporting university students we adopted a user centred design strategy. We designed and deployed an initial prototype of a mobile application, called “myUniMate”. A user group of 13 first year engineering students from an Australian university used the application during a regular semester. Pre- and post- questionnaires were used to gather qualitative feedback from participants. Additionally, participant interactions with the mobile application were tracked and used as quantitative data. The analysis of data gathered in this study allowed us to evaluate the merit of the prototype as a mobile application to support student transition into university. The results helped us identify a set of requirements that need to be included in the application in order to be valued by the students. In the remainder of this paper we present our findings and answer the following research questions: • What information should be given to first year university students? • How should the information be delivered to first year university students? • What are the mobile design guidelines that better support the transition of a first year university student? The “Five Senses Model” Lizzio (2006) proposed 5 areas of student needs that are relevant to their early success at university. The aim was to provide a framework that summarises ideas and practices that have been shown, either directly or indirectly, to enhance commencing students’ satisfaction, engagement and persistence in higher education. The five senses are: • Capability. The university experience is usually quite different from what a commencing student has experienced in her previous studies. Better prepared students tend to have early academic success, and are usually more satisfied with their university experience and persistent with their studies. • Connectedness. The university experience usually requires students to form new relationships with their peers or with university staff. Aside from relationships with other students, the identification or affiliation with their school or university is also important to become a successful student. • Purpose. Motivation is more effective when it is intrinsic (Deci & Ryan, 1985, 2000). A student with an intrinsic purpose of learning is more likely to choose the right degree, understand the relevance of the courses, and know how to systematically develop strengths and talents. • Resourcefulness. The ability to navigate the university system, get the help and information needed, and balance work and life, and the appropriate study commitment are all aspects that contribute to a successful university experience. • Academic culture. Students with a successful university experience usually know the value of learning, what is required for the learning process, and what is important or valued in this new culture. The details of our findings, the details of which are presented in a future section, are in agreement with this model. The “myUniMate” Mobile Application In this section we provide an overview of the application. The “myUniMate” mobile app was conceived as an aggregation of commonly acknowledged functionalities that are important to the experience of a university student such as a tightly integrated feedback loop. Android was chosen as the development platform due to the low cost of Android smartphones and relatively simpler deployment process. The mobile application was implemented asis a mobile client that communicates continuously with a server that handles data storages. Design Considerations The purpose of the study is to gather user feedback about the application, and solicit features that they would like to see included. “myUniMate” was designed in a “top-down” approach, in which functionalities were first drawn from similar studies by researchers in the project team and FP:331 343 implemented as a collection of clearly differentiated functional components. Each component provides information about one aspect of the student university experience. The first version of the application was designed with four components: Reminder, Mood & Health, Feedback, and Memo. All four components were implemented in their simplest form to let participants explore the desirable features and speed up the time to create an evolved version of the prototype. It follows a detailed description of each of the components. Reminder Module One of the most important aspects that first year university students need to understand is to manage multiple information sources by themselves. Apart from attending lectures, laboratories and tutorials, students may also work on part-time jobs or participate in other extracurricular activities. This variety of engagements can be an advantage, or a burden. There is a risk that the student feels overwhelmed by the tasks derived from these contexts. The “Reminder Component” was implemented to help students organize different tasks that they need to complete. Figure 1: The Reminder Component Figure 2: Detail Screen Figure 1 shows the appearance of this module. As it can be seen the screen shows a list of reminder items. Each item consists of a title, content, and a deadline for the task. On the left of each item, there is an icon indicating whether the task has been completed (green icon) or not (red icon). Each item is clickable, and if a user clicks on an item, she will be redirected to a screen similar to Figure 2. On this screen she can say whether the task has been completed and (optionally) make a comment. Mood & Health Module Self-perceived mood and health information has been shown by various researchers to be related to performance and wellbeing (Ryan & Deci, 2000, Huppert & So, 2013). And the support during the transition period should not only consider students’ performance in completing academic tasks, but wellbeing should also be included as a goal of the transition support. The measurement of students’ mood and health was implemented as a set of progress bars (as shown in Figure 3) because they have a straightforward interpretation. The ranges of the three progress bars go from a low magnitude to a high magnitude, but the individual wording is different for each measure. The mood scale goes from “negative” to “positive”, the energy scale from “low” to “high”, and the health scale from “very bad” to “very good”. Students report their current self-perceived mood and health by moving anchors along the progress bars and (optionally) making a comment about the values. Feedback Module An important aspect to support first year university students consists of suggesting alternatives to FP:332 344 their observed behaviour. Towards this end, self-reflection may be an effective way to foster these changes or reaffirm positive behaviour during their first year. The purpose of the feedback module is to allow students to reflect in their past performance and possibly adjust their behaviour based on what they have observed. Figure 3: The Reminder Figure 4: The Reminder Figure 5: The Reminder Component Component Component In Figure 4, the feedback component shows four measurements: • Compliance: the completion rate of tasks listed in the reminder component • Mood: the average of self-reported mood score • Energy: the average of the self-reported energy score • Health: the average of the self-reported health score. Each component is shown in a progress bar ranging from red to green. When the score of the corresponding measure is low, the progress will be positioned at the red part (left side of the bar). If the score of the measure is high, the progress will be at the green part (the right side of the bar). Memo Module This component is a conventional memo functionality that contains the aggregation of all text-based records entered through myUniMate. An example of this module is shown in Figure 5. Case Study: First Year Engineering Students The mobile application described in the previous section was deployed in a 6-week pilot study to gather feedback on the current design and solicit new functionalities from real-world users. 13 (11 male and 2 female) first year university students participated in this study. The current version of the mobile application was used as an example to stimulate participants’ imagination on thinking of novel functionalities. Interactions between participants and the mobile application were tracked and questionnaires were used to get feedback from participants. Sample To satisfy study goals, participants were required to be first-year undergraduate students using Android phones on a daily basis. We chose Android as the development platform because of the limitation of development skills of the project teamlow cost of Android mobile phones and its relatively easiersimpler deployment process. its Participants were recruited via email with a message that contained a brief description of the study and the participant requirements. After getting the approval from the course coordinator, a recruiting email was sent to each of the students enrolled in the first-year engineering course. A total of twenty students replied to this email FP:333 345 and expressed their interests in participating in this study. Fourteen of them ended up participating in the study and thirteen of them completed the entire study. Those who completed the study were compensated with a $20 gift card. Out of the 13 participants, 11 are male and 2 are female. All recruited participants were first-year undergraduate students in the Engineering Department of the university. All participants were regular Android smartphone users and had been using a smartphone for at least one year. Nine of the thirteen participants were raised in Australia, and the rest were international students. Method After recruiting all participants were required to attend a meeting (pre-questionnaire session) with the research team. Before the first questionnaire session, participants gave their consent and were informed that they can opt out at any time without affecting the relationship with the researchers or the university in which the study took place. All participants were required to use the mobile application for four weeks. During the first questionnaire session that took place on the first day of week 1, the researchers gave each participant a demonstration of the application. After the demonstration, participants were required to complete a questionnaire containing questions collecting demographic information, their experience with mobile phones and mobile applications, and their university experience. Participants were also required to complete a second questionnaire in which participants reported their expected activities or tasks that they needed to complete in the next 4 weeks. After the first questionnaire session, instructions on using “myUniMate” and tasks that they were required to complete were given to each participant through the application, and all of them were asked to use “myUniMate” for the following four weeks. The second questionnaire session took place at the end of the fourth week in which participants were required to complete two additional questionnaires about their university experience and their opinion about the application. The third questionnaire session was at the end of the sixth week, and participants were required to complete the same two questionnaires used in the second questionnaire session. Measurements The data obtained in the study can be categorised into three categories: questionnaire answers about the overall university experience, questionnaire answers regarding the mobile application, and usage data gathered through “myUniMate”. It follows a description of each of these data types: Current university experience The same sets of questions regarding participants’ current university experience were asked in all 3 questionnaire sessions to see if there were changes in students’ answers over the duration of the study. More specifically, the following questions were asked: • Since this is your first year at uni, how do you feel right now? • Are you confident? Why? Additionally, the following questions were included to obtain information about awareness of the institution: • How much do you know about university life? • What would you like to know right now? Or what information do you think would be most valuable to you? The learning aspect of their experience was captured with the following questions: • Do you know how to excel in a course? Why? • Do you think you are able to do well in a course? Why? • How do you plan to study? Why? Mobile application Since the aim of this study was to gather feedback about “myUniMate” and seek design suggestions, we asked the participants if they had used mobile applications that were designed to support their FP:334 346 study and help them adapt to a new lifestyle before they had their hands-on experience with “myUniMate”, and asked them what did they think of “myUniMate” after using it (questionnaire session 2 and 3). The questions used were: • Have you used a mobile application that aims at supporting your study? If yes, was it helpful? Why? • What features do you like best? Or what features do you think are most useful? Why? • What suggestions would you like to give on the design and implementation of myUniMate? Data from “myUniMate” This data source included the information entered by each participant through the application and additional metadata such as clicking a button, navigating to a screen, and entering some text. Study Results The questionnaire answers were analysed by coding the answers line by line. Two coding analysis were done on the data. In the first coding analysis, we tried to fit the answers into the “Five Senses Model”, if an answer does not fit in any of the 5 senses, it will be excluded from this part of the analysis. And for the second coding analysis we did not have a pre-defined coding scheme before the analysis. Instead, we identified themes from the data as we processed the answers. We used affinity diagrams (Foster, S. T., & Ganguly, K. K., 2007) to organise questionnaire answers in both coding analysis. Our major focus during the analysis was to find answers to the 3 research questions. Fitting the “Five Sense Model” • Capability. Participants generally have different capabilities, and when they were answering the questionnaires, most of them were comparing their high school experience with their current university experience. Capability 1 - “I'm enjoying it, much better then high school.” (Participant 3 – questionnaire session 1) Capability 2 - “… I felt quite stressful due to the unexpected transition from high school to university such as having to be responsible for assessments and tutorial preparations. However, I feel much more confident [now] in terms of the course load and maintaining a social life at university.” (Participant 10 – questionnaire session 3) No actual preparation for university was mentioned by participants. However, they did ask other people for information about what university life was like. Capability 3 - “Was told about it from siblings, family and friends.” (Participant 4 – questionnaire session 1) Capability 4 - “I live on campus in a college environment where making social connections and seeking academic assistance is trivial, I have positive working relationships with staff and students at the university and I am a member of the IT society.” (Participant 8 – questionnaire session 1) • Connectedness. Interactions with others (peers, relatives, university staffs) were mentioned by several participants, and the interactions were not limited with persons that they meet within the university, they also seek help or information from relatives or friends. Connectedness 1 - “…, lecturers and tutors have taught students how to succeed in a course during orientation. I've also gained some experience in the first semester.” (Participant 2 – questionnaire session 1) Connectedness 2 - “I feel that I know a fair amount - I feel settled and comfortable in my surroundings. Tips and advices from older students has [have] certainly helped.” (Participant 8 – questionnaire session 1) Connectedness 3 - “Not much but I suppose some discussion with older and experienced peers would really help with adapting to university life” (Participant 10 – questionnaire session 2) FP:335 347 • Purpose. Some participants already knew what is important to them at the time of study, which could turn into a successful university experience. Purpose 1 - “I would like to know more information on career such as internship, graduate program and also tips on work-life balance.” (Participant 2 – questionnaire session 1) Purpose 2 - “More information about unit elective choices, especially course content and what each course entails (in detail). More information about requirement for majors and enrolling in Honours, [and scholarship information]…” (Participant 8 – questionnaire session 1) Purpose 3 - “It’s my first year in this uni. Everything is exciting. Study is a little bit hard, but I want to learn the stull [stuff], which can help in the future. I am interested in electrical engineering. So it’s fine for me.” (Participant 9 – questionnaire session 1) • Resourcefulness. Universities have provided different types of information that are useful for students, however, not all students where the information is. Resourcefulness 1 - “The university provides a lot of help to students. There are plenty of resources the student can access, they can be very helpful.” (Participant 2 – questionnaire session 1) Resourcefulness 2 - “I would like to know more about options later on, i.e., postgrad research/ honours, and what steps I should take as an undergrad student to progress into honours/postgrad. I would also like to know more about what is expected in each of my current courses, e.g., assessment, required amounts of effort/ studying to achieve maximum marks.” (Participant 8 – questionnaire session 2) • Academic Culture. Most of the participants understood what they need to do in order to succeed academically. Academic 1 - “Preparing my study notes at the end of each week as this will reduce my workload as finals approach.” (Participant 5 – questionnaire session 1) Academic 2 - “Preview the lecture and tutorial, I can know what I can't understand, and I will focus on this during the class. Review the stuff after school. It can help me to enhance my knowledge. And I will know what I still don't understand. I find our or ask teachers.” (Participant 7 – questionnaire session 1) Academic 3 - “I was very unconfident as I wasn’t sure if I could pass.” (Participant 13 – questionnaire session 1) Features of first year university students and their needs Students start first year from different “levels” The first year experience may vary significantly from student to student. We have found a wide variety of perceptions of students about their first year experiences. Some students have prior information about the environment obtained from close relatives or friends. This gives them an edge over students who have no other reference about university. Capability 3, 4 illustrate this point. Some students expressed their general challenges when dealing with first year tasks, and there were participants specifically mention academic challenges (Academic 3): “I felt very lost and had trouble making friends.” (Participant 1 – questionnaire session 1) “Personally, I felt quite confused due to this new transition.” (Participant 5 – questionnaire session 1) Apart from the above two categories of students, there are also students who were passionate about what they were doing at the university, as shown in Purpose 3. Support should be provided from various angles From the point of view of a first year student, university life is not only about study. We should not only focus on the learning aspect to support student in their transition. By analysing participants’ answers to questions about their knowledge of university and learning, we found that although learning is the FP:336 348 main student concern, there are additional aspects that students would like to know, such as societies & events, information on courses, degrees, and postgraduate opportunities, career advices, time management & work-life balance etc. Purpose 1 and Purpose 2 are good examples of answers obtained about this aspect. The “emotional level” of students changes over time In this section, the three answers to the same questions are compared. Generally, most the students experience changes over time. Some students felt more confident when they had some successful experience at university, while some students understood more about what university is like. “I feel it’s really stressed, as high-school life is a bit more relaxing, but uni life needs more time onfocusing [focusing] on study.” (Participant 2 – questionnaire session 1) “Very stressful, but now I feel I’m doing better…” (Participant 2 – questionnaire session 2) “Everything seems to be better now, consider that I have found some ideas for studying and also enjoying my life, but coursework is still a bit stressed, which requires me put a lot effort into it.) (Participant 2 – questionnaire session 3) One interesting point to note is that 8 of the participants used similar texts to answer the same nd rd question in the 2 and 3 questionnaire sessions, which probably meant that there were no significant changes during this period. Students use different apps to get help Apart from the institutional mobile application (developed/sponsored by the university), mobile applications for time and task management, navigation, and instant messaging were the most used by our participants. When asked the reasons for using these applications, participants mentioned the requirement to remember and get notified of certain tasks, travel to, from, and within the university, and contact friends. Perception of the App All participants agreed that “myUniMate” was useful, because of its reminder functionality and its ability to allow a user to record her mood and health. The reason for including the “Mood &Health” component into “myUniMate” was to introduce the concept of “wellbeing” and foster its improvement throughout the first year. However, not all students understood or appreciated this idea. One participant stated that he didn’t know what it does while others felt that “it’s an interesting concept, can see why something like this would be useful for students/universities.” Different valuation for components Based on participants’ answers to questions on “myUniMate”, we found that the “Reminder” component, the “Mood & Health” component, and the “Feedback” component were consistently listed as the best features by all participants, and some participants thought the “Memo” component should be removed. Figure 6: Screen Access FP:337 349 In Figure 3, the first 4 columns show the number of screen accesses of the 4 major modules of “myUniMate”, and the last 3 columns show the number of times when participants were making comments on reminders, writing memos through the “SaySomething” button on the “Reminder” screen, and writing memos through the “Memo” screen. From Figure 3, we can see that the numbers of screen access of the “Reminder” component, “Feedback” component, and “Memo” component are significantly larger than that of the other screens, which suggests that those screens were heavily used by participants during the study. The high access rates of “Reminder” and “Feedback” are consistent with qualitative data, which means participants thought that the 2 component were useful and interesting and they used them more than the other screens. However, although participants would like to remove the “Memo” module, the module has the highest access rate. Though the “Mood & Health” component was not intensively used by participants, it is considered more interesting and useful by all participants (questionnaire answers). The remaining three columns, labelled “CommentReminder”, “SaySomething”, and “WriteMemo” are all related to information input by participants. The total number of reminders setup were 398 and at the end of the study 114 of them were completed (completion rate 28.6%). The low completion rate is acceptable, because the reminder module was not designed to encourage users to interact with the application and make comment. There is a significant difference between the “Memo” column and the “SaySomething” and “WriteMemo” columns, although participants frequently view the “Memo” screen, but they seldom write memos, and this might be the reason why participants considered the “Memo” module useless. Design Guidelines The app should include crucial aspects to students and be able to reflect student identity Participants were using multiple mobile applications, and they would like to know information about different things that related to their lives, from how to get a good grade to career advice. Therefore, the design of transition-supporting applications should take into account multiple facets about students. Those facets should all be closely related to a student’s life, and all those facets together should be unique for a specific student and able to reflect the stages of transition that students are on at a certain point. This can be accomplished by following the “5 Senses Model” (Lizzio, 2006), since it is an already established theory that identifies the key aspects to the successful transition of a first year student. The core functionalities of the app should be close to students’ everyday life Although students felt that the application was useful, it was commented by participants as “not ready for daily use”, as the current provided functionalities are limited and not appealing enough to enhance user engagement. Furthermore, in order for students to see the value of using this application, the functionalities should be directly linked to students’ everyday lives or at least focus on the core aspects of their lives. For example, an on-campus navigation system would be useful for all commencing students as looking for lecture theatres and tutorial rooms are challenging tasks for new students. Information delivered through the app should be personalised As students usually come from different background and have different abilities, the information should be tailored to suit their use. For a student who knows programming, some research project choices could be offered, while for a student who barely pass her courses, more detailed instructions on grasping the course material should be provided instead. As a student progresses though the transition, the amount of information of different aspects of her life should be adjusted as well. For example, at the beginning of the student’s degree, more information delivered should be focused on what it is like to study in a university, and later as she gets familiar with how universities work, the focus of the app should be shifted to the learning, socialising, and other aspects that she concerns. Apart from that, wellbeing should always be taken into account when trying to support students. FP:338 350 Information delivered through the app should be based on the level or “stage” or transition of the student With the passage of time, things change, so are students’ feelings about their university experience. The major consequence of the change is the change of their needs, and as a result, the support delivered through the mobile application should be updated accordingly. Limitation & Future Work The major limitation of this pilot study is its small sample size. Due to the small sample size, we were unable to statistically analyse if “myUniMate” served its purpose: supporting first year students’ transition to university. In the future, the next version of “myUniMate” could be designed and implemented based on the design guidelines proposed in the previous section. Longitudinal studies with larger sample sizes should be used to evaluate the effectivene ss of such applications. Collaboration with university-based transition-support programmes would also be a possibility for future work. Another limitation is that Conclusion To evaluate the feasibility of using a mobile application in supporting university students through their transition and solicit design requirements of such applications, an initial prototype of the mobile application was developed and deployed to thirteen first year engineering students from an Australian university during a normal semester. Through analysis of data gather in this study, we found that “myUniMate” is a good prototype as a mobile application for support student’s transition to university. However, more requirements need to be fulfilled before this type of mobile application can be accepted by students. The “Five Senses Model” could be used as a model of information that should be provided to first year university students, and information provided should not be limited to learning related information. When delivering information to students, the information should be personalised in order to suit individual student’s needs. What is more, when developing transition-supporting applications, the design guidelines presented in this paper could be considered. References Adam, A., Hartigan, C., & Brown, N. (2010). The value of an open, early academic development program to students' transition and first year experience: The UTAS UniStart program. https://fyhejournal.com/article/view/18/46 Briggs, A. R., Clark, J., & Hall, I. (2012). Building bridges: understanding student transition to university. Quality in Higher Education, 18(1), 3- 21. http://www.tandfonline.com/doi/pdf/10.1080/13538322.2011.614468 Cheong, C., Bruno, V., & Cheong, F. (2012). Designing a mobile-app-based collaborative learning system. Journal of Information Technology Education: Innovations in Practice, 11(1), 94-119. http://www.editlib.org/d/111726 Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behavior. Springer Science & Business Media. Deci, E. L., & Ryan, R. M. (2000). The" what" and" why" of goal pursuits: Human needs and the self- determination of behavior. Psychological inquiry, 11(4), 227-268. http://updatenet.eu/images/7/75/Selbstbestimmung_Deci_Ryan_2000.pdf Foster, S. T., & Ganguly, K. K. (2007). Managing quality: Integrating the supply chain (pp. 256-266). Upper Saddle River, New Jersey: Pearson Prentice Hall. Gale, T., & Parker, S. (2014). Navigating change: a typology of student transition in higher education. Studies in Higher Education, 39(5), 734-753. http://www.tandfonline.com/doi/pdf/10.1080/03075079.2012.721351 Huon, G., & Sankey, M. (2002, July). The transition to University: Understanding differences in success. In th Pacific Rim Conference on First Year in Higher Education, Christchurch NZ, July. http://www.fyhe.com.au/past_papers/papers/HuonPaper.doc Huppert, F. A., & So, T. T. (2013). Flourishing across Europe: Application of a new conceptual framework for defining well-being. Social Indicators Research, 110(3), 837-861. http://langleygroup.com.au/images/Flourishing-across-Europe_Huppert_2013-copy.pdf FP:339 351 Kinash, S., Brand, J., & Mathew, T. (2012). Challenging mobile learning discourse through research: Student perceptions of Blackboard Mobile Learn and iPads. Australasian journal of educational technology, 28(4). http://ajet.org.au/index.php/AJET/article/view/832/129 Krause, K. L. (2001). The university essay writing experience: a pathway for academic integration during transition. Higher Education Research and Development, 20(2), 147-168. http://www.tandfonline.com/doi/pdf/10.1080/07294360123586 Lizzio, A. (2006). Designing an orientation and transition strategy for commencing students. Griffith University first year experience project. https://cms- dev.itc.griffith.edu.au/__data/assets/pdf_file/0008/51875/Alfs-5-Senors-Paper-FYE-Project,- 2006.pdf Peat, M., Dalziel, J., & Grant, A. M. (2000). Enhancing the transition to university by facilitating social and study networks: results of a one-day workshop. Innovations in Education and Teaching International, 37(4), 293-303. http://www.tandfonline.com/doi/pdf/10.1080/135580000750052900 Pitkethly, A., & Prosser, M. (2001). The first year experience project: A model for university-wide change. Higher Education Research and Development, 20(2), 185-198. The first year experience project: A model for university-wide change Richardson, A., King, S., Garrett, R., & Wrench, A. (2012). Thriving or Just Surviving? Exploring student strategies for a smoother transition to university. The International Journal of the First Year in Higher Education, 3(2), 87-93. http://fyhe.com.au/past_papers/papers12/Papers/5E.pdf Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist, 55(1), 68. http://users.telenet.be/cr32258/Ryan_and_Deci_2000.pdf Steel, C. (2012). Fitting learning into life: Language students’ perspectives on benefits of using mobile apps. In ascilite (pp. 875-880). http://www.ascilite.org/conferences/Wellington12/2012/images/custom/steel,_caroline_- _fitting_learning.pdf Tinto, V. (1997). Colleges as communities: Taking research on student persistence seriously. The review of higher education, 21(2), 167-177. http://muse.jhu.edu/journals/review_of_higher_education/v021/21.2tinto.html Wilcox, P., Winn, S., & Fyvie‐Gauld, M. (2005). ‘It was nothing to do with the university, it was just the people’: the role of social support in the first-year experience of higher education. Studies in higher education, 30(6), 707-722. http://www.tandfonline.com/doi/pdf/10.1080/03075070500340036 Zhao, Y. & Pardo, A. (2015). A Mobile App in the 1st Year Uni-Life: A Pilot Study. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. FP:330-FP:340). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. FP:340 352 Learning maps: A design-based approach for capacity building in tertiary online learning and teaching Chie Adachi Mark O'Rourke Teaching Support Teaching Support Deakin University Deakin University This paper addresses the importance of creating high quality and contextualized resources for capacity building of academics for online learning and teaching. Drawing on a design-based research framework, the paper presents work-in-progress learning maps. Learning maps are an increasingly popular concept and resource among learning designers which capture and organize various theories and resources for the target learners. In a climate where the tertiary sector struggles to provide quality resources and support for teaching and learning practice, we argue that the creation and implementation of learning maps poses clear advantages and a successful model for teacher capacity building, and subsequently improves student learning. Keywords: Design-based research, Learning Map, Online Learning, Instructional Design Introduction This paper addresses current issues around the importance of creating and distributing high quality resources to assist in building teacher capacity for online learning and teaching contexts in higher education. In particular, we argue that drawing on a design based research framework will enable the interactive online resources to be underpinned by pedagogical theories which will subsequently inform teaching practice in online environments. As part of the mission to deliver improved student outcomes and build staff capacity in online learning and teaching at Deakin University, the learning map project was initiated by the central teaching and learning unit in Trimester 1, 2015. The aim of this project was to create a framework to encompass both pedagogical and technical parameters. The project delivered an effective, interactive and process-driven map which encapsulates and consolidates a diversity of resources useful for conducting assessments at Deakin University. While there are numerous theories and frameworks employed in e-learning contexts, there are difficulties for academics outside the field of education to come to terms with the application of these theories to their own disciplinary context. This is particularly true when academics are used to traditional face-to-face classroom settings, and not necessarily engaged with the discourse of e- learning. Similarly, e-learning practitioners have a need to understand the pedagogical context they are operating within. In the past the development of toolkits has been used as an effective strategy for addressing engagement with theory by offering support through careful design and prompting reflective practice (Conole, Dyke, Oliver, & Seale, 2004). In order to address this issue we propose that academics and e-learning practitioners should be supported with high quality and contextualized resources to develop their capacity. In the current climate of increased financial pressure on the university sector there is limited professional development opportunities for academics to build their skillsets, which impacts on their ability to deal with massification of student numbers, and in particular learn and implement educational technologies that may alleviate these pressures. For many academics the demands of maintaining currency in their disciplinary practice does not allow the time for gaining skills and knowledge in instructional design, e- learning and pedagogical theories in order to improve their teaching practice and address diverse learning needs of students. Coupled with varied student preparation and increased online activity impacting attendance patterns there are challenges for developing good learning and teaching st practices in the educational landscape of the 21 century. Taking such contexts into consideration, this paper presents a case study of creating capacity building resources underpinned by pedagogical frameworks, which we believe to be applicable to any other tertiary institutions. Drawing on design-based theory, we apply these frameworks in the conceptualization and creation of a learning map. Though the work presented here is based on a CP:1 353 relatively small project with primary findings only, it proposes an innovative model for the creation of contextualized interactive resources through the iterative involvement of both academics and theorists in a current learning and teaching context. Design-based research as a framework for creating resources Our approach for creating learning maps is underpinned by a design-based research framework. Design-based research has emerged and developed over the last few decades as a robust framework for not only a research methodology but also as an approach to designing technology-enhanced learning environments (Wang & Hannafin, 2005). The design-based research paradigm is described in the literature by a number of different terms including: design experiments; design research; development research; and formative research (Amiel 2008; Dede 2004; Wang & Hannafin 2005). Although each methodology has a slightly different focus, the underlying goals and approaches are the same (Amiel & Reeves, 2008). This paper applies the definition provided by Wang and Hannafin (2005) as an approach to the development of the learning map: Design-based research as a systematic but flexible methodology aimed to improve educational practices through iterative analysis, design, development, and implementation, based on collaboration among researchers and practitioners in real-world settings, and leading to contextually-sensitive design principles and theories (Wang & Hannafin, 2005, p. 7) Design-based research approach focuses on the fact that the collaborative work between practitioners (academics in our case) and researchers (instructional/educational designers) stems from the iterative process where multiple methodologies and frameworks can be applied and re- applied to generate an optimal outcome. We consider that design-based approach fits well with the concept of creating effective resources in which academics and instructional designers work closely together to better build capacity for good teaching practice. In particular, the five characteristics proposed by Wang and Hannafin (2005) provides a sound model for the creation of our learning map. The table below summarizes our approaches in relation to the five characteristics of design-based research. Table 1: Five characteristics of design-based research (Wang & Hannafin, 2005, p. 9) applied to the process of creating the learning map Characteristi Explanation of characteristic Creation of learning map c 1 Pragmatic • Design based research refines The learning map has a pedagogical focus and both theory and practice pragmatically informs practitioners about the • The value of theory is appraised improvement of their teaching. There is a shift by the extent to which principles of focus from the information conveyed to the inform and improve practice process of learning (Yelland, 2007). 2 Grounded • Design is theory-driven and The learning map is a concept founded within grounded in relevant research, the educational theory and is attributed to the theory and practice. real setting/process of leaning and teaching • Design is conducted in real- practice conducted by practitioners. This world settings and the design includes considering the contexts in which they process is embedded in, and are situated such as the physical and digital studied through, design-based space they operate in; and institutional, social research. and interactional elements (Ang et al., 2010; Laurillard, 2009; Moyle, 2010) 3 Interactive, • Designers are involved in the Instructional designers are involved in the iterative and design processes and work design and production of the learning map. flexible together with participants. The process is interactive and iterative in ways • Processes are iterative cycle of which practitioners and designers work analysis, design, together to analyse, produce and redesign the implementation, and redesign. learning map. Yelland and Tsembas (2008, p. • Initial plan is usually 107) propose that "pedagogies need to be insufficiently detailed so that reconceptualised to suit the new learning designers can make deliberate environments". Gagne (1985) presumes changes when necessary. knowledge is external and predefined, and transmitted from knowers to learners. This CP:2 354 method of instructional design is effective where content learning is fact or procedure focused. 4 Integrative • Mixed research methods are As new needs from practitioners arise, flexible used to maximize the credibility approaches for adopting various methods and of ongoing research. changes are necessary. In developing the • Methods vary during different learning map, the focus lies with both the phases as new needs and process-driven design but firmly grounded issues emerge and the focus of within sound pedagogical frameworks. This is the research evolves. representative of constructivist-oriented • Rigor is purposefully maintained learning, where the instructor guides the and discipline applied learner through dialogue, scaffolds new appropriate to the development concepts, and provides additional support for phase. learning (Jonassen, 2004). 5 Contextual • The research process, research The processes of designing, creating and findings, and changes from the improving the learning map are recorded in initial plan are documented. order to foster our approaches for future • Research results are connected capacity building of academics. This can be with the design process and the represented by instructional transaction theory setting. (Merrill, 2009) which describes a common • The content and depth of framework for specifying knowledge structure, generated design principles presentation, practice and learner guidance. varies. • Guidance for applying generated principles is needed. Learning map as a delivery mechanism for curated information Drawing on the above-mentioned design-based framework, the current paper showcases the learning map on assessment as the work-in-progress study. Various elements of assessments are addressed in this map – including discussion points for plagiarism and academic integrity. Figure 1 below illustrates the sample under discussion while acknowledging the limited display of interactive functionality of this map. Figure 1: Learning map interface As partially shown above, the learning map of assessment reveals five iterative steps involved in the pedagogical practice: 1) design, 2) build, 3) mark and provide feedback, 4) moderate and grade and CP:3 355 5) evaluate. Each step outlines short texts of what academics are expected to carry out. The links/resources embedded provide further information about the particular topics. Resources are categorized into four kinds for clarity via expandable sections: a) pedagogical, b) technical, c) institutional (e.g. Deakin University policies and resources) and d) external resources. Pedagogical resources consist of literature and latest research relevant to the outlined pedagogical concepts – both institutionally and externally sourced, while technical resources point to the how-to knowledge that academics need to know in conducting rather technical practices of teaching – e.g. building assessment tools within Deakin University’s learning management system (LMS) such as rubrics, assessment submission boxes, gradebooks. Discussion Our preliminary findings and anecdotal feedback from the practitioners reveal that the learning map is particularly beneficial and effective in the following aspects: 1. Just-in-time resource – the resources embedded within the learning map are arranged and sequenced to suit when academics need to access them. The process driven learning map guides learning and teaching practice by outlining clearly what needs to happen within the timeframe of the teaching period, allowing academics to access relevant resources at the time they are required. 2. Non-linear learning – At a macro level the learning map provides a linear structure, sequencing activities and resources in the order that teaching delivery occurs. However because learners can ‘jump’ between the embedded resources by opening the sections in which they would like to further explore, the learning map offers a non-linear interactive learning experience. 3. Aesthetic design – presenting content so it is aesthetically attractive and engaging will enhance the user experience and provide the simplicity required to enhance task completion. This can have a significant impact on cognition and learning (Heidig, Müller, & Reichelt, 2015) 4. Curated and contextualized resources – the resources provided within the learning map are varied yet contextualized to teaching and learning at Deakin University. By ensuring that the learning map is concise, information-overload for academics is avoided, and task specific information provides academics with the capacity to develop their skills as required. 5. Adaptive learning focus – the learning map offers flexible and adaptive learning paths. Information is provided in chunks and/or segments and learners can skim quickly through to discover the information required, or display the detail by expanding each section. The learning map offers different learning paths for capacity building based on the learners’ needs – either just in time or providing opportunity for further research. 6. Effective use of time – researching for good resources costs time and effort. The learning map provides currency of resources, and ensures they are pedagogically, technically and contextually appropriate. Conclusion The development of the learning map for assessment at Deakin University has provided an opportunity for academics to access current resources and theories about assessment. Firmly grounded in the design-based framework and pedagogical theories, the learning map also provides the capacity to expand teacher knowledge and skills relevant to their practice by structuring the resource in an interactive design that is process driven and aligned with trimester delivery and assessment milestones. Creating an easily accessible and re-useable resource is critical for academics trying to understand and redesign assessments in a changing higher education environment where increased student numbers and participation, issues of plagiarism, varied student preparation and an increase in online learning has significant impact. References Amiel, T., & Reeves, T. C. (2008). Design-based research and educational technology: Rethinking technology and the research agenda. Educational Technology & Society, 11:4, 29-40. Ang, C. S., Zaphiris, P., & Wilson, S. (2010). Computer Games and Sociocultural Play: An Activity Theoretical Perspective. Games and Culture, 5:4, 354-380. Conole, G., Dyke, M., Oliver, M., & Seale, J. (2004). Mapping pedagogy and tools for effective learning design. Computers & Education, 43:1, 17-33. CP:4 356 Dede, C. (2004). If design-based research is the answer, what is the question? A commentary on Collins, Joseph, and Birlaczyc; diSessa and Cobb; and Fishman, Marx, Blumenthal, Krajcik, and Soloway in the JLS Special Issue on Design-based research, Journal of the Learning Sciences, 13:1, 105-114. Gagne, R. M. (1985). The Conditions of Learning and Theory of Instruction. New York: Holt, Rinehart and Winston. Heidig, S., Müller, J., & Reichelt, M. (2015). Emotional design in multimedia learning: Differentiation on relevant design features and their effects on emotions and learning. Computers in Human Behavior, 44, 81-95. Jonassen, D. H. (2004). Learning to Solve Problems : An Instructional Design Guide. San Francisco: Pfeiffer. Laurillard, D. (2009). The pedagogical challenges to collaborative technologies. International Journal of Computer-Supported Collaborative Learning, 4:1, 5-20. Merrill, M. D. (2009). First Principles of Instruction. In C. M. Reigeluth & A. A. Carr-Chellman (Eds.), Instructional-Design Theories and Models III: Building a Common Knowledge Base (Vol. 3, pp. 41- 56). New York: Routledge. Moyle, K. (2010). Australian Education Review, Building innovation: learning with technologies. Camberwell, Australia. Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53:4, 5-23. Yelland, N. (2007). Shift to the Future: Rethinking Learning with New Technologies in Education. New York: Routledge. Yelland, N., & Tsembas, S. (2008). Elearning: Issues of pedagogy and practice in the information age. In P. Kell, W. Vialle, D. Konza, & G. Vogl (Eds.), Learning and The Learner: Exploring Learning for New Times. (pp. 95 -111). Newcastle, Australia: Faculty of Education, University of Wollongong. Adachi, C. & O’Rourke, M. (2015). Learning maps: A design-based approach for capacity building in tertiary online learning and teaching. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:1-CP:5). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:5 357 Using Learning Design to Unleash the Power of Learning Analytics Simon Paul Atkinson BPP University, United Kingdom New learning technologies require designers and faculty to take a fresh approach to the design of the learner experience. Adaptive learning, and responsive and predicative learning systems, are emerging with advances in learning analytics. This process of collecting, measuring, analysing and reporting data has the intention of optimising the student learning experience itself and/or the environment in which the experience of learning occurs. However, it is suggested here that no matter how sophisticated the learning analytics platforms, algorithms and user interfaces may become, it is the fundamentals of the learning design, exercised by individual learning designers and faculty, that will ensure that technology solutions will deliver significant and sustainable benefits. This paper argues that effective learning analytics is contingent on well structured and effectively mapped learning designs. Keywords: Learning analytics, learning design, SOLE Model, visualisation Learning Design Context To consciously misparaphrase American satirist Tom Lehrer, learning analytics are a little ‘like a sewer, you only get out of them, what you put into them’. Anyone who has ever designed a survey instrument knows that the quality of the data you get out depends on the quality of the structured instrument used to capture that data. Regardless of the excessive hyperbole around learning analytics by ill-fitted technology solution providers, the current mobile, social and saturated data environment in which we now work, live and learn (Siemens & Matheos, 2010) doubtless represents opportunities. For these PLEN to deliver on the promises of tailoring individual learning experiences, not just to competences and learning preferences but also to life contexts, we need to design learning capable of leveraging sophisticated learning analytics. We need to enable learning designers, often faculty, with the ability generate designs that are compatible with emerging analytical technologies. I think it important to differentiate fields of enquiry to be clear what is included and excluded across three definable educational realms, described as academic analytics (AA), educational data mining (EDM) and learning analytics and knowledge (LAK). Academic analytics (AA) is a field concerned primarily with organisational efficiencies derived from the intelligent use of business data in the educational context (Chacon, Spicer, & Valbuena, 2012), notably around student retention and faculty effectiveness (Campbell & Oblinger, 2007). Educational data mining (EDM) is a field heavily influenced by information science’s engagement with predictive computer based training methodology, in which large data sets are mined to identify predictive student behaviours, allowing faculty to alter course offerings or services on a cohort level. The overlaps between the fields of enquiry are malleable and many influential voices advocate for greater exchange and collaboration between them (Siemens & Baker, 2012). Why Learning Analytics Matters to Students and Learning Designers LAK is closely associated with both the AA and EDM fields but with a focus on the individual or personal learning journey. Learning analytics is the process of collecting, measuring, analysing and reporting data on the learner’s engagement with learning and, to a lesser extent, on the context of the learner, with a view to optimising both. LAK is concerned with how students develop competence and seeks to identify successful patterns of behaviour, relate that behaviour to known social variables, and identify probable future ‘optimal’ learning experiences. Data analysis, in the form of visualisations, models or maps, then supports adjustments to the learning environment or the individual learner trajectory to ensure an optimal learning opportunity. CP:6 358 LAK demonstrates a great deal of concern with semantic analysis and, increasingly, with contextual conditions that impact on the learner. There is a focus on how students develop competence, often by acknowledging the social dimensions of learning, and seeks to identify and facilitate optimal social engagement (Buckingham Shum & Ferguson, 2012). Concerned with ‘collecting traces that learners leave behind and using those traces to improve learning’, both the fields of data mining and visualisation are significant contributors to effective LAK (Duval & Verbert, 2012). Making sense of individuals’ behaviour and ‘optimising’ that behaviour within a given (possibly shifting) context against a backdrop of significant social variation, LAK is the exploration of the connections between factors. Much of the current research focus is on examining the validity of connection interrogation techniques (Guba & Lincoln, 2005). Significant learning analytics research is required around the design of learning content and its adaptivity (Vandewaetere, Vandercruysse, & Clarebout, 2012), on the responses and reaction to evolving learning spaces and roles (Atkinson, 2013), and on specific affordances within learning systems (Education Growth Advisors, 2013). However, the most evident data flowing from any learner’s engagement with a virtual learning system is likely to be in terms of intervention and adaption (does the student ask for help, does the student follow guidance) and in the field of assessment (Marinagi, Kaburlasos, & Tsoukalas, 2007; Rozali, Hassan, & Zamin, 2010; Silva & Restivo, 2012). Designing for Learning, Teaching and Analytics For any LAK system to be capable of interpreting student behaviour and acting on it, the learning activities undertaken by students need to be able to be disaggregated, mapped against desired outcomes, labelled against specific objectives and linked to specific tools and skills. Frameworks to establish quality indicators for learning analytics platforms and tools are emerging (Scheffel, Drachsler, Stoyanov, & Specht, 2014), but any LAK models rely on a measurement of engagement. However, this lacks the finesse required for students to adjust to the student’s individual context. Simply reminding a student to complete a neglected activity risks being demotivating. Much of higher education lacks alternatives. Knowing, for example, that a student watches and listens to all audio-visual elements in a course unit but neglects readings may prompt the learner to interrogate their study patterns. Are they studying on the train but find reading difficult in that context? Under these circumstances further information might be imparted through their preferred media or they might be advised to consider a text-to-speech application. Alternatively, perhaps the student could be encouraged to timetable in required reading. Regardless, learning should be presented in a context that suits the learner wherever possible. This requires learning designers, more often faculty themselves, to anticipate both the optimum media and activity mix to enable students to meet the learning outcomes prescribed and the alternatives. Articulating optimal, and alternative, pathways through learning content and activity is not as easy as it might at first appear. This challenge is evidenced by the paucity of alternative assessment provisions in most University courses (Williams, 2014). Disaggregating learning objectives into its constituent elements, activity and tools, is a precondition for a systematic presentation of alternatives. Most faculty find this challenging and toolkits serve a valuable function in reconciling practice with pedagogical theory (Conole & Fill, 2005). For students to accept these learning analytically driven pathways, I suggest that an annotated advanced organiser is the most suitable means available. Organisers ensure students have a clear idea of the learning completed and the learning required, ensuring they do not use valuable ‘working memory’ to retain syllabus structures in mind when there is no need to do so (Jong, 2010). Advanced organisers also enable students to see connections between concepts, themes or topics and develop a relational awareness not possible without such visual representations as well as supporting them in planning their workload, timing engagements and planning for activities they anticipate to be challenging (Atkinson, 2011) CP:7 359 Figure 1 – Excel view of a single topic view of the advanced organiser for the SOLE Toolkit Empowering students to see their progress and their future engagements is a fundamental part of effective design with future learning analytics in mind. The Student-Owned Learning Engagement (SOLE) model (www.solemodel.org) is a learning design model that also produces a toolkit in the form of an open and editable Excel workbook. It can allow students to use it as an advance organiser and for faculty to design and guide students through optimal pathways (Atkinson, 2011). Learning designers use the Excel workbook to design a constructively aligned course on a unit, topic or weekly basis (learning outcomes are mapped to topic level objectives), identifying activities across nine modes of engagement, and identifying the tools used. In doing so, a student sees clearly what they need to do, the suggested (but optional) sequence, the tools they should use and the mode of engagement anticipated. This generates two visual representations as pie charts, one displaying the modes of engagement and the other the tools of engagement, for each topic or weekly sheet. The intention here is to make learning a transparent process in which the learner chooses the extent to which they opt to engage, both in modes and with tools, and to make choices of future course selection based their own metacognitive development. The design is flexible enough to be implementable in the majority of VLE platforms. The mapping of intended learning outcomes (ILO) for a course or module to an individual topic or weekly objectives means that every activity that a student is encourage to engage in will be ‘traceable’ to the module ILOs. Aggregating the data from completed Excel workshops allows learning designers to identify where ‘misjudgements’ on the guidance for time allowances might be corrected, or to re-balance the tools being used to ensure modules are as inclusive as possible. Clearly the toolkit has advantages of use being based on unrestricted Excel workbooks, compatible with other spread sheet applications, but it has the disadvantage that is can be ‘broken’ if cells are over written. Aggregating data would be easier if the toolkit were also fully integrated into back end systems, and this is the focus for future research. Conclusions The implication is that each of these representational challenges, organisers, learner validated content, badges and aggregators, must also take into account the different contexts in which learners approach their learning experience. Gender, ethnicity, cultural milieu, language, will all impact on the degree to which a student wants to ‘see’ their learning journey mapped out in front of them, to have a ‘machine’ determine their next learning steps, or to be re-directed to correct an ‘error’ or deficiency in performance. We risk forgetting how fundamental assumptions about knowledge and the nature of learning underpin all our personal approaches to the learning experience; our personal epistemology matters greatly in any self-directed learning approach (Frambach, Driessen, Chan, & van der Vleuten, 2012). The advantages of representing analytical data to students is not so difficult to grasp, the challenges of doing so are significant. As learning designers, instructional designers and faculty, we must design units of learning that can be disassembled and reconstructed in meaningful ways to enable the LAK algorithms to work. CP:8 360 Experiences from early reusability projects demonstrated that learning content needed to be deliberately structured, assembled from carefully labelled parts, in such a way that the context of use could be recorded, interpreted and amended, and reuse made of all or part of the object (Churchill, 2007; Lukasiak et al., 2005; Muzio, Heins, & Mundell, 2002). The challenge for many current faculty and learning designers is that a granular model of design used by the SOLE Model relies less on raw ‘content’ than on the articulated relationships between different hermeneutical units, tools and modes of engagement within any given learning unit. The SOLE model and toolkit is not therefore simply a tool, it is a way of working. However sophisticated the learning analytics platforms, algorithms and user interfaces become in the next few years, it is the fundamentals of the learning design process which will ensure that learning providers do not need to ‘re-tool’ every 12 months. Much of the current commercial effort, informed by ‘big data’ and ‘every-click-counts’ models of Internet application development, is largely devoid of any educational understanding. Enquiries into discourse analysis, social network analysis, motivation, empathy and sentiment study, predictive modelling and visualisation, and engagement and adaptive uses of semantic content (Siemens, 2012) inform learning design itself. Grounded in meaningful pedagogical and andragogical theories of learning, these fields will ensure that technology solutions deliver significant and sustainable benefits. The SOLE model is an attempt to lay the foundations of learning designs that empower the learner with the own ability to make adjustments to their personal learning eco-system in partnership with learning analytics tools. References Atkinson, S. P. (2011). Embodied and Embedded Theory in Practice: The Student-Owned Learning-Engagement (SOLE) Model. The International Review of Research in Open and Distance Learning, 12(2), 1–18. Atkinson, S. P. (2013). Re-visioning Learning Spaces: Evolving faculty roles and emerging learning spaces. London: BPP University College. Buckingham Shum, S., & Ferguson, R. (2012). Social Learning Analytics. Educational Technology & Society, 15(3), 3–26. Campbell, J. P., & Oblinger, D. G. (2007). Academic Analytics (White Paper). EDUCAUSE. Chacon, F. J., Spicer, D., & Valbuena, A. (2012). Analytics in Support of Student Retention and Success (Research Bulletin No. 2, 2012). Louisville, CO: EDUCAUSE Center for Applied Research. Churchill, D. (2007). Towards a Useful Classification of Learning Objects. Educational Technology Research and Development, 55(5), 479–497. Conole, G., & Fill, K. (2005). A learning design toolkit to create pedagogically effective learning activities. Journal of Interactive Media in Education, 2005(1). Retrieved from http://jime.open.ac.uk/jime/article/viewArticle/2005-8/275 Duval, E., & Verbert, K. (2012). Learning Analytics. Eleed, 8(1). Retrieved from http://eleed.campussource.de/archive/8/3336 Education Growth Advisors. (2013). Learning to Adapt: Understanding the Adaptive Learning Supplier Landscape. Bill &Melinda Gates Foundation. Frambach, J. M., Driessen, E. W., Chan, L.-C., & van der Vleuten, C. P. M. (2012). Rethinking the globalisation of problem-based learning: how culture challenges self-directed learning. Medical Education, 46(8), 738–747. Guba, E. G., & Lincoln, Y. S. (2005). Paradigmatic controversies, contradictions, and emerging confluences. In N. K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research (3rd ed., Vol. 5, pp. 255–286). Thousand Oaks: Sage Publications. Jong, T. de. (2010). Cognitive load theory, educational research, and instructional design: some food for thought. Instructional Science, 38(2), 105–134. Lukasiak, J., Agostinho, S., Bennett, S., Harper, B., Lockyer, L., & Powley, B. (2005). Learning objects and learning designs: an integrated system for reusable, adaptive and shareable learning content. ALT-J, 13(2), 151–169. Marinagi, C. C., Kaburlasos, V. G., & Tsoukalas, V. T. (2007). An architecture for an adaptive assessment tool. In Frontiers In Education Conference - Global Engineering: Knowledge Without Borders, Opportunities Without Passports, 2007. FIE ’07. 37th Annual (pp. T3D–11– T3D–16). CP:9 361 Muzio, J. A., Heins, T., & Mundell, R. (2002). Experiences with reusable E-learning objects: From theory to practice. The Internet and Higher Education, 5(1), 21–34. Rozali, D. S., Hassan, M. F., & Zamin, N. (2010). A survey on adaptive qualitative assessment and dynamic questions generation approaches. In Information Technology (ITSim), 2010 International Symposium in (Vol. 3, pp. 1479–1484). Scheffel, M., Drachsler, H., Stoyanov, S., & Specht, M. (2014). Quality Indicators for Learning Analytics. Journal of Educational Technology & Society, 17(4), 117–132. Siemens, G. (2012). Learning analytics: envisioning a research discipline and a domain of practice. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 4–8). New York, NY, USA: ACM. Siemens, G., & Baker, R. S. J. (2012). Learning analytics and educational data mining: towards communication and collaboration. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 252–254). Vancouver. Siemens, G., & Matheos, K. (2010). Systemic Changes in Higher Education. In Education, 16(1). Retrieved from http://ineducation.ca/ Silva, J. F., & Restivo, F. J. (2012). An adaptive assessment system with knowledge representation and visual feedback. In 2012 15th International Conference on Interactive Collaborative Learning (ICL) (pp. 1–4). Vandewaetere, M., Vandercruysse, S., & Clarebout, G. (2012). Learners’ perceptions and illusions of adaptivity in computer-based learning environments. Educational Technology Research and Development, 60(2), 307–324. Williams, P. (2014). Squaring the circle: a new alternative to alternative-assessment. Teaching in Higher Education, 19(5), 565–577. Atkinson, S.P. (2015). Using Learning Design to Unleash the Power of Learning Analytics. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:6-CP:10). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:10 362 The future of practice-based research in educational technology: Small steps to improve generalisability of research Sakinah S. J. Alhadad Griffith University Implicit in the discourse of evidence-based practice are two fundamental concerns. One is the generalisability of research evidence where issues of external validity are integral to translation, relevance, and application in complex and multifaceted higher educational contexts. The other relates to practice-based evidence, where issues of internal validity impact on the design, interpretation, and dissemination of research. While practice-based research has an advantage in terms of high external validity, threats to internal validity can cause significant issues in terms of the subsequent inference, translation, and generalisability of findings. In educational technology, evaluation and research of e-learning in higher education is conducted by both practitioners and academics, each contributing different pieces of the puzzle towards a better understanding of the learning processes in complex real world settings. In this paper, I propose small, practical steps towards improving the generalisability of practice-based research. Keywords: Practice-based research, evaluation research, research methods, validity, measurement, generalisability The future of practice-based research: An evolving journey This paper is a product of reflections precipitated by my relatively recent move into an area of work that is at the nexus of research and practice. Due to my background as an educator and researcher in the psychological science domain, the challenges I currently face at this nexus trigger a self-motivated need to enhance the voice and value of practice-based research, specifically in the applicability and dissemination of education research in general (both basic and applied). As I begin the journey into an authentic understanding of fit-for-purpose practitioner research, these are my preliminary thoughts towards enhancing the generalisability of practice-based research. The principles of the suggestions made here are design- and measurement-agnostic, in the hope that practice-based researchers will be able to apply these to their contexts, whether they are involved in design research, focus group evaluation, or learning analytics research. The purpose of this paper is not to simply share my exploratory intellectual journey in this space, but to engage peers (both juniors and seniors) and leaders in an effort to contribute towards the enhancement of educational research in the broader community of others also living in and around the research-practice nexus. Practice-based research as educational technology research Evidence-based practice operates at two fundamental levels: the first is to use existing evidence and apply it to the practice; the second is to establish evidence where gaps exist in the current evidence base, or where existing evidence may be questionable, weak, or uncertain (Davies, 1999). In this paper, I focus on the latter. Practice-based research in educational technology (analogous to ‘evaluation research’ see Philips, Kennedy, & McNaught, 2011), provides macro- and micro- level views of teaching and learning-related phenomenon in dynamic and complex settings, which can in- turn provide a very rich source of directions for hypotheses for more controlled empirical research. The synergistic effect of both controlled empirical research and practice-based research increases the likelihood of studying the educational phenomenon more authentically. In this paper, I make some practical suggestions to optimise the balance of internal and external validity with the aim of enhancing the generalisability of this research (replicability/applicability in practice, in different contexts and populations of research findings in this domain). Other pertinent issues related to practice-based research are out of the scope of this paper and are covered elsewhere (for comprehensive guidelines and discourse on evaluation research at each stage of the e-learning CP:11 363 lifecycle, see Phillips, et. al, 2011; for critical discourse on the role of theory in evaluation research, see Hannon & Al-Mahmood, 2014, and Phillips, Kennedy, & McNaught, 2012). What do issues of internal and external validity mean for practice-based research? Campbell (1957) introduced the concept of internal and external validity as a means to evaluate the value of experimental designs in social settings. Internal and external validity in research design are often conflicting ideals. Optimising a research design for high internal validity incurs trade-offs for external validity, and vice versa. For any given research design, resources are limited, and this makes it is very difficult to identify and measure all variables that may be influencing the observed effect in any given research scenario. The question is, how do we make design decisions to optimise internal and external validity, and on what do we place emphasis when considering the purpose of our research? Broadly, external validity addresses the question of whether a particular finding is generalisable across a variety of contexts, settings, persons, and times. External validity in Campbell’s (1957) original definition referred to the generalisability of the studied effect (and of its underlying processes) across different participants, settings, and research methods. This was later distinguished into: (1) ecological validity, or the degree to which the research design replicates the actual occurrence of the scenario/circumstance in naturalistic settings; and (2) relevance of generalisability, or the degree to which the research findings can be generalised across different participant populations, contexts, and other related settings (Brewer, 2000). A research design with high external validity necessarily closely resembles or ideally replicates the authentic experience in the authentic setting. A benefit of having a design with high external validity is that any research finding can be seen to be generalisable to a real context. To give an example, observational research in real-world settings gathers genuine data on observable behaviours, and thus can be argued to be representative of real-world behaviours. Such a research approach has strength in identifying existing naturalistic relationships. However, optimising a research design for maximum external validity limits the degree to which genuine cause-and-effect relationships can be identified, owing to the impossibility of measuring the influence of the large number of variables in the naturalistic setting that may be influencing the target phenomenon. Internal validity refers to the extent to which we can accurately infer or conclude that the independent variable (or predictor) produced the observed effect on the dependent variable (or criterion) (Campbell, 1957). That is, internal validity is the degree to which we have confidence that a true causal relationship exists. In experimental research (often lab-based), ascertaining whether the observed effect (as measured by the dependent variable), is truly caused by or predicted by the independent variable is relatively less challenging than if it were tested in an applied environment. Determining confidently that the observed effect is solely a function of the one independent variable is dependent on controlling other potential influencing factors (i.e. extraneous variables: variables other than the one being investigated). In the controlled experimental context it is possible, to an extent, to keep the influence of extraneous variables (particularly ones that impact on the relationship systematically, named ‘confounds’) constant, consequently eliminating any differential influence these variables may have across the various levels of the independent variable. This ability to tightly control variables to optimise internal validity is the strength of lab-based research, but this approach is very difficult to implement in natural settings such as those typical in educational technology research. Further, when intending to optimise experimental control over extraneous variables it is necessary to recognise that variables are not equal – some variables are easier to control than others. Designing experiments that generalise within and across contexts involves a complex interplay of internal and external validity. Below I suggest ways to improve internal of external validity of practice-based research, and to increase the understanding of this psychometric property in research within complex environments. The way forward: Practical suggestions to enhance generalisability These strategies are suggested with the aim to increase the translation, utility, and application of the research outside of the research context, and to enhance the efficiency of the evaluative design process within context. 1. Nuance your evaluation research questions to increase the understanding of why or how the causal effect or relationship works in context CP:12 364 1.1. Consider factoring in control or comparison groups. In the academic capacity development domain, a frequently identified issue when evaluating academics’ practice-based classroom research is that claims of efficacy of intervention or effects studied often exclude an appropriate control condition or comparison group. (e.g., Benassi et al., 2014). Being thorough in identifying drivers of an effect requires appropriate controls in place to enable more accurate inferences or conclusions to be drawn. For example, in order to infer whether learning strategy A (e.g., structured reflective practice) was effective, having a control condition B such as, in this case, ‘unstructured reflective practice’, allows the researcher to more accurately infer that the pattern of findings was not merely due of the act of reflective practice, but because strategy A was structured for optimal reflection for the task or goal. Often in practice-based research, it may be either impossible or unethical to have the ideal control group. In this case, a strategy the researcher could adopt is to statistically control the measurable confounds or influencing variables (within reason). By quantifying variables or factors that may have influenced the observed effects, internal validity is enhanced as it now creates new plausible hypotheses. Using the example above, if having a control condition was not logistically possible, one may be able to quantify the amount of engagement with the act of structured reflection, and thus be able to quantify whether the extent of engagement with the reflection task impacts systematically on the learning outcome. Another example might be to measure related psychological constructs as covariates in the model (also see point 1.2 below); whether or not the student is a deep or surface learner may impact on the magnitude of effect observed as a whole group, so quantifying this will enable the researcher to understand more deeply the mechanisms behind successful adoption of this task, and how to further improve the design and application subsequently. 1.2 Use theory as a means to frame research design. Theories or models used as a research design framework can really enrich practice-based research design and hypotheses (see Figure 1 for example; see also Lizzio, Wilson, & Simons, 2002; Kember, McNaught, Chong, Lam, & Cheng, 2010). Mook (1983) purported that the component of an experiment that increases its generalisability capital is the theoretical process or understanding that accrues from the study. In the Figure 1 example, I suggest ways to use an example of an omnibus model, the 3P model (Biggs, 1989; 1993), to nuance your research question to enhance internal validity and generalisability of the research. The 3P model comprises three main components representing an integrated system of student learning: Presage, or pre-existing, (relatively) stable student characteristics that relate to learning, and to the instructional context; Process, the underlying factors related to the process of the learning task itself; and Product, the learning-related outcomes. Where research designs begin with the Presage and Product stages, a way to nuance the research question further is to look at the moderating or mediating effect of Process factors. For example, if the product/intervention leads to enhanced academic achievement, does this pattern of results change as a function of whether the students are high or low on self-efficacy? This nuanced question allows for more efficient refinement of either the design or the investigated intervention/product as a result of clearer understanding of why the effect is occurring. For the practitioner, this also provides an actionable strategy for design improvement in the classroom or curriculum design. This iterative cycle of the design and practice-based research process is indicated by the green arrow in Figure 1 (bottom panel). CP:13 365 Figure 1. Adapted from Biggs (1989). Top panel: Ways to enhance the 3P model as a framework for practice-based research. Bottom panel: * “Engagement with technology” here, can be either the Process (in this case, it can also act as a covariate) or Product (outcome), depending on the research question. Each factor can also be measured in multiple ways; carefully assess your environment as to what is available and possible to measure. For example, the uptake of technology may be measured through learning analytics (use/do not use, frequency of use, pattern of use over time). As a process, the critical design thinking may be centred around the questions you could ask – what data sources do you have access to; what is pragmatically measurable? For example, the question can change to: does the impact of technology use on learning change as a function of the frequency of use? As such, “engagement with technology” is now a covariate in this model. 2. Consider moving beyond student perceptions: convergent measures The likelihood of accurately measuring student perceptions as they relate to the intended effect or construct depends on the research question. If, for example, you are interested in assessing the effectiveness of an educational technology in terms of usability, asking students for their perceptions of their own attitudes and beliefs is appropriate, and is likely to be an accurate representation of the true effect (Note: for a good primer resource on survey/questionnaire development, see DeVellis, 2012). However, if you are interested in effectiveness or learning-related outcomes, asking students for their perceptions will give a false impression of the intervention’s or product’s effectiveness (Phillips et al., 2011). Findings pointing to similar conclusions are plentiful in the cognitive science literature – when students are asked to judge their own level of learning during study or in a test, students tend to misjudge their actual learning performance (e.g., Asher & Bjork, 2005; Castel, McCabe, & Roediger, 2007). Including learning-related process measures such as study strategies, engagement in formative assessment as aligned with learning design, or proxies of engagement or effort such as various sources of learning analytics (see Lockyer, Heathcote, & Dawson, 2013) will enable deeper explanations and inference of the learning phenomenon studied in context. In Figure 1 (Bottom panel) I suggest ways to move beyond student perceptions in evaluating the effectiveness of a product/intervention/design by including other dependent measures to be converged with measures of perceptions to increase our understanding of the studied phenomenon. Use of self-report is beneficial here if the aim is to assess attitudes or perceptions such as ease of use, satisfaction with use of technology for learning, and perceived development of skills and learning. CP:14 366 3. Consider the research questions you want answered and how to statistically test them before collecting data. Practical research is difficult to initiate in the first instance so opportunities to gather data need to be exploited. Consider how you would answer your research questions with statistics, and iteratively evaluate the levels of measurements of your dependent variables to ensure you have optimally designed your measures to answer your questions. Further, where quantifiable, report effect sizes as complementary to quantitative statistics. The size of the magnitude of the findings in the research may be reported if the learning-related dependent variables are measured on a quantitative scale. This allows for standardised comparison of the observed effect in terms of its quantified magnitude across studies (see for Cohen, 1992 for a primer; Cumming & Finch, 2001, or Cumming, 2012 for more depth; and Hattie, 2009 for discussion specific to quantification of learning measures). 4. Communication or dissemination: Report important information on contextual variables The overarching principle in enhancing generalisability of practice-based research studies is in the communication of the research findings and the details that facilitate generalisability. Be cognisant of, and acknowledge context specificity of the findings. Explicitly address the external and local realities in communication and dissemination of practice-based research. (Green, 2008). Important information on contextual settings such as representativeness of sample, reach, implementation methods, and other pertinent variables would help readers in assessing more accurately the applicability of the study results to their own context (Glasgow et al., 2006). Conclusion In this paper, I offer small, practical ways to optimise the balance of internal and external validity to facilitate the design, dissemination, and applicability of practice-based research. The utility of these suggestions are meant to be paradigm-agnostic, however the goodness of fit will often be less than ideal, as various factors in the multifaceted, dynamic, complex environments of practice-based researchers will interact differently. In the interpretation and application of research evidence, these principles are equally important in maintaining an appropriate level of skepticism and in establishing the quality and accuracy of inference to future research and practice, such that one is able to prevent acceptance and replication of poorly tested interventions or research. The strategies recommended in this paper are an effort to contribute towards the enhancement of educational research in the broader community of others also living in and around the research-practice nexus. References Asher, K. & Bjork, R. A. (2005). Illusions of competence in monitoring one’s own knowledge during study. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(2), 187-194. Benassi, V. A. Tappin, E. M., Overson, C. E., Lee, M. J., O’Brien, E. J. White, B. P., et al. (2014). Applying the science of learning: The cognition toolbox. In V. A. Benassi, C. E., Overson, & C. M. Hakala (Eds.), Applying the Science of Learning in education: Infusing psychological science into curriculum (pp194-205). Biggs, J.B. (1989). Approaches to the enhancement of tertiary teaching. Higher Education Research and Development, 8, 7–25. Biggs, J. B. (1993). From theory to practice: A cognitive systems approach, Higher Education Research & Development, 12(1), 73-85. Brewer, M. B. (2000). Research design and issues of validity. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 3-16). New York, US: Cambridge University Press. Campbell, D. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54 (4), 297-312. Castel, A. D., McCabe, D. P., & Roediger, H. L. (2007). Illusions of competence and overestimations of associative memory for identical items: Evidence from judgments of learning and encoding fluency. Psychonomic Bulletin & Review, 14, 107-111. Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. CP:15 367 Cumming, G. & Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Mesaurement, 61, 530-572. Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta- analysis. New York: Routledge. Davies, P. (1999). What is evidence-based education? British Journal of Educational Studies, 47 (2), 108-121. DeVellis, R. F. (2012). Scale development: Theory and applications. California: Thousand Oak. Glasgow, R.E., Green, L. W., Klesges, L. M., Abrams, D. B., Fisher, E. B., Goldstein, M. G., et. al. (2006). External validity: We need to do more. Annals of Behavioral Medicine, 31 (2), 105-108. Green, L. W. (2008). Making research relevant: If it is an evidence based-practice, where’s the practice based evidence? Family Practice, 25 (supp), i20-i24. Hannon, J., & Al-Mahmood, R. (2014). The place of theory in educational technology research. In B. Hegarty, J. McDonald, & S.K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014 (pp. 745-750). Hattie, J. A. C. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. London, UK: Routledge. Kember, D., McNaught, C., Chong, F. C. Y., Lam, P., & Cheng, K. F. (2010). Understanding the ways in which design features of educational websites impact upon student learning outcomes in blended learning environments. Computers & Education, 55, 1183-1192. Lizzio, A., Wilson, K., & Simons, R. (2002). University students’ perception of the learning environment and academic outcomes: Implications for theory and practice. Studies in Higher Education, 27(1), 27-52. Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57 (10), 1439-1459. Mook (1983). In defense of external validity. American Psychologist, 38(4), 379-387. Phillips, R. A., McNaught, C. & Kennedy, G. (2011). Evaluating e-learning: Guiding research and practice. New York and London: Routledge Phillips, R., Kennedy, G. & McNaught, C. (2012). The role of theory in learning technology evaluation research. Australasian Journal of Educational Technology, 28(7), 1103- 1118. http://www.ascilite.org.au/ajet/ajet28/phillips.html Alhadad, S.S.J. (2015). The future of practice-based research in educational technology: Small steps to improve generalisability of research. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:11-CP:16). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:16 368 Features of an online English language testing interface Zakiya Al Nadabi Language Centre, Sultan Qaboos University, Oman School of Education, University of Queensland, Australia This paper describes an online English language proficiency testing platform that uses Moodle- hosted selected and open response questions along with other useful features. These features include enhanced test security settings aided by the Safe Exam Browser; an embedded MP3 player for listening skills; and a split screen mode for reading tests. The paper highlights significant elements of this particular approach to testing as they apply to formal high-stakes e-exams (testing of learning) and for continuous assessment (testing for learning). Snapshots of sample online test materials illustrate these features. Issues of concern in the field of web-based, computer-assisted assessment will be discussed in light of experience gained from a recent pilot study in which this interface was used in a series of mock exams in 2015. Keywords: Moodle, web-based testing interface, technology-enhanced language assessment A Moodle-hosted testing interface Moodle (https://moodle.org/) is a course management system that is used by educators worldwide for teaching and learning purposes. The Moodle quiz feature is one important component of the teaching and learning process where quizzes and tests are set up using this quiz tool. Given the rapid developments Moodle has gone through in the past few years from one version to another, the Moodle quiz feature has been improved to accommodate a variety of educators’ testing needs. A number of plug-ins have also been developed for this purpose. An enhanced Moodle-hosted testing interface was developed by the researcher by combining several pre-existing technologies to provide a set of features targeting the testing of English language proficiency skills. The context is the Language Centre at Sultan Qaboos University in Oman, where Moodle is the university-wide e-learning platform. Moodle 1.9 was the version available at this institution at the time of the study and so it was used in the creation of the interface. However, the principles outlined here are equally applicable to more recent versions of Moodle. To reflect a testing philosophy embracing the need to create a better testing experience using Moodle as an e- assessment tool, the researcher embarked upon a research study to develop and trial a Moodle- hosted testing interface. The researcher made use of the Moodle 1.9 quiz feature and applied some other features to create this better testing experience for its users. The following sections describe the features and the rationale for applying each with references to the use of the interface features to trial mock exams in a recent 2015 pilot study. Illustrative snapshots of these features will be provided. Enhanced test security settings aided by Safe Exam Browser First of all, the most important feature of the interface is applying enhanced test security settings. The standard settings on the Moodle platform allow designers to create password-protected tests. This limits access to individuals or classes with knowledge of a common password (for example; the password can be displayed at the front of the room once all candidates are seated in the exam room). These tests can also be timed and a time remaining count-down timer can be displayed to each examinee. The number of attempts allowed for each test can also be set. However, heightened test security can be accomplished by using Moodle in conjunction with a security browser called Safe Exam Browser (version 2.0.3). This browser is an open source application that allows access to other computer functions and web resources during online exams to be controlled or prevented. When using the Safe Exam Browser, the test is seen in a full screen mode limiting web navigation and access to unauthorized internet resources. The computer operating system shortcuts and functions such as right-click to copy or print screen to take snapshots of exams, task manager or program switcher to control access to the operating system are disabled to prevent cheating during the exam. [See Safe Exam Browser (2015) for details on this browser]. The traditional approach of supervising CP:17 369 the exam to prevent cheating is still recommended when using this type of computerized exams. As Coy (2013) and Myrick (2010) recommend, for high-stakes tests, such security measures provided by Moodle settings should be combined with test proctoring or invigilation to achieve "high security" (Coy, 2013, p.59). Thoughtfully, applying the enhanced test security settings described above can be a very important precaution in the testing process in order to ensure that technology-related issues do not affect examinees’ test performance. Measurement error variance in the test scores might be attributed to the presence of construct-irrelevant (Brown, 2005; Davies, 1999; Standards for educational and psychological testing, 1999) technology-relevant issues that can threaten reliability and validity of the inferences of web-based assessment (Fulcher, 2003). From this stand point, the use of enhanced test security settings aided by the Safe Exam Browser can limit the effect of construct-irrelevant technology-relevant sources of measurement error leading to a better testing experience where examinees’ cheating behavior is monitored much more closely. Figure 1: e-exam with Safe Exam Browser As can be seen in Figure 1, no navigation elements are allowed. The way this browser was used in the study was that computers were set up to run the browser and then examinees were logged into the browser with a special log-in password. The browser was configured to open the SQU Moodle 1.9 e-learning platform automatically with no other websites permitted. Examinees were then logged into the Moodle course where the exam was located at which point they were able to start the exam. If they attempt to access the exam from a regular browser such as Internet Explorer or Chrome, they receive a message informing them that it can only be accessed using the Safe Exam Browser. Using the standard Moodle quiz security settings aided by the Safe Exam Browser and exam invigilation can, therefore, put much more enhanced security measures into operation. Matbury’s MP3 player for listening tests Since language tests usually include listening test components, finding a suitable mechanism to allow for control playback of audio during an e-exam is important. Typically, examinees are provided with recordings to respond to relevant exam questions. The researcher decided to use Matbury’s MP3 player for listening tests (Figure 2), which is a Flash MP3 audio player suitable for playing listening test audio recordings embedded in Moodle. CP:18 370 Figure 2: Embedded Matbury’s MP3 player for listening tests The way the player works is that it controls the number of times examinees can listen to test audio recordings and no playback, forward, pause, or stop functions can be controlled by examinees after hitting the play button. Test designers wishing to use the player for listening tests may refer to Matbury (2010a, 2010b) for instructions. To embed this player into the Moodle quiz in the study, the MP3 player file and audio files for listening tests were first stored in the Moodle course files. These files URLS were then linked to in the listening test Moodle Description questions by editing the player’s HTML code to load audio files within the listening test. As mentioned before, technology-relevant issues can be a major source of measurement errors caused by the fact that examinees’ test performance gets affected by the presence of technology- related problems (Fulcher, 2003). In the case of the Matbury’s MP3 player for listening tests, the player is intended to be used in the online testing interface to ensure that all examinees are exposed to the listening materials in a fair way. Fairness can be achieved here by enforcing a standard technological practice in exposing examinees to the same listening input so that everybody listens to the audio recordings the same number of times (e.g. once or twice). Another fair technology-related practice is not to permit pause, stop, or play backward or forward options for all examinees so that they cannot control the listening input. Once all examinees are fairly treated in such a web-based testing environment using standard practices assisted by the use of Matbury’s MP3 player, issues related to differences in the way technology is manipulated by examinees can be within the control of testers. To meet test fairness (Chapelle & Douglas, 2006; Fulcher, 2003; Kunnan, 2004) principles, such a player should be used to fairly expose examinees to the same listening materials in a similar manner leading to more reliable and valid test-based inferences and decisions. Of course, if it turns out that major issues in the use of such a player make it difficult to meet such test fairness goals, reliability and validity may still be questioned. The argument made in this paper is based on one of the exam trials carried out in this study in which the listening audio file did not load at all in one of the classes taking an exam due to a software-related update problem, namely Adobe Flash Player. The lesson learned from exam trials is that it is absolutely necessary to check all testing equipment for such online testing before going ahead with official exams upon which examinees’ futures are determined. The same can definitely be said about any other technology-enhanced web-based testing tool. A split screen mode for reading tests Part of the pilot study was the involvement of expert judges working as English language teachers and testers in the study context in a judgmental validation session to review an initial prototype of the Moodle-hosted online testing interface. A major feedback point was that it is preferable that examinees view reading tests and questions side by side in some kind of split screen mode. The researcher took this valuable viewpoint into consideration and started exploring ways to create a split screen mode for reading tests using the Moodle quiz feature. Eventually, it was possible for the researcher to work with Moodle to create this split screen mode from scratch. The way this was done was by creating a table with two cells, one for the reading text and another for the embedded questions using the embedded, multi-choice, cloze question type on Moodle 1.9. The final view shows the reading text on the left side of the screen and the questions on the right side. A code for a vertical scrolling bar to access the entire reading text was embedded in the HTML text-editor of Moodle CP:19 371 embedded question type with assistance from James Scully, the Language Centre’s E-Learning Coordinator. Figure 3: Split screen mode snapshot Using this split screen mode feature (Figure 3), examinees can scroll up and down to read the reading text paragraphs and simultaneously access the relevant questions on the same page. This is contrary to many paper-based exams in which examinees have to flip papers to connect the entire test questions and the reading text. It is assumed here that presenting the reading test materials in a split screen interface should increase concentration during exam-taking as it can reduce split attention and cognitive load demands caused by the way sources of information or testing materials are presented to examinees. Split attention principle as part of Sweller’s (1994) cognitive load theory (Ayres & Sweller, 2005) might help explain the assumed benefit of a split screen interface. The restructuring of the test format using the split screen interface can be of such benefit to examinees since it is assumed that it can reduce the need for accessing multiple sources of information within the test (i.e. reading passage and subsequent questions). This can best be explained by the split attention effect that examinees using multiple sources of information might encounter compared to others using integrated information (i.e. restructured test format using the split screen interface). By using the split screen mode for reading tests, the negative effect of the need for split attention and the subsequent increased cognitive load can be eliminated. In the pilot study feedback, examinees expressed their satisfaction with the split screen mode for the reading tests pointing to a much more positive testing experience than the paper-based exams. Remarks and conclusion This paper describes a Moodle-hosted online testing interface and its features. It does not describe the full detailed results of the pilot study trials of these features, but it does describe the application of these features and the rationale for their use for language testing purposes. Precautions should be exercised whether these features are applied for formal high-stakes e-exams (testing of learning) or for continuous assessment (testing for learning). This is to ensure that any construct-irrelevant technology-related issues or problems do not creep into the testing process and threaten reliability and validity of test-based inferences and relevant decisions. This paper is practical rather than theoretical in its exploration of issues that educational designers might need to consider and address. Considerations addressed here include design features to improve the examinee's experience; steps to minimize or prevent cheating; and ways of preventing tech-induced impediments to successful performance in exams. These considerations are important when designing a language testing interface and relevant regardless of the learning management system used. Having said that, however, this paper is limited in scope and presentation of theoretical grounding because it is descriptive rather than critical in nature. As clearly described in this paper, this was a pilot. A future publication should describe a larger scale study that followed this pilot employing improved design features of the online language testing interface. This other follow-up future publication will address research questions of the study and thoroughly present evidence-based theoretical and critical insights into the tech-induced issues briefly discussed here. CP:20 372 References Ayres, P., & Sweller, J. (2005). The split-attention principle in multimedia learning. In R.E. Mayer (Ed.), The Cambridge Handbook of Multimedia Learning (pp. 134-146). New York: Cambridge University Press. Brown, J. D. (2005). Testing in language programs: A comprehensive guide to English language assessment. New York: McGraw Hill. Chapelle, C. A., & Douglas, D. (2006). Assessing language through computer technology. Ernst Klett Sprachen. doi: 10.1017/CBO9780511733116 Coy, J. (2013). Instant Moodle quiz module how-to. Birmingham: Packt Publishing Ltd. Retrieved from https://www.packtpub.com/hardware-and-creative/instant-moodle-quiz-module-how-instant Davies, A. (1999). Dictionary of language testing (Vol. 7.). Cambridge; New York, NY: Press Syndicate of the University of Cambridge. Douglas, D., & Hegelheimer, V. (2007). Assessing language through computer technology. Annual Review of Applied Linguistics, 27, 115-132. doi: 10.1017/S0267190508070062 Fulcher, G. (2003). Interface design in computer-based language testing. Language Testing, 20(4), 384-408. doi: 10.1191/0265532203lt265oa Kunnan, A. J. (2004). Test fairness. In M. Milanovic and C. Weir (Eds.), European Language Testing in a Global Context: Proceedings of the ALTE Barcelona conference (pp. 27-48). Cambridge: Cambridge University Press. Matbury (2010a). How to use: How to deploy the player in web pages. Retrieved from https://code.google.com/p/moodle-mp3-player-for-tests/wiki/HowToUse Matbury (2010b). Moodle-mp3-player-for-tests: An MP3 player for embedding in tests. Retrieved from https://code.google.com/p/moodle-mp3-player-for- tests/downloads/detail?name=mp3_player_for_tests_2010_04_08_11.zip Myrick, J. (2010). Moodle 1.9 testing and assessment: Birmingham: Packt Publishing Ltd. Retrieved from https://www.packtpub.com/hardware-and-creative/moodle-19-testing-and-assessment National Council on Measurement in Education, American Psychological Association, and American Educational Research Association. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Safe Exam Browser. (2015). Retrieved from http://sourceforge.net/projects/seb/?source=typ_redirect and http://www.safeexambrowser.org/news_en.html. Sweller, J. (1994). Cognitive load theory, learning difficulty and instructional design. Learning and Instruction, 4, 295-312. doi:10.1016/0959-4752(94)90003-5 Al Nadabi, Z. (2015). Features of an online English language testing interface. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:17-CP:21). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:21 373 Fostering deep understanding in geography by inducing and managing confusion: an online learning approach Amaël Arguel Rod Lane School of Education School of Education Macquarie University Macquarie University Confusion is an emotion that is likely to occur when learning complex concepts. While this emotion is often seen as undesirable because of its potential to induce frustration and boredom, recent research has highlighted the vital role confusion can play in student learning. The learning of topics in geography such as tropical cyclone causes and processes can be particularly difficult because it requires the reconstruction of intuitive mental models that are often robust and resistant to change. This paper presents the design framework for an online module designed to enhance university students’ depth of knowledge of tropical cyclones. In particular, the intervention aims manage the level of confusion during learning. We hypothesise that in this way learners can engage with the cognitively demanding ideas in this topic and they are less likely to experience emotions such as frustration and boredom, which would be detrimental to the development of deep understanding. Keywords: Online module; geography; confusion; conceptual change; academic emotions Understanding complex systems Frequently, university students face situations in which the restructuring of existing knowledge is particularly difficult and represent an important step that can influence the other steps of further learning. For example, some learning of concepts from geography can be problematic if learners possess intuitive mental models that are inconsistent with current consensus in the discipline (Lane, 2008). For example, understanding the dynamics of tropical cyclones requires a complex integration of information from different domains, and the evaluation and restructuring of alternative conceptions (Lane & Coutts, 2012). To develop an understanding of cyclone causes and processes, learners require factual and conceptual knowledge of a range of key ideas including evaporation, air pressure and precipitation. These ideas act as threshold concepts (Meyer & Land, 2013) in this topic, enabling learners to develop a relational understanding of the links between the various causes and processes. This cognitive process known as conceptual change or conceptual reconstruction, is highly demanding and requires a significant level of engagement and metacognitive awareness from the learner (Vosniadou, 1994). Recent research suggests that innovative instructional design can improve learners’ motivation and engagement in complex learning tasks and that this can have a positive impact on learning outcomes (Moreno, 2006). Similarly, cognitive disequilibrium caused by an impasse, a contradiction, or some incongruity can also improve the engagement of students (Craig, Graesser, Sullins, & Gholson, 2004). This involves creating challenging learning opportunities that raise the interest of learners whilst building the essential skills for managing confusion. Learners’ confusion It is now widely understood that learning is more than “cold”, rational processes (Pintrich, Marx, & Boyle, 1993). Recent studies have emphasized the role of emotions in learning processes (Leutner, 2014; Mayer, 2014; Mayer & Estrella, 2014). In order to reconstruct their existing mental models students need to be both engaged and motivated. During the learning processes students often experience impasses, discrepancies, and contradictions between information from different sources. This information may come from instruction or from their prior knowledge of the topic (D’Mello, Lehman, Pekrun, & Graesser, 2014). These experiences are likely to induce a cognitive disequilibrium CP:22 374 that will provoke a specific kind of emotion: confusion. Because this emotion is unpleasant by nature (Russell, 2003), it is expected that learners will try to resolve it. This search for a reduction of confusion can increase learner’s engagement with the task, hence leading to a deeper understanding of the content. When building learning sequences to promote deep understanding it is possible to induce confusion to increase engagement. The technique of promoting cognitive disequilibrium by comparing students’ existing and new knowledge is well known by teachers, but perhaps underused in digital learning environments. The aim of the present study was to design a protocol to test the efficacy of an intervention for inducing confusion in an online learning platform used by university students. We hypothesized that, in the domain of geography, the learning of the dynamics of tropical cyclones should benefit from the induction of confusion because the processes involved are complex and students hold robust alternative conceptions about many of the core ideas (Lane & Coutts, 2012). Restructuring these mental models requires engagement and resilience of the learner. Intervention to promote cognitive disequilibrium The induction of confusion during learning can promote a deeper understanding of complex concepts by initiating effortful engagement and problem solving processes (Lehman, D’Mello, & Graesser, 2012). Several conditions are required to ensure that confusion is effectively managed in the learning process: (a) the cognitive disequilibrium that causes confusion must be relevant to the task and the pedagogical goals (D’Mello et al., 2014); (b) cognitive disequilibrium that causes the confusion must be resolved by learners (D’Mello & Graesser, 2014); (c) the environment must provide learners with appropriate scaffolding to help them manage their confusion and its duration. There are two attributes of interventions that employ confusion to promote learning. The first relate to the induction of confusion itself. Confusion can be induced by provoking cognitive disequilibrium, for example with the introduction of contradictory information. The second involve strategies for managing confusion. These two thresholds of the level of confusion during learning can be represented as the boundaries of a zone of optimal confusion (D’Mello et al., 2014), depicted in Figure 1. Figure 2: The boundaries of the zone of optimal confusion If the level of confusion is too high, or if confusion is too persistent, learners might experience frustration followed by boredom. This disengagement could prevent the restructuring of students existing understandings (mental models) (D’Mello & Graesser, 2014). To prevent this strategies can be introduced for managing confusion including improving students’ self-regulation skills and providing carefully targeted feedback. A student’s motivation during learning can also influence their performance. Recent research has noted, for example, the positive effects of incorporating motivational features in the design of educational interfaces including appealing graphics and challenging scenarios (Mayer, 2014). Another strategy for managing confusion and promoting its resolution is to enhance the resilience and CP:23 375 self-efficacy of learners. Anecdotal evidence suggests learners exhibit differences in terms of academic risk taking. Some learners are adventuresome while others are more cautious and likely to avoid situations promoting confusion (Clifford, 1991). Similarly, students who have confidence in their ability to resolve complex problems are more likely to persist in resolving cognitive disequilibrium. To encourage academic risk taking and build self-efficacy in the current study, we progressively introduced students to increasingly complex problems to ensure that they built their confidence and had positive experiences of confusion resolution. The aim of this instructional strategy was to prepare students to face problems in which their engagement in resolving the cognitive disequilibrium would be required. Materials and methods An online unit addressing the causes and processes of tropical cyclones was developed at Macquarie University on a Moodle platform. The unit included five modules focusing on the conceptual building blocks for understanding tropical cyclones (see Figure 2) and included a range of representations such as videos, instructional text, diagrams and animations to promote conceptual change. Participants were, for example, required to respond to a quiz eliciting their initial conceptions and manipulate an interactive animation to explore scientific processes. Activities were then included to stimulate reflection and foster conceptual change. Figure 2: Content structure of the online module After each module, students were asked to complete an online survey to self-report the frequency of the occurrence of selected emotions experienced during learning. This assessment of emotions was adapted from the retrospective affect judgment protocol (D’Mello et al., 2014) and consisted of nine Likert scale items relating to the emotions of anxiety, boredom, confuse/uncertainty, curiosity, delight, engagement/flow, frustration, surprise, and “neutral” for the absence of emotion. For each item, students were asked to provide a score between 1 and 10 to indicate whether they experienced this emotion “never” (score of 1) through to “all the time” (score of 10). A definition of each of these emotions was provided next to their respective rating scale. The unit also included both a pre-test and post-test to assess changes in students’ depth and accuracy of understanding. In addition content-specific quizzes were included at the end of each module. Participants The study was conducted at three Sydney universities and included a sample of 430 pre-service teachers (PSTs). For ethical reasons the universities are identified here as institution A, B and C. Most of the PSTs were from University A (n=228, 53%). There were 187 students from University B (44%) and a small number of students from University C (n=15, 3%). CP:24 376 Preliminary results The study is ongoing and to date we have collected data from 153 of the 430 participants. Strategies for managing confusion and promoting conceptual change This paper presents an innovative solution for promoting the understanding of complex systems. The solution involved using confusion and self-regulation to assist with the process of conceptual change/reconstruction. In learning situations where complex content and/or a conceptual restructuration is involved, controlling the navigation of students within the boundaries of a zone of optimal confusion is likely to be beneficial for learning (Craig et al., 2004; D’Mello et al., 2014). Indeed, instead of designing the instructions in an attempt to protect learners from confusion, the online unit was specifically created in order to induce confusion and to foster engagement. The module included a variety of tasks, for example, videos, simulations and problem solving activities designed to expose students’ existing mental models, explore scientific explanations, and reflect on inconsistencies caused by prior knowledge. The culminating task in the module encourages students to apply their new understandings to provide a rich explanation of the causes and processes of tropical cyclones. The idea of inducing cognitive disequilibrium and confusion in a progressive manner throughout the unit was hypothesized to improve learners’ confidence in dealing with challenging situations and to promote their engagement for a deeper understanding. Until recently, the benefits of inducing confusion have mainly been observed in laboratory-settings (D’Mello & Graesser, 2014; D’Mello et al., 2014; Lehman et al., 2012). In addition, few studies have focused specifically on techniques for confusion remediation despite their documented role in preventing the risks of frustration and boredom in the learning process (D’Mello et al., 2014). The study reported in this paper addresses this gap by examining three dimensions of confusion in the learning process: 1. The relationship between confusion and cognitive disequilibrium. 2. The role of confusion as a motivator and tool for engaging learners 3. Strategies for regulating/managing confusion in the learning process. Moreover, this study was designed to observe the effects of confusion in a real world setting, using an online learning environment and students from different universities. Unfortunately, this choice was also the source of some limitations such as the inability to control all aspects for the environment and employ a randomized controlled research. Nonetheless, we believe that the protocol presented here and the associated research will help to inform the design of digital learning environments to promote deep understanding of complex phenomena in both science and geography. References Clifford, M. M. (1991). Risk Taking: Theoretical, Empirical, and Educational Considerations. Educational Psychologist, 26(3-4), 263–297. http://doi.org/10.1080/00461520.1991.9653135 Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004). Affect and learning: an exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29(3), 241–250. D’Mello, S., & Graesser, A. (2014). Confusion and its dynamics during device comprehension with breakdown scenarios. Acta Psychologica, 151, 106–116. http://doi.org/10.1016/J.Actpsy.2014.06.005 D’Mello, S., Lehman, B., Pekrun, R., & Graesser, A. (2014). Confusion can be beneficial for learning. Learning and Instruction, 29, 153–170. http://doi.org/10.1016/j.learninstruc.2012.05.003 Lane, R. (2008). Students’ alternative conceptions in geography. Geographical Education, 21, 43. Lane, R., & Coutts, P. (2012). Students’ alternative conceptions of tropical cyclone causes and processes. International Research in Geographical and Environmental Education, 21(3), 205–222. Lehman, B., D’Mello, S., & Graesser, A. (2012). Confusion and complex learning during interactions with computer learning environments. Internet and Higher Education, 15(3), 184–194. http://doi.org/10.1016/J.Iheduc.2012.01.002 Leutner, D. (2014). Motivation and emotion as mediators in multimedia learning. Learning and Instruction, 29, 174–175. http://doi.org/10.1016/j.learninstruc.2013.05.004 CP:25 377 Mayer, R. E. (2014). Incorporating motivation into multimedia learning. Learning and Instruction, 29, 171–173. http://doi.org/10.1016/j.learninstruc.2013.04.003 Mayer, R. E., & Estrella, G. (2014). Benefits of emotional design in multimedia instruction. Learning and Instruction, 33(0), 12–18. http://doi.org/10.1016/j.learninstruc.2014.02.004 Meyer, J., & Land, R. (2013). Overcoming Barriers to Student Understanding: Threshold Concepts and Troublesome Knowledge. Routledge. Moreno, R. (2006). Does the modality principle hold for different media? A test of the method-affects- learning hypothesis VL -.22. Journal of Computer Assisted Learning, (3), 149–158. http://doi.org/10.1111/j.1365-2729.2006.00170.x Pintrich, P. R., Marx, R. W., & Boyle, R. A. (1993). Beyond Cold Conceptual Change: The Role of Motivational Beliefs and Classroom Contextual Factors in the Process of Conceptual Change. Review of Educational Research, 63(2), 167–199. http://doi.org/10.3102/00346543063002167 Russell, J. A. (2003). Core affect and the psychological construction of emotion VL -.110. Psychological Review, (1), 145–172. http://doi.org/10.1037/0033-295X.110.1.145 Vosniadou, S. (1994). Capturing and modeling the process of conceptual change. Learning and Instruction, 4(1), 45–69. Arguel, A. & Lane, R. (2015). Fostering deep understanding in geography by inducing and managing confusion: an online learning approach. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:22- CP:26). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:26 378 Using expectation confirmation theory to understand the learning outcomes of online business simulations Pierre Benckendorff Belina Gibbons Marlene Pratt The University of Queensland University of Wollongong Griffith University Australia Australia Australia The purpose of this paper is to contrast learners’ expectations of the knowledge and skills developed by an online business simulation at the start of the semester with their perceptions of how well the simulation performed in meeting these expectations at the end of the semester. The study draws on expectation confirmation theory to measure the expectations and perceived performance of two business simulations. Data were collected from 225 students studying at two Australian universities. The findings indicate that both online business simulations performed strongly in terms of helping learners understand strategy, real world problems and the importance of interaction and cooperation between different business departments. Both simulations also performed well in developing skills across all five levels of Bloom’s taxonomy. There were some notable differences between expectations and performance between the two cohorts and the implications of these differences for business simulation choice and design is discussed. Keywords: business, simulation, pedagogy, assessment, learning outcomes. Introduction It has been suggested that the growing emphasis on skills and graduate capabilities in business education requires a reframing and rethinking of pedagogy to support the development of desired learning outcomes (Biggs, 1999). In this context, innovations in technology enhanced learning have created new opportunities for business educators to create student-centred learning environments that foster the development of graduate capabilities. In particular, the use of simulations, ‘serious games’ and challenge-based learning has received recent attention in a number of fields. Simulations provide experiential learning environments that replicate workplace tasks or processes to allow students to apply knowledge and skills. Simulations are especially useful as a learning tool because they model aspects of reality in a safe environment, allowing learners to engage and make decisions in a risk-free environment (Kriz, 2010). Literature Review Thavikulwat (2004) defines a simulation as “an exercise involving reality of function in an artificial environment, a case study, but with the participants inside” (p. 243). Essentially, simulations provide a simplification of reality that facilitates participant’s exploration of different scenarios and outcomes (Hill, 2001). The focus of this paper is on online business simulations. Business simulations are typically experiential exercises wherein participants are “learning how to learn” (Penger, Znidarsic & Dimovski, 2011; Ncube, 2010; Akilli, 2007). Many business schools have adopted simulations as learning tools because they offer insights into the operational and strategic issues managers face (Adobor & Daneshfar, 2006; Tichon, 2012). The literature has increasingly focused on understanding what participants learn from simulations. Simulations provide more realistic scenarios than case studies often provide students with simultaneous objective feedback (Palmunen, 2013; Tompson & Dass, 2000). Simulations can be ‘fun’, providing participants with enthusiasm and motivation to actively learn. Simulations have been found to be effective in developing a range of employability skills including teamwork, communication and negotiation (Tichon, 2012; Gopinath & Sawyer, 1999), conflict-resolution (Seaton & Boyd, 2009) and interpersonal skills (Penger, Znidarsic & Dimovski, 2011). Business simulations can provide participants with a better understanding of financial reports, improve their computing skills and knowledge of programs such as Microsoft Excel, and enhance their quantitative analysis skills. This is because simulations often require working with numeric data, calculating outputs and understanding CP:27 379 the relationship between decisions and financial results (Fawcett, 2002). The focus of this paper is on contrasting the knowledge and skills learners expect to develop prior to participating in an online business simulation with their perceptions of how well simulations have performed at the end of the semester. The research draws on expectation confirmation theory (ECT) to provide a conceptual foundation. ECT was originally developed in the marketing field, but has subsequently been applied in psychology, consumer research and information systems research (Bhattacherjee, 2001; Oliver, 1980). ECT posits that satisfaction is influenced by the extent to which the performance of a product or service meets the expectations of the user. In a technology context, Bhattacherjee (2001) found that a user’s intention to continue using an information system is determined by their satisfaction, which in turn is influenced by the confirmation (or disconfirmation) of the users expectations based on performance. An understanding of learner expectations about online business simulations is important because ECT predicts that learners will be more satisfied when performance meets or exceeds expectations. As a consequence, learners who are satisfied with their simulation learning experience are likely to be more engaged and motivated to continue using the simulation throughout the semester. An understanding of expectations can also help inform the design of simulation-based pedagogy and assessment to ensure that performance meets expectations. Conversely, an understanding of initial learner expectations can help educators to identify and manage unrealistic expectations. The purpose of this paper is therefore to contrast the knowledge and skills learners expect to develop prior to participating in an online business simulation with their perceptions of how well simulations have performed at the end of the semester. Method Learner expectations were measured using a self-administered questionnaire completed in class by 107 business students at University A and 118 students at University B. Ninety per cent of students were final year undergraduate business students. University A had an even gender balance while University B had twice as many female students compared to male students. University A had an even split of domestic to international students compared to University B with 19 per cent domestic and 81 per cent international students. The students at University A were enrolled in an interdisciplinary final year capstone unit, which was designed to integrate disciplinary knowledge and responsible decision making through the application of ethical, socially responsible, and sustainable practice. The unit is based on a series of lectures and an action based learning project. In the action learning project learners form multidisciplinary teams and run a simulated business for a period of several weeks. Key performance areas include profit, quality, productivity, environmental impact, sustainability, social innovation and ethical performance. Team members compete as they make business performance decisions in the areas of operations, quality, marketing, HR, finance, production, corporate responsibility and sustainability. The total enterprise simulation was created by the university to address the lack of simulations dealing with responsible decision making. Students at University B were enrolled in a capstone hospitality management unit focussed on helping learners to integrate and apply knowledge from prior learning to solve management problems in a team environment. The learning experience is built around a simulation where learners develop strategies and implement decisions in ‘real time’ in order to develop a profitable hotel. The inputs into the simulation include strategic and tactical decisions on quality, refurbishment and extra facilities, room rates and discounting strategies, channel management, food and beverage options, marketing and advertising, environmental management, and human resources. Learners evaluate financial performance, seasonal trends, guest feedback, and staff satisfaction and alter their decisions accordingly. Students were surveyed about the knowledge and skills they expected to develop prior to participating in the simulation. The same students were then asked to complete a post-simulation survey at the end of the semester to measure the perceived performance of simulations in terms of knowledge and skills development. Student expectations and perceived performance were measured using a set of seven-point Likert scales (1 = Strongly Disagree … 7 = Strongly Agree) developed from the literature, student focus groups and trial surveys. The paper also draws on Bloom’s taxonomy to evaluate student expectations of skills (Bloom, Englehart, Furst, Hill, & Krathwohl, 1959). There are five levels in the taxonomy, moving through lowest order processes such as understanding and applying to higher order processes such as analysing, evaluating and creating. CP:28 380 Findings and Discussion The data analysis focuses on three areas: (1) overall patterns in the data, (2) differences between the two cohorts, and (3) differences between the expectations and performance of simulations regarding knowledge and skills development. The mean expectation and perceived performance ratings for both cohorts are presented in Figure 1. Means testing was conducted using the t-test statistic to identify whether differences between the two cohorts and between expectations and performance were significant. Significant differences (p < 0.05) are shown on the figures using arrows. Several observations are evident from the figure: 1. Learners had high expectations about the ability of simulations to develop further knowledge in all areas except financial knowledge (M=5.35). Furthermore, there were no significant differences between the two cohorts (t=-0.633; p=0.527). 2. Learners had high expectations about the ability of simulations to develop skills across all five levels of Bloom’s taxonomy with means ranging from 5.51 to 5.84. There were no significant differences in the expectation levels of the two cohorts. 3. Generally both simulations performed strongly in terms of helping learners understand strategy (M=5.63), developing an understanding of real world problems (M=5.60) and helping students to appreciate the need for interaction and cooperation between different business departments (M=5.81). Both simulations also performed well in terms of developing skills across all five levels of Bloom’s taxonomy. 4. There were some significant differences between expectations and performance across the two cohorts. The findings indicate that the performance of the University B simulation exceeded student expectations in the area of financial knowledge (M E = 5.40; M P = 5.68; t=-2.063, p=0.041). On the other hand, the performance of the University A simulation did not meet expectations in the areas of financial knowledge (M E = 5.29; M P = 4.90; t=2.339, p=0.021) and marketing (M E = 5.35; M P = 4.97; t=2.358, p=0.020). 5. The University B cohort generally rated the performance of their simulation more positively than students using the University A simulation for both knowledge and skills. Significant differences were evident between the two cohorts when students were asked to evaluate their understanding of finance (M A = 4.90; M B = 5.68; t=-4.321, p=0.000), marketing (M A = 4.97; M B = 5.64; t=-3.491, p=0.001) and operations (M A = 5.46; M B = 5.79; t=-1.997, p=0.047) at the end of the semester. Similarly, significant differences were also evident in the skills area, with University B students being significantly more likely than University A students to agree that the simulation had enhanced their ability to analyse data (M A = 5.50; M B = 5.87; t=-2.320, p=0.021), evaluate problems and make decisions (M A = 5.61; M B = 5.93; t=-2.143, p=0.033). Figure 1: Mean ratings for knowledge and skill Mean based on 1 = Strongly Disagree … 7 = Strongly Agree; Significant differences (p < 0.05) The findings generally support other studies that have found that business simulations are effective at CP:29 381 developing a range of knowledge and skill areas. This study differs from previous work by also identifying student expectations and by contrasting expectations and performance across two different simulations. This analysis identifies that both simulations are effective at providing an authentic context for the development of business knowledge and basic skills such as understanding and application as well as advanced skills such as analysis, evaluation and creation. Business educators often struggle to develop and assess these skills using more traditional pedagogies and assessment. The originality of this paper therefore resides in the implications for simulation design in the future. The expectations and perceptions of two student cohorts provide insight into how closely learning outcomes need to be matched with the selection and design of the simulation. Some of the differences that have been observed warrant further discussion. The University B students clearly underestimated the extent to which their simulation would develop their understanding of finance. These students generally major in tourism, hospitality and events management and are unlikely to have completed a substantial number of advanced accounting and finance units in their program. The University A cohort on the other hand included both business and commerce students and being in their final year, the business and commerce students are likely to have had an advanced understanding of finance concepts. It is therefore not surprising that these students did not learn a great deal more about finance and marketing from their simulation. This example highlights that one simulation is not necessarily better than another, but that the prior knowledge and skills of each cohort need to be considered when using a simulation to ensure that opportunities do exist for further advancement of knowledge and skills. The perceived performance of the University A simulation was also not rated as highly as the University B simulation in the areas of marketing and operations. These differences may be the result of differences between the learning objectives of the two simulators. The University B simulation is an operational and strategic planning simulation and has complex modules in the areas of operations, revenue management and marketing. On the other hand, the University A simulation was purpose built to develop student capacity in the areas of ethics, social responsibility and professional practice – all areas that were not measured by this study. Differences in the student perceptions of the skills developed by the two simulations may come down to differences in the complexity of the two simulations. Conclusion This paper has contrasted the knowledge and skills learners expect from participating in an online business simulation by comparing two different University simulations. Student learning expectations from both cohorts were high as were their perceived learning from the simulation. While both student cohorts varied in composition and business knowledge, each experienced a high level of engagement, learning interdependencies, strategy, real world knowledge and the development of skills across all levels of Bloom’s taxonomy. Differences arose with the influence of discipline upon the perceived performance of the two simulations, highlighting the importance of aligning the choice and/or design of simulation with the learning objectives. Acknowledgements: Support for this project has been provided by the Australian Government Office for Learning and Teaching. The views expressed in this paper do not necessarily reflect the views of the Australian Government Office for Learning and Teaching. References Adobor, H., & Daneshfar, A. (2006). Management simulations: determining their effectiveness. The Journal of Management Development, 25(2), 151-168. Akilli, G. K. (2007). Games and simulations: A new approach in education. Bhattacherjee, A. (2001). Understanding information systems continuance: an expectation- confirmation model. MIS quarterly, 25(3), 351-370. Biggs, J. (1999). Teaching for quality learning at university. Buckingham, UK: Open University Press Bloom, B. S., Englehart, M. D., Furst, E. D., Hill, W. H., & Krathwohl, D. R. (1959). Taxonomy of educational objectives: The classification of educational goals. New York: David McKay. Fawcett, S. L. (1996). Fear of accounts: Improving managers' competence and confidence through simulation exercises. Journal of European Industrial Training, 20(2), 17-24. CP:30 382 Fawcett, S. L. (2002). Group gaming and simulation in hospitality management: A user's guide. LTSN Hospitality, Leisure, Sport and Tourism. Gopinath, C., & Sawyer, J. E. (1999). Exploring the learning from an enterprise simulation. Journal of Management Development, 18(5), 477-489. Hill, C. C., & Semler, S. W. (2001). Simulation enhanced learning: Case studies in leadership development Retrieved 5 August, 2015, from http://www.learningsim.com/content/sim_enhanced_learning.pdf Kriz, W. C. (2010). A Systemic-Constructivist Approach to the Facilitation and Debriefing of Simulations and Games. Simulation & Gaming, 41(5), 553-680. Ncube, L. B. (2010). A Simulation of Lean Manufacturing: The Lean Lemonade Tycoon 2. Simulation & Gaming, 41(4), 568–586. Oliver, R. L. (1980). A cognitive model of the antecedents and consequences of satisfaction decisions. Journal of marketing research, 17, 460-469. Palmunen, L., Pelto, E., Paalumaki, A. & Lainema, T. (2013). Formation of Novice Business Students: Mental Models Through Simulation Gaming. Simulation & Gaming, 44(6), 846–868. Penger, S., Znidarsic, J. & Dimovski V. (2011). Experiential Learning and Management Education: Empirical Research and Implications for Practice in Higher Education in Slovenia. International Journal of Management and Information Systems, 15(1), 23–33 Seaton, L.J. & Boyd, M. (2008). The effective use of simulations in business courses. Academy of Educational Leadership Journal, 12, 107–118. Thavikulwat, P. (2004). The Architecture of Computerized Business gaming Simulations. Simulation & Gaming, 35(2), 242-269. Tichon, J. (2012). Evaluation of virtual reality training using affect. International Journal on E- Learning, 11(2), 107-116. Tompson, G. H., & Dass, P. (2000). Improving Students’ Self-Efficacy in Strategic Management: The Relative Impact of Cases and Simulations. Simulation & Gaming, 31(1), 22-41. Whiteley, T. R., & Faria, A. J. (1989). A Study of the Relationship Between Student Final Exam Performance and Simulation Game Participation. Simulation & Gaming, 20(1), 44-64. Benckendorff, P., Gibbons, B., & Pratt, M. (2015). Using expectation confirmation theory to understand the learning outcomes of online business simulations. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:27-CP:31). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:31 383 Towards a Pedagogy of Comparative Visualization in 3D Design Disciplines James R. Birt Jonathan Nelson Dirk Hovorka Faculty of Society and Design Faculty of Society and Design Business Information Systems Bond University, Australia Bond University, Australia University of Sydney, Australia Spatial visualization and interpretation are important skills for designers. However, these skills generally require significant experiential development over the course of years. Visualizations allow the human brain to convey complex spatial concepts in intuitive, navigable and manipulable forms improving learner outcomes and perceptions. But often these visualizations are studied as single modality solutions. Dual modality and multimedia presentation studies show positive improvements in learner outcomes but dual modality is often difficult to compare. This paper presents ongoing research in the use of comparative multimodal visualizations produced with emerging technology solutions in 3D Design classrooms. Presented are previous findings from multimedia design and a methodology to widen the scope of study. The context for this study is a university first year undergraduate course in architectural design. The presupposed outcome is that students become adept at interpretation and mental conversion at a rate greater than they would through more traditional curricular means. Keywords: Visualization; dual modality; 3D printing; virtual reality; multimedia; architecture; Introduction Visualizations and emerging technologies such as 3D printing and virtual reality are providing transformative change to education (Klerkx, Verbert & Duval, 2014). This is evident through technology enhanced teaching and learning (Keppell, Suddaby & Hard, 2011) and increased awareness (Johnson, et al., 2015a) and use (Johnson, et al., 2015b) of emerging media technology in higher education. Although transformative, technology should not diminish the foundational propositions in teaching and learning that, pedagogy is foremost, learning is the construction of knowledge, and collaboration is necessary to derive learning outcomes (Fowler, 2015; Ocepek et al., 2013). Learning is considered to be an active process influenced by prerequisites of the learner (Mayer 2005, 2008). The goal is to move a learner from shallow to deep learning through internal motivation with an intention to understand and environment(s) where students develop a strong personal interest through well-formed learning design. Visualization is the representation of abstract information and creation of approaches for conveying concepts in intuitive, navigable and manipulable forms including images, videos, virtual environments and physical representations (Höffler, 2010). In the context of this study, that is, 3D modelling, spatial visualization and interpretation are undoubtedly important skills for novice designers to develop (Wu & Chiang, 2013). These skills are involved in visualizing shapes, rotation of objects, and how pieces of a given design solution fit together. The ability to quickly, creatively and effectively interpret 3D spaces and forms from 2D drawings and the inverse, to reduce 3D ideas to 2D representations for communication purposes, is generally regarded as a hallmark of the profession. However, these skills generally require significant experiential development over the course of years and while experienced designers are adept at performing these translations there exists a communication barrier from instructor to learner due to this skills gap. Prior research in visualization has revealed strengths and weaknesses in the impact of any single modality on learning, and those learners themselves have different styles, needs and capabilities (Fowler, 2015; Höffler, 2010; Klerkx, Verbert & Duval, 2014; Mayer 2005, 2008; Ocepek et al., 2013). The use of multimedia visualizations and multiple modalities as positive learning design support tools are well documented and accepted (Moreno & Mayer, 2007). This research is therefore, not seeking single modality solutions but rather a systematic approach to multimodal modality and interactive presentation and instructions for curriculum designers and learners in courses that rely on visualizations and manipulations. The fundamental question is not whether technology, simulation or CP:32 384 visualization affects learning but how to guide the use of comparative multimodal visualization technology through, media affordances, lesson sequencing, learner perceptions and reflection to inform effective instruction and learning. This paper presents ongoing work on the effect and use of comparative visualization in the teaching and learning of 3d modelling design. Presented is a summary of the author’s previous pilot study (Birt & Hovorka, 2014) in the multimedia design discipline and methods to widen the scope of study and subsequent pedagogical approach to transition across disciplines to architectural design. Comparative visualization in multimedia design Previous work of the authors (Birt & Hovorka, 2014) explored a pilot study examining the effect of mixed media visualization pedagogy using 3D printing, 3D virtual reality and traditional 2D views on learning outcomes in multimedia 3d modelling design. The learning objectives and resulting objects and their use in the classroom afforded learner centered active engagement through physical and virtual interaction with the visualization technologies. Research measures from each of the weekly learning objectives were achieved through coding and analysis of learner blogs conducted during the 12 week semester. Students were asked to engage in deeper learning by answering questions related to the weekly learning objective and technology visualizations. This included questions on: engagement; cognitive memory; visualization advantages/limitations; contrast between visualization media; how each technology would assist in demonstration of the learning objective to a team of designers; and communication of the learning objective between themselves and the instructor. The direct and reflective comparison between technologies revealed a strong interaction among them for learning. Each visualization technology had positive, negative and mixed perceptions when it came to accessibility; usability; manipulability; navigability; visibility; communication; and creativity. With 3D printing offering positives in haptic feedback and connection between the virtual and physical environment; virtual reality offering real-time external and internal interaction, object scope and scale, improved spatial awareness and defect discovery; and traditional 2D offering high accessibility, ease of use and rapid versioning. The comparisons between delivery modes (visualization technologies) provided much more than different versions of the same material. The engagement with each technology required reinterpretation of the principles upon which the lesson was focused. This provided students a way to “reframe” their own understanding and to “fill in the gaps” they observed using other media. It was suggested that this is particularly applicable to foundational principles where a deep understanding and ability to understand the principle in different contexts is important. Project rationale In 3D architectural design as in 3D multimedia design, as spatial and geometric ideas become increasingly complex the industry standard 2D representations tend to convey less information about a design and how it is to be interpreted. Figure 1 illustrates this by showing: (a) 2d orthographic elevation drawing of a geometrically complex structure, (b) virtual 3D model perspective and, (c) the physical building. (a) (b) (c) Figure 1: A geometrically complex structure shown in 2D, virtual 3D and physical 3D While the 2D representation is useful in showing a simplified general arrangement of the building elements, many 2D drawings are required to fully illustrate the complexities and form of the design. In particular the region marked in Figure 1 (a) is not readily discernable from this projected vantage point as can be seen in Figure 1 (c). The virtual 3d model, while it serves to inform a more complete view of the tectonics and geometric characteristics, contains little to no data about physical assembly, nor does it facilitate a piecemeal selection of information about the structure which is the goal of the 2D projections. The physical building shown in Figure 1 (c) provides haptic feedback and navigation but lacks internal transition within the geometry and ways to view the structure in its entirety. These differences in utility and comprehensibility therefore necessitate the need for trainee designers to CP:33 385 develop the skills to quickly and effortlessly switch back and forth between various media both cognitively and physically. Visualizations can assist in teaching, learning and skills acquisition because the human brain is wired to ‘see’ and comprehend relationships between images faster and more efficiently than text or numbers (Höffler, 2010). Additionally, visualizations allow people to move between concrete reality, which means objects they can see and touch, to ideas and creations of objects and solutions that do not exist yet. Visualisation can enhance students’ conceptualisation, manipulation, application and retention of knowledge and skills provided they follow specific learning design (Mayer 2005, 2008; Moreno & Mayer, 2007). In part, visualizations must prime the learner’s perception - why do learners care?, draw on prior knowledge, avoid working memory overload through specific learning objectives, provide multiple presentation modalities, move learners from shallow to deeper learning and allow learners the opportunity to apply and build their own mental models (Hwang & Hu, 2013). Meta- analytic studies of 2D and 3D visualization show positive improvements in learning outcomes among low and high spatial learners (Höffler, 2010). However, there are many challenges to visualizing learning objectives including choosing between 2D and 3D interfaces, physical or virtual navigation, interaction methods, selecting an appropriate level of detail and availability of the visualization media. To assist with these challenges, technologies such as 3D modelling, game engines, 3D printing and VR are becoming available for use commercially and thus able to be incorporated into the classroom. The 2015 NMC Higher Education Horizon Report (Johnson, et al., 2015a) and Technology Outlook for Australian Tertiary Education Report (Johnson, et al., 2015b) specifically highlight these technologies as key educational technologies. VR technologies are mature, but the uptake in education has been hindered by cost, expertise and capability. This is now changing with the recent wave of low cost TM immersive 3D VR technology by vendors such as Oculus Rift (http://www.oculusvr.com/) and TM powerful interactive game engines such as Unity3D (http://unity3d.com/). However, there still remains an innate lack of physical haptic feedback that one gains through physical media manipulation (Fowler, 2015). In this way, 3D printing offers a way to bridge the gap between the virtual and the real. 3D printing has seen an explosion in the past five years due to low cost fused TM deposition modeling (FDM) systems by makers such as MakerBot (http://www.makerbot.com/). 3D printing at its basic level uses an additive manufacturing process to build objects up in layers using plastic polymer. Although the process is slow, 3D printing creates direct links between a virtual 3D based model and the formation of an accurate, scaled, physical representation from that model (Loy, 2014). This direct linking of object making to computer modeling changes the relationship of the learner to the making of the object and subsequent use, that is, it creates and enables a haptic feedback loop for learners. Project Methodology: Translating to architectural design The purpose of this study is to translate the previous findings and pedagogy of comparative visualization use in the classroom to additional disciplines in the hopes to (i) gain insight into spatial visualization skills in trainee students and (ii) form a body of knowledge to allow for future expansion to new skills and disciplines. The selected discipline for this proposed pilot study is in architectural design. This discipline was selected primarily because it is an accredited design discipline with coinciding learning outcomes with the first study in multimedia design. In line with the original pilot study the first research question is: RQ1: “How do learners perceive the comparative capabilities of visualization media to support learning?” and the second research question is: RQ2: “Do learner’s preferences for visualization technologies change with task or over time?” To answer these questions students will be given a series of eight dual coded comparative weekly media learning objectives highlighted in Table 1. Table 1: 3D modelling learning objectives and applied media conditions in architectural design Applied Media Learning Objective 2D VR Phys Introduce the basic theoretical paradigms of 3d modelling Y Y Y Demonstrate applied knowledge of 3d primitive construction and manipulation Y Y Demonstrate applied knowledge of curves and NURBS surfaces Y Y Demonstrate an understanding of 3d modelling as it relates to the human scale Y Y CP:34 386 Demonstrate the ability to construct complex surfaces Y Y Demonstrate an understanding of 3d modelling as it applies to architecture Y Y Demonstrate the ability to manage complex scenes with a high number of Y Y models Demonstrate applied knowledge of presenting a complex scene and ability to Y Y Y reflect and synthesize the course material It must be noted that although the word Demonstrate is used in Table 1, in the studied domain of architecture this refers to higher order skills of analyse, evaluate and create as highlighted by Blooms taxonomy and deeper learning.Through weekly learner blogs students will be asked a series of questions in line with the previous study and translation to deeper learning. The theme of the questions include: engagement; cognitive memory; visualization advantages/limitations; contrast between visualization media; how each technology would assist in demonstration of the learning objective to a team of designers; and communication of the learning objective between themselves and the instructor. These question themes and learning objectives have been formed in relation to the specific learning designs highlighted by Mayer (2005, 2008); Moreno & Mayer, (2007); Hwang & Hu, (2013) and others. The outcomes from the learner blogs will be analyzed using a thematic analysis TM through NVIVO (http://www.qsrinternational.com) and correlated against student outcomes. Over the course of the eight exercises students compare various forms of media including 2D, 3D print, built environments and 3D VR, culminating in comparison and demonstration (creation, evaluation and analysis) of all three. These exercises are intended to provide practical concept conveyance and higher order thinking. An illustrative example of the complex scene learning objective is provided in Figure 2. The scene represents an interactive VR visualization and simulated lighting cycle of a physical built environment on the learner’s campus highlighting complex shapes, surfaces, lighting and human scale. Figure 2: A geometrically complex scene in VR of a physical built environment on the learner’s campus Expected Outcomes and Future Project Direction The presupposed outcome of this study is that students become adept at 3D interpretation and mental conversion between 3D and 2D at a rate greater than they would through more traditional curricular means. More specifically as highlighted in Birt & Hovorka (2014) it would indicate that students would initially prefer the higher dimensional media as a means of rationalization and exploration due to the ease and familiarity permitted by them, but by the end of the study the students would be more adept at interpreting the lower dimensional media and thus prefer them for their convenience and availability. A future outcome of this study is to gain insight into the effectiveness of wholly 3D and VR representations of the built environment that can potentially help move the design industry toward working in higher dimensions. Additionally, the outcomes from this study and the previous work will look to extend the pedagogy and design to new skills and disciplines framing a body knowledge to develop a guideline of comparative visualization use in the classroom. References Birt, J. & Hovorka, D. (2014). Effect of mixed media visualization on learner perceptions and outcomes. In Proc 25th Australasian Conference on Information Systems. 8-10 December. Auckland, New Zealand. Fowler, C. (2015). Virtual reality and learning: Where is the pedagogy? British journal of educational technology, 46(2), 412-422. CP:35 387 Höffler, T. N. (2010). Spatial ability: Its influence on learning with visualizations - a meta-analytic review, Educational Psychology Review, 22(3), 245-269. Hwang, W.Y., & Hu, S.S. (2013). Analysis of peer learning behaviors using multiple representations in virtual reality and their impacts on geometry problem solving, Computers & Education, 62, 308- 319. Johnson, L., Adams Becker, S., Estrada, V., & Freeman, A. (2015a). NMC Horizon Report: 2015 Higher Education Edition. Austin, Texas: The New Media Consortium. Johnson, L., Adams Becker, S., & Hall, C. (2015b). NMC Technology Outlook for Australian Tertiary Education: A Horizon Project Regional Report. Austin, Texas: The New Media Consortium. Keppell, M., Suddaby, G. & Hard, N. (2011). Good Practice Report: Technology-Enhanced Learning. Sydney: Australian Learning and Teaching Council. Klerkx, J., Verbert, K., & Duval, E. (2014). Enhancing learning with visualization techniques. In Handbook of Research on Educational Communications and Technology (pp. 791-807). Springer New York. Loy, J. (2014). eLearning and eMaking: 3D Printing Blurring the Digital and the Physical, Education Sciences, 4(1), 108-121. Mayer, R. E. (2005). Cognitive theory of multimedia learning. The Cambridge handbook of multimedia learning, 31-48. Mayer, R. E. (2008). Applying the science of learning: evidence-based principles for the design of multimedia instruction. American Psychologist, 63(8), 760. Moreno, R., & Mayer, R. (2007). Interactive multimodal learning environments. Educational Psychology Review, 19(3),309-326. Ocepek, U., Bosnić, Z., Šerbec, I. N., and Rugelj, J. (2013). Exploring the relation between learning style models and preferred multimedia types, Computers and Education, 69, pp 343-355. Wu, C. F., and Chiang, M. C. (2013). Effectiveness of applying 2d static depictions and 3d animations to orthographic views learning in graphical course, Computers and Education, 63, pp 28-42. Birt, J.R., Nelson, J., & Hovorka, D. (2015). Towards a Pedagogy of Comparative Visualization in 3D Design Disciplines. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:32-CP:36). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:36 388 Implementing blended learning at faculty level: Supporting staff, and the ‘ripple effect’ Rosy Borland Birgit Loch Liam McManus Faculty of Science Faculty of Science Faculty of Science Swinburne University of Technology Swinburne University of Technology Swinburne University of Technology More and more Australian universities are mandating blended learning approaches, whether for efficiency reasons to reduce face-to-face classes or the need for scarce teaching spaces, to create more engaging learning environments by accessing the benefits online learning provides, or simply to keep up with competitors who have implemented such approaches. The challenges surrounding the adoption of online teaching approaches are not new. In the face of pressure to offer greater flexibility in their course offerings, Australian universities have, for a number of years, grappled with how to successfully embrace technology-supported learning in a way which engages both academic staff and their students. In this paper, we use an action research approach to describe how blended learning was introduced at a STEM faculty. We focus on how this has resulted in certain types of staff support provided. We also highlight the faster than expected diffusion of innovation that we have observed. Keywords: Blended learning, staff engagement, change management Introduction At the ascilite conference last year we presented a paper (Loch & Borland, 2014) on challenges faced by the discipline of mathematics when the blended classroom is implemented. We concluded with seven research questions that required investigation, given the increase of blended learning mandates in Australian universities. These questions similarly apply to other STEM disciplines. In this paper, we suggest answers to research question number 6 from last year’s paper (Loch & Borland 2014, p. 711): On a departmental level, what is the best approach for supporting teaching staff (including sessional staff) to develop and implement innovative pedagogy approaches, promote digital content creation and use technology to enhance learning and teaching outcomes? We do this in the context of an action research project to investigate introduction of blended learning in a STEM faculty at a university that has set a goal of achieving 50% of student learning online by 2020. We first provide the context of this study, then explain our theoretical framework and provide a description of how the introduction of blended learning was approached, and how this has led to the provision of certain types of staff support. We conclude with a discussion of the ‘ripple effect’. The context The faculty of STEM was created after a restructure of the university in 2014. In contrast to the other two faculties at the university that both teach into wholly online courses, the faculty of STEM had had little strategic engagement with online learning until the blended learning project commenced. There had been pockets of innovation, with lecturers trialing either their own ideas, or ideas they had learnt about at conferences and through discussions with colleagues (Abdekhodaee, Ekambaram & Borland 2011; But & Shobbrook 2012; Cain & Woodward 2012). Many of the lecturers previously in the engineering faculty were using tablet PCs provided through education equipment funding since 2011, as well as a large number of ‘clickers’. While individual lecturers used these tools to rethink their teaching style, there had not been any professional development on learning design to enable the CP:37 389 change required for blended learning. Many faculty staff had not moved much beyond uploading lecture notes and posting announcements via the Learning Management System. A university directive to increase the proportion of online learning to 50% resulted in the faculty executive nominating 20 first year units to be redeveloped in blended mode by the end of 2015. This initial proposal resulted in the establishment of the ‘Blended Learning Project (BLP)’. A small project team was subsequently convened comprising an academic leader and education developer and, as described later, a successful argument was made to appoint a project manager and technical support officer. A process for ongoing evaluation of the project was also put in place. Theoretical framework for the blended learning project We needed a research method that is participative and grounded in experience and that would reflect the context and objectives of the implementation of the blended learning project (Reushle & Loch, 2008). For this purpose, a qualitative action research method (Reushle, 2005), adapted by Reushle and Loch (2008) was modified to design and conduct the project. The model currently has two phases: the first phase corresponds to the pilot units that were redeveloped into blended mode first. The second phase relates to the remaining units to be redeveloped by the end of 2015. Figure 1 shows the iterative, cyclical process to develop, implement, evaluate, and modify the process. The evaluation of Phase 1 led to changes made to the process for Phase 2. In this paper, we will focus on just some parts of this model: the initiation of the project, the refining of the approach, and the unexpected outcomes, which we will call the ‘ripple effect’. Figure 1: Action research framework for the project Defining the task Requirements from the faculty executive as to the level of blended learning to be implemented (Alammary, Sheard & Carbone, 2014) were vague. Therefore, we needed to find a clear definition of blended learning situated in our particular context. From the various definitions available in the literature (see for example Alammary et al., 2014) we opted to embrace an understanding of blended learning as being an approach which increases opportunities for students to engage with content and resources online in order to make more time available in face-to-face classes for active learning. There was a directive from the executive not to set minimum standards. CP:38 390 Exploring approaches taken at other universities Once the task had been set, we identified how blended learning had been approached at other Australian universities. We spoke directly with colleagues from the University of Western Sydney, who were running a large blended learning project across the whole university. We also audited university websites that described policies and approaches to blended learning. These comparisons to our own situation led us to request funding to bring on board a project manager and a technical officer to support staff with their use of technology. Defining the process A multi-faceted process was put in place to support the blended learning project. The first step was to look at the teaching strategies currently being used. Unit teams for the pilot units attended a two-day learning design workshop based on the Carpe Diem process (Salmon & Wright, 2014). These workshops were conducted by the university’s central learning and teaching unit, and project team members attended the workshops to provide support for faculty participants. In these workshops the overall approach to teaching of the unit was considered and learning activities were reviewed and revised in light of the stated unit outcomes. An action plan was developed which outlined the changes to be implemented and any online content, assessments and activities to be developed. Following these workshops, teams were expected to work on their action plans, with support from members of the project team. Refining the approach As pilot units started to go through the re-development process, we realised the crucial importance of both allowing for flexibility in the approach taken and of encouraging unit teams to take small steps where appropriate to allow for familiarisation with the technology and the challenges presented at each stage of implementation (Weaver, Robbie & Borland, 2008). The faculty executive confirmed that this approach to delivering blended learning outcomes would satisfy the requirements of the faculty plan. The format of the learning design workshops was reviewed and it was decided to change it to a one- day format. This allowed more time following the workshop for unit teams to work on investigating different assessment strategies, making videos and restructuring unit sites. Designing staff support In putting together a strategy to support staff with unit development, we focused both on formal professional development and on fostering a community of practice which encouraged informal interactions between lecturers. We applied for and received faculty funding to purchase additional tablet PCs. We were also able to establish a ‘quiet recording studio’, a small office equipped with video and audio recording equipment for lecturers to book to create online resources. A series of workshops was run to provide advice on education design as well as development and support of resources, assessments, and activities. A number of different workshops were included in the mix: How-to Workshops to introduce tools and techniques, regular Lunchtime Support Sessions to give staff an opportunity to try things out and raise any issues they may have been experiencing, and Shared Practice Sessions to demonstrate what the more innovative adopters in the faculty had been doing and discuss the pros and cons of these implementations. Although these workshops were designed specifically for the purposes of the project, they were open to all staff in the faculty. They provided a valuable forum for establishing a collaborative relationship between academics working on blended learning developments across the faculty as well as members of the project team. In addition, hands-on education and technical support was provided for academic staff working on blended learning developments. A dedicated space was designated on the faculty wiki to share best practice, and provide access to guides for using strategies and tools and how-to articles as well as information about workshops and other related activities within the faculty. CP:39 391 Designing a communication strategy A communication strategy had not been necessary in the initiation stage of the project as the number of pilot units had been small. However, to achieve buy-in from teaching staff involved in the mandated units and to have wider dissemination of project outcomes, we developed a strategy to communicate regularly with faculty staff via two key mechanisms: 4. Teaching with Technology Snippets—a short weekly email as a teaser to introduce an available teaching technology, with additional information posted on the wiki 5. Regular emails advertising upcoming sessions, linking to information posted on the wiki. The ripple effect As seen above, getting people on board as a result of a faculty directive is not always straightforward. For this reason, we recognized that we needed to foster enthusiasm across the faculty in order for the gains made to ‘take hold’ and for the uptake of innovation to proceed at a reasonable pace while project resources were available. As Rogers (2003, p. 1) wrote: Many innovations require a lengthy period of many years from the time when they become available to the time when they are widely adopted. Therefore a common problem for many individuals and organisations is how to speed up the rate of diffusion of an innovation. Following the pilot phase, a number of nominated units started work on unit redevelopments. The implemented changes included developing ongoing assessment strategies using various online tools and online video to deliver content and demonstrate problem solving strategies, restructuring unit sites to support student learning and increasing use of interactive teaching strategies. These were the planned outcomes of the project. In addition to these anticipated outcomes, we have also witnessed unplanned, internally-motivated change; this is what we call the ‘ripple effect’. Even though units were initially nominated from above, a number of units beyond these were volunteered, including one of the pilot units. This has had the advantage of increasing momentum but also taking advantage of the enthusiasm of volunteers. Since then, further developments have occurred. Some of the teaching staff who attended the learning design workshops, have gone on to implement changes in other units they are teaching. Customised workshops were requested for staff in two departments within the faculty, one of which has elected to undertake its own ‘mini blended learning project’ with each unit team developing blended learning strategies within their unit. We believe that it is the flexibility and receptivity built into the design of the project that has made these unexpected and desirable outcomes possible. Conclusion In this paper, we described aspects of our action research framework to implement blended learning in our faculty. We explained the staff support we had decided was required to answer the question we had asked in our previous paper (Loch & Borland 2014, p. 711). While academics are notorious for ignoring emails, our short snippets resulted in responses from many asking for support or access to a particular technology. The existence of the ripple effect and its growing influence have shown that our efforts have been targeted in the right direction. In particular, the buy-in from a whole department was welcome but unexpected thinking back to the start of the project. References Abdekhodaee, A., Ekambaram, P., & Borland, R. (2009). Using Wiki as a facilitative tool for group work. In Developing engineers for social justice: Community involvement, ethics & sustainability Proceedings Australasian Association for Engineering Education Conference Fremantle 2011. http://search.informit.com.au/documentSummary;dn=258577984355051;res=IELENG Alammary, A., Sheard, J., & Carbone, A. (2014). Blended learning in higher education: Three different design approaches. Australasian Journal of Educational Technology, 30 (4), 440- 454. http://ascilite.org.au/ajet/submission/index.php/AJET/article/view/693 CP:40 392 But, J. & Shobbrook, R. (2012). Web-based lecture technologies and their effects on student performance. In Profession of Engineering Education: Advancing Teaching, Research and Careers Proceedings Australasian Association for Engineering Education Conference Melbourne 2012. http://www.aaee.com.au/conferences/2012/documents/abstracts/aaee2012-submission- 111.pdf Cain, A. & Woodward, C. J. (2012). Toward constructive alignment with portfolio assessment for introductory programming. In Proceedings of IEEE International Conference on Teaching, Assessment, and Learning for Engineering Hong Kong 2012. http://www.aaee.com.au/conferences/2012/documents/abstracts/aaee2012-submission- 111.pdf Loch, B. & Borland, R. (2014). The transition from traditional face-to-face teaching to blended learning – implications and challenges from a mathematics discipline perspective. In Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014. http://ascilite2014.otago.ac.nz/files/concisepapers/278-Loch.pdf Reushle, S. & Loch, B. (2008). Conducting a trial of web conferencing software: Why, how, and perceptions from the coalface. Turkish Online Journal of Distance Education, 9 (3), 19– 28. http://eprints.usq.edu.au/4254/1/Reushle_Loch_Publ_version.pdf Reushle, S. (2005). Inquiry into a transformative approach to professional development for online educators. Unpublished doctoral thesis, USQ. http://eprints.usq.edu.au/1494/ Salmon, G. & Wright, P. (2014). Transforming future teaching through ‘Carpe Diem’ learning design. Education Sciences. 4 (1), 52-63. http://www.mdpi.com/2227-7102/4/1/52 Weaver, D., Robbie, D. & Borland, R. (2008). The practitioner’s model: Designing a professional development program for online teaching. International Journal on E-Learning, 7 (4), 759- 774. http://www.editlib.org/p/24411/ Borland, R., Loch, B., & McManus, L. (2015). Implementing blended learning at faculty level: Supporting staff, and the ‘ripple effect’. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:37-CP:41). Note: All published papers are refereed, having undergone a double-blind peer-review process. e author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:41 393 The ethical considerations of using social media in educational environments Leanne Cameron Miriam Tanti Kim Mahoney Australian Catholic University Australian Catholic University Hills Grammar School Students in an undergraduate pre-service teacher education course were asked to utilise Twitter to access the professional educational community. Their tweets were to be used to promote the sharing of educational resources and establish a local supportive community of practice, to keep others informed of their teaching experiences and provide a vehicle for support and advice, both inside and outside the university. The ethical issues in relation to the use of social media in educational environments were wide- reaching and complex. This paper reports on a pilot study that begins an investigation on the practices of university students using social media in their studies. The ultimate aim of the project is to develop workable guidelines on the ethical use and practice of social media use in university education. Keywords: Social media, Ethical use of Social Media, Professional Learning Networks. Background 160 students in an undergraduate pre-service teacher education course were asked to utilise Twitter to access the professional educational community, both amongst their peers and outside the university. Their tweets were to be used to promote the sharing of educational resources and establish a local supportive community of practice, to keep others informed of their teaching experiences and provide a vehicle for support and advice. This approach was designed to provide opportunities for the pre-service teachers to create a sustainable culture of learning and build their own personal learning network (PLN) with contacts from both within, and outside, the university via social media. Treadwell (2008) defines a PLN as “a space and place where the learner creates, completes, documents and communicates their learning” within a community of learners. These PLNs offer opportunities for the pre-service teachers to discuss ideas or issues with experienced educators and workshop learning activities. Many practising teachers use their own PLNs to share ideas, resources and experiences and reflect on their own learning. The lecturers implementing the assignment in the study were aware the ethical issues of using social media in educational environments are wide-reaching and complex. With the increased use of these technologies in educational settings, it has been suggested that the onus be on institutions to develop protocols and policies to enable and support responsible use (Andrews, Dyson, Smyth & Wallace, 2011). This documentation is extremely common in Australian school systems and becoming more common in the Higher Education area. This paper reports on a pilot study that begins an investigation on the practices of university students using social media in their studies. The ultimate aim of the project is to develop workable guidelines on the ethical use and practice of social media use in university education. A description of the social media used in the project and its value in the educational sphere follows. Microblogging “Microblogging is the practice of posting small pieces of digital content—which could be text, pictures, links, short videos, or other media—on the Internet” (Educause, 2007, p.1). Twitter was the microblogging vehicle used in this project. Twitter is a social media tool that allows users to send and read short 140-character messages, and or photographs (tweets). Many lecturers see benefits of using social media with their students (Educause, 2007). They enthuse about its potential to promote the sharing of ideas, activities, events and interests within a learning community. It can broaden perspectives, beyond the local, into a more global, world-view. Used this way, social media could be regarded as a conduit to promote learning and an excellent professional development resource. The CP:42 394 potential of social media in education is also acknowledged by a number of the school education bodies in Australia: “ACCE firmly supports the potential educational affordances of online communication, including social media… Teachers at all levels are demonstrating innovative and educationally rewarding uses.” (Brandenburg, 2012) “Conversations in social media are a dialogue, an opportunity to listen, share, collaborate and respond to our colleagues and communities. We recognise the importance of participating in these conversations. Because the social media space is relatively new, and comments may be public and potentially permanent, we’ve developed these guidelines [the DEC Social Media Guidelines] to support staff as they engage in any conversations or interactions using digital media for official, professional and personal use.” (DEC, 2012) Methodology In this study, the authors analysed and compared the tweets of 160 pre-service teachers as they experienced their practicum experience in schools. All pre-service teachers who participated in the survey were completing a Bachelor of Teaching/Bachelor of Arts Secondary teaching degree. The pre-service teachers (PSTs) were required to tweet and retweet weekly as part of a compulsory assignment for a minimum 10-week period. The theoretical component of the unit included content on the ethical use of technology and cyber safety. The students’ tweets were analysed quantitatively for the four main areas of major teacher concern to teachers as identified in an earlier study by de Zwart, Lindsay, Henderson & Phillips (2011). Research questions • Can microblogging be used by pre-service teachers as a means of creating a ‘professional learning network’? • What are the ethical issues arise from the use of microblogging in a university course? General Findings The majority of tweets made by the pre-service teachers (PSTs) involved the sharing of ideas, resources and experiences. This was a key indicator that PSTs were using the media for professional learning: 1. To anyone teaching RE during their prac, this resource has helped me a tonne! https://www.smp.org/ Goodluck all #acuedu_s 2. Been looking for a great classroom ICT testing app that doesn't require log in & is fun. Thanks Emily Topher for showing #kahoot #ACUedu_s 3. Quick reminder for my fellow prac teachers: http://www.edutopia.org/blog/classroom- management-tips-novice-teachers-rebecca-alber … #ACUedu_s 4. Have you seen the programming support available for History K-10 on the BOSTES website? http://bit.ly/1Hra4EM #BOSTES #HSIE #HISTORY 5. Some of the PSTs’ tweets provided emotional support, especially when the group was about to embark on a practicum and examinations. 6. “Very concerned about not having a school, when it is almost the end of the semester. #acuedu_s” 7. “As a pre-service teacher it’s okay to not succeed at first. Regardless of what Aubrey’s father says. #ACUedu_s” 8. First day prac teaching. Nervous and excited. #ACUedu_s 9. Experienced teachers from outside the university also provided advice, support and posed reflective questions to the PSTs. Conversations via the media ensued. This provided evidence that the PSTs were networking with those currently working in the profession: CP:43 395 10. "When you're a teacher you have to convince the students that you are smarter than they are" - Dr Wendy Moran #ACUedu_s #truth 11. @amycottonteach giving great advice for casual teaching: don't handout worksheets and sit at front desk- use opp to teach & engage #acuedu_s 12. Do you think wealthy parents should be charged to send their children to public school? #TheProjectTV 13. Experiences should never be ignored as they are references to what will happen. TY to all those precious people shared theirs. #acuedu_s 14. 'Keep calm and pretend it's on the lesson plan' moral: always have a back up plan- @amycottonteach #teachertip #lifetip #ACUedu_s 15. Lecturers from the university also posed questions to the PSTs while they were out on Professional Experience. This created a three-way relationship between the university, the PSTs out in the schools and teachers currently in the field: 16. “Should mentors for pre-service or new teachers be experienced or new teachers themselves? Thoughts? Experiences? #acuedu_s #acuedu663” 17. Finland's education system always near the top in international rankings - why? http://www.theguardian.com/education/2015/jun/17/highly-trained-respected-and-free-why- finlands-teachers-are-different?CMP=share_btn_fb … #ACUedu_s 18. Your responsibility and professionalism as a teacher is always on display eg attendance at schl fete, email, school carnival, etc #ACUedu_s 19. I hope everyone is having a great practical teaching experience! #ACUedu_s Discussion: The Ethical Considerations From the evidence above, it is clear that this initial study provides evidence of the successful establishment of a professional community of practice in which PSTs, their university lecturers and currently practicing experienced teachers kept each other informed about resources, their teaching experiences and provided a means of support and advice. However, using social media in an educational context does not come without its dangers. Some educators shy away from using social media in their classrooms because of safety and/or classroom management concerns. Areas of teacher concern with using social media in the classroom The PSTs tweets were analysed against the four main areas of major teacher concern to teachers have been identified in an earlier study by de Zwart, Lindsay, Henderson & Phillips (2011). They were: 20. Privacy A major concern for this project was the issue of privacy. “Nearly every country in the world regards privacy as a fundamental human right in their constitution, either explicitly or implicitly” (Hartman, 2001). The ease of sharing this information via social media communities compounded the need for guidelines for social media usage in the classroom. In Australia, the Federal Privacy Act of 1998 outlines the basic forms of privacy which can be applied to social media (McNamee, 2005). When the PSTs tweets were examined, a number of issues regarding privacy were identified: One PST had set up her twitter account so that only people to whom she had permitted to be a follower were able to access any of her tweets; “@*****'s account is protected.” One student videoed another PST in an outside lecture and then posted it to the hash tag, raising the question was permission sought? One PST posted his email address; ” Pls email me - ###@acu.edu.au”and another posted her personal telephone number – “@xxxxx001 Hey this girl is looking for you, she wants you to text her # xxxx xxxx” CP:44 396 Another tweet included a photo of the student car park with a badly parked clearly showing the car and its number plate. “Not enough self-efficacy to get closer to the kerb? #acuparkingfail #acuedu_s” 21. Sharing inappropriate material There is an opportunity for photos, videos or sound recordings to be uploaded to social media sites, such as YouTube or FaceBook (Andrews, Dyson, Smyth & Wallace, 2011). The concern is not only that these materials are being used but the ease with which they can be copied, shared, widely distributed (Dunphy, Prendergast & O’Scolai, 2003) and the permanence of the posting (Reilly, 2009). Students can potentially face prosecution under the libel laws by passing what they consider a flippant comment, if it is found they have publicly humiliated a colleague. While no inappropriate material was posted by the PSTs, sometimes the analysis uncovered examples where it might be questioned if students remembered their tweets were publically accessible: “My dilemma: go to today's lecture or the mother's day afternoon tea at my son's daycare?” – Which was followed later by a photo of the PST and her son at the fore mentioned Mother’s Day afternoon tea. 22. Illegal downloading and Plagiarism Having access to vast amounts of information in easily malleable form is often a temptation for many students to make it their own. Breaches of the copyright law and the protection of intellectual property are commonly found on social media sites. This study found there were a lot of images posted to the unit hash tag. Fortunately it was found that the PSTs did not infringe copyright, however, because the posts are public, there is nothing stopping someone else from downloading them and infringing the law. The recommended YouTube clips may not be free from copyright if used in the classroom. Example of a tweet that falls into this category: Doing a unit of work on a fictional novel. Looked at Alice in Wonderland & found a hello kitty version.Ha! #acuedu_s https://www.youtube.com/watch?v=m6tllkG8flk … 23. Cyber bullying Schools have a fundamental duty of care to their students and site blocking is a necessary a key component of their cyber safety strategy (Millea, Galatis & McAllister, 2009). The use of social media by bullies who can hide behind its anonymity and send off offensive messages to their victims anytime is an issue with which our students and teachers must be made aware and be taught to recognize the symptoms in the students they will be supervising. The PSTs in the sample group had been given a lecture on cyber safety as part of their course work which could explain why there were not more examples of inappropriate and unethical tweets. This was an attempt to adopt an approach of appropriately managing the risk of students participating in unethical behaviour through highlighting the potential dangers. Most PSTs refrained from posting inappropriate comments and provided support to their peers. However, there were some examples where the PST tweets were not always kind. A number of tweets made disparaging comments about what people had for lunch, and another commented on a staff members’ shoes. The following comment was made about teachers at the school: Observed an ICT failure today + not being able to help students with an online assign cause teachers don't even have the skills #acuedu_s Conclusion This initial study provides evidence that by using social media, PSTs can successfully establish a professional community of practice in which the PSTs, their university lecturers and currently practicing experienced teachers can keep each other informed and provide a means of support and advice. Although this study did not discover any major ethical breaches in the data, it is recognised that it is extremely difficult for educators to monitor interactions on social media between their students and the wider public. For this reason, the authors will continue to explore class protocols and guidelines that could be established to ensure the safety and the ethical behaviour of all involved when using social media in an educational context. This study has identified that discussions on privacy, appropriate material, downloading, copyright law and cyber bullying would be minimum requirements for any educator planning to embark on using social media in their classroom. CP:45 397 References Andrews, T., Dyson, L.E., Smyth, R. & Wallace, R. (2011). The ethics of m-learning: Classroom threat or enhanced learner agency? Proceedings of the 10th World Conference on Mobile and Contextual Learning held in Beijing, China, 18-21 October, 2011, pp. 295-302. Brandenburg, A. (2012). Submission to the Joint Standing Committee on Cyber Safety. Retrieved from: http://acce.edu.au/sites/acce.edu.au/files/Joint%20Standing%20Committe%20Cybersafety.pdf on 17 July, 2015. De Zwart, L., Henderson, M. & Phillips, M. (2011). Teenagers, Legal Risks and Social Networking Sites. Victoria: Monash University. Dunphy, J.M., Prendergast, G.J., O’Scolai, P. (2003). The emergence of camera phones – Exploratory study on ethical and legal issues. Communications of the International Information Management Association, Vol. 3, pp. 121-133. DEC. (2012). Social media guidelines. Retrieved 14 July, 2015 from http://www.scribd.com/doc/54221457/Social-Media-Guidelines-for-Teachers Educause. (2007). 7 things you should know about … Twitter. Retrieved from www.educause.edu/eli on 17 July, 2015. Hartman, L. (2001). Technology and Ethics: Privacy in the Workplace . Retrieved from http://www3.nd.edu/~rbarger/privacyintheworkplace.html on 17 July, 2015. McNamee, A. (2005). Ethical issues arising from the real time tracking and monitoring of people using GPS-based location services, Wollongong University: Thesis. Millea, J., Galatis, H. & McAllister, A. (2009). [SICTAS] Annual report on emerging technologies: Planning for change, SICTAS report. Adelaide: Education.au Limited. Reilly, D. (2009). Ethics 101: 5 online pitfalls that destroy students. Viewed 10 March, 2012, http://www.educationau.edu.au/jahia/Jahia/home/SICTAS/pid/852. Treadwell, M. (2008). PLN: Personal Learning Network: Connect. Collaborate. Contribute. Retrieved from https://21clc.wikispaces.com/PLN on 17 July, 2015. Cameron, L., Tanti, M., & Mahoney, K. (2015). The ethical considerations of using social media in educational environments. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:42-CP:46). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:46 398 Teachers Cloud-based Content Creation in light of the TPACK Framework: Implications for Teacher Education Chris Campbell Aisha Al Harthi Arafeh Karimi The University of Queensland Sultan Qaboos University The University of Queensland With the advent ubiquitous computing, cloud-based content creation is becoming more popular and readily accessible. In Malaysia the government equipped 10,000 public primary and secondary schools with 4G Internet connectivity and a cloud-based learning environment called the Frog VLE. This study investigated the alignment and compatibility the TPACK framework to teachers’ learning designs. A rubric was developed, based on the TPACK framework, and after feedback from an expert panel, 152 cloud-based sites were analysed. Results show that most areas were somewhat aligned with the TPACK framework while three areas were fully aligned and one area was minimally aligned. The fully aligned areas were use of links, design navigation flow and design functionality. The minimally aligned area was interactivity. This research finding can potentially inform teacher education as if specifically taught this can empower teachers when creating cloud-based content. Keywords: learning design, TPACK, teacher education, cloud-based content Introduction With Internet speeds and access increasing cloud-based computing is becoming more common. Cloud-based computing is an “expandable, on demand service and tools that are served to the user via the Internet from specialised data centres” (Johnson, Adams Becker, Estrada & Freeman, 2014, p. 36). This study focused on cloud-based content created by Malaysian teachers as part of the 1BestariNet project which has equipped over 10,000 primary and secondary government schools with 4G Internet connectivity and a cloud-based learning environment called Frog VLE. By investigating the alignment and compatibility of the TPACK framework to teachers’ learning designs it becomes possible to then give some insight of cloud-based content creation to pre-service teachers. This paper will shed light on these implications for teacher education. The research question developed was to what extent are cloud-created learning designs produced by teachers’ compatible with the TPACK framework? This paper further discusses the implications of cloud-based learning design for teacher education. Literature Review As we know, TPACK is an increasingly common way of representing what teachers know about various technologies and how it applies to their teaching (Koehler & Mishra, 2005) while learning designs refer to a variety of ways that student learning experiences can be designed, generally using different digital technologies. Specifically, learning design is described as a ‘framework’ to make explicit the conceptual and practical underpinnings of “a sequence of educational activities” in an online environment (Dalziel, 2008). Oliver (1999) suggests that a learning design comprises of three key elements which are the tasks the learner is required to complete, the resources that support the learners to complete the task and the support mechanisms that exist from the teacher implementing the lesson. Digital technologies and in particular cloud-based learning resources have been evaluated from several perspectives (ie. Dinh et al., 2013). For the purpose of this research, following items, which are in accordance with TPACK and the Learning Design framework, have been selected to investigate the alignment of cloud-based learning resources. One item is the learning outcome and according to Hernández, Gütl, and Amado-Salvatierra (2014), the learning outcome of designed cloud-based learning resources should be clearly specified. Additionally, the instruction and guideline on how learners should interact with content should be clearly provided (Mikroyannidis, 2012). Several studies in the field of online and cloud-based learning have highlighted the importance of including interactive activities to give learners a chance to provide input and modify the information (i.e. Masud & Huang, 2012; McGee & Reis, 2012). The logical CP:47 399 alignment of such activities with the learning outcomes is significant in learning design (Oliver, Harper, Wills, Agostinho, & Hedberg, 2007). Similarly, learning tools should potentially stimulate a high level of learner engagement. These learning tools could include learning games, feedback and reflection tools (Lin, Wen, Jou, & Wu, 2014), or quizzes (Gusev & Armenski, 2014). Use of such interactive and engaging learning material should provide a balance between the use of multimodal materials and tools to accommodate multiple learning preferences (Chang, Chen, & Hsu, 2012). In the process of design, available tools and media in any digital learning system should suit content and learners’ needs (Thomas, 2011). As highlighted by Kop and Carrol (2011), in using additional links, the relevancy of the resources is at the most importance while all links remain functional. The visual design and consistency of the provided content is another important aspect highlighted by Sánchez-Franco et al. (2013). Finally, the navigation flow and transition between all components of designed resource shall remain clear and logical (Boyatt & Sinclair, 2012). A combined perspective of all of these items to view cloud-based designs in virtual learning environments provides good boundaries of potential guidelines for teachers to create digital content in light of the TPACK framework. Methodology and Results A rubric consisting of ten items was used to evaluate 152 cloud-based learning designs (CBLD) that were created by teachers in a virtual learning environment (VLE). Prior to the rubric being used it was evaluated by an expert panel and then changes were implemented (Campbell, Al Harthi & Karimi, 2015). Initial rubric reliability was measured through the internal consistency using Cronbach's alpha coefficient, which was .74, indicating an acceptable reliability (George & Mallery, 2003). After taking comments on board and further discussion the rubric was changed from 12 items to 10 items and two raters were employed to evaluate the 152 learning designs. Rubric items included: 1. Purpose and Objectives 2. Instructions and Guidelines 3. Interactivity 4. Engagement 5. Learning styles 6. Tools and Media 7. Links 8. Visual Consistency 9. Navigation Flow 10. Functionality To test the research question on the level of alignment of the learning designs with the TPACK framework, the following criteria was used: Table 1: Criteria for Rubric Scores Interpretation 6. Score Categories 7. Interpretation of Scores 8. 1-1.75 9. Not aligned 10. 1.76-2.5 11. Minimally aligned 12. 2.6-3.25 13. Somewhat aligned 14. 3.26-4 15. Fully aligned Based on these criteria, Table 2 shows that only three rubric items were found to be fully aligned with the TPACK framework across the 152 sites. These are the use of links, design navigation flow and design functionality. Only one rubric item was found to be minimally aligned with TPACK framework, which is the use of interactivity in the learning designs. The rest of the rubric items were found to be somewhat aligned with the TPACK framework. This was determined by the raters who both scored each site which determined the final score of whether overall the sites were aligned with TPACK and how much. Table 2: Rubric Item Alignment with the TPACK framework 16. Rubric Item 17. Mean 18. Std. 19. How aligned are the CP:48 400 Deviation CBLD? 1. Learning Outcomes 20. 2.632 21. 1.3747 22. Somewhat aligned 2. Instructions & Guidelines 23. 2.704 24. 1.1384 25. Somewhat aligned 3. Interactivity 26. 2.309 27. 1.2247 28. Minimally aligned 4. Engagement 29. 2.803 30. .9281 31. Somewhat aligned 5. Learning Preferences 32. 3.079 33. .7850 34. Somewhat aligned 6. Tools & Media 35. 2.941 36. 1.1053 37. Somewhat aligned 7. Links 38. 3.355 39. 1.1983 40. Fully aligned 8. Visual Consistency 41. 3.184 42. .8721 43. Somewhat aligned 9. Navigation Flow 44. 3.855 45. .5801 46. Fully aligned 10.Functionality 47. 3.730 48. .5635 49. Fully aligned To follow up with the previous analysis, a one sample t test was used. Test results showed that the difference in rubric item rating between the current sample of learning designs and the theoretical mean (2.5) were statistically significant for all rubric items, except learning outcome and interactivity at .05 significance level. Table 1: Descriptive Statistics and Results for a One Sample t Test 55. 95% Confidence 50. Rubric 53. Sig. 54. Mean Interval of the 51. t 52. df Item (2-tailed) Difference Difference 56. Lower 57. Upper 1. Learning 58. 1.180 59. 151 60. .240 61. .1316 62. -.089 63. .352 Outcomes 2. Instruction s & 64. 2.209 65. 151 66. .029 67. .2039 68. .022 69. .386 Guidelines 3. Interactivit 70. -1.921 71. 151 72. .057 73. -.1908 74. -.387 75. .005 y 4. Engagem 76. 4.020 77. 151 78. .000 79. .3026 80. .154 81. .451 ent 5. Learning 82. 9.093 83. 151 84. .000 85. .5789 86. .453 87. .705 Preferences 6. Tools & 88. 4.917 89. 151 90. .000 91. .4408 92. .264 93. .618 Media 7. Links 94. 8.800 95. 151 96. .000 97. .8553 98. .663 99. 1.047 8. Visual 100. 9.67 101. 1 102. .0 103. .684 104. .54 105. .82 Consistency 3 51 00 2 4 4 9. Navigation 106. 28.8 107. 1 108. .0 109. 1.35 110. 1.2 111. 1.4 Flow 03 51 00 53 62 48 10.Functiona 112. 26.9 113. 1 114. .0 115. 1.23 116. 1.1 117. 1.3 lity 19 51 00 03 40 21 N=152; Test Value = 2.5 Discussion and Conclusion From the results there are three areas that are fully aligned with TPACK. These are the use of links, the design of the navigation flow and the functionality design. As suggested in the literature review these are all important areas in site design and possibly most time and effort went into the design of these areas. Generally, from the 152 sties investigated the rest of the rubric items are somewhat aligned with TPACK. While the use of interactivity in the sites was minimally alighted with the TPACK framework. From the T-Test results neither the learning outcomes nor interactivity were statistically significant. This may mean that the teachers did not have enough knowledge and skills to design these areas well enough in the cloud-based environment. CP:49 401 Interactivity is one area where the sites are minimally alighted. This is an area where some sites benefit from learning design and time in creation, while for other sites it may not be needed. Possibility teachers need to think about this in more detail to ensure the optimal amount of interactivity is used in the site and then its relationship to TPACK may be increased. Implications for pre-service teachers This study highlights several important factors for teacher education students. These include that some areas of cloud-based learning design that are easier to relate to TPACK and some are more difficult. Those that are easier to relate to the TPACK framework include the use of links, navigation flow and functionality of the sites. More importantly when creating cloud-based learning design teacher education students should work on learning outcomes, instructions and guidelines, engagement, learning preferences as well as tools and media and visual consistency. Teacher education students may benefit from great understanding of interactivity when creating cloud-based learning designs. Thus, in teacher education, interactivity of cloud-based learning designs would benefit from a greater focus in teacher education programs. The other area is in learning outcomes as although somewhat aligned in this study to TPACK the area was not statistically significant. In conclusion, implications from this small-scale analysis suggest that teacher education programs need to reconceptualise design components in new ways that are more compatible with the virtual cloud environment. References Dalziel, J. (2008). Learning Design: Sharing pedagogical know-how. In T. Iiyoshi & M. S. V. Kumar (Eds.), Opening up education: The collective advancement of education through open technology, open content, and open knowledge. (pp. 375 - 387). Cambridge, Mass: MIT Press. Campbell, C., Al Harthi, A. & Karimi, A. (2015). Evaluation of the Learning Designs of Cloud-based Content using the TPACK Framework. In S. Carliner, C. Fulford & N. Ostashewski (Eds.), Proceedings of EdMedia: World Conference on Educational Media and Technology 2015 (pp. 302- 309). Association for the Advancement of Computing in Education (AACE). Boyatt, R., & Sinclair, J. (2012). Navigating the educational cloud. InWorkshop on Learning Technology for Education in Cloud (LTEC'12) (pp. 179-191). Springer Berlin Heidelberg. Chang, C. S., Chen, T. S., & Hsu, H. L. (2012). The Implications of Learning Cloud for Education: From the Perspectives of Learners. In Wireless, Mobile and Ubiquitous Technology in Education (WMUTE), 2012 IEEE Seventh International Conference on (pp. 157-161). IEEE. Conole, G. (2010). Learning design - Making practice explicit. Paper presented at the ConnectEd 2010: 2nd International conference on Design Education, Sydney, Australia. http://cloudworks.ac.uk/cloud/view/4001 Dalziel, J. (2008). Learning Design: Sharing pedagogical know-how. In T. Iiyoshi & M. S. V. Kumar (Eds.), Opening up education: The collective advancement of education through open technology, open content, and open knowledge. (pp. 375 - 387). Cambridge, Mass: MIT Press. Dinh, H. T., Lee, C., Niyato, D., & Wang, P. (2013). A survey of mobile cloud computing: architecture, applications, and approaches. Wireless communications and mobile computing, 13(18), 1587- 1611. George, D., & Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference. 11.0 update (4th ed.). Boston: Allyn & Bacon. Gusev, M., & Armenski, G. (2014). E-Assessment Systems and Online Learning with Adaptive Testing. In E-Learning Paradigms and Applications (pp. 229-249). Springer Berlin Heidelberg. Hernández, R., Gütl, C., & Amado-Salvatierra, H. R. (2014). Cloud learning activities orchestration for MOOC environments. In Learning Technology for Education in Cloud. MOOC and Big Data (pp. 25-36). Springer International Publishing. Johnson, L., Adams Becker, S., Estrada, V., and Freeman, A. (2014). NMC Horizon Report: 2014 K- 12 Edition. Austin, Texas: The New Media Consortium. Koehler, M. J., & Mishra, P. (2005). What happens when teachers design educational technology? The development of technological pedagogical content knowledge. Journal of Educational Computing Research, 32(2), 131-152. Kop, R., & Carroll, F. (2011). Cloud computing and creativity: Learning on a massive open online course. European Journal of Open, Distance and E-Learning. Lin, Y. T., Wen, M. L., Jou, M., & Wu, D. W. (2014). A cloud-based learning environment for developing student reflection abilities. Computers in Human Behavior, 32, 244-252. Masud, M. A. H., & Huang, X. (2012). An e-learning system architecture based on cloud computing. System, 10(11). CP:50 402 McGee, P., & Reis, A. (2012). Blended Course Design: A Synthesis of Best Practices. Journal of Asynchronous Learning Networks, 16(4), 7-22. Mikroyannidis, A. (2012). A semantic framework for cloud learning environments. Cloud Computing for Teaching and Learning: Strategies for Design and Implementation, 17-31. Oliver, R. (1999). Exploring strategies for on-line teaching and learning. Distance Education, 20(2), 240 - 254. Oliver, R., Harper, B., Wills, S., Agostinho, S., & Hedberg, J. (2007). Describing ICT-based learning designs that promote quality learning outcomes. Rethinking pedagogy for a digital age: Designing and delivering elearning, 64-80. Sánchez-Franco, M. J., Villarejo-Ramos, Á. F., Peral-Peral, B., Buitrago-Esquinas, E. M., & Roldán, J. L. (2013). Users’ Perception of Visual Design and the Usefulness of a Web-based Educational Tool. Procedia-Social and Behavioral Sciences, 93, 1916-1921. Thomas, P. Y. (2011). Cloud computing: A potential paradigm for practising the scholarship of teaching and learning. The Electronic Library, 29(2), 214-224. Wang, W. Y. C., Rashid, A., & Chuang, H. (2011). Toward the trend of cloud computing. Journal of Electronic Commerce Research, 12(4), 238-242. Campbell, C., Harthi, A.A., & Karimi, A. (2015). Teachers Cloud-based Content Creation in light of the TPACK Framework: Implications for Teacher Education. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:47-CP:51). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:51 403 The Next Wave of Learning with Humanoid Robot: Learning Innovation Design starts with “Hello NAO” Xin Ni Chua Esyin Chew Monash University Monash University Today, humanoid robotics research is a growing field and humanoid robots are now increasingly being used in the area such as education, hospitality and healthcare. They are expected to serve as humans’ daily companion and personal assistant in including in education. On the other hand, students may complain that the classroom today is boring and not engaging. Students are using mobile devices extensively but the traditional lectures remain PowerPoints. Is there a educational synergy for integrating a humanoid robot in daily teaching? Responding to the needs, the paper reports a work-in progress pilot study that designs the learning innovation with humanoid robot, NAO. Initial experiences are reported. Rule-based reasoning and progress test design are developed and recommended. The educational program is developed based on the design and pilot tested at the learning and teaching at Monash University Malaysia. Future work and recommendation are discussed in innovative technology engaging learning. Keywords: learning enhancement, NAO robot in education, IT education innovation Introduction and Literature Review Today, robotics is a growing field that received significant attention in the society (IEEE, 2015). One of the various types of robots is the humanoid robot. A humanoid robot is a robot that is built based on the human body structure. Most humanoid robots have a torso with a head, two arms and two legs (Roebuck, 2012). It is often seen to resemble human behavior and cognition, and to perform tasks in a similar way as human being (Aoyagi & Shirase, 2009). Humanoid robots are also being developed for scientific research purpose. In addition, humanoid robots are now widely used in other fields such as education, entertainment and healthcare (George, 2015). They are expected to serve as humans’ daily companion and personal assistant. Some robots act as a teaching assistant for elementary school students (Han & Kim, 2009). One of the famous humanoid robots, NAO, is released in 2008 by Aldebaran Robotics, a French robotics company (Aldebaran, 2015). The latest generation of NAO V5 Evolution launched in 2014. The functionality has improved for a better interaction between the robot and humans (Inbar, 2014). NAO has a powerful and fully programmable platform with various sensors and language capabilities. NAO is widely used around the world for research and education purposes. “In more than 70 countries, he was used in computer and science classes, from primary school through to university” (Aldebaran, 2015). NAO can be a true daily companion; it can sing, dance, play music and talk to people. Based on the comparative review by Chua (2015), it is found that NAO robot is the best choice of all the humanoid robots to use in enhancing learning and teaching based on the following justifications: (1) Language capability: it can speak up to 19 languages; (2) Mobility: it is small and light, easy to carry everywhere by lecturers or students; (3) Cost effectiveness: it is affordable, a lot cheaper compare to other expensive robot such as ASIMO (2015) and iCub (2015); (4) Sensors capability: it has all the general abilities needed to interact with the students and lecturers in a fun and humanoid way; (5) Durability: its battery life can stay longer; (6) Programmable and logic design: it has a powerful and fully programmable platform; (7) Attractiveness and motivation: it creates a “wow effect” for learning and teaching practitioners and develops further motivation to engage students with learning. With these functions, there is a high potentiality that NAO robot can be used to enhance learning and teaching for IT undergraduates and postgraduates. Generally, IT students are more demanding for technologies in learning and assessment. However, it seems to have another form of ‘digital gap’ between students and educators where educators may not meeting IT students’ expectation of blending educational technologies in class (Hiew & Chew, 2015). We need to thoughtfully explore and design the next wave of learning innovation, possibly with the aid of NAO robot. CP:52 404 The Next Wave of learning Innovation Design It is reported that an intelligent robot NAO is claimed to be ‘a star in the world of education’ and is being used in more than 70 countries for learning and teaching (Total Education, 2014). On the other hand, there are gap between student expectations and university learning and teaching. Students may complain that the classroom today is boring and not engaging. Students are using mobile devices extensively but the traditional lectures remain the use of Powerpoints and passive lecturers (Chew & Kalavally, 2014). In addition, a growing number of researches argue that modern technologies may cause concentration problem in the class. According to a recent survey conducted by Pew Research Centre with almost 2500 educators in the Unite States, they found that 87% of the educators feel that the digital gadgets have created an “easily distracted generation with short attention spans” (Jeffries, 2013). Hence, the aim of this work-in-progress research is to explore the thoughtfully integration of NAO robot in traditional class room setting to increase students’ engagement and learning experiences. These are the proposed research questions: (1) Can NAO enhances student learning and engagement experience, and how? (2) What are the design principles for NAO educational program? In addition, a research suggests that there is a lacking of students’ independent problem solving and communication skills in the pool of Malaysia Engineering and IT graduates (Tan, 2015). Students are taught and assessed in the same way the lecturers were being taught two decades ago. There is a need for rethinking and redesigning the learning and teaching. Since NAO robot has a powerful and fully programmable platform with various sensors and language capabilities, the paper presents the design and implementation of NAO educational program that is aimed to enhance learning and teaching innovation experience. Attracting and developing new generations of engineering and IT experts and conducting scientific research with NAO for seamless learning is the design principles. Engagement and motivation is the key driver for the design science of NAO in introducing IT education. The enhanced learning and teaching experience such as interactive learning, multi- language programming environment and NAO educational program implementation are the expected outcomes. Robot Model and Program Development The model of NAO robot used in this project is the latest version, which is the NAO V5 Evolution as depicted in Figure 1. The development environment is Python with Choregraphe 2.1 (2015). Choregraphe is a cross-platform development environment designed by Aldebaran Robotics that can implement NAO’s actions through logic- and graphics-based programming. It provides the functionality to create NAO robot application which includes the behaviours and dialogues, such as interacting with the audience, singing and dancing. Developers can monitor the behavior of the robot using the Robot View feature in Choregraphe. The strength of Choregraphe is that it allows developers to add customised behaviors to or further mechanisms of the robot using their own Python, C++ or Java code. Figure 1: NAO V5 Evolution Methods, Design principles and Limitations An educational program with Q&A sessions on NAO is developed and pilot tested in two teaching subjects: one undergraduate (with 240 students) and one postgraduate subjects (with 6 students). The NAO educational program we developed consists of interactive concept/ theories explanation and Q&A sessions. There is only one robot in the class that a lecturer can use for teaching more complex concepts and to engage students for interactive Q&A. Lecturers brought NAO to the class and integrated the newly developed NAO educational program in teaching and assessment. Students’ engagement were observed and reflected. The initial design principles for developing the NAO educational program for learning and teaching innovation are as follows: CP:53 405 (1) Developing the “Factbase ” using User Stories (e.g. challenging concepts / theories to teach with Q&A) • This will facilitate the interactive teaching and Q & A sessions between NAO and students. (2) Developing the interpreter for the rules [inference engine]: recognizes and executes a rule- based system whose conditions have been satisfied. This control is data driven (forward). • the interpreter of NAO robot, voice-to-text recognition engine need to be programmed with IF-THE-ELSE conditions to let NAO understand the interactive teaching and Q&A. (3) Developing the Rule-based: Sample of algorithm, Activity Diagram & Description of Design • The design of Q&A sessions, the flow / selections of the questions and a function to be able to calculate the total scores/ marks for students’ understanding and performance of the related topic. (4) Developing the NAO education program to send out email to students or lecturers for the engagement activities and scores. However, the above principles need to be further tested in a wider spectrum of subjects and with various students to be generalised. The learning materials and facts (data) in NAO robot might be outdated. Hence, data need to be updated frequently. Also, students might feel difficult to interact with NAO robot if this is their first time speaking with a humanoid robot. Therefore, a user manual guide needs to be provided for how to interact with NAO robot. NAO robot might not understand what the students or lecturers say if his/ her pronunciation is inaccurate or the voice is unclear. Thus, NAO robot should be able to react to the users and ask them to repeat their words. NAO robot can only recognise one voice at a time. If there are multiple voices at once, he cannot interpret voices correctly and hence, causing voice recognition problems. Internet connection is also another major constraint. NAO educational program needs to run on NAO robot with Internet connection. NAO robot will react slower and its voice recognition will become less accurate if the Internet / Wifi connection is weak. Hence, it is important to have a strong Internet connection to connect NAO with the computer that controls it. Initial Observation and Reflection The learning innovation with NAO is piloted in two teaching subjects, one first year degree programming subject and the other one is a master in business information system subject. These are some preliminary observations: 1. Not all students in the large student cohort of undergraduate subject (240 students) were fully engaged throughout all 2 hours class of teaching. However, the students’ engagement is tremendously high after introducing NAO educational program to explain certain concepts. All students were paying full attention during the Q&A sessions with NAO for fun interaction. 2. The learning with NAO experience at the postgraduate subjects (with 6 students) level is similar as described in point no 1. Students were prompted and energised to see NAO and engage with the learning process. 3. There is an impression that the ‘lecturer + students + power point’ is equal to a ‘boring lecture’. With the use of an interactive technology and ‘lively being’, NAO robot, students are motivated to learn and participate in the discussion. 4. Both students’ engagement and motivation are disrupted and enhanced by introducing NAO in the class. This phenomenon may decline after the initial ‘exciting moments’, comparable to those were the days when power point or mobile teaching were first introduced. More importantly, it is the design principles and best practices to embed NAO in enhancing learning experiences matters. These are the research gap to be investigated for future work. We would argue that the learning engagement paradigm has shifted from manual engagement to personal response system (i.e. clickers), and now with NAO robot. With the invention of robotic technologies in the 21st century, innovation in higher education using intelligent robots has become a challenging but transformative research in design and implementation. Students can learn the educational experiences in higher education with the human-NAO interactions. The design of proposed NAO educational programs enables students to practically connect theory with practice through problem solving, fun question and answers and high level of motivation for futurists’ perspectives. The level of learning engagement and experience is much enhanced. For programming subjects, students can design the algorithm and apply the programming concept to a moving robot CP:54 406 than a static system / website / mobile app (this aspect is not tested in the pilot study and yet to be explored). Concluding Remarks and Recommendation With the invention of robotic technologies in the 21st century, innovation in higher education using intelligent robots has become a challenging but transformative research in design and implementation. Students can learn the educational experiences in higher education with the human- NAO interactions. The design of the proposed and implementation of NAO educational programs enable students to practically connect theory with practice through problem solving, question and answers and high level of motivation for futurists’ perspectives. The learning innovation with NAO has been piloted in two teaching subjects with confirming experiences. The future work is to expand the innovation to more teaching subjects in School of IT at Monash University Malaysia and to design a framework of introducing NAO educational program based on a larger scale of experimental research. User experiences from both students and lecturers will be investigated. A comparative study between teaching IT and non-IT subjects with NAO can be explored. The level of learning engagement and motivation, enhancement or disruption of independent learning will be explored in the future work. We believe that the next wave of learning innovation no longer lies at e-learning or mobile learning but, a thoughtful integration of face-to-face learning with humanoid robot. References Aldebaran (2015). Who is NAO? Retrieved 11 Aug 2015 from: https://www.aldebaran.com/en/humanoid-robot/nao-robot Aoyagi, S. & Shirase, K. (2009) Service Robotics and Mechatronics, Springer Science and Business Media, pp.101-112 Retrieved 11 July 2015 from: https://books.google.com.my/books?id=aYye0cf1Kf4C&pg=PA101&lpg=PA101 ASIMO (2015). Retrieved 11 Aug 2015 from http://asimo.honda.com/ Chew, E. & Kalavally, V. (2014). Learning and Teaching Innovation in School of Engineering and School of Information Technology, Celebrating Innvative Learning Practices: Opportunity for Cmomunity Engagement and Collaboration, Monash University, 29 October 2014. Choregraphe (2015) Retrieved 11 Aug 2015 from: http://doc.aldebaran.com/2- 1/news/2.0/choregraphe_rn2.0.html Chua, X. N. (2015) FIT3060 FinalYear Project Report. School of IT, Monash University Malaysia, pp.4-10. George, D. (2015) Thirteen Advanced Humanoid Robots for Sale Today. Retrieved 11 July 2015 from: http://www.smashingrobotics.com/thirteen-advanced-humanoid-robots-for-sale-today Han, J. & Kim, D. (2009) r-Learning services for elementary school students with a teaching assistant robot, Human-Robot Interaction (HRI), 2009 4th ACM/IEEE International Conference, 11-13 March 2009, pp.255-256. Hiew & Chew (in press) Seams Remain in Seamless Learning, On the Horizon. iCLUB (2015). Retrieved 11 Aug 2015 from: http://www.icub.org/ IEEE (2015). IEEE-RAS Technical Committee on Humanoid Robotics. Retrieved 11 July 2015 from: IEEE (2015). IEEE-RAS Technical Committee on Humanoid Robotics. Retrieved 11 July 2015 from: http://www.humanoid-robotics.org/ Inbar, E. (2014). NAO Evolution V5 - Official Release, 2014. Retrieved 11 July 2015 from: http://content.robotslab.com/blog/nao-evolution-v5-official-release Jeffries, D. (2013) Is technology and the internet reducing pupils' attention spans? Retrieved 11 Aug 2015 from: http://www.theguardian.com/teacher-network/teacher-blog/2013/mar/11/technology- internet-pupil-attention-teaching Roebuck, K. (2012) Mobile Robot: High-impact Emerging Technology What You Need to Know: Definitions, Adoptions, Impact, Benefits, Maturity, Vendors. Retrieved 11 July 2015 from: https://books.google.com.my/books?id=dSEPBwAAQBAJ&pg=PA19&lpg=PA19 Tan, Y. T. (2015) “Pedagogy for Electronics and Electrical Engineering and Information Technology Education in Malaysia in the 21st Century” Global Multidisciplinary Academic Conference (GMAC 2015), 6-7 April 2015, Langkawi. Total Education (2014). Top 5 Robotics Companies in Education, 2014. Retrieved 11 Aug 2015 from: CP:55 407 http://blogs.terrapinn.com/total-education/2014/10/20/top-5-robotics-companies- education/?utm_campaign=Total%20Education%2013112014.html&utm_medium=email&utm_sou rce=Eloqua Chua, X.N. & Chew. E. (2015). The Next Wave of Learning with Humanoid Robot: Learning Innovation Design starts with “Hello NAO”. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:52- CP:56). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:56 408 Loop: A learning analytics tool to provide teachers with useful data visualisations Linda Corrin, Gregor Kennedy Paula de Barba University of Melbourne University of Melbourne University of Melbourne Aneesha Bakharia Lori Lockyer Dragan Gasevic University of South Australia Macquarie University University of Edinburgh David Williams Shane Dawson Scott Copeland University of Melbourne University of South Australia University of South Australia One of the great promises of learning analytics is the ability of digital systems to generate meaningful data about students’ learning interactions that can be returned to teachers. If provided in appropriate and timely ways, such data could be used by teachers to inform their current and future teaching practice. In this paper we showcase the learning analytics tool, Loop, which has been developed as part of an Australian Government Office of Learning and Teaching project. The project aimed to develop ways to deliver learning analytics data to academics in a meaningful way to support the enhancement of teaching and learning practice. In this paper elements of the tool will be described. The paper concludes with an outline of the next steps for the project including the evaluation of the effectiveness of the tool. Keywords: Learning Analytics, Higher Education, Learning Design, Data Visualisation. Introduction Learning analytics offers the potential to deliver data to teachers in ways that can inform the design and facilitation of learning activities. In order to realise this potential it is necessary to develop tools that allow student data to be analysed and visualised in ways that take into consideration the pedagogical design of learning tasks. This paper reports on the current progress of a project, funded by the Australian Office for Learning and Teaching (OLT), which explores what analytics would be useful for teachers in higher education to inform the development of a generic learning analytics tool. In previous papers we have outlined the overarching approach to the project and the preliminary findings from the first stage of the research (Kennedy et al., 2014). In this paper we will showcase the design of the tool and outline the next steps that will be taken to evaluate the effectiveness of the tool within real teaching scenarios. Advances in learning analytics approaches and methods have resulted in calls for tools that can help academics access and interpret data about their students’ engagement in online learning environments (Drachsler & Geller; 2012). Previous tools have provided different forms of visualisations of student data from learning management systems including access to resources (Di Bitonto, Pesare, Roselli & Rossano, 2015), assessment (Petropoulou, Kasimatis, Dimopoulos & Retalis, 2014), and social interaction analysis (Schreurs, Teplovs, Ferguson, De Laat & Buckingham Shum, 2013). While these tools provide different ways of viewing data about students’ engagement with online resources and activities, they do not facilitate an explicit connection with learning outcomes and pedagogical design. The Loop tool provides student data to teachers on students’ learning interactions and processes to inform teaching and feedback dialogues with students. A fundamental premise of the Loop tool is that the teachers’ pedagogical intent, which underpins the design of a learning activity, drives the framing of the analytics that are presented for each technology-based tool. This means that the data associated with students’ interactions with learning material can be used to determine the extent to which the pedagogical intent of the task has been reflected in students’ interactions. These premises are underpinned by the field of learning design and the Laurillard’s (2002) conversational framework. The learning design field provides a context through which more meaningful interpretation of student CP:57 409 data can be made (Lockyer, Heathcote, & Dawson, 2013) by articulating the pedagogical intent of the learning activities to which student engagement data can be compared. The conversational framework highlights the importance of interaction, dialogue and feedback between teachers and students, which should take place in an iterative cycle. Analysis of student data provides an evidence base to inform such feedback and remediation processes between teachers and students. The Loop Tool The Loop tool is an open source analytics application, developed as part of the OLT project, that allows users to visualise student data from learning management systems in a meaningful way. Loop is made up of two components: a log processing component that includes a data warehouse, and a web application that includes dashboards and reports. The Loop tool is programmed in Python using the Django web application framework. The tool is able to process logs from both the Moodle and Blackboard learning management systems to produce a number of different data representations and visualisations. Importantly, the tool is designed to integrate the course structure (i.e., course hierarchy) and course schedule in its associated visualisations to allow teachers to evaluate the effectiveness of learning activities scheduled throughout the course. The word ‘course’ has been used within the tool to represent a single subject or unit of study. Loop allows teachers to define key learning tasks or events (e.g. exam or non-instruction weeks) which are incorporated into the data representations in the tool to support the interpretation of patterns of student activity. These critical events allow for either isolated (e.g., mid-term exam) or recurrent events (e.g., weekly lectures) across the course to be defined. After its trial and before its formal release, the tool will be supplemented with a pedagogical helper tool that will assist teachers in disaggregating the learning objectives and learning design of their course. It will be this element of the Loop tool that will assist teachers to articulate their pedagogic intent and identify metrics that relate to their design intentions. When used together, the two tools will help teachers see the degree to which students’ interactions with learning tasks and assessments are manifest in LMS data corresponding with the pedagogical intent envisaged by teachers. The three main sections of the Loop tool are: dashboard, course access, and students. Dashboard The course dashboard provides a quick overview of activity within the course. Figure 1 provides a screenshot of the dashboard with data from one course with which Loop is being piloted. This dashboard summary of the data can be viewed for a particular week or for the entire course. The first representation of the dashboard is associated with “Daily Page Views” (Figure 1-A). This displays simple metrics of students’ levels of activity over time for three key three key areas of a course: content, communication, and assessment. The vertical lines on the daily page views graph represent the dates of teacher-identified critical learning events in the course (Figure 1-B). This visual representation allows teachers to easily see how students’ interactions with critical course materials are influenced (or unaffected) by critical events or key milestones within the course. A representation of “week metrics” (Figure 1-C) presents a digestible summary of key weekly activities in the course including the number of unique page views, the number of unique students who were active, the number of student sessions conducted, the average number of sessions, and the average number of page views per session. The dashboard also presents a list of the course material most visited by students over the time period (e.g., “Top Accessed Content”; Figure 1-D). CP:58 410 Figure 1: The Loop tool dashboard for week 1 of a course. A = Daily page views. B = Critical learning events. C = Week metrics. D = Top accessed content. Course Access The course access section provides teachers with the opportunity to explore, in more detail, students’ access and interaction with the course. The three-fold classification of content, communication, and assessment is again used to structure teachers’ exploration and investigation of the learning analytics data. Each of these sub-sections is described below. Content The content section displays all students’ interactions with the content material in the course. Teachers can elect to see the total number of student views for each item of content material within the course structure, per week and aggregated across the whole course. Teachers can also elect to see “unique student views” which shows how many unique students viewed each item of content material (either per week or in total aggregate). From either of these two sets of data teachers can access further details on any specific item of content. Figure 2 shows this more detailed view for a course content item. It and shows how the level of student access can be visualised in relation to key learning events over a three-month period. Teachers also have the option with this detailed view to see a histogram of access frequencies, and a list of students who have not yet accessed the particular item of content. Figure 2: A three-month visualisation of access to a course content item aligned with key learning events Communication The communication section contains visualisations of students’ interactions with discussion forums across the course. In this section it is possible to see the number of views and posts for each of the forums available in the discussion board, both per week and in total. The number of unique students CP:59 411 viewing each forum is also available. The numbers of views for each forum included in the discussion board in relation to a selected learning event date, are presented visually (see Figure 3). Figure 3 shows students’ access to discussion forums relative to the specific event of the weekly lecture. Teachers may, for example, be interested in whether students are actively engaging in or viewing discussion posts on a specific topic before or after a particular lecture or class. The visualisation shows how many discussion views have occurred (indicated by the size of the circle), as well as the proportion of discussion forum views that occurred before the learning event (blue circle segments) or after the learning event (red circle segments). Figure 3: A visualisation of student access to discussion forums in relation to course events Assessment The assessment section displays data related to students’ interaction with each assessment available across the course. The number of attempts for each assessment, per week and in total, are presented in a primary table. Teachers also have the ability to view the same data filtered by access by unique students. A “grade” table presents students’ grades for each assessment within the course. An “event” table presents students’ interaction with each piece of assessment in relation to a defined learning event, using the same visual representation techniques (i.e. size and colour) as those used in the content and communication sections. Students The students section of the tool provides the ability to monitor the activity, communication and progress of individual students. An aggregated table presents an overall count of activity for each student. When an individual student is selected, specific data related to the student’s interaction with content, communication and assessment can be viewed. This section can be used by teachers who are interested in investigating engagement patterns of students, potentially to identify those who are struggling with the course. For example, the teacher can check when a student has accessed content materials that relates to particular poor assessment attempts by the student, or whether a student is engaging with other peers on discussion forums after they have been set a topic or problem to investigate. Next steps The Loop tool will now be piloted in four courses across the three institutions participating in this project: a biomedical course and a pharmacology course at the University of Melbourne, an education course at Macquarie University, and an accounting course at the University of South Australia. These pilot studies will take place in the second semester of the 2015 academic year. Teaching staff involved in each pilot will take part in an interview at the beginning of semester during which they will be introduced to the functions of the tool and asked how they expect they will use the tool to support their teaching practice. Throughout the semester participants will be asked to complete a short diary entry each time they access the tool to record the purpose of their use and any outcomes and/or actions that results from viewing the data. At the end of semester they will be interviewed again and asked to reflect on their use of the tool, the results of any educational interventions made, and the usability of the tool. The findings of this data collection process will inform any changes to the tool CP:60 412 before it is released as an open source tool for other institutions to use. Moreover, the pedagogical helper tool will be developed, which will assist teachers in articulating their pedagogic intent of their course and its activities, based on its learning objectives and learning design. Conclusion The Loop tool represents a step forward in providing learning analytics that are able to support and inform learning design. The tool is unique in that it has been designed to provide dashboards, reports and visualisations that incorporate course hierarchy (i.e., content tree structure), course schedule (i.e., reporting for each week in the semester) and key course events (i.e., recurring events such as lectures and single assessment submission dates). This structure, manifest in Loop’s design and interface, is essential to give teachers an understanding of the how students’ learning behavior is temporally related to the pedagogical structure of their course, and the specific learning activities within it. References Kennedy, G., Corrin, L., Lockyer, L., Dawson, S., Williams, D., Mulder, R., Khamis, S., & Copeland, S. (2014). Completing the loop: returning learning analytics to teachers. In B. Hegarty, J. McDonald, & S.-K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014 (pp. 436-440). Di Bitonto, P., Pesare, E., Roselli, T., & Rossano, V. (2015). Digitally Enhanced Assessment in Virtual Learning Environments, In Proceedings of 21st International Conference on Distributed Multimedia Systems, Vancouver, Canada. Drachsler, H., & Greller, W. (2012). Confidence in learning analytics. In Proceedings of the Second International Conference on Learning Analytics and Knowledge (pp. 83-92).New York, NY: ACM. Laurillard, D. (2002). Rethinking University Teaching. A conversational framework for the effective use of learning technologies. London: Routledge. Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57(10) 1439–1459. Petropoulou, O., Kasimatis, K., Dimopoulos, I., & Retalis, S. (2014). LAe-R: A new learning analytics tool in Moodle for assessing students’ performance. Bulletin of the IEEE Technical Committee on Learning Technology, 16(1), 1-13. Schreurs, B., Teplovs, C., Ferguson, R., De Laat, M., & Buckingham Shum, S. (2013). Visualizing social learning ties by type and topic: Rationale and concept demonstrator. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 33-37). ACM. Corrin, L., Kennedy, G., Barba, P.D., Bakharia, A., Lockyer, L., Gasevic, D., Williams, D., Dawson, S., & Copeland, S. (2015). Loop: A learning analytics tool to provide teachers with useful data visualisations. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:57-CP:61). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:61 413 Teaching Complex Theoretical Multi-Step Problems in ICT Networking through 3D Printing and Augmented Reality Michael Cowling James Birt School of Engineering & Technology Faculty of Society and Design CQUniversity, Australia Bond University, Australia This paper presents a pilot study rationale and research methodology using a mixed media visualisation (3D printing and Augmented Reality simulation) learning intervention to help students in an ICT degree represent theoretical complex multi-step problems without a corresponding real world physical analog model. This is important because these concepts are difficult to visualise without a corresponding mental model. The proposed intervention uses an augmented reality application programmed with free commercially available tools, tested through an action research methodology, to evaluate the effectiveness of the mixed media visualisation techniques to teach ICT students networking. Specifically, 3D models of network equipment will be placed in a field and then the augmented reality app can be used to observe packet traversal and routing between the different devices as data travels from the source to the destination. Outcomes are expected to be an overall improvement in final skill level for all students. Keywords: mixed media visualization, networking, augmented reality, 3D printing, ICT Introduction As educators, we are increasingly surrounded by a new breed of individual - those that have never known a world where computers weren’t commonplace. These so-called ‘Digital Natives’ (often defined as those born after 1980) are described as being naturally fluent with a variety of digital technologies, with a distinctive set of characteristics that seems to be natural, including preference for speed, nonlinear processing, multitasking, and social learning thanks to their embedded life in digital technology during childhood and adolescence when neural plasticity is high (Prensky, 2001; Thompson, 2012). This new generation of students, later described as “Digitally Wise” by Prensky (2009), approach learning using multiple different types of available technology (Thompson, 2012), working with technology and their applications from a technology understanding rather than from a classical educational understanding. In particular, Jones, et al. (2009), points out that these students expect to be engaged by their environment, with participatory, interactive, sensory-rich, experimental activities (either physical or virtual) and opportunities for input. They are more oriented to visual media than previous generations and they prefer to learn visually by doing rather than by telling or reading. Students studying Information Communication Technology (ICT) could reasonably be expected to be the epitome of the “Digital Native” described above. Yet despite this new breed of student with a preference for learning visually, the representation of theoretical concepts without a corresponding real world physical analog model and the simulation of complex multi-step processes in the classroom is still a developing issue. For instance, in ICT the pedagogical approach of teaching programming has been discussed at length over a number of years by a number of researchers (Krpan, Mladenović, & Rosić, 2015; Pears et al., 2007), with the literature acknowledging that it is hard to teach students the problem solving and complex multi-step tasks required in the ICT discipline. In the teaching and learning of computer networking (the context for this study), this has been investigated with the development of virtual environments for modeling the processes (Dobrilovic, Jevtic & Odadzic, 2013; Powell et al., 2007) and abstract video based visualizations (https://youtu.be/-6Uoku- M6oY). However, networking models are complex to set up with software and require extensive reworking of existing network facilities. Abstract visualizations also don’t capture the complexity of the logical models, specifically the complexity and multi-step nature of the traversal of packets along the layers of the fundamental OSI-TCP/IP packet networking model. There is also a potential issue with interpretation of these models by students from varied cultures, as per previous work by the author on international students (Cowling & Novak, 2012). This paper therefore presents a pilot study rationale and research methodology to examine a mixed media visualization intervention using 3D printing and a mobile augmented reality application CP:62 414 programmed through freely available commercial grade visualization tools. The aim of the paper is to present a method to assist students in theoretical model understanding and applied use. In particular, to address the problem that these models are not physical in our existence but rather logical models used to describe packet behaviour at the software and hardware level. Pilot Study Rationale The use of visualizations as positive learning support tools are well documented and accepted (Mayer, 2005, 2008). Numerous academic disciplines incorporate a variety of 2D and 3D visualizations and haptic manipulations including medical anatomy, architecture, geography, chemistry and media/game design (Freitas & Neumann, 2009). This work also builds on previous work by the authors in multimedia design (Birt & Hovorka, 2014) studying the effects on learners building 3D models with applied mixed media visualizations, and paramedic science (Cowling, Moore and Birt, 2015), which studied the application of emerging technologies and comparative mixed media visualization on trainee paramedic science students studying airways management. The fundamental difference between this proposed study and the previous work of the authors is the availability of a direct physical real world model. In networking, and in particular in modelling packet flow network diagrams, this is not the case, with no corresponding physical model that represents the various layers of the networking model in a visual fashion for students. Tasker & Dalton (2008) argue that this creates a mental gap for students, providing a disconnect between their understanding of the concepts and their visual mental model. Further, they argue that visualisations can assist with this by providing students with an appropriate mental model that they can use to understand the “hidden” concepts, as outlined by Williamson et al (2012). This project therefore takes the work done by Tasker et al. and the previous work by the authors and extends it, with an aim to demonstrate that kinesthetic tools can be used to better form mental models (Paas & Sweller, 2014) and deliver improved pedagogy to teach networking concepts to 21st century students from varied cultures. Specifically, a combination of augmented reality through a mobile device and 3D printed models will be used to visualise how data travels through various network components from source to destination, addressing the following research questions: i) How does 3D printing and augmented reality impact 21st century student learning in ICT networking courses?; ii) How does 3D printing and augmented reality affect learning for students from varied cultures?; and iii) How does 3D printing and augmented reality assist ICT networking students in visualising complex multi-step processes? Experimental Design Participants in this work are students enrolled in the undergraduate networking course at the lead author’s institution. To conduct the experiment, the 3D printing and augmented reality intervention will be implemented into three standard tutorial exercises for the class. For each exercise, the student cohort will be split, with some students being given access to the new tools and some students using the traditional approach to the exercise. The groups completing the exercises with the new tools will be rotated to ensure that each individual student has equal access to both the intervention and the traditional methods. CP:63 415 Figure 1: An example simple network model & the equivalent view using augmented reality markers on an iPad The specific intervention involves the use of 3D printed networking components that are scanned by a mobile device using the Qualcomm Vuforia plug-in (www.vuforia.com) and an app developed in Unity3d (www.unity3d.com). Whilst previously being limited to game development and high end engineering projects, these tools are now becoming available to education. Specifically, 3D printing has seen an explosion in the past five years due to low cost fused deposition modeling (FDM) TM systems by makers such as MakerBot (www.makerbot.com). 3D printing at its basic level uses an additive manufacturing process to build objects up in layers using plastic polymer. Although the process is slow, 3D printing creates direct links between a virtual 3D based model and the formation of an accurate, scaled, physical representation from that model (Loy, 2014). This direct linking of object making to computer modeling changes the relationship of the learner to the making of the object and subsequent use, that is, it creates and enables a haptic feedback loop for learners. Using the 3D printed components as a tool, the app will then identify each component and use them to construct a custom network on the device based on the placement of the 3D printed items in the field by students. For instance, instead of using a traditional and static 2D model, 3D models of computers, switches and routers will instead be placed by students to construct a network that will then be imported into the mobile device (Figure 1). Once the network is in the mobile device, students will be able to simulate network traffic, visualising the complex multi-step process of the OSI and TCP/IP model (see Table 1). Students will also have the ability to rearrange 3D objects to understand how changes in the network infrastructure affect the performance of the network, providing them with a mental model for this complex process, in line with Tasker & Dalton (2008). Table 1: The Internet Protocol Suite (commonly TCP/IP model) TCP/IP Model 5. Application Includes protocols used by most applications (HTTP, FTP etc) Layer 4. Transport Layer Data Channel creation (end-to-end services, error and flow control etc) 3. Internet Layer Multiple network level (internetworking) 2. Data Link Layer Local network level 1. Physical Layer Actual physical wires and connections Research Method The theoretical framework underpinning this work will be action research (Kemmis, 2006), with each ‘loop’ in the research being conducted within a single term and with a different cohort of students, and the in-classroom implementation of the 3D printing and augmented reality intervention supplemented by research conducted with students to assess their feeling about the technology and its use in the classroom. The action research paradigm is appropriate because the researchers will work as practitioners in the classroom, implementing the change whilst simultaneously performing research to determine its effectiveness. Action research as a framework also implements an interactive inquiry process well suited to answering the research questions on student learning, teaching practice and visualisation of complex multi-step processes. To provide research data, a pre-test will first be conducted with students to assess their base knowledge, and then selected students will be asked to volunteer to complete an intervention. After the implementation of each exercise all students (both those completing the intervention and those completing the exercise in the traditional way) will be given a small post-exercise quiz to assess their knowledge of the concepts being covered. This will provide useful data on whether the implementation has made a difference to student results and address the research question “How st does 3D printing and augmented reality impact 21 century student learning in ICT networking courses?”. It is anticipated that approximately 50 students (domestic and international) will be able to participate in the experiment in total, after ethics approval is given and consent is sought from the students. CP:64 416 In addition to this experiment, at the end of the term students will also be issued with a survey asking how they felt about the use of the new tools and how they felt that they enhanced their learning. Survey questions will be developed based on existing theory on the digital competency of students and will include demographic questions as well as Likert scale quantitative questions to assess student feeling on the new tools, allowing for correlation between student demographics (such as international and domestic student details, age, gender etc) and the attitude to the research, amongst other factors, and answering the research question “How does 3D printing and augmented reality affect learning for students from varied cultures?”. Finally, as part of the end of term survey, an open-ended qualitative question will also be included for students to provide additional detail on their use of the new tools as desired, and it is here that answers may be found to the research question “How does 3D printing and augmented reality assist ICT networking students in visualising complex multi-step processes?”. However, due to the complexity of this question, and depending upon the survey results, an online focus group may also be conducted to collect further rich data on student experiences that relate to this research question. Ethics approval for this survey and the possible focus group will be obtained from the Human Ethics committee prior to administration. A combination of both quantitative and qualitative data will be collected from the quiz results and the survey instruments. Quantitative data will be analysed using SPSS to identify significant levels of difference in student satisfaction and to analyse whether a significant difference in student outcomes was identified. Qualitative data will be analysed using NVivo and coded to identify significant themes present in student comments. Conclusion This paper has presented a proposed pilot study involving a learning intervention using mixed media visualisation (3D printing and Augmented Reality simulation) to help teach complex multi-step problems to students studying computer networking in an ICT degree. Through the use of an action research paradigm, several tests will be performed at various stages to assess this assertion and student performance at the simulated task. In addition, a survey will be conducted to assess student attitude towards the intervention methods. Future work will report on the results of this study and provide correlations of various factors related to student performance, showing whether the use of these interventions have improved learning and whether the tools were accepted by the student cohort. Through this work, a greater understanding of the use of innovate technology tools and games simulation in the education space will be obtained, providing a foundation for future research. References Birt, J. & Hovorka, D. (2014). Effect of mixed media visualization on learner perceptions and outcomes. In Proc 25th Australasian Conference on Information Systems. 8-10 December. Auckland, New Zealand. Cowling, M. A., Moore, E, Birt, J. (2015). Augmenting Distance Education Skills Development in Paramedic Science through Mixed Media Visualisation, Proceedings of EUROMEDIA'2015, April 27-29, 2015, IST, Lisbon, Portugal. Cowling, M. A., Novak, J., (2012). Tweet the Teacher: Using Twitter as a Mechanism to Increase rd Classroom Engagement, Proceedings of ISANA International Education Association 23 Annual th th Conference, 4 – 7 December, 2012, Auckland, NZ. Dobrilovic, D. M., Jevtic, V. Z., & Odadzic, B. (2013). Expanding usability of virtual network laboratory in IT engineering education. International Journal of Online Engineering (iJOE), 9(1), pp-26. Freitas, S. d., and Neumann, T. 2009. The use of ‘exploratory learning’for supporting immersive learning in virtual environments, Computers & Education, (52:2), pp 343-352. Jones, C, Ramanau, R, Cross, S & Healing, G (2010), Net generation or Digital Natives: Is there a distinct new generation entering university?, Journal of Computers and Education, vol.54, pp.722-732 Kemmis, S. (2006). Exploring the relevance of critical theory for action research: Emancipatory action research in the footsteps of Jurgen Habermas. in P. Reason and H. Bradbury (eds.), Handbook of action research (pp. 94 - 105). SAGE Publications, London. CP:65 417 Krpan, D., Mladenović, S., & Rosić, M. (2015). Undergraduate Programming Courses, Students’ Perception and Success. Procedia-Social and Behavioral Sciences, 174, 3868-3872. Loy, J. (2014). eLearning and eMaking: 3D Printing Blurring the Digital and the Physical, Education Sciences, 4(1), 108-121. Mayer, R. E. (2005). Cognitive theory of multimedia learning. The Cambridge handbook of multimedia learning, pp 31-48. Paas, F., & Sweller, J. (2014). 2 Implications of Cognitive Load Theory for Multimedia Learning. The Cambridge Handbook of Multimedia Learning, 27. Pears, A, Seidman, S, Malmi, L, Mannila, L, Adams, E, Bennedsen, J, Devlin, M & Paterson, J (2007), A survey of literature on the teaching of introductory programming, SIGCSE Bull., 39, 4, 204-223. Powell, V. J., Turcheck, J., Wu, P. Y., Franzi, L. C., Johnson, R. S. & Parker, I. W. Vlabnet (2007). A virtual laboratory environment for teaching networking and data communications. Proceedings of ISECON, 1-9. Prensky, M. (2001). Digital natives, digital immigrants. On the Horizon, 9(5), 1-6. Prensky, M. (2009). H. sapiens digital: From digital immigrants and digital natives to digital wisdom. Innovate, 5(3). Retrieved from http://www.wisdompage.com/Prensky01.html Tasker, R., & Dalton, R. (2008). Visualizing the molecular world–Design, evaluation, and use of animations. In Visualization: Theory and practice in science education (pp. 103-131). Springer Netherlands. Thompson, P, (2012), The digital natives as learners: Technology use patterns and approaches to learning, Journal of Computers and Education, vol.65, pp. 12-33 Williamson, V. M., Lane, S. M., Gilbreath, T., Tasker, R., Ashkenazi, G., Williamson, K. C., & Macfarlane, R. D. (2012). The effect of viewing order of macroscopic and particulate visualizations on students’ particulate explanations. Journal of Chemical Education, 89(8), 979-987. Cowling, M. & Birt, J. (2015). Teaching Complex Theoretical Multi-Step Problems in ICT Networking through 3D Printing and Augmented Reality. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:62-CP:66). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:66 418 An investigation of blended learning experiences of first- year Chinese transnational program students at an Australian university Kun Dai University of Queensland The extensive uses of information and communication technologies (ICT) in higher education have reformed the traditional classroom-based study mode. Blended learning, the combination of online and offline learning methods, has become an essential teaching and learning strategy for both instructors and students. An increasing number of Chinese students choose to conduct their undergraduate study through China-Australia transnational programs. Due to the differences in teaching and learning styles between Chinese and Australian universities, the perceptions of transnational students on blended learning strategies may impact their study experience and the adaptation to a different environment. Although previous studies have investigated learning experiences and adaptation issues of Chinese students from various perspectives, limited studies have explored the perceptions of Chinese transnational program students on blended learning in their first-year Australian study. This study describes a series of preliminary qualitative findings of these students blended learning experiences, especially the online section, in an Australian university. Keywords: Blended learning; Online Learning; Transnational education; Chinese students; Higher education Introduction Blended learning has been proposed as a solution to achieve the goal of improving students’ engagement, accessibility and flexibility in the process of ICT-embedded study (Bonk & Graham, 2012). The term, “blended”, implies several meanings on the basis of different perspectives. Researchers (e.g. Bonk & Graham, 2012) summarize that blended learning is a study mode that combines traditional face-to-face styles with online learning methods. In this study, blended learning refers to the combination of offline and Internet-based study approach. The students’ online study mainly focuses on investigating experiences through Internet-based learning platform, for example, Blackboard. With the rapid development of ICTs, many unknown questions in blended learning emerge and need to be addressed. Kim, Bonk, and Teng (2009) argue that the topic of blended learning in Asian countries (e.g., China, Korea and Taiwan) and the use of blended learning in a cross-cultural phase should be investigated in depth and systematically. Transnational education has become an important part of modern higher education (Altbach, Reisberg, & Rumbley, 2009). A growing number of Chinese students choose to conduct their undergraduate study through transnational programs, for example, China-Australia mode, which allows students to experience different cultural environments and educational settings. In particular, transnational program students normally need to study in a Chinese university for at least one year. Due to the differences in culture and education between China and Australia, however, many Chinese students struggle with the transition from a domestic to an international environment. To understand students’ cross-cultural learning experiences, many educational researchers have explored relevant topics from their specific perspectives, such as blended learning in Chinese and Australian higher education (Bliuc, Ellis, Goodyear, & Piggott, 2011; Tang, 2013), issues of using blended learning (Bonk & Graham, 2012), and adaptation of ICTs in different cultural environments (Chen, Bennett, & Maton, 2008). Literature Review A number of researchers have investigated relevant issues about Internet-based blended learning in Australian and Chinese higher education. For the Australian context, Bliuc et al. (2011) explored the CP:67 419 use of blended learning in different curriculums, such as arts and engineering. Furthermore, students’ learning experiences and perception of blended learning strategies have also been identified through critical analysis (Bliuc et al., 2011). According to these research studies, researchers have concluded that students can experience both online and offline study approaches through blended learning depending on their study demands. The positive perceptions of Internet-based blended learning are mainly reflected in three phases: the flexibility of study mode, the abundant resources of the Internet, and offline interactions. Meanwhile, some negative aspects are identified as well, such as information overload, influences on learning and teaching productivity, and the balance between online and offline activities. For the Chinese context, many Chinese researchers have investigated students’ experiences of blended learning (Tang, 2013; Zhang & Han, 2012). Some studies have resulted in many similar findings to those of Australian researchers. For example, in accordance with Graham, Allen, and Ure (2003), Tang (2013) also suggested that the Internet-based blended learning approaches can help students to gain knowledge without the restrictions of time and space. Zhao (2008) resonated with Ellis, Goodyear, Prosser, and O'Hara (2006) who claimed that the Internet can provide a great many educational materials to students. Particularly, based on an investigation of using blended learning strategies in an English course, Zhang and Han (2012) identified that Internet-based blended learning strategies may motivate Chinese students’ self-learning interests and develop autonomous studying skills. Many Chinese students are used to rote and passive learning styles and lack critical and creative thinking (Chan, 1999). In a blended learning environment, students may use Internet-based tools to learn independently rather than only relying on instructors and textbooks (Zhang & Han, 2012). The mixed learning and teaching methods not only provide various educational recourses, but also establish a flexible environment for both instructors and learners (Rovai & Jordan, 2004). Hence, studying in such a blended learning environments may provide learners with more opportunities for thinking and studying autonomously and so enhance their motivation and self-regulation (Tang, 2013). Research Gaps and Questions According to the literature above, it is apparent that few studies have been conducted with the intent of identifying the learning experiences of Chinese transnational programs students to the blended learning environment in Australian universities. Therefore, this study aims to explore Chinese transnational program students’ first-year study experiences in Australian blended learning environments. On the basis of previous studies and introduction above, this study proposes to explore the following research questions: What are the learning experiences of first year China-Australia program students in Australian university through online environment? Research Methodology Six undergraduate students who studied in first-year China-Australia programs at an Australian university participated in this research project. Ethics approval for the project was obtained and the participants were each given a pseudonym to protect their privacy. They include four female (Yan, Hua, Min and Qian) and two male (Lun and Gang). Yan, Hua and Lun are from an accounting major. Min, Qian and Gang are from a design major. To investigate the students’ online learning experiences two focus group interviews were conducted. There were three participants in each of the two groups. Accounting students were in group one with the design students in the other group. Each interview took approximately one hour. Questions focused on exploring students’ online learning experiences of different educational settings and identifying the potential problems that they may struggle with. Based on students’ answers, initial understandings of using blended learning strategies in the selected transnational programs were analyzed. A thematic analysis was used to understand collected qualitative data systematically. In order to obtain precise and in-depth answers, the researcher used Chinese to interview each group and then translated the responses into English. CP:68 420 Results The experiences of learning online in Australian university On one hand, some participants identified that the Internet is fundamental tool in their study. With regards to the ways of using Internet in study, three participants from the accounting group and one student (Qian) from the design group thought that blended modes are more useful in their study, especially in the Australian university. These students provided various views of the reason why blended learning methods were important to their study in the Australian university. They pinpointed that online learning platform, for example, the Blackboard system, was one of the most useful learning online tools. According to interviews, these students agreed that many learning recourses can be found on Blackboard. On the Blackboard system, they can have a comprehensive understanding of the aim, goal, teaching/learning activities and assignments for a subject. For instance, Lun, from the accounting group, stated: There are a lot of online tools that I used during my study. Instructors also apply some online tools during teaching. In specific, I think the blackboard is the most useful online tool. It is a systematic online learning tool and it makes our study flexible and productive. For example, when I want to know the course content, such as reading list, videos and slides, I can easily find out them in the system and do not need to spend more time to search for relevant materials online by myself. Many instructors in my major usually upload the most important learning recourses in the system week by week. Therefore, students can clearly understand what they need to know before and after a class. However, when I study in my Chinese university, there are limited uses of such kind of systems during study. My Chinese university also did not have such useful system, which has all necessary functions for study. Compared to participants who agree that the blended learning mode is more helpful, two students (Min and Gang) from the design group prefer to study in offline environments. For example, Min claimed: Courses of design majors need to create art works through computer software, online tools and hand-drawing. When students need to ask questions about drawing something or request tutors’ feedback on specific drawing techniques, I think online learning strategies are difficult to help students to require feedback effectively because sometimes designing or drawing is hard to explain. So face-to-face learning is essential in my major. Although there is a great deal of online resources, for some courses in my major, I think that it is difficult to learn some specific knowledge through online platforms directly because communication with instructors and peers is important to have inspirations when I want to create something new or different. Therefore, I think online learning is not very useful sometimes for design major. I prefer to study in a face-to-face environment. Based on these statements above, it is apparent that Internet-based tools, for example, Blackboard, can provide various learning resources to students. Furthermore, students who study in different majors have their own understandings and requirements on the use of Internet-based learning tools in traditional face-to-face environments. On the other hand, some participants identify that their Chinese university does not provide useful online learning tools for supporting teaching and learning compared to the Australian university. According to the analysis of their feedback, main issues are highlighted by students: lack of useful online tools, unnecessary information overload, and limited uses of ICTs in teaching. Yan and Hua, from the accounting group, noticed that although the Chinese university has an online learning system, students did not use it because instructors let students use textbooks during study rather than uploading learning contents through the online learning system. Specifically, Hua also mentioned: The online learning system at my Chinese university is not very useful to study. For instance, the online library does not have enough resources and the interface and layout CP:69 421 design is not very good. That platform is different to Blackboard that is provided by the Australian university. The platform of the Chinese university seems an online forum, which has information that is irrelevant to study. I just use it to watch videos and do other types of online entertainment. Qian and Gang, from the design group, highlighted that they did not have any particular blended learning experiences in the Chinese university period of the whole transnational program. In particular, Gang stated: Compared to my Australian learning experience, I think I did not have impressive study activities that were taught through blended methods. In Australian university, instructors always use Blackboard and other online tools to assist in teaching. Some of them are good at using online tools to design specific interactive sections that encourage students to think of taught contents and question instructor and peers. However, in my previous Chinese classroom, it is difficult to have such kind of learning experiences. I just need to read textbooks and listen to the instructors. It seems to study in a high school rather than a university. The Chinese learning styles become obstacles to adapt to Australian university. For my transnational program, there are limited courses that aim to introduce Australian learning styles. Chinese university only arranges English course but this is not good for us to adapt to the real Australian classroom. According to students’ feedback, it is obvious that there are many differences of using online learning tools, recourses and study methods between Chinese and Australian universities, for example, instructors’ use of ICTs and the online learning systems provided by universities. These differences may become obstacles when students start to learning in an Australian university. Conclusion This study investigated the online learning experiences of six first-year Chinese transnational program students in an Australian university. The results revealed students’ experiences on the use of Internet- based learning strategies, including preferences for teaching and learning approaches, online study platforms and the issues of using online strategies. Depending on the differences of course contents, learning styles and educational environments, educators and students may consider how to use Internet-based tools in teaching and learning activities (Ellis et al., 2006). These findings resonate with previous research studies conducted by Bliuc et al. (2011) who concluded that the effective integration of face-to-face and online learning is an important aspect during teaching and learning. On the basis of these results, some results are similar to previous studies (Graham et al., 2003; Tang, 2013). For instance, students feel more flexible when studying in a online learning environment, which resonates with Graham et al. (2003) and Tang (2013), who identify that online learning can provide more flexible teaching and learning approaches in traditional study mode. Comparatively, this study also reveals potential research gaps. Due to the particular settings of transnational programs, students usually have learning experiences in both Chinese and Australian universities. When these students come to Australia, the blended learning environment provides a different study style to this particular group of learners. Therefore, understanding potential problems by both educators and students may be beneficial. For instance, the different ways of using blended learning in different majors of transnational programs. It may be beneficial for future studies to focus on exploring the blended learning experiences of Chinese transnational program students in Australian universities in depth. References Altbach, P. G., Reisberg, L., & Rumbley, L. E. (2009). Trends in global higher education: Tracking an academic revolution: Boston College Center for International Higher Education Chestnut Hill, MA. Bliuc, A.-M., Ellis, R. A., Goodyear, P., & Piggott, L. (2011). A blended learning approach to teaching foreign policy: Student experiences of learning through face-to-face and online discussion and their relationship to academic performance. Computers & Education, 56(3), 856-864. Bonk, C. J., & Graham, C. R. (2012). The handbook of blended learning: Global perspectives, local designs: John Wiley & Sons. CP:70 422 Chan, S. (1999). The Chinese learner–a question of style. Education+ Training, 41(6/7), 294-305. Chen, R. T. H., Bennett, S., & Maton, K. (2008). The adaptation of Chinese international students to online flexible learning: Two case studies. Distance Education, 29(3), 307-323. Ellis, R. A., Goodyear, P., Prosser, M., & O'Hara, A. (2006). How and what university students learn through online and face‐to‐face discussion: Conceptions, intentions and approaches. Journal of Computer Assisted Learning, 22(4), 244-256. Graham, C. R., Allen, S., & Ure, D. (2003). Blended learning environments: A review of the research literature. Unpublished manuscript, Provo, UT. Kim, K. J., Bonk, C. J., & Teng, Y. T. (2009). The present state and future trends of blended learning in workplace learning settings across five countries. Asia Pacific Education Review, 10(3), 299- 308. Rovai, A. P., & Jordan, H. (2004). Blended learning and sense of community: A comparative analysis with traditional and fully online graduate courses. The International Review of Research in Open and Distance Learning, 5(2). Tang, J. (2013). The research on blended learning of ESL based on Moodle platform. Studies in Literature and Language, 6(2), 30-34. Zhang, W., & Han, C. (2012). A case study of the application of a blended learning approach to web- based college English teaching platform in a medical university in eastern China. Theory and Practice in Language Studies, 2(9), 1961. Zhao, J. (2008). An examination of students’ perception of blended e-learning in Chinese higher education Technologies for E-Learning and Digital Entertainment (pp. 162-170): Springer. Dai, K. (2015). An investigation of blended learning experiences of first-year Chinese transnational program students at an Australian university. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:67-71). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:71 423 A comparison of undergraduate student experiences of assessed versus non-assessed participation in online asynchronous discussion groups: Lessons from a cross disciplinary study in health and sociology Tracy Douglas Carey Mather Sandra Murray School of Health Sciences School of Health Sciences School of Health Sciences University of Tasmania University of Tasmania University of Tasmania Louise Earwaker Allison James Jane Pittaway Library, Division of Students and Department of Maritime and Logistics School of Health Sciences Education University of Tasmania Management University of Tasmania Australian Maritime College Brady Robards Susan Salter School of Health Sciences School of Health Sciences University of Tasmania University of Tasmania This paper discusses a pilot study investigating perceptions from undergraduate students enrolled in units in which asynchronous online discussion boards were utilised formatively or linked to summative assessment. Of the influences that determine level of student engagement in online discussions, one key factor is whether discussions are assessed. Whilst assessing student discussions does motivate participation, this approach is not always valued by students as they are critical of the value of asynchronous discussion boards to their learning. The type of postings can be an influencing factor in student engagement, with effective facilitation, clear purpose and group participation perceived to be important. Students also viewed discussion boards as a platform in which peer engagement and information sharing occurred. Students who were enrolled in a unit in which discussion postings were assessed demonstrated emerging critical thinking skills. Students strongly indicated discussion boards must be fit-for-purpose and integrated into the curriculum regardless of whether they are assessed or not. Keywords: assessment, discussion boards, asynchronous, student, engagement, higher education Introduction Communication tools such as discussion boards form an integral part of online learning management systems and therefore are extensively used in higher education, particularly in an asynchronous context (Andreson 2009) as they provide a means for students to communicate and learn collaboratively. In many instances, these discussion boards are linked to assessment to facilitate engagement and promote development of critical thinking (Johnson and Johnson 1986). However, there is also a role for discussion boards as a formative learning tool. Student satisfaction about studying online has been well researched (Horzam, 2015; Lander 2014; Ladyshewsky 2013; Liaw 2008; Bouhnik and Marcus, 2006) with engagement often posed as difficult to achieve across the student cohort. As stated by Gregory (2015) discussion boards can be a collaborative learning tool, particularly for off campus students, and students usually participate when they are linked to assessment. Less evident in the literature is the student perspective of participation in asynchronous discussion groups that are non-assessed compared to perspectives on assessed discussion boards. According to Du et al (2008) active engagement with others promotes meaningful learning and in an online environment, the topic of discussion is important in determining the impact. Disengagement with asynchronous discussion boards may be related to facilitation (Northover 2002), with instructor facilitation preferred to student facilitation (Hew 2015). Students respond well to feedback in any learning and teaching paradigm and so instructor facilitation drives learning quality and student satisfaction in an online course (Ladyshewsky 2013). Disengagement may be related to the CP:72 424 ambiguous nature of discussion postings and the limited ability of students to construct knowledge through online discussion (Lander 2014) but once students are engaged they should be able to perceive the value of online discussion boards. The greater the level of student engagement, the higher the perceived value of asynchronous discussions (Northover 2005, Pena-Shaff et al 2005) Pena-Shaff et al (2005) reported student attitudes to online discussions ranged from enthusiastic to hostile and that some students perceived the asynchronous discussions as a chore lacking either substance or meaning. These authors also reported that some students rebelled against the assessment incentive, which they viewed as burdensome, with some students exhibiting resentment at forced participation. Clear purpose of a discussion board is essential for engagement (Gregory 2015) with identifiable student outcomes (Steen 2015). As a result, discussion boards are often linked to assessment. This paper reports on a pilot study investigating student perceptions of online discussion boards utilised as a key assessment item or formative learning tool. Methodology Undergraduate students studying in one of four units in sociology or health science were invited to participate by completing an anonymous online questionnaire. Two units utilised discussion boards as an assessment task in the unit, (10% of the overall assessment was determined by discussion board participation), with clear assessment criteria provided to the students. In the other two units, discussion boards were used as online communication tools for formative feedback purposes. Respondents were recruited by email with two reminders sent at two-week intervals. The survey questions were designed to elicit both quantitative and qualitative data. The first set of questions gathered information on the factors which motivated students to engage using online discussion boards and their overall experience as learners. The second group of questions were reflective and open-ended, designed to generate descriptive data on student experiences and asked about students’ proficiency and how they used discussion boards for learning. Research ethics approval was obtained for this study (H0013544). Results The students surveyed in this pilot study were enrolled in an undergraduate unit in health sciences or sociology in which discussion boards were utilised as either an assessment item (assessed) or a formative learning tool (non-assessed). A total of 78 students completed the questionnaire representing a small sample of the total cohort. Gender, level of education, and preferred language were similar for each group. The mean age of the assessed group was slightly older (aged over 25 years, 60%) than the non-assessed group (over 25 years, 41%). In addition, the non-assessed group were more likely to be studying part-time (81%) than the assessed group (36%) although a mixture of part-time and full-time status existed across the four units. Just over ninety percent (94%) of students in the assessed group were comfortable using the internet before starting their course, compared to 45% of the non-assessed students. However, differences identified between the nature of the two groups were not significant and were not related to discussion board access as 92% of the assessed group (and 72% of the non-assessed group) did not encounter any barriers to access. The assessed group were more comfortable in initiating (62%) and responding (66%) to discussion posts than the non-assessed group, in which only 37% were comfortable to respond to posts with 52% expressing some comfort in initiating posts in a discussion board. Seventy per cent of students who participated in assessed online discussion boards found the discussion valuable to their learning and 41% of these respondents stated that the online discussions did assist them with the completion of other assessment tasks in the unit. Respondents who were not assessed in their discussion postings did find the postings valuable (62%), however, not as valuable in relation to their assessment tasks (26%). In the assessed group, assessment was a motivating factor for participation according to 65%, while 33% of respondents in the non-assessed units indicated that linking assessment to discussion postings may motivate them to engage. Similarly, 56% of respondents in the assessed group indicated that discussion boards were useful to develop group engagement; however only 25% of the non-assessed group identified that this would be useful in their unit. Referencing of discussion posts was perceived similarly between the two groups with 43% in the assessed group finding referencing CP:73 425 of posts useful compared to 42% of respondents in the other group. Participants in the study provided answers to open-ended questions that explored the student perspective about: the purpose of discussion boards; most and least useful discussions; suggestions for improvement; and an opportunity to comment on any other aspect of the discussion boards. The non-assessed group were more homogeneous in their responses, stating that sharing information or interaction with other students was the purpose of discussion groups. One student stated: “To engage with the unit content as well as communicate with other members of the distance unit, while maintaining links with the unit coordinator” and “To share understandings and to discuss concepts being taught with peers”. Additionally, students in the assessed group also indicated the purpose was to gain marks and enable reflection by participating in online discussion. Some students in the assessed group were critical of the discussion tool, perceiving the purpose of the discussions for assessment as inconvenient, and therefore not directly related to their learning. Non-assessed group respondents indicated they preferred discussions that were compulsory (even if marks were not assigned) or where replies were posted. They liked the opportunity to gain or share information or be exposed to perspectives not already considered. For example, one student stated “… there were many different views and ideas presented that helped with a better understanding of things that may have been hidden/unknown”. Students in the assessed discussion groups commented they preferred the discussion posts that had meaning for them, including informal threads that developed from the assessable posts. One student stated: “Discussion kept me on track, so I found the discussions broad (and) to be a benefit. All discussions were engaging once you started”. However, over-sharing of personal information in discussion postings was not favoured by either group as this information was considered irrelevant and non-engaging. Lack of critical thinking or reasoned argument by other students was also frustrating according to respondents who were assessed on discussions. One student stated: (the least engaging were) “…the ones which only answered the question and did not have an opinion. What’s the point?” Referencing discussion posts was not always favoured among respondents in the assessed or non-assessed groups but non- referenced posts were also mentioned as being less engaging by students who were assessed. A number of respondents mentioned they preferred to post to a discussion board when they did not need to reference. Students in the non-assessed groups commented that they would like more engagement by others in the discussions. Comments included making the interaction compulsory or assessing the posts or participation. One student stated: “Assessing posts would encourage students to participate, then they would learn how valuable posting can be”. Conversely, some respondents in the assessed group sought to make the discussion groups non-assessable items. One student stated: “Do not make them assessments. It was a monumental fail… it was very difficult to participate and feel engaged in them, it became a hassle more than a learning tool” and “Don’t use them. Adult learners do not respond to them. I found the overall tone of the discussion to be fake/false designed to achieve a pass mark and nothing else”. Both groups indicated they would like the facilitator to guide and moderate the discussions more, and the assessed group students commented that they would like more engagement and feedback from facilitators. One student in the non-assessed group stated: “I think discussions could benefit from the lecturer’s contributions; to steer the topics and prevent the students from discussing too many personal issues”. One student stated: “I would have participated more if the lecturer was involved to keep the content of the discussions on the right track”. Respondents from the assessed group indicated would like the discussion groups to be comprised of a smaller number of students. The assessed group respondents also mentioned that technical difficulties and length of time involved to participate could be improved. One respondent stated: “It is very time consuming trying to prepare worthwhile discussion posts compared to the amount of marks they are worth”. The students in the assessed group focussed more on the inhibitors of discussions, citing disliking interaction, too much other work and too many discussions. A range of alternatives such as weekly quizzes, short answer questions or alternative assessment tasks such as an essay were suggested by respondents. One student stated: “would much rather just have online quizzes or assignments for learning, online discussions are a burden”. CP:74 426 Discussion Linking assessment to online discussions motivates student engagement particularly when discussion topics are facilitated to provide effective learning experiences. This is supported by previous studies in which students do value asynchronous discussions as an integral component of their online learning and assessment (Vonderwell et al. 2008). Participating in online discussions can enhance learning but the inclusion of referencing in posts may be detrimental to intended outcomes as indicated by respondents in this study. Lander (2014) found that referencing stunted conversation and halted learning and that some students were reluctant to state a value position that may diminish their perspective and have a negative impact on their knowledge construction. Referencing of posts was somewhat favoured among students in our groups and some students in the assessed group commented they did like referencing the information in their posts. However, some students did provide unfavourable comments with respect to the need to reference as they felt it impacted negatively on the quality of their post. Referencing of posts does add academic rigour to the postings and discussion threads and so should be encouraged for effective learning and teaching practice. Students in this pilot study, particularly in the assessed group, suggested improvements to online discussions could include facilitator guidance and feedback to students. This supports the literature in which effective facilitation has been shown to enhance the quality and satisfaction of the discussions for students (Ladyshewsky 2013). As indicated in previous studies the majority of respondents preferred a facilitator to direct the discussion, irrespective of whether the postings were assessed or not, (Hew 2015). Student perceptions indicated that discussion posts need to be engaging and fit for purpose, regardless of whether or not they are assessed. Effective facilitation, enables engagement by students.. Facilitator feedback can be scaffolded within assessed discussion boards to ensure that students are constructively building their online communication skills and knowledge effectively. This supports recent literature that indicates facilitators need to clearly indicate purpose for discussion boards and design tasks which provide constructive learning (Gregory 2015; Steen 2015). The nature of the group dynamics and motivation for participation in the online discussions will also influence student engagement (Robinson 2011). Assessment is regarded to be a key motivating factor in an online learning and teaching paradigm. The hostile responses from some students in the assessed group were similar to the findings of Penna-Shaff et al (2005) who reported that assessment hindered participation by some students who resented being forced to participate. These authors also found there was written apprehension anxiety, which was also a finding in both assessed and non-assessed groups in this study. Comfort levels of posting to discussions was more evident in the assessed group compared to the non- assessed group, which is most likely related to experience. In addition, students in the assessed group were more likely to voice their concern about the content of their posts than those in the non- assessed group. Du et al (2008) suggested that identifying patterns in which online discussions are conducted effectively could enable improvements in collaborative learning. The differences in the patterns of engagement, willingness to participate and behaviour between assessed and non- assessed students in discussion groups, provides opportunities for re-orientating online discussions to better suit the learning needs of students. Moreover, curriculum re-design could improve student perceptions and understanding of the value of this educational tool. The findings of this study suggest there were contributing factors that altered the student experience depending on whether discussion boards were assessed or not. Limitations of this study include different study status and ages of students, as well as the small sample size. Research into influence of discussion boards on student learning, in assessed and non-assessed groups across a range of disciplines and different undergraduate years, with a larger sample size, warrants further investigation. Conclusion This study found students focus on different aspects of asynchronous discussion groups depending on whether they are assessed or non-assessed. Students using online discussions that were assessed were more critical of the process, facilitator feedback and whether online discussions are a useful learning tool or a burden. The non-assessed group of students indicated the purpose of asynchronous discussions as a means of sharing information or engaging with their peers, with critical thinking being of less importance to this cohort. Online discussions, whether assessed or not, need to CP:75 427 have clear purpose, be authentic for engagement and enable meaningful learning. Assessing discussion postings does value add to their purpose pedagogically however effective facilitation also needs to be implemented to authenticate learning. Future studies investigating student perceptions of assessed and non-assessed asynchronous discussion boards across a wider range of disciplines and contexts are required to validate and extend the findings of this study. References Andresen, M.A. (2009). Asynchronous discussion forums: success factors, outcomes, assessments and limitations. Educational Technology & Society 12(1), 249-257. Bouhnik, D. & Marcus, T. (2006). Interaction in distance-learning courses. Journal of the American Society for Information Science and Technology, 57(3), 299–305. Du, J., Zhang, K., Olinzock, A., & Adams, J. (2008). Graduate students' perspectives on the meaningful nature of online discussions. Journal of Interactive Learning Research, 19(1), 21-36. Gregory, S (2015). Discussion boards as collaborative learning tools. Int.J.Cont.Engineering Education and Lifelong Learning, 25(1), 63-76. Hew, K.F. (2015). Student perceptions of peer versus instructor facilitation of asynchronous online discussions: further findings from three cases. Instr Sci, 43, 19-38. Horzum, M.B. (2015). Interaction, structure, social presence, and satisfaction in online learning. Eurasia Journal of Mathematics, Sciences and Technology Education, 11(3), 505-512. Johnson, D.W., Johnson, R.T (1986). Cooperative learning in the science classroom. Science and Children, 24, 31-32. Ladyshewsky, R.K. (2013). Instructor presence in online courses and student satisfaction. International Journal for the Scholarship of Teaching and Learning. 7(1), 13. Lander, J. (2014). The role of the interpersonal in online knowledge construction: unrealised or unrealisable potential. Proceedings ASCILITE Dunedin 2014, pp. 222-232. Liaw, S. (2008). Investigating students’ perceived satisfaction, behavioural intention, and effectiveness of e-learning: A case study of the Blackboard system. Computers and Education 51, 864-873. Northover, M. (2002). Online Discussion Boards – Friend or Foe? Proceedings ASCILITE Auckland 2002, pp. 477–484. Pena-Shaff, J., Altman, W. & Stephenson, H. (2005). Asynchronous online discussion as a tool for learning: Students’ attitudes, expectations and perceptions. Journal of International Learning Research 16(4), 409-430. Robinson (2011). Assessing the value of using an online discussion board for engaging students. Journal of Hospitality, Leisure, Sport and Tourism Education, 10(1), 13–22. Steen, T.M. (2015). Facilitating online learning activities through the discussion board: a first year university students’ perspective. Int.J.Cont.Engineering Education and Lifelong Learning, 25(1), 77-102. Vonderwell, S., Liang, X. & Alderman, K. (2008). Asynchronous Discussions and Assessment in Online Learning. Journal of Research on Technology in Education, 39(3), 309-328. Douglas, T., Mather, C., Murray, S., Earwaker, L., James, A., Pittaway, J., Robards, B., & Salter, S. (2015). A comparison of undergraduate student experiences of assessed versus non-assessed participation in online asynchronous discussion groups: Lessons from a cross disciplinary study in health and sociology. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:72-CP:76). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:76 428 Digital Futures research and society: action, awareness and accountability Joanne Doyle Lisa McDonald Michael Cuthill Australian Digital Futures Institute Australian Centre for Sustainable Institute for Resilient Regions University of Southern Queensland Business University of Southern Queensland and Development University of Southern Queensland Mike Keppell Learning Transformations Swinburne University of Technology The contemporary Higher Education research environment demands ‘real-world’ impact as a key means of accounting for public sector funding. As such, there is increased pressure on researchers and research institutions to ensure research delivers outcomes for public good. This paper reports on research focused on a Digital Futures collaborative research program. The aim of the research was to explore how researchers and research stakeholders understand research impact. Impact was articulated as ‘making a difference’ however that ‘difference’ was translated by research participants as meaning the tangible impacts relating to quantitative components of research activities. The subtler influences of research impact on society were less well articulated. Results from this research suggest that in the complex world of impact, action, awareness and accountability, as elements of research practice, are key to creating maximum value from knowledge creation initiatives. Keywords: research impact; technology; learning; evaluation; Higher Education; collaboration Introduction Higher Education research is seen to deliver two types of impact – academic impact and non- academic impact. Academic (scholarly) impact is ‘the intellectual contribution to one’s field of study within academia’ (Penfield, Baker, Scoble, & Wykes, 2013, p. 21) and is primarily evidenced through academic publications. Non-academic impact (referred to forthwith as ‘real-world’ impact) is the influence of research on ‘policy, managerial and professional practices, social behaviour or public discourse’ (Sumner, Crichton, Theobald, Zulu, & Parkhurst, 2011, p. 3). A range of interrelated terms have been used to denote this real-world impact, including non-scholarly impact, societal impact, non- traditional impact, external impact and secondary impact. However, real-world impact (and all these related terms) share in common a philosophical commitment to the public good. In contrast, while not necessarily ignoring the public good, it has been suggested that the publishing priority of scholarly impact preferences private reward to the researcher ahead of public benefit (Campbell, 2012). Assessment, and therefore understanding, of real-world impact has been seen to be problematic due to challenges of quantifying and recording the intangible influences of research. Recent developments in impact The role of research as being of benefit to society is well-recognised. Over recent years, the ways in which the use of research is conceptualised has changed. A preliminary database search reveals that, from the 1950s onwards, and with a noticeable increase from 1993 to 2013, many articles focused on ‘using research.’ In the 1990s, there was a general understanding that research findings would be ‘handed over’ to others, and implemented by those in the ‘real world’ (Jackson, 2014, p. 127). From 2003 to 2014, the term ‘translating research’ became popular in recognition of the fact that research often requires interpretation prior to implementation. A range of other terms including ‘knowledge production’ and ‘evidence utilisation’ were used commonly from 2000 onwards, with use of the term ‘knowledge mobilisation’ peaking in 2013. In the period between 2010 and 2014, ‘engaged scholarship’ was popularised. Engaged scholarship recognizes the need for multiple knowledge systems to be directly included in the knowledge creation process (Van de Ven, 2007). CP:77 429 The dynamic nature of the science-society relationship in discovering and integrating scientific knowledge has been explored by scholars. For example, Boyer (1990, p. 16) proposed a re-definition of scholarship to include ‘building bridges between theory and practice’, and Gibbons et al. (1994) identified contemporary knowledge production as a socially accountable process. Yet making use of research remains a haphazard process with many scholars lamenting the slow or nil application of research for the benefit of society (Shokar, 2014; Steffens, Weeks, Davidsson, & Isaak, 2014). Achieving real-world impact from research relies upon collaboration and knowledge-sharing between researchers and research users (Armstrong & Kendall, 2010; Hemsley-Brown & Sharp, 2003). There is extensive discussion in the literature about the impact of research on policy and practice. Yet the process of translating research is complex (Bastow, Dunleavy, & Tinkler, 2014). There is a well- recognised academic-practitioner gap in many fields (Steffens et al., 2014) and a ‘valley of death’ between discovery and application (Butler, 2008). While the health sector actively encourages evidence-based practice ( (Balakas & Sparks, 2010), many other disciplines, including education and business, struggle with bridging the gap between research and practice (Hemsley-Brown & Sharp, 2003; Skapinker, 2008). Research institutions across the globe have been increasingly required to demonstrate the relevance of research endeavours in terms of real-world impact. The focus on demonstrable impact is being driven by an international impact agenda that calls for greater accountability of public sector expenditure. The issue of accountability has been a key policy focus of governments since the early 1980s (Slavin, 2002). Identifying and articulating broad concept of research impact has become an important skill for researchers seeking engagement and investment (Chubb, 2014) Assessing Digital Futures program impact An intensive case study was undertaken at a regional university in Australia with the aim of enhancing understanding about the broad impacts of Higher Education research. The research sought to identify how Higher Education research delivers real-world benefit beyond contributions to academia. A phenomenological approach explored the lived experience of research impact from the perspective of researchers and research stakeholders in a collaborative multidisciplinary research program. A phenomenological method is concerned with illuminating certain aspects of lived experience that can be tied to experiences of the past, even though this can be ‘subject to reversals, surprises and readjustments’ (Diprose & Reynolds, 2014, p. 33). Meaning making through experiences can, therefore, involve the emergence of common themes where similar contexts of understanding are influential. Qualitative data was captured through a two-stage data collection process. In Stage 1, 27 semi-structured interviews were conducted. In Stage 2, ten researchers participated in two focus groups to discuss the main themes emerging from Stage 1 data. Data was analysed through thematic interpretation supported by NVivo data analysis software. A common theme that emerged during the data analysis process was that impactful research should ‘make a difference’. Researchers used a range of terms to denote the way research impact is conceptualised: change, variation, development, movement or difference. There was little reference to impact using terms such as worth or value, which have financial connotations, and no reference to impact having positive or negative dimensions, reinforcing the theme of ‘making a difference’ regardless of what the difference may be. Researchers demonstrated a good understanding of the distinction between research quality and research impact, as it is defined above, however there was little evidence of a direct or forced correlation between research quality and research impact. This supports literature which acknowledges that there is the potential for poor quality research to achieve high impact (Mendel, 2014). This can happen when highly-relevant but less-rigorous research is extensively promoted and highly cited as a result. The majority of researchers also demonstrated a good understanding of research impact in terms of the defined concept of ‘academic impact,’ and most researchers were confident that research impact would be achieved as a result of disseminating research findings. Academic outputs of publications, conference papers, presentations and citations were articulated as being the most tangible evidence of research impact. Additionally, researchers were comfortable sharing evidence of scholarly impact, yet found it difficult to articulate the real-world impact of projects that they were undertaking in the program. The CP:78 430 challenges of specifying real-world impact included time for impact to be realized and reliance upon other stakeholders to implement the findings of research. Researchers cited access to resources, time pressures and a lack of accountability as barriers to assessing the impact of research. A common theme emerging from the interviews was that real-world impact isn’t dependent on the production of research outputs. Impact frequently occurs during the research process through the interaction with research participants. Researchers perceived that impact had been achieved through altered opinions, improved understanding and broadened perspectives, reinforcing the concept that research impact is as much about creating awareness of research knowledge as it is about applying research knowledge. The manifestation of impact Assessing impact is not the same as evaluating impact. Impact evaluation suggests a quality judgement (Baehr, 2005) with a focus on financial and social measures. Guidelines for evaluating impact recommend activities such as baseline assessment in addition to understanding stakeholder needs and encouraging more than one source of evidence (Bromley & De Campos, 2014). In the case of impact assessment, the process is less clear. Assessing impact is more than identifying ‘additionality’ which is a quantitative measure that fails to recognize changes that are difficult to measure (Bromley & De Campos, 2014, p. 89). The results from the Digital Futures case study reinforced the two dimensions of research impact – academic impact and non-academic real-world impact. Additionally, the data analysis revealed an interdependency across the two dimensions that varies according to researcher accountability for mobilising research findings in terms of awareness and action. Awareness of research primarily occurs through dissemination of research findings and generates academic or scholarly impact. Activities that generate academic impact can be measured by counting publications, citations, tweets, blogs, etc, however this is a volume-oriented metrics-approach that is insufficient to fully capture the less tangible impacts of research activities. It is not possible to count or weigh every instance of impact and it is not phenomenologically possible to articulate the entirety of a particular researcher’s lived experience (Diprose & Reynolds, 2014), in this case, of research production. Research awareness therefore brings many benefits to society that are less obvious. _ENREF_23 Stanwick and Hargreaves (2012)_ENREF_25 identify four impact domains: producing knowledge, building capacity, informing policy and informing practice. The ‘producing knowledge’ dimension of impact was evident throughout the interviews with researchers conveying that participants in research activities benefitted as a result of participating in data collection activities and through community/industry/university collaborations. Participants gained an awareness of particular research objectives, methodologies, hypotheses and previous research findings, and researchers felt that they had made a real-world difference by imparting knowledge without the production of more tangible outputs. Action, as it relates to research impact, was reported by researchers as being largely outside the domain of their accountability. The implementation of research findings is dependent upon action by government, community organisations, industry associations, businesses and other research stakeholders. Specific examples of action arising from research was more challenging to articulate with researchers resorting to examples of academic impact as demonstrable evidence of their research impact. Researchers were familiar with the intent of research impact as ‘the demonstrable contribution that research makes to the economy, society, culture, national security, public policy or services, health, the environment, or quality of life, beyond contributions to academia’ (Australian Research Council, 2014) yet were less comfortable providing specific instances of having influenced these dimensions of real-world impact. Considerations for the future Research conducted by Higher Education institutions remains an essential activity for the health and wealth of a nation (Bauerlein, Gad-el-Hak, Grody, McKelvey, & Trimble, 2010). Whilst the contribution of research is well-understood in terms of scholarship of discovery (Boyer, 1990), assessing the influence of knowledge on society is a complex and challenging activity. Assessment frameworks that use a logic model approach to understanding impact are useful for assessing the academic and non- academic impacts of research where impact is directly related to research outputs. However, these CP:79 431 frameworks do not capture the more subtle and less tangible real-world impact that arises during the research activity, and which has perceptive, and therefore, interpretive experiential influences. In this sense, research impact is also created during collaboration and communication processes external to the research environment, and similarly brings interpretive qualities to such processes. Impact can also give rise to the intention of ‘making a difference’ (Chandler, 2014, p. 2). The researchers in this study were committed to making a difference, yet grappled with articulating the real-world impact of research beyond scholarly influences. Assessing impact extends beyond measuring ‘what can be measured’ to measuring ‘what should be measured’ (Wells & Whitworth, 2007, p. 1) and is an inherent complexity of impact assessment activities. Conclusion The Higher Education sector is under increasing pressure to demonstrate value to society (de Jong, Barker, Cox, Sveinsdottir, & Van den Besselaar, 2014) yet it is difficult to demonstrate value when there is no accepted framework for measuring real-world impact (Bornmann, 2012). More research is needed to understand research impact and clarify impact terminology that is inconsistent and confusing (Penfield et al., 2013). Assessing impact is feasible but current methods need to be improved (Ovseiko, Oancea, & Buchan, 2012). Understanding how impact manifests through action, awareness and accountability is an essential first step in re-conceptualising research impact. References Armstrong, K., & Kendall, E. (2010). Translating knowledge into practice and policy: the role of knowledge networks in primary health care. Health Information Management Journal, 39(2), 9. Australian Research Council. (2014). Research impact principles and framework. Retrieved 27 June 2014, 2014, from http://www.arc.gov.au/general/impact.htm Baehr, M. (2005). Distinctions between assessment and evaluation. Program Assessment Handbook, 7. Balakas, K., & Sparks, L. (2010). Teaching research and evidence-based practice using a service- learning approach. The Journal of nursing education, 49(12), 691-695. Bastow, S., Dunleavy, P., & Tinkler, J. (2014). The impact of the social sciences: how academics and their research make a difference: SAGE. Bauerlein, M., Gad-el-Hak, M., Grody, W., McKelvey, B., & Trimble, S. W. (2010, 13 June). We must stop the avalanche of low-quality research, The Chronicle of Higher Education. Retrieved from http://chronicle.com/article/We-Must-Stop-the-Avalanche-of/65890/ Bornmann, L. (2012). Measuring the societal impact of research: research is less and less assessed on scientific impact alone - we should aim to quantify the increasingly important contributions of science to society. EMBO Rep, 13(8), 673-676. doi: 10.1038/embor.2012.99 Boyer, E. L. (1990). Scholarship reconsidered: Priorities of the professoriate. New Jersey: The Carnegie Foundation for the Advancement of Teaching. Bromley, T., & De Campos, A. (2014). How can impact evaluation be planned? In P. Denicolo (Ed.), Achieving impact in research. London: SAGE Publications Ltd. Butler, D. (2008, 11 June 2008). Translational research: crossing the valley of death. Retrieved 30 September 2014, from http://www.nature.com/news/2008/080611/full/453840a.html Campbell, H. (2012). Lots of words… but do any of them matter? The challenge of engaged scholarship. Planning Theory & Practice, 13(3), 349-353. Chandler, C. (2014). What is the meaning of impact in relation to research and why does it matter? A view from inside academia. In P. Denicolo (Ed.), Achieving impact in research. London, UK: Sage Publications Ltd. Chubb, J. (2014). How does the impact agenda fit with attitudes and ethics that motivate research? In P. Denicolo (Ed.), Achieving impact in research. London, UK: Sage Publications Ltd. de Jong, S., Barker, K., Cox, D., Sveinsdottir, T., & Van den Besselaar, P. (2014). Understanding societal impact through productive interactions: ICT research as a case. Research Evaluation, 23(2), 89-102. doi: 10.1093/reseval/rvu001 Diprose, R., & Reynolds, J. (2014). Merleau-Ponty: key concepts: Routledge. Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: The dynamics of science and research in contemporary societies: Sage. CP:80 432 Hemsley-Brown, J., & Sharp, C. (2003). The use of research to improve professional practice: A systematic review of the literature. Oxford Review of Education, 29(4), 449-471. Jackson, A. (2014). How can knowledge exchange support the development of impact through partnerships and university infrastructures? In P. Denicolo (Ed.), Achieving impact in research (pp. 127-142). London: SAGE Publications Ltd. Mendel, J. (2014). Bad Research and High Impact: The Science: So What Campaign and Social Media Criticism. ACME: An International E-Journal for Critical Geographies, 13(1), 56-61. Ovseiko, P., Oancea, A., & Buchan, A. (2012). Assessing research impact in clinical medicine: a study using Research Excellence Framework pilot impact indicators. BMC Health Services Research, 12(478), 1-23. Penfield, T., Baker, M., Scoble, R., & Wykes, M. (2013). Assessment, evaluations, and definitions of research impact: A review. Research Evaluation, 23(1), 21-32. doi: 10.1093/reseval/rvt021 Shokar, N. K. (2014). Translating Research into Practice: The Colorectal Cancer Screening Experience. Journal of General Internal Medicine, 29(1), 10-12. Skapinker, M. (2008). Why business ignores business schools. Financial Times, 8, 11. Slavin, R. E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher, 31(7), 15-21. Stanwick, J., & Hargreaves, J. (2012). Measuring and maximising research impact in social science research settings: good practice guide. Adelaide: National Centre for Vocational Education Research. Steffens, P. R., Weeks, C. S., Davidsson, P., & Isaak, L. (2014). Shouting from the ivory tower: a marketing approach to improve communication of academic research to entrepreneurs. Entrepreneurship Theory and Practice, 38(2), 399-426. Sumner, A., Crichton, J., Theobald, S., Zulu, E., & Parkhurst, J. (2011). What shapes research impact on policy? Understanding research uptake in sexual and reproductive health policy processes in resource poor contexts. Health Research Policy and Systems, 9(Suppl 1), S3. Van de Ven, A. H. (2007). Engaged scholarship: a guide for organizational and social research: a guide for organizational and social research: Oxford University Press. Wells, R., & Whitworth, J. A. (2007). Assessing outcomes of health and medical research: do we measure what counts or count what we can measure? Australia and New Zealand Health Policy, 4(1), 14. Doyle, J., McDonald, L., Cuthill, M., & Keppell, M. (2015). Digital Futures research and society: action, awareness and accountability. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:77- CP:81). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:81 433 Making the Connection: Allowing access to digital higher education in a correctional environment Helen Farley Sharron Dove Stephen Seymour Australian Digital Futures Institute Australian Digital Futures Institute Australian Digital Futures Institute University of Southern Queensland University of Southern Queensland University of Southern Queensland John Macdonald Catherine Abraham Chris Lee Australian Digital Futures Institute Australian Digital Futures Institute Australian Digital Futures Institute University of Southern Queensland University of Southern Queensland University of Southern Queensland Susan Hopkins Jacinta Cox Louise Patching Open Access College Australian Digital Futures Institute Australian Digital Futures Institute University of Southern Queensland University of Southern Queensland University of Southern Queensland In most Australian correctional jurisdictions, prisoners are not allowed access to the internet precluding them from participating in higher education online. This paper reports on an Australian government-funded project, Making the Connection, which is taking digital technologies, that don’t require internet access, into correctional centres to enable prisoners to enroll in a suite of pre-tertiary and undergraduate programs. A version of the University of Southern Queensland’s learning management system has been installed onto the education server of participating correctional centres. The second stage of the project will see notebook computers preloaded with course materials, allocated to participating prisoners. At the time of writing, the project has been deployed at eight correctional centres in Queensland and Western Australia, with negotiations underway for further rollout to Victoria, New South Wales and South Australia. It is expected that the technologies and processes developed for this project will enable the delivery of higher education to other cohorts without access to reliable internet access. Keywords: correctional education; digital inclusion; digital divide; higher education; digital equity Introduction Prisoners in most Australian jurisdictions are not permitted to access online learning technologies due to procedural restrictions prohibiting prisoner access to the internet. Formal education and training delivery to prisoners is currently provided in non-digital forms, usually in the form of blocks of printed text. Although this method enables access to course materials, it does not develop digital literacies in incarcerated students, and these skills are becoming essential in order to pursue formal learning outside of correctional centres. Currently, there are few programs offered to incarcerated students that adequately prepare them for entry into higher education and even fewer that provide incarcerated students with the opportunity to use modern ICTs. Distance education has traditionally been viewed as means by which prisoners could access education in correctional centres, delivering education and resources to students who are unable to undertake traditional face-to-face education (Salane 2008). Formal education and training delivery to prisoners in Australia is currently provided in non-digital forms using large volumes of printed copies of the course materials and learning support resources, sometimes supplemented by CDs for use on in-cell laptops or in computer labs (Dorman and Bull 2003). This is costly for universities to assemble, print and post, is in no way interactive, and cannot incorporate all of the learning support resources of the course. Incarcerated students often have very little or no contact with each other and are not able to leverage the social learning supports that are available to students engaged in online courses. This undermines the social constructive pedagogy favoured in many post-secondary programs and poorly prepares students for a world in which employers expect their employees to be familiar with social networking and other web 2.0 resources (Erisman and Contardo 2005). Furthermore, the traditional forms of delivery to incarcerated students do not enable incarcerated students to develop the crucial graduate attributes including digital literacies, collaborative teamwork and critical thinking skills required to successfully complete studies in higher education and also to obtain meaningful employment after release from custody. This paper reports on a project, Making the Connection, that CP:82 434 is taking digital technologies into correctional centres, aiming to help students access higher education and obtain the digital literacies they need for work or study. The project: Making the Connection In the latter half of 2013, a team of researchers at the University of Southern Queensland (USQ) were awarded $4.39 million over three years by the Australian Government under the Higher Education Participation and Partnerships Program for a project titled Making the Connection: Improving Access to Higher Education for Low Socio-Economic Status Students with ICT Limitations. Beginning in early 2014, the project built on three previous projects led by USQ which trialed various digital technologies for learning in correctional centres. Most notable of these was the Office for Learning and Teaching- funded project, From Access to Success, which developed a version of USQ’s learning management system (LMS), a version of Moodle called USQ StudyDesk, which was installed onto the correctional centre education lab server. This server had no capacity to access the internet and was physically isolated from the main administrative correctional centre network. This new version of the LMS was called the USQ Offline StudyDesk and was installed by education officers from self-loading DVDs produced at USQ. The USQ Offline StudyDesk allowed incarcerated students to access course materials including interactive multimedia and assessments via computers in the education lab, without needing to access the internet. The From Access to Success project ran at two correctional centres in Queensland using two courses from the Tertiary Preparation Program, an enabling program run by USQ’s Open Access College. Students successfully completing this program are granted automatic entry into specific USQ undergraduate programs. Making the Connection is building on From Access to Success by continuing to develop the USQ Offline StudyDesk so that is robust, repeatable and reliable. One of the findings from the earlier project was that incarcerated students had only a few hours a week to access the correctional centre computer labs. This was because of the competition from other courses and programs, including vocational programs, for the space, and because students were typically employed in jobs in ‘industries’ within the correctional centre, restricting the time available to study. To help overcome these difficulties in access, the Making the Connection project will be providing notebook computers to participating students so that they can take them back to their cells and continue working in their personal time. Developing appropriate technologies is only a part of the challenge of providing higher education to incarcerated students. Appropriate courses and programs had to be adapted for use on the technologies and for use without access to the internet. Taking into account the levels of previous academic achievement in the correctional centres and jurisdictional sensitivities around students accruing HECS debt, there is a focus on the courses of the Tertiary Preparation Program and the Indigenous Higher Education Pathways Program, both Commonwealth-funded enabling programs. These programs are supplemented by three diploma programs: the Diploma of Arts (Social Sciences), Diploma of Science (Environment and Sustainability) and Diploma of Business Administration. A major part of the project is focused around engagement and outreach. The project has employed and Engagement Leader and also an Aboriginal and Torres Strait Islander Community Engagement Coordinator. The latter is in recognition of the overrepresentation of Aboriginal and Torres Strait Islander people in the correctional context. Making up some 2 per cent of the general population, they make up a staggering 28 per cent of the prisoner population nationally (ABS 2015). USQ Offline Study Desk At the beginning of the project, a detailed options analysis was undertaken to ensure that the USQ Offline StudyDesk installed onto a correctional centre education lab server was still the preferred technological approach. Various alternative options were examined including ‘Moodle-on-a-stick’ and secure cloud solutions. A team comprised of USQ ICT Services and Making the Connection personnel determined that the preferred solution remained installing the USQ Offline StudyDesk on a separate server linked to the education lab network via network switch. In the online environment, the USQ StudyDesk works with a Learning Objects Repository (LOR) which holds course content. Course content is vetted for copyright status and tagged with metadata to make it searchable. When a student accesses a resource via the StudyDesk, he or she is actually CP:83 435 accessing that resource through the Learning Objects Repository. This is obviously not feasible for those students using the USQ Offline StudyDesk. To address this issue, a bespoke piece of software, called a ‘compiler’, automatically harvests objects housed in the LOR and packages the resources with the course for export to the correctional centre. Another piece of software called a ‘checker’, goes through each course to ensure that files within each course are functional and that links to the internet have been removed. At the moment, the transfer of courses between USQ and the correctional centres occurs via DVD. In the near future, education officers will be able to download courses through a kiosk, hosted at USQ and accessed via the administrative network (which is internet-enabled). The version of the USQ Offline StudyDesk is approximately one version behind the main production version to allow for any glitches to be ironed out. The USQ Offline StudyDesk is currently installed in eight correctional centres in Queensland (7) and Western Australia (1). Notebook computers Because incarcerated students have limited access to the computer labs, it was decided that it would be desirable for students to have a personal device that they could take back to their cells. As with the modified LMS, these devices are not permitted to access the internet. Focus groups with incarcerated students participating in eBook reader trials in a previous project were critical of the small screen size and onscreen keyboard used in these devices. Taking this feedback onboard, the project team conducted a detailed options analysis of some 32 tablet computers, laptops and notebooks. It was decided that a Windows notebook would be most suitable as it had an almost full-size keyboard, adequate processing power and screen real estate was not compromised by an onscreen keyboard. In addition, students would be able to use Microsoft Office or OpenOffice to complete assessments. The project team are trialling the USQ Offline StudyDesk on the devices but are also considering using a HTML presentation layer to display course materials. These options will be trialled during the next phase of the project and hope to have the notebooks deployed into correctional centres before the end of 2015. Courses and programs The deployment of these technologies into correctional centres is just one part of the Making the Connection project. A suite of USQ courses and programs are being adapted for use without the need for internet access, to be used with the USQ Offline StudyDesk and the personal devices. A number of factors were taken into consideration when choosing the programs for modification. These included: 1. Average sentence length: Some 90 per cent of prisoners are sentenced for one year or less. 2. Previous academic achievements and experiences of the students: Most incarcerated students are from low socio-economic status backgrounds and have low levels of academic achievement. 3. Cultural background of the students: Aboriginal and Torres Strait Islander people are overrepresented in the criminal justice system. Low levels of education remain a key part of the ongoing cycle that leads to this over-representation. 4. Previous enrolment patterns for incarcerated students: USQ has been providing education to incarcerated students for around 25 years and has records of what programs incarcerated students have typically enrolled in. 5. Vocational outcomes: The project team consulted with careers advisors at the university about what careers ex-offenders could reasonably expect employment in and what programs would prepare them for these careers. 6. Practicality: The project team talked to course examiners (course coordinators), Heads of School, and Executive Deans about which courses could reasonably be adapted for delivery in the correctional environment. Courses with significant practical components or residential components were considered to be unsuitable. In addition, the jurisdictional owners expressed concern about the potential for incarcerated students to acquire a significant HECS debt. They worked closely with the project team to ensure that HECS debts would be kept to a minimum and would provide the best outcomes for students. CP:84 436 The Making the Connection project team selected the following courses to be used with the USQ Offline StudyDesk and personal devices. 1. Tertiary Preparation Program: Six courses from the Tertiary Preparation Program were selected for modification. These included general English and study skills courses, math courses and a humanities course. Successful completion of the Tertiary Preparation Program allows students automatic entry into selected USQ programs. This program is Commonwealth-funded enabling program and does not attract HECS fees. 2. Indigenous Higher Education Pathways Program: Six courses will be adapted from this program as part of the Making the Connection project. It is expected that this program will prove popular given the overrepresentation of Aboriginal and Torres Strait Islander prisoners and that Indigenous students are half as likely to have completed year 12 as non-Indigenous students. Again, this is a Commonwealth-funded enabling program for which students will not incur a HECS debt (Salane 2008). 3. Diploma of Arts (Social Sciences): Eight courses will be modified with an emphasis on community welfare and development. 4. Diploma of Science: This program will emphasize sustainability and the environment. Eight courses from this program will be modified. 5. Diploma of Business Administration: Historical data shows that most incarcerated students have enrolled in business programs. Again, eight courses from this program will be modified. Diploma programs were selected in acknowledgement of the typically short sentence length of most prisoners. Also, it was decided that it would be more beneficial to offer a selection of courses across a range of disciplines, rather than concentrate course modification efforts around one discipline as with a degree program. Course modification happens over an eight-week period called a ‘sprint’. A ‘sprint team’ comprised of learning designers, elearning designers, elearning technical support officers, copyright compliance officers, graphic designers, multimedia designers and other elearning professionals as needed, work with course examiners to create a plan for the modification. Online offerings of courses are moved to a specially designed Offline Course Development Area A number of activities are involved in course modification. 1. Courses are scoped so that the sprint team and the course examiners gain an idea as to how much work will be involved in modification. 2. Course materials are checked for copyright compliance. 3. Course materials are moved into the Learning Objects Repository. 4. Links to the internet are removed and alternative resources sourced if necessary. 5. The look and feel of the course is enhanced to ensure easy navigation. 6. Alternative assessments are designed if necessary. 7. Additional self-marking quizzes are incorporated to provide immediate feedback on knowledge recall. 8. A welcome video is created to help the incarcerated student feel connected to the course examiner. 9. The course is checked before being readied for deployment to the correctional centres. Typically, courses are modified in batches of six and are adapted in the semester before the next offer of that course. Course examiners’ time is bought out by the project. Funds are typically used for teaching or marking buyout. The course redesign process has been so successful that some course examiners are transferring resources developed as part of the project to the online offerings of their courses. Results The Making the Connection and the projects that preceded it provides a real and significant improvement to the traditional learning experience for incarcerated students by enabling students located in prisons across Australia to experience learning that is customised and personalised. So far, these projects have improved the learning experience across 239 enrolments in the in scope USQ courses (refer Figure 1). CP:85 437 Some testimonies from incarcerated students demonstrate the impact of the projects: It’s interesting that they treat education different to the core programs, when in fact it is the best form of rehabilitation. You are not going to change your person from some silly little 6 month course … educate a person and give them the skills they need to have a legitimate, successful employment status. If you don’t give them the tools they need, they are going to go nowhere. They definitely should be pushing the education flagship much, much further. I found it as an opportunity to redeem myself with my education. I really enjoy learning again. I was involved with drugs for a while but now my mind is clear I really enjoy learning again. Having my kids see me and see me move on to a career - so my kids can see I am going to turn my life around. Hopefully, I can turn things around because I don’t want them thinking it’s fine to come to jail because it’s not. I have been institutionalised my whole life. And I have another life sentence yet to do. I’m starting to think that I can help younger kids to not do the same mistakes that I did. Do courses, and get out and stay out. That’s my main motivation, is helping the younger generation and the youth in detention. I never had a computer while I was young but I learned to type while I was doing this course at the start of the semester. The future The Making the Connection team will be rolling the technologies and programs to additional correctional centres in Queensland and Western Australia before the end of the year. There is also strong interest from corrective services departments in New South Wales and Victoria. Jurisdictional owners have also expressed an interest in the availability of even shorter courses and programs to be offered to prisoners with very short sentences. To this end, the team are working with the Open University in the UK to make a selection of their Open Learn courses available in the offline environment. Perhaps the most exciting possibilities lie in making these technologies available for all those students without reliable internet access throughout Australia and the world. For example, broadband internet penetration is restricted in most countries within Southeast Asia due to the poor infrastructure. This is mostly attributable to a lack of private investment coupled with the severely limited capacity of the people to pay for services (Jeroschewski et al., 2013). The technologies and programs developed as part of the Making the Connection project have the potential to make higher education accessible to those otherwise unable to travel to a large city to study face-to-face, allowing people to remain in their communities and support the economic and social development of their regions. References Australian_Bureau_of_Statistics. 2015. Prisoners in Australia, 2014. Canberra, Australia: Australian Bureau of Statistics. Dorman, M. & Bull, D. (2003). Aligning educational needs with institutional priorities: Facilitating offender reintegration into contemporary society. IFECSA Conference. Retrieved from www.acea.org.au/.../2003%20papers/Paper%20Dorman_Bull.pdf Erisman, W. & Contardo, J. 2005. Learning to Reduce Recidivism: A 50-state analysis of postsecondary correctional education policy. The Institute for Higher Education Policy. CP:86 438 Jeroschewski, A., Levisse, A., Salazar, L., Tesoriero, R. and Ying, S. (2013). Connecting Southeast Asia through Broadband. In Enhancing ASEAN's connectivity,B. Das,ed. Singapore: Institute of Southeast Asia Studies: 72-90. Salane, F. (2008). Distance education in prisons: an educational right or a privilege? The case of "student inmates". Distances et savoirs, pp. 1-17. Retrieved from http://www.distanceandaccesstoeducation.org/contents/DS2008-Salane-English.pdf Wong, A. (2008), Build Communities, Not Prisons: The effects of the over-representation of Indigenous people in the criminal justice system. Hobart, University of Tasmania. Farley, H., Dove, S., Seymour, S., Macdonald, J., Abraham, C., Lee, C., Hopkins, S., Cox, J., & Patching, L. (2015). Making the Connection: Allowing access to digital higher education in a correctional environment. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:82-CP:87). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:87 439 Badging digital pathways of learning David Gibson Kathryn Coleman Leah Irving Curtin Learning and Teaching Melbourne Graduate School of Curtin Learning and Teaching Curtin University Education Curtin University University of Melbourne Educators worldwide are witnessing a change in thinking concerning digital learning, teaching and assessment resources as well as the theories and practices connected to making claims about learning based on digital evidence. These shifts are occurring as three elements have combined to form new digital pathways for learning: 1. Self- organizing online global communities engaged in informal learning activities, 2. A new globally supported mechanism for sharing and managing data, files, images and metadata concerning those activities known as ‘open badges’, and 3. Rapidly changing conceptions of higher education, continuing education, and the boundaries of informal to formal learning. So in addition to learners being on a personal learning journey to fulfill their aspirations for professional growth, higher education institutions world wide are also on learning journeys to modernize and respond to these changes, which have the potential for disruption and transformation of the university’s business model and role in society. Keywords: Digital badges, learning pathways, credentials, lifelong learning Introduction Digital pathways of learning are increasingly available to anyone, anywhere at anytime. Traditional learning pathways are ‘approved’ connections or ‘bridges’ that direct learners as they move in and out of courses in the same or different sector. Many learners also develop their own pathways as they traverse formal and informal learning opportunities, courses and programs related to their ideal identity and self, or career paths. These pathways often have some form of signifier of completion and recognition; either a credential, certificate or award plotted along them. A new form of exchange currency embodied in the technology of digital badges can serve as signposts along these pathways indicating points of interest on a learner’s journey. These might include new forms of apprenticeship, competency and so forth, providing transparency and access to a range of audiences and stakeholders. This disruptive form of acknowledgment and recognition of skill, experience and knowledge continues to realise its potential in education. Only recently Open edX launched a digital badge credential where “students will be able to earn badges upon completing a course and share these badges on Mozilla Backpack” (Baruah & Otto, 2015, para. 1). Educators worldwide are witnessing a change in thinking concerning digital learning, teaching and assessment resources as well as the theories and practices connected to making claims about learning based on digital evidence. These shifts are occurring as three elements have combined to form new digital pathways for learning; 1) Self-organising online global communities engaged in informal learning activities, 2) A new globally supported mechanism for sharing and managing data, files, images and metadata concerning those activities known as ‘digital badges’, 3) Rapidly changing conceptions of higher education, continuing education, and the boundaries of informal and formal learning. So in addition to learners being on a personal learning journey to fulfill their aspirations for professional growth, higher education institutions world wide are also on learning journeys to modernise and respond to these changes, which have the potential for disruption and transformation of the university’s business model and role in society. These numerous and diverse pathways of learning often arise outside of formal education, raising the question of when and in what ways higher education will participate in digital badging to give recognition to learning for a range of lifetime achievements that can sit around, within or on top of current credentials and grades. As Grant suggests, “…badges connect multiple spheres throughout lifelong learning, and make pathways of learning visible to others, opening up new opportunities for more people than the current system” CP:88 440 (2014, p.19). New business models for recognition of these learning paths for prior learning assessment (PLA) and evidence based learning are on the horizon with new credentialing organisations such as DeakinDigital (http://www.deakindigital.com) and the potential to unbundle and micro-credential targeted skills and capabilities. This paper presents the potential for technology and social forces to disrupt our current system of accreditation, trust and credentials and create new digital badge learning pathways. What is digital badging? A digital badge can be described through many lenses: through the technical and structural, multiple criteria and purposes, and the social, political and educational processes for award and issue. Most simply put, a digital badge is a web-based enabled technology that by virtue of its technical affordances (e.g. extensible digital format, accessibility, scalability, interoperability), has given rise to a complex global discussion about educational practices and possibilities that are centred on learning evidence, assessment of learning, and how learning is recognised and validated and by whom. Since assessment is key to the determination of status and value of someone’s knowledge, skills and capabilities, and is a key aspect of a formal education, digital badges acquired from anyone, anywhere at anytime represent a dramatic alternative assessment mechanism with powerful disruptive potential for higher education. The question in there lies, who do we trust to warrant evidence, and what evidence is credible? Digital badges are often referred to as a disruptive technology (Carlson & Blumenstyk, 2012). This disruption is twofold, because they can operate outside the conventional award of credentials in higher education and they rival current formal credentials that don’t necessarily represent the skills, experiences and knowledge required for employability and identity when bundled. The cultural practice of creating, awarding and displaying badges has its roots in social media and the open web; the practices “emerged from the intersection of digital games practices, online reputation systems used in commerce (e.g. eBay, Wikipedia and Amazon) and media culture as well as the historical custom of awarding recognition via physical status icons, such as ribbons, medals and trophies” (Gibson, Ostashewski, Flintoff, Grant, & Knight, 2013) with important implications for higher education learning to catch up with social practices on the Internet by becoming more inventive, collaborative, participatory and mobile (Davidson & Goldberg, 2009). Technically, a digital badge that adheres to the Mozilla Open Badge Infrastructure (OBI) is a ‘.png’ image file with metadata. PNG became an international standard when the World Wide Web Consortium adopted it in 2003 (International Standards Organization, 2004). Launched later by Mozilla in 2011, the OBI utilizes the .png standard to create trust networks among issuers, badge recipients, and other consumers, including organisations that recognise badges as signs of skills and achievement (Surman, 2011). Recently, the OBI has been adopted by a number of international professional learning organisations such as IMS Global Learning Consortium to issue digital credentials for professional learning, Pearson and the American Institute of Certified Public Accountants (AICPA). In education, the criteria for awarding a badge (Gibson et al., 2013) generally fall into one of three broad purposes: • Incentivise learners to engage in positive learning behaviours, • Identify and recognise progress in learning and content trajectories, and • Signify, warrant and credential engagement, skills, experiences, knowledge and achievement. Paths into learning A ‘learning journey’ perspective is helpful for thinking about the entry points, waypoints and possible futures for the processes and tools of digital badging in higher education in the form of micro- credentials and credentials. This perspective, views relationships with learners along three phases of their journey: 1. Before they are formally enrolled as higher education students, 2. While they are pursuing formal studies, and; 3. As they move on to other pursuits as lifelong learners in informal and formal learning spaces. We’ll refer to these as ‘paths into’ (Table 1) ‘paths during’ (Table 2) and the ‘lifelong pathway’ (Table 3) of formal learning in higher education. CP:89 441 Table 1. Badges on the path to formal learning Journey Waypoints Badging Examples Access to higher education Learner brings a collection of badges to the review process, which meets admissions criteria. Those badges will have been earned cost- free or at low cost from trusted issuers. Higher education admissions processes accept badges from trusted issuers, creating an alternative currency for pre-university and university-ready learning experiences. Recognition of prior learning Badges become a part of PLA review processes. Trusted issuers ease the burden of the review process. Badges can be stacked in a variety of ways to meet pre-requisites for courses and units. MOOC-like learning Free and low cost access to knowledge becomes ubiquitous; some experiences experiences include trusted badges that signify achievement and are accepted as prerequisites for courses and units. Paths during learning A second set of waypoints on the learning journey in higher education offers badges that might be earned during the time of formal engagement as an engaged or enrolled student. Engaged learners are those who are taking advantage of higher education offerings without a formal enrolment agreement or degree program plan, and enrolled learners are those who are registered to complete a planned degree or credential program. Table 2. Badges on the paths traversed during formal learning Journey Waypoints Badging Examples Personalising at scale – Badges become part of a continuum of personalization strategies by learner control, choice and offering alternative self-directed activities. ‘Badges as bridges’ adaptations facilitate new cross-disciplinary and trans-disciplinary approaches. Unbundling and rebundling Course and unit content is unbundled and badged in new (Bull, 2014) configurations, promoting openness and reuse of teaching and assessment materials. Assessment as networked Badges are awarded by flexible knowledge communities (e.g. peer credibility and expert groups, expert groups, global groups) within, across and extended authority from the university. Digital badges carry the university’s reputation in micro credentialing, while internal badging, points, and awards expand the creative use of motivational rewards and game-based learning in the learner’s digital experience. Scale and automation – In MOOC-like offerings, badging processes enable global scale and a integrating into the grades degree of automation while promoting quality learning experiences. and exams system Evidence-based & The tools and processes of badging (e.g. transparency and competency-focused transportability of outcomes) meld with portfolio assessment assessment processes, promoting the evolution of evidence-based competency- focused assessment in higher education. Lifelong pathways of learning The idea of a badge as a signpost of engagement, learning and achievement continues as the learner’s journey moves past formal education and into lifelong learning. The learner might return for additional advanced study in the future, or might begin to add credentials and experiences to their degree in order to professionally advance and develop their identities, either in the field, online or both. Some of the options during this phase of the relationship of the learner to the university or institute of higher education include, facilitating professional networking, acquiring certifications and credentials that are co-designed and co-developed or recognised by professional communities and associations, and engaging, learning and achieving new heights of knowledge and action across disciplines (Table 3). CP:90 442 Table 3. Badges on lifelong learning paths Journey Waypoints Badging Examples Alumni networks Badges from one’s degree-granting institution help alumni networks to form and adhere around common strengths, interests and aspirations. Professional certifications New certificate programs arise with flexible badge configurations that personalize the learning journey. Co-credentialing and Badges awarded from the institution are co-designed and Community Association endorsed by national and international associations to generate professional community recognition and new forms of leadership. Multi-disciplinary & Inter- Badges issued upon application of evidence to a disciplinary disciplinary recognition community you do not formally learn in, developing new opportunities for learning ecologies and pathways. Paths from informal to formal learning Given the relevance of graduate employability, outcomes based learning and competency based education in the changing space of higher education in Australia, one of the most exciting opportunities for digital badges are in the validation and warranting of those skills and capabilities that lay on the outskirts of the learning journey. The skills acquired in formal learning, bundled and hidden by grades that do not reflect their experiences and knowledge within, work based and work integrated leaning opportunities, internships and experiential learning. Digital badges have the potential to highlight the paths that lay between the ‘certification’ of informal learning, bridged with the many possible uses within a formal educational environment (Glover & Latif, 2013, NPN). The value of digital badges within these contextual situations highlights the issue of credibility and validity. “In order to compete with traditional credentials like degrees that boast centuries of credibility, organisations first need to create systems of badges that structure their educational offerings, serve audience needs, motivate learners to participate, and provide appropriate evidence to back up their claims” (Hickey et al, 2014, exec summary, para.1). Designing evidence-based badges can go one step toward creating an ecosystem that is trusted, valued and credible by involving key stakeholders in the co-design and co-endorsement of the badge. “Integrating experts in the badging process boosts the credibility of the credentials and its value in a knowledge-based economy. This contributes to the validation of the badge and its potential usefulness in professional settings” (Hickey et al, 2014, p.13). Conclusion Digital badges represent an opportunity to rigorously re-consider what evidence-based teaching, learning and assessment is, the role of the informal learning in formal learning pathways and the validity and credibility of our current credentials. Digital badges as markers, waypoints and signposts on a digital pathway have the disruptive potential to re-connect formal higher education systems to the wider world of professional, informal and lifelong learning, with a focus on building the individual capabilities of each learner. They offer new ways for the sector to consider what is learning, who are the credible assessors of learning and what learning evidence is. They also offer new opportunities for higher education institutions to modernise and respond to needs of learners, employers and our disciplines, which in turn, have the potential to transform the university’s role in society and disrupt the business model. References Baruah, P., & Otto, N. (2015). Digital Badges On the Open edX Platform. http://blog.edx.org/digital- badges-on-the-open-edx-platform/ [viewed July 2015] Blumenstyk, G. and Carlson, S. (2012), ‘For whom is college being reinvented?’, The Chronicle of Higher Education, 17 December, (verified 6 September 2015) http://chronicle. com/article/The- False-Promise-of-the/136305 Davidson, C. N. (2009). The future of learning institutions in a digital age Cambridge, Mass.: Cambridge, Mass. : MIT Press. CP:91 443 Gibson, D., Ostashewski, N., Flintoff, K., Grant, S., & Knight, E. (2015). Digital badges in education. Education and Information Technologies, 20(2), 403-410. doi: 10.1007/s10639-013-9291-7 Glover, I., & Latif, F. (2013). Investigating perceptions and potential of open badges in formal higher education. Paper presented at the Proceedings of the World Conference on Educational Multimedia, Hypermedia and Telecommunications 2013. ACCE, Chesapeake, VA. Grant, S. L. (2014). What Counts as Learning: Open Digital Badges for New Opportunities Connected Learning. Irvine, CA: The Digital Media and Learning Research Hub.http://dmlhub.net/wp-content/uploads/files/WhatCountsAsLearning_Grant.pdf [viewed 20 May 2015] Hickey, D., Otto, N., Itow, R., Schenke, K., Tran, C. & Chow, C. (2014). Badges Design Principles Documentation Project: Interim Report January 2014 Update, Center for Research on Learning and Technology Indiana University. Irvine, CA, (verified 9 July 2015) http://dpdproject.info/files/2014/05/DPD-interim-report-v4-january.pdf International Standards Organization. (2004). Portable Network Graphics (PNG): Functional specification. ISO/IEC 15948:2004. [verified 26 May 2013] http://www.iso.org/iso/catalogue_detail.htm?csnumber=29581 Surman, M. (2011). Mozilla Launches Open Badges Project. http://commonspace.wordpress.com/2011/09/15/openbadges-2/ [viewed 26 May 2013] Gibson, D., Coleman, K., & Irving, L. (2015). Badging digital pathways of learning. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:88-CP:92). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:92 444 The Agile Learning Model: Using big data to personalise the acquisition of accounting skills Brent Gregory Matthew Wysel Sue Gregory UNE Business School UNE Business School School of Education University of New England University of New England University of New England Big data mirrors the accounting process to the extent that it deals with how we capture, categorise, summarise and report information so that users can make informed decisions. By modelling this process, we can both demonstrate the future of accounting to our students, and build an agile learning environment that identifies for a student their ‘next crucial action’ in the learning process. Presented in this paper is a pilot study. Keywords: Agile learning, education big data, personalised learning, automated intervention Introduction Effective learning is predicated on feedback, as feedback enables the student to determine the effectiveness of their efforts in achieving the desired goal (Gregory, Uys & Gregory, 2014). This feedback can vary from: personalised to heuristic, continuous to discrete, adaptive to rigid, and is regarded as the key to self-regulated learning environments and the challenge in classical, didactic environments (Butler & Winne, 1995). In more sophisticated and effective learning environments both the student and the teacher actively participate in a mutual process of feeding back and processing information regarding the efficacy of the learning process to one another, with the common goal of refining the process (Cantillon & Wood, 2011). Feedback, therefore, relies on the provision and recording of data, its categorisation and summarisation, and timely reporting to one or more parties within the learning process. Historically, administrative overheads associated with evaluating the numbers of students typically found in school and university classes has enforced highly uniform, and therefore structured, evaluation processes with corresponding rigidity within tuition programs. In this case, institutions’ operational requirements to process sufficient volumes of students drive an increase in curriculum rigidity, a reduction in the number of assessments – and therefore a widening of feedback loops, and consequentially force students to increase their reliance on heuristic techniques to guide them through their learning environment. In order to develop our agile learning system a pilot study was undertaken through the selection of student activities, assessing learning and gathering feedback, we may need to step back from of our existing operational assumptions. This will enable us to adapt to realities such as the inputs to the system, (the students), are not homogeneous. A starting point may be to garner information about the student that does not come from the traditional assessment tasks. As well, we should develop resources that are outside the normal learning resources for a particular unit of study. One of our initial challenges will be to identify the questions that we need to ask. This suite of challenges is precisely where big data is most powerful. The commercial utilisation of big data hinges on the ability to collect large quantities of discrete sample points, assemble them for analysis and assessment and respond to the aggregated information in a timely manner. Typically, the commercial collection of sample points within big data applications involves an attempt to garner high quality information about the true position, activities, or knowledge of the subject, while minimising the impact of external influences associated with the assessment. Moreover, the systems created to capture and respond to the collection of big data are designed to retain sufficient agility to deliver personalised content to individuals based on the statistical correlation of one subjects’ current position to numbers of previous subjects’ next successful actions. Big data provides schools and universities the framework to: leverage data created by large numbers of students in the past; CP:93 445 facilitate sufficient continuity in assessment of current students to minimise external factors introduced by infrequent and high-value assessments; and provide the means by which a curriculum can be tailored to the point of personalisation. Big data is the existence of large quantities of discrete sample points, observed by a unified system, and assembled to yield the potential for analysis either between different sample points or across equivalent sample points at different time intervals. The U.S. National Institute of Standards and Technology summarises these same elements as volume, variety, velocity and variability (U.S. Department of Commerce, 2015) and they acknowledge that there are a range of working definitions, including the view of Drew “Big data, which started as a technological innovation in distributed computing, is now a cultural movement by which we continue to discover how humanity interacts with the world—and each other” (Drew as quoted in U.S. Department of Commerce, 2015, p. 11). Background AFM101 is the introductory accounting unit that is core to all business awards at the University of New England. As a result, it has a diverse student cohort and has been a unit that has experienced high attrition and failure. Typically, more than 50% of students that enroll in the unit do not successfully complete the unit. A range of data is used in order to match a particular student with the resources that will be most useful to them. Over recent years, a significant set of resources to solve particular learning problems has been developed. However, the general development of resources has been ad hoc rather than structured. This means benefits to students are piecemeal. To provide an example of the process, we will now peer through a narrow window of time and data collection points through a pilot study. Big data mirrors the accounting process to the extent that it deals with how we capture, categorise, summarise and report information so that users can make informed decisions. We have attempted to model this process not only as a demonstration of accounting but also to help us develop an agile learning system. Data is collected from assessment tasks and from other sources as well, including student questionnaires. A variety of vehicles are used to process the data (Moodle [Learning Management System], Excel, Qualtrics) and the results of the processing are imported to a central database (Excel). Reporting occurs on a number of levels and the key reporting aims to identify the resources that are likely to be most effective for a particular student and then guide the student to their next crucial action (NCA). When that is completed, the process reports the NCA. The process currently requires substantial human intervention and would benefit both students and the teaching staff if the systems were sufficiently compatible to improve the level of automation. Several other education systems have adapted to employ some version of an agile learning path (see, for example, http://www.lynda.com and http://www.khanacadamy.org), but noticeably traditional tertiary institutions have been slow to adopt this method. The growth in sophistication and adoption of distance-based learning has both undercut the ad hoc personalisation of education through socialisation that formally happened during tutorials or small-group meetings, and enabled a new breed of personalisation, centered around flexible learning patterns. Fundamentally, this personalisation of curriculums leverages the inherent agility facilitated by the application of big data to large quantities of data. Our progress to the agile learning system has been a gradual process. In this paper we highlight using components of the big data model to personalise the student learning experience. While students produce torrents of data as a by-product of their ordinary operations, and each of us is now a walking data generator (McAfee, Brynjolfsson, Davenport, Patil, & Barton, 2012), the key discipline is still collecting the data. In traditional data collection models, we collect and store data primarily around final results in assessment tasks. However, the increased opportunity for data collection, analysis, reporting and responding that online computer systems have created now enables us to collect data before, during and after the trimester. Using these additional data points, we are able to refine both our understanding of student’s needs and facilitate the management of these needs. In this paper we will focus on data associated with students’ basic math skills and their learning style as categorized by a representative systems bias test and the conclusions and potential interventions that can be automated as a result of this granular knowledge. Following we will describe the recent history and the consequences of adding these extra data points. CP:94 446 Preliminary results Methodology A pilot study was undertaken over two trimesters to see if interventions implemented in the second iteration of the teaching were successful, through the comparison of two cohorts of students, in 2014 and again in 2015, through a total of 332 students (155 in 2014 and 217 in 2015). All students were studying in off-campus mode (i.e., online, from a distance). 2014 Trimester 2 (115 off campus students) At the commencement of each trimester, students were provided with the opportunity to participate in two diagnostic tests: The Diagnostic Math Test (DMT) and the Representational Systems Biases Test (RSBT). The results of these tests are quantiatively assessed and used to personalise the learning of particular categories of students. Of particular interest are those students whose DMT and RSBT results align with the results of previous generations of students who went on to fail or under-perform in the unit. As more data collection points were added, there was the capacity to increase the level of personalisation and insight. This was especially true as we combined data points from different aspects and contexts within the same sample set. The DMT comprises of 10 reasonably straight forward math questions and generally takes around five to seven minutes to complete. Each question is awarded 10 marks. The RSBT comprised of five questions which provide some indication about the person’s preferred way to represent information (visual, auditory, kinaesthetic or auditory digital). Based on the results achieved in these tests, students were then guided toward different resource sets. To this stage, the personalisation has been unsophisticated. A student with a low DMT score (less than 8/10) was directed to math resources and other learning resources were highlighted as available to students depending upon their RSBT results. 2015 Trimester 1 (217 Off Campus Students) The same process was applied in the following trimester that this unit was taught, in 2015. However, based on outcomes from 2014 Trimester 2, two significant interventions were introduced: 1. The development of a Math Help Area on Moodle which was made up of a selection of basic math videos. All students that scored 70 or less in the DMT were directed to the Math Help area. Students who scored 80 were made aware of these resources in case they chose to utilise them; and, 2. A slightly more proactive approach identified to students at risk. This especially related to student non-participation in these activities, initially by automatically generated personalised emails and built to a phone call from an academic. Results 2014 trimester 2 (115 off campus students) The results of these diagnostic tests have strong predictive value in identifying students who could excel and those that may struggle. Table 1 indicates the key role math skills play in students succeeding in the unit. It provides an analysis of the DMT results for 115 off campus students in Trimester 2, 2014 (the full cohort that were enrolled). Students were divided into three categories based on their final grade in the unit. ‘Not Succeed’, includes all students that failed or did not complete the unit. ‘Pass’ includes all students who receive a grade of pass or credit and students who receive a distinction or better made up the ‘Excel’ group. The table reports student categories based on their score in the DMT. For instance, 84% of students that did not complete the DMT did not pass the unit. Whereas, for the students that scored 100% in the DMT, 39% did not complete the unit and 32% excelled. Table 1: Analysis of result in Diagnostic Math Test Unit result organised by score in DMT CP:95 447 2014, Trimester 2, Off Campus (n=115) 2015, Trimester 1, Off Campus (n=217) Score Excel Pass Not Succeed Score Excel Pass Not Succeed #NA 0% 16% 84% #NA 9% 22% 69% <70 0% 33% 67% <70 0% 67% 33% 70 0% 60% 40% 70 38% 38% 25% 80 25% 25% 50% 80 13% 63% 25% 90 17% 46% 38% 90 23% 39% 39% 100 32% 29% 39% 100 35% 31% 34% When the results are overlaid for the RSBT on the ‘Not Succeed’ group, it was found that 75% of failures were made up of students that did not complete the RSBT. At the other end of the spectrum, students who scored full marks in the DMT were identified as ‘auditory digital’ in the RSBT. Only 18% of this group did not pass the unit and 43% appeared in the ‘excel’ category. We can further improve the predictive value when we overlay other data points (repeat student, award enrolled, phone number available). These results also highlight that our ability to predict outcomes was superior to our ability to bring positive change, which in turn implied a need to review the existing intervention strategies. 2015 Trimester 1 (217 off campus students) A similar pattern of results occurred in Trimester 1, 2015 as displayed in Table 1 (2015). However there are some differences. When compared to 2014, we notice a general improvement in grades, with a substantial improvement in results for students with scores of 70 or below in the DMT. This group were directed to the Math Help resources. During the trimester, six videos were viewed a total of 130 times for a total of 194 minutes. However, no data was recorded on the extent to which individual students took up these resources. While it is possible that the math help resources made a contribution, the extent of that contribution may not be significant. For instance, all students who scored 70 in the DMT and still achieved a HD also identified as ‘auditory digital’ in the RSBT. So other factors could also at play. Limitations of the study The small sample size used means the value of this study is more as a pilot program and preparing for the future than in reporting outcomes that we can confidently act upon. Reporting the outcomes were also challenging because of the multitude of ways that the data can be presented. Further research In the immediate short term, we will extend the size of the database to include all students that have enrolled in the unit in the last four years and include all of the data points that have been collected. As well, we will investigate the utilisation of commercially available, large, unstructured database platforms to host and analyse all data which will support a more rigorous pattern analysis. Findings and conclusion Using principles from big data, we can develop an agile learning system that will enable a more personalised learning path for individuals. The appropriate use of data can signpost students to their next crucial action. Key to our future success will be how we better identify, collect and index the most useful data. This data must extend beyond results of assessment tasks. While the rich combinations of the data provides a challenge in global reporting, it does support precision in guiding the students’ study path. There is also a challenge in either developing or identifying the resource warehouse needed to support students and then in mapping the various paths through the resources. Asking the right questions will be key to developing an agile learning system. To date, the identification of these questions has been a cyclic process. For instance, we find a pattern in the data CP:96 448 (e.g. students that were directed to the math help resources did better than expected), but when we attempted to find what caused that pattern we found that we had not collected the data that would most help us identify the causes (to what extent did they use the resources? did those resources help?). In the words of Google’s director of research, Peter Norvig “We don’t have better algorithms. We just have more data” (McAffee et al., 2012, p. 63). We need not only more data, but we need the appropriate data. And, we need a unified system that will enable us to work with that data. Finally, when the data has identified what action is needed, we need the resources that will support the specific needs of each student. This requires the development, warehousing and indexing of the resources. Success requires cooperation – particularly between those with access to the data and those with access to the analytical tools and platforms. References Butler, D., & Winne, P. (1995), Feedback and self-regulated learning: A theoretical synthesis, Review of Educational Research, 64(3), 245–281. Cantillon, P., & Wood, D. (2011). ABC of Learning and Teaching in Medicine: Wiley. Gregory, B., Uys, P., & Gregory, S. (2014). The role of instant feedback in improving student understanding of basic accounting concepts. In B. Hegarty, J. McDonald, & S.-K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014 (pp. 634– 637). https://app.box.com/s/016cdyv8dq1pp0yhp1vw/3/2704865194/23032568124/1 McAfee, A., Brynjolfsson, E., Davenport, T. H., Patil, D., & Barton, D. (2012). Big data. The management revolution. Harvard Bus Rev, 90(10), 61–67. U.S. Department of Commerce. (2015). Draft NIST Big Data Interoperability Framework: Volume 1, Definitions (pp. 1–33). Gaithersburg, MD. Retrieved from http://dx.doi.org/10.6028/nist.sp.1500-1 Gregory, B., Wysel, M., & Gregory, S. (2015). The Agile Learning Model: Using big data to personalise the acquisition of accounting skills. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:93- CP:97). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:97 449 PST Online: Preparing pre-service teachers for teaching in virtual schools Steve Grono Yvonne Masters Sue Gregory School of Education School of Education School of Education University of New England University of New England University of New England Improvements in available technologies and an increased popularity of online learning spaces have seen a shift in the dominant ways students engage with formal and informal learning in their day-to-day lives. This is especially true for the distance education experience through the rise in virtual schools. As this shift occurs, it becomes increasingly important to reflect these new changes in curriculum design for pre-service teachers. Increasingly, these pre-service teachers will be engaging with students, not just in the traditional, physical classroom space, but also in online spaces and via distance. These new virtual learning environments require their own separate skillset to be properly navigated by both the learner and teacher to provide meaningful and rich learning experiences. In order to develop resources to facilitate the learning of these skills, current pre-service teachers have identified their own understandings of online learning and their readiness to teach within these new spaces. Keywords: virtual schools, pre-service teachers, online teaching, OER, distance education Introduction and background The growth of online collaboration and communication opportunities has expanded the potential for learners to create meaning across a range of new, modern teaching spaces. The online environment also creates the capacity for teaching to occur in new and more powerful formal and informal types of distance education. As dominant modes of learning shift, it has become increasingly important to recognise the need to ensure that pre-service teachers both are aware of the new skills required for teaching into increasingly online and virtual learning environments and are well supported in this transition. In particular, the recent emergence of virtual schools reflects new, purely online spaces that pre-service teachers may find themselves teaching into. The University of New England (UNE), Australia, is a world-leader in distance and blended learning, with the majority of its students studying by distance (over 80%) (University of New England, 2014). This provides academics with a higher education perspective of teaching and learning within the virtual space, and first-hand experience of the changing needs and expectations for high quality engagement, content delivery and behaviour management . However, despite this, many teacher education programs, especially those focused on Primary and Secondary Education, are not yet adapting to adequately cover this need of “preparing pre-service teachers to teach in this [virtual] environment successfully” (Bull, 2010, p. 29), instead focusing on face-to-face, physical teaching environments. This paper reports the first phase of a research project funded by the Office for Learning and Teaching (OLT), focused on preparing pre-service teachers for teaching in virtual schools. The project aims to create a suite of resources for pre-service teachers to use and draw on to support their ability to successfully engage in a virtual school teaching environment. The initial phase centres specifically on exploring the perceptions of current teacher education students regarding readiness for online teaching, their understanding of what virtual schooling is, and identifying key topics which they see as important as preparation for teaching into these virtual spaces. Literature review CP:98 450 It has been predicted that “by the end of the next decade, secondary schools will offer up to half of all courses in virtual formats” (Bull, 2010, p. 29). Indeed, in the United States of America, “virtual schooling is a fast growing option for K–12 students” (Kennedy, 2010, p. 21). Similar forays into virtual schooling are occurring in New South Wales where a new 7-12 virtual high school, Aurora College, opened this year, 2015. These changes in education delivery will necessitate a new approach to curriculum design: a re-shaping of discipline-based courses in higher education institutions with regard to teacher education and also re-definition of the use of information and communication technologies. In developing a suite of resources to assist pre-service teachers with the development of online teaching skills, the researchers draw on the currently available literature as to what constitutes best practice in online teaching. In Australia, there is very little written about such teaching in schools and it is necessary, for the Australian context, to draw on the literature around online teaching in higher education institutions (Downing & Dyment, 2013; Gregory & Salmon, 2013). A more specific bank of literature has emerged in North America, including not only research into school online teaching requirements (DiPietro, Ferdig, Black, & Preston, 2010; Murphy & Manzanares, 2008; Murphy, Rodríguez-Manzanares, & Barbour, 2011), but also the development of standards for teachers in online environments (International Association for K-12 Online Learning, 2011; International Society for Technology in Education, 2008). It is argued in the literature that teaching online necessitates a different range of skills from those currently covered in teacher education programs. It is also argued that classroom management, including behaviour management, is one of the most important challenges for teachers and one of the biggest concerns of pre-service teachers (O'Neill & Stephenson, 2011; Peters, 2009). While initial teacher education programs deliver a range of units designed to overcome these concerns by developing the requisite face-to-face skills, online teaching changes the dynamics and “necessitates a shift from a practice of controlling to engaging students’ attention” (Murphy & Manzanares, 2008, p. 1061). These researchers argue that there are contradictions in moving from face-to-face teaching in a conventional classroom to teaching online and teachers “may benefit from opportunities to develop new skills, techniques and strategies” (Murphy & Manzanares, 2008, p. 1070). The current project, in developing a resource to assist both teacher educators to teach new skills and pre-service teachers to understand these skills, has the capacity to begin to prepare new teachers for st the realities of 21 century education (Partnership for 21st Century Skills, 2009) and thereby, in the long term, to improve education opportunities for school age children. Method Before developing resources to support pre-service teachers’ understandings of online education, it was important to engage with current pre-service teacher education students to gauge their understanding of what virtual schooling is, their perceived strengths and weaknesses within this arena, and key areas they felt were important or problematic to be addressed within these resources. A survey was developed covering these and other questions around online education, with responses to inform future direction and focuses for the project. The survey was sent to current UNE students studying across twelve of the initial teacher education degrees offered at the university. This sample captures the views of pre-service teachers from a range of contexts including: Primary, Secondary and Early Childhood; on-campus and external cohorts; and those both early and late into their degree of study. The survey received 202 respondents from across these contexts, providing a rich and varied perspective regarding their knowledge and beliefs about online teaching. The participants ranged in age from under 21 to over 65, with the predominant age bracket being the 26 to 45 age group (two groups), as depicted in Figure 1. CP:99 451 0% Under 21 9% 5% 12% 21 to 25 1% 26 to 35 36 to 45 41% 46 to 55 30% 56 to 65 Over 65 Figure 1: Age of Student across all pre-service teacher courses Initial demographic questions provided the frequencies of age, gender and location of the participants. The majority of participants in the survey were female, 167 (83%), with 35 males (17%). This is typical of pre-service enrolment at UNE. Table 1 provides information in relation to where the participants were located. These results not only reflect the range of respondents, but also highlight patterns of average pre-service teaching degree participants. Of particular note, the data highlight the average pre-service teacher as being in their thirties, and thus not a direct school leaver themselves, and most frequently located in capital cities. This latter observation reflects pre-service teachers’ own engagement with online distance education as a viable learning environment in contrast to physical locations of study. This higher education experience, with either blended or wholly online learning, provides them with a learner’s perspective of virtual teaching and learning. However, the perspective requires explicit expansion to adapt these skills and experiences to provide effective virtual school teaching in pre-tertiary contexts. Table 1: Participant location statistics breakdown Participant Location Response % Rural 11 5% Rural Town 19 9% Small Regional Town / City 26 13% Small Non-Regional town / City 6 3% Regional City 35 17% Non-Regional City 3 1% Regional Major City 24 12% Non-Regional Major City 9 4% Capital City 69 34% Results Participants were asked to rate their knowledge of Information and Communication Technology (ICT) skills and online teaching capabilities on a 5 point Likert scale ranging from very low to very high. They were also asked to contrast this knowledge prior to beginning their course with their view of their current knowledge. In all cases, students identified their skills and knowledge as improving since beginning their course – with the mode response shifting from ‘average’ to ‘high’ in both instances (see Table 2). This shift is indicative of positive student engagement with technology enhanced learning throughout their studies, through both modelled use of learning technologies integrated into their units of study and explicit ICT in Education units of study. In the open-ended questions, participants were asked to consider important factors for developing a positive online learning experience and also what concerns they had or challenges they might face if appointed to teach using online technology. They identified a range of important factors in developing positive online learning experiences. Responses often drew from participants’ own higher education experiences with online learning, much of which parallels the needs of teaching within virtual schools, CP:100 452 including: timely access to teachers and support; online etiquette; quick response time; meaningful feedback; the ability to interact with other students to develop relationships and build meaning; tailoring content to individual needs; and pacing. In contrast, other responses highlighted the dissonance between important skills in tertiary online education and those involved in virtual schooling – a focus on time-management skills for online learners, over behaviour management in an online space. Table 2: Student self-perceptions of ICT skills & online teaching readiness Question Very Low Ave- High Very Mean Low rage High (1) (2) (3) (4) (5) Your skill level with respect to ICT in general 6 22 82 55 21 3.34 prior to commencing your current course Your skill level with respect to ICT now 1 2 65 89 29 3.77 Your knowledge level with respect to teaching online prior to commencing your 30 52 72 26 6 2.6 current course Your knowledge level with respect to 6 21 58 84 17 3.46 teaching online now Even experienced academics find the pedagogy involved to teach effectively online is quite different from learning online. A recent example of this was when a professional development workshop, presented by two experienced online teaching academics, elicited an interesting response from the academics. They were running a workshop for local teachers. However, not all could attend in person. Therefore, half the participants attend face-to-face whilst the remaining attended through video- conferencing. The academics stated that they were ‘struck by how different it was to work with the teacher mentors who were online compared to the face-to-face.’ There was a realisation that this particular academic ‘needed to reconsider [her] pedagogy for any future workshops presented for the project’. Hence, there is a need for pre-service teachers, with the changing technological world, to be able to teach online as opposed to learn online. Conclusions and next steps Presented is a snapshot of the wide range of pre-service teacher contexts and identified self- perceived strengths and knowledge in teacher education students towards ICT education and online learning. Interestingly, pre-service teacher responses were more focussed on their learning as opposed to their teaching (or future teaching). In addition to identified strengths and areas pre-service teachers identify as beneficial to the creation of positive online learning experiences, the data also reveals gaps and misconceptions in student understandings of virtual schooling and online education within the pre-tertiary space, which can be captured when designing resources to facilitate the further development of these pre-service teacher skills. The survey data provides a rich overview of current pre-service teachers’ understandings of virtual and online teaching, and identifies key problematic areas to be focused on in the next phase of the project – the creation of an open, online collection of resources for pre-service teacher use centred on virtual teaching. These resources will be developed in the form of an educational website containing a suite of resources for use by current and pre-service educators. The site aims to contain a series of modules around identified topic areas, and contain support materials, short videos and case studies, and classroom resources within each module. The use of open source learning platforms such as Moodle and WordPress will allow for adaptable yet accessible resources in a familiar and easily navigable format for pre-service teacher use, and allow resources to be built utilising a range of learning management tools and activities to facilitate a broad range of multi-modal materials across learning styles and needs. At the conclusion of the survey, participants were able to identify themselves and opt-in to a follow-up workshop to be held in Armidale or Parramatta. These workshops will further assess quality and use cases of the online resources developed and inform further improvements and modules to benefit pre- service teacher readiness and understanding of online education environments within their own pre- tertiary contexts. CP:101 453 References Bull, G. (2010). The always connected generation. Learning and Leading with Technology, 38(3), 28– 29. DiPietro, M., Ferdig, R. E., Black, E. W., & Preston, M. (2010). Best practices in teaching K-12 online: Lessons learned from Michigan Virtual School teachers. Journal of Interactive Online Learning, 9(3), 10–35. Downing, J. J., & Dyment, J. E. (2013). Teacher educators' readiness, preparation, and perceptions of preparing preservice teachers in a fully online environment: An exploratory study. The Teacher Educator, 48(2), 96–109. Gregory, J., & Salmon, G. (2013). Professional development for online university teaching. Distance Education, 34(3), 256–270. International Association for K-12 Online Learning. (2011). National standards for quality online teaching. Virginia, USA: International Association for K-12 Online Learning. International Society for Technology in Education. (2008). ISTE Standards Teachers. Washington DC: International Society for Technology in Education. Kennedy, K. (2010). Cross-reference of online teaching standards and the development of quality teachers for 21st century learning environments. [Article]. Distance Learning, 7(2), 21–28. Murphy, E., & Manzanares, M. A. (2008). Contradictions between the virtual and physical high school classroom: A third-generation Activity Theory perspective. British Journal of Educational Technology, 39(6), 1061–1072. Murphy, E., Rodríguez-Manzanares, M. A., & Barbour, M. (2011). Asynchronous and synchronous online teaching: Perspectives of Canadian high school distance education teachers. British Journal of Educational Technology, 42(4), 583–591.. O'Neill, S. C., & Stephenson, J. (2011). Classroom behaviour management preparation in undergraduate primary teacher education in Australia : A web-based investigation. Australasian Journal of Teacher Education, 36(10, Article 3), 35–52. Partnership for 21st Century Skills. (2009). 21st century learning environments. Washington DC. Peters, J. (2009). First year pre-service teachers’ learning about behaviour management. Paper presented at the Australian Association for Research in Education Annual Conference, Canberra. University of New England. (2014). UNE Overview. Retrieved from http://planning.une.edu.au/Statistics/overview/index.htm Grono, S., Masters, Y., & Gregory, S. (2015). PST Online: Preparing pre-service teachers for teaching in virtual schools. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:98-CP:102). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:102 454 Occupational Medicine Simulation Project Aaron Griffiths F/Xual Education Services In 2013 the Occupational and Aviation Medicine (OAM) unit of the University of Otago secured a project grant to develop a simulated virtual world environment for students of this unit, specifically those studying occupational medicine as distance learners. The simulation would be employed by facilitated student groups to contextualize occupational data for specific work processes, to re-enact occupational health examinations in the compiling of clinical assessments and to develop a research proposal for assessing health outcomes in these hazard environments. Developmentally, the underlying intent of the project was twofold; firstly, to investigate the virtual elements essential to the creation of an authentic context for learning and secondly, to explore those virtual aspects that might provide a supportive learning environment for the geographically dispersed student body. This paper details the pedagogical and design rationale employed by the author in the pursuit of this intent. Keywords: Virtual, simulation, occupational medicine, contextual learning, authenticity, presence. Project Overview The project brief from the Occupational and Aviation Medicine (OAM) unit was to create a virtual environment representative of areas of the cement manufacturing process, based on quarries and cement factories in the United Arab Emirates. Work processes are an integral part of occupational medicine’s week-long residential school, where group discussions around the occupational hazards in a particular environment, the compilation of workers’ occupational health histories and the role-playing of clinical assessments, all form part of an overall exercise in the development of a research proposal for assessing health outcomes in a particular hazard environment. Limited to one residential school per year due to the students’ geographical distribution, the project proposes to address this limitation by providing opportunities to engage in these exercises remotely. The open source server platform OpenSimulator was chosen as the test bed for the simulation build, its cost-effectiveness, agile development capability and scalability all components in the selection. The open source virtual world viewer Firestorm enables users to enter this environment as a customisable avatar and engage with both the virtual environment and each other; the former through touch and menu options, the latter through text and voice chat. The build itself would involve two main features; the manufacturing process areas and the clinic where the assessment of workers would take place. The process areas would be presented as identifiable immersive spaces, allowing participants to experience the occupational data in the context of the actual work environment from which it was gathered. While the simulation cannot represent all the occupational hazards, heat or vibration for example, the essence of these might still be experienced through viewing the data in association with the environment. As Heerington, Reeves and Oliver (2007, p.85) state “the physical reality of the learning situation is of less importance than the characteristics of the task design and the engagement of students in the learning environment.” In a similar manner the clinic would provide an immersive environment conducive to the task at hand, the occupational health assessment of workers, providing that “strong sense of situation” (Falconer, 2013, p.298) essential to an authentic experience. The clinic would enable students to perform different medical examinations and order a number of medical tests, effectively progressing through all the tasks related to an occupational health consultation. The worker would be role-played by a facilitating staff member, enabling students to also practice the communication skills necessary for effective patient examination. The Pedagogical Foundation As a general approach, the use of virtual environments for health and medicine education has been well documented (Creutzfeldt et al., 2010, Danforth et al., 2009, Loke, Blyth and Swan, 2012, Wiecha et al., 2010) and acknowledges the benefits of using these environments for simulated medical CP:103 455 practice. These studies also give credence to the immersive aspects of virtual environments as enabling situated learning, Loke, Blyth and Swan (2012, p.570) stating that students were able to “to call the shots and to live through the consequences of their decisions” in the simulated environment, constructing knowledge throughout the experience, rather than just responding to lineal, paper-based case studies. The use of well-executed virtual world simulations has also been evidenced to provide highly experiential learning spaces (Nygaard, Courtney and Leigh, 2012, Feinstein, Mann and Corsun, 2002) with an authenticity of learning experience that, significantly in the author’s view, is not necessarily bound to the physical fidelity of the simulation, but more related to the cognitive fidelity of the tasks provided (Heerington, Reeves and Oliver, 2007). In the context of the application virtual environments are also being investigated as a means of enabling social presence and co-presence for a student body that is highly dispersed geographically; aspects shown to provide both an enhanced sense of community and increased satisfaction for learners (Bulu, 2012, Edirisingha et al., 2009). Research suggests that virtual worlds do possess this capability, enabling a sense of being together and providing a possible means to negate the isolation and loneliness often experienced by distance learners (Hassel et al., 2012, Johnson, and Levine, 2008). Additionally, the provision of an educational space that offers an immersive experience predicated in context, would, through authentic learning, support an improved student performance (Chapman and Stone, 2010). The Simulation The Simulation Orientation Area The orientation area is the space first encountered when students log into the simulation. Its purpose can be likened to a halfway point between the physical world and the task-based environments, where students are introduced to the mechanics of moving, seeing, communicating, costuming their avatar and interacting with each other and the simulation. It also contains the introductory materials that relate to the course objectives and breakout areas for class discussions and round-ups. Visually it sets the scene, introduces users to the simulation’s conventions of use and readies them for engaging with the environments to come. Figure 1: Orientation Area/Process Area Environment (Quarry) The Process Area Environments For this trial project the worker selected for the clinical assessment was to be a 52 year old, male, heavy vehicle driver who, for the past 30 years, has been transporting rock from the quarry to the cement plant’s primary crusher. These areas therefore would be the required process areas necessary for an overview of this worker’s occupational environment. Operationally, occupational medicine clinicians do not normally access these environments; rather they assess health outcomes based upon field measurements taken by other agencies and consider these alongside the occupational history and clinical assessment of the worker. Situating the students in the simulated work environment was not then a necessity for the collection of data but more about giving consideration to the idea that information provided in an authentic context supports the integration of theory into practice (Falconer, 2013). From a cognitive task perspective, this method offered opportunity for more robust group discussions around aspects such as data accuracy as opposed to CP:104 456 just being presented with a data sheet as a fait accompli, encouraging higher levels of engagement and supporting the possibility of emergent learning behaviour in the participating student group (Kays and Sims, 2006). In the process areas’ development a limited budget did not allow for the creation of high fidelity models, nor were they deemed necessary from an immersive perspective. Other affordances of the virtual world were available; the perception of space, size and distance; atmospheric factors such as dust and noise; each adding to the immersive quality of the environment and supporting a sense of being there. Additionally, Gustafson (2001) considers that the meanings we give to a place are more often not just about our relationship with the environment, but an amalgamation of the relationship between one’s self, other occupants of the place and/or the environment. This concept was given consideration by not only providing environments that spoke to the participants as places relevant to their learning; occupational spaces; but by providing the ability to dress for those spaces appropriately, bringing authenticity to one’s own presence in the simulation and visual reinforcement through the presence of others, dressed for their roles. The locker room enabled this capability. The Locker Room Evidence suggests that the success of learning activities conducted in virtual world environments has a correlation to the degree of embodiment and presence students have been able to form (Peachy and Childs, 2011) and that identity and embodied presence are interconnected (Mennecke et al., 2011). Ganesh et al‘s (2012) work on self-identification with virtual agents points to the representation of self from a third person perspective as a facilitating factor in appropriating the avatar to the user’s self-identity, while other works additionally point to the importance of appearance for the construction of identity in virtual worlds (Martey and Consalvo, 2011, Neustaedter and Fedorovskaya, 2009). In light of this research, supporting presence through the creation of identity was considered an essential factor in enhancing not only the immersive experience but the learning experience as well. In this simulation however, appearance, i.e. dressing for the part, is not just tied to identity creation but is a cognitive task in itself, as appropriate outfitting in the required protective clothing for each process area environment, based on the hazard data for those areas, provided another aspect of situational context and learning. What the locker room then provided, over and above its usability intent, was an environment for the observation of self, where preparing for the role was the focus. This focus might be likened to a dressing room, the participants as actors in that preparatory stage for a play they are about to perform. Goffman (1956), in his consideration of self, talks of the performer’s belief in the part they are playing and how their impression of the reality of their performance influences in turn their audience’s; others; impression of reality. This creation of self as an in-world identity, appropriately dressed for a chosen process area environment would, in the author’s view, enhance that impression of reality and support that belief, or rather the suspension of disbelief, enhancing not only one’s own sense of presence and immersion in the virtual space but the sense of presence and immersion of the other participants; the audience. The Medical Assessment Clinic The clinic focuses on providing an environmental authenticity to the clinical assessment of the worker, with locker room costuming enabling students to assume an identity to support this authenticity, i.e. through the provision of theatre greens. Realistic simulations of the assessment tasks have not been developed, rather the students may choose from a varied number of physical examinations and medical tests, provided as selectable choices through a heads-up-display (HUD) and associated menu options, to assess the worker. This approach considered that there is no real educational benefit in constructing the physicality of taking, for example, the pulse and accruing the associated development cost. Rather the students just click on the take pulse button and the HUD responds with a programmed pulse rate response; in this manner the pulse taking has been acted out. This is a reiteration of the “cognitive realism” of tasks being of greater importance than the creation of realistic simulations of the events (Heerington, Reeves and Oliver, 2007). The Heads-Up-Display (HUD) CP:105 457 The HUD, mentioned above, also provides a consistent aspect across all areas of the simulation. It facilitates movement to all areas and costume assembly in the locker room and offers multiple forms of engagement with the process area environments through the presentation of hazard data, graphical depiction of specific hazards, hazard exposure calculators, etc., encouraging the possibility of learner-centric experiences (de Freitas and Nuemann, 2009) and active engagement, inside and outside of facilitated class activities. Figure 2: Locker Room/Accessing Hazard Data and Calculators Final Reflections Though yet to be provided with test students, there has been evidence, in the consultative demonstrations conducted, that this project will achieve its aims. It delivers an engaging, authentic and immersive experience, providing opportunities for situated, experiential and collaborative learning for the student community. It encourages a learner-centric cognitive approach with the tutor’s role as a facilitator of that process. A concluding statement from Heerington, Reeves and Oliver, (2007, p.94) seems appropriate; “When appropriate technologies can be selected as required and used as cognitive tools to solve complex problems, the responsibility for learning moves back to the learner, rather than the designer of the virtual environment.” References Bulu, S. (2012). Place presence, social presence, co-presence, and satisfaction in virtual worlds. Computers & Education. 58, 154–161. Chapman D. D. and Stone S. J. (2010). Measurement of outcomes in virtual environments. Advances in Developing Human Resources. 12 (6), 665-680. Creutzfeldt, J., Hedman, L., Medin, C., Heinrichs, W. L. and Felländer-Tsai, Li. (2010). Exploring virtual worlds for scenario-based repeated team training of cardiopulmonary resuscitation in medical students. Journal of Medical Internet Research. 12 (3). Danforth, D., Procter, M., Heller, R., Chen, R. and Johnson, M. (2009). Development of Virtual Patient Simulations for Medical Education. Journal of Virtual Worlds Research. 2 (2). de Freitas, S. and Nuemann, T. (2009). The use of ‘exploratory learning’ for supporting immersive learning in virtual environments. Computers and Education. 52 (2), 343-352. Edirisingha, P., Nie, M., Pluciennik, M. and Young, R. (2009). Socialisation for learning at a distance in a 3-D multi-user virtual environment. British Journal of Educational Technology. 40 (3), 458– 479. Falconer, L. (2013). Situated learning in virtual simulations: Researching the authentic dimension in virtual worlds. Journal of Interactive Learning Research. 24 (3), 285-300. Feinstein, A. H., Mann, S. and Corsun, D. L. (2002). Charting the experiential territory. Clarifying definitions and uses of computer simulation, games and role play. Journal of Management Development. 21 (10), 732-734. Ganesh, S., van Schie, H., de Lange, P., Thompson, E. and Wigboldus, D. (2012). How the human brain goes virtual: Distinct cortical regions of the person-processing network are involved in self- identification with virtual agents. Cerebral Cortex. 22 (7), 1577-1585. Goffman. E. (1956). The Presentation of Self in Everyday Life. London: Penguin. CP:106 458 Gustafson, P. (2001). Meanings of place: Everyday experience and theoretical conceptualizations. Journal of Environmental Psychology. 21, 5-16. Hassell, M., Goyal, S., Limayem, M. and Boughzala, I. (2011). Effects of presence, copresence, and flow on learning outcomes in 3D learning spaces. Administrative Issues Journal: Education, Practice & Research [online]. 2 (1), 62-73. Herrington, J., Reeves, T. C. and Oliver, R. (2007). Immersive learning technologies: Realism and online authentic learning. Journal of Computing in Higher Education [online]. 19 (1), 80-99. Johnson, L. F. and Levine, A. H. (2008). Virtual worlds: Inherently immersive, highly social learning spaces. Theory into Practice. 47 (2), 161-170. Kays, E. and Sims, R. (2006). Reinventing and reinvigorating instructional design: A theory for emergent learning. Proceedings of the 23rd Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education: Who’s Learning? Whose Technology? 256-265. Loke, S. K., Blyth, P. and Swan, J. (2012). Student views on how role-playing in a virtual hospital is distinctively relevant to medical education. In Brown, M. Hartnett, M. and Stewart, T. (Eds.), Future Challenges. Proceedings ascilite Wellington 2012. pp 565- 574. Martey, R. M. and Consalvo, M. (2011). Through the Looking-Glass Self: Group Identity and Avatar Design in Second Life. http://lamar.colostate.edu/~rmikealm/docs/costumes.pdf [viewed 15 July 2015]. Mennecke B. E., Triplett J. L., Hassall L. M., Conde Z. J. and Heer R. (2011). An examination of a theory of social presence in virtual worlds. Decision Sciences [online]. 42 (2), 413-450. Neustaedter, C. and Fedorovskaya, E. (2009). Presenting identity in a virtual world through avatar appearances. In Proceedings of Graphics Interface 2009 (GI '09). 183-190. Nygaard, C., Courtney, N. and Leigh, E. eds. (2012). Simulations, Games and Role Play in University Education. Oxfordshire: Libri Publishing. Peachy, A. and Childs, M. (2011). Reinventing Ourselves: Contemporary Concepts of Identity in Virtual Worlds. London: Springer Verlag. Wiecha, J., Heyden, R., Sternthal, E. and Merialdi, M. (2010). Learning in a virtual world: Experience with using Second Life for medical education. Journal of Medical Internet Research. 12 (1). Griffiths, A. (2015). Occupational Medicine Simulation Project. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:103-CP:107). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:107 459 Can learning analytics provide useful insights? An exploration on course level Eva Heinrich Massey University New Zealand This concise paper reports on an analysis of access logs of a first year university course that was delivered in a blended format. This analysis is an initial step in a wider project aimed at investigating if learning analytics can provided useful insights on course level, targeting both student learning and the needs of teachers. Preliminary findings show potential in noting when students need targeted help, a lack of correlation between access logs and grades, and insights into the degree by which course completion rates are affected by the lack of student engagement. Keywords: Learning analytics, first year courses, blended learning. Introduction In 2011 Long and Siemens wrote about the potential of learning analytics to “improve the quality and value of the learning experience” (p.40). Since then learning analytics research has been published in a special edition of the Journal of Educational Technology & Society and in a dedicated journal published by the Society for Learning Analytics Research. Valuable contributions have been made, yet, it seems that learning analytics is not ready to make widespread contributions in the sense of providing teachers with solutions, conceptually and technically, that are easy to apply, in terms of knowledge and time requirements, and provide meaningful insights. The 2014 review of learning analytics research by Papamitsiou and Economides confirms the potential but also alerts to a considerable number of gaps. While research articles highlight strength in approaches, it is fairly easy to spot the missing links. As there is no room for a comprehensive review in this concise paper only a few examples are given. Sophisticated statistical methods for preparing and analyzing log data are used by Andergassen et al. (2014), yet the results are not correlated with the wider learning context and no insights are provided on how to use the results to assist student learning. Promising results are reported for efficiently identifying students with poor evaluation skills (Gunnarsson et al., 2014), but a specialized blogging tool is required. The eLAT (exploratory Learning Analytics Toolkit) described by Dyckhoff et al. (2012) looks encouraging in allowing teachers to analyze teaching data presented in graphical form. Important considerations, such as not requiring specialist data mining knowledge, have been made, yet it remains to be seen if the relative simplicity of the indicators presented will be powerful enough. The research reported in this concise paper represents a first exploratory step in a wider research effort examining the potential of learning analytics in the context of first year university courses delivered in a blended format. The specific challenges of teaching first year courses are well documented (see, for example, Zepke, 2013). First year students are diverse in terms of readiness for study and subject knowledge. Engagement and retention are common problems. In this context the research presented here targets conventional applications of learning analytics around assisting teachers in helping students to learn better and in detecting problems early. Yet, the researchers are also interested in aspects less commonly emphasized. The current climate of higher education favors research outputs and makes even dedicated teachers question the wisdom of investing large amounts of time into preparing sophisticated teaching material (Ginns et al., 2010). Can learning analytics confirm the value of such material for student learning and provide re-assurance to teachers that their efforts are worthwhile? Governments (most certainly in New Zealand) set increasingly high completion rate targets. Institutions pass on the pressure to achieve these completion rates to teachers. These teachers work hard to provide learning opportunities to their students, but ultimately have no control over uptake. Learning analytics can help teachers document the level of engagement (or the lack of thereof) by students. In the longer run such data might be required to demonstrate the problem of widening the participation in higher education without higher resourcing but increasing completion rate expectations while supposedly not dropping standards. CP:108 460 This exploratory research step focused on a first year information technology course taught in a blended mode supported by the Learning Management System (LMS) Moodle and the multimedia- streaming platform Mediasite. The teacher who developed and taught the course is experienced in first year teaching, blended approaches and the use of technology for teaching. The course was delivered to one cohort of 59 students in the first semester of 2015 at a New Zealand university. The data analyzed were LMS records (access logs, marks for internal assessments), streaming records, exam marks and final grades. Data on student demographics were not accessed. Areas investigated Level of course participation by students In the blended course design access to the LMS was vital for full participation in the course. The LMS provided access to organizational information, topic introductions, lecture slides, links to screencasts and assessment specifications. Discussion forum contributions were required and two assignments had to be submitted via the LMS. Access records stored by the LMS can therefore be regarded as indicators for the degree of course participation. In total 31,226 access logs were recorded by the LMS (up to the hour of the final exam for the course). At the end of the course, the official classroll stood at 59 students. Of those, five students withdrew during the semester and six did not complete the assessment requirements (they did not sit the final exam), leaving 48 students who received a grade for the course. Records of the non-graded students, as well as records of additional five students who started the course but withdrew very early in the semester formed part of the LMS access logs. Looking at graded students only, the average number of access logs per student was 520, the minimum number 147 and the maximum number 1,321. Figure 1 shows the distribution of log entries across course areas: Access to course homepage; access to information about the course (e.g., teaching team, contact times, assessment details); access to assignment specifications, submission and feedback (as relevant as part of the internal assessment occurred face-to-face); access to forum discussions (reading could also occur via email digests); access to resources (text entered directly into the LMS, links to PDF lecture slides; links to streamed video recordings). The differences in how students interacted with the course site can be exemplified by looking at the proportion of access to the course home page to total access. Across all students this proportion was 32%, the minimum 11% and the maximum 48%. A lower proportion indicates that a student has done more work on the course site per session than a student with a higher proportion (as an entry for the home page would be triggered in most cases when starting a new session). Without additional data and much deeper investigation it is not possible to say how the different ways of interacting might be related to learning. Figure 1: Distribution of access logs The five students who started the course but withdrew within the first two weeks did so without financial or academic penalties and are of little concern to the teacher. Of the eleven students who remained on the classroll but did not complete all course requirements, three do not seem to have made any effort to study the course. These students had zero, eight and eleven log entries over a very short timespan. It is unlikely that the course design was a major factor in those students’ behaviour. The remaining eight students had on average 147 log entries each over up to the first eight weeks of the semester. Only one of those students attempted one of the internal assessment components. Monitoring of the access logs during the semester might have been able to pick up that these students were in danger of dropping out. CP:109 461 The course had six internal assessment components. Most of those were practical tasks to be shown to the teaching staff during lab times. Based on staff feedback students could rework their solutions and present those again for marking. This approach led to many constructive conversations between staff and students and to an opportunity to gain full marks by taking on the feedback and persisting with the work. In the end-of-course survey conducted by the university, which had an above average return rate of close to 50%, the questions on appropriateness of workload, assessment turn-around times and value of feedback given received the highest ratings of all questions (5.0, 5.2, and 5.1 out of 6), indicating that the assessment design was suitable. From a teaching perspective it was therefore disappointing to see that twelve students severely affected their final grades (and missed out on learning opportunities) by not attempting several assessment components. On average these students lost 18 marks out of 100 (assuming they would have achieved average marks for the work). Eight of those students failed the course, four passed with a ‘C’ grade. By doing all the course work to an average standard only two of those students would have failed and two would have lifted their grades to ‘A’. These numbers show the difficulties teachers face in achieving the completion rates expected. The twelve students who did not attempt substantial parts of the internal assessment amount to 20% of the classroll. The eleven students who according to LMS logs and assessment records did no to very little work for the course make up another 19%. Access to static material – lecture slides and readings Lecture slides and readings were made available via the LMS in PDF format. On average the graded students accessed 12.4 of the 17 sets of lecture slides available and 2.4 of the three readings. 48% of graded students accessed all 17 sets of lectures slides and 58% accessed all three readings. Access varied across the five topics of the course. All graded students accessed all lecture slides and the reading for Topic 3, a topic not directly linked to a practical course component. Some differences in total marks gained can be observed. The students who accessed all lecture slides had a mark average of 68, which was ten marks higher than the average across students who accessed less than half of the lecture slides. Students who accessed all three readings also had an average of 68, compared to 53 gained by students who accessed only one reading. Despite these differences, no statistically significant correlations could be observed. While the LMS access logs for lecture slides and readings might be indicators for engagement with the study material, they seem to carry limited meaning. Access to the material does not show the level to which students might have engaged. While no access to the material should be a strong indicator for a lack of engagement (assuming students did not get copies from friend without going through the LMS), this does not seem to carry any predictive value in terms of grades. In this course, like in others observed before, there were several students who received high grades, despite not accessing the material. This might be a sign of the diversity of students taking first year courses, where some students enter with a high level of subject knowledge and independent study skills. Access to dynamic material – screencasts Links to 15 screencasts were available for streaming from the university’s media server. The teacher used these screencast to demonstrate and explain how to use software to solve the practical course tasks. The content of the screencasts was directly relevant for the assessed work. The screencasts had an average length of 11 minutes 21 seconds and were accessed on average by 36 students. The students who accessed the screencasts watched 89% of the full lengths and repeated 1.7 times. Figure 2 shows the data on percentage watched and repeat factor for one of the screencasts used in CP:110 462 Week 3 of the course. Figure 2: Access to one of the screencasts used in Week 3 There were no significant correlations between the streaming of the screencasts and the marks for the directly related assessment tasks. A number of students did not access the recordings yet still completed the tasks to a high level. An explanation might again be related to student diversity. Log entries that show that some students have only played a few seconds of each recording. It would be interesting to investigate if this was intended behaviour or if there are technical reasons for those log entries. Monitoring of the access logs during the semester could provide opportunities to identify students who might need help. If a student watches one screencast many times more than other students, this might be an indication that the student struggles with the material. If a student has watched all screencasts for a topic but has not submitted the matching assessment work, the student might require extra assistance. For the teacher of the course the access statistics on the screencasts provided re-assurance that their effort in creating these recordings was worthwhile. In the past students had asked for more screencasts. The access logs confirmed that the students made use of the material. Knowledge, tool and time requirements For this research the analysis was carried out with a spreadsheet program using descriptive statistics. The work was time consuming and had to be approached with care. The data came from multiple sources (downloads from the LMS and from the media streaming system, spreadsheets with assessment details). Access logs for the streaming data had to be downloaded separately per recording. The LMS records contained data for anyone with access to the course at some stage and had to be separated into the various student cohorts (graded and others) and teachers and administrators. The LMS and media streaming logs contained student identifiers in different formats, which had to be converted. While none of these steps was difficult, the work was time consuming and error prone. Few teachers would be able to invest the time to analyse the data for a course after its completion. Even fewer teachers will be able to do such analysis on an on-going basis while teaching a course. Conclusions and recommendations Based on the analysis of this single course preliminary conclusions can be drawn. Looking at log data might provide two ways of assisting students. First, access logs indicate when student engagement with a course drops off. Noting this in a timely fashion might provide opportunities to engage with students directly, learn of issues and help. Second, over-engagement with material might indicate that a student is trying hard but is struggling with a topic. Again it might be possible to offer targeted help. Both of these scenarios will require some flexibility in course structures, for example, allowing a student to catch up with assessment components. Based on the data analyzed for this course it seems unlikely that a below average access to study material, such as lecture slides, readings, or recordings, is a reliable indicator for a lower study performance. Using analytics to encourage student access to material needs to be treated with care, as nothing will be gained by students just clicking on resource links to increase their rankings in access logs. In terms of assisting teachers the analytics gathered for this course showed promise. The degree to which students watched the screencast recordings was encouraging and confirmed to the teacher that the effort invested was worthwhile. While the data showed no statistically significant correlations between access logs and marks, they still allow a teacher to highlight when students fail to engage. As part of a more holistic assessment of a course, considering course design, student workloads and course evaluations, learning analytics can be a valuable part of a teacher’s argument should their completion rates be criticized. The next research steps will be to analyze several other first year blended courses, taking individual course characteristics into consideration. This will show if the preliminary findings can be confirmed. Should this investigation show how learning analytics can provide substantial benefits to students and teachers, further research steps will be to investigate an efficient toolset that makes it feasible to apply learning analytics. In this context bigger issues will have to be examined. Papamitsiou and Economides (2014) question if learning analytics should be the domain of individual teachers or of CP:111 463 specialists who have the required technical, statistical and pedagogical knowledge. Are there parallels to the areas of learning design and technology-enhanced learning, where research focused academics by large need specialist support? References Andergassen, M., Mödritscher, F., & Neumann, G. (2014). Practice and repetition during exam preparation in blended learning courses: correlations with learning results. Journal of Learning Analytics, 1 (1), 48-74. Dyckhoff, A.L., Zielke, D., Bültmann, M., Chatti, M.A., & Schroeder, U. (2012). Design and implementation of a learning analytics toolkit for teachers. Educational Technology and Society, 15 (3), 58-76. Ginns, P., Kitay, J., & Prosser, M. (2010). Transfer of academic staff learning in a research-intensive university. Teaching in Higher Education. 15 (3). 235- 246. http://dx.doi.org/10.1080/13562511003740783 Gunnarsson, B.L. & Alterman, R. (2014). Peer promotions as a method to promote content. Journal of Learning Analytics, 1 (2), 126-150. Long, P., & Siemens, G. (2011). Penetrating the fog: analytics in learning and education. EDUCAUSE Review, September/October 2011. Papamitsiou, Z. & Economides, A.E. (2014). Learning analytics and educational data mining in practice: a systematic literature review of empirical evidence. Educational Technology and Society, 17 (4), 49-64. Zepke, N. (2013). Student engagement: A complex business supporting the first year experience in tertiary education. The International Journal of the First Year in Higher Education, 4(2), 1- 14. http://dx.doi.org/10.5204/intjfyhe.v4i2.183 Heinrich, E. (2015). Can learning analytics provide useful insights? An exploration on course level. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:108-CP:112). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:112 464 A pedagogical end game for exams: a look 10 years into the future of high stakes assessment Mathew Hillier Belina Gibbons Monash University, Australia University of Tasmania, Australia This short paper looks ahead 10 years to a possible future for high stakes assessment in Australian higher education. The authors discuss some of the drivers pushing towards this future along with desirable operational features and pedagogical capabilities of an e- exam system for the year 2025. This paper represents a vision or road map to which a newly established, half million-dollar, Australian Government Office for Learning and Teaching national project on e-exams will be contributing over the next three years. Keywords: high-stakes assessment, forecasting, futures, e-exams Teaser of the Future The future of exams needs to be a fully open affair. This is a controversial statement in the context of the capabilities, mind set and technology available today. Imagine a future in which students are able to demonstrate their capabilities using a full range of twenty first century ‘tools of the trade’. They will be assessed in a manner that mimics, or is embedded in, the real world problem solving environments they will face in their work and social lives. Yet it provides for the strong integrity and assurance of contemporary high-stakes supervised examinations. Imagine that students will readily be able to gain access to the vast storehouse of information, tools and contacts available in the contemporary networked world. But this is easier said than done. Such highly authentic assessments (Crisp 2009, Herrington, Reeves & Oliver 2010) would provide students an opportunity to demonstrate their twenty first century abilities (Binkley et. al. 2012) in a very realistic, contextualised environment, but questions of fairness may well persist if appropriate frameworks are not established. Back to Today The world of today is awash with information sources, some high quality, the majority less so. Never the less, people facing problems in their daily life and work are able to draw upon a wide range of information and social resources. The technology to allow students to access these resources instantly is available to us today, but for educational authorities such access currently represents a significant threat to the integrity and validity of the assessment process, particularly at the high stakes end of the spectrum. An 'open borders' examination system would provide no comfort and assurance to educational testing authorities. The problem of bringing real world authenticity to the domain of high stakes testing currently seems insurmountable and we persist with locked down, limited, paper based or selected response examinations that get further away from the reality of practice of the 21st century every day. Forays into the world of technology enhanced high stakes assessment currently available largely replicate paper-based exams in a digital format; either 'armoured word processors' or glorified multiple choice quiz tools with limited pedagogical flexibility that only slightly expand the landscape of possible assessment activity. Call it "paper 1.1". Worse still, almost all solutions available today send institutions down a pedagogical and technological cul-de-sac with closed and 'black box' solutions that lock institutions out of the very data their own students produce. It is just this kind of data from which the most successful enterprises are increasingly mining insights to improve effectiveness, efficiency and spur innovation. Closed systems, while 'neat' and available today, are sacrificing opportunities for the future use of such data. In an increasingly data intensive future this could prove costly to institutions who choose ‘closed’ over that of open-standards based approaches. CP:113 465 Forward to the Future Instead, imagine a future in which authentic, complex and even 'wicked' problems can be set for students to address. A future where the 21st century 'tools of the trade' are provided to students to utilize as they see fit in constructing their responses. Imagine that all touch points a student makes in the information galaxy during the problem solving process can be known to the examiner. Where each student’s progress in solving a problem is captured in detail; where all sources, contacts and decision points are logged, mapped and presented in an easy to comprehend display, allowing examiners to see into the analytic mind of the student in order to assess their ability in solving real- world, wicked problems. The teacher will be presented with a graphical representation of the interactions in summary format, with drill down capabilities. The temporal dimension of student decision making and problem solving during the course of the assessment will be mapped and displayed with gains in marks or decision points made explicit. Varying degrees of automated or computer assisted marking will be available. Each problem set, question and fact presented to the student will be recorded, banked and tagged against a set of desired learning outcomes established by the institution. Program leaders and students will be able to see their progress towards personal and program learning goals as part of an overall educational analytics platform, of which e-exam performance data will be just one part. Program designers, teachers and managers will be able to conduct skills audits of assessments. With wider curriculum mapping and performance modelling in place, changes to assessments or curriculum will be reflected in the models of likely impact on learning outcomes and student performance. Students will be able to gain insight into their progress across a range of learning themes and 21st century skills. Assessments can be adaptive, modifying successive challenges as students progress. Teachers will gain insight into the performance of the very questions and problems set for students with the use of a range of statistical analysis approaches, all displayed in an easy to comprehend graphical display. Problematic questions will be highlighted for review and effective questions promoted to the top of the pile for sharing with other educators. The quality of questions developed will be reviewed prior to the exam via a secure online, integrated exam development, quality control and review process resulting in fewer errors and misunderstandings in the exam room. Computerised administrative processes for exam management will be made usable by non-technical administrators and teachers. A thoughtfully designed, open-technology based exam platform will enable scalability from the smallest classroom to the largest institution, so all in the community are able to benefit from advances in assessment techniques and technologies. Unknown futures will be catered for by storing data in 'open standards-based' secure store houses. This will ensure institutions will have future access to fine grained data that will facilitate the growth in learning about student performance, data and the evolution in analytics and presentation tools not yet known. As time goes on, deeper insights into student performance will be enabled by complete access to the response, action and click streams of each and every student's engagement with each and every question, fact and resource. Institutional policy may well still direct the release of marks, but the technology will greatly improve the timeliness of feedback from high stakes assessments. Feedback cycles will be streamlined such that computer marked items could provide instant feedback, while non-computer assessable items will be streamed to suitable or available markers. These markers will not need to diagnose increasingly messy student handwriting, and will be able to provide customised-on-the fly feedback based on a library of shared comments from colleagues. Moderation via linked systems will improve the consistency by which marking schemes are applied. Getting There The future outlined in this short paper is driven by a range of factors in the contemporary higher education world (Hillier & Fluck 2013). Added to this, the 'massification' of higher education has meant CP:114 466 that a bachelor degree is the new high school. In Australia 28% of the working age population have bachelor degrees or greater as of 2014 (ABS 2014). This is up from 21% in 2004. Similarly young adults aged 18-34 years are now much more engaged in higher education. For example in 1976 only 5% had a bachelor degree or greater, while in 2011 this had increased to 26% (ABS 2013). This vast increase in numbers has placed enormous pressure on higher education institutions to provide a quality education to larger class sizes at a time when per student funding is decreasing in real terms (Universities Australia, 2015, p. 7). The contemporary internet makes knowledge easily available. Higher education institutions have long since lost their monopoly on information. Instead universities need to play to their strategic strengths and expertise in critiquing knowledge. We guide students to acquire a critical mind, to analyse problems and assess the capabilities of students to perform against the criteria and standards suited for the world of the 21st century. Achieving this vision is indeed a truly 'wicked systems problem' (Rittel & Webber 1973, Ackoff 1999). A complex ‘wicked’ systems problem is an adequate metaphor for the ‘problem’ of e-exam implementation because there exists wide range of stakeholders and perspectives acting on the problem domain (Linstone 1999). These perspectives come from both within and external to institutions and include, university and government policy makers, institutional managers, students, teachers, finance, human resources, examinations officers, learning designers, parents, employers, technologists, campus facilitates, buildings and maintenance services. A national project (Transforming Exams 2015) has been convened in Australia to examine the problem in detail with plans to develop and pilot procedural and technical solutions in Universities. The project is located in most states and territories of Australia and will range from research intensive to teaching focused, from metropolitan to regional institutions over the period 2016 to 2018. The national project will continue work started at the University of Tasmania (Fluck, Pullen & Harper 2009) since 2007 and later at the University of Queensland via an OLT seed project in 2013-2014 (Hillier & Fluck 2014). The strategy developed to enable implementation of this 10 year plan is one of evolution rather than revolution. Procedural, policy and technological change of this magnitude would face overwhelming resistance unless all stakeholders are brought on the journey together. A transition strategy is planned of a gradual introduction and iterative development from paper-replacement to post-paper supervised exams to eventually arrive at an open borders approach to high stakes assessment. The process aims to address embedded cultured attitudes. 'Hearts and minds' must be won, professional development offered and technology infrastructure developed that meets a range of pedagogical, stability, efficiency and validity needs. The road map to this possible future is outlined in Table 1, starting with where we currently are, and through a gentle ramping up of technological, process and cultural change. Table 1: A possible road map to the future of high stakes assessment About now 2015-2020 2020-2025 2025 and beyond Medium for Paper Paper- Post-paper Fully high stakes replacement – exams computerised, assessments students can common. internet enabled opt to type All questions exams with instead of and materials candidates handwriting are digital, a using a range of (uses USB computer is software and drive to boot required to input devices. BYOD). respond to Some post- assessment paper exams challenges. appearing. CP:115 467 About now 2015-2020 2020-2025 2025 and beyond Connectivity None None to some use Mix of offline and Open internet of restricted ad- online exams access but all hoc networks for limited to selected transactions are response resources. fully logged reticulation in Connections inclusive of post-paper exams. logged. communication, timing, sources. Authenticity of Scenarios are Full colour High fidelity, data- Real-time links to assessment written diagrams and driven simulations global databases descriptions, with video begin to monochrome provide more illustrations authentic scenarios Candidate Manual Practice Practice Practices continue, identity comparison of continues, linked continues, but with two-factor assurance face with ID card to local database laptop camera authentication photo by a trusted via handheld takes pictures of incorporating supervisor device. the keyboard user biometrics such as at random face recognition. intervals. Materials A range of Digital equivalents e-books, high Practice continues provided/ published books, begin to replace resolution images, with increasing allowed electronic some materials. video, simulations, diversity of subject- calculators and E.g PDFs. all software tools specific software stationery are provided tools. equipment bought (open source). into the room by students Assessment Bundles of scripts Practice Digital responses, Digital response workflow are physically continues, but extends to data files are transported to digital response files created using accompanied by assessors scripts can be subject specific performance duplicated, software. E- metrics for archived and e- workflows, banked individual students, mailed. and tagged and interaction logs questions. Achievement On quality of Practice Practice Detailed analytics, measurement solution, and continues, continues, but keystrokes/screen written process analytics of analytics touches available – selected response increasingly the solution process items. detailed. E.g. time dominates taken per assessment. question, marks gain. Continuous Year-on-year bell- Some data on Individual Question ratings assessment curve overall ease or questions are take into account all improvement comparisons difficulty of rated for candidate process regulate overall individual discrimination and interactions within difficulty of exam. questions/ options reliability etc. the assessment. is available. It must be acknowledged that computerised and online exams already exist in the market but these are rarely used in higher education. They are generally limited in their pedagogical capabilities. What we are prosing is an e-exam 'platform', not just an application or web service. We argue that to st provide the full set of 21 century tools a 'whole computer environment' needs to be made available to each student, initially with no or restricted connectivity, but with a view to fully open, internet connected exams with comprehensive logging and auditing. CP:116 468 However, this vision is not pie-in-the-sky. The Tasmania Qualifications Authority has already moved beyond paper-replacement and post-paper examinations to utilise open, Internet connected examinations leading to Australian tertiary assessment ranks for Year 11/12 candidates (TQA, 2013). Marking was conducted electronically via iPad. Internationally, Finland is implementing a national 'Digabi' project to make all matriculation examinations digital by 2019 (Von Zansen, 2014). Conclusion Australia stands on a cusp for high stakes assessment. Institutions cannot afford to invest in fleets of computers reserved solely for semi-annual testing. Bring your own device (BYOD) solutions are therefore critical to this cultural transformation, bringing challenges of validity and reliability. But the prize is worth striving for, making academic awards more credible, and carrying the opportunity to reform curricula with powerful software tools. Further information on the national 'Transforming Exams' (2015) project is available. References ABS (2013) Young Adults: Now and Then, 4102.0 - Australian Social Trends, April 2013, Australian Bureau of Statistics http://www.abs.gov.au/AUSSTATS/

/Lookup/4102.0Main+Features40April+2013 ABS (2014) Non-school qualification at Bachelor Degree level or above, persons aged 20-64 years, 6227.0 - Education and Work, Australia, May 2014, Australian Bureau of Statistics http://www.abs.gov.au/AUSSTATS/

/DetailsPage/6227.0May%202014 Ackoff, R. (1999) Re-Creating the Corporation: a design of organizations for the 21st century. Oxford Univ. Press, New York Binkley, M., Erstad, O., Hermna, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining Twenty-First Century Skills. In Griffin, P., Care, E., & McGaw, B. Assessment and Teaching of 21st Century Skills, Dordrecht, Springer. http://www.atc21s.org/ Crisp, G. (2009). Towards Authentic e-Assessment Tasks. In G. Siemens & C. Fulford (Eds.), Proceedings of EdMedia: World Conference on Educational Media and Technology 2009 (pp. 1585-1590). Association for the Advancement of Computing in Education (AACE). http://www.editlib.org/p/31689/ Fluck, A., Pullen, D., & Harper, C. (2009). Case Study of a Computer Based Examination System. Australasian Journal of Educational Technology, 25(4), 509– 523. http://ajet.org.au/index.php/AJET/article/view/1126 Herrington, J., Reeves, T. C. & Oliver, R. (2010). A guide to authentic e-learning. New York: Routledge Hillier, M & Fluck, A. (2014) Transforming Exams: Processes and Platform for e-Exams in Supervised BYOD Environments, Australian Government Office for Learning and Teaching. http://www.olt.gov.au/resource-transforming-exams-byod-environments Hillier, M., & Fluck, A. (2013). Arguing again for e-exams in high stakes examinations. In H. Carter, M. Gosper, & J. Hedberg (Eds.), Electric Dreams (pp. 385–396). Macquarie University. Retrieved from http://www.ascilite.org.au/conferences/sydney13/program/papers/Hillier.pdf Linstone, H.A. (1999) Decision making for technology executives, Artech House, Boston Rittel, H. & Webber, M. (1973) Dilemmas in a General Theory of Planning, Policy Sciences, 4, pp. 155-169. Tasmanian Qualifications Authority (2013) Board Meetings. http://www.tqa.tas.gov.au/10270 & http://www.tqa.tas.gov.au/23727 Universities Australia (2015) Universities Australia Pre-Budget Submission 2015–16. Canberra, ACT. Von Zansen, A. (2015) Digabi Newsletter II/2015. http://translate.google.com.au/translate?hl=en&sl=sv&u=https://digabi.fi/%3Flang%3Dsv_s e&prev=search Transforming Exams (2015). 'e-Exam System' project, http://transformingexams.com Hillier, M. & Fluck, A. (2015). A pedagogical end game for exams: a look 10 years into the future of high stakes assessment. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:113-CP:118). CP:117 469 Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:118 470 Are Higher Education Institutions Prepared for Learning Analytics? Dirk Ifenthaler Curtin University Learning analytics may provide multiple benefits for higher education institutions and for involved stakeholders by using different data analytics strategies to produce summative, real-time and predictive insights and recommendations. However, are institutions and academic as well as administrative staff prepared for learning analytics? Considering a learning analytics benefits matrix, this study investigates the current capabilities for learning analytics at higher education institutions, explores the importance of data sources for a valid learning analytics framework, and builds an understanding on how important insights from learning analytics are perceived. Findings revealed a lack of staff and technology being available for learning analytics projects. It is concluded that more empirical research focussing on the validity of learning analytics frameworks and on expected benefits for learning and instruction is required to confirm the high hopes this promising emerging technology is suggesting. Keywords: learning analytics, benefits matrix, higher education, readiness Introduction The NMC Horizon Report: 2014 Higher Education Edition (Johnson, Adams Becker, Estrada, & Freeman, 2014) identified learning analytics as a mid-range trend driving changes in higher education within the next three to five years. Learning analytics (LA) uses dynamic information about learners and learning environments – assessing, eliciting and analysing them – for real-time modelling, prediction and optimization of learning processes, learning environments, and educational decision- making (Ifenthaler, 2015). Promising LA applications are being developed which use learner generated data and other relevant information in order to personalise and continuously adapt the learning environment (Long & Siemens, 2011). LA is expected to provide the pedagogical and technological background for producing real-time interventions at all times during the learning process. Students will benefit from LA through optimised learning pathways, personalised interventions and real-time scaffolds. LA will provide instructors detailed analysis and monitoring on the individual student level, allowing to identify particularly instable factors, like motivation or attention losses, before they occur. Instructional designers use LA information to evaluate learning materials, adjust difficulty levels and measure the impact of interventions (Lockyer, Heathcote, & Dawson, 2013). LA will further facilitate decision-making on institution level and help to analyse churn and identify gaps in curricular planning (Ifenthaler & Widanapathirana, 2014). However, are institutions and academic as well as administrative staff prepared for LA? The vast amount of available educational data requires flexible data mining tools and new statistical methods (Ifenthaler & Widanapathirana, 2014). Further, institutions need to develop and implement interactive data visualisations which provide students, instructors, instructional designers and administrators an overview of relevant information (Greller & Drachsler, 2012). Therefore, the purpose of this research is to explore the current state of LA in higher education and to help to identify challenges and barriers for applying LA. Benefits from learning analytics From a holistic point of view, LA may provide multiple benefits for higher education institutions and for involved stakeholders. Additionally, different data analytics strategies can be applied to produce summative, real-time and predictive insights. Table 1 provides a matrix outlining the benefits of LA for stakeholders using three analytics perspectives (Ifenthaler & Widanapathirana, 2014). However, it is not required to implement all features of the presented LA benefits matrix. Institutions need to carefully decide which features a LA frameworks shall include and provide the necessary infrastructure for a successful implementation. CP:119 471 Purpose of the study and research questions The implementation of a LA framework following the matrix of LA benefits (see Table 1) requires specialised staff and technological capabilities (d’Aquin, Dietze, Herder, Drachsler, & Taibi, 2014). Given the emerging field of LA, staff as well as technological solutions are scarce. Therefore, the purpose of this study was threefold: 1) to investigate the current capabilities for LA at higher education institutions, 2) to explore the importance of various data sources for a valid learning analytics framework, and 3) to build an understanding on how important insights from LA using a summative, real-time and predictive perspective are perceived. Table 1: Learning analytics benefits matrix Perspective Stakeholder Summative Real-time Predictive Governance • Apply cross- • Increase • Model impact of institutional productivity organizational comparisons • Apply rapid decision-making • Develop response to critical • Plan for change benchmarks incidents management • Inform policy • Analyze making performance • Inform quality assurance processes Institution • Analyze processes • Monitor processes • Forecast processes • Optimize resource • Evaluate resources • Project attrition allocation • Track enrollments • Model retention • Meet institutional • Analyze churn rates standards • Identify gaps • Compare units across programs and faculties Learning design • Analyze • Compare learning • Identify learning pedagogical models designs preferences • Measure impact of • Evaluate learning • Plan for future interventions materials interventions • Increase quality of • Adjust difficulty • Model difficulty curriculum levels levels • Provide resources • Model pathways required by learners Facilitator • Compare learners, • Monitor learning • Identify learners at cohorts and courses progression risk • Analyze teaching • Create meaningful • Forecast learning practices interventions progression • Increase quality of • Increase interaction • Plan interventions teaching • Modify content to • Model success rates meet cohorts’ needs Student • Understand learning • Receive automated • Optimize learning habits interventions and paths • Compare learning scaffolds • Adapt to paths • Take assessments recommendations • Analyze learning including just-in-time • Increase outcomes feedback engagement • Track progress • Increase success towards goals rates CP:120 472 Method Design In order to reach a large number of international higher education institutions, the principle means of data collection was an online survey which was conducted between August and October 2013. The survey was implemented on the Qualtrics platform (www.qualtrics.com). International listservs, forums, and social media channels focussing on educational technology and learning analytics were used to disseminate the link to the online survey. Participants The initial dataset consisted of 176 responses. After removing incomplete responses, the final dataset included 153 valid responses (21% female, 78% male, 1% indeterminate/intersex/unspecified). The average age of the participants was 44.68 years (SD = 9.10). 30% worked in a research position, 28% were research and teaching staff, 7% reported to be in a teaching position, 4% were in a senior management role, 1% reported to work in IT services, and 1% worked as library staff. 29% worked in other roles such as data analyst, statistician, or instructional designer. The majority of participants were located in the United States (28%) and Australia (19%). Other countries included United Kingdom (5%), Canada (5%), and the Netherlands (4%). 31% of the participants reported that they were currently involved in a project focussing on LA. Instrument The survey instrument consisted of the following sections: 1. Staff capabilities for learning analytics (7 items, Cronbach’s α = .89), 2. Available technology for learning analytics (13 items, Cronbach’s α = .98), 3. Barriers for implementing learning analytics (13 items, Cronbach’s α = .93), 4. Importance of student data (9 items, Cronbach’s α = .81), 5. Importance of learning environment data (13 items, Cronbach’s α = .85), 6. Benefits from learning analytics for the institution (20 items, Cronbach’s α = .93), 7. Importance of summative learning analytics (17 items, Cronbach’s α = .93), 8. Importance of real-time learning analytics (17 items, Cronbach’s α = .94), 9. Importance of predictive learning analytics (18 items, Cronbach’s α = .94), 10. Personal background (6 items). Most items were answered on a five-point Likert scale (e.g., 5 = very important; 4 = important; 3 = undecided; 2 = not very important; 1= not at all important). It took approximately 15 minutes to complete the survey. Data analysis All data stored on the Qualtrics platform was anonymised, exported, and analysed using SPSS V.22. Initial data checks showed that the distributions of ratings and scores satisfied the assumptions underlying the analysis procedures. Results Capabilities for learning analytics When asked about staff capabilities available for LA projects, over half of the participants reported that their institution had at least one learning management specialist (62.7%) and at least one learning designer (68.6%). Other staff capabilities available for LA projects included database analyst (41.2%), statistician (38.5%), and information management architect (22.9%). Only 25% of the participants reported that they had staff in the role of a learning analytics specialist. When asked about available technology for LA, only a small number of participants reported that their institution had a data warehouse in place (19.0%), used data visualisation capabilities (19.0%), and practised automated data reporting (21.6%) as well as predictive analytics (28.1%). One out of four participants indicated that their institution had interactive dashboards available for students and facilitators (25.5%). Interestingly, several institutions already utilised natural language processing (26.8), automated discussion board analytics (26.1), automated essay scoring (27.5%), and social network analysis (24.2%). CP:121 473 Importance of data sources In order to implement a valid LA framework, participants reported that it is important to have data sources from students available: socio-demographic data (94.1%), educational background (97.4%), learning history (85.0%), personal interest (92,8%), prior knowledge (95.5%), preferred learning strategies (79.7%), and computer literacy (90.2%). Less important data sources included social media preferences (18.3%) and social ties (18.9%). A valid LA framework also requires data sources from the learning environment (e.g., learning management system). Participants rated the importance of data sources as follows: use of learning materials (99.3%), discussion activity (92.1%), content navigation (92.2%), assessment results (98.7%), learning time (94.8%), use of external materials (89.6%), expected learning outcomes (98.1%), course difficulty level (94.7%), evaluation results (90.9%), expected learning paths (93.5%), and interaction of facilitators (96.1%). The location of learning was not regarded as being highly important (18.3%). Perceptions of learning analytics insights The three most important summative insights from LA reported by participants of the study were tracking student’s progress towards goals (99.3%), understanding of student’s learning habits (98.0%), and analyse student’s learning outcomes (98.7%). The most important real-time insights from LA included modifying content to meet students’ needs (96.7%), providing students with assessment including real-time feedback (98.0%), and creating meaningful interventions for students (98.0%). Participants rated the following insights from predictive LA being most important: increasing student’s engagement (98.0%), increasing student’s success rate (98.7%), and modelling student’s success rate (98.0%). Overall, participants reported that facilitators (96.0%) would benefit most from LA at their institution followed by students (95.4%) and learning designers (95.1%). The least benefits were expected for finance (15.0%) and facilities services (9.2%). Discussion and conclusions LA draws on an eclectic set of methodologies and data to provide summative, real-time, and predictive insights for improving learning, teaching, organisational efficiency and decision making (Lockyer et al., 2013; Long & Siemens, 2011). While the field of LA is receiving much attention for its capacity to provide lead indicators of student failure, it has to date focused on individual courses in isolation of the capabilities of higher education institutions. The findings of this work-in-progress study revealed a lack of staff being available for learning analytics projects. Specialised staff with a strong background in learning and teaching as well as data science are scarce. Similar, the findings clearly indicate that higher education institutions do not have the necessary technology available to implement valid LA frameworks. Accordingly, the high staff and technology requirements for LA frameworks can only be met by a small number of higher education institutions (Greller & Drachsler, 2012). Findings about the importance of data sources being relevant for a valid LA framework indicated that most of information from students and learning environments are perceived as equally important. Hence a current challenge for establishing LA frameworks is the interpretation of analysis results against the educational setting and its contextual idiosyncrasies (Coates, 2010). In other words, variables and indicators can carry different meanings and can therefore have different implications. This work-in-progress study has its obvious limitations which need to addressed. The nature of self- report data and the small sample size from a LA-aware group need to be considered when interpreting the results. Accordingly, future research shall provide further empirical evidence regarding the capabilities of higher education institutions for implementing LA frameworks. More importantly, the effectiveness of LA frameworks for improving learning and teaching needs to be addressed by rigorous empirical research. Last, questions about ownership of data and data security need to be critically reflected on national and international scale (Pardo & Siemens, 2014). To conclude, more educational data does not always make better educational data (Greller & Drachsler, 2012). LA has its obvious limitations and data collected from personal and educational sources (can) have multiple meanings. More importantly, empirical research focussing on the validity CP:122 474 of LA frameworks and on expected benefits for learning and instruction is required to confirm the high hopes this promising emerging technology is suggesting. References Coates, H. (2010). Defining and monitoring standards in Australian higher education. Higher Education Management and Policy, 22(1), 41-58. d’Aquin, M., Dietze, S., Herder, E., Drachsler, H., & Taibi, D. (2014). Using linked data in learning analytics. eLearning Papers, 36, 1-9. Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for learning analytics. Educational Technology & Society, 15(3), 42-57. Ifenthaler, D. (2015). Learning analytics. In J. M. Spector (Ed.), Encyclopedia of educational technology (Vol. 2, pp. 447–451). Thousand Oaks, CA: Sage. Ifenthaler, D., & Widanapathirana, C. (2014). Development and validation of a learning analytics framework: Two case studies using support vector machines. Technology, Knowledge and Learning, 19(1-2), 221-240. doi: 10.1007/s10758-014-9226-4 Johnson, L., Adams Becker, S., Estrada, V., & Freeman, A. (2014). NMC Horizon Report: 2014 Higher Education Edition. Austin, TX: The New Media Consortium. Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57(10), 1439–1459. doi: 10.1177/0002764213479367 Long, P. D., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 31-40. Pardo, A., & Siemens, G. (2014). Ethical and privacy principles for learning analytics. British Journal of Educational Technology. doi: 10.1111/bjet.12152 Ifenthaler, D. (2015). Are Higher Education Institutions Prepared for Learning Analytics? In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:119-CP:123). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:123 475 A blended learning ecosystem: What are the motivational issues for students? Maggie Hartnett, Alison Kearney and Mandia Mentis Massey University As technologies evolve, the places and spaces for learning are rapidly changing and learners are required to take increasing responsibility for directing their own learning. By doing so, students are presented with a range of opportunities and challenges within these complex learning environments. Research suggests that an important consideration is the effect on learner motivation. This paper reports on motivational issues for students working within an online post-graduate professional teacher education programme that blends lecturer-directed and student-directed learning. In 2014, students completed a survey about their experiences of setting their own learning goals and negotiating their own curriculum with an emphasis on motivation. This was followed by a series of interviews aimed at exploring these experiences in more depth. Preliminary findings highlight anxiety about choosing course content and setting learning goals were among key concerns identified by students. Results provide insight into motivational considerations for learners in complex learning eco-systems. Keywords: blended learning; self-directed learning; motivation; inquiry learning and inter-professional learning, learning ecosystems Background The ubiquity of digital technologies is changing the way individuals interact with each other and the world around them. Connecting, collaborating, and learning with and from each other are now possible in ways that are no longer constrained by time and space (Bates, 2005). This flexibility has a number of potential benefits, not least of which is that it provides learners with the power to choose when, where and how to learn (Harasim, 2012). But with flexibility also comes responsibility as students are increasingly expected to take ownership and direct their own learning. However ownership of learning, particularly in complex, digitally-rich learning environments, does not necessarily come naturally to all learners (Leach, 2000). This, in turn, can influence perceptions about learning experiences, learner beliefs and motivation (Hartnett, 2015). Perceptions of anxiety, confidence and/or preparedness are some indicators that can provide insight into the underlying motivation of learners (Brophy, 2010). This paper reports on the issues for students working within an online post-graduate professional teacher education programme that blends lecturer-directed learning with student-directed learning as they relate to motivation. e-Learning ecosystems The Encyclopaedia Britannica defines an ecosystem as a “complex of living organisms, their physical environment, and all their interrelationships in a particular unit of space” (2015, para 1). It has been argued that modern learning environments are complex systems that have many of the characteristics of living ecosystems (Gütl & Chang, 2008). Cowley et al. (2002) outline a range of components of an e-learning ecosystem. These include: the people – learners, teachers and support staff; the content to be learned; the structure of the learning; the environment in which the learning takes place; the technologies used that support learning; the skills required by learners (e.g., motivation, self-direction, subject matter, study and technical skills) and the support available to learners. Research context and participants The context for this study is a Post-graduate Diploma in Specialist Teaching (an innovative two year degree delivered jointly by two universities within New Zealand). The programme is designed for professional teachers working within special and inclusive education areas within the K-12 sector. Within the programme there are seven specialities. These are: Autism Spectrum Disorder, Blind and Low Vision, Complex Educational Needs, Deaf and Hearing Impaired, Early Intervention, Gifted and CP:124 476 Talented, and Learning and Behaviour. The programme is underpinned by a blended learning framework which integrates the three ecosystems of technology, pedagogy and context (Mentis, 2008). Blended learning is generally considered to be a coherent design approach that integrates the strengths of face-to-face teaching with online learning to provide more engaging learning experiences for students (Garrison & Vaughan, 2008). However, in the post-graduate specialist teaching programme, blended learning is conceptualised more broadly, and is used to describe more than the integration of traditional modes of teaching with new forms of technology. Blended learning in this model includes blending: specialised courses with more inter-professional generic content; lecturer- directed learning with more self-directed learning; case-work scenarios with authentic real-life practicum experiences; synchronous with asynchronous learning; formal and informal learning opportunities; structured approaches with more open pedagogies; formal and non-formal learning; as well as individual and co-operative learning. The Specialist Teaching programme also blends week long face-to-face courses (one at the beginning and one mid-year) with online modes throughout the year. From this description it is clear, the authors argue, that this programme fits the characterisation of an e-learning ecosystem. Using this blended learning approach allows for core courses to be taught inter-professionally across all specialities; a sharing of teaching time, expertise and resources and a robust inter-disciplinary approach to teaching and learning. This paper focuses on one aspect of the e-learning ecosystem, specifically how an inquiry-based learning approach (Spronken-Smith et al., 2011) impacts on the motivation of students. In this programme, lecturer-directed structured and guided inquiry is blended with more student-directed open inquiry where students set their own learning goals for their final portfolio assignment based on the stated learning competencies of the course, and make decisions regarding which aspects of the learning material or course content they will engage with. As part of the establishment of the programme, and to ensure its ongoing effectiveness, an annual survey is administered at the end of the academic year asking students to provide feedback about the programme design and its delivery. Survey data from the first two years (2011-2012), across all specialties, indicated that some students were having difficulties setting their own learning goals and making decisions regarding learning content to engage with (i.e. their learning path). As a result of these findings, further research was initiated to explore and understand what was occurring for students and how this related to their motivation in a comprehensive programme that requires them to direct aspects of their own learning. The over-arching research question that guided the research reported here is: “What are the issues for student motivation in a system where students are required to set their own learning path (goals and content)?” Participants were recruited by means of an invitation that was placed in the annual end-of-year online student survey (2013) that could be accessed by all students in the programme. Sixty-three students (out of 155 responses) volunteered to participate in the follow-up research. Of these, 21 students were selected to be interviewed. Criteria for selection were 1) specialist area and 2) responses to the end of year survey questions. This was to ensure that within the interview group as a whole there were participants representing each of the specialities in addition to a cross-section of those who agreed that a) most aspects of setting their own learning path (goals and content) were easy and enjoyable; b) some aspects were easy and/or enjoyable; c) neither agreed nor disagreed that setting their own learning path were easy and enjoyable; d) some aspects were difficult and/or unenjoyable; and e) most aspects were difficult and unenjoyable. All 21 participants were working in their professional roles as specialist teachers while enrolled in the programme. Data were gathered in 2014 via a short pre-interview survey, designed to gain some initial understanding of students’ experiences of setting their own learning goals and choosing learning content from the course to meet these learning goals. The survey comprised ten likert-type questions with responses ranging from 1 = strongly disagree to 5 = strongly agree. Participants completed the survey just prior to an individual face-to-face semi-structured interview, which was conducted either in person, or via Skype. At the time of the current research, twelve participants were in their final year of the two-year programme, and nine had completed the programme the previous year (i.e. end of 2013). The focus at this early stage of analysis is on potential motivational issues that emerged from the data rather than a comprehensive discussion of the findings to-date. CP:125 477 Preliminary findings Results from the pre-interview surveys are presented in Figure 1. Notably, from a motivational viewpoint, a sizeable percentage of participants reported feeling anxious about choosing content from the course (57% agreed or strongly agreed) and setting their own learning goals (43% agreed or strongly agreed). This compares with responses to questions about how prepared participants felt to choose content from the course (48% agreed); how prepared they felt to set their own learning goals (57% agreed); how confident they felt to choose content from the course (43% agreed or strongly agreed); and how confident they felt setting their own learning goals (57% agreed or strongly agreed). In other words, participants as a whole felt more confident and prepared to set their own learning goals than choosing their content from the course. Furthermore, the majority of participants reported feeling anxious about choosing content from the course. Preliminary analysis of the follow-up interviews reflected similar findings to the survey results, in terms of anxiety when choosing course content and setting learning goals, as the following comments indicate: Coming away from that first block course I understood what they were saying but actually thought was I capable of doing [it]? That was quite different. As I said I have done goals but to actually get them right and I was very concerned or worried or anxious that I was completely off track or that they were too broad or too simple, surely they can’t be that simple. So in that respect the more I prepared and the more I knew what to do of getting a good briefing, I suppose my belief in my own ability to get it right, there was doubt there. (Student 17) Because it was really hard to know what they wanted that was the biggest struggle well yes there is a competency, but really what do you want us to learn? Is it going to be enough, is it gutsy enough but still tight enough? I found that a real struggle. (Student 10) I think they had started saying to us at that point it [course content] is a smorgasbord you just pick what you want from it. But it is very difficult when you look at all that stuff and you think, as I said to you before here is stuff you would like to look at but you don’t have time. So actually choosing what you want to engage with can be quite daunting. (Student 15) CP:126 478 I felt prepared to set my own LGs 5% 14% 24% 57% I felt anxious choosing course content 5% 24% 14% 52% 5% I felt prepared to choose course content 24% 29% 48% I felt anxious about setting my own LGs 5% 33% 19% 29% 14% I felt confident about setting my own LGs 14% 24% 19% 38% 5% I felt confident choosing course content 14% 24% 19% 38% 5% 0% 20% 40% 60% 80% 100% strongly disagree disagree neither agree strongly agree Figure 1: Pre-interview survey responses The interview responses also revealed that students who reported that they felt prepared and confident to set their own learning goals had prior experience of doing so in other aspects of their lives. In a number of cases they also saw it as a purposeful activity because they were able to link their learning goals to their work: I think I am quite lucky is that I have been teaching a thing at school for five years and a lot of that has been making action plans. So it wasn't anything new to me to make an action plan and actually work through the steps of the action plan because that's what I had been teaching to the kids. So I think that, perhaps, prepared me a lot more than say someone who hadn't had that experience. I have also done audits in my job, Ministry of Ed[ucation] audits where I have had to actually deconstruct to reconstruct as part of an action plan. I think all of that really helps me. (Student 3) I like to be able to set my own learning goals so therefore it was relevant to me, that was really good and then I felt that what I was doing would benefit my job, myself and my colleagues as well because I could share stuff with them. (Student 5) Students who felt prepared and confident to choose their own content talked about the relevance of what they had chosen to their work as well as the need to be selective: For me in the end it just came down to selecting what was useful or what I thought might be useful to me at the time really. (Student 15) Yeah pretty confident I already knew that I needed to know specific things about my own practice, I needed to have a lot of the vision knowledge for example, a lot of the core stuff. I have been teaching for over thirty years so a lot of it was not rocket science you know, so I was quite selective in that thinking what really do I need to know to extend my practice. (Student 5) CP:127 479 When asked about the kind of learner they were, participants described themselves as “diligent and I’ve set myself high standards” (Student 1), “very enthusiastic, I absolutely love learning” (Student 7), “disciplined as a learner” (Student 15) and “I actually really enjoy the online study” (Student 5). Coupled with this, the average time since previous tertiary study was 12.6 years (SD = 9.2). Given the high expectations participants set for themselves together with the time elapsed since previous study, it is understandable why the many participants reported feeling anxious about choosing content and a considerable number felt anxious about setting their own learning goals (though to a lesser extent). Conclusion At this early stage these are only preliminary findings and further analysis of the data is necessary in order to unpack the complexity of learners’ experiences. What can be said at this point is that even though these students perceived themselves as able, diligent learners based on past learning success, the lack of familiarity and complexity of the blended learning environment in which they were situated caused them to question these judgements to some degree. Feelings of anxiety reported by participants were more salient when it came to choosing course content than the anxiety associated with setting their own learning goals. This has implications for the people, content and structure that makes up the rich, complex e-learning ecosystems such as the one described here as previous research has demonstrated that anxiety can undermine motivation to learn (Brophy, 2010). Primary among these is the need to offer differentiated guidance, particularly when it comes to choosing content to engage with, as even learners who perceive themselves as prepared and confident can feel anxious when learning in unfamiliar, complex environments which offer multiple learning options and pathways. Not to do so may detrimentally affect the motivation of learners. References Bates, A. W. (2005). Technology, e-learning and distance education (2nd ed.). New York: RoutledgeFalmer. Brophy, J. (2010). Motivating students to learn (3rd ed.). New York, NY: Routledge. Cowley, J., Chanley, S., Downes, S., Holstrom, L., Ressel, D., Siemens, G., et al. (2002). Preparing students for elearning. from Elearnspace website http://www.elearnspace.org/Articles/Preparingstudents.htm Ecosystem. (2015). In Encyclopaedia Britannica. Retrieved from http://www.britannica.com/EBchecked/topic/178597/ecosystem Garrison, D. R., & Vaughan, N. (2008). Blended learning in higher education: Framework, principles, and guidelines. San Francisco, CA: Jossey-Bass. Gütl, C., & Chang, V. (2008). Ecosystem-based theoretical models for learning in environments of the 21st century. International Journal of Emerging Technologies in Learning, 3(S3), 50-60. Harasim, L. (2012). Learning theory and online technologies. New York, NY: Routledge. Hartnett, M. (2015). Influences that undermine learners’ perceptions of autonomy, competence and relatedness in an online context. Australasian Journal of Educational Technology, 31(1), 86-99. Retrieved from http://ascilite.org.au/ajet/submission/index.php/AJET/issue/view/109 Leach, L. J. (2000). Self-directed Learning: Theory and Practice. Unpublished doctoral thesis, University of Technology, Sydney, Australia. Mentis, M. (2008). Navigating the e-learning terrain: Aligning technology, pedagogy and context. Electronic Journal of Elearning (EJEL), 6(3), 217- 226. Retreived from www.ejel.org. Spronken-Smith, R. A., Walker, R., Dickinson, K. J. M., Closs, G. P., Lord, J. M., & Harland, T. (2010). Redesigning a curriculum for inquiry: an ecology case study. Instructional Science, 39(5), 721– 735. http://doi.org/10.1007/s11251-010-9150-5 Hartnett, M, Kearney, A. & Mentis, M. (2015). A blended learning ecosystem: What are the motivational issues for students? In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:124-CP:128). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:128 480 Measuring creativity in collaborative design projects in pre-service teacher education Shannon Kennedy-Clark Sean Kearney Katrina Eddles-Hirsch University of Notre Dame, Australia University of Notre Dame, Australia University of Notre Dame, Australia Rod De La Hoz Vilma Galstaun Penny Wheeler University of Notre Dame, Australia University of Sydney Australian Catholic University Pre-service teacher education in the use of information and communication technologies (ICTs) has been the focus of numerous studies. In this paper, we further extend this body of research by examining the functions of creativity and how creative outputs are measured in pre-service teacher education, chiefly by discussing how students are assessed in terms of their creativities in design projects. The research aimed to evaluate the measures that had been put in place to ensure that the creative value of the student tasks was assessed objectively. Several strategies were used including a process-based task design, opportunities for students to revisit and refine designs, collaborative brainstorming, self-assessment, rubrics, panel marking by experts, and a design space that supported creativity. It was found that while interpretations of creativity were subjective, the students’ aim to develop creative outputs was fostered by the peer review and self-review processes adopted for the study. Keywords: creativity, assessment, collaboration, design learning, pre-service teacher education Introduction In pre-service teacher education in the use of Information and Communication Technologies (ICTs) there is a need to not only provide skills in evaluation and using technologies, but also to promote innovation and creative use of a range of tools that can enhance teaching and learning. Once pre- service teachers move into the classroom, they need to be able to work creatively within this environment. Creativity in this sense, is a functional skill in that the students need to develop a range of ICT enhanced learning experiences, whether it be a web quest, a website, or an app, for example, where the product itself needs to function. Designing and assessing creativity, however, is complex. Miller’s observation (1986) that a creative product is “something easy to recognise but hard to explain” captures the intangible, rather nebulous position of creativity. Hence, while we, as teachers, may easily recognise a design or artefact as creative, that does not mean that everyone will arrive at the same judgment about the level of creativity involved in the output, no matter what rubric or criteria are adopted. Creativity is also culturally bound, in that different people coming from different cultures, religions or races may show different criteria or judgment as to what they value as creative products. In the context of our study, we are assessing the creative design of artefacts that can be used for teaching and learning within the Australian school environment, itself a strongly diverse cultural setting. Drawing upon Cram et al. (2014), the students who participated in this study were provided with a solid grounding in the requisite design and technical skills before being given space to apply their own creative judgment in the design and development of a product. Functional creativity underpinned much of the design work in that the nature of the design space enabled and supported functional creativity. Functional creativity is a desireable quality for many professions, such as teaching, architecture and filmmaking, wherein creativity is valued highly. The affordances of technology for creative processes are equally attractive as students can work in three dimensions: they can revise a design in a low-stakes space and they can use innovative approaches to the design, from a dynamic viewpoint. Considering these possibilities led the research team to the question of how to assess creativity. More specifically, how does a teacher assess an artefact or design objectively as being more than mundane? This raised a subsequent question: How can we, as pre- service teacher educators, assesses the creative outputs of our students? The study was conducted in a pre-service teacher degree program at an Australian university. It is part of an ongoing research project on curriculum redesign in the use of ICT in education. The current paper focusses on the objectivity of the assessment and presents the preliminary results. CP:129 481 Creativity and assessment in higher education Creativity is gathering attention across the Australian higher education landscape as a graduate and as a desirable workplace attribute. Florida (2002) predicted that as much as one-third of the future workforce will be defined as being a ‘creative’ because of the nature of their roles. Creativity, in this respect, is favourable as it is linked with imaginative and innovative responses to future-oriented challenges and research, in a “workforce of generalists” (Altbach, Reisberg & Rumbley, 2009, p. 115). Given the responsive nature of higher education to global trends, the question of how to incorporate creative learning outcomes into programs has implications for course structure, task design and pedagogy. As with the concept of design, creativity can be understood to have a range of meanings. This paper will not participate in the arguments about mini-c (interpretive creativity), little-c (everyday creativity), pro-c (expert creativity) and big-c (legendary creativity) (Beghetto & Kaufman, 2013), but rather on how we assess creativity in the use of ICTs in design projects in pre-service teacher education. Sternberg (1991) declared that “assessments of creativity are in need of serious re- consideration and especially broadening,” implying that more consideration needs to be placed on how we assess creativity. There are a number of arguments that have been put forward about creativity; however, these can be distilled down to the judgments about the processes or products by an expert in the field (such as a teacher), in an educational context. So, in this sense, the rating of assessing of creativity depends on the subjectivity of one who is evaluating the creative output (Sternberg & Lubart, 1995). There is a general consensus across the field that it is an almost impossible task to set up objective criteria to qualify or assess students’ work as being creative. The authors adopted collaborative design assessment tasks to attempt to objectively assess the creative design of artefacts that can be used for teaching and learning in schools. Earlier papers by the authors have investigated the benefits of collaborative design (see for example, Authors, 2014) and agree with previous findings that within in the field of technology enhanced learning, the benefits of long-term design teams and the use of peer feedback have been well documented and successful (see for example, Jeong & Chi, 2007). What is missing from these studies is an understanding of how educators assess the creative outputs of their students. There are several strategies that have been put forward to establish measures of objectivity in assessing creativity in design or creative works to ensure equivalence across student outputs. What this means is that, without guidelines or structure, students have no boundaries on what the task may encompass, which makes it difficult for assessors to judge the equivalence of students’ creative works. There are countless strategies that have been applied in this endeavor, one of which is to in the actual design of the task. Tasks need to be designed as a process that enable students to receive guidance on their progress across three experiments, with each task using the skills and knowledge acquired in the previous task. The process should provide students with space to engage with two key aspects of creative design – the recognition and definition of the problem – both of which provide students with space to visualise the problem’s solution corresponding to the task specifications (Taylor, 1969). Research Design The first design task was two-dimensional, to give students a place to develop basic skills before progressing onto the open-ended design task. In the second task, the students were provided with a somewhat open-ended task. The marking criteria provided guidance on what features needed to be included. Therefore, while the tasks did not constrain students’ creative design of the product, they did constrain the boundaries of the task. Another strategy to support functional creativity is through group interactions that enable the generation of divergent ideas and critical reflection. Lucas et al. (2013) argue that one of the crucial aspects of creativity is divergent thinking, or, as they explain, the “ability to generate many ideas for a range of perspectives without being limited by preconceived thinking” (p. 14). They clarify that the ability to think divergently is important, but not a proxy of creativity. In regards to assessment design, there are a number of arguments that have been put forward on the value of having formative assessment built into the process (see, for example, Leahy and Wiliams, 2009). However, the nature of assessment in higher education necessitates an assessment instrument or rubric, and it is at that precise point that assessing creativity becomes objective. Ideas explored in the literature that support assessment of creativity include the use of descriptive rubrics, assessment by peers, assessment using portfolios, mixed methods of assessment and self- assessment (Lucas et al., 2013). The value of formative feedback in developing creativity is supported CP:130 482 by Beghetto and Kaufman (2013), who hold that educators can help students to develop creativity by providing timely and nuanced feedback relevant to the student’s own work. The rubrics were designed to identify the beyond-mundane functional designs using two qualitative criteria, “distinctive and significant”. For a design to be successful it needed to be both distinctive (demonstrating a unique combination of decisions) and significant (meaningful and relevant). Hence, the rubric was fairly flexible and did not limit or restrict the creative design process, nor did it judge designs using a more standard continuum of ‘developing’ or ‘developed’. The government regulates education degrees in Australia and all students must have exposure to a range of ICT in learning and teaching contexts as part of their course requirements. Students are assessed on their use of ICT during their professional experience placements. Hence, there is a focus on providing students with authentic (classroom) activities in order to develop the skills and knowledge to be able to use ICT. The study was conducted in a primary education course 4th year unit on teaching English, in which all students had previously completed the compulsory core first year unit on ICT in education. There was an expectation that students would have a basic understanding of interactive whiteboards, mobile devices and personal computers to support learning. The students were under-taking a weeklong summer school in intensive mode in January 2015. The students had lectures in the morning on English language teaching and language acquisition theory; they also had a design workshop in the afternoon. Students assigned themselves, before the start of the summer school, to groups of four on the basis of a school stage (1–3). There were four tutorial groups. The students were given their task in the week prior to the summer school so that they had time to think about curriculum areas. For the task, they had to find two ICT resources and build two ICT resources that could be used to support the development of English language and literacy in the context of their curriculum area/s (e.g. science, history, English, art, drama). The group assessment formed 70% of their course mark (30% for the rationale and 40% for the teaching) and the final 30% of the course mark was an essay exam on language teaching methodologies. The students needed to develop a sequence of three lessons using the resources and applying a range of language teaching methodologies. At the end of the week, the students would teach the class for 30 minutes using at least two of their resources. They would also submit a 500-word overview of their design approach and a rationale for why they used the language teaching approaches in that context. Students were advised that marks would be awarded for creative use of ICT; that is, they could pass the task if they found and built useable (authentic) resources, but students that developed innovative approaches to using ICT in education would be rewarded for their effort. The students worked with their tutors in the design workshops for the whole week. The workshops were unstructured in that they were able to select their own stage and curriculum areas. The students were also not given explicit technical support as they needed to be able to collaboratively problem solve and trouble shoot their technical issues. The assessment model built in two peer-review and feedback stages. The first was on the Monday, day one of the summer school, to clarify ideas and curriculum areas, and the second was on the Wednesday two days before the presentation of the final assessment on Friday. All of the tutorials were conducted in computer laboratories; however, students were also welcome to use other areas of the campus (such as the Library). For the assessment, students could use whatever resources they felt best supported language acquisition (e.g. iPads, interactive whiteboard, laptops). The group-teaching task was conducted on the Friday. The tutors video-recorded and took photos of the presentations for moderation purposes. All of the tutors used the same rubric and tasks were moderated post-assessment. The feedback from the tutors revealed that the tasks were difficult to assess in terms of creativity as there was only one person marking the task at the time. They noted that it would be of benefit in future offerings of the unit to have a peer-evaluation process in place. Despite tutor feedback, analysis of the unit grades indicated that the grades were evenly distributed across the trials. Results and discussion Students were administered with a survey in the lecture on the Thursday morning of the summer school, in order to gain an understanding of their views of the approach to the unit and assessment. The surveys were administered during the lecture. Preliminary analysis of the survey results indicated that students liked four aspects of the unit: a) that the problem was open-ended so that they could work to their limits; b) that they were not restricted by technology so that they could select resources that they felt would be of benefit to their students; c) that they had time to work with their peers in CP:131 483 class, so the three hour design workshops were long enough to get work completed; and d) that they had the opportunity to teach with their resources. The issues that the students raised were fairly consistent: firstly, finding and building four resources in a week was too time consuming; secondly, that there was not enough time to trial different language teaching methods, so once they committed to one method they had to stick to that method as there was not have enough time to change; finally, students also suggested that the assessment provided space for them to be creative in their approach to using technology, but that they felt restricted due to time constraints. They did however, responded positively to participating in their peer classes. When assessing creativity in the school environment two important questions need to be considered: firstly, we need to consider the definition of creativity that is being followed; and, secondly why we are assessing creativity in the first place. Both of these questions are important, as they make a significant impact on the form of assessment used. We also need to consider whether we are assessing programs or attempting to identify individuals who are gifted in this area. If we are assessing individuals then consideration of the type of definition for creativity is imperative. For example, if we consider creativity to be a concept that crosses all subject areas, then we would not be concerned about assessment tools that purposefully measured for creativity in specific subject areas. The purpose of creativity being assessed is also an important consideration, as different measures would be used for assessing a program versus an individual gifted in this area. The researchers also measured creativity in terms of the range of tools used and how they were used. We looked for groups that moved away from traditional teacher-led pedagogies and might bring innovation into the classroom. It was evident that assessing \ students’ resources in terms of creativity was difficult because it was subjective. In one instance, the tutors with cohort two indicated that they marked higher for students that had more innovative technologies. The tutors were felt that they were marking the tool and not the creative use of technology to support language learning. One technique for assessing creativity is the Consensual Assessment Technique (CAT), which utilises panel judging. According to Kaufman et al. (2008), in CAT experts evaluate an artefact or product, not the process. This method of panel judging is fairly common in other forms of talent assessment, such as Nobel prizes and grant applications. In creativity assessment there may be a range of formats for the judging panels, such as blind review or conferral; however, the common evaluation criteria is that the judging panel are all assumed to be experts. As such, after students had their design subjected to the rigours of self- and peer-review, the final stage of the assessment of creativity should be undertaken by a panel of experts. Hence, the validity of CAT is premised on the use of discipline experts as judges (Kaufman et al., 2008). The use of CAT or other similar panels of experts raises questions of who the most valid judges are in educational contexts. Are they the teachers or the industry experts? Is it necessary to bring in an external stakeholder to judge the designs as creative or valid? It is reasoned that perhaps the best choice of experts will depend on the purpose of the assessment. If the goal is to find the most accurate assessment of a creative design then the teaching staff would be the most logical choice of assessors. However, bringing in an industry expert may present students with another level of feedback to inform their designs. Conclusions This paper aimed to discuss processes that were put in place to attempt to objectively assess the creative design of artefacts made by pre-service teachers for use in the classroom. Several strategies were used including a process-based task design, opportunities to revisit and refine designs, collaborative brainstorming, self-assessment, rubrics, panel marking by experts, and a design space that supported creativity and was not constrained by two-dimensional limits. This paper does not intend to be a thorough investigation of the data, the technical capabilities of the pre-service teachers or the pedagogical underpinnings of their design rather it is a work-in-progress. The literature on creativity suggests that creativity is a quality perceived to be of value in the workplace. The authors offer a platform for discussions over how functional creativity can be measured in the context of learning design. If creativity is a desirable workplace attribute, we as educators need to provide space for our students to demonstrate their creativity and we also need to assess and provide valuable feedback to students so that their creative processes can develop. The question must be asked, how can we best design learning experiences that provide students with space to be creative? CP:132 484 And how does this creativity become functional in the workplace? Creative functionality is generally tied to a measurable output and needs to be tangible to students to have any real value References Altbach, P. G., Reisberg, L., & Rumbley, L. E. (2009). Trends in global higher education: tracking an academic revolution. Report prepared for the UNESCO 2009 World Conference on Higher Education. Paris: UNESCO Beghetto, R. A., & Kaufman, J. C. (2013). Creativity: Five fundamental insights that every educator should know. Educational Leadership, 70, 10-15. Cram, A., Lowe, R. & Lumkin, K. (2014). Assessing spatial design in virtual environments. In S. Kennedy-Clark, K. Everett & P. Wheeler, Cases on the assessment of scenario and game-based virtual worlds in Higher Education (pp. 74-116). Hershey, PA: IGI-Global. Florida, R. (2002). The rise of the creative class. New York: Basic Books. Jeong, H., & Chi, M. T. H. (2007). Knowledge convergence and collaborative learning. Instructional Science, 35, 287-315. Kaufman, J. C., Baer, J., Cole, J. C., & Sexton, J. D. (2008). A comparison of expert and nonexpert raters using the Consensual Assessment Technique. Creativity Research Journal, 20, 171–178. Ketelhut, D. J., Clarke, J., & Nelson, B. (2010). The development of River City, a multi-user virtual environment-based scientific inquiry curriculum: historical and design evolutions. In M. J. Jacobson & P. Reimann (Eds.), Designs for learning environments of the future: International perspectives from the learning sciences. Springer Publishing Company. pp. 89-110. Leahy, S., & Wiliam, D. (2009). From teachers to schools: scaling up formative assessment. 2009 AERA Annual Meeting on Disciplined Inquiry: Education research in the circle of knowledge, San Diego. Lucas, B., Claxton, G., & Spencer, E. (2013). Progression in student creativity in school: First steps towards new forms of formative assessments. OECD Education Working Papers. Paris: Organisation for Economic Cooperation and Development (OECD). Retrieved from http://ezproxy.library.usyd.edu.au/login?url=http://search.proquest.com/docview/1274643691 ?accountid=14757 Miller, W. C. (1986). The creative edge. New York: Addison-Wesley. Sternberg, R. S. (1991). Three facet model of creativity. In R. S. Sternberg (Ed.). The nature of creativity. Contemporary psychological perspectives (pp. 125-148). Cambridge, N.Y.: Cambridge University Press. Sternberg, R. S. & Lubart, T. I. (1995). Defying the crowd. London: The Free Press. Taylor, D. (1969). Creative design through functional visualization. The Journal of Creative Behavior, 3(2), 122–127. Kennedy-Clark, S., Kearney, S., Eddles-Hirsch, K., De La Hoz, R., Galstaun, V., & Wheeler, P. (2015). Measuring creativity in collaborative design projects in pre-service teacher education. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:129-CP:133). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:133 485 How to develop an online community for pre-service and early career teachers? Nick Kelly Marc Clarà Marlene Pratt University of Southern Queensland University of Lleida Queensland University of Technology Queensland University of Technology This paper contributes a number of design principles for developing large-scale online communities of pre-service and early career teachers (PS&ECTs). It presents the paradigms of connected learning, networked learning and communities of practice and contrasts them. It describes the potential for online communities to meet the needs of PS&ECTs and it identifies gaps that exist within certain types of existing online communities that currently support PS&ECTs. The paper proposes design principles for a new type of online community for PS&ECTs. These principles are drawn from the literature and from the preliminary outcomes of a pilot study. Keywords: teacher education, online community, early career, pre-service, design based research, connected learning, networked learning, community of practice Why an online community for teachers? There are many challenges to beginning a career as a teacher (Veenman, 1984). Support during this period of transition into service is critical and is particularly useful in the form of mentoring and induction programs (DeAngelis, Wall, & Che, 2013; Ingersoll & Strong, 2011). Online communities are a form of support that have the potential to stimulate collegiality between pre-service and early career teachers (PS&ECTs) (Herrington, Herrington, Kervin, & Ferry, 2006; Kelly, 2013). This paper aims to 1 present design principles from ongoing design-based research aimed at creating an online community of PS&ECTs across multiple institutions in the state of Queensland (Kelly, Reushle, Chakrabarty, & Kinnane, 2014). It is structured by presenting theoretical background and the argument for why there is a need to design and develop a new type of community for PS&ECTs; and then articulating strategies for how to develop such a community. There have been a number of recent attempts to augment the support for pre-service and early career teacher with the formation of online communities (e.g. Herrington et al., 2006; Lee & Brett, 2013; Lin, Lin, & Huang, 2008; Maher, Sanber, Cameron, Keys, & Vallance, 2013). Such attempts typically adopt one of three complementary paradigms, each of which make a commitment to valuing the connectedness between learners: (online) communities of practice (Lave & Wenger, 1991; Wenger, White, & Smith, 2009), connected learning (Ito et al., 2013) and networked learning (Goodyear, Banks, Hodgson, & McConnell, 2004). In this work we will refer to online communities with an understanding that they can be viewed through any or all of these lenses which place the emphasis respectively (and arguably, given the diversity of views that each term has come to represent) upon: • (communities of practice) The cultural norms and collaborative relationships that emerge within a group of practitioners with common purpose, where “communities of practice are groups of people who share a concern or a passion for something they do and learn how to do it better as they interact regularly” (Wenger, 2011). • (connected learning) The open nature of learning in a connected world allows for learning to be authentic and linked with society beyond classroom walls to promote interest and hence learning, where connected learning is “embedded within meaningful practices and supportive relationships” and is committed to recognising “diverse pathways and forms of knowledge and expertise” (Ito et al., 2013) • (networked learning) Learning is understood to take place through connections of learner-learner and learner-resource and this connectedness can be greatly enhanced through technology, where networked learning is “learning in which ICT is used to promote connections between one learner and other learners; between learners and tutors; between a learning community and its learning resources” (Goodyear et al., 2004) 1 For details of the ongoing project see http://www.stepup.edu.au CP:134 486 In short, research in these paradigms has shown that online communities of members with a shared practice can be extremely useful. They bring together in one place the people that a practitioner is likely to draw upon for questions about practice. They support the creation of such connections. Through interaction, they facilitate the development of rich stores of (third person, represented) knowledge that is accessible to all members. Whilst online communities can be a part of formal education or professional development, they are often informal. Globally, there has been a trend towards the adoption of online communities in which the term social network has become the successor to ‘Web 2.0’ (boyd & Ellison, 2007). Many professions and groups of practitioners now have online communities associated with them; and some have even transformed the nature of the practice associated with them (e.g. Mamykina, Manoim, Mittal, Hripcsak, & Hartmann, 2011). Large scale communities (with hundreds, thousands or even millions of members) offer the potential for facilitating valuable connections within the profession. This may be between members (e.g. a beginning teacher in a remote school might be connected with another beginning teacher in a similar situation) or between members and resources – the larger the network, the more likely that the individuals or resources needed can be found. There is, however, a trade-off with social presence and engagement being challenging to achieve in larger communities (Clará, Kelly, Mauri, & Danaher, In press). In this context, our argument is that large scale online communities have much potential to support PS&ECTs that is yet to be fulfilled. Firstly, what are the needs that PS&ECTs have from an online community? Six categories for the ways in which teachers can support one another online can be drawn following the work of Clarke, Triggs, and Nielsen (2014): (i) supporting reflection; (ii) modeling practice; (iii) convening relationships; (iv) advocating practical solutions; (v) promoting socialisation within the profession; and (vi) giving feedback. Many existing platforms that are used by PS&ECTs successfully enable teachers to convene relations, promote socialisation and advocate the practical. However, there is a dearth of large scale sites (i.e. more than 200 users) that promote reflection, feedback and modelling of practice. This is perhaps due to teachers feeling a need for privacy (a closed online space), trust (in other members of the community) and some kind of stability (in membership of that community) that is not met by the current generation of large scale online communities of PS&ECTs (Clará et al., In press). Early results from current work by the authors analysing the interactions of teachers in Facebook supports this hypothesis. There are many existing large scale online communities for teachers within Australia, however none fills all of these needs of PS&ECTs. Whilst an empirical survey of these communities is required to fully substantiate this claim, some types of online community available in Australia can be identified, Table 1, and limitations based upon anecdotal evidence described. “Scootle Community” is a national, government funded site that appears to have low levels of engagement and social presence amongst users, with low level activity on the site given the pool of potential users, possibly due to a lack of stability (constantly changing users), privacy (all data is owned by the government and is visible to all members) and, hence, trust. The Queensland state government supported site “The Learning Place” comes closest of the examples given to fulfilling the potential of online communities to meet PS&ECT needs. It has high levels of activity, with many widely-used resources that are the focus of discussion and for facilitating connections between users. However, the state government (who also employ many of the teachers using the site) owns the data and is heavily visible through logos and announcements on the site. This, along with broad visibility in most sections of the site, might be limiting trust for users of the site to share details of practice. There is little evidence of teachers developing the close connections needed for reflecting on practice, providing feedback or modelling practice (however, this may be occurring in private channels of communication). Many groups of PS&ECTs have arisen on the commercial platform “Facebook” (and similarly on “EdModo”). Some groups are visible and massive, whilst many are small and private. There is much variation between groups, however they have in common that: (i) the knowledge developed by the community is not searchable or reusable and, hence, is lost; and (ii) each new group springing up begins anew, losing the benefits of having a large established community. Many teacher education institutions also have their own intra-institutional online communities that can often support highly engaged, collegial support – however they are limited in size, cannot facilitate cross-institutional networks and are susceptible to fluctuating support from their host institutions (e.g. funding changes or key staff leaving). CP:135 487 Table 1: Types of online communities used by PS&ECTs in Australia with examples Type of Example of community Description of example community Nationwide, Scootle Community Federal Government supported site (run by government http://community.scootle.edu.au Education Services Australia) to facilitate a funded social network (Facebook style) around Scootle resources in particular and the teaching profession in general. Available to most educators in the country. Statewide, The Learning Place State Government supported site (run by government http://education.qld.gov.au/ Education Queensland) with a large and funded learningplace/ widely used collection of resources for classrooms and professional development, with social network support (chat, blogs, learning pathways) Commercial Facebook groups Widely-used commercial site that supports https://facebook.com many diverse groups of teachers. Some are openly available and some are private; ranging from the very small to the very large. Institutional Education Commons (USQ) A Moodle community of PS&ECTs supported https://open.usq.edu.au/course/ by motivated faculty members who provide a info.php?id=62 library of articles, videos and mentoring through the site (Henderson, Noble, & Cross, 2013). Design principles for “TeachConnect” With this understanding of the gap that remains, a group of academics from universities and teacher education providers across Queensland are working together to develop a community, TeachConnect, which will be launched in September 2015 and supported by the Queensland College of Teachers and an Office of Learning and Teaching grant. TeachConnect aims to augment current support for PS&ECTs by filling in the gaps identified above. A number of design principles for developing the site can be listed as: • It is independent and data (e.g. conversations) are private, owned by the members of the community – this is reflected in the lack of institutional presence (e.g. logos) on the site and the focus upon the profession (e.g. inspiring quotes about education). • It is single purpose (i.e. doesn’t have to meet government or institutional priorities) and its appearance and design make it clear that its goal is to facilitate PS&ECTs supporting one another. • It is free and universal in that all teachers have access to the site, regardless of school system or status of employment. • It is also restricted to individuals who have at some point been a pre-service teacher, to maintain the focus upon developing professional practice. • Knowledge that can be separated from its context and proponent is co-created and re-usable (e.g. where to find resources, how to get accredited, how to navigate schools) and develops over time. • There is a two-layer design that has clearly defined separation between what is publicly visible and a trusted, private space which is the focus of the site, where close relationships can develop, allowing for reflection upon practice between peers and facilitated by experienced teachers (a type of mentorship). • It is designed to be simple, quick and easy to use so that there is a minimal threshold to overcome to commence using the site (one-step sign on facilitated by close co-ordination with universities). • It is possible because it is widely supported by many universities within Queensland. It relies upon the shared purpose that all schools of education have in wanting the best possible outcomes for PS&ECTs, is inclusive in design and is freely accessible by all teacher education institutions. The process of developing TeachConnect: Lessons learned The process of developing TeachConnect has followed the principles of design-based research through multiple iterations of design involving the input of participants (Barab & Squire, 2004; Collins, Joseph, & Bielaczyc, 2004). The design-based paradigm is a good fit for this work, as educational CP:136 488 research is heavily context dependent, and at the same time the literature on developing online communities suggests that the exercise is far from being an exact science. Some heuristics for developing any kind of online community were distilled by Shirky (2010) as: (i) start small with a core community, as if you rely on being big it will probably never happen; (ii) understand and provide for what motivates your members (both intrinsic and extrinsic motivation); (iii) use the default options in the platform wisely to promote social connectivity; (iv) cater for all types of engagement (e.g. lurkers as well as active participants); (v) have as low a threshold as possible to get started on the site; (vi) tweak as you grow and be responsive to what the community is asking for. The vision for TeachConnect was informed in part by the literature, but also through focus groups (with PS&ECTs, teacher educators, experienced teachers and stakeholder organisations), a survey (Kelly et al., 2014; N=183) and a pilot study. Whilst details of this pilot and the development of TeachConnect are forthcoming, the essence of the lessons learnt can be distilled here. A pilot of a platform for PS&ECTs was conducted in 2014 (www.TeachQA.com) and involved over 200 pre- service teachers across two universities, and over 20 experienced teachers to develop a community. An evaluation of the problems experienced in this site revealed that it was: (i) Too difficult to sign up to; (ii) too restrictive in interactions (with not enough opportunity for dialogue; (iii) too public and did not allow for trust to develop (no private spaces for interaction); and (iv) not enough community engagement to remind PSTs that the site existed. In response, the TeachConnect platform is being integrated with a schedule of community engagement. Researchers will travel and talk to the lecturers, pre-service teachers and teachers who will be using the platform to build the community. The platform will be strongly customised to be specific to teachers’ needs, rather than using something “off-the-shelf”. We plan to work with an initial group of dedicated users to build a group culture, and help them as they do this. Ultimately, the use of the platform will only spread if it is fundamentally useful – there are no short cuts for building an online community. References Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. Journal of the Learning Sciences, 13(1), 1-14. boyd, d. m., & Ellison, N. B. (2007). Social network sites: Definition, history, and scholarship. Journal of Computer‐Mediated Communication, 13(1), 210-230. Clará, M., Kelly, N., Mauri, T., & Danaher, P. (In press). Challenges of teachers’ practice-oriented virtual communities for enabling reflection. Asia-Pacific Journal of Teacher Education. Clarke, A., Triggs, V., & Nielsen, W. (2014). Cooperating Teacher Participation in Teacher Education A Review of the Literature. Review of Educational Research, 84(2), 163-202. Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and methodological issues. Journal of the Learning Sciences, 13(1), 15-42. DeAngelis, K. J., Wall, A. F., & Che, J. (2013). The Impact of Preservice Preparation and Early Career Support on Novice Teachers’ Career Intentions and Decisions. Journal of Teacher Education. Goodyear, P., Banks, S., Hodgson, V., & McConnell, D. (2004). Research on networked learning: An overview Advances in research on networked learning (pp. 1-9): Springer. Henderson, R., Noble, K., & Cross, K. (2013). Additional professional induction strategy (APIS): Education Commons, a strategy to support transition to the world of work. Herrington, A., Herrington, J., Kervin, L., & Ferry, B. (2006). The design of an online community of practice for beginning teachers. Contemporary Issues in Technology and Teacher Education, 6(1), 120-132. Ingersoll, R. M., & Strong, M. (2011). The Impact of Induction and Mentoring Programs for Beginning Teachers A Critical Review of the Research. Review of Educational Research, 81(2), 201-233. Ito, M., Gutierrez, K., Livingstone, S., Penuel, B., Rhodes, J., Salen, K., . . . Watkins, S. C. (2013). Connected learning: An agenda for research and design: Digital Media and Learning Research Hub. Kelly, N. (2013). An opportunity to support beginning teachers in the transition from higher education into practice. Paper presented at the ASCILITE 2013, Macquarie University, Australia. Kelly, N., Reushle, S., Chakrabarty, S., & Kinnane, A. (2014). Beginning Teacher Support in Australia: Towards an Online Community to Augment Current Support. Australian Journal of Teacher Education, 39(4), 4. CP:137 489 Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation: Cambridge university press. Lee, K., & Brett, C. (2013). What are student inservice teachers talking about in their online Communities of Practice? Investigating student inservice teachers’ experiences in a double- layered CoP. Journal of Technology and Teacher Education, 21(1), 89-118. Lin, F.-r., Lin, S.-c., & Huang, T.-p. (2008). Knowledge sharing and creation in a teachers’ professional virtual community. Computers & Education, 50(3), 742-756. doi:http://dx.doi.org/10.1016/j.compedu.2006.07.009 Maher, D., Sanber, S., Cameron, L., Keys, P., & Vallance, R. (2013). An online professional network to support teachers’ information and communication technology development. Paper presented at the ASCILITE 2013, Macquarie University. Mamykina, L., Manoim, B., Mittal, M., Hripcsak, G., & Hartmann, B. (2011). Design lessons from the fastest q&a site in the west. Paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems. Shirky, C. (2010). Cognitive surplus: Creativity and generosity in a connected age: ePenguin. Veenman, S. (1984). Perceived problems of beginning teachers. Review of Educational Research, 54(2), 143-178. Wenger, E. (2011). Communities of practice: A brief introduction. Wenger, E., White, N., & Smith, J. D. (2009). Digital habitats: Stewarding technology for communities: CPsquare. Kelly, N., Clarà, M., & Pratt, M. (2015). How to develop an online community for pre-service and early career teachers? In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:134-CP:138). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:138 490 Collaboration between Primary Students and the Use of an Online Learning Environment: The Previous Collaborative Work Experiences Factor Aikaterini Kokkinaki University of Bristol This paper reports findings from a research study which involved the use of an Online Learning Environment by Greek primary students in their school classroom and from home for a period of six weeks for the development of a wiki for a school project. This research study sought to answer whether and how collaboration can be supported between primary students with the use of an Online Learning Environment. Although collaboration is often reported as the outcome from the use of technology in an educational context, this paper presents research findings to show that collaboration between primary students with the use of an Online Learning Environment is associated with students' previous collaborative work experiences. Keywords: Collaboration, Online Learning Environment, Primary Education, Previous Experiences Introduction In recent years, the rise of cloud computing and faster internet connections have not only been seen as opportunities for education to extend students' learning spaces beyond the walls of the classroom, but also as a means to bridge learning spaces across school, home, and the wide community (Jimoyiannis et al., 2013). Along with the rise of cloud computing and faster internet connections, the emergence of online tools which support synchronous and asynchronous communication, file sharing and creation of joint documents has generated interest regarding the opportunities for collaboration supported with these technologies in the context of education (Mader, 2007; Pilkington and Walker, 2003; Traxler, 2010). Thus, changing Information and Communications Technologies (ICT) contexts have increased opportunities for children to communicate and potentially collaborate from different locations online. Online Learning Environments (OLE) have been seen as online spaces where collaboration between students can be supported. Although previously published studies use terms such as: Virtual Learning Environments, Managed Learning Environments, Personal Learning Environments, Learning Platforms and Course Management Systems to refer to online environments that are used for educational purposes (British Educational Communications & Technology Agency, 2005), in this paper an OLE is understood as an online space that: Includes the components through which the learners and the tutors participate in online interactions including online learning (Joint Information Systems Committee, 2006, p. 6). The majority of studies that have researched collaboration with the use of an OLE concern the context of higher education. As discussed by Kennedy (2009), OLEs have been used in the context of higher education to support: communication and collaboration between students, assessment, the publication of online content as well as management and tracking of students. With regard to communication and collaboration, it has been shown that using an OLE increases the level of communication and collaboration between higher education students (Selinger, 1997) by giving students more chance to articulate their thoughts and understanding (Chou and Liu, 2005). In relation to collaboration and the use of an OLE in higher education, the study of Pilkington and Walker (2003) places particular interest on the exploration of student group collaboration with the use of an OLE. In this study, the authors investigated the participation of students in online debates using synchronous communication tools and they also explored whether students could work collaboratively in order to CP:139 491 compose joint reflections based on the material they discussed during the debate using an asynchronous discussion board. Pilkington and Walker (2003) found that the opportunities for online collaboration are not expanded by simply combining online tools, but by using the tools of an OLE respectively in order to support the purposes of the different tasks. Further to the work of Pilkington and Walker (2003), the results from the study of Timmis et al. (2010), showed that higher education students may not choose the online tool of an OLE that is best for the task but may instead migrate towards tools that full their social needs. The results from the studies of Pilkington and Walker (2003) and Timmis et al. (2010) provide confirmatory evidence that the integration of an OLE in education cannot alone support collaboration. The incorporation of online tools in an OLE, as for instance a wiki, is often perceived as a way in which collaboration between students can be supported (Bold, 2006; Kovacic et al., 2007; Lund, 2008). On the other hand, it has also been found that when students use a wiki they distribute the effort and each student, or pairs of students, take ownership for the part of the wiki that is assigned to them (Grant, 2006). Therefore, although a wiki is commonly considered a tool that supports collaboration between students by enabling them to create jointly developed content online, in practise collaboration between students may not be supported. That is because a wiki is nothing more than a collective website where a large number of participants are allowed to modify any page or create a new page using their web browser (Desilets and Paquet, 2005). There are a number of factors associated with collaboration and the use of online tools in education. Previously published research in the subject areas of collaboration and technology has suggested that higher education students' previous experiences with technology is associated with how students will use this technology to collaborate online (Kreijns et al., 2003; So, 2009). However, the term “previous experiences” has been mainly used to describe students' previous negative experiences with technology (Pauli et al.,2008; Vrasidas and McIsaac, 1999). This paper aims to shine new light and answer whether and how primary students' previous experiences into collaborative work impact collaboration with the use of technology and particularly with the use of an OLE. Methodology The study reported in this paper follows the multiple case studies research design that is comprised of three case studies and follows the literal replication logic, which means that the cases were designed to predict similar results (Yin, 2009). The type of case study followed is the explanatory case study. The “case”, are the students of a sixth grade primary classroom in Greece for the six week period of use of the designed OLE in the school classroom and from home for the development of a wiki for a school project. The 24 students of Case Study 1, the 12 students of Case Study 2 and the 12 students of Case Study 3 worked in groups of three or four at school and used the tools the were integrated within the designed OLE i.e., discussion forum, instant messaging and wiki at school as instructed by their teacher and in order to address the tasks that were designed by the teacher aiming to develop a joint wiki project. The topics of the wiki projects were: “Our Solar System”, “The Wonders of the Modern World” and “Species Near Extinction” for Case Studies 1, 2 and 3 respectively. A characteristic sample of the tasks, as given to students by the teacher in Case Study 1, is the following: “Work in groups and use the internet to find images for the planet which was allocated to your group and then upload the images found at the wiki”. Those students who had access to the OLE from home continued to use the online tools of the designed OLE and contributed content at the wiki from home. The following methods of data collection were utilised to support a holistic investigation of whether and how collaboration between primary students was supported with the use of an OLE: observation, focus group, questionnaire and data generated from the designed OLE. In this paper, results from the analysis of the observation and focus group data will be presented. The students were observed in a regular classroom session (before the data collection). Also, one group of students was observed in each case study every time the students used the designed OLE at school i.e. for one session every week and for a period of six weeks. The focus groups were conducted every two weeks. For the analysis of the collected data, different techniques were employed. For the analysis of the qualitative data, the thematic analysis framework was used. The quantitative data that were collected were analysed with the use of descriptive statistics. CP:140 492 Results The analysis of the observation data that were collected before the students start using the designed OLE at school, show that the students of Case Study 1, always worked in groups of four in the classroom. For the formation of the groups, the teacher took into consideration students' preferences but also regrouped students based on their abilities. The 12 students of Case Study 2 also worked in groups of four. Extra chairs were available in each group which allowed students to move around within the groups. All tasks that were designed by the teacher involved group work between the students. The way students worked in Case Study 3 differed from the way the students in Case Studies 1 and 2 did. It was observed that the students were sitting in pairs and that there was no mobility between the students. Moreover, all tasks that were assigned to students by their teacher did not involve group work. It was also observed that the students worked individually to the extent that they placed their note books and books vertically as desk dividers. According to the teacher of Case Study 3, this practice commonly occurred because students wanted to avoid their peers to cheat or copy their work. When the students used the designed OLE in their classroom, they shared information and gave help and feedback face-to-face (with the other students of their group) and online (via the designed OLE with the other groups). Sharing, help and feedback and joint work were the main themes that emerged from the thematic analysis conducted. In this paper, only qualitative data from the thematic analysis of the observation and focus group data will be presented and the results concern only face- to-face collaboration between primary students in their school classroom. Case Study 1 In Case Study 1, face-to-face collaboration was supported when the students worked with their group members for the development of a discussion forum or a wiki publication. Face-to-face within group collaboration wasn't supported when the students worked for the development of an instant messaging contribution. A characteristic extract from researcher's audio recorded reflection on classroom observations for Case Study 1 is given below: Nikos argues that he has identified the planet Jupiter and shares with his group information to support his argument. The information shared concerns the colour of this planet. Marina asks Nikos if he is completely sure about the colour of Jupiter and Nikos replies positive. Marina asks him to give further information over how he has come to know that and also asks him to say where he has read that. Nikos justifies his argument by explaining where the information can be found in their geography course book. The reasons that were given by students during the focus groups, in relation to the reasons for sharing information, giving help and feedback and participate in joint work with the other group members for the development of a discussion forum or a wiki publication, were: • For students to become assured and convinced about the accuracy of the information to be shared • For students to minimise potential negative comments to be received by other groups • For students to accommodate the different ideas shared Case Study 2 In Case Study 2, the students also collaborated face-to-face with the other members of their group. The situations that were interpreted as collaborative involved sharing of information, giving help and feedback and participate in joint work. The basic difference between Case Study 1 and Case Study 2 is that, in the latter, the students didn't only share information but rather created shared understanding over the information shared. A characteristic extract from researcher's audio recorded reflection on classroom observations for Case Study 2 is given below: Dimitris asks what counts as a modern wonder and explains that they have to be careful not to publish something irrelevant to what was asked. Katerina suggests to first decide CP:141 493 what the term wonder means and shares with her peers what the term “wonder” means for her. Dimitris asks whether the term modern wonder stands for places, monuments or constructions. Katerina argues that it could be everything as long they attract the attention of people. She suggests saying the Acropolis. Andreas asks Katerina what does it make her to believe that Acropolis is a wonder of the modern world and Katerina argues that the design of the Acropolis could be considered even nowadays contemporary and modern. Dimitris agrees but instead proposes to focus on constructions that were built in the last century in order to make their search more specic. All agree. This extract shows that students do not only construct a discussion forum post together, they are constructing a joint understanding over what the phrase “wonder of the modern world” stands for, for them. Case Study 3 The students of Case Study 3 also shared ideas, previous knowledge and information each time they participated in discussions for the development of discussion forum and/or instant messaging posts. However, face-to-face collaboration was not supported between the group members because it was difficult for those students to bring together the different ideas shared into a joint discussion forum or instant messaging post. A characteristic extract from researcher's audio recorded reflection on classroom observations for Case Study 3 is given below: Gianna expresses that nature disapproves humans because of the atmospheric pollution. Giorgos replies that their project is about the extinction of animals and suggests to focus on things that men do which have consequences to animals' extinction. Gianna disagrees and challenges him to read the teacher's question. She asks Soa and Vasia to take her side for being close friends. Giorgos asks them to develop a post for themselves informing them that he will publish alone whatever he believes is correct. A summary of the actions reported by the students of Case Study 3 to happen after a disagreement over ideas shared in their face-to-face discussions is given below: Follow the group leader's argument Publish individual posts at the discussion forum Publish one discussion forum post with all the arguments/ideas expressed No participation in the process of developing the discussion forum post Discussion Previously published research on collaboration and technology demonstrated an association between collaboration and students' previous experiences with technology (Harasim, 1995; So, 2009). However, the term “previous experiences” has been mainly used to describe students' previous negative experiences with technology (Pauli et al., 2008); Vrasidas and McIsaac, 1999). The findings of the study reported in this paper, reveal that not only previous negative experiences with technology but also the absence of previous collaborative work experiences affects how primary students will potentially collaborate by using an OLE in the school classroom. In Case Study 1 and Case Study 2, the students were working in groups before they start using the designed OLE at school and had no particular difficulties or concerns with regard to sharing ideas or information with the other students of their group or bringing together the different ideas expressed. In Case Study 2 especially, there was mobility between the groups of students which allowed them to make short visits to the other groups and ask questions relevant to the task given by the teacher. On the contrary, the students in Case Study 3, had no previous collaborative work experiences before they start using the OLE at school. The predominant culture in their classroom was to work individually. Although the students in Case Study 3 were sitting in pairs, they were working individually and rather competitively (e.g. they placed their notebooks and books vertically as desk-dividers). Furthermore, the tasks that were developed by the teacher (before students start using the OLE) involved students working individually to finish the work given. When the students in Case Study 3 started to work with the designed OLE at school, it was difficult for them to work collaboratively. It was CP:142 494 even difficult for them to bring together and accommodate the different ideas shared by group members in order to develop joint discussion forum, instant messaging or wiki publications. Conclusions In this paper it is argued that the way in which the sixth grade primary students worked with their peers at school without using technology, is reflected in the way in which they performed group work when they used technology i.e. the designed OLE. Situations that were interpreted as collaborative were found to occur with the students of Case Studies 1 and 2, whereas in only a few instances face- to-face collaboration was supported between the students of Case Study 3. This reveals two main aspects: Firstly, the predominant culture in the classroom affects online collaboration via an OLE and secondly, the absence of previous collaborative work experiences affects how students will potentially use an OLE to collaborate. Acknowledgements The author would like to thank Dr Jocelyn Wishart and Dr Federica Olivero for their contribution to the work being reported in this paper, especially for their scientific guidance. This research was funded by the Greek State Scholarships Foundation (IKY). References Bold, M. (2006). Use of wikis in graduate course work. Journal of Interactive Learning Research, 17(1), 5-14. Braun, V. and Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. British Educational Communications & Technology Agency (2005). An introduction to learning platforms for school leaders http://www.lancsng.ac.uk/ictservices/ictcentre/download/ [viewed 19 October 2013]. Chou, S.-W. and Liu, C.-H. (2005). Learning effectiveness in a web-based virtual learning environment: a learner control perspective. Journal of Computer Assisted Learning, 21(1), 65-76. Desilets, A. and Paquet, S. (2005). Wiki as a tool for web-based collaborative story telling in primary school: A case study. NRC Publications Archive, Canada. Grant, L. (2006). Using wikis in schools: A case study. Futurelab Archive. http://www.futurelab.org.uk/resources/publications reports articles/discussion papers/ Discussion Paper258 [viewed 21 December 2014]. Harasim, L. M. (1995). Learning networks: A field guide to teaching and learning online. Cambridge, MA: MIT press. Jimoyiannis, A. and Angelaina, S. (2012). Towards an analysis framework for investigating students' engagement and learning in educational blogs. Journal of Computer Assisted Learning, 28(3), 222-234. Joint Information Systems Committee (2006). Effective Use of VLEs: Introduction to VLEs. http://tools.jiscinfonet.ac.uk/downloads/vle/what-is-vle.pdf [viewed 19 October 2013]. Kennedy, D. J. (2009). Virtual Learning Environments (VLEs): here to stay, or on the brink of demise? The Plymouth Student Educator, 1(1), 58-66. Kovacic, A., Bubas, G. and Zlatovic, M. (2007). Evaluation of activities with a wiki system in teaching English as a second language. International Conference ICT for Language. Kreijns, K., Kirschner, P. A. and Jochems, W. (2003). Identifying the pitfalls for social interaction in computer-supported collaborative learning environments: a review of the research. Computers in Human Behavior, 19(3), 335-353. Lund, A. (2008). Wikis: A collective approach to language production. ReCALL, 20(1), 35. Mader, S. (2007). Using wiki in education. Stewart Mader. Pauli, R., Mohiyeddini, C., Bray, D., Michie, F. and Street, B. (2008). Individual differences in negative group work experiences in collaborative student learning. Educational Psychology, 28(1), 47-58. Pilkington, R. M. and Walker, S. A. (2003). Facilitating debate in networked learning: Reflecting on online synchronous discussion in higher education. Instructional Science, 31(1-2), 41-63. Selinger, M. (1997). Open learning, electronic communications and beginning teachers. European Journal of Teacher Education , 20(1), 71-84. So, H.-J. (2009). When groups decide to use asynchronous online discussions: Collaborative learning and social presence under a voluntary participation structure. Journal of Computer Assisted CP:143 495 Learning, 25(2), 143-160. Timmis, S., Joubert, M., Manuel, A. and Barnes, S. (2010). Transmission, transformation and ritual: An investigation of students' and researchers' digitally mediated communications and collaborative work. Learning. Media and Technology, 35(3), 307-322. Traxler, J. (2010), Students and mobile devices, Research in Learning Technology, 18(2), 3-15. Vrasidas, C. and McIsaac, M. S. (1999). Factors influencing interaction in an online course. American Journal of Distance Education, 13(3), 22-36. Yin, R. K. (2009). Case study research: Design and methods. Sage Publications. Kokkinaki, A. (2015). Collaboration between Primary Students and the Use of an Online Learning Environment: The Previous Collaborative Work Experiences Factor. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:139 -CP:144). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:144 496 A digital what? Creating a playspace to increase the quality of technology-enhanced teaching and learning Heather Lamond Andrew John Rowatt Massey University, New Zealand Massey University, New Zealand This paper outlines a work in progress to create a shared learning space that will enable teaching staff to be exposed to a broad range of established and emerging digital technologies with the aim of increasing their digital literacy and self-efficacy levels so that technologies can be integrated into teaching practice. The project is a partnership between the Centre for Teaching and Learning, and the Library and will facilitate easy, supported access to technologies that individual teaching staff would not otherwise be able to experience. Premised on the importance of experiential learning to develop knowledge, skills and confidence the space will be designed for collaborative and play- based learning and development. Keywords: barriers to adoption, emerging technology, digital literacy, teacher development Introduction To increase the quality of technology-enhanced teaching and learning, teaching staff must have the skills, confidence and technical ability to use technology in the teaching environment. This project is based on the premise that if teaching staff are to develop these skills they need access to technology in both a structured and unstructured way in order to discover, play and experiment with the outcome being confident use of technology in the teaching and learning environment. Through the creation of a “digital playspace” equipped with technologies that most individual departments or schools would not be able to justify or resource it is envisaged that teachers’ digital literacy will be developed and this will lead to comfortable and confident use of technology in teaching. The Vision The project is a partnership between the university Centre for Teaching and Learning and the Library and will result in the creation of a space in the Library that is equipped with a range of established and emerging technologies including a wide range of mobile devices, a small-group optimized and physically shared digital workspace in the form of a digital table, a large scale video wall, with the possibility of virtual reality technology and other emerging technologies including wearables and gesture-based input-devices. This space will be used for both facilitated hands on training sessions with academic staff and for unstructured hands on play. It is a widely held understanding (Buchanan, Sainter, & Saunders, 2013; MacCallum & Verhaart, 2014; Reid, 2014; Schneckenberg, 2010) that one of the barriers to technology adoption by academic staff is lack of access to the technology and this project sets out to begin to mitigate that barrier. MacCallum & Verhaart (2014) also found that lack of knowledge and skill was an impediment to teachers’ use of mobile devices so as well as providing access to technologies, the goal is to develop teachers’ self-efficacy in the use of modern digital technology in the classroom and beyond. While similar collections of technologies are present in some individual departments, schools or service units these are for specific teams to use (e.g.: software development testing) and not widely known about across the institution. By creating the space in the Library access to the technology is facilitated and it will be seen as a University resource. As the aim of the institution is to “strengthen the University’s leadership in digitally-mediated teaching and learning” (Massey University, 2014) it is critical that access to technology for learning and experimentation is made as open and easy as possible for staff. The Library is one of the few truly neutral spaces on campus, and therefore the logical place for a shared facility like this. The space will be set up to allow for informal learning through play as well as facilitated small group CP:145 497 sessions led by staff from the Centre for Teaching and Learning to support teachers in understanding how the technology can be utilised to create enhanced learning experiences for students. Research (Reid, 2014) has shown that a mixture of formal and informal training seems to be the most effective in adoption of technology by teachers so allowing for learning through play, latent learning and exploration (Burghardt, 2012; Matthews & Liu, 2012; Meyer, 2012) is important to this project. Planning The project began in earnest at the beginning of 2015 and is due for completion to usable stage in August 2015. A key element in the planning has been establishing productive working relationships across the University between the Centre for Teaching and Learning and the Library primarily, but also the Information Technology section (for procurement of digital technology) and Facilities Management (construction and design work). The physical space As the vision for the playspace is to provide a, collaborative zone, free of expectation or judgment for faculty to engage with the technologies the design of the space is intentionally flexible and informal with no space intended for solitary work – similar to the Learning Studio at Abilene Christian University (Lemley, 2013). The Library is the most logical place on campus for a facility like this however space in the Library is at a premium for students already so decisions around how the space can be designed so that precious student space is not lost have been critical. As the modern library role is changing from store for printed materials to shared learning space, planning has required thought about different pedagogies and learning experiences (Bennett, 2015). Fit-out includes a range of informal, flexible seating options to allow for small group discussions and is designed for a very physical, hands on learning experience rather than traditional training delivery. It is expected that this approach will facilitate experiential learning and full engagement with the technology for the purposes of sparking re-imagination of teaching practice. (Cheers, Eng & Postle, 2012; Steel & Andrews, 2012). The playspace is located near the current information commons and group study areas, so is an active zone of the building. The space will be open when not in use for staff development so that students can use the informal and formal seating as group study areas, and ultimately use the technologies in the space. The technologies The technologies going in to the space initially are a range of mobile devices, a digital table and video wall. Mobile device selection initially focused on tablet devices in a wide range of physical sizes and with Android, iOS and Windows operating systems all represented. A large number also included a high-resolution stylus input option. Devices will be provisioned with a small number of teaching and learning "apps", use of which will be included in the formal training sessions. Physical management will be by way of a dedicated charging/storage unit. An important factor in device selection (including brand) was internal experience with the device and the confidence to be able to provide technical support for the device internally or through established informal networks. Other factors for device selection included known compatibility with university network infrastructure and support and availability through preferred suppliers. Ongoing maintenance and support for devices and any specialist or non-standard “apps” will be the responsibility of educational technology staff within the CTL. One of the key premises for the selection of what to put in the space initially was to create a zone that enables digitally mediated small-group collaboration and knowledge co-construction through physically shared digital workspaces and the digital table and video wall will allow for this. Content from devices can be shared to the wall on different screens or simultaneously over the whole wall (an advantage over the standard video projector option). Software initially installed on the table included modules to facilitate group brainstorming sessions, co-exploration of content and locations, and co- creation of presentations and free-form visual designs. CP:146 498 As of writing, plans are underway to purchase one or more virtual reality headsets and potential future additions include a 3D printer, wearables and gesture-based input-devices. The use and support of the space In the initial launch phase the space will be used by Centre for Teaching and Learning (CTL) staff to facilitate small group technology familiarisation sessions and to work with individuals on an as needed basis. The focus of such sessions will be on emphasising the importance of playing and experimentation with the technology to enable meaningful development of digital literacy skills in teachers. As the CTL staff are not physically situated in the Library these sessions will be pre-booked rather than drop in. As more staff become comfortable and familiar with the technology it is envisaged that they will use the space with their students and with other colleagues, and ultimately students themselves will engage with the technology without facilitation. The future As with all investments of this sort planning for the future is critical to the success of the playspace as technology can quickly become outdated and/or unsupported. It is envisaged that the facilities provided in the playspace will be evaluated and technology added/removed/updated as part of the annual planning process (for both budgetary and project management reasons). As procurement can be a prolonged process a long lead time needs to be factored in for adjustments to the technologies provided, especially for additions. This process will be managed by the Centre for Teaching and Learning. The playspace is being created on one of three geographically dispersed campuses so the success of the project will be carefully measured to determine whether similar facilities on other campuses are required. Challenges and Risks While providing access to technology is a barrier to adoption by academic staff, it is not the only one (Kirkwood, 2015; Reid, 2014) and there are significant risks with this project that unless the underlying support and engagement strategies are in place it will not have the desired effect – access alone is not sufficient to guarantee adoption. A key challenge is getting engagement from teachers to use the space. Significant promotion and marketing of the space will be essential but it is also important that the right messages are coming from the university management at the highest level (Reid, 2014). The integration of technology into teaching and learning needs to be seen as a key direction for the entire institution through using a “joined-up” approach (Kirkwood, 2015) and this space is a key part of the process. In the current higher education environment of financial constraint, it is critical that this space is seen to be of strategic importance to the entire institution and not an underutilised space filled with expensive “toys” for a limited group of staff. Support for the technologies is a challenge to be addressed, particularly for library staff. As the physical space will be open for student use when not booked there is concern that library staff will be asked to support the technologies when they are not confident themselves in the use of the technologies. Strategies to mitigate this risk range from running sessions for the library staff to develop the skills and knowledge they need, to locking the technologies down if needed. This is a challenge that will be best understood when the space is live and available and will be iteratively managed. As the purpose of the space is to expose teachers to emerging technologies there is a challenge in future-proofing the investment and ensuring that it remains current. Digital technologies are evolving at such a rapid pace that selecting which technologies are showcased is challenging and will call for consistent financial investment in the space. The NMC Horizon Report (Johnson, Becker, Estrada, & Freeman, 2014) is one source that will be used to plan for future investment. CP:147 499 The digital playspace is a significant investment for the university, both in financial cost and also in time and space. It will become critical to be able to measure and demonstrate the success of the project and this is a challenge when much of the learning taking place may be informal or latent. Use of the space itself will be one measure, and ultimately the increased integration of digital technologies into teaching practice across the institution should be the clearest indicator of success. Conclusion Envisaged as an innovative environment to lift faculty digital literacy through formal and informal exposure to current and new technology, the ultimate aim of the digital playspace project is to enable confident use of technology by teachers in the classroom and beyond. Key challenges will be getting engagement from faculty and ensuring the resource is sustainable from both a technology and human resource perspective. The full benefits (or otherwise) of this project are, of course, yet to be realised as the true measure of success can only be observed over time. References Bennett, S.(2015). Putting Learning into Library Planning. portal: Libraries and the Academy 15(2), 215-231. The Johns Hopkins University Press. Buchanan, T., Sainter, P., & Saunders, G. (2013). Factors affecting faculty use of learning technologies: Implications for models of technology adoption. Journal of Computing in Higher Education, 25(1), 1–11. Burghardt, G. M. (2012). Play, exploration and learning. (N. M. Seel, Ed.)Encyclopedia of the Sciences of Learning. Boston, MA: Springer US. Cheers, C., Eng, C. S., & Postle, G. (2012). Experiential Space. In M. Keppell, K. Souter, & M. Riddle (Eds.) Physical and Virtual Learning Spaces in Higher Education: Concepts for the Modern Learning Environment (pp. 266-277). Hershey, PA: . doi:10.4018/978-1-60960-114-0.ch016 Johnson, L., Becker, S., Estrada, V., & Freeman, A. (2014). Horizon Report: 2014 Higher Education. http://www.editlib.org/p/130341/report_130341.pdf Kirkwood, A. (2015). Teaching and learning with technology in higher education: blended and distance education needs “joined-up thinking” rather than technological determinism. Open Learning: The Journal of Open, Distance and e-Learning, (June 2015), 1–16. Lemley, L. (2013). Connected: Collaborative Solutions to Technological Challenges. In R. Carpenter (Ed.), Cases on Higher Education Spaces: Innovation, Collaboration, and Technology (pp. 366- 386). Hershey, PA: MacCallum, K., & Verhaart, M. (2014). Adoption of mobile devices by academic staff: the reality. In B. Hegarty, J. McDonald, & S.-K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014 (pp. 658– 662). http://ascilite.org/conferences/dunedin2014/proceedings/ Massey University (2014). Shaping the nation: Taking the best to the world - The Road to 2025 (Strategy Document). Massey University, Palmerston North, NZ http://www.massey.ac.nz/massey/fms/About%20Massey/University- Management/documents/156512%20Road%202025%20WEB.pdf?E16946928C096DBABD84BC 20911E8BEC Matthews, R., & Liu, C. H. (2012). Play and its role in learning. Encyclopedia of the Sciences of Learning. Springer US. Meyer, P. (2012). Embodied learning at work: Making the mind-set shift from workplace to playspace. New Directions for Adult and Continuing Education, (134), 25–32. Reid, P. (2014). Categories for barriers to adoption of instructional technologies. Education and Information Technologies, 19(2), 383–407. Schneckenberg, D. (2010). Overcoming barriers for eLearning in universities-portfolio models for eCompetence development of faculty. British Journal of Educational Technology, 41(6), 979–991. Steel, C., & Andrews, T. (2012). Re-Imagining Teaching for Technology-Enriched Learning Spaces: An Academic Development Model. In M. Keppell, K. Souter, & M. Riddle (Eds.) Physical and Virtual Learning Spaces in Higher Education: Concepts for the Modern Learning Environment (pp. 242-265). Hershey, PA: . doi:10.4018/978-1-60960-114-0.ch015 CP:148 500 Lamond, H. & Rowatt, A.J. (2015). A digital what? Creating a playspace to increase the quality of technology-enhanced teaching and learning. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:145-CP:149). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:149 501 The three pillars to building staff capability to create digital learning experiences Catherine Manning Hero Macdonald University of Melbourne University of Melbourne Many institutions are grappling with building staff capability in the complex task of designing and creating high-quality, technology-rich digital learning experiences informed by pedagogy. This paper provides an overview of a pilot program with two interactions implemented at the University of Melbourne called the Digital Learning Design (DLD) program. Focused on building Library’s organisational capability the program was built on three pillars of staff capability; deep knowledge of learning theory, learning design principles and skills in selecting digital technologies. The DLD design drew on research in change management, effective capability building as well as best practice in developing digital technology skills. Learners experienced the learning theories taught with the program design including the concepts of the flipped classroom, authentic learning and community of practice. This paper showcases an innovative and successful approach to addressing the issue of enduring staff capability to create digital learning experiences. Keywords: digital learning, capability building, staff professional development, global challenges in education, digitally enabled learning for a global society. Introduction This paper discusses the implementation of two iterations of a professional development (PD) program piloted at the University of Melbourne in 2014-2015 called the Digital Learning Design (DLD) program. The DLD focused on building the University Library’s organisational capability for designing digital learning resources. The program was used to facilitate the launch of a new model of liaison librarianship, including the creation of new roles specialising in learning and teaching, during a period of significant organisational change. The program was designed to build capability in the three pillars of knowledge needed to create digital learning resources; knowledge of learning theory, learning design principles and educational technologies. Evaluation of the program has shown its overall design to be highly effective. Literature review Staff frequently are often comfortable with student-centred pedagogies in a face to face context. However, this often does not correlate with student-centred practice online (Owens, 2012). This is exacerbated by the fact that digital learning PD commonly separates the teaching of learning theory, learning design principles and technology skills into different programs. Staff are expected to coherently meld this disjointed PD when creating digital resources. Staff PD needs to address both the technical and pedagogical needs coherently in order to truly capitalise on the affordances of educational technologies (Owens, 2012). Staff PD on digital technologies is often characterised by fragmented, short, technology-focused workshops. These workshops do not address the ongoing pedagogical change needs of educators to teach with technology. Longer programs with follow-up that explicitly teach pedagogical practice is needed to reform teaching practice (Fullan, 2009, Lawless & Pellegrino, 2007). PD also needs to include authentic learning opportunities, peer learning and consideration of the working context (Macdonald & Poniatowska, 2011). Often staff have no experienced learning in a digital environment, despite being requested to design for this platform. Authentic experience of digital learning as a learner is needed to deepen their understanding of online pedagogies. This could include learning through implementing real projects or authentic engagement with digital learning (Macdonald & Poniatowska, 2011). CP:150 502 Furthermore, staff PD in digital technology is commonly designed with didactic teaching practices, rather than the student-centred pedagogical practice staff need to draw upon when implementing technology. Student-centred learning practices include ‘flipped classroom’ learning design, peer review activities, active learning, authentic learning and project based learning - all characteristics of constructivist and inquiry based learning approaches (Aditomo, Goodyear, Bliuc & Ellis, 2013). Effective staff PD design should model the student-centred practices that staff are requested to implement, as authentic experience of these pedagogies deepens learning engagement with them (Matzen & Edmunds (2007). Our understanding of how to best use technology to effectively impact on students outcomes is rapidly changing and is constantly being tested and redefined just as the digital technologies themselves are rapidly updating. For staff to adapt to this changing understanding ,Wegner’s (1998) ‘community of practice’ approach allows staff to discuss and create meaning to these changes (Armfiled, 2011). Program Learning Design The DLD’s design was underpinned by three pillars of knowledge; developing capability for learning theory, for learning design, and to support the effective use of digital technology. A key component, and deliberate pedagogical strategy, of the DLD was the organization of participants into small teams to work on authentic digital learning projects. Most teams were developing digital learning resources for particular subjects that they were already supporting, converting face to face delivery into the online environment. To support the three pillars, the learning objectives for the program were that participants would: • Have a foundational understanding of selected pedagogical approaches to blended and online learning • Have the skills to design an online digital resource using learning design principles • Know how and when a variety of technology tools might be used to support online teaching and learning • Have designed, developed and delivered an online or blended digital learning resource The DLD program was delivered over a five month period and included five face-to-face workshops, online learning modules, a peer review process of learning design plans (LDPs) and the completion of the authentic learning project. The program’s learning design and activities modeled the learning theory and pedagogy that the program was asking staff to adopt. The DLD was designed using the concept of the 'flipped classroom' with participants engaging with the didactic teaching aspects of the program via the online self-paced modules. The online modules introduced participants to core educational theories, learning design principles and frameworks for selecting technology tools. The five face-to-face workshops were designed on the constructivist principles of active learning. Key concepts were discussed and learning was applied through the creation of the LDPs and final learning resources for each project. Participants peer reviewed each other LDPs using guiding questions that reinforced the theory. As part of the program evaluation, experts analysed both the peer reviews and LDPs to asses the quality of engagement with core concepts. Community was further encouraged via a Yammer group designed to support skill development and the knowledge of digital technologies. After the final workshop, there was a two month period where project teams developed their digital resources. This period concluded with a ‘Showcase’ event where each project presented their completed digital learning resources. Program Evaluation Kirkpatrick’s (1998) Four Levels of Evaluation model was used to design the program evaluation. Kirkpatrick’s model asserts that evaluation of PD requires analysis of: participant reactions, learning, behaviours and results. Reactions were measured through post-program surveys and focus groups (pre and post), evidence of learning was found in the focus groups (pre and post) as well as expert analysis of peer review feedback, behaviors were evaluated via expert review of LDPs and expert CP:151 503 evaluation of final products. Due to the timeframes associated with the program and publication, no formal data has been gathered on the ‘Results’ level. This evaluation draws on a mix of qualitative and quantitative data, including: Focus Groups: Two rounds of focus groups with participants were conducted; before the program commenced (n=7) and at the conclusion of the formal teaching component of the program (n=4). Focus groups were used to explore changes in participants’ attitudes to teaching, learning design and the extent to which knowledge of pedagogies and learning theory informed their teaching. Thematic analysis was used to code data. Post-Program Surveys: Post-program surveys were used to obtain feedback from participants on the perceived strengths and weaknesses of the course content and design. Analysis of Participant Peer Review Feedback: Feedback from the peer review activity was analysed against the LDPs by the two university learning designers (LDs). A rubric was used to analyse the feedback on each of the peer review questions, rating the participants’ engagement with course concepts, their use of appropriate concepts and terminology and their ability to apply knowledge of learning design principles to critique the LDPs. Expert Review of Learning Design Plans and Digital Learning Resources: Expert review was also sought from the university LDs on the completed LDPs and the final digital learning resources. The LDs used the same set of peer review questions to identify areas of strength and weakness in the plans and rate the overall quality of the learning designs. This feedback, discussed in the Program Outcomes section, was used to both evaluate the program and provide feedback to participants about their learning and the quality of their learning designs. Discussion Reactions The focus groups and post-program surveys revealed strong reactions from participants about the content and structure of the course. 70% of pre-program survey respondents identified the program’s blended format and face to face workshops as being a key strength. Roughly 60% identified the project-based approach to learning as another key strength, while 50% mentioned the collaborative, collegial nature of the program. One survey respondent identified: “The opportunity to work on a practical project, to gain technical skills, to collaborate and share ideas, skills and perspectives with colleagues, and to learn from each other” as a highlight – emblematic of responses across the cohort. Similar sentiments were made by three of the four participants at the post-program focus group, where the group agreed that the workshops and application of learning to a real project facilitated the best learning. Learning In terms of learning, both the pre-program surveys and focus groups suggested that most staff commenced the course with very limited knowledge of learning theory and learning design principles. The post-program focus group participants unanimously agreed that their learning had progressed significantly during the course and that the course had achieved its intended learning outcomes. The expert ratings of peer review feedback and LDPs provided convincing evidence of this learning. Figure 1 below illustrates strong agreement between the expert reviewers and the peer reviewers, suggesting that participants have acquired the expected skills and knowledge of core educational and learning design concepts. Behaviour Expert ratings of the LDPs shows that participants have been able to successfully apply their learning to the production of new, high quality, pedagogically and technologically sound digital learning resources. This review rated 70% of the LDPs as proficient, and 20% at an 'excellent' standard. The expert reviews looked at the application of core course content including key learning design principles and the suitability of technologies. CP:152 504 Figure 1: Expert Review of Peer Review Feedback Program Outcomes The DLD has achieved a measurable increase in the Library’s capacity to produce high-quality, pedagogically-sound and curriculum-based digital resources as evidenced by both participant feedback and evaluation of the digital learning resources created during the program. The DLD project has demonstrated the effectiveness of an authentic, blended, research-based approach to PD. Outcomes for the DLD included: 1. An effective program designed in collaboration with two university groups and based in research 2. Creation of student-centred digital resources that focus on developing students' scholarly literacy skills 3. Design of digital learning resources informed by research and learning theory 4. Learning design and digital objectives created through partnerships with the subject coordinators in faculties 5. Team building in the library with the thirty librarians involved in each iteration 6. The successful implementation of a new model of library liaison References Aditomo, A., Goodyear, P., Bliuc, A.-M., & Ellis, R. A. (2013). Inquiry-based learning in higher education: principal forms, educational objectives, and disciplinary variations. Studies in Higher Education, 38(9), 1239-1258. doi:10.1080/03075079.2011.616584 Armfield, S. W. J. (2011). Technology leadership for school improvement Planning, designing, implementing and evaluating technology. In R. Papa (Ed.), Technology and Leadership for School Improvement (pp. 109-128). California: Sage. Fullan, M. (2009). Large-scale reform comes of age. Journal of Educational Change, 10(2/3), 101- 113. doi: 10.1007/s10833-009-91(08-z Lawless, K. A., & Pellegrino, J. W. (2007). Professional Development in Integrating Technology Into Teaching and Learning: Knowns, Unknowns, and Ways to Pursue Better Questions and Answers. Review of Educational Research, 77(4), 575-614. doi: 10.3102/0034654307309921 Kirkpatrick, D. L. (1998). The four levels of evaluation. In Evaluating corporate training: Models and issues (pp. 95-112). Springer Netherlands. Macdonald, J., & Poniatowska, B. (2011). Designing the professional development of staff for teaching online: an OU (UK) case study. Distance Education, 32(1), 119-134. doi: 10.1080/01587919.2011.565481 Matzen, N. J., & Edmunds, J. A. (2007). Technology as a Catalyst for Change: The Role of Professional Development. Journal of Research on Technology in Education (International Society for Technology in Education), 39(4), 417-430. CP:153 505 Owens, T. (2012). Hitting the nail on the head: the importance of specific staff development for effective blended learning. Innovations in Education & Teaching International, 49(4), 389-400. doi:10.1080/14703297.2012.728877 Wenger, E. (1998). Communities of Practice: Learning, Meaning and Identity. Cambridge: Cambridge University. Manning, C. & Macdonald. H. (2015). The three pillars to building staff capability to create digital learning experiences (2015). In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:150- CP:154). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:154 506 Developing Self-Regulated Learning through Reflection on Learning Analytics in Online Learning Environments Alexander Mikroyannidis Tracie Marie Farrell Frey The Open University, Milton Keynes, The Open University, Milton United Kingdom Keynes, United Kingdom This paper describes a conceptual framework for developing self-regulated learning through facilitated dialogue and reflection on learner activity in online learning environments. In particular, the framework focuses on the motivational and contextual aspects of self-regulated learning and how the field of learning analytics can support student metacognitive knowledge in these two areas and distribute instructional support. Keywords: learning analytics, self-regulated learning, critical pedagogy, inclusion Introduction A contemporary challenge of online learning is to create an environment in which a potentially highly diverse cohort of learners can be stimulated to interact with each other and engage in a meaningful learning process in a self-regulated way. This challenge is underlined by two observations: firstly, that growing class size has created a "teacher bandwidth problem" (Wiley & Edwards, 2002) and secondly, that the focus on learner autonomy and self-regulation in online learning may be leaving some learners behind (Beetham & Sharpe, 2013; Fruhmann, Nussbaumer & Albert, 2010). Both researchers and practitioners acknowledge the necessity to extend instructional support and scaffold the learning process in order to face these challenges. The primary question with which this paper is concerned is: How can students who do not currently possess the necessary skills in self-regulated learning utilise the structures and opportunities of online learning to develop those skills? The field of learning analytics presents new opportunities for "understanding and optimizing learning", through collecting, analysing and reporting upon valuable data about learner activity in online learning environments (SoLAR, 2015). This paper proposes a conceptual framework for online education (referred to as "Uplift") that aims to deal with certain challenges of self-regulated learning by investigating the affordances of learning analytics for building skills in self-regulation. Affordances of Learning Analytics for Self-regulated Learning Self-Regulated Learning (SRL) is broadly described as the ability to understand one's own learning processes and manipulate them (Schunk & Zimmerman, 2012). Common components of SRL models include four main areas of regulation (cognition, behaviour, motivation and context) across four broadly cyclic phases of learning (planning, execution, monitoring and evaluation) (Winne et al, 2000). For each area and each phase, there are strategies that learners can deploy to understand and control how they learn. Many tools for self-regulated learning are already available to help learners plan, set goals, map their activities and track their progress (Nussbaumer, Dahn, Kroop, Mikroyannidis & Albert, 2013). However, while these tools make it possible to practice self-regulated learning, they do not always help learners to acquire self-regulated learning skills (Beetham & Sharpe, 2013). Learning Analytics can support this process by providing valuable data to students and teachers that enhance metacognitive gains in certain areas. Context in Learning Contextual factors such as educational background, race, class and gender, for example, have been shown to affect the physical classroom experience. Perceived lack of representation can adversely affect motivation (Egalite, Kisida & Winters, 2013) and general participation (White, 2011), while hegemonic classroom dynamics can even entirely exclude learners from non-normative backgrounds (McLaren, 2003). In online learning, learning analytics can help uncover the impact of these factors and others, creating a picture of the learner in context and connecting learner profiles with learner CP:155 507 activity and outcomes. This information can then be utilised by learners to help gain metacognitive knowledge about their own learning experience. Interaction and Motivation in Learning Interaction is one of the means by which instructors attempt to keep learners engaged and motivated (Dabbagh & Kitsantas, 2012). However, the quality of interactions and the types of cognitive and emotional responses they elicit is what dictates the extent to which learning is positively impacted (Picciano, 2002). A quality interaction is one that improves self-knowledge about (meta) cognitive, behavioural, motivational and contextual experiences in learning (Hadwin & Oshige, 2011; Schunk, 1989). Learning analytics can help to identify and understand the nature of quality interactions, which has the potential not only for developing self-regulated learning skills, but also for influencing instructional design. Facilitation of Learning Research indicates that instructors play a vital role in facilitating meaningful learning online and in the development of self-regulated learning (Boyer, Maher & Kirkman, 2006). Instructors model strategies, moderate dialogue and track the engagement of students, all of which becomes more difficult as class sizes increase. Learning analytics can already support instructors through collecting baseline activity, identifying at-risk learners and (in some cases) recommending solutions (Arnold & Pistilli, 2012; SoLAR, 2015). Involving students in the process of interpreting learning analytics may provide an avenue for activating motivation and distributing instructional support (Sclater, 2015), as well as sharing the process of self-regulation among students and instructors (Hadwin & Oshige, 2011). The "Uplift" Framework "Uplift" is a conceptual framework that describes the affordances of learning analytics to support the development of self-regulated learning skills. Figure 1 illustrates a simple model of self-regulated learning based on Zimmerman (1990), overlaid with types of learning analytics that could be beneficial at each phase. The cycle of self-regulated learning begins at forethought and planning, where "Contextual Learning Analytics" (learner background, previous experiences, demographic data, etc.) can identify certain features that appear to influence learning, so that these can be acknowledged appropriately. The monitoring and control phase of the self-regulated learning cycle can be supported through "Performance and Behavioural Learning Analytics", to help the learner understand the connection between certain strategies and their learning outcomes. In the final phase of evaluation and reflection, learning analytics that support reflection, such as prediction-based analytics, trends and norms, can help learners to identify areas in which they need further support. This knowledge can be brought into the next cycle of self-regulation and also inform the next iteration of collecting, analysing and reporting on learner activities, raising the utility of learning analytics for self-regulated learning over time. Figure 1 - Learning Analytics Across Phases of Self-Regulation CP:156 508 Technological Structure The underlying system of Uplift will be an online learning platform collecting rich profile data on users ("contextual analytics"), married with complex capabilities in analysis of learner activities ("performance analytics", see fig 1). Learner activities will be collected through trace analysis, similarly to the software nStudy, developed at Simon Fraser University (Winne, 2015), as well as manually collected self-assessment of individual and group motivation through emotional proxies and latent variable modelling. On top of these capabilities, a variety of Web 2.0 features will be available to learners to engage with content and with each other, including rating systems, dialogue pages and comments. Finally, some simple, open source tools such as the Python Natural Language Toolkit and the R Text Mining Module will be adapted for use in basic sentiment analysis. This information will be used to track tendencies in participation, interpersonal relationships, knowledge and cognitive ability, motivation and environment, which can be expressed in the form of classroom learning analytics and used as relevant data for self-regulated learning. Pedagogical Structure The pedagogical companion to the system is its most unique aspect, in terms of the state-of-the-art. It involves delivering the data generated by the Uplift system back into the hands of students as a part of the regular classroom structure and learning goals. The data will be examined with students through facilitated dialogue based upon reflection protocols adapted from critical pedagogy (McLaren, 2003) and transformative learning (Mezirow, 1990), two educational traditions that place considerable emphasis on empowering students. The aim of the reflection is to illuminate the relationships between students and each other, students and instructors, and students and content, which impact the construction of knowledge and accessibility of education. Though they are not fool-proof, analytics can provide some general cues for beginning discussions about the contextualized learning experience, which can help target interventions and improve retention strategies. Opportunities and limitations Opportunities The structure of Uplift is intended to provide enough granularity and qualitative insight to consider self-regulated learning as both and event and an aptitude, which helps to forward the state-of-the-art (Winne & Perry, 2000) and combine approaches toward learning analytics from both educational data mining (EDM) and learning analytics & knowledge (LAK) perspectives (Siemens & Baker, 2012). Moreover, the collaborative inquiry aspect of the pedagogical component supports "socially shared regulated learning" (Hadwin & Oshige, 2011), which encourages learners to discuss learning experiences, model successful strategies and develop good practices for self-regulation over time. The dynamic nature of the information that Uplift collects, makes it a continuous source of new knowledge about oneself and others, improving social presence and motivation for participation for all learners, even if they are already skilled (Hadwin & Oshige, 2011). Uplift provides both teachers and students with many more data points to consider not only the efficacy of certain strategies, but also the possible reasons behind successes or failures in learning, addressing the "teacher bandwidth problem" (Wiley & Edwards, 2002), as well as some of the contextual features of education (race, class and gender) that may advantage some students over others. The overall effect of such an approach, richly reflecting with learners on dynamic aspects of their learning experience and highlighting the gaps in their knowledge, is expected to improve self-awareness and self-regulated learning more sustainably. Limitations Though the framework provides many opportunities for triggering learner curiosity and motivation to participate, the amount of data could also be over-stimulating for learners (Arnold & Pistilli, 2012) or take attention away from actual domain related content of a learning experience. However, the intention is that, over time, it would be possible to collect data on the types of analytics or facilitated dialogues that produce the most sustained, quality interactions, so that learners are not distracted by CP:157 509 superficial data. Another significant limitation is that Uplift relies heavily on the strong critical thinking and facilitation skills on part of the instructor to make sense of the vast data that will be possible to collect and analyse. One mechanism that can minimize the effects of this limitation is the distribution of analysis across the whole classroom. As students and instructors are equally encouraged to review and comment on data, it will be possible to uncover more insights (and more diverse insights) from the data. Conclusion Online education has made it possible for growing numbers of students from all over the world to participate in learning together. As the diversity and size of the classroom increases, it is necessary to ensure that the quality and accessibility of education are maintained. Uplift, as a framework, aims to leverage the unique qualities of online education, namely that it is possible to track the activities of learners in finer detail and make them transparent, to address those challenges and make learning analytics work more directly on behalf of students. Connecting learning analytics with the cycle of self- regulation can help instructors gain a more intimate picture of the cognitive, emotional and social life of their students. Likewise, delivering learning analytics into the hands of learners in meaningful ways can provoke curiosity, internal motivation and participation by giving learners a sense for the dynamic nature of their own learning process. Moreover, sharing the responsibility for drawing insights from these analytics distributes instructional support, builds rapport and presents a learning opportunity for both instructors and students (Maor, 2008). With more deep, social, contextualized information about all four areas of self-regulated learning in online environments available to students and instructors, the development, not only the practice, of SRL is achievable. References Arnold, K. E., & Pistilli, M. D. (2012, April). Course signals at Purdue: using learning analytics to increase student success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 267-270). ACM. Beetham, H., & Sharpe, R. (Eds.). (2013). Rethinking Pedagogy for a Digital Age: Designing for 21st Century Learning. Routledge. Boyer, N. R., Maher, P. A., & Kirkman, S. (2006). Transformative Learning in Online Settings: The Use of Self-Direction, Metacognition, and Collaborative Learning. Journal of Transformative Education, 4(4), 335-361. Dabbagh, N., & Kitsantas, A. (2012). Personal Learning Environments, social media, and self- regulated learning: A natural formula for connecting formal and informal learning. The Internet and higher education, 15(1), 3-8. Egalite, A. J., Kisida, B., & Winters, M. A. (2013). Representation in the Classroom: The Effect of Own-Race/Ethnicity Teacher Assignment on Student Achievement. Pridobljeno, 1(9), 2014. Fruhmann, K., Nussbaumer, A., & Albert, D. (2010, July). A psycho-pedagogical framework for self- regulated learning in a responsive open learning environment. In Proceedings of the International Conference eLearning Baltics Science (eLBa Science 2010) (pp. 1-2). Fraunhofer. Hadwin, A., & Oshige, M. (2011). Self-regulation, coregulation, and socially shared regulation: Exploring perspectives of social in self-regulated learning theory. Teachers College Record, 113(2), 240-264. Maor, D. (2008). Changing relationship: Who is the learner and who is the teacher in the online educational landscape?. Australasian Journal of Educational Technology, 24(5). McLaren, P. (2003). Critical pedagogy: A look at the major concepts. The critical pedagogy reader, 69-96. Mezirow, J. (1990). How critical reflection triggers transformative learning. Fostering critical reflection in adulthood, 1-20. Nussbaumer, A., Dahn, I., Kroop, S., Mikroyannidis, A., & Albert, D. (2015). Supporting Self- Regulated Learning. In Responsive Open Learning Environments (pp. 17-48). Springer International Publishing. Picciano, A. G. (2002). Beyond student perceptions: Issues of interaction, presence, and performance in an online course. Journal of Asynchronous learning networks, 6(1), 21-40. Society for Learning Analytics Research (SoLAR). (2015). Society for Learning Analytics Research (SoLAR). Retrieved 1 October 2015, from http://solaresearch.org/ Sclater, N. (2015) Student app for learning analytics: functionality and wireframes. Retrieved 15 September 2015, from http://analytics.jiscinvolve.org/wp/2015/08/21/student-app-for-learning- CP:158 510 analytics-functionality-and-wireframes/ Schunk, D. H. (1989). Social cognitive theory and self-regulated learning. Self-regulated learning and academic achievement (pp. 83-110). Springer New York. Schunk, D. H., & Zimmerman, B. J. (Eds.). (2012). Motivation and self-regulated learning: Theory, research, and applications. Routledge. Siemens, G., & d Baker, R. S. (2012, April). Learning analytics and educational data mining: towards communication and collaboration. In Proceedings of the 2nd international conference on learning analytics and knowledge (pp. 252-254). ACM. Wiley, D., & Edwards, E. (2002). Online self-organizing social systems. Quarterly Review of Distance Education, 3(1), 33. Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In Handbook of Self- Regulation, M. Boekaerts, P. Pintrich, and M. Zeidner, Eds., pp. 531–566, Academic Press, Orlando, Fla, USA. Winne, Philip H. (2015). Leveraging Big Data to Help Each Learner Upgrade Learning and Accelerate Learning Science. Manuscript in preparation. Winne, Philip H.; Perry, Nancy E.; Boekaerts, Monique (Ed); Pintrich, Paul R. (Ed); Zeidner, Moshe (Ed), (2000). Handbook of self-regulation. (p. 531-566). San Diego, CA, US: Academic Press. White, J. W. (2011). Resistance to classroom participation: Minority students, academic discourse, cultural conflicts, and issues of representation in whole class discussions. Journal of Language, Identity & Education, 10(4), 250-265. Zimmerman, B. J. (1990). Self-regulated learning and academic achievement: An overview. Educational psychologist, 25(1), 3-17. Mikroyannidis, A. & Frey, T.M.F. (2015). Developing Self-Regulated Learning through Reflection on Learning Analytics in Online Learning Environments. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:155-CP:159). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:159 511 Personalising professional learning mobility in Higher Education Maxine Mitchell Caroline Cottman University of the Sunshine Coast University of the Sunshine Coast The trends and impacts of digital technologies in the higher education sector mean that change is an ongoing, organic factor in response to the personalised nature in which society works, learns, lives, communicates, and connects. Such dynamic educational settings provide new environments for learning mobility that transcend boundaries of time, place, convention and learning community. This paper is fundamentally concerned with how educators, as adult learners, learn in a time when institutions, through their teaching staff, are attempting to address the fast pace innovations in learning and teaching. This paper describes a regional university’s approach to reconceptualising a model of professional learning that offers personalised, collaborative, and transformative learning experiences for its educators. The aim is to develop professional learning initiatives that are responsive to the educator’s learning mobility needs whilst also enriching the student learning experience and addressing institutional strategic priorities. Keywords: learning mobility, professional learning, digital technologies Introduction This paper is fundamentally concerned with how educators, as adult learners, learn. To give due attention to this, the investigation is approached from the higher education sector’s ability to respond to the socio-technical forces of change at the institutional level, faculty level and individual level. For the purpose of this paper the institutional level is conceived as the macro-level consisting of high level external forces such as the institution (e.g. policy, strategic direction, organisational structure), the sector (e.g. deregulation, government funding), and global factors (e.g. increased competition from non-university higher education providers, and globalisation and casualization of the academic workforce). The faculty level is conceived as the meso-level consisting of external forces such as the faculty, discipline and community. The individual- level is conceived as the micro-level consisting of the inner forces that drive or limit the individual’s motivation to engage in opportunities to change. The individual in this context is the higher education teacher, and their motivation to engage in change is concerned with those forces that drive or limit ways to deepen their understanding of their teaching practice. In addition to the use of the micro-, meso- and macro-level framework to examine the complexities of change in higher education, the paper also introduces the idea of the educator’s learning mobility. For the purposes of this discussion, learning mobility relates to people choosing to learn, work, communicate, collaborate and connect in any configuration, across learning contexts and boundaries, for continuous professional and personal growth, rather than the scholarly discourse on mobile learning and digital technologies. The paper explores the changing nature of higher education at the macro-, meso- and micro-levels, and the interconnectedness between the levels, through the lens of academic work and professional learning. Surfacing the complexities of change that may drive or limit the educator’s learning mobility across the three levels is applied to describe a regional university’s approach to reconceptualising a CP:160 512 model of professional learning that offers personalised, collaborative, and transformative learning experiences for its educators. Academic work: The complexities of change The essence of higher education academic work has been captured by Debowski (2012) as “one of the most rewarding yet frustrating and challenging roles anyone could undertake. It is complex, dynamic and rapidly evolving to accommodate the expectations of its many stakeholders” ( p. 3). As work gets more complex and informal learning emerges as an essential part of work, Boud and Brew (2012) emphasise a pragmatic approach where learning is viewed as a social process occurring within the context of practice which, in turn, leads to a fundamental shift to the perspective of academic work as professional learning. Macro-level and meso-level: The macro- and meso-levels have been combined for the purposes of this paper as the focus at this point is on the structures, conditions, and functions existing at the institutional and faculty levels embedded as top-down control that limit the ability organisationally to embrace change. For reconceptualised models of professional learning to have lasting impact, academics must feel confident, have a sense of control over their work, and professional self-efficacy and identity to assume personal responsibility in advancing their academic practice (Martin, McGill, & Sudweeks, 2013). This suggests attention needs to be given to the hierarchical structures at the macro- and meso-level when reconceptualising a whole-of-institution approach that fosters a bottom-up attitude where educators have a sense of agency in remodelling professional learning initiatives. Micro level: Learning mobility is much less concerned about specific structures, hierarchies, tasks and place. It shifts the fluidity of academic work to the activity of doing, being and acting in the world (crafting a sense of meaning and academic identity) that is not fixed by time, place and convention. A re- distribution of the function of academic work across networks, communities, and conversations shifts the responsibility to educator to personalise their own scholarly trajectory (Jewitt, 2009). In summary, learning mobility plays an essential part in the changing nature of workplace learning to enable continuity across the boundaries of time, space and the activity of learning (Jarche, 2012a, 2013b). Learning mobility advocates the invisible nature of workplace learning. There are three particular elements of learning mobility that underwrite the notion of invisible learning yet provide links back to the foundations of workplace learning. Firstly, learning can be formal or informal; secondly, knowledge can be explicit or implicit and finally, value can be tangible or intangible (Jarche, 2013a). Professional learning: The complexities of change The expectation today is that modern university teachers fully utilise the capacity of digital technologies to design engaging, authentic and personalised learning activities to enrich the educational experience (Phillips, McNaught, & Kennedy, 2011). We support the view that teachers are the single most important learning resource available to most higher education students (Villar & Alegre, 2007) and the heartbeat of the institution (Debowski, 2012). Given that educators are the pivot point at the macro-, meso-, and micro-level, a holistic approach is needed to rethink institutional- led professional development to design personalised, collaborative and transformative learning experiences for educators as part of their continuous professional learning (Boud & Brew, 2012; King, 2005). Macro-level: As social, informal learning has become an important driver for academic work it offers new professional learning opportunities (de Laat & Schreurs, 2013). The challenge is two-fold at the macro-level: institutions can no longer leave the responsibility for the educator’s engagement in their professional learning with their professional development department; and institutional structures need to adopt a wider, pragmatic, agile approach to professional learning practices to optimise the potential for individual and organisational learning (de Laat & Schreurs, 2013; Jarche, 2012a). CP:161 513 Meso-level: The meso-level, often referred to by educators as their ‘academic home’(Poole, 2009), can act as positive spaces to foster identity, opportunities for mutual support and collaboration, and generative sources of ideas; or they could serve to limit perspective and defend territories, and operate as places of resistance to change (Poole, 2009). The traditions of academe continue to challenge the value of university teaching as higher education teachers are ‘trained’ through the doctoral route and rewarded for research within their discipline field (Bates, 2015). There is no requirement to be qualified in teaching methods, or to engage in the learning and teaching discourse (Weimer, 2012). This creates tension when modern academic work requires highly qualified educators who are adaptive and responsive to discipline and pedagogical knowledge and skills in a changing higher education landscape. Micro-level: Steel (2004) concludes that many educators experience barriers that negate a sense of academic identity and institutional support to integrate innovations into their teaching practice. The barriers include time constraints, lack of resources, lack of understanding of educational theory and concepts, lack of knowledge of what is pedagogically possible, and lack of valuing learning and teaching. Furthermore, educators often view the traditions of institutionally-led professional development events as linear, out-of-step with personal needs, with limited effectiveness or relevance to immediate application to resolve their teaching challenge (Mitchell, 2015) rendering the event as ineffective or unappealing (Hart, 2015). In summary, for models of professional learning to have traction, the focus must firmly shift to the educator as adult learner. This refocus recognises that educators come with their own unique set of experiences, background and intentions, and emphasise the need for learning flexibility and mobility, where educators can personalise their own learning within, between and outside the traditions of professional development. An Institutional approach: Reconceptualising professional learning The regional university is in the early-phase of reconceptualising a model of professional learning from the perspective of an educator’s learning mobility. The revitalised model advocates that educators like to learn, work, communicate, collaborate and connect in any configuration, across learning contexts and boundaries, for continuous professional and personal growth. One of the biggest challenges to mainstreaming a personalising professional learning mobility mindset across the macro- (institution), meso- (faculty/school) and micro-levels (individual) is making it visible to the point that it advances professional recognition of university teaching. Designing a professional learning model that fosters an educator’s learning mobility enables individuals to take responsibility and control for their own learning and offers unlimited access to resources through personal learning networks (Jarche, 2012b; Kolowich, 2014). The networks become the learning and this goes to the core in the ways people like to work in a mobile society: in the flow of work; continuously; immediately; socially; and autonomously (Hart, 2013). However, such professional learning activities are mostly implicit, ad hoc, spontaneous, and invisible to others (de Laat & Schreurs, 2013). The development of the model acknowledges a whole-of-institution approach that fosters a bottom-up attitude to make professional learning visible. The model espouses a modularized approach, that is, scaffolded ‘learning chunks’. A modularized approach offers the advantage of the flexibility of choice as to which ‘bundle’ of professional learning activities educators engage in to meet their professional learning needs whilst addressing the collective needs of the institution by: • a systematic recognition of professional learning events completed within and outside the university that may be formal (e.g., completion of formal qualification within the domain of learning and teaching in higher education) or informal (e.g., active participation and collaboration within informal networks and communities to advance learning and teaching); an institutional initiative to advance the quality of university teaching (e.g. Foundations of University Teaching course, Peer Assisted Teaching Scheme (PATS)) or independent, self- discovery (e.g. participation in open access, collaborative and reflective learning activities to demonstrate advancement in teaching practice); • a credentialing mechanism to demonstrate achievement of professional teaching standards that: o Enables educators to build their academic career and professional identity through the CP:162 514 recognition of their teaching in a transparent, seamless and supported way; o Enables the institution to reconceptualise a professional learning model that cultivates a quality, transparent approach to continuously advance the quality and standard of university teaching; and o Offers a whole-of-institution approach to make visible to students and other stakeholders the professionalism that the institution and its staff bring to teaching and support for student learning. • a framework for sharing and collaborating across disciplinary thresholds of knowledge that enriches professional learning initiatives and creates opportunities for research partnership and networks that advances both the scholarship of learning and teaching, and disciplinary research (Weller, 2011); • a distributed teaching leadership model. Through shared and active engagement, strengthened by a sense of professional self-efficacy, distributed leadership builds a culture of respect and acceptance of change resulting in the development of leadership capacity to sustain improvement in the quality of learning and teaching (Office for Learning and Teaching Project, 2005). Next Stages The university is moving forward in developing a whole-of-institution approach to the design of a professional learning model that fosters a supportive environment that makes the educator’s learning mobility visible. The first stage is to advance the vision of change within the realms of the institutional level, faculty level, and possibly and most importantly, within the mindset of the individual. The bottom-up aspect of the model is fundamentally situated within the tenet of educator as adult learner and therefore asks the educator to take responsibility and being in control of their professional learning. From the authors’ experience in facilitating professional learning activities we know it to be true that educators have an intrinsic motivation to take ownership of their learning. The top-down aspect of the model is in shifting the locus of control in the constitution of professional learning events. Faculty level and institutional level structures need to provide pathways for, and recognition of, experience and expertise in university teaching. Currently the university is exploring partnership arrangements with other institutions driving sectoral change relating to professional recognition of university teaching, such as the UK’s Higher Education Academy (HEA) Fellowship scheme. This, in turn, creates platforms for educators to shape, choose, direct and take responsibility for their learning. The visibility of the educator’s learning mobility rewards the individual’s true sense of meaning making and identity which manifests as rewarding autonomy, mastery and purpose, leading to the engagement in professional practice (Pink, 2011). Creating an attitude of change that embraces a whole-of-institution mindset in partnership with a bottom-up approach to personalising professional learning means giving equal and due attention to academic work and professional learning activities that reward the educator’s learning mobility where the value may be explicit or implicit, formal or informal, tangible or intangible at the macro-, meso- and micro-levels. This paper recognises that higher education institutions serve many masters. Therefore the ongoing work in this area is to develop professional learning initiatives that are responsive to the educator’s learning mobility needs whilst also enriching the student learning experience, addressing institutional strategic priorities, and accommodating the expectations of its many stakeholders. References Bates, T. (2015). Teaching in a digital age. online: BC open textbooks. Retrieved from http://opentextbc.ca/teachinginadigitalage/ Boud, D., & Brew, A. (2012). Reconceptualising academic work as professional practice: Implications for academic development. International Journal for Academic Development, iFirst article, 1–14. http://doi.org/10.1080/1360144X.2012.671771 Debowski, S. (2012). The new academic: A strategic handbook. Berkshire, UK: McGraw Open University Press. CP:163 515 de Laat, M., & Schreurs, B. (2013). Visualising informal professional development networks: Building a case for learning analytics in the workplace. American Behavioral Scientist, online first version, 1–18. http://doi.org/10.1177/0002764213479364 Hart, J. (2013, April 25). 5 Characteristics of how knowledge workers like to learn at work. Retrieved from http://www.c4lpt.co.uk/blog/2013/04/25/5-characteristics/ Hart, J. (2015). Overview of modern workplace learning. Retrieved April 22, 2015, from http://janehart.com/books/modern-workplace-learning/overview-of-modern-workplace-learning/ Jarche, H. (2012a, June 17). Work is learning and learning is the work. Retrieved from http://www.jarche.com/2012/06/work-is-learning-and-learning-is-the-work/ Jarche, H. (2012b, July 12). PKM as pre-curation. Retrieved from http://www.jarche.com/2012/07/pkm-as-pre-curation/ Jarche, H. (2013a, May 21). Networked individuals trump organizations. Retrieved from http://www.jarche.com/2013/05/networked-individuals-trump-organizations/ Jarche, H. (2013b, August 7). A mobile workforce needs better on-site conversations. Retrieved from http://www.jarche.com/2013/08/a-mobile-workforce-needs-better-on-site-conversations/ Jewitt, C. (2009). Knowledge, creativity and communication. Bristol, U.K: FutureLab. Retrieved from http://www.beyondcurrenthorizons.org.uk/summative-report-knowledge-creativity-and- communication/ King, K. (2005). Bringing transformative learning to life. Florida, USA: Krieger Publishing Company. Kolowich, S. (2014, January 13). George Siemens gets connected. The Chronicle of Higher Education. Retrieved from http://chronicle.com/article/George-Siemens-Gets-Connected/143959/ Martin, R., McGill, T., & Sudweeks, F. (2013). Learning anywhere, anytime: Student motivators for M- learning. Journal of Information Technology Education: Research, 12(11), 51–67. Mitchell, M. (2015). Learning mobility in professional practice: Transforming workplace learning in higher education. (Unpublished doctoral thesis) University of Southern Queensland, Toowoomba, Australia. Office for Learning and Teaching Project. (2005). Distributed leadership in learning and teaching in Australian universities. Retrieved July 12, 2015, from https://emedia.rmit.edu.au/distributedleadership/?q=node/10 Phillips, R., McNaught, C., & Kennedy, G. (2011). Evaluating e-Learning: Guiding research and practice. USA: Routledge. Pink, D. (2011). Drive: The surprising truth about what motivates us. USA: Penguin Group. Retrieved from Poole, G. (2009). Academic disciplines: Homes or barricades? In C. Kreber (Ed.), The university and its disciplines: Teaching and learning within and beyond disciplinary boundaries (pp. 50–57). USA: Routledge. Steel, C. (2004). Establishing a zone where technology innovation is supported. In Roger Atkinson, Clare McBeath, Diana Jonas-Dwyer and Rob Phillips, Beyond the comfort zone ASCILITE 2004 (pp. 865–874). Perth, Western Australia. Villar, L., & Alegre, O. (2007). Measuring the learning of university teachers following online staff development courses: a Spanish case study. International Journal of Training and Development, 11(3), 200–213. Weimer, M. (2012). Learner-centred teaching and transformative learning. In E. Taylor & P. Cranton (Eds.), The Handbook of Transformative Learning: Theory, Research, and Practice. USA: Jossey- Bass. Weller, M. (2011). The Digital Scholar: How technology is transforming scholarly practice. Bloomsbury Publishing PLC. Retrieved from http://www.bloomsburyacademic.com/view/DigitalScholar_9781849666275/book-ba- 9781849666275.xml Mitchell, M. & Cottman, C. (2015). Personalising professional learning mobility in Higher Education. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:160-CP:164). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:164 516 Connecting fun and learning- an activity-theoretical approach to competency based game development Mark O'Rourke Deakin University, Australia Games-based learning has the potential to improve engagement and skill development. This research explores the development of the White Card Game and the impact that fun has on learning outcomes. The first-person shooter style game offers a contextualised, situated experience that equips learners with skills and an understanding of the socially complex world of work. The research has approached the analysis through an Activity Theoretical framework. This approach involved: analysing the interactions between components in the games-based learning activity system while they evolved; identifying contradictions and exploring the mediation that progressed the activity outcome; and examining fun within the games-based learning context. This analysis revealed significant increases in knowledge transfer, skill development and engagement with the curriculum in comparison to conventional pedagogical approaches. Keywords: games-based learning, immersive environments, activity system, fun, scaffold Introduction The use of game technologies can provide exciting ways to engage and educate new learners, especially those who may be disadvantaged in conventional learning environments. Games can offer a transformational change in pedagogical approaches by being intrinsically motivating, providing immediate feedback to learners and scaffolding skill and knowledge acquisition (de Freitas & Maharg, 2011). In this study a 3D immersive game environment was developed from an Activity Theoretical (Engestrom, 1987) perspective and some of the outcomes of the game as reported by the users has been analysed in the context of the development and production process. Trials of the White Card Game were undertaken with Certificate 3 in Construction (Carpentry) students who reported a preference for games-based learning compared to traditional delivery methods. They also expressed greater understanding for both the learning content and the relevance to course outcomes. The major concern that guided the research design of this project were the significant resources and time required for designing, developing and refining computer games to be used for education. The aim was to explore an approach to analyse tensions and facilitate productive interactions among developers, teachers and students that are involved in the design, application and use of games- based learning. The approach adopted an Activity Theoretical framework in order to facilitate the analysis of needs, tasks and outcomes in the games-based learning environment. This environment involved developing and trialling a computer game to achieve learning outcomes. The computer game components of the activity system that were examined included a number of parameters including narrative, gameplay and fun. This paper focuses on levels of fun and tests the impact it has on learning outcomes. Literature review Games-based delivery can establish a new paradigm where the critical constructs of learning are transformed from information and knowledge units relayed via curriculum to active learning experiences (de Freitas & Maharg, 2011). Critical aspects of games that can offer this transformational change in education include: the intrinsic motivation of gameplay; the responsiveness of the game environment in providing immediate user feedback; and the opportunity to scaffold the delivery of content in order to offer a complex, diverse and engaging learning opportunity. The fun had playing these games can influence these critical aspects subsequently influencing motivation and facilitating learning (Whitton, 2009) Perhaps the most frequently adopted framework in the study of games was proposed by CP:165 517 Csikszentmihalyi (1992) who conducted research into what makes experiences enjoyable and defined the term "flow". Flow is described as the process of optimal experience, whereby individuals are so involved in an activity that nothing else seems to matter. Making learning fun is a powerful incentive to engage students in the educational process. Players experience the results of decisions they make and are able to influence the game world with a responsive agency that delivers determination and empowerment to the player (Klimmt, Hartmann, & Frey, 2007). This is a challenging active experience and as expressed by Papert (1998) hard fun is the enjoyment had from mastering hard and complex gameplay. From this theoretical perspective serious games have enormous potential to offer alternative viewpoints with their capacity to combine realistic representation and imaginative fantasies in collaborative, participatory spaces. As the games demand instant reactions from player decisions, the feedback loops offer deeper thinking and learning opportunities (Gee, 2007). In a study devoted to measuring learners’ cognition of enjoyment Fu and colleagues (2009, p. 111) summarised their findings, "whether or not a game offers enjoyment to the player is a key factor in determining whether the player will become involved and continue to learn through the game." Methodology The White Card Game locates the player on a commercial building site. The player creates their avatar, puts on safety gear and independently moves through the building site achieving the game goals of identifying, reporting, assessing and controlling hazards over three levels of the building. The design and development of the White Card Game involved teachers and developers. Research data was collected from: observations of students and teachers in the classroom while students were engaged in playing the game; in-game data collection of students' gameplay activity; hard copy surveys; interviews with students; and communication documentation from teachers and game developers during planning, production and trialling of the games. Ethics approval was granted prior to the research. The teachers were introduced to the game development process and tools, and were given an indication of what was possible within the constraints of the budget available. This included a demonstration of game mechanics and limitations of user control over fine manipulation of game objects. They acquired an understanding of how the game environment could transfer knowledge through gameplay and also provide a more cognitive focus than practical skill acquisition. Similarly discussion with developers was conducted to ensure there was an understanding for aligning competency based learning criteria with the gameplay scenarios. This enabled the developers to envision how the game design would mediate the learning for the student participants, and in addition gave an occupational context for the game production. Communications data between teachers and developers indicated a transformation in their understanding of pedagogical game design that ensured successful learning and gameplay outcomes. Educational games provide a context for learning but also create a context through the continual interaction between users and the system. In the analysis of game components the research considers the goals, intentions and interactions of the teachers using the games-based resource, the designers developing the game and the students who are learning from the game. Engestrom's (1987) Activity Theory model describes purposeful interaction of active subjects with the objective world. Activity Theory is specifically concerned with how tools, which represent the accumulation and transmission of social cultural knowledge, mediate activity. This is represented in the top triangle in Figure 1. CP:166 518 Figure 1. Engestrom's Activity System Model Engestrom expanded the subject-object interaction to encompass collective activity by introducing "community", thereby creating a three-way interaction among subject, object and community. In addition other means of mediation include "rules" for the subject-community interaction; and "division of labour" for the object-community interaction (Kaptelinin & Nardi, 2012). An important principle of Engestrom's theory is that activity systems are constantly developing, and these developments are driven by contradictions. The games-based learning activity system in this research situates the learner as the subject of the activity. The object of the system is the game trial, which encompasses the activity, interactions and contradictions of the components, and as a product fulfills the goals or intentions of the activity through its transformation. This transformation moves the subject closer to the goal or outcome, producing knowledge acquisition in the learner. The community involves students, teachers, developers and the researcher, the nature of their social interactions and the beliefs and values that define or impact on the activity. This includes their styles and strategies for learning and their interactions with the technology. The communication or division of labour component describes technological and face-to-face communications in the design, development and trial of the games- based learning system. The rules constrain student users of the game in the subject content they are presented with and determine how game parameters feature in the design along with the learning content. The instruments of the activity system refer to the developed game. Results and Discussion Activities are socially and contextually bound, so can only be described in the context of the community they operate within. The community negotiates the rules and customs that define how it functions. In the context of the games-based activity the community includes teachers, designers and students. Individuals within these communities, with their different expectations, have had to alter their beliefs to adjust to the socially mediated expectations of the other groups. For instance the time spent on designing specific features of the game world is not necessarily of benefit to the learning experience. This has required a degree of mediation and transformation to address the conflicts between teachers, designers and the researcher who held the role of production manager. Within the context of the games-based activity system the rules defined constraints that facilitated a guided experiential learning process and impacted on the learner's capacity to apply knowledge acquired in the game, to problems or tasks presented in the game-playing context. The contradiction here is that at the same time it is critical to design the game so the user believes they have choice over movement and tasks, and that rewards provide feedback rather than appearing as a mechanism of control (Ryan, Rigby, & Przybylski, 2006). The results of trials supported a connection between learning and fun with observations of students playing the White Card Game indicating they were engaged with the content and interacting with their peers and teacher. Survey responses indicated that 74% of students enjoyed playing the game, 70% had fun and 78% found the game engaging. Coupled with 71% stating that they learnt about the topic playing the game we can infer that having fun and being engaged is linked to successful learning outcomes. The results also indicated that the game was more effective for enhancing learning outcomes than conventional teaching methods. This was supported by students' comments: CP:167 519 GG: "Game reinforces issues. You can always read a book but until you put it into action it it doesn't make sense. Book doesn't really show you the safety issues. You need to experience it to really understand it." ST: "Sometimes it is fun, but with my study, not that much information with my study I get, but when I compare things that are very important if I spend 1 hour playing this game I will learn the things that are very important but I will not get that much information in 1hr of study." Most of the communication focused on issues of translating pedagogical priorities into gameplay scenarios. The researcher activity was more subject oriented, enabled through delegating object oriented production coordination to the development team. Object oriented teamwork generates production competency and collective expertise (Engestrom, 2008) and in this instance made production communication more direct. However this also introduced contradictions in the activity system. As experienced as the development team was, a lot of the educational products they had worked on had implemented summative assessment processes in the form of quizzes. In these products the user was required to step outside the intrinsic gameplay and undertake more conventional delivery and assessment akin to a mastery learning model (Carroll, 1989). Communications with the team mediated this contradiction. The researcher discussed the priorities of embedding learning within gameplay and introduced concepts of action and goal-directed learning to the team. Also relayed to the team was the need to draw a distinction between virtual worlds and games. This required a significant shift in practice for the team from their experience in designing exploratory 3D world simulations, to including consequences and learning through failure within the gameplay. The development team’s productions to date had not included a capacity for having fun while exploring. This facilitated team development by allowing an opportunity for team members to reflect on the outcomes of the game player as learner. This was highlighted by one of the developers who spent a lot of the production in realistically creating hazardous situations with serious consequences for users who failed to perform safely on the building site. This allowed for the enhancement of a situated learning experience that embraced "context with consequentiality" (Barab et al., 2010). However, at the same time this expansive learning by the team member challenged prevalent practice and managerial values, and in doing so caused anxieties for the team leader whose priority was keeping the project within budget. This highlights a major shift in the motivations of members of the community in progressing the activity. The motivation of the researcher was the successful completion of the project in order to research the effectiveness of the game. The motivation of the team leader was to ensure financial viability of his company and responsibility to his development team in supplying ongoing employment. The dynamics created through these different motivations all focus on the objects of the activity, but the variations in the way the object is treated has an impact on the success of the activity outcome. As indicated through the transformations of the production team, there was a realignment of pedagogical focus in the development of the White Card Game, which successfully progressed the activity and expanded the existing boundaries of the team. Conclusion The White Card Game was developed to meet the needs of a large number of the student cohort with very low English skills and/or previous schooling, and aimed to engage and improve learning outcomes for all students. The findings have shown that the majority of students found that the game provided a richer learning experience than conventional methods. By adopting an Activity Theoretical framework analysis indicated that developmental transformations driven by contradictions were shown to occur in the components of the activity system. These mediated transformations involved teachers, developers and students in the games-based learning context, and resulted in revised knowledge and skills of the learner participants as the outcome of the activity. The significance and innovation of this research lies in its capacity to deliver new learning contexts that frame the development, integration and use of interactive games-based learning resources. The research outcomes make a significant contribution to sustainable production practices by targeting operational and technical developments and providing the framework to develop and evaluate new pedagogical approaches. This research supports innovation in higher education by identifying CP:168 520 pedagogical and technological barriers that impact on the use of games technology for learning acquisition. The White Card Game was funded through the VET E-learning strategy and to date there has been over 1000 downloads of the game, including institutions that only need a single copy for sitewide distribution. The game has been recognised as a successful pedagogical platform, having won Bronze in the IMS Global Learning Impact awards in San Diego, 2013, and Simulation Australia - Grand Prize 2013. References Barab, S. A., Dodge, T., Ingram-Goble, A., Peppler, K., Pettyjohn, P., Volk, C., & Solomoua, M. (2010). Pedagogical dramas and transformational play: Narratively rich games for learning. Mind, Culture, and Activity, 17(3), 235-264. Carroll, J. B. (1989). The Carroll Model: A 25-Year Retrospective and Prospective View. Educational Researcher, 18(1), 26-31. Csikszentmihalyi, M. (1992). Flow: the Psychology of Happiness. London: Random House. de Freitas, S., & Maharg, P. (2011). Digital Games and Learning: Modelling Learning Experiences in the Digital Age. In S. d. Freitas, H. Jenkins, & P. Maharg (Eds.), Digital Games and Learning (pp. 17-39). London: Continuum International Publishing Group. Engestrom, Y. (1987). Learning by expanding. An activity-theoretical approach to developmental research. Helsinki: Orienta-Konsultit Oy. Engestrom, Y. (2008). From teams to knots: Activity-theoretical studies of collaboration and learning at work: Cambridge University Press. Fu, F. L., Su, R. C., & Yu, S. C. (2009). EGameFlow: A scale to measure learners' enjoyment of e- learning games. Computers & Education, 52(1), 101-112. Gee, J. (2007). Good Video Games + Good Learning. New York: Peter Lang. Kaptelinin, V., & Nardi, B. (2012). Activity Theory in HCI: Fundamentals and Reflections (Vol. 5): Morgan and Claypool. Klimmt, C., Hartmann, T., & Frey, A. (2007). Effectance and control as determinants of video game enjoyment. CyberPsychology & Behavior, 10(6), 845-848. Papert, S. (1998). Does Easy Do It? Children, Games, and Learning. Game Developer, 88. Ryan, R., Rigby, C. S., & Przybylski, A. (2006). The motivational pull of video games: A self- determination theory approach. Motivation and Emotion, 30(4), 344-360. Whitton, N. (2009). Learning with Digital Games: A Practical Guide to Engaging Students in Higher Education O'Rourke, M. (2015). Connecting fun and learning- an activity-theoretical approach to competency based game development. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:165-CP:169). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:169 521 Learners’ confusion: faulty prior knowledge or a metacognitive monitoring error? Mariya Pachman Amael Arguel Lori Lockyer Macquarie university Macquarie university Macquarie university Australia Australia Australia Research often treats confusion as a turning point of the learners’ cognitive-affective dynamics in digital environments (e.g. D’Mello, Grasser and colleagues). The origin of confusion, however, is a topic of a debate. Could inaccurate prior knowledge serve as a source of confusion, or does confusion relate to metacognitive processes? In this paper we are attempting to address this question by employing case study analysis with fourteen participants who worked through simulated learning problems with feedback in a digital environment. Physiological and self-reported data were combined to examine problem-solving patterns. Preliminary findings highlighted the role of metacognitive monitoring in confusion development and its interrelation with inaccurate prior knowledge. Keywords: prior knowledge, metacognitive monitoring, confusion, self-regulated learning Background To effectively learn in technology-enhanced courses and digital learning environments students have to effectively process new complex information, know how to handle technical difficulties, and how to deal with the lack of immediate teacher's feedback. Learners may get confused trying to balance these multiple demands and end up frustrated and disengaged after reaching multiple impasses. Indeed, an unresolved confusion can easily turn into its non-constructive variety, in which case the learner’s interest will be completely lost (D'Mello & Greasser, 2014). At the same time, research demonstrates that cognitive disequilibrium, manifested by constructive confusion, plays a positive role in learning, promoting deep elaborative processing of the new information and transfer of learning (D’Mello et al., 2014). The problem lies in understanding of the origin of confusion. The four-tiered model of cognitive-affective dynamics described in details by D'Mello, Lehman, Pekrun and Graesser (2014) summarizes the findings of previous research and presents confusion within a network of the linked states. All four states — engagement, confusion, frustration and boredom — play an important role in learning according to these authors. Specifically, in this model, learners are initially in a state of engagement that can lead to confusion when an impasse is detected. If the impasse is resolved, learners will return to the original engagement state. Otherwise, with the failure to resolve the impasse learners may experience frustration. If learners stay confused for too long, for example while experiencing persistence failures to resolve an impasse, they may become bored and disengage from the learning task. Prolonged state of confusion, or joint confusion and frustration may have negative consequences for learning (D'Mello & Greasser, 2014; Liu et al., 2013). Confusion is likely to occur when students experience an impasse while processing new information that is inconsistent with their prior conceptions (Graesser, Lu, Olde, Cooper-Pye, & Whitten, 2005). Cognitive disequilibrium that triggers confusion is seen as an essential element in learning about complex systems, for which a shallow processing of information would potentially lead to critical errors of understanding and long lasting misconceptions. An alternative situation when the inaccuracy of prior conceptions is not detected, and learners engage in shallow processing could then be characterized by the Illusion of Understanding (Rozenblit & Keil, 2002). With Illusion of Understanding (IOU) learners have conviction of knowing or understanding something while in reality the knowledge or understanding is missing. Returning to the model of cognitive-affective states, the prior knowledge is not included in the model, although the authors of the model and the other researchers mention it throughout their work. The common assumption stating that a large knowledge base and good technical skills might help learners avoid confusion in online environments has never been challenged by considering the cases of inaccurate prior knowledge. In fact, if a learner experiences incongruences and a mismatch between prior conceptions and a new information, leading this learner to a cognitive disequilibrium CP:170 522 (e.g. Graesser et al., 2005), it calls for a reflection on the role of the prior knowledge in an “impasse” leading learner to confusion. Logically, the presence of a misguiding prior knowledge would contribute to the episode of impasse. Cases of inaccurate prior knowledge are quite common in learning math and science, and are tightly related to the notion of conceptual change. Specifically, conceptual change occurs when learners’ prior knowledge contains misconceptions and is inconsistent with the new upcoming information (Vosniadou, 1994). Inaccurate prior knowledge could lead to confusion, and some of the conceptual change literature encourages educators to seek a potentially conflicting for a learner situation when contradictions caused by the learner’s irrelevant conceptions are exposed and examined (Limón, 2001). Such method should potentially lead to re-evaluation of these outdated conceptions and foster a conceptual change. Inaccurate prior knowledge was also recently reviewed by self-regulated learning researchers in relation to the metacognitive monitoring accuracy. Van Loon et al. (2013) found that inaccurate prior knowledge not only leads to larger overestimations of the future performance in comparison with the cases in which no prior knowledge existed, but also that these overestimations were not corrected by actually taking the test (i.e. learners also hugely overestimate their on-test performance after taking the test). Finally, in line with previous research on metacognitive regulation, learners were not considering to re-study the materials they thought they knew (Thiede, et al., 2003; Van Loon et al., 2013), which would have also led to a better learning performance (Thiede, et al., 2003). Taking that metacognitive monitoring is often considered to be the most important part of self-regulation process (Thiede, et al., 2003; Winne, & Hadwin, 1998), these findings demonstrate how tightly self-regulatory processes are linked with prior knowledge. The present study is aimed at uncovering linkages between metacognitive processes, prior knowledge, and confusion. The present study The study was a part of a larger project which investigates confusion, feedback and self-regulation in digital learning environments. The study used insight problems (problems that require a shift of perspective or an ‘aha’ moment to reach a solution) in form of puzzles: this type of problems is notorious for activating non-relevant prior knowledge (Knobich, Ohlsson, & Raney, 2001). The problems presented to participants were transformation puzzles, in which the pieces could form two different layouts depicting different pictures. The specific shapes used for the pieces were causing a kind of visual illusion during the transformation. For Problem 1, both initial and final layouts looked similar but a gap between two pieces appeared for one of them. Problem 2 was of the same nature 1 but was showing the disappearance of a graphic element . Both problems were presented as on- screen simulations with learner control: learners could manipulate a scrollbar to move the puzzle pieces. Fourteen participants recruited via on-campus advertisement were tested individually. Their gaze trajectories were recorded via a Tobii-T120 eye tracker, two hints were provided and the solution times recorded. Then, individual gaze trajectories were shown to participants who retrospectively rated their confusion on a scale for each 1-minute interval of the problem-solving phase. They were also invited to report on their problem-solving approaches (think-aloud method). Thus, data from the reporting of confusion were triangulated with problem solving steps and trajectories obtained from eye tracker. Data analysis and results The data were coded by one rater, and the coding sheet was developed based on research questions and the emerging themes. Consequently, 35% of cases were coded by the second rater using a coding sheet. The inter-rater agreement was 90%. The remaining 10% were negotiated and the corresponding changes were made to all similar cases. Records were analyzed as a collection of case studies. Confusion ratings were higher for Problem 1 than for Problem 2: M p1 = 7.24 and M p2 = 4.97 on a 10-point scale, and a larger number of participants 1 http://en.wikipedia.org/wiki/Missing_square_Puzzle#/media/File:Missing_Square_Animation.gif and http://www.archimedes-lab.org/workshop13skulls.html respectively. CP:171 523 did not solve Problem 1 (7 non-solvers for Problem 1 and 3 for Problem 2). Hence it seems that Problem 1 would have served as a pre-training for the more challenging Problem 2 (which was confirmed by audio records analysis). Finally, patterns of cognitive disequilibrium and illusion of 2 understanding were assessed for all the participants (Fig. 1). Single or recurring instances of cognitive disequilibrium were recorded in 31% of cases for Problem 1 and 45% for Problem 2. We have operationalized cognitive disequilibrium as an increase in confusion ratings (often leading to a change of strategy as seen in gaze trajectory) after hint or after a wrong solution. Such changes in both confusion ratings and a gaze direction could have been a result of thinking about an incorrect solution of the problem, and then, realizing that it was a wrong solution path. Cognitive disequilibrium plus an illusion of understanding were recorded in 38% of cases for Problem 1 and 33% for Problem 2. An illusion of understanding (IOU) was operationalized as a decrease in confusion ratings/low confusion ratings before the hint or a wrong solution is delivered, increase after. IOU was often (86% of cases) followed by cognitive disequilibrium. While a case with the wrong solution clearly illustrates that participants were under the impression they knew the answer, a case with hints is not that straightforward. Often participants asked for a hint when they already had something in mind (as evident from audio records), in case of IOU this something or their existing ideas were rather misleading that why the primary decrease was followed by an increase in confusion ratings. If the participants did not ask for hints, hints were delivered 2 and 4 minutes after the start of the problem- solving, and similarly, if participants had something misleading in mind the primary decrease was followed by increase in confusion ratings. As we have already mentioned, these detected states were recurrent (as per D’Mello et al., 2014 model): one instance of cognitive disequilibrium could have been followed by the other when an additional impasse has been reached. Figure 1. Patterns during problem-solving with hints. “Other” cases (Fig. 1) are referring to the instances when the participants did not change their confusion ratings until the solution was reached or explained to them, or, alternatively, they were rating their confusion as decreasing through problem-solving process. Discussion While cognitive disequilibrium is discussed in the literature as the trigger for confusion (D’Mello & Graesser, 2014), incidents of IOU refer to the literature on metacognitive monitoring. IOU is especially harmful for learning because it influences the restudy efforts (e.g. Thiede, et al., 2003): learners who are convinced they know a vocabulary word, or an answer to a question, choose not to focus on this word or question anymore. Learners prone to IOU will try to reduce the time needed for the task and to deliver a rushed (often incorrect) answer, as it was observed with our participants. It is interesting to note that, although learners reported to be less confused with Problem 2 and produced a higher solution rate, the percentages of IOU instances were relatively similar amongst the problems (Fig. 1). Indeed, an increased familiarity with the task has been shown to increase participants’ confidence in 2 As two instances of cognitive disequilibrium experienced by the same subject while solving Problem 1 or 2. CP:172 524 ability to perform well on a test although it was not warranted by the test results (Baars et al., 2013). If Problem 1 was regarded as a pre-training for Problem 2, learners could feel increased confidence in their ability to solve this problem and, as a consequence, IOU was present at the same rate in Problem 2 as in Problem 1. Besides, learners were suggested to derive their monitoring judgments from the amount (but not the quality) of accessible information that comes to mind (Koriat, 1993). It equally explains the comparable IOU rates for both problems, and the presence of IOU at the first place. Since our pilot problems were insight problems learners were confident they possessed a certain prior knowledge. In reality, this prior knowledge was misleading, and its activation negatively reflected on metacognitive monitoring and, potentially, on performance (these data are being currently analysed) in line with the findings of Van Loon et al. (2013). Conclusion It could, then, be argued that the incorrect assessment of one’s potential or past performance (i.e. IOU) comprises a metacognitive component of confusion while inaccurate prior knowledge represents its cognitive component. “What learner believes to know […] influences his learning, not only directly” via prior knowledge, “but also indirectly by affecting metacognition and regulation of learning” (van Loon et al., 2013, p. 24). In this case any intervention aimed at reducing non-constructive confusion (via self-regulatory techniques) has to address the monitoring side of the process. Our recommendations for creators of and educators working with digital learning environments would then stress the importance of faded scaffolding (similar to Baars et al., 2013 techniques), asking students to self-explain or to draw concept maps of textual materials (e.g. Thiede et al., 2010). Overall, the above techniques were proven to improve both performance and monitoring accuracy for learners and to help them avoid illusion of understanding. Further research could also investigate additional techniques particular to technology-enabled environments. References Baars, M., Visser, S., van Gog, T., de Bruin, A., & Paas F. (2013). Completion of partially worked-out examples as a generation strategy for improving monitoring accuracy. Contemporary Educational Psychology, 38, 395-406. D’Mello, S. K. & Graesser A. C. (2014). Confusion and its Dynamics during Device Comprehension with Breakdown Scenarios. Acta Psychologica, 151, 106-116. D'Mello, S. K., Lehman, B. Pekrun, R., & Graesser, A. C. (2014). Confusion can be Beneficial for Learning. Learning & Instruction, 29(1), 153-170. Graesser, A. C., Lu, S., Olde, B. A., Cooper-Pye, E., & Whitten, S. (2005). Question asking and eye tracking during cognitive disequilibrium: Comprehending illustrated texts on devices when the devices break down. Memory & Cognition, 33(7), 1235-1247.Knoblich G., Ohlsson S. & Raney, G.E. (2001). An eye movement study of insight problem solving. Memory and Cognition, 29(7), 1000-1009. Knobich, G., Ohlsson, S., & Raney, G.E. (2001). An eye movement study of insight problem solving. Memory & Cognition, 29(7), 1000-1009. Koriat, A. (1993). How do we know what we know: the accessibility model of feeling of knowing. Psychological Review, 100, 609-639. Limón, M. (2001). On the cognitive conflict as an instructional strategy for conceptual change: acritical appraisal. Learning and Instruction, 11(4), 357-380. Liu, Z., Pataranutaporn, V., Ocumpaugh, J., & Baker, R. (2013). Sequences of Frustration and Confusion, and Learning. In D’Mello, S. K., Calvo, R. A., and Olney, A. (eds.) Proceedings of the 6th International Conference on Educational Data Mining Conference (Memphis, TN, July 6 - 9, 2013), 114-120. Rozenblit, L., & Keil, F., (2002). The misunderstood limits of folk science: an illusion of explanatory depth. Cognitive Science 26, 521–562. Thiede, K.W., Anderson, M. C. M., & Therriault, D. (2003). Accuracy of metacognitive monitoring affects learning of text. Journal of Educational Psychology, 95, 66-73. Thiede K.W., Griffin T., Wiley J., & Anderson M. C. M., (2010). Poor Metacomprehension Accuracy as a Result of Inappropriate Cue Use. Discourse Processes, 47(4) 331-362. van Loon, M., de Bruin, A., van Gog, T., & van Merrienboer, J. (2013). Activation of inaccurate prior knowledge affects primary-school students’ metacognitive judgments and calibration. Learning and Instruction, 24, 15-25. CP:173 525 Vosniadou, S. (1994). Capturing and modeling the process of conceptual change. Learning and Instruction, 4, 45-69. Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker & J. Dunlosky (Eds.), Metacognition in educational theory and practice (pp. 277-304). Mahwah, NJ: Erlbaum. Pachman, M., Arguel, A., & Lockyer, L. (2015). Learners’ confusion: faulty prior knowledge or a metacognitive monitoring error? In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:170- CP:174). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:174 526 Exploring my university students’ online learning activities in Wikis Choon Lang Gwendoline Cong Liu Quek Institute of Arts and Humanities, Learning Sciences and Technologies Shanghai Jiao Tong University Academic Group National Institute of Education, Singapore Students’ responses in an online learning environment serve as a powerful means to communicate feedback to instructors’ instructional design and facilitation of student learning. This study tapped on the metadata in wikis (supported by Google Sites) as online classroom data to investigate 72 university students’ online learning activities performed for their module weekly. The students were engaged most frequently in commenting and editing, but least frequently in updating and recovering files. Trends of students’ responses towards online learning over four semesters provided an insight for instructors to reflect on the appropriateness of their design and types of learning activities for their students. Keywords: Classroom data, online learning, online teaching, Wikis Introduction Both student learning and instructor teaching have great influence in the ‘happenings’ of online learning environment. Typically, the students receive instruction, perform tasks and spend significant amount of time learning materials provided by their instructors. Instead of collecting student feedback at the end of each course, a spontaneously available online data could be generated to capture the on-going learning process to help instructors make decisions in their facilitation of students’ learning activities. As both instructor and students are not online at the same time, students’ on-going weekly responses to the online teaching would help instructor understand the students’ participation and tasks performed. It is argued that the development of instructors’ knowledge and skills in using systemic and classroom data will help informing and generating improved students’ learning outcomes(Renshaw, Baroutsis, van Kraayenoord, Goos, & Dole, 2013). More specifically, the analysis of students’ learning activities can provide immense feedback on instructors’ task designs, facilitation and support for students leading to their overall teaching effectiveness. Online learning supported by social media such as Google sites, blogs and wikis is a common practice(Miller, 2014). Instructors use online learning as a pedagogical tool to reap teaching benefits such as structuring group collaboration and cooperative learning, promoting active student engagement in learning, using both synchronous and asynchronous activities, having round-the-clock access to the learning activities, and preparing discussion posts that invite insightful responses(Mayadas, Bourne, & Bacsich, 2009). Wikis is a social book marking tool that can be used for students to remain connected to the content being studied in both inside and outside of classroom settings(Oxford & Oxford, 2009). Students’ participation in online learning environments is as important to the instructors as they are to the students. It is only through students’ participation of online activities that instructors can gather insightful information about the students’ reaction towards the content, and how they can then better teach the content in order to achieve improved student learning outcomes (e.g. cognitively and affectively). Using the classroom data that is information gathered in the online learning environment can potentially lead to accounting any possible trend or describing the learning phenomenon that takes place(Renshaw et al., 2013). This paper seeks to demonstrate how an inquiry-based analysis of classroom data that focuses on the students’ online learning activities in Wikis could be used to inform and improve the instructors’ online teaching. Two research questions are: What activities do students engage in most frequently and least frequently in Wikis? How do students respond to types of online learning activities in Wikis? Methodology CP:175 527 Participants A total of 72 undergraduate students (Male: 14 ~ 19%, Female: 58 ~ 81%) enrolled over four semesters (starting from Jan 2013, Jan 2014, Aug 2013, Aug 2014) in a Singapore university participated in a module. Description of the students’ year and programme of study are shown in Table 1. Table 1: Year and programme of study (by semester in percentage) Jan 2013 Jan 2014 Aug 2013 Aug 2014 (A) Years 1 10.0% 21.0% 30.0% 25.0% 2 15.0% 27.0% 40.0% 42.0% 3 35.0% 26.0% 25.0% 33.0% 4 40.0% 26.0% 5.0% 0.0% (B) Ranks 1 (high) BUS (27%) PSY (39%) PSY (55%) PSY (46%) SOC (27%) 2 (low) PSY (20%) CHIN (11%) LMS (10%) ELH (15%) ELH (11%) Legend: BUS – Business SOC – Sociology PSY – Psychology CHIN – Chinese ELH – English LMS – Linguistics & Multilingual Studies Note Ranks refer to the enrolment of students by their programme of study Procedure Throughout the 16-week module, students were required to participate in a series of activities on wikis supported by Google Sites, a web application. Apart from accessing the course materials, other activities such as personalising a page for self-introduction, creating reflection logs (created); submitting work (attached); making revision to an existing page or subpage (edited); peer editing, making resource contribution (commented); removing work uploaded onto the page (deleted); making revision to an attachment uploaded for sharing or submission (updated); retrieving a previously deleted document (recovered). All learning activities were conducted in the online environment both within and outside of class time, and were tracked in the Google Sites metadata under the ‘Recent site activity’ page. Data analysis of students’ online activities was conducted in three parts: • Frequencies: Includes a) the most and least frequently performed activities, i.e. created, attached, edited, commented, deleted, updated, recovered, and b) the most and the least active week within a semester, • Trend analysis: Includes a) the frequencies of overall activities, and b) the frequencies of each activity type in each semester, and • Correlation analysis: Includes the correlation between types of activity performed by students Results In terms of frequencies, Figure 1 shows that “commented” and “edited” are the most frequently engaged activities (at 29.51% and 28.40% respectively) while “updated” and “recovered” are the least frequently engaged activities (at 2.92% and 0.17% respectively) across the four semesters. Figure 1 also shows students’ learning in terms of knowledge building, sharing and critique. Students were mostly engaged in commenting and editing. However, the students were found to be least engaged in updating and recovering their documents. This shows the students’ careful attitude when sharing information online since they were least likely to make revisions to their attachments (i.e. updated and recovered) that they had uploaded on the online platform; they would do self-correction if they were to make mistakes or when their information on the Wiki was no longer relevant. The frequency of student activities could also indicate the cohorts’ preferences towards specific activities. For example, two cohorts of students (Jan 2013 and Aug 2013) were more inclined towards commenting while two other cohorts (Jan 2014 and Aug 2014) were more inclined towards editing in the online learning activities. CP:176 528 15.00% 10.00% 5.00% 0.00% created attached edited commented deleted updated recovered Jan 2013 Jan 2014 Aug 2013 Aug 2014 Figure 1: Student Activities in Wikis supported by Google Sites for 4 semesters Figure 2 shows the students’ online actions over 16 weeks (inclusive of week 0, recess week and week 14) for all four semesters. Students were most active in wikis during week 3 (10.9%), followed by weeks 6, 8 and 10 (at 9.9%, 9.5% and 9.5% respectively) across all four semesters. 8.00% 7.00% 6.00% 5.00% 4.00% 3.00% 2.00% 1.00% 0.00% Jan 2013 Jan 2014 Aug 2013 Aug 2014 Figure 2: Overall Student Activity in Wikis supported by Google Sites for 4 semesters Figure 2 shows that for both Jan 2013 and Jan 2014 cohorts, students were relatively less active on wikis during weeks 4, 5 and 7 (at 0.12%, 0.69%, and 0.02% respectively for Jan 2013; at 0.55%, 0.19%, and 0.76% respectively for Jan 2014). The smaller percentage in the overall activity for these weeks could be attributed to public holidays (Chinese Lunar New Year) and students’ prelude to recess week (a week-long break) respectively, as students are either in festive mood and/ or more relaxed, and are hence less active in their online participation on the Wiki. However, the higher percentage in the overall activity for week 6, recess week and week 8 (at 0.33%, 0.43%, and 0.76% respectively for Jan 2013; at 2.33%, 2.78%, and 1.40% respectively for Jan 2014 ) could suggest that students’ work productivity tend to increase during the break (from week 7 to recess week - from 0.02% to 0.43% respectively for Jan 2013; from 0.76% to 2.78% respectively for Jan 2014) and/ or after break (from week 7 and week 8 – from 0.02% to 0.76% respectively for Jan 2013; from 0.76% to 1.40% respectively for Jan 2014). In particular, the rise in percentage in the overall activity from week 7 to recess week could be due to students’ using the recess week (refers to school break) to consolidate and catch up on work that was conducted in the first half of the semester. It is of interest to note that the slight decrease in the percentage of overall activity from recess week to week 8 for the cohort of Jan 2014. This could indicate that students from that particular cohort might have possibly coped or caught up with their work successfully. In sum, the students were less likely to participate in online learning during the festive period as compared to the recess/school break. This could mean that students used the recess to catch up their school work instead of participating in any online activities in Singapore context. Figure 2 shows the trend of student activity performed in the wiki increased steadily from 12.65% (in Jan 2013) to 35.52% (in Aug 2013), but declined at 29.04% (in Jan 2014) and at 21.66% (in Aug 2014). This trend is likely due to students enrolled in the module becoming more technology savvy as a result of their continued exposure to the online learning environment. For example, they could use CP:177 529 more learning features of Wiki to comment, share resources and tools for their projects. This could also be due to the university’s campus wide active promotion for technology-enabled pedagogy for teaching and learning. All these could have resulted in students from the later cohorts in becoming more adventurous in exploring new web-based tools and web-based resources in designing their projects in the subsequent courses. Correlation analysis was conducted across types of activities (refer to Table 2). The frequency of “created” is positive correlated with “attached” (r = .35, p < .01) and “edited” (r = .79, p < .01); the frequency of “attached” is positive correlated with “edited” (r = .47, p < .01), “commented” (r = .70, p < .01) “deleted” (r = .34, p < .01), and “updated” (r = .26, p < .05); the frequency of “edited” is positive correlated with “commented” (r = .39, p < .01); and the frequency of “commented” is positive correlated with “deleted” (r = .26, p < .05). The activity, “recovered” was not included in the correlation analysis due to its low frequency. Table 2: Correlation across different types of activities Created Attached Edited Commented Deleted Updated Created 1 .354** .790** .209 -.001 -.020 Attached 1 .469** .698** .343** .258** Edited 1 .385** .157 .083 Commented 1 .263* .060 Deleted 1 .133 Updated 1 *. Correlation is significant at the 0.05 level (2-tailed). **. Correlation is significant at the 0.01 level (2-tailed). Conclusion This paper has shown that through using an inquiry-based approach towards interpreting metadata stored in Google Sites, instructors can analyse student online activity so as to better improve their online pedagogy. The observed classroom data has revealed students’ behaviours in the online learning environment. It has shown that students in this study engaged most frequently in commenting and editing, but engaged least frequently in updating and recovering. Moreover, the student profile, the presence of school breaks, and the attitude towards online learning as reflected in the metadata do affect students’ response towards online learning. Students' participation in online platforms is not just important to the learners, but to the instructors as well. Hence, it is important that instructors take advantage of the activities that students engage in most frequently, as well as the presence of school breaks, so as to improve their students’ learning outcome. However, instructors should be cautioned against the use of simplistic analyses and comparisons derived from the metadata because the analysis has not taken into account the many underlying characteristics, such as profile of learners, socio-economic status or family background, which may explain the comparative performance of the students in terms of the grades obtained at the end of the module(Renshaw et al., 2013). In addition, careful attention and thoughtfulness should be exercised by instructors in scaffolding learners’ learning roadmaps through an understanding of their characteristics, the availability of curriculum time (online or blended) and the design of interesting student-centred learning activities, are necessary for their pedagogy to be effective in wikis(Author). References Author. Exploring Teachers' Perceptions of Wikis for Learning Classroom Uses. Australian Journal of Teacher Education , 39 (2). Fobes, C., & Kaufman, P. (2008). Critical Pedagogy in the Sociology Classroom: Challenges and Concerns. Teaching Sociology , 36 (1), 26-33. Gruba, P., & Hinkleman, D. (2012). Blending Technologies in second language classrooms. New York: Palgrave Macmillan. Mayadas, A. F., Bourne, J., & Bacsich, P. (2009). Online Education Today. Science Magazine , 323 (5910), 85-89. CP:178 530 Miller, M. D. (2014). Online Learning: Does It Work? In M. D. Miller, Minds online: teaching effectively with technology (pp. 23-24). London: Harvard University Press. Oxford, R., & Oxford, J. T. (2009). Second language teaching and learning in the Net Generation. Mānoa: National Foreign Language Resource Center, University of Hawai'i at Mānoa. Renshaw, P., Baroutsis, A., van Kraayenoord, C., Goos, M., & Dole, S. (2013). Teachers using classroom data well: Identifying key features of effective prctics. Final report. Brisbane: The University of Queensland. Quek, C.L.G. & Liu, C. (2015). Exploring my university students’ online learning activities in Wikis. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:175-CP:179). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:179 531 Learning to swim in an ocean of student data Carol Russel Office of DVC University of Western Sydney Like other Australian universities, Western Sydney University collects a large amount of data on student learning experiences, including their use of technologies. For busy discipline academics the task of mining and analysing all the data, to create meaningful evidence that informs teaching practice, can seem overwhelming. Graphs of responses to multiple choice questions are relatively straightforward to generate and share. But text comments in response to open-ended questions, although potentially very revealing, are often not used systematically. The University is making both quantitative and qualitative student survey responses available in a format that teaching staff can access directly through an institutional data dashboard. There has been some progress and there are some challenges. During 2015 we have been aiming to encourage teaching staff not just to dip their toes in the water but to take the plunge and use both quantitative and qualitative data actively and with purpose. Keywords: student feedback, data mining, text analytics. Launching into the strategy In late 2012, Western Sydney University embarked on an ambitious 3 year strategy to ramp up the use of technology-enhanced learning (TEL). In 2010, a student survey on learning technology had identified that students were expecting more and better use of technology than they were experiencing. In particular they wanted their teachers to engage with TEL (Gosper, Malfroy, & McKenzie, 2013; Russell, Malfroy, Gosper, & McKenzie, 2014). During 2011 and 2012, wifi was improved, lecture recording was transferred to an opt-out system and there were various other incremental improvements made to online learning facilities. At the beginning of 2013 there was a step change. The University issued all new students with iPads and began investing in enhanced support for redesigning curricula across all disciplines; recruiting blended learning support staff and in some cases also arranging for additional academic staff time. The new support teams were configured in a ‘hub and spoke’ model, with blended learning specialists available within disciplines to work hands-on with academics. Curriculum redesign initially focused on st 1 year undergraduate study, but in the following two years rolled out to other study levels, aiming to enhance flexibility for all and equity of access for students from low socioeconomic status backgrounds. Online and mobile technologies were combined with face-to-face campus classes. In the summer of 2013-4, condensed summer term options (many in blended or fully online mode) were introduced and in 2014 the University began expanding its fully online offerings. The University’s strategic plan for 2015-2020 has a central strategic objective of being ‘a distinctly student-centred university’; within which it aims to ‘transform its teaching and learning environments by integrating digital technologies with innovative curricula and work-integrated learning’. The challenge now is to ensure that these innovations are routinely being informed by evaluation evidence using data on students’ learning experiences and outcomes. This paper outlines work in progress to ensure that we are making effective use of the data we gather from and about students, to inform how we integrate technology into the curriculum. Evidence for strategic navigation A nationally funded Australian project on quality management for online learning environments in higher education found that while strategic plans are important, there is a need for distributed ownership and leadership – not just among teaching staff but also among the students who are supposed to be the beneficiaries (Holt et al., 2013). The project put forward a quality management framework with six components. The University has been addressing several of these: planning, technologies, resourcing and to some extent also organizational structure and governance. The sixth CP:180 532 component, evaluation that connects with a distributed leadership model, is the focus of this paper. Given the ‘student-centred’ strategic direction, gathering evidence from students has been a priority. Many teaching staff still needed convincing that students either benefit from or appreciate a shift away from established classroom teaching methods. To make sure that staff and students are on board and are pulling in the same direction, the overall approach in setting up the evaluation has been a pragmatic one; recognizing the need to accommodate multiple perspectives and to triangulate different sources of information (Phillips, McNaught, & Kennedy, 2012, pp. 77-78). Consistent with a pragmatic approach, the evaluation design has involved mixing qualitative and quantitative evidence; using a convergent research design to gather and merge complementary data from different sources (Creswell & Plano-Clark, 2011, pp. 77-81). We introduced a Blended Learning Survey (BLS) in September of 2013, including some questions st from the more comprehensive 2010 survey for comparison. The BLS went to 1 year undergraduate students and included both multiple choice questions about use of online and mobile technologies for learning, and requests for text comments (Russell & Qi, 2013). In 2014, questions on technology access and use were added to the regular Commencing Students Survey (CSS) and the BLS was run st nd again, this time with both 1 and 2 year undergraduate students. By then, student representatives on the University Senate were questioning the evidence that students wanted more ‘online lectures’ and asked for an additional survey. This was done (Russell, 2014). The extra survey included multiple choice questions about preferred study modes along with a request for text comments on their needs for flexibility. Other surveys in 2014 gathered student comments from open questions on ‘best aspects’ (BA) and ‘needs improvement’ (NI), which could be mined for comments on TEL (Table 1). The regular surveys were repeated in 2015, with the BLS now including all undergraduates. Table 1: Sources of data from 2014 students Survey Target respondents No of ‘BA’ No of ‘NI’ Evaluation purpose and timing comments comments Commencing newly enrolled students 985 907 experience of transition into Students Survey in weeks 3-4 of higher education (CSS) semester 1 Blended Learning all undergraduate 3137 3111 (on campus/ online study mode survey Part A students (flexibility) campus NI) preferences and flexibility (2014 only, with BLS) needs Blended Learning 1st and 2nd year 1976 1940 student use of technology for Survey (BLS) undergraduates in study Sept. Student Feedback all study units in all 43,630 35,399 design of study units and on Units (SFU) terms activities within them University sample of years 1 & 3 2989 2878 overall experience in degree Experience Survey undergraduates course Course Experience sample of graduates in 3697 3040 experience and value of Questionnaire the year following course after graduation (CEQ) graduation Various reports to senior management summarised the survey findings. For example, in one such report in 2015, a summary of evaluation evidence on use of mobile devices showed that in some disciplines laptops might be more useful for core study activities than iPads; suggesting a shift to providing multiple devices for students. Although we were collecting more evidence about the student experience of technology, teaching academics and their discipline team leaders still had no way of directly accessing and using this evidence. The SFUs are embedded in the established disciplinary curriculum and teaching review cycles, but do not include specific questions on TEL, while the surveys providing explicit information on TEL are not included in regular reviews. As a result, most teaching academics are largely unaware of what student feedback is available on TEL. So there is a need to streamline the gathering and use of survey data on TEL, to ensure that the students’ voices can be heard directly by their teachers, as well as at a more strategic level. CP:181 533 Data dashboard to the rescue For some years, the University has been running an institutional data visualization dashboard using Tableau software. The dashboard is used for planning academic programs and tracking institutional performance indicators. It also displays results from routine student surveys such as the SFUs and the CSS. The dashboard shows not only graphs of responses to multiple choice questions about the study experience, but also shows text analytics for student comments (Gozzard & Grebennikov, 2013). Targeted manual analysis of the BLS and other TEL data is time-consuming and requires skills with both quantitative and qualitative analysis methods. Given the amount of data available, and the wide variety of questions that could be asked of each dataset, it is impossible to generate and disseminate reports for all potential evaluation needs; especially when those who might use the evaluations do not know what is available. The dashboard is an ideal tool for academic teams to explore student feedback on technology use. Early in 2015, the institutional data analysis team extended the dashboard to display the BLS results from 2013 and 2014. Previous analyses of survey comments on TEL had used NVIVO for thematic analysis of student comments. The data analysis team used this earlier work and data (including the 2010 survey as well as 2013-14 comments) to update the text analytics to include categories and subcategories for TEL. The same text analytics programming is used for all student survey comments. So we can now use it to mine qualitative data on TEL from the other surveys. All the student data are de-identified, and the comments are cleaned to remove any references to individual teachers. Because the text analytics process is automated, it can cope with large amounts of text and can even deal with idiosyncratic student spelling. Figure 1 shows an example. nd Figure 1: Example of comment explorer dashboard display for 2 year Business students At this point, it seemed that we had solved the problem. Directors of Academic Programs and blended learning support teams could access to the dashboard and extract whatever was relevant to answer their own evaluation questions. Everyone concerned was duly informed about the availability of the dashboard displays, and those who requested dashboard access were able to see and select from all the blended learning survey data and other surveys where relevant. At the study unit level, academic coordinators could also use the analytics in the online learning management system to track student use of the online activities and digital resources they provide. We are also working to make aggregated analytics data available on student online activity across study units. So in principle, we had provided the tools for distributed leadership in curriculum evaluation and innovation. Still in danger of drowning After some initial presentations about the BLS dashboard to university committees and discussions with the blended learning support staff, it became clear that simply making the facility available would CP:182 534 not be enough to encourage its widespread use. Many potential users lack the time and/or skills for educational evaluations. They need help to develop evaluation questions, as well as to decide what data to extract and how to analyse it. Staff interviews were also part of the overall evaluation of the institutional TEL strategy – to gauge the effectiveness of staff development and curriculum development support in building staff capacity to introduce innovations. The semi-structured interviews covered a cross-section of 17 staff members across different roles, types of employment and discipline. Half of the interviews were in 2013 and and half in 2015, with 6 interviewees participating both times. Thematic analysis of the interview records showed a shift over the two years in the dominant themes. Among these was an increased frequency of discussion of learning activity design and of evaluation evidence. However, the main comments on evaluation evidence in 2015 related not to the BLS data, but to learning analytics, SFUs or other surveys at study unit level. These indicated that finding time to evaluate could be a barrier, for example: “…we are currently still analysing the data from the vUWS analytics. We can track weekly activity and preparation for class. We spent half a day extracting the data…..” “I don’t have time to track. I run two big units […] So there is no time for forward thinking.” The Quality Management Framework developed by Holt et al. (2013) implies that evaluation of a university’s use technology-enhanced learning should include stakeholder needs (staff and students), be embedded in governance structures and provide evidence not only for the selection of new technologies but also for ongoing assessment of performance, value and impact. In this case, we have a great deal of data from one set of stakeholders, the students, on how institutional decisions about technology have been changing their study experience, only some of which is being used effectively. We can track online activities through analytics in the online learning management system, and will soon make this available via a Tableau dashboard. We collect information from students about their experiences and already have this in a form that can be interrogated by academic groups. But we have work to do in building the feedback loops into institution-wide processes that engage the majority of teaching staff. This need was also picked up by the student representatives on the University Senate. When the results of the additional survey they requested in 2014 were reported, they asked that the University set up a body to regularly review the TEL strategy. Next steps: swimmer support In response to the student request the Senate Education Committee charged an established subgroup, the Student Experience & Engagement Committee, with developing a regular review process. Towards the end of 2015, there is a proposed framework and process for reporting annually, drawing in relevant student feedback from the regular surveys listed in Table 1. The proposal identifies components at three levels, drawing on a framework suggested by Gosper et al. (2013): institutional, academic-led and student-led technology use . It links these to specific objectives in the University’s strategic plan and identifies the data sources that can be used to draw out key messages for consideration by the Education Committee at the start of each year, so that the Committee, which includes discipline educational leaders, can discuss and recommend follow-up action. The intention is that an institution-wide formal review of student data on TEL will mean that resources and expertise are made available to support the distilling of key messages and translating these messages into actions within discipline groups. However this is still work in progress. There are outstanding questions on the details of who owns and analyses the data, what institutional expertise and tools are needed and how discipline academics will engage with the evidence produced. Our experience implies that many teaching staff will not dive into an ocean of student data and engage in meaningful evaluation unless support for evaluation is embedded at all institutional levels. Although a few of the more confident data swimmers may venture out into this ocean unaided, most will want at least one form of support – fins or flotation (support staff), swimming lessons (time to familiarise with data sources, analysis methods and tools) and navigation guidance (educational analysis). References CP:183 535 Creswell, J. W., & Plano-Clark, V. L. (2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage. Gosper, M., Malfroy, J., & McKenzie, J. (2013). Students' experiences and expectations of technologies: an Australian study designed to inform design and development decisions. Australasian Journal of Educational Technology, 29(2), 268-282. Gozzard, T., & Grebennikov, L. (2013). Hearing the Student Voice: Using Text Analytics to Measure Qualitative Feedback. Paper presented at the AAIR SIG Forum 2013, Leura, NSW. presentation retrieved from http://www.aair.org.au/app/webroot/media/SIGs/SIG%20Forum%202013/2013SIGTerryGozzardan dLeoGrebennikov.pdf Holt, D., Palmer, S., Munro, J., Solomonides, I., Gosper, M., Hicks, M., . . . Hollenbeck, R. (2013). Leading the quality management of online learning environments in Australian higher education. Australasian Journal of Educational Technology,, 29(3). Phillips, R., McNaught, C., & Kennedy, G. (2012). Evaluating e-Learning: guiding research and practice. London and New York: Routledge. Russell, C. (2014). Herding cats and measuring elephants: implementing and evaluating an institutional blended and mobile learning strategy. Paper presented at Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014, Dunedin. Russell, C., Malfroy, J., Gosper, M., & McKenzie, J. (2014). Using research to inform learning technology practice and policy: a qualitative analysis of student perspectives. Australasian Journal of Educational Technology, 30(1). Russell, C., & Qi, J. (2013, 1-4 December 2013). Evaluating an institutional blended & mobile learning strategy. Paper presented at the 30th ascilite conference, Sydney. Russel, C. (2015). Learning to swim in an ocean of student data. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:180-CP:184). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:184 536 Benchmarking for technology enhanced learning: Longer term benefits Michael Sankey Learning Environments and Media University of Southern Queensland It is one thing to undertake Benchmarking in the areas of technology enhanced learning (TEL) as a one-off activity, but it is quite another to build this form of activity into your strategy for future and long-term growth at an institution. This paper reports on a follow- up study conducted in 2015 with 22 of the 24 institutions who first participated in major inter-institutional benchmarking activity in June 2014, using the ACODE Benchmarks. The study was conducted eight months after the initial activity to understand how the institutions that had participated in the initial activity had used this to build their capacity for future growth. It will provide evidence of the longer-term value of this type of activity and will conclude with a series of recommendations on how an institution may apply this methodology to enhance its capacity to deal with the rapidly changing TEL space. Keywords: Benchmarking, technology enhanced learning, quality indicators, improvement. Introduction In June 2104 the Australasian Council of Online, Distance and eLearning (ACODE) facilitated a major Benchmarking Summit at Macquarie University in Sydney using the then recently reformatted ACODE Benchmarks for Technology Enhanced Learning (TEL). This was, without doubt, an unprecedented event within the Australasian higher education sector, with 24 institutions from five different countries coming together to Benchmark their capacity in this area of TEL. There were 15 Australian institutions, six from New Zealand, along with one each from the UK, South Africa and Fiji, present at the Benchmarking Summit. A paper describing this event in more detail, along with the finding of a major evaluation survey was presented as part of the 2014 ascilite conference proceedings (Author, 2014a). One of the reasons this event was so well attended is because benchmarking in the areas of TEL is becoming an increasingly important part of how many institutions are able to mediate a level of quality in their practices and then align this with the practice of other institutions. A further driver for this is that many of the issues being faced by Australasian universities, particularly in the area of quality in the online space, are coming under increased scruitin. This highlighted by Freeman (2014), who writes: Few university policy cycles include the value-adding stages of monitoring, evaluation and benchmarking. This suggests that many Australian universities [and by implication New Zealand universities] will face challenges meeting the Australian tertiary sector regulators’ requirements regarding evidence of implementation of policy, and improvement of policy over time. (P. 84) The purpose of the ACODE benchmarks for TEL (of which there are eight) has been to support the continuous quality improvement of many of the institutional practices around technology enhanced learning (ACODE, 2014). The approach adopted by this ACODE Benchmarking tool reflects an enterprise perspective, integrating the key issue of pedagogy with institutional dimensions, such as planning, staff development and infrastructure provision. These benchmarks have been developed for use at either an enterprise level, or by an organisational unit, and may also be used for self- assessment, or as part of a broader collaborative benchmarking activity. To participate in the Benchmarking Summit, held in June 2014, each institution had to first undertake a rigorous self-assess of their capacity in TEL against the embedded performance indicators (PIs) that are part of (used to validate) the Benchmarks. They then had to be willing to share that self- assessment with all the other institutions involved. As part of their commitment, each institution had to CP:185 537 participate in a minimum of two of the benchmarks, with some institutions doing three, four or five, and one institution doing all eight. During the summit, each institution took it in turns to briefly describe how they came to give themselves their particular rating. This, in many cases, generated quite lively discussion and debate as to why an institution gave themselves a particular rating and what would considered good/best practice. But more importantly, thanks to this debate and the open sharing of practice, each institution was then able to make a judgement on the veracity of their self-assessment. Here in lies the essence of the Benchmarking activity; having the opportunity to engage in broad ranging discussion around the PIs allows participants to form clear judgements as to the context of their own institutions practice, thereby allowing them to make qualitative determinations as to the acuracy of their self-assessment. Ultimately, the following two comments, in particular, exemplified the overall sentiment expressed by the participants of the Summit held in June 2014: “Great opportunity to meet and share where everyone is at. The benchmarking exercise is a great self reflective practice that is reinforced through the feedback and deliberation from other institutions” “I really enjoyed this Benchmarking Summit, I have learned a lot from the inter-institutional activity and will definitely be sharing and pushing for these benchmarks to be accepted at our institution. Thank you for facilitating this and look forward to the institution following up with the benchmarks in the future.” However, as with many activities of this nature, once it is all over people are inclined to go back to their day jobs and tend not to think too much about things like Benchmarking until it becomes time to do it all again. It was therefore seen as important to try and gauge a clear understanding of the longer-term benefits gained both individually and corporately by participating in, firstly undertaking the internal benchmarking activity and secondly attending the Benchmarking Summit. As a consequence, in March of 2015 a follow-up survey was provided to all those who attended the Summit to ascertain the level of follow-up activity that may have been generated by undertaking the benchmarking activity and participating in the Summit, once they had returned to their institutions. How this worked The twenty five participants responding to this survey, representing 22 institutions, had all previously undertaken the post-summit evaluation survey conducted in July 2014 and were all familiar with the approach being taken with the current online survey. The representatives from two institutions did not participate as the staff were no-longer at there institutions. The earlier survey had consisted of 30 questions, however, this current survey consisted of only seven questions, along with some basic demographic data to allow for the alignment with the previous collected data. Based on the fact that all the respondents had participated in the benchmarking activity eight months prior, each respondent was asked to: • reflect on their experience and on how useful, or otherwise, they felt it had been for both them personally and to their institution, • describe what they had formally done within your institution since the activity. e.g., had they written any formal reports to management? If so how were they received? Or was their participation in the event used more for internal purposes only? • describe how useful the follow-up documentation had been (the formal ACODE report on the benchmarking activity containing their data aligned with the data from the other institution involved), even though the data in the document had been anonymised. • comment on the proposition that ACODE would now formally facilitate a benchmarking activity every two years. • comment on how useful it would be for them to have access to an online tool to assist them in the collection and reporting of their institutional data. • comment on whether the data collected by that tool should be shareable with other participating institutions, and • lastly, although most of them only did some of the benchmarks, how useful was it to sit-in on all the discussions (which was the case), or should future events be broken-up into smaller CP:186 538 groups to try and streamline the activity, or was there more value to them in hearing what other institutions were doing across the other areas? Findings Eight months on and there was still a significantly positive affirmation of the usefulness of the benchmarking activity from both a personal and institutional perspective. All 22 institutions had, to some degree, continued their affiliation with the Benchmarks, using them as a way of base-lining their ongoing TEL activities, or by providing them a solid platform to advocate from. In some cases there had also bee follow-up activities within their institutions, while others stated that they would be returning to using the benchmarks again in the near future. About one third of the participants found that this activity served as a confirmatory activity, particularly in support of their current direction, while the other two thirds found that they had found them useful for providing evidence to their institution with a view to promote further growth. Importantly it was not just the Benchmarking activity that was seen as helpful, but the activities that each institution had to undertake to prepare for this activity, and then to align this with their ongoing strategic approach. To highlight this two institutions commented: “Many of our colleagues from across the campus who engaged with the workshop activity to set our initial scores were very positive and appreciative saying it was the best reflective engagement activity on their practice they had ever undertaken. We will use the ACODE data as a baseline measure of some of the benefits of our current refocus/ restructure” (4I3). “The strategic framework sitting under the benchmarks also gave me an idea of what it would look like if we as an institution were to adopt an integrated, strategic approach to the support of TEL” (4I8). When asked how the information had been fed back to the senior university management only in five of 22 instances had formal reports been written and presented to their senior management, although many had activly brought this to the attention of their senior managers. Far more common was the use of their findings for internal reporting and for informing their future practice, at a practical and stategic unit/department level. Not surprisingly, some institutions were undergoing some form of restructure, so the value was more localised, at the unit/department level. Indicative of the comments provided by participants: “I have fed back the comparative reports to the individuals who participated in the benchmarking process with me, and that was of interest to them, although I haven't followed up with whether they have taken it any further. I did provide a verbal and summarised report for my manager, although mostly this was for internal purposes” (5I18). To help assist institutions in their reporting, ACODE authored a formal report on the Benchmarking activity (Author, 2014b). This report provided institutions with a through description of the activity, nominated which institutions undertook which benchmarks and provided anonymised data from the institutions. When asked how useful this report was, the vast majority participants found this to be very helpful and where not overly concerned by the fact that the data had been anonymised. The following is indicative of this sentiument being expressed by the resondants: “It has been helpful because most of the time it is affirming to know that we are not the only ones who are struggling with some aspect or other. At other times, it is clear that we are outliers. It is always useful to get a sense of where one sits in comparison with others. The documentation has got us into good solid discussions at times” (6I15). When it was proposed that future benchmarking activities will be facilitated on a bi-yearly basis by ACODE, all the participants believed this was an excellent idea. More importantly, the proposed addition of an online tool to help them gather and collate their benchmarking data was overwhelmingly supported (95% agreement). However, in agreeing to this each institution was also very cautious about making their finding too transparent. Comments such as this exemplify the sentiment being CP:187 539 expressed: “It is important that we share our ideas and findings with others as long our anonymity as an institution is upheld” (7I22). “An online tool for collecting and reporting data would be invaluable. I don't see why it shouldn't be shareable with other participating organisations, provided the same confidentiality conditions apply as with the 2014 summit” (7I26). In the 2014 Benchmarking Summit all the institutions involved sat in on the discussions around all the benchmarks, regardless if they had done all eight or just two. In the initial evaluation some participants (those who had not undertaken many) suggested that this was too much, proposing alternative models of conducting the event, or lengthening the event (to 3 days) to allow for more discussions. It was therefore important to understand, prior to organising the 2016 event, if there had been a change in the participant’s sentiments, particularly since they had now now had more chance to reflect. On this occasion 77% of participants agreed that it was more helpful to sit in on all the sessions, 15% stated they would prefer more focused sessions, sessions focusing only on the benchmarks they participated in, while the remaining 8% made alternate suggestions. This posative sentiment was backed-up by comments such as the following: “The four we didn't do we got more learning from sitting in on those, as opposed to the four we had done as we already knew what we knew. But it would depend on how many were going to be there. It was definitely good to go to all of them” (8I2). “In our case, this proved to be even more valuable than sitting in on the benchmarks we had selected since we gained many new insights in a very short space of time” (8I12). Overall, the feedback received has provided ACODE with a clear way forward. Not only had the institutions benafited from the Benchmarking activity (self-assessment and particpating in the Summit), but they had also incorporated much of their learnings into creating a posative outcome for their institution. Conclusion Many of the issues we face in our institutions can be remediated by simply taking the time to self- assess against a set of quality indicators, like those found in the ACODE Benchmarks for TEL. However, when we then look to further extend that self-reflection, by sharing our current practice with those in similar circumstances, this provides the impetus for a truly dynamic learning activity. This study has confirmed that there are both short term and longer term benefits to undretaking a holostic benchmarking activity around the fast changing field of technology enhanced learning. This will become increasingly important as our regulatory bodies start to develop their formal measures for ensuring institutions are meeting their obligations to their students in relation to technology enhanced learning. An activity, like the Inter-institutional Benchmarking Summit that was facilitated by ACODE in June 2014, has provided the opportunity for many of the institutions involved to build relationships and stronger ties with their colleagues. In the broader context it has also provided these institutions with some of the wherewithal to meet the unique challenges of building a strong digital future. The ACODE Benchmarks for TEL have provided a catalyst to help make this happen for those who will take the time and subsequently benefited from the experience. References ACODE. (2014). Benchmarks for Technology Enhanced Learning. Retrieved from: http://www.acode.edu.au/mod/resource/view.php?id=216 Freeman, B., (2014): Benchmarking Australian and New Zealand university meta-policy in an increasingly regulated tertiary environment, Journal of Higher Education Policy and Management. 36(1), 74-87. Retrieved from: http://dx.doi.org/10.1080/1360080X.2013.861050 DOI: 10.1080/1360080X.2013.861050 TEQSA. (2013). TEQSA Information Sheet: eLearning and compliance with the Threshold Standards. Retrieved from: http://www.teqsa.gov.au/sites/default/files/TEQSAeLearningInformationSheet.pdf CP:188 540 Author, M. (2014a). Benchmarking for technology enhanced learning: taking the next step in the journey. In B. Hegarty, J. McDonald, & S.-K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014 (pp. 668-672). Author, M. (2014b). 'ACODE Benchmarks for Technology Enhanced Learning: Report on the refresh project 2014 and Inter Institutional Benchmarking Summit.' Australian Council on Online, Distance and E-Learning (ACODE). Sankey, M.D. (2015). Benchmarking for technology enhanced learning: Longer term benefits. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:185-CP:189). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:189 541 Building a framework for improved workplace assessment practice and better outcomes through online platforms Mark A Schier Louise Dunn Department of Biomedical & Health Sciences Swinburne University of Swinburne University of Technology, Technology, Hawthorn, Hawthorn, Australia Australia This paper discusses the development of an online platform used to build upon an existing system for assessing student workplace learning. It includes the background and rationale for the project, an overview of a rubric developed for the purpose of improving the understanding of the assessment criteria for all stakeholders. Our aim was to improve the pedagogical approach to student workplace learning in order to enhance learning outcomes for students as well as providing benefits to the university and workplace supervisors. To do this, we created a streamlined approach to assessment within the LMS at our university (Blackboard) enabling students to upload and submit their WIL portfolios. A more consistent process for academic supervisors to grade and provide timely feedback to the students, greater clarity in assessment requirements for students and workplace supervisors appears to have has been well achieved. Keywords: Work Integrated Learning, WIL, assessment, portfolio Background and rationale Work Integrated Learning programs (WIL) in contrast with classroom environments have learning objectives are vastly different to traditional classroom or other forms of learning (Cooper et al. 2010). One aspect that highlights the difference is the addition of a third stakeholder, the workplace organisation. The learning objectives are important as they direct learning in order to increase employability outcomes and provide evidence of what students have learnt on placement. Additionally, workplace evaluation is different to other forms of assessment as one critical element is self-evaluation by the student to demonstrate achievement of the learning objectives and the subsequent value of the placement (Biggs, 2003). To effectively and consistently assess submitted work, one method is the use of rubrics. These have an educational basis in clearly defining the important criteria along with a grading scheme which serves dual purposes: they allow students to make a pre-submission check of the readiness of their work for submission, and further they provide a clear framework for the assessment of the work by the examiner (Goodrich, 1996). This project built upon an existing framework of good pedagogical interaction with WIL for placements within the Public & Environmental Health and Biomedical Science programs (Author 1, year). This was particularly important, as students while on their placement are often isolated from traditional academic work in their year of practical learning. Others have reported the use of tools for placement activities (Nash et al. 2010; Hay, 2012; Shanahan, 2012). Approach An established portfolio paper-based assessment comprising of a number of components including a submission cover sheet, learning benchmark, experience record sheet and a series of project reports (Author 1, year) was used as the basis to develop the rubric and design the online submission platform. Although this assessment framework has been in operation for several years, the process for submission of the portfolio was identified as inefficient, resource intensive, environmentally poor and generally out-dated. The grading criteria were fairly under-developed, reducing the opportunity for quality and consistent feedback to students. Time pressures to meet deadlines and logistical issues also meant that good assessment practice was largely unshared. Once we had established that the constraints in terms of student access, supervisor requirements, health industry confidentiality and intellectual property issues could be managed via an online platform, we decided to utilise the existing resources of our LMS (Blackboard). The final portfolio was envisaged to be presented as a single CP:190 542 uploaded document with consistent marking criteria applied. Figure 1 illustrates the portfolio with some of the key elements shown. To plan the changes, a small working party met to consider the advantages and disadvantages of the proposed scheme and define the requirements to meet the constraints. The working party decided that using an electronic document portfolio was the best method of moving from the current paper- based current system. This would maintain the continuity of current portfolio objectives and meet the other constraints, and significantly reduce the administrative burden of managing distribution of portfolios and collection of feedback to return to the students. Additionally, it would allow sharing of current good assessment practice and feedback between academic supervisors through use of rubrics and improved moderation. A final benefit was the continuity of engagement for the students on placement, as they are well skilled in the use of Blackboard before taking their placement, but tend to lose some skills during their year away from the university. We faced the dilemma of creating a rubric with enough detail to capture the variety of the workplaces (hospital placement and public & environmental health) and the range of items in the portfolio, along with the keeping it simple enough to be meaningful and not too arduous for students or academic staff to utilise. The overarching goal of clarity and unambiguous criteria guided the creation of the rubric. Figure 1: Some key elements of the portfolio: learning benchmarks, experience record and reports. The outcome An assessment and feedback rubric was developed incorporating three levels of achievement (insufficient, developing and considerable). It was also designed to enable application to the various assessable components of the portfolio including the learning benchmark, experience record sheet, learning objectives, procedure, outcome, reflective summary, presentation, written expression, and overall. The rubric was also designed to be adjusted to add additional comments via an online drop down menu and facilitate the automatic recording of results in the students grade book, a feature not possible a using paper based system. Feedback from the student group at a mid-year workshop was enthusiastically positive. We have trialled the new online submission for semester 1 assessment (interim report in late July) and will refine the final submission rubric and submission process based upon student feedback from this semester (for submission in February 2016). Conclusions and future direction Initial indications, based on informal feedback from students and academics have been positive. A more consistent process for academic supervisors to grade and provide timely feedback to the students, greater clarity in assessment requirements for students and workplace supervisors appears to have has been achieved. Once the pilot submission stage is complete later in the year, we plan to carry out a more formal CP:191 543 analysis of the stakeholder responses. Future ideas include the possibility of digital authorisation and electronic verification of workplace skills evidence, and report endorsement from industry supervisors. References Biggs, J. (2003). Teaching for quality learning at university (2 ed.): The Society for Research into Higher Education & Open University Press Cooper, L., Orrell J. & Bowden, M. (2010). Work Integrated Learning: a guide to effective practice. Abingdon, Routledge. Dunn, L., Schier, M., Fonseca, L. (2012). An innovative multidisciplinary model for work placement assessment. Asia-Pacific Journal of Cooperative Education, 13(3), 135-145 Goodrich, H. (1996). “Understanding Rubrics.” Educational Leadership, 54 (4), 14-18. Hay, K. (2012). Moving down Stream: using e-technology to enhance social work field education. In M. Brown, M. Hartnett & T. Stewart (eds), Future Challenges, sustainable futures. Proceedings ascilite, Wellington 2012 (pp. 386–389). Nash, R., Sacre, S., Calleja, P. & Lock, J. (2010). Enhancing student learning in the workplace through developing the leadership capabilities of clinical supervisors in the nursing discipline. In C. H. Steele, M. J. Keppell, P. Gerbic & S. Housego (eds), Curriculum, technology & transformation for an unknown future. Proceedings ascilite Sydney 2010 (pp. 668–672). Shanahan, M. (2012). Mediated learning in the workplace: students’ perspectives on present and future value of knowledge tools. In M. Brown, M. Hartnett & T. Stewart (eds), Future Challenges, sustainable futures. Proceedings ascilite, Wellington 2012 (pp. 824–830). Schier, M.A. & Dunn, L. (2015). Building a framework for improved workplace assessment practice and better outcomes through online platforms. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:190-192). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:192 544 Promoting Critical Thinking in a Large Class through Outcomes-Based Approach by Means of an Audience Response System Teck Keong Seow Swee Kit Alan Soong Department of Biological Sciences Centre for Development of Teaching and College of Alice and Peter Tan Learning National University of Singapore National University of Singapore One of the first considerations that comes to bear in the design of a new course will inevitably be the learning outcomes. Some of the learning outcomes are specifically related to the subject matter while others may be more broad-based goals like the honing of critical thinking skills. The General Biology course that is offered at the National University of Singapore (NUS) is one such course in which the promotion of critical thinking skills is incrementally weaved into the various learning activities and assessment components of the course. The large enrolment of the course also necessitates taking into consideration the affordances of technology in the outcomes-based design of the course. This paper aims to share how the General Biology course, using the topic of fermentation as an example, could be designed using outcomes-based approach, with learning activities supported by an audience response system, in order to promote critical thinking in a large class setting. As this is a work-in-progress project, some preliminary findings from the feedback of the students of the course are presented here. Keywords: Outcomes-Based Education; Large Classes; Critical Thinking; Formative Assessment, Technology Introduction The General Biology course is a non-majors biology course that serves as a bridging course for those who are majoring in the Life Sciences but do not have a pass in A-Level Biology, as well as an elective course for non-Life Sciences students. Despite being offered every semester, including one of the special terms during the vacation, the enrolment for Semesters 1 typically ranges between 600 to 800 students. Inevitably, the challenge of crafting appropriate and yet logistically-manageable learning activities and assessment components would include the use of appropriate technology with the aim to better engage the learning of the students. According to Race (2010), the connections between the factors for successful learning may be compared to ripples on a pond, as illustrated in Figure 1. Using the ‘Ripples on a Pond’ model, which is based on the constructive alignment framework (Biggs, 2003; Biggs & Tang, 2007), the learning activities for each topic of the course were carefully scaffolded through design and development, and aligned with the intended learning outcomes, one of which is the ability to think critically, and to formulate and apply the concepts acquired to new contexts. CP:193 545 Figure 1: ‘Ripples on a Pond’ Model of Successful Learning (Race, 2010) While there are numerous definitions of critical thinking, in recent years, critical thinking has been defined as the development of ‘effective reasoning, interpretation, analysis, inference evaluation and the monitoring/adjustment of one’s own reasoning processes’ (Mummery & Morton-Allen, 2009). According to Hammer & Green (2011), many ‘universities and university teachers face increasing pressure to produce graduates who can think critically’, and they further indicated that many authors (Jones, 2007; Kirkpatrick & Mulligan, 2002; Paul et al., 1997) claim that a substantial number of university teachers ‘struggle to conceptualise and teach forms of critical thinking that are relevant for their specific disciplinary, teaching context’. An in-house audience response system developed by the Centre for Instructional Technology at NUS, known as questionSMS (qSMS), (Shyam & Musthafa, 2010) was used to support one of the learning activities, the in-class quizzes. Essentially, qSMS enables an instructor to receive responses during an in-class session (e.g. lecture or seminar) on a web browser without interrupting the flow of the class. When the service is enabled by the instructor, students will be able to send their responses (questions, feedback or answers) during the in-class session by accessing the Wi-Fi-enabled online system either through various mobile devices (e.g. smartphones, tablets or laptops) or short messaging service (SMS). Students will also have an opportunity to view and to vote for the responses of their classmates. For example, each question posed can be ranked in real-time based on the number of votes received. At the appropriate juncture during the in-class session, the instructor could address selected questions posed by the students. The polling feature can also be used the instructor to design higher-order questions for students to answer. Hence, qSMS serves as a useful tool to facilitate deeper learning. The following section provides an illustration on how, using the topic of fermentation as an example, the selected learning activities have been designed and scaffolded based on the intended learning outcome of promoting critical thinking, highlighting the use of the polling feature of qSMS in providing responses to good quality questions higher-order thinking through in-lecture quizzes. Besides in- lecture quizzes, other learning activities are also discussed. Vignette The intended learning outcomes for the topic of fermentation is that students will be able to describe the process of fermentation in living cells, identify the concepts behind the fermentation process, relate the process and concepts of fermentation with other energy-related biological processes in the cell, and employ the concepts to solve problems in various scenarios and settings. In addition, students should also be able to demonstrate the ability to think critically, formulate and apply the concepts to new contexts. Table 1 provides an overview summary of how the intended learning outcomes for the topic of fermentation were mapped out through the various learning activities. However the focus of this paper is on how critical thinking has been promoted through the design of learning activities which incorporates the use of qSMS for the in-lecture quizzes. CP:194 546 Table 1: Overview Summary of How Intended Learning Outcomes for the Topic of Fermentation Were Mapped Out Through Various Learning Activities Intended Learning In- In-Lecture In-Lecture Laboratory- Outcomes In-Laboratory Lecture Review Demonstration Based At the end of the topic, Discussions Quizzes* Questions Questions Assignments* students will be able to: a) describe the process of fermentation in     living cells; b) identify the concepts behind the fermentation     process; c) relate the process and concepts of fermentation with other energy-related    biological processes in the cell; d) employ the concepts to solve problems in various scenarios and    settings; e) think critically, formulate and apply the concepts to new      contexts. An asterisk (*) indicates that the learning activity involved the use of technology. In-Lecture Quizzes Quizzes are administered at appropriate junctures during the lectures using qSMS. The in-lecture quizzes consist of previous years’ final examination questions. Since the final examination of the course is an open-book exam, the questions are application-based higher order questions that require several steps of processing before arriving at the correct answer. The following example is one of the questions posed: The formation of dental cavities is due to the corrosion of the enamel layer of tooth surfaces by lactic acid. The lactic acid is produced by the bacteria that live as thin layers of sticky bacterial colonies on tooth surfaces known as dental plaques. The production of lactic acid may be best explained by the process of A. Aerobic cellular respiration by the bacteria as the enamel layer is rich in glucose from the food consumed. B. Alcoholic fermentation by the bacteria as the dental plaques are not very permeable to oxygen diffusion. C. Glycolysis by the bacteria as the enamel layer is rich in glucose from the food consumed. D. Anaerobic fermentation by the bacteria as the dental plaques are not very permeable to oxygen diffusion. E. Hydrolysis by the bacteria as the enamel layer is rich in glucose from the food consumed. The students were taught that there are two main types of fermentation processes, namely alcoholic fermentation in yeast cells and lactic acid fermentation in muscle cells. They were also told that, unlike aerobic cellular respiration, fermentation does not require oxygen. Since lactic acid is a product, the students were expected to eliminate options A, B, C and E as all the four processes do CP:195 547 not yield lactic acid. Furthermore, the statement that dental plaques are not very amenable to oxygen diffusion should help strengthen the choice of option D as the correct answer. The students were given about five minutes per question to submit their responses using qSMS, during which they were allowed to refer to their notes, check the internet, and even to discuss with those who were seated next to them. After the time limit, the instructor would disclose the answers. Explanations of how the answers may be derived, similar to the preceding paragraph above, would also be made known to the class (4th ripple of Figure 1). In addition to helping the students to recall what they have been taught, the quizzes are useful in helping the students to see and learn the processes involved in arriving at the answers, providing them with an opportunity to understand the thinking process of the instructor. Furthermore, as the answers were being explained, the students would also have the opportunity of interacting with the thoughts that they had when they were attempting the question earlier. In-Lecture Review Questions While explaining the fermentation process, the instructor had to review concepts that were already taught before. Instead of recapitulating the concepts, the instructor kept asking the class questions at appropriate junctures to help the class recall those concepts. As a result of the large class size, not everyone responds to the questions. However, it is highly probable that many do attempt to answer the questions mentally, if not orally. As such, the in-lecture review questions provide the students with an opportunity to make sense of their learning (3rd ripple of Figure 1) through the practice of answering questions (2nd ripple of Figure 1). In-Lecture Demonstration Questions To further reinforce the concepts taught, demonstrations are also held in the course of the lectures. Returning to the topic of fermentation, the instructor conducted a beer-brewing demonstration during the lecture (1st ripple of Figure 1). The instructor would pose questions as he added the various ingredients into the brewing tank, some of which were asked to help the students recall the concepts, while others to help the students to delve deeper into the topic. For example, the students were asked to predict what would happen if the lid of the tank was not properly closed. An answer directly related to the topic would be that the fermentation process might not occur since fermentation occurs in the absence of oxygen. However, the instructor would probe the students further to get them to come to the conclusion that there might also be a possibility of other microorganisms contaminating the brewing broth, resulting in other products. In-Laboratory Discussions Besides lectures, the learning activities of the course include laboratory-based practical sessions, in which the students deepen their learning by doing (2nd ripple of Figure 1). For the topic of fermentation, students were organised into groups of four and given the task of preparing the Korean pickled vegetable, kimchi (1st ripple of Figure 1). Besides the instructor, the students were also guided by well-trained teaching assistants, who would also use discussion questions (5th ripple of Figure 1) to help the students to relate what they were doing with what they had learned during the lecture. Laboratory-Based Assignments Additionally, the students are required to complete a graded assignment of 4 to 5 short-answer questions (2nd ripple of Figure 1) that are related to the topic of the practical after every laboratory session. The questions of the assignments serve to further develop the thinking skills of the students. For every question that a student had been unsuccessful in obtaining full marks, personalised feedback would be provided using the Gradebook tool of the in-house Learning Management System of NUS (4th ripple of Figure 1). Methodology and Preliminary Findings As bring-your-own-device (BYOD) open-book examinations were incorporated for the course, pre- CP:196 548 and post-exam online surveys were administered to solicit the perceptions of the students of for the course. One of the questions of the post-exam online survey was on whether students had found that the examination questions had helped them to either think deeper or provoke their thinking about the course. The questions of the survey were piloted with a few individuals and further fine-tuned before being administered to the students. Further to the survey, focus group discussion (FGD) sessions were held approximately 4 months after the BYOD exam. Open invitations were sent out to all the students for the FGD sessions, and a total of 4 sessions were held. To ensure that the students did not feel hindered in voicing their opinions, none of the instructors of the course were present during the FGD sessions. The instructor of the sessions led the participants in more elaborate discussions of the questions posed for the pre- and post-exam surveys. One of the questions asked during FGD session was how the course had helped them to think deeper and had provoked critical thinking. The responses of the online surveys and FGD sessions were collated and analysed, using the spreadsheet software, Microsoft Excel, and the text mining software, IBM SPSS Text Analytics. The results of the survey that was administered to the students of the Semester 1 2013 cohort found that 87.7% of the respondents (n=406) either agree or strongly agree, out of a 4-point Likert scale, that the course examination questions had helped them to think deeper and provoked critical thinking. The FGD question on how the course had helped them to think deeper and how the course had provoked critical thinking yielded the following key findings: Alignment of Learning Activities for Each Topic throughout the Course One of the feedback received from participants indicated that they appreciated that the lectures, laboratory sessions and examination questions were all interconnected for each of the topics. This implies that participants could relate the relevance of the various learning activities, including in- lecture quizzes, planned for the various modes of delivery for each of the topics. Questions Scaffolded Throughout the Course Many participants had in their feedback mentioned that they value the kinds of questions posed during face-to-face sessions, such as the laboratory sessions. Some commented that the kinds of questions posed by teaching assistants during laboratory sessions allowed them to think critically and helped to scaffold their learning. Demonstrations during Lectures Participants also appreciated the demonstrations presented during the lectures. For instance, real-life specimens such as transgenic fishes that fluoresced, plants and animal heart were brought to the lecture, and the participants also commented that the way lecturers presented the specimens with guiding questions engaged them at a deeper level. Bring-Your-Own-Laptop Examination Many participants also commented that the application-based multiple-choice questions that were posed for the BYOD final examination had triggered their critical thinking skills. One respondent commented, “I really liked how the questions tests us on our understanding of various biology concepts instead of questions just fully based on memory work”. Some had also given the feedback that media-rich comprehension-based MCQs helped them to appreciate the relevance of some biology concepts learnt in real-life application. One such example that was cited was the use of a news clip about the personal genomics company, 23andMe, for several of the final examination questions. Limitation of the study It is noted that in this work-in-progress preliminary study, there is less clear indication that qSMS has impacted students’ learning during in-class sessions. Similarly, more explicit questions could be posed during FGD. For the next round of study on this course, more questions focusing on students’ CP:197 549 perception on the use of qSMS during in-class sessions will be incorporated into the survey and FGD sessions. Students’ level of critical thinking before and after attending the course could also be measured in a systematic manner. This could carried out by administering the Cornell Critical Thinking Test Level Z (CCTT-Z), which has been described as a reliable and valid instrument in measuring critical thinking skills (Ennis et al., 2005). Next Step This preliminary study has provided an insight of how an outcomes-based course design (Race, 2010) to promote critical thinking has impacted learners, which is timely as literature has suggested the need to articulate the conceptualisation of critical thinking that is both discipline- and course- specific (Hammer & Green, 2011). Furthermore, Hammer & Green (2011) have also reported that studies on designing appropriate learning experiences to develop students’ critical thinking are still at an experimental phase. It is therefore proposed that a more in-depth evaluation study on the impact of such outcomes-based course design on students’ critical thinking skills be carried out. Considering that the course is being re-designed into a blended online course, studies on how critical thinking skills can be scaffolded, based Krathwohl’s recent revision (2002) of Bloom’s Taxonomy as one possible example, for such a blended online learning mode could be conducted. Acknowledgement The authors would like to thank NUS for the provision of a Teaching Enhancement Grant (WBS C- 154-000-049-001) and the very useful input from the other co-lecturers who had co-taught the course with one of the authors over the years. References Biggs, J. (2003). Teaching for quality learning at university: What the student does (2nd ed.). Maidenhead: Open University Press. Biggs, J. & Tang, C. (2007). Teaching for quality learning at university: What the student does (3rd ed.). Maidenhead: Open University Press. Ennis, R. H., Millman, J., & Tomko, T. N. (2005). Cornell critical thinking tests level X and level Z manual (5th ed.). Seaside, CA: The Critical Thinking Company. Hammer, S. J. & Green, W. (2011). Critical thinking in a first year management unit: The relationship between disciplinary learning, academic literacy and learning progression. Higher Education Research and Development 30, 303-315. Jones, A. (2007). Looking over our shoulders: Critical thinking and ontological insecurity in higher education. London Review of Education 5, 209-222. Kirkpatrick, A. & Mulligan, D. (2002). Cultures of learning: Critical reading in the social and applied sciences. Australian Review of Applied Linguistics 25, 73-99. Krathwohl, D. (2002). A revision of Bloom’s taxonomy: An overview. Theory Into Practice 41, 212-218. Mummery, J. & Morton-Allen, E. (2009). The development of critical thinkers: Do our efforts coincide with students’ beliefs? In The Student Experience: Proceedings of the 32nd HERDSA Annual Conference. http://www.herdsa.org.au/wp- content/uploads/conference/2009/papers/HERDSA2009_Mummery_J.pdf Paul, R., Elder, L. & Bartell, T. (1997). Study of 38 public universities and 28 private universities to determine faculty emphasis on critical thinking in instruction. In Research from The Center for Critical Thinking. http://www.criticalthinking.org/pages/study-of-38-public-universities-and-28- private-universities-to-determine-faculty-emphasis-on-critical-thinking-in-instruction/598 Race, P. (2010). Making learning happen: A guide for post-compulsory education (2nd ed.). London: SAGE Publications. Shyam, N. & Musthafa, Y. B. (2010). Question SMS – SMS based student feedback system. In Association of Pacific Rim Universities (APRU) Education and Research Technology Forum. http://apru.nus.edu.sg/_files/ert/1stERT/NUS_QuestionSMS.pdf CP:198 550 Seow, T.K. & Soong, S.K.A. (2015). Promoting Critical Thinking in a Large Class through Outcomes- Based Approach by Means of an Audience Response System. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:193-CP:199). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:199 551 Digital andragogy: A 21st century approach to tertiary education Rachel Sheffield Susan Ellen Blackley School of Education School of Education Curtin University Curtin University This paper revisits the term “andragogy” (adult education) and develops new ways of st working in tertiary education based upon an analysis of the skills and dispositions of 21 century learners through the lens of adult education, and the affordances of readily- accessible digital technologies. These ways of working constitute what we term “digital andragogy”. In order to engage and retain students and revitalise tertiary education, lecturers need to take account of the profiles of their learners and seek to create learning spaces that best suit their needs and wants. We posit that tertiary learners should be encouraged and supported to transition from pedagogical practices experienced in their school years to tertiary education contexts for learning that are grounded in digital andragogy. Described in this paper is a proof-of-concept project that is currently being undertaken with 88 undergraduate students in a Bachelor of Education Primary course. Keywords: digital andragogy, tertiary learners, digital affordances Introduction An increasing concern for educators in tertiary education is what they consider to be a lack of student engagement with course content (Biggs & Tang, 2007; Massingham & Herrington, 2006). Students approach their studies with a surface approach: enter the learning management system at the beginning of the semester, and then return to submit the assessments on the required due dates throughout the semester. They are more concerned in passing the units in the course and final certification, referred to as surface learning (Biggs & Tang, 2007) rather than their personal learning and the development of their professional identity (deep learning). Deep learning requires higher- order thinking, collaboration and conversation with peers, and reflection and feedback. In order for this to occur, learners need time to prepare, read widely, reflect, and communicate, and a disposition to do so. Most of the current learners in tertiary institutions were exposed to pedagogical practices throughout their primary and secondary years of schooling (McGrath, 2009), and as a result may expect the same practices to be used by their lecturers in the tertiary context. When mature adult learners are confronted with pedagogical approaches in their tertiary studies, existing predispositions to surface learning may emerge. Whilst surface learning and pedagogical practices may require less energy than deep learning and andragogical practices on the part of both the student and teacher, we believe st that neither is conducive to developing 21 century skills or profession-readiness. Profile of 21st century tertiary learners Survey data were collected from 1238 Bachelor of Education students over three years (2013 – 2015) to determine their use of and confidence with various digital technologies, and how they managed their studies and other life commitments. Whilst students engage with various digital technologies, it appears that their confidence and use extends only as far as their immediate needs: this includes the Internet, emails, social media (Facebook and twitter) and to a lesser extent YouTube. This finding is supported by Henderson, Selwyn, and Aston (2015, p. 10) who concluded that these are not the “creative, collaborative, participatory and hyper-connected practices” touted in the discourses of digital learning. We suspect that the high use of and confidence with email is in response to preferred communication means with the university, rather than a preferred way of contacting friends and family. Students are much less confident with the other nominated tools such as Dropbox, Wikis, Blogs, Keynote, and Vokis, and often do not use them. When they want to learn to use a new digital technology they do not look to the University or other formalised learning; they go to on-line tutorials, YouTube videos and the support and experience of others, either unknown on-line or known including CP:200 552 family and friends to help them. Digital technologies are used to acknowledge others and to form personal identities (Seely-Brown, 2004). Students demonstrate an approach to learning aligned with “what they need right now” (personalised) learning rather than the “just in case” (directed) learning of the past. We conclude that students want personalised flexible learning, and instantaneous, personally-directed feedback and communication. In regards to how they engage with their studies, the survey and interview data indicated that the ability to move in and out of the university landscape quickly and easily, leaving digital bookmarks to know what they have done, what needs to be completed and when, rank highly. Tertiary students prefer to multi-task rather than complete tasks in a linear fashion. Students’ lives are complex having many facets, and they are reluctant to deprive themselves of social contact, relaxation or hours of paid work in favour of a deeper commitment to their studies. They have a spread of technological competences and therefore a range of abilities to manage these aspects of their university lives as opposed to their socially-mediated lives. There is also a somewhat misguided or naïve belief about the tech-savviness of these students. We make many assumptions about their ability to solve basic technical issues including file managing, selecting browsers, accessing materials, and effectively navigating learning management systems. Digital andragogy Traditional teaching (using pedagogical practices) in tertiary settings no longer provides the best fit for the learners and their lifestyles, and does not adequately align with how knowledge is accessed and constructed in our Web 2.0 world. This is the optimum time and place to embrace andragogical practices within a digitally expanded educational context: which we coin as “digital andragogy”. Our st notion of digital andragogy draws on our profile of 21 century learners, the affordances of Web 2.0 st technologies, and the desire to promote 21 Century Learning Skills. Silva (2009) states that “an “emphasis on what students can do with knowledge, rather than what units of knowledge they have, is st the essence of 21 century skills” (p. 630). Whilst in the literature there are varied descriptions and st lists of what constitutes “21 Century Learning Skills”, there are four components that are consistent: Critical thinking, communication, collaboration, and creativity. Cleary these are not new skills, but perhaps the point is that they have new importance (Silva, 2009); they have been singled out as highly desirable assets for employability in our digital world. Our definition of digital andragogy, distilled by from our investigation and analysis, is “the practice of educators to equip and encourage adult learners to choose and use the affordances of accessible digital technologies to personalise their learning and facilitate their interactions with peers and tutors”. However to achieve this, we contend that particular ways of working need to be made explicit for both the educator and the learner. Table 1 provides details of these ways of working. Table 1: Ways of Working for Successful Digital Andragogy Educator actions Learner actions Navigation through the unit is scaffolded by Self-directed navigation through the content and “chunking” content and tasks. tasks is undertaken. The immediate application of learning is made Internal motivation is developed and personal obvious. progress monitored. Tasks and activities are designed to require Collaboration with peers occurs in teams with collaborative team work. complementary skill sets. Creative and innovative solutions and practices Past experience and prior learning is drawn are modelled. upon. Opportunities for creative development and Contextual creativity is developed. reflection are provided. Engagement with a variety of modes and Engagement with a variety of modes and mediums of communication. mediums of communication. Enacting digital andragogy To enact digital andragogy successfully, we suggest that the following principles are addressed by the unit designer: CP:201 553 • The learners are made very aware of the rationale for the non-pedagogical approach, and the expected ways of working (Table 1). • The learning modules are chunks of information/skills/strategies that encourage learner collaboration and reflection to construct meaning and connections to prior knowledge. • The assessment tasks serve three purposes, not just determination of grades. The tasks are assessment of learning, for learning, and as learning. • Task (formative and summative) feedback is prompt, personal, and provided in different formats (written, video, and sound bite). The one-year proof-of-concept (POC) project that the authors are conducting is piloting a digital andragogical approach in two related and consecutive units in the Bachelor of Education (Primary) at Curtin University undertaken by the same student cohort (N = 88, 10 male, 78 female). Pre- and post- unit implementation data have been collected for Semester 1, 2015 (anonymous online survey and semi-structured email interviews. The Learning Management System being used (Canvas by Instructure™) has been interrogated to investigate site analytics that will also contribute to a picture of the effectiveness of this approach to tertiary education. Briefly the characteristics of the digital andragogical approach taken in the POC are: • 5 mandatory Masterclasses as opposed to the traditional 12 tutorial sessions. Students can choose which timetabled Masterclass they attend. • the remaining 7 timetabled tutorials are for drop-in sessions: students may work in groups or receive individual attention from the tutor who is present for the whole time or choose not to attend and manage their work elsewhere. • unit content has been chunked into manageable portions & learning is demonstrated by the submission of weekly tasks. These are commented on and feedback provided by the tutor within one week, and they are not part of the summative assessment. • the LMS being used has important functionality: students can choose multiple ways in which they are notified of announcements, grades, and feedback: ranging from their university student email to Facebook and SMS messages to their phones; they also choose the frequency of the messages (from as soon as sent to once a week), the LMS also has an app for easy access. • the tutors have digitised information as much as possible using GoAnimations (© 2005 GoAnimate), Vokis (© 2015 Oddcast Inc), Kahoots (© Kahoot! 2014), and video clips. Findings The participant response rate was 96.6%, and in this concise paper we shall present the major post- unit implementation findings. Table 2 shows the level of agreement with the importance of various aspects of unit delivery. Table 2: Post-unit implementation survey data (Question 4: How important are the following to you?) Unit delivery aspect % strongly agree/agree Having access to my tutor 96 Being able to access my unit online 92 Being able to access my unit progression (know what I have done and what to 89 do next) Being able to attend any workshop in the week I want to 73 Attending workshops on campus 72.5 Accessing recorded materials (lectures) 56 Receiving notifications in multiple ways (SMS, Facebook, email) 55 Using my phone to access unit information through the app 47 Contributing to peer conversation (eg. Discussion board) 34 CP:202 554 Upon completion of the unit, students reflecting on whether or not the importance placed upon these functions of the LMS and mode of unit delivery had changed revealed that 51% indicated that it was more important for them to be able to access their unit progression, closely followed by 49% indicating that having access to their tutor was more important. Interestingly the lowest scoring aspect from this survey item as shown in Table 1 (contributing to peer conversion) revealed that 62.5% stated that there was no change in their opinion and 8% stated that this aspect was now less important. The mandatory weekly tasks (scaffolded chunking of unit content and reflection) were contentiously viewed; approximately two-thirds of the cohort valued them and understood the connection to their learning, whilst the remaining third considered them an imposition. From the survey responses, 87.7% indicated that the strongly agreed or agreed with the statement “The weekly tasks were related to my forthcoming practicum and future professional identity”, and the second highest scoring response was 79.5% agreement with the statement “The weekly tasks scaffolded my progress through the unit”. The purpose of the Masterclasses was to provide more flexibility for student engagement in the unit. Table 3 shows the high level of success of this mode. Table 3: Survey responses reflecting upon the Masterclasses Statement % strongly agree/agree The focus of each Masterclass was clear and relevant 98.6 The Masterclasses were engaging and student-focused 94.5 The Masterclasses allowed me to collaborate with my peers in real time 93 The 5 Masterclasses in combination with drop-in sessions supported my 90.5 learning and life-style commitments The schedule of the Masterclasses allowed me to make choices about my 87.8 attendance The unit delivery worked better for me than the traditional 12 weeks attendance 83.8 approach There were many positive quotes from students about the digital andragogical approach to the unit delivery that seem to support the ways of working identified in Table 1. Some representative quotes are: The notifications were great and also the ability to upload the weekly tasks and get feedback was good. The Masterclasses were GREAT, quality over quantity! This has been a very successful learning experience. Having achievable weekly tasks and readings to complete was something I found extremely useful, along with the accessible syllabus and module resources. I found it really effective having both the Masterclasses which were quite intensive but also having the time to sort of consolidate that learning and speak with you[rself]” which was a sentiment reflected by the majority of the interviewees. I thought it was good that we had weekly activities because it keeps you thinking about the unit and you don’t just forget about it for a few weeks until you come back for the next assignment” which related to the chunking of the unit content. Conclusions The survey data was drawn from on-campus as well as fully online students in this particular unit. Overall, satisfaction was high with the on-campus students and much less so with the online students. The primary reason for this was access to the Masterclasses; despite being recorded live during workshops using iLecture, feedback from the online students was that they felt excluded, could not hear the on-campus student responses, and had considerable “void periods” whilst the on-campus students were discussing or individually reflecting in the flipped classroom. The other main complaint was in regards to having to use both the Blackboard site (to submit assessment and receive summative assessment grades) and the Canvas site for unit implementation. We suspect that the CP:203 555 issue is one of time; typically the online students have work and family commitments, so having to use two LMSs and simultaneously become familiar with a new one, was viewed as time wasted. In regards to unit delivery using a digital andragogical approach, the steps we undertook in this project matched the ways of working outlined in Table 1. For the second professional studies unit that the on-campus students undertake in semester 2, some minor modifications have been made but the same methodology is being employed. The modifications include changing the cut-off time for weekly tasks, adjusting the course time and local time for submission on Canvas, and not recording the Masterclasses in real time, rather using Camtasia Studio (© 1995 - 2015, TechSmith Corporation) software to video and audio record the slide presentations (screen captured) with the tutor’s voiceover. This can then be saved and uploaded as an mp4 file for Regional online students to access as they require, and on-campus students to revise Masterclasses. The increasing use of digital spaces in tertiary education needs to be accompanied by negotiations between educators and learners to ensure engagement and deep learning; we believe this can be achieved by embracing digital andragogy. References Biggs, J., & Tang, C. (2007). Teaching for Quality Learning at University (3rd ed.). Berkshire, UK: McGraw-Hill. Henderson, M., Selwyn, N., & Aston, R. (2015). What works and why? Student perceptions of ‘useful’ digital technology in university teaching and learning. Studies in Higher Education. 1-13. DOI:10.1080/03075079.2015.1007946. http://newmediaresearch.educ.monash.edu.au/lnm/new- article-what-works-and-why-student-perceptions-of-useful-digital-technology-in-university-teaching- and-learning/ Massingham, P., & Herrington, T. (2006). Does Attendance Matter? An Examination of Student Attitudes, Participation, Performance and Attendance. Journal of University Teaching & Learning Practice, 3(2), 82-103. Retrieved 20 April, 2015 from http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1046&context=jutlp McGrath, V. (2009). Reviewing the Evidence on How Adult Students Learn: An
Examination of Knowles' Model of Andragogy. The Irish Journal of Adult and Community Education, 99- 110. http://eric.ed.gov/?id=EJ860562 Seely-Brown, J. (2004). Growing up digital: How the web changes work, education, and the way people learn. Retrieved 21 April, 2015 from http://www.johnseelybrown.com/Growing_up_digital.pdf Silva, E. (2009). Measuring Skills for 21st - Century Learning. The Phi Delta Kappan, 90(9), 630-634. Retrieved 18 April, 2015 from http://pdk.sagepub.com/content/90/9/630.full.pdf+html Sheffield, R. & Blackley, S. E. (2015). Digital andragogy: A 21st century approach to tertiary education. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:200-CP:204). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:204 556 Blended Learning Adoption Monitoring Simon Douglas Smith University of South Australia A debate exists regarding blended learning definitions; current research relies heavily on concepts developed in online and distance education contexts. A recent review of blended learning studies reveals that colleges and universities do not readily keep records of who teaches blended courses, and faculty are not fully cognizant of whether they are teaching in blended learning format (Skrypnyk et al, 2015). Driven by needs such as improved course delivery and student retention, tertiary institutions are strategically increasing their blended learning offerings, yet there exists no widely accepted reporting mechanism to monitor blended learning adoption. This paper introduces a practical method for monitoring blended learning adoption at an institution, and recommends an approach towards semi-automating the process. Keywords: Blended learning adoption, evaluation, monitoring Introduction Many tertiary educational institutions are currently developing and implementing digital learning strategies. These strategies encompass multiple modes of learning including the use of digital tools in more traditional on-campus modes through to online learning with no requirements for on-campus attendance. While these strategies embrace more contemporary use of technology and learning theory it is difficult to assess and evaluate the progress in increasing blended modes. For instance, some courses may appear to be “flipped”, or utilize technological resources to create active learning environments during lecture times, or may have included an array of interactive tools accessible via the online course site. Other courses may have barely scratched the surface when it comes to leveraging benefits that a blended learning approach may offer. How do we measure this effectively? How can we create a method to easily capture changes in blended learning adoption across programs and over time that assists us in making informed institution-wide decisions? Managing the adoption of blended learning to understand return on investment requires an effective way to monitor and report on the rate of change. This paper addresses two specific issues that will reduce barriers in Blended Learning Adoption Monitoring (BLAM) at an institutional level. The first is the consideration of an applicable blended learning definition for this context, and the second is the judicious demarcation of what does and does not constitute progressive ‘blended learning categories’ for the purpose of semi-autonomously monitoring that utilizes readily accessible resources. Defining blended learning contextually Three competing approaches to defining blended learning exist (Graham et al, 2014) and are summarised as follows: • Combining online and face-to-face instruction • Combining instructional modalities (or delivery media) • Combining instructional methods The first approach predominates research, with attempts at refinement such as an institutionally defined percentage of face-to-face time being replaced by online activities, or the percentage of content being delivered online (e.g. 0% termed traditional, 1%-29% termed web facilitated, 30%-79% termed blended, and 80% or more termed online) (Allen and Seaman, 2007). Other more descriptive attempts have been made such as the thoughtful integration of classroom and online learning experiences (Garrison & Kanuka, 2004), or the planned, pedagogically valuable integration of traditional and online activities (Picciano, 2006). However, descriptions of various blends do not explain why specific models work within certain contexts, therefore we are left to approximate a best- fit based approach on the goal we are trying to achieve (Graham et al, 2014). CP:205 557 Blended learning concepts are often grounded in either online or distance education while lacking their own theories to address blending itself (Skrypnyk et al, 2015). Garrison and Kanuka (2004) emphasise the importance of effective integration of traditional face-to-face and technology whereby we are not just adding on to an existing approach; indeed, some blends seem to transform instruction (Bonk & Graham 2005). Transformative potential intimates the synergistic affordances possible in blended approaches. A blended learning definition that assists in determining the rate of blended learning adoption should include a migratory path from the discrete components of traditional and online towards a synergistic combination that facilitates transformation of learning. For instance, a mobile learning activity can consist of traditional synchronous components, yet be mediated online to facilitate geographic flexibility. In this situation a level of transformation occurs via geolocation and sharing attributes that facilitate active learning, whilst also retaining some traditional face-to-face guidance. We cannot predict where along the continuum of traditional to online a transformative blend will occur, as this depends on how a blend is applied and not by the percentage of component parts. It is therefore of limited value to derive a scaled measurement from traditional, to a percentage of online, to transformed blended learning. We can, however, map an ordinal set of categories to represent in the broadest sense, traditional learning, an intermediate blended learning stage without transformation, then finally transformed blended learning. This method is not only inclusive of transformative blended learning, but also identifies a progressive stage towards it. The following section discusses a literature-based approach to determining appropriate blended learning categories to distinguish these three levels of adoption. Judicious demarcation of blended learning categories Various exploratory models have been formed that characterise categories of blended learning (Graham et al, 2014). Examples include models that categorise based on activity, course, program and institutional level blends (Graham, 2006, as cited in Graham et al, 2014), physical and pedagogical characteristics of blended learning (Sharpe, Benfield, Roberts and Francis, 2006), and pedagogical interventions distinguishing enabling, enhancing, or transforming blends (Graham and Robison, 2007). Other models exist that consider both physical and pedagogical structuring of blended learning. The need for a categorical approach that recognises at minimum an initial state (traditional), and a progressive state towards a synergistic state that transforms learning has been presented above. The approach determined via observational best-fit by Graham & Robison (2007) indicates that there is a progressive blended learning pathway of three categories; enabled, enhanced and transformed, and is considered as being strongly aligned to the above described need to determine blended learning adoption categories in that it represents a migration path from traditional learning to transformed blended learning. These three categories have been regarded as observable archetypes. Graham & Robison (2007) did not attempt to categorise all seventy case studies used to formulate these categories as it was regarded that each case did not neatly fit into one of the categories. In this paper an attempt is made to co-opt Graham & Robison’s three categories, with some modification to fit into a schema that can be applied to BLAM based upon analytics and course site category indicators. Graham & Robison (2007, p.96, 100, 104) use themes of scope, purpose and nature to illuminate criteria used to evaluate cases as belonging to the categories of enabled, enhanced or transformed blended learning; the enhanced category being divided into two levels. A summary of criteria is presented below: Transformed Blend: Large scope, purpose is to improve pedagogy, affordances move towards active learning. Enhanced Blend I: Similar to Transformed Blends although the scope is small. Enhanced Blend II: Small or large scopes that improve productivity within the traditional paradigm. For example: greater content provision, increased communication, flexible access to content, visual demonstrations. Enabled Blend: Focus primarily on providing access and convenience to students. The institutional wide blended learning adoption framework developed by Graham et al (2013) identifies a range of areas to address via three stages of adoption. These stages are Awareness/Exploration, Adoption/Early Implementation, and Mature Implementation. The categories CP:206 558 for each stage are listed as Strategy (purpose, advocacy, implementation, definition, policy), Structure (governance, models, scheduling, evaluation), and Support (technical, pedagogical, incentives). This framework represents a comprehensive picture of institutional-wide dependencies for successful blended learning adoption. However, for our purposes, we are only interested in the area of evaluation component within such a framework, and specifically in potential monitoring capabilities that can be semi-automated from available online resources. Evaluation measures include inputs such as quality standards, and outputs such as reported levels of satisfaction, student opportunities and achievement (Graham et al, 2013). While these measures address the wider institutional goals of a blended learning environment, without the inclusion of any course delivery specific data it is difficult to map these measures to practical course level implementations. Learning management systems are currently embedded within higher educational institutions and allow for increasingly sophisticated learning analytics capabilities. As educational technologies have an increased presence in active learning situations, we can anticipate access to richer learning analytics data that can further inform blended learning adoption status. We can therefore expect continual development of adoption monitoring that utilises increasingly available data to provide an aggregated, and therefore richer institutional wide overview of the state of blended learning. As such, this paper seeks to determine the extent to which course site information and associated analytics can be utilised to determine the adoption state of blended learning as a specific focus, rather than using the broader evaluation measures described above. This requires a rethink of the three blended learning categories outlined above in order to repurpose within the course site and associated learning analytics context. An observational best-fit process (Graham & Robison, 2007) determined blended learning categories that informs the institutional wide blended learning adoption framework. To determine evaluative measures for adoption monitoring based upon course site and analytics data, a similar observational best—fit process was conducted. This consisted of a review of four hundred course sites (approximating 40% of all courses offered during a single study period within an Australian university) to determine a practical approach of assigning courses to progressive levels within the three aforementioned blended learning categories. The review process was guided by site resources analytics that summarised what resources were in the course, followed by individual course site visits to determine how resources were being applied. Review observations resulted in a set of criteria for necessary modifications to the original three blended learning category descriptions, and a four star system for each category being derived. The Transformed Blend is described above as being large in scope, improving pedagogy, and containing active learning affordances. During the review process it was found that the scale of scope was difficult to determine from course site information alone. However, sophistication of scope can be approximated via an aggregate of transformed components wherein each transformed type identified receives a single star rating within a four star system. Pedagogy was considered to have been transformed via blended learning if there was evidence of affordances that are above and beyond what might have been achieved via traditional or online implementations alone. Examples include identification of technology integrated activities such as eSims (educational simulations), integrated role plays, gamification, integrated mobile learning, and even virtual classrooms to the extent that it integrates geographically disperse students and facilitate real-time interactions. Finally, the presence of active learning as a goal in the original criteria has been expanded upon, as it was considered to be just one of the many possible examples of improved pedagogy possible via transformation. A course exhibiting four differing types of transformative elements would be regarded as highly transformed (4 stars). The Enhanced Blend description above contains two levels, the first given as a smaller scale version of the transformed blend. In our modified transformed description there is a differing approach to describing scope, such that a four star system approximates sophistication of scope. Therefore, the first level of enhanced blend described becomes absorbed into the transformed blend category, and is likely to be represented via a low star rating. The second level is described as non-transformative enhancements such as greater content provision and video demonstrations that improve productivity. The course site review process determined observationally four types of productivity improvements: content inclusion, facilitated interactions, site navigation (look & feel), and personal presence to assist with course engagement. A course site that exhibits all four components would be regarded as highly enhanced (4 stars). CP:207 559 The Enabled Blend description above focuses on access and convenience. During course review observations it was regarded that the Enabled Blend best-fit representation would be via a four star system consisting of the following four criteria respectively: some course materials in the course site, all necessary course materials in the course site, guidance text associated with the course materials, and a brief course orientation in the form of introduction or welcome. A course exhibiting all four components would be regarded as highly enabled (4 stars). A summary of the modified description of blended learning categories for BLAM is presented below: Transformed Blend (BLAM): Primary purpose to improve pedagogy, affordances include synergistic delivery. Enhanced Blend (BLAM): Improved productivity within the traditional paradigm. For example: greater content provision, increased communication, flexible access to content, video with visual demonstrations. Enabled Blend (BLAM): Focus primarily on providing access and convenience to students. Enabled Enhanced Transformed Some course materials Look & Feel (navigation, Virtual Classroom available media) Peer assess online (e.g. All necessary course materials Content (external resources) SPARK) available Personal presence (e.g. video) Integrated learning (e.g. eSim, Guidance text Interaction (groups, sharing, gamification, mobile learning) Introduction or welcome peer assessment, chat, Other (please list) quizzes) Figure 1: Blended Learning Rating Form (abbreviated) 140 120 100 Number of Courses 80 60 40 20 0 -20 0 2 4 6 8 10 12 Cumulative Blended Rating (total stars ) Enabled Enhanced Transformed Rating 0 1 2 3 4 Rating 0 1 2 3 4 Rating 0 1 2 3 4 # Courses 35 23 28 65 249 # Courses 234 98 48 16 4 # Courses 372 25 3 0 0 Figure 2: Blended learning ratings distribution – SP2 2015 (~40% of all courses) The graph above provides a snapshot in time of the level of blended learning adopted institutionally as determined by the BLAM process. Data may also be viewed in different ways, for example at division level, program level, or individual course level, and over time. Mapping review data to learner, teacher, course, and curriculum analytics may reveal correlations that will automate part or all of the review process. Next Steps & Conclusion The current process is semi-automated as it is guided by analytics and uses a form. The next step is to seek correlations between the review process output and other existing analytics data to predict review results, thereby reducing or eliminating review effort. In conclusion, a blended learning adoption monitoring method has been devised that is non-invasive, utilises readily accessible resources, has the potential to increase semi-automated reporting, and is useful in providing progress reports during blended learning deployment initiatives. CP:208 560 References Alammary, A., Sheard, J., & Carbone, A. (2014). Blended learning in higher education: Three different design approaches. Australasian Journal of Educational Technology, 30(4), 440-454. Allen, I. E., Seaman, J., & Garrett, R. (2007). Blending in: The extent and promise of blended education in the United States. Newburyport, MA: Sloan Consortium. Bonk, C. J., & Graham, C. R. (Eds.). (2005). Handbook of blended learning: Global perspectives, local designs. San Francisco, CA: Pfeiffer Publishing. Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. The Internet and Higher Education, 7(2), 95-105. Graham, C. R., Henrie, C. R., & Gibbons, A. S. (2014). Developing models and theory for blended learning research. In A.G. Picciano, C. D. Dziuban & C. R. Graham (eds), Blended learning: Research perspectives 2, (pp.13-33). New York: Routledge. Graham, C. R., & Robison, R. (2007). Realizing the transformational potential of blended learning: Comparing cases of transforming blends and enhancing blends in higher education. In A.G. Picciano & C. D. Dziuban, Blended learning: Research perspectives, (pp.83-110). The Sloan Consortium. Graham, C. R., Woodfield, W., & Harrison, J. B. (2013). A framework for institutional adoption and implementation of blended learning in higher education. The Internet and Higher Education, 18, 4- 14. Picciano, A. G. (2006). Blended learning: Implications for growth and access. Journal of Asynchronous Learning Networks, 10(3), 95-102. Sharpe, R., Benfield, G., Roberts, G., & Francis, R. (2006). The undergraduate experience of blended e-learning: a review of UK literature and practice. York, UK: Higher Education Academy. Skrypnyk, O., Joksimovic, S., Kovanovic, V., Dawson, S., Gasevic, D., Siemens, G. (2015). This History and State of Blended Learning. In G. Siemens, D. Gasevic, & S. Dawson, (2015). Preparing for the Digital University. http://linkresearchlab.org/PreparingDigitalUniversity.pdf Smith, S.D. (2015). Blended Learning Adoption Montoring. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:205-CP:209). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:209 561 The value of digital critical reflection to global citizenship and global health Lee Stoner Massey University Auckland, New Zealand This paper will contend that digital critical reflection can play a key role in tackling contemporary global health concerns. More specifically, institutes of higher education can utilize study abroad to foster global citizenship, which in turn may empower students to become civically engaged and potentially drive social change. However, global citizenship, as an educational outcome, is optimally facilitated when educational experiences are married with appropriate pedagogy, including the shaping of subsequent understandings and actions with critical reflection. This paper will discuss a pre-existing global health study abroad course, and outline: (1) why critical reflection is an essential step to fostering global citizenship, and (2) how digital story telling is being utilized to enrich the critical reflection process. Keywords: digital stories; critical reflection; transformative learning; educational travel; mobile pedagogy INTRODUCTION There have been increasing calls (Lewin, 2009; Stearns, 2009), from both the political and academic arenas, to ensure the capacity of higher education students to think and act globally in order to effectively address political, social, economic, and environmental problems on a global scale. This paper seeks to recognize this call and extend it to include global health, which together with climate change are, arguably, the two greatest challenges facing humanity today. In particular, health systems are struggling under the escalating burden of chronic diseases, including diabetes and cardiovascular diseases, which in turn are driven be poor lifestyle choices. Worldwide there were 57 million deaths in 2008, 63% of which can be attributed to chronic diseases, with over 80% of these deaths occurring in low- and middle-income countries (Hunter & Reddy, 2013; WHO, 2015). Clearly, lives can be saved and the global economy would be much stronger if people did more to avoid poor lifestyle choices such as physical inactivity and unhealthy eating habits. Yet despite growing public awareness about chronic disease and the consequences of such lifestyle choices, chronic diseases continue to rise. Based on this observation, perhaps “personal”-responsibility is not the answer, and conceivably the answer is “global”-responsibility, manifested as global citizenship (Stoner, Perry, Wadsworth, Stoner, & Tarrant, 2014). Higher education and the process of internationalization can play a key role in the fight against chronic diseases (Stoner, et al., 2014). Specifically, institutes of higher education can utilize study abroad to foster global citizenship, which in turn may empower students to become civically engaged and potentially drive social change. However, global citizenship, as an educational outcome, is optimally facilitated when educational experiences are married with appropriate pedagogy, including the shaping of subsequent understandings and actions with critical reflection. This paper will discuss a pre-existing global health study abroad course, and outline: (1) why critical reflection is an essential step to fostering global citizenship, and (2) how digital story telling is being utilized to enrich the critical reflection process. WHAT IS GLOBAL CITIZENSHIP? Global citizenship, like other complex psychosocial concepts, being framed by a single definition does not typically capture the complexity of the phenomenon. Considering this, in conjunction with the highly contested and multi-faceted nature of the term global citizenship, there have been three key dimensions identified by (Schattle, 2009), which serve as commonly accepted denominators of global citizenship: (1) global awareness (understanding and appreciation of one’s self in the world and of world issues), (2) social responsibility (concern for others, for society at large, and for the CP:215 562 environment), and (3) civic engagement (active engagement with local, regional, national and global community issues). HOW DO WE FOSTER GLOBAL CITIZENSHIP? Nurturing a globally-minded citizen has typically been associated with a transformative learning experience (Bell, Gibson, Tarrant, Perry, & Stoner, 2014; Jack Mezirow, 1991). This includes those pedagogies that engage the student with alternative lenses, orientations, or points of view related to a complex issue (such as global health), ultimately leading to a change in perspective. Arguably, a key to transformation is educative experiences coupled with critical reflection (Dewey, 1938; Kolb, 1984). An experience without critical reflection is solely an experience, which does not necessarily provide an individual with the opportunity to shape perspective – it actually has the possibility of being mis- educative (Dewey, 1938). An educative experience should serve as a departure point for learning, not an end result, and subsequently should foster an opportunity for deeper inquiry of the questions borne from the experience and subsequent reflection (Dewey, 1938). THE IMPORTANCE OF CRITICAL REFLECTION Critical reflection is the mechanism by which students begin to make meaning out of their experiences and adjust their frames of reference. By engaging students in critical reflection, students are encouraged to ‘scratch below the surface’ and become “critically aware of how and why their assumptions have come to constrain the way they perceive, understand, and feel about their world” (Jack Mezirow, 1991). From this process it is plausible that a learner’s reinvestment in informed application can lead to greater sensitivity, stronger acumen, and more informed approach to the issues that are affecting the well-being of our communities both local and global. Simply stated, the attributes of an engaged global citizen do not just happen, they accumulate through an educative experience, conscious engagement, critical reflection, and informed application. THE VALUE OF SHORT-TERM STUDY ABROAD We assert that experientially based, short-term educational travel programs provide a relevant learning site for students to experience, grapple with, reframe, and reflect on issues global in nature— ultimately fostering the conditions necessary for transformative experiences (Bell, et al., 2014; Tarrant et al., 2014; Tarrant, Rubin, et al., 2014). Such programs may provide an experience of cultural immersion and exposure to values and beliefs that differ to students’ own beliefs, can highlight common challenges faced by all societies (Tarrant, Rubin, & Stoner, 2014), and can serve as the disorientating dilemma necessary to initiative perspective transformation (J. Mezirow, 1978). However, it is important to note that while the experience is indeed a key component of transformative learning, the experience must be coupled with “integrating circumstances” whereby students begin to search consciously and unconsciously for the “missing piece” (Clark, 1991). As stated above, the catalyst for this transformation is critical reflection. INTEGRATING TECHNOLOGY AND REFLECTION Digital storytelling can serve as a robust medium for capturing the essence of a student’s perspective and level of understanding by utilizing “multimedia tools to engage individuals in authentic learning experiences that provide real-world relevance and personal-value within a situated context” (Walters, Green, Liangyan, & Walters, 2011). Our position is not one that is anti-paper-based reflection; traditional forms of critical reflection can and do work in the context of short-term study abroad (Bell, et al., 2014; Tarrant et al., 2014; Tarrant, Rubin, et al., 2014). Nonetheless, we argue that reflective experiences can be further enhanced by using technologies and services many students are intimately familiar with and use on a daily basis (Figure 1). In this regard, digital storytelling can provide students with a louder, clearer voice, utilizing a presentational form (Taylor & Ladkin, 2009) to reflectively articulate themselves and develop the foundation of a civically engaged citizen. CP:216 563 Experience (A) 1 Global Citizenship (C) ? Critical Global Reflection (B) Citizenship (C) 2 Experience (A) 3 Digital Critical Reflection (B) Global Citizenship (C) + Figure 1. Pathway from experience to global citizenship. Pathway 1 presents the ‘just do it’ approach, where it is expected that experiential education (A) is sufficient to foster global citizenship (C). Pathway 2 couples experiential education (A) with a traditional critical reflection (e.g., paper-based) (B) approach, an approach demonstrated to lead to global citizenship (C) within the context of international education (Bell, et al., 2014; Tarrant, Lyons, et al., 2014; Tarrant, Rubin, et al., 2014). Pathway 3 replaces traditional critical reflection with digital critical reflection (C), an opportunity to meet learners on the platforms and forums where they live, communicate, and already engage, and subsequently enhance reflective process. Reproduced from(Perry et al., 2015). Reflective digital stories, when compared to traditional reflective journals, have been demonstrated to be more indicative of the impact experiences had on students’ learning and competency (Walters, et al., 2011). Walters et al., (2011) state: “While journals recorded a catalogue of events, the digital stories, even at the lowest-level of reflection, were more indicative of the impact of the experience… than journals” (p.49). While it has been clearly presented that critical reflection methods are imperative for students to make sense of experiences, the use of digital media could be a medium that provides students with familiar space to be authentic and true-to-self. Moreover, this forum has greater potential to take the learning experience beyond the classroom, helping students to connect with the global-community, and to potentially become truly engaged global citizens empowered with voices to evoke change (Perry, et al., 2015). CONCLUSION Arguably, personal-responsibility is not the answer to tackling contemporary global health issues. Conceivably the answer is global-responsibility, manifested as global citizenship. Institutes of higher education can utilize study abroad to foster global citizenship, which in turn may empower students to become civically engaged and potentially drive social change. Global citizenship, as an educational outcome, is optimally facilitated when educational experiences are married with appropriate pedagogy, including the shaping of subsequent understandings and actions with critical reflection. While traditional forms of critical reflection can and do work in the context of study abroad, reflective experiences can be further enhanced by drawing on mediums familiar to students in the form of digital storytelling. References Bell, H. L., Gibson, H. J., Tarrant, M. A., Perry, L. G., & Stoner, L. (2014). Transformational learning through study abroad: US students’ reflections on learning about sustainability in the South Pacific. Leisure Studies. Dewey, J. (1938). Experience and education. New York: Collier Books. Hunter, D. J., & Reddy, K. S. (2013). Noncommunicable diseases. N Engl J Med, 369(14), 1336- 1343. CP:217 564 Kolb, D. A. (1984). Experiential learning : experience as the source of learning and development. Englewood Cliffs; London: Prentice-Hall. Lewin, R. (2009). Transforming the study abroad experience into a collective priority. Association of American Colleges and Universities Peer Review, Fall, 8-11. Mezirow, J. (1978). Perspective Transformation. Adult Education, 28(2), 100-110. Mezirow, J. (1991). Transformative dimensions of adult learning (1st ed.). San Francisco: Jossey- Bass. Perry, L., Stoner, L., Schleser, M., Stoner, K., Wadsworth, D., Page, R., & Tarrant, M. A. (2015). Digital Mediums as Reflective Tools: Finding Appropriate Spaces for Students to Get Introspective. Compare: A Journal of Comparative and International Education, 42(2), 323-330. Schattle, H. (2009). Global citizenship in theory and practice. In R. Lewin (Ed.), The handbook of practice and research in study abroad : higher education and the quest for global citizenship (pp. 3-18). London: Routledge. Stearns, P. N. (2009). Educating global citizens in colleges and universities : challenges and opportunities. London: Routledge. Stoner, L., Perry, L., Wadsworth, D., Stoner, K. R., & Tarrant, M. A. (2014). Global citizenship is key to securing global health: the role of higher education. Prev Med, 64, 126-128. Tarrant, M. A., Lyons, K., Stoner, L., Kyle, G., Wearing, S., & Poudyal, N. (2014). Global Citizenry, Educational Travel and Sustainable Tourism: Evidence from Australia and New Zealand. Journal of Sustainable Tourism, 403-420. Tarrant, M. A., Rubin, D., & Stoner, L. (2014). The Added Value of Study Abroad: Fostering a Global Citizenry. Journal of Studies in International Education, 18(2), 141–161. Taylor, S. S., & Ladkin, D. (2009). Understanding Arts-Based Methods in Managerial Development. Academy of Management Learning & Education,, 8(1), 55-69. Walters, L. M., Green, M. R., Liangyan, W., & Walters, T. (2011). From Heads to Hearts: Digital Stories as Reflection Artifacts of Teachers’ International Experience. Issues in Teacher Education, 20(2), 37-52. WHO. (2015). WHO highlights need for countries to scale up action on noncommunicable diseases Retrieved March 23, 2015, from http://www.who.int/mediacentre/news/notes/2014/action-on- ncds/en/ Douglas, T., Mather, C., Murray, S., Earwaker, L., James, A., Pittaway, J., Robards, B., & Salter, S. (2015). The value of digital critical reflection to global citizenship and global health. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:214-CP:218). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:218 565 Authentic context as a foundation for gamification and game-based learning Hanna Teräs Marko Teräs Jarmo Viteli School of Education School of Information Systems School of Information Sciences Murdoch University Curtin University University of Tampere, Finland Engage learners, the results of these endeavours are varied and there is still limited understanding of the success factors and design principles of pedagogically meaningful gamified and game-based learning Gamified and game-based learning are becoming increasingly widespread in formal education. While the primary motivation for employing gamification and game-based learning tends to be the attempt to motivate and. This paper suggests that understanding the role of an authentic context is a useful starting- point for a meaningful gamified learning design. Drawing from human-computer interaction and educational research in situated and authentic learning it proposes the first steps for a roadmap towards a deeper understanding of the phenomena of gamification and game-based learning, venturing beyond the “fun factor”. Keywords: Authentic learning, context, gamification, game-based learning, interaction design Introduction Video games are growing as an entertainment phenomenon. Based on Entertainment Software Association report from 2015, 155 million people play video games in the United States (Entertainment Software Association, 2015). The average age of a game is 35 while only 26 precent are under 18. Records from Australia show similar results with 76 precent of adults playing, while the average age has risen during the last 10 years from 24 to 32 (Brand, Lorentz, & Mathew, 2014). As gaming has become more popular, the interest of other fields and industries towards digital games has also increased. Some have postulated we are living the time of wider implementation of various forms of virtual environments, games and ‘gamification’ also for non-entertainment contexts such as business and learning (Gregory et al., 2013; Johnson, Becker, Estrada, & Freeman, 2014). While the primary motivation for employing gamification and game-based learning tends to be the attempt to motivate and engage learners, the results of these endeavours are varied and there is still limited understanding of the success factors and design principles of these learning approaches. The “fun factor” can be difficult to capture in a gamified learning design, especially in a way that is simultaneously pedagogically meaningful. More research is needed in order to better understand what constitutes meaningful game-based or gamified learning. In this paper we suggest that the understanding of context is a key in designing game-based and gamified learning. Although the question of context is of a central importance in both game design and learning design, it has not been discussed in much depth in game-based learning/gamification literature. On the other hand, its role and importance has been earlier emphasised in human-computer interaction (e.g. Dourish, 2001; Greenberg, 2001; Moran & Dourish, 2001; Moran, 1994) and interaction design (Garrett, 2010; Svanæs, 2013). In this paper, the concept of context and its relevance for the pedagogy of games and game-like environments is discussed and implications on educational design are suggested. The aim of the paper is to propose the first steps for a roadmap towards a deeper pedagogical understanding of gamification and game-based learning and identify questions for further research. Games, gamification and education – venturing beyond the hype It is no wonder educators have turned their eyes towards video games. Video games have been claimed to contain various positive affordances from improving general decision making skills, spatial awareness, overall health and wellbeing, and variety of professional skills in a safe surrounding (de CP:219 566 Freitas, 2006; Fröding & Peterson, 2013; Susi, Johannesson, & Backlund, 2007). Games also tend to hold a property that formal education often lacks: they are known to keep players engaged and motivated for extended periods of time. Such user experiences have lead to understand the deep engagement in interactive games (Takatalo, Häkkinen, & Nyman, 2015) through the concept of flow (Csikszentmihalyi, 2014). Different approaches have been employed in the attempt of harnessing these user experience outcomes for learning purposes. Gamification is an approach that is gaining popularity at a fast rate. The term refers to the application of game elements in non-game contexts, such as education (Deterding, Dixon, Khaled, & Nacke, 2011; Johnson et al., 2014), usually with the prospect to improve students’ motivation and learning engagement (Hamari, Koivisto, & Sarsa, 2014). On the other hand, serious games, games that have been purposefully designed for a non-entertainment purpose, are also on the increase. Serious games attempt to reach a balance between fun and learning (Bellotti, Kapralos, Lee, Moreno-Ger, & Berta, 2013; Ott, Popescu, Stănescu, & de Freitas, 2013). Finding a functional relationship between game elements and learning has been a hot topic in the area of education and training (Kapp, 2012). In order to achieve it, for example traditional instructional design models, such as the ADDIE model, have been compared to and combined with game design (Becker & Parker, 2012; Buendía-garcía, García-martínez, Navarrete-ibañez, & Jesús, 2013). Yet, the “silver bullet” remains to be found. The development costs of game-based learning projects tend to amount on the high side, and many of the initiatives have failed to redeem the high hopes placed on them (Moreno-Ger, Burgos, Martínez-Ortiz, Sierra, & Fernández-Manjón, 2008). Furthermore, the research reporting effectiveness of applications in this area of study have been claimed to be filled with questions of validity (Susi et al., 2007). The most commonly used gamification strategies appear to be the incorporation of digital badges, rewards or points into the learning environment. However, it must be kept in mind that game-based learning – and gamification to an extent - combines games design and learning design. This is no easy and straightforward task and it should go without saying that a sloppy design with regard to one or the other will not result in effective and meaningful learning outcomes. Merely adding badges or leaderboards to traditional learning activities will hardly constitute quality gamified learning (see e.g. Kapp, 2012). As Gregory et al. (2015) point out, game mechanics that are applied without adequate pedagogical planning may turn out to be counterproductive and result in unintended consequences. Research in the actual impact of gamification is still sparse and sometimes methodologically restricted. Moreover, there are grey areas in definitions: serious games, gamification and simulations seem to sometimes be used interchangeably, which makes comparison of results challenging. The available research knowledge suggests that game and simulation-based learning shows promise. For example, D’Angelo and her colleagues (2014) examined 260 STEM simulation studies and found a total of 59 unique studies that were either experimental (i.e., random assignment with treatment and control groups) or quasi-experimental (i.e., not randomized but with treatment and control groups). The results from the meta-analysis indicated that, overall, simulations have a beneficial effect over treatments in which there were no simulations. However, the studies analysed consisted predominantly of science education at the K-12 level, suggesting that there is a need for a more robust pool of high quality research studies in other domains. Hamari, Koivisto and Sarsa (2014) conducted an extensive quantitative literature review in order to examine the effects of gamification. Their findings highlighted that the manifold nature of gamification often not regarded in related studies. They introduce two aspects that are of a central importance in gamification: context and qualities of the user. This paper concentrates on examining the aspect of context and its role in the design of gamified or game-based learning. Context in human-computer interaction Before the advent of gamification, the importance of context has been discussed when designing technological applications. In human-computer interaction it has been examined especially in the area of context-aware computing (Moran & Dourish, 2001), which aims to create seamless people, process, place and time appropriate computing applications. Dey, Abowd, and Salber (2001) proposed a definition of context as: any information that can be used to characterize the situation of entities (i.e., whether a person, place, or object) that are considered relevant to the interaction between a user CP:220 567 and an application, including the user and the application themselves. Context is typically the location, identity, and state of people, groups, and computational and physical objects. (p. 106) Humans make meaning through interaction (Dourish, 2001). At the same time, human perception is actively directed towards the world and its objects, and it is shaped by previous experiences (Svanæs, 2013). This means that context is dynamic and ever-changing (Greenberg, 2001), and under constant redefinition by those who act in it. Whenever a technological system is introduced to an existing context, the context will impact and change it (Moran, 1994). At the same time, changes in the context impact on how existing technologies might be used, valued and supported. As such, contextual understanding is an important part of user experience design that aims to support everyday practices (Garrett, 2010). Intentions, roles, time and place affect how users interact with a technology, and how they perceive it. Phenomenological and ethnographic descriptions, in addition to on-going design research, can provide rich accounts that can advice interaction design that supports everyday practices (Cilesiz, 2011; Greenberg, 2001). Authentic context in learning design and virtual environments The role of context is not only important in human-computer interaction, but in learning as well. The pedagogical model of authentic learning (Herrington, Reeves, & Oliver, 2010) has proved to be a useful foundation for learning design in different types of virtual environments (Teräs & Kartoglu, forthcoming; Teräs, 2014). The authentic learning framework provides practical guidelines for operationalizing pedagogical ideas deriving from situated learning (Brown, Collins, & Duguid, 1989). Situated and authentic learning models emphasise the contextualised nature of effective learning. The models have been developed to bridge the all-too-common gap between academic/school activities and the activities undertaken by practitioners in the actual contexts where the knowledge and skills will be used. (Brown et al., 1989; Herrington et al., 2010). While authentic learning is often associated with game-like environments, the concept of authenticity tends to be used rather lightly, typically referring to the visual realism of the 3-dimensional environment. Caird (1996) makes a distinction between physical and psychological fidelity in the design of virtual environment training systems, and points out that the aspect of physical fidelity tends to be overemphasised, even to the point of naivety. From a learning perspective, the psychological fidelity, or cognitive realism of the learning environment may be of a far greater importance (Herrington et al., 2010). The idea of cognitive realism puts the role of an authentic context in the spotlight: it is essential that the learning tasks activate similar thought processes and actions as the ones required in the actual real-life context. Suggestions for further research Drawing from previous research in the areas of human-computer interaction and education opens new avenues for investigating gamified and game-based learning. Specifically, it is crucial to shift the focus from the “fun factor” and the novelty appeal towards a more pedagogically-driven research agenda in order to find meaningful and sustainable ways of integrating gamified learning and educational games in the curriculum. Moreover, in addition to controlled environments, gamification would benefit from being studied in natural settings in order to gain a richer understanding of the complexities brought about by contextualization. Examples of research questions yet to be explored include the following: 1) What are the user qualities, intentions and roles that affect how learners interact with educational games and gamified learning? 2) How does curriculum as a context affect and change an educational game / gamified learning experience? 3) What are the design principles of authentic game-based / gamified learning? 4) What factors create an authentic context in game-based / gamified learning? CP:221 568 The research methodologies appropriate for such research questions include phenomenology, ethnography, case studies, grounded theory and design-based research. Discussion Understanding the role of context in both games and learning is crucial for the development of meaningful game-based learning. Merely attempting to convey curriculum content through a game and attempting to use gameplay to motivate students in consuming that content is not a very in-depth approach to game-based learning. In particular, it is important to draw attention to the education philosophical underpinnings that inform the development of game-based and gamified learning environments. Focussing on physical fidelity at the expense of psychological fidelity or cognitive realism of the learning environment may result in insufficient attention to the learning side of the design. Consequently, the learning environment may be based on traditional views of learning as memorizing content and teaching as instruction, instead of reflecting contemporary constructivist pedagogies. Knowledge construction always takes place in a context. In the words of Dourish (2001), it is “creation, manipulation and sharing of meaning through engaged interaction” (p. 126), and it is this very process that defines the context in which we operate. This has direct implications on learning: action creates understanding. The essence of an authentic context consists of processes, skills and actions that take place in a certain setting. Therefore, for game-based or gamified learning to be meaningful, it must allow for and promote actions and interactions that create understanding and meaning. There must be cognitive realism, as the authentic context is socially constructed through the actions and interactions that resemble real-life settings. This transcends striving for learning engagement, and also finding the “fun factor”, which some have debated as a dead end for games research (Calleja, 2011). Gamification and learning connect people from various disciplines and fields. These people are still unfortunately working too often in silos, inside their own disciplines and worldviews. In Where the Action Is: The Foundations of Embodied Interaction, Dourish (2001) proceeded to lay the foundations of multidisciplinary conversation between technologists, practitioners, designers and researchers to build interactive systems with higher quality. He underlined the value people with different disciplines and roles can bring to development. The authors of this paper recommend connecting expertise and understandings in authentic learning, HCI and game studies in order to gain a deeper understanding of meaningful game-based learning that goes beyond the engagement hype. References Becker, K., & Parker, J. (2012). Serious Instructional Design: ID for Digital Simulations and Games. Society for Information Technology & Teacher Education International Conference 2012 (pp. 2480–2485). Chesapeake, VA: AACE. Retrieved April 16, 2014, from http://www.editlib.org/p/39955 Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., & Berta, R. (2013). Assessment in and of serious games: An overview. Advances in Human-Computer Interaction, 2013. Brand, J. E., Lorentz, P., & Mathew, T. (2014). Digital Australia 2014. Gold Coast. Retrieved from http://igea.wpengine.com/wp-content/uploads/2013/11/Digital-Australia-2014-DA14.pdf Brown, J. S., Collins, A., & Duguid, P. (1989). Situated Cognition and the Culture of Learning. Educational Researcher, 18(1), 32–42. Buendía-garcía, F., García-martínez, S., Navarrete-ibañez, E. M., & Jesús, M. (2013). Designing Serious Games for getting transferable skills in training settings. Interaction Design and Architecture(s), 19(Winter 2013/14), 47–62. Retrieved from http://www.mifav.uniroma2.it/inevent/events/idea2010/doc/19_4.pdf Caird, J. K. (1996). Persistent issues in the application of virtual environment systems to training. Proceedings Third Annual Symposium on Human Interaction with Complex Systems. HICS’96, 124–132. Calleja, G. (2011). In-Game: From Immersion to Incorporation. Cambridge, Massachusetts: MIT Press. Cilesiz, S. (2011). A phenomenological approach to experiences with technology: current state, promise, and future directions for research. Educational Technology Research & Development, 59(4), 487–510. CP:222 569 Csikszentmihalyi, M. (2014). Flow and the Foundations of Positive Psychology: The Collected Works of Mihaly Csikszentmihalyi. Dordrecht: Springer Netherlands. Retrieved from http://link.springer.com/10.1007/978-94-017-9088-8 Deterding, S., Dixon, D., Khaled, R., & Nacke, L. E. (2011). From game design elements to gamefulness: defining gamification. Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments (pp. 9–15). ACM. Retrieved April 17, 2014, from http://dl.acm.org/citation.cfm?id=2181040 Dey, A. K., Abowd, G. D., & Salber, D. (2001). A Conceptual Framework and a Toolkit for Supporting the Rapid Prototyping of Context-Aware Applications. Human-Computer Interaction, 16(2-4), 97– 166. Dourish, P. (2001). Where the Action Is: The Foundations of Embodied Interaction. Cambridge, Massachusetts: The MIT Press. Dourish, P. (2004). What we talk about when we talk about context. Personal and Ubiquitous Computing, 8, 19–30. Entertainment Software Association. (2015). Essential facts about the computer and video game industry: 2015 sales, demographic and usage data. Retrieved from http://www.theesa.com/wp- content/uploads/2015/04/ESA-Essential-Facts-2015.pdf De Freitas, S. (2006). Learning in Immersive worlds: A review of game-based learning. Fröding, B., & Peterson, M. (2013). Why computer games can be essential for human flourishing. Journal of Information, Communication and Ethics in Society, 11(2), 81–91. Retrieved March 30, 2014, from http://www.emeraldinsight.com/10.1108/JICES-01-2013-0001 Garrett, J. J. (2010). The Elements of User Experience: User-Centered Design for the Web and Beyond (2nd. ed.). Berkeley, CA: New Riders. Greenberg, S. (2001). Context as a Dynamic Construct. Human-Computer Interaction, 16(2), 257– 268. Gregory, S., Gregory, B., Reiners, T., Hillier, M., Lee, M. J. W., Jacka, L., & Larson, I. (2013). Virtual worlds in Australian and New Zealand higher education: Remembering the past, understanding the present and imagining the future. Electric Dreams. Proceedings ascilite 2013 Sydney (pp. 312– 324). Retrieved from http://www.ascilite.org/conferences/sydney13/program/papers/Gregory, Sue.pdf Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does Gamification Work? -- A Literature Review of Empirical Studies on Gamification. 2014 47th Hawaii International Conference on System Sciences, 3025–3034. IEEE. Retrieved from http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6758978 Herrington, J., Reeves, T. C., & Oliver, R. (2010). A guide to authentic e-learning. London and New York: Routledge. Johnson, L., Becker, S., Estrada, V., & Freeman, A. (2014). Horizon Report: 2014 Higher Education. The New Media Consortium. Retrieved from http://cdn.nmc.org/media/2014-nmc-horizon-report- he-EN-SC.pdf Kapp, K. M. (2012). The Gamification of Learning and Instruction: Game-based Methods and Strategies for Training and Education. San Francisco, California: Pfeiffer. Moran, T. P. (1994). Introduction to This Special issue on Context in Design. Human-Computer Interaction, 9(1), 1–2. Moran, T. P., & Dourish, P. (2001). Introduction to This Special Issue on Context-Aware Computing. Human-Computer Interaction, 16(2-4), 87–95. Moreno-Ger, P., Burgos, D., Martínez-Ortiz, I., Sierra, J. L., & Fernández-Manjón, B. (2008). Educational game design for online education. Computers in Human Behavior, 24(6), 2530–2540. Noussis, A. (2012). The Mechanics of Gaming & Learning: Intersecting Paradigms Through Design. Ott, M., Popescu, M. M., Stănescu, I. A., & de Freitas, S. (2013). Game-Enhanced Learning: Preliminary Thoughts on Curriculum Integration. New Pedagogical Approaches in Game Enhanced Learning: Curriculum Integration (pp. 38–59). Retrieved from http://www.scopus.com/inward/record.url?eid=2-s2.0-84898154495&partnerID=tZOtx3y1 Susi, T., Johannesson, M., & Backlund, P. (2007). Serious Games: An Overview. Technical Report HS- IKI -TR-07-001 (Vol. 73). Skövde, Sweden. Retrieved July 1, 2014, from http://www.autzones.com/din6000/textes/semaine12/SusiEtAl(2005).pdf Svanæs, D. (2013). Interaction Design for and with the Lived Body: Some Implications of Merleau- Ponty’s Phenomenology. ACM Transactions on Computer-Human Interaction, 20(1), 1–30. Takatalo, J., Häkkinen, J., & Nyman, G. (2015). Understanding Presence, Involvement, and Flow in Digital Games. In R. Bernhaupt (Ed.), Game User Experience Evaluation. Cham: Springer International Publishing. CP:223 570 Teräs, H. (2014). Collaborative online professional development for teachers in higher education. Professional Development in Education, 1–18. Routledge. Retrieved from http://dx.doi.org/10.1080/19415257.2014.961094 Teräs & Kartoglu (2015). Authentic learning with technology for professional development in vaccine management. Manuscript submitted for publication. Teräs, H., Teräs, M., & Viteli, J. (2015). Authentic context as a foundation for gamification and game- based learning. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:219-CP:224). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:224 571 A gamified eLearning approach to teaching food regulation Danielle Teychenne Deakin University Australia Knowledge of food regulation in Australia and New Zealand is fundamental for higher education nutrition students. Despite its importance, students are often disengaged with the learning content as it involves legislation, regulatory bodies, complex application procedures, food safety testing and political debates that often dismiss scientific fact. At a university in Victoria, students were taught this content in a passive, 2-hour, face-to-face lecture. This lecture did not provide any active learning opportunities for the students to apply their newfound knowledge. This paper describes a proposed pilot project to address learner disengagement through a gamified eLearning tool, The Story of Hemp. This digitally immersive teaching approach aims to reengage students with a real world context for their learning, leaving them with a greater sense of identity and significance as budding nutritionists. Keywords: eLearning, gamification, student engagement, game-based learning, narrative-based learning The learning problem: student disengagement Nutrition students are required to develop an understanding of the complex regulatory structures and processes of the Australian and New Zealand food industry. Traditional delivery of this content has involved a passive learning experience that has not engaged students, resulting in superficial learning outcomes. A partnership between academic staff and the digital resource team at a university in Victoria set out to transform the student experience of the topic through the application of multimedia resources designed to provide opportunities for students to actively participate in their learning about the food regulatory environment. Popular culture, and the controversy surrounding the use of hemp as a food product suggested this would make a lively and engaging topic for the exploration of the food regulatory process. The processes involved in the regulation and legalisation of such a product would generate interest for, and intrigue the students. An animated video was originally proposed as the medium, but after considering the script, the content was still uninteresting and un-engaging. In order to improve engagement, the academic and resource teams proposed that students needed to play an active part in learning about food legislation and its political processes. Our plan was to enable students to become virtual stakeholders in the food regulation environment; as a result, an eLearning project was decided upon, The Story Of Hemp eLearning game was developed. Why gamified eLearning? Games-based learning was chosen as it had flexibility to incorporate a wide range of user interactions and learning activities drawn from game design. Game design elements such as role-play, narrative and reward have been embedded to encourage and motivate students. Students begin the module in a realistic environment, inhabiting the role of a graduate nutritionist. Their aim is to solve a number of strategically placed real world problems. These learning challenges begin when the user is introduced to the main character Gary, a health food shop owner from Nimbin. Upon introduction, Gary expresses his desires to sell hemp products for human consumption but is restricted by legislation. This sparks a journey to Canberra to campaign for legislation change. Gary has no experience with the food regulation system and so the user must guide him by making all decisions and providing solutions. In order to provide correct information, students must actively search for and utilise relevant information that is stored in the form of documents, websites, articles and videos within the eLearning module. The main character’s lack of knowledge creates an expectation for students to teach. This expectation can influence higher levels of knowledge retention compared with the expectation to be tested (Nestojko et al, 2014, p.1045). CP:225 572 Upon completion of a learning activity, students are rewarded for their efforts with a digital badge and encouragement from the main character. These situation based learning activities are encompassed in a risk-free setting for learners to explore and develop their newfound knowledge. This provides a safe environment for students to practice new skills without worry of failure or consequences (p.48, Kapp, 2012). It is aimed that through role-play, students will build confidence to demonstrate knowledge and skills as nutritionists in real world settings (p.149, Kapp, 2012). By incorporating game design elements in a non-game context (Werbach, 2015), we are able to provide an enjoyable and memorable learning experience for the learners. Narrative Narrative has been incorporated to enhance content engagement by including the user in the story and allowing them to explore narrative pathways. Whitton and White (2012, p.45) describe narrative as a device that encapsulates story. Story consists of a number of events, settings and characters, with a clear beginning, middle and end (Whitton et al. 2012, p.45-46). In this eLearning experience, the user is immersed in the story from the beginning and plays an important decision making role until the end. Upon meeting the main character Gary, the user is guided through a number of scenarios in which they are presented with a range of pathways. These are decision-making points that provide the learner with an opportunity to explore the influence of their actions and solidify knowledge acquisition. This grants power to the learner as they “are no longer passive observers of the story, they are agents within it”(Whitton et al. 2012, p.46). Despite the overall storyline following a set structure, giving the user partial agency will provide a more memorable experience for knowledge retention and future application. Teaching others In this eLearning experience, users play the role of a graduate nutritionist but they are also expected to take on the role of a teacher. The main character Gary has been deliberately designed with no experience of the Australian and New Zealand food regulatory framework. Gary must rely on the students to teach him the processes of the food regulation system and make informed decisions for him based on their newfound knowledge. Students will learn deeply and recall more when they know they will soon need to teach the material to someone else (Nestojko et al. 2014, p.1046). By prompting the students to teach the main character, it is providing them with “authentic and purposeful tasks that map on to real world activities”(Whitton et al, 2012, p.11). This approach is designed to be an active learning experience for students that provides context to build confidence in their knowledge and abilities, which will develop their identity as a future graduate nutritionist. Interactive environments Throughout the eLearning module, students can interact with digital environments and learning activities that produce auditory, animated or written feedback (see Figure 1). This feedback guides the learner to make the right decisions to progress the game. Some learning activities include building an application to change the food standards code by dragging and dropping contents (See Figure 3); interviewing community stakeholders by clicking on provided questions and navigating the main character around Canberra by clicking the correct location. The ability for user actions to affect change within situations and environments of the game is powerful as it gives freedom to the learner to explore. Interactions that don’t contain learning activities are also included to help students navigate around the environment and maintain interest for the duration of the game In Figure 1, the game instructions state: “Click on the health food shop to go inside”. If the user decides against these instructions and clicks on the Kombi van, its horn will beep twice. From this auditory feedback, students will understand that this action does not affect physical change in the environment and attempt another action. “Feedback for actions is essential to be able to reflect on their effectiveness and modify them for future occasions”(Whitton et al. 2012, p.16). These interactive activities and environments provide students with a far more active learning experience in comparison with a face- to-face lecture. CP:226 573 Figure 1: Interactive environment set in the town of Nimbin Content investigation Relevant learning content is available in a repository within the game called the ‘library resource’, which is located at the bottom of the screen at all times via the dashboard tab (see Figure 1 and 2). The library resource contains relevant documents, websites, videos and news articles. Students must pull content from the library resource when they are prompted by learning activities and challenges. “Instead of focusing on creating a universal design that pushes…content, [the game is focusing] on crafting the right types of reasons a person needs to pull the content” (Kuhlmann, 2009). Digital learning activities include drag and drop interactions (see Figure 3), short answer questions and multiple choice questions. These activities are designed to motivate students to further investigate regulatory documents by providing real world context through the game’s use of narrative. This is a more effective alternative to bombarding students with endless documents and articles with no goals or direction. Students have real world reasons to motivate them to investigate learning content. Figure 2: Library resources CP:227 574 Figure 3: Drag and drop application writing activity Digital badges Students have the ability to earn digital badges within the eLearning game. These badges provide short-term goals; illustrate overall progression towards completion and hallmark real world knowledge and understanding. Students can earn six badges within the eLearning experience, with the first badge being achievable easily and early. This badge is titled the “Food stakeholder” badge and in order to receive it, students must research which government body they need to visit in order to start the campaign to change hemp food legislation. Gary’s representation as a normal citizen empowers the user in becoming a change agent in the Australian food regulatory system. This badge is received early in the game to give users confidence in their abilities and sense of achievement. The frequency at which these badges are distributed is spaced out purposely to maintain a consistent level of motivation and sense of achievement from the learner. Muntean (2011, p.328) stresses the importance of using digital badges to compensate students for their academic achievements, especially difficult tasks or exercises. It is hoped that these badges will encourage students’ perseverance to complete the eLearning module. Research method The research method that has been chosen is a mixed methods approach. Students will be aware that research is taking place. The control and experimental group will be chosen randomly and contain 100 students. The control group will be subject to the normal teaching methods (one 2-hour passive lecture presenting food regulation/legislation/associated political debates). With the experimental group being exposed to the new eLearning unit (30-45 minutes duration). Upon completion of these teaching approaches, all participants will complete a closed book, in class test that will consist of 10 multiple choice and 2 short answer questions. These questions will address food regulation processes and their associated political debates as presented in both the lecture and eLearning unit. This test will investigate levels of knowledge retention for teaching approaches across both student groups. Results from both groups will be compared and analysed for any differences. In the interests of fairness, students from the control group will have access to the eLearning unit after the in class test is completed. Qualitative data collection will also involve focus groups. Both the control and experimental group will have the opportunity to volunteer and assess their feelings about the use and value of the eLearning unit and its various components. Conclusion This interactive eLearning game has several learning outcomes that are relevant to the nutrition industry. It is anticipated that by exposing students to the dealings between different regulatory bodies, politicians and stakeholders they will gain a first hand experience of the environment of food regulation. It is intended that students will see the food regulatory system from different perspectives, CP:228 575 including that of the main character and various campaign stakeholders. Students will be prompted to investigate the hemp foods debate in a wider context by interviewing certain characters. Specific application timelines that food regulation bodies follow will be apparent through the illustrated passing of time. Political debate on food regulation will be presented through short videos and narrative turns in the story. It is hoped that students will live through eLearning as if they are in a graduate nutritionist internship role. It is hypothesised that this pilot study will provide positive results regarding an eLearning teaching approach when compared with a lecture-based approach. It is expected that the eLearning game will provide higher student engagement and knowledge retention through the use of gamification elements such as narrative, badges, interactive environments, role-play and feedback. Acknowledgement The author wishes to thank Dr Julie Woods for her support, collaboration and conception of this eLearning project. References Kapp, Karl M. The Gamification Of Learning And Instruction. San Francisco, CA: Pfeiffer, 2012. Print, (p. 48, 149) Are your E-learning courses pushed or pulled?, Kuhlmann, T. (2009, May) [Blog post] http://blogs.articulate.com/rapid-elearning/are-your-e-learning-courses-pushed-or-pulled/ Muntean, Cristina Ioana. 'Raising Engagement In E-Learning Through Gamification, ICVL Models And Methodologies'. Proceedings of The 6th International Conference on Virtual Learning (2015): 328 http://icvl.eu/2011/disc/icvl/documente/pdf/met/ICVL_ModelsAndMethodologies_paper42.pdf Nestojko, J., Bui, D., Kornell, N., Bjork, E., (2014) Expecting to teach enhances learning and organisation of knowledge in free recall of text passages, Memory and Cognition V.42: 7 (p.1045- 1046) Coursera, Werbach, K. (2015). Gamification [Video file] https://www.coursera.org/course/gamification Whitton, Nicola, and Alex Moseley. Using Games To Enhance Learning And Teaching. New York: Routledge, 2012. Print. (p.11,16) Image attributions Figure 1 Pepper heart, Darya Pino, https://www.flickr.com/photos/summertomato/3540719101/ , Creative Commons license 2.0 Palm Tree - Outrigger Kanaloa at Kona October 2014, Ron Cogswell,https://www.flickr.com/photos/22711505@N05/16361503190/, Creative Commons license 2.0 Blue Sky 2, Fabio Marini, https://www.flickr.com/photos/fabiomarini/1340979055/, Creative Commons license 2.0 Nimbin, Brian Yap, www.flickr.com/photos/yewenyi/3021716830/, Creative Commons license 4.0 © Shopfront in the sun – classic store front with colored awnings, ConceptWDollar Photo Club © Shopfront with two doors and large windows. White store facade, ConceptW/Dollar Photo Club © Old sport car, Konstantinos Moraiti/Dollar Photo Club © Classic VW 21 Window Mini Bus Image, Keith Webber Jr/123rf 2015, Danielle Teychenne, Deakin University Teychenne, D. (2015). A gamified eLearning approach to teaching food regulation. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:225 –CP:229). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:229 576 Pre-service teachers’ reflections on their participation in 1:1 laptop programs Rebecca Maria Walker Susan Ellen Blackley Curtin University Curtin University Australia Australia A number of government and non-government schools have implemented a one-laptop- per-student (1:1) policy. Whilst there was initial interest in the implementation of these programs, little has been done to track the uptake of digital learning technologies afforded by access to the laptops. This study examined tertiary students’ reflections on their experiences with 1:1 laptop programs after graduating from secondary school and at the commencement of their Bachelor of Education course. It is an extension of a previous study conducted by the researchers (authors, 2015) that presented findings about teachers’ use of laptops in 1:1 laptop program schools. The objectives of this second-phase research were to: • Capture recollections of the students’ experience of 1:1 laptop programs • Categorise these recollections into positive and negative experiences • Investigate the impact of 1:1 laptop programs on students’ perceptions of teaching with ICTs and their personal learning at University. Keywords: ICTs, laptops, pre-service teachers Background One-to-one laptop initiatives in schools have been expanding significantly over the last decade. This is due to a number of factors: less expensive hardware, improved Internet connectivity, and promotion by governments and education authorities. Inspired by the Digital Education Revolution Policy document released in 2007 (Rudd, Smith, & Conroy, 2007) many secondary schools throughout Australia opted to participate in the revolution with the intention of equipping every student in Years 9 to 12 access to “world class information and communications technology” (p.1). Some schools made the decision to implement a 1:1 laptop policy throughout the year levels 8 – 12, and many independent schools placed the onus on the provision of these laptops squarely on the shoulders of the parents, mandating that students bring these for every class. There is a body of research around the uptake by teachers of digital technologies in secondary schools (e.g., Handal, Campbell, Cavanagh, Petocz, & Kelly, 2013; Hsu, Wu, & Hwang, 2007; Kopcha, 2012; Mumtaz, 2006; Sang, Valcke, Van Braak, & Tondeur, 2009), and a growing body of research specifically designed to investigate the use of laptops (e.g., Inan & Lowther, 2010; Penuel, 2006; Rosen & Beck-Hill, 2012; Weston & Bain, 2010). The use of laptops by students at the direction of teachers has mostly involved note taking, assessment writing, homework, organisation, drill practices, communication and searching the internet (Authors, 2015; Keengwe, Schnellert, & Mills, 2012; Penuel, 2006). Whilst research has been conducted on teacher uptake of 1:1 laptop programs and, to a lesser degree, some measure of student outcomes whilst at secondary school, little investigation has been conducted in regards to an examination of the ongoing impact of the school experience on students entering pre-service teacher higher education. This is a key consideration as teacher beliefs based on past experiences is reported as a major challenge to technology effective integration (Mouza, 2008). The research presented in this paper contributes to the body of research as it focused upon the experiences of 1:1 laptop programs in secondary education from a student perspective in their first post-secondary school year of education. It was envisaged that the data collected would be flavoured by the participants’ involvement in pre-service teacher education programs; perhaps influencing the reflective lens of each participant as they being to develop their professional identity. Research method The research undertaken was a qualitative approach within the parameters of a case study of first- CP:230 577 year Bachelor of Education students. The research questions were: 1. What was the nature of laptop use of students in 1:1 laptop program schools? 2. How has the 1:1 laptop program impacted students’ learning at University? 3. How has the 1:1 laptop program impacted students’ perceptions of teaching with ICTs? Participants First year Bachelor of Education students in a common first year unit (undertaken by primary, early childhood education, and secondary students) were invited to participate in the anonymous, online survey, and also a series of semi-structured focus group interviews. Due to the nature of the research, only students who had undergone a minimum of one year in a 1:1 laptop program in their secondary schooling were eligible to participate. The data set presented in this paper was sourced from two iterations of surveys and interviews: 2014 (N = 20) and 2015 (N = 10): 27 female students and three male students. Data collection methods Data were collected using two methods: an anonymous, online survey and semi-structured focus group interviews. The Qualtrics survey of 50 statements used two 5-point Likert scale arrangements from Very often to Never and Strongly disagree to Strongly agree to obtain students’ reflections, and five semi-structured focus group interviews that were 45 – 60 minutes in length; audio-recorded; transcribed verbatim; and cross-checked by the researchers. To ensure consistency, the same researcher conducted the interviews. The anonymous, online survey comprised both demographic and reflective components. The demographics targeted aged, gender, and number of years since leaving secondary schooling. The reflective components used the stem “At my secondary school, I used my laptop to …” and were aligned to four different categories: productivity activities, education-specific activities, communication activities, and creation activities. These categories and statements were similar to those used by Handal et al. (2013) in their study, and were adapted to reflect the capabilities of the Apple Mac environment within a context of a 1:1 laptop program. Data analyses Data analyses were conducted on the survey responses and interview transcripts. The survey responses were grouped in two positive responses (strongly agree and agree; very often and often), a mid-way response (sometimes or neutral), and two negative responses (strongly disagree and disagree; seldom and never). The data were analysed in single-fields and a selection of multiple-fields (cross-tabulations) in order to gain insights into trends and relationships. The interview recordings were transcribed by one researcher, and cross-checked by the other researcher. Both researchers coded the transcripts, and then collaborated to reach consensus on the final coding. The transcripts were then analysed independently, and further collaboration resulted in a consensus. Findings Productivity activities usage The productivity activities that participants rated as using most often were word processing and creation of presentations using PowerPoint: both scoring 90% “very often + often”. Interestingly 100% of the interviewees confided that although the laptop was “handy for typing up assignments” there was a prevailing practice of doubling up of work: “We would still handwrite our work and then transfer it onto the laptop in Word” (Participant 2d1). The use of PowerPoint presentations was predictable: “We did a lot of PowerPoints at school for assignments” (Participant 2d2). The two lowest rated productivity activities were draw diagrams (scoring 60% seldom or never: “Trying to draw diagrams on the laptop was too hard so you’d always need your file with paper.” Participant 2d2) and create desktop publications (scoring 80% seldom or never). Education-specific activities The highest rated activity in this category was gain information from websites scoring 90% very often or often. (Participant 2d2) Actually the only thing we really used it for in class was research on the Internet. (Participant 2a2) It was useful to be able to do searches on the Internet anytime. (Participant 2c1) We would use it a lot for research. For example in Art, we would start a new topic, CP:231 578 like the Renaissance, and the teacher would tell us to research it. Two other reasonably high-scoring activities were do my homework (80% very often or often) and investigate simulations, access videos & movies and complete assessment tasks (both at 60% very often or often). Communication activities The two highest-scoring activities for communication were access the school intranet (90% very often or often) and access emails (80% very often or often). Creation activities The three creation activities listed in the survey (Create videos/movies; create animations; create pod/vodcasts) scored highly negative: respectively, 53% seldom or never; 63% seldom or never; and 73% seldom or never. Participant reflections on the school 1:1 laptop program This group of statements used the stem “The 1:1 laptop program at my secondary school…” and Table 1 summarises the participants’ responses. Table 1: Summary of participant reflections on the personal impact of the 1:1 laptop program Statement Scoring advanced my productivity ICT skills 80% strongly agree or agree advanced my creativity ICT skills 70% strongly agree or agree made it easier for me to achieve to the best of my ability 60% strongly agree or agree assisted me to be organised 60% strongly agree or agree advanced my communication ICT skills 60% strongly agree or agree is something I would recommend to other schools 60% strongly agree or agree provided me with a platform to take responsibility for my learning & 50% strongly agree or agree supported my preferred learning style motivated me to engage with my classes 40% agree (no strongly agree) allowed me to choose the time and place for engagement with the 60% neutral curriculum Cross-tabulation: number of years attended a 1-1 laptop school + something I would recommend to other schools Interestingly the results of this cross-tabulation indicate that the longer the participants were involved in their school 1:1 laptop program, the less likely they would be to recommend the program to another school (0 – 1 year involvement scored 30% agreement; whilst 4+ - 5 years involvement scored 30% neutral or disagree. Discussion Three key issues were identified from the interview data: school policy versus teachers’ beliefs, student misuse of the technology, and teacher and student preference for pen-and-paper use. The following quotes provide an indication of why these were issues. 1. School policy versus teachers’ beliefs 2d2: [the teachers] would tell us we wouldn’t be using it in their lessons but we still have to bring them to class because they said it was school policy. 2d1: we would have the laptop open and pens and paper out. We would have the laptop at the top of the desk open because we wouldn’t be using it, but we had to have it open, so that would be the best place to get it out of the way so we could write. 2. Student misuse of the technology 2c2: There wasn’t any block on the Internet so we would be on Facebook and MySpace all the time. We didn’t do much work that year. We would take photos of teachers and edit these and show them CP:232 579 around. 2c1: You weren’t meant to be able to get onto Facebook and Gmail but we found a way around that. The teacher was up the front and we were watching a movie on the projector and kids would be gaming on their laptops instead of word processing notes. Someone would take turns in taking the notes and then send the others those notes. 2d1: it is also really easy to have your finger on the escape key and get out of something you shouldn’t be in when the teacher comes close … only some teachers walked around while we were working. 3. Teacher and student preference for pen and paper 2c1: In English the teacher made us hand write everything. Most of our work was still pen and paper, I preferred that. These conveyed sentiments from participants align with research reporting pre-service teachers’ 1:1 laptop integration within their practicums involves directing students to utilize them for note-taking, assessment writing, communication, and internet searching (Mouza & Karchmer-Klein, 2013). These findings begin to demonstrate the cyclical nature of past learning experiences impacting future teaching practices. The interviewees recommended four key actions for teachers to undertake to make 1:1 laptop programs more effective: (1) laptop use needs to be monitored well; (2) they need to integrate them properly into lessons; (3) they should not block so many websites (“if you are going to give kids the technology then don’t take it away because you don’t trust them”); and (4) it is important that teachers know how to use the technology. Finally, the interviewees reflected on how they would use the affordances of 1:1 laptop programs in their future teaching. These included allocating set times in the school day to use the laptops, using videos to demonstrate real life applications of concepts, prioritising handwriting over laptop use, not building the lesson around the laptop, infrequent use, and student-directed individual use. In addition, participants expressed their uncertainty on how to utilise the laptop within their teaching practices and very narrow views on how they would use it, for example Internet curation tools such as Scoop It and a discussion board. These reflections demonstrate the necessity of more explicit attention to integrating technology within student learning experiences (Hughes, 2013). Furthermore, the importance of pre-service teachers being given opportunities to analyse and critically reflect on integrating technology into students learning experiences in a meaningful way (Mouza & Karchmer-Klein, 2013) rather than negatively past schooling experiences impacting their teaching practice (Cullen & Greene, 2011). References Cullen, T. & Green, B. (2011). Pre-service teachers’ beliefs, attitudes, and motivation about technology integration. Journal of Educational Computing Research, 45(1), 29-47. Retrieved from http://jec.sagepub.com/content/45/1/29.refs?patientinform-links=yes&legid=spjec;45/1/29 Handal, B., Campbell, C., Cavanagh, M., Petocz, P., & Kelly, N. (2013). Technological pedagogical content knowledge of secondary mathematics teachers. Contemporary Issues in Technology and Teacher Education, 13(1), 22-40. Retrieved from www.citejournal.org/articles/v13i1mathematics1.pdf Hsu, T-S., Wu, H-K., & Hwang, F-K. (2007). Factors influencing junior high school teachers’ computer-based instructional practices regarding their instructional evolution stages. Educational Technology & Society, 10(94), 118-130. Retrieved from www.ifets.info/journals/10_4/12.pdf Hughes, J. (2013). Descriptive indicators of future teachers’ technology integration in the PK-12 classroom: Trends from a laptop-infused teacher education program. Journal of Educational Computing Research, 48(4), 491-516. Retrieved from http://jec.sagepub.com/content/48/4/491.refs Inan, F. A. & Lowther, D. L. (2010). Laptops in the K-12 classrooms: Exploring factors impacting instructional use. Computers & Education, 55, 937-944. Retrieved from http://sttechnology.pbworks.com/f/Inan_(2010)_Laptops%20in%20the%20K- 12%20classrooms.pdf Keengwe, J., Schnellert, G., & Mills, C. (2012). Laptop initiative: Impact on instructional technology integration and student learning. Education and Information Technologies, 17(2), 137-146. Retrieved CP:233 580 from http://www.researchgate.net/publication/227245942_Laptop_initiative_Impact_on_instruction al_technology_integration_and_student_learning Kopcha, T. J. (2012). Teachers’ perceptions of the barriers to technology integration and practices with technology under situated professional development. Computers & Education, 59, 1109-1121. Retrieved from http://marianrosenberg.wiki.westga.edu/file/view/KopchaTTeachersPerceptions.pdf Mouza, C. (2008). Learning with laptops: Implementation and outcomes in an urban, under-privileged school. Journal of Research on Technology in Education, 40(4), 447-472. Retrieved from http://files.eric.ed.gov/fulltext/EJ826086.pdf Mouza, C. & Karchmer-Klein, R. (2013). Promoting and assessing pre-service teachers’ technological, pedagogical content knowledge (TPACK) in the context of case development. Journal of Educational Computing Research, 48(2), 127-152. Retrieved from http://jec.sagepub.com/content/48/2/127.refs Mumtaz, S. (2013). Factors affecting teachers’ use of information and communication technology: A review of the literature. Journal of Information technology for Teacher Education, 9(3), 319-342, DOI: 10.1080/14759390000200096. Penuel, W. R. (2006). Implementation and effects of one-to-one computing initiatives: A research synthesis. Journal of Research on technology in Education, 38(3), 329-348. Retrieved from www.editlib.org/p/99387/ Rosen, Y. & Beck-Hill, D. (2012). Intertwining digital content and a one-to-one laptop environment in teaching and learning: Lessons from the Time to Know Program. Journal of Research on Technology in Education, 44(3), 225-241. Retrieved from www.editlib.org/p/54959/ Rudd, K., Smith, S., & Conroy, S. (2007). A Digital Revolution. Retrieved from http://www.pixel.com.au/documentation//products/netsupport/netsupport_school/labors_digital _education_revolution_campaign_launch.pdf. Sang, G., Valcke, M., Van Braak, J., & Tondeur, J. (2009). Factors support or prevent teachers form integrating ICT into classroom teaching: A Chinese perspective. In S.C. Kong et al. (Eds.), th Proceedings of the 17 International Conference on Computers in Education (pp. 808-815). Hong Kong: Asia-Pacific Society for Computers in Education. Authors (2015). One-one laptop programs: Transformation or stagnation? Issues in Educational Research, 25(2), 99-117. Retrieved from http://www.iier.org.au/iier25/XXX.html Weston, M. E. & Bain, A. (2010). The end of techno-critique: The naked truth about 1-1 laptop initiatives and educational change. The Journal of Technology, Learning, and Assessment, 9(6), 6- 25. Retrieved from http://www.jtla.org. Walker, R.M. & Blackley, S.E. (2015). Pre-service teachers’ reflections on their participation in 1:1 laptop programs. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:230-CP:234). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:234 581 Mind the Gap: Exploring knowledge decay in online sequential mathematics courses Brian Webby Diana Quinn Amie Albrecht University of South Australia University of South Australia University of South Australia Kevin White University of South Australia Open access, digitally-enabled learning can provide freedom and choice for new learners − not only in how and what they study, but when. With this freedom comes risk. One potential risk lies in the timing of enrolment in courses, particularly where fundamental knowledge is built across a year and where extended gaps between sequential courses might cause knowledge decay. Mathematics may be susceptible here. Our concerns were allayed; an examination of data suggested that new students preferentially minimise gaps and found no significant evidence for knowledge decay over periods of up to 12 months. Nevertheless, to support student learning in open online learning environments, it could be important to provide resources for student self-assessment of knowledge deficiencies, and the facility to refresh and regain understanding. Keywords: Online education, mathematics, knowledge decay, timing of courses Introduction More students are accessing online education, in part because of the flexibility that digitally-enabled courses allow (Mayadas, Bourne, & Bacsich, 2009). These students may be entering university studies without having had any academic experience, nor having met any academic benchmarks (Stone, 2012). Along with the ease of access to university study, there is risk; for example, students without study experience or adequate support may flounder and inefficiently use computers, materials and the online spaces available to them (Anderson, Lee, Simpson, & Stein, 2011; Marshall, 2014; Author 2 et al., 2012). Another naivety posing a potential risk for learners is the sequence and number of courses taken at any one time. Traditionally, students are strongly encouraged to take certain courses in succession, constructing a linear learning path through a degree program. Given the flexibility in offerings of online courses, students may take a more oscillatory learning path, perhaps moving between levels of study, returning to earlier levels to refresh knowledge, building a more self-organised learning model (George Siemens, pers. comm. 2015). The gap between sequential courses may be important. The Unified Learning Model, intended to reflect the principles of the mind’s neural plasticity, supposes two memory states, with practical repetition required to transfer knowledge from short-term (working) memory to the better-retained long-term memory state (Chiriacescu, Soh, & Shell, 2013). The transfer depends on: the degree of repetition (within a number of time steps) of an idea, the motivation and emotional state of the learner, and the connectedness of the idea to already known ideas. An exponential forgetting curve (Chiriacescu et al., 2013) models decay of knowledge as a function of time and sparsity of connection, meaning that as the elapsed time since learning an idea increases, and as the number of connections between associated ideas decreases, chunks of knowledge are lost. Knowledge decay has been studied extensively in high schools, where it is referred to as summer learning loss (Cooper, Valentine, Charlton, & Melson, 2003) and more recently in on-campus tertiary environments (Dills, Hernández-Julian, & Rotthoff, 2015). In this large, cross-discipline study, Dills and colleagues assessed knowledge decay between sequential courses, (e.g. Japanese 101 and Japanese 102) examining whether a 2 month or a 4 month gap between sequential courses had a detrimental impact on the final mark in the subsequent course. Overall, they found no evidence for knowledge decay with the longer gap, indeed they interpret their findings as evidence against the CP:235 582 concept of summer learning loss. The exception to these findings was in language courses where a statistically significant detrimental effect was found for the longer gap. Our study explores whether students exhibit wisdom in the design of their online study plans to minimise knowledge decay and support academic success. Any evidence would inform potential guidance to students and input to policy regarding structured enrolment in sequential online courses. Methodology Data were available for two foundation level mathematics courses. These are sequential courses at the same year level with one (Course 1) as a prerequisite of the second (Course 2). Since 2012, Course 1 has been offered online 11 times and Course 2, 10 times. Course topics are dissimilar – the first course in the sequence presents basic algebra and trigonometry, the second introductory calculus - but Course 2 relies on a familiarity with the mathematical language and methods developed in Course 1. We define the gap as the time in between the teaching periods of the sequential courses. The timing of offerings and duration of teaching produces gaps which are integer multiples of 3 months. Thus a student taking the follow-on course immediately after its prerequisite ends, will experience a gap of 0 months, a student following on one study period later experiences a gap of 3 months. Rather than have a negative gap value, we denote the gap when students take both courses simultaneously as concurrent (abbreviated as cc). Students who achieved a pass in the first course and had attempted the second (attempting at least one assessment) were identified. Students’ final marks for Course 2 were mapped against their study gaps, with box and whisker graphs used to display the distribution of the data (Spitzer, Wildenhain, Rappsilber, & Tyers, 2014). Age and final mark in Course 2 were plotted as a series of scattergrams to represent the gaps between Course 1 and Course 2. Results The observed enrolment pattern amongst the 305 students comprising our data set is shown in Table 1. Most students took Course 2 at its next available delivery, more than 90% did so within 6 months. Table 1: Proportion of students with observed study gap (cc represents concurrent enrolments) Gap (months) cc 0 3 6 9 12 Othe r Proportion 3.3 61.6 21.0 7.9 2.6 2.3 1.3 (%) The whiskers in the boxplots of Figure 1 represent the highest and lowest marks for Course 2 for a given gap, the dark horizontal line is the median and a rectangle shows where the central 50% of data lie. We have added a cross to show the mean value and the number of data points in each category is shown at the base of the boxplot. CP:236 583 Figure 1: Course 2 marks against gap between Course 1 and Course 2. Students selecting a gap of 0 months achieved the highest median mark. An independent-samples t- test was conducted to assess whether average marks obtained in Course 2 after a gap of 0 months or after a gap of 3 months are different. The two distributions are not statistically distinguishable, t(111)= -.25, p = .807. The median (and mean) mark in Course 2 appears to decline if the gap allowed is greater than 6 months. However a t-test to compare average mark obtained with a gap of 0 or 3 months, against marks obtained with a gap of 9 or more months, showed no significant difference in the distributions, t(16) = .28, p =.393. Our results suggest there is no discernible effect of knowledge decay for gaps of up to 12 months. Age and Gap In Figure 2, a series of graphs display student age on enrolment in Course 2 (horizontal axis) against final mark in Course 2 (vertical axis). The different graphs represent the gap (in units of 3 months) between starting Course 1 and Course 2. The apparent random scatter of dots indicates that students from any age group do not show a preference for gaps of a particular duration between Course 1 and Course 2, and that age does not seem to influence Course 2 mark. CP:237 584 Figure 2: Relationships between Course 2 final mark, age and gap between Course 1 and Course 2. Discussion Knowledge decay has been an argument used to promote a change in high school calendars from 9 to 12 months (Cooper et al., 2003). Studies at tertiary level show evidence of knowledge decay in a few disciplines (Deslauriers & Wieman, 2011; Dills et al., 2015). In this study we explore whether knowledge decay plays a role in the success of online mathematics students of various ages who are free to determine their own gap between successive courses. To focus on knowledge decay, rather than online learning capabilities, we limited our study to those students who had a successful online mathematics experience (defined as passing Course 1). We excluded from the data set a population of students who enrol and pay their fees but never log in to the course web site (here third parties often finance enrolment). Only those students who submitted an assessment were included in the analysis. In examining data from 3 years of deliveries of sequential foundational mathematics courses, we found no definite support for evidence of knowledge decay across gaps of up to 12 months. Many students selected minimal gaps in their pattern of enrolments; perhaps they do not require direct guidance on this issue. Online students exhibit considerable age diversity but this apparently does not affect course outcome or the gap selected by students. However many of the online students report as being new to university study. Perhaps a detectable knowledge decay experienced across a sizeable gap is confounded with the effect of new students becoming more effective in their learning. Variables known to impact on mathematical knowledge retention include learning approaches (De Smedt et al., 2010), teaching method (Deslauriers & Wieman, 2011), structure (Thiel, Peterman, & Brown, 2008) and emotion (Kim, Park, & Cozart, 2014). Digitally-enabled courses can support knowledge retention and regaining in ways that may not be easily available in traditional learning environments. For example, an extensive quiz can be used to identify deficiencies in students’ current knowledge and direct students to modules where they can re-learn and review specific concepts. In this way students’ brains may be able to quickly rebuild the neural connections despite the time between the original and new learning (Chiriacescu et al., 2013). We speculated that the older students in our cohort might possess some academic wisdom concerning the potential effect of interrupted practice of their mathematics knowledge, and would therefore preferentially select shorter gaps between sequential courses. Such a pattern was not demonstrated in our data but neither was there a discernable impact on study success. It may be that online learners, being largely self-directed learners, recognise the potential effect of a considerable CP:238 585 gap and make use of resources provided to refresh their knowledge before resuming study. While knowledge decay is not evident amongst these students, the university should continue to develop resources so that students can self-assess their incoming knowledge and be directed to materials specifically chosen to suit the level and topics relevant to courses they will study. Conclusion A university has obligations to provide students with accurate and complete advice in order to help them achieve success in their studies. However online students assembling programs of study in reference to their perceived needs, are able to make wise choices about the length of time to allow between sequential courses. Knowledge decay seems less of an issue in tertiary environments and for online students compared to high school environments. Available self-assessment tools and resources to promote recall and revision are possibly important components of open online learning environments for supporting students in overcoming any knowledge decay that may occur between sequential courses. References Anderson, B., Lee, S. W., Simpson, M. G., & Stein, S. J. (2011). Study orchestrations in distance learning: Identifying dissonance and its implications for distance educators. The International Review of Research in Open and Distributed Learning, 12(5), 1- 17. http://www.irrodl.org/index.php/irrodl/article/view/977/1886 Chiriacescu, V., Soh, L.-K., & Shell, D. F. (2013). Understanding human learning using a multi-agent simulation of the unified learning model. Paper presented at the Cognitive Informatics & Cognitive Computing (ICCI* CC), 2013 12th IEEE International Conference on. http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1081&context=computerscidiss Cooper, H., Valentine, J. C., Charlton, K., & Melson, A. (2003). The effects of modified school calendars on student achievement and on school and community attitudes. Review of Educational Research, 73(1), 1-52. http://sitemaker.umich.edu/carss_education/files/extended_day_cooper.pdf De Smedt, B., Ansari, D., Grabner, R. H., Hannula, M. M., Schneider, M., & Verschaffel, L. (2010). Cognitive neuroscience meets mathematics education. Educational Research Review, 5(1), 97- 105. https://www.uni-trier.de/fileadmin/fb1/prof/PSY/PAE/Team/Schneider/DeSmedtEtAl2011.pdf Deslauriers, L., & Wieman, C. (2011). Learning and retention of quantum concepts with different teaching methods. Physical Review Special Topics-Physics Education Research, 7(1), 010101. http://journals.aps.org/prstper/pdf/10.1103/PhysRevSTPER.7.010101 Dills, A. K., Hernández-Julian, R., & Rotthoff, K. W. (2015). Knowledge decay between semesters. http://pirate.shu.edu/~rotthoku/Liberty/Dills,%20Hernandez,%20Rotthoff_%20Knowled ge%20Decay%20Between%20Semesters.pdf Kim, C., Park, S. W., & Cozart, J. (2014). Affective and motivational factors of learning in online mathematics courses. British Journal of Educational Technology, 45(1), 171-185. Marshall, H. R. (2014). Digital Technology and Adult Online Learner Preparedness: Providing Appropriate Support for Developing Computer Comperacy. Walden University. Mayadas, A. F., Bourne, J., & Bacsich, P. (2009). Online education today. Science, 323(5910), 85- 89. http://www.sciencemag.org/content/323/5910/85.full Author 2 et al. (2012). Spitzer, M., Wildenhain, J., Rappsilber, J., & Tyers, M. (2014). BoxPlotR: a web tool for generation of box plots. Nature methods, 11(2), 121- 122. http://www.nature.com/nmeth/journal/v11/n2/pdf/nmeth.2811.pdf Stone, C. (2012). Engaging students across distance and place. Journal of the Australia and New Zealand Student Services Association, 39, 49-54. Thiel, T., Peterman, S., & Brown, M. (2008). Addressing the Crisis in College Mathematics: Designing Courses for Student Success. Change: The Magazine of Higher Learning, 40(4), 44- 49. http://www.changemag.org/Archives/Back%20Issues/July-August%202008/abstract-college- mathematics.html CP:239 586 Webby, B., Quinn, D., Albrecht, A., & White, K. (2015). Mind the Gap: Exploring knowledge decay in online sequential mathematics courses. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:235-CP:240). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:240 587 Clearing the Fog: A Learning Analytics Code of Practice Simon Welsh Stewart McKinney Adaptive Learning and Teaching Blackboard Analytics Services Charles Sturt University Charles Sturt University Learning Analytics is an area of practice that impacts the legal and ethical obligations of educational institutions. New legislative regimes, growing concern about online privacy, and the affordances of the data being collected mean Learning Analytics could represent a risk to universities to the same extent that it represents an opportunity. These risks augur the need for institutions to develop formal practice and/or policy frameworks around Learning Analytics to define supported practice, actively manage risks and begin to build trust and ethical practice through transparency. There is a danger for Australian universities that the development of such “checks and balances” are not keeping pace with the technological advancements in this field. This paper outlines how one university is seeking to provide a frame for lawful and ethical practice of Learning Analytics through a Code of Practice. Keywords: Learning Analytics; Ethics; Privacy; Learning Technology; Code of Practice; Higher Education The Need for a Learning Analytics Code of Practice It is four years since Long and Siemens published their now oft-cited paper Penetrating the Fog (2011), which provided a brief context and direction for the new field of Learning Analytics. Since then, as Learning Analytics has grown and evolved, so to have concerns around its potential impacts on the privacy and agency of University students and staff. Beattie, Woodley and Souter (2014), for example, examined a number of ethical issues around Learning Analytics and argued for a Charter of Learner Data Rights. Despite awareness of the ethical concerns, it appears that technological advancements and analytics capabilities within Australian universities are out-pacing the development of controls around what is collected, why and how such data is used. In early 2015, as an initial aspect in the development of a Code of Practice, Charles Sturt University (CSU) issued a “call out” to members of the Australasian Council on Open, Distance and e-Learning (ACODE) and the New South Wales Learning Analytics Working Group to share any a formal practice or policy framework for Learning Analytics they might have in place. No such documents were reported. A number of factors strongly suggest that Learning Analytics as a field of endeavor needs to be practiced within a defined framework of lawful and ethical practice: • the law - Learning Analytics embodies the collection, storage and use of personal information and, as such, is subject to relevant privacy laws (eg Privacy and Personal Information Protection Act 1998 (NSW)) and the Australian Privacy Principles. Our work also suggests that Learning Analytics activities undertaken by Universities (or other research bodies) are subject to the National Statement on Ethical Conduct in Human Research; • general societal concern around monitoring of online behavior and collection of personal information via tracking technologies; • relative immaturity of the discipline with institutions, practitioners and technology vendors still figuring out what works and finding the boundaries of “acceptable” practice; and • potential for inadvertent misuse and/or abuse. That is, Learning Analytics can offer new ways to discourage, disadvantage or even discriminate against students and staff. Importantly, a Code of Practice is not just a means for defining how institutions want to practice in order to maximise effectiveness or minimise risk; it is also an essential step in building trust between the institution and its students and staff through openness and transparency. A Code of Practice that guides the institution towards transparency and openness can serve to clear the fog around how students and staff are being monitored, why and how data is used, and start to dispel some of the fears about what lurks within. Such transparency in itself can also build a propensity for ethical practice as it provides a mechanism for staff and students to “watch the watcher”. CP:241 588 The University’s Approach to a Learning Analytics Code of Practice Practice and Policy The trouble with Codes of Practice is that they don’t necessarily have the weight of policy. The trouble with policies is that nobody reads them. Early in our work, it was recognised that CSU would need a readily accessible Code of Practice that worked from relevant legislation and University obligations, in order to provide staff and students with clear boundaries for the lawful and ethical use of Learning Analytics. This Code of Practice would be supported with a suite of professional learning resources and activities to further contextualise the Code around the use of specific learning technologies, Learning Analytics approaches and interventions. Importantly, the key elements of the Code would be reflected within a Policy Framework. A key goal for the Policy Framework was to mainstream Learning Analytics by embedding it into the everyday practice of the University. Therefore, it was decided not to develop a separate Learning Analytics Policy, but rather to renew existing policies to reflect the legal and ethical challenges around Learning Analytics and the tenets of the Code of Practice. Thus, the Policy Framework became a document defining how relevant existing policies would be changed to support the Code of Practice, legislation and other obligations. The existing policies to be renewed go beyond just technology- related policies to include policies around admissions, learning and teaching, intellectual property, staff codes of conduct and others. These key policies create obligations for University staff, students and systems, as well as for third-party learning technology parties. A central component of the Policy Framework is an Analytics Consent Statement, which explicitly addresses the key features and practices of our approach to Learning Analytics collection, storage and use in order to enable informed consent by staff and students. A Multi-disciplinary Approach The Code of Practice and Policy Framework –still in draft form at time of writing – were developed through a literature review and individual consultations with major stakeholders across CSU. This latter activity provided a mechanism for multi-disciplinary input to the Code, which is critical, given the scope of impacts of Learning Analytics practice across a range of professional discipline areas. To provide a coherent Code that is integrated with the broader operations and obligations of the University it was necessary to engage with areas of the institution representing those disciplines, including the CSU Privacy Officer, legal, information technology, Corporate Affairs, Academic Governance, research ethics committee, human resources, University records, Faculties and the Office for Students. A key outcomes of this consultation was not just the input and reshaping of the draft Code but the raising of awareness and understanding of a) privacy issues in relation to learning and teaching, and the use of learning technologies in particular, b) the extent to which the collection of personal information is possible within University systems and external learning technologies and c) who needs access to such personal information and for what purposes. Students were also consulted, via their representative bodies, as part of the Code development. The student response (like that of staff) was very positive: affirming the need for a Code, appreciating that the University was undertaking this work and strongly embracing the notions of the openness and transparency that permeate the draft Code. Principles and Commitments The Code of Practice was developed with both CSU staff and students as intended audiences. It is structured around three themes: i) Ethical Intent; ii) Student Success; and iii) Transparency and Informed Participation. Within each theme are the “Governing Principles”. These are more than mere ‘guiding’ principles. They are positioned as the core ethical and legal foundations of Learning Analytics at CSU with which all practices must be consistent. Alongside of the Governing Principles are a series of “Commitments”, which describe the University’s assurances – our promises – towards an ethical and open practice of Learning Analytics. Theme 1: Ethical Intent CSU acknowledges that Learning Analytics raises a number of ethical and legal issues (including privacy rights). However, given the University’s educational context, the benefits offered by Learning CP:242 589 Analytics for students and staff justify its practice in supporting learning and teaching insofar as those ethical and legal issues can be managed to respect all who are the subject of data collection. The body of literature makes frequent reference to how institutions need to have in place clear guidelines on ethical considerations surrounding such aspects as the rights and dignity of individuals, and openness about processes and practices (Pardo & Siemens, 2014; Siemens, 2013; Slade & Prinsloo, 2013). The literature is equally insistent on higher education institutions ensuring that their legal obligations are being met in relation to personal privacy, data collection and information protection (Kay, Korn & Oppenheim, 2012; Siemens, 2013). The Governing Principles and Commitments for the category of Ethical Intent are shown in Figure 1. Figure 1: Draft CSU Learning Analytics Code of Practice – Ethical Intent Principles and Commitments Theme 2: Student Success Principles 4 and 5 align with the CSU Learning Analytics Strategy (2013), whereby the analysis of learning and teaching related behaviours and data are argued to provide valuable insights into the student experience. Collected data is used for the purpose of better understanding and supporting student progress and retention, and promoting teaching excellence and scholarship. Students are engaged as active agents in the implementation of Learning Analytics, and placed at the centre of the learning experience by accommodating diverse individual characteristics in the learning process, by providing choice, and by allowing them to be active ‘managers’ of their own learning through the use of analytics. Elemental to gaining a better understanding of and supporting student progress and retention is the recognition and respect given to all students’ knowledge, experiences, strengths and needs (Boyle & Wallace, 2011). Of particular relevance, consonant with the University Strategy Objectives for improved educational outcomes and lives for Indigenous Australians, is ensuring learning data is used in ways that optimise all students’ engagement and advances successful learning outcomes according to their understandings and aspirations. Our Governing Principles and Commitments under Student Success can be seen in Figure 2. CP:243 590 Figure 2: Draft CSU Learning Analytics Code of Practice – Student Success Principles and Commitments Theme 3: Transparency and Informed Consent The final two principles show how the University will be clear and open in its purpose and scope for Learning Analytics, and maintain an established pathway for staff and students to understand their rights of access and privacy and regularly update their consent to data collection and storage. In order for the University to confirm Learning Analytics as a trusted activity within a community of practice for learning and teaching, then “its very policy of transparency” will inspire confidence in the institution’s efforts in Learning Analytics (Kruse & Pongsajapan, 2012). Forthrightness in processes and practices will ensure all staff and students have access to descriptions “of how Learning Analytics is carried out and […] informed of the type of information being collected, including how it is collected, stored and processed” (Creagh, 2014, p. 15). Our Governing Principles and Commitments under Transparency and Informed Consent can be seen in Figure 3. Figure 3: Draft CSU Learning Analytics Code of Practice – Transparency and Informed Consent Principles and Commitments CP:244 591 Conclusion The essential argument for a Learning Analytics Code of Practice is to recognise that the collection, retention and analysis of student and staff data from learning and teaching systems is an impingement on privacy. However, this impingement is justified to the extent that it is undertaken for an ethical purpose (e.g. to provide a meaningful benefit for those whose privacy it impinges upon) and conducted in accordance with clear, transparent and lawful governing principles and policies that define acceptable practice consistent with that purpose. Without the latter, any means could be argued to justify the ends. The authors do not suggest though that all institutions should adopt the draft Code outlined here. Rather, the argument is simply to have a Code. Indeed, there is great value in the development of a Learning Analytics Code of Practice through a broad consultative process across an institution as this a) raises awareness and understanding of the issues, b) identifies the opportunities for connections between existing policies and practices unique to each institution and c) enables the institution to contextualise the Code to their Learning Analytics strategy and stakeholder needs and expectations. References Beattie, S., Woodley, C., & Souter, K. (2014). Creepy Analytics and Learner Data Rights. In B.Hegarty, J. McDonald, & S.-K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educationaltechnology. Proceedings ascilite Dunedin 2014 (pp. 421-425). Boyle, A. & Wallace, R. (2011). Indigenous people and e-nabling technologies: An analysis of recent experiences in northern and central Australia. Kulumun Indigenous Online Journal, 1(1), pp. 1-14. Retrieved from https://novaojs.newcastle.edu.au/ojs/index.php/kulumun/article/view/53/39 Creagh, T. (2014). Analytics: Privacy and ethical issues related to data collection. Unpublished draft paper. Kay, D., Korn, N. & Oppenheim, C. (2012). Legal, risk and ethical aspects of analytics in Higher Education. JISC CETIS Analytics Series, 1(6). Retrieved from http://publications.cetis.org.uk/wp- content/uploads/2012/11/Legal-Risk-and-Ethical-Aspects-of-Analytics-in-Higher-Education-Vol1- No6.pdf Kruse, A. & Pongsajapan, R. (2012). Student-centered learning analytics. Retrieved from https://cndls.georgetown.edu/m/documents/thoughtpaper-krusepongsajapan.pdf Long, P. & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. Educause Review, September/October, 2011. Retrieved from https://net.educause.edu/ir/library/pdf/ERM1151.pdf Pardo, A. & Siemens, G. (2014). Ethical and privacy principles for Learning Analytics. British Educational Research Association, 45(3). doi: 10.1111/bjet.12152 Siemens, G. (2013).Learning Analytics: The emergence of a discipline. American Behavioral Scientist, 57(10). doi: 10.1177/0002764213498851 Slade, S. & Prinsloo, P. (2013). Learning Analytics: Ethical issues and dilemmas. American Behavioral Scientist, 57(10). doi: 10.1177/000276421347936 Welsh, S. & Mckinny, S. (2015). Clearing the Fog: A Learning Analytics Code of Practice. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:241-CP:245). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:245 592 Dreaming of Electric Sheep: CSU’s Vision for Analytics- Driven Adaptive Learning and Teaching Simon Welsh Philip Uys Adaptive Learning and Teaching Learning Technologies Services Charles Sturt University Charles Sturt University Current institutional approaches to Learning Analytics which focus on student risk and engagement are problematic in terms of their ability to support improved student learning and success outside of retention. Charles Sturt University’s (CSU’s) deductive work on defining its institutional model of Learning Analytics has led it to reconfigure its Learning Analytics activities into an Adaptive Learning and Teaching program. Adaptive Learning and Teaching is defined as any educational approach that utilises feedback or analytics on student learning to adapt content, teaching, systems and/or design to enhance learning effectiveness. A key feature of the CSU vision is to focus analytic processes on students’ representations of knowledge and integrate with the student “digital footprint” to provide real-time adaptation of online learning experiences and personalise online learning. Concurrently, CSU’s Adaptive Learning and Teaching Services team is working to build capability in using Learning Analytics to inform adaptation in learning and teaching practices. Keywords: Learning Analytics; Adaptive Learning; Deductive; Inductive; Analytics Strategy; Organisational Design; Student Success; Personalised Learning; Online Learning A Brief History of Learning Analytics at Charles Sturt University In 2013, Charles Sturt University (CSU) established a Learning Analytics Working Party (LAWP), a multidisciplinary body bringing together stakeholders from across faculties, technology, business intelligence, library, student support and administration. The second author of this paper is the founder and chair of LAWP. The LAWP then developed a Learning Analytics Strategy and CSU appointed a staff member, the first author of this paper, to drive the implementation of that strategy. An initial step in the implementation was to define how the institution wanted to apply Learning Analytics across CSU. A Model of Learning Analytics was developed by the LAWP that identifies and defines the elements required for the implementation of Learning Analytics at CSU and how those elements interact (see Figure 1). The Model moves deductively from the definition of what the institution is trying to do with Learning Analytics (enhance student success), through a theoretical understanding of the drivers of student success to how Learning Analytics is to be embedded in the organisation to drive adaptation among students, staff and systems, and how impacts will be evaluated. The Model can be thought of as a map of all the areas of complexity that need to be resolved. CP:246 593 Figure 1: CSU Institutional Model of Learning Analytics From Learning Analytics to Adaptive Learning and Teaching Current institutional approaches to Learning Analytics – as distinct from the work being done in Learning Analytics research or in innovative small-scale applications – are often focused on predicting student attrition risk and/or monitoring student engagement to inform interventions usually around enhancing retention. Purdue University’s Course Signals program is an exemplar of such approaches. Australian institutions are also active in this space whether that be through the development of institutional approaches to the use of analytics tools embedded in Learning Management Systems (LMS) (eg Retention Centre in Blackboard) or the development of dedicated engagement/retention predictive engines which encompass a broader range of analytics sources. The University of New England, University of South Australia, UTS, Griffith University and CSU are just some examples of institutions with the latter (Siemens, Dawson and Lynch, 2013; Let’s Talk Learning Analytics and Retention National Forum, 2015; Alexander, n.d.). Typically, these predictive engines and analytics tools use behavioural indicators (ie number of log-ins or clicks in an LMS) or learning outcomes (ie failed an assessment, failed to submit, GPA, etc) as their metrics. In a review of four American institutions’ engagement/retention analytics models, Sharkey (2014) reports that most predictive models around student engagement/risk tend to use the same sorts of variable (behaviours or learning outcomes) in the same kinds of ways. This paper argues that, as a direction for institutional Learning Analytics strategies, an over-emphasis on the kinds of engagement/retention approaches currently observed is limiting for a number of reasons: • Where’s the “learning” in Learning Analytics? Engagement is defined in behavioural terms (eg whether students accessed a site/resource), rather than in terms of actual learning quality. Lodge and Lewis (2012) discuss the issues associated with a behaviourally-focussed approach to the measurement of learning, concluding that: “strict behavioural data such as this lacks the power to contribute to the understanding of student learning in a complex social context such as higher education” (p.3). Such an approach places emphasis on the management of student behaviour, either micro (increasing clicks/activity) or macro (course completion), rather than enhancement of learning per se. This also begs the question of whether these approaches are truly Learning Analytics or more akin to Academic Analytics (see Ferguson, 2012, for a discussion of the distinction); • Institutions can develop an over-reliance on inductive analytics processes, where analytics are gathered and analysed for predictive associations without integration into a deductive model with a clear focus on student success. This may be viable if the goal is to predict distinct outcomes, like withdrawal from a course or program, but learning is a process and inductive approaches alone may fail to support the complexities of enhancing the quality of CP:247 594 student learning. Furthermore, many metrics which are readily available for inductive analysis are simplistic, not context specific and lead to “counting clicks” rather than monitoring the effectiveness of learning. As Lodge and Lewis (2012) comment, taking a constructivist approach to learning, “the emphasis here is on “how” [students interact with knowledge] and not “how much” as appears to be the nature of the data collected using LA” (p.3). For example, the number of forum posts or LMS log-ins by a student does not tell us about the quality of those posts/sessions, or their relevance to the learning design. Without a deductive model to drive the development of analytics capabilities there can be too much weight placed upon on such simplistic metrics and we end up focusing on what we’ve got rather than asking: how do we get what we need? As Gasevic, Dawson and Siemens (2015) state: “Learning analytics resources should be well aligned to established research on effective instructional practice. In so doing we can move from static prediction of a single academic outcome, to more sustainable and replicable insights into the learning process” (p.66); • Learning Analytics systems that provide students (via dashboards or notifications) with general or summative indicators of behavioural engagement have little utility in improving learning as they fail to provide the kind of specific instructive feedback to the student on where their learning is ineffective and how to improve it that Hattie and Timperley (2007) argue is critical. Rather, all they indicate to the student is a need for more activity (e.g. more forum posts or library searches) … precipitating “The Boxer Response” (named for the horse in George Orwell’s Animal Farm), where the student is asked to embrace the mantra “I will work harder” but with little guidance on what they need to work on or how. Feedback is needed at the point of learning and that feedback needs to be about the specific learning process/activity that is occurring (Hattie and Timperley, 2007). Furthermore, there is intractable complexity in attempting to predict the occurrence of quality learning at scale across students, across learning designs, across content and across disciplines. There is substantial variation in what quality learning looks like in different contexts and attempting to implement institutional-scale systems to address this may miss the point, even if it could be done. That is, any kind of system that provides students with a summative or lag indicator of the quality of their learning would still not meet the need for feedback at the point of learning about the specific learning process/activity that is occurring. The goal is to support adaptation during learning, not adaptation by re-learning; • some learning designs problematise meaningful analytic measurement (eg work placements); • there are serious ethical issues around a) the extent to which students would reasonably expect to be surveilled and b) the University’s obligation to act once it has information about students at risk; and • such Learning Analytics approaches are typically focused on “raising the floor” (supporting students at risk) and ignore opportunities to “raise the ceiling” (supporting high-achieving students to optimise their talents). To address the above, and informed by work on the Model of Learning Analytics, CSU moved from a Learning Analytics program to an Adaptive Learning and Teaching (ALT) program. Adaptive Learning and Teaching is defined as any educational approach that utilises feedback or analytics on student learning to adapt content, teaching, systems and/or design to enhance learning effectiveness. The focus is not only on monitoring and managing the student relationship, but on providing a “data engine” to enable adaptations across practice (by staff and students), systems (and the learning experiences they enable) and processes that support improved student learning. The ALT approach incorporates traditional feedback mechanisms (eg student evaluations) and other data and analytics sources. However, it employs a reconfigured view of Learning Analytics as a learning design challenge, in the first instance. That is, rather than Learning Analytics being a capability that is applied to an extant learning design, it is something that needs to be designed into the learning activity such that by engaging in the activity the student intrinsically generates analytics about the learning process that are meaningful for both themselves and the teacher. For this occur, there is a need to re-direct Learning Analytics such that the point of focus for analyses is not, primarily, the student digital footprint but the representations of knowledge created through the interaction with analytics-enabled learning activities. A key feature of the ALT approach is using learning technologies (designed to support specific CP:248 595 pedagogies) to create and capture representations (relevant to the pedagogy in question) of student thinking and knowledge, which can be coupled with other data and analyzed to provide insight on student learning. These insights are then used to enable adaptation at four levels: 1. Real-time adaptation of learning activities to personalise the student experience and promote deep learning; 2. Adaptation for students by supporting development of their meta-cognitive skills, learner dispositions and learning strategies; 3. Adaptation in teaching and learning design; and 4. Adaptation of learning technologies and systems. A Pathway to Personalised Online Learning For CSU, Adaptive Learning and Teaching is a pathway to delivering personalised online learning. The key to this is the real-time adaptation of online learning activities providing adaptation during learning that is responsive to: • The knowledge of the student – as represented via the learning technology; and • Their learning behaviours – as captured via a student’s “digital footprint”. Multi-dimensional analytics are critical to paint a holistic picture of student learning and CSU is currently working on integrating data sources in a way consistent with “Phase 3” of the Learning Analytics Sophistication Model proposed by Siemens, Dawson and Lynch (2013). The ability to couple knowledge representations and cross-systems data on learning behaviours will enable the personalisation of a) the pathway within a specific learning activity (as done in many existing adaptive learning tools) and b) the feedback/interventions provided – where feedback is provided based on what the particular student has/has not done in their broader learning context. For example, if there are key resources associated with a learning activity, have they been reviewed? Have “lead up” or pre-requisite activities been completed satisfactorily? Where there are gaps, the student can be directed to address these specifically and/or provided with any additional support resources that are embedded in the activity. By using students’ “digital footprints” to inform feedback/intervention the opportunity is created to also employ “big data style” recommender processes: students who also struggled with X, did Y and Z. A critical challenge is developing technologies that can “read” a wider variety of knowledge representations. Current adaptive learning tools rely heavily on multiple choice or open numerical responses (Education Growth Advisors, 2013), which work well for questions with clear “right or wrong” answers. We need to broaden this – for example, employing capabilities like natural language processing – and deal with the challenge of content specificity. To address the latter challenge, new ALT technologies would focus more on what’s happening in the learning process. That is, focus on the form of the knowledge representation more than its content. ALT technologies would look for patterns in the knowledge representation (and any changes therein) that suggest deep learning is occurring and feedback to the student would seek to promote deep learning. For example, in the analysis of free text, an ALT technology would look for evidence of deep learning in the patterns of language used – connection and critique of ideas, development of hypotheses, etc – and the feedback to the student guides deeper engagement (ie scaffolds deep learning). Importantly, ALT technologies should be viewed (and used) as a complement to the teacher, not a replacement. Such technologies would deal with basic pedagogies (eg practice-mastery paradigms) and/or construct learning experiences to guide students toward patterns of (deep) engagement with content (as defined by the form of the knowledge representation), but the quality of students’ ideas, analyses and conclusions remains the realm of the teacher. Indeed, the use of ALT technologies may create more space for teachers to focus on these higher-order dimensions with their students. Building Capacity Not Just Apps The CSU ALT program is not just about building “smart” learning technologies, a critical part is building capacity of staff and students in using Learning Analytics to inform practice and adaptation. The unit implementing the ALT program is Adaptive Learning and Teaching Services and was named to deliberately position it as a service provider to those using Learning Analytics at the university, primarily teaching staff and students. The objective is to avoid Learning Analytics being seen as CP:249 596 something that is done by the “data geeks”, to become something that is just part of everyday practice and experience. It is not about Adaptive Learning and Teaching Services doing Learning Analytics for the University, rather it’s about this unit mainstreaming Learning Analytics. The ALT program seeks to enhance organizational capability, whether that be through professional learning for academics or through developing innovative learning applications. Thus, the role of the Adaptive Learning and Teaching Services is to promote, enable and support the application of ALT approaches and technologies. The key functions of ALTS are shown in Figure 2. Figure 2: Key Functions of Adaptive Learning and Teaching Services at CSU We “background” Learning Analytics to talk about ALT because Learning Analytics informs adaptation in learning and teaching by people and systems, but it is not the outcome that drives the institution. Learning Analytics is a means to an end and that end is a rich and responsive student-centred learning experience that integrates analyses of student learning processes and learning behaviours to enhance student success. References Alexander, S. (n.d.). Student retention and learning analytics. In Slideshare. Retrieved from http://www.slideshare.net/ShereelAlex/olt-conference-learning-analytics?qid Education Growth Advisors. (2013). Learning to Adapt: Understanding the Adaptive Learning Supplier Landscape. Retrieved from http://tytonpartners.com/library/understanding-the-adaptive-learning- supplier-landscape/ Ferguson, R. (2012). Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5/6) pp. 304–317. Gasevic, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning Analytics are about learning. TechTrends, 59(1), 64-71. Retrieved from http://link.springer.com.ezproxy.csu.edu.au/article/10.1007/s11528-014-0822-x Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81–112. doi:10.3102/003465430298487 Let’s Talk Learning Analytics and Retention OLT Project National Forum – Case Studies. (2015). Retrieved from http://www.letstalklearninganalytics.edu.au/national-forum/ Lodge, J., & Lewis, M. (2012). Pigeon pecks and mouse clicks: Putting the learning back into learning analytics. Proc. ascilite Conference: Future Challenges/Sustainable Futures, (25 – 28 November, Wellington, NZ). Retrieved from: http://www.ascilite.org/conferences/Wellington12/2012/images/custom/lodge%2c_jason_- _pigeon_pecks.pdf Sharkey, M. (2014). Your Secret Sauce is not so Secret. In Blue Canary Blog. Retrieved from http://bluecanarydata.com/your-secret-sauce-is-not-so-secret/ Siemens, G., Dawson, S., & Lynch, G. (2013). Improving the Quality and Productivity of the Higher Education Sector – Policy and Strategy for Systems-Level Deployment of Learning Analytics. Society of Learning Analytics Research. Retrieved from: http://solaresearch.org/core/improving- the-quality-and-productivity-of-the-higher-education-sector/ CP:250 597 Welsh, S. & Uys, P. (2015). Dreaming of Electric Sheep: CSU’s Vision for Analytics-Driven Adaptive Learning and Teaching. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:246-CP:251). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:251 598 SkillBox: a pilot study Rachel Anne Whitsed Joanne Parker Charles Sturt University, Australia Charles Sturt University, Australia The aim of this project is to research, develop and evaluate a set of tools that can be used in tertiary subjects to formatively scaffold the skill base of students. The SkillBox instrument uses text, video and quizzes to deliver learning materials and formative assessment to students on a specific topic within a discipline area. A pilot project evaluated the use of a Matrix SkillBox in a Charles Sturt University (CSU) Distance Education (DE) subject and found its use appeared to increase knowledge and confidence in the topic areas covered. These findings will be further investigated in ongoing research involving larger numbers of students. Keywords: SkillBox; discipline-based skills, web-based learning tools Introduction Some university subjects require a certain level of skill in a discipline that may not feature strongly in the students’ academic background, such as mathematical or statistical skills in a subject that is not strictly a mathematics subject. An example is Geographic Information Systems (GIS). Some GIS subjects require certain mathematical skills, such as basic understanding of matrices. However, students commence these subjects with a wide range of pre-existing skill levels, ranging from very little to a high level of competency. Often a lot of time is devoted in these subjects to bring all students to the same skill level, rather than focussing on the more important application of these skills to GIS. This is a common issue in subjects requiring mathematics, statistics or other foundation knowledge (Galligan, 2013). The importance of teaching mathematics and other subjects in an online environment has been explored for at least the past 15 years (Allen et al. 1998; Porter 1997). The types of technology employed in these situations can be broadly described as web-based learning tools (WBLTs) (Kay 2011), which include online whiteboards, video podcasts and tablet technologies. The rationale given for researching and developing such tools includes improving student retention (Anderson and Jacoby 2013; Faridhan et al. 2013), improving subject readiness (Kay and Kletskin 2012) and maximising learning opportunities (Galligan et al. 2010). Researchers have found the use of these technologies to be overwhelmingly positive, particularly in a distance education environment. The use of online formative self-assessment quizzes has also been shown to improve student engagement, leading to increased subject pass rates (Nagel and van Eck 2012). In addition to using online quizzes, a key aspect of our approach is to curate existing material such as video tutorials alongside purpose-built material such as text-based explanations and online quizzes. Antonio et al. (2012) claim that digital curation can increase student motivation, engagement and learning outcomes. Based on our observations above and the research mentioned, we identified a need for a set of tools that can be accessed by students independently and as needed, in a non-threatening environment, to learn and scaffold the skills needed prior to or in conjunction with the application of skills in the subject. The need was also identified for the tool to require no active intervention from the subject coordinator and for it to be reusable in other subjects. As such, our pilot study aimed to build students’ confidence and skills in mathematical concepts required for postgraduate study of GIS through the integration of a Matrix Skillbox in SPA403 Algorithms in GIS and Modelling. Methods Our project used a mix of existing technology to develop a curated set of tools consisting of learning materials and formative assessment tools. As a pilot, the Matrix SkillBox was developed for use by students in the subject SPA403 Algorithms in GIS and Modelling in Session 1 2015 at CSU. The SkillBox is designed to be a self-paced optional module that students can dip in and out of at any time to learn and reinforce basic matrix concepts. For the Matrix SkillBox, the technology and tools used CP:252 599 included Khan Academy videos and quizzes (Khan Academy 2015), textbook explanations and exercises, and specially developed explanatory text with worked examples, and formative quizzes. Students enrolled in the subject were also invited to participate in a research project designed to evaluate its impact and effectiveness (Figure 1). Strategies for evaluating the initiative and measuring its impact included surveys, to measure shifts in attitude and confidence, and quizzes, to measure shifts in knowledge and competence. The surveys were based on research by Fogarty et al. (2001), who validated a questionnaire designed to measure general mathematics confidence, general confidence with using technology, and attitudes towards the use of technology for mathematics learning. Our first 11 survey questions are drawn from their statements on confidence when learning mathematics. Our remaining survey questions are on confidence with specific matrix operations (see Appendix). Students were asked to complete the survey at the commencement of the subject and again after working through the Matrix SkillBox. Responses were recorded on a 5-point Likert scale, plus a category of Don’t Know / Not Applicable. Figure 1. Sequencing of research project around SkillBox Students were also asked to complete a quiz (separate to the formative quizzes within the SkillBox) to summatively assess their knowledge of the topic area, both before and after engaging with the Matrix SkillBox. Six questions were related to matrix basics (dimensions of matrices and addition and multiplication) and four questions were related to determinants and inverses of 2x2 matrices. Answers in the quizzes were recorded as Incorrect, Don’t Know or Correct. The questions in both quizzes were of the same format and type, but with different numerical details, and were taken approximately 4 weeks apart by most students. Results The survey questions can be broadly divided into questions about positive attitude towards mathematics, questions about negative attitude towards mathematics, and questions on general understanding and confidence with matrices. Six students (33% response rate) participated in the pilot study. Changes in attitudes towards mathematics were minimal – positive attitudes increased slightly and negative attitudes decreased slightly. However understanding and confidence with matrices increased substantially (44% agree or strongly agree to 83% agree or strongly agree) (Figure 2). Figure 2. Survey responses pre- and post- intervention with the Matrix SkillBox (SA=strongly CP:253 600 agree, A=agree, N=neutral, D=disagree, SD=strongly disagree, DK/NA=don’t know/not applicable) With a sample size of six it is not possible to draw strong conclusions from these results. We did however test the shifts in responses for statistical significance using the Wilcoxon-Mann-Whitney test (Wilcox 2009). At a 95% confidence level ( ==6)0.05, n there was a statistically significant increase in confidence in calculating the determinant of a 2x2 matrix (Q18, p = 0.11, Z = -2.33) and calculating the inverse of a 2x2 matrix (Q19, p = 0.11, Z = -2.33). In other words confidence increased after intervention with the Matrix SkillBox, while attitudes towards mathematics only improved slightly. The quiz questions can be categorised as “Matrix Basics” and “Determinants and Inverses”. On matrix basics, the number of correct responses increased from 50% to 83%, and on determinants and inverses, correct responses increased from 21% to 100% (Figure 3), after intervention with the Matrix SkillBox. This shows that knowledge and competence also increased after intervention with the Matrix SkillBox. Figure 3. Quiz responses pre- and post- intervention with the Matrix SkillBox Self-reported time spent on the Matrix SkillBox ranged from less than 1 hour to around 20 hours (median 3 hours). When asked what they found most useful about the Matrix SkillBox, students mentioned the videos, quizzes, accessibility and repeatability. The only response to what they found least useful was a request for randomised questions – this suggestion has since been incorporated into the next version of the Matrix SkillBox. Suggestions for improvements included expanding the Matrix SkillBox to cover more concepts. Discussion and conclusions Previous research has identified the importance of using web-based learning tools for developing skills, improving student retention, improving subject readiness, maximising learning opportunities, increasing student engagement and improving learning outcomes. Our pilot study using the Matrix SkillBox suggests similar outcomes. With minimal time cost to students, and no active intervention from the subject coordinator once the Matrix SkillBox was in place, students increased their knowledge, competencies and confidence around matrices. Due to the sample size of the pilot study we have not yet been able to measure impact on student retention or learning outcomes. The concept of the Matrix SkillBox can easily be translated into other disciplines, with the structure of text, videos, quizzes and other resources remaining constant. In this way the look and feel of the tool is familiar to students who have used a previous SkillBox, and they will know what to expect. This familiarity should mean students will be more likely to engage, if they have found a SkillBox useful in the past. The replicability means that the burden on subject coordinators to implement a SkillBox in a subject is lessened, and should mean that uptake of SkillBoxes in subjects is preferred over each subject curating and developing their own resources for topics that are covered by an existing SkillBox. In addition, because the SkillBox is designed as a module that sits alongside a subject, rather than embedded in the subject curriculum, it can be implemented as a course-wide approach to address potential gaps in students’ skills and knowledge. By keeping the SkillBoxes accessible, self- paced, and taking less than 10 hours to complete, student equity is also improved. In the pilot study, the subject coordinator found that teaching time and resources could be devoted to more advanced topics, knowing that students had the resources in SkillBox to bring themselves up to CP:254 601 speed on matrices if necessary. As a result the subject was able to more adequately cover relevant subject material. In future research, the perceived impact on the subject itself will be elicited by surveying the subject coordinators involved. This research is ongoing, with the implementation of the Matrix SkillBox in further CSU subjects, and the development of SkillBoxes in other disciplines including statistics and programming. For each discipline, a new SkillBox will be developed, with each SkillBox following the same structure. Each completed SkillBox can then be embedded in multiple relevant subjects, with little ongoing investment from the Subject Coordinator. As the number of students participating in the research increases, we will be able to measure with more accuracy the impact on students of implementing a relevant SkillBox within their subject. Acknowledgements Ethics approval was granted by the CSU Human Research Ethics Committee with protocol number 2015/007. References Allen, G.D., Stecher, M. and Yasskin, P.B. (1998) The web-based mathematics course, a survey of the required features for an on-line math course, Syllabus Magazine Nov/Dec 1998 Anderson, C. and Jacoby, J. (2013) Using simple technologies to improve student engagement and success in an online applied-science course: a case study. In H. Carter, M. Gosper and J. Hedberg (Eds.) Electric Dreams. Proceedings ascilite Sydney 2013 (pp. 47- 51). http://www.ascilite.org.au/conferences/sydney13/program/papers/Anderson.pdf Antonio, A., Martin, N. and Stagg, A. (2012) Engaging higher education students via digital curation. In M. Brown, M. Hartnett and T. Stewart (Eds.) Future challenges, sustainable futures. Proceedings ascilite Wellington 2012 (pp. 55- 59) http://www.ascilite.org.au/conferences/wellington12/2012/images/custom/antonio%2c_amy_- _engaging.pdf Faridhan, Y.E., Loch, B. and Walker, L. (2013) Improving retention in first-year mathematics using learning analytics. In H. Carter, M. Gosper and J. Hedberg (Eds.) Electric Dreams. Proceedings ascilite Sydney 2013 (pp. 278- 282). http://www.ascilite.org.au/conferences/sydney13/program/papers/Faridhan.pdf Fogarty, G.J., Cretchley, P., Harman, C., Ellerton, N. and Konki, N. (2001) Validation of a questionnaire to measure mathematics confidence, computer confidence and attitudes towards the use of technology for learning mathematics. Mathematics Education Research Journal 13(2), 154- 160 Galligan, L. (2013) A systematic approach to embedding academic numeracy at university. Higher Education Research and Development 32(5), 734-737 Galligan, L., Loch, B., McDonald, C. and Taylor, J.A (2010) The use of tablet and related technologies in mathematics teaching. Australian Senior Mathematics Journal 24(1), 38-51 Kay, R.H. (2011) Evaluating learning, design, and engagement in web-based learning tools (WBLTs): the WBLT Evaluation Scale. Computers in Human Behaviour 27, 1849-1856 Kay, R.H. and Kletskin, I. (2012) Evaluating the use of problem-based video podcasts to teach mathematics in higher education. Computers & Education, 59, 619-627 Khan Academy (2015) Precalculus: matrices. https://www.khanacademy.org/math/precalculus/precalc-matrices Nagel, L. and van Eck, L. (2012) Sustainable learning through formative online assessment: using quizzes to maintain engagement. In M. Brown, M. Hartnett and T. Stewart (Eds.) Future challenges, sustainable futures. Proceedings ascilite Wellington 2012 (pp. 692- 699) http://www.ascilite.org.au/conferences/ wellington12/2012/images/custom/nagel%2c_lynette_-_sustainable.pdf Porter, L.R. (1997) Creating the virtual classroom: distance learning with the Internet. Wiley: New York. Wilcox, R.R. (2009) Basic statistics: understanding conventional methods and modern insights. Oxford; New York: Oxford University Press CP:255 602 Whitsed, R.A. & Parker, J. (2015). SkillBox: a pilot study. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:252-CP:256). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:256 603 Digital equity: A social justice issue for staff, not just students Julie Willems Office of the Vice Provost Learning and Teaching, Monash University It can be forgotten that it is not simply students who face the challenges of digital equity in higher education. Staff can also face digital challenges, and employment at an institution is not necessarily a safety net to protect staff from the digital divide. This paper attempts to give this voice to this issue. The digital equity challenges that they may face can range from internet accessibility, diversity in skills, or access to the required equipment and software, including necessary upgrades. This process is, however, is compounded when staff are geographically dispersed from the institution, disconnected by time, or where access to technology and Internet connectivity varies greatly between the institution’s sites. Much of these issues can be beyond the control and capacity of staff to alter. However, in terms of a staff-led approach to address such issues and empower others, a robust professional development program on digital technology is but one means to help stem the digital divide between staff ‘haves’ and ‘have nots’. Keywords: digital equity; digital divide; social justice; educational technology; higher education; professional development; educational equity Introduction Digital equity is a social justice issue which involves giving voice to the twin, and often intertwined, issues of underrepresentation and under-empowerment. Both themes are identifiable in the broad arena of educational equity. Underrepresentation relates to participation of particular sub-groups within a given population in light of their actual percentage in the overall population. For students in higher education, growing concern at an Australian governmental level drew our attention to these underrepresented in the mid 1960’s, later operationally defining these sub-groups in the mid 1990’s in the Martin Report (Martin, 1994). The focus on underrepresentation of staff in higher education has been a different matter. While it is slowly being addressed – such as Indigenous academics and women in management – there is still much to be done. However, this discussion is beyond the parameters of this paper. In the mid 1990’s, and around the same time as the operationalisation of the underrepresented student groups in higher education, the concept of digital equity crept into our academic consciousness, emerging in relation to the introduction of computers for teaching and learning. Digital equity relates to the second aspect of social justice – underempowerment – wherein individuals are unable to use technology in ways that would enable them participating fully and equitably in society (Gorski, 2009). The solution for underempowerment is “a multi-dimensional social process that helps people gain control over their own lives” (Page & Czuba, 1999, p. 1). Digital equity Digital equity is defined as “equal access and opportunity to digital tools, resources, and services to increase digital knowledge, awareness, and skills. [D]igital equity is more than a comparable delivery of goods and services, but fair distribution based on…needs (Davis et al., 2007, n.p.). This definition highlights that digital equity is not simply about access and distribution. It is also about knowledge skills and awareness so that the technology can be used to its full capacity. Initially the digital equity concerns for students in all education sectors was over the digital divide – the gap between the ‘haves’ and ‘have nots’ – and focussed on access to computer hardware and connectivity. By the end of the decade however, theorists in the fields of multicultural education, CP:257 604 critical theory and feminist studies (cf. Spender, 1995) were contributing to the expansion of the concepts of digital equity and the digital divide. Their scholarship, and those since, have helped move the notion of the digital divide beyond a simplistic view around hardware and connectivity (cf. Gorski, 2009), to refocus our understanding of it as a complex and multi-dimensional (cf. Willems, 2010) field of study. As such, digital equity goes far beyond access to hardware and connectivity. For example, when access to hardware and connectivity is resolved, questions still remain about how current is the hardware, can the necessary software or updates can be afforded, how swift is the Internet connection speed, or how is the technology is used? These are but some of the complex considerations. As Makinen (2006) has noted, bridging gaps in physical access to technologies is not sufficient a solution to the complex problem, especially “if we fail to address the gaps in opportunity actually to use the technologies in ways that empower people to participate more fully and equitably” (Gorski, 2009, p.352). The context of higher education in Australia is no exception. While this issue of under-empowerment has been investigated concerning students (cf. Willems, 2010), digital equity as a staff issue in higher education has received less attention in the literature (Willems, 2011). When discussed, it often relates to the digital divide in terms of a skills and knowledge comparison between students and staff, with digital immigrants attempting to colonise the world of digital natives (cf. Prensky, 2001a, 2001b). Yet this is but the tip of the iceberg. Digital equity and the empowerment of staff in higher education What are some of the digital equity issues for staff in higher education in terms of their disempowerment in the workplace? Some relate to personal factors such as knowledge and skills. Others relate to geographical location and/or isolation. And still others relate to technological access. Sometimes these various facets compound. When added to other staff equity factors, the experienced issues have the potential to compound for the individual staff members. Table 1 attempts to identify some of these aspects. Table 1. Facets of digital equity for staff in higher education Staff Social Facet Potential Impacts Potential Digital Equity Justice Aspect Impact(s) Underempowermen Geographical Lack of connectivity and Widening gap between t dispersion or equality in terms of technology technological advantage isolation access and updates and access Disability Accessible technology and Widening gap of digital content participation Technological Technological inequality in Widening gap of digital diversity terms of technology diversity participation of technology in the workplace Technological Lack of, or impaired, access to Widening gap of digital access equipment and software, access including necessary upgrades. Diversity in Gap in skills and knowledge Digital marginalization; knowledge base for staff Widening gap of digital and skills literacy The social justice goal of the digitally underempowered, or even disempowered, is to facilitate empowerment (Marullo & Edwards, 2000). Yet many of the digital equity issues can be beyond the control and capacity of staff to alter. This may at first appear deterministic, and that there may be little which can be done about these external forces upon the individual. Yet the work of Bandura (1989) suggests that through personal agency, solutions can be found to change what one can. As Bandura notes, personal agency: is achieved through reflective and regulative thought, the skills at one's command, and other tools of self-influence that affect choice and support selected courses of action. Self-generated influences operate deterministically on behavior the same way as external sources of influence do… It is because self-influence operates deterministically CP:258 605 on action that some measure of self-directedness and freedom is possible. (1989, p.1182) That is, through personal agency, there are things that staff can do to change their situation both reactively and proactively to empower themselves and others in spite of the factors external to one’s control or influence, such as institutional technology choices. One solution for a staff-led approach to address such issues and empower others is the creation of a robust professional development program on digital technology to help stem the digital divide between staff ‘haves’ and ‘have nots’, so that they can participate in the workplace fully and equitably. Driven by a needs and skills analysis to identify technological issues encountered by staff in their daily work functions and existing skills gaps, a locally generated but centrally supported professional development approach could be adopted. Facilitated where possible by staff champions to address issues identified in the needs analysis and showcasing, for example, technological work-arounds identified in the workplace, it represents a ‘bottom up’ approach for personal agency. Some issues to consider in the constructions of technology-related professional development programs is to examine the developed program through the lens of various underrepresented and underempowered sub-groups of the academic workforce to see where the program weaknesses are. Can all staff access physically and/or virtually the content? What mechanisms are built in to review the program from their perspective? Where skill development is the target, is the learning scaffolded appropriately? Do all staff have supported access to the programs? Is the content created in such a way that it can be accessed by those who it is intended to target? The list goes on. The development of the program need not be a costly exercise. It could involve gathering staff together to view, resources already created by others, or hearing the enthusiasm of others. Exemplar sessions could include the following ideas: • establishing a regular time and space for local professional development to take place as an investment for all stakeholders • viewing as a team a video such as a TEDx talk on a particular subject, followed by a staff discussion around some pre-determined reflective questions • organising a group participation in webinars run by professional agencies or organisations such as ASCILITE with a brief staff discussion to follow on how the technology and information might be useful locally • holding ‘speed dating’ sessions in which staff are divided into small groups and rotate around the room in their group, visiting pre-chosen colleagues showcasing a particular technology or app that they have found useful for teaching and learning and why • arranging a structured staff excursion to a local technology store to play with the devices on display The building of a robust technology-related professional development program to address these complex challenges by scaffolding knowledge and skills, and assisting access to and use of technology tools (ISTE, 2006). It also helps address one of the issues highlighted as part of empowering the disadvantaged and that is social connectivity, so that “the individual and community are fundamentally connected” (Page & Czuba, 1999, p. 1). The value of staff empowering staff through the development of their own digital technology professional development programs driven by localised needs analysis is that it meets some of the objectives in overcoming underempowerment by the reduction of staff isolation and potential alienation from others and the institution by helping them feel connected and part of the ‘whole’. As Marullo and Edwards (2000) sum up, “Such changes are to come about through altering institutional arrangements by redistributing resources and enhancing capacities of those with less, so that such institutional operations no longer maintain such inequities” ( p.898). Digital equity for staff in higher education is indeed is an area that requires ongoing research. Conclusion In summary, to bolster staff technological empowerment, it is beyond simply redistributing resources that digital inequity will be overcome. It is also through the enhancing staff capacity through CP:259 606 connectedness between individual and the broader community that is a crucial facet. In line with this, the paper argues that a concerted and targeted staff professional development is but one means to help stem the continuing digital divide. It is a means of overcoming helplessness and fostering personal agency. In terms of social justice issues, digital equity is an issue concerning not only students in higher education, but also staff. This paper has demonstrated that employment at an institution is not necessarily safe guard staff from the digital divide. Promoting awareness of this issue through publication and research is a way in which these challenges can begin to become visible and start being addressed. If institutions of higher education wish to be seen as actively striving to overcome aspects of digital inequities, then these inconsistencies need to be identified, articulated, faced, and actively pursued. It is an issue of academic integrity. References Bandura, A. (1989). Human agency is social cognitive theory. American Psychologist, 44(9), 1175- 1184. Gorski, P.C. (2009). Insisting on digital equity: Reframing the dominant discourse on multicultural education and technology. Urban Education, 44 (3), 348-364. Davis, T., Fuller, M., Jackson, S., Pittman, J., & Sweet, J. (2007). A National Consideration of Digital Equity. International Society for Technology in Education (ISTE). http://eric.ed.gov/?id=ED497214 Makinen, M. (2006). Digital empowerment as a process for enhancing citizens’ participation. E- learning, 3(3), 381-395. Martin, L.M. (1994). Equity and General Performance Indicators in Higher Education. Canberra: AGPS. Marullo, S., & Edwards, B. (2000). From charity to justice. American Behavioral Scientist, 43(5), 895- 912. Page, N., & Czuba, C.E. (1999). Empowerment: What it is? Journal of Extension, 37(5), 1-7. Presnky, M. (2001a). Digital Natives, Digital Immigrants. On the Horizon, 9(5), 1-6. Prensky, M. (2001b). Digital Natives, Digital Immigrants: Do they really think different? On the Horizon, 9(6), 1-6. Spender, D. (1995). Nattering on the Net: Women, power and cyberspace. Melbourne: Spinifex Press. Willems, J. (2010). The Equity Raw-Score Matrix – a multidimensional indicator of potential disadvantage in higher education. HERDSA Journal, 29(6), 603-621. Willems, J. (2011). Equity: A key benchmark for students and staff in an era of changing demands, changing directions In G. Williams, P. Statham, N. Brown & B. Cleland (Eds.), Changing Demands, Changing Directions. Proceedings ascilite Hobart 2011. (pp. 1305- 1311). http://www.ascilite.org.au/conferences/hobart11/procs/Willems-full.pdf Willems. J. (2015). Digital equity: A social justice issue for staff, not just students. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. CP:257-CP:260). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. CP:260 607 Metacognitive Development in Professional Educators: NZ teacher experiences using mobile technologies in a tertiary education environment Reem Abu Askar University of Auckland, NZ This research focuses on three areas: 1) The interaction between practising teachers’ metacognitive knowledge and regulation skills in relation to their classroom practices using mobile technologies; 2) perceived barriers and facilitators to the successful integration and use of mobile technology in the classroom; and 3) the impact of introducing a professional development programme (iPads Professional Development Programme) (iPDP) aimed at developing tertiary teachers’ metacognitive knowledge and regulation skills in order to improve their classroom practices. The main purpose of this study is to determine whether the development of teachers’ metacognitive knowledge and skills improves teachers’ pedagogical practices and integration of mobile technologies, such as iPads, and increases their proficiency using mobile devices for teaching and learning in tertiary blended classroom environments in New Zealand. This aligns with the “educational design research’s” (EDR) characteristics of offering practical solutions to real-world problems from the perspectives of both the participants and the researchers. Keywords: iPad use, Teachers’ metacognition, Educational design research, Professional development, Tertiary education. Research Background It is argued that mobile technologies have the potential to be powerful teaching and learning tools (Al- Zahrani & Laxman, 2014; Herrington, Ostashewski, Reid, & Flintoff, 2014) . They have the potential to offer teachers a flexible, relevant, personalised, metacognitive, and innovative way of teaching and supporting students in the 21st century (Ertmer, 1999; Ertmer & Ottenbreit-Leftwich, 2010). In New Zealand, a growing number of mobile devices such as iPads are being used across the spectrum, ranging from an early childhood education and care setting (Spencer, Coutts, Fagan, & King, 2013) to schools that have made a strong commitment to iPads (Henderson & Yeow, 2012; Tasman-Jones, 2012). However, a review of the literature reveals that although teachers are considered key to transforming teaching and learning (Gong & Wallace, 2012; Kinash, Brand, & Mathew, 2012), little research has examined teachers’ practices in relation to the opportunities technology provides, particularly in the tertiary education sector, where class sizes tend to be large, and the technological infrastructure is undergoing rapid change (King & Toland, 2014; Melhuish & Falloon, 2010; Ovens, Garbett, Heap, & Tolosa, 2013). In addition, examining the factors that influence technology integration indicates that teachers’ metacognition is one of, if not the most, influential factor for adopting new practices (Borg, 2006; Zohar & Barzilai, 2015). Moreover, as tertiary education moves toward mobile learning, there is clear evidence that introducing technology without supporting professional learning can undermine the best of intentions (Cavanaugh & Hargis, 2013; Schuck et al., 2013). Given the scarcity of research on teachers’ metacognition, and the lack of research analysing the relationship between developing teachers’ metacognitive knowledge and skills and teachers’ pedagogical practices and integration of mobile technologies, such as iPads, in tertiary classroom environments much more investigation is required. Research questions 1. What is the relationship between practising teachers’ metacognitive knowledge and skills and their classroom practices using mobile technologies in tertiary education? 2. What are the perceived barriers and facilitators to the successful implementation of mobile devices in the classroom? 3. To what extent will a supportive online professional development programme (iPDP) enhance teachers’ metacognition in order to develop their practices with mobile technologies? P:1 608 Description of proposed intervention The current study identifies the iPDP as the intervention to be developed collaboratively by the researcher and teacher participants. It aims to enhance teachers’ metacognitive knowledge and skills to improve their classroom practices. The intervention design will be guided by five major principles identified as a research hypothesis, focusing on the power of inner self “SELF”, outer self “PEER”, “COMMUNITY” and “CONTEXT”. It will also be directed by two major theoretical frameworks: “Mobile Professional Learning Community (MPLC)”(Cochrane & Antonczak, 2013) and “Metacognitive Technological Pedagogical Content Knowledge” (M-TPACK) (Wilson, Zygouris-Coe, & Cardullo, 2015), which are based on the concept of “Situated Cognition theory” (SC) (Brown, Collins, & Duguid, 1989). The iPDP intervention will include a short 5- to 10-minute video, posted weekly on the Blackboard online system, as a practical guide to using iPads in different subject areas. Regular posting and the brevity of the video may encourage teachers improve their practices by enhancing their knowledge and applying what they have learned. It will also save time compared with traditional lecturer PD sessions. The content of each video will focus on an area such as iPedagogy, different practices with iPads, and keeping up with upcoming innovations and inspiration for creative practices with iPads. Before and during the implementation of iPDP, strategies such as online surveys, goal setting activities, self-observation and self-reflection, peer observation, and peer feedback will be used. A MPLC will also be established to help teachers interact with each other, share their experiences and learning, and get support from their peers. Research design This study will employ “educational design research” (EDR) (McKenney & Reeves, 2014a) to make learning research more relevant to classroom practices. Wang and Hannafin,(2005) defined EDR as “a systematic but flexible methodology aimed to improve educational practices through iterative analysis, design, development, and implementation, based on collaboration among researchers and practitioners in real-world settings, and leading to contextually sensitive design principles and theories” (pp. 6–7). This study aims to provide an in-depth “picture” of teachers’ experiences using mobile technologies as it occurs using questionnaires, think aloud sessions, observations, interviews, and focus groups. Data from these methods will be analysed using content analysis and thematic analysis strategies to examine the interaction between practising teachers’ metacognitive knowledge and regulation skills in relation to their classroom practices using mobile technologies. A generic model for design research (GMDR) (McKenney & Reeves, 2014) will be used to provide an outline of the proposed study. The GMDR includes three central phases: analysis and exploration, design and construction, and evaluation and reflection, which lead to the two ultimate outputs of increased theoretical understanding and effective intervention maturation (see Figure 1). Figure 1 Generic model for conducting educational design research (GMDR) (McKenney & Reeves, 2014) It is expected that the design of the project – Educational Design Research (EDR, and the intervention – iPDP may have direct benefits to participants. It will provide teachers with an opportunity to develop their own professional development strategies that match their needs, and offer a useful methodological toolkit to better understand variables within naturalistic contexts. In addition, research results will be beneficial for educating professionals and policy makers and contribute to wider public understanding of educational policy and practice. P:2 609 References Al-Zahrani, H., & Laxman, K. (2014). Factors that Enhance or Hinder Acceptance and Use of Mobile Devices for Learning: A Meta-analysis of 60 Studies on Mobile Learning. Computer Communication & Collaboration, 2(4), 39-60. Borg, S. (2006). Teacher cognition and language education: Research and practice. London: Continuum. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42. Cavanaugh, C., & Hargis, J. (2013). iPads as cognitive toolboxes in higher education. In S. Dowling (Ed.), Redefining learning, book 2 in the HCT educational technology series. Abu Dhabi: HCT Press. Cochrane, T., & Antonczak, L. (2013). A Mobile Learning Community of Practice: Facilitating Conceptual Shifts in Pedagogy. In Hernandez-Leo, Ley, Klamma & Harrer (Ed.). Scaling up Learning for Sustained Impact (pp. 430–435). Berlin: Springer Ertmer, P. A. (1999). Addressing first-and second-order barriers to change: Strategies for technology integration. Educational Technology Research and Development, 47(4), 47–61. Ertmer, P. A., & Ottenbreit-Leftwich, A. T. (2010). Teacher technology change: How knowledge, confidence, beliefs, and culture intersect. Journal of Research on Technology in Education, 42(3), 255–284. Gong, Z., & Wallace, J. D. (2012). A comparative analysis of iPad and other M-learning technologies: Exploring students’ view of adoption, potentials, and challenges. Multiple Literacies in the Technical Editing Classroom: An Approach to Teaching, 13(2), 2- 29. Henderson, S., & Yeow, J. (2012). IPad in education: A case study of iPad adoption and use in a primary school. Proceedings of the 45th Annual Hawaii International Conference on System Sciences, January 4– 7, 2012, Computer Society Press, 2012 (78–87). Retrieved from http://www.computer.org/csdl/proceedings/hicss/2012/4525/00/4525a078.pdf Herrington, J., Ostashewski, N., Reid, D., & Flintoff, K. (2014). Mobile Technologies in Teacher Education. In Successful Teacher Education (pp. 137–151). DOI 10.1007/978-94-6209-677-6_9 Kinash, S., Brand, J., & Mathew, T. (2012). Challenging mobile learning discourse through research: Student perceptions of Blackboard Mobile Learn and iPads. Australasian Journal of Educational Technology, 28(4), 639–655. King, K., & Toland, J. (2014). iPads and the paperless office: The impact of tablet devices on paper consumption in higher education. Journal of Applied Computing and Information Technology, 18(1). McKenney, S., & Reeves, T. C. (2014). Educational design research. In Handbook of research on educational communications and technology (pp. 131–140). Springer. Melhuish, K. & Falloon, G. (2010). Looking to the future: M-learning with the iPad. Computers in New Zealand Schools: Learning, Leading, Technology, 22 (3). 1- 16. Ovens, A., Garbett, D., & Heap, R. (2013). Sustaining high quality pedagogy in the changing technological landscape. Computers in New Zealand Schools: Learning, Teaching, Technology, 25(1-3), 21– 37. Schuck, A. R., Vliegenthart, R., Boomgaarden, H. G., Elenbaas, M., Azrout, R., van Spanje, J., & De Vreese, C. H. (2013). Explaining campaign news coverage: How medium, time, and context explain variation in the media framing of the 2009 European parliamentary elections. Journal of Political Marketing, 12(1), 8–28. Spencer, K. M., Coutts, T., Fagan, T., & King, A. (2013). Connections, diversity, coherence: Three vignettes exploring learning with iPads in primary schools. Computers in New Zealand Schools: Learning, Teaching, Technology, 25(1–3), 38–55. Tasman-Jones, J. (2012, May 15). School iPad revolution may go nationwide. Stuff.co.nz. Retrieved from http://www.stuff.co.nz/auckland/local-news/6919789/School-iPad-revolution-may-go- nationwide Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53(4), 5–23. Wilson, N. S., Zygouris-Coe, V. I., & Cardullo, V. M. (2015). Teacher Development, Support, and Training with Mobile Technologies. In Professional Development and Workplace Learning: Concepts, Methodologies, Tools, and Applications (pp. 556-580). Hershey, PA: Business Science Reference. doi:10.4018/978-1-4666-8632-8.ch033 Zohar, A., & Barzilai, S. (2015). Metacognition and teaching higher order thinking (HOT) in science education: Students' thinking, teachers' knowledge and instructional practices. In R. Wegerif, J. P:3 610 Kaufman & L. Li (Eds.), Routledge international handbook of research on teaching thinking (pp. 229-242). Routledge: Oxon, UK. Abu Askar, R. (2015). Metacognitive Development in Professional Educators: NZ teacher experiences using mobile technologies in a tertiary education environment. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:1-P4). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:4 611 Digitise Your Dreams the Indigenous Way Aaron Matthews Rachna Aggarwal Siew Leng Lim Centre for Aboriginal Studies Centre for Aboriginal Studies Curtin Teaching and Learning Curtin University Curtin University Curtin University Dreamtime stories are the Indigenous way of understanding the world. These stories gave unity and purpose to Indigenous societies in the past and are important today in maintaining their identity and culture. They are seen to be the beginning of knowledge and thus make them good artefacts for capturing learning experiences. Research has shown that the sharing of stories from experience helps student see the purpose of learning hypothetical or conceptual content (Bittel & Bettoi, 2014). As such, the key to learning would lie with the choice and design of stories to make sure their connections with real world problems and prior knowledge are prominent. A digital story strategy captures the entire enquiry process by acting as the channel for self-expression in a digital era, including students’ information fluency towards constructing knowledge based on what they have observed and reflected on, to developing the ability to apply this new knowledge to a problem later (Kervin et. al., 2014). Riesland (2005) wrote that visual literacy education will empower the twenty-first century students with the skill to survive in a dynamic and fast revolving online world as they learn to decipher hypermedia information to develop critical thinking and analytical skills. Keywords: Enabling course, Indigenous, dreamtime, digital story, visual literacy, learning style, traditional storyline, technology Introduction Indigenous Tertiary Enabling Course is designed to offer Indigenous students an alternate entry path way into mainstream university degree programs. Although there is no strong evidence of a specific Indigenous learning style, they do have recurrent styles for preferring to learn by observation over verbal instruction and reflective learning (Hughes & More, 1997). Description This presentation will showcase how we integrate traditional storytelling technique with technology to develop engaging multimedia-rich digital stories for use to capture the attention of students with information (resources), and also to facilitate discussion and reflection (activities). We will share students’ feedback and address tutors’ concerns. Through this poster presentation session, we hope to offer some suggestions to tutors with the intention to implement this strategy in their classes and also gather opinions from those who have had experience with digital stories. References Bittel, N. & Bettoni, M. (2014). From Complex Maths to Simple Stories: a Knowledge Management Approach to Education. Switzerland: Swiss Distance University of Applied Sciences. Hughes, P. & More, A. J. (1997). Aboriginal Ways of Learning and Learning Styles. Paper presented on 4th December in Brisbane at the Annual Conference of the Australian Association for Research in Education. Kervin, L., O’Shea, S., Harwood, V. and McMahon, S. (2014). Digital storytelling: Capturing the stories of mentors in Australian Indigenous Mentoring Experience. Sage Research Methods Cases London, U.K.: Sage Publications, Ltd. Riesland. E. (2005). Visual Literacy and the Classroom. New Horizons for Learning. http://education.jhu.edu/PD/newhorizons/strategies/topics/literacy/articles/visual-literacy-and-the- classroom/ http://www.australia.gov.au/about-australia/australian-story/dreaming P:5 612 Matthews, A., Aggarwal, R., and Lim, S.L. (2015). Digitise Your Dreams the Indigenous Way. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:5-P:6). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:6 613 Introducing StatHand: A Mobile Application Supporting Students’ Statistical Decision Making Peter Allan Lynne Roberts Frank Baughman School of Psychology School of Psychology School of Psychology and Speech Pathology and Speech Pathology and Speech Pathology Curtin University Curtin University Curtin University Quantitative research methods are essential to the development of professional competence across a broad range of disciplines. They are also an area of weakness for many students. In particular, students are known to struggle with the skill of selecting quantitative analytical strategies appropriate for common research questions, hypotheses and data types, and this skill is not often practiced in class. Decision trees (or graphic organisers) are known to facilitate this decision making process, but extant trees have limitations. Furthermore, research indicates that students are more likely to access mobile-based material than content delivered via the web or face-to-face. It is within this context, and with funding from the Australian Government Office for Learning and Teaching, that we developed StatHand (see https://stathand.net), a cross-platform mobile application to designed to support students’ statistical decision making. In this poster, we will briefly articulate the rationale behind StatHand, highlight ongoing research into its efficacy and provide delegates with hands-on experience with the application. Keywords: Statistics; decision tree; graphic organizer; mobile application; iPad; iPhone; iOS. Background Decision trees (also commonly referred to as “graphic organisers”) to guide statistical decision-making have been used for at least half a century (e.g., Siegel, 1956) and are now commonly included in statistics textbooks (see, for e.g., Allen, Bennett, & Heritage, 2014). Their popularity is supported by both theoretical and empirical work. Theoretically, they rest on the idea that knowledge must be organised or structured to be accessible from long-term memory. Decision trees provide this structure by explicitly highlighting the interconnectedness (and differentiation) between important statistical concepts (Schau & Mattern, 1997). Empirically, the work by Carlson and colleagues (Carlson, Protsman, & Tomaka, 2005; Protsman & Carlson, 2008) has demonstrated that decision trees can facilitate significantly faster and more accurate (by a multiple of three) statistical decision-making, compared to more traditional methods of statistical test selection (e.g., by searching through a familiar textbook). StatHand Despite their popularity, traditional, paper-based statistical decision-trees also have limitations. Furthermore, research indicates that contemporary students are more likely to access mobile based material than content delivered via the web or face-to-face (Stowell, 2011). It is within this context that we have developed StatHand, a free cross-platform mobile application designed to support students’ statistical decision making (see https://stathand.net). This application, developed with the support of the Australian Government Office for Learning and Teaching, guides users through a series of simple, annotated questions to ultimately offer them the guidance necessary to conduct, interpret and report a statistical test suitable for their circumstances. References Allen, P., Bennett, K., & Heritage, B. (2014). SPSS Statistics: A practical guide (version 22). Melbourne, Australia: Cengage Learning. Carlson, M. Protsman, L. & Tomaka, J. (2005). Graphic organizers can facilitate selection of statistical tests. Part 1 - Analysis of group differences. Journal of Physical Therapy Education, 19, 57-66. Protsman, L., & Carlson, M. (2008). Graphic organizers can facilitate selection of statistical tests: Part 2 – Correlation and regression analysis. Journal of Physical Therapy Education, 22, 36-41. P:7 614 Schau, C., & Mattern, N. (1997). Use of map techniques in teaching applied statistics courses. The American Statistician, 51, 171-175. doi:10.1080/00031305.1997.10473955 Siegel, S. (1956). Nonparametric statistics for the behavioural sciences. New York, NY: McGraw-Hill. Stowell, J. R., (2011). Emerging technologies to improve teaching and learning in a digital world. In D. S. Dunn, J. H. Wilson, J. E. Freeman, & J. R. Stowell (Eds.), Best practices for technology- enhanced teaching & learning: Connecting to psychology and the social sciences (pp. 299-316). New York, NY: Oxford University Press. Allen, P., Roberts, L., & Baughman, F. (2015). Introducing StatHand: A Mobile Application Supporting Students’ Statistical Decision Making. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P7:-P:8). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:8 615 E-learning, resilience and change in higher education: A case study of a College of Business Kofi Ayebi-Arthur Niki Davis Una Cunningham University of Canterbury e-Learning University of Canterbury e-Learning University of Canterbury e-Learning Lab Lab Lab Christchurch, New Zealand Christchurch, New Zealand Christchurch, New Zealand What can e-learning offer in a crisis that closes the University campus? This paper presents the emerging findings in a case study of one College of Business impacted in 2011 by earthquakes in New Zealand. Analyses from interviews of nine staff and documents they recommended were used to describe processes of increasing resilience with e-learning over the worst seismic events. Increasing deployment of the University’s learning management system by staff and students plus audio recordings and video recordings of lectures enabled the College to continue its teaching. The Technology Acceptance Model (Davis, Bagozzi, & Warshaw, 1989) and the generic model of organisational resilience by Resilient Organisations (Resilient Organisations, 2012) will be used to evaluate the adoption and adaptation of e-learning when a crisis occurs. Keywords: E-learning, crisis, resilience, higher education, Technology Acceptance Model (TAM). Introduction There has been rapid evolution in the range of software since the beginning of the 21st century to support learning in universities. “Online education is established, growing, and here to stay” (Mayadas, Bourne, & Bacsich, 2009, p. 1). When disasters and crises, both man-made and natural, occur, resilient higher education institutions adapt in order to continue teaching and research. The University of Canterbury (UC) was affected by seismic events, which resulted in the closure of the University for two weeks at the start of the 2011 academic year (Agnew & Hickson, 2012; Dabner, 2012). Case study research aimed to provide a rich illustration of the ways in which e-learning assisted the University to keep open and improve learning and teaching as it recovered. The larger study of the University includes an embedded case study of how the UC College of Business and Law (CoBL) adapted with e-learning in the wake of the seismic activities. The emerging findings of the embedded case study are presented here. Methodology A qualitative intrinsic embedded/nested single case study design was chosen for the study (Gray, 2009). Sources of data included interviews and documents and the UC Progressive Restart website. Nonprobability purposive sampling was employed in the study to select the sample for the study (Cohen, Manion & Morrison, 2007). Seven academics who used e-learning were purposively selected and interviewed plus two members of the e-learning support staff (flexible learning advisors). Key informants for the University case identified the first key informant in the CoBL who then identified other academics in the College who used e-learning, both before and after the earthquake of 2011. These participants also identified relevant documents. The responses from the primary and secondary sources were coded and analysed using NVivo 10 qualitative analysis software (QSR International, 2012). The nine interviews were coded into three deductive categories: positive to e- learning, negative to e-learning, and mixed before further inductive analysis into themes. The themes were then also reviewed with the documentary sources of evidence and a timeline of the main seismic events and adaptations. Results This analysis has not yet been completed. Across all three categories a total of 18 themes were identified. The seven themes found in all three categories were: Perceived usefulness, Access to support, Organisation direction, Earthquake motivating factor, Attitude of students, Skills and Perceived ease of use. The most common theme was Perceived usefulness, which was mentioned by P:9 616 eight of the nine interviewees in a total of 62 units of meaning. For example: “…in the short term it [e- learning] was very useful when there was no physical campus” (CoB 6). This is particularly interesting because of the fit with the theoretical models that will be used in further analyses. Next steps The case study will be presented as an account of the adaptations made by the College from the first earthquake in 2010 until 2014, as perceived by the staff interviewed. It will also be interpreted using the Technology Acceptance Model (Davis et al., 1989). Finally, the Indicators of Resilience Model (Resilient Organisations, 2012) will be applied, if possible, to determine the value of e-learning as part of the resilience in the College of Business and Law in the aftermath of the 2010-2011 seismic events. The study aims to contribute to increased resilience for universities and might be used as a scenario by senior managers in their disaster planning exercises, which adds another challenge to the research analysis and reporting. References Agnew, S., & Hickson, S. (2012). Using online assessment to replace invigilated assessment in times of natural disaster: Are some online assessment conditions better than others? Journal of Open, Flexible and Distance Learning, 16(1), 1-13. http://journals.akoaotearoa.ac.nz/index.php/JOFDL/article/view/86 Cohen, L., Manion, L., & Morrison, K. (2007). Research methods in education. New York, NY: Routledge. Dabner, N. (2012). "Breaking Ground" in the use of social media: A case study of a university earthquake response to inform educational design with facebook. Internet and Higher Education, 15(1), 69-78. doi: 10.1016/j.iheduc.2011.06.001 Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: a comparison of two theoretical models. Management science, 35(8), 982-1003. Gray, D. E. (2009). Doing research in the real world (2 ed.). Thousand Oaks, CA: SAGE Publications Inc. Mayadas, A. F., Bourne, J., & Bacsich, P. (2009). Online education today. Journal of Asynchronous Learning Networks, 13(2), 49-56. http://www.sciencemag.org/content/323/5910/85 QSR International. (2012). NVivo 10 for Windows. http://www.qsrinternational.com [viewed 24 Sept 2015]. Resilient Organisations. (2012). About resilience. http://www.resorgs.org.nz/Content/what-is- organisational-resilience.html [viewed 24 Sept 2015]. Ayebi-Arthur, K., Davis, N., Cunningham, U. (2015). E-learning, resilience and change in higher education: A case study of a College of Business. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:9-P:10). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:10 617 Enhancing Student Learning Outcomes with Simulation- based Pedagogies Pierre Benckendorff Gui Lohmann Marlene Pratt The University of Queensland Griffith University Griffith University Paul Reynolds Paul Strickland Paul Whitelaw University of South Australia La Trobe University William Angliss Institute This poster reports on an Australian Government Office for Learning and Teaching (OLT) project to assist business educators to embed simulations into the curriculum. The purpose of this project was to gather and disseminate good practice in the design of pedagogy and assessment in simulation-based units in business. Data collection included interviews with educators and decision makers, student focus groups and surveys. The project included the development of an online toolkit consisting of case studies, a good practice guide and a simulation learning barometer. A ‘framework for simulation-based pedagogy’ is presented as a key outcome of the project. Keywords: business, simulation, pedagogy, assessment, learning outcomes. Introduction Enrolments in business fields such as management, marketing, accounting, finance, tourism and hospitality have expanded dramatically over the last decade. However, this popularity has resulted in large class sizes, which create challenges for developing graduate capabilities. It has been suggested that technology enhanced learning may overcome some of these challenges in business education (Karakaya, Ainscough, & Chopoorian, 2001). In particular, ‘gamification’ and the use of simulations have received attention in a number of fields. Online business simulations provide experiential learning environments that replicate workplace tasks or processes to allow students to apply knowledge and skills. Simulations are especially useful as a learning tool because they model aspects of reality in a safe environment, allowing learners to make errors that do not have real repercussions (Adobor & Daneshfar, 2006). Description The poster (see next page) reports on an Australian Government Office for Learning and Teaching (OLT) project to assist business educators to embed simulations into the curriculum. The purpose of this project was to gather and disseminate good practice in the design of pedagogy and assessment in simulation-based units in business. The project makes several key contributions regarding the learning outcomes, adoption, pedagogy, assessment and evaluation of online business simulations. These five areas form the basis for the ‘Framework for Simulation-based Pedagogy’ included on the poster: 1. Learning outcomes: The adoption of a simulation starts with a consideration of learning outcomes. Simulations are particularly effective in helping learners to integrate and apply business knowledge as well as providing opportunities to practise analysis, evaluation, creation and collaboration skills. 2. Simulation adoption: Simulation-based pedagogies require tactful management of the institutional constraints and challenges that have been identified. A champion is needed to promote and sustain the use of a simulation. Active engagement with supportive program directors, senior managers and decisions makers is a necessity. The background and needs of students should also be considered. 3. Pedagogy: Key suggestions for pedagogy include the use of non-traditional pedagogy that incorporates authentic learning tasks and activities, providing learners with opportunities to experience multiple perspectives, supporting collaboration, and coaching and scaffolding learning. 4. Assessment: The development of higher order graduate capabilities can be encouraged by designing authentic assessment tasks that require students to practice these capabilities. Common methods included assessing team interaction; using reports and presentations to P:11 618 communicate proposals, plans, company performance and competitor analyses; student reflections; and vivas. 5. Evaluation: The project has developed a Simulation Learning Barometer for benchmarking and evaluating student engagement, learning activities and assessment, team dynamics, learning outcomes, and satisfaction. References Adobor, H., & Daneshfar, A. (2006). Management simulations: determining their effectiveness. The Journal of Management Development, 25(2), 151-168. Karakaya, F., Ainscough, T. L., & Chopoorian, J. (2001). The effects of class size and learning style on student performance in a multimedia-based marketing course. Journal of Marketing Education, 23(2), 84-90. Benckendorff, P., Lohmann, G., Pratt, M., Reynolds, P., Strickland, P., & Whitelaw, P. (2015). Enhancing Student Learning Outcomes with Simulation-based Pedagogies. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:11-12). Note: All published papers are refereed, having undergone a double-blind peer-review process. P:12 619 The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:13 620 Creating concept vignettes as a module supplement for active and authentic learning Chandrima Chatterjee SUTD, Singapore Teaching Quantum Mechanics can be a daunting task for instructors. Typical classroom lectures may not be sufficient at times for proper understanding of the fundamental concepts. Hence there is a need to incorporate an effective scheme in the present teaching curriculum to further the learning experience of the students thereby enhancing their understanding of complex and abstract concepts. As such developing short educational and instructional videos known as Concept Vignettes on selected topics can help to supplement the existing lesson materials in quantum mechanics (Garik et al, 2005; Kohnle et al, 2010). Concept Vignette videos have been created on various topics previously by MIT’s Teaching and Learning Laboratory and are specially designed to enable students to learn a key concept in Science or Engineering (McKagan et al, 2008; Muller R et al, 2002) . My study will involve developing similar videos (in collaboration with MIT lecturers) with focus on the fundamentals of Quantum Mechanics. Keywords: Concept vignettes, curriculum, active and interactive learning Project objectives and deliverables The main aim for this research is to provide the students with an online platform to overcome their conceptual difficulties related to the lectures in Quantum Mechanics. My research will be focused on devising an effective way by means of which students will be able to revisit certain core concepts post regular lectures. This pedagogical strategy will also ensure that students are able to gauge their understanding of the lesson materials by answering some fundamental questions that will be integrated in this user-friendly online platform. Methodology My research will involve developing Concept Vignettes on topics mentioned earlier. Designing of Concept Vignettes comprises of the following steps: a) Creating short videos whereby difficult concepts will be revisited using commercially available software b) Embedding these videos in Microsoft PowerPoint to create an interactive platform c) Online quiz will be developed to test students’ understanding. After introducing a concept, a student needs to answer some questions related to the topic before moving on to a new one. d) An online student survey would be conducted in order to gauge how much it has benefitted them and should such strategies be adopted in the future. The survey would be mostly online supplemented by some student interviews. Significance and impact of study This is a pilot project that would help students enrolled in General Chemistry course. It is meant to address the key areas of student difficulty and is expected to remove some of their common misconceptions. The core and the fundamental concepts will be presented through a user-friendly video to reinforce the confidence of the students. Students are expected to watch the video prior to answering the embedded quiz. This pedagogical strategy will not only benefit the students, but will also facilitate the instructors to attain the pre-identified learning objectives of the module and therefore, enhance student learning outcomes. Such videos can be used by educators, students or anyone with an interest in Science. P:14 621 References Garik P et al. (2005). “Modernizing General Chemistry for the Year 2050: Why Are General Chemistry Instructors Hesitant to Teach Quantum Concepts?,” Presented at NARST, Dallas, TX. Kohnle A. et al. (2010). “Developing and evaluating animations for teaching quantum mechanics concepts,” Eur. J. Phys. 31, 1441. McKagan SB et al. (2008). “Developing and Researching PhET simulations for Teaching Quantum Mechanics,” Am J of Phys, 76, 408. Muller R & Wiesner H. (2002). , “Teaching quantum mechanics on an introductory level,” American Journal of Physics, 70, 200, (2002). Chatterjee, C. (2015). Creating concept vignettes as a module supplement for active and authentic learning. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:14-P:15). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:15 622 Preparing Students for Future Learning Jasmine Cheng Sally Payne Jennifer Banks UTS:Insearc, Australia UTS:Insearc, Australia UTS:Insearc, Australia Abstract UTS:Insearch is the premium pathway provider to the University of Technology Sydney (UTS). With education increasingly moving towards technology enhanced delivery, we identified the need to appraise our teaching approaches to better prepare students for future learning. This proposal represents the Blended Learning Framework adopted for the process of designing and implementing blended learning within the academic subjects. We initiated a suite of strategies with the intention to create classroom environment where learning occurs through seamless integration of technology enhanced strategies and face-to-face activities, characterised by the best features of interaction within a subject, that will promote academic enhancement and innovation in learning and teaching. The ‘hands on’ strategies allowed teaching staff to experience first-hand how students could be engaged with content through the meaningful use of technologies. This has led to 76% of our subjects either well progressed or fully compliant with a blended learning approach within a year. Keywords: technology enhanced, blended learning, engagement, innovation Introduction As part of the blended learning project, UTS:Insearch has defined blended learning as: “A learning environment where students learn through seamless integration of technology enhanced strategies and face-to-face activities, characterised by the best features of interaction within a subject”. In creating such learning environment, UTS:Insearch aims to seamlessly integrate a variety of technologies into the delivery of the curriculum, encouraging students to access learning opportunities and resources where the technologies are not segmented and structured or perceived as being ‘added on’ or an ‘extra’ workload. As far as possible the technologies used within this pedagogically planned framework should feel ‘invisible’ to students, and teachers, simply forming part of the way, along with face-to-face teaching, they interact with content and each other to provide a complementary learning experience that enhances and adds value to their studies and the students’ depth of understanding (Torrisi-Steele 2011). The Blended Learning Frameworks The technology enhanced strategies adopted by UTS:Insearch have been informed by Blended Learning Frameworks that provide overarching models that assist teachers to design their subjects, and support student-centred learning. These models are the 3E Framework and the Eight Phases of Blended Learning Framework (Figure 1). The Eight Phases of Blended Learning Framework has been adapted for the process of designing and implementing blended learning within subjects at UTS:Insearch. The different ‘phases’ of the framework emphasise the personal nature of each students’ journey through the learning process as this process becomes more student-centred (Woodall and Hovis 2010). The 3 E Framework The 3E Framework is based on an existing and tested Enhance-Extend-Empower continuum using technology for teaching and assessment, and supporting student learning. This framework considers how to include learning activities as a minimum (Enhance), through to further uses of technology that facilitate more student responsibility and control (Extend), and to reinforce more advanced, collaborative learning activities used in academic and professional environments (Empower). These three stages can be conceptualised in Table 1 below. By approaching the introduction of the concept in a gradual, supportive manner where we continuously stressed that pedagogy leads technology, we were able to encourage teaching staff to P:16 623 adopt a non-traditional approach to their practice and to begin building a dynamic and innovative teaching culture. This has led to 76% of our subjects either well progressed or fully compliant with a blended learning approach within a year. The Eight Phases of Blended Learning Framework can be characterised into the following stages: Preparation: Phase 1 Prepare Me: (Readiness Phase) Instruction: Phase 2 Tell Me: (Presentation Phase) Phase 3 Show Me: (Demonstration Phase) Figure 1: The Eight Phases of Blended Phase 4 LetFramework Learning Me: (Practice Phase) Table 1: 3E (Enhance-Extend-Empower) Framework Validation: ENHANCE EXTEND EMPOWER Phase 5 Check Me: (Assessment Phase) Phase 6 Support Me: (Assistance Phase) Phase 7 Coach Me: (Experience Phase) Phase 8 Connect Me: (Collaboration Adopting technology in simple Further use of technologyPhase) to Developed use of technology and effective ways to actively facilitate key aspects of that requires higher order support students and increase students’ individual and individual and collaborative that their activity and self- collaborative learning and reflects how learning is created responsibility. assessment by increasing their and used in academic and choice and control. professional contexts. An example of how this might be applied to assist students to engage with and better understand key concepts: ENHANCE EXTEND EMPOWER Encouraging early engagement in key concepts Students take turns in defining Students work in pairs to Students use online resources one or two key terms or create an online guide for a (collaborative spaces, links to concepts each week using a particular topic (for example, online readings, links to video class glossary or wiki. an online ‘scavenger hunt’ for clips etc.) that students can use fellow students to explore). in problem based learning tasks. P:17 624 References Association for the Advancement of Computing in Education (2007), Perspectives on blended learning in higher education. http://ablendedmaricopa.pbworks.com, accessed 30/06/2015. Partridge, H., Ponting, D., & McCay, M. (2011) Good Practice Report: Blended Learning. Australian Learning and Teaching Council Ltd, an initiative of the Australian Government Department of Education, Employment and Workplace Relations. Picciano, A.G. (2009). Blending with a Purpose: The Multimodal Model. Journal of Asynchronous Learning Networks, 13(1), pp. 7-18. Sagor, R. (2000). Guiding School Improvement with Action Research. Alexandria: Association for Supervision & Curriculum Development (ASCD). Takahashi, E., Austin, T. and Morimoto, Y. (2000). ‘Social Interaction and Language Development in a FLES Classroom’. Second and foreign language learning through classroom interaction. Hall, J. C. and Verplaetse, L. S. (eds), New Jersey: Lawrence Erlbaum Associates, pp.139- 157. Technology Enhanced Learning Quality Framework 2012-15, York St John University. https://www.yorksj.ac.uk/pdf/TEL%20Framework%20FINAL.pdf, accessed 30/06/2015. Torrisi-Steele, Geraldine (2011) This Thing Called Blended Learning — A Definition and Planning Approach in Research and Development in Higher Education Volume 34. Woodall, Dorman and Hovis, Sally (2010) “Eight Phases of Workplace Learning: A Framework for Designing Blended Programs”. http://www.skillsoft.com/infocenter/white_papers.asp, accessed 03/07/2015. Cheng, J., Payne, S., & Banks, J. (2015). Preparing Students for Future Learning. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:16-P:18). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:18 625 The use of rubrics for the assessment of digital products in language learning Neil Cowie Okayama University, Japan Many language teachers incorporate the use of digital technology into their classrooms in a various forms such as videos, blogs and slideshares. However, both teachers and students need a new level of awareness in assessing such web-authored products. A possible way for both teachers and students to learn to assess such digital products is for both parties to get involved in the process of assessment, specifically in rubric construction. This poster presentation will investigate the process in which English as a Foreign Language (EFL) teachers and students in a Japanese university collaboratively negotiate the process of rubric construction and the use of such an assessment tool throughout one academic semester. The collaborative process highlights two challenges that the teachers and students face: 1) how to assess the combination of language use and digital products; and, 2) how to empower teachers and students in the digital age. Keywords: language learning, digital products, assessment, rubrics, action research Introduction and description Rubrics are tools showing what criteria are expected at different levels of achievement along a continuum. There are a number of reasons why rubrics are useful for assessment: they give structure to assessment tasks, enable teachers to give clear feedback, and encourage consistency and fairness (Atkinson & Lim, 2013; Jeong, 2015); they can also be used to assess class participation and collaboration (University of New South Wales, 2015). For these reasons rubrics may be particularly appropriate for language learners carrying out project based work using Web 2.0 tools. This poster presentation will describe an action research project in a national university in western Japan. EFL teachers at the school developed a 16-week blended e-learning course during which the students created four digital presentations that included voice, photographs, video, and animation. Rubrics made collaboratively by the teachers and the students were used to guide this process and as a method of assessment. Data was collected through lesson observations, surveys, and interviews with the teachers and students. Key lessons concerning the creation and use of rubrics, issues of teacher role and identity, and future suggestions for further research and teacher development will be shown. References Atkinson, D., & Lim, S. (2013). Improving assessment processes in higher education: Student and teacher perceptions of the effectiveness of a rubric embedded in an LMS. Australian Journal of Educational Technology, 29 (5), 651-66. Jeong, H. (2015). Rubrics in the classroom: Do teachers really follow them? Language Testing in Asia, 5 (6). DOI 10.1186/s40468-015-0013-5. University of New South Wales. (2015) UNSW Assessment as Learning Toolkit. Retrieved from https://teaching.unsw.edu.au/assessment-toolkit P:19 626 The use of rubrics for the assessment of digital products in language learning Background: Approaches and features of language learning and digital technology APPROA Out-of-class Blended language Blended Web 2.0 projects Online CH practice FEATURE Need for an LMS Yes/No Yes/No Yes Vital Software tools Web 2.0 e-books (language Web 2.0 (language skills) Web 2.0 (collaborative tools) Virtual classrooms skills) Teaching Traditional Audiolingual Task-based Project-based Online approach In or out of class Out Both Both Out Challenges Choice of e-books and websites Choice of software tools Project design and tool Materials Development choices Problem: How to guide and assess Web 2.0 digital projects? Solution: Action research with rubrics Creating rubrics by Student Final version of Rubric formats (Partial) assessment presented by teacher and utilizing the rubrics surveys and rubrics completed teacher students interviews 1. Teachers survey 2. Students 3. Teachers and 4. Surveys and 5. Teachers reflect rubrics and ‘can-do’ understand the students conduct interviews about on 1 to 4. Create final statements in order purpose of rubrics ongoing assessment experience in versions of rubrics to show students and create their own using the rubrics creating rubrics and and clear guidelines rubric rationale, which define completed in Step 2. assessment process. for process of using concepts and basic learning goals. them. framework. Issues and Future Research Points  Principles of rubrics and ‘can do’  Assessment of technological products and/or statements language?  Principles of student involvement  Language teacher and/or technology teacher?  Principles of teacher involvement  Future online materials development roles P:20 627 Cowie, N. 2015). The use of rubrics for the assessment of digital products in language learning. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:19-P21). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:21 628 Developing an online challenge-based learning platform David Gibson Katy Scott Leah Irving Curtin University Curtin University Curtin University This poster provides an overview of the early development of a platform to facilitate online challenge-based learning that has potential for widespread global application. Challenge is a highly scalable platform that can personalise education for a massive global audience. Two challenges delivering learning activities and interactive content with gamified incentives to promote learner engagement have been developed and piloted. The primary concepts underpinning the student learning experience are individual and group-based problem solving, globally relevant challenges, personalisation and gamification of outcomes. Keywords: challenge-based learning, gamification, Snapshot of Challenge-based learning Challenge-based learning facilitates a multi-disciplinary approach to solving real world problems. The key focus of challenge-based learning is that rather than being content driven it takes an inquiry approach to identifying and analyzing a problem, finding and developing a solution and publishing the results (Johnson et al., 2009). Curtin University’s Challenge Platform Challenge is designed to support:  self-directed learning,  self-organizing international teams,  open-ended problem solving that requires complex thinking  cross-disciplinary engagement,  automated documentation and assessment of learning, social network validation processes,  peer evaluation and feedback,  expert judging and a variety of levels of recognition and awards. Challenge enables individuals to build up a private, safe and trusted longitudinal record of digital engagement and to make progress at the individual level while working alone or with others. Individuals can participate numerous times in any number of different challenges and can gain a collection of micro-credentials that stand as evidence of meeting university-level progress and achievement. The platform is designed for delivering learning activities and interactive content with gamified incentives that seek to promote learner engagement. A data collection and analysis capability is being developed to gather data about user actions, behaviors and achievements at a highly granular a (‘high resolution’) level to enable aggregations at higher levels based on domain models of the expertise being exhibited in the user’s actions, performances and products. The platform is not specifically a game, nor a game platform in and of itself, but it uses elements that are game-inspired. Current Challenges Two Challenges have been developed and piloted; Leadership Challenge and Careers Illuminate Challenge. Students participating in a challenge earn points towards badges, are able to level up to become a community mentor and expert, and completion contributes to the Curtin Extra Certificate. A Challenge under development has self-forming teams choosing a real world problem associated with one of the United Nations sustainable development goals, creating a solution and presenting the solution to a panel of experts. P:22 629 Pilot In the ten months since the Challenge platform was launched over four and a half thousand students have engaged with one of two Challenges. Over forty two thousand activities have been completed and over five hundred badges have been issued to students. The administration dashboard provides data at a granular level to identify issues with activities and student progress enabling staff respond swiftly in addressing instructional design problems and student engagement. References Johnson, Laurence F.; Smith, Rachel S.; Smythe, J. Troy; Varon, Rachel K. (2009). Challenge-Based Learning: An Approach for Our Time. Austin, Texas: The New Media Consortium. Gibson, D.C., Scott, K., & Irving, L. (2015). Developing an online challenge-based learning platform. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:22-P:23). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:23 630 Let’s Talk Learning Analytics and Student Retention David Heath Deborah West Henk Huijser Charles Darwin University Charles Darwin University Charles Darwin University This poster presents a summary of an Australian Government Office for Learning and Teaching strategic commissioned project titled Learning Analytics: Assisting Universities with Student Retention. The project was descriptive and exploratory, with data collection occurring between July, 2014 and March, 2015. A mixed method design was employed. The project occurred at a time when many institutions were actively exploring their options so a primary focus was on highlighting crucial issues in relation to learning analytics implementation. Following the data collection phase, a framework and accompanying set of discussion questions were developed to emphasise the importance of systematic discussion in making sense of and harnessing the opportunities afforded by learning analytics for student retention purposes. Keywords: Learning Analytics, Student Retention; Analytics Implementation; Introduction and Description The project focused on the following two research questions: 1. hat factors are relevant or need to be considered where the implementation of learning analytics for student retention purposes is concerned? 2. How do these factors impact on the implementation of learning analytics for student retention purposes? The research question were investigated via a mixed method project design, deployed in line with an ‘expansion’ purpose (Greene, Caracelli & Graham, 1989). Four data collection methods were employed:  An institution level survey (n = 24), focusing on sector readiness and decision making around the use of learning analytics for retention purposes;  An academic level survey (n = 353), focusing on teaching staff and other academic staff potentially involved with student retention. Questions focused on progress, aspirations and support needs;  A series of follow-up interviews (n = 23), with academic level survey participants designed to expand on the implications of different activities and experiences with learning analytics to date; and,  A suite of case studies (n = 5) developed by each of the research partner institutions detailing their experiences with learning analytics and demonstrating why elements in the framework are important. Results, Conclusion and Further Resources Following the data collection phase of the project a number of headline findings emerged relating to progress in the sector with regards to learning analytics and student retention. These were: 1. The sector in Australia is at an early stage of implementation and understanding around learning analytics; 2. Institutional context is critical and underpins the development, implementation and use of learning analytics; 3. Tensions exist around the extent to which learning analytics can drive actions and behaviours or take the functions of people; 4. Tensions exist between ‘business’ and ‘educational’ perspectives, aspirations and P:24 631 opportunities; 5. People across institutions have a key role to play in leveraging the opportunities of learning analytics which must take account of the relationships between strategy, planning, policy and action; and, 6. The sheer variety of data means establishing relevant business and educational questions is critical. The data was used to inform the development of the Let’s Talk Learning Analytics and Student Retention Framework, which provides a framework and accompanying set of discussion questions to facilitate systematic, institutional discussion around implementing learning analytics. More information can be found on the project website www.letstalklearninganalytics.edu.au References Green, J., Caracelli, V. & Graham, W. (1989). Toward a conceptual framework for mixed- method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255. http://epa.sagepub.com/content/11/3/255.abstract Heath, D., West, D., & Huijser, H. (2015). Let’s Talk Learning Analytics and Student Retention. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:24-P:25). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. \ P:25 632 Experiential Learning in Accounting: Engaging a diverse student cohort through the use of role-plays Rosemary Kerr Ross Taplin Alina Lee Curtin University Curtin University Curtin University Abhi Singh Curtin University Accounting is a client focused profession requiring interpersonal skills; however multiple offshore and onshore locations and large student numbers preclude all students experiencing work placements. This poster reports the outcomes of experiential learning activities, in the form of short role plays, designed to enhance accounting students’ communication skills, problem solving, ethical decision making and application of accounting knowledge. Online video, using YouTube, provided teacher training and student support in how to do role plays in tutorial classes. Online students were encouraged to participate through any electronic medium. Teachers and students from all locations reported the video was a vital resource for the class activity. Students and teachers enjoyed the role plays and perceived the activity was effective in building communication confidence. Online students did not engage with role plays and delivering role play activities to these cohorts presents challenges. Keywords: Experiential learning, role plays, online video, multi-location course delivery Practicing being an accountant through role plays Accounting is a client focused profession and requires high-level interpersonal skills as well as technical skills. Large cohorts of undergraduate accounting students make opportunities to practice being an accountant, such as work placements, difficult. Research in other disciplines reports that role plays enable students to practice skills in communication, collaboration, problem solving, ethical decision making and application of their knowledge (Taplin, 2007). Therefore, experiential learning (Kolb & Fry, 1975) through multiple small role plays to give students authentic learning experiences were trialed for both face-to face and online cohorts in large enrolment accounting units in an Australian university. A learning support video was developed and distributed to all locations using YouTube (Sherer & Shea, 2011). The video aimed to show teachers and students what the classroom activity looked like. A teacher guide document outlined the value of role plays to be communicated to students and gave instructions for facilitating the activities and the post- role play reflective class discussion. The role plays were designed to be a quick classroom activity, drawn from real life scenarios. Students performed the role plays in pairs, playing client and consultant, swapping roles to give everyone the opportunity to practice being an accountant. Tutors lead post- role play discussions to give feedback on the accounting issues and to get students to reflect on their role play experience. Students enrolled online were asked to perform the role plays with a partner through skype, email or telephone. Feedback to online students was given at the end of the week via Blackboard™. The student performance in the role plays was not assessed but the content covered was assessable. Feedback was gathered through email correspondence with teachers and focus groups and questionnaires provided student perceptions of the role plays. Tutors across all locations appreciated the learning activity and found the video and guide resource helped them understand how to facilitate role plays in the classroom. Students across all locations enjoyed the role plays and appreciated the safe environment to practice their communication and interpersonal skills. Role plays were novel in the accounting course and the video helped them understand what to do in the activity. Some students felt challenged by the ethical dilemmas and many international students experienced language challenges. Online students, typically already in the workplace, did not engage in the role play activities because they did not see the value. P:26 633 References Kolb, D.A., & Fry, R. (1975). Toward an applied theory of experiential learning. In C. Cooper (Ed.), Theories of Group Process. London: John Wiley. Sherer,P., & Shea, T. (2011). Using Online Video to Support Student Learning and Engagement. College Teaching, 59(2), 56-59.doi: http://dx.doi.org/10.1080/87567555.2010.511313 Taplin, R.H. (2007). Enhancing Statistical Education by Using Role-plays of Consultations, with discussion, Journal of the Royal Statistical Society, Series A, 170, 267-300. Kerr, R., Taplin, R., Lee, A., & Singh, A. (2015). Experiential Learning in Accounting: Engaging a diverse student cohort through the use of role-plays. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:26 -P:27). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:27 634 The CSU Online Learning model Tim Klapdor Charles Stuart University One of the key components of the CSU Distance Education Strategy is the articulation of an Online Learning and Teaching Model consisting of a set of elements which are known to result in increased student engagement. Increasing student engagement and connectedness is an important goal because of its link to measures of teaching quality, retention and overall satisfaction. This poster is a visual representation of those key elements and provides a unique way contextualizing learning design, activity and technology that results in increased student engagement. Keywords: online, engagement, online learning, elearning, e-learning, pedagogy, online pedagogies, practice, Description The concept of engaged learning that underpins this teaching-learning model builds on Moore’s (1989) highly cited model of Distance Education engagement, which incorporates learner-teacher, learner-learner and learner-content interaction. The CSU model adds learner-community engagement as a key element of professional courses as well as institutional engagement as a key additional element of the student’s overall connected experience. This then leads to five key interactive elements within this broad notion of student engagement: • learner-teacher engagement 
 • learner-learner engagement 
 • learner-content engagement 
 • learner-community-workplace engagement and 
 • learner-institution engagement. 
 The poster provides a visualization of the key elements of the model: • Small Group Support; • Personalised Support; • Teacher Presence; • Interaction Between Students; • Interaction With Workplaces; • Interactive Resources; and • E-Assessment. Each element is described and marked with a clearly identifiable icon. The poster will also provide links to access additional resources, supporting research and exemplars of the model. References Moore, M. G. (1989). Editorial: Three types of interaction. American Journal of Distance Education, 3(2), 1-7. doi: 10.1080/08923648909526659 P:28 635 Klapdor, T. (2015). The CSU Online Learning model. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:28-P:29). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:29 636 MOOCs as spaces to innovate Alison Lockley Charles Darwin University MOOCs have gained momentum in recent years and offer a new opportunity to interact with potential students to the university. While MOOCs have been seen as a disruptive force for higher education they have provided spaces to explore innovative approaches and emerging technology. The poster will showcase CDU’s process and experiences in this innovative space. Lockley, A. (2015). MOOCs as spaces to innovate. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:30-P:30). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:30 637 Mobile devices in an Interprofessional Community of Practice #NPF14LMD Mandia Mentis Wendy Holley-Boen Massey University Massey University The use of mobile devices shows promise in supporting practitioners to develop professional ePortfolios to document their ongoing learning and practice. This poster illustrates how practitioners within an interprofessional community of practice use mobile devices to develop professional identities. The affordances of mobile technology enable transformative ways of using multi-media in ePortfolios to showcase authentic practice and field-based learning in developing professional identities. The experiences of a practitioner focus group using mobile devices is analysed using a cultural historical activity theory (CHAT) framework to foreground changes in conceptions about Professional Learning and Identity Development (PLID). Keywords: mobile devices, mLearning, ePortfolios, Interprofessional practice, professional identities Introduction and context This study is part of a wider project #NPF14LMD (funded by Ako Aotearoa) looking at the uses of mobile devices by learners and professionals. The poster complements other panel and sharing practice sessions submitted for the conference (tagged with #NPF14LMD). The context for this project is an ongoing professional development course for practitioners within Inclusive Education. The practitioners included Resource Teachers from the seven specialist areas of Autism, Blind and Low Vision, Deaf and Hearing Impairment, Learning and Behaviour, Gifted Education, Complex Educational Needs and Early Intervention. The practitioners were invited to use mobile devices to document and narrate their professional learning and identity development in ePortfolios. The Questions A small focus group (10) of practitioners representative of the larger cohort were invited to consider whether using mobile devices changed the way they were able to document their ongoing professional learning and reflect on their professional practice. Practitioners were asked to identify which mobile devices they used, and for which practices. Participants were asked to comment on the affordances that mobile devices provided and how this influenced their reflection on practice. Description A Cultural Historical Activity Theory framework was used to analyse the activity of using mobile devices for professional identity development. This poster illustrates the dimensions of the CHAT analysis of using mobile devices in an Interprofessional community of practice, showing how the use of mobile devices contributed to changed conceptions of professional learning and identity development. These dimensions included:  Subjects: Resource Teachers from seven specialist areas within Inclusive Education  Tools: Mobile devices as outlined by the practitioner participants  Object: Examples of ePortfolios on professional learning, development and identity  Rules, Community, Division of Labour: Tensions and opportunities for learning & practice as identified by participants  OUTCOME: changed conceptions of professional learning and identity development PLID Mentis, M. & Holley-Boen, W. (2015). Mobile devices in an Interprofessional Community of Practice #NPF14LMD. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:31-P:32). Note: All published papers are refereed, having undergone a double-blind peer-review process. P:31 638 The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:32 639 Technology for Learning: How Do Medical Students Use Technology for Education? Michelle Moscova David Bruce Porter Kate Schreiber Educational Development Educational Technology Educational Developer University of Wollongong Graduate University of Wollongong Graduate University of Wollongong Graduate School of Medicine School of Medicine School of Medicine To assist in the design/selection and implementation of educational technologies in a regional medical program, first-year students were surveyed to determine the technologies used for academic purposes and their technology usage habits. The perceived usefulness and usability of technologies have been noted as important factors in technology adoption, as well as student engagement with technology. To address these conditions, the researchers surveyed students regarding the technologies they used for specific educational tasks. While still in our early stages of research, the results suggest that smartphones and tablets, while popular with students, still have not displaced laptops as the preferred devices for most tasks. Keyword: medical education, educational technology, adoption, usability, mobile devices, byod Background The Bachelor of Medicine Bachelor of Surgery (MBBS) program at the University of Wollongong (UOW) emphasizes rural and regional medicine and relies heavily on blended and online modes of delivery. Examining students’ understanding and use of technology is of inherent concern in program quality assurance, particularly in the design/selection and implementation of educational technologies. In a period of rapidly changing technological innovation, university programs must address the implementation of technologies in support of student learning. The importance of the perceived usefulness and ease of use of technologies underscore the major theories of technology adoption (Agarwal & Prasad, 1997; Davis, 1989; Straub, 2009). The failure of technology to meet the needs of users inhibits the adoption of these innovations. Furthermore, technical difficulties have been linked to lower test scores and higher attrition rates (Sitzmann, Ely, Bell, & Bauer, 2010). To address the technological needs of students, the Educational Technology team at UOW’s Graduate School of Medicine examined students’ use of technology as applied to their education. The annual study by the Educause Center for Applied Research recommends that institutions assess students’ technological literacy and conduct research to help students connect with technology in ways that enhance engagement and learning (Dahlstrom & Bischel, 2014). The Educational Technology team emphasises the user experience in the implementation of educational technologies. To target the medical program’s use of instructional technologies and to inform design decisions, the team engages in annual assessment of medical students’ use of technology for educational purposes. This study extends current research by understanding not only the devices students are using, but also the purposes for which students use the devices. Method First-year medical students were surveyed regarding the technology they used for specific educational tasks. During their first week of the program, the students indicated their responses via personal response devices (i.e., clickers) to survey items that asked their age, the devices they used for educational purposes, and the devices they used for specific educational tasks. Multiple response items were used to identify all devices students used for each task, therefore only frequencies and proportions of the cohort are reported. The study has been approved by UOW’s ethics committee. Key Findings  The majority (88.57%, N=70) of the students said they used multiple devices for educational purposes. P:33 640  In terms of educational uses, the largest proportion (87.14%, N=70) of students are still using laptops to access the learning management system, other online materials, email, and library resources.  While the majority of students used electronic means to access library resources, 25.71% (N=70) of students preferred to access these services in person.  Managing a calendar was the only educational task for which the largest proportion of students (37.68%, N=69) used smartphones.  Of students under 25 (57%, N=70), none of the students used tablets for educational purposes, compared with the 47.62% of mature-age students who used tablets (n=21). This may be due to the cost of these devices. Implications and Areas for Future Research While the current study reflects the technology preferences and usage behaviours of one cohort of medical students, the results begin to illustrate student technology practices. The survey data collected to date provide insights into the technologies students are using and suggest considerations for future program development. Over the long term, the second-, third-, and fourth-year cohorts will be surveyed to compare their educational technology preferences. Future studies might consider additional factors, including technical and pedagogical support in the use of technologies. Furthermore, studies conducted at other medical institutions will provide more generalizable observation and, henceforth, inform the integration of educational technologies across medical institutions. References Agarwal, R., & Prasad, J. (1997). The role of innovation characteristics and perceived voluntariness in the acceptance of information technologies. Decision sciences, 28(3), 557-582. Dahlstrom, E., & Bischel, J. (2014). ECAR study of undergraduate students and information technology. Louisville, CO: Educause Center for Applied Research. Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly, 319-340. Sitzmann, T., Ely, K., Bell, B. S., & Bauer, K. N. (2010). The effects of technical difficulties on learning and attrition during online training. Journal of Experimental Psychology: Applied, 16(3), 281. Straub, E. T. (2009). Understanding technology adoption: Theory and future directions for informal learning. Review of educational research, 79(2), 625-649. Moscova, M., Porter, D.B., & Schreiber, K.. Technology for Learning: How Do Medical Students Use Technology for Education? In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:33- P:34). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:34 641 The Flipped Teacher and the Flipped Learner Framework Jorge Luis Reyna Zeballos University of Technology Sydney We propose an 11 step framework to support educators and students to teach and learn with the Flipped Classroom (FC) model. Based on principles of blended and student- centred learning, organisational appearance, universal design and evaluation, the framework acts as a conduit between theory and good practice. Elements of the framework include: (1) planning stage, why and what to flip; (2) storyboard and lesson plan; (3) timing for activities; (4) online, (pre or post classroom) activities; (5) classroom work; (6) organisation of content; (7) visual design; (8) usability and accessibility; (9) building, testing and deployment; (10) communication of the benefits of the flipped model to students; and (11) evaluation and improvement. This paper will present the evidence behind each of these elements in a practical way to guide teachers and students through a flipped model of teaching and learning. Keywords: flipping the classroom, flipped learning, flipped classrooms, blended learning Introduction Blended learning is not a new concept (Bonk & Graham, 2012; Garrison & Vaughn, 2008). However it has gained prominence recently with the use of the term ‘Flipped Classroom’ (Bergmann & Sams, 2012). This approach replaces the traditional transmissive lecture for pre-class preparation, active in- class tasks and post-class work (Abeysekera & Dawson, 2015). Considered planning and implementation of flipped classroom (FC), can lead to increased teacher-student interaction and more effective learning (Moffett, 2015). It is crucial that the teacher is present when students attempt to analyse and apply new knowledge (Johnson, 2013). It has been postulated that FC can promote student’s self-direction and lead them to taking responsibility of their own learning (Bergmann & Sams, 2012). However, there are two major limitations associated with flipped classrooms. From the academic perspective, it is a time consuming exercise to set it up and it will require constant monitoring and improvement (Della Ratta, 2015; Shimamoto 2012; Snowden 2012; Wagner et al. 2013). From the student side, it will not work if they fail to engage with pre-class work (Kachka, 2012). The research gap The term ‘Flipped Classroom’ is gaining traction with academics but they are experiencing difficulties implementing effective learning designs (Chen et al., 2014). There is limited evidence-based research of the effectiveness of FC (Jensen et al., 2014). The approach is under-evaluated, under-theorised and under-researched in general (Abeysekera and Dawson, 2015). A recent search of the FC literature since 2012, returns publications mainly in the form of conference proceedings supplemented by a few journal papers. Most refer to case studies and none of them rely on particularly rigorous research designs. Examples can be found from many disciplines including education, sociology, languages, nutrition, chemistry, nursing, engineering and medical education. A planning template for FC design that considers before, during and after class activities and assessments was described by Gilboy et al., (2015). This template was based on Bloom's taxonomy but does not address the student’s experience. In contrast, Moffett (2015) described 12 tips for flipping the classroom but this was not comprehensive. At the time of writing, a holistic model to guide students and academics with flipped learning and teaching has not being described. As educators, we believe there are several variables or elements that could influence the success of a FC approach. This conceptual paper proposes an 11 step framework to support educators and students to teach and learn with the FC model. Based on principles of blended and student-centred learning, organisational appearance, universal design and evaluation, the framework acts as a conduit between theory and good practice. P:35 642 The framework At the University of Technology Sydney, we are promoting the FC as a way to further engage our students and foster independent learning skills. Additionally, we are promoting flexible and collaborative learning as we value the practice-oriented nature of our courses. We have developed new collaborative learning spaces and incorporated active learning theory in the design blended learning subjects. In this regard, we are currently refining the ‘Flipped Teacher and the Flipped Learner Framework’ (Figure 1), a tool to inform academics on how to FC. Additionally, it will inform our students the advantages of this educational strategy. Figure 1: The Flipped Teacher and the Flipped Learner Framework Our approach takes into consideration the planning on the FC, for example; why and what to flip, emphasise the use of storyboards (digital content) and lesson plans (face-to-face). Timing for activities and to ensure online and classroom activities are integrated seamless. Additionally, the framework embraces good visual design, usability and accessibility and building, testing and deployment of digital resources. The importance of communicating with the students the FC intervention is discussed. References Abeysekera, L. and P. Dawson (2015). "Motivation and cognitive load in the flipped classroom: definition, rationale and a call for research." Higher Education Research & Development 34(1): 1-14. Bergmann & Sams (2012). Flip Your Classroom: Reach Every Student in Every Class Every Day. International Society for Technology in Education. Bonk, Curtis J., and Charles R. Graham. The handbook of blended learning: Global perspectives, local designs. John Wiley & Sons, 2012. Chen, H. Y. L., et al. (2014). "Design and Evaluation of a Flipped Course Adopting the Holistic Flipped Classroom Approach." 2014 14th Ieee International Conference on Advanced Learning Technologies (Icalt): 627-631. Della Ratta, C. B. (2015). "Flipping the Classroom with Team-Based Learning in Undergraduate Nursing Education." Nurse Educator 40(2): 71-74. Garrison, D. R., & Vaughan, N. D. (2008). Blended learning in higher education: Framework, principles, and guidelines. John Wiley & Sons. Gilboy, M. B., et al. (2015). "Enhancing Student Engagement Using the Flipped Classroom." Journal of Nutrition Education and Behavior 47(1): 109-114. Jensen, J. L., et al. (2015). "Improvements from a Flipped Classroom May Simply Be the Fruits of Active Learning." Cbe-Life Sciences Education 14(1). Moffett, J. (2015). "Twelve tips for "flipping" the classroom." Medical Teacher 37(4): 331-336. P:36 643 Reyna Zeballos, J.L. (2015). The Flipped Teacher and the Flipped Learner Framework. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:35-P:37). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:37 644 Enhancing Workplace Learning through Mobile Technology: Designing the GPS for WPL Lina Markauskaite Peter Goodyear Franziska Trede Centre for Research on Computer- Centre for Research on Computer- The Education For Practice Institute Supported Learning and Cognition Supported Learning and Cognition Charles Sturt University The University of Sydney The University of Sydney Freny Tayebjee Celina McEwen Susie Macfarlane Careers Unit The Education For Practice Institute School of Exercise and Nutrition Sciences Faculty of Health, Deakin University of Western Sydney Charles Sturt University University Technology-mediated learning (TML) and workplace learning (WPL) are major priorities for universities. TML is core to the dynamic growth and modernization of university education, and WPL is an essential strategy used by universities to prepare students for future work. In Australia, both are rapidly changing practices, providing new possibilities and challenges. Though these two areas have largely remained separate in educational literature and practice, the integration of TML and WPL can provide important opportunities to bridge university and the workplace as well as build students’ digital capacities and online professional identities. This poster presents a mobile resource for students, named the “GPS for WPL”, aimed at helping students, academics and workplace educators to improve professional learning experiences by making better use of mobile technology. This resource was designed as part of a project funded by the Office for Learning and Teaching, entitled “Enhancing Workplace Learning through Mobile Technology”. Keywords: Mobile learning, mobile resource, workplace learning Project background “Enhancing Workplace Learning Through Mobile Technology” is a two-year research project that commenced in February 2015. It is a multi- site project led by CSU and conducted in collaboration with The University of Sydney, The University of Western Sydney and Deakin University. The aim is to develop and pilot a set of resources to help students, academics and workplace educators make better use of personal, mobile technologies to connect learning and work, and improve workplace practices. These resources will be integrated in what we are calling a mobile technology capacity- building framework for workplace learning – a set of materials and methods that can help all participants clarify their understandings of the main issues and opportunities, and improve their technology-mediated learning, practice and teaching skills. Conceptual framework This project draws on three sets of theoretical ideas: a) fostering the development of students’ agency (capacity to act) in WPL (Billett, 2011); b) translating research-based evidence into tools and resources that university teachers can use in course and curriculum design (Goodyear & Markauskaite, 2012); and, c) theorising the relations between technology, workplaces and work practices to sharpen conceptions of learning to participate in technology-mediated practices (Moen et al., 2012). The GPS for WPL is the first outcome of this project. It is primarily realizes the first two sets of ideas and aims to provide students with a resource that helps to enhance their agency to use mobile technologies skillfully and knowledgeably in workplace. This initial resource has been developed to support students’ WPL by focusing on how mobile technology can help with their learning on placement and prepare them for practice. The GPS is a resource that complements general P:38 645 preparation for WPL and could be used in various professional courses (nursing, education, etc.). Further, as digital technology is rapidly changing, this resource is neither focusing on providing a list of apps nor is it solely about enhancing digital literacies. Figure 1: GPS for WPL homepage The design and structure The development process of GPS for WPL was based on iterative 5 steps model: 1. Review of the literature on mobile technology and WPL 2. Consultation with local and international expert reference groups 3. Initial (Stage 1) design and development of the resource 4. Test by student users as well as academics, learning and workplace educators 5. Follow-up (Stage 2) refinement and development The GPS for WPL (https://gps4wpl.wordpress.com) has been created using Wordpress. It includes two main and four complementary entry points: The Landscape; Guiding questions; FAQ; Quiz; Sitemap; and Search, respectively. The landscape represents common purposes of mobile technology in WPL, such as staying connected, making informed decisions, and integrating theory and practice (Figure 1). The guiding questions of What, When, Where, How, Who, and Why to use technology provide tips, reflective questions, exercises and further links. The resource includes internal webpages and blog posts and links to external objects (e.g., webpages, videos, documents). Most content can be accessed through between 1 to 3 clicks (Figure 2). Further, students’ and educators’ use of the resource is integrated with possibilities to participate in a discussion forum. P:39 646 Figure 2: GPS for WPL structure Conclusion Though the GPS for WPL has been designed for Health and Education students to better and appropriately use mobile devices and apps on placement to enhance their learning to become a future professional, it is also hoped that, ultimately, the resource will help enhance the use of mobile technology for learning on placement for students of all disciplines. References Billett, S. (2011) Integrating experiences in workplace and university settings: A conceptual perspective. In S. Billett, S. & Henderson, A. (Eds.) Developing learning professionals. Dordrecht, Springer. Goodyear, P., Markauskaite, L (2012). Pedagogic designs, technology and practice-based education. In Higgs, J., Barnett, R., Billett, S., Hutchings, M. & Trede, F. (Eds.) Moen, A., Mørch, A. & Paavola, S. (Eds.) (2012). Collaborative knowledge creation: Practices, tools, concepts, Rotterdam, Sense. F. Trede, L. Markauskaite, P. Goodyear, S. Macfarlane, F. Tayebjee, C. McEwen. Enhancing Workplace Learning through Mobile Technology: Designing the GPS for WPL (2015). Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:38-P:40). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:40 647 Refocussing support on locally connected, digitally enabled communities of practice Susan Tull University of Canterbury Investigation of a new support model for professional development in the pedagogical use of technologies found that local communities of practice were preferred over a pan- university online community of practice. The support model was refocussed to digitally enable the development of locally connected communities of practice. This poster displays the two models, the research findings which supported their development, and recommendations for future developments. Keywords: professional development, communities of practice, technology support Introduction Introducing new technologies university-wide poses the problem of creating effective professional development opportunities which go “beyond the provision of technical information and training to encompass the development of a deeper understanding of the capability of learning technologies based on sound teaching and learning principles” (Gosper et al., 2011, p.92). A community of practice (Lave & Wenger, 1991) approach to professional development offers the opportunity for community members to learn from practitioners who have developed that deeper understanding and to share their practice with others who can, in turn, learn from them. Description Professional development in the pedagogically sound use of a video capture technology (Echo360) at the University of Canterbury was initially provided through a model of support centred on a digitally enabled online community of practice (CoP). An easily accessible online space, structured around common teaching and learning problems, was developed to encourage lecturers to connect with each other and to share their practice with colleagues from across the university. This top-down approach included shared examples of lecturers’ good practice solutions, with supporting explanations, alongside opportunities for engagement in forum discussions. Research was conducted on the first six months of this model’s implementation (Tull, 2014). Qualitative data collected from a purposive sample of nine lecturers from across most colleges of the university, and quantitative data collected from the online space, gave little indication that an online community of practice had developed. A strong preference for interacting face to face, rather than with an unseen body of peers, was expressed by lecturers. Non-judgmental personal support from known colleagues was highly valued, and was identified as a low risk way for lecturers to improve their practice in the use of this video capture technology. Lecturers who worked in teams or collaborated with colleagues in their use of Echo360 spoke highly of the benefits of doing so. Small local CoPs had developed where practitioners had become known to each other, and these groups had been able to provide both support and encouragement. The support model was refocused to take account of the research findings. New elements were incorporated alongside those of value from the initial model. Most significantly, rather than seeking to foster one pan-university online CoP, the new model supports practitioners in making connections within their local context. A restructured home page includes the addition of a ‘Colleague support list’ database, in which practitioners can volunteer their willingness to support others, and local practitioners offering support can be found. The database provides volunteers’ location information and preferred means of contact, as well as the areas of Echo360 use in which they are happy to provide collegial support. The areas are chosen from a prepopulated list, and those offering support can choose a primary support area and supplementary area(s). By enabling connections with ‘local’ users the site facilitates the development of local CoPs as a peer support network. P:41 648 The success of this new support model is contingent on faculty being made aware of it, as well as on the degree to which practitioners choose to become involved. Alerting all Echo360 users to the existence of the online space, its purpose and its facility to enable local peer support, is recommended. Future community development should be driven by the practitioners, enabling the bottom-up development of digitally enabled, locally connected communities of practice. References Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge, UK: Cambridge University Press. Gosper, Maree, McNeill, Margot, Woo, Karen, Phillips, Rob, Preston, Greg, & Green, David. (2011). Web-Based Lecture Technologies and Learning and Teaching: A Study of Change in Four Australian Universities. Journal of Asynchronous Learning Networks, 15(4), 84-95. Tull, S. (2014) Finding a Solution: Solution Focussed Professional Development for a Web Based Lecture Technology. In Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications, 2014 (pp. 1933-1938). Chesapeake, VA: AACE. Retrieved from http://www.editlib.org/p/147739/ Acknowledgements The author would like to thank Echo360 for the provision of their Active Learning Grants Programme, without which the initial research for this support model would not have been undertaken. Tull, S. (2015). Refocussing support on locally connected, digitally enabled communities of practice. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:41-P:42). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. P:42 649 Enhancing Queensland Pre-service Teachers’ Self-efficacy to Teach STEM By the Use of Remote Access Laboratories: A Mixed Methods Study Ting, Wu University of Southern Queensland Education for Science, Technology, Engineering, and Mathematics (STEM) is acknowledged as a priority around the world. However, many primary and secondary teachers are inadequately prepared for teaching STEM because of their limited exposure in their own schooling and teacher preparation. The Remote Access Laboratories for Fun, Innovation and Education (RALfie) project offer opportunities to provide a variety of STEM experiences available to students and teachers in schools, especially those in remote locations. They also have potential for influencing teachers’ self-efficacy to teach STEM by building up their capacities and capabilities to teach technologies. The mixed methods research is investigating how engagement with RALfie influence teachers’ self- efficacy for teaching STEM. Keywords: Self-efficacy, Remote Access Laboratories, STEM Overview The Remote Access Laboratories for Fun, Innovation and Education (RALfie) project aims to develop children’s Science, Technology, Engineering and Maths (STEM) concepts whilst fostering a positive attitude towards STEM learning. RALfie is creating a learning environment and the associated technical systems to offer low cost RAL, using tools such as the Lego Mindstorms EV3 Programmable Brick, and share them with other learners online. Others can use the RAL creating two types of participants: Makers and Users of RAL. This study focused on a trial of the system with Pre- Service Teachers (PSTs) who worked with hands-on and online experiments. In the Maker Event, PSTs used Lego to build an experiment and then connect it to the RALfie environment using the interface called a RALfie Box. They also connected IP cameras to the RALfie Box allowing remote viewing of the experiment in action. A web-based interface enabled remote control. PSTs were then able to view the experiment and control it remotely. For the User Event, the RALfie team designed four online experiments which enabled PSTs to access STEM experiments remotely. This research is about Queensland pre-service teachers’ self-efficacy to teach technology. Remote Access Laboratories (RAL) is being used as a vehicle to influence pre-service teachers’ self-efficacy to teach technology. The main research question is to investigate in what ways engagement with Remote Access Labs influences pre-service teachers’ self-efficacy to teach STEM content. Bandura’s self-efficacy theory (Bandura, 1977) is the conceptual framework in this research. Self-efficacy beliefs are derived from four principal sources of information, namely enactive mastery experience, vicarious experience, verbal persuasion, and physiological and emotional status. Based on Bandura’s theory, the Science Teaching Efficacy Belief Instrument (STEBI-B) was developed for pre-service teachers (Enochs & Riggs, 1990). This research modifies the STEBI-B for use as the Technology Teaching Efficacy Belief Instrument (T-TEBI). Mixed methods were used to collect data. Participants were pre-service STEM teachers at the University of Southern Queensland. They study a technology course in which RAL activities were used as part of the course. The pre-test and post-test of the T-TEBI surveys were analysed to trace changes in their self-efficacy. Interview was used to investigate in what ways engagement with RAL influences their self-efficacy. The outcome of this research is to investigate the effects on primary and secondary pre-service teachers’ self-efficacy to teach technology in schools using RALfie. Teachers’ attitudes and beliefs about their capability to teach STEM have a great impact on students’ attitudes and achievements in STEM learning. This research extends the application of self-efficacy in the RAL context. It expands the understanding of relationship between teachers’ self-efficacy and their capacity for teaching STEM. It will also inform the ways of impacting and influencing pre-service teachers’ self-efficacy using RAL as a vehicle. P:43 650 Acknowledgement Supported by the Digital Futures (CRN) Project funded through the Australian Government's Collaborative Research Networks Program. RALfie team members:Dr Alexander Kist, Prof Peter Albion, Dr Andrew Maxwell, Dr Lindy Orwin, Mr Ananda Maiti and Ms Wu Ting. QR code for RALfie’s CoLLAB website: RALfie YouTube videos: Contact us by email:

Visit the websites RALfie Research Project: research.ralfie.org and the RALfie’s CoLLAB: ralfie.org References Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2), 191-215. doi: 10.1037/0033-295X.84.2.191 Enochs, L. G., & Riggs, I. M. (1990). Further development of an elementary science teaching efficacy belief instrument: A preservice elementary scale. School Science and Mathematics, 90(8), 694- 706. doi: 10.1111/j.1949-8594.1990.tb12048.x Wu, T. (2015). Enhancing Queensland Pre-service Teachers’ Self-efficacy to Teach STEM By the Use of Remote Access Laboratories: A Mixed Methods Study. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. P:43-P:44). Note: All published papers are refereed, having undergone a double-blind peer-review process. P:44 651 Decisions and designs for building enterprise learning systems within an enabled learning paradigm: The case of third party technologies Garry Allan Zosh Pawlaczek RMIT University RMIT University Student learning data is now a currency of value for both our educational institutions and the increasing number of third party providers that complement and extend the university learning management system. Detailed awareness of the data management practices of these providers is of increasing relevance to the governance of enterprise learning system design, and in parallel educators need to be cognisant of the core data practices of third party technologies that they deploy within their teaching environments. Keywords: Third party technology, no-fee technology, enterprise learning systems, governance Introduction The enhancement of enterprise learning spaces requires a framework of evaluation for the purpose of establishing the pedagogical imperatives for the learning environment. The proliferation of enterprise level third party software development has opened up unique and diverse practices in learning and teaching (Hallam, 2012). Ideas of innovation have also been closely aligned with the use of technologies that have been socially marketed as both disruptive and a desirable personal commodity, and so educators and academic managers that identify themselves as innovative may make additional enterprise-level integration demands to support an innovation agenda as part of university practice. Enterprise learning environments, however, require careful management to ensure that both staff and students are able to carry out activities in a reliable, interconnected and supported space (Keppell et al, 2011). Risk management in an era of enhanced innovation, which often means applying new third party technology, carries with it some consideration that academic and technology managers ought to both address. The integrity of enterprise learning environments and the ability to provide a pedagogically cogent learning experience needs to be balanced against an agenda of strategically planned experimentation and then evaluation that advances learning and teaching practice. Who is designing the enterprise learning environment? Assertive technologists, practitioners and administrators have become expert at negotiating with relevant university domains to drive specific technical enhancements that are not necessarily focused on enterprise learning and teaching work-flow. The tensions start to emerge when requests are made for the enterprise-level inclusion of third party and licenced no-fee products, which can include services such as multiple and diverse publisher integrations with the LMS, that can take staff and students into what is in effect an alternate LMS, that transacts high stakes summative assessment independent of the university. The perception that licensed, no-fee services are ‘cost-neutral’ to a university pervades these requests and resonates as a sensible investment amongst decision- makers. In turn, IT departments, who are service-oriented and financially constrained, can approach these requests as potentially good solutions. Resisting these requests, as a process of evaluation, has become increasingly contentious because of the emerging availability of enterprise-level third party technologies, resulting in risk-management practices being interpreted negatively; particularly with products that are socially marketed, have significant social presence, and are new to enterprise level integration. Stringent risk management practices and innovation are not necessarily good companions (Keppell et al, 2011) but they are necessary to ensure a due-diligence, evaluation-based approach to designing enterprise digital learning spaces. The question then rests as to who is designing the learning environment? To answer this question satisfactorily it would be necessary to include many stakeholders. This in-itself reveals complexity that includes governance, user- expectation, pedagogy, vendors, risk, finance and a further question of who owns learning? DP:1 652 Discussion Who owns learning? Educators are eager to locate learning and teaching designs within a space that meets the demands of learning. Many contemporary collaboration tools also happen to be no-fee technologies, which present a win-win for cash-strapped universities. The often socially salient technologies are also easy to access with or without integration and so, creating curriculum experiences outside of a managed learning environment is entirely within the scope of individual teachers. The problem with this approach, however, is that accessibility, privacy and identity linked data footprints are being compromised. Students should own their learning profile, however with such models it is tacitly transitioning ownership via data transactions to third parties external to the institution. And if students expect us to care about them then perhaps the compromise ought to be a greater awareness of risk, and its management, associated with teaching using the external technologies. However, the appeal and benefits of third party, no-fee technologies is that it enables universities to provide the connectivity and agility that learners want and it posits itself as a solution to financial constraints. It also introduces the notion of seamless user experience by using technologies that already familiar to learners prior to them entering a particular institution. The university student and staff communities are demographically desirable targets for the provisioning of no-fee web services, where sustainable revenue is derived from aggregated services and an advertising income model. Under this model, over the last decade, services offered have advanced from basic email to web conferencing, social media and collaborative office personal productivity services (Vaidhyanathan, 2009). This strategy of “migratable” services, where a large user base will move from service to service within an ecosystem without giving deep consideration to the consequences of moving between theses services, nor the terms of service applicable to the usage context, is now being exploited in Universities, and is likely to further drive ecosystem lock-in. Acceptance of migratability within an ecosystem is such that most users would not consider it unusual that a search engine company is now in the web TV business, and a key provider of collaborative learning technologies for universities. The disparate products and technologies that are able to be linked within an overarching migratable ecosystem, continues to broaden and strengthen the contextual richness of the user experience. The advent of ‘app store’ extensions to these services has multiplied the value of the technology suite available to the university community. Part of the reason for this has been because of the need for learning engagement. The historical development of LMSs has seen few differences between providers with regards to tools for designing learning collaboration, with a common suite of tools: forum, chat, document sharing, quiz, assessment submission, and a focus on content management (Dalziel, 2013); these constitute a critical part of learning design, but not necessarily learning engagement. Third party services are typically complex and described in terms of service documentation that can be rapidly changing and not fully understood by the staff who are using the technology and responsible for the student learning experience. Informed vetting of the associated risks of data handling, privacy and security practices of ‘app store’ and other extended third party systems is beyond what can be reasonably expected of teaching practitioners, and “the process of education” (Dalziel, 2013) however, the affordance of these technologies could not be reasonably matched by fully university funded and managed technologies that more completely uphold the teacher and institutional oversight that has previously characterised the academic learning exchange. Additionally, for teaching staff, the affordance of the third party technologies, expressed in the portability of information, flexibility of collaboration, and fluency of communication, has progressively engendered a trust of these services. Staff have ‘bought into’ the concept of no-fee and third party commercial services being intrinsic to the university information dynamic, and as a consequence the display of trust and acceptance by staff communicates a message to our students that trusting your learning exchange, in the form of personal information and communication, to no-fee services is an acceptable thing. In fact, in some cases, personal technical allegiances can be so extensive, that staff adopt a somewhat evangelical approach to the promotion of no-fee services with their students. This mindset has been questioned (Vaidhyanathan, 2009), and as universities now place greater emphasis on all dimensions of student data for their own analysis purposes, it is likely that the separate and explicitly commercially oriented data management practices of licenced and no fee services will come under greater institutional scrutiny. DP:2 653 Third party technologies- the role of Learning Analytics The aggregation of data made possible by increasingly sophisticated and interconnected university learning environments, has given rise to the burgeoning field of Learning Analytics (Siemens, 2012). Where contemporary large data analysis techniques, are able to be applied to the inherently complex nature of the online learning experience, and provide an evidence basis for educational decision making. Universities have traditionally invested in the learner experience, and now are beginning to explicitly invest in the aggregation, management and analysis of learner data. Learner data is progressively becoming a currency of value to both educational institutions and the third party providers. This trajectory is considered likely to continue (Chatti et al, 2014), as increasingly sophisticated analytics will become available with predictive and personalised capabilities focused on individual learner support across a wide range of factors that underpin academic performance and student success in University education. An important ongoing aspect of the implementation of Learning Analytics in University systems is the challenge of upholding ethical practices and ensuring transparency of process to the university community. Particularly as learning is or should be owned by the student. The maintenance of ethical data practices within an institution can only be sustained with a staff body that is informed and engaged with data practices and the risks associated with abuse of the ethical management of both student and staff data. This practice must be extended to focus on how, to what extent, and under what conditions, licenced no-fee technologies are integrated within the university learning ecosystem. After all, the contemporary student expects us to care about them and their future (Worley, 2011). Conclusions Third party licenced and no-fee services introduce data management practices into university learning systems that require academic staff to transition to a more complete knowledge of the risks associated with such systems and the full responsibilities inherent in the technology choices made in the academic learning context. Specifically, as university Learning Analytics initiatives are advanced, priority should be placed on ensuring that staff knowledge and understanding of data management practices of third party systems are a strong component of the development dialogue. Equally, institution-level decision making needs to be receptive and responsive to the academic voice to ensure that the LMS and its associated ecosystem as fully as possible meets learning and teaching requirements, and that the context of use of licenced no-fee technologies are bounded in a manner that best secures the responsibilities that educators have for their students. References Chatti, M. A., Lukarov, V., Thüs, H., Muslim, A., Yousef, A. M. F., Wahid, U., Greven, C., Chakrabarti, A., & Schroeder, U. (2014). Learning Analytics: Challenges and Future Research Directions. eleed, Issue. 10, [verified 20 June 2015] https://eleed.campussource.de/archive/10/4035 Dalziel, J. (2013), The LAMS community: Building communities of designers, In H. Beetham, & R. Sharpe, R. (Eds.), Rethinking pedagogy for a digital age: Designing for 21st century learning. Routledge. Hallam, G. (2012). Briefing paper on eTextbooks and third party eLearning products and their implications for Australian university libraries. Council of Australian University Librarians. http://eprints.qut.edu.au/55244/ [viewed 22 June 2015] Keppell, M., Souter, K., & Riddle, M. (Eds.). (2011). Physical and Virtual Learning Spaces in Higher Education : Concepts for the Modern Learning Environment. Hershey, PA, USA: IGI Global Siemens. G. (2012). Learning analytics: envisioning a research discipline and a domain of practice. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK '12), Simon Buckingham Shum, Dragan Gasevic, & Rebecca Ferguson (Eds.). ACM, New York, NY, USA, 4-8. http://doi.acm.org/10.1145/2330601.2330605 Vaidhyanathan, S. (2009). The Googlization of Universities, The NEA 2009 Almanac of Higher Education, http://www.nea.org/home/32967.htm [viewed 28 May 2015] Worley, K. (2011) Educating College Students of the Net Generation Adult Learning, Adult Learning 22(3), 31-39. DP:3 654 Pawlaczek, Z. & Allan, G. (2015). Decisions and designs for building enterprise learning systems within an enabled learning paradigm: The case of third party technologies. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. DP:1-DP:4). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. DP:4 655 Designing for “Flexibility”: Exploring the Complexities of Dual-Mode Teaching Karin Barac Lynda Davies Lenka Boorer Griffith University, Australia Griffith University, Australia Griffith University, Australia The demand to offer students more flexibility in their university study options has seen a growth in multiple course offerings in different modes of learning such as on-campus, online or a mix of both (blended). In line with this demand for flexibility there has been a need for universities to streamline practices to meet shrinking budgets. This environment has facilitated the growth of dual-mode teaching where faculties attempt to teach online and on-campus cohorts together with variable results. The curriculum, staffing and student expectation demands of these different modes of delivery are often at odds and it is becoming more difficult to meet these demands while maintaining a high quality teaching and learning environment. We would like to share experiences and discuss with like-minded colleagues how to approach this particular design challenge in the hopes of developing some guidelines and practical examples that can inform us all. Keywords: Online and Blended Learning, Academic Staffing, Student Expectations Introduction Nationally and internationally universities are striving to attract and retain students through offering flexibility in study options as a response to the ever-increasing competitive environment. This idea of flexibility centers on the idea of study occurring at “any time, any place” allowing students to “balance” study with work and other life commitments. The increasing demand for flexibility in study options has seen a growth in online and blended learning offerings of courses (or units) within university programs. While this demand for flexibility initially presents as an issue of curriculum design, the reality is that it becomes a complex interplay between curriculum design, staffing, and administration systems to be successful. The current university climate of shrinking budgets, however, has meant schools look to rationalize offerings and workload allocations to balance the finances. One of the effects of rationalization has seen an increase in the merging of on-campus and online offerings taught by single teaching teams as a single cohort. Our university calls this dual-mode teaching. While we have developed a framework of course design (Barac, Davies, Duffy, Aitkin, & Lodge, 2013) that makes best use of evidence-based research to deliver courses that serve this new single cohort there has been significant dissent from the different sub-cohorts of students (domestic, international and online) to the dual-mode approach. In the past, to offer the best learning experience to students in different modes required approaches that were often at odds. In this new world of learning, how do we move forward in designing courses that will serve the need for flexibility while also serving the expectations of both domestic and international students? Where do we find the balance between quality course delivery; equitable workload and staffing allocations; and meeting student expectations based on their personal learning needs and perceptions? Problem For the past two years we have been implementing dual-mode teaching in a considered design approach that makes the best use of evidence-based practice to ensure the quality of the course design serves on-campus and online students. This has required a balancing act between student and academic expectations of how these learning environments should operate. First, we have found a dissonance between student expectations of their learning experience and their demand for flexibility. Domestic students demand flexibility but do not want to lose their timetabled lectures and tutorials. International students travel to another country to get an on-campus experience. In our postgraduate degrees especially, this on-campus, in-country experience is threatened by what they see as a privileging of independent study over daily face-to-face contact. DP:5 656 These divergent student perceptions are problematic given that, in design terms, flexibility relies on a move to student-centred approaches that use technologies to facilitate successful learning. As blended learning designs (such as those using flipped classrooms) proliferate the success of these learning environments rely more and more on students to accept responsibility for their role in the learning environment. Recognising and fulfilling that personal responsibility is something with which many students seem to struggle. Research has also shown, unfortunately, that as course design moves towards a blended approach students equate less time on campus with less time on task (Vaughan, 2007). Studies in student perceptions of online and blended learning environments have found that students still place a high value on face-to-face interactions with their teachers (Conole, De Laat, Dillon, & Darby, 2008; Russell, Malfroy, Gosper, & McKenzie, 2014). We have found that students struggle with the idea of contact with the teaching staff consisting of something other than the traditional face- to-face style. Consequently, many do not see online interactions with staff as “contact”. This is particularly worrying as we move more blended with our on-campus offerings. Second, for academics, the design process for dual mode teaching is challenging as the move to teaching online requires a fundamental shift in their ideas of teaching and learning. Many who are comfortable with the on-campus lecture-tutorial model struggle with how to design courses in both spaces that do not privilege one mode over the other and maintains an equitable learning experience. The difficulties staff encounter as they redesign their courses are exacerbated when course development and online teaching is under-recognised during workload allocation processes. This often occurs because faculty executives under-estimate the complexity of the task and how much time it takes to re-design pedagogy, curriculum progression, student interaction with their course sites, content, peers and teaching staff, and content resources. Contrary to popular expectations, as students experience these re-designed courses for the first time (and particularly early in the semester), their demands on the teaching team often exceed those experienced during a traditional course delivery. It takes time and effort to bring students successfully into the new culture of learning. We have seen that as students take more courses designed well for the online environment, many change their behavior accordingly, although there are those who still resist the new methods. If faculty executive do not understand and allow for these challenges, the inadequate workload allocation presents a danger to the quality of the over all course delivery and hence the student experience. Sharing Session In the Arts, Education and Law Group the learning and teaching support team developed a design process that makes academics preclude mode from their initial decisions about course design. Once the initial framework of content and learning activities had been decided then they were asked to make decisions on where, when, and how each cohort would interact with these. In this way, the course would be designed towards best practice student-centered approaches rather than the traditional ideas of face-to-face delivery. In this context we advocated that academics thought of the amount of time students needed to give their study per week remaining the same: that is, a 10CP course equates to 10 hours work by students. What changes is how students interact with teaching staff, peers and content for the 3 hours (e.g., including listening/watching recorded mini-lectures) and then work for a further 7 hours independently (and with peers if appropriate). This independent study includes doing readings, assessment preparation, learning activities etc. We found in our initial course designs, however, that we did not adequately align incoming students’ expectations of study in the course with how the course was to be taught. Despite instructional text and tips throughout the course site, many students did not make the shift in their thinking of what contact looks like in the new environment, nor did they actively understand that they were equal partners in the learning transaction. It is for this reason we are currently working on embedding info- graphics that will communicate this to the students in a way that our previous textual and teaching methods did not convey. Discussion Points and Goals DP:6 657 We would like to get together with like-minded colleagues and have the opportunity to discuss the following questions: -  How do you approach the design of courses that are taught both on-campus and online?  Does your university allow students to ‘dip’ in and out between on-campus and online? How, is this achieved?  How do you accommodate ideas of flexibility in programs with large (and/or significant) international student cohorts?  What is the ideal contact time per week between teaching team and students in flexible environment? What do you define as contact?  What workload allocation does your university provide academics for the design and teaching of online or blended courses? Should it be different? The outcomes of this session will be to work up some general guidelines and examples of good practice that participants can take back to their individual contexts to inform their decisions around program and course design and staffing implications for online and dual-mode teaching. References Barac, K., Davies, L., Duffy, S., Aitkin, N., & Lodge, J. (2013). Five stages of online course design: Taking the grief out of converting courses for online delivery. Paper presented at the Ascilite, Sydney. Conole, G., De Laat, M., Dillon, T., & Darby, J. (2008). ‘Disruptive technologies’,‘pedagogical innovation’: What’s new? Findings from an in-depth study of students’ use and perception of technology. Computers & Education, 50(2), 511-524. Russell, C., Malfroy, J., Gosper, M., & McKenzie, J. (2014). Using research to inform learning technology practice and policy: a qualitative analysis of student perspectives. Australasian Journal of Educational Technology, 30(1). Vaughan, N. (2007). Perspectives on blended learning in higher education. International Journal on E- Learning, 6(1), 81-94. Barac, K., Davies, L., & Boorer, L. (2015). Designing for “Flexibility”: Exploring the Complexities of Dual-Mode Teaching. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. DP:5-DP:7). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. DP:7 658 Connecting or constructing academic literacies on Facebook Mark Bassett Auckland University of Technology This paper outlines proposed doctoral research into how postgraduate students develop academic literacies within the bounds of learning theories and Web 2.0 tools that their lecturers select. Lea and Street’s (1998) academic literacies approach, which views literacies as contested social practices, forms the overarching view of literacy in this research. Over one semester, multiple case studies of postgraduate students will be conducted as they complete a paper within their subject of study. Students will use a private Facebook community to complete learning tasks and engage in student initiated discussions. The learning tasks will provide opportunities to examine the student experience of both the constructivist and connectivist paradigms. The aim is to further understanding of the student experience that can inform the creation of sound, theory driven Web 2.0-based learning tasks that effectively assist students in the development of their academic literacies. Feedback on the proposed research is sought from the Ascilite community. Keywords: Facebook, academic literacies, constructivism, connectivism Introduction Postgraduate students have varying levels of academic experience and must also develop numerous academic literacies (Lea & Street, 1998) as they engage with their chosen subject of study. As they do this, students are increasingly provided with tasks that make use of Web 2.0 tools. Web 2.0 learning tasks are often underpinned by constructivist learning theory (Cochrane, 2012; Conole, 2010), with the more recent connectivist theory of learning (Siemens, 2004) also gaining currency. This paper outlines a proposed PhD research project that aims to describe the student experience of developing academic literacies through the use of Facebook within the constructivist and connectivist pedagogies. In-depth case studies, using constructivist grounded theory methods of data analysis, will provide rich descriptions of how students develop their academic literacies during a semester-long paper that embeds academic literacies learning into course content via Web 2.0 tools. The paper ends with a call for critical input from the Ascilite community. Facebook Social integration into university life is a clear function of Facebook when used in learning and teaching. Duffy (2011) argues that Facebook can connect students with peers and teachers in communities, and that being part of such communities is crucial to successful learning experiences. McCarthy’s (2010) qualitative study into the use of Facebook in a blended learning environment for first-year tertiary students indicated that interactions between EAL learners and learners who had English as a first language were enhanced; communities were started online and then those networks realised during face to face classes. The measurable impact of Facebook on academic achievement is uncertain, with perceptions of its usefulness as a learning tool mixed at best. In their quantitative study of student perceptions of Facebook, Kabilan, Ahmad and Abidin (2010) found that when English language learners focused on learning tasks more than socialising, they found Facebook useful for improving writing and communication skills, with the added incentive of not feeling embarrassed about making mistakes. In a three year study of Facebook as an online learning environment for first year undergraduate students in Australia and Singapore, McCarthy (2013) found that involvement in virtual discussions allowed time for students to create and measure their responses. Kirschner and Karpinski (2010) surveyed 102 undergraduate and 117 graduate students in the United States to examine the impact of Facebook use on Grade Point Average (GPA). Facebook users had a lower GPA and spent less time studying than non-users. Similar results were obtained in another DP:8 659 survey of 1,839 undergraduate students in the United States (Junco, 2012); with Facebook using students having a lower GPA compared to non-users. Also, a study of 239 undergraduate students in Sweden reported that Facebook use negatively influenced assignment preparation (Rouis et al., 2011). In all of the research summarised in the previous paragraph, Facebook was not used as a learning tool; its use was social only and outside the bounds of the intended learning and teaching context. Therefore, if students use Facebook for purposes other than learning, it can be a disruption that can impact negatively on academic achievement. In contrast, if lecturers purposefully employ Facebook based on sound pedagogy, it can have a positive effect (Duffy, 2011; Kabilan et al., 2010; & McCarthy, 2010, 2013). However, in their quantitative study of 210 undergraduate and 32 postgraduate students’ perceptions of Facebook, Irwin, Ball, Desbrow and Leveritt (2012) found that only 51% of the students thought it was an effective learning tool. Paradoxically, 76% of the students recommended that lecturers use Facebook in future courses, with Irwin et al. concluding that further research would be necessary in order to ascertain whether and how it could enhance learning. Academic literacies Since the 1970s, the term literacy itself has taken on new life as it is attached to various issues or disciplines, such as “‘oral literacy’, ‘visual literacy’, ‘information literacy’, ‘media literacy’, ‘science literacy’ and even ‘emotional literacy’” (Lankshear & Knobel, 2006, p. 20). Each of these different literacies can be seen as “a specific kind of competence, an ability to function with informational tools in the named domain, be it computers, geography, or something else” (Newman, 2002, p. 33). The academic literacies approach (Lea & Street, 1998, 2006) defines literacies in the plural. From this perspective, academic literacy is not definable in a singular form as it is not the same for individual students and is influenced by their own background, as well as the specific subject they are studying and the institutional context. Lea and Street (2006) state that the academic literacies approach “is concerned with meaning making, identity, power and authority, and foregrounds the institutional nature of what counts as knowledge in any particular academic context” (p. 369). Constructivist learning theory Social constructivism has been the learning theory of choice for a considerable amount of research into the use of Web 2.0 in tertiary education (Cochrane, 2012; Conole, 2010); Web 2.0’s characteristics of peer to peer collaboration and user generated content appear to resonate well with the constructivist focus on student-centred, social, and collaborative activities. There are numerous examples of Web 2.0-based academic literacies learning initiatives that take a constructivist approach (Beckett, Amaro-Jiménez & Beckett, 2010; Snodgrass, 2011; Wingate & Dreiss, 2009), with students enabled to discover new information. Having the teacher provide minimal / no direction with students solving problems, either on their own or in groups (Biggs & Tang, 2011) can assist students with gaining entrance to the discourse of their discipline by discovery. However, for complex tasks, such as examining educational research methods, students may struggle to learn effectively without sufficient guidance from a teacher. From the cognitivist perspective, “[m]inimally guided instruction appears to proceed with no reference to the characteristics of working memory, long-term memory, or the intricate relations between them” (Kirschner, Sweller & Clark, 2006, p. 76). Connectivist learning theory The focus of constructivist learning theory is the knowledge construction of the individual (Harasim, 2012), but in a Web 2.0 context, learners can collaborate with other learners across networks, enabling shared knowledge creation. Differing from constructivist views of knowledge construction occurring within individuals, connectivist learning theory posits that knowledge construction occurs within networks between individuals (Siemens, 2004). Criticisms of connectivism are that it is not really a learning theory, but more of a guide for online pedagogy, and that existing theories can be be adapted to sufficiently explain learning in a digital age (Kop & Hill, 2008). However, in attempting to reconcile the academic literacies approach with a theory of learning, connectivism perhaps offers an appropriate landscape. The academic literacies approach seeks to redress imbalances between what institutions prescribe academic literacy to be and what academic literacy actually is for individual students (Lea & Street, 1998) who come from highly individualised backgrounds. Because academic literacy can be taken as socially contested and individual, it is not appropriate for a lecturer to then DP:9 660 define one academic literacy for all students. Examples of connectivist approaches to learning and teaching academic literacies include: the use of social bookmarking for managing reading lists and notes (Dujardin, Edwards & Beckingham, 2012); students collaborating with peers to refine their academic writing through blogging (Dujardin, 2012); and the development of critical thinking skills to choose information when a learner needs it, and to have capacity to learn what is not yet conceptualised (Ravenscroft, 2011). Also, Cochrane worked with Journalism lecturers to shift assessment practices (Cochrane, Antonczak, Gordon, Sissons & Withell, 2012). The assessment involved students using Storify to collate comments from social media on a current news item, and then using mobile devices to provide critique of the social media comments. Compared with ttraditional essay assessments, student work on Storify demonstrated both more critique and creativity. Method Multiple case studies will be the design for data collection because case studies enable deeper understandings of how individuals act and interact within a particular context (Berg, 2007). The development of student academic literacies in a Web 2.0-based constructivist learning environment is the contemporary phenomenon to be investigated. The context is a postgraduate qualification at a New Zealand university. The bounded system (Yin, 2009) that the phenomenon occurs within is a semester long paper, with the units of analysis being individual students enrolled in the paper. To enable a potentially deep understanding of each case (Ary, Jacobs, Razavieh, and Sorensen, 2006), three sources of evidence will be analysed: a test of student academic literacy; face-to-face interviews; and samples of student writing both online and through traditional written assessments. Constructivist grounded theory methods of data analysis (Charmaz, 2006) will be employed. Because “[t]he analysis of case study evidence is one of the least developed and most difficult aspects of doing case studies” (Yin, 2009, p. 127), the well-established constant comparative analytic techniques of grounded theory provide a clear framework for data analysis. The research will focus on the postgraduate student experience of one semester-long blended learning paper at a New Zealand university. The researcher is based in a university Student Learning Centre, and will work in collaboration with a Faculty-based lecturer. Academic literacies learning is embedded into the course content of this paper, with some student activity completed off-campus. For formative assessment tasks, students must generate their own blog posts, and also critique the blog posts of peers using a private Facebook community. Students also engage in informal discussions within the Facebook community, some of which are led by the lecturer, while others are spontaneously generated by students. Student experience of Facebook is varied, with most cohorts having had little or no experience of its use for learning and teaching purposes. Expectations and call for feedback This research can make positive contributions to the learning experiences of students and the teaching experiences of lecturers. Both of these communities grapple with the lived experiences of learning and teaching in an increasingly digital landscape. Rhetoric permeates this landscape: the educational benefits of Web 2.0; the argued virtues of both constructivist and connectivist learning theories; and the complexity of academic literacies learning which is embedded into subject content. Understanding more clearly academic literacies learning tasks that are embedded into subject content, that are Web 2.0-based, and that draw on either constructivist or connectivist learning theory could help teachers to enhance their practice. This could occur through the rigorous design of tasks for academic literacies learning embedded into course content; and appropriate use of Web 2.0 tools to facilitate these tasks. Furthermore, as constructivism and connectivism are likely to influence the pedagogical decisions that lecturers make about how students can and should learn, the student experience of those decisions needs to be analysed. For any cohort of students who learn with Web 2.0, their learning experience is mediated by the lecturer and how Web 2.0 is blended with the overall curriculum. An analysis of this particular cohort of students may contribute to the creation of sound, theory driven Web 2.0-based learning tasks that effectively assist students in the development of their academic literacies. An outcome of this project could be the establishment of a robust blueprint for further research into how these tasks could be adapted for use with other student cohorts in a variety of disciplines. The author requests the invaluable feedback of learning technologists and academics within the Ascilite DP:10 661 community in order to refine and/or augment this proposed PhD research project. References [First level heading style] Ary, D., Jacobs, L., Razavieh, A., & Sorensen, C. (2006). Introduction to research in education (7th ed.). Belmont, CA: Thomson Wadsworth. Beckett, G.H., Amaro-Jiménez, C., & Beckett, K.S. (2010). Students' use of asynchronous discussions for academic discourse socialisation. Distance Education, 31(3), 315-335. doi: 10.1080/01587919.2010.513956 Berg, B. (2007). Qualitative research methods for the social sciences (5th ed.). Boston, MA: Pearson Education. Biggs, J., & Tang, C. (2011). Teaching for quality learning at university (4th ed.). Maidenhead, England: Open University Press. Charmaz, K. (2006). Constructing grounded theory. Thousand Oaks, CA: Sage. Cochrane, T. D. (2012). Critical success factors for transforming pedagogy with mobile Web 2.0. British Journal of Educational Technology 45(1), 65-82. doi: 10.1111/j.1467-8535.2012.01384.x Cochrane, T., Antonczak, L., Gordon, A., Sissons, H., & Withell, A. (2012). Heutagogy and mobile social media: post Web 2.0 pedagogy. In M. Brown, M. Hartnett & T. Stewart (Eds.), Future challenges, sustainable futures. Proceedings ascilite 2012 Wellington. (pp.204-214). Conole, G. (2010). Stepping over the edge: The implications of new technologies for education. In M. Lee & C. McLoughlin (Eds.), Web 2.0-based e-learning: Applying social informatics for tertiary teaching (pp. 394-415). Hershey, PA: IGI Global. Dujardin, F. (2012). Educators’ digital literacies: The role of pedagogical design in innovation. In C. M. Stracke (Ed.), Proceedings of the European conference LINQ, 2012 (pp. 26-40). Berlin, Germany: GITO. Dujardin, F., Edwards, K., Beckingham, S. (2012). Diigo: Social bookmarking in a professional communication course. In C. Cheal, J. Coughlin, & S. Moore (Eds.), Transformation in teaching: Social media strategies in higher education (pp. 243-274). Santa Rosa, CA: Informing Science Press. Irwin, C., Ball, L., Desbrow, B., & Leveritt, M. (2012). Student perceptions of using Facebook as an interactive learning resource at university. Australasian Journal of Educational Technology, 28(7), 1221-1232. Retrieved from http://ascilite.org.au/ajet/submission/index.php/AJET/article/view/798/98 Junco, R. (2012). Too much face and not enough books: The relationship between multiple indeces of Facebook use and academic performance. Computers in Human Behaviour, 28(1), 187-198. doi:10.1016/j.chb.2011.08.026 Kabilan, M.K., Ahmad, N., & Abidin, M.J.Z. (2010). Facebook: An online environment for learning of English in institutions of higher education? The Internet and Higher Education, 13(4), 179-187. doi: 10.1016/j.iheduc.2010.07.003 Kirschner, P. A., & Karpinski, A. C. (2010). Facebook and academic performance. Computers in Human Behaviour, 26(6), 1237-1245. doi:10.1016/j.chb.2010.03.024 Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75-86. doi: 10.1207/s15326985ep4102_1 Kop, R., & Hill, A. (2008). Connectivism: Learning theory of the future or vestige of the past? International Review of Research in Open and Distance Learning, 9(3), 1-13. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/523/1103 nd Lankshear, C., & Knobel, M. (2006) New literacies: Everyday practices and classroom learning (2 ed.). Maidenhead, England: McGraw-Hill. Lea, M. R., & Street, B. (1998). Student writing in higher education: An academic literacies approach. Studies in Higher Education, 23(2), 157-172. doi: 10.1080/03075079812331380364 Lea, M. R., & Street, B. (2006). The academic literacies model: Theory and applications. Theory Into Practice, 45(4), 368-377. Retrieved from http://www.jstor.org/ McCarthy, J. (2010). Blended learning environments: Using social networking sites to enhance the first year experience. Australasian Journal of Educational Technology, 26(6), 729-740. Retrieved from http://ascilite.org.au/ajet/submission/index.php/AJET/article/view/1039/300 Newman, M. (2002). The designs of academic literacy: A multiliteracies examination of academic achievement. Westport, CT: Greenwood. Ravenscroft, A. (2011). Dialogue and connectivism: A new approach to understanding and promoting dialogue-rich networked learning. International Review of Research in Open and Distance DP:11 662 Learning, 12(3), 139-160. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/934/1829 Rouis, S., Limayem, M., & Salehi-Sangari, E. (2011). Impact of Facebook usage on students’ academic achievement: Role of self-regulation and trust. Electronic Journal of Research in Educational Psychology, 9(3), 961-994. Retrieved from http://www.investigacion- psicopedagogica.org/revista/articulos/25/english/Art_25_620.pdf Siemens, G. (2004). Connectivism: A learning theory for the digital age. Retrieved September 21, 2014, http://www.elearnspace.org/Articles/connectivism.htm Snodgrass, S. (2011). Wiki activities in blended learning for health professional students: Enhancing critical thinking and clinical reasoning skills. Australasian Journal of Educational Technology, 27(4), 563-580. Retrieved from http://ascilite.org.au/ajet/submission/index.php/AJET/article/view/938/214 Wingate, U., & Dreiss, C. (2009). Developing students’ academic literacy: An online approach. Journal of Academic Language & Learning, 3(1), 14-25. Retrieved from http://journal.aall.org.au/index.php/jall/article/view/65/58 Yin, R. K. (2009). Case study research: Design and methods (4th ed.). Thousand Oaks, CA: Sage. Bassett, M.. (2015). Designing for “Flexibility”: Connecting or constructing academic literacies on Facebook. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. DP:8-DP:12). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. DP:12 663 Technology issues in blended synchronous learning Barney Dalgarno Matt Bower Mark J. W. Lee uImagine School of Education School of Education Charles Sturt University Macquarie University Charles Sturt University Gregor Kennedy Centre for the Study of Higher Education University of Melbourne Universities have responded to demand from students for increased time flexibility by providing online alternatives to face-to-face education, typically centered around the provision of online learning resources along with asynchronous online learning activities. More recently, synchronous options afforded by the capabilities of web conferencing tools, video conferencing tools and virtual worlds have emerged, providing the potential to bring together face-to-face and remote students using blended synchronous learning strategies. In the OLT-funded project Blended Synchronicity: Uniting on-campus and distributed learners through rich-media real-time collaboration tools, seven case studies of the use of blended synchronous learning strategies were explored. This discussion paper highlights the technology considerations and technology setup issues emerging from the case studies, as background material for a round table discussion session at the conference. Keywords: blended synchronous learning, video conferencing, web conferencing, virtual worlds Introduction Changes in the lifestyle patterns and expectations of university students, along with the availability of new technologies, have presented both challenges and opportunities to university educators in recent years. The increasing diversity of the student population and the rising cost of higher education have led to a student population more constrained by the time demands of work and family than ever before (Gosper, Green, McNeill, Phillips, Preston & Woo, 2008; James, Krause & Jennings, 2010). This has given rise to an ever-growing demand for more flexible alternatives to face-to-face study. Alongside this, new online learning technologies and the ubiquitous availability of mobile devices have provided new affordances for anywhere, anytime work and study (Kearney, Schuck, Burden & Aubusson, 2012). Universities have largely responded to these changing demands of students by providing online learning resources such as reading materials and recorded lectures along with asynchronous online learning activities through discussion forums, supported by the capabilities of learning management systems. More recently, the availability of new online and mobile technologies, coupled with a renewed focus on communication and collaboration skills within graduate outcomes, has led to increased interest in the provision of synchronous learning opportunities for online students. In this context, web conferencing tools such as Adobe Connect (Butz et al., 2014) and Blackboard Collaborate (Spann, 2012), video conferencing tools such as Skype (Cunningham, 2014), and virtual worlds such as Second Life (Beltrán Sierra, Gutiérrez & Garzón-Castro, 2012) have been used to bring together on-campus and remote students in real time. In the study, Blended synchronicity: Uniting on-campus and distributed learners through media-rich real-time collaboration tools (an Australian Office of Learning and Teaching funded project) seven case studies involving blended learning designs were explored (see Bower, Kennedy, Dalgarno, Lee, Kenney & de Barba, 2012; Bower, Kenney, Dalgarno, Lee & Kennedy, 2014; Bower, Kennedy, Dalgarno, Lee & Kenney, 2015). This paper builds on earlier publications, which provide details of a large survey of usage of these approaches (Bower et al., 2012) along with in-depth case studies (Bower et al., 2014; Bower et al., 2015), to provide a discussion of the key technology issues emerging from the study and considerations for selection and appropriation of technologies. DP:13 664 Data collection The seven case studies from which the data presented here was collected were identified drawing on 1748 responses to a 2011-2012 survey of Australian and New Zealand tertiary educators on rich- media synchronous technology usage (Bower et al., 2012). Table 1 provides a summary of the technologies, discipline foci and learning tasks within these seven case studies. See Bower et al.(2014) and Bower et al.(2015) for more detailed descriptions of the cases. Face to face learning activities during the lessons were video recorded as were selected online activities within the web conferencing, video conferencing or virtual worlds session. Student reflections on the lesson were gathered through a questionnaire and students also participated in focus group interviews following the lessons. Teaching staff were also interviewed before and after the lesson. Table 1: The seven case studies of blended synchronous learning strategies Case Technology Discipline and content Learning task Study focus 1 Web conferencing using Finance, investment Collaborative evaluation Adobe Connect 2 Room-based video Healthcare, quality Collaborative evaluation conferencing using Access improvement Grid 3 Web conferencing using Histology, microscopic Large-group Q & A and Adobe Connect tissue analysis small group problem solving 4 Web conferencing using Statistics, hypothesis Collaborative problem Blackboard Collaborate testing solving 5 Virtual worlds using Second Mandarin language, Paired role-play Life authentic communication 6 Web conferencing using Sexology, exploration Lecture discussions Blackboard Collaborate of personal experiences 7 Virtual worlds using Teacher education, Collaborative evaluation AvayaLive technology in learning and design Emergent technology issues In the analysis of the data collected during the study across the seven cases a number of technology- related issues emerged which have implications for anybody planning to use blended synchronous learning approaches or anybody responsible for the technological infrastructure needed to support such approaches. These emergent technological issues and considerations are discussed in the following sections. Technology setup When setting up the audio for blended synchronous learning classes it was important to avoid audio feedback loops caused by sound coming out of one set of computer speakers and being detected by the microphone of another computer. In most cases this was managed by having all audio into and out of the face-to-face classroom run through the teacher’s machine. Another alternative to this was to have all participants use earphones. The main technology setup issue reported by students was that the microphone in the face-to-face classroom was unable to capture all of the student comments. This meant that the teacher needed to re-articulate student comments into the microphone in order for them to hear what was said. In these cases teachers paraphrased comments or in some instances did not relay them to remote students. This also led to interference with the flow of the lesson as remote students experienced periods of inaudible student commentary or face-to-face students listened to comments twice as their teacher repeated them. One important decision for teachers is whether to let students make audio contributions or to only allow them to use text chat. The advantage of enabling audio contributions is that it can enable more DP:14 665 rapid and more extensive contributions. It could also enhance the sense of co-presence and reduce cognitive overload caused by trying to work with two visual modalities such as the text chat and notes area at the same time. On the other hand, text was more reliable than audio and enabled many simultaneous contributions by participants. Using audio and text communication at the same time was perceived by some students to result in fragmented conversation that was hard to follow. Additionally, slow typing speed sometimes led to text appearing after the issue being discussed had moved on. The importance of being able to record the session was observed across cases, with students who participated in web conferencing sessions citing this as an advantage of the approach. The inability to easily record and disseminate the virtual worlds and room-based video conferencing lessons was perceived to be a limitation by both teachers and students. Also, one teacher noted that the inability to record individual breakout rooms during web conferencing sessions was a distinct weakness of the system. Problems and constraints associated with the technological approach Technological issues were reported in each case, ranging from minor to substantial in impact. Internet speed and technology reliability were reported in all case studies, resulting in delayed or choppy audio or in some cases, temporary inaudibility. Some teachers indicated that they needed to monitor the system for audio feedback loops, and disable the microphone rights of other participants if feedback loops occurred. In addition to these general issues, the following are some of the issues emerging from specific cases: • Not noticing that the teacher’s microphone was muted for 2 minutes of the lesson; • Latency on the interactive whiteboard slowing the pace of the lesson; • The web conferencing system crashing, possibly because too many breakout rooms were open at once; • Students experiencing temporary difficulty accessing the features of breakout rooms; • Students being inexplicably logged out of the system; • Inability of iPad client software to allow students to draw on the web conferencing whiteboard; • The teacher’s browser crashing during review of group work responses in the virtual world; and • Difficulty navigating and interacting within a 3D environment. Some difficulties with the technological approach related not to the technology itself, but conventions and etiquette relating to its use. For instance, some remote students reported that it could be hard to know when to talk because of the lack of visual cues. Strategies for working with technology A range of strategies for managing the technology-mediated nature of the environment emerged from the student and teacher questionnaires and interviews and researcher observations. All teachers started their session at least 10 minutes before the scheduled lesson start time so that they and students could test the technology setup. One teacher pointed out that it was important to prompt students for contribution at regular intervals – this not only promoted engagement and learning, but also enabled the teacher to assess whether or not the technology was working as intended. One teacher also recommended micro-strategies for working with text chat. For instance it is useful to ask distance students whether or not they have any questions because it can take time for them to write, in which case the lesson might have already moved on. Asking students to use the prefix “Q” enabled the teacher to more easily distinguish text chat questions (requiring responses) from comments. If there are ever problems with audio, it is important to remember to use text chat to ask students whether they can hear. Teachers were in general agreement that it was useful for them to develop skills in troubleshooting technological issues, because immediate technical assistance may not always be available on demand. Conclusion This paper has highlighted the key technology issues emerging from seven case studies of blended synchronous learning. Through the proposed round table discussion session at the conference participants will have the opportunity to ask questions about the particular issues highlighted and DP:15 666 share their own experiences in order to build collective understanding about this important aspect of blended synchronous learning design and support. The Blended Synchronous Learning Design Framework which was developed during the study will be used to help frame the discussion (Bower, Dalgarno, Kennedy, Lee & Kenney, 2014). References Beltrán Sierra, L.M., Gutiérrez, R.S. & Garzón-Castro, C.L. (2012). Second Life as a support element for learning electronic related subjects: A real case. Computers & Education, 58, 291-302. Bower, M., Kenney, J., Dalgarno, B., Lee, M. J., & Kennedy, G. E. (2014). Patterns and principles for blended synchronous learning: Engaging remote and face-to-face learners in rich-media real-time collaborative activities. Australasian Journal of Educational Technology, 30(3) 261-272. Bower, M., Kennedy, G. E., Dalgarno, B., Lee, M. J. W., & Kenney, J. (2015). Design and implementation factors in blended synchronous learning environments: Outcomes from a cross- case analysis. Computers & Education, 86, 1–17. http://dx.doi.org/10.1016/j.compedu.2015.03.006 Bower, M., Kennedy, G.E., Dalgarno, B., Lee, M.J.W., Kenney, J. & de Barba, P. (2012). Use of media-rich real-time collaboration tools for learning and teaching in Australian and New Zealand universities. In Future challenges, sustainable futures. Proceedings ascilite Wellington 2012. Bower, M., Dalgarno, B., Kennedy, G. E., Lee, M. J. W., & Kenney, J. (2014). Blended Synchronous Learning: A Handbook for Educators. https://blendsync.org/handbook Butz, N.T., Stupnisky, R.H., Peterson, E.S. & Majerus, M.M. (2014). Motivation in synchronous hybrid graduate business programs: A self-determination approach to contrasting online and on-campus students. MERLOT Journal of Online Learning and Teaching, 10(2), 211-227. Gosper, M., Green, D., McNeill, M., Phillips, R., Preston, G. & Woo, K. (2008). The impact of web- based lecture technologies on current and future practices in learning and teaching. Sydney: Australian Learning and Teaching Council. James, R., Krause, K.-L. & Jennings, C. (2010). The first year experience in Australian universities: Findings from 1994 to 2009. Melbourne, Australia: The University of Melbourne, Centre for the Study of Higher Education. Kearney, M., Schuck, S., Burden, K. & Aubusson, P. (2012). Viewing mobile learning from a pedagogical perspective. Research in Learning Technology, 20. Spann, D. (2012). 5 innovative ways to use virtual classrooms in higher education. In M. Brown, M. Hartnett & T. Stewart (Eds), Future challenges, sustainable futures. Proceedings ascilite Wellington 2012 (pp. 864-866). Dalgarno, B., Bower, M., Lee, M.J.W., & Kennedy, G. (2015). Technology issues in blended synchronous learning. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. DP:13-DP:16). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. DP:16 667 On the Evaluation of OLEs Using the HEART Framework Ilias Flaounas Aikaterini Kokkinaki Atlassian, Australia University of Bristol, United Kingdom In 2010 Google's researchers introduced the HEART framework for the evaluation of online products. HEART, which stands for Happiness, Engagement, Adoption, Retention and Tasks, tries to provide guidance on a set of key metrics that need to be measured in order to evaluate an online product in an objective and holistic manner. While each metric quantifies an angle of key factors, we need all of them in order to achieve safe conclusions. Our position is that the same framework could be used in the assessment of the deployment of an OLE. We present the framework and an example of its application. Keywords: evaluation framework, online learning environment Evaluation of Online Learning Environments Every modern educational institution offers access to an Online Learning Environment (OLE), or as interchangeably used in the literature: Virtual Learning Environments, Managed Learning Environments, Personal Learning Environments and Learning Platforms. An OLE has been characterised as an online space that includes the components through which the learners and the tutors participate in online interactions including online learning (Joint Information Systems Committee, 2006, p. 6). Without argument, an OLE has become an online space where a significant amount of the teaching experience of students takes place. However, it is not clear how an institution can measure the effectiveness and the impact of their OLE. This is even more difficult if the institution needs to measure the impact from students' perspective. Simple metrics such as the number of Daily Active Users, or Monthly Active Users that measure the number of students that login on a daily or monthly basis, may have significant hidden issues. To give an example, students may login every day to the OLE to access core materials for their courses because they were given no alternative option and not because they necessarily enjoy using the OLE. Traditional surveys may offer some limited insights and actually they are part of the HEART framework especially when the survey concerns measuring students' satisfaction. Nevertheless, the question on whether the evaluation of an OLE is complete and as objective as possible remains unanswered. In this paper, the Happiness-Engagement-Adoption-Retention-Tasks (HEART) framework is presented and its potential use as a measuring framework for OLEs in higher education is discussed. The original purpose of HEART framework was to help software designers to create online products, monitor their quality, detect problems and give directions for future modifications. Currently, the HEART framework is used by software companies for the evaluation of their online products. Although the HEART framework doesn't discuss specific metrics, it provides a set of recommendations on perspectives that need to be taken into account. The exact metrics can be decided by the higher education institutions based on their aims and needs. The HEART framework The HEART framework was presented by Google's User Experience (UX) researchers (Rodden et al., 2010), as an effort to establish a framework around user-centered metrics in the era of big data and analytics. In particular, Rodden et al. (2010) discuss not only the established practices in the UX community, i.e., small scale analysis of attitudinal and behavioral data, but also the opportunities created by taking advantage of the large scale data created by the instrumentation of online products. Goals, Signals and Metrics The HEART framework doesn't focus on specific metrics but rather presents a structured way to organise the metrics that should be captured in an evaluation in order to ensure that all the useful aspects are captured. According to HEART, the set of key themes are organised as rows and the set DP:17 668 of Goals-Signals-Metrics as columns as illustrated in Table 1. Each piece of new content, as for instance a new course or a new activity, is uploaded at the OLE for a particular purpose and with specific goals. The goals need to be well defined and each one should be measured by the use of one or more signals. Signals are what most people refer to as “metrics”. The distinction between signals and metrics is technical. A signal is a high level description of the quantity that a non-technical person wants to capture. For instance, a signal could be: “The number of students that are active within a day”. Metrics are more formal and low-level technical descriptions of signals and reflect the underline infrastructure of the OLE. To give an example, the metric of the afore mentioned signal could be: “The number of registered users in the OLE; which have a student status; who perform one or more actions of the set: accessing material, making comments or submitting coursework; within the time period of one day; and as captured by the analysis of the log files that store the meta-data of the OLE”. In another example, we may upload some new material at the OLE that we believe could enhance students' interaction with the OLE. The question that emerges is how can we assess whether this particular action was successful or not. Suppose that our goal is to increase by 50% the overall engagement of the new students with the OLE. In this case, one related signal could potentially be “the time spent by students at the OLE”. However, the actual metric that implements that signal requires some low-level details. In this hypothetical scenario, it would be necessary to split users' timeline in 5 mins slots because of the nature of the logs available and then capture if each student was active within each time slot. Further decisions involve: The identification of students who have left their browser tab open without interacting with the OLE, or those students who could be considered as outliers because of untypical high frequency of usage. Table 1 – Summary of the goals, signals and metrics for an OLE component based on the HEART framework Theme Goal Signal Metric Run survey during the 3rd week of the Run a survey with the We want 80% of course; answers will question “How would students to provide be in scale 1 to 5 and Happiness you rate the online positive or very we count the environment for the positive feedback. percentage of course?” students that answered 4 or 5. Measure number of We want 80% of logins from distinct students that use the Measure number of users that have Engagement OLE to visit it at least logins per week. student status per once per week. week based on log files. Measure the percentage of logins We want 100% of from distinct users students to access Measure number of that have student Adoption the OLE within the logins within the first status, per week first week at least 2 week. based on log files times. within the first week that they enrol to the course. Measure the fraction of students that use We want every the OLE out of the student that used the Measure the number number of students Retention OLE at least once of returning students. that logged in at least before, to revisit once within the every week. semester for the course. DP:18 669 We want 50% of Measure the number students, to complete Measure the number of active students, at least one self- of tests that are and the number of assessment test. completed and the self-assessment tests Task Success Also, less than 20% number of tests that that each student of the students are abandoned each completes or should drop out from week per student. abandons week over a started test. week. Themes of the Student Experience with the OLE The goals, signals and metrics should capture different and complementary aspects of students' experience. These aspects are organised in the core themes of the HEART framework as described below: Happiness. This theme answers the question: “How happy are the students from using the OLE?” The theme Happiness can be measured with the use of a traditional survey that asks a few simple questions such as: “How satisfied are you with the OLE?”, where students can answer in a Likert type scale. Alternatively, the survey could include open ended questions such as: “What do you like the most when you use the OLE?”. Qualitative techniques can be deployed for the analysis of this particular set of data. There is no need to ask students whether they use the OLE or not or which part they use the most or how often they use the OLE because these questions can be answered with the use of the OLEs analytics. Moreover, these questions are covered by the other themes of the HEART framework. Engagement. This theme measures the level of engagement of students who use an OLE with the use of analytics. In this theme, it is important to measure how frequently students visit the OLE, how much time they spend, what type of interactions students have with the different features of the OLE and the available content within a certain period of time (e.g. per month or per semester). Summary statistics can be produced per course, per department or any other segmentation that is useful for feature action. Adoption. New students have different needs compared to students who have used the OLE before. For example, they need to learn how to interact with the OLE. This explains why new students should be treated as a distinct cohort and the focus for them should be on the identification of problems and issues relevant to the adoption of the OLE. For the new students we may be interested to find out how easy it has been to get value out of the OLE, how many different features of the OLE have they used, or whether they have accessed all the available content or just a subset of it. A low usage of a specific feature, like direct interaction with other students via instant messaging, can potentially indicate that this feature is not easily “discoverable” by students. Retention. This theme aims to identify how often students re-visit the OLE. Here, we try to identify issues relevant with retention. For instance, the identification of cases where students visit specific page only once to get the course material but do not return, could be an indication that those students use the OLE as a repository for downloading material rather than a true online environment for learning. It is important to point out that retention is different to engagement as the former monitors whether students return to specific pages of the OLE despite the fact that they know what type of material is available on those pages. A low retention may reveal that the OLE doesn't provide long standing value. Tasks. Depending on the exact OLE setup, it may provide a set of different tasks that students may complete. For example a task could be the submission of coursework via the OLE instead of submitting it via email. The number of students that complete each task should be measured separately. Students may interact with the OLE and spend significant time using it, however this does not imply that they complete the tasks that we would like them to complete. Application Example We want to setup the OLE for a new course offered to students. Students use the OLE to find course DP:19 670 material, perform self-assessment tests and submit coursework. We want to evaluate the effectiveness OLE. We start by completing the goals column of Table 1 for each theme of the HEART framework. This answers the simple question of what success looks like. Then we define the signals and the metrics that we would need to measure to quantify each goal. As the students start using the OLE we can start monitoring the different metrics and start assessing how close or far we are from the original goals. Depending on the collected data, we can either support the argument that students get the value we targeted or detect issues that need to be tackled. We may discover for instance that adoption is high whereas engagement is low. This may imply that students try out the OLE at first, however they use it less often as time passes. In this case, we should take appropriate actions as for example, increase the quality of the material or decrease the quantity of the material offered. We may also discover that adoption, engagement and retention are high, but happiness is low. In this case it should be examined whether students use the OLE not because they like it but because they have no alternative choice. References Joint Information Systems Committee (2006). Effective Use of VLEs: Introduction to VLEs. http://tools.jiscinfonet.ac.uk/downloads/vle/what-is-vle.pdf [viewed 19 October 2013]. Rodden, K., Hutchinson, H., Fu, X. (2010). Measuring the User Experience on a Large Scale: User- Centered Metrics for Web Applications. In Proceedings of CHI 2010, ACM Press. http://research.google.com/pubs/pub36299.html Flaounas, I. & A. Kokkinaki, A. (2015). On the Evaluation of OLEs Using the HEART Framework. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. DP:17-DP:20). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. DP:20 671 A practitioner’s guide to learning analytics Cathy Gunn Jenny McDonald Claire Donald University of Auckland University of Otago University of Auckland New Zealand New Zealand New Zealand John Milne Mark Nichols Eva Heinrich Massey University Open Polytechnic Massey University New Zealand New Zealand New Zealand A growing body of literature identifies learning analytics as an emergent field of research that can deepen our understanding of learning and inform learning design practice. However, realizing the potential is not straightforward, as even defining learning analytics is vexed. For teachers and learning designers, the practical issue of how to engage with learning analytics data is problematic. This discussion paper begins by outlining the background to learning analytics at a practice level. Next, we introduce learning analytics frameworks, and one in particular that serves our aim to develop a guide for practitioners wishing to engage with learning analytics for different purposes. We will develop and refine the guide by mapping it to case-studies at NZ tertiary institutions, and through discussion with practitioners internationally. Our goal is to make analytics data more accessible and useful to teachers, learning designers and institutions. Keywords: Learning analytics; learning technology research; case study Background Learning analytics has been variously defined (e.g. Barneveld, Arnold & Campbell, 2012; Elias, 2011; Society for Learning Analytics Research, 2012). Cooper (2012a) suggests that a single definition is impossible because of the broad range of perspectives and motivations involved. For our purpose, which is working with teachers, learning designers and learners, Cooper’s description is useful: Analytics is the process of developing actionable insights through problem definition and the application of statistical models and analysis against existing and/or simulated future data. (p.7) Analytics data provides opportunities for informed decision-making at both institutional and practice level. Davenport et al (2010; p1) noted that 'putting analytics to work is about improving performance in key domains using data and analysis'. While this quote refers to the business world, the same principle can be applied to education, with potential benefits including:  Increased understanding of the effects of learning design decisions, learning contexts and what works in relation to stated learning objectives;  Deeper insight into the impact of different processes and practices in learning environments;  A rationale for change and the agility to respond to changing circumstances;  Ability to detect patterns and trends;  Enhanced decision-making where logic and supporting data is consistently applied;  Testing assumptions made in learning design and course planning. Testing assumptions in learning design and course planning is the focus of our research. As Lockyer, Heathcote and Dawson (2013) point out, learning analytics studies reported thus far tend to focus on relatively coarse indicators such as student retention and progression. The means and methods for more fine-grained analysis have yet to be fully explored, to: … inform teachers on the success and outcomes of their design of learning experiences and activities alongside monitoring student learning for direct support during the academic semester. (Lockyer, Heathcote and Dawson, 2013, p1441) DP:21 672 Many educational technology researchers (e.g. Reeves, 2011; Gunn & Steel, 2012) have noted the limitations of subjective data and the high level objective sources that reveal what is happening but not why. There is also a lack of meaningful ways to interpret trends that the data reveal at the level of practice. Learning analytics could be an important missing link for evidence-based practice if it offers ways to leverage insights from objective and passively collected data sources. The aim to develop evidence-based educational practice has been on the agenda for many years (e.g., Biesta, 2007; Clegg, 2005; Denzin, 2009); whether learning analytics is the means to move this agenda forward remains to be seen. Learning analytics frameworks Frameworks designed to map the landscape of analytics in general and learning analytics in particular represent a range of application levels and perspectives (e.g. Davenport, Harris, & Morison, 2010; Greller & Drachsler, 2012; Lockyer, Heathcote and Dawson, 2013). We have chosen to adopt as a starting point for our project, A Framework of Characteristics of Learning Analytics (Cooper, 2012b: Figure 1). Cooper identifies two key uses for his framework: first, to analyze existing applications and second, to develop principled designs for new projects. He also notes that some of the framework dimensions, which move from simple information gathering and extrapolation to actionable insight, may require capability beyond that currently available within educational institutions. The simple framework from Davenport, Harris, & Morison, 2010 (Table 1) highlights the move from more straightforward reporting and extrapolation to actionable insights such as modelling, experimentation, recommendations and predictions. Figure 1. The Framework of Characteristics of Learning Analytics (From Cooper, 2012b p.5) DP:22 673 While frameworks are useful to educational technology researchers and early adopters with a reasonable grasp of analytics capabilities, they are too conceptual for most teachers and learning designers to use as a basis to develop an analytics practice. Our experience shows that a degree of translation is required to make the application of learning analytics concepts accessible to those who might benefit most. A central goal of our project is to support that translation through the development of reporting templates and guidelines. Past Present Future What happened? What is happening What will happen Information now? (Reporting) (Extrapolation) (Alerts) How and why did it What is the next best What is the best / worst Insight happen? action? possible outcome? (Purpose) (Modeling, experimental (Recommendation) (Prediction, optimization, design) simulation) Table 1: Key Questions Addressed by Analytics (From Davenport, Harris & Morison, 2010 p7) Our project and case studies A scan of four participating institutions and the New Zealand tertiary sector identified a number of early adopters working on the analytics capabilities of different elearning systems. An invitation to become case studies for a national initiative attracted a number of willing participants. The formal project includes eight case studies covering a range of educational aims and contexts, including:  Development and enterprise-wide implementation of a reporting system to identify and record action taken with at risk students;  Design analytics functions and reports to answer specific questions posed by lecturers in an iterative learning design process;  Analyze data from different sources to identify course elements most strongly associated with student engagement in large lectures;  Explore the data produced by online skills development modules to determine the extent to which it can inform learning re/design;  Process student free text responses to identify features indicative of developing disciplinary fluency;  Translate LMS log data into inform students about their learning and help teachers design more effective teaching strategies. We were aware early on that the learning analytics field is complex; the data we will use has many dimensions and characteristics, various possible interpretations and different methods we may use. The case study method maintains the focus on practice, and creates the opportunity for intended users of the analytics tools to help us shape the data collection and reporting capabilities. In some cases, we have the ability to modify the elearning systems we are using to generate the data we want, while in others we are constrained by what is available through tools such as the institutional LMS. We are also aware that learning analytics is not just about data and statistical analysis. Various forms of feedback, observation, discussion, focus groups and interviews can all help to deepen our understanding of the teaching and learning process. The use of Cooper’s framework does not preclude this and we endorse the point in the Open University’s Guide to Learning Analytics (The Open University n.d.) that students should never be judged on their data alone. Our aims are modest compared to researchers such as Greller & Drachsler (2012). We do not aim to scope an entire field of research, but to develop a practical guide for teachers without specialist data analysis skills and institutions making tentative moves into the field of learning analytics to make use of data they already have. A current challenge is to manage access, analysis and interpretation of the data in a timely manner. Adding this dimension to elearning practice is a formidable task for some academics who may lack confidence in the applications of technology in education. Like all other areas of elearning practice, we believe it is important to lower the barriers to entry to a level where DP:23 674 non-technical users can participate. Open access is also a significant shift for institutions where privacy concerns and security measures had become the guiding principles of data management. These changes to individual and institutional practice require evidence to inform decisions as well as the subsequent actions. We appreciate Cooper’s invitation to make creative use of his framework to scaffold development of a design rationale and functional aspects of our own analytics practice. In turn, we hope readers find this discussion paper a useful step towards the co-construction of meaning and methods to make learning analytics an integral part of evidence-based learning design practice. References van Barneveld, A., Arnold, K. E., & Campbell, J. P. (2012). Analytics in higher education: Establishing a common language. EDUCAUSE learning initiative, 1, 1-11. Biesta, G. (2007). Why "what works" won't work: Evidence-based practice and the democratic deficit in educational research. Educational Theory, 57 (1), 1–22. Retrieved from http://dx.doi.org/10.1111/j.1741-5446.2006.00241.x doi:10.1111/j.1741-5446.2006.00241.x Clegg, S. (2005). Evidence-based practice in educational research: A critical realist critique of systematic review. British Journal of Sociology of Education, 26 (3), pp. 415-428. Retrieved from http://www.jstor.org/stable/30036076 Cooper, A. (2012a) “What is ‘Analytics’? Definition and Essential Characteristics, CETIS Analytics Series, vol. 1, no. 5 Cooper, A. (2012b) A Framework of Characteristics for Analytics, CETIS Analytics Series, vol. 1, no. 7 Davenport, T.H., Harris, J.G and Morison, R. (2010) Analytics at Work: Smarter Decisions, Better Results. Harvard Business Press, 2010. Denzin, N. K. (2009). The elephant in the living room: or extending the conversation about the politics of evidence. Qualitative Research, 9 (2), 139-160. Retrieved from http://qrj.sagepub.com/content/9/2/139.abstract doi: 10.1177/1468794108098034 Elias, T. (2011). Learning analytics: Definitions, processes and potential. Retrieved from http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf Greller, W and Drachsler, H. (2012) Translating Learning into Numbers: A Generic Framework for Learning Analytics, Educational Technology & Society, vol. 15, no. 3, pp. 42-57 Gunn, C. and Steele, C. (2012) Linking theory to practice in learning technology research. Research in Learning Technology 2012, 20. Lockyer, L., Heathcote, E., and Dawson, S. (2013) Informing Pedagogical Action: Aligning Learning Analytics With Learning Design. American Behavioral Scientist 57(10) 1439–1459 Society for Learning Analytics Research (2012) About. Retrieved from http://www.solaresearch.org/about/ Reeves, T. C. (2011). Can Educational Research be Both Rigorous and Relevant? Educational Designer 1(4). http://www.educationaldesigner.org/ed/volume1/issue4/article13/ The Open University (n.d.) Using Information to Support Student Learning, Retrieved from http://www.open.ac.uk/students/charter/sites/www.open.ac.uk.students.charter/files/files/ecms/web -content/using-information-to-support-student-learning.pdf Gunn, C., McDonald, J., Donald, C., Milne, J., Nichols, M., & Heinrich, E. (2015). A practitioner’s guide to learning analytics. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. DP:21- DP:24). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. DP:24 675 STEMming the flow: content delivery through digital media for enhancing students’ construction of knowledge Elaine Huber Institute for Interactive Media & Learning University of Technology Sydney It may be that the greatest gain in aggregate student learning in STEM is achieved not through the adoption of optimal teaching practices in each classroom but through the elimination of the worst practices (Fairweather, 2008, p. 8). Keywords: digital media, content delivery, STEM Introduction STEM disciplines rely heavily on the delivery of facts, formulas, concepts and definitions to lay the groundwork for later application. With the move to blended and flipped learning, teachers are being encouraged to replace the traditional face-to-face lecture with video recordings of mini lectures placed online. Students are then expected to engage with this digital media as pre-work before coming to class to actively engage with the content through collaborative activities. This discussion paper examines the modality of content delivery in the STEM subjects and its effectiveness. During the session, participants will be guided to share ideas and practices through an online collaborative medium. The shared resources bank can then be accessed and used for curriculum design and development activities. Background There is widespread acknowledgement that active engagement with subject content leads to better student learning outcomes (Biggs & Tang, 2013; Kuh, Kinzie, Buckley, Bridges & Kayek, 2007; Prince, 2004). The idea of flipping the classroom is growing in popularity as curricula are redesigned to remove the didactic lecture (which may encourage passivity) and replace it with a blend of online preparatory materials followed by active, collaborative, face-to-face learning activities. However, not everyone agrees that lectures should be replaced or that they are passive instruments. For example, Richardson (2008) suggests a rethink of lecture delivery, breaking it up into smaller chunks (creating mini-lectures) interspersed with student centred activities designed to engage the students with concepts essential for understanding and knowledge creation. Nonetheless we return to the basic premise of delivering content and how students engage with it. So what makes for an effective learning experience in terms of the content delivery? The push to blend and flip has resulted in many institutions experimenting with a variety of content delivery modes. The availability of flexible multimodal resources is a requirement of today’s student. It’s impossible to classify all students neatly into lecture attendees or non-attendees (Gysbers, Johnston, Hancock & Denyer, 2011) and the reasons for an individual’s choice are numerous and beyond the scope of this discussion. Many students prefer the freedom and accessibility of online lectures whereas others have been known to complain when content is placed online because they feel they are being deprived. They want to hear their lecturer - the expert - provide the information directly to them (Gysbers et al., 2011). The creation of video lectures, either recording the original lecture or recording mini-lectures at the content expert’s desk, in front of a webcam or in an empty room in front of a camera, has been a response to this. But the question remains; is this delivery any better (or worse) than the face-to-face (f2f) equivalent in terms of effectiveness? Discipline based design Pedagogical strategies most effective in enhancing student learning outcomes are not discipline dependent (Pascarllea & Terenzini, 2005). However certain disciplines such as those included in the term STEM (Science, Technology, Engineering and Mathematics) can be heavily content driven, DP:25 676 particularly in the early stages. There is an abundance of terminology of anatomical structures, ecosystems, information systems and formulae that define actions, processes, laws and the like. Such content driven learning tends to be located in the lower order objectives categorised in Bloom’s (1956) cognitive domain such as remembering, and understanding. Deeper learning takes place at the higher order objectives such as analysing, evaluating and creating, but the design of these higher order learning experiences is dependent on students having attained the prerequisite knowledge and skills (Orlich, Harder, Callahan, Trevisan, & Brown, 2007). A review of the recent literature on STEM educational research uncovers papers and studies predominantly within two areas. The need for directed attention to STEM due to a lack of future graduates (See for example Alkhasawneh & Hargraves, 2014; Williams, Kaui & Ernst, 2015) and retention studies to find out what support mechanisms can be employed within these subjects (Jackson, Charleston & Gilbert, 2014). There is little extant literature on the practicalities of how to deliver the content through effective and engaging methods. One study of engineering students looked at the sequence of delivering content. In most scenarios, content is delivered first then applied to simulation exercises. In this study, the authors observed classes where the reverse was true (simulation followed by delivery of concepts) and found more engagement and deeper level of understanding of the content (Bowen & Deluca, 2015). As a result of student led enquiry (or student-centred learning) rather than teacher driven provision of content, the students are forced to investigate, observe, challenge and question before the concepts are given to reinforce the observation. Transformational change in learning and teaching Scientists have been described as ‘naturally sceptical’ though presenting them with research and data on student learning alone is seldom compelling enough to change their pedagogy (Wieman, Perkins & Gilbert, 2010). Studies have clearly shown that in courses taught with a traditional lecture format students are not assimilating many of the fundamental concepts of physics (Yoder & Cook, 2014). But since improving STEM undergraduate education is the aim, how does one entice STEM faculty to change? “Additional research evidence will play only a small role in this process” (Fairweather, 2008, p. 13). It is postulated that “a combination of easy to use adoption tools and a broad spectrum of adoption choices combined with convincing research” (Golter et al., 2012, p. 53) might be the answer. There are many supplementary resources available for students including the traditional textbooks, lecturer’s notes, lecturer’s verbal descriptions, images, and 3D models. More recently, simulations, eBooks, animations and other open educational resources are easy to find on the Internet and through university libraries. But still we return to the question of how we can use such digital media to enhance the student experience. To provide a way for students to interact with the expert; to feel the passion and depth of knowledge a f2f experience can provide. Is a video interspersed with quiz questions to test understanding of concepts the answer? Indeed, many lecturers are converting their (traditional) f2f lectures into recorded videos of various modality which can be placed in an online environment for ease of access by students. These same students may be expected to engage with these different forms of content before attending a f2f class where they will then use it for more active participation. This classic blend of online and f2f content is very good in theory but is it effective for students to master the content knowledge? In conclusion, we do need to provide content for today’s students in more contemporary formats but there are still many unanswered questions. What form does such digital media need to take, to be most effective? Who will create these artefacts? And will students engage with them? Proposal This discussion session intends to investigate how content is being delivered in these STEM disciplines as the uptake of active, collaborative and blended learning grows. By collecting a range of examples of good practice, a picture may emerge. Furthermore, how can digital media be used to deliver concepts in more effective and engaging ways? DP:26 677 Components Science Technology Engineering Mathematics Video Video followed by quizzes eBooks with embedded quizzes Interactive online modules Table 1. Summary of practical methods for delivering content A pre-conference Google document will be created and shared to conference participants through social media channels. Conference participants will be invited to collaborate and develop the number of examples during the conference, based on other presentations and sessions. In the allocated session time, questions can be raised, and descriptions clarified. Table 1 is an example of what this may look like, populated with a few scenarios. In this way a practical toolkit of options can be made available for educational designers, developers and teaching academics to take back to their institutions for use and dissemination. These, coupled with a growing body of evidence may provide strong grounds to convince academics in the STEM disciplines to revisit the way they deliver content in their subjects. References Alkhasawneh, R. & Hargraves, R.H. (2014). Developing a hybrid model to predict student first year retention in stem disciplines using machine learning techniques. Journal of STEM Education 15(3) 35-42. Biggs, J. & Tang, C. (2007). Teaching for Quality Learning at University (3rd ed.). Berkshire, England: Open University Press. Bloom, Benjamin S. (1956). Taxonomy of Educational Objectives. Allyn and Bacon, Boston, MA. Bowen, B. & Deluca, W. (2015). Comparing traditional versus alternative sequencing of instruction when using simulation modelling. Journal of STEM Education 16(1) 5-10. Fairweather, J. (2008). Linking evidence and promising practices in science, technology, engineering, and mathematics (stem) undergraduate education. A Status Report for The National Academies National Research Council Board of Science Education. http://sites.nationalacademies.org/cs/groups/dbassesite/documents/webpage/dbasse_072637.pdf. Golter, P.B.,Thiessen, D.B., Van Wie, B.J. & Brown, G.R. (2012). Adoption of a non-lecture pedagogy in chemical engineering: insights gained from observing an adopter. Journal of STEM Education 13(5) 52-61. Gysbers, V., Johnston, J., Hancock, D. & Denyer, G. (2011). why do students still bother coming to lectures, when everything is available online? International Journal of Innovation in Science and Mathematics Education, 19(2), 20-36. Kuh, G., Kinzie, J., Buckley, J., Bridges, B. & Kayek, J. (2007). Piecing together the student success puzzle: Research, propositions, and recommendations. Washington, D.C.: Association for the Study of Higher Education. Journal of STEM Education 15(1), 11-19. Jackson, J., Charleston, L.V. & Gilbert, J.E. (2014). The Use of Regional Data Collection to Inform University Led Initiatives: The Case of a STEM Education SWOT Analysis. Orlich, D., Harder, R., Callahan, R., Trevisan, M. & Brown, A. (2004). Teaching strategies: a guide to effective instruction (7th ed.). Houghton Mifflin. Pascarella, E.T. & Terenzini, P. T. (2005). How College Affects Students: A Third Decade of Research, Volume 2. Jossey-Bass Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education 93(3), 223-231. http://www.rlillo.educsalud.cl/Capac_Docente_BecadosAPS/Metodologias/Aprendizaje%20Activo %20Prince_2004.pdf Richardson, D. (2008). Don't dump the didactic lecture: fix it. Advances in Physiology Education 32(1), 23-24 DOI: 10.1152/advan.00048.2007 Wieman, C., Perkins, K. & Gilbert, S. (2010). Transforming science education at large research universities: a case study in progress. Change: The Magazine of Higher Learning, 42(2), 6-1. DP:27 678 Williams, T., Kaui,T. & Ernst, J. (2015). Special populations at-risk for dropping out of school: a discipline-based analysis of stem educators. Journal of STEM Education 16(1) 41-45. http://jstem.org/index.php?journal=JSTEM&page=article&op=view&path%5B%5D=1921&path%5B %5D=1643 Yoder, G. & Cook, J. (2014). Rapid conversion of traditional introductory physics sequences to an activity-based format. Journal of STEM Education 15(2) 16-23. Huber, E. (2015). STEMming the flow: content delivery through digital media for enhancing students’ construction of knowledge. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. DP:25- DP:28). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. DP:28 679 The “I”s have it: Development of a framework for implementing Learning Analytics Hazel Jones Australian Digital Futures Institute University of Southern Queensland Learning Analytics are increasingly becoming commonplace in tertiary institutions and there are many frameworks and implementation strategies that have been developed to assist institutions in effective take up. Most of these are aimed at an institutional level and at strategic development, often with a key aim of improving student retention. This paper briefly discusses and compares these frameworks and introduces an alternative, complementary framework that is aimed at a practical level of implementation for groups or teams, be this a discipline group or a project team. The framework is built on 6 “I”s – impetus, input, interrogation, intervention and impact, all within an institutional context, Keywords: Learning Analytics, implementation framework, higher education, implementation model Introduction Higher education has changed significantly in recent decades with the increasing digitisation of administration, learning and teaching. The field of Learning Analytics (LA) has come to prominence over the past five years through its proposal that the data produced by higher education can be used much more effectively to improve higher education than it is currently the case (Siemens, Dawson & Lynch, 2013). Through efficient use of data it is claimed that universities can “improve teaching, learning, organizational efficiency, and decision making and, as a consequence, it can serve as a foundation for systemic change” (Long & Siemens, 2011, p32). The most recent Horizon Report for Australian Tertiary Education lists a “growing focus on measuring learning” as one of the top three trends, and LA on the one-year or less time to adoption horizon (Johnson, Adams-Becker, & Hall, 2015). With the increasing adoption of LA, many frameworks and implementation strategies have been developed to assist institutions in effective take up. Most of these are aimed at an institutional level and at strategic development, with a theoretical or conceptual approach and often with a key aim of improving student retention. There are a number of frameworks at the general level and at the institutional level but a dearth at the level of departments. This paper aims to provoke discussion on how a shift can be made from a “top-down” institutional approach to a more distributed approach that empowers staff and encourages collaboration and sharing of practice. Literature Review Current literature reveals a variety of representations for models of LA implementation with cyclical, linear or combination models being used to discuss approaches that generally have either an outcomes or process focus. The work of a current OLT Project led by Shane Dawson (2015) has identified two distinct clusters and approaches for implementation of LA in Australian universities. The first group which is outcomes focussed, gives emphasis to retention outcomes and cost savings and budgetary concerns and has limited reference to LA as a means to improve learning; whilst the second group is process focussed, with a “broader view of learning analytics and its application into learning and teaching practice.” (p 25). The project considers the benefits and limitations of both approaches and concludes that a combined model beginning with small projects that can demonstrate impact from pedagogical and technological viewpoints may be the most effective approach. The process focussed approach allows for LA to realise its potential for evidence driven change in all levels of higher education and is the type of approach adopted in several other frameworks (eg DP:29 680 Campbell & Oblinger, 2007; Clow, 2012; Dron & Anderson, 2009; Elias, 2011; Siemens, 2013). The outcomes focussed view is typified by projects such as the OLT project titled Let’s Talk Learning Analytics and Student Retention investigated institutional level implementation of LA, from the perspective of improving student retention. The project team has developed a framework that “supports systematic discussion and reflection around the use of learning analytics for retention purposes.” (West et al., 2015, p 1). A cyclical process is portrayed in several of the process focussed frameworks and models (e.g. Clow, 2012; Siemens, 2013) whilst some frameworks with either a process or outcomes focus have adopted a more linear approach with defined steps or stages (e.g. Campbell & Oblinger, 2007; West et al., 2015). Some process focussed frameworks have combined cycles and stages (e.g. Dron & Anderson, 2009; Elias, 2011). Whilst some of the process focussed models only include processes (Campbell & Oblinger, 2007; Dron & Anderson, 2009; Siemens, 2013); others include additional components such as people, including learners (Clow, 2012; Elias 2011) and resources such as organization, theory, computers (Elias , 2011). There are also a smaller number of frameworks that adopt different approaches, referring neither to processes nor outcomes. The importance of different dimensions is discussed by Greller & Drachsler (2012) who outline a framework consisting of six critical dimensions: stakeholders, objectives, data, instruments, external limitations and internal limitations that can be implemented at multiple levels within an institution. Another alternative approach is offered by the IRAC framework (Information, Representation, Affordances and Change) developed by Jones, Beer & Clark (2013) which aims for user-centred design of analytics. This user-centred approach was further developed by Beer, Tickner & Jones (2014) who discuss three paths for institutional implementation of LA of doing it to, for and with teachers, and suggest a balance of the three is the most effective approach. What is common across the frameworks and models is the importance of action or intervention as part of a closing the loop process and situating all projects within the specific institutional context. The “I” framework Most of the above frameworks are theoretical or conceptual in nature with a focus on institutional level implementation and a description of the “what” of the LA process. They have also been written from a LA perspective by authors who have expertise and experience in this field. What is missing from these frameworks is a practical implementation strategy – the “how” of the LA process – and a discussion of questions that it would be beneficial to discuss at the school/discipline or team level. With this in mind the “I” framework is being suggested as an additional and alternative framework which builds on aspects of the above frameworks and situates the cyclical process of LA within the specific institutional context. Focusing on questions that can be discussed will encourage constructive conversations and help staff to focus on working together to ensure efficient and effective implementation. Institutional context: includes the policies and strategic directions that have been set for implementation of LA. This context also incorporates the support structures, including technologies and /or data warehouses. Although discipline groups or project teams will rarely have the opportunity to have any input into this, they do though need to be aware of these and situate their implementation within these contexts. When a team has resolved to implement LA and gained knowledge of the institutional context they can follow through the implementation framework by considering the following questions and taking appropriate actions: Impetus: who will be driving the implementation and what are the specific questions to be addressed, for example is this related to student retention, student engagement with learning content or how are students performing on a particular quiz? From this an implementation plan can be developed that will address specific actions, timeframes and responsibilities. The implementation plan would also consider who and /or what will be influenced by this – will it be students (to become more responsible for their own learning) and or staff – to encourage interest in data and use of the data for positive change Input: what data is available to address the question, who has access to this information and how do staff access this in a format that is easily analysed? Interrogation: how is the data going to be analysed and interpreted and who will be responsible for this. Who will be provided with the results of the interrogation? DP:30 681 Intervention: What actions are planned as a result of the interrogation and who will be responsible for taking those actions? Impact: How successful was the process of implementation and what was the impact of interventions? Depending on the results of this the process could be repeated, using similar impetus or a deeper level of investigation. Whilst the framework is generally unidirectional it can be an iterative process returning to any of the early phase as reflection occurs within the impact phase. Discussion questions As part of the discussion for this paper the following questions will be raised to stimulate conversation:  Is this something that non LA experts will be able to utilise? For example, is the terminology non-technical and is the framework compatible with current educational research?  Is there sufficient or too much emphasis on all aspects of the framework? Some earlier models (eg Siemens, 2013) place a high emphasis on data including collection and acquisition, storage and cleansing, all of which would generally be an institutional level responsibility and hence included in the overarching institutional context for this framework. Whilst it is not intended that each stage of the “I” framework will have equal weighting, there does need to be a balance between the stages and that implementation will flow continually through the stages.  Are there any dimensions missing from the framework? By building on a wide range of frameworks and models it is hoped that the “I” framework does encompass all the essential stages of LA implementation in a logical progression. Visualisations of the framework will also be presented to elicit feedback on which best conveys the intent of the framework. As LA is a new field, information dissemination is growing exponentially and a limitation of this research may be that some new frameworks and models have been published since the submission of this paper, so it would be beneficial to know if there are other models or frameworks that have already been developed that have not been considered in this paper. This preliminary description of the “I” framework has been developed to offer an implementation strategy for LA that can be readily adopted by small teams such as discipline groups or project teams. It is planned to introduce the framework to participants in a forthcoming PhD research project and to seek feedback on the suitability of the framework for specific contexts. References Beer, C., Tickner, R., & Jones, D. (2014). Three paths for learning analytics and beyond: moving from rhetoric to reality. In B. Hegarty, J. McDonald, & S.-K. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014 pp. 242-250. Retrieved from http://ascilite.org/conferences/dunedin2014/files/fullpapers/185-Beer.pdf Campbell, J.P. & Oblinger, D.G. (2007). Academic Analytics. EDUCAUSE Quarterly. October (2007) Retrieved July 1, 2015 from https://net.educause.edu/ir/library/pdf/PUB6101.pdf Clow, D. (2012). The learning analytics cycle: closing the loop effectively. In LAK '12 Proceedings of the 2nd International Conference on Learning Analytics and Knowledge 134-138 doi:10.1145/2330601.2330636 Dawson, S. (2015). A tale of two constructs: Understanding learning analytics adoption.[PowerPoint slides] Retrieved July 1, 2015 from http://www.slideshare.net/shaned07/june-presentations- orgadoptionlearninganalytics Dron, J. & Anderson, T. (2009). On the design of collective applications. Proceedings of the 2009 International Conference on Computational Science and Engineering, Volume 04, 368-374. Retrieved July 11, 2015 from http://ieeexplore.ieee.org.ezproxy.usq.edu.au/stamp/stamp.jsp?tp=&arnumber=5284087 Elias, T. (2011). Learning analytics: Definitions, processes and potentials. Retrieved July 11, 2015 from http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf Greller, W., & Drachsler, H. (2012). Translating Learning into Numbers: A Generic Framework for Learning Analytics. Educational Technology & Society, 15 (3), 42–57. Retrieved from http://www.jstor.org/stable/jeductechsoci.15.3.42 DP:31 682 Johnson, L., Adams Becker, S., & Hall, C. (2015). 2015 NMC Technology Outlook for Australian Tertiary Education: A Horizon Project Regional Report. Austin, Texas: The New Media Consortium. Jones, D., Beer, C., & Clark, D. (2013). The IRAC framework: Locating the performance zone for learning analytics. In H. Carter, M. Gosper and J. Hedberg (Eds.), Electric Dreams. Proceedings ascilite 2013 Sydney. 446-450. Retrieved from http://www.ascilite.org.au/conferences/sydney13/program/papers/Jones.pdf Long, P. & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. Educause Review. 46(5) 31-40. Retrieved from https://net.educause.edu/ir/library/pdf/ERM1151.pdf Macfadyen, L.P. & Dawson, S. (2012). Numbers are not enough. Why e-Learning Analytics failed to inform an institutional strategic plan. Educational Technology & Society 15(3), 149-163. Retrieved from http://www.ifets.info/journals/15_3/11.pdf Siemens, G. (2013). Learning Analytics: The Emergence of a Discipline. American Behavioral Scientist, 57(10), 1371–1379. doi:10.1177/0002764213498851 Siemens, G., Dawson, S., & Lynch, G. (2013). Improving the Quality and Productivity of the Higher Education Sector: Policy and Strategy for Systems-Level Deployment of Learning Analytics. Society for Learning Analytics Research. Retrieved from http://www.olt.gov.au/resource-improving- quality-and-productivity-higher-education-sector-2013 West, D., Huijser, H., Lizzio, A. Miles, C., Toohey, D., Heath, D., …Searle, B. (2015). Let’s Talk Learning Analytics and Student Retention. The National Forum 9th April 2014, Griffith University, Brisbane. Jones, H. (2015). The “I”s have it: Development of a framework for implementing Learning Analytics. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. DP:29-DP:32). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. DP:32 683 Learning analytics - are we at risk of missing the point? Danny Yen-Ting Liu Tim Rogers Abelardo Pardo Macquarie University University of South Australia The University of Sydney Australia Australia The rise of learning analytics in the last few years has seen fervent development from institutions, researchers, and vendors. However, it seems to have had a laggard reception in higher education. Peering behind some barriers to adoption, we question whether common approaches that address the economics of low hanging fruit distract us from asking and answering deeper questions about student learning. This may lead to destructive feedback loops where learning analytics, swept by the currents of institutional agendas and cultures, does not deliver upon its promises to those who need it most - students and educators. Keywords: learning analytics, predictive analytics, retention, barriers to adoption. Introduction The education sector is currently undergoing a paradigm shift towards the collection and analysis of detailed datasets about user interactions with resources and other users (Long & Siemens, 2011). The field of learning analytics (LA) has recently emerged with the objective of measuring, collecting, analysing, and reporting data about learners and their contexts, for the purpose of understanding and optimizing learning and the environments in which it occurs (Siemens, Dawson, & Lynch, 2013). Initially, identifying students at risk of abandoning a course or the institution was an obvious target for the field, where data could have direct economic impact (De-La-Fuente-Valentín, Pardo, & Kloos, 2013). The rationale was that through the use of their data, ‘at risk’ students could be identified early and, ideally, the right set of actions could be taken to avoid attrition. After this initial problem, experts defined the ideal trajectories to be followed by educational institutions to embrace the use of data. Corresponding maturity models for LA have been framed around progression from reporting to predictive modelling to automatic triggering of business processes (Goldstein & Katz, 2005), or from reports on single systems to organisational and sector transformation through sharing of innovations (Siemens et al., 2013), or from static reporting to a pervasive culture of data-driven optimisation (Norris & Baer, 2013). Despite lofty ambitions, adoption of LA has been slower than anticipated with most US institutions only using data for simple reporting purposes (Bichsel, 2012), and the situation in Australia is by no means more advanced (Colvin et al., 2015). Why is LA yet to be fully realised in learning and teaching? Effective LA is predicated on having the right tools, processes, and people to understand and optimise learning (Bichsel, 2012), and it appears that to date, developments in LA are primarily driven by diverse (and sometimes esoteric) research agendas, bespoke institutional hackery to achieve quick retention wins, or vendors designing business intelligence-like dashboards replete with bar graphs and colour-coded text – indeed, the authors and their institutions are perpetrators of this. Who of these players, if any, is missing the point of LA? In this discussion paper, we pose intentionally provocative conjectures regarding the relatively laggard uptake of LA around its development and validity, competing institutional agendas and cultures, and the needs of educators and students. Three conjectures for debate Muddied waters: the seduction of predictive analytics and retention rates The preponderance of LA projects in the short history of the field have focussed on predicting student underperformance or dropout (Siemens et al., 2013). Purdue Course Signals (Arnold & Pistilli, 2012), the traffic light early warning system for identifying students at risk, is a poster child for this element of LA. The rash of projects that arose in its wake have followed somewhat of a formula: correlate independent variables (e.g. demographics, past performance, LMS usage) with an outcome variable (e.g. student performance or attrition), and apply resultant models to new data to identify students at risk and recommend resources and/or services. For many in higher education this has become the DP:33 684 sine qua non of LA, and a recent comprehensive overview of the Australian LA landscape has found that many institutions mistakenly regard this form of predictive analytics as coextensive with LA as a whole (Colvin et al., 2015). However, it is difficult to argue against the institutional agendas served by this form of LA. Student attrition is a pain point for many universities and the widening of participation coupled with massification of higher education means it is now much more difficult to rely on traditional processes to identify students at risk, right at the juncture where there are likely to be more of them. Or, it may be that this is simply a stage of LA maturity that must be passed through: the identification of risk is low hanging fruit, especially in comparison with the more complex and difficult longitudinal questions that would need to be addressed for LA to cast light on learning processes (Gašević, Dawson, & Siemens, 2015). There has certainly been uptake of this kind of LA. However, the algorithmic identification of students at risk implies that there is something wrong with the student, ostensibly reverting to a ‘theory 1’ perspective (Ramsden, 1993) of education. When algorithms are black boxes, this prevents academics from identifying teaching or curriculum issues that may be at play. Perhaps more problematic is the increasingly common business model to outsource at-risk student contact to internal or external outreach services, which means that the feedback to academics is usually very superficial and the feedback to students is disconnected from their learning. Overall, these processes typically distance educators from students and alienate educators from their practice. Whatever its benefits to the institution, these interventions are not about learning per se, but the outcomes of the assessment and engagement activities. Very little in LA has so far penetrated the relevant learning processes (Gašević et al., 2015), yet this is precisely where educators can act to make changes. Rather, predictive analytics is more about the identification of antecedent conditions in order to control behaviour (e.g. log in more, post more comments). It should not be a surprise that some see this as the advent of a new behaviourism (Lodge & Lewis, 2012; Dietrichson, 2013). Killing the pipeline: too much promise, not enough delivery LA, to some extent, is analogous to the uptake of a new product or service and will follow some of the same principles. Studies of the uptake of organisational innovations sometimes present this as a ‘pipeline’ problem (Warren, 2008), in this case moving educators through specific states: from unaware of LA, to aware but not yet interested, to interested, to using, and finally integrating. At each stage, the hope is to ‘pump’ academics from one state to another. However, these flows may not go in the intended direction. At each flow point, initiatives are needed which are specific to the state educators are in. The ‘marketing’ of LA has been largely successful judging by the number of educators who have moved from unaware to aware (and some even interested) during the short history of the field. But the number of useful tools and products is low, potentially leaving interested educators unable to progress and therefore vulnerable to flowing back to the ‘aware but not yet interested’ pool. One issue contributing to the developmental lag of LA is the propensity for universities to either develop their own tools individually or buy them from vendors. Evidence for this binary approach is implicit in a recent overview of LA in Australia (Colvin et al., 2015). What was not mentioned in this report, but known to the investigators ([author 2] being one of them), was the number of similar technical initiatives that were replicated at each university, and the low levels of knowledge sharing. By way of example, one university had developed competence in extracting a range of data from Moodle and other teaching sources while another university had a well-developed ‘business’ model for presenting Moodle data to educators in a way that could be harnessed for student intervention. Both universities could have benefitted from a close collaboration, and neither were in the same State (so could not be considered ‘competitors’), yet even when aware of each other’s initiatives they proceeded along the slow and expensive path of developing their missing piece of the picture from scratch. This low level of sector knowledge sharing slows down the development of tools, leaving academics who are interested in LA either unable to secure the right tool for their interest, or only able to access tools that are barely adequate and locked in a slow development cycle. The result may be frustration and flowing backwards rather than forwards on the adoption pipeline. What do students and educators want? Although educators and students receive the most benefit from LA (Drachsler & Greller, 2012), very little work has been done to ask them what they really want or need. Yet, simply asking is no panacea DP:34 685 for uptake as they may be labouring under false understandings and expectations, and their conceptions may be affected by unconscious processes that distort rather than clarify. For example, a recent exercise carried out by JISC featured naive (with respect to LA) students designing ‘app-like’ interfaces for the kind of data they would like to see (Sclater, 2015). While probably a necessary exercise, it is not sufficient to inform the choice of data and its method of presentation. Several of the example screenshots show feedback suggestions that indicate students may be influenced by a performative rather than a mastery learning orientation (presumably unconsciously), and unhelpfully focus on outcomes rather than processes (Hattie & Timperley, 2007), potentially demotivating students with low self-efficacy while encouraging complacency in higher achievers (Corrin & de Barba, 2014). Conclusions and provocations for the future So is LA, as currently being courted by institutions, missing the point? The rise and rise of predictive analytics has certainly piqued interest in the field, but has also crowded out the development of tools focussed on the process of learning. In this context, idealised models of institutional LA maturity (e.g. Norris & Baer, 2013) usually do not help because although they may mention personalised learning, they are predicated on increasing levels of predictive utility. The issues that LA would need to address if it were to focus on learning would be predicated instead on explanatory adequacy, which may or may not increase predictive accuracy (Bhaskar, 1975). The question should be why did a learning event or program succeed or not, rather than which students did or did not succeed. While these questions are not necessarily mutually exclusive, the predictive analytics paradigm offers little help with the former (Rogers, Dawson, & Gašević, forthcoming). In order for LA to better address the deeper questions, universities and vendors need to work out mutual relationships that can springboard LA development (Siemens, 2012). We need to avoid a destructive feedback loop where a lack of innovative tools hinders uptake, the lack of uptake signals cautious investment in tools, leading to slow tool development, contributing to a lack of innovative tools… These phenomena are well- known in business (Sterman, 2000) and there is no reason to think higher education would be immune. Finally, as LA sits at the intersection of theory and practice, it needs to pay heed to its users as well as better integrate with significant learning theories to isolate the data and feedback that will truly be valuable to educators and students. References Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: using learning analytics to increase student success. Paper presented at the 2nd International Conference on Learning Analytics and Knowledge. Bhaskar, R. (1975). A realist theory of science: Routledge. Bichsel, J. (2012). Analytics in higher education: Benefits, barriers, progress, and recommendations: EDUCAUSE Center for Applied Research. Colvin, C., Rogers, T., Wade, A., Dawson, S., Gašević, D., Buckingham Shum, S., et al. (2015). Student retention and learning analytics: a snapshot of Australian practices and a framework for advancement (draft report). Sydney, Australia: Australian Office for Learning and Teaching. Corrin, L., & de Barba, P. (2014). Exploring students’ interpretation of feedback delivered through learning analytics dashboards. Paper presented at the Proceedings of the ascilite 2014 conference. De-La-Fuente-Valentín, L., Pardo, A., & Kloos, C. D. (2013). Addressing drop-out and sustained effort issues with large practical groups using an automated delivery and assessment system. Computers & Education, 61, 33-42. Dietrichson, A. (2013). Beyond clickometry: Analytics for constructivist pedagogies. International Journal on E-Learning, 12(4), 333-351. Drachsler, H., & Greller, W. (2012). Confidence in learning analytics. Paper presented at the LAK12: 2nd International Conference on Learning Analytics & Knowledge. Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64-71. Goldstein, P. J., & Katz, R. N. (2005). Academic analytics: The uses of management information and technology in higher education: EDUCAUSE Centre for Analysis and Research. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112. DP:35 686 Lodge, J., & Lewis, M. (2012). Pigeon pecks and mouse clicks: Putting the learning back into learning analytics. Future challenges, sustainable futures. Proceedings ascilite Wellington, 560-564. Long, P., & Siemens, G. (2011). Penetrating the Fog: Analytics in Learning and Education. EDUCAUSE review, 46(5), 30. Norris, D. M., & Baer, L. L. (2013). Building organizational capacity for analytics. Retrieved from http://www.educause.edu/library/resources/building-organizationalcapacity-analytics Ramsden, P. (1993). Theories of Learning and Teaching and the Practice of Excellence in Higher Education. Higher Education Research & Development, 12(1), 87-97. Rogers, T., Dawson, S., & Gašević, D. (forthcoming). Do Counts of Digital Traces Count for Learning: The Imperative for Theory in Learning Analytics. In C. Haythornthwaite, R. Andrews, J. Fransman & M. Kazmer (Eds.), Sage Handbook of E-Learning Research. London: Sage Publications. Sclater, N. (2015). What do students want from a learning analytics app? Retrieved from http://analytics.jiscinvolve.org/wp/2015/04/29/what-do-students-want-from-a-learning-analytics- app/ Siemens, G. (2012). Learning analytics: envisioning a research discipline and a domain of practice. Paper presented at the Proceedings of the 2nd International Conference on Learning Analytics and Knowledge. Siemens, G., Dawson, S., & Lynch, G. (2013). Improving the quality and productivity of the higher education sector: Society for Learning Analytics Research. Sterman, J. D. (2000). Business dynamics: systems thinking and modeling for a complex world (Vol. 19): Irwin/McGraw-Hill Boston. Warren, K. (2008). Strategic management dynamics: John Wiley & Sons. Liu, D.Y-T., Rogers, T., & Pardo, A. (2015). Learning analytics - are we at risk of missing the point? In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. DP:33-DP:36). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. DP:36 687 The impact of digital technology on postgraduate supervision Dorit Maor Murdoch University There is a need to improve supervision of higher degree students to increase completion rates, reduce attrition and improve quality. This discussion paper explores the contribution that technology can make to higher degree research supervision. It focuses on research studies that support supervision through the application of digital technology. In reviewing current research, I discuss whether web-based tools can influence the training of Higher Degree Research (HDR) students, are effective in supporting students, and can reduce breakdowns in supervisory relationships. A major trend in higher education is the re-purposing of Web 2.0 systems, not only to access knowledge collaboratively, but also to create and sustain communities of learners. In critically reviewing current research-based papers, I was able to assess the impact of web-based tools on the training and support of doctoral students. The longer-term aim of this research project is to create a digital platform that can assist postgraduate students and their supervisors. Keywords: higher degree research supervision, Web 2.0 systems, supervisors. Background to the supervision project Addressing high attrition rates among HDR students (more than 25%) and the quality of research undertaken by such students are areas of great concern to many universities (Norton, 2012). In response, supervision has moved from an individual relationship to a team approach, leading to a reduction in independent research and providing access to a range of supervisors with various forms of expertise (Green and Bowden, 2012). Redirection from a product-oriented thesis to a process- oriented one; and from a person-centred to a community-centred method has revolutionised many universities approach to providing higher degree education (Stubb et al., 2012). These modifications may instigate changes in the pedagogy of supervision, heighten critical thinking about research questions, and enhance positive relationships with supervisors (Lee, 2008). According to Hammond et al. (2010), using technology as a tool in supervision can transform the character of higher degree training and raise the research outcomes for Australian universities. Therefore, using technology should be able to boost completion rates and reduce the time taken to complete degrees (Hammond, Ryland, Tennant, & Boud 2010). One of the reasons for looking toward technology to solve Australian universities’ research problems is that digital environments are already changing the way people interact (Danby & Lee, 2012). Students, whether undergraduate or postgraduate, are demanding greater flexibility in the way that they study. They want their supervisors to be available 24/7 and to respond to their texts/emails, etc. There is also more remote and distance learning among postgraduate students, which exacerbates the need for communication that bridges the geographical gap. These burgeoning needs can be mediated by software that takes advantage of common computer literacies and is accessible regardless of the choice of device. With this background in mind, the current paper examines the literature regarding how technology can support the supervision of HDR students. The literature presented will provide a starting point for collegial discussion, which aimed at developing criteria appropriate for designing a web-based supervision platform. The overall outcome of this process is to inform a larger project to design a comprehensive communication platform for supervisors and their HDR students. Literature search To investigate the research that is currently available about how technology is used in the supervision process, empirical research articles were identified on that topic based on these methods. The DP:37 688 academic referencing system Sente 6 was used to search the following databases: ScienceDirect, Editlib, ERIC, Academic Onfile ProQuest and SAGE Journals. This yielded limited results so the keywords were expanded to include pedagogical concepts, like ‘face-to-face training’, ‘reflective practice’ and ‘distance education’, ‘doctoral student training’, ‘doctoral process’ and ‘doctoral education’. Results from this second stage search numbered in the several thousands. Then a restricted time period was used to recognize how technology changes rapidly: 2006 – 2013. The search focused on the Web browser as a participation platform for which software applications are built. From several thousand, the final set of articles comprised 196 Peer reviewed papers, 64 Conference Proceedings, 8 Dissertations, and 16 Reports. When using the filter concepts related to technology, supervision and pedagogical supervision, and supervisor–supervisee relationships, a final set of 18 papers considered most relevant to this project was selected. Findings The literature review revealed a clear shift towards participatory pedagogy with the supervision relationship becoming more reciprocal and less hierarchical (e.g., Andrew, 2012; Fenge, 2012). While research regarding HDR supervision is comprehensive, there appears to be limited investigation into the role of technology within supervisory relationships. Nevertheless, several large-scale national studies, such as the United Kingdom’s Researchers of Tomorrow (Carpenter, et al., 2010; Carpenter et al., 2011; Carpenter, 2012) and the Grattan Institute’s Mapping Australian Higher Education (Norton, 2012), concluded that in general supervisor’s competency with regard to technology was substantially lagging behind that of their students. A major impetus for using Web 2.0 technologies was to initiate doctoral students into scholarly communities where the highly interactive relationships and focused on more a participatory pedagogy (Danby, & Lee (2012). Using collaborative-based technology, such as ‘Google docs’, delivered a sense of connectedness that promoted social and academic outcomes. However, not all the studies revealed that supervision was moving towards a more participatory pedagogy (Halse (2011). In some, the relationships did not change as a result of using technology: the supervisor still maintained the role of advisor and mentor and provided support and quality control, but with the advantage of better communication (de Beer, & Mason (2009). However, in most countries there is mounting pressure to implement a more open and flexible type of supervision (Stelma (2011). The technology used in these studies varied considerably and included the following: Skype, Elluminate, Wimba, Second Life, telephone, and MSN messenger in distance education; Wikis, Microblogging, Social Bookmarking and email; ePortfolio (PebblePad) and an in-house virtual portfolio as a dialog tool; Microsoft Office 365 (Share-Point and OneNote for collaborative writing, Yammer for collaborative ideas and Lync/Skype for communication) and WebCT in more traditional supervision. Two crucial studies were extremely salient because they created completely new web 2.0 technology environments: Doctoralnet (Danby, & Lee, 2012) and Form@doct (Melingre, Serres, Sainsot, Men, 2013). These environments enabled doctoral candidates who were geographically isolated to collaborate with their supervisors and other doctoral students. The use web 2.0 technology facilitated discussions, videoconferencing, linked homepages, and collaboration in writing spaces (Danby, & Lee, 2012). In particular, Danby & Lee identify the synergy between technology and pedagogy in relation to achieving supervisory goals. It was apparent that digital tools enabled greater dialogue and interaction between the students and their supervisors (de Beer & Mason, 2009; Cumming, 2010; Carpenter, 2012). Moreover, students generally desired more web-based communication and support from their supervisors (Rockinson- Szapkiw, 2011). The digital environments created virtual spaces that combined technology and pedagogy into a process where research projects could be developed in a more collegial and collaborative way (Le, 2012). Through combining supervision pedagogy with new technologies, these research projects reflected a shift to a process of creating communities of scholars. In one study focusing on Generation Y doctoral students who used Web 2.0 technologies, most participants confirmed that their supervisors were not very interested or competent in new Web technology applications and continued to supervise in a traditional manner (Carpenter, Tanner, Smith, & Goodman, 2011). There was no strong synergy between students and supervisors in spite of the opportunities to use social learning technologies. Apparently their supervisors’ knowledge and competency in using technology for the advancement of the process was lagging behind that of their DP:38 689 students. Other pedagogies involved collaborative processes through using either ePortfolio as a resource (Le, 2012) and communication tool or collaborative Website workspace (Rockinson- Szapkiw , 2011). Conclusion This literature review identified the need for an iterative process based on scholarly conversations about supervision pedagogy. In particular, such discussions need to focus on two paradigms in order to design a comprehensive web-based supervisory communication platform. The two paradigms that emerged were: technological and pedagogical. It was apparent from the literature that there was a necessity to redefine these concepts in terms of a united digital pedagogy, because one term in isolation was not sufficient for success without the other. Social media is essential to create a sustainable community for students and their supervisors in the future. It is clear that current students are demanding greater use of technology and want their supervisors to become competent in the use of it to make their relationships more flexible and interactive. A digital pedagogy model that brings about these multidimensional changes using a social media application could help to create the next generation of supervision pedagogy. Ideally it would develop a more participatory relationship to shift supervision from an intense personal relationship to a more professional one. A technological tool to assist in implementing this vision is only a first step in providing the foundation for a sustainable bridge between technology and supervision pedagogy. Critically, further empirical studies are needed. Such research, in focussing on the multidimensionality of contemporary supervision, is likely to contribute to the recognition of doctoral supervision as a field of scholarly work. References Andrew, M. (2012). Supervising doctorates at a distance: Three trans-Tasman stories. Quality Assurance in Education, 20(1), 42-53. Carpenter, J. (2012). Researchers of tomorrow: The research behaviour of generation Y doctoral students. Information Services and Use, 32(1), 3-17. Carpenter, J., Tanner, S., Smith, N., and Goodman, M. (2011). Researchers of tomorrow: A three year (BL/JISC) study tracking the research behaviour of ‘generation Y’ doctoral students: Second annual report 2010–2011. London: Education for Change. Carpenter, J., Wetheridge, L., Smith, N., Goodman, M., and Struijvé, O. (2010). Researchers of tomorrow: A three year (BL/JISC) study tracking the research behaviour of ’generation Y’ doctoral students: Annual report 2009–2010. London: Education for Change. Cumming, J. (2010). Doctoral enterprise: A holistic conception of evolving practices and arrangements. Studies in Higher Education, 35(1), 25-39. de Beer, M., and Mason, R.B. (2009). Using a blended approach to facilitate postgraduate supervision. Innovations in Education and Teaching International, 46(2), 213-226. Danby, S., & Lee, A. (2012). Researching doctoral pedagogy close up: Design and action in two doctoral programmes. Australian Universities Review, 54(1), 19-28. Fenge, L.-A. (2012). Enhancing the doctoral journey: The role of group supervision in supporting collaborative learning and creativity. Studies in Higher Education, 37(4), 401-414. Green, P., and Bowden, J. (2012). Completion mindsets and contexts in doctoral supervision. Quality Assurance in Education, 20(1), 66-80. Halse, C. (2011). ‘Becoming a supervisor’: The impact of doctoral supervision on supervisors’ learning. Studies in Higher Education, 36(5), 557-570. Hammond, J., Ryland, K., Tennant, M., and Boud, D. (2010). Building research supervision and training across Australian universities. Strawberry Hills, NSW: Australian Learning and Teaching Council. Le, Q. (2012). E-Portfolio for enhancing graduate research supervision. Quality Assurance in Education, 20(1), 54-65. Lee, A. (2008). How are doctoral students supervised? Concepts of doctoral research supervision. Studies in Higher Education, 33(3), 267-281.doi:10.1080/03075070802049202. Malingre, L., Serres, A., Sainsot, A., & Men, L. (2013). Form@doct: Designing innovative online tutorials for PhD students in France. IFLA Journal, 39(1), 45-57. DP:39 690 Norton, A. (2012). Mapping Australian higher education. Melbourne: Grattan Institute. Stubb, J., Pyhältö, K., and Lonka, K. (2012). Conceptions of research: The doctoral student experience in three domains. Studies in Higher Education, 1-14. Stelma, J. (2011). An ecological model of developing researcher competence: The case of software technology in doctoral research. Instructional Science, 39(3), 367-385. Rockinson-Szapkiw, A. (2011). Improving doctoral candidates persistence in the online dissertation process. In proceedings of Global Learn Asia Pacific, 1162-1166. Maor, D. (2015). The impact of digital technology on postgraduate supervision. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. DP:37-DP:40). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. DP:40 691 Is Student Transition to Blended Learning as easy as we think (and what do they think)? Carol A. Miles University of Newcastle This is about the students. In the move to ‘flipped’ or blended modes of delivery, universities are spending all of their energies focusing on course design and upskilling academics, and assuming that students will easily embrace the new methodologies that are integral to blended learning approaches. We make this assumption based on the belief that they are au fait with all things technology when that may not be true. What we are doing is radically changing what they are experiencing as learning delivery methods, compared to what they had expected. Through implementation of these new blended learning delivery models, we have fundamentally changed what they are expected to do as students. We do this without sufficient warning and support mechanisms for this radical new way of learning. We must engage the students in this discussion and really LISTEN to what they want and need. We must conduct robust research that will inform our course design and teaching practices, our student advising and support, and we must begin now. Keywords: Blended Learning, Flipped Classroom, Student Support Introduction As teaching and learning professionals scramble to provide skills to academics teaching in flipped and blended modes, it appears that few are actively helping the students understand the changes they need to make to effectively engage with learning in this new university environment. Blended/Hybrid/Flipped learning represents the most dramatic change to university teaching that has ever occurred. As we work towards removing the lecture as the main form of content delivery, students maintain ongoing expectations for ‘traditional’ university teaching and study. There is a real paradigm shift from what students have expected when applying for a university degree, and relatively little has been done to date to prepare them for engaging successfully with these new teaching and learning models. Flipped delivery assumes that general content knowledge has been achieved by the student (commonly through viewing video clips or engaging in other online resources) prior to completion of authentic tasks in class. As these teaching strategies develop from their infancy, the quality of the online materials is in many cases less than optimal, and students are expected to learn from materials that do not replicate the same interactive quality or format as the existing lecture model. The increasing use of purposeful video or other activities designed to engage the student in the independent mastery of content is an essential part of the ‘flip’. This represents a different way of learning and organisation of study that is not only unfamiliar to students’ parents, siblings, and previous teachers, but also, frankly, to most of their university teachers who are implementing the new strategies. There is a definitive shift from students as consumers of content to creators of their own knowledge through a shift to deeper learning approaches (Johnson, Adams Becker & Hall, 2015). These developments, especially the reduction in face-to-face teaching hours, place a greater emphasis on the student as curator of their own learning and assign them greater responsibility for maintaining sufficient involvement in their courses. Students will no longer have a timetable of hours of contact that directs their mastery of core course content. For the first time they have the responsibility and the opportunity to determine their own approaches to mastery of content and concepts. While this may on the surface appear to be a positive development, it must be acknowledged that students are being required to do this with little consideration for the impact the changes will have on their workload and their approaches to learning. The previously held belief that it was the responsibility of the university to ensure that students are being provided learning opportunities is now being, to a large extent, transferred to the individuals themselves. DP:41 692 How new teaching methods affect our students Across Australia, universities are developing blended learning experiences and designing corresponding learning spaces that increasingly leverage the growing number of educational technologies available (Johnson, et al., 2015). This is in stark opposition to the traditional lecture model. These developments challenge the relevance of the traditional lecture format as the most effective model, and in fact, represent a renaissance of teaching and learning methods in the university setting. Adopting these approaches will necessitate fundamental changes to how most courses and especially assessments are designed and delivered. As the majority of universities attempt to facilitate these changes, attention is focussed on the redevelopment of courses, activities, and assessments and the re-training of teaching staff to allow for successful implementation of blended learning models. Much less attention is being given to supporting students through these changes. The expectation that students will master content through online engagement/viewing videos prior to attending classes may, for many, look no different than the common traditional expectation that they read the chapter prior to the lecture – which many students assume is unnecessary as they expect and even demand that the content be taught during the lecture. If they approach their studies with these traditional expectations and habits, it will be difficult, if not impossible, for them to succeed in the flipped environment. Student learning support will need to be offered in virtual, asynchronous environments as well as the traditional face-to-face consultative meetings (Foggett, 2015). This must address not only standard learning development topics such as writing and study skills, but also instruction in technologies that students are being asked to utilise along with new forms of time management to address self-mastery of content. This shift in study requirements will be easier for some students than others. As universities compete in the race to develop virtual and physical learning spaces that will facilitate the changes in pedagogy required to assure student success – specifically engaged, and often group activities – an assumption appears to be made that students will be naturally drawn to this form of learning. Because most of these new methods involve some form of learning technologies, there is an underlying belief that students will easily embrace the changes in study and learning habits. This has not been the case to date (Dalstrom & Bischel, 2014) with students preferring and expecting more traditional methods of course delivery. The increasing use of learning technologies will require students to radically change their methods of organising their study and general life as a student. Although technology is commonly woven into most aspects of students’ lives, students are not as adept at leveraging technology to succeed in their studies as may be assumed. Longitudinal data from past student studies shows us that most (but not all) students access a variety of technologies on a daily basis, with a division between learning technologies and technologies used for personal purposes. While they recognise the value of technology, students may still require guidance when using technology in meaningful and engaging ways for academic study (Gosper, Malfroy & McKenzie, 2013). Active provision of this support has been largely ignored in discussions and program development surrounding blended learning. Students enrolling in what they assume are ‘traditional’ university programs will not have an expectation of multiple use of educational technologies or of self-directed learning. (Calderon, Ginsberg & Ciabocchi, 2015). Many will approach university with an understanding of study requirements that have been firmly established through high school, other institutions, and from parents, siblings, and friends. Most are not aware that they will need to develop a whole new skill set that allows them to be effective students when exposed to these changes to pedagogy. The major change they will need to adopt is a far greater requirement to independently manage their own learning processes. It is important to distinguish between students faced with studying in a blended environment (which includes face-to-face learning), and those who have intentionally chosen to study online. Students enrolling in fully online courses would be expected to have some knowledge that their mode of study would be different than previous face-to-face experience. Online course information often indicates the need to study independently using technology and the course materials provided. Students discovering that they are enrolled in newly-designed blended courses often begin their studies assuming that they will be getting a traditional university education, not dissimilar to their high school DP:42 693 or other previous educational experiences and expectations. Dalstrom and Bischel (2014) reported that after sufficient exposure and experience, many students do prefer blended learning environments, and their expectations are increasing for these hybrid online/face-to-face experiences. Critically, however, many still expect (and even embrace) the face-to- face lecture model. It must also be acknowledged that the predominance of video-produced lectures presented as a standard for many blended models is relatively low tech and low-engagement. It does not emulate the engaged experience of many mobile, social media and other platforms that students have come to expect in their daily lives. The assumption that students will embrace these new learning methods because of their technology components is arguable at best. As these blended delivery models become increasingly popular universities are providing a plethora of programs to support academics in this style of teaching and course design. A broad variety of incentives and programming is provided to assure that teaching academics are redesigning their courses, as well as their teaching methods to address the requirements of the new models. Despite these efforts, one of the toughest things for our students is the skill deficit of many of their university teachers. Many of these people are using these strategies / technologies as learners themselves, with compulsion to change their teaching methods through university policy and strategy, and not necessarily through their own choice (Llamas, 2014). With students unfamiliar with these new learning methods, and course coordinators (often grudgingly) attempting to deliver courses in new ways we have set up a scenario that is tantamount to the blind leading the blind. Call to Action In order to maximise the effectiveness of the teaching approaches university communities must gather evidence that supports the continued use of these approaches. Learning analytics is now yielding critical information about students’ engagement with their course materials and activities (Miles, 2015). This involves a form of learner profiling, a process of gathering and analysing large amounts of detail about individual student interactions. The goal is to build better pedagogies, empower students to take an active part in their learning, target at-risk student populations, and assess factors affecting student success. Despite this work there seems to be a paucity of input from the students themselves. We must engage students in our decisions surrounding provision of support for them – relating to both technical expertise and study strategies. Students with different backgrounds, experiences, circumstances and learning styles will necessarily require different support mechanisms to take advantage of new approaches to teaching. We are telling students what is best for their learning when we are all in our infancy in this new blended world. There is tremendous pressure on the instructor to design engaged pre-class activities that allow students to master the content independently. We need to engage the students through action research to determine which content mastery activities actually yield the best learning results. These empirical findings will allow us to convince our academics that all of the effort put into course redesign will support student learning in our new teaching spaces – physical and virtual. Engagement with student groups on a national level, as well as careful liaison with secondary schools will be required to prepare students for this entirely new way of university study (and, consequently, career preparation). We not only need to guide our students on how to use the technologies and learning resources available, but when and why specific tools would best assist them in achieving academic success. It is time to work carefully and closely with all students and listen to them regarding how they want to construct their learning! Considerable research is required to determine the optimal institutional and course-based supports required for students embarking on a completely different university journey than has previously existed. This is a call for action to Australian universities and those around the world to partner with our students in empirical and action research to provide a solid basis for our assumptions about student learning needs. This will allow us to construct student support mechanisms that will prepare our students to embark on radically different learning journeys and do it successfully. This will provide the groundwork for course design and instruction practices for our generations of students to come. We are not there yet. We are not close. It’s time to begin. References DP:43 694 Calderon, O., Ginsberg, A., Ciabocchi, L. (2015). Multidimensional Assessment of Blended Learning: Maximizing Program Effectiveness Based on Student and Faculty Feedback. International Journal of Education and Development using Information and Communication Technology, 11(1) 80-100. Dalstrom,E., and Bichsel, J. (2014). ECAR Study of Undergraduate Students and Information Technology, Research report. Louisville, CO: ECAR, October 2014. Available from http://www.educause.edu/ecar. Foggett, K. (2015). Moving student consultations to the cloud: meeting new demand through eConsult ©. AACE EdMedia 2015, Montreal, Quebec. Published Proceedings. Gosper, M., Malfroy, J., & McKenzie, J. (2013). Students' experiences and expectations of technologies: An Australian study designed to inform planning and development decisions. Australasian Journal of Educational Technology, 29(2), 268-282. Johnson, L., Adams Becker, S., and Hall, C. (2015). 2015 NMC Technology Outlook for Australian Tertiary Education: A Horizon Project Regional Report. Austin, Texas: The New Media Consortium. Llamas, R. (2014). Worldwide Wearable Computing Device 2014–2018 Forecast and Analysis, IDC Report, March 2014. Miles (2015). Australian university teachers’ engagement with learning analytics: still early days. AACE EdMedia 2015, Montreal, Quebec. Published Proceedings. Miles, C.A. (2015). Is Student Transition to Blended Learning as easy as we think (and what do they think)? In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. DP:41-DP:44). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. DP:44 695 Learning through doing: Creating a makerspace in the academic library Karen Miller Curtin University Australia Makerspaces are becoming increasingly common in higher education institutions, and academic libraries can be regarded as an ideal location for such collaborative learning environments, as they provide a neutral space which encourages cross-disciplinary engagement and collaboration. This paper discusses the potential role of the library in facilitating the development of important lifelong learning skills through hands-on, problem-solving, and participatory making activities. It describes Curtin Library’s initial steps to establish a makerspace, which it is doing by providing a physical and virtual space, organizing events, workshops and activities and engaging in collaborative projects. Keywords: makerspaces, libraries, creativity, collaboration Introduction: What is a makerspace? Makerspaces are collaborative learning environments where people come together to share tools, materials and expertise, and develop digital literacy and other skills through hands-on ‘making’ activities. Fostering community building, they encourage collaboration, creativity, experimentation, exploration, and the sharing of knowledge and experience. They provide opportunities to engage in problem solving activities using technologies such as 3D modelling and printing, electronic circuitry, robotics, coding, data visualisation, virtual and augmented reality, video-creation, animation and digital storytelling, as well as art, craft and design. There are many kinds of makerspaces: they come in many different shapes and sizes and can take any form - big or small, portable or permanent. They are adaptable, and can be shaped to educational goals or the interests of individuals or groups. As a learning environment, a makerspace is generally underpinned by a constructivist approach, which holds that effective learning occurs by involving the learner in the primary construction of knowledge through hands-on enquiry based learning. It also applies connectivist learning theories, emphasising learning through doing in a socially networked environment and that recognizes the student as a creator of information, not just a consumer (Dunaway, 2001). These theories have informed the way maker cultures are built, incorporating multiple learning styles and allowing students to take control of their own learning, providing an intersection between formal and informal learning to include “designing, playing, tinkering, collaborating, inquiring, mentoring experimenting, problem solving and inventing” (Loertscher, 2012). By fostering learning in this way, it encourages the development of valuable, essential skills that students need to meet society’s challenges, both now and in the future. Makerspaces exist in a variety of contexts, as shared environments for ‘making’ things have essentially been around for many hundreds of years. However in the past decade or so makerspaces have been established as dedicated learning environments in community organisations (an exciting example being The Tinkering Studio at the Exploratorium); educational institutions (such as Makernow, the Digital Fabrication Lab at Falmouth University, Cornwall); and pubic libraries (of which The Edge, in Brisbane, Queensland is a successful Australian example). In the higher education sector the academic library is beginning to position itself to play a central role in this emerging trend. In the US some fine exemplars are the University of Virginia Scholars Lab and Makerspace, and the De La Mare Science and Engineering library in Nebraska. Changes in technology, education, and the needs of their communities have seen academic libraries adapt by repositioning their spaces as well as rethinking their purpose. By establishing makerspaces many academic libraries “aim to fulfill aspects of their current and emerging roles, such as fostering life-long learning, acting as catalysts for collaboration, and providing support and resources for the creation of knowledge”(Miller, 2015). Such a shift provides an opportunity for academic libraries to expand their traditional areas of teaching information literacy to experiment with different approaches, DP:45 696 thus enabling libraries to learn about new and emerging learning technologies and innovative approaches to teaching and learning. This means exploring, and being informed by, new concepts of information literacy that are redefined to empower learners. It also means being informed by changes in technology, encompassing emerging literacies such as digital literacy, mobile literacy and visual literacy within an overarching and unifying framework of “metaliteracy” (Mackey & Jacobson, 2011). To establish and facilitate a makerspace, the library does not need to be an expert in the technologies or in the skills that are taught through the makerspace activity. There is little doubt that maker education inspires deep learning and encourages student ownership of their own learning. The benefits are clear in that an educational makerspace: fosters curiosity, tinkering, and iterative learning, which in turn leads to better thinking through better questioning. This learning environment fosters enthusiasm for learning, student confidence, and natural collaboration. Ultimately, the outcome of maker education and educational makerspaces leads to determination, independent and creative problem solving, and an authentic preparation for the real world by simulating real-world challenges. (Kurti, Kurti & Fleming, 2014, p. 11) Curtin Library Makerspace During 2015, Curtin University Library is currently implementing a strategic initiative to build a maker community through the establishment of a makerspace. It aims to do this by facilitating ‘making’ events, activities and projects, both within the library and the wider Curtin community, as well as seeking opportunities to engage with the community more broadly. In collaboration with Curtin Teaching and Learning, the Curtin Library makerspace intends to be primarily a learning space which encourages openness, sharing, experimentation and play, and facilitates cross disciplinary interaction inquiry, learning, collaboration, outreach and research. The makerspace will allow the Library to supplement its traditional information literacy teaching role to facilitate the development of digital literacy and other skills, not only by providing a creative space for people to use for their own maker projects, but by coordinating and facilitating workshops, drop in sessions and events, as well as by participating in shared projects, including research. A further important role is to act as a conduit for the various making facilities that exist within the university, to enable the sharing of the wealth of facilities, equipment, knowledge, skills and experience for mutual benefit. As a current initiative, the library makerspace is still embryonic at the time of writing. We are in the midst of establishing a small space (approx. 30m 2) and a fledgling maker community, with library staff gradually acquiring the necessary skills, experience and knowledge to facilitate the resource. Remaining open minded as to how it develops as a space, we aim to be responsive to the changing needs of the communities we serve, with its evolution driven by the community itself. However, some of the themes we are interested in encouraging, and which reflect new developments in teaching and learning within the university include:  visualization technologies, including augmented reality and virtual reality  coding, programming, electronics, circuitry, robotics  media, animation, digital storytelling and games based learning  3D scanning, modeling and printing In addition to these themes there are four main areas that we are focussing on to establish the makerspace. First, equipping a space with appropriate resources; second, running events to build the maker community; third, facilitating learning; and finally, collaborating on cross-disciplinary maker projects. Space/ resources The physical space was established in July 2015 in a small room located next to the student lounge area, to provide an area to meet and work, and some tools, equipment and software to enable creative activity to occur. The makerspace also spills into the open lounge area by making craft activities available, such as knitting, crochet, origami and papercraft, as well as board games and DP:46 697 puzzles, to library visitors. Events / community Arguably it is the makerspace community that is the most important aspect of the makerspace. We have sought to connect with the Curtin maker community, not only through discussions with individuals and groups, but through organising events. In March 2015 Curtin Library ran a ‘popup’ makerspace during the Curtin Festival of Learning, a week long event that showcases innovative teaching and learning practices from Curtin teaching staff and researchers. We offered a range of hands-on ‘making’ activities around 3D scanning and printing, virtual and augmented reality, coding and electronics, as well as non-digital art, craft and design activities. Again, in August 2015, funded by a small grant for National Science Week, we ran another week-long makerspace event on the theme of ‘light’, offering workshops that include illuminated origami, paper and soft circuitry, electronics and coding using Arduinos, virtual reality and light painting. More recently we have participated in campus events such as Loud Shirt Day and Curtin Creative Festival. These events have been a very important way to engage with the maker community both internal and external to the university. Learning / activities & workshops A tangible way in which we have explored the new learning approach that making enables was by creating games-based learning activities that involved both mobile devices and augmented reality to develop digital literacy skills, teach research skills and raise awareness of the Library’s services and resources. We are also in the process of developed a number of maker activities, as well as drawing on the existing expertise of those within the Curtin community. For example, an engineering academic has run Arduino workshops for us, and student mentors have developed Makey Makey activities to use with groups. Projects / collaborations/ research An example of a collaborative project based in the library makerspace involves creating a 360 degree panoramic ‘virtual tour’ of Curtin Library with digital storytelling elements embedded in it, providing library clients with an immersive 3D virtual experience of the physical spaces of the Library. While navigating around and experiencing the 3-dimensionality of the spaces online, clients will be able to interact with a range of learning objects embedded throughout the virtual tour. The Library is collaborating with Curtin’s HIVE (Hub for Immersive Visualisation) as well as visualisation students studying Film & TV through Curtin’s Work Integrated Learning Program. There are many other opportunities to foster collaborative projects, for example in of K-12 education/STEM research, and through school outreach or community programs such as Curtin AHEAD (Addressing Higher Education Access Disadvantage), both internal and external to the university. One of the most exciting opportunities in establishing the makerspace is to undergo research studies of the makerspace itself, and its development. Conclusion Curtin Library’s initiative to establish a makerspace in the library has been both challenging and rewarding as it has attempted to be flexible and adaptable to its particular circumstances, and responsive to the needs of the community it serves. Above all, it has invested in fostering the maker community by forging collaborative relationships, undoubtedly the key to a successful makerspace. Our approach has been to start small, and shape it as we respond to the ideas and desires of our growing maker community. The challenge ahead is to find ways to foster the continued organic evolution of makerspace. Issues to consider include:  The legitimacy of the role of the library in the higher education context in fostering learning through making  The importance or otherwise of the digital literacy skills that can be acquired through makerspace activities, and its contribution to ‘lifelong learning’  Role of the makerspace in developing knowledge about learning technologies and their use in the higher education environment  How to evaluate the benefits, measure success and demonstrate efficacy of the makerspace DP:47 698  Formulating research questions which could make a contribution to scholarship in this area References Dunaway, M. K. (2011). Connectivism: Learning theory and pedagogical practice for networked information landscapes. Reference Services Review, 39(4), 675-685. Loertscher, D. V. (2012). Maker Spaces and the Learning Commons. Teacher Librarian, 40(1), 45- 46,63. Kurti, R. S., Kurti, D., & Fleming, L. (2014). The Philosophy of Educational Makerspaces Part 1 of Making an Educational Makerspace. Teacher Librarian, 41(5), 8-11 Miller, K (2015) Review of Makerspaces: A Practical Guide for Librarians, by John J. Burke. Australian Academic and Research Libraries, 2015. DOI: 10.1080/00048623.2015.1063188 Mackey, T. P., & Jacobson, T. E. (2011). Reframing Information Literacy as a Metaliteracy. College & Research Libraries, 72(1), 62-78. Slatter, Diane & Howard, Zaana (2013) A place to make, hack, and learn: makerspaces in Australian public libraries. The Australian Library Journal,62(4), pp. 272-284. Miller, K. (2015). Learning through doing: Creating a makerspace in the academic library. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. DP:45-DP:48). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. DP:48 699 Engaged and connected: embedding, modelling and practising what works Carolyn J. Woodley Benjamin A. Kehrwald Charles Sturt University Charles Sturt University Australia Australia The notion that there are no ‘e-pedagogies’ per se but rather ‘e-flavours’ of existing pedagogical approaches emphasises that ‘good teaching is good teaching’, irrespective of technologies – educational or otherwise. Charles Sturt University (CSU) recently released a Distance Education Strategy (2015) that promotes engagement and connectedness as key ideas in technology-enhanced teaching. Rather than prescribing particular activities to particular spaces or technologies, CSU’s Online Learning and Teaching Model foregrounds seven elements known to support learning: small group support; personalised support; teacher presence; interaction between students; interaction with workplaces; interactive resources; and e-assessment. This paper argues the merits of an approach to learning and teaching which uses these seven elements to inform online teaching practices. The literature that supports each element is considered alongside examples of elements. The discussion considers curriculum that embeds, models and explicitly teaches these seven elements of the Learning and Teaching model to the University’s academic staff. Keywords: learning design, online learning, pedagogical labels, professional development. Introduction As argued by Mayes and de Freitas (2004), there are no ‘e-pedagogies’ per se, there are merely ‘e- flavours’ of existing pedagogical approaches. Yet, the search for ‘e-pedagogies’ which might galvanize academics’ understanding of and approach to flexible, learner-centred, technology- enhanced learning continues—possibly to the detriment of good teaching practice and sensible, evidence-based discussion. In the context of both blended learning and flipped approaches, for example, despite a considerable buzz around the notion of widely applicable approaches to integrating technology into place-based teaching, there are no universally-agreed upon definitions. Both blended and flipped approaches allow for substantial variation in the sorts of teaching and learning activities promoted and the degrees of technology integration required. Thus, all too often, the basis of many flipped and blended learning definitions are unexamined generalisations about the use of technology, the roles of teachers to support productive learning activity and the relationships between the learners’ activity and the achievement of learning outcomes. These generalisations result in several problems with technology-enhanced teaching, including a view of educational technologies as prescriptive of particular approaches to teaching, a limited view of the range of teaching practices that can be enhanced with technology and how, the implementation of a limited range of online activities and axiomatic claims about what constitutes good teaching. While Charles Sturt University (CSU) still uses the term ‘blended’ in its learning and teaching discourses, the university’s recently released Distance Education Strategy, Destination 2020: A Road Map for CSU’s Online Future (Wills, Dalgarno & Olcott, 2015), promotes engagement and connectedness as key ideas in technology-enhanced teaching. Rather than prescribing particular activities to particular spaces or technologies, CSU’s Online Learning and Teaching Model foregrounds seven elements that are known to support learners and learning: small group support; personalised support; teacher presence; interaction between students; interaction with workplaces; interactive resources; and e-assessment. This paper argues the merits of an approach to learning and teaching which uses these seven elements to inform online teaching practices and considers the literature that supports each element DP:49 700 as well as examples of each element. The basis of this argument is that “good teaching is good teaching” (Ragan, 1998) and that the principles which underpin good teaching in campus-based education are the same as those which underpin good technology enhanced (or online) teaching. The differences between these modes are in how the principles are enacted. The discussion considers the redesign of a subject in CSU’s professional development program, the Graduate Certificate in Learning and Teaching in Higher Education (GCLTHE), to embed, model and explicitly teach the seven elements of the Online Learning and Teaching model. A central point is that these seven elements simply emphasise good teaching practices without particular reference to technology. In considering the implementation of the elements, the context of a technology-enhanced subject informs practical decision making about enacting the seven elements of the CSU Online Learning and Teaching Model. So, in the example provided, the elements provide both neat triggers for academic teaching staff to audit and reflect on their current practices in addition to highly practical pegs from which to hang learning and teaching activities. Working through practical examples of engagement and connectedness in the GCLTHE and setting assessment tasks that require learners to interact with their workplace, provide teaching academics with scaffolded support to redesign their own curriculum to, in turn, better support their students with engaging and connected programs. Posing the problem Teaching is not a science. Historically, teaching practices in higher education have been heavily influenced by teachers’ prior experiences and less by informed debate about how students learn (Biggs, 2003; Ramsden, 2003), how to teach particular disciplines (Young, 2010), how particular cohorts learn (Arkoudis, 2010) and how to best use technologies to support learning (Conole & Oliver, 2007). Teachers can be a passionate bunch; indeed, passion for both teaching and one’s discipline are key characteristics that define the teaching professional. Arguably, however, any teaching approach – technology-enhanced or not – holds no inherent guarantee of either student engagement or learning. The art of teaching and the lived learning experience are far more complex than any single teaching approach or technology can accommodate or claim credit for. Experienced teachers use their knowledge, expertise and understanding of their learning cohort to develop effective, engaging learning experiences and it seems that much of this learning design work is unconsciously done. In part because of the proliferation of technologies and the multiple learning affordances of technologies, we need teachers to be explicit, conscious and deliberate about learning design (Conole, 2010). Common perceptions of online teaching practices are often negative: “Many online learning platforms consist of passive video lectures and podcasts” (Pedago, 2014). The practice of digitizing existing materials and ‘putting things online’ is not going to improve bad teaching practice. Price reminds us that “a lack of imagination in course design can’t be rescued simply by being digitised” (Price, 2013: 137). Good teachers need to be good designers for an online context. A similar case was argued in the context of comparing distance education with on campus teaching. As Ragan (1998) points out, ‘Good teaching is good teaching.’ The fundamentals of good teaching practice remain unchanged across modes of delivery and medium. What changes is how those fundamental principles of good teaching are enacted. Educators should focus on effective teaching approaches that are known to engage and connect students – with other students, with ideas, with teaching and support staff or with professional networks. The rhetoric of educational labels often align particular teaching and learning activities with specific spaces and technologies. Courses that are ‘blended’ or ‘flipped’ attract attention, but those approaches must be applied thoughtfully. A focus on sound teaching practices informed by what the student does could both prove less controversial and provide a more accurate picture of teaching activities (Land & Hannafin, 2000). Of course, a focus on what the student does is what good teachers do. Education is perennially plagued by binarily represented arguments – online and face-to- face, traditional and progressive, lectures or flipped – when, really, good teachers will use whatever teaching approaches or technologies that are appropriate. CSU’s focus on engagement and connectedness in relation to teaching approaches irrespective of technology or mode, then, is useful and timely. An approach: the relationship between pedagogy and practice The networked learning community has described relationships between pedagogy and practical activity within an organisational context (Steeples, Jones & Goodyear, 2002). The networked learning model includes a pedagogical framework that both influences and is influenced by the activity within an educational setting. The pedagogical framework is conceived in four levels of activity, from most DP:50 701 abstract, to most concrete: philosophical commitments, high level pedagogy, pedagogical strategy and pedagogical tactics. Careful alignment between these four levels of the pedagogical framework supports coherent pedagogical practice. Those pedagogical practices manifest within the educational setting as: a) the development of learning tasks; b) the selection or creation and sequencing of learning resources; c) technology and media choices as part of the structure of the learning environment; and, d) situated teaching practices. Each of these, in turn, influence learner activity and, ultimately, learning outcomes. This networked learning model (Steeples, Jones & Goodyear, 2002) can be used to describe the relationship between a set of pedagogical commitments and the practical activities which are implied by those commitments. In the case of the CSU Online Learning Model (Wills, Dalgarno & Olcott, 2015), at the philosophical level, the pedagogical framework is influenced by subjectivist epistemology and relativist ontology. The high level pedagogical influences are constructivist, particularly social constructivist. Learning is viewed as an active, constructive process in which learners are generators of meaning. Learning is essentially a meaning-making endeavour in which learners acquire and apply knowledge, skills and other capabilities to respond to authentic problems. Steeples et al.’s (2002) networked learning model is predicated upon the idea of connectedness; it has strong correlations with CSU’s Online Learning Model which emphasises learner engagement, i.e., engagement with the subject or topic, with other learners, with the teacher as an authoritative supporter of learning, with the organisation or institution which accredits the learning and with the community, workplace or other setting which provide contexts for authentic learning. As part of the CSU Online Learning Model, seven elements are identified which describe pedagogical strategies (Steeples, Jones & Goodyear, 2002) promoted in online learning at CSU. Steeples, Jones and Goodyear’s (2002) pedagogical framework consists of a set of pedagogical commitments that are applied within the organisational context. The philosophical level includes the organisational mission and values as well as the epistemology, ontology and axiology (among others) that inform the choice of a high level pedagogical approach. The high level pedagogy describes a general approach to learning and teaching that is relatively abstract, but instantiates the theoretical commitments established in at the ‘philosophy’ level. The levels of pedagogical strategy and pedagogical tactics describe increasingly concrete pedagogical intention and action. ‘Strategy’ describes intentions for coherent, coordinated action, ‘high level pedagogy’ and ‘pedagogical tactics’ describe responsive, situated activity – such as engagement and connectedness, specific types of engagement and the seven elements of Online Learning and Teaching respectively. CSU’s Online Learning and Teaching Model Charles Sturt University (CSU)’s recently released Distance Education (DE) strategy (Wills, Dalgarno & Olcott, 2015) has moved away from labels like ‘blended’ learning and instead promotes engagement and connectedness as key strategies in curriculum design. The high level pedagogy of the Online Learning and Teaching model builds on Moore’s (1989) ideas about engagement to include:  Learner-teacher engagement  Learner-learner engagement  Learner-content engagement  Learning-workplace/community engagement  Learner-institution engagement. The five types of engagement are essential features of a holistic learning experience which provide a rich context for seven pedagogical tactics known to support learning: small group support; personalised support; teacher presence; interaction between students; interaction with workplaces; interactive resources; and e-assessment. Teacher presence The relationship between learners and the teacher is a powerful influence on learner activity, engagement and, ultimately, learning (Ramsden, 2003). In online learning, in which the learner and teacher are physically removed from one another and communication and interaction are mediated by technology, teacher presence facilitates the development of learner-teacher relationships in online learning by enhancing students’ experience of the teacher as not only present in the online environment, but playing a supportive role as an agent of the university (see Garrison, Anderson & Archer, 1999). Teachers make visible demonstrations of their presence and activity through the way DP:51 702 the learning materials are presented, the structure of the learning environment, facilitation and participation in learning dialogues and forms of direct instruction such as responding to student questions and providing feedback (Anderson, Rourke, Garrison & Archer, 2001). Ultimately, teacher presence works to facilitate the social and cognitive processes that constitute learning. Interaction between learners Interaction is nearly taken for granted as part of learning processes (Mayes, 2006). However, this element focuses specifically on interaction between learners and the possibilities created by mediating technologies for peer interaction amongst distributed groups of learners. Beuchot and Bullen suggest that “the potential for interaction is the most salient and most influential characteristic of computer conferencing; it alters the nature of learning and increases its quality” (Beuchot & Bullen, 2005: 69). The focus on interaction between learners emphasises the view of learning as a social process. A number of pedagogical approaches and models leverage social processes to support learners’ efforts to engage in productive activity and to make sense of their experiences (for example, social constructivism in general (Prawat & Flowden, 1994; Hung and Chen, 2001) and specific approaches including Community of Inquiry (Garrison et al., 1999) and Communities of Practice (Wenger, 1998)). Small group activity and support Further to the previous points about the teacher presence, learner-learner interaction and social learning, online social structures such as study groups are an important way to support learners’ purposeful learning activity (Kehrwald, 2005). The technology in online learning provides opportunities for social connectivity and the formation of groups or other social structures which transcend physical and temporal constraints. The formation of groups as part of learning activity can provide supportive structure for productive learning activity (Thorpe, 2002). Under the guidance of skilled online facilitators, small groups can provide learners with academic, administrative, organisational and effective support within structured learning processes (Ryan, 2001). Personalised support Personalisation of learning is an important theme in contemporary higher education. A focus on personalisation emphasises learners’ agency in learning processes and responsibility for their own learning (McLaughlin & Lee, 2008). However, as pedagogical approaches increasingly acknowledge shared control with and greater responsibility of learners, the needs for learner support need to be redefined to address the need for a different kind of responsive, learner-centred support for learning. Partly, this approach relies on interpersonal interaction to support individuals in the terms that they wish to express themselves (Thorpe, 2002). But, increasingly, learner experiences and learner support can be personalised through the use of flexible (or open) pathways, inclusive teaching practices and learner support strategies informed by learning analytics (see, for example, Buckingham Shum & Ferguson, 2012; Siemens & Long, 2011). By identifying both the needs of individual students and students at risk then using adaptive learning approaches, institutions can cultivate more productive relationships with each learner and provide a more coherent learning experience. Interactive resources The use of interactive resources provides an additional form of interaction and engagement beyond the previously described learner-teacher and learner-learner interaction. Dynamic content and rich media create opportunities for experiential engagement with learning materials and content. Quite simply, rich media learning objects provide access to information and ideas (Sosteric & Hesemeier, 2002) and can improve learners’ access to information by presenting ideas in multiple modes. Rich media can improve the cognitive accessibility of information through the integration of still images, moving images, audio and text. The addition of interactivity changes the nature of the user experience to emphasise active engagement and create the potential for a more dynamic learning experience. Used as part of authentic, interactive learning designs and online learning experiences facilitated by skilled teaching staff, interactive resources can enhance student engagement and cater to a greater range of learning preferences. DP:52 703 Interaction with workplaces The use of authentic problems and real-world contexts supports learning (Herrington, Reeves and Oliver, 2006). Moreover, learning occurs in a diverse range of sites, most of which are beyond the edges of university campuses. An emphasis on learners’ interaction with workplaces addresses the need for authenticity and acknowledges the learning that takes place as part of professional practice. The use of online and mobile technologies create opportunities to more explicitly link workplace activity and formal learning and to extend higher education beyond the university campus (see, for example, Pachler, Pimmer & Seipold (2011) for a collection of cases). As a reflective and communal space, too, online sites provide vital spaces for students in disparate working roles to connect: ‘the workplace’ is effectively multiplied and amplified and students are better able to generalise their personal learning at work (Woodley & Beattie, 2011). Learners can move from an individual workplace experience to a community of fully participatory novice professionals in a structured and safe online place (Woodley & Beattie, 2011). e-assessment Assessment and feedback are critical parts of education and learning. Therefore, it is essential to maintain high standards in the design and implementation of assessment and the provision of feedback to support learning. e-assessment helps online educators enact good practice by supporting flexible and inclusive learning and teaching practices. Educational technologies facilitate a diverse set of authentic assessment practices ranging from computer-based exams, dynamic online presentations and remote exam invigilation to digital versions of traditional scholarly writing, to the creation of rich-media records of authentic professional practice. These technologies also support the provision of timely, personalised feedback in a variety of media that can reinforce the teacher presence. In the next section, we consider the operationalisation of these seven elements in one CSU subject. Embedding, modelling and practising The Graduate Certificate in Learning and Teaching in Higher Education (GCLTHE), like most university teacher development programs, provides a vehicle through which the university can disseminate learning and teaching policies, values and quality assurance processes. Learning within the program explicitly refers to university systems, support people and policies and encourages academics to become conscious designers of effective learning experiences (Conole, 2010). The program aims to also model basic good teaching. Using the seven elements of CSU’s Learning and Teaching model to structure and provide content for a subject in the program offers a chance to both demonstrate and evaluate the utility of the model. The case in point is the GCLTHE online subject Designing for Blended Learning in Higher Education. Within the design of Designing for Blended Learning in Higher Education, the seven elements from the Online Learning and Teaching Model provide both neat triggers for academic staff to audit and reflect on their current teaching practices and highly practical pegs from which to hang learning and teaching activities. The subject design provides teaching academics with scaffolded support to examine and redesign their own curriculum to, in turn, better support their students with engaging and connected programs. Within that process, the seven elements have the capacity to provide a pattern and a structure to a learning experience. A key reason to include explicit reference to the elements is to encourage academics to consciously consider them when designing learning activities and to align their practice to CSU Learning and Teaching initiatives. Embedding An early collaborative activity can be completed either as a Wiki or in Google docs. The seven elements of the Learning and Teaching Model are presented in tabular form: learners are asked where, in their own curriculum, they can find examples of each element and to describe the example. The following elements of the Learning and Teaching model are embedded: interaction between learners, interaction with workplaces and teacher presence (self-reflexively, the teacher begins the population of the table with examples from Designing for Blended Learning in Higher Education). The DP:53 704 activity results in a collaboratively produced document that shares learning and teaching activities. Furthermore, CSU’s Online Learning and Teaching can be seen as building on what teachers already do. Modelling Personalised support is explicitly modelled through frequent communications in Announcements, the Discussion Forum, in feedback on formative and summative assessment (including using audio) and in emails. The tone of each communication is crafted to be friendly but professional – and, mostly, to be enthusiastic and encouraging. One simple example of communication that also incorporates analytics is that Announcements go straight to university email. If learners have not yet logged into the subject, they will get an email along the lines of: “According to Blackboard analytics, 6 people have not yet accessed the site and 4 people have not yet posted in Discussion – if that is you, expect an email later today! ” Learners are reminded of the ease with which even basic analytics of Blackboard can provide prompts for early reminders to learners to engage. The capacity of Blackboard to monitor learners’ activity is one thing: reminding learners that it is their role as teachers to remind their learners is the modelling. Blackboard’s Survey tool, too, is used to gauge learner’s awareness of particular theories, university policies and technologies. Survey results also serve to support a more tailored if not personalised learning design. Beyond analytics, personalised support is evidenced in personal, tailored responses to students both in Discussion threads and via emails. Individual responses to Posts acknowledging ideas and suggesting resources create a personal learning experience in a social context. These teaching approaches have the teacher presence at their core but also embed personalised support. Practising Various activities see learners practising, in supported ways, learning with unfamiliar technologies or Blackboard functions that are typically underutilised. For example, small group support is worked into an e-assessment task that asks students to present and facilitate discussion online in groups of 4-5. The activity embeds small group support, interaction between learners, teacher presence (the teacher provides feedback on each session) and e-assessment. Other engaged and connected activities include contributing to the subject’s glossary, collaborating on a Wiki that aligns particular technologies with scaffolding learning activities as well as activities that model teacher presence (such as emailing draft assessment tasks in a formative sequence). The most useful aspect of the seven elements of CSU’s Learning and Teaching Model is that they are entirely practical and decidedly sensible to academic teaching staff who are not from educational backgrounds: that is, they make sense, are easily operationalised and support teachers in designing curriculum. More broadly, a range of teaching and learning activities in Designing for Blended Learning in Higher Education exemplify the elements of CSU’s Learning and Teaching model. Teacher presence Teacher presence can readily be seen in activities such as regular teacher-generated bulletins, video snippets to provide multi-modal teacher presence, regular participation in ongoing subject discussions, personalised responses to each student’s posts, personalised contact with students who are identified ‘at risk’ according to CSU metrics and personalised and contextualised feedback on assessment items. Interaction between learners and small group activity and support Activities that encourage interaction between learners and small group activity and support are evident in some assessment tasks that require incorporating peer review and in other explicitly designed collaborative activities that ask for whole group input. The purposeful formation of learning groups for the final presentation also exemplifies these elements as do learning tasks which require regular interaction between learners, using Google Docs and Blackboard’s Wiki, and collaborative assessment. Personalised support While many of the examples of teacher presence are also examples of personalised support, this element is also evidenced in the creation of flexible study pathways, flexible arrangements about assessment and using learning analytics and other CSU tools and metrics to identify students at risk DP:54 705 and to provide individual support. Interactive resources Of all of the elements in CSU’s Online Learning and Teaching model, the area of appropriate interactive resources perhaps needs a greater lead in time for designers and teachers. While the use of rich media learning objects is planned for the subject, they are not yet in use. However, various other dynamic resources, multimodal resources, open and flexible resources are in use. An additional aspect of this elements is an assessment task that asks students to devise a plan to make their own open educational resources (OERs). Interaction with workplaces The Graduate Certificate in Learning and Teaching in Higher Education is for CSU staff, so the context, the purpose and the content all focus on interaction with the workplace. In-house teaching qualifications seek to develop academics as teachers and are part of a national push to improve the quality of teaching at Australian universities (Ling, 2009). The range of learners from disciplines as diverse as veterinary science and dentistry to information systems and agriculture mean that the communal reflections (Woodley and Beattie, 2011) of this online community serve to multiply and expand CSU as a workplace and to provide students with a rich sense of the university. All assessment is highly experiential with a focus on teachers’ own practices - their assessment, their teaching approaches and their students. Learning and assessment tasks are linked to authentic activity of academics as well as CSU processes, tools, policies and support staff. e-assessment E-assessment in this subjects is interpreted both broadly (as in assessment activities that are supported, completed and submitted online) and narrowly (as in the use of the CSU-developed Electronic Assignment Submission Tracking System (EASTS)). Feedback, too, on assessment is via personalised comments and track changes function in word, videos for whole of class feedback as well as a range digital marketing approaches including audio feedback. Each assessment piece requires students to use and/or explore different technologies and to especially exploit the collaborative capacity of online communication tools. Conclusion Beyond pedagogical labels, as Cedar Riener Tweeted in a text speak Tweet: “many teachers just trying to tweak pedagogy to be better, build bridges to students, etc. Many don't care abt label” (Reiner, 2013). CSU’s seven elements manage to sidestep anything like controversial or ill-defined educational labels to focus on what the student does. Kearsley (2000) argues that “the most important role of the instructor in online classes is to ensure a high degree of interactivity and participation” (Kearsely, 2000: 78). CSU’s Learning and Teaching model supports that idea. “Learning is interactive when learners are actively engaged in a variety of activities, and along with their peers and teaching, they are co-constructors of knowledge” (Chamberlain & Vrasidas, 2007: 79). This broad definition of interactivity gestures towards the idea of students as generators of meaning: a role that the internet facilitates. Engaged and developmental learning through a mix of activities that acknowledges constructivism as an effective learning design is not new (Bornstein, 1989). Such approaches recognise that content acquisition is not sufficient for an education and that a broader engagement is needed: “in an engaged learning environment, each learner’s actions contribute not only to individual knowledge but to overall communication development as well” (Conrad & Donaldson, 2004: 5). Engaged learning can be collaborative. It includes students collaborating with lecturers to establish learning goals or negotiating assessment, students locating, critiquing and sharing appropriate resources and ongoing assessment – including peer assessment (Conrad & Donaldson, 2004). New media offers ever increasing opportunities for engaged and connected collaborative learning experiences. Engaged learning does not emphasise technology for any particular type of teaching. Engaged learning is concerned with what the student does: face-to-face, online, in the community and in the workplace. Is teaching more art than science? No single pedagogical approach or theory is likely to define or accurately depict what goes on in the teaching and learning space. We need to have more teaching approaches in out arsenal, not just a chosen few, and approaches do not need to be hierarchised – they just need to be available. Teaching approaches need to focus on the learner – not DP:55 706 the definition, not the technology. CSU’s Online Learning and Teaching model seeks to develop a range and a pattern of activities that creates a learning continuum. The engagement of students has often been measured by indicators such as time spent studying, class hours, time spent in extra-curricular activities. The model aims to focus on approaches that “combine pedagogy and learning technologies in ways that extend to large numbers of student’s opportunities for deep learning through application and consolidation” (Sankey & Hunt, 2013: 787). CSU’s Learning and Teaching model recognises that there are no e-learning models, only e-enhancements of existing learning (Mayes & De Freitas, 2004). The move away from hyperbolic educational labels augurs well for a focus on what learners actually do. References Anderson, T. Rourke, L., Garrison, D. R. & Archer, W. (2001). Assessing teaching presence in a computer conferencing context. Journal of Asynchronous Learning Networks, 5(2): 1-17. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.95.9117&rep=rep1&type=pdf Anderson, C., Day, K., Haywood, D., Haywood, J., Land, R. & Macleod, H. (2002). Evaluating networked learning: Developing a multi-disciplinary, multi-method approach. In C. Steeples and C. Jones (Eds.) Networked Learning: Perspectives and Issues (pp. 169-192). London: Springer. Arkoudis, S. (2010) Teaching International Students Strategies to enhance learning, Melbourne: CSHE. www.cshe.unimelb.edu.au/resources_teach/ teaching_in_practice/docs/international.pdf Beuchot, A., & Bullen, M. (2005). Interaction and interpersonality in online discussion forums. Distance Education, 26(1), 67-87. Biggs, J. (2003). Teaching for quality learning at university. Open University Press/Society for Research into Higher Education: Buckingham. Buckingham Shum, S. & Ferguson, R. (2012). Social Learning Analytics. Educational Technology and Society, 15 (3), 3–26. Conole, G. & M. Oliver (2007). Contemporary perspectives in e-learning research: Themes, methods and impact on practice. London: Routledge. Conole, G. (2010). Review of Pedagogical Models and their use in e-Learning. www.slideshare.net/grainne/pedagogical-models-and-their-use-in-elearning-20100304 Conrad, R-M & Donaldson, J. A. (2004) Engaging the Online Learner: Activities and Resources for Creative Instruction. San Francisco: Jossey-Bass. Garrison, D.R., Anderson, T. & Archer, W. (1999). Critical inquiry in a text-based environment: Computer conferencing in higher education, The Internet and Higher Education 2(2): 87-105. Herrington, J. A., Reeves, T. C. & Oliver, R. (2006). A model of authentic activities for online learning. In C. Juwah (Ed.) Interactions in online education. Abingdon: Routledge. Hung, D. W. L. & D.-T. Chen (2001). Situated cognition, Vygotskian thought and learning from the communities of practice perspective: Implications for the design of web-based e-learning, Education Media International 38(1), 3-12. Kearsley, G. (2000). Online Education: Learning and teaching in cyberspace. Wadsworth/Thomson Learning: Belmont, CA. Kehrwald, B. A. (2005). Learner support in online learning communities: opportunities & challenges. In J. B. Son & S. O'Neill (Eds.) Enhancing learning & teaching: Pedagogy, technology & language, Post Pressed: Flaxton, Australia: 133-148. Land, S.M. & Hannafin, M.J. (2000). Student-centered learning environments. In D.H. Jonassen & S.M. Land DP:56 707 (Eds.) Theoretical foundations of learning environments (pp. 1-26). Lawrence Earlbaum Associates: Mahwah, NJ. McLoughlin, C., & Lee, M.J.W. (2008). The 3 P's of pedagogy for the networked society: Personalization, participation, & productivity. International Journal of Teaching & Learning in Higher Education, 20(1), 10–27. Mayes, T. (2006). Theoretical perspectives on interactivity in e-learning. In C. Juwah (Ed.) Interactions in online education (pp. 9-26). Routledge: Abingdon, UK. Mayes, T. & de Freitas, S. (2013) In H. Beetham & R. Sharpe (Eds.) Rethinking Pedagogy for a Digital Age: Designing and Delivering E-Learning, Routledge: London. Moore, M. G. (1989). Three types of interaction. American Journal of Distance Education, 3(2), 1-7. Pachler, N., Pimmer, C. & Seipold, J. (Eds.) (2011). Work-based mobile learning: Concepts and cases. Peter- Lang: Bern. Pedago (2014). The Brave New (Wired) World of Online Education. http://blog.smart.ly/tag/education/ Prawat, R. S. & R. E. Floden (1994). Philosophical perspectives on constructivist views of learning. Educational Psychologist, 29(1), 37-48. Ragan, L.C. (1998). Good Teaching is Good Teaching: An Emerging Set of Guiding Principles & Practices for the Design & Development of Distance Education. DEOSNEWS 8(12). www.ed.psu.edu/ACSDE/deosnews8.12.htm Ramsden, P. (2003). Learning to Teach in Higher Education. Routledge: London. Reiner, C. (2013) Tweet at https://twitter.com/jhrees/status/393547132776353793 Ryan, Y. (2001). The provision of learner support services online: the changing faces of virtual education. The Commonwealth of Learning: Vancouver, BC, 71-94. Sankey, M.D. & Hunt, L. (2013). Using technology to enable flipped classrooms whilst sustaining sound pedagogy. In H. Carter, M. Gosper & J. Hedberg (Eds.) Electric Dreams. Proceedings ascilite Sydney, 785-795. Siemens, G. & Long, P. (2011) Penetrating the Fog: Analytics in Learning and Education, EDUCAUSE Review, 46(5), pp. 30-32, 34, 36, 38, 40. Sosteric, M. & Hesemeier, S. (2002). When is a Learning Object not an Object: A first step towards a theory of learning objects. International Review of Research in Open and Distance Learning Journal, 3(2). http://www.irrodl.org/index.php/irrodl/article/view/106/185 Steeples, C., Jones, C. & Goodyear, P. (2002). Beyond e-learning: a future for networked learning. In Steeples, C & Jones, C (Eds.) Networked Learning: Perspectives and Issues, (pp. 323- 341). London: Springer. Thorpe, M. (2002). Rethinking learner support: The challenge of collaborative online learning, Open Learning 17(2), 105-119. Trigwell, K. (2001). Judging university teaching. International Journal for Academic Development, 6 (1), 65- 73. Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge University Press: Cambridge and New York. Wills, S., Dalgarno, B. and Olcott, D. (2015) Destination 2020: A Road Map for CSU’s Online Future. Division of Student Learning, Charles Sturt University: Wagga Wagga. Woodley, C. & Beattie, S. (2011). Communal reflections on the workplace: Locating learning for the legal professional, Asia-Pacific Journal of Cooperative Education. 12(1), 19-30. Young, P. (2010) Generic or discipline‐specific? An exploration of the significance of discipline‐ specific issues DP:57 708 in researching and developing teaching and learning in higher education, Innovations in Education and Teaching International. 47(1), 115-124. Woodley, C.J. & Kehrwald, B.A. (2015). Engaged and connected: embedding, modelling and practising what works. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. DP:49-DP:58). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. DP:58 709 Developing the Scholarship of Technology Enhanced Learning (SOTEL) Thomas Cochrane Vickel Narayan Centre for Learning and Teaching Centre for Learning and Teaching Auckland University of Technology Auckland University of Technology Embedding a reflective practice framework based upon SOTEL within collaborative curriculum design critically informs the evaluation and impact of the curriculum redesign process and provides a mechanism for dissemination to a broader, global audience. This symposium explores the potential of developing collaborative open scholarship networks for SOTEL and educational design research. Keywords: Open Scholarship, Collaborative Scholarship, Scholarship of teaching and learning. Introduction Laurillard (2012) describes teaching as a design science and argues that this should involve collaborative curriculum design enabled by digital technologies. We propose a framework in which we embed an explicit focus upon the scholarship of teaching and learning (SOTL) to inform a deeper level of collaborative curriculum design (Weaver, Robbie, Kokonis, & Miceli, 2012). Originally conceptualised by Boyer (1990) as a way to validate reflective practice as a viable and valued research focus, a range of researchers have made a case for updating SOTL for the social media environment of twenty first century education (Garnett & Ecclesfield, 2011; Greenhow & Gleason, 2014; Haigh, 2010). Education is often seen as a transformative experience for learners, however the role of technology in mediating transformation in education has been hotly debated (JISC, 2011; Keane & Blicbau, 2012; Puentedura, 2006; Reeves, 2005). The application of SOTL to technology enhanced learning is one way to critically evaluate the broader impact of technology in education, and has led to the emerging development of the scholarship of technology enhanced learning (SOTEL) (Wickens, 2006). SOTEL links the scholarship of teaching and learning with the growing body of literature surrounding the exploration and impact of technology enhanced learning. Critical reflection on the experience of designing and implementing collaborative curricula have included conference proceedings, book chapters, and journal papers. These have helped refine and reshape the course design and also informed the design and development of subsequent courses. SOTEL also provides the opportunity to conceptualise the design of a course by making a direct link between theory and practice, and also provides a model for our lecturer colleagues within a department and the wider university to explore. The adoption of a SOTEL model has resulted in a wide body of research within a variety of educational contexts that now encompasses a network of over 37 lecturers as collaborative curriculum designers and reflective practice co-authors, producing over 100 peer reviewed publications. Embedding critical reflection upon the impact and effectiveness of our learning designs by an explicit focus and nurturing of the scholarship of technology enhanced learning enables a deeper reflective process and a wider impact via scholarly peer-reviewed publications. We encourage a culture of publication within open access journals, conference proceedings, and establishing research profiles on emerging social media research communities such as Researchgate.net and Academia.edu, but acknowledge that this is still an emergent avenue for academic scholarship. For many of the lecturers that we work in partnership with, SOTEL represents a new field of research that can compliment their discipline-based research activities. The symposium will provide some practical examples of how we attempt to build these collaborative research networks and invite participants to begin establishing their own SOTEL networks. Format 1. Introduction to SOTEL (5 mins) • Short participant SurveyMonkey Survey – what collaborative scholarship tools do you use and why? SP:1 710 2. Examples of open scholarship networks and collaboration tools for developing SOTEL (15 mins) • Embedding SOTEL within an educational design research methodology (EDR) • Researchgate and Academia.edu • Evernote, and Google Docs - Shared collaborative writing • Twitter – creating a global SOTEL network • Mendeley • Flipboard, ScoopIt, and curation tools • Google Plus - Building collaborative communities 3. Participant Brainstorm: (10 mins) 1. How could you use these tools in your context? 2. What other social scholarship tools would you suggest? 3. Discuss on Todaysmeet: http://todaysmeet.com/sotel and Twitter: #soteled Strategies The session will utilise a range of participant interaction strategies including and introductory survey, a Twitter conversation stream, and time for participants to sign up on collaborative scholarship platforms such as ResearchGate, Academia and Mendeley. Audience Any academic wanting to explore collaborative open scholarship. Participants will need to BYOD: laptop, smartphone, and/or wireless tablet. Biographies of Presenters Thomas Cochrane Thomas Cochrane is an Academic Advisor and Senior Lecturer in educational Technology at AUT University's Centre for Learning and Teaching (CfLAT). Thomas has managed and implemented over 50 mobile learning projects, with a recent focus upon Android and iOS smartphones and the iPad as catalysts to enable student-generated content and student-generated learning contexts, bridging formal and informal learning environments. He has over 100 peer reviewed publications, receiving best paper awards at Ascilite 2009, ALT-C 2011, and ALT-C 2012. Research profile: https://www.researchgate.net/profile/Thomas_Cochrane/ Vickel Narayan Vickel Narayan a learning and teaching consultant at the Centre for Learning and Teaching (CfLAT) at the Auckland University of Technology. Previously, Vickel was an Academic Advisor (eLearning) at Unitec Institute of Technology from 2009 to 2011. He has a keen interest in Web 2.0 technologies and its potential to engage students and teachers in the teaching and learning process. Vickel is particularly interested in exploring mobile Web 2.0 tools for creating, nurturing and maintaining virtual communities, social connectedness, fostering social constructivism, student generated content and contexts. References Boyer, E. (1990). Scholarship reconsidered: priorities of the professoriate. Princeton, NJ: Carnegie Foundation for the Advancement of Teaching. Garnett, F., & Ecclesfield, N. (2011). Towards a framework for co-creating Open Scholarship. In D. Hawkridge, K. Ng & S. Verjans (Eds.), Proceedings of ALT-C 2011 - Thriving in a colder and more challenging climate: The 18th international conference of the Association for Learning Technology (pp. 199-216). University of Leeds, UK: ALT Association for Learning Technology. Greenhow, C., & Gleason, B. (2014). Social scholarship: Reconsidering scholarly practices in the age of social media. British Journal of Educational Technology, 45(3), 392-402. Haigh, N. (2010). The Scholarship of Teaching & Learning: a practical introduction and critique. Ako Aotearoa, National Office, Wellington, New Zealand: Ako Aotearoa. JISC. (2011). Transforming curriculum delivery through technology: Stories of challenge, benefit and change (Internet). Bristol: Higher Education Funding Council for England. Keane, T., & Blicbau, A. (2012). (Trans) Formation through educational technologies. Paper presented at the ascilite 2012: Future challenges, sustainable futures, Wellington, New Zealand. SP:2 711 Laurillard, D. (2012). Teaching as a design science: Building pedagogical patterns for learning and technology. New York: Routledge. Puentedura, R. (2006). Transformation, Technology, and Education. Retrieved 18 February, 2013, from http://hippasus.com/resources/tte/puentedura_tte.pdf Reeves, T. (2005). No significant differences revisited: A historical perspective on the research informing contemporary online learning. In G. Kearsley (Ed.), Online learning: Personal reflections on the transformation of education (pp. 299-308). Englewood Cliffs, NJ: Educational Technology Publications. Weaver, D., Robbie, D., Kokonis, S., & Miceli, L. (2012). Collaborative scholarship as a means of improving both university teaching practice and research capability. International Journal for Academic Development, 18(3), 237-250. Wickens, R. (2006). SoTEL: Toward a Scholarship of Technology Enhanced Learning. Canadian Journal of University Continuing Education 32(2), 21-41. Cochrane, T. & Narayan, V. (2015). Developing the Scholarship of Technology Enhanced Learning (SOTEL). In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. SP:1-SP:3). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. SP:3 712 #NPF14LMD Learners and Mobile Devices: Sharing Practice Thomas Cochrane Stanley Frielick Vickel Narayan Centre for Learning And Teaching, Centre for Learning and Teaching Centre for Learning And Teaching, Auckland University of Technology, New Auckland University of Technology, New Auckland University of Technology, Zealand. Zealand. New Zealand. Acushla Dee Sciascia Centre for Learning And Teaching, Auckland University of Technology, New Zealand. NPF14LMD is a National AKO Aotearoa funded two year project exploring the key issues surrounding learners and mobile devices. The project encompassed six tertiary institutions across New Zealand, involving over 50 practitioners and several hundred students. The sharing practice session will report on some of the key findings from this project after two years of implementation. Keywords: mobile learning, learner engagement, new pedagogies. Introduction The field of mobile learning has a significant and growing body of literature and research, however there has been little longitudinal research or meta analysis on the impact of mobile learning upon learners and the key issues around integrating mobile devices into educational environments. After two years of exploration and implementation the #NPF14LMD project (Cochrane et al., 2014) has a wealth of practitioner and learner stories to enrich the field of mobile learning. Practitioners from the six collaborating tertiary education institutions across New Zealand will share their journeys, and introduce the development of an innovative new framework linking mobile learning and Maori pedagogies. The project utlises a commuity of practice hosted via a Google Plus Community (http://bit.ly/1zP2S0T) that links the six case studies creating a national network of researchers and practitioners. practitioner stories are curated via a variety of social media platforms using a common hashtag #npf14lmd (e.g. see the Twitter conversation analysis at http://bit.ly/MwhcLi) and a collaborative Google map http://bit.ly/npf14lmdmap. Format 1. Introduction of the #NPF14LMD project team (5 mins) • Short participant SurveyMonkey Survey – what is your experience of implementing mobile learning? 2. We will showcase three examples of mobile learning implementations from six collaborating institutions across New Zealand selected from a variety of contexts including: (15 mins) • Paramedicine • Occupational Therapy • Computer Science • Game Design • Carpentry • Teacher education • Introduction to the Framework: “He Whare Ako, He Whare Hangarau” 3. Participant Question and Answer: (10 mins) 4. How could you use mobile learning in your context? 5. What are the key practivcal lessons learnt? 6. Discuss on Todaysmeet: http://todaysmeet.com/npf14lmd and Twitter: #npf14lmd Strategies The session will utilise a range of participant interaction strategies including an introductory survey, a SP:4 713 Twitter conversation stream, and time for participants to ask in depth questions of each of the case studies represented. Audience Any academic wanting to explore mobile learning. Participants will need to BYOD: laptop, smartphone, and/or wireless tablet. Biographies of Presenters Thomas Cochrane Thomas Cochrane is an Academic Advisor and Senior Lecturer in educational Technology at AUT University's Centre for Learning and Teaching (CfLAT). Thomas has managed and implemented over 50 mobile learning projects, with a recent focus upon Android and iOS smartphones and the iPad as catalysts to enable student-generated content and student-generated learning contexts, bridging formal and informal learning environments. He has over 100 peer reviewed publications, receiving best paper awards at Ascilite 2009, ALT-C 2011, and ALT-C 2012. Research profile: https://www.researchgate.net/profile/Thomas_Cochrane/ Stanley Frielick Dr Stanley Frielick is Director of Learning and Teaching at AUT University in Auckland – a central role in a network of staff, students and enabling technologies that increases the capability of the university for educational development and innovation. His involvement in educational computing began as a teacher with the Apple IIe and LOGO in schools in 1984, and since then has participated in the evolution of networked technologies that interlink with the development of new theories and approaches. Vickel Narayan Vickel Narayan a learning and teaching consultant at the Centre for Learning and Teaching (CfLAT) at the Auckland University of Technology. Previously, Vickel was an Academic Advisor (eLearning) at Unitec Institute of Technology from 2009 to 2011. He has a keen interest in Web 2.0 technologies and its potential to engage students and teachers in the teaching and learning process. Vickel is particularly interested in exploring mobile Web 2.0 tools for creating, nurturing and maintaining virtual communities, social connectedness, fostering social constructivism, student generated content and contexts. Acushla Dee Sciascia Dr. Acushla Dee Sciascia (O’Carroll) Ngaruahine Rangi, Ngati Ruanui, Te Ati Awa Senior Researcher Centre for Learning and Teaching (CfLAT), AUT University Te Wananga Aronui o Tamaki Makaurau Mandia Mentis Mandia Mentis coordinates and teaches the post graduate Special Education Programme at Massey University, New Zealand and manages the online teaching and community environments for the School of Education. My research interests include inclusive education, e-learning within communities of practice and assessment and teaching practices for learners with diverse needs. James Oldfield James Oldfield is a Senior Lecturer in Information Systems and an SP:5 714 Academic Advisor within Te Puna Ako at Unitec Institute of Technology. James currently provides advice and support to teaching staff working with new teaching spaces, eLearning and mobile learning. James has recently designed and implemented a mobile learning initiative in the Bachelor of Business programme where iPads are used by all students to enable pedagogical change. His research interests are focused on mobile learning and authentic learning and is an Apple Distinguished Educator. Adrienne Moyle Adrienne Moyle (Learning Designer, Centre for the Creative Application of Technology in Education, Faculty of Education, University of Auckland) Adrienne’s background in screen production, learning design and education has resulted in a unique mix of skills with which to contribute to the Centre for the Creative Application of Technology in Education (CreATE). Adrienne really enjoys empowering educators to integrate technology into their learning and teaching in pedagogically sound ways. The learners and mobile devices project (#NPF14LMD) is a great way for her to collaborate nationally with other participant researchers, and learn more about the affordances of mobile devices. References Cochrane, T., Narayan, V., Brannigan, R., Frenchman, K., Nicholson, E., Rutherford, S., et al. (2014). #NPF14LMD AUT University Case Studies. Poster presented at the Rhetoric and Reality, proceedings of the 31st Ascilite Conference. Retrieved from http://ascilite2014.otago.ac.nz/posters/ - 291 Cochrane, T., Frielick, S., Narayan, V., & Sciascia. A.D. (2015). #NPF14LMD Learners and Mobile Devices: Sharing Practice. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. SP:4-SP:6). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. SP:6 715 Learning Analytics Special Interest Group: Recognising Outstanding Practice in Learning Analytics Grace Lynch Abelardo Pardo Simon Welsh Adaptive Learning & Teaching Services Society for Learning Analytics Research School of Electrical and Charles Sturt University Information Engineering University of Sydney Keywords: Learning Analytics, higher education, learning and teaching, awards Audio/Visual Requirements AV/Data projector, screen and whiteboard Abstract The ascilite Learning Analytics Special Interest Group (LA SIG) aims to promote and develop awareness and resources surrounding Learning Analytics and its application to learning and teaching. The LA SIG sharing practice session will:  facilitate networking for LA researchers and practitioners;  facilitate sharing of outstanding practice in LA via presentations from the three finalists in the LA SIG Awards program. The Awards finalists will share their experiences and lessons learned in relation to their nominated projects/practices. The 2015 Award winner will also be announced during the session; and  provide updates on what’s happening in the Learning Analytics community within Australia/New Zealand and beyond. The session will have the following structure: 1. Latest developments in the Learning Analytics community in Australia/NZ and an update from the Society of Learning Analytics Research – Dr Grace Lynch and Dr Abelardo Pardo 2. Presentations from the finalists for the 2015 Award for Excellence in Learning Analytics – moderated by Simon Welsh 3. Discussion and Q&A – moderated by Simon Welsh 4. Announcement of the 2015 Award winner – Dr Grace Lynch and Dr Abelardo Pardo About the Awards In 2015, the LA SIG launched the Awards for Excellence in Learning Analytics with the purpose of providing an opportunity for those working with Learning Analytics to share their practice and be recognised for their achievements in this field, while creating resources for SIG members around effective practices in Learning Analytics. Lynch, G., Pardo, A., & Welsh, S. (2015). Learning Analytics Special Interest Group: Recognising Outstanding Practice in Learning Analytics. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. SP:7- SP:7). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. SP:7 716 Easing into mobile learning Angela Murphy Helen Farley Australian Digital Futures Institute Australian Digital Futures Institute University of Southern Queensland University of Southern Queensland Keywords: Mobile learning, m-learning, smart mobile technologies, BYOD, e-learning, higher education Audio/Visual Requirements Room set up workshop style (group tables of 6 – 8) with the following: AV - Data projector and screen, whiteboard. Abstract Research has identified that students have access to and use a wide range of mobile devices to informally support their learning practices, however few students have access to educator-led initiatives to support mobile learning (Farley, Lane, Hafeez-Baig & Carter, 2014). Many educators are keen to leverage the affordances of mobile technologies to improve collaboration, interactivity and personalization within their courses, yet tight budgets and lack of training opportunities leave them wondering where to begin. This session will discuss a eight principles that have emerged from a recent study conducted by researchers from the University of Southern Queensland (USQ), in partnership with researchers at the Australian National University (ANU) and the University of South Australia (UniSA) (Farley, Murphy & Johnson, 2015). The aim of the project is to develop a Mobile Learning Evaluation Framework (MLEF) and is funded by the Australian Government’s Collaborative Research Network (CRN) program. The eight principles offer educators practical, low-cost tactics that will facilitate their engagement with mobile learning and encourage them to challenge their current teaching methods. This discussion will also report on the findings of the research on the differences between students studying primarily on-campus compared to those who study primarily in an online environment. This discussion will assist educators who have both on-campus and online students with adjusting their approach to ensure that all students have equal opportunities to maximize the benefits of mobile learning and feel included and integrated despite their geographic location. Session Agenda 1. The session will begin with an overview of the Mobile Learning Evaluation Framework which is aims to support leaders and educators in higher education institutions to provide sustainable mobile learning opportunities to students. This will be streamlined by the provision of a pamphlet giving details of the projects so that too much time is not taken up with this portion of the program. 2. Attendees will be asked about their previous experience with implementing mobile learning initiatives in their teaching practices and how their students are spontaneously using mobile technologies to support their learning. Attendees will be encouraged to share these experiences with the group. 3. Attendees will be provided with information gathered during the project about the needs, issues and practices of students engaging mobile technologies in their learning, including issues such as wireless connectivity. 4. We will consider as a group the issues and challenges identified by educators in attempting to implement mobile learning initiatives within their teaching practice or learning institutions. 5. The session will conclude with a discussion around the eight principles that emerged from the SP:8 717 study and potential for additional strategies that may be employed to encourage the sustainable implementation of mobile learning initiatives within higher education institutions. References Farley, H., Murphy, A. & Johnson, C. (2015) How Do Students Use Their Mobile Devices to Support Learning? A Case Study from an Australian Regional University. Journal of Interactive Multimedia in Education. Murphy, A., Farley, H., Lane, M., Hafeez-Baig, A. & Carter, B. (2014). Mobile learning anytime, anywhere: What are our students doing? Australasian Journal of Information Systems, 18 (3), 331- 345. Murphy, A. & Farley, H. (2015). Easing into mobile learning. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (ppSP:8-SP:9). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. SP:9 718 Clinical Logs: Best Practices in the Design and Implementation David Bruce Porter Michelle Moscova Graduate School of Medicine Graduate School of Medicine University of Wollongong University of Wollongong Clinical logs as used in medical education provide data to track students’ clinical experiences and patient encounters. These data have been used to ensure consistency of experience across clinical placements, to measure compliance with course outcomes, and to enable students to identify and address gaps in their learning. A variety of paper-based and technology-assisted logging methods have been used. Electronic logs provide the advantages of convenient data access, storage, and real- time analytics, with the potential for both student and faculty access. To ensure the soundness of design and implementation of their clinical log, researchers at the University of Wollongong Graduate School of Medicine conducted a review of the pitfalls and best practices in using clinical logs, particularly electronic logging platforms, to ensure soundness of design and implementation of their clinical log. Briefly, the clinical log provides an excellent tool to administratively track the experiences of students in their clinical placements and to assess the gaps in their learning. However, the usefulness of the logs may be limited by the validity of the student-reported data and the students’ perceived educational benefits of the log. This presentation will discuss the purpose, benefits, challenges, and recommendations in the implementation of clinical logs in medical education. Porter, D.B. (2015). Clinical Logs: Best Practices in the Design and Implementation. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. AB:1-AB:1). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. AB:1 719 Institution wide information privacy frameworks to support academics in the use of learning analytics Eva Dobozy Jennifer Heath Pat Reynolds School of Education Student Support & Education Analytics Professor Emeritus Curtin University University of Wollongong King's College London Eeva Leinonen Deputy Vice-Chancellor (Academic) University of Wollongong Lecturers invest time and effort in developing, implementing and evaluating their learning designs. They are also increasingly interested in and engaged with the capture of changes in student engagement and utilization patterns and learning outcomes using learning analytics tools. These new analytics tools make individual student surveillance possible. Given these rapid developments, there is now an urgent need for educators and learning analytics researchers to think about the ethics of learning analytics and the protection of individual privacy. This presentation will consider the importance of an Institution wide privacy framework to support learning analytics. Institution wide frameworks provide protection for both students and academic staff as they engage with learning analytics and should provide the academic staff with clarity regarding ethical matters that often arise in this domain. Key features of institution wide frameworks include: governance structures; responsibility for action; maximum transparency; privacy principles consistent across diverse education delivery methods; legislative requirements met; suitable student consent mechanisms and; a clear secondary use of data policy. Keywords: Learning Design, Learning Analytics, information privacy, data-driven education, personal privacy Dobozy, E., Heath, J., Reynolds, P., & Leinonen, E. (2015). Institution wide information privacy frameworks to support academics in the use of learning analytics. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. AB:2-AB:2). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. AB:2 720 Digitally enabled learning through Bb+ Ruth Greenaway Susie Vergers Maxine Mitchell University of the Sunshine Coast University of the Sunshine Coast University of the Sunshine Coast Ensuring the learning journey of each student is digitally enabled and supported through the effective use of technology was the driver for the introduction of what we have called Blackboard Plus (Bb+). The University of the Sunshine Coast (USC) uses the learning management system Blackboard (Bb) and all courses have a Blackboard course site, whether the course is taught face-to-face or online, locally or globally. Academics are encouraged to consider their Blackboard course site as an extension of their physical classroom to digitally enable student learning. Students require instructions to navigate the Bb course site just as they do in a face-to-face learning environment. This notion ensures curriculum is student-focused, explicit and relevant with intentional integration and sequencing of knowledge, skills and attitudes to enhance their learning experience (Nelson et al., 2014). However, all too often Bb course sites are organised in different ways and students have told us that the inconsistency of navigation and organisation of materials across courses and programs is confusing and frustrating. To improve the student experience the Centre for Support and Advancement of Learning and Teaching (C~SALT) at USC developed Bb+ and used a bottom-up approach to the initiative’s implementation. Bb+ is a University-wide initiative coordinated by the C~SALT curriculum support team and designed to assist academic staff in making the change from a face-to-face to a blended learning approach. Bb+ aims to bring consistency of course design and presentation across courses and programs whilst ensuring that the underlying pedagogy enhances learning, regardless of delivery mode. The goal is to improve the student experience when navigating Bb, which in turn will improve the opportunity to meet the course learning outcomes. There are a number of parts to Bb+. The Core Elements identify important requirements informed by universal design principles (CAST, 2011) that assists students to connect with the course and the learning outcomes. Core Elements are presented in a checklist, supported by a user guide and an example course. Two other key components of Bb+ are the course templates (automatically uploaded for academics) and a visual design toolkit which enables academics to improve the “look and feel” of their Bb course sites. Ultimately, it is academics that manage the Bb course sites and facilitate the students’ blended learning experiences. Therefore a bottom-up approach to the implementation of Bb+ is utilised so academics are empowered to create the change and to design blended educational programs suited for local and global needs (Carbonell et al., 2013). A bottom-up approach to Bb+ harnessed the enthusiasm of the early adopters and created momentum for lasting change across the University. The course coordinators of a number of first year courses from each faculty participated in the pilot to impact as many students as possible. Through a series of hands-on workshops, course coordinators were assisted in developing their courses to meet the core elements. These courses will be used as examples for colleagues to view as the Bb+ innovation is rolled out across the whole University in 2016. References Carbonell, K. B., Dailey-Hebert, A., & Gijselaers, W. (2013). Unleashing the creative potential of faculty to create blended learning. The Internet and Higher Education, 18, 29-37. CAST (2011). Universal Design for Learning Guidelines version 2.0. Wakefield, MA: Author. http://www.udlcenter.org/aboutudl/udlguidelines [viewed 14/7/15] Nelson, K. J., Creagh, T., Kift, S. M., & Clarke, J. A. (2014). Transition Pedagogy Handbook: A Good Practice Guide for Policy and Practice in the First Year Experience at QUT. Greenaway, R., Vergers, S., & Mitchell, M. (2015). Digitally enabled learning through Bb+. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. AB:3-AB:4). AB:3 721 Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. AB:4 722 Using interactive multimedia for “flipped lecture” preparation: does it make a difference to student learning? Michelle Moscova Tracey Kuit Karen Fildes School of Medicine School of Biological Sciences School of Medicine University of Wollongong University of Wollongong University of Wollongong Kate Schreiber Teresa Treweek School of Medicine School of Medicine University of Wollongong University of Wollongong Traditional lecture format is largely based on a passive “stand and deliver” model. In this model, students’ first exposure to new material occurs in the lecture. The more challenging aspects of learning – development of critical thinking and application of the content typically occur after the lecture, when the students have less opportunities to discuss content with peers and educators and clarify any misconceptions. “Flipped” or active lecture model reverses this process, with the students being asked to prepare before coming to class. In this model lecture time is utilised as a platform for student discussion and deep engagement with the key concepts. Although teaching methods based on active learning have been well described, they can be perceived as logistically difficult to apply. As described in the literature, some barriers to implementation of active learning lecture models are time required for academics to prepare pre-lecture materials and poor student engagement with the pre- lecture materials. To address this, we have developed an alternative model of multi-disciplinary content delivery through the use of interactive technology for pre-lecture preparation. It consists of a multimedia module designed to prepare students from two different programs – science and medicine – for a “flipped” biochemistry lectures. This allowed academics from different disciplines to collaborate and share the workload, reducing time required to prepare pre-lecture materials. The design of the module allows sections or entire module to be adapted in the future by different academics. The module provided background content for face-to-face lectures using active inquiry-based learning, creating space for discussion to build on core concepts previously explored online. While the content of the module was mostly the same for science and medical students, the “flipped” lectures were specifically developed for each cohort based on the learning objectives for each program. Student acceptance and effectiveness of our model of content delivery was evaluated. Fifty-two second-year medical and 291 first-year science students were surveyed to evaluate students’ perception of the module and the “flipped” lecture. A knowledge-based quiz was also given to the 291 science students one week after the lecture to assess short-term knowledge retention. Results show 75-100% of students felt that the module was easy to understand and 60-89% found it engaging and wanted similar modules available across a variety of topics. However, less than 30% of science students felt that the “flipped” lecture format helped them learn effectively. Science students who did the module and attended the lecture did significantly better on the knowledge quiz compared to students who either attended the lecture or completed the online module. Medical students who completed the module before the lecture were more likely to report feeling prepared for the class discussion and to prefer “flipped” lecture format to traditional lecture compared to students who did not complete the module. In our example, the model combining interactive multimedia module and “flipped lecture” was successfully used across two disciplines to prepare students for active learning lectures. Work is underway to expand and further evaluate this model of content delivery. Moscova, M., Kuit, T., Fildes, K., Schreiber, K., & Treweek, T. (2015). Using interactive multimedia for “flipped lecture” preparation: does it make a difference to student learning? In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. AB:5-AB:5). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. AB:5 723 Attention as skill: Contemplation in online learning environments Ratna Malar Selvaratnam Taylors College The University of Western Australia This is an exploration of the need to cultivate attention as a skill in online learning ecosystems. Taylor’s College is an alternative pathway provider to the University of Western Australia. It is redesigning its diploma program delivery, equivalent to the first year of university, to include non-traditional spaces both online and physical. One of the concerns of online spaces is in equipping the students with the skill of attention control. In online environments it is easy to have reactive attention to stimuli that is not always within one’s control. I suggest having an internal locus of control for attention is a skill to be cultivated to ensure effective learning in online environments. This research looks at the field of contemplative education to see what is offered in this space. Contemplation involves attention and awareness. Keywords: contemplation, mindfulness, online, tertiary education, learning, e- learning, digital Technology is open for many unmindful uses. As an example, problems can emerge when we post content online, without first taking a breath and considering what the implications might be. Other issues of unmindful use of technology include online bullying, disconnection from face-to-face social relationships, developing a passive learning style, and literally training ourselves to be inattentive and unfocused (Hassed & Chambers, 2014). Today our attention is pulled in a multitude of directions through technology and the media. Miller (2014) argues that this results in fragmented consciousness, where we are pushed and pulled by the outside world. However, from contemplative awareness, we see things as they are now. There are already several campus initiatives across America in the contemplative studies space including Brown University’s Contemplative Studies Initiative, Centre for New Designs and Scholarship at Georgetown, Emory Collaborative for Contemplative Studies, Mindful Awareness Research Centre at UCLA, The Center for compassion and Altruism Research and Education at Stanford University, and UCSD Center for Mindfulness (The Association for Contemplative Mind in Higher Education, n.d.). In Australia There has been some work in UNSW in contemplative studies through a symposium on Contemplative Education at UNSW, May 15th, 2014 (University of New South Wales, n.d.). Additionally, Australia now has its first government accredited tertiary institution grounded in Buddhist values and wisdom. Incorporating the mindfulness arm of contemplation, it organised the International Conference on Mindfulness, Education and Transformation 2014 (Nan Tien Institute, n.d.). These are leading institutions which provide both face-to-face and online learning environments and have considerations for the students’ use of attention especially with regards to completion of the programs. Digital learning ecosystems give rise to the need for enhanced digital contemplative methods (Bush, 2010) which in turn can assist in enhancing attention. There is some effort to outline ways in which the body, social isolation, identity and aesthetics in online education can be approached mindfully. Douglass (2007) suggests that a way to investigate whether the format really serves students or contributes to their isolation is by honestly assessing our own relationship with technology. Some have gone as far as referring to contemplation’s esoteric roots. In an asynchronous environment, where students can participate on their own schedules, it is possible to require every student to participate in each session, which is more difficult in synchronous settings (Coburn, 2013). In online learning, there is a built-in opportunity for reflectiveness, although that opportunity needs to be cultivated. Educators are seeking to increase attention, contemplation, wisdom, and compassion by using the very digital media that seems to be decreasing these capacities (Barbezat & Bush, 2014). AB:6 724 References Barbezat, D., & Bush, M. (2014). Contemplative Practices in Higher Education. San Francisco: John Wiley & Sons. Bush, M. (2010). Contemplative Higher Education in Contemporary America. Northampton: Association for Contemplative Mind in Higher Education. Coburn, T. (2013). Peak Oil, Peak Water, Peak Education. In L. Sanders (Ed.), Contemplative Studies in Higher Education (pp. 3-12). San Francisco: Wiley Subscription Services. Douglass, L. (2007). Contemplative Online Learning Environments. Journal of Online Education . Hassed, C., & Chambers, R. (2014). Mindful Learning. Wollombi: Exisle. Miller, J. (2014). The Contemplative Practitioner (Second ed.). Toronto: University of Toronto Press. Nan Tien Institute. (n.d.). Retrieved July 15, 2015, from http://www.nantien.edu.au/ The Association for Contemplative Mind in Higher Education. (n.d.). Retrieved July 15, 2015, from www.contemplativemind.org/resources/higher-education/links University of New South Wales. (n.d.). Retrieved July 15, 2015, from http://unsw.academia.edu.au Selvaratnam, R.M. (2015). Attention as skill: Contemplation in online learning environments. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. AB:6-AB:7). Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. AB:7 725 Applying Adaptive Comparative Judgement to videos as an indicator of ‘At Risk’ teaching performance Ruth Elizabeth Geer University of South Australia The quality of students entering initial teacher education programs is being criticised in the media and political arenas with the Teacher Education Ministerial Advisory Group identifying rigorous selection as a key priority. In general, students are chosen on academic achievement which may not necessarily be an accurate measure of who will become successful teachers. Teacher education institutions are seeking effective and sustainable strategies for selecting students who may become exceptional teachers, while also giving reliable judgements on who may require additional support. The first actual opportunity for screening occurs in vivo when preservice teachers undertake their professional experience placements which may be considered as too late. With effective communication seen as an essential skill for good teaching, the production of a short video was seen as a tool to identify preservice teachers who struggle to clearly express themselves and thus may be ‘At Risk’ on their placement. The use of an adaptive comparative judgement ranking system was explored as a possible approach to rank preservice teachers providing an indicator of future teaching performance. Adaptive Comparative Judgement, which was derived from Thurstone’s (1927) discovery that people are unreliable when making absolute judgements but are more dependable for relative judgements, requires educators to compare the work of two students deciding which is better. From many such comparisons a ranking scale is created showing the relative quality of students’ performance. The study describes a number of processes that were used in the research design to explore how a brief video by first year Master of Teaching (Secondary) preservice teachers might be used to identify those who may require additional support to be successful or who may be unsuited to the teaching profession. Preservice teachers were asked to prepare a one and a half minute video of a talk that they would present to 14 year olds on “things you can do to help you with your learning”. Using the Adaptive Comparative Judgement web-based program, a group of six teacher educators compared 83 videos. Many comparisons of the videos to determine their preference for one of two videos in a pair, based on the criteria of who was best at communicating their ideas in a clear, concise and well sequenced manner using appropriate language. A high reliability in excess of 0.93 was achieved with each video being judged between eleven and thirteen times. Since this was the first time the software had been used, the videos were judged by the same educators a second time based on the criteria of whose talk was more likely to engage and interest the learner. The findings are being analysed to determine whether there is any correlation between the two rankings and to ascertain the importance of identifying appropriate criteria. This presentation will discuss whether the findings of this investigative study have been effective in identifying ‘At Risk’ preservice teachers. The video rankings will further be analysed against their performance in their three professional experience placements. Finally, conclusions will be drawn on the predictive use of videos for this particular cohort of preservice teachers in identifying potential teaching performance. Further research on other cohorts is currently underway to further ascertain reliability of videos as an indicator of ‘At Risk’ teaching performance. References Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273–286 Geer, R.E. (2015). Applying Adaptive Comparative Judgement to videos as an indicator of ‘At Risk’ teaching performance. In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. AB:8-AB:9). Note: All published papers are refereed, having undergone a double-blind peer-review process. AB:8 726 The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. AB:9 727 Vertical learning in Agricultural Science: It’s all fun and games until… Emma Yench Sylvia Grommen La Trobe University La Trobe University Australia Australia As with other courses, the Agricultural/Animal Science degree program has been challenged with students seeing their core and elective subjects as individual learning points rather than a coherent set of experiences that are closely linked and ultimately lead to their development as an animal or agricultural scientist. This atomization of the curriculum is particularly apparent in multidisciplinary courses, such as agricultural sciences, with subjects as diverse as physiology, statistics and business. In addition, concepts and skills learned within one particular year level are often considered as ‘completed’ rather than as key knowledge for further development in subsequent years. We therefore wanted to create a learning experience that emphasises the interconnectivity of subjects by providing a real-life, engaging background that will help students look past subjects as ‘individual’ entities and helps them see the ‘bigger picture’ of the agricultural and animal science disciplines. To this end we are developing what we call ‘the vertical learning environment’, a learning environment that ties together the vertical strands that link the curriculum across year levels. This vertical learning strategy is being developed as an integrated suite of online learning experiences for students enrolled in the Bachelor of Animal and Veterinary Biosciences and Bachelor of Agricultural Science degrees. The vertical learning environment will take the form of a simple interactive ‘biosphere’ that contains a number of narratives and characters with which the students can interact in different ways, depending on the subject and the task, and this throughout their course. Teaching staff can use the narratives in this biosphere to support and contextualize their learning activities and assessment tasks, and these narratives can easily be expanded depending on the need. In time, the online biosphere will contain a large repository of linked narratives that students will explore during the course of their degree, making the connections between (and relevance of) subjects more explicit. The online biosphere will incorporate elements of gamification, enabling students to make decisions in any given narrative resulting in adaptations to their experience of that narrative. The objective is to increase student engagement, encourage problem-solving and transferable skills, and subsequently develop more advanced discipline expertise. This discussion paper will report on the development, trial and evaluation of an initial pilot, and invites suggestions and constructive criticism from interested peers. The online biosphere will be trialled in semester 2, 2015, with a prototype biosphere environment consisting of background stories contextualising one narrative and associated problem-solving learning activity relevant to two second year subjects, Animal Nutrition (AGR2AN) and Biochemistry for Agricultural and Animal Sciences (AGR2BAA). These two subjects share a strong disciplinary link that will be emphasised by this common narrative. Evaluating the students’ experience of these narrative linkages will be a precursor for the further development of the vertical learning concept across all year levels of the aforementioned degrees. Yench, E. & Grommen, S. (2015). Vertical learning in Agricultural Science: It’s all fun and games until... In T. Reiners, B.R. von Konsky, D. Gibson, V. Chang, L. Irving, & K. Clarke (Eds.), Globally connected, digitally enabled. Proceedings ascilite 2015 in Perth (pp. AB:10- AB:10. Note: All published papers are refereed, having undergone a double-blind peer-review process. The author(s) assign a Creative Commons by attribution licence enabling others to distribute, remix, tweak, and build upon their work, even commercially, as long as credit is given to the author(s) for the original creation. AB:10 728 ascilite 2015 Reviewers Reem Abu Askar University of Auckland Chie Adachi Deakin University Mifrah Ahmad Sultan Qaboos University, Oman Zakiya Al Nadabi University of Queensland Peter Albion University of Southern Queensland Sakinah Alhadad Griffith University Garry Allan RMIT University, Melbourne Australia Hussain Alyami The University of Auckland Cris Antona Otago Polytechnic Douglas Atkinson Curtin University Simon Atkinson BPP University Kofi Ayebi-Arthur University of Canterbury Karin Barac Griffith University Mark Bassett Auckland University of Technology Pierre Benckendorff The University of Queensland Sue Bennett University of Wollongong Trevor Billany Charles Darwin University James Birt Bond University Carina Bossu University of Tasmania Scott Bradey James Cook University Rachel Byars Otago Polytechnic Leanne Cameron Australian Catholic University Chris Campbell The University of Queensland Gail Casey Deakin University, Geelong Mark Caukill Nelson Marlborough Institute of Technology Vanessa Chang Curtin University WeiFong Cheng Tunku Abdul Rahman University College, Malaysia Jasmine Cheng UTS:Insearch Frederick Chew The Australian National University Rashmi Chhetri University of Houston Lyn Fung Choy National Institute of Education Tom Clark Otago Polytechnic Thomas Cochrane Auckland University of Technology Meg Colasante RMIT University, Melbourne Linda Corrin University of Melbourne Michael Cowling CQUniversity Australia Iwona Czaplinski Queensland University of Technology Kun Dai The University of Queensland Barney Dalgarno Charles Sturt University Kashmira Dave University of Western Sydney, Parramatta 729 ascilite 2015 Reviewers Amanda Davies Charles Sturt University Irwin DeVries Thompson Rivers University Eva Dobozy Curtin University Tracy Douglas University of Tasmania Linda Fang Temasek Polytechnic Helen Farley University of Southern Queensland Victor Fester Unitec Institute of Technology Kim Flintoff Curtin University Katharina Freund Australian National University Stanley Frielick Auckland University of Technology Steve Gallagher University of Otago Timna Garnett Flinders University Dragan Gasevic University of Edinburgh Belma Gaukrodger Nelson Marlborough Institute of Technology Megan Gibbons Otago Polytechnic David Gibson Curtin University Sue Gregory University of New England Tim Griffin University of Western Sydney paul gruba The University of Melbourne Eugene Gvozdenko The University of Melbourne Patrick Halloran Curtin University Maggie Hartnett Massey University Michael Henderson Monash University Vincent Horner University of South Africa, Pretoria Dirk Ifenthaler Curtin University Leah Irving Curtin University Tomayess Issa Curtin University Martin Jenkins Coventry University Deborah Jones The University of Melbourne Hazel Jones University of Southern Queensland Anthony Jones The University of Melbourne Doris Jung University of Waikato Jo-Anne Kelder University of Tasmania Shannon Kennedy-Clark University of Notre Dame Wai Jing Kwok Multimedia University, Malaysia David Kwok Republic Polytechnic, Singapore Jo Lander University of Sydney Romy Lawson University of Wollongong Yik Sheng Lee Tunku Abdul Rahman University College, Malaysia Sabrina Leone Università Politecnica delle Marche, Italy Fui-Theng Leow INTI International University, Malaysia Danny Liu Macquarie University Jason Lodge The University of Melbourne Swee-Kin Loke University of Otago 730 ascilite 2015 Reviewers Kulari Lokuge Dona Swinburne University of Technology, Melbourne Susie Macfarlane Deakin University Stephen Marshall Victoria University of Wellington Romana Martin Curtin University Yvonne Masters University of New England Jenny McDonald University of Otago Mark McGuire University of Otago David McMeekin Curtin University Salome Meyer EIT Institute of Technology, Hawke's Bay Carol Miles University of Newcastle Aidrina Mohamed Sofiadin Curtin University Heng Ngee Mok Singapore Management University Michelle Moscova University of Wollongong Adrienne Moyle The University of Auckland Ariel Muvhunzwi Zimbabwe Library Association Tse Kian Neo Multimedia University, Malaysia Mai Neo Multimedia University, Malaysia Maria Northcote Avondale College of Higher Education Mark O'Rourke Deakin University Mariya Pachman Macquarie university Abelardo Pardo The University of Sydney Helen Partridge University of Southern Queensland Robyn Philip Queensland University of Technology Rob Phillips Murdoch University Oleksandra Poquet University of South Australia David Porter University of Wollongong Petrea Redmond University of Southern Queensland Torsten Reiners Curtin University Emanuel Reiterer Curtin University Xiang Ren University of Southern Queensland Pauline Roberts Edith Cowan University Andrew Rowatt Massey University Carol Russell Western Sydney University Michael Sankey University of Southern Queensland Mark Schier Swinburne University of Technology Ratna Selvaratnam The University of Western Australia Simon Smith University of South Australia Hugh Soord Curtin Univeristy Erica Southgate The University of Newcastle Lee Stoner Massey University Kenneth Strang State University of New York Jennie Swann Auckland University of Technology Marko Teras Curtin University Danielle Teychenne Deakin University 731 ascilite 2015 Reviewers Lisa Thomas University of Wollongong Sarah Thorneycroft University of New England Susan Tull University of Canterbury Deborah Veness Australian National University Elena Verezub Swinburne University of Technology Susie Vergers University of the Sunshine Coast Brian von Konsky Curtin University Nick Wallingford Bay of Plenty Polytechnic, New Zealand Debbi Weaver La Trobe University brian webby University of South Australia Simon Welsh Charles Sturt University Rachel Whitsed Charles Sturt University Yvonne Wisbey Australian National University College of Law Lincoln Wood Auckland University of Technology Carolyn Woodley Charles Sturt University Emma Yench La Trobe University Nau Zaung Curtin Unversity Yu Zhao The University of Sydney 732 ascilite 2015 Reviewers