(PDF) Innovative Model, Tools, and Learning Environments to Promote Active Learning for Undergraduates in Computational Science

Journal of Computational Science Education Volume 8, Issue 3 Innovative Model, Tools, and Learning Environments to Promote Active Learning for Undergraduates in Computational Science & Engineering Hong Liu, Andrei Ludu, Michael Spector Matthew Ikle Jerry Klein North Texas University, Adams State University Embry-Riddle Aeronautical University Denton, TX 76201 Alamosa CO 81102 Daytona Beach, FL 32114 US United States United States +1 386 226 7741 +1 940 369 5070 +1 719 588 4487 {liuho, ludua, kleinj7}@erau.edu

ABSTRACT This paper presents an innovative hybrid learning model as well General Terms as the tools, resources, and learning environment to promote Experimentation, Verification. active learning for both face-to-face students and online students. Most small universities in the United States lack adequate Keywords resources and cost justifiable enrollments to offer Computational Hybrid Learning, Cyberlearning, Educational Technology, Science and Engineering (CSE) courses. The goal of the project MOOC, Model-based Learning, and Computational Science & was to find an effective and affordable model for small Engineering Education. universities to prepare underserved students with marketable analytical skills in CSE. As the primary outcome, the project created a cluster of collaborating institutions that combined 1. INTRODUCTION The digital technology in the 21st century is characterized by students into common classes and used cyberlearning learning omnipresent smart devices and ubiquitous computing that enables tools to deliver and manage instruction. The instrumental tools for computing to occur anytime and anywhere. This contributes to big educational technologies included Smart Podium, digital data challenges, increasing complexity and rapid changes in projector, teleconference systems such as AdobeConnect, auto technologies. Consequently, marketable skills for technical tracking camera and high quality audio in both local and remote classrooms. As an innovative active learning environment, an careers emerge and rapidly change. To harness complex R&D process was used to provide a coherent framework for technologies and make sense of the big data, undergraduate students majoring in STEM (Science, Technology, Engineering, designing instruction and assessing learning. Course design and Mathematics) need to learn how to model the associated centered on model-based learning which proposes that students problems in mathematical formalisms and leverage the computing learn complex content by elaborating on their mental model, resources to simulate the problems. In [18], Levy summarized one developing a conceptual model, refining a mathematical model, of three major educational initiatives of a SIAM (Society for and conducting experiments to validate and revise their Industrial and Applied Mathematics) as follows: conceptual and mathematical models. A wave lab and underwater robotics lab were used to facilitate the experimental components “The Modeling Across the Curriculum, was built around the idea of hands-on research projects. Course delivery included that modeling can build many job skills students need and can be interactive live online help sessions, immediate feedback to an important educational tool at not only the secondary and students, peer support, and teamwork which were crucial for undergraduate levels, but throughout the educational experience”. student success. Another key feature of instruction of the project was using emerging technologies such as HIMATT (Highly The SIAM-NSF-ASA Workshop on Modeling Cross the integrated model assessment technology and tools) [11] to Curriculum II [13, 14, and 20] made the following two evaluate how students think through and model complex, ill- recommendations to math teachers and STEM education policy defined and ill-structured realistic problems. makers: Building a pipeline for K16 education in mathematical modeling and connecting math to reality. Computational Science and Engineering (CSE) is an emerging multidisciplinary field of Categories and Subject Descriptors study that incorporates both endeavors recommended above. CSE K.3.1 [Computer Uses in Education]: Collaborative learning, focuses on the integration of knowledge and methodologies from Distance learning. K.3.2 [Computer and Information Science computer science, applied mathematics, engineering, and science Education]: Computer science education, Curriculum. to solve real-world problems. CSE provides the critical mathematical modeling skills and data analytical skills that apply Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are to all STEM fields. Therefore, it is critical that CSE courses and not made or distributed for profit or commercial advantage and that curricula are a viable option for every undergraduate STEM copies bear this notice and the full citation on the first page. To copy major. [19]. Based on the literature above, the authors of this otherwise, or republish, to post on servers or to redistribute to lists, project identified the following two courses as the corner stones of requires prior specific permission and/or a fee. Copyright ãJOCSE, a a CSE curriculum: Mathematical Modeling and Simulation supported publication of the Shodor Education Foundation Inc. (MMS) and Data Mining and Visualization (DMV). DOI: https://doi.org/10.22369/issn.2153-4136/8/3/2 December 2017 ISSN 2153-4136 11 Volume 8, Issue 3 Journal of Computational Science Education Many non-research focused higher-education institutions, 2.2 Cyberlearning Technology to Facilitate including most minority-serving and small institutions that the authors of this paper are affiliated, lack the necessary resources Tri-located Course Deliverance and minimal enrollments to justify offering such CSE courses. To Each of the three courses was taught by a professor in the help provide CSE educational opportunities for students at small classroom at one location with physically present students while a institutions, a proof-of-concept project was funded by National small number of students in the other two universities attended the Science Foundation (NSF). The project created a cluster of same class in remote classrooms using live two-way collaborating institutions that combined students into common communications. The approach of this project is actually a hybrid classes and used cyberlearning technologies [1] to deliver and learning environment that combines traditional classroom and manage instruction. The primary goal of the project was to offer cyberlearning with live interactions between the classmates as three cyberlearning courses in CSE through a coalition of three well as students and instructors. small universities by sharing resources and pooling students. Another goal was to use short-term summer projects to reinforce The courses at ERAU were delivered at a state-of-art student learning and assess student ability to solve real-world Teleconference Classroom. The major hardware devices are: (1) problems. The following three key activities were implemented: HD voice audio system with one microphone sitting in front of the (a) establish a multi-institution coalition that collaboratively podium and two mics hanging on the ceiling of the classroom and provides CSE courses and projects by combining students from multiple well positioned speakers, (2) an auto video tracking different colleges into a single class and leveraging each college's camera that was installed on the back of the classroom to video capabilities, (b) provide general CSE courses organized by the instructor, (3) a SmartPodium, which integrated the computational methods (e.g. mathematical modeling and whiteboard with the computer screen by SmartTechnology simulation) rather than domain specific courses such as (home.smarttech.com) on the front podium, and a digital projector computational biology; and (c) use cyberlearning technologies to installed in the ceiling of the classroom. Besides standard deliver courses. This project provided a feasible model and software settings, the computer in this classroom installed the innovative configuration of educational technology to AdobeConnect teleconferencing system to support two-way live significantly increase student participation in CSE and a cost communications to the students in the remote classroom. Among effective method to renew courses and curricula. all the devices mentioned above, the most important one is the reliable teleconference system that makes the two-way communication possible. The second most critical instrumental 2. STRATEGY AND IMPLEMENTATION factor is the quality of the audio system. Poor quality mics and This section presents the implementation details of the hybrid speakers can be very annoying to the students in remote learning model, innovative educational toolset, new MOOCs campuses. Before a new semester starts, a new instructor for such (Massive Open Online Course) in CSE, and the active learning a synchronous online learning course should practice setting up environment for both face-to-face and online students. and testing all relevant systems under the supervision of an experienced member of the IT staff. 2.1 Infrastructure of Coalition of Universities In our tri-located class, we have a local class and two remote and Hybrid Cyberlearning Model classes. Two modes of the teleconference features most frequently The infrastructure of the coalition was built on the contract that all used were the video mode that uses the full screen to show the participating institutions share the workload, resources, and videos of local and all remote classes and the presentation mode benefits equally through faculty partnership over a period of two that use the whole screen to show the lecture notes or other years. The Coalition of Universities includes two campuses of software documents of the presenter. The instructor has the ability Embry-Riddle Aeronautical University (ERAU) - Daytona Beach to promote a participant in a remote campus as the presenter. In (DB) Campus in Florida and Prescott Campus in Arizona, and presentation mode, about three quarters of the screen of the Adams State University (ASU) in Colorado. The authors of this remote classes shares the screen of the presenter and the right paper who are associated with the coalition of universities took panel of the other quarter screen shows the small size videos of turns developing, reviewing and teaching the three courses in the instructors and other classes. The CSE courses require the Mathematical Modeling (I and II) and Data Mining. In instructors to use the presentation mode most of the time. In this particularly, Liu at ERAU developed the first draft of MMS case, students in remote campuses are invisible to the instructor course, Ikle at ASU reviewed, revised and adapted Liu's course unless they ask questions. Because the online students do not have for ASU one year later. Meanwhile, Ikle developed the first draft the physical presence in the classroom, students are reluctant to of DMV, then, Liu reviewed, revised and adapted the course for ask questions and consequently can be easily neglected by the ERAU. While Liu offered the first course in MMS in spring 2014, inexperienced online teacher. Therefore, instructors of such a Ikle sat with his ASU students and monitored the course delivery hybrid synchronous learning course need to learn how to online. Reciprocally, when Ikle at ASU offered the DMV courses periodically query the online students in remote campuses. An in fall 2014, Liu sat with his student at DB Campus of ERAU effective technique we learned was to set a laptop near our online. Moreover, Liu and Ikle exchanged courses to teach in the SmartPodium which in effect turns the local class into an next round (e.g. Ikle taught MMS in ASU in spring 2015 and Liu additional virtual class. The laptop is used to connect the second taught DMV in fall 2015). Therefore, Liu and Ikle both served as video camera shooting towards to physical present students so that the co-developers and co-teachers of each other's course. it enables students in the remote classes to see the students in the Consequently, both universities gained a new course in their local classroom. While the main computer connected with Smart curricula that were originally developed by a peer university in the Podium is set in presentation mode, we can set the laptop in video coalition. This model not only provided students access to new mode to monitor the facial expressions of the remote students. In CSE courses in each year but also provide a cost-effective method this mode, the screen of our virtual classroom is equally divided for faculty development and curriculum enrichment. into four sections, one for the instructor shot by the auto tracking 12 ISSN 2153-4136 December 2017 Journal of Computational Science Education Volume 8, Issue 3 camera, the second one is the students in local classroom shot by ability. The workshops were based on the ACE program: the second camera, and the other two display the two classes of Analysis, Computation, and Experiment [5]. remote students. We will use a couple of examples from the MMS to illustrate how the course objectives were implemented and accomplished. MMS 2.3 Innovative Learning Environment (http://modelsim.wordpress.com ) was designed for college An R&D process was used to provide a coherent framework for sophomore and junior students who have taken multivariate designing instruction and assessing learning in which the Calculus and are familiar with at least one programming language. instructional and assessment methods were aligned with a The goal of the course was to learn how to use the advanced common idea: Model-based learning and reasoning. Besides mathematics language such as matrix algebra and calculus as well traditional grading, the project used emerging technologies such as software tools to solve real-world problems. The topics of the as HIMATT to evaluate how students think through and model course covered broad interdisciplinary problems whose solutions complex, ill-defined and ill-structured realistic problems [16]. heavily depended on mathematical modeling and simulation. All courses employed model-based learning and reasoning from More specifically, objectives were to first principles and were aligned with SIAM's CSE educational 1. Introduce the major categories of mathematical models as well goals [19]: as their modeling languages and tools 1. Students can construct qualitative models using first principles 2. Expose students to a broad variety of real-world applications of of the domain. computational mathematics 2. Students can translate their qualitative models to mathematical 3. Train students how to follow mathematical modeling process models which require them to understand the underlying to conceptualize problems, validate their models and verify equations governing first principles. their solutions 3. Students can use mathematical modeling and computational 4. Improve students' capability to make judicious tradeoffs in software tools to create their mathematical models, design their modeling assumptions to abstract complex problems data collection systems, and interpret experimental data. 5. Provide students with hands-on experience in the use of The three cyberlearning courses were: (1) Mathematical Modeling computational software tools such as MATLAB, NetLogo, and Simulation-1 (MMS-I). This course was often titled STELLA, etc. to model and simulate mathematical problems, Introduction to Computational Science and is included in most and present visual representations of the problem space as CSE curricula. It was first taught at ERAU-Daytona Beach. (2) well as alternative solutions. Mathematical Modeling and Simulation-2 (MMS-II). This course 6. Provide students with teamwork experience to solve problems was a revised version of the computational physics course and beyond the scope of textbook exercises and typically beyond was first taught at ERAU-Prescott. This course provided a test the scope of effort for one person. case of modifying a domain-specific CSE course and making it The course includes five modules: 1: Model Classification and non-domain specific: structuring the course by computational Modeling Methodology, 2, Matrix Algebra, 3: Data and Error method methods rather than by types of physics problems. (3) Models, 4: Agent-based Modeling, and 5: Modeling System Data Mining and Visualization (DMV). This course reflected a Dynamics. The system engineering modeling methodology in change in CSE degree programs [2]. Data explosion is a major module 1 ([7]) emphasizes (1) building of conceptual models trend and the resulting challenges have created tremendous before the mathematical models to increase traceability of model employment opportunities in data-driven business, scientific assumptions and first principles, (2) separating concerns and research, and counterterrorism. All courses consisted of modules refining models iteratively to divide and conquer complexity, and organized by computational method rather than organized by (3) verification and validation (V&V) based on empirical data and applications. When the organizational unit was based on the mathematical analysis. The other four modules were divided into method of computation rather than the domain of application, it three units and each unit starts with an interesting application and made the development of course modules for distributed teaching develops a core concept to model. For example, to illustrate how much easier. Also, modularization provided the flexibility needed we designed the course to develop deep learning, but starting at to adapt courses to the particular interests and needs of students. low thresholds, the concept of Eigenvectors was revisited four Each module included a recorded demonstration and lecture and times with increasing depths at each time. In the first unit, we interactive instructional activities. adapted a module about the Leslie transition matrix and its The project has been committed to course-based research applications to Biology from a paper by A. Shiflet and G. Shiflet experiences (CURE) which connect math with reality [14]. The [12]. The Eigenvector was informally and intuitively introduced primary instructional strategy of the project was for students to as the stable population distributions of the Leslie population work in study teams solving problems using online resources model. In the second unit, we inspected the direction changes of created or selected for the particular topic. Since students the vectors under geometric transformations such as mirror represented multiple academic disciplines, each module included reflection and projection. The students found that most problem sets and supporting materials for the different domains. transformations have some invariant directions except rotational Each module consists of instruction on (1) a prototype problem, matrices. Therefore, the concept of Eigenvectors was identified (2) tools for creating a conceptual model, (3) mathematical informally again. We provided its formal definition in the third modeling tools, and (4) resources for each of the various academic unit and adapted an example of its application to the Google page disciplines. Each module was designed by clarifying and defining ranking problem from a popular paper [3] by Bryan and Leise. In objectives and selecting or developing problems and examples. the fifth module, the Eigenvectors and their geometric After completing the team course projects the students in remote interpretations were identified again from the solutions of the campuses had the opportunities to participate in a two-week long initial value problem of a linear ODE system. summer research workshop. The purposes of the workshop were The third module demonstrates the uncertainty and inevitable to reinforce learning and assess the students' problem-solving errors for modeling real world applications. Instead of seeking December 2017 ISSN 2153-4136 13 Volume 8, Issue 3 Journal of Computational Science Education exact solutions, the students learned that when dealing with real comparison with fewer experienced persons, and that pattern held world applications it is more practical to search for optimal up in this study. solutions that minimize the estimated errors based on observable data. Random variables, Monte Carlo methods, Markov chain Student feedback included one survey in the beginning and two were introduced in the first unit. Students learned how to use the evaluations in the middle and the end of the semester. The survey had 21 questions about the students' academic and demographic stochastic transition matrices to model and simulate the uncertainty of outcomes for real-world financial and business background as well as their beliefs about CSE courses, problems. Students also discovered that great differences of cyberlearning vs face-to-face learning, and team work. The two outcomes can be significantly mitigated after the corresponding evaluations consisted of four items focusing on assessing the model was simulated thousands of times. The second unit focused students learning outcomes and the need for improvement: (1) on multivariate data fitting techniques and applications of data- List the primary ways your learning has been enhanced in this driven prediction models. The third unit presented the concepts of course, (2) List the primary ways that your learning has been error models, linear and nonlinear Kalman filters, and their hindered in the course, (3) List the primary ways you could enhance your own learning in the course, and (4) Some applications to GPS. MATLAB was used to simulate how the Kalman filters could help to pinpoint our positions in feet range recommendations. An independent evaluator conducted the error by using the signals from 4-6 semi-geosynchronous GPS evaluations, summarized the data, and then, reported to both the satellites that were 25,000 km above the earth. Liu learned about instructor and the external evaluator. The summary report this application at a conference in 2005 which was the major included the samples and frequencies of positive and negative inspiration to initiate the MMS course [10]. It takes 27 class hours feedbacks as well as recommendations. It is difficult to quantify to cover all lessons of the first three modules. The course included the success of this type of data and summarize the feedback of all a mid-term test and an individual conceptual modeling project 8 classes. It is obvious, however, that positive feedback comments were dominant, and constructive criticism was evident; in was due two weeks later. In the last month of the course, the focus was shifted to team projects and the instructor met each team addition, there was a very low drop/fail rates (0 to 10%) of all separately at least once a week to check their progress and answer classes comparing with other math courses at ERAU (10%-30%). questions. The instructor only gave lectures once a week to give Indeed, the evaluations and formative assessments were more an overview to both agent-based modeling and system dynamical helpful in making timely instructional adjustments than measuring modeling. The students were encouraged, but not mandated to success. For example, in the middle term evaluation of MMS I in learn the last two modules online based on the needs of their team the spring of 2014, the remote students complained that the poor projects. audio quality hindered their learning. To respond to this issue, the instructor obtained advice from IT experts and purchased three sets of the high-end quality audio and mic systems called Konftel. 2.4 Learning Assessment In two weeks, the desktop computer based audio systems in all Our summative learning assessments included: (1) student three classrooms were all replaced with the Konftel systems. As feedbacks and survey data, (2) peer reviews of the online another example, we observed that students paid little attention to published course materials and student paper work samples by the lectures that seemed irrelevant to their own team projects, external experts such as the external evaluator Dr. Michael which were assigned in the last month of the course. Therefore, Spector and instructional designer Dr. Jerry Klein, and (3) student we changed the course delivery method for the last month research outcomes from coauthored publications and accordingly in next term: Instead of lecturing to the whole class presentations. We used traditional assessment instruments to three hours per week, the class met once a week to address measure student learning including tests that required students to logistics and common issues and an hour mandatory meeting with explain and predict events as well as rubrics to grade student the instructor was scheduled with each team to report on progress, conceptual and mathematical models. However, knowledge-based obstacles, and work plans for the next week. Consequently, the tests are not sufficient. These instruments do not measure how an students were more engaged and prepared for their meetings with individual thinks or might approach other projects and problems the teacher. [4]. We created assessment instruments using the HIMATT learning assessment methods and tools [11] and 16]. HIMATT is Team projects assigned for the MMS and DMV courses are a validated technology that essentially captures the student's presented here to illustrate how course objectives were met. In the conceptualization or model of a complex situation and compares it last month of a semester, the emphasis shifted to team projects. with a reference model that could be an expert's model, to assess The intention of projects was to cultivate students' ability to solve progress towards expertise, or models of that student at an earlier real-world problems. Project problems and grading rubrics are time to assess progress from previous states of complex thinking. similar to those of the Mathematical Contest in Modeling (MCM, In order to implement this technology, the project faculty created https://www.comap.com/ ) sponsored by COMAP (Consortium problem scenarios that were open-ended and not fully specified so for Mathematics and its Applications). After the 10th week of the as to require students to think about the nature of the problem and semester, the instructor provided two or more open-ended possible alternative solution approaches but not so detailed that problems for their team projects. One is a continuous modeling students could actually provide concrete solutions. Four questions problem and the other is a discrete modeling problem. In were then asked of students and experts: (1) what are the key particular, the rubrics of the project grading included: 20% paper factors influencing this problem situation, (2) describe the nature presentation, 10% for oral presentation, 20% for conceptual and of each factor, (3) how are the factors related, (4) describe the math modeling, 10% for simulation, 20% for mathematical nature of each relationship among the factors. HIMATT uses analysis, and 20% for verification and validation of the model. analyzing pairs of resulting conceptualizations with regard to One MMS problem was to model and simulate a safe landing gear surface, structural and semantic similarity. In general terms, for aero robots. Students had two weeks to build conceptual experts tend to see more relationships among factors and tend to models that captured the major factors and their qualitative identify key nodes or concepts that influence the situation in relationships. The conceptual model required students to answer 14 ISSN 2153-4136 December 2017 Journal of Computational Science Education Volume 8, Issue 3 the four questions mentioned listed above. A month before the project was to use the biomass data collected around the artificial end of the course, teams of 4 students were assigned to build the reefs provided by the Beach Safety Office of Volusia County to mathematical model and simulate for the problem using Stella. predict the healthiness and effectiveness of the artificial reefs The won team of students had the opportunity to participate in a towards to marine ecosystem preservation. The team continued to paid summer research workshop and conduct the experiments in improve the data mining results in the summer with support from ERAU's Wave Lab (see Figure 1, safe landing gear for weather the Prepare Industrial Career Math mini-grant. The project was balloon problem). An example of discrete modeling is students in co-mentored by Liu and a staff member at the Beach Safety MMS were asked to model and simulate the emergent evacuation Office. The Project was presented as a poster at the SIAM Annual of the vulnerable residents in a coastal city when a tsunami hits Conference in 2016. Because of this project, one student found an the city. The basic geographic information such as the elevation internship opportunity and part time job in Volusia County and the population density of each city district were given in a government. In summary, the strategies we used to engage student data sheet. The territories of the districts are approximated as learning were (1) teach mathematical concepts using relevant grids of uniform sizes and the streets are horizontal and vertically application contexts and (2) provide team research projects that lined up to separate districts. The students were given 40 school will facilitate the development of marketable problem-solving buses and asked to select only one exclusively used street from skills. east to west to evacuate the vulnerable residents who could not drive or had no cars. Two teams chose this project and both used 3. OUTCOMES the agent-based modeling approach for the problem. One team As the project progressed, the following new educational wrote their own Java program, and the other team chose to use resources were created: (1) Multimedia MOOC materials for three NetLogo to model and simulate their solutions. While EXCELL, courses in Mathematical Modeling and Data Mining, (2) MATLAB, Stella and NetLogo were frequently used in case Documents for formative and summative learning assessments; studies and students learned how to read the programs and and (3) Five peer-reviewed research papers coauthored by the diagrams, the instructor did not teach students how to program in undergraduate participants of three summer research workshops those languages and tools. Instead, free online tutorials were from 2013-2015. provided to students so they could learn as needed. Teams were assigned based on complementary knowledge and skills so that Each college contributed one faculty member and one course each team had at least one student proficient in programming. An resulting in each university in the coalition having three new observable outcome was that students often demonstrated their courses. All courses were thoroughly reviewed by peer instructors ability of learning-on-demand when inspired by interesting and upgraded multiple times providing each university three high problems ([6]). For example, one student who used NetLogo to quality innovative cyberlearning courses. The students in each program their evacuation model learned how to use NetLogo from campus have more course choices while the campuses save costs scratch and did an excellent job in less than a month. He is by not having faculty members teaching extra courses: We simply continuing his project and intends to submit a paper for shared resources and pooled students to make the class size reach publication. the ideal number of students. The feedback collected formally and informally from students indicated that they benefited from the The DMV (http://datamininedvis.wordpress.com/) instructional multimedia course website, which allowed them to learn the design and course delivery strategy are similar to that of the material at their own pace. They read textual materials for the MMS. Since DMV is a course that is offered by most universities, theory and concepts and watched the videos for the examples. the content selection was less challenging than the MMS course. Three course websites were built using WordPress technology and Two aspects that made our DMV course differently were the team contributed to the MOOC. projects: (1) The teams sometimes had to collaborate with students in a remote campus and (2) some projects have an Three two weeks workshops were offered for a total of 18 industrial co-mentor in addition to the instructor. We also students from three universities in summer 2013-2015. Five allocated the last month of the course for the team projects. The students in ASU and Prescott campus were funded by the NSF instructor selected the won team project and encouraged the grant to participate in the summer workshops at Daytona Beach. students to continue the project in order to earn internship or job All students started their mandatory team projects six weeks opportunities. Of particular interests are two projects that were before they completed their CSE courses in spring semesters. The successfully completed in spring 2015 and spring 2016. The first focus of course-based research experiences (CURE) is on the project involved data mining for profiling incoming students and modeling, computation, simulation and analysis of open-ended predicting student retention rate based on the authentic training application problems. All 18 students chose to continue their dataset provided by the Office of Student Success and Retention research projects in their CSE courses but shifted their focus at ERAU. A team of four students from ERAU Prescott selected towards model validation based on authentic data. The three teams the project and did an excellent job when they took the course of students used the ACE methodology [5] to model and analyze offered at the Daytona Beach campus in spring term 2015. In physics applications such as safe landing gears of weather balloon middle May of 2015, the four students were invited to Daytona payload, buoy motions and underwater light scattering patterns in Beach in the summer research workshop and they were co- a Wave Tank. They obtained their data from hands-on mentored by Liu and Mr. Steven Lehr who had over 15 years of experiments in the Wave Lab and the Leverage Robotics Lab. A industrial data analytics experiences. After the research outcomes team of five students from the DMV course used data from the were presented to administrators at Office of Student Success and ERAU admission and registration office to predict student Retention, they organized a committee to investigate the impact of retention and attrition. The course-based researcher and summer gateway courses such as pre-calculus or calculus to the student research workshops resulted in four student co-authored papers [6, retentions in 2016. The research results [14] was published in an 8, 9, and 17], three conference presentations, and two conference IEEE conference proceeding in 2016. The second project was presentations. Figures 1-4 shows the five student coauthors of [9] assigned to two students as a team project in spring 2016. The from ASU designing and testing a safe landing system for weather balloon payload in summer 2015. December 2017 ISSN 2153-4136 15 Volume 8, Issue 3 Journal of Computational Science Education Because students of the cyberlearning courses represented more game of "Tanks" in MATLAB. The students especially enjoyed diverse STEM majors than the typical students at any single the team projects on the application of authentic weather data, campus and course assessment approaches were quite different, genetic data, public health data, real-estate data, and student there was not a proper control group to compare learning retention data. outcomes quantitatively. In addition, class sizes were between 12- Although the student evaluation metadata in section 2.4 had 20 students from three campuses and consequently sample sizes were too small to draw statistically significant conclusions. limited value in helping us to accurately measure our success, the Suggested by our external evaluator Spector, we compared the numerous positive feedbacks including the constructive criticisms learning outcomes of MMS with an Ordinary Differential clearly indicated that all three courses were effectively delivered Equation Course (ODE, MA345 in ERAU), which was one of and improved in the second and third round of offerings. Many of most similar courses taught by the same instructor of the MMS these student comments were included in the final project course at the same time. The two courses were similar in these evaluation reported by the external evaluator to NSF. In the final aspects, (1) same prerequisite, (2) similar contents, matrices project evaluation to NSF, Spector presented the following evaluation summary: algebra and system ODE, (3) similar small class sizes (17 vs 29 students). Besides the delivery method (MMS used cyberlearning “In terms of the three primary goals of this project (establish an & blended learning, ODE used traditional face-to-face lectures), initial cluster of three collaborating colleges; create and offer a the other major differences between MMS and ODE are: (1) ODE computational science and engineering program at collaborating students were more homogeneously from physics and engineering colleges; create the infrastructure and processes to extend the majors and the students in MMS came from more diversified collaboration cluster), this program has been successful and all STEM majors including biology and meteorology majors, (2) three goals have been achieved. …… Problems encountered have MMS is a problem-based learning course with model-based been addressed and refinements made. Students taking the instructional design and assessments, query guided learning, collaboration courses at a distance are performing as well as teamwork, etc., the ODE course was taught and tested in students in comparable classes as indicated by the grades conventional methods, and (3) the focus of the MMS course was awarded. Understanding of complex computational engineering on the depth of their understanding of the key concepts and problems is occurring as shown by an analysis of problem problem solving ability with the use of computational tools while conceptualizations and solutions previously reported. Interest is students in ODE course covered a broad range of content and especially high as shown by voluntary participation in the two solved several types of ODEs by hand. Comparing the student summer workshops, which should be better funded if such non- evaluations of the two courses showed that the students in MMS formal but highly productive efforts are to be continued.” were much more motivated. They gained more confidence from the MMS course to solve relevant problems to their careers by using system engineering methodology and computational tools. Grades of ODE in spring 2014, 7 A's, 7 B's, 10 C's. 2 F's, and 3 W's, which is a typical grade distribution at ERAU. The grades awarded for MMS in the same semester were 6 A's, 6 B's and 4 C's, and 1 W. Though the grades of the courses might not tell how much the students really learned, we believe that low attrition and failure rate of the classes is a good indicator of the course success. We offered DMV three times, the first time by IKle, the second time by Liu, and third time by Liu and Lehr. Not a single student did withdraw from the class. This is very rare for any other mathematics course: The typical withdrawal rate of similar level mathematics courses are 10-20% at ERAU. Our student feedback showed that most of them loved the courses because of the relevant and marketable skills they gained for industrial careers. The withdrawal (W) or failure (D or F) rates of the three MMS courses were very low: one to two students in each course. Spector examined all the student evaluations reports presented in Figure 1. Students are preparing the safe landing system section 2.4 and also evaluated online course materials, sample student test papers, project reports, etc. for each course. He communicated with the instructors frequently so that the student concerns were addressed timely and the project efforts were aligned with the major goal of these courses: fostering student problem-solving ability and promoting deep learning. As a result, the students in MMS courses learned how to use the system engineering modeling methodology and procedures to translate real-world problems into mathematical models. Students also gained hands-on experience in using software tools such as MATLAB and STELLA to model and simulate real-world projects. A set of sample projects of the MMV-II indicates that students developed deep learning including: (1) A Model of the Rings of Saturn illustrating the appearance of the Cassini Division for certain parameters of the shepherd moon(s) and (2) Writing a 16 ISSN 2153-4136 December 2017 Journal of Computational Science Education Volume 8, Issue 3 In addition to the MOOC websites, this project resulted in the five publications [6, 8, 9, 15, and 17] (undergraduate coauthors are marked by asterisks). PIs conducted the following personal dissemination events: Liu and A. Shiflet co-chaired a miniSymposium on Educational Innovations in CSE Education in the SIAM Annual Conference, July 2014 in Chicago. Liu and Ludu Co-Chaired a miniSymposium in SIAM SEA Conference 2014 in Melbourne, FL, 2014. Spector and Liu Co-Chaired a miniSymposium on Cyberlearning Technology and Deep Learning in a CSE Conference in March 2015 in Salt Lake City, Utah. In addition, three students gave a presentation at the 2015 Kappa Mu Epsilon National Convention. The social benefit to students included providing equal learning opportunities to under-represented students: Adams State University is a federally designated Hispanic Serving Institution (HSI). The summer research workshops brought minority students Figure 2. Sensor and Arduino board of the safe landing gear from Adams State and students from the Prescott campus in Arizona to meet the students from the Daytona Beach campus. These events not only facilitated student collaboration on research, but also helped them build friendships and develop the mutual understanding of other students from different cultural and socioeconomic backgrounds. In summary, the project is sustainable and scalable because it benefits small universities and provides a cost effective solution to advance CSE education. As a continuation of this project, we have undertaken a new project that included more institutions, more STEM disciplines, and more students. In addition to ASU and ERAU, the newly funded IUSE project created a coalition that included two more institutional partners—Hampden-Sydney College (HSC) in Virginia and Bethune-Cookman University (BCU), a Historic Black University (HBCU) in Florida. The four small universities are collaborating to integrate CDSE (Computational Data-enabled Sciences and Engineering) coursework into the undergraduate curriculum and embed authentic research experiences based on a CURE pedagogical model. Four new courses were added: (a) Figure 3. Prepare test for landing on grasses Database Design, (b) Genomics and Bioinformatics, (c) Problems in Atmosphere and Hydrosphere, and (d) Advanced Computing Resources in Biology. Except for the Database course, the other three new courses were organized by applications instead of computational methods. We are also creating of a virtual educational observatory to ensure that the effort will be ongoing after project funding ends and consequently have an even broader impact, especially on small, regional, and minority-serving institutions. The continuous project will help us collect adequate survey data so that we can draw statistically significant conclusions using evidence based and data driven evaluations. For example, we have added more questions in the end of term evaluation so that we can compare changes in student beliefs about CSE and team work after they have completed the course. The end of term evaluation will also ask each student to rank his Figure 4. Captured the moment for landing on water or her perceived importance and difficulty level of the course contents. In addition, the new project will adopt new technologies (e.g. educational data mining, learning analytics), use proven 4. BROADER IMPACT AND FUTURE instructional approaches (e.g., experiential, problem-centered, WORK collaborative learning), and integrate them into a flexible This project provided a feasible model to significantly increase curriculum involving the co-creation of resources and activities by student participation in CSE. Few small colleges have the participating partners. resources to provide CSE courses and programs for undergraduate students. The project also demonstrated a viable method for 5. ACKNOWLEDGMENTS scaling up: Adding more colleges to a cluster and then creating a The project for CSE education and summer REU workshops network of clusters. Our strategies to advance CSE education can presented in this paper was sponsored by the National Science be straightforwardly extended to other disciplinary domains and Foundation (NSF) of United States under the program of TUES other small universities. Grant 1244967 from 2013-2016. The authors would like also give December 2017 ISSN 2153-4136 17 Volume 8, Issue 3 Journal of Computational Science Education thanks for the NSF of United States for the continuous support for [10] Pratap Misra, Per Enge, (2004), Global Positioning System, the following-up project under the IUSE Grant 1626602 from Signals, Measurements, and Performance, Ganga-Jamuna 2017-2019. Press. [11] Pirnay-Dummer, P., Ifenthaler, D., & Spector, J. M. (2010), 6. REFERENCES Highly integrated model assessment technology and tools, Educational Technology Research & Development, 58(1), [1] Borgman, C.L. (2008) Fostering Learning in the Networked 3-18. World: The Cyberlearning Opportunity and Challenge. [12] Shiflet, Angela and George, (2013) Vol. 4, Issue 1, Journal Report of the NSF Task Force on Cyberlearning. of Computational Science Education, pp 11-15, [2] Borne, K., Wallin, J. & Weigel, R. (2009), The New http://www.shodor.org/media/content/jocse/volume4/issue1/s Computational and Data Sciences Undergraduate Program at hiflet_2013. George Mason University ICCS 2009 Proceedings of the 9th [13] Society of Industrial and Applied Mathematics, Modeling International Conference on Computational Science. 74-83. Across the Curriculum, Report on a SIAM-NSF Workshop*, [3] Kurt Byran and Tanya Leise, (2006), The $25,000,000,000∗ Arlington, Virginia, August 30–31, 2012 Eigenvector the linear algebra behind Google, www.rose- https://www.siam.org/reports/modeling_12.pdf. hulman.edu/~bryan/googleFinalVersionFixed.pdf. [14] Society for Industrial and Applied Mathematics. 2015. [4] Cognitive and Technology Group at Vanderbilt University, Modeling across the Curriculum II: Report on the 2nd 1992. The Jasper series as an example of anchored SIAM-NSF workshop. Philadelphia, PA: Society for instruction: Theory, program description, and assessment Industrial and Applied Mathematics. data. Educational Psychologist 27:291–315. [15] Simpson, A. Ludu, H. J. Cho and H. Liu, 2014, Experimental [5] Liu, L. & Ludu, A. (2012) ACE - A Model Centered REU and theoretical studies on visible light attenuation in water, Program Standing on the Three Legs of CSE: Analysis, Atmospheric and Oceanic Physics, retrievable from Computation and Experiment. Proceeding of ICCS2012, http://arxiv.org/abs/1408.3883 . Omaha, NE. [16] Spector, M. (2010) Assessing Progress of Learning in [6] Liu, H. Klein, J. (2013), Using REU Project and Complex Domains. The 11th International Conference on Crowdsourcing to Facilitate Learning on Demand, Education Research: New Educational Paradigm for Proceedings of IADIS International Conference on Cognition Learning and Instruction. and Explorative Learning at Digit Ages, Fort Worth, Texas, [17] Steven Lehr, Hong Liu, Sean Klinglesmith*, Alex Konyha *, pp 251-258. Natalia Robaszewska *, Jacob Medinilla *, Use Educational [7] Hong Liu, Jayathi Raghavan, A Mathematical Modeling Data Mining to Predict Undergraduate Retention, Submitted Module with System Engineering Approach for Teaching to the 16th IEEE International Conference on Advanced Undergraduate Students to Conquer Complexity, The Learning Technologies - ICALT2016, Austin, Texas. Proceedings of Conference ICCS 09 , Part II, LNCS5545, pp [18] Transforming Post-Secondary Education in Mathematics 93-102, 2009. (TPSE Math), Report of a Workshop Joint Mathematics, [8] Liu, H., Shi, X., Shao, J., Zhou, Q., Joseph-Ellison, S., Meetings, San Antonio, Texas, January 2015, Jaworski *, J., and Wen, C *., The Mechatronic System of [19] Turner, P. & Petzold, L. (2011) Undergraduate Eco-Dolphin - a Fleet of Autonomous Underwater Vehicles, Computational Science and Engineering Education, SIAM Proceeding of International Conference of Advanced Review, 53 (3), 561–574. Mechatronics System 2015, Beijing, August 24-27, 2015. [20] Turner, P. Levy, R., and Fowler, K., Collaboration in the [9] Lyons, T*, Ginther, M*. Mascarenas, P*. Rickard, E. Mathematical Sciences Community on Mathematical Robinson*, J. Braeger*, J, Liu, H., and Ludu A. 2014, Modeling Across the Curriculum, Chance, Using Data to Nonlinear Decelerator for Payloads in Aerial Delivery Advance Science, Education, and Society, Systems. I: Design and Testing, Nonlinear Dynamics, http://chance.amstat.org/2015/11/math-modeling. retrievable from http://arxiv.org/abs/1408.4190. 18 ISSN 2153-4136 December 2017