Papers by John Haggerty
Analysis of the empirical research in the journal of second language writing at its 25th year (1992–2016)
In this historical survey, we review 272 empirical research articles published in the Journal of ... more In this historical survey, we review 272 empirical research articles published in the Journal of Second Language Writing (JSLW) over its first quarter century of publication. We report overall and periodic analyses (1992–1999, 2000–2010, 2011–2016) in respect to the following themes: (1) contexts and participants, (2) research foci and theoretical orientations, and (3) research methodology and data sources. The typical research contexts and participants were undergraduates in U.S. universities or colleges. The most common research foci were feedback and writing instruction and the main theoretical orientations were cognitive, social, socio-cognitive, genre, contrastive rhetoric, and critical theories. The most frequently used research methodology was qualitative and the top three data sources used by L2 writing researchers were multiple sources, text samples, and elicitation. Based on the findings, we make suggestions for future research in studies of L2 writing. Along with Tony Silva’s reflections on our results, the present analysis gives readers a birds-eye view of the scholarship on L2 writing over the last 25 years as represented in the JSLW.
Language Testing in Asia, Jul 26, 2015
Drawing on second language (L2) motivation constructs modelled on Dörnyei's (2009) L2 Motivationa... more Drawing on second language (L2) motivation constructs modelled on Dörnyei's (2009) L2 Motivational Self System, this study explores the relationship between language testing experience and the motivation to learn English among young adolescents (aged 12-15) in South Korea. A 40-item questionnaire was administered to middle-school students (N = 341) enrolled in a private language school (hakwan). Exploratory factor analysis (EFA) identified five salient L2 motivation factors. These factors were compared to four learner-background characteristics: gender, grade level, L2 test-preparation time, and experience taking a high-stakes university-level language test. The results suggest that second language motivation, based on the L2 motivation factors identified as most salient in this educational context, was significantly associated with the amount of time spent preparing for language tests and experience taking a high-stakes language test intended primarily for university-entrance purposes. Young South Korean adolescent learners’ testing experiences and their motivation to learn English are discussed in relation to the social consequences of test use and ethical assessment practices.
Asian EFL Journal, Dec 2011
This paper explores the potential impact of high-stakes English testing on young English language... more This paper explores the potential impact of high-stakes English testing on young English language learners’ (ELL) attitudes, beliefs, and motivations. A more meaningful role for consequential validity in language testing is sought through engagement with sociocultural theory, specifically Vygotskian conceptions of identity formation, in order to more fully contextualize a high-stakes learning environment, its effects on younger ELLs, and its implications for the test validation process. This initial phase, to be followed by a larger study, consisted of a pilot questionnaire developed from a working model of Language Proficiency Identity (LPID), and subsequently administered to 202 ELLs of various education levels. An analysis of responses to 20 questionnaire items was conducted on the basis of a) education level, and b) the completion of one of two high-stakes English tests. There were significant correlations found for 15/20 responses from middle-school students, 4/20 from high school students, and 0/20 from university students. These preliminary results suggest that high-stakes English testing has a more dramatic effect on the language proficiency identity of younger ELLs. Some implications for test validity are discussed in light of current theory and research on identity formation.
The Validity of Consequences: High-Stakes Language Testing and the Polarization of Young ELLs
This paper presents the main findings of an exploratory study that probes the attitudinal differe... more This paper presents the main findings of an exploratory study that probes the attitudinal differences of young ELLs based on completion of cognitively-demanding high-stakes language testing. A pilot questionnaire was administered to 202 ELLs from three different education levels: middle school, high school, and university. An analysis of responses to 20 questionnaire items was conducted on the basis of a) education level, and b) the completion of one of two high-stakes English tests: the TOEFL or the TEPS. There were significant correlations found for 15/20 responses from middle-school students, 4/20 from high school students, and 0/20 from university students. These preliminary results suggest that high-stakes English testing has a potentially polarizing effect on the attitudes, beliefs, and motivations of younger ELLs. Possible implications for modern notions of test validity, specifically consequential validity, will be explored and some recommendations for improvements to the test validation process will be offered.
Book chapters by John Haggerty
Test intensity is a concept that is increasingly discussed in considerations of washback (Watanab... more Test intensity is a concept that is increasingly discussed in considerations of washback (Watanabe, 2004). The greater the role that tests play in an educational setting, the greater their potential intensity, and the greater their potential impact on teaching and learning. This chapter reports on the main results of a questionnaire study involving 341 young adolescent language learners in the test-intensive context of South Korea. It examines the interaction between language testing experience (with high-stakes proficiency tests designed for adult, university-level, learners) and the motivation to learn English. Results suggest subtle but undermining effects of testing experience on the motivation to learn, and the need to reconsider the ethics and fairness of using such tests on children in middle school young adolescent language learners.
The global movement of students, the concomitant linguistic and cultural diversity of university ... more The global movement of students, the concomitant linguistic and cultural diversity of university classrooms, and mounting concerns about retention and program completion have prompted the increased use of post-entry diagnostic assessment (Read, 2008; 2015), which identifies students at-risk and provides them with early academic support. In this chapter we report on findings from a multistage-evaluation mixed methods study (Creswell, 2015), now in its sixth year, which is evaluating the impact of a diagnostic assessment procedure on retention, student engagement, and academic success in an undergraduate engineering program. Findings to date were examined through the lens of Activity Theory, which views socio-cultural object-oriented human activity as mediated through the use of symbolic (e.g., language) and material (e.g., pen) tools (Engeström, 1987; Leont'ev, 1981; Vygotsky, 1987). Changes in activities and their interrelationships are of central interest. In this Chapter we report on catalysts for such changes that have increased the impact of the diagnostic assessment procedure by: 1) applying a disciplinary (rather than generic) approach, which was fine grained enough to trigger optimal academic support; 2) embedding the diagnostic assessment procedure within a required first-year engineering course, which dramatically increased the numbers of students who voluntarily sought academic support; and 3) paying increased attention to the development of social connections, which are often crucial in student retention and success (e.g. Fox, Cheng, & Zumbo, 2014; Tinto, 1993). The findings reported here suggest that developing social connections/networks along with academic support increases the positive impact of a diagnostic assessment procedure.
Conference Presentations by John Haggerty
Performing the “graduate class” online: Academic discourse socialization practices among domestic and international doctoral students in a blended classroom.
Research exploring the academic socialization experiences of university students has increased ov... more Research exploring the academic socialization experiences of university students has increased over the last decade (e.g. Morita, 2004; Bronson, 2005; Zappa-Hollman, 2007). These studies illustrate the complexity of the socialization process for domestic and international students as they become accustomed to new social practices within a variety of educational spaces (digital and non-digital). Within North American higher education contexts, there has been concerted effort to incorporate (or “blend”) modern information and communication technologies with traditional classroom practices (Vaughan, Cleveland-Innes & Garrison, 2013). “Blended” classrooms combine both online learning management systems (such as Connect) and the face-to-face classroom (Gruba & Hinkelman, 2012). Although previous studies (e.g. Yim, 2011; Potts 2005) have explored student experience within blended classrooms, there is still limited understanding of how these ‘online’ experiences impact the academic socialization of students. This multiple case study explores the online academic socialization experiences of a group of international and domestic students (and the instructor) participating in a required doctoral course in a Canadian university. Theoretically, we draw on a language socialization (LS) theoretical framework (Duff & Talmy, 2011; Ochs, 2002). LS is a language-mediated process which involves communication among and between ‘novices’ and ‘experts’ through which novices (implicitly and explicitly) learn the norms of language use along with underlying values, ideologies, and potential identities available to community members (Duff, 2010). Data collected includes online transcripts, critical episode narratives, and interviews with other cohort members and the instructor). Data analysis includes discourse analysis of turn taking (Seedhouse, 2004), indexicality (Johnstone, 2008), and membership categorization (Baker, 2000). Initial results indicate that participants perform textual positionality (e.g. ‘novice’), affinity groups (Gee, 2007), and idiosyncratic communicative practices. Results help foster a deeper discussion of the kinds of challenges international and domestic graduate students face and how institutions can more effectively respond to these issues.
Reaching students-at-risk: The case of diagnostic assessment in first-year engineering
Increasingly, universities are using post-entry diagnostic assessment (Alderson, 2005. 2007) to i... more Increasingly, universities are using post-entry diagnostic assessment (Alderson, 2005. 2007) to identify entering students at-risk and provide early, individualized academic support. However, whether such support should be mandatory has remained a question of considerable debate (Read, 2008). This paper presents findings from a longitudinal study in a Canadian university which utilizes diagnostic assessment to better inform its undergraduate engineering program. In the study’s first year, 489 students (50% of the engineering cohort) were assessed with a modified, engineering-based version of the University of Auckland’s Diagnostic English Language Needs Assessment (DELNA). Students were informed of their results and invited on a voluntary basis to meet with peer mentors. Only 12 students (2%) sought additional feedback, including 3 of the 27 students (11%) who were identified at-risk. At the end of the year, 10 from the at-risk group had dropped out or were failing; 7 were borderline failures; and 10 were performing well (including the 3 at-risk students who had sought feedback). In the study’s second year, 899 students (95% of cohort) were assessed, but only 33 students (4%) voluntarily followed-up on their results. However, there was evidence that 3 of these students remained in the engineering program because of early diagnosis and pedagogical intervention. In year three, the diagnostic assessment was embedded within a required engineering course and students seeking feedback dramatically increased. Findings suggest that voluntary uptake of diagnostic feedback may not work; whereas embedding uptake in the context of a required course dramatically increases the potential of reaching students-at-risk.
Playing “writing games”: Academic discourse socialization into first-year doctoral studies in a Canadian university
There is now a fairly wide body of research exploring the academic experiences of post-secondary ... more There is now a fairly wide body of research exploring the academic experiences of post-secondary students as they make their transition to university (e.g., Casanave, 2002; Leki, 1985; Morita, 2004). However, to our knowledge this process has never been investigated from a collaborative student (and authorial) perspective. This study utilizes a language socialization approach (Duff, 2008, 2010) to better understand the academic socialization of four first-year PhD students (two domestic and two international) in the same department in a Canadian university. According to Duff (2010), language socialization is a “dynamic, socially and culturally situated, multimodal, and often multilingual process with unpredictable uptake, intentions, behind-the-scenes power plays, investment on the part of learners, and outcomes” (p. 186). This paper explores academic discourse socialization processes (a sub-field within language socialization) with a specific focus on written academic enculturation. As part of the same cohort, the authors participated in two mandatory doctoral courses during their first year of study. These courses required the completion of more traditional written assignments as well as computer-mediated communication (CMC) among class members. Data collection includes online communications (as posted during each course), written assignments (including instructor feedback), and post-course interactive reflections (on a collaborative wiki). Discourse analysis focuses on how power relations and cultural capital (Bourdieu, 1977) work to enable or constrain the academic socialization process. The results of this study will be of great interest to current and incoming doctoral students as well as those who are charged with facilitating their successful transition.
Mitigating risk in first-year engineering: Post-admission diagnostic assessment in a Canadian university
The use of diagnostic assessment in the post-admission context of first-year engineering requires... more The use of diagnostic assessment in the post-admission context of first-year engineering requires finding the right balance in the interplay of marketing (Read, 2008), task, rater, scale, and feedback. This longitudinal study investigates the potential of diagnostic assessment to identify students-at-risk, provide more effective academic support, and help prevent failure.
When the goal of a diagnostic assessment is positive washback on learning (Cheng, 2005), finding ... more When the goal of a diagnostic assessment is positive washback on learning (Cheng, 2005), finding the right balance in the interplay of marketing (Read, 2009), task, rater, scale, and feedback that results in optimal use that optimizes the use of assessment information is a critical issue. This paper presents initial findings from a longitudinal study investigating the potential of diagnostic assessment to identify students-at-risk early in their first year of engineering, to provide academic support, and help prevent failure. In the first year of the study, 489 students (50% of the first-year engineering cohort) were assessed with a modified version of the DELNA, which reflected the engineering context in: 1) tasks (i.e., the writing prompt used an engineering graph; a mathematics diagnostic was added); and 2) raters (i.e., 4 were drawn from with engineering; and 4 from with writing studies backgrounds). Raters were trained to use the generic DELNA scale and to write detailed feedback for students-at-risk. Students were informed of their results by e-mail and invited on a voluntary basis to discuss them with engineering and writing studies peer mentors. Only 12 (2%) of 489 students sought additional feedback, including 3 (11%) of the 27 students who were identified at-risk. At the end of the year, 10 of from the at-risk group had dropped out or were failing; 7 were borderline failures; and 10 were performing well. Case studies (Creswell, 1998; Merriam, 1998) of conducted with first-year students identified three additional risk factors: second language (L2) learners without language support, transfers from other contexts, and students repeating first-year. Amongst at-risk students who were successful in their first year, 2 had sought additional feedback on their diagnostic assessment results. Other key factors were: evidence of social networks, making connections with learning support, and strategic management of course demands. In the second year of the study, 899 students (95% of cohort) were assessed. While changes to the scale, raters, and tasks increased the consistency and quality of feedback, these changes failed to resolve the problem of voluntary uptake. Only 33 students (4%) followed up on their results. Proposed revisions to the diagnostic process include using it as a placement test. Implications of this change in purpose and use will be discussed.
Envisioning the Future: Pre-service Teachers' Imagined Community and Identity
This study explores the inter-connected factors affecting the imagined identities and communities... more This study explores the inter-connected factors affecting the imagined identities and communities of pre-service English teachers in South Korea. Through observation and reflective one-to-one and group interviews, I explore the complex processes involved in pre-service teachers' socialization into their respective communities of practice as well as their evolving language teacher identities. Participants for this study include nine pre-service English teachers who had completed (or were in the process of completing) their teacher education program and were currently preparing for a national exam to obtain their public school teaching certificate at either the middle school or high school level. Results suggest that a number of factors identified in previous research (e.g. Thomas & Cheng, 2009) are also of considerable influence for these participants; however, the L2 assessment environment appears to be having a dominant influence on pre-service teachers' imagined self and imagined community (what the teaching environment will be like and what will be possible for them to accomplish within it), a somewhat unique finding based on the literature reviewed. These findings suggest that there is a pressing need for much greater sensitivity to the L2 assessment environment surrounding teacher education and certification programs, particularly in (increasingly) test-intensive educational environments.
Managing the consequences of test use: L2 motivation, L2 identity, and L2 testing in globalized education environments
A great deal of research has been conducted into L2 motivation and its impact on the achievement ... more A great deal of research has been conducted into L2 motivation and its impact on the achievement of language proficiency (Dörnyei, 2001). This has helped to better understand the multitude of factors influencing a student’s beliefs, attitudes, and motivations to learn a second language as well as their tendency (or lack thereof) to actually follow through and reach their desired goals. This has lead to a great deal of research into identity formation among various language, the malleability of identity over time and place, and its continual negotiation within various ‘sites of struggle’ in which the willingness (or ability) to ‘invest’ in a language is fundamentally tied to issues of power (Norton, 2000). Despite this explosion of interest into L2 identity formation, the impact of high-stakes standardized language testing on language learner identity, particularly among young learners, has received scant attention at best. Despite the intensive role that language testing plays in many meritocratic educational environments in Asia (Ross, 2008) and its growing importance in the U.S. as a result of educational initiatives such as ‘No Child Left Behind,’ this is a consideration that has been largely overlooked in our increasingly globalized educational environment.
This paper reports on two studies conducted in the Korean context: a pilot study (Haggerty, in press) investigating ELLs at three educational levels (middle school, high school and university, N= 202), and a subsequent MA dissertation that focused more exclusively on L2 motivational factors influencing middle school students (N= 341). The results of both these studies strongly suggest that young learners are differentially affected by their language testing experiences and these differences are for more pronounced for younger learners than older ones. Through engagement with modern conceptions of test validity and Vygotskian conceptions of socio-cognitive development, the potential effects of language testing on L2 motivation and L2 identity formation will be explored in the hope of spurring additional research into an issue that profoundly affects English language learners of all ages around the world.
Thesis Chapters by John Haggerty
Most universities in English-dominant countries have been competing to attract multilingual learn... more Most universities in English-dominant countries have been competing to attract multilingual learners for some time, inspired by the dual need for brain power and income generation (Lee, Maldonado-Maldonado, & Rhodes, 2006). In the Canadian context, this has resulted in rising international student populations (Anderson, 2015) and the expansion of increasingly sophisticated academic language programs (Fox, Cheng, & Zumbo, 2014). Despite this, external research into the effectiveness and appropriateness of these programs from the perspectives of the students enrolled remains scarce (Keefe & Shi, 2017).
This multiple case study involves six multilingual learners enrolled in a newly-designed academic language program in a Canadian university. This first-year program provided content and academic language courses in two disciplinary areas (Arts and Sciences), which upon successful completion, qualified students for their second year in the university mainstream. In this study, I investigate how students responded to program design features and academic writing instruction. I incorporate multiple interviews with students, collection of their written assignments and feedback, observations of classrooms and other educational events, interviews with other program stakeholders, and collection of program documents.
Of the six student participants in this study, four were successful and two were less successful. For the four successful students, participation in the sheltered program was perceived as an overall beneficial experience that helped them make a positive transition to mainstream studies. However, responses to academic writing instruction and practice were highly variable and influenced by students’ backgrounds and their educational or disciplinary beliefs. For two less successful students, notions of agency, identity, and appropriation became influential in their transitions as they increasingly reported confusion, frustration, and conflict in meeting academic expectations. Results suggest there are several opportunities and challenges involved in the integration of sophisticated theoretical and pedagogical approaches, some of which may not be realized for some time after instruction has ceased. The study highlights an ongoing need to: 1) (re)consider the time needed and the degree of complexity involved in academic writing instruction and, 2) maximize alignment of pedagogical objectives with multilingual learners’ backgrounds as well as their perceived academic and disciplinary writing needs.
Uploads
Papers by John Haggerty
Book chapters by John Haggerty
Conference Presentations by John Haggerty
This paper reports on two studies conducted in the Korean context: a pilot study (Haggerty, in press) investigating ELLs at three educational levels (middle school, high school and university, N= 202), and a subsequent MA dissertation that focused more exclusively on L2 motivational factors influencing middle school students (N= 341). The results of both these studies strongly suggest that young learners are differentially affected by their language testing experiences and these differences are for more pronounced for younger learners than older ones. Through engagement with modern conceptions of test validity and Vygotskian conceptions of socio-cognitive development, the potential effects of language testing on L2 motivation and L2 identity formation will be explored in the hope of spurring additional research into an issue that profoundly affects English language learners of all ages around the world.
Thesis Chapters by John Haggerty
This multiple case study involves six multilingual learners enrolled in a newly-designed academic language program in a Canadian university. This first-year program provided content and academic language courses in two disciplinary areas (Arts and Sciences), which upon successful completion, qualified students for their second year in the university mainstream. In this study, I investigate how students responded to program design features and academic writing instruction. I incorporate multiple interviews with students, collection of their written assignments and feedback, observations of classrooms and other educational events, interviews with other program stakeholders, and collection of program documents.
Of the six student participants in this study, four were successful and two were less successful. For the four successful students, participation in the sheltered program was perceived as an overall beneficial experience that helped them make a positive transition to mainstream studies. However, responses to academic writing instruction and practice were highly variable and influenced by students’ backgrounds and their educational or disciplinary beliefs. For two less successful students, notions of agency, identity, and appropriation became influential in their transitions as they increasingly reported confusion, frustration, and conflict in meeting academic expectations. Results suggest there are several opportunities and challenges involved in the integration of sophisticated theoretical and pedagogical approaches, some of which may not be realized for some time after instruction has ceased. The study highlights an ongoing need to: 1) (re)consider the time needed and the degree of complexity involved in academic writing instruction and, 2) maximize alignment of pedagogical objectives with multilingual learners’ backgrounds as well as their perceived academic and disciplinary writing needs.