Books by Susan Ramlo, PhD
USING Q METHODOLOGY AND Q FACTOR ANALYSIS IN MIXED
METHODS RESEARCH
The purpose of this chapt... more USING Q METHODOLOGY AND Q FACTOR ANALYSIS IN MIXED
METHODS RESEARCH
The purpose of this chapter is to identify two
multivariate techniques that can be used to
facilitate the interpretation of mixed methods
research. These techniques can aid researchers
in answering their research questions by
demonstrating how to disaggregate or aggregate
their data. More specifically, our purpose
is to introduce two multivariate techniques,
Q methodology and Q factor analysis, to
readers by describing these techniques as well
as by giving examples that should assist readers
in performing their own Q methodology
and Q factor analysis studies. Finally, we discuss
how researchers can take their research a
step farther by answering more-sophisticated
research questions that include groups of
people. For instance, these groups, whether
derived empirically such as via Q factor analysis
or theoretically such as psychosocial stages,
can be used as variables within other types of
multivariate analyses.
Published papers by Susan Ramlo, PhD
Operant Subjectivity, 2023
Our world is full of algorithms that operate within social media, face identification, et cetera.... more Our world is full of algorithms that operate within social media, face identification, et cetera. Is the algorithm the secret to uncovering meaning and understanding within Q methodology or are algorithms also full of subjectivity? Stephenson predicted the problems others would have with Q methodology. Steven Brown suggested that one must get "a feeling for the organism" during interpretation, yet some Q researchers seem to seek formulaic approaches to their Q studies, sans a sense of feeling for the subjectivities uncovered. In this way, these formulaic researchers seek a set of algorithms to perform Q studies much like novice physics students often approach problem solving. However, as a famous jazz bassist stated, "It's not the notes that makes it jazz … it's not music without structure." This assertion was the impetus for addressing how Q, jazz music, and natural science use structure to frame exploration, interaction, and inquiry. Thus, the various structures within the stages of any Q study can help frame any study but in a way that is always new and inclusive while embracing inquiry and exploration.
Journal of Cybersecurity Education, Research and Practice, 2020
Cybersecurity is a worldwide issue and concern. Prior studies indicate that many people do not us... more Cybersecurity is a worldwide issue and concern. Prior studies indicate that many people do not use cybersecurity best practices. Although these prior studies used large-scale surveys or interviews, this study used Q methodology [Q] because Q provides greater insight than Likert-format surveys. In fact, Q was created to scientifically study subjectivity. Within a Q study, various stages as well as philosophical, epistemological, and ontological principles represent a complete methodology. At first, Q researchers collect items that represent the broad range of communications about the topic (called the concourse). Although the items can be pictures, scents, or other means of communication, statements are the most common. Q researchers reduce the items of the concourse to create the Q-sample while preserving the range of communications. Subsequently, participants sort these items into a grid to provide a snapshot of their viewpoint on the topic. Statistical analysis reveals the multiple, diverse viewpoints in a way that allows for detailed descriptions of those views. In this study, the researchers collected statements about cybersecurity. Students in technical degree programs, including computer information systems (CIS), sorted these statements into a grid with a range of “most like my view” to “most unlike my view” of cybersecurity. Items placed on the extreme ends of this grid represent those statements most salient with each student’s views. Analyses revealed three divergent viewpoints: 1) Cybersecurity best practices, 2) No worries, and 3) No sense of urgency. Although the CIS majors identified with View 1, the other technical degree program students were represented across all three views. Certainly, students who hold the No worries and No sense of urgency viewpoints are unprepared to deal with cybersecurity issues in the workplace. The descriptions of these views have implications for cybersecurity course and program development, including assessments. Additionally, this study’s outcomes indicate a need to replicate this investigation in other settings to estimate risk of employees introducing cyber threats at their workplace. Similarly, these outcomes have implications for workforce development training regarding improved cybersecurity viewpoints and, therefore, behaviors.
Examining urban, American, middle-school students' divergent views of nature before and after a field trip to a university field station and nature preserve
The Urban Review, 2018
Nature field trips offer opportunities for urban students’ exploration, discovery, and learning w... more Nature field trips offer opportunities for urban students’ exploration, discovery, and learning which they may not experience otherwise. Seventh grade students at an economically disadvantaged urban school in the US Midwest sorted statements related to school, nature, and science before and after their field trip experience. These sorts provide a snapshot of students’ subjective thoughts on the topic. The statements were developed from student writings from the previous year’s 7th grade field trip. As a qualitatively focused mixed method, Q methodology [Q] requires only a relatively small group of participants (here just under 50). The sorts were then statistically grouped based on similarity of the sorts with the resulting three perspectives: Active Nature Learners, Not a Nature Lover, and The Environmentalists. Thus, Q provided differentiation of student views about nature before and after the field trip. Thirty-percent of those who provided sorts before and after the field trip changed their viewpoint to one that was more positive about nature (Active Nature Learners) or the environment (The Environmentalists). Students’ written comments and the descriptive viewpoints stakeholder feedback that can be used for program improvement. For instance, The Environmentalists view provides an outcome goal for the field trip experience for students.
Divergent viewpoints about the statistical stage of a mixed method: Qualitative versus quantitative orientations
International Journal of Research & Method in Education, 2019
Within mixed methods research, a key issue is how to mix qualitative and quantitative research or... more Within mixed methods research, a key issue is how to mix qualitative and quantitative research orientations. Similar debates arise within a mixed method called Q methodology [Q] that has an 80-year history and strong qualitative leanings. Q is often identified with the sorting of statements and the factor analysis (grouping) of people with similar viewpoints. However, Q is a complete methodology with statistical, philosophy-of-science, epistemological, and psychological principles. This study sought to empirically study the viewpoints concerning Q’s factor analytic stage using Q to reveal viewpoints that reflect divergent worldviews about research and mixed methods. Although the idea of Q as a mixed method is represented across all three perspectives, these views diverge in relation to the factor analytic stage. The three perspectives that emerged are: Inherently Mixed-Focus, Quantitative-Focus, and Skeptical Novice. Thus, viewpoints include one accepting qualitative-quantitative focused factor analysis, another accepting traditional factor analysis, and another that seems doubtful of the other viewpoints. These distinct viewpoints mirror discussions within the
larger mixed methods community and in social science research in general about how research methods and philosophies can be mixed. Implications for examining debates within mixed methods communities and future research are discussed.
Studies in Higher Education, 2018
Recently, campus free speech has become a focus of contentious debate and increased scrutiny. Thi... more Recently, campus free speech has become a focus of contentious debate and increased scrutiny. This study confirms that although university stakeholders may generally embrace the concept of free speech on campus, they also disagree about its limits and purpose within higher education, even if they are from the same institution. This investigation used Q methodology to scientifically study the subjectivity (viewpoints) among a diverse set of university stakeholders within the United States. Participants provided a snapshot of their views by sorting 55 statements related to speech on campus. The analyses revealed consensus, distinguishing statements, and rich descriptions of five unique speech on campus viewpoints: Idealistic, Social Justice, Speech Crisis, Sage on the Stage, and Fox News. These viewpoints provided insights about how university stakeholders perceive speech on campus. Consensus includes acceptance that ideas taught and expressed at institutions of higher education should rest with the faculty. Implications are discussed.
Instructors in higher education are very familiar with the Likert scale Students' Evaluation of T... more Instructors in higher education are very familiar with the Likert scale Students' Evaluation of Teaching (SET) used to evaluate teaching. Researchers have raised concerns about biases affecting the results of SET surveys, as well as their validity and reliability and use in high-stakes decision making. Here, we demonstrate that Q methodology, identified as an 80 year old mixed method, is better suited to determine the differing student views about a college math education course. We will discuss how Q used for SET can produce results that are more helpful in assisting faculty in improving their teaching effectiveness and in redesigning courses while also providing more informative SET results for faculty evaluations.
Flipped classrooms are a relatively new teaching strategy where the typical lecture and homework ... more Flipped classrooms are a relatively new teaching strategy where the typical lecture and homework elements of a course are reversed. Although flipped classrooms are gaining popularity, evaluations of this type of pedagogical model are limited. The purpose of this study was to investigate student views related to the effectiveness of a flipped physics classroom in a way that could inform the instructor for improving upon the course's effectiveness for all students. This study involved the use of Q methodology, a measure of subjectivity, to examine students' views of their flipped physics classroom experiences at a large, public university within a first-semester physics course for non-majors. Twenty-one of the 22 students enrolled in the class participated. The sample of statements sorted by students was selected from various student-instructor communications. Two distinct views, as well as consensus, emerged from the analyses and revealed a positive view of the flipped classroom experience, named "Active Learners," and a more passive learner view named "Unprepared Traditionalists." Students' written comments were used further to examine these views and to provide further insights related to these unique views of the flipped course. This mid-semester evaluation provided insights that led to changes within the course to improve students' learning experiences during the remainder of the semester and the following semester. The instructor began to provide a rationale for flipping the physics course, demonstrated how to access videos and other materials online, and offered time at the start of each class for students to ask questions regarding video content.
Student views regarding online freshmen physics courses
Background: Nationally, many public universities have started to
move into the online course and ... more Background: Nationally, many public universities have started to
move into the online course and program market that was previously
associated with for-profit institutions of higher education. Public
university administrators state that students seek the flexibility of
online courses. But do students want to take courses online, especially
freshmen-level science courses perceived to be difficult?
Purpose: This study investigated student views related to the
potential of a physics course they were currently enrolled in being
offered online.
Sample: This study took place at a large, public, mid-western university
and involved students enrolled in either the first or second semester
of a face-to-face flipped physics course for engineering technology
majors.
Design and methods: Discussions with students during the semester
about their online course experiences and expectations were used to
develop the concourse and subsequently the Q sample to perform
a Q methodology study about students’ views regarding taking
physics courses online. Additional statements for the concourse and
Q sample were taken from communications with administrators at
the university. In this way, the statements sorted by the students
included those from students and those from administrators. Factor
analysis of the Q sorts resulted in three factors, each representing a
unique perspective. Interpretations of these perspectives included
the analyses of the Q sorts, the researcher’s interactions with students
and administrators, and students’ written responses regarding their
previous online course experiences and their sorting decisions.
Results: Three unique student views emerged were named: keeping it
real and face-to-face, Online could be ok depending upon the course
and instructor, and Online not for STEM classes. Consensus among
the views is also discussed.
Conclusions: Overall, students’ views are negative concerning having
physics courses, including labs, online and those views conflict with
statements expressed by administrators regarding students’ desires
for online courses at the university.
Students' Views about Potentially Offering Physics Courses Online
This manuscript’s purpose is to introduce Q as a methodology before providing clarification about... more This manuscript’s purpose is to introduce Q as a methodology before providing clarification about the preferred factor analytical choices of centroid and theoretical (hand) rotation. Stephenson, the creator of Q, designated that only these choices allowed for scientific exploration of subjectivity while not violating assumptions associated with other choices like principal components (PCA) and Varimax. Although Q software offers Stephenson’s preferred choices as factor analytic options, today most Q methodologists use the more “modern” factor analytical choices of PCA and Varimax. Similarly, reviewers and critics of Q research often question the use of centroid with theoretical rotation, further discouraging their use. Researchers who took statistics coursework since the advent of statistical computer software are unfamiliar with centroid and theoretical rotation, their history, their processes, and why they offer a means of best scientifically exploring pragmatic, meaningful factor analytical solutions within Q methodology studies. Statistical versus theoretical considerations are discussed.
Students' Views about Potentially Offering Physics Courses Online
Nationally, many public universities have started to move into the online course and program mark... more Nationally, many public universities have started to move into the online course and program market that is most often associated with for-profit institutions of higher education. Administrators in public universities make statements regarding benefits students' desire for flexibility and profit-margins related to online courses. But do students attending a large public university want to take courses online especially science courses perceived to be difficult such as freshmen level physics courses? This study took place at a large, public, Midwestern university and involved students enrolled in the first semester of a face-to-face, flipped physics course for engineering technology majors. Statements were collected from comments about online courses made by the university's administration and students in the course. Twenty students sorted 45 statements. Two student views emerged with one rejecting online courses in general and the other primarily rejecting online math, science, and technology courses, including physics. Students' descriptions of their previous online course experiences were used to inform the analyses and to assist in describing the two views that emerged in conjunction with the distinguishing statements. Consensus among the two views is also discussed. Overall, the results indicate a potential divergence between student views and what university administrators believe students want.
In higher education, reading circles are not uncommon teaching and learning professional developm... more In higher education, reading circles are not uncommon teaching and learning professional development
experiences. Yet we found no studies in the literature that included an evaluation of the effectiveness of
reading circles. In this study, faculty who selected a teaching and learning themed book were part of selfdirected
reading circle groups. The evaluation used Q methodology, a measure of subjectivity (Brown, 1986;
Stephenson, 1953). The analysis revealed two distinct perspectives, with 23 out of 24 Q sorters represented by
one factor. This factor indicated that the reading circles were successful in getting nearly all participants to
reflect on and transform their teaching.
Q methodology blends qualitative and quantitative, yet was only recently identified as a mixed me... more Q methodology blends qualitative and quantitative, yet was only recently identified as a mixed method. Q has had a challenging 80-year history that can inform the broader but younger mixed methods community. This article introduces Q methodology and its position within mixed methods before discussing Q’s struggles against dismissive voices and faulty assumptions. The benefits of research communities and mentoring are also discussed within the context of Q’s history. Many of the struggles of mixed methods currently can be seen within Q’s history. The purpose of this article is to use the experiences within the Q community to benefit and inform the practice of mixed methods researchers in ways that assist all of us to best study our world.
Mid-Western Educational Researcher, Aug 2015
Program assessment is now commonplace at most colleges and universities and is required for accre... more Program assessment is now commonplace at most colleges and universities and is required for accreditation of specific degree programs. Key aspects of program assessment include program improvement, improved student learning, and adequate student preparation for the workforce. Thus, program assessment is a key ingredient to program health. Although surveys are often used within program assessment in higher education, this study demonstrates the weaknesses
of this method and, instead, introduces Q methodology as a means of program assessment especially in the area of needs assessment. In essence, Q offers an objective way to measure subjectivity about any topic. Unlike Likert-scale surveys, Q is a mixed method that reveals the multiple unique views as well as
consensus within the group of participants. In this study, Q was used to determine views of a construction engineering technology program. How the results will be used to improve the program is presented.
Operant Subjectivity 36(4), Jul 2013
represented populations in higher education by allowing students to earn an Associate's degree or... more represented populations in higher education by allowing students to earn an Associate's degree or up to two years of college credit by the time they graduate from high school. A previous qualitative study examined transition experiences of students in their senior year. The current study was conducted at a large Midwestern ECHS where students take one to three college classes during their first two years of high school (freshman and sophomore years) and full college course loads during their last two years (junior and senior years) of high school. The transition between the sophomore and junior year is the most difficult for many of the students. Therefore, there is a need to understand persistence of these students beyond the sophomore year and through to graduation. The authors developed the Q sample based upon the aforementioned qualitative study. Statements from within the six categories that emerged from that study were used to balance the Q sample. The purpose of this study is to investigate ECHS students' views of their high school experience with the goal of determining which perspectives lead to persistence between the students' sophomore and junior years and what views best represent "college-ready" beyond typical academic criteria, such as ACT scores, for this and other similar programs. Further planned research will both present longitudinal findings following from this initial study and seek to replicate the perspectives found here. For this study, 42 students that were at the rank of high school sophomore and junior at the end of the 2011-2012 academic year sorted 51 statements that pertained to their experiences and perspectives of being an ECHS student. The study found two factors: "College Ready" and "Academically Immature." Implications of these results are discussed.
The purpose of this study was to investigate in-service science teachers’ understanding of force ... more The purpose of this study was to investigate in-service science teachers’ understanding of force and motion concepts after a week-long professional development opportunity at a large, public Midwestern university. In this study, views of learning physics were investigated using Q methodology. Previous studies {{439 Ramlo, Susan 2008;413 Ramlo, Susan 2008; }} found that engineering-technology students in a first semester college physics course posses a variety of views about their learning of physics and that these views are related to their learning of physics concepts. The in-service teacher professional development study is especially important considering issues in the United States related to out of field teaching in subjects such as physics {{421 Escalada,Lawrence T. 2006; }}. The findings should assist those providing professional development opportunities to in-service science teachers. Previous studies investigating teachers’ views about learning science with Q methodology were not found in the literature.
Awareness of effective weight management strategies is necessary to prepare exercise science stud... more Awareness of effective weight management strategies is necessary to prepare exercise science students for future work with obesity. Exercise science faculty developed a course related to exercise as a therapeutic tool and options available for weight loss. Purpose: This study investigates students’ views of weight management and obesity which can be a difficult task utilizing traditional methods such as Likert scale surveys. Methods: To investigate students’ views relative to the course content, Q Methodology, which is a mixed method research approach, was used at the beginning and at the end of the course. Students completed a Q-sort, ranking 44 statements related to obesity in accordance to their own points of view. The pretest/posttest design enabled an opportunity to determine if course goals were accomplished and if students’ perceptions related to obesity were transformed. Results: Factor analysis of pre-course sorts revealed students possessed a naïve singular view of weight related concepts. Analysis of sorts at course completion resulted in a two-factor solution revealing changes in perspective; some students took on a health professional type of view while others demonstrated anti-fat views. Conclusion: The findings revealed key variables that should be addressed in curricula involving therapeutic obesity interventions. Improved curriculum based on these results should better encourage students to develop broad comprehension of multifactorial causes and treatment for weight management and obesity. Implementing the use of Q Methodology may serve as a valuable asset to measure and evaluate subjectivity of obesity discrimination. Anti-fat attitudes among exercise science students may influence future clients’ potential achievements.
Operant Subjectivity: The International Journal for Q Methodology, 2011
In volume 32 of this journal, Paul Stenner suggests that Stephenson was resistant to Q methodolog... more In volume 32 of this journal, Paul Stenner suggests that Stephenson was resistant to Q methodology being placed within other theoretical frameworks. Yet in this same piece, Stenner states that it is time for Q methodology to be brought into a greater dialogue with contemporary social theory and research practice. This article seeks to demonstrate how Q fits into the contemporary research practice of mixed methods and argues that this perspective is not in conflict with Stephenson’s positions on Q as a methodology. Further, our position reflects recent calls for the development of new techniques and procedures to be used in mixed-methods research. Those making the call will find interest in what Q has to offer the social and behavioral sciences now, 75 years after it emerged in Stephenson’s 1935 letter to Nature, and even though the term mixed-methods research has only emerged in last couple of decades. Q methodology is shown to fit well methodologically into the mixed-methods continuum as described by prominent mixed-methods scholars, which further supports a position that Q represents a mixed research methodology.
Stephen Gourlay's comments about our article "Q Methodology and Its Position in the are greatly a... more Stephen Gourlay's comments about our article "Q Methodology and Its Position in the are greatly appreciated and we are delighted that he agrees with us that Q methodology is a mixed method. We have had discussions similar to what Gourlay presents as the qualification of aspects of what is typically considered highly quantitative-for example, R-factor analysis where the naming of the factors requires a qualitative and often subjective interpretation of the numerical results. Simply, we cannot agree more with Gourlay as well as those he references, for example, Michell (1999). The truth is that when Stephenson first developed Q technique, there was no separation of the qualitative and quantitative in the research venue like there is today. Perhaps this is why Q, as Gourlay describes, consists of the quantification that is at the heart of factor analysis but also the quantification of that which is inherently qualitative (the concourse). In this way, Q consists of a meshing of qualitative and quantitative. Some may distinguish the two major aspects of Q methodology: Q sort and Q factor analysis. The Q factor analysis appears purely quantitative to some but the interpretation of the factors, like the factors determined within R factor analysis, must be interpreted and this involves qualitative aspects. Similarly, the Q sort is an act that follows the determination of the concourse and the selection of the Q samplesomething inherently qualitative. Yet, as Gourlay describes, we then "quantify" this concourse by allowing sorters to determine where each of the Q-sample items is placed within the grid. In this way, we could use the phrase "interactive continuum" to better describe how the qualitative and quantitative aspects of Q methodology work together such that each piece informs the other. We hope others join Gourlay in embracing this mixture as representative of Q as a mixed method and allow us all to help extend the use of Q methodology.
Uploads
Books by Susan Ramlo, PhD
METHODS RESEARCH
The purpose of this chapter is to identify two
multivariate techniques that can be used to
facilitate the interpretation of mixed methods
research. These techniques can aid researchers
in answering their research questions by
demonstrating how to disaggregate or aggregate
their data. More specifically, our purpose
is to introduce two multivariate techniques,
Q methodology and Q factor analysis, to
readers by describing these techniques as well
as by giving examples that should assist readers
in performing their own Q methodology
and Q factor analysis studies. Finally, we discuss
how researchers can take their research a
step farther by answering more-sophisticated
research questions that include groups of
people. For instance, these groups, whether
derived empirically such as via Q factor analysis
or theoretically such as psychosocial stages,
can be used as variables within other types of
multivariate analyses.
Published papers by Susan Ramlo, PhD
larger mixed methods community and in social science research in general about how research methods and philosophies can be mixed. Implications for examining debates within mixed methods communities and future research are discussed.
move into the online course and program market that was previously
associated with for-profit institutions of higher education. Public
university administrators state that students seek the flexibility of
online courses. But do students want to take courses online, especially
freshmen-level science courses perceived to be difficult?
Purpose: This study investigated student views related to the
potential of a physics course they were currently enrolled in being
offered online.
Sample: This study took place at a large, public, mid-western university
and involved students enrolled in either the first or second semester
of a face-to-face flipped physics course for engineering technology
majors.
Design and methods: Discussions with students during the semester
about their online course experiences and expectations were used to
develop the concourse and subsequently the Q sample to perform
a Q methodology study about students’ views regarding taking
physics courses online. Additional statements for the concourse and
Q sample were taken from communications with administrators at
the university. In this way, the statements sorted by the students
included those from students and those from administrators. Factor
analysis of the Q sorts resulted in three factors, each representing a
unique perspective. Interpretations of these perspectives included
the analyses of the Q sorts, the researcher’s interactions with students
and administrators, and students’ written responses regarding their
previous online course experiences and their sorting decisions.
Results: Three unique student views emerged were named: keeping it
real and face-to-face, Online could be ok depending upon the course
and instructor, and Online not for STEM classes. Consensus among
the views is also discussed.
Conclusions: Overall, students’ views are negative concerning having
physics courses, including labs, online and those views conflict with
statements expressed by administrators regarding students’ desires
for online courses at the university.
experiences. Yet we found no studies in the literature that included an evaluation of the effectiveness of
reading circles. In this study, faculty who selected a teaching and learning themed book were part of selfdirected
reading circle groups. The evaluation used Q methodology, a measure of subjectivity (Brown, 1986;
Stephenson, 1953). The analysis revealed two distinct perspectives, with 23 out of 24 Q sorters represented by
one factor. This factor indicated that the reading circles were successful in getting nearly all participants to
reflect on and transform their teaching.
of this method and, instead, introduces Q methodology as a means of program assessment especially in the area of needs assessment. In essence, Q offers an objective way to measure subjectivity about any topic. Unlike Likert-scale surveys, Q is a mixed method that reveals the multiple unique views as well as
consensus within the group of participants. In this study, Q was used to determine views of a construction engineering technology program. How the results will be used to improve the program is presented.