Papers by Stephen MacNeil
arXiv (Cornell University), Mar 14, 2024
Grasping complex computing concepts often poses a challenge for students who struggle to anchor t... more Grasping complex computing concepts often poses a challenge for students who struggle to anchor these new ideas to familiar experiences and understandings. To help with this, a good analogy can bridge the gap between unfamiliar concepts and familiar ones, providing an engaging way to aid understanding. However, creating effective educational analogies is difficult even for experienced instructors. We investigate to what extent large language models (LLMs), specifically ChatGPT, can provide access to personally relevant analogies on demand. Focusing on recursion, a challenging threshold concept, we analyzed the analogies generated by 385 first-year computing students. They were provided with a code snippet and tasked to generate their own recursion-based analogies using ChatGPT, optionally including personally relevant topics in their prompts. We observed a great deal of diversity in the analogies produced with student-prescribed topics, in contrast to the otherwise generic analogies, highlighting the value of student creativity when working with LLMs. Not only did students enjoy the activity and report an improved understanding of recursion, but they described more easily remembering analogies that were personally and culturally relevant. • Social and professional topics → Computing education.
Generating Multiple Choice Questions for Computing Courses Using Large Language Models
Generative AI in Computing Education: Perspectives of Students and Instructors
Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models
The Effects of Generative AI on Computing Students’ Help-Seeking Preferences
arXiv (Cornell University), Nov 2, 2023
arXiv (Cornell University), Aug 2, 2023
The recent advent of large language models (LLM) has resulted in high-performing conversational a... more The recent advent of large language models (LLM) has resulted in high-performing conversational agents such as chatGPT. These agents must remember key information from an ongoing conversation to provide responses that are contextually relevant to the user. However, these agents have limited memory and can be distracted by irrelevant parts of the conversation. While many strategies exist to manage conversational memory, users currently lack affordances for viewing and controlling what the agent remembers, resulting in a poor mental model and conversational breakdowns. In this paper, we present Memory Sandbox, an interactive system and design probe that allows users to manage the conversational memory of LLM-powered agents. By treating memories as data objects that can be viewed, manipulated, recorded, summarized, and shared across conversations, Memory Sandbox provides interaction affordances for users to manage how the agent should 'see' the conversation.
Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology
The recent advent of large language models (LLM) has resulted in high-performing conversational a... more The recent advent of large language models (LLM) has resulted in high-performing conversational agents such as ChatGPT. These agents must remember key information from an ongoing conversation to provide responses that are contextually relevant to the user. However, these agents have limited memory and can be distracted by irrelevant parts of the conversation. While many strategies exist to manage conversational memory, users currently lack afordances for viewing and controlling what the agent remembers, resulting in a poor mental model and conversational breakdowns. In this paper, we present Memory Sandbox, an interactive system and design probe that allows users to manage the conversational memory of LLM-powered agents. By treating memories as data objects that can be viewed, manipulated, recorded, summarized, and shared across conversations, Memory Sandbox provides interaction afordances for users to manage how the agent should 'see' the conversation.
Recent breakthroughs in Large Language Models (LLMs), such as GPT-3 and Codex, now enable softwar... more Recent breakthroughs in Large Language Models (LLMs), such as GPT-3 and Codex, now enable software developers to generate code based on a natural language prompt. Within computer science education, researchers are exploring the potential for LLMs to generate code explanations and programming assignments using carefully crafted prompts. These advances may enable students to interact with code in new ways while helping instructors scale their learning materials. However, LLMs also introduce new implications for academic integrity, curriculum design, and software engineering careers. This workshop will demonstrate the capabilities of LLMs to help attendees evaluate whether and how LLMs might be integrated into their pedagogy and research. We will also engage attendees in brainstorming to consider how LLMs will impact our field. • Social and professional topics → Computing education; • Computing methodologies → Natural language generation.
CausalMapper: Challenging designers to think in systems with Causal Maps and Large Language Model
Creativity and Cognition, Jun 19, 2023
DesignNet: a knowledge graph representation of the conceptual design space
Creativity and Cognition, Jun 19, 2023
arXiv (Cornell University), Feb 27, 2023
A Context-Aware Browser Extension for Just-in-Time Learning of Data Literacy Skills
arXiv (Cornell University), Jul 3, 2023
Figure : Three methods to connect user interface components to large language models. 1) static p... more Figure : Three methods to connect user interface components to large language models. 1) static prompts are predefined prompts that can be selected directly from the UI, 2) template-based prompts generate prompts based on selected options in the UI, 3) free-form prompts provide a direct way of interacting with prompts.
Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2
Curating course material that aligns with students' learning goals is a challenging and time-cons... more Curating course material that aligns with students' learning goals is a challenging and time-consuming task that instructors undergo when preparing their curricula. For instance, it is a challenge to find multiple-choice questions or example codes that demonstrate recursion in an unlabeled question bank or repository. Recently, Large Language Models (LLMs) have demonstrated the capability to generate high-quality learning materials at scale. In this poster, we use LLMs to identify programming concepts found within code snippets, allowing instructors to quickly curate their course materials. We compare programming concepts generated by LLMs with concepts generated by experts to see the extent to which they agree. The agreement was calculated using Cohen's Kappa. • Social and professional topics → Computing education; • Computing methodologies → Natural language generation.
Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1
Reasoning about code and explaining its purpose are fundamental skills for computer scientists. T... more Reasoning about code and explaining its purpose are fundamental skills for computer scientists. There has been extensive research in the field of computing education on the relationship between a student's ability to explain code and other skills such as writing and tracing code. In particular, the ability to describe at a high-level of abstraction how code will behave over all possible inputs correlates strongly with code writing skills. However, developing the expertise to comprehend and explain code accurately and succinctly is a challenge for many students. Existing pedagogical approaches that scaffold the ability to explain code, such as producing exemplar code explanations on demand, do not currently scale well to large classrooms. The recent emergence of powerful large language models (LLMs) may offer a solution. In this paper, we explore the potential of LLMs in generating explanations that can serve as examples to scaffold students' ability to understand and explain code. To evaluate LLM-created explanations, we compare them with explanations created by students in a large course (𝑛 ≈ 1000) with respect to accuracy, understandability and length. We find that LLM-created explanations, which can be produced automatically on demand, are rated as being significantly easier to understand and more accurate summaries of code than student-created explanations. We discuss the significance of this finding, and suggest how such models can be incorporated into introductory programming education.
Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 2
The recent advent of highly accurate and scalable large language models (LLMs) has taken the worl... more The recent advent of highly accurate and scalable large language models (LLMs) has taken the world by storm. From art to essays to computer code, LLMs are producing novel content that until recently was thought only humans could produce. Recent work in computing education has sought to understand the capabilities of LLMs for solving tasks such as writing code, explaining code,
Freeform Templates: Combining Freeform Curation with Structured Templates
Creativity and Cognition
The Implications of Large Language Models for CS Teachers and Students
Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2
arXiv (Cornell University), Nov 4, 2022
Advances in natural language processing have resulted in large language models (LLMs) that are ca... more Advances in natural language processing have resulted in large language models (LLMs) that are capable of generating understandable and sensible written text. Recent versions of these models, such as OpenAI Codex and GPT-3, can generate code and code explanations. However, it is unclear whether and how students might engage with such explanations. In this paper, we report on our experiences generating multiple code explanation types using LLMs and integrating them into an interactive e-book on web software development. We modified the e-book to make LLM-generated code explanations accessible through buttons next to code snippets in the materials, which allowed us to track the use of the explanations as well as to ask for feedback on their utility. Three different types of explanations were available for students for each explainable code snippet; a line-by-line explanation, a list of important concepts, and a high-level summary of the code. Our preliminary results show that all varieties of explanations were viewed by students and that the majority of students perceived the code explanations as helpful to them. However, student engagement appeared to vary by code snippet complexity, explanation type, and code snippet length. Drawing on our experiences, we discuss future directions for integrating explanations generated by LLMs into existing computer science classrooms. • Social and professional topics → Computing education; • Computing methodologies → Natural language generation.