Conversation Analysis in Online Discussions: Insights for LLM-Based Chat Systems

This thesis explored how large language model–based chatbots can move beyond sycophantic and overly accommodating behaviors to offer more reflective, provocative conversations. Currently available commercial chat systems tend to flatter users, evade persistent challenges, and refrain from disagreeing. To design systems that remain engaging without descending into unproductive argument, it is essential to monitor conversational trajectories dynamically and to track signals of misunderstanding, ill divergence, and healthy engagement. The ability to detect alarming trajectories may help diversify response strategies. As a case study, we examined human-to-human discussions on the subreddit ChangeMyView, which is specifically dedicated to thoughtful, debate-like conversations where participants aim to challenge each other's opinions. Drawing on Conversation Analysis (CA) as a theoretical framework, we analyzed whether features of written online discourse align with CA premises and whether such features distinguish more “successful” (delta-awarded) conversations from less “successful” (deltaless) ones. In the process of analysis, we iterated through a range of computational methods, pivoting from mixed-methods analysis of conversations using API calls to LLMs to approximating discussions through a variety of NLP-computed linguistic characteristics. Our findings suggest that human-employed conversational approaches to sustaining challenging conversations are much more variable than what is apparent from our experience using LLM-based chatbots. Consistent with prior research, our results reveal no clear, consistent markers that separate delta from deltaless threads. Nevertheless, a combination of features based on CA theoretical postulates and observed in online discussions could capture discussion trajectories in an interpretable manner. This suggests that CA-based insights can inform the design of conversational agents, even when success metrics remain elusive and highly variable across human-human conversations.