Written by Paulina Bondaronek and Siobhán Healy-Cullen
Machine-assisted topic analysis (MATA) aims to use the efficiency of Artificial Intelligence and combines it with the nuanced and rich insights derived from qualitative analysis. I call this a “meaningful AI-Human collaboration”. MATA was developed in response to a significant challenge during the COVID-19 pandemic; I was tasked with analysing and providing actionable insights based on 16,000 free-text responses to the question “How could we improve the service” (rapidly). The service in question was the NHS Test & Trace, which managed the pandemic response in England. With only my eyeballs to rely on (…and my expertise as a Behavioural Scientist), I recognised the potential of technology to speed up this task.
The Role of AI in Qualitative Analysis
MATA uses Artificial Intelligence for topic modelling. Put simply, MATA automatically identifies and organises key patterns in large volumes of textual data. I call this “community of words”. Researchers then qualitatively analyse this structured data, applying theoretical approaches and exploring the “community of meaning” in the representative quotes from each model. This process involves negotiating meanings beyond words and descriptions, creating a contextual understanding that AI alone is unlikely to achieve, but can enable.
Although MATA was developed to assess data related to free-text data, I have since applied the method to different datasets, which has led to great collaboration and learning. For example, I applied MATA to analyse experiences with menstrual tracking apps (MTAs), often not evaluated despite their widespread use. By scraping Twitter (X) data, my colleague Tris Papakonstantinou and I conducted a social listening exercise to explore the potential negative impacts of these apps (we chose to focus on negative impacts for many reflexive reasons that I won’t detail here!). With MATA, we analysed over 34,000 tweets, which would be impossible using qualitative analysis (and not the best use of time for unique qualitative research skills).
With this data, I travelled from London to Wellington, to Massey University for further analysis – one of the few hubs of critical thought in digital health.
At Massey University, I visited Sarah Riley and Siobhán Healy-Cullen, whose research areas include critical digital health. I was eager to find space where we could discuss MATA and get our hands dirty, to analyse the machine-generated output. During a pilot data workshop, I showed the topics generated by topic modelling and explored how different researchers might interpret and analyse the data, forming the “community of meaning.” The purpose of the pilot workshop was to assess the method’s usefulness, especially with a complex, messy dataset like social media tweets that were not created for this purpose.
During the data workshop, we discussed a handful of topics, including users’ experiences of MTAs as expressed in tweets, including distress and frustration from the app’s inaccurate predictions and unrealistic expectations (though these expectations might seem realistic given the unsubstantiated benefit claims made by the companies). One topic grouped together by the machine was “privacy and security concerns”: concerns were raised about app companies sharing data with the government, for example, in cases of law enforcement requests during investigations of unexplained pregnancy loss, and the potential use of personal data in legal cases related to reproductive health.
See below the mindmap of the analysis, with “topics” indicating automatically identified topics from large datasets, and “themes” analysis through qualitative methods by a Human.
Overall, the workshop was insightful, and I wish we could discuss the dataset for a whole day rather than 2 hours! The participants found it useful, and I learned a lot about how to run future workshops. I aim to run such workshops regularly – where the “AI-human collaboration” would be close, iterative, and, most importantly, meaningful. What does meaning mean in AI-Human collaboration? For me, it means bringing together experts, practitioners and, most importantly, the community that the data is about, to provide feedback to the machine to improve its analysis, and to inform actionable insights grounded in lived experience.
From MATA to HUMBLE
MATA is evolving. The analysis produces broad findings, obscuring the nuanced experiences of diverse groups over time. An unmeaningful, superficial AI analysis can further marginalise the voices of underrepresented groups and minority communities.
However, if we develop sophisticated, comprehensive, slow-pace, iterative algorithms, we have the opportunity to bring the less-heard voice to the surface and address existing social inequalities. I would like to help contribute to realising this vision of AI, hence, I’m introducing HUMBLE (Human-Centred Method for Bias-Reducing Algorithms with Natural Language Processing and Qualitative Analysis).
A Balanced Approach
While AI can efficiently process large datasets and identify broad patterns, it has its limitations. Qualitative research is inherently complex, involving different epistemologies, lenses, and theoretical frameworks that AI alone cannot fully apply. Human insight is crucial in interpreting nuanced meanings, understanding context, and negotiating the deeper significance of data. Yet, I would argue that there is a time and place for AI in qualitative research— when the free-text data is overwhelming for human analysis (like making sense of the voluminous and messy social media data) and/or when rapid analysis is important, such as during a global pandemic.
If you are interested in meaningful AI-Human collaboration, please get in touch. I have started a journey to bring together what might be seen as divergent disciplines, data science, social science, humanities, and whatever different disciplinary labels you might associate yourself with. Join me on the journey!
Acknowledgments and gratitude
Thank you to Sarah Riley and Siobhán Healy-Cullen for hosting me at Massey University. I would also like to express my gratitude to all the students who participated in the workshop, and other academics who attended, including Antonia Lyons and Chris Stephens.
About the Authors
Paulina Bondaronek. I am a multidisciplinary research fellow with a background in health psychology. I lead the HUMBLE project, which integrates natural language processing, qualitative analysis, and health psychology to analyse large datasets that might not be manageable by qualitative researchers alone but can be facilitated by AI. I also focus on social inequalities and bias mitigation. I believe that technology has the potential to bring the voices of underserved communities and minorities to the surface, if we approach the development of AI models with transparency, openness, and, most importantly, in collaboration with both researchers and the community.
Siobhán Healy-Cullen is a post-doctoral researcher in critical health psychology at Massey University Te Kunenga ki Pūrehuroa, Aotearoa New Zealand. Siobhán uses critical, qualitative methods to explore sexual identities and sexual citizenship.
More about MATA here:
1Bondaronek, P., Papakonstantinou, T., Stefanidou, C., & Chadborn, T. (2023). User feedback on the NHS Test & Trace Service during COVID-19: the use of machine learning to analyse free-text data from 37,914 UK adults. Public Health in Practice, 100401.
2 Towler, L., Bondaronek, P., Papakonstantinou, T., Amlôt, R., Chadborn, T., Ainsworth, B., & Yardley, L. (2023). Applying machine-learning to rapidly analyze large qualitative text datasets to inform the COVID-19 pandemic response: comparing human and machine-assisted topic analysis techniques. Frontiers in Public Health, 11, 1268223.