Using NLP to collaboratively surface patterns of meaning in large corpora

I'm working on an empirical project with two co-authors, where a significant part of the central phenomenon we are studying unfolded online on twitter. 

We have a massive multi year tweet and factiva news corpus that we sense-make together by periodically meeting to discuss the data and to theorize qualitatively from algorithmically generated representations of and algorithmically sampled texts from the corpus, previously drawn models of the phenomenon, memos, manuscripts, etc. We each of us have varying familiarity/expertise around the special phenomenon, the substrate upon which it unfolds (online discourse), the theoretical/research conversation in which are participating, and NLP. In collaboration, we see things. 

I'm currently working on creating a visual representation of key themes (based on a BERTopic model) in the corpus, in particular the twitter discourse, over time and across our own interpreted actor categories. An interesting macro pattern emerged in conversation last time around a longitudinal heatmap, where one category of actors had a meaningfully different trend than the other categories. For the next meeting, I'm trying to tease out more detail to interpret together. But that's easier said than done. 

Note the time elapsed in the screenshot. My last run ended up taking ~6 hours to execute. I’m implementing the analysis in python (away from my beloved R, but I’m backfilling an NLP course at CBS this upcoming year that utilizes python, so this made sense).  I'm currently using Google colab for the compute. My main computer now (until August when I start at CBS and get access to some very nice toys) is a Macbook Neo, which while zippy doesn’t have the memory to handle the type of NLP algorithms we need for this effort. Colab is great because you have both free and for-pay cloud compute, so you don’t need the laptop or desktop you’re directly working on to have the juice. However, if you don't have a monthly subscription to Colab, you don’t get priority, and you can sometimes get booted from a session midway. Some of these algorithms take hours, even on beefy GPU/CPU configs, and woe betide if your laptop goes to sleep or Colab decides to allocate your compute to a subscriber midway through a run. I've been burned.

Anyhow, thought I'd noodle around on describing what each of the algorithms constitutive of the BERTopic method does to the text. This helps clarify my own understanding and ability to articulate what the algorithms are doing. This understanding and ability to articulate the computational aspects of computationally entangled interpretive research affords the human interpreter (even a non-technical one) a certain reflexivity about the how the phenomenon is being represented and thus interpreted. It is essential, I strongly assert, to the methodology.

Demystifying BERTopic

BERTopic is a semi-supervised machine learning method to extract human-interpretable "topics" or themes from a large text dataset. It is, in my estimation, an approach that will rise in prominence in management research in the next 5 years. It uses a few distinct algorithms. 

The pre-trained transformer algorithm SBERT converts each text (in this case a tweet) into a list of numbers, a location in a high-dimensional mathematical space (you can call it a hypercube if you're looking to impress/bore your friends) where tweets that mean similar things end up near each other: the result is a vector of 384 numbers encoding the text's semantic content. Each of the 384 numbers captures a different dimension of meaning. Less dimensions in the sense of something you could name cleanly, like sentiment, and more abstract features the model learned from vast amounts of text (labeled by humans, often by assigning into sets of sentences that contrast, paraphrase, imply, etc. each other). One number might encode something roughly like financial connotation, another something like technical register, maybe even something like formalism. No single number means anything on its own; meaning emerges from the full pattern of 384 values together, the way a color isn't in any single pixel but in the combination of red, green, and blue values. Two tweets that mean roughly the same thing will have similar patterns across all 384 numbers, even if they share no words at all. 

Now to identify locations in this space that are human-interpretable, we need to use clustering methods to find dense clumps of meaning, or perhaps, tweets that are about the same thing. However, 384 dimensions is computationally bonkers for clustering, and most of those dimensions are noise. The dimensionality reduction algorithm UMAP compresses the 384 dimensions down to 5 while preserving the neighborhood structure: tweets that were close together in the high-dimensional space should still be close together after compression. Think of it like flattening an intricate origami figure while trying to keep points that were touching each other in the 3D figure close to each other in the flat version. Unfolding it completely flat renders close points apart, while simply smushing it flat might damage or distort it. The solution, it turns out, is more origami.

Once the tweets are in that compressed 5-dimensional space, HDBSCAN does the actual clustering, or finding clumps of meaning in the hundreds of thousands of tweets or factiva articles. It's a density-based algorithm: it finds regions in the 5 dimensional space where tweets are packed closely together and calls those clusters or topics, while treating sparse regions as noise (or outliers). Unlike k-means, you don't tell it how many clusters or topics to find, it discovers that from the data's shape. This is unsupervised machine learning (the SBERT algo mentioned above is pre-trained on human-labeled text datasets, a core part of how it "knows" human languages, which is why I referred to this method as semi-supervised).

The BERTopic technique ends with using c-TF-IDF to identify which terms (words or two-word sequences) appear disproportionately in each cluster (or, which words best characterize each topic). 

Combined with information about time or when these topics were most prevalent in the corpus (in the context of our own knowledge of the broader phenomenon, which is somewhat exogenous to the data), identity or in which actors' discourse the topic is more present, representativeness or in which tweets these topics are most present, scope or how prevalent or restricted the participating actors or vocabularies related to the topic are, etc., we can interpret what the topic is about.

Progress note

Based on the above, I have been representing the two corpora as partitioned longitudinal topic heatmaps, where one can see the prevalence of the various topics change over time and across actor categories, paired with tables of central and specific words and bigrams for each topic, topic prevalence and coherence measures, representative tweets, as well as significant time periods for each topic in the twitter discourse.

As of the screenshot above I had been fighting an oversize Topic 0 problem where I get one gigantic discourse blob topic that is a bulk of the discourse, which is HDBSCAN's density detector failing to find enough dense clumps in the UMAP projection: most tweets end up in one sprawling low-density mass and very few separated into coherent clusters. I'm tinkering with UMAP on the compression and with how picky HDBSCAN is about what counts as a dense region worth clumping into a topic. But going is slow, each run takes a while.

EDIT: A day later, I did it! I have a satisfactory model with a much less dominant Topic 0 representing what I interpret as a sort of baseline discourse, with what appears to be many more coherent topics on top, rising and falling over time. Now on to making visuals that my co-authors and I can collaboratively interpret.