Topic modelling (before LLMs)

(Migrated from old blog)

As the name suggests topic modeling refers to the determination of the topic of the text. Imagine creating a news platform where the news that comes is automatically categorized based on the type of news it is, either it is technology, politics, weather or anything like that.

The input to the algorithm is a document-term matrix. Where each topic consists of a set of words where order doesn’t matter, So it is a Bag Of Word implementation.

We have two assumptions for Topic Modelling Algorithms that we are going to discuss,

• each document consists of a mixture of topics
• each topic consists of a collection of words

LSA: Latent Semantic Analysis

In this, we generate a matrix $A^{m\times n}$ where m is the number of documents and n is the number of words or terms. So each row represents a document and each column represents a word. Each cell has a raw count of the appearance of the word but the better solution is to have a tf-idf score of that word. and tf-idf is $w_{i,j}=t{f}_{i,j}\times \log \frac{N}{d{f}_j}$

Where $w_i,j$ is the tf-idf score. $t{f}_{i,j}$ Number of occurrences of term in document $N$ Total Documents $d{f}_j$ Number of documents containing words Once we have A we can start thinking about latent topics but the problem is that A is a very large, sparse and noisy matrix. To perform computation we will have to use Dimensionality Reduction. Let’s use truncated SVD for that. (More on SVD later or you can study using google )

So, $A=U_t{S}_t{V}_t^T$ where $U_t\in {\mathbb{R}}^{m\times t}$ is document-topic matrix and $V_t\in {\mathbb{R}}^{m\times t}$ is a term-topic matrix. In $U$ column represent $t$ topics and row represents document vector of topics. In $V$ column too represents $t$ topics and row represents term vector of topics. Here $t$ represents the truncated topics which are most significant dimentions in our transformed space.

With these documents, we can easily apply measures such as cosine similarities to evaluate.

• The similarity of different documents
• The similarity of different words

LSA is quick and easy to use but has few drawbacks

• lack of interpretable embeddings that is we don’t know whether the topics are arbitrary positive or negative
• We need huge datasets for it to perform well.
• less efficient representation

pLSA: Probabilistic Latent Semantic Analysis

pLSA is a Bayesian version of LSA. It uses Dirichlet priors for a document-topic and word-topic distribution. We assume how the text is generated is that you think of a topic and then for the words in that topic you pick a word and write it. The goal of this is to choose distributions such that the probability of the generation of the collection is MAX.

Okay So there is good math in this, let us talk about notations. $d$ is document and $w$ is word where $d\in D$ and $w\in W_d$ where $D$ is number of documents and $W_d$ is the word from the document d $n_{wd}$ count of word in document d ${\varphi }_{wt}$ is the probability of the word in topic. ${\theta }_{td}$ is the probability of the topic in that document.

According to the law of total probability, $P(w)=\sum _{t\in T}P(w|t)P(t)$

Similarly, $pLSA=\sum _{t\in T}P(w|t,d)P(t|d)=\sum _{t\in T}P(w|t)P(t|d)$ This is an assumption we make and it is called as assumption of conditional Independence i.e $P(w|t,d)=P(w|t)$ we don’t consider Probability of the word is dependent on the document, only dependent on topic.

We now will calculate the log likelihood and maximize its arguments: $\log \prod _{d\in D}p(d)\prod _{w\in d}P(w|d{)}^{n_{dw}}\to max(\varphi ,\theta )$

Which is equal to $\sum _{d\in D}\sum _{w\in d}{n}_{dw}\log \sum _{t\in T}{\varphi }_{wt}{\theta }_{td}\to max(\varphi ,\theta )$

We have normalized and non-negative constraints that means ${\varphi }_{wt}\ge 0\mathrm{\&}{\theta }_{td}\ge 0$ Also, $\sum _{w\in W}{\varphi }_{wt}=1\mathrm{\&}\sum _{t\in T}{\theta }_{td}=1$

So how it works, really… The Optimization generally used is E-M optimization. So we start with $$ P(t|d,w) = \frac{P(w| t,d)}{P(w|d)} = \frac{P(w|t)P(t|d)}{P(w|d)} $

The E step: $P(t|d,w)=\frac{P(w|t)P(t|d)}{P(w|d)}=\frac{{\varphi }_{wt}{\theta }_{td}}{\sum _{s\in T}{\varphi }_{ws}{\theta }_{sd}}$ The M Step: ${\varphi }_{wt}=\frac{n_{wt}}{\sum _w{n}_{wt}}{n}_{wt}=\sum _d{n}_{dw}P(t|w,d)$ ${\theta }_{td}=\frac{n_{td}}{\sum _t{n}_{td}}{n}_{td}=\sum _W{n}_{dw}P(t|d,w)$

The Algorithm looks like this:

http://www.machinelearning.ru/wiki/images/1/1f/Voron14aist.pdf