Gensim Applications

Introduction

Gensim is an open-source Python package for natural language processing used mainly for unsupervised topic modeling. It uses state-of-the-art academic models and modern statistical machine learning to perform complex NLP tasks. In terms of convenience, Gensim is much better than other libraries, such as sklearn. The one reason why Gensim stands out from other libraries is that it can handle large text files without even loading the whole file into the memory, and since it uses unsupervised models, Gensim in NLP does not require tagging of documents.

Features Of Gensim

• Robustness: Gensim is used in various systems over a wide range of applications.
• Scalability: Gensim is highly scalable. It uses incremental online training algorithms to contribute to this cause. While working with Gensim, the large text data does not need to reside fully in the Random Access Memory, which means the algorithms used are independent of the corpus size.
• Model Agnostic: Gensim is based on the Python language so it can be used in a variety of operating systems such as Windows, UNIX, LINUX, etc.

Uses of Gensim

• Word2Vec: This is a very popular natural language processing model that is used to produce word embeddings. Word embeddings are a form of representation of words available in the corpus in the shape of vectors. This model is a group of shallow and two-layer neural network models. The main purpose of word embeddings is to extract the linguistic context of each word in the corpus. Gensim provides a very easy way to implement word2vec using the Word2Vec model Code implementation

Output:-

We can also see similar words:-

Output:-

• Doc2Vec: This model is analogous to the word2vec model. The only difference is it is used for representing documents as a vector and is a generalization of the word2vec method. Gensim has the Doc2Vec model for this purpose which provides a vectorized representation of a group of words taken collectively as a single unit. Code Implementation

Now, the training data for the Doc2Vec model should be a list of tagged documents

Next, we train our model.

To get the document vector, we pass a list of words to the infer_vector() method.

Output:-

• fasttext: This neural network-based model was developed by Facebook’s AI Research (FAIR). Fasttext, like word2vec is also used to produce word embeddings by using supervised or unsupervised algorithms. Gensim has the FastText model to implement this.
• Topic modeling: Topic modeling is the process of recognizing words from different topics present in the corpus of data. It discovers the abstract topics that are present in a collection of documents. It helps in building a topic-per-document model and words-per-topic model, modeled as Dirichlet distributions. For the code implementation, we are using the "api8" dataset of Gensim and FastText model. First, we load the data:-

Then we preprocess the data.

Finally we use the model

Output:-

• Latent Dirichlet Allocation(LDA): This is one of the most popular methods for performing topic modeling. LDA aims to extract topics from documents on the basis of the words contained in them. For the purpose of LDA, Gensim has the LdaModel model. Code Implementation

Output:-

The above output shows the words that contributed to the seven topics, along with the weightage of the word’s contribution to that topic.

• Latent Semantic Analysis(LSA): This natural language processing method analyzes relationships between documents and their related terms. It uses the mathematical technique named Singular Value Decomposition , which finds hidden relationships between terms and concepts in unstructured data. We use LDA primarily for concept searching and automated document categorization. LSI is implemented in Gensim using LsiModel. Code Implementation

Output:-

• Compute Similarity Matrices: We can compute similarity against a corpus of documents by storing the index matrix in memory. This similarity measure used is the cosine between two vectors. Gensim has the gensim.similarities.MatrixSimilarity for the purpose of computing the similarity matrix. Sample code:-

Output:-

• Document Summarization: This process uses natural language processing techniques to generate a summary for documents. In other words, it is the task of rewriting a document into its shorter form while still retaining its important content. In Gensim, for the purpose of summarization, we use the from gensim.summarization.summarizer

Code Implementation

Here's what the document looks like:-

Output:-

Output:-