Acoustic Properties of Speech

Pre-requisites

Before learning about the acoustic properties of speech in NLP, let us first learn some basics about NLP itself.

• NLP stands for Natural Language Processing. In NLP, we analyze and synthesize the input and the trained NLP model then predicts the necessary output.
• NLP is the backbone of technologies like Artificial Intelligence and Deep Learning.
• In basic terms, we can say that NLP is nothing but the computer program's ability to process and understand the provided human language.
• The NLP process first converts our input text into a series of tokens (called the Doc object) and then performs several operations on the Doc object.
• A typical NLP process consists of various stages like tokenizer, tagger, lemmatizer, parser, and entity recognizer. In every stage, the input is the Doc object and the output is the processed Doc.
• Every stage makes some kind of respective change to the Doc object and feeds it to the subsequent stage of the process.

Introduction

As we know that NLP uses various stages to convert the user input text into a machine-understandable Doc object.

Tokenization is the first stage in which the model creates a Doc object out of the provided text and its main work is to convert the large text into segments and then converting into the form of tokens. The tagger component is the next step and it assigns the parts-of-speech tag to the tokens. The tagger component takes the Doc object as input and creates a Token.tag. The parser component is the third component in the entire process which parses the tagged tokens and then assigns the dependency labels to them. The lemmatizer component then assigns the base form to the labelled tokens. It is the overall process of the NLP.

Now, the Parts of Speech tagging is one of the prime stages of the entire NLP process. It helps in achieving a high level of accuracy in extracting the meaning of the word. It also helps us to remove the Word Sense Disambiguation. To perform the POS tagging, we need to use a dictionary of the language that we are working on. The dictionary contains all the POS tags and their meaning.

The parts of speech tagging are mainly governed by acoustic and lexical properties. Let us now discuss the acoustic properties in detail in the next section.

Properties of Speech

We have two types of modalities involved in human speech analysis- lexical and acoustic. So, depending on the use case, we use either of the two modalities. The lexical information modality is used to understand the opinion of human beings. But if we want to get the emotion of the voice, we use acoustic information. For example, with NLP, online translators can translate languages more accurately and present grammatically-correct results. This is infinitely helpful when trying to communicate with someone in another language. Not only that but when translating from another language to your own, tools now recognize the language based on inputted text and translate it.

We use the acoustic information along with the lexical information to train our machine model so that it can completely comprehend human speech with accuracy. These two properties can now be fused easily with the help of modern-day neural network architectures.

So to deal with the acoustic properties, we use Acoustic Language Processing.

Acoustic Processing of Speech

Let us now take the example of an experiment to understand the processing of acoustic properties of speech in NLP.

In an experiment (binary classification problem), we used the voices of more than 3,000 speakers to train a model. We mixed different types of emotions in a certain way. The happy and angry (not sad) emotions are mixed as high-arousal emotions. On the other hand, sad and happy emotions are mixed or grouped as low-arousal emotions. So, we have found out that if we mixed the acoustic properties along with the text, the impact would be positive.

In the processing of the data, the model used the ALP (Acoustic Language Processing) to combine the classical word embeddings and the acoustics to create certain data sets which can train the model with more accurate arousal.

In another experiment, we moved a bit forward with a ternary classification problem. We have three types of states: happy, sad, and angry. We first grouped the two negative valences i.e. sad and angry emotions. On the other hand, we treated the happy emotion as a single positive valence emotion. Now if we train the model using the ALP with the current data set, we can observe a major change in the error rate as compared to the previous experiment.

Computing Acoustic Probabilities

In the above first experiment, when we tried to use the same data set in a normal NLP model, we discovered that the Acoustic Language Processing training has shown 40% less error than the normal only-NLP model.

In the second experiment, when we tried to use the same data set in a normal NLP model, we discovered that the Acoustic Language Processing training has shown 33% less error than the normal only-NLP model.

By looking at the previous two experiments and their results, we can conclude that if we create an embedding conversation using the acoustic properties and Acoustic Language Processing, the model is better trained with less error than the normal NLP model.