How to Analyse and Monitor MongoDB Logs?

How does MongoDB Store its Data?

MongoDB uses a flexible schema in which documents in a collection can have different fields and structures. In MongoDB, we can view the data in JSON format, but it stores its data in a collection of BSON(Binary JSON) documents. BSON is a binary-encoded serialization of JSON documents. The advantages of BSON over JSON for storage are less space on disk, support for more data types, and faster processing. Logs in MongoDB are stored as JSON files.

Logs in MongoDB

In MongoDB, logs are the recorded events that provide information on the activities, errors, and performance of the database. MongoDB logs are essential for monitoring and troubleshooting database operations. The most common logs on mongoDB are,

• The diagnostic log records information about the operation performed on mongoDB, such as database connection events, read/write operations, and server errors.
• The profiling log, on the other hand, tracks the queries that the MongoDB server executes and the time taken to complete each query.
• The audit log captures a record of all actions performed by MongoDB, such as authentication events, database changes, and access control events

MongoDB logs are stored in a file called mongod.log, which is located in the /var/log/MongoDB directory by default. The location of the file can be changed through the MongoDB.conf` file.

The following is the layout of a log entry in MongoDB, and this follows the Relaxed Extended JSON v2.0 format,

Monitoring Event Messages in MongoDB Logs

MongoDB logs contain various types of event messages that can be used to monitor database operations. Some of the most common event messages that should be monitored are,

• Connection messages indicate when a client connects or disconnects from the database.
• Query messages messages record the queries executed on the database.
• Indexing messages messages indicate when an index is created or dropped on a collection.
• Security messages refers to messages that are related to the authorization and authentication in the MongoDB instance.
• Replication messages messages indicate when replication operations are performed on the database.

MongoDB Log Messages

MongoDB log messages are categorized into different levels, including debug, info, warning, and error. Debug messages provide the most amount of information and are typically used for development and debugging purposes. Info messages provide information about normal database operations, while warning and error messages indicate potential problems or errors.

A startup warning log message will be like this,

The above log message is caused by the initandlisten thread with a severity of Warning, and the c field type is used to identify the category of the logged entry. Some components are COMMAND, ACCESS etc...

Introduction to Mtools

Mtools is a collection of Python-based command-line tools for analyzing MongoDB logs. It includes various tools, such as login, mlogfilter, and mplotqueries. These tools can be used to extract and filter log data, analyze query patterns, and generate visualizations. We can install tools through the pip manager using the following command,

The login tool can be used to extract metadata from MongoDB logs. It provides detailed information about the log file, such as the number of lines, the size of the file, and the date range of log entries.

The mlogfilter tool can be used to filter MongoDB log entries based on various criteria, such as time range, log level, and keywords. It allows you to extract specific log entries that are relevant to your analysis.

The mplotqueries tool can be used to visualize MongoDB log data. It generates various types of charts and graphs, such as query frequency histograms and query response time scatter plots. To analyze query performance, you can run the following command in your terminal,

The mlogfiler command separates logs regarding queries and the mplotqueris command is used to visualize the queries.

How to Visualise MongoDB Log Files?

Visualizing MongoDB log files can help you understand query patterns, identify performance bottlenecks, and optimize database operations. One of the most common visualization tools used for MongoDB logs is the mplotqueries tool.

To use the mplotqueries tool, you first need to install the mtools package. Once installed, you can run the tool with the following command:

This command will generate various types of charts and graphs, such as query frequency histograms and query response time scatter plots. To visualizes a certain type of log, you can use mlogfilter like the command in the previous section.