What is System Logging in Linux?

Why are System Logs Important?

System logs are crucial for computer system management and system maintenance. Let's discuss some reasons why a System log in Linux is crucial.

Troubleshooting and Diagnostics: Logs document system faults and their causes. System administrators can troubleshoot and fix the problem by analyzing these logs.

Monitoring System Health: Logs help administrators track system health and performance. They can find patterns, spot anomalies, and prevent issues by analyzing logs routinely. This proactive strategy keeps the system running smoothly.

Security Analysis: System logs are essential for security analysis. They record login attempts, user actions, and system events. These logs allow administrators to identify security breaches, unauthorized access attempts, and questionable system activity.

Auditing and Compliance: Many businesses and organizations require log collecting and keeping. System logs record user and system events. This data is essential for regulatory compliance.

Capacity Planning and Resource Management: System logs show CPU, memory, disc, and network usage. Administrators can discover resource-intensive operations and plan capacity by analyzing these logs.

Software and Application Debugging: Developers and support teams need application logs to debug software faults. These logs help find and fix defects by detailing program behavior, faults, and exceptions.

Performance Optimization: System logs can optimize system settings and performance. Performance logs help administrators improve system responsiveness and efficiency.

Historical Analysis: Logs record system activity. Historical data is useful for trend analysis, pattern identification, and long-term system management and improvement decisions.

What is Syslog?

Linux logging system known as Syslog, allows devices, applications, and systems send log messages to a central logging server or store them locally. Syslog messages provide system events, activities, problems, and more.

Basic Syslog Terms

Some basic terms commonly used in the context of syslog are:

Syslog: Syslog, system log in Linux sends log messages from devices, applications, and systems to a centralized logging server or stores them locally.

Syslog Message: The kernel, applications, daemons, and services generate Syslog messages. The source, timestamp, and log content of each message describe the event or action.

Syslog Daemon (syslogd or rsyslogd): This background process receives syslog messages. It reads log messages from files or named pipes and listens to UDP or TCP ports. Based on the severity and facility codes, the syslog daemon transmits incoming messages to log files..

Facility: A numeric code categorizes logs by message source. Kernel messages, user-level applications, mail systems, and others have facility codes. Kernel communications have facility code 0, user-level messages have code 1, and so on.

Severity: Syslog message severity indicates its importance or urgency. From "emerg" (system unusable) to "debug" (debugging information), Syslog categorizes by severity. Syslog headers reflect severity.

Syslog Configuration Files: Configuration files regulate syslog daemon behavior. Older systems' main configuration file is " /etc/syslog.conf." On newer systems, like many Linux distributions, the syslog daemon is replaced by "rsyslog," and its configuration file is commonly " /etc/rsyslog.conf" or " /etc/rsyslog.d/*.conf". These configuration files describe log filters, log message storage locations, and log entry actions.

Log Files: Log files, system log in Linux hold Syslog messages. "/var/log" and its subdirectories hold Linux log files. "Messages," "secure," "auth.log," "kernel.log," "syslog," and application-specific logs are common.

Remote Syslog: Remote Syslog forwards Syslog messages between systems. Centralized logging and monitoring allow log messages from many systems to be managed and analyzed.

Log rotation: Log rotation involves archiving or compressing old log files and producing new ones to avoid them from taking up too much disc space. Log rotation is commonly handled by "logrotate."

Understanding syslog and Linux logging message management, starts with these basic terminologies.

Syslog Facilities

Syslog facilities are numeric codes that classify log messages by source. A facility identifies the process or component that created each log message. The facility code in the syslog message header adds context.

Syslog defines 24 facility codes with distinct numeric values. Common Syslog facilities and their numeric codes are:

Kernel Messages: Facility Code 0

These messages originate from the Linux kernel and provide information about low-level system activities and events.

User-level Messages: Facility Code 1

This facility is used for general user-level messages generated by applications and processes running on the system.

Mail System: Facility Code 2

The mail system facility is used to log messages related to mail servers and email-related events.

System Daemons: Facility Code 3

This facility is used to log messages from system daemons, such as cron (scheduler), syslogd (syslog daemon itself), etc.

Security/Authorization Messages: Facility Code 4

The security facility is used for logging security-related events, authentication attempts, and authorization information.

Internal Messages: Facility Code 5

The internal facility is used for logging messages related to the syslogd implementation and internal events.

Line Printer Subsystem: Facility Code 6

The line printer subsystem facility is used for logging messages related to printing and print spooling.

Network News Subsystem: Facility Code 7

This facility is used for logging messages related to network news servers and related events.

UUCP Subsystem: Facility Code 8

The UUCP subsystem facility is used for logging messages related to Unix-to-Unix Copy (UUCP) communications.

Clock Daemon: Facility Code 9

The clock daemon facility is used for logging messages related to the system clock and time synchronization.

Security/Authorization Messages (Private Use): Facility Code 10

Similar to Facility Code 4, this facility is reserved for private use and may be used for customized security-related logging.

Syslog Priorities

Priorities (sometimes called severity levels or log levels) categorize log messages by importance or urgency in the syslog system. Priorities assist identify the importance of log messages. To provide context, Syslog message headers include priority levels.

The Syslog protocol defines eight standard priority levels, each with a unique keyword and numeric value. Lower numbers indicate increased severity. Syslog priorities along with their corresponding keywords and numeric values:

Emergency (emerg) - Priority Code 0:

Represents a system-wide panic or catastrophic failure, where the system is unusable.

Example: Critical hardware failure or loss of network connectivity.

Alert (alert) - Priority Code 1:

Indicates an urgent condition that requires immediate attention.

Example: Software component or service failure that affects critical operations.

Critical (crit) - Priority Code 2:

Signifies a critical condition that demands immediate action.

Example: A service or application failure that impacts essential functionality.

Error (err) - Priority Code 3:

Indicates a non-urgent error condition that still requires attention.

Example: A configuration error or a resource limitation.

Warning (warning) - Priority Code 4:

Represents a non-urgent warning or notification.

Example: Non-critical system events or potential issues.

Notice (notice) - Priority Code 5:

Indicates a normal but significant condition.

Example: Normal system events or user actions that need attention.

Informational (info) - Priority Code 6:

Represents purely informational messages.

Example: System status updates or application activities.

Debug (debug) - Priority Code 7:

Used for debugging and development purposes.

Example: Detailed debug messages useful for troubleshooting.

Basic Commands for Linux Logging

Several basic Linux commands and utilities are used for logging and log management. View, manipulate, and analyze log files with these commands. Some key Linux logging commands are mentioned below.

View Logs - cat and less:

The cat command is used to display the content of log files directly in the terminal.

For example:

The less command is more versatile and allows you to scroll through large log files easily. It also provides search functionality. For example:

View Logs with Timestamp - tail:

The tail command is used to display the last few lines of a log file. By default, it shows the last 10 lines. It's useful for monitoring real-time log updates. To show the last 50 lines:

Follow Logs in Real-Time - tail -f:

The -f option with tail allows you to follow the log file in real-time as new log entries are added. This is particularly useful for monitoring logs as they update, such as watching system messages during troubleshooting.

Search Logs - grep:

The grep command is used to search for specific patterns or keywords in log files. It's helpful for filtering out relevant information from large log files. For example, to find all occurrences of "error" in a log file:

Log Rotation - logrotate:

The logrotate command is used for log file rotation, which compresses or archives old log files and creates new ones to manage disk space efficiently. It is typically configured to run automatically at scheduled intervals. For example, to manually trigger log rotation:

Check System Logs - dmesg:

The dmesg command displays the kernel ring buffer, which contains boot messages and other kernel-related information. It's useful for reviewing early system messages and hardware-related events.

View Specific Log - journalctl:

On systems using systemd, journalctl is used to view the system's systemd journal logs. It provides various filtering and querying options to access specific log information.

These basic Linux logging commands will help you access and manage log files, search for specific events, and monitor real-time log updates efficiently. Depending on the Linux distribution and log configuration, some commands may have slight variations or additional options.

The logger Command

The logger command in Linux is a utility that allows you to send log messages from the command line to the system's logging infrastructure. It is a convenient way to generate custom log entries, which can be useful for testing, debugging, or adding your own log messages to scripts or processes.

The basic syntax of the logger command is as follows:

Here, options are optional flags that modify the behavior of the command, and message is the log message you want to send.

Some commonly used options with the logger command are:

-p, --priority: Specifies the log priority (severity level) of the message. You can use standard syslog priorities like "emerg," "alert," "crit," "err," "warning," "notice," "info," or "debug." For example:

logger -p err "This is an error message."

-t, --tag: Sets a custom tag for the log message, which helps identify the source of the log entry. For example:

logger -t myscript "Custom log message from my script."

-f, --file: Reads the log message from a file rather than specifying it directly on the command line. For example:

logger -f /path/to/logfile.txt

-i, --id: Includes the process ID (PID) of the logger command in the log message. This can be useful when debugging or tracking multiple instances of the logger command.

-s, --stderr: Sends the log message to the standard error stream (stderr) instead of the system log. This is helpful when capturing log messages in scripts or capturing output in specific locations.

-u, --socket: Sends the log message to a specific UNIX socket for logging. This is typically used when a system has a custom logging setup.

Common Linux Logs and Their Locations

System, application, and service logs are created in Linux. These logs aid troubleshooting, monitoring, and security analysis. Common Linux logs and their default locations:

System Logs:

/var/log/syslog : General system messages and events.

/var/log/kern.log : Kernel messages.

/var/log/auth.log : Authentication-related messages (login, logout, authentication failures, etc.).

/var/log/daemon.log : Messages from system daemons.

/var/log/messages : A catch-all log file that may contain messages from various services and daemons.

Boot Logs:

/var/log/boot.log : Boot messages and information.

Package Management Logs:

/var/log/dpkg.log : Debian package management system (APT) installation and removal logs.

/var/log/yum.log: YUM package manager installation and removal logs (used in RPM-based systems).

Authentication Logs:

/var/log/wtmp : Records successful user logins and logouts.

/var/log/btmp : Records failed login attempts.

/var/log/lastlog : Stores the last login information for each user.

Security Logs:

/var/log/auth.log : Authentication-related events, including user login attempts, password changes, etc.

/var/log/secure : Similar to auth.log, found on some Linux distributions.

Cron Jobs Logs:

/var/log/cron : Records cron job activities.

Application Logs:

/var/log/apache2/access.log and /var/log/apache2/error.log : Apache web server access and error logs.

/var/log/nginx/access.log and /var/log/nginx/error.log : Nginx web server access and error logs.

/var/log/mysql/error.log : MySQL database server error log.

/var/log/maillog : Mail server log (Postfix, Sendmail, etc.).

Systemd Journal (Available on systems using systemd):

/var/log/journal/ : systemd journal logs that contain detailed information about various system events.

Kernel Ring Buffer:

/var/log/dmesg : Displays the kernel ring buffer, containing boot messages and kernel-related events.

Application-specific Logs:

Various applications may have their own log files located in /var/log or within their installation directories. To view log contents, you can use commands like cat, less, or tail.

Learn More

Troubleshooting in Linux Using Syslog, DMESG, & Sar

To learn more about Linux follow our Linux Tutorial