Important Linux Log Files You Must Be Monitoring on RHEL 9 and 10

Every problem on a Linux server leaves a trace somewhere: a failed login, a service that would not start, a package update that changed a library, a disk that started throwing errors. Knowing which log holds which trace is the difference between a five minute diagnosis and an afternoon of guessing. This guide covers the logs that matter on Red Hat Enterprise Linux 9 and 10, and on compatible systems such as Oracle Linux, AlmaLinux and Rocky Linux, with the commands we use to read them.

Two places logs live on RHEL: the journal and /var/log

On RHEL 9 and 10 almost everything is first collected by systemd-journald. The journal is a structured, indexed store that you read with journalctl. On a standard install, rsyslog then reads the journal and writes the familiar text files in /var/log: messages, secure, cron, maillog and boot.log. Some services keep their own files regardless, notably the audit daemon, the web servers, the databases and dnf.

Check that rsyslog is present and running before you rely on the text files. Minimal and container images may not include it:

rpm -q rsyslog
systemctl status rsyslog
# if it is missing:
dnf install rsyslog
systemctl enable --now rsyslog

Most files in /var/log are readable only by root, so run the commands below as root or with sudo.

The systemd journal

The journal is the first place to look, because it holds the output of every systemd unit, the kernel and anything that logs through syslog, all with precise timestamps and metadata you can filter on.

Commands worth memorising

# everything from one service, newest last
journalctl -u nginx

# follow a service live, like tail -f
journalctl -u sshd -f

# only errors and worse, from the current boot
journalctl -p err -b

# warnings and worse from the previous boot (useful after a crash)
journalctl -p warning -b -1

# a time window
journalctl --since "1 hour ago"
journalctl -u php-fpm --since "2026-10-22 08:00" --until "2026-10-22 09:00"

# kernel messages only (the replacement for reading a dmesg file)
journalctl -k -b

# search message text
journalctl -u sshd -g "Failed password"

# list the boots the journal knows about
journalctl --list-boots

Several -u options can be combined, which is handy when a request passes through more than one service: journalctl -u nginx -u php-fpm --since "10 min ago".

Make the journal persistent

On a default RHEL install the journal is kept in /run/log/journal, in memory, and it is gone after a reboot. That is exactly the moment you want it most. To keep it on disk, create a drop-in and restart journald:

mkdir -p /etc/systemd/journald.conf.d
vi /etc/systemd/journald.conf.d/persistent.conf

[Journal]
Storage=persistent
SystemMaxUse=2G

systemctl restart systemd-journald
journalctl --flush
ls -ld /var/log/journal
journalctl --disk-usage

SystemMaxUse caps the space the journal may use; pick a value that suits the disk. After the next reboot, journalctl --list-boots shows more than one entry and journalctl -b -1 works.

/var/log/messages: the general system log

This is the catch-all file: service starts and stops, kernel messages, hardware and driver warnings, network changes and anything else logged at info level or above that does not belong to the mail, authentication or cron categories. It is the text equivalent of a large part of the journal, and it is convenient for grep and for tools that expect plain files.

tail -f /var/log/messages
grep -iE "error|fail|oom" /var/log/messages | tail -50
grep -i "out of memory" /var/log/messages

A line such as Out of memory: Killed process means the kernel ended a process because the server ran out of RAM. The answer is to give the workload more memory or reduce what it uses (fewer PHP-FPM or database workers, a cache, fixing a leak), not to hide the symptom.

/var/log/secure: the Linux security log for authentication and privilege

Every SSH login, every failed password, every sudo and su, and PAM messages land here. On RHEL, Oracle Linux, AlmaLinux and Rocky Linux this is the file to watch for access to the server.

# failed SSH passwords and the addresses they came from
grep "Failed password" /var/log/secure | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head

# successful logins
grep "Accepted" /var/log/secure

# who used sudo, and for what
grep "COMMAND=" /var/log/secure

Two related tools complete the picture. last lists successful logins from /var/log/wtmp, and lastb lists failed ones from /var/log/btmp. If accounts lock after failed attempts, faillock --user alice shows the attempts and faillock --user alice --reset clears them.

/var/log/audit/audit.log: the audit trail

The Linux audit system records security relevant events at kernel level: logins, privilege changes, SELinux decisions and any file or system call you choose to watch. It is written by auditd and read with ausearch and aureport rather than grep, because those tools understand its format.

# summary of everything in the log
aureport --summary

# failed logins
aureport --login --failed

# authentication attempts
aureport -au

# today's events, with numeric IDs turned into names
ausearch -ts today -i

# failed login events only
ausearch -m USER_LOGIN -sv no -i

Watch the files that matter

Audit rules let you record every change to a sensitive file. Put them in a file under /etc/audit/rules.d/, for example /etc/audit/rules.d/50-watch.rules:

-w /etc/ssh/sshd_config -p wa -k sshd_config
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity

augenrules --load
auditctl -l
ausearch -k sshd_config -i

The last command shows who changed the SSH configuration, when, and with which program.

SELinux denials

RHEL runs with SELinux enforcing, and it should stay that way. When SELinux blocks something, the denial is recorded as an AVC record in the audit log. Look there before changing anything else:

getenforce
ausearch -m AVC,USER_AVC -ts recent -i
ausearch -m AVC -ts today | audit2why

For a plain language explanation with suggested fixes, install setroubleshoot and run sealert against the log:

dnf install setroubleshoot-server
sealert -a /var/log/audit/audit.log

Most denials have a simple cause: a file with the wrong label (fix with restorecon -Rv <path>), content placed in a non-standard directory (label it with semanage fcontext), or a feature that needs a boolean switched on (getsebool -a | grep httpd, then setsebool -P). Work through those before considering a custom policy module.

/var/log/cron: scheduled jobs

Every job started by crond and anacron is logged here, with the user and the command. When a backup or report "did not run", this file tells you whether it was started at all.

grep CROND /var/log/cron | tail -20
grep "(root) CMD" /var/log/cron | tail -20

Many RHEL housekeeping tasks now run as systemd timers instead of cron jobs. List them with systemctl list-timers and read their output with journalctl -u <name>.service, for example journalctl -u logrotate.service.

/var/log/maillog: mail delivery

Postfix and other mail software log every message they accept, relay, defer or bounce here. If a web application says it sent an email and nobody received it, the answer is usually in this file.

# the full path of one message, by recipient
grep "user@example.org" /var/log/maillog

# bounces and deferrals
grep -E "status=(bounced|deferred)" /var/log/maillog | tail -20

# what is still waiting in the queue
postqueue -p

Each Postfix message has a queue ID (for example 4F2A81C0B3E). Grep for that ID to see every step of its delivery, including the reply from the receiving server.

/var/log/dnf.log and dnf history: what changed

When something breaks after maintenance, the first question is what changed. On RHEL 9 and 10 package operations are handled by dnf, which writes /var/log/dnf.log (the dnf run itself), /var/log/dnf.rpm.log (each package installed, upgraded or removed) and /var/log/dnf.librepo.log (repository downloads). The quickest view is the history database:

# every transaction, newest first
dnf history

# exactly what one transaction did
dnf history info last
dnf history info 42

# every transaction that touched a package
dnf history list nginx

# packages changed recently, from the log file
grep -E "Install|Upgrade|Erase" /var/log/dnf.rpm.log | tail -30

# does anything need a restart after updates (dnf-plugins-core)
dnf needs-restarting -r

If an update has to be reversed, dnf history undo <id> rolls back a single transaction, provided the older packages are still available in the repositories.

/var/log/boot.log and the boot journal

/var/log/boot.log holds the start-up status messages you would see on the console. It is useful for a quick look, but the complete record of a boot is in the journal:

cat /var/log/boot.log
journalctl -b -p err
journalctl -b -1 -p err      # previous boot, needs the persistent journal
systemctl --failed           # units that did not start
systemd-analyze blame | head # what took longest

Web server and PHP logs

Each web server keeps its own access and error logs:

• nginx: /var/log/nginx/access.log and /var/log/nginx/error.log, plus any per-site files you define with access_log and error_log.
• Apache httpd: /var/log/httpd/access_log, /var/log/httpd/error_log, and ssl_access_log and ssl_error_log when mod_ssl is installed.
• PHP-FPM: /var/log/php-fpm/error.log for the FPM master and /var/log/php-fpm/www-error.log for PHP errors from the default pool. The pool can also write a slow log with a stack trace of every slow request once request_slowlog_timeout is set.

# top client addresses (default nginx log format)
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

# count of server errors (5xx) by URL
awk '$9 ~ /^5/ {print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

# the latest errors
tail -50 /var/log/nginx/error.log
tail -50 /var/log/php-fpm/www-error.log

A burst of 502 or 504 responses in the access log, matched by "upstream timed out" or "connect() failed" in the nginx error log, points at PHP-FPM: either all workers were busy or a script ran too long.

Database logs

• MariaDB (the database in the RHEL repositories): /var/log/mariadb/mariadb.log, set by log-error in /etc/my.cnf.d/mariadb-server.cnf.
• MySQL from the RHEL repositories: /var/log/mysql/mysqld.log.
• PostgreSQL: /var/lib/pgsql/data/log/, one file per weekday by default.

The error log shows start-up problems, crashes, corrupted tables and replication errors. For performance, enable the slow query log. For MariaDB, add to the [mysqld] section of /etc/my.cnf.d/mariadb-server.cnf and restart:

slow_query_log = 1
slow_query_log_file = /var/log/mariadb/mariadb-slow.log
long_query_time = 1

Then summarise it with mysqldumpslow -s t /var/log/mariadb/mariadb-slow.log | head -40 to see which queries cost the most total time.

Keeping logs under control with logrotate

RHEL rotates log files with logrotate, run once a day by a systemd timer (logrotate.timer). The global defaults are in /etc/logrotate.conf (weekly, four rotations, dated file names such as messages-20261018), and each package adds its own rules under /etc/logrotate.d/. The system files (messages, secure, cron, maillog) are covered by /etc/logrotate.d/rsyslog.

systemctl list-timers logrotate.timer
cat /etc/logrotate.d/nginx

# dry run: show what would happen, change nothing
logrotate -d /etc/logrotate.d/nginx

# force a rotation now, verbosely
logrotate -vf /etc/logrotate.d/nginx

For your own application, add a file such as /etc/logrotate.d/myapp:

/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp myapp
    postrotate
        /usr/bin/systemctl reload myapp.service >/dev/null 2>&1 || true
    endscript
}

Keep application logs under /var/log: files there get the SELinux label that logrotate is allowed to manage. If the log directory is writable by a user other than root, add an su myapp myapp line so logrotate can rotate it safely.

The journal rotates itself within the SystemMaxUse limit. To trim it by hand, use journalctl --vacuum-time=30d or journalctl --vacuum-size=1G.

Centralising logs with rsyslog

Logs that live only on the server they describe have two weaknesses: you have to log in to each machine to read them, and an attacker who gets root can edit them. Forwarding every server's logs to a central collector solves both, and lets you search one place during an incident. The collector can be another RHEL server running rsyslog, or a log platform such as Graylog.

On the collector

Create /etc/rsyslog.d/10-collector.conf. It listens on TCP port 20514, which the SELinux policy already labels as a syslog port (check with semanage port -l | grep syslogd), and writes one file per host and program:

module(load="imtcp")

template(name="RemotePerHost" type="string"
         string="/var/log/remote/%HOSTNAME:::secpath-replace%/%PROGRAMNAME:::secpath-replace%.log")

ruleset(name="remote") {
    action(type="omfile" dynaFile="RemotePerHost")
}

input(type="imtcp" port="20514" ruleset="remote")

Open the port only to your servers, here through the firewalld internal zone, then validate and restart:

firewall-cmd --permanent --zone=internal --add-source=10.0.10.0/24
firewall-cmd --permanent --zone=internal --add-port=20514/tcp
firewall-cmd --reload
rsyslogd -N1
systemctl restart rsyslog
ss -tlnp | grep 20514

If you choose a port that is not already labelled, add it first with semanage port -a -t syslogd_port_t -p tcp <port>, on the collector and on the senders. Add a logrotate rule for /var/log/remote/*/*.log as shown above.

On every server

Create /etc/rsyslog.d/90-central.conf. TCP with a disk-assisted queue means that a collector restart or a network interruption delays messages instead of losing them:

*.* action(type="omfwd" target="logs.example.net" port="20514" protocol="tcp"
           template="RSYSLOG_SyslogProtocol23Format"
           queue.type="LinkedList" queue.filename="fwd_central"
           queue.maxDiskSpace="1g" queue.saveOnShutdown="on"
           action.resumeRetryCount="-1")

rsyslogd -N1
systemctl restart rsyslog

To ship audit events as well, set active = yes in /etc/audit/plugins.d/syslog.conf and restart auditd with service auditd restart (auditd does not accept a restart through systemctl).

Prove it arrives

Send a test message and find it on the collector:

# on the server
logger -t logtest "central logging test from $(hostname)"

# on the collector
tail /var/log/remote/*/logtest.log

Repeat the test after every change to the collector or the firewall, and alert on silence: a server that has stopped sending logs is easy to miss, because nothing in the logs tells you about it.

A quick daily checklist

• journalctl -p err -b: anything new at error level since boot?
• aureport --login --failed and lastb | head: unusual failed logins?
• ausearch -m AVC -ts today: any SELinux denials?
• dnf history: anything changed that you did not plan?
• tail /var/log/nginx/error.log and the database error log: application trouble?
• journalctl --disk-usage and df -h /var/log: room to keep logging?

Reading these takes a few minutes. Automating them, through a central collector and alerts on the patterns that matter, is what turns logs from an archive into an early warning.

EuroVPS engineers build, run and watch log pipelines like this for customers on our managed servers.