Mini Case Study: Service Problem (service + logs)

In the real world you rarely get a “perfect error message.” Instead you get symptoms (service doesn’t respond, log spam, failed login). This case study shows a repeatable approach that stays mostly no-sudo.

Simulated case study (practice files)

This case study can be done either on your live system with systemctl/journalctl, or using the simulated snapshots in ~/playground/chapter9. The simulated version is useful for learning the reasoning, especially when some log files are blocked.

Scenario

“The web server isn’t responding” or “SSH logins fail.” You are allowed to inspect, but not to change system configuration.

Step 1: Check the service status (read-only if allowed)

# Try a status check (might be readable without sudo)
systemctl status sshd --no-pager

# Or for a web server on Red Hat, common units are httpd or nginx
systemctl status httpd --no-pager

Step 2: Pull the most relevant logs from the last window

# Live system approach (if journalctl works)
# For SSH (replace sshd with your unit)
journalctl -u sshd --since "30 min ago" --no-pager

# Only errors for that unit in the time window
journalctl -u sshd -p err..alert --since "30 min ago" --no-pager

# If you need to search for a keyword inside recent logs
journalctl -u sshd -g "failed" --since "today" --no-pager

# Simulated approach (no sudo, repeatable)
cd ~/playground/chapter9

# Show all sshd-related log lines from the simulated journal
sed -n '/sshd/p' journal_after.txt

# Show only error lines from the simulated journal
sed -n '/ERROR/p' journal_after.txt

Step 3: Connect logs to disk usage (common cause)

If the journal shows storage messages like no space left on device or repeated “write failed,” go back to the disk workflow:

• Run df -h to see which mount point is full
• Use du to find the largest directories under that filesystem
• Then inspect the folder that contains logs (often /var/log)

# Simulated "why did it start failing?" check:
cd ~/playground/chapter9

# Compare df output (what got fuller?)
diff -u df_before.txt df_after.txt

# Compare how /var changed
diff -u du_var_before.txt du_var_after.txt

# Extract the /var/log line to confirm where the space went
sed -n '/\/var\/log/p' du_var_after.txt

What to write down (for students)

• What was the symptom?
• What command narrowed it down to a service/time?
• What was the first relevant error you saw?
• Did it point to disk space, config, or permissions?