Defensive security: hardening and monitoring

Offense gets the headlines. Defense keeps the lights on. This article covers the practical side of defending systems: hardening them so attacks fail, monitoring them so attacks are detected, and building detection rules so you catch adversaries before they reach their objectives.

Prerequisites

You should understand how attackers operate, post-exploitation techniques, and Linux security basics.

CIS benchmarks

CIS (Center for Internet Security) benchmarks are consensus-based security configuration guides. They exist for every major OS, cloud platform, and application. Each benchmark contains hundreds of specific settings with rationale and implementation steps.

Key areas covered by CIS benchmarks:

# Example CIS checks for Ubuntu (manual)
# 1. Ensure /tmp is a separate partition with noexec
mount | grep /tmp
# Expected: /tmp on /dev/sda3 type ext4 (rw,nosuid,nodev,noexec)

# 2. Ensure SSH root login is disabled
grep "^PermitRootLogin" /etc/ssh/sshd_config
# Expected: PermitRootLogin no

# 3. Ensure password authentication is disabled for SSH
grep "^PasswordAuthentication" /etc/ssh/sshd_config
# Expected: PasswordAuthentication no

# 4. Check for world-writable files
find / -xdev -type f -perm -0002 2>/dev/null
# Expected: empty or very limited output

Automated tools like CIS-CAT or Lynis can run these checks:

# Lynis security audit
sudo apt install lynis -y
sudo lynis audit system

Output (summary section):

  Hardening index : 72 [##############      ]
  Tests performed : 256
  Plugins enabled : 2

  -[ Suggestions ]-
  * Set a password on GRUB bootloader to prevent unauthorized modifications [BOOT-5122]
  * Enable process accounting [ACCT-9622]
  * Install a PAM module for password strength testing like pam_cracklib [AUTH-9262]

Attack surface reduction

Every service, port, and user account is a potential entry point. Reduce the attack surface by:

# List listening services
ss -tlnp

For each service, ask:

1. Does it need to be running?
2. Does it need to be accessible from the network?
3. Is it the latest patched version?
4. Is it configured securely?

# Disable unnecessary services
sudo systemctl disable --now cups       # Printing (if not needed)
sudo systemctl disable --now avahi-daemon  # mDNS (if not needed)

# Remove unnecessary packages
sudo apt autoremove --purge

SIEM basics

A SIEM (Security Information and Event Management) collects logs from across your infrastructure, normalizes them, and lets you search, correlate, and alert on security events.

Common SIEMs: Splunk, Elastic SIEM, Microsoft Sentinel, Wazuh (open source).

What to feed into your SIEM:

• Authentication logs (SSH, application logins, Active Directory)
• Firewall logs
• Web server access and error logs
• DNS query logs
• Endpoint agent logs (EDR)
• Application logs

Log aggregation

Centralize logs so an attacker who compromises a system cannot simply delete the local logs.

# Configure rsyslog to send logs to a central server
# On the client (/etc/rsyslog.d/50-remote.conf):
*.* @@logserver.company.com:514    # TCP
*.* @logserver.company.com:514     # UDP

# Or use journald to forward to a remote server
# In /etc/systemd/journal-upload.conf:
# URL=https://logserver.company.com:19532

EDR concepts

EDR (Endpoint Detection and Response) agents run on every endpoint (workstation, server) and monitor:

• Process creation and command lines
• File system changes
• Network connections
• Registry modifications (Windows)
• Memory injection attempts

When suspicious activity is detected, EDR can:

• Alert the security team
• Isolate the endpoint from the network
• Kill the malicious process
• Collect forensic data

MITRE ATT&CK

MITRE ATT&CK is a knowledge base of adversary tactics and techniques. It maps what attackers do (techniques) to why they do it (tactics).

Tactics (the “why”):

Example 1: Map 5 MITRE ATT&CK techniques to log sources and detection rules

graph TD
  T1["T1059: Command-Line Interface"] -->|Detect via| L1["Process creation logs<br/>Sysmon, auditd"]
  T2["T1053: Scheduled Task/Cron"] -->|Detect via| L2["Crontab changes<br/>/var/log/syslog"]
  T3["T1021: Remote Services SSH"] -->|Detect via| L3["auth.log<br/>SSH connection logs"]
  T4["T1048: Exfiltration Over C2"] -->|Detect via| L4["Network flow logs<br/>DNS query logs"]
  T5["T1078: Valid Accounts"] -->|Detect via| L5["Authentication logs<br/>Unusual login patterns"]
  style T1 fill:#ef5350,stroke:#c62828,color:#fff
  style T2 fill:#ef5350,stroke:#c62828,color:#fff
  style T3 fill:#ef5350,stroke:#c62828,color:#fff
  style T4 fill:#ef5350,stroke:#c62828,color:#fff
  style T5 fill:#ef5350,stroke:#c62828,color:#fff

Detection 1: T1059 (Command-Line Interface)

What to detect: suspicious command execution (encoded PowerShell, reverse shells, recon commands).

Log source: auditd (Linux), Sysmon (Windows)

# Linux auditd rule: log all execve calls
sudo auditctl -a always,exit -F arch=b64 -S execve -k command_exec

# Detection query (in your SIEM):
# Alert when processes execute base64 decoding, netcat, or curl to suspicious URLs
grep -E "base64|ncat|nc -e|/dev/tcp" /var/log/audit/audit.log

Detection 2: T1053 (Scheduled Task/Cron)

What to detect: new cron jobs or scheduled tasks created by non-admin users.

Log source: syslog, cron logs

# Monitor crontab changes
grep "crontab" /var/log/auth.log

# Detection rule: alert on crontab edits by non-root users
# Filter: user != root AND action == "EDIT"

Detection 3: T1021 (Remote Services, SSH)

What to detect: SSH connections from unusual IPs, at unusual times, or to unusual accounts.

Log source: auth.log, journalctl

# Baseline: build a list of normal SSH source IPs
grep "Accepted" /var/log/auth.log | awk '{print $11}' | sort -u

# Detection: alert on SSH logins from IPs not in the baseline
grep "Accepted" /var/log/auth.log | awk '{print $11}' | sort -u | \
  while read ip; do
    grep -q "$ip" /etc/ssh/allowed_ips || echo "ALERT: Unknown IP $ip"
  done

Detection 4: T1048 (Exfiltration Over C2)

What to detect: large data transfers to external IPs, DNS tunneling, unusual traffic patterns.

Log source: network flow logs, DNS logs

# Detection: unusually large DNS responses (possible tunneling)
# Normal DNS response: ~100-500 bytes
# DNS tunneling: responses with encoded data can be much larger

# Detection: high volume of DNS queries to a single domain
grep "query" /var/log/named/query.log | awk '{print $NF}' | sort | uniq -c | sort -rn | head -10
# Alert if any domain has > 1000 queries per hour

Detection 5: T1078 (Valid Accounts)

What to detect: use of legitimate credentials in suspicious contexts.

Log source: authentication logs

# Detection: same account logging in from multiple geographic locations simultaneously
# Detection: logins outside business hours for accounts that never work late
# Detection: service accounts used for interactive logins

grep "Accepted" /var/log/auth.log | awk '{print $9, $11}' | sort | uniq -c
# Alert if a service account (e.g., deploy, www-data) has interactive SSH sessions

Example 2: Building a detection rule workflow

Step 1: Choose a technique from ATT&CK (e.g., T1053.003: Cron)

Step 2: Identify what log data you have

# Do we log crontab changes?
grep -c "crontab" /var/log/auth.log

Step 3: Write the detection logic

# Pseudocode for SIEM rule:
# IF source = auth.log
# AND message CONTAINS "crontab"  
# AND user NOT IN ["root", "deploy-bot"]
# THEN alert "Unexpected crontab modification"
#   severity: HIGH
#   include: user, timestamp, source_ip

Step 4: Test with known-good and known-bad events

# Simulate: legitimate cron edit
sudo crontab -e    # Should NOT trigger (user is root)

# Simulate: suspicious cron edit
sudo -u www-data crontab -e    # SHOULD trigger

Step 5: Tune to reduce false positives

Defense in depth

No single defense is sufficient. Layer your defenses:

1. Perimeter: Firewall, WAF, DDoS protection
2. Network: Segmentation, IDS/IPS, network monitoring
3. Endpoint: EDR, antivirus, hardening
4. Application: Input validation, authentication, secure coding
5. Data: Encryption, access controls, backup
6. People: Security awareness training, phishing simulation
7. Process: Incident response plan, patch management

What comes next

The next article covers Incident response, where you will learn the structured process for handling security incidents from detection to lessons learned.

For practical Linux hardening, see Linux security basics.