Malware types and analysis basics
In this series (16 parts)
- How attackers think: the attacker mindset
- Networking fundamentals for security
- Cryptography fundamentals
- Public key infrastructure and certificates
- Authentication and authorization
- Web application security: OWASP Top 10
- Network attacks and defenses
- Linux privilege escalation
- Windows security fundamentals
- Malware types and analysis basics
- Reconnaissance and OSINT
- Exploitation basics and CVEs
- Post-exploitation and persistence
- Defensive security: hardening and monitoring
- Incident response
- CTF skills and practice labs
Malware is software designed to do harm. It ranges from simple scripts that delete files to sophisticated programs that encrypt entire networks for ransom. Understanding what malware looks like, how it behaves, and how to safely analyze it is essential for incident responders, security analysts, and anyone who deals with compromised systems.
Prerequisites
You should understand how attackers think and basic Linux command line tools.
Types of malware
Virus
A virus attaches itself to a legitimate program or file. When the host file runs, the virus code executes too. Viruses need human action to spread (opening a file, running a program).
Worm
A worm is self-replicating malware that spreads automatically across networks without human interaction. It exploits vulnerabilities in network services to move from machine to machine. Notable example: WannaCry (2017) exploited an SMB vulnerability to spread across networks in minutes.
Trojan
A trojan disguises itself as legitimate software. The user installs it thinking it is useful, but it contains hidden malicious functionality. Unlike viruses, trojans do not self-replicate.
Ransomware
Ransomware encrypts files and demands payment for the decryption key. Modern ransomware also steals data before encrypting (double extortion: pay to decrypt AND pay to prevent data leak).
The encryption typically uses a combination of symmetric and asymmetric cryptography: files are encrypted with AES (fast), and the AES key is encrypted with the attacker’s RSA public key (so only the attacker can decrypt it).
Rootkit
A rootkit hides deep in the system, modifying the operating system to conceal its presence. Kernel-level rootkits are particularly dangerous because they can hide files, processes, and network connections from all userspace tools.
Spyware
Spyware silently collects information: keystrokes, screenshots, browsing history, credentials. It sends the collected data to the attacker’s server.
Static vs dynamic analysis
| Aspect | Static analysis | Dynamic analysis |
|---|---|---|
| Execution | Malware is NOT run | Malware IS run |
| Environment | Any machine | Isolated sandbox |
| Speed | Fast initial triage | Slower, requires setup |
| Evasion | Cannot detect runtime behavior | Can trigger anti-analysis checks |
| Tools | strings, file, hexdump, disassemblers | Sandboxes, debuggers, process monitors |
Static analysis: examining without executing
Step 1: File identification
# Determine file type
file suspicious_file
Output:
suspicious_file: PE32+ executable (GUI) x86-64, for MS Windows
# Calculate hash for threat intel lookup
sha256sum suspicious_file
Output:
a1b2c3d4e5f6...7890abcdef suspicious_file
# Check the hash against VirusTotal (using their API or web)
# https://www.virustotal.com/gui/file/a1b2c3d4e5f6...
Step 2: String extraction
# Extract readable strings
strings suspicious_file | head -50
Output:
!This program cannot be run in DOS mode.
.text
.rdata
cmd.exe
powershell.exe -enc
http://evil.com/beacon
/tmp/.hidden_config
CreateRemoteThread
VirtualAllocEx
WriteProcessMemory
HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run
Red flags:
- References to
cmd.exeandpowershell.exewith encoded commands - A URL that looks like a command-and-control (C2) server
- Windows API calls related to process injection (CreateRemoteThread, VirtualAllocEx)
- Registry paths for persistence (Run key)
- Hidden config files in
/tmp
Step 3: Examine metadata
# For PE files, check import tables
objdump -x suspicious_file | grep -A 5 "Import"
# For scripts
cat suspicious_script.ps1 | head -30
Dynamic analysis: observing behavior
⚠ Only run malware in an isolated sandbox environment. Never on your host machine, never on a network connected to production systems.
Common sandbox tools:
- Cuckoo Sandbox / CAPE: automated malware analysis
- Any.run: interactive cloud sandbox
- REMnux: Linux distribution for malware analysis
What to observe:
- File system changes (what files does it create, modify, delete?)
- Network connections (what IPs/domains does it contact?)
- Process creation (what child processes does it spawn?)
- Registry changes (Windows, for persistence)
- System calls
Example 1: Triage a suspicious file
You received a suspicious email attachment. Here is the triage workflow:
# Step 1: Identify the file
file attachment.pdf
Output:
attachment.pdf: PE32 executable (GUI) Intel 80386, for MS Windows
The file claims to be a PDF but is actually a Windows executable. This is a classic social engineering trick.
# Step 2: Hash it
sha256sum attachment.pdf
md5sum attachment.pdf
Output:
e3b0c44298fc1c149afbf4c8996fb924... attachment.pdf
d41d8cd98f00b204e9800998ecf8427e attachment.pdf
# Step 3: Check strings for indicators
strings attachment.pdf | grep -iE "http|cmd|powershell|exe|dll|registry|tmp|temp"
Output:
cmd.exe /c
http://198.51.100.23:8080/payload
C:\Users\Public\update.exe
HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run
powershell -WindowStyle Hidden -enc JABzAD0ATg...
Indicators found:
- C2 server at 198.51.100.23:8080
- Drops a file to C:\Users\Public\update.exe
- Adds registry run key for persistence
- Executes encoded PowerShell
# Step 4: Decode the Base64 PowerShell
echo "JABzAD0ATg..." | base64 -d
# Step 5: Search the hash on VirusTotal or other threat intel platforms
# If the hash is already known, you get detection names and behavioral reports
Example 2: Identify common malware indicators
Here are patterns to look for when analyzing files:
Network indicators (IOCs):
# Extract URLs and IPs from a binary
strings suspicious_file | grep -oP 'https?://[^\s"]+'
strings suspicious_file | grep -oP '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}'
Output:
http://evil.com/gate.php
http://198.51.100.23/update
198.51.100.23
203.0.113.42
Persistence indicators:
# Windows registry run keys in strings
strings suspicious_file | grep -i "currentversion\\\\run"
# Linux persistence in strings
strings suspicious_file | grep -iE "crontab|\.bashrc|systemd|init\.d"
Obfuscation indicators:
# High entropy sections (encrypted or compressed payloads)
# Check if strings output is very sparse compared to file size
echo "File size: $(wc -c < suspicious_file) bytes"
echo "Strings found: $(strings suspicious_file | wc -l) lines"
If a 500KB file only has 10 readable strings, it is likely packed or encrypted.
Behavioral categories to document:
| Category | Indicator | Meaning |
|---|---|---|
| C2 communication | URLs, IPs, DNS lookups | Calls home for commands |
| Persistence | Registry keys, cron, services | Survives reboot |
| Credential theft | Strings referencing SAM, shadow, LSASS | Stealing passwords |
| Lateral movement | SMB, WMI, SSH references | Spreading to other systems |
| Data exfiltration | Upload functions, external IPs | Stealing data |
| Anti-analysis | VM detection, debugger checks | Trying to evade sandboxes |
What comes next
The next article covers Reconnaissance and OSINT, where you will learn how attackers gather information before launching an attack, and what defenders can do about it.
For the incident response process that follows malware discovery, see article 15.
Future article ideas:
- Reverse engineering with Ghidra
- Writing YARA rules for malware detection
- Memory forensics with Volatility