Incident Response Playbook for Linux/Unix Intrusion Detection (Updated 2025)

This Incident Response Playbook outlines a structured process for detecting, containing, remediating, and recovering from Linux or Unix-based intrusions. It incorporates modern security practices such as the use of Endpoint Detection and Response (EDR), advanced forensic tools, and threat intelligence integration to enhance incident handling capabilities.

1. Preparation

Objective: Establish procedures, deploy tools, and document critical system configurations to ensure efficient handling of a Linux/Unix intrusion.

Key Actions:

1. Deploy Security Tools:

   • Implement EDR on Linux/Unix systems to monitor and alert on security events.

   • Install auditing tools such as Auditd, AppArmor, and SELinux.

   • Ensure that intrusion detection systems (e.g., Snort, Suricata) are operational.

2. Maintain Documentation:

   • Keep a list of critical services, ports, and file hashes for system binaries.

   • Document expected system activity, including normal user access patterns and resource usage.

3. Define Response Procedures:

   • Create a response plan that includes roles and responsibilities for SOC analysts, IT administrators, and other stakeholders.

   • Establish secure communication channels for incident coordination.

4. Implement Log Management:

   • Configure centralized logging using syslog or Logstash.

   • Ensure that key logs (e.g., /var/log/auth.log, /var/log/syslog) are monitored and retained according to compliance requirements.

2. Identification

Objective: Detect suspicious activities, determine the scope of the intrusion, and escalate the incident as necessary.

Detection Techniques:

1. Unusual Accounts:

   • Check for unexpected entries in /etc/passwd, /etc/group, and /etc/shadow.

   • Identify orphaned files that may have been created by deleted accounts:

     find / \( --nouser -o --nogroup \) --print

Unusual Files:

• Search for SUID and GUID files, which may have been modified to gain elevated privileges:

find / -uid 0 \( -perm -4000 -o -perm 2000 \) -print

• Check directories like /proc and /tmp for hidden or unusual files.

• Unusual Services and Processes:

  • Review running processes and services:

ps aux chkconfig --list

• Investigate unknown processes using lsof:

lsof -p <PID>

Network Activity:

• Detect sniffers and suspicious network traffic:

ip link | grep PROMISC netstat -antp lsof -i

Look for unusual IP addresses or MAC entries in the network.

Log Analysis:

• Search logs for anomalies such as failed login attempts or unexpected reboots:

grep "authentication failure" /var/log/auth.log

dmesg | grep "promiscuous mode"

• Review key log files in /var/log.

3. Containment

Objective: Limit the attack's impact by isolating affected systems and preserving evidence for investigation.

Steps:

1. Isolate Systems:

   • Disconnect compromised systems from the network using EDR or firewall rules.

   • Disable remote access services (e.g., SSH) if necessary.

2. Preserve Evidence:

   • Create bit-by-bit images of hard drives and capture volatile memory (RAM).

   • Save important log files and system snapshots to secure storage.

3. Monitor for Lateral Movement:

   • Use network monitoring tools to detect any signs of further propagation.

   • Block suspicious IP addresses and domain names.

4. Remediation

Objective: Remove the threat and secure the environment against future intrusions.

Steps:

1. Remove Malicious Artifacts:

• Delete or quarantine all identified malware and backdoors.

• Terminate malicious processes and clear any unauthorized scheduled tasks:

crontab -u root -l

cat /etc/crontab

1. Revoke Compromised Access:

   • Disable compromised user accounts and reset all passwords.

   • Update SSH keys and disable unused services.

2. Apply Security Patches:

   • Patch all known vulnerabilities that may have been exploited.

3. Restore System Integrity:

   • Verify file integrity using checksums (e.g., SHA256).

   • Replace compromised binaries with verified versions from secure sources.

4. ---

   5. Recovery

   Objective: Restore affected systems to normal operations and verify that the environment is secure.

   Steps:

   1. System Reinstallation (Best Practice):

      • If feasible, perform a clean reinstallation of the operating system and applications.

   2. Account and Credential Security:

      • Enforce secure password policies and implement multi-factor authentication (MFA).

      • Rotate all credentials, including service account passwords and API keys.

   3. Monitoring and Validation:

      • Conduct a post-remediation scan using updated security tools.

      • Monitor system logs and network traffic for signs of residual threats.

5. ---

   6. Lessons Learned

   Objective: Improve the organisation’s security posture by documenting the incident and refining response processes.

   Steps:

   1. Incident Reporting:

      • Create a detailed report including:

        • Detection timeline and key events.

        • Actions taken and their effectiveness.

        • Indicators of compromise (IOCs).

   2. Root Cause Analysis:

      • Determine how the intrusion occurred and identify gaps in security controls.

   3. Process Improvements:

      • Update incident response playbooks and detection rules.

      • Enhance employee training and awareness programs.

   4. Security Enhancements:

      • Implement stronger access controls, network segmentation, and regular vulnerability assessments.

6. ---

   Conclusion

   This Linux/Unix Intrusion Detection Playbook provides a comprehensive approach to handling security incidents on Linux/Unix systems. Regular updates to tools, playbooks, and threat intelligence are crucial to maintaining effective incident response capabilities.

   For additional resources, such as forensic tool recommendations, log analysis scripts, or integration guides for EDR platforms, please reach out!