With the increasing popularity of open-source products, it is crucial for a backend engineer to be able to clearly identify whether an abnormal machine has been compromised. Based on my personal work experience, I have compiled several common scenarios of machines being hacked for reference.

This article is first published in the medium MPP plan. If you are a medium user, please follow me in medium. Thank you very much.

1. Intruders May Delete Machine Logs

Check if log information still exists or has been cleared using the following commands:

2. Intruders May Create a New File for Storing Usernames and Passwords

Check /etc/passwd and /etc/shadow files for any alterations using the following commands:

3. Intruders May Modify Usernames and Passwords

Examine the contents of /etc/passwd and /etc/shadow files for any changes using the following commands:

4. Check Recent Successful and Last Unsuccessful Login Events on the Machine

Refer to the log “/var/log/lastlog” using the following commands:

5. Use who to View All Currently Logged-in Users on the Machine

Refer to the log file “/var/run/utmp”:

6. Use last to View Users Logged in Since Machine Creation

Refer to the log file “/var/log/wtmp”:

7. Use ac to View Connection Time (in Hours) for All Users on the Machine

Refer to the log file “/var/log/wtmp”:

8. If Abnormal Traffic is Detected

Use “tcpdump” to capture network packets or “iperf” to check traffic.

9. Review the /var/log/secure Log File

Attempt to identify information about intruders using the following commands:

10. Identify Scripts Executed by Abnormal Processes

a. Use the top command to view the PID of abnormal processes:

b. Search for the executable file of the process in the virtual file system directory:

11. File Recovery After Confirming Intrusion and Deletion of Important Files

1. When a process opens a file, even if it’s deleted, it remains on the disk as long as the process keeps it open. To recover such files, use lsof from the /proc directory.

2. Most lsof information is stored in directories named after the process’s PID, such as /proc/1234, containing information for PID 1234. Each process directory contains various files providing insight into the process’s memory space, file descriptor list, symbolic links to files on disk, and other system information. lsof uses this and other kernel internal state information to generate its output.

Using the information above, you can retrieve the data by examining /proc/<PID>/fd/<descriptor>.

For example, to recover /var/log/secure, follow these steps:

a. Check /var/log/secure, confirming its absence:

b. Use lsof to check if any process is currently accessing /var/log/secure:

c. From the information above, PID 1264 (rsyslogd) has opened the file with a file descriptor of 4. It’s marked as deleted. Therefore, you can check the corresponding information in /proc/1264/fd/4:

d. You can recover the data by redirecting it to a file using I/O redirection:

e. Confirm the existence of /var/log/secure again. This method is particularly useful for many applications, especially log files and databases.

The above is the method I summarized for dealing with Linux intrusion. It can generally handle most problems. If you encounter an unresolved issue, it is best to seek advice from a professional IT operations and maintenance engineer.

I may not have written it completely correctly, so if you have different opinions, please leave a comment and let me know.