This post was originally published here by Chris Sanders.
Processes are executed on systems in a variety of ways. A process might be executed by a user double-clicking an icon, by an automated service running at startup, or by a script referencing a third-party application. While not every attack requires the use of malware, most of them do require the execution of some type of process, even if it is a legitimate process being used for nefarious purposes. In this post, I’ll discuss how hunting for command line process executions can be a useful strategy for finding evil and how Sqrrl can make finding and investigating these events easier.
Command-Line Process Execution
Whenever an attacker gets a foothold on a system, they are typically interacting with it via the command-line interface rather than a GUI – at least at first. The CLI provides access to a robust set of tools that allow the attacker to browse the file system, read file contents, create accounts, connect to remote systems, created scheduled jobs, and even download and run arbitrary code.
If you’re logging command line execution, you’ll have visibility into executed processes. This visibility could come from Windows process auditing, Powershell command line logging, Sysmon process execution logs, bash command line auditing, or your third-party endpoint detection and response platform. Once you have this visibility it opens a lot of doors for investigating alerts, but that’s not all. It also provides ample opportunity for hunting through this data to find compromises not discovered by your IDS signatures.
Hunting for Odd Processes
Every investigation begins with a question. The question I’ll pose here is, “For systems on my network, did any unexpected or malicious processes execute from the command line?”
There are a couple of ways to go about this, but I think the best place to start is with a simple aggregation. An aggregation is a data transformation technique where you pick a specific field in a data set and collect all unique values of the field and the number of occurrences for each value. We can then take that data and sort it by the number of occurrences. I’m not too concerned about the processes with the most number of executions in this case as that is most likely comprised of legitimate scripts, startup processes, and application updates. Instead, I’m concerned about the least frequently executed processes as that list might reveal something unexpected. After all, most malicious commands are likely going to be used just a few times.
For this example, the network we’re working with has Carbon Black installed on their endpoints which means we’re provided with a treasure-trove of information. In our data model, command line process executions are mapped to the command_line field from the CarbonBlack data source. Given that, I can construct the following query using Sqrrl Query Language:
SELECT COUNT(*),command_line FROM CarbonBlack GROUP BY command_line ORDER BY COUNT(*) ASC LIMIT 20
This query selects the command_line field from the CarbonBlack data source and groups and counts all unique entries for that field. The results are sorted by the count, with the least frequent occurrences at the top.
Based on this output, a few things stand out to me. First, I see DNSChanger referenced several times. DNSChanger is malware that modifies the DNS cache of a system to point requests for common URLs to malicious servers. It looks like a system might be infected with this malware. Also of interest is what appears to be someone using the Linux command line tool sed to parse Cisco ASA logs. This could be a legitimate administrator troubleshooting a network issue, but it could also be an attacker attempting to learn more about the network. Both anomalies require further investigation, but now we have a launch point to jump from.
Tips for Investigating Suspicious Processes
What constitutes a suspicious process? In general, it’s anything you can’t explain. When you run this query the first time, there will be quite a few things that fit this description, especially if you’re not familiar with common host processes. As you gain proficiency in this type of hunting, you’ll develop an eye for things that might represent adversarial activity. In the meantime, here are things you can look for to help point you in the right direction:
- Processes that are named to look like legitimate process (svch0st.exe instead of svchost.exe)
- Processes whose names are random letters and numbers
- Legitimate applications that are often used by attackers for nefarious purposes (psexec, sc, wmic)
- User management commands used in unexpected places (useradd, net user)
- Script execution (powershell, python, perl, bash, etc)
- Downloads from the internet (wget, curl)
You can decrease the clutter in your hunting query results by filtering out the following:
- Administrative users who frequently execute processes via the command line
- Commands issues on systems of administrative users
- Whitelisted processes that are expected in your environment
One you’ve found something interesting, you should pivot off the computer name or IP address and look for evidence of other malicious activity. This might include other process executions, suspicious user accounts, or connections to suspicious URLs/IPs on the internet. The more robust your data, the more pivots you’ll have available.
Automating Data Retrieval with Reports
You can’t automate the investigation process, but you can automate the retrieval of data used for hunting input. This process aggregation output in this example is useful enough that it warrants examining on a regular basis. This makes it a great candidate to build a report from. By building a report in Sqrrl the aggregation is available with a single click.
Conclusion
Attacker success often hinges on the ability to execute processes on victim systems. By leveraging this and ensuring visibility into command line process execution, you can better enable yourself to spot malicious activity. In this article, I showed a simple aggregation that allows you to spot potentially malicious processes. By automating the collection of this data with Sqrrl, you’ll be better enabled to find post-compromise activity.
