Week 8 – Advanced Pen Testing Paper
Hunting takes cybersecurity to the next level by making it an active process in which security analysts search for indicators of cyber attackers and go in trying to track down and “hunt” their prey, which, in this case, are cyber-attackers (Ashford, 2015). In anticipating opposition to cyberattacks, organizations can build stronger defenses, because they can find and fix vulnerabilities in their networks and systems before they are attacked maliciously. Proactive defense is key to mitigating operational risk, because cleaning up the aftermath of an attack is much more costly than proactive defense strategies. Nowadays, there is special, sophisticated technology and highly skilled analysts that actively patrol network environments and locate abnormalities and indicators of compromise. According to Lee, cyber hunting is a “focused and iterative approach to searching out, identifying, and understanding adversaries internal to the defender’s networks” (2016).
Hunters typically look at all processes, tools, commands, and network file shares that are running in an environment to find potential vulnerabilities that typical security systems, like firewalls, antiviruses, etc., would miss because they are not malicious in and of themselves, but a trained eye can recognize if something is inappropriate, unlikely, or unusual, which can signal that something is wrong. According to an interview by Computer Weekly, Ben Johnson of Bit9 + CarbonBlack says that this innovation in cybersecurity arose because large, well-resourced companies are getting hacked on a daily basis (Ashford, 2015). Because attackers are always innovating and evolving their capabilities, there must also be innovation and evolution of defense capabilities. Hunting typically involves the most enthusiastic, passionate, and security driven security analysts, because it is these individuals that enjoy proactively investigating and not waiting for alerts or emergency calls to come in. They know how to think like an attacker, act like an attacker, attack like an attacker, how to communicate with the attackers, and the good ones can even infiltrate cyber criminals’ minds and organizations to learn their techniques and find out what their plans and deeds are. For example, according to another article by Ashford, many hunters that work for security companies, such as RSA FraudAction, do this, and are long-standing members of hacker forums, talking directly to hackers (2016). This kind of proactive security is a bit extreme, and as such, these actions are carried out by only the most dedicated hunters. At the most basic level, hunters are looking for abnormal, unusual or suspicious behavior, especially in relation to high-value data assets, wherever there is risk and attackers may be active (Ashford, 2015), which could be anywhere on a network, at any time, with or without real login information or administrator privileges.
One of the reasons hunters must exist and are in high demand is because attackers can mask their attacks to look like normal network and/or system usage, which doesn’t get flagged by automated security systems. For example, when an attacker steals valid user credentials and uses them to log on to a network or network device, it is difficult to detect them because there is no malware or malicious code; it simply looks like a user has logged in to their account. A hunter would look for multiple logins at the same time. A hunter could look for the terminal or command line command to pull password hashes into a file, like the bkhive command, which dumps the syskey bootkey from a Windows system hive, and the samdump2 command, which dumps Windows (up to Vista) passwords and hashes. This command is not a command that a typical user would know, and so a hunter could collect all processes and commands running on all endpoints of a network, making it possible to identify compromised computers by tracking commands, like the aforementioned Windows command, that most people don’t know about.
Advanced persistent threats present significant challenges to the security community and changes how organization need to view, implement and manage security operations, according to Rackspace (2017). Advanced persistent threats occur when attackers breach data infrastructure by continuously targeting the infrastructure. Attackers then remain within that infrastructure, undetected, to locate and access valuable information, and as Daniel Clayton, a former British intelligence officer who now serves as a director of security operations at Rackspace, describes, advanced persistent threats are typically “groups of individuals that have the resources and manpower to persistently target a company or organization 24 hours a day for as long as it takes to get the job done” (Rackspace, 2017). While prevention measures, like web application firewalls, intrusion detection and preventions systems, and anti-virus software, can be effective against some attacks, like DDoS, viruses, Trojans, and other attacks that remain consistent across all platforms, the reality of advanced persistent threats has made many of these measures obsolete in the modern world of cyber security. Effective security now requires organizations to be in the mindset that they will be attacked continually and, as such, actively scan their systems, networks, and endpoints for malicious activity.
Lee states that the “formal process of threat hunting should not be confused with an attempt to prevent adversaries from breaching the environment or for defenders to eliminate vulnerabilities in the network” (2016), but I would partly disagree on this statement when taken as is, because, although threat hunting in and of itself is just hunting, the idea behind any security operation is to remediate threats and make systems and networks less vulnerable by patching and, ultimately, removing vulnerabilities and preventing attackers from breaching systems. He is right in the sense that hunting itself is just that, hunting – digging through the network and searching for traces of attacks. This said, it is bad practice for an organization to isolate any one department or group of individuals, because when the teams work together, better results can be achieved. For example, if a hunter found traces of an attack, but didn’t do anything about it, or even eliminated the attacker from the network, the vulnerability that the attacker originally exploited would still be open, and other attackers – or even the same one – would still be able to exploit that vulnerability. With this in mind, best practices would be to not only hunt for attacks but to also use that data to mitigate risk.
Organizations must collect high quality data in order to hunt for cyberattacks appropriately. They must also use appropriate tools to access and analyze this data. Finally, they must have analysts with the proper skills to use the data and tools to find security breaches and incidents. Bianco describes a hunting maturity model based on primarily the skills of the analysts, because they are the ones who turn data into detections (2015). He also states that the quality of the data that an organization routinely collects from its IT environment is also a strong factor in determining the HMM level. Better quality data and more of it allows expert hunters to find more results. How an organization collects and analyzes the data is important as well, but not as important, because there are many ways, as we will see later, to collect data, and many different types of data that can be collected which will yield great results. The hunting maturity model ranges from HMM0 to HMM4, HMM0 being the initial stage of maturity, and HMM4 being the leading stage of maturity. An HMM0 organization relies heavily on automated alerting tools, such as IDS, SIEM, or antivirus, and they may regularly update their systems as per best practices, but they routinely collect little or no data. Organizations that fall under the HMM4 category automate their successful data analysis procedures and routinely collect high levels of quality data. In HMM4, organizations will turn any successful hunting process into operational, automated detection. This process frees analysts from having to monotonously run the same processes over and over again. Instead they can spend their time concentrating on improving existing processes and creating new ones. This makes HMM4 organizations great at resisting malicious attacks, because it allows them to focus their efforts on creating new hunting processes. This results in constant improvement to the organization’s detection program as a whole, as well as allows them to be industry leaders (Bianco, 2015). This, in turn, improves the entire world of cybersecurity, by advancing lower HMM-level organizations’ security policies and automations. It should be noted that both HMM0 and HMM4 organizations carry out automation, but the automation that they carry out is different. HMM4 organizations always have automation in the front of their minds as they create new hunting techniques, whereas HMM0 organizations rely entirely on their automated detection, whether it’s provided by a vendor or created in-house. They may spend time improving their detection by creating new signatures or looking for new threat intel feeds to consume, but they are not fundamentally changing the way they find adversaries in their network. Even if they employ the most sophisticated security analytics tools available, if they are sitting back and waiting for alerts, they are not hunting. HMM4 organizations, on the other hand, are actively trying new methods to find the threat actors in their systems. They try new ideas all the time, knowing that some won’t pan out but others will. They are inventive, curious, and agile, qualities you can’t get from a purely automated detection product. Although a good hunting platform can certainly give your team a boost, you can’t buy your way to HMM4. Bianco recommends HMM2 for CISOs looking to start hunting operations (2015). HMM2 describes organizations that are able to learn and apply procedures developed by others, and may make minor changes, but are not yet capable of creating wholly new procedures themselves. HMM2 organizations must have schedules to apply security procedures on a regular basis.
A couple recommendations I would make for organizations looking to implement hunting operations would be to monitor endpoint process creation, as well as searching for indicators of compromise. Many organizations look for logs to analyze but as Carvey describes in his Dell SecureWorks presentation, a malicious attacker can repurpose syslog so that logs aren’t giving proper information, and this would have to be detected by monitoring for these processes (Carvey, 2015). Organizations should look for endpoint processes that show artifacts or indicators that malicious activity is occurring in the network. Indicators, like endpoint process artifacts, can show lateral movements in internal networks. Web shells can be used to gain access to an infrastructure, by compromising a web server, and then moving to internal systems. Examples include is a Windows server running Apache and WordPress or by manipulating an IIS server. An attacker can also gain access with a web shell to an SQL server from a web server. The attacker can gain access to a web server, put a web shell on it, and with RDP access on both servers, the attacker can access the web shell in Internet Explorer by connecting it to localhost. Then they use the web shell to issue SQL injection commands, using xp_cmdshell and then create a user account on the SQL server. This can be found by looking through the browsing history, to see where the attacker was accessing localhost. In this case, the organization wouldn’t have event logs, because the attacker deleted the web shell after they were done, but there would be logs in the web server. Another file system indicator can be found on IIS servers with ASPX web shells, because the first time it is accessed, the .NET framework creates a page called the_name_of_the_web_shell.compile. In other words, the framework actually compiles it. These are file system artifacts that a hunter should look for when looking for advanced persistent threats, because attackers can come in, install a web shell, delete it after use, and repeat this process as much as they want, all the while going undetected in the network, because they actually created a legitimate login to the SQL or IIS server. Only if someone was actively looking for those indicators would they find out that web shells had been installed by malicious users. If an attacker crashes a web browser, it will create a session restore file, which, if the attacker doesn’t reinitiate the browser to delete that file, will remain on the system. Parsing through a compromised system after it has been taken offline will allow the forensics team to find these files and see what commands were issued through the web shell, as well as the username and password that the attacker used to access the SQL server, because the username and password would get stored in a config file. Carvey states that clusters of indicators, not individual artifacts, should be looked for, “because there are a lot of things that go on within an infrastructure that, if you look at them in isolation from everything else, could look like threat actor activity, because a lot of the stuff that we see threat actors doing is stuff that a normal admin might do” (2015).
Process creation monitoring is useful in live detection of attacks being carried out. This enables security professionals to see commands used by attackers, as they are being used, like checking the time of the remote system, checking to see if the task is completed, reissuing the task. Hunters should look to see if a process was created, when it was created and compare that to the hours of operation of that organization or the working hours of the person that normally uses that endpoint and other clusters of indicators like registry keys, passwords that were used, event logs, file systems, etc. Take, for example, the sticky keys attack. In the Windows registry, there is a key called image file execution options with spaces between all the words, that Microsoft left in place so that users can add debugging capabilities to binaries. An attacker can modify this registry key, via RDP access to the system, with the reg.exe command line utility. The attacker creates a subkey for one of the two accessibility tools, hc.exe or utilman.exe, and points the debugger value to cmd.exe. Even if all the passwords in the organization’s infrastructure are changed, and the attacker can still access the infrastructure, all they must do is RDP to that system, and when the login screen shows up, instead of inputting credentials, they just hit the shift key five times, and get a system level command prompt. Attackers use command line tools to do anything on a system. Once in, they can create users, change passwords, dump passwords, and anything else. The only way to detect this is to monitor for process creation and see that cmd.exe is being launched in places that it shouldn’t be, perhaps at times or on systems that should show no use, or on systems on which users should not be launching cmd.exe. Another suggestion I would make is to make use of shimcache and amcache. This allows systems administrators to see what has been run on a system and when and for how long. This can be started by running it through Python directly or by making a Windows EXE from the Python script, provided on its GitHub Page, https://github.com/mandiant/ShimCacheParser. ShimCache data should be collected and analyzed from all Windows endpoints in an organization, both clients and servers. Servers are particularly important, because they are “usually the number one initial entry point for breaches, especially internet-facing servers, or other servers and DMZs,” says David Sharpe, in his DerbyCon 2015 talk (Sharpe, 2015). Amcache replaced Shimcache, starting with Windows Server 2012 and Windows 8, and provides the same function as Shimcache, but has more useful fields for hunting, such as an SHA1 hash of the file, as well as more useful timestamp fields. Data from these caches should be stacked and analyzed for sequences of recon activity, net commands, pings, archivers, like RAR, being ran, and EXEs running out of abnormal locations on the disk. An example would be if an Amcache timeline were created, and EXEs were found, being run from the C:users location, this would be an abnormal location for EXEs to be run, as this does not normally occur.
I would also recommend mining server antivirus logs, because they are a consistent, high yield data source to hunt for intrusions, which is especially true for internet-facing assets. According to Sharpe, about 20% of all targeted intrusions have AV fire somewhere along the timeline (2015). If an intrusion attempt has progressed far enough along to where an AV product triggers, then that is helpful. At best, there will be a blocked intrusion, but there will still be an exploitable hole that needs to be addressed. The worst case scenario is that the intrusion is far along and AV picked up one tool in a long series of events that need to be addressed. Things to look for include web shells, AV detections while the file is under webroot or C:windows, any kind of backdoors and malware street names identified by intelligence sources and experience. This should be supplemented by custom host intrusion prevention systems detection, with HIPS rules targeting how malware tools work. Look for credential dumpers, like WCE, pwdump, gsecdump, fgdump, or Mimikatz.
Netstat data should be mined to find rogue listeners across all endpoints, especially servers on the network edges. The command netstat -nabo to pull the data and mine it. An example of an indicator of compromise could be if one TCP port has bound to it multiple process names and paths on a single system. This would be impossible on a normally-running system. In this case, intruder activity could be interleaved with legitimate SQL server activity. Netstat data output should be stacked for all internet-accessible servers by listening port, and see how many ports show up just once. This data should also be stacked by the full path to the process’ binary, and see how many paths show up just once. Additionally, all output should be preserved as a baseline, and all new listeners that appear over time, especially those across internet-facing systems, should be tracked (Sharpe, 2015).
There are many companies that offer proactive hunting services for fees, but I would recommend that an organization also have in-house hunters that are proactively seeking out cyberattacks. Outside consultation should be utilized in order to improve in-house hunting. In striving to be an organization with competent cybersecurity measures in place, the organization should collect very large amounts of data from across the enterprise and at all endpoints. All the suggestions I have made in this paper have involved compiling large amounts of data sets to find abnormalities in the operations of the organization. It is only with these data sets that we can analyze the data and find indicators of compromise.
Ashford, W. (2015, October 13). Cyber security innovation is crucial, says security evangelist. Retrieved December 15, 2017, from http://www.computerweekly.com/news/4500255332/Cyber-security-innovation-is-crucial-says-security-evangelist
Ashford, W. (2016, March). Hunters: a rare but essential breed of enterprise cyber defenders. Retrieved December 15, 2017, from http://www.computerweekly.com/feature/Hunters-a-rare-but-essential-breed-of-enterprise-cyber-defenders
Bianco, D. (2015, October 15). A Simple Hunting Maturity Model. Retrieved December 15, 2017, from http://detect-respond.blogspot.com/2015/10/a-simple-hunting-maturity-model.html
Carvey, H. (2015, July 25). BsidesCincy 2015 01 Lateral Movement Harlan Carvey. Retrieved December 15, 2017, from https://www.youtube.com/watch?v=dYoYMsJ5aIc
Lee, R. M. (2016, February). The Who, What, Where, When, Why and How of Effective Threat Hunting. Retrieved December 15, 2017, from https://www.sans.org/reading-room/whitepapers/analyst/who-what-where-when-effective-threat-hunting-36785
Rackspace. (2017, September 29). AGE OF THE CYBER HUNTER: HOW A NEW GENERATION OF THREATS CHANGED THE CYBERSECURITY PARADIGM. Retrieved December 15, 2017, from https://www.rackspace.com/sites/default/files/white-papers/age-of-the-cyber-hunter-white-paper_1.pdf
Rackspace. (2017). ENTERPRISE SECURITY TODAY – WHY SPEED MATTERS. Retrieved December 15, 2017, from http://go.rackspace.com/brand-deepdives26.html
Sharpe, D. (2015, September 28). Fix Me19 Intrusion Hunting for the Masses A Practical Guide David Sharpe. Retrieved December 15, 2017, from https://www.youtube.com/watch?t=1&v=MUUseTJp3jM