This post was originally published here by Sqrrl Team.
Danny Akacki currently works as part of a hunt team at a Fortune 100 company in the financial sector. He started threat hunting at Mandiant as part of David Bianco’s team, and has spent the past four years working in threat hunting and incident response.
Key Takeaways:
- Embrace “purple teaming.” The best SOCS have have red team and blue team analysts that closely coordinate with each other to share information.
- A good way to establish baselines for network behaviour is to use blogs to establish a timeline of events. This can serve as a useful jumping off point for pivoting through data.
- Hunting is useless without documentation. There’s no use going down rabbit holes without having data to feed back into your program. You need to be able to retrace your incident investigation steps.
How do you make the most out of the data you’ve collected when threat hunting? How do you gather high-fidelity indicators to guide future hunts? Recently, we sat down with Danny Akacki (@DAkacki) to talk about concrete steps that every SOC can take to improve their hunting maturity level.
Question: How did you make the transition from being a tier-one analyst to doing proactive threat hunting? Were there a lot of techniques or skillsets that you were already familiar with when you started?
Danny Akacki (DA): The whole methodology of being “proactive versus reactive” was not new, but to actually be doing it getting a feel for it was like, “Oh man, this is what this means, right?” I had worked in a couple different SOCs before that and Mandiant was my first startup because that was really when the whole hunting wheels got rolling.
I’d worked in a couple of SOCs which was pretty much just putting out fires all day. When you’re a level one analyst or level two analyst, you are always reacting. You get these alerts and notifications, which is triage. That’s following up something that cued you into something else. We decided to not wait for that anymore. That was no longer our charter to just wait around for these things to kind of slap you in the face. We really started to say, “Okay, let’s look past this. Let’s not wait for that. What is our threat landscape?”
Since I worked at Mandiant and then at GE and where I’m at now, it’s been important to know what your client’s risk levels and how bad guys would come at them. I’ve said it for a long time: hunting is finding ways for evil to do evil things. So, in that spirit you build these different use cases of “if they were to go after this resource, or if they were to use this tactic, and here’s the current new hotness in malware or lan exploit kit, how would this look?” Does it apply to us?. If it does, can we see it? No. You find out through your hunting, like, “Hey, we have a perceived gap in A, B and C locations. Okay, cool. Let’s attack that now.” Trying to stay ahead of the curve pretty much boils down to the whole reactive versus proactive approaches.
Q: One of the common questions we get is “how do I decide what to hunt for? Do you do it by data sets that you have available to you? Do you do it by kill chain analysis? Are you looking for gaps in your other detections systems? Are you trying to figure out where you’re most vulnerable or what your crown jewel assets might be? Was it kind of a mix of all of them?
DA: My answer to all of those is yes. Those are all factors in building out your hunting program in starting any kind of small hunting task. It really depends. It’s really hard to answer this as a blanket statement over multiple business verticals, over multiple companies because everybody values something differently. If you’re kind of at that low end of that hunting maturity model, you’re going to start looking at, “Okay, let’s just start from the beginning. What data do we have?” That comes from the very basic level of let’s figure out what we’re already getting, figure out how we can put it in one place and then have somebody to look at it. If you know you’re only getting some proxy logs and firewall logs and maybe some BPM logs, your approach is going to be vastly different from another org that is more mature that has logs coming from their endpoints and DNS. DNS sounds easy but in my experience and a lot of our experiences, that one can be tricky. Not everybody is even doing that– or doing it correctly.
It really comes from what level you’re at, if you’re just kind of starting out, like I said, you go and look for those things. As you mature and you’re finding those gaps in visibility and you’re getting more of those log sources and, okay, cool. Let’s build up some use cases around these. If we were going to have CQ beaconing, X fill, that kind of thing, we know that we have a really good baseline of all of our log sources. Let’s build up use cases around each one and how is this evil going to look coming across all of these different data services. Yeah, there’s just all of those things you mentioned at the very outset can be a starting point for depending on where your organization is at.
Q: Would you say that one of the skills of an analyst should be in deciding how you’re going to prioritize what you’re looking for based on that? That decision’s being made at a lot of different levels from the manager level.
DA: Right. It is. Your analysts, especially if you’re having analysts that are themselves mature enough to be doing the hunting, should have input into this. It’s also a really important factor here when you’re building up your program. We give all this great advice but doing this at scale gets harder and harder. Just because your analysts can see something doesn’t mean they own it. They’re going to have to get different leaders out of different spots and organizations on board with this saying, “Hey, here’s why we need this data. Can you help us get this?” Then also speaking to those different owners saying, “What’s important to you? What do you need to protecting? If this certain box got popped, if a bad guy got in, what’s going to do the most damage for you? Going all the way up to your C levels, what are the company’s crown jewels?”
In the bigger orb that you’re in, those answers are really going to vary. A lot of times it could be a very daunting task. So, where do you start? Let’s pinpoint this down and say, “These are our most important things. Okay, let’s start building our program around these things and guarding these things,” and inevitably you’re going to find gaps in other areas. You’re going to find other things that are as important, if not more important. That’s just a natural byproduct of this. I always stress that this is not a sole exercise of any one security program with any five security guys. You need to have buy-in and support from a lot of different aspects within your businesses. I think that’s something that gets lost a lot of times when people will kind of spin their wheels wondering, “Hey, why aren’t we getting anywhere? Why isn’t this being fruitful?” Those channels have to be made and kept open so that’s also a very important aspect of how you can pinpoint where you want to focus your efforts in hunting.
Q: What are the things that you would need to do if you’re a junior level analyst that is trying to work your way up? What can you focus on that’s really going to be the most useful to you?
DA: This answer can be kind of two-fold. I think there are definitely skills that can be acquired and improved upon but the first one I really want to stress is that the most important skill that any hunter can have isn’t necessarily a skill, it’s kind of innate, is a curiosity. A curiosity about the data, what do you have? What is it telling you? What looks weird? What should be normal? Your curiosity should start at let’s have a baseline. What is our network actually telling us? What data do we have going across? Let’s have a curiosity in the normal first, right?
To get a good baseline over everything and then expand that out. “Okay, what looks abnormal in here? Oh, that looks neat.” I’d say 90% of what we do is just kind of not looking for the nation states and not looking for the specific hacker groups and all this sexy stuff that we always hear about but it’s the pursuit of the, “Huh, that’s weird.” That’s really what it comes down to.
Say you’re a limited level one junior guy but you’re super curious and you’re always going to want to dig into things and find out how these things work. You’re going to hit a well at a certain point as a blue teamer, as defense, as a hunter if you don’t embrace and really understand the methods of your attackers. If you don’t understand their techniques how could you ever really be as effective a hunter as you could be? That’s really important.
There are people that put premiums on all kinds of skills like, “Oh, you have to be a programmer,” and, “Oh, you have to do Python and all this.” It’s not necessarily that but it’s cool to be able to look at code and be able to understand what it’s telling you. You don’t have to be able to light a program but that’s also helpful. A basic understanding of malware and how it gets downloaded, droppers and all this kind of thing, understanding those methodologies are very important to being an effective hunter.
Q: How can a hunter make the most out of adversary techniques that they discover?
DA: Probably the most important byproduct of discovering these techniques that these adversaries are using is being able to put that somewhere. Documentation in what we do is huge. There’s no point in going through a hunt, asking a question, forming a use case and digging through all the data, going down rabbit holes, not finding anything and not doing it all over again if you are not feeding this back into your program.
The other really important thing is that there’s been a big push in the past year or so of this “purple team” mentality. The more you understand what your red team is doing, then the better off you’re going to be. Being able to take what you’ve learned, work with your red team, have them explain certain topics to you that maybe you understand and if they just got done doing a campaign, having a red team, blue team debrief is so incredibly useful and it’s not something that we see a whole lot. There’s usually this kind of “good guy, bad guy” situation where it’s like “Hey, we’ve got this but are not going to tell you,” and then, “Hey, we found you but we’re not going to tell you that either.” No. That communication has to happen and it will bear fruit, I promise you. That’s really how you can make the most out of studying the techniques, knowing your enemy is being able to feed that information, back feed a program and working through the people that you have working for you to try to get in. Yeah, that’s probably the most important thing that you’re going to do.
Q: What’s something that you often look for — maybe your go-to piece of information or piece of data or intel that you’re going to be using to do what you were just talking about?
DA: I’m a log monkey. That’s what I’ve been doing forever because … that’s how we all started. Even in the very early days in this whole “hunting” thing, it was really just combing through logs, looking at logs. Again, that goes back to knowing what’s normal so you can recognize the abnormal, and that was before we had fancy things like Sqrrl and Tap and all these kinds of things, being able to just do something as simple as go through logs and just grab all day long and being able to try to make a timeline. Say you know that there was an incident or a red team exercise and being able to pivot through different log sources so that you can form that picture and then create that timeline. Timeline is hugely important so that you can try to get a fuller picture of what happened.
My go-to has always been we have to start at the network level because a lot of times, especially when you’re in a consulting role and you’re doing these very limited engagements, you might not have access to the host or the host visibility’s going to be very low. Maybe you’re not getting or they don’t have an endpoint solution or they’re AV solution is kind of wonky. I think that happens a lot. Mine’s always been firewall, proxy, BPN, DNS, DHCP, that kind of thing.
Q: How do you determine the fidelity of different levels of indicators?
DA: That can be really subjective as to what’s worth it to look at and what means.
Sometimes just having IPs and domains and hashes is nice and those have their place. But at a certain point the higher fidelity stuff is, being able to chain of events together based on behaviors and based on when A happened then B and then C, that kind of thing.
Chaining this these things together, knowing those behaviors is probably the most high fidelity “indicator” that you might have that something wonky is going on like, “Hey, here’s the service account that is logging onto this different machine and it shouldn’t ever be doing this,” or, “Here’s an advent account that’s logging on at these weird times a day,” and that looks weird to you.
The most high fidelity things you can have are based on behaviors and that goes back to knowing your environment. It’s that kind of a behavioral thing of what an attacker might do or how they might come at you. It’s not just one little thing anymore. It’s not one IP anymore. It’s a bunch of different things happening in a certain sequence but in relation to each other that can really cue you off to, “hey, something really wonky is going on right now.”
Threat Hunting Academy’s ongoing Hunter Spotlight series features conversations with top-level threat hunters to discuss a range of topics, from spotting adversary tactics, techniques, and procedures to leading hunt teams. Each interview is loosely based around their “Threat Hunter Profile” which can be found on the Sqrrl blog. The original interview is available here.
Photo:BetaNews
