This post was originally published here by Sqrrl Team.
What does the future of threat hunting look like? We brought together some of the world’s premier threat hunters to find out.
Sqrrl partnered with Richard Bejtlich from TaoSecurity to bring together a panel discussion comprised of the original General Electric CIRT incident handler team. These responders helped to pioneer some of the most common tools and techniques used in hunting today. In this roundtable conversation, they covered the some of the techniques they prefer to use when hunting, how to build a hunt team, and how the practice is likely to evolve in the future.
You can listen to the full panel discussion here.
How does setting up a hunt team affect an organization?
Richard Bejtlich (RB): All right. I think we did a good job taking a look at some of the past of this area. I’d like to talk about what’s happening now, the present. I’d like to use the idea of no two organizations are the same. Everyone on the call is working in another company or has worked in other companies along the way since our time at General Electric. Bamm, can you talk a little bit about that dynamic and how that affects standing up a threat hunt mission at a new organization?
Bamm Visscher (BV): Yeah. As I alluded to earlier, I always think of visibility driving any detection, whether it’s real time batch or hunting so that right there starts the differences. I think it was our time at GE was the first time that I really saw visibility and disparate visibilities was really a focus. We called it enterprise visibility initiative where we were specifically going after a network or specifically going after a host or specifically going after log data.
In any organization, being able to get any of those three types is going to have their own challenges. Honestly we deployed a ton of NSM sensors. We went through and got logging which was insane amount that we were doing. I’ve never seen as much host and application logging in one organization. At the same time, it was the first time that I can think of that you could really just pivot between those three times of data at speed. Even though if you called it hunting, batch or real time the ability to do investigative work and then, later, scoping during an incident was to me unparalleled at the time. That, to me, is the biggest driver.
RB: What does that mean for getting started? Do you just come up with some hypothesis that says, “This worked somewhere else and I just go ahead and try it,” or do you have to do a whole bunch of research? How would you get started in a new environment?
BV: I used things that have worked in the past in a new environment and then alter things as they change. As I learn about the network infrastructures, I learn about what operating systems are involved and what internet presence they have to change where I may need to go next. Then Ken makes fun of me because I’m always talking about small bytes and I really do try to focus on doing something small, stop, stop analyze what we did and figure out what the next byte’s going to be, what makes the most sense.
I think if you look from the perspective of I need to get visibility first. I’m going to get a chunk of network data. I’m going to get a chunk of my log data and I’m going to get a chunk of my host data, and focus on not only just getting that data but getting it complete. If you try to say, “I’m going to do my entire network,” that means both my ingress, egress, east, west, communications and all that all at once and then you’re going to have a project that’s going to last years.
Get your egress done and then at the same time maybe you got some resources that you can get specific application logging done. Maybe it’s your crown jewel so to speak. Do that first. Get your visibility. Start getting the layout of the land. Hopefully you start seeing a couple of incidents and you can kind of understand why threats are targeting you and you can start adjusting your hunting techniques based on that.
What goes into mentoring new hunters?
RB: I think those are really great points. I’ve never gone to any organization that said, “Yes, we are going to thoroughly instrument ourselves with a multi-year, multimillion-dollar project. Go ahead and do that.” It’s always quite a challenge to get a toehold and hopefully with that toehold you show some results which builds some confidence in your abilities and your processes. That then allows you to expand the visibility to another part of the enterprise and you get a little bit more success there and you just keep building success upon success. Generally the big bang approach won’t make it through the budgeting process, let alone all the other hurdles you might encounter.
Speaking of getting started in today’s world, David, you do a lot of public speaking and blogging and such. As a result of that you’re influencing a lot of the new generation of analysts. Can you talk a little bit about what’s it like to be an analyst these days who wants to get into that mission or advice you might have for them or mentoring and the personnel aspects of threat hunting?
David Bianco (DB): Absolutely. I would not presume to speak for who is trying to get into the field because I think the security field has always been pretty diverse in terms of backgrounds, right? There have always been straight up computer science people, IT background people. There have even been a lot of nontraditional paths. I worked once with a person who was in the security field who was trained as an army musician, right? There’s a lot of variability there and I wouldn’t even begin to know how to categorize who wants to become a security analyst these days. I can say maybe a little bit about how to get started in that field, especially if you’re trying to become less of a traditional kind of sock puppet monkey where you’re pressing the keys when the alert comes on and more of a engaged incident responder, incident investigator and threat hunter. I think those three things, responder, investigator and threat hunter, have actually a lot of overlap and they have a lot of skills and tool sets in common and requirements and what they need in order to do their job.
I’ve found worked for me and has worked for some others is just kind of take it slow and easy. I had the luxury, before I got into the security field full time, to be a pretty experienced Unix system administrator. I remember very well doing really interesting things just to make the infrastructure of the network work correctly. That led me to really have a good understanding of what all the pieces are in an IT environment and how they’re supposed to work together and how they shouldn’t work together sometimes and, therefore, even today when I see something that is a little bit off I can kind of get a quick gut check of whether I think that this is most likely to be a misconfiguration in the IT environment or an error that’s benign rather than possibly malicious activity. I won’t say that I discount the malicious activity hypothesis in all those cases but it gives me a little bit of a better idea of what I might be looking for.
A lot of people getting into this are coming in from the big data side and the data analytics and data science side, too. I think a really successful threat hunting program is going to have that mixture of the IT environment expertise, the security environment expertise and the data analytics expertise. They don’t all have to be in the same person but it’s actually reasonable for one person to have at least a little bit of experience in all three of those and maybe ideally a lot of experience in one of those. I think if you’re trying to get started in this, have a look and see what your skillset already is. If you are coming from you just came out of a university program that focused a lot on the infosec and cybersecurity pieces, maybe decide whether you are more interested in filling some of your background with some extra IT experience or maybe some data analytics experience and do some online courses or do some practical projects on your own time.
That’s pretty much exactly how I started getting involved in the data analytics and data science pieces, not being a data science person coming just from the security and the IT part. It can be kind of a challenge but if you give it time that’s perfectly doable. I would say try to figure out where you stand on the those three areas and fill in some of the gaps. Be creative a little bit in how you get that experience and that knowledge and just volunteer.
RB: That’s great. Great advice. David mentioned something I’d like to emphasize for all the hiring managers out there. There are no security people who are experts with all forms of network data, all forms of host data, all forms of log data who can reverse engineer, and who also can do forensics. You’re lucky if you find someone who’s really strong in one with a backup and a second and has some familiarity with the rest. That was one of the ways I was able to justify hiring these six gentlemen to be my first six incident handlers was convincing my boss: that you needed a mix of capabilities in order to accomplish the mission. Now you should always be trying to develop your skills as an individual and such but there aren’t unicorns who can do all of these things equally well because the field is just so broad these days. You can’t go deep in all those areas.
What will Threat Hunting look like in the future?
Richard Bejtlich (RB): I’d like to turn a little bit to the future and I’ve got two challenging questions I think. I’ll have one for Ken and then the last one for Tyler. Ken, a lot of what we’ve done in the past has focused on looking within the enterprise but there are more and more companies these days who, if they’re a startup for example, they might not have an enterprise. Their enterprise consists of however they get their WiFi access and everything else is in the cloud. I’m wondering if you could talk a little bit about any ways that you’ve been dealing with those sorts of challenges and is it possible to have a threat hunting mission when all of your data is essentially some place else?
Ken Bradley (KB): Man, why did you ask me that? Wow, that’s a tough one. I will say that right now I consider myself very fortunate that I’m not having to answer those questions. We’re definitely moving to those areas in certain small pieces of our business but I haven’t been asked to solve any yet. I think it probably starts with really examining the relationship, specifically the contract, as far as service agreements that you have with your cloud provider. I think that with the cyber security term as it grows now is pretty common these days that anybody you contract with, regardless of whether you’re talking about a cloud provider. I think that’s probably the first area that I would look at is what type of cyber security, what type of security mechanism does this provider have in place and agreements that you can get, services that they offer to help deal with those challenges.
I talked a little bit about collecting information off of your hosts and your enterprise. Those are all in your space, that’s easier to do than if you have all of your hosts or all of your main servers virtual inside its own cloud space. If that was the way we were transitioning I would examine what type of logging capability they have, or administer those virtuals in that cloud space. Maybe it’s a series of jump boxes or some way to get into that space and be able to put your own scripts or your own services on there to produce information that we’ve been talking about again in the host space. I just want maybe logs or I’d like to be able to collect a list of all the AutoRun data off of my servers that I have running.
RB:That does remind me of the fact that there are differences among cloud providers. There are some cloud providers who provide you very granular access and they’re constantly innovating in this way that you can get access to this data and do something with it. Then there are other cloud providers where the first time you learn that there’s been a problem is when they tell you your data has been deleted and they’re trying to do a backup.
It reminds me a little bit about banks and credit card companies. I have a couple of relationships where I can move money from one account to another or make a big purchase and I hear zip from the bank or from the credit card company. Then there’s another company that comes to mind, and I’m not here to advertise so I won’t say who it is, but I can be exceptionally granular down to any purchase over a dollar at a gas station, overseas, whatever. I get a lot of everything I buy basically and I’ve caught so many problems that way. Then add on top of that, their security team is good and they don’t let the person with a foreign accent who’s trying to empty my account do that. It’s just a different way of thinking but I think that’s sort of the way I have to approach it.
Again, in the theme of challenging questions, I’m going to ask this one to Tyler. Tyler, I’m constantly amazed by people who want to get into the reverse engineering or malware analysis fields because it seems like it was hard enough when you started a couple decades ago but it seems like things just keep getting tougher and tougher and even just trying to get someone to the point of the skill level that you’re at, which seemed to me to take forever, what do you see? Are things really getting tougher for the reverse engineer? Are there tools that you’re using now that you didn’t have before that make life a little easier? What do you see going forward in that area, as it relates to our threat hunt mission?
Tyler Hudak (TH): Yeah. I’ll definitely agree with you on that in that it is much harder, I think, nowadays to get into reverse engineering. I’m not even going to call it reverse engineering because I think malware analysis kind of better describes the scope of everything. When I first started we were really just focused on Windows executables and that’s all we had to worry about. Very quickly it moved into having to be able to analyze documents, PDFs, Microsoft Office documents, even weird Korean format documents that nobody had ever heard of, we had to start learning to pick apart.
It’s just gotten more and more complex in that now, in addition to all the stuff that you have to learn about before, the Windows executables and the documents, now you have to do script analysis or just the Powershell analysis and Java script analysis and script analysis. There are so many other operating systems out there, especially with proliferation of IOT devices. Now you have to understand how to analyze that format or a Mac binary or a Linux binary or some weird format binary so there’s just so much out there and hopefully I’m not scaring anybody away. I think Bamm said it best earlier is that you just take one small chunk at a time and learn it and then move on.
Fortunately, when I first started I was really kind of just learning on my own how to do this. I taught a lot of it to myself and just doing a lot of the various malware analysis contests that were fortunately around that I could kind of learn how to do stuff and go from there. Nowadays there are so many resources out there for pretty much any type of thing you’re coming up against that you can go and you find information on and learn how to do things. If you get stuck looking at something like a Powershell document or a Powershell script, there are probably tons of YouTube videos that will show you how to do that. If you need to look at something like a PDF or some brand new weird attack from an Office document, there are presentations online from conferences where people will go through and tell you how to pick those apart.
Really it’s just take it one step at a time. Learn what you need to and then move on because inevitably you’re going to come across something that you’ve never had to analyze before. I’ve been doing malware analysis for a while now and I’m always coming across things that I have no clue how to analyze. It’s just you go back to the basics. You start with the basic techniques and kind of go from there.
