Threat Hunting, TTPs, Indicators, and MITRE ATT&CK – Bingo

November 25, 2018

This blog is a continuation of a fantastic discussion with Richard Bejtlich. He responded to a question online, I blogged about it here and then he responded here and in response to another question I posed here. In this blog I’ll reply to his replies. The main point of this blog to me though is not to debate but to document my own experiences and look briefly at the evolution of terms and schools of thought as it applies to our cybersecurity field (the term cybersecurity vs. infosec is actually a great example of a term that has evolved yet to many still have two very different meanings and to some are interchangeable).

TL;DR Version: Richard and I seem to disagree on what an indicator is (reasonably so as his definition of one is perfectly supportable I just have a different view based on different experiences) which I didn’t previously realize was impacting how both of us viewed threat hunting. In doing so it seems that we’ve both come up under different “schools of thought” and therefore “hunting” can be leveraged in many different ways but you should strive to document and put structure to it if possible to really get the most value out of it.

Historical Context Matters But We Shouldn’t be Bound To It

In Richard’s blog he masterfully brings in some historical context of hunting’s origins (which few people really bring in history in a way Richard does but more should), or at least what was first documented (much which was done by him), in reference to the U.S. Air Force and National Security Agency. He also masterfully gives credit (another thing we should all strive more to do) to folks like Tony Sager and where his journey took him from the Air Force CERT to GE-CIRT where his use of the term was solidified even more with the wonderful analysts he cites.

What hunting was in the U.S. Air Force in the mid 2000’s and what it was at GE-CIRT shortly after doesn’t have to define the term for the rest of the field. It’s an amazing contribution, but not definitive. I think that’s, on face value, a frustrating topic. If I was at the GE-CIRT as an example and I really fleshed out a term my natural reaction would be to push against people using it differently (Richard is not doing that, I’m making a different point) but in reality terms, use-cases, and our field’s knowledge in general is constantly growing. There’s actually a logical fallacy called Appeal to Tradition which essentially states that something is what it is because it’s always been that way. But what’s interesting here is “hunting” was something numerous groups did. None documented or arguably had as much impact as the folks like Tony Sager, Richard, NSA VAO, AF-CERT, and the GE-CIRT but the experience is not less valid.

As an example, in my own experience in the National Security Agency one of my jobs was the Discovery Lead for one of the NSA Threat Operations Centers (NTOC) (would have been close proximity to Tony’s group but operating independently). Prior to this I had no knowledge of the term “hunting” and was not exposed to the schools of thought that were being developed at GE-CIRT or, interestingly enough, the use of the term and practice that Tony Sager and VAO were pursuing. I was part of a small team that established a new NTOC office at a field site and we were tasked with finding the “unknown unknowns”. This was admittedly very vague guidance but it was explained to us that the role of our NTOC office was explicitly finding the cyber threats that weren’t already documented and being tracked. Find what’s evading our insight to reveal collection gaps. We called this “Discovery” which, on a day-to-day basis, became known as “hunting.” The line between hunting and cyber threat intelligence though were very blurred for us because of our requirements; I would note that hunting was one way we went about satisfying our cyber threat intelligence requirements by identifying previously unknown intrusions (hunting) that we would then analyze (CTI). What we effectively were doing was taking an intelligence requirement, collecting against it through hunting, analyzing the intrusions we observed, producing insights, and distributing it to others as blogs, defensive mitigations and new detections, and reports. We used models such as the newly developed Diamond Model (previous to the paper’s publication) under the tutelage of Sergio Caltagirone and if I remember correctly we were the first team or one of the first to do so outside of his where he, Andrew Pendergast, and Chris Betz created the model. The interesting thing to the discussion here is that “hunting” was something that developed for my team without, to my knowledge, external prompt although I imagine there was cross-pollination from Sergio, Andrew, and Chris with the various teams they interacted with. For us, Discovery and our use of the term “hunting” was always about “finding new threats” but the main value in doing that was in identifying new defensive recommendations and gaps in collection even though we also found plenty of new threats to track. (For anyone curious I ended up choosing industrial control systems as the focus for my Discovery team since the collection would be so different and thus maybe the threats would be to; it turns out they were. This was a defining moment for me in ICS cyber security and a lot of what I had to develop in knowledge there at the NTOC helped in my journey and greatly inspired my work today at Dragos). Interestingly though one of the ways we found new threats was in the application of adversary tactics, techniques, and procedures as analytics/patterns instead of specific indicators. This aspect seems to distance Richard and I further which I’ll cover in the next section. But to close out the topic on the value you get out of hunting…

Richard acknowledges that you can identify visibility and resistance gaps as part of hunting, but it’s not the reason to go hunting.


In response to that I would say it may not be the reason that he and others hunt but it was always the reason my team hunted early in my career and it shaped how I view hunting now. His school of thought is simply different. I would also wonder aloud if a term needs to change when the actions are the exact same but the value propositions you’re aiming for are numbered differently. If you do exactly the same things and use exactly the same approach but you hope to find threats instead of collection gaps should you rename your effort? I’m not sure on that yet but my initial thoughts would say no. The “school of thought” for hunting and how to achieve it was very specific for Richard and me; I assume there are others in other organizations and parts of the world who have similar experiences that make their schools of thought much different. Richard and I are obviously both heavily influenced by a U.S. Department of Defense flavoring. I’d be interested if anyone else had their own journeys to document around the same time period.

We Were Bound To Disagree

It is clear that Richard and I disagree on the term hunting and how it’s used but the important part is why we disagree. I actually have no issues whatsoever with how Richard is defining the term and its background for his uses; his experiences are unique and many including myself have benefited from them. I don’t think there is a right or wrong to this discussion, just a friendly exploration to flesh out the topic for ourselves (or at least mine) and others. If we go back to the original points though it was clear Richard and I were bound to disagree and I didn’t see it initially.

I referenced but then moved past a point early in my other blog that Richard referred to all threat detection falling into two categories of either “matching” or “hunting”. I thought it was an interesting discussion which I slightly disagreed with but hurried past to get to the core debate. I noted that everything is “Matching” or “Unknowns” but not that hunting is contained in one or the other; it can be across both. If you are matching indicators you are not hunting in my opinion which I think Richard would agree with. If you are matching adversary behaviors though you might be depending on how you’re doing it (if the behavior is already a defined detection then no, but if it’s a pattern you’re searching out to find new activity that isn’t defined, then yes). When in fact, that was my error and in reality the disagreement is rooted there. A subtle but important point because I didn’t realize (although to his credit he’s definitely written on it before) that Richard considers adversary TTPs to be a sub-class of indicators of compromise (IOCs). I asked him that question after his first blog and he was kind enough to answer it here.

Again, Richard is great to identify and credit influential folks such as David Bianco and Chris Sanders, both who I consider friends and have written things that have definitely helped me further my thoughts too. He references David’s Pyramid of Pain (an extremely useful model by the way) where it very clearly calls out TTPs as a type of Indicator. I’m going to disagree again, surprise surprise, but this is where everything “came together” for me in that the disagreement is in the evolution of terms and schools of thought and nothing more. David’s use of the term indicator is mostly in keeping with another foundational work by Eric Hutchins, Mike Cloppert, and Rohan Amin titled Intelligence-Driven Computer Network Defense Informed by Analysis of Adversary Campaigns and Intrusion Kill ChainsYes the kill chain paper. Here they define out indicators to be atomic, computer, or behavioral. Behavioral indicators here are defined as those indicators that are made up of atomic and computer indicators to describe adversary activity, the example given in the paper is “the intruder would initially used a backdoor which generated network traffic matching [regular expression] at the rate of [some frequency] to [some IP address] and then replace it with one matching the MD5 hash [value] once access was established.” Where David’s use of the term seems to be in disagreement is “behavioral” which on its face would speak to TTPs but in reality TTPs can be described and leveraged now without any atomic or computed indicators to accompany them. My second point in the next paragraph will dive deeper into that.

Why is this such an important point to this discussion? For two main reasons.

  • First, terms and schools of thought evolve. “Indicator” today is almost exclusively associated with indicator feeds and the atomic and computed form of indicators. Some still use behavioral indicators and talk about them quite a bit to great value. But for the majority of the industry if you say “indicator” they’ve come to learn about them, use them, and see value propositions defined by atomic and computed values. Security professionals using indicator that way or not wrong. We even see another “school of thought” coming up which is based around MITRE’s ATT&CK; quite a few of their use-cases would fit nicely into mine by using ATT&CK as one framework for guiding threat hunts. In one of their papers they specifically note the focus on tactics and techniques (TT in the TTP) for them is important to go beyond “traditional indicators of compromise.” Here it would seem they are not using TTPs as a form of IOC either.
  • Second, what you can to do today for detection was not necessarily possible for most teams as little as a decade ago. The cybersecurity community has evolved quickly made many more advancements than we often credit ourselves. One such advancement is in the form of analytics. Analytics are effectively questions you ask of the data. Analytics have been around a long time in many forms but with the advancement of the community and of computing power we can now use analytics in more large scale ways and in a distributed fashion at scale (run the analytics at the edge where the data exists instead of pulling all the data back and then running analytics across it). One type of analytic, that I wrote about and referenced in the last blog when I mentioned the four types of detection paper, are threat analytics. Threat analytics effectively are adversary behaviors, i.e. TTPs or tradecraft (different things by the way). But they are not behavioral indicators in the way Hutchins, Cloppert, and Amin identified them. They don’t include any atomic or computed indicators; post detection there will be indicators but you don’t define the indicators ahead of the detection. The entire analytic might say “alert any time a file is dropped to a system, opens up a network port, generates network traffic to an external IP address, and then downloads an additional file once communications are established”. This analytic would get to the example given in the kill chain paper but without knowing the hash or IP address of the backdoor or anything about the adversary leveraging the behavior. This is done through an analytics engine now which have been around for awhile but are more common and accessible than ever before. When the analytic is defined it is “matching” to Richard’s point. But when I’m leveraging the TTP or tradecraft outside of the analytic to go search for and find new threats (numerous threats can leverage identical TTPs, so you’re searching for “unknown” threats using “known” tradecraft) I’m not matching any indicators and instead am using an intelligence-driven hypothesis to identify new threats. That’s EXACTLY what we did at the NTOC site I was at and we called that hunting.

As an aside, I think my second point is even how Richard is doing  his hunting to some degree because in his blog he gives an example that you could tell an analyst to go look go HTTP user agents as they are being leveraged by the adversary for malicious network communications but not tell them what user agents to look for, that at a high level is a tactic of the adversary which would classify as a component of a TTP and not be an indicator. It is not some anomaly that is occurring to filter through but a hypothesis generated from some insight the defender had such as seeing adversaries do it before (intelligence-driven) or based on their own expertise in how the protocol should work (domain expertise). I don’t want to argue points for Richard though so maybe I’m not interpreting it correctly but I think if we spent more time (preferably over beer) on this we’d probably root out more commonalities in approaches.

All of that long winded way of saying: Richard and I fundamentally seem to disagree on what an indicator is which is having a flow down effect I didn’t previously realize to how we view and define threat hunting. We would still end up debating back and forth on the ordering of the value propositions but both agree that the value propositions all exist (find threats, identify gaps, develop new detections, etc.) which is another reason everyone should be hunting regardless of how you order the value you get out of it.

This has been fantastic. I have thoroughly enjoyed this back and forth, and thank you again Richard for being such a gracious person to explore it all with me while challenging me to document my experiences in this blog.

Hunting vs. Incident Response vs. Just Doing Your Job

November 22, 2018

One of the things that motivates me to write is either being really pissed off or finding someone I really respect and having a different opinion. This blog is about the latter. Richard Bejtlich tweeted (oh the horror) that to him, hunting is just another form of detection. Now his real argument is of course more nuanced and well thought out but for the purpose of my blog and putting forth my view I’ll try to make his as simplistic as possible.


Now in reality, debating with Richard about topics related to detection, network security monitoring, and consequently threat hunting is probably similar to debating Plato about societies in the Republic. But I’ve always found myself learning more from debating smart people about small disagreements than debating the ones I outright disagree with. So enough compliments, let’s get into it.

The Argument

The heart of the argument, as I’m positioning it, is about what hunting is and is not and thus how folks use it. The initial prompt was from Twitter user @hellor00t about where “hunt teams” should be placed in the CIRT and whether or not hunt is just a new term for IR analyst. The comment was “seems like title semantics to me.” Which is totally fair. Richard, as previously stated, noted that hunt is just a form of detection. That incident response teams detect intruders using two major modes: matching and hunting. There’s a lot to unravel here but I’ll bypass the modes of detection argument slightly to say I actually agree but wouldn’t call one mode hunting. You’re either matching knowns (behaviors of adversaries, indicators, behaviors of users, etc.) or exploring unknowns (anomalies). You should cover your knowns first and then explore the unknowns but your strategy for detection should be more well thought out. I covered that topic with Sergio Caltagirone in a paper on the four types of threat detection which actually lines up nicely with Richard’s point (except the term hunting) here.

My Position

Honestly you can do whatever you want that gets you excited about security and proves to be an efficient and effective way to do your job for the organization. But I do think hunting goes beyond Richard’s point. I’ve written about threat hunting (not nearly as much as Richard) previously a few times before but here are two papers that help phrase my “school of thought” a bit that I wrote with David Bianco on generating hypotheses and another about what hunting is and isn’t with the “other” Rob Lee at SANS. My main thought process is that threat hunting is a proactive and iterative approach to detecting threats. It seems to align directly with what Richard wrote but I’d add that it really isn’t, to me, about detecting threats. I know that’s super confusing so I’ll go into depth a bit more.

Hunting is a hypothesis-led approach to testing your environment for threats. The purpose, to me, is not in finding threats but in determining what gaps you have in your ability to detect and respond to them. As an example, how do you know whether or not your organization would ever be able to detect and respond to the latest tradecraft observed by $ActivityGroup? You should generate a hypothesis about how that tradecraft would look in your environment played out against what you have to protect. You could determine the “route” the intrusion might take based on what you’ve observed previously and see if you have the people, process, and technology required at each point to successfully detect the tactics and procedures of the threat as well as the ability to respond to them. That’s only one way to generate hypotheses but hopefully serves as a tangible example. You’re looking to achieve a layered defense while figuring out your gaps. As an output of hunting, you should have gaps to address as well as detections you can now create as a result of your hunting (i.e. threat hunting cannot be fully automated but the output of your hunting should be more automation in your environment against the threats you hunted for). You could also create playbooks, or step-by-step guidance, on what an analyst should do to validate malicious activity and how to respond based on what you learned. Additionally, you could update your collection management framework (a topic I’ve talked about before and will have a paper out later this month on) to discuss what collection and data sources you do and don’t have in your own environment.

In short, hunting, to me, is a way to assess your security (people, process, and technology) against threats while extending your automation footprint to better be prepared in the future. Or simply stated, it’s incident response without the incident that’s done with a purpose and contributes something. Ad-hoc hunting to me isn’t hunting, it’s just proactively searching for threats in your environment (although admittedly saying you’re “hunting for threats” rolls off the tongue a lot better than “log correlating and searching”). Responding to alerts isn’t hunting either, it’s just triaging and investigating alerts. However, there are many “schools of thought” and you’re welcome to do whatever works for you in your environments.

Threat hunting is also an extremely useful way to measure the effectiveness and breadth of your threat intelligence efforts. Dan Gunter wrote an excellent Masters paper at SANS here on the topic.

Back to the original question of “where do you put your hunt team?” I really wouldn’t have a dedicated threat hunting team. I’d treat threat hunting as a multiple times a year or at least once a year assessment against the scenarios that are most concerning to you and your organization. Bringing together people from across the security organization and even outside that organization can be really useful to thinking outside the box about your hypotheses generation efforts. Having a dedicated team can make the process stale and cause too much group think. I’d rather see people “take turns” in that role if there’s a need to have a dedicated function.

In Conclusion

In conclusion (for now I assume) I will note I probably don’t disagree with Richard much but it was a useful prompt to have me think about threat hunting and put some thoughts in a blog to hopefully help others flesh out their views. My view is that hunting is not a form of detection, or isn’t just a form of detection. If you’re measuring the value of threat hunting on how many threats you find you’re likely not to be able to justify it at all compared to just doing proactive security work. Threat hunting, in my opinion, should be a much more structured test against hypotheses that is pushing your organization forward and ensuring you’re prepared against “styles” of threats. I do completely agree with Richard’s view though that Response is too narrowly defined as just incident response and after the fact to folks. I talk about “detection and response” largely because the terms can be useful in communication and when defining skill sets of folks. However, I do agree that detection and response both should be tightly coupled in people, process, and technology and that response is, in its best form, proactive.


What It’s Like to Raise Venture Capital (Just My Experience)

November 22, 2018

Last week it was announced that Dragos, Inc. raised $37M in our Series B financing event. That brings the total amount of money raised for the company to $48.2M since the company was founded in 2016 and marks my third time raising (Seed Round of $1.2M, A Round of $10M, and now the B Round). There’s a lot written explaining venture capital so I won’t attempt to explain it here in a short blog; but the purpose of this blog is to share some insights into things I’ve learned as a security practitioner that were new and surprising to me or just general observations that were interesting enough to document. Over the past two years I’ve taken more than 300 calls with more than 150 venture capitalists, officially “pitched” to more than 50, and had term sheets from about a dozen. I’ll start off with a comment about the team and company before diving into some of the observations I’ve had across those experiences.

“If you want to go fast, go alone; but if you want to go far, go together”

The proverb/quote above has some controversy around it regarding its origins but I like the quote nevertheless. It explains a lot about startups (to be fair so does the TV show Silicon Valley). When Jon Lavender, Justin Cavinee, and I started Dragos as co-founders we had previously built CyberLens together back in 2012/2013 time frame under the name “Dragos Security.” The tool was simply an assessment tool to process out packet captures and draw a map including some deep packet inspection of industrial control system (ICS) protocols. Getting it together as three people was really easy. It was a good year into the life of Dragos, Inc. when we were still adding things into the Dragos Platform that were already accomplished in CyberLens. The difference of course was the scale and stability of the Dragos Platform, our technology at Dragos, was significantly more vast than CyberLens. As we started adding a “feature” into the product it would break into numerous features and requirements. In theory what our technology does is simple: passively map out industrial environments, utilize threat analytics to identify the M.O./behavior/pattern of adversaries instead of just indicator or anomaly based detection, and offer up playbooks in an analyst workbench that act as step-by-step guidance to validate the alerts and scope the investigation. Not too bad right? Over the last two years we’ve seen the need to scale to 10 network appliances back to a server, to 40, to 300, and beyond. All on-prem without being able to take advantage of the cloud (in most cases) because of the sensitivity of industrial environments. At Dragos we’re about to be 100 people strong. You move so quickly you don’t really stop to think about the numbers and scale and all the things you’re doing. But one of the reasons startups tend to outpace larger companies is the ability to move those folks in a single direction and throw the entire organization behind it without as much communication challenges as large companies; flexibility is key. It’s amazing to see that occur and yet moving forward always seems like it takes place at a snail’s pace. You look back over 1 quarter and you realize that’s not true; but going far definitely requires the team and creating a scalable, deployable, stable, and usable technology often requires more than just a few people. I’m proud of the team we have today and the growth we’re achieving. Every now and then I sit back, pour a stiff drink, and just have to feel the exhaustion overcome me all at once. But then you sit forward and keep going again. You see the difference you’re making for your customers, the lives of the people on your team changing and growing, and you just feel so fortunate. Luckily, my eight month old son and wonderful wife provide more centering for me than exhaustion making the journey a bit more doable.

Venture Capital is Neither Good nor Bad

As a security practitioner I often found myself opining about venture capital. Bankers, private equity folks, finance majors, and “business people” were what I always assumed made up all the ranks of Venture Capital and to be honest I couldn’t stand the idea of them. I didn’t despise anyone, everyone takes their own path in life. But non-security practitioners trying to fund and decide “market winners” without an appreciation of the technology always bothered me. For anyone that’s seen my writings before it’s probably pretty apparent that I don’t think too highly of “management” positions leading security teams or technology teams without an understanding of security or technology. So it extended beyond venture capital. I also saw venture capitalists as just interested in making a buck (that is their job but the amount of “let’s make the world a better place” I’ve seen even in the hard times has been stunning coming from them) and largely figured it was better to bootstrap a business than take capital. While a lot of bad venture capitalists exist I learned that the bad apples were really giving a bad name to a much larger and more diverse field. Here’s some areas that I have evolved my own thinking:

Watch the Burn, Spend the Money, Watch the Burn

One of the reasons I hated the idea of venture capital is that I figured and had heard they would try to get you to spend so much money that you would have to raise more money later. They’d “get you over a barrel” and you’d be too deep in debt to do anything but go forward and raise more. That impression I had was entirely wrong, maybe I’ve just seen a different group of venture capitalists but there’s a whole science and art to throttling the business correctly. I’ve heard “watch the burn” from my board members (made up of our investors) to the point that it’s annoying not because it’s not accurate but because it’s been stated so much it couldn’t possibly have left my mind. The burn is how fast you’re burning through capital. Venture capitalists don’t want you to spend too much money, they want you to spend the right amount of money to invest in the business’ future. The logic is actually pretty simple when you think about it. In an enterprise software styled company where you are investing heavily in R&D you’re going to need to raise venture money to support it (sales can’t support the R&D costs especially before you have a product). But once you have the product (though it’s never done) you still have to invest heavily in sales and marketing. If I put $1 into sales and marketing today I likely won’t see any return on that for 9+ months. It takes time to generate the lead, make the contact, get the meeting, make the pitch, fit into this or next year’s budget, etc. The sales cycle can easily be 9-18 months long. But if I can invest $1 into sales and marketing today and get a return of $2 in 12 months that’s a 100% growth. Throttling the business to be “profitable” especially early on is robbing the future of the business. If I can lose $5M this year and make $10M in sales next year, that’s a great thing for the business. Thus venture capital is needed so that you can spend what you need to, but you don’t just want to spend money you want to understand what the return on investment is and monitor your burn as closely as possible to make sure you’re not over investing or under investing. This can cause a “foot on the gas foot off the gas foot on the gas” back and forth in the company on an almost weekly basis. There’s risk into taking this approach and it’s not the only one but previous-me would have stated “well just grow as you can support it and don’t try to lose money in hopes of earning more business next year.” Today I understand that the speed of the market, demand from the customers, growth of competitors, marketing from incumbents, and more all goes into deciding that spending large sums of money to grow the business is hyper critical especially in a software company where deal sizes and sales cycles are both large.

Lots of Different Types of Venture Capitalists 

There are a lot of different types of venture capitalists, and I’m talking about the people not the firms yet, I’ll approach those next. Some venture capitalists are the bankers, private equity folks, business grads, and financial analysts that I had previously thought about. But many are also “operators” that have built and led companies before. In reality there are good and bad in both forms. I’ve found that as a founder I really resonate with operators. Having a few of them on my board early on gave me good guidance and made a collaborative environment. They were also more empathetic to the needs of the business instead of spreadsheets and trying to math the company to death. As we grew it was also important to add those who had more experience as the “typical” type of finance people who helped provide a bit of devil’s advocacy to the board on various decisions and help make sure that the “emotion” of running the business and the “experience” of having done it before wasn’t completely out of touch with what the numbers could support. It’s obvious of course that it takes all types, but to me seeing the different value propositions of people and their experience and how they influenced the company was interesting. As a gaming nerd there’s so much that role playing games have taught me about business, more so than any of the free online classes I’ve taken on business (seriously you should watch those free videos like the free MIT and Harvard classes and you should throw in some good RPGs as homework). Class skills and development of different professions while understanding people’s skill tree maximums and strengths/weaknesses is much easier when you just imagine everyone you meet has a playing card or a stats page that you can see when you hover over them.

Lots of Different Types of Venture Capital Firms

There are many different types of VC firms. Some have experience in your field, some do not. VC Firms vs. Corporate VCs are a major difference in how they view the world as well. There is even the informal ranking system of “Tiers” that is very loosely made but can help you think about the VCs to determine which are right for you. From the very beginning the idea at Dragos was to IPO the company. Making it to the stock exchange with a public listing of your stock sounds like a good thing from a monetary perspective but it provides something more important to my team. We want to keep Dragos around and really make the world a more secure place for our most important and critical infrastructures. The industrial world is huge and deserves protection. Having an “exit” by getting bought by another company generally doesn’t allow that. Acquisitions of technology companies are not always bad but especially early in the company’s life when they are just really getting their culture and tech and value propositions right, getting adopted into another company can be a death sentence. It’s going to take years to put a real dent into industrial cybersecurity. When you raise venture capital though you have to be able to give your VC’s an exit. They need to be able to exit the company at some point and recoup their investment plus how much their investment has grown. As it turns out, the appetite for how this occurs differs from firm to firm. There is not simply 1 “persona” at any firm but their track record, size of fund, and “tier” can help inform what they’re looking for in how you exit.

As an example, if your VC firm has a small “fund” they’re investing out of ($100M-$200M) and they haven’t had a lot of IPOs before then it should be pretty obvious they are looking for an acquisition as an exit. If they invest $5M in your company early on and you can exit for $200-$300M they get enough return to put a real “dent” in their overall fund size to show returns to their investors (which surprisingly enough are not just rich people but more often college endowments, retirement funds, etc.). But if your investor has done IPOs before and is investing out of a large fund ($500M+ and often times $700M+) then the check sizes they write are going to be larger (they’re going to look to invest more like $15M instead of $5M so they can put their capital to work) and they’re going to need the exit to be larger (which is more likely to be an IPO than an acquisition) to get an exit that actually puts a dent in their overall fund. There’s way more to evaluating VC firms including their networks, who the board member is that will actually sit on your board, and how they enable their investments but figuring out what they need in terms of an exit is a really important part of the equation to make sure it aligns with the company you want to build. An IPO is an “exit” for many of the investors but it’s a growth stage for your company. The founding team and its members shouldn’t be thinking about an exit but should be aware that taking venture capital is more of an investment/loan than it is anything else.

Raise What You Need Not What You Can

If you’re outside of a company you rarely get deep insight into what’s going on in that company. One of the ways we measure companies though, at least when consider venture capital, is how much they raised and at what valuation. At one point I would have thought raising the maximum amount and then not raising again for a long time is a good strategy; turns out that’s very silly. The simple version is easy to understand, if I raise $10M at a $15M pre-money valuation you give up a lot of the company in terms of dilution. If you raise $5M at $15M and then raise another $5M in a year at a $30M valuation you’ve just raised money without giving up as much of your company. It’s never that simple though and you have to figure out how much you can raise and how much you can put to use to get you to the next milestone for your company. If the venture community is hot, then investments might come easy, if it’s not then no matter how good you do the money will “dry up” to some extent. Additionally, you always want to make sure you can hit the goals to get to the next milestone.

As an example, did you raise $30M at $100M pre-money valuation? Congrats that’s awesome. Did you raise $60M at $150M valuation? Looking a bit dicey by comparison. But why? Let’s assume those are C series numbers since they are really big amounts (although in the industrial community a lot of the Series B raises are more like Series C, our $37M raise at a B is more like a C round for comparable IT security companies). If you raise $30M on $100M pre-money valuation then your post money valuation is $130M. To raise a D round on this trajectory you really want to show investors that you can 2-3x their investment, this entices the new investors and shows good health for the company. I.e. at the D round you’d really want a $260-$390M pre-valuation. Ideally you could get there in a 2-3 year period showing rapid growth. But if you raise $60M at a $150M valuation for your D round now you have a $210M post money valuation. Which means you’d ideally want to hit $420M-630M pre-money valuation to show the same level of growth. But the larger the numbers the slower the growth might be and there is a limit in how much you can invest in your company each year to hit a return especially if you’re being compared to the growth of the company that took $30M. I.e. pouring $50M onto a company instead of $30M may not actually give you any extra value depending on market size, customer adoption, sales cycles, and more. So if you raise more than you need just because it’s available you are potentially taking more dilution in the company than you need, you are possibly not doing anything more for your growth rate, and you may make it incredibly difficult or even impossible to raise your next round which means you have to get acquired as you’ll be burning too much to go cash flow positive or raise more. There’s no perfect formula nor only one school of thought, but this was all interesting to me as I learned more about raising capital and watched our own experience. If you’re wondering, we raised the $37M specifically to get us to the next round which we have a target valuation and date for already (I found it useful to always be thinking about your next round and walking backwards into it for the current round).

Terms, Terms, Terms

It would also seem obvious that a company that raises $30M at $100M valuation is a better company than the one that raises $30M at $90M valuation (pre-money on both). However, what most people never get a chance to see are the terms of the investment. The term sheet is the “offer” that an investor makes a company for the terms of the investment that then get ironed out in a much larger package of closing documents. Those aren’t made public but in my opinion are far more important than the valuation of the round. A seemingly bigger and better investment may have horrible terms. As an example, one term that could included in the investment is “participating preferred” or even have a multiple on it. A multiple is idiocy in my opinion and would only make sense to a company in distress and participating preferred in general is a term I refused to ever accept. There are numerous terms you have to watch out for that can flavor the entire trajectory of the company but let’s dive into participating preferred which I think of as one of the worst terms.

Let’s imagine an investor puts in $10M into your company and you add equity for them which gives them 10% of the company. And then your company eventually exits for $200M. In a simple world where that was all the investment you had then the $200M would be split between all the preferred stock holders (the investor) and the common stock (the founders and employees). So $100M would be split according to how much equity everyone has. The investor’s $10M turns into $20M and that’s what they leave with. The rest of the $180 is split between the founders and employees depending on how much they own each. But if the investment was participating preferred at a 1x then the investor would get back their $10M and then get to participate in 10% of the remaining $190M. That means they’d walk away with $29M and the company would split $171M. In reality the investors would actually get more because of interest accrued on the initial $10M. This is very simplistic but even in the simplistic view you can see why this would be problematic. Even if your goal isn’t everyone splitting money and instead is just growing the company, you could potentially have lots of extra money flowing out of the company at an IPO. Simply put, no matter the situation you really don’t want money leaving your company that you could be investing in your company or splitting between the people who were putting in the day-to-day work. In my experience, my goal was a simple term sheet. An adviser of mine told me to demand that the terms of the deal be so simple that they could be written on a napkin. We aimed for that and succeeded in doing it but it is way harder than it sounds.

Boards Are Equally Important and Not Important

This is a weird statement to write because I get a lot of value out of my board as advisers and a sounding board. But previously I would have thought a board had a lot of control over a company. I don’t know why but I just always assumed a lot of decisions get made at the board level. Turns out that’s not the case. The person running the company day-to-day is the CEO. The CEO knows so much more about the company than the board members could possibly know. In relation to venture capital the board wants to know that the company is growing, monitor their investment, and provide input and connections that could benefit the company to help it grow. As an example, one of the ways I used my board is to help me with interviewing key hires especially VP’s because they’ve seen many people and have experience I don’t have on to act as a sounding board. They provide good connections throughout the various industries we work in and also provide some guidance to make sure that I’m growing the business in the right way at the right speed and am building a company that others will want to invest in which will help our team build the type of company that can really impact the industry. But they don’t make a lot of decisions. I assume a lot of people think they do because I even have people outside my company that want to do business with us or want something from us hit up my board members directly assuming they have some significant influence over me; very little makes me more likely to ignore someone.

It turns out my view of how involved boards were was entirely off. They provide help that I ask for, but they don’t make a lot of choices. Salary and compensation packages get approved at the board level (which I find to be useful actually), the annual budget gets approved there, and the rest is just providing them updates and getting advice. You could get bad terms that dictate a board has more say but I’d be concerned about that for a lot of reasons. Opening a new office, choices around the product, hiring and firing, strategy for the company, go-to-market, etc. that’s all on my team internal to the company. The board is interested and active but entirely disconnected from the “decisions” that get made around that. The approval of an annual budget has a high level of impact but it’s something my team and I put together and put in front of them, their input is “that looks normal” “that looks like it’ll get you what you want” “have you considered upping the investment in X?” “I think you’ve over invested in Y”. It’s useful, it’s their approval, but it’s still the work from inside the company and whatever I put in front of them is going to be within the range of what gets approved.

I imagine this is all super obvious to most other people but I was surprised to find how little boards actually do and yet how they remain vitally important. One of the things I’ve really found value in my board is simply having a well experienced team of folks as a sounding board. It really can’t be overstated. As the CEO whether or not its true you constantly feel you’re the person who’s suppose to have all the answers and lead the ship. Sure everyone else actually does all the work, and the decisions are super collaborative in the company, but that feeling remains. At the board level though it feels far more like a discussion with less stress. I’ve found my board to be my “safe space” to just think about stuff out loud without repercussion and have folks more experienced than myself in areas that are different but important provide thoughts or validation. At the end of the day it’s not lost on me that so many in the community, so many customers, and all the families of all my employees have a very vested interest in me not screwing up my part. The act of “yea Rob that makes total sense” or “I really think you should dig into that more” provides far more comfort than I can put into words.

Every Company is Different and Your Journey Will Be Filled with Friends and at Times Lonely

These are my thoughts. These are some of my observations. Yours are bound to be different but I hope it’s useful to see one person’s thoughts on all this. I’ve never felt more excited and more proud than leading this team at Dragos and I’ve also never felt so very alone in my career. That sounds weird but let me clarify. Being the CEO and founder of an industrial cybersecurity company that has received venture capital, now above $48M, and is forging into a market that’s largely been undefined…is a lonely and exciting endeavor. Industry analysts and financial folks have tons of opinions, but none of them have actually been down this journey. I’m empathetic to the various industry analysts trying to define the market or provide hot takes on where the market will or won’t go; and it’s useful to the community because buyers and community members cannot just listen to the obviously biased companies leading the way, but those same industry analysts and financial folks have far less experience in this topic. The number of peers I have for the very specific thing I’m doing is tiny. And to be honest it’s not like the founders of competitive companies really get together and have open discussions other than “hey thanks for helping also build the market, good luck to you, I hope I utterly destroy you on the battlefield, but I respect you” lol. I often look at the industrial cybersecurity community and industry practitioners that I respect pontificating on the market, who will be winners, what that means, what products will be needed, etc. and I struggle not to roll my eyes a bit as they have never built companies or products in that space. I have to force myself to realize that what they’re doing is still an important part of the equation and very useful to folks. But I do also understand that they are also extremely biased, operating with completely different experience, and sometimes are advising on things that they have far less experience in than people realize especially compared to the companies that live it day to day and are on that unique journey. I’ve had heated arguments with people I respect and call friends simply because their experience guides them to view things so differently than the experience I have. And yet, that’s ok. Actually it’s really useful for everyone. And taking a step back to just keep that in mind is vital to not becoming arrogant or being misguided yourself. To solve a problem as complex as any that guides you down such a unique path is going to require thought process and input from everyone, welcoming it is important especially when it seems counter to your own experiences.

Why say all this? To stress the point that in this community, this industry, this time, etc. things are so specific that any lessons learned I have may not really apply to you in your journey. And it’s so different that even the founders that have built a company before may not have experiences that help or relate to you; I found the “entrepreneur” networks to be largely disappointing with experiences that didn’t translate to the challenges my team is facing. I found the folks in my company and their insights as well as the experiences and insights of our customers to be far more useful to me. But maybe those networks work for you. I don’t think I’m special, but I think this journey is unique. Yours is too. Hopefully my thoughts are useful but don’t take them as the only thoughts nor any of my takes as the only way to view those topics (hell maybe I’m wrong on plenty and will have different views in a few years). Instead, I’d focus more on how my opinions have changed and be open minded to have yours change as well.

What It’s Like to Testify to the U.S. Senate

July 21, 2018

One of my SANS students challenged me recently that I haven’t posted on my blog in awhile; I realized I hadn’t and checked today and found that it’s been almost a year since my last post. So first of all, apologies on the delay. I’ve been busy (Dragos, Inc. has been expanding rapidly and we’re about to reach 60 employees, my wife and I welcomed our first born into the world, and industrial cyber attacks haven’t exactly slowed down) however, no excuses, I should post and share my experiences more and appreciate the folks who take time to read. This post comes as a request to some attendees at one of my SANS @Night talks where they asked me what the experience was like and how it happened. I feel that it’s really not transparent how someone gets to testify in front of the Senate and what all goes into it so I hope this post is useful to illuminate the experience a bit. The recorded testimony can be found here.

How I Got Invited

I imagine most folks’ path to testifying in front of the Senate is different. I also assume for most they are seen so widely outside their own community as an expert that it’s a natural thing. My experience was a bit different. One of the things that motivates me a lot is educating people especially on topics of intelligence and industrial security. The intersection of those fields with policy is pretty clear and unavoidable. I’ve also always been a bit outspoken about individuals speaking on and influencing technical discussions without having technical experience. This has led me to getting involved in educating policy folks on topics of industrial cybersecurity, especially critical infrastructure. An ex-Senate staffer I ran across asked if I’d be interested in offering some sessions for Senate and House staffers on the topic of electric grid cybersecurity. Over the course of about a year I would routinely go to DC and spend a few hours talking about how the power grid works, what cyber threats actually do, and demystifying a lot of the hype (yes our infrastructure operators have made fairly resilient infrastructure, no cyber threats aren’t magic or all powerful, and no Ted Koppel’s book is not accurate in the real risk). In addition, I got invited to speak on a panel on cyber threats to the grid at the Siebel Scholars’ conference with a panel made up of Richard Clarke, Kevin Mandia, and Liam O’Murchu (which was moderated by Ted Koppel…super nice guy, he just didn’t write a technically accurate book). While at the dinner another Senate staffer liked what I had to say and offered to keep in touch, I exchanged cards and that was that.

Eventually it was the staffer from the Siebel event that emailed me out of the blue one day asking if I’d be willing to testify in front of the Senate’s Committee on Energy and Natural Resources on cyber threats to U.S. infrastructure. Because I had spent time with other staffers in the Senate the recommendation went over well as a few of the other staffers knew me and agreed I’d be good to have on the panel. What I also really enjoyed was finding out that both Republican and Democrat staffers had recommended me; I joked with them that they all thought I was on their “side” and that they couldn’t figure out where my politics actually fell (to be honest I’m not political, just opinionated) but in truth they were all just on the same page of wanting to protect infrastructure and didn’t care about the politics around it.

Preparing for the Testimony

In preparation for the testimony I used YouTube to watch other testimony to the committee on the subject of cybersecurity and determine what type of questions the Senators had previously. I also pulled the written testimony of Thomas Rid and Kevin Mandia, two individuals I respect in the field that have also testified, and looked at their style of writing and what they chose to highlight. I then prepared my written testimony on the key points I wanted to get across. For my testimony the three key points were essentially:

  • that industrial cyber threat landscape is largely unknown so we cannot just adopt IT cybersecurity frameworks/regulations/best-practices into industrial networks because they were built off of risk observed against cyber threats targeting the Enterprise networks and how to handle them
  • that regulation has served a purpose (such as NERC CIP for the power grid) and helped the industry but that we have exhausted reasonable regulations and the industry is struggling to innovate or do real defense in the face of new regulations that come out every few years
  • that the Department of Energy’s Office of Cybersecurity, Energy Security, and Emergency Response (CESER) that was being formed at the time has a unique opportunity to help work with private sector and form partnerships not only with asset owners and operators but also the private sector security community; many of the government’s actions right now in reaction to fear of cyber threats are quickly pushing them into being competitive with private sector firms that are actually better equipped to deal with certain situations and we should all attempt to play to our strengths instead of competing

It was interesting to try to distill everything I ever wanted to say down to effectively seven pages (found here) and to do so in a non-technical audience of policy makers. Some of my points, such as highlighting that technologies such as artificial intelligence aren’t a silver bullet for security, were put in to directly counter some of the things the Senators had been told in previous testimony that I felt mislead them. I then sent out my testimony to people I knew and trusted around the community who helped me steer clear of any wording or language that might come off wrong. As an example, I was mentioning “NERC” “DOE” and “energy companies” so I sent my written testimony to people I trusted at NERC, DOE, and numerous energy companies to make sure I was representing them the best I could. I.e. don’t get in front of the Senate on TV broadcast across the country and talk out of turn about matters that impact others.

What I didn’t expect was that the list of people testifying goes public well before the hearing so people reach out to you and the Senators about you. I had various interests groups email me trying to get their points included in my testimony; almost exclusively it was groups I’d include in “the crazies” category. The two most surprising was a group that wanted me to bash NERC and tell the Senators how the grid was so fragile and needed new oversight and something about EMPs (please stop with the EMP stuff) and then a member of the industrial community emailed some Senators directly and, from what was explained to me later by a staffer, essentially petitioned them not to listen to whatever was going to be said about cyber threats and instead to realize that level 0 sensor cybersecurity is the only relevant topic. I consider this person a friend and long-time community member and do not think he was being malicious but to be honest I found that to pretty rude because it, intentionally or not, was a distraction and could have subverted the points of the people that were presenting on the topic. (I’ve avoided the level 0 debate in the community because it becomes almost fanatical but in essence my position is that all the risk we’re seeing due to cyber threats does not start at a sensor level and if you do not learn what the adversaries are doing, how they’re doing it, and counter them before it ever gets to that level you will lose out). I only include this discussion of unexpected components to testifying to note to everyone that Senators and staffers get all sorts of things sent to them all the time. You really need to be active in the process to counterbalance the viewpoints but also ensure you do so in a manner that is conducive to the overall narrative.

Delivering the Testimony

For the verbal testimony itself you cannot read from your seven page written testimony. The Senators have a very limited amount of time to hear testimony and ask questions and they want every second to answer as many questions as possible. They are extremely busy and this is genuinely the time they’ve dedicated to focus on this specific topic. So you get five minutes to summarize your points, they’ve read your written testimony in advance, and they get a few minutes to ask you questions back. Who they ask questions to is entirely dependent on the Senators and who they want to talk about and on what subjects.

I have to admit I was actually pretty nervous delivering my written testimony. This was a recorded event and not even live. I’ve been on live news before on channels such as CNN and Fox in front of millions of people and felt more confident. I speak at a lot of venues. I’ve keynoted numerous conferences. And none of it compared. This was nerve wracking and I had to steady my hand by holding the paper in front of me. I also made a deliberate choice to speak fast and get through as much material as possible in the five minutes, I knew the Senators had already read the testimony (or at least their staffers did) and that they had prepared questions, so my verbal testimony was intended more for those that watched the hearing after the fact. When it got time for the QA session I was back in my element and felt confident. But that verbal testimony was the most nervous I’ve been in awhile.

I found the most important part of the verbal testimony was to try to answer the Senator’s questions as quickly and coherently as possible. I also found that all of their questions were extremely reasonable. Even those that appeared to have an agenda actually only were getting clarification on points they had previously been told by others. And to be honest, more lobbyists and self-interested people try to speak to Senators than folks doing the mission. What that meant to me is a clear understanding that some of the Senator’s views in topics such as just disconnecting the grid from the internet were based off of other people and not some agenda of their own. To be honest I found that the Senators all cared deeply about the topic and were highly professional. The country definitely feels divided these days but in the hearing I was in, at least, everyone there seemed to sincerely care about infrastructure security with no partisan political games tied to it.

At the end of the testimony I also felt that I won some major kudos points at home. My son was being born that day and my wife and I both felt that I should still go and testify; she is an immigrant from Holland and I served my country in the military so we had this extra sense that this was something patriotic that had to be done; so she gave me the go-ahead (luckily I didn’t miss my son’s birth though). The Senate was apparently informed about my son’s imminent birth and just how awesome my wife was being by letting me go…so on official record, at around 2:06:17…Senator Murkowski and the committee congratulated my wife and thanked her, and I got on official record “she’s awesome” so in essence…I win.

Final Thoughts

It’s really difficult these days to talk about politics with how decisive it gets especially in the media. We have things that make us very passionate and strike at the core of who we are and what we believe in. I’m not political at all and even I’ve been extremely bothered by some of the things I’ve seen such as the consistent attack on the U.S. Intelligence Community. However, testifying to the U.S. Senate was the most lifting political experience I’ve ever had. I swelled with pride in a way not easy to describe and felt the joy in this grand experiment we call democracy in a way I found unexpected and amazing. I would encourage the technical practitioner community to engage your elected officials and their staffers. Seek to educate others. And if you get the chance to testify to take the opportunity and try your best to represent the community well in doing so; it was an amazing experience that I highly recommend.






Russian Election Meddling, GRIZZLYSTEPPE, and Bananas

August 17, 2017

It’s been awhile since I’ve been able to post to my blog (as it turns out doing a Series A raise for my company Dragos has been time consuming so I apologize for the absence in writing).  But it is fitting that my first blog post in awhile has something to do with the GRIZZLYSTEPPE report. I almost got sucked back into writing when I saw the Defense Intelligence Agency (DIA) tweet out the Norse cyber attack map.

Matt jumped on it pretty quickly though which was great.

I tried to attempt to fill the person in running the account just in case they didn’t understand why folks were less than excited about their presentation.

But in their responses to me it seemed they didn’t fully understand. They articulated that they use unclassified data for the conference but use classified data at work. Of course the problem wasn’t the data (even though it’s not just unclassified but completely bad/fake data) it’s the idea that a cyber attack map aka “pew pew map” is not a good way to communicate to any audience as its simply a marketing ploy. However, it’s not worth a full blog post so I’ll just instead request everyone to do their homework (should only be a quick Google search) on why pew pew maps are stupid and everyone serious in the discussion should stop using them.

On To the Main Discussion

But on to the main topic. What does Russian election meddling, the GRIZZLYSTEPPE report, and bananas all have in common? Absolutely nothing. Each are individually completely unrelated to each other and people should stop putting any of them together as it ultimately just makes people look silly (to be fair no one’s associated bananas with the election interference yet but it might be a better correlation than the GRIZZLYSTEPPE report).

This discussion was all spawned by an article that the New York Times released on August 16th, 2017 titled “In Ukraine, a Malware Expert Who Could Blow the Whistle on Russian Hacking“. Spoiler alert: he can’t. I went on a bit of a Twitter rant to explain why the article wasn’t good, it can be found here, but I felt it was a complex and an important enough topic to cover in a blog.

The NYT piece posits that a hacker known by his alias “Profexer” was responsible for writing the P.A.S. tool and is now a witness for the FBI after coming forward to Ukrainian police. The P.A.S. tool, the article puts forward, was leveraged by Russia’s intelligence services without his knowledge (not sure how he can be a “witness” then but I digress). The authors of the article previously explicitly stated P.A.S. was used in the break-in of the Democratic National Committee  (DNC) but they had to issue a correction to that (to their credit, folks from NYT reached out to me after I critiqued it on Twitter to try to get the story correct after it was published; I asked for the correction as I’m sure others did but in reading the updated article the correction doesn’t actually address the larger issues so I wanted to cover them here in the blog).


Figure 1: Correction Related to P.A.S. and the DNC

Where did they get this assertion that P.A.S. was used in the DNC breach? By tying the GRIZZLYSTEPPE report (which does note that P.A.S. has been used by Russian security service members before) to the DNC breach. The GRIZZLYSTEPPE report has nothing to do with the DNC breach though and was a collection of technical indicators the government compiled from multiple agencies all working different Russian related threat groups. The threat group that compromised the DNC was Russian but not all Russian groups broke into the DNC. The GRIZZLYSTEPPE report was also highly criticized for its lack of accuracy and lack of a clear message and purpose. I covered it here on my blog but that was also picked up by numerous journalists and covered elsewhere. In other words, there’s no excuse for not knowing how widely criticized the GRIZZLYSTEPPE report was before citing it as good evidence in a NYT piece. Interestingly, the journalists didn’t even link to the “Enhanced Analysis” version of the GRIZZLYSTEPPE report which was published afterwards (and is actually much better) as a response to the critiques of the first one.

A major issue exists though with the correction to the NYT article. It changes the entire point of the story. If Profexer isn’t actually a “witness” to the case because P.A.S. wasn’t used in DNC then what’s the message the journalists are trying to get across? Someone who wasn’t working with the Russians, developed a tool that the Russians didn’t use in the DNC case, and didn’t have any insight into any of the Russian threat groups or campaigns cannot be a good witness.

Even after the correction though the journalists draw the readers attention to the breach early and often to continue to reinforce that this gives new insight into that case.

Figure 2: Snippet from NYT Article Referencing DNC Breach and Profexer

And again the journalists explicitly state that Profexer is somehow a witness to what occurred and reference him back again to the election hacking.

Figure 3: Snippet from NYT Article Claiming Profexer is a Witness

The article goes on to note how this changes our thoughts on the Russian groups (APT28 / APT29 or COZYBEAR / FANCYBEAR) and how they operate; the journalists state that using publicly available tools or outsourcing tool development to cyber criminals is against the modus operandi (MO) of the Russian security services. I do not know where the journalists get this claim but they do not source it; I disagree with the claim but I’ll note the burden of proof here is on them with regards to showing where they’re claiming the previous MO and I’ll simply state that there have been numerous publications and reports showcasing Russian threat groups including the security services using other groups and people’s tools and exploits. This isn’t new information and it’s fairly common for many threat groups to operate in this way.

The attribution on APT28 and APT29 is some of the most solid attribution the community has ever done. Numerous cybersecurity firms have covered this group including FireEye, CrowdStrike, Kaspersky, TrendMicro, and F-Secure but we’ve also had government attribution before by the German intelligence services on a breach into their government that pre-dates the DNC breach. A cursory look will reveal that organizations have been tracking this Russian threat group for about a decade now. Yet none of the people who’ve actually covered these groups were cited in the NYT article. Instead the journalists chose to cite Jeffrey Carr and his quote is confusing to most readers because he is trying to detract from the attribution where he states: “there is not now and never has been a single piece of technical evidence produced that connects the malware used in the D.N.C. attack to the G.R.U., F.S.B. or any agency of the Russian government.” It’s almost as the journalists just wanted a contrarian view to look balanced but what an odd selection if not just set up their witness to be even more important.

I want to be very clear on my next critique: I actually don’t think Jeffrey Carr is a bad person. I know he ruffles the feathers of a lot of folks in the community (mine included at times) but on the two occasions I’ve met him in person he’s been an absolutely nice person to me and was civil and well articulated. That being said, he is not an expert on attribution, not an expert on these groups, nor has any reason to be cited in conjunction with them. He’s often widely criticized in the community when he tries to do attribution and it’s often painfully full of 101 intelligence analysis failures. The NYT didn’t do him any favors by including him in this article and seriously detracted from the idea that they understood enough about this topic to cover it. Simply stated: “cyber” is not an expertise, if you are covering a niche topic like attribution or a further niche topic like Russian group attribution you need to use folks who have experience in that subject matter.

Please Stop Arguing About Attribution Without Expertise In It

This is a bit of a big request but it’d be very useful if people stop taking a stance on why attribution is difficult or not and whether or not attribution is right or not if they have never had experience in doing attribution. This is important because the journalists in this article seem to want to help bolster the case against the Russian intelligence services yet make it more confusing. At one point they try to set up their witness as some new smoking gun to be added to the case as a push back to people like President Trump.

Figure 4: Snippet from NYT Article Setting Up the Importance of the “Witness”

Attribution is not about having a smoking gun. Attribution is a good example of doing true intelligence analysis; there are no certainties and you only can come to an assessment such as low, moderate, or high confidence. Almost every single piece of data put forward in that assessment can and should have counters to it. Very reasonable counters as well. It’s why when anyone arguing for attribution argues a single piece of evidence they almost always lose the argument or look silly. It’s simply very rarely about one piece of evidence and is instead the analysis over the total data set. The attribution levied towards Russia for meddling in the U.S. elections is solid. The reason President Trump and others don’t want to accept that has nothing to do with the fact that there hasn’t been a witness or a “single piece of technical evidence produced that connects the malware used in the D.N.C. attack to the G.R.U.” it is because they do not want to accept the conclusion or the reality it presents. There’s nothing that’s going to change this. I’m convinced that if President Putin came out and said “yea it was us” we’d have critics coming forward saying how it’s a false flag operation and it’s actually not true.

But what’s the problem with people arguing these points? It detracts from the already solid assessment. It’s similar to when the FBI wanted to release IP addresses and some technical indicators during the Sony hack to talk about how they knew it was North Korea. I critiqued that approach when it happened here. The basis of my argument was that the FBI’s attribution to North Korea was likely correct but their presentation of evidence as proof was highly misleading. Obviously the FBI didn’t just use those technical indicators to do the attribution, so how could anyone be expected to look at those and be convinced?  And rightfully so people came out and argued against those technical indicators noting they could easily be wrong and that adversaries of any origin could have leveraged the IP addresses for their operations. And the critiques were correct. The technical evidence in isolation was not good. The totality of the data set though was very sound and the analysis on top of it though were very sound.

I often think of this like climate change arguments. You can have 100 scientists with a career in climate studies posit forth an assessment and then two people with absolutely no experience argue on the subject. One of the people arguing for the climate scientists’ position could grab out a single data point to argue and now the person arguing against that first person is arguing against an uninformed opinion on a single data point instead of the combined analysis and work of the scientists. The two people arguing both leave understandably feeling like they won the argument: the original assessment by the scientists was likely right but the person arguing against the data point was also probably right about their argument against that data point. The only people who lost in this debate were the scientists who weren’t involved in the argument and who’s research wasn’t properly presented.

Closing Thoughts

I never like to just rant about things, I try to use these opportunities as things to learn from. All of this is actually extremely relevant to my SANS FOR578 – Cyber Threat Intelligence course so a lot of times I write these blog posts and reference them in class. So with that theme in mind here’s the things I want you to extract from this blog as learning moments (to my students, to the journalists, and to whomever else finds it valuable).

  • If you are doing research/writing on niche topics please find people with expertise in that niche topic (Jeffrey Carr is not an expert on attribution)
  • If you are going to posit that the entire public understanding of a nation-state group’s MO has changed because a single piece of evidence you’re likely wrong (do more homework)
  • If you are going to posit that there is a witness that can change the narrative about a case please talk to people familiar on the case (determine if that type of evidence is even important)
  • If you are going to write on a topic that is highly controversial research the previous controversy first (GRIZZLYSTEPPE was entirely unrelated to the DNC case)
  • Attribution is not done with single pieces of evidence or a smoking gun it is done as analysis on complex data sets most of which is not even technical (attribution is hard but doable)
  • The most interesting data for attribution isn’t highly classified but instead just hard work/analysis on complex scenarios (classification standards don’t imply accuracy or relevancy)
  • Just because someone’s code was used by an adversary does not imply the author knows anything about how it was used or by whom (the threat is the human not the malware)
  • Stop using pew pew maps (seriously just stop; it makes you look like an idiot)


Marketing ICS Vulnerabilities and POC Malware – You’re Doing it Wrong

April 30, 2017

There’s been two cases recently of industrial control system (ICS) security firms identifying vulnerabilities and also creating proof of concept (POC) malware for those vulnerabilities. This has bothered me and I want to explore the topic in this blog post; I do not pretend there is a right or wrong answer here but I recognize by even writing the post I am passing judgement on the actions and I’m ok with that. I don’t agree with the actions and in the interest of a more public discussion below is my rationale.


At the beginning of April 2017 CRITIFENCE an ICS security firm published an article on Security Affairs titled “ClearEnergy ransomware aim to destroy process automation logics in critical infrastructure, SCADA, and industrial control systems.” There’s a good overview of the story here that details how it ended up being a media stunt to highlight vulnerabilities the company found. The TL;DR version of the story is that the firm wanted to highlight the vulnerabilities they found in some Schneider Electric equipment that they dubbed ClearEnergy so they built their own POC malware for it that they also dubbed ClearEnergy. But they published an article on it leaving out the fact that it was POC malware. Or in other words, they led people to believe that this was in-the-wild (real and impacting organizations) malware. I don’t feel there was any malice by the company, as soon as the article was published I reached out to the CTO of CRITIFENCE and he was very polite and responded that he’d edit the article quickly. I wanted to write a blog calling out the behavior and what I didn’t like about it as a learning moment for everyone but the CTO was so professional and quick in his response that I decided against it. However, after seeing a second instance of this type of activity I decided a blog post was in order for a larger community discussion.

On April 27th, 2017 Security Week published an article titled “New SCADA Flaws Allow Ransomware, Other Attacks” based on a presentation by ICS security firm Applied Risk at SecurityWeek’s 2017 Signapore ICS Cyber Security Conference. The talk, and the article, highlighted ICS ransomware that the firm dubbed “Scythe” that targets “SCADA devices.” Applied Risk noted that the attack can take advantage of a firmware validation bypass vulnerability and lock out folks’ ability to update to new firmware. The firm did acknowledge in their presentation though and in the article that this too was POC malware.




Figure: Image from Applied Risk’s POC Malware

Why None of this is Good (In My Opinion):

In my opinion both of these firms have been irresponsible in a couple of ways.

First, CRITIFENCE obviously messed up by not telling anyone that ClearEnergy was POC malware. In an effort to promote their discovery of vulnerabilities they were quick to write an article and publish it and that absolutely contributed to hype and fear. Hype around these types of issues ultimately leads to the ICS community not listening to or trusting the security community (honestly with good reason). However, what CRITIFENCE did do that I liked (besides being responsive which is a major plus) was work through a vulnerability disclosure process that led to proper discussion by the vendor as well as an advisory through the U.S.’ ICS-CERT. In contrast, Applied Risk did not do that so far as I can tell. I do not know what all Applied Risk is doing about the vulnerabilities but they said they contacted the vendors and two of the vendors (according to the SecurityWeek article) acknowledged that the vulnerability is important but difficult to fix. The difference with the Applied Risk vulnerabilities is that the community is left unaware of what devices are impacted, the vendors haven’t been able to address the issues yet, and there are no advisories to the larger community through any appropriate channel. Ultimately, this leaves the ICS community in a very bad spot.

Second, CRITIFENCE and Applied Risk are both making a marketing spectacle out of the vulnerabilities and POC malware. Now, this point is my opinion and not necessarily a larger community best-practice, but I absolutely despise seeing folks name their vulnerabilities or naming POC malware. It comes off as a pure marketing stunt. Vulnerabilities in ICS are not uncommon and there’s good research to be done. Sometimes, the things the infosec community sees as vulnerabilities may have been designed that way on purpose to allow things like firmware updates and password resets for operators who needed to get access to sensitive equipment in time-sensitive scenarios. I’m not saying we can’t do better – but it’s not like the engineering community is stupid (far from it) and highlighting vulnerabilities as marketing stunts can often have unintended consequence including the vendors not wanting to work with researchers or disclose vulnerabilities. There’s no incentive for ICS vendors to work with firms who are going to use issues in their products for marketing for a firm’s security product.

Third, vulnerability disclosure can absolutely have the impact of adversaries learning how to attack devices in ways they did not know previously. I do not advocate for security through obscurity but there is value in following a strict vulnerability disclosure policy even in normal IT environments because this has been an issue for decades. In ICS environments, it can be upwards of 2-3 years for folks to be able to get a patch and apply it after a vulnerability comes out. That is not due to ignorance of the issue or lack of concern for the problem but due to operations constraints in various industries. So in essence, the adversaries get informed about how to do something they previously didn’t know about while system owners can’t accurately address the issues. This makes vulnerability disclosure in the ICS community a very sensitive topic to handle with care. Yelling out to the world “this is the vulnerability and oh by the way here’s exactly how you should leverage it and we even created some fake malware to highlight the value to you as an attacker and what you can gain” is just levels of ridiculousness. It’s why you’ll never see my firm Dragos or myself or anyone on my team finding and disclosing new vulnerabilities in ICS devices to the public. If we ever find anything it’ll only be worked through the appropriate channels and quietly distributed to the right people – not in media sites and conference presentations. I’m not a huge fan of disclosing vulnerabilities at conferences and in media in general but I do want to acknowledge that it can be done correctly and I have seen a few firms (Talos, DigitalBond, and IOActive come to mind) do it very well. As an example, Eireann Leverett and Colin Cassidy found vulnerabilities in industrial Ethernet switches and worked closely with the vendors to address them. After working through a very intensive process they wanted to do a series of conference presentations about them to highlight the issues. They invited me to take part to show what could be done from a defense perspective. So I stayed out of the “here’s the vulnerabilities” and instead focused on “these exists so what can defenders do besides just patch.” I took part in that research because the work was so important and Eireann and Colin were so professional in how they went about it. It was a thrill to use the entire process as a learning opportunity to the larger community. Highlighting vulnerabilities and creating POC malware for something that doesn’t even have an advisory or where the vendor hasn’t made patches yet just isn’t appropriate.

Closing Thoughts:

There is a lot of research to be done into ICS and how to address the vulnerabilities that these devices have. Vendors must get better at following best-practices for developing new equipment, software, and networking protocols. And there are good case-studies of what to do and how to carry yourself in the ICS security researcher community (Adam Crain and Chris Sistrunk’s research into DNP3 and all the things that it led to is a fantastic example of doing things correctly to address serious issues). But the focus on turning vulnerabilities into marketing material, discussing vulnerabilities in media and at conferences before vendors have addressed them and before the community can get an advisory through proper channels, and creating/marketing POC malware to draw attention to your vulnerabilities is simply, in my opinion, irresponsible.

Try these practices instead:

  • Disclose vulnerabilities to the impacted vendors and work with them to address them
    • If they decide that they will not address the issue or do not see the problem talk to impacted asset owners/operators to ensure what you see as a vulnerability is an issue that will introduce risk to the community and use appropriate channels such as the ICS-CERT to push the vendor or develop defensive/detection signatures and bring it to the community’s attention; sometimes you’re left without a lot of options but make sure you’ve exhausted the good options first
  • After the advisory is available (for some time you feel comfortable with) if you or your firm would like to highlight the vulnerabilities at a conference or in the media that’s your choice
    • I would encourage focusing the discussion on what people can do besides just patch such as how to detect attacks that might leverage the vulnerabilities
  • Avoid naming your vulnerabilities, there’s already a whole official process for cataloging vulnerabilities
  • (In my opinion) do not make POC malware showing adversaries what they can do and why they should do it (the argument “the adversaries already know” is wrong in most cases)
  • If you decide to make POC malware anyway at least avoid naming it and marketing it (comes off as an extremely dirty marketing approach)
  • Avoid hyping up the impact (talking about power grids coming down and terrorist attacks in the same article is just a ridiculous attempt to illicit fear and attention)

In my experience, ICS vendors are difficult to work with at times because they have other priorities too but they care and want to do the right thing. If you are persistent you can move the community forward. But the vendors of the equipment are not the enemy and they will absolutely blacklist researchers, firms, and entire groups of folks for doing things that are adverse to their business instead of working with them. Research is important, and if you want to go down the route of researching and disclosing vulnerabilities there’s value there and proper ways to do it. If you’re interested in vulnerability disclosure best practices in the larger community check out Katie Moussouris who is a leading authority on bug bounty and vulnerability disclosure programs. But please, stop naming your vulnerabilities, building marketing campaigns around them, and creating fake malware because you don’t think you’re getting enough attention already.

Analytical Leaps and Wild Speculation in Recent Reports of Industrial Cyber Attacks

December 31, 2016

“Judgement is what analysts use to fill gaps in their knowledge. It entails going beyond the available information and is the principal means of coping with uncertainty. It always involves an analytical leap, from the known into the uncertain.”

– Chapter 4, Psychology of Intelligence Analysis, Richards J. Heuer.


Analytical leaps, as Richards J. Heuer said in his must-read book Psychology of Intelligence Analysis, are part of the process for analysts. Sometimes though these analytical leaps can be dangerous, especially when they are biased, misinformed, presented in a misleading way, or otherwise just not made using sound analytical processes. Analytical leaps should be backed by evidence or at a minimum should include evidence leading up to the analytical leap. Unfortunately, as multiple analytical leaps are made in series it can lead to entirely wrong conclusions and wild speculation. There have been three interesting stories relating to industrial attacks this December as we try to close out 2016 that are worth exploring in this topic. It is my hope that looking at these three cases will help everyone be a bit more critical of information before alarmism sets in.

The three cases that will be explored are:

  • IBM Managed Services’ claim of “Attacks Targeting Industrial Control Systems (ICS) Up 110%”
  • CyberX’s claim that “New Killdisk Malware Brings Ransomware Into Industrial Domain”
  • The Washington Post’s claim that “Russian Operation Hacked a Vermont Utility, Showing Risk to U.S. Electrical Grid Security, officials say”


“Attacks Targeting Industrial Control Systems (ICS) Up 110%”

I’m always skeptical of metrics that have no immediately present quantification. As an example, the IBM Managed Security Services posted an article stating that “attacks targeting industrial control systems increased over 110 percent in 2016 over last year’s numbers as of Nov. 30.” But there is no data in the article to quantify what that means. Is 110% increase an increase from 10 attacks to 21 attacks? Or is it 100 attacks increased to 210 attacks?

The only way to understand what that percentage means is to leave this report and go download the IBM report from last year and read through it (never make your reader jump through extra hoops to get information that is your headline). In their 2015 report IBM states that there were around 1,300 attacks in 2015 (Figure 1). This would mean that in 2016 IBM is reporting they saw around 2,700 ICS attacks.


Figure 1: Figure from IBM’s 2015 Report on ICS Attacks


However, there are a few questions that linger. First, this is a considerable jump from what they were tracking previously and from their 2014 metrics. IBM states that the “spike in ICS traffic was related to SCADA brute-force attacks, which use automation to guess default or weak passwords.” This is an analytical leap that they make based on what they’ve observed. But, it would be nice to know if anything else has changed as well. Did they bring up more sensors, have more customers, increase staffing, etc. as the stated reason for the increase would not alone be responsible.

Second, how is IBM defining an attack. Attacks in industrial contexts have very specific meaning – an attempt to brute-force a password simply wouldn’t qualify. They also note that a pentesting tool on GitHub was released in Jan 2016 that could be used against the ICS protocol Modbus. IBM states that the increase in metrics was likely related to this tools’ release. It’s speculation though as they do not give any evidence to support their claim. However, it leads to my next point.

Third, is this customer data or is this honeypot data? If it’s customer data is it from the ICS or simply the business networks of industrial companies? And if it’s honeypot data it would be good to separate that data out as it’s often been abused to misreport “SCADA attack” metrics. From looking at the discussion of brute-force logins and a pentesting tool for a serial protocol released on GitHub, my speculation is that this is referring mostly to honeypot data. Honeypots can be useful but must be used in specific ways when discussing industrial environments and should not be lumped into “attack” data from customer networks.

The article also makes another analytical leap when it states “The U.S. was also the largest target of ICS-based attacks in 2016, primarily because, once again, it has a larger ICS presence than any other country at this time.” The leap does not seem informed by anything other than the hypothesis that the US has more ICS. Also, again there is no quantification. As an example, where is this claim coming from, how much larger is the ICS presence than other countries, and are the quantity of attacks proportional to the US ICS footprint when compared to other nations’ quantity of industrial systems? I would again speculate that what they are observing has far more to do with where they are collecting data (how many sensors do they have in the US compared to China as an example).

In closing out the article IBM cites three “notable recent ICS attacks.” The three case studies chosen were the SFG malware that targeted an energy company, the New York dam, and the Ukrainian power outage. While the Ukrainian power outage is good to highlight (although they don’t actually highlight the ICS portion of the attack), the other two cases are poor choices. As an example, the SFG malware targeting an energy company is something that was already debunked publicly and would have been easy to find prior to creating this article. The New York dam was also something that was largely hyped by media and was publicly downplayed as well. More worrisome is that the way IBM framed the New York dam “attack” is incorrect. They state: “attackers compromised the dam’s command and control system in 2013 using a cellular modem.” Except, it wasn’t the dam’s command and control system it was a single read-only human machine interface (HMI) watching the water level of the dam. The dam had a manual control system (i.e. you had to crank it to open it).

Or more simply put: the IBM team is likely doing great work and likely has people who understand ICS…you just wouldn’t get that impression from reading this article. The information is largely inaccurate, there is no quantification to their numbers, and their analytical leaps are unsupported with some obvious lingering questions as to the source of the data.


“New Killdisk Malware Brings Ransomware Into Industrial Domain”

CyberX released a blog noting that they have “uncovered new evidence that the KillDisk disk-wiping malware previously used in the cyberattacks against the Ukrainian power grid has now evolved into ransomware.” This is a cool find by the CyberX team but they don’t release digital hashes or any technical details that could be used to help validate the find. However, the find isn’t actually new (I’m a bit confused as to why CyberX states they uncovered this new evidence when they cite in their blog an ESET article with the same discovery from weeks earlier. I imagine they found an additional strain but they don’t clarify that). ESET had disclosed the new variant of KillDisk being used by a group they are calling the TeleBots gang and noted they found it being used against financial networks in Ukraine. So, where’s the industrial link? Well, there is none.

CyberX’s blog never details how they are making the analytical leap from “KillDisk now has a ransomware functionality” to “and it’s targeting industrial sites.” Instead, it appears the entire basis for their hypothesis is that Sandworm previously used KillDisk in the Ukraine ICS attack in 2015. While this is true, the Sandworm team has never just targeted one industry. iSight and others have long reported that the Sandworm team has targeted telecoms, financial networks, NATO sites, military personnel, and other non-industrial related targets. But it’s also not known for sure that this is still the Sandworm team. The CyberX blog does not state how they are linking Sandworm’s attacks on Ukraine to the TeleBots usage of ransomware. Instead they just cite ESET’s assessment that the teams are linked. But ESET even stated they aren’t sure and it’s just an assessment based off of observed similarities.

Or more simply put: CyberX put out a blog saying they uncovered new evidence that KillDisk had evolved into ransomware although they cite ESET’s discovery of this new evidence from weeks prior with no other evidence presented. They then make the claim that the TeleBots gang, the one using the ransomware, evolved from Sandworm but they offer no evidence and instead again just cite ESET’s assessment. They offer absolutely no evidence that this ransomware Killdisk variant has targeted any industrial sites. The logic seems to be “Sandworm did Ukraine, KillDisk was in Ukraine, Sandworm is TeleBots gang, TeleBots modified Killdisk to be ransomware, therefore they are going to target industrial sites.” When doing analysis always be aware of Occam’s razor and do not make too many assumptions to try to force a hypothesis to be true. There could be evidence of ransomware targeting industrial sites, it does make sense that they would eventually. But no evidence is offered in this article and both the title and thesis of the blog are completely unfounded as presented.


“Russian Operation Hacked a Vermont Utility, Showing Risk to U.S. Electrical Grid Security, officials say”

This story is more interesting than the others but too early to really know much. The only thing known at this point is that the media is already overreacting. The Washington Post put out an article on a Vermont utility getting hacked by a Russian operation with calls from the Vermont Governor condemning Vladimir Putin for attempting to hack the grid. Eric Geller pointed out that the first headline the Post ran with was  “Russian hackers penetrated U.S. electricity grid through utility in Vermont, officials say” but they changed to “Russian operation hacked a Vermont utility, showing risk to U.S. electrical grid, officials say.” We don’t know exactly why it was changed but it may have been due to the Post overreacting when they heard the Vermont utility found malware on a laptop and simply assumed it was related to the electric grid. Except, as the Vermont (Burlington) utility pointed out the laptop was not connected to the organization’s grid systems.

Electric and other industrial facilities have plenty of business and corporate network systems that are often not connected to the ICS network at all. It’s not good for them to get infected, and they aren’t always disconnected, but it’s not worth alarming anyone over without additional evidence.  However, the bigger analytical leap being made is that this is related to Russian operations.

The utility notes that they took the DHS/FBI GRIZZLY STEPPE report indicators and found a piece of malware on the laptop. We do not know yet if this is a false positive but even if it is not there is no evidence yet to say that this has anything to do with Russia. As I pointed out in a previous blog, the GRIZZLY STEPPE report is riddled with errors and the indicators put out were very non-descriptive data points. The one YARA rule they put out, which the utility may have used, was related to a piece of malware that is publicly downloadable meaning anyone could use it. Unfortunately, after the story ran with its hyped-up headlines Senator Patrick Leahy released a statement condemning the “attempt to penetrate the electric grid” as a state-sponsored hack by Russia. As Dimitri Alperovitch, CTO of CrowdStrike who responded to the Russian hack of the DNC, pointed out “No one should be making attribution conclusions purely from the indicators in the USCERT report. It was all a jumbled mess.”

Or more simply put: a Vermont utility acted appropriately and ran indicators of compromise from the GRIZZLY STEPPE report as the DHS/FBI instructed the community to do. This led to them finding a match to the indicator on a laptop separated from the grid systems but it’s not yet been confirmed that malware was present. The Vermont Governor Peter Shumlin then publicly chastised Vladimir Putin and Russia for trying to hack the electric grid. U.S. officials then inappropriately gave additional information and commentary to the Washington Post about an ongoing investigation which lead them to run with the headline that the this was a Russian operation. After all, the indicators supposedly were related to Russia because the DHS and FBI said so – and supposedly that’s good enough. Unfortunately, this also led a U.S. Senator to come out and condemn Russia for state-sponsored hacking of the utility.

Closing Thoughts

There are absolutely threats to industrial environments including ICS/SCADA networks. It does make sense that ICS breaches and attacks would be on the rise especially as these systems become more interconnected. It also makes perfect sense that ransomware will be used in industrial environments just like any other environment that has computer systems. And yes, the attribution to Russia compromising the DNC is very solid based on private sector data with government validation. But, to make claims about attacks and attempt to quantify it you actually have to present real data and where that data is coming from and how it was collected. To make claims of new ransomware targeting industrial networks you have to actually provide evidence not simply make a series of analytical leaps. To start making claims of attribution to a state such as Russia just because some poorly constructed indicators alerted on a single laptop is dangerous.

Or more simply put: be careful of analytical leaps especially when they are made without presenting any evidence leading into them. Hypotheses and analysis requires evidence else it is simply speculation. We have enough speculation already in the industrial industry and more will only lead to increasingly dangerous or embarrassing scenarios such as a US governor and senator condemning Russia for hacking the electric grid and scaring the public in the process when we simply do not have many facts about the situation yet.

Critiques of the DHS/FBI’s GRIZZLY STEPPE Report

December 30, 2016

On December 29th, 2016 the White House released a statement from the President of the United States (POTUS) that formally accused Russia of interfering with the US elections, amongst other activities. This statement laid out the beginning of the US’ response including sanctions against Russian military and intelligence community members.  The purpose of this blog post is to specifically look at the DHS and FBI’s Joint Analysis Report (JAR) on Russian civilian and military Intelligence Services (RIS) titled “GRIZZLY STEPPE – Russian Malicious Cyber Activity”. For those interested in a discussion on the larger purpose of the POTUS statement and surrounding activity take a look at Thomas Rid’s and Matt Tait’s Twitter feeds for good commentary on the subject.

Background to the Report

For years there has been solid public evidence by private sector intelligence companies such as CrowdStrike, FireEye, and Kaspersky that has called attention to Russian-based cyber activity. These groups have been tracked for a considerable amount of time (years) across multiple victim organizations. The latest high profile case relevant to the White House’s statement was CrowdStrike’s analysis of COZYBEAR and FANCYBEAR breaking into the DNC and leaking emails and information. These groups are also known by FireEye as the APT28 and APT29 campaign groups.

The White House’s response is ultimately a strong and accurate statement. The attribution towards the Russian government was confirmed by the US government using their sources and methods on top of good private sector analysis. I am going to critique aspects of the DHS/FBI report below but I want to make a very clear statement: POTUS’ statement, the multiple government agency response, and the validation of private sector intelligence by the government is wholly a great response. This helps establish a clear norm in the international community although that topic is best reserved for a future discussion.

Expectations of the Report

Most relevant to this blog, the lead in to the DHS/FBI report was given by the White House in their fact sheet on the Russian cyber activity (Figure 1).




Figure 1: White House Fact Sheet in Response to Russian Cyber Activity

The fact sheet lays out very clearly the purpose of the DHS/FBI report. It notes a few key points:

  • The report is intended to help network defenders; it is not the technical evidence of attribution
  • The report contains a combination of private sector data and declassified government data
  • The report will help defenders identify and block Russian malware – this is specifically declassified government data not private sector data
  • The report goes beyond indicators to include new tradecraft and techniques used by the Russian intelligence services

If anyone is like me, when I read the above I became very excited. This was a clear statement from the White House that they were going to help network defenders, give out a combination of previously classified data as well as validate private sector data, release information about Russian malware that was previously classified, and detail new tactics and techniques used by Russia. Unfortunately, while the intent was laid out clearly by the White House that intent was not captured in the DHS/FBI report.

Because what I’m going to write below is blunt feedback I want to note ahead of time, I’m doing this for the purpose of the community as well as government operators/report writers who read to learn and become better. I understand that it is always hard to publish things from the government. In my time working in the U.S. Intelligence Community on such cases it was extremely rare that anything was released publicly and when it was it was almost always disappointing as the best material and information had been stripped out. For that reason, I want to especially note, and say thank you, to the government operators who did fantastic work and tried their best to push out the best information. For those involved in the sanitation of that information and the report writing – well, read below.

DHS/FBI’s GRIZZLY STEPPE Report – Opportunities for Improvement

Let’s explore each main point that I created from the White House fact sheet to critique the DHS/FBI report and show opportunities for improvement in the future.

 The report is intended to help network defenders; it is not the technical evidence of attribution

There is no mention of the focus of attribution in any of the White House’s statements. Across multiple statements from government officials and agencies it is clear that the technical data and attribution will be a report prepared for Congress and later declassified (likely prepared by the NSA). Yet, the GRIZZLY STEPPE report reads like a poorly done vendor intelligence report stringing together various aspects of attribution without evidence. The beginning of the report (Figure 2) specifically notes that the DHS/FBI has avoided attribution before in their JARs but that based off of their technical indicators they can confirm the private sector attribution to RIS.



Figure 2: Beginning of DHS/FBI GRIZZLY STEPPE JAR

The next section is the DHS/FBI description which is entirely focused on APT28 and APT29’s compromise of “a political party” (the DNC). Here again they confirm attribution (Figure 3).



Figure 3: Description Section of DHS/FBI GRIZZLY STEPPE JAR

But why is this so bad? Because it does not follow the intent laid out by the White House and confuses readers to think that this report is about attribution and not the intended purpose of helping network defenders. The public is looking for evidence of the attribution, the White House and the DHS/FBI clearly laid out that this report is meant for network defense, and then the entire discussion in the document is on how the DHS/FBI confirms that APT28 and APT29 are RIS groups that compromised a political party. The technical indicators they released later in the report (which we will discuss more below) are in no way related to that attribution though.

Or said more simply: the written portion of the report has little to nothing to do with the intended purpose or the technical data released.

Even worse, page 4 of the document notes other groups identified as RIS (Figure 4). This would be exciting normally. Government validation of private sector intelligence helps raise the confidence level of the public information. Unfortunately, the list in the report detracts from the confidence because of the interweaving of unrelated data.



Figure 4: Reported RIS Names from DHS/FBI GRIZZLY STEPPE Report

As an example, the list contains campaign/group names such as APT28, APT29, COZYBEAR, Sandworm, Sofacy, and others. This is exactly what you’d want to see although the government’s justification for this assessment is completely lacking (for a better exploration on the topic of naming see Sergio Caltagirone’s blog post here). But as the list progresses it becomes worrisome as the list also contains malware names (HAVEX and BlackEnergy v3 as examples) which are different than campaign names. Campaign names describe a collection of intrusions into one or more victims by the same adversary. Those campaigns can utilize various pieces of malware and sometimes malware is consistent across unrelated campaigns and unrelated actors. It gets worse though when the list includes things such as “Powershell Backdoor”. This is not even a malware family at this point but instead a classification of a capability that can be found in various malware families.

Or said more simply: the list of reported RIS names includes relevant and specific names such as campaign names, more general and often unrelated malware family names, and extremely broad and non-descriptive classification of capabilities. It was a mixing of data types that didn’t meet any objective in the report and only added confusion as to whether the DHS/FBI knows what they are doing or if they are instead just telling teams in the government “contribute anything you have that has been affiliated with Russian activity.”


The report contains a combination of private sector data and declassified government data

This is a much shorter critique but still an important one: there is no way to tell what data was private sector data and what was declassified government data. Different data types have different confidence levels. If you observe a piece of malware on your network communicating to adversary command and control (C2) servers you would feel confident using that information to find other infections in your network. If someone randomly passed you an IP address without context you might not be sure how best to leverage it or just generally cautious to do so as it might generate alerts of non-malicious nature and waste your time investigating it. In the same way, it is useful to know what is government data from previously classified sources and what is data from the private sector and more importantly who in the private sector. Organizations will have different trust or confidence levels of the different types of data and where it came from. Unfortunately, this is entirely missing. The report does not source its data at all. It’s a random collection of information and in that way, is mostly useless.

Or said more simply: always tell people where you got your data, separate it from your own data which you have a higher confidence level in having observed first hand, and if you are using other people’s campaign names, data, analysis, etc. explain why so that analysts can do something with it instead of treating it as random situational awareness.


The report will help defenders identify and block Russian malware – this is specifically declassified government data not private sector data

The lead in to the report specifically noted that information about the Russian malware was newly declassified and would be given out; this is in contrary to other statements that the information was part private sector and part government data. When looking through the technical indicators though there is little context to the information released.

In some locations in the CSV the indicators are IP addresses with a request to network administrators to look for it and in other locations there are IP addresses with just what country it was located in. This information is nearly useless for a few reasons. First, we do not know what data set these indicators belong to (see my previous point, are these IPs for “Sandworm”, “APT28” “Powershell” or what?). Second, many (30%+) of these IP addresses are mostly useless as they are VPS, TOR exit nodes, proxies, and other non-descriptive internet traffic sites (you can use this type of information but not in the way being positioned in the report and not well without additional information such as timestamps). Third, IP addresses as indicators especially when associated with malware or adversary campaigns must contain information around timing. I.e. when were these IP addresses associated with the malware or campaign and when were they in active usage? IP addresses and domains are constantly getting shuffled around the Internet and are mostly useful when seen in a snapshot of time.

But let’s focus on the malware specifically which was laid out by the White House fact sheet as newly declassified information. The CSV does contain information for around 30 malicious files (Figure 5). Unfortunately, all but two have the same problems as the IP addresses in that there isn’t appropriate context as to what most of them are related to and when they were leveraged.



Figure 5: CSV of Indicators from the GRIZZLY STEPPE Report

What is particularly frustrating is that this might have been some of the best information if done correctly. A quick look in VirusTotal Intelligence reveals that many of these hashes were not being tracked previously as associated to any specific adversary campaign (Figure 6). Therefore, if the DHS/FBI was to confirm that these samples of malware were part of RIS operations it would help defenders and incident responders prioritize and further investigate these samples if they had found them before. As Ben Miller pointed out, this helps encourage folks to do better root cause analysis of seemingly generic malware (Figure 6).


Figure 6: Tweet from Ben Miller on GRIZZLY STEPPE Malware Hashes

So what’s the problem? All but the two hashes released that state they belong to the OnionDuke family do not contain the appropriate context for defenders to leverage them. Without knowing what campaign they were associated with and when there’s not appropriate information for defenders to investigate these discoveries on their network. They can block the activity (play the equivalent of whack-a-mole) but not leverage it for real defense without considerable effort. Additionally, the report specifically said this was newly declassified information. However, looking the samples in VirusTotal Intelligence (Figure 7) reveals that many of them were already known dating back to April 2016.



Figure 7: VirusTotal Intelligence Lookup of One Digital Hash from GRIZZLY STEPPE

The only thing that would thus be classified about this data (note they said newly declassified and not private sector information) would be the association of this malware to a specific family or campaign instead of leaving it as “generic.” But as noted that information was left out. It’s also not fair to say it’s all “RIS” given the DHS/FBI’s inappropriate aggregation of campaign, malware, and capability names in their “Reported RIS” list. As an example, they used one name from their “Reported RIS” list (OnionDuke) and thus some of the other samples might be from there as well such as “Powershell Backdoor” which is wholly not descriptive. Either way we don’t know because they left that information out. Also as a general pet peeve, the hashes are sometimes given as MD5, sometimes as SHA1, and sometimes as SHA256. It’s ok to choose whatever standard you want if you’re giving out information but be consistent in the data format.

Or more simply stated: the indicators are not very descriptive and will have a high rate of false positives for defenders that use them. A few of the malware samples are interesting and now have context (OnionDuke) to their use but the majority do not have the required context to make them useful without considerable effort by defenders. Lastly, some of the samples were already known and the government information does not add any value – if these were previously classified it is a perfect example of over classification by government bureaucracy.


The report goes beyond indicators to include new tradecraft and techniques used by the Russian intelligence services

The report was to detail new tradecraft and techniques used by the RIS and specifically noted that defenders could leverage this to find new tactics and techniques. Except – it doesn’t. The report instead gives a high-level overview of how APT28 and APT29 have been reported to operate which is very generic and similar to many adversary campaigns (Figure 8). The tradecraft and techniques presented specific to the RIS include things such as “using shortened URLs”, “spear phishing”, “lateral movement”, and “escalating privileges” once in the network. This is basically the same set of tactics used across unrelated campaigns for the last decade or more.



Figure 8: APT28 and APT29 Tactics as Described by DHS/FBI GRIZZLY STEPPE Report

This description in the report wouldn’t be a problem for a more generic audience. If this was the DHS/FBI trying to explain to the American public how attacks like this were carried out it might even be too technical but it would be ok. The stated purpose though was for network defenders to discover new RIS tradecraft. With that purpose, it is not technical or descriptive enough and is simply a rehashing of what is common network defense knowledge. Moreover, if you would read a technical report from FireEye on APT28 or APT29 you would have better context and technical information to do defense than if you read the DHS/FBI document.

Closing Thoughts

The White House’s response and combined messaging from the government agencies is well done and the technical attribution provided by private sector companies has been solid for quite some time. However, the DHS/FBI GRIZZLY STEPPE report does not meet its stated intent of helping network defenders and instead choose to focus on a confusing assortment of attribution, non-descriptive indicators, and re-hashed tradecraft. Additionally, the bulk of the report (8 of the 13 pages) is general high level recommendations not descriptive of the RIS threats mentioned and with no linking to what activity would help with what aspect of the technical data covered. It simply serves as an advertisement of documents and programs the DHS is trying to support. One recommendation for Whitelisting Applications might as well read “whitelisting is good mm’kay?”  If that recommendation would have been overlaid with what it would have stopped in this campaign specifically and how defenders could then leverage that information going forward it would at least have been descriptive and useful. Instead it reads like a copy/paste of DHS’ most recent documents – at least in a vendor report you usually only get 1 page of marketing instead of 8.

This ultimately seems like a very rushed report put together by multiple teams working different data sets and motivations. It is my opinion and speculation that there were some really good government analysts and operators contributing to this data and then report reviews, leadership approval processes, and sanitation processes stripped out most of the value and left behind a very confusing report trying to cover too much while saying too little.

We must do better as a community. This report is a good example of how a really strong strategic message (POTUS statement) and really good data (government and private sector combination) can be opened to critique due to poor report writing.



The DHS released an updated version which I thought did a great job of analysis; my analysis of it can be found here: