Attribution is not Transitive – Tribune Publishing Cyber Attack as a Case Study

December 31, 2018

I made a number of tweets on this subject but then the voice of Richard Bejtlich entered my head and told me that all twitter threads should be a blog post, and here I am. This blog looks at the cyber attack on Tribune Publishing and the claims that North Korea is responsible as an opportunity to highlight that attribution is not a transitive property.

 

Shortly after Tribune Publishing lost operations and ability to print papers the press highlighted that there was a cyber attack. The attack was highlighted as a targeted attack by a nation-state. This was all related to one anonymous insider at the company telling the media. Thus, early on I, and many others on social media, called for calm and patience while the details became public. The details are still not public and the company hasn’t officially responded but an insider told media sources that the malware used in the attack was Ryuk which is a family of ransomware (Checkpoint did a great write-up on it here). Checkpoint did some great analysis on the malware and noted that there is commonality in some aspects of the malware and another family of malware called Hermes. They appropriately highlight that while Hermes has been attributed in use to Lazarus Group before, there are alternative explanations including the group who developed Ryuk having access to the source code of the Hermes malware. There are likely alternative hypotheses not explored here as well. However, that link is seemingly being used by others to draw the conclusion that Tribune Publishing was attacked by North Korea.

Here is Forbes making that claim. They are not the only ones though. Fortunately, some journalists took a different approach. Here the New York Times accurately notes that just because (and if) Ryuk was used doesn’t mean it has government ties (thanks David and Nicole!). They also introduce alternative hypotheses including Adam Meyers from CrowdStrike stating that CrowdStrike tracks an eastern european cyber crime group that leverages the malware.

So what is the logic that led to the North Korean claims and what lessons can we extract?

Seemingly, the logic leveraged was that Ryuk has a link to Hermes. Hermes has links to Lazarus Group. Lazarus Group has been attributed to North Korea. Therefore, all uses of Ryuk must be North Korea. That is transitive attribution and is an association fallacy.

The logic seems kind of sound though, so what’s the problem?

There are a few large issues at play for us to explore.

First, we all have a collection bias. I.e. what we analyze is based off what we collect. We cannot know the true extent of collection available, so it is common for analysts to assume their collection is pretty good in comparison to what’s available. In fact though, it’s almost always the opposite where our collection is much worse than we realize. If Ryuk or Hermes malware was leveraged by teams other than Lazarus that would pose a big issue for the attribution claims. The “uniqueness” of malware is directly tied to collection. In perfect collection you could factually state if malware is unique to one team or not. But without perfect collection, and no one has perfect collection, we must understand that malware may appear unique to a specific team but may not be unique to them at all. It just may be unique to them in our collection.

Second, the links Checkpoint drew were not definitive and had other hypotheses identified. Therefore, if an assessment is going to be drawn out for attribution purposes and not the malware analysis purposes done in their blog, we’d need to do a more structured assessment including more data sources such as additional intrusions and cluster those intrusions using some model like the Diamond Model for Activity Groups. Moving the malware usage to a cluster of intrusions would reveal more data points to then start working on a more structured assessment.

Third, I have not looked deeply into Hermes but we’d want to explore the connections Hermes had to Lazarus Group and do the same type of analysis on the links mentioned in the second point for Ryuk.

Fourth, Lazarus Group is a collection of clusters of intrusions from across multiple researchers, teams, and organizations. Whereas Lazarus Group was at one point decently well defined it has come to represent to many a larger clustering of anything North Korean in nature with links to any aspect of known Lazarus Group activity. This is not a put down on any team that tracks the Lazarus Group, it’s simply a realization that the analyst bias that goes into selecting intrusions and putting them into the Lazarus Group is done differently across different teams and thus a super group is not likely to be granular in its accuracy. (I talk about the problem with this type of threat tracking here). This may not matter at all if you want to attribute the principal group responsible for Lazarus Group as North Korea. But attribution, especially when you want to attribute all parties and not just the one chiefly responsible, is not binary. Not every single intrusion that goes into the clustering of “Lazarus Group” is going to be accurate. Not every single intrusion is going to be North Korea developed malware used by North Korea operators. There are alliances (North Korea allies in other states or organizations), there are supply chains (where they source exploits, code, etc.), there are operators vs. developers, there are different operations teams, there are different customers of the operation’s intelligence requirements, etc. to consider. All this means that if you want to do attribution to North Korea off of Lazarus Group you can get to a pretty good confidence level (likely Moderate Confidence if you’re just basing it off of intrusion analysis). If you’re wanting to reverse engineer individual aspects of that grouping though the attribution wouldn’t necessarily hold. I.e. individual families of malware, intrusions, aspects of malware such as encoding routines, etc. could all be an important puzzle piece in multiple puzzles, not just Lazarus Group.

The fourth point is the biggest hindrance in attribution being transitive. All the puzzle pieces that go into doing an assessment can be important. But by themselves they are likely not. I’ve seen so many people ask for the “smoking gun” when talking about intelligence analysis. The FBI’s attribution of North Korea to the Sony Attack comes to mind (which I wrote about here in Wired) where the FBI’s assessment was sound but the infosec community wanted them to “prove it” so they released some technical pieces of evidence, which to the FBI probably seemed pretty good in hindsight but to the public were not conclusive. This is a common analyst mistake. When you do analysis there are pieces of evidence that become really important to you, but only in context of all the analysis you did. I.e. it’s really important to you now with all the knowledge you have about the case. But you needed a lot of other data and context to have it be important. So releasing just “the important stuff” externally will not likely resonate with others who cannot come to the same conclusions you did on just partial data. Even with identical data sets two analysts will likely come to different conclusions anyway. To address this you never get in the habit of arguing about evidence, you position and argue your assessment. The totality of the data and your analysis, not just pieces of data.

All this is a round about way of saying that if you take a piece of data from an assessment (such as links to Hermes malware) and take it away from all the other data, then you cannot take the assessment with that piece of data. You cannot just simply look for Hermes malware to pop up and go “yup that’s Lazarus Group”. Further, links of Hermes to other malware families like Ryuk and thus attacks where Ryuk show up further complicate the issue. The more analytical leaps you make the less likely your assessment is going to be sound.

This doesn’t mean that the attack wasn’t done by North Korea. If it was knowing their intention would be an entirely different and especially difficult assessment to make. But at this point, no actual assessments have been done. The only thing being highlighted in certain media outlets is transitive attribution because of links observed in different malware families. This is sloppy and will lead to numerous inaccuracies. Additionally, there can be political issues if high profile targets like the New York Times and Wall Street Journal (luckily they haven’t) come out and attribute the attack to North Korea. That puts pressure on the US government as well as the North Korea government. There are real impacts to attribution claims between states.

In summary, as an analyst you should be aware that assessments do not often have a transitive property. Understand your collection biases and what goes into the assessments you make. From there, if you need to make a new assessment, then you need to go through the process of collecting data and analyzing and producing an assessment, short cuts such as transitive analysis will not be better than a low confidence assessment. Do not strive for perfection where you have analysis paralysis (sometimes it’s ok to make gut calls as an analyst) but understand when something is a guess, a hypothesis that’s missing plenty of data sources and other hypotheses are also equally possible (low confidence), or when you’ve done structured analysis across multiple data sources to achieve a higher level of confidence (moderate or high).

Threat Hunting, TTPs, Indicators, and MITRE ATT&CK – Bingo

November 25, 2018

This blog is a continuation of a fantastic discussion with Richard Bejtlich. He responded to a question online, I blogged about it here and then he responded here and in response to another question I posed here. In this blog I’ll reply to his replies. The main point of this blog to me though is not to debate but to document my own experiences and look briefly at the evolution of terms and schools of thought as it applies to our cybersecurity field (the term cybersecurity vs. infosec is actually a great example of a term that has evolved yet to many still have two very different meanings and to some are interchangeable).

TL;DR Version: Richard and I seem to disagree on what an indicator is (reasonably so as his definition of one is perfectly supportable I just have a different view based on different experiences) which I didn’t previously realize was impacting how both of us viewed threat hunting. In doing so it seems that we’ve both come up under different “schools of thought” and therefore “hunting” can be leveraged in many different ways but you should strive to document and put structure to it if possible to really get the most value out of it.

Historical Context Matters But We Shouldn’t be Bound To It

In Richard’s blog he masterfully brings in some historical context of hunting’s origins (which few people really bring in history in a way Richard does but more should), or at least what was first documented (much which was done by him), in reference to the U.S. Air Force and National Security Agency. He also masterfully gives credit (another thing we should all strive more to do) to folks like Tony Sager and where his journey took him from the Air Force CERT to GE-CIRT where his use of the term was solidified even more with the wonderful analysts he cites.

What hunting was in the U.S. Air Force in the mid 2000’s and what it was at GE-CIRT shortly after doesn’t have to define the term for the rest of the field. It’s an amazing contribution, but not definitive. I think that’s, on face value, a frustrating topic. If I was at the GE-CIRT as an example and I really fleshed out a term my natural reaction would be to push against people using it differently (Richard is not doing that, I’m making a different point) but in reality terms, use-cases, and our field’s knowledge in general is constantly growing. There’s actually a logical fallacy called Appeal to Tradition which essentially states that something is what it is because it’s always been that way. But what’s interesting here is “hunting” was something numerous groups did. None documented or arguably had as much impact as the folks like Tony Sager, Richard, NSA VAO, AF-CERT, and the GE-CIRT but the experience is not less valid.

As an example, in my own experience in the National Security Agency one of my jobs was the Discovery Lead for one of the NSA Threat Operations Centers (NTOC) (would have been close proximity to Tony’s group but operating independently). Prior to this I had no knowledge of the term “hunting” and was not exposed to the schools of thought that were being developed at GE-CIRT or, interestingly enough, the use of the term and practice that Tony Sager and VAO were pursuing. I was part of a small team that established a new NTOC office at a field site and we were tasked with finding the “unknown unknowns”. This was admittedly very vague guidance but it was explained to us that the role of our NTOC office was explicitly finding the cyber threats that weren’t already documented and being tracked. Find what’s evading our insight to reveal collection gaps. We called this “Discovery” which, on a day-to-day basis, became known as “hunting.” The line between hunting and cyber threat intelligence though were very blurred for us because of our requirements; I would note that hunting was one way we went about satisfying our cyber threat intelligence requirements by identifying previously unknown intrusions (hunting) that we would then analyze (CTI). What we effectively were doing was taking an intelligence requirement, collecting against it through hunting, analyzing the intrusions we observed, producing insights, and distributing it to others as blogs, defensive mitigations and new detections, and reports. We used models such as the newly developed Diamond Model (previous to the paper’s publication) under the tutelage of Sergio Caltagirone and if I remember correctly we were the first team or one of the first to do so outside of his where he, Andrew Pendergast, and Chris Betz created the model. The interesting thing to the discussion here is that “hunting” was something that developed for my team without, to my knowledge, external prompt although I imagine there was cross-pollination from Sergio, Andrew, and Chris with the various teams they interacted with. For us, Discovery and our use of the term “hunting” was always about “finding new threats” but the main value in doing that was in identifying new defensive recommendations and gaps in collection even though we also found plenty of new threats to track. (For anyone curious I ended up choosing industrial control systems as the focus for my Discovery team since the collection would be so different and thus maybe the threats would be to; it turns out they were. This was a defining moment for me in ICS cyber security and a lot of what I had to develop in knowledge there at the NTOC helped in my journey and greatly inspired my work today at Dragos). Interestingly though one of the ways we found new threats was in the application of adversary tactics, techniques, and procedures as analytics/patterns instead of specific indicators. This aspect seems to distance Richard and I further which I’ll cover in the next section. But to close out the topic on the value you get out of hunting…

Richard acknowledges that you can identify visibility and resistance gaps as part of hunting, but it’s not the reason to go hunting.

 

In response to that I would say it may not be the reason that he and others hunt but it was always the reason my team hunted early in my career and it shaped how I view hunting now. His school of thought is simply different. I would also wonder aloud if a term needs to change when the actions are the exact same but the value propositions you’re aiming for are numbered differently. If you do exactly the same things and use exactly the same approach but you hope to find threats instead of collection gaps should you rename your effort? I’m not sure on that yet but my initial thoughts would say no. The “school of thought” for hunting and how to achieve it was very specific for Richard and me; I assume there are others in other organizations and parts of the world who have similar experiences that make their schools of thought much different. Richard and I are obviously both heavily influenced by a U.S. Department of Defense flavoring. I’d be interested if anyone else had their own journeys to document around the same time period.

We Were Bound To Disagree

It is clear that Richard and I disagree on the term hunting and how it’s used but the important part is why we disagree. I actually have no issues whatsoever with how Richard is defining the term and its background for his uses; his experiences are unique and many including myself have benefited from them. I don’t think there is a right or wrong to this discussion, just a friendly exploration to flesh out the topic for ourselves (or at least mine) and others. If we go back to the original points though it was clear Richard and I were bound to disagree and I didn’t see it initially.

I referenced but then moved past a point early in my other blog that Richard referred to all threat detection falling into two categories of either “matching” or “hunting”. I thought it was an interesting discussion which I slightly disagreed with but hurried past to get to the core debate. I noted that everything is “Matching” or “Unknowns” but not that hunting is contained in one or the other; it can be across both. If you are matching indicators you are not hunting in my opinion which I think Richard would agree with. If you are matching adversary behaviors though you might be depending on how you’re doing it (if the behavior is already a defined detection then no, but if it’s a pattern you’re searching out to find new activity that isn’t defined, then yes). When in fact, that was my error and in reality the disagreement is rooted there. A subtle but important point because I didn’t realize (although to his credit he’s definitely written on it before) that Richard considers adversary TTPs to be a sub-class of indicators of compromise (IOCs). I asked him that question after his first blog and he was kind enough to answer it here.

Again, Richard is great to identify and credit influential folks such as David Bianco and Chris Sanders, both who I consider friends and have written things that have definitely helped me further my thoughts too. He references David’s Pyramid of Pain (an extremely useful model by the way) where it very clearly calls out TTPs as a type of Indicator. I’m going to disagree again, surprise surprise, but this is where everything “came together” for me in that the disagreement is in the evolution of terms and schools of thought and nothing more. David’s use of the term indicator is mostly in keeping with another foundational work by Eric Hutchins, Mike Cloppert, and Rohan Amin titled Intelligence-Driven Computer Network Defense Informed by Analysis of Adversary Campaigns and Intrusion Kill ChainsYes the kill chain paper. Here they define out indicators to be atomic, computer, or behavioral. Behavioral indicators here are defined as those indicators that are made up of atomic and computer indicators to describe adversary activity, the example given in the paper is “the intruder would initially used a backdoor which generated network traffic matching [regular expression] at the rate of [some frequency] to [some IP address] and then replace it with one matching the MD5 hash [value] once access was established.” Where David’s use of the term seems to be in disagreement is “behavioral” which on its face would speak to TTPs but in reality TTPs can be described and leveraged now without any atomic or computed indicators to accompany them. My second point in the next paragraph will dive deeper into that.

Why is this such an important point to this discussion? For two main reasons.

  • First, terms and schools of thought evolve. “Indicator” today is almost exclusively associated with indicator feeds and the atomic and computed form of indicators. Some still use behavioral indicators and talk about them quite a bit to great value. But for the majority of the industry if you say “indicator” they’ve come to learn about them, use them, and see value propositions defined by atomic and computed values. Security professionals using indicator that way or not wrong. We even see another “school of thought” coming up which is based around MITRE’s ATT&CK; quite a few of their use-cases would fit nicely into mine by using ATT&CK as one framework for guiding threat hunts. In one of their papers they specifically note the focus on tactics and techniques (TT in the TTP) for them is important to go beyond “traditional indicators of compromise.” Here it would seem they are not using TTPs as a form of IOC either.
  • Second, what you can to do today for detection was not necessarily possible for most teams as little as a decade ago. The cybersecurity community has evolved quickly made many more advancements than we often credit ourselves. One such advancement is in the form of analytics. Analytics are effectively questions you ask of the data. Analytics have been around a long time in many forms but with the advancement of the community and of computing power we can now use analytics in more large scale ways and in a distributed fashion at scale (run the analytics at the edge where the data exists instead of pulling all the data back and then running analytics across it). One type of analytic, that I wrote about and referenced in the last blog when I mentioned the four types of detection paper, are threat analytics. Threat analytics effectively are adversary behaviors, i.e. TTPs or tradecraft (different things by the way). But they are not behavioral indicators in the way Hutchins, Cloppert, and Amin identified them. They don’t include any atomic or computed indicators; post detection there will be indicators but you don’t define the indicators ahead of the detection. The entire analytic might say “alert any time a file is dropped to a system, opens up a network port, generates network traffic to an external IP address, and then downloads an additional file once communications are established”. This analytic would get to the example given in the kill chain paper but without knowing the hash or IP address of the backdoor or anything about the adversary leveraging the behavior. This is done through an analytics engine now which have been around for awhile but are more common and accessible than ever before. When the analytic is defined it is “matching” to Richard’s point. But when I’m leveraging the TTP or tradecraft outside of the analytic to go search for and find new threats (numerous threats can leverage identical TTPs, so you’re searching for “unknown” threats using “known” tradecraft) I’m not matching any indicators and instead am using an intelligence-driven hypothesis to identify new threats. That’s EXACTLY what we did at the NTOC site I was at and we called that hunting.

As an aside, I think my second point is even how Richard is doing  his hunting to some degree because in his blog he gives an example that you could tell an analyst to go look go HTTP user agents as they are being leveraged by the adversary for malicious network communications but not tell them what user agents to look for, that at a high level is a tactic of the adversary which would classify as a component of a TTP and not be an indicator. It is not some anomaly that is occurring to filter through but a hypothesis generated from some insight the defender had such as seeing adversaries do it before (intelligence-driven) or based on their own expertise in how the protocol should work (domain expertise). I don’t want to argue points for Richard though so maybe I’m not interpreting it correctly but I think if we spent more time (preferably over beer) on this we’d probably root out more commonalities in approaches.

All of that long winded way of saying: Richard and I fundamentally seem to disagree on what an indicator is which is having a flow down effect I didn’t previously realize to how we view and define threat hunting. We would still end up debating back and forth on the ordering of the value propositions but both agree that the value propositions all exist (find threats, identify gaps, develop new detections, etc.) which is another reason everyone should be hunting regardless of how you order the value you get out of it.

This has been fantastic. I have thoroughly enjoyed this back and forth, and thank you again Richard for being such a gracious person to explore it all with me while challenging me to document my experiences in this blog.

Hunting vs. Incident Response vs. Just Doing Your Job

November 22, 2018

One of the things that motivates me to write is either being really pissed off or finding someone I really respect and having a different opinion. This blog is about the latter. Richard Bejtlich tweeted (oh the horror) that to him, hunting is just another form of detection. Now his real argument is of course more nuanced and well thought out but for the purpose of my blog and putting forth my view I’ll try to make his as simplistic as possible.

 

Now in reality, debating with Richard about topics related to detection, network security monitoring, and consequently threat hunting is probably similar to debating Plato about societies in the Republic. But I’ve always found myself learning more from debating smart people about small disagreements than debating the ones I outright disagree with. So enough compliments, let’s get into it.

The Argument

The heart of the argument, as I’m positioning it, is about what hunting is and is not and thus how folks use it. The initial prompt was from Twitter user @hellor00t about where “hunt teams” should be placed in the CIRT and whether or not hunt is just a new term for IR analyst. The comment was “seems like title semantics to me.” Which is totally fair. Richard, as previously stated, noted that hunt is just a form of detection. That incident response teams detect intruders using two major modes: matching and hunting. There’s a lot to unravel here but I’ll bypass the modes of detection argument slightly to say I actually agree but wouldn’t call one mode hunting. You’re either matching knowns (behaviors of adversaries, indicators, behaviors of users, etc.) or exploring unknowns (anomalies). You should cover your knowns first and then explore the unknowns but your strategy for detection should be more well thought out. I covered that topic with Sergio Caltagirone in a paper on the four types of threat detection which actually lines up nicely with Richard’s point (except the term hunting) here.

My Position

Honestly you can do whatever you want that gets you excited about security and proves to be an efficient and effective way to do your job for the organization. But I do think hunting goes beyond Richard’s point. I’ve written about threat hunting (not nearly as much as Richard) previously a few times before but here are two papers that help phrase my “school of thought” a bit that I wrote with David Bianco on generating hypotheses and another about what hunting is and isn’t with the “other” Rob Lee at SANS. My main thought process is that threat hunting is a proactive and iterative approach to detecting threats. It seems to align directly with what Richard wrote but I’d add that it really isn’t, to me, about detecting threats. I know that’s super confusing so I’ll go into depth a bit more.

Hunting is a hypothesis-led approach to testing your environment for threats. The purpose, to me, is not in finding threats but in determining what gaps you have in your ability to detect and respond to them. As an example, how do you know whether or not your organization would ever be able to detect and respond to the latest tradecraft observed by $ActivityGroup? You should generate a hypothesis about how that tradecraft would look in your environment played out against what you have to protect. You could determine the “route” the intrusion might take based on what you’ve observed previously and see if you have the people, process, and technology required at each point to successfully detect the tactics and procedures of the threat as well as the ability to respond to them. That’s only one way to generate hypotheses but hopefully serves as a tangible example. You’re looking to achieve a layered defense while figuring out your gaps. As an output of hunting, you should have gaps to address as well as detections you can now create as a result of your hunting (i.e. threat hunting cannot be fully automated but the output of your hunting should be more automation in your environment against the threats you hunted for). You could also create playbooks, or step-by-step guidance, on what an analyst should do to validate malicious activity and how to respond based on what you learned. Additionally, you could update your collection management framework (a topic I’ve talked about before and will have a paper out later this month on) to discuss what collection and data sources you do and don’t have in your own environment.

In short, hunting, to me, is a way to assess your security (people, process, and technology) against threats while extending your automation footprint to better be prepared in the future. Or simply stated, it’s incident response without the incident that’s done with a purpose and contributes something. Ad-hoc hunting to me isn’t hunting, it’s just proactively searching for threats in your environment (although admittedly saying you’re “hunting for threats” rolls off the tongue a lot better than “log correlating and searching”). Responding to alerts isn’t hunting either, it’s just triaging and investigating alerts. However, there are many “schools of thought” and you’re welcome to do whatever works for you in your environments.

Threat hunting is also an extremely useful way to measure the effectiveness and breadth of your threat intelligence efforts. Dan Gunter wrote an excellent Masters paper at SANS here on the topic.

Back to the original question of “where do you put your hunt team?” I really wouldn’t have a dedicated threat hunting team. I’d treat threat hunting as a multiple times a year or at least once a year assessment against the scenarios that are most concerning to you and your organization. Bringing together people from across the security organization and even outside that organization can be really useful to thinking outside the box about your hypotheses generation efforts. Having a dedicated team can make the process stale and cause too much group think. I’d rather see people “take turns” in that role if there’s a need to have a dedicated function.

In Conclusion

In conclusion (for now I assume) I will note I probably don’t disagree with Richard much but it was a useful prompt to have me think about threat hunting and put some thoughts in a blog to hopefully help others flesh out their views. My view is that hunting is not a form of detection, or isn’t just a form of detection. If you’re measuring the value of threat hunting on how many threats you find you’re likely not to be able to justify it at all compared to just doing proactive security work. Threat hunting, in my opinion, should be a much more structured test against hypotheses that is pushing your organization forward and ensuring you’re prepared against “styles” of threats. I do completely agree with Richard’s view though that Response is too narrowly defined as just incident response and after the fact to folks. I talk about “detection and response” largely because the terms can be useful in communication and when defining skill sets of folks. However, I do agree that detection and response both should be tightly coupled in people, process, and technology and that response is, in its best form, proactive.

 

What It’s Like to Raise Venture Capital (Just My Experience)

November 22, 2018

Last week it was announced that Dragos, Inc. raised $37M in our Series B financing event. That brings the total amount of money raised for the company to $48.2M since the company was founded in 2016 and marks my third time raising (Seed Round of $1.2M, A Round of $10M, and now the B Round). There’s a lot written explaining venture capital so I won’t attempt to explain it here in a short blog; but the purpose of this blog is to share some insights into things I’ve learned as a security practitioner that were new and surprising to me or just general observations that were interesting enough to document. Over the past two years I’ve taken more than 300 calls with more than 150 venture capitalists, officially “pitched” to more than 50, and had term sheets from about a dozen. I’ll start off with a comment about the team and company before diving into some of the observations I’ve had across those experiences.

“If you want to go fast, go alone; but if you want to go far, go together”

The proverb/quote above has some controversy around it regarding its origins but I like the quote nevertheless. It explains a lot about startups (to be fair so does the TV show Silicon Valley). When Jon Lavender, Justin Cavinee, and I started Dragos as co-founders we had previously built CyberLens together back in 2012/2013 time frame under the name “Dragos Security.” The tool was simply an assessment tool to process out packet captures and draw a map including some deep packet inspection of industrial control system (ICS) protocols. Getting it together as three people was really easy. It was a good year into the life of Dragos, Inc. when we were still adding things into the Dragos Platform that were already accomplished in CyberLens. The difference of course was the scale and stability of the Dragos Platform, our technology at Dragos, was significantly more vast than CyberLens. As we started adding a “feature” into the product it would break into numerous features and requirements. In theory what our technology does is simple: passively map out industrial environments, utilize threat analytics to identify the M.O./behavior/pattern of adversaries instead of just indicator or anomaly based detection, and offer up playbooks in an analyst workbench that act as step-by-step guidance to validate the alerts and scope the investigation. Not too bad right? Over the last two years we’ve seen the need to scale to 10 network appliances back to a server, to 40, to 300, and beyond. All on-prem without being able to take advantage of the cloud (in most cases) because of the sensitivity of industrial environments. At Dragos we’re about to be 100 people strong. You move so quickly you don’t really stop to think about the numbers and scale and all the things you’re doing. But one of the reasons startups tend to outpace larger companies is the ability to move those folks in a single direction and throw the entire organization behind it without as much communication challenges as large companies; flexibility is key. It’s amazing to see that occur and yet moving forward always seems like it takes place at a snail’s pace. You look back over 1 quarter and you realize that’s not true; but going far definitely requires the team and creating a scalable, deployable, stable, and usable technology often requires more than just a few people. I’m proud of the team we have today and the growth we’re achieving. Every now and then I sit back, pour a stiff drink, and just have to feel the exhaustion overcome me all at once. But then you sit forward and keep going again. You see the difference you’re making for your customers, the lives of the people on your team changing and growing, and you just feel so fortunate. Luckily, my eight month old son and wonderful wife provide more centering for me than exhaustion making the journey a bit more doable.

Venture Capital is Neither Good nor Bad

As a security practitioner I often found myself opining about venture capital. Bankers, private equity folks, finance majors, and “business people” were what I always assumed made up all the ranks of Venture Capital and to be honest I couldn’t stand the idea of them. I didn’t despise anyone, everyone takes their own path in life. But non-security practitioners trying to fund and decide “market winners” without an appreciation of the technology always bothered me. For anyone that’s seen my writings before it’s probably pretty apparent that I don’t think too highly of “management” positions leading security teams or technology teams without an understanding of security or technology. So it extended beyond venture capital. I also saw venture capitalists as just interested in making a buck (that is their job but the amount of “let’s make the world a better place” I’ve seen even in the hard times has been stunning coming from them) and largely figured it was better to bootstrap a business than take capital. While a lot of bad venture capitalists exist I learned that the bad apples were really giving a bad name to a much larger and more diverse field. Here’s some areas that I have evolved my own thinking:

Watch the Burn, Spend the Money, Watch the Burn

One of the reasons I hated the idea of venture capital is that I figured and had heard they would try to get you to spend so much money that you would have to raise more money later. They’d “get you over a barrel” and you’d be too deep in debt to do anything but go forward and raise more. That impression I had was entirely wrong, maybe I’ve just seen a different group of venture capitalists but there’s a whole science and art to throttling the business correctly. I’ve heard “watch the burn” from my board members (made up of our investors) to the point that it’s annoying not because it’s not accurate but because it’s been stated so much it couldn’t possibly have left my mind. The burn is how fast you’re burning through capital. Venture capitalists don’t want you to spend too much money, they want you to spend the right amount of money to invest in the business’ future. The logic is actually pretty simple when you think about it. In an enterprise software styled company where you are investing heavily in R&D you’re going to need to raise venture money to support it (sales can’t support the R&D costs especially before you have a product). But once you have the product (though it’s never done) you still have to invest heavily in sales and marketing. If I put $1 into sales and marketing today I likely won’t see any return on that for 9+ months. It takes time to generate the lead, make the contact, get the meeting, make the pitch, fit into this or next year’s budget, etc. The sales cycle can easily be 9-18 months long. But if I can invest $1 into sales and marketing today and get a return of $2 in 12 months that’s a 100% growth. Throttling the business to be “profitable” especially early on is robbing the future of the business. If I can lose $5M this year and make $10M in sales next year, that’s a great thing for the business. Thus venture capital is needed so that you can spend what you need to, but you don’t just want to spend money you want to understand what the return on investment is and monitor your burn as closely as possible to make sure you’re not over investing or under investing. This can cause a “foot on the gas foot off the gas foot on the gas” back and forth in the company on an almost weekly basis. There’s risk into taking this approach and it’s not the only one but previous-me would have stated “well just grow as you can support it and don’t try to lose money in hopes of earning more business next year.” Today I understand that the speed of the market, demand from the customers, growth of competitors, marketing from incumbents, and more all goes into deciding that spending large sums of money to grow the business is hyper critical especially in a software company where deal sizes and sales cycles are both large.

Lots of Different Types of Venture Capitalists 

There are a lot of different types of venture capitalists, and I’m talking about the people not the firms yet, I’ll approach those next. Some venture capitalists are the bankers, private equity folks, business grads, and financial analysts that I had previously thought about. But many are also “operators” that have built and led companies before. In reality there are good and bad in both forms. I’ve found that as a founder I really resonate with operators. Having a few of them on my board early on gave me good guidance and made a collaborative environment. They were also more empathetic to the needs of the business instead of spreadsheets and trying to math the company to death. As we grew it was also important to add those who had more experience as the “typical” type of finance people who helped provide a bit of devil’s advocacy to the board on various decisions and help make sure that the “emotion” of running the business and the “experience” of having done it before wasn’t completely out of touch with what the numbers could support. It’s obvious of course that it takes all types, but to me seeing the different value propositions of people and their experience and how they influenced the company was interesting. As a gaming nerd there’s so much that role playing games have taught me about business, more so than any of the free online classes I’ve taken on business (seriously you should watch those free videos like the free MIT and Harvard classes and you should throw in some good RPGs as homework). Class skills and development of different professions while understanding people’s skill tree maximums and strengths/weaknesses is much easier when you just imagine everyone you meet has a playing card or a stats page that you can see when you hover over them.

Lots of Different Types of Venture Capital Firms

There are many different types of VC firms. Some have experience in your field, some do not. VC Firms vs. Corporate VCs are a major difference in how they view the world as well. There is even the informal ranking system of “Tiers” that is very loosely made but can help you think about the VCs to determine which are right for you. From the very beginning the idea at Dragos was to IPO the company. Making it to the stock exchange with a public listing of your stock sounds like a good thing from a monetary perspective but it provides something more important to my team. We want to keep Dragos around and really make the world a more secure place for our most important and critical infrastructures. The industrial world is huge and deserves protection. Having an “exit” by getting bought by another company generally doesn’t allow that. Acquisitions of technology companies are not always bad but especially early in the company’s life when they are just really getting their culture and tech and value propositions right, getting adopted into another company can be a death sentence. It’s going to take years to put a real dent into industrial cybersecurity. When you raise venture capital though you have to be able to give your VC’s an exit. They need to be able to exit the company at some point and recoup their investment plus how much their investment has grown. As it turns out, the appetite for how this occurs differs from firm to firm. There is not simply 1 “persona” at any firm but their track record, size of fund, and “tier” can help inform what they’re looking for in how you exit.

As an example, if your VC firm has a small “fund” they’re investing out of ($100M-$200M) and they haven’t had a lot of IPOs before then it should be pretty obvious they are looking for an acquisition as an exit. If they invest $5M in your company early on and you can exit for $200-$300M they get enough return to put a real “dent” in their overall fund size to show returns to their investors (which surprisingly enough are not just rich people but more often college endowments, retirement funds, etc.). But if your investor has done IPOs before and is investing out of a large fund ($500M+ and often times $700M+) then the check sizes they write are going to be larger (they’re going to look to invest more like $15M instead of $5M so they can put their capital to work) and they’re going to need the exit to be larger (which is more likely to be an IPO than an acquisition) to get an exit that actually puts a dent in their overall fund. There’s way more to evaluating VC firms including their networks, who the board member is that will actually sit on your board, and how they enable their investments but figuring out what they need in terms of an exit is a really important part of the equation to make sure it aligns with the company you want to build. An IPO is an “exit” for many of the investors but it’s a growth stage for your company. The founding team and its members shouldn’t be thinking about an exit but should be aware that taking venture capital is more of an investment/loan than it is anything else.

Raise What You Need Not What You Can

If you’re outside of a company you rarely get deep insight into what’s going on in that company. One of the ways we measure companies though, at least when consider venture capital, is how much they raised and at what valuation. At one point I would have thought raising the maximum amount and then not raising again for a long time is a good strategy; turns out that’s very silly. The simple version is easy to understand, if I raise $10M at a $15M pre-money valuation you give up a lot of the company in terms of dilution. If you raise $5M at $15M and then raise another $5M in a year at a $30M valuation you’ve just raised money without giving up as much of your company. It’s never that simple though and you have to figure out how much you can raise and how much you can put to use to get you to the next milestone for your company. If the venture community is hot, then investments might come easy, if it’s not then no matter how good you do the money will “dry up” to some extent. Additionally, you always want to make sure you can hit the goals to get to the next milestone.

As an example, did you raise $30M at $100M pre-money valuation? Congrats that’s awesome. Did you raise $60M at $150M valuation? Looking a bit dicey by comparison. But why? Let’s assume those are C series numbers since they are really big amounts (although in the industrial community a lot of the Series B raises are more like Series C, our $37M raise at a B is more like a C round for comparable IT security companies). If you raise $30M on $100M pre-money valuation then your post money valuation is $130M. To raise a D round on this trajectory you really want to show investors that you can 2-3x their investment, this entices the new investors and shows good health for the company. I.e. at the D round you’d really want a $260-$390M pre-valuation. Ideally you could get there in a 2-3 year period showing rapid growth. But if you raise $60M at a $150M valuation for your D round now you have a $210M post money valuation. Which means you’d ideally want to hit $420M-630M pre-money valuation to show the same level of growth. But the larger the numbers the slower the growth might be and there is a limit in how much you can invest in your company each year to hit a return especially if you’re being compared to the growth of the company that took $30M. I.e. pouring $50M onto a company instead of $30M may not actually give you any extra value depending on market size, customer adoption, sales cycles, and more. So if you raise more than you need just because it’s available you are potentially taking more dilution in the company than you need, you are possibly not doing anything more for your growth rate, and you may make it incredibly difficult or even impossible to raise your next round which means you have to get acquired as you’ll be burning too much to go cash flow positive or raise more. There’s no perfect formula nor only one school of thought, but this was all interesting to me as I learned more about raising capital and watched our own experience. If you’re wondering, we raised the $37M specifically to get us to the next round which we have a target valuation and date for already (I found it useful to always be thinking about your next round and walking backwards into it for the current round).

Terms, Terms, Terms

It would also seem obvious that a company that raises $30M at $100M valuation is a better company than the one that raises $30M at $90M valuation (pre-money on both). However, what most people never get a chance to see are the terms of the investment. The term sheet is the “offer” that an investor makes a company for the terms of the investment that then get ironed out in a much larger package of closing documents. Those aren’t made public but in my opinion are far more important than the valuation of the round. A seemingly bigger and better investment may have horrible terms. As an example, one term that could included in the investment is “participating preferred” or even have a multiple on it. A multiple is idiocy in my opinion and would only make sense to a company in distress and participating preferred in general is a term I refused to ever accept. There are numerous terms you have to watch out for that can flavor the entire trajectory of the company but let’s dive into participating preferred which I think of as one of the worst terms.

Let’s imagine an investor puts in $10M into your company and you add equity for them which gives them 10% of the company. And then your company eventually exits for $200M. In a simple world where that was all the investment you had then the $200M would be split between all the preferred stock holders (the investor) and the common stock (the founders and employees). So $100M would be split according to how much equity everyone has. The investor’s $10M turns into $20M and that’s what they leave with. The rest of the $180 is split between the founders and employees depending on how much they own each. But if the investment was participating preferred at a 1x then the investor would get back their $10M and then get to participate in 10% of the remaining $190M. That means they’d walk away with $29M and the company would split $171M. In reality the investors would actually get more because of interest accrued on the initial $10M. This is very simplistic but even in the simplistic view you can see why this would be problematic. Even if your goal isn’t everyone splitting money and instead is just growing the company, you could potentially have lots of extra money flowing out of the company at an IPO. Simply put, no matter the situation you really don’t want money leaving your company that you could be investing in your company or splitting between the people who were putting in the day-to-day work. In my experience, my goal was a simple term sheet. An adviser of mine told me to demand that the terms of the deal be so simple that they could be written on a napkin. We aimed for that and succeeded in doing it but it is way harder than it sounds.

Boards Are Equally Important and Not Important

This is a weird statement to write because I get a lot of value out of my board as advisers and a sounding board. But previously I would have thought a board had a lot of control over a company. I don’t know why but I just always assumed a lot of decisions get made at the board level. Turns out that’s not the case. The person running the company day-to-day is the CEO. The CEO knows so much more about the company than the board members could possibly know. In relation to venture capital the board wants to know that the company is growing, monitor their investment, and provide input and connections that could benefit the company to help it grow. As an example, one of the ways I used my board is to help me with interviewing key hires especially VP’s because they’ve seen many people and have experience I don’t have on to act as a sounding board. They provide good connections throughout the various industries we work in and also provide some guidance to make sure that I’m growing the business in the right way at the right speed and am building a company that others will want to invest in which will help our team build the type of company that can really impact the industry. But they don’t make a lot of decisions. I assume a lot of people think they do because I even have people outside my company that want to do business with us or want something from us hit up my board members directly assuming they have some significant influence over me; very little makes me more likely to ignore someone.

It turns out my view of how involved boards were was entirely off. They provide help that I ask for, but they don’t make a lot of choices. Salary and compensation packages get approved at the board level (which I find to be useful actually), the annual budget gets approved there, and the rest is just providing them updates and getting advice. You could get bad terms that dictate a board has more say but I’d be concerned about that for a lot of reasons. Opening a new office, choices around the product, hiring and firing, strategy for the company, go-to-market, etc. that’s all on my team internal to the company. The board is interested and active but entirely disconnected from the “decisions” that get made around that. The approval of an annual budget has a high level of impact but it’s something my team and I put together and put in front of them, their input is “that looks normal” “that looks like it’ll get you what you want” “have you considered upping the investment in X?” “I think you’ve over invested in Y”. It’s useful, it’s their approval, but it’s still the work from inside the company and whatever I put in front of them is going to be within the range of what gets approved.

I imagine this is all super obvious to most other people but I was surprised to find how little boards actually do and yet how they remain vitally important. One of the things I’ve really found value in my board is simply having a well experienced team of folks as a sounding board. It really can’t be overstated. As the CEO whether or not its true you constantly feel you’re the person who’s suppose to have all the answers and lead the ship. Sure everyone else actually does all the work, and the decisions are super collaborative in the company, but that feeling remains. At the board level though it feels far more like a discussion with less stress. I’ve found my board to be my “safe space” to just think about stuff out loud without repercussion and have folks more experienced than myself in areas that are different but important provide thoughts or validation. At the end of the day it’s not lost on me that so many in the community, so many customers, and all the families of all my employees have a very vested interest in me not screwing up my part. The act of “yea Rob that makes total sense” or “I really think you should dig into that more” provides far more comfort than I can put into words.

Every Company is Different and Your Journey Will Be Filled with Friends and at Times Lonely

These are my thoughts. These are some of my observations. Yours are bound to be different but I hope it’s useful to see one person’s thoughts on all this. I’ve never felt more excited and more proud than leading this team at Dragos and I’ve also never felt so very alone in my career. That sounds weird but let me clarify. Being the CEO and founder of an industrial cybersecurity company that has received venture capital, now above $48M, and is forging into a market that’s largely been undefined…is a lonely and exciting endeavor. Industry analysts and financial folks have tons of opinions, but none of them have actually been down this journey. I’m empathetic to the various industry analysts trying to define the market or provide hot takes on where the market will or won’t go; and it’s useful to the community because buyers and community members cannot just listen to the obviously biased companies leading the way, but those same industry analysts and financial folks have far less experience in this topic. The number of peers I have for the very specific thing I’m doing is tiny. And to be honest it’s not like the founders of competitive companies really get together and have open discussions other than “hey thanks for helping also build the market, good luck to you, I hope I utterly destroy you on the battlefield, but I respect you” lol. I often look at the industrial cybersecurity community and industry practitioners that I respect pontificating on the market, who will be winners, what that means, what products will be needed, etc. and I struggle not to roll my eyes a bit as they have never built companies or products in that space. I have to force myself to realize that what they’re doing is still an important part of the equation and very useful to folks. But I do also understand that they are also extremely biased, operating with completely different experience, and sometimes are advising on things that they have far less experience in than people realize especially compared to the companies that live it day to day and are on that unique journey. I’ve had heated arguments with people I respect and call friends simply because their experience guides them to view things so differently than the experience I have. And yet, that’s ok. Actually it’s really useful for everyone. And taking a step back to just keep that in mind is vital to not becoming arrogant or being misguided yourself. To solve a problem as complex as any that guides you down such a unique path is going to require thought process and input from everyone, welcoming it is important especially when it seems counter to your own experiences.

Why say all this? To stress the point that in this community, this industry, this time, etc. things are so specific that any lessons learned I have may not really apply to you in your journey. And it’s so different that even the founders that have built a company before may not have experiences that help or relate to you; I found the “entrepreneur” networks to be largely disappointing with experiences that didn’t translate to the challenges my team is facing. I found the folks in my company and their insights as well as the experiences and insights of our customers to be far more useful to me. But maybe those networks work for you. I don’t think I’m special, but I think this journey is unique. Yours is too. Hopefully my thoughts are useful but don’t take them as the only thoughts nor any of my takes as the only way to view those topics (hell maybe I’m wrong on plenty and will have different views in a few years). Instead, I’d focus more on how my opinions have changed and be open minded to have yours change as well.

What It’s Like to Testify to the U.S. Senate

July 21, 2018

One of my SANS students challenged me recently that I haven’t posted on my blog in awhile; I realized I hadn’t and checked today and found that it’s been almost a year since my last post. So first of all, apologies on the delay. I’ve been busy (Dragos, Inc. has been expanding rapidly and we’re about to reach 60 employees, my wife and I welcomed our first born into the world, and industrial cyber attacks haven’t exactly slowed down) however, no excuses, I should post and share my experiences more and appreciate the folks who take time to read. This post comes as a request to some attendees at one of my SANS @Night talks where they asked me what the experience was like and how it happened. I feel that it’s really not transparent how someone gets to testify in front of the Senate and what all goes into it so I hope this post is useful to illuminate the experience a bit. The recorded testimony can be found here.

How I Got Invited

I imagine most folks’ path to testifying in front of the Senate is different. I also assume for most they are seen so widely outside their own community as an expert that it’s a natural thing. My experience was a bit different. One of the things that motivates me a lot is educating people especially on topics of intelligence and industrial security. The intersection of those fields with policy is pretty clear and unavoidable. I’ve also always been a bit outspoken about individuals speaking on and influencing technical discussions without having technical experience. This has led me to getting involved in educating policy folks on topics of industrial cybersecurity, especially critical infrastructure. An ex-Senate staffer I ran across asked if I’d be interested in offering some sessions for Senate and House staffers on the topic of electric grid cybersecurity. Over the course of about a year I would routinely go to DC and spend a few hours talking about how the power grid works, what cyber threats actually do, and demystifying a lot of the hype (yes our infrastructure operators have made fairly resilient infrastructure, no cyber threats aren’t magic or all powerful, and no Ted Koppel’s book is not accurate in the real risk). In addition, I got invited to speak on a panel on cyber threats to the grid at the Siebel Scholars’ conference with a panel made up of Richard Clarke, Kevin Mandia, and Liam O’Murchu (which was moderated by Ted Koppel…super nice guy, he just didn’t write a technically accurate book). While at the dinner another Senate staffer liked what I had to say and offered to keep in touch, I exchanged cards and that was that.

Eventually it was the staffer from the Siebel event that emailed me out of the blue one day asking if I’d be willing to testify in front of the Senate’s Committee on Energy and Natural Resources on cyber threats to U.S. infrastructure. Because I had spent time with other staffers in the Senate the recommendation went over well as a few of the other staffers knew me and agreed I’d be good to have on the panel. What I also really enjoyed was finding out that both Republican and Democrat staffers had recommended me; I joked with them that they all thought I was on their “side” and that they couldn’t figure out where my politics actually fell (to be honest I’m not political, just opinionated) but in truth they were all just on the same page of wanting to protect infrastructure and didn’t care about the politics around it.

Preparing for the Testimony

In preparation for the testimony I used YouTube to watch other testimony to the committee on the subject of cybersecurity and determine what type of questions the Senators had previously. I also pulled the written testimony of Thomas Rid and Kevin Mandia, two individuals I respect in the field that have also testified, and looked at their style of writing and what they chose to highlight. I then prepared my written testimony on the key points I wanted to get across. For my testimony the three key points were essentially:

  • that industrial cyber threat landscape is largely unknown so we cannot just adopt IT cybersecurity frameworks/regulations/best-practices into industrial networks because they were built off of risk observed against cyber threats targeting the Enterprise networks and how to handle them
  • that regulation has served a purpose (such as NERC CIP for the power grid) and helped the industry but that we have exhausted reasonable regulations and the industry is struggling to innovate or do real defense in the face of new regulations that come out every few years
  • that the Department of Energy’s Office of Cybersecurity, Energy Security, and Emergency Response (CESER) that was being formed at the time has a unique opportunity to help work with private sector and form partnerships not only with asset owners and operators but also the private sector security community; many of the government’s actions right now in reaction to fear of cyber threats are quickly pushing them into being competitive with private sector firms that are actually better equipped to deal with certain situations and we should all attempt to play to our strengths instead of competing

It was interesting to try to distill everything I ever wanted to say down to effectively seven pages (found here) and to do so in a non-technical audience of policy makers. Some of my points, such as highlighting that technologies such as artificial intelligence aren’t a silver bullet for security, were put in to directly counter some of the things the Senators had been told in previous testimony that I felt mislead them. I then sent out my testimony to people I knew and trusted around the community who helped me steer clear of any wording or language that might come off wrong. As an example, I was mentioning “NERC” “DOE” and “energy companies” so I sent my written testimony to people I trusted at NERC, DOE, and numerous energy companies to make sure I was representing them the best I could. I.e. don’t get in front of the Senate on TV broadcast across the country and talk out of turn about matters that impact others.

What I didn’t expect was that the list of people testifying goes public well before the hearing so people reach out to you and the Senators about you. I had various interests groups email me trying to get their points included in my testimony; almost exclusively it was groups I’d include in “the crazies” category. The two most surprising was a group that wanted me to bash NERC and tell the Senators how the grid was so fragile and needed new oversight and something about EMPs (please stop with the EMP stuff) and then a member of the industrial community emailed some Senators directly and, from what was explained to me later by a staffer, essentially petitioned them not to listen to whatever was going to be said about cyber threats and instead to realize that level 0 sensor cybersecurity is the only relevant topic. I consider this person a friend and long-time community member and do not think he was being malicious but to be honest I found that to pretty rude because it, intentionally or not, was a distraction and could have subverted the points of the people that were presenting on the topic. (I’ve avoided the level 0 debate in the community because it becomes almost fanatical but in essence my position is that all the risk we’re seeing due to cyber threats does not start at a sensor level and if you do not learn what the adversaries are doing, how they’re doing it, and counter them before it ever gets to that level you will lose out). I only include this discussion of unexpected components to testifying to note to everyone that Senators and staffers get all sorts of things sent to them all the time. You really need to be active in the process to counterbalance the viewpoints but also ensure you do so in a manner that is conducive to the overall narrative.

Delivering the Testimony

For the verbal testimony itself you cannot read from your seven page written testimony. The Senators have a very limited amount of time to hear testimony and ask questions and they want every second to answer as many questions as possible. They are extremely busy and this is genuinely the time they’ve dedicated to focus on this specific topic. So you get five minutes to summarize your points, they’ve read your written testimony in advance, and they get a few minutes to ask you questions back. Who they ask questions to is entirely dependent on the Senators and who they want to talk about and on what subjects.

I have to admit I was actually pretty nervous delivering my written testimony. This was a recorded event and not even live. I’ve been on live news before on channels such as CNN and Fox in front of millions of people and felt more confident. I speak at a lot of venues. I’ve keynoted numerous conferences. And none of it compared. This was nerve wracking and I had to steady my hand by holding the paper in front of me. I also made a deliberate choice to speak fast and get through as much material as possible in the five minutes, I knew the Senators had already read the testimony (or at least their staffers did) and that they had prepared questions, so my verbal testimony was intended more for those that watched the hearing after the fact. When it got time for the QA session I was back in my element and felt confident. But that verbal testimony was the most nervous I’ve been in awhile.

I found the most important part of the verbal testimony was to try to answer the Senator’s questions as quickly and coherently as possible. I also found that all of their questions were extremely reasonable. Even those that appeared to have an agenda actually only were getting clarification on points they had previously been told by others. And to be honest, more lobbyists and self-interested people try to speak to Senators than folks doing the mission. What that meant to me is a clear understanding that some of the Senator’s views in topics such as just disconnecting the grid from the internet were based off of other people and not some agenda of their own. To be honest I found that the Senators all cared deeply about the topic and were highly professional. The country definitely feels divided these days but in the hearing I was in, at least, everyone there seemed to sincerely care about infrastructure security with no partisan political games tied to it.

At the end of the testimony I also felt that I won some major kudos points at home. My son was being born that day and my wife and I both felt that I should still go and testify; she is an immigrant from Holland and I served my country in the military so we had this extra sense that this was something patriotic that had to be done; so she gave me the go-ahead (luckily I didn’t miss my son’s birth though). The Senate was apparently informed about my son’s imminent birth and just how awesome my wife was being by letting me go…so on official record, at around 2:06:17…Senator Murkowski and the committee congratulated my wife and thanked her, and I got on official record “she’s awesome” so in essence…I win.

Final Thoughts

It’s really difficult these days to talk about politics with how decisive it gets especially in the media. We have things that make us very passionate and strike at the core of who we are and what we believe in. I’m not political at all and even I’ve been extremely bothered by some of the things I’ve seen such as the consistent attack on the U.S. Intelligence Community. However, testifying to the U.S. Senate was the most lifting political experience I’ve ever had. I swelled with pride in a way not easy to describe and felt the joy in this grand experiment we call democracy in a way I found unexpected and amazing. I would encourage the technical practitioner community to engage your elected officials and their staffers. Seek to educate others. And if you get the chance to testify to take the opportunity and try your best to represent the community well in doing so; it was an amazing experience that I highly recommend.

 

 

 

 

 

Russian Election Meddling, GRIZZLYSTEPPE, and Bananas

August 17, 2017

It’s been awhile since I’ve been able to post to my blog (as it turns out doing a Series A raise for my company Dragos has been time consuming so I apologize for the absence in writing).  But it is fitting that my first blog post in awhile has something to do with the GRIZZLYSTEPPE report. I almost got sucked back into writing when I saw the Defense Intelligence Agency (DIA) tweet out the Norse cyber attack map.

Matt jumped on it pretty quickly though which was great.

I tried to attempt to fill the person in running the account just in case they didn’t understand why folks were less than excited about their presentation.

But in their responses to me it seemed they didn’t fully understand. They articulated that they use unclassified data for the conference but use classified data at work. Of course the problem wasn’t the data (even though it’s not just unclassified but completely bad/fake data) it’s the idea that a cyber attack map aka “pew pew map” is not a good way to communicate to any audience as its simply a marketing ploy. However, it’s not worth a full blog post so I’ll just instead request everyone to do their homework (should only be a quick Google search) on why pew pew maps are stupid and everyone serious in the discussion should stop using them.

On To the Main Discussion

But on to the main topic. What does Russian election meddling, the GRIZZLYSTEPPE report, and bananas all have in common? Absolutely nothing. Each are individually completely unrelated to each other and people should stop putting any of them together as it ultimately just makes people look silly (to be fair no one’s associated bananas with the election interference yet but it might be a better correlation than the GRIZZLYSTEPPE report).

This discussion was all spawned by an article that the New York Times released on August 16th, 2017 titled “In Ukraine, a Malware Expert Who Could Blow the Whistle on Russian Hacking“. Spoiler alert: he can’t. I went on a bit of a Twitter rant to explain why the article wasn’t good, it can be found here, but I felt it was a complex and an important enough topic to cover in a blog.

The NYT piece posits that a hacker known by his alias “Profexer” was responsible for writing the P.A.S. tool and is now a witness for the FBI after coming forward to Ukrainian police. The P.A.S. tool, the article puts forward, was leveraged by Russia’s intelligence services without his knowledge (not sure how he can be a “witness” then but I digress). The authors of the article previously explicitly stated P.A.S. was used in the break-in of the Democratic National Committee  (DNC) but they had to issue a correction to that (to their credit, folks from NYT reached out to me after I critiqued it on Twitter to try to get the story correct after it was published; I asked for the correction as I’m sure others did but in reading the updated article the correction doesn’t actually address the larger issues so I wanted to cover them here in the blog).

 

Figure 1: Correction Related to P.A.S. and the DNC

Where did they get this assertion that P.A.S. was used in the DNC breach? By tying the GRIZZLYSTEPPE report (which does note that P.A.S. has been used by Russian security service members before) to the DNC breach. The GRIZZLYSTEPPE report has nothing to do with the DNC breach though and was a collection of technical indicators the government compiled from multiple agencies all working different Russian related threat groups. The threat group that compromised the DNC was Russian but not all Russian groups broke into the DNC. The GRIZZLYSTEPPE report was also highly criticized for its lack of accuracy and lack of a clear message and purpose. I covered it here on my blog but that was also picked up by numerous journalists and covered elsewhere. In other words, there’s no excuse for not knowing how widely criticized the GRIZZLYSTEPPE report was before citing it as good evidence in a NYT piece. Interestingly, the journalists didn’t even link to the “Enhanced Analysis” version of the GRIZZLYSTEPPE report which was published afterwards (and is actually much better) as a response to the critiques of the first one.

A major issue exists though with the correction to the NYT article. It changes the entire point of the story. If Profexer isn’t actually a “witness” to the case because P.A.S. wasn’t used in DNC then what’s the message the journalists are trying to get across? Someone who wasn’t working with the Russians, developed a tool that the Russians didn’t use in the DNC case, and didn’t have any insight into any of the Russian threat groups or campaigns cannot be a good witness.

Even after the correction though the journalists draw the readers attention to the breach early and often to continue to reinforce that this gives new insight into that case.

Figure 2: Snippet from NYT Article Referencing DNC Breach and Profexer

And again the journalists explicitly state that Profexer is somehow a witness to what occurred and reference him back again to the election hacking.

Figure 3: Snippet from NYT Article Claiming Profexer is a Witness

The article goes on to note how this changes our thoughts on the Russian groups (APT28 / APT29 or COZYBEAR / FANCYBEAR) and how they operate; the journalists state that using publicly available tools or outsourcing tool development to cyber criminals is against the modus operandi (MO) of the Russian security services. I do not know where the journalists get this claim but they do not source it; I disagree with the claim but I’ll note the burden of proof here is on them with regards to showing where they’re claiming the previous MO and I’ll simply state that there have been numerous publications and reports showcasing Russian threat groups including the security services using other groups and people’s tools and exploits. This isn’t new information and it’s fairly common for many threat groups to operate in this way.

The attribution on APT28 and APT29 is some of the most solid attribution the community has ever done. Numerous cybersecurity firms have covered this group including FireEye, CrowdStrike, Kaspersky, TrendMicro, and F-Secure but we’ve also had government attribution before by the German intelligence services on a breach into their government that pre-dates the DNC breach. A cursory look will reveal that organizations have been tracking this Russian threat group for about a decade now. Yet none of the people who’ve actually covered these groups were cited in the NYT article. Instead the journalists chose to cite Jeffrey Carr and his quote is confusing to most readers because he is trying to detract from the attribution where he states: “there is not now and never has been a single piece of technical evidence produced that connects the malware used in the D.N.C. attack to the G.R.U., F.S.B. or any agency of the Russian government.” It’s almost as the journalists just wanted a contrarian view to look balanced but what an odd selection if not just set up their witness to be even more important.

I want to be very clear on my next critique: I actually don’t think Jeffrey Carr is a bad person. I know he ruffles the feathers of a lot of folks in the community (mine included at times) but on the two occasions I’ve met him in person he’s been an absolutely nice person to me and was civil and well articulated. That being said, he is not an expert on attribution, not an expert on these groups, nor has any reason to be cited in conjunction with them. He’s often widely criticized in the community when he tries to do attribution and it’s often painfully full of 101 intelligence analysis failures. The NYT didn’t do him any favors by including him in this article and seriously detracted from the idea that they understood enough about this topic to cover it. Simply stated: “cyber” is not an expertise, if you are covering a niche topic like attribution or a further niche topic like Russian group attribution you need to use folks who have experience in that subject matter.

Please Stop Arguing About Attribution Without Expertise In It

This is a bit of a big request but it’d be very useful if people stop taking a stance on why attribution is difficult or not and whether or not attribution is right or not if they have never had experience in doing attribution. This is important because the journalists in this article seem to want to help bolster the case against the Russian intelligence services yet make it more confusing. At one point they try to set up their witness as some new smoking gun to be added to the case as a push back to people like President Trump.

Figure 4: Snippet from NYT Article Setting Up the Importance of the “Witness”

Attribution is not about having a smoking gun. Attribution is a good example of doing true intelligence analysis; there are no certainties and you only can come to an assessment such as low, moderate, or high confidence. Almost every single piece of data put forward in that assessment can and should have counters to it. Very reasonable counters as well. It’s why when anyone arguing for attribution argues a single piece of evidence they almost always lose the argument or look silly. It’s simply very rarely about one piece of evidence and is instead the analysis over the total data set. The attribution levied towards Russia for meddling in the U.S. elections is solid. The reason President Trump and others don’t want to accept that has nothing to do with the fact that there hasn’t been a witness or a “single piece of technical evidence produced that connects the malware used in the D.N.C. attack to the G.R.U.” it is because they do not want to accept the conclusion or the reality it presents. There’s nothing that’s going to change this. I’m convinced that if President Putin came out and said “yea it was us” we’d have critics coming forward saying how it’s a false flag operation and it’s actually not true.

But what’s the problem with people arguing these points? It detracts from the already solid assessment. It’s similar to when the FBI wanted to release IP addresses and some technical indicators during the Sony hack to talk about how they knew it was North Korea. I critiqued that approach when it happened here. The basis of my argument was that the FBI’s attribution to North Korea was likely correct but their presentation of evidence as proof was highly misleading. Obviously the FBI didn’t just use those technical indicators to do the attribution, so how could anyone be expected to look at those and be convinced?  And rightfully so people came out and argued against those technical indicators noting they could easily be wrong and that adversaries of any origin could have leveraged the IP addresses for their operations. And the critiques were correct. The technical evidence in isolation was not good. The totality of the data set though was very sound and the analysis on top of it though were very sound.

I often think of this like climate change arguments. You can have 100 scientists with a career in climate studies posit forth an assessment and then two people with absolutely no experience argue on the subject. One of the people arguing for the climate scientists’ position could grab out a single data point to argue and now the person arguing against that first person is arguing against an uninformed opinion on a single data point instead of the combined analysis and work of the scientists. The two people arguing both leave understandably feeling like they won the argument: the original assessment by the scientists was likely right but the person arguing against the data point was also probably right about their argument against that data point. The only people who lost in this debate were the scientists who weren’t involved in the argument and who’s research wasn’t properly presented.

Closing Thoughts

I never like to just rant about things, I try to use these opportunities as things to learn from. All of this is actually extremely relevant to my SANS FOR578 – Cyber Threat Intelligence course so a lot of times I write these blog posts and reference them in class. So with that theme in mind here’s the things I want you to extract from this blog as learning moments (to my students, to the journalists, and to whomever else finds it valuable).

  • If you are doing research/writing on niche topics please find people with expertise in that niche topic (Jeffrey Carr is not an expert on attribution)
  • If you are going to posit that the entire public understanding of a nation-state group’s MO has changed because a single piece of evidence you’re likely wrong (do more homework)
  • If you are going to posit that there is a witness that can change the narrative about a case please talk to people familiar on the case (determine if that type of evidence is even important)
  • If you are going to write on a topic that is highly controversial research the previous controversy first (GRIZZLYSTEPPE was entirely unrelated to the DNC case)
  • Attribution is not done with single pieces of evidence or a smoking gun it is done as analysis on complex data sets most of which is not even technical (attribution is hard but doable)
  • The most interesting data for attribution isn’t highly classified but instead just hard work/analysis on complex scenarios (classification standards don’t imply accuracy or relevancy)
  • Just because someone’s code was used by an adversary does not imply the author knows anything about how it was used or by whom (the threat is the human not the malware)
  • Stop using pew pew maps (seriously just stop; it makes you look like an idiot)

 

Marketing ICS Vulnerabilities and POC Malware – You’re Doing it Wrong

April 30, 2017

There’s been two cases recently of industrial control system (ICS) security firms identifying vulnerabilities and also creating proof of concept (POC) malware for those vulnerabilities. This has bothered me and I want to explore the topic in this blog post; I do not pretend there is a right or wrong answer here but I recognize by even writing the post I am passing judgement on the actions and I’m ok with that. I don’t agree with the actions and in the interest of a more public discussion below is my rationale.

Background:

At the beginning of April 2017 CRITIFENCE an ICS security firm published an article on Security Affairs titled “ClearEnergy ransomware aim to destroy process automation logics in critical infrastructure, SCADA, and industrial control systems.” There’s a good overview of the story here that details how it ended up being a media stunt to highlight vulnerabilities the company found. The TL;DR version of the story is that the firm wanted to highlight the vulnerabilities they found in some Schneider Electric equipment that they dubbed ClearEnergy so they built their own POC malware for it that they also dubbed ClearEnergy. But they published an article on it leaving out the fact that it was POC malware. Or in other words, they led people to believe that this was in-the-wild (real and impacting organizations) malware. I don’t feel there was any malice by the company, as soon as the article was published I reached out to the CTO of CRITIFENCE and he was very polite and responded that he’d edit the article quickly. I wanted to write a blog calling out the behavior and what I didn’t like about it as a learning moment for everyone but the CTO was so professional and quick in his response that I decided against it. However, after seeing a second instance of this type of activity I decided a blog post was in order for a larger community discussion.

On April 27th, 2017 Security Week published an article titled “New SCADA Flaws Allow Ransomware, Other Attacks” based on a presentation by ICS security firm Applied Risk at SecurityWeek’s 2017 Signapore ICS Cyber Security Conference. The talk, and the article, highlighted ICS ransomware that the firm dubbed “Scythe” that targets “SCADA devices.” Applied Risk noted that the attack can take advantage of a firmware validation bypass vulnerability and lock out folks’ ability to update to new firmware. The firm did acknowledge in their presentation though and in the article that this too was POC malware.

 

 

Scythe

Figure: Image from Applied Risk’s POC Malware

Why None of this is Good (In My Opinion):

In my opinion both of these firms have been irresponsible in a couple of ways.

First, CRITIFENCE obviously messed up by not telling anyone that ClearEnergy was POC malware. In an effort to promote their discovery of vulnerabilities they were quick to write an article and publish it and that absolutely contributed to hype and fear. Hype around these types of issues ultimately leads to the ICS community not listening to or trusting the security community (honestly with good reason). However, what CRITIFENCE did do that I liked (besides being responsive which is a major plus) was work through a vulnerability disclosure process that led to proper discussion by the vendor as well as an advisory through the U.S.’ ICS-CERT. In contrast, Applied Risk did not do that so far as I can tell. I do not know what all Applied Risk is doing about the vulnerabilities but they said they contacted the vendors and two of the vendors (according to the SecurityWeek article) acknowledged that the vulnerability is important but difficult to fix. The difference with the Applied Risk vulnerabilities is that the community is left unaware of what devices are impacted, the vendors haven’t been able to address the issues yet, and there are no advisories to the larger community through any appropriate channel. Ultimately, this leaves the ICS community in a very bad spot.

Second, CRITIFENCE and Applied Risk are both making a marketing spectacle out of the vulnerabilities and POC malware. Now, this point is my opinion and not necessarily a larger community best-practice, but I absolutely despise seeing folks name their vulnerabilities or naming POC malware. It comes off as a pure marketing stunt. Vulnerabilities in ICS are not uncommon and there’s good research to be done. Sometimes, the things the infosec community sees as vulnerabilities may have been designed that way on purpose to allow things like firmware updates and password resets for operators who needed to get access to sensitive equipment in time-sensitive scenarios. I’m not saying we can’t do better – but it’s not like the engineering community is stupid (far from it) and highlighting vulnerabilities as marketing stunts can often have unintended consequence including the vendors not wanting to work with researchers or disclose vulnerabilities. There’s no incentive for ICS vendors to work with firms who are going to use issues in their products for marketing for a firm’s security product.

Third, vulnerability disclosure can absolutely have the impact of adversaries learning how to attack devices in ways they did not know previously. I do not advocate for security through obscurity but there is value in following a strict vulnerability disclosure policy even in normal IT environments because this has been an issue for decades. In ICS environments, it can be upwards of 2-3 years for folks to be able to get a patch and apply it after a vulnerability comes out. That is not due to ignorance of the issue or lack of concern for the problem but due to operations constraints in various industries. So in essence, the adversaries get informed about how to do something they previously didn’t know about while system owners can’t accurately address the issues. This makes vulnerability disclosure in the ICS community a very sensitive topic to handle with care. Yelling out to the world “this is the vulnerability and oh by the way here’s exactly how you should leverage it and we even created some fake malware to highlight the value to you as an attacker and what you can gain” is just levels of ridiculousness. It’s why you’ll never see my firm Dragos or myself or anyone on my team finding and disclosing new vulnerabilities in ICS devices to the public. If we ever find anything it’ll only be worked through the appropriate channels and quietly distributed to the right people – not in media sites and conference presentations. I’m not a huge fan of disclosing vulnerabilities at conferences and in media in general but I do want to acknowledge that it can be done correctly and I have seen a few firms (Talos, DigitalBond, and IOActive come to mind) do it very well. As an example, Eireann Leverett and Colin Cassidy found vulnerabilities in industrial Ethernet switches and worked closely with the vendors to address them. After working through a very intensive process they wanted to do a series of conference presentations about them to highlight the issues. They invited me to take part to show what could be done from a defense perspective. So I stayed out of the “here’s the vulnerabilities” and instead focused on “these exists so what can defenders do besides just patch.” I took part in that research because the work was so important and Eireann and Colin were so professional in how they went about it. It was a thrill to use the entire process as a learning opportunity to the larger community. Highlighting vulnerabilities and creating POC malware for something that doesn’t even have an advisory or where the vendor hasn’t made patches yet just isn’t appropriate.

Closing Thoughts:

There is a lot of research to be done into ICS and how to address the vulnerabilities that these devices have. Vendors must get better at following best-practices for developing new equipment, software, and networking protocols. And there are good case-studies of what to do and how to carry yourself in the ICS security researcher community (Adam Crain and Chris Sistrunk’s research into DNP3 and all the things that it led to is a fantastic example of doing things correctly to address serious issues). But the focus on turning vulnerabilities into marketing material, discussing vulnerabilities in media and at conferences before vendors have addressed them and before the community can get an advisory through proper channels, and creating/marketing POC malware to draw attention to your vulnerabilities is simply, in my opinion, irresponsible.

Try these practices instead:

  • Disclose vulnerabilities to the impacted vendors and work with them to address them
    • If they decide that they will not address the issue or do not see the problem talk to impacted asset owners/operators to ensure what you see as a vulnerability is an issue that will introduce risk to the community and use appropriate channels such as the ICS-CERT to push the vendor or develop defensive/detection signatures and bring it to the community’s attention; sometimes you’re left without a lot of options but make sure you’ve exhausted the good options first
  • After the advisory is available (for some time you feel comfortable with) if you or your firm would like to highlight the vulnerabilities at a conference or in the media that’s your choice
    • I would encourage focusing the discussion on what people can do besides just patch such as how to detect attacks that might leverage the vulnerabilities
  • Avoid naming your vulnerabilities, there’s already a whole official process for cataloging vulnerabilities
  • (In my opinion) do not make POC malware showing adversaries what they can do and why they should do it (the argument “the adversaries already know” is wrong in most cases)
  • If you decide to make POC malware anyway at least avoid naming it and marketing it (comes off as an extremely dirty marketing approach)
  • Avoid hyping up the impact (talking about power grids coming down and terrorist attacks in the same article is just a ridiculous attempt to illicit fear and attention)

In my experience, ICS vendors are difficult to work with at times because they have other priorities too but they care and want to do the right thing. If you are persistent you can move the community forward. But the vendors of the equipment are not the enemy and they will absolutely blacklist researchers, firms, and entire groups of folks for doing things that are adverse to their business instead of working with them. Research is important, and if you want to go down the route of researching and disclosing vulnerabilities there’s value there and proper ways to do it. If you’re interested in vulnerability disclosure best practices in the larger community check out Katie Moussouris who is a leading authority on bug bounty and vulnerability disclosure programs. But please, stop naming your vulnerabilities, building marketing campaigns around them, and creating fake malware because you don’t think you’re getting enough attention already.

Analytical Leaps and Wild Speculation in Recent Reports of Industrial Cyber Attacks

December 31, 2016

“Judgement is what analysts use to fill gaps in their knowledge. It entails going beyond the available information and is the principal means of coping with uncertainty. It always involves an analytical leap, from the known into the uncertain.”

– Chapter 4, Psychology of Intelligence Analysis, Richards J. Heuer.

 

Analytical leaps, as Richards J. Heuer said in his must-read book Psychology of Intelligence Analysis, are part of the process for analysts. Sometimes though these analytical leaps can be dangerous, especially when they are biased, misinformed, presented in a misleading way, or otherwise just not made using sound analytical processes. Analytical leaps should be backed by evidence or at a minimum should include evidence leading up to the analytical leap. Unfortunately, as multiple analytical leaps are made in series it can lead to entirely wrong conclusions and wild speculation. There have been three interesting stories relating to industrial attacks this December as we try to close out 2016 that are worth exploring in this topic. It is my hope that looking at these three cases will help everyone be a bit more critical of information before alarmism sets in.

The three cases that will be explored are:

  • IBM Managed Services’ claim of “Attacks Targeting Industrial Control Systems (ICS) Up 110%”
  • CyberX’s claim that “New Killdisk Malware Brings Ransomware Into Industrial Domain”
  • The Washington Post’s claim that “Russian Operation Hacked a Vermont Utility, Showing Risk to U.S. Electrical Grid Security, officials say”

 

“Attacks Targeting Industrial Control Systems (ICS) Up 110%”

I’m always skeptical of metrics that have no immediately present quantification. As an example, the IBM Managed Security Services posted an article stating that “attacks targeting industrial control systems increased over 110 percent in 2016 over last year’s numbers as of Nov. 30.” But there is no data in the article to quantify what that means. Is 110% increase an increase from 10 attacks to 21 attacks? Or is it 100 attacks increased to 210 attacks?

The only way to understand what that percentage means is to leave this report and go download the IBM report from last year and read through it (never make your reader jump through extra hoops to get information that is your headline). In their 2015 report IBM states that there were around 1,300 attacks in 2015 (Figure 1). This would mean that in 2016 IBM is reporting they saw around 2,700 ICS attacks.

figure1

Figure 1: Figure from IBM’s 2015 Report on ICS Attacks

 

However, there are a few questions that linger. First, this is a considerable jump from what they were tracking previously and from their 2014 metrics. IBM states that the “spike in ICS traffic was related to SCADA brute-force attacks, which use automation to guess default or weak passwords.” This is an analytical leap that they make based on what they’ve observed. But, it would be nice to know if anything else has changed as well. Did they bring up more sensors, have more customers, increase staffing, etc. as the stated reason for the increase would not alone be responsible.

Second, how is IBM defining an attack. Attacks in industrial contexts have very specific meaning – an attempt to brute-force a password simply wouldn’t qualify. They also note that a pentesting tool on GitHub was released in Jan 2016 that could be used against the ICS protocol Modbus. IBM states that the increase in metrics was likely related to this tools’ release. It’s speculation though as they do not give any evidence to support their claim. However, it leads to my next point.

Third, is this customer data or is this honeypot data? If it’s customer data is it from the ICS or simply the business networks of industrial companies? And if it’s honeypot data it would be good to separate that data out as it’s often been abused to misreport “SCADA attack” metrics. From looking at the discussion of brute-force logins and a pentesting tool for a serial protocol released on GitHub, my speculation is that this is referring mostly to honeypot data. Honeypots can be useful but must be used in specific ways when discussing industrial environments and should not be lumped into “attack” data from customer networks.

The article also makes another analytical leap when it states “The U.S. was also the largest target of ICS-based attacks in 2016, primarily because, once again, it has a larger ICS presence than any other country at this time.” The leap does not seem informed by anything other than the hypothesis that the US has more ICS. Also, again there is no quantification. As an example, where is this claim coming from, how much larger is the ICS presence than other countries, and are the quantity of attacks proportional to the US ICS footprint when compared to other nations’ quantity of industrial systems? I would again speculate that what they are observing has far more to do with where they are collecting data (how many sensors do they have in the US compared to China as an example).

In closing out the article IBM cites three “notable recent ICS attacks.” The three case studies chosen were the SFG malware that targeted an energy company, the New York dam, and the Ukrainian power outage. While the Ukrainian power outage is good to highlight (although they don’t actually highlight the ICS portion of the attack), the other two cases are poor choices. As an example, the SFG malware targeting an energy company is something that was already debunked publicly and would have been easy to find prior to creating this article. The New York dam was also something that was largely hyped by media and was publicly downplayed as well. More worrisome is that the way IBM framed the New York dam “attack” is incorrect. They state: “attackers compromised the dam’s command and control system in 2013 using a cellular modem.” Except, it wasn’t the dam’s command and control system it was a single read-only human machine interface (HMI) watching the water level of the dam. The dam had a manual control system (i.e. you had to crank it to open it).

Or more simply put: the IBM team is likely doing great work and likely has people who understand ICS…you just wouldn’t get that impression from reading this article. The information is largely inaccurate, there is no quantification to their numbers, and their analytical leaps are unsupported with some obvious lingering questions as to the source of the data.

 

“New Killdisk Malware Brings Ransomware Into Industrial Domain”

CyberX released a blog noting that they have “uncovered new evidence that the KillDisk disk-wiping malware previously used in the cyberattacks against the Ukrainian power grid has now evolved into ransomware.” This is a cool find by the CyberX team but they don’t release digital hashes or any technical details that could be used to help validate the find. However, the find isn’t actually new (I’m a bit confused as to why CyberX states they uncovered this new evidence when they cite in their blog an ESET article with the same discovery from weeks earlier. I imagine they found an additional strain but they don’t clarify that). ESET had disclosed the new variant of KillDisk being used by a group they are calling the TeleBots gang and noted they found it being used against financial networks in Ukraine. So, where’s the industrial link? Well, there is none.

CyberX’s blog never details how they are making the analytical leap from “KillDisk now has a ransomware functionality” to “and it’s targeting industrial sites.” Instead, it appears the entire basis for their hypothesis is that Sandworm previously used KillDisk in the Ukraine ICS attack in 2015. While this is true, the Sandworm team has never just targeted one industry. iSight and others have long reported that the Sandworm team has targeted telecoms, financial networks, NATO sites, military personnel, and other non-industrial related targets. But it’s also not known for sure that this is still the Sandworm team. The CyberX blog does not state how they are linking Sandworm’s attacks on Ukraine to the TeleBots usage of ransomware. Instead they just cite ESET’s assessment that the teams are linked. But ESET even stated they aren’t sure and it’s just an assessment based off of observed similarities.

Or more simply put: CyberX put out a blog saying they uncovered new evidence that KillDisk had evolved into ransomware although they cite ESET’s discovery of this new evidence from weeks prior with no other evidence presented. They then make the claim that the TeleBots gang, the one using the ransomware, evolved from Sandworm but they offer no evidence and instead again just cite ESET’s assessment. They offer absolutely no evidence that this ransomware Killdisk variant has targeted any industrial sites. The logic seems to be “Sandworm did Ukraine, KillDisk was in Ukraine, Sandworm is TeleBots gang, TeleBots modified Killdisk to be ransomware, therefore they are going to target industrial sites.” When doing analysis always be aware of Occam’s razor and do not make too many assumptions to try to force a hypothesis to be true. There could be evidence of ransomware targeting industrial sites, it does make sense that they would eventually. But no evidence is offered in this article and both the title and thesis of the blog are completely unfounded as presented.

 

“Russian Operation Hacked a Vermont Utility, Showing Risk to U.S. Electrical Grid Security, officials say”

This story is more interesting than the others but too early to really know much. The only thing known at this point is that the media is already overreacting. The Washington Post put out an article on a Vermont utility getting hacked by a Russian operation with calls from the Vermont Governor condemning Vladimir Putin for attempting to hack the grid. Eric Geller pointed out that the first headline the Post ran with was  “Russian hackers penetrated U.S. electricity grid through utility in Vermont, officials say” but they changed to “Russian operation hacked a Vermont utility, showing risk to U.S. electrical grid, officials say.” We don’t know exactly why it was changed but it may have been due to the Post overreacting when they heard the Vermont utility found malware on a laptop and simply assumed it was related to the electric grid. Except, as the Vermont (Burlington) utility pointed out the laptop was not connected to the organization’s grid systems.

Electric and other industrial facilities have plenty of business and corporate network systems that are often not connected to the ICS network at all. It’s not good for them to get infected, and they aren’t always disconnected, but it’s not worth alarming anyone over without additional evidence.  However, the bigger analytical leap being made is that this is related to Russian operations.

The utility notes that they took the DHS/FBI GRIZZLY STEPPE report indicators and found a piece of malware on the laptop. We do not know yet if this is a false positive but even if it is not there is no evidence yet to say that this has anything to do with Russia. As I pointed out in a previous blog, the GRIZZLY STEPPE report is riddled with errors and the indicators put out were very non-descriptive data points. The one YARA rule they put out, which the utility may have used, was related to a piece of malware that is publicly downloadable meaning anyone could use it. Unfortunately, after the story ran with its hyped-up headlines Senator Patrick Leahy released a statement condemning the “attempt to penetrate the electric grid” as a state-sponsored hack by Russia. As Dimitri Alperovitch, CTO of CrowdStrike who responded to the Russian hack of the DNC, pointed out “No one should be making attribution conclusions purely from the indicators in the USCERT report. It was all a jumbled mess.”

Or more simply put: a Vermont utility acted appropriately and ran indicators of compromise from the GRIZZLY STEPPE report as the DHS/FBI instructed the community to do. This led to them finding a match to the indicator on a laptop separated from the grid systems but it’s not yet been confirmed that malware was present. The Vermont Governor Peter Shumlin then publicly chastised Vladimir Putin and Russia for trying to hack the electric grid. U.S. officials then inappropriately gave additional information and commentary to the Washington Post about an ongoing investigation which lead them to run with the headline that the this was a Russian operation. After all, the indicators supposedly were related to Russia because the DHS and FBI said so – and supposedly that’s good enough. Unfortunately, this also led a U.S. Senator to come out and condemn Russia for state-sponsored hacking of the utility.

Closing Thoughts

There are absolutely threats to industrial environments including ICS/SCADA networks. It does make sense that ICS breaches and attacks would be on the rise especially as these systems become more interconnected. It also makes perfect sense that ransomware will be used in industrial environments just like any other environment that has computer systems. And yes, the attribution to Russia compromising the DNC is very solid based on private sector data with government validation. But, to make claims about attacks and attempt to quantify it you actually have to present real data and where that data is coming from and how it was collected. To make claims of new ransomware targeting industrial networks you have to actually provide evidence not simply make a series of analytical leaps. To start making claims of attribution to a state such as Russia just because some poorly constructed indicators alerted on a single laptop is dangerous.

Or more simply put: be careful of analytical leaps especially when they are made without presenting any evidence leading into them. Hypotheses and analysis requires evidence else it is simply speculation. We have enough speculation already in the industrial industry and more will only lead to increasingly dangerous or embarrassing scenarios such as a US governor and senator condemning Russia for hacking the electric grid and scaring the public in the process when we simply do not have many facts about the situation yet.