Monthly Archives

December 2015

Facts, Hype, and Takeaways from Reports on Iranian Activity Against the Power Grid and a Dam

December 21, 2015

This was first posted on the SANS ICS blog here.


Yesterday a report on Iranian activity focused on a small dam in New York was released by Danny Yadron at the Wall Street Journal. Today a report was released by Garance Burke and Jonathan Fahey at the Associated Press reporting on Iranian activity linked to the OpCleaver report by CYLANCE where documents related to Calpine were stolen. So what’s the hype, what are the facts, and what are the takeaways? Let’s explore:
The Facts:

I’ve worked with both Danny and Garance before and have a high amount of respect for the effort they put into their reporting. Reporting on technical content can be very difficult and sometimes leads to inaccurate reports especially when the topic of security is combined with control systems. Garance and Danny do their homework though. In that regards I instantly feel more positive about the articles. That’s also why I was willing to contribute a quote to Garance when she called and wanted a quote for the story. I didn’t get to see the story, didn’t know all that was going to be written, but understanding the type of data that was stolen that was related to Calpine — yes it absolutely is something an adversary would want and defenders should protect and she was correct to in emphasizing that.

In theWSJ story there were namedindividuals in the town who were present for, and recalled, the FBI response to the activity. Additionally, there are unclassified reports from the FBI and ICS-CERT that could likelycorrelated to the dam event. Both stories are credible in the fact that they occurred. Not all the details are properly fleshed out though for the ICS community and there’s a few areas to leave you wanting.

When looking at the ICS Cyber Kill Chain where did the dam and Calpine cases fall? Neither of them were attacks. I’d put the dam activity under Reconnaissance in Stage 1 and I’d put the Calpine case under Act under Stage 1 but not in Calpine networks. This is important to note; the “Act” was the exfiltration of sensitive documents related to Calpine but the intrusion was not in Calpine but instead in contractor networks. Or simply put, neither Calpine nor the dam were compromised. But both showed a focused effort by an adversary, possibly Iran but attribution is always tricky, against infrastructure.

Also, in the case of the dam the WSJ report notes that U.S. authorities confused the dam with other similarly named dams. The cell modem the infrastructure used would have been distributed out in a manner that likely made physical location difficult to determine. This may have confused the adversaries as well.

I’d highlight the following facts:

  • Both reports are credible news organizations and reporters
  • The WSJ report on the dam is additionally credible with regards to the event having taken place (the details could always be wrong though). This is due to correlated details with other unclassified reports, timing considerations, and a named source noting that the FBI did respond
  • The WSJ report identifies that the activity was “probing” but likely not scanning activity; the focused effort on queries and searches by the adversaries is more of a targeted Reconnaissance than random scans
  • The AP report is additionally credible given that named sources identified and provided samples of the stolen documents of internal information, passwords, and system diagrams in Calpine
  • The AP report identified that sensitive data about Calpine was stored on contractor networks and was not stolen from internal to the ICS
  • Neither the dam nor Calpine were compromised. There was no intrusion into ICS networks nor were there any attacks.


The Hype:

To anyone in the ICS community these reports likely have some cringe worthy statements to you. This has been the discussion in various social media circles already where ICS security professionals have taken offense to statements such as the AP report’s “cyberattackers had opened a pathway into the networks running the United States power grid.” The comments, which I agree with, are that there is no open pathway based off of these stolen documents. With the WSJ report the title states “Iranian Hackers Infiltrated New York Dam” which obviously did not actually occur since there was no intrusion. These issues are consistent with any news reports regardless of how good the journalists are on the subject matter. This is where I’m both empathetic and exhausted.

I’m empathetic because there are a lot of eyes on these reports and hands in the proverbial cookie jar. Very rarely do journalists get to choose their report’s title. Additionally, the reporters’ main audience is not the ICS community. It’s a more laymen non-technical audience. Any report that those of us would come up with in the community focused only on measured facts would likely be incredibly boring or entirely too technical for a laymen audience. I’m exhausted because this type of activity is understandable but not excusable. If we continue to hype threats and accidently miseducate the audience people will pay attention to that. Outside our community there are folks who impact our community. Policy makers and the general public have a lot of impact on ICS. Journalists and news organizations need to do better for sure, but we should also take into consideration they are trying to make something out of reports from a community who does not like sharing these types of events. Overall I felt positively about the articles but I’d like to see the news reporting community as a whole do better with regards to ICS and security.

I’d highlight the following hype:

  • The WSJ’s title is misleading as there was no intrusion
  • The AP’s statements around the impact of this data loss is (in some places) misleading. It is valuable data but does not make the grid any more vulnerable today than it was before
  • Both reports provide very little evidence and rely on unnamed sources for the attribution to Iran; given the number of reports and correlation of events the case is stronger than usual but still not enough to truly validate that the Iranian government was responsible


The Takeaways:

If I’m going to highlight the flaws of the journalist community I’m certainly going to highlight our own community’s flaws. News organizations need to do better in general but the ICS community needs to get better at identifying issues and being willing to share lessons learned. Some organizations are amazing at this but as an overall community there’s work to be done. The identification of the loss of data related to Calpine only came from a member at CYLANCE identifying the sensitive documents on one of the adversary’s FTP servers. The amount of documents stolen from multiple sites should have been detected by someone internal to those networks (such as Calpine’s contractor) and not waiting for a 3rd party notification. Additionally, anyone in the ICS community who’s been here long enough can think about a couple of close calls and actual incidents that are not public. If the community cannot figure out a way to responsibly share case studies and lessons learned then we will have to accept other people outside our community writing the narrative. It’s a hard task but we have to figure it out.

Defense is definitely getting better. ICS is not as vulnerable as people make it out to be. And defenders are taking a more proactive approach to security than ever before. But we as a community have some ground to cover. Taking an active defense approach to monitoring our networks, performing incident response, and sharing the non-sensitive details for the community to learn is required for us to raise the bar and have the ICS security story be written by the ICS community. Journalists are going to tell the stories regardless. It’s up to us to identify and guide a proper narrative or to not complain about it.

Additionally, the stories both highlight a focused effort by foreign adversaries targeting infrastructure. It also highlights sensitive ICS data being stored on non-ICS networks. This reinforces the need to bridge the IT/OT gap and have ICS and IT professionals work more closely together. The one thing I’ll push back on a bit from the ICS perspective is the comment from Calpine that the stolen data, diagrams, and passwords were old and thus pose no threat. Calpine may bean industry leader in this area but ICS diagrams, passwords, and data does not change that quickly at all and even when old can be useful. This type of information is definitely useful to an adversary for reconnaissance and learning purposes — but no it is not a threat to bringing down the grid or Calpine’s facilities.

I’d highlight the following takeaways:

  • The cultural and technical barriers to identifying incidents, responding to them, and sharing lessons learned need to be reduced in the community so the proper narrative can be written and security can be elevated
  • The IT/OT gap is a divide that must be bridged if for no other reason than the fact that all the sensitive information about an ICS does not just rely on the ICS networks; IT networks including contractor networks can reveal data about the ICS that we do not want adversaries to have
  • The data from the AP story would be useful to adversaries but should not be overvalued. The biggest takeaway is a focused effort by adversaries to learn about infrastructure and target it
  • The power grid or infrastructure such as dams are not as easy to impact as folks like to make it sound, but adversaries are getting smarter and focusing harder on this challenge — defenders too must get smarter and focus on the threat to keep the opportunity to damage infrastructure out of the hands of malicious actors

Minimum Viable Products are Dubious in Critical Infrastructure

December 4, 2015

Minimum Viable Products in the critical infrastructure community are increasingly just mislabeled BETA tests; that needs to be communicated correctly.

The concept of a Minimum Viable Product (MVP) is catching on across the startup industry. The idea of the MVP is tied closely to The Lean Startup model created by Eric Ries in 2011 and has very sound principals focused around maximizing the return on investment and feedback from creating new products. Eric defines the MVP as the “version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.” This enforces the entrepreneurial spirit and need for innovation combined with getting customer feedback about a new technology without having to develop a perfect plan or product first. An MVP is also meant to be sold to customers so that revenue is generated. In short, be open to testing things out publicly earlier, pivot based off of technical and market feedback, and earn some money to raise the valuation of the company and entice investors.

Personally, I believe the lean startup model as a whole is smart. I use some aspects of the model as CEO of Dragos Security. However, I chose not to use the concept of an MVP. Minimum Viable Products are dubious in critical infrastructure. I state this understanding that the notion of getting the product out the door and gaining feedback to guide its development is a great idea. And when I say critical infrastructure I’m focusing heavily on the industrial control system (ICS) portion of the community (energy, water, manufacturing, etc.). The problem I have though is that I have observed a number of startups in the critical infrastructure startup space taking advantage of their customers, albeit unintentionally, when they push out MVPs. This is a bold claim, I won’t point fingers, and I don’t want to come across as arrogant. But I want to make it very clear: the critical infrastructure community deals with human lives; mistakes, resource drains, and misguiding expectations impact the mission.

My observations of the startups abusing the MVP concept:

  • Bold claims are made about the new technologies seemingly out of a need to differentiate against larger and more well established companies
  • Technologies are increasingly deployed earlier in the development cycle because the startups do not want to have to invest in the industry specific hardware or software to test the technology
  • The correct customers that should be taking part in the feedback process are pushed aside in favor of easier to get customers because successes are needed as badly as cash; there is pressure to validate the company’s vision to entice or satisfy Angel or Seed investors
  • The fact that the technology is an MVP, is lightly (if at all) tested, and will very likely change in features or even purpose is not getting communicated to customers in an apparent attempt to get a jump start on the long acquisition cycles in critical infrastructure and bypass discussions on business risk
  • Customers are more heavily relied upon for feedback, or even development, costing them time and resources often due to the startups’ lack of ICS expertise; the startup may have some specific ICS knowledge or general ICS knowledge but rarely does it have depth in all the markets its tackling such as electric, natural gas, oil, water, etc. although it wants to market and sell to those industries

What is the impact of all this? Customers are taking bigger risks in terms of time, untested technologies, changing technologies, and over hyped features than they recognize. If the technology does not succeed, if the startup pivots, or if the customers burn out on the process all that’s been accomplished is significant mistrust between the critical infrastructure stakeholders and their desire to “innovate” with startups anymore. And all of this is occurring on potentially sensitive networks and infrastructure which have the potential to impact safety or the environment.

My recommendations to startups: if you are going to deploy technologies into critical infrastructure early in the development cycle make sure the risks are accurately conveyed and ensure that the customer knows that they are part of a learning process for your technology and company. This begs instant push-back: “If we communicate this as a type of test or a learning process they will likely not trust our company or technology and choose to go with other more established products and companies. We are trying to help. We are innovators.” And to my straw man here, I empathize greatly. Change is needed in this space and innovation is required. We must do better especially with regards to security. But even if we ignore the statistics around the number of failed technologies and startups that would stress why many should never actually touch an ICS environment I could comfortably state that the community is not as rigid as folks think. The critical infrastructure community, especially in ICS, gets cast in a weird light by many outside the community. My experience shows that the critical infrastructure community is just as innovative, and I would argue more so, than any other industry but they are much more careful to try to understand the potential impact and risks…as they should be.

My experience in a new technology startup: when the Dragos Security team was developing our CyberLens software we needed to test it out. Hardware was expensive and we could not afford to build out networks for every type of vendor’s ICS hardware and network communications. Although we have a lot of ICS knowledge on the team we all were keenly aware that we are not experts in every aspect of every ICS industry we wanted to sell to. Customer feedback was (and still is) vital. To add to this we were pressed because we were competing with larger more established companies and technologies but on a very limited budget. So, instead of trying to sell an MVP we simply launched a BETA instead; the BETA lasted over twelve months. How did we accomplish this? We spent $0 on marketing and sales and focused entirely on staying lean and developing and validating our technology. We made contacts in the community, educated them on what we wanted to do, advised where the technology was tested and safe to deploy, and refused to charge our BETA participants for our time or product since they were greatly helping us and keeping our costs down. In turn we offered them discounts for when our product launched and offered some of our time to educate them in matters we did have expertise. This created strong relationships with our BETA participants that carried over when we launched our product to have them join us as customers. We even found new customers when we launched based on referrals from BETA participants vouching for our company. Or more simply stated: we were overly honest and upfront, avoided hype and buzzwords, and brought value so we were seen as fellow team members and not snake oil salesmen. I recommend more startups take this approach even under pressure and when it is difficult to differentiate in the market.

My conclusion: the MVP model in its intended form is not a bad model. In many communities it is an especially smart model. And just because a company is using an MVP route in this space does not mean they are abusing anyone or falling into the pitfalls I listed above. But, as a whole, in the critical infrastructure community it is a process that is more often abused than used correctly and it is damaging in the long term. Customers are not cash cows and guinea pigs – they are investors in your vision and partners. Startups should still push out technologies earlier than trying to wait to create a perfect product without the right feedback, but call these early pushes what they are. It is not a Minimum Viable Product it is a BETA test of core features. Customers should not be asked to spend limited budgets on top of their time and feedback for it nor should they be misled as to what part of the process they are helping in. You will find the community is more likely to help when they know you are being upfront even with understandable shortcomings.