Browsing Tag

ICS

Minimum Viable Products are Dubious in Critical Infrastructure

December 4, 2015

Minimum Viable Products in the critical infrastructure community are increasingly just mislabeled BETA tests; that needs to be communicated correctly.

The concept of a Minimum Viable Product (MVP) is catching on across the startup industry. The idea of the MVP is tied closely to The Lean Startup model created by Eric Ries in 2011 and has very sound principals focused around maximizing the return on investment and feedback from creating new products. Eric defines the MVP as the “version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.” This enforces the entrepreneurial spirit and need for innovation combined with getting customer feedback about a new technology without having to develop a perfect plan or product first. An MVP is also meant to be sold to customers so that revenue is generated. In short, be open to testing things out publicly earlier, pivot based off of technical and market feedback, and earn some money to raise the valuation of the company and entice investors.

Personally, I believe the lean startup model as a whole is smart. I use some aspects of the model as CEO of Dragos Security. However, I chose not to use the concept of an MVP. Minimum Viable Products are dubious in critical infrastructure. I state this understanding that the notion of getting the product out the door and gaining feedback to guide its development is a great idea. And when I say critical infrastructure I’m focusing heavily on the industrial control system (ICS) portion of the community (energy, water, manufacturing, etc.). The problem I have though is that I have observed a number of startups in the critical infrastructure startup space taking advantage of their customers, albeit unintentionally, when they push out MVPs. This is a bold claim, I won’t point fingers, and I don’t want to come across as arrogant. But I want to make it very clear: the critical infrastructure community deals with human lives; mistakes, resource drains, and misguiding expectations impact the mission.

My observations of the startups abusing the MVP concept:

  • Bold claims are made about the new technologies seemingly out of a need to differentiate against larger and more well established companies
  • Technologies are increasingly deployed earlier in the development cycle because the startups do not want to have to invest in the industry specific hardware or software to test the technology
  • The correct customers that should be taking part in the feedback process are pushed aside in favor of easier to get customers because successes are needed as badly as cash; there is pressure to validate the company’s vision to entice or satisfy Angel or Seed investors
  • The fact that the technology is an MVP, is lightly (if at all) tested, and will very likely change in features or even purpose is not getting communicated to customers in an apparent attempt to get a jump start on the long acquisition cycles in critical infrastructure and bypass discussions on business risk
  • Customers are more heavily relied upon for feedback, or even development, costing them time and resources often due to the startups’ lack of ICS expertise; the startup may have some specific ICS knowledge or general ICS knowledge but rarely does it have depth in all the markets its tackling such as electric, natural gas, oil, water, etc. although it wants to market and sell to those industries

What is the impact of all this? Customers are taking bigger risks in terms of time, untested technologies, changing technologies, and over hyped features than they recognize. If the technology does not succeed, if the startup pivots, or if the customers burn out on the process all that’s been accomplished is significant mistrust between the critical infrastructure stakeholders and their desire to “innovate” with startups anymore. And all of this is occurring on potentially sensitive networks and infrastructure which have the potential to impact safety or the environment.

My recommendations to startups: if you are going to deploy technologies into critical infrastructure early in the development cycle make sure the risks are accurately conveyed and ensure that the customer knows that they are part of a learning process for your technology and company. This begs instant push-back: “If we communicate this as a type of test or a learning process they will likely not trust our company or technology and choose to go with other more established products and companies. We are trying to help. We are innovators.” And to my straw man here, I empathize greatly. Change is needed in this space and innovation is required. We must do better especially with regards to security. But even if we ignore the statistics around the number of failed technologies and startups that would stress why many should never actually touch an ICS environment I could comfortably state that the community is not as rigid as folks think. The critical infrastructure community, especially in ICS, gets cast in a weird light by many outside the community. My experience shows that the critical infrastructure community is just as innovative, and I would argue more so, than any other industry but they are much more careful to try to understand the potential impact and risks…as they should be.

My experience in a new technology startup: when the Dragos Security team was developing our CyberLens software we needed to test it out. Hardware was expensive and we could not afford to build out networks for every type of vendor’s ICS hardware and network communications. Although we have a lot of ICS knowledge on the team we all were keenly aware that we are not experts in every aspect of every ICS industry we wanted to sell to. Customer feedback was (and still is) vital. To add to this we were pressed because we were competing with larger more established companies and technologies but on a very limited budget. So, instead of trying to sell an MVP we simply launched a BETA instead; the BETA lasted over twelve months. How did we accomplish this? We spent $0 on marketing and sales and focused entirely on staying lean and developing and validating our technology. We made contacts in the community, educated them on what we wanted to do, advised where the technology was tested and safe to deploy, and refused to charge our BETA participants for our time or product since they were greatly helping us and keeping our costs down. In turn we offered them discounts for when our product launched and offered some of our time to educate them in matters we did have expertise. This created strong relationships with our BETA participants that carried over when we launched our product to have them join us as customers. We even found new customers when we launched based on referrals from BETA participants vouching for our company. Or more simply stated: we were overly honest and upfront, avoided hype and buzzwords, and brought value so we were seen as fellow team members and not snake oil salesmen. I recommend more startups take this approach even under pressure and when it is difficult to differentiate in the market.

My conclusion: the MVP model in its intended form is not a bad model. In many communities it is an especially smart model. And just because a company is using an MVP route in this space does not mean they are abusing anyone or falling into the pitfalls I listed above. But, as a whole, in the critical infrastructure community it is a process that is more often abused than used correctly and it is damaging in the long term. Customers are not cash cows and guinea pigs – they are investors in your vision and partners. Startups should still push out technologies earlier than trying to wait to create a perfect product without the right feedback, but call these early pushes what they are. It is not a Minimum Viable Product it is a BETA test of core features. Customers should not be asked to spend limited budgets on top of their time and feedback for it nor should they be misled as to what part of the process they are helping in. You will find the community is more likely to help when they know you are being upfront even with understandable shortcomings.

Security Awareness and ICS Cyber Attacks: Telling the Right Story

October 7, 2015

This was first posted on the SANS ICS blog here.

 

A lack of security awareness and the culture that surrounds security is a widely understood problem in the cyber security community. In the ICS community this problem is impactful towards operations and understanding the scope of the threats we face. A recent report by the Chatham House titled “Cyber Security at Civil Nuclear Facilities”shined a light on these issues in the nuclear industry through an 18 month long project.

The report highlights a number of prevailing problems in the nuclear sector that make security more difficult; the findingsdo not represent all nuclear sector entities but take a look at the sector as a whole. Friction between IT and OT personnel, the prevailing myth that the air gap is an effective single security solution, and a lack of understanding the problem are all cited as major findings of the research group.

The group recommends a number of actions which need to be taken and these can be mapped along the Sliding Scale of Cyber Security. A big focus is placed on better designing the systems to have security built into them which can be understood in the Architecture phase of the scale. Another focus was on leveraging whitelisting and intrusion detection systems as well as other Passive Defense mechanisms instead of just an air gap. Lastly, one of the most significant recommendations was towards getting more personnel trained in cybersecurity practices (SANS offers ICS410 and ICS515 to address these types of concerns) and take a proactive approach versus a reactive approach towards finding threats in the environment — this recommendations maps to the Active Defense component of the scale which focuses on empowering analysts and security personnel to hunt for and respond to threats.

One of the more interesting major recommendations put forth by the report was:
“The infrequency of cyber security incident disclosure at nuclear facilities makes it difficult to assess the true extent of the problem and may lead nuclear industry personnel to believe that there are few incidents. Moreover, limited collaboration with other industries or information-sharing means that the nuclear industry tends not to learn from other industries that are more advanced in this field.”

At SANS we have consistently observed this as an issue in the wider community and try to bring the community together with events such as the ICS Summit to help address the concernand promote community sharing. No single event or effort alone though can fix the problem. A lack of information sharing and incident disclosure has led to a false sense of security while also allowing fake or hyped up stories in news media to become the representation of our industry to people in our community and external to it.

This aspect of infrequency of cyber security incident disclosure can be observed in multiple places. As an example, an article from 2014 by Inside Energy compiled incident reporting to the Department of Energy about electric grid outages and over 15 years noted that there were 14 incidents related to a cyber event. The earliest cyber attack was identified in 2003 but then there was a lack of events until 2011-2014 which made up the other 13 cases. It should be noted that the reporting for a cyber attack was any type of unauthorized access to the system including the hardware, software, and data.

We in the industry need to have better data so that we can more fully understand and categorize attacks along models such as the ICS Cyber Kill Chain to extract lessons learned. What is revealing about the Department of Energydata though is the lack of visibility into the ICS networked environment. As an example, in the data set there is a measured understanding of impact for physical attacks, fires, storms, etc. showing great visibility into the ICS as a whole but for every single event regarding cyber the impact was either labeled as zero or unknown; that in combination with no data for 2003-2011 is less representative of the number of events and more representative of missing data. It has become clear over the years that a significant number of ICS organizations do not have personnel that are trained and empowered to look into the network to find threats. This must change and the findings must be shared, anonymously and appropriately, with the community if we are ever to scope the true threat in the community and determine the appropriate resource investments and responses to address the issues.

The ICS community stands a unique opportunity to have our story told by our ICS owners, operators, and security personnel to understand and address the problem ourselves. Valuable compilations of data such as that by Inside Energy using the Department of Energy reports as well as the Chatham House report help reinforce this need. Without involvement from the community, the ICS security story will be told by others who may not have the appropriate experience to make the right conclusions and offer helpful solutions. The need for cyber security will influence change in the ICS community through national level policies, regulations, vendor practices, and culture shifts – it is imperative that the right people with real data are writing the story that will drive those changes.

Three Takeaways from the State of Security in Control Systems Survey

July 7, 2015

This was first posted on the SANS ICS blog here.

 

The State of Security in Control Systems Today was a SANS survey conducted with 314 ICS community members and was released on June 25th. The whitepaper can be found here and the webcast here. A few things stuck out from the survey that I felt it appropriate to highlight in this blog.

  1. Energy/Utilities Represent

Energy/Utilities made up the most of the respondents with 29.3% in total. While the variables impacting this cannot be narrowed down it is likely that pressure from organizations such as NERC, heavy focus on energy protection in the U.S. in national media and politics, and market interest has at least driven security awareness. We also see an energy bias in other metrics on reporting such as the ICS-CERT’s quarterly reports. This is a both a good thing and an area for improvement. It is great to see the energy sector get heavily involved in events such as this survey, in training conferences, and major events like the electric sector’s GridEx. Personally, I’ve interacted with groups such as the ES-ISAC and been extremely impressed. Getting data from this segment of the community helps understand the problem better so that we can all make the appropriate investments in security.

Takeaway: We really need to do more to reach the other communities. Energy tends to be a hot topic item but it is far from the only industry that has security issues. Each portion of the ICS community from water to pharmaceuticals face similar issues. In the upcoming years hopefully reports like this SANS survey will be able to capture more of those audiences. I feel this is likely given the increased awareness in other industries I have seen even in the last few years.

 

  1. IT/OT Convergence Seen as 2nd Most Likely Threat

The number one vector the respondents felt was the most significant threat to their ICS was external threats. This makes sense given the increased understanding in the community regarding external actors and the cyber security of operations. However, interestingly the second top threat identified as the integration of IT into control system networks. I really liked seeing this metric because I too believe it presents one of the largest threat vectors to operations. ICS targeted nation state malware tends to get the most media attention. BlackEnergy2, Stuxnet, and Havex were all very concerning. However, it is far more likely on a day to day basis that not architecting and maintaining the network correctly will lead to decreased or stopped operations. The integration of OT and IT also presents a number of challenges with incidental malware that, while non-targeted, presents a significant risk as has been documented numerous times when important systems halt due to accidental malware infections such as Conficker.

Takeaway: The ICS community needs to be aware of external threats and realize that they pose the most targeted threat to operations. However, it was great seeing that issues revolving around the integration of IT and OT is accurately seen as a concern. Architecting and maintaining the OT network correctly to include safe and segmented integration, structuring such as the Purdue model, and ultimately reducing the risks associated with IT/OT convergence will go a long way for the security of the environment. The type of efforts required to reduce the risk of IT/OT convergence is also the same foundational efforts that help identify, respond, and learn from external threats and threat vectors.

 

  1. Lack of Visibility is Far Reaching

A significant portion of the group, 48.8%, stated that they simple did not have visibility into their environment. This could mean a number of things to include IT and OT not having visibility into each other’s processes and environment, lack of understanding of the networked environment, inability to collect data such as network traffic or logs, and a lack of a plan to pull together all stakeholders when appropriate. Each of these has been observed and continually documented as problems in the ICS community. What is interesting about this single metric though is that it impacts most of the other metrics. For example, respondents who do not have visibility into their environment will not be able to fully identify threats in their environment; 48.8% stated that they were not aware of any infiltration or infection of their control systems. Additionally, when a breach occurs it is difficult to respond correctly without visibility; 34% of the participants who had identified breaches stated that they had been breached multiple times in the last 12 months.

Takeaways: Nearly half of the respondents to the survey indicated that they did not have visibility into the environments. This makes it incredibly difficult to know if they have been impacted by breaches. It also makes it difficult to scope a threat and respond appropriately. I would bet that a significant portion of those participants who indicated they were breached multiple times had links between the breaches that they were unaware of due to a lack of visibility. Re-infections that occur due to not fully cleaning up after a breach are common in the IT and OT communities. ICS community members need to ensure that they are developing plans to increase their visibility. That means including all stakeholders (in both IT and OT), ensuring that at least sampling from the environment can be taken in the form of logs and network traffic, and talking with vendors to plan better visibility into system upgrades and refreshes. For example, a mirrored port on a network switch is a great resource to gain invaluable network traffic data from the OT environment that can help identify threats and reduce time and cost of incident response.

Follow on: To help with the discussion of visibility into the environment I will post two entries to the SANS ICS blog in the upcoming weeks. They will be focused on two of the beginning labs in SANS ICS515 — Active Defense and Incident Response. The first will cover using Mandiant’s free incident response tool: Redline and how to use it in an ICS to gather critical data. The second will cover using some basic features in Wireshark to sample network traffic and identify abnormalities.

Final Thoughts

I was very impressed with the participants of the SANS survey. Their inputs help give a better understanding into the community and its challenges. While the takeaways above focus on areas for improvement it is easy to look at the past few years and realize that security is increasing overall. Security awareness, trained security professionals, and community openness are all increasing. We have a long way to go in the community but we are getting better. However, there are many actions that can and should be taken today to drastically help security. First, we must be more open with data and willing to participate in spot checks, like surveys, on the community. Secondly, wherever there is a lack of a plan forward, such as IT/OT convergence strategies, the appropriate stakeholders need to meet and discuss with the intent to act. Thirdly, incidents are happening whether or not the community is ready for it. Appropriate visibility into the environments we rely on, incident response plans, and identified personnel to involve are all requirements. We can move the bar forward together.