We caught Jens Weisner at S4 and he talks about cybersecurity in Germany – progress, challenges and a little comparing of the German approach to what he sees happening in North America.

Jens Weisner, Industrial Cybersecurity Unit Head Bundesamt für Sicherheit in der Informationstechnik (BSI)”

          

READ HERE THE FULL TRANSCRIPT:

Intro: The Industrial Security podcast with Andrew Ginter and Nate Nelson, sponsored by Waterfall Security Solutions.

Nate: Welcome, all, to the Industrial Security podcast. My name as usual is Nate Nelson, I’m here as usual with Andrew Ginter, vice president of Industrial Security at Waterfall Security Solutions. He’s going to introduce today’s guest. Andrew, how’s it going?

Andrew: I’m doing well, thank you, Nate. Our guest today is Jens Wiesner he is the director of the industrial cyber security unit at the German BSI. BSI is German, it’s Bundesamt für Sicherheit in der Informationstechnik, which is a crude approximation of the English translation Federal Office for Information Security. So, let’s go to my interview with Jens.

Hello, Jens.

Jens: Hello, Andrew.

Andrew: Thank you for joining us. Can we get started with the basics? I know that the BSI is vaguely analogous to the DHS here in North America, what is the BSI? What do you do over there?

Jens: So, the BSI is the German Federal Office for Information Security. It’s somewhat different from the DHS because our main focus was in the past time, securing governmental networks, and that has changed now to critical infrastructure protection, but only the IT side of critical infrastructure protection. So, if we are not doing the FEMA part, we are doing a bit ICS-CERT, a bit GOV-CERT and the bit NIST stuff. So, certification is also part of the BSI.

Nate: Okay, Andrew, can we start by having you define some of the terms that Jens used?

Andrew: Sure. He talked about an ICS-CERT, that’s the industrial control system computer emergency response team. In North America, that’s the team with the flyaway team, when there’s an incident at an industrial site, they pick up their gear, they fly to the site, they help respond. They’ve got obviously a center where they have experts and can offer advice as well. The government cert is the same thing, but for presumably government agencies. NIST is the United States National Institute of Standards and Technologies, they do standards. Now, he mentioned that BSI does some certification, I’m thinking he might mean enforcement, I’m thinking the BSI might have a regulatory role. I used the term DHS, but he points out rightfully that the DHS is a very big agency in the United States and stuff like the FEMA, the Federal Emergency Management Agency, is not part of the BSI mandate. The DHS does everything from physical security at airports to hurricane response, and it’s the ICS security part that is relevant to BSI.

Nate: I want to hear more about what Jens does, let’s get back to your interview with him.

Andrew: The topic near and dear to my heart, this is The Industrial Security podcast is the ICS part, what are you doing in the ICS space?

Jens: So, Germany has a lot of critical infrastructure and they need to be secured, meaning the technical aspects of industrial control systems which are being used in critical infrastructures have to be understood, have to be secured. And I’m heading the team which is responsible for the technical aspects of critical infrastructure protection.

Andrew: I know that over here, we work with the DHS, used to be called the ICS-CERT, it’s called something else now. They had conferences, they produced literature and guidance, are there comparable functions in the BSI?

Jens: Sure. A big part of our work is awareness, meaning we have, at the moment, I think 19 papers which are also available in English. Some of them are very short, something like incident stories or how to do remote access, easy things up to 200 pages or 150 pages at the moment, which are something like OT for IT guys, how to do secure OT networks.

Andrew: How about conferences? Waterfall goes to a lot of conferences in North America, I’m much less aware of what’s happening in Germany.Jens: So, conferences in Germany focusing on ICS are extremely rare. There’s only 1 which is called IT Meets Industry which focuses on the chemical industry in Germany, and there’s CCC which is the something like the black head of Germany.

Nate: Andrew, you told me that you got a chance to look at some of BSI’s papers, correct?

Jens: I did, I did, they’ve got some good stuff there. The one I particularly liked was their top 10 threats and countermeasures. They go through 10 kinds of attacks, 10 kinds of threats and they talk about how to deal with them. It’s, in a sense, it’s a nice getting started paper. If you’ve got a smaller critical infrastructure site, you’ve got a smaller than industrial site and you haven’t been terribly active in the security space until now, this is a great starting point to understand the basic threats everyone has to deal with and understand the usual ways to address those threats in the ICS space. But there’s other papers there too and that in fact was my next question to Jens here.

Your best practice guidance, can you give me an example, what do you guys talk about?

Jens: For example, last year, we published a paper on medical device security. We’re focused on best practices which are easy to achieve, not comprehensive, not complete checklists, but things you have to think about and to implement it. And for medical devices, the medical device is also I know doctors don’t like it when I say this, but it’s like a factory with PLCs inside and they have to work, and if they don’t work, people are going to die. It’s the same with medical devices.

Nate: It’s a very interesting idea to compare medical devices to industrial ones, but maybe that’s also somewhat of a communication problem that someone like Jens can see it this way, but of course, he mentioned doctors may not see medicine in the same way that industrial workers see factories.

Andrew: So, yeah, medical devices are not like factories, but there’s a general perception, a widespread perception that medical device security is lagging the security initiatives in the industrial space. And a lot of the issues are the same, especially when it comes to safety systems. And in fact, safety systems is what Jens talked about next. Let’s listen in.

 

So, that’s interesting, the medical devices, I’ve seen interest in that field lately, there’s more and more people talking about that field. What else is hot? What are you working on lately? What have you produced lately?

Jens: So, everybody knows Triton/Trisis, and in fact, we were working on the topic securing safety systems before everybody talked about Triton/Trisis. So, we started 2016 already and finalized the paper which was published in December 2017, and then Triton/Trisis was published and all hell broke loose. And in the aftermath of Triton/Trisis, we generated open source Snort rules which everybody can deploy in their networks if they have the capability to, and they can use them to connect changes in the safety system. And they have to connect this to a change management system that every change in the programming of the controller has to be matched with a downtime with some process which is authorized by a third party or by their superiors, that unauthorized changes have to be tracked. And that’s something it’s very hard for an attacker to circumvent these measures.

Nate: Andrew, I’ve got a few clarification questions for you. Firstly, what is Triton/Trisis?

Andrew: Well, Triton/Trisis had 2 different names for the August 2018 attack on the Saudi refinery safety system. This was the first time that we had observed that anyone had had observed in the wild a piece of malware taking on a safety system and trying to impair the operation of the safety system. And this is of course very bad because, if the attack had succeeded, if the safety system had been disabled, it means that unsafe conditions at the plant would not have been diagnosed and we could have had disaster, we could have had loss of life. The purpose of a safety system is to protect human life at complex dangerous industrial sites, and this was an attack on one of those systems, the first we’d observed.

Nate: Okay, my second question, what are Snort rules?

Andrew: Well, Snort is an open source intrusion detection system. It’s a pun, when we look at network traffic, we often say we’re sniffing the traffic, we’re seeing a copy of the traffic. The traffic continues, we don’t impair the traffic, it’s looking at it from the outside, it’s called sniffing. So, Snort is a great big sniff that pulls a lot of packets in and analyzes them. It’s like an anti-virus for network traffic, raises alerts when it sees suspicious patterns.

Nate: Okay. My last clarification question about what Jens talks about, what is change control?

Andrew: That’s a good question, it could mean a couple of different things. I in fact asked him about that next, so let’s go back to the recording here.

Let me understand what you’re talking about. Is this a change control process or is this a technology for detecting unauthorized changes?

Jens: The Snort rules, it’s a simply intrusion detection system for detecting changes in the system, but a change could be legit, so it has to be connected to a other management system owned by the operator and these 2 have to be connected. So, an intrusion detection system might generate a false positive in this way.

Andrew: Right. So, what Jens was talking about was what I’ve heard called elsewhere as the engineering change control process. The Snort rules are detecting, changes over the network chain to the configuration of the safety system. And of course safety systems don’t change very often. Any change to a safety system is a big deal. Any change to a safety system really should be accompanied by a work order, should be accompanied by a planning process, by an engineering process that went through an inquiry to make sure that the change that’s proposed to the safety system is safe, is not going to cause problems with safety at the plant. And so, every time you touch the safety system, it better be a work order detailing what’s going to change. And every time that Snort rule goes off saying, “Hey, someone changed the configuration,” there’d better be a work order to correspond to it. So, he’s not talking about any kind of automation to automatically do any of that, what he’s talking about is the BSI published Snort rules to detect changes in the Triconex safety systems that the Trisis/Triton attack targeted, and of course, the attack was changing those configurations. And so, if you ever see that rule fire and say, “Hey, there’s a change,” you’ve got an opportunity to go back to your work orders and say, “But there’s no work order, this is suspicious,” and trigger an investigation, and if necessary, trigger a safety shut down until you’ve repaired your safety systems. So, I think this is the long winded version of what Jens was talking about. So, after this topic we changed gears and I started asking about sort of the difference between his experience of what he sees in Germany and in the rest of the world.

So, Jens, you get around, you’re here at S4, I’ve seen you at some of the other North American events, how would you compare what’s going on in Germany to what’s going on in other parts of the world that you’re visiting and you’re your understanding?

Jens: So, honestly, I’ve never been to a critical infrastructure in the United States. So, it’s a bit difficult to go in depth and compare make the comparison in depth, but I see it seems to me we are struggling with the same issues, meaning legacy products, meaning insufficient awareness on the operator sides, starting with a really easy stuff, not having basic knowledge of the inventory of the network, not having an overview of the network itself, not having been prepared for incidents, for cyber incidents. And, for me, most companies are in the early steps, early stages. For example, electricity generation transmission grid, they’re quite mature, it seems it, for me, like if in the United States. Same in Germany, but others, for example, wastewater treatment facilities are mostly small companies, at least in Germany, they are very small companies and there are lots of them, consisting of very few personnel, electricians mostly, and not trained for anything which includes cyber.

Nate: So, Andrew, does that sound similar to you?

Andrew: It does, it sounds very similar. I asked because we always wonder, I always wonder if somebody else in the world hasn’t already solved these problems and I’m just not aware of them because of language barriers or other barriers to communication. But it sounds like it’s very similar in Germany to the situation in North America, some industries are further ahead, some are behind, the electric sector leads, this is a very familiar circumstance for someone who’s well-versed in in the North American industry. So, given that other industries and some industries and especially smaller sites are further behind, my next question had to do with small sites.

So, that’s very interesting. I think it’s very much the same in North America. I saw a statistic a little while ago saying, in the United States alone, there’s something like 20,000 drinking water treatment utilities and over 200,000 wastewater treatment utilities, almost all of which must be tiny. Do you have a stream of advice for small sites, sites that don’t have full-time staff?

Jens: In Germany, a critical infrastructure is defined that they have an effect on 500,000 people and more. So, we’re not covering every small utility so it a might be a bit different to the United States. On the other hand, the first thing is not to put your head in the sand like the ostrich and to start. And to start, for example, we have documents, top 10 threats and countermeasures, anybeverybody has them, you have them as well. You took the number 20 and not the number 10, it doesn’t matter. You just start with them, you find your crown jewels, “What hurts? What’s really critical when it stops?” And the other point is, “What’s your exposure?” For example, we still find sites on Shodan in Germany, it’s happened 2 weeks ago, which were doing easy simple stuff, but they were fully exposed to the internet with VNC, with the HMI, with service ports. And talking to the when to the integrator, they said, “Oh, we should shift the ports, we should change the ports.” No, they shouldn’t do that, they should do something different and they should do it properly. So, it’s not only an issue of this site operator, but also of the personnel who built the sites and to maintain the sites, which are often different people.

Nate: Can you explain Shodan?

Andrew: Yes, Shodan is a search engine, it’s analogous to Google. What Google does is searches web servers and so, really, I imagine the Google search engine connects to more or less every IP address on the planet looking for port 80 or port 443; these are TCP ports that are serving web sites or encrypted websites. And once it connects to them, it asks for data and indexes the data. Shodan looks at ports 80 and 443, but also looks at a lot of other ports. The goal of Shodan is to connect to every IP address on the planet and ask it, “What ports have you got open?” So, some ports are VNC, which is like a remote desktop, other ports are remote desktop, some ports are Windows File Sharing, other ports are FTP and other kinds of file sharing. There’s all sorts of standard ports and non-standard ports for that matter, and Shodhan tries to find out what’s out there, what’s listening. In particular, Shodan has been used by security researchers. Once the database exists, you can ask the database, “Okay, I know that this kind of control system listens on these 7 ports. This port has the remote desktop, that port has the other thing, this port has a copy of the HMI and so on.” And now you can ask Shodan, “Which IP addresses in the world expose those ports?” Now, you connect to them and go, “Oh, look, some fool has connected a control system directly to the internet.” I thought what Jens’ says answer they’re saying when they discover these things in their jurisdiction in Germany and they go to the owner and say, “Look, you really shouldn’t be connecting your control system to the internet, this is not safe,” and the answer is, “Oh, I should shift the ports,” which means adjust the firewall so that instead of the 7 usual ports being exposed, they change the port numbers so that they are 7 unusually numbered ports, but they’re still exposed and they’re saying, “Don’t do this, this is a bad idea.”

Nate: Right. Is shifting the port’s worth anything?

Andrew: Well, shifting the ports is what they call security through obscurity, your no more secure than you used to be. If somebody knows you’ve shifted the ports, they can still attack you, all you’ve done is made the numbering the ports non-standard. So, now you have to do just a little bit more research to go, “Oh, look there’s 7 ports there and they match this control system, somebody’s mapped the ports.” Really what you need to do is stop connecting the control systems to the internet, do not let packets from the internet route into the control system and attack the control system, that’s what you got to do, not renumber things so that it looks a little bit different from everything else in the world. Looking a little different is not actually much more secure.

Nate: But then of course, does that introduce the other conversation about whether there’s any tangible benefit to connecting to the internet, or are we going to say that, in general, as a principle, you should be hiding yourself entirely?

Andrew: Well, I didn’t ask Jens that, but in my own experience, it’s a bad idea, it’s certainly a bad idea to connect control systems directly to the internet. If you need access to the control system for whatever reason, there’s ways to do it that are much more secure than, “Quick, connect directly to the internet and let every IP address on the planet send attack packets at me,” that’s just such a bad idea.

Nate: Alright, fair enough. But before we get on too much of a tangent, let’s shift gears here maybe.

Andrew: Sure. My next question had to do with regulations.

Talking about critical infrastructure in Europe, there was a directive a couple of years, the NIST directive. What is that? How does it affect Germany? How do you interact with that?

Jens: So, the NIST directive is, as you said, an over Europe directive and Germany is one of the early adopters. So, we already covered I would say 95, 97% of the installations and of the methods these installations have to be monitored. So, it’s a directive given by some State or even European body this very high level. And you have to break it down into actionable pieces, and this we already did before the directive was in place. Together with France, we were the early adopters and we were testing how we could do it. And, at the moment, it turns out that the first round is done, so meaning it was implemented in 2015, and now we are going to refine it, we’re going to rework it. And the European Commission has seen that there might be some or there could be some further changes, but at the moment, there is at least to my knowledge, not an official change planned. This is policy stuff which I’m not so good at or not, it’s boring.

Andrew: I see. I was going to ask you if you could be more specific. There’s stuff they ordered you to do, I had the impression it had something to do with you’ve got to identify sites, what was the order? It’s great that you’ve done it already, but what did you do?

Jens: Okay, okay. It was specified which sector is a critical infrastructure, meaning wastewater was a critical infrastructure. For example, food generation and food processing was, according to the European Union, not a critical infrastructure, but we made it in Germany as a critical infrastructure. Transportation systems for example, railway, airports are critical infrastructures. But if you say as started earlier with, affected people, 500,000 and how to break it down. So, if you take for example an airport, 500,000 people per day, per week, per month. So, it’s a fine-tuning here which was not specified in the European directive, but has to be done by every member State. And every member State in return has to gather incident reports and makes statistics and submit these statistics to the European Union.

Nate: Jens there hit on a theme it seems of our show which is sort of how regulatory bodies define critical infrastructure.

Andrew: That’s right. And it was interesting to see that what the kind of way that they identify critical infrastructure in Germany is, in a sense, the same as all over the world. Everyone does it a little differently, but ultimately, they all seem to be measuring consequences. Critical infrastructure is defined pretty much universally in terms of consequences to society. So, the yardstick that the BSI is using is an undesirable impact, a serious impact on a half-million people. Anything that impacts that many people or more that has a consequence for that many people are more is critical infrastructure. It’s interesting because there’s lots of ways to measure risk, consequences, exposure, but the critical infrastructure people, the regulators seem pretty consistent in terms of consequence as the measure.

Nate: Following up on that idea, if they are consistent in defining this according to consequences, what do we think about their consistency regarding where that line is? Presumably, each of these bodies or these governments has to draw a line somewhere of what consequences are deemed critical and what are sort of manageable. Did you get any sense from Jens or sort of after all of these talks with all these experts about where different people are during the lines?

Jens: That’s a good question, but I didn’t go into that with Jens. Take for instance NERPSIP in North America, I can give you an answer from the NERPSIP standard. If we talk power generation the NERPSIP standard says, “A site is low impact or medium impact or high impact, not based on the number of people affected, but rather the amount of electrical energy affected.” So, a generating site is defined as medium impact if it produces 1500 megawatts or more and it’s defined as medium impact if a cyber-attack could impair the production of 1500 megawatts or more. So, different regulators, different geographies, they use different measures, but in all cases, they’re measuring consequences.

So, you said earlier that regulation and rules is stuff that we all have to deal with, we all understand this. But you personally, the industrial control system security field, what part of it do you find the most exciting? What part do you like to stir?

Jens: For me, it’s the impact driven thing, the identifying which parts, which small piece breaks and which has a major effect which you didn’t think about it, identifying these pieces. From a background, I’m a technical guy, I love to playing around with computer systems. And finding there the small weak points which might destroy the whole system, that’s for me the most interesting part.

Andrew: Cool. Can you give me an example of 1 or 2 of these weak points? What would that look like?

Jens: We recently dealt with a patient monitor and they were going into an infinitive reboot loop if they both had the same IP address. The easy stuff, as I said, which shouldn’t have been happening, on a on a larger scale, it’s for me, I’m not decided if in most cases, it’s people. It’s the people because, the technical things, I wouldn’t say control, but you can measure it, you can find a way to stir to steer it. But people, they just do stupid things, plug in USB sticks, want to watch some movies at night during the night shift, and they are the biggest threat in my opinion to the system; and not by intention, but just because they don’t know better or they want an easy way.

Andrew: So, Jens answer here reminded me of Ralph Langner’s book, ‘Robust Control System Networks’. Ralph is a German, he basically says, “Guys, use good engineering, make the system robust.” Small mistakes like duplicate IP addresses should not completely shut down a device. Functions that are not dependent, the device functions that do not depend on the network, those functions should continue to operate normally, you should continue monitoring the patient normally. The fact that you’ve got a problem in your networking stack doesn’t mean the entire device needs to reboot. So, there’s some similarities here I’m sensing in the sort of German perspective on cyber-security.

Nate: Right. But of course, we could talk engineering for days and then there’s always the matter of people. And it seems like, anytime you talk about security, people are always going to be the wild card.

Andrew: That’s right, and Jens did talk about that. That was my next question to him, so let’s listen in.

So, is there something that the BSI is doing or you’d like to see done to help that? What’s the right step there? Is it more documentation? Is it conferences? Is it teaching the people who’ll teach the people? What is the right step?

Jens: It’s very difficult. So, we are trying everything, and at the moment, I don’t think we are having a big impact. So, it’s likely with this conference, Dale is on stage and said, “Oh, we’re talking about making systems more secure since 15 years now, and have we succeeded? No.” And I have the same impression here, but we can’t stop, we have to continue. And Germany is very proud of their engineering skills, of their quality of work, and to give these people an incentive to catch them by their honor, to catch them by their honor. And most of them, if they are given the opportunity, they will follow it, and we are now creating tabletop exercises which we are going to distribute which give you an easy playful access to this kind of evil thinking, because most people still think in use cases and not in abuse cases.

Nate: It’s kind of nice that he mentioned the words ‘honor’ and ‘proud’, I think he had ‘pride’ there, and pride isn’t usually something that I tend to hear in these sorts of conversations.

Andrew: Yeah, the Germans are rightfully proud of their engineering abilities and he’s saying there should be a way to tap into that; and I think he’s right. And he also talked about teaching people about attacks, and I’m utterly convinced that if you want to do security, you have to understand the threats, we have to understand the attacks that are coming at us. And if we understand the attacks, if you understand the threats and the risks, we can start to apply engineering expertise to understanding the problem and dealing with the problem.

Nate: We’re now reaching the tail end of your interview, let’s check back in for the final question that you asked Jens.

Andrew: This has been great, Jens. Is there a parting thought? Is there a lesson you’d like to leave with our listeners?

Jens: There are actually 2. The first is, it can be done. So, you have to try. And it won’t work the first time and things might not work as expected, but we will get there. And we will get there, not on ourselves, but together with others, meaning, find someone, someone you know and you trust, find a group, if you trust the State, go to the State, if you have some superior companies, wouldn’t sign competitor, but someone you think, “Okay, cool guy, we could work together,” just try to do that. And try to exchange information, try to exchange your best practices. And there’s this old discussion about IT and OT and the old fight. So, putting someone from IT in the OT department and someone from the IT department in the OT Department and just switching the people for a few weeks, a few months helps a lot. What people told me what was most successful is seeing the thing through the eyes of the other person.

Nate: Alright, well, before we get into another argument about IT and OT on this podcast, how about we just end it there? Andrew, thank you as always for sitting with me.

Andrew: My pleasure, Nate. I think you’re right, it’s good to end on a positive note. Jens point that, it can’t be done, we have to try, even if it doesn’t work the first time, keep going and cooperate, I think that’s a great message to carry forward.

Nate: Great. Then on that note, thank you, Andrew, thank you to Jens for sitting with us. This has been The Industrial Security podcast, I’ve been Nate Nelson, see you next time.