Commvault sees ResOps as a business model, not malware prevention/recovery mechanics

Resilience Operations is a board-level sponsored IT and security operating model for enterprises and not specific and separate IT (CIO) and security (CISO) prevention and restoration operations; that’s Commvault’s ResOps message.

Darren Thomson, Commvault’s VP and CTO for EMEA, says that the individual CIO and CISO ResOps activities are necessary but need to be co-ordinated as part of a business-level recovery plan. ResOps is a business-level activity and should not be fragmented into lower-level operations with potential gaps between them; the stakes are too high.

Thomson tells us: “I was in the insurance industry for four years. I saw this every day. Now what's going on there? Well, it's more than we don't have the right technology, although that's part of it. It actually starts with culturally, those organisations are doing a great job in their own light, but they have not been spliced together and are not collaborating on something that answers the real question. Because the real question isn't how quickly can they get your backups back? Because if those backups come back really quickly but are corrupted by malware, who cares? We still don't have a business. The answer isn't how quickly we can detect malware or how quickly we can isolate that malware because we've been briefed. It didn't work. We did our best, but we're post breach now.”

So: “What we've come up with with ResOps is a governance framework and part of what that's about is asking the right question. Within ResOps, we have this concept of mean time to clean recovery, MTCR. Now get the CEO to ask his leaders this question: “How quickly can you get my minimal viable company, the bit of the company that really matters; how quickly can we get that back?”"

“Now that question can't even be answered today because those two groups (IT and security) are operating separately. So the backup people, they would tell you all kinds of good stories about how quickly the data can come back, but you need the security vote to do the forensics to tell you whether it's clean or not.”

“Unless they're collaborating, you just don't have an answer to the MTCR question. And so my advice to boards is start asking that question. You're going to get some really disturbing answers, but it's like you've now admitted you've got a problem and now you can start to solve the problem.”

“in our experience at Commvault, the first step on that, by the way, is start running some workshops with the security folk and the infrastructure folk in a room to start brainstorming what this plan's going to look like. What does a cyber recovery plan look like as opposed to a disaster recovery plan? Because it involves skills and domains from both security and infrastructure.”

In summary; “ResOps is our governance framework as well as our platform from a technology perspective that starts to bring security and infrastructure together and starts to enable us to decide what's important, focus in on that, and create a system whereby data can not only come back quickly, but can be given a stamp of approval to say it's clean.”

Suppose, we suggested, the CISO reported to the CIO or vice versa. Wouldn’t there be separation problem go away? Well, yes, in theory, but such reporting structures are not the usual thing.

Thomson said: “We see all kinds of permutations of this. So I'd say the most common is that they're separate still. In many cases, the CIO is on the board and the CSO isn't, that's wrong in my view, but that's often the case. Sometimes they’re peers on the board,. Sometimes one reports to the other. There's a problem with that, which I'll come back to. I've seen that not work." 

"Very often and increasingly what we're seeing is the emergence of a new role, which is something like the Chief Risk Officer or the IT Risk Officer or someone, and those folks don't necessarily report to the Risk Officer, but the Risk Officer at board level has responsibility for this. So they are asking the right question, they are encouraging that collaboration and they are asking business questions as opposed to technical questions. And we see all permutations of those and others because this is a work in progress.”

This led us to suppose that, unless there was experience of a massive cyber attack at board level and CIO/CISO level then the organization would not perceive it had a problem. We posed this notion to Thomson: “If you suggest to them that let's run through a simulated cyber attack, let's put your senior affected people who are going to be affected by this in a single room and let's run through a cyber attack scenario and see what happens within your organization's constraints. Is that something that you'd be proposing to them?

Thomson was ahead of us: “What you're describing is something that's called Minutes to Meltdown. It's an initiative that we've been running for two years now. It's actually a marketing-led initiative. Our head of marketing, Anna, cites it as being the marketing initiative with the greatest return the company has ever seen.”

“What it means essentially is we do it in great locations with the smoke, lights, and drama required. … We put executives in a room together. We try and make sure there's a good mix of security and infrastructure folk. Preferably we're sponsored by somebody [at] board level.and we put them through three hours of hell in a nice way.”

“Basically we put people through three hours of ransomware attack. It's not a technical tabletop exercise, it's almost purely emotion. My role typically in those exercises is to ask a bunch of questions. We put them through some scenarios. One of the scenarios is, right, the hacker wants to talk to somebody. Who's going to talk to the hacker? Is that the CEO? Is that the CSO, somebody else? We go through that.”

And: “By the end of the three hours … what’s lovely to watch is you start to have a conversation between the security folk and the infrastructure folk about what good might look like here. How can each of those groups contribute to something that would fix this problem?”

In a way it's a team building exercise.

Thomson agreed: “It really is. Often these people have never met and they're talking different languages at the beginning and talking similar languages at the end. Essentially nobody comes out of that room thinking, "Oh, we've got this. This is not a problem for us." I've never seen that.”

With that foundation laid then: “Now we can start to think about the planning. What might the team look like? How might this come together as a programme of work? What would the plan look like? What is its relationship to disaster recovery? Because clearly it's related, but it's not the same thing. Well, technology would enable it and clearly Commvault has got a vested interest there.”

He added: ”It's a great way of getting this going. But I'd like to think that even if Commvault were not engaged after that exercise, something happened in that organisation. They're now having a different conversation and I think that's almost definitely true. We get fantastic feedback from this event.”

At an event: “I actually had one of the executives from Maersk in one of my sessions. They went through one of the biggest cyber attacks in human history. So they had a few things to say and it was very positive, I think. And … we gave them the stage for a while just to talk because the value there is immense for the others in the room.”

Thomson emphasized one point: “We don't touch Commvault product in these sessions. We don't mention it. It really isn't the point of the session. The point of the session is for them to have a moment whereby they go, 'We have a problem and we know that problem is going to be driven by culture and people and process and there's going to be some necessary disruptive technology for us to get this right.' And that's as far as we go. Really it’s like we're creating an impetus for change.”

He said: “There are a couple of takeaways that the audience will get from us, which is, and nothing has to do with our product, but everything to do with things like, well, look, by the way, if you've got a plan that isn't tested, that's no plan at all. … wouldn't it be great if ,actually, that testing discipline came pre-breach, came before we had a problem, not after it.”

There is a way to do this: “The ability to build infrastructure as code; we can take infrastructure, translate that to code, build that in the cloud. Technology like air gapping, the ability to create a logical and physical segmentation, again, leveraging cloud in order for us to stand up what we would call a clean room to do the testing.”

“The ability to do live malware scanning on your backups, massively disruptive technology that helps us with this. AI, massive; part machine learning, the ability to learn how to put what we call run books together to build these infrastructures dynamically based on when we need them. All of this translates to we can now build a testing discipline that's constant, it's continuous,” for the minimum viable company infrastructure.

“We're starting to see this happening in the build now for people to be constantly testing the recovery of that. So every time there's a business change, every time there's some sort of change to the threat landscape, every time we have an application, the test evolves and the plan evolves so that when, heaven forbid, the breach actually happens, we're pretty good.”

Companies can, if they have a good MTCR for their minimum viable company infrastructure, consider not acquiescing to ransom demands. And the less money reward for cyber attacks then the less likely they are to happen. 

Commvault’s ResOps concept seems like a darn good idea.

Comment

Cohesity runs Ransomware Resiliency Workshops along similar lines to Minutes from Meltdown.