Inhibitors to Remediation

How to Get Out of Our Own Way

This article is based on a paper that I wrote during college where I studied improving security operations and using lessons learned to emphasize the topic of inhibitors to our ability to remediate incidents in our environments. I used a real-life (but dramatized and anonymized) example of how our SOC ended up tripping over ourselves and potentially had a breach of an endpoint, but it helped me highlight and understand where our processes went wrong and what I could do to share the message far and wide: we sometimes cause ourselves problems, so let’s open our eyes to when it happens.

What Are Inhibitors?

Before we move forward, it is important to define an inhibitor to remediation. However, if you try to research it in depth, an inhibitor to remediation is not readily defined by the security community in my research, so instead, we will break down the phrase itself and build a definition.

As defined by Merriam-Webster Dictionary, an inhibitor is one that inhibits, or prohibits from doing something, such as restraining. Remediation is the act or process of remedying, as to correct or counteract something. Combined, an inhibitor to remediation is something that prohibits the act of correcting or counteracting something, in this case, the resolution to a security incident.

The topic of inhibitors actually spawned from my time studying for my CySA+, where the CompTIA Cybersecurity Analyst+ certification retains an objective for covering the topic of inhibitors to remediation. As part of this, the book “All-In-On CompTIA CySA+ Exam Guide” by Fernando J. Maymi and Brent Chapman discusses it with the following excerpt:

“Even a solid plan for remediation that has a stakeholder buy-in sometimes faces obstacles. Many of the challenges arise from processes that have major dependencies on the IT systems or from a stale policy that fails to adequately address the changing technological landscape.” - Maymi & Chapman, 2018

The book itself does not provide any specific technical description of inhibitors beyond the paragraph, but it does give a starting point, and also provides some examples of when inhibitors arise within our people, processes, and technologies. Because of this, we can translate that inhibitors can be classified into three categories that are already in common use by the industry today, the three standard types of security controls: Technical, Operational, and Managerial.

When Our Tools Don’t Work

Covering technical inhibitors first, we can define them as:

Inhibitors that arise when tools and technologies (in general, ‘systems’) are not properly implemented, maintained, or utilized (either over or under) by an organization.

SIEM instance takes too long to run a search on a accelerated index? EPP didn’t detect that malicious word doc? Email server is down? All of these can accumulate up to technical inhibitors where our systems prevent us from doing our jobs. In order for us to correct them, we have to take into consideration this question before we continue: Is this system vital to our process or can it be circumvented by other systems we have/will have?

If the answer is no (it isn’t vital), you probably should be dropping this system from your processes over the course of a few weeks, developing workarounds and decommissioning its use, as this is a cheap and effective way to also improve operational inhibitors. If you argue that you spent millions on this tool and to simply stop using it would waste money, I’d argue that it sounds like you either had only managerial buy-in on the tool (where no one on the ground-level actually advocated for it or knew about the procurement), or the sales people misrepresented functionality. Ultimately, my message is: spending money on tools doesn’t make them effective, and should be a null answer towards justifying use of a tool. If it doesn’t work for the process, for the team, for remediation: get rid of it and stop wasting time on maintaining it and allocating analyst hours to use it when they don’t need to.

If the answer is yes (it is vital), you have two options:

1. Engage the provider/vendor of the tools and technologies and get proper implementation of the product as soon as possible to prevent these types of flaws in the future.

2. If the products can’t be corrected in a manner that is sufficient to resolve the issues, then a change management process should be engaged to begin procurement of either replacement tools and technologies or additional ones to cover the gap that the existing tools have.

When We Don’t Know What To Do

Operational inhibitors are arguably some of the most common in security operation centers around the world. When discussing the role an analyst has to perform, there are many topics that require a foundational knowledge in order for them to succeed. Defining these inhibitors is easy:

Inhibitors that come from the people who orchestrate the security mission, and can come from a general lack of understanding or knowledge, but also can be manipulated via social engineering.

While entire papers and studies have been dedicated to the improvement of the security skills gap have already been written, including this one from myself, the simple answer to this is training on the implemented tools and processes. If the analysts do not understand the tools and processes, then further self-development should be utilized. As often is the case, analysts are encouraged to gain industry certifications to improve their theoretical (things they know) and practical knowledge (things they know how to do), which subsequently improves the team and removes the operational inhibitor of just not knowing given a specific scenario.

However, while self-development is the current trend of the security industry, who are full of ‘do it yourself’ types, professional organizations also have a responsibility in training their teams. In a reflection essay I wrote a few years ago, I stated:

“… its just as vital for them to learn to do things on their own as it is for our organizations, both academic and professional, to build structures and teaching tools to help degree programs stay current and relevant, and on-the-job training flexible and meaningful… Most organizations today are […] attempting to pivot on hiring based on similar skillsets and low-level certifications that have a much easier barrier to entry to cross, instead choosing to foster a learning environment to help individuals grow on the job and receiving mentorship from more experienced security professionals, almost similar to an apprenticeship.” - Mark Rogers, 2020

There is also an overlooked inhibitor that goes beyond the SOC here. I love referencing this one because we often forget exactly what our annual security training tells us: everyone is responsible for the security of the organization. This means that even at the lowest levels, our peers have to be vigilant toward the security objectives which are orchestrated by our policies. Thus, if a user receives a phishing email and clicks on the link, they themselves became an operational inhibitor by violating policy and creating a security incident. This one is more food for thought but it is important to make that distinction here as well.

When Our Workflows Lack Flow

Finally, the most difficult inhibitors to overcome are the ones our leadership put in place. To describe them:

Inhibitors that stem from policy, procedures, and guidelines that are either too specific or non-descript, but also can be outdated and undermaintained, or not followed by an organization.

The easiest example of a managerial inhibitor is the leader who micromanages decision-making practices. Another example is when employees simply choose to circumvent policies set in place by the organization. Finally, not keeping up-to-date processes, like our SOP’s, can inhibit our ability to respond to new and dynamic situations that then requires levels of leadership be involved in a cumbersome process to develop on the spot (though arguably these situations always produce the best after-action reviews.)

Some ways to counteract managerial inhibitors should include:

• Review policies on direct oversight of SOC actions (need for approval by watch officers or third parties, especially those who aren’t working directly in daily SOC Operations), and consider indirect (auditor style) policies instead. This reduces our Mean-time-to-Resolution (MTTReso) by reducing hands involved, but still allows us to review actions taken and revert where needed to minimize impact on wrong decisions.

• Review policies on the firm separation of duties between the SOC team’s remediation requirements and I.T. systems. Requiring the mail admin to approve every email purge or a firewall engineer to approve every IP block is cumbersome and heavily increases lag-time to our incident response process. If your SOC isn’t allowed to do SOC things, you have effectively neutered your team and actually have a security advisory group that doesn’t do anything but churn alerts all day and annoy other team members with those alerts. This actually impacts service across your organization’s entire I.T. operations by causing impromtu taskings to fix problems that, otherwise, a set of permissions and an analyst could fix on their own since they’re already working the problem. Your least privilege is essentially causing you a security problem…

• Implement a SOP review process in order to keep procedures and guidelines up-to-date with the latest information, tools, and best practices. Keeping documentation current is a vital, yet often overlooked and a daunting task given the rate on how fast security information changes. To support this function, we need to be having more after-action reviews or lessons learned in our post-mortems. If we can’t be prepared to review ourselves at least once a month, then we aren’t being vigilant in ensuring we are ready when the worst happens. This should happen at every level too, from the junior/tier 1 in the trenches to the SOC Manager overseeing the battles; everyone should be having input in how we evolve our processes to be the most efficient and effective.

Takeaways and Where To Go From Here

This isn’t a complete and exhaustive list of examples but the ones referenced are top of mind based on my experiences of working in and with the SOC’s I’ve been around. If something on the list resonates with you though, what are your next steps?

Well, time to start documenting. If the organization cares about not wasting time, money, or other resources, I’m going to highly recommend you start documenting your inhibitors and how they wasted your time (analyst or engineer hours), how they are wasting other resources, like cloud computing hours, server space, etc, or how they make us vulnerable or create gaps to securing our environment. There are bonus points if you can point out how they actually violate or don’t meet the best practices or industry standards, or things your org is required to follow by their security policy, and local/state/federal regulation or law.

This documentation should be a collective effort and not an effort that can be misconstrued as 'the spiteful analyst is complaining again’. A group effort of team members agreeing and conjoining their efforts to documenting flaws for the benefit of correcting them is going to improve moral, help peer review findings, brain storm effective solutions, and impress any SOC Leadership team (unless you’ve got bad leaders who don’t want to acknowledge the problems.) This leads to the ‘brain storm effective solutions part’. We shouldn’t just complain to complain. We need to create a strategy and recommendations on how to remove the inhibitors. Our SIEM doesn’t work? Let’s look at new ones that have features that improve operations where our current one has failed. We need to update SOP’s? Let’s talk about where to update them and why.

Ultimately, we should be looking to approach the situation by breaking it down to the issues, by documenting, then creating solutions we can recommend to our leadership. Remember to incorporate improvements to Return on Investments (ROI), reduce Total Cost of Ownership (TCO), reduction in Mean-Time-To-Detection (MTTD), Mean-Time-To-Response (MTTResp), and Mean-Time-To-Resolution (MTTReso), and other metrics that our leadership look to have when they make decisions to procure, make changes, etc.