The problem with buying too many security tools
Securing a company’s tech stack and effectively mitigating risk is a balancing act. There are many different aspects of an effective security program to consider. You have to be able to effectively identify risk, implement controls, continuously monitor your controls to ensure they still solve the risk appropriately, maintain the solutions and ensure they are working as expected, etc. All the while making sure you still have enough bandwidth to solve the next risk. If your solutions are not scalable, they will hamper you down the line.
You need tools and software to help your team accomplish their goals. This in itself is walking a tightrope. Buy too many tools and you may have a bloated budget or tech debt that will be very difficult to reverse. Buy too few tools and your team may feel immense pressure and be too far outside their comfort zone.
In my experience, there is a wide spectrum of approaches security programs tend to take with regards to deploying tools. Some companies will be heavily dependent on vendor purchased software to solve issues and risks. Other companies (though, a minority) will only build their own solutions. There is no single right answer to this and every company has different needs, but I believe that too many in the industry are on the side of purchasing software and assuming that means they are safe and secure.
Security Tech Debt
If your primary means of mitigating risks identified in your environment is to purchase a vendor provided tool, you are very likely to create technical debt, or put another way, residual risk. This is because there is not a single tool in the market that can sufficiently address all risks associated with its use case. No DLP tool is flawless and may have blind spots depending on the systems you are monitoring. No EDR will give you perfect workstation security. Some phishing detection tools can be fooled and bypassed. You get the idea.
This can be called the Circle Sticker problem, as covered by Ross Haleliuk. Essentially, your attack surface grows over time as the company adopts new tech, new risks are uncovered, or new techniques to bypass typical controls are released. If you’re dependent on a vendor solution, as this attack surface grows, you will be waiting and praying for them to release new features or detections. And hopefully they don’t charge you more for that!
Even if you can avoid creating excessive technical debt in this way, you will need to have a team of people dedicated to procuring tools as novel risks are identified. The velocity with which you’ll need to purchase tools will be so high, that it will take significant time to navigate. Attending demos, executing proof of concepts, negotiating contracts, configuring and deploying the new tool, etc. are all very time intensive tasks that can easily become a full time job (or many).
Speaking of more people, you’re going to need to hire a large team to handle alerts from these solutions, because there’s going to be a ton. Even the best vendors that provide the highest quality detections are not going to be able to deliver alerts with high fidelity without a significant investment into tuning these systems. Depending on what you purchase, you may or may not be able to influence what does/doesn’t create an alert, which could leave you in a corner, unable to influence this pain point (beyond begging your vendors to update their detections).
While alert volume isn’t necessarily in the same category as tech debt, it has a similar impact. The majority of day to day work that I’ve seen in a Detection Engineering team is not creating new detections, but rather, making sure their detections only fire if absolutely necessary. Our primary goal is fewer alerts (with high fidelity), not more.
Impacts to Security Budget
One of the biggest problems security teams face today is the never ending need for more budget. Beyond the natural expansion of attack surface, a contributing factor is that most vendors that you use will be looking for ways to increase their revenue from your contract, every year.
Imagine this scenario for a second - you work for a company who is firmly of the belief that the way to do security is to -
Identify an issue
Buy a tool to solve that issue
Hire security personnel that are either less experienced, or those with very narrow, but vendor specific experience to manage and maintain the tool
Rinse and repeat
How many tools will you end up with at the end of 1, 2, or 3 years of this approach? If you’re in a fairly modern company of moderate size, I’d bet it would be dozens. The majority of new security product companies I get solicited by solve very narrow problems that only exist because of deficiencies in other tools the business needs to operate day to day (which is an entirely separate issue, for another day).
Now you’re in a situation where you have many, many vendors, all swarming to get more value for your contract, every year. The compounding effect of this can become quite expensive after a few years.
Impacts to Recruiting and Retention
Security organizations that tend toward a “tools oriented” model of mitigating risks by purchasing a tool (or a suite of tools by the same vendor) will typically look to train or hire security professionals to only work with that vendor’s stack. This usually takes form by aligning specific teams to specific tools. In some cases, this may be justified. But most of the time, this approach opens up several risks that are not only technological risks.
First, this can severely restrict the talent pool when hiring. If your expectation is that new hires have to have experience with your selected tools, you immediately remove anywhere from 50-80% of your candidate pool. Perhaps you are using a tool that has massive adoption, but even then, you are still tightening your pool. Maybe instead you hire candidates who are earlier in their career, but this still doesn’t totally solve the problem. You may increase your candidate pool this way, but chances are, you’ll need to train and develop them. This could mean just internal training, which is a fine solution in most cases assuming you have time, skill, a solid teaching structure, and patience. If you cannot provide sufficient training internally, this could result in another large financial cost. Some vendors charge a very pretty penny for facilitated training on their products.
Another balancing act emerges - give security functions and teams a wide range of tools to operate and the training burden is significant and sprawled. Give them too narrow a scope, and you may be hurting skill development.
Impact to Security Skill Development
Having a tools oriented security program could also mean that employees who desire a broad range of experience and knowledge may be railroaded into small, narrow scope roles. They may cycle through these for a while, but the best engineers want to have skill in a diverse set of domains and they may end up feeling trapped. Additionally, most engineers will desire experience in fundamental disciplines in security, not just a suite of tools.
Ambitious and experienced security engineers will understand that vendor solutions come and go, so the fundamental skills underlying the tools they use are important. Fully understanding how a security product is solving a problem means they can leverage this knowledge towards solving more difficult problems, thus boosting their careers. If a tool is solidified as the solution to a problem from on high, it can leave the engineer responsible for that solution in a position with little authority or influence, possibly resulting in less overall job satisfaction. At the very least, this setup would not likely result in highly motivated employees who feel encouraged to dig deep into problems to find novel solutions.
Being restricted to a tools oriented mindset makes it harder for employees to earn more salary over time, as they will have less marketable skill sets. What happens to their career path if you go away from a tool or vendor that is the primary focus of their job? Or what if that vendor is rapidly losing market share and no longer the standard for their vertical?
So essentially, you could inadvertently create an environment that is inherently pushing your most talented and ambitious people away from your organization, in search of more opportunity that you could just provide them yourself.
Security Engineering Culture
My belief that to solve this problem, more security teams and organizations should adopt a Security Engineering culture. You can think of this as very similar to a software engineering mindset, where you are breaking down problems into first principles and using fundamental IT, security, and coding experience to create solutions. This means hiring engineers who have those fundamental skills, which can be challenging. When I got the privilege of hiring engineers for my current team, we honed in on candidates' basic understanding of security, exploitation, and problem solving. I tried to avoid asking about specific vendors and tools, but given how we aligned responsibilities, this was not 100% avoidable. However, I did not let lack of experience with a tool prohibit me from considering them.
This does not mean that I am against purchasing tools, but rather approaching the market fully understanding the problem you wish to solve. This means that your security engineers will need hands-on experience with the problem and intimate details of exactly where the risk lies.
In a lot of cases, this can significantly increase the speed and efficiency with which the team can solve problems. If you are performing a proof of concept on a tool with poorly defined requirements (due to a lack of fundamental skills or understanding), you will end up with a protracted POC, and a product that does not solve the problem in totality, increasing the chances you have to buy a new product to supplement. Or, in worse cases, just plain rip and replace the original product, starting the process over completely. Rather, if you have a team of empowered engineers with detailed knowledge, they will test that product to its ends and ensure it provides enough of a solution to justify the cost (because no security product is perfect).
You will also have the option to forgo a purchase all together and build a solution of your own. I’ve seen a lot of cases of this comparison in action, and you absolutely want this option available to you as a security leader. This is likely to result in a solution that is significantly cheaper, even if you count the time your engineers need to spend on creating and maintaining the solution. I’ve also seen instances where the spend remains roughly the same, but the home grown solution fits the need much better than a vendor tool, increasing overall efficiency.
Though building your own solutions can be intimidating, it should be approached with more seriousness. The ability to create custom tools is increasing by the day and the complexity of deploying these solutions is rapidly decreasing. Even if you deploy a custom solution to a relatively small problem, you still forgo an entire project to purchase a tool to solve that problem. Even some security products that solve narrow problems can stack up to a large expense.
Additionally, having a security engineering mindset will open up the ability to purchase vendor tools that provide solutions with a high level of customizability. A SIEM that provides detections as code that is fully managed by the customer provides the opportunity to have highly tuned alerts to your environment. Fewer alerts means more time to reduce risk in other areas. High fidelity alerts means your incident response can get to the mitigation of an incident faster, especially in those precious moments during the early stages of an attack. More efficient detections means you can run a leaner team that costs less to operate, without burning out your employees (this too, is a balancing act, but it is very possible to win here).
Finally, this is an excellent way to attract and retain the best talent, in my opinion. The most talented security personnel I’ve worked with genuinely want to make an impact and will work hard to do so, given they are empowered to solve these problems. This can be accomplished by giving members of the team ownership over specific technical functions. An engineer tasked with managing the security of the company's email is far more likely to dive deep into the technology if they get to be the primary driver of the solution. They may still opt for a vendor tool, but again, this rarely is a silver bullet and a strong security engineer will be able to articulate exactly where a solution falls short, giving leadership a strong position from which to make decisions in the infamous buy vs build paradigm.