Lessons learned in AppSec

Notes from going down an unplanned career path

After many years of focus in Detection and Response, I got an incredible opportunity to shift my focus to Application Security. I honestly never expected to have that be a part of my career path, but when I took OSCP, I fell in love with web application penetration testing. Combine that with Portswigger’s amazing Web App Security Academy and I was completely hooked.

After some time, I have taken away some interesting lessons. None of this is particularly revolutionary, especially for seasoned AppSec pros, but I thought this might still be helpful to some.

Lessons and Things I’ve Learned

Play the prioritization game

I have a pretty standard quip I say (that probably drives my team mad at this point) but a big part of working cyber security is learning how to assess risk of a given issue and prioritize it against all your other issues. In most security teams, you’ll have more problems to deal with than time to solve them. However, you can easily end up spending a huge chunk of your time remediating an issue that has no clear path to being exploited or does not have some material impact on your security posture. This isn’t to say those things shouldn’t get done but rather that you must get really good at running this equation where you are ruthlessly focused on issues that create the most risk, which is to say - the things with the biggest impact to your company and the highest likelihood of occurring.

For instance, if you have publicly facing assets with a known CVE, that has a public exploit, that CISA has said is actively being exploited… you should probably address that before you worry about the weak cipher beg bounty report you received. The problem is, the concerns you have to tackle typically aren’t this clear cut. Usually, there is a massive amount of nuance and ambiguity. Which is why I say this is an important skill.

Some people have built systems where they can effectively score these issues and build an “engine” that can prioritize issues for them. But this can quickly turn into a massive project with a ton of administrative overhead and lots of meetings, if you’re not careful. If you’re engineering capabilities are high enough, you could theoretically create a technical solution to do this for you, but that also is a massive undertaking.

Figuring out what works for you and your organization is very tricky and if I’m honest, I tend to lean on an unstructured approach where I depend on my knowledge of hacking techniques and strong contextual understanding of the business to help me determine the overall risk of a situation. It’s obviously not perfect and I always check with my team, in case I am missing something (not uncommon). Regardless, having this be a regular part of your day to day is crucial, in my opinion.

The job isn’t just about technical prowess

One thing that I’ve learned that really stands out to me is just how much the job is about communication, education, relationship building, and sometimes compromise. When you’re working with your Engineering department, you have to really be able to talk to them in a way that supports both missions - build stuff that people will buy AND make sure it’s secure. It’s not us vs. them, but a very delicate balancing act of moving fast in competitive markets while ensuring you’re not creating excessive risk that will undermine the company’s financial success.

This, in my opinion, is the hardest part of the job. It takes a combination of hard and soft skills to be successful at this, which many engineers struggle with. Not only do you have to be capable of identifying risks and prioritizing them, you must also effectively communicate them, typically to an audience that, while technical, doesn’t have an in depth understanding of vulnerabilities and exploitation. You must effectively become a teacher in order to successfully communicate how these issues can affect a business.

I’ve seen a lot of people working in security (myself included) get irritated when the business makes a decision that does not completely mitigate an identified risk. This is where the balancing act comes in and understanding that every business has to carefully manage risk tradeoffs across multiple realms, not just security. And I think this is what makes working in AppSec, particularly for tech/SaaS businesses, so arduous at times. It can be really easy to get slanted about risk and feel undervalued because your recommended solution doesn’t get totally adopted. It takes a lot of perspective, perseverance, and selflessness to withstand this, at times.

Zoom out

A skill that will set you apart and massively increase your effectiveness is being able to zoom out and understand the big picture. Understand the infrastructure your application(s) run on, how the data flows, what integrations exist, typical coding patterns used, etc.

Knowing all of that, or knowing where to find the info, will make any engineer better at handling the topics mentioned prior to this one, especially communicating risk to the organization. If you know how data flows through an application, it’s that much easier to identify the highest risks but also makes it very clear for other audiences why the issue is so impactful.

One way to really get good at this is to know how to effectively search your code base. Documentation is rarely ever 100% accurate and up to date, but seeing code in production doesn’t lie. So being able to search your repositories in Github (or whatever you may use) is a pretty critical skill. Github Search can be easy to pick up initially for finding some things, but there’s also a ton of quirks with using it, so spending time getting familiar with it can pay huge dividends.

Coding skills are not optional

This may come off as a very obvious take, especially for Application Security, but I am still seeing a ton of people who are hopeful to get into cyber security asking if they really need to learn to code. The answer is an unequivocal yes. Even in other niches within security, knowing how to code in at least bash and/or python today isn’t really negotiable. I’m going to break down my opinion into two different perspectives - coding for general purposes in security and then what it means for working in AppSec.

General purpose - Are there jobs you can get without coding? Yes, but you severely limit your career advancement opportunities, especially if you want to have an engineering focused job/career path (i.e. where you can fetch the highest salaries, typically). I’ll be completely honest - I was very intimidated by coding early in my career. And even after I learned, I was so bad at it. As of today, I don’t consider myself an exceptional coder, but I know enough to stitch together things that work well enough. And honestly, 99% of the coding you’d do in security is usually limited to - call an API, get some data, organize that data in a way that works to solve your problem, return that data somewhere else for future use and/or to be consumed via API for another script. There are so many low cost and free resources available today there really is no reason not to dive in.

For Application Security though, it’s completely crucial to learn to read and write in the language(s) of the code base you’re working on. Even if you have a role that is only focused on pentesting, being able to understand source code will make your life so much easier. You can spend less time on attacks that won’t work and really hone in what does. And obviously, if you’re more defensive focused, you need to be able to read the code and identify problem areas. That is typically more difficult and requires more skill/knowledge, but that will come with time as you read code more often. So, you don’t need to be an expert on these languages or web development in general, but you do need a thorough understanding of how everything works in order to be effective at your job.

In addition to that, I think in the future we will see many more security engineers who started off as software developers who then switch to Application Security. They already know how the code fits together exceptionally well, so just adding security relevant training to their skillset is very easy and can be picked up very quickly. So, if you’re not starting out as a developer but you wish to be in AppSec, you’re already at a disadvantage to other candidates who did previously work as a developer.

Bug Bounty is a massively useful tool

Running a bug bounty program is one of the most effective things you can do for your application security program, assuming you’re ready for it. I’m not an expert on that part of things, so if you don’t have a program but would like one, be sure to learn more about preparing for this. It can be a huge leg up in terms of labor saved, but it does not mean that there is no labor involved.

That said, having bug bounty researchers constantly testing your application and maintaining positive relationships with them can help you scale your program tremendously. If you’re in an organization that is constantly shipping code and deploying new services (which is most these days, it seems), you’ll never be able to thoroughly test everything yourself, without sacrificing time that may be better spent elsewhere (see: prioritization).

Of course, you should have the basics in place to make sure your organization is not pushing obviously vulnerable code, but things like static analysis cannot catch everything. Once you get rid of that kind of low hanging fruit, most vulnerabilities today happen because of complex logic that has to be built into an application in order for it to function as your engineers intend it to.

Another awesome benefit of running a bug bounty program is that triaging reports yourself will actually make you a better hunter. You’ll get to see what kinds of techniques people are using and their overall level of effectiveness. It is a huge learning opportunity that can yield positive results in a lot of different directions.

That said, the big downside to bug bounty programs are the constant influx of very low quality reports. Some of them are just mind boggling. It can be a real drag to constantly triage drab reports and my least favorite ones always include the phrase “this may lead to data leaking”. But, turning a negative into a positive - this will also give you perspective that while bug bounty is highly competitive, a huge number of hunters are not going very deep.

That’s not all…

There’s obviously a ton more that can and should be said on this topic, but there is always so much more to learn. I am planning to shift this blog a little in the future to be more focused on technical details of things I am exploring, but that will come in time.

In the meantime, I’m hopeful this is at least interesting to some!