Penetration testing – or pen-testing as we colloquially call it – is a crucial component to a robust security programme in any organisation. As management, it’s critical you understand where pen-testing fits into your programme and what it can do for your organisation. Equally important is what it can’t do, and how the different “flavours” of pen-testing impact what we the testers need from you, and the outcomes of the testing.

In this two-part series, we’re going to explore both black box and white box testing, where they differ, and when to use one over the other. Before all that though, we should run through penetration testing so we’re all on the same page.

Pen-testing is the art of simulating how a threat actor would break into your product or infrastructure. It sits in between a vulnerability assessment (which only attacks your product and not the environment it sits in) and a red team – a full-blown assault on your entire organisation with specific mission-critical components as the objective, in order to test your resilience to persistent cyber threats and stress-test your security operations centre.

You then take results of the pen-test and reduce your organisation’s attack surface – and therefore cyber-risk – by applying the accompanying recommendations.

A pen-test is defined by a scoping document: a legal agreement between us and our client to define exactly the applications and infrastructure we’re allowed to investigate. This could include the allowed URLs, the IP addresses, or the backend servers and databases which we can interact with, which, under typical circumstances, would be considered hacking under Australian law. It’s important to get this document right as either party can land in some serious hot water if it’s wrong.

Under the umbrella that is pen-testing, there fall several different categories, most notably black box and white box. Grey box is somewhere between the two, but not often used in the industry (if it’s not truly white box, we often default to calling it black box).

Black box penetration testing is pen-testing where all the testers know is what is defined in the scope: no source code, configurations, documentation, nor access to the development team. By definition, black box means “anything that has mysterious or unknown internal functions or mechanisms.” When testing black box, we don’t have access to documentation of the internals of the application, nor the configurations of the infrastructure.

While simply reviewing source code, documentation, and configurations are never enough on their own, they can help us answer questions we may have more quickly than the guesswork required with black box testing. Some things stand out as glaring issues in source code, saving us time and providing better value for money. Some things are hidden without knowing how to find then; source code and configurations often act as a map. Some test cases are even straight up dangerous to check without the proper knowledge to facilitate the testing, and sometimes we just can’t know what is or isn’t going to be risky.

Testing black box also often reduces the level of assurance testing can provide. As the Open Web Application Security Project (OWASP) point out in their Application Security Verification Standard (ASVS), once you move past low assurance applications to ones that deal with sensitive data or are mission-critical systems, black box pen-testing alone doesn’t cut it. You have to start looking at white box testing to find the hidden/out of reach vulnerabilities that can still lead to serious breaches and break-ins.

Some say that black box testing is good since it’s close to how a real threat actor would attack your systems. However, the major issue with this statement in this context is that we as pen-testers have limited resources to allocate to an engagement.

The client’s budget and priorities heavily dictate what techniques we can use, how much time we can spend on an engagement, the number of people we can put on a job etc. In contrast to this, there’s no theoretical limit to the resources a dedicated threat could allocate to their activities. Buying zero-days, having very large teams, and spending 12+ months trying to compromise your network isn’t unheard of for sophisticated threat actors.

As a consequence of reducing the level of assurance that the pen-testing provides and the limited resources we can allocate to an engagement, black box testing limits the class of attacker it defends against. It takes exponentially more time to thwart high skilled attackers such as criminal organisations and APTs if the only risk assessment approach is black box testing whereas with source code and documentation we can not only find problems quicker, there are often obscure vulnerabilities that may not be found without “following the code” so to speak.

But what does a black box penetration test look like from the client’s perspective? Well, a black box pen-test requires much less input and resources from the client; typically, we just need a well-defined scope, credentials to access any required resources, and in the case of black box web application penetration testing, an exception to any Web Application Firewall (WAF) that may in place (remember that we have limited time to bypass a WAF compared to threat actors).

Black box penetration testing is a type of pen-testing that involves going in blind: often the only thing we know is what URLs and resources we’re allowed to test. While it’s less involved for a business to organise, the downsides include a reduced level of assurance compared to other forms of pen-testing; it leaves you less protected from advanced threat actors, and provides you with less bang-for-your-buck. That said, it provides a simple way to add some level of security to products and infrastructure.