Lately, customers have been asking for a black box approach when performing a penetration test of their system/web application. During meetings, it has been clear to me that the stakeholders do not fully understand the differences between a white box and a black box approach and what benefits they provide. Therefore I would like to write down some of my thoughts on the subject, perhaps it will give an additional perspective when choosing between these approaches.

When most people think of security testing, they think of a black box approach. In a black box approach, the tester is unaware of the program internals (or has limited knowledge of them) and tests the system as an external user. The tester is trying to simulate an attack by a malicious outsider by running automated scripts and manual inspection to identify errors. A white box approach, on the other hand, is an approach when a tester has knowledge of the design of the application and tries to identify vulnerabilities in the application.

This article will try to give an overview of both approaches and will try to highlight reasons why white box is more preferred over a black box approach, depending on the goal of the test.

Throughout this article I will argue from the perspective of a web application. However, the same can be said for other systems.

The black box approach

Black box testing is a form of security testing that focuses on the functionality of an application from an external perspective, without knowledge of the internal structure or code. The benefits of this approach is that it is possible to test what kind of information can be extracted from a system, and how it can be used to uncover vulnerabilities. Because the test is being performed in a black box, the security tester and an attacker have the same information available to them. Therefore this is a good approach for simulating a real attack or a given threat actor. However, this can be a more challenging and time-consuming approach.

By having access to the internal system, security testers can more easily uncover attack paths that would otherwise be difficult to identify in a black box approach. While this is clearly a benefit in terms of cost and speed for the customer, it may not be the overall goal of the test. As previously described, there are valid reasons for a black box approach.

The white box approach

White box security testing has a number of advantages over black box testing. First, it can help identify potential vulnerabilities in the code that can be exploited by attackers. Second, it can help assess the effectiveness of security controls such as access control mechanisms and input validation routines. Third, it can help determine the impact of security vulnerabilities on the overall system security. Finally, white box security testing can provide insights into the design and implementation of security controls.

Additionally, a white box test provides more value than a black box test from a business perspective. It provides a better understanding of the system under test, which in turn leads to more comprehensive testing and generally covers a larger amount of attack paths.

Moreover, a white box approach is highly valuable early in the software development life cycle (SDLC) because it can uncover systematic issues and catch vulnerabilities early before they reach production. For example, if a system has chosen a poor cryptographic algorithm, it may be difficult to change once the system has been released. Mitigating early prevents the headache of changing code relating to the foundation of a system. It also reduces the technical debt that most development teams experience.

Meeting the needs of both sides

Both approaches satisfy different needs and goals. Each approach should be evaluated if they will cover the goal of the security test, as defined by the stakeholders. The goal should be defined with the current state of the system in mind. A system that has undergone multiple security tests which led to vulnerabilities being mitigated, can consider a black box test. Because the system has hopefully been updated with improved security controls, and relevant teams have increased their security awareness, it can be beneficial to understand what an outside attacker can accomplish with the new security controls in place.

A system that has never been tested by security professionals or has just been released, should consider a white box approach. The goal in this scenario is much broader than that of a previously tested system. By having internal knowledge of a system, the test will be deeper and provide the stakeholders with a wider set of recommendations leading to an overall increased security posture.

Conclusion

Both approaches provide different advantages depending on what the goal of the security test is to achieve. As described, testing a system which has previously been tested may not necessarily need an additional white box test. Instead, the stakeholders may be interested in how the updated system holds against an external adversary. Testing a system which has not undergone any security testing does not benefit from a black box approach in terms of cost and depth that a white box approach provides.

In conclusion, identify at which stage of the software development life cycle the system is in and what the objective should be of the security test, and make a decision on which approach to take based on that.