Pros and Cons of Different Forms of Technical Security Assessments Including VA DAST AI PT YMMV

Let’s start by defining our terms:

VA (Vulnerability Assessment)

This is a fully automated scan which seeks to identify known vulnerabilities that are present.  Common tools for this are Nessus, Qualys and OpenVAS.

DAST (Dynamic Application Security Test)

This is an automated assessment of a web application.  It’s like a vulnerability scan, but with more ability to authenticate and navigate an application.  Common tools for this are Burp, ZAP, and AppCheck.

AI (Artificial Intelligence)

This is a newer type of automated assessment, which uses artificial intelligence (AI) to replace the consultant/tester.  It aims to replicate a penetration test.

PT (Penetration Test or Pen Test)

This is an assessment where a consultant/tester makes use of automated and manual tooling to conduct a technical security assessment.

YMMV (Your Mileage May Vary)

URM recognises that each of these services differ, and that they each have their place in maintaining an organisation’s security posture.  However, care should be taken to ensure that the service selected fits the organisation’s objectives.

Now that we know what each term means, we should look at the differentiators of each type of assessment. Each of them has their advantages and drawbacks.

Vulnerability Assessment

VAs are a key part of security assessment.  Due to their relatively low cost and convenience factor, they can be run regularly to ensure that an organisation’s security posture is maintained.

The drawback to vulnerability assessments is their coverage.  Being fully automated, they perform a pre-defined series of checks and tests, expecting specific responses in order to identify a vulnerability.  This leads to false positives (things reported as vulnerable erroneously), as well as false negatives (where things that are vulnerable are not reported).  Between these two, it can take quite some time to validate the findings present in a VA report.

Finally, due to the fully automated nature of VAs, VA reports are solely technical findings.  They often fail to discuss impact, and won’t group or contextualise findings in order to aid the organisation in understanding or remediating findings.

Dynamic Application Security Test

DASTs are ultimately a more focussed form of VA that solely target web applications.  Their capabilities are improved over a VA when assessing applications, as they can authenticate to applications and have more thoroughly defined sets of tests to perform.

Just like a VA though, DASTs are prone to false positives and negatives, and their report will be machine generated.

Artificial Intelligence

AI testing makes use of the above assessments, but uses some form of artificial intelligence to try and extend the testing performed, as well as aid in the quality of the report.

The effectiveness of AI assessments varies wildly by vendor, and, as a new and expanding market segment, marketing is running rampant with wild claims.  Pricing can also be surprisingly high for AI-based assessments given their automated nature.

Based on URM’s visibility of AI guided testing, it should be viewed with similar scepticism as self-driving cars. A great idea, still in its infancy, and probably a long way from being ready for widespread use – despite what marketing teams may claim!

Penetration Testing

Penetration testing is the assessment that takes the most time from an organisation to conduct; scoping calls need to be performed, pre-requisites need to be provided, and the consultant may need technical assistance during the test.  Then, once the test is completed, there is a delay whilst a report is written and goes through a quality control process.  This, combined with the (sometimes) higher cost, means that pen tests are performed less frequently than fully automated testing.

However, the time investment from the organisation allows a human expert to better understand the scope of the test and perform checks and tests in context (even linking multiple tests together), resulting in security findings that automated tests (even with the guidance of AI) cannot identify.

The consultant will make use of the automated testing types identified above, but will validate the false positives and negatives, ensuring that the findings present in the report are as accurate as possible.  The report will also contain more than just the technical findings, with information about business impact, areas of general weakness/strength, and interpretive language, which makes the report easier to read and quicker to action.

‍

‍

Hopefully the above has provided some insight into the different types of technical security assessment available.  Beware of marketing or sales tactics which claim to offer all of the benefits, but none of the drawbacks – as the old adage goes: if it sounds too good to be true, it probably is.

By utilising the above types of assessment appropriately, organisations can gain good visibility of their security posture and maintain that posture, all whilst minimising time and financial expenditure.