IT Audit - The Institute Of Internal Auditors  


Reference Library

Discussion Board

Contact the Editor

About ITAudit

Authors Wanted


Previous Issues

print Print Article

Beyond Vulnerability Scans — Security Considerations for Auditors

September 15, 2005
Craig Wright, Director of Computer Assurance, BDO CAS, CISM, CISSP, ISSAP, ISSMP


Poorly conducted penetration tests and ethical attacks may not fully identify security vulnerabilities in IT systems, leaving organizations exposed to external threats.


External vulnerability scanning techniques, such as penetration testing or ethical attacks, are used by many organizations to identify security vulnerabilities in a computer network. Unfortunately, these scanning techniques do not focus their analysis on high-end security problems, such as determining internal control failures that could lead to a security breach. Rather, they often report vulnerabilities without providing enough information about the problem or fail to detect system flaws, even though internal controls may not be working properly. To help IT departments better detect security risks, internal auditors need to ensure organizations that use external scans do so as part of a complete audit program and not as their sole vulnerability detection method.


Recent compromises of credit card companies exemplify the need for organizations to properly assess vulnerabilities that may lead to security breaches. To determine the effectiveness of their security practices, some organizations rely on external vulnerability scans such as ethical attacks to simulate security breaches and identify points of entry to an organization's IT systems.

When properly conducted, penetration tests and ethical attacks help organizations reduce the total number of security vulnerabilities in networks and other IT systems. However, the problem arises when organizations use automated vulnerability tests that are not advanced enough to provide consistent significant results, are high in cost, or are used exclusively as the only detection mechanism. In addition, some external vulnerability scans don't identify all security problems in an enterprise's IT systems, require an expert analysis to be used meaningfully, and may lead organizations to have a false sense of security. For example, ethical attacks sometimes generate false positives by identifying system vulnerabilities that don't really exist. In addition, ethical attacks generally do not test the organization's entire attack space for security gaps — they only test single points within that space — or often find security gaps IT departments already know about.

Finally, many organizations have the misconception that external vulnerability scans simulate real external attacks all the time. Any service, audit or otherwise, is limited by organizational constraints such as time, which may not apply to an external attacker. As a result, most external vulnerability scans do not simulate real attacks properly. For example, many successful attacks by hackers are planned over a period exceeding six to 12 months. This level of testing may not be feasible for many organizations from a budget perspective.


Because of the problems associated with external vulnerability tests, audit assessments provide a better alternative to determine an organization's level of security and whether or not internal controls are working properly. What's more, different studies have documented the problem with external vulnerability scans and the advantages of internal audits. In a study conducted at the University of New Haven, U.S. computer security expert Dr. Fred Cohen found that penetration tests such as ethical attacks bypassed security vulnerabilities, even when IT controls were not operating correctly. Furthermore, researchers at the University of Newcastle in Australia published a study in 2005 — "Exposing the Inadequacy of Ethical Attacks in Order to Reinforce the Need to Implement Good Audit Practice" — which found that penetration tests only identified 13.3 percent of the total number of high-level vulnerabilities on the systems tested. However, a comprehensive audit assessment on the same systems identified nearly 97 percent of the high-level vulnerabilities (see graph 1 below).

New Graph

Graph 1: High-level vulnerability test results. (The results where tested at a 99 percent confidence interval.)

Besides identifying system flaws more effectively than external vulnerability scans, internal audits help executives and senior managers understand the organization's infrastructure and associated risks, as well as produce a "security roadmap" that shows the organization's key IT security areas. On the other hand, external vulnerability scans identify some of the problems, but fail to report all significant risks. For example, a recent security issue reported by The SANS Institute is the use of Secure Shell (SSH) Trojans — malicious code designed to act as the original SSH protocol to gain unrestricted access to computer networks. SSH is a program that provides strong authentication and secure communications over unsecure channels by allowing users to log into a networked computer, execute commands in a remote machine, and move files from one machine to another. A comprehensive audit would check the software versions and controls of all key external services such as SSH to make sure they are up-to-date, including the program's hash code, which cannot be replicated mathematically. Thus, if an attacker alters the SSH program, its hash code will change also. If an ethical attack of the system is performed, the report would find nothing wrong with the hacked SSH program. An audit, on the other hand, would detect and report the security problem by identifying the changed hash code.

In the credit card industry, some of the world's major credit card companies joined forces to tighten e-commerce security through the establishment of the Payment Card Industry (PCI) Data Security Standard. As part of its requirements, the PCI mandates that online vendors implement one primary function per server only. However, many systems are configured with multiple Internet protocol (IP) addresses and assign only one virtual server to each of the IP addresses on the system. Because an ethical attack would be unable to differentiate between the system and the virtual server, it would report each online transaction as taking place on a separate server. Conversely, an audit would detect that a single host is running multiple virtual servers and report the security breach.

Finally, external vulnerability scans are commonly used in organizations during random blind tests, or attacks that are conducted with zero information about the systems tested. Studies have found repeatedly that random blind testing does not help to determine a system's security level (for more information, refer to the end of the article). Unfortunately, most ethical attacks use random blind tests. To perform more detailed risk analyses, auditors need to have as much information about the systems being audited as possible. By definition, open, "white box-style" audits (i.e., when detailed knowledge of the systems is supplied to the tester) deliver a better result than “black box-style" assessments (i.e., when testers have to discover the details of the systems they are auditing), commonly used during ethical attacks. Both audits and ethical attacks require expert knowledge and time. However, audits provide the best basis to determine if an organization is securing its IT systems properly by identifying that all IT controls are working according to established rules and guidelines.


The use of external vulnerability assessments is not unproductive; detecting IT security risks to reduce the total number of vulnerabilities in a network provides an easily measured result. The problem arises when organizations do not know what their security levels are or the effectiveness of the IT controls used to secure an organization's network. Unfortunately, penetration tests and ethical attacks are unable to provide this level of detail. Because internal audits provide more in-depth information of a system's security vulnerabilities, they are more accurate than external scans. Audits also allow organizations to take a risk-based approach to IT security and prepare for external attacks more effectively.

To ensure organizations are properly protected from external attacks, internal auditors must work with chief security officers or individuals in a similar role to make sure organizations are conducting assessments that detect all vulnerabilities and are testing internal controls properly. To detect security loopholes and test internal controls more accurately, organizations should use audits as the principal method to review their security efforts.

Before IT departments employ ethical attacks or penetration tests to detect vulnerabilities, auditors should recommend organizations take into consideration the methodology's limitations. In addition, IT departments should use external vulnerability scans as way to gauge their IT process vulnerabilities only, but not to determine how secure the organization is from attacks. To do this, organizations should use the skills of an internal auditor trained in this area. When combined with audit assessments, ethical attacks provide organizations with valuable information regarding their security levels and "hot areas." However, in organizations with scarce financial resources, audits will prove to be a far better and more thorough alternative to an ethical attack or penetration test.


To learn more about penetration tests, read Kenneth van Wyk's article, “Finding the Elusive Value in Penetration Testing.”

Fred Cohen has written numerous articles on blind testing:

  • Managing Network Security, Penetration Testing?” Cohen, Fred (1997).
  • “National Info-Sec Technical Baseline, Intrusion Detection and Response.” Lawrence Livermore National Laboratory, Sandia National Laboratories. Cohen, Fred (1996).