Designing and validating assessments: scope, rules of engagement and authorization

Most assessment failures are planning failures. A penetration test that knocks over a production database, a scan that floods a fragile medical device, a consultant who tests the wrong IP range and probes a neighbor's network, a report that answers a question nobody asked: each of these traces back to a missing scope statement, unclear rules or absent authorization. This lesson covers the planning that turns a test from a risky activity into a controlled, legal and useful one. It applies equally to internal tests, hired consultants and audits.

The planning documents go by many names, such as statement of work, test plan, rules of engagement, letter of authorization or engagement letter, but they answer the same questions. Why are we testing? What exactly is in and out of scope? What methods are allowed, and when? Who can authorize, who must be informed, and who can stop the test? How will sensitive data and findings be protected? What will the deliverable look like? Getting written answers before work begins protects the organization, the testers and anyone else whose systems might be affected.

Objectives and questions

Every assessment starts with an objective expressed as a question the results will answer. Good objectives are specific and tied to a decision. 'Can an external attacker with no credentials reach customer payment data?' leads to a different test from 'Does our cloud configuration meet the Center for Internet Security benchmark?' or 'Would our staff report a convincing phishing email within an hour?' Vague objectives such as 'test our security' produce long reports full of low-value findings and leave the important question unanswered.

Objectives also determine the type of test and the level of knowledge given to testers. In a black-box (zero-knowledge) test, testers start with only public information, simulating an outside attacker; this is realistic but spends much of the budget on discovery. In a white-box (full-knowledge) test, testers receive architecture diagrams, source code and credentials, which gives the most thorough coverage per hour. Gray-box (partial-knowledge) testing sits between, for example giving testers a normal user account to simulate a compromised employee. Separately, a test is announced (blue team aware) or unannounced (only a few leaders know), the latter testing detection and response as well as prevention. The exam often describes these in terms of knowledge given to the tester and knowledge given to defenders.

Scope: in, out and boundaries

The scope statement lists exactly what may be tested: IP ranges and host names, applications and their environments, cloud accounts and subscriptions, physical locations, people and processes for social engineering, and any third-party services. Equally important is what is out of scope: production systems that cannot tolerate disruption, systems owned by other parties, specific attack types such as denial of service, and any data the testers must not access. Boundaries should be explicit enough that a tester who finds a new host can tell instantly whether it is in scope; when in doubt, they stop and ask.

Third-party and cloud systems need special care. An organization can authorize testing only of assets it owns or controls. A cloud provider's shared infrastructure, a payment processor's systems or a software-as-a-service application belong to someone else, and testing them without that party's permission may breach contracts and the law. Major cloud providers publish penetration testing policies that allow customers to test their own resources within limits, often prohibiting denial-of-service testing and attacks on the provider's underlying platform; read the current policy before scoping. For a hosted service, the provider's own assessment reports, such as a SOC 2 report, usually replace direct testing.

Rules of engagement

Rules of engagement (RoE) translate scope into operating instructions. They cover permitted and prohibited techniques; testing windows (for example, outside business hours for disruptive scans, or business hours only so staff are available if something breaks); rate limits for scanning; how credentials and accounts provided for testing will be handled and removed; what testers do if they find evidence of a real compromise or illegal content (stop and escalate immediately); how and to whom critical findings are reported during the test rather than waiting for the final report; and the emergency stop procedure. They also name points of contact on both sides, with out-of-hours phone numbers.

Data handling deserves its own section. Testers may encounter personal information, health records, card numbers or trade secrets. The rules should require minimum necessary access (prove access with a screenshot of a record count, not a download of the table), encryption of any evidence held, retention limits, and secure destruction at the end. Where privacy law applies, the engagement is a disclosure of personal information to a service provider, so contracts should include the protections that law requires.

Written authorization

Written authorization is the single most important document in any intrusive test. It is signed by someone with authority over the assets, such as the system owner or an executive, and states who may test what, when, and by what means. Testers carry it, sometimes literally as a 'get out of jail' letter during physical tests, so that if a guard or the police intervene, the authority is clear. Without it, a well-intentioned tester may be committing an offense, and an employee who 'just ran a quick scan' may be disciplined or prosecuted.

In the United States, the main federal law is the Computer Fraud and Abuse Act, 18 U.S.C. 1030, which prohibits accessing a computer without authorization or exceeding authorized access, and causing damage. In Van Buren v. United States (2021) the Supreme Court read 'exceeds authorized access' narrowly, as entering parts of a system that are off limits rather than misusing information one is entitled to see. In May 2022 the Department of Justice revised its charging policy so that federal prosecutors should not charge good-faith security research under the CFAA. That policy does not bind states, which have their own computer crime laws, or private parties who can still sue. Written authorization from the owner remains the tester's real protection, and many organizations add safe-harbor language to their vulnerability disclosure policies.

Authorization must come from the right person. A department head cannot authorize testing of another department's systems, and nobody inside the organization can authorize testing of a supplier's systems. If an engagement involves social engineering of employees, human resources and legal should approve the approach, and in unionized workplaces collective agreements may also apply. For physical tests, building owners and landlords may need to consent, and guards' managers should know how to verify the letter.

Safety, change management and communication

Even authorized tests can cause harm. Scans can crash older systems, account lockout policies can lock out real users when testers guess passwords, and exploitation can leave systems unstable. Mitigations include testing in a staging environment that mirrors production where possible, taking backups beforehand, using safe scanner profiles for fragile devices such as industrial control systems and medical equipment, rate limiting, and scheduling around business peaks. Intrusive tests should go through the change management process like any other change, so the operations center knows what to expect and can tell a test from a real attack.

Communication plans decide who is told what. For announced tests the security operations center is informed, and testers' source addresses are shared so alerts can be recognized. For unannounced tests, a small 'white cell' or trusted agent knows, so that a real response can be stopped before police are called or customers are notified. Either way, the plan should say how testers identify themselves if challenged and how a suspected real incident discovered during testing is handled. A test must never mask a real attack.

# Checking a list of discovered hosts against the authorized scope
import ipaddress
scope = [ipaddress.ip_network(n) for n in ('203.0.113.0/26', '198.51.100.16/28')]
excluded = [ipaddress.ip_address(a) for a in ('203.0.113.10',)]  # fragile legacy server
found = ['203.0.113.5', '203.0.113.10', '203.0.113.70', '198.51.100.20', '192.0.2.44']
for h in found:
    ip = ipaddress.ip_address(h)
    if ip in excluded:
        status = 'OUT: explicitly excluded'
    elif any(ip in n for n in scope):
        status = 'in scope'
    else:
        status = 'OUT: not authorized, stop and ask'
    print(f'{h:15s} {status}')

203.0.113.5     in scope
203.0.113.10    OUT: explicitly excluded
203.0.113.70    OUT: not authorized, stop and ask
198.51.100.20   in scope
192.0.2.44      OUT: not authorized, stop and ask

The output shows why explicit scope matters: two of five discovered hosts may not be touched, one because it is fragile and one because it lies outside the authorized ranges. Professional testers run checks like this before every action, and well-run engagements keep an activity log with timestamps and source addresses so that any incident can be matched to, or ruled out from, the test.

Validating assessment design and quality

Before signing an engagement, a manager should validate that the design will answer the objective. Are the testers qualified, with relevant certifications and references? Does the method follow a recognized standard such as NIST SP 800-115, the Penetration Testing Execution Standard (PTES) or the Open Source Security Testing Methodology Manual (OSSTMM)? Is the time budget realistic for the scope? Will findings be risk-rated in a way the organization uses? Are retesting and a management summary included? Is insurance in place? These questions belong in procurement as much as in security.

After the test, validation continues: were all scoped items actually tested, and were limitations recorded? A good report states what was not covered and why, for example because an application was unavailable for two days. Without that statement, readers will assume coverage that never happened.