Log reviews, monitoring and synthetic transactions

Logs are the memory of an information system. They record who signed in, what changed, which requests failed and when a service slowed down. Without them, an organization cannot investigate an incident, prove compliance or notice that a control has stopped working. With them, but without anyone reviewing them, it has an expensive archive that tells the story only after the damage is done. Sub-domain 6.2 of the outline names log reviews and synthetic transactions as testing activities, and 6.3 treats log data as part of the security process data that management collects.

This lesson covers log review as an assurance activity: what to log, how to protect and retain logs, how to organize reviews so they find problems rather than generate noise, and how synthetic transactions and real user monitoring test whether services and controls behave as users and attackers would experience them. It stays at the level of program design and evidence. Detection engineering and incident response are treated in the security operations material and in ITIL-140 for service monitoring.

Why log review is a control test

Reviewing logs tests controls in two ways. First, it tests whether preventive controls are working: if the access control system is supposed to block logins from disabled accounts, a review that finds successful logins by disabled accounts has found a control failure. Second, log review is itself a detective control that must be tested: auditors ask whether reviews happen as scheduled, whether the reviewer had the skills and independence to spot problems, and whether findings were followed up. A signed checklist saying that logs were reviewed, with no evidence of what was examined, is weak evidence.

Deciding what to log

Logging everything is neither affordable nor useful. A logging standard should start from the risks and controls that matter and define, for each class of system, which events must be recorded. Common baselines include authentication successes and failures, use of privileged accounts, changes to accounts, groups and permissions, changes to security configuration and to the logging itself, access to sensitive data, and errors that indicate attacks or failures. NIST SP 800-92, Guide to Computer Security Log Management, remains the foundational reference, and NIST has published draft and revised guidance on log management planning (SP 800-92 Revision 1 was released in draft in 2023; check its current status).

Each log record should answer the classic questions: what happened, when, where, by whom or by which process, from which source, and with what outcome. Time is critical. If servers disagree about the time by even a few minutes, reconstructing a sequence of events across systems becomes guesswork. Clock synchronization to a trusted time source, and recording time zones or using UTC consistently, are prerequisites for meaningful review.

Protecting log integrity

Logs are evidence, and evidence is worthless if it can be altered. An administrator who can delete the record of their own actions defeats the accountability that logging is meant to provide. Good practice separates duties so that system administrators cannot modify central logs, forwards logs promptly off the system that generated them, restricts access to the log platform, stores logs on write-once or immutable storage for the retention period, and monitors for logging being disabled. Hashing log files or using a platform that records integrity information allows tampering to be detected. A sudden gap in a system's logs is itself an event worth investigating.

In Canada, retention periods come from several sources. Federal departments follow Treasury Board policy and the Library and Archives of Canada Act for records; the Canadian Centre for Cyber Security recommends retaining security logs long enough to support investigations, commonly at least a year. Organizations subject to PIPEDA must keep a record of every breach of security safeguards involving personal information for 24 months after determining the breach occurred, which in practice depends on logs. In Quebec, Law 25 requires a confidentiality incident register. Sector regulators such as OSFI expect logging adequate to detect and investigate technology and cyber incidents under Guideline B-13.

Organizing log reviews

Effective reviews are scheduled, scoped and recorded. Scheduled means the frequency matches the risk: privileged activity on critical systems may be reviewed daily through automated alerts and weekly by a person, while lower-risk systems are reviewed monthly. Scoped means each review has a defined question, such as whether any administrator account was used outside approved change windows. Recorded means the reviewer documents what was examined, the time range, what was found and what action followed, so that an auditor can re-perform or at least understand the review.

Automation does the heavy lifting. A security information and event management (SIEM) platform collects and normalizes logs, applies correlation rules and produces alerts and reports. Rules encode the review questions: a disabled account with a successful login, a change to a firewall rule outside the approved window, more than a threshold of failed logins across many accounts from one source. Human reviewers then examine what the rules surface and periodically look at raw data for patterns the rules do not cover.

import re
from collections import Counter
# Synthetic authentication log lines (illustrative format)
log = '''2026-10-12T02:14:03Z INFO auth user=amira src=10.1.4.20 result=success mfa=yes
2026-10-12T02:15:11Z WARN auth user=svc_backup src=203.0.113.9 result=failure mfa=no
2026-10-12T02:15:12Z WARN auth user=svc_backup src=203.0.113.9 result=failure mfa=no
2026-10-12T02:15:14Z WARN auth user=svc_backup src=203.0.113.9 result=failure mfa=no
2026-10-12T02:15:15Z WARN auth user=svc_backup src=203.0.113.9 result=failure mfa=no
2026-10-12T02:15:17Z WARN auth user=svc_backup src=203.0.113.9 result=failure mfa=no
2026-10-12T02:15:19Z INFO auth user=svc_backup src=203.0.113.9 result=success mfa=no
2026-10-12T08:01:44Z INFO auth user=jlee src=10.1.6.31 result=success mfa=yes
2026-10-12T09:30:02Z ERROR auth user=former.contractor src=10.1.9.4 result=success mfa=yes'''
disabled = {'former.contractor'}
pat = re.compile(r'(\S+) (\w+) auth user=(\S+) src=(\S+) result=(\w+) mfa=(\w+)')
fails, findings = Counter(), []
for line in log.splitlines():
    ts, level, user, src, result, mfa = pat.match(line).groups()
    if result == 'failure':
        fails[(user, src)] += 1
    elif result == 'success':
        if fails[(user, src)] >= 5:
            findings.append(f'{ts} success for {user} from {src} after {fails[(user, src)]} failures')
        if user in disabled:
            findings.append(f'{ts} success for disabled account {user}')
        if mfa == 'no' and not src.startswith('10.'):
            findings.append(f'{ts} external sign-in without MFA for {user}')
for f in findings:
    print('FINDING', f)

FINDING 2026-10-12T02:15:19Z success for svc_backup from 203.0.113.9 after 5 failures
FINDING 2026-10-12T02:15:19Z external sign-in without MFA for svc_backup
FINDING 2026-10-12T09:30:02Z success for disabled account former.contractor

The three findings illustrate how log review tests other controls. A burst of failures followed by success suggests a password-guessing attempt that succeeded, which tests the lockout and password controls. An external sign-in without multifactor authentication tests the conditional access policy, and here shows a service account exempted from it. A successful sign-in by a disabled account tests the deprovisioning process. Each finding becomes an incident ticket, a control deficiency or both.

Clipping levels and noise

A clipping level is a threshold below which events are recorded but not alerted, because isolated occurrences are normal. Three failed logins in an hour is ordinary; fifty is not. Setting clipping levels is a balance: too low and reviewers drown in alerts and start ignoring them, too high and real attacks pass under the threshold. Clipping levels should be documented, justified by data, reviewed when systems or user behaviour change, and tested by occasionally checking what fell below them.

Synthetic transactions and real user monitoring

Logs record what happened. Synthetic transactions test what would happen, on demand, by running scripted actions against a system as a user would: loading a login page, signing in with a test account, adding an item to a cart, checking a balance. Synthetic monitoring runs these scripts on a schedule from different locations and measures availability, response time and correctness. Because the script and its expected results are known, any deviation is meaningful: the page is down, slow, or returns the wrong content.

Synthetic transactions also test security controls. A scheduled script can confirm that a disabled test account is rejected, that an administrative interface is not reachable from the internet, that a page requires authentication, or that a security header is present. These are positive tests of controls that would otherwise be checked only during an annual assessment. Real user monitoring (RUM) complements synthetic monitoring by measuring the experience of actual users through instrumentation in the application, revealing problems that scripted paths miss, but it cannot test what users have not yet done.

import random
random.seed(2026)
# Synthetic transaction: a login journey checked every 5 minutes for 30 days
checks_per_day = 24 * 12
results = []
for day in range(30):
    for k in range(checks_per_day):
        ok = random.random() > 0.0015          # rare random failures
        if day == 17 and 80 <= k < 104:      # a two-hour outage on day 18
            ok = False
        results.append(ok)
up = sum(results)
avail = up / len(results)
print(f'checks run: {len(results)}  failed: {len(results) - up}')
print(f'measured availability: {avail:.4%}')
for target in (0.999, 0.9995):
    allowed = (1 - target) * 30 * 24 * 60
    print(f'target {target:.2%}: allowed downtime in 30 days = {allowed:.1f} min -> {"met" if avail >= target else "missed"}')

checks run: 8640  failed: 38
measured availability: 99.5602%
target 99.90%: allowed downtime in 30 days = 43.2 min -> missed
target 99.95%: allowed downtime in 30 days = 21.6 min -> missed

The calculation shows how synthetic monitoring produces evidence for a service-level commitment. One two-hour outage was enough to miss a 99.9 % target, which allows about 43 minutes of downtime in 30 days. A manager reviewing these results would ask whether the outage was detected promptly, whether the cause was found and whether the commitment, the architecture or both need to change.

Reporting log review results

Log review results feed the reporting covered in lesson 11. Useful measures include the percentage of in-scope systems sending logs to the central platform, the age of the oldest unreviewed high-priority alert, the time from alert to triage, the number of findings by category and the percentage closed within target. Coverage matters as much as volume: a review program that works well for the systems it sees, but sees only 70 % of critical systems, has a 30 % blind spot that belongs in the risk register.