Logging, monitoring and detection

Most security incidents are discovered through monitoring: a rule fires, an analyst notices a pattern, or a system reports an anomaly. Some are discovered much later, by a customer, a journalist, a regulator or a ransom note. The difference between those two outcomes is largely the quality of an organization's logging and monitoring program. Sub-domain 7.2 of the CISSP outline, conduct logging and monitoring activities, covers the tools and processes that turn raw events into timely detection: security information and event management, continuous monitoring, egress monitoring, log management, threat intelligence, user and entity behaviour analytics and security orchestration.

CISSP-115 dealt with log review as a control test: deciding what to log, protecting log integrity, clipping levels and organizing periodic reviews. This lesson takes the operational view. It looks at how a monitoring program is built and run day to day, how much it costs to collect and keep data, how alerts are triaged, how analysts' capacity limits what can be monitored, and how intelligence and automation help a small team keep up.

From events to incidents

Monitoring deals with a funnel. At the top are events: any observable occurrence, such as a log-in, a file access or a network connection. Large organizations generate billions a day. Detection rules and analytics select a small fraction of those as alerts, which are events or combinations of events that may indicate a problem and deserve human or automated attention. Triage examines each alert and decides whether it is a false positive, a benign true positive (real activity that is allowed), or a true positive that warrants investigation. Some investigations confirm an incident: an event that actually or potentially jeopardizes the confidentiality, integrity or availability of information or systems, or violates security policy.

Each step in the funnel is a design choice with trade-offs. Too few rules and real attacks pass unnoticed; too many and analysts drown in alerts, start closing them without proper review, and miss the one that mattered. The craft of security operations is tuning the funnel so that analysts spend their limited time on alerts with a reasonable chance of being real.

SIEM, log management and data pipelines

A security information and event management (SIEM) system collects logs and events from many sources, normalizes them into a common format, correlates related events, applies detection rules and presents alerts, dashboards and search tools to analysts. It also supports compliance reporting and retention. A log management platform may handle collection, storage and search more cheaply for high-volume sources, forwarding only security-relevant data to the SIEM. Many organizations now use a data pipeline layer that filters, enriches and routes logs before they reach expensive storage.

Coverage matters more than tool choice. A SIEM that receives logs from only half the environment gives false comfort. Security operations should keep an inventory of log sources mapped to assets and to the detection use cases they support, and should monitor the monitoring: alert when an important source stops sending data, because silence from a firewall can mean failure or tampering.

How much data, kept for how long?

Retention requirements come from several directions. Legal and regulatory rules set minimums for some records; investigations need enough history to establish when an intrusion began, and studies of real incidents regularly find dwell times of weeks or months; and storage costs push in the other direction. A common pattern is tiered retention: recent data, perhaps 30 to 90 days, is kept in fast, searchable storage for daily detection, while older data is kept for a year or more in cheaper archive storage that can be restored for investigations.

Capacity planning starts with simple arithmetic: events per day multiplied by average size multiplied by retention days, adjusted for compression and replication. The numbers grow quickly, and they drive both licence costs (many SIEMs charge by volume ingested) and storage costs.

# Log retention sizing for tiered storage (illustrative volumes)
sources = {           # events per day, average bytes per event
    'endpoint EDR': (180_000_000, 600),
    'firewall':     (250_000_000, 350),
    'DNS':          (400_000_000, 200),
    'identity':     (12_000_000, 900),
    'cloud audit':  (20_000_000, 1_200),
}
hot_days, archive_days = 90, 365
compression = 8          # archive compresses about 8:1
replicas = 2             # hot tier stored twice for resilience
raw_day = sum(n * b for n, b in sources.values())
print(f'Raw data per day: {raw_day / 1e9:,.1f} GB')
hot = raw_day * hot_days * replicas
archive = raw_day * archive_days / compression
print(f'Hot tier ({hot_days} days, x{replicas}): {hot / 1e12:,.1f} TB')
print(f'Archive ({archive_days} days, {compression}:1): {archive / 1e12:,.1f} TB')
for name, (n, b) in sorted(sources.items(), key=lambda kv: -kv[1][0] * kv[1][1]):
    print(f'  {name:12s} {n * b / raw_day:5.1%} of daily volume')

Raw data per day: 310.3 GB
Hot tier (90 days, x2): 55.9 TB
Archive (365 days, 8:1): 14.2 TB
  endpoint EDR 34.8% of daily volume
  firewall     28.2% of daily volume
  DNS          25.8% of daily volume
  cloud audit   7.7% of daily volume
  identity      3.5% of daily volume

The result shows why data pipelines matter. Endpoint, firewall and DNS logs make up almost 90 % of the daily volume, yet many of their records are routine. Filtering duplicate or low-value records before ingestion, or keeping high-volume sources in cheaper storage and sending only summaries to the SIEM, can cut costs substantially without losing detection value, provided the full data remains available for investigations.

Alert triage and analyst capacity

Analysts are the scarcest resource in monitoring. If each alert takes on average 15 minutes to triage and an analyst has six productive hours per shift, one analyst can properly handle about 24 alerts per shift. A SOC that generates 500 alerts a day therefore needs many analysts, or better rules, or automation. Two metrics guide tuning: the precision of each rule (the share of its alerts that turn out to be true positives) and the alert volume it produces.

import random
random.seed(7)
# Alert volume and precision by detection rule over one month (synthetic)
rules = {
    'impossible travel':         (420, 0.04),
    'new admin account':         (35,  0.40),
    'mass file download':        (160, 0.12),
    'malware hash match':        (60,  0.85),
    'login from Tor exit node':  (900, 0.01),
}
minutes_per_alert = 15
print(f"{'rule':26s} {'alerts':>6s} {'true':>5s} {'precision':>9s} {'analyst h':>9s}")
total_h = 0
for name, (n, p) in rules.items():
    tp = sum(random.random() < p for _ in range(n))
    hours = n * minutes_per_alert / 60
    total_h += hours
    print(f'{name:26s} {n:6d} {tp:5d} {tp / n:9.1%} {hours:9.0f}')
print(f'Total analyst hours this month: {total_h:.0f} (about {total_h / 6 / 21:.1f} analysts at 6 h x 21 days)')

rule                       alerts  true precision analyst h
impossible travel             420    18      4.3%       105
new admin account              35    18     51.4%         9
mass file download            160    19     11.9%        40
malware hash match             60    49     81.7%        15
login from Tor exit node      900    11      1.2%       225
Total analyst hours this month: 394 (about 3.1 analysts at 6 h x 21 days)

Two rules in this synthetic month consume most of the workload while finding almost nothing. The right response is not simply to disable them but to tune them: for example, suppress known corporate VPN egress points from the travel rule, or route the low-precision Tor rule to an automated enrichment step that only escalates when combined with a second indicator. The high-precision malware rule deserves fast, perhaps automatic, response. Tuning decisions should be documented, reviewed and tested, because a careless suppression can hide a real attack.

Continuous monitoring and egress monitoring

Continuous monitoring, in the sense used by NIST SP 800-137 and many regulators, means maintaining ongoing awareness of security posture, vulnerabilities and threats to support risk decisions. It covers more than alerts: it includes configuration drift, patch status, asset inventory accuracy and control effectiveness, reported at a frequency matched to risk. The operations team provides much of this data.

Egress monitoring watches what leaves the network: outbound connections, uploads, email attachments and data moving to cloud services. It supports detection of data exfiltration and of compromised systems contacting external control servers. Data loss prevention (DLP) tools inspect content for sensitive information such as card numbers or health identifiers and can alert, block or encrypt. Egress monitoring must respect privacy and employment law and the organization's own monitoring notices, and encryption increasingly limits what network tools can see, which shifts monitoring toward endpoints and cloud service logs.

Threat intelligence and user behaviour analytics

Threat intelligence is information about threats that has been collected, analysed and put in context so that it supports decisions. Strategic intelligence informs executives about trends and adversary motives; operational intelligence describes campaigns and adversary behaviour; tactical and technical intelligence includes indicators of compromise such as malicious domains and file hashes, and descriptions of techniques. Many SOCs map their detection coverage to a framework of adversary behaviours, such as MITRE ATT&CK, to find gaps. Intelligence comes from commercial feeds, government sources such as CCCS, Canada's national cyber security agency, and the US Cybersecurity and Infrastructure Security Agency, industry information sharing and analysis centres, and the organization's own incidents. Intelligence is only useful if it is relevant, timely and acted on; a feed of a million indicators that nobody tunes can create more noise than value.

User and entity behaviour analytics (UEBA) builds baselines of normal behaviour for users, devices and services and flags deviations, such as an account that suddenly reads thousands of files it has never touched. UEBA helps detect insider threats and compromised accounts that use valid credentials, which rule-based detection often misses. Its outputs are probabilistic risk scores, so they need human judgement and, because they profile individuals, privacy review and clear policy.

Security orchestration, automation and response

Security orchestration, automation and response (SOAR) tools connect security systems and execute playbooks: when an alert of a certain type arrives, automatically gather context from threat intelligence and asset systems, open a ticket, notify the right analyst and, for well-understood cases, take a containment action such as disabling an account or isolating an endpoint. Automation reduces triage time and makes responses consistent. It also carries risk: an automated action that is wrong at scale can disrupt the business. Good practice is to automate enrichment freely, automate low-impact containment with clear criteria, and keep high-impact actions behind human approval, with every automated action logged and periodically reviewed.