Detection Engineering Lesson 1: Telemetry and Evidence
Table of Contents
Return to the Detection Engineering and Incident Response Course
A detection is only as strong as its telemetry contract. The contract states which system produced an event, which fields exist, how time is recorded, and where collection gaps appear.
Build an Inventory
Create one row per source before opening the investigation files.
| Source | Key fields | Useful question | Limit |
|---|---|---|---|
| Identity | User, source IP, device, result, authentication protocol, MFA state | Did a principal authenticate from a new context? | A successful sign-in does not prove who used the credential |
| Endpoint | Host, user, process, parent, command line, hash | What executed, and which process launched the child? | Missing events do not prove missing execution |
| DNS | Host, user, query, response, answer | Which names did an endpoint resolve? | A lookup does not prove a later connection |
Record retention, clock source, ingestion delay, field transformations, and access owner. Do not merge similarly named fields until their meanings match. A device name reported by an identity provider might describe registration state rather than the endpoint which sent traffic.
Preserve Evidence
- Copy the sample files into a read-only source folder.
- Create a separate folder for filters, timelines, and notes.
- Record a SHA-256 hash for each source file.
- Use UTC for every derived timeline entry.
- Keep raw values beside normalized values.
shasum -a 256 auth-events.jsonl endpoint-events.jsonl dns-events.csv
Expected output:
<64 hexadecimal characters> auth-events.jsonl
<64 hexadecimal characters> endpoint-events.jsonl
<64 hexadecimal characters> dns-events.csv
The exact hashes depend on the downloaded bytes. The verification target is stable hashes across your work session. Recalculate after the lab and compare the values.
Normalize a Timeline
List the JSON events in chronological order without changing the source.
jq -sr 'sort_by(.timestamp)[] | [.timestamp, (.host // .device), .user, .event_type, .correlation_id] | @tsv' \
auth-events.jsonl endpoint-events.jsonl
Inspect the CSV separately, then place selected DNS rows in your analyst timeline. Keep the original correlation ID in every derived row.
Use these columns:
timestamp_utc | source | asset | principal | action | result | correlation_id | analyst_note
Observation: A-1004 records a successful sign-in from 203.0.113.77.
Conclusion: the sign-in is suspicious because earlier failures, a new country, an unknown device, and a legacy protocol surround the event.
The conclusion depends on several observations. Keep the distinction visible.
Practice Steps
- Download the sample data README and its three linked files.
- Record file hashes.
- Build the source inventory table.
- Add the first five events to a normalized timeline.
- Mark each row as observation, conclusion, or open question.
- Write one collection gap for each source.
Expected Result: you have a source inventory, stable hashes, five ordered events, and three documented limits.
Troubleshooting
| Problem | Fix |
|---|---|
jq reports invalid JSON | Process the JSONL files one object per line or use jq -s as shown |
| Events sort incorrectly | Keep ISO 8601 UTC timestamps with the trailing Z |
| A field is absent | Use unknown in notes instead of copying a value from another source |
| Hashes changed | Restore the source copies and move analyst notes to a separate file |
Verify Your Work
- Source files stayed unchanged
- Every derived row retains a correlation ID
- All timestamps use UTC
- Every conclusion points to at least one observation
- Every source has a stated collection limit
Primary reference, checked 2026-10-10: CISA logging and monitoring guidance .
Next Steps
Continue to Lesson 2: ATT&CK Detection Strategies . You will connect observed behavior to current ATT&CK defensive objects.
