Logging strategies: structured events, selection, security, and retention

Logs are timestamped event records with severity, body, resource, and attributes. OpenTelemetry can correlate them with trace and span identifiers when context is valid. Correlation does not make a log a span, and trace tail-sampling semantics must not be silently applied to an independent log stream.

A logging design specifies event schema, clock and observed time, severity, routing, overload behavior, redaction, access control, integrity, retention, deletion, and query expectations. Retention durations are determined by business, security, legal, contractual, privacy, and cost requirements; SOC 2, PCI DSS, and SOX do not imply one universal number for every log.

Модель и предпосылки

• Applications write structured events to a bounded local/export path. A collector validates, normalizes and redacts before routing to hot search, archive, or security analysis.
• High-value security/audit and error events can be preserved while repetitive success events are deliberately sampled or aggregated; queries must disclose sampling and must not report sampled counts as complete totals.
• Untrusted fields are encoded/sanitized so control characters cannot forge log records. Secrets, tokens and unnecessary personal data are excluded before durable storage.

Проверяемые утверждения

• C1. The OpenTelemetry log model distinguishes event timestamp from observed timestamp and provides severity, body, resource, attributes, TraceId and SpanId fields.
• C2. Trace/span correlation enriches logs but does not turn log selection into trace tail sampling or guarantee complete context.
• C3. RFC 5424 defines a syslog header, structured data and message format; transport and application schemas still need validation.
• C4. Log management includes generation, transmission, storage, analysis and disposal, with organization-specific retention and protection requirements.
• C5. Logging should exclude or protect secrets/sensitive data and sanitize untrusted event fields against injection.

Исполняемые сценарии

Structured correlated event. The application emits a bounded schema with trace context, then the pipeline validates and redacts it before storage.

Priority selection under load. The pipeline preserves security/error classes and deliberately samples repetitive success events while disclosing the bias.

Retention and deletion matrix. Data class, purpose, access, legal hold, location and deletion evidence determine lifecycle instead of a universal compliance duration.

Log injection and secret guard. Untrusted fields and credentials are blocked before they can forge records or enter durable stores.

Ошибки проектирования

• Не называйте отбор логов tail sampling, если это не trace-wide decision над полным набором spans.
• Не используйте trace_id как authorization proof и не записывайте tokens, credentials или лишние PII.
• Не обещайте единый срок хранения из названия compliance framework; храните только обоснованный срок.
• Не показывайте sampled success events как полный event count без rate/weight metadata.

Границы гарантии

• At-least-once export can create duplicates; queries need event identity or deduplication policy.
• Clock skew and delayed delivery mean Timestamp and ObservedTimestamp answer different questions.
• During overload, drop/block/spill priority must be explicit and monitored; no policy is lossless under unbounded input.

Связанные материалы

Официальные и первичные источники

1. OpenTelemetry Logs data model
2. OpenTelemetry logging trace-context compatibility
3. RFC 5424: The Syslog Protocol
4. NIST SP 800-92: Guide to Computer Security Log Management
5. OWASP Logging Cheat Sheet