Skip to main content
SSV Node uses structured logging with Zap to provide detailed operational insights. This page covers log configuration, formats, and analysis techniques.

​
Log Configuration

Configure logging through your config.yaml or environment variables.

​
Configuration Options

Based on the source code (source: observability/config.go:12 and observability/log/global.go:33), the logger supports:
config.yaml
# Logging configuration
LogLevel: "info"        # debug, info, warn, error, fatal
LogFormat: "json"       # json or console
LogLevelFormat: "capital" # capital, capitalColor, lowercase
LogFilePath: "/var/log/ssv/node.log"  # Optional: write logs to file
LogFileSize: 100        # MB per log file (when using file output)
LogFileBackups: 10      # Number of old log files to keep

​
Log Levels

Use for: Development and detailed troubleshootingIncludes verbose information about:
  • Message routing and processing
  • Duty scheduling details
  • Network events and peer connections
  • Consensus round details
  • Cache hits/misses
Debug logging generates significant output and may impact performance. Use only when necessary.
Use for: Production monitoring (recommended)Includes:
  • Node startup and shutdown events
  • Validator lifecycle events
  • Duty handler start/stop
  • Successful duty submissions
  • Network connection status
  • Important state changes
This is the recommended level for production deployments.
Use for: Highlighting potential issuesIncludes:
  • Validator not found warnings
  • Misaligned slot and duty timing
  • Subscription channel congestion
  • Recoverable errors
  • Performance degradation signals
Use for: Tracking operational failuresIncludes:
  • Failed duty submissions
  • Beacon node connection failures
  • Database operation errors
  • Validator errors
  • Message processing failures
Error-level logs indicate issues requiring operator attention.
Use for: Critical failures that stop the nodeIncludes:
  • Panic recovery events with stack traces
  • Unrecoverable initialization failures
  • Critical configuration errors
Fatal logs indicate the node has stopped or will stop immediately.

​
Log Formats

​
JSON Format (Production)

Structured logs suitable for log aggregation and analysis. 
{
  "level": "INFO",
  "time": "2026-03-04T10:15:30.123456Z",
  "caller": "duties/attester.go:145",
  "msg": "starting duty handler",
  "name": "ssv.operator.duties.attester"
}
JSON format is recommended for production as it integrates seamlessly with logging platforms like ELK, Splunk, or Grafana Loki.

​
Console Format (Development)

Human-readable format for local development. 
2026-03-04T10:15:30.123456Z	INFO	duties/attester.go:145	starting duty handler	{"name": "ssv.operator.duties.attester"}

​
Level Format Options

Based on source: observability/log/global.go:20
  • capital: INFO, ERROR, DEBUG (default)
  • capitalColor: Colored level names (useful for console viewing)
  • lowercase: info, error, debug

​
File Output Configuration

SSV Node supports rotating log files using [lumberjack](source: observability/log/global.go:109).
config.yaml
LogFilePath: "/var/log/ssv/node.log"
LogFileSize: 100      # Megabytes before rotation
LogFileBackups: 10    # Keep 10 old log files
File rotation behavior:
  • New file created when size exceeds LogFileSize MB
  • Keeps LogFileBackups number of old files
  • Old files retained for 28 days maximum
  • Files are not compressed by default
When file output is configured, logs are written to both stdout and the log file. The file always contains all log levels (debug and above) regardless of the LogLevel setting.

​
Log Fields and Context

SSV Node uses structured logging with consistent field names across the codebase.

​
Standard Log Fields

Based on source: observability/log/fields/fields.go and log examples:
Field NameDescriptionExample
epochBeacon chain epoch{"epoch": 12345}
slotBeacon chain slot{"slot": 394560}
validatorValidator public key (hex){"validator": "0x8234...abcd"}
duty_idUnique duty identifier{"duty_id": "..."}
roleValidator duty role{"role": "ATTESTER"}
addressNetwork address{"address": "0.0.0.0:15000"}
nameLogger component name{"name": "ssv.operator.duties"}
callerSource file and line{"caller": "attester.go:145"}

​
Component Names

Logger names follow a hierarchical pattern (source: observability/log/names.go):
  • ssv.observability - Observability system
  • ssv.operator.duties - Duty handlers
    • ssv.operator.duties.attester - Attester duty handler
    • ssv.operator.duties.proposer - Proposer duty handler
    • ssv.operator.duties.committee - Sync committee handler
  • ssv.operator.validator - Validator lifecycle management
  • ssv.runner - Duty execution runners
  • ssv.api - API server
  • ssv.metrics - Metrics handler

​
Log Examples

​
Startup Logs

{"level":"INFO","time":"2026-03-04T10:15:30.000000Z","msg":"starting duty handler","name":"ssv.operator.duties.attester"}
{"level":"INFO","time":"2026-03-04T10:15:30.001000Z","msg":"setup collection","name":"ssv.metrics","address":"0.0.0.0:15000","enableProf":true}
{"level":"INFO","time":"2026-03-04T10:15:30.002000Z","msg":"Serving SSV API","addr":":13000"}

​
Duty Execution Logs

{"level":"DEBUG","time":"2026-03-04T10:16:00.000000Z","msg":"🛠 ticker event","epoch_slot_pos":"12345/394560/0"}
{"level":"DEBUG","time":"2026-03-04T10:16:00.100000Z","msg":"🗂 got duties","epoch":12345,"duties_count":42}
{"level":"DEBUG","time":"2026-03-04T10:16:00.500000Z","msg":"successfully fetched attestation data","slot":394560}

​
Reorg and Index Change Events

{"level":"INFO","time":"2026-03-04T10:16:05.000000Z","msg":"🔀 reorg event received","epoch_slot_pos":"12345/394561/1","event":{"slot":394560,"depth":1}}
{"level":"INFO","time":"2026-03-04T10:17:00.000000Z","msg":"🔁 indices change received","epoch_slot_pos":"12345/394565/5"}

​
Warning Logs

{"level":"WARN","time":"2026-03-04T10:16:10.000000Z","msg":"validator not found","validator":"0x8234567890abcdef..."}
{"level":"WARN","time":"2026-03-04T10:16:11.000000Z","msg":"subscriber channel full, dropping the message"}
{"level":"WARN","time":"2026-03-04T10:16:12.000000Z","msg":"validator not yet participating","validator_index":12345}

​
Error Logs

{"level":"ERROR","time":"2026-03-04T10:16:15.000000Z","msg":"failed to fetch duties for current epoch","error":"connection refused","epoch":12345}
{"level":"ERROR","time":"2026-03-04T10:16:16.000000Z","msg":"failed to submit beacon committee subscription","error":"timeout exceeded"}
{"level":"ERROR","time":"2026-03-04T10:16:17.000000Z","msg":"exporter API request failed","error":"invalid request"}

​
Log Analysis

​
Using jq for JSON Logs

Filter and analyze JSON-formatted logs: 
# Filter ERROR level logs
cat node.log | jq 'select(.level == "ERROR")'

# Show all logs for a specific validator
cat node.log | jq 'select(.validator == "0x8234...")'

# Count errors by message
cat node.log | jq -r 'select(.level == "ERROR") | .msg' | sort | uniq -c

# Extract duty execution timeline
cat node.log | jq 'select(.msg | contains("duty")) | {time, msg, epoch, slot}'

# Find logs from specific component
cat node.log | jq 'select(.name | startswith("ssv.operator.duties"))'

​
Using grep for Pattern Matching

# Find all reorg events
grep "reorg event" node.log

# Track validator lifecycle
grep -E "(starting|stopping|removed)" node.log

# Find connection errors
grep -i "connection" node.log | grep -i "error\|fail"

# Monitor duty submissions
grep "submission" node.log

​
Log Aggregation with Grafana Loki

Grafana Loki provides powerful log aggregation for SSV Node’s JSON logs.
Example LogQL queries: 
# All error logs from SSV node
{job="ssv-node"} | json | level="ERROR"

# Validator-related warnings
{job="ssv-node"} | json | name=~".*validator.*" | level="WARN"

# Duty execution timeline
{job="ssv-node"} | json | msg=~".*duty.*"

# Rate of errors per minute
sum(rate({job="ssv-node"} | json | level="ERROR" [1m]))

# Errors grouped by component
sum by (name) (count_over_time({job="ssv-node"} | json | level="ERROR" [5m]))

​
Common Log Patterns

​
Normal Operation

{"level":"INFO","msg":"starting duty handler"}
{"level":"DEBUG","msg":"🛠 ticker event"}
{"level":"DEBUG","msg":"🗂 got duties"}
{"level":"DEBUG","msg":"successfully fetched attestation data"}

​
Beacon Node Connectivity Issues

{"level":"ERROR","msg":"failed to fetch duties for current epoch","error":"connection refused"}
{"level":"ERROR","msg":"couldn't fetch node version","error":"context deadline exceeded"}
Persistent beacon node errors indicate network connectivity or beacon node availability issues. Check your beacon node endpoint configuration.

​
Performance Degradation

{"level":"WARN","msg":"current slot and duty slot are not aligned"}
{"level":"WARN","msg":"subscriber channel full, dropping the message"}

​
Chain Reorg Events

{"level":"INFO","msg":"🔀 reorg event received","event":{"slot":394560,"depth":1}}
Reorgs are normal but frequent deep reorgs may indicate beacon chain issues.

​
Logging Best Practices

Production: Use JSON

Always use JSON format in production for easier parsing and integration with log aggregation tools.

Set Appropriate Level

Use info level for production. Enable debug only when troubleshooting specific issues.

Implement Log Rotation

Configure file output with rotation to prevent disk space exhaustion.

Monitor Error Rates

Track error log frequency as a key health indicator. Sudden increases warrant investigation.

​
Log Retention Recommendations

  • Production: Retain 7-30 days of logs
  • Development: 1-7 days sufficient
  • Compliance: Adjust based on regulatory requirements

​
Privacy Considerations

Logs may contain validator public keys and operational data. Ensure proper access controls and consider data privacy regulations when sharing logs.

​
Next Steps

Prometheus Metrics

Set up Prometheus metrics for quantitative monitoring

Troubleshooting Guide

Debug common issues using logs and metrics

Build docs developers (and LLMs) love

Get started for freeTalk to us