Skip to main content
Dynamic Configuration enables runtime tuning of Cadence server behavior without requiring restarts. This allows operational flexibility, gradual rollouts, and per-domain customization.

​
Overview

Dynamic config provides:
  • Runtime Updates: Change settings without restarting services
  • Scoped Configuration: Apply settings globally, per-domain, or per-task-list
  • Gradual Rollout: Test changes on specific domains before global deployment
  • Operational Safety: Tune throttling, timeouts, and limits during incidents
  • Multi-Tenancy: Customize behavior per tenant (domain)

​
Configuration Sources

Cadence supports multiple dynamic config backends:

​
File-Based Configuration

Default backend: YAML file loaded at startup and refreshed periodically. Configuration (config.yaml): 
dynamicConfigClient:
  filepath: "config/dynamicconfig/development.yaml"
  pollInterval: "60s"
Dynamic Config File (development.yaml): 
# Global setting
system.enableAuthorization:
  - value: false

# Domain-specific setting
history.defaultWorkflowTaskStartToCloseTimeout:
  - value: "10s"
  - value: "30s"
    constraints:
      domainName: "critical-domain"

# Domain and task list specific
matching.maxTaskListSize:
  - value: 1000
  - value: 5000
    constraints:
      domainName: "high-volume-domain"
      taskListName: "important-tasks"

​
Database-Based Configuration

Enterprise feature: Store config in Cassandra/SQL for centralized management. Configuration: 
dynamicConfigClient:
  configstore:
    pollInterval: "30s"
Requires separate config service deployment.

​
Environment-Based Overrides

Quick overrides: Use environment variables for specific settings. 
DYNAMIC_CONFIG_OVERRIDE_frontend.rps=2000 ./cadence-server start

​
Configuration Format

​
Value Structure

Each setting is a list of values with optional constraints: 
key:
  - value: <default-value>
  - value: <override-value>
    constraints:
      domainName: "<domain>"
      taskListName: "<task-list>"
      taskType: <0=Decision, 1=Activity>

​
Constraint Types

  1. domainName: String - Apply to specific domain
  2. taskListName: String - Apply to specific task list
  3. taskType: Integer - Apply to task type (0=Decision, 1=Activity)

​
Constraint Matching

Value is selected if all constraints match exactly: 
# Match: domain=foo, taskList=bar
- value: 100
  constraints:
    domainName: "foo"
    taskListName: "bar"

# Match: domain=foo only (no task list specified)
- value: 50
  constraints:
    domainName: "foo"

# Match: any domain/task list (no constraints)
- value: 10
Query with domain=foo, taskList=bar returns 100 (most specific match).

​
Common Configuration Keys

​
Throttling and Rate Limits

​
Frontend Rate Limits

# Global RPS limit per frontend instance
frontend.rps:
  - value: 2000

# RPS limit per domain
frontend.domainRPS:
  - value: 500
  - value: 2000
    constraints:
      domainName: "high-priority-domain"

# Max concurrent requests
frontend.maxConcurrentRequests:
  - value: 1000

​
History Service Limits

# Task processing RPS
history.taskProcessRPS:
  - value: 2000

# Transfer queue max reader count
history.transferTaskMaxReadLevel:
  - value: 500

# History event count limit per decision
history.historyCountLimitWarn:
  - value: 10000
history.historyCountLimitError:
  - value: 50000

​
Matching Service Limits

# Task list max size
matching.maxTaskListSize:
  - value: 1000
  - value: 10000
    constraints:
      domainName: "large-scale-domain"

# Task dispatch RPS
matching.taskDispatchRPS:
  - value: 5000

​
Timeouts and Retention

# Default workflow execution timeout
frontend.defaultWorkflowExecutionTimeout:
  - value: "72h"

# Default decision task timeout
history.defaultWorkflowTaskStartToCloseTimeout:
  - value: "10s"
  - value: "30s"
    constraints:
      domainName: "long-decision-domain"

# Domain retention period
frontend.domainRetentionDays:
  - value: 7

# Max page size for list operations
frontend.visibilityMaxPageSize:
  - value: 1000

# Enable advanced visibility
system.advancedVisibilityWritingMode:
  - value: "dual"  # off, on, dual

# Valid search attributes
frontend.validSearchAttributes:
  - value:
      CustomerId: "Keyword"
      OrderStatus: "Keyword"
      OrderAmount: "Double"
      Passed: "Bool"

​
Archival Configuration

# Archival worker concurrency
history.numArchiverConcurrency:
  - value: 50

# Archival processing rate
history.archivalProcessRPS:
  - value: 100

​
Feature Flags

# Enable authorization
system.enableAuthorization:
  - value: false

# Enable gRPC
frontend.enableGRPCOutbound:
  - value: true

# Enable sticky scheduling
history.enableStickyQuery:
  - value: true

​
Key Categories

Configuration keys are organized by component:

​
System-Wide

  • system.* - Global system settings
  • limit.* - Global rate limits

​
Service-Specific

  • frontend.* - Frontend service settings
  • history.* - History service settings
  • matching.* - Matching service settings
  • worker.* - Worker service settings

​
Feature-Specific

  • archival.* - Archival configuration
  • visibility.* - Visibility configuration
  • authorization.* - Authorization settings

​
Best Practices

​
Configuration Management

  • Version Control: Track dynamic config files in Git
  • Testing: Test changes in development before production
  • Documentation: Document non-standard settings and reasoning
  • Monitoring: Watch for config-related errors in logs

​
Performance Tuning

  • Start Conservative: Begin with default values
  • Measure Impact: Monitor metrics before and after changes
  • Gradual Rollout: Test on single domain before global deployment
  • Domain Isolation: Use domain-specific limits for noisy neighbors

​
Operational Safety

  • Emergency Response: Keep emergency config changes documented
  • Rollback Plan: Test reverting to previous config
  • Throttling: Prefer throttling over rejection during incidents
  • Validation: Validate config syntax before deployment

​
Multi-Tenancy

# Isolate tenants with different limits
frontend.domainRPS:
  - value: 100
  - value: 1000
    constraints:
      domainName: "premium-tenant"
  - value: 10
    constraints:
      domainName: "trial-tenant"

matching.maxTaskListSize:
  - value: 1000
  - value: 10000
    constraints:
      domainName: "premium-tenant"

​
Configuration Discovery

​
List All Keys

See available dynamic config keys in source code: 
# Find all dynamic config keys
grep -r "GetBoolProperty\|GetIntProperty\|GetDurationProperty" \
  common/dynamicconfig/

​
Key Naming Convention

Keys follow pattern: <component>.<feature><PropertyType> Examples:
  • frontend.rps - Frontend RPS limit
  • history.taskProcessRPS - History task processing rate
  • matching.maxTaskListSize - Matching task list size

​
Monitoring Dynamic Config

​
Log Entries

Config changes and errors are logged: 
[INFO] Dynamic config reloaded successfully
[WARN] Failed to fetch key: frontend.rps, using default value
[DEBUG] Dynamic config not set for key: custom.setting, use default value

​
Metrics

Monitor config-related metrics: 
# Config fetch errors
sum(rate(cadence_dynamic_config_errors_total[5m]))

# Config reload success rate
sum(rate(cadence_dynamic_config_reloads_total[5m]))

​
Validation

Verify config is applied: 
# Check logs for config reload
grep "Dynamic config reloaded" /var/log/cadence/frontend.log

# Monitor specific metric affected by config
# e.g., after changing frontend.rps

​
Examples

​
Gradual Feature Rollout

# Enable new feature for canary domain
feature.newQueryEngine:
  - value: false              # Default: disabled
  - value: true
    constraints:
      domainName: "canary"    # Enable for canary

# After validation, enable for all
feature.newQueryEngine:
  - value: true               # Enable globally

​
Emergency Throttling

# Reduce load during incident
frontend.domainRPS:
  - value: 500              # Default
  - value: 100              # Emergency: reduce to 20%
    constraints:
      domainName: "noisy-domain"

history.taskProcessRPS:
  - value: 1000             # Reduce task processing

​
Custom Timeouts for Workflows

history.defaultWorkflowTaskStartToCloseTimeout:
  - value: "10s"
  - value: "60s"
    constraints:
      domainName: "ml-training"  # ML workflows need longer
  - value: "30s"
    constraints:
      domainName: "data-pipeline"

​
Task List Prioritization

matching.taskDispatchRPS:
  - value: 1000
  - value: 5000
    constraints:
      domainName: "priority-domain"
      taskListName: "critical-tasks"

matching.maxTaskListSize:
  - value: 1000
  - value: 10000
    constraints:
      domainName: "priority-domain"
      taskListName: "critical-tasks"

​
Troubleshooting

​
Config Not Applied

Problem: Changes to dynamic config have no effect Solution: 
# Check file path in config
grep dynamicConfigClient config.yaml

# Verify file syntax
yaml-lint config/dynamicconfig/development.yaml

# Check poll interval (may take time to apply)
grep pollInterval config.yaml

# Force reload (if supported)
kill -SIGHUP <cadence-pid>

# Check logs for reload events
grep "Dynamic config" /var/log/cadence/*.log

​
Constraint Not Matching

Problem: Domain-specific config not applying Solution: 
# Ensure exact match of ALL constraints
- value: 100
  constraints:
    domainName: "my-domain"     # Must match exactly
    # If taskListName is in query, it must be in constraints too

​
Default Values Used

Problem: Seeing “dynamic config not set” warnings Solution:
  • This is normal for optional configs
  • Only add config entries for values you want to override
  • Default values are defined in code

​
Next Steps

Build docs developers (and LLMs) love

Get started for freeTalk to us