Lemline uses Protocol Buffers for internal messaging to achieve efficient serialization, strong typing, and backward compatibility.
Why Protocol Buffers?
Compact serialization
Binary format reduces message size by 50-80% compared to JSON
Strong typing
Compile-time validation prevents serialization errors
Backward compatibility
Add new fields without breaking existing consumers
Cross-language
Generated clients for Kotlin, TypeScript, Python, Go, etc.
Schema overview
Internal schemas are defined in lemline-messages-proto/src/main/proto/internal/:
internal/
├── common.proto # Shared types (timestamps, errors)
├── state/
│ ├── node_state.proto # Task state variants
│ └── node_stack.proto # Execution stack frames
└── workflow/
├── commands.proto # Workflow command messages
├── events.proto # Workflow event messages
├── configs.proto # Configuration messages
└── envelope.proto # Message envelopes
Workflow commands
Commands flow through the commands-in and commands-out channels:
WorkflowCommand
message WorkflowCommand {
string workflow_id = 1;
string namespace = 2;
string name = 3;
string version = 4;
bytes workflow_state = 5; // Compressed InstanceState
optional string input_json = 6;
optional string zone_id = 7;
optional string correlation_id = 8;
}
InstanceState (embedded in workflow_state)
message InstanceState {
map<string, NodeState> states = 1; // Node position -> state
NodePosition current_position = 2;
WorkflowStatus status = 3;
optional string output_json = 4;
optional ErrorDetail error = 5;
}
enum WorkflowStatus {
WORKFLOW_STATUS_UNSPECIFIED = 0;
WORKFLOW_STATUS_RUNNING = 1;
WORKFLOW_STATUS_WAITING = 2;
WORKFLOW_STATUS_COMPLETED = 3;
WORKFLOW_STATUS_FAULTED = 4;
}
Workflow events
Events flow through the events-out channel for durable operations:
WorkflowEvent
message WorkflowEvent {
string workflow_id = 1;
string event_type = 2;
bytes payload = 3;
int64 sequence = 4;
string created_at = 5;
}
Event types
| Event Type | Payload | Purpose |
|---|
RETRY_SCHEDULED | RetryPayload | Task retry scheduled |
WAIT_SCHEDULED | WaitPayload | Wait task created |
FORK_CREATED | ForkPayload | Fork branches spawned |
LISTEN_CREATED | ListenPayload | Listen task waiting |
SCHEDULE_CREATED | SchedulePayload | Workflow scheduled |
Node state variants
Each task type has a specific state structure:
message NodeState {
oneof state {
RootState root = 1;
DoState do = 2;
SwitchState switch = 3;
ForState for = 4;
TryState try = 5;
ForkState fork = 6;
// ... other task types
}
}
message DoState {
int32 current_index = 1;
repeated string completed_tasks = 2;
}
message ForkState {
repeated string branch_ids = 1;
map<string, BranchStatus> branch_statuses = 2;
}
message TryState {
bool in_catch = 1;
optional ErrorDetail caught_error = 2;
}
Error representation
message ErrorDetail {
string type = 1; // RFC 7807 error type URI
int32 status = 2; // HTTP-style status code
string title = 3; // Human-readable summary
optional string detail = 4; // Detailed error message
optional string instance = 5; // Error instance identifier
}
{
"type": "https://lemline.com/errors/validation",
"status": 400,
"title": "Validation error",
"detail": "Invalid input: missing required field 'orderId'",
"instance": "wf-123/root.do.0"
}
Message envelope
All broker messages are wrapped in an envelope:
message MessageEnvelope {
string message_id = 1; // UUIDv7
string message_type = 2; // COMMAND, EVENT, LIFECYCLE
bytes payload = 3; // Serialized message
string created_at = 4; // ISO 8601 timestamp
optional string correlation_id = 5;
}
Transport (broker messages)
Internal messages use Protobuf binary format:
// Serialize
val bytes = workflowCommand.toByteArray()
// Deserialize
val command = WorkflowCommand.parseFrom(bytes)
Storage (database)
Database payloads use Protobuf JSON for human readability:
// Serialize to JSON
val json = JsonFormat.printer().print(workflowCommand)
// Deserialize from JSON
val builder = WorkflowCommand.newBuilder()
JsonFormat.parser().merge(json, builder)
val command = builder.build()
Backward compatibility rules
Follow these rules to maintain backward compatibility when evolving schemas:
Never reuse field numbers
Once a field number is assigned, it’s permanently reserved.message Example {
string old_field = 1 [deprecated = true]; // Don't remove
string new_field = 2; // Use next number
}
When removing a field, reserve both its number and name.message Example {
reserved 3; // Field number
reserved "removed_field"; // Field name
string active_field = 4;
}
Use optional for new fields
New fields must be optional to maintain compatibility with older messages.message Example {
string required_field = 1; // Existing
optional string new_field = 2; // New (optional)
}
Enums must have _UNSPECIFIED = 0
First enum value must be *_UNSPECIFIED = 0 for default behavior.enum Status {
STATUS_UNSPECIFIED = 0; // Required
STATUS_ACTIVE = 1;
STATUS_COMPLETED = 2;
}
Code generation
Protobuf schemas are compiled to Kotlin classes:
# Generate Kotlin code
./gradlew :lemline-messages-proto:build
com.lemline.messages.internal.workflow.WorkflowCommandcom.lemline.messages.internal.workflow.WorkflowEventcom.lemline.messages.internal.state.NodeState
Usage examples
Creating a workflow command
import com.lemline.messages.internal.workflow.WorkflowCommand
val command = WorkflowCommand.newBuilder()
.setWorkflowId("wf-123")
.setNamespace("production")
.setName("order-fulfillment")
.setVersion("1.0.0")
.setWorkflowState(compressedState)
.setInputJson("{}")
.build()
// Serialize to bytes
val bytes = command.toByteArray()
Parsing a workflow event
import com.lemline.messages.internal.workflow.WorkflowEvent
// Deserialize from bytes
val event = WorkflowEvent.parseFrom(bytes)
when (event.eventType) {
"RETRY_SCHEDULED" -> {
val payload = RetryPayload.parseFrom(event.payload)
handleRetry(payload)
}
"WAIT_SCHEDULED" -> {
val payload = WaitPayload.parseFrom(event.payload)
handleWait(payload)
}
}
Schema validation
Lemline uses buf for schema linting and breaking change detection:
# Lint schemas
buf lint
# Check for breaking changes
buf breaking --against .git#branch=main
docs/protobuf-messaging.md in the source repository for the full policy.
Next steps
Messaging architecture
Learn about dual-channel messaging
Outbox pattern
Reliable message delivery
State management
Workflow state structure
Protocol Buffers
Official Protobuf documentation