Message Queue Implementations

Message queues enable asynchronous communication between distributed components, improving system performance through decoupling, traffic smoothing, and fault tolerance.

Message Queue Fundamentals

What is a Message Queue?

A message queue is an inter-application communication mechanism where:

Producers send messages without waiting for processing
Message broker ensures reliable delivery
Consumers retrieve and process messages independently

Design Pattern Connection: Message queues implement the Observer pattern - publishers emit events without knowing who will consume them, and subscribers receive events without knowing who published them.

Core Components

Producer

Sends messages to the queue

Publishes without waiting
No knowledge of consumers
Can continue processing immediately

Message Broker

Central message processing system

Stores messages reliably
Routes to consumers
Ensures delivery guarantees

Consumer

Retrieves and processes messages

Pulls from queue
No knowledge of producers
Processes asynchronously

Why Use Message Queues?

Asynchronous Processing
Decoupling
Traffic Smoothing

Improve system performance by deferring non-critical operations

Synchronous problem

Issues: Long response time, user waits for all operations

Asynchronous solution

Benefits: Fast response, background processing

Real Example: E-commerce Order

Critical path (synchronous):

Validate inventory
Process payment
Create order record

Background tasks (async via queue):

Send SMS notifications
Send email receipts
Update analytics
Log audit trail

Reduce dependencies between system componentsBefore (tightly coupled):

public void placeOrder(Order order) {
    inventoryService.reserve(order);  // Direct dependency
    paymentService.charge(order);      // Direct dependency
    smsService.notify(order);          // Direct dependency
    emailService.send(order);          // Direct dependency
    logService.record(order);          // Direct dependency
}

Problems:

All services must be available
Changes cascade across services
Difficult to add new integrations

After (loosely coupled):

public void placeOrder(Order order) {
    inventoryService.reserve(order);
    paymentService.charge(order);
    
    // Publish event, don't call directly
    eventPublisher.publish("order.created", order);
    
    return order; // Fast response
}

Benefits:

Services work independently
Easy to add new consumers
Failures isolated

Reliability Guarantees

Durability

Messages must not be lost

Persist to disk
Replicate across nodes
Survive broker restarts

At-Least-Once Delivery

Every message consumed at least once

Consumer acknowledgment (ACK)
Retry on failure
Idempotent processing

Ordering

Preserve message order when required

Per-partition ordering
Single consumer per partition
Trade-off with parallelism

Redis as Message Queue

Redis is primarily a cache, but its data structures can implement simple message queues. Three approaches:

List (PUSH/POP)
Pub/Sub
Streams (Redis 5.0+)

Production-Consumption with Redis Lists

Pattern: Producer-Consumer (Point-to-Point)Redis Lists are doubly-linked lists with O(1) insert/delete at both ends - perfect for FIFO queues.

Approach 1: LPUSH + RPOP

# Producer adds to left (head)
LPUSH queue:orders '{"orderId": 123, "amount": 99.99}'

# Consumer removes from right (tail) - FIFO
RPOP queue:orders

✅ Advantages:

O(1) time complexity
FIFO ordering guaranteed
Redis persistence protects server-side data

❌ Critical Issues:

1. Performance Risk - Busy Polling

// BAD: CPU-intensive busy loop
while(true) {
    $result = $redis->rpop("queue");
    if($result) {
        $data = json_decode($result, true);
        // process...
    }
    // Empty queue = wasted CPU cycles!
}

Consumers constantly poll even when queue empty
Wastes CPU resources
Increases Redis load

2. Point-to-Point Only

Single consumer gets each message
No broadcast/fanout capability
First consumer wins (based on speed)

3. Data Safety - Client-Side Risk

RPOP removes from server immediately
↓
Data now only in client memory
↓
If client crashes → Message lost forever!

No acknowledgment mechanism
Cannot retry failed processing
Network issues = data loss

Approach 2: LPUSH + BRPOP (Recommended)

Blocking pop solves the busy-polling problem:

# Consumer blocks until data available or timeout
BRPOP queue:orders 30  # Wait up to 30 seconds
# Returns: 1) "queue:orders" 2) "{message}"

# Wait indefinitely (timeout=0)
BRPOP queue:orders 0

✅ Benefits:

No busy-waiting: Client sleeps until message arrives
Reduced Redis load: No constant polling
Instant processing: Wakes immediately when message published
Multiple queues: BRPOP queue1 queue2 queue3 0 (priority order)

How BRPOP works:

If queue has data → Returns immediately
If queue empty → Client connection blocks
When new data arrives → Client instantly awakened
If timeout reached → Returns nil

Exception Handling Required

while True:
    try:
        result = redis.brpop('queue:orders', timeout=30)
        if result:
            process_message(result[1])
    except redis.exceptions.ConnectionError:
        # Server disconnected idle connection
        reconnect()

Redis may close idle connections to save resources. Always handle connection errors!

❌ Still has issues:

Client data safety problem remains
No ACK mechanism
Cannot handle failed processing

Approach 3: LPUSH + LRANGE + RPOP

Strategy: Read first, then consume after processing

# Consumer peeks at message (doesn't remove)
LRANGE queue:orders -1 -1  # Get last element

# Process message...
# If successful:
RPOP queue:orders  # Now remove it

✅ Improves safety:

Message stays on server during processing
Client crash doesn’t lose message

❌ New problems:

Not blocking (back to busy-polling)
Duplicate processing risk: If consumer crashes after processing but before RPOP

Approach 4: LPUSH + BRPOPLPUSH + LREM (Most Reliable)

Atomic move to backup queue:

# Atomically move from main queue to processing queue
BRPOPLPUSH queue:orders queue:processing 30

# Consumer processes message...
# On success, remove from processing queue:
LREM queue:processing 1 "{message}"

How It Works

✅ Advantages:

Atomic operation: Message never lost between queues
Server-side safety: All operations on Redis
Crash recovery: Unacknowledged messages stay in processing queue
Blocking: No busy-polling

❌ Remaining issue:

Stuck messages: If consumer crashes, messages stuck in queue:processingSolution: Monitoring daemon

# Separate process monitors processing queue
def rescue_stuck_messages():
    while True:
        # Find messages older than timeout
        old_messages = find_old_messages('queue:processing', timeout=300)
        for msg in old_messages:
            # Move back to main queue
            redis.rpoplpush('queue:processing', 'queue:orders')
        time.sleep(60)

This creates a circular queue for automatic retry.

Summary: List-Based Patterns

Approach	Blocking	Data Safety	Duplicate Risk	Complexity
LPUSH + RPOP	❌	❌ Server only	❌	Low
LPUSH + BRPOP	✅	❌ Server only	❌	Low
LPUSH + LRANGE + RPOP	❌	⚠️ Better	✅ Possible	Medium
LPUSH + BRPOPLPUSH	✅	✅ Best	✅ Possible*	High

*Requires monitoring daemon for stuck message recovery

Pattern: Message Broadcasting (One-to-Many)Unlike List queues (one consumer per message), Pub/Sub sends each message to all subscribers.

Basic Commands

# Subscribers listen to channel
SUBSCRIBE channel:orders
# Output:
# 1) "subscribe"
# 2) "channel:orders"
# 3) (integer) 1  # Number of subscriptions

# Publisher sends message
PUBLISH channel:orders '{"orderId": 123}'
# Returns: (integer) 3  # Number of subscribers who received it

# All subscribers receive:
# 1) "message"           # Type
# 2) "channel:orders"    # Channel name
# 3) '{"orderId": 123}'  # Message payload

# Unsubscribe
UNSUBSCRIBE channel:orders

Pattern Matching with PSUBSCRIBE

Subscribe to multiple channels with wildcards:

# Subscribe to all payment-related channels
PSUBSCRIBE pay.*

# Receives messages from:
# - pay.success
# - pay.failed
# - pay.refund
# - pay.anything

Message format (4 fields):

1) "pmessage"           # Type (pattern message)
2) "pay.*"              # Pattern matched
3) "pay.success"        # Actual channel
4) '{"amount": 99.99}'  # Payload

Use Cases

Event Broadcasting

Example: Order created event

# Publisher
redis.publish('order.created', json.dumps({
    'orderId': 123,
    'userId': 456
}))

# Multiple consumers:
# - Inventory service (reduce stock)
# - Email service (send confirmation)
# - Analytics service (track metrics)
# - Notification service (push alert)

Real-Time Messaging

Example: Chat application

# User joins chat room
redis.subscribe(f'chat:room:{room_id}')

# User sends message
redis.publish(f'chat:room:{room_id}', message)

# All room participants receive instantly

Critical Limitations

🚨 Fire-and-Forget ModelPub/Sub has NO persistence or reliability:

No message storage
- Messages exist only in transit
- If no subscribers → Message dropped
- Cannot retrieve historical messages
No delivery guarantees
- Network disconnect → Messages lost
- Redis restart → All in-flight messages lost
- No acknowledgment mechanism
No retry capability
- Failed processing → Message gone forever
- Cannot replay messages
- No dead-letter queue

When NOT to use Pub/Sub

Avoid for critical operations:❌ Payment confirmations ❌ Order processing ❌ Financial transactions ❌ Anything requiring guaranteed deliveryUse professional MQ instead: RabbitMQ, Kafka, RocketMQ

Internal Implementation

How Redis Pub/Sub Works Internally

Redis maintains a dictionary (hash map) for Pub/Sub:

# Simplified internal structure
pubsub_channels = {
    'channel:orders': [client1, client2, client3],  # Linked list of subscribers
    'channel:payments': [client2, client4],
    'news.tech': [client1, client5, client6]
}

SUBSCRIBE flow:

Client subscribes to channel
Redis adds client to channel’s subscriber list

PUBLISH flow:

Look up channel in dictionary
Iterate through subscriber list
Send message to each subscriber
Return count of recipients

Why it’s fast: O(1) lookup, O(N) broadcast where N = subscribers

Best Practices

✅ Good use cases:

Real-time dashboards
Live notifications (can tolerate loss)
Cache invalidation signals
Presence detection (user online/offline)

⚠️ Acceptable with care:

Chat applications (users understand potential message loss)
Live sports scores
Stock price updates

❌ Never use for:

Business-critical events
Financial transactions
Audit logs
Anything requiring compliance

Redis Streams - Production-Grade MQ

Introduced: Redis 5.0 (2018)Inspiration: Apache Kafka conceptsStreams combine the best of Lists and Pub/Sub while adding persistence, consumer groups, and acknowledgments.

Key Features

Persistent Log

Append-only message log
Messages stored on disk
Survives Redis restart
Each message has unique ID

Consumer Groups

Multiple consumers per group
Load balancing within group
Independent group progress
Exactly-once semantics

Acknowledgments

Track pending messages (PEL)
Retry unacknowledged messages
Prevent message loss
Handle consumer failures

Random Access

Read from any position
Time-based queries
Message replay capability
Historical data access

Basic Operations

# Producer: Add message to stream
XADD mystream * orderId 123 amount 99.99
# Returns: "1638360000000-0" (timestamp-sequence ID)

# Consumer: Read new messages
XREAD COUNT 1 STREAMS mystream $
# $ means "only new messages from now on"

Independent Consumption

Each consumer independently reads the stream:

# Consumer 1 reads from beginning
XREAD COUNT 10 STREAMS mystream 0

# Consumer 2 reads only new messages
XREAD COUNT 10 STREAMS mystream $

# Blocking read (like BRPOP)
XREAD BLOCK 5000 COUNT 1 STREAMS mystream $
# Blocks up to 5 seconds waiting for new messages

Special ID $: Represents the maximum ID currently in the stream. Use it to read only messages that arrive after you start listening.

Consumer Groups - The Power Feature

Consumer groups enable distributed processing like Kafka:Key concepts:

Consumer Group Components

Consumer Group: Named group of consumers
- Has independent position (last_delivered_id)
- Tracks which messages delivered to whom
- Multiple groups can read same stream
PEL (Pending Entries List):
- Per-consumer list of unacknowledged messages
- Grows when messages read
- Shrinks when messages acknowledged
- Enables failure recovery
Consumer: Individual processor within group
- Gets different messages (load balanced)
- Must acknowledge after processing
- Can claim abandoned messages from failed consumers

Consumer Group Commands

# 1. Create consumer group
XGROUP CREATE mystream email-group 0
#                                   ^
#                                   Start from beginning (0)
#                                   Use $ for "only new messages"

# 2. Read as part of group
XREADGROUP GROUP email-group consumer1 COUNT 1 STREAMS mystream >
#                                                                ^
#                                                                Read undelivered messages

# Returns:
# 1) 1) "mystream"
#    2) 1) 1) "1638360000000-0"
#          2) 1) "orderId"
#             2) "123"
#             3) "amount"
#             4) "99.99"

# 3. Acknowledge successful processing
XACK mystream email-group 1638360000000-0
# Returns: (integer) 1

Comparison: Independent vs. Group Consumption

Independent (Fan-Out)
Consumer Group (Load Balanced)

Pattern: Every consumer gets every message

# Each consumer reads independently
messages = redis.xread(
    count=10,
    streams={'mystream': last_id}
)

Use case: Broadcasting events to multiple services✅ Each service processes all events ❌ No load distribution

Pattern: Messages distributed among group members

# Multiple workers share the load
messages = redis.xreadgroup(
    groupname='worker-group',
    consumername='worker-1',
    count=10,
    streams={'mystream': '>'}
)

Use case: Parallel processing of tasks✅ Horizontal scaling ✅ Load distribution ✅ Fault tolerance

Handling Failures

Consumer crashes

Messages remain in PEL (unacknowledged)

Monitor pending messages

# View pending messages for a consumer
XPENDING mystream worker-group - + 10 consumer1

Claim abandoned messages

# Another consumer takes over
XCLAIM mystream worker-group consumer2 3600000 1638360000000-0
#                                       ^
#                                       Min idle time (1 hour)

Process and acknowledge

# After successful processing
XACK mystream worker-group 1638360000000-0

Stream Management

# Limit stream size (automatic trimming)
XADD mystream MAXLEN ~ 10000 * field value
#                     ^
#                     ~ means approximate (more efficient)

# Delete specific messages
XDEL mystream 1638360000000-0

# Get stream info
XINFO STREAM mystream

# Get consumer group info
XINFO GROUPS mystream

# Get consumers in group
XINFO CONSUMERS mystream worker-group

No Built-In Partitioning

Difference from Kafka: Redis Streams don’t have built-in partitioning.To achieve Kafka-like partitioning:

# Client-side routing based on key
def get_stream_name(order_id):
    partition = hash(order_id) % NUM_PARTITIONS
    return f"orders:partition:{partition}"

# Send to appropriate partition
stream = get_stream_name(order['id'])
redis.xadd(stream, order)

Production Considerations

Memory Management

Streams grow indefinitely - use trimming:

# Keep only recent messages
XTRIM mystream MAXLEN ~ 100000

# Or use TTL strategy
XADD mystream MAXLEN ~ 10000 * data value

Monitoring

Track key metrics:

Stream length (XLEN)
Consumer lag (XPENDING)
Processing rate
Memory usage

When to Use Streams

✅ Perfect for:

Event sourcing
Activity feeds
Sensor data collection
Real-time analytics pipelines
Task queues with persistence

⚠️ Consider alternatives for:

Multi-datacenter replication (use Kafka)
Massive throughput >1M msg/s (use Kafka/Pulsar)
Complex routing rules (use RabbitMQ)
Guaranteed cross-region delivery (use cloud-native MQ)

Redis MQ Pattern Comparison

Feature	List (BRPOP)	List (BRPOPLPUSH)	Pub/Sub	Streams
Persistence	✅ Yes	✅ Yes	❌ No	✅ Yes
At-Least-Once	❌ No	⚠️ With monitoring	❌ No	✅ Yes (ACK)
Multiple Consumers	❌ Competing	❌ Competing	✅ Broadcast	✅ Both modes
Consumer Groups	❌ No	❌ No	❌ No	✅ Yes
Message Replay	❌ No	❌ No	❌ No	✅ Yes
Blocking Read	✅ Yes	✅ Yes	✅ Yes	✅ Yes
Complexity	Low	Medium	Low	High
Best For	Simple queues	Reliable queues	Events	Production MQ

Design Considerations

Idempotency

Handle duplicate messages gracefully

# Use unique message ID
processed_ids = set()

def process_message(msg_id, data):
    if msg_id in processed_ids:
        return  # Already processed
    
    # Do work...
    processed_ids.add(msg_id)

Store processed IDs in Redis with expiration

Message TTL

Prevent queue bloat

# Add timestamp to messages
message = {
    'data': payload,
    'timestamp': time.time()
}

# Consumer checks age
age = time.time() - msg['timestamp']
if age > MAX_AGE:
    # Discard or move to DLQ
    pass

Error Handling

Implement dead-letter queue

MAX_RETRIES = 3

try:
    process(message)
except Exception as e:
    retries = get_retry_count(msg_id)
    if retries < MAX_RETRIES:
        requeue(message)
    else:
        move_to_dlq(message, error=str(e))

Monitoring

Track queue healthKey metrics:

Queue depth (backlog)
Processing rate
Error rate
Consumer lag
Message age

Production Message Queue Alternatives

When Redis isn’t enough:For mission-critical message processing with strict reliability requirements, use dedicated message queue systems:

RabbitMQ

Best for: Complex routing, traditional MQ patterns✅ Rich routing (exchanges, bindings) ✅ Strong delivery guarantees ✅ Management UI ✅ Multi-protocol (AMQP, MQTT, STOMP)❌ Lower throughput than Kafka ❌ More operational complexity

Apache Kafka

Best for: High-throughput event streaming✅ Massive scalability (millions msg/s) ✅ Long-term storage ✅ Stream processing (Kafka Streams) ✅ Distributed partitioning❌ Complex setup ❌ Requires ZooKeeper (pre-3.x)

RocketMQ

Best for: E-commerce, financial systems✅ Designed for transactional messages ✅ Scheduled/delayed messages ✅ Alibaba battle-tested ✅ Lower latency than Kafka❌ Smaller ecosystem ❌ Less documentation (English)

Decision Matrix

Use Redis When...
Use Professional MQ When...

✅ Good fit for Redis MQ:

Already using Redis for caching
Simple queue requirements
Low-to-medium message volume (less than 10K msg/s)
Message loss tolerance (Pub/Sub) or using Streams
Need minimal infrastructure
Development/testing environments

Example use cases:

Email/SMS notification queues
Background job processing
Cache invalidation
Real-time activity feeds

Best Practices Summary

Choose the right pattern

Simple tasks → List with BRPOP
Reliable delivery → List with BRPOPLPUSH or Streams
Broadcasting → Pub/Sub (if loss acceptable) or Streams
Production critical → Redis Streams or dedicated MQ

Design for failure

Implement idempotent processing
Use acknowledgments (Streams)
Set up dead-letter queues
Monitor and alert on queue depth

Manage resources

Trim streams/lists to prevent memory bloat
Set message TTLs
Monitor Redis memory usage
Plan for scaling (add consumers, partition streams)

Test failure scenarios

Consumer crashes mid-processing
Network partitions
Redis restarts
Message replay after failures

Load Balancing

Distribute synchronous traffic across servers

Service Discovery

Dynamic service location for distributed messaging

Java & Spring

Databases

Algorithms & Data Structures

System Design

​Message Queue Fundamentals

​What is a Message Queue?

​Core Components

Producer

Message Broker

Consumer

​Why Use Message Queues?

Real Example: E-commerce Order

Peak Shaving Example

​Reliability Guarantees

Durability

At-Least-Once Delivery

Ordering

​Redis as Message Queue

​Production-Consumption with Redis Lists

​Approach 1: LPUSH + RPOP

​Approach 2: LPUSH + BRPOP (Recommended)

​Approach 3: LPUSH + LRANGE + RPOP

​Approach 4: LPUSH + BRPOPLPUSH + LREM (Most Reliable)

How It Works

​Summary: List-Based Patterns

​Publish-Subscribe Pattern

​Basic Commands

​Pattern Matching with PSUBSCRIBE

​Use Cases

Event Broadcasting

Real-Time Messaging

​Critical Limitations

When NOT to use Pub/Sub

​Internal Implementation

​Best Practices

​Redis Streams - Production-Grade MQ

​Key Features

Persistent Log

Consumer Groups

Acknowledgments

Random Access

​Basic Operations

​Independent Consumption

​Consumer Groups - The Power Feature

​Consumer Group Commands

​Comparison: Independent vs. Group Consumption

​Handling Failures

​Stream Management

​No Built-In Partitioning

​Production Considerations

Memory Management

Monitoring

​When to Use Streams

​Redis MQ Pattern Comparison

​Design Considerations

Idempotency

Message TTL

Error Handling

Monitoring

​Production Message Queue Alternatives

RabbitMQ

Apache Kafka

RocketMQ

​Decision Matrix

​Best Practices Summary

​Related Topics

Load Balancing

Service Discovery

Build docs developers (and LLMs) love

Message Queue Fundamentals

What is a Message Queue?

Core Components

Why Use Message Queues?

Reliability Guarantees

Redis as Message Queue

Production-Consumption with Redis Lists

Approach 1: LPUSH + RPOP

Approach 2: LPUSH + BRPOP (Recommended)

Approach 3: LPUSH + LRANGE + RPOP

Approach 4: LPUSH + BRPOPLPUSH + LREM (Most Reliable)

Summary: List-Based Patterns

Publish-Subscribe Pattern

Basic Commands

Pattern Matching with PSUBSCRIBE

Use Cases

Critical Limitations

Internal Implementation

Best Practices

Redis Streams - Production-Grade MQ

Key Features

Basic Operations

Independent Consumption

Consumer Groups - The Power Feature

Consumer Group Commands

Comparison: Independent vs. Group Consumption

Handling Failures

Stream Management

No Built-In Partitioning

Production Considerations

When to Use Streams

Redis MQ Pattern Comparison

Design Considerations

Production Message Queue Alternatives

Decision Matrix

Best Practices Summary

Related Topics