Redis Beyond Caching: Streams, Pub/Sub, and Data Structures

Redis: More Than a Cache

Most developers know Redis as a cache. Set a key, get a key, maybe set an expiration. But Redis is a full-featured data structure server with capabilities that solve problems caching alone cannot.

Understanding Redis's advanced features opens up new architectural possibilities: real-time messaging, distributed coordination, rate limiting, leaderboards, and event streaming - all with sub-millisecond latency.

Redis Data Structures

Each data structure is optimized for specific access patterns:

Strings: The basics - caching, counters, distributed locks. Atomic operations like INCR make counters safe without transactions.

Hashes: Object storage with field-level access. Get or update individual fields without reading the entire object.

Lists: Ordered sequences. Efficient push/pop from both ends. Perfect for queues and recent items lists.

Sets: Unique collections with O(1) membership testing. Set operations (union, intersection, difference) enable powerful queries.

Sorted Sets: Sets with scores. Automatically sorted by score. Ideal for leaderboards, priority queues, and time-series with timestamp scores.

HyperLogLog: Probabilistic counting of unique items using minimal memory. Count billions of unique visitors with 12KB.

Redis Streams

Streams bring Kafka-like event streaming to Redis. Messages are persisted in an append-only log. Consumer groups enable parallel processing with exactly-once delivery semantics.

Unlike Pub/Sub, Streams persist messages. Consumers can read history, resume from where they left off, and acknowledge processing. Failed messages can be reclaimed and reprocessed.

Use Streams for event sourcing, activity feeds, and any scenario where you need durable, ordered message delivery without deploying Kafka.

Pub/Sub for Real-Time

Pub/Sub provides fire-and-forget messaging. Publishers send to channels. All subscribers receive messages in real-time. No persistence - if a subscriber is offline, messages are lost.

Perfect for real-time notifications, cache invalidation, and live updates where missing a message is acceptable. For guaranteed delivery, use Streams instead.

Lua Scripting for Atomicity

Redis is single-threaded - commands execute atomically. But multi-step operations need Lua scripts to maintain atomicity.

Classic example: rate limiting. Check current count, increment, set expiration if new - all must happen atomically. A Lua script executes as a single operation, preventing race conditions.

Scripts are cached by SHA hash. Send the script once, then call by hash for efficiency. Keep scripts small and focused.

Clustering and High Availability

Redis Sentinel: Automatic failover for master-replica setups. Monitors health, elects new master on failure, updates clients.

Redis Cluster: Horizontal scaling across multiple nodes. Data is sharded by key hash. Adds complexity but enables datasets larger than one machine's memory.

Choose based on your needs: Sentinel for high availability with moderate data size, Cluster when you need to scale beyond a single node.

Best Practices

Use pipelines: Batch commands to reduce round trips

Choose persistence wisely: RDB for snapshots, AOF for durability

Set maxmemory policy: Define behavior when memory is full

Monitor memory: Redis is in-memory - plan capacity

Use appropriate data structures: Match structure to access pattern

Lua for atomicity: Multi-step operations need scripts