Redis Caching System

Status: ✅ Production Ready Version: 1.0.0 Last Updated: November 2, 2025

Overview

Noumaris uses Redis for distributed caching to improve API response times and reduce database load. The caching system implements a three-layer hierarchy with graceful degradation when Redis is unavailable.

Key Features

✅ Distributed caching - Shared across all worker processes
✅ Graceful degradation - App continues working if Redis goes down
✅ Automatic invalidation - Cache cleared when data changes
✅ Connection pooling - Efficient connection reuse
✅ Health monitoring - Built-in health checks and metrics
✅ TTL-based expiration - Automatic cache expiry (5 minutes)

Architecture

Three-Layer Cache Hierarchy

┌─────────────────────────────────────────┐
│          API Request                    │
└─────────────────────────────────────────┘
                  ↓
┌─────────────────────────────────────────┐
│ Layer 1: Redis Cache (distributed)     │
│ • Shared across all workers             │
│ • TTL: 5 minutes                        │
│ • Speed: 1-2ms                          │
└─────────────────────────────────────────┘
                  ↓ (if miss)
┌─────────────────────────────────────────┐
│ Layer 2: In-Memory Cache (process-local│
│ • Per-worker cache                      │
│ • TTL: 5 minutes                        │
│ • Speed: 0.1ms                          │
└─────────────────────────────────────────┘
                  ↓ (if miss)
┌─────────────────────────────────────────┐
│ Layer 3: PostgreSQL (source of truth)  │
│ • Authoritative data                    │
│ • Speed: 10-20ms                        │
└─────────────────────────────────────────┘

Graceful Degradation

If Redis becomes unavailable, the system automatically falls back:

Redis available → Use Redis cache (fastest)
Redis down → Use in-memory cache (still fast)
Both caches miss → Query database (slower but reliable)

Critical: The app never crashes due to Redis issues. It just runs slightly slower.

Configuration

Environment Variables

bash

# Optional - Redis will use defaults if not set
REDIS_HOST=localhost        # Default: localhost
REDIS_PORT=6379            # Default: 6379
REDIS_DB=0                 # Default: 0 (database number)
REDIS_PASSWORD=            # Default: none (for local dev)

Docker Compose Configuration

Located in /docker-compose.yml:

yaml

redis:
  image: redis:7-alpine
  container_name: redis_cache
  command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
  ports:
    - "6379:6379"
  volumes:
    - redis_data:/data
  healthcheck:
    test: ["CMD", "redis-cli", "ping"]
    interval: 10s
    timeout: 3s
    retries: 3

Configuration Explained:

--appendonly yes - Persist data to disk (survives restarts)
--maxmemory 256mb - Limit memory usage to 256MB
--maxmemory-policy allkeys-lru - Evict least recently used keys when full

Usage

Basic Usage

python

from noumaris_backend.database import get_redis_client

# Get Redis client (singleton)
redis = get_redis_client()

# Check if Redis is available
if redis.is_available:
    print("✅ Redis is ready")
else:
    print("⚠️  Redis unavailable - using fallback")

# Store data with TTL
redis.set_json("user:123", {"name": "John"}, ttl=300)  # 5 minutes

# Retrieve data
user_data = redis.get_json("user:123")

# Delete cache
redis.delete("user:123")

# Check if key exists
if redis.exists("user:123"):
    print("Key exists")

# Get remaining TTL
ttl_seconds = redis.ttl("user:123")

Permission Service Integration

The Permission Service automatically uses Redis:

python

from noumaris_backend.services.permission_service import get_permission_service

# Redis client is automatically injected
service = get_permission_service(session)

# Permission check (uses Redis cache)
has_access = service.has_permission("user123", "clinical.scribe_full")

# Invalidate cache when permissions change
service.invalidate_user_cache("user123")

Advanced Usage

python

# Get Redis client with custom config
redis = get_redis_client(
    host="redis.example.com",
    port=6380,
    password="secret"
)

# Get server info
info = redis.get_info()
print(f"Redis version: {info['redis_version']}")
print(f"Memory used: {info['used_memory_human']}")

# Test connection
if redis.ping():
    print("Redis is responding")

# Flush database (DANGER - deletes all keys)
redis.flushdb()  # Use with extreme caution!

Performance Benchmarks

Before Redis (Database Only)

Operation	Queries	Time
Permission check	1	10-20ms
List 50 residents with permissions	50	500-1000ms
Resident dashboard load	3	30-60ms

After Redis (Cached)

Operation	Time	Improvement
Permission check (cached)	1-2ms	10x faster
List 50 residents (cached)	50-100ms	10x faster
Resident dashboard (cached)	3-5ms	10x faster

Cache Hit Rates

In production, we expect:

First request: Cache miss (10-20ms) - fetches from database
Subsequent requests (within 5 min): Cache hit (1-2ms) - from Redis
Typical hit rate: 85-95% for permission checks

Cache Invalidation

When Cache is Cleared

Cache is automatically invalidated when:

Permission granted/revoked → User's cache cleared
Institution features changed → All residents' cache cleared
Scribe access level updated → User's cache cleared
Manual invalidation → Specific user or all users

Invalidation Methods

python

# Invalidate single user
service.invalidate_user_cache("user123")

# Invalidate all residents in institution
service.invalidate_institution_resident_cache(institution_id=5)

# Invalidate ALL caches (use sparingly)
service.invalidate_all_cache()

Automatic Expiration

All cache entries automatically expire after 5 minutes (TTL).

This ensures:

Stale data doesn't persist forever
Memory usage stays bounded
Cache eventually consistent with database

Monitoring

Startup Logs

When the application starts, you'll see:

✅ Redis cache initialized successfully
Redis version: 7.2.0
Redis memory used: 1.2M

Or if Redis is unavailable:

⚠️  Redis unavailable - running without distributed cache

Debug Logging

Enable debug logging to see cache behavior:

python

import logging
logging.getLogger("noumaris_backend.services.permission_service").setLevel(logging.DEBUG)

You'll see:

Cache HIT (Redis) for user abc123
Cache HIT (memory) for user def456
Cache MISS for user xyz789, querying database

Health Checks

Check Redis health:

bash

# From command line
docker exec redis_cache redis-cli ping
# Output: PONG

# Check memory usage
docker exec redis_cache redis-cli INFO memory

# Check connected clients
docker exec redis_cache redis-cli CLIENT LIST

Troubleshooting

Redis Not Starting

Problem: Redis container won't start

Solutions:

bash

# Check if port 6379 is already in use
lsof -i :6379

# Check Docker logs
docker logs redis_cache

# Restart Redis
docker-compose restart redis

# Full reset (DELETES ALL CACHED DATA)
docker-compose down
docker volume rm noumaris_redis_data
docker-compose up -d redis

Cache Not Working

Problem: Changes not reflected immediately

Cause: Cache still has old data

Solutions:

Wait 5 minutes - Cache auto-expires

Manual invalidation:

bash

docker exec redis_cache redis-cli FLUSHDB

Restart application - Clears in-memory cache

Connection Errors

Problem: RedisConnectionError: Connection refused

Diagnosis:

bash

# Check Redis is running
docker ps | grep redis

# Test connection from host
redis-cli -h localhost -p 6379 ping

# Check firewall/network
telnet localhost 6379

Solutions:

Ensure Redis is running: docker-compose up -d redis
Check environment variables are correct
App will fall back to in-memory cache automatically

Memory Issues

Problem: Redis using too much memory

Check current usage:

bash

docker exec redis_cache redis-cli INFO memory

Solutions:

Redis is configured with 256MB limit and LRU eviction

Increase limit in docker-compose.yml:

yaml

command: redis-server --maxmemory 512mb --maxmemory-policy allkeys-lru

Manually flush old data: docker exec redis_cache redis-cli FLUSHDB

Production Deployment

Google Cloud Memorystore

For production, use managed Redis (Google Memorystore):

python

# Environment variables for production
REDIS_HOST=10.128.0.5           # Private IP
REDIS_PORT=6379
REDIS_PASSWORD=<secure-password>

Production Configuration

yaml

# cloudbuild.yaml or Cloud Run environment
env:
  - name: REDIS_HOST
    value: "${REDIS_HOST}"
  - name: REDIS_PASSWORD
    valueFrom:
      secretKeyRef:
        name: redis-password
        key: latest

High Availability

For critical workloads, consider:

Redis Sentinel - Automatic failover
Redis Cluster - Horizontal scaling
Read replicas - Distribute load

Security

Best Practices

✅ DO:

Use password authentication in production (REDIS_PASSWORD)
Limit network access (private VPC only)
Enable TLS for connections (Cloud Memorystore supports this)
Monitor access patterns
Set appropriate memory limits

❌ DON'T:

Expose Redis port publicly
Store sensitive data unencrypted
Use default Redis port in production (if possible)
Disable authentication
Allow FLUSHDB in production

Data Stored in Redis

Currently cached:

User permissions (feature IDs only, not sensitive data)
No PII, passwords, or confidential information

Cache keys format:

permissions:abc123    # User ID → Set of feature_ids

File Structure

backend/src/noumaris_backend/database/
├── __init__.py                  # Exports get_redis_client()
├── manager.py                   # DatabaseManager (PostgreSQL)
├── redis_client.py              # RedisClient implementation
└── REDIS_CACHING.md            # This file

backend/src/noumaris_backend/services/
└── permission_service.py        # Uses Redis for permission caching

API Reference

RedisClient Class

Full API documentation in redis_client.py.

Key Methods:

Method	Description	Returns
`get(key)`	Get string value	`str` or `None`
`set(key, value, ttl)`	Set string value	`bool`
`get_json(key)`	Get JSON value	`Any` or `None`
`set_json(key, value, ttl)`	Set JSON value	`bool`
`delete(*keys)`	Delete keys	`int` (count deleted)
`exists(*keys)`	Check if exists	`int` (count exists)
`expire(key, seconds)`	Set TTL	`bool`
`ttl(key)`	Get remaining TTL	`int` (seconds)
`ping()`	Test connection	`bool`
`flushdb()`	Clear all keys	`bool`
`get_info()`	Server stats	`dict` or `None`

FAQ

Q: What happens if Redis goes down? A: The app continues working normally, just slightly slower. It falls back to in-memory cache, then database.

Q: Do I need Redis for local development? A: No, it's optional. The app works fine without it. Redis provides better performance but isn't required.

Q: How do I disable Redis? A: Just don't start the Redis container. The app will detect it's unavailable and use in-memory cache.

Q: Can I clear the cache? A: Yes, either wait 5 minutes for auto-expiry, or manually flush: docker exec redis_cache redis-cli FLUSHDB

Q: Does Redis store sensitive data? A: No, only permission mappings (feature IDs). No PII, passwords, or confidential data.

Q: How much memory does Redis use? A: Typically 10-50MB for normal usage. Limited to 256MB max by configuration.

Q: Is the cache always consistent? A: Eventually consistent. Cache expires after 5 minutes. For immediate consistency, cache is invalidated when data changes.

Version History

1.0.0 (2025-11-02) - Initial release with permission caching
- Three-layer cache hierarchy
- Graceful degradation
- Automatic invalidation
- Connection pooling
- Health monitoring

/docs/architecture/backend.md - Backend architecture overview
/docs/guides/development.md - Development setup guide
/project-management/active/backend-improvements-2025.md - Performance optimization plan
redis_client.py - Complete Redis client implementation
permission_service.py - Permission service with caching

Questions or issues? Check the troubleshooting section or review the implementation in redis_client.py.

Redis Caching System ​

Overview ​

Key Features ​

Architecture ​

Three-Layer Cache Hierarchy ​

Graceful Degradation ​

Configuration ​

Environment Variables ​

Docker Compose Configuration ​

Usage ​

Basic Usage ​

Permission Service Integration ​

Advanced Usage ​

Performance Benchmarks ​

Before Redis (Database Only) ​

After Redis (Cached) ​

Cache Hit Rates ​

Cache Invalidation ​

When Cache is Cleared ​

Invalidation Methods ​

Automatic Expiration ​

Monitoring ​

Startup Logs ​

Debug Logging ​

Health Checks ​

Troubleshooting ​

Redis Not Starting ​

Cache Not Working ​

Connection Errors ​

Memory Issues ​

Production Deployment ​

Google Cloud Memorystore ​

Production Configuration ​

High Availability ​

Security ​

Best Practices ​

Data Stored in Redis ​

File Structure ​

API Reference ​

RedisClient Class ​

FAQ ​

Version History ​

Related Documentation ​

Redis Caching System

Overview

Key Features

Architecture

Three-Layer Cache Hierarchy

Graceful Degradation

Configuration

Environment Variables

Docker Compose Configuration

Usage

Basic Usage

Permission Service Integration

Advanced Usage

Performance Benchmarks

Before Redis (Database Only)

After Redis (Cached)

Cache Hit Rates

Cache Invalidation

When Cache is Cleared

Invalidation Methods

Automatic Expiration

Monitoring

Startup Logs

Debug Logging

Health Checks

Troubleshooting

Redis Not Starting

Cache Not Working

Connection Errors

Memory Issues

Production Deployment

Google Cloud Memorystore

Production Configuration

High Availability

Security

Best Practices

Data Stored in Redis

File Structure

API Reference

RedisClient Class

FAQ

Version History

Related Documentation