​

Materi Go API Service

​

🏗️ Hybrid Collaboration Architecture

​

Implemented Features

• Collaboration Controller: 6 REST endpoints for collaboration operations (placeholders)
• Document Management: Full CRUD operations with version control
• AI Integration: Content generation, summarization, and intelligent suggestions
• Event System Foundation: Redis-based event publishing framework (simplified)
• Models & Types: Complete collaboration data models and operation types
• Standardized Logging: Consistent structured logging across all services

​

🚧 Architectural Work Remaining

• Service Layer Integration: Connect controllers to actual collaboration business logic
• Real-time WebSocket Sync: Implement operational transformation algorithms
• Permission Caching: High-performance authorization with Redis caching
• Cross-service Events: Full event-driven sync between API and Collaboration services
• Performance Optimization: Sub-50ms response times with intelligent caching

​

Current Service Responsibilities

• Document CRUD operations (basic implementation)
• AI-powered content generation and processing
• File upload and storage management (MinIO integration)
• Basic event publishing framework
• Collaboration REST endpoints (placeholder responses)

• Real-time sync to collaboration service
• Collaboration service integration for permission validation
• Bidirectional synchronization between HTTP and WebSocket operations
• Document permission caching and real-time cache invalidation
• Operational transformation algorithms

​

🔄 Collaboration Service Integration Status

​

Current Implementation (Type-Safe Placeholders)

// ✅ Implemented endpoints (with placeholder business logic)
POST   /documents/{id}/operations     // Apply collaborative operations
GET    /documents/{id}/presence      // Get active users in document
PUT    /documents/{id}/presence      // Update user presence/cursor
GET    /documents/{id}/sync-status   // Get document synchronization status
POST   /documents/{id}/broadcast     // Broadcast document updates
GET    /stats                        // Get collaboration statistics
POST   /webhook                      // Process collaboration webhooks

• ✅ Type-safe event structures and routing
• ✅ Redis integration framework
• 🚧 Business logic requires implementation

• ✅ Complete operation transformation types
• ✅ Document presence and sync models
• ✅ Vector clock and conflict resolution structures

​

🎯 Next Implementation Phase

1. Service Layer Development:
   • Implement actual collaboration business logic
   • Connect to operational transformation algorithms
   • Add Redis-based state management
2. Real-time Integration:
   • WebSocket connection management
   • Bidirectional sync with Rust collaboration service
   • Event-driven state synchronization
3. Performance Optimization:
   • Permission caching layer
   • Operational transform conflict resolution
   • Sub-50ms response time optimization

​

🔄 Event-Driven Architecture

​

Published Events

// Document events
DocumentCreatedEvent    // New document creation
DocumentUpdatedEvent    // Content or metadata changes
DocumentDeletedEvent    // Document deletion
DocumentSharedEvent     // Permission changes

// User events (when acting as proxy)
UserUpdatedEvent        // Profile changes via API

​

Consumed Events

// From Shield service
UserCreatedEvent        // New user registration
UserUpdatedEvent        // Profile updates
UserDeletedEvent        // Account deletion

// From Collaboration service
CollaborationSessionStartedEvent  // User joins document
OperationAppliedEvent            // Real-time edits

​

Event Flow Example

​

🚀 Getting Started

​

Prerequisites

• Go 1.21+
• Redis 7.0+
• PostgreSQL 15+
• Docker & Docker Compose

​

Development Setup

1. Clone and Install Dependencies
   Copy

   cd api
   go mod tidy
   
2. Configure Environment
   Copy

   cp .env.example .env
   # Edit .env with your configuration
   
3. Start Infrastructure
   Copy

   # Start event-driven infrastructure
   docker-compose -f ../docker-compose.event-driven.yml up -d
   
4. Run Database Migrations
   Copy

   go run cmd/migrate/main.go up
   
5. Start API Service
   Copy

   # Development mode
   go run cmd/api/main.go
   
   # With event publishing enabled (placeholder implementation)
   go run cmd/api/main.go --enable-events
   

​

🔧 Collaboration Development Notes

• All endpoints return valid responses for front-end integration
• Business logic returns mock data (documented in controller TODOs)
• Event publishing logs events but doesn’t implement full Redis streams
• Ready for service layer implementation without breaking changes

​

Event System Integration

• Documents are created, updated, or deleted
• User profiles are modified via API
• Document permissions change
• File uploads complete

• User synchronization from Shield service
• Collaboration state updates from Rust service

​

📊 Performance & Monitoring

​

Key Metrics

• Response Time: P95 < 100ms for document operations
• Throughput: 1000+ requests/second sustained
• Event Processing: Sub-50ms event publishing latency
• Cache Hit Rate: >90% for user context validation

​

Monitoring Endpoints

• Health Check: GET /health
• Metrics: GET /metrics (Prometheus format)
• Event Status: GET /admin/events/status

​

Performance Optimizations

• Database Connection Pooling: Optimized for concurrent requests
• Redis Caching: User context and document metadata caching
• Event Batching: Batch processing for high-volume operations
• Background Processing: Async AI processing and file operations

​

🔧 Configuration

​

Environment Variables

# Core API Configuration
PORT=8080
ENV=development
LOG_LEVEL=info

# Database Configuration
DATABASE_URL=postgresql://user:pass@localhost:5432/materi_api
DATABASE_MAX_CONNECTIONS=25
DATABASE_MAX_IDLE=10

# Redis Configuration (Events & Caching)
REDIS_URL=redis://localhost:6379
REDIS_PASSWORD=materi_redis_2024
REDIS_DB=0

# Event System Configuration
EVENT_CONSUMER_GROUP=api-service
EVENT_CONSUMER_NAME=api-service-1
ENABLE_EVENT_PUBLISHING=true
ENABLE_EVENT_CONSUMPTION=true

# External Services
SHIELD_BASE_URL=http://localhost:8001
COLLABORATION_BASE_URL=http://localhost:8002
MINIO_ENDPOINT=localhost:9000
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin

# AI Integration
OPENAI_API_KEY=your_openai_key
ANTHROPIC_API_KEY=your_anthropic_key
AI_RATE_LIMIT=100

# Authentication
JWT_SECRET=your_jwt_secret_key
JWT_EXPIRY=24h
API_GATEWAY_SHARED_SECRET=shared_secret

​

Event Configuration

# Event streams the API service subscribes to
event_streams:
    - 'topic:user.created'
    - 'topic:user.updated'
    - 'topic:user.deleted'
    - 'topic:collaboration.session.started'
    - 'topic:operation.applied'

# Event publishing configuration
event_publishing:
    batch_size: 10
    flush_interval: 100ms
    retry_attempts: 3
    dead_letter_queue: true

​

🧪 Testing

​

Unit Tests

go test ./...

​

Integration Tests

go test ./test/integration/...

​

Event System Tests

# Test event publishing
go test ./test/events/publisher_test.go

# Test event consumption
go test ./test/events/consumer_test.go

​

Load Testing

# Install k6
brew install k6

# Run load tests
k6 run test/load/api_load_test.js

​

🐳 Deployment

​

Docker Build

# Standard build
docker build -t materi/api:latest .

# Event-enabled build
docker build -f Dockerfile.events -t materi/api:events .

​

Docker Compose Deployment

# Deploy with event infrastructure
docker-compose -f docker-compose.event-driven.yml up -d

# Scale API service
docker-compose -f docker-compose.event-driven.yml up -d --scale api-service=3

​

Kubernetes Deployment

# Apply Kubernetes manifests
kubectl apply -f k8s/

# Check deployment status
kubectl get pods -l app=materi-api

​

📈 Scaling & Operations

​

Horizontal Scaling

• Stateless Design: No local state, all data in databases/Redis
• Event Consumer Groups: Multiple instances process events in parallel
• Database Connection Pooling: Shared connection management
• Load Balancer Ready: Health checks and graceful shutdown

​

Event Processing at Scale

// Consumer group configuration for scaling
consumerConfig := EventConsumerConfig{
    Group:     "api-service",
    Consumers: []string{
        "api-service-1",
        "api-service-2",
        "api-service-3",
    },
    BatchSize: 10,
    Concurrency: 4,
}

​

Operational Considerations

• Graceful Shutdown: 30-second grace period for event processing
• Circuit Breakers: Protection against downstream service failures
• Rate Limiting: Per-user and global rate limits
• Database Migrations: Zero-downtime schema changes
• Event Replay: Ability to replay events for data recovery

​

🔒 Security

​

API Security

• JWT token validation with Redis caching
• Request rate limiting and DDoS protection
• Input validation and sanitization
• SQL injection prevention with parameterized queries

​

Event Security

• Redis AUTH for event stream access
• Event payload encryption for sensitive data
• Consumer group isolation
• Audit trail for all published events

​

📚 API Documentation

​

Core Endpoints

• Documents: GET|POST|PUT|DELETE /api/v1/documents
• AI Generation: POST /api/v1/ai/generate
• File Uploads: POST /api/v1/files
• User Context: GET /api/v1/users/me

​

Event Management

• Event Status: GET /api/v1/admin/events/status
• Consumer Health: GET /api/v1/admin/events/consumers
• Event Replay: POST /api/v1/admin/events/replay

​

🤝 Contributing

1. Fork the repository
2. Create a feature branch: git checkout -b feature/event-driven-sync
3. Make changes and add tests
4. Ensure all tests pass: go test ./...
5. Test event integration: docker-compose -f docker-compose.event-driven.yml up
6. Submit a pull request

​

🆘 Troubleshooting

​

Common Issues

# Check Redis connectivity
redis-cli -u redis://localhost:6379 ping

# Monitor event streams
redis-cli XINFO STREAM materi:events:all

# Check Go memory stats
curl http://localhost:8080/debug/pprof/heap

# Monitor Redis memory
redis-cli INFO memory

# Check connection pool status
curl http://localhost:8080/admin/db/status

# Test database connectivity
go run cmd/migrate/main.go status

​

Performance Debugging

# Enable Go profiling
go tool pprof http://localhost:8080/debug/pprof/profile

# Monitor event processing lag
curl http://localhost:8080/admin/events/lag

# Check API response times
curl -w "@curl-format.txt" http://localhost:8080/api/v1/documents

​

Collaboration Service Development

# Test collaboration endpoints (returns placeholder data)
curl http://localhost:8080/api/v1/collaboration/documents/{id}/presence
curl -X POST http://localhost:8080/api/v1/collaboration/documents/{id}/operations \
     -H "Content-Type: application/json" \
     -d '{"operations": [{"type": "insert", "position": 0, "content": "test"}]}'

1. Phase 1: Service layer business logic (operational transformation)
2. Phase 2: Redis state management and event streaming
3. Phase 3: WebSocket integration with Rust collaboration service
4. Phase 4: Performance optimization and caching