Skip to main content

ManuScript

The unified event schema for distributed collaboration at scale. ManuScript is the Protocol Buffer schema system that serves as the single source of truth for all events flowing through the Materi collaboration platform. It enables type-safe, auditable, and replayable event processing across 7+ microservices handling 95B+ events per year.

Features

Production Ready
  • Compiled Protocol Buffers with Go, TypeScript, and Python bindings
  • Type-safe event definitions with protoc
  • 50+ domain-specific event types across 8 domains
  • Beautiful TUI explorer for schema introspection
  • Published packages: github.com/materi/domain/proto (Go), @materi/proto (npm), materi-proto (PyPI)
🎯 Competitive Advantages
  • Unified Schema: Single source of truth vs fragmented implicit schemas
  • Introspectable Metadata: Custom proto options enable runtime inspection
  • Deterministic Replay: Complete event history replay for debugging
  • Forensic Auditing: Immutable audit trails with strong consistency guarantees
  • Distributed Tracing: Built-in trace correlation across services
  • Causal Ordering: Hybrid Logical Clock (HLC) for deterministic ordering
🔒 Designed for Scale
  • Supports 95B+ events/year across 7+ services
  • Redis stream integration for high-throughput publishing
  • Consumer group management with offset tracking
  • 30-90 day retention policies with compliance metadata

Architecture

Installation

Go

TypeScript / JavaScript

Python

Development Setup

Prerequisites:
  • Go 1.22+
  • Node.js 18+
  • Python 3.11+
  • protoc v3.0+ (install via brew install protobuf on macOS)

Usage

MSX Inspector - Interactive Schema Explorer

Controls:
  • ↑/↓ or j/k - Navigate between domains
  • Enter - View event details
  • h - Return home
  • q - Quit
Features:
  • 📋 Browse all domains (user, document, collaboration, etc.)
  • 🔍 View event metadata (consumers, stream names, retention)
  • 🏷️ See which services consume each event
  • ✅ Check audit & replay flags for compliance
  • 📊 View schema statistics

Programmatic Usage

Event Domains

User (12 events)

  • Account creation, authentication, profile updates
  • Consumers: Shield, API, Aria, Folio

Document (6 events)

  • Document lifecycle (create, update, delete, archive)
  • Consumers: API, Relay, Aria, Folio

Collaboration (5 events)

  • Real-time collaborative editing synchronization
  • Consumers: Relay, API, Folio

Workspace (5 events)

  • Team management, workspace creation, settings
  • Consumers: API, Shield, Folio

Aria (AI Safety) (5 events)

  • AI analysis, safety gate decisions, content moderation
  • Consumers: Aria, API, Folio

Notification (4 events)

  • Email, webhooks, in-app notifications, SMS
  • Consumers: Shield, API, Folio

Audit (3 events)

  • Security events, compliance logs, forensic trails
  • Consumers: Arena, Shield, Folio

Metadata & Custom Options

Every event includes enriched metadata for introspection:

Compilation

All Languages

Individual Languages

Build Packages

Version Management

Watch Mode

Generated Output:
  • Go: proto/materi/v1/*.pb.go (10 files)
  • TypeScript: ts/generated/domains/*.ts (12 files)
  • Python: py/src/materi_proto/domains/*_pb2.py (9 files + stubs)

Testing

Go

TypeScript

Python

Test Coverage:
  • Schema registry initialization & retrieval
  • Event metadata validation
  • Consumer mapping
  • Stream routing
  • Serialization/deserialization round-trips
  • Type safety verification

Distributed Tracing Support

Every DomainEvent includes trace correlation fields for observability:
Integrates with:
  • Folio (centralized observability hub)
  • Prometheus (metric collection)
  • Jaeger (distributed tracing)

Deployment

Docker

Kubernetes

CI/CD

GitHub Actions automatically:
  • ✅ Compile proto definitions for Go, TypeScript, and Python on every PR
  • Drift detection - fails if generated code is out of sync
  • ✅ Run test suite for all languages
  • ✅ Build and type-check packages
  • ✅ Publish to npm and PyPI on release

Workflows

Manual Publishing

See .github/workflows/ for configuration

Performance

Benchmarks (Go on M2 MacBook Pro):
  • Schema registry initialization: ~100µs
  • Event lookup by type: ~50ns
  • Domain retrieval: ~1µs
  • Consumer mapping: ~500ns
Memory Usage:
  • Registry with 50 events: ~2MB
  • Full descriptor set: ~500KB

Competitive Analysis

Contributing

Add a New Event Type

  1. Define in .msx/domains/{domain}/v1.proto
  2. Add metadata options (consumers, stream, retention)
  3. Regenerate all languages:
  4. Update re-exports:
    • Go: proto/wrapper.go
    • TypeScript: ts/src/index.ts
    • Python: py/src/materi_proto/__init__.py
  5. Add tests and commit generated files

Add a New Domain

  1. Create .msx/domains/{new-domain}/v1.proto
  2. Reference envelope and shared types
  3. Update Makefile to include the new proto file
  4. Regenerate with make compile-all
  5. Update all re-export wrappers

Update Package Version

Proto File Guidelines

  • Always include go_package option pointing to github.com/materi/domain/proto/materi/v1/{domain}
  • Use semantic versioning in directory names (v1, v2, etc.)
  • Add comprehensive field comments for documentation
  • Include metadata options for introspection

Roadmap

  • ✅ Phase 1: Proto schema + barrel file architecture (COMPLETE)
  • ✅ Phase 2: Custom metadata options (COMPLETE)
  • ✅ Phase 3: Go bindings compilation (COMPLETE)
  • ✅ Phase 4: TypeScript bindings + npm package (COMPLETE)
  • ✅ Phase 5: Python bindings + PyPI package (COMPLETE)
  • ✅ Phase 6: Multi-language CI/CD (COMPLETE)
  • 🔄 Phase 7: Bubble Tea TUI inspector (IN PROGRESS)
  • ⏳ Phase 8: gRPC reflection service
  • ⏳ Phase 9: JSON schema generation
  • ⏳ Phase 10: Event migration/versioning tools

Support

Documentation: Issues & Questions: Open an issue on GitHub or contact @devops-devil

License

Proprietary - Materi Platform Internal Use Only
Made with ❤️ for distributed collaboration at scale MSX v0.1.0 | Protocol Buffers 3 | Go 1.22 | TypeScript 5.x | Python 3.11+