seaweedfs/KAFKA_GATEWAY_PLAN.md

Kafka Gateway Development Plan

Overview

Implementation plan for SeaweedFS Kafka Gateway feature completion, organized by difficulty and dependencies.

Phase 0: Quick Wins and Hygiene (2-3 days)

Goal: Foundation cleanup and basic test coverage

Tasks

• Unit tests: gateway connection/refusal in test/kafka/unit/gateway_test.go
• Schema Registry connectivity test in test/kafka/integration/docker_test.go
• Offsets: legacy key parsing in weed/mq/kafka/offset/smq_storage.go
• Offsets: leaderEpoch placeholder (return 0 instead of -1)
• ApiVersions audit: align return set with validation
• Admin polish: error codes in DeleteTopics/CreateTopics
• Logging: replace fmt.Printf with structured logging
• Throttle: consistent throttle_time_ms across responses

Exit Criteria: All unit/integration tests green, no debug prints, consistent throttling

Phase 1: Data Plane + Admin Visibility (1 week)

Goal: Core fetch/produce robustness and group management

Tasks

• Fetch: record batch concatenation in fetch.go:getMultipleRecordBatches
• Fetch: GZIP compression implementation
• Group introspection: DescribeGroups (15) and ListGroups (16) handlers
• Client compatibility tests expansion
• Metrics: basic request/error/latency tracking

Exit Criteria: Multi-batch fetch works, gzip validated, group introspection functional

Phase 2: Consumer Groups Robustness (1 week)

Goal: Stable consumer group rebalancing

Tasks

• Rebalancing test with new framework
• Cooperative-sticky assignment implementation (feature-flagged)
• Consumer group state management hardening

Exit Criteria: Deterministic rebalances, cooperative-sticky tests passing

Phase 3: Schema Management Readiness (2 weeks)

Goal: Schema validation and encoding fixes

Tasks

• Avro union re-encoding fix (re-enable skipped tests)
• Protobuf index varints + descriptor field APIs
• Schema topic detection and metadata lookup
• Produce: per-topic schema validation

Exit Criteria: Schema decode/encode tests pass, produce validation enabled

Phase 4: Schematized Fetch Integration (1-2 weeks)

Goal: End-to-end schema round-trip

Tasks

• fetchSchematizedRecords SeaweedMQ integration
• Batch reconstruction with compression/CRC
• End-to-end schema tests

Exit Criteria: Round-trip produce/fetch for all schema formats

Risk Mitigation

• Avro fix blocks schema features - prioritize early
• Cooperative-sticky can be feature-flagged
• Compression beyond gzip can be deferred
• Performance targets: 100k messages, p99 < reasonable latency

Release Criteria

• All test suites green (unit/integration/e2e/client-compat)
• Consumer groups stable under churn
• Schema topics round-trip correctly
• No debug logging in production paths