seaweedfs/test/kafka/README.md

Kafka Integration Testing with Docker Compose

This directory contains comprehensive integration tests for SeaweedFS Kafka Gateway using Docker Compose to set up all required dependencies.

Overview

The Docker Compose setup provides:

• Apache Kafka with Zookeeper
• Confluent Schema Registry for schema management
• SeaweedFS complete stack (Master, Volume, Filer, MQ Broker, MQ Agent)
• Kafka Gateway that bridges Kafka protocol to SeaweedFS MQ
• Test utilities for schema registration and data setup

Quick Start

Prerequisites

• Docker and Docker Compose
• Go 1.21+
• Make (optional, for convenience)

Basic Usage

1. Start the environment:

   make setup-schemas
   

   This starts all services and registers test schemas.

2. Run integration tests:

   make test-integration
   

3. Run end-to-end tests:

   make test-e2e
   

4. Clean up:

   make clean
   

Architecture

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Kafka Client  │───▶│  Kafka Gateway   │───▶│   SeaweedFS MQ  │
│   (Sarama/      │    │  (Port 9093)     │    │   (Broker/Agent)│
│    kafka-go)    │    │                  │    │                 │
└─────────────────┘    └──────────────────┘    └─────────────────┘
                              │
                              ▼
                       ┌──────────────────┐
                       │ Schema Registry  │
                       │  (Port 8081)     │
                       └──────────────────┘
                              │
                              ▼
                       ┌──────────────────┐
                       │  Native Kafka    │
                       │  (Port 9092)     │
                       └──────────────────┘

Services

Core Services

Service	Port	Description
Zookeeper	2181	Kafka coordination
Kafka	9092	Native Kafka broker
Schema Registry	8081	Confluent Schema Registry
SeaweedFS Master	9333	SeaweedFS master server
SeaweedFS Volume	8080	SeaweedFS volume server
SeaweedFS Filer	8888	SeaweedFS filer server
SeaweedFS MQ Broker	17777	SeaweedFS message queue broker
SeaweedFS MQ Agent	16777	SeaweedFS message queue agent
Kafka Gateway	9093	Kafka protocol gateway to SeaweedFS

Test Services

Service	Description
test-setup	Registers schemas and sets up test data
kafka-producer	Creates topics and produces test messages
kafka-consumer	Consumes messages for verification

Available Tests

Unit Tests

make test-unit

Tests individual Kafka components without external dependencies.

Integration Tests

make test-integration

Tests with real Kafka, Schema Registry, and SeaweedFS services.

Schema Tests

make test-schema

Tests schema registry integration and schema evolution.

End-to-End Tests

make test-e2e

Complete workflow tests with all services running.

Performance Tests

make test-performance

Benchmarks and performance measurements.

Client-Specific Tests

make test-sarama      # IBM Sarama client tests
make test-kafka-go    # Segmentio kafka-go client tests

Development Workflow

Start Individual Components

# Start only Kafka ecosystem
make dev-kafka

# Start only SeaweedFS
make dev-seaweedfs

# Start Kafka Gateway
make dev-gateway

Debugging

# Show all service logs
make logs

# Show specific service logs
make logs-kafka
make logs-schema-registry
make logs-seaweedfs
make logs-gateway

# Check service status
make status

# Debug environment
make debug

Interactive Testing

# List Kafka topics
make topics

# Create a topic
make create-topic TOPIC=my-test-topic

# Produce messages interactively
make produce TOPIC=my-test-topic

# Consume messages
make consume TOPIC=my-test-topic

# Open shell in containers
make shell-kafka
make shell-gateway

Test Configuration

Environment Variables

Variable	Default	Description
KAFKA_BOOTSTRAP_SERVERS	localhost:9092	Kafka broker addresses
KAFKA_GATEWAY_URL	localhost:9093	Kafka Gateway address
SCHEMA_REGISTRY_URL	http://localhost:8081	Schema Registry URL
TEST_TIMEOUT	10m	Test timeout duration

Running Tests with Custom Configuration

KAFKA_BOOTSTRAP_SERVERS=localhost:9092 \
KAFKA_GATEWAY_URL=localhost:9093 \
SCHEMA_REGISTRY_URL=http://localhost:8081 \
go test -v ./test/kafka/ -run Integration

Test Schemas

The setup automatically registers these test schemas:

User Schema

{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"},
    {"name": "email", "type": ["null", "string"], "default": null}
  ]
}

User Event Schema

{
  "type": "record",
  "name": "UserEvent",
  "fields": [
    {"name": "userId", "type": "int"},
    {"name": "eventType", "type": "string"},
    {"name": "timestamp", "type": "long"},
    {"name": "data", "type": ["null", "string"], "default": null}
  ]
}

Log Entry Schema

{
  "type": "record",
  "name": "LogEntry",
  "fields": [
    {"name": "level", "type": "string"},
    {"name": "message", "type": "string"},
    {"name": "timestamp", "type": "long"},
    {"name": "service", "type": "string"},
    {"name": "metadata", "type": {"type": "map", "values": "string"}}
  ]
}

Troubleshooting

Common Issues

1. Services not starting:

   make status
   make debug
   

2. Port conflicts:

   • Check if ports 2181, 8081, 9092, 9093, etc. are available
   • Modify docker-compose.yml to use different ports if needed

3. Schema Registry connection issues:

   curl http://localhost:8081/subjects
   make logs-schema-registry
   

4. Kafka Gateway not responding:

   nc -z localhost 9093
   make logs-gateway
   

5. Test timeouts:

   • Increase TEST_TIMEOUT environment variable
   • Check service health with make status

Performance Tuning

For better performance in testing:

1. Increase Docker resources:

   • Memory: 4GB+
   • CPU: 2+ cores

2. Adjust Kafka settings:

   • Modify docker-compose.yml Kafka environment variables
   • Tune partition counts and replication factors

3. SeaweedFS optimization:

   • Adjust volume size limits
   • Configure appropriate replication settings

CI/CD Integration

GitHub Actions Example

name: Kafka Integration Tests

on: [push, pull_request]

jobs:
  kafka-integration:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-go@v3
        with:
          go-version: '1.21'
      - name: Run Kafka Integration Tests
        run: |
          cd test/kafka
          make ci-test          

Local CI Testing

# Run full CI test suite
make ci-test

# Run end-to-end CI tests
make ci-e2e

Contributing

When adding new tests:

1. Follow naming conventions:

   • Integration tests: TestDockerIntegration_*
   • Unit tests: Test*_Unit
   • Performance tests: TestDockerIntegration_Performance

2. Use environment variables:

   • Check for required environment variables
   • Skip tests gracefully if dependencies unavailable

3. Clean up resources:

   • Use unique topic names with timestamps
   • Clean up test data after tests

4. Add documentation:

   • Update this README for new test categories
   • Document any new environment variables or configuration

References

• Apache Kafka Documentation
• Confluent Schema Registry
• SeaweedFS Documentation
• IBM Sarama Kafka Client
• Segmentio kafka-go Client