seaweedfs/test/kafka/simple-publisher/README.md

Simple Kafka-Go Publisher for SeaweedMQ

This is a simple publisher client that demonstrates publishing raw messages to SeaweedMQ topics with "_" prefix, which bypass schema validation.

Features

• Schema-Free Publishing: Topics with "_" prefix don't require schema validation
• Raw Message Storage: Messages are stored in a "value" field as raw bytes
• Multiple Message Formats: Supports JSON, binary, and empty messages
• Kafka-Go Compatible: Uses the popular kafka-go library

Prerequisites

1. SeaweedMQ Running: Make sure SeaweedMQ is running on localhost:17777 (default Kafka port)
2. Go Modules: The project uses Go modules for dependency management

Setup and Run

# Navigate to the publisher directory
cd test/kafka/simple-publisher

# Download dependencies
go mod tidy

# Run the publisher
go run main.go

Expected Output

Publishing messages to topic '_raw_messages' on broker 'localhost:17777'
Publishing messages...
- Published message 1: {"id":1,"message":"Hello from kafka-go client",...}
- Published message 2: {"id":2,"message":"Raw message without schema validation",...}
- Published message 3: {"id":3,"message":"Testing SMQ with underscore prefix topic",...}

Publishing different raw message formats...
- Published raw message 1: key=binary_key, value=Simple string message
- Published raw message 2: key=json_key, value={"raw_field": "raw_value", "number": 42}
- Published raw message 3: key=empty_key, value=
- Published raw message 4: key=, value=Message with no key

All test messages published to topic with '_' prefix!
These messages should be stored as raw bytes without schema validation.

Topic Naming Convention

• Schema-Required Topics: user-events, orders, payments (require schema validation)
• Schema-Free Topics: _raw_messages, _logs, _metrics (bypass schema validation)

The "_" prefix tells SeaweedMQ to treat the topic as a system topic and skip schema processing entirely.

Message Storage

For topics with "_" prefix:

• Messages are stored as raw bytes without schema validation
• No Confluent Schema Registry envelope is required
• Any binary data or text can be published
• SMQ assumes raw messages are stored in a "value" field internally

Integration with SeaweedMQ

This client works with SeaweedMQ's existing schema bypass logic:

1. isSystemTopic() function identifies "_" prefix topics as system topics
2. produceSchemaBasedRecord() bypasses schema processing for system topics
3. Raw storage via seaweedMQHandler.ProduceRecord() stores messages as-is

Use Cases

• Log ingestion: Store application logs without predefined schema
• Metrics collection: Publish time-series data in various formats
• Raw data pipelines: Process unstructured data before applying schemas
• Development/testing: Quickly publish test data without schema setup