Branch: feature/mq-kafka-gateway-m1

add-ec-vacuum

add-foundation-db

add_fasthttp_client

add_remote_storage

adding-message-queue-integration-tests

avoid_releasing_temp_file_on_write

changing-to-zap

collect-public-metrics

create-table-snapshot-api-design

data_query_pushdown

dependabot/maven/other/java/client/com.google.protobuf-protobuf-java-3.25.5

dependabot/maven/other/java/examples/org.apache.hadoop-hadoop-common-3.4.0

detect-and-plan-ec-tasks

do-not-retry-if-error-is-NotFound

fasthttp

feature/mq-kafka-gateway-m1

filer1_maintenance_branch

fix-GetObjectLockConfigurationHandler

fix-versioning-listing-only

ftp

gh-pages

improve-fuse-mount

improve-fuse-mount2

logrus

master

message_send

mount2

mq-subscribe

mq2

original_weed_mount

random_access_file

refactor-needle-read-operations

refactor-volume-write

remote_overlay

revert-5134-patch-1

revert-5819-patch-1

revert-6434-bugfix-missing-s3-audit

s3-select

sub

tcp_read

test-reverting-lock-table

test_udp

testing

testing-sdx-generation

tikv

track-mount-e2e

volume_buffered_writes

worker-execute-ec-tasks

0.72

0.72.release

0.73

0.74

0.75

0.76

0.77

0.90

0.91

0.92

0.93

0.94

0.95

0.96

0.97

0.98

0.99

1.00

1.01

1.02

1.03

1.04

1.05

1.06

1.07

1.08

1.09

1.10

1.11

1.12

1.14

1.15

1.16

1.17

1.18

1.19

1.20

1.21

1.22

1.23

1.24

1.25

1.26

1.27

1.28

1.29

1.30

1.31

1.32

1.33

1.34

1.35

1.36

1.37

1.38

1.40

1.41

1.42

1.43

1.44

1.45

1.46

1.47

1.48

1.49

1.50

1.51

1.52

1.53

1.54

1.55

1.56

1.57

1.58

1.59

1.60

1.61

1.61RC

1.62

1.63

1.64

1.65

1.66

1.67

1.68

1.69

1.70

1.71

1.72

1.73

1.74

1.75

1.76

1.77

1.78

1.79

1.80

1.81

1.82

1.83

1.84

1.85

1.86

1.87

1.88

1.90

1.91

1.92

1.93

1.94

1.95

1.96

1.97

1.98

1.99

1;70

2.00

2.01

2.02

2.03

2.04

2.05

2.06

2.07

2.08

2.09

2.10

2.11

2.12

2.13

2.14

2.15

2.16

2.17

2.18

2.19

2.20

2.21

2.22

2.23

2.24

2.25

2.26

2.27

2.28

2.29

2.30

2.31

2.32

2.33

2.34

2.35

2.36

2.37

2.38

2.39

2.40

2.41

2.42

2.43

2.47

2.48

2.49

2.50

2.51

2.52

2.53

2.54

2.55

2.56

2.57

2.58

2.59

2.60

2.61

2.62

2.63

2.64

2.65

2.66

2.67

2.68

2.69

2.70

2.71

2.72

2.73

2.74

2.75

2.76

2.77

2.78

2.79

2.80

2.81

2.82

2.83

2.84

2.85

2.86

2.87

2.88

2.89

2.90

2.91

2.92

2.93

2.94

2.95

2.96

2.97

2.98

2.99

3.00

3.01

3.02

3.03

3.04

3.05

3.06

3.07

3.08

3.09

3.10

3.11

3.12

3.13

3.14

3.15

3.16

3.18

3.19

3.20

3.21

3.22

3.23

3.24

3.25

3.26

3.27

3.28

3.29

3.30

3.31

3.32

3.33

3.34

3.35

3.36

3.37

3.38

3.39

3.40

3.41

3.42

3.43

3.44

3.45

3.46

3.47

3.48

3.50

3.51

3.52

3.53

3.54

3.55

3.56

3.57

3.58

3.59

3.60

3.61

3.62

3.63

3.64

3.65

3.66

3.67

3.68

3.69

3.71

3.72

3.73

3.74

3.75

3.76

3.77

3.78

3.79

3.80

3.81

3.82

3.83

3.84

3.85

3.86

3.87

3.88

3.89

3.90

3.91

3.92

3.93

3.94

3.95

3.96

3.97

dev

helm-3.65.1

v0.69

v0.70beta

v3.33

Branches Tags

${ item.name }

Create tag ${ searchTerm }

Create branch ${ searchTerm }

from 'feature/mq-kafka-gateway-m1'

${ noResults }

Commit Graph

8 Commits (feature/mq-kafka-gateway-m1)

Author	SHA1	Message	Date
chrislu	3f53262988	feat: implement per-topic schema validation in produce handler - Add comprehensive schema validation in produce.go with validateSchemaCompatibility - Implement performSchemaValidation with topic schema detection and format validation - Add validateMessageContent with format-specific validation (Avro, Protobuf, JSON Schema) - Add helper methods: parseSchemaID, isStrictSchemaValidation, getTopicCompatibilityLevel - Create comprehensive test suite in produce_schema_validation_test.go - Update fetch.go to use proper schema format detection and metadata building - Fix variable naming conflicts between schema package and schema variables - Add proper error handling and validation for schema management integration Tests: All schema validation tests pass, 2 expected failures due to missing schema registry	4 days ago
chrislu	90dd673888	kafka: fix Avro union re-encoding issue - Modify Avro decoder to preserve union type information by storing unions as records - Update union detection logic in encoding to properly reconstruct Avro union format - Fix test verification logic to handle new union storage format - Re-enable previously skipped Avro union tests in decode_encode_test.go and integration_test.go - All Avro union round-trip tests now pass This fixes the core issue where Avro unions like {'int': 42} were being converted to simple scalar values and losing the union type information needed for re-encoding. The fix stores unions as RecordValue with the union type as the field name, preserving the information needed for proper round-trip encoding.	4 days ago
chrislu	9ea6ef0bf8	fix tests	6 days ago
chrislu	deb315a8a9	persist kafka offset Phase E2: Integrate Protobuf descriptor parser with decoder - Update NewProtobufDecoder to use ProtobufDescriptorParser - Add findFirstMessageName helper for automatic message detection - Fix ParseBinaryDescriptor to return schema even on resolution failure - Add comprehensive tests for protobuf decoder integration - Improve error handling and caching behavior This enables proper binary descriptor parsing in the protobuf decoder, completing the integration between descriptor parsing and decoding. Phase E3: Complete Protobuf message descriptor resolution - Implement full protobuf descriptor resolution using protoreflect API - Add buildFileDescriptor and findMessageInFileDescriptor methods - Support nested message resolution with findNestedMessageDescriptor - Add proper mutex protection for thread-safe cache access - Update all test data to use proper field cardinality labels - Update test expectations to handle successful descriptor resolution - Enable full protobuf decoder creation from binary descriptors Phase E (Protobuf Support) is now complete: ✅ E1: Binary descriptor parsing ✅ E2: Decoder integration ✅ E3: Full message descriptor resolution Protobuf messages can now be fully parsed and decoded Phase F: Implement Kafka record batch compression support - Add comprehensive compression module supporting gzip/snappy/lz4/zstd - Implement RecordBatchParser with full compression and CRC validation - Support compression codec extraction from record batch attributes - Add compression/decompression for all major Kafka codecs - Integrate compression support into Produce and Fetch handlers - Add extensive unit tests for all compression codecs - Support round-trip compression/decompression with proper error handling - Add performance benchmarks for compression operations Key features: ✅ Gzip compression (ratio: 0.02) ✅ Snappy compression (ratio: 0.06, fastest) ✅ LZ4 compression (ratio: 0.02) ✅ Zstd compression (ratio: 0.01, best compression) ✅ CRC32 validation for record batch integrity ✅ Proper Kafka record batch format v2 parsing ✅ Backward compatibility with uncompressed records Phase F (Compression Handling) is now complete. Phase G: Implement advanced schema compatibility checking and migration - Add comprehensive SchemaEvolutionChecker with full compatibility rules - Support BACKWARD, FORWARD, FULL, and NONE compatibility levels - Implement Avro schema compatibility checking with field analysis - Add JSON Schema compatibility validation - Support Protobuf compatibility checking (simplified implementation) - Add type promotion rules (int->long, float->double, string<->bytes) - Integrate schema evolution into Manager with validation methods - Add schema evolution suggestions and migration guidance - Support schema compatibility validation before evolution - Add comprehensive unit tests for all compatibility scenarios Key features: ✅ BACKWARD compatibility: New schema can read old data ✅ FORWARD compatibility: Old schema can read new data ✅ FULL compatibility: Both backward and forward compatible ✅ Type promotion support for safe schema evolution ✅ Field addition/removal validation with default value checks ✅ Schema evolution suggestions for incompatible changes ✅ Integration with schema registry for validation workflows Phase G (Schema Evolution) is now complete. fmt	1 week ago
chrislu	4ed2604c71	Phase 6: Add JSON Schema decoder support for Kafka Gateway - Add gojsonschema dependency for JSON Schema validation and parsing - Implement JSONSchemaDecoder with validation and SMQ RecordValue conversion - Support all JSON Schema types: object, array, string, number, integer, boolean - Add format-specific type mapping (date-time, email, byte, etc.) - Include schema inference from JSON Schema to SeaweedMQ RecordType - Add round-trip encoding from RecordValue back to validated JSON - Integrate JSON Schema support into Schema Manager with caching - Comprehensive test coverage for validation, decoding, and type inference This completes schema format support for Avro, Protobuf, and JSON Schema.	1 week ago
chrislu	71b2615f4a	fmt	1 week ago
chrislu	394f49a25f	Phase 5: Add Protobuf decoder support for Kafka Gateway - Add ProtobufDecoder with dynamic message handling via protoreflect - Support Protobuf binary data decoding to Go maps and SMQ RecordValue - Implement Confluent Protobuf envelope parsing with varint indexes - Add Protobuf-to-RecordType inference with nested message support - Include Protobuf encoding for round-trip message reconstruction - Integrate Protobuf support into Schema Manager with caching - Add varint encoding/decoding utilities for Protobuf indexes - Prepare foundation for full FileDescriptorSet parsing in Phase 8 This enables the Kafka Gateway to process Protobuf-schematized messages.	1 week ago
chrislu	7b47ad613b	Phase 4: Integrate schema decoding into Kafka Produce path - Add Schema Manager to coordinate registry, decoders, and validation - Integrate schema management into Handler with enable/disable controls - Add schema processing functions in Produce path for schematized messages - Support both permissive and strict validation modes - Include message extraction and compatibility validation stubs - Add comprehensive Manager tests with mock registry server - Prepare foundation for SeaweedMQ integration in Phase 8 This enables the Kafka Gateway to detect, decode, and process schematized messages.	1 week ago