Tree:
87829d52f5
add-admin-and-worker-to-helm-charts
add-ec-vacuum
add_fasthttp_client
add_remote_storage
adding-message-queue-integration-tests
also-delete-parent-directory-if-empty
avoid_releasing_temp_file_on_write
changing-to-zap
collect-public-metrics
copilot/fix-helm-chart-installation
create-table-snapshot-api-design
data_query_pushdown
dependabot/go_modules/github.com/arangodb/go-driver-1.6.9
dependabot/go_modules/github.com/klauspost/compress-1.18.2
dependabot/go_modules/github.com/prometheus/procfs-0.19.2
dependabot/go_modules/github.com/shirou/gopsutil/v4-4.25.11
dependabot/go_modules/github.com/ydb-platform/ydb-go-sdk/v3-3.121.0
dependabot/maven/other/java/client/com.google.protobuf-protobuf-java-3.25.5
dependabot/maven/other/java/examples/org.apache.hadoop-hadoop-common-3.4.0
detect-and-plan-ec-tasks
do-not-retry-if-error-is-NotFound
enhance-erasure-coding
fasthttp
filer1_maintenance_branch
fix-GetObjectLockConfigurationHandler
fix-versioning-listing-only
ftp
gh-pages
improve-fuse-mount
improve-fuse-mount2
logrus
master
message_send
mount2
mq-subscribe
mq2
original_weed_mount
pr-7412
random_access_file
refactor-needle-read-operations
refactor-volume-write
remote_overlay
revert-5134-patch-1
revert-5819-patch-1
revert-6434-bugfix-missing-s3-audit
s3-select
sub
tcp_read
test-reverting-lock-table
test_udp
testing
testing-sdx-generation
tikv
track-mount-e2e
upgrade-versions-to-4.00
volume_buffered_writes
worker-execute-ec-tasks
0.72
0.72.release
0.73
0.74
0.75
0.76
0.77
0.90
0.91
0.92
0.93
0.94
0.95
0.96
0.97
0.98
0.99
1.00
1.01
1.02
1.03
1.04
1.05
1.06
1.07
1.08
1.09
1.10
1.11
1.12
1.14
1.15
1.16
1.17
1.18
1.19
1.20
1.21
1.22
1.23
1.24
1.25
1.26
1.27
1.28
1.29
1.30
1.31
1.32
1.33
1.34
1.35
1.36
1.37
1.38
1.40
1.41
1.42
1.43
1.44
1.45
1.46
1.47
1.48
1.49
1.50
1.51
1.52
1.53
1.54
1.55
1.56
1.57
1.58
1.59
1.60
1.61
1.61RC
1.62
1.63
1.64
1.65
1.66
1.67
1.68
1.69
1.70
1.71
1.72
1.73
1.74
1.75
1.76
1.77
1.78
1.79
1.80
1.81
1.82
1.83
1.84
1.85
1.86
1.87
1.88
1.90
1.91
1.92
1.93
1.94
1.95
1.96
1.97
1.98
1.99
1;70
2.00
2.01
2.02
2.03
2.04
2.05
2.06
2.07
2.08
2.09
2.10
2.11
2.12
2.13
2.14
2.15
2.16
2.17
2.18
2.19
2.20
2.21
2.22
2.23
2.24
2.25
2.26
2.27
2.28
2.29
2.30
2.31
2.32
2.33
2.34
2.35
2.36
2.37
2.38
2.39
2.40
2.41
2.42
2.43
2.47
2.48
2.49
2.50
2.51
2.52
2.53
2.54
2.55
2.56
2.57
2.58
2.59
2.60
2.61
2.62
2.63
2.64
2.65
2.66
2.67
2.68
2.69
2.70
2.71
2.72
2.73
2.74
2.75
2.76
2.77
2.78
2.79
2.80
2.81
2.82
2.83
2.84
2.85
2.86
2.87
2.88
2.89
2.90
2.91
2.92
2.93
2.94
2.95
2.96
2.97
2.98
2.99
3.00
3.01
3.02
3.03
3.04
3.05
3.06
3.07
3.08
3.09
3.10
3.11
3.12
3.13
3.14
3.15
3.16
3.18
3.19
3.20
3.21
3.22
3.23
3.24
3.25
3.26
3.27
3.28
3.29
3.30
3.31
3.32
3.33
3.34
3.35
3.36
3.37
3.38
3.39
3.40
3.41
3.42
3.43
3.44
3.45
3.46
3.47
3.48
3.50
3.51
3.52
3.53
3.54
3.55
3.56
3.57
3.58
3.59
3.60
3.61
3.62
3.63
3.64
3.65
3.66
3.67
3.68
3.69
3.71
3.72
3.73
3.74
3.75
3.76
3.77
3.78
3.79
3.80
3.81
3.82
3.83
3.84
3.85
3.86
3.87
3.88
3.89
3.90
3.91
3.92
3.93
3.94
3.95
3.96
3.97
3.98
3.99
4.00
4.01
dev
helm-3.65.1
v0.69
v0.70beta
v3.33
${ noResults }
10 Commits (87829d52f5e929262465cf9a1ea5bc333b1040fa)
| Author | SHA1 | Message | Date |
|---|---|---|---|
chrislu
|
deb315a8a9 |
persist kafka offset
Phase E2: Integrate Protobuf descriptor parser with decoder - Update NewProtobufDecoder to use ProtobufDescriptorParser - Add findFirstMessageName helper for automatic message detection - Fix ParseBinaryDescriptor to return schema even on resolution failure - Add comprehensive tests for protobuf decoder integration - Improve error handling and caching behavior This enables proper binary descriptor parsing in the protobuf decoder, completing the integration between descriptor parsing and decoding. Phase E3: Complete Protobuf message descriptor resolution - Implement full protobuf descriptor resolution using protoreflect API - Add buildFileDescriptor and findMessageInFileDescriptor methods - Support nested message resolution with findNestedMessageDescriptor - Add proper mutex protection for thread-safe cache access - Update all test data to use proper field cardinality labels - Update test expectations to handle successful descriptor resolution - Enable full protobuf decoder creation from binary descriptors Phase E (Protobuf Support) is now complete: ✅ E1: Binary descriptor parsing ✅ E2: Decoder integration ✅ E3: Full message descriptor resolution Protobuf messages can now be fully parsed and decoded Phase F: Implement Kafka record batch compression support - Add comprehensive compression module supporting gzip/snappy/lz4/zstd - Implement RecordBatchParser with full compression and CRC validation - Support compression codec extraction from record batch attributes - Add compression/decompression for all major Kafka codecs - Integrate compression support into Produce and Fetch handlers - Add extensive unit tests for all compression codecs - Support round-trip compression/decompression with proper error handling - Add performance benchmarks for compression operations Key features: ✅ Gzip compression (ratio: 0.02) ✅ Snappy compression (ratio: 0.06, fastest) ✅ LZ4 compression (ratio: 0.02) ✅ Zstd compression (ratio: 0.01, best compression) ✅ CRC32 validation for record batch integrity ✅ Proper Kafka record batch format v2 parsing ✅ Backward compatibility with uncompressed records Phase F (Compression Handling) is now complete. Phase G: Implement advanced schema compatibility checking and migration - Add comprehensive SchemaEvolutionChecker with full compatibility rules - Support BACKWARD, FORWARD, FULL, and NONE compatibility levels - Implement Avro schema compatibility checking with field analysis - Add JSON Schema compatibility validation - Support Protobuf compatibility checking (simplified implementation) - Add type promotion rules (int->long, float->double, string<->bytes) - Integrate schema evolution into Manager with validation methods - Add schema evolution suggestions and migration guidance - Support schema compatibility validation before evolution - Add comprehensive unit tests for all compatibility scenarios Key features: ✅ BACKWARD compatibility: New schema can read old data ✅ FORWARD compatibility: Old schema can read new data ✅ FULL compatibility: Both backward and forward compatible ✅ Type promotion support for safe schema evolution ✅ Field addition/removal validation with default value checks ✅ Schema evolution suggestions for incompatible changes ✅ Integration with schema registry for validation workflows Phase G (Schema Evolution) is now complete. fmt |
3 months ago |
|
chrislu
|
71b2615f4a |
fmt
|
3 months ago |
|
chrislu
|
9cfbc0d4a1 |
Phase 7: Implement Fetch path schema reconstruction framework
- Add schema reconstruction functions to convert SMQ RecordValue back to Kafka format - Implement Confluent envelope reconstruction with proper schema metadata - Add Kafka record batch creation for schematized messages - Include topic-based schema detection and metadata retrieval - Add comprehensive round-trip testing for Avro schema reconstruction - Fix envelope parsing to avoid Protobuf interference with Avro messages - Prepare foundation for full SeaweedMQ integration in Phase 8 This enables the Kafka Gateway to reconstruct original message formats on Fetch. |
3 months ago |
|
chrislu
|
26eae1583f |
Phase 1: Enhanced Kafka Gateway Schema Integration
- Enhanced AgentClient with comprehensive Kafka record schema - Added kafka_key, kafka_value, kafka_timestamp, kafka_headers fields - Added kafka_offset and kafka_partition for full Kafka compatibility - Implemented createKafkaRecordSchema() for structured message storage - Enhanced SeaweedMQHandler with schema-aware topic management - Added CreateTopicWithSchema() method for proper schema registration - Integrated getDefaultKafkaSchema() for consistent schema across topics - Enhanced KafkaTopicInfo to store schema metadata - Enhanced Produce API with SeaweedMQ integration - Updated produceToSeaweedMQ() to use enhanced schema - Added comprehensive debug logging for SeaweedMQ operations - Maintained backward compatibility with in-memory mode - Added comprehensive integration tests - TestSeaweedMQIntegration for end-to-end SeaweedMQ backend testing - TestSchemaCompatibility for various message format validation - Tests verify enhanced schema works with different key-value types This implements the mq.agent architecture pattern for Kafka Gateway, providing structured message storage in SeaweedFS with full schema support. |
3 months ago |
|
chrislu
|
440fd4b65e |
feat: major Kafka Gateway milestone - near-complete E2E functionality
✅ COMPLETED: - Cross-client Produce compatibility (kafka-go + Sarama) - Fetch API version validation (v0-v11) - ListOffsets v2 parsing (replica_id, isolation_level) - Fetch v5 response structure (18→78 bytes, ~95% Sarama compatible) 🔧 CURRENT STATUS: - Produce: ✅ Working perfectly with both clients - Metadata: ✅ Working with multiple versions (v0-v7) - ListOffsets: ✅ Working with v2 format - Fetch: 🟡 Nearly compatible, minor format tweaks needed Next: Fine-tune Fetch v5 response for perfect Sarama compatibility |
3 months ago |
|
chrislu
|
f6da3b2920 |
fix: Fetch API version validation and ListOffsets v2 parsing
- Updated Fetch API to support v0-v11 (was v0-v1) - Fixed ListOffsets v2 request parsing (added replica_id and isolation_level fields) - Added proper debug logging for Fetch and ListOffsets handlers - Improved record batch construction with proper varint encoding - Cross-client Produce compatibility confirmed (kafka-go and Sarama) Next: Fix Fetch v5 response format for Sarama consumer compatibility |
3 months ago |
|
chrislu
|
2a7d1ccacf |
fmt
|
3 months ago |
|
chrislu
|
23f4f5e096 |
fix: correct Produce v7 request parsing for Sarama compatibility
✅ MAJOR FIX: Produce v7 Request Parsing - Fixed client_id, transactional_id, acks, timeout parsing - Now correctly parses Sarama requests: * client_id: sarama ✅ * transactional_id: null ✅ * acks: -1, timeout: 10000 ✅ * topics count: 1 ✅ * topic: sarama-e2e-topic ✅ 🔧 NEXT: Fix Produce v7 response format - Sarama getting 'invalid length' error on response - Response parsing issue, not request parsing |
3 months ago |
|
chrislu
|
5595dfd476 |
mq(kafka): Add comprehensive protocol compatibility review and TODOs
- Create PROTOCOL_COMPATIBILITY_REVIEW.md documenting all compatibility issues - Add critical TODOs to most problematic protocol implementations: * Produce: Record batch parsing is simplified, missing compression/CRC * Offset management: Hardcoded 'test-topic' parsing breaks real clients * JoinGroup: Consumer subscription extraction hardcoded, incomplete parsing * Fetch: Fake record batch construction with dummy data * Handler: Missing API version validation across all endpoints - Identify high/medium/low priority fixes needed for real client compatibility - Document specific areas needing work: * Record format parsing (v0/v1/v2, compression, CRC validation) * Request parsing (topics arrays, partition arrays, protocol metadata) * Consumer group protocol metadata parsing * Connection metadata extraction * Error code accuracy - Add testing recommendations for kafka-go, Sarama, Java clients - Provide roadmap for Phase 4 protocol compliance improvements This review is essential before attempting integration with real Kafka clients as current simplified implementations will fail with actual client libraries. |
3 months ago |
|
chrislu
|
59f1c3dda5 |
mq(kafka): implement Fetch handler with record batch construction, high watermark tracking, and comprehensive test coverage for consumer functionality
|
3 months ago |