Tree:
26eae1583f
add-admin-and-worker-to-helm-charts
add-ec-vacuum
add-foundation-db
add_fasthttp_client
add_remote_storage
adding-message-queue-integration-tests
also-delete-parent-directory-if-empty
avoid_releasing_temp_file_on_write
changing-to-zap
collect-public-metrics
create-table-snapshot-api-design
data_query_pushdown
dependabot/maven/other/java/client/com.google.protobuf-protobuf-java-3.25.5
dependabot/maven/other/java/examples/org.apache.hadoop-hadoop-common-3.4.0
detect-and-plan-ec-tasks
do-not-retry-if-error-is-NotFound
enhance-erasure-coding
fasthttp
filer1_maintenance_branch
fix-GetObjectLockConfigurationHandler
fix-versioning-listing-only
ftp
gh-pages
improve-fuse-mount
improve-fuse-mount2
logrus
master
message_send
mount2
mq-subscribe
mq2
original_weed_mount
pr-7412
random_access_file
refactor-needle-read-operations
refactor-volume-write
remote_overlay
revert-5134-patch-1
revert-5819-patch-1
revert-6434-bugfix-missing-s3-audit
s3-select
sub
tcp_read
test-reverting-lock-table
test_udp
testing
testing-sdx-generation
tikv
track-mount-e2e
volume_buffered_writes
worker-execute-ec-tasks
0.72
0.72.release
0.73
0.74
0.75
0.76
0.77
0.90
0.91
0.92
0.93
0.94
0.95
0.96
0.97
0.98
0.99
1.00
1.01
1.02
1.03
1.04
1.05
1.06
1.07
1.08
1.09
1.10
1.11
1.12
1.14
1.15
1.16
1.17
1.18
1.19
1.20
1.21
1.22
1.23
1.24
1.25
1.26
1.27
1.28
1.29
1.30
1.31
1.32
1.33
1.34
1.35
1.36
1.37
1.38
1.40
1.41
1.42
1.43
1.44
1.45
1.46
1.47
1.48
1.49
1.50
1.51
1.52
1.53
1.54
1.55
1.56
1.57
1.58
1.59
1.60
1.61
1.61RC
1.62
1.63
1.64
1.65
1.66
1.67
1.68
1.69
1.70
1.71
1.72
1.73
1.74
1.75
1.76
1.77
1.78
1.79
1.80
1.81
1.82
1.83
1.84
1.85
1.86
1.87
1.88
1.90
1.91
1.92
1.93
1.94
1.95
1.96
1.97
1.98
1.99
1;70
2.00
2.01
2.02
2.03
2.04
2.05
2.06
2.07
2.08
2.09
2.10
2.11
2.12
2.13
2.14
2.15
2.16
2.17
2.18
2.19
2.20
2.21
2.22
2.23
2.24
2.25
2.26
2.27
2.28
2.29
2.30
2.31
2.32
2.33
2.34
2.35
2.36
2.37
2.38
2.39
2.40
2.41
2.42
2.43
2.47
2.48
2.49
2.50
2.51
2.52
2.53
2.54
2.55
2.56
2.57
2.58
2.59
2.60
2.61
2.62
2.63
2.64
2.65
2.66
2.67
2.68
2.69
2.70
2.71
2.72
2.73
2.74
2.75
2.76
2.77
2.78
2.79
2.80
2.81
2.82
2.83
2.84
2.85
2.86
2.87
2.88
2.89
2.90
2.91
2.92
2.93
2.94
2.95
2.96
2.97
2.98
2.99
3.00
3.01
3.02
3.03
3.04
3.05
3.06
3.07
3.08
3.09
3.10
3.11
3.12
3.13
3.14
3.15
3.16
3.18
3.19
3.20
3.21
3.22
3.23
3.24
3.25
3.26
3.27
3.28
3.29
3.30
3.31
3.32
3.33
3.34
3.35
3.36
3.37
3.38
3.39
3.40
3.41
3.42
3.43
3.44
3.45
3.46
3.47
3.48
3.50
3.51
3.52
3.53
3.54
3.55
3.56
3.57
3.58
3.59
3.60
3.61
3.62
3.63
3.64
3.65
3.66
3.67
3.68
3.69
3.71
3.72
3.73
3.74
3.75
3.76
3.77
3.78
3.79
3.80
3.81
3.82
3.83
3.84
3.85
3.86
3.87
3.88
3.89
3.90
3.91
3.92
3.93
3.94
3.95
3.96
3.97
3.98
3.99
4.00
dev
helm-3.65.1
v0.69
v0.70beta
v3.33
${ noResults }
7 Commits (26eae1583ff32578327ad9dd567cc8dfc618d6fa)
| Author | SHA1 | Message | Date |
|---|---|---|---|
chrislu
|
26eae1583f |
Phase 1: Enhanced Kafka Gateway Schema Integration
- Enhanced AgentClient with comprehensive Kafka record schema - Added kafka_key, kafka_value, kafka_timestamp, kafka_headers fields - Added kafka_offset and kafka_partition for full Kafka compatibility - Implemented createKafkaRecordSchema() for structured message storage - Enhanced SeaweedMQHandler with schema-aware topic management - Added CreateTopicWithSchema() method for proper schema registration - Integrated getDefaultKafkaSchema() for consistent schema across topics - Enhanced KafkaTopicInfo to store schema metadata - Enhanced Produce API with SeaweedMQ integration - Updated produceToSeaweedMQ() to use enhanced schema - Added comprehensive debug logging for SeaweedMQ operations - Maintained backward compatibility with in-memory mode - Added comprehensive integration tests - TestSeaweedMQIntegration for end-to-end SeaweedMQ backend testing - TestSchemaCompatibility for various message format validation - Tests verify enhanced schema works with different key-value types This implements the mq.agent architecture pattern for Kafka Gateway, providing structured message storage in SeaweedFS with full schema support. |
2 months ago |
|
chrislu
|
440fd4b65e |
feat: major Kafka Gateway milestone - near-complete E2E functionality
✅ COMPLETED: - Cross-client Produce compatibility (kafka-go + Sarama) - Fetch API version validation (v0-v11) - ListOffsets v2 parsing (replica_id, isolation_level) - Fetch v5 response structure (18→78 bytes, ~95% Sarama compatible) 🔧 CURRENT STATUS: - Produce: ✅ Working perfectly with both clients - Metadata: ✅ Working with multiple versions (v0-v7) - ListOffsets: ✅ Working with v2 format - Fetch: 🟡 Nearly compatible, minor format tweaks needed Next: Fine-tune Fetch v5 response for perfect Sarama compatibility |
2 months ago |
|
chrislu
|
f6da3b2920 |
fix: Fetch API version validation and ListOffsets v2 parsing
- Updated Fetch API to support v0-v11 (was v0-v1) - Fixed ListOffsets v2 request parsing (added replica_id and isolation_level fields) - Added proper debug logging for Fetch and ListOffsets handlers - Improved record batch construction with proper varint encoding - Cross-client Produce compatibility confirmed (kafka-go and Sarama) Next: Fix Fetch v5 response format for Sarama consumer compatibility |
2 months ago |
|
chrislu
|
2a7d1ccacf |
fmt
|
2 months ago |
|
chrislu
|
23f4f5e096 |
fix: correct Produce v7 request parsing for Sarama compatibility
✅ MAJOR FIX: Produce v7 Request Parsing - Fixed client_id, transactional_id, acks, timeout parsing - Now correctly parses Sarama requests: * client_id: sarama ✅ * transactional_id: null ✅ * acks: -1, timeout: 10000 ✅ * topics count: 1 ✅ * topic: sarama-e2e-topic ✅ 🔧 NEXT: Fix Produce v7 response format - Sarama getting 'invalid length' error on response - Response parsing issue, not request parsing |
2 months ago |
|
chrislu
|
5595dfd476 |
mq(kafka): Add comprehensive protocol compatibility review and TODOs
- Create PROTOCOL_COMPATIBILITY_REVIEW.md documenting all compatibility issues - Add critical TODOs to most problematic protocol implementations: * Produce: Record batch parsing is simplified, missing compression/CRC * Offset management: Hardcoded 'test-topic' parsing breaks real clients * JoinGroup: Consumer subscription extraction hardcoded, incomplete parsing * Fetch: Fake record batch construction with dummy data * Handler: Missing API version validation across all endpoints - Identify high/medium/low priority fixes needed for real client compatibility - Document specific areas needing work: * Record format parsing (v0/v1/v2, compression, CRC validation) * Request parsing (topics arrays, partition arrays, protocol metadata) * Consumer group protocol metadata parsing * Connection metadata extraction * Error code accuracy - Add testing recommendations for kafka-go, Sarama, Java clients - Provide roadmap for Phase 4 protocol compliance improvements This review is essential before attempting integration with real Kafka clients as current simplified implementations will fail with actual client libraries. |
2 months ago |
|
chrislu
|
59f1c3dda5 |
mq(kafka): implement Fetch handler with record batch construction, high watermark tracking, and comprehensive test coverage for consumer functionality
|
2 months ago |