Branch: feature/mq-kafka-gateway-m1

add-ec-vacuum

add-foundation-db

add_fasthttp_client

add_remote_storage

adding-message-queue-integration-tests

avoid_releasing_temp_file_on_write

changing-to-zap

collect-public-metrics

create-table-snapshot-api-design

data_query_pushdown

dependabot/maven/other/java/client/com.google.protobuf-protobuf-java-3.25.5

dependabot/maven/other/java/examples/org.apache.hadoop-hadoop-common-3.4.0

detect-and-plan-ec-tasks

do-not-retry-if-error-is-NotFound

fasthttp

feature/mq-kafka-gateway-m1

filer1_maintenance_branch

fix-GetObjectLockConfigurationHandler

fix-versioning-listing-only

ftp

gh-pages

improve-fuse-mount

improve-fuse-mount2

logrus

master

message_send

mount2

mq-subscribe

mq2

original_weed_mount

random_access_file

refactor-needle-read-operations

refactor-volume-write

remote_overlay

revert-5134-patch-1

revert-5819-patch-1

revert-6434-bugfix-missing-s3-audit

s3-select

sub

tcp_read

test-reverting-lock-table

test_udp

testing

testing-sdx-generation

tikv

track-mount-e2e

volume_buffered_writes

worker-execute-ec-tasks

0.72

0.72.release

0.73

0.74

0.75

0.76

0.77

0.90

0.91

0.92

0.93

0.94

0.95

0.96

0.97

0.98

0.99

1.00

1.01

1.02

1.03

1.04

1.05

1.06

1.07

1.08

1.09

1.10

1.11

1.12

1.14

1.15

1.16

1.17

1.18

1.19

1.20

1.21

1.22

1.23

1.24

1.25

1.26

1.27

1.28

1.29

1.30

1.31

1.32

1.33

1.34

1.35

1.36

1.37

1.38

1.40

1.41

1.42

1.43

1.44

1.45

1.46

1.47

1.48

1.49

1.50

1.51

1.52

1.53

1.54

1.55

1.56

1.57

1.58

1.59

1.60

1.61

1.61RC

1.62

1.63

1.64

1.65

1.66

1.67

1.68

1.69

1.70

1.71

1.72

1.73

1.74

1.75

1.76

1.77

1.78

1.79

1.80

1.81

1.82

1.83

1.84

1.85

1.86

1.87

1.88

1.90

1.91

1.92

1.93

1.94

1.95

1.96

1.97

1.98

1.99

1;70

2.00

2.01

2.02

2.03

2.04

2.05

2.06

2.07

2.08

2.09

2.10

2.11

2.12

2.13

2.14

2.15

2.16

2.17

2.18

2.19

2.20

2.21

2.22

2.23

2.24

2.25

2.26

2.27

2.28

2.29

2.30

2.31

2.32

2.33

2.34

2.35

2.36

2.37

2.38

2.39

2.40

2.41

2.42

2.43

2.47

2.48

2.49

2.50

2.51

2.52

2.53

2.54

2.55

2.56

2.57

2.58

2.59

2.60

2.61

2.62

2.63

2.64

2.65

2.66

2.67

2.68

2.69

2.70

2.71

2.72

2.73

2.74

2.75

2.76

2.77

2.78

2.79

2.80

2.81

2.82

2.83

2.84

2.85

2.86

2.87

2.88

2.89

2.90

2.91

2.92

2.93

2.94

2.95

2.96

2.97

2.98

2.99

3.00

3.01

3.02

3.03

3.04

3.05

3.06

3.07

3.08

3.09

3.10

3.11

3.12

3.13

3.14

3.15

3.16

3.18

3.19

3.20

3.21

3.22

3.23

3.24

3.25

3.26

3.27

3.28

3.29

3.30

3.31

3.32

3.33

3.34

3.35

3.36

3.37

3.38

3.39

3.40

3.41

3.42

3.43

3.44

3.45

3.46

3.47

3.48

3.50

3.51

3.52

3.53

3.54

3.55

3.56

3.57

3.58

3.59

3.60

3.61

3.62

3.63

3.64

3.65

3.66

3.67

3.68

3.69

3.71

3.72

3.73

3.74

3.75

3.76

3.77

3.78

3.79

3.80

3.81

3.82

3.83

3.84

3.85

3.86

3.87

3.88

3.89

3.90

3.91

3.92

3.93

3.94

3.95

3.96

3.97

dev

helm-3.65.1

v0.69

v0.70beta

v3.33

Branches Tags

${ item.name }

Create tag ${ searchTerm }

Create branch ${ searchTerm }

from 'feature/mq-kafka-gateway-m1'

${ noResults }

Commit Graph

3 Commits (feature/mq-kafka-gateway-m1)

Author	SHA1	Message	Date
chrislu	6e1b96fb4a	Phase 6: Complete testing, validation, and documentation FINAL PHASE - SMQ Native Offset Implementation Complete ✅ - Create comprehensive end-to-end integration tests covering complete offset flow: - TestEndToEndOffsetFlow: Full publish/subscribe workflow with offset tracking - TestOffsetPersistenceAcrossRestarts: Validation of offset persistence behavior - TestConcurrentOffsetOperations: Multi-threaded offset assignment validation - TestOffsetValidationAndErrorHandling: Comprehensive error condition testing - All integration tests pass, validating complete system functionality - Add extensive performance benchmarks for all major operations: - BenchmarkOffsetAssignment: Sequential and parallel offset assignment - BenchmarkBatchOffsetAssignment: Batch operations with various sizes - BenchmarkSQLOffsetStorage: Complete SQL storage operation benchmarks - BenchmarkInMemoryVsSQL: Performance comparison between storage backends - BenchmarkOffsetSubscription: Subscription lifecycle and operations - BenchmarkSMQOffsetIntegration: Full integration layer performance - BenchmarkConcurrentOperations: Multi-threaded performance characteristics - Benchmarks demonstrate production-ready performance and scalability - Validate offset consistency and system reliability: - Database migration system with automatic schema updates - Proper NULL handling in SQL operations and migration management - Comprehensive error handling and validation throughout all components - Thread-safe operations with proper locking and concurrency control - Create comprehensive implementation documentation: - SMQ_NATIVE_OFFSET_IMPLEMENTATION.md: Complete implementation guide - Architecture overview with detailed component descriptions - Usage examples for all major operations and integration patterns - Performance characteristics and optimization recommendations - Deployment considerations and configuration options - Troubleshooting guide with common issues and debugging tools - Future enhancement roadmap and extension points - Update development plan with completion status: - All 6 phases successfully completed with comprehensive testing - 60+ tests covering all components and integration scenarios - Production-ready SQL storage backend with migration system - Complete broker integration with offset-aware operations - Extensive performance validation and optimization - Future-proof architecture supporting extensibility ## Implementation Summary This completes the full implementation of native per-partition sequential offsets in SeaweedMQ, providing: ✅ Sequential offset assignment per partition with thread-safe operations ✅ Persistent SQL storage backend with automatic migrations ✅ Complete broker integration with offset-aware publishing/subscription ✅ Comprehensive subscription management with seeking and lag tracking ✅ Robust error handling and validation throughout the system ✅ Extensive test coverage (60+ tests) and performance benchmarks ✅ Production-ready architecture with monitoring and troubleshooting support ✅ Complete documentation with usage examples and deployment guides The implementation eliminates the need for external offset mapping while maintaining high performance, reliability, and compatibility with existing SeaweedMQ operations. All tests pass and benchmarks demonstrate production-ready scalability.	1 week ago
chrislu	1e2ad6c1c0	Update development plan with Phase 1-3 completion status - Mark Phase 1 (Protocol Schema Updates) as completed - Mark Phase 2 (Offset Assignment Logic) as completed - Mark Phase 3 (Subscription by Offset) as completed - Add detailed implementation summaries for each completed phase - Update next steps to focus on Phase 4 (Broker Integration) - Document comprehensive test coverage (40+ tests) and robust functionality	1 week ago
chrislu	450db29c17	Phase 1: Add native offset support to SMQ protobuf definitions - Add EXACT_OFFSET and RESET_TO_OFFSET to OffsetType enum - Add start_offset field to PartitionOffset for offset-based positioning - Add base_offset and last_offset fields to PublishRecordResponse - Add offset field to SubscribeRecordResponse - Regenerate protobuf Go code - Add comprehensive tests for proto serialization and backward compatibility - All tests pass, ready for Phase 2 implementation	1 week ago