Tree: 491404b3f6

add-ec-vacuum

add_fasthttp_client

add_remote_storage

adding-message-queue-integration-tests

adjust-fsck-cutoff-default

also-delete-parent-directory-if-empty

avoid_releasing_temp_file_on_write

changing-to-zap

collect-public-metrics

copilot/fix-helm-chart-installation

copilot/fix-s3-object-tagging-issue

copilot/make-renew-interval-configurable

copilot/make-renew-interval-configurable-again

copilot/sub-pr-7677

create-table-snapshot-api-design

data_query_pushdown

dependabot/maven/other/java/client/com.google.protobuf-protobuf-java-3.25.5

dependabot/maven/other/java/examples/org.apache.hadoop-hadoop-common-3.4.0

detect-and-plan-ec-tasks

do-not-retry-if-error-is-NotFound

ec-disk-type-support

enhance-erasure-coding

fasthttp

feature/mini-port-detection

feature/modernize-s3-tests

filer1_maintenance_branch

fix-GetObjectLockConfigurationHandler

fix-mount-http-parallelism

fix-mount-read-throughput-7504

fix-s3-object-tagging-issue-7589

fix-versioning-listing-only

ftp

gh-pages

improve-fuse-mount

improve-fuse-mount2

logrus

master

message_send

mount2

mq-subscribe

mq2

nfs-cookie-prefix-list-fixes

optimize-delete-lookups

original_weed_mount

pr-7412

raft-dual-write

random_access_file

refactor-needle-read-operations

refactor-volume-write

remote_overlay

remove-implicit-directory-handling

revert-5134-patch-1

revert-5819-patch-1

revert-6434-bugfix-missing-s3-audit

s3-remote-cache-singleflight

s3-select

sub

tcp_read

test-reverting-lock-table

test_udp

testing

testing-sdx-generation

tikv

track-mount-e2e

upgrade-versions-to-4.00

volume_buffered_writes

worker-execute-ec-tasks

0.72

0.72.release

0.73

0.74

0.75

0.76

0.77

0.90

0.91

0.92

0.93

0.94

0.95

0.96

0.97

0.98

0.99

1.00

1.01

1.02

1.03

1.04

1.05

1.06

1.07

1.08

1.09

1.10

1.11

1.12

1.14

1.15

1.16

1.17

1.18

1.19

1.20

1.21

1.22

1.23

1.24

1.25

1.26

1.27

1.28

1.29

1.30

1.31

1.32

1.33

1.34

1.35

1.36

1.37

1.38

1.40

1.41

1.42

1.43

1.44

1.45

1.46

1.47

1.48

1.49

1.50

1.51

1.52

1.53

1.54

1.55

1.56

1.57

1.58

1.59

1.60

1.61

1.61RC

1.62

1.63

1.64

1.65

1.66

1.67

1.68

1.69

1.70

1.71

1.72

1.73

1.74

1.75

1.76

1.77

1.78

1.79

1.80

1.81

1.82

1.83

1.84

1.85

1.86

1.87

1.88

1.90

1.91

1.92

1.93

1.94

1.95

1.96

1.97

1.98

1.99

1;70

2.00

2.01

2.02

2.03

2.04

2.05

2.06

2.07

2.08

2.09

2.10

2.11

2.12

2.13

2.14

2.15

2.16

2.17

2.18

2.19

2.20

2.21

2.22

2.23

2.24

2.25

2.26

2.27

2.28

2.29

2.30

2.31

2.32

2.33

2.34

2.35

2.36

2.37

2.38

2.39

2.40

2.41

2.42

2.43

2.47

2.48

2.49

2.50

2.51

2.52

2.53

2.54

2.55

2.56

2.57

2.58

2.59

2.60

2.61

2.62

2.63

2.64

2.65

2.66

2.67

2.68

2.69

2.70

2.71

2.72

2.73

2.74

2.75

2.76

2.77

2.78

2.79

2.80

2.81

2.82

2.83

2.84

2.85

2.86

2.87

2.88

2.89

2.90

2.91

2.92

2.93

2.94

2.95

2.96

2.97

2.98

2.99

3.00

3.01

3.02

3.03

3.04

3.05

3.06

3.07

3.08

3.09

3.10

3.11

3.12

3.13

3.14

3.15

3.16

3.18

3.19

3.20

3.21

3.22

3.23

3.24

3.25

3.26

3.27

3.28

3.29

3.30

3.31

3.32

3.33

3.34

3.35

3.36

3.37

3.38

3.39

3.40

3.41

3.42

3.43

3.44

3.45

3.46

3.47

3.48

3.50

3.51

3.52

3.53

3.54

3.55

3.56

3.57

3.58

3.59

3.60

3.61

3.62

3.63

3.64

3.65

3.66

3.67

3.68

3.69

3.71

3.72

3.73

3.74

3.75

3.76

3.77

3.78

3.79

3.80

3.81

3.82

3.83

3.84

3.85

3.86

3.87

3.88

3.89

3.90

3.91

3.92

3.93

3.94

3.95

3.96

3.97

3.98

3.99

4.00

4.01

4.02

4.03

4.04

dev

helm-3.65.1

v0.69

v0.70beta

v3.33

Branches Tags

${ item.name }

Create tag ${ searchTerm }

Create branch ${ searchTerm }

from '491404b3f6'

${ noResults }

Commit Graph

4 Commits (491404b3f6b4dcdcbd6a13e89216c18d76206ee1)

Author	SHA1	Message	Date
chrislu	6e1b96fb4a	Phase 6: Complete testing, validation, and documentation FINAL PHASE - SMQ Native Offset Implementation Complete ✅ - Create comprehensive end-to-end integration tests covering complete offset flow: - TestEndToEndOffsetFlow: Full publish/subscribe workflow with offset tracking - TestOffsetPersistenceAcrossRestarts: Validation of offset persistence behavior - TestConcurrentOffsetOperations: Multi-threaded offset assignment validation - TestOffsetValidationAndErrorHandling: Comprehensive error condition testing - All integration tests pass, validating complete system functionality - Add extensive performance benchmarks for all major operations: - BenchmarkOffsetAssignment: Sequential and parallel offset assignment - BenchmarkBatchOffsetAssignment: Batch operations with various sizes - BenchmarkSQLOffsetStorage: Complete SQL storage operation benchmarks - BenchmarkInMemoryVsSQL: Performance comparison between storage backends - BenchmarkOffsetSubscription: Subscription lifecycle and operations - BenchmarkSMQOffsetIntegration: Full integration layer performance - BenchmarkConcurrentOperations: Multi-threaded performance characteristics - Benchmarks demonstrate production-ready performance and scalability - Validate offset consistency and system reliability: - Database migration system with automatic schema updates - Proper NULL handling in SQL operations and migration management - Comprehensive error handling and validation throughout all components - Thread-safe operations with proper locking and concurrency control - Create comprehensive implementation documentation: - SMQ_NATIVE_OFFSET_IMPLEMENTATION.md: Complete implementation guide - Architecture overview with detailed component descriptions - Usage examples for all major operations and integration patterns - Performance characteristics and optimization recommendations - Deployment considerations and configuration options - Troubleshooting guide with common issues and debugging tools - Future enhancement roadmap and extension points - Update development plan with completion status: - All 6 phases successfully completed with comprehensive testing - 60+ tests covering all components and integration scenarios - Production-ready SQL storage backend with migration system - Complete broker integration with offset-aware operations - Extensive performance validation and optimization - Future-proof architecture supporting extensibility ## Implementation Summary This completes the full implementation of native per-partition sequential offsets in SeaweedMQ, providing: ✅ Sequential offset assignment per partition with thread-safe operations ✅ Persistent SQL storage backend with automatic migrations ✅ Complete broker integration with offset-aware publishing/subscription ✅ Comprehensive subscription management with seeking and lag tracking ✅ Robust error handling and validation throughout the system ✅ Extensive test coverage (60+ tests) and performance benchmarks ✅ Production-ready architecture with monitoring and troubleshooting support ✅ Complete documentation with usage examples and deployment guides The implementation eliminates the need for external offset mapping while maintaining high performance, reliability, and compatibility with existing SeaweedMQ operations. All tests pass and benchmarks demonstrate production-ready scalability.	4 months ago
chrislu	6aba7e6620	Phase 5: Implement SQL storage backend for offset persistence - Design comprehensive SQL schema for offset storage with future _index column support - Implement SQLOffsetStorage with full database operations: - Partition offset checkpoints with UPSERT functionality - Detailed offset mappings with range queries and statistics - Database migration system with version tracking - Performance optimizations with proper indexing - Add database migration manager with automatic schema updates - Create comprehensive test suite with 11 test cases covering: - Schema initialization and table creation - Checkpoint save/load operations with error handling - Offset mapping storage and retrieval with sorting - Range queries and highest offset detection - Partition statistics with NULL value handling - Cleanup operations for old data retention - Concurrent access safety and database vacuum - Extend BrokerOffsetManager with SQL storage integration: - NewBrokerOffsetManagerWithSQL for database-backed storage - Configurable storage backends (in-memory fallback, SQL preferred) - Database connection management and error handling - Add SQLite driver dependency and configure for optimal performance - Support for future database types (PostgreSQL, MySQL) with abstraction layer Key TODOs and Assumptions: - TODO: Add _index as computed column when database supports it - TODO: Implement database backup and restore functionality - TODO: Add configuration for database path and connection parameters - ASSUMPTION: Using SQLite for now, extensible to other databases - ASSUMPTION: WAL mode and performance pragmas for production use - ASSUMPTION: Migration system handles schema evolution gracefully All 11 SQL storage tests pass, providing robust persistent offset management.	4 months ago
chrislu	82fb366968	Phase 3: Implement offset-based subscription and SMQ integration - Add OffsetSubscriber for managing offset-based subscriptions - Implement OffsetSubscription with seeking, lag tracking, and range operations - Add OffsetSeeker for offset validation and range utilities - Create SMQOffsetIntegration for bridging offset management with SMQ broker - Support all OffsetType variants: EXACT_OFFSET, RESET_TO_OFFSET, RESET_TO_EARLIEST, RESET_TO_LATEST - Implement subscription lifecycle: create, seek, advance, close - Add comprehensive offset validation and error handling - Support batch record publishing and subscription - Add offset metrics and partition information APIs - Include extensive test coverage for all subscription scenarios: - Basic subscription creation and record consumption - Offset seeking and range operations - Subscription lag tracking and end-of-stream detection - Empty partition handling and error conditions - Integration with offset assignment and high water marks - All 40+ tests pass, providing robust offset-based messaging foundation	4 months ago
chrislu	161866b269	Phase 2: Implement offset assignment logic and recovery - Add PartitionOffsetManager for sequential offset assignment per partition - Implement OffsetStorage interface with in-memory and SQL storage backends - Add PartitionOffsetRegistry for managing multiple partition offset managers - Implement offset recovery from checkpoints and storage scanning - Add OffsetAssigner for high-level offset assignment operations - Support both single and batch offset assignment with timestamps - Add comprehensive tests covering: - Basic and batch offset assignment - Offset recovery from checkpoints and storage - Multi-partition offset management - Concurrent offset assignment safety - All tests pass, offset assignment is thread-safe and recoverable	4 months ago