Branch:
feature/mq-kafka-gateway-m1
add-ec-vacuum
add-foundation-db
add_fasthttp_client
add_remote_storage
adding-message-queue-integration-tests
avoid_releasing_temp_file_on_write
changing-to-zap
collect-public-metrics
create-table-snapshot-api-design
data_query_pushdown
dependabot/maven/other/java/client/com.google.protobuf-protobuf-java-3.25.5
dependabot/maven/other/java/examples/org.apache.hadoop-hadoop-common-3.4.0
detect-and-plan-ec-tasks
do-not-retry-if-error-is-NotFound
fasthttp
feature/mq-kafka-gateway-m1
filer1_maintenance_branch
fix-GetObjectLockConfigurationHandler
fix-versioning-listing-only
ftp
gh-pages
improve-fuse-mount
improve-fuse-mount2
logrus
master
message_send
mount2
mq-subscribe
mq2
original_weed_mount
random_access_file
refactor-needle-read-operations
refactor-volume-write
remote_overlay
revert-5134-patch-1
revert-5819-patch-1
revert-6434-bugfix-missing-s3-audit
s3-select
sub
tcp_read
test-reverting-lock-table
test_udp
testing
testing-sdx-generation
tikv
track-mount-e2e
volume_buffered_writes
worker-execute-ec-tasks
0.72
0.72.release
0.73
0.74
0.75
0.76
0.77
0.90
0.91
0.92
0.93
0.94
0.95
0.96
0.97
0.98
0.99
1.00
1.01
1.02
1.03
1.04
1.05
1.06
1.07
1.08
1.09
1.10
1.11
1.12
1.14
1.15
1.16
1.17
1.18
1.19
1.20
1.21
1.22
1.23
1.24
1.25
1.26
1.27
1.28
1.29
1.30
1.31
1.32
1.33
1.34
1.35
1.36
1.37
1.38
1.40
1.41
1.42
1.43
1.44
1.45
1.46
1.47
1.48
1.49
1.50
1.51
1.52
1.53
1.54
1.55
1.56
1.57
1.58
1.59
1.60
1.61
1.61RC
1.62
1.63
1.64
1.65
1.66
1.67
1.68
1.69
1.70
1.71
1.72
1.73
1.74
1.75
1.76
1.77
1.78
1.79
1.80
1.81
1.82
1.83
1.84
1.85
1.86
1.87
1.88
1.90
1.91
1.92
1.93
1.94
1.95
1.96
1.97
1.98
1.99
1;70
2.00
2.01
2.02
2.03
2.04
2.05
2.06
2.07
2.08
2.09
2.10
2.11
2.12
2.13
2.14
2.15
2.16
2.17
2.18
2.19
2.20
2.21
2.22
2.23
2.24
2.25
2.26
2.27
2.28
2.29
2.30
2.31
2.32
2.33
2.34
2.35
2.36
2.37
2.38
2.39
2.40
2.41
2.42
2.43
2.47
2.48
2.49
2.50
2.51
2.52
2.53
2.54
2.55
2.56
2.57
2.58
2.59
2.60
2.61
2.62
2.63
2.64
2.65
2.66
2.67
2.68
2.69
2.70
2.71
2.72
2.73
2.74
2.75
2.76
2.77
2.78
2.79
2.80
2.81
2.82
2.83
2.84
2.85
2.86
2.87
2.88
2.89
2.90
2.91
2.92
2.93
2.94
2.95
2.96
2.97
2.98
2.99
3.00
3.01
3.02
3.03
3.04
3.05
3.06
3.07
3.08
3.09
3.10
3.11
3.12
3.13
3.14
3.15
3.16
3.18
3.19
3.20
3.21
3.22
3.23
3.24
3.25
3.26
3.27
3.28
3.29
3.30
3.31
3.32
3.33
3.34
3.35
3.36
3.37
3.38
3.39
3.40
3.41
3.42
3.43
3.44
3.45
3.46
3.47
3.48
3.50
3.51
3.52
3.53
3.54
3.55
3.56
3.57
3.58
3.59
3.60
3.61
3.62
3.63
3.64
3.65
3.66
3.67
3.68
3.69
3.71
3.72
3.73
3.74
3.75
3.76
3.77
3.78
3.79
3.80
3.81
3.82
3.83
3.84
3.85
3.86
3.87
3.88
3.89
3.90
3.91
3.92
3.93
3.94
3.95
3.96
3.97
dev
helm-3.65.1
v0.69
v0.70beta
v3.33
${ item.name }
${ noResults }
4 Commits (feature/mq-kafka-gateway-m1)
| Author | SHA1 | Message | Date |
|---|---|---|---|
chrislu
|
ad63d06ac7 |
address some TODOs
|
3 days ago |
|
chrislu
|
ccd48feefb |
fix test errors
|
7 days ago |
|
chrislu
|
6aba7e6620 |
Phase 5: Implement SQL storage backend for offset persistence
- Design comprehensive SQL schema for offset storage with future _index column support - Implement SQLOffsetStorage with full database operations: - Partition offset checkpoints with UPSERT functionality - Detailed offset mappings with range queries and statistics - Database migration system with version tracking - Performance optimizations with proper indexing - Add database migration manager with automatic schema updates - Create comprehensive test suite with 11 test cases covering: - Schema initialization and table creation - Checkpoint save/load operations with error handling - Offset mapping storage and retrieval with sorting - Range queries and highest offset detection - Partition statistics with NULL value handling - Cleanup operations for old data retention - Concurrent access safety and database vacuum - Extend BrokerOffsetManager with SQL storage integration: - NewBrokerOffsetManagerWithSQL for database-backed storage - Configurable storage backends (in-memory fallback, SQL preferred) - Database connection management and error handling - Add SQLite driver dependency and configure for optimal performance - Support for future database types (PostgreSQL, MySQL) with abstraction layer Key TODOs and Assumptions: - TODO: Add _index as computed column when database supports it - TODO: Implement database backup and restore functionality - TODO: Add configuration for database path and connection parameters - ASSUMPTION: Using SQLite for now, extensible to other databases - ASSUMPTION: WAL mode and performance pragmas for production use - ASSUMPTION: Migration system handles schema evolution gracefully All 11 SQL storage tests pass, providing robust persistent offset management. |
1 week ago |
|
chrislu
|
161866b269 |
Phase 2: Implement offset assignment logic and recovery
- Add PartitionOffsetManager for sequential offset assignment per partition - Implement OffsetStorage interface with in-memory and SQL storage backends - Add PartitionOffsetRegistry for managing multiple partition offset managers - Implement offset recovery from checkpoints and storage scanning - Add OffsetAssigner for high-level offset assignment operations - Support both single and batch offset assignment with timestamps - Add comprehensive tests covering: - Basic and batch offset assignment - Offset recovery from checkpoints and storage - Multi-partition offset management - Concurrent offset assignment safety - All tests pass, offset assignment is thread-safe and recoverable |
1 week ago |