Browse Source
Phase 1: Add native offset support to SMQ protobuf definitions
Phase 1: Add native offset support to SMQ protobuf definitions
- Add EXACT_OFFSET and RESET_TO_OFFSET to OffsetType enum - Add start_offset field to PartitionOffset for offset-based positioning - Add base_offset and last_offset fields to PublishRecordResponse - Add offset field to SubscribeRecordResponse - Regenerate protobuf Go code - Add comprehensive tests for proto serialization and backward compatibility - All tests pass, ready for Phase 2 implementationpull/7231/head
chrislu
2 months ago
7 changed files with 622 additions and 86 deletions
-
287SMQ_OFFSET_DEVELOPMENT_PLAN.md
-
3weed/pb/mq_agent.proto
-
37weed/pb/mq_agent_pb/mq_agent.pb.go
-
102weed/pb/mq_agent_pb/publish_response_test.go
-
4weed/pb/mq_schema.proto
-
182weed/pb/schema_pb/mq_schema.pb.go
-
93weed/pb/schema_pb/offset_test.go
@ -0,0 +1,287 @@ |
|||||
|
# SMQ Native Offset Development Plan |
||||
|
|
||||
|
## Overview |
||||
|
|
||||
|
Add native per-partition sequential offsets to SeaweedMQ to eliminate the need for external offset mapping and provide better interoperability with message queue protocols. |
||||
|
|
||||
|
## Architecture Changes |
||||
|
|
||||
|
### Data Model |
||||
|
- Add `offset` field (int64) to each record alongside existing `ts_ns` |
||||
|
- Offset domain: per `schema_pb.Partition` (ring range) |
||||
|
- Offsets are strictly monotonic within a partition |
||||
|
- Leader assigns offsets; followers replicate |
||||
|
|
||||
|
### Storage |
||||
|
- Use `_index` as hidden SQL table column for offset storage |
||||
|
- Maintain per-partition offset counters in broker state |
||||
|
- Checkpoint offset state periodically for recovery |
||||
|
|
||||
|
## Development Phases |
||||
|
|
||||
|
### Phase 1: Proto and Data Model Changes |
||||
|
|
||||
|
**Scope**: Update protobuf definitions and core data structures |
||||
|
|
||||
|
**Tasks**: |
||||
|
1. Update `mq_schema.proto`: |
||||
|
- Add `offset` field to record storage format |
||||
|
- Add offset-based `OffsetType` enums |
||||
|
- Add `offset_value` field to subscription requests |
||||
|
2. Update `mq_agent.proto`: |
||||
|
- Add `base_offset` and `last_offset` to `PublishRecordResponse` |
||||
|
- Add `offset` field to `SubscribeRecordResponse` |
||||
|
3. Regenerate protobuf Go code |
||||
|
4. Update core data structures in broker code |
||||
|
5. Add offset field to SQL schema with `_index` column |
||||
|
|
||||
|
**Tests**: |
||||
|
- Proto compilation tests |
||||
|
- Data structure serialization tests |
||||
|
- SQL schema migration tests |
||||
|
|
||||
|
**Deliverables**: |
||||
|
- Updated proto files |
||||
|
- Generated Go code |
||||
|
- Updated SQL schema |
||||
|
- Basic unit tests |
||||
|
|
||||
|
### Phase 2: Offset Assignment Logic |
||||
|
|
||||
|
**Scope**: Implement offset assignment in broker |
||||
|
|
||||
|
**Tasks**: |
||||
|
1. Add `PartitionOffsetManager` component: |
||||
|
- Track `next_offset` per partition |
||||
|
- Assign sequential offsets to records |
||||
|
- Handle offset recovery on startup |
||||
|
2. Integrate with existing record publishing flow: |
||||
|
- Assign offsets before storage |
||||
|
- Update `PublishRecordResponse` with offset info |
||||
|
3. Add offset persistence to storage layer: |
||||
|
- Store offset alongside record data |
||||
|
- Index by offset for efficient lookups |
||||
|
4. Implement offset recovery: |
||||
|
- Load highest offset on partition leadership |
||||
|
- Handle clean and unclean restarts |
||||
|
|
||||
|
**Tests**: |
||||
|
- Offset assignment unit tests |
||||
|
- Offset persistence tests |
||||
|
- Recovery scenario tests |
||||
|
- Concurrent assignment tests |
||||
|
|
||||
|
**Deliverables**: |
||||
|
- `PartitionOffsetManager` implementation |
||||
|
- Integrated publishing with offsets |
||||
|
- Offset recovery logic |
||||
|
- Comprehensive test suite |
||||
|
|
||||
|
### Phase 3: Subscription by Offset |
||||
|
|
||||
|
**Scope**: Enable consumers to subscribe using offsets |
||||
|
|
||||
|
**Tasks**: |
||||
|
1. Extend subscription logic: |
||||
|
- Support `EXACT_OFFSET` and `RESET_TO_OFFSET` modes |
||||
|
- Add offset-based seeking |
||||
|
- Maintain backward compatibility with timestamp-based seeks |
||||
|
2. Update `SubscribeRecordResponse`: |
||||
|
- Include offset in response messages |
||||
|
- Ensure offset ordering in delivery |
||||
|
3. Add offset validation: |
||||
|
- Validate requested offsets are within valid range |
||||
|
- Handle out-of-range offset requests gracefully |
||||
|
4. Implement offset-based filtering and pagination |
||||
|
|
||||
|
**Tests**: |
||||
|
- Offset-based subscription tests |
||||
|
- Seek functionality tests |
||||
|
- Out-of-range offset handling tests |
||||
|
- Mixed timestamp/offset subscription tests |
||||
|
|
||||
|
**Deliverables**: |
||||
|
- Offset-based subscription implementation |
||||
|
- Updated subscription APIs |
||||
|
- Validation and error handling |
||||
|
- Integration tests |
||||
|
|
||||
|
### Phase 4: High Water Mark and Lag Calculation |
||||
|
|
||||
|
**Scope**: Implement native offset-based metrics |
||||
|
|
||||
|
**Tasks**: |
||||
|
1. Add high water mark tracking: |
||||
|
- Track highest committed offset per partition |
||||
|
- Expose via broker APIs |
||||
|
- Update on successful replication |
||||
|
2. Implement lag calculation: |
||||
|
- Consumer lag = high_water_mark - consumer_offset |
||||
|
- Partition lag metrics |
||||
|
- Consumer group lag aggregation |
||||
|
3. Add offset-based monitoring: |
||||
|
- Partition offset metrics |
||||
|
- Consumer position tracking |
||||
|
- Lag alerting capabilities |
||||
|
4. Update existing monitoring integration |
||||
|
|
||||
|
**Tests**: |
||||
|
- High water mark calculation tests |
||||
|
- Lag computation tests |
||||
|
- Monitoring integration tests |
||||
|
- Metrics accuracy tests |
||||
|
|
||||
|
**Deliverables**: |
||||
|
- High water mark implementation |
||||
|
- Lag calculation logic |
||||
|
- Monitoring integration |
||||
|
- Metrics and alerting |
||||
|
|
||||
|
### Phase 5: Kafka Gateway Integration |
||||
|
|
||||
|
**Scope**: Update Kafka gateway to use native SMQ offsets |
||||
|
|
||||
|
**Tasks**: |
||||
|
1. Remove offset mapping layer: |
||||
|
- Delete `kafka-system/offset-mappings` topic usage |
||||
|
- Remove `PersistentLedger` and `SeaweedMQStorage` |
||||
|
- Simplify offset translation logic |
||||
|
2. Update Kafka protocol handlers: |
||||
|
- Use native SMQ offsets in Produce responses |
||||
|
- Map SMQ offsets directly to Kafka offsets |
||||
|
- Update ListOffsets and Fetch handlers |
||||
|
3. Simplify consumer group offset management: |
||||
|
- Store Kafka consumer offsets as SMQ offsets |
||||
|
- Remove timestamp-based offset translation |
||||
|
4. Update integration tests: |
||||
|
- Test Kafka client compatibility |
||||
|
- Verify offset consistency |
||||
|
- Test long-term disconnection scenarios |
||||
|
|
||||
|
**Tests**: |
||||
|
- Kafka protocol compatibility tests |
||||
|
- End-to-end integration tests |
||||
|
- Performance comparison tests |
||||
|
- Migration scenario tests |
||||
|
|
||||
|
**Deliverables**: |
||||
|
- Simplified Kafka gateway |
||||
|
- Removed offset mapping complexity |
||||
|
- Updated integration tests |
||||
|
- Performance improvements |
||||
|
|
||||
|
### Phase 6: Performance Optimization and Production Readiness |
||||
|
|
||||
|
**Scope**: Optimize performance and prepare for production |
||||
|
|
||||
|
**Tasks**: |
||||
|
1. Optimize offset assignment performance: |
||||
|
- Batch offset assignment |
||||
|
- Reduce lock contention |
||||
|
- Optimize recovery performance |
||||
|
2. Add offset compaction and cleanup: |
||||
|
- Implement offset-based log compaction |
||||
|
- Add retention policies based on offsets |
||||
|
- Cleanup old offset checkpoints |
||||
|
3. Enhance monitoring and observability: |
||||
|
- Detailed offset metrics |
||||
|
- Performance dashboards |
||||
|
- Alerting on offset anomalies |
||||
|
4. Load testing and benchmarking: |
||||
|
- Compare performance with timestamp-only approach |
||||
|
- Test under high load scenarios |
||||
|
- Validate memory usage patterns |
||||
|
|
||||
|
**Tests**: |
||||
|
- Performance benchmarks |
||||
|
- Load testing scenarios |
||||
|
- Memory usage tests |
||||
|
- Stress testing under failures |
||||
|
|
||||
|
**Deliverables**: |
||||
|
- Optimized offset implementation |
||||
|
- Production monitoring |
||||
|
- Performance benchmarks |
||||
|
- Production deployment guide |
||||
|
|
||||
|
## Implementation Guidelines |
||||
|
|
||||
|
### Code Organization |
||||
|
``` |
||||
|
weed/mq/ |
||||
|
├── offset/ |
||||
|
│ ├── manager.go # PartitionOffsetManager |
||||
|
│ ├── recovery.go # Offset recovery logic |
||||
|
│ └── checkpoint.go # Offset checkpointing |
||||
|
├── broker/ |
||||
|
│ ├── partition_leader.go # Updated with offset assignment |
||||
|
│ └── subscriber.go # Updated with offset support |
||||
|
└── storage/ |
||||
|
└── offset_store.go # Offset persistence layer |
||||
|
``` |
||||
|
|
||||
|
### Testing Strategy |
||||
|
- Unit tests for each component |
||||
|
- Integration tests for cross-component interactions |
||||
|
- Performance tests for offset assignment and recovery |
||||
|
- Compatibility tests with existing SMQ features |
||||
|
- End-to-end tests with Kafka gateway |
||||
|
|
||||
|
### Commit Strategy |
||||
|
- One commit per completed task within a phase |
||||
|
- All tests must pass before commit |
||||
|
- No binary files in commits |
||||
|
- Clear commit messages describing changes |
||||
|
|
||||
|
### Rollout Plan |
||||
|
1. Deploy to development environment after Phase 2 |
||||
|
2. Integration testing after Phase 3 |
||||
|
3. Performance testing after Phase 4 |
||||
|
4. Kafka gateway migration after Phase 5 |
||||
|
5. Production rollout after Phase 6 |
||||
|
|
||||
|
## Success Criteria |
||||
|
|
||||
|
### Phase Completion Criteria |
||||
|
- All tests pass |
||||
|
- Code review completed |
||||
|
- Documentation updated |
||||
|
- Performance benchmarks meet targets |
||||
|
|
||||
|
### Overall Success Metrics |
||||
|
- Eliminate external offset mapping complexity |
||||
|
- Maintain or improve performance |
||||
|
- Full Kafka protocol compatibility |
||||
|
- Native SMQ offset support for all protocols |
||||
|
- Simplified consumer group offset management |
||||
|
|
||||
|
## Risk Mitigation |
||||
|
|
||||
|
### Technical Risks |
||||
|
- **Offset assignment bottlenecks**: Implement batching and optimize locking |
||||
|
- **Recovery performance**: Use checkpointing and incremental recovery |
||||
|
- **Storage overhead**: Optimize offset storage and indexing |
||||
|
|
||||
|
### Operational Risks |
||||
|
- **Migration complexity**: Implement gradual rollout with rollback capability |
||||
|
- **Data consistency**: Extensive testing of offset assignment and recovery |
||||
|
- **Performance regression**: Continuous benchmarking and monitoring |
||||
|
|
||||
|
## Timeline Estimate |
||||
|
|
||||
|
- Phase 1: 1-2 weeks |
||||
|
- Phase 2: 2-3 weeks |
||||
|
- Phase 3: 2-3 weeks |
||||
|
- Phase 4: 1-2 weeks |
||||
|
- Phase 5: 2-3 weeks |
||||
|
- Phase 6: 2-3 weeks |
||||
|
|
||||
|
**Total: 10-16 weeks** |
||||
|
|
||||
|
## Next Steps |
||||
|
|
||||
|
1. Review and approve development plan |
||||
|
2. Set up development branch |
||||
|
3. Begin Phase 1 implementation |
||||
|
4. Establish testing and CI pipeline |
||||
|
5. Regular progress reviews and adjustments |
||||
@ -0,0 +1,102 @@ |
|||||
|
package mq_agent_pb |
||||
|
|
||||
|
import ( |
||||
|
"testing" |
||||
|
"google.golang.org/protobuf/proto" |
||||
|
) |
||||
|
|
||||
|
func TestPublishRecordResponseSerialization(t *testing.T) { |
||||
|
// Test that PublishRecordResponse can serialize/deserialize with new offset fields
|
||||
|
original := &PublishRecordResponse{ |
||||
|
AckSequence: 123, |
||||
|
Error: "", |
||||
|
BaseOffset: 1000, // New field
|
||||
|
LastOffset: 1005, // New field
|
||||
|
} |
||||
|
|
||||
|
// Test proto marshaling/unmarshaling
|
||||
|
data, err := proto.Marshal(original) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to marshal PublishRecordResponse: %v", err) |
||||
|
} |
||||
|
|
||||
|
restored := &PublishRecordResponse{} |
||||
|
err = proto.Unmarshal(data, restored) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to unmarshal PublishRecordResponse: %v", err) |
||||
|
} |
||||
|
|
||||
|
// Verify all fields are preserved
|
||||
|
if restored.AckSequence != original.AckSequence { |
||||
|
t.Errorf("AckSequence = %d, want %d", restored.AckSequence, original.AckSequence) |
||||
|
} |
||||
|
if restored.BaseOffset != original.BaseOffset { |
||||
|
t.Errorf("BaseOffset = %d, want %d", restored.BaseOffset, original.BaseOffset) |
||||
|
} |
||||
|
if restored.LastOffset != original.LastOffset { |
||||
|
t.Errorf("LastOffset = %d, want %d", restored.LastOffset, original.LastOffset) |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
func TestSubscribeRecordResponseSerialization(t *testing.T) { |
||||
|
// Test that SubscribeRecordResponse can serialize/deserialize with new offset field
|
||||
|
original := &SubscribeRecordResponse{ |
||||
|
Key: []byte("test-key"), |
||||
|
TsNs: 1234567890, |
||||
|
Error: "", |
||||
|
IsEndOfStream: false, |
||||
|
IsEndOfTopic: false, |
||||
|
Offset: 42, // New field
|
||||
|
} |
||||
|
|
||||
|
// Test proto marshaling/unmarshaling
|
||||
|
data, err := proto.Marshal(original) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to marshal SubscribeRecordResponse: %v", err) |
||||
|
} |
||||
|
|
||||
|
restored := &SubscribeRecordResponse{} |
||||
|
err = proto.Unmarshal(data, restored) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to unmarshal SubscribeRecordResponse: %v", err) |
||||
|
} |
||||
|
|
||||
|
// Verify all fields are preserved
|
||||
|
if restored.TsNs != original.TsNs { |
||||
|
t.Errorf("TsNs = %d, want %d", restored.TsNs, original.TsNs) |
||||
|
} |
||||
|
if restored.Offset != original.Offset { |
||||
|
t.Errorf("Offset = %d, want %d", restored.Offset, original.Offset) |
||||
|
} |
||||
|
if string(restored.Key) != string(original.Key) { |
||||
|
t.Errorf("Key = %s, want %s", string(restored.Key), string(original.Key)) |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
func TestPublishRecordResponseBackwardCompatibility(t *testing.T) { |
||||
|
// Test that PublishRecordResponse without offset fields still works
|
||||
|
original := &PublishRecordResponse{ |
||||
|
AckSequence: 123, |
||||
|
Error: "", |
||||
|
// BaseOffset and LastOffset not set (defaults to 0)
|
||||
|
} |
||||
|
|
||||
|
data, err := proto.Marshal(original) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to marshal PublishRecordResponse: %v", err) |
||||
|
} |
||||
|
|
||||
|
restored := &PublishRecordResponse{} |
||||
|
err = proto.Unmarshal(data, restored) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to unmarshal PublishRecordResponse: %v", err) |
||||
|
} |
||||
|
|
||||
|
// Offset fields should default to 0
|
||||
|
if restored.BaseOffset != 0 { |
||||
|
t.Errorf("BaseOffset = %d, want 0", restored.BaseOffset) |
||||
|
} |
||||
|
if restored.LastOffset != 0 { |
||||
|
t.Errorf("LastOffset = %d, want 0", restored.LastOffset) |
||||
|
} |
||||
|
} |
||||
@ -0,0 +1,93 @@ |
|||||
|
package schema_pb |
||||
|
|
||||
|
import ( |
||||
|
"testing" |
||||
|
"google.golang.org/protobuf/proto" |
||||
|
) |
||||
|
|
||||
|
func TestOffsetTypeEnums(t *testing.T) { |
||||
|
// Test that new offset-based enum values are defined
|
||||
|
tests := []struct { |
||||
|
name string |
||||
|
value OffsetType |
||||
|
expected int32 |
||||
|
}{ |
||||
|
{"EXACT_OFFSET", OffsetType_EXACT_OFFSET, 25}, |
||||
|
{"RESET_TO_OFFSET", OffsetType_RESET_TO_OFFSET, 30}, |
||||
|
} |
||||
|
|
||||
|
for _, tt := range tests { |
||||
|
t.Run(tt.name, func(t *testing.T) { |
||||
|
if int32(tt.value) != tt.expected { |
||||
|
t.Errorf("OffsetType_%s = %d, want %d", tt.name, int32(tt.value), tt.expected) |
||||
|
} |
||||
|
}) |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
func TestPartitionOffsetSerialization(t *testing.T) { |
||||
|
// Test that PartitionOffset can serialize/deserialize with new offset field
|
||||
|
original := &PartitionOffset{ |
||||
|
Partition: &Partition{ |
||||
|
RingSize: 1024, |
||||
|
RangeStart: 0, |
||||
|
RangeStop: 31, |
||||
|
UnixTimeNs: 1234567890, |
||||
|
}, |
||||
|
StartTsNs: 1234567890, |
||||
|
StartOffset: 42, // New field
|
||||
|
} |
||||
|
|
||||
|
// Test proto marshaling/unmarshaling
|
||||
|
data, err := proto.Marshal(original) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to marshal PartitionOffset: %v", err) |
||||
|
} |
||||
|
|
||||
|
restored := &PartitionOffset{} |
||||
|
err = proto.Unmarshal(data, restored) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to unmarshal PartitionOffset: %v", err) |
||||
|
} |
||||
|
|
||||
|
// Verify all fields are preserved
|
||||
|
if restored.StartTsNs != original.StartTsNs { |
||||
|
t.Errorf("StartTsNs = %d, want %d", restored.StartTsNs, original.StartTsNs) |
||||
|
} |
||||
|
if restored.StartOffset != original.StartOffset { |
||||
|
t.Errorf("StartOffset = %d, want %d", restored.StartOffset, original.StartOffset) |
||||
|
} |
||||
|
if restored.Partition.RingSize != original.Partition.RingSize { |
||||
|
t.Errorf("Partition.RingSize = %d, want %d", restored.Partition.RingSize, original.Partition.RingSize) |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
func TestPartitionOffsetBackwardCompatibility(t *testing.T) { |
||||
|
// Test that PartitionOffset without StartOffset still works
|
||||
|
original := &PartitionOffset{ |
||||
|
Partition: &Partition{ |
||||
|
RingSize: 1024, |
||||
|
RangeStart: 0, |
||||
|
RangeStop: 31, |
||||
|
UnixTimeNs: 1234567890, |
||||
|
}, |
||||
|
StartTsNs: 1234567890, |
||||
|
// StartOffset not set (defaults to 0)
|
||||
|
} |
||||
|
|
||||
|
data, err := proto.Marshal(original) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to marshal PartitionOffset: %v", err) |
||||
|
} |
||||
|
|
||||
|
restored := &PartitionOffset{} |
||||
|
err = proto.Unmarshal(data, restored) |
||||
|
if err != nil { |
||||
|
t.Fatalf("Failed to unmarshal PartitionOffset: %v", err) |
||||
|
} |
||||
|
|
||||
|
// StartOffset should default to 0
|
||||
|
if restored.StartOffset != 0 { |
||||
|
t.Errorf("StartOffset = %d, want 0", restored.StartOffset) |
||||
|
} |
||||
|
} |
||||
Write
Preview
Loading…
Cancel
Save
Reference in new issue