chrislu
|
593c1ebef2
|
fix: Resolve High Priority TODOs - Real MQ Broker Integration
✅ COMPLETED HIGH PRIORITY TODOs:
🔧 **Real FilerClient Integration** (engine.go:131)
- Implemented GetFilerClient() method in BrokerClient
- Added filerClientImpl with full FilerClient interface compliance
- Added AdjustedUrl() and GetDataCenter() methods
- Real filerClient connection replaces nil fallback
🔧 **Partition Discovery via MQ Broker** (hybrid_message_scanner.go:116)
- Added ListTopicPartitions() method using topic configuration
- Implemented discoverTopicPartitions() in HybridMessageScanner
- Reads actual partition count from BrokerPartitionAssignments
- Generates proper partition ranges based on topic.PartitionCount
📋 **Technical Fixes:**
- Fixed compilation errors with undefined variables
- Proper error handling with filerClientErr variable
- Corrected ConfigureTopicResponse field usage (BrokerPartitionAssignments vs PartitionCount)
- Complete FilerClient interface implementation
🎯 **Impact:**
- SQL engine now connects to real MQ broker infrastructure
- Actual topic partition discovery instead of hardcoded defaults
- Production-ready broker integration with graceful fallbacks
- Maintains backward compatibility with sample data when broker unavailable
✅ All tests passing - High priority TODO resolution complete!
Next: Schema-aware message parsing and time filter optimization.
|
1 month ago |
|
chrislu
|
fe41380d51
|
feat: Phase 2 - Add DDL operations and real MQ broker integration
Implements comprehensive DDL support for MQ topic management:
New Components:
- Real MQ broker connectivity via BrokerClient
- CREATE TABLE → ConfigureTopic gRPC calls
- DROP TABLE → DeleteTopic operations
- DESCRIBE table → Schema introspection
- SQL type mapping (SQL ↔ MQ schema types)
Enhanced Features:
- Live topic discovery from MQ broker
- Fallback to cached/sample data when broker unavailable
- MySQL-compatible DESCRIBE output
- Schema validation and error handling
- CREATE TABLE with column definitions
Key Infrastructure:
- broker_client.go: gRPC communication with MQ broker
- sql_types.go: Bidirectional SQL/MQ type conversion
- describe.go: Table schema introspection
- Enhanced engine.go: Full DDL routing and execution
Supported SQL Operations:
✅ SHOW DATABASES, SHOW TABLES (live + fallback)
✅ CREATE TABLE table_name (col1 INT, col2 VARCHAR(50), ...)
✅ DROP TABLE table_name
✅ DESCRIBE table_name / SHOW COLUMNS FROM table_name
Known Limitations:
- SQL parser issues with reserved keywords (e.g., 'timestamp')
- Requires running MQ broker for full functionality
- ALTER TABLE not yet implemented
- DeleteTopic method needs broker-side implementation
Architecture Decisions:
- Broker discovery via filer lock mechanism (same as shell commands)
- Graceful fallback when broker unavailable
- ConfigureTopic for CREATE TABLE with 6 default partitions
- Schema versioning ready for ALTER TABLE support
Testing:
- Unit tests updated with filer address parameter
- Integration tests for DDL operations
- Error handling for connection failures
Next Phase: SELECT query execution with Parquet scanning
|
1 month ago |
|
chrislu
|
ad86637e0b
|
feat: Phase 1 - Add SQL query engine foundation for MQ topics
Implements core SQL infrastructure with metadata operations:
New Components:
- SQL parser integration using github.com/xwb1989/sqlparser
- Query engine framework in weed/query/engine/
- Schema catalog mapping MQ topics to SQL tables
- Interactive SQL CLI command 'weed sql'
Supported Operations:
- SHOW DATABASES (lists MQ namespaces)
- SHOW TABLES (lists MQ topics)
- SQL statement parsing and routing
- Error handling and result formatting
Key Design Decisions:
- MQ namespaces ↔ SQL databases
- MQ topics ↔ SQL tables
- Parquet message storage ready for querying
- Backward-compatible schema evolution support
Testing:
- Unit tests for core engine functionality
- Command integration tests
- Parse error handling validation
Assumptions (documented in code):
- All MQ messages stored in Parquet format
- Schema evolution maintains backward compatibility
- MySQL-compatible SQL syntax via sqlparser
- Single-threaded usage per SQL session
Next Phase: DDL operations (CREATE/ALTER/DROP TABLE)
|
1 month ago |
|
chrislu
|
26dbc6c905
|
move to https://github.com/seaweedfs/seaweedfs
|
3 years ago |
|
Chris Lu
|
939e4b57a8
|
go fmt
|
6 years ago |
|
Chris Lu
|
f8d4b7d1c0
|
support basic json filtering and selection
|
6 years ago |
|
Chris Lu
|
e26670c67a
|
add some basic sql types
copied some code from vitness
|
6 years ago |
|
Chris Lu
|
cf47f657af
|
scaffold for volume server query feature
|
6 years ago |
