Chris Lu
cc05874d06
Add message queue agent ( #6463 )
* scaffold message queue agent
* adjust proto, add mq_agent
* add agent client implementation
* remove unused function
* agent publish server implementation
* adding agent
4 months ago
chrislu
ec155022e7
"golang.org/x/exp/slices" => "slices" and go fmt
5 months ago
Chris Lu
dc784bf217
merge current message queue code changes ( #6201 )
* listing files to convert to parquet
* write parquet files
* save logs into parquet files
* pass by value
* compact logs into parquet format
* can skip existing files
* refactor
* refactor
* fix compilation
* when no partition found
* refactor
* add untested parquet file read
* rename package
* refactor
* rename files
* remove unused
* add merged log read func
* parquet wants to know the file size
* rewind by time
* pass in stop ts
* add stop ts
* adjust log
* minor
* adjust log
* skip .parquet files when reading message logs
* skip non message files
* Update subscriber_record.go
* send messages
* skip message data with only ts
* skip non log files
* update parquet-go package
* ensure a valid record type
* add new field to a record type
* Update read_parquet_to_log.go
* fix parquet file name generation
* separating reading parquet and logs
* add key field
* add skipped logs
* use in memory cache
* refactor
* refactor
* refactor
* refactor, and change compact log
* refactor
* rename
* refactor
* fix format
* prefix v to version directory
6 months ago
chrislu
ff66155302
fix tests
9 months ago
chrislu
edc87c16f9
balance subscribers
need to ensure the offsets are persisted
12 months ago
chrislu
554ae09f82
rename
1 year ago
chrislu
b5099263a4
rename file
1 year ago
chrislu
d5abffa42c
rename Balancer to PubBalancer
1 year ago
chrislu
d218fe54fa
go fmt
1 year ago
chrislu
b1871427c3
persist consumer group offset
1. use one follower
2. read write consumer group offset
1 year ago
chrislu
b166597c83
fix test
1 year ago
chrislu
ab9fc0ad70
simplify
1 year ago
chrislu
ea2601799c
fix test
1 year ago
chrislu
529fa8c4c8
go fmt
1 year ago
chrislu
49869eec83
edge cases for broker assignment
1 year ago
chrislu
fe03b1b522
handle single broker case
1 year ago
chrislu
50c5dd7313
test low active brokers
1 year ago
chrislu
7598922b41
assign followers
1 year ago
chrislu
1b4484bf0a
go fmt
1 year ago
Chris Lu
0d74ac9224
fix divide by zero
1 year ago
chrislu
428fea45f3
updated and added assignments
1 year ago
chrislu
e8b05ecc91
add/remove assigned partitions
1 year ago
chrislu
b0a2e9aea3
fix assignments if brokers changed
1 year ago
chrislu
9ce7c482b3
refactor
1 year ago
chrislu
34839237ab
refactor
1 year ago
chrislu
2828ccbb30
refactor
1 year ago
chrislu
be0c426dc7
simplify to LookupTopicPartitions(topic)
1 year ago
chrislu
db3670a3a5
simplify api
1 year ago
chrislu
34a78ffad0
remove isForPublish from LookupTopicBrokers
also adds a return parameter: whether the topic exists or not
1 year ago
chrislu
f782165638
rename
1 year ago
chrislu
dd1ec70e62
rename
1 year ago
chrislu
6a7a679137
passing partition generation timestamp
1 year ago
chrislu
35869b5c80
subscriber can be notified of the assignment change when topic is just configured
Next: Subscriber needs to read by the timestamp offset.
1 year ago
chrislu
efb695fd93
lookup existing topic partitions
1 year ago
chrislu
093fdc1621
subscriber can get assignments
1 year ago
chrislu
e0727071c8
go fmt
1 year ago
chrislu
61b05fad95
fix test
1 year ago
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
1 year ago