chrislu
ec155022e7
"golang.org/x/exp/slices" => "slices" and go fmt
3 days ago
Chris Lu
dc784bf217
merge current message queue code changes ( #6201 )
* listing files to convert to parquet
* write parquet files
* save logs into parquet files
* pass by value
* compact logs into parquet format
* can skip existing files
* refactor
* refactor
* fix compilation
* when no partition found
* refactor
* add untested parquet file read
* rename package
* refactor
* rename files
* remove unused
* add merged log read func
* parquet wants to know the file size
* rewind by time
* pass in stop ts
* add stop ts
* adjust log
* minor
* adjust log
* skip .parquet files when reading message logs
* skip non message files
* Update subscriber_record.go
* send messages
* skip message data with only ts
* skip non log files
* update parquet-go package
* ensure a valid record type
* add new field to a record type
* Update read_parquet_to_log.go
* fix parquet file name generation
* separating reading parquet and logs
* add key field
* add skipped logs
* use in memory cache
* refactor
* refactor
* refactor
* refactor, and change compact log
* refactor
* rename
* refactor
* fix format
* prefix v to version directory
2 months ago
chrislu
ff66155302
fix tests
4 months ago
chrislu
edc87c16f9
balance subscribers
need to ensure the offsets are persisted
7 months ago
chrislu
554ae09f82
rename
7 months ago
chrislu
b5099263a4
rename file
7 months ago
chrislu
d5abffa42c
rename Balancer to PubBalancer
7 months ago
chrislu
d218fe54fa
go fmt
7 months ago
chrislu
b1871427c3
persist consumer group offset
1. use one follower
2. read write consumer group offset
7 months ago
chrislu
b166597c83
fix test
8 months ago
chrislu
ab9fc0ad70
simplify
9 months ago
chrislu
ea2601799c
fix test
9 months ago
chrislu
529fa8c4c8
go fmt
9 months ago
chrislu
49869eec83
edge cases for broker assignment
10 months ago
chrislu
fe03b1b522
handle single broker case
10 months ago
chrislu
50c5dd7313
test low active brokers
10 months ago
chrislu
7598922b41
assign followers
10 months ago
chrislu
1b4484bf0a
go fmt
10 months ago
Chris Lu
0d74ac9224
fix divide by zero
11 months ago
chrislu
428fea45f3
updated and added assignments
11 months ago
chrislu
e8b05ecc91
add/remove assigned partitions
11 months ago
chrislu
b0a2e9aea3
fix assignments if brokers changed
11 months ago
chrislu
9ce7c482b3
refactor
11 months ago
chrislu
34839237ab
refactor
11 months ago
chrislu
2828ccbb30
refactor
11 months ago
chrislu
be0c426dc7
simplify to LookupTopicPartitions(topic)
11 months ago
chrislu
db3670a3a5
simplify api
11 months ago
chrislu
34a78ffad0
remove isForPublish from LookupTopicBrokers
also adds a return parameter: whether the topic exists or not
11 months ago
chrislu
f782165638
rename
11 months ago
chrislu
dd1ec70e62
rename
11 months ago
chrislu
6a7a679137
passing partition generation timestamp
12 months ago
chrislu
35869b5c80
subscriber can be notified of the assignment change when topic is just configured
Next: Subscriber needs to read by the timestamp offset.
12 months ago
chrislu
efb695fd93
lookup existing topic partitions
12 months ago
chrislu
093fdc1621
subscriber can get assignments
12 months ago
chrislu
e0727071c8
go fmt
1 year ago
chrislu
61b05fad95
fix test
1 year ago
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
1 year ago