Chris Lu
dc784bf217
merge current message queue code changes ( #6201 )
* listing files to convert to parquet
* write parquet files
* save logs into parquet files
* pass by value
* compact logs into parquet format
* can skip existing files
* refactor
* refactor
* fix compilation
* when no partition found
* refactor
* add untested parquet file read
* rename package
* refactor
* rename files
* remove unused
* add merged log read func
* parquet wants to know the file size
* rewind by time
* pass in stop ts
* add stop ts
* adjust log
* minor
* adjust log
* skip .parquet files when reading message logs
* skip non message files
* Update subscriber_record.go
* send messages
* skip message data with only ts
* skip non log files
* update parquet-go package
* ensure a valid record type
* add new field to a record type
* Update read_parquet_to_log.go
* fix parquet file name generation
* separating reading parquet and logs
* add key field
* add skipped logs
* use in memory cache
* refactor
* refactor
* refactor
* refactor, and change compact log
* refactor
* rename
* refactor
* fix format
* prefix v to version directory
2 months ago
chrislu
554ae09f82
rename
7 months ago
chrislu
d218fe54fa
go fmt
7 months ago
chrislu
6e5075e14e
move read write topic config into filer client accessor
7 months ago
chrislu
53d1d2b78a
save schema when configuring topic
8 months ago
Chris Lu
656b78d1af
proxy to broker leader
11 months ago
Chris Lu
4dc6681833
proxy requests to lock owner
11 months ago
chrislu
9e6ea80dff
read existing topic conf
11 months ago
chrislu
d268fbe18a
when configure, cancel existing assignments
11 months ago
chrislu
c3f8530f97
refactor
11 months ago
chrislu
5f35857cd0
no need to assign partitions explicitly
11 months ago
chrislu
e8b05ecc91
add/remove assigned partitions
11 months ago
chrislu
861ad732ca
refactor
11 months ago
chrislu
47924afa1c
refactor
11 months ago
chrislu
7121ae9617
separate out read topic conf and ensure topic active assignments
11 months ago
chrislu
a5b906c0dd
refactor
11 months ago
chrislu
bc8d2a01cc
create local topic partition from config on filer
11 months ago
chrislu
be0c426dc7
simplify to LookupTopicPartitions(topic)
11 months ago
chrislu
db3670a3a5
simplify api
11 months ago
chrislu
34a78ffad0
remove isForPublish from LookupTopicBrokers
also adds a return parameter: whether the topic exists or not
11 months ago
chrislu
ba73199174
reuse local partition
11 months ago
chrislu
fa59a5d67e
read from disk if not in memory
11 months ago
chrislu
6a7a679137
passing partition generation timestamp
12 months ago
chrislu
8af61dec91
avoid empty spaces in file name
12 months ago
chrislu
aed54eda61
refactor
12 months ago
chrislu
d51efddf5c
flush to disk
Need to do: read from disk
12 months ago
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
1 year ago