Chris Lu
dc784bf217
merge current message queue code changes ( #6201 )
* listing files to convert to parquet
* write parquet files
* save logs into parquet files
* pass by value
* compact logs into parquet format
* can skip existing files
* refactor
* refactor
* fix compilation
* when no partition found
* refactor
* add untested parquet file read
* rename package
* refactor
* rename files
* remove unused
* add merged log read func
* parquet wants to know the file size
* rewind by time
* pass in stop ts
* add stop ts
* adjust log
* minor
* adjust log
* skip .parquet files when reading message logs
* skip non message files
* Update subscriber_record.go
* send messages
* skip message data with only ts
* skip non log files
* update parquet-go package
* ensure a valid record type
* add new field to a record type
* Update read_parquet_to_log.go
* fix parquet file name generation
* separating reading parquet and logs
* add key field
* add skipped logs
* use in memory cache
* refactor
* refactor
* refactor
* refactor, and change compact log
* refactor
* rename
* refactor
* fix format
* prefix v to version directory
3 months ago
chrislu
554ae09f82
rename
9 months ago
chrislu
d218fe54fa
go fmt
9 months ago
chrislu
6e5075e14e
move read write topic config into filer client accessor
9 months ago
chrislu
53d1d2b78a
save schema when configuring topic
10 months ago
Chris Lu
656b78d1af
proxy to broker leader
1 year ago
Chris Lu
4dc6681833
proxy requests to lock owner
1 year ago
chrislu
9e6ea80dff
read existing topic conf
1 year ago
chrislu
d268fbe18a
when configure, cancel existing assignments
1 year ago
chrislu
c3f8530f97
refactor
1 year ago
chrislu
5f35857cd0
no need to assign partitions explicitly
1 year ago
chrislu
e8b05ecc91
add/remove assigned partitions
1 year ago
chrislu
861ad732ca
refactor
1 year ago
chrislu
47924afa1c
refactor
1 year ago
chrislu
7121ae9617
separate out read topic conf and ensure topic active assignments
1 year ago
chrislu
a5b906c0dd
refactor
1 year ago
chrislu
bc8d2a01cc
create local topic partition from config on filer
1 year ago
chrislu
be0c426dc7
simplify to LookupTopicPartitions(topic)
1 year ago
chrislu
db3670a3a5
simplify api
1 year ago
chrislu
34a78ffad0
remove isForPublish from LookupTopicBrokers
also adds a return parameter: whether the topic exists or not
1 year ago
chrislu
ba73199174
reuse local partition
1 year ago
chrislu
fa59a5d67e
read from disk if not in memory
1 year ago
chrislu
6a7a679137
passing partition generation timestamp
1 year ago
chrislu
8af61dec91
avoid empty spaces in file name
1 year ago
chrislu
aed54eda61
refactor
1 year ago
chrislu
d51efddf5c
flush to disk
Need to do: read from disk
1 year ago
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
1 year ago