chrislu
11201ac95f
adding agent
3 weeks ago
Chris Lu
dc784bf217
merge current message queue code changes ( #6201 )
* listing files to convert to parquet
* write parquet files
* save logs into parquet files
* pass by value
* compact logs into parquet format
* can skip existing files
* refactor
* refactor
* fix compilation
* when no partition found
* refactor
* add untested parquet file read
* rename package
* refactor
* rename files
* remove unused
* add merged log read func
* parquet wants to know the file size
* rewind by time
* pass in stop ts
* add stop ts
* adjust log
* minor
* adjust log
* skip .parquet files when reading message logs
* skip non message files
* Update subscriber_record.go
* send messages
* skip message data with only ts
* skip non log files
* update parquet-go package
* ensure a valid record type
* add new field to a record type
* Update read_parquet_to_log.go
* fix parquet file name generation
* separating reading parquet and logs
* add key field
* add skipped logs
* use in memory cache
* refactor
* refactor
* refactor
* refactor, and change compact log
* refactor
* rename
* refactor
* fix format
* prefix v to version directory
2 months ago
chrislu
270e91b0be
adds locking
4 months ago
chrislu
b6fd1ff4ce
refactor
4 months ago
chrislu
b1871427c3
persist consumer group offset
1. use one follower
2. read write consumer group offset
7 months ago
chrislu
e568e742c9
renaming
9 months ago
chrislu
ab9fc0ad70
simplify
9 months ago
chrislu
1b4484bf0a
go fmt
10 months ago
chrislu
fdf0ea8e11
passing partition info
11 months ago
chrislu
45994641e9
lint
12 months ago
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
1 year ago
chrislu
734178093e
refactor TopicPartition struct
1 year ago
chrislu
23f334dd13
comment
1 year ago
chrislu
c7e05e4e71
ensure latest stats are reported
1 year ago
chrislu
1492bf7552
fix listing topics
1 year ago
chrislu
6c6655de9d
wip
1 year ago
chrislu
a65088d739
broker report stats to balancer
1 year ago
chrislu
89a1fd1751
Squashed commit of the following:
commit 4827425146
Author: chrislu <chris.lu@gmail.com>
Date: Sat Sep 16 15:05:38 2023 -0700
balancer works
commit 3b50139f68
Author: chrislu <chris.lu@gmail.com>
Date: Fri Sep 15 22:22:32 2023 -0700
comments
commit 7f685ce7ba
Author: chrislu <chris.lu@gmail.com>
Date: Fri Sep 15 22:20:05 2023 -0700
adjust APIs
commit 436d99443b
Author: chrislu <chris.lu@gmail.com>
Date: Thu Sep 14 23:49:05 2023 -0700
receive broker stats
commit b771fefa37
Merge: 0a851ec00
890881037
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 13 00:03:47 2023 -0700
Merge branch 'master' into sub
commit 0a851ec00b
Author: chrislu <chris.lu@gmail.com>
Date: Sun Sep 10 22:01:25 2023 -0700
Create balancer.go
commit 39941edc0b
Author: chrislu <chris.lu@gmail.com>
Date: Thu Sep 7 23:55:19 2023 -0700
add publisher shutdown
commit 875f562779
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 6 23:16:41 2023 -0700
server side send response at least once per second
commit 984b6c54cf
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 6 23:15:29 2023 -0700
ack interval 128
commit 2492a45499
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 6 22:39:46 2023 -0700
ack interval
commit ba67e6ca29
Author: chrislu <chris.lu@gmail.com>
Date: Mon Sep 4 21:43:50 2023 -0700
api for sub
commit 9e4f985698
Author: chrislu <chris.lu@gmail.com>
Date: Mon Sep 4 21:43:30 2023 -0700
publish, benchmark
commit cb470d44df
Author: chrislu <chris.lu@gmail.com>
Date: Fri Sep 1 00:36:51 2023 -0700
can pub and sub
commit 1eb2da46d5
Author: chrislu <chris.lu@gmail.com>
Date: Mon Aug 28 09:02:12 2023 -0700
connect and publish
commit 504ae8383a
Author: chrislu <chris.lu@gmail.com>
Date: Mon Aug 28 09:01:25 2023 -0700
protoc version
commit dbcba75271
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 18:59:04 2023 -0700
rename to lookup
commit c9caf33119
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 18:33:46 2023 -0700
move functions
commit 4d6c18d86f
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 17:50:59 2023 -0700
pub sub initial tests
commit 4eb8e8624d
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 13:14:39 2023 -0700
rename
commit 1990456670
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 13:13:14 2023 -0700
sub
commit 905911853d
Author: chrislu <chris.lu@gmail.com>
Date: Sat Aug 26 13:39:21 2023 -0700
adjust proto
1 year ago
chrislu
4827425146
balancer works
1 year ago
chrislu
cb470d44df
can pub and sub
1 year ago
chrislu
01d70c21f3
Squashed commit of the following:
commit 32f4b1a130
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 22:52:19 2023 -0700
fix compilation
commit e77ad33b7c
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 22:46:44 2023 -0700
pub
commit f431f30cc7
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 13:27:39 2023 -0700
fix generic type
commit 4e9dcb1829
Merge: 30f942580 16e3f2d52
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 12:47:14 2023 -0700
Merge branch 'master' into pubsub
commit 30f942580ad1bb32ae94aade2e3a21ec3ab63e21
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 11:10:58 2023 -0700
wip
commit f8b00980bc2f3879bb43decffd9a08d842f196f2
Author: chrislu <chris.lu@gmail.com>
Date: Tue Jul 25 09:14:35 2023 -0700
add design document
commit 08d2bebe42a26ebc39f1542f54d99e73620727dd
Author: chrislu <chris.lu@gmail.com>
Date: Tue Jul 25 09:14:06 2023 -0700
minor
commit bcfa7982b262a40fcdce6fc6613fad2ce07c13da
Author: chrislu <chris.lu@gmail.com>
Date: Tue Jul 25 09:13:49 2023 -0700
rename
1 year ago