chrislu
d218fe54fa
go fmt
7 months ago
chrislu
b1871427c3
persist consumer group offset
1. use one follower
2. read write consumer group offset
7 months ago
chrislu
8d5bb7420d
rename
7 months ago
chrislu
ccdd9cd8de
refactor
9 months ago
chrislu
f37c0d0d7a
comment out println
9 months ago
chrislu
f07875e8e1
send flush message to follower before shutting down logBuffer
9 months ago
chrislu
47975e75bf
purge old code
9 months ago
chrislu
237e68059d
shutdown follower if no clients, not only publishers
to avoid overhead when a publisher repeatedly send one message and close
9 months ago
chrislu
c9df613b6b
add publisher name for debugging
9 months ago
chrislu
ca4f89a6f6
simplify
9 months ago
chrislu
0d55bddc75
logs
9 months ago
chrislu
e3c8be7f89
LocalPartition shutdown
9 months ago
chrislu
ab9fc0ad70
simplify
9 months ago
chrislu
7463058299
change visibility
9 months ago
chrislu
dab5454332
publish and send to follower
9 months ago
chrislu
f249f831b1
MaybeShutdownLocalPartition refactor
9 months ago
chrislu
2a0ab7e90d
fix
9 months ago
chrislu
059a120708
refactor
9 months ago
chrislu
7d87c1d2bb
refactor
9 months ago
chrislu
5cc94a05b9
separate goroutine to send ack to publisher
9 months ago
chrislu
f67c8309d8
rename
9 months ago
chrislu
205d2285f3
setup follower by publisher
* the subscriber would getOrGen a local partition and wait
* the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.
9 months ago
chrislu
6ef3e6bcad
logging
9 months ago
chrislu
8e5068fd2f
notify
9 months ago
chrislu
529fa8c4c8
go fmt
9 months ago
chrislu
6f75a0af55
ask follower to follow
10 months ago
chrislu
3b28433cb1
change LogBuffer visibility in LocalPartition
10 months ago
chrislu
1b4484bf0a
go fmt
10 months ago
chrislu
3795d8dca8
release local topic partition if no publisher and subscribers
11 months ago
chrislu
ba73199174
reuse local partition
11 months ago
chrislu
fa59a5d67e
read from disk if not in memory
11 months ago
chrislu
6a7a679137
passing partition generation timestamp
12 months ago
chrislu
d51efddf5c
flush to disk
Need to do: read from disk
12 months ago
chrislu
49428a303b
add batch index for each memory buffer
12 months ago
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
1 year ago
chrislu
89a1fd1751
Squashed commit of the following:
commit 4827425146
Author: chrislu <chris.lu@gmail.com>
Date: Sat Sep 16 15:05:38 2023 -0700
balancer works
commit 3b50139f68
Author: chrislu <chris.lu@gmail.com>
Date: Fri Sep 15 22:22:32 2023 -0700
comments
commit 7f685ce7ba
Author: chrislu <chris.lu@gmail.com>
Date: Fri Sep 15 22:20:05 2023 -0700
adjust APIs
commit 436d99443b
Author: chrislu <chris.lu@gmail.com>
Date: Thu Sep 14 23:49:05 2023 -0700
receive broker stats
commit b771fefa37
Merge: 0a851ec00
890881037
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 13 00:03:47 2023 -0700
Merge branch 'master' into sub
commit 0a851ec00b
Author: chrislu <chris.lu@gmail.com>
Date: Sun Sep 10 22:01:25 2023 -0700
Create balancer.go
commit 39941edc0b
Author: chrislu <chris.lu@gmail.com>
Date: Thu Sep 7 23:55:19 2023 -0700
add publisher shutdown
commit 875f562779
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 6 23:16:41 2023 -0700
server side send response at least once per second
commit 984b6c54cf
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 6 23:15:29 2023 -0700
ack interval 128
commit 2492a45499
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 6 22:39:46 2023 -0700
ack interval
commit ba67e6ca29
Author: chrislu <chris.lu@gmail.com>
Date: Mon Sep 4 21:43:50 2023 -0700
api for sub
commit 9e4f985698
Author: chrislu <chris.lu@gmail.com>
Date: Mon Sep 4 21:43:30 2023 -0700
publish, benchmark
commit cb470d44df
Author: chrislu <chris.lu@gmail.com>
Date: Fri Sep 1 00:36:51 2023 -0700
can pub and sub
commit 1eb2da46d5
Author: chrislu <chris.lu@gmail.com>
Date: Mon Aug 28 09:02:12 2023 -0700
connect and publish
commit 504ae8383a
Author: chrislu <chris.lu@gmail.com>
Date: Mon Aug 28 09:01:25 2023 -0700
protoc version
commit dbcba75271
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 18:59:04 2023 -0700
rename to lookup
commit c9caf33119
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 18:33:46 2023 -0700
move functions
commit 4d6c18d86f
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 17:50:59 2023 -0700
pub sub initial tests
commit 4eb8e8624d
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 13:14:39 2023 -0700
rename
commit 1990456670
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 13:13:14 2023 -0700
sub
commit 905911853d
Author: chrislu <chris.lu@gmail.com>
Date: Sat Aug 26 13:39:21 2023 -0700
adjust proto
1 year ago
chrislu
4827425146
balancer works
1 year ago
chrislu
1eb2da46d5
connect and publish
1 year ago
chrislu
dbcba75271
rename to lookup
1 year ago
chrislu
1990456670
sub
1 year ago
chrislu
905911853d
adjust proto
1 year ago
chrislu
01d70c21f3
Squashed commit of the following:
commit 32f4b1a130
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 22:52:19 2023 -0700
fix compilation
commit e77ad33b7c
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 22:46:44 2023 -0700
pub
commit f431f30cc7
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 13:27:39 2023 -0700
fix generic type
commit 4e9dcb1829
Merge: 30f942580 16e3f2d52
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 12:47:14 2023 -0700
Merge branch 'master' into pubsub
commit 30f942580ad1bb32ae94aade2e3a21ec3ab63e21
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 11:10:58 2023 -0700
wip
commit f8b00980bc2f3879bb43decffd9a08d842f196f2
Author: chrislu <chris.lu@gmail.com>
Date: Tue Jul 25 09:14:35 2023 -0700
add design document
commit 08d2bebe42a26ebc39f1542f54d99e73620727dd
Author: chrislu <chris.lu@gmail.com>
Date: Tue Jul 25 09:14:06 2023 -0700
minor
commit bcfa7982b262a40fcdce6fc6613fad2ce07c13da
Author: chrislu <chris.lu@gmail.com>
Date: Tue Jul 25 09:13:49 2023 -0700
rename
1 year ago