chrislu
270e91b0be
adds locking
4 months ago
chrislu
d218fe54fa
go fmt
7 months ago
chrislu
f37c0d0d7a
comment out println
9 months ago
chrislu
e568e742c9
renaming
9 months ago
chrislu
66a878af39
add control message to data message
9 months ago
chrislu
c9df613b6b
add publisher name for debugging
9 months ago
chrislu
0d55bddc75
logs
9 months ago
chrislu
dab5454332
publish and send to follower
9 months ago
chrislu
059a120708
refactor
9 months ago
chrislu
7d87c1d2bb
refactor
9 months ago
chrislu
5cc94a05b9
separate goroutine to send ack to publisher
9 months ago
chrislu
17806cde2a
separate context
If using the same context, the downstream grpc would be cancelled when the parent grpc is cancelled.
9 months ago
chrislu
f67c8309d8
rename
9 months ago
chrislu
d6a066495b
adjust logs
9 months ago
chrislu
1b1b96ea1e
close local partition if no more publishers and subscribers
9 months ago
chrislu
d558a58946
refactor
9 months ago
chrislu
205d2285f3
setup follower by publisher
* the subscriber would getOrGen a local partition and wait
* the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.
9 months ago
chrislu
7003ce7425
publisher is able to fully send data and receive acks
still needs to close the pipes cleanly
9 months ago
chrislu
529fa8c4c8
go fmt
9 months ago
chrislu
1b5c547cfe
logs
9 months ago
chrislu
55714f54ab
local partition is generated or not
9 months ago
chrislu
6f75a0af55
ask follower to follow
10 months ago
chrislu
0b2e5ddc7c
wait 3 seconds before shutting down publish client, to wait for all messages to be received
11 months ago
chrislu
a507069d19
lock for creating local partition
11 months ago
chrislu
2beaa2d0b3
pub/sub brokers check filer for assigned partitions
11 months ago
chrislu
c77d35313e
pub/sub broker only check local assigned partitions
11 months ago
chrislu
a5b906c0dd
refactor
11 months ago
chrislu
2828ccbb30
refactor
11 months ago
chrislu
c8090b1f99
send assignments to newly connected subscribers
11 months ago
chrislu
c616081ab9
refactor
11 months ago
chrislu
bc8d2a01cc
create local topic partition from config on filer
11 months ago
chrislu
3795d8dca8
release local topic partition if no publisher and subscribers
11 months ago
chrislu
ee41dbb7fc
rename functions
12 months ago
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
1 year ago
chrislu
89a1fd1751
Squashed commit of the following:
commit 4827425146
Author: chrislu <chris.lu@gmail.com>
Date: Sat Sep 16 15:05:38 2023 -0700
balancer works
commit 3b50139f68
Author: chrislu <chris.lu@gmail.com>
Date: Fri Sep 15 22:22:32 2023 -0700
comments
commit 7f685ce7ba
Author: chrislu <chris.lu@gmail.com>
Date: Fri Sep 15 22:20:05 2023 -0700
adjust APIs
commit 436d99443b
Author: chrislu <chris.lu@gmail.com>
Date: Thu Sep 14 23:49:05 2023 -0700
receive broker stats
commit b771fefa37
Merge: 0a851ec00
890881037
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 13 00:03:47 2023 -0700
Merge branch 'master' into sub
commit 0a851ec00b
Author: chrislu <chris.lu@gmail.com>
Date: Sun Sep 10 22:01:25 2023 -0700
Create balancer.go
commit 39941edc0b
Author: chrislu <chris.lu@gmail.com>
Date: Thu Sep 7 23:55:19 2023 -0700
add publisher shutdown
commit 875f562779
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 6 23:16:41 2023 -0700
server side send response at least once per second
commit 984b6c54cf
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 6 23:15:29 2023 -0700
ack interval 128
commit 2492a45499
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 6 22:39:46 2023 -0700
ack interval
commit ba67e6ca29
Author: chrislu <chris.lu@gmail.com>
Date: Mon Sep 4 21:43:50 2023 -0700
api for sub
commit 9e4f985698
Author: chrislu <chris.lu@gmail.com>
Date: Mon Sep 4 21:43:30 2023 -0700
publish, benchmark
commit cb470d44df
Author: chrislu <chris.lu@gmail.com>
Date: Fri Sep 1 00:36:51 2023 -0700
can pub and sub
commit 1eb2da46d5
Author: chrislu <chris.lu@gmail.com>
Date: Mon Aug 28 09:02:12 2023 -0700
connect and publish
commit 504ae8383a
Author: chrislu <chris.lu@gmail.com>
Date: Mon Aug 28 09:01:25 2023 -0700
protoc version
commit dbcba75271
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 18:59:04 2023 -0700
rename to lookup
commit c9caf33119
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 18:33:46 2023 -0700
move functions
commit 4d6c18d86f
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 17:50:59 2023 -0700
pub sub initial tests
commit 4eb8e8624d
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 13:14:39 2023 -0700
rename
commit 1990456670
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 13:13:14 2023 -0700
sub
commit 905911853d
Author: chrislu <chris.lu@gmail.com>
Date: Sat Aug 26 13:39:21 2023 -0700
adjust proto
1 year ago
chrislu
4eb8e8624d
rename
1 year ago
chrislu
905911853d
adjust proto
1 year ago
chrislu
01d70c21f3
Squashed commit of the following:
commit 32f4b1a130
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 22:52:19 2023 -0700
fix compilation
commit e77ad33b7c
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 22:46:44 2023 -0700
pub
commit f431f30cc7
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 13:27:39 2023 -0700
fix generic type
commit 4e9dcb1829
Merge: 30f942580 16e3f2d52
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 12:47:14 2023 -0700
Merge branch 'master' into pubsub
commit 30f942580ad1bb32ae94aade2e3a21ec3ab63e21
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 11:10:58 2023 -0700
wip
commit f8b00980bc2f3879bb43decffd9a08d842f196f2
Author: chrislu <chris.lu@gmail.com>
Date: Tue Jul 25 09:14:35 2023 -0700
add design document
commit 08d2bebe42a26ebc39f1542f54d99e73620727dd
Author: chrislu <chris.lu@gmail.com>
Date: Tue Jul 25 09:14:06 2023 -0700
minor
commit bcfa7982b262a40fcdce6fc6613fad2ce07c13da
Author: chrislu <chris.lu@gmail.com>
Date: Tue Jul 25 09:13:49 2023 -0700
rename
1 year ago
chrislu
afef014832
scaffold
2 years ago