Chris Lu
69553e5ba6
convert error fromating to %w everywhere ( #6995 )
2 weeks ago
Chris Lu
51543bbb87
Admin UI: Add message queue to admin UI ( #6958 )
* add a menu item "Message Queue"
* add a menu item "Message Queue"
* move the "brokers" link under it.
* add "topics", "subscribers". Add pages for them.
* refactor
* show topic details
* admin display publisher and subscriber info
* remove publisher and subscribers from the topic row pull down
* collecting more stats from publishers and subscribers
* fix layout
* fix publisher name
* add local listeners for mq broker and agent
* render consumer group offsets
* remove subscribers from left menu
* topic with retention
* support editing topic retention
* show retention when listing topics
* create bucket
* Update s3_buckets_templ.go
* embed the static assets into the binary
fix https://github.com/seaweedfs/seaweedfs/issues/6964
2 weeks ago
Chris Lu
cc05874d06
Add message queue agent ( #6463 )
* scaffold message queue agent
* adjust proto, add mq_agent
* add agent client implementation
* remove unused function
* agent publish server implementation
* adding agent
6 months ago
chrislu
554ae09f82
rename
1 year ago
chrislu
6e5075e14e
move read write topic config into filer client accessor
1 year ago
Chris Lu
656b78d1af
proxy to broker leader
1 year ago
Chris Lu
4dc6681833
proxy requests to lock owner
1 year ago
chrislu
d40225ecd2
return the lookup results
2 years ago
chrislu
841fafd0a8
publish to input buffer
currently, the input buffer may not exist when start to publish
2 years ago
chrislu
e8b05ecc91
add/remove assigned partitions
2 years ago
chrislu
581cf6644c
lookup reads from filer instead of local memory
2 years ago
chrislu
be0c426dc7
simplify to LookupTopicPartitions(topic)
2 years ago
chrislu
db3670a3a5
simplify api
2 years ago
chrislu
34a78ffad0
remove isForPublish from LookupTopicBrokers
also adds a return parameter: whether the topic exists or not
2 years ago
chrislu
efb695fd93
lookup existing topic partitions
2 years ago
chrislu
034db049a0
comment fix
2 years ago
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
2 years ago
chrislu
734178093e
refactor TopicPartition struct
2 years ago
chrislu
1492bf7552
fix listing topics
2 years ago
chrislu
3cf9b8d621
create topic and report topic
2 years ago
chrislu
b3f94feede
do create topic
2 years ago
chrislu
d74348048a
implement create topic
2 years ago
chrislu
0361c321b4
add CreateTopic API
2 years ago
chrislu
b18112ae10
fix compilation bugs during merge
2 years ago
chrislu
a65088d739
broker report stats to balancer
2 years ago
chrislu
89a1fd1751
Squashed commit of the following:
commit 4827425146
Author: chrislu <chris.lu@gmail.com>
Date: Sat Sep 16 15:05:38 2023 -0700
balancer works
commit 3b50139f68
Author: chrislu <chris.lu@gmail.com>
Date: Fri Sep 15 22:22:32 2023 -0700
comments
commit 7f685ce7ba
Author: chrislu <chris.lu@gmail.com>
Date: Fri Sep 15 22:20:05 2023 -0700
adjust APIs
commit 436d99443b
Author: chrislu <chris.lu@gmail.com>
Date: Thu Sep 14 23:49:05 2023 -0700
receive broker stats
commit b771fefa37
Merge: 0a851ec00
890881037
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 13 00:03:47 2023 -0700
Merge branch 'master' into sub
commit 0a851ec00b
Author: chrislu <chris.lu@gmail.com>
Date: Sun Sep 10 22:01:25 2023 -0700
Create balancer.go
commit 39941edc0b
Author: chrislu <chris.lu@gmail.com>
Date: Thu Sep 7 23:55:19 2023 -0700
add publisher shutdown
commit 875f562779
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 6 23:16:41 2023 -0700
server side send response at least once per second
commit 984b6c54cf
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 6 23:15:29 2023 -0700
ack interval 128
commit 2492a45499
Author: chrislu <chris.lu@gmail.com>
Date: Wed Sep 6 22:39:46 2023 -0700
ack interval
commit ba67e6ca29
Author: chrislu <chris.lu@gmail.com>
Date: Mon Sep 4 21:43:50 2023 -0700
api for sub
commit 9e4f985698
Author: chrislu <chris.lu@gmail.com>
Date: Mon Sep 4 21:43:30 2023 -0700
publish, benchmark
commit cb470d44df
Author: chrislu <chris.lu@gmail.com>
Date: Fri Sep 1 00:36:51 2023 -0700
can pub and sub
commit 1eb2da46d5
Author: chrislu <chris.lu@gmail.com>
Date: Mon Aug 28 09:02:12 2023 -0700
connect and publish
commit 504ae8383a
Author: chrislu <chris.lu@gmail.com>
Date: Mon Aug 28 09:01:25 2023 -0700
protoc version
commit dbcba75271
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 18:59:04 2023 -0700
rename to lookup
commit c9caf33119
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 18:33:46 2023 -0700
move functions
commit 4d6c18d86f
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 17:50:59 2023 -0700
pub sub initial tests
commit 4eb8e8624d
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 13:14:39 2023 -0700
rename
commit 1990456670
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 27 13:13:14 2023 -0700
sub
commit 905911853d
Author: chrislu <chris.lu@gmail.com>
Date: Sat Aug 26 13:39:21 2023 -0700
adjust proto
2 years ago
chrislu
1eb2da46d5
connect and publish
2 years ago
chrislu
dbcba75271
rename to lookup
2 years ago
chrislu
c9caf33119
move functions
2 years ago