You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

43 lines
1.2 KiB

Merge accumulated changes related to message queue (#5098) * balance partitions on brokers * prepare topic partition first and then publish, move partition * purge unused APIs * clean up * adjust logs * add BalanceTopics() grpc API * configure topic * configure topic command * refactor * repair missing partitions * sequence of operations to ensure ordering * proto to close publishers and consumers * rename file * topic partition versioned by unixTimeNs * create local topic partition * close publishers * randomize the client name * wait until no publishers * logs * close stop publisher channel * send last ack * comments * comment * comments * support list of brokers * add cli options * Update .gitignore * logs * return io.eof directly * refactor * optionally create topic * refactoring * detect consumer disconnection * sub client wait for more messages * subscribe by time stamp * rename * rename to sub_balancer * rename * adjust comments * rename * fix compilation * rename * rename * SubscriberToSubCoordinator * sticky rebalance * go fmt * add tests * balance partitions on brokers * prepare topic partition first and then publish, move partition * purge unused APIs * clean up * adjust logs * add BalanceTopics() grpc API * configure topic * configure topic command * refactor * repair missing partitions * sequence of operations to ensure ordering * proto to close publishers and consumers * rename file * topic partition versioned by unixTimeNs * create local topic partition * close publishers * randomize the client name * wait until no publishers * logs * close stop publisher channel * send last ack * comments * comment * comments * support list of brokers * add cli options * Update .gitignore * logs * return io.eof directly * refactor * optionally create topic * refactoring * detect consumer disconnection * sub client wait for more messages * subscribe by time stamp * rename * rename to sub_balancer * rename * adjust comments * rename * fix compilation * rename * rename * SubscriberToSubCoordinator * sticky rebalance * go fmt * add tests * tracking topic=>broker * merge * comment
1 year ago
  1. package pub_balancer
  2. /*
  3. Sequence of operations to ensure ordering
  4. Assuming Publisher P10 is publishing to Topic Partition TP10, and Subscriber S10 is subscribing to Topic TP10.
  5. After splitting Topic TP10 into Topic Partition TP11 and Topic Partition TP21,
  6. Publisher P11 is publishing to Topic Partition TP11, and Publisher P21 is publishing to Topic Partition TP21.
  7. Subscriber S12 is subscribing to Topic Partition TP11, and Subscriber S21 is subscribing to Topic Partition TP21.
  8. (The last digit is ephoch generation number, which is increasing when the topic partitioning is changed.)
  9. The diagram is as follows:
  10. P10 -> TP10 -> S10
  11. ||
  12. \/
  13. P11 -> TP11 -> S11
  14. P21 -> TP21 -> S21
  15. The following is the sequence of events:
  16. 1. Create Topic Partition TP11 and TP21
  17. 2. Close Publisher(s) P10
  18. 3. Close Subscriber(s) S10
  19. 4. Close Topic Partition TP10
  20. 5. Start Publisher P11, P21
  21. 6. Start Subscriber S11, S21
  22. The dependency is as follows:
  23. 2 => 3 => 4
  24. | |
  25. v v
  26. 1 => (5 | 6)
  27. And also:
  28. 2 => 5
  29. 3 => 6
  30. For brokers:
  31. 1. Close all publishers for a topic partition
  32. 2. Close all subscribers for a topic partition
  33. 3. Close the topic partition
  34. */