-
Notifications
You must be signed in to change notification settings - Fork 103
Open
Description
In current consistent hash allocate strategy, we will get unbalanced assignment caused by hash collision.
For example, we have a topic named TopicTest, the topic allocate at 3 brokers: broker0, broker1, broker2, 10 queues per broker, and we have 32 task managers. The allocation result using the ConsistentHashAllocateStrategy as follows:
{7=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=0], 8=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=1], 9=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=2], 10=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=3], 11=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=4], 12=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=5], 13=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=6], 14=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=7], 15=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=8], 16=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=9], 17=MessageQueue [topic=TopicTest, brokerName=broker1, queueId=9], 18=MessageQueue [topic=TopicTest, brokerName=broker0, queueId=9]}
allocation details, format is taskIndex=allocate num:
{7=1, 8=2, 9=3, 10=3, 11=3, 12=3, 13=3, 14=3, 15=3, 16=3, 17=2, 18=1}
We have 32 task manager, but only 12 task manager can work normally, which means we have 20 wasted task manager.
We should resolve the hash collision, I'm glad for your suggestion.
Metadata
Metadata
Assignees
Labels
No labels