Skip to content

[Bug] unbalanced assignment caused by hash collision in ConsistentHashAllocateStrategy.java #113

@humkum

Description

@humkum

In current consistent hash allocate strategy, we will get unbalanced assignment caused by hash collision.
For example, we have a topic named TopicTest, the topic allocate at 3 brokers: broker0, broker1, broker2, 10 queues per broker, and we have 32 task managers. The allocation result using the ConsistentHashAllocateStrategy as follows:

{7=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=0], 8=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=1], 9=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=2], 10=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=3], 11=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=4], 12=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=5], 13=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=6], 14=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=7], 15=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=8], 16=MessageQueue [topic=TopicTest, brokerName=broker2, queueId=9], 17=MessageQueue [topic=TopicTest, brokerName=broker1, queueId=9], 18=MessageQueue [topic=TopicTest, brokerName=broker0, queueId=9]}
allocation details, format is taskIndex=allocate num:
{7=1, 8=2, 9=3, 10=3, 11=3, 12=3, 13=3, 14=3, 15=3, 16=3, 17=2, 18=1}

We have 32 task manager, but only 12 task manager can work normally, which means we have 20 wasted task manager.

We should resolve the hash collision, I'm glad for your suggestion.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions