Skip to content

Conversation

@alexander-held
Copy link
Member

@alexander-held alexander-held commented Jan 20, 2026

The custom processing caused ReduceSchedulingError instances which seemed to be more frequent with more complex workloads. Switch to dask.distributed.as_completed instead, which does not run into these issues based on my testing.

@alexander-held
Copy link
Member Author

In its current state this may have helped partially but hit a new case I have previously not seen:

ReduceSchedulingError: dataset [...] has 3 items in merge-buffer (should only be 1); chunks left to merge: 0

The more common version prior to the changes here was:

ReduceSchedulingError: dataset [...] has 0 items in merge-buffer (should only be 1); chunks left to merge: 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants