Checklist
Related Issues and Possible Duplicates
Related Issues
Possible Duplicates
Brief Summary
GroupResult.completed_count() (and related methods like .ready()) determine group progress by iterating every child AsyncResult and checking its state individually — one backend round-trip per task. For large groups (e.g. 10,000+ tasks), this means thousands of backend calls just to answer "how many are done."
Celery already solves an equivalent problem for chords: on_chord_part_return() in the Redis (and GCS) backends maintains an atomic, backend-side counter (.j/.s/.t keys in Redis) so the backend can determine "are all chord members done?" in a single pipelined operation, without polling each child. That mechanism exists specifically to decide when to fire the chord callback — but the same primitive seems like it could answer a more general question: "how many of N tasks in this group are done?" for plain groups too, not just chord headers.
This issue proposes discussing whether it's worth exposing an efficient, backend-native progress query for plain GroupResults, reusing the bookkeeping pattern that already exists for chords, rather than requiring O(N) client-side polling.
Design
Architectural Considerations
None — this would be implemented within Celery's existing backend abstraction (celery/backends/base.py, with a concrete implementation in celery/backends/redis.py, and potentially gcs.py). No new external dependencies. Backends that don't implement the new capability would fall back to the current O(N) behavior, so this would be additive rather than a breaking change to the Backend interface.
Proposed Behavior
This is a starting hypothesis, not a fixed design — I'd like feedback on the direction before settling on specifics.
At a high level:
- Today: querying group progress means iterating every child result and checking its state — O(N) backend calls.
- Proposed: for backends that support it, group progress could be tracked via a backend-maintained counter (similar to the existing chord
.j/.s key pattern in the Redis backend), so a progress query costs O(1) instead of O(N).
Open questions I don't want to prematurely answer:
- Should this be opt-in per-group (to avoid writing extra backend state for every group that never asks for progress), or could it be made cheap enough to be the default path for
completed_count()?
- Should retries/re-runs of a task be handled by having the counter track distinct task attempts, or final states only?
- How should this interact with
result_expires / TTL on the counter keys?
Proposed UI/UX
Not locked in — but a plausible shape, to make the discussion concrete:
- A way to opt a group into progress tracking (e.g. a flag when creating/applying the group).
- A new read method (e.g.
GroupResult.progress() returning (completed, total)) that costs one backend call, distinct from the existing completed_count() so existing behavior/cost characteristics aren't silently changed for anyone relying on them.
- Feature-detection so calling code (and Celery itself) can tell whether a given backend supports the fast path, falling back gracefully to the current iteration-based approach when it doesn't.
Happy to adjust any of this — flag names, method names, whether it's opt-in vs. default — based on what maintainers think fits Celery's existing conventions better.
Diagrams
Current:
GroupResult.completed_count()
↓
iterate every AsyncResult in the group
↓
N backend state checks (O(N))
Proposed:
GroupResult.progress()
↓
backend-maintained aggregate counter
↓
single query (O(1)), for backends that support it
For reference, the existing chord mechanism this would build on lives in celery/backends/redis.py, in on_chord_part_return() / add_to_chord() / set_chord_size().
Alternatives
- Application-level workarounds: applications needing efficient progress today could maintain their own counter (e.g. incrementing a key in Redis from within each task, or attaching a no-op chord callback purely to piggyback on the existing chord bookkeeping). These work but push the responsibility for correct, race-safe counting onto every application individually rather than solving it once in Celery.
- Leaving
completed_count() as-is and only documenting the cost characteristics, so users know to avoid it at scale. Simpler, but doesn't actually solve the underlying gap.
Checklist
Related Issues and Possible Duplicates
Related Issues
Possible Duplicates
Brief Summary
GroupResult.completed_count()(and related methods like.ready()) determine group progress by iterating every childAsyncResultand checking its state individually — one backend round-trip per task. For large groups (e.g. 10,000+ tasks), this means thousands of backend calls just to answer "how many are done."Celery already solves an equivalent problem for chords:
on_chord_part_return()in the Redis (and GCS) backends maintains an atomic, backend-side counter (.j/.s/.tkeys in Redis) so the backend can determine "are all chord members done?" in a single pipelined operation, without polling each child. That mechanism exists specifically to decide when to fire the chord callback — but the same primitive seems like it could answer a more general question: "how many of N tasks in this group are done?" for plain groups too, not just chord headers.This issue proposes discussing whether it's worth exposing an efficient, backend-native progress query for plain
GroupResults, reusing the bookkeeping pattern that already exists for chords, rather than requiring O(N) client-side polling.Design
Architectural Considerations
None — this would be implemented within Celery's existing backend abstraction (
celery/backends/base.py, with a concrete implementation incelery/backends/redis.py, and potentiallygcs.py). No new external dependencies. Backends that don't implement the new capability would fall back to the current O(N) behavior, so this would be additive rather than a breaking change to theBackendinterface.Proposed Behavior
This is a starting hypothesis, not a fixed design — I'd like feedback on the direction before settling on specifics.
At a high level:
.j/.skey pattern in the Redis backend), so a progress query costs O(1) instead of O(N).Open questions I don't want to prematurely answer:
completed_count()?result_expires/ TTL on the counter keys?Proposed UI/UX
Not locked in — but a plausible shape, to make the discussion concrete:
GroupResult.progress()returning(completed, total)) that costs one backend call, distinct from the existingcompleted_count()so existing behavior/cost characteristics aren't silently changed for anyone relying on them.Happy to adjust any of this — flag names, method names, whether it's opt-in vs. default — based on what maintainers think fits Celery's existing conventions better.
Diagrams
For reference, the existing chord mechanism this would build on lives in
celery/backends/redis.py, inon_chord_part_return()/add_to_chord()/set_chord_size().Alternatives
completed_count()as-is and only documenting the cost characteristics, so users know to avoid it at scale. Simpler, but doesn't actually solve the underlying gap.