fix: run evaluation on main thread when num_threads=1 - #9328
Merged
Conversation
isaacbmiller
requested changes
Mar 3, 2026
Collaborator
There was a problem hiding this comment.
Thanks for the PR! Left some comments
…m docstring - Extract _process_outcome() and _report_progress() helpers used by both _execute_sequential and _execute_parallel - Remove issue reference and framework-specific justification from test docstring
Contributor
Author
|
Hey, pushed a fix for both comments. Extracted shared _process_outcome/_report_progress helpers so sequential and parallel use the same logic now, and trimmed the test docstring. Ready for another look when you get a chance! |
Collaborator
|
I like this! |
isaacbmiller
approved these changes
Mar 4, 2026
Collaborator
|
Thanks for the contribution @zamal-db ! |
MaximeRivest
pushed a commit
to MaximeRivest/dspy
that referenced
this pull request
Mar 5, 2026
* fix: run evaluation on main thread when num_threads=1 (stanfordnlp#9150) * refactor: address review share logic between sequential/parallel, trim docstring - Extract _process_outcome() and _report_progress() helpers used by both _execute_sequential and _execute_parallel - Remove issue reference and framework-specific justification from test docstring
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9150
Problem
When using
dspy.Evaluatewithnum_threads=1, work is still dispatched to aThreadPoolExecutor. This breaks frameworks like AppWorld and Terminal Bench that callsignal.signal()inside their task execution, becausesignal.signal()can only be called from the main thread:\
ValueError: signal only works in main thread of the main interpreter
\\
Even setting
num_threads=1doesn't help, sinceParallelExecutorunconditionally routes throughThreadPoolExecutor.Solution
Add a
_execute_sequential()method toParallelExecutorthat runs items directly on the calling thread whennum_threads == 1. The sequential path preserves all existing behavior:max_errorscancellationfailed_indicesandexceptions_maptrackingcompare_resultsmode)KeyboardInterrupthandlingWhen
num_threads > 1, the existing_execute_parallel()path is used unchanged.Changes
dspy/utils/parallelizer.py: Add_execute_sequential()and route to it fromexecute()whennum_threads == 1tests/utils/test_parallelizer.py: Add 5 tests for sequential execution (main-thread verification, error handling, max_errors, failed index tracking, compare_results)tests/evaluate/test_evaluate.py: Add 1 end-to-end test verifyingEvaluate(num_threads=1)runs the metric on the main thread