Observed
The bug shows up as negative scaling — the same build (32k trivial fork
bodies) gets dramatically slower and spawns ever more threads as --jobs
increases. Sampling the daemon with the below reproduction code (macOS, 14 cores, mill 1.2.0-RC1-42-dc1f60,
TASKS=2000 FAN=16 WORK=0):
--jobs |
wall |
peak threads BLOCKED on executor monitor |
live pool threads |
| 16 |
7.3s |
114 |
2,010 |
| 256 |
14.7s |
478 |
3,009 |
| 1024 |
47.5s |
1,625 |
5,560 |
| 2000 |
149s |
2,366 |
8,644 |
| 4000 |
did not finish (>15 min) |
— |
— |
More parallelism makes it ~20× slower (16 → 2000) and the pool balloons to
thousands of live threads. Thread dumps show hundreds/thousands of task threads
BLOCKED (on object monitor) on the shared executor, matching what we saw in
production (~130 blocked, pool names reaching ...thread-619162):
"execution-contexts-threadpool-6-thread-57" ... daemon
java.lang.Thread.State: BLOCKED (on object monitor)
at mill.exec.ExecutionContexts$ThreadPool.leaveBlocking(ExecutionContexts.scala:69)
- waiting to lock <0x...> (a java.util.concurrent.ThreadPoolExecutor)
at mill.exec.ExecutionContexts$ThreadPool.blocking(ExecutionContexts.scala:77)
at mill.exec.ExecutionContexts$ThreadPool.await(ExecutionContexts.scala:48)
at build_.package_$Work.out$$anonfun$... (build.mill:23)
Suspect Analysis
ExecutionContexts.ThreadPool.blocking { ... } resizes the shared daemon
ThreadPoolExecutor on every span:
// core/exec/src/mill/exec/ExecutionContexts.scala
def enterBlocking(): Unit = executor.synchronized { // :60
val n = executor.getCorePoolSize + 1
if (n > executor.getMaximumPoolSize) executor.setMaximumPoolSize(n)
executor.setCorePoolSize(n)
}
def leaveBlocking(): Unit = executor.synchronized { // :69
executor.setCorePoolSize(executor.getCorePoolSize - 1)
executor.setMaximumPoolSize(executor.getMaximumPoolSize - 1)
}
def blocking[T](t: => T): T = { enterBlocking(); try t finally leaveBlocking() }Every Task.fork.await, every managed-blocking wait, and every internal
task-lock wait/reacquire goes through this. When many tasks are inside a
blocking { } span at once (normal for a wide graph under a high --jobs), two
things go wrong:
- Monitor convoy — all of them serialize on the single
executor monitor
in enter/leaveBlocking, so hundreds of task threads sit BLOCKED (on object monitor) and no task makes progress.
- Unbounded thread churn —
setCorePoolSize is bumped up on every enter and
down on every leave; under a storm of spans this spawns/reaps workers
continuously, creating hundreds of thousands of threads over a run.
We hit this in production on a wide __.prepareOffline --all: the daemon burned
CPU for ~70 min with ~130 threads BLOCKED on the executor monitor and pool
thread names reaching ...thread-619162 (~619k threads created), while
zero threads did any actual work.
Reproduction (single command, no fork, public build)
build.mill:
package build
import mill.*
// Many tasks, each forks FAN children and awaits them. `Task.fork.await` is
// `ThreadPool.blocking { Await.result(...) }`, so every await enters/leaves the
// managed-blocking section that resizes the shared executor under one monitor.
// WORK=0 makes the fork bodies trivial so coordination cost dominates.
private val tasks = sys.env.getOrElse("TASKS", "2000").toInt
private val fan = sys.env.getOrElse("FAN", "16").toInt
private val workIters = sys.env.getOrElse("WORK", "300000").toLong
object work extends Cross[Work]((0 until tasks).map(_.toString))
trait Work extends Cross.Module[String] {
def out = Task {
val futs = (0 until fan).map { i =>
Task.fork.async(dest = Task.dest / s"f$i", key = i.toString, message = s"f$i") { _ =>
var h = 0L; var j = 0L
while (j < workIters) { h = h * 31 + j; j += 1 }
h
}
}
futs.map(f => Task.fork.await(f)).sum
}
}Set WORK=0 so the fork bodies are trivial and coordination dominates, then
raise --jobs and take a thread dump mid-run:
WORK=0 mill --jobs 1024 'work[__].out' # then: jstack <daemon-pid>
Observed
The bug shows up as negative scaling — the same build (32k trivial fork
bodies) gets dramatically slower and spawns ever more threads as
--jobsincreases. Sampling the daemon with the below reproduction code (macOS, 14 cores, mill
1.2.0-RC1-42-dc1f60,TASKS=2000 FAN=16 WORK=0):--jobsMore parallelism makes it ~20× slower (16 → 2000) and the pool balloons to
thousands of live threads. Thread dumps show hundreds/thousands of task threads
BLOCKED (on object monitor)on the shared executor, matching what we saw inproduction (~130 blocked, pool names reaching
...thread-619162):Suspect Analysis
ExecutionContexts.ThreadPool.blocking { ... }resizes the shared daemonThreadPoolExecutoron every span:Every
Task.fork.await, every managed-blocking wait, and every internaltask-lock wait/reacquire goes through this. When many tasks are inside a
blocking { }span at once (normal for a wide graph under a high--jobs), twothings go wrong:
executormonitorin
enter/leaveBlocking, so hundreds of task threads sitBLOCKED (on object monitor)and no task makes progress.setCorePoolSizeis bumped up on every enter anddown on every leave; under a storm of spans this spawns/reaps workers
continuously, creating hundreds of thousands of threads over a run.
We hit this in production on a wide
__.prepareOffline --all: the daemon burnedCPU for ~70 min with ~130 threads BLOCKED on the executor monitor and pool
thread names reaching
...thread-619162(~619k threads created), whilezero threads did any actual work.
Reproduction (single command, no fork, public build)
build.mill:Set
WORK=0so the fork bodies are trivial and coordination dominates, thenraise
--jobsand take a thread dump mid-run: