Describe the issue
When running BFCL against a self-hosted inference server (llama-server, vllm, etc.), the generate phase produces intermittent 400 responses — typically in bursts of 3–5 — then recovers. The errors are not caused by invalid request bodies, exhaustively verified it was not a server configuration issue. Turn concurrency down to 1 on your server and it happens all the time, if the model is local and fast.
ID datapoint
Logging request.content in an httpx send hook shows a valid, fully-serialized JSON body. Server-side verbose logging for the same request (llama-server) shows an empty body received. So the connection likely was cut before delivery.
What is the issue
OpenAICompletionsHandler creates one OpenAI client that persists for the entire benchmark run. The underlying httpx connection pool keeps connections alive between requests. Between test cases, the inference server may close an idle connection. httpx then reuses that dead connection for the next POST: the server receives only the TCP headers before the RST, sees an empty body, and returns 400. The httpx client does not retry non-idempotent POST requests by design, so the error surfaces as a BFCL failure. This throws off your error rate.
Proposed Changes
Pass a custom http_client with max_keepalive_connections=0 to the OpenAI() constructor. This forces a new TCP connection per request. The overhead is negligible relative to inference latency (LLM inference >> TCP handshake). httpx is already a transitive dependency via the openai package, so no new dependency is iintroduced.
The fix belongs in _build_client_kwargs in openai_completion.py. Here's the patch:
File: bfcl_eval/model_handler/api_inference/openai_completion.py
import json
import os
import time
from typing import Any
+import httpx
from bfcl_eval.constants.type_mappings import GORILLA_TO_OPENAPI
def _build_client_kwargs(self):
kwargs = {}
if api_key := os.getenv("OPENAI_API_KEY"):
kwargs["api_key"] = api_key
if base_url := os.getenv("OPENAI_BASE_URL"):
kwargs["base_url"] = base_url
if headers_env := os.getenv("OPENAI_DEFAULT_HEADERS"):
kwargs["default_headers"] = json.loads(headers_env)
+ kwargs["http_client"] = httpx.Client(limits=httpx.Limits(max_keepalive_connections=0, max_connections=100))
return kwargs
Additional Context:
Disable keep-alive to prevent stale connections from causing 400s. Between benchmark requests, the server may close an idle connection. httpx will attempt to reuse it, sending the POST body on a dead socket. The server receives only headers (empty body) and returns 400. Disabling keep-alive forces a fresh connection per request, which is negligible overhead compared to inference latency.
Describe the issue
When running BFCL against a self-hosted inference server (llama-server, vllm, etc.), the generate phase produces intermittent 400 responses — typically in bursts of 3–5 — then recovers. The errors are not caused by invalid request bodies, exhaustively verified it was not a server configuration issue. Turn concurrency down to 1 on your server and it happens all the time, if the model is local and fast.
ID datapoint
Logging request.content in an httpx send hook shows a valid, fully-serialized JSON body. Server-side verbose logging for the same request (llama-server) shows an empty body received. So the connection likely was cut before delivery.
What is the issue
OpenAICompletionsHandler creates one OpenAI client that persists for the entire benchmark run. The underlying httpx connection pool keeps connections alive between requests. Between test cases, the inference server may close an idle connection. httpx then reuses that dead connection for the next POST: the server receives only the TCP headers before the RST, sees an empty body, and returns 400. The httpx client does not retry non-idempotent POST requests by design, so the error surfaces as a BFCL failure. This throws off your error rate.
Proposed Changes
Pass a custom http_client with max_keepalive_connections=0 to the OpenAI() constructor. This forces a new TCP connection per request. The overhead is negligible relative to inference latency (LLM inference >> TCP handshake). httpx is already a transitive dependency via the openai package, so no new dependency is iintroduced.
The fix belongs in _build_client_kwargs in
openai_completion.py. Here's the patch:File:
bfcl_eval/model_handler/api_inference/openai_completion.pyAdditional Context:
Disable keep-alive to prevent stale connections from causing 400s. Between benchmark requests, the server may close an idle connection. httpx will attempt to reuse it, sending the POST body on a dead socket. The server receives only headers (empty body) and returns 400. Disabling keep-alive forces a fresh connection per request, which is negligible overhead compared to inference latency.