LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models

Unless otherwise specified, the implementation details below describe our primary Qwen2.5-VL-7B (3) experiments, which use the full 100K local visual cue search corpus. Additional backbone experiments follow the same data construction and training objective, but may use model-specific training lengths.

We construct the local visual cue search corpus from COCO train2014 object annotations (12). Each example contains a complete image, a localized visual cue cropped from the same image, and the corresponding ground-truth bounding box in the original image. The default training split contains 99,500 training examples and 500 validation examples. We sample tiny and small cue regions at a 70%:30% ratio, where tiny regions occupy less than 1% of the image area and small regions occupy 1–5%. For Qwen2.5-VL-7B (3), all target boxes are represented using absolute pixel coordinates. The resulting corpus statistics are summarized in Table 8.

We perform reinforcement post-training with GRPO using the EasyR1 framework (51; 23). The reward is computed from the model response by parsing the box inside the <answer> tag. The localization reward is the IoU between the predicted box and the ground-truth box, and the final reward combines this IoU reward with a format reward that checks whether the response follows the required <think> and <answer> structure. The format reward weight is 0.1. The main training hyperparameters are listed in Table 9.

All LOCUS post-training runs were conducted on a single node equipped with eight NVIDIA A100-80G GPUs. Table 10 reports the wall-clock time and total GPU-hours for one training run of each backbone. GPU-hours are calculated as the number of GPUs multiplied by wall-clock time. These costs cover reinforcement post-training and exclude benchmark evaluation.

The training prompt provides the full image and the localized visual cue as two image inputs, and asks the model to recover the cue location in the full image. The prompt template used for the primary Qwen2.5-VL-7B training is shown below.

We evaluate all models with the same benchmark adapters and vLLM-based inference pipeline. Unless otherwise specified, decoding uses greedy generation with temperature 0 and a maximum generation length of 8192 tokens. For checkpoints trained with the reasoning format, we append the same <think>/<answer> instruction to evaluation prompts; for models with native or model-specific thinking formats, we use their corresponding inference templates.

Our evaluation covers fine-grained perception benchmarks, including V*Bench (35), HR-Bench-4K/8K (31), CV-Bench (27), and MME-RealWorld-EN (49); visual grounding benchmarks, including RefCOCO, RefCOCO+ (40; 10), and RefCOCOg (19); hallucination benchmarks, including POPE (11) and HallusionBench (7); general perception benchmarks, including MMStar (5), RealWorldQA (1), OCRBench (16), ScienceQA-IMG (SQA^I) (18), and BabyVision (4); and reasoning benchmarks, including MathVision (29), MathVerse (46), WeMath (21), and LogicVista (37). For multiple-choice benchmarks, each adapter extracts the final option letter from the generated response. For open-ended perception benchmarks, we apply the benchmark-specific normalization and matching rules implemented in the corresponding adapter; RealWorldQA is handled according to its multiple-choice or open-ended format. For grounding benchmarks, we parse the generated bounding box and report ACC@0.5. Raw model outputs are saved without truncation before answer parsing, which is important for long reasoning outputs. For the Qwen2.5-VL-7B baseline comparison, all models are evaluated using the same benchmark adapters and answer-parsing protocols. PixelReasoner^† is evaluated under a restricted single-round full-image setting: each example contains only the original image and question, and the model generates one response without inference-time cropping, zooming, visual-tool execution, or additional visual observations. This setting matches the standard image-question inference interface of LOCUS and evaluates the capability encoded in the checkpoint rather than the additional benefits of tool-augmented inference.

Fig. 6 shows representative examples from our local visual cue search training data. Each sample consists of a full image, a localized visual cue cropped from the same image, and the ground-truth target box of the cue in the full image. The examples cover both tiny and small cue regions, illustrating that the model must recover visually subtle local evidence from cluttered full-image context. The visual cue is used only during training to construct a verifiable localization objective; inference uses the standard image-question input without any crop.

For the SFT baseline in Table 4 of the main paper, we use the same 100K local visual cue search examples as LOCUS. To construct reasoning-format supervision, we use Qwen2.5-VL-72B-Instruct (3) as a teacher model. During annotation, the teacher is given the full image, the crop image, and the ground-truth bounding box, and is asked to generate a brief rationale explaining how the crop can be localized in the full image. We then extract the generated rationale and pair it with the ground-truth bounding box as the final answer, yielding responses in the same <think> and <answer> format as the RL training output.

Importantly, the ground-truth box is used only for teacher-side rationale annotation. During SFT training, the student receives the same input as in RL training, namely the full image, the crop image, and the instruction to locate the crop in the full image; the ground-truth box is not included in the prompt. This baseline tests whether supervised imitation of teacher-generated rationales and ground-truth coordinate answers is sufficient, compared with directly optimizing localization quality through the IoU-based RL reward.

The text-cue baseline in Table 5 is designed to isolate the effect of cue modality while keeping the target regions and optimization setup unchanged. We start from the same 100K tiny/small training examples used by LOCUS, each containing a full image, a crop corresponding to the target region, and the ground-truth bounding box in pixel coordinates. For each example, we use Qwen3-VL-235B-A22B-Instruct (2) to generate a short referring expression conditioned on both the full image and the target crop. The annotation prompt asks the model to produce a single concise expression under 25 words that uniquely identifies the cropped object in the full image, focusing on category, appearance, size, and spatial relations to nearby objects.

After generation, each sample is converted into a text-grounding training instance by replacing the visual crop with the generated referring expression. The model receives only the full image and a text query of the form: “Please find the object described by the following text in the image,” followed by the referring expression, and is trained to output the same ground-truth box as $[x_{1},y_{1},x_{2},y_{2}]$ in pixel coordinates. The response format follows the same reasoning template as LOCUS, with intermediate reasoning enclosed by <think> and the final box enclosed by <answer>.

This construction ensures a controlled comparison between text and visual cues: both variants use identical images, target boxes, data split, backbone, GRPO training configuration, and IoU-based localization reward. The only difference is the query modality. The text-cue baseline uses a generated linguistic description as the localization query, whereas LOCUS uses the local visual crop itself.

We use Attention-in-Box (AIB) to quantify whether a model allocates more attention to the ground-truth evidence region during ordinary full-image inference. The analysis is conducted on V*Bench samples with manually annotated object bounding boxes. For each model and each sample, we run the same image-question prompt as in evaluation and extract the attention from the position that predicts the first answer token to all image-patch tokens. We use HuggingFace inference with output_attentions=True; the language-model attention uses the eager implementation to expose attention weights, while the vision encoder uses memory-efficient attention. Image preprocessing follows the evaluation pipeline, including the same chat template, qwen_vl_utils image processing, and processor pixel limits.

For each layer, we average attention weights over heads and keep only the entries corresponding to image-patch tokens. These values are reshaped to the spatial image-token grid derived from image_grid_thw after the model’s spatial merge. Let $A_{l}\in\mathbb{R}^{H\times W}$ denote the resulting attention map at layer $l$. We project the ground-truth bounding box onto the same grid and construct a binary mask $M\in\{0,1\}^{H\times W}$, where a grid cell is included if it intersects any annotated box. The layer-wise AIB is defined as

The mean AIB reported in Fig. 4(a) of the main paper is computed from the layer-averaged attention map, while Fig. 4(b) reports $\mathrm{AIB}_{l}$ across layers. We also compute Peak-in-Box, which checks whether the maximum-attention image token lies inside the ground-truth box, but use AIB as the primary metric because it measures total attention mass assigned to the evidence region.

For qualitative visualization, we upsample the layer-averaged attention map to the original image resolution with bilinear interpolation and overlay it on the image as a heatmap. To reduce visual clutter, low-attention pixels below a percentile threshold are rendered transparent, and the remaining values are shown with a bounded opacity. For side-by-side comparisons, Base and LOCUS heatmaps are normalized with shared peak-ratio scaling so that color intensity remains comparable across the two models. Ground-truth boxes are drawn on the original image to indicate the annotated evidence region.

To examine whether the correlation between localization quality and fine-grained VQA correctness also holds beyond Qwen2.5-VL, we repeat the analysis in Fig. 2 of the main paper using Qwen3-VL-4B-Thinking on the V*Bench direct-attribute subset. As shown in Fig. 7, correctly answered samples exhibit higher grounding IoU and grounding success rate than incorrectly answered samples, and VQA accuracy generally increases with grounding IoU. This provides additional evidence that reliable localization of task-relevant regions is closely associated with fine-grained perception.

We further evaluate whether LOCUS directly improves the proxy task it is trained on, namely local visual cue search. Table 11 reports localization performance on the held-out validation split of our cue-search data, consisting of 500 samples with the same 70% tiny and 30% small cue distribution as the main training setting. The base models show limited zero-shot ability to match a localized visual cue back to its spatial support in the full image, achieving only 16.4 and 21.7 mean IoU for Qwen2.5-VL-7B and Qwen3-VL-4B, respectively. This indicates that local visual cue search is a non-trivial capability even when the target crop is explicitly provided.

After LOCUS training, both backbones obtain large gains across all localization metrics. Qwen2.5-VL improves from 16.4 to 43.0 in mean IoU and from 16.4 to 48.8 in ACC@0.5, while Qwen3-VL improves from 21.7 to 43.2 in mean IoU and from 19.6 to 48.4 in ACC@0.5. These results confirm that the RL objective effectively teaches the model to perform precise local visual matching, providing direct evidence that the downstream gains are grounded in improved cue localization ability.

We additionally report visual grounding results on RefCOCO, RefCOCO+ (40; 10), and RefCOCOg (19) in Table 12. These results are provided as a supplementary evaluation of localization ability under standard text-based referring expressions. On Qwen2.5-VL-7B, LOCUS improves the average ACC@0.5 from 81.4 to 82.7, indicating that local visual cue search does not harm general grounding ability while improving fine-grained perception.

Fig. 8 provides the complete layer-wise Attention-in-Box (AIB) curves on V*Bench. The two models exhibit similar AIB values in early layers, while the gap becomes substantially larger from around layer 19 onward. This pattern suggests that the integration of fine-grained visual evidence for answer generation is more pronounced in later layers, where LOCUS assigns more attention mass to the ground-truth evidence region than the base model.

We show the training dynamics of the primary Qwen2.5-VL-7B LOCUS experiment in Fig. 9. The IoU-based localization reward steadily improves during GRPO training, while the format reward quickly converges, indicating stable optimization of the local visual cue search objective.

Fig. 10 provides additional qualitative examples on V*Bench. Across these cases, the queried evidence is small and easily overlooked within the full-image context. The base model often fails to locate the target object and either predicts an incorrect attribute or concludes that the object is absent. In contrast, LOCUS attends more strongly to the relevant local region and recovers the correct answer under the same full-image input. These examples further illustrate that local visual cue search improves fine-grained perception by strengthening evidence retrieval without requiring inference-time cropping or zooming.

Although LOCUS improves the model’s ability to retrieve local evidence, fine-grained perception also requires correctly interpreting the retrieved visual content. Fig. 11 shows representative failure cases where both the base model and LOCUS produce incorrect answers. In these examples, the models often attend to or describe the relevant region, but still fail to infer the correct fine-grained attribute. For instance, in the Apple-logo example, both models focus on the logo area but interpret its color as a single dominant color rather than recognizing it as polychromatic. Similarly, in the tablecloth example, LOCUS attends to the local region around the tablecloth, yet misidentifies its color under cluttered background and low-resolution visual evidence.

These cases suggest that local visual cue search mainly addresses the evidence retrieval bottleneck, but does not fully solve all fine-grained perception failures. Accurate answers may still depend on attribute-level recognition, semantic disambiguation, and robust interpretation of small or visually ambiguous regions. We therefore view LOCUS as a complementary mechanism that improves access to decisive evidence, while stronger visual understanding remains necessary for resolving cases where the evidence itself is difficult to interpret.