CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance

Off-the-shelf encoders, pre-trained on large amounts of unimodal data, possess a rich semantic understanding of their respective modalities, providing a strong foundation for generating meaningful feature representations that can effectively guide lightweight models in multi-modal training. As an initial step of CLIP-PING, we leverage these pre-trained encoders — denoted as $\mathcal{F}_{I}^{*}$ and $\mathcal{F}_{T}^{*}$ — as our image and text feature extractors (i.e., teachers). Given a dataset of (image, text) pairs, $\mathcal{D}=\{(I_{k},T_{k})\}_{k=1}^{|\mathcal{D}|}$, we compute feature representations of teacher encoders for each modality prior to the multi-modal training stage. Specifically, for each $I_{k},T_{k}\in\mathcal{D}$, we obtain image features $\tilde{z}_{k}^{I}=\mathcal{F}_{I}^{*}(I_{k})$ and text features $\tilde{z}_{k}^{T}=\mathcal{F}_{T}^{*}(T_{k})$. To minimize computational requirements, we process each modality separately on a single GPU, ensuring that only one large-scale encoder is loaded into memory at a time. This preliminary feature extraction process enables efficient use of features generated by high-capacity teacher encoders, containing billions of parameters.

Once features are extracted from teacher encoders, they are frozen in reusable, auxiliary feature banks, i.e., $\mathcal{B}_{I}^{*}=\{\tilde{z}_{k}^{I}\}_{k=1}^{|\mathcal{D}|}$ for image and $\mathcal{B}_{T}^{*}=\{\tilde{z}_{k}^{T}\}_{k=1}^{|\mathcal{D}|}$ for text. These feature banks allow frozen features to be accessed efficiently for neighbor retrieval, ensuring intrinsic guidance from the pre-trained teacher encoders during the student training. This one-time unimodal feature extraction significantly streamlines the multi-modal training process, enabling computational efficiency, making CLIP-PING a resource-efficient solution for boosting the performance of lightweight vision-language models.

Following CLIP [36], we consider a dual-encoder architecture, comprising an image encoder $\mathcal{E}_{I}$ and a text encoder $\mathcal{E}_{T}$, to jointly learn meaningful feature representations across modalities. For a batch of $N$ (image, text) pairs, i.e., $\{(I_{k},T_{k})\}_{k=1}^{N}$, image and text features, $z_{k}^{I}=\mathcal{E}_{I}(I_{k})$ and $z_{k}^{T}=\mathcal{E}_{T}(T_{k})$, are obtained through each encoder branch. The objective is to learn the shared feature space, where corresponding images and texts are semantically aligned. To this end, we employ the standard CLIP objective [36] that ensures $N$ positive pairs to closely aligned while $N^{2}-N$ irrelevant pairs remain apart. Mathematically, this can be formalized as a symmetric function based on the InfoNCE [31, 55] loss, consisting of:

and

where $\mathcal{L}_{I\rightarrow T}^{\text{CLIP}}$ aligns image feature $z_{k}^{I}$ with its corresponding text feature $z_{k}^{T}$ by maximizing the similarity in-between, while $\mathcal{L}_{T\rightarrow I}^{\text{CLIP}}$ mirrors this. The similarity $\mathcal{S}(\cdot,\cdot)$ is typically measured by the dot product between features, scaled by a learnable temperature parameter $\tau$. In summary, the overall image-text contrastive loss, $\mathcal{L}^{\text{CLIP}}$, is formulated as:

We leverage frozen features from each auxiliary feature bank (either image or text) to encourage each sample to be in close proximity with its semantically similar or nearest-neighbor (NN) samples within the feature bank. To achieve this, we maintain two support sets, $\mathcal{Q}_{I}\subset\mathcal{B}_{I}^{*}$ and $\mathcal{Q}_{T}\subset\mathcal{B}_{T}^{*}$, capturing representative subsets of frozen features from auxiliary feature banks. Specifically, image feature support set is $\mathcal{Q}_{I}=\{\tilde{z}_{k}^{I}\}_{k=1}^{|\mathcal{Q}_{I}|}$, and text feature support set is $\mathcal{Q}_{T}=\{\tilde{z}_{k}^{T}\}_{k=1}^{|\mathcal{Q}_{T}|}$. The underlying concept of NN retrieval is shown in Fig. 4, i.e., $\text{NN}(\tilde{z}):=\underset{q\in\mathcal{Q}}{\mathrm{argmin}}||\tilde{z}-q||_{2}$ for each modality. Mathematically, intra-modal contrastive supervision through NN samples can be formalized as:

and

where both losses are symmetric. Thus, supervision from image NN is defined by:

Likewise, supervision from text NN is defined by:

The overall objective for intra-modal contrastive supervision through frozen NN samples, $\mathcal{L}_{\text{NN}}^{\text{PING}}$, is formulated as:

By incorporating intra-modal contrastive supervision in this way, CLIP-PING enables student encoders to capture rich semantic features of teacher encoders, which improves the robustness and strengthens alignment within each modality.

We explore cross nearest-neighbor (XNN) samples within each modality, by cross-referencing frozen NN features to obtain more diverse supervisory signals from the dataset. As illustrated in Fig. 4, for a frozen image feature $\tilde{z}_{k}^{I}$, its XNN is identified as the image sample associated with NN of frozen text feature $\tilde{z}_{k}^{T}$, i.e., $\text{XNN}(\tilde{z}_{k}^{I}):=\tilde{z}_{i}^{I}\in\mathcal{Q}_{I}$, where $\tilde{z}_{i}^{T}=\text{NN}(\tilde{z}_{k}^{T})$ for $i$-th pair $(\tilde{z}_{i}^{I},\tilde{z}_{i}^{T})$. Likewise, for a frozen text feature $\tilde{z}_{k}^{T}$, $\text{XNN}(\tilde{z}_{k}^{T}):=\tilde{z}_{i}^{T}\in\mathcal{Q}_{T}$, where $\tilde{z}_{i}^{I}=\text{NN}(\tilde{z}_{k}^{I})$ for $i$-th pair $(\tilde{z}_{i}^{I},\tilde{z}_{i}^{T})$. Mathematically, inter-modal contrastive supervision through XNN samples can be formalized as:

and

where both losses are symmetric. Thus, supervision from image XNN is defined by:

Likewise, supervision from text XNN is defined by:

The overall objective for inter-modal contrastive supervision through frozen XNN samples, $\mathcal{L}_{\text{XNN}}^{\text{PING}}$, is formulated as:

By incorporating inter-modal contrastive supervision in this way, CLIP-PING encourages indirect alignment for semantically similar pairs of frozen features, enabling student encoders to capture richer latent features across modalities.

Together, the intra-modal and inter-modal supervision enables the learning of more generic features with enriched semantic diversity. The overall PING objective can be summarized as:

Here, $\alpha$ is a tunable hyperparameter to control the weight, balancing between intra-modal and inter-modal guidance.

By integrating CLIP objective with PING supervision, we leverage the strengths of cross-modal contrastive alignment with rich, diverse supervision introduced by intrinsic neighbors (i.e., NN and XNN). Thus, the final CLIP-PING objective, $\mathcal{L}^{\text{CLIP-PING}}$, is:

where $\lambda$ is a tunable hyperparameter that controls the contribution of $\mathcal{L}^{\text{PING}}$ relative to the standard CLIP loss, $\mathcal{L}^{\text{CLIP}}$. This combined CLIP-PING objective effectively boosts lightweight models by leveraging knowledge from pre-trained encoders, while significantly reducing computational requirements, thus making CLIP-PING ideal for multi-modal training in resource-constrained settings.

We use the COCO [27] dataset, which contains 600K (image, text) pairs, i.e., 118K images, each with 5 text descriptions. We also explore another training set, where we combine COCO [27] with the Conceptual Captions 3M (CC3M) [38]. Due to download issues, we only obtained 2.3M pairs from CC3M, bringing the total size of the combined dataset to about 3M (image, text) pairs. As our primary focus is on resource-constrained scenarios, we intentionally avoid using larger Internet-scale datasets, commonly employed in several recent works, to prioritize computational efficiency and reduce storage demands.

By default, we use ResNet-v2-50 [19, 15] for image features and BERT-Base [18] (with BertTokenizer and a maximum context length of 55) for text, both initialized with pre-trained weights available on HuggingFace (i.e., timm/resnetv2_50x1_bit.goog_in21k_ft_in1k and google-bert/bert-base-uncased). Feature representations are extracted without data augmentation and stored in pickle format for efficient access during training.

We consider MobileBERT_TINY [39] (with MobileBertTokenizer and a maximum context length of 55) as our text encoder and explore three variations of image encoders, each representing different architectural paradigms: ViT-XS [8] (a smaller transformer-based model), ConvNeXt-Pico [29] (a compact convolutional network), and MNv4-Hybrid-M [35] (a hybrid variant of MobileNetv4 model). Specifications of encoders are listed in Table 1. Unless otherwise stated, we use ViT-XS and MobileBERT_TINY as the default pair for ablations and analysis. Each encoder is paired with a projection head, a two-layer MLP with GELU non-linearity for the first layer.

We compare CLIP-PING against the original CLIP [36] and CLIP-KD [48], which combines feature distillation (FD), interactive contrastive learning (ICL), and contrastive relational distillation (CRD). For experiments with COCO [27], we include additional comparisons with traditional CLIP distillation (CLIP-D), and an efficient CLIP distillation using auxiliary feature banks (CLIP-F).

All lightweight models are trained from scratch for 35 epochs, applying a cosine learning rate scheduler with a linear warm-up over the first 5 epochs. Training batch size is 1024 and AdamW optimizer with a weight decay of 1e-5 is used. Initial learning rates of image and text encoders are 3e-3 and 1e-3, respectively. Input images are resized to $224\times 224$, with RandomResizedCrop as the sole data augmentation during training. We use gradient checkpointing and automatic mixed precision to improve memory efficiency and accelerate training. The learnable temperature $\tau$ is initialized at 0.07, and the projection dimension is 256. By default, we set the supervision loss weight values as $\alpha=0.25$ and $\lambda=0.6$. Our support set is implemented as a first-in-first-out (FIFO) queue, with a size of 32,768. More details are provided in the supplementary material.

Models are evaluated across several downstream tasks, including cross-modal retrieval, zero-shot classification, and linear evaluation to assess the efficacy of CLIP-PING. For cross-modal retrieval, we use the CC3M [38] validation set, i.e., 13K (image, text) pairs, and the COCO [27] validation set, i.e., 5K images, each with 5 text descriptions. Additionally, we use the Flickr30K [51] test set, i.e., 1K images, each with 5 text descriptions, for zero-shot cross-modal retrieval. Retrieval performance is measured with Recall@K metrics, i.e., R@1 for image-to-text (I2T) and text-to-image (T2I). For zero-shot classification, evaluations are conducted on STL-10 [5], CIFAR-10, CIFAR-100 [20], Stanford 40 Actions [50], and ImageNet1K [6], while linear evaluation is performed on STL-10, CIFAR-10, CIFAR-100, and ImageNet1K. We follow the prompt engineering of CLIP, with specific settings for the Stanford 40 Actions dataset provided in the supplementary material. Top-1 accuracy (Acc) is used as the primary metric for classification tasks, and we report the average across all datasets as “Avg”.

Tables 2 and 4 show performance comparison on cross-modal retrieval task, highlighting CLIP-PING’s consistent improvements across all three lightweight vision-language models. Table 2 summarizes results for models trained on COCO [27], and Tab. 4 provides results for the combined dataset. Notably, CLIP-PING demonstrates robustness across various architectures. The computational overhead of CLIP-PING is minimal compared to explicit distillation methods like CLIP-D and CLIP-KD [48], while remaining as efficient as the original CLIP [49]. Essentially, CLIP-PING achieves competitive performance without the resource-intensive processes required by distillation-based methods. This efficiency likely stems from the way CLIP-PING leverages frozen pre-trained features, which reduces the need for repeated calculations typically associated with training. We also observed that increasing computational resources further enhances performance. Specifically, we find that Active CLIP-PING (A-CLIP-PING) with active teacher encoders for stronger guidance during training, replacing the feature extraction stage, delivers additional performance boosts, for instance, 2.9% and 1.8% improvements on zero-shot Flickr30K [51] retrieval for ViT-XS [8] image encoder trained on COCO+CC3M [27, 38] dataset with computational demands remain comparable to CLIP-KD [48].

We train a linear classifier for 30 epochs with a batch size of 512. The Adam optimizer is used, with a cosine learning rate scheduler and an initial learning rate of 1e-2. As shown in Tab. 3, CLIP-PING outperforms other methods by a notable gap, illustrating an improvement of 4.6% over CLIP-KD [48] for average when evaluating the ViT-XS [8] image encoder trained on COCO+CC3M [27, 38]. Moreover, the A-CLIP-PING variant delivers an additional 4.9% performance boost.

As shown in Tab. 5, CLIP-PING consistently outperforms competing methods in zero-shot image classification across all datasets. Specifically, it achieves an average performance improvement of 4.7% over CLIP-KD [48] with the ViT-XS [8] image encoder trained on COCO+CC3M [27, 38] dataset. The A-CLIP-PING variant provides additional 4.4% performance boost.

As shown in Tab. 6(a), we explore the effect of supervision loss weight $\alpha$ in Eq. 14. As $\alpha$ increases, emphasizing more weight on XNN supervision, the performance gradually declines. However, the drop in performance saturates at $\alpha=0.25$, suggesting that CLIP-PING benefits both from NN and XNN supervision. This indicates that both forms of supervision contribute positively, with $\alpha=0.25$ being the optimal choice to balance them.

As shown in Tab. 6(b), we examine the effect of loss weight $\lambda$ in Eq. 15. Performance gradually improves with increasing $\lambda$ up to 0.6, after which it plateaus, indicating that intrinsic neighbors guidance boosts training while CLIP-PING continues to benefit from foundational cross-modal contrastive alignment.

Table 7(a) presents the results of varying the support set size $|Q|$. As the support set grows, it increases the chance of retrieving a closer NN within the dataset. However, increasing the support set size beyond 32,768 does not yield additional performance gains.

In Tab. 7(b), we explore the impact of selecting one random neighbor from the top-$k$ NN instead of the closest neighbor. We find that increasing $k$ beyond 1 slightly degrades performance, suggesting that the nearest-neighbor strategy offers clearer guidance, while other neighbors may introduce noise.

In Tab. 8(a), we employ ViT-B/16 [8], pre-trained on ImageNet [6] (i.e., vit_base_patch16_224) as our image teacher or feature extractor and analyze its effect on performance. Notably, our CLIP-PING variants still outperform other methods.

In Tab. 8(b), we analyze the effect of supervision sources: both image and text supervision (default), with only text supervision (w/o $\mathcal{L}_{I}^{\text{NN}}$ and $\mathcal{L}_{I}^{\text{XNN}}$ in Eqs. 8 and 13), and with only image supervision (w/o $\mathcal{L}_{T}^{\text{NN}}$ and $\mathcal{L}_{T}^{\text{XNN}}$ in Eqs. 8 and 13). We observed that CLIP-PING benefits equally from both supervision.