PinEqualizer: Full Funnel Content Exploration and Debiasing System at PinterestConference: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of KoreaProceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’26), August 09–13, 2026, Jeju Island, Republic of KoreaDOI: 10.1145/3770855.3818461ISBN: 979-8-4007-2259-2/2026/08CCS: Information systems Recommender systems

The objective of full-funnel content exploration is to maximize the long-term user value generated by newly ingested content. We measure this via a long-term fresh content holdout that is applied consistently across all major discovery surfaces (Homefeed, Related Pins, and Search). In this design, the holdout cohort is limited to the content created before the start of the experiment, while the production cohort incorporates newly created content. To remove the inherent advantage of a larger candidate pool, we randomly remove an equivalent amount of content in the production group (based on content ID hash to ensure the same content is removed consistently for all requests), illustrated in Figure 1.a. The observed engagement delta therefore quantifies the incremental benefit of the newly created and explored corpus (Figure 1.b). Instead of demonstrating that a bigger explored content corpus leads to higher user engagement (already well established in (Su et al. 2024)), our main goal is to ensure that the freshest explored corpus provides incremental value over the stale content corpus in the holdout (which has a sizable, fully-explored content corpus).

While long-term holdouts provide the ground truth for fresh content value, they suffer from significant feedback latency. The cumulative benefits of full funnel exploration often persist well beyond the initial intervention window. Therefore, we define a content graduation metric as a robust intermediate proxy. Our methodology is grounded in the observation that successful exploration transitions content from a high-variance "exploration" stage to a stable "exploitation" stage. Formally, we define the content graduation metric as the quantity (and proportion) of newly ingested content accumulating $X$ positive engagements within $Y$ days of ingestion, where we identified specific cumulative engagement threshold $X$ based our empirical data where the variance of engagement rate decreases significantly (Figure 1.c).

In addition, differently from (Su et al. 2024), we want to avoid spending traffic on exploring content with low engagement potential. Therefore, we define the explored low-engaging content metric as the quantity of fully-explored content with the upper confidence bound of engagement rate less than some threshold $Z$. Combining both metrics, we then formally define under-explored content as content that is neither "graduated" nor "explored low-engaging".

As previous work (Su et al. 2024) mentioned, user-segmented A/B experiments are not able to evaluate content-level metrics due to the shared corpus between arms. In our work, we use under-explored content engagement, defined as the volume of positive engagements received by under-explored content, in traditional user-segmented A/B experiments to measure system improvements.

There are several advantages of this experiment design: (1) this metric is strongly correlated with content graduation since more engagement to under-explored content is directly leading to graduation, (2) using positive engagement instead of impressions as the success criteria encourages exploration algorithms that have high relevance and low engagement tradeoffs, (3) we can confidently evaluate any overall user-level engagement tradeoffs when making launch decisions, and (4) since a standard user A/B setup is used, experimentation speed is maximized.

Theoretically, content level leakage is still possible in this setup, which causes under-estimation of the gain of this metric (i.e., better exploration in treatment could cause under-explored content to gain traffic in control). In reality, we observe that this is minimal, as (1) once the treatment accelerates content graduation, the graduated content will not be considered in the metric of the following day, and (2) the content a user sees in the control vs treatment group is typically distinct, especially when experiment traffic is low.

Figure 2 shows the diagram of a typical industry-scale multi-stage funnel for major recommendation and search product surfaces at Pinterest. Each product surface has its unique context as query input to the system: homefeed is a recommendation surface that only takes user information as input, while search takes the current query from the user to ensure relevance of the results in addition to user information for personalization.

To successfully introduce and gather user feedback for cold-start content, the entire content funnel requires stage-by-stage improvements. A significant bottleneck at any stage can throttle distribution to users. A key finding and contribution of this work is the recognition that cold-start content must effectively compete with existing content during the ranking and selection processes at every stage of the funnel and it is critical to reduce the system’s overall bias towards old content to ensure new content has a fair chance.

The corpus selection stage is responsible for choosing the active corpus to index from the entire corpus by removing low quality and/or policy violating content and identifying high quality, engaging content to serve. Here we built new components to effectively select the most promising fresh content to explore since it is very costly, in both infrastructure and engagement, to fully explore every piece of new content at Pinterest’s scale. The retrieval stage contains various types of candidate generators to retrieve the most relevant content given the context query to pass to the full ranking, e.g., based on text token like search query, graph random-walk (Eksombatchai et al. 2018), embedding or ML-based two tower architecture. Here we improved existing mechanisms and built dedicated candidate generators for cold-start content. In the full ranking stage, our main goal is to reduce the model’s bias on both new and old content via improvements on features, architecture, loss function, calibration, as well as optimization in the final utility function for exploration. In the offline components, we improved training data sampling, and deployed a full funnel analysis to identify bottlenecks to guide our prioritization for further development.

The stages of a multi-layer recommendation funnel are highly interdependent and the realized impact of improvements at a given stage (e.g. ranking) can be constrained by the state of preceding stages (e.g. corpus selection and retrieval). Consequently, it is critical to maintain a comprehensive view of bottlenecks and highest-ROI intervention points across the entire funnel. To this end, we employ a full-funnel log and bottleneck analysis that examines, for each stage, the input composition and output survival of cold-start versus existing content, specifically:

• •

  Input availability: fraction of input content that is under-explored and ingested within Y days;

• •

  Output survival rate: probability of surviving in the top-k results among under-explored and newly ingested content.

Intuitively, a given stage can be bottlenecked, or at least inefficient, when fresh content is sufficiently present in the input but the survival rate in the output is low. Due to differences in product surface characteristics and the current state of underlying production systems, these inefficiencies can vary across surfaces, making this full funnel perspective critical to sequence work effectively.

On the Search surface during the same time frame, we have found the bottleneck starts at the retrieval stage. Given the strong relevance bar on search, we also hypothesized if there is enough relevant content for the query in the corpus. To be able to measure it, for a small sample of queries we over-retrieved a much larger portion of fresh content compared with the production system and ran the relevance model on the results; this validated the availability of relevant content at the corpus level and therefore confirmed that the bottleneck is at retrieval.

We have a large volume of Pins uploaded to Pinterest daily, making it infeasible to index and explore all of them. Therefore, we built a corpus selection pipeline that effectively identifies a subset of high-potential fresh items to explore. By maintaining a dedicated exploration corpus and prioritizing the most promising fresh items, the system learns more quickly while reducing wasted traffic to content with low engagement potential.

While detailed implementation of this model is not critical to our main finding, at a high level: for each item, the model predicts its engagement rate as $\alpha$. We then combine it with the empirical engagement rate to calculate the posterior, which is used for final selection:

where $e$ is the number of observed engagements, $n$ is the number of impressions, and $N$ controls the strength of the prior.

Having the right features is critical for debiasing the ML model, and in this section we will discuss the successful techniques we developed. Some of them are effective in improving the prediction on fresh content, while others can reduce the model’s bias favoring existing old content. Most of these feature improvements can be applied to both retrieval stage model (e.g., two-tower learned retrieval) as well as the final ranking model.

To mitigate this issue, we apply dropout to historical item engagement features during training, forcing the model to generalize better and depend more on content signals. We evaluate two random dropout strategies: (1) Uniform dropout, where all item engagement features are dropped simultaneously. (2) Individual dropout, where each item engagement feature is dropped independently. Empirically, individual dropout consistently yields better performance. A feature-importance analysis confirms the effectiveness: models trained with individual dropout place less importance on engagement features and shift weight toward content and context based features. We do not find it to be effective to drop out user-level historical engagement features, therefore user-level personalization is not impacted by this.

At Pinterest, we use three different retrieval strategies: Token Retrieval, Graph Retrieval, and Embedding Retrieval. Token Retrieval is a term-matching retrieval setup where documents are indexed into an inverted index and queries are expressed as structured query trees to fetch matching documents efficiently. Graph Retrieval (Eksombatchai et al. 2018) performs random walks within a graph from the query node and returns the visited nodes (documents) as candidates. Embedding Retrieval is an ANN (Approximate Nearest Neighbor)-based search based on query vs document embedding similarity.

As we move to the late stage of the search and recommendation funnels, we have a typical setup where one or multiple multi-task heavy ranking models with the most advanced architecture and features predict the various rewards of the items to the user (e.g., probability of a save, a click, or a relevance score); and a final utility layer that combines these rewards and other global factors, such as diversity, to form the final order of results to present to the user.

There are two key challenges at this stage. First, due to factors such as sparsity of cold start items in training data, ranking models often tend to under-predict rewards for fresh content and over-predict for old content. To solve this problem, in addition to the feature improvements discussed previously, we developed the following additional techniques to debias the model:

After model prediction bias, the second major challenge in a large scale search and recommendation system is the feedback loop: the training data for predicting user engagement is collected from actual user engagement history, which depends on what the production system decides to show to the user in the first place. This is a well known issue, and in the context of this paper it means that if the production system severely under-predicts fresh content, the amount of training data on fresh content will be further reduced, and simple up-sampling or off-policy correction is not sufficient. To solve this problem, we developed several techniques to improve the final utility layer to allow more exploration of cold-start items:

where $\alpha$ controls the exploration strength and $\beta$ determines how quickly the bonus decays with accumulated impressions.

We experimented with two integration approaches: (1) Adding the UCB term directly as a logit adjustment within the ranking model, and (2) incorporating it as an explicit term in the utility layer. Both implementations demonstrated strong performance in reducing selection bias and improving training data quality, though the utility layer integration offered greater flexibility for tuning independently of model retraining.

To integrate our solution, our ranking model training process consists of two steps: 1) large-scale training of the main model architecture using large volumes of sampled training data, 2) a lightweight calibration step using a smaller sample set, unbiased from sampling logic, to calibrate predictions against empirical engagement rates. We use a feature representation $\phi(x)$ learned during step (1) but update the covariance matrix with the latest data in step (2).

To handle inherently binary tasks, such as click and save predictions, similarly to (Su et al. 2024), we avoid explicitly computing the ridge regression parameters $\theta$ but instead rely on the logit of the pretrained binary prediction as a cheap surrogate for the mean reward. Another practical issue we faced with this setup stems from the magnitude of the design matrix $H$ in Eq. 4 roughly scaling with the sample size $N$ and thus the posterior uncertainty tends proportional to $1/\sqrt{N}$. Since the sample size $N$ from step (2) can vary independently of the pre-training process in step (1) this risks unintentionally varying the overall magnitude of exploration from time to time. To overcome this we add a scaling term to the $\alpha$ parameter in Eq. 3, $\alpha=\sqrt{\frac{N}{N_{o}}}\alpha_{o}$ where $N_{o}$ is set as a reference batch size to maintain stability.

We started to deploy and iteratively improve various parts of the overall system since 2024, and have delivered significant impact to user engagement and content eco-system health with specific side-wide metrics gains in Table 1.

Throughout our work, we have launched a series of experiments that delivered statistically significant lifts in our target experiment metrics. To provide insights into the impact of the improvements in each component, we group these launches by the component they primarily target and summarize the results in Table 2. Note, different from the overall impact gain in Table 1 measured by surface-level long term holdout, the metrics here are simply aggregating the gains from individual launches since 2024. To make the comparison across components more meaningful, we pick a single recommendation surface (Related Pins) to report, and findings and conclusions from other product surfaces are directionally consistent as well. To provide additional insights on cold start improvement, we present engagement lift for both all fresh (created < N days) as well as under-explored content. All experiments we launched had strict guardrail on overall engagement tradeoff.

As evident from Table 2, final metric lifts are observed across different components of the overall full-funnel system. However, introducing a dedicated fresh content exploration corpus for retrieval, together with ranking feature improvements (such as improved content embeddings) disproportionately improved under-explored performance. For UCB based exploration, we compare both the heuristic realtime impression-based method in Eq. 2 and Neural Liner Bandit (NLB) in Eq. 3. We observe that NLB achieves incremental gains over the heuristic approach. However, due to simplicity and easier interpretability of the heuristic, we choose that over the more complicated NLB in certain production scenarios. For example, on the Related Pins surface we deploy the heuristic approach in a light-weighted early funnel ranker and only use NLB in the most advanced final late stage ranking model.

Lastly, to achieve the final impact a key practical lesson from our development journey is that it is important to have the right sequencing and prioritization on which stage to focus on in a multi-stage production system. At high level, we started our effort at the corpus and retrieval stage, to ensure enough fresh content sent to the ranking stage and were able to realize significant impact even with a suboptimal ranking stage. Later we observed that the ranking stage bottleneck became much more significant via our full-funnel analysis, which led us to invest more on ranking improvements. Within the ranking stage, we also examine the availability of cold-start content in training data to prioritize feature / architecture improvements vs explicit UCB-exploration in utility.