CoRCi: Cross-Reconstruction of Coherent Interests Modeling in Cross-Domain Sequential RecommendationConference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NYWoodstock ’18: ACM Symposium on Neural Gaze Detection, June 03–05, 2018, Woodstock, NYISBN: 978-1-4503-XXXX-X/18/06CCS: Information systems Recommender systems

Multi-head attention (MHA) (Vaswani et al. 2017) is a core technique in modern sequence models. It enables parallelized, context-aware modeling. However, the definitions and architectures associated with encoders and decoders differ slightly in various works. For instance, the Transformer’s encoder does not employ the causal mask, and its decoder uses two MHA layers (Vaswani et al. 2017). Recent LLMs adopt a decoder-only structure, where each decoder consists of one MHA layer with causal mask (Touvron et al. 2023; Achiam et al. 2023; Bai et al. 2023), and a Feedforward network (FFN). SASRec employs MHA with the causal mask (Kang and McAuley 2018), and most MHA-based sequential recommenders leverage SASRec as their sequence encoder (Cao et al. 2022a; Ye et al. 2023; Bian et al. 2025). In contrast, BERT4Rec (Sun et al. 2019) employs bidirectional MHA without the causal mask. This section defines the attention module (ATTN), comprising an MHA layer and an FFN.

Formally, let the query, key, and value matrices be denoted by $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}\in\mathbb{R}^{L\times d}$, where $L$ is the input length, and $d$ is the embedding dimension. The scaled dot-product attention is defined as:

where $d_{k}$ is the dimension of the key vectors. The causal mask $\mathbf{M}^{c}\in\mathbb{R}^{L\times L}$ is defined as $\mathbf{M}^{c}_{ij}=\begin{cases}0,&\text{if}\ j\leq i,\\ -\infty,&\text{if}\ j>i,\end{cases}$ where $1\leq i,j\leq L$, which ensures that each position in a sequence can only attend to previous or current positions, and not to any future positions.

To extend single-head attention to multiple parallel heads, the MHA mechanism projects the inputs $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ into $h$ subspaces using learned parameter matrices $\mathbf{W}_{i}^{Q}\in\mathbb{R}^{d\times d_{k}}$, $\mathbf{W}_{i}^{K}\in\mathbb{R}^{d\times d_{k}}$, and $\mathbf{W}_{i}^{V}\in\mathbb{R}^{d\times d_{v}}$. The outputs of the attention heads are concatenated and sent into an output projection. Specifically, the MHA mechanism is given by:

where $d_{v}=d_{k}=d/h$, and $\mathbf{W}^{O}\in\mathbb{R}^{d\times d}$ denotes the learnable output projection .

To endow the model with nonlinearity and to consider interactions between different latent dimensions effectively, we adopt the FFN from Llama (Touvron et al. 2023) and define it as:

where $\otimes$ denotes Hadamard product, $\mathbf{W}_{1}\in\mathbb{R}^{d\times\frac{8}{3}d}$, $\mathbf{W}_{2}\in\mathbb{R}^{d\times\frac{8}{3}d}$, and $\mathbf{W}_{3}\in\mathbb{R}^{\frac{8}{3}d\times d}$ are learnable matrices, and the Swish activation function is defined as $\text{Swish}(x)=\frac{x}{1+e^{-x}}$ (Touvron et al. 2023).

We also add residual connection, dropout, and layer normalization into the pipeline to mitigate overfitting:

In implementation, all computations will be batchified for acceleration. Finally, the attention block ATTN can be abbreviated as:

In this paper, we focus on the dual-target CDSR task, which is defined as follows. Let $\mathcal{I}_{\mathsf{A}}$ and $\mathcal{I}_{\mathsf{B}}$ denote the complete item sets for non-overlapping domains A and B, respectively. For a user, let its A-domain sequence $T_{\mathsf{A}}=(a_{1},a_{2},\ldots,a_{k})\in\mathcal{I}_{\mathsf{A}}$, and B-domain sequence $T_{\mathsf{B}}=(b_{1},b_{2},\ldots,b_{q})\in\mathcal{I}_{\mathsf{B}}$. The objective is to transfer knowledge between these sequences to generate top-N recommendations for the next items, $a_{k+1}$ and $b_{q+1}$, in their respective domains. The dual-target CDSR task can be formulated as:

Continuing with the above example, we define the combination of domains A and B as the mixed-domain, M. Its item set is then denoted by $\mathcal{I}_{\mathsf{M}}=\mathcal{I}_{\mathsf{A}}\cup\mathcal{I}_{\mathsf{B}}$. As shown in Figure 4, we define the user’s mixed-domain interaction sequence $T_{\mathsf{M}}$ $=$ $(a_{1},b_{1},a_{2},b_{2},b_{3},\ldots,a_{k-1},\allowbreak b_{q-1})$. Every domain-specific sequences can be redefined as the domain-masked mixed-domain sequences as follows: $T_{\mathsf{A}}$ $=$ $(a_{1},o,\allowbreak a_{2},o,o,\ldots,a_{k-1},o)$ for domain A, and $T_{\mathsf{B}}$ $=$ $(o,b_{1},o,b_{2},b_{3},\ldots,o,\allowbreak b_{q-1})$ for domain B, where $o$ represents the all-zero padding item. After padding, $T_{\mathsf{A}}$ and $T_{\mathsf{B}}$ have the same length as $T_{\mathsf{M}}$ to accelerate subsequent tensor calculations. The positional index of the interactive items in each sequence will be assigned separately in reverse chronological order, ignoring padding.

We employ one shared item embedding table $\mathbf{E}^{\mathcal{I}}\in\mathbb{R}^{|\mathcal{I}_{\mathsf{M}}|\times d}$ and one shared positional embedding table $\mathbf{E}^{\mathcal{P}}\in\mathbb{R}^{L\times d}$ to project each numeric item index and positional index into a $d$-dimensional embedding vector, respectively. $L$ denotes the maximum input length. Finally, we represent the embedding of $T_{\mathsf{M}}$, $T_{\mathsf{A}}$, and $T_{\mathsf{B}}$ as $\mathbf{E}_{\mathsf{M}}$, $\mathbf{E}_{\mathsf{A}}$, and $\mathbf{E}_{\mathsf{B}}$, respectively, where each embedding is the summation of corresponding item embedding and positional embedding.

Inspired by SASRec (Kang and McAuley 2018), we leverage the self-attention mechanism to conduct sequential encoding. Specifically, at this stage, we instantiate one self-attention encoder (SAE) for each specific domain, A or B, respectively. The process can be described as follows:

To reconnect domain-invariant interest dispersed in specific-domain sequences, we design a cross-attention module under the Cross-Reconstruction approach, denoted CR_M, which produces mixed-domain representations from the specific-domain representations pre-encoded in Equation 9. Specifically, CR_M leverages a cross-attention layer (Kang and McAuley 2018) to conduct the reconstruction. Because at each time step at most one of $\mathbf{H}_{\mathsf{A}}$ or $\mathbf{H}_{\mathsf{B}}$ holds a non-padding token, we use the sum of these two representations as the K and V input for cross-attention. For the Q input, we employ the $\mathbf{E}_{\mathsf{M}}$ to introduce the mixed domain position information. The overall process of CR_M can then be formulated as follows:

To transfer the reconstructed mixed-domain knowledge back into the specific-domain representations, we again leverage cross-attention. Unlike token-wise addition, which only aligns corresponding positions, cross-attention enables each token in a specific domain to attend to the entire history of mixed-domain interactions, yielding a more comprehensive transfer. Specifically, we instantiate a Cross-Attention Transfer (CAT) for each specific domain A or B, respectively, which are formulated as follows:

where $\hat{\mathbf{H}}_{\mathsf{M}}$ denotes stop-gradient $\mathbf{H}_{\mathsf{M}}$. This prevents mixed-domain modeling from being distorted by specific-domain supervision, thereby maintaining the coherence of domain-invariant interests.

In this subsection, we introduce the losses used for optimizing specific and mixed domains. For specific domains, we employ vanilla InfoNCE (Oord et al. 2018). Given a query representation $h$, its ground truth embedding $e^{+}$, and a candidate set $E^{\pm}$ formed by $e^{+}$ and the embeddings of $N_{neg}=128$ unobserved negative samples within the same domain, InfoNCE is defined as:

where $\tau$ denotes the temperature hyperparameter. Hence, the losses for domains A and B are computed as:

where $\hat{h}_{m,j}$ denotes the corresponding stop-gradient encoded mixed-domain representation of the same item. Here, we regard the mixed-domain embedding $\hat{h}_{m,j}$ as a task-agnostic backbone, and the specific-domain representations derived in Equation 11 are then trained to learn the residuals that tailor this backbone to specific downstream objectives in each domain.

For the mixed domain, let the query representation be $h_{m}$. We form the candidate set as: $E^{\pm}_{\mathsf{m}}=\{e^{+}_{\mathsf{m}}\}\cup E^{-}_{\mathsf{A}}\cup E^{-}_{\mathsf{B}}$, where $e^{+}_{\mathsf{m}}$ is the embedding of the ground-truth item (positive), and $N_{\text{neg}}$ unobserved negatives are sampled uniformly from each specific domain to form the negative sets $E^{-}_{\mathsf{A}}$ and $E^{-}_{\mathsf{B}}$, respectively. Considering that items drawn from the same domain tend to have higher similarity, inner-product similarity without differentiating domains may tend to introduce intra-domain bias. To counteract this issue, we embed the Focal loss (Lin et al. 2017) into the InfoNCE objective, proposing FocalNCE. Given $\alpha{>}0$ is the focusing parameter. FocalNCE is then defined as:

FocalNCE regards each similarity score as a class probability. Similarities involving the same-domain negatives constitute naturally difficult classes and are penalized extra. Consequently, FocalNCE suppresses this intra-domain bias that would otherwise skew the mixed-domain space, thereby fostering coherent domain-invariant interest representations in CoRCi. Hence, the mixed domain loss is:

Eventually, given the weight parameter $\beta$, the overall objective function is a linear combination of the losses from each domain:

For FLOPs in one forward pass, the MHA module requires $4L^{2}d+8Ld^{2}$, and the FFN requires $168Ld^{2}$. For ABXI, the shared SA and FFN process three sequences, and each of the three projectors (using FFN structure) processes one sequence. Therefore, the total FLOPs are 3 SA plus 6 FFN, which gives $12L^{2}d+120Ld^{2}$. In comparison, for CoRCi, each SA and FFN processes only one sequence, resulting in 5 SA plus 5 FFN, which is $20L^{2}d+120Ld^{2}$ around 1.06× that of ABXI. We omit normalization for both models and LoRA for ABXI. Hence, CoRCi requires a similar number of FLOPs but has more parameters, consistent with the scaling laws of LLMs and generative recommenders (Zhai et al. 2024; Zhang et al. 2024a). Excluding item embeddings, the model contains a total of 5,528,624 learnable dense parameters. In terms of training time, each epoch of all datasets takes less than 30 seconds to train on an NVIDIA RTX4090.

Scaling CoRCi from $2$ to $k+2$ domains adds $k$ additional SAEs and $k$ additional CATs, while the number of CR remains fixed at one. As more domains are added, VRAM usage increases roughly proportionally, since both models follow a similar architectural paradigm.

We conduct experiments on three two-domain datasets constructed from six subsets of the Amazon review dataset (McAuley et al. 2015; He and McAuley 2016b). These datasets include Food-Kitchen (AFK: combining ‘Grocery and Gourmet Food’ as Food and ‘Home and Kitchen’ as Kitchen), Beauty-Electronics (ABE: combining ‘Beauty’ as Beauty and ‘Electronics’ as Electronics), and Movie-Book (AMB: combining ‘Movies and TV’ as ‘Movie and ‘Books’ as Book). During preprocessing, each review is treated as an interaction. First, we select users with valid interaction history in each specific domain, and merge them chronologically to construct mixed-domain sequences. Next, we filter out items with fewer than ten interactions. Following prior studies (Bian et al. 2025; Kang and McAuley 2018), we retain a maximum of 50 recent interactions for each mixed-domain sequence. Lastly, we exclude users whose specific-domain sequences contain fewer than five interactions in any domain. The statistics of these datasets are summarized in Table 1.

We train each model five times using different random seeds to assess stability and robustness. The performance is evaluated using three metrics: Hit Rate (HR), Normalized Discounted Cumulative Gain (NDCG), and Mean Reciprocal Rank (MRR).

We compare CoRCi to three types of models: single-target CDSR (ST-CDSR) models, dual-target CDSR (DT-CDSR) models, and Sequential Recommendation (SR) models. Furthermore, SR models can be subdivided into three groups based on target settings: single-target (ST-SR), dual-target (DT-SR), and mixed-target (MT-SR).

Specifically, ST-SR models are trained on a single domain at a time. ST-CDSR models are trained on the combined dataset but designate a single domain as the target for each run. Both MT and DT models consider all domains as targets and are trained on a complete cross-domain dataset. The MT-SR models do not distinguish between domains during modeling or loss calculation. In contrast, both DT-SR and DT-CDSR models aggregate per-domain losses. The DT-CDSR model incorporates domain information during training, while DT-SR does not. The baselines are listed as follows: SASRec (Kang and McAuley 2018) (SR) is a milestone SR model that leverages self-attention to model sequential dynamics. BERT4Rec (Sun et al. 2019) (SR) leverages the Cloze objective with bidirectional self-attention to model sequential dynamics. CD-SASRec (Alharbi and Caragea 2022) (ST-CDSR) aggregates source and target domain sequential dynamics using multiplicative attention and self-attention, respectively, to generate the final representation. CD-ASR (Alharbi and Caragea 2021) (ST-CDSR) employs self-attention to extract knowledge from source domains and encode the target-domain sequence; these representations are then aggregated to predict. MGCL (Xu et al. 2023) (ST-CDSR) designs a multi-view framework with contrastive learning to fuse graphical and sequential information from both mixed-domain and specific-domain perspectives. C²DSR (Cao et al. 2022a) (DT-CDSR) establishes contrastive learning by applying random replacement to create negative sequences, reinforcing alignment between mixed- and specific-domain representations. DREAM (Ye et al. 2023) (DT-CDSR) enhances mixed-domain representations by injecting adaptively extracted knowledge from encoded specific-domain sequential representations. ABXI (Bian et al. 2025) (DT-CDSR) unifies the mixed- and specific-domain tasks by employing task-guide alignment and conducts domain-invariant interests adaptation on one shared encoder.

Following (Kang and McAuley 2018; Tang and Wang 2018; Sun et al. 2019; Bian et al. 2025), we adopt leave-one-out strategy. As shown in Figure 4, the last and penultimate interactions of the mixed-domain sequence serve as testing and validation targets, regardless of domain. Metrics are computed per domain. For simplicity and fairness, we follow (Kang and McAuley 2018; Tang and Wang 2018; Sun et al. 2019; Bian et al. 2025) and sample $N_{\text{mtc}}$ = 999 negative items from the ground truth’s domain in metric calculation. Table 1 reports the number of ground truths per domain in the validation and testing sets. For a fair comparison, unspecified hyperparameter settings follow (Bian et al. 2025) and are detailed in Table 2.

For each dataset, we designate the domain with better metrics as the easy domain and the other as the hard domain. Table 3 summarizes the performance of CoRCi relative to the baselines, from which we derive the following key findings.

Our proposed CoRCi model consistently achieves optimal performance across all evaluated metrics and tasks, with all improvements statistically significant (paired t-tests, $p\leq 0.01$). The average per-domain gains across four metrics are 4.99% and 6.33% on AFK, 5.77% and 11.06% on ABE, and 3.18% and 4.68% on AMB. Notably, CoRCi delivers greater improvements on the hard domains while also making decent progress on easy domains, demonstrating its effectiveness in capturing domain-invariant interests and adaptively reintegrating them into domain-specific modeling.

SASRec-mt and BERT4Rec-mt also perform strongly on the dual-target CDSR task by merely modifying the loss calculation. However, aside from ABXI, no baseline consistently outperforms them across all datasets. Since both CoRCi and ABXI emphasize the capturing of domain-invariant interests, these results validate the importance of this capability in improving CDSR performance.

The DT-CDSR models outperform the ST-CDSR models overall. This is because the dual-target tasks lack noticeable sparsity differences between domains. Therefore, their advantage in transferring knowledge from rich domains to sparse domains is weakened.

In this subsection, we conduct ablation studies to quantitatively evaluate the design choices of CoRCi via five ablation variants:

We report individual and aggregated domain performance on MRR for each dataset to provide a comprehensive evaluation. As reported in Table 4 (RQ2), the variants V${}_{\text{1enc}}$ and V${}_{\text{3enc}}$ underperform CoCRi on every metric. V${}_{\text{3enc}}$, which deploys three independent encoders, regains only a marginal portion of the lost performance relative to V${}_{\text{1enc}}$. These results verify that the proposed SAE–CR hybrid is more effective in modeling domain-invariant interests than either a single shared encoder or fully decoupled encoders. Additionally, the inferior results of V${}_{\text{w/o-cat}}$ highlight the crucial role of cross-attention in enabling sequence-level fusion. Likewise, the marked performance declines of V${}_{\text{w/o-sg}}$ and V${}_{\text{w/o-}\mathcal{L}{\mathsf{M}}}$ verify that both the stop-gradient operation and the standalone mixed-domain objective are indispensable.

We analyze two factors from the perspective of mitigating domain discrepancies: the loss selections and the focusing parameter $\alpha$.

For each variant V, we report the performance of final specific-domain representations and the performance of encoded mixed-domain representations (M-V, cf. Equation 10).

In the upper part of Table 4, we observe that altering FocalNCE and InfoNCE degrades the easy domains of every dataset and the hard domain of AFK. For the hard domains of ABE and AMB, there are statistically insignificant fluctuations.

The lower half of Table 4 reports the performance of the encoded mixed-domain representations. Except for V${}_{\text{dt}}$, the results match those of the upper half on AFK, AMB, and the Beauty domain of ABE. On the Electronics domain of ABE, the gap between the two halves is statistically insignificant. This suggests that when CoRCi does not extract useful domain-specific information, it also refrains from injecting harmful noise. These findings suggest that significant mixed-domain performance gains are crucial for enhancing downstream performance.

Notably, V_dt degenerates consistently across both evaluations. This aligns with the finding in Section 1, which emphasizes that preserving coherent, domain-invariant interests is crucial for improving mixed-domain performance.

Figure 6 illustrates the concave relationship between $\beta$ and MRR performance, where too large or too small a value will degrade performance. The optimum occurs at $\beta=1.0$ for AFK and ABE, and $\beta=0.7$ for AMB. The latter corroborates earlier findings in Section 1 that AMB is driven more by domain-specific interests. Besides, the Beauty domain exhibits a slight gain at $\beta=5$, but the Electronics domain collapses, confirming that the optimal $\beta$ may differ between paired domains in one dataset. Overall, it is practical to tune the weights between mixed- and specific-domain losses.

To further assess the generalizability of CoRCi, we extend the experiments to three domains and non-overlapping users, which better reflects real-world applications. Specifically, we utilize the Douban dataset (Zhu et al. 2019), incorporating all three available domains (Movie, Book, and Music), namely DMBM. We preprocess the data by removing items with fewer than five occurrences, truncating mixed-domain sequences to the most recent 100 interactions, and filtering users with fewer than five interactions. Users are not required to interact in all domains. Timestamps are recorded at day granularity. For ties within a day, we impose the priority Movie > Book > Music, reflecting the descending interaction volume. This ordering intentionally heightens domain imbalance: Book and Music interactions occur later in the sequence, increasing their likelihood of serving as validation or test ground truths while reducing their effective training lengths. Table 1 summarizes the resulting DMBM statistics.

Table 5 contrasts CoRCi with ABXI (the best-performing from RQ1) across three evaluation metrics. CoRCi achieves significant gains over ABXI in all nine domain-metric combinations ($p$ < 0.01).

Figure 1 depicts a shortened snippet from user A11FFLD0GV82CQ in the AFK dataset. The complete training sequence is¹¹ 1 Some item pages have been removed, so we identify them based on logged reviews.: Granola Cereal (F) → Grill Pan (K) → Griddle Pan (K) → Large Water Dispenser With Standard Filter (K) → Fruit and Nut Bar (F) → Gourmet Coffee (F) → Decaffeinated Green Tea (F) → Golden Honey Oat Graham (F) → Oat Chocolate Chip Coconut Bar (F) → Airtight Container (K).

The recent interactions reveal a preference for healthy breakfast snacks (Food) and corresponding storage (Kitchen). The validation and testing items, Sunrise Crunchy Vanilla Cereal (F) and Freshware Silicone Molds (K), are thematically consistent. CoRCi ranks them 3rd and 4th, respectively, demonstrating its ability to preserve coherent domain-invariant interests and thus deliver accurate downstream recommendations.