BAHSD: Bridging the Long-tail Gap via Adaptive Distillation in Black-box Sequential Recommendation

Sequential recommendation captures users’ dynamic preferences by modeling temporal dependencies in interaction histories [1, 20, 12, 14, 18, 13]. Transformer-based architectures, notably SASRec [10] and BERT4Rec [22], have become the standard due to their ability to encode long-range patterns via self-attention. In practice, these models are often deployed as black-box APIs that expose only predictions while concealing internal parameters and gradients, necessitating distillation techniques that operate under such constraints [5, 21].

Knowledge distillation transfers knowledge from a teacher model to a student model [8]. Traditional distillation methods in recommendation systems follow white-box [35] or gray-box paradigms [4, 9, 11, 17, 35], which rely on intermediate features, attention maps, or gradient information that are inaccessible under black-box API settings. Recent black-box distillation efforts, including DHKD [30], ABKD [24], ICD [16], and RLD [23], primarily address loss conflicts and capacity gaps in general tasks such as image classification but lack recommendation-specific design. These methods treat teacher outputs as static distributions, overlooking both the temporal dynamics inherent in user sequences and the signal heterogeneity induced by long-tail user distributions. Approaches such as RCE-KD [36], Ekd4rec [26], and DLLM2Rec [3] attempt to refine ranking alignment through item partitioning, yet they operate at the output level without addressing the underlying quality disparity in teacher signals across different user groups.

Long-tail issues have been extensively studied in recommendation systems [34, 6, 28, 25, 33, 27]. Methods such as HPSERec [29] and UnKD [2] focus on mitigating item-side popularity bias through stratified learning or unbiased distillation. Nevertheless, these methods exhibit two critical limitations: they primarily target item-side rather than user-side disparities, and they implicitly assume uniform teacher signal quality across all users. More fundamentally, their reliance on white-box access or explicit metadata renders them inapplicable under strict black-box constraints where only logits are available.

In contrast to prior work, we identify that the core bottleneck in black-box sequential recommendation lies in user-side signal heterogeneity—specifically, the dual degradation of teacher outputs into preference solidification for head users and information vacuum for tail users. BAHSD represents the first framework designed to address this problem under strict black-box constraints, achieving adaptive knowledge transfer through multi-scale consistency probing without requiring explicit user stratification or internal features.

Prediction performance analysis. Table 1 reports the performance of teachers at the user levels. On the dense MovieLens-1M dataset, we observe a striking performance inversion: tail users outperform head users, e.g., SASRec Recall@10: 0.875 vs. 0.713. This finding validates the phenomenon of preference solidification, in which Transformer-based teachers overfit to long head sequences, thereby compromising generalization. For the sparse Amazon Beauty dataset, head-tier performance exceeds that of the tail as expected, although tail signals remain noisy and exhibit low confidence. Information-theoretic analysis. Kernel Density Estimation (KDE) of the teacher logits shown in Fig. 2 reveals two distinct degradation modes. For tail users, a discriminative vacuum emerges where the probability mass concentrates near zero ($p<0.05$) with high entropy, indicating that signals are dominated by noise and rendering distribution matching ineffective. For head users, entropy-saturated solidification occurs where probability mass shifts non-linearly to high-confidence intervals ($p>0.15$). This low-entropy state reflects overconfidence and overfitting to local patterns.

If an oracle could perfectly assess signal reliability $\mathcal{R}(u)$, the optimal stratified loss would take the form:

where $\gamma_{1}$ and $\gamma_{2}$ denote user stratification thresholds. However, explicit stratification is infeasible under black-box constraints. Therefore, our core motivation is to implicitly approximate this oracle by designing a distillation framework that perceives signal reliability and adaptively adjusts its optimization strategies without requiring metadata or internal access. BAHSD achieves this through multi-scale sequence probing and adaptive loss synergy, directly addressing the identified degradation modes.

We define the set of users as $\mathcal{U}=\{u_{1},\dots,u_{M}\}$ and items as $\mathcal{I}=\{i_{1},\dots,i_{N}\}$. For a user $u\in\mathcal{U}$, the chronologically ordered interaction sequence is denoted as $S_{u}=[i_{1},\dots,i_{L}]\in\mathcal{I}^{L}$, where $L=|S_{u}|$ (sequences with $L<5$ are filtered).

• •

  Teacher Model ($\mathcal{T}$): A black-box API outputting logits $\mathbf{z}^{\mathcal{T}}(S)\in\mathbb{R}^{N}$. The predictive distribution is $\mathbf{p}^{\mathcal{T}}(S_{u})=\sigma(\mathbf{z}^{\mathcal{T}}(S_{u}))$, where $\sigma(\cdot)$ is the softmax function.

• •

  Student Model ($\mathcal{S}$): A lightweight homogeneous architecture. It outputs logits $\mathbf{z}^{\mathcal{S}}(S_{u})$ and hidden states $\mathbf{H}^{\mathcal{S}}(S_{u})\in\mathbb{R}^{L\times D}$.

• •

  Probing Views: $S_{p}$ denotes the multi-scale view $p\in\{short,mid,full\}$, e.g., corresponding to truncated prefixes (50%, 75%, 100%) of $S_{u}$.

• •

  Latent Representation: $\mathbf{h}_{p}^{\mathcal{S}}=\text{L2\_Norm}(\mathbf{H}^{\mathcal{S}}(S_{p}))\in\mathbb{R}^{D}$ represents the aggregated global representation of view $S_{p}$.

To extract knowledge at different granularities, we define a joint loss for each view $p$:

where $\mathcal{L}_{rank}$ denotes the pairwise Bayesian Personalized Ranking (BPR) loss and $\mathcal{L}_{KL}^{(p)}$ represents the temperature-scaled Kullback-Leibler divergence:

A large global temperature $\tau_{kd}$ enables adaptive gradient adjustment that responds to signal heterogeneity. For head users exhibiting preference solidification, $\tau_{kd}$ smooths extreme probability peaks to expose dark knowledge, allowing $\mathcal{L}_{KL}$ to dominate fine-grained knowledge transfer. Conversely, for tail users where distributions collapse, the temperature scaling drives both teacher and student distributions toward uniformity, which naturally attenuates the KL divergence gradient. In this regime, the model adaptively transitions to $\mathcal{L}_{rank}$-dominated optimization, focusing on robust ranking learning rather than matching uninformative distributions.

Empirical analysis reveals that tail users often exhibit deceptive false confidence due to sparse and homogeneous interactions. To filter such transient perturbations, we introduce a self-supervised symmetric KL divergence that constrains the student model’s output consistency across short-scale views:

where $\mathbf{p}^{\mathcal{S}}=\text{softmax}(\mathbf{z}^{\mathcal{S}}(S_{p})/\tau_{\text{kd}})$. Genuine user intent evolves smoothly over time, whereas pseudo-sharpness, a form of false confidence arising from sparse data, exhibits significant distributional shifts when the input sequence is truncated. When $\mathcal{L}_{inv}$ captures such divergence, it generates strong penalty gradients that guide the student model back toward a more robust representation space. Notably, this constraint excludes the full sequence $S_{\text{full}}$ to preserve its capacity for capturing recent concept drift while ensuring stable and reliable representations for historical intents. This design complements the other loss components by explicitly filtering noise at the distribution level, thereby reinforcing the framework’s overall robustness to signal heterogeneity.

To mitigate preference solidification in head users and enhance semantic discriminability for tail users, we introduce a latent-space contrastive module based on InfoNCE. Transformer-based models such as SASRec and BERT4Rec share a structural property where the sequence representation $\mathbf{h}_{\text{full}}^{\mathcal{S}}$ and the item embedding matrix $\mathbf{W}^{\mathcal{S}}$ lie in the same semantic manifold, enabling direct sequence-item alignment:

All components adhere to black-box constraints. The anchor $\mathbf{h}_{\text{full}}^{\mathcal{S}}$ is the L2-normalized final hidden state of the student for the full sequence. The positive sample $\mathbf{e}^{+}$ is obtained by extracting the top-1 item index $c_{1}$ from the teacher’s logits and mapping it to the student’s embedding matrix via $\mathbf{e}^{+}=\text{normalize}(\mathbf{W}^{\mathcal{S}}[c_{1}])$. Negative samples are uniformly drawn from the student’s embedding space, with $N_{neg}=256$.

The temperature parameter $\tau_{\text{cl}}$ controls geometric repulsion in the latent space with two complementary benefits. It mitigates head-user representation collapse by enlarging inter-item distances to alleviate preference solidification, and it addresses tail-user discriminative vacuum through negative sampling that establishes well-defined item boundaries.

The overall optimization objective for the student model is formulated as a weighted combination of the three proposed components:

where the hyper-parameters $\lambda_{A}$, $\lambda_{B}$, and $\lambda_{C}$ balance the contribution of each loss term, with dataset- and architecture-specific values provided in the experimental section.