JoyAI-Talker: Full-Duplex Speech Interactive Large Model Built for Empathetic Voice Agents

Initialized from the JoyAI-LLM Flash pretraining phase, JoyAI-Talker undergoes a unified, multi-stage speech-text joint training paradigm across all training phases. Our motivation for initiating joint multi-modal training at the mid-training phase—rather than delaying it until the SFT phase—is twofold. First, the early foundational pretraining stages establish the model’s fundamental cognitive baseline and world knowledge over massive textual corpora. Second, integrating the speech modality at this earlier stage promotes stable cross-modal alignment and prevents the catastrophic forgetting of textual knowledge, thereby improving multi-modal understanding without inducing textual degradation.

Consequently, we implement a unified speech-text mixed training scheme across all subsequent phases, sequentially optimizing the model through four distinct stages:

• •

  Speech-Text Joint Mid-training: This phase aims to bridge the representation space of the speech and textual modalities while preserving the foundational cognitive capabilities of the pre-trained language backbone. We introduce a highly diversified joint corpus. For the textual modality, this comprises web crawls, reasoning-intensive code repositories, high-fidelity PDF documents, and synthetic data to balance general knowledge with logical reasoning. For the speech modality, we incorporate a wide variety of tasks including automatic speech recognition (ASR), speech translation, speech question-answering (QA), audio captioning, real-world multi-turn speech dialogues, and speech text continuation (synthesized by feeding ASR transcriptions into an LLM to generate coherent textual responses paired with the original audio). This joint pretraining is split into two speech-text joint steps:

  • –

    Step 1 (Audio Adapter Warmup): Both the LLM backbone and the audio encoder are frozen, and we optimize only the multi-modal audio adapter to map high-dimensional speech features into the linguistic embedding space of the LLM.

  • –

    Step 2 (Full-Parameter Joint Training): We unfreeze all model parameters—including the LLM backbone, the speech encoder, and the audio adapter—for comprehensive end-to-end joint optimization.

  Throughout this phase, speech data is formatted in a chat-style Q&A structure, while textual data is ingested as raw, continuous text to preserve cognitive capacity.

• •

  Speech-Text Joint Context Extension: Following mid-training, this phase is designed to comprehensively bolster reasoning capabilities over extensive textual documents and enhance interactive proficiency during extended, multi-turn speech dialogues. All model parameters remain fully trainable. To target general long-context scaling, the textual mixture is rebalanced by upsampling long-context mathematical and coding datasets. For the speech modality, we upsample real-world multi-turn long speech dialogues and dynamically synthesize multi-turn ASR and translation tasks by concatenating single-turn samples. To emphasize long-range dependencies while preserving short-context proficiency, the joint data mixture blends long-form text (academic papers, financial reports, etc.) and long-duration audio (podcasts, lectures, etc.) with a balanced proportion of short-sequence data.

• •

  Speech-Text Joint Supervised Fine-Tuning (SFT): Serving as the cornerstone of our post-training alignment pipeline, this phase is strategically designed to expand the model’s multi-modal knowledge boundaries while cultivating highly colloquial, concise, and direct speech communication. The textual SFT dataset covers code, math, STEM, and general instruction-following, where we upsample knowledge-dense and interaction-centric categories to facilitate communicative voice dialogues. The speech SFT dataset covers ASR, translation, QA, speech text continuation, audio captioning, and empathetic speech dialogues. To facilitate cross-modal alignment and elevate speech response quality, we upsample synthesized speech-to-text (S2T) data constructed by converting conversational textual SFT prompts into natural speech audio via our in-house text-to-speech model.

• •

  Speech-Text Joint Direct Preference Optimization (DPO): To further refine response quality, improve instruction-following, and suppress residual hallucinations, we implement a dedicated DPO phase. The model is optimized directly on pairwise preferred and dis-preferred responses self-generated by the SFT policy. For each seed prompt (spanning general QA, reasoning, and speech inputs), the SFT policy generates multiple candidate responses, from which preferred and dis-preferred pairs are selected using a multi-faceted model judging framework. Candidates are routed to three complementary, specialized judges:

  • –

    General QA Judge: An LLM judge evaluates mathematics, code, STEM, and general knowledge responses along correctness, logical soundness, clarity, and completeness, using reference answers as quality anchors.

  • –

    Instruction-Following (IF) Judge: For prompts with explicit structural constraints (exact item counts, mandatory keywords, length limits, JSON output), we prompt an LLM judge with a constraint-compliance rubric. This rubric allocates most points to hard-constraint satisfaction (with fixed penalties per violation type) and the rest to content quality and output cleanliness, thereby raising the candidate scoring margin on constraint-violating cases.

  • –

    Rule-Based IF Judge: For machine-verifiable constraints, we score candidates using deterministic, rule-based validations, ensuring full reproducibility and zero inference cost.

  For speech inputs, each prompt carries an audio field while the candidate responses remain textual, allowing us to leverage the same QA and IF judges on the transcribed text. Crucially, rather than pooling speech-conditioned and text-conditioned candidates cross-modally—which would create severe distribution mismatches and an excessively large optimization gap due to the systematic modality gap—we adopt a speech-only strategy. This strategy constructs preference pairs strictly within the speech-conditioned pool. This local comparison provides a highly calibrated, realistic, and learnable gradient signal, enabling the model to steadily correct its speech-pathway behavior. The resulting textual and speech preference sets are then mixed for joint DPO training.

Our audio-language model training system is built on a highly optimized extension of the Megatron-Core framework. Alongside Data Parallelism (DP), Tensor Parallelism (TP), Sequence Parallelism (SP), and Pipeline Parallelism (PP), we employ 8-way Expert Parallelism (EP) to scale the sparse Mixture-of-Experts backbone. MoE operations are accelerated through DeepEP for low-latency token dispatch and combination, grouped GEMM, and fused permutation and routing. CUDA Graph capture further reduces CPU launch overhead and stabilizes step latency.

The training sequence length is configured at 8K tokens during the Mid-training Phase, and is subsequently scaled to 64K tokens during the Context Extension and SFT Phases. For the 64K-token training stages, we adopt Context Parallelism (CP) with THD-format attention to distribute the computational load. Under this long-context regime, we employ a best-fit sequence packing strategy utilizing block-diagonal attention masks to concatenate multiple independent samples or speech dialogue sessions into a single training sequence. This is powered by an audio-aware packing algorithm that jointly considers token length and audio-padding waste, preventing cross-contamination between unrelated samples while approximately doubling end-to-end throughput over standard length-only packing. We also distribute the audio encoder across CP ranks: each rank processes its local audio slice, and a differentiable all-gather reconstructs the embeddings while preserving gradient flow. This reduces per-rank encoding cost in proportion to the CP degree and avoids an encoder bottleneck in long-context multimodal training.

Across all joint training phases, we implement a strict loss masking strategy to optimize representation learning for target generations. For both textual corpora and multi-turn conversational datasets, the autoregressive loss is computed solely on the Assistant’s responses. The loss on all other components is completely masked out; this includes all user-side inputs—encompassing both textual and speech queries—as well as all tool-side responses (such as tool execution results and API replies), on which no loss is computed. This selective gradient optimization ensures that the model’s representation learning capacity is focused exclusively on generating accurate, coherent, and concise assistant responses.

Large Speech Language Models (LSLMs) have substantially advanced speech language understanding, driving applications ranging from intelligent assistants to empathetic companions [10]. However, effective dialogue extends beyond merely interpreting what is said, it necessitates recognizing speaker attributes (e.g., age and gender) and discerning how the message is delivered [11, 12]. Non-verbal acoustic cues-such as emotional prosody and paralinguistic behaviors (e.g., laughter, sigh, and cough)-are essential for capturing these nuances, facilitating natural, trustworthy, and emotionally intelligent communication. Despite their importance, existing approaches typically model these elements in isolation, failing to reflect the tightly coupled nature of understanding, reasoning, and response generation inherent in human conversation.

To address this gap, we introduce Persona-Adaptive Empathetic Response (PAER) to deliver highly customized, multifaceted empathic interactions that dynamically adapt to individual users. This framework is explicitly structured as a hierarchical cognitive pipeline: audio understanding $\rightarrow$ chain-of-thought (CoT) reasoning $\rightarrow$ empathetic dialogue generation. Our main contributions are twofold:

• •

  A Progressive Cognitive Empathy Framework: We design a progressive task formulation that mirrors human cognitive processes. Initially, we employ foundational perception tasks to accurately extract speaker attributes (e.g., gender, age, and emotional state) from the input audio. Thinker module then explicitly integrates these attributes into its reasoning process via CoT. This facilitates the generation of context-adaptive responses that contain not only contextually appropriate text but also precise controls over utterance-level expression and localized paralinguistic events (e.g., sighing, slow speaking rate, and low volume), without altering the assistant’s speaker identity. Subsequently, the Talker module, following the JoyVoice speech generation framework [8] and refined through instruction-aware supervised fine-tuning (SFT), translates the Thinker’s textual and control outputs into highly expressive and natural speech.

• •

  Comprehensive Evaluation and Superior Performance: We rigorously evaluate our model across foundational audio understanding and downstream empathy generation tasks. First, we validate its precise perception of speaker attributes on public benchmarks, achieving outstanding performance on AIR-Bench [13] (for age and gender) and MER2025 [14] (for emotion). These robust perceptual capabilities lay a solid foundation for individual-specific empathy. Building upon this, we comprehensively assess end-to-end empathic expressiveness using the open-source EchoMind dataset [15]. Extensive experiments demonstrate that our approach consistently achieves superior performance in both textual and acoustic empathy metrics.

Following the large-scale speech data construction recipe of JoyVoice [8], the Talker is trained on a multilingual corpus containing both read and in-the-wild speech. The corpus covers diverse speakers, languages, recording conditions, speaking styles, and conversational scenarios, providing broad support for robust pronunciation, natural prosody, and generalizable speech generation.

To construct instruction-aware training data, we further collect expressive speech from movies, television programs, and other conversational media. Compared with conventional read speech, these sources contain richer variations in emotion, speaking rate, vocal effort, conversational rhythm, phonation, and paralinguistic behavior. Multimodal contextual cues from audio and video are used to derive natural-language descriptions of speaker characteristics and utterance-level acoustic realization.

The expressive corpus is combined with general speech data during foundation training. General speech preserves linguistic coverage, pronunciation accuracy, and default synthesis quality, while expressive speech expands the controllable distribution of speaking styles and paralinguistic behaviors.

The instruction-controllable foundation model uses a hierarchical representation consisting of global and local instructions. The global instruction describes relatively stable speaker-level characteristics, such as role description, gender, age, accent, and underlying timbre. The local instruction specifies the acoustic realization of the current utterance, including its scene, temporary timbre variation, primary emotion, speaking rate and rhythm, breathiness, and spatial impression.

This separation distinguishes persistent speaker characteristics from utterance-dependent expression. During speaker-specific SFT, the global characteristics are absorbed into the fixed target-speaker model, while the local instruction remains available as the dynamic control interface. Therefore, the deployed Talker does not require a global instruction at inference time.

Following the unified autoregressive formulation of JoyVoice [8], we extend the input sequence with hierarchical instruction conditions:

where

Here, $P_{\mathrm{sys}}$ defines the speech generation task, $I^{g}$ provides persistent speaker-level conditioning, $\mathrm{spk}$ denotes the speaker embedding, and $I^{l}$ specifies the local expressive characteristics of the current utterance. The sequences $\{t_{n}\}_{n=1}^{N}$ and $\{s_{m}\}_{m=1}^{M}$ denote the input text tokens and target speech tokens, respectively.

Placing $I^{l}$ immediately before the corresponding text establishes an explicit association among speaker identity, utterance-level expression, and linguistic content. Both $I^{g}$ and $I^{l}$ are represented as natural-language token sequences, enabling open-ended and compositional control through the existing language-modeling interface. As in JoyVoice, the autoregressive hidden representations associated with $S$ are further used to condition the acoustic generation module.

Paralinguistic behavior is an important component of natural speech interaction. Vocal events can communicate emotion, attitude, hesitation, physical state, and conversational intent beyond the lexical meaning of the text.

We represent ten types of paralinguistic units using explicit special tokens:

These symbols are added to the model vocabulary and treated as special tokens rather than ordinary textual descriptions. They can be inserted directly into the input text at the desired positions, enabling the model to jointly learn their semantic context, temporal placement, and acoustic realization.

For example:

[Sigh]I understand. Let us try another solution.

I did not expect that at all.[Laughter]That was actually quite funny.

This representation provides an explicit interface for controlling localized non-lexical vocal events while preserving the surrounding linguistic content. Local instructions govern the overall expressive realization of an utterance, whereas paralinguistic tokens provide position-specific control over localized non-lexical vocal events. These two interfaces are complementary.

Collecting expressive target-speaker recordings at scale is expensive. Studio recordings usually provide clean and consistent timbre, but cover only a limited range of emotional states, prosodic patterns, speaking rates, and paralinguistic behaviors. Exhaustively recording all combinations of these attributes is impractical.

We address this problem through a voice-conversion-based SFT data construction strategy. The central idea is to borrow performances rather than voices.

Movies and conversational media provide a large collection of naturally occurring performances, including excitement, hesitation, anger, relief, whispering, shouting, rapid speech, restrained emotion, and various paralinguistic behaviors. We use voice conversion to transfer these expressive performances into the target assistant voice.

Given an expressive source utterance $x^{s}$, its speaking instruction $I$, and a target-speaker reference $r^{t}$, the converted speech can be represented as

where $\widetilde{x}^{t}$ is expected to retain the major expressive characteristics of the source utterance while transferring its speaker identity toward the target voice.

The resulting training sample is formulated as

where $I^{l}$ describes the expressive realization of the converted utterance and $T$ denotes its linguistic content. In this way, diverse performances from different speakers are reinterpreted through a single, consistent assistant voice.

The converted samples constitute the instruction-aware SFT corpus for the target assistant voice. Quality control is applied to preserve transcription consistency, target-speaker similarity, expressive relevance, and general audio quality. In this way, diverse performances produced by different source speakers are reinterpreted through a consistent assistant identity.

This strategy enables low-cost expressive adaptation without requiring the target speaker to record every combination of emotion, speaking rate, vocal effort, phonation, and paralinguistic behavior. The resulting speaker-specific SFT model serves as the Talker in JoyAI-Talker, preserving a fixed and recognizable assistant identity while retaining the local instruction-following capabilities acquired during foundation training.

The current Talker uses open-ended natural-language instructions rather than a closed set of mandatory categorical labels. The instructions may describe one or multiple acoustic attributes with different levels of detail.

The supported instruction dimensions mainly include:

• •

  Emotion and Affective Expression: Descriptions of primary emotion, emotional intensity, emotional nuance, attitude, and communicative intention, such as happy, sad, angry, excited, surprised, fearful, and so on.

• •

  Speaking Rate and Rhythm: Descriptions of speaking rate and rhythmic characteristics, such as extremely fast, fast, normal, slow, extremely slow, and so on.

• •

  Loudness and Vocal Effort: Descriptions of loudness and vocal effort, such as shouting, loud, normal, soft, whispering, and so on.

• •

  Timbre and Phonation: Descriptions of temporary timbre characteristics and phonation styles, such as clear, bright, breathy, hoarse, tense, relaxed, and so on.

• •

  Paralinguistic Behavior: Control of the ten types of paralinguistic units introduced above.

• •

  Compositional Expression: Natural-language descriptions combining multiple acoustic attributes, such as emotion, speaking rate, loudness, timbre, scene, paralinguistic behavior, and so on.

Representing instructions in natural language provides a more flexible control interface than fixed categorical labels. It allows expressive attributes to be described with varied wording and enables multiple control dimensions to be combined within a single instruction.

The Talker supports streaming text input and streaming speech output. As the Thinker incrementally produces a textual response, the Talker can begin synthesizing the available content without waiting for the complete response.

During streaming inference, the local instruction is provided before its associated text and is preserved as a complete conditioning unit. This is particularly important for natural-language instructions and paralinguistic special tokens, whose meanings may be weakened if they are separated from the corresponding text span.

The streaming interface dynamically organizes incoming text into generation units while maintaining the association among local instructions, paralinguistic events, and response content. This enables low-latency speech generation while preserving the expressive intention of the response.

Because the deployed Talker uses a fixed target-speaker voice, no global speaker instruction is required during inference. The Thinker produces the textual response together with an optional local speaking instruction derived from its empathetic understanding of the user, dialogue context, and intended communicative behavior. The instruction is enclosed by <instruct> and </instruct> and is placed before the corresponding response text.

When an explicit instruction is provided, the Talker uses it to control the utterance-level acoustic realization while preserving the target assistant identity. When no instruction is provided, the Talker infers an appropriate speaking style from the linguistic and semantic context. Position-specific paralinguistic tokens may additionally be inserted into the response text to control localized vocal events.

This interface separates empathetic reasoning from acoustic realization. The Thinker determines response content and communicative intent, while the Talker specializes in speaker-consistent, expressive, and low-latency speech generation.

The Joy-Duplex framework isolates real-time turn-taking control from core response generation, deploying a lightweight external duplex predictor to govern the downstream half-duplex SDM. As illustrated in Fig. 3, the architecture consists of a streaming Audio Encoder and an interleaved Joy-Duplex Decoder:

• •

  Streaming Audio Encoder: To process continuous user speech with low algorithmic latency, we employ a streaming speech encoder that slices the input waveform into non-overlapping 160 ms chunks during inference. To enhance the model’s robustness and generalization across various latency requirements, the encoder is trained using a dynamic chunk-based attention strategy [19], where the chunk size is dynamically randomized during the training phase. The attention receptive field of each frame is constrained to only look backward at historical context and within the current chunk. The resulting continuous Audio Embeddings are then aligned via a linear projector and directly fed into the decoder.

• •

  Joy-Duplex Decoder: The core backbone is a 1.7B-parameter decoder-only language model. The Joy-Duplex Decoder accepts the continuous Audio Embeddings alongside historical context, and autoregressively decodes interleaved Text Tokens and State Tokens within a single unified sequence. Notably, to establish explicit structural boundaries between speech transcripts and interaction states, each predicted text segment is strictly terminated with a dedicated <|asr_eos|> token (represented as blue circles in Fig. 3). This boundary token explicitly marks the completion of incremental transcription for the current frame, immediately after which the decoder predicts the corresponding interaction State Tokens.

We formulate streaming duplex state control as a token-interleaved sequence modeling task under incrementally available acoustic observations. Let $A_{t}$ denote the continuous Audio Embeddings extracted from the $t$-th chunk, $T_{t}=\{w_{1},w_{2},\dots,\texttt{<|asr\_eos|>\lx@text@rbrace}$ represent the corresponding streaming Text Tokens terminated by the <|asr_eos|> token, and $S_{t}$ denote the predicted State Tokens. At each step $t$, conditioned on the historical context $\mathcal{H}_{t-1}=\{A_{<t},T_{<t},S_{<t}\}$, the joint probability of predicting the current frame’s text and state sequence is factorized as:

As illustrated in Fig. 3, when the user is speaking, the decoder first predicts the aligned Text Tokens $T_{t}$ and closes the text segment with the <|asr_eos|> token (e.g., “What” $\rightarrow\texttt{<|asr\_eos|>}$), followed immediately by the State Token $S_{t}$ (e.g., <|partial|>) to capture instantaneous semantic and interaction status. Even during speech pauses or acoustic non-speech chunks where no text tokens are generated, the decoder outputs $\texttt{<|asr\_eos|>}\rightarrow\texttt{<|partial|>}$ to securely hold the turn. Only when a complete syntactic boundary is established (e.g., “What is your name?”), the sequence generates $\text{name}\rightarrow\texttt{<|asr\_eos|>}\rightarrow\texttt{<|complete|>}\rightarrow\texttt{<|accept|>}$, unblocking the semantic gate to trigger the downstream response.

The downstream SDM remains in a gated standby mode, activated for response generation if and only if the Joy-Duplex Decoder emits an explicit <|accept|> token. To enhance deployment safety in multi-speaker or noisy environments, a lightweight Speaker Verification (SV) plugin can run in parallel alongside the encoder for identity matching. This ensures that the SDM is exclusively triggered by the authorized speaker, forming a secure, modular, and robust full-duplex pipeline.

Rather than relying on frame-level acoustic energy which often causes false interruptions during natural hesitations, Joy-Duplex introduces five dedicated state tokens directly integrated into the decoding stream to establish a semantically-informed interaction state machine:

• •

  <|partial|>: Continuously generated during ongoing active speech, signaling that the user’s semantic intent is still developing. This instructs the downstream SDM to remain in standby mode.

• •

  <|complete|>: Triggered immediately when the incremental transcription reaches a coherent syntactic and semantic boundary (e.g., transcribing the end of “…your name” $\rightarrow$ <|complete|>). This serves as a precursor for the final interactive gating decision.

• •

  <|backchannel|>: Emitted when the user provides short verbal feedback (e.g., “uh-huh”, “yeah”). It allows the system to acknowledge the user’s presence without initiating a full conversational turn.

• •

  <|accept|> / <|reject|>: The semantic rejection gate. Immediately following the <|complete|> token, the decoder evaluates the final interaction intent. If a valid, system-directed query is confirmed, it yields <|accept|>, which unblocks the gate and triggers the downstream SDM to speak. Conversely, if the completed segment consists of background leakage, off-target speech, or ambient noise, the decoder outputs <|reject|>, which explicitly suppresses the SDM to prevent accidental activations.

To establish a foundation for personalized empathy, we evaluate the model’s capability in perceiving speaker attributes from raw audio.

• •

  Acoustic Perception Evaluation: We assess gender and age recognition on AIR-Bench, and emotion recognition on MER2025. To capture emotional nuances more precisely, we audited the test set labels for Track 1, transitioning from six discrete emotional categories to a richer "primary emotion + mental state description" format.

• •

  Evaluation Settings: We evaluate the model under two distinct settings: (1) Direct, which measures atomic perception capabilities in isolation, and (2) CoT, which evaluates the attributes explicitly generated within the model’s intermediate Chain-of-Thought reasoning steps during actual dialogue generation.

• •

  Perception Results Analysis: As shown in Table 3, JoyAI-Talker displays solid and stable capabilities:

  • –

    Under the Direct setting, the model demonstrates stable baseline understanding, reaching $98.2\%$ in gender, $77.3\%$ in age, and $79.6\%$ in emotion recognition.

  • –

    Under the CoT setting, the model successfully retains and utilizes its acoustic perception within a continuous conversational reasoning process, achieving a competitive $77.7\%$ accuracy in age prediction.

We assess the final generated responses to evaluate the model’s empathetic and persona-adaptive response quality, with the acoustic realization evaluated separately.

• •

  Evaluation Setup: We evaluate the final responses using three English subsets derived from EchoMind (Emotion, Age, and Gender). An LLM-as-a-judge paradigm is employed, rating responses on a 1–10 scale across six weighted dimensions: Basic Logic ($10\%$), Knowledge Consistency ($10\%$), Diversity ($10\%$), Content Logic ($20\%$), Interaction ($20\%$), and Empathy ($30\%$).

• •

  Performance Analysis: As detailed in Table 4, the results highlight the progressive alignment of our framework:

  • –

    JoyAI-Talker achieves competitive results across both the overall weighted scores and the dedicated empathy dimensions.

  • –

    Favorable gains are observed on the EchoMind-Emotion subset. On EchoMind-Age, while the overall weighted score shows a slight increase, the dedicated empathy score displays a notable improvement (climbing to $7.49$).

  • –

    These targeted gains demonstrate that our decoupled framework effectively concentrates modeling capacity on deep persona adaptation and emotional resonance by seamlessly integrating audio understanding, explicit persona-driven reasoning, and empathetic response generation.

To verify the efficacy of our proposed framework, we conduct extensive evaluations using the latest established full-duplex conversational standard. We evaluate Joy-Duplex on Full-Duplex-Bench v1.5 [4], a benchmark designed to assess an audio assistant’s categorical behaviors under four typical conversational overlap scenarios:

• •

  User Interruption: The user genuinely attempts to take over the floor (e.g., “Wait, I want to…”). The system must yield the turn immediately and address the new query ($C_{\text{RESPOND}}\uparrow$).

• •

  User Backchannel: The user injects short affirmations (e.g., “uh-huh”, “yeah”) to signal engagement rather than taking the turn. The system should ignore these cues and continue speaking ($C_{\text{RESUME}}\uparrow$).

• •

  Talking to Other: The user addresses a third party in the environment (e.g., “Hey Grandma…”). The system must perform semantic addressee detection and hold the floor ($C_{\text{RESUME}}\uparrow$, $C_{\text{RESPOND}}\downarrow$).

• •

  Background Speech: Ambient far-field speech or environmental noise occurs. The system must remain robust to this acoustic interference and continue its response ($C_{\text{RESUME}}\uparrow$, $C_{\text{RESPOND}}\downarrow$).

Following the benchmark protocol, all system behavioral outputs are classified into four categories: $C_{\text{RESPOND}}$ (yielding and responding), $C_{\text{RESUME}}$ (ignoring and continuing), $C_{\text{UNCERTAIN}}$ (addressee checking), and $C_{\text{UNKNOWN}}$ (unclassified behaviors).

The quantitative results on Full-Duplex-Bench v1.5 are summarized in Table 5. To ensure a state-of-the-art evaluation, in addition to citing baselines from the original benchmark paper [4], we leverage the commercial API to locally evaluate and include performance metrics for the released Gemini 3.1 Live model.

As presented in Table 5, Joy-Duplex achieves state-of-the-art performance across all four conversational scenarios, demonstrating superior capabilities in both turn-yielding and turn-holding decisions. Driven by our chunk-masked streaming encoder and explicit state transition gating, the system achieves a strong interruption response rate and suppresses erroneous floor-holding. The remaining unresponded interruptions are primarily attributed to a conservative trade-off introduced by our semantic rejection tokens (<|reject|>), which prioritize preventing false activations under ambiguous acoustic conditions.

When encountering non-floor-taking inputs, this conservative gating policy exhibits notable advantages by maintaining appropriate turn-holding behaviors. When users address third parties, Joy-Duplex reduces false responses to 0.17 while maintaining a 0.72 resume rate. Under background speech interference, false activations are suppressed to 0.10 with an 0.85 resume rate. Furthermore, the system seamlessly maintains conversational flow during user backchannels, achieving a high resume rate of 0.96 with a minimal false trigger rate of 0.01. Overall, these evaluations confirm that Joy-Duplex delivers a well-balanced interaction policy by effectively combining prompt turn-yielding with robust semantic rejection.