PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generation

Given an input video and a natural language text $t$, we leverage the language-aligned discrete representation produced by PyraTok to obtain zero-shot, text-guided spatio-temporal masks. Specifically, we first pass the video through the frozen PyraTok encoder and its Language-aligned Pyramidal Quantization (LaPQ) hierarchy and extract the quantized features from the last quantization block, denoted by $\mathbf{q}^{(L)}\in\mathbb{R}^{T^{\prime}\times H^{\prime}\times W^{\prime}\times d}$, which capture high-level, text-aligned semantics at a compressed spatio-temporal resolution. We then decompose the input text into a set of semantic units (typically content words or short phrases), $\{w_{1},\dots,w_{K}\}$, and obtain a language embedding $\mathbf{e}_{w_{k}}$ for each unit using the same vision–language model employed during PyraTok training. For every semantic unit $w_{k}$, we compute a similarity score between $\mathbf{e}_{w_{k}}$ and each token in $\mathbf{q}^{(L)}$ (e.g., via cosine similarity in the shared embedding space), yielding a token-level relevance map $\mathbf{S}_{w_{k}}^{\text{tok}}(t^{\prime},h^{\prime},w^{\prime})$. This relevance map is then upsampled to the original video resolution, following the encoder downsampling pattern (or via decoder-aligned projection), to produce a dense per-pixel score volume $\mathbf{S}_{w_{k}}(x,y,t)$ for each semantic unit. We treat these volumes as unary potentials in a fully connected 3D Conditional Random Field (CRF) defined over the spatio-temporal lattice $(x,y,t)$, with pairwise terms encouraging spatial smoothness aligned to image edges and temporal consistency across adjacent frames. Running mean-field inference in this 3D-CRF refines the raw scores into a binary segmentation mask $\mathbf{M}_{w_{k}}(x,y,t)\in\{0,1\}$ that assigns each pixel in each frame to semantic unit $w_{k}$. Repeating this procedure iteratively over all semantic units in the prompt yields a set of word-level, zero-shot, text-guided segmentation masks that are both spatially precise and temporally coherent across the video.

We compare our language-guided tokenizer with OmniTokenizer [57], LARP [55], and VideoVAE+ [63], on diverse scenes in Fig. 10 and novel-category examples in Fig. 11. Existing tokenizers often yield coarse, blob-like masks with strong label confusion: OmniTokenizer and LARP tend to over-smooth object boundaries and merge adjacent instances (e.g., bus and road, trees and background), while VideoVAE+ frequently misses thin structures such as bike frames, surfboards, and traffic signs, or hallucinates spurious regions in uniform areas. These methods also struggle with rare or fine-grained concepts, leading to incomplete segmentation of small objects (e.g., cat ears, surfboard tips) and inconsistent labeling across the image. In contrast, PyraTok produces masks that are both sharper and more semantically aligned with the ground truth, accurately separating foreground from background and preserving thin structures. Fig. 11 further demonstrates strong zero-shot generalization: PyraTok cleanly segments unseen categories such as millennium falcon, tordelli, golden retriever, Pikachu, and axolotl, and simultaneously grounds multiple text queries (e.g., “golden retriever / puppy / grass field / vegetation”) in the correct regions, highlighting that our language-aligned tokens carry richer semantic information than prior VAE-based tokenizers.

For all question answering results, we adopt Qwen2.5-VL-3B [1] as the default vision–language (VLM) backbone to generate open-ended answers from our video representations. Given an input clip, we first encode the video with our proposed PyraTok VAE and extract the discrete representations from all quantization blocks. These multi-scale features are projected into the language embedding space and prepended to the question tokens, yielding a unified conditioning sequence for the autoregressive decoder. The Qwen2.5-VL-3B model then performs conditional text generation to produce the final answer. All VQA inferences are executed using the Text Generation Inference (TGI) pipeline from HuggingFace [17], which provides a stable and reproducible deployment for our qualitative analysis.

Furthermore, across Fig. 12 to Fig. 14, we compare PyraTok against Qwen2.5-3B, VideoVAE+, OmniTokenizer, and LARP on diverse video scenarios, including action sequences (helicopter crash, motorcycle chase, aircraft destruction), transformation events (monster emergence, firetruck-to-robot), and emotional interactions (a surprise proposal). The lower-capacity baselines (Qwen2.5-3B and VideoVAE+) often produce vague or partially incorrect explanations, while OmniTokenizer and LARP capture events more reliably but still miss finer details. PyraTok consistently provides the most accurate, complete, and context-aware interpretations across all scenarios, demonstrating stronger temporal reasoning, causal understanding, and fine-grained visual grounding compared to competing models.

We tackle temporal action localization in long, untrimmed videos by directly operating in the discrete latent space of PyraTok. Given a video of $N$ RGB frames and a textual description of the target action, we first encode every frame with our pyramidal tokenizer. Empirically, we observe that $\mathbf{q}^{(1)}$ offers the best trade-off between semantic expressiveness and temporal resolution: it preserves subtle motion cues (e.g., arm swing before an arrow release, the instant of impact in a punch, see Fig. 15) that are strongly smoothed out in deeper levels. We therefore use $q^{(1)}$ as our frame-level features. For each frame $t$, we spatially pool the tokens $\mathbf{q}^{(1)}_{t}$ (mean-pooling over space) to obtain a compact frame descriptor $\mathbf{v}_{t}\in\mathbb{R}^{d}$. The textual query is encoded by the same language backbone used for PyraTok’s cross-modal training, producing a normalized embedding $\mathbf{z}\in\mathbb{R}^{d}$. We compute cosine similarity scores $s_{t}\!=\!\langle\mathbf{v}_{t},\mathbf{z}\rangle$ for all frames, which yield a dense text–video alignment signal over time.

To robustly localize an action interval, we evaluate similarities in a sliding-window fashion. The video is partitioned into overlapping chunks $(t,t+K-1)$ of length $K{=}25$ frames (with stride 1 in all experiments). For each chunk we aggregate the frame scores, $S_{t}\!=\!\frac{1}{K}\sum_{i=t}^{t+K-1}s_{i}$, resulting in a 1D confidence trajectory $\{S_{t}\}_{t=1}^{N-K+1}$ that reflects how strongly the query is grounded in each temporal neighborhood. We then decode this trajectory into contiguous segments using a longest-connected-sequence algorithm: (i) we threshold $S_{t}$ at a fixed confidence $\tau$ to obtain a binary sequence; (ii) identify all maximally connected high-confidence segments; and (iii) select the segment with the highest average score as the predicted action interval. For multi-action queries, we iteratively remove the selected interval and repeat, merging overlapping segments when necessary. The resulting segments define our temporal action predictions.

Fig. 15 and Fig. 16 visualize localized action segments for different tokenizers on several challenging examples. For each text query, the ground-truth (GT) segment is shown in blue, and the predictions obtained from VideoVAE [63],+, SweetTok [46], LARP [55], and PyraTok are displayed as colored bars beneath. The baselines consistently exhibit temporally diffuse and fragmented activations: their similarity signals tend to fire on visually similar but semantically off-target frames, producing multiple short segments or systematically shifted intervals. For instance, in Fig. 15, in the clip “A girl shoots an arrow”, both VideoVAE + and SweetTok activate broadly over the whole sequence and fail to concentrate probability on the actual release moment, while LARP on several disjoint intervals before and after the shot. In contrast, PyraTok yields a single, compact segment that tightly aligns with the GT span around the arrow release. A similar pattern appears for text query “A person fires a shotgun”, where baseline tokenizers localize earlier or later segments, whereas PyraTok localizes correctly.

The advantages of our fine-grained features are even more evident for actions with multiple sub-events. In Fig. 15 example “An MMA fighter knocks down his opponent with a punch to the face” and “…with a kick to the face”, the motion unfolds rapidly and is preceded by visually similar feints. VideoVAE + and SweetTok tend to spread confidence over the entire exchange, leading to overly long or misaligned segments, while LARP often localizes only part of the motion (e.g., the wind-up but not the impact). PyraTok, by contrast, localizes a concise window centered around the decisive contact, closely matching the GT. In Fig. 16, for “A person performs two overhead presses”, PyraTok produces two high-confidence video segments that track both overhead press repetitions, whereas baselines either miss the second repetition or merge the two into one coarse interval. For complex, extended actions such as “A man and a woman engage in sword fighting” and “Three missiles are launched from a desert”, baseline tokenizers again show scattered activations, localizing short segments around high-motion frames or transient explosions, and resulting in under-coverage of the GT. In contrast, PyraTok yields more accurate localization.

To assess the usefulness of our tokens for generative modeling, we couple PyraTok with a conditional video decoder built on Qwen-2.5VL [1]. Concretely, we treat the text encoder of Qwen-2.5VL as a frozen condition network and fine-tune its video decoder to autoregressively predict PyraTok codes. Given a textual prompt, we first encode the prompt into language features, which are injected into a transformer-based decoder that models the joint distribution over all spatio–temporal tokens from our four quantizers. The decoder predicts the next token conditioned on the text and all previously generated tokens, until a full sequence of discrete video codes is obtained. These codes are then passed through the PyraTok VAE decoder to synthesize the final video. Thanks to PyraTok’s compact yet expressive representation, this pipeline can generate videos at 20 FPS with resolutions up to 4K.

Fig. 17 and Fig. 18 show qualitative comparisons on text-to-video generation where we keep the Qwen-2.5VL decoder architecture fixed and only swap the underlying tokenizer. OmniTokenizer and LARP tend to under-utilize fine-grained textual cues, often missing localized attributes such as the “two egg halves” in the ramen bowl or the “motion blur on pedestrians” in the neon street scene, and producing over-smoothed or distorted structures in complex compositions like the tree city and Mars spaceport. SweetTok better preserves global layout but still struggles with high-frequency details and subtle style descriptors (e.g., HDR interior lighting, crisp spray around the polar bear), leading to muted textures and inconsistent object shapes.

In contrast, PyraTok yields samples that more faithfully reflect both global scene descriptions and fine-grained phrases in the prompts. The additional objects specified in the text appear at the correct locations, motion-related cues are rendered more plausibly, and material and lighting properties (glossy chocolate surface, bioluminescent foliage, cinematic city glow) are captured with higher fidelity. Fig. 19 further illustrates 4K text-to-video generation for a 3-second clip, where PyraTok renders fine-grained details and maintains sharp structures, demonstrating that our multi-scale quantization supports high-resolution, text-aligned video synthesis.

We further evaluate PyraTok on 4K frame reconstruction in Fig. 20. At this resolution, prior tokenizers struggle to preserve fine structures and high-frequency textures. VideoVAE+ [63] produces strong over-smoothing: the coral branches, tree leaves, and fur on the buffalo become noticeably blurred, and small objects such as distant boats and fire lamps nearly vanish in the zoomed-in crops. OmniTokenizer [57] improves sharpness but introduces ringing and aliasing along high-contrast boundaries (e.g., the product watch edges and mountain silhouettes), and often exhibits color bleeding in specular regions. SweetTok [46] and LARP [55] retain more detail yet still suffer from blocky artifacts on repetitive textures (grass, foliage, brick walls) and inconsistent reconstruction of tiny highlights, such as reflections on the watch bezel and lights on the night harbor. In contrast, our PyraTok reconstructions remain consistently crisp and coherent. Objects across all scenes—from coral polyps and reef fish to product shots and distant architectural details—retain sharp contours and clean textures without haloing. Fine-grained elements such as fur strands, leaf veins, and small fruits are faithfully preserved, demonstrating that our pyramidal tokenization scales effectively to ultra-high resolutions while avoiding the blurring and aliasing present in prior methods.

In qualitative video reconstruction comparisons (Figs. 21–24), existing tokenizers show consistent limitations across diverse scenes. TokenFlow [37] and SweetTok often oversmooth high-frequency content, causing foliage, clothing textures, and facial details to blur, and small or thin structures to distort or disappear; they also introduce blocky artifacts under large motion. LARP better preserves local contrast but frequently produces ringing around boundaries and unstable illumination, leading to flickering highlights and shadows. MotionAura [44] improves temporal smoothness yet still suffers from identity drift in talking-head sequences and ghosting around fast movements, reducing perceptual realism. Moreover, as previous methods were trained on low-resolution data, their high-resolution reconstructions exhibit temporal artifacts such as frame stuttering. In contrast, our 4K-trained PyraTok preserves high-frequency detail and temporal coherence, producing smooth and stable video.

We further study whether PyraTok can serve as a drop-in tokenizer for existing text-to-video priors. To this end, we replace the original VAE/tokenizer in three pretrained models, i.e., Open source version of MAGVIT-V2 [33, 70] and OmniGenV2 [60] (autoregressive priors) and MotionAura [44] (discrete diffusion prior), and fine-tune only the prior on 10k clips from OpenVid-1M [35] so that it models PyraTok codes. Under identical prompts and sampling hyper-parameters, and across all architectures, using the native tokenizer leads to typical failure modes: colors and exposure drift over time, geometry “breathes” (e.g., wobbling backgrounds and object contours), high-frequency details such as dough surface texture or water droplets quickly collapse into smooth blobs, and object semantics are weakly preserved (e.g., inconsistent shape of the claw-machine robot or citrus slices). After swapping in PyraTok, the same priors produce videos that are both more semantically aligned with the prompts and markedly more temporally consistent.

In Figure 25, MAGVITv2+PyraTok maintains stable neon lighting in the arcade, preserves the dough’s volume and hand pose across frames, and keeps the boiling dumplings sharp with coherent bubble motion. In Figure 26, OmniGenV2+PyraTok yields crisp tree trunks and facial details with reduced frame-to-frame jitter, while the splashing juice exhibits smoother, physically plausible trajectories.

Similarly, MotionAura+PyraTok in Figure 27 suppresses diffusion-induced flicker in backgrounds. These improvements indicate that PyraTok’s multi-scale discrete representation reduces quantization artifacts and exposes a more structured latent space, making it easier for both autoregressive and diffusion priors to model long-range spatio-temporal dependencies and maintain object identity over time, even with minimal fine-tuning data.

As reported in Table 7, PyraTok attains the lowest LPIPS and competitive PSNR/SSIM on MCL-JCV [54] at a bitrate of 0.034, surpassing traditional codecs like HEVC [43] and VCC [6] in perceptual fidelity (LPIPS) by preserving fine texture and temporal coherence through semantically guided quantization.

We evaluate our tokenizer and generator on class-guided video generation using the UCF-101 [42] dataset. Given a target action class, the model is conditioned on the class label and asked to synthesize a short video clip from scratch. This setting measures not only low-level visual fidelity (appearance, motion smoothness) but also whether the generated sequence is semantically consistent with the requested action category.

We compare PyraTok against a broad set of video generative models that rely on different tokenizers and generator architectures, including MAGVIT/MAGVIT-V2, LARP-L, CogVideo, TATS, Video-LaVIT, OmniTok, and SweetTok. For all methods, we report the generative Fréchet Video Distance (gFVD), where lower values indicate better alignment with the distribution of real videos. As shown in Table 8, our method achieves the lowest gFVD on UCF-101, improving upon the strongest prior tokenizer by a substantial margin. These results indicate that our representation is better suited for high-quality, temporally coherent video synthesis, and that scaling the generator on top of our tokens directly translates into stronger video generation performance.

Table 9 presents an ablation of the quantization module in PyraTok, where each row corresponds to a different way of discretizing the encoder features, specified by its quantization type, vocabulary size, and embedding dimensionality. The simple single-codebook baseline VQ [52] (4096 / 256), with vocal size of 4096 and a dimension of 256, yields the weakest reconstruction quality on both COCO [31] and WebVid [2], confirming that a single global codebook is insufficient to capture the variability of natural image–video data. Introducing a group structure in GVQ [20] (4096 / 256) slightly improves PSNR and SSIM, and reduces LPIPS; however, the gains are modest because each group still operates with a relatively small shared codebook. The lookup-free single-block variant, LFQ [70] (32800 / 16), increases the effective vocabulary while reducing the per-code dimension, resulting in a clear improvement in PSNR and SSIM, and a lower LPIPS, indicating that finer local code assignment is beneficial.

Residual quantization with a higher-dimensional code space, RVQ [24] (8000 / 512), further reduces distortion over vanilla VQ, and replacing the residual codebook with our latent product quantizer, LaPQ (8000 / 512), yields another consistent improvement, showing that decomposing the latent channels into product codebooks makes better use of the same vocabulary size. When we combine residual modeling with LFQ-style factorization, RVQ (32800 / 16) achieves even better performance, but our full LaPQ (Ours, 48000 / 16) achieves the best performance across all metrics on both validation sets, with the highest PSNR/SSIM and lowest LPIPS, while incurring only a small increase in inference time compared to simpler schemes. Overall, results demonstrate that LaPQ’s combination of lookup-free factorization and product–residual coding provides a significantly more expressive and distortion-resilient discrete representation than standard VQ, GVQ, LFQ, or RVQ under comparable computational budgets.

We further analyze how the effectiveness of our tokenizer scales with input resolution by measuring the percentage of active codewords at different spatial resolutions (see Fig. 28). As the resolution increases from $240$p to $4320$p, codebook utilization rises monotonically from $55.23\%$ to $97.12\%$, indicating that higher-resolution inputs naturally excite a richer subset of the learned vocabulary rather than collapsing to a small set of frequently used tokens. In particular, utilization already exceeds $79\%$ at $1080$p and surpasses $90\%$ in the $4$K regime ($2160$p and $4320$p), suggesting that the proposed pyramidal design can effectively exploit the larger spatial support to express more diverse and fine-grained semantics. This trend confirms that our discrete latent space remains expressive and well-populated as we scale to high-resolution video, which is critical for both faithful reconstruction and downstream video-language understanding tasks.

We ablate each component of the training objective on three video understanding benchmarks: THUMOS14 [19] and ActivityNet v1.3 [7] for temporal action localization, and MVBench for video question answering (Table 10). With the full objective, PyraTok achieves $33.17/29.11$ Avg. mAP on THUMOS14/ActivityNet and $86.03$ mAP on MVBench. Removing the drift regularizer $\mathcal{L}_{\text{drift}}$, which anchors the adapted encoder to the pretrained VAE manifold, degrades performance by $1.90/1.49$ mAP on THUMOS14/ActivityNet and by $2.71$ points on MVBench, indicating that maintaining a stable latent space is important for robust transfer across both localization and QA.

The autoregressive alignment loss $\mathcal{L}_{\text{AR}}$ has a different effect: dropping it leads to a relatively small drop on temporal localization ($0.72/1.13$ mAP), but causes a pronounced $6.58$-point decline on MVBench. This suggests that sequence-level token modeling is especially critical for high-level video reasoning, where the model must integrate information over longer temporal horizons. When we remove both the DINO-guided visual loss and the autoregressive loss ($\mathcal{L}_{\text{dino}}+\mathcal{L}_{\text{AR}}$), performance drops most severely on TAL (by $3.88$ and $2.33$ mAP on THUMOS14 and ActivityNet, respectively) and by $4.46$ points on MVBench, highlighting the complementarity between discriminative visual supervision and global token prediction.

We further study the codebook-related objectives, as described in Equation 2. Ablating the text-conditioned alignment term $\mathcal{L}_{\text{text-cond.}}$ reduces performance by $2.95/1.56$ mAP on THUMOS14/ActivityNet and by $2.47$ points on MVBench, while removing the text–codebook alignment $\mathcal{L}_{\text{text-codebook}}$ yields a similar degradation ($2.06/2.04$ mAP and $2.12$ points). These results confirm that both local token–text alignment and global codeword–text alignment are necessary to maintain semantically structured latents that generalize well across detection and QA tasks. In contrast, dropping the vision-comment loss $\mathcal{L}_{\text{vision\_commitment}}$ produces the smallest degradation (at most $0.50/0.90$ mAP on THUMOS14/ActivityNet and $1.80$ points on MVBench), suggesting that, for downstream understanding, the semantic shaping of the codebook is more critical than the pure vision commitment penalty. Overall, the complete loss formulation is consistently superior, validating our multi-part objective for unified video understanding.

Droplet-10M [73] (Subset). We curate a subset of the Droplet-10M dataset, consisting of approximately 4–5 million HD videos (720p). This subset serves as the foundation for pretraining, providing broad coverage of human activities, natural scenes, and diverse motion patterns, and a dense caption distribution and consistent temporal dynamics, crucial for learning fine-grained video–text alignment. To ensure data quality and maintain high spatial fidelity, only videos at 720p or higher resolution are retained.

OpenVid-1M [35] (300K Subset). We supplement training with 300K high-quality video–caption pairs sampled from OpenVid-1M. Only HD videos are selected to maintain visual consistency. This subset contributes to expanding linguistic diversity and contextual variation, improving open-domain caption understanding and cross-modal reasoning.

UltraVideo [64] (40K with Reconstructed Captions). To enrich representation at extreme resolutions, we incorporate 40K ultra–high-definition videos (4K and 8K) from the UltraVideo dataset. Since many of these videos lack high-quality textual descriptions, we generate captions using a multimodal LLM pipeline. This enables the model to learn from high-fidelity visual data and supports scalability to higher-resolution downstream applications.

OpenVid-1M [35] (Test Split). We employ 100K samples from the OpenVid-1M test split for evaluating generalization to unseen open-domain video–text pairs. This ensures consistency with the distribution of the training data while validating model generalization under identical data conditions.

WebVid-10M [2] (Validation) and COCO [31] (Validation). For generative evaluation, we follow the WebVid-10M and COCO-Val validation protocols. For class-guided video generation, we further evaluate on the UCF-101 [42] dataset.

MCL-JCV [54] (Compression Validation). To assess the effectiveness of our video compression and reconstruction, we employ the MCL-JCV benchmark. This dataset provides a controlled setup for evaluating perceptual quality and rate–distortion tradeoffs under varying compression levels.

To evaluate generalization beyond supervised training, we test the model under zero-shot conditions across diverse downstream video understanding tasks. For zero-shot action localization, we evaluate on ActivityNet [7] and THUMOS14 [19], which contain diverse and complex activities. For zero-shot video segmentation, we benchmark on YouTube-VIS 2021 [66] and OVIS [36]. Both datasets present challenging dynamic scenes with multiple interacting objects and frequent occlusions. For video classification, we utilize Kinetics [22], while for VideoQA, we adopt MVBench [25], a comprehensive multi-task benchmark covering spatiotemporal reasoning, action understanding, and commonsense interpretation.