"Open Source"
Visual document retrieval—ranking and searching raw PDF page screenshots directly without fragile OCR parsing—has emerged as a foundational building block for enterprise visual RAG systems. However, prevailing state-of-the-art visual retrievers (such as ColPali and ColQwen) rely heavily on adapting generative Visual Language Models (VLMs). These architectures stitch a separate pretrained vision tower (e.g., SigLIP) through a projection layer into an autoregressive causal language model. Because information retrieval and vector search require representation learning rather than autoregressive token generation, carrying over the parameter bulk, unidirectional attention constraints, and memory overhead of a generative decoder imposes severe latency and storage penalties.
On September 3, 2026, H Company introduced NeoMME (pronounced "nee-oh-me"), an open-source family of 260M and 800M parameter multilingual, multimodal foundation encoders released under the Apache 2.0 license. By abandoning dual-tower and VLM-based designs in favor of a single bidirectional Transformer that natively ingests both text tokens and raw 32×32 image patches, NeoMME delivers state-of-the-art retrieval performance on the ViDoRe v3 benchmark while matching models 14× its size, doubling indexing throughput, and slashing late-interaction storage by up to 255× down to 6 kB per page.
Key Breakthroughs.
1. Single-Tower
Unified Bidirectional Architecture Unlike conventional multimodal encoders that adapt existing dual-tower models or strip decoders from autoregressive VLMs, NeoMME processes multimodal inputs in a single computational path:
Native Patch & Token Embedding: Text tokens pass through factorized token embeddings, while raw images are segmented into non-overlapping 32×32 pixel patches and projected via a lightweight MLP directly into the same hidden dimension. Dynamic Resolution & Aspect Ratio Preservation: Images retain their native aspect ratios and resolutions, dynamically assigning more tokens to dense, information-rich document layouts without artificial distortion. Long Bidirectional Attention Stack: Supports a native context length of 16,384 tokens (accommodating up to two uncompressed 4K UHD document pages). The backbone incorporates grouped-query attention (GQA), query-key normalization, gated attention, 2D rotary position embeddings (RoPE), and alternating sliding-window attention with global attention every sixth layer.
2. Pretraining via
Discrete Masked-Diffusion Denoising NeoMME was trained from scratch across 524 billion packed multimodal tokens without bootstrapping from existing vision or language models:
Masked Diffusion Text Denoising: Text-only samples undergo stochastic token masking with corruption rates sampled uniformly between 0 and 1. Multimodal samples apply high corruption rates between 0.3 and 1.0 while leaving image patches fully visible. Forced Visual Grounding: Under high masking rates, the model cannot rely on local language priors to infer missing tokens (e.g., recovering specific financial figures or table entries), compelling the bidirectional attention layers to ground textual semantics directly in the visual page patches.
- NorMuon Optimization: Pretrained using the data-efficient NorMuon optimizer, maximizing convergence speed and representation quality across a compact training budget.
3. Dual-Head
Architecture: Dense & Late-Interaction in One Pass For downstream visual document retrieval, NeoMME-Retriever integrates two complementary retrieval heads trained jointly atop the shared backbone:
Dense Retrieval Head: Applies mean pooling across hidden states into a single normalized vector, enabling instant approximate nearest neighbor (ANN) retrieval using standard vector databases (Qdrant, Milvus, Weaviate). Late-Interaction Head: Projects each token and visual patch into a 128-dimensional normalized embedding, enabling fine-grained ColBERT/ColPali-style token-level maximum similarity (MeanMaxSim) matching between query tokens and localized image regions. Unified Forward Pass: Both representations are generated in a single execution pass, allowing production pipelines to run fast dense filtering followed by high-precision late-interaction reranking without redundant inference.
4. 255x
Storage Compression for High-Resolution Late-Interaction A primary barrier to scaling late-interaction visual search has been index storage: storing uncompressed multi-vector embeddings for high-resolution 2048×2048 documents generates roughly 4,200 vectors (~1.5 MB to 2.1 MB per page). NeoMME-Retriever solves this through two synergistic compression techniques:
Hierarchical Token Pooling: Spatially and semantically clusters adjacent document patch vectors, reducing the vector count by up to 10×. Asymmetric Quantization: Quantizes document vectors to 1-bit binary representations while maintaining query embeddings in higher precision (int8). Compression Frontier: A pooling factor of 8 paired with binary document quantization shrinks index footprint from 1.5 MB down to 6 kB per page (a 255× compression) while preserving over 95% of baseline retrieval accuracy (nDCG@10).
Technical Specifications & Benchmark Overview
& Benchmark Overview Model Parameters ViDoRe v3 (nDCG@10) ViDoRe v2 (nDCG@5) ViDoRe v1 (nDCG@5) Throughput (2048×2048, L40S) ColModernVBERT 250M 0.261 0.407 0.806 26 pages/sec ColSmol-256M 256M 0.207 0.348 0.797 ~30 pages/sec NeoMME-260M 260M 0.523 0.522 0.860 51 pages/sec ColSmol-500M 500M 0.340 0.455 0.825 22 pages/sec NeoMME-800M 800M 0.556 0.559 0.874 34 pages/sec Vultron Flash 850M 0.565 0.604 0.882 28 pages/sec ColPali v1.3 2.92B 0.430 0.547 0.848 9 pages/sec ColQwen2.5-v0.2 3.75B 0.524 0.601 0.895 7 pages/sec
Note: NeoMME-260M matches the retrieval quality of 3.75B parameter ColQwen2.5 with 14× fewer parameters and over 7× faster encoding throughput
Verified Integration & API Usage
NeoMME-Retriever is natively supported in Hugging Face Transformers and Sentence Transformers:
pythonimport torch from PIL import Image import requests from transformers import NeoMMEProcessor, NeoMMEForRetrieval from sentence_transformers.util import cos_sim, mean_maxsim model_name = "Hcompany/NeoMME-260M-Retriever" processor = NeoMMEProcessor.from_pretrained(model_name) model = NeoMMEForRetrieval.from_pretrained(model_name, device_map="auto") # Load document page image and query doc_url = "https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/shift_kazakhstan.jpg?raw=true" image = Image.open(requests.get(doc_url, stream=True).raw) query = "What percentage of offshore oil production is planned for 2026?" doc_inputs = processor.apply_chat_template( [[{"role": "user", "content": [{"type": "image", "image": image}]}]], task="document", tokenize=True, return_tensors="pt" ).to(model.device) query_inputs = processor.apply_chat_template( [[{"role": "user", "content": query}]], task="query", tokenize=True, return_tensors="pt" ).to(model.device) with torch.inference_mode(): doc_out = model(**doc_inputs) query_out = model(**query_inputs) # Compute fine-grained late-interaction score late_score = mean_maxsim( query_out.embeddings, doc_out.embeddings, a_mask=query_inputs["attention_mask"], b_mask=doc_inputs["attention_mask"] ) # Compute dense cosine similarity dense_score = cos_sim(query_out.dense_embeddings, doc_out.dense_embeddings) print(f"Late-Interaction Score: {late_score.item():.4f}") print(f"Dense Similarity Score: {dense_score.item():.4f}")