论文精选 82arXiv

Latent Core Tokenizer: Compress, but Meaningfully· 潜在核心分词器：有意义的压缩

Tokenizers are commonly optimized for compression, but a compact vocabulary does not necessarily distribute its capacity evenly across languages. We introduce the Latent Core Tokenizer (LCT), a language-agnostic approach that separates structural discovery from vocabulary construction. LCT uses Minimum Description Length, entropy-based boundary signals, and morphotactic constraints to identify reusable linguistic units before constructing a shared vocabulary. Across 104 languages with a 200K-token vocabulary, LCT achieves lower fertility and higher MorphScore than BPE, Unigram, and parity-aware BPE, while maintaining comparable cross-lingual disparity in tokenization cost. Across four multilingual downstream benchmarks, LCT improves aggregate score by 1.48, 1.83, and 2.00 points over BPE,

领域：cs.CL作者：Felermino D. M. A. Ali、Millicent Ochieng、Ogbemi Ekwejunor-Etchie

#cs.CL

📎 发布于 10-09 01:32🔗 阅读原文（arxiv.org）📄 PDF

Low-Bit N-Gram Embeddings for Language Models· 大模型低精度 n-gram 嵌入

Many recent open-weight language models use n-gram embedding tables to improve performance…

领域：cs.CL作者：Stephen Chung、Wenyu Du

📎 arXiv🕒 10-23 01:56🔗 arxiv.org

论文精选 82

Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2· LLM 能否调试网络语料库？

Multilingual pipelines such as FineWeb2 apply one recipe across a thousand languages, leav…

领域：cs.CL作者：Marek Šuppa、Ivan Vykopal、Jaroslav Kopčan

📎 arXiv🕒 10-23 01:52🔗 arxiv.org

论文精选 83

Bilingual Language Models Capture Cross-Linguistic Influence in Second-Language Reading· 双语模型捕捉二语阅读的跨语言影响

While monolingual language model (LM) surprisal is widely established as a predictor of re…

领域：cs.CL作者：Abderrahmane Issam、Irene E. Winther、Stefan L. Frank

📎 arXiv🕒 10-23 01:42🔗 arxiv.org

论文精选 82

How Ground Truth Shapes Retrieval Scores: Evidence-Aware Evaluation of a Retrieval-Augmented Generation System for Breast Lesions· 乳腺病变管理文献中的检索系统评估

Purpose: To describe an evidence-aware retrieval evaluation protocol for a clinical retrie…

领域：cs.CL作者：Miguel Mochizuki Silva、Samuel Ismael、Mateus C. Dias

📎 arXiv🕒 10-23 01:36🔗 arxiv.org

论文精选 80

Would an LLM Match into a Residency? Uncertainty, Reasoning and Image Use of Open Language Models on the Spanish MIR Examination· 大语言模型能通过医学考试吗？

Accuracy on medical examinations is common evidence offered for the clinical use of langua…

领域：cs.CL作者：Miguel Fernandez-de-Retana、Unai Zulaika、Ruben Sanchez-Corcuera

📎 arXiv🕒 10-23 01:23🔗 arxiv.org

本站内容由 LLM 精选聚合，原文版权归 arXiv 所有 · 摘录仅供参考