Grading the Unannotated: Insights from the WorkRB Skill Extraction Challenge at RecSys-HR 2026· 技能提取挑战赛：RecSys-HR 2026 的见解
Skill extraction benchmarks score models against a few human-annotated positives and treat every other skill in the vocabulary as equally wrong. At the scale of ESCO (13,939 skills) this binary view can reward models that rank the annotated positive high for the wrong reason, and hides semantically nonsensical predictions. The WorkRB Challenge at RecSys-HR 2026 replaces it with graded relevance: every candidate skill in a model's ranking window carries a 0 to 4 grade, and the leaderboard scores nDCG@100 against these grades. Since ternary judgements proved too coarse for objective annotation, relevance is decomposed into a cascade of four binary questions (same domain, activity mentioned, right granularity, replacement of a gold positive) that a smaller LLM judge can answer reliably; the l
技能提取挑战赛引入分级相关性评估，改进模型评价方式。
- 核心方法
- 使用四个二元问题（同一领域、活动提及、正确粒度、替代黄金正例）的级联来分解相关性，通过小规模LLM进行可靠评估。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 现有的技能提取基准测试只能评估少数人类标注的技能，忽略了模型对未标注技能的错误预测。
- 关键实验
- 在RecSys-HR 2026技能提取挑战赛中进行了实验，评估了模型在100个排名窗口中的nDCG表现。
- 主要贡献
- 提出了分级相关性评估方法，提高了技能提取模型的评价精度。
- 意义与局限
- 该方法能更准确地反映模型的性能，避免因误判带来的奖励，但可能增加标注成本。