20 results for “evaluation pipeline”
CS papers onlyHybrid search: Keyword + semantic, ranked by combined score.ⓘ
Want pure semantic search? Try claim verification →
This paper describes the use of a trie data structure to enhance experiment efficiency in comparative pipeline experiments for cascading retrieval pipelines using PyTerrier, observing a 26% reduction…
Fangyijie Wang, Jianjun Yu, Wentao Shi, Haixia Huang +3 more
A clinician-centered pipeline is proposed for remote annotation and evaluation of ultrasound AI studies.
BADGER is a unified, production-grade evaluation framework that integrates text-to-SQL assessment with agentic behavior evaluation, significantly outperforming existing benchmarks on industry queries.
The paper introduces MedCase-Structured, a synthetic, FHIR-formatted dataset designed to benchmark diagnostic reasoning in realistic EHR settings, showing that LLMs perform worse on structured data th…
DS@GT ARC evaluates the effectiveness of Corrective RAG (CRAG) and CiteFix in improving citation faithfulness and answer grounding in Retrieval-Augmented Generation (RAG) QA systems, revealing a trade…
This paper presents an open-source computer vision pipeline for classifying vehicle body types from naturalistic roadway video.
This paper systematically evaluates the consistency of popular causal discovery benchmarks against real-world scientific literature, revealing significant variability in their accuracy.
Yijun Shen, Minghao Shao, Yichen Zhao, Zhuoyan Yu +3 more
The paper introduces VHDLSuite, an infrastructure for evaluating Large Language Models in VHDL, including a data pipeline, benchmark, and evaluation framework.
The paper introduces a data-centric optimization pipeline to improve coding agents' ability to interact with a branching lakehouse, showing significant accuracy gains by treating agent evaluation as a…
The paper introduces a Deep Research pipeline that significantly improves literature search recall and demonstrates that human-curated citation lists are often unreliable and do not serve as a true gr…
The paper evaluates prompt-injection defenses for educational LLM tutors, demonstrating that optimal security requires balancing adversarial robustness, usability, and latency, and proposing a compreh…
This paper proposes a structured pipeline using LLMs to generate and evaluate obfuscated XSS payloads, demonstrating that while LLMs can generate samples, they currently struggle to ensure payloads ma…
The paper introduces GTA, a scalable framework for generating realistic, multi-hop web-agent tasks with dense, executable trajectories, addressing the current lack of process-level supervision in web…
Shahbaz Siddeeq, Mateen Abbasi, Jussi Rasku, Zheying Zhang +3 more
This paper compares the quality and coverage of epic-organized LLM-generated Gherkin acceptance criteria with requirement-aligned generation, using four requirements documents from the PURE dataset.
This paper introduces MARS, a framework for repeat-order food delivery recommendation that combines pre-trained language models with collaborative retrieval and contextual filtering.
Minjae Lee, Hee-Soo Heo, Youngki Kwon, Han-Gyu Kim +2 more
The paper introduces Target Speaker Tagging (TST), a task that combines speaker diarization, verification, and identification into a single workflow for multi-speaker conversations. It presents TST-Be…
This paper conducts a large-scale audit of human annotation reporting in NLP, finding that while reporting has improved, critical details needed to assess annotation validity, such as training and agr…
The paper introduces MMGenre, a benchmark for multi-genre singing voice synthesis diagnosis, revealing limited genre discrimination and proposing lightweight genre-specific continued training.
This paper introduces KaliBench, a fine-grained benchmark and dataset for evaluating natural-language-to-CLI translation in cybersecurity workflows.
Rongxu Cui, Zongzheng Zhang, Jingrui Pang, Haohan Chi +10 more
This paper introduces a safety benchmark and dataset to evaluate the safety of Vision-Language-Action models.