Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval
This paper introduces AlgoREval, a benchmark to evaluate the capability of large language models in parametric code retrieval.
This paper introduces a new benchmark for evaluating the capability of large language models in parametric code retrieval, which is a distinct and measurable capability.
Before reading this…
Applications
- →Code generation
- →Algorithmic code retrieval
To understand this paper, make sure you know these concepts first:
- Large language modelsfind papers →
- Code generationfind papers →
Abstract
More Like ThisLarge language models (LLMs) have demonstrated strong performance in code generation, where success depends on both recalling relevant algorithmic knowledge and reasoning about how to apply it. However, existing LLM pipelines are opaque, with no explicit separation between these two components. We argue that for well-known algorithms whose canonical implementations are widely accessible in pretraining corpora, code generation is better measured as \textit{parametric code retrieval}: reproducing a named algorithm from internalised knowledge rather than synthesizing a novel one. We introduce AlgoREval, a benchmark of 599 problems spanning classical 77 algorithms across 14 domains, 7 programming languages, and 4 graph-input representations to evaluate this capability in isolation, and assess 15 models (7B--34B parameters) in a zero-shot setting. We find substantial variation in retrieval accuracy across languages and input representations, even for widely documented algorithms and show that prompt augmentation with retrieved code snippets or structured algorithmic hints improve accuracy on complex algorithms, while SFT achieves broader language gains and GRPO achieves larger per-language gains on specific languages. Together, our results establish parametric code retrieval as a distinct, measurable capability and caution against deploying AI-generated algorithmic code without systematic validation.\footnote{Code and dataset are available at https://github.com/Nickil21/AlgoREval