Kajetan Schweighofer

The paper introduces TSVD, a novel framework that efficiently pre-trains LLMs by enforcing both low rank and strict weight orthonormality, achieving performance comparable to full-parameter models with significantly reduced computational cost.

This paper introduces Anchored Weight Decay (AWD), a regularization technique that effectively prevents prior-task forgetting during LLM fine-tuning with Evolution Strategies (ES), positioning ES as a viable method for continual learning.

This paper introduces RREDCoT, a method for approximating optimal reward redistribution in Chain-of-Thought reasoning language models without additional generation.

This paper introduces AutoBackSwap, a method to reduce reliance of classifiers on spurious backgrounds in image classification tasks using a secondary network and infilling.