Researchers from T-Technologies, working with MIPT, Innopolis University and Sber, have developed SimplexLoRA, a fine-tuning method that directs more resources to the layers that matter most for a given task, T-Bank reported. The work was presented at the A* Conference on Empirical Methods in Natural Language Processing, held October 24-29 in Budapest.

The widely used LoRA method keeps a model's core weights frozen and fine-tunes it through small adapter modules built into its layers. But it gives every adapter the same number of trainable parameters, even though layers contribute unequally to a specific task. SimplexLoRA first identifies which layers are most important and redistributes the total pool of trainable parameters accordingly, giving key layers more and the rest less. That distribution is then fixed and the model is fine-tuned using the standard LoRA scheme, improving quality without raising the parameter count or compute costs.

The team tested SimplexLoRA on 12 tasks across four models, covering text understanding, summarization, question answering and reasoning. In most tests it matched or beat standard LoRA at the same parameter count and cost, performing especially well when the number of trainable parameters was small. The method is aimed at teams adapting off-the-shelf language models to their own tasks, including question answering, document key-point extraction, sentiment analysis and mathematical reasoning, and is particularly useful when compute is limited. The source code has been released publicly.

"When adapting a language model to a new task, it's important to understand where additional trainable parameters will bring the most benefit," said Stanislav Moiseev, head of the T-Technologies research and development center. He said SimplexLoRA lets developers choose that distribution at the start of fine-tuning, improving results at the same parameter count while keeping the familiar LoRA foundation.

Source: RIA Novosti