A new method for generating synthetic training data is showing significant promise for improving how large language models learn to code. Researchers have developed a workflow that creates targeted datasets by focusing on specific programming concepts, moving beyond the traditional approach of simply amassing vast quantities of text.

The technique starts with a detailed taxonomy—a hierarchical map—of thousands of programming concepts, from basic strings to advanced graph algorithms. This map was built by analyzing existing code datasets. Using this structure, the team can instruct a data generation system to produce exercises that combine selected concepts, controlling for difficulty and scope.

As a practical test, the method was used to create a new dataset called Nemotron-Pretraining-Code-Concepts. It contains 15 million unique Python programming problems, all validated as functional code, derived from 91 core concepts identified as foundational. When approximately 10 billion tokens from this synthetic dataset were blended into the final stages of training for the Nemotron-Nano-v3 model, the results were clear: a six-point jump in accuracy on the standard HumanEval benchmark, rising from 73 to 79.

Beyond the numbers, engineers observed the model handling edge cases and complex algorithmic reasoning with greater reliability. The entire framework and the new coding dataset have been released under an open license, offering a template for other teams to build specialized training data for different skills, potentially reshaping how models are taught.

Source: Hugging Face Blog