Specializing a model doesn't mean a loss of general ability. @bespokelabsai trained Inkling on debugging a singular repo and produced a model that's better at coding across the board, while using fewer tokens to get the answer right.
Bespoke Labs post-trains Inkling on one code repo and reports broader coding gains
AISummary
Bespoke Labs post-trained the Inkling base model on a single GitHub repository using supervised fine-tuning and GRPO reinforcement learning. The post reports a 57-point improvement on the held-out fontTools evaluation over the base model, along with gains on Terminal-Bench 2.1 and SWE-bench Lite. It also says the post-trained model uses about 40% fewer tokens.
Post on XView on X
@tinkerapi
How to post-train a model to personalize it on your code repo? In our latest research in Bespoke Labs, we post-trained a model to improve its performance on a given Github repository. Starting from Inkling base, we use supervised fine-tuning (SFT) with trajectories coming from a strong teacher model, and reinforcement learning (GRPO) on repository-specialized environments that we curated. SFT gave a 52pp improvement in performance on the held-out fontTools evaluation set. Further RL training lifts the total improvement to 57pp compared to the base Inkling model. In addition to the in-distribution evaluation our post-trained Inkling shows good performance on Terminal-Bench 2.1 and SWE-Bench Lite while becoming 40% more token efficient due to post-training. Read our full research blog post here: https://bespokelabs.ai/blog/personalizing-inkling-for-your-code-repository-with-post-training Many thanks to Thinking Machines Lab for their credit contribution that helped support this research.
Source: Tinker · x.comPublished · added here