Is Programming by Example solved by LLMs?

Summary

This paper talks about the ability of large language models (LLMs) to solve Programming by Example (PBE) tasks, which involve generating algorithms based on input-output examples. The authors investigate whether LLMs can effectively perform PBE and how their performance can be improved.

What's the problem?

Programming by Example is a method where users provide examples of inputs and expected outputs, and the system generates the code that produces those outputs. While LLMs have shown success in generating code, there are questions about whether they can consistently handle PBE tasks, especially when the problems presented are not similar to those they were trained on. This raises concerns about their effectiveness in real-world applications where examples can vary widely.

What's the solution?

The authors conducted experiments using LLMs on classic programming tasks involving lists and strings, as well as a more challenging graphics programming domain. They found that while pretrained models struggled with PBE tasks, fine-tuning these models significantly improved their performance, especially when the test problems were similar to the training data. They also analyzed why these models succeeded or failed and explored ways to enhance their ability to generalize to new types of problems.

Why it matters?

This research is important because it highlights both the progress and limitations of LLMs in solving PBE tasks. By understanding how LLMs can be improved through fine-tuning and identifying their weaknesses, researchers can develop better programming tools that are more flexible and applicable in various scenarios. This could lead to more user-friendly systems for coding that help people generate algorithms without needing deep programming knowledge.