New technique improves the reasoning capabilities of large language models

According to the researchers, Natural Language Embedded Programs (NLEPs) improve the transparency and trustworthiness of AI models in several ways.

Firstly, NLEPs enable large language models to achieve higher accuracy on a wide range of reasoning tasks. This is because NLEPs involve prompting a language model to create and execute a Python program to solve a user's query, and then output the solution as natural language. This approach has been found to be generalizable, meaning one NLEP prompt can be reused for multiple tasks.

Secondly, NLEPs improve transparency by allowing a user to check the program to see exactly how the model reasoned about the query. This means that if the model gave a wrong answer, the user could identify where the error occurred and fix the program accordingly. This traceability of the model's reasoning process can enhance trust in the model's outputs.

As MIT postdoc Hongyin Luo explains, "We want AI to perform complex reasoning in a way that is transparent and trustworthy. There is still a long way to go, but we have shown that combining the capabilities of programming and natural language in large language models is a very good potential first step toward a future where people can fully understand and trust what is going on inside their AI model."

Lastly, the NLEP approach offers greater efficiency than some other methods. If a user has many similar questions, they can generate one core program and then replace certain variables without needing to run the model repeatedly. This can save time and computational resources, further enhancing the practicality and appeal of these models.

Natural language embedded programs (NLEPs) is a new technique that enables large language models to solve natural language, math, data analysis, and symbolic reasoning tasks by generating programs2. The approach involves prompting a language model to create and execute a Python program to solve a user's query, and then output the solution as natural language.

NLEPs function in four steps. First, the model calls the necessary packages or functions it will need to solve the task. Second, it imports natural language representations of the knowledge required for the task, such as a list of U.S. presidents' birthdays. Third, the model implements a function that calculates the answer. Finally, the model outputs the result as a line of natural language, possibly with an automatic data visualization.

This technique improves transparency as users can check the program to see how the model reasoned about the query and fix the program if the model gave a wrong answer. NLEPs also offer greater efficiency, as users can generate one core program and then replace certain variables without needing to run the model repeatedly for similar questions.