How to make System 1 LLMs show System 2 competencies?

Abstract

From their inception, Large Language Models were best understood as a sort of giant external System 1 for the humanity at large (see the right part of the figure). Trained as they are on the collective digital footprints of all of humanity's articulated knowledge, LLMs have shown that their next token prediction mechanism can exhibit impressive System 1 like verbal behaviors. Unlike most AI systems preceding them, which showed “deep and narrow” System 2 competencies on particular tasks (AlphaGo, Deep Blue, etc.), LLMs were the first AI systems that brought forth broad System 1 competence across many (almost all?) tasks.

But why only System 1? How about System 2 behaviors? Despite many claims to the contrary in the early 2022–24 literature, standard LLMs have been shown to fall far short of System 2 behaviors–such as planning and reasoning. Initial excitement based on anecdotal performance of LLMs on reasoning tasks has dissipated to some extent by the later spate of studies questioning the robustness of such behaviors–be it planning (as shown by our own work), simple arithmetic and logic, or general mathematical and abstract benchmarks. Performance of LLMs on reasoning tasks only started improving substantially since fall 2024, when the first so-called “reasoning models” started coming out– OpenAI's o1, DeepSeek's R1 and so forth. The interesting part, however is that these models are still trained auto-regressively and do next token prediction at the inference stage, thus sticking close to the System 1 operation. How then did they acquire reasoning abilities?

Kahneman's account already provides a possible direction–you can “compile” System 2 behaviors into System 1. While humans, with their own System 2, might be essentially memoizing what they do through deliberation once into reflexive reasoning for later use, LLMs depend on external synthetic training data. The currently popular approach can be understood as a way of fine tuning the base model on a huge amount of synthetic data with verified solutions in the so-called “post-training” phase. We will unpack this below.

Let's start with the generally accepted fact that standard LLMs are very good generators of plausible solutions for any task including reasoning tasks. With a generator in hand, we can do a Generate-Test search to identify correct solution candidates for the reasoning tasks. While generate-test is normally viewed as a highly inefficient search technique, this can change if we have access to a good generator–and LLMs, it turns out, can fill this role rather well as they have been trained on our collective digital footprints. The question is who is going to test the candidates that an LLM generates.

While there is a temptation to make the LLMs themselves be the judges of the correctness of the candidates they generate, research (including ours) suggests that by and large, LLMs may be no better at verification (unless they have been specifically trained for verification over particular distributions of tasks) than they are at generation. (The hope that they might be better at verification seems to rest on arguments about verification being of lower computational complexity than generation. But there is little reason to support the stance that the System 1-like performance of LLMs is meaningfully correlated with computational complexity.) There are however many domains for which we have classical (pre-LLM) verifiers and/or simulators with soundness/completeness guarantees. You can verify the correctness of a plan or a schedule in polynomial time; you can provide partial verification of a generated piece of Python code by feeding it to the python interpreter along with unit tests with known correct input/output pairs, and finally, you can provide logical verification of a mathematical proof with interactive theorem provers like Lean. Even when there are no ready made verifiers and simulators, it is possible to learn them from experience. In fact, the current interest in learning the so-called “world models” can be interpreted as a way of providing a means of learned verification for the guesses of the underlying base model.

With such (external) verifiers on hand, we can put LLMs in a Generate-Test framework to extract solutions that have some form of correctness guarantee. This way of using LLM as a generator of plausible candidates and having additional mechanisms to improve the correctness of the candidate returned to the end user has been categorized as test-time scaling. In our own work, we called this the LLM-Modulo framework. In the LLM-Modulo framework, the verifier can do more than just simply give binary feedback on the correctness of the candidate produced by the LLM. The verifier can also point out specific parts of the current candidate that lead to incorrectness, and even suggest partial replacements. Such feedback is sent back to LLM as a “back prompt,” which affects the next candidate generated by the LLM.

Generate-Test techniques on LLMs, such as LLM-Modulo, can indeed get LLMs to exhibit more correct behaviors on System 2 tasks such as planning, code generation, and theorem proving. However, they are also going to consume valuable end-user facing time during inference. Can we make the base model learn to generate better solutions? Well, the answer is clear enough. Any time the generate-test process produces a solution that passes the verifier, this problem and solution pair can be made part of the training data for the LLM. A more aggressive technique would be not to wait for problems to be given at the inference time, but self learn during the training phase itself. In particular, in the so-called “post-training” phase the LLM is given a large curriculum of problem instances that it attempts to solve using the generate-test framework. These attempts become additional synthetic data to fine tune the underlying generating LLM. While the specifics of the fine tuning vary, they involve incrementally modifying the parameters of the LLM's transformer to be in tune with the pairs of problems and their correct solutions. DeepSeek R1 popularized the use of a particular type of policy gradient framework to support this learning phase. While the term RLVR–or reinforcement learning from verified reward has been used to refer to this process, it is easy to see that RLVR is essentially an RL/learning step added on top of a generate-test framework like LLM-Modulo.

With such a post-training step, that may well take significant computational resources rivaling even those used for the original pre-training of the LLM, the finetuned LLM could become a significantly better generator of solutions for reasoning problems than the original base model. In other words, post-training effectively compiles the signal from the verifiers into the base model weights, so the model is more likely to generate candidate solutions that will pass muster with the verifiers. In a way, post-training can be viewed as a realization of Marvin Minsky's dictum “Intelligence is taking the test part of the generate-test to the generate part!”

Notice that this post-trained LLM still acts very much like a next token prediction model. While the candidate solutions it generates are likely to be more accurate, there is still no guarantee of correctness. If guarantees are needed, we will still need verifiers in the loop at inference time.

The only conspicuous difference between the post-trained models and standard LLMs is that they are trained to output a stream of intermediate tokens before outputting the tokens corresponding to their solution guess. At the training time, the intermediate tokens for the post-training examples can be some summary of the interaction between the base model and the verifiers (e.g. the back prompts and the successive solutions). This is however just one possible way to populate intermediate tokens–they could also come from the base model pre-trained to output pseudo “thinking step by step” traces before outputting the solution. At the most elementary level, these additional tokens change the conditional distribution of the ensuing tokens enough to make them be closer to what the verifiers would have endorsed. Thus, the intermediate tokens serve as a scaffold to the base model to fit itself to the solution tokens. While they certainly seem to help the accuracy of the post-trained LLMs, there is considerable difference of opinion as to whether the traces necessarily have any interpretable semantics to the end users and verifier. Not surprisingly, the DeepSeek R1 training regime, which made intermediate tokens popular, advertises itself as an “outcome-based RL” method that only gives reward based on the verifiability of the solution tokens. (They do however effectively divide the reward of the solution tokens over all the tokens of the trajectory, including intermediate tokens). It is likely that the interpretability of the intermediate tokens is only an accidental reflection of imitating the “chains of thought” present in the training data (to the point that some AI safety researchers issue entreaties to the frontier companies to “preserve interpretability of chains of thought” as a fragile opportunity to monitor LLMs!).

This account also makes clear that neither standard LLMs nor the reasoning models have their own experience of the world, but instead depend profoundly on our (humanity's) experience–expressed linguistically. Our digital footprints include both declarative information (facts, opinions, etc.), and procedural information (verifiers, simulators, etc.). The post training phase can be viewed as a way of “digesting” the procedural information into the base model. (In this sense, the progress on LLMs can be interpreted as a 21st century continuation of the quest for expert systems–except that we no longer are required to manually encode our knowledge–the LLM training regime takes all our digital footprints directly).

To summarize, the way System 1 LLMs get better at System 2 tasks is by painstakingly compiling the signal from external verifiers into the base model via their post-training phase. While this clearly seems to have made the underlying models better at generating plausible solutions to reasoning problems, many questions remain about both the computational efficiency of such an approach, and, more critically, its ability to generalize out of distribution to handle novel tasks. The fact that LLMs are approximating System 2 behaviors this way through compilation into the same base level architecture might explain the “jagged intelligence” exhibited by the frontier models–doing research mathematics proofs on one hand, while still failing at trivial unarticulated common sense challenges on the other.

Further Reading:

(How) Do Reasoning Models Reason?

Subbarao Kambhampati, Kaya Stechly, Karthik Valmeekam

Annals of the New York Academy of Sciences. April 2025.

Can Large Language Models Reason and Plan?

Subbarao Kambhampati

Annals of New York Academy of Sciences. March 2024.

The author declare no conflicts of interest.