NIno Scherrer
Research Areas
Authored Publications
Sort By
Preview abstract
Large language models have achieved remarkable capabilities across domains, yet mechanisms underlying sophisticated reasoning continue to be explored1,2. Recent reasoning-reinforced models, including OpenAI’s o-series and DeepSeek-r1, outperform other merely instruction-tuned models on complex cognitive tasks3,4, attributed to extended test-time computation through longer chains of thought5. Here we show that enhanced reasoning emerges not from extended computation alone, but from the systematic simulation of complex, multi-agent interactions—a society of thought—which enables the deliberate diversification and debate among internal cognitive perspectives characterized by distinct personality traits and domain expertise. Through quantitative analysis using classified outputs and mechanistic interpretability methods applied to reasoning traces6–8, we find that reasoning models like DeepSeek-r1 exhibit much greater perspective diversity than baseline models, activating broader and more conflict between heterogeneous personality- and expertise-related features during reasoning. This multi-agent structure manifests in conversational behaviors including question-answering sequences, perspective shifts, and reconciliation of conflicting views, as well as in socio-emotional roles that characterize back-and-forth conversation, which together account for over 60% of the accuracy advantage in reasoning tasks through both direct and indirect facilitation of cognitive strategies9,10. Controlled reinforcement learning experiments further reveal that priming models with conversational scaffolding—even when dialogues lead to incorrect solutions—substantially accelerates reasoning improvement compared to answer-only training. These findings indicate that the social organization of thought, rather than correctness alone, enables effective exploration of solution spaces. We suggest that reasoning models establish a computational parallel to collective intelligence in human groups11–13, where diversity enables superior problem-solving when systematically structured and suggest new opportunities for agent organization to harness the wisdom of crowds.
View details
Preview abstract
This article is targeting an external publication like CACM and is meant to be an opinion piece.
Abstract:
Large Language Models (LLMs) have revolutionized the AI landscape, demonstrating remarkable capabilities across a wide range of tasks. Each new model seemingly reinforces the notion that modern transformer-based AI can conquer any challenge if armed with sufficient compute and data. However, the scaling-driven paradigm is far from a universal solution to AI’s diverse challenges. For example, while scaling has accelerated certain applications, such as robotics, it has yet to show significant impact in others, such as identifying misinformation. Currently, there is no clear framework for distinguishing which use cases thrive from scaling with more data and which demand alternative approaches.
We are beginning to observe that the shape of data itself may hold valuable clues that could inform the success of data-driven scaling. For instance, insights from topological data analysis suggest that examining structural patterns and stability of data across multiple scales can help determine when scaling will be advantageous.
Moreover, the practicalities of data acquisition impose additional constraints that we must factor into the scaling equation upfront. Factors such as availability of high quality data, with its highly nuanced definition, complexity and resource intensity of data collection, and availability of proper evaluation benchmarks determine not just the effectiveness but also viability of scaling.
We have translated these emerging insights about data shape and nature of data acquisition into a practical framework of questions that evaluate predictiveness of historical data, stability of data patterns, clarity of data requirements, feasibility of high-quality data collection, and ease of assessing data quality. Together, these answers can help practitioners make more informed decisions about when scaling is more likely to yield successful outcomes. We have applied the framework to several AI use cases as an example. These early observations highlight a critical need for continued research in this domain.
Full draft link: https://docs.google.com/document/d/1f-HQ69KA4Ec7lNeWI-lTHUurhC0WMO4lNUiDUAEPKes/edit?usp=sharing
View details
MesaNet: Sequence Modelling by Locally Optimal test-Time Training
Songlin Yang
Razvan Pascanu
Alexander Meulemans
Seijin Kobayashi
Yanick Schimpf
João Sacramento
Maximilian Schlegel
Luca Versari
Oliver Sieberling
2025
Preview abstract
Sequence modeling is currently dominated by causal transformer architectures that
use softmax self-attention. Although widely adopted, transformers require scaling
memory and compute linearly during inference. A recent stream of work linearized
the softmax operation, resulting in powerful recurrent neural network (RNN)
models with constant memory and compute costs such as DeltaNet, Mamba or
xLSTM. These models can be unified by noting that their recurrent layer dynamics
can all be derived from an in-context regression objective, approximately optimized
through an online learning rule. Here, we join this line of work and introduce a
numerically stable, chunkwise parallelizable version of the recently proposed Mesa
layer (von Oswald et al., 2024), which could only run sequentially in time and was
therefore not scalable. This layer again stems from an in-context loss, but which is
now minimized to optimality at every time point using a fast conjugate gradient
solver. Through an extensive suite of experiments study up to the billion-parameter
scale, we show that optimal test-time training enables reaching lower language
modeling perplexity and higher downstream benchmark performance than previous
RNNs, especially on tasks requiring long context understanding. This performance
gain comes at the cost of additional flops spent during inference time. Our results
are therefore intriguingly related to recent trends of increasing test-time compute to
improve performance – here by spending compute to solve sequential optimization
problems within the neural network itself.
View details
