Iterative Hypothesis Generation for Scientific Discovery with Monte Carlo Nash Equilibrium Self-Refining Trees (2025)

Limitations of Existing Approaches.

Previous works have attempted to address these gaps through approaches like zero-shot hypothesis generation but suffer from critical limitations: 1) Lack of Iterative Refinement: Hypotheses may be theoretically sound but lack iterative refinement[20]. 2) Imbalanced Exploration-Exploitation: Conventional approaches struggle to balance novel hypothesis exploration with established patterns, leading to biased or suboptimal results[9].

Addressing Challenges with MC-NEST.

MC-NEST integrates Nash Equilibrium strategies with LLM-based self-refinement to address these limitations: 1) Dynamic Adaptation: MC-NEST balances exploration and exploitation using Nash Equilibrium, enabling adaptability to emerging scientific contexts. For example, in protein engineering, it explores less-studied modifications (e.g., alanine in a glycine-rich linker) while leveraging well-known substitutions (e.g., lysine-for-arginine in the NLS). 2) Iterative Self-Refinement: MC-NEST employs MCTS with iterative self-critique, refining hypotheses against known principles. For instance, it identifies trade-offs (e.g., reduced binding affinity) and incorporates additional modifications (e.g., alanine in the glycine-rich linker). 3) Strategic Exploration: MC-NEST uses sampling approaches to prioritize high-potential hypotheses while maintaining diversity, ensuring robust hypothesis generation.

2.2.1 Initialization.

In MC-NEST, the root node represents the initial hypothesis state, with edges denoting potential transformations or refinements through iterative self-critique and exploration strategies. To initialize the root node, we use a pre-trained LLM with a Zero-Shot Chain-of-Thought (ZSCoT) strategy[10]. Specifically, the LLM is prompted with the input instance $p$ to generate an initial hypothesis without relying on task-specific fine-tuning or prior search history. This approach leverages the LLM’s broad, pre-trained knowledge to establish a well-reasoned starting point, enhancing adaptability and promoting a wide, unbiased exploration of the hypothesis space. The initialization is represented as: $\smash{\text{root}=\text{Node}(\text{hypothesis}=\text{ZSCoT\_LLM}(p))}$

2.2.2 Candidate Node Generation.

Child nodes are generated by applying a structured process of self-refinement and self-evaluation to the parent node’s hypothesis. Self-refinement focuses on improving the hypothesis itself by prompting the LLM with the current hypothesis and customizing instructions, such as increasing specificity, enhancing novelty, or aligning better with empirical data. The LLM determines what to refine using predefined heuristics—such as logical coherence, relevance to the research goal, and consistency with known information—that guide the refinement. Following refinement, self-evaluation updates the hypothesis against these metrics to ensure each child node represents an improvement over its parent. Nodes are visited using a breadth-first search (BFS) strategy[4], where a node is expanded if it has not reached its maximum allowed children and none of its children have a higher quality score $Q$ than the node itself. If no candidate nodes meet these criteria, the method refines the root node, reinitializing the search by generating a new hypothesis using the ZSCoT strategy. This approach balances exploration (generating new hypotheses) and exploitation (refining existing ones) by dynamically adjusting based on the quality scores of hypotheses. While global optimality is not guaranteed, the iterative refinement process aims to converge towards high-quality hypotheses, with higher $Q$-scores indicating better solutions.

2.2.3 Nash Equilibrium Strategy for Node Selection.

The hypothesis generation process in MC-NEST begins with an initial hypothesis generated by a pre-trained LLM at the root node of a search tree. Each node represents a unique hypothesis state, and edges signify possible refinements through iterative self-critique. Child nodes are created by refining the parent node’s hypothesis using structured prompts, employing self-refinement and self-evaluation techniques to iteratively enhance the hypothesis.

During node selection, MC-NEST uses the UCT, where each node is assigned a quality score $Q$ derived from evaluation metrics such as logical coherence, novelty, and empirical alignment. The UCT score balances the exploration of under-explored nodes and the exploitation of high-quality hypotheses, guiding the search toward optimal solutions. A node is considered fully expanded if it reaches the maximum allowed number of children or if any child exhibits a reward $Q$ greater than or equal to that of the current node. For a set of candidate nodes, $Node(Hypothesis)=\{h_{1},h_{2},\dots,h_{n}\}$, the Nash Equilibrium strategy assigns a uniform probability distribution over possible actions: $\pi(h_{i})=\frac{1}{n},\quad\forall i=1,2,\dots,n$, where $n$ is the number of candidate nodes. This uniform probability ensures fair exploration of the hypothesis space, preventing premature convergence to suboptimal solutions. The MC-NEST framework employs three selection policies to balance exploration and exploitation:

These policies systematically balance exploration and exploitation, ensuring that the search process prioritizes high-reward nodes while maintaining a broad exploration of the hypothesis space.

2.2.4 Upper Confidence Bound (UCT) Update.

The UCT update guides node refinement by computing: $UCT(i)=Q(i)+C\sqrt{\frac{\ln(N_{\text{parent}})}{N(i)+\epsilon}}$,where $Q(i)$ is the hypothesis reward, $C$ controls exploration, $N_{\text{parent}}$ is parent visits, $N(i)$ is node visits, and $\epsilon$ avoids division by zero. The score is adjusted with Nash equilibrium probability: $\text{UCT}(i)=Q(i)+C\sqrt{\frac{\ln(N_{\text{parent}})}{N(i)+\epsilon}}+\frac{$.The node with the highest score, $i^{*}=\arg\max_{i}[\text{Score}(i)]$, is selected for refinement or as the final hypothesis, ensuring robust exploration and exploitation of the hypothesis space.

2.2.5 Expansion.

Following node selection, MC-NEST expands the search tree by generating a refined child node. Given a selected node $n_{s}$, a new child $n_{c}$ is created via self-refinement: $n_{c}=\text{SelfRefine}(n_{s})$. This process critiques and improves the solution at $n_{s}$, storing the refined version in $n_{c}$: $n_{s}\text{.children}\leftarrow n_{s}\text{.children}\cup\{n_{c}\}$. The critique is formulated as $\text{Critique}(a_{s})=\text{LLMCritique}(p,a_{s})$, where $p$ is the problem instance. The refined answer $a_{c}$ is: $a_{c}=\text{RefineAnswer}(p,a_{s},\text{Critique}(a_{s}))$ and assigned to $n_{c}$. This structured expansion enables MC-NEST to enhance solutions iteratively, driving systematic search improvement.

2.2.6 Backpropagation.

MC-NEST updates node quality scores $Q$ and visit counts from the newly expanded node $n_{c}$ up to the root. This propagates deeper exploration insights into higher-level decisions. Given a child node $n_{c}$ and its parent $n_{p}$, backpropagation updates $Q(n_{p})$ using: $Q(n_{p})=\frac{Q(n_{p})+\max(Q(n_{c}))}{2}$. This balances the exploitation of known values with exploration. The visit count is incremented:$\text{Visit}(n_{p})=\text{Visit}(n_{p})+1.$The process recurses from $n_{c}$ to the root, ensuring informed node selection in MC-NEST.

2.2.7 Self-Refine.

2.2.8 Self-Evaluation.

MC-NEST iteratively improves candidate solutions via LLM-based critique and refinement. Given a node $n$ with answer $A_{n}$, a critique $C_{n}$ is generated using:$C_{n}=\text{LLM}(P+A_{n}).$Using $C_{n}$, the answer is refined:$A_{n+1}=\text{LLM}(P+A_{n}+C_{n}).$The refined answer $A_{n+1}$ is stored in a new child node, iteratively enhancing solutions in MC-NEST.

2.2.9 Human-AI Collaboration.

MC-NEST is designed to facilitate iterative human-AI collaboration, enabling researchers to refine and validate hypotheses dynamically. Upon generating a final hypothesis, MC-NEST enables human experts to evaluate its novelty, clarity, significance, and verifiability, with the option to iteratively refine the process as needed based on researcher input. This iterative loop ensures that the generated hypotheses align with domain-specific knowledge and scientific rigor while also incorporating human intuition and expertise. By integrating human judgment at critical stages, MC-NEST not only enhances the reliability of its outputs but also fosters a collaborative environment where AI augments human creativity rather than replacing it.

Prompting Strategies.

Table2 summarizes the performance of different prompting strategies on the Social Science dataset. ZSCoT consistently outperforms ZS and FS approaches across all evaluated LLMs. For DeepSeek-32B, ZSCoT achieves an average score of 2.65, compared to 2.44 for ZS and 2.52 for 2-FS. Similarly, DeepSeek-7B with ZSCoT attains an average score of 2.56, outperforming ZS with 2.34 and 2-FS with 2.45. GPT-4o also shows significant improvements with ZSCoT, achieving an average score of 2.46 compared to 2.12 for ZS.

MC-NEST Sampling Strategies.

Table3 presents the results of MC-NEST evaluations using Greedy, Importance Sampling, and Pairwise Importance Sampling. For GPT-4o, Greedy sampling with an eight-step rollout achieves the highest overall score of 2.81. Pairwise Importance Sampling, however, excels in novelty with 2.74 while maintaining competitive clarity and significance scores. DeepSeek-32B shows similar trends, with Greedy sampling achieving the best overall results with 2.91 at an eight-step rollout. For DeepSeek-7B, Importance Sampling performs best at a four-step rollout with 2.78, while Pairwise Importance Sampling achieves balanced performance at eight steps with 2.76. These results highlight the effectiveness of MC-NEST in enhancing the quality of social science hypothesis generation.

Prompting Strategies.

As shown in Table4, ZSCoT again demonstrates outstanding performance on the Computer Science dataset. DeepSeek-32B achieves an average score of 2.74 with ZSCoT, compared to 2.48 with ZS and 2.60 with 2-FS. Similarly, DeepSeek-7B with ZSCoT attains an average score of 2.71, outperforming ZS with 2.53 and 2-FS with 2.29. Notably, verifiability scores improve significantly with ZSCoT, reaching 3.00 for DeepSeek-32B, highlighting the usefulness of structured reasoning for factual consistency.

MC-NEST Sampling Strategies.

Table5 presents the results of MC-NEST evaluations on the Computer Science dataset. For GPT-4o, Pairwise Importance Sampling outperforms other strategies at an eight-step rollout, achieving an average score of 2.82. DeepSeek-32B achieves its highest score with Pairwise Importance Sampling at an eight-step rollout with 2.85, while DeepSeek-7B performs best with Greedy and Importance Sampling at a four-step rollout with 2.78. These results suggest that longer rollouts and adaptive sampling strategies enhance computer science hypothesis generation quality.

Prompting Strategies.

Table6 summarizes the performance of prompting strategies on the Biomedicine dataset. ZSCoT consistently improves performance across LLMs, with DeepSeek-32B achieving an average score of 2.80, compared to 2.40 with ZS and 2.48 with 2-FS. Qualitative metrics, such as novelty and significance, also show substantial improvements with ZSCoT. For instance, DeepSeek-32B with ZSCoT achieves a novelty score of 2.55 and a significance score of 2.90, compared to 1.95 and 2.65 with ZS, respectively.

MC-NEST Sampling Strategies.

Table7 presents the results of MC-NEST evaluations on the Biomedicine dataset. For GPT-4o, Greedy and Pairwise Importance Sampling perform best at an eight-step rollout, achieving an average score of 2.84. DeepSeek-32B achieves its highest score with Importance Sampling at a four-step rollout with 2.87, while DeepSeek-7B performs best with Pairwise Importance Sampling at eight steps with 2.88. These results demonstrate the importance of adaptive sampling strategies using MC-NEST for optimizing hypothesis generation in biomedicine domains.

Our experiments demonstrate that structured reasoning and adaptive sampling strategies with MC-NEST significantly enhance hypothesis generation quality across domains. Increasing rollout lengths generally improves performance, with Pairwise Importance Sampling offering a competitive balance between novelty and verifiability. These findings underscore the importance of MC-NEST with sampling strategies for optimizing LLM performance in scientific hypothesis generation.

4.3.1 Human Evaluation and Case Study.

The human evaluation results in Table8 highlight the outstanding performance of MC-NEST with Greedy, Importance Sampling, and Pairwise Importance Sampling strategies compared to other approaches. MC-NEST with ZSCoT prompting achieved the highest average score of 2.62 in Social Science and 2.37 in Computer Science, while Importance Sampling achieved the best performance in Biomedicine with a score of 2.37. Notably, MC-NEST with Importance Sampling prompting outperformed other methods in Biomedicine, achieving a score of 2.44 with Deepseek-32B, and in Social Science, scoring 2.41 with Deepseek-7B. Similarly, Pairwise Importance Sampling demonstrated strong performance for GPT-4o in Computer Science, achieving a score of 2.38, though it slightly underperformed for Deepseek-32B with a score of 2.37. These results underscore the effectiveness of MC-NEST with different sampling strategies—Greedy, Importance Sampling, and Pairwise Importance Sampling—in optimizing hypothesis generation across domains, outperforming traditional approaches.

4.3.2 Usefulness of the MC-NEST Framework.

MC-NEST is a powerful framework for hypothesis generation, combining MCTS with Nash Equilibrium strategies to dynamically balance exploration and exploitation. It iteratively refines hypotheses through self-critique and validation, ensuring novelty and empirical grounding. In experiments, MC-NEST outperformed baselines across multiple domains, achieving higher BertScore and qualitative metrics (novelty, clarity, significance, and verifiability). For example, in optimizing synthetic peptide sequences for nuclear localization and solubility, MC-NEST proposed experimentally validated modifications. Its ability to incorporate emerging scientific literature and adapt to new discoveries distinguishes it from frameworks lacking iterative refinement. These features make MC-NEST a versatile and effective tool for advancing scientific discovery through automated hypothesis generation.