Title: Boosting Zero-Shot AI Reasoning with Self-Adaptive Prompting
Introduction:
Recent advancements in large language models (LLMs) have shown great potential in solving complex problems with minimal training. However, generating sample prompts for these models can be challenging. To overcome this, we introduce a new method called Consistency-Based Self-Adaptive Prompting (COSP) that bridges the gap between zero-shot and few-shot reasoning. Additionally, we present Universal Self-Adaptive Prompting (USP), an extension of COSP that works for a wide range of natural language tasks.
Prompting LLMs with their own outputs:
We discovered that LLMs could benefit from their own zero-shot outputs as demonstrations. However, we needed to ensure the reliability of these demonstrations since incorrect demonstrations could lead to inaccurate answers. To select robust self-generated demonstrations, we relied on the LLM’s confidence and consistency in its predictions. By measuring the uncertainty and self-consistency of the model’s answers, we could identify demonstrations likely to be correct.
COSP Method:
1. The unlabeled questions are inputted into the LLM multiple times to generate rationales and answers.
2. The most frequent and consistent answers are selected as pseudo-demonstrations, while repetition is penalized.
3. The pseudo-demonstrations are concatenated into test questions and fed back into the LLM for a final predicted answer.
USP for General NLP Tasks:
USP extends the COSP method to different natural language understanding and generation tasks:
– Classification: Uncertainty is measured through the probability distribution of each class predicted by the neural network.
– Short-form Generation: Similar to COSP, but without the rationale-generating step if not needed.
– Long-form Generation: For open-ended tasks like summarization, model outputs are compared using an overlap metric.
Key Results:
– COSP significantly outperforms the standard zero-shot baseline in arithmetic and reasoning problems.
– USP improves zero-shot performance across various tasks, including classification, short-form generation, and long-form generation.
Conclusion:
COSP and USP provide effective methods for enhancing zero-shot reasoning in AI models. By leveraging the models’ own outputs and measuring their confidence and consistency, we can generate reliable self-generated demonstrations. This approach has the potential to advance AI capabilities in problem-solving and natural language understanding.