Teach Ai “I Don’t Know”: A new dataset alleviates the illusion of enhanced decoration
Strengthening the box uses reward signals to guide the large language model to achieve ideal behavior. This approach cultivates the model’s ability to generate logical and structured output by enhancing the correct response. However, the challenge has been ensuring that these models also know when to not respond, especially in the face of incomplete or misleading questions without clear answers.
Problems arise when the language model (enhanced reinforcement fill) begins to lose the ability to refuse to answer unclear or ambiguous query. These models are not signal uncertainty, but tend to produce confident responses that are incorrect. This phenomenon identified in this article as the “illusion tax” emphasizes an increasing number of risks. As models perform better, they may also become more likely to hallucinate when silence is more appropriate. This is particularly dangerous in areas where high trust and precision are required.
Current tools used to train large language models often ignore the importance of rejection behavior. Strengthening the framework tends to reward only the correct answers while punishing the wrong answers while ignoring the situation where effective responses have no answers at all. The reward system used does not adequately enhance rejection, resulting in overconfident models. For example, the paper shows that after standard RFT, the rejection rate of multiple models falls to near zero, indicating that the current training cannot correctly resolve hallucinations.
Researchers at the University of Southern California have developed a comprehensive unanswerable math (SUM) dataset. The sum introduces implicitly unanswered mathematical questions by modifying existing questions by criteria such as lack of critical information or creating logical inconsistencies. The researchers used DeepScaler as the basic dataset and adopted the O3-Mini model to generate high-quality unanswerable questions. This synthetic dataset is designed to teach the model to recognize when the problem lacks sufficient information and respond accordingly.
Sum’s core technology is a mix of answerable and unanswered questions during training. Modify the problem to make it ambiguous or unsolvable while maintaining rationality. The training prompt indicates the model “I don’t know” with irrefutable input. By introducing only 10% of the sum data into the reinforcement box, the model begins to use inference time reasoning to evaluate uncertainty. This structure allows them to reject answers more appropriately without compromising their performance on problems that can be solved.
Performance analysis shows significant improvements. After training with the sum, the rejection rate of the QWEN2.5-7B model increased from 0.01 to 0.73 from the total benchmark, and its rejection rate increased from 0.01 to 0.81 on the UMWP benchmark. On the self-aware dataset, rejection accuracy rose sharply from 0.01 to 0.94. Llama-3.1-8b-Instruct shows a similar trend, with the rejection rate increasing from 0.00 to 0.75 of the sum and the rejection rate of UMWP from 0.01 to 0.79. Despite the benefits of these rejection behaviors, the accuracy on responsive datasets such as GSM8K and Math -500 remained stable, with most variations ranging from 0.00 to -0.05. Minimum declines suggest that rejection training can be introduced in case of significant sacrifices in task performance.
This study outlines a clear tradeoff between improving reasoning and trustworthiness. Powerful enhanced decorations, while powerful, tend to curb cautious behavior. The summation dataset corrects this by teaching models to identify what they cannot solve. Only a small portion has been added to the training data, the language model has become better at determining the boundaries of knowledge. This approach marks an important step to make AI systems not only smarter, but also more honest.
View paper and dataset on hugging face. All credits for this study are to the researchers on the project.
🆕 did you know? Marktechpost is the fastest growing AI media platform with more than 1 million readers per month. Book a strategy call to discuss your campaign goals. Also, please stay tuned for us twitter And don’t forget to join us 95k+ ml reddit And subscribe Our newsletter.
Nikhil is an intern consultant at Marktechpost. He is studying for a comprehensive material degree in integrated materials at the Haragpur Indian Technical College. Nikhil is an AI/ML enthusiast and has been studying applications in fields such as biomaterials and biomedical sciences. He has a strong background in materials science, and he is exploring new advancements and creating opportunities for contribution.
