The article introduces EDINET-Bench, a novel open-source Japanese financial benchmark designed to evaluate Large Language Models (LLMs) on complex financial tasks. This benchmark addresses the scarcity of challenging Japanese financial datasets for LLM evaluation, crucial for tasks like accounting fraud detection, earnings forecasting, and industry prediction. The EDINET-Bench dataset is automatically compiled from ten years of Japanese annual reports available through the Electronic Disclosure for Investors’ NETwork (EDINET). Initial evaluations indicate that even state-of-the-art LLMs perform only marginally better than logistic regression in some complex financial tasks, highlighting the need for domain-specific adaptation and further research. The project makes its dataset, benchmark construction code, and evaluation code publicly available to foster advancements in LLM applications within the financial sector.

The article introduces AutoThink, an innovative approach designed to enhance the inference efficiency and accuracy of reasoning Large Language Models (LLMs). AutoThink addresses the challenge of LLMs generating excessive or insufficient reasoning tokens, which leads to computational inefficiency and suboptimal performance. This system comprises two main components: a query complexity classifier that dynamically allocates the optimal number of reasoning tokens, and a dataset of control vectors derived from "pivotal tokens" to guide the LLM's reasoning path. Experimental results demonstrate that AutoThink significantly reduces output tokens while substantially improving accuracy on complex reasoning tasks, suggesting a more strategic approach to LLM resource allocation rather than simply increasing computation.

The article introduces System Prompt Learning (SPL), an innovative approach enabling Large Language Models (LLMs) to learn and refine problem-solving strategies through practical experience. This method addresses the current disparity where most developers lack the sophisticated system prompts that make advanced AI assistants so capable. SPL represents a "third paradigm" of LLM learning, augmenting traditional pretraining and finetuning by allowing models to classify problems, apply relevant strategies, and continuously improve these strategies over time. The system maintains a dynamic database of human-readable strategies, demonstrating significant performance improvements across various benchmarks and offering benefits like cumulative learning, transparency, and adaptability. Implemented as an open-source plugin in optillm, SPL offers a practical way to integrate this adaptive intelligence into LLM applications.