Researchers from UC Berkeley and Anyscale Introduce RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing
What improvements were seen with data augmentation in RouteLLM?
Data augmentation in RouteLLM significantly improved router performance. By augmenting the training data with golden-label datasets or LLM-judge-labeled datasets, the overall distribution of preference data aligned more closely with benchmarks, increasing the benchmark-dataset similarity score and resulting in performance improvements3. This allowed routers to achieve cost savings while maintaining high-quality responses, demonstrating the effectiveness of data augmentation in enhancing routing performance4.
What data sources were used in RouteLLM's training process?
RouteLLM's training process utilized preference data from Chatbot Arena, which consists of prompts and comparisons of response quality between two models. Additionally, data augmentation techniques were employed, including golden-label datasets and an LLM judge, to enhance router performance.
What is the primary goal of RouteLLM?
The primary goal of RouteLLM is to provide a cost-effective and efficient solution for deploying Large Language Models (LLMs) by balancing performance and cost. It achieves this through a routing framework that dynamically selects between stronger and weaker LLMs during inference, optimizing the balance between response quality and cost.