China's DeepSeek Coder becomes first open-source coding model to beat GPT-4 Turbo

DeepSeek Coder V2, an open-source mixture of experts (MoE) code language model, has shown to outperform state-of-the-art closed-source models, including GPT-4 Turbo, Claude 3 Opus, and Gemini 1.5 Pro. It supports more than 300 programming languages and has achieved top-ranking performance with only a small number of activated parameters2.

When tested on benchmarks such as MBPP+, HumanEval, and Aider, designed to evaluate code generation, editing, and problem-solving capabilities of Large Language Models (LLMs), DeepSeek Coder V2 scored 76.2, 90.2, and 73.7, respectively. This places it ahead of most closed and open-source models, including GPT-4 Turbo, Claude 3 Opus, Gemini 1.5 Pro, Codestral, and Llama-3 70B.

Additionally, DeepSeek Coder V2 has shown strong performance in general reasoning and language understanding tasks. In the MMLU benchmark, which evaluates language understanding across multiple tasks, it scored 79.2. This is better than other code-specific models and similar to the score of Llama-3 70B.

Overall, DeepSeek Coder V2 has demonstrated exceptional proficiency in code generation and understanding, as well as in mathematical tasks, significantly narrowing the performance gap between open-source models and leading AI models like GPT-4 Turbo, Claude 3 Opus, and Gemini 1.5 Pro.

The DeepSeek Coder V2 model introduces several key features that differentiate it from its predecessor:

1. Improved Language Support: The new V2 offering expands language support from 86 programming languages to 338, allowing it to handle a wider range of coding tasks.
2. Increased Context Window: The context window has been increased from 16K to 128K, enabling the model to handle more complex and extensive coding tasks.
3. Enhanced Performance: When tested on various benchmarks, DeepSeek Coder V2 demonstrated improved performance in code generation, editing, and problem-solving capabilities, as well as in mathematical tasks.
4. Mixture of Experts Framework: The model is based on DeepSeek V2, which uses a Mixture of Experts framework. This allows the model to activate only a subset of "expert" parameters (2.4B and 21B for the 16B and 236B parameter options, respectively) to address the tasks at hand while optimizing for diverse computing and application needs.
5. Strong Performance in General Reasoning and Language Understanding: In addition to excelling at coding and math-related tasks, DeepSeek Coder V2 also delivers decent performance in general reasoning and language understanding tasks, as demonstrated by its score on the MMLU benchmark.