DeepSeek R-1 Model Overview and how it Ranks Versus OpenAI’s O1

DeepSeek is a Chinese AI business “devoted to making AGI a reality” and open-sourcing all its designs. They began in 2023, but have actually been making waves over the previous month approximately, and especially this past week with the release of their 2 newest thinking designs: DeepSeek-R1-Zero and the advanced DeepSeek-R1, likewise known as DeepSeek Reasoner.

They have actually launched not only the models but likewise the code and assessment prompts for public usage, along with a comprehensive paper detailing their technique.

Aside from creating 2 extremely performant models that are on par with OpenAI’s o1 design, the paper has a great deal of valuable info around support knowing, chain of thought reasoning, timely engineering with thinking models, and more.

We’ll start by focusing on the training process of DeepSeek-R1-Zero, which uniquely relied exclusively on reinforcement learning, rather of standard monitored learning. We’ll then move on to DeepSeek-R1, how it’s reasoning works, and some prompt engineering finest practices for thinking models.

Hey everyone, Dan here, co-founder of PromptHub. Today, we’re diving into DeepSeek’s latest model release and comparing it with OpenAI’s thinking models, particularly the A1 and A1 Mini models. We’ll explore their training process, reasoning capabilities, and some crucial insights into prompt engineering for reasoning models.

DeepSeek is a Chinese-based AI company devoted to open-source development. Their current release, the R1 thinking model, is groundbreaking due to its open-source nature and ingenious training methods. This includes open access to the models, prompts, and research study papers.

Released on January 20th, DeepSeek’s R1 accomplished excellent efficiency on different benchmarks, rivaling OpenAI’s A1 models. Notably, they also released a precursor model, R10, which works as the structure for R1.

Training Process: R10 to R1

R10: This design was trained exclusively using reinforcement learning without monitored fine-tuning, making it the first open-source model to achieve high efficiency through this approach. Training included:

– Rewarding right responses in deterministic jobs (e.g., math problems).
– Encouraging structured reasoning outputs utilizing templates with “” and “” tags

Through countless iterations, R10 developed longer reasoning chains, self-verification, and even reflective behaviors. For example, during training, the model demonstrated “aha” moments and self-correction habits, which are uncommon in conventional LLMs.

R1: Building on R10, R1 added several enhancements:

– Curated datasets with long Chain of Thought examples.
– Incorporation of R10-generated reasoning chains.
– Human choice alignment for refined responses.
– Distillation into smaller models (LLaMA 3.1 and 3.3 at various sizes).

Performance Benchmarks

DeepSeek’s R1 design carries out on par with OpenAI’s A1 models across lots of thinking benchmarks:

Reasoning and Math Tasks: R1 competitors or outperforms A1 models in precision and depth of thinking.
Coding Tasks: A1 models usually perform much better in LiveCode Bench and CodeForces tasks.
Simple QA: R1 frequently outmatches A1 in structured QA jobs (e.g., 47% precision vs. 30%).

One significant finding is that longer reasoning chains usually improve efficiency. This lines up with insights from Microsoft’s Med-Prompt structure and OpenAI’s observations on test-time compute and thinking depth.

Challenges and Observations

Despite its strengths, R1 has some constraints:

– Mixing English and Chinese reactions due to an absence of supervised fine-tuning.
– Less polished actions compared to talk models like OpenAI’s GPT.

These issues were attended to during R1’s refinement process, including supervised fine-tuning and human feedback.

Prompt Engineering Insights

A remarkable takeaway from DeepSeek’s research study is how few-shot prompting abject R1’s performance compared to zero-shot or concise tailored prompts. This aligns with findings from the Med-Prompt paper and OpenAI’s suggestions to limit context in thinking models. Overcomplicating the input can overwhelm the model and lower precision.

DeepSeek’s R1 is a considerable step forward for open-source reasoning models, showing capabilities that rival OpenAI’s A1. It’s an amazing time to try out these models and their chat user interface, which is totally free to use.

If you have concerns or wish to find out more, have a look at the resources linked listed below. See you next time!

Training DeepSeek-R1-Zero: A reinforcement learning-only method

DeepSeek-R1-Zero stands apart from many other advanced models because it was trained utilizing just reinforcement learning (RL), no supervised fine-tuning (SFT). This challenges the current standard technique and opens up brand-new opportunities to train reasoning designs with less human intervention and effort.

DeepSeek-R1-Zero is the very first open-source design to confirm that sophisticated thinking abilities can be established simply through RL.

Without pre-labeled datasets, the design learns through trial and error, improving its behavior, specifications, and weights based entirely on feedback from the services it creates.

DeepSeek-R1-Zero is the base model for DeepSeek-R1.

The RL procedure for DeepSeek-R1-Zero

The training procedure for DeepSeek-R1-Zero involved providing the design with different reasoning tasks, ranging from mathematics issues to abstract reasoning challenges. The model produced outputs and was assessed based on its efficiency.

DeepSeek-R1-Zero received feedback through a reward system that assisted guide its knowing process:

Accuracy rewards: Evaluates whether the output is proper. Used for when there are deterministic results (math problems).

Format rewards: Encouraged the design to structure its thinking within and tags.

Training prompt design template

To train DeepSeek-R1-Zero to generate structured chain of idea series, the scientists used the following prompt training template, changing timely with the thinking concern. You can access it in PromptHub here.

This template triggered the design to clearly outline its thought procedure within tags before providing the last answer in tags.

The power of RL in reasoning

With this training process DeepSeek-R1-Zero began to produce sophisticated reasoning chains.

Through countless training actions, DeepSeek-R1-Zero progressed to fix progressively complex problems. It found out to:

– Generate long thinking chains that enabled deeper and more structured problem-solving

– Perform self-verification to cross-check its own responses (more on this later).

– Correct its own errors, showcasing emerging self-reflective habits.

DeepSeek R1-Zero performance

While DeepSeek-R1-Zero is primarily a precursor to DeepSeek-R1, it still achieved high efficiency on several criteria. Let’s dive into some of the experiments ran.

Accuracy improvements during training

– Pass@1 accuracy started at 15.6% and by the end of the training it improved to 71.0%, similar to OpenAI’s o1-0912 design.

– The red solid line represents efficiency with bulk ballot (comparable to ensembling and self-consistency methods), which increased accuracy further to 86.7%, exceeding o1-0912.

Next we’ll look at a table comparing DeepSeek-R1-Zero’s efficiency across numerous reasoning datasets versus OpenAI’s thinking designs.

AIME 2024: 71.0% Pass@1, somewhat below o1-0912 but above o1-mini. 86.7% cons@64, beating both o1 and o1-mini.

MATH-500: Achieved 95.9%, beating both o1-0912 and o1-mini.

GPQA Diamond: Outperformed o1-mini with a score of 73.3%.

– Performed much worse on coding jobs (CodeForces and LiveCode Bench).

Next we’ll take a look at how the action length increased throughout the RL training procedure.

This chart shows the length of responses from the design as the training process advances. Each “action” represents one cycle of the model’s knowing procedure, where feedback is provided based on the output’s efficiency, evaluated using the timely template talked about previously.

For each concern (corresponding to one step), 16 reactions were sampled, and the typical precision was determined to ensure steady assessment.

As training advances, the design produces longer reasoning chains, permitting it to resolve increasingly complex thinking jobs by leveraging more test-time calculate.

While longer chains don’t always ensure better results, they generally associate with improved performance-a pattern also observed in the MEDPROMPT paper (read more about it here) and in the original o1 paper from OpenAI.

Aha minute and self-verification

Among the coolest aspects of DeepSeek-R1-Zero’s development (which also uses to the flagship R-1 design) is just how great the design became at reasoning. There were advanced thinking behaviors that were not explicitly programmed but arose through its reinforcement discovering process.

Over thousands of training steps, the design started to self-correct, review flawed logic, and confirm its own solutions-all within its chain of thought

An example of this noted in the paper, described as a the “Aha moment” is below in red text.

In this circumstances, the model literally said, “That’s an aha minute.” Through DeepSeek’s chat function (their version of ChatGPT) this type of reasoning typically emerges with expressions like “Wait a minute” or “Wait, but … ,”

Limitations and difficulties in DeepSeek-R1-Zero

While DeepSeek-R1-Zero was able to perform at a high level, there were some drawbacks with the model.

Language mixing and coherence issues: The design sometimes produced actions that combined languages (Chinese and English).

Reinforcement knowing compromises: The lack of supervised fine-tuning (SFT) implied that the design lacked the refinement needed for totally polished, human-aligned outputs.

DeepSeek-R1 was developed to deal with these concerns!

What is DeepSeek R1

DeepSeek-R1 is an open-source reasoning model from the Chinese AI laboratory DeepSeek. It constructs on DeepSeek-R1-Zero, which was trained completely with support learning. Unlike its predecessor, DeepSeek-R1 incorporates supervised fine-tuning, making it more refined. Notably, it surpasses OpenAI’s o1 design on numerous benchmarks-more on that later on.

What are the main distinctions between DeepSeek-R1 and DeepSeek-R1-Zero?

DeepSeek-R1 builds on the foundation of DeepSeek-R1-Zero, which serves as the base design. The 2 vary in their training methods and total efficiency.

1. Training method

DeepSeek-R1-Zero: Trained entirely with reinforcement learning (RL) and no monitored fine-tuning (SFT).

DeepSeek-R1: Uses a multi-stage training pipeline that includes supervised fine-tuning (SFT) initially, followed by the very same support finding out process that DeepSeek-R1-Zero wet through. SFT helps enhance coherence and readability.

2. Readability & Coherence

DeepSeek-R1-Zero: Dealt with language mixing (English and Chinese) and readability issues. Its reasoning was strong, however its outputs were less polished.

DeepSeek-R1: Addressed these issues with cold-start fine-tuning, making reactions clearer and more structured.

3. Performance

DeepSeek-R1-Zero: Still an extremely strong reasoning model, often beating OpenAI’s o1, but fell the language blending concerns reduced usability significantly.

DeepSeek-R1: Outperforms R1-Zero and OpenAI’s o1 on many thinking standards, and the reactions are far more polished.

In short, DeepSeek-R1-Zero was an evidence of concept, while DeepSeek-R1 is the fully enhanced version.

How DeepSeek-R1 was trained

To tackle the readability and coherence concerns of R1-Zero, the researchers included a cold-start fine-tuning phase and a multi-stage training pipeline when developing DeepSeek-R1:

Cold-Start Fine-Tuning:

– Researchers prepared a premium dataset of long chains of idea examples for preliminary monitored fine-tuning (SFT). This information was collected using:- Few-shot triggering with comprehensive CoT examples.

– Post-processed outputs from DeepSeek-R1-Zero, improved by human annotators.

Reinforcement Learning:

DeepSeek-R1 underwent the exact same RL procedure as DeepSeek-R1-Zero to improve its further.

Human Preference Alignment:

– A secondary RL stage enhanced the model’s helpfulness and harmlessness, making sure much better alignment with user needs.

Distillation to Smaller Models:

– DeepSeek-R1’s reasoning abilities were distilled into smaller sized, efficient models like Qwen and Llama-3.1 -8 B, and Llama-3.3 -70 B-Instruct.

DeepSeek R-1 benchmark efficiency

The researchers evaluated DeepSeek R-1 throughout a range of criteria and versus leading designs: o1, GPT-4o, and Claude 3.5 Sonnet, o1-mini.

The criteria were broken down into numerous categories, shown below in the table: English, Code, Math, and Chinese.

Setup

The following criteria were applied across all designs:

Maximum generation length: 32,768 tokens.

Sampling configuration:- Temperature: 0.6.

– Top-p worth: 0.95.

– DeepSeek R1 outperformed o1, Claude 3.5 Sonnet and other models in the bulk of thinking criteria.

o1 was the best-performing model in 4 out of the 5 coding-related criteria.

– DeepSeek carried out well on innovative and long-context task job, like AlpacaEval 2.0 and ArenaHard, exceeding all other models.

Prompt Engineering with reasoning models

My favorite part of the post was the researchers’ observation about DeepSeek-R1’s level of sensitivity to prompts:

This is another datapoint that aligns with insights from our Prompt Engineering with Reasoning Models Guide, which recommendations Microsoft’s research study on their MedPrompt framework. In their research study with OpenAI’s o1-preview design, they found that frustrating thinking models with few-shot context broken down performance-a sharp contrast to non-reasoning models.

The essential takeaway? Zero-shot prompting with clear and succinct instructions seem to be best when using thinking models.