{"id":3585,"date":"2025-02-10T20:18:43","date_gmt":"2025-02-10T12:18:43","guid":{"rendered":"https:\/\/techduker.kinsta.cloud\/?p=3585"},"modified":"2025-02-19T22:55:41","modified_gmt":"2025-02-19T14:55:41","slug":"deepseek-v3","status":"publish","type":"post","link":"https:\/\/techduker.kinsta.cloud\/en\/ai\/deepseek-v3\/","title":{"rendered":"DeepSeek V3 model: the strongest open source AI, 18 technology highlights, a must-have for AI players!"},"content":{"rendered":"<p class=\"wp-block-paragraph\">With the rapid development of artificial intelligence technology, there is a growing demand for open source AI models in China.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As one of the most advanced open-source AI models in China, DeepSeek V3 has been recognized as one of the most advanced open-source AI models in China, with its<strong>Unique structure and efficient training techniques<\/strong>The program demonstrates strong reasoning and computational skills.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Compared with traditional models, DeepSeek V3 has made breakthroughs in cost-effectiveness, performance and long context processing capabilities, and has become the focus of great attention from the Chinese open source community and industry. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This post.<a href=\"https:\/\/techduker.kinsta.cloud\/en\/\">Techduker<\/a>We will take you on an in-depth exploration of <strong>Technical Highlights of DeepSeek V3<\/strong>The AI model is a powerful, powerful model that allows you to quickly master the core technology of this powerful AI model.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>DeepSeek V3 Features and Technology<\/strong><\/h2>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"818\" height=\"600\" src=\"https:\/\/techduker.kinsta.cloud\/wp-content\/uploads\/2025\/02\/DeepSeek-V3-2.webp\" alt=\"DeepSeek V3 Features and Technology\" class=\"wp-image-3592\" srcset=\"https:\/\/techduker.kinsta.cloud\/wp-content\/uploads\/2025\/02\/DeepSeek-V3-2.webp 818w, https:\/\/techduker.kinsta.cloud\/wp-content\/uploads\/2025\/02\/DeepSeek-V3-2-300x220.webp 300w, https:\/\/techduker.kinsta.cloud\/wp-content\/uploads\/2025\/02\/DeepSeek-V3-2-768x563.webp 768w, https:\/\/techduker.kinsta.cloud\/wp-content\/uploads\/2025\/02\/DeepSeek-V3-2-16x12.webp 16w, https:\/\/techduker.kinsta.cloud\/wp-content\/uploads\/2025\/02\/DeepSeek-V3-2-600x440.webp 600w\" sizes=\"auto, (max-width: 818px) 100vw, 818px\" \/><figcaption class=\"wp-element-caption\">DeepSeek V3 Features and Technology<\/figcaption><\/figure>\n<\/div>\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/chat.deepseek.com\/\" rel=\"noopener\">DeepSeek V3<\/a> It has gained widespread attention since its release, and its innovative techniques covering decentralized reasoning, Mixed Expert Model (MoE), Multi-Token Prediction (MTP), and efficient FP8 training have made it a leader among open-source AI models.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Its training cost is only US$5.57 million, which is extremely cost-effective compared to the industry average.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The model has a total of 671 billion parameters, with 37 billion parameters activated per token, and the computational cost is dramatically reduced by FP8 mixed precision technology.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In addition, DeepSeek V3 has been optimized specifically for math, programming, and long contextual understanding, allowing it to perform well in a number of benchmark tests.<\/p>\n\n\n\n<p class=\"is-style-sme-alert-remark wp-block-paragraph\">Extended Reading:<a href=\"https:\/\/techduker.kinsta.cloud\/en\/ai\/deepseek\/\">What is DeepSeek? China's New AI Powerhouse: Technology Innovation or Plagiarism?<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>DeepSeek V3: 18 Core Technology Highlights to Enhance Model Performance<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">DeepSeek V3 breaks through the limits of traditional AI models with innovative architecture and efficient training strategies.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The 671 billion-parameter MoE design, Multi Token Prediction (MTP) mechanism, and FP8 Mixed Precision Computing not only improve inference and training efficiency, but also achieve leading performance in the areas of multilingual understanding, long context processing, and mathematical reasoning.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Below is an analysis of each of the 18 core technology highlights.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong><strong>Model Framework<\/strong><\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 1<\/span>: Model of Mega-scale Mixed Expertise (MoE)<\/strong>\n<ul class=\"wp-block-list\">\n<li>A 671 billion parameter Model of Mixed Expertise (MoE) is used, with 37 billion parameters activated per token.<\/li>\n\n\n\n<li>Each MoE layer contains 1 shared expert and 256 dedicated experts with 2048 hidden dimensions within the experts.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 2<\/span>: Multiple Latent Attention (MLA)<\/strong><\/strong>\n<ul class=\"wp-block-list\">\n<li>Reducing KV caching requirements and improving inference efficiency through low-rank union compression.<\/li>\n\n\n\n<li>Design 128 attention heads, each with a dimension of 128 and a KV compression dimension of 512.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 3<\/span>: Load balancing strategy without auxiliary losses<\/strong>\n<ul class=\"wp-block-list\">\n<li>Innovatively eliminates the impact of traditional load balancing strategies on model performance, giving experts more flexibility to specialize in different areas and improving overall performance.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 4<\/span>: Multi Token Prediction (MTP) Training<\/strong>\n<ul class=\"wp-block-list\">\n<li>At the same time, 2 future tokens are predicted to increase training signal density and improve data efficiency.<\/li>\n\n\n\n<li>The predicted success rate of the second token is 85%-95%, which greatly accelerates the decoding speed.<\/li>\n\n\n\n<li>A single-layer MTP module is used to predict additional tokens sequentially, and at the same time ensure that each prediction step maintains a complete causal relationship to ensure contextual coherence.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Efficient Training<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 5<\/span>: FP8+ mixed precision calculation<\/strong>\n<ul class=\"wp-block-list\">\n<li>FP8 is used for computation and storage to reduce GPU memory usage and accelerate training.<\/li>\n\n\n\n<li>Most of the matrix operations (e.g. Fprop, Dgrad, Wgrad) are run under FP8, which is 2x faster than BF16.<\/li>\n\n\n\n<li>High-precision calculations (e.g. embedded modules and MoE gate modules) are reserved to ensure the stability and accuracy of the numerical calculations.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 6<\/span>: DualPipe<strong>+<\/strong> Algorithms to improve training efficiency<\/strong>\n<ul class=\"wp-block-list\">\n<li>Reduces pipeline bubbles and improves GPU utilization by overlapping compute and communication.<\/li>\n\n\n\n<li>Split each block into four components: attention mechanism, all-to-all distribution, MLP, and all-to-all combination, and manually adjust the resource allocation ratio of GPU streaming multiprocessors (SMs) to improve operational efficiency.<\/li>\n\n\n\n<li>Using bi-directional pipeline scheduling, micro-batches are fed simultaneously from both ends of the pipeline, minimizing communication delays and increasing operational efficiency.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 7<\/span>: Extreme Memory Optimization<\/strong>\n<ul class=\"wp-block-list\">\n<li>Recalculated RMSNorm+ and MLA projection to reduce memory usage.<\/li>\n\n\n\n<li>Stores index-weighted average (EMA+) parameters in the CPU to reduce GPU memory pressure.<\/li>\n\n\n\n<li>The MTP module shares the embedding and output layers with the main model, which effectively reduces the memory footprint and improves the overall memory efficiency.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 8<\/span>: Highly stable training<\/strong>\n<ul class=\"wp-block-list\">\n<li>There were no irrecoverable loss peaks, no rollbacks, and a training success rate of 100%.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 9<\/span>: Very low training costs<\/strong>\n<ul class=\"wp-block-list\">\n<li>The training cost was only $5.57 million, totaling 2.788 million H800 GPU hours, which is much lower than the training cost of any known model of its class in the world.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p class=\"is-style-sme-alert-remark wp-block-paragraph\">Extended Reading:<a href=\"https:\/\/techduker.kinsta.cloud\/en\/ai\/deepseek-ai\/\">DeepSeek Beginner's Guide: Even newbies can start from scratch and get started easily!<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Data Processing and Pre-training<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 10<\/span>: High-quality, diverse training data<\/strong>\n<ul class=\"wp-block-list\">\n<li>Pre-training on 14.8 megabytes of tokens, covering multilingualism, math, programming, etc.<\/li>\n\n\n\n<li>Increase the proportion of math and programming samples, and expand multilingual support beyond English and Chinese.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 11<\/span>: File Packaging and FIM Policies<\/strong>\n<ul class=\"wp-block-list\">\n<li>Adoption of file packing technique maintains contextual integrity and avoids cross-sample masking effects.<\/li>\n\n\n\n<li>Introducing the Fill-in-Middle (FIM) strategy, 10% rates use a structured fill-in-middle format with the following structured data: `` pre  suf  middle `'', which improves the model fill-in-middle capability.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 12<\/span>: Multilingual Parser Optimization<\/strong>\n<ul class=\"wp-block-list\">\n<li>Scale vocabulary to 128K tokens using byte-level BPE.<\/li>\n\n\n\n<li>Special tokens containing punctuation and line breaks are also introduced to improve the compression efficiency of multilingual texts.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 13<\/span>: Long Context Extension Technology<\/strong>\n<ul class=\"wp-block-list\">\n<li>Scale context length from 4K to 128K with two-stage training.<\/li>\n\n\n\n<li>Adopts YaRN technology to set<code><code><code><code><code><code><code><code><code><code>`scale = 40, base = 1, factor = 32`<\/code><\/code><\/code><\/code><\/code><\/code>In the case of a long contextual extension, this ensures stability.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p class=\"is-style-sme-alert-remark wp-block-paragraph\">Extended Reading:<a href=\"https:\/\/techduker.kinsta.cloud\/en\/ai\/deepseek-basic\/\">The five basic rules of deepseek and the basic use of the process!<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Post-training and Performance Enhancement<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 14<\/span>: Supervisory Fine Tuning (SFT)<\/strong>\n<ul class=\"wp-block-list\">\n<li>Use 1.5 million instruction fine-tuning samples, covering a wide range of areas such as reasoning, math, programming, and more.<\/li>\n\n\n\n<li>Inference data is also generated through the internal DeepSeek-R1 model, ensuring both accuracy and format clarity.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 15<\/span>: Reinforced Learning (RL)<\/strong>\n<ul class=\"wp-block-list\">\n<li>Optimizing performance on complex reasoning tasks by combining rule-based and model-based incentives.<\/li>\n\n\n\n<li>Grouped Relative Strategy Optimization (GRPO) is used to further enhance the model performance by estimating the baseline through group scores.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 16<\/span>: Knowledge Distillation&nbsp;<\/strong>\n<ul class=\"wp-block-list\">\n<li>Reasoning distillation through the DeepSeek-R1 family of models significantly enhances performance in mathematical and programming tasks.<\/li>\n\n\n\n<li>Model performance is dramatically improved in LiveCodeBench and MATH-500 benchmark tests.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Performance<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 17<\/span>: Leading in many areas of measurement<\/strong>\n<ul class=\"wp-block-list\">\n<li>The model accuracy reaches 85.6% in the MMLU benchmark test and 92.3% in the GSM8K mathematical test.<\/li>\n\n\n\n<li>In addition, the pass rate of HumanEval code generation task has increased by 15%.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong><span style=\"background-image: linear-gradient(transparent 60%, rgba(252, 185, 0, 0.5) 60%)\" class=\"sme-highlighter\">Highlight 18<\/span>::<strong>The results are comparable to the best closed-source models.<\/strong><\/strong>\n<ul class=\"wp-block-list\">\n<li>In the LongBench v2 long context benchmark, the model F1 score is 91.6, which is comparable to GPT-4o.<\/li>\n\n\n\n<li>In the FRAMES benchmarking test, the model demonstrated a far superior ability to handle contexts of 100K token length.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p class=\"is-style-sme-alert-remark wp-block-paragraph\">Extended Reading:<a href=\"https:\/\/techduker.kinsta.cloud\/en\/ai\/chatgpto1\/\">Chat GPT o1 model has an IQ of over 120! The most powerful AI has mastered the human mindset!<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Technical Breakthroughs and Application Value of DeepSeek V3<\/strong><\/h2>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"818\" height=\"600\" src=\"https:\/\/techduker.kinsta.cloud\/wp-content\/uploads\/2025\/02\/DeepSeek-V3-3.webp\" alt=\"DeepSeek V3 Technology Applications\" class=\"wp-image-3593\" srcset=\"https:\/\/techduker.kinsta.cloud\/wp-content\/uploads\/2025\/02\/DeepSeek-V3-3.webp 818w, https:\/\/techduker.kinsta.cloud\/wp-content\/uploads\/2025\/02\/DeepSeek-V3-3-300x220.webp 300w, https:\/\/techduker.kinsta.cloud\/wp-content\/uploads\/2025\/02\/DeepSeek-V3-3-768x563.webp 768w, https:\/\/techduker.kinsta.cloud\/wp-content\/uploads\/2025\/02\/DeepSeek-V3-3-16x12.webp 16w, https:\/\/techduker.kinsta.cloud\/wp-content\/uploads\/2025\/02\/DeepSeek-V3-3-600x440.webp 600w\" sizes=\"auto, (max-width: 818px) 100vw, 818px\" \/><figcaption class=\"wp-element-caption\">DeepSeek V3 Technology Applications<\/figcaption><\/figure>\n<\/div>\n\n\n<p class=\"wp-block-paragraph\">DeepSeek V3 is the leader in open source AI through its innovative MoE architecture, FP8 training technology, and efficient memory management strategies.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It not only reduces training costs, but also demonstrates strengths in areas such as math, programming and long context processing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In the future, DeepSeek V3 is expected to play a greater role in intelligent customer service, AI assistants, program development assistance and other areas, and continue to promote the development of open source AI.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Conclusion<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">DeepSeek V3, a new benchmark in the AI field, has succeeded in taking the lead in open source AI modeling through its innovative technical architecture and efficient training methods.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Its excellent performance, cost-effectiveness, and wide range of applications make it an important technology for AI developers and researchers.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">With the advancement of hardware technology, DeepSeek V3 will be even more desirable in the future, and may become an important engine to promote the development of artificial intelligence!<\/p>","protected":false},"excerpt":{"rendered":"<p>\u96a8\u8457\u4eba\u5de5\u667a\u6167\u6280\u8853\u7684\u5feb\u901f\u767c\u5c55\uff0c\u4e2d\u570b\u5e02\u5834\u5c0d\u958b\u6e90 AI \u6a21\u578b\u7684\u9700\u6c42\u65e5\u76ca\u589e\u9577\u3002 DeepSeek V3 \u4f5c\u70ba\u76ee\u524d\u4e2d\u570b\u6700 [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":3595,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[53,144,52],"class_list":["post-3585","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","tag-ai","tag-deepseek","tag-openai"],"_links":{"self":[{"href":"https:\/\/techduker.kinsta.cloud\/en\/wp-json\/wp\/v2\/posts\/3585","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techduker.kinsta.cloud\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techduker.kinsta.cloud\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techduker.kinsta.cloud\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/techduker.kinsta.cloud\/en\/wp-json\/wp\/v2\/comments?post=3585"}],"version-history":[{"count":10,"href":"https:\/\/techduker.kinsta.cloud\/en\/wp-json\/wp\/v2\/posts\/3585\/revisions"}],"predecessor-version":[{"id":3857,"href":"https:\/\/techduker.kinsta.cloud\/en\/wp-json\/wp\/v2\/posts\/3585\/revisions\/3857"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techduker.kinsta.cloud\/en\/wp-json\/wp\/v2\/media\/3595"}],"wp:attachment":[{"href":"https:\/\/techduker.kinsta.cloud\/en\/wp-json\/wp\/v2\/media?parent=3585"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techduker.kinsta.cloud\/en\/wp-json\/wp\/v2\/categories?post=3585"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techduker.kinsta.cloud\/en\/wp-json\/wp\/v2\/tags?post=3585"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}