Podcast: 𝐇𝐨𝐰 𝐃𝐞𝐞𝐩𝐬𝐞𝐞𝐤 𝐥𝐞𝐯𝐞𝐫𝐚𝐠𝐞𝐝 ‘𝐊𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞 𝐃𝐢𝐬𝐭𝐢𝐥𝐥𝐚𝐭𝐢𝐨𝐧’ 𝐭𝐨 𝐨𝐮𝐭𝐬𝐦𝐚𝐫𝐭 𝐎𝐩𝐞𝐧𝐀𝐈

Website: https://debabratapruseth.com/

There has been a wave of speculation and awe surrounding how Deepseek, a relatively new player, managed to challenge industry giants like OpenAI with its competitive LLM models—achieving comparable performance in a fraction of the time and cost.
One term that frequently comes up in this discussion is “𝑲𝒏𝒐𝒘𝒍𝒆𝒅𝒈𝒆 𝑫𝒊𝒔𝒕𝒊𝒍𝒍𝒂𝒕𝒊𝒐𝒏.” In fact, Microsoft, OpenAI, and even the U.S. government are reportedly investigating whether Deepseek leveraged this technique in ways that might raise ethical or legal concerns.

𝐖𝐡𝐚𝐭 𝐢𝐬 𝐊𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞 𝐃𝐢𝐬𝐭𝐢𝐥𝐥𝐚𝐭𝐢𝐨𝐧?

Knowledge distillation is a machine learning technique where a smaller, more efficient model (the 𝐬𝐭𝐮𝐝𝐞𝐧𝐭) learns from a larger, highly trained model (the 𝐭𝐞𝐚𝐜𝐡𝐞𝐫), rather than directly from raw data. This allows the student model to retain much of the teacher’s intelligence while being significantly more compact and resource-efficient.

𝐇𝐨𝐰 𝐝𝐨𝐞𝐬 𝐢𝐭 𝐰𝐨𝐫𝐤?

1) 𝑻𝒓𝒂𝒊𝒏 𝒂 𝑩𝒊𝒈 𝑻𝒆𝒂𝒄𝒉𝒆𝒓 𝑴𝒐𝒅𝒆𝒍
– A very powerful AI model learns from a huge dataset.
– This model becomes super accurate but is too big and slow

2) 𝑻𝒉𝒆 𝑻𝒆𝒂𝒄𝒉𝒆𝒓 𝑮𝒊𝒗𝒆𝒔 “𝑺𝒐𝒇𝒕” 𝑨𝒏𝒔𝒘𝒆𝒓𝒔
– Instead of just saying “This is a cat” or “This is a dog,” the teacher gives confidence scores. Example: “I’m 85% sure it’s a cat and 5% sure it’s a dog”
-These soft answers help the student model understand relationships between different classes.

3) 𝑻𝒓𝒂𝒊𝒏 𝒂 𝑺𝒎𝒂𝒍𝒍𝒆𝒓 𝑺𝒕𝒖𝒅𝒆𝒏𝒕 𝑴𝒐𝒅𝒆𝒍
– The student model is trained using both 𝒓𝒆𝒂𝒍 𝒅𝒂𝒕𝒂 and the teacher’s 𝒔𝒐𝒇𝒕 𝒂𝒏𝒔𝒘𝒆𝒓𝒔.
– The student learns to mimic the teacher but with fewer resources.

4) 𝑵𝒐𝒘 𝑾𝒆 𝑯𝒂𝒗𝒆 𝒂 𝑭𝒂𝒔𝒕𝒆𝒓, 𝑺𝒎𝒂𝒍𝒍𝒆𝒓 𝑨𝑰!
– The student model is much smaller and faster
– Even though it’s not as big as the teacher, it performs almost as well.

𝐃𝐢𝐝 𝐃𝐞𝐞𝐩𝐬𝐞𝐞𝐤 𝐔𝐬𝐞 𝐎𝐩𝐞𝐧𝐀𝐈 𝐚𝐬 𝐈𝐭𝐬 “𝐓𝐞𝐚𝐜𝐡𝐞𝐫”?

In this case, the suspected 𝒕𝒆𝒂𝒄𝒉𝒆𝒓 𝒊𝒔 𝑶𝒑𝒆𝒏𝑨𝑰, and the 𝒔𝒕𝒖𝒅𝒆𝒏𝒕 𝒊𝒔 𝑫𝒆𝒆𝒑𝒔𝒆𝒆𝒌. This raises an important question: Did OpenAI directly assist Deepseek? The answer is no—but OpenAI’s models and datasets are accessible via APIs that can be purchased. What regulators and industry experts are now scrutinizing is whether Deepseek systematically leveraged OpenAI’s APIs for large-scale knowledge distillation.

In the coming days, we may gain clarity on whether Deepseek’s approach violated ethical AI development guidelines—or if it simply found a clever way to compete with the biggest names in AI.

Until then, enjoy both Deepseek and OpenAI.