Rohan Paul· @rohanpaul_ai · X·· 3 小时前AI 评分42
AI 导读
Sam Altman 谈本地推理。 “从全局资源配置的角度来看，让每个人把大量全球有限的 DRAM 存放在自己的电脑里，却只发挥 5% 的利用率，这是相当糟糕的” 按每个 token 计算，数据中心赢得轻而易举。从任何利用率指标来看，批大小为 1 的本地模型都是一种浪费，而且这也是技术栈中唯一没有人能为你废弃的部分。 https://x.com/morqon/status/2115838856524120128/video/1
正文 · 原文
Sam Altman on local inference.
"from a global allocation perspective, it's quite terrible to have everybody, a bunch of the world's limited supply of DRAM sitting in their own computer and being used 5% of the time"
per token, the datacenter wins easily. A local model at batch size 1 is wasteful by any utilization metric, and it’s also the only part of the stack nobody can deprecate on you. https://x.com/morqon/status/2115838856524120128/video/1
来源：Rohan Paul · x.com