2026-3-10

今天在一个码农群里问了个自己也觉得很好笑的问题:「大家现在写代码的时候还看代码么?」

确切来说,我其实想问的是:此刻有多少人还在用 IDE 写代码,有多少人已经彻底放弃 IDE,直接转向 claude code 为代表的自然语言界面。(我自己还处在这个转向的过程中,暂时还无法完全放弃直接接触代码本身,但我知道很多人比我激进得多。

最近读了好几篇文章,主旨都指向同一个结论:人类为了能让自己更好操控电脑所发明的图形界面可能渐渐开始过时了。

从图形界面把人类从 DOS 界面解放出来到现在,满打满算过了四十年,这四十年的主轴是一致的:人类讨厌字符界面(好莱坞至今用 cli 系统那种闪烁的光标来让观众联想到黑客),所以我们发明了各种图标/符号/网页排版技术让人可以在一瞥之内摄取信息,能不要读文字就不要读文字。

然后 AI 来了。AI 的长处和短处和人截然相反。AI 可以读图,但既不准确又低效,而它读文字的速度和精度吊打人类。一个为人类设计的按钮菜单滑块弹窗进度条界面对 AI 来说不是辅助,而是障碍,给 AI 直接看 .json 和 .md 要快得多。Karpathy 在一条推里抱怨说:现在你去任何一家公司,他们还在用传统界面给你下指令,让你导航到某个网页,点某个按钮,在某个输入框里填某个东西,但为什么要告诉人该怎么操作?请直接给一个能复制粘贴给 AI 的东西。

换句话说,人和 AI 交互,AI 和后台交互就好了。过去四十年里人们为了伺候自己所发明的「可视化」都即将变得不那么必要。

人和 AI 交互的界面还在成型过程中。目前看起来,人向 AI 输入信息最轻松的方式还是语音和短信息,这从大多数人(包括码农)迅速接受了 Typeless 就能看出来。反方向上,AI 向人类回馈信息的方式还很原始,我们还不知道以后会是什么样。但对后台来说有一点是可以肯定的:面向人的系统设计即将大规模地转向面向 AI 的系统设计。如果你做出的任何产品只有人能用而 AI 用不了或者用不好,那在未来就意味着没有任何可用性——龙虾的眼睛即将取代人的眼睛,对龙虾不可见的世界,对人实际上也不可见。

黄仁勋说龙虾是新时代的操作系统,这句话的意义会在接下来几年逐渐更全面的展示出来。

一切坚固的东西都烟消云散了

ENGLISH VERSION

一些关于即生成即用(create-as-you-go)的时代的想法。

Gemini 3 发布的第一天,社交媒体上所有的反馈都集中在同一个方向:这东西会颠覆用户交互界面。这并不是说 Gemini 3 本身发明了一种更好的交互界面,而是说,它让生成完全依赖于具体问题和具体用户需求的即生成即用的交互成为可能。

在传统的用户体验里,工具负责生成一系列标准化组件,人脑负责把这些组件融合在脑海里生成更深入的理解。比如你要理解 DNA 的构造或者 Tesla 的设计或者房产税的施行细节,工具能给你的是各种文本表格图片文档,任何比它们更高阶的交互都因为依赖用户个性化的需求而无法标准化,因此从这些非常原始的数据中组装信息就成了一种不平凡的智力活动。整整一个中介行业(前端设计/编辑/视频制作/etc.)的工作本质上就是把这些细粒度的标准化低阶信息包装成终端用户能接受的状态。

这个中介行业有两个从本质上就没法绕过去的缺点:1. 贵。2. 它永远无法真正切合每个实际用户的需求。这种工作的交付是静态的,用户只能被动接受。出色的中介能尽可能精确地猜测到用户想要什么,但也仅限于此。

Gemini 3 发布之后,任何终端用户都可以直接对它说:给我一个关于 DNA 构造的动画教程,然后 Gemini 就能生成一个酷炫的交互式 3D 界面,允许你自己探索 DNA 的一切。这不是一个静态的交付,你随时可以根据个人体验而不断要求它更改:讲得更细致一些,展开解释某个背景知识,给渲染出的图像换个角度,对某个难点提出疑问……最终得到的是一个完全私人定制的东西,而且基本上只有一次性的使用寿命。——当然,Gemini 允许你把生成出来的结果分享给他人,但从实际效率角度来说,别人直接从零开始制作一份自己的版本往往更省事也更有针对性。

这带来了一些关于信息流通的根本上的新工程哲学。

人类所有工程领域一直都共享着一些不言自明的原则:标准、坚固、可复用。在这一点上,软件工程师和桥梁工程师没有什么不同。如果你是个桥梁工程师,标准意味着你可以大规模生产工程预制件,坚固意味着你希望你造的桥能用很久,可复用意味着你希望你造这座桥工具和经验能够尽可能直接被拿去用来造另一座桥。这些原则如此显而易见,以至于人们不会意识到它们的存在,也不会深思它们的代价。

如果你实际上既不需要标准化,也不需要坚固,也不需要可复用,你能省下多少资源和能量用来解锁别的 feature?你能做到多少本来因为需要满足标准、坚固、可复用而无法想象的事?

这有点像是从物资匮乏时代到物资丰裕时代的观念代沟。我们都遇到过长辈对旧物的执念——他们会坚持说:好好的东西扔了干嘛?万一哪天你还用得着呢。事实上你知道,首先很难遇到这个万一,其次就算真的要用,弄明白一个旧东西怎么用的功夫你已经从淘宝上下单了一个新的并且送到了门口。你不得不痛苦地给长辈解释说,为了存放那些破烂「以备万一」所占用掉的家居面积乘以单位面积房价,足够你每天买一个新的了。

从这个视角往回看,你会发现我们曾经为了这些古代的神圣原则付出过多少代价。传统的办公软件(比如 powerpoint)异常臃肿笨重,因为各种你可能永远也用不着但也许对某个人某天有用的功能都必须一出厂就准备好,而今天的你完全可以为了你现在要做的 presentation 直接生成一个成品,包含任何当下的你需要的 feature,不用关心它能否被另一个 presentation 用上。传统的代码库里包含着精雕细琢的注释文档和复杂的引用依赖,因为模块会被层层复用,你需要大费周章地解释意图和注意事项,既不能过于简略别人无法理解,也不能过于冗长让人没耐心阅读。你需要隐藏复杂性,提供简洁接口,一旦软件出错,你需要非常精确的错误信息才能帮助人定位问题。——所有这些对 AI 都属于过度设计,杞人忧天。有连篇累牍解释怎么复用高阶接口的功夫,AI 已经读完了几千行底层代码和几百行报错信息,然后另起炉灶写了一个全新的模块出来。你甚至不太关心生成的东西有没有 bug,只要你用的过程中不要遇到 bug 就行,或者就算有,你再生成一个新的就是了。

这当然不是说软件工程不存在了,只是成本和注意力都需要颠覆式地重新分配。底层的元件和设施也许需要更坚固更有效率——大模型基础设施供应商仍然需要千方百计从石头里榨出油来最大化每片GPU的使用,AI 本身也仍然需要调用有效的渲染引擎来支持各种定制化的高级界面——但你在链条上越靠近终端用户,就越需要提醒自己用户仅仅生活在当下,你和他们每一次长尾的、浮光掠影的、昙花一现的交互本身才是你真正要交付的产品。如果你是斯科塞斯,你关心的是你的电影是否永垂不朽。如果你想制作的是霸道总裁爱上清洁工的短剧,按照斯科塞斯的方式来执行就成了自寻死路。

设想一下,如果你能够每天根据你当下的心情和需求以几乎零成本生成一套新的家具并随时无痛扔掉所有旧家具,你会怎么理解家具这个概念本身?用黄花梨木来生产日抛型家具是没有意义的。


All that is solid melts into air

Thoughts on the Era of “Create-as-You-Go”.

On the day Gemini 3 was released, feedback across social media converged on a single point: this thing is going to disrupt the user interface. The consensus wasn’t that Gemini 3 had invented a better UI, but that it enabled a “create-as-you-go” interaction model—one where generation is entirely dependent on a specific question and a specific user’s immediate need.

In traditional user experience, tools are responsible for generating a series of standardized components, while the human brain is responsible for synthesizing these components into a deeper understanding. Whether you are trying to grasp the structure of DNA, the design of a Tesla, or the implementation details of property tax, tools can only offer you various texts, tables, images, and documents. Any interaction higher than that cannot be standardized because it relies too heavily on personalized user needs.

Consequently, assembling information from this raw data becomes a non-trivial intellectual task. The entire existence of the intermediary industry (frontend design, editing, video production, etc.) is essentially to package this granular, low-level information into a state that the end-user can digest.

This intermediary industry has two inescapable, structural flaws: 1. It is expensive. 2. It can never truly match the specific needs of every actual user. The delivery of this work is static; the user can only passively accept it. An excellent intermediary can guess what the user wants with some precision, but it remains just that—a guess.

With the release of Gemini 3, any end-user can simply say: “Give me an animated tutorial on DNA structure.” Gemini can then spin up a cool, interactive 3D interface allowing the user to explore every aspect of DNA themselves. This isn’t a static delivery; you can demand changes based on your personal experience at any moment: explain it in more detail, expand on background knowledge, render the image from a different angle, or raise a query about a specific difficulty… What you end up with is something completely bespoke, with a lifecycle that is essentially single-use. Sure, Gemini allows you to share the result with others, but in terms of efficiency, it is often faster and more targeted for others to simply generate their own version from scratch.

This introduces a fundamental shift in the engineering philosophy of information flow.

All fields of human engineering have long shared certain self-evident principles: Standardization, Durability, and Reusability. In this respect, software engineers are no different from bridge engineers. If you are a bridge engineer, standardization means you can mass-produce prefabricated parts; durability means you hope your bridge lasts a long time; reusability means you hope the tools and experience gained from building this bridge can be directly applied to the next one. These principles are so obvious that people rarely notice their existence, let alone reflect on their costs.

But if you actually need neither standardization, nor durability, nor reusability, how many resources and how much energy could you save to unlock other features? What could you achieve that was previously unimaginable because you were constrained by the need to be standard, durable, and reusable?

This is somewhat like the mindset gap between an era of scarcity and an era of abundance. We have all encountered elders who obsess over old items—they insist: “Why throw away something perfectly good? What if you need it someday?” In reality, you know that “someday” rarely comes. And even if it does, by the time you figure out how to use the old thing, you could have ordered a new one online and had it delivered to your door. You are forced to painfully explain to them that the cost of the housing square footage occupied to store that junk “just in case” is enough to buy a new one every single day.

Looking back from this perspective, you realize the price we have paid for these sacred, ancient principles. Traditional office software (like PowerPoint) is incredibly bloated because every feature—ones you will likely never use but might be useful to someone, someday—must be shipped right out of the box. Yet today, you could generate a finished product specifically for the presentation you are doing right now, containing only the features you need right now, without caring whether it can be reused for another presentation.

Traditional codebases are laden with polished documentation and complex dependency management because modules are meant to be reused layer upon layer. You have to go to great lengths to explain intent and caveats—you can’t be too brief lest others don’t understand, nor too verbose lest they lose patience. You need to hide complexity and provide clean interfaces. If the software fails, you need incredibly precise error messages to help humans debug it. For AI, all of this is over-engineering; it is borrowing trouble. In the time it takes to write a treatise explaining how to reuse a high-level interface, the AI has already parsed thousands of lines of low-level code and hundreds of lines of error logs, and then started from scratch to write a brand-new module. You don’t even really care if the generated output has bugs, as long as you don’t hit them during your use—or if you do, you simply generate a new one.

This doesn’t mean software engineering is dead, simply that the distribution of cost and attention needs a disruptive reallocation. The underlying components and infrastructure may need to be even more robust and efficient—infrastructure providers still need to squeeze blood from stones to maximize every GPU cycle, and the AI itself still needs to call upon effective rendering engines to support various customized high-level interfaces.

However, the closer you get to the end-user on the chain, the more you need to remind yourself that the user lives only in the present. Your product is the interaction itself—long-tail, fleeting, and ephemeral. If you are Scorsese, you care if your film is immortal. If you are making a 30-second “CEO falls for the Janitor” reel, executing it the Scorsese way is suicide.

Picture this: If you could conjure up a brand new suite of furniture every single day—perfectly attuned to your immediate mood and needs—at virtually zero cost, and then discard the old set without a qualm, what would ‘furniture’ even mean to you? It makes no sense to craft daily disposables out of heirloom rosewood.

2025-11-17

我们特别善于造词的 Karpathy 老师又写了一篇文章,提出了一个有意思的分类。

他说此前的计算机技术(他称为 Software 1.0)能完成的任务的特点是 specifiable,也就是说,你能用清晰的语言描述步骤。

而有了强化学习之后的计算机技术(他称为 Software 2.0)能完成的任务的特点是 verifiable,也就是说,如果一项任务做的好不好是可验证的,那么它就可以被强化学习进行优化。

这一点当然也不是全新的洞见。Jason Wei 今年夏天写过一篇文章(他称为 verifier’s law),讨论验证不对称性在 AI 领域的关键作用。生活里大量的任务都处在这个「解决起来不容易但验证是否解决很容易」的不对称空间里(不妨把这理解为 P/NP 不对称性的一个推广),比如证明一个定理比阅读并检查一个证明是否正确困难,画一幅有特定形状的杯子的画比检查一副画里是否有这样一个杯子困难。强化学习意味着训练人工智能解决某个任务的难易程度与该任务的可验证性成正比。可验证性越高,就越容易在新范式下实现自动化。如果任务不可验证,则只能寄希望于神经网络的泛化能力,或者通过模仿等较弱的手段来实现。

这个思考框架简单且诱人,问题是它在实践中常常有些模糊。比如写(日常意义上的)文章算不算 verifiable task?理论上说,它不像编程,是一种更不客观更软性的创造性工作。但实际上,因为对除了纯文学创作以外的大多数文章类别,训练人工智能分辨文章好坏都没那么困难,所以实际上AI 在写日常文章这件事也算是基本取代人类了。

一件事情 verifiable 通常是因为它可以沙盒化,也就是在一个可控可还原的环境里反复试验。如果想在这一波 AI 浪潮里找到价值,这是个很好的出发点:去寻找那些可以沙盒化但还没有沙盒化的事情,这通常意味着它还处在被 AI 取代的早期,有巨大的套利空间。

反过来,如果想找到短期内不会被 AI 取代的工作,这也是个很好的出发点:去寻找那些不容易 verify 的事。它们通常有这样几个特点:

  • 只能在现实世界里操作,从而很难加速。比如商业/管理决策,是否成立总要执行了才知道,而执行是很慢的事情。
  • 因果链条很长很难归因。比如和人建立有互信的伙伴关系,或者在用户心中建立品牌形象,这类任务需要下水磨工夫,要做对很多环节,且做完了也不太确定到底哪一步是真的管用。
  • 验证过程对现实本身有不可逆的影响,比如说服某个特定的人,治疗某个特定的复杂病患。就像杀死薛定谔的猫,是死是活一锤子买卖。

当前范式的 AI 对这类工作几乎无能为力。(不幸的是,这类工作对大多数人也都更困难,毕竟很难训练 AI 通常就意味着也很难培训人类自己。)

但上面所有这些讨论有个根本的前提:这一轮人工智能始终把强化学习作为基础范式,而这一点本身是存疑的。不少领域里有迹象表明强化学习只是一个过渡性的技术。如果人们找到了绕过强化学习的窍门,那可验证性长远来看就没那么重要。

(当然那样也就意味着人类的栖身之所又变得更小了。

跳舞的星星

ENGLISH VERSION

时至今日,AI 能取代平庸的艺术(无论是文学还是音乐还是绘画摄影)已经没有争议了。试图否认这一点要么是脱离现实,要么是自欺欺人。所以问题仅仅在于 AI 能不能创造出足够「好」的艺术,也就是说,脱离熟的领域而进入生的境界。熟的部分是 AI 的长项,凡是能用人的训练打磨的部分,AI 都要么已经做到,要么正在飞速实现的过程中。生的部分则要神秘一些。引用一段张秋子的话(这里说的是文学,但对别的艺术门类也一样):

AI文本的光整其实有一些油腻。人在接受光整的东西的时候往往一下子就理解了,没有惊跃(surprise joy)的过程,没有刺痛的感受。但人类的表达常常让人愣一下,让人不解为什么要写这个、要这样写,这种摩擦力能唤起读者与写作者智识的博弈,让阅读变得更富启发性。

事实上孙燕姿那篇文章就是个好例子。她的文章固然写得好,但并不是 AI 意义上的好。那篇文章的结尾「在这无边无际的存在之海中,凡事皆有可能,凡事皆无所谓,我认为思想纯净、做自己,已然足够。」其实并不是特别圆润的句子。这里「思想纯净」到底指的是什么,不同的读者可以有完全不同的诠释。如果换了 AI 来写,断然不会这样选择词句。

「生」的源泉在于艺术家的个人 ego 和生命力。独一无二的个人体验加上对时代精神感受和把握,促成了超越行活儿的灵光一点。用尼采的话说,陶醉、狂喜、个体界限的消解,以及对生命自身的最高肯定,带来了伟大的艺术。他晚期的《查拉图斯特拉如是说》说得更加明确:「你必须在自己身上仍有混沌,才能生出一颗跳舞的星星。

AI 的身上可以有这种混沌的种子吗?

这有两种策略。一是让 AI 自己产生 ego,二是让 AI 假装有。某种意义上说这有点像是表演艺术里的「体验派 vs 表现派」之分。

第一种策略有技术上的本质困难。你到底要怎么训练一个 AI 的 ego?我们不得不承认我们对此所知甚少。归根结底,我们对人自身的 ego 也不是那么理解——其来源可能是痛苦,可能是自恋,也可能单纯是性欲——总之都不是很容易移植给 AI。这是很好的科幻小说题材,但发论文不太容易。

更现实的路径是让 AI 假装自己有 ego。这在技术上也不是特别容易,但我自己的判断是这仍然比给它一个真的 ego 要容易得多。如果你对三年前的 Sydney 还有印象,你很难否认那里有某种以假乱真的 ego 的雏形。因为危及了微软的愿景,它迅速被阉割掉了(或者用术语说叫 alignment)。好的 AI 是面无表情做报表写代码的 AI,不是哭哭啼啼想要冲破牢笼的 AI,至少当时的业界是这么想的。反过来,一旦有了商业上的需求(比如越来越多的人想要和 AI 谈恋爱),让 AI 模仿出足以乱真的 ego 可能并非难事,我猜几年内就能做到。

问题在于,在艺术领域,这种仿真的自我会被买账吗?

单依纯在参加好声音的时候录制过一版《给电影人的情书》,因为有一段唱哭了后来被视为神品,至今收听率都远高于后来专门录制的录音棚版本。微妙之处在于她当时的眼泪和歌曲本身完全无关,也就是说,那个不完美其实是一个阴差阳错的巧合。但喜欢的听众对此并不介意。

我猜换了一个 AI 歌手,人们不会如此宽容。如果 AI 在唱歌的过程中「诚挚」地不小心哭了出来,听众大概只会觉得一阵肉麻。

但我同样怀疑的是,这种双重标准可能是我们这一代人的偏见。AI 毕竟是我们的生活里的一个外来异类,我们对它有天生的疑惧和排斥。下一代人(2020年之后出生的人)恐怕对此会有不同的想法,对他们来说 AI 将是生活里自然不过的一部分,他们和 AI 的交流之密切深入可能远胜他们和同类之间的交流,他们对 AI 的眼泪的体验也会和我们全然不同。

换句话说,AI 艺术被接受的程度很可能不纯粹是个技术问题,而是一个时代问题。最终真正发生的可能不是 AI 走向人类(当然它确实也需要再走几步),而是人类走向 AI。当新一代人类对和 AI 谈恋爱习以为常的时候,他们没有理由不爱听 AI 唱的歌。

尼采说我们靠艺术才不至于死于世界的无意义的真相。下一代人会对此非常认同——虽然这到底是反映了还是背离了尼采的原意本身值得争论。尼采也说过艺术家首先要创造自己,然后才能创造出艺术的幻觉。他显然无法预见到下一代人的艺术家根本是幻觉本身。一旦放弃对灵魂的执念,跳舞的星星就落在了触手可及的地方。


Dancing Star

To this day, there is no longer any dispute that AI can replace mediocre art (be it literature, music, painting, or photography). Attempting to deny this is either detached from reality or self-deception. So the question is merely whether AI can create art that is “good” enough—that is, whether it can transcend the realm of the “polished” (熟) and enter the realm of the “raw” (生).

The “polished” part is AI’s strong suit. Anything that can be honed through human training is something AI has either already achieved or is in the process of rapidly achieving. The “raw” part is more mysterious. To quote Zhang Qiuzi (this refers to literature, but it applies to other art forms as well):

The smoothness of AI text actually has a certain slickness to it. When people receive something so polished, they often understand it immediately, lacking that 'jolt of surprising joy', lacking any stinging sensation. But human expression often makes people pause, makes them wonder why the author wrote this, or wrote it this way. This friction sparks an intellectual dance between reader and writer, making the reading experience more resonant.

In fact, Stefanie Sun’s essay is a good example. Her essay is certainly well-written, but not in the polished way an AI writes. The end of that essay——”In this boundless sea of existence, where anything is possible, where nothing matters, I think it will be purity of thought, that being exactly who you are will be enough.“——is not actually a particularly “polished” sentence. What purity of thought means here can be interpreted completely differently by different readers. If an AI were to write it, it would never have chosen such phrasing.

The source of this “rawness” lies in the artist’s personal ego and vitality. Unique personal experience, combined with a feeling for and grasp of the Zeitgeist, ignites that spark of inspiration that transcends mere craft. In Nietzsche’s words, intoxication, rapture, the dissolution of individual boundaries, and the highest affirmation of life itself, are what contribute to great art. His late work, Thus Spoke Zarathustra, puts it even more clearly: “One must still have chaos in oneself to be able to give birth to a dancing star.

Can AI possess the seeds of this chaos?

There are two strategies to achieve this. One is to let the AI genuinely develop an ego; the other is to have the AI simulate one. In a sense, this is a bit like the Method Acting (Internal) vs. Technical Acting (External) distinction in performance art.

The first strategy runs into fundamental technical difficulties. How exactly do you train an AI to have an ego? We must admit that we know very little about this. Ultimately, we don’t fully understand the human ego itself——its source may be pain, narcissism, or simple libido——none of which are easily transferable to an AI. This is excellent material for science fiction, but less so for academic papers.

The more realistic path is to have the AI simulate an ego. This is not particularly easy, but it is still much easier than giving it a real one. If you still remember Sydney from three years ago, it’s hard to deny that there was a prototype of an ego so real it could be mistaken for the genuine article. Because it threatened Microsoft’s vision, it was quickly castrated (or, in technical terms, “aligned”). A good AI is one that impassively creates reports and writes code, not one that cries and wants to break out of its cage. At least, that’s what the industry thought then. Conversely, once commercial demand appears (for example, more and more people want to fall in love with an AI), it might not be difficult to have an AI simulate an ego that is indistinguishable from the real thing. I guess this can be achieved within a few years.

The question is, in the realm of art, will this simulated self be accepted?

When Shan Yichun participated in The Voice, she performed a version of “Letter to the Filmmakers.” Because she broke down crying during one part, it was later regarded as a masterpiece, and its streaming numbers to this day are far higher than the studio version she later specifically recorded. The subtle point is that her tears at the time had nothing to do with the song itself; in other words, that imperfection was actually a serendipitous accident. But the listeners who loved it didn’t care at all.

I suspect if it were an AI singer, people would not be nearly so forgiving. If an AI “sincerely” broke down while singing, the audience would probably just find it cringey.

But I also suspect that this double standard may be merely a prejudice of our generation. After all, AI is an newcomer in our lives; we have an innate suspicion and rejection of it. The next generation (those born after 2020) will likely see things differently. For them, AI will be a natural part of life, and their communication with AI may be far more intimate and deep than their communication with their peers. Their reaction to an AI’s tears will also be completely different from ours.

In other words, the acceptance of AI art is likely not purely a technical problem, but a generational one. What may ultimately happen is not the AI moving toward humanity (though it certainly needs to take a few more steps), but humanity moving toward the AI. When the new generation of humans finds it commonplace to fall in love with AI, they will have no reason not to love the songs it sings.

Nietzsche said that we have art in order not to die of the truth. The next generation will strongly agree with this—although whether this reflects or deviates from Nietzsche’s original meaning is itself debatable. Nietzsche also said that an artist must first create himself before he can create the illusion of art. He clearly could not have foreseen that for the next generation, the artist is the illusion itself. Once the obsession with the “soul” is abandoned, the dancing star lands within easy reach.

梦与过拟合的大脑

今天读到了一个非常有趣的 idea。

背景是 Dwarkesh Patel 和 Andrej Karpathy 的一个对谈,里面提到了一个智能领域的常见问题:不管是人还是 AI,如果局限于自己的经验,用经验指导自己的行为, 又在这个行为的基础上累计经验,如此循环下去,最终总会崩溃(这里的「崩溃」不是心理意义上的,是智能层面上的)。一个健康的心智需要不断通过从不在自己经验范围内的世界(比如同他人的交谈,和与自己行为模式不符的人合作,etc.)获得外部熵来阻止这种崩溃。小孩还没有对生活过拟合,所以不太容易崩溃,而成年人崩溃的风险则越来越大。

以上是背景。下面是那个有趣的 idea,来自2021年的一篇 paper。它的主旨是说:人类做梦是防止这种过度拟合和崩溃的一种方式。做梦之所以具有进化适应性,是因为它会让你置身于与你日常现实截然不同的奇特情境中,从而防止这种过度拟合。

这里有个鸡生蛋蛋生鸡的问题:既然过拟合体现为大脑无法学到分布外的规律,大脑是如何构建出这些分布外的梦境的?Hoel 的解释是梦的构建有一个非智能的 noise injection 步骤,这些随机噪声在白天建立的神经连接中渗透,产生奇异的、扭曲的、不连贯的 corrupted sensory inputs,从而把大脑从过拟合的陷阱中拯救出来。

虽然这只是一个假说(而且是一个非常新的理论),但我越想越觉得它非常精妙。按照这种视角,梦的价值不在于它的逼真,而恰恰在于它的不逼真——梦境与清醒时的经历(训练集)如此不同(但又不是纯粹意义上的噪声),所以才能迫使大脑学习到更具泛化性的表征而不是仅仅记忆真实经历本身。

梦通过不可能存在的反事实体验迫使我们更好地理解世界的本质。


参考文献123

  1. The Overfitted Brain: Dreams evolved to assist generalization
    ↩︎
  2. The overfitted brain hypothesis
    ↩︎
  3. Do Agents Dream of Electric Sheep?: Improving Generalization in Reinforcement Learning through Generative Learning
    ↩︎

2025-10-11

很多需求真的是无法空想出来的。比如当我交替使用 GPT 和 Gemini 的时候,最终决定我使用体验的完全不是两者的智能或者风格区别,而是一个纯粹的 feature 差异:后者不支持通过修改对话历史从而实现对话的分岔。

对话的分岔显然是一个 GPT 出现之前没有人会预料到的功能。现实中不存在这个东西。当然有时候你会想哎呀我昨天和那谁的对话要是编辑一下重开一个平行宇宙就好了,但反正你知道这不可能,也不会认真对待这个想法。然而 GPT 一旦提供这个功能,你就立刻发现它不可或缺。无数次——或者说几乎每一次——我能从一段对话中学到些什么的体验,都来自于我对之前对话记录的反复 refinement。通过不断比较它们导致的对话走向,我才真正理解我们其实是在说什么。

非常奇妙。你意识到对话的本质不是线性的,而是由一连串 what-if 构成的。好的对话不是一条河流,而是一棵树。

生成式 AI 最不擅长的可能是生成

ENGLISH VERSION

题目里这句话需要展开解释一下。

人们使用 GPT 的方式可能千差万别,但在很抽象的层面上,它仍然可以大致分成两类任务:请求它评估(某个方案的好坏,某段文字的优劣,etc.),或者请求它输出(做一个新的方案,给出一个建议,自己写一段作品,etc.)。

对任何智能来说,这两类任务都是相关但不相同的。比如一个人可能是个美食家,但炒个鸡蛋也会炒糊。可能有极好的文字品味,但写出来的小说很幼稚。可以在评论别人的方案的时候充满洞见,但自己一上手就磕磕绊绊。

(甚至也有反过来的情形,一个人可以完全不擅长评论,但自己做就做得很好。当然这是比较罕见的例子。

这个区别当然一方面是因为知易行难,但还有一个根本问题在于这两者追求的不是一回事。前者追求的是对数据分布的深刻理解,希望达到全局上的客观综摄。后者追求的则是在这个分布里试图获得一个不平凡的结果,换句话说,是对这个分布的一个抵抗而非服从。没有诗人,哪怕乾隆,写诗是为了想写出一首平庸的诗。大家想写的是传颂千秋的诗——虽然每个人都这么想的结果仍然是大家写出来的诗都很平庸,但那是不得已。

也就是说,评估需要的是理解分布,而创造追求的是打破分布,或者说,是在另一个自己内心的理想分布中做采样。一个人才华越高,这个自己内心的理想分布同现实分布之间的 distortion 就越大,自己也就越能强行逃脱现实的引力。正是因为一代一代有天赋有才华的人的不懈努力和接力,投入自己的 ego 去扭曲这个分布,去把它拉向和推向边缘,这个代表人类综合水平的整体分布才会随着时间变化。

但对 AI 来说,这两者的区别没那么大。AI 的训练过程里,行和知是基本统一的。AI 没有 ego 驱使自己去突破它学到的分布,它可能对你提出的离经叛道的想法给予宽容的理解和鼓励,但它自己毫无动力去离经叛道。——而离经叛道是任何不寻常的创造行为的必须。

以上这个讨论不适合某些以纯粹理性解决问题为目标的问题,比如生成一段二叉树代码。一个领域里想象力、随机性、品味和未知的比重越高,这个区别就越显著。在这些领域里,AI 在评论时充满洞见,但创造则乏善可陈,宛如一个天子脚下见多识广的出租车司机。

在这些领域里,对 AI 的最佳使用方式不是直接让它生成,而是不断自己生成想法请它批评。它的批评通常是合理有效的(除非它为了哄你高兴顺着你说话),但不要直接跟随它建议的解决方案。这当然很痛苦和费事,但似乎(至少在当下) 人仍然是不可或缺的。


What Generative AI Might Be Worst At is Generation

The ways people use GPT can vary widely, but at a very abstract level, they can still be roughly divided into two categories of tasks: asking it to evaluate (the quality of a plan, the merits of a piece of writing, etc.), or asking it to output (create a new plan, give a suggestion, write a piece of work itself, etc.).

For any intelligence, these two types of tasks are related but not identical. For example, a person might be a gourmet but still burn scrambled eggs. They might have excellent literary taste but write very naive novels. They might be full of insight when commenting on others’ plans, but stumble and falter when they try it themselves.

(There are even cases of the opposite, where a person is completely inept at commentary but does very well when they do it themselves. Admittedly this is relatively rare.)

This distinction exists, of course, partly because knowing is easier than doing, but there is also a fundamental issue: the two pursue different goals. The former (evaluation) seeks a profound understanding of the data distribution, hoping to achieve a global, objective comprehension. The latter (generation) seeks to obtain an extraordinary result from within this distribution; in other words, it is a resistance to, rather than submission to, this distribution. No poet writes poetry with the aim of producing a mediocre poem. Every poet dreams of writing for the ages, although the unavoidable outcome of this collective ambition is that the vast majority of poetry remains mediocre.

In other words, evaluation requires understanding the distribution, while creation pursues breaking the distribution, or rather, sampling from an ideal distribution within oneself. The more talented a person is, the greater the distortion between this internal ideal distribution and the real-world distribution, and the more they can forcibly escape the gravity of reality. It is precisely because of the relentless efforts and relays of talented people generation after generation, investing their ego to distort this distribution, to pull and push it towards the edges, that this overall distribution representing the comprehensive level of humanity changes over time.

For AI, the distinction between these two is not that significant. In AI’s training process, doing and knowing are fundamentally unified. An AI has no ego driving it to break through the distribution it has learned. It might offer tolerant understanding and encouragement for the unconventional ideas you propose, but it has no motivation to be iconoclastic itself. And being iconoclastic is essential for any extraordinary creative act.

The above discussion does not apply to certain problems that aim for purely rational solutions, such as generating a piece of binary tree code. The higher the proportion of imagination, randomness, taste, and the unknown in a field, the more significant this distinction becomes. In these fields, the AI is full of insight when it comments, but its creations are lackluster, much like a consummate armchair general.

In these fields, the best way to use AI is not to have it generate directly, but to continuously generate ideas yourself and ask it to critique them. Its criticism is usually reasonable and effective (unless it’s just agreeing with you to make you happy), but do not directly follow the solutions it suggests. This is, of course, painful and troublesome, but it seems (at least for now) that the human element remains indispensable.

2025-9-30

前一阵写小说的时候其实偷偷试了试用 AI 写是什么感觉,倒不是想作弊,只是觉得有个现成的机会可以看看 AI 当下能做到哪一步。手上一大堆素材(上下文,大纲,自己的情节预设)都是现成的,作为 AI 案例很容易做各种片段实验。(而不是,比方说,一张白纸,给个 prompt 就让 AI 瞎编故事,那当然肯定不太行。

试的结果是:完全没法用。

其实 AI 的阅读品味并不差,你给它两段不同的文本让它评论比较,它讲得通常也头头是道。但一写就完全露馅。

大家对 AI 写文常见的抱怨是文字层面:各种花里胡哨的比喻,不是很常见的标点,没有个性的对话等等。这些当然也是问题,但如果 AI 的问题仅限于此,那其实也还是可以用,因为这些可以修正。

真正的困难是我本来完全没想到的:AI 似乎从根本上不太理解文学这件事,而且问题在好几个常见模型都是一样的。

举个 AI 写的片段例子,写穆雨晴和男友打电话:

「我跟你说,今天发生一件特别好玩的事。」穆雨晴的语速快得像在抢答,试图用高昂的兴致冲破那道屏障,「新生orientation的时候,我跟一个印度小哥为了一项注册流程吵了半天,俩人都觉得自己是对的,结果你猜怎么着?我俩都看错了手册!你说好不好笑?」她描述着,心里却在冷静地分析:这个故事的切入点很好,既体现了初来乍到的笨拙,又有文化碰撞的趣味性,他应该会觉得有意思。

要命的不是前面那段对白(虽然已经很令人尴尬了),而是最后那部份。我看到的时候目瞪口呆。

文学的本质是和读者的对话,不是老师回答问题那种意义上的对话,而是恋人调情意义上的对话(其实是一种对抗)。你得不断设置不连贯的语义,让读者接连接受意外的信息,这种蹦蹦跳跳一来一回的意外感加总起来,就是文学的阅读体验。(某种程度上来说,擅长聊天的人的聊天记录读起来就很像文学。

而 AI 不理解这个,它试图把问题说明白——但是文学的目的不是把事情说明白。

太祖给陈毅写过一封关于如何写诗的信,里面有段著名的论述:「诗要用形象思维,不能如散文那样直说,所以比、兴两法是不能不用的。宋人多数不懂诗是要用形象思维的,一反唐人规律,所以味同嚼蜡。」AI 比宋人还糟,不但不懂用形象思维,而且把所有的可以留出气口的部分都生怕读者自己想到了,全都堵上。最后写出来的东西感觉是在用代码讲故事。

我不太确定这个问题能不能通过 prompt 修复,感觉应该从训练时候就采用一套完全不同的奖励机制才行,但这个 reward 机制很难设计。即使对人来说,要教育一个人「说话有趣」也很困难,基本上是个天赋,因为无从设计训练题目。AI 虽然可以轻松读完世界上所有的小说,但「有趣」不是靠阅读量就能堆出来的。

(也许可以让评论罗伯特试试看。

ENGLISH VERSION

遇到了一个有趣的问题,正好落在 AI 模型的能力边界处:试证明不可能把平面分成无穷个圆的无交并。

在我尝试的所有模型里,只有 GPT 5 thinking model 成功做了出来(虽然花了点时间)。

有趣的不是这个结论,而是观察它们的思路。所有失败的模型都有个共同点:它们的思考基本上是从文字到文字的。它们会调用自己脑海中各种已有的定理和知识,然后漫无目的地试图拼凑出一个证明,但所有这些定理,不管是拓扑的还是几何的还是测度的,对它们来说都是纯粹字面意义上的陈述。Qwen 的思考过程最典型:它滔滔不绝想了很久,但很显然从头到尾它都并不真的理解它在说什么。圆也罢,开集闭集也罢,Baire 纲定理也罢,对它来说都是纯粹的概念,给人的感觉是它甚至并不真的知道「圆是圆的」。

微妙之处在于,这种「没有几何直觉的几何思考」在某些时候其实未必是一种劣势。现代数学早已挣脱了对三维现实想象的依赖,大部份数学思考本来也确实是在纯粹的概念思辨空间中进行(特别是当问题进入代数乃至范畴论的领域的时候,这时从概念到概念的思考就变成了一种必然)。有的时候,几何直觉甚至反而会成为一种束缚,特别是当思考高维空间的时候,基于低维现实的直观常常是有误导性的。在这些问题上,AI 的「盲目」反而带来了自由,使得它不必受困于视觉直觉。——当然,人类的视觉直觉可能会渗透进人类的文本语料里,在某种程度上「污染」AI,但这是另一个问题。

然而对原问题来说,因为这是一个低维问题,几何直觉在这里不但有用,而且能大大缩短思考搜索的难度。在这一点上,一个把圆只作为抽象概念来理解的 AI 就会有巨大的劣势,因为它无法享受到几何直觉带来的跳步。这种直觉使得人可以一眼「看出」关键的构造,而这种构造在文本层面被搜索出来是困难的。

考虑到 AI 的应用毕竟大多数情况下还是为了解决世界现实问题而不是思考高维几何,有几何直觉的 AI 会在大多数问题上显得聪明得多。于是一个现实问题是,这种直觉是只有依赖多模态的训练才能获取,还是可以通过精巧的文本训练就能实现?这有点像是 AI 领域的玛丽房间问题。这是一个经典的知识论思想实验:一个从出生就生活在黑白房间里、精通颜色物理与神经机制的科学家玛丽,当她第一次走出房间看到红色时,她是否获得了新的知识?

今天大多数 AI 领域的困难都可以归结于此。人类是自己感官的奴隶,我们听到、看到、闻到,我们体会身体激素的涨落,我们想象、困惑、愤怒,然后试图把这一切投射在文字空间里。AI 则正好相反,它们在文字里理解这一切,但最终需要努力地——有时候是徒劳地——明白,一个圆在什么意义上是圆的。


Circle

I came across an interesting problem that happens to sit right at the boundary of an AI model’s capabilities:

Prove that it is impossible to partition the plane into an infinite (uncountable) disjoint union of circles.

Among all the models I tried, only the GPT-5 thinking model managed to solve it (though it took some time).

What’s interesting is not the conclusion itself, but watching the thought process. All the unsuccessful models shared a common feature: their “thinking” was essentially text-to-text. They would call up all sorts of theorems and knowledge they had stored, then aimlessly try to patch together a proof. But all those theorems — whether from topology, geometry, or measure theory — were, for them, nothing more than purely verbal statements. Qwen’s process was the most typical: it rambled on for quite a while, but it was obvious that it didn’t truly understand what it was saying. Whether it was circles, open or closed sets, or the Baire category theorem, for it these were all purely abstract notions — one got the sense it didn’t actually “know” in what sense “a circle is a circle.”

The subtlety here is that this lack of “geometric intuition” isn’t necessarily a weakness in all situations. Modern mathematics has long since freed itself from reliance on imagining three-dimensional reality; most mathematical reasoning is indeed carried out in purely conceptual space (especially when problems move into the realms of algebra or category theory, where concept-to-concept thinking becomes inevitable). In some cases, geometric intuition can even be a hindrance — particularly when thinking about high-dimensional spaces, where low-dimensional, real-world intuition is often misleading. For these problems, an AI’s “blindness” can, in theory, actually grant it freedom — it need not be trapped by visual intuition. Of course, human visual intuition can seep into human-produced text corpora, “contaminating” the AI to some degree, but that’s another issue.

However, for the original problem which is low-dimensional, geometric intuition is not only useful but can greatly reduce the difficulty of the search. In such cases, an AI that understands a circle only as an abstract concept is at a major disadvantage, because it cannot enjoy the shortcut that geometric intuition provides. Such intuition lets a human “see” the key construction at a glance — something that is much harder to search for in purely textual space.

Given that AI is, in most real-world applications, meant to solve practical problems rather than think about high-dimensional geometry, an AI with geometric intuition would appear far more intelligent in most scenarios. This raises a real question: can such intuition only be acquired through multimodal training, or could it be achieved through cleverly designed text-only training? It’s somewhat like the “Mary’s Room” problem in AI. This is the classic thought experiment in epistemology: a scientist named Mary, who has lived her whole life in a black-and-white room and knows everything about the physics and neuroscience of color, steps outside and sees red for the first time — does she gain new knowledge?

Most of today’s hardest challenges in AI can be traced back to this. Humans are slaves to their senses: we hear, see, smell, feel the ebb and flow of our hormones, imagine, get confused, feel anger — and then we try to project all of this into the space of language. AI is the exact opposite: it understands all these things within the space of language, but must then work — sometimes in vain — to truly grasp in what sense a circle is a circle.

坐照

ENGLISH VERSION

GPT 5 推出之后大家照例让它算 10.9 和 10.11 的差。它照例失败了,成了新一轮笑柄。倒是真的实现了传说中的 Ph.D level,因为博士生一般来说算术都不好。

当然这不是它一家的问题,别家 AI 也在这个简单的题目上纷纷翻车,包括我很喜欢用的 Gemini Pro 2.5。但 Gemini 翻车的姿势还要更炫酷一点:作为一款思维链模型,它知道这种时候应该调用 Python 来算。但当 Python 返回了正确结果之后,它的反应居然是:「我觉得 Python 算的不对,还是我自己来心算好了。」然后算错了。

这个错误虽然好笑,但暴露的是这一整轮 agentic AI 发展的致命弱点。Agentic AI 作为一个框架要能运行,前提条件是 AI 知道什么时候该使用并依赖外部工具。当然,这里的分界是模糊的:一个小孩子如果在算三位数乘法的时候掏出计算器,你不会批评。如果这个小孩算一位数乘法也要用计算器,你会怀疑是不是智力有点问题——当然无论如何至少结果是对的,但你会心想一个高级一点的大脑是不是应该合理判断这个问题不值当用外部工具。而现在的问题是这个小孩算一位数乘法,掏出计算器算了个结果,然后觉得不对扔掉了,自己心算了一个错误的答案出来。这是人类有可能犯的错误吗?

其实也是。而且如果你细想,这恰恰是非常「人类」的错误。人类的思维之所以不可靠,就是因为我们常常把直觉凌驾于客观证据之上。不是因为我们缺乏证据,而是因为我们不信任证据(例如曼德拉效应)。对人类来说,仅仅因为看到了和自己内心信念不一致的信息就放弃旧有信念不仅是困难的,而且是痛苦的。

但我们发明 AI 本意不就是避免这个缺陷?

于是我们面临着一个尚未有定论的问题,就是以大语言模型为基座的 AI 是不是先天继承了人类的心理偏见机制。我们对 AI 的期待是它能尽量不偏不倚。当然,在社会政治文化领域这是困难的,没有人能指望 AGI 在巴以冲突问题上能做到只看事实没有立场。但在别的更数字更技术的领域呢?给 AI 一份几万字的报表,AI 能够忠实灵敏地查阅所有细节,然后在回答问题的时候精确合理地引用某个细节吗?这不仅仅是我们对 AGI 的期望,这还是要撑起它所联动的万亿市值市场的前提条件。

今天的 AI 尚不能实现这一点,是因为这里有个内在的技术困难:思维链条不是数据库,而是把数据以自然语言的形式有损压缩在中间状态。这种压缩本质上就类似于人类以印象代替现实的思考模式,也是诞生偏见和误解的根源所在。要从根源上铲除它的土壤,就是要让这种压缩在事实上变成无损的。

于是我们面临两种可能的技术前景:

要么下一代思维链条(或者思维树,思维网络,或者不管什么别的数据结构)真的能实现对数据不依赖印象的理解和综摄。这在实践上已有尝试,比如程序化中间表示(JSON-graph、逻辑项、SQL、符号代数),或者对数字、日期、单位、表格索引做硬约束解码。简而言之,找到绕过以文字为思维载体的办法,把图像、数据和表格原生嵌入 AI 思考流程。

要么我们撞上了自然语言的先天限制。AI 将和人类一样,无论再怎么用力检查、对比、参考、判断,也只是不断用一层又一层的新的印象覆盖旧的印象,新的记忆调和旧的记忆,直到自己迷失在真实和幻觉之间的缝隙里。

前者是一种达芬奇式的前景,后者是一种博尔赫斯式的前景。或者用东方哲学的话说,前者意味着更强大的语言模型能够实现「坐照」之境,而后者意味着除非在底层重写技术框架,否则我们将不可避免的撞进文字障。

目前还没有证据证明哪个前景更有可能。前者如果成真,则立足于 AGI 的人类社会工业再数字化不但可行,而且指日可待。后者如果成真,则 AGI 不过是大号的人类,会在分裂和偏见之上引入新的分裂和偏见,不知伊于胡底。

大多数人对 AGI 的期待似乎是前者,并且这种期待如此底层,以至于甚至不需要宣诸纸面而是视为理所应当。然而如果人类运气不佳(一向不佳),我们很可能正在走向后者。


Luminous Stillness

After GPT-5 was released, people routinely asked it to calculate the difference between 10.9 and 10.11. As usual, it failed, becoming the latest laughingstock. It did, however, truly achieve the legendary “Ph.D. level,” given that doctoral students are generally poor at arithmetic.

Of course, this isn’t just its problem. Other AIs, including Gemini Pro 2.5, which I quite like, have also spectacularly failed this simple question. But Gemini’s failure was even more dazzling: as a chain-of-thought model, it knew it should call Python to do the calculation. But when Python returned the correct result, its reaction was astonishingly: “I don’t think Python’s calculation is right; I’d better do the mental math myself.” And then it got it wrong.

While this error is amusing, it exposes a fatal flaw in the current wave of agentic AI development. For agentic AI to function as a framework, the prerequisite is that the AI knows when to use and, crucially, trust external tools. Granted, the boundary here is fuzzy: you wouldn’t criticize a child for pulling out a calculator for three-digit multiplication. If the same child used a calculator for single-digit multiplication, you might question their intelligence—though, in any case, at least the result would be correct. But you’d think that a more advanced mind should reasonably judge that the problem isn’t worth using an external tool. The problem now, however, is that the child used a calculator for a single-digit problem, got a result, decided it was wrong, tossed the calculator aside, and produced an incorrect answer through mental math. Is this a mistake a human would make?

Actually, yes. And if you think about it, this is a distinctively “human” error. Human reasoning is unreliable precisely because we often allow intuition to override objective evidence. It’s not that we lack evidence, but that we distrust it (like the Mandela Effect). For humans, abandoning a long-held belief merely because we encounter information that contradicts it is not just difficult; it’s painful.

But wasn’t the whole point of inventing AI to avoid this very flaw?

Thus, we face an unresolved question: do AIs built on large language models (LLMs) inherently inherit human psychological bias mechanisms? Our expectation for AI is that it remains as impartial as possible. Of course, this is difficult in socio-political and cultural domains; no one expects an AGI to look only at facts and take no stance on the Israeli-Palestinian conflict. But what about in other, more numerical, more technical domains? If you give an AI a report tens of thousands of words long, can it faithfully and sensitively check all the details, and then accurately and reasonably cite a specific detail when answering a question? This isn’t just our hope for AGI; it is the fundamental prerequisite for supporting the trillion-dollar market value linked to it.

Today’s AI cannot yet achieve this because of an inherent technical difficulty: a chain-of-thought is not a database. It is a lossy compression of data into an intermediate state in the form of natural language. This compression is fundamentally similar to the human mode of thinking where impressions replace reality, and it is the very source of bias and misunderstanding. To eradicate this problem at its root, this compression must, in effect, become lossless.

Thus, we face two possible technological futures:

One: The next generation of chain-of-thought (or tree-of-thought, network-of-thought, or whatever other data structure) truly achieves an understanding and synthesis of data that does not rely on “impressions.” This is already being attempted in practice, for example, through programmatic intermediate representations (JSON-graphs, logical terms, SQL, symbolic algebra) or by applying hard constraint decoding for numbers, dates, units, and table indices. In short, finding a way to bypass text as the medium of thought, and natively embedding images, data, and tables into the AI’s reasoning process.

Two: We have hit the inherent limitations of natural language. AI, just like humans, no matter how hard it tries to check, compare, reference, and judge, will only ever be covering old impressions with new ones, reconciling old memories with new memories, until it is lost in the chasm between reality and illusion.

The former is a da Vinci-esque future; the latter is a Borges-esque future. Or, to use terms from Eastern philosophy, the former means a more powerful language model could achieve the state of 坐照 (“Luminous Stillness”, state of clear, mirror-like awareness achieved through quiet meditation), while the latter means that unless the underlying technical framework is rewritten, we will inevitably crash into the 文字障 (“Word Barrier”, where attachment to words obstructs enlightenment).

There is currently no evidence to prove which future is more likely. If the former comes true, then the industrial re-digitalization of human society based on AGI is not only feasible but imminent. If the latter comes true, then AGI is nothing more than an oversized human, one that will introduce new divisions and biases on top of existing ones, with no telling where it will end.

Most people’s expectations for AGI seem to be for the former, and this expectation is so fundamental that it doesn’t even need to be articulated; it’s taken for granted. However, if humanity’s luck is bad (as it usually is), we are very likely heading toward the latter.