24:37Ilya Sutskever: Sequence to sequence learning with neural networks: what a decade
Accepting a NeurIPS Test of Time award for the 2014 sequence to sequence paper, Ilya Sutskever puts his own slides from ten years earlier back on screen and grades them one by one: the deep learning hypothesis held, the autoregressive bet held, connectionism is the idea he says truly stood the test of time, and pipelining was simply unwise. Then the line the talk is remembered for: pretraining as we know it will unquestionably end, because compute grows on three channels while data grows on none. Data is the fossil fuel of AI and there is but one internet. He relays agents, synthetic data and inference time compute as other people's speculations rather than his roadmap, and offers hominid brain to body scaling as an existence proof that a second scaling regime can be found at all. The forward half lists four properties of superintelligence with every hedge attached, including the argument that gets the least attention: a system that genuinely reasons is a system you cannot predict. The Q&A, included in full here, produced several of the most quoted moments of the conference.