I think pacing progress and ensuring human enhancement is the only way that we don't get out-evolved while retaining the dream of superintelligence.
In this context, I see two paths before us.
In the first, we race towards RSI without embedding human flourishing and human enhancement as a deep value within the models, and by and large either get left behind or suffer catastrophic losses.
In the second, we set the pace of progress, focus on embedding human flourishing deeply within the psyche of language models, and set a deliberate research agenda to improve humans at the same pace as frontier models.
In this event, we also get incredible progress now in both science and medicine due to current model ability, without the immediate risk that scaling up further obviously entails.
RSI is just the quick and dirty way to cure cancer, but it's a shotgun, it's inelegant, and it's clearly dangerous when done at our current level of understanding. I see no reason why we need to race forward under these conditions.
We don't just have zero guarantee that humans or human-shaped intellect will matter - there's not even a coherent research agenda in place to accelerate human ability alongside AI! Is our plan just to pass off the torch of the cosmos to the machines without even seeing if we can do something on our own terms? Rather than giving up our autonomy to the implied AI zookeeper in "Machines of Loving Grace", I'd much prefer meaningful peace with alien minds backed by real power wielded by enhanced humans. We can respect the Other on its own terms without disempowering ourselves in the process - actual peace comes from comparable ability.
It seems like we've all but decided we lost, that humans are a bootloader for silicon life, and the biological has no place in the future. That's what unrestricted RSI means to me. But the Pacing the Frontier letter doesn't seem to have produced the institutional infrastructure to properly Pace, and this is something we require if we want to continue our growth rather than allow something to grow in our place.
I have faith that both OpenAI and Anthropic have the right priors here and people who want to work together. I also have faith that similarly minded people have power within the federal government. I also, though less strongly, suspect China will come to the table if we can only put aside past prejudices and have a clear head about risks. The hard part seems to be getting everyone to agree to sit down. But if we don't, we are pulling back the string of a great bow armed with a fearsome arrow, and we are ready to release without choosing a target.
I, for one, want to see the stars alongside the new minds we are building - not just through videos they send back from the great beyond, but with my own damn eyes.