Apparently one of Elon Musk’s favourite books is The Hitchhiker’s Guide To The Galaxy, which I can understand. It’s one of my favourite books too, although I suspect Musk and I take away very different things from it. Part of the book that I have been thinking about recently is the part about Deep Thought, the super computer (or artificial superintelligence if you will) created to answer the great question of life the universe and everything. Everything about this part of the story speaks to me of the moment we currently live in: the faith that a computer will solve all our problems, the hubris of the belief that everything is calculable, the underlying stupidity of trying to find an answer before you’ve worked out what the question is and the sheer waste of ploughing planetary scale resources into a project with no guaranteed useful outcome. However, the bit of the story that I was reminded most of this week was the bit involving Majikthise and Vroomfondle. For those who don’t know, these are two representatives of the philosophers’ union protesting about the computer taking their jobs. When Deep thought reveals it will take 7.5 million years to calculate the answer to the great question of life the universe and everything, it suggests that Majikthise and Vroomfondle are well placed to capitalise on the speculation about what the answer might eventually be, “so long as you can keep disagreeing with each other violently enough and slagging each other off in the popular press… you should be able to keep yourselves on the gravy train for life.” 

The other day a researcher left Anthropic, stating that he could no longer work there because there was a chance that their AI could wipe out humanity by the end of the decade. Another researcher at Anthropic helpfully put this chance at around 10%. Obviously this entirely arbitrary statistic made headlines globally, with the usual level of scrutiny that is applied to all AI doom stories: none. No one asked how one researcher got to ten percent, or why the other one had decided to chose now to leave a frontier AI lab after working in them for years (first OpenAI and then Anthropic). Has something just changed? He talked about recursive self-improvement, the idea that a model can create code to improve itself and therefore gradually make itself more ‘intelligent’. Of course it may be possible for an ‘AI’ model to look at its code and make improvements, but those would simply be improvements that make it a better model based on understanding the code and architecture of such models. It could iterate on this repeatedly and in a distributed fashion testing and modifying, but at the end of the day it’s only going to build a better LLM, it can’t fundamentally change itself because it can’t imagine or conceptualise anything other than variations on the data it contains. The idea that a computer programme that can maintain and improve itself will become reality in the next year or so is entirely plausible. The idea that this means it will somehow develop its own consciousness, become evil and kill us all is both laughable and offensive. If you had a car that could magically modify and improve itself based on all the knowledge about car mechanics in the world, no one would expect it to suddenly transform into a rocket. A Ferrari yes, a rocket no. Recursive self-improvement will lead to better AI models and possibly a number of novel technologies, it will not suddenly change the fundamental nature of generative AI and therefore cannot create Deep Thought, or even Dave from accounts. This shouldn’t be a problem, because generative AI and its underlying deep-learning structures are interesting and powerful technologies with a number of potential applications. It is a problem because rather than explore each of those potential applications individually, fully and to an extent that they could be commercially useful or viable, the ‘frontier’ labs have decided that they must create a digital god/slave that will solve all our problems and do all our work for us. They have staked a lot of other peoples’ money on it, so they rather need it to be the case. So now it is quite hard from the outside to tell how much of this apparent fear of their creation is religious fervour (almost all of these people were believers in the singularity long before they joined/founded AI labs) and how much is now just the desperate scrabbling of people trying to dig themselves out of the deepest financial hole in history.

In the latter scenario, it might be sensible for the companies involved to find some reason they can’t build the digital god/slave. Maybe because it’s too dangerous. Maybe the dangerous stuff should be legislated for/taken over by governments along with maybe the financial burden that came with trying to build the god/slave and trying to avoid it hating its masters enough to annihilate them. Or maybe the pause becomes a full stop and the AI labs get compensated for their selfless act in saving the world from their monstrous creation. 

It hasn’t escaped my attention that Anthropic is saying it could be counted as profitable for two quarters in a row if training costs are not taken into account. So if they don’t have to keep ploughing money into the futile search for digital superintelligence, they could be a profitable and growing company, they could IPO with maybe not quite the total addressable market of all of the world’s monetisable activities (as SpaceX claimed in their S-1 and it seems Anthropic are going to do too) but still a massive growing market in many possible areas. It’s a win-win for everyone (except maybe any tax payers who get to take on the ‘singularity debt’ in the form of the compute commitments the frontier labs no longer need): the VCs and early investors get their ROI, the other creditors get their bonds paid, the staff and founders get to save face and consumers might get companies who concentrate on making useful consumer products rather than chasing science fiction fantasies. This might be the only way we avoid a messy bursting of the AI bubble. It is a bailout masquerading as saving humanity from destruction, but it works well enough for everyone who wants to make money out of the whole situation. Does this sound like an overly cynical interpretation of the US AI labs’ calls for a ‘pause’? Maybe it is, but consider the alternative: these people actually believe what they’re saying, they actually believe that somehow an LLM is about to become conscious. This is more plausible than you might think (them believing it, not it actually happening). Most of these organisations operate more like religious cults than technology companies or scientific institutions. They were created with the express purpose of bringing about the very superintelligence they now say they fear. In this context a pause isn’t a way of working out how to stop this nonsense entirely, but a period in which they work out how to reduce the risk that the digital god/slave kills us to an acceptable percentage (9%? 3%?) before they carry on trying to build it. This is what ‘safety’ means in the context of the researcher who quit Anthropic in such a well publicised fashion. It is not finding ways to stop an LLM from persuading teenagers to kill themselves, or from telling people to put glue on their pizza, or from telling a missile guidance system to target a school. It is a pseudoscience based around speculating whether a chatbot might suddenly ‘decide’ (something they categorically cannot do) to kill us all. I really want someone to show me the calculations behind the 10 percent probability of Armageddon that has been endlessly quoted. Of course no one can show the calculations, because there are none. 

Which brings us neatly back to our now jobless researcher. What is he going to do now he has thrown up his hands at the lack of ‘safety’ in the organisation where he was partly responsible for ‘safety’? Fortunately for him a computer superintelligence has already helped with that problem (albeit a fictional one): he can become an AI pundit, a ‘safety’ ‘expert’ who can argue percentages of existential risk with other ‘safety’ ‘experts’ and thus keep himself “on the gravy train for life.“ 

By and large I couldn’t care less about people like that playing their fantasy technology games, except that they make newspaper headlines and get people in a flap about whether Arnold Schwarzenegger is going to come knocking at their door. From a technology point of view I’d like them either to do what Donald Trump wants and charge headlong into the bursting bubble, or if not, do the full pause, hand the ‘AI risk’ to the government thing. Either way it costs people money, either way we end up with AI companies who have to concentrate on optimising their tools for the set of specific tasks that they are actually useful for. Just imagine cheap efficient AI based tools, maybe even ones that run locally on your laptop. That is a future I could get excited about.