Emma 🐎🍆 & LLMs with Chris Best

Hey y’all! The other day,

Chris Best

and I discussed a widely-experienced spammer who became a legend on Substack as well as some themes related to LLMs / “AI.” I felt pretty bad about my capacity to articulate my issues with LLMs; it hadn’t been on my mind for a few days, and I couldn’t recall my own positions! I thought I should share it anyway, but also expand on what I presently believe about them.

I cannot caveat this strongly enough, though: I could absolutely be wrong; I know everyone always knows this, but I have good reason to emphasize it. Chief among them is this: I have already been wrong about “how much can be inferred from ‘mere’ language”! If you had asked me ten years ago e.g. if LLMs would be able to produce accurate responses to questions in the variety of domains that they currently can, I would have said “no” for many of the same sorts of reasons I now doubt their further progress. It’s not possible for me to assess whether I was simply mistaken about where the limit is, or am in fact mistaken about whether there’s a limit at all. People smarter than I am in every sense say that I am wrong.

That said: a few thoughts I often have about LLMs these days follow!

LLMs and Jobs

If you’re worried about LLMs automating your job, you should ask yourself:

1. Is my job more or less complicated than driving a car?

2. Does my job occur in a context more or less complicated than the rather constrained system of roads, signage, lane markings, curbs, laws, and types of phenomena that self-driving cars face?

3. Is what I do “digitizable” —that is, capturable and modelable by computers— to a greater or lesser extent than a car’s operation, taking into account LIDAR and GPS and computer vision and so on? (Relatedly: is there more or less training data for what you specifically do than there is for cars driving on streets?).

4. Are companies pouring billions of dollars into making the automation of my job, year after year?

5. How does your current boss or customer feel about hallucinations or lies?

To get to where we are today with self-driving cars required shocking quantities of hardware and software engineering, but also and crucially top-down efforts like “highly precisely mapping and describing cities and laws” and “setting up costly human-in-the-loop manual intervention triggers and capacities” and so on. And where we are today is limited in an interesting way: it is not the case that “given hardware and software, cars self-drive”; it is the case that “given hardware, software, fallback humans on standby, and a giant amount of top-down programming and mapping and investment, there are some environments in which cars self-drive.” I assume most of the work in self-driving cars is as much in “scaling and reducing the cost of making a particular environment or city one in which these cars can drive” as in “improving the general driving-ability of these cars.” This may all be true for lots of other jobs, too. Whether it is will shape a lot of what happens to the industry over the next decade: whether for example all returns accrue to “general model companies,” or whether there’s money to be made “making different work environments ones in which LLMs can be successful.”¹

It’s also true that there are jobs where the answers to these questions suggest that the job could be automated now or soon. But there are many jobs where the answers to these questions suggest otherwise, barring some major development which may or not be inevitable.

More than Mere LLMs

Inevitability is of course hard to reason about. So far as I understand the relevant fields, there is no reason compatible with naturalism or materialism that “a mind like ours” couldn’t run on contemporary types of computers.² I am not a materialist; I am perfectly comfortable with the dualist theory that mind or aspects of mind are supernatural phenomena, in which case they may or may not be achievable on contemporary types of computers. There are all sorts of possibilities there: that they require new types of computers; that they require new types of programming; or that they cannot ever be achievable on computers for supernatural reasons.³

If we imagine that minds are achievable on current computers, I still think it’s likely that they require approaches we don’t yet have. This claim is common among cranks like me but not exclusively among us: that e.g. symbolic structures that correlate to how our minds may work are needed, that new instrumentalities or mechanisms that have yet to be developed will at a minimum need to be combined with LLMs or at a maximum will work totally differently. In either case, one cannot extrapolate from improvements in LLM progress whether or not we’ll devise such technologies, any more than one could look at the rapid pace of progress in physics in the 20th century and correctly conclude that a theory unifying general relativity and quantum mechanics was soon to arrive.

Certain processes can be predicted, but all we can say about uninvented technologies or yet to be developed knowledge is whether they violate the laws of physics as we understand them today. If they don’t, they should come, but on what timeline no one can say, and there’s always the possibility that our current understanding of either physics or the relevant other domains of knowledge is incomplete or erroneous. It’s possible that LLMs will match our minds entirely, soon or later. It’s also possible that they will not, either soon or later.

LLMs and New Knowledge

The generation of new knowledge is something

Dwarkesh Patel

has noted repeatedly seems to be surprisingly absent from the repertoire of LLMs. It’s also not something we understand well in ourselves.⁴

In general, the way it works for us is:

• We become aware of a problem and internally imagine ways it might be resolved. This imaginative process entails what Karl Popper called “creative conjecture,” and like all creative acts seems to be unusually sensitive to the particulars of the mind performing it.

• We test potential solutions against (1) existing knowledge, (2) intuition, and (3) reality, in the form of experiments or attempts or efforts or what-have-you.

The way that imaginative step works is slightly mysterious. We often heavily weight “recombination” in our conceptions of it: that is, a semi-random or random process of trying things we already know, perhaps in variations or rearrangements, to see if they “fit.” But there is, I suspect, something else at play. I don’t know what, but I don’t think this “brute force” recombination is the sole source of imaginative leaps. Again, being a dualist makes this a comfortable position for me, but for a materialist, it’s important to believe that nothing else is at play except recombination, because new knowledge must be a matter of “information that’s in there being recombined” and nothing else; there is of course no “divine spark” to a eureka moment, in the materialist view.

If we grant that all new knowledge arises from recombinations of existing knowledge —which it certainly could; recombination at different levels, of cross-cutting types of knowledge or orthogonal modes of intellection, seems near-limitlessly rich to me anyway— there remain problems with the “testing” step. LLMs have some sense of (1) existing knowledge, in an unusual way and with the hallucination problem constituting a major dilemma for them in working with it, none of (2) intuition, and questionable access to (3) reality. Debating whether (2) intuition is important leads to similar terrain as debating whether imagination is mere recombination: anyone who maintains that imagination is more than recombination or that intuition is important is rapidly disqualified from serious conversations because they must resort to something like dualism, in effect claiming that “the human mind is more than math” or “cannot be reduced to math.” Efforts can be made to improve access to (3) reality, of course, but I suspect that this is its own very long-term and very complex effort, with substantial last-mile challenges. It wouldn’t surprise me if the entanglements between sense-perception and intellectual operations (and language) are as vast and difficult an area as those LLMs constitute solutions to now.

So: it’s somewhat easy to imagine 1M LLMs trying to recombine all known physics knowledge in novel ways to develop a unification of quantum mechanics and general relativity. It’s much harder to imagine what can review and evaluate their outputs; you could connect them —perhaps— to all sorts of sensors and experiment-conducting devices; you could have other LLMs adversarially grade their outputs; etc. But if this were possible, it would already be possible! So why haven’t they advanced medicine or physics yet, beyond accelerating the work of scientists (which is already really good! LLMs rule!)?

An answer I feel but cannot defend is: because there’s no free energy. You cannot “hack” the knowledge-generation process any more than you can “hack” energy generation processes. Why not? I have my answer, of course, but I don’t know what the materialist explanation is; there may well be one.⁵ If it’s akin to why we cannot have free energy, which has to do with the laws of physics, then such an answer probably has to do with “laws” of information, but I can’t say what those might be. What I do know is that the latest gate for an LLM-skeptic is this, and it’s a real one no matter how many prior gates have been breached. It’s silly to cite LLMs, given their very high capacity for error and all sorts of distortions, but o3 has a decent overview of the situation, and suggests that fairly substantial work remains to be done before this changes, although it’s optimistic.

Relatedly: a hell of a lot of human life involves “the creation of new knowledge,” which we tend to think of as “how to cure cancer” but which also constitutes “how to do this specific thing, today, for this customer, in these conditions, with these constraints.” I am skeptical of LLM art, LLM writing, LLM movies, LLM programming, LLM automation of all jobs, and LLM science for many reasons, but certainly in part because I think all of these things involve the creation of new knowledge from ever-changing and highly-specific and information-dense contexts, which is something I don’t think LLMs are good at and which human minds are sort of good at. I further think that they’re structurally unlikely to get good at them through scaling. If I’m wrong, I’ll apologize, of course, and again: I am non-technical and smarter people already think these arguments are foolish.

RIP Emma, and thanks

Chris Best

as always for the conversation!

1. For another angle on the success of self-driving cars, see this much sunnier post.

2. Many argue that “minds” aren’t required for AI, but I think most people have always meant “mind on a computer” by AI / AGI and that far more jobs require “full mind” than technologists typically think, which accounts for many a failed startup!

3. One other possibility for the non-materialist is that “whether this is possible” can change; that is: God alters how the universe works, in response to human activities or for any other reason.

4. I have not seen good answers to his question that suggest imminent or inevitable “AGI.”

5. I personally believe that this world will tend not to have “objectively accessible” sign-posts to the supernatural, such that there must now or eventually be a materialist explanation if this remains the case.