Don't LLMs really have understanding?

In articles and podcasts discussing relationship of LLMs and Artificial General Intelligence (AGI), I have multiple times noted claim that LLMs are "actually very stupid" or "not even approaching AGI" since "they just generate stream of tokens from some algorithm" and "do not have any real understanding" of what they are saying. ChatGPT can itself summarize these views by its answer to question "Do the large language models understand anything really?":

"LLMs don't truly "understand" anything in the way that humans do. Instead, they process and generate text based on patterns learned from vast amounts of data, without consciousness, emotions, or personal experiences. Any semblance of understanding is more an artifact of clever pattern prediction than actual comprehension. LLMs can only hence simulate understanding: true understanding involves more than pattern matching; it requires a conscious framework of reasoning and an awareness of reality that LLMs lack."

Ultimately the question of understanding is of course matter of definition and multiple definitions are possible. But I have been recently inclining to think that it is not useful to couple the definition of "true understanding" to "consciousness" in the way expressed above. On low level, LLMs are immensely complex neural networks where the neuronal connections have been trained by data to create neural circuits for reasoning, memory, associations and pattern-recognition so that they generate reasonable language output from language input. On low level, human brains are immensely complex neural networks where the neuronal connections have been trained by sensory data to create neural circuits for reasoning, memory, associations and pattern-recognition so that they can generate reasonable language output from language input. An alien investigating human and LLM neural networks could conclude that depending on definition of "understanding" they either both have true understanding or neither has true understanding (only semblance or simulation of understanding).

The distinguishing difference in LLMs and human brains is indeed in consciousness. The so-called "hard problem" of consciousness - explanation of why and how consciousness arises from the neuronal activity in the brain - remains stubbornly unsolved one and some philosophers of mind claim even that it is inherently unsolvable. Since we don't know the exact requirements of consciousness, there is at least a theoretical possibility that LLMs are conscious. This possibility was risen to news cycle when Google fired their engineer Blake Lemoine in 2022 after he had convinced himself Googles AI-model was in fact conscious or sentient and raised related ethical concerns. This would of course reduce the difference of LLM undrstanding and Human understanding to zero, but it's a rather fringe view and I will continue the discussion here from the more mainstream point-of-view that LLMs unlike human brains are not conscious.

When people say that we have real understanding of, say, that cats are animals, it's usually partly real we are consciously aware of such understanding ie we feel its reality in our consciousness. When we are working on some problem and thinking of different possibilities, making logical connections and comparing alternative solutions, we have a stream of consciousness experiencing these thoughts. On abstract level we feel that "we" are our consciousness, that when we make decisions and think, "we" are in conscious control. We are hence tempted to think that any entity without consciousness cannot have real understanding.

But here comes the humbling revelation: While we do not really know how consciousness arises in human brain, we do know from decades of neuroscience research that our consciousness is not in the drivers seat in our brain. Our consciousness is not doing thinking when our brain is thinking and our consciousness is not making decisions when our brain is making decisions. As tempting and natural it is to think that our consciousness is in control, this turns out to be just a powerful illusion. Studies in neuropsychology and fMRI brain imaging have shown that our consciousness is more like a passenger on the back seat of the car watching the scenery pass by. The heavy lifting of logic, memory, reasoning, knowledge (like "cat is an animal"), decisions and even language production all happen in the unconscious parts of out brain. We only become conscious of a fraction of these things and even then only minimum of a few hundred milliseconds after the decision, memory or sentence has been forming. See for example the research of Roger Koenig-Robert & Joel Pearson on the topic. Our consciousness is just an after-the-fact sensation of these things happening.

Hence the fact that humans can become (occasionally and partly) conscious of facts, relationships and conclusions should not be considered the criterion making our underlying understanding real. Our brain networks that encode the relationships like "cat is an animal" are not conscious, yet the understanding of this relationship exists only in this encoding, not in the later viewing of the encoding on the canvas of consciousness. Hence it would be misleading to include conscious awareness as a criterion of "understanding": such awareness is only helplessly, without control and after-the-fact observing whatever stream of signals our unconscious brain-circuits happens to produce. Our consciousness cannot distinguish between true and false tokens, fantasy or reality - it just streams out whatever signals the deeper parts of our brains generate as we well know from study of psychosis and psychedelics.

But when we drop consciousness from the criterion for "true understanding", the processes of reasoning and understanding in human brain and LLMs reduce to similar token-generating trained neural networks. Understanding in both cases is nothing more than encodings in trained connections between neurons and the resulting responses to inputs. If the language model is having neuronal connections and weights that represents some relationship in the world in similar way as human brains represent the same relationship then we can equally well claim understanding for both systems.

So the flaw that the critics of "LLM understanding" make is not that they would be giving too little credit to the LLM mode of operation: admittely LLM intelligence is "just a blind algorithm generating outputs from complex neural networks" as they sometimes put it. The flaw is that they give too much credit to the operation of our human brains, whose intelligence fundamentally turns out to be also just a blind algorithm generating outputs from complex neural networks. The only non-blind part in our brain - consciousness - is contributing to intelligence and understanding only in an imaginary way.

By being more humble about our own limitations and nature (an understandably emotionally difficult feat from our self-centered vantage point-of-view), we can see that in many ways LLMs understanding and intelligence is qualitatively like our own - and quantitatively often superior. The implication for AGI is radical: instead of the current models not even being on a trajectory towards AGI, they might be already beyond AGI.