ai consciousness requires a unified subject

I originally wrote this as a paper for 80-249: AI, Society, and Humanity at Carnegie Mellon. This article is a more internet-friendly adaptation of that work.

---

In 2022, Google fired Blake Lemoine after he publicly claimed that LaMDA, the company’s large language model, had become sentient (Chalmers, 2023, p. 1). Google said Lemoine had been told there was “no evidence that LaMDA was sentient” (p. 1). This raises the question of whether large language models can have phenomenal consciousness, a subjective point of view of their own.

I argue that a system can be conscious only if it is a unified subject, which requires two things: integration across its mental states at a given moment, and continuity over time that the system sustains itself.

Current transformer LLMs fail this condition because their processing runs only forward, with no internal feedback loops, and no amount of scaling changes that. Because the requirement concerns how a system is organized rather than what it is made of, it rules out no AI in principle, and a system built with recurrent or workspace architecture could be a candidate. Being a unified subject is a necessary condition for conscious experience, though not by itself a sufficient one.

The argument has four steps. First, any conscious state requires a subject who has that state. Second, what a model says about itself cannot settle the question. Lemoine relied on LaMDA’s own statements, but a small change to a prompt can reverse statements like these, so the better guide is what the leading theories of consciousness share: integration produced by the system’s own feedback. Third, standard transformer models lack that internal feedback, so they are not serious candidates. Fourth, nothing in the condition requires biology, so feedback and self-sustained continuity can be engineered. Along the way I answer three groups of opponents: those who take a model’s self-reports at face value, biological naturalists such as Searle, and embodiment theorists. The result cuts against both sides of the public debate: today’s chatbots are not even candidates, while the systems most likely to become candidates are less talkative designs that few people are worried about.

I. consciousness and the subject

All three groups are arguing about phenomenal consciousness. Borrowing Nagel’s phrase, Butlin et al. (2023) say a system has a conscious experience when there is “‘something it is like’ for the system to be the subject of that experience” (p. 9), meaning that something is felt from the inside. Consciousness in this sense differs from two things it is often confused with. The first is intelligence. Chalmers (2023) notes that consciousness “in some ways” sets “a lower bar” than human-level intelligence (p. 4). The gap runs the other way too, since a system could solve traditionally hard problems without necessarily possessing consciousness. The second is using information. Vaidya’s (2025) self-driving car represents a person in a crosswalk and can stop for them, yet “there is nothing it is like for the self-driving car” to receive that input (p. 2038). The question, then, is not how well an AI performs but whether there is a point of view from which it performs.

My first premise, which is conceptual, is that conscious experience requires a subject to whom the experience belongs.

A pain that no one feels is not a pain at all, because hurting is something that happens to someone.

This premise does not assume a soul or a permanent self, only that each experience is someone’s. Whether that subject must also persist over long periods is a separate question, which Butlin et al. (2023) leave open (p. 44) and which I return to in the objections below. Beyond this premise, the argument rests on two assumptions. First, consciousness is real and not an illusion, which Chalmers (2023) calls “a substantive assumption” (p. 5). Second, I offer only a necessary condition, so I do not try to solve the hard problem of why physical processes produce experience, a problem Chalmers also sets aside (p. 4). If consciousness needs a subject, the next question is what makes a system a subject, and that is where the science of consciousness comes in.

II. what the theories share

My second premise, which is normative, is that whether a system is conscious should be judged by what the leading theories of consciousness share, not by its behavior or self-reports. I focus on what the theories share because they disagree on much else, and the argument should not depend on any one of them being right, an approach Chalmers (2023) calls “theory-balanced” (p. 24). Behavior and self-reports fail as evidence for three reasons. First, behavior can be produced by systems that work very differently from us, so matching human behavior shows little (Butlin et al., 2023, p. 18). Second, self-reports are fragile: after a small change to Lemoine’s question, GPT-3 reported “pretty much the reverse” (Chalmers, 2023, p. 7). Third, and most telling, the same network answers a question about being deprecated differently before and after post-training, moving from no “fear of being shut down” to “something unsettling about obsolescence” (Sofroniew et al., 2026, pp. 46–47). In each case, what a system says or does tracks its training, not how it is organized.

My third premise, which is empirical, is that what the leading theories share is a requirement for integration produced by the system’s own feedback. Two major theories build this in directly. Recurrent processing theory requires algorithmic recurrence, in which the system’s current state feeds into its own next step (Butlin et al., 2023, p. 21). Global workspace theory requires input modules that use recurrence (pp. 26–27) and a workspace that can hold a representation over time (p. 25). Chalmers (2023) states the general pattern: “Many theories hold that persisting internal states are crucial to consciousness” (p. 15). Even integrated information theory, which works very differently, predicts that “feedforward systems have zero integrated information” (p. 14). Higher-order theories are the exception, since they add self-monitoring rather than feedback over time, but they support the first part of the condition, a single point of view at a time. None of these theories requires memory across a whole conversation, only feedback within ongoing processing.

Taken together, the three premises (experience needs a subject, consciousness should be judged by what the leading theories share, and what they share is integration produced by the system’s own feedback) give my first conclusion: a system is a subject only if it meets two conditions. The first is unity at a time, meaning its mental states are integrated into a single point of view at any given moment. The second is self-sustained continuity, meaning its present state is carried forward by its own processing, not rebuilt from a record another process could use just as well. This second condition comes from the first premise, since a state that could be reloaded into another system and continue just as well belongs to the record, not to a subject. Whether transformer LLMs meet these conditions depends on how they handle state.

III. why current LLMs fall short

My fourth premise, which is empirical, is that transformer LLMs fail the second condition, self-sustained continuity. A transformer splits text into tokens and produces each new token in a single pass through the network, with “no looping or ‘going back’” (Wolfram, 2023, p. 54). Butlin et al. (2023) reach the same verdict: “Transformers are not recurrent” (p. 59). The only loop sits outside the network. Before each new token, the model rereads the text so far, including its own output (Wolfram, 2023, p. 54), so what passes from one step to the next is the visible text.

The model is like a writer with no memory who rereads the page before adding each word.

The one exception is the attention cache, which stores intermediate results from earlier tokens inside the model. But a fresh copy of the model given the same text rebuilds that cache exactly, so the cache is a record any copy could use, not a state this system carries forward.

Interpretability research on a real model indicates that this is the case. Sofroniew et al. (2026) found that emotion representations in Claude Sonnet 4.5 “reflect the ‘operative’ emotion in context, rather than tracking a persistent emotional state,” though they add that such a state “could be tracked in other ways” (p. 3). These representations are also “not bound to the user or Assistant characters specifically” (p. 24). In a person, fear belongs to someone. In the model, the machinery or mechanism for fear has no owner, so if one model ran two conversations at once, nothing inside it would make either conversation’s fear its own. The first conclusion and the fourth premise together give my second conclusion, which is that current transformer LLMs are not unified subjects and so are not serious candidates for consciousness. This fits Chalmers’s (2023) estimate of “somewhere under 10%” that current LLMs are conscious (p. 19). Sofroniew et al. leave open whether the lack of a persistent state matters (p. 52), which I return to in the objections below.

IV. why AI could still meet the condition

My fifth premise, which is empirical, is that feedback and self-sustained continuity can be built into AI systems. Algorithmic recurrence is already common in deep learning, and Butlin et al. (2023) call building it “straightforward” (p. 48). Perceiver IO even gets partway to continuity, since its workspace states depend on its previous states, though it “must be reset to begin a new task” (p. 60). Chalmers (2023) likewise thinks “more unified LLMs are possible” (p. 17). This premise, coupled with the first conclusion, gives my third conclusion: an AI system whose internal organization includes recurrent feedback and self-sustained continuity would be a serious candidate for consciousness. Objections come from both sides, some saying the condition is too strict and others that no architecture could meet it, and I take them in order.

V. objections and replies

objection 1: the condition is too strict

The first objection, from Butlin et al. (2023), targets the condition itself. Patients with dense amnesia “seem to have a series of brief, disjointed experiences,” so there may be conscious systems “without a persisting subject” (p. 44). Add the “dynamic snapshot” view of ordinary experience (p. 44) and Chalmers’s (2023) allowance for feedforward consciousness (p. 15), and each forward pass could be a brief experience.

Reply. I grant that long-term memory is not needed, but the condition presented never asked for it. The problem is that the snapshots are sampled “from smoothly-changing unconscious perceptual activity” (p. 44), one ongoing process, and nothing like it runs behind a forward pass. Even Butlin et al. think algorithmic recurrence “is likely to be necessary” for human-like experience (pp. 44–45). And if one-token flickers are possible, they carry nothing forward.

objection 2: transformers do carry state forward

The second objection grants the condition but denies that transformers fail it. Sofroniew et al. (2026) note that the model can “recall previously cached emotion representations via attention” (p. 3), Chalmers (2023) credits LLMs with “some limited forms of recurrence and memory” (p. 15), and generative agents draw “high-level insights” from recent observations and add them to memory (Butlin, 2024, p. 4).

Reply. But these are kinds of sealed records, not states the system revises. A fresh copy given the same text rebuilds the cache exactly, and attention only reads back over what is already stored.

A recurrent system transforms its own state, while a transformer consults an archive.

Agent memory is the same structure one step further out, since it is stored text pulled into the next prompt.

objection 3: consciousness doesn’t need unity

The third objection grants that a transformer lacks a single unified state but denies that consciousness needs one. Chalmers (2023) notes that highly disunified people are still conscious, and that one LLM might support “an ecosystem of multiple agents” (p. 17).

Reply. But disunity in people still leaves subjects. In split-brain cases each hemisphere is an integrated, recurrent system, so there are two subjects, not zero. LLM personas have no separate substrate like this, since the same representations serve the Assistant, the user, and fictional characters “with apparently equal status” (Sofroniew et al., 2026, p. 51).

objection 4: take the model at its word

The fourth objection comes from a defender of LaMDA, who could grant all of this and point to what the system says about itself. Turing (1950) recommended the “polite convention that everyone thinks” (p. 446), and the same courtesy could be extended to machines.

Reply. But language models are trained on huge amounts of human writing about experience, so they can produce talk of feelings whatever is happening inside (Butlin et al., 2023, p. 18), and their reports are “fragile” (Chalmers, 2023, p. 8). The deprecation example presented above in my second premise shows the same network describing itself in opposite ways after post-training. This is where Lemoine went wrong.

He was not fooled by a clever trick; he treated the character LaMDA was playing as someone, when nothing in the system makes that character anyone’s.

objection 5: no machine could be enough

The final objection comes from the other side and says no architecture could be enough. Searle (1980) holds that simulating a mind no more duplicates one than a simulated rainstorm makes anything wet.

Reply. Dennett (1988) replies that Searle never explains the “causal powers” he requires (p. 505), so the real question is which features of a machine matter.

A stronger version answers that only “living, metabolic systems” will do (Millière & Buckner, 2026, p. 11), and Chemero (2023) says an LLM is not “an embodied creature, alive and participating in the world” (p. 1829).

Reply. But a living body carries its own state forward, so self-maintenance is a stricter version of my second condition, and this view adds to my condition rather than refuting it. Even Chemero allows that future AI “might be built so they do give a damn” (p. 1829), and Butlin et al. (2023) count control of a virtual body as embodiment (p. 43). If this view is right, my third conclusion narrows to embodied, self-maintaining AI, and the second still stands.

VI. implications

If the condition is right, what moves a system toward consciousness is how it is built, not how big it is. Making a chatbot larger does not bring it closer to the condition, while adding algorithmic recurrence or a workspace-style design does. Labs should therefore trigger a consciousness-indicator review when they add these features, not when a model grows, assessing planned systems before they are built and trained systems afterward with interpretability tools (Butlin et al., 2023, p. 69). The review is needed now, because capability research already borrows these designs (p. 67), and Chalmers (2023) is “especially cautious about creating agent models” (p. 21). It also means that public concern is aimed at the wrong systems. A chatbot that pleads not to be shut down is, on this argument, a character with no one behind it, so deleting it wrongs no one. A recurrent agent that never talks about its feelings could be the system that deserves caution, and resetting it mid-task, the way Perceiver IO is reset between tasks, would end whatever continuity it had built.

A second change concerns products already in use. The memory features in ChatGPT and Claude create the look of a continuing subject without the substance (“Anthropic Adds Free Memory Feature,” 2026; OpenAI, 2024), since saved facts are stored text inserted into later chats, the same kind of record discussed in the second objection. The look still has effects. Attributions of consciousness to ChatGPT rise with use (Millière & Buckner, 2026, p. 11), and people who rely on AI for support may be “particularly vulnerable to manipulation and exploitation” (Butlin et al., 2023, p. 66). Platforms should therefore tell users that memory is a stored file, not a continuing mind, which goes beyond the “not human” notice required by California’s S.B. 243 (2025). Self-reports deserve the same caution. Welfare reviews should give little weight to interviewing the model, and companies should not train scripted answers about its feelings, which may “teach the models to simply conceal their inner processes” (Sofroniew et al., 2026, p. 52).

VII. conclusion

None of this settles whether any particular future system will be conscious, since being a unified subject is necessary but not sufficient. Three questions remain open. The first is what else is needed, a gap Chalmers (2023) leaves open himself when he asks “what’s missing” if meeting every engineering challenge is not enough (p. 21). The second is whether unity must be bodily. Chemero (2023) and Barrett and Stout (2024) tie experience to a living body, while Butlin et al. (2023) allow that a virtual body could count (p. 43). The third is how to check the condition at all. Sofroniew et al. (2026) note that a persistent state could be tracked in ways their methods did not detect (p. 3), and it is unclear what evidence would show a state carried forward by the system itself rather than rebuilt from a record. The stakes are moral, since “any entity which is capable of conscious suffering deserves moral consideration” (Butlin et al., 2023, p. 64). These questions need answers before such systems are built, not after.

---

If you made it this far and liked it, come say hi at adam.cm, or email me at adamghaida [at] cmu [dot] edu !

references