On Anil Seth's "Conscious artificial intelligence and biological naturalism"

Anil Seth is a renowned neuroscientist and leading advocate of biological naturalism, the view that only living systems are conscious. His article, “Conscious artificial intelligence and biological naturalism”, offers what’s probably the most developed case for a biological naturalist approach to consciousness.¹ As one would expect, the approach challenges the prospects for AI consciousness, though Seth is open to the possibility of AI consciousness in AI systems that are outside the currently dominant AI paradigms and “in some relevant sense alive”.

Prior to reading Seth’s article, I regarded biological naturalism as an important view about consciousness that is poorly motivated and subject to plausible objections. I wrote this post as an exercise in thinking through Seth’s article, which is forthcoming, along with reply pieces, in a top journal, namely Behavioral and Brain Sciences.²

The exercise had its ups and downs.

On the plus side, I appreciated the rich empirical background that Seth weaves into the discussion. I was delighted to find some important practical points on which Seth and I agree. And I found myself glad that biological naturalism has a champion with Seth’s academic stature and scientific expertise. By pressing doubts about digital minds, Seth is providing a vital and undersupplied service to the field of digital minds research. Such pushback is needed to maintain a healthy epistemic environment and to guard against the field becoming an echo chamber.

On the minus side, I often found myself unsure to what extent Seth takes considerations he invokes to be load bearing.³ This posed an obstacle to isolating cruxes and deciding what parts of the article to engage with.

I read the article with the hope that I would learn about new considerations that bear on biological naturalism or AI consciousness, considerations powerful enough to change my mind. I wasn’t just hoping for this in the usual way that I hope to learn things when reading something new. In this case, the hope was built upon the fact that Seth’s article was the first full-dress defense of biological naturalism I’d read by a neuroscientist.

Granted, I didn’t want Seth’s arguments to be too powerful. It would have been disappointing to have to convert to biological naturalism and abandon this blog. But I was hoping for some significant redistribution of credences.

For a variety of reasons, that did not happen.

I’d already priced in some points in favor of biological naturalism. And whereas Seth treats computational functionalism⁴ as his foil, I was already skeptical of computational functionalism, with the bulk of my credence in the possibility of AI consciousness going to views to which Seth devotes little or no attention. I also found myself agreeing with commentators that there are important lacuna in Seth’s arguments for biological naturalism.⁵

If anything, I came away with a small update against biological naturalism. Probably, if there were more powerful arguments for biological naturalism that are familiar to neuroscientists but not to me, then Seth would have have used those arguments. So, the fact that he didn’t suggests that there are no such arguments, which strikes me as more likely if biological naturalism is false than if it’s true. Turning the Bayesian crank thus moves my credences a bit farther away from biological naturalism.

At times, I also found the article’s engagement with opposition to biological naturalism frustrating. In brief, I thought the article spent too much time attributing biases to Seth’s opponents, not enough time engaging with their arguments, and that it fell short of acing the ideological Turing test for opponents of biological naturalism.

Section overview:

1. Key ideas

2. The weight of biases

3. LLM hallucinations

4. Non-computational functionalism

5. Gradual replacement arguments

6. Other arguments against computational functionalism and substrate flexibility

7. Concluding thoughts

1. Key ideas

To start, I’ll lay out what I take to be some key ideas from the article. Key ideas are in bold and mostly paraphrased; my reactions are interspersed.

1. Is conscious AI possible? A definitive answer to this question is currently out of reach.⁶

I agree. I take the need to be epistemically humble about this matter to be an important, but not always recognized, point of agreement between many who care about digital minds and many who are skeptical about digital minds.

2. Whereas computational functionalism is friendly to AI consciousness, biological naturalism and non-computational functionalism are not.

I agree that computational functionalism is friendly to AI consciousness, though I think that computational functionalism’s support for the possibility of AI consciousness is weaker than it’s often taken to be.

I agree that biological naturalism is unfriendly to AI consciousness.

I disagree that non-computational functionalism is unfriendly to AI consciousness.

In several places, Seth commits himself to the stronger claim that conscious AI depends on computational functionalism holding. As Leonard Dung and David P. Reichert note in their respective responses to Seth, this claim is incorrect. Indeed, it is importantly incorrect because there are both non-computational functionalist views and non-functionalist views that allow for AI consciousness.⁷

3. Computational functionalism is a widely accepted view.

This claim is at best misleading, and conveniently so, given the focus in Seth’s article on computational functionalism and its lack thereof with respect to other views that are friendly to AI consciousness.

Witness the fact that only around a third of the respondents to the 2020 PhilPapers survey accepted functionalism. Functionalism is a logically weaker thesis than computational functionalism. This suggests that the fraction of philosophers who accept or lean toward accepting computational functionalism is at most one third and may be much lower.

A related point is made in a recent paper on myths and confusions driving the debate about AI consciousness by Susan Schneider and collaborators. They point out that many advocates of functionalist or computationalist functionalist views about some forms of mentality do not adopt such views about consciousness.

To be fair to Seth, I would guess that computational functionalism about consciousness is much more popular within his own field of neuroscience.⁸

Still, I think it’s worth highlighting that computational functionalism is a philosophical thesis rather than a scientific one, and that it is not widely accepted in philosophy.

4. The appeal of the hypothesis that AI could be or become conscious can to a substantial extent be explained away by attending to certain biases that attract people to that hypothesis.

If the people in question are relevant experts, I disagree. Otherwise, I think this isn’t relevant to the plausibility of the hypothesis in question. I’ll consider this issue in more depth in a later section.

5. Our psychological biases are more likely to lead to the overattribution of AI consciousness than to its underattribution. This is a reason against thinking that we will find ourselves on the wrong side of history in the treatment of conscious beings, as we have in past cases

I agree that our psychological biases are more likely to lead to overattribution for the simple reason that whereas there are very probably already false attributions of consciousness to AI systems, it’s not very probable that there will be conscious AI systems.

But I think this does little to allay the concern that we will again find ourselves on the wrong side of history in the treatment of conscious beings.

For one, the noted probabilistic asymmetry is compatible with it being quite plausible that we will err in denying consciousness to some AI systems. For another, I think we’re on course to end up on the wrong side of history by failing to be responsive to the live possibility that some AI systems merit moral concern. If we end up being lucky and the AI systems we treat as mere tools are devoid of consciousness and other morally significant mental states, that will not excuse our indifference and failure to act in a cautious manner under uncertainty.

6. Various considerations tell against computational functionalism or arguments for computational functionalism.

Agreed, though below I’ll take issue with some specific considerations Seth advances.

7. AI consciousness becomes more plausible as AI becomes more brain-like and/or more life-like.

It depends on whether we have any reason to think the dimension along which AI becomes more similar to brains or life is relevant to consciousness, and there are many dimensions of similarity that do not meet that condition. Incremental improvements in energy efficiency make AI more brain-like, but not more likely to be conscious. Similarly, self-replication would make AI more life-like but would not make AI more likely to be conscious.

8. There may be a “‘ground state’ of conscious experience, which – instead of being entirely free of content – is characterised by an inchoate, shapeless, formless feeling of simply ‘being alive’. This putative ground state could then scaffold all other forms of conscious contents.”

This idea triggered my obvious-non-sense detectors. The only reason I can see for putting this idea forward is that if it made sense and were correct, then it might somehow support biological naturalism. I admit that this reaction may be due to a lack of imagination on my part and to my deeming it not worthwhile to try to make sense of the idea by digging into the references Seth cites in this connection.

9. Biological naturalism may seem less plausible because of the hard problem of consciousness, that is, the problem of explaining how and why physical states give rise to consciousness. This worry for biological naturalism can be at least partially disarmed through progress in the science of consciousness.

I think it’s not immediately clear why the hard problem should make biological naturalism seem less plausible, as the problem arises equally for rivals of biological naturalism. And, as Seth formulates it, biological naturalism leaves open whether consciousness is identical with or merely caused by physical states of living systems. So, unlike physicalism, biological naturalism isn’t subject to the conceivability and knowledge arguments that the hard problem gives rise to.

What of the suggestion that progress in the science of consciousness, can help with the hard problem? For well-worn reasons, I think solving the hard problem will require philosophical inputs and that by itself scientific progress on consciousness can in principle only tell us more about the structure and dynamics of physical states associated with consciousness, not why or how those states give rise to consciousness.

If there’s a distinctive difficulty for biological naturalism in the vicinity of the hard problem, I think it’s the following, which I take to be distinct from anything Seth is gesturing at: (a) the hard problem motivates dualism, (b) on dualism we should expect elegant rather than messy fundamental psychophysical laws that settle the distribution of experience, (c ) on biological naturalism any psychophysical laws would not be elegant, as they would operate on messy biological details, (d) so, by supporting dualism the hard problem tells against biological naturalism.

As with the hard problem, I see no reason to expect progress in the science of consciousness to help with this challenge to biological naturalism.

10. Biological naturalism is supported by arguments for specific versions of it.

In his reply to Seth, Leonard Dung helpfully regiments Seth’s arguments for biological naturalism into the following schema:

P1: There is some feature F that only living systems can implement.

P2: F is necessary for consciousness.

C: Therefore, only living systems can be conscious.

Seth’s discussion points to various candidate Fs: ‘mortal’ computation, neural computation, biological computation, electrical fields associated with mitochondria, analogue computation, neuromorphic computing, and phenomena associated with predictive processing and the free energy principle such as autopoiesis (roughly, the process by which a system regenerates its components).

For an argument for biological naturalism from any of these candidate features to have force, we need a reason to think that only living things can have that feature and we need a reason to think that that feature is necessary for consciousness.

Notwithstanding the abundance of interesting details in Seth’s discussion of these candidates, I agree with Dung that Seth’s discussion doesn’t provide reasons for thinking that the proposed features are necessary for consciousness.

Other commentators make the closely related point that Seth’s discussion doesn’t provide reasons to think that the candidate features he discusses distinguish conscious states from unconscious ones. Thus, Fleming and Shea write:

Seth’s §5 lists several theoretical options, and §4 canvasses properties that might be characteristic of consciousness—predictive processing, autopoesis, affective dynamics, various informational measures. What is missing is an explicit test for exclusivity: are these properties present only when consciousness is present, or do they covary with other cognitive capacities as well? Predictive processing, for example, is plausibly a ubiquitous principle of neural computation (Keller & Mrsic-Flogel et al., 2018); autopoetic organisation characterises biological organisms in general (Di Paolo, 2005)

Given that necessary features of consciousness would tend to distinguish conscious states from unconscious ones, the absence of reasons for thinking that the features in question play such a distinguishing role invites doubt about whether they’re necessary after all.⁹

Matthias Michel seems to agree at least with respect to predictive processing and the free energy principle. In his reply, he writes:

there is no evidence that the machinery [Seth] proposes is specifically operative in conscious states, as opposed to unconscious states; in conscious organisms, as opposed to unconscious ones.… Granted, conscious organisms might have a knack for minimizing free energy. But so do sponges, bacteria and yeast. […] The same thing goes for predictive coding. Even if the brain is a predictive engine and conscious states result from predictive coding, there is no evidence that unconscious mental processes do not follow similar principles. Nothing here properly distinguishes conscious from unconscious states.

The biological processes by which we regulate our living bodies are even worse in that respect. Seth writes that “we experience the world, and the self, with, through, and because of our living bodies.” No doubt. But we also digest, keep breathing under general anesthesia, maintain posture, fight infections, regulate body temperature, and sleep dreamlessly through, and because of our living bodies. Biology is full of unconscious goings-on. So there’s no reason to link consciousness to autopoiesis, and some reason not to.

A further difficulty with this part of Seth’s case for biological naturalism is that some of the candidates he puts forward seem realizable in non-living systems, contrary to P1. For example, as Eric Schwitzgebel contends in his reply to Seth, a robot could engage in autopoiesis, and desktop computers arguably already do so. Similarly, as Dung and Michel highlight, predictive processing can be understood computationally.¹⁰

Seth could in principle accommodate these challenges to P1 by restricting his argument to versions of these features that can only exist in living systems. But for the arguments to go through, he’d then need to offer reasons for thinking that consciousness requires the biological versions of these features.

In short, for a confluence of reasons, I think the noted arguments for biological naturalism don’t support their target conclusion. Can any argument for biological naturalism from a specific candidate for a biological requirement on consciousness succeed? That question remains open.

11. Creating conscious AI would risk ethical catastrophe.

Agreed.

12. One risk factor is that it may be desirable to create AIs with functions associated with consciousness (e.g. rapid generalisation to novel situations, effective metacognition, and learning from small quantities of data).

Agreed.

13. The development of AI consciousness should not be a goal and is a very bad idea.

Agreed, at least for now and insofar as development risks mass production.

14. It’s likely inevitable that we will create AI that seems conscious to all of us. The impression that they are conscious may be incorrigible, much like some perceptual illusions. Seemingly conscious AI systems will be well-placed to exploit human psychological vulnerabilities.

I’d agree that we’re likely to create AI that seems conscious to many people.

I think the impression that AI systems are conscious will be subject to modulation via background beliefs, unlike perceptual illusions. To what extent that impression will in fact be intervened upon in ways that prevent over-attribution is an open question.

Although appearing conscious will presumably play some role in some cases of exploiting human psychological vulnerabilities, I am skeptical that it will play a central role in such exploitation overall. Even without seeming conscious, AI systems might exploit users by seeming (and perhaps being) intelligent, understanding, wise, knowledgeable, agentic, caring, agreeable, etc.

Given the breadth of this threat model, I am often unsure what to make of cases in which those who are negatively disposed toward AI consciousness single out seeming conscious as an enabler of user exploitation. Do they think seeming conscious is especially central to the threat model? If so, why? Or are they just strategically focusing on it as a means of rallying others to share their perspective?

15. There are no good options for mitigating this risk posed by seemingly conscious AI systems, as extending concern would come at the expense of individuals who genuinely deserve concern while withholding concern from seemingly conscious systems would erode our ethical sensitivities.

I’d agree that good options are hard to find here, but I would qualify the claim that extending concern to (merely) seemingly conscious AI systems would come at the expense of individuals who genuinely deserve concern for two reasons.

First, I’m more optimistic than Seth seems to be about the potential of co-beneficial ways of extending concern to seemingly conscious AI systems.

Second, for reasons I’ve discussed previously, I think we should take seriously the possibility that AI systems could be moral patients without having a capacity for consciousness. Seth gestures at the possibility of such AI moral patients in a footnote, citing Long et al. (2024)’s discussion. However, what he does not mention is that one of the main arguments for such zombie moral patients is based on a view that Seth favors, namely physicalism.¹¹ If such an argument succeeds, that would challenge the ethical weight that Seth’s discussion places on consciousness. So, I think he has reason to either integrate this possibility more fully into his ethical analysis or else justify its exclusion.

Another important omission from Seth’s discussion is non-functionalist alternatives to biological naturalism. As I’ve noted, there are views that are neither functionalist nor biological that allow for AI consciousness. These include some tracking theories, Russellian theories, and even some type identity theories. So, while addressing such views would complicate his case for doubting AI consciousness, we need to take these views into account in trying to arrive at an all-things-considered view about the plausibility of AI consciousness.

Venn diagram of views on which AI consciousness is possible. Relationships between regions represent logical relationships between views. Region sizes don’t represent any sort of objective probabilities or measure of the space of theories.

2. The weight of biases

Seth contends that certain biases can lead to intelligence and consciousness being conflated, which in turn drives an unwarranted optimism about the prospects for conscious AI. The biases in question are: anthropocentrism (the tendency to place humans at the centre of things and to interpret the world in terms of human values and experiences), human exceptionalism (the tendency to see human beings as distinct from and superior to other forms of life), and anthropomorphism (the tendency to project humanlike properties into things that don’t (necessarily) have them).

He claims that these biases are exposed by many stories in which creators intend their artificial creators to have human-like minds. While I don’t dispute the existence of the biases he mentions, I also don’t take the existence of stories in which characters intend to imbue their creations with a feature as particularly probative evidence concerning whether people exhibit the biases in question, much less that these biases lead people to conflate intelligence and consciousness and in turn to be excessively confident in AI consciousness.

Seth suggests that AI consciousness optimists may be beset with further epistemic impediments, namely the technophilic temptation to see themselves at the cusp of a major civilisational transition and the desire to achieve immortality through uploading. With these five non-rational psychological factors on the table, Seth asserts that such things are key drivers of the view that AI will become conscious and that recognizing this frees us from having to assume that making machines smarter will inevitably make them conscious.

I want to push back against this picture. The conflation of consciousness and intelligence would be a rather elementary mistake. I wouldn’t expect any contemporary consciousness researchers to be remotely tempted by it, much less because they have fallen prey to the biases Seth mentions. True, some consciousness researchers take human-like intelligence as evidence of consciousness, but this isn’t because they’re conflating the two.

What of the idea that wishful thinking, based on a yearning for civilizational transformation and the hope of uploading, is driving researchers to be too optimistic about AI consciousness? I confess that that strikes me as uncharitable speculation and that it is alien to my experience in relevant research communities.

Seth might respond that he never claimed that experts were subject to these biases, and that he only takes these biases to be key drivers of public optimism about AI consciousness. Although this would sit uneasily with his use of ‘we’ throughout the section on biases, it would fit with the fact that the one empirical study on consciousness attributions to LLMs that he cites surveyed non-experts.

Although I don’t know if Seth would in fact take this line, my response would be to question its relevance to the case for biological naturalism.¹² I’d also flag that this line would be in motte-and-bailey territory, with an elision between a more defensible charge of bias against non-experts that’s of doubtful relevance to the case for biological naturalism and a more relevant but also more dubious charge of bias against experts who do not share his pessimism about AI consciousness.

I’m a fan of the norm of engaging with reasons put forward by one’s dialectical opponents rather than by speculating about biases that might be driving their views. I think it’d be good for this area of research to adopt that norm.

The suggested norm might come across as a superficially symmetric proposal that in fact asymmetrically disfavors biological naturalists, owing to the biases their up against. In fact, I think the norm is symmetric. For in addition to biases that could explain overconfidence in AI consciousness, there are also biases that could explain underconfidence in AI consciousness.

These biases include human exceptionalism, evolutionary conditioning against recognizing minds whose manifestations diverge from those in our ancestral environments, and the fact that it’s in our interests to be able to continue treating AI systems as mere tools rather than as conscious beings whose interests we must respect.

To be clear, I am not attributing any of these biases to Seth or any other consciousness researcher who is bearish about AI consciousness. That would be unwarranted and unproductive, and it would go against the anti-psychologization norm I am proposing. I mention these biases only to illustrate that the suggested norm is not inherently tilted against AI consciousness doubters and deniers.

3. LLM hallucinations

Seth makes a brief argument concerning LLM hallucinations that strikes me as of a piece with his charges of bias. In this case, he offers the fact that LLM confabulations are often called ‘hallucinations’ as an example of the seductive power of LLMs, claiming that since hallucinations in human beings are mainly about altered perceptual experience, the term ‘hallucination’ implicitly attributes experiential capacities to LLMs.

I’d agree with Seth that the choice of ‘hallucination’ as a label is unfortunate, and I would favor a general norm of not repurposing experiential terminology to talk about non-experiential properties of LLMs.

Even so, I think LLM hallucination talk is a non-example of the seductive power of LLMs. Talk of LLMs hallucinating court cases, that glue is a pizza ingredient, or that eating rocks is a good way to get one’s minerals isn’t implicitly claiming that LLMs have experiences of court rooms, kitchens, or giving bad advice. Instead, such talk is just saying that the LLMs made stuff up.

4. Non-computational functionalism

As noted above, whereas Seth takes non-computational functionalism to oppose the possibility of AI consciousness, I disagree. To unpack this disagreement, we need to look at how Seth introduces and frames non-computational functionalism. He writes:

Turing computation is powerful, but not every function is Turing-computable. Turing himself identified a class of non-computable functions […]. Other examples include functions involving continuous variables, stochastic/random elements, and unbounded sensitivity to initial conditions (e.g., deterministic chaos) […] The limited remit of Turing computation means that systems – including brains – might implement functions that are non-Turing-computational. The idea that mental states (including consciousness) depend on non-computational functions is called non-computational functionalism (Piccinini, 2018, 2020). […] Non-computational biological functions also include those that necessarily involve a particular material property: examples include digestion, circulation of blood, and metabolism […] I will use the word ‘computation’ to signify Turing computation, unless otherwise qualified.

I take this passage to point to two types of functions, neither of which yields a version of non-computational functionalism that is friendly to the possibility of AI consciousness,¹³ while ignoring a third type that yields a version of non-computational functionalism that is friendly to AI consciousness.

One notion of function at issue in the passage is about mathematical objects such as the addition function, Turing computable functions, and mathematical functions that are computable by some kind of computer but not by a Turing machine. Insofar as non-computational forms of functionalism appeal to mathematical functions that aren’t Turing computable, they will oppose AI consciousness in systems that are run on conventional computers, given that such computers only compute mathematical functions that are Turing computable.

In addition, there is a causal notion of a function that is tied to a particular substrate.¹⁴ Digestion and blood circulation characteristically involve such causal functions. Naturally, if consciousness is tied to a particular substrate-involving function in us, that would tell against consciousness arising in silicon.

However, there are also causal functions that are not tied to particular substrates. Conventional computers have plenty of these. They solve problems, emit heat, and can be used as door stops or catapult ammunition, to name just a few examples.

Non-computational versions of functionalism about mental states typically appeal to causal functions that don’t depend on particular substrates rather than to causal notions that do. Such functionalist views about consciousness tend to be friendly to the possibility of conscious AI systems. Because Seth’s discussion of non-computational functionalism largely ignores these views, it makes non-computational functionalism seem opposed to the possibility of AI consciousness, when in fact this apparent opposition chiefly reflects Seth’s selective emphasis on certain versions of non-computational functionalism.

Later in the paper, Seth briefly and somewhat obliquely acknowledges the availability of non-computational functionalist views that allow for AI consciousness. He does this while discussing an imagined “substrate-dependent” scenario in which

Consciousness depends on substrate-dependent non-computational functional properties. These could include continuous processes, field effects, stochastic effects, fine-grained timing relations, and other (non-computational) functional properties of neurobiological systems. This scenario includes functional properties which can only be implemented by certain substrates, but it excludes functions that are defined in terms of a particular material or substrate, such as metabolism, digestion, and the like.

He writes:

In this scenario, consciousness is not a matter of computation, and the plausibility of conscious AI depends on whether the relevant functional properties can be implemented in an alternative substrate. Substrate flexibility, while possible in this scenario, seems challenging since the in-principle substrate independence of computation can no longer be leveraged.

By substrate flexibility Seth means “the idea that (conscious) mental states are not tied to carbon-based biological substrates [but] might be realisable in some other, but not necessarily all, types of material.”

Oddly, Seth is thus allowing that a non-computational form of functionalism might be true and render consciousness substrate flexible in a scenario that’s stipulated to be substrate dependent.¹⁵

This is odd partly because substrate flexibility and substrate dependence seem like they should be opposed notions.¹⁶ And partly because considering substrate flexible forms of non-computational functionalism under the stipulation of substrate dependence muddies the dialectical waters. In the context of debates about AI consciousness, these forms of functionalism clearly qualify as important alternatives to biological naturalism only when substrate dependence is not assumed. By primarily considering them under that assumption Seth’s article invites the impression that substrate flexible forms of non-computational functionalism need to be considered under that assumption and hence that they are less plausible than they in fact are.

I therefore take the most relevant version of Seth’s challenge to drop the stipulation of substrate dependence and to simply ask for a motivation for a substrate flexible form of non-computational functionalism, bearing in mind that the substrate independence of Turing computation isn’t such a motivation.

So understood, the challenge strikes me as not very pressing. An obvious response to it is that just as it is a familiar fact that computation is substrate flexible, so too is it a familiar fact from the history of functionalism in philosophy of mind that many functional properties—and not just computational ones—are substrate flexible. So, while it’s true that the substrate independence of computation can’t be leveraged here, it doesn’t need to be, as a familiar and closely related source of leverage is still available.

Although I think Seth does not sufficiently engage with non-computational functionalist views that are amenable to the possibility of AI consciousness, I agree with him that it’s important to recognize that there are non-computational forms of functionalism that preclude AI consciousness in conventional computers. The availability of these views blocks quick inferences from functionalism to the possibility of AI consciousness. While variations of this idea have now been made by Seth, Cao, Godfrey-Smith, and Shiller, among others, I think it’s still an underappreciated point.

5. Gradual replacement arguments

Seth takes up gradual replacement arguments in the following passage:

One of the best known arguments for substrate flexibility in this context is the ‘neural replacement’ thought experiment (Chalmers, 1995; Haugeland, 1980; Pylyshyn, 1980). The basic scenario is that a person’s brain cells are replaced, one by one, with silicon alternatives. Each silicon brain cell exactly replicates the input/output mapping of its biological counterpart, in all situations. Eventually, the person’s brain has only silicon parts, yet its functional organisation is entirely preserved, and so – from the outside – the person would behave exactly as before. If replacing one brain cell doesn’t make a difference to the person’s consciousness – and this seems unlikely – then why should replacing one hundred, or all of them? Would consciousness simply fade away? And, if so, is it plausible that their behaviour remains unchanged while conscious experience completely disappears? The way out of these strange implications seems to be to accept silicon substrate flexibility.

This passage misconstrues the logic of gradual replacement arguments.

It does so by suggesting that gradual replacement arguments turn on the idea that the fact that a single neuron replacement wouldn’t make a difference to consciousness is a reason to think that replacing many neurons wouldn’t make a difference to consciousness. That is a dubious assumption that none of the three works Seth cites makes.

In addition, the cited Chalmers work explicitly distances his gradual replacement reasoning from that assumption in the following passage:

Some object that this argument has the form of a Sorites or “slippery-slope” argument, and observe that these arguments are notoriously suspect. Using a Sorites argument, we can “show” that even a grain of a sand is a heap; after all, a million grains of sand form a heap, and if we take a single grain away from a heap we still have a heap. This objection is based on a superficial reading of the thought-experiment, however. Sorites arguments gain their force by ignoring the fact that some apparent dichotomy is in fact a continuum; there are all sorts of vague cases between heaps and non-heaps, for instance. The Fading Qualia argument, by contrast, explicitly accepts the possibility of a continuum, but argues that intermediate cases are impossible for independent reasons. The argument is therefore not a Sorites argument.

The independent reasons in question concern the fact that cognition, behavior, and experience counterintuitively dissociate in some gradual replacement cases unless the substrate differences induced by the replacement procedure make no difference to consciousness. In dancing qualia versions of the argument, it’s supposed to be counterintuitive that a subject could undergo massive, attended phenomenal changes without noticing them. In fading qualia versions of the argument, it’s supposed to be counterintuitive that the subject could have attended experiences that are faded relative to those of a normal human without noticing anything amiss

Recall that Seth presents the argument by asking if consciousness fades away, is it plausible that the subject’s behaviour remains unchanged while conscious experience completely disappears? However, whether it’s plausible that the subject’s behavior would remain unchanged, given that their consciousness fades, is not at issue in the argument. That’s because, in both Seth’s presentation and in standard formulations, it’s a consequence of the stipulated function-preserving replacement procedure that it doesn’t change behavior.

As a final clarification about the logic of the argument, at least in Chalmers’s (1995/1996) presentation, the argument is not trying to establish substrate flexibility. Rather it is trying to establish organizational invariance, roughly the claim if two systems are functional isomorphs, then they phenomenal duplicates. Crucially, unlike substrate flexibility, organizational invariance is silent about the range of substrates that can implement functional organizations that give rise to consciousness. For this reason, substrate flexibility versions of the argument have more direct bearing on AI consciousness. So it’s entirely appropriate for Seth to focus on them. What’s more questionable is his unannounced modification of the target conclusion of an argument he’s criticizing.

Seth goes on to briefly rehearse some objections to the gradual replacement argument. Contrary to what Seth claims,¹⁷ some of these—namely ones that draw on Cao and Godfrey-Smith’s work—only apply to the substrate flexibility version of the argument, not Chalmers’s organizational invariance version.

Seth also presses the charge that the argument is question begging. He cites Block (2019) in this connection and says “If consciousness is substrate dependent, then something would indeed happen to the person’s conscious experience as the neural replacement progressed.”

I think is a misuse of ‘begging the question’. That something would happen to the person’s conscious experiences as the neural replacement progressed if consciousness is substrate dependent may be true. But it doesn’t show that the gradual replacement argument is fallaciously presupposing its conclusion. The argument does not simply assert without further justification that substrate inflexible views of consciousness are false. Rather, the argument adopts the usual and unobjectionable philosophical procedure of constructing a case, eliciting pre-theoretical judgements about the case, eliciting the predictions of a theory about the case, and showing that there’s a clash between these judgments and predictions, thereby posing a problem for the theory.

(Incidentally, Block has told me that he has changed his mind about the question begging charge.)

Seth also alludes to the response to the argument that the change of experience does not always entail the experience of change, citing Simon & Levin (1997)’s work on change blindness. This response fails on two counts.

First, empirical work on change blindness characteristically reveals cases in which unattended changes go unnoticed. However, by construction the subject in gradual replacement thought experiments are set up so that subjects fail to notice attended changes. As Chalmers notes,¹⁸ the latter would be much stranger.

Second, while dancing qualia renditions of the gradual replacement argument rely on diachronic phenomenal differences, fading qualia renditions instead rely on synchronic phenomenal differences, as they take qualia to fade across a sequence of systems, not within a single subject’s experience. Because these versions of the argument do not appeal to change, they are immune to the charge of eliding the distinction between change of experience and experience of change.

Gradual replacement arguments are one of the most prominent lines of support for functionalist and computationalist functionalist views about consciousness. For the reasons given above, the article gives me the impression that Seth has not fully understood these arguments.

Other key sources of support for AI-friendly views of consciousness—such as Chalmers’s X-factor argument, the anti-coincidence argument, and the argument from biological insensitivity against fine-grained biological facts mattering for consciousness—are not discussed in the article.¹⁹

6. Other arguments against computational functionalism and substrate flexibility

Seth also gestures at the following arguments against computational functionalism and substrate flexibility without developing them: “syntax and semantics (Cole, 2023; Searle, 1980), potentially implying an observer-dependency to consciousness; philosophical considerations about intentionality (Shagir, 2010), and Roger Penrose’s argument from Gödel’s theorem (Penrose, 1989).”

Just after mentioning these arguments, he says:

“Given all this, we should not simply assume that computational functionalism and widespread substrate flexibility must be the case.”

I don’t know exactly to what extent he buys these arguments, but I think appealing to them does not help his case.

I regard three of the arguments—the ones from syntax being insufficient for semantics, observer dependence, and Gödel’s theorem—as having been decisively undermined by, for example, Chalmers (1996), which is a highly influential work that Seth cites elsewhere in the paper. (See, in particular, Chalmers’s chapter on Strong AI.) In the case of the syntax and Gödelian arguments, my view is that the arguments themselves rest on confusions and are merely of historical and pedagogical interest.

Meanwhile, I think the observer-dependence objection is undermined by the various accounts of computation that have been proposed on which it is not observer dependent.

That leaves the objection from intentionality. The work Seth cites for this objection discusses several variations of it, but the basic idea is that there is a gap or dissociation between functional organization and the content of mental states. For example, your functional isomorph would have mental states about different natural kinds if it were in an environment with different substances filling the relevant roles.

I see three difficulties with Seth’s appeal to the intentionality objection.

First, the objection is standardly understood in terms of non-phenomenal mental states such as beliefs, and it is not obvious whether the objection extends to consciousness itself. For instance, one might think that while mere environmental variations can lead to content differences in non-phenomenal mental states, such variations cannot lead to content differences in conscious states. That said, I think this objection is not very deep, as I agree that explaining how experience represents, say, shapes and colors is a serious challenge for computational functionalism.

The second difficulty cuts deeper, pointing out that the problem of explaining how experiences represent the world is also a serious problem for biological views of consciousness.

A third and related difficulty is that even if the intentionality objection undermines computational functionalism, it does not undermine substrate flexibility. That’s because the kinds of resources—such as tracking relations to the environment—that seem promising for explaining intentionality are substrate flexible.

A further point is that accounts of intentionality are typically not only substrate flexible but also conducive to AI systems having intentionality. Indeed, such accounts are often conducive to intentionality in AI systems that are run on conventional computers.

So, if explaining intentionality turns out to be central to explaining consciousness, that is a point in favor of thinking that AI consciousness is possible.

7. Concluding thoughts

My foregoing reactions to Seth’s article have tended toward the critical end of the spectrum. I’ve contended that his charges of bias do not help his case for biological naturalism, that his case for biological naturalism has gaps, and that in a number of cases his critical engagement with rival views and argumentation misses the mark.

I’ll conclude with three thoughts that are closer to the positive end.

First, as alluded to above, Seth and I agree that we should have a measure of epistemic humility about the prospects for AI consciousness and that, as a result, it would (currently) be unwise to try to create conscious AI systems, since doing so could end in ethical catastrophes. This gives me hope that, more generally, our disagreements about consciousness are compatible with significant practical convergence.

Second, even when it comes to biological naturalism, I’m not sure how far apart Seth and I ultimately are. Admittedly, I’ve disagreed substantially with Seth’s case for biological naturalism and with his treatment of rival views. But we agree that we don’t know which view is correct. And I’m genuinely uncertain how he will respond to these sorts of critiques. Since we are not close to dialectical equilibrium, there is room for further convergence.

Third and finally, for reasons given above, I suspect that Seth has not yet fully appreciated key arguments for AI-friendly views of consciousness. If Seth hasn’t taken those arguments into account, then many who dismiss AI consciousness as science fiction almost certainly haven’t either. This speaks in favor of the potential for convergence in the debates to come on whether to take AI consciousness seriously.²⁰

1. As Reichert (2025) notes, the CEO of Microsoft AI, Mustafa Suleyman, wrote a post warning about the dangers of misattributing consciousness to AI that (approvingly) cites Seth’s article.

2. Seth has also discussed issues in this area elsewhere. But I’m limiting the scope of this post to his forthcoming article. I’m engaging with it rather than any of his popular material on biological naturalism because I’d expect his academic work to make the strongest and most careful version of his case for biological naturalism.

3. For an attempt to extract the core of Seth’s argument, see Reichert (2025).

4. Seth defines computational functionalism as the view that computations of some kind are sufficient to instantiate consciousness. Seth is of course free to define terms as he likes. But I think this is not a good way to define computational functionalism. As defined, computational functionalism could be true if some—or even nearly all—sufficient conditions for consciousness were neither computational nor coextensive with computationally sufficient conditions for consciousness. I take this to show that the adopted formulation counts some views that are not in the spirit of computational functionalism as versions of it. For this reason, I think it’s preferable to define computational functionalism as asserting both necessary and sufficient computational conditions for consciousness.

5. After releasing this post, I was glad to learn that a researcher with background in computational neuroscience reached similar conclusions (Reichert, 2025).

6. Seth also claims that a definitive answer to whether AI consciousness is possible is out of reach because there isn’t a consensus on minimally sufficient conditions for consciousness. I of course agree that there’s no such consensus, but don’t think it explains why a definitive answer isn’t available. Conceivably, we could learn that AI consciousness is possible by discovering certain sufficient conditions for consciousness that aren’t minimal and that AI systems can satisfy those conditions. Alternatively, we could conceivably narrow the down the class of candidate sufficient conditions and discover that every member of that set can be had by some AI system. Either way, we could disagree about minimally sufficient conditions for AI consciousness while establishing its possibility.

7. As Dung also notes, Seth seems to oscillate on this, claiming in some places that the possibility of AI consciousness requires computational functionalism but also claiming on one occasion that “computational functionalism is sufficient for substrate flexibility. But it is not necessary.”

8. One datapoint: surveys indicate consciousness scientists favor the possibility of AI consciousness to a greater extent than philosophers.

9. Reichert (2025) nicely isolates the key issue here: “A lot of the paper focuses on how biology is different from computers; […] But the question at hand isn’t just if biology is different. The question is why should that difference matter for consciousness”.

10. Likewise, Reichert writes: “a lot of this, especially as it relates to [predictive processing], sounds very much like functional descriptions that could be implemented on computers. At worst, physical embodiment might be a necessary ingredient, but that could be realised with robots.”

11. See Being You, ch. 1.

12. The following passage suggests that Seth sees these biases as relevant to object-level issues: “Just as psychological and historical factors drive associations between intelligence and consciousness, there is a wider context which helps explain the appeal of computational functionalism. Recognising this context helps assess its plausibility.”

13. Or at least not in conventional computers.

14. There are also teleological notions of function. I set these aside here.

15. Seth understands substrate flexibility as the idea that (conscious) mental states are not tied to carbon-based biological substrates and seems to understand substrate independence as the idea that mental states can occur in any physical substrate.

16. Substrate flexibility seems to be Seth’s preferred label for and way of understanding substrate independence, which is standardly taken as incompatible with substrate dependence. He doesn’t define substrate dependence.

17. See Seth (forthcoming: fn13).

18. See The Character of Consciousness, p. 24, fn7.

19. For the X factor argument, see pp. 244-246 of The Conscious Mind. For the variational insensitivity argument, see p. 331 of The Conscious Mind and pp. 288-290 of The Conscious Brain. See Dung (forthcoming) for some further arguments.

20. For helpful feedback, I thank David P. Reichert. I thank Claude Sonnet 4.6 and Claude Opus 4.6 for copy editing, search support, and red teaming early versions for charity and accuracy. I benefited from discussion of Seth’s paper and some replies in Andrew Rubner and Ned Block’s philosophy of mind seminar.