Below is a blog comment posting to KurzweilAI.net. The problem I have with the fantasies dreamed up by the proponents of an AI (Artificial Intelligence) - one day, "real soon now" - becoming "sentient" (I put that in quotes because word has been bastardised by widespread ignorance) is the very fact that because they aren't alive they can't be sentient in the true and original meaning of the term. As animals we are embodied in our environment thanks to literally billions of years of interaction with it. Yes, on one level the brain is a machine and the mind an emergent property of this impossibly complex machinery but it's embodied in its environment and all functioning is contingent on environmental stimuli. The brain's function collapses in the absence of environmental stimuli (sensory deprivation is used a means of torture for that very reason) because it cannot function as a disembodied entity. And part of that environment is the very body the brain is in, that body with its autonomic, reflexive, hard-wired genetic responses to millennia old cues, the fish, the lizard, the rat, the ape - all our past lives, encoded in our DNA during the headlong rush on the river out of Eden. A computer or computer system, however exquisitely engineered, could only ever be an imitation of life. Whatever it would be it would not be human, it would not be animal. It would not, could not be moral because conscience is a function of our biology and irrelevant without it. (Don't believe me? See, If It Feels Good to Be Good, It Might Be Only Natural.)
J. Storrs Hall, Kurzweil, et al. just don't get it. They're dualists in an age where science has rendered dualism meaningless, an age where observer and observed cannot exist one without the other, where matter and energy are simply different ways of looking at the same thing. The AI believers embrace a religion of sciolism, they've reduced the spiritual to algorithms in a finite state machine. This is an idea so fantastically (and obviously) wrong that I'm at a loss to imagine how they can ever be disabused of their folly.
'Here are we, one magical movement from Kether to Malkuth.
There are you, you drive like a demon from Station to Station.'
- Bowie, Station to Station