Chatbot technologies are prompting predictions that automation is going to enter the white-collar space. This inevitability leads to concerns that AI is going to replace humanity. Prophets are using words like “intelligent,” “sentient,” and “conscious” to describe their assistants.

This is all based upon the criteria for intelligence proposed by Alan Turing. The problem is that Turing’s test (can I tell if I am conversing with a computer?) is not a meaningful test of intelligence. Intelligence is the ability to change behavior in response to a change in the environment. The environment known to a chatbot is grossly impoverished in comparison to the environment experienced by humans. The capacity of the chatbot to navigate that environment is almost non-existent – it does so only under the rules defined by its training algorithm. What these systems actually do is propagate human intelligence and combine language in novel ways.

Without intelligence, claims of sentience and consciousness fall aside.

The real problem with these technologies is that other people will use them to create the impression that they are intelligent and moral actors. Copying the speech of Gandhi or MLK Jr. is going to become easy. We are going to have to invest in deeper means of assessing capabilities – such as actually observing what people do.