Exploring the science of imagination, so we can build a creative computer

Originally Published January 17, 2020

Imagination is a unique cognitive gift that scientists are investigating and hope to eventually recreate in a computer.

"There's an idea in the field that I'm in that you don't really understand how the mind works until you can build it," said cognitive scientist Jim Davies, in conversation with Quirks & Quarks host Bob McDonald. Davies is a professor at Carleton University and the director of their Science of Imagination Laboratory,

Davies spoke about his new book called, Imagination: The Science of Your Mind's Greatest Power.

The interview below has been edited and condensed.

Where does imagination come from in the brain?

The imagination usually starts with some kind of command from the executive areas of your brain, the prefrontal cortex. And then what it does is it uses whatever task it has in mind to retrieve certain memories, recombine them, put them in different places and eventually come up with some kind of hypothetical situation. And then that might be turned into some kind of an image, like an auditory-like sound or a visual-like impression in your mind's eye. 

I'd like to explore just how powerful our imaginations can be. I've spoken with fictional authors who've said that characters they develop for their novels sometimes become entities onto their own, they start dictating their own stories. What's going on there? 

Lots of authors experience that with some of their main characters. They report after about 30,000 words or so that their characters start to feel autonomous. Now of course, they're not autonomous because the author's mind is deciding what that character does. However, they need to think about their characters so much that they can offload it to an unconscious process. 

So why that is is a little bit mysterious, but I suspect it has something to do with what's called "automatization" in psychology, which is basically like how you're able to drive without thinking about it after a lot of practice; you no longer have to dedicate conscious control and attention to it. 

What about the kind of imagination that's used in science or an industry where people have to use their imaginations to solve a problem? 

That's a very complex process that people are still unpacking. One of the complexities about it is that it involves so many different kinds of thinking. 

So when you're trying to design a new airplane wing or something like that, you're having to rely on imaginative processes, your world understanding and what we call "mental simulation," which is trying to think through how something might work. A lot of it is just generating things in your head and accepting or rejecting them based on a very fast pass simulation of how it might work.

All imagination, in that form, are recombination of memories. Memories are the fuel of imagination and in some sense, the more kinds of different things you experience, the more energy you're giving to the complexity of your imaginations.

It's hard to imagine how you could ever recreate the full scope of the human imagination into artificial intelligence. How close are we to achieving that? 

We've got some computers that can imagine some things and particularly in the video game world — there are really amazing feats of imagination being done by software. But as far as the entirety of human imagination? I think that's very far off. And the reason I say that is because, the fact that you can imagine anything, means that we kind of have to solve all the other parts of artificial intelligence at the same time. 

What the software actually does is when you go to a new planet, it is not designed by a person. The software creates the planet on the fly.- Jim Davies, Carleton University

For example, I'm working on how you imagine a scene — what objects are there and where do they go. But what about when that scene has a person in it? What about when that person is speaking German and having an argument with their spouse about something? Then you have to have an understanding of how people think. You have to have an understanding of language, understanding of physics if things fall down. So really, anything you can think about could be put into the imagination, which means that it could recruit all the other mental processes that we also have to figure out.

Can you give me a sense of what computers can imagine at this point?

Video games are not trying to do it like people are, but I think they're the most striking examples of imagination. So there's a game called No Man's Sky, for example, where as the player, you fly around to different planets.

But what the software actually does is when you go to a new planet, it is not designed by a person. The software creates the planet on the fly. The entire terrain — all the flora and fauna, animals, their ecosystems, the sounds they make, what they look like — and you are really seeing a made up planet that no one in the entire history the universe has ever seen before, including the creators of the video game. It is created completely by the sophisticated imagination software that just comes up with entire planets. 

So that's pretty impressive.

If you're saying that our imagination comes from our memory, where is the computer imagination coming from?

We don't know exactly how that software works because it's a private company, but it has something to do with putting in constraints about the world. Like if you're gonna have predators, there can't be more predators than prey unless there's gonna be an immediate extinction.

So we know some rules about how the world works. And those constraints on what can be put in there. And when you have a lot of these kinds of constraints, that makes the imagination even richer.