The future of AI is not in sentient robots, but it might be in gaming

Machines have been learning to challenge humans in games for years. Poker, chess, Go - the bots have conquered these and more, with investment from some of the world's largest firms.

Most of these games are strategic, but slow: players (and bots) have plenty of time to decide their next move. That is what made the work of research group OpenAI in online team brawler Dota 2 - which requires real-time decision-making between potentially dozens of choices in a single frame - so different.

OpenAI's bots, the OpenAI Five, went head-to-head against teams of professional players at Dota 2's annual championship, The International, this August. Although the bots lost, the matches provided an insight into how reinforcement learning is changing the game when it comes to artificial intelligence.

The OpenAI Five bots were originally limited to Necrophos, Viper, Sniper, Crystal Maiden and Lich

It's safe to say that AI has a reputation in gaming: many gamers consider a match to be an instant loss if they have to play with a bot, and a player disconnect is often accompanied by "GG". However Mike Cook, an AI researcher at Falmouth University, told Computing that is a simplistic view:

"I think game AI is generally quite good, but people often don't appreciate how difficult the task of creating AI for games is. Game developers have a very difficult problem to solve: an AI can't be too good, because it'll beat the player all the time, but it can't be too bad either, or it won't be fun to play against.

"On top of this, game AI has to feel natural and contextual. If I'm playing a game against a bot I want it to feel like it's another human player, not like a piece of code with millisecond reaction times. I want to be able to trick them or distract them or surprise them, none of which make sense for an AI."

Thanks to its virtual years of training, OpenAI's system can in fact act in a similar manner to a human. It utilises tactics like setting traps and focusing fire, and even creep-blocking (a technique whereby a player character slows down friendly minions by walking in front of them, to coalesce multiple groups into one). Modern bots that players are familiar with are incapable of similar actions.

The machine threat

That isn't to say that today's bots are limited by their tech: if they wanted, the developers could make them play perfectly, to the point that it is near-impossible to win against them - but that isn't what a game is about. "The difficulty comes in knowing how to restrict the AI without making it stupid, and how to make it seem intelligent without letting it unleash the perfect machine abilities it has," says Cook.

In a one-on-one situation, today's bots can perform capably; however, their inability to communicate with teammates exposes their weaknesses in co-operative play.

"Co-operative AI is much harder, because we have to guess what the player wants to do, and avoid getting in their way or becoming a liability."

OpenAI's bots have "a huge advantage" over their human counterparts. As well as split-second reaction times (these actually had to be artificially slowed down before the matches) and the fact that they will never fumble a command, they have an almost perfect knowledge of the entire battlefield.

The OpenAI Five are constantly scanning data to make decisions. In the bottom left is a chart tracking their value function, or expectation of a reward: think of it as their confidence in the outcome of a fight

Despite that, the OpenAI Five still lost, which shows that the weaknesses of an AI are still significant: so significant that their advantages don't make up for it yet. Other changes - like switching to a single, mortal courier (a unit that ferries items to players) instead of five invulnerable ones - also certainly had an impact. One of those changes was picking heroes for the AI, in the interests of time, rather than allowing it to draft its own.

Even bots have 'comfort' heroes, just like humans do

"A lot of the bots' advantage came through [the draft] - we saw in earlier exhibition matches that the bots typically started a game with a 90+ per cent confidence they would win, simply based on the heroes they'd chosen to use… But this was a fair change I think: humans are used to playing with the standard 115 hero lineup, so restricting them to just 18 would have invalidated a lot of their knowledge and skill about the game."

Obviously, the difference between a hero pool of 18 and 115 has a significant impact on the balance of the metagame.

"If we remove most of these heroes, as OpenAI have, this balance is thrown off, and humans won't be used to the new relationships between each hero. I think some heroes play to OpenAI's strengths more than others… Gyrocopter and Sniper are two heroes the bots clearly love: heroes with big damaging ultimate spells and high damage. Other heroes, like Tidehunter, feel like they suit the bots' style less, even though they may do well playing against other bots… I think that even bots have 'comfort' heroes, just like humans do!"

A long way to go before Blade Runner

The hallmark of human intelligence is that it can do many things to a standard: we can read, walk, play games and many, many more things. AI platforms, however, are extremely specialised: AlphaGo can't play chess, and OpenAI is incapable of making a cup of tea.

Although some systems composite multiple techniques into a single one (smartphone cameras with face recognition and post-shot editing are a good example), these are not truly a multi-skilled AI.

"We can't keep glueing software together until we end up with an artificial brain," says Cook, "[but] will we ever need to?"

It's a good question. Frankly, we don't need robots that are as multi-talented as humans. A system that could perform many different tasks to achieve a single outcome is certainly desirable (Cook's example is saving people from a burning building), but that AI wouldn't need to know how to play lawn bowls.

There's just no point to multi-talented AI, say experts

"It's often easier to build a system that solves a specific problem rather than trying to build a general one… So one thing that will delay this a lot will be the simple fact that there often won't be an incentive to build something as generally talented as we are."

It's one of the hardest problems humanity is working on right now - it's the cold fusion of computer science

Lack of incentive has never stood in the way of progress, though, especially if it is driven by passion - and multi-talented AI is the sci-fi dream. A robot that can walk the dog, cook dinner and clean the house without - to paraphrase Pratchett - stripping the plaster off the walls and making a furious cup of cat could actually be in the future, now that technologists have both the vision and the money to put towards it.

"We may begin to see systems which can perform a lot of related but distinct tasks over the next decade. There's a few different ways to approach these problems on a structural level: some of them just involve more money and time, while others require the invention of new algorithms and AI techniques.

"It's one of the hardest problems humanity is working on right now, though - along with quantum computing, it's the cold fusion of computer science."

The future of gaming

There is no doubt that OpenAI's show matches were a powerful demonstration of a new approach to artificial intelligence: one that, with time, can challenge the best human players. Challenging a human doesn't equate to thinking like one, though.

We play games against other people to get the best of our opponents: to out-think them, as well as out-play them. "Human-like intelligence doesn't make an AI human in all the other ways that are important for games - being social, being emotional," says Cook.

"That's why we need to appreciate that game AI - all AI, really - has this theatrical element, this layer on top that's all about how the user or the player perceives the system. We don't want our AI to act like real human beings do, we need our AI to be actors: to understand the drama of a situation, what the player wants to achieve or avoid, and what they can do to help make that happen.

"That might involve playing a game really well, but that's just one small aspect of modern AI, and I think truly great game AI will need to embrace these ideas about storytelling and social interaction and modelling the player."

That is why OpenAI - despite its tactical approach in Dota 2 - is more of a demonstration about the future of AI in the real world, rather than gaming. Human-versus-machine contests like those at The International are entertaining and technologically impressive, but don't tell us much about how an AI compares to a human.

"It's useful, it's interesting, it's impressive," says Cook. "But the future of games will need a very different kind of AI."

Developments like OpenAI are useful because they represent a research milestone: a task that developers weren't good enough to accomplish before, but now can. Cook warns, however, "It's important that we remember that the journey is what matters, not the milestone at the end."