The Role of Evolutionary Computation in Game AI

Discover the world at Leiden UniversityDiscover the world at Leiden University
The Role of
Evolutionary Computation
in Game AI
Mike Preuss @ S3 ACM GECCO summer school, Praha
July 10, 2019

Discover the world at Leiden University
● my roots in Evolutionary Computation
● game AI today: a forming field
● Procedural Content Generation
● Balancing
● GVGAI/competitions/rolling horizon EA
roadmap
2

Evolutionary Computation and me
●most likely the last PhD student of Hans-Paul
Schwefel (co-inventor of the Evolution Strategy)
●most important message: expect the unexpected
●focus on multi-modal optimization (detect
several good, different solutions simultaneously)
●a bit on multi-objective optimization
●some work on experimental methodology
3

PhD (2013) and book (2015)
most important results:
●the advantage of niching (if any) can only come from
coordinated placement
●in contrast: SLS is randomized placement
●simulations/theory: unfortunately, we cannot speed
up finding the global optimum
●but if we go for coverage (find most if not all good
optima) we can gain a lot
●invented NEA2 as top algorithm (2015), lesson
learned: for niching, also use the objective values
modern name for this stuff: quality diversity
4

GECCO multimodal optimization competition
• builds on a fixed selection of 20 problems made
from 12 functions in different dimensions
• averages over 3 different rankings:
- average PR (peak rate) values
- static F1 (takes the amount of points in the
answer set into account)
- dynamic F1 (same but as integral over time
for the whole runtime)
• only global optima are sought
• the number of optima is low for most problems
http://www.epitropakis.co.uk/gecco2019/
5

what we did not expect so early
6

bla
a forming field
a forming field

what about
game AI?
current game AI is:
• playing well
• generating content
• modeling players
see 2018 book:
Artificial Intelligence
and Games, by
Yannakakis/Togelius
http://gameaibook.org/book.pdf
video: fully generated
game worlds
in No Man’s Sky (2016)
8

when game AI becomes… artificially intelligent
ongoing discussion in game AI: how intelligent is current game AI?
see recent gamasutra post:
http://www.gamasutra.com/view/news/253974/When_artificial_intelligence_in_video_games_becomesartificially_intellige
nt.php
"My prediction is a little bit controversial.
I think the next giant leap of game AI is actually artificial intelligence."
- Alex J. Champandard at GDC 2013
Ken de Jong (at GECCO 2017, Berlin) on the question:
What do you see as dominant research direction in Evolutionary Computation for
the next years?
“It is time to apply our methods to real problems now.”
9

classical AI techniques in games
even in the early 2000s, lots of standard techniques in use (and developers showing
little interest for a change):
• finite state machines (FSM)
• hierarchical FSM
• scripting
basically mostly used for Non-Player Character (NPC) control, also:
• cellular automata
• influence maps
• basic optimization approaches
• game theory
(still useful for the right problem, but we have so much more now)
10

modern approach
• we treat all aspects of games, now also game design/production
• we use what fits best for a specific problem
everything is an agent:
AI director, opponent AI controller, content generator, ....
game AI
specific algorithms
classic AI /
machine learning
in-game AI use
computational intelligence
game design/production
11

game ai overall view
• game AI traditionally encompasses mostly NPC behavior and bots
• control happens during the game, design support before deployment
• design support is rather new, but a highly dynamic research area
game AI
design supportcontrol
directors
strategic (bots)
/supportive AI
PCGNPC behavior balancing
NPC = non-player character, PCG = procedural content generation
12

panoramic view
game AI components, player centric view, from:
Yannakakis, G.N.; Togelius, J., "A Panorama of Artificial and Computational Intelligence in Games," in IEEE
TCIAIG, 2014
13

why apply CI methods to games?
contrary to board games,
• game trees often not applicable
• incomplete information
• concurrency: during planning phase, game situation
changes
• quantifying a game situation is not trivial
➔ good and fast approximations are needed
evolutionary optimization is
• versatile, flexible, still works (somehow)
• copes with noise and strange search spaces
• can be asked to deliver a result at any time
14

Evolutionary and AI
•not fully up to date, but clearly shows relations
•boundaries much more fuzzy nowadays
15

CI and games: a field forming
• 1999: Blondie24, learning checkers with CI
• 2005 first Computational Intelligence in Games (CIG) conference
• IEEE TCIAIG Journal (Transactions on CI and Artificial
Intelligence in Games) since 2009
• EvoGames track in Evo*, DETA track in GECCO since 2009/2011
• many “neighbor” conferences, etc. AIIDE, FDG, gameai/nucl.ai
conf. (not strictly CI, but CI welcome)
• general approach target oriented, not technique oriented
• 2012: first Dagstuhl seminar on AI and CI in Games
• 2015: second Dagstuhl seminar (integration)
• 2017: third Dagstuhl seminar (AI-supported design)
• 2018/2019: journal and conference renamed to IEEE Transactions
on Games and IEEE Conference on Games
• 2019: first Shonan seminar on AI for Games
16

one small step for [a] man….
what recently happened:
17
AlphaGo: Go
(MCTS, expert knowledge, self-play)
AlphaGoZero: Go
(MCTS-, self-play)
Deep Reinforcement Learning: ALE
(pixel input, self-play)
AlphaStar: StarCraft
(MCTS?, pixel input, self-play, expert knowledge)
AlphaZero: Go, Chess, Shogi
(MCTS-, self-play)
Go-Explore: Montezuma
(population based exploration)
Obstacle Tower Challenge
(population based training)
ForTheWin: Capture the Flag
(2 speed rnns, population based
learning, population based training)

combining EC and Deep Reinforcement Learning
•some (limited) successes in weight optimization with EC
•population-based learning:
- GANs, multiple agent architectures
- resembles co-evolutionary approaches, same problems (how to control)
•population-based based training:
- generating test cases as needed
- procedural content generation (PCG), mainly done via EC methods
•not really exploited yet: multi-modal/multi-objective approaches
•maybe more things to transfer?
18

what I did in Game AI
●procedural generation of strategy game maps
● organizing StarCraft I competitions, survey
● some works on balancing (also using EC):
○there are usually many solutions in balancing
○the problem is very multi-modal
●recently: facet orchestration (how to combine
several generators, e.g. visuals, audio)
●very recently: transfer AlphaAnything to new
problems (not from the games domain)
●understand how it works on small games
19

Obstacle Tower Challenge
current DRL challenge
for artificial general
intelligence (AGI)
• levels increase in
difficulty
• levels procedurally
generated
• decorations change
• includes minigames
(Sokoban)
runs in 2 phases:
- qualification (gone)
- finals (up to July
15th, 2019)
https://www.aicrowd.com/challenges/unity-
obstacle-tower-challenge
20

bla
Procedural Content Generation

search-based vs. generate-and-test
22

Evolutionary Algorithms
• initial solutions needed: existing or randomly generated solutions
• fitness function shall measure the targeted properties (designer input)
• matching variation operators need to be established
23

why using EAs in PCG?
pros:
- flexible, only needs good utility function
- not very sensitive concerning noise/mistakes
- anytime method: we can obtain a result whenever needed
cons:
- optimality cannot be guaranteed (except for very few special cases)
- stagnation phases of unpredictable length (break?)
- depends on matching representation and operators
- slow on simple problems
24

back in 2009: galactic arms race
25

how are the new weapons produced?
• very small population, 3 solutions (candidates, individuals)
• interactive selection: player removes existing solution when new one is generated
• weapons are created via a parametrized particle system by complex neural
network
• cgNEAT: neural network topology optimized by means of evolutionary algorithm
26

a sample gallery
27

StarCraft I map evolution (2009-2012)
28

RTS map design with Sentient Sketchbook
Preuss, Liapis, Togelius. Searching for Good and Diverse Game Levels. IEEE CIG 2014.
29

gamesbyangelina and the procjam
• Michael Cook’s AI that is designed to intelligently design video games:
http://www.gamesbyangelina.org/
• PROCJAM: yearly game jam with strong emphasis on PCG
http://itch.io/jam/procjam
• coming soon: facet orchestration
30

Project seminar
nd
balancing

Game AI hot topics
• procedural content generation (PCG)
- large game worlds are expensive to make
- represent specific (designer) styles
• believability of non-player character (NPC) behavior
- more humanlike behavior (also evoke and show emotions)
- better cooperation of units (team AI)
- human/AI interaction
• personalization
- preference modeling (what do they like?), player type analysis
- dynamic adaptation of game content and mechanics
balancing is
everywhere
32

balancing project with BlueByte
• prototype from BlueByte
• ZombieVillage game, related to tower defense
• several parameters: can we balance automatically?
33

balancing project with BlueByte
• number of (most important) parameters reduced to 5
• interchange between manual and automated (integrated) balancing
• core result: both steps needed, many different solutions exist
Marlene Beyer, Aleksandr Agureikin, Alexander Anokhin, Christoph Laenger, Felix Nolte, Jonas Winterberg, Marcel Renka,
Martin Rieger, Nicolas Pflanzl, Mike Preuss, Vanessa Volz: An integrated process for game balancing. CIG 2016
34

tackling a complex RTS
Mike Preuss, Thomas Pfeiffer, Nicolas Pflanzl, Vanessa Volz: Integrated Balancing of an RTS Game:
Case Study and Toolbox Refinement. CIG 2018
35

bla
nd
general game AI/competitions

can we also have general (game) AI?
• instead of this:
• we rather want this:
game 1
game 2
game 3AI 3
AI 2
AI 1
develop for
develop for
develop for
game 1
game 2
game 3
AI 1
learn to play
learn to play
learn to play
37

GVGAI ENVIRONMENT
• general video-game playing (GVGP) succeeds general game-playing
• targeted at more complex, non-deterministic games
• language first shaped during Dagstuhl seminar CI and AI in games 2012
• we want to obtain controllers that can play “any game”
• important publications:
John Levine, Clare B. Congdon, Michal Bída, Marc Ebner, Graham Kendall, Simon Lucas, Risto Miikkulainen,
Tom Schaul, and Tommy Thompson. General Video Game Playing. Dagstuhl Follow-up, 6:1–7, 2013.
Tom Schaul. A Video Game Description Language for Model-based or Interactive Learning. In Proceedings of the
IEEE Conference on Computational Intelligence in Games, Niagara Falls, 2013. IEEE Press
http://www.gvgai.net
38

GVGAI competition
• build general AI controllers
• learn to play many games, not one
• many different challenges
• many successful controllers
use Monte-Carlo Tree search
• but Rolling Horizon EA also an option
• but how can this work?
• “traditional track”: 1-player planning
• employs forward model (action simulation, may be non-deterministic)
• new: 2-player planning, learning (no forward model)
• now also: level generation, rule generation
39

some example games
Infection:
Missile Command:
Pacman
40

VGDL – game and level definitions
41

A SAMPLE (RANDOM) CONTROLLER
42

Rolling Horizon EA for search based planning
• basic idea: evolve whole sequence of moves at once, apply only first action
• hybrids with MCTS like playouts are also possible
• simple extension with shift buffer (only remove latest action) performs better
Raluca Gaina, Simon Lucas, Diego Perez. Rolling Horizon Evolution Enhancements in General Video
Game Playing. IEEE CIG 2017
43

current competitions held at COG 2019
• 4th Angry Birds Level Generation Competition
• Bot Bowl I
• Fighting Game AI Competition
• First TextWorld Problems: A Reinforcement and Language Learning Challenge
• Geometry Friends Game AI Competition
• General Video Game AI Competitions (Learning Track)
• Hanabi Competition
• Hearthstone AI competition
• MicroRTS AI Competition
• Short Video Competition
• StarCraft AI Competition
• Strategy Card Game AI Competition
lots of demo and explanation videos included:
http://ieee-cog.org/competitions_conference/
44

conclusions
•EC is part of AI and has always been
•Procedural Content Generation getting very important for games
industry but also for Deep Learning
•evolutionary approaches have great potential for improving other
algorithms, e.g. RHEA (rolling horizon EA) instead of MCTS
•3 important applications of EC algorithms in game AI:
PCG, balancing, search-based planning
45

Discover the world at Leiden UniversityDiscover the world at Leiden University
Don’t
panic!
46

The Role of Evolutionary Computation in Game AI

Recommandé

Recommandé

Contenu connexe

Similaire à The Role of Evolutionary Computation in Game AI

Similaire à The Role of Evolutionary Computation in Game AI (20)

Dernier

Dernier (20)

The Role of Evolutionary Computation in Game AI