Log of my adventure creating an AI Mortal Kombat player (round 2)

If you haven’t read the round 1article yet, I recommend you check it out before you start reading this article. If you already read round one, let’s get started.

Picture by Marlon Marçal on Behance

Introduction

After discussing my ideas for this project a bit on Reddit, some fellows told me that Google Colab supports JavaScript code in the environment by using ijavascript kernel. So it’s not necessary to migrate the game environment code to Python. The only drawback with this approach is that the use of GPUs and TPUs is not optimized so well for JavaScript.

Ijavascript logo by their Github

So, after investigating the pros and cons, I decided to use colab for javascript since using a different environment for training will probably cause an overfitting, in fact, the most important thing for training is the quality of the data we use, and in our case, the “dataset” is the environment in which the agent “learns” with its possible actions.

“We don’t have better algorithms. We just have more data.” — Peter Norvig, director of research at Google.

So our plan is going to change course completely by removing the Python code from the equation.

Running javascript in google colab

In the first block of your file .ipynb add these command lines.

# After running this code, reload the page

#downloads and installs ijavascript as a node package
!npm install -g --unsafe-perm ijavascript

# installs ijavascript globally
!ijsinstall --install=global

The ijavascript kernel doesn’t have clean and simple way to execute shell commands, so we can use the following helper function.

var { spawn } = require('child_process');
var sh = (cmd) => {
  $$.async();
  var sp = spawn(cmd, { cwd: process.cwd(), stdio: 'pipe', shell:true, encoding: 'utf-8' });
  sp.stdout.on('data', data => console.log(data.toString()));     sp.stderr.on('data', data => console.error(data.toString()));  sp.on('close', () => $$.done());
};
sh('npm init -y');

This is a Colab open example.

Steps (Update)

1. Fork mk.js project into my Github.
2. Adapt the project for my propose.
3. ̶M̶i̶g̶r̶a̶t̶e̶ ̶t̶h̶e̶ ̶g̶a̶m̶e̶ ̶e̶n̶v̶i̶r̶o̶n̶m̶e̶n̶t̶ ̶f̶r̶o̶m̶ ̶J̶S̶ ̶t̶o̶ ̶P̶y̶t̶h̶o̶n̶.̶ Move the javascript source to Colab using ijavascript kernel.
4. Develop an agent using Deep Q-learning techniques in Google Colab.
5. Export and convert the agent in a Tensorflow.js model.
6. Import the model in the JS app and setup the controls (to allow the agent to control one fighter).
7. Deploy the app.

Colab integration

Finally, I started the migration from the js app to Colab, but I realized that migrating the code from a browser application to an interpreter is not as simple as it seems, after investigating a bit, seems like the javascript scopes in the ijavascript environment works a bit different, so I had to rewrite many of the functionalities.

Photo by Brett Jordan on Unsplash

After a few headaches, I was finally able to create a standalone gaming environment that we can interact with via 3 functions.

• startGame()
• getState(index)
• dispatchAction(action)

startGame()

Starts a new fight, this function will be useful to restart the fight on every iteration of our reinforcement learning algorithm.

getState(index)

The getState function returns the actual state of the game (position and life of each fighter) from a given index, where index === 0 represents Player 1 (Subzero) and index === 1 represents Player 2 (Kano).

This function is useful to update the current state of the environment during the agent training, for instance, if the opponent gets close to you, you should probably take an evade or hit action.

dispatchAction(action)

This is probably the most important one. To allow the agent to interact with the environment and generate actions that produce a reward or punishment, we need an action callback. In this case, the parameter is an “action” that is represented by an ASCII code from the keyboard, these codes is mapped as follows.

And the action will be executed using the following function

Modeling

Once the integration was achieved and having discarded all the UI functionalities, we can start the training process.

If I want to train this by playing against myself, I will spend hours playing the game blindly (without UI) and programmatically, but if I train it by playing against a player who doesn’t move, the agent will learn just to get close and hit.

So the solution is … more machine learning! By creating two neural networks that fight each other and share memory of the game. Many reinforcement learning algorithms use this approach to train the model, such as AlphaZero.

Finally, our reinforcement learning diagram looks like this

Model representation by the author

Testing the game

The first test I did to verify if it’s working fine is a simple fight simulation, I bring them closer and once the agents are close enough I make them fight until one wins, it’s a dummy behavior, but works fine.

Once we create two agents that interact with the environment, the next thing we must do is to make their actions being evaluated according to the state of the environment and their historical (action, reward) peers, and not according to an arbitrary conditional.

Conclusion

Creating an environment for agents to interact is an arduous task that requires a lot of time and effort, so it’s always recommended (when possible) to try to use environments that already exist or variants of them.

Fortunately there are many libraries available to solve this problem such as OpenAI Gym or DeepMind OpenSepiel.

In the next article I will try to develop the deep q learning algorithm, to allow our agent to learn to fight without predefined rules.

Then I will analyze the results to see if the learning really works and creates emergent properties that maximize your chance of winning a fight.

You can check all the colab source code here and the game github here.