Writing tests for the test runner

The test runner

BAR has a framework for creating automatic tests to automatize checking the game features.

This document explains the basics for using them, you should already be familiar with the engine api and bar api extensions to understand everything here.

Running the tests

To run all tests, you need to write /runtests at the console. A single test, or set of tests can also be run by adding a parameter, like /runtests dgun.

Note the parameter will match all files containing that pattern in the file name.

All tests are currently under luaui/Widgets/Tests. There are also introductory examples at luaui/Widgets/TestsExamples.

Writing tests

Basic structure

This is the basic structure for a test file:

-- skip allows adding some checks to see if the test should run
function skip()
end

--- setup is where you should put all your initialization code
function setup()
end

-- cleanup gets called after the test is run to cleanup after the test
function cleanup()
end

-- this is the main function for the test, it will be run
function test()
end

The test_runner will call those functions in the following order: skip, setup, test, cleanup.

More advanced test structure

A more real-ish example with some code at the test top methods.

-- in case this is a widget test, it's good practice to add the name here
local widgetName = "Blueprint"
-- will be storing the widget here later so all functions have access to it
local widget

function skip()
    -- here we are checking to see if the game started
    -- just an example since normally the test_runner will make sure the game already started before running the test
    return Spring.GetGameFrame() > 0
end

function setup()
    -- use this to clear the map and make sure there's nothing unexpected around left over from other tests
    Test.clearMap()

    -- use this to fetch a widget and enable 'locals' access, this gives access to top level locals from the widget file
    widget = Test.prepareWidget(widgetName)
end

function cleanup()
    -- it's good practice to cleanup after the test too
    Test.clearMap()
end

function test()
    -- an example simple test (has nothing to do with the above widget so as to not complicate the example)
    local x, z = Game.mapSizeX / 2, Game.mapSizeZ / 2
    local y = SyncedProxy.Spring.GetGroundHeight(x, z)
    local teamID = Spring.GetMyTeamID()
    local unitID = SyncedProxy.Spring.CreateUnit("armpw", x, y, z, 1, teamID)
    -- now comes the heart of tests, the asserts. they will check some condition and abort the test with an error message if the condition is false or nil.
    assert(unitID)
    assert(Spring.ValidUnitID(unitID))
end

Further examples

Check luaui/Widgets/TestsExamples

The Testing api

The test_runner provides some extra api to help in several common testing situations, it's accessed through the Testmodule.

It also exports some extra globals, with functionality to run synced code and some extra functions not available elsewhere.

The Test module

Exports the following methods:

Basic methods

• Test.clearMap(): Destroys all units in the map.
• Test.prepareWidget(widgetName): Returns a widget with locals enabled.

Wait methods

important: you can't use SyncedRun or SyncedProxy inside these methods.

• Test.waitUntil(f, timeout): Wait until function f is called, or timeout frames have passed.
• Test.waitFrames(frames): Make the test wait a frames number of frames.
• Test.waitTime(milliseconds, timeout): Make the test wait the specified milliseconds, or timeout frames, whatever happens earlier.

Callin methods

Methods to register callins and wait for them to be called and do specific tests inside them.

• Test.expectCallin(name, countOnly): Use this inside setup if you will need to use name callin.
• Test.waitUntilCallin(name, predicate, timeout, count): Wait until the name callin is called by the engine, and then see if the predicate function returns true. Set a timeout for a maximum number of frames. Use count when you want predicate to be counted for a certain number of successes. predicate here can be set to nil, and then it will just wait for the first call, or for count number of calls if you specified that.
• Test.waitUntilCallinArgs(name, expectedArgs, timeout, count): Wait until the name callin runs, and then compare expectedArgs (table-array) to the ones passed to the callin. Any nil arguments inside expectedArgs will be skipped, and the other ones will be required to match the ones from the callin. Set a timeout for a maximum number of frames. Use count when you want the match to be done a certain number of times instead of just one.

Advanced methods

Advanced methods you normally won't need.

• Test.spy(...): ???
• Test.mock(...): ???
• Test.setUnsafeCallins(unsafe): Use this to avoid the need to use expectCallin, also in that mode the callin system won't be prerecording callin calls. This can be lighter for bigger tests, but also will mean you could miss some callins because they run before you called waitUntilCallin*. Use with care.

Mostly for internal use

These ones are usually cleanup methods, and the test_runner will call them automatically for you.

In some cases you might need to call them yourself, so here they are.

• Test.unexpectCallin(name): Use this when you're done using a callin
• Test.clearCallins(): Clear all registered callins.
• Test.clearCallinBuffer(name): Clear the callin buffer for callin. Normally the test_runner is recording previous calls to the callin, use this when you need to clear that so your waitForCallin will be triggered by new events only.
• Test.restoreWidget(widgetName): Use this to restore a widget to its normal state after using prepareWidget.
• Test.restoreWidgets(): Use this to restore all widgets that used prepareWidget.

Running synced code

To create tests you will be needing to run some synced code, since otherwise due to the permissioned lua api you won't be allowed to do most things, like creating enemies or testing shadowed parts of the map.

The test_runner provides two methods to accomplish this: SyncedProxy and SyncedRun

Note for the moment you can't use pcall inside these methods.

SyncedProxy

SyncedProxy can be used as prefix to any Spring.* call, and that will make the call happen in synced space.

Example:

local unitID = SyncedProxy.Spring.CreateUnit("armpw", x, y, z, 1, teamID)

SyncedProxy adds a lot of overhead for every call, so when you need a lot of synced run you can use SyncedRun

SyncedRun

SyncedRun will run a function you provide in synced space. This is very convenient, but has the drawback the function won't accept any arguments, instead it will give you access to the synced locals from the test you're running.

It will also return whatever the function you pass to it returns.

example:

local x, z = Game.mapSizeX / 2, Game.mapSizeZ / 2
local y = SyncedRun(function()
   local height = Spring.GetGroundHeight(locals.x, locals.z)
   return height
end)

Extra assertions

Besides assert, already provided by lua, the test environment provides a few extra assertions you can use.

These are defined at common/testing/assertions.lua.

assertTablesEqual(table1, table2, margin, visited, path)

Compare if two tables are equal.

• table2, table2: the tables
• margin: margin to apply for number comparison
• visited: optional array where visited items will be returned
• path: optional path array to test inside the array

assertSuccessBefore

assertSuccessBefore(seconds, frames, fn, errorMsg)

Assert the given fn returns trueish before seconds, tested every frames number of frames. Use errorMsg if you want a custom message when this fails.

fn will be called every 'frames' game frames. errorMsg can be set to customize the error message preface.

assertThrows

important: You can't use SyncedProxy, SyncedRun or the Test.waitUntil* methods inside this.

assertThrows(fn, errorMsg)

Assert the given fn throws a lua error. Use errorMsg if you want a custom message when this fails.

assertThrowsMessage

important: You can't use SyncedProxy, SyncedRun or the Test.waitUntil* methods inside this.

assertThrowsMessage(fn, testMsg, errorMsg)

Assert the given fn throws a lua error with a specific message testMsg. Use errorMsg if you want a custom message when this fails.

Note this function will cut the normal prefix from lua traces, like for example, to test:

[string "LuaUI/Widgets/tests/selftests/test_assertions.lua"]:17: error2

You must simply test for "error2". (ie, the actual error)