ImmediateTestFeedback

This repo serves as the starting point for installing and tweaking the ImmediateTestFeedback Prototype build inside the PLCTE seminar at the software architecture chair of the Hasso-Plattner-Institute. This screencast demonstrates the functionality.

Introduction

Changing software without correctly understanding it often leads to confusion, as developers do not understand how the change corresponds to the new observed behaviour of the system. Today, many software systems are equipped with a test suite. Test suites document code and give feedback on changed program behaviour. We explored ways to use test suites for software comprehension. We implemented a tool that provides additional visualization and gives immediate feedback on software changes. Information about changes in the software and their implications to the test suite are collected using mutation testing. The tool uses this information to present relevant test cases for developers, and additionally prioritize test executions for immediate feedback.

This system uses mutation testing data to present relevant test cases for developers and additionally prioritize test executions for immediate feedback.

Setting Up The Prototype

The prototype system consists of several parts:

• TestingPlugin - The plugin that integrates and visualizes tests into Visual Studio Code
• TestingBackend - The backend that provides the plugin with the necessary information.
• pytest-immediate - The plugin for pytest that enables immediate response times for failing tests and test reordering

Installation Steps

1. Clone the repository you want to get immediate test feedback during development. Examplatory, we will use flask. (The version is pinned to make sure that all test run through with this concrete setup. Unfortunatley, currently flask uses dependencies for its tests on the master branches of other repos like Werkzeug. This makes the setup really flaky.)

   git clone https://github.com/pallets/flask.git
   git checkout e0d7d402
   

   The required datasets for the flask repository are already present in the TestingBackend repository, which simplifies the setup. See Using Custom Data for more information.
2. Create a virtual environment that can run the test using pytest Example for flask:

   cd flask
   python -m venv venv
   . venv/bin/activate
   pip install -e . -r requirements/dev.txt 
   

   A run of pytest should now complete without errors.
3. Clone this repository including all submodules.

   git clone --recurse-submodules https://github.com/XPerianer/ImmediateTestFeedback
   

4. Install pytest-immediate into the flask virtual environment.

   cd flask
   . venv/bin/activate
   cd ../pytest-immediate
   pip install . -r requirements.txt
   

   If you now call pytest in the flask folder, it should mention immediate under plugins.
5. Install and start the Backend:

   cd TestingBackend
   

   and follow the instructions in the TestingBackend repository
6. Install and start the Frontend:

   cd TestingPlugin
   

   and follow the instructions in the TestingPlugin repository

Datasets

The relevancy data, as well as the prediction model for predicting failing tests can be customized to other repositories.

For the installation steps, we used the standard data which is already included in the TestingBackend repo.

Using Custom Data

There are two different datasets needed for the prototype:

• The mutation testing dataset
• The Prediction Model

The Mutation Testing Dataset

The mutation testing dataset can be generated by Mutester. The Mutester repo also has additional information on setup requirements and optional parameters.

Based on the mutation testing dataset the prediction model is generated, that can be loaded by the TestingBackend. This gives the possibility to quickly try out different predictor models. It also enables use other embeddings than TSNE for the visualization of tests.

The Predictor Model

Given the mutation testing dataset the necessary files can be generated by using the preprocessing jupyter notebook. Its dependencies can be installed in a virtual environment with these commands:

python -m venv venv
. venv/bin/activate
pip install -r requirements.txt

Now the notebook can be run

jupyter notebook preprocess.ipynb

Follow it to create the necessary files, that can then be inserted into the TestingBackend.