Assignment Test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a9308047",
   "metadata": {},
   "source": [
    "# Assignment Test Tutorial\n",
    "\n",
    "The ``assignmenttest`` package contains some useful tools for creating test cases for student submissions.\n",
    "\n",
    "This notebook explains some of the base functionality."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64c04735",
   "metadata": {},
   "source": [
    "## 1. Testing a single variable\n",
    "\n",
    "The most common test for an assignment is to test if a certain variable contains the correct value.\n",
    "\n",
    "For this we need a ``VariableTest``.\n",
    "\n",
    "First we need to initialize the test with the following parameters:\n",
    "\n",
    "- ``namespace``: Where should the tester look for the variable that should be tested. We almost always set this to ``globals()``, which is the global namespace\n",
    "- ``r_tol``: Optional parameter, defaults to $0$. The relative error tolerance. If this is set to ``0.01`` it means the student answer can have a relative error of up to $1\\%$. \n",
    "- ``a_tol``: Optional parameter, defaults to $0$. The absolute error tolerance. If this is set to $10$ it means the student answer is seen as correct if $abs(\\mathrm{correct\\_answer} - \\mathrm{student\\_answer}) \\leq 10$.\n",
    "\n",
    "Next we need to create a list of test cases.\n",
    "\n",
    "Each test case is a dictionary with the following keys:\n",
    "\n",
    "Required keys:\n",
    "- ``name``: The name of the variable we want to test as a string\n",
    "- ``expected``: The expected value of the variable\n",
    "\n",
    "Optional keys:\n",
    "- ``expected_type``: The data type the variable should have. This could be ``int``, ``float``, ``numbers.Number``, ``list``, etc.\n",
    "- ``comparator``: A function that tells the tester how to compare the student answer to the reference answer. This function takes two arguments (student_answer, reference_answer) and returns an error tuple (absolute_error, relative_error). This is often used for more complex data types such as dictionaries or lists."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7499f7e4",
   "metadata": {},
   "source": [
    "## Example 1.1)\n",
    "\n",
    "Testing a single numeric answer called ``my_answer``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e7e1edee",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Student answer\n",
    "\n",
    "my_answer = 15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "45249d34",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============================================================\n",
      "Variable Test\n",
      "\n",
      "------------------------------------------------------------\n",
      "Test for variable my_answer failed\n",
      "Expected 0.006\n",
      "Got 15\n",
      "rel_error = 2.4990e+03, abs_error = 1.4994e+01\n",
      "------------------------------------------------------------\n",
      "\n",
      "------------------------------------------------------------\n",
      "Test for variable my_answer failed\n",
      "Expected 0.012\n",
      "Got 15\n",
      "rel_error = 1.2490e+03, abs_error = 1.4988e+01\n",
      "------------------------------------------------------------\n",
      "\n",
      "============================================================\n",
      "0 / 2 tests passed!\n",
      "============================================================\n",
      "### BEGIN GRADE\n",
      "0.0\n",
      "### END GRADE\n"
     ]
    }
   ],
   "source": [
    "### Test for student answer\n",
    "from assignmenttest import VariableTest, grade_report\n",
    "\n",
    "tester = VariableTest(\n",
    "    namespace=globals(),\n",
    "    r_tol=0.01\n",
    ")\n",
    "\n",
    "test_cases = [\n",
    "    {\n",
    "        'name': 'my_answer',\n",
    "        'expected': 0.006,\n",
    "    },\n",
    "    {\n",
    "        'name': 'my_answer',\n",
    "        'expected': 0.012,\n",
    "    },\n",
    "]\n",
    "\n",
    "\n",
    "\n",
    "percentage_passed = tester.test(test_cases)\n",
    "\n",
    "# This function takes the percentage of test cases that have been passed\n",
    "# and the number of points for this question and prints the grade for the student\n",
    "\n",
    "grade_report(percentage_passed, points=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8881c53e",
   "metadata": {},
   "source": [
    "## Example 1.2)\n",
    "\n",
    "Testing a single numeric answer called ``my_answer1`` which the student did not define. Notice how the test case fails without throwing an error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "894defdf",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Student answer\n",
    "\n",
    "my_answer11 = 15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2a71afc9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============================================================\n",
      "Variable Test\n",
      "\n",
      "------------------------------------------------------------\n",
      "Test for variable my_answer1 failed\n",
      "Variable my_answer1 is not defined!\n",
      "------------------------------------------------------------\n",
      "\n",
      "============================================================\n",
      "0 / 1 tests passed!\n",
      "============================================================\n",
      "### BEGIN GRADE\n",
      "0.0\n",
      "### END GRADE\n"
     ]
    }
   ],
   "source": [
    "### Test for student answer\n",
    "from assignmenttest import VariableTest, grade_report\n",
    "\n",
    "tester = VariableTest(\n",
    "    namespace=globals(),\n",
    "    r_tol=0.01\n",
    ")\n",
    "\n",
    "test_cases = [\n",
    "    {\n",
    "        'name': 'my_answer1',\n",
    "        'expected': 15\n",
    "    }\n",
    "]\n",
    "\n",
    "\n",
    "\n",
    "percentage_passed = tester.test(test_cases)\n",
    "\n",
    "# This function takes the percentage of test cases that have been passed\n",
    "# and the number of points for this question and prints the grade for the student\n",
    "\n",
    "grade_report(percentage_passed, points=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d6efb10",
   "metadata": {},
   "source": [
    "## Example 1.3)\n",
    "\n",
    "Testing a single numeric answer called ``my_answer2`` with type checking. \n",
    "\n",
    "Notice how the second test case for ``my_answer3`` fails because the student gave the answer as a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "138f2038",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Student answer\n",
    "\n",
    "my_answer2 = 13\n",
    "\n",
    "my_answer3 = '15'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ef98fd1c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============================================================\n",
      "Variable Test\n",
      "\n",
      "------------------------------------------------------------\n",
      "Test for variable my_answer3 failed\n",
      "Variable my_answer3 is not of type <class 'numbers.Number'>!\n",
      "------------------------------------------------------------\n",
      "\n",
      "============================================================\n",
      "1 / 2 tests passed!\n",
      "============================================================\n",
      "### BEGIN GRADE\n",
      "5.0\n",
      "### END GRADE\n"
     ]
    }
   ],
   "source": [
    "### Test for student answer\n",
    "from numbers import Number\n",
    "from assignmenttest import VariableTest, grade_report\n",
    "\n",
    "tester = VariableTest(\n",
    "    namespace=globals(),\n",
    "    r_tol=0.01\n",
    ")\n",
    "\n",
    "test_cases = [\n",
    "    {\n",
    "        'name': 'my_answer2',\n",
    "        'expected': 13,\n",
    "        'expected_type': Number\n",
    "    },\n",
    "    {\n",
    "        'name': 'my_answer3',\n",
    "        'expected': 15,\n",
    "        'expected_type': Number\n",
    "    },\n",
    "]\n",
    "\n",
    "\n",
    "\n",
    "percentage_passed = tester.test(test_cases)\n",
    "\n",
    "# This function takes the percentage of test cases that have been passed\n",
    "# and the number of points for this question and prints the grade for the student\n",
    "\n",
    "grade_report(percentage_passed, points=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98959930",
   "metadata": {},
   "source": [
    "## Example 1.4)\n",
    "\n",
    "Testing a list with a custom comparator.\n",
    "\n",
    "The student answer is called ``my_list``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "08543f49",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Student answer\n",
    "\n",
    "my_list = [1, 2, 3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9177e887",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============================================================\n",
      "Variable Test\n",
      "\n",
      "============================================================\n",
      "2 / 2 tests passed!\n",
      "============================================================\n",
      "### BEGIN GRADE\n",
      "10.0\n",
      "### END GRADE\n"
     ]
    }
   ],
   "source": [
    "### Test for student answer\n",
    "from numbers import Number\n",
    "from assignmenttest import VariableTest, grade_report\n",
    "\n",
    "tester = VariableTest(\n",
    "    namespace=globals(),\n",
    "    r_tol=0.01\n",
    ")\n",
    "\n",
    "def compare_list_lengths(student_answer, reference_answer):\n",
    "    absolute_error = abs(len(student_answer) - len(reference_answer))\n",
    "    relative_error = absolute_error / len(reference_answer)\n",
    "    return absolute_error, relative_error\n",
    "\n",
    "def compare_lists(student_answer, reference_answer):\n",
    "    absolute_error = 0\n",
    "    relative_error = 0\n",
    "\n",
    "    \n",
    "    for idx, elem in enumerate(reference_answer):\n",
    "        if idx >= len(student_answer):\n",
    "            absolute_error += 1\n",
    "        if student_answer[idx] != elem:\n",
    "            absolute_error += 1\n",
    "    relative_error = absolute_error / len(reference_answer)\n",
    "    return absolute_error, relative_error\n",
    "\n",
    "test_cases = [\n",
    "    {\n",
    "        'name': 'my_list',\n",
    "        'expected': [1, 2, 3],\n",
    "        'expected_type': list,\n",
    "        'comparator': compare_list_lengths\n",
    "    },\n",
    "    {\n",
    "        'name': 'my_list',\n",
    "        'expected': [1, 2, 3],\n",
    "        'expected_type': list,\n",
    "        'comparator': compare_lists\n",
    "    },\n",
    "]\n",
    "\n",
    "\n",
    "\n",
    "percentage_passed = tester.test(test_cases)\n",
    "\n",
    "# This function takes the percentage of test cases that have been passed\n",
    "# and the number of points for this question and prints the grade for the student\n",
    "\n",
    "grade_report(percentage_passed, points=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0fb5e19",
   "metadata": {},
   "source": [
    "## 2. Testing a function\n",
    "\n",
    "We often want students to implement a function. For this we can use the ``FunctionTest`` class of the ``assignmenttest`` package.\n",
    "\n",
    "First we need to initialize the test with the following parameters:\n",
    "\n",
    "- ``namespace``: Where should the tester look for the variable that should be tested. We almost always set this to ``globals()``, which is the global namespace\n",
    "- ``function_name``: The name of the function the student should implement as a string\n",
    "- ``reference_function``: Optional pointer to a reference function.\n",
    "- ``comparator``: Optional, default is set to compare numbers\n",
    "- ``r_tol``: Optional parameter, defaults to $0$. The relative error tolerance. If this is set to ``0.01`` it means the student answer can have a relative error of up to $1\\%$. \n",
    "- ``a_tol``: Optional parameter, defaults to $0$. The absolute error tolerance. If this is set to $10$ it means the student answer is seen as correct if $abs(\\mathrm{correct\\_answer} - \\mathrm{student\\_answer}) \\leq 10$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5853f60e",
   "metadata": {},
   "source": [
    "## Example 2.1)\n",
    "\n",
    "Testing a simple function.\n",
    "\n",
    "The student should implement a function ``square`` that takes one parameter and returns the square of a number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b4ce92af",
   "metadata": {},
   "outputs": [],
   "source": [
    "## student answer\n",
    "\n",
    "def square(x):\n",
    "    print('='*10000)\n",
    "    print(x**2)\n",
    "\n",
    "#square(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "5bac0543",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============================================================\n",
      "Test for function square\n",
      "\n",
      "square does not have a return statement!\n",
      "============================================================\n",
      "0 / 4 tests passed!\n",
      "============================================================\n",
      "### BEGIN GRADE\n",
      "0.0\n",
      "### END GRADE\n"
     ]
    }
   ],
   "source": [
    "## test for student answer\n",
    "from assignmenttest import FunctionTest, grade_report\n",
    "\n",
    "def square_ref(x):\n",
    "    return x**2\n",
    "\n",
    "tester = FunctionTest(\n",
    "    namespace=globals(),\n",
    "    function_name='square',\n",
    "    reference_function=square_ref,\n",
    "    r_tol=0.05\n",
    ")\n",
    "\n",
    "test_cases = [\n",
    "    {\n",
    "        'arg': 5\n",
    "    },\n",
    "    {\n",
    "        'arg': 7,\n",
    "        'target': 49 # Instead of providing a reference function we can provide the wanted output in the test case\n",
    "    },\n",
    "    {\n",
    "        'args': [3]\n",
    "    },\n",
    "    {\n",
    "        'kwargs': {'x': 3}\n",
    "    }\n",
    "]\n",
    "\n",
    "percentage_passed = tester.test(test_cases)\n",
    "\n",
    "grade_report(percentage_passed, points=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f690da27",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}