-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathTest_Harness.mw
139 lines (116 loc) · 4.77 KB
/
Test_Harness.mw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
A Test Harness is an automated testing framework.
This article contains information about the input, output and tests for
systems implementing the [[CCS]]
'''Comment''' This needs to be split into two articles, keep requirements in this one, and move other information to another article [[User:Douglas Lane|Douglas Lane]] 17:47, 17 April 2011 (CEST)
==Input to the Test Harness==
{| class="wikitable"
! File name
|-
| [[Contest_Control_System#contest.yaml | contest.yaml ]] file
|-
| [[Problem_format#Configuration_file_format | problem.yaml]] and all associated files
|-
| [[Contest_Control_System#group.tsv | group.tsv]] file
|-
| [[Contest_Control_System#teams.tsv | teams.tsv]] file
|-
| [[Contest_Control_System#userdata.tsv | userdata.tsv ]] file
|-
|}
==Testing Configuration Changes==
{| class="wikitable"
! Description
! Input
! Compare to
! Notes
|-
| Test contest information || Load [[Contest_Control_System#contest.yaml | contest.yaml ]] file || [[Event Feed]] || missing default-clars
|-
| Test problems configuration || Load [[Contest_Control_System#problemset.yaml | problemset.yaml ]] and [[Problem format]] files || [[Event Feed]] || missing everything except: <br> letter, name, balloon color, balloon color RGB
|-
| Test language configuration || Load [[Contest_Control_System#languages.yaml | languages.yaml ]] file || [[Event Feed]] || missing compiler, compiler-args, <br> runner, runner-args
|-
| Test testcase configuration || Load [[Problem format]] file || [[Event Feed]] || [[TBD]]
|-
| Test group configuration || Load [[Contest_Control_System#group.tsv | group.tsv]] file || [[Event Feed]] || missing Group ID
|-
| Test team configuration || Load [[Contest_Control_System#teams.tsv | teams.tsv]] file || [[Event Feed]] || missing Reservation ID, Group ID, Country
|-
| Test validator || Load [[Problem format]] file || [[TBD]] || [[TBD]]
|-
| Test judgement names|| contest.yaml || [[Event Feed]] || [[TBD]]
|-
| Test notification || Judgement || [[Event Feed]] || [[TBD]]
|}
==Testing Run Submissions==
{| class="wikitable"
! Description
! Input
! Compare to
|-
| Test run submission || [[Contest_Control_System#Scriptable_Submissions | Scriptable Submissions]] runs || [[Event Feed]]
|-
| Test run judgement || [[Contest_Control_System#Scriptable_Submissions | Scriptable Submissions]] runs || [[Event Feed]]
|}
==Testing Clarifications==
{| class="wikitable"
! Description
! Input
! Compare to
|-
| Test clar submission || [[Contest_Control_System#Scriptable_Submissions | Scriptable Submissions]] clars || [[Event Feed]]
|-
| Test clar answered || [[Contest_Control_System#Scriptable_Submissions | Scriptable Submissions]] clars || [[Event Feed]]
|}
:'''Comment''': There is currently no CCS-defined way to automatically answer a clarification [[User:Douglas Lane|Douglas Lane]] 03:57, 20 February 2011 (CET)
==Testing Standings==
{| class="wikitable"
! Description
! Input
! Compare to
|-
| Test standings change || [[Contest_Control_System#Scriptable_Submissions | Scriptable Submissions]] runs || [[Contest_Control_System#scoreboard.tsv | scoreboard.tsv]]
|}
==Testing Notifications==
{| class="wikitable"
! Description
! Input
! Compare to
|-
| Test notification sent || [[Contest_Control_System#Scriptable_Submissions | Scriptable Submissions]] runs || Not defined
|}
==Testing [[Contest_Control_System#Prohibited_Operations|Prohibited Operations]]==
In order to test each of the Prohibited Operation a set of standardized messages
needs to be defined.
{| class="wikitable"
! Operation
! Expected Message
|-
| Using libraries except those explicitly allowed || Team's program used unauthorized library/function
|-
| Executing other programs || Team's program executed program "/bin/perl"
|-
| Creating new processes || Team's program tried to start new process
|-
| Creating new threads || Team's program attempted to create a thread
|-
| Reading any files || Team's program tried to read file "c.in"
|-
| Creating files || Team's program tried to create file "c.in"
|-
| Sending signals to other programs || Team's program tried to use a signal N
|-
| Side-stepping time or memory limits || Team's program attempted to exceed limit <br> Team's program attempted to exceed memory limit
|-
| Sending or receiving network traffic || Team's program tried to write network output <br> Team's program tried to read network input
|}
==Testing [[Contest_Control_System#Validators|Validators]]==
Per the [[Output validator]] article test validators to report proper results, esp. file format.
:'''Comment''' TODO complete writing this section [[User:Douglas Lane|Douglas Lane]] 23:31, 28 February 2011 (CET)
{| class="wikitable"
! Test
|-
| [[Output_validator#Reporting_a_judgment|Reporting a judgment]] [[TBD]]
|-
| [[Output_validator#Reporting_Additional_Feedback|Reporting Additional Feedback]] [[TBD]]
|}