Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add session/download-pdf endpoints and add graph to represent values #164

Merged
merged 55 commits into from
Oct 4, 2024
Merged
Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
e16f20c
Update script to add in pdf and session endpoints in to benchmark result
VirajP1002 Jul 29, 2024
33ab283
Update tests
VirajP1002 Jul 29, 2024
64cce07
Update happy path benchmark and mock_stats
VirajP1002 Jul 29, 2024
e2e4b9e
Update visualise_results to create a graph for session and PDF endpoints
VirajP1002 Jul 29, 2024
d25d376
Fix and add/parameterize tests to support new changes
VirajP1002 Jul 29, 2024
21f6f41
Format file
VirajP1002 Jul 29, 2024
f68894e
Temporary change to check actions
VirajP1002 Jul 29, 2024
93c98f0
Revert changes
VirajP1002 Jul 29, 2024
82b64a4
Refactor file location
VirajP1002 Jul 30, 2024
a5faa9f
Revert file location change
VirajP1002 Jul 30, 2024
4022aff
Update how pdf_percentile is calculated and update tests
VirajP1002 Jul 30, 2024
37e3e15
Add & parameterize tests
VirajP1002 Jul 30, 2024
2cffa1d
Format files
VirajP1002 Jul 30, 2024
9c77e50
Temporarily remove call to create graph of percentiles
VirajP1002 Jul 30, 2024
838608a
Update visualise script logic
VirajP1002 Jul 31, 2024
4e0bf3b
Update formatting of pdf_percentile and tests
VirajP1002 Jul 31, 2024
eb7fef9
Update test
VirajP1002 Jul 31, 2024
e65f920
Format files
VirajP1002 Jul 31, 2024
8d81135
Allow slack to output new graph
VirajP1002 Jul 31, 2024
85b67b6
Update graphs to be in one image using subplots
VirajP1002 Aug 1, 2024
75e02ec
Test different image size
VirajP1002 Aug 1, 2024
937565d
Resize graph image
VirajP1002 Aug 1, 2024
84be880
Resize graph image
VirajP1002 Aug 1, 2024
99539bd
Resize graph image
VirajP1002 Aug 1, 2024
23f3703
Resize graph image
VirajP1002 Aug 1, 2024
0d2533e
Update tests and format code
VirajP1002 Aug 2, 2024
9e94186
Update output for GH
VirajP1002 Aug 2, 2024
3b40e9e
Update and parametrize tests
VirajP1002 Aug 5, 2024
1231b67
Remove blank line
VirajP1002 Aug 5, 2024
29a7a66
Merge branch 'main' into add-additional-endpoints
VirajP1002 Aug 6, 2024
664a155
Merge main
VirajP1002 Aug 13, 2024
216e133
Refactor else statement
VirajP1002 Aug 13, 2024
b02d491
Address comments
VirajP1002 Aug 13, 2024
aed7d61
Formatting
VirajP1002 Aug 13, 2024
a4f2fd6
Remove suffix from create_dataframe method
VirajP1002 Aug 14, 2024
a966b7b
Remove Any type hinting
VirajP1002 Aug 14, 2024
4660843
Add formatting to all percentiles
VirajP1002 Aug 14, 2024
ebf1b15
Format
VirajP1002 Aug 14, 2024
5c4d3c8
Refactor percentile formatting
VirajP1002 Aug 16, 2024
d0942b7
Address comments
VirajP1002 Aug 21, 2024
b7d5666
Merge branch 'main' into add-additional-endpoints
VirajP1002 Aug 23, 2024
39b3199
Change casting from int to float to have more accurate values in benc…
VirajP1002 Aug 23, 2024
3e7d911
Format
VirajP1002 Aug 23, 2024
243d890
Merge branch 'main' into add-additional-endpoints
VirajP1002 Aug 28, 2024
7c59d16
Minor tweaks for PR - continuing from @VirajP1002
liamtoozer Sep 6, 2024
f02228e
Linting
liamtoozer Sep 6, 2024
d234f7f
Merge branch 'main' into add-additional-endpoints
liamtoozer Sep 6, 2024
b910fe6
Respond to comments
VirajP1002 Sep 18, 2024
131b109
Address comments
VirajP1002 Sep 19, 2024
d0df0c6
Refactor code, format and add new test
VirajP1002 Sep 20, 2024
f04995a
Remove format string call in test
VirajP1002 Sep 20, 2024
d53cf05
Update test_create_graph to utilise the new get_dataframes() method
VirajP1002 Sep 20, 2024
2a9ef0f
Update formatted percentiles definitions and references
VirajP1002 Sep 23, 2024
21759af
Merge branch 'main' into add-additional-endpoints
VirajP1002 Sep 23, 2024
09bbf48
Increase test coverage fail_under value
VirajP1002 Sep 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions requests/test_benchmark_business_happy_path.json
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@
"method": "POST",
"url": "/questionnaire/block-383/",
"data": {
"answer-439": "Hi eQ team \ud83d\ude0d"
"answer-439": "Hi eQ Team"
}
},
{
Expand Down Expand Up @@ -119,8 +119,8 @@
"method": "POST",
"url": "/questionnaire/people/add-person/",
"data": {
"first-name": "Erling",
"last-name": "Haaland"
"first-name": "Kylian",
"last-name": "Mbappé"
}
},
{
Expand Down Expand Up @@ -540,6 +540,10 @@
"method": "GET",
"url": "/submitted/view-response/"
},
{
"method": "GET",
"url": "/submitted/download-pdf"
},
{
"method": "GET",
"url": "/submitted/thank-you/"
Expand Down
103 changes: 68 additions & 35 deletions scripts/benchmark_stats.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ def __init__(self, folder_paths: List[str]):
"POST": {"response_times": [], "total": 0},
}

self._session_percentile: List[int] = []
self._pdf_percentile: List[int] = []
self._total_failures: int = 0
self._percentiles: Mapping[int : List[float]] = defaultdict(list) # noqa: E203

Expand All @@ -30,15 +32,25 @@ def __str__(self):
f"{percentile}th: {self.percentiles[percentile]}ms"
for percentile in self.PERCENTILES_TO_REPORT
)
formatted_metrics = {
berroar marked this conversation as resolved.
Show resolved Hide resolved
"average_get": self.formatted_percentile(self.average_get),
"average_post": self.formatted_percentile(self.average_post),
"pdf_percentile": self.formatted_percentile(self.average_pdf_percentile),
"session_percentile": self.formatted_percentile(
self.average_session_percentile
),
}
if self.output_to_github:
formatted_percentiles = formatted_percentiles.replace(os.linesep, "<br />")
return (
f'{{"body": "'
f"**Benchmark Results**<br /><br />"
f"Percentile Averages:<br />"
f"{formatted_percentiles}<br />"
f"GETs (99th): {self.average_get}ms<br />"
f"POSTs (99th): {self.average_post}ms<br /><br />"
f"GETs (99th): {formatted_metrics['average_get']}<br />"
f"POSTs (99th): {formatted_metrics['average_post']}<br /><br />"
f"PDF: {formatted_metrics['pdf_percentile']}<br />"
f"Session: {formatted_metrics['session_percentile']}<br /><br />"
f"Total Requests: {self.total_requests:,}<br />"
f"Total Failures: {self._total_failures:,}<br />"
f'Error Percentage: {(round(self.error_percentage, 2))}%<br />"}}'
Expand All @@ -48,8 +60,11 @@ def __str__(self):
f"Percentile Averages:\n"
f"{formatted_percentiles}\n"
f"---\n"
f"GETs (99th): {self.average_get}ms\n"
f"POSTs (99th): {self.average_post}ms\n"
f"GETs (99th): {formatted_metrics['average_get']}\n"
f"POSTs (99th): {formatted_metrics['average_post']}\n"
f"---\n"
f"PDF: {formatted_metrics['pdf_percentile']}\n"
f"Session: {formatted_metrics['session_percentile']}\n"
f"---\n"
f"Total Requests: {self.total_requests:,}\n"
f"Total Failures: {self._total_failures:,}\n"
Expand All @@ -60,34 +75,48 @@ def _process_file_data(self):
for file in self.files:
with open(file) as fp:
for row in DictReader(fp, delimiter=","):
request_count = int(
row.get("Request Count") or row.get("# requests")
)
if row["Name"] == "Aggregated":
failure_count = row.get("Failure Count") or row.get(
"# failures"
)
self._total_failures += int(failure_count)
else:
weighted_request_count = self._get_weighted_request_count(
request_count
)
for percentile in self.PERCENTILES_TO_REPORT:
weighted_percentile = (
float(row[f"{percentile}%"]) * weighted_request_count
)
self._percentiles[percentile].append(weighted_percentile)

percentile_response_time = float(
row[f"{self.PERCENTILE_TO_USE_FOR_AVERAGES}%"]
)
weighted_response_time = (
percentile_response_time * weighted_request_count
)
self._requests[row["Type"]]["response_times"].append(
weighted_response_time
)
self._requests[row["Type"]]["total"] += request_count
self._process_row(row)

def _process_row(self, row):
# Handle special 'Aggregated' row
if row["Name"] == "Aggregated":
failure_count = row["Failure Count"] or row["# failures"]
self._total_failures += int(failure_count)
return

if row["Name"] == "/submitted/download-pdf":
self._pdf_percentile.append(
int(row[f"{self.PERCENTILE_TO_USE_FOR_AVERAGES}%"])
)
elif row["Name"] == "/session":
self._session_percentile.append(
int(row[f"{self.PERCENTILE_TO_USE_FOR_AVERAGES}%"])
)

request_count = int(row.get("Request Count") or row.get("# requests"))
weighted_request_count = self._get_weighted_request_count(request_count)

for percentile in self.PERCENTILES_TO_REPORT:
weighted_percentile = float(row[f"{percentile}%"]) * weighted_request_count
self._percentiles[percentile].append(weighted_percentile)

percentile_response_time = float(row[f"{self.PERCENTILE_TO_USE_FOR_AVERAGES}%"])
weighted_response_time = percentile_response_time * weighted_request_count
self._requests[row["Type"]]["response_times"].append(weighted_response_time)
self._requests[row["Type"]]["total"] += request_count

@property
def average_pdf_percentile(self) -> int | None:
if self._pdf_percentile:
average_pdf_percentile = sum(self._pdf_percentile) / len(
self._pdf_percentile
)
return round(average_pdf_percentile)
return None

@property
def average_session_percentile(self) -> int:
return round(sum(self._session_percentile) / len(self._session_percentile))

@property
def files(self) -> List[str]:
Expand All @@ -96,7 +125,7 @@ def files(self) -> List[str]:
@property
def percentiles(self) -> Mapping:
return {
percentile: int(
percentile: round(
sum(values) / self._get_weighted_request_count(self.total_requests)
)
for percentile, values in self._percentiles.items()
Expand All @@ -108,14 +137,14 @@ def total_requests(self) -> int:

@property
def average_get(self) -> int:
return int(
return round(
sum(self._requests["GET"]["response_times"])
/ self._get_weighted_request_count(self._requests["GET"]["total"])
)

@property
def average_post(self) -> int:
return int(
return round(
sum(self._requests["POST"]["response_times"])
/ self._get_weighted_request_count(self._requests["POST"]["total"])
)
Expand All @@ -124,5 +153,9 @@ def average_post(self) -> int:
def error_percentage(self) -> float:
return (self._total_failures * 100) / self.total_requests

@staticmethod
def formatted_percentile(percentile) -> str:
return f"{percentile}ms" if percentile else "N/A"

def _get_weighted_request_count(self, request_count: int) -> float:
return request_count * self.PERCENTILE_TO_USE_FOR_AVERAGES / 100
68 changes: 53 additions & 15 deletions scripts/visualise_results.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,38 +6,59 @@
from scripts.get_summary import get_results, parse_environment_variables

PERCENTILES_TO_GRAPH = (50, 90, 95, 99)
PERCENTILES_TO_PLOT = ("50th", "90th", "95th", "99th")

ADDITIONAL_METRICS_TO_GRAPH = ("PDF", "Session")


class GraphGenerationFailed(Exception):
pass


def plot_data(df, number_of_days_to_plot):
try:
plt.style.use("fast")
def plot_data(dataframes, number_of_days_to_plot):
plt.style.use("fast")
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(15, 6))

for i, dataframe in enumerate(dataframes):
"""
We use the .subplot() method below to switch the indexes of the plots themselves,
if we do not switch plots or remove this method, the visuals (such as the graph background)
and, most importantly, the axes values will only be applied to the second subplot.
"""
plt.subplot(1, 2, i + 1)
berroar marked this conversation as resolved.
Show resolved Hide resolved
if (
number_of_days_to_plot and number_of_days_to_plot <= 45
): # To make the chart still easily digestible
df.plot.line(marker="o", markersize=8)
dataframe.plot.line(marker="o", markersize=8, ax=axes[i])
plt.grid(True, axis="both", alpha=0.3)
else:
df.plot.line()
dataframe.plot.line(ax=axes[i])

plt.margins(0.03, 0.07)
plt.legend(frameon=True, prop={"size": 17})
plt.xticks(df.index, df["DATE"], size="small", rotation=90)
plt.legend(frameon=True, prop={"size": 10})
plt.xticks(dataframe.index, dataframe["DATE"], size="small", rotation=90)
plt.yticks(size="small")
plt.ylabel("Average Response Time (ms)")
plt.xlabel("Run Date (YYYY-MM-DD)", labelpad=13)

plt.savefig("performance_graph.png", bbox_inches="tight")
print("Graph saved as performance_graph.png")

def create_graph(dataframes, number_of_days_to_plot, filename):
try:
plot_data(dataframes, number_of_days_to_plot)
plt.savefig(filename, bbox_inches="tight")
print("Graph saved as", filename)
except Exception as e:
raise GraphGenerationFailed from e


def get_data_frame(results):
def create_dataframe(result_fields, values_to_plot):
return DataFrame(
result_fields,
columns=["DATE", *(f"{percentile}" for percentile in values_to_plot)],
)


def get_performance_data_frame(results):
result_fields = [
[
result.date,
Expand All @@ -49,16 +70,33 @@ def get_data_frame(results):
for result in results
]

percentile_columns = (f"{percentile}th" for percentile in PERCENTILES_TO_GRAPH)
return DataFrame(result_fields, columns=["DATE", *percentile_columns])
return create_dataframe(result_fields, PERCENTILES_TO_PLOT)


def get_additional_metrics_data_frame(results):
result_fields = [
[
result.date,
result.statistics.average_pdf_percentile,
result.statistics.average_session_percentile,
]
for result in results
]

return create_dataframe(result_fields, ADDITIONAL_METRICS_TO_GRAPH)


if __name__ == "__main__":
parsed_variables = parse_environment_variables()
number_of_days = parsed_variables["number_of_days"]

folders = sorted(glob(f"{parsed_variables['output_dir']}/*"))
results = get_results(folders, number_of_days)
dataframe = get_data_frame(results)

plot_data(dataframe, number_of_days)
results = list(get_results(folders, number_of_days))

performance_dataframe = get_performance_data_frame(results)
additional_metrics_dataframe = get_additional_metrics_data_frame(results)

dataframe_values = [performance_dataframe, additional_metrics_dataframe]
berroar marked this conversation as resolved.
Show resolved Hide resolved

create_graph(dataframe_values, number_of_days, "performance_graph.png")
Loading