Suppress most cli errors, but still warn if we get too many

[plars]

Description

When running a job while polling with the cli, there may be an occasional network delay that can cause you to get errors like this:

ERROR: 2024-11-20 19:04:11 client.py:64 -- Timeout while trying to communicate with the server.
WARNING: 2024-11-20 19:04:11 __init__.py:874 -- Unable to retrieve job state.
unknown

These can usually be ignored since it will try again, but can be often be interpreted by the user to think that something is wrong when it isn't. However, there's also a possibility that the server is unreachable for a long time, and we don't want to hide that from the user if it's happening.

I think this takes a pretty balanced approach and silences most of these warnings and errors (except when it's going to be fatal), while running a counter for consecutive timeout/connection errors. It will warn the user that something could be wrong at every interval of $TESTFLINGER_ERROR_THRESHOLD (default 3) consecutive errors, but also indicate that it will keep retrying.

Resolved issues

CERTTF-283

Documentation

Added a reference section to the documentation about the testflinger config command, and the supported configuration settings.

Web service API changes

N/A

Tests

Additional unit tests added

[boukeas]

The changes are very useful as it is indeed the case that users may not know how to interpret these messages.

I do believe that a requirement for 10 consecutive failures will effectively suppress all messages, even when there are actual networking issues, thus making these messages ineffective as a diagnostic tool. So my suggestions are to:

• retrieve the number of messages required to display a warning from the config file, so that we are able to control it more easily
• reduce the number to 5 or even 3

[tang-mm]

Thanks for adding the reference! I just need a few clarifications on the command usage.
Also, in the previous how to authentication guide, we asked users to reload an .env file to update their environment variable. should we also change that doc to recommend the method using cli?