Raise aio-max-nr on GHA

[cole-miller]

No description provided.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.23%. Comparing base (61fc84c) to head (e814a1c).
Report is 22 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #596      +/-   ##
==========================================
+ Coverage   81.21%   81.23%   +0.01%     
==========================================
  Files         191      192       +1     
  Lines       27033    27061      +28     
  Branches     4944     4980      +36     
==========================================
+ Hits        21956    21982      +26     
+ Misses       3483     3436      -47     
- Partials     1594     1643      +49     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cole-miller]

Ah, it's io_setup in probeAsyncIO that's failing with EAGAIN due to aio-nr reaching aio-max-nr. I'm not sure what probeAsyncIO should actually do in this case -- it probably shouldn't just report success, but on the other hand it doesn't seem good to fail dqlite_node_create because of a temporary condition. Maybe probeAsyncIO should have a retry loop with backoff internally?

[cole-miller]

Summary: the issue with the flaky test cluster/restart is that it somewhat consistently hits the aio-max-nr ceiling in CI during raft_uv startup (probeAsyncIO). This PR addresses that by introducing a retry loop into test_server_start (with exponential backoff). That seems to fix the problem for cluster/restart, but conn/query/one now runs into a similar issue (somewhat reproducibly), with the error originating here:

dqlite/src/raft/uv_writer.c

Lines 44 to 63 in c9eda92

	static int uvWriterIoSetup(unsigned n, aio_context_t *ctx, char *errmsg)
	{
	int rv;
	rv = UvOsIoSetup(n, ctx);
	if (rv != 0) {
	switch (rv) {
	case UV_EAGAIN:
	ErrMsgPrintf(errmsg,
	"AIO events user limit exceeded");
	rv = RAFT_TOOMANY;
	break;
	default:
	UvOsErrMsg(errmsg, "io_setup", rv);
	rv = RAFT_IOERR;
	break;
	}
	return rv;
	}
	return 0;
	}

This is trickier to fix because the error is propagated all the way out to the client. We don't want to put a retry loop inside dqlite or raft library code; that should live inside the test suite, wherever we call clientSendExec, etc. But this is a large refactoring that I'm not going to attempt right now.

I think aio-max-nr wasn't a problem before in raft or dqlite CI because we didn't run as many tests in parallel. Now that we have the raft tests and the dqlite tests running in the same CI job and at the same time, there's more contention for this scarce resource.

[cole-miller]

Raising aio-max-nr to 1000000 seems to reduce the rate at which these tests fail.