Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry with backoff #30

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open
17 changes: 12 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# effection-contrib
# [effection-contrib](https://effection-contrib.deno.dev/)

This repository contains a collection of contributions by the community. This
repository automatically publishes to JSR and NPM. All packages that were used
Expand All @@ -8,7 +8,14 @@ more than a few times are welcome.

1. Create a directory
2. Add a deno.json file with
`{ "name": "@effection-contrib/<you_package_name>", version: "0.1.0", "exports": "./mod.ts", "license": "MIT" }`
```json
{
"name": "@effection-contrib/<you_package_name>",
"version": "0.1.0",
"exports": "./mod.ts",
"license": "MIT"
}
```
3. Add a README.md (the first sentence of the README will be used as the
description)
4. Add your source code and export it from `mod.ts`
Expand All @@ -17,6 +24,6 @@ more than a few times are welcome.

## To publish a new project

1. Member of [jsr.io/@effection-contrib](https://jsr.io/@effection-contrib) has
too add that project to the scope
2. It should be publish on next merge to main
1. A member of [jsr.io/@effection-contrib](https://jsr.io/@effection-contrib) has
to add that project to the scope
2. Your new package will be published on the next merge to main
4 changes: 2 additions & 2 deletions deno.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
"dev": "deno run -A --watch www/main.ts"
},
"imports": {
"effection": "npm:[email protected]",
"effection-www/": "https://raw.githubusercontent.com/thefrontside/effection/4982887c1677b847d256402e8709f2b1d49437e6/www/",
"revolution": "https://deno.land/x/[email protected]/mod.ts",
"revolution/jsx-runtime": "https://deno.land/x/[email protected]/jsx-runtime.ts",
Expand Down Expand Up @@ -38,7 +37,8 @@
"./deno-deploy",
"./task-buffer",
"./tinyexec",
"./websocket"
"./websocket",
"./retry-backoff"
],
"deploy": {
"project": "aa1dbfaa-d7c1-49d7-b514-69e1d8344f95",
Expand Down
37 changes: 37 additions & 0 deletions retry-backoff/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# retry-backoff

Retry operations with incremental backoff.

---

There's a default timeout set to 30 seconds. If you'd like to set a different
timeout, you'll need to either pass in options to `useRetryWithBackoff`:

```js
import { main } from "effection";
import { useRetryWithBackoff } from "@effection-contrib/retry-backoff";

await main(function* () {
yield* useRetryWithBackoff(function* () {
yield* call(fetch("https://foo.bar/"));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't be compatible with v4, so let's use function syntax: yield* call(() => fetch(...))

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

}, { timeout: 45_000 });
});
```

Or initialize the context so that the same timeout can be applied to all of your
retry operations:

```js
import { main } from "effection";
import {
initRetryWithBackoff,
useRetryWithBackoff,
} from "@effection-contrib/retry-backoff";

await main(function* () {
yield* initRetryWithBackoff({ timeout: 45_000 });
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove this function.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you talking about the example myOperation function? or main?

Copy link
Member

@taras taras Dec 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the example myOperation function, so the example reads like

await main(function*() {
  yield* retryBackoff(function*() {
     const result = call(() => fetch(....));
     ...
  });
})

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh okay. I took care of that from this comment

yield* retryWithBackoff(function* () {
yield* call(fetch("https://foo.bar/"));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, let's use the function syntax that's compatible with v4

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And done

});
});
```
6 changes: 6 additions & 0 deletions retry-backoff/deno.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"name": "@effection-contrib/retry-backoff",
"version": "0.1.0",
"exports": "./mod.ts",
"license": "MIT"
}
4 changes: 4 additions & 0 deletions retry-backoff/mod.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
export {
useRetryWithBackoff,
initRetryWithBackoff,
} from './retry-backoff.ts';
39 changes: 39 additions & 0 deletions retry-backoff/retry-backoff.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
import { run } from "npm:[email protected]";
import { describe, it } from "bdd";
import { expect } from "expect";
import { useRetryWithBackoff } from "./retry-backoff.ts"

describe("RetryBackoff", () => {
it("retries operation and returns output if operation finishes on time", async () => {
await run(function* () {
let attempts = 0;
let result = 0;
yield* useRetryWithBackoff(function* () {
if (attempts < 2) {
attempts++;
throw new Error("operation failed");
} else {
result = 1;
}
}, { timeout: 2_000 });
expect(attempts).toBe(2);
expect(result).toBe(1);
})
});

it("retries operation and handles timeout when operation exceeds limit", async () => {
await run(function* () {
let attempts = 0;
let result = 0;
yield* useRetryWithBackoff(function* () {
if (attempts < 2) {
attempts++;
throw new Error("operation failed");
} else {
result = 1;
}
}, { timeout: 500 });
expect(result).toBe(0);
})
});
});
87 changes: 87 additions & 0 deletions retry-backoff/retry-backoff.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
import {
createContext,
type Context as ContextType,
type Operation,
race,
sleep,
} from "npm:[email protected]";
import prettyMilliseconds from "npm:[email protected]";

interface UseRetryBackoffOptions {
timeout?: number;
}

interface RetryWithContextDefaults {
timeout: number;
}

const RetryBackoffContext = createContext<RetryWithContextDefaults>(
"retry-with-context",
{
timeout: 30_000,
}
);

export function* useRetryWithBackoff<T> (
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need docs for this function.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be the right way to write docs for this context?

/**
  * Description
  *
  * @property {function} set
  * @param {{ timeout: number }}
*/
export const RetryBackoffContext = createContext<RetryWithContextDefaults>(

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's right. You don't need to describe the parameters and properties because those will be generated from typescript

fn: () => Operation<T>,
options?: UseRetryBackoffOptions,
) {
const defaults = yield* RetryBackoffContext.expect();
const _options = {
...defaults,
...options,
};

let attempt = -1;
let name = fn.name || "unknown";

function* body() {
while (true) {
try {
const result = yield* fn();
if (attempt !== -1) {
console.log(`Operation[${name}] succeeded after ${attempt + 2} retry.`);
}
return result;
} catch {
// https://aws.amazon.com/ru/blogs/architecture/exponential-backoff-and-jitter/
const backoff = Math.pow(2, attempt) * 1000;
const delayMs = Math.round((backoff * (1 + Math.random())) / 2);
console.log(`Operation[${name}] failed, will retry in ${prettyMilliseconds(delayMs)}.`);
yield* sleep(delayMs);
attempt++;
}
}
}

function* timeout() {
yield* sleep(_options.timeout);
console.log(`Operation[${name}] timedout after ${attempt + 2}`);
}

yield* race([
body(),
timeout(),
]);
}

export function* initRetryWithBackoff(
defaults: RetryWithContextDefaults,
) {
// deno-lint-ignore require-yield
function* init(): Operation<RetryWithContextDefaults> {
return defaults;
}

return yield* ensureContext(
RetryBackoffContext,
init(),
);
}

function* ensureContext<T>(Context: ContextType<T>, init: Operation<T>) {
if (!(yield* Context.get())) {
yield* Context.set(yield* init);
}
return yield* Context.expect();
}