diff --git a/README.md b/README.md index 23bc0b34..a252445f 100644 --- a/README.md +++ b/README.md @@ -191,7 +191,7 @@ There are three main ways you can use Conductor when building durable, resilient 2. Create Conductor workflows that implement application state - A typical workflow implements the saga pattern. 3. Use Conductor SDK and APIs to manage workflows from your application. -### [Create and Run Conductor Workers] +### [Create and Run Conductor Workers](docs/readme/workers.md) ### [Create Conductor Workflows] diff --git a/docs/readme/workers.md b/docs/readme/workers.md index ca3a3ca0..f5e6a8ca 100644 --- a/docs/readme/workers.md +++ b/docs/readme/workers.md @@ -1,68 +1,185 @@ -# Writing Workers with the C# SDK +# Writing Workers -A worker is responsible for executing a task. -Operator and System tasks are handled by the Conductor server, while user defined tasks needs to have a worker created that awaits the work to be scheduled by the server for it to be executed. +A Workflow task represents a unit of business logic that achieves a specific goal, such as checking inventory, initiating payment transfer, etc. A worker implements a task in the workflow. -Worker framework provides features such as polling threads, metrics and server communication. +## Content -### Design Principles for Workers -Each worker embodies design pattern and follows certain basic principles: + + -1. Workers are stateless and do not implement a workflow specific logic. -2. Each worker executes a very specific task and produces well-defined output given specific inputs. -3. Workers are meant to be idempotent (or should handle cases where the task that partially executed gets rescheduled due to timeouts etc.) -4. Workers do not implement the logic to handle retries etc, that is taken care by the Conductor server. +- [Implementing Workers](#implementing-workers) + - [Managing Workers in Application](#managing-workers-in-application) +- [Design Principles for Workers](#design-principles-for-workers) +- [System Task Workers](#system-task-workers) + - [Wait Task](#wait-task) + - [Using Code to Create Wait Task](#using-code-to-create-wait-task) + - [JSON Configuration](#json-configuration) + - [HTTP Task](#http-task) + - [Using Code to Create HTTP Task](#using-code-to-create-http-task) + - [JSON Configuration](#json-configuration-1) + - [Javascript Executor Task](#javascript-executor-task) + - [Using Code to Create Inline Task](#using-code-to-create-inline-task) + - [JSON Configuration](#json-configuration-2) + - [JSON Processing using JQ](#json-processing-using-jq) + - [Using Code to Create JSON JQ Transform Task](#using-code-to-create-json-jq-transform-task) + - [JSON Configuration](#json-configuration-3) +- [Worker vs. Microservice/HTTP Endpoints](#worker-vs-microservicehttp-endpoints) +- [Deploying Workers in Production](#deploying-workers-in-production) -### Creating Task Workers -Example worker + -```csharp -public class SimpleWorker : IWorkflowTask +## Implementing Workers + +The workers can be implemented by writing a simple C# function and annotating the function with the `@worker_task`. Conductor workers are services (similar to microservices) that follow the [Single Responsibility Principle](https://en.wikipedia.org/wiki/Single_responsibility_principle). + +Workers can be hosted along with the workflow or run in a distributed environment where a single workflow uses workers deployed and running in different machines/VMs/containers. Whether to keep all the workers in the same application or run them as a distributed application is a design and architectural choice. Conductor is well suited for both kinds of scenarios. + +You can create or convert any existing C# function to a distributed worker by adding `@worker_task` annotation to it. Here is a simple worker that takes name as input and returns greetings: + +```cs +To Do +``` + +A worker can take inputs which are primitives - `str`, `int`, `float`, `bool` etc. or can be complex data classes. + +Here is an example worker that uses `dataclass` as part of the worker input. + +```cs +To Do +``` + +### Managing Workers in Application + +Workers use a polling mechanism (with a long poll) to check for any available tasks from the server periodically. The startup and shutdown of workers are handled by the `conductor.client.automator.task_handler.TaskHandler` class. + +```cs +To Do +``` + +## Design Principles for Workers + +Each worker embodies the design pattern and follows certain basic principles: + +1. Workers are stateless and do not implement a workflow-specific logic. +2. Each worker executes a particular task and produces well-defined output given specific inputs. +3. Workers are meant to be idempotent (Should handle cases where the partially executed task, due to timeouts, etc, gets rescheduled). +4. Workers do not implement the logic to handle retries, etc., that is taken care of by the Conductor server. + +## System Task Workers + +A system task worker is a pre-built, general-purpose worker in your Conductor server distribution. + +System tasks automate repeated tasks such as calling an HTTP endpoint, executing lightweight ECMA-compliant javascript code, publishing to an event broker, etc. + +### Wait Task + +>[!tip] +>Wait is a powerful way to have your system wait for a specific trigger, such as an external event, a particular date/time, or duration, such as 2 hours, without having to manage threads, background processes, or jobs. + +#### Using Code to Create Wait Task + +```cs +To Do +``` + +#### JSON Configuration + +```json { - public string TaskType { get; } - public WorkflowTaskExecutorConfiguration WorkerSettings { get; } - - public SimpleWorker(string taskType = "test-sdk-csharp-task") - { - TaskType = taskType; - WorkerSettings = new WorkflowTaskExecutorConfiguration(); - } - - public TaskResult Execute(Task task) - { - return task.Completed(); - } + "name": "wait", + "taskReferenceName": "wait_till_jan_end", + "type": "WAIT", + "inputParameters": { + "until": "2024-01-31 00:00 UTC" + } } ``` -## Starting Workers -You can use `WorkflowTaskHost` to create a worker host, it requires a configuration object and then you can add your workers. +### HTTP Task -```csharp -using Conductor.Client.Worker; -using System; -using System.Threading.Thread; +Make a request to an HTTP(S) endpoint. The task allows for GET, PUT, POST, DELETE, HEAD, and PATCH requests. -var host = WorkflowTaskHost.CreateWorkerHost(configuration, new SimpleWorker()); -await host.startAsync(); -Thread.Sleep(TimeSpan.FromSeconds(100)); +#### Using Code to Create HTTP Task + +```cs +To Do +``` + +#### JSON Configuration + +```json +{ + "name": "http_task", + "taskReferenceName": "http_task_ref", + "type" : "HTTP", + "uri": "https://orkes-api-tester.orkesconductor.com/api", + "method": "GET" +} +``` + +### Javascript Executor Task + +Execute ECMA-compliant Javascript code. It is useful when writing a script for data mapping, calculations, etc. + +#### Using Code to Create Inline Task + +```cs +To Do +``` + +#### JSON Configuration + +```json +{ + "name": "inline_task", + "taskReferenceName": "inline_task_ref", + "type": "INLINE", + "inputParameters": { + "expression": " function greetings() {\n return {\n \"text\": \"hello \" + $.name\n }\n }\n greetings();", + "evaluatorType": "graaljs", + "name": "${workflow.input.name}" + } +} +``` + +### JSON Processing using JQ + +[Jq](https://jqlang.github.io/jq/) is like sed for JSON data - you can slice, filter, map, and transform structured data with the same ease that sed, awk, grep, and friends let you play with text. + +#### Using Code to Create JSON JQ Transform Task + +```cs +To Do +``` + +#### JSON Configuration + +```json +{ + "name": "json_transform_task", + "taskReferenceName": "json_transform_task_ref", + "type": "JSON_JQ_TRANSFORM", + "inputParameters": { + "key1": "k1", + "key2": "k2", + "queryExpression": "{ key3: (.key1.value1 + .key2.value2) }", + } +} ``` -Check out our [integration tests](https://github.com/conductor-sdk/conductor-csharp/blob/92c7580156a89322717c94aeaea9e5201fe577eb/Tests/Worker/WorkerTests.cs#L37) for more examples +## Worker vs. Microservice/HTTP Endpoints -Worker SDK collects the following metrics: +>[!tip] +>Workers are a lightweight alternative to exposing an HTTP endpoint and orchestrating using HTTP tasks. Using workers is a recommended approach if you do not need to expose the service over HTTP or gRPC endpoints. +There are several advantages to this approach: -| Name | Purpose | Tags | -| ------------------ | :------------------------------------------- | -------------------------------- | -| task_poll_error | Client error when polling for a task queue | taskType, includeRetries, status | -| task_execute_error | Execution error | taskType | -| task_update_error | Task status cannot be updated back to server | taskType | -| task_poll_counter | Incremented each time polling is done | taskType | -| task_poll_time | Time to poll for a batch of tasks | taskType | -| task_execute_time | Time to execute a task | taskType | -| task_result_size | Records output payload size of a task | taskType | +1. **No need for an API management layer** : Given there are no exposed endpoints and workers are self-load-balancing. +2. **Reduced infrastructure footprint** : No need for an API gateway/load balancer. +3. All the communication is initiated by workers using polling - avoiding the need to open up any incoming TCP ports. +4. Workers **self-regulate** when busy; they only poll as much as they can handle. Backpressure handling is done out of the box. +5. Workers can be scaled up/down quickly based on the demand by increasing the number of processes. -Metrics on client side supplements the one collected from server in identifying the network as well as client side issues. +## Deploying Workers in Production -### Next: [Create and Execute Workflows](https://github.com/conductor-sdk/conductor-csharp/blob/main/docs/readme/workflow.md) +Conductor workers can run in the cloud-native environment or on-prem and can easily be deployed like any other C# application. Workers can run a containerized environment, VMs, or bare metal like you would deploy your other C# applications.