workers

RizaFarheen · Apr 5, 2024 · cf24ada · cf24ada
1 parent 60596bc
commit cf24ada
Show file tree

Hide file tree

Showing 2 changed files with 167 additions and 50 deletions.
diff --git a/README.md b/README.md
@@ -191,7 +191,7 @@ There are three main ways you can use Conductor when building durable, resilient
 2. Create Conductor workflows that implement application state - A typical workflow implements the saga pattern.
 3. Use Conductor SDK and APIs to manage workflows from your application.
 
-### [Create and Run Conductor Workers]
+### [Create and Run Conductor Workers](docs/readme/workers.md)
 
 ### [Create Conductor Workflows]
 

diff --git a/docs/readme/workers.md b/docs/readme/workers.md
@@ -1,68 +1,185 @@
-# Writing Workers with the C# SDK
+# Writing Workers 
 
-A worker is responsible for executing a task. 
-Operator and System tasks are handled by the Conductor server, while user defined tasks needs to have a worker created that awaits the work to be scheduled by the server for it to be executed.
+A Workflow task represents a unit of business logic that achieves a specific goal, such as checking inventory, initiating payment transfer, etc. A worker implements a task in the workflow.
 
-Worker framework provides features such as polling threads, metrics and server communication.
+## Content 
 
-### Design Principles for Workers
-Each worker embodies design pattern and follows certain basic principles:
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
 
-1. Workers are stateless and do not implement a workflow specific logic. 
-2. Each worker executes a very specific task and produces well-defined output given specific inputs. 
-3. Workers are meant to be idempotent (or should handle cases where the task that partially executed gets rescheduled due to timeouts etc.)
-4. Workers do not implement the logic to handle retries etc, that is taken care by the Conductor server.
+- [Implementing Workers](#implementing-workers)
+  - [Managing Workers in Application](#managing-workers-in-application)
+- [Design Principles for Workers](#design-principles-for-workers)
+- [System Task Workers](#system-task-workers)
+  - [Wait Task](#wait-task)
+    - [Using Code to Create Wait Task](#using-code-to-create-wait-task)
+    - [JSON Configuration](#json-configuration)
+  - [HTTP Task](#http-task)
+    - [Using Code to Create HTTP Task](#using-code-to-create-http-task)
+    - [JSON Configuration](#json-configuration-1)
+  - [Javascript Executor Task](#javascript-executor-task)
+    - [Using Code to Create Inline Task](#using-code-to-create-inline-task)
+    - [JSON Configuration](#json-configuration-2)
+  - [JSON Processing using JQ](#json-processing-using-jq)
+    - [Using Code to Create JSON JQ Transform Task](#using-code-to-create-json-jq-transform-task)
+    - [JSON Configuration](#json-configuration-3)
+- [Worker vs. Microservice/HTTP Endpoints](#worker-vs-microservicehttp-endpoints)
+- [Deploying Workers in Production](#deploying-workers-in-production)
 
-### Creating Task Workers
-Example worker
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
-```csharp
-public class SimpleWorker : IWorkflowTask
+## Implementing Workers
+
+The workers can be implemented by writing a simple C# function and annotating the function with the `@worker_task`. Conductor workers are services (similar to microservices) that follow the [Single Responsibility Principle](https://en.wikipedia.org/wiki/Single_responsibility_principle).
+
+Workers can be hosted along with the workflow or run in a distributed environment where a single workflow uses workers deployed and running in different machines/VMs/containers. Whether to keep all the workers in the same application or run them as a distributed application is a design and architectural choice. Conductor is well suited for both kinds of scenarios.
+
+You can create or convert any existing C# function to a distributed worker by adding `@worker_task` annotation to it. Here is a simple worker that takes name as input and returns greetings:
+
+```cs
+To Do
+```
+
+A worker can take inputs which are primitives - `str`, `int`, `float`, `bool` etc. or can be complex data classes.
+
+Here is an example worker that uses `dataclass` as part of the worker input.
+
+```cs
+To Do
+```
+
+### Managing Workers in Application
+
+Workers use a polling mechanism (with a long poll) to check for any available tasks from the server periodically. The startup and shutdown of workers are handled by the `conductor.client.automator.task_handler.TaskHandler` class.
+
+```cs
+To Do
+```
+
+## Design Principles for Workers
+
+Each worker embodies the design pattern and follows certain basic principles:
+
+1. Workers are stateless and do not implement a workflow-specific logic.
+2. Each worker executes a particular task and produces well-defined output given specific inputs.
+3. Workers are meant to be idempotent (Should handle cases where the partially executed task, due to timeouts, etc, gets rescheduled).
+4. Workers do not implement the logic to handle retries, etc., that is taken care of by the Conductor server.
+
+## System Task Workers
+
+A system task worker is a pre-built, general-purpose worker in your Conductor server distribution.
+
+System tasks automate repeated tasks such as calling an HTTP endpoint, executing lightweight ECMA-compliant javascript code, publishing to an event broker, etc.
+
+### Wait Task
+
+>[!tip]
+>Wait is a powerful way to have your system wait for a specific trigger, such as an external event, a particular date/time, or duration, such as 2 hours, without having to manage threads, background processes, or jobs.
+
+#### Using Code to Create Wait Task
+
+```cs
+To Do
+```
+
+#### JSON Configuration
+
+```json
 {
-    public string TaskType { get; }
-    public WorkflowTaskExecutorConfiguration WorkerSettings { get; }
-
-    public SimpleWorker(string taskType = "test-sdk-csharp-task")
-    {
-        TaskType = taskType;
-        WorkerSettings = new WorkflowTaskExecutorConfiguration();
-    }
-
-    public TaskResult Execute(Task task)
-    {
-        return task.Completed();
-    }
+  "name": "wait",
+  "taskReferenceName": "wait_till_jan_end",
+  "type": "WAIT",
+  "inputParameters": {
+    "until": "2024-01-31 00:00 UTC"
+  }
 }
 ```
 
-## Starting Workers
-You can use `WorkflowTaskHost` to create a worker host, it requires a configuration object and then you can add your workers.
+### HTTP Task
 
-```csharp
-using Conductor.Client.Worker;
-using System;
-using System.Threading.Thread;
+Make a request to an HTTP(S) endpoint. The task allows for GET, PUT, POST, DELETE, HEAD, and PATCH requests.
 
-var host = WorkflowTaskHost.CreateWorkerHost(configuration, new SimpleWorker());
-await host.startAsync();
-Thread.Sleep(TimeSpan.FromSeconds(100));
+#### Using Code to Create HTTP Task
+
+```cs
+To Do
+```
+
+#### JSON Configuration
+
+```json
+{
+  "name": "http_task",
+  "taskReferenceName": "http_task_ref",
+  "type" : "HTTP",
+  "uri": "https://orkes-api-tester.orkesconductor.com/api",
+  "method": "GET"
+}
+```
+
+### Javascript Executor Task
+
+Execute ECMA-compliant Javascript code. It is useful when writing a script for data mapping, calculations, etc.
+
+#### Using Code to Create Inline Task
+
+```cs
+To Do
+```
+
+#### JSON Configuration
+
+```json
+{
+  "name": "inline_task",
+  "taskReferenceName": "inline_task_ref",
+  "type": "INLINE",
+  "inputParameters": {
+    "expression": " function greetings() {\n  return {\n            \"text\": \"hello \" + $.name\n        }\n    }\n    greetings();",
+    "evaluatorType": "graaljs",
+    "name": "${workflow.input.name}"
+  }
+}
+```
+
+### JSON Processing using JQ
+
+[Jq](https://jqlang.github.io/jq/) is like sed for JSON data - you can slice, filter, map, and transform structured data with the same ease that sed, awk, grep, and friends let you play with text.
+
+#### Using Code to Create JSON JQ Transform Task
+
+```cs
+To Do
+```
+
+#### JSON Configuration
+
+```json
+{
+  "name": "json_transform_task",
+  "taskReferenceName": "json_transform_task_ref",
+  "type": "JSON_JQ_TRANSFORM",
+  "inputParameters": {
+    "key1": "k1",        
+    "key2": "k2",
+    "queryExpression": "{ key3: (.key1.value1 + .key2.value2) }",
+  }
+}
 ```
 
-Check out our [integration tests](https://github.com/conductor-sdk/conductor-csharp/blob/92c7580156a89322717c94aeaea9e5201fe577eb/Tests/Worker/WorkerTests.cs#L37) for more examples
+## Worker vs. Microservice/HTTP Endpoints
 
-Worker SDK collects the following metrics:
+>[!tip]
+>Workers are a lightweight alternative to exposing an HTTP endpoint and orchestrating using HTTP tasks. Using workers is a recommended approach if you do not need to expose the service over HTTP or gRPC endpoints.
 
+There are several advantages to this approach:
 
-| Name               | Purpose                                      | Tags                             |
-| ------------------ | :------------------------------------------- | -------------------------------- |
-| task_poll_error    | Client error when polling for a task queue   | taskType, includeRetries, status |
-| task_execute_error | Execution error                              | taskType                         |
-| task_update_error  | Task status cannot be updated back to server | taskType                         |
-| task_poll_counter  | Incremented each time polling is done        | taskType                         |
-| task_poll_time     | Time to poll for a batch of tasks            | taskType                         |
-| task_execute_time  | Time to execute a task                       | taskType                         |
-| task_result_size   | Records output payload size of a task        | taskType                         |
+1. **No need for an API management layer** : Given there are no exposed endpoints and workers are self-load-balancing.
+2. **Reduced infrastructure footprint** : No need for an API gateway/load balancer.
+3. All the communication is initiated by workers using polling - avoiding the need to open up any incoming TCP ports.
+4. Workers **self-regulate** when busy; they only poll as much as they can handle. Backpressure handling is done out of the box.
+5. Workers can be scaled up/down quickly based on the demand by increasing the number of processes.
 
-Metrics on client side supplements the one collected from server in identifying the network as well as client side issues.
+## Deploying Workers in Production
 
-### Next: [Create and Execute Workflows](https://github.com/conductor-sdk/conductor-csharp/blob/main/docs/readme/workflow.md)
+Conductor workers can run in the cloud-native environment or on-prem and can easily be deployed like any other C# application. Workers can run a containerized environment, VMs, or bare metal like you would deploy your other C# applications.