Skip to content

Commit

Permalink
Update README (#60)
Browse files Browse the repository at this point in the history
  • Loading branch information
@Pedrobc89
Pedrobc89 authored Dec 11, 2024
1 parent 9c3c742 commit 14d24c2
Showing 1 changed file with 89 additions and 88 deletions.
177 changes: 89 additions & 88 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,40 +1,69 @@
# Hyperon Distributed Atom Space Algorithm Node

The Distributed Atom Space Algorithm Node (aka DAS Node) is a component of DAS
(<https://github.com/singnet/das>) which allows the implementation of
distributed algorithms using one or more DAS as shared knowledge base.
The Hyperon Distributed Atom Space Algorithm Node (aka DAS Node) is a core
component of the Distributed AtomSpace (DAS)
(<https://github.com/singnet/das>), enabiling the implementation of distributed
algorithms across DAS instances using one or more shared knowledge base. This
documentation provides an overview of the DAS Node's purpose, architecture, and
usage, catering to users with no prior knowledge of the system.

This component is aimed at two different use cases:

DAS' query engine uses it to implement a distributed algorithm for query
resolution. Users of DAS libraries can use it to implement distributed
algorithms by exploring primitives like leadership election and message
exchanging built in the component.
## Introduction

## Features
Remote DAS queries will be triggered by a specialization of our
DistributedALgorithmNode which will connect client (query caller) and server
(DAS Server running a Redis-mongo atomdb) using command lines (this is the way
AtomSpaceNode exchange messages

### DistributedAlgorithmNode
<!--TODO: Add a chart representing an arbitrary number of DAS Node's quering on
an arbitrary number of DAS Servers-->

## Key Concepts and Components

DistributedAlgorithmNode is an abstract class that represents a node in a network,
designed for distributed processing. Users must extend this class to implement
custom distributed networks.
### Messages

Communication between nodes is done through Message objects. These objects act
like commands exchanged between nodes, although the actual code isn't
transmitted. Instead, a command identifier is sent, and the receiving node
reconstructs and executes the corresponding Message object. The command format
resembles a command-line interface, with commands and arguments.

Message is an abstract class representing messages exchanged between nodes.
Each subclass should implement the act() method, which defines the behavior
that will be executed on the receiving node.

`Messages` are exchanged through a `MessageBroker`, which implements which
communication protocol we will be using. Currently there are the following
implementations available:

- SynchronousSharedRAM
- SynchronousGRPC
- AsynchronousMQTT

### Leadership

In distributed algorithms, a common problem is coordination, there are a number
of known algorithms to solve for that such as Raft, Paxos among others.
LeadershipBroker defines the API for leader election in the network. Users
typically don't interact with this class directly; it's managed by the
DistributedAlgorithmNode.

### DistributedAlgorithmNode

DistributedAlgorithmNode is an abstract class that represents a node in a
network, designed for distributed processing. Users must extend this class to
implement custom distributed networks.

Key Points for Extending DistributedAlgorithmNode:

- DistributedAlgorithmNode builds Message objects because it inherits from
MessageFactory. The base class can create basic Message objects for common
tasks, such as handling new nodes joining the network. However, subclasses
should override message_factory() to handle application-specific messages.
- DistributedAlgorithmNode builds Message objects because it inherits from
MessageFactory. The base class can create basic Message objects for common
tasks, such as handling new nodes joining the network. However, subclasses
should override message_factory() to handle application-specific messages.

- Message execution is threaded. If commands update shared state in
DistributedAlgorithmNode or its subclasses, you must protect this state using mutual
exclusion mechanisms.
DistributedAlgorithmNode or its subclasses, you must protect this state using
mutual exclusion mechanisms.

- The constructor for DistributedAlgorithmNode requires a MessageBroker and a
LeadershipBroker, both of which are abstract. You must choose concrete
Expand All @@ -43,96 +72,68 @@ Key Points for Extending DistributedAlgorithmNode:
may be needed, depending on the network topology and application
requirements.

- DistributedAlgorithmNode has several pure virtual methods that must be implemented by
subclasses. These methods handle fundamental tasks, such as leadership
elections and notifying nodes when new peers join the network.

### Message

Message is an abstract class representing messages exchanged between nodes.
Each subclass should implement the act() method, which defines the behavior
that will be executed on the receiving node.

Messages aren't serialized for transmission. Instead, an identifier is sent,
allowing the recipient to instantiate the appropriate Message subclass and
invoke its act() method.

### LeadershipBroker

LeadershipBroker defines the API for leader election in the network. Users
typically don't interact with this class directly; it's managed by the
DistributedAlgorithmNode.
- DistributedAlgorithmNode has several pure virtual methods that must be
implemented by subclasses. These methods handle fundamental tasks, such as
leadership elections and notifying nodes when new peers join the network.

#### SingleMasterServer

This is the only current implementation of LeadershipBroker. It assumes a
topology where a single master server (the leader) communicates with multiple
clients.
## Examples

### MessageBroker
We provide bindings to Python to ease usability and fast prototyping.
Import the python package and create your own DistributedAlgorithmNode:

MessageBroker defines the API for the communication layer between nodes. Users
of DistributedAlgorithmNode don't interact with MessageBroker directly.
```python
from hyperon_das_node import (
DistributedAlgorithmNode,
LeadershipBrokerType,
MessageBrokerType
)

Currently there are two implementations: RAM and GRPC
class MyDasNode(DistributedAlgorithmNode):
def __init__(self, node_id):
super().__init__(
node_id,
LeadershipBrokerType.SINGLE_MASTER_SERVER,
MessageBrokerType.GRPC,
)

#### SynchronousSharedRAM (RAM)

This implementation uses shared memory to exchange Message objects between
nodes running within the same process. It's suitable for scenarios where all
nodes run in a shared memory space.
def message_factory(self, command: str, args: list[str]) -> Message:
message = super().message_factory(command, args)
if message is not None:
return Message

#### SynchronousGRPC (GRPC)
# Create you custom Messages objects here
...

This implementation uses gRPC for inter-node communication. It sets up gRPC
servers and clients, using queues and threads to handle incoming and outgoing
messages asynchronously.
```

When broadcast() or send() methods are called, the command is added to the
outgoing queue and sent asynchronously. Each node listens for incoming requests
via a gRPC server, processes them, and queues them for execution.
Inside the examples folder you can find a number of examples of how to use the
library. The `simple_node.py` shows a very simple example of how to use the
library.

No rpc answer is used in these GRPC calls. So if a command expects an answer to
return, this answer is supposed to be implemented as a separate Message going
back from the target node to the node that originated the request.

## Building from source

The build process involves two steps: first building the C++ code with Bazel,
and then building and installing the Python package.

We are using a docker image with `bazel` installed. To build the image you can run:

```sh
./scripts/docker_image_build.sh
# Once the image is built we can run the container
./scripts/container_tty.sh
# This will build the cpp code and copy files of interest to bazel_assets dir
cd src
../scripts/bazel_build.sh
cd ..
```

Once the image is built and the code is built, all the libraries will be copied to a folder named `bazel_assets`.
This folder will be used to build the python package.
We are using a docker image with `bazel` installed. To build the image you can
run:

To install the package inside the docker container, you can run:
```sh
pip install .
# This allows to run tests in python inside the container.
make cpp-image
make cpp-build
make wheel-image
make wheel-build
```

### Exporting the python package
Once the image is built and the code is built, all the libraries will be copied
to a folder named `bazel_assets`. This folder will be used to build the python
package.

If you wish to install the python package in another machine (currently only ubuntu is supported), run the following command:
To install the package inside the docker container, you can run: ```sh pip
install . # This allows to run tests in python inside the container. ```

```sh
pip wheel .

make install
```

The output of the wheel can be copied to the local machine and installed manually.

## Examples

Inside the examples folder you can find a number of examples of how to use the library. The `simple_node.py` shows a very simple example of how to use the library.

0 comments on commit 14d24c2

Please sign in to comment.