-
Notifications
You must be signed in to change notification settings - Fork 120
Liota Developer Guide
Little IoT Agent (liota) is an open source project, offering convenience for IoT solution developers in creating IoT Edge System data orchestration applications. Liota allows interaction with data-center components over any transport for any IoT edge system. It is easy-to-use and provides the enterprise-quality platform for interacting with IoT solutions.
Python 2.7.9
BSD 2-Clause
The six basic abstractions of liota represent a complete data flow from a device attached to the edge system to an application in a data-center. These are 'Device' (often called “data source” on or attached to the edge system), 'DeviceComms' (communication protocols used between devices and the edge system), 'EdgeSystem' (the edge system hardware and software platforms), 'Metric' (represents a time-series stream from a data source to a data-center application), 'DCCComms' (communication protocols between the edge system and a data-center), and 'DCC' (data-center component, that is an ingestion application in a data-center). We have abstract classes for all six of these constructs in a small object hierarchy and a number of concrete classes that comprise this release of liota and allows a liota package or set of packages to create complete data flows from a data source to a component in a data-center.
Entities represent abstractions for the three main constructs: System, Devices, and Metrics that can be, depending on the DCC, registered with the DCC. Such registration typically creates a concrete representation (let's call these 'Resources' simply to differentiate from the local representation we'll call objects) in the data-center component for the local object. Various pieces of metadata can be associated with the Resource typically created during or after registration. Examples of such meta-data associated with the Resource, metrics, entries in a key/value store, relationships to other Resources, alerts, and actions.
Entity: Abstract base class for all the entities(Device, Edge System, and Metric are
considered as entities in Liota terminology), it requires following arguments for initializing:
name: Entity name,
entity_id: Entity ID,
entity_type: Entity TypeEach DCC must implement a register() method and return a RegisteredEntity object. The registered entity object may include any specific data, e.g., uuid, that the DCC may need to refer to the Resource representing the entity object. An entity may be registered with multiple DCCs in the same liota package or set of packages.
RegisteredEntity: Represents the registered (Device, Edge System & Metric) entity object. Requires
following argument for initializing:
ref_entity: Entity that is represented by a RegisteredEntity Object,
ref_dcc: DCC with which the Entity is registered,
reg_entity_id: RegisteredEntity ID
set_properties(properties): instance method to set properties for the registered entity objectThe abstract subclasses of Entities, EdgeSystem and Device contains logic abstractions that are common to all edge system or connected devices. We expect, as concrete implementations are created over time that we'll see some common data and logic that we can move up into the abstract classes.
EdgeSystem: Abstract base class for all edge systems (Gateways), requires following argument for initializing:
name: EdgeSystem name
entity_id: Entity ID
entity_type: Entity type which is set to "EdgeSystem"Note: We recommend creating EdgeSystem Objects before creating DCCComms or DeviceComms objects.
Device: Abstract base class for the all the edge systems (Gateways), requires following argument
for initializing:
entity_id: locally generated id using Liota,
entity_type: Entity type which is set to "Device"The Metric subclass of Entity is the local object representing a stream of (number, timestamp) tuples. Metrics may be registered with one or more DCCs and the DCC returns a registered metric object. The metric object includes a sampling function which is a user defined method (udm), a sampling frequency stating the interval between subsequent executions of the udm and an aggregation count stating how many executions of the udm to aggregate before sending to the DCCs to which the metric has been registered. An important piece of meta-data liota supports are SI units and a prefix eliminating any confusion as to what the stream of numbers represents.
Metric: Defines the metric object which requires following argument for initializing:
name: metric name,
unit: unit defined for metric, liota uses by default Pint library to set SI units,
interval: time interval required to collect the metric value,
aggregation_Size: the number of values to be aggregated before publishing to DCC,
sampling_function: Method responsible for collecting the sensor values
register (dcc_obj, reg_entity_id): this method is used to return the registered metric object
requires parameters dcc_obj(DCC Object) & reg_entity_id (Registered entity unique identifier)The abstract class DeviceComms represents mechanisms through which devices send and receive data to/from edge systems. Some examples are CAN bus, Modbus, ProfiNet, Zibgee, GPIO pins, Industrial Serial Protocols as well as sockets, websockets, MQTT & CoAP. The DeviceComms abstract class is a representation for the various device communication mechanisms.
device_comms: Abstract class for all device communications
_connect(): Establish connection to device
_disconnect(): Disconnect the connection to device
send(message): Publish message to device if it receives messages or acts as an actuator
receive(): Receive the data from the deviceThe abstract class DCCComms represents communication protocols between edge systems and DCCs. Currently, liota supports MQTT, WebSocket, and plain old BSD sockets. In the near future, it will also support CoAP. MQTT, WebSocket and CoAP are 'Application' or layer-7 protocols.
MQTT is a pub-sub system using TCP and CoAP that implements reliable UDP datagrams and a data format specification. These protocols are capable of satisfying most of the use cases for transferring data from the IoT gateways to data-center components. With the current implementation, the gateway acts as MQTT, WebSocket or a traditional Socket client.
dcc_comms: Abstract class for all the DCC communications.
_connect(): Establish connection to DCC
_disconnect(): Disconnecting the connection to DCC
send(message,msg_attr): Publish sensor values to DCC, msg_attr is used
to hold the specific parameters required to publish the data
receive(msg_attr): Receive messages from the DCC, msg_attr is used to
specify the required parametersThe abstract class DCC represents an application in a datacenter. It is potentially the most important and complex abstraction of liota. It provides flexibility to developers for choosing the datacenter components they need using API's provided by liota. With the help of this abstraction, developers may build custom solutions. The abstract class states basic methods and encapsulates them into unified common API's required to send data to various DCC's. Graphite and Pulse IoT are currently the data-center components supported with AWS, BlueMix, and ThingWorx to come soon. New DCCs can easily be integrated into the abstraction.
DCC: Abstract class for all DCC's
register(entity_obj): Register entity object with DCC
create_relationship(reg_entity_parent, reg_entity_child): Create child-parent relationship
between two entity objects
_format_data(reg_metric): Formats the collected reg_metric(Registered Metric) data, sensor
values as per the DCC format before publishing
publish(reg_metric): An instance method used to publish the data to DCC, it also uses
reg_metric.msg_attr for holding the various parameters which might be required to publish
the message.
set_properties(reg_entity, properties): Used to set properties for the required reg_entity.
unregister(entity_obj): Unregister the entity object from DCCOnly required if you do not use the Pulse IoT Center bootstrap package. If you would like to use this package send an e-mail to [email protected]
In general, liota can be installed using Python Package Index (PyPI):
$ pip install liotaCloning the Git Project and installing using setup.py:
$ git clone https://github.com/vmware/liota.git
$ cd liota
$ python setup.py install After installing Liota you can manually copy the configuration files from "/usr/lib/liota/config/" to "/etc/liota" and create "/var/log/liota" directory.
Or you can use the helper script "post-install-setup.sh" to copy the config files which exist at the path "/usr/lib/liota".
The script by default checks if the "liota" non-root user exists and if it doesn't then non-root "Liota" user is required to be created manually. If you require Liota to be installed as the different non-root user which pre-exists on the system, then the script will be required to be executed in the following way:
$ cd /usr/lib/liota
$ LIOTA_USER="non-root user" ./post-install-setup.shIf Liota is required to be installed as a root user (not the preferred way), then the script should be executed in the following way:
$ cd /usr/lib/liota
$ LIOTA_USER="root" ./post-install-setup.shThe Liota configuration file and logging.json file exist at this path after installation:
$ cd /usr/lib/liota/config/liota.conf
$ cd /usr/lib/liota/config/logging.jsonLiota supports BSD sockets, WebSocket, and MQTT communication protocols. Refer MQTT Section to learn the MQTT configuration options that are available.
Identity and TLS Configurations
-
Identity class encapsulates certificates or credentials related to a connection used at both DCC and Device side.
-
TLSConf class encapsulates parameters related to TLS configuration.
-
CRLs(Certificate Revocation Lists) check has been implemented: VERIFY_CRL_CHECK_CHAIN.
In this mode CRLs of all certificates in the peer cert chain are checked. In order to enable the CRLs set the 'crl_path' in '/etc/liota/liota.conf', it requires a valid path to CRL in PEM format (preferable directory '/usr/lib/liota'). If the check is required to be disabled 'crl_path' should be left blank.
Liota supports SI units and the conversion of the units with help of Pint library which, is included in the liota package to provide developers the capability to use SI units in their code. We have also included the example simulated_graphite_temp.py which uses the library to convert temperature values from Celsius to Fahrenheit and Kelvin. More details on the usage of the Pint library and conversion of units can be found at this Pint Link.
The default location for log files generated during Liota operation can be found at:
/var/log/liotaIf the above directory is not available or is not writeable then modify the log location in the file '/etc/liota/logging.json'.
Liota can be easily uninstalled as per the steps below:
$ pip uninstall liota
$ rm -rf /usr/lib/liota/ /etc/liota/ /var/log/liota/Liota applications can be broken into small package modules that can be loaded and unloaded into a running liota process. We recommend putting the Edge Systems, Devices, Metrics and DCC(s) into separate packages. Then each construct can be loaded and unloaded at will. See the README in the package directory for complete details.
The Liota package manager consists of two parts:
(1) A PackageThread that loads files, maintains global data structures and runs package initialization/clean-up code.
(2) A PackageMessengerThread which listens on one or more specific communication channels, which is a named pipe for now, to provide an interface for users and automated agents to send commands to the package manager.
The Package manager will initialize the data structures and threads when its own module package_manager.py is imported. Once imported, PackageMessengerThread will start listening on the pipe and PackageThread will load all packages specified in the packages_auto.txt in a batch.
Currently, supported commands include package action commands and statistical commands.
Package action commands:
- load package_name <sha1_checksum>
Load a package with the specified name and its sha1 checksum, even multiple packages can be loaded in the single command by giving the other package name followed by its checksum. For example, linux os user can first use "$ sha1sum filename" cmd to get checksum, and then load package:
$ ./liotapkg.sh load filename <sha1_checksum>A python file cal_sha1sum.py is also provided to help you calculate checksum for a file:
$ python cal_sha1sum.py file_name
(could be relative or absolute file name).
For example, under /usr/lib/liota/packages the path will be
$ python cal_sha1sum.py iotcc_mqtt.pyIf the specified package provides a list of dependencies, recursively load all its dependencies. If more than one package name is specified, load them (as well as their dependencies) in a batch and no package will be loaded twice or reloaded. Liota packages must follow certain formats for the package manager to process them correctly. It is up to the package developer to follow the format requirements. If the dependency lists of specified packages and their dependencies contain loops, then all packages involved in the loop will not be loaded.
- unload package_name
Unload a package with the specified name. If the specified package has dependents loaded, all its dependents are unloaded recursively. If more than one package names are specified, unload them (as well as their dependents) in a batch.
To unregister an entity while unloading, set the following flag in /usr/lib/liota/packages/sampleProp.conf to True:
ShouldUnregisterOnUnload = "True"- reload package_name <sha1_checksum>
Unload a package with the specified name and attempt to reload the same package using the same file name. Batch operation is not supported for reloading. If the specified package is not loaded when this command is invoked, the command will fail.
- update package_name <sha1_checksum>
Unload a package with the specified name and attempt to reload the same package, even multiple packages can be updated in the single command by providing the other package name followed by its checksum. If the specified package has dependents loaded, attempt to recursively update all these dependents. If the specified package is not loaded when this command is invoked, skip unloading and load the specified package directly.
- delete package_name
Remove a package with the specified name. By default, the removed package will be stashed into a separate folder in the package path, so package manager will not find it. However, if package manager fails to create the folder, or fails to move the file, the package file will be deleted from the file system.
Note: If the package_name contains space or special characters, it should be enclosed with in single/double quotes. e.g.,
$ ./liotapkg.sh load -r 'filename 1' <sha1_checksum>Load Liota Packages automatically when the Package Manager starts by listing package names and checksums in the file specified by pkg_list in [PKG_CFG] of '/etc/liota/liota.conf', e.g., by default '/usr/lib/liota/packages/liotad/packages_auto.txt' (This file should NOT have blank spaces while specifying the below command):
package_name:<sha1_checksum>There are 2 options to add liota package names and checksum:
- manually write into pkg_list file;
- add at run time through command [load, reload, update] by specifying the "-r" option, e.g.,
$ cd /usr/lib/liota/packages
$ ./liotad/liotapkg.sh load -r filename <sha1_checksum>To be reminded, unload command will remove it from pkg_list file.
-
stat met|col|th Print statistical data in the Liota log about metrics, collectors and Python threads respectively.
-
stat pkg package_name Print the status of a package (loaded or not)
-
list pkg|res|th Print a list of package, resources (shared objects) and threads respectively. Note: PackageThread is just a single thread. If your package contains time-consuming or blocking lines in its run() or clean_up() methods, it will cause PackageThread to wait/block and stop accepting coming commands.
PackageMessengerThread listens on a named pipe whose location is defined in '/etc/liota/liota.conf'. It then parses texts received from the pipe into commands and arguments and sends them to PackageThread.
Different techniques can be supported in PackageMessengerThread in the future.
There is a LiotaPackage class defined in package_manager.py which looks like
class LiotaPackage:
"""
LiotaPackage is the abstract base class (ABC) of all package classes.
It should define abstract methods that developers should implement.
"""
__metaclass__ = ABCMeta
@abstractmethod
def run(self, registry):
raise NotImplementedError
@abstractmethod
def clean_up(self):
raise NotImplementedErrorDevelopers are required to implement their packages by inheriting from LiotaPackage and implementing the two abstract methods run() and clean_up().
run() is called upon initialization and may establish necessary communication channels, parse configuration files and create metrics using user-defined methods as sampling functions. Developers need to explicitly register shared objects in run(), if they want these objects to be accessible by other packages. clean_up() is called upon package unloading and may involve notifying auxiliary threads to stop, notify metrics to stop collecting and other cleaning up.
Note: Package manager will not do things that it cannot do, for example, terminating an auxiliary thread that is blocked on I/O. It is up to the developers to write non-blocking code and create only stoppable threads (or not at all). If the package file contains syntax errors or significant logic errors, the behavior of package manager is undefined. Please also refer to packages and packages/examples for example packages we have provided for better understanding of package manager.
Basic user packages which we believe should exist on the edge system to publish the basic health stats to Pulse IoT Control Center. These packages should be placed at path '/usr/lib/liota/packages'.
The sample package contains specifications of GeneralEdgeSystem, and replaces the "EdgeSystemName" with the logic to auto-generate the unique name every time this package is loaded based on the user flow. The unique identifier can be from the system (e.g. the MAC address) or devices can generate their unique name.
This is a sample user package which creates a IoTControlCenter DCC object and registers edge system on IoTCC over MQTT Protocol to acquire "registered edge system", i.e. iotcc_edge_system. This package has the dependency on credentials package which is pre-loaded during the installation in order to provide the required credentials and configuration parameters in the liota package manager registry. The properties for the edge system can be set as 'key:value' pair, you can also set the location by passing the 'latitude:value' and 'longitude:value' as properties in the user package. If the unregister call is included in the clean-up method then the resource will be unregistered and the entire history will be deleted from Pulse IoT Control Center so comment the logic if the unregsitration of the resource is not required to be done on the package unload. The retry mechanism has been implemented for important calls like registration, create_relationship or set_property in case of the exception. User Configurable Retry and Delay Settings can be tweaked by the user as per the targeted scale. The user can also override by default MQTT “conn_disconn_timeout” value in the package while defining “MqttDccComms”. It is the configurable timeout in seconds required for MQTT protocol connection and disconnection.
This is a sample user package to publish the basic edge system stats which we believe are required to monitor the health status of the edge system from Pulse IoT Control Center. Optional mechanism: If the device raises an intermittent exception during metric collection process it will be required to be handled in the user code otherwise if an exception is thrown from user code the collection process will be stopped for that metric. If the None value is returned by User Defined Method(UDM) then the metric value for that particular collector instance won't be published.
This is a sample device package which registers five devices to Pulse IoT Control Center and then a relationship is established to Edge System. A basic UDM returns random value it should be tweaked by the user in order to collect device-specific metric, all the five devices are loaded with dev_metric. The retry mechanism has been implemented for important calls like registration, create_relationship or set_property in case of the exception. User Configurable Retry and Delay Settings can be tweaked by the user as per the targeted scale.
Send e-mail to [email protected] if you would like to have the bootstrap package installation for Pulse IoT Center.