Skip to content

Latest commit

 

History

History
345 lines (299 loc) · 10.5 KB

model_server_rest_api.md

File metadata and controls

345 lines (299 loc) · 10.5 KB

OpenVINO™ Model Server RESTful API Documentation

Introduction

In addition with gRPC APIs OpenVINO™ model server also supports RESTful APIs which follows the documentation from tensorflow serving REST API. Both row and column format of the request are implemented in these APIs. REST API is recommended when the primary goal is in reducing the number of client side python dependencies and simpler application code.

Note : Only numerical data type is supported.

This document covers following API:

Note : The implementations for Predict, GetModelMetadata and GetModelStatus function calls are currently available. These are the most generic function calls and should address most of the usage scenarios.

Model Status API

  • Description

Get information about the status of served models

  • URL
GET http://${REST_URL}:${REST_PORT}/v1/models/${MODEL_NAME}/versions/${MODEL_VERSION}

Note : Including /versions/${MODEL_VERSION} is optional. If omitted status for all versions is returned in the response.

  • Response format

If successful, returns a JSON of following format :

{
  'model_version_status':[
    {
      'version': <model version>|<string>,
      'state': <model state>|<string>,
      'status': {
        'error_code': <error code>|<string>,
        'error_message': <error message>|<string>
      }
    }
  ]
}
  • Usage Example
$ curl http://localhost:8001/v1/models/person-detection/versions/1
{
  'model_version_status':[
    {
      'version': '1', 
      'state': 'AVAILABLE', 
      'status': {
        'error_code': 'OK', 
        'error_message': ''
      }
    }
  ]
}

Read more about Get Model Status API usage here

Model Metadata API

  • Description

Get the metadata of a model in the model server.

  • URL
GET http://${REST_URL}:${REST_PORT}/v1/models/${MODEL_NAME}/versions/${MODEL_VERSION}/metadata

Note :Including ${MODEL_VERSION} is optional. If omitted the model metadata for the latest version is returned in the response.

  • Response format

If successful, returns a JSON representation of GetModelMetadataResponse protobuf.

  • Usage example
$ curl http://localhost:8001/v1/models/person-detection/versions/1/metadata
{
  "modelSpec": {
    "name": "person-detection",
    "version": "1"
  },
  "metadata": {
    "signature_def": {
      "@type": "type.googleapis.com/tensorflow.serving.SignatureDefMap",
      "signatureDef": {
        "serving_default": {
          "inputs": {
            "data": {
              "name": "data_2:0",
              "dtype": "DT_FLOAT",
              "tensorShape": {
                "dim": [
                  {
                    "size": "1"
                  },
                  {
                    "size": "3"
                  },
                  {
                    "size": "400"
                  },
                  {
                    "size": "600"
                  }
                ]
              }
            }
          },
          "outputs": {
            "detection_out": {
              "name": "detection_out_2:0",
              "dtype": "DT_FLOAT",
              "tensorShape": {
                "dim": [
                  {
                    "size": "1"
                  },
                  {
                    "size": "1"
                  },
                  {
                    "size": "200"
                  },
                  {
                    "size": "7"
                  }
                ]
              }
            }
          },
          "methodName": "tensorflow/serving/predict"
        }
      }
    }
  }
}

Read more about Get Model Metadata API usage here

Predict API

  • Description

Sends requests via TensorFlow Serving RESTful API using images in numpy format. It displays performance statistics and optionally the model accuracy.

  • URL
POST http://${REST_URL}:${REST_PORT}/v1/models/${MODEL_NAME}/versions/${MODEL_VERSION}:predict
  • Request Header
{
  // (Optional) Serving signature to use.
  // If unspecifed default serving signature is used.
  "signature_name": <string>,

  // Input Tensors in row ("instances") or columnar ("inputs") format.
  // A request can have either of them but NOT both.
  "instances": <value>|<(nested)list>|<list-of-objects>
  "inputs": <value>|<(nested)list>|<object>
}

Note Read How to specify input tensors in row format and How to specify input tensors in column format for more details.

  • Response

A request in row format has response formatted as follows :

{
  "predictions": <value>|<(nested)list>|<list-of-objects>
}

A request in column format has response formatted as follows :

{
  "outputs": <value>|<(nested)list>|<object>
}

Read more about Predict API usage here

Config Reload API

  • Description

Sends requests via RESTful API to trigger config reloading and gets models and DAGs statuses as a response.This endpoint can be used with disabled automatic config reload to ensure changes in configuration are applied in a specific time and also to get confirmation about reload operation status. Typically this option is to be used when OVMS is started with a parameter --file_system_poll_wait_seconds 0. Reload operation does not pass new configuration to OVMS server. The configuration file changes needs to be applied by the OVMS administrator. The REST API call just initiate applying the configuration file which is already present.

  • URL
POST http://${REST_URL}:${REST_PORT}/v1/config/reload
  • FLOW

Flow after receiving request:

  1. If config file was changed - reload config.
  2. If any model version directory was changed or new version was added - reload this model.
  3. If any model that is part of a DAG was changed or new version was added - reload this pipeline.
  4. In case there are no errors in the reload operation, the response includes the status of all models and DAGs, otherwise error message is returned.
  • Request
    To trigger reload, HTTP POST request with empty body should be sent on given URL. Example curl command:
curl --request POST http://${REST_URL}:${REST_PORT}/v1/config/reload
  • Response
    In case of config reload success, response contains JSON with aggregation of getModelStatus responses for all models and DAGs after reload is finished, along with operation status:
{ 
"<model name>": 
{ 
  "model_version_status": [     
  { 
     "version": <model version>|<string>,
     "state": <model state>|<string>, 
     "status":
{ 
  "error_code": <error code>|<string>, 
  "error_message": <error message>|<string>       
} 
  }, 
  ...  
] 
}, 
... 
}

In case of any failure during execution:

{ 
  "error": <error message>|<string> 
}

When operation succeeds HTTP response status code is

  • 201 when config(config file or model version) was reloaded
  • 200 when reload was not required, already applied or OVMS was started in single model mode When operation fails another status code is returned.

Possible messages returned on error:

  • obtaining config file change time failed (file is not exisiting or cannot be accessed):
{
  "error": "Config file not found or cannot open."
}
  • config file was changed and config reloading failed (file content is not a valid JSON, any of model or DAG config is incorrect):
{
  "error": "Reloading config file failed. Check server logs for more info."
}
  • config file was not changed and model versions reloading failed (model directory was removed):
{
  "error": "Reloading models versions failed. Check server logs for more info."
}
  • retrieving status of one of the models failed:
{
  "error": "Retrieving all model statuses failed. Check server logs for more info."
}
  • converting model status responses to json failed:
{
  "error": "Serializing model statuses to json failed. Check server logs for more info."
}

Even if one of models reload failed other may be working properly. To check state of loaded models use Config Status API. To detect exact cause of errors described above analyzing sever logs may be necessary.

Config Status API

  • Description

Sends requests via RESTful API to get response that contains aggregation of getModelStatus responses for all models and DAGs.

  • URL
GET http://${REST_URL}:${REST_PORT}/v1/config
  • Request
    To trigger this API HTTP GET request should be sent on given URL.Example curl command:
curl --request GET http://${REST_URL}:${REST_PORT}/v1/config
  • Response
    In case of success, response contains JSON with aggregation of getModelStatus responses for all models and DAGs, along with operation status:
{ 
"<model name>": 
{ 
  "model_version_status": [     
  { 
     "version": <model version>|<string>,
     "state": <model state>|<string>, 
     "status":
{ 
  "error_code": <error code>|<string>, 
  "error_message": <error message>|<string>       
} 
  }, 
  ...  
] 
}, 
... 
}

In case of any failure during execution:

{ 
  "error": <error message>|<string> 
}

When operation succeeded HTTP response status code is 200, otherwise another code is returned. Possible messages returned on error:

  • retrieving status of one of the models failed:
{
  "error": "Retrieving all model statuses failed. Check server logs for more info."
}
  • converting model status responses to json failed:
{
  "error": "Serializing model statuses to json failed. Check server logs for more info."
}