Configuration Setup

Configuration and setup

The transfer2go tool relies on two configuration files: the server config file and catalog one. Both should be provided in JSON data-format, see details below. The transfer2go tool combines both server and a client based on the way you use it, see below.

Server examples

# start the server based on provided configuration file
transfer2go -config config.json

# start the server with given config file
# as well as provide registration end-point (another agent where this one will be registered)
# and increase verbosity level
transfer2go -config config.json -agent http://localhost:8989 -verbose 1

Client examples

# we can register local files in our agent by providing records.json files, e.g.

[
    {"lfn":"/store/file.root",
     "pfn":"/path/file3.root",
     "block":"/a/b/c#123",
     "dataset":"/a/b/c"}
]

# and now we can upload these records to the agent catalog
transfer2go -register=records.json -agent=http://localhost:8989

# if we want to transfer a file to T2_site_name, we'll do
transfer2go -agent http://localhost:8989 -src=file.root -dst=T4_site_name

# similarly we can transfer a block (/a/b/c#123) or a dataset (/a/b/c), e.g.
transfer2go -model=pull -agent=http://localhost:8989 -src=SourceAgent-2:a/b/c -dst=DestinationAgent -auth=false
transfer2go -model=pull -agent=http://localhost:8989 -src=SourceAgent-2:a/b/c#123 -dst=DestinationAgent -auth=false

we also may acquire some info about running agents via plain HTTP requests, e.g.

# get dump of agent TFC (Trivial File Catalog)
curl http://localhost:8987/tfc

# get status of specific agent
curl http://localhost:8987/status

and, we can update running agent:

# change agent's protocol to new one
curl -X POST -d '{"protocol":"srm", "backend":"srmv2", "tool":"/opt/VDT/srm-client-lbnl/bin/srmcp"}' http://localhost:8989/protocol

# reset agent's protocol to default one (http)
curl http://localhost:8989/reset

Configuration example

Configuration file is JSON based and contains the following fields:

• catalog, a name of catalog configuration file
• protocol, a name of protocol to be used by the agent, e.g. srmv2
• backend, a backend end-point
• tool, a unix path to internal tool used for transfer, e.g. /usr/bin/bla
• mfile, name of the metrics file, i.e. file where various agent's metrics will be stored
• minterval, a metrics interval, i.e. how often agent metrics values will be acquired
• verbose, verbosity level
• name, a name of the agent (a node name in PhEDEx)
• register, a remote agent URL where we can register this one
• type, pull/push in order to active the respective models. In pull model start main-agent in pull type while rest of them should be push type.
• MonitorTime, time after which need to reset and recalculate the system param.
• router, make it true to use the smart router.
• trinterval, parameter for the router to retrain the ML model after certain interval. Here is an example of such file with simple UNIX cp command as internal tool.

{
    "catalog":"test/catalog/main.json",
    "protocol":"cp",
    "backend":"/tmp",
    "tool":"/bin/cp",
    "url":"http://localhost:8989",
    "mfile":"metrics.log",
    "csvfile":"history.csv",
    "minterval":60,
    "verbose":0,
    "name":"Test",
    "staticdir":"static",
    "type":"pull",
    "MonitorTime":3600,
    "trinterval": "10s",
    "router":true
}

Catalog example

A catalog file is simple JSON file which describes catalog nature, e.g.

{
    "type":"sqlite3",
    "uri":"/path/catalog.db"
}

Here it is SQLite database (type sqlite3) and its uri. The Catalog file may contain also Login/Password/Owner fields necessary for other databases.

Test Setup

Create three different database catalog.db, catalog2.db, main.db under test/catalog folder using schema of static/sql/sqlite3/schema.sql file.

$ bash test.sh