Test setup

Introduction

To perform the tests we need 3 nodes, the main agent (coordinator), the source agent (which keeps our data) and destination agent (where we'll relocate the data). We define them as following:

mainhost=main-agent.cern.ch
sourcehost=source-server.cern.ch
desthost=uibo-cms-02.cr.cnaf.infn.it

Each host should have transfer2go executable which can be obtained as

git clone [email protected]:vkuznet/transfer2go.git

You may compile code as simple as running make within transfer2go area.

There is a pair of configuration files per given host test/catalog/main.json and test/config/main.json, and similar for source and destination hosts.

Before running the code you need to have local DB setup, make it as following:

sqlite3 <dbFileName> < static/sql/sqlite3/schema.sql

and place these db files into test/catalog area

NOTE: Make sure your dbFileName is same as mentioned in test/catalog/main.json file.

Need go version >= 1.8 to compile it.

If you don't have complete access to agent's ports you need to setup SSH tunnels to them.

Setting up SSH tunnels

By default the 8989, 8000, 9000 ports are used for main, source and destination hosts, respectively. You can do it as following:

# PORT_NUMBER is a port number on remote host, it will be forwarded to the same port on localhost
$ ssh -S none -L PORT_NUMBER:localhost:PORT_NUMBER hostname -fN

For example:

# tunnel ports from destination to source and main hosts
$ ssh -S none -L 8000:localhost:8000 $sourcehost -fN
$ ssh -S none -L 8989:localhost:8989 $mainhost -fN

# tunnel ports from main host to source and destination
$ ssh -S none -L 8000:localhost:8000 $sourcehost -fN
$ ssh -S none -L 9000:localhost:9000 $desthost -fN

# tunnel ports from source to main host
$ ssh -S none -L 8989:localhost:8989 $mainhost -fN
$ ssh -S none -L 9000:localhost:9000 $desthost -fN

Starting main agent

# login to mainhost and cd to transfer2go area
# we do not need to register our main agent with anyone
$ bash supervisor.sh ./transfer2go -config test/config/main.json -auth=false

Starting destination agent

# login to desthost and cd to transfer2go area
# here we register our agent with main one accessed at http://localhost:8989
$ bash supervisor.sh ./transfer2go -config test/config/destination.json -auth=false -agent http://localhost:8989

Start source agent

# login to sourcehost and cd to transfer2go area
# here we register our agent with main one accessed at http://localhost:8989
$ bash supervisor.sh ./transfer2go -config test/config/source.json -auth=false -agent http://localhost:8989

Test agents

You can check all agents status using simple curl command:

curl http://localhost:8000
Default page: 2017-09-02 01:28:47.345237749 +0200 CEST
agents: map[Test:http://localhost:8989 destination:http://localhost:9000 source:http://localhost:8000]

Here we sent query to source agent (http://localhost:8000) and it replied with list of known agents: Test (the main one), source and destination.

Register data in source agent

To register data in source agent we can use any file or you make create one by yourself. Then you need to add file information into records.json file. Here is an example of such file

[
 {"lfn":"file.root", "pfn":"/real/path/file.root", "block":"/a/b/c#123", "dataset":"/a/b/c"}
]

This file contains list of JSON records describing your dataset/block/file attributes. With this file you may upload your data into source agent (here we use data from test/data/records.json file)

$ ./transfer2go -register=test/data/records.json -agent=http://localhost:8000 -auth=false

The data will be registered in source agent.

Transfer data from source to destination

To transfer the data you need to submit a client request to main agent to initiate the transfer. This can be done as following

# we specify file on source agent as source:file where source is agent alias name and file is an LFN
# for destination we can use either its alias name or its URL
./transfer2go -agent=http://localhost:8989 -src=source:file.root -dst=destination -verbose 1

Similarly, we can transfer entire dataset or block, e.g.

./transfer2go -agent=http://localhost:8989 -src=source:/a/b/c -dst=destination -verbose 1
./transfer2go -agent=http://localhost:8989 -src=source:/a/b/c#123 -dst=destination -verbose 1

This request will be send to main agent for approval. Now we should visit main agent URL and approve our request. This can be done either via web or CLI interfaces.

For web interface please visit main agent URL (if necessary create an SSH tunnel to it) at http://localhost:8989. The pending request will looks like this, see Main Agent Pending page link.

For CLI interface you will issue the following command:

# fetch list of pending requests
curl -s "https://localhost:8989/list?type=pending"
# results will look like this output
[{"ts":0,"file":"file.root","block":"","dataset":"",
  "srcUrl":"http://localhost:8000","srcAlias":"sourceAgent",
  "dstUrl":"http://localhost:9000","dstAlias":"destinationAgent",
  "delay":0,"id":1505303694,"priority":0,"status":"","FailedRecords":null}]

# now you can approve specific request
./transfer2go -agent=http://localhost:8989 -approve=1505303694

Once transfer is approved the appropriate file/data/blocks will be transferred to a destination node.
Please note, at the moment anyone can approve requests in CLI/web interface. Later, the proper authorization will be put in place and will be based on policies, such as certain group of people will have rights to do the approval.

For test 5-6

mainhost=main-agent.cern.ch
sourcehost1=source-server.cern.ch
sourcehost2=uibo-cms-02.cr.cnaf.infn.it
desthost=uibo-cms-03.cr.cnaf.infn.it

Follow the same setup steps.