-
Notifications
You must be signed in to change notification settings - Fork 0
Transfer your case to lumi
The following describes how to transfer data for selected cases stored on atos to the common storage on lumi.
As we have been assigned to two projects on lumi there are two options:
- The Destination Earth project_465000527
- The original project_462000140 if you do not yet have an account for the new project.
The preferred project is the first one, but if you don't yet have an account there use the second one. To make unattended transfers create a second ssh key without a passphrase, upload it to lumi using their web service, and use this one for transfers. Add the following to you .ssh/config file on atos.
Host lumi_transfer
HostName lumi.csc.fi
ForwardX11Trusted yes
ForwardX11 yes
ForwardAgent yes
User YOUR_LUMI_USER
IdentityFile /home/YOUR_ATOS_USER/.ssh/SOME_SSH_KEY
where SOME_SSH_KEY is the name of your new ssh key.
git clone [email protected]:DEODE-NWP/dcmdb.git
cd dcmdb
Edit the cases/case/meta.yaml file (where case is the subdirectory you created, see dcmdb/README.md) and add the path on lumi like
deode_cy46ref:
file_templates : ['fc%Y%m%d%H+%LLLgrib_sfxs','fc%Y%m%d%H+%LLLgrib2_fp']
atos:
path_template : 'ec:/smb/harmonie/deode_cy46ref/%Y/%m/%d/%H/'
lumi_transfer:
path_template : '/scratch/project_462000140/de_33050_common_data/cases/gavle_2021/deode_cy46ref/%Y/%m/%d/%H/'
...
where lumi_transfer is the name of your ssh connection as defined above. Use scratch/PROJECT/de_33050_common_data/cases/ where PROJECT is one of the above mentioned projects. Please remember to commit and push your changes so that it's documented where your data is located on lumi.
To transfer one or more cases we use transfer2lumi.py -c transfer.yml where all required settings are given and described in transfer.yml.
Note that the remote (in the example lumi_transfer) should be the same in .ssh/config, case/meta.yml and in transfer.yml. If everything is correct you should be able to start the transfer for the given run. Data will be downloaded from ecfs and transferred per cycle. If data is already on lumi the data will not be transferred again.
[snh@ac6-102 dcmdb]$ ./transfer2lumi.py -c transfer.yml
Loaded: ['gavle_2021']
Fetch: 2021-07-15 00:00:00
all files already in place for this date
Fetch: 2021-07-15 03:00:00
all files already in place for this date
Fetch: 2021-07-15 06:00:00
ecp ec:/smb/harmonie/deode_cy46ref/2021/07/15/06//fc2021071506+000grib_sfxs /ec/res4/scratch/snh/gavle_2021/deode_cy46ref/2021/07/15/06/
ecp ec:/smb/harmonie/deode_cy46ref/2021/07/15/06//fc2021071506+001grib_sfxs /ec/res4/scratch/snh/gavle_2021/deode_cy46ref/2021/07/15/06/
ecp ec:/smb/harmonie/deode_cy46ref/2021/07/15/06//fc2021071506+002grib_sfxs /ec/res4/scratch/snh/gavle_2021/deode_cy46ref/2021/07/15/06/
ecp ec:/smb/harmonie/deode_cy46ref/2021/07/15/06//fc2021071506+003grib_sfxs /ec/res4/scratch/snh/gavle_2021/deode_cy46ref/2021/07/15/06/
ecp ec:/smb/harmonie/deode_cy46ref/2021/07/15/06//fc2021071506+000grib2_fp /ec/res4/scratch/snh/gavle_2021/deode_cy46ref/2021/07/15/06/
ecp ec:/smb/harmonie/deode_cy46ref/2021/07/15/06//fc2021071506+001grib2_fp /ec/res4/scratch/snh/gavle_2021/deode_cy46ref/2021/07/15/06/
ecp ec:/smb/harmonie/deode_cy46ref/2021/07/15/06//fc2021071506+002grib2_fp /ec/res4/scratch/snh/gavle_2021/deode_cy46ref/2021/07/15/06/
ecp ec:/smb/harmonie/deode_cy46ref/2021/07/15/06//fc2021071506+003grib2_fp /ec/res4/scratch/snh/gavle_2021/deode_cy46ref/2021/07/15/06/
ssh lumi_transfer "mkdir -p /scratch/project_462000140/de_33050_common_data/cases/gavle_2021/deode_cy46ref/2021/07/15/06/"
rsync -vaux /ec/res4/scratch/snh/gavle_2021/deode_cy46ref/2021/07/15/06// lumi_transfer:/scratch/project_462000140/de_33050_common_data/cases/gavle_2021/deode_cy46ref/2021/07/15/06//
sending incremental file list
./
fc2021071506+000grib2_fp
...
To check which data are available on lumi do
./chase.py -host lumi_transfer -l -v
All possible problems are not yet considered. Please report any problems to ulf.andrae_at_smhi.se.