-
-
- Uprade of OSDF to Pelican for Collaborations
+
+ Uprade of the OSDF origin to Pelican for Collaborations
@@ -465,19 +465,25 @@
Pelican
- Uprade of OSDF to Pelican for Collaborations
-The OSG Collab AP had an OSDF door deployed on the access point that provided users with authenticated access to a Ceph storage cluster for shared projects from HTCondor jobs to the OSPool running at remote Execution Points (EPs). Users running jobs on the OSPool access the filesystem either via a client tool (stashcp) or via an HTCondor plugin that is invoked in their submit scripts. The storage is also mounted on the AP at /ospool/uc-shared/project.
-The general purpose documentation found here (https://portal.osg-htc.org/documentation/htc_workloads/managing_data/osdf/) is also applicable for the users on the OSG Collab AP. It describes using the HTCondor plugin to move data to and from the OSDF Origin.
-In a nutshell, if you are not using the client tool then:
+ Uprade of the OSDF origin to Pelican for Collaborations
+The OSG Collab AP had an OSDF door deployed on the access point (ap23.uc.osg-htc.org) that provided users with authenticated access to a Ceph cluster, that provides high capacity storage to shared project directories. HTCondor jobs running in the OSPool at remote Execution Points (EPs) can access the filesystem either via a client tool or via an HTCondor plugin that is invoked in their submit scripts. The storage is also mounted on the AP at /ospool/uc-shared/project.
+On 11/21/2024, OSG/PATh staff migrated the OSDF door from the OSG Collab AP to a separate infrastructure to allow upgrading the origin to the Pelican Platform (https://pelicanplatform.org/) and provide shared project access to users at other APs (ap20.uc.osg-htc.org and ap21.uc.osg-htc.org).
+The migration should have been transparent to the OSG Collab AP. If using the HTCondor plugin, no changes are needed in your submission scripts. For reference, the general purpose documentation found here (https://portal.osg-htc.org/documentation/htc_workloads/managing_data/osdf/) is also applicable to the users of the OSG Collab AP. It describes using the HTCondor plugin to move data to and from the OSDF/Pelican Origin.
+In a nutshell then:
-
-
Use this for your job running on the OSPool to a file from your project directory:
-transfer_input_files = osdf:///ospool/apXX/data//
- 5. Use this for your jobs running on the OSPool to write a file into your project directory:
-OSDF_LOCATION = osdf:///ospool/
-transfer_input_files = $(OSDF_LOCATION)/
+Include the following in your submit script for an OSPool job read a file from your project directory at the EP:
+transfer_input_files = osdf:///ospool/uc-shared/project//
+ 5. Include the folloow in your submit script for an OSPool job to write a file in your project directory from an EP
+OSDF_LOCATION = osdf:///ospool/uc-shared/project//
+transfer_input_files = $(OSDF_LOCATION)/
+The upgrade to the Pelican platform, which uses federated urls for the origin, still keeps the same prefix osdf://. After the upgrade, osdf:// simply points to pelican://osg-htc.org/
+If you are using a client in your runtime script at the EP, then the previous tool, stashcp, will continue to work as long as the version is > 6.12 except recursive access (-r flag). Thefore, we recommend that groups migrate to using the pelican client instead.
+The pelican equivalent of the stashcp client read command: stashcp -d osdf:///ospool/uc-shared/project/<your_project>/<file> is pelican object get -d osdf:///ospool/uc-shared/project/xenon/ppaschos/testfile .
+Similarly, the pelican equivalent of the stashcp write comamnd, stashcp -d <file> osdf:///ospool/uc-shared/project/<your_project/<file> is pelican object put -d <file> osdf:///ospool//uc-shared/project/<your_project/<file>
+If your container image was built using one of the OSG base environments, then the pelican client tool is already included. The pelican client can be installed from the osg repo with dnf install pelican. Refer to this documentation (https://osg-htc.org/docs/common/yum/) to add the osg repository.
diff --git a/search/search_index.json b/search/search_index.json
index baed967..cafde0d 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Mission \u00b6 The mission of OSG Collaboration Support is to help Midsize Collaborations leverage the OSG Consortium services and technologies for their research. A list of services can be found here . Among them, is provisioning access to an OSG-managed Collab Access Point as a single point of submission of dHTC jobs to the Open Science Pool (OSPool). Users from different institutions can join a collaboration specific project. This simplifies access to compute and storage resources as users are not bound by institutional barriers - such as the need for insitutional based identies. For example Institutions A and B need to work on a project. It is not easy for students/postdocs from Institution A to obtain an identity that will allow them to collaborate with their collegues at the Institution B. On the Collab AP, PIs can create a single project with users from both institutions. This allows for a collaborative environment where the research team can launch jobs to the OSPool, levereage multi-TB ephemeral storage and even run computational workflows - like post-processing or mereging jobs on the access point (the login node). Obtain accounts on OSG's Collab Access Point. \u00b6 A PI can visit the accounting portal (https://ci-connect.net) sign-up using CI Logon credentials, create a project and add users. A consultation with someone from Collaboration Support is required. A walk through through the process is linked here . News \u00b6 11/21/2024 The Open Science Data Platorm (OSDF) origin that was deployed on the OSG Collab AP, ap23.uc.osg-htc.org, has moved to a separate infrastructure to accommodate upgrading to the Pelican data distribution platform and allow for users at other APs use the backend storage. For more details refer to this [section](misc/pelican.md). 04/18/2023 A new Collab AP host, ap23.uc.osg-htg.org. has been deployed which will replace the existing one, login.collab.ci-connect.net. Users and projects from login.collab.ci-connect.net have moved to the new one. Users are responsible for migrating their data and make any necessary corrections to their workflows. Collaboration Support staff will be available to assist. Collaboration Institutions and Compute Sites \u00b6 Regular Meeting schedule \u00b6 Collaboration support holds and participates in meetings during the week to assist and coordinate on projects. A list of the weekly meeting schedule is maintained here List of supported collaboratoins \u00b6 Collaboration list Contact Us \u00b6 Open a Ticket by sending an email to help@osg-htc.org and request support for your collaboration project.","title":"OSG Collaboration Support"},{"location":"#mission","text":"The mission of OSG Collaboration Support is to help Midsize Collaborations leverage the OSG Consortium services and technologies for their research. A list of services can be found here . Among them, is provisioning access to an OSG-managed Collab Access Point as a single point of submission of dHTC jobs to the Open Science Pool (OSPool). Users from different institutions can join a collaboration specific project. This simplifies access to compute and storage resources as users are not bound by institutional barriers - such as the need for insitutional based identies. For example Institutions A and B need to work on a project. It is not easy for students/postdocs from Institution A to obtain an identity that will allow them to collaborate with their collegues at the Institution B. On the Collab AP, PIs can create a single project with users from both institutions. This allows for a collaborative environment where the research team can launch jobs to the OSPool, levereage multi-TB ephemeral storage and even run computational workflows - like post-processing or mereging jobs on the access point (the login node).","title":"Mission"},{"location":"#obtain-accounts-on-osgs-collab-access-point","text":"A PI can visit the accounting portal (https://ci-connect.net) sign-up using CI Logon credentials, create a project and add users. A consultation with someone from Collaboration Support is required. A walk through through the process is linked here .","title":"Obtain accounts on OSG's Collab Access Point."},{"location":"#news","text":"11/21/2024 The Open Science Data Platorm (OSDF) origin that was deployed on the OSG Collab AP, ap23.uc.osg-htc.org, has moved to a separate infrastructure to accommodate upgrading to the Pelican data distribution platform and allow for users at other APs use the backend storage. For more details refer to this [section](misc/pelican.md). 04/18/2023 A new Collab AP host, ap23.uc.osg-htg.org. has been deployed which will replace the existing one, login.collab.ci-connect.net. Users and projects from login.collab.ci-connect.net have moved to the new one. Users are responsible for migrating their data and make any necessary corrections to their workflows. Collaboration Support staff will be available to assist.","title":"News"},{"location":"#collaboration-institutions-and-compute-sites","text":"","title":"Collaboration Institutions and Compute Sites"},{"location":"#regular-meeting-schedule","text":"Collaboration support holds and participates in meetings during the week to assist and coordinate on projects. A list of the weekly meeting schedule is maintained here","title":"Regular Meeting schedule"},{"location":"#list-of-supported-collaboratoins","text":"Collaboration list","title":"List of supported collaboratoins"},{"location":"#contact-us","text":"Open a Ticket by sending an email to help@osg-htc.org and request support for your collaboration project.","title":"Contact Us"},{"location":"misc/meeting-schedule/","text":"Schedule of collaborative meetings/calls \u00b6 OSG Area Coordinators meeting \u00b6 When: Wednesdays 2-3pm Central URL: https://unl.zoom.us/j/865025296 IGWN-CESER meetting \u00b6 When: Tuesdays 12:30pm-1:30pm Central Details in the IGWN Collaboration page Icecube-CESER meeting \u00b6 When: Biweeklyy on Mondays 3:00 pm-4:00pm Central Detais in the Icecube Collaboration page XENON Analysis/Compute meeting \u00b6 When: Thursdays 10:00am-11:00am Central Details in the XENON Collaboration page IGWN production meeting \u00b6 When: Thursdays 9:00am-10:00am Central Details in the IGWN Collaboration page Koto support meetting \u00b6 When: Thursdays 3:00pm - 4:00pm Central URL: https://uchicago.zoom.us/s/4250591437 xCache meetting \u00b6 When: Thursdays 1:00pm-2:00pm Central URL: https://unl.zoom.us/j/651969661 Internet2-CESER meeting \u00b6 When: Fridays 2:00pm-3:00pm Central URL: https://internet2.zoom.us/j/668736847","title":"Meeting/Call schedule"},{"location":"misc/meeting-schedule/#schedule-of-collaborative-meetingscalls","text":"","title":"Schedule of collaborative meetings/calls"},{"location":"misc/meeting-schedule/#osg-area-coordinators-meeting","text":"When: Wednesdays 2-3pm Central URL: https://unl.zoom.us/j/865025296","title":"OSG Area Coordinators meeting"},{"location":"misc/meeting-schedule/#igwn-ceser-meetting","text":"When: Tuesdays 12:30pm-1:30pm Central Details in the IGWN Collaboration page","title":"IGWN-CESER meetting"},{"location":"misc/meeting-schedule/#icecube-ceser-meeting","text":"When: Biweeklyy on Mondays 3:00 pm-4:00pm Central Detais in the Icecube Collaboration page","title":"Icecube-CESER meeting"},{"location":"misc/meeting-schedule/#xenon-analysiscompute-meeting","text":"When: Thursdays 10:00am-11:00am Central Details in the XENON Collaboration page","title":"XENON Analysis/Compute meeting"},{"location":"misc/meeting-schedule/#igwn-production-meeting","text":"When: Thursdays 9:00am-10:00am Central Details in the IGWN Collaboration page","title":"IGWN production meeting"},{"location":"misc/meeting-schedule/#koto-support-meetting","text":"When: Thursdays 3:00pm - 4:00pm Central URL: https://uchicago.zoom.us/s/4250591437","title":"Koto support meetting"},{"location":"misc/meeting-schedule/#xcache-meetting","text":"When: Thursdays 1:00pm-2:00pm Central URL: https://unl.zoom.us/j/651969661","title":"xCache meetting"},{"location":"misc/meeting-schedule/#internet2-ceser-meeting","text":"When: Fridays 2:00pm-3:00pm Central URL: https://internet2.zoom.us/j/668736847","title":"Internet2-CESER meeting"},{"location":"misc/onboard/","text":"This is a test to build up","title":"Access to Collab AP"},{"location":"misc/pelican/","text":"Uprade of OSDF to Pelican for Collaborations \u00b6 The OSG Collab AP had an OSDF door deployed on the access point that provided users with authenticated access to a Ceph storage cluster for shared projects from HTCondor jobs to the OSPool running at remote Execution Points (EPs). Users running jobs on the OSPool access the filesystem either via a client tool (stashcp) or via an HTCondor plugin that is invoked in their submit scripts. The storage is also mounted on the AP at /ospool/uc-shared/project. The general purpose documentation found here (https://portal.osg-htc.org/documentation/htc_workloads/managing_data/osdf/) is also applicable for the users on the OSG Collab AP. It describes using the HTCondor plugin to move data to and from the OSDF Origin. In a nutshell, if you are not using the client tool then: Use this for your job running on the OSPool to a file from your project directory: transfer_input_files = osdf:///ospool/apXX/data/ / 5. Use this for your jobs running on the OSPool to write a file into your project directory: OSDF_LOCATION = osdf:///ospool/ transfer_input_files = $(OSDF_LOCATION)/","title":"Pelican"},{"location":"misc/pelican/#uprade-of-osdf-to-pelican-for-collaborations","text":"The OSG Collab AP had an OSDF door deployed on the access point that provided users with authenticated access to a Ceph storage cluster for shared projects from HTCondor jobs to the OSPool running at remote Execution Points (EPs). Users running jobs on the OSPool access the filesystem either via a client tool (stashcp) or via an HTCondor plugin that is invoked in their submit scripts. The storage is also mounted on the AP at /ospool/uc-shared/project. The general purpose documentation found here (https://portal.osg-htc.org/documentation/htc_workloads/managing_data/osdf/) is also applicable for the users on the OSG Collab AP. It describes using the HTCondor plugin to move data to and from the OSDF Origin. In a nutshell, if you are not using the client tool then: Use this for your job running on the OSPool to a file from your project directory: transfer_input_files = osdf:///ospool/apXX/data/ / 5. Use this for your jobs running on the OSPool to write a file into your project directory: OSDF_LOCATION = osdf:///ospool/ transfer_input_files = $(OSDF_LOCATION)/","title":"Uprade of OSDF to Pelican for Collaborations"},{"location":"misc/services/","text":"Collab Access Point (AP) \u00b6 Onboarding to the OSG managed and Collaboration dedicated Access Point. Users will be trained in job submission and software development appropriate for the dHTC jobs. In addition, the project will be supported with visible performance metrics coming from the central OSG Accounting dashboard Gracc . Collaboration Support (CS) staff will provide consultation services on how to best optimize the workflow, help with debugging any issues encountered and facilitate coordination with other institutional partners that require access to produded data products. Data management and storage \u00b6 All projects will be provisioned with distributed storage on the Open Science Data Federation (OSDF) and local storage on the AP. Collaboration support will work with the collaboration in managing the storage within the limits agreed upon during the onboarding process. Staff will train users on how to best use data management tools to monitor, track and distribute data to jobs or other institutional storage locations. Collaboration support staff will also work with the PI in understanding the long-term needs of the project and architect solutions where the collaboration can integrate with the OSDF federation. These include assistance in setting up Origins of data which will distribute them to the network of Caches which are placed in close proximity to execution points. CS staff, in coordination with other OSG teams, will also assist in provisioning services of institutional managed caches in proximity to locations where compute cycles are available or deploy Hosted CEs on behalf on the collaboration. Additional tools, such as Rucio, are also available to provide a service for distributing data without the need for an Origin. Data in this case is hosted in part at each member institution with Rucio providing a centralized management service of where the data are and where they need to be replicated - either for use in dHTC jobs or local batch systems. Follow this link to look at the Kibana dashboard tracking FTS transfers for Icecube, LIGO and XENON and Rucio information on XENON. Integration of Collaboration owned resources \u00b6 This process results in a pool of resources where a job submission from an AP, e.g. the Collab AP, would be run with priority because the collaboration owns the available compute cycle. This is accomplished by deploying Compute Entry (CE) points on the edge infrastructure of institutions. A project is affiliated with an entity named Virtual Organization, which is used to allow factory operators to directly send the project's jobs to the dedicated compute resources behind the CE. An international collaboration can then achieve a global computing scale if CEs are deployed nearby member institutions, all running jobs on their local clusters for the collaboration. Those clusters can either be wholly owned by the collaboration or the latter might have a compute allocation there. In the end, the project can leverage both the Open Science Pool and it's own pool to maximize throughput.","title":"Services"},{"location":"misc/services/#collab-access-point-ap","text":"Onboarding to the OSG managed and Collaboration dedicated Access Point. Users will be trained in job submission and software development appropriate for the dHTC jobs. In addition, the project will be supported with visible performance metrics coming from the central OSG Accounting dashboard Gracc . Collaboration Support (CS) staff will provide consultation services on how to best optimize the workflow, help with debugging any issues encountered and facilitate coordination with other institutional partners that require access to produded data products.","title":"Collab Access Point (AP)"},{"location":"misc/services/#data-management-and-storage","text":"All projects will be provisioned with distributed storage on the Open Science Data Federation (OSDF) and local storage on the AP. Collaboration support will work with the collaboration in managing the storage within the limits agreed upon during the onboarding process. Staff will train users on how to best use data management tools to monitor, track and distribute data to jobs or other institutional storage locations. Collaboration support staff will also work with the PI in understanding the long-term needs of the project and architect solutions where the collaboration can integrate with the OSDF federation. These include assistance in setting up Origins of data which will distribute them to the network of Caches which are placed in close proximity to execution points. CS staff, in coordination with other OSG teams, will also assist in provisioning services of institutional managed caches in proximity to locations where compute cycles are available or deploy Hosted CEs on behalf on the collaboration. Additional tools, such as Rucio, are also available to provide a service for distributing data without the need for an Origin. Data in this case is hosted in part at each member institution with Rucio providing a centralized management service of where the data are and where they need to be replicated - either for use in dHTC jobs or local batch systems. Follow this link to look at the Kibana dashboard tracking FTS transfers for Icecube, LIGO and XENON and Rucio information on XENON.","title":"Data management and storage"},{"location":"misc/services/#integration-of-collaboration-owned-resources","text":"This process results in a pool of resources where a job submission from an AP, e.g. the Collab AP, would be run with priority because the collaboration owns the available compute cycle. This is accomplished by deploying Compute Entry (CE) points on the edge infrastructure of institutions. A project is affiliated with an entity named Virtual Organization, which is used to allow factory operators to directly send the project's jobs to the dedicated compute resources behind the CE. An international collaboration can then achieve a global computing scale if CEs are deployed nearby member institutions, all running jobs on their local clusters for the collaboration. Those clusters can either be wholly owned by the collaboration or the latter might have a compute allocation there. In the end, the project can leverage both the Open Science Pool and it's own pool to maximize throughput.","title":"Integration of Collaboration owned resources"},{"location":"projects/Redtop/","text":"REDTOP \u00b6 Support areas of REDTOP include data management, provisioned storage on Stash (~100 TB), and assistance in executing their job workloads. Computing Model for REDTOP using OSG resources: https://redtop.fnal.gov/computing-model/ OSG Collaboration Support Services supports REDTOP workflows on the OpenScienceGrid and has provided a compute plan in support of the colllaboration's LOI to Snowmass21","title":"REDTOP"},{"location":"projects/Redtop/#redtop","text":"Support areas of REDTOP include data management, provisioned storage on Stash (~100 TB), and assistance in executing their job workloads. Computing Model for REDTOP using OSG resources: https://redtop.fnal.gov/computing-model/ OSG Collaboration Support Services supports REDTOP workflows on the OpenScienceGrid and has provided a compute plan in support of the colllaboration's LOI to Snowmass21","title":"REDTOP"},{"location":"projects/clas12/","text":"CLAS12 \u00b6 The CLAS12 experiment at Jefferson Lab is supported in part by the U.S. Department of Energy, the National Science Foundation (NSF), the Italian Istituto Nazionale di Fisica Nucleare (INFN), the French Centre National de la Recherche Scientifique (CNRS), the French Commissariat pour l\u2019Energie Atomique, the UK Science and Technology Facilities Council and other international funding CLAS12 data and production management: Maurizio Ungaro ungaro@jlab.org","title":"GLAS12"},{"location":"projects/clas12/#clas12","text":"The CLAS12 experiment at Jefferson Lab is supported in part by the U.S. Department of Energy, the National Science Foundation (NSF), the Italian Istituto Nazionale di Fisica Nucleare (INFN), the French Centre National de la Recherche Scientifique (CNRS), the French Commissariat pour l\u2019Energie Atomique, the UK Science and Technology Facilities Council and other international funding CLAS12 data and production management: Maurizio Ungaro ungaro@jlab.org","title":"CLAS12"},{"location":"projects/eic/","text":"EIC \u00b6 EIC (Electron-Ion Collider) is a proposed High Energy Physics experiment, a joint effort between Brookhaven and Jefferson national labs. Overall project management lead: Jerome Lauret eromel@bnl.gov BNL project coordinator: Xin Zhao xzhao@bnl.gov JLAB project coordinators: Markus Diefenthaler, mdiefent@jlab.org and David Lawrence, davidl@jlab.org","title":"EIC"},{"location":"projects/eic/#eic","text":"EIC (Electron-Ion Collider) is a proposed High Energy Physics experiment, a joint effort between Brookhaven and Jefferson national labs. Overall project management lead: Jerome Lauret eromel@bnl.gov BNL project coordinator: Xin Zhao xzhao@bnl.gov JLAB project coordinators: Markus Diefenthaler, mdiefent@jlab.org and David Lawrence, davidl@jlab.org","title":"EIC"},{"location":"projects/gluex/","text":"GLUEX \u00b6 The GlueX experiment at Jefferson Lab is supported in part by the U.S. Department of Energy, the U.S. National Science Foundation, the German Research Foundation, and other international foundations and institutions. The UConn group acknowledges support for work on GlueX from the U.S. National Science Foundation under grant 1812415. GlueX data and production management: Richard Jones richard.t.jones@uconn.edu JLAB GlueX coordinators: Thomas Britton tbritton@jlab.org and Bryan Hess bhess@jlab.org","title":"GLUEX"},{"location":"projects/gluex/#gluex","text":"The GlueX experiment at Jefferson Lab is supported in part by the U.S. Department of Energy, the U.S. National Science Foundation, the German Research Foundation, and other international foundations and institutions. The UConn group acknowledges support for work on GlueX from the U.S. National Science Foundation under grant 1812415. GlueX data and production management: Richard Jones richard.t.jones@uconn.edu JLAB GlueX coordinators: Thomas Britton tbritton@jlab.org and Bryan Hess bhess@jlab.org","title":"GLUEX"},{"location":"projects/icecube/","text":"IceCube-PATh \u00b6 Icecube is supported by OSG Collaborations via the PATh project. This page will include meeting-minutes and location along with status reports","title":"IceCube"},{"location":"projects/icecube/#icecube-path","text":"Icecube is supported by OSG Collaborations via the PATh project. This page will include meeting-minutes and location along with status reports","title":"IceCube-PATh"},{"location":"projects/igwn/","text":"International Gravitational-Wave Observatory Network \u00b6 Support areas for IGWN include data management, infrastructure (StashCaches, FTS service for Rucio managed third-party transfers), best practices for software and job management, and in-person participation in the Open Science - LIGO weekly meetings. The LIGO component of the IGWN collaboration has 119 institutions and universities in the US that make use of the Open Science distributed infrastructure for computing. LIGO is supported by the NSF award # PHY-2110594 for the LIGO Data Grid (LDG) and NSF award # PHY-1764464 for the LIGO Lab in computing, R&D, and operations. A detailed list of technical contacts and leads for the various workloads is maintained by the LIGO and Open Science group here. For general purpose data and job management, the lead is: James Clark james.clark@ligo.org Institutions Compute Resources Weekly IGWN-PATh meeting \u00b6 Topic: PATh-IGWN technical call Time: 11:00 PM Central Time (US and Canada) Every week on Thur Join Zoom Meeting: TB-updated Weekly IGWN Production meeting \u00b6 Time: 9:00 AM Central Time (US and Canada) Every Week on Thu Links to be updated. Currently access is only restricted to people with ligo credentials. Weekly IGWN Condor meeting \u00b6 TB updated","title":"IGWN"},{"location":"projects/igwn/#international-gravitational-wave-observatory-network","text":"Support areas for IGWN include data management, infrastructure (StashCaches, FTS service for Rucio managed third-party transfers), best practices for software and job management, and in-person participation in the Open Science - LIGO weekly meetings. The LIGO component of the IGWN collaboration has 119 institutions and universities in the US that make use of the Open Science distributed infrastructure for computing. LIGO is supported by the NSF award # PHY-2110594 for the LIGO Data Grid (LDG) and NSF award # PHY-1764464 for the LIGO Lab in computing, R&D, and operations. A detailed list of technical contacts and leads for the various workloads is maintained by the LIGO and Open Science group here. For general purpose data and job management, the lead is: James Clark james.clark@ligo.org Institutions Compute Resources","title":"International Gravitational-Wave Observatory Network"},{"location":"projects/igwn/#weekly-igwn-path-meeting","text":"Topic: PATh-IGWN technical call Time: 11:00 PM Central Time (US and Canada) Every week on Thur Join Zoom Meeting: TB-updated","title":"Weekly IGWN-PATh meeting"},{"location":"projects/igwn/#weekly-igwn-production-meeting","text":"Time: 9:00 AM Central Time (US and Canada) Every Week on Thu Links to be updated. Currently access is only restricted to people with ligo credentials.","title":"Weekly IGWN Production meeting"},{"location":"projects/igwn/#weekly-igwn-condor-meeting","text":"TB updated","title":"Weekly IGWN Condor meeting"},{"location":"projects/koto/","text":"KOTO \u00b6 KOTO is an international collaboration of an experiment based at the JPARC HEP facility in Japan. Access Point \u00b6 KOTO submits out of the Connect node: login.collab.ci-connect.net. To request an account, visit the portal at https://ci-connect.net and ask to join the PATh Collaborations group. KOTO is a subgroup under PATh collaborations and a separate request needs to be made to join by clicking the request memberhip button. Storage access \u00b6 Besides their home directories on login.collab.ci-connect.net, users have access to local storage in /scratch/ and distributed storage in /collab/user/ . In addition, the collaboration has access to /collab/project/KOTO for sharing data products between the users. The distributed storage is http accessible: http://stash.osgstorage.org/collab/KOTO and data can be downloaded using a browser or command line utilities like wget or curl. Jobs to the OSPool \u00b6 Inlined below is a submit script to run a KOTO job on the OSPool 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 #!/bin/bash Universe = Vanilla Executable = run_koto.sh Requirements = HAS_SINGULARITY == TRUE +SingularityImage = \"/cvmfs/singularity.opensciencegrid.org/ppaschos/koto-dev:latest\" transfer_input_files = e14_201605.mac Error = output.err. $( Cluster ) - $( Process ) Output = output.out. $( Cluster ) - $( Process ) Log = output.log. $( Cluster ) should_transfer_files = YES WhenToTransferOutput = ON_EXIT request_cpus = 1 request_memory = 2GB request_disk = 2GB +ProjectName = \"collab.KOTO\" Queue 1 KOTO uses a software stack that is included in the Singularity image - shown in the script above. The image is deployed on the remote execution nodes and jobs run within the Singularity container. The input file, e14_201605.mac, is sent to the execution node as part of the job. The execution script, run_koto.sh, sets the environment and includes all invocations to executables. The following script, run_koto.sh, runs a benchmark example from the E14 library. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 #!/bin/bash echo $HOSTNAME source /opt/koto/root/v6.22.02/bin/thisroot.sh source /opt/koto/geant4/10.05.p01/bin/geant4.sh echo \"PATH --> \" $PATH echo \"LD_LIBRARY_PATH --> \" $LD_LIBRARY_PATH echo \"ROOTSYS --> \" $ROOTSYS echo \"G4 data environent\" env | grep G4 export E14_TOP_DIR = \"/opt/koto/e14/201605v6.2/e14\" echo \"E14_TOP_DIR=\" $E14_TOP_DIR source /opt/koto/e14/201605v6.2/setup.sh output = \"output.root\" nevent = 1000 seed = 0 echo \"Running Code\" /opt/koto/e14/201605v6.2/e14/examples/gsim4test/bin/gsim4test e14_201605.mac ${ output } ${ nevent } ${ seed } ${ seed } Pegasus Workflows \u00b6 The Pegasus project encompasses a set of technologies that help workflow-based applications execute in a number of different environments including desktops, campus clusters, grids, and clouds. Pegasus bridges the scientific domain and the execution environment by automatically mapping high-level workflow descriptions onto distributed resources. It automatically locates the necessary input data and computational resources necessary for workflow execution. Pegasus enables scientists to construct workflows in abstract terms without worrying about the details of the underlying execution environment or the particulars of the low-level specifications required by the middleware. Some of the advantages of using Pegasus include: Portability / Reuse - User created workflows can easily be run in different environments without alteration. Pegasus currently runs workflows on compute systems scheduled via HTCondor, including the OSPool, and other other systems or via other schedulers (e.g. ACCESS resources, Amazon EC2, Google Cloud, and many campus clusters). The same workflow can run on a single system or across a heterogeneous set of resources. Performance - The Pegasus mapper can reorder, group, and prioritize tasks in order to increase the overall workflow performance. Scalability - Pegasus can easily scale both the size of the workflow, and the resources that the workflow is distributed over. Pegasus runs workflows ranging from just a few computational tasks up to 1 million tasks. The number of resources involved in executing a workflow can scale as needed without any impediments to performance. Provenance - By default, all jobs in Pegasus are launched via the kickstart process that captures runtime provenance of the job and helps in debugging. The provenance data is collected in a database, and the data can be summarized with tools such as pegasus-statistics or directly with SQL queries. Data Management - Pegasus handles replica selection, data transfers and output registrations in data catalogs. These tasks are added to a workflow as auxiliary jobs by the Pegasus planner. Reliability - Jobs and data transfers are automatically retried in case of failures. Debugging tools such as pegasus-analyzer help the user to debug the workflow in case of non-recoverable failures. Error Recovery - When errors occur, Pegasus tries to recover when possible by retrying tasks, retrying the entire workflow, providing workflow-level checkpointing, re-mapping portions of the workflow, trying alternative data sources for staging data, and, when all else fails, providing a rescue workflow containing a description of only the work that remains to be done. Pegasus keeps track of what has been done (provenance) including the locations of data used and produced, and which software was used with which parameters. OSG has no read/write enabled shared file system across the resources. Jobs are required to either bring inputs along with the job, or as part of the job stage the inputs from a remote location. The following examples highlight how Pegasus can be used to manage KOTO workloads in such an environment by providing an abstraction layer around things like data movements and job retries, enabling the users to run larger workloads, spending less time developing job management tools and babysitting their computations. Pegasus workflows have 4 components: Site Catalog - Describes the execution environment in which the workflow will be executed. Transformation Catalog - Specifies locations of the executables used by the workflow. Replica Catalog - Specifies locations of the input data to the workflow. Workflow Description - An abstract workflow description containing compute steps and dependencies between the steps. We refer to this workflow as abstract because it does not contain data locations and available software. When developing a Pegasus Workflow using the Python API , all four components may be defined in the same file. For details, please refer to the Pegasus documentation . KOTO workflow example \u00b6 The GitHub repo https://github.com/pegasus-isi/koto-pegasus contains a sample Pegasus workflow for KOTO processing. It is a very simple Monte Carlo example, containing just N independent jobs: This example is using OSG StashCache for data transfers. Credentials are transparant to the end users. Additionally, this example uses a custom container to run jobs. This ensures a consistent and complete environment for the jobs. Which container to use is defined in workflow.py with the +SingularityImage attribute. When invoked, the workflow script ( workflow.py ) does the following: Writes the file pegasus.properties . This file contains configuration settings used by Pegasus and HTCondor. # --- Properties --------------------------------------------------------------- props = Properties() props[\"pegasus.data.configuration\"] = \"nonsharedfs\" # Provide a full kickstart record, including the environment, even for successful jobs props[\"pegasus.gridstart.arguments\"] = \"-f\" # Limit the number of idle jobs for large workflows props[\"dagman.maxidle\"] = \"1000\" # use timestamps for all our dirs - this make sure we have unique names in stash props[\"pegasus.dir.useTimestamp\"] = \"true\" # seperate output dir for each workflow run props[\"pegasus.dir.storage.deep\"] = \"true\" # Help Pegasus developers by sharing performance data (optional) props[\"pegasus.monitord.encoding\"] = \"json\" props[\"pegasus.catalog.workflow.amqp.url\"] = \"amqp://friend:donatedata@msgs.pegasus.isi.edu:5672/prod/workflows\" props[\"pegasus.metrics.app\"] = \"KOTO\" Writes the file sites.yml . This file describes the execution environment in which the workflow will be run. # --- Sites -------------------------------------------------------------------- sc = SiteCatalog() # local site (submit machine) local_site = Site(name=\"local\") local_shared_scratch = Directory(directory_type=Directory.SHARED_SCRATCH, path=WORK_DIR / \"scratch\") local_shared_scratch.add_file_servers(FileServer(url=\"file://\" + str(WORK_DIR / \"scratch\"), operation_type=Operation.ALL)) local_site.add_directories(local_shared_scratch) local_storage = Directory(directory_type=Directory.LOCAL_STORAGE, path=WORK_DIR / \"outputs\") local_storage.add_file_servers(FileServer(url=\"file://\" + str(WORK_DIR / \"outputs\"), operation_type=Operation.ALL)) local_site.add_directories(local_storage) local_site.add_env(PATH=os.environ[\"PATH\"]) sc.add_sites(local_site) # stash site (staging site, where intermediate data will be stored) stash_site = Site(name=\"stash\", arch=Arch.X86_64, os_type=OS.LINUX) stash_staging_path = \"/collab/user/{USER}/staging\".format(USER=getpass.getuser()) stash_shared_scratch = Directory(directory_type=Directory.SHARED_SCRATCH, path=stash_staging_path) stash_shared_scratch.add_file_servers( FileServer( url=\"stash:///osgconnect{STASH_STAGING_PATH}\".format(STASH_STAGING_PATH=stash_staging_path), operation_type=Operation.ALL) ) stash_site.add_directories(stash_shared_scratch) sc.add_sites(stash_site) # condorpool (execution site) condorpool_site = Site(name=\"condorpool\", arch=Arch.X86_64, os_type=OS.LINUX) condorpool_site.add_pegasus_profile(style=\"condor\") condorpool_site.add_condor_profile( universe=\"vanilla\", requirements=\"HAS_SINGULARITY == True && HAS_AVX2 == True\", request_cpus=1, request_memory=\"4 GB\", request_disk=\"10 GB\", ) condorpool_site.add_profiles( Namespace.CONDOR, key=\"+SingularityImage\", value='\"/cvmfs/singularity.opensciencegrid.org/chiehlin0212/koto-dev:latest\"' ) condorpool_site.add_profiles( Namespace.CONDOR, key=\"+ProjectName\", value='\"collab.KOTO\"' ) sc.add_sites(condorpool_site) # write SiteCatalog to ./sites.yml sc.write() Writes the file transformations.yml . This file specifies the executables used in the workflow and contains the locations where they are physically located. In this example, we only have run_koto.sh . # --- Transformations ---------------------------------------------------------- run_koto = Transformation( name=\"run_koto\", site=\"local\", pfn=TOP_DIR / \"bin/run_koto.sh\", is_stageable=True ).add_pegasus_profile(clusters_size=1) tc = TransformationCatalog() tc.add_transformations(run_koto) # write TransformationCatalog to ./transformations.yml tc.write() Writes the file replicas.yml . This file specifies the physical locations of any input files used by the workflow. In this example, there is an entry for each file in the inputs/ directory, and all files matching /collab/project/KOTO/external/run87/accidental/64kW/**/*.root # --- Replicas ----------------------------------------------------------------- rc = ReplicaCatalog() input_files = [] # all files in the inputs/ dir input_files = glob.glob(str(TOP_DIR / \"inputs/*\")) for f in input_files: lfn = os.path.basename(f) rc.add_replica(site=\"local\", lfn=lfn, pfn=f) # also add all wave files wave_files = glob.glob(\"/collab/project/KOTO/external/run87/accidental/64kW/**/*.root\", recursive=True) for f in wave_files: lfn = os.path.basename(f) rc.add_replica(site=\"local\", lfn=lfn, pfn=f) # write ReplicaCatalog to replicas.yml rc.write() Builds the wordfreq workflow and submits it for execution. When wf.plan is invoked, pegasus.properties , sites.yml , transformations.yml , and replicas.yml will be consumed as part of the workflow planning process. Note that in this step there is no mention of data movement and job details as these are added by Pegasus when the workflow is planned into an executable workflow. As part of the planning process, additional jobs which handle scratch directory creation, data staging, and data cleanup are added to the workflow. # --- Workflow ----------------------------------------------------------------- wf = Workflow(name=\"koto\") random.seed() for n in range(10): # arguments decay_mode = \"KL3pi0\" seed = str(random.randint(0, 10000)) wave = os.path.basename(random.choice(wave_files)) # TODO: check if we already have a job with the seed/wave combination print(\" ... added mc job with wave={} / seed={}\".format(wave, seed)) out_root = File(\"{}_BasicVetoAndCuts_Accidental_{}.root\".format(decay_mode, seed)) job = Job(run_koto)\\ .add_args(decay_mode, seed, wave)\\ .add_outputs(out_root) # inputs, only use the lfn for inp in input_files + [wave]: job.add_inputs(os.path.basename(inp)) wf.add_jobs(job) # plan and run the workflow wf.plan( dir=WORK_DIR / \"runs\", sites=[\"condorpool\"], staging_sites={\"condorpool\": \"stash\"}, output_sites=[\"local\"], cluster=[\"horizontal\"], submit=True ) Submit the workflow by executing workflow.py . $ ./workflow.py Note that when Pegasus plans/submits a workflow, a workflow directory is created and presented in the output. In the following example output, the workflow directory is /home/user/workflows/runs/pegasus/run0014 . 2020.12.18 14:33:07.059 CST: ----------------------------------------------------------------------- 2020.12.18 14:33:07.064 CST: File for submitting this DAG to HTCondor : koto-workflow-0.dag.condor.sub 2020.12.18 14:33:07.070 CST: Log of DAGMan debugging messages : koto-workflow-0.dag.dagman.out 2020.12.18 14:33:07.075 CST: Log of HTCondor library output : koto-workflow-0.dag.lib.out 2020.12.18 14:33:07.080 CST: Log of HTCondor library error messages : koto-workflow-0.dag.lib.err 2020.12.18 14:33:07.086 CST: Log of the life of condor_dagman itself : koto-workflow-0.dag.dagman.log 2020.12.18 14:33:07.091 CST: 2020.12.18 14:33:07.096 CST: -no_submit given, not submitting DAG to HTCondor. You can do this with: 2020.12.18 14:33:07.107 CST: ----------------------------------------------------------------------- 2020.12.18 14:33:11.352 CST: Your database is compatible with Pegasus version: 5.1.0dev [WARNING] Submitting to condor koto-workflow-0.dag.condor.sub 2020.12.18 14:33:12.060 CST: Time taken to execute is 5.818 seconds Your workflow has been started and is running in the base directory: /home/user/workflows/runs/koto-workflow/run0014 *** To monitor the workflow you can run *** pegasus-status -l /home/user/workflows/runs/koto-workflow/run0014 *** To remove your workflow run *** pegasus-remove /home/user/workflows/runs/koto-workflow/run0014 This directory is the handle to the workflow instance and is used by Pegasus command line tools. Some useful tools to know about: pegasus-status -v [wfdir] Provides status on a currently running workflow. ( more ) pegasus-analyzer [wfdir] Provides debugging clues why a workflow failed. Run this after a workflow has failed. ( more ) pegasus-statistics [wfdir] Provides statistics, such as walltimes, on a workflow after it has completed. ( more ) pegasus-remove [wfdir] Removes a workflow from the system. ( more )","title":"KOTO"},{"location":"projects/koto/#koto","text":"KOTO is an international collaboration of an experiment based at the JPARC HEP facility in Japan.","title":"KOTO"},{"location":"projects/koto/#access-point","text":"KOTO submits out of the Connect node: login.collab.ci-connect.net. To request an account, visit the portal at https://ci-connect.net and ask to join the PATh Collaborations group. KOTO is a subgroup under PATh collaborations and a separate request needs to be made to join by clicking the request memberhip button.","title":"Access Point"},{"location":"projects/koto/#storage-access","text":"Besides their home directories on login.collab.ci-connect.net, users have access to local storage in /scratch/ and distributed storage in /collab/user/ . In addition, the collaboration has access to /collab/project/KOTO for sharing data products between the users. The distributed storage is http accessible: http://stash.osgstorage.org/collab/KOTO and data can be downloaded using a browser or command line utilities like wget or curl.","title":"Storage access"},{"location":"projects/koto/#jobs-to-the-ospool","text":"Inlined below is a submit script to run a KOTO job on the OSPool 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 #!/bin/bash Universe = Vanilla Executable = run_koto.sh Requirements = HAS_SINGULARITY == TRUE +SingularityImage = \"/cvmfs/singularity.opensciencegrid.org/ppaschos/koto-dev:latest\" transfer_input_files = e14_201605.mac Error = output.err. $( Cluster ) - $( Process ) Output = output.out. $( Cluster ) - $( Process ) Log = output.log. $( Cluster ) should_transfer_files = YES WhenToTransferOutput = ON_EXIT request_cpus = 1 request_memory = 2GB request_disk = 2GB +ProjectName = \"collab.KOTO\" Queue 1 KOTO uses a software stack that is included in the Singularity image - shown in the script above. The image is deployed on the remote execution nodes and jobs run within the Singularity container. The input file, e14_201605.mac, is sent to the execution node as part of the job. The execution script, run_koto.sh, sets the environment and includes all invocations to executables. The following script, run_koto.sh, runs a benchmark example from the E14 library. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 #!/bin/bash echo $HOSTNAME source /opt/koto/root/v6.22.02/bin/thisroot.sh source /opt/koto/geant4/10.05.p01/bin/geant4.sh echo \"PATH --> \" $PATH echo \"LD_LIBRARY_PATH --> \" $LD_LIBRARY_PATH echo \"ROOTSYS --> \" $ROOTSYS echo \"G4 data environent\" env | grep G4 export E14_TOP_DIR = \"/opt/koto/e14/201605v6.2/e14\" echo \"E14_TOP_DIR=\" $E14_TOP_DIR source /opt/koto/e14/201605v6.2/setup.sh output = \"output.root\" nevent = 1000 seed = 0 echo \"Running Code\" /opt/koto/e14/201605v6.2/e14/examples/gsim4test/bin/gsim4test e14_201605.mac ${ output } ${ nevent } ${ seed } ${ seed }","title":"Jobs to the OSPool"},{"location":"projects/koto/#pegasus-workflows","text":"The Pegasus project encompasses a set of technologies that help workflow-based applications execute in a number of different environments including desktops, campus clusters, grids, and clouds. Pegasus bridges the scientific domain and the execution environment by automatically mapping high-level workflow descriptions onto distributed resources. It automatically locates the necessary input data and computational resources necessary for workflow execution. Pegasus enables scientists to construct workflows in abstract terms without worrying about the details of the underlying execution environment or the particulars of the low-level specifications required by the middleware. Some of the advantages of using Pegasus include: Portability / Reuse - User created workflows can easily be run in different environments without alteration. Pegasus currently runs workflows on compute systems scheduled via HTCondor, including the OSPool, and other other systems or via other schedulers (e.g. ACCESS resources, Amazon EC2, Google Cloud, and many campus clusters). The same workflow can run on a single system or across a heterogeneous set of resources. Performance - The Pegasus mapper can reorder, group, and prioritize tasks in order to increase the overall workflow performance. Scalability - Pegasus can easily scale both the size of the workflow, and the resources that the workflow is distributed over. Pegasus runs workflows ranging from just a few computational tasks up to 1 million tasks. The number of resources involved in executing a workflow can scale as needed without any impediments to performance. Provenance - By default, all jobs in Pegasus are launched via the kickstart process that captures runtime provenance of the job and helps in debugging. The provenance data is collected in a database, and the data can be summarized with tools such as pegasus-statistics or directly with SQL queries. Data Management - Pegasus handles replica selection, data transfers and output registrations in data catalogs. These tasks are added to a workflow as auxiliary jobs by the Pegasus planner. Reliability - Jobs and data transfers are automatically retried in case of failures. Debugging tools such as pegasus-analyzer help the user to debug the workflow in case of non-recoverable failures. Error Recovery - When errors occur, Pegasus tries to recover when possible by retrying tasks, retrying the entire workflow, providing workflow-level checkpointing, re-mapping portions of the workflow, trying alternative data sources for staging data, and, when all else fails, providing a rescue workflow containing a description of only the work that remains to be done. Pegasus keeps track of what has been done (provenance) including the locations of data used and produced, and which software was used with which parameters. OSG has no read/write enabled shared file system across the resources. Jobs are required to either bring inputs along with the job, or as part of the job stage the inputs from a remote location. The following examples highlight how Pegasus can be used to manage KOTO workloads in such an environment by providing an abstraction layer around things like data movements and job retries, enabling the users to run larger workloads, spending less time developing job management tools and babysitting their computations. Pegasus workflows have 4 components: Site Catalog - Describes the execution environment in which the workflow will be executed. Transformation Catalog - Specifies locations of the executables used by the workflow. Replica Catalog - Specifies locations of the input data to the workflow. Workflow Description - An abstract workflow description containing compute steps and dependencies between the steps. We refer to this workflow as abstract because it does not contain data locations and available software. When developing a Pegasus Workflow using the Python API , all four components may be defined in the same file. For details, please refer to the Pegasus documentation .","title":"Pegasus Workflows"},{"location":"projects/koto/#koto-workflow-example","text":"The GitHub repo https://github.com/pegasus-isi/koto-pegasus contains a sample Pegasus workflow for KOTO processing. It is a very simple Monte Carlo example, containing just N independent jobs: This example is using OSG StashCache for data transfers. Credentials are transparant to the end users. Additionally, this example uses a custom container to run jobs. This ensures a consistent and complete environment for the jobs. Which container to use is defined in workflow.py with the +SingularityImage attribute. When invoked, the workflow script ( workflow.py ) does the following: Writes the file pegasus.properties . This file contains configuration settings used by Pegasus and HTCondor. # --- Properties --------------------------------------------------------------- props = Properties() props[\"pegasus.data.configuration\"] = \"nonsharedfs\" # Provide a full kickstart record, including the environment, even for successful jobs props[\"pegasus.gridstart.arguments\"] = \"-f\" # Limit the number of idle jobs for large workflows props[\"dagman.maxidle\"] = \"1000\" # use timestamps for all our dirs - this make sure we have unique names in stash props[\"pegasus.dir.useTimestamp\"] = \"true\" # seperate output dir for each workflow run props[\"pegasus.dir.storage.deep\"] = \"true\" # Help Pegasus developers by sharing performance data (optional) props[\"pegasus.monitord.encoding\"] = \"json\" props[\"pegasus.catalog.workflow.amqp.url\"] = \"amqp://friend:donatedata@msgs.pegasus.isi.edu:5672/prod/workflows\" props[\"pegasus.metrics.app\"] = \"KOTO\" Writes the file sites.yml . This file describes the execution environment in which the workflow will be run. # --- Sites -------------------------------------------------------------------- sc = SiteCatalog() # local site (submit machine) local_site = Site(name=\"local\") local_shared_scratch = Directory(directory_type=Directory.SHARED_SCRATCH, path=WORK_DIR / \"scratch\") local_shared_scratch.add_file_servers(FileServer(url=\"file://\" + str(WORK_DIR / \"scratch\"), operation_type=Operation.ALL)) local_site.add_directories(local_shared_scratch) local_storage = Directory(directory_type=Directory.LOCAL_STORAGE, path=WORK_DIR / \"outputs\") local_storage.add_file_servers(FileServer(url=\"file://\" + str(WORK_DIR / \"outputs\"), operation_type=Operation.ALL)) local_site.add_directories(local_storage) local_site.add_env(PATH=os.environ[\"PATH\"]) sc.add_sites(local_site) # stash site (staging site, where intermediate data will be stored) stash_site = Site(name=\"stash\", arch=Arch.X86_64, os_type=OS.LINUX) stash_staging_path = \"/collab/user/{USER}/staging\".format(USER=getpass.getuser()) stash_shared_scratch = Directory(directory_type=Directory.SHARED_SCRATCH, path=stash_staging_path) stash_shared_scratch.add_file_servers( FileServer( url=\"stash:///osgconnect{STASH_STAGING_PATH}\".format(STASH_STAGING_PATH=stash_staging_path), operation_type=Operation.ALL) ) stash_site.add_directories(stash_shared_scratch) sc.add_sites(stash_site) # condorpool (execution site) condorpool_site = Site(name=\"condorpool\", arch=Arch.X86_64, os_type=OS.LINUX) condorpool_site.add_pegasus_profile(style=\"condor\") condorpool_site.add_condor_profile( universe=\"vanilla\", requirements=\"HAS_SINGULARITY == True && HAS_AVX2 == True\", request_cpus=1, request_memory=\"4 GB\", request_disk=\"10 GB\", ) condorpool_site.add_profiles( Namespace.CONDOR, key=\"+SingularityImage\", value='\"/cvmfs/singularity.opensciencegrid.org/chiehlin0212/koto-dev:latest\"' ) condorpool_site.add_profiles( Namespace.CONDOR, key=\"+ProjectName\", value='\"collab.KOTO\"' ) sc.add_sites(condorpool_site) # write SiteCatalog to ./sites.yml sc.write() Writes the file transformations.yml . This file specifies the executables used in the workflow and contains the locations where they are physically located. In this example, we only have run_koto.sh . # --- Transformations ---------------------------------------------------------- run_koto = Transformation( name=\"run_koto\", site=\"local\", pfn=TOP_DIR / \"bin/run_koto.sh\", is_stageable=True ).add_pegasus_profile(clusters_size=1) tc = TransformationCatalog() tc.add_transformations(run_koto) # write TransformationCatalog to ./transformations.yml tc.write() Writes the file replicas.yml . This file specifies the physical locations of any input files used by the workflow. In this example, there is an entry for each file in the inputs/ directory, and all files matching /collab/project/KOTO/external/run87/accidental/64kW/**/*.root # --- Replicas ----------------------------------------------------------------- rc = ReplicaCatalog() input_files = [] # all files in the inputs/ dir input_files = glob.glob(str(TOP_DIR / \"inputs/*\")) for f in input_files: lfn = os.path.basename(f) rc.add_replica(site=\"local\", lfn=lfn, pfn=f) # also add all wave files wave_files = glob.glob(\"/collab/project/KOTO/external/run87/accidental/64kW/**/*.root\", recursive=True) for f in wave_files: lfn = os.path.basename(f) rc.add_replica(site=\"local\", lfn=lfn, pfn=f) # write ReplicaCatalog to replicas.yml rc.write() Builds the wordfreq workflow and submits it for execution. When wf.plan is invoked, pegasus.properties , sites.yml , transformations.yml , and replicas.yml will be consumed as part of the workflow planning process. Note that in this step there is no mention of data movement and job details as these are added by Pegasus when the workflow is planned into an executable workflow. As part of the planning process, additional jobs which handle scratch directory creation, data staging, and data cleanup are added to the workflow. # --- Workflow ----------------------------------------------------------------- wf = Workflow(name=\"koto\") random.seed() for n in range(10): # arguments decay_mode = \"KL3pi0\" seed = str(random.randint(0, 10000)) wave = os.path.basename(random.choice(wave_files)) # TODO: check if we already have a job with the seed/wave combination print(\" ... added mc job with wave={} / seed={}\".format(wave, seed)) out_root = File(\"{}_BasicVetoAndCuts_Accidental_{}.root\".format(decay_mode, seed)) job = Job(run_koto)\\ .add_args(decay_mode, seed, wave)\\ .add_outputs(out_root) # inputs, only use the lfn for inp in input_files + [wave]: job.add_inputs(os.path.basename(inp)) wf.add_jobs(job) # plan and run the workflow wf.plan( dir=WORK_DIR / \"runs\", sites=[\"condorpool\"], staging_sites={\"condorpool\": \"stash\"}, output_sites=[\"local\"], cluster=[\"horizontal\"], submit=True ) Submit the workflow by executing workflow.py . $ ./workflow.py Note that when Pegasus plans/submits a workflow, a workflow directory is created and presented in the output. In the following example output, the workflow directory is /home/user/workflows/runs/pegasus/run0014 . 2020.12.18 14:33:07.059 CST: ----------------------------------------------------------------------- 2020.12.18 14:33:07.064 CST: File for submitting this DAG to HTCondor : koto-workflow-0.dag.condor.sub 2020.12.18 14:33:07.070 CST: Log of DAGMan debugging messages : koto-workflow-0.dag.dagman.out 2020.12.18 14:33:07.075 CST: Log of HTCondor library output : koto-workflow-0.dag.lib.out 2020.12.18 14:33:07.080 CST: Log of HTCondor library error messages : koto-workflow-0.dag.lib.err 2020.12.18 14:33:07.086 CST: Log of the life of condor_dagman itself : koto-workflow-0.dag.dagman.log 2020.12.18 14:33:07.091 CST: 2020.12.18 14:33:07.096 CST: -no_submit given, not submitting DAG to HTCondor. You can do this with: 2020.12.18 14:33:07.107 CST: ----------------------------------------------------------------------- 2020.12.18 14:33:11.352 CST: Your database is compatible with Pegasus version: 5.1.0dev [WARNING] Submitting to condor koto-workflow-0.dag.condor.sub 2020.12.18 14:33:12.060 CST: Time taken to execute is 5.818 seconds Your workflow has been started and is running in the base directory: /home/user/workflows/runs/koto-workflow/run0014 *** To monitor the workflow you can run *** pegasus-status -l /home/user/workflows/runs/koto-workflow/run0014 *** To remove your workflow run *** pegasus-remove /home/user/workflows/runs/koto-workflow/run0014 This directory is the handle to the workflow instance and is used by Pegasus command line tools. Some useful tools to know about: pegasus-status -v [wfdir] Provides status on a currently running workflow. ( more ) pegasus-analyzer [wfdir] Provides debugging clues why a workflow failed. Run this after a workflow has failed. ( more ) pegasus-statistics [wfdir] Provides statistics, such as walltimes, on a workflow after it has completed. ( more ) pegasus-remove [wfdir] Removes a workflow from the system. ( more )","title":"KOTO workflow example"},{"location":"projects/lsst/","text":"LSST via Duke \u00b6 LSST is supported by OSG Collaboration Services via the duke.lsst osg project Sign up to get access to the OSG submit node here: Duke CI-Connect Access Portal News \u00b6 You can now use stashcp to transfer data to and from OSG as part of your computational workflow. You must transfer your input files to a new namespace. From your active session on the duke login node the namespace is located in: /collab/users and /collab/project/lsst Users can create their own private directories in users or shared directories in lsst. New accounts will create the user directories automatically and provide read,write permissions in the project/lsst folder To transfer files to a remote site via a job enter this stashcp command in your execution script: stashcp /osgconnect/collab/users/ / . To transfer files, such as output, from the remote site where your jobs is running to your space on stash, enter this command in your execution script stashcp stash:///osgconnect/collab/users/ /.","title":"LSST"},{"location":"projects/lsst/#lsst-via-duke","text":"LSST is supported by OSG Collaboration Services via the duke.lsst osg project Sign up to get access to the OSG submit node here: Duke CI-Connect Access Portal","title":"LSST via Duke"},{"location":"projects/lsst/#news","text":"You can now use stashcp to transfer data to and from OSG as part of your computational workflow. You must transfer your input files to a new namespace. From your active session on the duke login node the namespace is located in: /collab/users and /collab/project/lsst Users can create their own private directories in users or shared directories in lsst. New accounts will create the user directories automatically and provide read,write permissions in the project/lsst folder To transfer files to a remote site via a job enter this stashcp command in your execution script: stashcp /osgconnect/collab/users/ / . To transfer files, such as output, from the remote site where your jobs is running to your space on stash, enter this command in your execution script stashcp stash:///osgconnect/collab/users/ /.","title":"News"},{"location":"projects/moller/","text":"MOLLER \u00b6 The Measurement of a Lepton-Lepton Electroweak Reaction is a proposed high energy physics experiment at JLAB. A pilot project for jobs at the Open Science fabric is already in place on the OSG Connect infrastructure. The main focus for the collaboration at the moment is to carry out extensive simulations that will further validate the detector design. There is no overall project or data management person for the Moller experiment at this point. The PI is Prof Wouter Deconick Wouter.Deconinck@umanitoba.ca","title":"MOLLER"},{"location":"projects/moller/#moller","text":"The Measurement of a Lepton-Lepton Electroweak Reaction is a proposed high energy physics experiment at JLAB. A pilot project for jobs at the Open Science fabric is already in place on the OSG Connect infrastructure. The main focus for the collaboration at the moment is to carry out extensive simulations that will further validate the detector design. There is no overall project or data management person for the Moller experiment at this point. The PI is Prof Wouter Deconick Wouter.Deconinck@umanitoba.ca","title":"MOLLER"},{"location":"projects/project-list/","text":"List of Supported Projects \u00b6 IGWN(LIGO,VIRGO,KAGRA) IceCube REDTOP South Pole Telescope (SPT3g) CMBS4 KOTO XENON Futurecolliders CLAS12 GLUEX EIC/AID2E EIC/ePIC MOLLER HepSim LSST DES Event Horizon Telescope DUNE UHE Neutrino Obsevatory Trinity EUSO-SPB2 VERITAS Other Fermi Lab Collaborations","title":"Overview"},{"location":"projects/project-list/#list-of-supported-projects","text":"IGWN(LIGO,VIRGO,KAGRA) IceCube REDTOP South Pole Telescope (SPT3g) CMBS4 KOTO XENON Futurecolliders CLAS12 GLUEX EIC/AID2E EIC/ePIC MOLLER HepSim LSST DES Event Horizon Telescope DUNE UHE Neutrino Obsevatory Trinity EUSO-SPB2 VERITAS Other Fermi Lab Collaborations","title":"List of Supported Projects"},{"location":"projects/snowmass/","text":"Snowmass21 \u00b6 The Snowmass21 effort is supported with a submit/analysis node on the Connect infrastructure, local and distributed storage, a CVMFS data origin at UChicago, and managed singularity containers for software distribution. The Snowmass effort currently has contributions from more than 50 campuses and institutions across the world with more than 150 users with access to the Snowmass21 submit host with 180 TB of local storage and 50 TB of distributed storage on stash. Data production and analysis lead: John Stupak john.stupak@ou.edu Snowmass has a dedicated operational page found here https://maniaclab.uchicago.edu/snowmass-connect-docs/","title":"Snowmass"},{"location":"projects/snowmass/#snowmass21","text":"The Snowmass21 effort is supported with a submit/analysis node on the Connect infrastructure, local and distributed storage, a CVMFS data origin at UChicago, and managed singularity containers for software distribution. The Snowmass effort currently has contributions from more than 50 campuses and institutions across the world with more than 150 users with access to the Snowmass21 submit host with 180 TB of local storage and 50 TB of distributed storage on stash. Data production and analysis lead: John Stupak john.stupak@ou.edu Snowmass has a dedicated operational page found here https://maniaclab.uchicago.edu/snowmass-connect-docs/","title":"Snowmass21"},{"location":"projects/spt/","text":"South Pole Telescope SPT-3G \u00b6 Support areas for SPT include data management, infrastructure (Submit host, Backups to Federated Facilities, Globus Service, dCache and local storage at UChicago for experimental data from the South Pole and production-quality data from OSG), and best practices for software and job management, SPT is a collaboration that involves over 30 University campuses in the US along with National Laboratories. Examples of main contributions come from the University of Chicago, University of California Berkeley and Davis, the University of Illinois at Urbana/Champaign, Michigan State, and Argonne and Fermi National Labs. SPT-3G is supported by the NSF award # 1852617 (South Pole Telescope Operations and Data Products). Data production and analysis Lead: Sasha Rahlin arahlin@fnal.gov Data management coordination: Tom Crawford tcrawfor@kicp.uchicago.edu Account Portal and sign up for Access to Resources SPT Local Storage Dashboard SPT Grid Storage Dashboard Information fo Users \u00b6 Guide to access OSG resources Guid to obtain Grid Certificates General Information on Software availability and documenation \u00b6","title":"South Pole Telescope"},{"location":"projects/spt/#south-pole-telescope-spt-3g","text":"Support areas for SPT include data management, infrastructure (Submit host, Backups to Federated Facilities, Globus Service, dCache and local storage at UChicago for experimental data from the South Pole and production-quality data from OSG), and best practices for software and job management, SPT is a collaboration that involves over 30 University campuses in the US along with National Laboratories. Examples of main contributions come from the University of Chicago, University of California Berkeley and Davis, the University of Illinois at Urbana/Champaign, Michigan State, and Argonne and Fermi National Labs. SPT-3G is supported by the NSF award # 1852617 (South Pole Telescope Operations and Data Products). Data production and analysis Lead: Sasha Rahlin arahlin@fnal.gov Data management coordination: Tom Crawford tcrawfor@kicp.uchicago.edu Account Portal and sign up for Access to Resources SPT Local Storage Dashboard SPT Grid Storage Dashboard","title":"South Pole Telescope SPT-3G"},{"location":"projects/spt/#information-fo-users","text":"Guide to access OSG resources Guid to obtain Grid Certificates General Information on Software availability and documenation","title":"Information fo Users"},{"location":"projects/spt/#_1","text":"","title":""},{"location":"projects/veritas/","text":"VERITAS Collaboration \u00b6","title":"VERITAS"},{"location":"projects/veritas/#veritas-collaboration","text":"","title":"VERITAS Collaboration"},{"location":"projects/xenon/","text":"XENON \u00b6 Support areas for XENON include data management, infrastructure (Submit host, dCache storage, FTS service for Rucio managed third-party transfers, Rucio Service), and best practices for software and job management. XENON is supported by NSF institutional grants PHY-2112796 at UChicago, PHY-2112851 at Columbia University, PHY-2112801 at Rice University, PHY-2112802 at UC San Diego and PHY-21128-3 at Purdue University. Data Production team Lead: Evan Shockley eshockley@physics.ucsd.edu MC Production team Lead: Diego Ramirez diego.ramirez@physik.uni-freiburg.de At present, XENON consumes ~6 PB of distributed storage f in the US & Europe managed by Rucio. Total storage utilization has remained constant in time, as the collaboration is deleting older data - from XENON 1T - while it continues to gather fresh data from the current phase - XENON nT. Nonetheless, the overall replicated distributed storage is expected to increase by the end of the 5-year lifetime of the experiment because the amount of data volume for the nT phase is projected to be five times larger than the 1T volume. Status of Services \u00b6 Rucio Server Ganglia FTS Server Ganglia Check-mk Xenon Rucio Group Rucio Postgress XENON login node Weekly XENON analysis/Compute call \u00b6 URL: https://riceuniversity.zoom.us/j/4815153031 Minutes:https://xe1t-wiki.lngs.infn.it/doku.php?id=xenon:xenon1t:analysistools:200402","title":"XENON"},{"location":"projects/xenon/#xenon","text":"Support areas for XENON include data management, infrastructure (Submit host, dCache storage, FTS service for Rucio managed third-party transfers, Rucio Service), and best practices for software and job management. XENON is supported by NSF institutional grants PHY-2112796 at UChicago, PHY-2112851 at Columbia University, PHY-2112801 at Rice University, PHY-2112802 at UC San Diego and PHY-21128-3 at Purdue University. Data Production team Lead: Evan Shockley eshockley@physics.ucsd.edu MC Production team Lead: Diego Ramirez diego.ramirez@physik.uni-freiburg.de At present, XENON consumes ~6 PB of distributed storage f in the US & Europe managed by Rucio. Total storage utilization has remained constant in time, as the collaboration is deleting older data - from XENON 1T - while it continues to gather fresh data from the current phase - XENON nT. Nonetheless, the overall replicated distributed storage is expected to increase by the end of the 5-year lifetime of the experiment because the amount of data volume for the nT phase is projected to be five times larger than the 1T volume.","title":"XENON"},{"location":"projects/xenon/#status-of-services","text":"Rucio Server Ganglia FTS Server Ganglia Check-mk Xenon Rucio Group Rucio Postgress XENON login node","title":"Status of Services"},{"location":"projects/xenon/#weekly-xenon-analysiscompute-call","text":"URL: https://riceuniversity.zoom.us/j/4815153031 Minutes:https://xe1t-wiki.lngs.infn.it/doku.php?id=xenon:xenon1t:analysistools:200402","title":"Weekly XENON analysis/Compute call"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Mission \u00b6 The mission of OSG Collaboration Support is to help Midsize Collaborations leverage the OSG Consortium services and technologies for their research. A list of services can be found here . Among them, is provisioning access to an OSG-managed Collab Access Point as a single point of submission of dHTC jobs to the Open Science Pool (OSPool). Users from different institutions can join a collaboration specific project. This simplifies access to compute and storage resources as users are not bound by institutional barriers - such as the need for insitutional based identies. For example Institutions A and B need to work on a project. It is not easy for students/postdocs from Institution A to obtain an identity that will allow them to collaborate with their collegues at the Institution B. On the Collab AP, PIs can create a single project with users from both institutions. This allows for a collaborative environment where the research team can launch jobs to the OSPool, levereage multi-TB ephemeral storage and even run computational workflows - like post-processing or mereging jobs on the access point (the login node). Obtain accounts on OSG's Collab Access Point. \u00b6 A PI can visit the accounting portal (https://ci-connect.net) sign-up using CI Logon credentials, create a project and add users. A consultation with someone from Collaboration Support is required. A walk through through the process is linked here . News \u00b6 11/21/2024 The Open Science Data Platorm (OSDF) origin that was deployed on the OSG Collab AP, ap23.uc.osg-htc.org, has moved to a separate infrastructure to accommodate upgrading to the Pelican data distribution platform and allow for users at other APs use the backend storage. For more details refer to this [section](misc/pelican.md). 04/18/2023 A new Collab AP host, ap23.uc.osg-htg.org. has been deployed which will replace the existing one, login.collab.ci-connect.net. Users and projects from login.collab.ci-connect.net have moved to the new one. Users are responsible for migrating their data and make any necessary corrections to their workflows. Collaboration Support staff will be available to assist. Collaboration Institutions and Compute Sites \u00b6 Regular Meeting schedule \u00b6 Collaboration support holds and participates in meetings during the week to assist and coordinate on projects. A list of the weekly meeting schedule is maintained here List of supported collaboratoins \u00b6 Collaboration list Contact Us \u00b6 Open a Ticket by sending an email to help@osg-htc.org and request support for your collaboration project.","title":"OSG Collaboration Support"},{"location":"#mission","text":"The mission of OSG Collaboration Support is to help Midsize Collaborations leverage the OSG Consortium services and technologies for their research. A list of services can be found here . Among them, is provisioning access to an OSG-managed Collab Access Point as a single point of submission of dHTC jobs to the Open Science Pool (OSPool). Users from different institutions can join a collaboration specific project. This simplifies access to compute and storage resources as users are not bound by institutional barriers - such as the need for insitutional based identies. For example Institutions A and B need to work on a project. It is not easy for students/postdocs from Institution A to obtain an identity that will allow them to collaborate with their collegues at the Institution B. On the Collab AP, PIs can create a single project with users from both institutions. This allows for a collaborative environment where the research team can launch jobs to the OSPool, levereage multi-TB ephemeral storage and even run computational workflows - like post-processing or mereging jobs on the access point (the login node).","title":"Mission"},{"location":"#obtain-accounts-on-osgs-collab-access-point","text":"A PI can visit the accounting portal (https://ci-connect.net) sign-up using CI Logon credentials, create a project and add users. A consultation with someone from Collaboration Support is required. A walk through through the process is linked here .","title":"Obtain accounts on OSG's Collab Access Point."},{"location":"#news","text":"11/21/2024 The Open Science Data Platorm (OSDF) origin that was deployed on the OSG Collab AP, ap23.uc.osg-htc.org, has moved to a separate infrastructure to accommodate upgrading to the Pelican data distribution platform and allow for users at other APs use the backend storage. For more details refer to this [section](misc/pelican.md). 04/18/2023 A new Collab AP host, ap23.uc.osg-htg.org. has been deployed which will replace the existing one, login.collab.ci-connect.net. Users and projects from login.collab.ci-connect.net have moved to the new one. Users are responsible for migrating their data and make any necessary corrections to their workflows. Collaboration Support staff will be available to assist.","title":"News"},{"location":"#collaboration-institutions-and-compute-sites","text":"","title":"Collaboration Institutions and Compute Sites"},{"location":"#regular-meeting-schedule","text":"Collaboration support holds and participates in meetings during the week to assist and coordinate on projects. A list of the weekly meeting schedule is maintained here","title":"Regular Meeting schedule"},{"location":"#list-of-supported-collaboratoins","text":"Collaboration list","title":"List of supported collaboratoins"},{"location":"#contact-us","text":"Open a Ticket by sending an email to help@osg-htc.org and request support for your collaboration project.","title":"Contact Us"},{"location":"misc/meeting-schedule/","text":"Schedule of collaborative meetings/calls \u00b6 OSG Area Coordinators meeting \u00b6 When: Wednesdays 2-3pm Central URL: https://unl.zoom.us/j/865025296 IGWN-CESER meetting \u00b6 When: Tuesdays 12:30pm-1:30pm Central Details in the IGWN Collaboration page Icecube-CESER meeting \u00b6 When: Biweeklyy on Mondays 3:00 pm-4:00pm Central Detais in the Icecube Collaboration page XENON Analysis/Compute meeting \u00b6 When: Thursdays 10:00am-11:00am Central Details in the XENON Collaboration page IGWN production meeting \u00b6 When: Thursdays 9:00am-10:00am Central Details in the IGWN Collaboration page Koto support meetting \u00b6 When: Thursdays 3:00pm - 4:00pm Central URL: https://uchicago.zoom.us/s/4250591437 xCache meetting \u00b6 When: Thursdays 1:00pm-2:00pm Central URL: https://unl.zoom.us/j/651969661 Internet2-CESER meeting \u00b6 When: Fridays 2:00pm-3:00pm Central URL: https://internet2.zoom.us/j/668736847","title":"Meeting/Call schedule"},{"location":"misc/meeting-schedule/#schedule-of-collaborative-meetingscalls","text":"","title":"Schedule of collaborative meetings/calls"},{"location":"misc/meeting-schedule/#osg-area-coordinators-meeting","text":"When: Wednesdays 2-3pm Central URL: https://unl.zoom.us/j/865025296","title":"OSG Area Coordinators meeting"},{"location":"misc/meeting-schedule/#igwn-ceser-meetting","text":"When: Tuesdays 12:30pm-1:30pm Central Details in the IGWN Collaboration page","title":"IGWN-CESER meetting"},{"location":"misc/meeting-schedule/#icecube-ceser-meeting","text":"When: Biweeklyy on Mondays 3:00 pm-4:00pm Central Detais in the Icecube Collaboration page","title":"Icecube-CESER meeting"},{"location":"misc/meeting-schedule/#xenon-analysiscompute-meeting","text":"When: Thursdays 10:00am-11:00am Central Details in the XENON Collaboration page","title":"XENON Analysis/Compute meeting"},{"location":"misc/meeting-schedule/#igwn-production-meeting","text":"When: Thursdays 9:00am-10:00am Central Details in the IGWN Collaboration page","title":"IGWN production meeting"},{"location":"misc/meeting-schedule/#koto-support-meetting","text":"When: Thursdays 3:00pm - 4:00pm Central URL: https://uchicago.zoom.us/s/4250591437","title":"Koto support meetting"},{"location":"misc/meeting-schedule/#xcache-meetting","text":"When: Thursdays 1:00pm-2:00pm Central URL: https://unl.zoom.us/j/651969661","title":"xCache meetting"},{"location":"misc/meeting-schedule/#internet2-ceser-meeting","text":"When: Fridays 2:00pm-3:00pm Central URL: https://internet2.zoom.us/j/668736847","title":"Internet2-CESER meeting"},{"location":"misc/onboard/","text":"This is a test to build up","title":"Access to Collab AP"},{"location":"misc/pelican/","text":"Uprade of the OSDF origin to Pelican for Collaborations \u00b6 The OSG Collab AP had an OSDF door deployed on the access point (ap23.uc.osg-htc.org) that provided users with authenticated access to a Ceph cluster, that provides high capacity storage to shared project directories. HTCondor jobs running in the OSPool at remote Execution Points (EPs) can access the filesystem either via a client tool or via an HTCondor plugin that is invoked in their submit scripts. The storage is also mounted on the AP at /ospool/uc-shared/project. On 11/21/2024 , OSG/PATh staff migrated the OSDF door from the OSG Collab AP to a separate infrastructure to allow upgrading the origin to the Pelican Platform (https://pelicanplatform.org/) and provide shared project access to users at other APs (ap20.uc.osg-htc.org and ap21.uc.osg-htc.org). The migration should have been transparent to the OSG Collab AP. If using the HTCondor plugin, no changes are needed in your submission scripts. For reference, the general purpose documentation found here (https://portal.osg-htc.org/documentation/htc_workloads/managing_data/osdf/) is also applicable to the users of the OSG Collab AP. It describes using the HTCondor plugin to move data to and from the OSDF/Pelican Origin. In a nutshell then: Include the following in your submit script for an OSPool job read a file from your project directory at the EP: transfer_input_files = osdf:///ospool/uc-shared/project/ / 5. Include the folloow in your submit script for an OSPool job to write a file in your project directory from an EP OSDF_LOCATION = osdf:///ospool/uc-shared/project/ / transfer_input_files = $(OSDF_LOCATION)/ The upgrade to the Pelican platform, which uses federated urls for the origin, still keeps the same prefix osdf:// . After the upgrade, osdf:// simply points to pelican://osg-htc.org/ If you are using a client in your runtime script at the EP, then the previous tool, stashcp , will continue to work as long as the version is > 6.12 except recursive access ( -r flag). Thefore, we recommend that groups migrate to using the pelican client instead. The pelican equivalent of the stashcp client read command: stashcp -d osdf:///ospool/uc-shared/project// is pelican object get -d osdf:///ospool/uc-shared/project/xenon/ppaschos/testfile . Similarly, the pelican equivalent of the stashcp write comamnd, stashcp -d osdf:///ospool/uc-shared/project/ is pelican object put -d osdf:///ospool//uc-shared/project/ If your container image was built using one of the OSG base environments, then the pelican client tool is already included. The pelican client can be installed from the osg repo with dnf install pelican . Refer to this documentation (https://osg-htc.org/docs/common/yum/) to add the osg repository.","title":"Pelican"},{"location":"misc/pelican/#uprade-of-the-osdf-origin-to-pelican-for-collaborations","text":"The OSG Collab AP had an OSDF door deployed on the access point (ap23.uc.osg-htc.org) that provided users with authenticated access to a Ceph cluster, that provides high capacity storage to shared project directories. HTCondor jobs running in the OSPool at remote Execution Points (EPs) can access the filesystem either via a client tool or via an HTCondor plugin that is invoked in their submit scripts. The storage is also mounted on the AP at /ospool/uc-shared/project. On 11/21/2024 , OSG/PATh staff migrated the OSDF door from the OSG Collab AP to a separate infrastructure to allow upgrading the origin to the Pelican Platform (https://pelicanplatform.org/) and provide shared project access to users at other APs (ap20.uc.osg-htc.org and ap21.uc.osg-htc.org). The migration should have been transparent to the OSG Collab AP. If using the HTCondor plugin, no changes are needed in your submission scripts. For reference, the general purpose documentation found here (https://portal.osg-htc.org/documentation/htc_workloads/managing_data/osdf/) is also applicable to the users of the OSG Collab AP. It describes using the HTCondor plugin to move data to and from the OSDF/Pelican Origin. In a nutshell then: Include the following in your submit script for an OSPool job read a file from your project directory at the EP: transfer_input_files = osdf:///ospool/uc-shared/project/ / 5. Include the folloow in your submit script for an OSPool job to write a file in your project directory from an EP OSDF_LOCATION = osdf:///ospool/uc-shared/project/ / transfer_input_files = $(OSDF_LOCATION)/ The upgrade to the Pelican platform, which uses federated urls for the origin, still keeps the same prefix osdf:// . After the upgrade, osdf:// simply points to pelican://osg-htc.org/ If you are using a client in your runtime script at the EP, then the previous tool, stashcp , will continue to work as long as the version is > 6.12 except recursive access ( -r flag). Thefore, we recommend that groups migrate to using the pelican client instead. The pelican equivalent of the stashcp client read command: stashcp -d osdf:///ospool/uc-shared/project// is pelican object get -d osdf:///ospool/uc-shared/project/xenon/ppaschos/testfile . Similarly, the pelican equivalent of the stashcp write comamnd, stashcp -d osdf:///ospool/uc-shared/project/ is pelican object put -d osdf:///ospool//uc-shared/project/ If your container image was built using one of the OSG base environments, then the pelican client tool is already included. The pelican client can be installed from the osg repo with dnf install pelican . Refer to this documentation (https://osg-htc.org/docs/common/yum/) to add the osg repository.","title":"Uprade of the OSDF origin to Pelican for Collaborations"},{"location":"misc/services/","text":"Collab Access Point (AP) \u00b6 Onboarding to the OSG managed and Collaboration dedicated Access Point. Users will be trained in job submission and software development appropriate for the dHTC jobs. In addition, the project will be supported with visible performance metrics coming from the central OSG Accounting dashboard Gracc . Collaboration Support (CS) staff will provide consultation services on how to best optimize the workflow, help with debugging any issues encountered and facilitate coordination with other institutional partners that require access to produded data products. Data management and storage \u00b6 All projects will be provisioned with distributed storage on the Open Science Data Federation (OSDF) and local storage on the AP. Collaboration support will work with the collaboration in managing the storage within the limits agreed upon during the onboarding process. Staff will train users on how to best use data management tools to monitor, track and distribute data to jobs or other institutional storage locations. Collaboration support staff will also work with the PI in understanding the long-term needs of the project and architect solutions where the collaboration can integrate with the OSDF federation. These include assistance in setting up Origins of data which will distribute them to the network of Caches which are placed in close proximity to execution points. CS staff, in coordination with other OSG teams, will also assist in provisioning services of institutional managed caches in proximity to locations where compute cycles are available or deploy Hosted CEs on behalf on the collaboration. Additional tools, such as Rucio, are also available to provide a service for distributing data without the need for an Origin. Data in this case is hosted in part at each member institution with Rucio providing a centralized management service of where the data are and where they need to be replicated - either for use in dHTC jobs or local batch systems. Follow this link to look at the Kibana dashboard tracking FTS transfers for Icecube, LIGO and XENON and Rucio information on XENON. Integration of Collaboration owned resources \u00b6 This process results in a pool of resources where a job submission from an AP, e.g. the Collab AP, would be run with priority because the collaboration owns the available compute cycle. This is accomplished by deploying Compute Entry (CE) points on the edge infrastructure of institutions. A project is affiliated with an entity named Virtual Organization, which is used to allow factory operators to directly send the project's jobs to the dedicated compute resources behind the CE. An international collaboration can then achieve a global computing scale if CEs are deployed nearby member institutions, all running jobs on their local clusters for the collaboration. Those clusters can either be wholly owned by the collaboration or the latter might have a compute allocation there. In the end, the project can leverage both the Open Science Pool and it's own pool to maximize throughput.","title":"Services"},{"location":"misc/services/#collab-access-point-ap","text":"Onboarding to the OSG managed and Collaboration dedicated Access Point. Users will be trained in job submission and software development appropriate for the dHTC jobs. In addition, the project will be supported with visible performance metrics coming from the central OSG Accounting dashboard Gracc . Collaboration Support (CS) staff will provide consultation services on how to best optimize the workflow, help with debugging any issues encountered and facilitate coordination with other institutional partners that require access to produded data products.","title":"Collab Access Point (AP)"},{"location":"misc/services/#data-management-and-storage","text":"All projects will be provisioned with distributed storage on the Open Science Data Federation (OSDF) and local storage on the AP. Collaboration support will work with the collaboration in managing the storage within the limits agreed upon during the onboarding process. Staff will train users on how to best use data management tools to monitor, track and distribute data to jobs or other institutional storage locations. Collaboration support staff will also work with the PI in understanding the long-term needs of the project and architect solutions where the collaboration can integrate with the OSDF federation. These include assistance in setting up Origins of data which will distribute them to the network of Caches which are placed in close proximity to execution points. CS staff, in coordination with other OSG teams, will also assist in provisioning services of institutional managed caches in proximity to locations where compute cycles are available or deploy Hosted CEs on behalf on the collaboration. Additional tools, such as Rucio, are also available to provide a service for distributing data without the need for an Origin. Data in this case is hosted in part at each member institution with Rucio providing a centralized management service of where the data are and where they need to be replicated - either for use in dHTC jobs or local batch systems. Follow this link to look at the Kibana dashboard tracking FTS transfers for Icecube, LIGO and XENON and Rucio information on XENON.","title":"Data management and storage"},{"location":"misc/services/#integration-of-collaboration-owned-resources","text":"This process results in a pool of resources where a job submission from an AP, e.g. the Collab AP, would be run with priority because the collaboration owns the available compute cycle. This is accomplished by deploying Compute Entry (CE) points on the edge infrastructure of institutions. A project is affiliated with an entity named Virtual Organization, which is used to allow factory operators to directly send the project's jobs to the dedicated compute resources behind the CE. An international collaboration can then achieve a global computing scale if CEs are deployed nearby member institutions, all running jobs on their local clusters for the collaboration. Those clusters can either be wholly owned by the collaboration or the latter might have a compute allocation there. In the end, the project can leverage both the Open Science Pool and it's own pool to maximize throughput.","title":"Integration of Collaboration owned resources"},{"location":"projects/Redtop/","text":"REDTOP \u00b6 Support areas of REDTOP include data management, provisioned storage on Stash (~100 TB), and assistance in executing their job workloads. Computing Model for REDTOP using OSG resources: https://redtop.fnal.gov/computing-model/ OSG Collaboration Support Services supports REDTOP workflows on the OpenScienceGrid and has provided a compute plan in support of the colllaboration's LOI to Snowmass21","title":"REDTOP"},{"location":"projects/Redtop/#redtop","text":"Support areas of REDTOP include data management, provisioned storage on Stash (~100 TB), and assistance in executing their job workloads. Computing Model for REDTOP using OSG resources: https://redtop.fnal.gov/computing-model/ OSG Collaboration Support Services supports REDTOP workflows on the OpenScienceGrid and has provided a compute plan in support of the colllaboration's LOI to Snowmass21","title":"REDTOP"},{"location":"projects/clas12/","text":"CLAS12 \u00b6 The CLAS12 experiment at Jefferson Lab is supported in part by the U.S. Department of Energy, the National Science Foundation (NSF), the Italian Istituto Nazionale di Fisica Nucleare (INFN), the French Centre National de la Recherche Scientifique (CNRS), the French Commissariat pour l\u2019Energie Atomique, the UK Science and Technology Facilities Council and other international funding CLAS12 data and production management: Maurizio Ungaro ungaro@jlab.org","title":"GLAS12"},{"location":"projects/clas12/#clas12","text":"The CLAS12 experiment at Jefferson Lab is supported in part by the U.S. Department of Energy, the National Science Foundation (NSF), the Italian Istituto Nazionale di Fisica Nucleare (INFN), the French Centre National de la Recherche Scientifique (CNRS), the French Commissariat pour l\u2019Energie Atomique, the UK Science and Technology Facilities Council and other international funding CLAS12 data and production management: Maurizio Ungaro ungaro@jlab.org","title":"CLAS12"},{"location":"projects/eic/","text":"EIC \u00b6 EIC (Electron-Ion Collider) is a proposed High Energy Physics experiment, a joint effort between Brookhaven and Jefferson national labs. Overall project management lead: Jerome Lauret eromel@bnl.gov BNL project coordinator: Xin Zhao xzhao@bnl.gov JLAB project coordinators: Markus Diefenthaler, mdiefent@jlab.org and David Lawrence, davidl@jlab.org","title":"EIC"},{"location":"projects/eic/#eic","text":"EIC (Electron-Ion Collider) is a proposed High Energy Physics experiment, a joint effort between Brookhaven and Jefferson national labs. Overall project management lead: Jerome Lauret eromel@bnl.gov BNL project coordinator: Xin Zhao xzhao@bnl.gov JLAB project coordinators: Markus Diefenthaler, mdiefent@jlab.org and David Lawrence, davidl@jlab.org","title":"EIC"},{"location":"projects/gluex/","text":"GLUEX \u00b6 The GlueX experiment at Jefferson Lab is supported in part by the U.S. Department of Energy, the U.S. National Science Foundation, the German Research Foundation, and other international foundations and institutions. The UConn group acknowledges support for work on GlueX from the U.S. National Science Foundation under grant 1812415. GlueX data and production management: Richard Jones richard.t.jones@uconn.edu JLAB GlueX coordinators: Thomas Britton tbritton@jlab.org and Bryan Hess bhess@jlab.org","title":"GLUEX"},{"location":"projects/gluex/#gluex","text":"The GlueX experiment at Jefferson Lab is supported in part by the U.S. Department of Energy, the U.S. National Science Foundation, the German Research Foundation, and other international foundations and institutions. The UConn group acknowledges support for work on GlueX from the U.S. National Science Foundation under grant 1812415. GlueX data and production management: Richard Jones richard.t.jones@uconn.edu JLAB GlueX coordinators: Thomas Britton tbritton@jlab.org and Bryan Hess bhess@jlab.org","title":"GLUEX"},{"location":"projects/icecube/","text":"IceCube-PATh \u00b6 Icecube is supported by OSG Collaborations via the PATh project. This page will include meeting-minutes and location along with status reports","title":"IceCube"},{"location":"projects/icecube/#icecube-path","text":"Icecube is supported by OSG Collaborations via the PATh project. This page will include meeting-minutes and location along with status reports","title":"IceCube-PATh"},{"location":"projects/igwn/","text":"International Gravitational-Wave Observatory Network \u00b6 Support areas for IGWN include data management, infrastructure (StashCaches, FTS service for Rucio managed third-party transfers), best practices for software and job management, and in-person participation in the Open Science - LIGO weekly meetings. The LIGO component of the IGWN collaboration has 119 institutions and universities in the US that make use of the Open Science distributed infrastructure for computing. LIGO is supported by the NSF award # PHY-2110594 for the LIGO Data Grid (LDG) and NSF award # PHY-1764464 for the LIGO Lab in computing, R&D, and operations. A detailed list of technical contacts and leads for the various workloads is maintained by the LIGO and Open Science group here. For general purpose data and job management, the lead is: James Clark james.clark@ligo.org Institutions Compute Resources Weekly IGWN-PATh meeting \u00b6 Topic: PATh-IGWN technical call Time: 11:00 PM Central Time (US and Canada) Every week on Thur Join Zoom Meeting: TB-updated Weekly IGWN Production meeting \u00b6 Time: 9:00 AM Central Time (US and Canada) Every Week on Thu Links to be updated. Currently access is only restricted to people with ligo credentials. Weekly IGWN Condor meeting \u00b6 TB updated","title":"IGWN"},{"location":"projects/igwn/#international-gravitational-wave-observatory-network","text":"Support areas for IGWN include data management, infrastructure (StashCaches, FTS service for Rucio managed third-party transfers), best practices for software and job management, and in-person participation in the Open Science - LIGO weekly meetings. The LIGO component of the IGWN collaboration has 119 institutions and universities in the US that make use of the Open Science distributed infrastructure for computing. LIGO is supported by the NSF award # PHY-2110594 for the LIGO Data Grid (LDG) and NSF award # PHY-1764464 for the LIGO Lab in computing, R&D, and operations. A detailed list of technical contacts and leads for the various workloads is maintained by the LIGO and Open Science group here. For general purpose data and job management, the lead is: James Clark james.clark@ligo.org Institutions Compute Resources","title":"International Gravitational-Wave Observatory Network"},{"location":"projects/igwn/#weekly-igwn-path-meeting","text":"Topic: PATh-IGWN technical call Time: 11:00 PM Central Time (US and Canada) Every week on Thur Join Zoom Meeting: TB-updated","title":"Weekly IGWN-PATh meeting"},{"location":"projects/igwn/#weekly-igwn-production-meeting","text":"Time: 9:00 AM Central Time (US and Canada) Every Week on Thu Links to be updated. Currently access is only restricted to people with ligo credentials.","title":"Weekly IGWN Production meeting"},{"location":"projects/igwn/#weekly-igwn-condor-meeting","text":"TB updated","title":"Weekly IGWN Condor meeting"},{"location":"projects/koto/","text":"KOTO \u00b6 KOTO is an international collaboration of an experiment based at the JPARC HEP facility in Japan. Access Point \u00b6 KOTO submits out of the Connect node: login.collab.ci-connect.net. To request an account, visit the portal at https://ci-connect.net and ask to join the PATh Collaborations group. KOTO is a subgroup under PATh collaborations and a separate request needs to be made to join by clicking the request memberhip button. Storage access \u00b6 Besides their home directories on login.collab.ci-connect.net, users have access to local storage in /scratch/ and distributed storage in /collab/user/ . In addition, the collaboration has access to /collab/project/KOTO for sharing data products between the users. The distributed storage is http accessible: http://stash.osgstorage.org/collab/KOTO and data can be downloaded using a browser or command line utilities like wget or curl. Jobs to the OSPool \u00b6 Inlined below is a submit script to run a KOTO job on the OSPool 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 #!/bin/bash Universe = Vanilla Executable = run_koto.sh Requirements = HAS_SINGULARITY == TRUE +SingularityImage = \"/cvmfs/singularity.opensciencegrid.org/ppaschos/koto-dev:latest\" transfer_input_files = e14_201605.mac Error = output.err. $( Cluster ) - $( Process ) Output = output.out. $( Cluster ) - $( Process ) Log = output.log. $( Cluster ) should_transfer_files = YES WhenToTransferOutput = ON_EXIT request_cpus = 1 request_memory = 2GB request_disk = 2GB +ProjectName = \"collab.KOTO\" Queue 1 KOTO uses a software stack that is included in the Singularity image - shown in the script above. The image is deployed on the remote execution nodes and jobs run within the Singularity container. The input file, e14_201605.mac, is sent to the execution node as part of the job. The execution script, run_koto.sh, sets the environment and includes all invocations to executables. The following script, run_koto.sh, runs a benchmark example from the E14 library. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 #!/bin/bash echo $HOSTNAME source /opt/koto/root/v6.22.02/bin/thisroot.sh source /opt/koto/geant4/10.05.p01/bin/geant4.sh echo \"PATH --> \" $PATH echo \"LD_LIBRARY_PATH --> \" $LD_LIBRARY_PATH echo \"ROOTSYS --> \" $ROOTSYS echo \"G4 data environent\" env | grep G4 export E14_TOP_DIR = \"/opt/koto/e14/201605v6.2/e14\" echo \"E14_TOP_DIR=\" $E14_TOP_DIR source /opt/koto/e14/201605v6.2/setup.sh output = \"output.root\" nevent = 1000 seed = 0 echo \"Running Code\" /opt/koto/e14/201605v6.2/e14/examples/gsim4test/bin/gsim4test e14_201605.mac ${ output } ${ nevent } ${ seed } ${ seed } Pegasus Workflows \u00b6 The Pegasus project encompasses a set of technologies that help workflow-based applications execute in a number of different environments including desktops, campus clusters, grids, and clouds. Pegasus bridges the scientific domain and the execution environment by automatically mapping high-level workflow descriptions onto distributed resources. It automatically locates the necessary input data and computational resources necessary for workflow execution. Pegasus enables scientists to construct workflows in abstract terms without worrying about the details of the underlying execution environment or the particulars of the low-level specifications required by the middleware. Some of the advantages of using Pegasus include: Portability / Reuse - User created workflows can easily be run in different environments without alteration. Pegasus currently runs workflows on compute systems scheduled via HTCondor, including the OSPool, and other other systems or via other schedulers (e.g. ACCESS resources, Amazon EC2, Google Cloud, and many campus clusters). The same workflow can run on a single system or across a heterogeneous set of resources. Performance - The Pegasus mapper can reorder, group, and prioritize tasks in order to increase the overall workflow performance. Scalability - Pegasus can easily scale both the size of the workflow, and the resources that the workflow is distributed over. Pegasus runs workflows ranging from just a few computational tasks up to 1 million tasks. The number of resources involved in executing a workflow can scale as needed without any impediments to performance. Provenance - By default, all jobs in Pegasus are launched via the kickstart process that captures runtime provenance of the job and helps in debugging. The provenance data is collected in a database, and the data can be summarized with tools such as pegasus-statistics or directly with SQL queries. Data Management - Pegasus handles replica selection, data transfers and output registrations in data catalogs. These tasks are added to a workflow as auxiliary jobs by the Pegasus planner. Reliability - Jobs and data transfers are automatically retried in case of failures. Debugging tools such as pegasus-analyzer help the user to debug the workflow in case of non-recoverable failures. Error Recovery - When errors occur, Pegasus tries to recover when possible by retrying tasks, retrying the entire workflow, providing workflow-level checkpointing, re-mapping portions of the workflow, trying alternative data sources for staging data, and, when all else fails, providing a rescue workflow containing a description of only the work that remains to be done. Pegasus keeps track of what has been done (provenance) including the locations of data used and produced, and which software was used with which parameters. OSG has no read/write enabled shared file system across the resources. Jobs are required to either bring inputs along with the job, or as part of the job stage the inputs from a remote location. The following examples highlight how Pegasus can be used to manage KOTO workloads in such an environment by providing an abstraction layer around things like data movements and job retries, enabling the users to run larger workloads, spending less time developing job management tools and babysitting their computations. Pegasus workflows have 4 components: Site Catalog - Describes the execution environment in which the workflow will be executed. Transformation Catalog - Specifies locations of the executables used by the workflow. Replica Catalog - Specifies locations of the input data to the workflow. Workflow Description - An abstract workflow description containing compute steps and dependencies between the steps. We refer to this workflow as abstract because it does not contain data locations and available software. When developing a Pegasus Workflow using the Python API , all four components may be defined in the same file. For details, please refer to the Pegasus documentation . KOTO workflow example \u00b6 The GitHub repo https://github.com/pegasus-isi/koto-pegasus contains a sample Pegasus workflow for KOTO processing. It is a very simple Monte Carlo example, containing just N independent jobs: This example is using OSG StashCache for data transfers. Credentials are transparant to the end users. Additionally, this example uses a custom container to run jobs. This ensures a consistent and complete environment for the jobs. Which container to use is defined in workflow.py with the +SingularityImage attribute. When invoked, the workflow script ( workflow.py ) does the following: Writes the file pegasus.properties . This file contains configuration settings used by Pegasus and HTCondor. # --- Properties --------------------------------------------------------------- props = Properties() props[\"pegasus.data.configuration\"] = \"nonsharedfs\" # Provide a full kickstart record, including the environment, even for successful jobs props[\"pegasus.gridstart.arguments\"] = \"-f\" # Limit the number of idle jobs for large workflows props[\"dagman.maxidle\"] = \"1000\" # use timestamps for all our dirs - this make sure we have unique names in stash props[\"pegasus.dir.useTimestamp\"] = \"true\" # seperate output dir for each workflow run props[\"pegasus.dir.storage.deep\"] = \"true\" # Help Pegasus developers by sharing performance data (optional) props[\"pegasus.monitord.encoding\"] = \"json\" props[\"pegasus.catalog.workflow.amqp.url\"] = \"amqp://friend:donatedata@msgs.pegasus.isi.edu:5672/prod/workflows\" props[\"pegasus.metrics.app\"] = \"KOTO\" Writes the file sites.yml . This file describes the execution environment in which the workflow will be run. # --- Sites -------------------------------------------------------------------- sc = SiteCatalog() # local site (submit machine) local_site = Site(name=\"local\") local_shared_scratch = Directory(directory_type=Directory.SHARED_SCRATCH, path=WORK_DIR / \"scratch\") local_shared_scratch.add_file_servers(FileServer(url=\"file://\" + str(WORK_DIR / \"scratch\"), operation_type=Operation.ALL)) local_site.add_directories(local_shared_scratch) local_storage = Directory(directory_type=Directory.LOCAL_STORAGE, path=WORK_DIR / \"outputs\") local_storage.add_file_servers(FileServer(url=\"file://\" + str(WORK_DIR / \"outputs\"), operation_type=Operation.ALL)) local_site.add_directories(local_storage) local_site.add_env(PATH=os.environ[\"PATH\"]) sc.add_sites(local_site) # stash site (staging site, where intermediate data will be stored) stash_site = Site(name=\"stash\", arch=Arch.X86_64, os_type=OS.LINUX) stash_staging_path = \"/collab/user/{USER}/staging\".format(USER=getpass.getuser()) stash_shared_scratch = Directory(directory_type=Directory.SHARED_SCRATCH, path=stash_staging_path) stash_shared_scratch.add_file_servers( FileServer( url=\"stash:///osgconnect{STASH_STAGING_PATH}\".format(STASH_STAGING_PATH=stash_staging_path), operation_type=Operation.ALL) ) stash_site.add_directories(stash_shared_scratch) sc.add_sites(stash_site) # condorpool (execution site) condorpool_site = Site(name=\"condorpool\", arch=Arch.X86_64, os_type=OS.LINUX) condorpool_site.add_pegasus_profile(style=\"condor\") condorpool_site.add_condor_profile( universe=\"vanilla\", requirements=\"HAS_SINGULARITY == True && HAS_AVX2 == True\", request_cpus=1, request_memory=\"4 GB\", request_disk=\"10 GB\", ) condorpool_site.add_profiles( Namespace.CONDOR, key=\"+SingularityImage\", value='\"/cvmfs/singularity.opensciencegrid.org/chiehlin0212/koto-dev:latest\"' ) condorpool_site.add_profiles( Namespace.CONDOR, key=\"+ProjectName\", value='\"collab.KOTO\"' ) sc.add_sites(condorpool_site) # write SiteCatalog to ./sites.yml sc.write() Writes the file transformations.yml . This file specifies the executables used in the workflow and contains the locations where they are physically located. In this example, we only have run_koto.sh . # --- Transformations ---------------------------------------------------------- run_koto = Transformation( name=\"run_koto\", site=\"local\", pfn=TOP_DIR / \"bin/run_koto.sh\", is_stageable=True ).add_pegasus_profile(clusters_size=1) tc = TransformationCatalog() tc.add_transformations(run_koto) # write TransformationCatalog to ./transformations.yml tc.write() Writes the file replicas.yml . This file specifies the physical locations of any input files used by the workflow. In this example, there is an entry for each file in the inputs/ directory, and all files matching /collab/project/KOTO/external/run87/accidental/64kW/**/*.root # --- Replicas ----------------------------------------------------------------- rc = ReplicaCatalog() input_files = [] # all files in the inputs/ dir input_files = glob.glob(str(TOP_DIR / \"inputs/*\")) for f in input_files: lfn = os.path.basename(f) rc.add_replica(site=\"local\", lfn=lfn, pfn=f) # also add all wave files wave_files = glob.glob(\"/collab/project/KOTO/external/run87/accidental/64kW/**/*.root\", recursive=True) for f in wave_files: lfn = os.path.basename(f) rc.add_replica(site=\"local\", lfn=lfn, pfn=f) # write ReplicaCatalog to replicas.yml rc.write() Builds the wordfreq workflow and submits it for execution. When wf.plan is invoked, pegasus.properties , sites.yml , transformations.yml , and replicas.yml will be consumed as part of the workflow planning process. Note that in this step there is no mention of data movement and job details as these are added by Pegasus when the workflow is planned into an executable workflow. As part of the planning process, additional jobs which handle scratch directory creation, data staging, and data cleanup are added to the workflow. # --- Workflow ----------------------------------------------------------------- wf = Workflow(name=\"koto\") random.seed() for n in range(10): # arguments decay_mode = \"KL3pi0\" seed = str(random.randint(0, 10000)) wave = os.path.basename(random.choice(wave_files)) # TODO: check if we already have a job with the seed/wave combination print(\" ... added mc job with wave={} / seed={}\".format(wave, seed)) out_root = File(\"{}_BasicVetoAndCuts_Accidental_{}.root\".format(decay_mode, seed)) job = Job(run_koto)\\ .add_args(decay_mode, seed, wave)\\ .add_outputs(out_root) # inputs, only use the lfn for inp in input_files + [wave]: job.add_inputs(os.path.basename(inp)) wf.add_jobs(job) # plan and run the workflow wf.plan( dir=WORK_DIR / \"runs\", sites=[\"condorpool\"], staging_sites={\"condorpool\": \"stash\"}, output_sites=[\"local\"], cluster=[\"horizontal\"], submit=True ) Submit the workflow by executing workflow.py . $ ./workflow.py Note that when Pegasus plans/submits a workflow, a workflow directory is created and presented in the output. In the following example output, the workflow directory is /home/user/workflows/runs/pegasus/run0014 . 2020.12.18 14:33:07.059 CST: ----------------------------------------------------------------------- 2020.12.18 14:33:07.064 CST: File for submitting this DAG to HTCondor : koto-workflow-0.dag.condor.sub 2020.12.18 14:33:07.070 CST: Log of DAGMan debugging messages : koto-workflow-0.dag.dagman.out 2020.12.18 14:33:07.075 CST: Log of HTCondor library output : koto-workflow-0.dag.lib.out 2020.12.18 14:33:07.080 CST: Log of HTCondor library error messages : koto-workflow-0.dag.lib.err 2020.12.18 14:33:07.086 CST: Log of the life of condor_dagman itself : koto-workflow-0.dag.dagman.log 2020.12.18 14:33:07.091 CST: 2020.12.18 14:33:07.096 CST: -no_submit given, not submitting DAG to HTCondor. You can do this with: 2020.12.18 14:33:07.107 CST: ----------------------------------------------------------------------- 2020.12.18 14:33:11.352 CST: Your database is compatible with Pegasus version: 5.1.0dev [WARNING] Submitting to condor koto-workflow-0.dag.condor.sub 2020.12.18 14:33:12.060 CST: Time taken to execute is 5.818 seconds Your workflow has been started and is running in the base directory: /home/user/workflows/runs/koto-workflow/run0014 *** To monitor the workflow you can run *** pegasus-status -l /home/user/workflows/runs/koto-workflow/run0014 *** To remove your workflow run *** pegasus-remove /home/user/workflows/runs/koto-workflow/run0014 This directory is the handle to the workflow instance and is used by Pegasus command line tools. Some useful tools to know about: pegasus-status -v [wfdir] Provides status on a currently running workflow. ( more ) pegasus-analyzer [wfdir] Provides debugging clues why a workflow failed. Run this after a workflow has failed. ( more ) pegasus-statistics [wfdir] Provides statistics, such as walltimes, on a workflow after it has completed. ( more ) pegasus-remove [wfdir] Removes a workflow from the system. ( more )","title":"KOTO"},{"location":"projects/koto/#koto","text":"KOTO is an international collaboration of an experiment based at the JPARC HEP facility in Japan.","title":"KOTO"},{"location":"projects/koto/#access-point","text":"KOTO submits out of the Connect node: login.collab.ci-connect.net. To request an account, visit the portal at https://ci-connect.net and ask to join the PATh Collaborations group. KOTO is a subgroup under PATh collaborations and a separate request needs to be made to join by clicking the request memberhip button.","title":"Access Point"},{"location":"projects/koto/#storage-access","text":"Besides their home directories on login.collab.ci-connect.net, users have access to local storage in /scratch/ and distributed storage in /collab/user/ . In addition, the collaboration has access to /collab/project/KOTO for sharing data products between the users. The distributed storage is http accessible: http://stash.osgstorage.org/collab/KOTO and data can be downloaded using a browser or command line utilities like wget or curl.","title":"Storage access"},{"location":"projects/koto/#jobs-to-the-ospool","text":"Inlined below is a submit script to run a KOTO job on the OSPool 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 #!/bin/bash Universe = Vanilla Executable = run_koto.sh Requirements = HAS_SINGULARITY == TRUE +SingularityImage = \"/cvmfs/singularity.opensciencegrid.org/ppaschos/koto-dev:latest\" transfer_input_files = e14_201605.mac Error = output.err. $( Cluster ) - $( Process ) Output = output.out. $( Cluster ) - $( Process ) Log = output.log. $( Cluster ) should_transfer_files = YES WhenToTransferOutput = ON_EXIT request_cpus = 1 request_memory = 2GB request_disk = 2GB +ProjectName = \"collab.KOTO\" Queue 1 KOTO uses a software stack that is included in the Singularity image - shown in the script above. The image is deployed on the remote execution nodes and jobs run within the Singularity container. The input file, e14_201605.mac, is sent to the execution node as part of the job. The execution script, run_koto.sh, sets the environment and includes all invocations to executables. The following script, run_koto.sh, runs a benchmark example from the E14 library. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 #!/bin/bash echo $HOSTNAME source /opt/koto/root/v6.22.02/bin/thisroot.sh source /opt/koto/geant4/10.05.p01/bin/geant4.sh echo \"PATH --> \" $PATH echo \"LD_LIBRARY_PATH --> \" $LD_LIBRARY_PATH echo \"ROOTSYS --> \" $ROOTSYS echo \"G4 data environent\" env | grep G4 export E14_TOP_DIR = \"/opt/koto/e14/201605v6.2/e14\" echo \"E14_TOP_DIR=\" $E14_TOP_DIR source /opt/koto/e14/201605v6.2/setup.sh output = \"output.root\" nevent = 1000 seed = 0 echo \"Running Code\" /opt/koto/e14/201605v6.2/e14/examples/gsim4test/bin/gsim4test e14_201605.mac ${ output } ${ nevent } ${ seed } ${ seed }","title":"Jobs to the OSPool"},{"location":"projects/koto/#pegasus-workflows","text":"The Pegasus project encompasses a set of technologies that help workflow-based applications execute in a number of different environments including desktops, campus clusters, grids, and clouds. Pegasus bridges the scientific domain and the execution environment by automatically mapping high-level workflow descriptions onto distributed resources. It automatically locates the necessary input data and computational resources necessary for workflow execution. Pegasus enables scientists to construct workflows in abstract terms without worrying about the details of the underlying execution environment or the particulars of the low-level specifications required by the middleware. Some of the advantages of using Pegasus include: Portability / Reuse - User created workflows can easily be run in different environments without alteration. Pegasus currently runs workflows on compute systems scheduled via HTCondor, including the OSPool, and other other systems or via other schedulers (e.g. ACCESS resources, Amazon EC2, Google Cloud, and many campus clusters). The same workflow can run on a single system or across a heterogeneous set of resources. Performance - The Pegasus mapper can reorder, group, and prioritize tasks in order to increase the overall workflow performance. Scalability - Pegasus can easily scale both the size of the workflow, and the resources that the workflow is distributed over. Pegasus runs workflows ranging from just a few computational tasks up to 1 million tasks. The number of resources involved in executing a workflow can scale as needed without any impediments to performance. Provenance - By default, all jobs in Pegasus are launched via the kickstart process that captures runtime provenance of the job and helps in debugging. The provenance data is collected in a database, and the data can be summarized with tools such as pegasus-statistics or directly with SQL queries. Data Management - Pegasus handles replica selection, data transfers and output registrations in data catalogs. These tasks are added to a workflow as auxiliary jobs by the Pegasus planner. Reliability - Jobs and data transfers are automatically retried in case of failures. Debugging tools such as pegasus-analyzer help the user to debug the workflow in case of non-recoverable failures. Error Recovery - When errors occur, Pegasus tries to recover when possible by retrying tasks, retrying the entire workflow, providing workflow-level checkpointing, re-mapping portions of the workflow, trying alternative data sources for staging data, and, when all else fails, providing a rescue workflow containing a description of only the work that remains to be done. Pegasus keeps track of what has been done (provenance) including the locations of data used and produced, and which software was used with which parameters. OSG has no read/write enabled shared file system across the resources. Jobs are required to either bring inputs along with the job, or as part of the job stage the inputs from a remote location. The following examples highlight how Pegasus can be used to manage KOTO workloads in such an environment by providing an abstraction layer around things like data movements and job retries, enabling the users to run larger workloads, spending less time developing job management tools and babysitting their computations. Pegasus workflows have 4 components: Site Catalog - Describes the execution environment in which the workflow will be executed. Transformation Catalog - Specifies locations of the executables used by the workflow. Replica Catalog - Specifies locations of the input data to the workflow. Workflow Description - An abstract workflow description containing compute steps and dependencies between the steps. We refer to this workflow as abstract because it does not contain data locations and available software. When developing a Pegasus Workflow using the Python API , all four components may be defined in the same file. For details, please refer to the Pegasus documentation .","title":"Pegasus Workflows"},{"location":"projects/koto/#koto-workflow-example","text":"The GitHub repo https://github.com/pegasus-isi/koto-pegasus contains a sample Pegasus workflow for KOTO processing. It is a very simple Monte Carlo example, containing just N independent jobs: This example is using OSG StashCache for data transfers. Credentials are transparant to the end users. Additionally, this example uses a custom container to run jobs. This ensures a consistent and complete environment for the jobs. Which container to use is defined in workflow.py with the +SingularityImage attribute. When invoked, the workflow script ( workflow.py ) does the following: Writes the file pegasus.properties . This file contains configuration settings used by Pegasus and HTCondor. # --- Properties --------------------------------------------------------------- props = Properties() props[\"pegasus.data.configuration\"] = \"nonsharedfs\" # Provide a full kickstart record, including the environment, even for successful jobs props[\"pegasus.gridstart.arguments\"] = \"-f\" # Limit the number of idle jobs for large workflows props[\"dagman.maxidle\"] = \"1000\" # use timestamps for all our dirs - this make sure we have unique names in stash props[\"pegasus.dir.useTimestamp\"] = \"true\" # seperate output dir for each workflow run props[\"pegasus.dir.storage.deep\"] = \"true\" # Help Pegasus developers by sharing performance data (optional) props[\"pegasus.monitord.encoding\"] = \"json\" props[\"pegasus.catalog.workflow.amqp.url\"] = \"amqp://friend:donatedata@msgs.pegasus.isi.edu:5672/prod/workflows\" props[\"pegasus.metrics.app\"] = \"KOTO\" Writes the file sites.yml . This file describes the execution environment in which the workflow will be run. # --- Sites -------------------------------------------------------------------- sc = SiteCatalog() # local site (submit machine) local_site = Site(name=\"local\") local_shared_scratch = Directory(directory_type=Directory.SHARED_SCRATCH, path=WORK_DIR / \"scratch\") local_shared_scratch.add_file_servers(FileServer(url=\"file://\" + str(WORK_DIR / \"scratch\"), operation_type=Operation.ALL)) local_site.add_directories(local_shared_scratch) local_storage = Directory(directory_type=Directory.LOCAL_STORAGE, path=WORK_DIR / \"outputs\") local_storage.add_file_servers(FileServer(url=\"file://\" + str(WORK_DIR / \"outputs\"), operation_type=Operation.ALL)) local_site.add_directories(local_storage) local_site.add_env(PATH=os.environ[\"PATH\"]) sc.add_sites(local_site) # stash site (staging site, where intermediate data will be stored) stash_site = Site(name=\"stash\", arch=Arch.X86_64, os_type=OS.LINUX) stash_staging_path = \"/collab/user/{USER}/staging\".format(USER=getpass.getuser()) stash_shared_scratch = Directory(directory_type=Directory.SHARED_SCRATCH, path=stash_staging_path) stash_shared_scratch.add_file_servers( FileServer( url=\"stash:///osgconnect{STASH_STAGING_PATH}\".format(STASH_STAGING_PATH=stash_staging_path), operation_type=Operation.ALL) ) stash_site.add_directories(stash_shared_scratch) sc.add_sites(stash_site) # condorpool (execution site) condorpool_site = Site(name=\"condorpool\", arch=Arch.X86_64, os_type=OS.LINUX) condorpool_site.add_pegasus_profile(style=\"condor\") condorpool_site.add_condor_profile( universe=\"vanilla\", requirements=\"HAS_SINGULARITY == True && HAS_AVX2 == True\", request_cpus=1, request_memory=\"4 GB\", request_disk=\"10 GB\", ) condorpool_site.add_profiles( Namespace.CONDOR, key=\"+SingularityImage\", value='\"/cvmfs/singularity.opensciencegrid.org/chiehlin0212/koto-dev:latest\"' ) condorpool_site.add_profiles( Namespace.CONDOR, key=\"+ProjectName\", value='\"collab.KOTO\"' ) sc.add_sites(condorpool_site) # write SiteCatalog to ./sites.yml sc.write() Writes the file transformations.yml . This file specifies the executables used in the workflow and contains the locations where they are physically located. In this example, we only have run_koto.sh . # --- Transformations ---------------------------------------------------------- run_koto = Transformation( name=\"run_koto\", site=\"local\", pfn=TOP_DIR / \"bin/run_koto.sh\", is_stageable=True ).add_pegasus_profile(clusters_size=1) tc = TransformationCatalog() tc.add_transformations(run_koto) # write TransformationCatalog to ./transformations.yml tc.write() Writes the file replicas.yml . This file specifies the physical locations of any input files used by the workflow. In this example, there is an entry for each file in the inputs/ directory, and all files matching /collab/project/KOTO/external/run87/accidental/64kW/**/*.root # --- Replicas ----------------------------------------------------------------- rc = ReplicaCatalog() input_files = [] # all files in the inputs/ dir input_files = glob.glob(str(TOP_DIR / \"inputs/*\")) for f in input_files: lfn = os.path.basename(f) rc.add_replica(site=\"local\", lfn=lfn, pfn=f) # also add all wave files wave_files = glob.glob(\"/collab/project/KOTO/external/run87/accidental/64kW/**/*.root\", recursive=True) for f in wave_files: lfn = os.path.basename(f) rc.add_replica(site=\"local\", lfn=lfn, pfn=f) # write ReplicaCatalog to replicas.yml rc.write() Builds the wordfreq workflow and submits it for execution. When wf.plan is invoked, pegasus.properties , sites.yml , transformations.yml , and replicas.yml will be consumed as part of the workflow planning process. Note that in this step there is no mention of data movement and job details as these are added by Pegasus when the workflow is planned into an executable workflow. As part of the planning process, additional jobs which handle scratch directory creation, data staging, and data cleanup are added to the workflow. # --- Workflow ----------------------------------------------------------------- wf = Workflow(name=\"koto\") random.seed() for n in range(10): # arguments decay_mode = \"KL3pi0\" seed = str(random.randint(0, 10000)) wave = os.path.basename(random.choice(wave_files)) # TODO: check if we already have a job with the seed/wave combination print(\" ... added mc job with wave={} / seed={}\".format(wave, seed)) out_root = File(\"{}_BasicVetoAndCuts_Accidental_{}.root\".format(decay_mode, seed)) job = Job(run_koto)\\ .add_args(decay_mode, seed, wave)\\ .add_outputs(out_root) # inputs, only use the lfn for inp in input_files + [wave]: job.add_inputs(os.path.basename(inp)) wf.add_jobs(job) # plan and run the workflow wf.plan( dir=WORK_DIR / \"runs\", sites=[\"condorpool\"], staging_sites={\"condorpool\": \"stash\"}, output_sites=[\"local\"], cluster=[\"horizontal\"], submit=True ) Submit the workflow by executing workflow.py . $ ./workflow.py Note that when Pegasus plans/submits a workflow, a workflow directory is created and presented in the output. In the following example output, the workflow directory is /home/user/workflows/runs/pegasus/run0014 . 2020.12.18 14:33:07.059 CST: ----------------------------------------------------------------------- 2020.12.18 14:33:07.064 CST: File for submitting this DAG to HTCondor : koto-workflow-0.dag.condor.sub 2020.12.18 14:33:07.070 CST: Log of DAGMan debugging messages : koto-workflow-0.dag.dagman.out 2020.12.18 14:33:07.075 CST: Log of HTCondor library output : koto-workflow-0.dag.lib.out 2020.12.18 14:33:07.080 CST: Log of HTCondor library error messages : koto-workflow-0.dag.lib.err 2020.12.18 14:33:07.086 CST: Log of the life of condor_dagman itself : koto-workflow-0.dag.dagman.log 2020.12.18 14:33:07.091 CST: 2020.12.18 14:33:07.096 CST: -no_submit given, not submitting DAG to HTCondor. You can do this with: 2020.12.18 14:33:07.107 CST: ----------------------------------------------------------------------- 2020.12.18 14:33:11.352 CST: Your database is compatible with Pegasus version: 5.1.0dev [WARNING] Submitting to condor koto-workflow-0.dag.condor.sub 2020.12.18 14:33:12.060 CST: Time taken to execute is 5.818 seconds Your workflow has been started and is running in the base directory: /home/user/workflows/runs/koto-workflow/run0014 *** To monitor the workflow you can run *** pegasus-status -l /home/user/workflows/runs/koto-workflow/run0014 *** To remove your workflow run *** pegasus-remove /home/user/workflows/runs/koto-workflow/run0014 This directory is the handle to the workflow instance and is used by Pegasus command line tools. Some useful tools to know about: pegasus-status -v [wfdir] Provides status on a currently running workflow. ( more ) pegasus-analyzer [wfdir] Provides debugging clues why a workflow failed. Run this after a workflow has failed. ( more ) pegasus-statistics [wfdir] Provides statistics, such as walltimes, on a workflow after it has completed. ( more ) pegasus-remove [wfdir] Removes a workflow from the system. ( more )","title":"KOTO workflow example"},{"location":"projects/lsst/","text":"LSST via Duke \u00b6 LSST is supported by OSG Collaboration Services via the duke.lsst osg project Sign up to get access to the OSG submit node here: Duke CI-Connect Access Portal News \u00b6 You can now use stashcp to transfer data to and from OSG as part of your computational workflow. You must transfer your input files to a new namespace. From your active session on the duke login node the namespace is located in: /collab/users and /collab/project/lsst Users can create their own private directories in users or shared directories in lsst. New accounts will create the user directories automatically and provide read,write permissions in the project/lsst folder To transfer files to a remote site via a job enter this stashcp command in your execution script: stashcp /osgconnect/collab/users/ / . To transfer files, such as output, from the remote site where your jobs is running to your space on stash, enter this command in your execution script stashcp stash:///osgconnect/collab/users/ /.","title":"LSST"},{"location":"projects/lsst/#lsst-via-duke","text":"LSST is supported by OSG Collaboration Services via the duke.lsst osg project Sign up to get access to the OSG submit node here: Duke CI-Connect Access Portal","title":"LSST via Duke"},{"location":"projects/lsst/#news","text":"You can now use stashcp to transfer data to and from OSG as part of your computational workflow. You must transfer your input files to a new namespace. From your active session on the duke login node the namespace is located in: /collab/users and /collab/project/lsst Users can create their own private directories in users or shared directories in lsst. New accounts will create the user directories automatically and provide read,write permissions in the project/lsst folder To transfer files to a remote site via a job enter this stashcp command in your execution script: stashcp /osgconnect/collab/users/ / . To transfer files, such as output, from the remote site where your jobs is running to your space on stash, enter this command in your execution script stashcp stash:///osgconnect/collab/users/ /.","title":"News"},{"location":"projects/moller/","text":"MOLLER \u00b6 The Measurement of a Lepton-Lepton Electroweak Reaction is a proposed high energy physics experiment at JLAB. A pilot project for jobs at the Open Science fabric is already in place on the OSG Connect infrastructure. The main focus for the collaboration at the moment is to carry out extensive simulations that will further validate the detector design. There is no overall project or data management person for the Moller experiment at this point. The PI is Prof Wouter Deconick Wouter.Deconinck@umanitoba.ca","title":"MOLLER"},{"location":"projects/moller/#moller","text":"The Measurement of a Lepton-Lepton Electroweak Reaction is a proposed high energy physics experiment at JLAB. A pilot project for jobs at the Open Science fabric is already in place on the OSG Connect infrastructure. The main focus for the collaboration at the moment is to carry out extensive simulations that will further validate the detector design. There is no overall project or data management person for the Moller experiment at this point. The PI is Prof Wouter Deconick Wouter.Deconinck@umanitoba.ca","title":"MOLLER"},{"location":"projects/project-list/","text":"List of Supported Projects \u00b6 IGWN(LIGO,VIRGO,KAGRA) IceCube REDTOP South Pole Telescope (SPT3g) CMBS4 KOTO XENON Futurecolliders CLAS12 GLUEX EIC/AID2E EIC/ePIC MOLLER HepSim LSST DES Event Horizon Telescope DUNE UHE Neutrino Obsevatory Trinity EUSO-SPB2 VERITAS Other Fermi Lab Collaborations","title":"Overview"},{"location":"projects/project-list/#list-of-supported-projects","text":"IGWN(LIGO,VIRGO,KAGRA) IceCube REDTOP South Pole Telescope (SPT3g) CMBS4 KOTO XENON Futurecolliders CLAS12 GLUEX EIC/AID2E EIC/ePIC MOLLER HepSim LSST DES Event Horizon Telescope DUNE UHE Neutrino Obsevatory Trinity EUSO-SPB2 VERITAS Other Fermi Lab Collaborations","title":"List of Supported Projects"},{"location":"projects/snowmass/","text":"Snowmass21 \u00b6 The Snowmass21 effort is supported with a submit/analysis node on the Connect infrastructure, local and distributed storage, a CVMFS data origin at UChicago, and managed singularity containers for software distribution. The Snowmass effort currently has contributions from more than 50 campuses and institutions across the world with more than 150 users with access to the Snowmass21 submit host with 180 TB of local storage and 50 TB of distributed storage on stash. Data production and analysis lead: John Stupak john.stupak@ou.edu Snowmass has a dedicated operational page found here https://maniaclab.uchicago.edu/snowmass-connect-docs/","title":"Snowmass"},{"location":"projects/snowmass/#snowmass21","text":"The Snowmass21 effort is supported with a submit/analysis node on the Connect infrastructure, local and distributed storage, a CVMFS data origin at UChicago, and managed singularity containers for software distribution. The Snowmass effort currently has contributions from more than 50 campuses and institutions across the world with more than 150 users with access to the Snowmass21 submit host with 180 TB of local storage and 50 TB of distributed storage on stash. Data production and analysis lead: John Stupak john.stupak@ou.edu Snowmass has a dedicated operational page found here https://maniaclab.uchicago.edu/snowmass-connect-docs/","title":"Snowmass21"},{"location":"projects/spt/","text":"South Pole Telescope SPT-3G \u00b6 Support areas for SPT include data management, infrastructure (Submit host, Backups to Federated Facilities, Globus Service, dCache and local storage at UChicago for experimental data from the South Pole and production-quality data from OSG), and best practices for software and job management, SPT is a collaboration that involves over 30 University campuses in the US along with National Laboratories. Examples of main contributions come from the University of Chicago, University of California Berkeley and Davis, the University of Illinois at Urbana/Champaign, Michigan State, and Argonne and Fermi National Labs. SPT-3G is supported by the NSF award # 1852617 (South Pole Telescope Operations and Data Products). Data production and analysis Lead: Sasha Rahlin arahlin@fnal.gov Data management coordination: Tom Crawford tcrawfor@kicp.uchicago.edu Account Portal and sign up for Access to Resources SPT Local Storage Dashboard SPT Grid Storage Dashboard Information fo Users \u00b6 Guide to access OSG resources Guid to obtain Grid Certificates General Information on Software availability and documenation \u00b6","title":"South Pole Telescope"},{"location":"projects/spt/#south-pole-telescope-spt-3g","text":"Support areas for SPT include data management, infrastructure (Submit host, Backups to Federated Facilities, Globus Service, dCache and local storage at UChicago for experimental data from the South Pole and production-quality data from OSG), and best practices for software and job management, SPT is a collaboration that involves over 30 University campuses in the US along with National Laboratories. Examples of main contributions come from the University of Chicago, University of California Berkeley and Davis, the University of Illinois at Urbana/Champaign, Michigan State, and Argonne and Fermi National Labs. SPT-3G is supported by the NSF award # 1852617 (South Pole Telescope Operations and Data Products). Data production and analysis Lead: Sasha Rahlin arahlin@fnal.gov Data management coordination: Tom Crawford tcrawfor@kicp.uchicago.edu Account Portal and sign up for Access to Resources SPT Local Storage Dashboard SPT Grid Storage Dashboard","title":"South Pole Telescope SPT-3G"},{"location":"projects/spt/#information-fo-users","text":"Guide to access OSG resources Guid to obtain Grid Certificates General Information on Software availability and documenation","title":"Information fo Users"},{"location":"projects/spt/#_1","text":"","title":""},{"location":"projects/veritas/","text":"VERITAS Collaboration \u00b6","title":"VERITAS"},{"location":"projects/veritas/#veritas-collaboration","text":"","title":"VERITAS Collaboration"},{"location":"projects/xenon/","text":"XENON \u00b6 Support areas for XENON include data management, infrastructure (Submit host, dCache storage, FTS service for Rucio managed third-party transfers, Rucio Service), and best practices for software and job management. XENON is supported by NSF institutional grants PHY-2112796 at UChicago, PHY-2112851 at Columbia University, PHY-2112801 at Rice University, PHY-2112802 at UC San Diego and PHY-21128-3 at Purdue University. Data Production team Lead: Evan Shockley eshockley@physics.ucsd.edu MC Production team Lead: Diego Ramirez diego.ramirez@physik.uni-freiburg.de At present, XENON consumes ~6 PB of distributed storage f in the US & Europe managed by Rucio. Total storage utilization has remained constant in time, as the collaboration is deleting older data - from XENON 1T - while it continues to gather fresh data from the current phase - XENON nT. Nonetheless, the overall replicated distributed storage is expected to increase by the end of the 5-year lifetime of the experiment because the amount of data volume for the nT phase is projected to be five times larger than the 1T volume. Status of Services \u00b6 Rucio Server Ganglia FTS Server Ganglia Check-mk Xenon Rucio Group Rucio Postgress XENON login node Weekly XENON analysis/Compute call \u00b6 URL: https://riceuniversity.zoom.us/j/4815153031 Minutes:https://xe1t-wiki.lngs.infn.it/doku.php?id=xenon:xenon1t:analysistools:200402","title":"XENON"},{"location":"projects/xenon/#xenon","text":"Support areas for XENON include data management, infrastructure (Submit host, dCache storage, FTS service for Rucio managed third-party transfers, Rucio Service), and best practices for software and job management. XENON is supported by NSF institutional grants PHY-2112796 at UChicago, PHY-2112851 at Columbia University, PHY-2112801 at Rice University, PHY-2112802 at UC San Diego and PHY-21128-3 at Purdue University. Data Production team Lead: Evan Shockley eshockley@physics.ucsd.edu MC Production team Lead: Diego Ramirez diego.ramirez@physik.uni-freiburg.de At present, XENON consumes ~6 PB of distributed storage f in the US & Europe managed by Rucio. Total storage utilization has remained constant in time, as the collaboration is deleting older data - from XENON 1T - while it continues to gather fresh data from the current phase - XENON nT. Nonetheless, the overall replicated distributed storage is expected to increase by the end of the 5-year lifetime of the experiment because the amount of data volume for the nT phase is projected to be five times larger than the 1T volume.","title":"XENON"},{"location":"projects/xenon/#status-of-services","text":"Rucio Server Ganglia FTS Server Ganglia Check-mk Xenon Rucio Group Rucio Postgress XENON login node","title":"Status of Services"},{"location":"projects/xenon/#weekly-xenon-analysiscompute-call","text":"URL: https://riceuniversity.zoom.us/j/4815153031 Minutes:https://xe1t-wiki.lngs.infn.it/doku.php?id=xenon:xenon1t:analysistools:200402","title":"Weekly XENON analysis/Compute call"}]}
\ No newline at end of file
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index 3d97505..dbb3b19 100644
Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ