CHANGES

Complete change log for the Pilot 1
-----------------------------------


73.0

Processes
- Some minor cleanup; removed unused printProcessTree() + import of subprocess (processes)
- Corrected log message for SIGKILL in killProcesses() (processes)
- Added missing variable in log message in getMaxMemoryUsageFromCGroups() (processes)

Containers
- Added removeContainerImage() which removes any --containerimage XYZ options from job parameters (feature only supported in Pilot 2) (SiteInformation)

Timing info
- Added t0 to Job class (Job)
- Added getOsTimesTuple() (FileHandling)
- Writing t0 to file in executePayload() (RunJob)
- Setting CPU consumption time in __updateJobs() (i.e. just before heartbeat update) (Monitor)
- Using get_instant_cpu_consumption_time() in __updateJobs() for checking CPU consumption on the fly (Monitor)
- Updated code to use get_cpu_consumption_time() instead of setTimeConsumed() (RunJob*)
- Removed unused setTimeConsumed() (pUtil)
- Creating t0_times.txt file also in RunJobEvent (RunJobEvent)
- Writing and reading child pid to/from file (RunJob, RunJobEvent, Monitor)
- Created get_current_cpu_consumption_time() (processes)

Tracing
- Removed any present escape characters in the stateReason field in sendTrace() (mover)
( out-commented for now, take back in release 73.1+
- Switched off sending tracing report if rucio copytool is used (and back on if necessary), in stagein_real(), stageout() (mover)
)

Google related fix
- Added https:// protocol to allowed_replica_schemas in is_directaccess() (Job)
- Added escape character for & (needed for google turls); replace with & (happens in google turls) in createPoolFileCatalog() (pUtil)

LAPP debugging
- Added command output to log (detailed rucio output), in stageIn() (rucio_sitemover)

Updates from Nicolo Magini:

  https://github.com/PanDAWMS/pilot/pull/216

Commit Summary

Use killpg instead of kill to timeout also all child processes

File Changes

M TimerCommand.py (6)

Updates from Wen Guan:

  https://github.com/PanDAWMS/pilot/pull/215

Commit Summary

allow esmerge to select closest inputs

File Changes

M movers/rucio_sitemover.py (7)

1. on line 314, to allow remote replicas if fdata.storageId is defined (only for esmerge jobs the storageId is defined).
Normally if there is no replication rule, es unmerge files will only be stored at one place. This modification will not
change anything. If es unmerge fiels are replicated to somewhere else, this update will allow esmerge job to read from
all available places (currently this situation will only happen on NERSC es jobs).
2. fix the checksum and filesize warning messages(it only changes the log messages).

b8e5d35 allow esmerge to read from different place if unmerge file is replaced

1. file.replicas is None but we still try to stagein. It's not normal jobs(for normal jobs if file.replicas is None,
the job will fail with no replicas.). It can only happen for ES merge jobs.
2. isDeterministic is True. It means it's not objectstore. Currently it's only NERSC_DATADISK. Doug wants to move the
es outputs from NERSC_DATADISK to somewhere else and then the esmerge job can stagein from that place(currently es
merge job can run at CERN or somewhere, and remotely stagein from NERSC_DATADISK).

Mario: This will allow to pull in data from anywhere in the world

  https://github.com/PanDAWMS/pilot/pull/217

Commit Summary

fix localSite
File Changes

M movers/mover.py (12)

Updates from Mario Lassnig:

  https://github.com/PanDAWMS/pilot/pull/214

Commit Summary

allow https URLs for Google SE

File Changes

M movers/mover.py (2)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.1

Log tailing (requested by R. Walker)
- Created get_files(), removed old implementation of tail(), added a new version of tail() [FileHandling]
- Now adding the tail of the latest found log file in the payload workdir (debug mode), in __updateJobs() [Monitor]

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.2

list_replicas()
- Specifying --pfn in rucio download, stageIn(), which will prevent list_replicas() from being used on server side (rucio_sitemover)

Overwritten "Payload exceeded max allowed memory" fix
- Now setting ERR_PAYLOADEXCEEDMAXMEM if MEMORYEXCEEDED file detected at the end of interpretPayloadStdout() to prevent
  signal error from being set instead. Requested by R. Walker (ATLASExperiment)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.3

Large log files
- It was reported in JIRA ticket https://its.cern.ch/jira/browse/ADCSUPPORT-5078 that once again log files are too big, this time due to new sub directory _joproxy15
- Added _joproxy15 to list of redundant directories in removeRedundantFiles() (ATLASExperiment)

Updates from A. Anisenkov:

  https://github.com/PanDAWMS/pilot/pull/220

* movers stage-in: manually sort replicas according to priority values exposed by Rucio
* fixes applied to mover.py file can be reverted once Rucio server-side patch will be delivered

Commit Summary

movers: fixed logic to properly consider priority of replica protocols for stage-in
typo fix
fix
cosmetic
File Changes

M movers/base.py (27)
M movers/mover.py (12)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.4

Code cleanup
- Removed deprecated functions: getTURLFileInfoDic(), updatePFN(), getDefaultStorage() (Mover)

Process kills
- Switched to using killpg() instead of kill -9 in killOrphans(), as suggested by P. McGuigan while
  debugging left processes after looping job kill (processes)

Log message fix
- Corrected % -> %s in log message in stageIn() (base)

Updates from W. Guan:

  https://github.com/PanDAWMS/pilot/pull/221

Commit Summary

fix for es merge stagein

File Changes

M movers/mover.py (17)

  https://github.com/PanDAWMS/pilot/pull/224

Commit Summary

verify objectstore stagein

File Changes

M movers/base.py (12)
M movers/objectstore_sitemover.py (23)
M movers/rucio_sitemover.py (16)

Updates from A. Anisenkov:

  https://github.com/PanDAWMS/pilot/pull/222

Commit Summary

movers: resolve_replica logic updated to look up fist protocol allowed for direct access mode for files requested direct read
cosmetic

File Changes

M movers/base.py (6)
M movers/mover.py (7)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.5

Bug fixes:
- Now making sure that stdout_tail is not None before trying to use it in updatePandaServer() (PandaServerClient)
- Added safer handling of stdout_tail variable to prevent discovered lost heartbeat as suggested in GGUS ticket https://ggus.eu/?mode=ticket_info&ticket_id=137637,
  in updatePandaServer() (PandaServerClient)
- Stopped setting PQ as default local/remoteSite in Rucio traces Created getRSE(ddmEndPoints) (Mover)
- Added missing string to int conversion in os.killpg() call for orphaned processes (processes)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.6

New error code
- ERR_BADXML = 1247, "Badly formed XML" - set if the PoolFileCatalog.xml contains illegal characters, as happened
  in recent user job (RunJob)

Replicas
- Now always setting geoip in list_replicas() call, resolve_replicas() (mover)

Log file creation, requested by R. Walker:
- Added HAHM_* to removeRedundantFiles() (ATLASExperiment)
- Added --one-file-system to the tar command in createLogFile() (JobLog)

UCORE scaling correction, requested by R. Walker:
- Now scaling memory limit 2*maxRSS with job.coreCount/schedconfig.corecount, in __check_memory_usage() (Monitor)

Support for new prodSourceLabel, requested by A. da Silva et al:
- Added -i ALRB and prodSourceLabel=rc_alrb to getProdSourceLabel() (pilot)

Updates from Tomas Javurek: (reverted)

  https://github.com/PanDAWMS/pilot/pull/227

Commit Summary

ruciomover: api only from now on, traces off

File Changes

M movers/rucio_sitemover.py (100)

Updates from P. Svirin:

  https://github.com/PanDAWMS/pilot/pull/226

Commit Summary

Added RUCIO tracing

File Changes

M movers/mover.py (10)
M movers/rucio_sitemover.py (19)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.7:

Rucio mover:
- Replaced rucio mover with version from Tomas's github repo
- Removed code that prevents tracing from being used when rucio copytool is used (Mover)

writeToFile:
- Now updating LFNs to TURLs in writeToFile input file list in direct access mode (RunJob)

Updates from Tomas Javurek:

https://github.com/PanDAWMS/pilot/pull/229/

[missing comments; ask Tomas]

Comments added by Paul:
1) Again using Rucio API for file transfers
2) Now verifying that the replica is present at the destination
3) Now using transfer time-out based on file size

Updates from Frank Berghaus:

Test jobs running against the Dynafed at CERN attempting to benchmark its performance failed because the pilot does not
allow dav for direct access. We would like to proceed with these test, also for benchmarking WebDAV against XRootD in
general. For this we would like to allow direct access with davs.

From the ROOT perspective both WebDAV and HTTP are implemented by TDavixFile. Both protocols should be allowed for direct access on LAN.

You can view, comment on, or merge this pull request online at:

  https://github.com/PanDAWMS/pilot/pull/232

Commit Summary

Allow direct access on WebDAV for local access

File Changes

M Job.py (2)
M movers/mover.py (2)

Updates from Wen Guan:

You can view, comment on, or merge this pull request online at:

  https://github.com/PanDAWMS/pilot/pull/233

Commit Summary

switch to use different ddm protocols based on retries
fix issues caused by failing update eventranges

File Changes

M Job.py (3)
M RunJobEvent.py (27)
M movers/mover.py (14)
M movers/objectstore_sitemover.py (15)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.8:

Davs in direct access
- Removed davs protocol from allowed direct access schemas (Job, mover)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.9:

Patch for an unhandled exception in rucio mover stageOut() function leading to a false-positive situation where a failed
stage-out was reported as successful (rucio_sitemover)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.10:

Local monitoring
- Avoid port 9256 in TCP server code since it is used for local monitoring (Requested by A. de Salvo) (PilotTCPServer)

Updates from Wen Guan:

  https://github.com/PanDAWMS/pilot/pull/237

Commit Summary

disable verify for objectstore stageout

File Changes

M movers/objectstore_sitemover.py (8)
M movers/rucio_sitemover.py (10)

Updates from Tomas Javurek:

  https://github.com/PanDAWMS/pilot/pull/238

Commit Summary

switching off timeouts since they appear to cause a problem for the Rucio API

File Changes

M movers/rucio_sitemover.py (8)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.11:

Benchmarking bug fix
- Added missing break after benchmark termination. Previously benchmark command could not be terminated if they ran
  for longer than the allowed time (RunJob)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.12:

Checksum verification update
- For stage-out checksum verification the pilot now uses checksum value from XML which is calculated right after the
  payload finishes rather than the value recalculated after the stage-out has finished. This optimization will discover
  any file corruptions that happen between end of payload execution and end of stage-out.
- Copied get_metadata_from_xml() from Pilot 2 (pUtil)
- stageout() now populates fspec object with checksum value from XML (mover)

Updates from Wen Guan:

  https://github.com/PanDAWMS/pilot/pull/240

Commit Summary

report corrupted es files
Merge branch 'main-dev' of ssh://github.com/PanDAWMS/pilot into main-dev

File Changes

M Job.py (3)
M PandaServerClient.py (4)
M RunJob.py (18)
M RunJobUtilities.py (4)
M UpdateHandler.py (4)

Updates from Tomas Javurek

  https://github.com/PanDAWMS/pilot/pull/239

This change was requested by Ivan and Rod. I implemented logger in silly way before, when every exception, that a log from rucio wasn't propagated. With this change, it should be streamed directly.

Commit Summary

rucio mover: proper logger handler in place of flushing
rucio mover: proper logger handler in place of flushing, II
rucio mover: proper logger handler in place of flushing, III
rucio mover: proper logger handler in place of flushing, IV

File Changes

M movers/rucio_sitemover.py (72)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.13

Nordugrid patch
- Moved back offending rucio imports from module header to where they are used (rucio_sitemover)

XML update:
- Always calling createMetadataForOutput() in postJobTask(), not only in non-failed jobs as before, to get a set strXML (JobLog)

Exception handling:
- Added exception handling to findProcessesInGroup() to avoid crash when ps command has failed (-> invalid literal for int())

Debug update
- Now calling get_files with different file name pattern in case TRF produced logs with name log.* instead of *.log, __updateJobs() (Monitor)
- Corrected 'False' -> False for debug variable (Job)

Updates from Wen Guan:

  https://github.com/PanDAWMS/pilot/pull/241

Commit Summary

fix bug in corruptedfiles

File Changes

M RunJob.py (16)
Patch Links:

https://github.com/PanDAWMS/pilot/pull/241.patch
https://github.com/PanDAWMS/pilot/pull/241.diff

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.14

IPv6
- New version of Rucio function detect_client_location() that is IPv6 compatible (pilot)
- Deprecated detect_client_location() in pilot, updated duplicated function in mover code (pilot, mover)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.15

User defined containers
- Discarding any @-syntax in cmtconfig if present (used for user defined containers; ALRB_USER_PLATFORM encoded in cmtconfig) (Job)

IPv6
- Some cleanup related to the detect_client_location() function (pilot, mover)

Updates from Alexey Anisenkov:

  https://github.com/PanDAWMS/pilot/pull/244

Commit Summary

detect_location: pass ATLASSite name as site value for client location to Rucio.resolve_replica()

File Changes

M movers/mover.py (8)


////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.16

Looping job update:
- Fixing a bug in __loopingJobKiller() which now identifies the most recently modified file and sets lastTimeFilesWereModified which will be compared to the current
  time (Monitor)
- NOTE: update get_time_for_last_touch() in Pilot 2 as well

Unset variables bug fix:
- Removed usages of globalJob and globalSite in monitor_job(), discovered by P. Svirin (Monitor)

Wrong architecture:
- Added new error code 1248, ERR_WRONGARCHITECTURE: "Job built on wrong architecture" (PilotErrors)
- Added new error code 1249, ERR_RUNGENFAILURE: "RunGen failure (consult log file)" (PilotErrors)
- Refactored interpretPayloadStdout(); created isSQLiteLockingProblem(), isBuiltOnWrongArchitecture() (ATLASExperiment)

maxwdir:
- Added a 10% grace margin to size checks against maxwdir, __checkWorkDir() (Monitor)
- Discussed in JIRA ticket https://its.cern.ch/jira/browse/ATLASPANDA-482

getJobStatus:
- Fixed undefined variable in exception handling in getJobStatus() (pUtil)

No shell code returned for proxy error
- Added NOPROXY error code to list in shellExitCode (pUtil)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.17

Exceptions in looping job killer
- Protection for exception in find_latest_modified_file() if file no longer exists (Monitor)
- Created verified_files(), used by get_files() and find_latest_modified_file() (FileHandling)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

73.18

Job log
- Added merged_lhef._0.events-new and Process to redundant files list, to be removed before job log is created (ATLASExperiment)

Looping job killer
- Changed from getctime() to getmtime() in find_latest_modified_file() (FileHandling)
- Same change in __updateJobs() where the function is used to find the latest updated log file (Monitor)

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

TODO:

todo: remove the explicit usages of schedconfig.lfchost and replace with an experiment specific method (getFileCatalog())
todo: rename pUtil.getExperiment to pUtil.getExperimentObject, correct import in SiteInformation

#### add new error codes 1217-1219 to proddb
Update prodDB for ERR_RUNJOBEXC : "Exception caught by runJob" -> "Exception caught by RunJob*" ? not necessary??
Added new error codes; 1224 (ERR_ESRECOVERABLE), 1225 (ERR_ESMERGERECOVERABLE) (PilotErrors)