Skip to content

Brainstorm meeting (Aug 19th 2020)

Jump to bottom
Kenneth Hoste edited this page Aug 19, 2020 · 7 revisions
  • Wed Aug 19th 2020, 11am CEST
  • attending: Kenneth Hoste, Bob Droge, Peter Stol, Carlos Fenoy

Recent progress

  • filesystem layer
    • conf call with CVMFS developers (see https://github.com/EESSI/meetings/wiki/meeting-CernVM-FS-Aug-17th-2020)
    • organize catalog stuff for compat layer?
    • figure out the whole publishing procedure for compat/software layers
      • focus on publisher/gateway mechanism (even though it's not 100% stable according to CernVM-FS devs)
    • updates to compat layer were still done directly on Stratum-0 for now
      • but should also work with publisher (see Ansible playbooks)
  • software layer
    • binutils: going forward with leveraging binutils from compat layer
      • no technical reason to use binutils provided through EasyBuild
      • especially since building binutils in Prefix env with EasyBuild is painful
      • also what Compute Canada does
      • binutils in Prefix is also very recent (close to latest)
      • doesn't really limit us: if needed we can shadow binutils from Prefix through EB
    • Caspar/Kenneth still playing around in their own Prefix setup
      • will apply that in pilot repo once they're happy with it

Packages

  • What is part of the compatibility layer and what is not, i.e. which things should (not) be included?
  • binutils: settled for now: using the one from Prefix (definitely for pilot setup)
  • balance between keeping minimal set of pkgs to make management easier vs security updates
    • in place updates may break stuff in compat layer (examples: Bison, flex, CMake)
  • documented decision of why packages are included in "set" files in compat layer
    • can have separate set for e.g. POWER or Arm systems
  • things we definitely need:
    • Python 3
    • Lmod (+ Lua pkgs)
    • binutils + GCC (incl. g++)
  • Singularity?
    • does that work with setuid binaries via CVMFS mount
    • security concerns?
    • open options w.r.t. containers
    • doesn't make sense to have in software layer (you really want in place updates for security reasons)
  • for Open MPI we probably don't need anything special
    • building Open MPI on top of UCX/PMIx/libfabric should support a wide range of network stacks
  • should we optimize packages installed in Prefix?
    • no, those packages should be as generic as possible (which is probably the default)
    • can test on old systems and maybe in QEMU environment emulating the most basic x86_86 CPU
  • in place updates to libc can cause trouble
    • cfr. problems with Intel compilers (segfaults) and recently broken GCC after updating from CentOS 8.1 to 8.2 (see ADD LINK HERE)
    • good reason to have separate test repo + snapshots so you can roll back transactions
    • also motivates the need for a test suite in software layer

Management

  • How do we manage this thing? Who can modify it and how?
    • work through pull request in EESSI/compat-layer
    • workflow
      • make local changes through overlay FS + test
      • issue PR
      • merge PR
      • apply changes to test repo (transaction, etc.)
      • later also apply to production repo
  • Security updates
    • only do security updates in current Prefix installation (2020.06)
      • disable sync of ebuild tree
    • update all packages in next version (2021.01)
      • "emerge world" or reinstall from scratch
  • we need a way to keep a thorough changelog that provides details on what was changed, by whom, when
    • does CVMFS support something like this? (log messages with transaction)
    • keep separate changelogs for compat and software layer (maybe even in subdirs)
    • have some scripting around this to verify current status & auto-update changelog files
  • are we able to fully control & reproduce package updates?
    • can we assume changes in test repo can be reproduced in production repo?
    • for example being able to cope with sources disappearing upstream
      • or SourceForge/GitHub being down
    • probably involves having our own mirror (see https://wiki.gentoo.org/wiki/Local_distfiles_cache
)
      • update mirror when updating test repo
      • use mirror when updating prod repo
  • Bob is playing with Ansible playbooks to install compatibility layer
  • Peter: some interesting stuff in https://github.com/awesomebytes/gentoo_prefix_ci that we can leverage

Testing

  • How do we test things/updates? We probably need a dev branch/repo for this as well. Can we use the CI / github actions with a daily Gentoo Prefix build that Carlos found? https://github.com/EESSI/compatibility-layer/issues/3#issuecomment-674910901 .

  • Use cvmfs snapshots?

  • comparing package list with installed packaged in Prefix can be done in GitHub Actions CI

    • maybe even through overlay in GitHub Actions
    • could gradually grow over time with specific checks (e.g. for libc, EasyBuild, Lmod)
      • for example install bzip2 from source with EasyBuild

  • thorough test suite to test applications will be available in software layer

Directory structure

Installation procedure

Do we just keep it as it is for now, or does someone want to look into alternatives?

Other

  • Client init script (though this also overlaps with the software layer)

TODO

  • open issues in compatibility-layer based on these notes for things to do
Clone this wiki locally