Software Compatibility ======================= **e4s-cl** wraps itself around a MPI command and inserts a container launch before the command is run. This implies an understanding of both the process launcher and the container technology to use. This section details the software **e4s-cl** was developped for and tested with. MPI implementations -------------------- .. _mpi_compatibility: **e4s-cl** is able to launch executables compiled with a different version than the one available on the system at runtime. MPI families +++++++++++++ With the introduction of the MPICH ABI initiative and the MPI implementations that followed it, we see the emergence of multiple 'families' of MPI implementations. Members within a family can usually swap their libraries at runtime, while this is impossible accross families. If you are unsure about the implementation used to compile a binary, a simple check is to run :code:`ldd` on the executable and looking at the MPI library listed as a dependency. The below table lists the usual libray name in the :code:`soname` column. .. csv-table:: :file: mpi.csv Enabling translation +++++++++++++++++++++ **e4s-cl** is able to detect the MPI implementation used on the host from the input parameters; however the implementation used in the container cannot be ascertained easily. MPI translation is thus not needed when running executables with an implementation from the same family as the host's; when this is not the case, one can enable it from the command line using the :code:`--from` flag: .. code:: # Enable translation from an OpenMPI binary to a MPICH-based profile e4s-cl --from openmpi srun -n 4 [...] path/to/executable This will install and leverage :code:`WI4MPI` (git repos `here `_), which is a wrapper interface that performs translations from one MPI implementation to another. This allows :code:`e4s-cl` to function even when substituting two MPI libraries that are ABI-incompatible. Container backends ------------------- During a :code:`launch` command, a list of required files will be computed on each target node, and bound to a container spawned on those nodes. Container backends are the container technologies **e4s-cl** is able to use. :code:`apptainer`, :code:`podman`, :code:`shifter` and :code:`singularity` are supported. The :code:`docker` backend can be enabled by installing **e4s-cl** with the :code:`docker` extra feature. Most of the container backends have some degree of configuration available to them. Configuration is done by setting the corresponding values in the :code:`backends` section of the configuration file, or by passing a given environment variable. An option passed in the environment will override the same option set in the configuration file. :code:`apptainer` ++++++++++++++++++ :code:`apptainer` support is achieved using its command line interface. :code:`apptainer` has to be installed on the system and available for it to be used by **e4s-cl**. The configuration file section for :code:`apptainer` is :code:`backends/apptainer`. The available options are: .. list-table:: :widths: 10 10 5 5 20 :header-rows: 1 * - Configuration variable - Environment variable - Type - Default - Description * - :code:`executable` - :code:`E4S_CL_APPTAINER_EXECUTABLE` - String - :code:`""` - Path to the :code:`apptainer` executable to use. * - :code:`options` - :code:`E4S_CL_APPTAINER_OPTIONS` - List - :code:`[]` - Options to pass to the spawned :code:`apptainer` process. * - :code:`run_options` - :code:`E4S_CL_APPTAINER_EXEC_OPTIONS` - List - :code:`[]` - Options to pass to the :code:`exec` command of the spawned :code:`apptainer` process. :code:`docker` ++++++++++++++ .. admonition:: :code:`docker` support :code:`docker` support in HPC is limited at best. Due to its architecture, it will most likely not work with your MPI implementation. To enable :code:`docker` support from **e4s-cl**, :ref:`install` with the :code:`docker` extra enabled. :code:`docker` support is achieved using the :code:`docker-py` `module `_. The :code:`docker` daemon has to be installed and running on the system to be accessed. There are no configuration options for the :code:`docker` backend. .. warning:: Using **docker** with MPI Several MPI implementations expect their processes to inherit opened file descriptors; because of docker's client-daemon architecture, this is not possible. To use docker images with MPI, it is encouraged to used :ref:`podman`. .. _podman: :code:`podman` +++++++++++++++ :code:`podman` support is achieved using its command line interface. :code:`podman` has to be installed on the system and available for it to be used by **e4s-cl**. The configuration file section for :code:`podman` is :code:`backends/podman`. The available options are: .. list-table:: :widths: 10 10 5 5 20 :header-rows: 1 * - Configuration variable - Environment variable - Type - Default - Description * - :code:`executable` - :code:`E4S_CL_PODMAN_EXECUTABLE` - String - :code:`""` - Path to the :code:`podman` executable to use. * - :code:`options` - :code:`E4S_CL_PODMAN_OPTIONS` - List - :code:`[]` - Options to pass to the spawned :code:`podman` process. * - :code:`run_options` - :code:`E4S_CL_PODMAN_RUN_OPTIONS` - List - :code:`[]` - Options to pass to the :code:`run` command of the spawned :code:`podman` process. :code:`shifter` ++++++++++++++++ The :code:`shifter` container technology is an interesting case, as it offers much of what **e4s-cl** is trying to propose, but in a much more restrictive way. .. warning:: :code:`shifter` MPI modules Some MPI libraries will be configured as modules for :code:`shifter` and imported automatically. You might not need **e4s-cl** ! Check the configuration file in :code:`/etc/shifter/udiRoot.conf` to see if the MPI library you are trying to use is imported. .. warning:: Binding files with :code:`shifter` :code:`shifter` is the only container backend that does not support binding files to the container. :code:`shifter` also forbids binding directories to certain locations, like :code:`/etc`. This is circumvented by **e4s-cl** by copying all required libraries in a temporary directory, then binding it to the container. Files are ignored. The configuration file section for :code:`shifter` is :code:`backends/shifter`. .. list-table:: :widths: 10 10 5 5 20 :header-rows: 1 * - Configuration variable - Environment variable - Type - Default - Description * - :code:`executable` - :code:`E4S_CL_SHIFTER_EXECUTABLE` - String - :code:`""` - Path to the :code:`shifter` executable to use. * - :code:`options` - :code:`E4S_CL_SHIFTER_OPTIONS` - List - :code:`[]` - Options to pass to the spawned :code:`shifter` process. :code:`singularity` ++++++++++++++++++++ :code:`singularity` support is achieved using its command line interface. :code:`singularity` has to be installed on the system and available for it to be used by **e4s-cl**. The configuration file section for :code:`singularity` is :code:`backends/singularity`. The available options are: .. list-table:: :widths: 10 10 5 5 20 :header-rows: 1 * - Configuration variable - Environment variable - Type - Default - Description * - :code:`executable` - :code:`E4S_CL_SINGULARITY_EXECUTABLE` - String - :code:`""` - Path to the :code:`singularity` executable to use. * - :code:`options` - :code:`E4S_CL_SINGULARITY_OPTIONS` - List - :code:`[]` - Options to pass to the spawned :code:`singularity` process. * - :code:`run_options` - :code:`E4S_CL_SINGULARITY_EXEC_OPTIONS` - List - :code:`[]` - Options to pass to the :code:`exec` command of the spawned :code:`singularity` process. :code:`barebones` ++++++++++++++++++++ :code:`barebones` support is achieved by bypassing the use of a container technology. The configuration file section for :code:`barebones` is :code:`backends/barebones`. The available options are: .. list-table:: :widths: 10 10 5 5 20 :header-rows: 1 * - Configuration variable - Environment variable - Type - Default - Description * - :code:`options` - :code:`E4S_CL_BAREBONES_OPTIONS` - List - :code:`[]` - Options to pass to the spawned process. * - :code:`run_options` - :code:`E4S_CL_BAREBONES_EXEC_OPTIONS` - List - :code:`[]` - Options to pass to the script of the spawned process. Other container backends ++++++++++++++++++++++++++ More container technologies can be supported. Create an issue on github or write a dedicated module in :code:`e4s_cl/cf/containers`. Refer to :code:`e4s_cl/cf/containers/__init__.py` for details. Process launchers ------------------ The following process managers were successfully tested with **e4s-cl**: - The stock :code:`mpirun` of multiple MPI distributions; - LLNL's SLURM using :code:`srun`; - CRAY's ALPS using :code:`aprun`; - IBM's JSM using :code:`jsrun`. Support implies the automatic detection of parameters. If a launcher is not supported, use the ':code:`--`' syntax to separate launcher and process arguments. Additional options can be configured through the configuration file or the environment: .. list-table:: :widths: 10 10 5 5 20 :header-rows: 1 * - Configuration variable - Environment variable - Type - Default - Description * - :code:`launcher_options` - :code:`E4S_CL_LAUNCHER_OPTIONS` - List - :code:`[]` - List of options to pass to the launcher