8. FUGAKU specific topics¶

FUGAKU is a supercomputer designed for versatility, but there are some restrictions for various reasons. This section is a summary of them specific to FUGAKU environment for users who are already expert with Singularity.

8.1. Available versions¶

As of June 2024, following versions are available on FUGAKU.

3.11-7, installed at /opt/singularitypro311/bin/singularity
4.1.2-2, installed at /opt/singularitypro41/bin/singularity

Default version (/usr/bin/singularity) is linked to 4.1.2. Although there are very few cases where 3.11 is needed,when you dare to use it, execute it by specifying the full path, or give priority to that in the PATH variable.No additional configuration is needed.

Both versions of Singularity are installed to both of login and compute nodes. Note that the difference of architecture, x86_64 and aarch64. When we pull images from external repositories, such as Docker Hub, images for corresponding architecture may not exist. In such cases, the one that exists may be acquired unintentionally. Container does not have a hardware emulation feature, it could not be working on mismatched architecture. Image processing without container launch, such as conversion, digital signatures, etc. can be performed regardless of architecture. We can check which architecture the content of the SIF image are in by performing the following.

$ singularity sif list sample-arm.sif
------------------------------------------------------------------------------
ID   |GROUP   |LINK    |SIF POSITION (start-end)  |TYPE
------------------------------------------------------------------------------
1    |1       |NONE    |32176-32362               |Def.FILE
2    |1       |NONE    |32362-33398               |JSON.Generic
3    |1       |NONE    |33398-33594               |JSON.Generic
4    |1       |NONE    |36864-51032064            |FS (Squashfs/*System/arm64)

Architecture is found in the TYPE column ID=4.

8.2. Restrictions about Sandox image¶

A sandbox image is a extracted image into filesystem. Even if you are using a SIF image, the –fakeroot option will extract it to the directory $TMPDIR as an sandbox image. –fakeroot which enables user namespace separation, the user mapping is not effective on shared filesystem. This cause write permission fail. This means that we can not use our home directory for sandbox with –fakeroot and –writable options.

The other hand, buliding singularity image with –fakeroot option also extract temporal image as sandbox in $TMPDIR. It would be converted to SIF image at the last stage. This means that users use sandbox unconciously. As a result, image creation fails with default settings. There are following two workarounds to avoid this problem.

8.2.1. Workaround using /worktmp¶

In generic, $TMPDIR is set to /var/tmp or somewhere else. However, it is set to one’s home directory as default in FUGAKU. To avoid problems described in above, it is necessary to use a local storage. On compute nodes in FUGAKU, There is only one option, /worktmp. When we use –fakeroot, including image building, set it the environment variable as follows.

$ export TMPDIR=/worktmp

However, /worktmp is mounted as shmfs filesystem. This shares capacity of conventional memory. Since compute nodes have only 32GB of emmory for each, the maximum image size is estimated to be around 10 GB.

8.2.2. Workaround using proot¶

When the singularity find the third party utility 'proot' in $PATH, it enables building images without –fakeroot options or using /worktmp directory. So it allows the building of images of a size that exceeds the limit of /worktmp. Singularity will use proot if it finds it in the $PATH, unless the –fakeroot option is specified. However, there are some functional limitations.

Since proot is not installed by default, follow the steps below to obtain and build proot and talloc by yourself.

$ git clone https://github.com/proot-me/proot.git
$ cd proot
$ wget https://www.samba.org/ftp/talloc/talloc-2.4.2.tar.gz
$ tar xzf talloc-2.4.2.tar.gz
$ cd talloc-2.4.2
$ ./configure --disable-python
$ gcc -Ilib/replace -Ibin/default -D__STDC_WANT_LIB_EXT1__=1 -c talloc.c  -o talloc.o
$ cd ../src

Remove some python-related optoins from GNUMakefile and rewrite to link talloc.o statically. See belows for details.

--- a/src/GNUmakefile
+++ b/src/GNUmakefile
@@ -16,15 +16,18 @@ OBJDUMP  = $(CROSS_COMPILE)objdump
 PYTHON   = python3

 HAS_SWIG := $(shell swig -version 2>/dev/null)
-PYTHON_MAJOR_VERSION = $(shell ${PYTHON} -c "import sys; print(sys.version_info.major)" 2>/dev/null)
-PYTHON_EMBED = $(shell ${PYTHON} -c "import sys; print('--embed' if sys.hexversion > 0x03080000 else '')" 2>/dev/null)
-HAS_PYTHON_CONFIG := $(shell ${PYTHON}-config --ldflags ${PYTHON_EMBED} 2>/dev/null)

 CPPFLAGS += -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -I. -I$(VPATH) -I$(VPATH)/../lib/uthash/include
 CFLAGS   += -g -Wall -Wextra -O2
 CFLAGS   += $(shell pkg-config --cflags talloc)
 LDFLAGS  += -Wl,-z,noexecstack
-LDFLAGS  += $(shell pkg-config --libs talloc)
+LDFLAGS  += ../talloc-2.4.2/talloc.o

After modifying, make proot only and install it to the directoy in $PATH.

$ make proot
$ mkdir -p ~/.local/$(arch)/bin
$ cp proot ~/.local/$(arch)/bin
$ chmod 755 ~/.local/$(arch)/bin/proot

Due to the difference of the architecture, the usable binary is differed for login and compute nodes.In this example, dedicated directories are created as the install targets. Therefore, set the $PATH as follows.

export PATH=~/.local/$(arch)/bin:$PATH

Now let’s build an image without –fakeroot option. Singularity find the proot automatically without any config.

$ cat p.def
Bootstrap: localimage
From: alma.sif
%post
        yum -y install python3-pip
        pip3 install numpy scipy
        yum clean all

$ singularity build p.sif p.def
INFO:    Using proot to build unprivileged. Not all builds are supported. If build fails, use --remote or --fakeroot.
INFO:    Starting build...
<<..snip..>>
INFO:    Creating SIF file...
INFO:    Build complete: p.sif

Currently, there are following restrictions to use proot. As shown in above, if the building is not successful, we should give up to use proot but –fakeroot option to have higher level privileges.

As a Bootstrap in definition file, we cannot specify yum, debootstrap, arch and zypper. it’s necessary to use localimage, library, oras and docker instead.
In definition file, we cannot use %pre and %setup, as pre-build processes.
The root privleges by proot is emulated and incomplete.

8.3. Restriction by permission of user home directories¶

The home directory tree for normal user in FUGAKU has the following permissions. As a consequence, those are not accessible for users who do not blong to fugaku group.

$ ls -dl /vol0004/mdt0/home
drwxr-x--- 281 root fugaku 16384 Feb 23 17:50 /vol0004/mdt0/home
$ ls -dl /vol0004/mdt0/home/uXXXXX
drwx------ 8 uXXXXX fugaku 4096 Apr 25 18:45 /vol0004/mdt0/home/uXXXXXX

All normal users belong to the fugaku group as a subgroup, but not the primary group. Because of the limitations of the Linux kernel’s UID namespace separation feature, group privileges can only be mapped to the primary group, so users in containers cannot have access to the fugaku group, since group privileges can only be mapped to the primary group. Therefore, the home directory cannot be mounted, resulting in the error “failed to mount /home/uXXXXX to /root: permission denied.” There are two workaround to avoid this restriction.

Before executing singularity build command, change the primary group to the fugaku group by using newgrp fugaku. However, the file owner and group permissions will be set to fugaku group.
Use the –no-setgroups option added in SingularityPRO 3.11-4 to address this issue.

Both method have been confirmed to work successfully.

8.4. Multi-node parallel with MPI¶

Because of restrictions on communication between nodes allocated by the scheduler, users cannot use MPI parallelism across multiple nodes with their own MPI. For the more information on how to run it, see Multi-node parallel on FUGAKU.

8.5. OCI mode¶

New feature OCI mode from version 4.1 enables OCI(Docker compatible) container utilization. However, current FUGAKU environment, it is not be available for security measures on OS configuration.