5. Creating an image¶
Learn how to create an SIF image for execution. There are some methods depending on the setup you need, so choose the one that suits your situation.
5.1. How to use a sandbox¶
When building a system by directly editing an image, the actual work is done by manually. This is convenient for simple image creation and modification, but there are problems such as no work history.
5.1.1. If you don’t need to start the container¶
As already mentioned in the previous section, if the user brings files etc. Into the sandbox or edits the settings etc.,those could be converted to SIF images. This is useful way if you just need to expand the application or reference data that has already been created. The image of the work procedure will be as follows.
$ singularity build --sandbox ubi_py38 docker://registry.access.redhat.com/ubi8/python-38
$ tar xvzf ~/application.tar.gz -C ubi_py38
usr/local/bin/application
usr/local/bin/subcommand
usr/local/lib/libappr.so.1
<<..snip..>>
$ vi ubi_py38/usr/local/etc/application.config
$ cp ~/tmp/config.json ubi_py38/usr/local/etc
$ singularity build appl.sif ubi_py38
INFO: Starting build...
INFO: Creating SIF file...
INFO: Build complete: appl.sif
$ singularity exec appl.sif /usr/local/bin/application
This gets the Redhat UBI image which has Python-3.8 installed and creates the sandbox image.
The tarball of the application that has been expanded under /usr/local is expanded in sandbox.
Editing the expanded file and bring in the configuration file from the outside.
Generating an SIF image from a sandbox image that has been modified.
Launching an application from the resulting SIF image.
If you can build an environment just by importing the created file in this way, image creation will be completed very easily.
5.1.2. When starting a container and changing the system¶
All files and directories in sandbox that were originally owned by root were owned by the user. If you start the container with –writable option, it will start in the same state about the file owner and you can edit the system file with user privileges.
$ singularity shell --writable ubuntu2004
Singularity> touch /hogefuga
Singularity> ls -l /
total 20
lrwxrwxrwx 1 a0XXXX fugaku 7 Apr 27 02:46 bin -> usr/bin
drwxr-xr-x 2 a0XXXX fugaku 6 Apr 15 2020 boot
drwxr-xr-x 21 root root 3860 May 15 03:19 dev
lrwxrwxrwx 1 a0XXXX fugaku 36 May 11 14:28 environment -> .singularity.d/env/90-environment.sh
drwxr-xr-x 31 a0XXXX fugaku 4096 Apr 27 02:49 etc
-rw-rw-r-- 1 a0XXXX fugaku 0 May 16 14:28 hogefuga
drwxr-xr-x 3 a0XXXX fugaku 60 May 11 14:28 home
<<..snip..>>
Singularity> exit
$ singularity build from_sandbox.sif ubuntu2004
INFO: Starting build...
INFO: Creating SIF file...
INFO: Build complete: from_sandbox.sif
$ ./from_sandbox.sif ls -l /hogefuga
-rw-rw-r-- 1 a0XXXX fugaku 0 May 16 14:28 hogefuga
When you want to make changes to the system in the image as root, for example by installing additional packages, this requires the –fakeroot option to be used in conjunction with the –writable option. –fakeroot, separates the user ID namespace in the container from that in the host OS, mapping UID=0, or root, in the container to tue normal user on the host. This makes it look as if the user is working with normal user privileges from the host environment, while using root privileges in the container. At this time, the user’s home directory is bound to /root.
However, there is a problem "FUGAKU" specific here. The home directories of normal users have the following permission settings, and cannot be accessed unless users belong to the fugaku group.
$ ls -dl /vol0004/mdt0/home
drwxr-x--- 281 root fugaku 16384 Feb 23 17:50 /vol0004/mdt0/home
$ ls -dl /vol0004/mdt0/home/uXXXXX
drwx------ 8 uXXXXX fugaku 4096 Apr 25 18:45 /vol0004/mdt0/home/uXXXXXX
All normal users belong to the fugaku group as a subgroup, but not the primary group.Because of the limitations of the Linux kernel’s UID namespace separation feature, group privileges can only be mapped to the primary group, so users in containers cannot have access to the fugaku group, since group privileges can only be mapped to the primary group. Therefore, the home directory cannot be mounted, resulting in the error “failed to mount /home/uXXXXX to /root: permission denied. There are two ways to work around this.
Before executing singularity build command, change the primary group to the fugaku group by using newgrp fugaku. However, the file owner and group permissions will be set to fugaku group.
Use the –no-setgroups option added in SingularityPRO 3.11-4 to address this issue.
From now on, we will assume that you have done newgrp fugaku. Using –fakeroot to make sure that you are root inside the container.
$ newgrp fugaku
$ singularity shell --writable --fakeroot ubuntu2004
WARNING: Skipping mount /etc/localtime [binds]: /etc/localtime doesn't exist in container
Singularity> whoami
root
Singularity> pwd
/root
Singulraity> ls -l /home
total 0
However, there is another problem to be solved. The isolated ID is valid only on the node starting the container. So with local storage, file owners can follow suit, but with shared file systems such as FEFS and NFS, storage can’t handle identity isolation. Therefore, the sandbox image in the home directory cannot be used with –fakeroot and the environment construction does not work properly. There is /worktmp as a local storage available to normal user in FUGAKU environment, but this is shared from the memory of the compute node as storage. Note that a compute node has only 32GB of memory, so it is not possible to create a image larger than 10GB. If you need to create such a image, consider to use an external command proot. Refer to the Workaround using proot about details.
The flow of creating a sandbox image from the SIF image in the /worktmp directory, installing any package in the –fakeroot environment, and returning it to the SIF image should be as follows.
$ newgrp fugaku
$ singularity build --sandbox /worktmp/ubuntu-sandbox ubuntu2004.sif
$ singularity exec --fakeroot --writable /worktmp/ubuntu-sandbox apt install -y XXXXXXX # コマンド実行の例
$ singularity shell --fakeroot --writable /worktmp/ubuntu-sandbox # シェル環境でインタラクティブ作業する例
Singularity> apt install XXXXXXX
Reading package lists... Done
Building dependency tree
<<..snip..>>
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Singularity> exit
$ singularity build ~/from-sandbox.sif /worktmp/ubuntu-sandbox
If you have less work, you can just execute the command with exec as in the example instead of shell.
5.2. Custom image with definition file¶
All operation of image creation can be described in a definition file, for example, where to get the base image, bringing in files, set up the system inside the container, define the execution environment, etc. Properly setting the execution environment could greatly improve the convenience to use the container.
The definition file used to be called as recipe file, and the name has not been decided, but now it is officially a definition file, and the extension is def. Please refer to the official documentation for more details, as this is just a simple usage.
The definition file consists of four parts: header, construction, execution environment definition, and description. however, it works with at least the header.
$ cat example.def
Bootstrap: docker
From: registry.access.redhat.com/ubi8/python-38
%file
some_data.tar.gz /
%post
dnf install -y python3-libxml2
tar xzf /some_data.tar.gz -C /usr/share
rm -f /some_data.tar.gz
dnf clean all
%environment
export LC_ALL=en_US.utf-8
export CONFIG_FILE=~/applicatin/etc/some.config
%runscript
python $*
%labels
Author Tarou.Fugaku@fugaku.org
Version v0.99.1
This example describes the following:
Get the UBI8 Python3.8 environment with Bootstrap: and From: in the header and create a sandbox image in $TMPDIR.
Import the data at hand (some_data.tar.gz) to the root directory in the container.
Start the container and install python3-libxml2 with dnf.
Extract the data captured by %file to /usr/ share and delete the file and dnf cache.
Set environment variables in %environment for starting the container.
In %runscript, set the command to be executed when the image file is executed directly.
Describe the container information in %labels
As you can see, the %post process is done in sandbox, the dnf command can only be used with root privileges. Therefore, creating an image with this definition file requires root privileges or –fakeroot execution. Additionally, in FUGAKU, since the environment variable $TMPDIR is set to be the home directory, the build will fail as it is. then execute as follows.
$ export TMPDIR=/worktmp
$ newgrp fugaku
$ singularity build --fakeroot ~/from_def.sif example.def
The time required varies depending on the processing content, but from_def.sif should be completed. If you cannot change $TMPDIR, you can specify only the image expansion destination of singularity by setting $SINGULARITY_TMPDIR.
5.3. Embedding the execution configuration¶
In the definition file of the completed image, %runscript contains the command line 'python $'. This means that python would be started with the passed options at the same time as the container is started. In other words, the following two do the same thing.
$ singularity exec from_def.sif python hoge.py 1024
$ ./from_def.sif hoge.py 1024
For example, if the file name of the image is the same as the command of the application and it is placed in a directory in the PATH, you do not need to be aware that you are using a container. It is also possible to embed standard pre-processing, etc., and you can also use %environment to prevent troubles due to setting omissions. These are things that have been done in job scripts in a batch job environment, but by embedding them in an image with Singularity make job scripts simple to avoid mistakes.
In addition to this, there are %app for bundling multiple applications and using them properly, and %help for writing a brief explanation. See documentation for details. If you manage this definition file in the repository, it will be clear how the calculation was performed with an image how it created, and it will be easy to reproduce even if you create the same environment again. However, if you use a tag like 'latest' in From: at the initial image building, the content will change each time you build the image.
5.4. Creating an image by submitting a job¶
Let’s put these tasks into a job for FUGAKU. Create a simple script like the one below and submit the job.
$ cat job.sh
#!/bin/bash
export TMPDIR=/worktmp
newgrp fugaku
singularity biuld --fakeroot create_job.sif example.def
$ pjsub -L "rscunit=rscunit_ft01,rscgrp=small,node=1,elapse=0:15:00" ./job.sh
You can create an image in the same way.
5.5. Importing TCSDS into a container¶
FUGAKU has its own repository, language environment and library.You can embed a specific version of the runtime in the container image by getting the package from it and installing it.At the moment, only the rpm file is prepared, so it cannot be installed on Debian OS such as Ubuntu,so we will show you how to install it on RHEL and its derivative UBI etc.UBI is the subject here for the sake of simplicity, but please note that UBI may be difficult to use in practice because development packages etc. are only supplied in a limited way.
The repository where TCSDS is located is /home/apps/singularity/tcs/, but documents are also included, so please refer to it as necessary. However, it is inconvenient if it is under /home to import it into the container, so we will devise it. first, prepare the following repository file.
$ cat tcs.repo
[tcs]
enabled=1
name=tcs
gpgcheck=0
baseurl=file:///opt/repo/tcs/
Copy this file to /tmp. Generate sandbox image of RHEL type OS such as CentOS, UBI in /worktmp with sandbox. Temporarily create a directory named 'source'.
Next, boot this image with –writable and –fakeroot. However, you still can’t access the local repository under /home. So we use -B to bind the repository directory to another directory in the container, here /opt/repo.When you use –writable, the destination directory will not be created automatically, so create it in advance.Then use the repo file to add the repository information inside the container.The repo file is saved in /tmp which is bound by default, so you don’t need to bring it in explicitly.
$ singularity build --sandbox -f source docker://registry.access.redhat.com/ubi8/python-38
<<..snip..>>
$ mkdir source/opt/repo
$ singularity shell --writable --fakeroot -B /home/apps/singularity:/opt/repo source
Singularity> dnf config-manager --add-repo=/tmp/tcs.repo
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Adding repo from: file:///tmp/tcs.repo
Singularity> dnf repolist
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
repo id repo name
tcs tcs
ubi-8-appstream Red Hat Universal Base Image 8 (RPMs) - AppStream
ubi-8-baseos Red Hat Universal Base Image 8 (RPMs) - BaseOS
ubi-8-codeready-builder Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder
Singularity> dnf search FJSV
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Last metadata expiration check: 0:03:40 ago on Wed 20 Oct 2021 12:07:04 PM JST.
===================================== Name Matched: FJSV =====================================
FJSVxtclanga-common-TCSDS10R24.noarch : Generation Management for Language
FJSVxtclanga-fc-compiler-CECA023.aarch64 : Fujitsu Fortran/C/C++ Compiler
FJSVxtclanga-fc-compiler-doc-CMLN016.noarch : Fujitsu Fortran/C/C++ Compiler Online Documents
<<..snip..>>
Once you have confirmed that you can access the repository in this way, you can select and install the required packages as usual.
Singularity> dnf install -y FJSVxtclanga-fc-compiler-CECA023.aarch64
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Last metadata expiration check: 0:07:29 ago on Wed 20 Oct 2021 12:07:04 PM JST.
Dependencies resolved.
===============================================================================================
Package Architecture Version Repository Size
===============================================================================================
Installing:
FJSVxtclanga-fc-compiler-CECA023 aarch64 4.4.0-03 tcs 88 M
Transaction Summary
===============================================================================================
Install 1 Package
Total size: 88 M
Installed size: 516 M
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : FJSVxtclanga-fc-compiler-CECA023-4.4.0-03.aarch64 1/1
Verifying : FJSVxtclanga-fc-compiler-CECA023-4.4.0-03.aarch64 1/1
Installed products updated.
Installed:
FJSVxtclanga-fc-compiler-CECA023-4.4.0-03.aarch64
Complete!
$ dnf clean all
It is recommended to clear the cache to make the image smaller.This is also recommended the case when installing packages from the repository with %post description in the definition file.Once you’ve installed the package or application, convert the sandbox image to a SIF image.
$ singularity build -f ~/tcs-installed.sif source
WARNING: 'nodev' mount option set on /worktmp, it could be a source of failure during build process
INFO: Starting build...
INFO: Creating SIF file...
INFO: Build complete: /home/uXXXXX/tcs-installed.sif
When this document was first written, Singularity PRO did not allow the use of -B at build time, so it was necessary to go through the sandbox once, as described above. For this reason, processing by definition files and environment variable embedding were separately performed for the created sandbox image using the following definition files.
$ cat from_source.def
Bootstrap: localimage
From: /worktmp/source
%post
<<..snip..>>
$ singularity build -f ~/from_def_with_tcs.sif from_source.def
Since version 3.9 or later, the -B option works at building images, it enables direct image building without any manual operations written in above. Use definition files describeed importing repo file by %file or creating it in %post. Note that the target directory for -B optoin must exist inside the source image. In this case, /opt/repo directory must be created before building.We will end up working in the sandbox. However, if the specified directory does not exist at the time of execution using the 'singularity exec', it will be prepared automatically.
$ singularity build --sandbox ubi_py38 docker://registry.access.redhat.com/ubi8/python-38
$ mkdir ubi_py38/opt/repo
$ cat direct_build.def
Bootstrap: localimage
From: ubi_py38
%files
tcs.repo /
%post
dnf config-manager --add-repo=/tcs.repo
dnf install -y FJSVxtclanga-fc-compiler-CECA023.aarch64
dnf clean all
rm -f /tcs.repo
<<..snip..>>
$ singularity build -f -B /home/apps/singularity:/opt/repo ubi_tcs.sif direct_build.def
5.6. Image management by digital signature¶
Singularity SIF Images are managed in one file, so if you change the file name, you will not be able to trace who created the image and how. This can cause security issues as well as management trouble. Therefore, by adding a digital signature to the image by the creator, it is possible to prevent problems which image being altered or replaced.
Digital signatures use the public key mechanism. First, calculate the hash value of the image, then encrypt it with the private key, and add (sign) it to the image. The public key should be managed in secure, separately from the image. The user who receives the image calculates the hash value of the image and decryption the signature cryptography with the creator’s public key to verify whether it is exactly same image signed by the creator.
5.6.1. Creating a key pair¶
First, create a key pair of private and public keys. Use the 'key' command. You will be asked if you want to push to the keystore on the way, but for now, leave it as no. even if it is Y, the key pair should be created.
$ singularity key newpair
Enter your name (e.g., John Doe) : Fugaku Tarou
Enter your email address (e.g., john.doe@example.com) : tarou@fugaku.org
Enter optional comment (e.g., development keys) : Test Keypair
Enter a passphrase :
Retype your passphrase :
Would you like to push it to the keystore? [Y,n] n
Generating Entity and OpenPGP Key Pair... done
NOT pushing newly created key to: https://keys.sylabs.io
The keys can be checked with the list option as shown below for the public key and private key, and each is stored in ~/.singularity/sypgp.
$ singularity key list
Public key listing (/home/rccs-aot/a0XXXX/.singularity/sypgp/pgp-public):
0) U: Fugaku Tarou (Test Keypair) <tarou@fugaku.org>
C: 2021-03-31 13:00:00 +0900 JST
F: B70ADXXXXFB3CEZZZZ67F0A304YYYYY122B0WWWWW
L: 4096
--------
$ singularity key list --secret
Public key listing (/home/rccs-aot/a0XXXX/.singularity/sypgp/pgp-secret):
1) U: Fugaku Tarou (Test Keypair) <tarou@fugaku.org>
C: 2021-03-31 13:00:00 +0900 JST
F: B70ADXXXXFB3CEZZZZ67F0A304YYYYY122B0WWWWW
L: 4096
--------
$ ls -l ~/.singularity/sypgp
-rw------- 1 a0XXXX a0XXXX 6625 Mar 31 13:00 pgp-public
-rw------- 1 a0XXXX a0XXXX 14455 Mar 31 13:00 pgp-secret
Let’s use this to sign the image and verify it.
5.6.2. How to sign and verify¶
To sign with singularity, use the 'sign' command. only for SIF images.
$ singularity sign ubuntu2004.sif
Enter key passphrase :
Signature created and applied to ulsb.sif
If you have created multiple key pairs, you will be asked which one to use, but if you have one, just enter the passphrase which you entered at creating. The file size will be slightly larger by the amount of the signature. next, let’s verify the image.
$ singularity verify ubuntu2004.sif
Verifying image: ubuntu2004.sif
[LOCAL] Signing entity: Fugaku Tarou (Test Keypair) <tarou@fugaku.org>
[LOCAL] Fingerprint: B70ADXXXXFB3CEZZZZ67F0A304YYYYY122B0WWWWW
Objects verified:
ID |GROUP |LINK |TYPE
------------------------------------------------
1 |1 |NONE |Def.FILE
2 |1 |NONE |FS
Container verified: ubuntu2004.sif
If you give this image to someone, share your public key in another way. We sometimes use a service called a key store, but here we will try a method of converting keys into files and exchanging them. In this case, you output your public key to a file called public.asc and someone else imports it for verification.
$ singularity key export ./public.asc
Public key with fingerprint B70ADXXXXFB3CEZZZZ67F0A304YYYYY122B0WWWWW correctly exported to file: ./public.asc
$ singularity key import ./public.asc
Key with fingerprint B70ADXXXXFB3CEZZZZ67F0A304YYYYY122B0WWWWW successfully added to the public keyring
Using this imported public key, you can check whether the acquired image is as intended and whether it is exactly the same as when it was created. In addition, since multiple signatures can be applied, it is possible to add signatures from both the creator and the approver.