3. What is Singularity?¶
Singularity is an open source Linux container implementation developed in 2015 to enhance the convenience of supercomputers at the Lawrence Berkeley National Laboratory in the United States.
Currently Singularity is being developed separately by Sylabs, a company founded by the developers of the product, and by an independent community. The free of charge open source version is called the Community Edition (CE) and is published on Github under a modified BSD License (with some code under the Apache License). In contrast, there is Singularity PRO, which is based on CE with well-tested functionality and paid services such as security support and bug fix backports, provided by Sylabs, the main developer of the software. “FUGAKU” uses Singularity PRO for security and support reasons.
With Singularity, of the system configuration required to execute an application, the parts other than the kernel can be independently configured and maintained, and can be integrated with the application, so if the environment has the same architecture, the entire execution environment can be carried.
Also, if the runtime libraries required for an application are updated frequently or if you need to make full use of a wide variety of applications, you don’t have to think about setting up and switching between each requirement separately. It is also easier to keep old runtime environments, making it easier to revalidate data.
There are 4 distinctive features of Singularity compared to other container implementations:
The image is created in a single file, which makes managing container images intuitive and easy.
The image can be used as a single file without extraction and decompression resulting in the reduction of the I/O workload to the shared file system.
The ID and privileges of the started user are inherited inside the container. As a result, files, Hardware, and processes can be accessed as the same user.
Secure operation is possible because no root authority or daemon is involved when creating an image or executing a container.
3.1. Container¶
Containers are one of the virtualization technologies. Whereas full virtualization installs and runs the entire OS on a hardware emulator, containers share a host-side kernel and only switch between user environments. Therefore, there are following benefits.
Since it runs on a booted kernel, there is no need to boot the entire OS as in full virtualization, booting is fast and there is little overhead.
Since the system, application, and user settings can be used together, it achieves high reproducibility and easy care for inter-generational verification.
There is no need to set up the environment multiple times on multiple systems or to verify the operation.
In addition to Singularity, there are many container implementations such as Podman, Docker, LXE, Shifter, CharieCloud, etc., each of which has its own characteristics. What they have in common is to switch under the root directory and run the application on a dedicated environment. The cut out of the user land (in short, below the root directory) is called an image, and the format and usage method differs depending on the container implementation.
The contents of the container are no different from the existing OS environment. Therefore, it is not possible to bring in applications of different architectures; it is not possible to run an image created on ARM on an x86_64 environment, or vice versa. Also, since the kernel is from the host, it may not be possible to run applications that are strongly dependent on the kernel version or that require additional drivers to be installed.
3.2. Image¶
To create an image of a container, deploy an OS environment separate from the host side in an empty file system, and build the required runtime and application environment. The resulting file system can be saved individually as a file or retrieved from a repository on the network.
An image of Singularity consists of a single file in a format called SIF (Singularity Image Format). Therefore, the user can have an image in which the application is installed for each runtime in one file, and handling such as saving and transferring is intuitive and easy.
On the other hand, Singularity does not require that the image be created in SIF format. You can also run container images saved in other formats directly.
3.3. Execution¶
Basically, launching a container means expanding and mounting an image and preparing the application execution environment, and it has a different meaning from launching an application in it. However, Singularity integrates the launch of the container and the launch of the application, so you can run it as if you were dealing with a statically linked binary, without being aware that you are using the container.
There are some following ways to run your application inside a container with Singularity:
Start the shell when the container starts and operate it manually.
Execute any command in the image at the same time as starting the container.
Make sure that the preset commands for the image are executed as soon as the container is started.
Start only the container and issue a command to the specified container from the outside.
In a batch processing environment such as “FUGAKU”, the second and third execution methods will be used except for verification purpose. The fourth is when the same environment is used over and over again to process a large amount of data, or when a service such as Jupyter Notebook is launched and accessed from the outside