3.3. LLVM

This section describes how to use the C/C++/Fortran compilers based on LLVM. LLVM, which generates binaries for compute nodes, is available on both the login nodes (Intel) and the compute nodes.

3.3.1. Usage of LLVM compilers

3.3.1.1. Environment setting

These compiler is located at:

On login node (cross environment) /vol0004/apps/r/OSS_CN/llvm-22.1.0/cross_clangfx
On compute node (native environment) /vol0004/apps/r/OSS_CN/llvm-22.1.0/own_clangfx

  • Previous versions are still available for use.

  • If you prefer to use LLVM 19.1.4, ‘LLVM v19.1.4’ remains available.

  • If you prefer to use LLVM 21.1.4, please specify LLVM/llvmorg-21.1.0 in the usage environment settings described below.

Execute the following command to setup the environment before running compilers.

login node

[_LNIlogin]$ module load LLVM/llvmorg-22.1.0

compute node

[_CNlogin]$ module load LLVM/llvmorg-22.1.0

Available languages are following

Software

Language

Version

Command

Clang

C

22.1.0

clang

Clang++

C++

22.1.0

clang++

Flang

Fortran

22.1.0

flang

3.3.1.2. How to Use

Here is an example of how to compile C. The commands are common for both nodes. For compilation options, please refer to those provided by LLVM’s Clang/Flang.

In particular, with flang, the compile process may sometimes take longer. In such cases, please allocate a larger cache size on the compute node, as described below.

[Sequential execution]

login node
[_LNIlogin]$ clang -O3 source_file
compute node
[_CNlogin]$ clang -O3 source_file

[OpenMP]

Please add “-fopenmp” option for OpenMP library.

login node
[_LNIlogin]$ clang -O3 -fopenmp source_file
compute node
[_CNlogin]$ clang -O3 -fopenmp source_file

[MPI]

You can use MPI library via mpiclang/mpiclang++/mpiflang commands.

login node
[_LNIlogin]$ mpiclang -O3 -fopenmp source_file
compute node
[_CNlogin]$ mpiclang -O3 -fopenmp source_file

See also

The same procedure applies when using C++ (mpiclang++) and Fortran (mpiflang).

Fujitsu MPI in LLVM22 uses libraries within the Fujitsu compiler environment, and due to module incompatibility, it may output the following error message during compilation with the mpiflang command, resulting in a link error.

  • Example of a linking error message

undefined reference to `_QMmpi_f08_callbacksPmpi_type_null_copy_fn'
undefined reference to `_QMmpi_f08_typesPompi_comm_op_eq'
flang-22: error: linker command failed with exit code 1 (use -v to see invocation)

This error occurs when all of the following conditions are met:

  1. Use mpi_f08.

  2. Use module subprograms defined in the mpi_f08_callbacks module or the mpi_f08_types module, which are components of mpi_f08.

If an error occurs, add the following to the mpiflang command options:

-L/vol0004/apps/r/OSS_CN/llvm-22.1.0/fj-mpi-tcs/libmpi_usempif

Also, add the following path to LD_LIBRARY_PATH during execution.

/vol0004/apps/r/OSS_CN/llvm-22.1.0/fj-mpi-tcs/libmpi_usempif

Attention

/vol0004/apps/r/OSS_CN/llvm-22.1.0/fj-mpi-tcs/libmpi_usempif specified in LD_LIBRARY_PATH must be listed before ${LLVM_BASEDIR}/lib64.

Example: Compilation in a job script

When assigning a job to a compute node, specify the pjsub option –llio cn-cache-size=1Gi.
If omitted, the translation time may increase.
#!/bin/sh -x
#PJM -L  "node=1"
#PJM -L  "rscgrp=small"
#PJM -L  "elapse=01:00:00"
#PJM -x PJM_LLIO_GFSCACHE=/vol0004
#PJM -g groupname
#PJM --llio cn-cache-size=1Gi
#PJM -s
#

module purge
module load LLVM/llvmorg-22.1.0

mpiclang -o sample1.out sample1.c

3.3.1.3. How to execute

The compiled binary can be executed either from an interactive job or from a script. Here is an example of script. To improve performance, transfer the executable and LLVM libraries using llio_transfer.

#!/bin/sh -x
#PJM -L  "node=128"
#PJM --mpi "max-proc-per-node=4"
#PJM -L  "rscgrp=small"
#PJM -L  "elapse=01:00:00"
#PJM -x PJM_LLIO_GFSCACHE=/vol0004
#PJM -g groupname
#PJM -s
#

module purge
module load LLVM/llvmorg-22.1.0
llio_transfer `find ${LLVM_BASEDIR}/lib64 -type f -name \*.so\*`
llio_transfer sample1.out

mpiexec ./sample1.out

See also

For C++ and Fortran programs, the same execution procedure should be followed.

Note

When using LLVM21.1.0, transfer MPI libraries (File under ${MPI_HOME}/lib6) using llio_transfer.

3.3.1.4. Using Libraries

With this LLVM, the OpenBLAS and FFTW libraries are available for use.

Please note that this library cannot be used with other compilers.

3.3.1.4.1. Usage of OpenBLAS

The libraries are located at the following path:

Library

Path of Library

LP64, sequential

/vol0004/apps/r/OSS_CN/llvm/openblas-seq

LP64, thread-parallel

/vol0004/apps/r/OSS_CN/llvm/openblas-omp

ILP64, sequential

/vol0004/apps/r/OSS_CN/llvm/openblas-seq-ilp64

ILP64, thread-parallel

/vol0004/apps/r/OSS_CN/llvm/openblas-omp-ilp64
3.3.1.4.1.1. How to compile

When compiling a program that uses OpenBLAS, please first configure the environment for LLVM. Then, add the option -lopenblas at the linking stage.

For multi-threaded execution, please also add the option -fopenmp.

In addition, please add the following options depending on the programming language.

For C/C++:

  • Please specify the library and header files of the OpenBLAS installation directory (${OPENBLAS_DIR}) using the -L and -I options.

  • Please add -lflang_rt.runtime -lm.

For Fortran:

  • Please specify the library files of the OpenBLAS installation directory (${OPENBLAS_DIR}) using the -L option.

3.3.1.4.1.2. How to execute

When running a program that uses OpenBLAS, please add ${OPENBLAS_DIR}/lib to the LD_LIBRARY_PATH.

Modifications from the original OpenBLAS

Compared with the original OpenBLAS 0.3.26, the following routines have been tuned for A64FX.

  • DGEMM

  • SGEMM

3.3.1.4.2. Usage of FFTW

FFTW libraries are located at /vol0004/apps/r/OSS_CN/llvm/fftw3. (FFTW_DIR)

3.3.1.4.2.1. How to compile

When compiling a program that uses FFTW, please first configure the environment for LLVM. Then, add the following options at the linking stage:

  • Single-precision sequential library : -lfftw3f

  • Single-precision multi-threaded library : -lfftw3f_omp -lfftw3f

  • Double-precision sequential library : -lfftw3

  • Double-precision multi-threaded library : -lfftw3_omp -lfftw3

When using the multi-threaded library, please add the option -fopenmp.

In addition, Please specify the library and header files of the FFTW installation directory (${FFTW_DIR}) using the -L and -I options.

3.3.1.4.2.2. How to execute

When running a program that uses FFTW, please add ${FFTW_DIR}/lib to the LD_LIBRARY_PATH.

Modifications from the original FFTW

  • Header files supporting various SIMD instructions have been adapted for SVE instructions.

  • In the automatic tuning mechanism called plan generation, efficiency is achieved by removing processing routes that are not selected.

3.3.1.5. Cautions

3.3.1.5.1. Caution for REAL(KIND=16) and COMPLEX(KIND=16) types

On the cross-compilation environment flang available on the login nodes, the following error occurs when compiling Fortran programs that use REAL(KIND=16) or COMPLEX(KIND=16) types.

Example for the failure

flang complex16.f90
error: Semantic errors in complex16.f90
 ./complex16.f90:3:3: error: COMPLEX(KIND=16) is not an enabled type for this target
 complex(kind=16) :: x
 ^^^^^^^^^^^^^^^^^^^^^

flang real16.f90
error: Semantic errors in real16.f90
 ./real16.f90:3:3: error: REAL(KIND=16) is not an enabled type for this target
 real(kind=16) :: x, y, z
 ^^^^^^^^^^^^^^^^^^^^^^^^

Workaround

When using these types, please compile in the native environment (i.e., on the compute nodes).

3.3.1.5.2. Notes for Interlanguage Linkage

3.3.1.5.2.1. Compilation Command and Required Options during Linking

When linking combined Fortran, C, and C++ object programs created with the LLVM compiler, please note the following points.

When linking C++ objects (Using libstdc++)

When linking using the command clang or flang, please specify the option -lstdc++.

When linking C++ objects (Using libc++)

When linking using the command clang or flang, please specify the option -lc++.

When linking Fortran objects

When linking using the command clang or clang++, please specify the following options.

-lflang_rt.runtime -lm

3.3.2. Migration from Fujitsu compiler

This chapter explains the differences in specifications and behavior between the Fujitsu compiler and the LLVM (https://llvm.org/) compiler environment to facilitate the smooth migration of applications developed and built with the Fujitsu compiler to the LLVM environment.

3.3.2.1. Compiler Commands

Here describes about differences between the Fujitsu compiler and the LLVM compiler. For the Fujitsu compiler, the command names differ between native compilers and cross compilers. In contrast, the LLVM compiler uses the same command names regardless of whether they are native or cross compilers.

Item

Fujitsu compiler

LLVM compiler

C compiler
fcc
fccpx

clang

C++ compiler
FCC
FCCpx

clang++

Fortran compiler
frt
frtpx

flang

3.3.2.2. Compiler Options

3.3.2.2.1. Comparison of Compiler Option

Fujitsu compilers (trad mode, clang mode) and LLVM compilers differ in their compile option systems and how to specify their options. This section presents a comparison table of compiler options for both compilers as reference information for migration.

Table: Compiler Options with an Identical Function lists compiler options that share the same name and have identical functional descriptions in both the Fujitsu compiler and the LLVM compiler. Table: Compiler Options with a Similar Function (For C/C++Compiler) and Table: Compiler Options with a Similar Function (For Fortran Compiler) list compiler options that have different names but identical functional descriptions.

Note

  • Fujitsu compiler-specific options(Example:-K or -ffj) are not available in the LLVM compiler. Specifying these will result in a compilation error and cause the process to terminate. For examples of Fujitsu compiler-specific options, please refer to Incompatible features.

  • Even if options have the same functional overview, optimization results may differ. After migration, we recommend performing functional verification.

Compiler Options with an Identical Function

Option Name

-Dname[=tokens]

-E

-Idir

-Ldir

-P

-S

-Uname

-Wtool,arg1[,arg2]…

-c

-g

-g0

-lname

-o pathname

-shared

-v

-w
Compiler Options with a Similar Function (For C/C++Compiler)

Fujitsu Compiler trad mode Option Name

Fujitsu Compiler clang mode Option Name

LLVM Compiler Option Name

-KA64FX

-mcpu=a64fx

-mcpu=a64fx

-KARMV8_1_A

-march=armv8.1-a

-march=armv8.1-a

-KARMV8_2_A

-march=armv8.2-a

-march=armv8.2-a

-KARMV8_3_A

-march=armv8.3-a

-march=armv8.3-a

-KARMV8_A

-march=armv8-a

-march=armv8-a

-KGENERIC_CPU

-mcpu=generic

-mcpu=generic

-K{PIC|pic}

-f{PIC|pic}

-f{PIC|pic}

-Kcmodel={small|large}

-mcmodel={small|large}

-mcmodel={small|large}

-K{eval|noeval}

-f{fast-math|no-fast-math}

-f{fast-math|no-fast-math}

-Kfast

-Ofast

-O3 -ffast-math

-Kfp_contract

-ffp-contract=fast

-ffp-contract={fast|on|fast-honor-pragmas}

-Knofp_contract

-ffp-contract=off

-ffp-contract=off

-Kfp_relaxed

-ffj-fp-relaxed

-freciprocal-math -mrecip

-Knofp_relaxed

-ffj-no-fp-relaxed

-fno-reciprocal-math or -mrecip=none

-K{largepage|nolargepage}

-ffj-{largepage|no-largepage}

Refer to Large Page Function

-K{lib|nolib}

-f{builtin|no-builtin}

-f{builtin|no-builtin}

-Kloop_fission

-ffj-loop-fission

-mllvm -enable-loop-distribute

-K{omitfp|noomitfp}

-f{omit-frame-pointer|no-omit-frame-pointer}

-f{omit-frame-pointer|no-omit-frame-pointer}

-K{openmp|noopenmp}

-f{openmp|no-openmp}

-f{openmp|no-openmp}

-K{openmp_simd|noopenmp_simd}

-f{openmp-simd|no-openmp-simd}

-f{openmp-simd|no-openmp-simd}

-Koptmsg=2

-Rpass=.*

-Rpass=.*

-K{simd|nosimd}

-f{vectorize|no-vectorize}

-f{vectorize|no-vectorize}

-K{strict_aliasing|nostrict_aliasing}

-f{strict-aliasing|no-strict-aliasing}

-f{strict-aliasing|no-strict-aliasing}

-Kswp

-ffj-swp

-mllvm -aarch64-enable-pipeliner

-K{unroll|nounroll}

-f{unroll-loops|no-unroll-loops}

-f{unroll-loops|no-unroll-loops}

-N{exceptions|noexceptions}

-f{exceptions|no-exceptions}

-f{exceptions|no-exceptions}

-V

--version

--version

{-x-|-x0}

-f{inline-functions|no-inline-functions}

-f{inline-functions|no-inline-functions}
Compiler Options with a Similar Function (For Fortran Compiler)

Fujitsu Compiler Option Name

LLVM Compiler Option Name

-KA64FX

-mcpu=a64fx

-KARMV8_1_A

-march=armv8.1-a

-KARMV8_2_A

-march=armv8.2-a

-KARMV8_3_A

-march=armv8.3-a

-KARMV8_A

-march=armv8-a

-KGENERIC_CPU

-mcpu=generic

-K{PIC|pic}

-f{PIC|pic}

-Kcmodel={small|large}

-mcmodel={small|large}

-K{eval|noeval}

-f{fast-math|no-fast-math}

-Kfast

-O3 -ffast-math -fstack-arrays

-Kfp_contract

-ffp-contract={fast|on}

-Knofp_contract

-ffp-contract=off

-Kfp_relaxed

-freciprocal-math -mrecip

-Knofp_relaxed

-fno-reciprocal-math or -mrecip=none

-K{largepage|nolargepage}

Refer to Large Page Function

-K{omitfp|noomitfp}

-f{omit-frame-pointer|no-omit-frame-pointer}

-K{openmp|noopenmp}

-f{openmp|no-openmp}

-Koptmsg=2

-Rpass=.*

-K{simd|nosimd}

-f{vectorize|no-vectorize}

-Kswp

-mllvm -aarch64-enable-pipeliner

-K{unroll|nounroll}

-f{unroll-loops|no-unroll-loops}

-V

--version

3.3.2.2.2. Conditions of Enabling Compiler Options and Default Behavior

For some compiler options in the LLVM compiler, the conditions under which they become active when explicitly specified (such as optimization level or the presence of prerequisite options) and the default behavior when omitted differ from those of the Fujitsu compiler.

The table below shows effective conditions and induction conditions for the Fujitsu Compiler and the LLVM compiler.

For details on compiler options and negative compiler options, refer to the official documentation (*).

(*) The documentation you should refer to varies depending on the LLVM version. Please visit the LLVM Release site at https://releases.llvm.org/ and check the documentation corresponding to the version you are using.

Effective Conditions and Induction Conditions for the Fujitsu Compiler and the LLVM compiler(C/C++)
Fujitsu(trad)
1) Effective Condition
2) Induction Condition
Fujitsu(clang)
1) Effective Condition
2) Induction Condition
LLVM
1) Effective Condition
2) Induction Condition
-x-
1) -O1 or higher
2) -O3 or higher
-finline-functions
1) -O2 or higher
2) -O2 or higher
-finline-functions
1) -O1 or higher
2) -O1 or higher
-Kswp
1) -O1 or higher
2) -O2 or higher
-ffj-swp
1) -O1 or higher
2) None (Explicit specification only)
-mllvm -aarch64-enable-pipeliner
1) -O1 or higher
2) None (Explicit specification only)
-Kfp_contract
1) -O1 or higher
2) -Kfast
-ffp-contract=fast (*1)
1) -O1 or higher
2) -Ofast
-ffp-contract=fast (*2)
1) -O1 or higher
2) -ffast-math
-Kunroll
1) -O1 or higher
2) -O2 or higher
-funroll-loops
1) -O1 or higher
2) -O2 or higher
-funroll-loops
1) -O1 or higher
2) -O2 or higher
-Ksimd[=auto]
1) -O2 or higher
2) -O2 or higher
-fvectorize
1) -O1 or higher
2) -O2 or higher
-fvectorize
1) -O1 or higher
2) -O2 or higher
-Kfp_relaxed
1) -O1 or higher
2) -Kfast
-ffj-fp-relaxed
1) -O1 or higher
2) -Ofast
-freciprocal-math -mrecip
1) -O1 or higher, and -fno-honor-infinities or -freciprocal-math or -ffast-math
2) None (Explicit specification only) (*3)
-Kloop_fission (*4)
1) -O2 or higher
2) -O2 or higher
-ffj-loop-fission (*4)
1) -O2 or higher
2) -O2 or higher
-mllvm -enable-loop-distribute
1) -O1 or higher
2) None (Explicit specification only)
-Komitfp
1) -O1 or higher
2) -Kfast
-fomit-frame-pointer （*5, *6）
1) None (Regardless of the optimization level)
2) -Ofast
-fomit-frame-pointer （*5, *6）
1) None (Regardless of the optimization level)
2) None (Explicit specification only) (*3)

(*1) The Fujitsu compiler (clang mode) also supports the -ffp-contract=on option. -ffp-contract=on is enabled regardless of the optimization level. If the -ffp-contract option is omitted, -ffp-contract=on is applied.

(*2) The LLVM compiler also supports the -ffp-contract=on and the -ffp-contract=fast-honor-pragmas options. -ffp-contract=on is enabled regardless of the optimization level. -ffp-contract=fast-honor-pragmas is enabled at -O1 or higher. If the -ffp-contract option is omitted, -ffp-contract=on is applied.

(*3) While -freciprocal-math is implicitly enabled by -ffast-math, -mrecip is not.

(*4) In Fujitsu compilers, specifying -Kloop_fission (trad mode) / -ffj-loop-fission (clang mode) alone does not perform loop fission. To enable loop fission, you need to specify the target loop with the optimization control specifier “loop_fission_target” or “fission_point” (trad mode only).

(*5) When -fno-omit-frame-pointer is specified for the Fujitsu compiler (clang mode), it also enables -mno-omit-leaf-frame-pointer. In contrast, specifying -fno-omit-frame-pointer for the LLVM compiler does not induce -mno-omit-leaf-frame-pointer.

(*6) When -fomit-frame-pointer is specified for the Fujitsu compiler (clang mode), it also enables -momit-leaf-frame-pointer. In contrast, specifying -fomit-frame-pointer for the LLVM compiler does not induce -momit-leaf-frame-pointer. However, due to the LLVM implementation, the frame pointers for leaf functions are also removed, resulting in behavior equivalent to that of the Fujitsu compiler (in clang mode).

Effective Conditions and Induction Conditions for the Fujitsu Compiler and the LLVM compiler(Fortran)
Fujitsu
1) Effective Condition
2) Induction Condition
LLVM
1) Effective Condition
2) Induction Condition
-Kswp
1) -O1 or higher
2) -O2 or higher
-mllvm -aarch64-enable-pipeliner
1) -O1 or higher
2) -O1 or higher
-Kfp_contract
1) -O1 or higher
2) -Kfast
-ffp-contract=fast
1) -O1 or higher
2) -ffast-math
-Kunroll
1) -O1 or higher
2) -O2 or higher
-funroll-loops
1) -O1 or higher
2) -O2 or higher
-Ksimd[=auto]
1) -O2 or higher
2) -O2 or higher
-fvectorize
1) -O1 or higher
2) -O2 or higher
-Kfp_relaxed
1) -O1 or higher
2) -Kfast
-freciprocal-math -mrecip
1) -O1 or higher, and -fno-honor-infinities or -ffast-math
2) None (Explicit specification only) (*)
-Komitfp
1) -O1 or higher
2) -Kfast
-fomit-frame-pointer
1) None (Regardless of the optimization level)
2) None (Explicit specification only)

(*) While -freciprocal-math is implicitly enabled by -ffast-math, -mrecip is not.

3.3.2.3. Incompatible features

The LLVM compiler does not support Fujitsu compiler and A64FX-specific features. Main incompatible features are listed below.

  1. Fujitsu Compiler-Specific Options

    Specifying Fujitsu compiler-specific compiler options such as the following to the LLVM compiler will result in a compilation error and cause the process to terminate.

    • -K option (Example: -Kfast)

    • -N option (Example: -NRtrap)

    • -ffj options (Example: -ffj-fjprof)

    • Options related to the math libraries(Example: -SSL2, -lfjlapacksve)

    • Other Fujitsu compiler-specific options（Example: -A’name(token)’）

    For options with similar functionality, refer to Table: Compiler Options with a Similar Function (For C/C++Compiler) and Table: Compiler Options with a Similar Function (For Fortran Compiler) in Compiler Options.

  2. Fujitsu Compiler-Specific Optimization Control Specifiers

    If Fujitsu compiler-specific optimization control specifiers are written within the source code, they will not be recognized by the LLVM compiler and will be ignored.

  3. Fujitsu Compiler-Specific Environment Variables

    The following Fujitsu compiler-specific environment variables that used during compilation with the Fujitsu compiler and when executing binaries created by the Fujitsu compiler has no effect on operation with the LLVM compiler and they are ignored.

    • Environment variable for compile command(Example: fcc_ENV)

    • Environment variable for execution(Example: FLIB_C_MESSAGE)

  4. Predefined Macro Names

    Fujitsu compilers predefine macros such as __FUJITSU or __CLANG_FUJITSU, which may be used in conditional branching within source code. These macros are not defined in LLVM compilers. If you are using them for conditional branching, we recommend verifying that the expected processing occurs.

    Refer to Differences in Predefined Macro Names for a detailed list of macros

  5. A64FX-Specific Features (Available through environment variables or options)

    The following features were primarily controlled in Fujitsu compilers through environment variables, compiler options, and optimization control specifiers; however, these control mechanisms are not provided in LLVM compilers. Applications that utilized these features may experience operational differences during migration; therefore, we recommend verifying their functionality through operational testing.

    • Hardware barrier (Example: Environment variables FLIB_BARRIER=HARD)

    • Sector cache function (Example: Optimization control specifier scache_isolate_way L2=n1)

    • Hardware prefetch assistance function (Example: -Khpctag)

  6. Fujitsu Fortran Compiler Runtime Options

    The executable binary created with the Fujitsu Fortran compiler provides “runtime options” that control its behavior using command-line arguments when invoked. The LLVM compiler (flang) does not provide functionality equivalent to these runtime options. Specifying runtime options for an LLVM-compiled executable will have no effect.

  7. Fujitsu Profiler (fipp/fapp)

    The Fujitsu Compiler supports the Fujitsu Profiler (fipp/fapp) for performance analysis, but the LLVM Compiler does not. When using the LLVM compiler, please use general performance analysis tools such as perf.

3.3.2.4. Differences in Predefined Macro Names

There are differences in predefined macros between Fujitsu compilers and LLVM compilers. Some macros are defined only in Fujitsu compilers, and there may be cases where the values of the same macros differ between the two compilers. If these macros are used for conditional branching in the source code, it may affect the behavior. Therefore, it is recommended to check the differences in macro definitions.

3.3.2.4.1. Macros Defined Only in Fujitsu Compiler

The macros defined only in the Fujitsu compiler are shown.

The macros defined only in the Fujitsu C/C++ compiler (trad mode)
Macro Name
Note
__FCC_major__
Common C/C++ Compiler
__FCC_minor__
Common C/C++ Compiler
__FCC_patchlevel__
Common C/C++ Compiler
__FCC_version__
Common C/C++ Compiler
__FUJITSU
Common C/C++ Compiler
__STDC_NO_ATOMICS__
C compiler only
__STDC_NO_THREADS__
C compiler only
__MT__
Common C/C++ Compiler
__SIGNED_CHARS__
Common C/C++ Compiler
_BOOL
C++ compiler only
_GNU_SOURCE
C++ compiler only
_WCHAR_T
C++ compiler only
__ARRAY_OPERATORS
C++ compiler only
__EXCEPTIONS
C++ compiler only
__FJ_RUNTIME_USES_NAMESPACES
C++ compiler only
__GNUG__
C++ compiler only
__PLACEMENT_DELETE
C++ compiler only
__RTTI
C++ compiler only
__FMTL_PARALLEL
C++ compiler only
__FMTL_SERIAL
C++ compiler only
__FMTL_SSL2
C++ compiler only
__FMTL_SSL2BLAMP
C++ compiler only
The macros defined only in the Fujitsu C/C++ compiler (clang mode)
Macro Name
Note
__CLANG_FUJITSU
Common C/C++ Compiler
__FCC_major__
Common C/C++ Compiler
__FCC_minor__
Common C/C++ Compiler
__FCC_patchlevel__
Common C/C++ Compiler
__FCC_version__
Common C/C++ Compiler
The macros defined only in the Fujitsu Fortran compiler
Macro Name
Note
__FRT_major__

__FRT_minor__

__FRT_patchlevel__

__FRT_version__

__FUJITSU

__linux
Undefined in flang, but defined in clang/clang++
__unix
Undefined in flang, but defined in clang/clang++
linux
Undefined in flang, but defined in clang/clang++
unix
Undefined in flang, but defined in clang/clang++

3.3.2.4.2. Macros with Different Values

The macros defined with the same name but different values in both Fujitsu compiler and LLVM compiler are shown.

Macro with a different value compared to Fujitsu C/C++ Compiler (trad mode)
Macro Name
Value of Macro (Fujitsu)
Value of Macro (LLVM)
Note
__GNUC_MINOR__
1
2
Common C/C++ Compiler
__GNUC_PATCHLEVEL__
0
1
Common C/C++ Compiler
__GNUC__
6
4
Common C/C++ Compiler
__STDC_VERSION__
201112L
201710L
C compiler only
__WCHAR_TYPE__
int
unsigned int
Common C/C++ Compiler
_OPENMP
201107
202011
Common C/C++ Compiler
Macro with a different value compared to Fujitsu C/C++ Compiler (clang mode)
Macro Name
Value of Macro (Fujitsu)
Value of Macro (LLVM)
Note
__STDC_VERSION__
201112L
201710L
C compiler only
__clang_major__
7
22
Common C/C++ Compiler
__clang_version__
“7.1.0 “
“22.1.0 “
Common C/C++ Compiler
_OPENMP
201107
202011
Common C/C++ Compiler
Macro with a different value compared to Fujitsu Fortran Compiler
Macro Name
Value of Macro (Fujitsu)
Value of Macro (LLVM)
_OPENMP
201511
202011

3.3.2.5. Differences in Optimization Control Specifiers

There are differences in optimization control specifiers between Fujitsu compiler and LLVM compiler. The correspondence of optimization directives for both compilers, as far as is known at present, is shown in the table below. Note that the conditions of enabling optimization are the same as the conditions of enabling compiler options. For details, refer to Conditions of Enabling Compiler Options and Default Behavior.

Correspondence of optimization control specifiers (C/C++)
Fujitsu Compiler (trad mode)
Fujitsu Compiler (clang mode)
LLVM Compiler
#pragma[ fj] loop simd
#pragma clang loop vectorize(enable)
Same as clang mode
#pragma[ fj] loop nosimd
#pragma clang loop vectorize(disable)
Same as clang mode
(None)
#pragma clang loop vectorize_width(n, scalable)
Same as clang mode
(None)
#pragma clang loop vectorize(assume_safety)
Same as clang mode
#pragma[ fj] loop unroll
#pragma clang loop unroll(enable)
Same as clang mode
#pragma[ fj] loop unroll n
#pragma clang loop unroll_count(n)
Same as clang mode
#pragma[ fj] loop unroll “full”
#pragma clang loop unroll(full)
Same as clang mode
#pragma[ fj] loop nounroll
#pragma clang loop unroll(disable)
Same as clang mode
#pragma loop fp_contract
#pragma clang fp contract(fast)
Same as clang mode
#pragma loop nofp_contract
#pragma clang fp contract(off)
Same as clang mode (*1)
(None)
#pragma clang fp contract(on)
Same as clang mode
#pragma loop loop_fission_target
#pragma fj loop loop_fission_target
#pragma clang loop distribute(enable) (*2)
#pragma procedure fp_relaxed
(None)
#pragma clang fp reciprocal(on)
#pragma procedure nofp_relaxed
(None)
#pragma clang fp reciprocal(off)

(*1) If compiler option -ffp-contract=fast is specified, the option takes precedence and ignores the “#pragma clang fp contract(off)” specification. If compiler option -ffp-contract=fast-honor-pragmas is specified, the “#pragma clang fp contract(off)” specification takes precedence.

(*2) “#pragma clang loop distribute(enable)” in the LLVM compiler instructs loop distribution, similar to the” loop_fission_target” optimization control specifier in the Fujitsu compiler, but the algorithms differ, so the same effect is not guaranteed.

Correspondence of optimization control specifiers (Fotran)
Fujitsu Compiler
LLVM Compiler
OCL SIMD
dir$ vector always
OCL NOSIMD
dir$ novector
OCL UNROLL[(n)]
dir$ unroll [n]
OCL NOUNROLL
dir$ nounroll

3.3.2.6. Substitute

3.3.2.6.1. Large Page Function

The Fujitsu compiler had large page functionality enabled by default, making it available without requiring any special configuration. This may result in improved performance depending on the environment and application. On the other hand, this feature is not enabled by default in the LLVM compiler, so explicit specification is required to use it.

In the Fugaku environment, you can use the large page feature of the HPC extension by adding the following to the compiler (linker) options.

-L/opt/FJSVxos/mmm/lib64 -lmpg -Wl,-T/opt/FJSVxos/mmm/util/bss-2mb.lds

Specify this option before any other libraries. For details on the Large Page function, refer to the manual “Job Operation Software End-user’s Guide for HPC Extensions”.

Compilation Example:

clang a.c -L/opt/FJSVxos/mmm/lib64 -lmpg \
  Wl,-T/opt/FJSVxos/mmm/util/bss-2mb.lds

Note

The memory allocation behavior in the LLVM compiler may differ from that in the Fujitsu compiler. We recommend verifying that the migration functions as intended through actual builds and operational testing.

3.3.2.7. Object and Source Incompatibility

3.3.2.7.1. Object Incompatible

Object files, assembler source files, module files (.mod) and etc. generated by Fujitsu compilers (trad mode or clang mode) and LLVM compilers are incompatible. Therefore, all source code must be recompiled with the LLVM compiler.

Although Fujitsu clang mode is LLVM-based, it includes Fujitsu’s unique extensions, and its behavior differs from pure LLVM. Even if partial compatibility appears to exist, reusing objects is not recommended as it may lead to unintended behavior.

3.3.2.7.2. Source Incompatibility

The LLVM compiler adheres to standard language specifications (C/C++/Fortran), but proprietary intrinsic functions and service routines that were available with Fujitsu compilers are not supported. Source code that depends on these may require modifications when using LLVM.

3.3.2.7.3. Execution Result Incompatibility

Even with identical source code, there may be differences in execution results between Fujitsu compilers and LLVM compilers. This incompatibility primarily stems from the following two points. During post-migration verification, it is recommended to confirm numerical consistency and output format compatibility.

  1. Differences in numerical calculation precision and error
    Differences in compiler optimization and implementations of numerical calculation libraries may lead to variations in the precision and error of numerical calculations.
  2. Differences in output specifications
    The appearance and format of output results, such as floating-point display format (number of digits, exponential notation, handling of spaces, etc.), may differ.

3.3.2.8. About Filename Suffixes

Indicates whether filename suffixes used with Fujitsu compilers (trad mode/clang mode) are available for use with LLVM compilers (clang, clang++, flang).

C compiler clang can compile files with the suffixes listed in the table below as input.

File type

File suffix

C source file

.c

C source file

.i

Assembler source file

.s

Assembler source file that must be preprocessed

.S

Object file

.o

C++ compiler clang++ can compile files with the suffixes listed in the table below as input.

File type

File suffix

C++ source file

.cc

C++ source file

.CC

C++ source file

.cpp

C++ source file

.CPP

C++ source file

.cxx

C++ source file

.CXX

C++ source file

.c++

C++ source file

.C++

C++ source file

.C

C++ source file

.cp

C++ source file

.ii

Assembler source file

.s

Assembler source file that must be preprocessed

.S

Object file

.o

Fortran compiler flang can compile files with the suffixes listed in the table below as input. The file format (fixed format/free format) is automatically determined based on the file extension, so no special option specification is required.

Assembler source files (with the .s extension) cannot be compiled with flang and will cause errors during compilation. Please use clang or clang++ for compiling.

File type

File suffix

Format

Fortran source program

.f

Fixed Format

Fortran source program

.for

Fixed Format

Fortran source program

.f90

Free Format

Fortran source program

.f95

Free Format

Fortran source program

.f03

Free Format

Fortran source program

.f08

Free Format

Preprocessing directive Fortran source program

.F

Fixed Format

Preprocessing directive Fortran source program

.FOR

Fixed Format

Preprocessing directive Fortran source program

.F90

Free Format

Preprocessing directive Fortran source program

.F95

Free Format

Preprocessing directive Fortran source program

.F03

Free Format

Preprocessing directive Fortran source program

.F08

Free Format

Object file

.o

-

3.3.2.9. SIMD Built-in Functions

Fujitsu Compiler (clang mode) supports the SIMD built-in functions documented in “Arm C Language Extensions” version 00bet1 (First public release).

The LLVM compiler (clang) is progressively implementing support for SIMD built-in functions based on the ACLE (Arm C Language Extensions) specification published by Arm. However, it does not fully comply with the latest ACLE specification, and due to specification changes or implementation differences, it may not function as expected. Therefore, when migrating, we recommend verifying whether the SIMD built-in functions you intend to use are supported in the LLVM environment through actual builds and functional verification.

For details on SIMD built-in functions, please refer to the documentation published by Arm for developers. The specifications can be found on GitHub at <https://github.com/ARM-software/acle/releases>.

3.3.2.10. Compatibility with GNU Specifications (C/C++ and Fortran)

The LLVM compilers (clang and flang) support some of the extensions and compatibility options available in the GNU C/C++ and GNU Fortran compilers. Many features used in Fujitsu compilers (trad/clang mode) are likely available in the LLVM environment as well. However, specification changes or implementation differences may affect behavior. Therefore, during migration, we recommend verifying that the extended syntax and compatibility options you use function as intended in the LLVM environment through actual builds and functional testing.

3.3.2.11. Language Specifications

3.3.2.11.1. Supported Language Standard

The following table shows the differences in the language standard supported by the Fujitsu compiler and the LLVM compiler. The notation “(partial)” indicates that the compiler in question does not support all features of the specified standard, but only provides partial compliance.

For details on the status of each feature, please refer to the following documentation.

  • Fujitsu Compiler Support Status
    “Supercomputer Fugaku User Guide - Language and development environment”
  • LLVM Compiler (clang/flang) Support Status
    C Support in Clang <https://clang.llvm.org/c_status.html>
    C++ Support in Clang <https://clang.llvm.org/cxx_status.html>
    Flang Fortran Standards Support (*)
    C++ Standard Library (*)
    (*) Since the documentation you should refer to varies depending on the LLVM version, please visit the LLVM Release site (https://releases.llvm.org/) and check the documentation corresponding to the version you are using.
Comparison of language standards between Fujitsu compiler and LLVM compiler

Item

Fujitsu Compiler trad mode

Fujitsu Compiler clang mode

LLVM Compiler

C Language Standard


C11 Standard
C99 Standard
C89 Standard


C11 Standard
C99 Standard
C89 Standard
C23 Standard (part)
C17 Standard (part)
C11 Standard (part)
C99 Standard
C89 Standard

C++ Language Standard


C++17 Standard (part)
C++14 Standard
C++11 Standard
C++03 Standard


C++17 Standard
C++14 Standard
C++11 Standard
C++03 Standard
C++23 Standard (part) (*1)
C++20 Standard (part) (*1)
C++17 Standard
C++14 Standard
C++11 Standard
C++03 Standard

Fortran Language Standard

Fortran2018 Standard (part)
Fortran2008 Standard
Fortran2003 Standard
Fortran95 Standard
Fortran90 Standard
Fortran77 Standard

None
Fortran2023 Standard (part)
Fortran2018 Standard (part)
Fortran2008 Standard (part)
Fortran2003 Standard (part)
Fortran95 Standard
Fortran90 Standard
Fortran77 Standard

(*1): Refer to Note When Using the Latest Standards of C++20 or Later.

3.3.2.11.1.1. Note When Using the Latest Standards of C++20 or Later

When processing based on the language specifications of the latest C++ language standard, it is recommended to use the -stdlib=libc++ option. On the other hand, when using libstdc++, by default, the libstdc++ included in the standard GCC 8.5 of RHEL 8.10 is used, and this libstdc++ only supports up to part of the C++20 standard. To use newer standards, refer to How to Use the Latest Standards with libstdc++.

Following is an example of a command to compile a C++ program that conforms to the C++20 standard using libc++.

clang++ -std=c++20 -stdlib=libc++ samplecxx20.cpp

3.3.2.11.1.2. How to Use the Latest Standards with libstdc++

When using the latest standards (C++20 or later) with libstdc++, you need to set the GCC installation directory (GCC_INSTALL_DIR) as follows during both compilation and execution. clang++ uses the libstdc++ located in the specified installation directory.

In the Fugaku environment, the GCC provided by Spack is usually used. The installation directory will be mentioned later.

[Setting at Compilation] Please add the following to the compiler options.

In the case of a native compiler

--gcc-install-dir=${GCC_INSTALL_DIR}

In the case of a cross compiler

--gcc-install-dir=${GCC_INSTALL_DIR} -L${GCC_INSTALL_DIR}/../../../../lib64

[Setting at Execution] Please add the following to the environment variable LD_LIBRARY_PATH.

export LD_LIBRARY_PATH=${GCC_INSTALL_DIR}/../../../../lib64:${LD_LIBRARY_PATH}

In the Fugaku environment, the installation directory of GCC provided by Spack is used. The current versions of GCC available on Fugaku and their associated installation directories are shown in Table GCC version and the installation directory below.

GCC version and the installation directory
GCC Version
Installation Directory (GCC_INSTALL_DIR)
GCC 12.2.0
/vol0004/apps/oss/spack-v1.0.1/opt/spack/linux-a64fx/gcc-12.2.0-f57uyl2rzc74cow54td7bdy77xajibir/lib/gcc/aarch64-unknown-linux-gnu/12.2.0
GCC 13.4.0
/vol0004/apps/oss/spack-v1.0.1/opt/spack/linux-a64fx/gcc-13.4.0-qiftl5lzmfkexynrq6bam2ioglq4v6ex/lib/gcc/aarch64-unknown-linux-gnu/13.4.0
GCC 14.3.0
/vol0004/apps/oss/spack-v1.0.1/opt/spack/linux-a64fx/gcc-14.3.0-cnjx4mhazorywcmgf6qpzv5y5qfqr4la/lib/gcc/aarch64-unknown-linux-gnu/14.3.0
GCC 15.1.0
/vol0004/apps/oss/spack-v1.0.1/opt/spack/linux-a64fx/gcc-15.1.0-c3wm4pbp52oyk3h4hzdg4pg43kvcy4bf/lib/gcc/aarch64-unknown-linux-gnu/15.1.0

In a Spack environment, when the compiler environment being referenced changes due to a GCC version upgrade, you can check the installation directory by using the gcc command option -print-search-dirs after setting up the target GCC environment on the compute node.

Please use the installation directory path obtained from this command output.

$ . /vol0004/apps/oss/spack/share/spack/setup-env.sh
$ spack load gcc@12.2.0  # 使用するgccを指定する
$ gcc -print-search-dirs |grep install
install: /vol0004/apps/oss/spack-v1.0.1/opt/spack/linux-a64fx/gcc-12.2.0-f57uyl2rzc74cow54td7bdy77xajibir/lib/gcc/aarch64-unknown-linux-gnu/12.2.0/

3.3.2.11.2. OpenMP API Specifications

The following table shows the differences in the supported OpenMP API specification. The notation “(partial)” indicates that the compiler in question does not support all features of the specified standard, but only provides partial compliance. For specifications without “(Part)”, only the latest supported version based on official documentation is listed.

For Fujitsu compilers, the scope of support is defined as part of the product’s operational guarantee and technical support. For LLVM compilers, the support status is indicated based on the presence or absence of implemented features.

Item

Fujitsu Compiler trad mode

Fujitsu Compiler clang mode

LLVM Compiler

C/C++ OpenMP


OpenMP 4.5 (part)
OpenMP 4.0 (part)
OpenMP 3.1

OpenMP 5.0 (part)
OpenMP 4.5
OpenMP 5.1 (part)
OpenMP 5.0 (part)
OpenMP 4.5

Fortran OpenMP
OpenMP 5.0 (part)
OpenMP 4.5

なし


OpenMP 4.0 (part)
OpenMP 3.1
  • Fujitsu Compiler Support Status
    “C User’s Guide”, “C++ User’s Guide”, “Fortran User’s Guide”
    “Supercomputer Fugaku User Guide - Language and development environment”
  • LLVM Compiler (clang/flang) Support Status
    clang OpenMP Support (*)
    flang OpenMP Support (*)
    (*) Since the documentation you should refer to varies depending on the LLVM version, please visit the LLVM Release site (https://releases.llvm.org/) and check the documentation corresponding to the version you are using.

3.3.2.12. Case Studies

We also have a separate case study collection that summarizes specific challenges and solutions when the migration from Fujitsu compiler to LLVM compiler.

Please refer to ‘Case Studies’.

3.3.2.13. Unsupported Usage

An experimental linkage of Fujitsu BLAS and LAPACK library in the LLVM environment is shown below.

Use of libraries provided with the Fujitsu compiler with the other compiler environments is not supported. Please be aware of this limitation in advance.

The following two libraries are considered.

-lfjlapacksve : BLAS and LAPACK, serial version
-lfjlapackexsve : BLAS and LAPACK, multi-threaded version

Example specification of libraries at compile time

We have confirmed that the following configuration completes successfully at link time.

INCLUDE="-idirafter /opt/FJSVxtclanga/tcsds-latest/include"
LIBS="-L /opt/FJSVxtclanga/tcsds-latest/lib64"
FLIB="-lfjlapacksve -lfj90i -lfj90f -lfjsrcinfo -lfjcrt -lelf" # BLAS, LAPACK serial version
FLIB_M="-lfjlapackexsve -lfjomphk -lfjomp -lfj90i -lfj90f -lfjsrcinfo -lfjcrt -lfjompcrt -lelf" # BLAS, LAPACK, multi-thread version

clang++ -O2 sample.cpp ${INCLUDE} ${LIBS} ${FLIB}
clang++ -O2 sample.cpp ${INCLUDE} ${LIBS} ${FLIB_M}
flang -O2 sample.f ${LIBS} ${FLIB}
flang -O2 sample.f ${LIBS} ${FLIB_M}

If any issue occur at runtime, please discontinue use.