kmrrungenscript.py − generates a job-script for kmrrun
kmrrungenscript.py
[-q queue] [-t rsctime] [-e node] [-s shape]
[-p proc] [-d dir] [-n task_proc]
-m mapper [-k kvgenerator] [-r reducer]
[-C] [-R ckptdir] [-S sched] [-w file]
kmrrungenscript.py generates a job-script that executes kmrrun once for a specified job scheduler. Currently, it supports "Parallelnavi" on K computer and SLURM on FOCUS supercomputer as job schedulers.
The following options are supported:
-q queue, --queue=queue
Specifies a queue to submit the generated job-script.
-t rsctime, --resource-time=rsctime
Specifies time limit in the job resources. This is given by "00:00:00" format. Default is "00:10:00".
-e node, --number-of-exec-node=node
Specifies the number of nodes (processes) to execute. Default is 12.
-s shape, --shape=shape
Specifies the MPI process shape. Valid only on K scheduler. Default is 1.
-p procs, --proc=procs
Specifies the number of MPI processes. Valid only on K scheduler. Default is 8.
-d indir, --inputdir=indir
Specifies the input directory. When used on K computer, this directory should be one located in K global storage that is staged-in. Default is "./input".
-n task_proc, --task-proc=task_proc
Specifies the number of processes to run each mapper and reducer. Default is 1.
-m mapper, --mapper=mapper
Specifies mapper program and its arguments. A mapper program can be a serial or an MPI program.
-k kvgenerator, --kvgen=kvgenerator
Specifies key-value generator program and its arguments. A key-value generator program should be a serial program.
-r reducer, --reducer=reducer
Specifies reducer program and its arguments. A reducer program can be a serial or an MPI program.
-C, -ckpt
Enables checkpoint/restart.
-R ckptdir, --restart-filename=ckptdir
Specifies prefix of directories where checkpoint files are located. This option should be given when restarting on a system that requires staging. Valid only on K scheduler.
-S sched, --scheduler=sched
Specifies a job scheduler. Specify Scheduler ’K’ or ’FOCUS’. ’K’ supports K computer/FX10 and ’FOCUS’ supports Focus supercomputer. Default is ’K’.
-w scrfile, --write-scriptfile=scrfile
Specifies a script output file. Default is STDOUT.
For example, the following script is generated.
$
kmrrungenscript.py -q small -e 10 -s 2 -p 2 -d input \
-m pi.mapper -k pi.kvgen.sh -r pi.reducer -C -R
ckptdir_12345
It will generate the following job script.
#!/bin/bash -x
#
#PJM --rsc-list "rscgrp=small"
#PJM --rsc-list "elapse=00:10:00"
#PJM --rsc-list "node=10"
#PJM --mpi "shape=2"
#PJM --mpi "proc=2"
#PJM --rsc-list "proc-core=unlimited"
#PJM --stg-transfiles "all"
#PJM --stgin "./kmrrun ./kmrrun"
#PJM --stgin "./pi.mapper ./pi.mapper"
#PJM --stgin "./pi.kvgen.sh ./pi.kvgen.sh"
#PJM --stgin "./pi.reducer ./pi.reducer"
#PJM --stgin "./input/* ./input/"
#PJM --stgin "./ckptdir_12345.00000/*
./ckptdir00000/"
#PJM --stgin "./ckptdir_12345.00001/*
./ckptdir00001/"
#PJM --stgin "./ckptdir_12345.00002/*
./ckptdir00002/"
#PJM --stgin "./ckptdir_12345.00003/*
./ckptdir00003/"
#
# !!WRITE STGOUT HERE!!
#
#PJM --stgout "./ckptdir00000/*
./ckptdir_%j.00000/"
#PJM --stgout "./ckptdir00001/*
./ckptdir_%j.00001/"
# !!PJM --stgout "./core* ./"
#PJM -S
. /work/system/Env_base
mpiexec -n 2
./kmrrun -n 1 -m "./pi.mapper" -k
"./pi.kvgen.sh" \
-r "./pi.reducer" --ckpt ./input