kmrshell_mpi

NAME
SYNOPSIS
DESCRIPTION
OPTIONS

NAME

kmrshell_mpi − map-reduce by shell command pipeline

SYNOPSIS

kmrshell_mpi -n m_num[:r_num] -m mapper [-k kvgenerator] [-r reducer] inputfile

DESCRIPTION

kmrshell starts map-reduce shell processes (for "streaming"). It runs MPI program as mapper and reducer. The output files of the mapper are parsed by a key-value generator program (specified by ’-k’ option) to generate key-value pairs and then they are shuffled and written to files on each rank with ’key’-named as a text file whose line represents a key-value separated by a space. The file is passed to the reducer as the last parameter.

OPTIONS

The following options are supported:

-n m_num[:r_num]

......

-m mapper

Specifies a mapper program. The program can have arguments, where they are separated by a whitespace.

Mapper specification: A mapper reads data from STDIN and outputs key-value data to STDOUT. Output data is a sequence of a line "key value\n", where the fields are separated by a whitespace.

-k kvgenerator

......

-r reducer

Specifies a reducer program. The program can have arguments, where they are separated by a whitespace.

Reducer Specification: A reducer reads key-value data from STDIN and outputs the result to STDOUT. Input data is a sequence of a line "key value\n", where the fields are separated by a whitespace. Lines with the same keys constitutes consecutive lines.