kmrshell_mpi.c File Reference

kmrshell_mpi is command line version of KMR and it runs a MapReduce program whose mapper and reducers are user specified MPI programs. More...

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdarg.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/param.h>
#include <dirent.h>
#include <unistd.h>
#include <mpi.h>
#include "kmr.h"

Go to the source code of this file.

Classes
struct	cmdinfo

Macros
#define	ARGSIZ   8
#define	ARGSTRLEN   (8 * 1024)
#define	COMMANDLEN   1024
#define	DEFAULT_MPI   2
#define	LINELEN   32767

Functions
static int	add_command_kv (KMR_KVS *, int, char **, char *, int)
static int	delete_file (const struct kmr_kv_box, const KMR_KVS *, KMR_KVS *, void *, long)
static int	generate_mapcmd_kvs (const struct kmr_kv_box, const KMR_KVS *, KMR_KVS *, void *, long)
static int	generate_redcmd_kvs (const struct kmr_kv_box, const KMR_KVS *, KMR_KVS *, void *, long)
static void	kmrsh_abort (int, const char *,...)
int	main (int argc, char *argv[])
static void	parse_args (char *, char *[])
static int	run_kv_generator (const struct kmr_kv_box, const KMR_KVS *, KMR_KVS *, void *, long)
static int	sleep_wait (const struct kmr_kv_box, const KMR_KVS *, KMR_KVS *, void *, long)
static int	write_kvs (const struct kmr_kv_box[], const long, const KMR_KVS *, KMR_KVS *, void *)

Detailed Description

kmrshell_mpi is command line version of KMR and it runs a MapReduce program whose mapper and reducers are user specified MPI programs.

When kmrshell_mpi is used to run a MapReduce program, user should specify a simple program that generates key-value pairs from the output of mapper. The key-value generator program can be specified by '-k' option and can be implemented by reading outputs of mapper and then writing key-value pairs to the standard output. After shuffling the key-value paris, key-value pairs are written to files on each rank with 'key'-named text files whose line represents a key-value separated by a space. The file is passed to the reducer as the last parameter.

kmrshell_mpi can run Map-only MapReduce where no reducer is run. This is very useful if you want to run multiple tasks as a single job.

Options

• -m mapper [Mandatory]
  • Specify a mapper program (MPI)
• -k key_value_generator [Optional]
  • Specify a key-value pair generator program (serial)
• -r reducer [Optional]
  • Specify a reducer program (MPI)
• -n m_num[:r_num] [Optional]
  • Specify number of MPI processes to run mapper and reducers. When r_num is specified, each mapper runs with m_num processes and each reducer runs with r_num processes. When r_num is not specified each mapper and reducer runs with m_num processes. The default is 2.

Usage

$ mpiexec -machinefile machines -n 4 \
./kmrshell_mpi -n m_num[:r_num] -m mapper [-k kvgenerator] [-r reducer] \
inputfile

Examples

e.g.1) Run MPI mapper and MPI reducer with 2 MPI processes each.
    $ mpiexec -np 2 ./kmrshell_mpi -n 2 -m "./mpi_pi.mapper" -k "./mpi_pi.kvgen.sh" -r "./mpi_pi.reducer" ./work

e.g.2) Run MPI mapper with 2 MPI processes and MPI reducer with 4 processes.
    $ mpiexec -np 2 ./kmrshell_mpi -n 2:4 -m "./mpi_pi.mapper" -k "./mpi_pi.kvgen.sh" -r "./mpi_pi.reducer" ./work

e.g.3) Only run MPI mapper with 2 MPI processes
    $ mpiexec -np 2 ./kmrshell2 -n 2 -m "./mpi_pi.mapper" ./work

Definition in file kmrshell_mpi.c.