KMR
Macros | Enumerations | Functions | Variables
kmrshell.c File Reference

KMR-Shell for Streaming. More...

#include <stdio.h>
#include <stdlib.h>
#include <stddef.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <signal.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <sys/param.h>
#include <dirent.h>
#include <assert.h>

Go to the source code of this file.

Macros

#define ARGSIZ   8
 Maximum number of arguments to mapper and reducer programs. More...
 
#define ARGSTRLEN   (8 * 1024)
 Buffer string size of arguments to mapper and reducer programs. More...
 
#define LINELEN   32767
 Maximum length of a line of data. More...
 

Enumerations

enum  { PIPE_IN = 1, PIPE_OUT = 2 }
 Parameter for pipeops(). More...
 

Functions

static int execute (char *prog, char *const args[], pid_t *cldpid)
 Forks and execs a process. More...
 
static int filereader (char *path)
 Reads files (file-reader). More...
 
int main (int argc, char *argv[])
 Starts map-reduce shell processes (for "streaming"). More...
 
static void parse_args (char *argstr, char *argary[])
 Parses command parameters given for mapper and reducer arguments. More...
 
static int pipeops (int *pipefd, int direc)
 Sets up pipe states. More...
 
static void putfilecontents (char *)
 
static void reapchild (int exitstat)
 Handles SIGCHLD signal. More...
 

Variables

static char * shuffler = "kmrshuffler"
 

Detailed Description

KMR-Shell for Streaming.

It forks processes of a mapper, a shuffler, and a reducer, then, it reads a number of specified input files and passes their data to a mapper via a pipe. A mapper, a shuffler, and a reducer are shell executables.

Definition in file kmrshell.c.

Macro Definition Documentation

◆ LINELEN

#define LINELEN   32767

Maximum length of a line of data.

Definition at line 25 of file kmrshell.c.

◆ ARGSIZ

#define ARGSIZ   8

Maximum number of arguments to mapper and reducer programs.

Definition at line 28 of file kmrshell.c.

◆ ARGSTRLEN

#define ARGSTRLEN   (8 * 1024)

Buffer string size of arguments to mapper and reducer programs.

Definition at line 31 of file kmrshell.c.

Enumeration Type Documentation

◆ anonymous enum

anonymous enum

Parameter for pipeops().

Enumerator
PIPE_IN 

Attach stdin to pipe.

PIPE_OUT 

Attach stdout to pipe.

Definition at line 34 of file kmrshell.c.

Function Documentation

◆ pipeops()

static int pipeops ( int *  pipefd,
int  direc 
)
static

Sets up pipe states.

It attaches pipes to stdin or stdout.

Definition at line 61 of file kmrshell.c.

◆ execute()

static int execute ( char *  prog,
char *const  args[],
pid_t *  cldpid 
)
static

Forks and execs a process.

Parameters
progprogram path.
argsargments to program.
cldpidpid of child (out).

Definition at line 102 of file kmrshell.c.

◆ reapchild()

static void reapchild ( int  exitstat)
static

Handles SIGCHLD signal.

Definition at line 51 of file kmrshell.c.

◆ filereader()

static int filereader ( char *  path)
static

Reads files (file-reader).

It reads possibly multiple files and writes their contents to stdout. If "path" is a directory, it enumerates the files under the directory and reads each file.

Parameters
pathpath to file (or directory).

Definition at line 186 of file kmrshell.c.

◆ parse_args()

static void parse_args ( char *  argstr,
char *  argary[] 
)
static

Parses command parameters given for mapper and reducer arguments.

It scans an argument string like "mapper arg0 arg1" for the -m and -r options, and generates an argv array {"mapper", "arg0", "arg1", 0}. The separator is a whitespace.

Parameters
argstrstring given for -m or -r options.
argaryarray to be filled by argument strings.

Definition at line 257 of file kmrshell.c.

◆ main()

int main ( int  argc,
char *  argv[] 
)

Starts map-reduce shell processes (for "streaming").

It forks and execs a mapper, a shuffler, and a reducer, which are connected via pipes.

  • -m mapper program.
  • -r reducer program.
  • input file or directory.

Definition at line 293 of file kmrshell.c.