split-exec

NAME

split-exec - Fork the command for its split of input

SYNOPSIS

split-exec [ options ] command ...

split-exec --help

split-exec --man

DESCRIPTION

This program will fork a new child for each chunk of input.

The assumption is that the child will take a "long time" before it produces any output but that once it starts producing output, it will complete "quickly".

The potential performance gain comes from the ability to overlap this "long time", for example, input-output tasks like "count the number of lines in all files in the given directory".

Both the command and the environment are implicitly trusted and no checks are made to ensure they will not result in something undesired. For example, $PATH is propagated as-is.

OPTIONS

--help

Print this brief help message and exit.

--man

Display the manual page.

--children=i

The maximum number of children to have at any one time. This only includes the children this program creates --- the provided command may fork multiple children. The default is 4.

--lines=i

How many sequential lines to pass to each child. The last child will not get this number of lines if the number of input lines is not a whole multiple of this number. The default is 1.

To avoid the obvious deadlock, all the lines are read in and then child is executed.

--separation=i

The minimum number of wallclock seconds separation between successive forks to create children. The default is 0.

--timeout=i

Children that have executed longer than this many wall-clock seconds may be terminated. This is a minimum and children may survive longer before being killed. The default, 0, means no timeout is used.

EXAMPLES

Here are some contrived examples, mainly to demonstrate the syntax and how this is meant to work. Hopefully, you have real work to do and this script will be of some use to you.

    perl -e 'print join "\n", 1..20, "\n";' | split-exec factor

    perl -e 'print "dummy\n" x 20' | split-exec --children=5 -- perl -pe 'sleep rand 10'

    perl -e 'print "dummy\n" x 20' | split-exec --timeout=5 -- perl -pe 'sleep rand 10'

EXIT STATUS

This script will exit with zero upon successful completion, non-zero on any error. All debugging or diagnostic output is written to STDERR. Please do not ignore STDERR.

VERSION

    $Id: split-exec.html,v 1.4 2017/01/11 01:40:39 suter Exp $

COPYRIGHT

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.