9.21 Handle Too-Long Command Lines with xargs

xargs is one of those UNIX utilities that seems pretty useless when you first hear about it - but turns into one of the handiest tools you can have.

xargs reads a group of arguments from its standard input, then runs a UNIX command with that group of arguments. It keeps reading arguments and running the command until it runs out of arguments. The shell's backquotes (9.16) do the same kind of thing, but they give all the arguments to the command at once. This can give you a Too many arguments (9.20) error.

Here are a couple of examples:

• If you want to print most of the files in a large directory, put the output of ls into a file. Edit the file to leave just the filenames you want printed. Give the file to xargs' standard input:

  <	% ls > allfiles.tmp % vi allfiles.tmp % xargs lpr < allfiles.tmp

  What did that do? With lines like these in allfiles.tmp:

  % cat allfiles.tmp
  afile
  application
     ...
  yoyotest
  zapme

  xargs ran one or more lpr commands, each with a group of arguments, until it had read every word in the file:

  lpr afile application ...
     ...
  lpr ... yoyotest zapme

• The standard output of xargs is the standard output of the commands it runs. So, if you'd created allfiles.tmp above but you wanted to format the files with pr (43.7) first, you could type:

  % xargs pr < allfiles.tmp | lpr

  Then xargs would run all of these pr commands. The shell would pipe their standard outputs [4] to a single lpr command:

  [4] Actually, the shell is piping the standard output of xargs. As I said above, xargs sends the standard output of commands it runs to its own standard output.

  pr afile application ...
     ...
  pr ... yoyotest zapme

  In this next example, find (17.1) gets a list of all files in the directory tree. Next, we use xargs to read those filenames and run grep -l (15.7) to find which files contain the word "WARNING." Next, we pipe that to a setup with pr and lpr, like the one in the previous example:

  % find . -type f -print | xargs grep -l WARNING | xargs pr | lpr

  "Huh?" you might say. Just take that step by step. The output of find is a list of filenames, like ./afile ./bfile ... ./adir/zfile and so on. The first xargs gives those filenames to one or more grep -l commands:

  grep -l WARNING ./afile ./bfile ...
     ...
  grep -l WARNING ./adir/zfile ...

  The standard output of all those greps is a (shortened) list of filenames that match. That's piped to another xargs-it runs pr commands with the filenames that grep found.

  UNIX is weird and wonderful!

• Sometimes you don't want xargs to run its command with as many arguments as it can fit on the command line. The -n option sets the maximum number of arguments xargs will give to each command. Another handy option, -p, prompts you before running each command.

  Here's a directory full of files with errors (whose names end with .bad) and corrected versions (named .fixed). I use ls to give the list of files to xargs; it reads two filenames at once, then asks whether I want to run diff -c to compare those two files. It keeps prompting me and running diff -c until it runs out of file pairs:

  % ls
  chap1.bad
  chap1.fixed
  chap2.bad
  chap2.fixed
      ...
  chap9.bad
  chap9.fixed
  % ls | xargs -p -n2 diff -c
  diff -c chap1.bad chap1.fixed ?...y
     ...Output of diff command for chap1...
  diff -c chap2.bad chap2.fixed ?...n
  diff -c chap3.bad chap3.fixed ?...y
     ...Output of diff command for chap3...
      ...

As the next article (9.22) explains, xargs can have trouble if an argument has white space inside a word. Luckily, the GNU xargs (read about it there) solves the problem.