29.7 Count How Many Times Each Word Is Used

Here's this book's Preface run through wordfreq:

% wordfreq ch00
 141 the
  98 to
  84 and
  84 of
  71 a
  55 in
  44 that
  38 book
  32 we
  ...

The script was taken from a long-ago Usenet (1.33) posting by Carl Brandauer. Here is Carl's original script (with a few small edits):

tr sort uniq -4

cat $* | # tr reads the standard input tr "[A-Z]" "[a-z]" | # Convert all uppercase to lowercase tr -cs "a-z'" "\012" | # replace all characters not a-z or ' # with a new line. i.e. one word per line sort | # uniq expects sorted input uniq -c | # Count number of times each word appears sort +0nr +1d | # Sort first from most to least frequent, # then alphabetically pr -w80 -4 -h "Concordance for $*" # Print in four columns

The version on the disc is somewhat different. It adjusts the tr commands for the script's -i option. The disc version also doesn't use pr to make output in four columns, though you can add that to your copy of the script - or just pipe the wordfreq output through pr on the command line when you need it.

The second tr command above (with the -cs options) is for the Berkeley version of tr. For System V tr, the command should be:

tr -cs "[a-z]'" "[\012*]"

If you aren't sure which version of tr you have, see article 35.11. You could use deroff (29.10) instead.

One of the beauties of a simple script like this is that you can tweak it if you don't like the way it counts. For example, if you want hyphenated words like copy-editor to count as one, add a hyphen to the tr -cs expression: "[a-z]'-" (System V) or "-a-z'" (Berkeley).