How do I apply a shell command to many files in nested (and poorly escaped) subdirectories?

Hi! I'm trying to do something like the following:

for file in `find . *.foo`
do
somecommand $file
done

But the command isn't working because $file is very odd. Because my directory tree has crappy file names (including spaces), I need to escape the find command. But none of the obvious escapes seem to work: -ls gives me the space-delimited filename fragments -fprint doesn't do any better.

I also tried: for file in "find . *.foo -ls"; do echo $file; done - but that gives all of the responses from find in one long line.

Any hints? I'm happy for any workaround, but am frustrated that I can't figure this out.

Thanks, Alex

(Hi Matt!)

From stackoverflow

• Instead of relying on the shell to do that work, rely on find to do it:

  find . -name "*.foo" -exec somecommand "{}" \;
  

  Then the file name will be properly escaped, and never interpreted by the shell.

  dfa : for GNU find, -exec somecommand {} + is better

• xargs is your friend. You will also want to investigate the -0 (zero) option with it. find (with -print0) will help to produce the list. The Wikipedia page has some good examples.

  Another useful reason to use xargs, is that if you have many files (dozens or more), xargs will split them up into individual calls to whatever xargs is then called upon to run (in the first wikipedia example, rm)

  Steve Jessop : "xargs will split them up into individual calls". Or if this isn't desirable, as happens in rare cases, then use "-n1 -r" to get the same behaviour as the for loop. -r is a GNU extension, -n is POSIX.

  Alister Bulman : +1 for the tip, onebyone

  lhunath : -1 because xargs is broken unless used with the -0 option and using the -0 option with find is pretty silly seeing as find itself has a -exec {} + predicate that does exactly the same thing.

  Alister Bulman : '-0 |xargs' is easier to remember than escaping the call to '-exec {}' however, which can get complicated.

• find . -name '*.foo' -print0 | xargs -0 -n 1 somecommand
  

  It does get messy if you need to run a number of shell commands on each item, though.

  Steve Jessop : Actually, you need -r (and hence GNU find or similar) to exactly emulate the for loop. xargs will otherwise execute the command with no arguments if the input is 0-length.

  lhunath : or you could use find's -exec {} + to avoid the convolution: find . -name '*.foo' -exec somecommand {} \;

• find . -name '*.foo' -print0 | xargs -0 sh -c 'for F in "${@}"; do ...; done' "${0}"
  

• You have plenty of answers that explain well how to do it; but for the sake of completion I'll repeat and add to it:

  xargs is only ever useful for interactive use (when you know all your filenames are plain - no spaces or quotes) or when used with the -0 option. Otherwise, it'll break everything.

  find is a very useful tool; put using it to pipe filenames into xargs (even with -0) is rather convoluted as find can do it all itself with either -exec command {} \; or -exec command {} + depending on what you want:

  find /path -name 'pattern' -exec somecommand {} \;
  find /path -name 'pattern' -exec somecommand {} +
  

  The former runs somecommand with one argument for each file recursively in /path that matches pattern.

  The latter runs somecommand with as many arguments as fit on the command line at once for files recursively in /path that match pattern.

  Which one to use depends on somecommand. If it can take multiple filename arguments (like rm, grep, etc.) then the latter option is faster (since you run somecommand far less often). If somecommand takes only one argument then you need the former solution. So look at somecommand's man page.

  More on find: http://mywiki.wooledge.org/UsingFind

  In bash, for is a statement that iterates over arguments. If you do something like this:

  for foo in "$bar"
  

  you're giving for one argument to iterate over (note the quotes!). If you do something like this:

  for foo in $bar
  

  you're asking bash to take the contents of bar and tear it apart wherever there are spaces, tabs or newlines (technically, whatever characters are in IFS) and use the pieces of that operation as arguments to for. That is NOT filenames. Assuming that the result of a tearing long string that contains filenames apart wherever there is whitespace yields in a pile of filenames is just wrong. As you have just noticed.

  The answer is: Don't use for, it's obviously the wrong tool. The above find commands all assume that somecommand is an executable in PATH. If it's a bash statement, you'll need this construct instead (iterates over find's output, like you tried, but safely):

  while read -r -d ''; do
      somebashstatement "$REPLY"
  done < <(find /path -name 'pattern' -print0)
  

  This uses a while-read loop that reads parts of the string find outputs until it reaches a NULL byte (which is what -print0 uses to separate the filenames). Since NULL bytes can't be part of filenames (unlike spaces, tabs and newlines) this is a safe operation.

  If you don't need somebashstatement to be part of your script (eg. it doesn't change the script environment by keeping a counter or setting a variable or some such) then you can still use find's -exec to run your bash statement:

  find /path -name 'pattern' -exec bash -c 'somebashstatement "$1"' -- {} \;
  find /path -name 'pattern' -exec bash -c 'for file; do somebashstatement "$file"; done' -- {} +
  

  Here, the -exec executes a bash command with three or more arguments.

  1. The bash statement to execute.
  2. A --. bash will put this in $0, you can put anything you like here, really.
  3. Your filename or filenames (depending on whether you used {} \; or {} + respectively). The filename(s) end(s) up in $1 (and $2, $3, ... if there's more than one, of course).

  The bash statement in the first find command here runs somebashstatement with the filename as argument.

  The bash statement in the second find command here runs a for(!) loop that iterates over each positional parameter (that's what the reduced for syntax - for foo; do - does) and runs a somebashstatement with the filename as argument. The difference here between the very first find statement I showed with -exec {} + is that we run only one bash process for lots of filenames but still one somebashstatement for each of those filenames.

  All this is also well explained in the UsingFind page linked above.

• I had to do something similar some time ago, renaming files to allow them to live in Win32 environments:

  #!/bin/bash
  IFS=$'\n'
  function RecurseDirs
  {
  for f in "$@"
  do
    newf=echo "${f}" | sed -e 's/[\\/:\*\?#"\|<>]/_/g'
    if [ ${newf} != ${f} ]; then
      echo "${f}" "${newf}"
      mv "${f}" "${newf}"
      f="${newf}"
    fi
    if [[ -d "${f}" ]]; then
      cd "${f}"
      RecurseDirs $(ls -1 ".")
    fi
  done
  cd ..
  }
  RecurseDirs .
  

  This is probably a little simplistic, doesn't avoid name collisions, and I'm sure it could be done better -- but this does remove the need to use basename on the find results (in my case) before performing my sed replacement.

  I might ask, what are you doing to the found files, exactly?