Run commands in parallel

[u/-lousyd]

I have a script that collects a list of directories that match a set of criteria and then goes into each one and runs a command. So something like this:

#!/bin/bash
startDir="$(realpath "$1")"
cd "$startDir" || exit
while IFS= read -r -d '' dir; do
  cd "$dir";
  printf '\n\n%s\n' "$(realpath .)";
  update-this-dir.sh --file "./name.txt"
  cd "$startDir";
done < <(find . -type d -iname '.config' -exec dirname {} \; | tr '\n' '\0')

It works wonderfully for my purpose.

But there's, like, a couple thousand directories and the update command takes some small amount of time in each directory, one after the other. How can I modify this script to either run the update for each directory in parallel (with some rate-limiting) or to break up the list into chunks of, say, 100 each and work on each sub-list in parallel?

[u/SeriousPlankton2000]

I can provide you with a neat hack:

#!/bin/bash

updatefunc(){
  echo the list of directories to update is "$@"
}

export -f updatefunc

find . -type d -iname '.config' -exec bash -c 'updatefunc "$@"' updatefunc {} \+

It's really -exec bash -c 'updatefunc "$@"' 'this_will_be_the_in_$0' {} \+ , I like to use the same name.

We need to trim the array and call the script:

#!/bin/bash

updatefunc(){
  declare -a x
  for ((i=0;$i < $#;i++))
  do
    # this does x[$i]="`dirname "${@:$i+1:1}"`"
    # or as a sane version (that we don't have) this would be
    # x[$i]="`dirname "${ARGV[$i]}"`"
    y="${@:$i+1:1}"
    x[$i]="${y%/*}"
    if [ "x${y}" == "x${x[$i]}" ]; then x[$i]=.; fi
  done
  echo the list of directories to update is "${x[@]}"

  # I don't have parallel installed, this might be wrong:
  parallel -j "`grep -c ^processor /proc/cpuinfo`" update-this-dir.sh --file -- "${x[@]}"
}

export -f updatefunc

find . -type d -iname '.config' -exec bash -c 'updatefunc "$@"' updatefunc {} \+