How do you run commands across 1000s of machines?

Good answers will mention some or all of the following:
  • ssh
  • Setting up password-less ssh for the machines, or assuming that they are already set up
  • A command-line for loop which iterates over machines doing something
    • Perl script also OK
    • Stellar candidates will mention “xargs -P” which will allow them to run some commands in parallel (say, for 1000 machines, process them in 100 batches, doing 10-machine batches in parallel

The worst possible answer is some form of “I log into all the machines one by one”.