Linux - Running The Same Command on Many Machines at Once

Question

I have about a dozen Linux boxes that I occasionally need to run the same command(s) on. Is there an easier way (or automated way) to do this asides from logging on to each machine and running the command, one at a time? It's not the same commands all the time and it's not at a preset time so it's not something ideal for tools like cron.

Answer 1

There's also ClusterSSH, which, well, does exactly what you think it does. Yay for descriptive naming!

Answer 2

Hm, maybe pconsole is the tool you want.

Answer 3

You should check out Puppet here is a linux.com article on its setup and use

Answer 4

You may try Capistrano. Originally it is a Rails deployment tool, but it grew to do many things with remote machines.

For a more complete aproach to administration of multiple machine you may want to try Chef.

Answer 5

Don't overlook pssh and dsh either; they're both much like gsh that @Philip Durbin mentioned.

Answer 6

Ansible has ability to do ad hoc commands and is also expandable to do full config management. The ability to match based on groups is excellent.

http://docs.ansible.com/intro_adhoc.html

Answer 7

If you are on a KDE desktop, konsole has the "Send Input to all sessions" option. It applies to all the sessions opened in the same konsole window.

Answer 8

I guess I'll copy and paste my answer from Stack Overflow, which people seemed to like. . .

---

The issue of how to run commands on many servers at once came up on a Perl mailing list the other day and I'll give the same recommendation I gave there, which is to use gsh: http://outflux.net/unix/software/gsh

gsh is similar to the "for box in box1_name box2_name box3_name" solution already given but I find gsh to be more convenient. You set up a /etc/ghosts file containing your servers in groups such as web, db, RHEL4, x86_64, or whatever (man ghosts) then you use that group when you call gsh.

[pdurbin@beamish ~]$ gsh web "cat /etc/redhat-release; uname -r"
www-2.foo.com: Red Hat Enterprise Linux AS release 4 (Nahant Update 7)
www-2.foo.com: 2.6.9-78.0.1.ELsmp
www-3.foo.com: Red Hat Enterprise Linux AS release 4 (Nahant Update 7)
www-3.foo.com: 2.6.9-78.0.1.ELsmp
www-4.foo.com: Red Hat Enterprise Linux Server release 5.2 (Tikanga)
www-4.foo.com: 2.6.18-92.1.13.el5
www-5.foo.com: Red Hat Enterprise Linux Server release 5.2 (Tikanga)
www-5.foo.com: 2.6.18-92.1.13.el5
[pdurbin@beamish ~]$

You can also combine or split ghost groups, using web+db or web-RHEL4, for example.

I'll also mention that while I have never used shmux, its website contains a list of software (including gsh) that lets you run commands on many servers at once. Capistrano has already been mentioned and (from what I understand) could be on that list as well.

Answer 9

You could also try ssh [user@]hostname [command] and then loop through the host names.

Answer 10

Also take a look at func. Func lets you use a module architecture to control different aspects of a system, from dealing with packages, to running commands, etc. It works on a client/server module over SSL and has a fairly flexible authentication scheme as well.

Answer 11

I've used ClusterSSH and it works. However, beyond a certain number of terminals, it doesn't scale well.

I'm surprised no one has mentiond xargs. Xargs is perfect for most things. In fact, I've gotten so comfortable with it that I never use clusterssh anymore.

Example

echo -n host1 host2 host3 host4|xargs -d" " -n1 -P30 -I+ ssh + "sudo service apache2 restart"

(Restart apache on all web hosts.)

Answer 12

If you are already a tmux user, you might enjoy tmux-cssh (stands for TMUX-C(luster)-SSH). You give it a list of server names and it opens a new tmux window with an SSH session open to each server in its own pane. All keystrokes are copied to each window. Depending on the number of hosts the output can be hard to read.

Answer 13

Other tools are:

• Rundeck
• ControlTier

RunDeck could also be descriped as the lightweight ControlTier.

These tools not only give you a cluster shell, but also a web frontend and you can save your jobs for future use.

Answer 14

You can also try Cluster SSH (cssh). I haven't used it, but there's an article on LinuxJournal.com called "Manage Multiple Servers Efficiently" that talks about it.

Answer 15

Use dish, simple and very powerful ! This script changed my life :) One script to rule them all !

http://nic-nac-project.org/~drimiks/gnu/dish.shtml

The diligence shell 'dish' executes commands via ssh/rsh/telnet/mysql simultaneously on several systems. Changing password, updating configuration, copying files, checking status, etc. on hundreds of nodes is made then simple. Dish is similar to the IBM's 'dsh' (distributed shell) but furthermore allows remote execution of commands requiring authentication. Compared to other open source packages which include a distributed shell (or similar tools) like ClusterIt, dancer's shell, parallel distributed shell, or C3, dish is a slim and easy to use remote-shell wrapper for cases where a flexible solution is needed. You can browse the dish's manual page on-line here.

Answer 16

Here are some examples using the commands mentioned in other answers:

pdsh

Can take numeric ranges, outputs with the host name first, seems actually maintained:

simplest version:

pdsh -w hostname echo done

output:

hostname: done

More complicated example:

pdsh -w username@hostname[7-8],username@otherhost "sleep 5 && echo done"

output:

hostname8: done
hostname7: done
otherhost: done

mac: brew install pdsh

dsh

dsh -m user@host,user@host --remoteshell ssh --concurrent-shell -- "sleep 5 && echo done"

need remote shell setting otherwise it'll use "rsh" which may or may not even be available.

mac: brew install dsh

pssh

pssh -i -H user@host -H user@host "sleep 5 && echo done"

They can all take list of machines from a file, optionally, as well.

Feel free to add more examples here, it's a comunity wiki

mac: brew install pssh

Cluster SSH

Cluster SSH (the "mac only version" apparently there are two versions the other is here) actually opens "real terminals" with a big terminal at that bottom that sends input to all the others. So it's like a UI that sends it out to multiple.

Ex:

csshX hostname0[1-2] hostname3 then type into the red box and it goes to each terminal window, one per host, that it pops up.

mac: brew install csshx (an old version but still works)

there is also a brew install ansible FWIW

Answer 17

Easiest way I found without installing or configuring much software is using plain old tmux. Say you have 9 linux servers. Pick a box as your main. Start a tmux session:

tmux

Then create 9 split tmux panes by doing this 8 times:

ctrl-b + %

Now SSH into each box in each pane. You'll need to know some tmux shortcuts. To navigate, press:

ctrl+b <arrow-keys>

Once your logged in to all your boxes on each pane. Now turn on pane synchronization where it lets you type the same thing into each box:

ctrl+b :setw synchronize-panes on

now when you press any keys, it will show up on every pane. to turn it off, just make on to off. to cycle resize panes, press ctrl+b < space-bar >.

This works alot better for me since I need to see each terminal output as sometimes servers crash or hang for whatever reason when downloading or upgrade software. Any issues, you can just isolate and resolve individually.