Home

execute_on_all_hosts.sh




Purpose

The script execute_on_all_hosts.sh can be used to execute a script or binary on multiple hosts. The script/binary to execute on the hosts is copied via scp to each host and then executed on every host via ssh. execute_on_all_hosts.sh also supports executing a command on all hosts without copying something to the hosts.

execute_on_hosts.sh supports customer defined templates for the scp and ssh commands. Therefor access via a Golden Host or something similar is also supported.

The script can run in sequential or parallel order on the target hosts. In the later case you can restrict the maximum number of parallel running processes and customize some other values regarding the handling of the parallel processes (see parameter -W and -w).

In addition, some not that usual features which might be useful in some environments like

- scp with Forward Agent enabling

- cleanup of the known_hosts

- different user and ssh keys for scp and ssh

are supported (see the Usage and the Known debug switches below)

The results are written to one or more logfiles.



Back to top

License


# CDDL HEADER START
#
# The contents of this file and the script are subject to the terms of the
# Common Development and Distribution License, Version 1.0 only
# (the "License").  You may not use this file except in compliance
# with the License.
#
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
# or http://www.opensolaris.org/os/licensing.
# See the License for the specific language governing permissions
# and limitations under the License.
#
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
# If applicable, add the following below this CDDL HEADER, with the
# fields enclosed by brackets "[]" replaced with your own identifying
# information: Portions Copyright [yyyy] [name of copyright owner]
#
# CDDL HEADER END


Back to top

History


Version
Releasedate
         Description
v0.0.1
28.02.2008 initial release
v0.0.2
29.02.2008
##   28.02.2008 v0.0.2 /bs
##     added parameter hostlist, scriptfile, outputfile, and user
##     added the parameter -U
##     the script now supports multiple hostlists
##
v0.0.3
03.03.2008
##   03.03.2008 v0.0.3 /bs
##     added additional workarounds for CYGWIN
##     added the parameter -I
##     added a Summary about failed hosts
##     changed __ONLY_ONCE to ${__FALSE}
##     added code to print a warning if ssh-agent is not running
##
v0.0.4
04.03.2008
##   04.03.2008 v0.0.4 /bs
##     changed the default output filename to "/var/tmp/${__SCRIPTNAME##*/}.$$.log"
##     added code to call dos2unix for the script file if running in CYGWIN
##
v0.0.5
01.08.2008
##   01.08.2008 v0.0.5 /bs
##     corrected a syntax error in the routine to print the short usage
##     set read permissions for the script before executing scp if necessary
##     added the parameter  -p scpoptions and  -P sshoptions
##     added code to sort the list of hosts and remove duplicate entries
##
v0.0.6
07.08.2008
##   07.08.2008 v0.0.6 /bs
##     corrected a syntax error

v0.0.7
20.09.2008
##   20.09.2008 v0.0.7 /bs
##     added the option -K / --nostrictkeys

v0.0.10
30.03.2011 ##   30.03.2011 v0.0.10 /bs
##     added the parameter -b / --ssh_keyfile
##     added the parameter -B / --do_not_copy
##




v1.0.0
31.05.2013
##   31.05.2013 v1.1.0 /bs
##     executeCommandAndLog rewritten using coprocesses (see also credits)
##     Info update: executeCommandAndLog does now return the RC of the executed
##                  command even if a logfile is defined
##     replaced the runtime system with scriptt.sh v2.0.0
##     the commands can now run in parallel in the background
##     (parameter -d, -w, and -W)

v1.1.0
06.06.2013
##   06.06.2013 v1.1.0 /bs
##     the parameter -W and now supports a syntax like -W /5 and the
##     "/" can be replaced by ","
##     the parameter -w and now supports a syntax like -w /5, -w /5/20,
##     or -w //10. "/ "can be replaced by ","
##     You can now use seconds, minutes, or hours for the time values
##     of the parameter -w and -W can now

v1.2.0
28.05.2014
##   28.05.2014 v1.2.0 /bs
##     use scriptt.sh v2.0.0.6
##     added the parameter -x (--excludehost)
##     added the parameter -A (--includehost)
##     enhanced support for binaries
##     reworked the usage help
##     replaced "which" whith "whence"

v1.3.0
19.09.2014
##   19.09.2014 v1.3.0 /bs
##     the script now supports the format user@host

v1.3.1
27.04.2014
##   27.09.2014 v1.3.1 /bs
##     added the keyword ignore_user_in_file for the parameter -D
##     added the keyword do_not_sort_hostlist for the parameter -D
##     added the keyword ssh_binary for the parameter -D
##     added the keyword scp_binary for the parameter -D
##     added the keyword dos2unix_binary for the parameter -D

v2.0.0
24.07.0215
##   24.07.2015 v2.0.0 /bs
##     added support for different user for ssh and scp (parameter -u)
##     added support for different ssh key files for the ssh and scp user
##     added an option to list all hosts to the question to start the script
##     added parameter to enable the forward agent for scp (parameter -D enable_ForwardAgent_for_scp)
##     added support for templates for the scp usage (parameter -t scp:scp_template)
##     added support for templates for the ssh usage (parameter -t ssh:ssh_template)
##       The scp/ssh templates are useful if a golden host is used to access the machines
##     added t{SED_SEP}he debug options (parameter -D) from scriptt.sh version 2.1.0.7 21.07.2015
##     added single step mode for sequential usage (parameter -D singlestep)
##     added dry run mode (parameter -D dryrun)
##     added a parameter to cleanup the known_hosts (parameter -D clean_known_hosts)
##     added an example for golden host usage (parameter -D sls_db)
##     added the parameter "-D if=xx"
##     the parameter -K now also adds the options "-o NumberOfPasswordPrompts=0 -o ConnectTimeout=10"
##       to scp and ssh if running in parallel mode


v2.0.1
009.10.2015
##   09.10.2015 v2.0.1 /bs
##     added support for scp via sls; the sls parameter are now:
##       -D sls_scp_db     sls for scp for DB
##       -D sls_ssh_db     sls for ssh for DB
##       -D sls_ssh_fms    sls for ssh for FMS
##
##       -D sls_fms        sls for scp and ssh for FMS
##       -D sls_db         sls for scp and ssh for DB

v2.1.0
24.10.2015
##   24.10.2015 v2.1.0 /bs
##     added support for field separators in the hostlist files (filename[:separator])
##     added the keyword fieldsep=x to the parameter -D to change the default field separator
##     added the keyword printargs to the parameter -D
##     now the parameter -i hostlist is optional if -A hostname is used
##     now the parameter -x hostexcludelist supports a leading "?" for
##       optional hostexcludelist files
##     reworked the messages written by the script (in normal mode and
##     in verbose mode)
##     the parameter "-x none" did not work --fixed
##     the usage with "hostfile scriptfile .." did not work --fixed
##     the script now reset the current terminal using "stty sane" at
##     script end
##     the parameter -K now also adds the options "-o BatchMode=yes -o PasswordAuthentication=no"
##       to scp and ssh if running in parallel mode


v2.2.0
27.10.2015
##   27.10.2015 v2.2.0 /bs
##     execute_on_all_hosts.sh now creates up to 10 backups of the logfile
##     and rewrites the logfile every time.
##     To overwrite the number of backups use the following syntax for
##     the parameter -l:
##
##        -l logfile,[no_of_backups_of_the_logfile]
##
##     The default number of backups for the log file is configured in the variable
##     MAX_NO_OF_LOGFILES
##

v2.2.1
25.11.2015
##   25.11.2015 v2.2.1 /bs
##      the internal sed field separator is now "CTRL-X" so the pipe character
##      "|" can be used in commands to execute again (see the variable SED_SEP)




Back to top

Operating system


Tested with Solaris and Linux;  the previous version was also tested with Cygwin; the script should work on other Unix Operating systems with a kornshell (ksh).



Back to top

Language / type


Kornshell Script



Back to top

Prerequisites

ssh and scp must be installed and configured; the sshd daemon must run on the hosts.

In addition, ssh-agent or something similar should be configured and login via public ssh key should be enabled on the target hosts.




Back to top

Usage


[xtrnaw7@t540p scripts]$ ./execute_on_all_hosts.sh -v -h
[25.11.2015 21:01:29] execute_on_all_hosts.sh v2.2.1 started at Mi 25. Nov 21:01:29 CET 2015.
[25.11.2015 21:01:29] Reading the config file "/data/develop/scripts/execute_on_all_hosts.conf" ...
  execute_on_all_hosts.sh v2.2.1 - copy a script to a list of hosts via scp and execute it on the hosts using ssh

  Usage: execute_on_all_hosts.sh [-v|+v] [-q|+q] [-h] [-l logfile|+l] [-y|+y] [-n|+n]
                    [-D debugswitch] [-a|+a] [-O|+O] [-f|+f] [-C] [-H] [-X] [-S n] [-V] [-T]
                    [-i hostlist{,hostlist1{,...}] [-s scriptfile] [-o outputfile] [-u sshuser|scp:scpuser|ssh:sshuser] [-I basedir]
                    [-c shell] [-k] [-U] [-p scpoptions] [-P sshoptions] [-K] [-R] [-B|+B] [-b ssh_keyfile|scp:scp_keyfile|ssh:ssh_keyfile]
                    [-x excludehost] [-A includehost] [-t ssh:ssh_template] [-t scp:scp_template]
                    [-d|+d] [-W [timeout[/intervall]] [-w NoOfBackgroundProcs[/intervall[/timeout]]]]
                    [hostlist [scriptfile [outputfile [sshuser]]]]

  The parameters scriptfile, outputfile, and sshuser overwrite the options; hostlist (or includehost) and scriptfile are mandatory either
  as parameter or as option.

  For optimal usage ssh login via public key should be enabled on the target hosts and
  the ssh agent should run on this host (the agent is currently running).

  Use "-D help" to view the known debug options.

  Use the parameter "-v -h [-v]" to view the detailed online help; use the parameter "-X" to view some usage examples.

  It is strongly recommended to test the script execution in dry run mode (parameter "-D dryrun") before doing the real work!

  see also http://bnsmb.de/solaris/execute_on_all_hosts.html



 Note: Use -{switch} or --{longswitch} to turn an option on;
       use +{switch} or ++{longswitch} to turn an option off

       The long format of the parameter (--parameter/++parameter) is not supported by all ksh implementations


    Parameter:

      -v|+v - turn verbose mode on/off; current value: y
              Long format: --verbose / ++verbose
      -q|+q - turn quiet mode on/off; current value: n
              Long format: --quiet / ++quiet
      -h    - show usage
              Long format: --help
      -l    - set the logfile
              current value: /var/tmp/execute_on_all_hosts.LOG
              use "-l [logfile,no_of_backups]" to define the number of retained backups
              of the logfile; the default no of backups is 10
              Long format: --logfile
      +l    - do not write a logfile
              Long format: ++logfile
      -y|+y - assume yes to all questions or not
              Long format: --yes / ++yes
      -n|+n - assume no to all questions or not
              Long format: --no /++no
      -D|+D - debug switch
              current value:
              use "-D help" to list the known debug switches
              Long format: --debug / ++debug
      -a|+a - turn colors on/off; current value: n
              Long format: --color / ++color
      -O|+O - overwrite existing files or not; current value: n
              Long format: --overwrite / ++overwrite
      -f|+f - force; do it anyway; current value: n
              Long format: --force / ++force
      -C    - write a default config file in the current directory and exit
              Long format: --writeconfigfile
      -H    - write extended usage to STDERR and exit
              Long format: --doc
      -X    - write usage examples to STDERR and exit
              Long format: --view_examples
      -S n  - print error/warning summaries:
              n = 0 no summariess, 1 = print error msgs,
              2 = print warning msgs, 3 = print error and warning mgs
              Current value: 0
              Long format: --summaries
      -V    - write version number to STDOUT and exit
              Long format: --version
      -T    - append STDOUT and STDERR to the file "/var/tmp/execute_on_all_hosts.sh.375.tee.log"
              Long format: --tee

      -i hostlist[:fieldseparator]
              "hostlist" is the file with the list of hosts to process.
              Format: one hostname per line (use user@host to use an explicit
              user for a host; this user is then used for scp and ssh to this host);
              empty lines and lines beginning with a hash "#" are ignored.
              The script only reads the first field of each line. The field
              separator is " ". To change the field separator
              add "[:fieldseparator]" to the filename.
              The char "," can not be used as field separator; use
                -D fieldsep=","
              to change the default field separator instead.
              Use a comma "," to separate multiple hostlists or use the
              parameter "-i" more than one time.
              All hostlists are merged and sorted. Each host in the list(s)
              is only processed one time.
              Missing hostlist files are treated as error. To ignore a
              missing hostlist file add a leading "?" to the filename.
              Current Value: ""
              Long format: --hostlist

      -x hostname
      -x hostexcludelist[:fieldseparator]
              exclude the host "hostname" from the execution
              This parameter can be used multiple times; regular expressions
              for "hostname" are allowed.
              Use "-x none" to clear the list of hosts to exclude.
              Use a relative or absolute filename for "hostname" to read the
              hosts to exclude from a file. Format of the file:
              one hostname per line; empty lines and lines beginning with "#"
              are ignored.
              The script only reads the first field of each line. The field
              separator is " ". To change the field separator
              add "[:fieldseparator]" to the filename.
              The char "," can not be used as field separator; use
                -D fieldsep=","
              to change the default field separator instead.
              Missing hostexcludelist files are treated as error. To ignore a
              missing hostexcludelist file add a leading "?" to the filename.
              Current Value: ""
              Long format: --excludehost

      -A hostname
              add the host "hostname" to the list of hosts
              This parameter can be used multiple times; regular expressions
              for "hostname" are not allowed.
              Use "-A none" to clear the list of hosts to include.
              Current Value: ""
              Long format: --includehost

      -s scriptfile
              scriptfile is the script or executable to execute on all hosts.
              It will be copied to each host using the user "scpuser" and
              executed on each host with the user "sshuser".
              If "scriptfile" is a binary it will be executed directly;
              if "scriptfile" is not a binary it will be executed
              either by the default shell for "sshuser" or by the
              shell specified with the parameter "-c".
              If this script is running in a cygwin session and the binary
              dos2unix is available via the path the script will be converted
              to Unix style using dos2unix before copying it to the hosts.
              If "-B" is used, the value for this parameter is a command to
              execute on the hosts without first copying it to the host. In
              this case the command is executed by the user "sshuser"
              without calling a shell.
              Use \; in this case to separate multiple commands to execute.
              e.g: -s "uname -a \; uptime "
              Current value: ""
              Long format: --scriptfile

      -u sshuser
              "sshuser" is the userid to use for ssh and scp on the target
              hosts
              To use different user for ssh and scp use:
              -u ssh:sshuser
              -u scp:scpuser
              If "-u scp:scpuser" is not used the sshuser is also used for scp.
              Current value for sshuser is "support"
              Current value for scpuser is ""
              Long format: --sshuser

      -c shell_to_use
              "shell_to_use" is the shell to execute the script on the hosts
              Use "none" to execute the script with the default shell
              of the "sshuser".
              If "-B" is used or if the command to execute (parameter "-s")
              is a binary this parameter is ignored and the command
              is always executed directly.
              Current value "/usr/bin/ksh"
              Long format: --shell

      -I basedir
              "basedir" is the directory with the hostlist and the
              script files; if this parameter is found the script searches
              the hostlist files and the script files in this directory also
              Current value: ""
              Long format: --basedir

      -o outputfile
              "outputfile" is the name of the file for the output generated
              by the commands. If "-U" is also used the output for each host
              is logged into a separate file named "<outputfile>.<hostname>"
              Current value "/var/tmp/execute_on_all_hosts.sh.375.cmds.log"
              Long format: --outputfile

      -p scpoptions
              "scpoptions" are additional options for scp, e.g
                 -p "-o StrictHostKeyChecking=no"
              Current value ""
              Long format: --scpoptions

      -P sshoptions
              "sshoptions" are additional options for ssh
              Current value ""
              Long format: --sshoptions

      -t ssh:ssh_template
      -t scp:scp_template
              Define a new template for the scp or ssh command
              (see below for the known template placeholder)
              The scp/ssh templates are useful if a golden host is used
              to access the machines
              Current value:  ssh template:  %b %o %k -l %u %i %c %s
                              scp template:  %b %o %k %S %u@%i:%s
              Long format --template

      -K|+K use the option "StrictHostKeyChecking=no" for ssh and scp.
              In parallel mode also add the ssh and scp options
                "-o NumberOfPasswordPrompts=0 -o ConnectTimeout=10"
              Current value: n
              Long format: --nostrictkeys

      -k|+k add comments to the output file
              Current value: y
              Long format: --nocomments

      -B|+B do not upload a script to the hosts
              In this case the value for the parameter "-s" is interpreted
              as command to execute and the value for the shell to
              use (parameter "-c") is ignored.
              Current value: n
              Long format: --do_not_copy

      -b ssh_keyfile
              Use the ssh key file "ssh_keyfile" for ssh and scp.
              Default:
              Use the default ssh key files for the user "sshuser" for ssh
              and the default key file for the user "scpuser" for scp
              To use different key files for ssh and scp use:
              -b ssh:ssh_keyfile
              -b scp:ssh_keyfile
              Current value for ssh_keyfile is: ""
              Current value for scp_keyfile is: ""
              Long format: --ssh_keyfile

      -U|+U create unique log files
              (see parameter "-o")
              Current value: n
              Long format: --uniquelogfiles

      -R|+R  use RCM methods
              The default value depends on the existence of copy_table.pl
              in the PATH.
              Use the environment variables RCM_USERID and RCM_PASSWORD
              to set the RCM userid and password. If these variables are
              not set the script will prompt the user for them.
              Current value: n
              Long format: --rcm

      -d|+d execute the commands in parallel in the background
              Current value: n
              Long format: --parallel

      -W timeout[/intervall]
              "timeout" is the timeout for background processes and
              "intervall" is the wait intervall for background processes.
              Use "-1" or "none" for the timeout value to disable the
              timeout.
              Current values:
                Timeout: 1h (= 3600 second(s) )
                Intervall: 2s (= 2 second(s) )
              Use a trailing "m" for times in minutes or a trailing "h"
              for times in hours; "s" is for seconds which is the default
              if neither "m" nor "h" is used.
              Use "default" for any value to use the default value.
              Use " -W /intervall" to only change the intervall.
              You can also use "," to separate the values.
              This parameter is only used if the commands are executed
              in parallel (parameter "-d")
              Long format: --timeout

      -w noOfBackgroundprocesses[/startIntervall[/maxStartTime]]
              "noOfBackgroundprocesses" is the max. number of background
              processes running at the same time, "startIntervall" is the
              wait intervall and "maxStartTime" is the timeout for starting
              the background processes.
              Use "-1" or "none" for an unlimited number of parallel
              background processes.
              Use "-1" or "none" for the timeout value to disable
              the timeout.
              Current values:
                Max. number of parallel running background processes: -1
                Wait intervall: 5s (= 5 second(s) )
                Timeout value: 5m (= 300 second(s) )
              Use a trailing "m" for times in minutes or a trailing "h"
              for times in hours; "s" is for seconds which is the default
              if neither "m" nor "h" is used.
              Use "default" for any value to use the default value.
              Use " -w /startIntervall" to only change the start intervall.
              Use " -w //maxStartTime" to only change the timeout
              You can also use "," to separate the values.
              This parameter is only used if the commands are executed
              in parallel (parameter "-d")
              Long format: --noOfbackgroundProcesses


Placeholder in the ssh and scp templates are:

Placeholder  usage                              used for scp?    used for ssh?     parameter
-------------------------------------------------------------------------------------------------
%%           %                                  yes              yes               n/a
%b           scp binary                         yes              no                -D scp_binary
%b           ssh binary                         no               yes               -D ssh_binary

%S           source script or command           yes              yes               -s
%s           script/binary on the target host   yes              yes               n/a
%c           shell to use                       no               yes               -c

%u           scp user                           yes              no                -u scp:
%u           ssh user                           no               yes               -u ssh:

%h           target host (FQN)                  yes              yes               n/a
%H           target host (short hostname)       yes              yes               n/a
%d           DNS domain                         yes              yes               n/a
%i           target host interface              yes              yes               n/a

%o           scp options                        yes              no                -p
%o           ssh options                        no               yes               -P

%k           ssh key for ssh                    no               yes               -b ssh:
%k           ssh key for scp                    yes              no                -b scp:

Do not use the the character ^A or ^X in any parameter!

[25.11.2015 21:01:29] The log file used was "/tmp/execute_on_all_hosts.sh.375.TEMP" 
[25.11.2015 21:01:29] execute_on_all_hosts.sh v2.2.1 started at Mi 25. Nov 21:01:29 CET 2015 and ended at Mi 25. Nov 21:01:29 CET 2015.
[25.11.2015 21:01:29] The RC is 1.
[xtrnaw7@t540p scripts]$

 


back to top

Example



In this example we only execute a "uname -a" on a list of hosts:

# the file with the list of hosts:
#
xtrnaw7@t30:/data/develop/scripts$ cat hosts
# machines in my testenvironment
sol9
pb001
ferrari
sol8
sol2
sol3
ultra30

# The file with the commands to execute on each host:
#
xtrnaw7@t30:/data/develop/scripts$ cat testscript0
uname -a

# run the script (using the user root on the target hosts):
#
xtrnaw7@t30:/data/develop/scripts$ ./execute_on_all_hosts.sh -i hosts -s testscript0  -u root
[28.02.2008 21:01:03] execute_on_all_hosts.sh started on Thu Feb 28 21:01:03 CET 2008 
[28.02.2008 21:01:03] Reading the config file "./execute_on_all_hosts.conf" ...
[28.02.2008 21:01:03] Using the log file "/var/tmp/execute_on_all_hosts.LOG" 
[28.02.2008 21:01:03] 
[28.02.2008 21:01:03] Executing the script 
[28.02.2008 21:01:03]     testscript0
[28.02.2008 21:01:03] as ssh user 
[28.02.2008 21:01:03]     root
[28.02.2008 21:01:03] on every host listed in the file 
[28.02.2008 21:01:03]     hosts
[28.02.2008 21:01:03] 
[28.02.2008 21:01:03] The shell to execute the script is 
[28.02.2008 21:01:03]     /usr/bin/ksh
[28.02.2008 21:01:03] The output of the commands will be logged in the file
[28.02.2008 21:01:03]     /var/tmp/execute_on_all_hosts.sh.19003.log
[28.02.2008 21:01:03] 
Okay (y/N)? y
[28.02.2008 21:01:06] Starting processing ...
[28.02.2008 21:01:06] ---- Processing "sol9" ...
testscript0                                                                   100%    9     0.0KB/s   00:00   
SunOS sol9 5.11 snv_78 sun4u sparc sun4u
[28.02.2008 21:01:09] ---- Processing "pb001" ...
testscript0                                                                   100%    9     0.0KB/s   00:00   
SunOS pb001 5.11 snv_78 i86pc i386 i86pc
[28.02.2008 21:01:10] ---- Processing "ferrari" ...
ssh: connect to host ferrari port 22: No route to host
lost connection
[28.02.2008 21:01:13] ERROR: Error copying the script to the host "ferrari" - could not execute the script on that host
[28.02.2008 21:01:13] ---- Processing "sol8" ...
ssh: connect to host sol8 port 22: Connection refused
lost connection
[28.02.2008 21:01:13] ERROR: Error copying the script to the host "sol8" - could not execute the script on that host
[28.02.2008 21:01:13] ---- Processing "sol2" ...
testscript0                                                                   100%    9     0.0KB/s   00:00   
SunOS sol2 5.10 Generic_127111-03 sun4u sparc SUNW,Ultra-5_10
[28.02.2008 21:01:16] ---- Processing "sol3" ...
ssh: connect to host sol3 port 22: No route to host
lost connection
[28.02.2008 21:01:19] ERROR: Error copying the script to the host "sol3" - could not execute the script on that host
[28.02.2008 21:01:19] ---- Processing "ultra30" ...
Warning: the RSA host key for 'ultra30' differs from the key for the IP address '192.168.1.20'
Offending key for IP in /home/xtrnaw7/.ssh/known_hosts:13
Matching host key in /home/xtrnaw7/.ssh/known_hosts:40
testscript0                                                                   100%    9     0.0KB/s   00:00   
Warning: the RSA host key for 'ultra30' differs from the key for the IP address '192.168.1.20'
Offending key for IP in /home/xtrnaw7/.ssh/known_hosts:13
Matching host key in /home/xtrnaw7/.ssh/known_hosts:40
SunOS ultra30 5.10 Generic_120011-14 sun4u sparc SUNW,Ultra-30
[28.02.2008 21:01:21] 
[28.02.2008 21:01:21] All done
[28.02.2008 21:01:21] 
[28.02.2008 21:01:21] The output of the commands is logged in the file
[28.02.2008 21:01:21]     /var/tmp/execute_on_all_hosts.sh.19003.log
[28.02.2008 21:01:21] 
[28.02.2008 21:01:21] The log file used was "/var/tmp/execute_on_all_hosts.LOG
[28.02.2008 21:01:21] execute_on_all_hosts.sh ended on Thu Feb 28 21:01:21 CET 2008.
[28.02.2008 21:01:21] The RC is 0.

# The output of the commands is written to the logfile:

xtrnaw7@t30:/data/develop/scripts$ cat /var/tmp/execute_on_all_hosts.sh.19003.log

# ### ---- Log of the script executed on host "sol9" --- start ---
SunOS sol9 5.11 snv_78 sun4u sparc sun4u
# ### ---- Log of the script executed on host "sol9" --- end ---

# ### ---- Log of the script executed on host "pb001" --- start ---
SunOS pb001 5.11 snv_78 i86pc i386 i86pc
# ### ---- Log of the script executed on host "pb001" --- end ---

# ### ---- Log of the script executed on host "sol2" --- start ---
SunOS sol2 5.10 Generic_127111-03 sun4u sparc SUNW,Ultra-5_10
# ### ---- Log of the script executed on host "sol2" --- end ---

# ### ---- Log of the script executed on host "ultra30" --- start ---
Warning: the RSA host key for 'ultra30' differs from the key for the IP address '192.168.1.20'
Offending key for IP in /home/xtrnaw7/.ssh/known_hosts:13
Matching host key in /home/xtrnaw7/.ssh/known_hosts:40
SunOS ultra30 5.10 Generic_120011-14 sun4u sparc SUNW,Ultra-30
# ### ---- Log of the script executed on host "ultra30" --- end ---



Back to top

Other Examples


Note:

Use ./execute_on_all_hosts.sh -X to get some other examples


# execute the script testscript0 on all hosts listed in the file hostlist0 as user support
# and log the output to the file mylogfile.log
#
execute_on_all_hosts.sh ./hostlist0 ./testscript0 mylogfile.log


# execute the command "uname -a" as user support on all hosts listed in the files hostlist1 and hostlist2

#
execute_on_all_hosts.sh -i ./hostlist1,hostlist2 -s "uname -a" -B


# execute the script /var/tmp/testscript.sh which already exists on the host as user root on all hosts

# listed in the files hostlist1 and hostlist2 using the shell /usr/bin/ksh.
#
execute_on_all_hosts.sh -u root -i ./hostlist1,hostlist2 -s "/var/tmp/testscript.sh" -B -c /usr/bin/ksh


# execute the script myscript.sh on all hosts listed in the file myhostlist as user "support" in sequential order;
# the output will be logged in the file ./execute_on_all_hosts.sh.$$.log (where $$ is the PID of the proccess
# running the script):
#
execute_on_all_hosts.sh myhostlist myscript.sh

# alternative usage:
#
execute_on_all_hosts.sh -i myhostlist -s myscript.sh


# execute the command "uptime" on all hosts listed in the files myhostlist1 and myhostlist2 in parallel:
#
execute_on_all_hosts.sh -i myhostlist1,myhostlist2 -s uptime -B -d

# execute the commands "uname -a; uptime" on all hosts listed in the file myhostlist in parallel restricting the
# number of background processes running at the same time to 4. The output is written in separate files for each host;
# the filenames are /var/tmp/test.log.<hostname>:
#
execute_on_all_hosts.sh -i myhostlist -s "uname -a ; uptime" -B -d -w 4 -o /var/tmp/test.log -U



# execute the command "uname -i" on all hosts in the files hostlist, hostsA, hosts1, and hosts2 and exclude the host myhost.isbs.de
# and all hosts listed in the file ./hosts.
# There are different field separator used for each hostlist and the hostlist file hostsA is optional
#
execute_on_all_hosts.sh -i hostlist -B -s "uname -i"  -i ?hostsA:y,hosts1:\;,hosts2::   -D fieldsep="," -x myhost.isbs.de -x ./hosts



Templates


execute_on_all_hosts.sh uses templates for the scp and ssh commands to use. The default templates are:

    ssh template:  %b %o %k -l %u %i %c "%s"

    scp template:  %b %o %k %S %u@%i:%s

These templates evaluate to "standard" scp and ssh commands, e.g:

    ssh -o StrictHostKeyChecking=no -l zzz t61p.akabank.de /usr/bin/ksh /tmp/tmp_28922_execute_on_all_hosts.sh

    scp -o StrictHostKeyChecking=no -i scp_key monitor_netstat.sh yyy@t61p.akabank.de:/tmp/tmp_28922_execute_on_all_hosts.sh


For access via an Golden Host you can define custom templates, e.g. the ssh template

    %b %k %o -t -A -l user4root sls.isbs.de access_host -c %u@%H \"%s\"

evaluates to

    ssh -i ~/.ssh/my_ssh_key -o StrictHostKeyChecking=no -t -A -l user4root sls.isbs.de access_host -c myuser@develop "/tmp/tmp_29185_execute_on_all_hosts.sh"

The known placeholder for the scp and ssh templates are:

Placeholder in the ssh and scp templates are:

Placeholder  usage                              used for scp?    used for ssh?     parameter
-------------------------------------------------------------------------------------------------
%%           %                                  yes              yes               n/a
%b           scp binary                         yes              no                -D scp_binary
%b           ssh binary                         no               yes               -D ssh_binary

%S           source script or command           yes              yes               -s
%s           script/binary on the target host   yes              yes               n/a
%c           shell to use                       no               yes               -c

%u           scp user                           yes              no                -u scp:
%u           ssh user                           no               yes               -u ssh:

%h           target host (FQN)                  yes              yes               n/a
%H           target host (short hostname)       yes              yes               n/a
%d           DNS domain                         yes              yes               n/a
%i           target host interface              yes              yes               n/a

%o           scp options                        yes              no                -p
%o           ssh options                        no               yes               -P

%k           ssh key for ssh                    no               yes               -b ssh:
%k           ssh key for scp                    yes              no                -b scp:


Note:

Do not use the character ^A  or ^X in any of the parameter!



Back to top

Hints & Troubleshooting

This section contains some hints and tips using the script.


Delete the tempoary file on the hosts

execute_on_all_hosts.sh does not delete the temporary file created on the hosts . To delete the script after execution add the command

rm $0

to your script


Back to top

Internas


Note: See also the documentation for the script template used for this script: scriptt.sh

The script execute_on_all_hosts.sh copies the script to the each host via scp and executes the script on a the host via ssh.

The script uses the scp and ssh binaries found in the PATH. You should start and configure ssh-agent (or a replacement) before using this script.

To change the ssh and scp binaries used by execute_on_all_host.sh, either change the PATH variable before calling execute_on_all_hosts; or create a config using execute_on_all_host.sh -C and change the variables DEFAULT_SSH_BINARY and/or  DEFAULT_SCP_BINARY in the config file, or use the parameter -D ssh_binary=x and/or -D scp_binary=x.

If running under Cygwin the script execute_on_all_hosts.sh converts the script to execute with dos2unix before copying it to the target hosts. The script searches dos2unix in the PATH. To change the binary used for dos2unix use one of the methods described above for ssh/scp; the parameter to change the binary is -D dos2unix_binary=x, the variable is DEFAULT_DOS2UNIX_BINARY.



Back to top

Known debug switches

execute_on_all_hosts.sh supports various so called debug switches (-D <debugswitch>) for customizing the script behavior and for trouble shooting.

The known debug switches in version 2.2.1 are:

[xtrnaw7@t540p scripts]$ ./execute_on_all_hosts.sh -D help
[25.11.2015 21:03:43] execute_on_all_hosts.sh v2.2.1 started at Mi 25. Nov 21:03:43 CET 2015.
[25.11.2015 21:03:43] Reading the config file "/data/develop/scripts/execute_on_all_hosts.conf" ...
Known debug switches (for -D / --debug):

  help          -- show this usage and exit
  create_documentation
                -- create the script documentation
  list_rc       -- list return codes used by this script
  msg           -- log debug messages to the file /tmp/execute_on_all_hosts.sh.475.debug
  trace         -- activate tracing to the file /tmp/execute_on_all_hosts.sh.475.trace
  fn_to_stderr  -- print the function names to STDERR
  fn_to_tty     -- print the function names to /dev/tty
  fn_to_handle9 -- print the function names to the file handle 9
  fn_to_device=filename
                -- print the function names to the file "filename"
  debugcode="x" -- execute the debug code "x" at every function start
  printargs     -- print the script arguments
  tracefunc=f1[,...,f#]
                -- enable tracing for the functions f1 to f#
                   Note: Use either debugcode=x or tracefunc=f1 - but NOT both
  debug         -- start debug env
  setvar:name=value
                -- set the variable "name" to "value"
  listfunc      -- list all functions defined and exit
  create_dump=d -- enable environment dumps; target directory is d
  SyntaxHelp    -- print syntax usage examples for the functions in the template
                   and exit
  dryrun        -- dry run only, do not execute commands
                   default prefix for dryrun is: "/usr/bin/echo "
  dryrun=prefix -- dry run only, add the prefix "prefix" to all commands

  fieldsep=x    -- change the default field separator to x
  ignore_user_in_file
               -- ignore user names from the host lists
  do_not_sort_hostlist
               -- do not sort the list of hosts
  ssh_binary=x
               -- use the ssh binary x
  scp_binary=x
               -- use the scp binary x
  dos2unix_binary=x
               -- use the dos2unix binary x
  if=x         -- use the host interface x, e.g.
                  if=a1; hostname=myhost.mydom.net -> interface to use is myhosta1.mydom.net
  enable_ForwardAgent_for_scp
               -- enable "ForwardAgent yes" for scp
  disable_ForwardAgent_for_scp
               -- disable "ForwardAgent yes" for scp
  singlestep   -- execute the commands in single step mode
                  (only used in sequentiell mode)
  print_cmd    -- write scp and ssh commands to STDOUT
  clean_known_hosts
               -- remove each host from the known_hosts file before doing
                  the scp and ssh
  restore_known_hosts
               -- restore the known_hosts file before the script ends
  rcm_server   -- RCM server to use
  rcm_user     -- RCM user
  rcm_password -- RCM password

    ---- Macros ----

  enable_gh    -- enable ForwardAgent for ssh and scp,
                  cleanup and restore the known_hosts

  sls_db       -- use DB SLS for ssh and scp access,
                  enable ForwardAgent for ssh and scp,
                  cleanup and restore the known_hosts,
                  and set the ssh user and scp user to root

  sls_scp_db   -- use DB SLS for scp access,
                  enable ForwardAgent for ssh and scp,
                  cleanup and restore the known_hosts,
                  and set the scp user to root

  sls_ssh_db   -- use DB SLS for ssh access,
                  enable ForwardAgent for ssh and scp,
                  cleanup and restore the known_hosts,
                  and set the ssh user to root

  sls_fms      -- use FMS SLS for ssh and scp access,
                  enable ForwardAgent for ssh and scp,
                  and set the ssh and the scp user to root

  sls_ssh_fms  -- use FMS SLS for ssh access,
                  enable ForwardAgent for ssh and scp,
                  and set the ssh to root and the scp user to support
                  set the interface to use to a1
[25.11.2015 21:03:43] The log file used was "/tmp/execute_on_all_hosts.sh.475.TEMP" 
[25.11.2015 21:03:43] execute_on_all_hosts.sh v2.2.1 started at Mi 25. Nov 21:03:43 CET 2015 and ended at Mi 25. Nov 21:03:43 CET 2015.
[25.11.2015 21:03:43] The RC is 0.
[xtrnaw7@t540p scripts]$




Back to top

Notes

Please see the source code of the script for additional information



Back to top

Download


Download execute_on_all_hosts.sh


Back to top