Home

execute_on_all_hosts.sh



Purpose

The script execute_on_all_hosts.sh can be used to execute a script or binary on multiple hosts. The script/binary to execute on the hosts is copied via scp to each host and then executed on every host via ssh. execute_on_all_hosts.sh also supports executing a command on all hosts without copying something to the hosts.

execute_on_hosts.sh supports customer defined templates for the scp and ssh commands. Therefor access via a Golden Host or something similar is also supported.

The script can run in sequential or parallel order on the target hosts. In the later case you can restrict the maximum number of parallel running processes and customize some other values regarding the handling of the parallel processes (see parameter -W and -w).

In addition, some not that usual features which might be useful in some environments like

- scp with Forward Agent enabling

- cleanup of the known_hosts

- different user and ssh keys for scp and ssh

are supported (see the Usage and the Known debug switches below)

The results are written to one or more logfiles.


Back to top

License


# CDDL HEADER START
#
# The contents of this file and the script are subject to the terms of the
# Common Development and Distribution License, Version 1.0 only
# (the "License").  You may not use this file except in compliance
# with the License.
#
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
# or http://www.opensolaris.org/os/licensing.
# See the License for the specific language governing permissions
# and limitations under the License.
#
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
# If applicable, add the following below this CDDL HEADER, with the
# fields enclosed by brackets "[]" replaced with your own identifying
# information: Portions Copyright [yyyy] [name of copyright owner]
#
# CDDL HEADER END

Back to top

History


Version
 Releasedate
          Description
v0.0.1
 28.02.2008 initial release
v0.0.2
 29.02.2008
 ##   28.02.2008 v0.0.2 /bs
##     added parameter hostlist, scriptfile, outputfile, and user
##     added the parameter -U
##     the script now supports multiple hostlists
##
v0.0.3
 03.03.2008
 ##   03.03.2008 v0.0.3 /bs
##     added additional workarounds for CYGWIN
##     added the parameter -I
##     added a Summary about failed hosts
##     changed __ONLY_ONCE to ${__FALSE}
##     added code to print a warning if ssh-agent is not running
##
v0.0.4
 04.03.2008
 ##   04.03.2008 v0.0.4 /bs
##     changed the default output filename to "/var/tmp/${__SCRIPTNAME##*/}.$$.log"
##     added code to call dos2unix for the script file if running in CYGWIN
##
v0.0.5
 01.08.2008
 ##   01.08.2008 v0.0.5 /bs
##     corrected a syntax error in the routine to print the short usage
##     set read permissions for the script before executing scp if necessary
##     added the parameter  -p scpoptions and  -P sshoptions
##     added code to sort the list of hosts and remove duplicate entries
##
v0.0.6
 07.08.2008
 ##   07.08.2008 v0.0.6 /bs
##     corrected a syntax error
##
v0.0.7
 20.09.2008
 ##   20.09.2008 v0.0.7 /bs
##     added the option -K / --nostrictkeys
v0.0.10
 30.03.2011 ##   30.03.2011 v0.0.10 /bs
##     added the parameter -b / --ssh_keyfile
##     added the parameter -B / --do_not_copy
##



v1.0.0
 31.05.2013
 ##   31.05.2013 v1.1.0 /bs
##     executeCommandAndLog rewritten using coprocesses (see also credits)
##     Info update: executeCommandAndLog does now return the RC of the executed
##                  command even if a logfile is defined
##     replaced the runtime system with scriptt.sh v2.0.0
##     the commands can now run in parallel in the background
##     (parameter -d, -w, and -W)
##
v1.1.0
 06.06.2013
 ##   06.06.2013 v1.1.0 /bs
##     the parameter -W and now supports a syntax like -W /5 and the
##     "/" can be replaced by ","
##     the parameter -w and now supports a syntax like -w /5, -w /5/20,
##     or -w //10. "/ "can be replaced by ","
##     You can now use seconds, minutes, or hours for the time values
##     of the parameter -w and -W can now
##
v1.2.0
 28.05.2014
 ##   28.05.2014 v1.2.0 /bs
##     use scriptt.sh v2.0.0.6
##     added the parameter -x (--excludehost)
##     added the parameter -A (--includehost)
##     enhanced support for binaries
##     reworked the usage help
##     replaced "which" whith "whence"
##
v1.3.0
 19.09.2014
 ##   19.09.2014 v1.3.0 /bs
##     the script now supports the format user@host
##
v1.3.1
 27.04.2014
 ##   27.09.2014 v1.3.1 /bs
##     added the keyword ignore_user_in_file for the parameter -D
##     added the keyword do_not_sort_hostlist for the parameter -D
##     added the keyword ssh_binary for the parameter -D
##     added the keyword scp_binary for the parameter -D
##     added the keyword dos2unix_binary for the parameter -D
##
v2.0.0
 24.07.0215
 ##   24.07.2015 v2.0.0 /bs
##     added support for different user for ssh and scp (parameter -u)
##     added support for different ssh key files for the ssh and scp user
##     added an option to list all hosts to the question to start the script
##     added parameter to enable the forward agent for scp (parameter -D enable_ForwardAgent_for_scp)
##     added support for templates for the scp usage (parameter -t scp:scp_template)
##     added support for templates for the ssh usage (parameter -t ssh:ssh_template)
##       The scp/ssh templates are useful if a golden host is used to access the machines
##     added t{SED_SEP}he debug options (parameter -D) from scriptt.sh version 2.1.0.7 21.07.2015
##     added single step mode for sequential usage (parameter -D singlestep)
##     added dry run mode (parameter -D dryrun)
##     added a parameter to cleanup the known_hosts (parameter -D clean_known_hosts)
##     added an example for golden host usage (parameter -D sls_db)
##     added the parameter "-D if=xx"
##     the parameter -K now also adds the options "-o NumberOfPasswordPrompts=0 -o ConnectTimeout=10"
##       to scp and ssh if running in parallel mode
##
v2.0.1
 009.10.2015
 ##   09.10.2015 v2.0.1 /bs
##     added support for scp via sls; the sls parameter are now:
##       -D sls_scp_db     sls for scp for DB
##       -D sls_ssh_db     sls for ssh for DB
##       -D sls_ssh_fms    sls for ssh for FMS
##
##       -D sls_fms        sls for scp and ssh for FMS
##       -D sls_db         sls for scp and ssh for DB
##
v2.1.0
 24.10.2015
 ##   24.10.2015 v2.1.0 /bs
##     added support for field separators in the hostlist files (filename[:separator])
##     added the keyword fieldsep=x to the parameter -D to change the default field separator
##     added the keyword printargs to the parameter -D
##     now the parameter -i hostlist is optional if -A hostname is used
##     now the parameter -x hostexcludelist supports a leading "?" for
##       optional hostexcludelist files
##     reworked the messages written by the script (in normal mode and
##     in verbose mode)
##     the parameter "-x none" did not work --fixed
##     the usage with "hostfile scriptfile .." did not work --fixed
##     the script now reset the current terminal using "stty sane" at
##     script end
##     the parameter -K now also adds the options "-o BatchMode=yes -o PasswordAuthentication=no"
##       to scp and ssh if running in parallel mode
##
v2.2.0
 27.10.2015
 ##   27.10.2015 v2.2.0 /bs
##     execute_on_all_hosts.sh now creates up to 10 backups of the logfile
##     and rewrites the logfile every time.
##     To overwrite the number of backups use the following syntax for
##     the parameter -l:
##
##        -l logfile,[no_of_backups_of_the_logfile]
##
##     The default number of backups for the log file is configured in the variable
##     MAX_NO_OF_LOGFILES
##
v2.2.1
 25.11.2015
 ##   25.11.2015 v2.2.1 /bs
##      the internal sed field separator is now "CTRL-X" so the pipe character
##      "|" can be used in commands to execute again (see the variable SED_SEP)
v2.2.2
 25.08.2017
 ##   25.08.2017 v2.2.2 /bs
##      the parameter -t did not support templates with colon ":" - fixed
##      the script printed in some circumstances the wrong number of hosts to process in the summary (There are 2 hosts to process) - fixed
##
v2.2.3
 26.08.2017
 ##   26.08.2017 v2.2.3 /bs
##      New macros for the parameter -D:
##
##       -D sls_scp_db_unxxx4     sls for scp for DB as user unxxx4
##       -D sls_ssh_db_unxxx4     sls for ssh for DB as user unxxx4
##       -D sls_db_unxxx4         sls for scp and ssh for DB as user unxxx4
##       -D ticket_id=x           define ticket id for DB access
##       -D use_ssh_wrapper       use my ssh and scp wrapper script for SLS
##       -D no_use_ssh_wrapper    do not use my ssh and scp wrapper script for SLS
##
##      the ticket string set with the parameter "-D ticket_id=x" is now part of the
##      default ssh and scp template (in the default the ticket string is empty)
##
##      improved some error messages to be more clear
##      in the code to enable the forward agent for scp (ENABLE_FORWARD_AGENT_FOR_SCP_CODE)
##        was a "=" missing so that it did not work -- fixed
##
v2.3.0
 21.07.2021
 ##   21.07.2021 v2.3.0 /bs
##     the parameter "-W timeout/intervall" now defines the timeout for the scp and ssh commands and the intervall
##       between the ssh/scp commands for the machines if running in sequential mode
##       (note: a timeout value will only work if the executable "timeout" is available via the PATH; see the source code for
##              a timeout implementation in a ksh script)
##     the script now prints a summary at script end like this :
##       (the 2nd line is only shown in sequential mode):
##
##       [16.07.2021 08:52:23] The script runtime is (day:hour:minute:seconds) 0:00:04:33 (= 273 seconds) for 84 hosts
##       [16.07.2021 08:52:23]   ( -> about 4 second(s) her host)
##
##     added the template sls_db_unxxx4_w_timeout (-D sls_db_unxxx4_w_timeout) :
##       Use sls as user unxxx4 to connect with a timeout of 15 seconds for each ssh/scp command
##       Set the variable SSH_SCP_CMD_TIMEOUT before starting the script change the timeout value
##       (note: The template will only work if the executable "timeout" is available via the PATH; see the source code for
##              a timeout implementation in a ksh script)
##     added the alias "sls" for sls_db_unxxx4_w_timeout
##     the macrco sls_db_unxxx4 (and also sls_db_unxxx4_w_timeout) now uses the scp and ssh binaries defined
##       with the parameter "-D ssh_binary=x" or "-D scp_binary=x" if defined (def. values are /usr/bin/ssh and /usr/bin/scp)
##     the function die will now call the al<<ias __unsettraps if the script is running in Solaris
##     LogRuntimeInfo rewritten -- the old syntax did not work in the ksh from Solaris 11
##     added the keyword log_ssh_cmds for the parameter -D
##     added the keyword do_not_log_ssh_cmds for the parameter -D
##     the value for the setting "-o ConnectTimeout" used if the parameter "-K" is used in parallel mode can be defined using the
##       environment variable SSH_SCP_CMD_TIMEOUT or the parameter -W
##     the script now prints the list of hosts on which the executed command ended with a non-zero return code
##       at script end if running in sequential mode
##     disabled a wrong error message regarding restoring the known_hosts file
##     the script ignored the return code of the scp command in sequential mode -- fixed
##     the output of the ssh and scp commands in sequential mode was not written to the logfile -- fixed
##       (use the parameter "-D do_not_log_ssh_cmds" to disable the logging of the ssh command output to the logfile)
##     fixed some typos
##     fixed some minor errors
##     ${__FUNCTION_EXIT} was executed twice in the function isNumber  -- fixed
##     the initial runtime code was missing in the functions LogHeader, rand, PrintLockFileErrorMsg, and USER_SIGNAL_HANDLER -- fixed
##     the script did not print the list of failed hosts at script end in sequential mode -- fixed
##
v2.3.1
 08.09.2022
 ##   08.09.2022 v2.3.1 /bs
##     the script now also removes the entries for the IP addresses from the known_hosts file if the parameter "-D clean_known_hosts" is used
##     the script now also uses ssh-keygen to remove entries from the known_hosts file if the parameter "-D clean_known_hosts" is used
##       (-> hashed entries in the file will also be removed)
##     added the keyword ssh-keygen_binary for the parameter -D
##     added the keyword nameserver for the parameter -D
##     added the keyword delete_known_hosts for the parameter -D
##     added the keyword clean_and_restore_known_hosts for the parameter -D
##     added the keyword delete_and_restore_known_hosts for the parameter -D
##     added the keyword ignore_known_hosts for the parameter -D
##     added the keyword SLS_db_unxxx4 for the parameter -D
##     added the keyword SLS_db_unxxx4_sudo for the parameter -D
##     added the keyword sls_db_unxxx4_sudo for the parameter -D
##     added the variable TRACE_MAIN (-> trace the main function if the variable TRACE_MAIN is set to 0 before starting the script)
##     added the variable TRACE_PROMPT (-> PS4 prompt used for tracing the main function; set this variable before starting the script)
##    
v2.3.2 09.12.2022 ##   09.12.2022 v2.3.2 /bs
##     the macro sls_ssh_db_unxxx4 was not working -- fixed
##
v2.3.3 05.02.2023 ##   05.02.2023 v2.3.3 /bs
##     the function CalculateSeconds did not handle values ending with a "h" correct -- fixed
##
v2.3.4 20.02.2023 ##   20.02.2023 v2.3.4 /bs
##     the script did not like leading white spaces in the files with the host lists -- fixed
##
v2.3.5 12.01.2024 ##   12.01.2024 v2.3.5 /bs
##     replaced "egrep" with "grep -E" (see variable EGREP)
##     corrected some minor bugs in the commands to print messages
##






Back to top

Operating system


Tested with Solaris and Linux; the previous version was also tested with Cygwin; the script should work on other Unix Operating systems with a kornshell (ksh).


Back to top

Language / type


Kornshell Script


Back to top

Prerequisites

ssh and scp must be installed and configured; the sshd daemon must run on the hosts.

In addition, ssh-agent or something similar should be running and configured on the machine used to execute this script and login via public ssh key should be enabled on the target hosts.



Back to top

Usage

[xtrnaw7@t15g /data/develop/scripts]$ ./execute_on_all_hosts.sh -h
[09.09.2022 09:56:55] execute_on_all_hosts.sh v2.3.1 started at Fri Sep  9 09:56:55 CEST 2022 (The PID of this process is 4026935).
[09.09.2022 09:56:55] Reading the config file "/data/develop/scripts/execute_on_all_hosts.conf" ...
  execute_on_all_hosts.sh v2.3.1 - copy a script to a list of hosts via scp and execute it on the hosts using ssh

  Usage: execute_on_all_hosts.sh [-v|+v] [-q|+q] [-h] [-l logfile|+l] [-y|+y] [-n|+n]
                    [-D debugswitch] [-a|+a] [-O|+O] [-f|+f] [-C] [-H] [-X] [-S n] [-V] [-T]
                    [-i hostlist{,hostlist1{,...}] [-s scriptfile] [-o outputfile] [-u sshuser|scp:scpuser|ssh:sshuser] [-I basedir]
                    [-c shell] [-k] [-U] [-p scpoptions] [-P sshoptions] [-K] [-R] [-B|+B] [-b ssh_keyfile|scp:scp_keyfile|ssh:ssh_keyfile]
                    [-x excludehost] [-A includehost] [-t ssh:ssh_template] [-t scp:scp_template]
                    [-d|+d] [-W [timeout[/intervall]] [-w NoOfBackgroundProcs[/intervall[/timeout]]]]
                    [hostlist [scriptfile [outputfile [sshuser]]]]

  The parameters scriptfile, outputfile, and sshuser overwrite the options; hostlist (or includehost) and scriptfile are mandatory either
  as parameter or as option.

  For optimal usage ssh login via public key should be enabled on the target hosts and
  the ssh agent should run on this host (the agent is currently running).

  Use "-D help" to view the known debug options.

  Use the parameter "-v -h [-v]" to view the detailed online help; use the parameter "-X" to view some usage examples.

  It is strongly recommended to test the script execution in dry run mode (parameter "-D dryrun") before doing the real work!

  see also http://bnsmb.de/solaris/execute_on_all_hosts.html


[09.09.2022 09:56:55] Use "-v -h", "-v -v -h", "-v -v -v -h" or "+h" for a long help text
[09.09.2022 09:56:55] The log file used was "/tmp/execute_on_all_hosts.sh.4026935.TEMP" 
[09.09.2022 09:56:55] execute_on_all_hosts.sh v2.3.1 started at Fri Sep  9 09:56:55 CEST 2022 and ended at Fri Sep  9 09:56:55 CEST 2022.
[09.09.2022 09:56:55] The script runtime is (day:hour:minute:seconds) 0:00:00:00 (= 0 seconds) for  hosts
[09.09.2022 09:56:55] The RC is 1.
[xtrnaw7@t15g /data/develop/scripts]$


Detailed Usage help
[xtrnaw7@t15g /data/develop/scripts]$ ./execute_on_all_hosts.sh -h -v -v
[09.09.2022 09:58:28] execute_on_all_hosts.sh v2.3.1 started at Fri Sep  9 09:58:28 CEST 2022 (The PID of this process is 4027196).
[09.09.2022 09:58:28] Reading the config file "/data/develop/scripts/execute_on_all_hosts.conf" ...
[09.09.2022 09:58:28] Parameter after processing the default parameter are:  "-h -v -v" 
  execute_on_all_hosts.sh v2.3.1 - copy a script to a list of hosts via scp and execute it on the hosts using ssh

  Usage: execute_on_all_hosts.sh [-v|+v] [-q|+q] [-h] [-l logfile|+l] [-y|+y] [-n|+n]
                    [-D debugswitch] [-a|+a] [-O|+O] [-f|+f] [-C] [-H] [-X] [-S n] [-V] [-T]
                    [-i hostlist{,hostlist1{,...}] [-s scriptfile] [-o outputfile] [-u sshuser|scp:scpuser|ssh:sshuser] [-I basedir]
                    [-c shell] [-k] [-U] [-p scpoptions] [-P sshoptions] [-K] [-R] [-B|+B] [-b ssh_keyfile|scp:scp_keyfile|ssh:ssh_keyfile]
                    [-x excludehost] [-A includehost] [-t ssh:ssh_template] [-t scp:scp_template]
                    [-d|+d] [-W [timeout[/intervall]] [-w NoOfBackgroundProcs[/intervall[/timeout]]]]
                    [hostlist [scriptfile [outputfile [sshuser]]]]

  The parameters scriptfile, outputfile, and sshuser overwrite the options; hostlist (or includehost) and scriptfile are mandatory either
  as parameter or as option.

  For optimal usage ssh login via public key should be enabled on the target hosts and
  the ssh agent should run on this host (the agent is currently running).

  Use "-D help" to view the known debug options.

  Use the parameter "-v -h [-v]" to view the detailed online help; use the parameter "-X" to view some usage examples.

  It is strongly recommended to test the script execution in dry run mode (parameter "-D dryrun") before doing the real work!

  see also http://bnsmb.de/solaris/execute_on_all_hosts.html



 Note: Use -{switch} or --{longswitch} to turn an option on;
       use +{switch} or ++{longswitch} to turn an option off

       The long format of the parameter (--parameter/++parameter) is not supported by all ksh implementations


    Parameter:

      -v|+v - turn verbose mode on/off; current value: y
              Long format: --verbose / ++verbose
      -q|+q - turn quiet mode on/off; current value: n
              Long format: --quiet / ++quiet
      -h    - show usage
              Long format: --help
      -l    - set the logfile
              current value: /var/tmp/execute_on_all_hosts.LOG
              use "-l [logfile,no_of_backups]" to define the number of retained backups
              of the logfile; the default no of backups is 10
              Long format: --logfile
      +l    - do not write a logfile
              Long format: ++logfile
      -y|+y - assume yes to all questions or not
              Long format: --yes / ++yes
      -n|+n - assume no to all questions or not
              Long format: --no /++no
      -D|+D - debug switch
              current value:
              use "-D help" to list the known debug switches
              Long format: --debug / ++debug
      -a|+a - turn colors on/off; current value: n
              Long format: --color / ++color
      -O|+O - overwrite existing files or not; current value: n
              Long format: --overwrite / ++overwrite
      -f|+f - force; do it anyway; current value: n
              Long format: --force / ++force
      -C    - write a default config file in the current directory and exit
              Long format: --writeconfigfile
      -H    - write extended usage to STDERR and exit
              Long format: --doc
      -X    - write usage examples to STDERR and exit
              Long format: --view_examples
      -S n  - print error/warning summaries:
              n = 0 no summariess, 1 = print error msgs,
              2 = print warning msgs, 3 = print error and warning mgs
              Current value: 0
              Long format: --summaries
      -V    - write version number to STDOUT and exit
              Long format: --version
      -T    - append STDOUT and STDERR to the file "/var/tmp/execute_on_all_hosts.sh.4027196.tee.log"
              Long format: --tee

      -i hostlist[:fieldseparator]
              "hostlist" is the file with the list of hosts to process.
              Format: one hostname per line (use user@host to use an explicit
              user for a host; this user is then used for scp and ssh to this host);
              empty lines and lines beginning with a hash "#" are ignored.
              The script only reads the first field of each line. The field
              separator is " ". To change the field separator
              add ":fieldseparator" to the filename.
              The comma "," can not be used as field separator; use
                -D fieldsep=","
              to change the default field separator instead.
              Use a comma "," to separate multiple hostlists or use the
              parameter "-i" more than one time.
              All hostlists are merged and sorted. Each host in the list(s)
              is only processed one time.
              Missing hostlist files are treated as error. To ignore a
              missing hostlist file add a leading "?" to the filename.
              Current Value: ""
              Long format: --hostlist

      -x hostname
      -x hostexcludelist[:fieldseparator]
              exclude the host "hostname" from the execution
              This parameter can be used multiple times; regular expressions
              for "hostname" are allowed.
              Use "-x none" to clear the list of hosts to exclude.
              Use a relative or absolute filename for "hostname" to read the
              hosts to exclude from a file. Format of the file:
              one hostname per line; empty lines and lines beginning with "#"
              are ignored.
              The script only reads the first field of each line. The field
              separator is " ". To change the field separator
              add ":fieldseparator" to the filename.
              The comma "," can not be used as field separator; use
                -D fieldsep=","
              to change the default field separator instead.
              Missing hostexcludelist files are treated as error. To ignore a
              missing hostexcludelist file add a leading "?" to the filename.
              Current Value: ""
              Long format: --excludehost

      -A hostname
              add the host "hostname" to the list of hosts
              This parameter can be used multiple times; regular expressions
              for "hostname" are not allowed.
              Use "-A none" to clear the list of hosts to include.
              Current Value: ""
              Long format: --includehost

      -s scriptfile
              scriptfile is the script or executable to execute on all hosts.
              It will be copied to each host using the user "scpuser" and
              executed on each host with the user "sshuser".
              If "scriptfile" is a binary it will be executed directly;
              if "scriptfile" is not a binary it will be executed
              either by the default shell for "sshuser" or by the
              shell specified with the parameter "-c".
              If this script is running in a cygwin session and the binary
              dos2unix is available via the path the script will be converted
              to Unix style using dos2unix before copying it to the hosts.
              If "-B" is used, the value for this parameter is a command to
              execute on the hosts without first copying it to the host. In
              this case the command is executed by the user "sshuser"
              without calling a shell.
              Use \; in this case to separate multiple commands to execute.
              e.g: -s "uname -a \; uptime "
              Current value: ""
              Long format: --scriptfile

      -u sshuser
              "sshuser" is the userid to use for ssh and scp on the target
              hosts
              To use different user for ssh and scp use:
              -u ssh:sshuser
              -u scp:scpuser
              If "-u scp:scpuser" is not used the sshuser is also used for scp.
              Current value for sshuser is "support"
              Current value for scpuser is ""
              Long format: --sshuser

      -c shell_to_use
              "shell_to_use" is the shell to execute the script on the hosts
              Use "none" to execute the script with the default shell
              of the "sshuser".
              If "-B" is used or if the command to execute (parameter "-s")
              is a binary this parameter is ignored and the command
              is always executed directly.
              Current value "/usr/bin/ksh"
              Long format: --shell

      -I basedir
              "basedir" is the directory with the hostlist and the
              script files; if this parameter is found the script searches
              the hostlist files and the script files in this directory also
              Current value: ""
              Long format: --basedir

      -o outputfile
              "outputfile" is the name of the file for the output generated
              by the commands. If "-U" is also used the output for each host
              is logged into a separate file named "<outputfile>.<hostname>"
              Current value "/var/tmp/execute_on_all_hosts.sh.4027196.cmds.log"
              Long format: --outputfile

      -p scpoptions
              "scpoptions" are additional options for scp, e.g
                 -p "-o StrictHostKeyChecking=no"
              Current value ""
              Long format: --scpoptions

      -P sshoptions
              "sshoptions" are additional options for ssh
              Current value ""
              Long format: --sshoptions

      -t ssh:ssh_template
      -t scp:scp_template
              Define a new template for the scp or ssh command
              (see below for the known template placeholder)
              The scp/ssh templates are useful if a golden host is used
              to access the machines
              Current value:  ssh template:  %b %o %k %t%u@%i %c %s
                              scp template:  %b %o %k %S %t%u@%i:%s
              Long format --template

      -K|+K use the option "StrictHostKeyChecking=no" for ssh and scp.
              In parallel mode also add the ssh and scp options
                "-o NumberOfPasswordPrompts=0 -o ConnectTimeout= -o BatchMode=yes -o PasswordAuthentication=no"
              The timeout value used can be changed with the variable SSH_SCP_CMD_TIMEOUT
              Current value: n
              Long format: --nostrictkeys

      -k|+k add comments to the output file
              Current value: y
              Long format: --nocomments

      -B|+B do not upload a script to the hosts
              In this case the value for the parameter "-s" is interpreted
              as command to execute and the value for the shell to
              use (parameter "-c") is ignored.
              Note: For commands with options like "uname -a" you should use a script
              Current value: n
              Long format: --do_not_copy

      -b ssh_keyfile
              Use the ssh key file "ssh_keyfile" for ssh and scp.
              Default:
              Use the default ssh key files for the user "sshuser" for ssh
              and the default key file for the user "scpuser" for scp
              To use different key files for ssh and scp use:
              -b ssh:ssh_keyfile
              -b scp:ssh_keyfile
              Current value for ssh_keyfile is: ""
              Current value for scp_keyfile is: ""
              Long format: --ssh_keyfile

      -U|+U create unique log files
              (see parameter "-o")
              Current value: n
              Long format: --uniquelogfiles

      -R|+R  use RCM methods
              The default value depends on the existence of copy_table.pl
              in the PATH.
              Use the environment variables RCM_USERID and RCM_PASSWORD
              to set the RCM userid and password. If these variables are
              not set the script will prompt the user for them.
              Current value: n
              Long format: --rcm

      -d|+d execute the commands in parallel in the background
              Current value: n
              Long format: --parallel

      -W timeout[/intervall]
              Usage in parallel mode:
             
              "timeout" is the timeout for background processes and
              "intervall" is the wait intervall for background processes.
              Use "-1" or "none" for the timeout value to disable the
              timeout.
              Current values:
                Timeout: 1h (=  second(s) )
                Intervall: 2s (= 2 second(s) )
              Use a trailing "m" for times in minutes or a trailing "h"
              for times in hours; "s" is for seconds which is the default
              if neither "m" nor "h" is used.
              Use "default" for any value to use the default value.
              Use " -W /intervall" to only change the intervall.
              You can also use "," to separate the values.
             
              Usage in sequentiell mode:
             
              In sequentiell mode the parameter
             
              -W timeout/intervall
             
              can be used to define the timeout for the scp and ssh commands
              and the intervall between the ssh/scp commands for the hosts.
              "timeout" is the timeout for the scp and ssh commands (This
              will only work if the executable "timeout" is available via PATh variable).
              "intervall" is the time to wait between the ssh/scp commands
              for the hosts.
              Alternatively, the timeout for ssh/scp commands can be defined in the
              environment variable SSH_SCP_CMD_TIMEOUT.
              (see above for the supported format for both values)
              Long format: --timeout

      -w noOfBackgroundprocesses[/startIntervall[/maxStartTime]]
              "noOfBackgroundprocesses" is the max. number of background
              processes running at the same time, "startIntervall" is the
              wait intervall and "maxStartTime" is the timeout for starting
              the background processes.
              Use "-1" or "none" for an unlimited number of parallel
              background processes.
              Use "-1" or "none" for the timeout value to disable
              the timeout.
              Current values:
                Max. number of parallel running background processes: -1
                Wait intervall: 5s (= 5 second(s) )
                Timeout value: 5m (= 300 second(s) )
              Use a trailing "m" for times in minutes or a trailing "h"
              for times in hours; "s" is for seconds which is the default
              if neither "m" nor "h" is used.
              Use "default" for any value to use the default value.
              Use " -w /startIntervall" to only change the start intervall.
              Use " -w //maxStartTime" to only change the timeout
              You can also use "," to separate the values.
              This parameter is only used if the commands are executed
              in parallel (parameter "-d")
              Long format: --noOfbackgroundProcesses


Placeholder in the ssh and scp templates are:

Placeholder  usage                              used for scp?    used for ssh?     parameter
-------------------------------------------------------------------------------------------------
%%           %                                  yes              yes               n/a
%b           scp binary                         yes              no                -D scp_binary
%b           ssh binary                         no               yes               -D ssh_binary

%S           source script or command           yes              yes               -s
%s           script/binary on the target host   yes              yes               n/a
%c           shell to use                       no               yes               -c

%u           scp user                           yes              no                -u scp:
%u           ssh user                           no               yes               -u ssh:
%t           ticket ID string                   yes              yes               -D ticket_id=x

%h           target host (FQN)                  yes              yes               n/a
%H           target host (short hostname)       yes              yes               n/a
%d           DNS domain                         yes              yes               n/a
%i           target host interface              yes              yes               n/a

%o           scp options                        yes              no                -p
%o           ssh options                        no               yes               -P

%k           ssh key for ssh                    no               yes               -b ssh:
%k           ssh key for scp                    yes              no                -b scp:

Do not use the characters ^A or ^X in any of the parameter!

Environment variables that are used if set (0 = TRUE, 1 = FALSE):

  __DEBUG_CODE (Current value: "")
  __RT_VERBOSE_LEVEL (Current value: "1")
  __QUIET_MODE (Current value: "1")
  __VERBOSE_MODE (Current value: "0")
  __VERBOSE_LEVEL (Current value: "2")
  __OVERWRITE_MODE (Current value: "1")
  __USER_BREAK_ALLOWED (Current value: "0")
  __NO_TIME_STAMPS (Current value: "1")
  __NO_HEADERS (Current value: "1")
  __USE_COLORS (Current value: "1")
  __USE_RBAC (Current value: "1")
  __RBAC_BINARY (Current value: "/usr/bin/pfexec")
  __TEE_OUTPUT_FILE (Current value: "/var/tmp/execute_on_all_hosts.sh.4027196.tee.log")
  __INFO_PREFIX (Current value: "INFO:")
  __DEBUG_PREFIX (Current value: "DEBUG:")
  __WARNING_PREFIX (Current value: "WARNING:")
  __ERROR_PREFIX (Current value: "ERROR:")
  __RUNTIME_INFO_PREFIX (Current value: "RUNTIME INFO:")
  __NO_CLEANUP (Current value: "1")
  __NO_EXIT_ROUTINES (Current value: "1")
  __NO_TEMPFILES_DELETE (Current value: "1")
  __NO_TEMPMOUNTS_UMOUNT (Current value: "1")
  __NO_TEMPDIR_DELETE (Current value: "1")
  __NO_FINISH_ROUTINES (Current value: "1")
  __CLEANUP_ON_ERROR (Current value: "1")
  __CREATE_DUMP (Current value: "")
  __DUMP_ALREADY_CREATED (Current value: "")
  __DUMPDIR (Current value: "")
  __USE_ONLY_KSH88_FEATURES (Current value: "1")
  CONFIG_FILE (Current value: "")
  SSH_SCP_CMD_TIMEOUT (Current value: "")
  TRACE_MAIN (Current value: "")
  TRACE_PROMPT (Current value: "")
[09.09.2022 09:58:28] The log file used was "/tmp/execute_on_all_hosts.sh.4027196.TEMP" 
[09.09.2022 09:58:28] execute_on_all_hosts.sh v2.3.1 started at Fri Sep  9 09:58:28 CEST 2022 and ended at Fri Sep  9 09:58:28 CEST 2022.
[09.09.2022 09:58:28] The script runtime is (day:hour:minute:seconds) 0:00:00:00 (= 0 seconds) for  hosts
[09.09.2022 09:58:28] The RC is 1.
[xtrnaw7@t15g /data/develop/scripts]$
 





back to top

Example

In this example we only execute a "uname -a" on a list of hosts:

(Note: This example was created with one of the previous versions of the script)


# the file with the list of hosts:
#
xtrnaw7@t30:/data/develop/scripts$ cat hosts
# machines in my testenvironment
sol9
pb001
ferrari
sol8
sol2
sol3
ultra30

# The file with the commands to execute on each host:
#
xtrnaw7@t30:/data/develop/scripts$ cat testscript0
uname -a

# run the script (using the user root on the target hosts):
#
xtrnaw7@t30:/data/develop/scripts$ ./execute_on_all_hosts.sh -i hosts -s testscript0  -u root
[28.02.2008 21:01:03] execute_on_all_hosts.sh started on Thu Feb 28 21:01:03 CET 2008 
[28.02.2008 21:01:03] Reading the config file "./execute_on_all_hosts.conf" ...
[28.02.2008 21:01:03] Using the log file "/var/tmp/execute_on_all_hosts.LOG" 
[28.02.2008 21:01:03] 
[28.02.2008 21:01:03] Executing the script 
[28.02.2008 21:01:03]     testscript0
[28.02.2008 21:01:03] as ssh user 
[28.02.2008 21:01:03]     root
[28.02.2008 21:01:03] on every host listed in the file 
[28.02.2008 21:01:03]     hosts
[28.02.2008 21:01:03] 
[28.02.2008 21:01:03] The shell to execute the script is 
[28.02.2008 21:01:03]     /usr/bin/ksh
[28.02.2008 21:01:03] The output of the commands will be logged in the file
[28.02.2008 21:01:03]     /var/tmp/execute_on_all_hosts.sh.19003.log
[28.02.2008 21:01:03] 
Okay (y/N)? y
[28.02.2008 21:01:06] Starting processing ...
[28.02.2008 21:01:06] ---- Processing "sol9" ...
testscript0                                                                   100%    9     0.0KB/s   00:00   
SunOS sol9 5.11 snv_78 sun4u sparc sun4u
[28.02.2008 21:01:09] ---- Processing "pb001" ...
testscript0                                                                   100%    9     0.0KB/s   00:00   
SunOS pb001 5.11 snv_78 i86pc i386 i86pc
[28.02.2008 21:01:10] ---- Processing "ferrari" ...
ssh: connect to host ferrari port 22: No route to host
lost connection
[28.02.2008 21:01:13] ERROR: Error copying the script to the host "ferrari" - could not execute the script on that host
[28.02.2008 21:01:13] ---- Processing "sol8" ...
ssh: connect to host sol8 port 22: Connection refused
lost connection
[28.02.2008 21:01:13] ERROR: Error copying the script to the host "sol8" - could not execute the script on that host
[28.02.2008 21:01:13] ---- Processing "sol2" ...
testscript0                                                                   100%    9     0.0KB/s   00:00   
SunOS sol2 5.10 Generic_127111-03 sun4u sparc SUNW,Ultra-5_10
[28.02.2008 21:01:16] ---- Processing "sol3" ...
ssh: connect to host sol3 port 22: No route to host
lost connection
[28.02.2008 21:01:19] ERROR: Error copying the script to the host "sol3" - could not execute the script on that host
[28.02.2008 21:01:19] ---- Processing "ultra30" ...
Warning: the RSA host key for 'ultra30' differs from the key for the IP address '192.168.1.20'
Offending key for IP in /home/xtrnaw7/.ssh/known_hosts:13
Matching host key in /home/xtrnaw7/.ssh/known_hosts:40
testscript0                                                                   100%    9     0.0KB/s   00:00   
Warning: the RSA host key for 'ultra30' differs from the key for the IP address '192.168.1.20'
Offending key for IP in /home/xtrnaw7/.ssh/known_hosts:13
Matching host key in /home/xtrnaw7/.ssh/known_hosts:40
SunOS ultra30 5.10 Generic_120011-14 sun4u sparc SUNW,Ultra-30
[28.02.2008 21:01:21] 
[28.02.2008 21:01:21] All done
[28.02.2008 21:01:21] 
[28.02.2008 21:01:21] The output of the commands is logged in the file
[28.02.2008 21:01:21]     /var/tmp/execute_on_all_hosts.sh.19003.log
[28.02.2008 21:01:21] 
[28.02.2008 21:01:21] The log file used was "/var/tmp/execute_on_all_hosts.LOG" 
[28.02.2008 21:01:21] execute_on_all_hosts.sh ended on Thu Feb 28 21:01:21 CET 2008.
[28.02.2008 21:01:21] The RC is 0.

# The output of the commands is written to the logfile:

xtrnaw7@t30:/data/develop/scripts$ cat /var/tmp/execute_on_all_hosts.sh.19003.log

# ### ---- Log of the script executed on host "sol9" --- start ---
SunOS sol9 5.11 snv_78 sun4u sparc sun4u
# ### ---- Log of the script executed on host "sol9" --- end ---

# ### ---- Log of the script executed on host "pb001" --- start ---
SunOS pb001 5.11 snv_78 i86pc i386 i86pc
# ### ---- Log of the script executed on host "pb001" --- end ---

# ### ---- Log of the script executed on host "sol2" --- start ---
SunOS sol2 5.10 Generic_127111-03 sun4u sparc SUNW,Ultra-5_10
# ### ---- Log of the script executed on host "sol2" --- end ---

# ### ---- Log of the script executed on host "ultra30" --- start ---
Warning: the RSA host key for 'ultra30' differs from the key for the IP address '192.168.1.20'
Offending key for IP in /home/xtrnaw7/.ssh/known_hosts:13
Matching host key in /home/xtrnaw7/.ssh/known_hosts:40
SunOS ultra30 5.10 Generic_120011-14 sun4u sparc SUNW,Ultra-30
# ### ---- Log of the script executed on host "ultra30" --- end ---


Other Example
Note: "..." is a replacment for the list of other hosts processed in this example


xtrnaw7:~ $ time /data/develop/scripts/execute_on_all_hosts.sh   -D print_cmd -i machine_list_dev  -i machine_list_uat    +R  -K  -s ./ls_messages  -o ./ls_messages.out  -W 10 -D if=a1   -u admin01 -y
[24.07.2021 13:00:54] execute_on_all_hosts.sh v2.3.0 started at Sat Jul 24 13:00:54 CEST 2021 (The PID of this process is 456676).
[24.07.2021 13:00:54] No config file ("execute_on_all_hosts.conf") found (use -C to create a default config file)
[24.07.2021 13:00:54] DEBUG: Enabling printing the scp and ssh commands to STDOUT.
[24.07.2021 13:00:54] DEBUG: Using the host interface a1
[24.07.2021 13:00:54] Using the log file "/var/tmp/execute_on_all_hosts.LOG"
[24.07.2021 13:00:54]
[24.07.2021 13:00:54] Executing the script
[24.07.2021 13:00:54]     ./ls_messages
[24.07.2021 13:00:54] as ssh user
[24.07.2021 13:00:54]     admin01 (using the default ssh key file)
[24.07.2021 13:00:54] on every host listed in the file(s)
[24.07.2021 13:00:54]     machine_list_dev (the field separator is " ")
[24.07.2021 13:00:54]     machine_list_uat (the field separator is " ")
[24.07.2021 13:00:54] The scp user to copy the files is
[24.07.2021 13:00:54]     admin01 (using the default ssh key file)
[24.07.2021 13:00:54]
[24.07.2021 13:00:54] The shell to execute the command is
[24.07.2021 13:00:54]     /usr/bin/ksh
[24.07.2021 13:00:54]
[24.07.2021 13:00:54] Using the ssh binary "ssh".
[24.07.2021 13:00:54] Using the scp binary "scp".
[24.07.2021 13:00:54] The additional options for scp are:
[24.07.2021 13:00:54]      -o StrictHostKeyChecking=no
[24.07.2021 13:00:54] The additional options for ssh are:
[24.07.2021 13:00:54]      -o StrictHostKeyChecking=no
[24.07.2021 13:00:54] The template for scp commands used is:
[24.07.2021 13:00:54]      %b %o %k %S %t%u@%i:%s
[24.07.2021 13:00:54] The template for ssh commands used is:
[24.07.2021 13:00:54]      %b %o %k %t%u@%i %c %s
[24.07.2021 13:00:54] The output of the commands will be logged in the file
[24.07.2021 13:00:54]     /home/xtrnaw7/ls_messages.out
[24.07.2021 13:00:54]
[24.07.2021 13:00:54] RCM support is not used
[24.07.2021 13:00:54]
[24.07.2021 13:00:54] The scp/ssh processes will run sequential one after the other.
[24.07.2021 13:00:54] Timeout value for the sequential ssh/scp commands is 10 seconds
[24.07.2021 13:00:54]
[24.07.2021 13:00:54] There are 472 hosts to process
[24.07.2021 13:00:54]
[24.07.2021 13:00:54] Starting processing ...
 
[24.07.2021 13:00:54] ---- Processing "isbdadapp4.isbs.com" (ssh user is "admin01", scp user is "admin01")  ... ( 1 from 472)
[24.07.2021 13:00:54] The scp command is:  /usr/bin/timeout --foreground --preserve-status 10   scp  -o StrictHostKeyChecking=no   ./ls_messages admin01@isbdadapp4a1.isbs.com:/tmp/tmp_456676_execute_on_all_hosts.sh
*** INFO: Hostname "isbdadapp4a1" replaced with "isbdadapp4a1.isbs.com"
*** INFO: Adding the host key for isbdadapp4a1.isbs.com to the known_hosts
getaddrinfo isbdadapp4a1.isbs.com: Name or service not known
getaddrinfo isbdadapp4a1.isbs.com: Name or service not known
getaddrinfo isbdadapp4a1.isbs.com: Name or service not known
*** INFO: Executing now :  /usr/bin/scp -S /tmp/ssh_wrapper_for_scp   "-o"  "StrictHostKeyChecking=no"  "./ls_messages"  ticket_id=9999@admin01@isbdadapp4a1.isbs.com:'/tmp/tmp_456676_execute_on_all_hosts.sh'
ssh: Could not resolve hostname isbdadapp4a1.isbs.com: Name or service not known
lost connection
[24.07.2021 13:00:54] ERROR: Error copying the script to the host "isbdadapp4.isbs.com" - could not execute the script on that host
 
[24.07.2021 13:00:54] ---- Processing "isbdaem0001.isbs.com" (ssh user is "admin01", scp user is "admin01")  ... ( 2 from 472)
[24.07.2021 13:00:54] The scp command is:  /usr/bin/timeout --foreground --preserve-status 10   scp  -o StrictHostKeyChecking=no   ./ls_messages admin01@isbdaem0001a1.isbs.com:/tmp/tmp_456676_execute_on_all_hosts.sh
*** INFO: Hostname "isbdaem0001a1" replaced with "isbdaem0001a1.isbs.com"
*** INFO: Hostname "isbdaem0001a1.isbs.com" removed from the known_hosts file
*** INFO: Adding the host key for isbdaem0001a1.isbs.com to the known_hosts
# isbdaem0001a1.isbs.com:22 SSH-2.0-SSH
# isbdaem0001a1.isbs.com:22 SSH-2.0-SSH
# isbdaem0001a1.isbs.com:22 SSH-2.0-SSH
*** INFO: Executing now :  /usr/bin/scp -S /tmp/ssh_wrapper_for_scp   "-o"  "StrictHostKeyChecking=no"  "./ls_messages"  ticket_id=9999@admin01@isbdaem0001a1.isbs.com:'/tmp/tmp_456676_execute_on_all_hosts.sh'
[24.07.2021 13:00:56] The ssh command is:  /usr/bin/timeout --foreground --preserve-status 10   ssh  -o StrictHostKeyChecking=no   admin01@isbdaem0001a1.isbs.com /usr/bin/ksh /tmp/tmp_456676_execute_on_all_hosts.sh 
*** INFO: Hostname "isbdaem0001a1" replaced with "isbdaem0001a1.isbs.com"
*** INFO: Hostname "isbdaem0001a1.isbs.com" removed from the known_hosts file
*** INFO: Adding the host key for isbdaem0001a1.isbs.com to the known_hosts
# isbdaem0001a1.isbs.com:22 SSH-2.0-SSH
# isbdaem0001a1.isbs.com:22 SSH-2.0-SSH
# isbdaem0001a1.isbs.com:22 SSH-2.0-SSH
*** INFO: Executing now :  /usr/bin/ssh   "-o"  "StrictHostKeyChecking=no"  ticket_id=9999@admin01@isbdaem0001a1.isbs.com "/usr/bin/ksh"  "/tmp/tmp_456676_execute_on_all_hosts.sh"
####  isbdaem0001.isbs.com : Linux :  -rw-r----- 1 root support 71642 Jul 24 13:00 /var/log/messages
 
[24.07.2021 13:00:58] ---- Processing "isbdbfcm12.isbs.com" (ssh user is "admin01", scp user is "admin01")  ... ( 3 from 472)
[24.07.2021 13:00:58] The scp command is:  /usr/bin/timeout --foreground --preserve-status 10   scp  -o StrictHostKeyChecking=no   ./ls_messages admin01@isbdbfcm12a1.isbs.com:/tmp/tmp_456676_execute_on_all_hosts.sh
*** INFO: Hostname "isbdbfcm12a1" replaced with "isbdbfcm12a1.isbs.com"
*** INFO: Hostname "isbdbfcm12a1.isbs.com" removed from the known_hosts file
*** INFO: Adding the host key for isbdbfcm12a1.isbs.com to the known_hosts
# isbdbfcm12a1.isbs.com:22 SSH-2.0-SSH
# isbdbfcm12a1.isbs.com:22 SSH-2.0-SSH
# isbdbfcm12a1.isbs.com:22 SSH-2.0-SSH
*** INFO: Executing now :  /usr/bin/scp -S /tmp/ssh_wrapper_for_scp   "-o"  "StrictHostKeyChecking=no"  "./ls_messages"  ticket_id=9999@admin01@isbdbfcm12a1.isbs.com:'/tmp/tmp_456676_execute_on_all_hosts.sh'
[24.07.2021 13:01:01] The ssh command is:  /usr/bin/timeout --foreground --preserve-status 10   ssh  -o StrictHostKeyChecking=no   admin01@isbdbfcm12a1.isbs.com /usr/bin/ksh /tmp/tmp_456676_execute_on_all_hosts.sh 
*** INFO: Hostname "isbdbfcm12a1" replaced with "isbdbfcm12a1.isbs.com"
*** INFO: Hostname "isbdbfcm12a1.isbs.com" removed from the known_hosts file
*** INFO: Adding the host key for isbdbfcm12a1.isbs.com to the known_hosts
# isbdbfcm12a1.isbs.com:22 SSH-2.0-SSH
# isbdbfcm12a1.isbs.com:22 SSH-2.0-SSH
# isbdbfcm12a1.isbs.com:22 SSH-2.0-SSH
*** INFO: Executing now :  /usr/bin/ssh   "-o"  "StrictHostKeyChecking=no"  ticket_id=9999@admin01@isbdbfcm12a1.isbs.com "/usr/bin/ksh"  "/tmp/tmp_456676_execute_on_all_hosts.sh"
####  isbdbfcm12.isbs.com : Linux :  -rw-r-----. 1 root support 82039 Jul 24 13:01 /var/log/messages
 
...

 
[24.07.2021 13:38:17] ---- Processing "isbrusap12.isbs.com" (ssh user is "admin01", scp user is "admin01")  ... ( 471 from 472)
[24.07.2021 13:38:17] The scp command is:  /usr/bin/timeout --foreground --preserve-status 10   scp  -o StrictHostKeyChecking=no   ./ls_messages admin01@isbrusap12a1.isbs.com:/tmp/tmp_456676_execute_on_all_hosts.sh
*** INFO: Hostname "isbrusap12a1" replaced with "isbrusap12a1.isbs.com"
*** INFO: Hostname "isbrusap12a1.isbs.com" removed from the known_hosts file
*** INFO: Adding the host key for isbrusap12a1.isbs.com to the known_hosts
# isbrusap12a1.isbs.com:22 SSH-2.0-SSH
# isbrusap12a1.isbs.com:22 SSH-2.0-SSH
# isbrusap12a1.isbs.com:22 SSH-2.0-SSH
*** INFO: Executing now :  /usr/bin/scp -S /tmp/ssh_wrapper_for_scp   "-o"  "StrictHostKeyChecking=no"  "./ls_messages"  ticket_id=9999@admin01@isbrusap12a1.isbs.com:'/tmp/tmp_456676_execute_on_all_hosts.sh'
[24.07.2021 13:38:19] The ssh command is:  /usr/bin/timeout --foreground --preserve-status 10   ssh  -o StrictHostKeyChecking=no   admin01@isbrusap12a1.isbs.com /usr/bin/ksh /tmp/tmp_456676_execute_on_all_hosts.sh 
*** INFO: Hostname "isbrusap12a1" replaced with "isbrusap12a1.isbs.com"
*** INFO: Hostname "isbrusap12a1.isbs.com" removed from the known_hosts file
*** INFO: Adding the host key for isbrusap12a1.isbs.com to the known_hosts
# isbrusap12a1.isbs.com:22 SSH-2.0-SSH
# isbrusap12a1.isbs.com:22 SSH-2.0-SSH
# isbrusap12a1.isbs.com:22 SSH-2.0-SSH
*** INFO: Executing now :  /usr/bin/ssh   "-o"  "StrictHostKeyChecking=no"  ticket_id=9999@admin01@isbrusap12a1.isbs.com "/usr/bin/ksh"  "/tmp/tmp_456676_execute_on_all_hosts.sh"
####  isbrusap12 : Linux :  -rw-r-----. 1 root support 75183 Jul 24 13:38 /var/log/messages
 
[24.07.2021 13:38:21] ---- Processing "isbrvic01.isbs.com" (ssh user is "admin01", scp user is "admin01")  ... ( 472 from 472)
[24.07.2021 13:38:21] The scp command is:  /usr/bin/timeout --foreground --preserve-status 10   scp  -o StrictHostKeyChecking=no   ./ls_messages admin01@isbrvic01a1.isbs.com:/tmp/tmp_456676_execute_on_all_hosts.sh
*** INFO: Hostname "isbrvic01a1" replaced with "isbrvic01a1.isbs.com"
*** INFO: Hostname "isbrvic01a1.isbs.com" removed from the known_hosts file
*** INFO: Adding the host key for isbrvic01a1.isbs.com to the known_hosts
# isbrvic01a1.isbs.com:22 SSH-2.0-SSH
# isbrvic01a1.isbs.com:22 SSH-2.0-SSH
# isbrvic01a1.isbs.com:22 SSH-2.0-SSH
*** INFO: Executing now :  /usr/bin/scp -S /tmp/ssh_wrapper_for_scp   "-o"  "StrictHostKeyChecking=no"  "./ls_messages"  ticket_id=9999@admin01@isbrvic01a1.isbs.com:'/tmp/tmp_456676_execute_on_all_hosts.sh'
[24.07.2021 13:38:24] The ssh command is:  /usr/bin/timeout --foreground --preserve-status 10   ssh  -o StrictHostKeyChecking=no   admin01@isbrvic01a1.isbs.com /usr/bin/ksh /tmp/tmp_456676_execute_on_all_hosts.sh 
*** INFO: Hostname "isbrvic01a1" replaced with "isbrvic01a1.isbs.com"
*** INFO: Hostname "isbrvic01a1.isbs.com" removed from the known_hosts file
*** INFO: Adding the host key for isbrvic01a1.isbs.com to the known_hosts
# isbrvic01a1.isbs.com:22 SSH-2.0-SSH
# isbrvic01a1.isbs.com:22 SSH-2.0-SSH
# isbrvic01a1.isbs.com:22 SSH-2.0-SSH
*** INFO: Executing now :  /usr/bin/ssh   "-o"  "StrictHostKeyChecking=no"  ticket_id=9999@admin01@isbrvic01a1.isbs.com "/usr/bin/ksh"  "/tmp/tmp_456676_execute_on_all_hosts.sh"
####  isbrvic01.isbs.com : AIX :  -rw-r-----    1 root     support      341754 Jul 24 13:38 /var/log/messages
 
[24.07.2021 13:38:26] ---- Processing "isbrxbe01.isbs.com" (ssh user is "admin01", scp user is "admin01")  ... ( 473 from 472)
[24.07.2021 13:38:26] The scp command is:  /usr/bin/timeout --foreground --preserve-status 10   scp  -o StrictHostKeyChecking=no   ./ls_messages admin01@isbrxbe01a1.isbs.com:/tmp/tmp_456676_execute_on_all_hosts.sh
*** INFO: Hostname "isbrxbe01a1" replaced with "isbrxbe01a1.isbs.com"
*** INFO: Hostname "isbrxbe01a1.isbs.com" removed from the known_hosts file
*** INFO: Adding the host key for isbrxbe01a1.isbs.com to the known_hosts
# isbrxbe01a1.isbs.com:22 SSH-2.0-SSH
# isbrxbe01a1.isbs.com:22 SSH-2.0-SSH
# isbrxbe01a1.isbs.com:22 SSH-2.0-SSH
*** INFO: Executing now :  /usr/bin/scp -S /tmp/ssh_wrapper_for_scp   "-o"  "StrictHostKeyChecking=no"  "./ls_messages"  ticket_id=9999@admin01@isbrxbe01a1.isbs.com:'/tmp/tmp_456676_execute_on_all_hosts.sh'
ticket_id=9999@admin01@isbrxbe01a1.isbs.com: Permission denied (publickey).
lost connection
[24.07.2021 13:38:28] ERROR: Error copying the script to the host "isbrxbe01.isbs.com" - could not execute the script on that host
[24.07.2021 13:38:28]
[24.07.2021 13:38:28] All done, 473 host(s) processed:
     isbdadapp4.isbs.com isbdaem0001.isbs.com isbdbfcm12.isbs.com ... isbrusap12.isbs.com isbrvic01.isbs.com isbrxbe01.isbs.com
[24.07.2021 13:38:28]
[24.07.2021 13:38:28] The following hosts could NOT be processed due to an error:
    isbdadapp4.isbs.com
    isbdbw01.isbs.com
   
 ...

    isbrucr2002.isbs.com
    isbrxbe01.isbs.com
[24.07.2021 13:38:28]
[24.07.2021 13:38:28]
[24.07.2021 13:38:28] The log file used was "/var/tmp/execute_on_all_hosts.LOG"
[24.07.2021 13:38:28] execute_on_all_hosts.sh v2.3.0 started at Sat Jul 24 13:00:54 CEST 2021 and ended at Sat Jul 24 13:38:28 CEST 2021.
[24.07.2021 13:38:28] The script runtime is (day:hour:minute:seconds) 0:00:37:34 (= 2254 seconds) for 473 hosts
[24.07.2021 13:38:28]   ( -> about 5 second(s) her host)
[24.07.2021 13:38:28] The RC is 0.

real    37m34.849s
user    1m58.061s
sys    1m16.764s
xtrnaw7:~ $

 
Back to top

Other Examples


Note:

Use the parameter -X to get some usage examples


xtrnaw7@t540p /tools/scripts]$ ./execute_on_all_hosts.sh -X
[18.07.2021 15:40:00] execute_on_all_hosts.sh v2.3.0 started at Sun Jul 18 15:40:00 CEST 2021 (The PID of this process is 189152).
[18.07.2021 15:40:00] No config file ("execute_on_all_hosts.conf") found (use -C to create a default config file)
 -----------------------------------------------------------------------------------------------------
 execute_on_all_hosts.sh v2.3.0
 Documentation - Examples
 -----------------------------------------------------------------------------------------------------


 # create the documentation for the script

   ./execute_on_all_hosts.sh -H 2>./execute_on_all_hosts.sh.doc

 # get the verbose usage for the script

   ./execute_on_all_hosts.sh -v -v -h

 # write a config file for the script

   ./execute_on_all_hosts.sh -C


 # use another config file

   CONFIG_FILE=myconfig.conf ./execute_on_all_hosts.sh

 # use no config file at all

   CONFIG_FILE=none ./execute_on_all_hosts.sh


 # write all output (STDOUT and STDERR) to the file
 #   /var/tmp/mylog.txt

    __TEE_OUTPUT_FILE=/var/tmp/mylog.txt ./execute_on_all_hosts.sh -T


 # create a dump of the environment variables in the files
 #   /tmp/execute_on_all_hosts.sh.envvars.$$
 #   /tmp/execute_on_all_hosts.sh.exported_envvars.$$
 # ($$ is the PID of the process running the script)

    __CREATE_DUMP=1 ./execute_on_all_hosts.sh

 # create a dump of the environment variables in the files
 #   /var/tmp/debug/execute_on_all_hosts.sh.envvars.$$
 #   /var/tmp/debug/execute_on_all_hosts.sh.exported_envvars.\##EXAMPLE## # ($$ is the PID of the process running the script)
 # ($$ is the PID of the process running the script)

    __CREATE_DUMP=/var/tmp/debug ./execute_on_all_hosts.sh

 # use logger instead of echo to print the messages

    LOGMSG_FUNCTION="logger -s -p user.info execute_on_all_hosts.sh"  ./execute_on_all_hosts.sh

 # execute the script "./ls_tmp" as user "support" on the list of hostnames from the file "machine_list"
 # in sequential mode using a timeout value for the ssh and scp commands from 10 seconds
 
 ./execute_on_all_hosts.sh -D print_cmd  -W 10 -i machine_list  -K -s ./ls_tmp

 
 # hint: add the parameter "-d" to the examples below to execute them in parallel sessions
 #

 # execute the command "uname -a" as user "support" on the list of hostnames from the file "hostlist"
 #
 ./execute_on_all_hosts.sh  -i hostlist -s "uname -a" -B


 # execute the script "testscript" as user "support" on the list of hostnames from the file "hostlist"
 # and for the host myhost.mydomain.de but not for the host host1.mydomain.de
 # Use the interface "a1" to access the hosts
 #
 ./execute_on_all_hosts.sh  -i hostlist -s "testscript"  -K -D if=a1 -x host1.mydomain.de -A myhost.mydomain.de


 # execute the command "uname -a" as "root" on the list of hostnames from the file "hostlist",
 # enable ForwardAgent for ssh and scp, cleanup and restore the known_hosts, use the user "root" for the ssh access
 #
 ./execute_on_all_hosts.sh  -i hostlist -s "uname -a" -B  +R -D enable_gh -u ssh:root


 # execute the script "testscript" on the list of hostnames from the file "hostlist_fms" as user "root"
 # via the user "ghuser" on the jump server "goldenhost" and the executable "sls" on the golden host:
 #
 ./execute_on_all_hosts.sh  -i hostlist_fms -s testscript -u ssh:root -t ssh:"%b %k %o -t -t -A -l ghuser goldenhost sls -c %u@%H '%s' "

 #
[18.07.2021 15:40:00] The log file used was "/tmp/execute_on_all_hosts.sh.189152.TEMP" 
[18.07.2021 15:40:00] execute_on_all_hosts.sh v2.3.0 started at Sun Jul 18 15:40:00 CEST 2021 and ended at Sun Jul 18 15:40:00 CEST 2021.
[18.07.2021 15:40:00] The script runtime is (day:hour:minute:seconds) 0:00:00:00 (= 0 seconds) for  hosts
[18.07.2021 15:40:00] The RC is 0.
[xtrnaw7@t540p /tools/scripts]$


Templates


execute_on_all_hosts.sh uses templates for the scp and ssh commands to use. The default templates are:

    ssh template:  %b %o %k -l %u %i %c "%s"

    scp template:  %b %o %k %S %u@%i:%s

These templates evaluate to "standard" scp and ssh commands, e.g:

    ssh -o StrictHostKeyChecking=no -l zzz t61p.akabank.de /usr/bin/ksh /tmp/tmp_28922_execute_on_all_hosts.sh

    scp -o StrictHostKeyChecking=no -i scp_key monitor_netstat.sh yyy@t61p.akabank.de:/tmp/tmp_28922_execute_on_all_hosts.sh


For access via an Golden Host you can define custom templates, e.g. the ssh template

    %b %k %o -t -A -l user4root sls.isbs.de access_host -c %u@%H \"%s\"

evaluates to

    ssh -i ~/.ssh/my_ssh_key -o StrictHostKeyChecking=no -t -A -l user4root sls.isbs.de access_host -c myuser@develop "/tmp/tmp_29185_execute_on_all_hosts.sh"

The known placeholder for the scp and ssh templates are:

Placeholder in the ssh and scp templates are:

Placeholder  usage                              used for scp?    used for ssh?     parameter
-------------------------------------------------------------------------------------------------
%%           %                                  yes              yes               n/a
%b           scp binary                         yes              no                -D scp_binary
%b           ssh binary                         no               yes               -D ssh_binary

%S           source script or command           yes              yes               -s
%s           script/binary on the target host   yes              yes               n/a
%c           shell to use                       no               yes               -c

%u           scp user                           yes              no                -u scp:
%u           ssh user                           no               yes               -u ssh:

%h           target host (FQN)                  yes              yes               n/a
%H           target host (short hostname)       yes              yes               n/a
%d           DNS domain                         yes              yes               n/a
%i           target host interface              yes              yes               n/a

%o           scp options                        yes              no                -p
%o           ssh options                        no               yes               -P

%k           ssh key for ssh                    no               yes               -b ssh:
%k           ssh key for scp                    yes              no                -b scp:

Note:

Do not use the character ^A  or ^X in any of the parameter!


Back to top

Hints & Troubleshooting

This section contains some hints and tips using the script.

Delete the temporary file on the hosts

execute_on_all_hosts.sh does not delete the temporary file created on the hosts . To delete the script after execution add the command

rm $0

to your script

Back to top

Internas


Note: See also the documentation for the script template used for this script: scriptt.sh

The script execute_on_all_hosts.sh copies the script to the each host via scp and executes the script on a the host via ssh (Note: If the parameter "-B" is used nothing is copied to the hosts)

The script uses the scp and ssh binaries found in the PATH. You should start and configure the ssh-agent (or a replacement) before using this script.

To change the ssh and scp binaries used by execute_on_all_host.sh, either change the PATH variable before calling execute_on_all_hosts; create a config using execute_on_all_host.sh -C and change the variables DEFAULT_SSH_BINARY and/or  DEFAULT_SCP_BINARY in the config file, or use the parameter -D ssh_binary=x and/or -D scp_binary=x.

If running under Cygwin the script execute_on_all_hosts.sh converts the script to execute with dos2unix before copying it to the target hosts. The script searches dos2unix in the PATH. To change the binary used for dos2unix use one of the methods described above for ssh/scp; the parameter to change the binary is -D dos2unix_binary=x, the variable is DEFAULT_DOS2UNIX_BINARY.


Back to top

Known debug switches

execute_on_all_hosts.sh supports various so called debug switches (-D <debugswitch>) for customizing the script behavior and for trouble shooting.

The known debug switches in version 2.3.1 are:

xtrnaw7@t15g /data/develop/scripts]$ ./execute_on_all_hosts.sh -D help
[09.09.2022 09:59:56] execute_on_all_hosts.sh v2.3.1 started at Fri Sep  9 09:59:56 CEST 2022 (The PID of this process is 4027368).
[09.09.2022 09:59:56] Reading the config file "/data/develop/scripts/execute_on_all_hosts.conf" ...
Note:

Be aware that the order of some of the -D parameter is important (see below)!

Known debug switches (for -D / --debug):

  help          -- show this usage and exit
  create_documentation
                -- create the script documentation
  printargs     -- print the script arguments
  list_rc       -- list return codes used by this script and exit
  listfunc      -- list all functions defined and exit
  msg           -- log debug messages to the file /tmp/execute_on_all_hosts.sh.4027368.debug
  trace         -- activate tracing to the file /tmp/execute_on_all_hosts.sh.4027368.trace
                   Note: set the variable TRACE_MAIN to 0 to enable tracing the main function before starting the script
                         set the variable TRACE_PROMPT to the prompt to use for tracing before starting the script
  fn_to_stderr  -- print the function names to STDERR
  fn_to_tty     -- print the function names to /dev/tty
  fn_to_handle9 -- print the function names to the file handle 9
  fn_to_device=filename
                -- print the function names to the file "filename"
  debugcode="x" -- execute the debug code "x" at every function start
  tracefunc=f1[,...,f#]
                -- enable tracing for the functions f1 to f#
                   Note: Use either debugcode=x or tracefunc=f1 - but NOT both
  debug         -- start debug env
  setvar:name=value
                -- set the variable "name" to "value"
  create_dump=d -- enable environment dumps; target directory is d
  SyntaxHelp    -- print syntax usage examples for the functions in the template
                   and exit
  dryrun        -- dry run only, do not execute commands
                   default prefix for dryrun is: "/usr/bin/echo "
  dryrun=prefix -- dry run only, add the prefix "prefix" to all commands

  fieldsep=x    -- change the default field separator to x
  ignore_user_in_file
               -- ignore user names from the host lists
  do_not_sort_hostlist
               -- do not sort the list of hosts
  ssh_binary=x
               -- use the ssh binary x
  scp_binary=x
               -- use the scp binary x
  ssh-keygen_binary=x
               -- use the ssh-keygen binary x
  dos2unix_binary=x
               -- use the dos2unix binary x
  if=x         -- use the host interface x, e.g.
                  if=a1; hostname=myhost.mydom.net -> interface to use is myhosta1.mydom.net
  enable_ForwardAgent_for_scp
               -- enable "ForwardAgent yes" for scp
  disable_ForwardAgent_for_scp
               -- disable "ForwardAgent yes" for scp
  singlestep   -- execute the commands in single step mode if running in sequential mode
  print_cmd    -- write scp and ssh commands to STDOUT
  clean_known_hosts
               -- remove each host from the known_hosts file before doing the scp and ssh
  clean_and_restore_known_hosts
               -- remove each host from the known_hosts file before doing the scp and ssh
                  and restore the known_hosts file at script end
  delete_known_hosts
               -- remove all hosts from the known_hosts file before doing the scp and ssh
  delete_and_restore_known_hosts | ignore_known_hosts
               -- remove all hosts from the known_hosts file before doing the scp and ssh
                  and restore the known_hosts file at script end
  restore_known_hosts
               -- restore the known_hosts file before the script ends
  nameserver   -- define the name server to use
              
  do_not_log_ssh_cmds
               -- do not log the output of the ssh commands in the log file of the script
  log_ssh_cmds
               -- log the output of the ssh commands in the log file of the script

  rcm_server=x    -- RCM server to use
  rcm_user=x      -- RCM user
  rcm_password=x  -- RCM password

    ---- Macros ----

  enable_gh    -- enable ForwardAgent for ssh and scp,
                  cleanup and restore the known_hosts

  ticket_id=x  -- set the ticket id for the SLS access to x; use
                  "ticket_id=none" for no ticket id; use "ticket_id=default"
                  to use the default ticket number which is 9999.
                  Current value is ""
                 
  use_ssh_wrapper
               -- use my ssh and scp wrapper script
                  Current value is y

  Hints for the sls* Parameter listed below:
 
  To use another ticket id with the sls* parameter add the parameter "-D ticket_id=n" BEFORE the
  sls* Parameter!

  sls_db_unxxx4
               -- use DB SLS for ssh and scp access using the user unxxx4
                  enable ForwardAgent for ssh and scp,
                  cleanup and restore the known_hosts,
                  enable RCM access,
                  and set the ssh user and scp user to unxxx4

  sls_db_unxxx4_sudo
               -- use DB SLS for ssh and scp access using the user unxxx4
                  use sudo to execute the command
                  enable ForwardAgent for ssh and scp,
                  cleanup and restore the known_hosts,
                  enable RCM access,
                  and set the ssh user and scp user to unxxx4

  SLS_db_unxxx4
               -- use DB SLS for ssh and scp access using the user unxxx4
                  enable ForwardAgent for ssh and scp,
                  use the option "StrictHostKeyChecking=no" for ssh and scp,
                  temporary remove all entries from the known_hosts,
                  enable RCM access,
                  and set the ssh user and scp user to unxxx4

  SLS_db_unxxx4_sudo
               -- use DB SLS for ssh and scp access using the user unxxx4
                  use sudo to execute the command
                  enable ForwardAgent for ssh and scp,
                  use the option "StrictHostKeyChecking=no" for ssh and scp,
                  temporary remove all entries from the known_hosts,
                  enable RCM access,
                  and set the ssh user and scp user to unxxx4

  sls_db_unxxx4_w_timeout or sls
               -- use DB SLS for ssh and scp access using the user unxxx4
                  enable ForwardAgent for ssh and scp,
                  cleanup and restore the known_hosts,
                  enable RCM access,
                  and set the ssh user and scp user to unxxx4
                  use a timeout of 15 seconds for each ssh and sls command
                  (set the variable SSH_SCP_CMD_TIMEOUT to change the timeout
                   value)
                 
  sls_scp_db_unxxx4
               -- use DB SLS for scp access using the user unxxx4,
                  enable ForwardAgent for ssh and scp,
                  cleanup and restore the known_hosts,
                  enable RCM access,
                  and set the scp user to unxxx4

  sls_ssh_db_unxxx4
               -- use DB SLS for ssh access using the user unxxx4
                  enable ForwardAgent for ssh and scp,
                  cleanup and restore the known_hosts,
                  enable RCM access,
                  and set the ssh user to unxxx4
 
  sls_db       -- use DB SLS for ssh and scp access,
                  enable ForwardAgent for ssh and scp,
                  cleanup and restore the known_hosts,
                  enable RCM access,
                  and set the ssh user and scp user to root

  sls_scp_db   -- use DB SLS for scp access,
                  enable ForwardAgent for ssh and scp,
                  cleanup and restore the known_hosts,
                  enable RCM access,
                  and set the scp user to root

  sls_ssh_db   -- use DB SLS for ssh access,
                  enable ForwardAgent for ssh and scp,
                  cleanup and restore the known_hosts,
                  enable RCM access,
                  and set the ssh user to root

  sls_fms      -- use FMS SLS for ssh and scp access,
                  enable ForwardAgent for ssh and scp,
                  enable RCM access,
                  set the interface to use to a1,
                  and set the ssh and the scp user to root

  sls_ssh_fms  -- use FMS SLS for ssh access,
                  enable ForwardAgent for ssh and scp,
                  enable RCM access,
                  set the interface to use to a1,
                  and set the ssh to root and the scp user to support
[09.09.2022 09:59:56] The log file used was "/tmp/execute_on_all_hosts.sh.4027368.TEMP" 
[09.09.2022 09:59:56] execute_on_all_hosts.sh v2.3.1 started at Fri Sep  9 09:59:56 CEST 2022 and ended at Fri Sep  9 09:59:56 CEST 2022.
[09.09.2022 09:59:56] The script runtime is (day:hour:minute:seconds) 0:00:00:00 (= 0 seconds) for  hosts
[09.09.2022 09:59:56] The RC is 0.
[xtrnaw7@t15g /data/develop/scripts]$




Back to top

Notes

Please see the source code of the script for additional information


Back to top

Download


Download execute_on_all_hosts.sh

Back to top