execo

Handles launch and control of several OS level processes concurrently.

Handles remote executions and file copies with ssh or similar tools.

Overview

This module offers a high level API for parallel local or remote processes execution with the execo.action.Action class hierarchy, and a lower level API with the execo.process.Process class hierarchy, for handling individual local or remote subprocesses.

Process class hierarchy

A execo.process.Process and its subclass execo.process.SshProcess are abstractions of local or remote operating system process, similar to the subprocess.Popen class in the python standard library (actually execo execo.process.Process use them internally). The main differences is that a background thread (which is started when execo module is imported) takes care of reading asynchronously stdout and stderr, and of handling the lifecycle of the process. This allows writing easily code in a style appropriate for conducting several processes in parallel.

You can automatically instanciate the right class execo.process.Process or execo.process.SshProcess with the “factory” function execo.process.get_process.

execo.process.ProcessBase and execo.process.TaktukProcess are used internally and should probably not be used, unless for developing code similar to execo.action.TaktukRemote.

get_process

execo.process.get_process(*args, **kwargs)

Instanciates a execo.process.Process or execo.process.SshProcess, depending on the existence of host keyword argument

ProcessBase

Inheritance diagram of execo.process.ProcessBase
class execo.process.ProcessBase(cmd, timeout=None, ignore_exit_code=False, nolog_exit_code=False, ignore_timeout=False, nolog_timeout=False, ignore_error=False, nolog_error=False, ignore_expect_fail=False, nolog_expect_fail=False, ignore_write_error=False, nolog_write_error=False, default_expect_timeout=None, default_stdout_handler=True, default_stderr_handler=True, lifecycle_handlers=None, stdout_handlers=None, stderr_handlers=None, name=None)

Bases: object

An almost abstract base class for all kinds of processes.

There are no abstract methods, but a execo.process.ProcessBase by itself is almost useless, it only provides accessors to data shared with all subclasses, but no way to start it or stop it. These methods have to be implemented in concrete subclasses.

It is possible to register custom lifecycle and output handlers to the process, in order to provide specific actions or stdout/stderr parsing when needed. See execo.process.ProcessLifecycleHandler and execo.process.ProcessOutputHandler.

A ProcessBase and its subclasses can act as a context manager object, allowing to write code such as:

with Process(...).start() as p:
  [...do something...]

When exiting the contex manager scope, the process is automatically killed.

Parameters:
  • cmd – string or tuple containing the command and args to run.
  • timeout – Timeout (in seconds, or None for no timeout) after which the process will automatically be sent a SIGTERM.
  • ignore_exit_code – Boolean. If True, a process with a return code != 0 will still be considered ok.
  • nolog_exit_code – Boolean. If False, termination of a process with a return code != 0 will cause a warning in logs.
  • ignore_timeout – Boolean. If True, a process which reaches its timeout will be sent a SIGTERM, but will still be considered ok (but it will most probably have an exit code != 0).
  • nolog_timeout – Boolean. If False, a process which reaches its timeout and is sent a SIGTERM will cause a warning in logs.
  • ignore_error – Boolean. If True, a process raising an OS level error will still be considered ok.
  • nolog_error – Boolean. If False, a process raising an OS level error will cause a warning in logs.
  • ignore_expect_fail – Boolean. If True, on a failure to find an expect match (reach expect timeout of eof or stream error before finding a match) the process will still be considered ok.
  • nolog_expect_fail – Boolean. If False, a failure to find an expect match (reach expect timeout of eof or stream error before finding a match) will cause a warning in logs.
  • ignore_write_error – Boolean. If True, on a write error to the process stdin, the process will still be considered ok.
  • nolog_write_error – Boolean. If False, on a write error to the process stdin, will cause a warning in the logs.
  • default_expect_timeout – The default timeout for expect invocations when no explicit timeout is given. Defaults to None, meaning no timeout.
  • default_stdout_handler – If True, a default handler sends stdout stream output to the member string self.stdout.
  • default_stderr_handler – if True, a default handler sends stderr stream output to the member string self.stderr.
  • lifecycle_handlers – List of instances of execo.process.ProcessLifecycleHandler for being notified of process lifecycle events.
  • stdout_handlers – List which can contain instances of execo.process.ProcessOutputHandler for handling activity on process stdout, or existing file descriptors (positive integer), or existing file objects, or filenames, to which stdout will be sent. If a filename is given, it will be opened in write mode, and closed on eof.
  • stderr_handlers – List which can contain instances of execo.process.ProcessOutputHandler for handling activity on process stderr, or existing file descriptors (positive integer), or existing file objects, or filenames, to which stderr will be sent. If a filename is given, it will be opened in write mode, and closed on eof.
  • name – User-friendly name. A default is generated and can be changed.
cmd = None

Process command line: string or tuple containing the command and args to run.

default_expect_timeout = None

The default timeout for expect invocations when no explicit timeout is given. Defaults to None, meaning no timeout.

default_stderr_handler = None

if True, a default handler sends stderr stream output to the member string self.stderr

default_stdout_handler = None

if True, a default handler sends stdout stream output to the member string self.stdout

end_date = None

Process end date or None if not yet ended

ended = None

Whether the process has ended or not

ended_condition = None

Condition notified when the process has actually ended

ended_event = None

Event raised when the process has actually ended

error = None

Whether there was an error starting the process. This is not the process return code.

error_reason = None

Operating system level errno, if there was an error starting the process, or None

exit_code = None

Process exit code if available (if the process ended correctly from the operating system point of view), or None

expect(regexes, timeout=False, stream=1, backtrack=True, expect_output_handler=None)

searches the process output stream(s) for some regex. It mimics/takes ideas from Don Libes expect, or python-pexpect.

It is a more user-friendly interface than execo.process.ExpectOutputHandler (in particular, it manages backtracking).

It waits for a regex to match, then returns the tuple (regex_index, match_object), or (None, None) if timeout reached, or if eof reached, or stream in error, without any match.

It uses thread local storage such that concurrent expects in parallel threads do not interfere which each other.

Parameters:
  • regexes – a regex or list of regexes. May be given as string or as compiled regexes (If given as compiled regexes, do not forget flags, most likely re.MULTILINE. regex passed as string are compiled with re.MULTILINE)
  • timeout – wait timeout after which it returns (None, None) if no match was found. If False (the default): use the default expect timeout. If None: no timeout. If 0: return immediately, no blocking if no match found.
  • stream – stream to monitor for this process, STDOUT or STDERR.
  • backtrack – If True (the default), the first expect is done from the beginning of the process start. If False, the first expect is done from the next received process output.
  • expect_output_handler – If not None, a specific ExpectOutputHandler instance to use. Otherwise, if None (the default), a thread local ExpectOutputHandler is instanciated.
expect_fail = None

Whether an expect invocation failed to match (reach expect timeout, or stream eof or error before finding any match).

finished_ok

Check process has ran and is ok.

A process is finished_ok if it has started and ended and it is ok.

forced_kill = None

Whether the process was killed forcibly. When a process is killed with SIGTERM (either manually or automatically, due to reaching a timeout), execo will wait some time (constant set in execo source) and if after this timeout the process is still running, it will be killed forcibly with a SIGKILL.

host = None

Remote host, if relevant

ignore_error = None

Boolean. If True, a process raising an OS level error will still be considered ok

ignore_exit_code = None

Boolean. If True, a process with a return code != 0 will still be considered ok

ignore_expect_fail = None

Boolean. If True, on a failure to find an expect match (reach expect timeout of eof or stream error before finding a match) the process will still be considered ok.

ignore_timeout = None

Boolean. If True, a process which reaches its timeout will be sent a SIGTERM, but will still be considered ok (but it will most probably have an exit code != 0)

ignore_write_error = None

Boolean. If True, on a write error to the process stdin, the process will still be considered ok.

killed = None

Whether the process was killed.

lifecycle_handlers = None

List of instances of execo.process.ProcessLifecycleHandler for being notified of process lifecycle events.

name = None

User-friendly name. A default is generated and can be changed.

nolog_error = None

Boolean. If False, a process raising an OS level error will cause a warning in logs

nolog_exit_code = None

Boolean. If False, termination of a process with a return code != 0 will cause a warning in logs

nolog_expect_fail = None

Boolean. If False, a failure to find an expect match (reach expect timeout of eof or stream error before finding a match) will cause a warning in logs.

nolog_timeout = None

Boolean. If False, a process which reaches its timeout and is sent a SIGTERM will cause a warning in logs

nolog_write_error = None

Boolean. If False, on a write error to the process stdin, will cause a warning in the logs.

ok

Check process is ok.

A process is ok, if:

  • did not failed on an expect invocation (or was instructed to ignore it)
  • has no write error (or was instructed to ignore it)
  • it is not yet started or not yet ended
  • it started and ended and:
    • has no error (or was instructed to ignore them)
    • did not timeout (or was instructed to ignore it)
    • returned 0 (or was instructed to ignore a non zero exit code)
reset()

Reinitialize a process so that it can later be restarted.

If it is running, this method will first kill it then wait for its termination before reseting;

running

If the process is currently running.

start_date = None

Process start date or None if not yet started

started = None

Whether the process was started or not

started_condition = None

Condition notified when the process is actually started

started_event = None

Event raised when the process is actually started

stats()

Return a dict summarizing the statistics of this process.

see execo.report.Report.stats.

stderr = None

Process stderr

stderr_handlers = None

List which can contain instances of execo.process.ProcessOutputHandler for handling activity on process stderr, or existing file descriptors (positive integer), or existing file objects, or filenames, to which stderr will be sent. If a filename is given, it will be opened in write mode, and closed on eof.

stdout = None

Process stdout

stdout_handlers = None

List which can contain instances of execo.process.ProcessOutputHandler for handling activity on process stdout, or existing file descriptors (positive integer), or existing file objects, or filenames, to which stdout will be sent. If a filename is given, it will be opened in write mode, and closed on eof.

timeout = None

timeout (in seconds, or None for no timeout) after which the process will automatically be sent a SIGTERM

timeout_date = None

The date at wich this process will timeout or none if not available.

timeouted = None

Whether the process has reached its timeout, or None if we don’t know yet (process still running, timeout not reached).

write_error = None

Whether there was a write error to the process stdin.

Process

Inheritance diagram of execo.process.Process
class execo.process.Process(cmd, shell=False, pty=False, kill_subprocesses=None, **kwargs)

Bases: execo.process.ProcessBase

Handle an operating system process.

In coordination with the internal _Conductor I/O and process lifecycle management thread which is started when execo is imported, this class allows creation, start, interruption (kill), and waiting (for the termination) of an operating system process. The subprocess output streams (stdout, stderr), as well as various informations about the subprocess and its state can be accessed asynchronously.

Example usage of the process class: run an iperf server, and connect to it with an iperf client:

>>> server = Process('iperf -s')
>>> server.ignore_exit_code = server.nolog_exit_code = True
>>> server.start()
Process('iperf -s')
>>> client = Process('iperf -c localhost -t 2').start()
>>> client.started
True
>>> client.ended
False
>>> client.wait()
Process('iperf -c localhost -t 2')
>>> client.ended
True
>>> server.ended
False
>>> server.kill()
Process('iperf -s')
>>> server.wait()
Process('iperf -s')
>>> server.ended
True
Parameters:
  • cmd – string or tuple containing the command and args to run.
  • shell – Whether or not to use a shell to run the cmd. See subprocess.Popen.
  • pty – If True, open a pseudo tty and connect process’s stdin and stdout to it (stderr is still connected as a pipe). Make process a session leader. If lacking permissions to send signals to the process, try to simulate sending control characters to its pty.
  • kill_subprocesses – If True, signals are also sent to subprocesses. If None, automatically decide based on shell = True/False.
kill(sig=<Signals.SIGTERM: 15>, auto_force_kill_timeout=True)

Send a signal (default: SIGTERM) to the subprocess.

Parameters:
  • sig – the signal to send
  • auto_force_kill_timeout – Sets whether or not execo will check that the subprocess has terminated after beeing sent SIGTERM/SIGINT, and automatically send SIGKILL if the subprocess is not yet terminated. If None/False, will not auto-send SIGKILL. If True, will auto-send SIGKILL after the default timeout from execo.configuration['kill_timeout']. Otherwise, the given value is used as the timeout.

Sending signals to processes automatically ignores and disable logs of exit code != 0.

kill_subprocesses = None

If True, signals are also sent to subprocesses. If None, automatically decide based on shell = True/False.

pid = None

Subprocess’s pid, if available (subprocess started) or None

pty = None

If True, open a pseudo tty and connect process’s stdin and stdout to it (stderr is still connected as a pipe). Make process a session leader. If lacking permissions to send signals to the process, try to simulate sending control characters to its pty.

run(timeout=None)

Start subprocess then wait for its end.

shell = None

Whether or not to use a shell to run the cmd. See subprocess.Popen

start()

Start the subprocess.

stderr_fd = None

the subprocess stderr filehandle or None if not available.

stdin_fd = None

the subprocess stdin filehandle or None if not available.

stdout_fd = None

the subprocess stdout filehandle or None if not available.

wait(timeout=None)

Wait for the subprocess end.

write(s)

Write on the Process standard input

Allows process instances to behave as file-like objects. You can for example print to a Process.

This method automatically waits for the standard input to be actually writable if the process was just started (as the start is asynchronous).

SshProcess

Inheritance diagram of execo.process.SshProcess
class execo.process.SshProcess(cmd, host, connection_params=None, **kwargs)

Bases: execo.process.Process

Handle a remote command execution through ssh or similar remote execution tool.

Note: the closing of the remote process upon killing of the SshProcess depends on the ssh (or ssh-like) command behavior. With openssh, this can be obtained by passing options -tt (force tty creation), thus these are the default options in execo.config.default_connection_params.

Parameters:
  • cmd – string containing the command and args to run.
  • host – execo.host.Host to connect to
  • connection_params – connection parameters
connection_params = None

Remote connection params

remote_cmd = None

The command executed remotely

TaktukProcess

Inheritance diagram of execo.process.TaktukProcess
class execo.process.TaktukProcess(cmd, host, **kwargs)

Bases: execo.process.ProcessBase

Dummy process similar to execo.process.SshProcess.

start()

Notify TaktukProcess of actual remote process start.

This method is intended to be used by execo.action.TaktukRemote.

Serial

Inheritance diagram of execo.process.Serial
class execo.process.Serial(device, speed, **kwargs)

Bases: execo.process.Process

Create a connection to a serial port.

Parameters:
  • device – the serial device
  • speed – serial port speed

SerialSsh

Inheritance diagram of execo.process.SerialSsh
class execo.process.SerialSsh(host, device, speed, connection_params=None, **kwargs)

Bases: execo.process.SshProcess

Create a connection to a serial port through ssh.

Parameters:
  • host – remote side of the ssh connection
  • device – the serial device on the remote host
  • speed – serial port speed
  • connection_params – connection params for connecting to host

PortForwarder

Inheritance diagram of execo.process.PortForwarder
class execo.process.PortForwarder(host, remote_host, remote_port, local_port=None, bind_address=None, connection_params=None, **kwargs)

Bases: execo.process.SshProcess

Create an ssh port forwarder process (ssh -L).

This port forwarding process opens a listening socket on localhost, and forwards it to the given remote host / port on the remote side of the ssh connection.

Parameters:
  • host – remote side of the ssh connection
  • remote_host – the remote host to connect to on the remote side
  • remote_port – the remote port to connect to on the remote side
  • local_port – the port to use locally for the listening socket. If None (the default), a port is automatically selected with execo.utils.get_port
  • bind_address – the bind address to use locally for the listening socket. If None (the default), it uses 127.0.0.1, so the socket is only available to localhost.
  • connection_params – connection params for connecting to host
forwarding = None

threading.Event which can be waited upon to be notified when the forwarding port is actually open

local_port = None

The local port of the tunnel

ProcessLifecycleHandler

class execo.process.ProcessLifecycleHandler

Bases: object

Abstract handler for execo.process.ProcessBase lifecycle.

end(process)

Handle execo.process.ProcessBase’s end.

Parameters:process – The ProcessBase which ends.
reset(process)

Handle execo.process.ProcessBase’s reset.

Parameters:process – The ProcessBase which is reset.
start(process)

Handle execo.process.ProcessBase’s start.

Parameters:process – The ProcessBase which starts.

ProcessOutputHandler

class execo.process.ProcessOutputHandler

Bases: object

Abstract handler for execo.process.ProcessBase output.

ProcessOutputHandler constructor. Call it in inherited classes.

read(process, stream, string, eof, error)

Handle string read from a execo.process.ProcessBase’s stream.

Parameters:
  • process – the ProcessBase which outputs the string
  • stream – the index of the process’ stream which gets data (STDOUT / STDERR)
  • string – the string read

:param eof:(boolean) true if the stream is now at eof.

Parameters:error – (boolean) true if there was an error on the stream
read_line(process, stream, string)

Handle string read line by line from a execo.process.ProcessBase’s stream.

Parameters:
  • process – the ProcessBase which outputs the line
  • stream – the index of the process’ stream which gets data (STDOUT / STDERR)
  • string – the line read

ExpectOutputHandler

class execo.process.ExpectOutputHandler

Bases: execo.process.ProcessOutputHandler

Handler for monitoring stdout / stderr of a Process and being notified when some regex matches. It mimics/takes ideas from Don Libes expect, or python-pexpect.

To use this execo.process.ProcessOutputHandler, instanciate one, add it to the monitored process(es) stdout / stderr handlers. As soon as added, it records the stream output. Then call the expect method to expect some regexes in the stream.

One instance of ExpectOutputHandler can handle stdout/stderr of several processes. It tracks each process’s stream search position independently. For subsequent matches, the search start position in the stream is the end of the previous match.

expect(regexes, timeout=None)

Expect a regex or a list of regexes.

Parameters:
  • regexes – a regex or list of regexes. May be given as string or as compiled regexes (If given as compiled regexes, do not forget flags, most likely re.MULTILINE. regex passed as string are compiled with re.MULTILINE)
  • timeout – wait timeout after which it returns {} if no match was found. If None (the default): no timeout. If 0: return immediately, no blocking if no match found.

Returns a dict whose keys are tuples (process, stream) and whose values are tuples (regex_index, match_object). The processes in the keys are the processes where a match was found. The stream in the keys are the streams (STDIN, STDOUT) where the match was found. The regex_index in the values is the index of the regex which matched, when given several regexes to search, or always 0 if only one regex was search. The match_object in the values is the regex match object. The (process, stream) tuples that are monitored are the ones to which this ExpectOutputHandler has been added as ProcessOutputHandler. For each (process, stream) monitored, only the first match is returned by a call to expect. To get more matches, call expect again. When timeout == None or there is a timeout, if no match found, will block until at least one match is found for a (process, stream), or until reaching the timeout (in this case, it will return an empty dict). If timeout == 0 and no match were immediately found, will return immediately with an empty dict.

read(process, stream, string, eof, error)

Handle string read from a execo.process.ProcessBase’s stream.

Parameters:
  • process – the ProcessBase which outputs the string
  • stream – the index of the process’ stream which gets data (STDOUT / STDERR)
  • string – the string read

:param eof:(boolean) true if the stream is now at eof.

Parameters:error – (boolean) true if there was an error on the stream

Action class hierarchy

An execo.action.Action is an abstraction of a set of parallel processes. It is an abstract class. Child classes are: execo.action.Remote, execo.action.TaktukRemote, execo.action.Get, execo.action.Put, execo.action.TaktukGet, execo.action.TaktukPut, execo.action.Local. A execo.action.Remote or execo.action.TaktukRemote is a remote process execution on a group of hosts. The remote connection is performed by ssh or a similar tool. execo.action.Remote uses as many ssh connections as remote hosts, whereas execo.action.TaktukRemote uses taktuk internally (http://taktuk.gforge.inria.fr/) to build a communication tree, thus is more scalable. Put and Get are actions for copying files or directories to or from groups of hosts. The copy is performed with scp or a similar tool. execo.action.TaktukPut and execo.action.TaktukGet also copy files or directories to or from groups of hosts, using taktuk (see taktuk documentation about limitations of using taktuk for file transfers). A execo.action.Local is a local process (it is a very lightweight execo.action.Action on top of a single execo.Process.Process instance).

execo.action.Remote, execo.action.TaktukRemote, execo.action.Get, execo.action.TaktukGet, execo.action.Put, execo.action.TaktukPut require a list of remote hosts to perform their tasks. These hosts are passed as an iterable of instances of execo.host.Host. The execo.host.Host class instances have an address, and may have a ssh port, a ssh keyfile, a ssh user, if needed.

A configurable execo.action.ActionFactory can be created to choose which kind of actions to instanciate, ssh/scp or taktuk. Functions execo.action.get_remote, execo.action.get_fileput, execo.action.get_fileget use the default execo.action.ActionFactory: execo.action.default_action_factory.

As an example of the usage of the execo.action.Remote class, let’s launch some commands on a few remote hosts:

a = Remote(cmd='whoami ; uname -a',
           hosts=(Host('nancy'), Host('Rennes'))).start()
b = Remote(cmd='cd ~/project ; make clean ; make'
           hosts=(Host('lille'), Host('sophia'))).start()
a.wait()
# do something else....
b.wait()

Action

Inheritance diagram of execo.action.Action
class execo.action.Action(lifecycle_handlers=None, name=None, default_expect_timeout=None)

Bases: object

Abstract base class. A set of parallel processes.

An execo.action.Action can be started (execo.action.Action.start), killed (execo.action.Action.kill). One can wait (execo.action.Action.wait) for an Action, it means waiting for all processes in the Action to finish. An Action can be run (execo.action.Action.run), it means start it then wait for it to complete.

An Action and its subclasses can act as a context manager object, allowing to write code such as:

with Remote(...).start() as r:
  [...do something...]

When exiting the contex manager scope, the remote is automatically killed.

Parameters:
  • lifecycle_handlers – List of instances of execo.action.ActionLifecycleHandler for being notified of action lifecycle events.
  • name – User-friendly name. A default is generated and can be changed.
  • default_expect_timeout – The default timeout for expect invocations when no explicit timeout is given. Defaults to None, meaning no timeout.
default_expect_timeout = None

The default timeout for expect invocations when no explicit timeout is given. Defaults to None, meaning no timeout.

ended = None

whether this Action has ended (boolean)

expect(regexes, timeout=False, stream=1, backtrack=True, expect_output_handler=None)

searches the process output stream(s) for some regex. It mimics/takes ideas from Don Libes expect, or python-pexpect, and does so on multiple processes running in parallel.

It waits for a regex to match on all processes. Then it returns a list of tuples (process, regex index, match object). For processes / streams for which there was no match before reaching the timeout or the stream is eof or error, the tuple is (process, None, None). The returned list has the same process sort order than self.processes.

It uses execo.process.ProcessBase.expect, see its documentation for more details.

It waits for a regex to match, then returns the tuple (regex_index, match_object), or (None, None) if timeout reached, or if eof reached, or stream in error, without any match.

It uses thread local storage such that concurrent expects in parallel threads do not interfere which each other.

Parameters:
  • regexes – a regex or list of regexes. May be given as string or as compiled regexes (If given as compiled regexes, do not forget flags, most likely re.MULTILINE. regex passed as string are compiled with re.MULTILINE)
  • timeout – wait timeout after which it returns (None, None) if no match was found. If False (the default): use the default expect timeout. If None: no timeout. If 0: return immediately, no blocking if no match found.
  • stream – stream to monitor for this process, STDOUT or STDERR.
  • backtrack – If True (the default), the first expect is done from the beginning of the process start. If False, the first expect is done from the next received process output.
  • expect_output_handler – If not None, a specific ExpectOutputHandler instance to use. Otherwise, if None (the default), a thread local ExpectOutputHandler is instanciated.
finished_ok

Action has started, ended, and is not in error.

kill()

Kill all processes not yet ended.

Returns immediately, without waiting for processes to be actually killed.

return self

lifecycle_handlers = None

List of instances of execo.action.ActionLifecycleHandler for being notified of action lifecycle events.

name = None

User-friendly name. A default is generated and can be changed.

ok

Returns a boolean indicating if all processes are ok.

refer to execo.process.ProcessBase.ok for detailed semantics of being ok for a process.

processes = None

Iterable of all execo.process.ProcessBase

reset()

Reinitialize an Action so that it can later be restarted.

If it is running, this method will first kill it then wait for its termination before reseting.

run(timeout=None)

Start all processes then wait for them to complete.

return self

start()

Start all processes.

return self

started = None

whether this Action was started (boolean)

stats()

Return a dict summarizing the statistics of all processes of this Action.

see execo.report.Report.stats.

wait(timeout=None)

Wait for all processes to complete.

return self

Remote

Inheritance diagram of execo.action.Remote
class execo.action.Remote(cmd, hosts, connection_params=None, process_args=None, **kwargs)

Bases: execo.action.Action

Launch a command remotely on several host, with ssh or a similar remote connection tool.

One ssh process is launched for each connection.

Parameters:
cmd = None

The command to run remotely. substitions described in execo.substitutions.remote_substitute will be performed.

connection_params = None

A dict similar to execo.config.default_connection_params whose values will override those in default_connection_params for connection.

hosts

Iterable of execo.host.Host to which to connect and run the command.

kill()

Kill all processes not yet ended.

Returns immediately, without waiting for processes to be actually killed.

return self

process_args = None

Dict of keyword arguments passed to instanciated processes.

start()

Start all processes.

return self

write(s)

Write on the Remote processes standard inputs

Allows Remote instances to behave as file-like objects. You can for example print to a Remote.

TaktukRemote

Inheritance diagram of execo.action.TaktukRemote
class execo.action.TaktukRemote(cmd, hosts, connection_params=None, process_args=None, **kwargs)

Bases: execo.action.Action

Launch a command remotely on several host, with taktuk.

One taktuk instance is ran, which itself connects to hosts through an ssh tree.

Behavior should be identical to execo.action.Remote. Current limitation are:

  • we can provide per-host user with taktuk, but we cannot provide per-host port or keyfile, so a check is made that all hosts and connection_params have the same port / keyfile (or None). If not, an exception is raised during initialization.
  • remote processes are not killed when killing the TaktukRemote. See ‘hanged commands’ in http://taktuk.gforge.inria.fr/taktuk.html#bugs. With ssh the workaround is to pass options -tt but passing these options to taktuk connector causes immediate closing of the connector upon connection.
Parameters:
cmd = None

The command to run remotely. substitions described in execo.substitutions.remote_substitute will be performed.

connection_params = None

A dict similar to execo.config.default_connection_params whose values will override those in default_connection_params for connection.

hosts

Iterable of execo.host.Host to which to connect and run the command.

kill()

Kill all processes not yet ended.

Returns immediately, without waiting for processes to be actually killed.

return self

process_args = None

Dict of keyword arguments passed to instanciated processes.

start()

Start all processes.

return self

wait(timeout=None)

Wait for all processes to complete.

return self

RemoteSerial

Inheritance diagram of execo.action.RemoteSerial
class execo.action.RemoteSerial(hosts, device, speed, connection_params=None, process_args=None, **kwargs)

Bases: execo.action.Remote

Open a serial port on several hosts in parallel through ssh or a similar remote connection tool.

The serial port can be read (standard output) and written to (standard input).

Parameters:
  • hosts – iterable of execo.host.Host to which to connect and open the serial device.
  • device – The Path to the serial device on the remote hosts (for example: /dev/ttyUSB1). Substitions described in execo.substitutions.remote_substitute will be performed.
  • speed – the speed of the serial port (for example: 115200)
  • connection_params – a dict similar to execo.config.default_connection_params whose values will override those in default_connection_params for connection.
  • process_args – Dict of keyword arguments passed to instanciated processes.
connection_params = None

A dict similar to execo.config.default_connection_params whose values will override those in default_connection_params for connection.

device = None

Path to the serial devices on the remote hosts. (for example: /dev/ttyUSB1). Substitions described in execo.substitutions.remote_substitute will be performed.

process_args = None

Dict of keyword arguments passed to instanciated processes.

speed = None

The speed of the serial port (for example: 115200)

Put

Inheritance diagram of execo.action.Put
class execo.action.Put(hosts, local_files, remote_location='.', connection_params=None, **kwargs)

Bases: execo.action.Remote

Copy local files to several remote host, with scp or a similar connection tool.

Parameters:
connection_params = None

A dict similar to execo.config.default_connection_params whose values will override those in default_connection_params for connection.

local_files = None

An iterable of string of file paths. substitions described in execo.substitutions.remote_substitute will be performed.

remote_location = None

The directory on the remote hosts were the files will be copied. substitions described in execo.substitutions.remote_substitute will be performed.

TaktukPut

Inheritance diagram of execo.action.TaktukPut
class execo.action.TaktukPut(hosts, local_files, remote_location='.', connection_params=None, **kwargs)

Bases: execo.action.TaktukRemote

Copy local files to several remote host, with taktuk.

Parameters:
connection_params = None

Dict similar to execo.config.default_connection_params whose values will override those in default_connection_params for connection.

local_files = None

Iterable of string of file paths. substitions described in execo.substitutions.remote_substitute will not be performed, but taktuk substitutions can be used (see http://taktuk.gforge.inria.fr/taktuk.html#item_put__2a_src__2a__2a_dest__2a)

remote_location = None

The directory on the remote hosts were the files will be copied. substitions described in execo.substitutions.remote_substitute will not be performed, but taktuk substitutions can be used (see http://taktuk.gforge.inria.fr/taktuk.html#item_put__2a_src__2a__2a_dest__2a)

ChainPut

Inheritance diagram of execo.action.ChainPut
class execo.action.ChainPut(hosts, local_files, remote_location='.', connection_params=None, **kwargs)

Bases: execo.action.SequentialActions

Broadcast local files to several remote host, with an unencrypted, unauthenticated chain of host to host copies (idea taken from kastafior).

Each broadcast is performed with a chain copy (simultaneously: host0 sending to host1, host1 sending to host2, … hostN to hostN+1)

ChainPut relies on:

  • running a bourne shell and netcat being available both on remote hosts and on localhost.
  • direct TCP connections allowed between any nodes among localhost and remote hosts. The exact chain of TCP connections is: localhost to first remote host, first remote host to second remote host, and so on up to the last remote host.

On the security side, data transfers are not crypted, and ChainPut briefly opens a TCP server socket on each remote host, accepting any data without authentication. Insecure temporary files are used. It is thus intended to be used in a secured network environment.

Parameters:
  • hosts – iterable of execo.host.Host onto which to copy the files.
  • local_file – iterable of source file (local pathes).
  • remote_location – destination directory (remote path).
  • connection_params – a dict similar to execo.config.default_connection_params whose values will override those in default_connection_params for connection.

Get

Inheritance diagram of execo.action.Get
class execo.action.Get(hosts, remote_files, local_location='.', connection_params=None, **kwargs)

Bases: execo.action.Remote

Copy remote files from several remote host to a local directory, with scp or a similar connection tool.

Parameters:
connection_params = None

Dict similar to execo.config.default_connection_params whose values will override those in default_connection_params for connection.

local_location = None

The local directory were the files will be copied. substitions described in execo.substitutions.remote_substitute will be performed.

remote_files = None

Iterable of string of file paths. substitions described in execo.substitutions.remote_substitute will be performed.

TaktukGet

Inheritance diagram of execo.action.TaktukGet
class execo.action.TaktukGet(hosts, remote_files, local_location='.', connection_params=None, **kwargs)

Bases: execo.action.TaktukRemote

Copy remote files from several remote host to a local directory, with taktuk.

Parameters:
connection_params = None

Dict similar to execo.config.default_connection_params whose values will override those in default_connection_params for connection.

local_location = None

The local directory were the files will be copied. Substitions described in execo.substitutions.remote_substitute will not be performed, but taktuk substitutions can be used (see http://taktuk.gforge.inria.fr/taktuk.html#item_get__2a_src__2a__2a_dest__2a)

remote_files = None

Iterable of string of file paths. Substitions described in execo.substitutions.remote_substitute will not be performed, but taktuk substitutions can be used (see http://taktuk.gforge.inria.fr/taktuk.html#item_get__2a_src__2a__2a_dest__2a)

Local

Inheritance diagram of execo.action.Local
class execo.action.Local(cmd, process_args=None, **kwargs)

Bases: execo.action.Action

Launch a command localy.

Parameters:
  • cmd – the command to run.
  • process_args – Dict of keyword arguments passed to instanciated processes.
cmd = None

the command to run

kill()

Kill all processes not yet ended.

Returns immediately, without waiting for processes to be actually killed.

return self

process_args = None

Dict of keyword arguments passed to instanciated processes.

start()

Start all processes.

return self

ParallelActions

Inheritance diagram of execo.action.ParallelActions
class execo.action.ParallelActions(actions, **kwargs)

Bases: execo.action.Action

An execo.action.Action running several sub-Actions in parallel.

Will start, kill, wait, run every action in parallel.

hide_subactions = None

Wether to hide sub actions in stats.

kill()

Kill all processes not yet ended.

Returns immediately, without waiting for processes to be actually killed.

return self

reset()

Reinitialize an Action so that it can later be restarted.

If it is running, this method will first kill it then wait for its termination before reseting.

start()

Start all processes.

return self

stats()

Return a dict summarizing the statistics of all processes of this Action.

see execo.report.Report.stats.

SequentialActions

Inheritance diagram of execo.action.SequentialActions
class execo.action.SequentialActions(actions, **kwargs)

Bases: execo.action.Action

An execo.action.Action running several sub-actions sequentially.

Will start, kill, wait, run every Action sequentially.

hide_subactions = None

Wether to hide sub actions in stats.

kill()

Kill all processes not yet ended.

Returns immediately, without waiting for processes to be actually killed.

return self

reset()

Reinitialize an Action so that it can later be restarted.

If it is running, this method will first kill it then wait for its termination before reseting.

start()

Start all processes.

return self

stats()

Return a dict summarizing the statistics of all processes of this Action.

see execo.report.Report.stats.

ActionLifecycleHandler

Inheritance diagram of execo.action.ActionLifecycleHandler
class execo.action.ActionLifecycleHandler

Bases: object

Abstract handler for execo.action.Action lifecycle.

end(action)

Handle execo.action.Action’s end.

Parameters:action – The Action which ends.
reset(action)

Handle execo.action.Action’s reset.

Parameters:action – the Action which is reset.
start(action)

Handle execo.action.Action’s start.

Parameters:action – The Action which starts.

wait_any_actions

execo.action.wait_any_actions(actions, timeout=None)

Wait for any of the actions given to terminate.

Parameters:

returns: iterable of execo.action.Action which have terminated.

wait_all_actions

execo.action.wait_all_actions(actions, timeout=None)

Wait for all of the actions given to terminate.

Parameters:

returns: iterable of execo.action.Action which have terminated.

filter_bad_hosts

execo.action.filter_bad_hosts(action, hosts)

Returns the list of host filtered from any host where any process of the action has failed.

Parameters:
  • action – The action from which the processes are checked
  • hosts – The iterable of hosts or host addesses to be filtered and returned as a list

ActionFactory

class execo.action.ActionFactory(remote_tool=None, fileput_tool=None, fileget_tool=None)

Instanciate multiple remote process execution and file copies using configurable connector tools: ssh, scp, taktuk

Parameters:
  • remote_tool – can be execo.config.SSH or execo.config.TAKTUK
  • fileput_tool – can be execo.config.SCP, execo.config.TAKTUK or execo.config.CHAINPUT
  • fileget_tool – can be execo.config.SCP or execo.config.TAKTUK
get_fileget(*args, **kwargs)

Instanciates a execo.action.Get or execo.action.TaktukGet

get_fileput(*args, **kwargs)

Instanciates a execo.action.Put, execo.action.TaktukPut or execo.action.ChainPut

get_remote(*args, **kwargs)

Instanciates a execo.action.Remote or execo.action.TaktukRemote

get_remote

execo.action.get_remote(*args, **kwargs)

Instanciates a execo.action.Remote or execo.action.TaktukRemote with the default factory

get_fileput

execo.action.get_fileput(*args, **kwargs)

Instanciates a execo.action.Put or execo.action.TaktukPut with the default factory

get_fileget

execo.action.get_fileget(*args, **kwargs)

Instanciates a execo.action.Get or execo.action.TaktukGet with the default factory

substitutions for Remote, TaktukRemote, Get, Put

In the command line given for a execo.action.Remote, execo.action.TaktukRemote, as well as in pathes given to execo.action.Get and execo.action.Put, some patterns are automatically substituted:

  • all occurences of the literal string {{{host}}} are substituted by the address of the execo.host.Host to which execo connects to.
  • all occurences of {{<expression>}} are substituted in the following way: <expression> must be a python expression, which will be evaluated in the context (globals and locals) where the execo.action.Remote, execo.action.Put, execo.action.Get is instanciated, and which must return a sequence. {{<expression>}} will be replaced by <expression>[index % len(<expression>)].

For example, in the following code:

execo.Remote('iperf -c {{[host.address for host in hosts2]}}', hosts1)

When execo runs this command on all hosts of hosts1, it will produce a different command line for each host, each command line connecting a host from hosts1 to a host from hosts2

if hosts1 contains six hosts a, b, c, d, e, f and hosts2 contains 3 hosts 1, 2, 3 the mapping between hosts1 and hosts2 could be:

  • a -> 1
  • b -> 2
  • c -> 3
  • d -> 1
  • e -> 2
  • f -> 3

execo.substitutions.remote_substitute is the function used internally by execo to perform these substitutions:

execo.substitutions.remote_substitute(string, all_hosts, index, frame_context)

Perform some tag substitutions in a specific context.

Parameters:
  • string – the string onto which to perfom the substitution.
  • all_hosts – an iterable of execo.host.Host which is the context into which the substitution will be made. all_hosts[index] is the execo.host.Host to which this string applies.
  • index – the index in all_hosts of the execo.host.Host to which this string applies.
  • frame_context – a tuple of mappings (globals, locals) in the context of which the expressions (if any) will be evaluated.
  • Replaces all occurences of the literal string {{{host}}} by the execo.host.Host address itself.
  • Replaces all occurences of {{<expression>}} in the following way: <expression> must be a python expression, which will be evaluated in the context of frame_context (globals and locals), and which must return a sequence. {{<expression>}} will be replaced by <expression>[index % len(<expression>)].

Miscellaneous classes

Host

class execo.host.Host(address, user=False, keyfile=False, port=False)

Bases: object

A host to connect to.

  • Has an address (mandatory)
  • Can optionaly have a user, a keyfile, a port, which are used for remote connection and authentification (with a ssh like remote connection tool).

Has an intuitive comparison and hashing behavior: two execo.host.Host with the same members (address, user, keyfile, port) will hash equally and will be seen as identical keys in a set or dict.

>>> h1 = Host('localhost')
>>> h1.user = 'root'
>>> h1
Host('localhost', user='root')
>>> h2 = Host('localhost', user = 'root')
>>> h1 == h2
True
>>> d = dict()
>>> d[h1] = True
>>> d[h2]
True
Parameters:
  • address – (string or execo.host.Host) the host address or another execo.host.Host instance which will be copied into this new instance
  • user – (string) optional user whith which to connect. If False (default value), means use the default user. If None, means don’t use any user.
  • keyfile – (string) optional keyfile whith which to connect. If False (default value), means use the default keyfile. If None, means don’t use any keyfile.
  • port – (integer) optional port to which to connect. If False (default value), means use the default port. If None, means don’t use any port.
address = None

the host address

Report

class execo.report.Report(stats_objects=None)

Bases: object

Summary of one or more execo.action.Action.

A Report gathers the results of actions or (recursively) of other reports.

Parameters:stats_objects –
add(stats_objects)

Add some sub-execo.report.Report or execo.action.Action to this report.

Parameters:stats_objects –
static empty_stats()

Return a stats initialized to zero.

name

Name given to this report. If None, a default name will be given.

stats()

Return a dict summarizing the statistics of all execo.action.Action and sub-execo.report.Report registered to this report.

This stats dict contains the following metrics:

  • start_date: earliest start date (unix timestamp) of all Action or None if none have started yet.
  • end_date: latest end date (unix timestamp) of all Action or None if not available (not all started, not all ended).
  • num_processes: number of processes in all Action.
  • num_started: number of processes that have started.
  • num_ended: number of processes that have ended.
  • num_errors: number of processes that went in error when started.
  • num_timeouts: number of processes that had to be killed (SIGTERM) after reaching their timeout.
  • num_forced_kills: number of processes that had to be forcibly killed (SIGKILL) after not responding for some time.
  • num_non_zero_exit_codes: number of processes that ran correctly but whose return code was != 0.
  • num_expect_fail: number of processes on which there was an expect failure, i.e. the expect search did not find a match before reaching the expect timeout or the stream eof/error.
  • num_write_error: number of processes on which there was a write error.
  • num_ok: number of processes which:
    • did not started
    • started and not yet ended
    • started and ended and did not went in error (or where launched with flag ignore_error) , did not timeout (or where launched with flag ignore_timeout), and had an exit code == 0 (or where launched with flag ignore_exit_code).
  • num_finished_ok: number of processes which started, ended, and are ok.
to_string(wide=False, brief=False)

Returns a formatted string with a human-readable stats of all Action results.

Parameters:
  • wide – if False (default), report format is designed for 80 columns display. If True, output a (175 characters) wide report.
  • brief – when True, only the Total stats is output, not each Action or Report stats. Default is False.

Timer

class execo.time_utils.Timer(timeout=None)

Bases: object

Keep track of elapsed time.

Create and start the Timer.

elapsed()

Return this Timer’s instance elapsed time since start.

remaining()

Returns the remaining time before the timeout.

start_date()

Return this Timer’s instance start time.

wait_elapsed(elapsed)

Sleep until the given amount of time has elapsed since the Timer’s start.

Parameters:elapsed – the delay to sleep in one of the formats handled (or None) (see execo.time_utils.get_seconds).

Exceptions

class execo.exception.ProcessesFailed(processes)

Bases: Exception

Raised when one or more execo.process.Process have failed.

Parameters:processes – iterable of failed processes
processes = None

iterable of failed processes

Utilities

sleep

execo.time_utils.sleep(delay=None, until=None)

Sleep until a given delay has elapsed or until a given date.

If both present, will sleep at least for the delay and at least until the date.

Parameters:

get_unixts

execo.time_utils.get_unixts(d)

Convert a date to a unix timestamp (float).

Parameters:d –

a date in one of the supported types. if date == None, returns None. Supported types

  • datetime.datetime: see execo.time_utils.datetime_to_unixts
  • string: see execo.time_utils.str_date_to_unixts
  • numeric type: see execo.time_utils.numeric_date_to_unixts

get_seconds

execo.time_utils.get_seconds(duration)

Convert a duration to a number of seconds (float).

Parameters:duration –

a duration in one of the supported types. if duration == None, returns None. Supported types

  • datetime.timedelta: see execo.time_utils.timedelta_to_seconds
  • string: see execo.time_utils.str_duration_to_seconds
  • numeric type: see execo.time_utils.numeric_duration_to_seconds

format_date

execo.time_utils.format_date(d, showms=False)

Return a string with the formatted date (year, month, day, hour, min, sec, ms) in locale timezone and in rfc-3339 format for pretty printing.

Parameters:
  • d – a date in one of the formats handled (or None) (see execo.time_utils.get_unixts).
  • showms – whether to show ms or not. Default False.

format_duration

execo.time_utils.format_duration(duration, showms=False)

Return a string with a formatted duration (days, hours, mins, secs, ms) for pretty printing.

Parameters:
  • duration – a duration in one of the formats handled (or None) (see execo.time_utils.get_seconds).
  • showms – whether to show ms or not. Default False.

get_hosts_list

execo.host.get_hosts_list(hosts)

Deep copy an iterable of execo.host.Host or hostnames to a list of execo.host.Host.

order is preserved

get_hosts_set

execo.host.get_hosts_set(hosts)

Deep copy an iterable of execo.host.Host or hostnames to a set of execo.host.Host.

memoize

execo.utils.memoize(obj)

memoizing decorator

works on functions, methods, or classes, and exposes the cache publicly. From https://wiki.python.org/moin/PythonDecoratorLibrary#Alternate_memoize_as_nested_functions warning: not thread-safe, but as dict is a primitive type, worst case race condition would be that the underlying obj is called two times for the same args/kwargs, and the cache is updated serially with both (necessarily identical) values for the same key.

get_port

execo.utils.get_port()

Thread safely returns a round-robbed port in the range configuration['port_range']

Logging

execo uses the standard logging module for logging. By default, logs are sent to stderr, logging level is logging.WARNING, so no logs should be output when everything works correctly. Some logs will appear if some processes don’t exit with a zero return code (unless they were created with flag ignore_non_zer_exit_code), or if some processes timeout (unless they were created with flag ignore_timeout), or if process instanciation resulted in a operating system error (unless they were created with flag ignore_error).

logger

execo.log.logger

The execo logger.

Configuration

This module may be configured at import time by modifiing execo module variables execo.config.configuration, execo.config.default_connection_params or by defining two dicts configuration and default_connection_params in the file ~/.execo.conf.py

configuration

The configuration dict contains global configuration parameters.

execo.config.configuration = {'color_mode': False, 'color_styles': {'log_header': ('yellow',), 'object_repr': ('blue',), 'emph': ('cyan',), 'report_warn': ('magenta',), 'report_error': ('red', 'bold'), 'command': ('blue', 'bold'), 'host': ('magenta', 'bold'), 'user1': ('green', 'bold'), 'user2': ('yellow', 'bold'), 'user3': ('cyan', 'bold'), 3: ('green', 'reverse'), 5: ('green', 'reverse'), 10: ('green',), 12: ('green', 'bold'), 15: ('magenta', 'bold'), 20: ('magenta',), 30: ('cyan',), 40: ('red',), 50: ('yellow', 'on_red')}, 'compact_output_threshold': 4096, 'fileget_tool': 1, 'fileput_tool': 1, 'intr_period': 1, 'kill_childs_at_end': True, 'kill_timeout': 60, 'log_level': 20, 'port_range': (25500, 26700), 'remote_tool': 0}

Global execo configuration parameters.

  • log_level: the log level (see module logging)
  • remote_tool: default tool to use when instanciating remote processes. Can be execo.config.SSH or execo.config.TAKTUK
  • fileput_tool: default tool to use to put files remotely. Can be execo.config.SCP or execo.config.TAKTUK
  • fileget_tool: default tool to use to get remote files. Can be execo.config.SCP or execo.config.TAKTUK
  • compact_output_threshold: only beginning and end of stdout / stderr are displayed by execo.process.ProcessBase.dump when their size is greater than this threshold. 0 for no threshold
  • kill_timeout: number of seconds to wait after a clean SIGTERM kill before assuming that the process is not responsive and killing it with SIGKILL
  • intr_period: number of seconds between periodic check interrupt, for correct handling of ctrl-c
  • port_range: a tuple (start port, end port) of ports to use for the function execo.utils.get_port. As all python ranges, start is inclusive, end is exclusive.
  • kill_childs_at_end: Whether to try sending SIGTERM to all subprocesses started through execo when the script terminates. Warnings:
    • this config option must be set at execo import time, changing it later will be ignored
    • SIGTERM is only sent to all childs or subchilds which did not try to daemonize by changing their process group.
  • color_mode: whether to colorize output (with ansi escape sequences)
  • color_styles: mapping of identifiers to iterables of ansi attributes identifiers (see execo.log._ansi_styles)

Its default values are:

configuration = {
    'log_level': logging.INFO,
    'remote_tool': SSH,
    'fileput_tool': SCP,
    'fileget_tool': SCP,
    'compact_output_threshold': 4096,
    'kill_timeout': 60,
    'intr_period': 1,
    'port_range': (25500, 26700),
    'kill_childs_at_end': True,
    'color_mode': checktty(sys.stdout)
                  and checktty(sys.stderr),
    'color_styles': {
        'log_header': ('yellow',),
        'object_repr': ('blue',),
        'emph': ('cyan',),
        'report_warn': ('magenta',),
        'report_error': ('red', 'bold'),
        'command': ('blue', 'bold'),
        'host': ('magenta', 'bold'),
        'user1': ('green', 'bold'),
        'user2': ('yellow', 'bold'),
        'user3': ('cyan', 'bold'),
        IODEBUG: ('green', 'reverse'),
        FDEBUG: ('green', 'reverse'),
        logging.DEBUG: ('green',),
        TRACE: ('green', 'bold',),
        DETAIL: ('magenta', 'bold'),
        logging.INFO: ('magenta',),
        logging.WARNING: ('cyan',),
        logging.ERROR: ('red',),
        logging.CRITICAL: ('yellow', 'on_red')
        },
    }

default_connection_params

The default_connection_params dict contains default parameters for remote connections.

execo.config.default_connection_params = {'chainput_chain_retry': 10, 'chainput_host_retry': 5, 'chainput_nc_client_timeout': 5, 'chainput_nc_server_timeout': 30, 'chainput_try_delay': 1, 'forwarding_timeout': 25, 'host_rewrite_func': None, 'keyfile': None, 'nc': '/bin/nc.traditional -v -v', 'port': None, 'pty': False, 'scp': 'scp', 'scp_options': ('-o', 'BatchMode=yes', '-o', 'PasswordAuthentication=no', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'ConnectTimeout=20', '-rp'), 'ssh': 'ssh', 'ssh_options': ('-tt', '-o', 'BatchMode=yes', '-o', 'PasswordAuthentication=no', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'ConnectTimeout=20'), 'taktuk': 'taktuk', 'taktuk_connector': 'ssh', 'taktuk_connector_options': ('-o', 'BatchMode=yes', '-o', 'PasswordAuthentication=no', '-o', 'StrictHostKeyChecking=no', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'ConnectTimeout=20'), 'taktuk_options': ('-s',), 'user': None}

Default connection params for ssh/scp/taktuk connections.

  • user: the user to connect with.
  • keyfile: the keyfile to connect with.
  • port: the port to connect to.
  • ssh: the ssh or ssh-like command.
  • scp: the scp or scp-like command.
  • taktuk: the taktuk command.
  • ssh_options: tuple of options passed to ssh.
  • scp_options: tuple of options passed to scp.
  • taktuk_options: tuple of options passed to taktuk.
  • taktuk_connector: the ssh-like connector command for taktuk.
  • taktuk_connector_options: tuple of options passed to taktuk_connector.
  • nc: the netcat command to use
  • chainput_nc_client_timeout: timeout for client connection to next hop
  • chainput_nc_server_timeout: timeout for server to wait for incoming connection from previous hop
  • chainput_host_retry: number of times each hop in the transfer chain retries to connect to next hop
  • chainput_chain_retry: number of times each hop in the transfer chain tries a new next hop if all tries to current next hop fail. If given a float (between 0.0 and 1.0), this is expressed as a ratio of the total number of hosts in the chain.
  • chainput_try_delay: delay in seconds between TCP client to server connection attempts.
  • forwarding_timeout: max time to wait for a forwarding port to be available
  • pty: boolean. Wether to allocate or not a pty for ssh/scp.
  • host_rewrite_func: function called to rewrite hosts addresses. Takes a host address, returns a host address.

Its default values are:

    default_connection_params = {
        'user':        None,
        'keyfile':     None,
        'port':        None,
        'ssh':         'ssh',
        'scp':         'scp',
        'taktuk':      'taktuk',
        'ssh_options': ( '-tt',
                         '-o', 'BatchMode=yes',
                         '-o', 'PasswordAuthentication=no',
                         '-o', 'StrictHostKeyChecking=no',
                         '-o', 'UserKnownHostsFile=/dev/null',
                         '-o', 'ConnectTimeout=20' ),
        'scp_options': ( '-o', 'BatchMode=yes',
                         '-o', 'PasswordAuthentication=no',
                         '-o', 'StrictHostKeyChecking=no',
                         '-o', 'UserKnownHostsFile=/dev/null',
                         '-o', 'ConnectTimeout=20',
                         '-rp' ),
        'taktuk_options': ( '-s', ),
        'taktuk_connector': 'ssh',
        'taktuk_connector_options': ( '-o', 'BatchMode=yes',
                                      '-o', 'PasswordAuthentication=no',
                                      '-o', 'StrictHostKeyChecking=no',
                                      '-o', 'UserKnownHostsFile=/dev/null',
                                      '-o', 'ConnectTimeout=20'),
        'chainput_nc_client_timeout': 5,
        'chainput_nc_server_timeout': 30,
        'nc': '/bin/nc.traditional -v -v',
        'chainput_host_retry': 5,
        'chainput_chain_retry': 10,
        'chainput_try_delay': 1,
        'forwarding_timeout': 25,
        'pty': False,
        'host_rewrite_func': None
        }

These default connection parameters are the ones used when no other specific connection parameters are given to execo.process.SshProcess, execo.action.Remote, execo.action.TaktukRemote, execo.action.Get, execo.action.TaktukGet, execo.action.Put, execo.action.TaktukPut, or given to the execo.host.Host. When connecting to a remote host, the connection parameters are first taken from the execo.host.Host instance to which the connection is made, then from the connection_params given to the execo.process.SshProcess / execo.action.TaktukRemote / execo.action.Remote / execo.action.Get / execo.action.TaktukGet / execo.action.Put / execo.action.TaktukPut, if there are some, then from the default_connection_params, which has default values which can be changed by directly modifying its global value, or in ~/.execo.conf.py

ssh/scp configuration for SshProcess, Remote, TaktukRemote, Get, TaktukGet, Put, TaktukPut

For execo.process.SshProcess, execo.action.Remote, execo.action.TaktukRemote, execo.action.Get, execo.action.TaktukGet, execo.action.Put, execo.action.TaktukPut to work correctly, ssh/scp connections need to be fully automatic: No password has to be asked. The default configuration in execo is to force a passwordless, public key based authentification. As this tool is growing in a cluster/grid environment where servers are frequently redeployed, default configuration also disables strict key checking, and the recording of hosts keys to ~/.ssh/know_hosts. This may be a security hole in a different context.

Miscellaneous notes

Time: timestamps and durations

Two notions are used in execo for dealing with time: the notion of a date, also called a timestamp, which uniquely identifies a moment in time, and the notion of duration. Internally all dates are unix timestamps, ie. number of seconds elapsed since the unix epoch (00:00:00 on Jan 1 1970), possibly with or without subsecond precision (float or integer). All durations are in seconds, also possibly with or without subsecond precision.

When passing parameters to execo api, all timestamps and durations can be expressed in various formats (see execo.time_utils.get_unixts, execo.time_utils.get_seconds) and will be automatically converted to dates as unix timestamps, and durations in seconds.

All dates in execo are UTC. When giving a date to execo, you can give it with an explicit timezone or if no timezone is given, execo will assume that datetime.datetime or dates expressed as strings are in local timezone. For timestamps given directly as unix timestamps, it will assume they are in UTC.

If a developer wants to take advantage of the time facilities of execo, the main functions are execo.time_utils.get_unixts for properly converting the various timestamp formats to a unix timestamp (with proper handling of timezone), and execo.time_utils.get_seconds for properly converting the various duration formats to seconds. execo.time_utils.format_date and execo.time_utils.format_duration can be used to pretty-print a timestamp or a duration, expressed in any of the format supported by execo.time_utils.get_unixts or execo.time_utils.get_seconds (they actually call them internally before pretty-printing).

Exceptions at shutdown

Some exceptions may sometimes be triggered at python shutdown, with the message most likely raised during interpreter shutdown. They are most likely caused by a bug in shutdown code’s handling of threads termination, and thus can be ignored. See http://bugs.python.org/issue1856