API¶
awsfabrictasks.conf¶
-
class
awsfabrictasks.conf.
Settings
[source]¶ Bases:
object
Settings object inspired by django.conf.settings.
-
clear_settings
()[source]¶ Clear all settings (intended for testing). Deletes all uppercase attributes.
-
awsfabrictasks.utils¶
-
exception
awsfabrictasks.utils.
InvalidLogLevel
[source]¶ Bases:
exceptions.KeyError
Raised when
getLoglevelFromString()
gets an invalidloglevelstring
.
-
awsfabrictasks.utils.
compute_localfile_md5sum
(localfile)[source]¶ Compute the hex-digested md5 checksum of the given
localfile
.Parameters: localfile – Path to a file on the local filesystem.
-
awsfabrictasks.utils.
configureStreamLogger
(loggername, level)[source]¶ Configure a stdout/stderr logger (logging.StreamHandler) with the given
loggername
andlevel
. If you are configuring logging for a task, useconfigureStreamLoggerForTask()
.This is suitable for log-configuration for a single task, where the user specifies a loglevel.
Returns: The configured logger.
-
awsfabrictasks.utils.
configureStreamLoggerForTask
(modulename, taskname, loglevel)[source]¶ Configure logging for a task.
Shortcut for:
configureStreamLogger(modulename + '.' + taskname, loglevel)
Example (note that what you put in the loglevel docs for your task depends on how you use the logger):
@task mytask(loglevel='INFO'): """ Does some task. :param loglevel: Controls the amount of output: QUIET --- No output. INFO --- Only produce output for changes. DEBUG --- One line of output for each file. Defaults to "INFO". """ log = configureStreamLoggerForTask(__name__, 's3_syncupload_dir', getLoglevelFromString(loglevel)) log.info('Hello world')
-
awsfabrictasks.utils.
force_slashend
(path)[source]¶ Return
path
suffixed with/
(path is unchanged if it is already suffixed with/
).
-
awsfabrictasks.utils.
getLoglevelFromString
(loglevelstring)[source]¶ Lookup
loglevelstring
inloglevel_stringmap
.Raises: InvalidLogLevel – If loglevelstring is not in loglevel_stringmap
.Returns: The loglevel. Return type: int
-
awsfabrictasks.utils.
guess_contenttype
(filename)[source]¶ Return the content-type for the given
filename
. Usesmimetypes.guess_type()
.
-
awsfabrictasks.utils.
parse_bool
(data)[source]¶ Return
True
if data is one of::'true', 'True', True
. Otherwise, returnFalse
.
-
awsfabrictasks.utils.
rsyncformat_path
(source_dir, sync_content=False)[source]¶ rsync uses
/
in the source directory to determine if we should sync a directory or the contents of a directory. How rsync works:- Sync contents:
- Source path ending with
/
means sync the contents (just as if we used/*
except that*
does not include hidden files). - Sync the directory:
- Source path NOT ending with
/
means sync the directory. I.e.: If the source is/etc/init.d
, and the destination is/tmp
, the contents of/etc/init.d
is copied into/tmp/init.d/
.
This is error-prone, and the consequences can be severe if combined with
--delete
. Therefore, we use a boolean to distinguish between these two methods of specifying source directory, and reformat the path usingforce_slashend()
andforce_noslashend()
.Parameters: - source_dir – The source directory. May be a remote directory (i.e.: [USER@]HOSTNAME:PATH), or a local directory.
- sync_content – Normally the function automatically makes sure
local_dir
is not suffixed with/
, which makes rsync copylocal_dir
intoremote_dir
. Withsync_content=True
, the content oflocal_dir
is synced intoremote_dir
instead.
-
awsfabrictasks.utils.
sudo_chattr
(remote_path, owner=None, mode=None)[source]¶ Run
sudo_chown()
andsudo_chmod()
onremote_path
. If owner or mode is None, their corresponding function is not called.
-
awsfabrictasks.utils.
sudo_mkdir_p
(remote_path, **chattr_kw)[source]¶ sudo mkdir -p <remote_path>
followed by :func:`sudo_chattr`(remote_path, **chattr_kw).
-
awsfabrictasks.utils.
sudo_upload_dir
(local_dir, remote_dir, **chattr_kw)[source]¶ Upload all files and directories in
local_dir
toremote_dir
. Directories are created withsudo_mkdir_p()
and files are uploaded withsudo_upload_file()
.chattr_kw
is forwarded in both cases.
-
awsfabrictasks.utils.
sudo_upload_file
(local_path, remote_path, **chattr_kw)[source]¶ Use sudo to upload a file from
local_path
toremote_path
and runsudo_chattr()
with the givenchattr_kw
as arguments.
-
awsfabrictasks.utils.
sudo_upload_string_to_file
(string_to_upload, remote_path, **chattr_kw)[source]¶ Create a tempfile containing
string_to_upload
, and usesudo_upload_file()
to upload the tempfile. Removes the tempfile when the upload is complete or if it fails.Parameters: - string_to_upload – The string to write to the tempfile.
- remote_path – See
sudo_upload_file()
. - chattr_kw – See
sudo_upload_file()
.
-
awsfabrictasks.utils.
loglevel_stringmap
= {'INFO': 20, 'QUIET': 50, 'WARN': 30, 'CRITICAL': 50, 'ERROR': 40, 'DEBUG': 10}¶ Map of strings to loglevels (for the logging module)
awsfabrictasks.ubuntu¶
Ubuntu utilities.
awsfabrictasks.ec2.api¶
-
exception
awsfabrictasks.ec2.api.
Ec2RegionConnectionError
(region)[source]¶ Bases:
exceptions.Exception
Raised when we fail to connect to a region.
-
exception
awsfabrictasks.ec2.api.
InstanceLookupError
[source]¶ Bases:
exceptions.LookupError
Base class for instance lookup errors.
-
exception
awsfabrictasks.ec2.api.
MultipleInstancesWithSameNameError
[source]¶ Bases:
awsfabrictasks.ec2.api.InstanceLookupError
Raised when multiple instances with the same nametag is discovered. (see:
Ec2InstanceWrapper.get_by_nametag()
)
-
exception
awsfabrictasks.ec2.api.
NoInstanceWithNameFound
[source]¶ Bases:
awsfabrictasks.ec2.api.InstanceLookupError
Raised when no instace with the requested name is found in
Ec2InstanceWrapper.get_by_nametag()
.
-
exception
awsfabrictasks.ec2.api.
NotExactlyOneInstanceError
[source]¶ Bases:
awsfabrictasks.ec2.api.InstanceLookupError
Raised when more than one instance is found when expecting exactly one instance.
-
exception
awsfabrictasks.ec2.api.
WaitForStateError
[source]¶ Bases:
exceptions.Exception
Raises when
wait_for_state()
times out.
-
class
awsfabrictasks.ec2.api.
Ec2InstanceWrapper
(instance)[source]¶ Bases:
object
Wraps a
boto.ec2.instance.Instance
with convenience functions.Variables: instance – The boto.ec2.instance.Instance
.Parameters: instance – A boto.ec2.instance.Instance
object.-
add_instance_to_env
()[source]¶ Add
self
tofabric.api.env.ec2instances[self.get_ssh_uri()]
, and register the key-pair for the instance infabric.api.env.key_filename
.
-
classmethod
get_by_instanceid
(instanceid)[source]¶ Connect to AWS and get the EC2 instance with the given instance ID.
Parameters: instanceid_with_optional_region – Parsed with
parse_instanceid()
to find the region and name.Raises: - Ec2RegionConnectionError – If connecting to the region fails.
- LookupError – If the requested instance was not found in the region.
Returns: A
Ec2InstanceWrapper
contaning the requested instance.
-
classmethod
get_by_nametag
(instancename_with_optional_region)[source]¶ Connect to AWS and get the EC2 instance with the given Name-tag.
Parameters: instancename_with_optional_region – Parsed with
parse_instancename()
to find the region and name.Raises: - Ec2RegionConnectionError – If connecting to the region fails.
- InstanceLookupError – Or one of its subclasses if the requested instance was not found in the region.
Returns: A
Ec2InstanceWrapper
contaning the requested instance.
-
classmethod
get_by_tagvalue
(tags={}, region=None)[source]¶ Connect to AWS and get the EC2 instance with the given tag:value pairs.
- :param tags
- A string like ‘role=testing,fake=yes’ to AND a set of ec2 instance tags
Parameters: region – optional. Raises: Ec2RegionConnectionError – If connecting to the region fails. Returns: A list of :class:`Ec2InstanceWrapper`s containing the matching instances.
-
classmethod
get_exactly_one_by_tagvalue
(tags, region=None)[source]¶ Use
get_by_tagvalue()
to find instances bytags
, but raiseLookupError
if not exactly one instance is found.
-
classmethod
get_from_host_string
()[source]¶ If an instance has been registered in
fabric.api.env
usingadd_instance_to_env()
, this method can be used to get the instance identified byfabric.api.env.host_string
.
-
get_ssh_key_filename
()[source]¶ Get the SSH indentify filename (.pem-file) for the instance. Searches
awsfab_settings.KEYPAIR_PATH
for"<instance.key_name>.pem"
.Raises: LookupError – If the key is not found.
-
-
class
awsfabrictasks.ec2.api.
Ec2LaunchInstance
(extra_tags={}, configname=None, configname_help='Please select one of the following configurations:', duplicate_name_protection=True)[source]¶ Bases:
object
Launch instances configured in
awsfab_settings.EC2_LAUNCH_CONFIGS
.Example:
launcher = Ec2LaunchInstance(extra_tags={'Name': 'mytest'}) launcher.confirm() instance = launcher.run_instance()
Note that this class is optimized for the following use case:
- Create one or more instances (initialize one or more Ec2LaunchInstance).
- Confirm using
confirm()
orconfirm_many()
. - Launch each instance using meth:Ec2LaunchInstance.run_instance or
Ec2LaunchInstance.run_many_instances()
. - Use
Ec2LaunchInstance.wait_for_running_state_many()
to wait for all instances to launch. - Do something with the running instances.
Example of launching many instances:
a = Ec2LaunchInstance(extra_tags={‘Name’: ‘a’}) b = Ec2LaunchInstance(extra_tags={‘Name’: ‘b’}) Ec2LaunchInstance.confirm_many([a, b]) Ec2LaunchInstance.run_many_instances([a, b]) # Note: that we can start doing stuff witha
andb
that does not # require the instances to be running, such as setting tags. Ec2LaunchInstance.wait_for_running_state_many([a, b])Initialize the launcher. Runs
create_config_ask_if_none()
.Parameters: - configname – Name of a configuration in
awsfab_settings.EC2_LAUNCH_CONFIGS
. If it isNone
, we ask the user for the configfile. - configname_help – The help to show above the prompt for configname input (only used
if
configname
isNone
.
-
confirm
()[source]¶ Use
prettyprint()
to show the user their choices, and ask for confirmation. Runsfabric.api.abort()
if the user does not confirm the choices.
-
classmethod
confirm_many
(launchers)[source]¶ Loop through Use
prettyprint()
to show the user their choices, and ask for confirmation. Runsfabric.api.abort()
if the user does not confirm the choices.
-
create_config_ask_if_none
()[source]¶ Set
kw
andconf
usingconfigname
. Prompt the user for a configname if bool(configname
) isFalse
.
Merge tags from the awsfab_settings.EC2_LAUNCH_CONFIGS config, and the
extra_tags
parameter for __init__, and return the resulting dict.
-
run_instance
()[source]¶ Run/launch the configured instance, and add the tags to the instance (
get_all_tags()
).Returns: The launched instance.
-
classmethod
run_many_instances
(launchers)[source]¶ Loop through
launchers
and runrun_instance()
.Parameters: - launchers – List of Ec2LaunchInstance objects.
- kwargs – Forwarded to
wait_for_running_state()
.
-
classmethod
wait_for_running_state_many
(launchers, **kwargs)[source]¶ Loop through
launchers
and runwait_for_running_state()
.Parameters: - launchers – List of Ec2LaunchInstance objects that have been lauched with
Ec2LaunchInstance.run_instance()
. - kwargs – Forwarded to
wait_for_running_state()
.
- launchers – List of Ec2LaunchInstance objects that have been lauched with
-
conf
= None¶ A config dict from awsfab_settings.EC2_LAUNCH_CONFIGS.
-
configname
= None¶ See the docs for the __init__ parameter.
-
configname_help
= None¶ See the docs for the __init__ parameter.
-
instance
= None¶ The instance launced by
run_instance()
. None when run_instance() has not been invoked.
-
kw
= None¶ Keyword arguments for
run_instances()
.
-
tag_retry_count
= 4¶ Number of times to retry when adding tags gets EC2ResponseError.
-
tag_retry_sleep
= 2¶ Number of seconds to sleep before retrying when adding tags gets EC2ResponseError.
-
awsfabrictasks.ec2.api.
ec2_rsync
(*args, **kwargs)[source]¶ Deprecated since version 1.0.13: Use
ec2_rsync_upload()
instead.
-
awsfabrictasks.ec2.api.
ec2_rsync_download
(remote_dir, local_dir, rsync_args='-av', sync_content=False)[source]¶ rsync
remote_dir
on the current EC2 instance (the one returned byEc2InstanceWrapper.get_from_host_string()
) intolocal_dir
.Parameters: sync_content – Normally the function automatically makes sure local_dir
is not suffixed with/
, which makes rsync copylocal_dir
intoremote_dir
. Withsync_content=True
, the content oflocal_dir
is synced intoremote_dir
instead.
-
awsfabrictasks.ec2.api.
ec2_rsync_download_command
(instancewrapper, remote_dir, local_dir, rsync_args='-av', sync_content=False)[source]¶ Returns the rsync command used by
ec2_rsync_download()
. Takes the same parameters asec2_rsync_download()
, except for the first parameter,instancewrapper
, which is aEc2InstanceWrapper
object.
-
awsfabrictasks.ec2.api.
ec2_rsync_upload
(local_dir, remote_dir, rsync_args='-av', sync_content=False)[source]¶ rsync
local_dir
intoremote_dir
on the current EC2 instance (the one returned byEc2InstanceWrapper.get_from_host_string()
).Parameters: sync_content – Normally the function automatically makes sure local_dir
is not suffixed with/
, which makes rsync copylocal_dir
intoremote_dir
. Withsync_content=True
, the content oflocal_dir
is synced intoremote_dir
instead.
-
awsfabrictasks.ec2.api.
ec2_rsync_upload_command
(instancewrapper, local_dir, remote_dir, rsync_args='-av', sync_content=False)[source]¶ Returns the rsync command used by
ec2_rsync_upload()
. Takes the same parameters asec2_rsync_upload()
, except for the first parameter,instancewrapper
, which is aEc2InstanceWrapper
object.
-
awsfabrictasks.ec2.api.
parse_instanceid
(instanceid_with_optional_region)[source]¶ Parse instance id with an optional region-name prefixed. Region name is specified by prefixing the instanceid with
<regionname>:
.Returns: (region, instanceid) where region defaults to awsfab_settings.DEFAULT_REGION
if not prefixed to the id.
-
awsfabrictasks.ec2.api.
parse_instancename
(instancename_with_optional_region)[source]¶ Just like
parse_instanceid()
, however this is for instance names. We keep them as separate functions in case they diverge in the future.Returns: (region, instanceid) where region defaults to awsfab_settings.DEFAULT_REGION
if not prefixed to the name.
-
awsfabrictasks.ec2.api.
print_ec2_instance
(instance, full=False, indentspaces=3)[source]¶ Print attributes of an ec2 instance.
Parameters: - instance – A
boto.ec2.instance.Instance
object. - full – Print all attributes? If not, a subset of the attributes are printed.
- indentspaces – Number of spaces to indent each line in the output.
- instance – A
-
awsfabrictasks.ec2.api.
wait_for_running_state
(instanceid, **kwargs)[source]¶ Shortcut for
wait_for_state(instanceid, 'running', **kwargs)
.
-
awsfabrictasks.ec2.api.
wait_for_state
(instanceid, state_name, sleep_intervals=[15, 5], last_sleep_repeat=40)[source]¶ Poll the instance with
instanceid
until itsstate_name
matches the desiredstate_name
.The first poll is performed without any delay, and the rest of the polls are performed according to
sleep_intervals
.Parameters: - instanceid – ID of an instance.
- state_name – The state_name to wait for.
- sleep_intervals – List of seconds to wait between each poll for state. The first poll
is made immediately, then we wait for sleep_intervals[0] seconds before the next poll,
and repeat for each item in sleep_intervals. Then we repeat for
last_sleep_repeat
using the last item insleep_intervals
as the timout for each wait. - last_sleep_repeat – Number of times to repeat the last item in
sleep_intervals
. If this is 20, we will wait for a maximum ofsum(sleep_intervals) + sleep_intervals[-1]*20
.
awsfabrictasks.decorators¶
awsfabrictasks.s3.api¶
-
exception
awsfabrictasks.s3.api.
S3ConnectionError
(msg='Could not connect S3')[source]¶ Bases:
exceptions.Exception
Raised when we fail to connect to S3.
-
exception
awsfabrictasks.s3.api.
S3ErrorBase
[source]¶ Bases:
exceptions.Exception
Base class for all S3 errors. Never raised directly.
-
exception
awsfabrictasks.s3.api.
S3FileDoesNotExist
(s3file)[source]¶ Bases:
awsfabrictasks.s3.api.S3FileErrorBase
Raised when an
S3File
does not exist.Parameters: s3file – A S3File
object.
-
exception
awsfabrictasks.s3.api.
S3FileErrorBase
(s3file)[source]¶ Bases:
awsfabrictasks.s3.api.S3ErrorBase
Base class for all
S3File
errors. Never raised directly.Parameters: s3file – A S3File
object.
-
exception
awsfabrictasks.s3.api.
S3FileExistsError
(s3file)[source]¶ Bases:
awsfabrictasks.s3.api.S3FileErrorBase
Raised when trying to overwrite an existing
S3File
, unless overwriting is requested.Parameters: s3file – A S3File
object.
-
exception
awsfabrictasks.s3.api.
S3FileNoInfo
(s3file)[source]¶ Bases:
awsfabrictasks.s3.api.S3FileErrorBase
Raised when trying to use
S3File
metadata before performing a HEAD request.Parameters: s3file – A S3File
object.
-
class
awsfabrictasks.s3.api.
S3ConnectionWrapper
(connection)[source]¶ Bases:
object
S3 connection wrapper.
Parameters: bucket – A boto.rds.bucket.DBInstance
object.-
classmethod
get_bucket
(bucketname)[source]¶ Get the requested bucket.
Shortcut for:
S3ConnectionWrapper.get_connection().connection.get_bucket(bucketname)
Parameters: bucketname – Name of an S3 bucket.
-
classmethod
get_bucket_using_pattern
(bucketname)[source]¶ Same as
get_bucket()
, however thebucketname
is filtered throughsettingsformat_bucketname()
.
-
classmethod
-
class
awsfabrictasks.s3.api.
S3File
(bucket, key)[source]¶ Bases:
object
Simplifies working with keys in S3 buckets.
-
delete
()[source]¶ Delete the key/file from the bucket.
Raises: S3FileDoesNotExist – If the key does not exist in the bucket.
-
etag_matches_localfile
(localfile)[source]¶ Return
True
if the file at the path given inlocalfile
has an md5 hex-digested checksum matching the etag of this S3 key.
-
set_contents_from_filename
(localfile, overwrite=False)[source]¶ Upload
localfile
.Parameters: overwrite – If True
, overwrite if the key/file exists.Raises: S3FileExistsError – If overwrite==True
and the key exists in the bucket.
-
set_contents_from_string
(data, overwrite=False)[source]¶ Write
data
to the S3 file.Parameters: overwrite – If True
, overwrite if the key/file exists.Raises: S3FileExistsError – If overwrite==True
and the key exists in the bucket.
-
-
class
awsfabrictasks.s3.api.
S3Sync
(bucket, local_dir, s3prefix)[source]¶ Bases:
object
Makes it easy to sync files to and from S3. This class does not make any changes to the local filesyste, or S3, it only makes it easy to write function that works with hierarkies of files synced locally and on S3.
A good example is the sourcecode for
awsfabrictasks.s3.tasks.s3_syncupload_dir()
.Parameters: - bucket – A
boto.rds.bucket.DBInstance
object. - local_dir – The local directory.
- local_dir – The S3 key prefix that corresponds to
local_dir
.
-
iterfiles
()[source]¶ Iterate over all files both local and within the S3 prefix. Yields
S3SyncIterFile
objects.How it works:
- Uses
dirlist_absfilenames()
to get all local files in thelocal_dir
. - Uses
s3list_s3filedict()
to get all S3 files in thes3prefix
. - Uses these two sets of information to create
S3SyncIterFile
objects.
- Uses
- bucket – A
-
class
awsfabrictasks.s3.api.
S3SyncIterFile
[source]¶ Bases:
object
Objects of this class is yielded by
S3Sync.iterfiles()
. Contains info about where the file exists, its local and S3 path (even if it does not exist).-
both_exists
()[source]¶ Returns
True
iflocalexists
ands3exists
.
-
download_s3file_to_localfile
()[source]¶ create_localdir()
and download the file ats3path
tolocalpath
.
-
localexists
= None¶ Local file exists?
-
localpath
= None¶ The local path. Always set. Use
localexists
if you want to know if the local file exists.
-
s3exists
= None¶ S3 file exists?
-
-
awsfabrictasks.s3.api.
dirlist_absfilenames
(dirpath)[source]¶ Get all the files within the given
dirpath
as a set of absolute filenames.
-
awsfabrictasks.s3.api.
iter_bucketcontents
(bucket, prefix, match, delimiter, formatter=<function <lambda>>)[source]¶ Iterate over items given bucket, yielding items formatted for output.
Parameters: - bucket – A class:boto.s3.bucket.Bucket object.
- prefix – The prefix to list. Defaults to empty string, which lists all items in the root directory.
- match –
A Unix shell style pattern to match. Matches against the entire key name (in filesystem terms: the absolute path).
Uses the
fnmatch
python module. The match is case-sensitive.Examples:
*.jpg *2012*example*.log icon-*.png
- delimiter – The delimiter to use. Defaults to
/
. - formatter – Formatter callback to use to format each key. Not used on Prefix-keys (directories). The callback should take a key as input, and return a string.
-
awsfabrictasks.s3.api.
localpath_to_s3path
(localdir, localpath, s3prefix)[source]¶ Convert a local filepath into a S3 path within the given
s3prefix
.Parameters: - localdir – The local directory that corresponds to
s3prefix
. - localpath – Path to a file within
localdir
. - s3prefix – Prefix to use for the file on S3.
Example:: >>> localpath_to_s3path(‘/mydir’, ‘/mydir/hello/world.txt’, ‘my/test’) ‘my/test/hello/world.txt’
- localdir – The local directory that corresponds to
-
awsfabrictasks.s3.api.
s3list_s3filedict
(bucket, prefix)[source]¶ Get all the keys with the given
prefix
as a dict with key-name as key and the key-object wrappen in aS3File
as value.
-
awsfabrictasks.s3.api.
s3path_to_localpath
(s3prefix, s3path, localdir)[source]¶ Convert a s3 filepath into a local filepath within the given
localdir
.Parameters: - s3prefix – Prefix used for the file on S3.
- s3path – Path to a file within
s3prefix
. - localdir – The local directory that corresponds to
s3prefix
.
Example:: >>> s3path_to_localpath(‘mydir/’, ‘mydir/hello/world.txt’, ‘/my/test’) ‘/my/test/hello/world.txt’