5. Examples

This is a collection of python scripts that demonstrate the usage of the kamaki clients API library

5.1. Initial steps

5.1.1. Initialize clients

Initialize the identity (auth), storage (store), compute, network, block storage (volume) and image clients, given an authentication URL and TOKEN.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
from kamaki.clients import astakos, pithos, cyclades, image
from kamaki.clients.utils import https

https.patch_with_certs('/etc/ssl/certs/ca-certificates.crt')

identity_client = astakos.AstakosClient(URL, TOKEN)

pithosURL = identity_client.get_endpoint_url(pithos.PithosClient.service_type)
storage_client = pithos.PithosClient(pithosURL, TOKEN)
storage_client.account = identity_client.user_info['id']
storage_client.container = 'pithos'

imageURL = identity_client.get_endpoint_url(image.ImageClient.service_type)
image_client = image.ImageClient(imageURL, TOKEN)

computeURL = identity_client.get_endpoint_url(
    cyclades.CycladesComputeClient.service_type)
compute_client = cyclades.CycladesComputeClient(computeURL, TOKEN)

networkURL = identity_client.get_endpoint_url(
    cyclades.CycladesNetworkClient.service_type)
network_client = cyclades.CycladesNetworkClient(networkURL, TOKEN)

volumeURL = identity_client.get_endpoint_url(
    cyclades.CycladesBlockStorageClient.service_type)
volume_client = cyclades.CycladesBlockStorageClient(volumeURL, TOKEN)

Warning

Line 4 sets the CA certificates bundle to support secure connections. Secure connections are enabled by default and must be managed before setting any clients. See SSL authentication for more details.

5.1.2. Authentication URL and TOKEN from config file

Drag the URL and TOKEN information from the kamaki configuration file, using the “Config” class from kamaki CLI.

1
2
3
4
5
6
from kamaki.cli.config import Config

cnf = Config()
CLOUD = cnf.get('global', 'default_cloud')
URL = cnf.get_cloud(CLOUD, 'url')
TOKEN = cnf.get_cloud(CLOUD, 'token')

Note

The cloud URL and TOKEN are stored under a cloud name. Kamaki can be configured to use multiple clouds. It is common practice to set the value of the default cloud using the global.default_cloud configuration option. Here it was assumed that the value global.default_cloud is the name of the preferred cloud

5.1.3. Log HTTP

Instruct kamaki to output the HTTP logs on the console.

1
2
3
4
from kamaki.cli import logger

logger.add_stream_logger('kamaki.clients.recv', fmt='< %(message)s')
logger.add_stream_logger('kamaki.clients.send', fmt='> %(message)s')

5.2. Containers and files

5.2.1. Information on object

List all objects in the storage_client’s default container and ask user to pick one of them for more information.

1
2
3
4
5
6
7
8
import json

names = [o['name'] for o in storage_client.list_objects()]
print 'Remote objects:\n\t', '\n\t'.join(names)

pick = raw_input('Pick one: ')
remote_object = storage_client.get_object_info(pick)
print json.dumps(remote_object, indent=2)

5.2.2. Backup container

Back up the contents of the default container to a new container.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
import time

backup_container = 'backup_%s' % time.time()
storage_client.create_container(backup_container)

for o in storage_client.list_objects():
    try:
        storage_client.copy_object(
            storage_client.container, o['name'], backup_container)
    except ClientError as ce:
        print "Failed to backup object %s" % o['name']
        print ce

5.2.3. Empty and delete containers

Delete all containers if their names start with “backup”.

1
2
3
4
5
6
7
8
9
current_container = storage_client.container
try:
    for c in storage_client.list_containers():
        if c['name'].startswith('backup_'):
            storage_client.container = c['name']
            storage_client.del_container(delimiter='/')
            storage_client.purge_container()
finally:
    storage_client.container = current_container

Note

The “del_container” method will empty the container. The “purge_container” method will destroy an empty container. If the container is not empty, it cannot be destroyed.

Note

The “try-finally” clause is used to preserve the original container settings of the client (usually “pithos”)

5.2.4. Upload and Download

Upload a local file

1
2
3
4
5
6
import os.path

local_file_name = raw_input('File to upload: ')
with open(local_file_name) as f:
    remote_object = os.path.basename(f.name)
    storage_client.upload_object(remote_object, f, upload_cb=_gen)

Download a remote object as local file

1
2
3
4
5
6
import os.path

obj = raw_input('Pick remote object to download:')
target = raw_input('Local file name: ')
with open(target, 'w+') as f:
    storage_client.download_object(obj, f, download_cb=_gen)

Note

The _gen callback function is used to show upload/download progress. It is optional. It must be a python generator, for example:

1
2
3
4
5
6
7
import sys

def _gen(length):
    for i in range(length):
        sys.stderr.write('*')
        yield
    yield

5.2.5. Asynchronous batch upload

Upload all files in a directory asynchronously

1
2
3
4
5
6
7
8
9
from os import walk

def upload_one_file(fname):
    with open(fname) as f:
        storage_client.upload_object(fname, f)

local_path = raw_input('Dir to upload: ')
for top, subdirs, files in walk(local_path):
    storage_client.async_run(upload_one_file, [dict(fname=fn) for fn in files])

5.2.6. Reassign container

Each resource is assigned to a project, where the resource quotas are defined. With this script, users are prompted to choose a project to assign the default container.

1
2
3
4
5
import json

projects = [{p['id']: p['name']} for p in identity_client.get_projects()]
print 'These are your projects:', json.dumps(projects, indent=2)
storage_client.reassign_container(raw_input('Assign container to project id: '))

5.2.7. Download and stream in parallel

Download an object in chunks. Stream them as they are being downloaded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
obj = raw_input('Pick object to stream: ')
destination = raw_input('Stream it where? ')

obj_size = int(storage_client.get_object_info(obj)['content-length'])
BLOCK_SIZE = int(storage_client.get_container_info()['x-container-block-size'])
CHUNK_SIZE = 4 * BLOCK_SIZE

def stream(i, output):
    """Stream the contents of buf[i] to output"""
    output.write(bufs[i])

from kamaki.clients import SilentEvent

with open(destination, 'w+') as output:
    event = None
    bufs = ['', '']
    for i in range(1 + (obj_size / CHUNK_SIZE)):
        buf_index = i % 2
        start, end = CHUNK_SIZE * i, min(CHUNK_SIZE * (i + 1), obj_size)
        bufs[buf_index] = storage_client.download_to_string(
            obj, range_str='%s-%s' % (start, end))
        if event and not event.is_alive():
            event.join()
        event = SilentEvent(stream, buf_index, output)
        event.start()

Note

The kamaki.clients.SilentEvent class extends threading.Thread in order to simplify thread handling.

5.3. Images

5.3.1. Register image

Upload an image to container “images” and register it to Plankton.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
storage_client.container, path = 'images', 'my_image.diskdump'
with open(path) as f:
    storage_client.upload_object(path, f)

location = (identity_client.user_info()['id'], storage_client.container, path)
properties = dict(osfamily='linux', users='root', os='debian')
img = image_client.register('My New Image', location, properties=properties)

import json
print 'Image created sucesfully', json.dumps(img, indent=2)

Note

Properties are mandatory in order to create a working image. In this example it is assumed a Debian Linux image. The suggested method for creating, uploading and registering custom images is by using the snf-image-creator tool.

5.3.2. Find image

Find images belonging to current user, by its name.

1
2
3
4
image_name = raw_input('Image name to look for:')
my_images = [img for img in image_client.list_public() if (
    img.get('owner') == identity_client.user_info()['id']) and (
    img.get('name') == image_name)]

5.3.3. Modify image

Change the name and add a properties to an image. Use the image created above. One of the properties (description) is new, the other (users) exists and will be updated with a new value.

1
2
3
new_props = img.get('properties', dict()).update(
    dict(description='Debian Wheezy', users='root user'))
image_client.update_image(img['id'], name='New Name', properties=new_props)

5.3.4. Unregister image

Unregister the image created above.

1
image_client.unregister(img['id'])

5.4. Virtual Machines (Servers)

5.4.1. Find flavors

Find all flavors with 2048 MB of RAM, 2 CPU cores and disk space between 20 GB and 40 GB.

1
2
3
for flv in compute_client.list_flavors(detail=True):
    if flv['ram'] == 2048 and flv['vcpus'] == 2 and 20 <= flv['disk'] <= 40:
        print 'Flavor', flv['id'], 'matches'

5.4.2. Create server

To create a server, pick a name, a flavor id and an image id. In this example, assume the image from a previous step.

1
2
3
4
5
srv_name = raw_input('Server name: ')
flv = compute_client.get_flavor_details(raw_input('Flavor id: '))

srv = compute_client.create_server(
    srv_name, flavor_id=flv['id'], image_id=img['id'])

Note

To access the virtual server, a password is returned by the creation method. This password is revealed only once, when the server is created and it’s not stored anywhere on the service side.

A popular access method is to inject the user ssh keys, as shown bellow.

1
2
3
4
5
6
7
8
from base64 import b64encode
ssh_keys = dict(
    contents=b64encode(open('~/.ssh/id_rsa.pub').read()),
    path='/root/.ssh/authorized_keys',
    owner='root', group='root', mode=0600)
ssh_srv = compute_client.create_server(
    srv_name,
    flavor_id=flv['id'], image_id=img['id'], personality=[ssh_keys, ])

5.4.3. Connection information

There are many ways to connect to a server: using a password or ssh keys, through a VNC console, the IP address or the qualified domain name.

Credentials for connection through a VNC console:

1
2
3
4
5
import json

vnc_console = compute_client.get_server_console(srv['id'])
print 'The following VNC credentials will be invalidated shortly'
print json.dumps(vnc_console, indent=2)

The following script collects all network information available: the F.Q.D.N. (fully qualified domain name) and the IP addresses (v4 as well as v6).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
conn_info = dict(fqdn=srv['SNF:fqdn'], ipv4=[], ipv6=[])

nics = compute_client.get_server_nics(srv['id'])
for port in nics['attachments']:
    if port['ipv4']:
        conn_info['ipv4'].append(port['ipv4'])
    if port['ipv6']:
        conn_info['ipv6'].append(port['ipv6'])

print json.dumps(conn_info, indent=2)

5.4.4. Update server

Rename the server and then add/change some metadata.

1
2
3
compute_client.update_server_name(srv['id'], 'New Server Name')
compute_client.update_server_metadata(
    srv['id'], modified=True, description='No description')

5.4.5. Start, Shutdown, Reboot or Delete server

First, get the current status of the server, and write a method for handling the wait results.

1
2
3
4
5
def assert_status(new, expected):
    assert new is not None, 'Timeout while waiting for status %s' % expected
    assert new == expected, 'Server did not reach status %s' % expected

srv['status'] = compute_client.get_server_details(srv['id'])['status']

Shutdown a server, assuming it is currently active.

1
2
3
4
5
compute_client.shutdown_server(srv['id'])
print 'Wait for server to shutdown'
srv['status'] = compute_client.wait_server(srv['id'], srv['status'])
assert_status(srv['status'], 'STOPPED')
print '... OK'

Start the stopped server.

1
2
3
4
5
compute_client.start_server(srv['id'])
print 'Wait for server to start'
srv['status'] = compute_client.wait_server(srv['id'], srv['status'])
assert_status(srv['status'], 'ACTIVE')
print '... OK'

Reboot the active server.

1
2
3
4
5
compute_client.reboot_server(srv['id'])
print 'Wait for server to reboot'
srv['status'] = compute_client.wait_server(srv['id'], 'REBOOT')
assert_status(srv['status'], 'ACTIVE')
print '... OK'

Destroy the server.

1
2
3
4
5
compute_client.delete_server(srv['id'])
print 'Wait for server to be deleted'
srv['status'] = compute_client.wait_server(srv['id'], srv['status'])
assert_status(srv['status'], 'DELETED')
print '... OK'

5.5. Server snapshots

5.5.1. Lookup volume

Each virtual server has at least one volume. This information is already stored in the “srv” object retrieved in a previous step.

1
print 'Volumes of server', srv['id'], ':', srv['volumes']

Retrieve this information using the Block Storage client and check if it matches.

1
2
3
4
5
6
7
8
volumes = []
for vol in volume_client.list_volumes(detail=True):
    for att in vol['attachments']:
        if att['server_id'] == srv['id']:
            volumes.append(int(vol['id']))
            continue

assert sorted(srv['volumes']) == sorted(volumes), 'Volumes do not match!'

5.5.2. Create and delete volume

Create an extra volume. A server can have multiple volumes attached.

Note

In this example, the size of the volume is retrieved from the size of the server flavor. This is the safest method to set a fitting size.

1
2
3
flv = compute_client.get_flavor_details(srv['flavor']['id'])
new_volume = volume_client.create_volume(
    size=flv['disk'], server_id=srv['id'], display_name='New Volume')

Destroy the volume.

1
volume_client.delete_volume(new_volume['id'])

Warning

volume creation and deletion may take some time to complete.

5.5.3. Lookup snapshot

Find the snapshots of the server’s first volume.

1
2
snapshots = [snp for snp in volume_client.list_snapshots(detail=True) if (
    snp['volume_id'] == srv['volumes'][0])]

5.5.4. Create and delete snapshot

Create a snapshot of the first server volume.

1
new_snapshot = volume_client.create_snapshot(srv['volumes'][0], 'New Snapshot')

Delete the snapshot.

1
volume_client.delete_snapshot(new_snapshot['id'])

5.5.5. Backup and restore snapshot

A snapshot can be thought as a backup, stored at users “snapshots” container.

Restore server from snapshot (assume the one created in the previous step).

1
2
compute_client.create_server(
    name='Reserve', image_id=new_snapshot['id'], flavor_id=srv['flavor']['id'])

To be safer, download the snapshot to local disk.

1
2
3
4
obj = new_snapshot['display_name']
storage_client.container = 'snapshots'
with open(obj, 'w+') as f:
    storage_client.download_object(obj, f)

If the snapshot has been removed from the system, the server can be restored by uploading it from the local copy.

1
2
3
4
5
6
7
8
9
with open(obj, 'r') as f:
    storage_client.upload_object(obj, f)

location = (storage_client.account, storage_client.container, obj)
props = dict(exclude_all_taks=True)
snp = image_client.register('Restored snapshot', location, properties=props)

compute_client.create_server(
    name='Reserve', image_id=snp['id'], flavor_id=srv['flavor']['id'])

Note

By uploading from a local copy, we must register the snapshot as an image. Use the “exclude_all_taks” to register such images.

5.6. Networks

Term Description
network A public or private network
sunet A subnet of a network
port A connection between a device (e.g., vm) and a network
floating_ip An external IP v4, reserved by the current user for a network

5.6.1. Public and private networks

Public networks are created by the system. Private networks are created and managed by users.

Separate public from private networks.

1
2
3
4
5
6
public_nets, vpns = [], []
for net in network_client.list_networks(detail=True):
    if net['public']:
        public_nets.append(net)
    else:
        vpns.append(net)

5.6.2. Create and destroy virtual private network

Create a VPN.

1
2
3
vpn = network_client.create_network(
    type=cyclades.CycladesNetworkClient.network_types[1], name='New Net')
print 'Created VPN with id', vpn['id'], 'with name', vpn['name']

Note

The “type” of the network is a Cyclades-specific parameter. To see all network types:

1
print 'Network types:', cyclades.CycladesNetworkClient.network_types

Delete the VPN.

1
network_client.delete_network(vpn['id'])

5.6.3. Lookup IP

Find the ID of an IP.

1
2
3
4
5
6
ip_addr, ip_id = '123.45.67.89', None

for ip in network_client.list_floatingips():
    if ip_addr == ip['floating_ip_address']:
        ip_id = ip['id']
        break

5.6.4. Lookup server from IP

Find the server ID from an IP ID.

1
2
3
ip = network_client.get_floatingip_details(ip_id)
if ip['instance_id']:
    print 'IP', ip['floating_ip_address'], 'is used by', ip['instance_id']

5.6.5. Reserve and release IP

Reserve an IP.

1
2
ip = network_client.create_floatingip()
print 'Reserved new IP', ip['floating_ip_address']

Note

Reserving an IP means “make it available for use”. A freshly reserved IP is not used by any servers.

Release an IP.

1
2
network_client.delete_floatingip(ip['id'])
print 'Released IP', ip['floating_ip_address']

5.6.6. Attach and dettach IP

Attach IP to server, by creating a connection (port) between the server and the network related to the IP.

Note

The “srv” object and the “assert_status” method from an earlier script are used here too.

1
2
3
4
5
6
7
8
9
port = network_client.create_port(
    network_id=ip['floating_network_id'],
    device_id=srv['id'],
    fixed_ips=[dict(ip_address=ip['floating_ip_address']), ])

print 'Attaching IP', ip['floating_ip_address'], 'to server', port['device_id']
port['status'] = network_client.wait_port(port['id'], port['status'])
assert_status(port['status'], 'ACTIVE')
print '... OK'

Detach IP from server.

1
2
3
4
5
6
network_client.delete_port(port['id'])

print 'Detaching IP', ip['floating_ip_address'], 'from', port['device_id']
port['status'] = network_client.wait_port(port['id'], port['status'])
assert_status(port['status'], 'DELETED')
print '... OK'