This is a collection of python scripts that demonstrate the usage of the kamaki clients API library
Initialize the identity (auth), storage (store), compute, network, block storage (volume) and image clients, given an authentication URL and TOKEN.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | from kamaki.clients import astakos, pithos, cyclades, image
from kamaki.clients.utils import https
https.patch_with_certs('/etc/ssl/certs/ca-certificates.crt')
identity_client = astakos.AstakosClient(URL, TOKEN)
pithosURL = identity_client.get_endpoint_url(pithos.PithosClient.service_type)
storage_client = pithos.PithosClient(pithosURL, TOKEN)
storage_client.account = identity_client.user_info['id']
storage_client.container = 'pithos'
imageURL = identity_client.get_endpoint_url(image.ImageClient.service_type)
image_client = image.ImageClient(imageURL, TOKEN)
computeURL = identity_client.get_endpoint_url(
cyclades.CycladesComputeClient.service_type)
compute_client = cyclades.CycladesComputeClient(computeURL, TOKEN)
networkURL = identity_client.get_endpoint_url(
cyclades.CycladesNetworkClient.service_type)
network_client = cyclades.CycladesNetworkClient(networkURL, TOKEN)
volumeURL = identity_client.get_endpoint_url(
cyclades.CycladesBlockStorageClient.service_type)
volume_client = cyclades.CycladesBlockStorageClient(volumeURL, TOKEN)
|
Warning
Line 4 sets the CA certificates bundle to support secure connections. Secure connections are enabled by default and must be managed before setting any clients. See SSL authentication for more details.
Drag the URL and TOKEN information from the kamaki configuration file, using the “Config” class from kamaki CLI.
1 2 3 4 5 6 | from kamaki.cli.config import Config
cnf = Config()
CLOUD = cnf.get('global', 'default_cloud')
URL = cnf.get_cloud(CLOUD, 'url')
TOKEN = cnf.get_cloud(CLOUD, 'token')
|
Note
The cloud URL and TOKEN are stored under a cloud name. Kamaki can be configured to use multiple clouds. It is common practice to set the value of the default cloud using the global.default_cloud configuration option. Here it was assumed that the value global.default_cloud is the name of the preferred cloud
Instruct kamaki to output the HTTP logs on the console.
1 2 3 4 | from kamaki.cli import logger
logger.add_stream_logger('kamaki.clients.recv', fmt='< %(message)s')
logger.add_stream_logger('kamaki.clients.send', fmt='> %(message)s')
|
List all objects in the storage_client’s default container and ask user to pick one of them for more information.
1 2 3 4 5 6 7 8 | import json
names = [o['name'] for o in storage_client.list_objects()]
print 'Remote objects:\n\t', '\n\t'.join(names)
pick = raw_input('Pick one: ')
remote_object = storage_client.get_object_info(pick)
print json.dumps(remote_object, indent=2)
|
Back up the contents of the default container to a new container.
1 2 3 4 5 6 7 8 9 10 11 12 | import time
backup_container = 'backup_%s' % time.time()
storage_client.create_container(backup_container)
for o in storage_client.list_objects():
try:
storage_client.copy_object(
storage_client.container, o['name'], backup_container)
except ClientError as ce:
print "Failed to backup object %s" % o['name']
print ce
|
Delete all containers if their names start with “backup”.
1 2 3 4 5 6 7 8 9 | current_container = storage_client.container
try:
for c in storage_client.list_containers():
if c['name'].startswith('backup_'):
storage_client.container = c['name']
storage_client.del_container(delimiter='/')
storage_client.purge_container()
finally:
storage_client.container = current_container
|
Note
The “del_container” method will empty the container. The “purge_container” method will destroy an empty container. If the container is not empty, it cannot be destroyed.
Note
The “try-finally” clause is used to preserve the original container settings of the client (usually “pithos”)
Upload a local file
1 2 3 4 5 6 | import os.path
local_file_name = raw_input('File to upload: ')
with open(local_file_name) as f:
remote_object = os.path.basename(f.name)
storage_client.upload_object(remote_object, f, upload_cb=_gen)
|
Download a remote object as local file
1 2 3 4 5 6 | import os.path
obj = raw_input('Pick remote object to download:')
target = raw_input('Local file name: ')
with open(target, 'w+') as f:
storage_client.download_object(obj, f, download_cb=_gen)
|
Note
The _gen callback function is used to show upload/download progress. It is optional. It must be a python generator, for example:
1 2 3 4 5 6 7 | import sys
def _gen(length):
for i in range(length):
sys.stderr.write('*')
yield
yield
|
Upload all files in a directory asynchronously
1 2 3 4 5 6 7 8 9 | from os import walk
def upload_one_file(fname):
with open(fname) as f:
storage_client.upload_object(fname, f)
local_path = raw_input('Dir to upload: ')
for top, subdirs, files in walk(local_path):
storage_client.async_run(upload_one_file, [dict(fname=fn) for fn in files])
|
Each resource is assigned to a project, where the resource quotas are defined. With this script, users are prompted to choose a project to assign the default container.
1 2 3 4 5 | import json
projects = [{p['id']: p['name']} for p in identity_client.get_projects()]
print 'These are your projects:', json.dumps(projects, indent=2)
storage_client.reassign_container(raw_input('Assign container to project id: '))
|
Download an object in chunks. Stream them as they are being downloaded.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | obj = raw_input('Pick object to stream: ')
destination = raw_input('Stream it where? ')
obj_size = int(storage_client.get_object_info(obj)['content-length'])
BLOCK_SIZE = int(storage_client.get_container_info()['x-container-block-size'])
CHUNK_SIZE = 4 * BLOCK_SIZE
def stream(i, output):
"""Stream the contents of buf[i] to output"""
output.write(bufs[i])
from kamaki.clients import SilentEvent
with open(destination, 'w+') as output:
event = None
bufs = ['', '']
for i in range(1 + (obj_size / CHUNK_SIZE)):
buf_index = i % 2
start, end = CHUNK_SIZE * i, min(CHUNK_SIZE * (i + 1), obj_size)
bufs[buf_index] = storage_client.download_to_string(
obj, range_str='%s-%s' % (start, end))
if event and not event.is_alive():
event.join()
event = SilentEvent(stream, buf_index, output)
event.start()
|
Note
The kamaki.clients.SilentEvent class extends threading.Thread in order to simplify thread handling.
Upload an image to container “images” and register it to Plankton.
1 2 3 4 5 6 7 8 9 10 | storage_client.container, path = 'images', 'my_image.diskdump'
with open(path) as f:
storage_client.upload_object(path, f)
location = (identity_client.user_info()['id'], storage_client.container, path)
properties = dict(osfamily='linux', users='root', os='debian')
img = image_client.register('My New Image', location, properties=properties)
import json
print 'Image created sucesfully', json.dumps(img, indent=2)
|
Note
Properties are mandatory in order to create a working image. In this example it is assumed a Debian Linux image. The suggested method for creating, uploading and registering custom images is by using the snf-image-creator tool.
Find images belonging to current user, by its name.
1 2 3 4 | image_name = raw_input('Image name to look for:')
my_images = [img for img in image_client.list_public() if (
img.get('owner') == identity_client.user_info()['id']) and (
img.get('name') == image_name)]
|
Change the name and add a properties to an image. Use the image created above. One of the properties (description) is new, the other (users) exists and will be updated with a new value.
1 2 3 | new_props = img.get('properties', dict()).update(
dict(description='Debian Wheezy', users='root user'))
image_client.update_image(img['id'], name='New Name', properties=new_props)
|
Find all flavors with 2048 MB of RAM, 2 CPU cores and disk space between 20 GB and 40 GB.
1 2 3 | for flv in compute_client.list_flavors(detail=True):
if flv['ram'] == 2048 and flv['vcpus'] == 2 and 20 <= flv['disk'] <= 40:
print 'Flavor', flv['id'], 'matches'
|
To create a server, pick a name, a flavor id and an image id. In this example, assume the image from a previous step.
1 2 3 4 5 | srv_name = raw_input('Server name: ')
flv = compute_client.get_flavor_details(raw_input('Flavor id: '))
srv = compute_client.create_server(
srv_name, flavor_id=flv['id'], image_id=img['id'])
|
Note
To access the virtual server, a password is returned by the creation method. This password is revealed only once, when the server is created and it’s not stored anywhere on the service side.
A popular access method is to inject the user ssh keys, as shown bellow.
1 2 3 4 5 6 7 8 | from base64 import b64encode
ssh_keys = dict(
contents=b64encode(open('~/.ssh/id_rsa.pub').read()),
path='/root/.ssh/authorized_keys',
owner='root', group='root', mode=0600)
ssh_srv = compute_client.create_server(
srv_name,
flavor_id=flv['id'], image_id=img['id'], personality=[ssh_keys, ])
|
There are many ways to connect to a server: using a password or ssh keys, through a VNC console, the IP address or the qualified domain name.
Credentials for connection through a VNC console:
1 2 3 4 5 | import json
vnc_console = compute_client.get_server_console(srv['id'])
print 'The following VNC credentials will be invalidated shortly'
print json.dumps(vnc_console, indent=2)
|
The following script collects all network information available: the F.Q.D.N. (fully qualified domain name) and the IP addresses (v4 as well as v6).
1 2 3 4 5 6 7 8 9 10 | conn_info = dict(fqdn=srv['SNF:fqdn'], ipv4=[], ipv6=[])
nics = compute_client.get_server_nics(srv['id'])
for port in nics['attachments']:
if port['ipv4']:
conn_info['ipv4'].append(port['ipv4'])
if port['ipv6']:
conn_info['ipv6'].append(port['ipv6'])
print json.dumps(conn_info, indent=2)
|
Rename the server and then add/change some metadata.
1 2 3 | compute_client.update_server_name(srv['id'], 'New Server Name')
compute_client.update_server_metadata(
srv['id'], modified=True, description='No description')
|
First, get the current status of the server, and write a method for handling the wait results.
1 2 3 4 5 | def assert_status(new, expected):
assert new is not None, 'Timeout while waiting for status %s' % expected
assert new == expected, 'Server did not reach status %s' % expected
srv['status'] = compute_client.get_server_details(srv['id'])['status']
|
Shutdown a server, assuming it is currently active.
1 2 3 4 5 | compute_client.shutdown_server(srv['id'])
print 'Wait for server to shutdown'
srv['status'] = compute_client.wait_server(srv['id'], srv['status'])
assert_status(srv['status'], 'STOPPED')
print '... OK'
|
Start the stopped server.
1 2 3 4 5 | compute_client.start_server(srv['id'])
print 'Wait for server to start'
srv['status'] = compute_client.wait_server(srv['id'], srv['status'])
assert_status(srv['status'], 'ACTIVE')
print '... OK'
|
Reboot the active server.
1 2 3 4 5 | compute_client.reboot_server(srv['id'])
print 'Wait for server to reboot'
srv['status'] = compute_client.wait_server(srv['id'], 'REBOOT')
assert_status(srv['status'], 'ACTIVE')
print '... OK'
|
Destroy the server.
1 2 3 4 5 | compute_client.delete_server(srv['id'])
print 'Wait for server to be deleted'
srv['status'] = compute_client.wait_server(srv['id'], srv['status'])
assert_status(srv['status'], 'DELETED')
print '... OK'
|
Each virtual server has at least one volume. This information is already stored in the “srv” object retrieved in a previous step.
1 | print 'Volumes of server', srv['id'], ':', srv['volumes']
|
Retrieve this information using the Block Storage client and check if it matches.
1 2 3 4 5 6 7 8 | volumes = []
for vol in volume_client.list_volumes(detail=True):
for att in vol['attachments']:
if att['server_id'] == srv['id']:
volumes.append(int(vol['id']))
continue
assert sorted(srv['volumes']) == sorted(volumes), 'Volumes do not match!'
|
Create an extra volume. A server can have multiple volumes attached.
Note
In this example, the size of the volume is retrieved from the size of the server flavor. This is the safest method to set a fitting size.
1 2 3 | flv = compute_client.get_flavor_details(srv['flavor']['id'])
new_volume = volume_client.create_volume(
size=flv['disk'], server_id=srv['id'], display_name='New Volume')
|
Destroy the volume.
1 | volume_client.delete_volume(new_volume['id'])
|
Warning
volume creation and deletion may take some time to complete.
Find the snapshots of the server’s first volume.
1 2 | snapshots = [snp for snp in volume_client.list_snapshots(detail=True) if (
snp['volume_id'] == srv['volumes'][0])]
|
Create a snapshot of the first server volume.
1 | new_snapshot = volume_client.create_snapshot(srv['volumes'][0], 'New Snapshot')
|
Delete the snapshot.
1 | volume_client.delete_snapshot(new_snapshot['id'])
|
A snapshot can be thought as a backup, stored at users “snapshots” container.
Restore server from snapshot (assume the one created in the previous step).
1 2 | compute_client.create_server(
name='Reserve', image_id=new_snapshot['id'], flavor_id=srv['flavor']['id'])
|
To be safer, download the snapshot to local disk.
1 2 3 4 | obj = new_snapshot['display_name']
storage_client.container = 'snapshots'
with open(obj, 'w+') as f:
storage_client.download_object(obj, f)
|
If the snapshot has been removed from the system, the server can be restored by uploading it from the local copy.
1 2 3 4 5 6 7 8 9 | with open(obj, 'r') as f:
storage_client.upload_object(obj, f)
location = (storage_client.account, storage_client.container, obj)
props = dict(exclude_all_taks=True)
snp = image_client.register('Restored snapshot', location, properties=props)
compute_client.create_server(
name='Reserve', image_id=snp['id'], flavor_id=srv['flavor']['id'])
|
Note
By uploading from a local copy, we must register the snapshot as an image. Use the “exclude_all_taks” to register such images.
Term | Description |
---|---|
network | A public or private network |
sunet | A subnet of a network |
port | A connection between a device (e.g., vm) and a network |
floating_ip | An external IP v4, reserved by the current user for a network |
Public networks are created by the system. Private networks are created and managed by users.
Separate public from private networks.
1 2 3 4 5 6 | public_nets, vpns = [], []
for net in network_client.list_networks(detail=True):
if net['public']:
public_nets.append(net)
else:
vpns.append(net)
|
Create a VPN.
1 2 3 | vpn = network_client.create_network(
type=cyclades.CycladesNetworkClient.network_types[1], name='New Net')
print 'Created VPN with id', vpn['id'], 'with name', vpn['name']
|
Note
The “type” of the network is a Cyclades-specific parameter. To see all network types:
1 | print 'Network types:', cyclades.CycladesNetworkClient.network_types
|
Delete the VPN.
1 | network_client.delete_network(vpn['id'])
|
Find the ID of an IP.
1 2 3 4 5 6 | ip_addr, ip_id = '123.45.67.89', None
for ip in network_client.list_floatingips():
if ip_addr == ip['floating_ip_address']:
ip_id = ip['id']
break
|
Find the server ID from an IP ID.
1 2 3 | ip = network_client.get_floatingip_details(ip_id)
if ip['instance_id']:
print 'IP', ip['floating_ip_address'], 'is used by', ip['instance_id']
|
Reserve an IP.
1 2 | ip = network_client.create_floatingip()
print 'Reserved new IP', ip['floating_ip_address']
|
Note
Reserving an IP means “make it available for use”. A freshly reserved IP is not used by any servers.
Release an IP.
1 2 | network_client.delete_floatingip(ip['id'])
print 'Released IP', ip['floating_ip_address']
|
Attach IP to server, by creating a connection (port) between the server and the network related to the IP.
Note
The “srv” object and the “assert_status” method from an earlier script are used here too.
1 2 3 4 5 6 7 8 9 | port = network_client.create_port(
network_id=ip['floating_network_id'],
device_id=srv['id'],
fixed_ips=[dict(ip_address=ip['floating_ip_address']), ])
print 'Attaching IP', ip['floating_ip_address'], 'to server', port['device_id']
port['status'] = network_client.wait_port(port['id'], port['status'])
assert_status(port['status'], 'ACTIVE')
print '... OK'
|
Detach IP from server.
1 2 3 4 5 6 | network_client.delete_port(port['id'])
print 'Detaching IP', ip['floating_ip_address'], 'from', port['device_id']
port['status'] = network_client.wait_port(port['id'], port['status'])
assert_status(port['status'], 'DELETED')
print '... OK'
|