8.5. Clients reference¶
Kamaki library API consists of clients corresponding to the Synnefo API, which is equivalent to the OpenStack API with some extensions. In some cases, kamaki implements the corresponding OpenStack libraries as separate clients and the Synnefo extensions as class extensions of the former.
The kamaki library API consists of the following clients:
In kamaki.clients.astakos
:
AstakosClient An Identity and Account client for Synnefo API
OriginalAstakosClient The client of the Synnefo astakosclient package
LoggedAstakosClient The original client with kamaki-style logging
CachedAstakosClient Some calls are cached to speed things up
Note
Use AstakosClient
if you are not sure
TO BE COMPLETED
8.5.1. Astakos / Identity¶
Synnefo API: https://www.synnefo.org/docs/synnefo/latest/identity-api-guide.html
The core functionality of this module is to authenticate a user and provide user data (e.g., email, unique user id)
8.5.1.1. Authenticate user¶
Example: Authenticate user, get name and uuid
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | from kamaki.clients.astakos import AstakosClient, ClientError
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
try:
user = astakos.authenticate()
except ClientError as ce:
if ce in (401, ):
print "Authentication failed, {0}".format(ce)
exit(0)
raise
user_info = user['access']['user']
print "Authentication was successful for {name}, with uuid {uuid}".format(
name=user_info['name'], uuid=user_info['id'])
|
Note
the authenticate
method returns a dict, which is defined by the
Synnefo API (not by kamaki)
8.5.2. Astakos / Resources and Quotas¶
Synnefo API: https://www.synnefo.org/docs/synnefo/latest/api-guide.html#resource-and-quota-service-api-astakos
This API provides information on available resources, resource usage and quota limits.
8.5.2.1. Resource quotas¶
Example: Resource usage and limits for number of VMs and IPs
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | from kamaki.clients.astakos import AstakosClient
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
# Get user uuid
user = astakos.authenticate()
uuid = user['access']['user']['id']
# Get resources assigned on my personal (system) project
all_resources = astakos.get_quotas()
my_resources = all_resources[uuid]
# Print usage and limits for VMs and IPs
vms = my_resources["cyclades.vm"]
ips = my_resources["cyclades.floating_ip"]
print "You are using {vms}/{vm_limit} VMs and {ips}/{ip_limit} IPs".format(
vms=vms["usage"], vm_limit=vms["limit"],
ips=ips["usage"], ip_limit=ips["limit"])
|
Note
Quotas are defined by projects (see next section). Every user is
member to a personal project (the “system” project) which is identified by
the uuid of the user, but they may draw resources from other projects as
well. In this script we only got the quota information related to the system
project and we did that with this line of code
my_resources = all_resources[uuid]
8.5.3. Astakos / Projects¶
Synnefo API: https://www.synnefo.org/docs/synnefo/latest/api-guide.html#project-service-api
The relation between projects, users and resources:
cloud resources: VMs, CPUs, RAM, Volumes, IPs, VPNs, Storage space
a cloud user --- is member to --- projects
a cloud resource --- must be registered to --- a project
A user creates a resource: registers a resource to a project he is member of
What information is found in a project:
- members: cloud users who can use the project resources
- project limit: usage limits per resource for the whole project
- member limit: usage limits per resource per cloud user
- usage: current usage per resource per cloud user
Note
By default, every user has a personal (system) project. By default when a user creates a resource, it is registered to this project, except if they explicitly request to register a resource to another project.
8.5.3.1. Query my projects¶
Example: Get information for all projects I am member to
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | from kamaki.clients.astakos import AstakosClient
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
# Get user uuid
user = astakos.authenticate()
uuid = user['access']['user']['id']
# Get all projects information
projects = astakos.get_projects()
for project in projects:
print "Project {name} ({uuid})\n\t{description}\n\t{url}".format(
name=project["name"], uuid=project["id"],
description=project["description"], url=project["homepage"])
|
The results should look like this:
system:a1234567-a890-1234-56ae-78f90bb1c2db (a1234567-a890-1234-56ae-78f90bb1c2db)
System project for user user@example.com
CS333 lab assignments (a9f87654-3af2-1e09-8765-43a2df1098765)
Virtual clusters for CS333 assignments
https://university.example.com/courses/cs333
8.5.3.2. Quotas per project¶
Example: Get usage and total limits per resource per project
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | from kamaki.clients.astakos import AstakosClient
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
# Get user uuid
user = astakos.authenticate()
uuid = user['access']['user']['id']
# Get resources assigned on my personal (system) project
all_resources = astakos.get_quotas()
for project, resources in all_resources.items():
print "For project with id {project_id}".format(project_id=project)
for resource, values in resources.items():
print " {resource}: {used}/{limit}".format(
resource=resource, used=values["usage"], limit=values["limit"])
|
The results should look like this:
a1234567-a890-1234-56ae-78f90bb1c2db
cyclades.cpu: 1/2
cyclades.disk: 40/40
cyclades.floating_ip: 1/1
cyclades.network.private: 0/2
cyclades.ram: 2147483648/2147483648
cyclades.vm: 1/2
pithos.diskspace: 20522192022/20522192022
a9f87654-3af2-1e09-8765-43a2df1098765
cyclades.cpu: 4/8
cyclades.disk: 80/120
cyclades.floating_ip: 1/4
cyclades.network.private: 1/5
cyclades.ram: 4294967296/53687091200
cyclades.vm: 3/4
pithos.diskspace: 20522192022/53687091200
8.5.3.3. Allocate resource to a project¶
Example: Create an IP assigned to a specific project
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesNetworkClient
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
# I already know the project id
project = "a9f87654-3af2-1e09-8765-43a2df1098765"
# Initialize cyclades_network client
service_type = CycladesNetworkClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
cyclades_network = CycladesNetworkClient(endpoint, TOKEN)
# Create a new floating ip
ip = cyclades_network.create_floatingip(project_id=project)
print "A new IP {0} is reserved for you".format(ip["floating_ip_address"])
|
Note
All “create_something” methods take an optional “project_id” argument which instructs Synnefo to register this resource to a specific project
8.5.3.4. Reassign resource to another project¶
Example: Reassign a storage container to different project
In the following scenario we assume that course_container
is a storage
container on Pithos, which is assigned to the system (personal) project and
suffers from low quota limits. Fortunately, we have an extra project with enough
storage available. We will reassign the container to benefit from this option.
We will check the quota limits of the project of this container and, if they are used up, we will reassign it to a different project.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
# Our data
container_name = "course_container"
user = astakos.authenticate()
uuid = user["access"]["user"]["id"]
# Initialize a Pithos client
service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN, uuid, container_name)
# To what project is this container assigned to?
container = pithos.get_container_info(container_name)
container_project = container["x-container-policy-project"]
# Get quota info
quotas = astakos.get_quotas()
container_quotas = quotas[container_project]["pithos.diskspace"]
usage, limit = container_quotas["usage"], container_quotas["limit"]
if usage < limit:
print "Quotas for container {0} are OK".format(container_name)
else:
# We need to reassign to another project
new_project = "a9f87654-3af2-1e09-8765-43a2df1098765"
pithos.reassign_container(project_id=new_project)
print "Container {name} is reassigned to project {id}".format(
name=container_name, id=new_project)
|
Note
All quotable resources can be reassigned to different projects, including composite resources (aka: depended on other cloud resources) like VMs.
8.5.4. Cyclades / Compute¶
Synnefo API: https://www.synnefo.org/docs/synnefo/latest/compute-api-guide.html
A server or VM (Virtual Machine) is a complex resource: you need other resources to built it, namely CPU cores, RAM memory, Volume space and, optionally, VPNs and IPs.
8.5.4.1. List server quotas¶
Example: Check server-related resources and report to user
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | from kamaki.clients.astakos import AstakosClient
# Initialize Astakos
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
# Check quotas
total_quotas = astakos.get_quotas()
resources = (
"cyclades.vm", "cyclades.cpu", "cyclades.ram", "cyclades.disk",
"cyclades.network.private", "cyclades.floating_ip")
for project, quotas in total_quotas.items():
print "Project {0}".format(project)
for r in resources:
usage, limit = quotas[r]["usage"], quotas[r]["limit"]
if usage < limit:
print "\t{0} ... OK".format(r)
else:
print "\t{0}: ... EXCEEDED".format(r)
|
8.5.4.2. Create server¶
Example: Create a server with an IP and a server without networking (assume the IP is reserved for use by current user)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesComputeClient
# Initialize Astakos
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
# Initialize CycladesComputeClient
service_type = CycladesComputeClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
compute = CycladesComputeClient(endpoint, TOKEN)
# Create a public server (with IP)
name = "My public server"
flavor = "420"
image = "image-id-for-debian"
IP = "123.45.67.89"
network_id_for_this_ip = 204
networks = dict(uuid=network_id_for_this_ip, fixed_ip=IP)
project = "a1234567-a890-1234-56ae-78f90bb1c2db"
public_server = compute.create_server(
name, flavor, image, networks=networks, project=project)
# Create a private server (without networking)
name = "My private server"
networks = []
private_server = compute.create_server(
name, flavor, image, networks=networks, project=project)
|
Note
The “networks” parameter sets the connectivity attributes of the new
server. If it is None (default), networking is configured according to the
default policy of the cloud (e.g., automatically assign the first available
public IP). To set up a server without networking: networks=[]
.
8.5.4.3. Wait server to built¶
Example: Create a new server (default settings) and wait until it is built. Then, print the server id and its IP (assume that the default networking policy of the cloud is to automatically set an IP to every new server).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesComputeClient
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
service_type = CycladesComputeClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
compute = CycladesComputeClient(endpoint, TOKEN)
# Create a server (automatic IP)
name = "My public server"
flavor = "420"
image = "image-id-for-debian"
project = "a1234567-a890-1234-56ae-78f90bb1c2db"
server = compute.create_server(name, flavor, image, project=project)
print "Server status is {0}".server["status"]
new_status = compute.wait_server_until(server["id"], "ACTIVE")
if new_status != "ACTIVE":
print "Waiting for server to build: time out..., server is in {0}".format(
new_status)
else:
# Find ip
nics = filter(lambda nic: bool(nic["ipv4"]), server["attachments"])
ip = nics[0]["ipv4"]
# Create server
print "Server {id} is now {status}\n\tName: {name}\n\tIP: {ip}".format(
id=server["id"], status=server["status"], name=server["name"], ip=ip)
|
Note
The wait_server_while
and wait_server_until
methods work for
all valid server states (ACTIVE, BUILD, STOPPED, ERROR, etc.) and
can be used to block a program under some conditions. These blockers are
based on the kamaki.clients.wait
, which blocks for as long as a
user-provided method returns true.
8.5.4.4. Query images and flavors¶
Example: Find the appropriate image and flavor for a server
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesComputeClient
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
service_type = CycladesComputeClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
compute = CycladesComputeClient(endpoint, TOKEN)
# Find flavor with 2 cores, 20GB disk and 2048MB of ram
pick_flavor = lambda flavor: all(
flavor["vcpus"] == 2,
flavor["disk"] == 20,
flavor["ram"] == 2048
)
all_flavors = compute.list_flavors(detail=True)
flavors = filter(pick_flavor, all_flavors)
# Find images with debian in their name
pick_image = lambda image: "debian" in image["name"].lower()
all_images = compute.list_images(detail=True)
images = filter(pick_image, all_images)
# Show results
flavor_ids = '\n\t'.join([f["id"] for f in flavors])
print "{num} flavors match\n\t{ids}".format(num=len(flavors), ids=flavor_ids)
image_ids = '\n\t'.join([i["id"] for i in images])
print "{num} images match: {ids}".format(num=len(images), ids=image_ids)
|
8.5.4.5. Reboot server¶
Example: Reboot a server.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesComputeClient
# Initialize Astakos
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
# Initialize CycladesComputeClient
service_type = CycladesComputeClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
compute = CycladesComputeClient(endpoint, TOKEN)
# Get server current status
server_id = "my-server-id"
server = compute.get_server_details(server_id)
status = server["status"]
# Shutdown server
if status == "ACTIVE":
compute.server_reboot(server_id)
elif status == "STOPPED":
compute.server_start(server_id)
new_status = compute.wait_sever_while(server_id, "REBOOT")
print "Server {id} is now {status}".format(id=server_id, status=new_status)
|
Note
Similarly you can start_server
, shutdown_server
as well as
resize_server
and delete_server
.
8.5.4.6. Reassign and resize a server¶
- Example: We need to make our server more powerful, but it’s assigned to a
- project which is out of resources. We will shutdown the server, reassign it to another project and resize it to another flavor.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesComputeClient
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
service_type = CycladesComputeClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
compute = CycladesComputeClient(endpoint, TOKEN)
# Data
server_id = "my-server-id"
new_flavor = "new-flavor-id-able-in-resources"
new_project = "a9f87654-3af2-1e09-8765-43a2df1098765"
# Shutdown server
compute.shutdown_server(server_id)
compute.wait_server_until(server_id, "STOPPED")
# Reassign and resize
compute.reassign_server(server_id, new_project)
compute.resize_server(server_id, new_flavor)
# Start server
compute.start_server(server_id)
compute.wait_server_until(server_id, "ACTIVE")
|
Note
Servers must be stopped in order to resize or reassign it.
8.5.5. Cyclades / Network¶
Synnefo API: https://www.synnefo.org/docs/synnefo/latest/network-api-guide.html
The Synnefo approach to the Network API diverges from the OpenStack semantics. Check the Synnefo documentation for more details.
8.5.5.1. Create private network¶
Example: Create a new private network between two servers.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesNetworkClient
# Initialize Astakos
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
# Initialize CycladesComputeClient
service_type = CycladesNetworkClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
network = CycladesNetworkClient(endpoint, TOKEN)
# Servers
server_1 = "id-for-server-1"
server_2 = "id-for-server-2"
# Create a new network
type_ = CycladesNetworkClient.types[0]
mynet = network.create_network(type_, name="My Network")
# Connect servers to network
port_1 = network.create_port(mynet["id"], device_id=server_1)
port_2 = network.create_port(mynet["id"], device_id=server_2)
# Wait until ready
network.wait_port_until(port_1["id"], "ACTIVE")
network.wait_port_until(port_2["id"], "ACTIVE")
|
Note
In Synnefo, ports are the connections between a network and a server.
8.5.5.2. Reserve IP¶
Example: Check if there are free IPs, reserve one if not and use it with a server.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import CycladesNetworkClient
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
service_type = CycladesNetworkClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
network = CycladesNetworkClient(endpoint, TOKEN)
# Data
server_id = "id-for-server-1"
# Check for unused IPs
unused_ips = filter(lambda ip: not ip["port_id"], network.list_floatingips())
# Reserve an IP
ip = unused_ips[0] if unused_ips else network.create_floatingip()
# Retrieve network id and IPv4 address
net = ip["floating_netowrk_id"]
fixed_ips = [dict(ip_address=ip["floating_ip_address"]), ]
# Connect IP to server
port = network.create_port(net["id"], device_id=server_id, fixed_ips=fixed_ips)
# Wait until ready
network.wait_port_until(port["id"], "ACTIVE")
|
Note
IPs are connected to networks, which are connected to servers.
8.5.5.3. Create cluster¶
Example: Create a cluster of three servers, where only one has a public IP.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import (
CycladesComputeClient, CycladesNetworkClient)
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
service_type = CycladesComputeClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
compute = CycladesComputeClient(endpoint, TOKEN)
service_type = CycladesNetworkClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
network = CycladesNetworkClient(endpoint, TOKEN)
# Create VPN and reserve an IP
type_ = CycladesNetworkClient.types[0]
vpn = network.create_network(type_, name="Cluster Network")
unused_ips = filter(lambda ip: not ip["port_id"], network.list_floatingips())
ip = unused_ips[0] if unused_ips else network.create_floatingip()
ip_net = ip["floating_network_id"]
# Server data
flavor = 420
image = "image-id-for-a-debian-image"
# Create nodes
networks = [dict(uuid=vpn["id"]), ]
node_1 = compute.create_server("Node 1", flavor, image, networks=networks)
node_2 = compute.create_server("Node 2", flavor, image, networks=networks)
# Create gateway
networks.append(dict(uuid=ip_net, fixed_ip=ip["floating_ip_address"]))
gateway = compute.create_server("Gateway", flavor, image, networks=networks)
# Wait servers to get ready
compute.wait_server_until(node_1["id"], "ACTIVE")
compute.wait_server_until(node_2["id"], "ACTIVE")
compute.wait_server_until(gateway["id"], "ACTIVE")
|
8.5.6. Cyclades / BlockStorage¶
Synnefo API: https://www.synnefo.org/docs/synnefo/latest/blockstorage-api-guide.html
8.5.6.1. Unplug and plug volume¶
Example: Create a volume for server_1, then unplug it and plug it on server_2, as you would do with a USB stick.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.cyclades import (
CycladesBlockStorageClient, CycladesComputeClient)
# Initialize Astakos
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
# Initialize CycladesComputeClient
service_type = CycladesComputeClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
compute = CycladesComputeClient(endpoint, TOKEN)
# Servers
server_id_1, server_id_2 = "id-for-sever-1", "id-for-sever-2"
# Initialize CycladesBlockStorageClient
service_type = CycladesBlockStorageClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
blockstorage = CycladesBlockStorageClient(endpoint, TOKEN)
# Create new volume on server_1
size_ = 20 # in GB
name = "USB stick"
usb = blockstorage.create_volume(size_, name, server_id=server_id_1)
blockstorage.wait_volume_until(usb["id"], "in_use")
# Unplug and plug to the other server
compute.detach_volume(server_id_1, usb["id"])
blockstorage.wait_volume_while(usb["id"], "in_use")
compute.attach_volume(server_id_2, usb["id"])
blockstorage.wait_volume_until(usb["id"], "in_use")
|
8.5.7. Image¶
Synnefo API: https://www.synnefo.org/docs/synnefo/latest/image-api-guide.html
In Synnefo, an image is loaded as a file to the storage service (Pithos+), and then is registered to the image service (Plankton). The image location is unique and can be used as an image identifier.
Image location formats:
pithos://<user_uuid>/<container>/<object path>
e.g., pithos://user-uuid/images/debian_base.diskdump
or, if the user uuid os implied
/<container>/<object path>
e.g., /images/debian_base.diskdump
8.5.7.1. Register¶
Example: Register the image file my-image.diskdump
, currently stored
locally. It will be uploaded to images
.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | import json
from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient
from kamaki.clients.image import ImageClient
# Initliaze astakos client
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
user = astakos.authenticate()
uuid = user["access"]["user"]["id"]
# Initliaze Pithos
service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN)
pithos.account = uuid
# Initialize Image
service_type = ImageClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
image = ImageClient(endpoint, TOKEN)
# Our data
local = "my-image.diskdump"
local_meta = "my-image.diskdump.meta"
with open(local_meta) as f:
meta = json.load(f)
# Upload the image and meta files
pithos.container = "images"
with open(local) as f:
pithos.upload_object(local, f)
with open(local_meta) as f:
pithos.upload_object(local_meta, f)
# Register image
name, location = meta.pop("name"), meta.pop("location")
properties = meta.pop("properties")
image.register(name, location, properties=properties, params=meta)
|
It is a common practice to keep the image registration details in a json meta file (e.g., to register images in the future). This metafile is typically uploaded along with the image.
kamaki file cat /images/my-image.diskdump.meta
{
"name": "Debian Base With Extras",
"checksum": "3cb03556ec971f...e8dd6190443b560cb7",
"updated-at": "2013-06-19 08:01:00",
"created-at": "2013-06-19 08:00:22",
"properties": {
"OS": "linux",
"USER": "root"
},
"location": "pithos://user-uuid/images/my-image.diskdump",
"is-public": "False",
"owner": "user-uuid",
"disk-format": "diskdump",
"size": "903471104",
"deleted-at": "",
"container-format": "bare"
}
8.5.7.2. List¶
Example: List the names and Pithos locations of the images registered by me
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.image import ImageClient
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
user = astakos.authenticate()
uuid = user["access"]["user"]["id"]
service_type = ImageClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
image = ImageClient(endpoint, TOKEN)
# Get all images owned/registered by me
images = filter(lambda img: img["owner"] == uuid, image.list_public())
print "My images:\n"
for i in images:
print "\t{name} ({location})".format(
name=i["name"], location=i["location"])
|
8.5.7.3. Unresgister¶
Example: Unregister and delete an image.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient
from kamaki.clients.image import ImageClient
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
user = astakos.authenticate()
uuid = user["access"]["user"]["id"]
service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN)
pithos.account = uuid
service_type = ImageClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
image = ImageClient(endpoint, TOKEN)
# Find the image by id
image_id = "my-image-id"
my_image = image.get_meta(image_id)
# Check if it is my image
if my_image["owner"] == uuid:
image.unregister(image_id)
# Delete the image files
pithos.container = "images"
separator = "{uuid}/{container}/".format(
uuid=uuid, container=pithos.container)
_, location = my_image["location"].split(separator)
meta_object = "{0}.meta".format(location)
pithos.del_object(location)
pithos.del_object(meta_object)
else:
print "This image wasn't registered by me"
|
Note
Unregistering an image does not delete the image dump from pithos. In order to do that, you need to have the appropriate permissions (aka, the image file must by stored on your Pithos account), so that you can delete it as a file.
8.5.8. Pithos¶
Synnefo API: https://www.synnefo.org/docs/synnefo/latest/object-api-guide.html Pithos+ is the storage service of Synnefo.
Each user has their own storage space, organized in containers. Each container contains objects. In most cases we can think of objects as files, but in reality they are not the same thing. Typically, it is the responsibility of the application to simulate the functionality of folders and files, if they need it.
Here is an example, where the containers are pithos
, images
, music
and trash
:
user-uuid
pithos
myfile.txt
myfolder/
myfolder/anotherfile.txt
my-linux-distro.diskdump
images
debian-stable.disckdump
my-special-image.diskdump
music
The Beatles - White Album/
The Beatles - White Album/Back in the U.S.S.R.
BoC - Music has the right to children/
BoC - Music has the right to children/Wildlife Analysis
BoC - Music has the right to children/An eagle in your mind
trash
my deleted folder/
my deleted folder/some old file.txt
my deleted folder/removed by accident.png
Quotas are applied at project level. Each container is registered to a project (by default, the personal/system project of the owner). Objects (“files”) inherit the project policy of the container they are in.
8.5.8.1. Initialize pithos client¶
Example: Initialize a pithos client to handle the objects in the container
pithos
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient
# Initliaze astakos client
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
# Initliaze pithos
service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN)
# Set user UUID and Container (optional)
user = astakos.authenticate()
uuid = user["access"]["user"]["id"]
pithos.account = uuid
pithos.container = "pithos"
|
Note
To access the objects of another user, set the account
parameter
to their uuid and Pithos will have access to the objects the other user
allows you to see or edit.
8.5.8.2. List and information¶
Example Recursively list the contents of all my containers.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient
# initliaze astakos client
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
# Initliaze pithos
service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN)
# Set user UUID and Container (optional)
user = astakos.authenticate()
uuid = user["access"]["user"]["id"]
pithos.account = uuid
containers = pithos.list_containers()
for container in containers:
pithos.container = container["name"]
project = container["x_container_policy"]["project"]
print "Listing contents of {container} (project: {project})".format(
container=pithos.container, project=project)
for object_ in pithos.list_objects():
name = object_["name"]
size = object_["bytes"]
type_ = object_["content_type"]
print "\t{name} \t{type_} \t{size} bytes".format(
name=name, size=size, type_=type_)
|
The results should look like this:
Listing contents of pithos (project: a1234567-a890-1234-56ae-78f90bb1c2db)
myfile.txt text/plain 202 bytes
myfolder/ application/directory 0 bytes
myfolder/anotherfile.txt text/plain 333 bytes
my-linux-distro.diskdump applcation/octet-stream 539427293 bytes
Listing contents of images (project: a9f87654-3af2-1e09-8765-43a2df1098765)
debian-stable.disckdump application/octet-stream 309427093 bytes
my-special-image.diskdump applcation/octet-stream 339427293 bytes
Listing contents of music (project: a9f87654-3af2-1e09-8765-43a2df1098765)
The Beatles - White Album/ application/directory 0 bytes
The Beatles - White Album/Back in the U.S.S.R.mp3 media/mpeg 3442135 bytes
BoC - Music has the right to children/ application/directory 0 bytes
BoC - Music has the right to children/Wildlife Analysis.mp3 media/mpeg 4442135 bytes
BoC - Music has the right to children/An eagle in your mind.mp3 media/mpeg 23442135 bytes
Listing contents of trash (project: a1234567-a890-1234-56ae-78f90bb1c2db)
my deleted folder/ application/directory 0 bytes
my deleted folder/some old file.txt text/plain 10 bytes
my deleted folder/removed by accident.png image/png 20 bytes
Note
In the above example, half of the projects (pithos, trash) are registered to the user’s system (personal) project, while the rest ( images, music) at another project, probably one offering more storage resources.
8.5.8.3. Upload and download¶
- Example: Download
my-linux-distro.diskdump
from container “pithos” to a - local file as
local.diskdump
and upload it to container “images” asrecovered.diskdump
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient
# Initliaze astakos client
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN)
user = astakos.authenticate()
uuid = user["access"]["user"]["id"]
pithos.account = uuid
# Download from container "pithos"
pithos.container = "pithos"
source = "my-linux-distro.diskdump"
target = "local.diskdump"
with open(target, "rb+") as f:
pithos.download_object(source, f)
# Upload to container "images"
pithos.container = "images"
with open(target) as f:
pithos.upload_object(source, f)
|
Note
The file is now stored at three locations: the container “pithos”, the container “images” and, with a different name, at the local hard disk.
Note
The upload_object
and download_object
methods are optimized in
many ways: they feature dynamic simultaneous connections and automatic
resume. In this case for instance, the data of the uploaded file will not
be uploaded, as they are already on the server.
8.5.8.4. Move / Copy / Delete¶
- Example: Move
/pithos/my-linux-distro.diskdump
totrash
, delete /images/my-pithos-distro.diskdump
and then copy it from trash.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN)
user = astakos.authenticate()
uuid = user["access"]["user"]["id"]
pithos.account = uuid
# Move to "trash"
object_ = "my-linux-distro.diskdump"
pithos.move_object("pithos", object_, "trash")
# Delete from "images"
pithos.container = "images"
pithos.del_object(object_)
# Copy from "trash" to "images"
new_object = "recovered.diskdump"
pithos.copy_object("trash", object_, "images", new_object)
|
This is the status after the execution of the script above:
user-uuid
pithos
myfile.txt
myfolder/
myfolder/anotherfile.txt
images
debian-stable.disckdump
my-special-image.diskdump
recovered.diskdump
music
The Beatles - White Album/
The Beatles - White Album/Back in the U.S.S.R.
BoC - Music has the right to children/
BoC - Music has the right to children/Wildlife Analysis
BoC - Music has the right to children/An eagle in your mind
trash
my deleted folder/
my deleted folder/some old file.txt
my deleted folder/removed by accident.png
my-linux-distro.diskdump
8.5.8.5. Reassign container¶
Example: Reassign container “images” to the same project as the one of “pithos”.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | from kamaki.clients.astakos import AstakosClient
from kamaki.clients.pithos import PithosClient
AUTHENTICATION_URL = "https://astakos.example.com/identity/v2.0"
TOKEN = "User-Token"
astakos = AstakosClient(AUTHENTICATION_URL, TOKEN)
service_type = PithosClient.service_type
endpoint = astakos.get_endpoint_url(service_type)
pithos = PithosClient(endpoint, TOKEN)
user = astakos.authenticate()
uuid = user["access"]["user"]["id"]
pithos.account = uuid
# Get the project containers we care for
containers = filter(
lambda c: c["name"] in ("pithos", "images"),
pithos.list_containers())
# Construct dict of the form {CONTAINER_NAME: PROJECT_ID, ...}
projects = dict([(
c["name"],
c["x_container_policy"]["project"]) for c in containers])
# Check projects and reassign if needed
if projects["pithos"] != projects["images"]:
pithos.container = "images"
pithos.reassign_container(projects["pithos"])
|