Using Object Storage Clients¶

Standard client tools can be used to browse objects in VPSA Object Storage. This section will help configuring Object Storage Client Tools to work against VPSA Object Storage. In order to access the VPSA Object Storage the client tool must be configured with the user’s authentication credentials.

The VPSA Object Storage support two API interfaces:

AWS S3 API
Openstack Swift API

The required parameters can be found in the Object Storage User Information page. Information for the user currently logged in to the Object Storage management interface displayed by clicking the user name on the management interface’s upper right corner.

AWS S3 Compatible clients¶

Supported S3 APIs¶

The VPSA Object Storage is utilizing Openstack Swift’s S3 Middleware. As S3 is an AWS product, it includes some features that are AWS oriented and are outside of the scope of Zadara’s Object Storage offering.

Zadara supports the following S3 operations:

Object operations¶

Bucket operations¶

Authentication information¶

For Object Storage connectivity, it is required to gather the following information from the VPSA Object Storage management UI:

VPSA Object Storage Endpoint
VPSA Object Storage region.
S3 API Access Key/Secret Key

In the VPSA Object Storage GUI, navigate to the User Information section (top right corner, by clicking the logged in username).

s3-auth-details

S3 Browser¶

S3 Browser can be used to administrate and perform object operations against Zadara’s VPSA Object Storage. The account information in S3 Browser should be configured according to the following example (S3 Compatible Storage):

s3-browser-create-account

Once the Endpoint and authentication details are configured properly, click on the Advanced S3-compatible storage settings

In the advanced settings select the following:

Signature version - Signature V4
Addressing model - Path style
Override storage regions - specify the VPSA Object Storage region name; the format is Region Name=<region name>.

Close and save the account information.

Note

S3 Browser client is hard-coded to use us-east-1 as the default region, In order to use Object Storage v4 signatures, ensure the same region value is configured in your VPSA Object Storage or override the default S3Browser region name in the Advanced Settings options.

S3cmd¶

The credentials can be retrieved from the VPSA Object Storage logged in “User Information” properties.

/etc/.s3cfg

[default]
access_key = <S3 Access Key>
secret_key = <S3 Secret Key>
host_base = vsa-00000001-cloud-01.zadara.com
host_bucket = vsa-00000001-cloud-01.zadara.com
use_https = True

Note

access_key is the user S3 Access Key
secret_key is the user S3 Secret Key
host_base is the HTTPS path to the VPSA Object Storage being accessed

AWS Command Line Interface¶

Update the default/create new profile for the VPSA Object Storage within aws configuration file.

~/.aws/config

[profile zadara]
s3 =
    signature_version = s3v4

Note

It is possible to use both AWS v4/v2 signatures with S3-compatible storage such as Zadara VPSA Object Storage.

~/.aws/credentials

[zadara]
aws_access_key_id = <S3 Access Key>
aws_secret_access_key = <S3 Secret Key>

The credentials can be retrieved from the VPSA Object Storage logged in “User Information” properties.

Example of usage:

$ aws s3 --profile=zadara --endpoint-url=https://vsa-00000001-cloud-01.zadara.com --region=us-east-1 ls s3://zadara-test

2018-04-01 19:00 mytestfile1
2018-04-01 19:10 mytestfile2
2018-04-01 19:20 mytestfile3

Note

profile is the name of the credentials and config profile specified above (in this case, “zadara”)
endpoint-url is the HTTPS path to the VPSA Object Storage being accessed
region should match the Region defined in the VPSA Object Storage settings page (Zadara default: us-east-1)

Common operations examples¶

Creating a pre-signed URL - this allows anyone who receives the pre-signed URL to retrieve the object with HTTP GET request. The operation uses the S3 credentials and region field to generate the pre-signed URL.
```
$ aws s3 presign --profile <AWS CLI profile name> --endpoint \
https://<object storage api endpoint> \
s3://<container/bucket name>/<object name> --expires-in <expiration in seconds>
```
For more information please refer to the official AWS CLI Command Reference.

boto3 python library¶

Update the default/create new profile for the VPSA Object Storage within aws configuration file.

~/.aws/config

[profile zadara]
s3 =
    signature_version = s3v4

Note

It is possible to use both AWS v4/v2 signatures with S3-compatible storage such as Zadara VPSA Object Storage.

~/.aws/credentials

[zadara]
aws_access_key_id = <S3 Access Key>
aws_secret_access_key = <S3 Secret Key>

The credentials can be retrieved from the VPSA Object Storage logged in “User Information” properties.

In your python code:

#!/usr/bin/env python

import boto3

session = boto3.session.Session(profile_name='zadara')

s3_client = session.client(
    service_name='s3',
    region_name='us-east-1',
    endpoint_url='https://vsa-00000001-cloud-01.zadara.com',
)

print('Buckets')
print(s3_client.list_buckets())

print('')

print('Objects')
print(s3_client.list_objects(Bucket='test'))

Note

profile_name is the name of the credentials and config profile specified above (in this case, “zadara”)
endpoint_url is the HTTPS path to the VPSA Object Storage being accessed
region should match the Region defined in the VPSA Object Storage settings page (Zadara default: us-east-1)

AWS S3 Java SDK (aws-java-sdk)¶

AWS Provides a comprehensive S3 Java SDK that can be used with Zadara’s VPSA Object Storage. Getting started guide is available in Zadara’s Support Knowledge Base article - How to use AWS S3 Java SDK with VPSA Object Storage.

AWS S3 PHP SDK (aws-sdk-php)¶

AWS Provides a comprehensive S3 PHP SDK that can be used with Zadara’s VPSA Object Storage. Getting started guide is available in Zadara’s Support Knowledge Base article - How to use AWS S3 PHP SDK with VPSA Object Storage.

AWS S3 JavaScript SDK (aws-sdk)¶

AWS Provides a comprehensive S3 JavaScript SDK that can be used with Zadara’s VPSA Object Storage. Getting started guide is available in Zadara’s Support Knowledge Base article - How to use AWS S3 JavaScript SDK with VPSA Object Storage.

Openstack Swift Interface¶

The management interface generates a new Swift API token upon login. This means that if you logout and login again you’ll notice a new token. The token presented by the management interface is always the latest and valid to be used.

The API tokens created by the management interface are generated based on the Object Storage global configuration for token validity (default: 24 hours).

Example of validating an API token using the CLI:

# Get two consecutive API tokens from the management interface and store it
$ TOKEN1=gAAAAABiuwu4P55M2V...
$ TOKEN2=gAAAAABiuwvc8BEYc8...

$ curl -X GET -H "Content-Type: application/json" \
  -H "X-Access-Key: $TOKEN1"  \
  "https://<object storage endpoint>:8443/api/zios/accounts/AUTH_<account ID>/users.json"

{"response":{"users":[],"count":0,"status":0}}%

curl -X GET -H "Content-Type: application/json" \
 -H "X-Access-Key: $TOKEN2" \
 "https://<object storage endpoint>:8443/api/zios/accounts/AUTH_<account ID>/users.json"

{"response":{"users":[],"count":0,"status":0}}%

# We can validate these tokens using the Openstack Keystone auth service as well:

$ curl -s   -H "X-Subject-Token: $TOKEN2" -H "X-Auth-Token: $TOKEN1"  \
  "https://<object storage endpoint>:5000/v3/auth/tokens" | python3 -m json.tool
{
    "token": {
       "is_domain": false,
      "methods": [
         "password"
      ],
      "roles": [
         {
            "id": "fedeff6db6df47959e96d8dd33963cfe",
            "name": "zios_admin"
         }
      ],
      "expires_at": "2022-06-29T14:10:36.000000Z",
 ....
      "issued_at": "2022-06-28T14:10:36.000000Z"
   }
}

Important

By default, the API token is valid for 24 hours. the preferred option to identify/renew the API token via an API call is to use a Keystone authentication request and not using VPSA Object Storage command indicated in the Zadara VPSA Object Storage REST API Guide. Example for authentication against the Keystone service is provided in the next section.

cURL (swift API)¶

cURL can be used for Object Storage operations. The connectivity information is available in the User Information view.

curl_account_urls

In this example, we will use the Front End Network Account URL or Public API Network Account URL, and API Token in order to create a new container:

$ curl -H "x-auth-token: <user_token>" -X PUT <account_url>/test-bucket/

For example:

$ URL=<Front End Network Account URL or Public API Network Account URL>
$ TOKEN=<MYAPI TOKEN>
$ curl -H "x-auth-token: $TOKEN" -X PUT $URL/test-bucket/

The following example describes how to get the token programmatically using the Swift API:

$ curl -i -H "Content-Type: application/json" \
-d '{ "auth": \
{ "identity": { "methods": ["password"], "password": \
{ "user": {"name": "<USERNAME>", "domain": { "id": "default" }, \
"password": "<USER PASSWORD>" }} }, "scope": { "project": \
{ "name": "<ACCOUNT_NAME>", "domain": { "id": "default" } } } } }' \
"https://vsa-00000001-mycloud-01.zadara.com:5000/v3/auth/tokens" ;

and use the returned token for the subsequent API calls.

HTTP/1.1 201 Created
Date: Thu, 19 Nov 2020 16:05:28 GMT
Server: Apache/2.4.29 (Ubuntu)
Content-Length: 1114
X-Subject-Token: gAAAAABftpfIAiuo2tRZZP8VVtomU1knVG7xNUONV4b2u....

Additional examples of using the Openstack Swift API can be found at the Openstack Swift API documentation

Cloudberry Explorer for OpenStack (v3 authentication)¶

Use the logged-in User Information properties to set the authentication fields of Cloudberry Explorer

CyberDuck¶

Cyberduck version: 7.7.1 (33788)

Cyberduck client support “Openstack Swift (Keystone 3)” API interface.

Use the logged-in User Information properties to set the authentication field of CyberDuck client.

Server - the VPSA Object Storage v3 Auth Endpoint.
Port - 5000
Project:Domain:Username - <VPSA Object Storage Account>:default:<VPSA Object Storage Username>

cyber-duck-config