Using Object Storage Clients

Standard client tools can be used to browse objects in the Object Storage. This section will help configuring Object Storage Client Tools to work against Object Storage. In order to access the Object Storage the client tool must be configured with the user’s authentication credentials.

The Object Storage support two API interfaces:

  1. AWS S3 API

  2. Openstack Swift API

The required parameters can be found in the Object Storage User Information page. Information for the user currently logged in to the Object Storage management interface displayed by clicking the user name on the management interface’s upper right corner.

ngos-user-info

AWS S3 Compatible clients

Supported S3 APIs

The Object Storage is utilizing Openstack Swift’s S3 Middleware. As S3 is an AWS product, it includes some features that are AWS oriented and are outside of the scope of Zadara’s Object Storage offering.

Zadara supports the following S3 operations:

Object operations

Bucket operations

Authentication information

For Object Storage connectivity, it is required to gather the following information from the Object Storage management UI:

  1. Object Storage Endpoint - an endpoint entry will be available according to to the networking layout of the Object Storage (i.e. for Object Storage with a Public IP interface we’ll have a record matching to the public network interfaces)

  2. Object Storage region.

  3. S3 API Access Key/Secret Key

In the Object Storage GUI, navigate to the User Information section (top right corner, by clicking the logged in username).

ngos-user-info-full

S3 Browser

S3 Browser can be used to administrate and perform object operations against Zadara’s Object Storage. The account information in S3 Browser should be configured according to the following example (S3 Compatible Storage):

s3-browser-create-account

Once the Endpoint and authentication details are configured properly, click on the Advanced S3-compatible storage settings

In the advanced settings select the following:

  1. Signature version - Signature V4

  2. Addressing model - Path style

  3. Override storage regions - specify the Object Storage region name; the format is Region Name=<region name>.

Close and save the account information.

Note

S3 Browser client is hard-coded to use us-east-1 as the default region, In order to use Object Storage v4 signatures, ensure the same region value is configured in your Object Storage or override the default S3Browser region name in the Advanced Settings options.

S3cmd

The credentials can be retrieved from the Object Storage logged in “User Information” properties.

/etc/.s3cfg

[default]
access_key = <S3 Access Key>
secret_key = <S3 Secret Key>
host_base = vsa-00000001-cloud-01.zadara.com
host_bucket = vsa-00000001-cloud-01.zadara.com
use_https = True

Note

  • access_key is the user S3 Access Key

  • secret_key is the user S3 Secret Key

  • host_base is the HTTPS path to the Object Storage being accessed

AWS Command Line Interface

Update the default/create new profile for the Object Storage within aws configuration file.

~/.aws/config

[profile zadara]
s3 =
    signature_version = s3v4

Note

It is possible to use both AWS v4/v2 signatures with S3-compatible storage such as Zadara Object Storage.

~/.aws/credentials

[zadara]
aws_access_key_id = <S3 Access Key>
aws_secret_access_key = <S3 Secret Key>

The credentials can be retrieved from the Object Storage logged in “User Information” properties.

Example of usage:

$ aws s3 --profile=zadara --endpoint-url=https://vsa-00000001-cloud-01.zadara.com --region=us-east-1 ls s3://zadara-test

2018-04-01 19:00 mytestfile1
2018-04-01 19:10 mytestfile2
2018-04-01 19:20 mytestfile3

Note

  • profile is the name of the credentials and config profile specified above (in this case, “zadara”)

  • endpoint-url is the HTTPS path to the Object Storage being accessed

  • region should match the Region defined in the Object Storage settings page (Zadara default: us-east-1)

Common operations examples

  • Creating a pre-signed URL - this allows anyone who receives the pre-signed URL to retrieve the object with HTTP GET request. The operation uses the S3 credentials and region field to generate the pre-signed URL.

    $ aws s3 presign --profile <AWS CLI profile name> --endpoint \
https://<object storage api endpoint> \
s3://<container/bucket name>/<object name> --expires-in <expiration in seconds>

    For more information please refer to the official AWS CLI Command Reference.

CORS configuration examples

Version: 23.09

CORS configurations can be configured using AWS and Swift APIs.

Important

For best practice, it is highly recommended not to use both AWS and Swift methods for CORS configurations.

CORS settings that are configured using AWS APIs take precedence over CORS settings that are configured via Swift APIs.

When using an AWS API to update a CORS configuration, all existing Swift CORS headers are removed. However, if a CORS configuration that was configured using AWS APIs is updated using Swift APIs, the Swift CORS headers are ignored. For CORS settings configured via AWS APIs, Swift CORS headers are always ignored, irrespective of whether Swift or AWS APIs were used first.

PUT Bucket CORS

To configure a bucket to allow cross-region requests, use the AWS S3 PutBucketCors bucket operation.

For example:

aws s3api put-bucket-cors --bucket my-bucket --profile=zadara --endpoint-url=https://vsa-00000103-public-zadara-qa19.zadarazios.com --cors-configuration file://cors.json

A CORS configuration is a list of CORS rules to apply to a bucket. It is mandatory to define at least one rule in a CORS configuration.

Command line parameters:

  • --bucket: The name of the NGOS object container on which the CORS rules will be configured.

  • --profile: The profile name of the configuration in the ~/.aws/config and ~/.aws/credentials files, for account credentials as described above.

  • --endpoint: The account’s API Endpoint or Public API Endpoint.

  • --cors-configuration: The file comprising the CORS configuration rules.

    In this example, the CORS configuration rules are defined in the cors.json configuration file:

    {
   "CORSRules": [
      {
      "ID": "xyz-abc-def-wxy",
      "AllowedOrigins": ["https"],
      "AllowedHeaders": ["*"],
      "AllowedMethods": ["PUT", "POST", "DELETE"],
      "MaxAgeSeconds": 3000,
      "ExposeHeaders": ["x-amz-server-side-encryption"]
      },
      {
      "AllowedOrigins": ["*"],
      "AllowedHeaders": ["x-abc-*"],
      "AllowedMethods": ["GET"]
      },
      {
      "AllowedOrigins": ["http://*.example.com", "https://xyz.com"],
      "AllowedHeaders": ["x-def"],
      "AllowedMethods": ["PUT", "POST", "DELETE"],
      "MaxAgeSeconds": 4000
      }
   ]
}

CORS Rule Properties:

CORSRules: A CORS configuration is a list of CORS rules. It is mandatory to define at least one rule.

Each rule in a CORS configuration list must have the following mandatory properties:

  • ID (mandatory)

    A unique string identifying the rule, up to a maximum length of 255 characters.

  • AllowedMethods (mandatory)

    A comma-separated list of one or more permitted REST methods allowed on the bucket for the current rule.

    • Only the "GET", "PUT", "POST", "DELETE" and "HEAD" methods can be specified, and the AllowedMethods list must comprise at least one method.

    • Wildcards and empty strings are not permitted.

  • AllowedOrigins (mandatory)

    A comma-separated list of origin sites allowed access to the bucket for the current rule.

    • An asterisk (*) wildcard can be specified in an origin string.

      • A maximum of one asterisk can be specified in an origin string.

      • An origin string that includes an asterisk specifies permitted access to the bucket from all origins that match the permutations. For example, "AllowedOrigins": ["http://*.example.com"] specifies permitted access from all subdomains of example.com.

      • AllowedOrigins with the value of a single origin comprising only an asterisk ("AllowedHeaders": ["*"]) specifies permitted access from all origins.

    • AllowedOrigins comprising strings without an asterisk must be fully defined, and only CORS requests from origins with an exact match are permitted access.

Each rule in a CORS configuration list can have the following optional properties:

  • ID (optional)

    A string to identify the rule, up to 255 characters.

  • AllowedHeaders (optional)

    A comma-separated list of headers that are allowed on this bucket for the current rule.

    • An asterisk (*) wildcard can be specified in a header.

      • A maximum of one asterisk can be specified in a header.

      • An allowed header string that includes an asterisk specifies that all matching permutations of the header are allowed. For example, "AllowedHeaders": ["x-abc-*"] specifies that all headers prefixed by “x-abc-” are allowed.

      • AllowedHeaders with the value of only an asterisk ("AllowedHeaders": ["*"]) specifies that headers with any value are permitted.

    • AllowedHeaders comprising strings without an asterisk must be fully defined, and only CORS requests with headers having an exact match are permitted.

  • ExposeHeaders (optional)

    A comma-separated list of headers that can be exposed to the client from a CORS request.

    • Wildcards are not permitted.

  • MaxAgeSeconds (optional)

    A positive integer specifying the maximum number of seconds that an OPTIONS request result can be cached by the browser for the current rule.

GET Bucket CORS

To view the CORS configuration on a bucket, use the AWS S3 GetBucketCors bucket operation.

For example:

aws s3api get-bucket-cors --bucket my-bucket --profile=zadara --endpoint-url=https://vsa-00000103-public-zadara-qa19.zadarazios.com

The CORS configuration is returned in JSON format.

Sample response:

{
"CORSRules": [
   {
      "ID": "xyz-abc-def-wxy",
      "AllowedHeaders": [
      "*"
      ],
      "AllowedMethods": [
      "PUT",
      "POST",
      "DELETE"
      ],
      "AllowedOrigins": [
      "https"
      ],
      "ExposeHeaders": [
      "x-amz-server-side-encryption"
      ],
      "MaxAgeSeconds": 3000
   },
   {
      "AllowedHeaders": [
      "x-abc-*"
      ],
      "AllowedMethods": [
      "GET"
      ],
      "AllowedOrigins": [
      "*"
      ]
   },
   {
      "AllowedHeaders": [
      "x-def"
      ],
      "AllowedMethods": [
      "PUT",
      "POST",
      "DELETE"
      ],
      "AllowedOrigins": [
      "http://*.example.com",
      "https://xyz.com"
      ],
      "MaxAgeSeconds": 4000
   }
]
}

Note

If the CORS configuration was configured using a Swift API and not by an AWS API, then the GetBucketCors operation does not return any data.

DELETE Bucket CORS

To delete a CORS configuration from a bucket, use the AWS S3 DeleteBucketCors bucket operation.

For example:

aws s3api delete-bucket-cors --bucket my-bucket --profile=zadara --endpoint-url=https://vsa-00000103-public-zadara-qa19.zadarazios.com

Note

Only AWS CORS configurations are erased from the container collection.

CORS headers configured using Swift APIs are not affected.

OBJECT Options (CORS)

To determine whether the server will permit a request on a CORS-enabled bucket or on any of its objects to proceed, an OPTIONS request can be invoked to direct the browser to send a preflight request (i.e. a preliminary probe) to the same URL.

Examples of typical signatures for an OPTIONS request:

  • For a CORS-enabled bucket

    curl -i -XOPTIONS -H "X-Auth-Token: <token>" \
-H "Origin: http://abc.com" \
-H "Access-Control-Request-Method: POST" \
-H "Access-Control-Request-Headers: <header1>" "https://(endpoint)/(bucket-name)"

  • For an object in a CORS-enabled bucket

    curl -i -XOPTIONS -H "X-Auth-Token: <token>" \
-H "Origin: http://abc.com" \
-H "Access-Control-Request-Method: POST" \
-H "Access-Control-Request-Headers: <header1>" "https://(endpoint)/(bucket-name)/(object-key)"

Note

Some versions of curl might require the headers in single quotes, and the URL in double quotes.

OPTIONS request headers:

  • Mandatory header:

    • Origin header (mandatory):

      The Origin header is checked against the list of AllowedOrigins in the CORS configuration rules.

      If there are no rules that match this header, the OPTIONS request fails with the HTTP_UNAUTHORIZED (401) status.

    • Access-Control-Request-Method header (mandatory):

      The Access-Control-Request-Method header is checked against the list of AllowedMethods in the CORS configuration rules.

      If there are no rules that match this header, the OPTIONS request fails with the HTTP_UNAUTHORIZED (401) status.

    Each rule in the CORS configuration is checked for matches to both AllowedOrigins and AllowedMethods. If there is no match, the check proceeds to the next rule.

    Success is determined when a match is found. No further rules are checked, and the browser can proceed with invoking the request.

    If none of the rules match, the HTTP_UNAUTHORIZED status is raised on OPTIONS request, indicating to the browser that it cannot invoke the request on the target CORS-enabled bucket using the requested Origin and method.

  • Optional header:

    • Access-Control-Request-Headers header (optional):

      The Access-Control-Request-Headers header is checked against the list of the optional AllowedHeaders in the CORS configuration rule that matches the madatory headers (AllowedOrigins and AllowedMethods).

      If AllowedHeaders is specified in the CORS rule, but if any of the Access-Control-Request-Headers do not match, the OPTIONS request returns a success status, but without any Access-Control-* headers in the response.

      In addition to a success status code, a successful OPTIONS request response should also contain all Access-Control-* headers, indicating the origin, method and request headers that are allowed on the bucket.

OPTIONS request example:

curl -i -XOPTIONS -H 'X-Auth-Token: <token>' \
   -H 'Origin: https://example.com' \
   -H 'Access-Control-Request-Method: PUT' \
   -H 'Access-Control-Request-Headers: x-123-abc' "https://vsa-0000003d-zadara-qa21.zadarazios.com:443/v1/AUTH_2eb509f93b0c4790890061007cdd62a4/corsbucket/delete.json"

OPTIONS request response example for a method that is configured in the CORS rule:

HTTP/1.1 200 OK
Allow: HEAD, POST, GET, OPTIONS, DELETE, PUT
access-control-allow-origin: https://example.com
vary: Origin, Access-Control-Request-Headers
access-control-max-age: 3000
Access-Control-Allow-Methods: PUT, POST, DELETE
access-control-allow-headers: x-123-abc
x-trans-id: txb7997e22016c4e449733e-006527a288
x-openstack-request-id: txb7997e22016c4e449733e-006527a288
Server: Zadara
Content-Length: 0
Date: Thu, 18 Apr 2024 07:38:49 GMT

In the response example, the following matching elements have been returned in response headers:

Returned response header

Returned elements

access-control-allow-origin

Origin in the request that matches the CORS rule’s AllowedOrigins.

access-control-allow-methods

All AllowedMethods in the matching CORS rule.

access-control-allow-headers

All request headers that match AllowedHeaders in the CORS rule.

access-control-max-age

MaxAgeSeconds, if specified in the CORS rule that matches the origin and method request headers.

Important

If even only one of the request headers does not match AllowedHeaders in the CORS rule, the OPTIONS response does not return any of the access-control-* headers, including the access-control-allow-origin and access-control-allow-methods.

ExposeHeaders is not returned in an OPTIONS response.

OPTIONS request response example for a method that is not authorized in the CORS rule:

HTTP/1.1 401 Unauthorized
Content-Type: text/html; charset=UTF-8
Allow: OPTIONS, HEAD, PUT, POST, DELETE, GET
Content-Length: 131
WWW-Authenticate: Keystone uri='<Keystone URI>'
x-trans-id: txf9e61f89a7424a84827e9-006641c489
x-openstack-request-id: txf9e61f89a7424a84827e9-006641c489
Server: Zadara
Date: Thu, 18 Apr 2024 07:36:23 GMT

<html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html>

Object Lifecycle Policy configuration examples

Version: 23.09

An optional Object Lifecycle Policy can be configured for a container, to determine the retention period for the container’s objects.

By creating one or more policy rules for a container, an Object Lifecycle Policy is established for the container.

The AWS S3 API examples in this section depict operations creating and managing a container’s Object Lifecycle Policy, with the following assumptions:

  • The container is named my-container.

  • The container is in an account that has the public endpoint URL https://vsa-00000103-public-zadara-qa19.zadarazios.com.

  • The account and its credentials are defined in a profile named zadara, configured in ~/.aws/config and ~/.aws/credentials as described in AWS Command Line Interface.

PUT Bucket Lifecycle Configuration

To configure rules for a container’s Object Lifecycle Policy, use the AWS S3 PutBucketLifecycleConfiguration bucket operation.

The AWS S3 CLI can invoke a configuration file that specifies lifecycle rules. For example, the following lifecycle configuration file lifecycle.json specifies two rules for the container’s Object Lifecycle Policy:

  • The first rule specifies:

    • Objects in the container have a retention period of 30 days from object creation.

    • In this example, there is no Filter parameter, indicating that the rule applies to all objects in the container.

  • The second rule specifies:

    • Objects in the container have a retention period up to 00:00 on 1st July 2024, irrespective of their age.

    • In this example, the Filter parameter applies this rule only to objects in the log folder.

    • Note that this rule is set initially to be disabled.

{
   "Rules": [
      {
            "Expiration": {
               "Days": 30
            },
            "ID": "30-day expiration rule",
            "Status": "Enabled"
      },
      {
            "Expiration": {
               "Date": "2024-07-01T12:00:00"
            },
            "ID": "2024 first half year expiration",
            "Filter": {
               "Prefix": "log/"
            },
            "Status": "Disabled"
      }
   ]
}

An AWS S3 CLI example that applies the rules specified in the lifecycle configuration file lifecycle.json, to configure a Lifecycle Policy for a container named my-container:

aws s3api put-bucket-lifecycle-configuration \
--bucket my-container \
--profile=zadara \
--endpoint-url=https://vsa-00000103-public-zadara-qa19.zadarazios.com  \
--lifecycle-configuration file://lifecycle.json
GET Bucket Lifecycle Configuration

To retrieve a container’s Object Lifecycle Policy configuration, use the AWS S3 GetBucketLifecycleConfiguration bucket operation.

For example, to retrieve the Object Lifecycle Policy configuration in JSON output format for a container named my-container:

aws s3api get-bucket-lifecycle-configuration \
--bucket=my-container \
--output=json \
--profile=zadara \
--endpoint-url=https://vsa-00000103-public-zadara-qa19.zadarazios.com

JSON response:

{
   "Rules": [
      {
            "Expiration": {
               "Days": 30
            },
            "ID": "30-day expiration rule",
            "Status": "Enabled"
      },
      {
            "Expiration": {
               "Date": "2024-07-01T12:00:00"
            },
            "ID": "2024 first half year expiration",
            "Filter": {
               "Prefix": "log/"
            },
            "Status": "Disabled"
      }
   ]
}
DELETE Bucket Lifecycle

To delete a container’s Object Lifecycle Policy configuration, use the AWS S3 DeleteBucketLifecycle bucket operation.

For example, to delete the Object Lifecycle Policy configuration for a container named my-container:

aws s3api delete-bucket-lifecycle \
--bucket=my-container \
--profile=zadara \
--endpoint-url=https://vsa-00000103-public-zadara-qa19.zadarazios.com

boto3 python library

Update the default/create new profile for the Object Storage within aws configuration file.

~/.aws/config

[profile zadara]
s3 =
    signature_version = s3v4

Note

It is possible to use both AWS v4/v2 signatures with S3-compatible storage such as Zadara Object Storage.

~/.aws/credentials

[zadara]
aws_access_key_id = <S3 Access Key>
aws_secret_access_key = <S3 Secret Key>

The credentials can be retrieved from the Object Storage logged in “User Information” properties.

In your python code:

#!/usr/bin/env python

import boto3

session = boto3.session.Session(profile_name='zadara')

s3_client = session.client(
    service_name='s3',
    region_name='us-east-1',
    endpoint_url='https://vsa-00000001-cloud-01.zadara.com',
)

print('Buckets')
print(s3_client.list_buckets())

print('')

print('Objects')
print(s3_client.list_objects(Bucket='test'))

Note

  • profile_name is the name of the credentials and config profile specified above (in this case, “zadara”)

  • endpoint_url is the HTTPS path to the Object Storage being accessed

  • region should match the Region defined in the Object Storage settings page (Zadara default: us-east-1)

AWS S3 Java SDK (aws-java-sdk)

AWS Provides a comprehensive S3 Java SDK that can be used with Zadara’s Object Storage. Getting started guide is available in Zadara’s Support Knowledge Base article - How to use AWS S3 Java SDK with Object Storage.

AWS S3 PHP SDK (aws-sdk-php)

AWS Provides a comprehensive S3 PHP SDK that can be used with Zadara’s VPSA Object Storage. Getting started guide is available in Zadara’s Support Knowledge Base article - How to use AWS S3 PHP SDK with Object Storage.

AWS S3 JavaScript SDK (aws-sdk)

AWS Provides a comprehensive S3 JavaScript SDK that can be used with Zadara’s VPSA Object Storage. Getting started guide is available in Zadara’s Support Knowledge Base article - How to use AWS S3 JavaScript SDK with Object Storage.

Openstack Swift Interface

The management interface generates a new Swift API token upon login. This means that if you logout and login again you’ll notice a new token. The token presented by the management interface is always the latest and valid to be used.

The API tokens created by the management interface are generated based on the Object Storage global configuration for token validity (default: 24 hours).

Example of validating an API token using the CLI:

# Get two consecutive API tokens from the management interface and store it
$ TOKEN1=gAAAAABiuwu4P55M2V...
$ TOKEN2=gAAAAABiuwvc8BEYc8...

$ curl -X GET -H "Content-Type: application/json" \
  -H "X-Access-Key: $TOKEN1"  \
  "https://<object storage endpoint>:8443/api/zios/accounts/AUTH_<account ID>/users.json"

{"response":{"users":[],"count":0,"status":0}}%

curl -X GET -H "Content-Type: application/json" \
 -H "X-Access-Key: $TOKEN2" \
 "https://<object storage endpoint>:8443/api/zios/accounts/AUTH_<account ID>/users.json"

{"response":{"users":[],"count":0,"status":0}}%

# We can validate these tokens using the Openstack Keystone auth service as well:

$ curl -s   -H "X-Subject-Token: $TOKEN2" -H "X-Auth-Token: $TOKEN1"  \
  "https://<object storage endpoint>:5000/v3/auth/tokens" | python3 -m json.tool
{
    "token": {
       "is_domain": false,
      "methods": [
         "password"
      ],
      "roles": [
         {
            "id": "fedeff6db6df47959e96d8dd33963cfe",
            "name": "zios_admin"
         }
      ],
      "expires_at": "2022-06-29T14:10:36.000000Z",
 ....
      "issued_at": "2022-06-28T14:10:36.000000Z"
   }
}

Important

By default, the API token is valid for 24 hours. the preferred option to identify/renew the API token via an API call is to use a Keystone authentication request and not using Object Storage command indicated in the Zadara Object Storage REST API Guide. Example for authentication against the Keystone service is provided in the next section.

cURL (swift API)

cURL can be used for Object Storage operations. The connectivity information is available in the User Information view.

curl_account_urls

In this example, we will use the Front End Network Account URL or Public API Network Account URL, and API Token in order to create a new container:

$ curl -H "x-auth-token: <user_token>" -X PUT <account_url>/test-bucket/

For example:

$ URL=<Front End Network Account URL or Public API Network Account URL>
$ TOKEN=<MYAPI TOKEN>
$ curl -H "x-auth-token: $TOKEN" -X PUT $URL/test-bucket/

The following example describes how to get the token programmatically using the Swift API:

$ curl -i -H "Content-Type: application/json" \
-d '{ "auth": \
{ "identity": { "methods": ["password"], "password": \
{ "user": {"name": "<USERNAME>", "domain": { "id": "default" }, \
"password": "<USER PASSWORD>" }} }, "scope": { "project": \
{ "name": "<ACCOUNT_NAME>", "domain": { "id": "default" } } } } }' \
"https://vsa-00000001-mycloud-01.zadara.com:5000/v3/auth/tokens" ;

and use the returned token for the subsequent API calls.

HTTP/1.1 201 Created
Date: Thu, 19 Nov 2020 16:05:28 GMT
Server: Apache/2.4.29 (Ubuntu)
Content-Length: 1114
X-Subject-Token: gAAAAABftpfIAiuo2tRZZP8VVtomU1knVG7xNUONV4b2u....

Additional examples of using the Openstack Swift API can be found at the Openstack Swift API documentation

Cloudberry Explorer for OpenStack (v3 authentication)

Use the logged-in User Information properties to set the authentication fields of Cloudberry Explorer

image81

CyberDuck

Cyberduck version: 7.7.1 (33788)

Cyberduck client support “Openstack Swift (Keystone 3)” API interface.

Use the logged-in User Information properties to set the authentication field of CyberDuck client.

  1. Server - the Object Storage v3 Auth Endpoint.

  2. Port - 5000

  3. Project:Domain:Username - <Object Storage Account>:default:<Object Storage Username>

cyber-duck-config