# Using Object Storage Clients

Standard client tools can be used to browse objects in the Object Storage.
This section will help configuring Object Storage Client Tools to work
against Object Storage. In order to access the Object Storage the
client tool must be configured with the user’s authentication credentials.

The Object Storage support two API interfaces:

1. AWS S3 API
2. OpenStack Swift API

The required parameters can be found in the Object Storage User Information
page. Information for the user currently logged in to the Object Storage
management interface displayed by clicking the user name on the management
interface’s upper right corner.

![ngos-user-info](media/ngos-user-info.png)

## AWS S3 Compatible clients

### Supported S3 APIs

The Object Storage is utilizing Openstack Swift’s S3 Middleware. As S3 is an AWS product,
it includes some features that are AWS oriented and are outside of the scope of
Zadara’s Object Storage offering.

Zadara supports the following S3 operations:

#### Object operations

- [GET Object](http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectGET.html)
- [HEAD Object](http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectHEAD.html)
- [PUT Object](http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectPUT.html)
- [PUT Object Copy](http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectCOPY.html)
- [DELETE Object](http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectDELETE.html)
- [DELETE Multiple Objects](http://docs.amazonwebservices.com/AmazonS3/latest/API/multiobjectdeleteapi.html)
- [GET Object ACL](http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectGETacl.html)
- [PUT Object ACL](http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectPUTacl.html)
- [Initiate Multipart Upload](http://docs.amazonwebservices.com/AmazonS3/latest/API/mpUploadInitiate.html)
- [Upload Part](http://docs.amazonwebservices.com/AmazonS3/latest/API/mpUploadUploadPart.html)
- [Upload Part Copy](http://docs.amazonwebservices.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html)
- [Complete Multipart Upload](http://docs.amazonwebservices.com/AmazonS3/latest/API/mpUploadComplete.html)
- [Abort Multipart Upload](http://docs.amazonwebservices.com/AmazonS3/latest/API/mpUploadAbort.html)
- [List Parts](http://docs.amazonwebservices.com/AmazonS3/latest/API/mpUploadListParts.html)
- [List Multipart Uploads](http://docs.amazonwebservices.com/AmazonS3/latest/API/mpUploadListMPUpload.html)
- [Object Retention Support](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectRetention.html)

#### Bucket operations

- [GET Bucket List Objects](http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGET.html)
- [HEAD Bucket](http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketHEAD.html)
- [PUT Bucket](http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketPUT.html)
- [DELETE Bucket](http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketDELETE.html)
- [GET Bucket ACL](http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGETacl.html)
- [PUT Bucket ACL](http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketPUTacl.html)
- [GET Bucket Location](http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTBucketGETlocation.html)
- [GET Bucket Versioning](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketVersioning.html)
- [GET Bucket Logging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLogging.html)
- [PUT Bucket Logging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLogging.html)
- [GET Bucket CORS](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketCors.html) Version: 23.09
- [PUT Bucket CORS](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketCors.html) Version: 23.09
- [DELETE Bucket CORS](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketCors.html) Version: 23.09
- [OPTIONS Object](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTOPTIONSobject.html) Version: 23.09
- [PUT Bucket Lifecycle Configuration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html) Version: 23.09
- [GET Bucket Lifecycle Configuration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLifecycleConfiguration.html) Version: 23.09
- [DELETE Bucket Lifecycle Configuration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketLifecycle.html) Version: 23.09

### Authentication information

For Object Storage connectivity, it is required to gather the following
information from the Object Storage management UI:

1. Object Storage Endpoint - an endpoint entry will be available according to
   to the networking layout of the Object Storage (i.e. for Object Storage
   with a Public IP interface we’ll have a record matching to the public network
   interfaces)
2. Object Storage region.
3. S3 API Access Key/Secret Key

In the Object Storage GUI, navigate to the User Information section
(top right corner, by clicking the logged in username).

![ngos-user-info-full](media/ngos-user-info-full.png)

### S3 Browser

S3 Browser can be used to administrate and perform object operations against
Zadara’s Object Storage. The account information in S3 Browser should
be configured according to the following example (S3 Compatible Storage):

![s3-browser-create-account](media/s3-browser-create-account.png)

Once the Endpoint and authentication details are configured properly, click
on the Advanced S3-compatible storage settings

In the advanced settings select the following:

1. Signature version - Signature V4
2. Addressing model - Path style
3. Override storage regions - specify the Object Storage region name;
   the format is `Region Name=<region name>`.

Close and save the account information.

### S3cmd

The credentials can be retrieved from the Object Storage logged in
“User Information” properties.

**/etc/.s3cfg**

```ini
[default]
access_key = <S3 Access Key>
secret_key = <S3 Secret Key>
host_base = vsa-00000001-cloud-01.zadara.com
host_bucket = vsa-00000001-cloud-01.zadara.com
use_https = True
```

<!-- end -->

<a id="aws-cli"></a>

### AWS Command Line Interface

Update the default/create new profile for the Object Storage within
aws configuration file.

**~/.aws/config**

```ini
[profile zadara]
s3 =
    signature_version = s3v4
```

<!-- end -->

**~/.aws/credentials**

```ini
[zadara]
aws_access_key_id = <S3 Access Key>
aws_secret_access_key = <S3 Secret Key>
```

<!-- end -->

The credentials can be retrieved from the Object Storage logged in
“User Information” properties.

Example of usage:

```console
$ aws s3 --profile=zadara --endpoint-url=https://vsa-00000001-cloud-01.zadara.com --region=us-east-1 ls s3://zadara-test

2018-04-01 19:00 mytestfile1
2018-04-01 19:10 mytestfile2
2018-04-01 19:20 mytestfile3
```

<!-- end -->

#### Common operations examples

* Creating a presigned URL - this allows anyone who receives the presigned
  URL to retrieve the object with HTTP GET request. The operation uses the
  S3 credentials and region field to generate the pre-signed URL.
  ```console
  $ aws s3 presign --profile <AWS CLI profile name> --endpoint \
  https://<object storage api endpoint> \
  s3://<container/bucket name>/<object name> --expires-in <expiration in seconds>
  ```

  For more information please refer to the official [AWS CLI Command
  Reference](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/presign.html).

<a id="ngos-aws-cli-cors-examples"></a>

#### CORS configuration examples

Version: 23.09

CORS configurations can be configured using AWS and Swift APIs.

##### PUT Bucket CORS

To configure a bucket to allow cross-region requests, use the AWS S3
`PutBucketCors` bucket operation.

For example:

```console
aws s3api put-bucket-cors --bucket my-bucket --profile=zadara --endpoint-url=https://vsa-00000103-public-zadara-qa19.zadarazios.com --cors-configuration file://cors.json
```

A CORS configuration is a list of CORS rules to apply to a bucket.
It is mandatory to define at least one rule in a CORS configuration.

Command line parameters:

* `--bucket`: The name of the NGOS object container on which the CORS
  rules will be configured.
* `--profile`: The profile name of the configuration in the
  `~/.aws/config` and `~/.aws/credentials` files, for account
  credentials as described above.
* `--endpoint`: The account’s API Endpoint or Public API Endpoint.
* `--cors-configuration`: The file comprising the CORS configuration rules.

  In this example, the CORS configuration rules are defined in the
  `cors.json` configuration file:
  ```console
  {
     "CORSRules": [
        {
        "ID": "xyz-abc-def-wxy",
        "AllowedOrigins": ["https"],
        "AllowedHeaders": ["*"],
        "AllowedMethods": ["PUT", "POST", "DELETE"],
        "MaxAgeSeconds": 3000,
        "ExposeHeaders": ["x-amz-server-side-encryption"]
        },
        {
        "AllowedOrigins": ["*"],
        "AllowedHeaders": ["x-abc-*"],
        "AllowedMethods": ["GET"]
        },
        {
        "AllowedOrigins": ["http://*.example.com", "https://xyz.com"],
        "AllowedHeaders": ["x-def"],
        "AllowedMethods": ["PUT", "POST", "DELETE"],
        "MaxAgeSeconds": 4000
        }
     ]
  }
  ```

**CORS Rule Properties**:

`CORSRules`: A CORS configuration is a list of CORS rules.
It is mandatory to define at least one rule.

Each rule in a CORS configuration list must have the following
**mandatory** properties:

* **ID** (mandatory)

  A unique string identifying the rule, up to a maximum length of 255
  characters.
* **AllowedMethods** (mandatory)

  A comma-separated list of one or more permitted REST methods allowed
  on the bucket for the current rule.
  * Only the `"GET"`, `"PUT"`, `"POST"`, `"DELETE"` and `"HEAD"`
    methods can be specified, and the AllowedMethods list must comprise
    at least one method.
  * Wildcards and empty strings are not permitted.
* **AllowedOrigins** (mandatory)

  A comma-separated list of origin sites allowed access to the bucket
  for the current rule.
  * An asterisk (`*`) wildcard can be specified in an origin string.
    * A maximum of one asterisk can be specified in an origin string.
    * An origin string that includes an asterisk specifies permitted
      access to the bucket from all origins that match the permutations.
      For example, `"AllowedOrigins": ["http://*.example.com"]`
      specifies permitted access from all subdomains of `example.com`.
    * AllowedOrigins with the value of a single origin comprising
      only an asterisk (`"AllowedHeaders": ["*"]`) specifies
      permitted access from all origins.
  * AllowedOrigins comprising strings without an asterisk must be
    fully defined, and only CORS requests from origins with an exact
    match are permitted access.

Each rule in a CORS configuration list can have the following **optional**
properties:

* **ID** (optional)

  A string to identify the rule, up to 255 characters.
* **AllowedHeaders** (optional)

  A comma-separated list of headers that are allowed on this bucket for
  the current rule.
  * An asterisk (`*`) wildcard can be specified in a header.
    * A maximum of one asterisk can be specified in a header.
    * An allowed header string that includes an asterisk specifies
      that all matching permutations of the header are allowed.
      For example, `"AllowedHeaders": ["x-abc-*"]`
      specifies that all headers prefixed by “x-abc-” are allowed.
    * AllowedHeaders with the value of only an asterisk
      (`"AllowedHeaders": ["*"]`) specifies that headers with any
      value are permitted.
  * AllowedHeaders comprising strings without an asterisk must be
    fully defined, and only CORS requests with headers having an exact
    match are permitted.
* **ExposeHeaders** (optional)

  A comma-separated list of headers that can be exposed to the client
  from a CORS request.
  * Wildcards are not permitted.
* **MaxAgeSeconds** (optional)

  A positive integer specifying the maximum number of seconds that an
  OPTIONS request result can be cached by the browser for the current
  rule.

##### GET Bucket CORS

To view the CORS configuration on a bucket, use the AWS S3
`GetBucketCors` bucket operation.

For example:

```console
aws s3api get-bucket-cors --bucket my-bucket --profile=zadara --endpoint-url=https://vsa-00000103-public-zadara-qa19.zadarazios.com
```

The CORS configuration is returned in JSON format.

Sample response:

```console
{
"CORSRules": [
   {
      "ID": "xyz-abc-def-wxy",
      "AllowedHeaders": [
      "*"
      ],
      "AllowedMethods": [
      "PUT",
      "POST",
      "DELETE"
      ],
      "AllowedOrigins": [
      "https"
      ],
      "ExposeHeaders": [
      "x-amz-server-side-encryption"
      ],
      "MaxAgeSeconds": 3000
   },
   {
      "AllowedHeaders": [
      "x-abc-*"
      ],
      "AllowedMethods": [
      "GET"
      ],
      "AllowedOrigins": [
      "*"
      ]
   },
   {
      "AllowedHeaders": [
      "x-def"
      ],
      "AllowedMethods": [
      "PUT",
      "POST",
      "DELETE"
      ],
      "AllowedOrigins": [
      "http://*.example.com",
      "https://xyz.com"
      ],
      "MaxAgeSeconds": 4000
   }
]
}
```

##### DELETE Bucket CORS

To delete a CORS configuration from a bucket, use the AWS S3
`DeleteBucketCors` bucket operation.

For example:

```console
aws s3api delete-bucket-cors --bucket my-bucket --profile=zadara --endpoint-url=https://vsa-00000103-public-zadara-qa19.zadarazios.com
```

##### OBJECT Options (CORS)

To determine whether the server will permit a request on a CORS-enabled
bucket or on any of its objects to proceed, an OPTIONS request can be
invoked to direct the browser to send a preflight request
(i.e. a preliminary probe) to the same URL.

Examples of typical signatures for an OPTIONS request:

* For a CORS-enabled bucket
  ```console
  curl -i -XOPTIONS -H "X-Auth-Token: <token>" \
  -H "Origin: http://abc.com" \
  -H "Access-Control-Request-Method: POST" \
  -H "Access-Control-Request-Headers: <header1>" "https://(endpoint)/(bucket-name)"
  ```
* For an object in a CORS-enabled bucket
  ```console
  curl -i -XOPTIONS -H "X-Auth-Token: <token>" \
  -H "Origin: http://abc.com" \
  -H "Access-Control-Request-Method: POST" \
  -H "Access-Control-Request-Headers: <header1>" "https://(endpoint)/(bucket-name)/(object-key)"
  ```

**OPTIONS request headers**:

* **Mandatory** header:
  * **Origin** header (mandatory):

    The Origin header is checked against the list of AllowedOrigins in the
    CORS configuration rules.

    If there are no rules that match this header, the
    OPTIONS request fails with the HTTP_UNAUTHORIZED (401) status.
  * **Access-Control-Request-Method** header (mandatory):

    The Access-Control-Request-Method header is checked against the list
    of AllowedMethods in the CORS configuration rules.

    If there are no rules that match this header, the
    OPTIONS request fails with the HTTP_UNAUTHORIZED (401) status.

  Each rule in the CORS configuration is checked for matches to both
  AllowedOrigins and AllowedMethods. If there is no match, the check
  proceeds to the next rule.

  Success is determined when a match is found. No further rules are
  checked, and the browser can proceed with invoking the request.

  If none of the rules match, the HTTP_UNAUTHORIZED status is raised on
  OPTIONS request, indicating to the browser that it cannot invoke the
  request on the target CORS-enabled bucket using the requested Origin and
  method.
* **Optional** header:
  * **Access-Control-Request-Headers** header (optional):

    The Access-Control-Request-Headers header is checked against the list
    of the optional AllowedHeaders in the CORS configuration rule that
    matches the madatory headers (AllowedOrigins and AllowedMethods).

    If AllowedHeaders is specified in the CORS rule, but if any of the
    Access-Control-Request-Headers do not match, the OPTIONS request
    returns a success status, but without any Access-Control-\* headers in
    the response.

    In addition to a success status code, a successful OPTIONS request
    response should also contain all Access-Control-\* headers,
    indicating the origin, method and request headers that are allowed on
    the bucket.

**OPTIONS request example**:

```console
curl -i -XOPTIONS -H 'X-Auth-Token: <token>' \
   -H 'Origin: https://example.com' \
   -H 'Access-Control-Request-Method: PUT' \
   -H 'Access-Control-Request-Headers: x-123-abc' "https://vsa-0000003d-zadara-qa21.zadarazios.com:443/v1/AUTH_2eb509f93b0c4790890061007cdd62a4/corsbucket/delete.json"
```

**OPTIONS request response example for a method that is configured in the CORS rule**:

```console
HTTP/1.1 200 OK
Allow: HEAD, POST, GET, OPTIONS, DELETE, PUT
access-control-allow-origin: https://example.com
vary: Origin, Access-Control-Request-Headers
access-control-max-age: 3000
Access-Control-Allow-Methods: PUT, POST, DELETE
access-control-allow-headers: x-123-abc
x-trans-id: txb7997e22016c4e449733e-006527a288
x-openstack-request-id: txb7997e22016c4e449733e-006527a288
Server: Zadara
Content-Length: 0
Date: Thu, 18 Apr 2024 07:38:49 GMT
```

In the response example, the following matching elements have been returned
in response headers:

| Returned response header       | Returned elements                                                                                    |
|--------------------------------|------------------------------------------------------------------------------------------------------|
| `access-control-allow-origin`  | **Origin** in the request that matches the CORS rule’s **AllowedOrigins**.                           |
| `access-control-allow-methods` | All **AllowedMethods** in the matching CORS rule.                                                    |
| `access-control-allow-headers` | All request headers that match **AllowedHeaders** in the CORS rule.                                  |
| `access-control-max-age`       | **MaxAgeSeconds**, if specified in the CORS rule that matches the origin and method request headers. |

**OPTIONS request response example for a method that is not authorized in the CORS rule**:

```console
HTTP/1.1 401 Unauthorized
Content-Type: text/html; charset=UTF-8
Allow: OPTIONS, HEAD, PUT, POST, DELETE, GET
Content-Length: 131
WWW-Authenticate: Keystone uri='<Keystone URI>'
x-trans-id: txf9e61f89a7424a84827e9-006641c489
x-openstack-request-id: txf9e61f89a7424a84827e9-006641c489
Server: Zadara
Date: Thu, 18 Apr 2024 07:36:23 GMT

<html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html>
```

<a id="ngos-object-lifecycle-policy-asw-s3api-examples"></a>

#### Object Lifecycle Policy configuration examples

Version: 23.09

An optional [Object Lifecycle Policy](ngos-console.md#ngos-object-lifecycle-policy) can be configured for a
container, to determine the retention period for the container’s objects.

By creating one or more policy rules for a container, an Object Lifecycle
Policy is established for the container.

The AWS S3 API examples in this section depict operations creating and
managing a container’s Object Lifecycle Policy, with the following
assumptions:

* The container is named `my-container`.
* The container is in an account that has the public endpoint URL
  `https://vsa-00000103-public-zadara-qa19.zadarazios.com`.
* The account and its credentials are defined in a profile named
  `zadara`, configured in `~/.aws/config` and `~/.aws/credentials`
  as described in [AWS Command Line Interface](#aws-cli).

##### PUT Bucket Lifecycle Configuration

To configure rules for a container’s Object Lifecycle Policy,
use the AWS S3 `PutBucketLifecycleConfiguration` bucket operation.

The AWS S3 CLI can invoke a configuration file that specifies lifecycle
rules.
For example, the following lifecycle configuration file `lifecycle.json`
specifies two rules for the container’s Object Lifecycle Policy:

* The first rule specifies:
  * Objects in the container have a retention period of 30 days from
    object creation.
  * In this example, there is no `Filter` parameter,
    indicating that the rule applies to all objects in the container.
* The second rule specifies:
  * Objects in the container have a retention period up to 00:00 on
    1st July 2024, irrespective of their age.
  * In this example, the `Filter` parameter applies this rule only to
    objects in the `log` folder.
  * Note that this rule is set initially to be disabled.

```console
{
   "Rules": [
      {
            "Expiration": {
               "Days": 30
            },
            "ID": "30-day expiration rule",
            "Status": "Enabled"
      },
      {
            "Expiration": {
               "Date": "2024-07-01T12:00:00"
            },
            "ID": "2024 first half year expiration",
            "Filter": {
               "Prefix": "log/"
            },
            "Status": "Disabled"
      }
   ]
}
```

An AWS S3 CLI example that applies the rules specified in the lifecycle
configuration file `lifecycle.json`, to configure a Lifecycle Policy for
a container named `my-container`:

```console
aws s3api put-bucket-lifecycle-configuration \
--bucket my-container \
--profile=zadara \
--endpoint-url=https://vsa-00000103-public-zadara-qa19.zadarazios.com  \
--lifecycle-configuration file://lifecycle.json
```

##### GET Bucket Lifecycle Configuration

To retrieve a container’s Object Lifecycle Policy configuration,
use the AWS S3 `GetBucketLifecycleConfiguration` bucket operation.

For example, to retrieve the Object Lifecycle Policy configuration in JSON
output format for a container named `my-container`:

```console
aws s3api get-bucket-lifecycle-configuration \
--bucket=my-container \
--output=json \
--profile=zadara \
--endpoint-url=https://vsa-00000103-public-zadara-qa19.zadarazios.com
```

JSON response:

```console
{
   "Rules": [
      {
            "Expiration": {
               "Days": 30
            },
            "ID": "30-day expiration rule",
            "Status": "Enabled"
      },
      {
            "Expiration": {
               "Date": "2024-07-01T12:00:00"
            },
            "ID": "2024 first half year expiration",
            "Filter": {
               "Prefix": "log/"
            },
            "Status": "Disabled"
      }
   ]
}
```

##### DELETE Bucket Lifecycle

To delete a container’s Object Lifecycle Policy configuration,
use the AWS S3 `DeleteBucketLifecycle` bucket operation.

For example, to delete the Object Lifecycle Policy configuration for a
container named `my-container`:

```console
aws s3api delete-bucket-lifecycle \
--bucket=my-container \
--profile=zadara \
--endpoint-url=https://vsa-00000103-public-zadara-qa19.zadarazios.com
```

#### Multipart Object example

Multipart object calls are used in scenarios where large objects need to be uploaded or downloaded efficiently.
<br/>
Multipart uploads are recommended for files larger than 100MB, and are mandatory for files larger than 5GB.
<br/>
Parts can be uploaded in parallel, and are reassembled into a single object at the destination container folder.
<br/>

The following example illustrates the steps to upload a large file `largeFile.dat`
that has a size of 725,135,360 bytes (approximately 700MB):

1. Split the file into smaller parts, in this example, three parts (files)
   of 200MB each, and one part (file) of the remainder.
   ```console
   $ ls -l largeFile.dat

   -rwxrwxrwx 1 user1 zadara 725135360 Feb  3 15:22 largeFile.dat

   $ split -b 200m largeFile.dat

   $ ls -l xa*

   -rwxrwxrwx 2 user1 zadara 209715200 Feb 10 08:27 xaa*
   -rwxrwxrwx 2 user1 zadara 209715200 Feb 10 08:27 xab*
   -rwxrwxrwx 2 user1 zadara 209715200 Feb 10 08:27 xac*
   -rwxrwxrwx 2 user1 zadara  95989760 Feb 10 08:27 xad*
   ```
2. Initialize the upload process with directives that the multipart file
   `largeFile.dat` must be assembled into the destination folder
   `partsExample`
   in the bucket `mybucket`.
   ```console
   $ aws s3api create-multipart-upload \
   --profile=zadara \
   --endpoint-url=https://vsa-00000001-cloud-01.zadara.com \
   --bucket mybucket \
   --key 'partsExample/largeFile.dat'
   ```

   Response:
   ```console
   {
      "Bucket": "mybucket",
      "Key": "partSexample/largeFile.dat",
      "UploadId": "OWM1YTdlOWUtODdlMC00NGY1LTk3ZWYtOWNiNTE2ZjBjODJj"
   }
   ```
3. From the response, note the value of `UploadId`, which is needed for
   all the subsequent calls of this multipart upload.

   Upload the first part, `--part-number 1`, the `xaa` file, and
   specify the `UploadId`:
   ```console
   $ aws s3api upload-part --profile=zadara \
   --endpoint-url=https://vsa-00000001-cloud-01.zadara.com \
   --bucket mybucket \
   --key "partsExample/largeFile.dat" \
   --part-number 1 \
   --body xaa \
   --upload-id OWM1YTdlOWUtODdlMC00NGY1LTk3ZWYtOWNiNTE2ZjBjODJj
   ```

   Response:
   ```console
   {
      "ETag": "\"6eac0b608eee111876931ddbd6907429\""
   }
   ```
4. From the response, note the value of `ETag` for each part.
   These will be required in the multipart upload completion step,
   where the parts are reassembled into a single object at the destination.

   Upload the second part, `--part-number 2`, the `xab` file,
   taking note of the value of its `ETag` from the response:
   ```console
   $ aws s3api upload-part --profile=zadara \
   --endpoint-url=https://vsa-00000001-cloud-01.zadara.com \
   --bucket mybucket \
   --key "partsExample/largeFile.dat" \
   --part-number 2 \
   --body xab \
   --upload-id OWM1YTdlOWUtODdlMC00NGY1LTk3ZWYtOWNiNTE2ZjBjODJj
   ```

   Response:
   ```console
   {
      "ETag": "\"aaffcae340799e25223970903f917cd6\""
   }
   ```
5. Upload the third part, `--part-number 3`, the `xac` file,
   taking note of the value of its `ETag` from the response:
   ```console
   $ aws s3api upload-part --profile=zadara \
   --endpoint-url=https://vsa-00000001-cloud-01.zadara.com \
   --bucket mybucket \
   --key "partsExample/largeFile.dat" \
   --part-number 3 \
   --body xac \
   --upload-id OWM1YTdlOWUtODdlMC00NGY1LTk3ZWYtOWNiNTE2ZjBjODJj
   ```

   Response:
   ```console
   {
      "ETag": "\"d6d7687673a728c8dc5683ce8ada395f\""
   }
   ```
6. Upload the fourth part, `--part-number 4`, the `xad` file,
   taking note of the value of its `ETag` from the response:
   ```console
   $ aws s3api upload-part --profile=zadara \
   --endpoint-url=https://vsa-00000001-cloud-01.zadara.com \
   --bucket mybucket \
   --key "partsExample/largeFile.dat" \
   --part-number 4 \
   --body xad \
   --upload-id OWM1YTdlOWUtODdlMC00NGY1LTk3ZWYtOWNiNTE2ZjBjODJj
   ```

   Response:
   ```console
   {
      "ETag": "\"ab246aba52b0c3e78ee3be1f43843d3b\""
   }
   ```
7. Edit and save a JSON file, ensuring that it includes the `ETag` value
   returned from the responses from each `upload-part` of the multipart
   object.

   In this example, the file is named `parts.json` containing the
   following:
   ```console
   {
      "Parts": [
         { "ETag": "6eac0b608eee111876931ddbd6907429", "PartNumber":1 },
         { "ETag": "aaffcae340799e25223970903f917cd6", "PartNumber":2 },
         { "ETag": "d6d7687673a728c8dc5683ce8ada395f", "PartNumber":3 },
         { "ETag": "ab246aba52b0c3e78ee3be1f43843d3b", "PartNumber":4 }
      ]
   }
   ```
8. Complete the multipart upload, using the `parts.json` file to
   specify the file parts reassembly.
   ```console
   $ aws s3api complete-multipart-upload \
   --profile=zadara \
   --endpoint-url=https://vsa-00000001-cloud-01.zadara.com \
   --bucket mybucket \
   --key "partsExample/largeFile.dat"  \
   --upload-id OWM1YTdlOWUtODdlMC00NGY1LTk3ZWYtOWNiNTE2ZjBjODJj \
   --multipart-upload file://parts.json
   ```

   Response:
   ```console
   {
      "Location": "https://vsa-00000103-public-zadara-qa19.zadarazios.com/tmptest/partsExample/largeFile.dat",
      "Bucket": "mybucket",
      "Key": "partsExample/largeFile.dat",
      "ETag": "\"2e0b93de0eb3f28c6734b864dd157ccd-4\""
   }
   ```
9. Use the `head-object` command to return information on the uploaded
   multipart object:
   ```console
   $ aws s3api head-object --profile=zadara \
   --endpoint-url=https://vsa-00000001-cloud-01.zadara.com \
   --bucket mybucket \
   --key "partsExample/largeFile.dat"
   --part-number 2
   ```

   Response:
   ```console
   {
      "AcceptRanges": "bytes",
      "LastModified": "Mon, 10 Feb 2025 07:35:18 GMT",
      "ContentLength": 209715200,
      "ETag": "\"2e0b93de0eb3f28c6734b864dd157ccd-4\"",
      "ContentType": "application/octet-stream",
      "Metadata": {},
      "PartsCount": 4
   }
   ```

   Included in the response above:
   * `"ContentLength": 209715200`:

     The size of this part (`--part-number 2`) is 200 MB (209715200 bytes),
     matching the size of the original uploaded part 2, `xab`.
   * `"PartsCount": 4`:

     The total number of parts in the uploaded multipart object.

### boto3 python library

Update the default/create new profile for the Object Storage within
aws configuration file.

**~/.aws/config**

```ini
[profile zadara]
s3 =
    signature_version = s3v4
```

<!-- end -->

**~/.aws/credentials**

```ini
[zadara]
aws_access_key_id = <S3 Access Key>
aws_secret_access_key = <S3 Secret Key>
```

<!-- end -->

The credentials can be retrieved from the Object Storage logged in
“User Information” properties.

In your python code:

```default
#!/usr/bin/env python

import boto3

session = boto3.session.Session(profile_name='zadara')

s3_client = session.client(
    service_name='s3',
    region_name='us-east-1',
    endpoint_url='https://vsa-00000001-cloud-01.zadara.com',
)

print('Buckets')
print(s3_client.list_buckets())

print('')

print('Objects')
print(s3_client.list_objects(Bucket='test'))
```

### AWS S3 Java SDK (aws-java-sdk)

AWS Provides a comprehensive S3 Java SDK that can be used with Zadara’s Object Storage. Getting started guide is available in Zadara’s Support
Knowledge Base article -
[How to use AWS S3 Java SDK with Object Storage](https://support.zadarastorage.com/hc/en-us/articles/360040983591-How-to-use-AWS-S3-Java-SDK-with-VPSA-Object-Storage).

### AWS S3 PHP SDK (aws-sdk-php)

AWS Provides a comprehensive S3 PHP SDK that can be used with Zadara’s VPSA
Object Storage. Getting started guide is available in Zadara’s Support
Knowledge Base article -
[How to use AWS S3 PHP SDK with Object Storage](https://support.zadarastorage.com/hc/en-us/articles/360040991211-How-to-use-AWS-S3-PHP-SDK-with-VPSA-Object-Storage).

### AWS S3 JavaScript SDK (aws-sdk)

AWS Provides a comprehensive S3 JavaScript SDK that can be used with Zadara’s VPSA
Object Storage. Getting started guide is available in Zadara’s Support
Knowledge Base article -
[How to use AWS S3 JavaScript SDK with Object Storage](https://support.zadarastorage.com/hc/en-us/articles/360060083472-How-to-use-AWS-S3-JavaScript-SDK-with-VPSA-Object-Storage).

### Cyberduck for AWS S3

**Cyberduck Version**: 9.1.7 (43177)

Cyberduck is a graphical client that supports the S3 API and is compatible
with Zadara Object Storage.

* Download the Cyberduck client for Windows and macOS at
  [https://cyberduck.io/download/](https://cyberduck.io/download/).
* See the [Cyberduck Reference](https://docs.cyberduck.io/cyberduck/)
  for the official Cyberduck documentation.

<a id="ngos-cyberduck-configure-connection"></a>

#### Configuring a Cyberduck S3 Connection to Zadara Object Storage

To connect to your Zadara Object Storage using the Cyberduck client, use
the S3 Access Key, S3 Secret Key, and Public API Endpoint from your
Zadara Object Storage account’s information page.

1. At the top right of the Zadara Object Storage UI, click your login
   username.

   The user profile, account, credentials and connectivity details appear.
2. In the Cyberduck client, configure a new Connection:

   Configuring a Connection sets up a new Bookmark in Cyberduck.

   Bookmarks are listed by their Nicknames at the end of the
   **Bookmark** submenu.
   1. Go to one of:
      * **File > Open Connection** or click **Open Connection +**.

        The **Open Connection** dialog displays.
      * **Bookmark > New**.

        The **Bookmark** dialog displays.
   2. From the protocol dropdown, select **Amazon S3**.

      If **Amazon S3** does not appear in the dropdown’s initial list,
      click **More Options…** to display, locate and select it.
   3. From the Zadara Object Storage UI’s **Account Information**
      section, copy the following mandatory values to their equivalent
      fields in the Cyberduck **Open Connection** or **Bookmark**
      dialog window:

      |                                   | Copy from Zadara Object Storage UI   | Paste to Cyberduck         |
      |-----------------------------------|--------------------------------------|----------------------------|
      | Page Section                      | Field                                | Open Connection / Bookmark |
      | **Connectivity - Public Network** | Public API Endpoint                  | Server                     |
      | **Authentication**                | S3 Access Key                        | Access Key ID              |
      | **Authentication**                | S3 Secret Key                        | Secret Access Key          |
   4. To save the Connection configuration:
      * **Open Connection** dialog:

        Click **Connect**.

        The dialog closes, and the root folder of the connected Zadara
        Object Storage container appears in Cyberduck’s main view.

        You can expand the root folder to navigate, explore and manage
        the container’s branches and objects.
      * **Bookmark** dialog:

        Close the dialog window.

        The new Connection is listed in Cyberduck’s main view,
        and at the end of Bookmark submenu.

        To navigate, explore and manage the Zadara Object Storage
        container’s branches and objects, select the newly listed
        Bookmark.

<a id="ngos-cyberduck-configure-region-preferences"></a>

#### Configuring a Cyberduck Custom S3 Region

To configure a custom region:

1. In the Cyberduck client, go to **Edit > Preferences**.

   The **Preferences** dialog opens.
2. In the **Preferences** dialog, click **S3**.

   From the **Default Bucket Location** dropdown, select the
   Region that matches the **Region** in the Authentication
   section of your Zadara Object Storage UI.

   If your Zadara Object Storage’s **Region** is set to a custom value
   that is **not** listed in Cyberduck’s **Default Bucket Location**
   dropdown:
   1. Close the Cyberduck client.
   2. Contact your Object Storage Administrator
      to refer to
      [Creating a Custom S3 Region Configuration for Cyberduck (Admin)](#ngos-create-custom-cyberduck-s3-region-config).

      The Object Storage Administrator will provide you
      with a `.cyberduckprofile` configuration file, that will enable
      access to your Zadara Object Storage from Cyberduck.
      1. Copy the `.cyberduckprofile` configuration file to the
         Cyberduck installation on your client machine:
         * Windows: `%APPDATA%\Cyberduck\Profiles\`
         * macOS: `~/Library/Group Containers/G69SCX94XU.duck/Library/Application Support/duck/Profiles/`
      2. Reopen the Cyberduck client, and go to **Edit > Preferences**.

         In the **Preferences** dialog, click **S3**, and in the
         **Default Bucket Location** dropdown, locate and select the
         matching new custom Region.
3. Close the **Preferences** dialog window to save the Region
   configuration.

   Continue with [Configuring a Cyberduck S3 Connection to Zadara Object Storage](#ngos-cyberduck-configure-connection).

<a id="ngos-create-custom-cyberduck-s3-region-config"></a>

#### Creating a Custom S3 Region Configuration for Cyberduck (Admin)

Scope: Object Storage Administrator

To connect Cyberduck successfully to a Zadara Object Storage container,
the Region settings in the Cyberduck client and the Zadara Object Storage
must be identical.

A custom Zadara Object Storage Region requires a customized Cybderduck
`.cyberduckprofile` configuration file, with customized parameter values
that are accessed in the Cyberduck client Preferences.

To create a `.cyberduckprofile` configuration file:

1. In the Zadara Object Storage UI, navigate to **System > Settings**.
2. In the **General** section of the **General & Connectivity** tab,
   click **Edit** to the right of **Region**.
   * **Region**: Enter a custom name for the Region, for example, zadara.
   * **Advanced: Honor custom AWSV4 Region field provided by the client**:

     Mark the checkbox.

   Click **Save**.
3. Create a Cyberduck profile XML text file.
   1. Name the profile file with a meaningful prefix, and the suffix
      `.cyberduckprofile`, for example, `zadara.cyberduckprofile`.
   2. Edit the file, making sure that it includes the correct Hostname,
      and optionally the custom Region.

      For example:
      ```console
      <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
      <plist version="1.0">
      <dict>
         <key>Protocol</key>              <string>s3</string>
         <key>Vendor</key>                <string>Zadara</string>
         <key>Default Nickname</key>      <string>S3 - zadara - vsa-00000001 - yuser1@NGOS123</string>
         <key>Description</key>           <string>Zadara Object Storage - S3 Compatible</string>
         <key>Default Hostname</key>      <string>vsa-00000001-public.zadarazios.com</string>
         <key>Hostname Configurable</key> <false/>
         <key>Default Port</key>          <string>443</string>
         <key>Port Configurable</key>     <false/>
         <key>Username Configurable</key> <true/>
         <key>Password Configurable</key> <true/>
         <key>Authorization</key>         <string>AWS4HMACSHA256</string>
         <key>Region</key>                <string>zadara</string>
         <key>Path Style</key>            <true/>
      </dict>
      </plist>
      ```
4. Test connectivity in the Cyberduck client.
   1. Close the Cyberduck client if it was open.
   2. Copy the `.cyberduckprofile` configuration file to the
      Cyberduck installation on your client machine:
      * Windows: `%APPDATA%\Cyberduck\Profiles\`
      * macOS: `~/Library/Group Containers/G69SCX94XU.duck/Library/Application Support/duck/Profiles/`
   3. Open the Cyberduck client.
      1. Go to **Edit > Preferences**.

         The **Preferences** dialog opens.
      2. In the **Preferences** dialog, click **S3**.

         From the **Default Bucket Location** dropdown, select the
         Region that matches the **Region** in the Authentication
         section of your Zadara Object Storage UI.
      3. Close the **Preferences** dialog window to save the Region
         configuration.
      4. Go to **File > Open Connection** or click **Open Connection +**.

         The **Open Connection** dialog displays.
         1. Select your custom profile from the Protocol dropdown list.

            If it does not appear in the dropdown list,
            click **More Options…** to locate and select it.
         2. Click **Connect**.
         3. At the prompt, enter your Access Key and Secret Key.

            The dialog closes, and the root folder of the connected Zadara
            Object Storage container appears in Cyberduck’s main view.

            You can expand the root folder to navigate, explore and manage
            the container’s branches and objects.
5. Share the `.cyberduckprofile` configuration with Cyberduck users.

### Rclone

Starting from version v1.74, Zadara Object Storage is supported as a native
backend in [Rclone](https://rclone.org/), an open-source command-line
program for managing files on cloud storage.

For full configuration details and usage examples, refer to the official
Rclone documentation: [Rclone S3 - Zadara](https://rclone.org/s3/#zadara).

## OpenStack Swift Interface

The management interface generates a new Swift API token upon login. This
means that if you logout and login again you’ll notice a new token.
The token presented by the management interface is always the latest and valid
to be used.

The API tokens created by the management interface are generated based on
the Object Storage global configuration for token validity (default: 24 hours).

Example of validating an API token using the CLI:

```bash
# Get two consecutive API tokens from the management interface and store it
$ TOKEN1=gAAAAABiuwu4P55M2V...
$ TOKEN2=gAAAAABiuwvc8BEYc8...

$ curl -X GET -H "Content-Type: application/json" \
  -H "X-Access-Key: $TOKEN1"  \
  "https://<object storage endpoint>:8443/api/zios/accounts/AUTH_<account ID>/users.json"

{"response":{"users":[],"count":0,"status":0}}%

curl -X GET -H "Content-Type: application/json" \
 -H "X-Access-Key: $TOKEN2" \
 "https://<object storage endpoint>:8443/api/zios/accounts/AUTH_<account ID>/users.json"

{"response":{"users":[],"count":0,"status":0}}%

# We can validate these tokens using the Openstack Keystone auth service as well:

$ curl -s   -H "X-Subject-Token: $TOKEN2" -H "X-Auth-Token: $TOKEN1"  \
  "https://<object storage endpoint>:5000/v3/auth/tokens" | python3 -m json.tool
{
    "token": {
       "is_domain": false,
      "methods": [
         "password"
      ],
      "roles": [
         {
            "id": "fedeff6db6df47959e96d8dd33963cfe",
            "name": "zios_admin"
         }
      ],
      "expires_at": "2022-06-29T14:10:36.000000Z",
 ....
      "issued_at": "2022-06-28T14:10:36.000000Z"
   }
}
```

<!-- end -->

### cURL (swift API)

cURL can be used for Object Storage operations. The connectivity information is
available in the User Information view.

![curl_account_urls](media/curl_account_urls.png)

In this example, we will use the **Front End Network Account URL** or
**Public API Network Account URL**, and **API Token** in order to create a new container:

```console
$ curl -H "x-auth-token: <user_token>" -X PUT <account_url>/test-bucket/
```

<!-- end -->

For example:

```console
$ URL=<Front End Network Account URL or Public API Network Account URL>
$ TOKEN=<MYAPI TOKEN>
$ curl -H "x-auth-token: $TOKEN" -X PUT $URL/test-bucket/
```

<!-- end -->

The following example describes how to get the token programmatically using
the Swift API:

```console
$ curl -i -H "Content-Type: application/json" \
-d '{ "auth": \
{ "identity": { "methods": ["password"], "password": \
{ "user": {"name": "<USERNAME>", "domain": { "id": "default" }, \
"password": "<USER PASSWORD>" }} }, "scope": { "project": \
{ "name": "<ACCOUNT_NAME>", "domain": { "id": "default" } } } } }' \
"https://vsa-00000001-mycloud-01.zadara.com:5000/v3/auth/tokens" ;
```

<!-- end -->

and use the returned token for the subsequent API calls.

```console
HTTP/1.1 201 Created
Date: Thu, 19 Nov 2020 16:05:28 GMT
Server: Apache/2.4.29 (Ubuntu)
Content-Length: 1114
X-Subject-Token: gAAAAABftpfIAiuo2tRZZP8VVtomU1knVG7xNUONV4b2u....
```

#### Presigned URL for POST

The following example uses the boto3
[S3 Client generate_presigned_post](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/generate_presigned_post.html)
to build the URL and form fields used for a presigned S3 POST.

1. Create  a python script that uses boto3 functions to presign a URL for
   POST.

   In this example, the python script is `post_presign.py`.
   ```console
   # post_presign.py

   import boto3

   s3_client = boto3.client(
         's3',
         aws_access_key_id='<S3 Access Key>',
         aws_secret_access_key='S3 Secret Key',
         endpoint_url='https://vsa-00000103-public-zadara-qa19.zadarazios.com',
         region_name='us-east-1',
         config=boto3.session.Config(signature_version='s3v4')
   )
   url = s3_client.generate_presigned_post(
         Bucket='Container1',
         Key='obj1',
         ExpiresIn=3600
   )

   print(url)
   ```
2. Run the python script to presign the URL for POST:
   ```console
   python3 post_presign.py
   ```

   Response:
   ```console
   {
   'url': 'https://vsa-00000103-public-zadara-qa19.zadarazios.com:443/Container1',
   'fields': {
               'key': 'obj1',
               'x-amz-algorithm': 'AWS4-HMAC-SHA256',
               'x-amz-credentia_request',
               'x-amz-date': '20250217T055444Z',
               'policy': 'eyJleHBpcmF0aW9uIjogIjIwMjUtMDItMTdUMDY6NTQ6NDRaIiwgImNvbmRpdGlvbnMiOiBbeyJidWNrZXQiOiAidG1wdGVzdCJ9LCNIQTI1NiJ9LCB7IngtYW16LWNyZWRlbnRpYWwiOiAiOTZhYzdiYzdmZGEzNGMxYzllZDA0YmRiMzkxNmNhMjAvMjAyNTAyMTcvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LCB7IngtYW16LWRhdGUiOiAi35fab89dde858fc3046c37fa6a340f7159166c2a8422701e6f4'
               }
   }
   ```
3. Using cURL, upload the object to the presigned URL,
   using the parameter values returned in the response from the script:
   ```console
   curl -i 'https://vsa-00000103-public-zadara-qa19.zadarazios.com:443/Container1' \
        -F 'key=obj1' -F 'x-amz-algorithm=AWS4-HMAC-SHA256' \
        -F 'st-1/s3/aws4_request' \
        -F 'x-amz-date=20250217T055444Z' \
        -F 'policy=eyJleHBpcmF0aW9uIjogIjIwMjUtMDItMTdUMDY6NTQ6NDRaIiwgImNvbmRpdGlvbnMiOiBbeyJidWNrZXQiOiAidG1wNC1ITUFDLVNIQTI1NiJ9LCB7IngtYW16LWNyZWRlbnRpYWwiOiAiOTZhYzdiYzdmZGEzNGMxYzllZDA0YmRiMzkxNmNhMjAvMjAyNTAyMTcvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LCB7IngtYW16LW6c5398a1f35fab89dde858fc3046c37fa6a340f7159166c2a8422701e6f4'
   ```
4. Verify the upload by listing the objects in the container:
   ```console
   aws s3api list-objects \
   --bucket Container1 \
   --profile zadara
   --endpoint https://vsa-00000103-public-zadara-qa19.zadarazios.com:443
   ```

   Response:
   ```console
    {
      "Contents": [
         {
            "Key": "obj1",
            "LastModified": "2025-02-17T05:57:43.000Z",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "Size": 65959,
            "StorageClass": "STANDARD",
            "Owner": {
               "DisplayName": "<NGOS account:username>",
               "ID": "<NGOS account:username>"
            }
         }
      ],
      "RequestCharged": null,
      "Prefix": ""
   }
   ```

Additional examples of using the OpenStack Swift API can be found at the
[OpenStack Swift API documentation](https://docs.openstack.org/api-ref/object-store/).

### Cloudberry Explorer for OpenStack (v3 authentication)

Use the logged-in User Information properties to set the authentication
fields of Cloudberry Explorer

![image81](media/image81.png)

### Cyberduck for OpenStack Swift

**Cyberduck Version**: 9.1.7 (43177)

Cyberduck is a graphical client that supports the “OpenStack Swift (Keystone 3)”
API interface and is compatible with Zadara Object Storage.

To connect to your Zadara Object Storage using the Cyberduck client, use
the V3 Auth Endpoint URL, Account and Username from your Zadara Object
Storage account’s information page.

1. At the top right of the Zadara Object Storage UI, click your login
   username.

   The user profile, account, credentials and connectivity details appear.
2. In the Cyberduck client:
   1. Go to **File > Open Connection** or click **Open Connection +**.

      The **Open Connection** dialog displays.
   2. From the protocol dropdown, select **OpenStack Swift (Keystone 3)**.
   3. Use the Zadara Object Storage’s logged-in User Information
      properties to set the authentication fields of the Cyberduck client.
      1. Server - the Object Storage v3 Auth Endpoint.
      2. Port - 5000
      3. Project:Domain:Username - <Object Storage Account>:default:<Object Storage Username>

      ![cyber-duck-config](media/cyber-duck-config.png)