Console¶

Scope: Object Storage Administrator Account Administrator Account Member

The Object Storage Console provides management access for Object Storage accounts. It is not a tool for read/write operations from/to the object storage. With the console, you can create/delete and view containers, and list their content, create and delete folders to better organize the objects, and set permissions and other management configurations.

The Object Storage Console Window¶

The Console Window comprises:

Containers pane - list the containers in the Object Storage.
Folders / Objects pane - Note that the view shows either folders or objects. Click on a container to display the content folders. Click on a folder to display the content objects. To return from object view to folder view, or in the case of nested folders, to move back within the folder hierarchy click on .. above the object pane.
Details pane - shows different properties and permissions depending on whether a container, folder, or object is selected in the top pane.

Encrypted Containers¶

Encryption management of Data-at-Rest (data on the disk drives) is applied by the Object Storage on a per-container basis. Encrypted and unencrypted containers can coexist in the same account.

An Object Storage generates a random 256-bit unique Encryption Key per encrypted container and uses the Advanced Encryption Standard (AES) to encrypt and decrypt the objects’ data.

The Encryption Keys are stored on disk as ciphertext, using AES with a 256-bit Master Encryption Key, which is generated from a user-supplied Master Encryption Password.

The user owns the Master Encryption Password. It is never stored on any persistent media. Instead, only its SHA3 hash-sum is saved on disk for password validation.

Caution

Since the system does not keep the Master Encryption Password, you are fully responsible to retain and protect the Master Encryption Password.

During Object Storage operation, the Master Encryption Password itself is held in kernel memory of the Object Storage. Core-dumping any User Mode process within the Object Storage will not reveal the Master Encryption Key.

This method ensures that encrypted Data-at-Rest cannot be accessed without explicitly knowing the user-supplied Master Encryption Password, thus providing you full protection if you enable Data-at-Rest Volume encryption.

Caution

The encryption attribute of a container cannot be changed! If you’d like to encrypt the objects of a non-encrypted container, or vice versa, you will need to create a new container and copy the data.

Setting Encryption Password¶

Scope: Object Storage Administrator

To create a Master Encryption Password, go to the Settings page, Security tab and click Edit in the Encryption section. Read the instructions and warning. Enter your password and click Save.

Store your Master Encryption Password in a secure place.

Create Containers¶

To create a new container in the account, open the console, and click Add in the toolbar above the console pane. The Create Container dialog will open.

Enter the following information:

Name - Enter container name. Note the Object Storage is both S3 and Swift API protocol compatible. S3 containers are expected to contain only lowercase letters, numbers, periods and dashes. The Swift API is less restrictive; container names can start with any character and contain any pattern. The container name cannot contain a slash (/) character because this character delimits the container and object name. The creation wizard will verify the proposed container name, a warning message will be displayed if a non-s3 compatible name was chosen. This restriction can be overridden by checking the Override S3 Naming Rules option.
Storage Policy - the target storage policy for the container creation
Encrypted - select if the container should be encrypted.
Object Lock - selecting object lock will prevent the deletion or modification of any object prior to its retention period expiration. For more information on Object Lock, see Object Lock Containers.

Click Create. The new container will be displayed in the Containers pane.

Delete Containers¶

To remove a container, open the console, go to the containers pane, select the container to be deleted and click Delete. The system will prompt you for deletion confirmation.

Delete a non-empty container¶

Object-Lock enabled container which contains objects cannot be deleted from the console. In order to delete it all contents (including object versions) should be deleted. Empty container can be deleted from the console.
Container without object-lock enabled can be deleted directly from the console, once the operation is confirmed by the user, the system will empty the container’s data and will delete the container itself.

Note

The delete operation of a container may take a while to complete. During the delete operation all new writes to the container will fail. Once the operation completes, the container will no longer be listed in the console.

Caution

After deletion confirmation, the container with all its content will be deleted. This operation is permanent and the data can not be recovered.

Adding folders¶

By definition, containers are flat and there is no hierarchy structure for storing the objects. However, since many users are accustomed to the folders tree concept of file systems, Object Storage Console gives you an option to simulate a hierarchical structure within the Object Storage Containers.

To create a folder, open the console, select a container in the containers pane, navigate to the hierarchy level where you want to create the new folder, and click Add Folder. Give it a name and click Submit.

Navigation within the container’s folders tree is done in a way similar to the common user experience of file systems explorer. By double clicking a folder, you enter it and see its content (objects and sub-folders). By double clicking the .. at the top of the objects pane, you navigate one level up to the parent Folder. The path indicator above the objects pane always show you current position in the tree.

Removing folders¶

To remove a folder, navigate to its parent folder, select the folder to be removed and click Delete from the toolbar above the folder pane.

After confirmation, the folder with all its content will be deleted.

Details Pane¶

The details pane at the bottom of the console screen includes the following tabs:

Properties - displays read only properties of selected container, folder, or object.
HTTP Headers - display, edit, add, or delete the HTTP headers used in the object storage operations.
Permissions - for information on assigning permissions to containers, see Setting Container Permissions (Account Admin).
Versioning - if enabled, versioning supports storing multiple versions of an object in the same container thus allowing recovery from unintended actions or failures. Note that once versioning is enabled for a container it cannot be disabled.
Versions - displays versions for all folders and objects within a container. Note this tab is only available for containers for which versioning is enabled.
Event Log - displays log of events related to selected container.

Versioned Container¶

Zadara’s Object Storage supports container versioning which is the ability to maintain multiple versions of an object instance in the same container.

Object Storage PUT operation which normally would replace (overwrite) an existing object would yield an additional a new object version. The versioning mechanism intends to protect against unintended deletion.

Versioning is a pre-requisite for enabling Object Lock Containers.

“Versioned” container would add a version ID attribute to each object and a “Latest” flag to the most recent object’s version which is retrieved by default.

S3 API allows the user to list, get or delete a specific version while using its version ID.

Enable Versioning¶

Post creating a new bucket, navigate to the south pane of the console and open the versioning tab.

Deleting an object¶

Deleting an object in a container with versioning enabled would yield a new 0-byte version of the same object with a a DeleteMarker tombstone. The DeleteMarker version will list the object as deleted however, all previous versions are retained.

In the following example, we have a container with two versions of the same object:

$ aws s3api list-object-versions --endpoint-url=$ENDPOINT --profile=$PROFILE \
  --bucket $BUCKET --output yaml

Versions:
- ETag: '"49e5c77426e2e3f5b635f7965f0020e3"'
  IsLatest: true
  Key: my-data.csv
  LastModified: '2023-04-11T21:07:45.053000+00:00'
  Owner:
    DisplayName: <truncated>
    ID: <truncated>
  Size: 22032
  StorageClass: STANDARD
  VersionId: '1681247265.05321'
- ETag: '"49e5c77426e2e3f5b635f7965f0020e3"'
  IsLatest: false
  Key: my-data.csv
  LastModified: '2023-04-11T21:05:02.306000+00:00'
  Owner:
    DisplayName: <truncated>
    ID: <truncated>
  Size: 22032
  StorageClass: STANDARD
  VersionId: '1681247102.30656'

Deleting the object without specifying a version ID would remove the object from the bucket listing while retaining the existing versions.

Delete the object:

$ aws s3api delete-object --endpoint-url=$ENDPOINT --profile=$PROFILE  --bucket $BUCKET --key my-data.csv

Listing the container’s object would output an empty list.

$ aws s3api list-objects --endpoint-url=$ENDPOINT --profile=$PROFILE  --bucket $BUCKET --output yaml

While listing the object’s versions:

$ aws s3api list-object-versions  --endpoint-url=$ENDPOINT --profile=$PROFILE  --bucket $BUCKET --output yaml

DeleteMarkers:
- IsLatest: true
  Key: my-data.csv
  LastModified: '2023-04-11T21:14:41.045000+00:00'
  Owner:
    DisplayName: <truncated>
    ID: <truncated>
  VersionId: '1681247681.04512'
Versions:
- ETag: '"49e5c77426e2e3f5b635f7965f0020e3"'
  IsLatest: false
  Key: my-data.csv
  LastModified: '2023-04-11T21:07:45.053000+00:00'
  Owner:
    DisplayName: <truncated>
    ID: <truncated>
  Size: 22032
  StorageClass: STANDARD
  VersionId: '1681247265.05321'
- ETag: '"49e5c77426e2e3f5b635f7965f0020e3"'
  IsLatest: false
  Key: my-data.csv
  LastModified: '2023-04-11T21:05:02.306000+00:00'
  Owner:
    DisplayName: <truncated>
    ID: <truncated>
  Size: 22032
  StorageClass: STANDARD
  VersionId: '1681247102.30656'

Object Lock Containers¶

Zadara Object Storage Immutability ensures data integrity by stopping stored objects from being deleted or overwritten during a specific retention timeframe. When an object is locked in compliance mode, its retention mode can’t be changed, and its retention period can’t be shortened. Immutability ensures object version integrity and availability throughout the defined retention period.

This feature can be leveraged directly from the S3 Compatible backup software (i.e. Veeam Backup and Replication) to ensure the integrity and availability of the backup as required. A configuration guide for Veeam Backup & Replication can be found in Zadara’s Knowledge-Base portal.

S3 Object Lock¶

The NextGen Object Storage uses the S3 Object Lock feature (Compliance Mode) in order to set a retention period to a given object and mark it as an immutable object. Deleting an object will be blocked until its retention period has expired. Object Lock should be enabled when creating a new container, directly from the management interface or by using AWS S3 Tools (CLI/SDK).

Note

Retention ensures that an object is WORM protected (cannot be deleted or overwritten) for a defined period of time that can be extended. It can be expressed in seconds, days or years.
Object Lock Retention is set at the object level. Retention period on an object version can be set either explicitly on the object level, or through a container default setting. Configuring a default container setting is supported via an S3 compatible client. A default lock configuration set at the container level does not apply retroactively to versions of objects created earlier. It only applies to objects that are created from the time that the configuration is applied. When a retention period is applied to an object version explicitly, you specify a Retain Until Date for the object version. The Retain Until Date setting is stored in the object version’s metadata and ensures the object version’s integrity until the retention period expires. A retention period can be extended by submitting a new lock request (put-object-retention).
Zadara’s Object Storage supports Object Lock Compliance mode, which is restrictive. It cannot be undone within the retention period. As a result, no user including the object storage administrator user, will be able to delete objects during their retention period.
Although the system’s s3api allows setting Object Lock Governance mode, it assumes the same restrictions as Compliance mode.
Object Lock cannot be enabled for existing containers.

Enable Object Lock from the management interface¶

Object Lock can be enabled for a new container during its creation. In order to create a new container with Object Lock:

Log in to the management interface.
Navigate to the Object Storage Console section.
In the upper options menu, click on the Add button.
Enter a new container name.
Check the “Object Lock” option.
Clicking Create to create the new container.

On creation, the Versioning feature will be automatically enabled for the new container.

Note

Automatic enabling of versioning for the new container could lead to additional storage consumption. Object Lock will prevent the deletion or modification of any object prior to its retention period expiry.

A container’s Object Lock property set to “true” identifies the container as being Object Lock enabled.

Enable Object Lock using the AWS S3 CLI¶

In the following examples, Object Lock is enabled using AWS Tools for PowerShell.

Currently, Object Lock can be enabled and reviewed only from the NextGen Object Storage S3 API interface.

Note

The following examples use PowerShell syntax. Equivalent API calls will achieve the same results using the language of your choice.

Enabling Object Lock¶

Object Lock should be enabled on the container level at creation time. Object versioning will be enabled automatically.

Make sure that the Object Storage credentials are set.

Define the NextGen Object Storage as an endpoint:

$ENDPOINT="https://vsa-0000000b-zadara-qa13.zadara.com"

Container creation

$BUCKET="immutable-container"
aws s3api --endpoint-url=$ENDPOINT create-bucket --bucket $BUCKET --object-lock-enabled-for-bucket

The expected result should be:

{
    "Location": "/immutable-container"
}

Confirm that Object Lock was enabled for the newly created container**

aws s3api --endpoint-url=$ENDPOINT get-object-lock-configuration --bucket $BUCKET

The expected result should be:

{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled"
    }
}

Upload a new object

$OBJECT="new-object-with-lock.log"
aws s3api --endpoint-url=$ENDPOINT put-object --bucket $BUCKET --key $OBJECT --body $OBJECT

#Response

{
   "ETag": "\"c6125a47483a2823d993da3d31ba6a50\"",
   "VersionId": "MzMxNjlmNzItOWQ3Ni00MWI0LTllOGYtZDQyN2RkMjRlN2Jk"
}

Set Object retention mode and date

aws s3api --endpoint-url=$ENDPOINT put-object-retention --bucket $BUCKET --key $OBJECT --retention Mode=COMPLIANCE,RetainUntilDate=2020-04-01

Retrieve Object Lock configuration

aws s3api --endpoint-url=$ENDPOINT get-object-retention --bucket $BUCKET --key $OBJECT

#Response

{
   "Retention": {
      "Mode": "COMPLIANCE",
      "RetainUntilDate": "2020-04-01T00:00:00"
   }
}

In this example, the object will remain locked until April 1st, 2020.

List an object’s versions and attempt to delete a specific version

List an object’s versions:

aws s3api --endpoint-url=$ENDPOINT list-object-versions --bucket $BUCKET  --prefix $OBJECT
{
   "Versions": [
      {
            "ETag": "%22c6125a47483a2823d993da3d31ba6a50%22",
            "Size": 14871255,
            "StorageClass": "STANDARD",
            "Key": "new-object-with-lock.log",
            "VersionId": "MzMxNjlmNzItOWQ3Ni00MWI0LTllOGYtZDQyN2RkMjRlN2Jk",
            "IsLatest": true,
            "LastModified": "2020-03-08T16:54:30.225Z",
            "Owner": {
               "DisplayName": "veeam:client",
               "ID": "veeam:client"
            }
      }
   ]
}

Attempt to delete a specific version of an object:

aws s3api --endpoint-url=$ENDPOINT delete-object --bucket=$BUCKET --key=$OBJECT --version-id=$VERSION

An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied.

Large objects support¶

Zadara’s Object Storage has a 5GB limit on the size of a single uploaded object. However, leveraging segmentation a single object size is unlimited. Segments of the larger object are uploaded and a special manifest file is created that, when downloaded, sends all the segments concatenated as a single object. This also offers much greater upload speed with the possibility of parallel uploads of the segments.

Dynamic Large Object (DLO) is supported out-of-the-box.

The majority of object storage clients support multi-part upload and the allows the user to set the segment size.

Failed upload handling¶

In case the multipart upload doesn’t complete, the NextGen Object Storage will not assemble the object parts and will not create any object. The parts will remain stored in the Object Storage for a period of 15 days, this until the Object Storage segment tracker will cleanup automatically the orphan parts. In this case aborted/failed uploads incomplete parts will be considered as part of <<<<<<< HEAD the account used capacity. ======= the account used capacity. >>>>>>> bbe335af (ngos - versioning and immutability)