Console¶
Scope: Object Storage Administrator Account Administrator Account Member
The Object Storage Console provides management access for Object Storage accounts. It is not a tool for read/write operations from/to the object storage. With the console, you can create/delete and view containers, and list their content, create and delete folders to better organize the objects, and set permissions and other management configurations.
The Object Storage Console Window¶
The Console Window comprises:
Containers pane - list the containers in the Object Storage.
Folders / Objects pane - Note that the view shows either folders or objects. Click on a container to display the content folders. Click on a folder to display the content objects. To return from object view to folder view, or in the case of nested folders, to move back within the folder hierarchy click on .. above the object pane.
Details pane - shows different properties and permissions depending on whether a container, folder, or object is selected in the top pane.
Encrypted Containers¶
Encryption management of Data-at-Rest (data on the disk drives) is applied by the Object Storage on a per-container basis. Encrypted and unencrypted containers can coexist in the same account.
An Object Storage generates a random 256-bit unique Encryption Key per encrypted container and uses the Advanced Encryption Standard (AES) to encrypt and decrypt the objects’ data.
The Encryption Keys are stored on disk as ciphertext, using AES with a 256-bit Master Encryption Key, which is generated from a user-supplied Master Encryption Password.
The user owns the Master Encryption Password. It is never stored on any persistent media. Instead, only its SHA3 hash-sum is saved on disk for password validation.
Caution
Since the system does not keep the Master Encryption Password, you are fully responsible to retain and protect the Master Encryption Password.
During Object Storage operation, the Master Encryption Password itself is held in kernel memory of the Object Storage. Core-dumping any User Mode process within the Object Storage will not reveal the Master Encryption Key.
This method ensures that encrypted Data-at-Rest cannot be accessed without explicitly knowing the user-supplied Master Encryption Password, thus providing you full protection if you enable Data-at-Rest Volume encryption.
Caution
The encryption attribute of a container cannot be changed! If you’d like to encrypt the objects of a non-encrypted container, or vice versa, you will need to create a new container and copy the data.
Setting Encryption Password¶
Scope: Object Storage Administrator
To create a Master Encryption Password, go to the Settings page, Security tab and click Edit in the Encryption section. Read the instructions and warning. Enter your password and click Save.
Store your Master Encryption Password in a secure place.
Create Containers¶
To create a new container in the account, open the console, and click Add in the toolbar above the console pane. The Create Container dialog will open.
Enter the following information:
Name - Enter container name. Note the Object Storage is both S3 and Swift API protocol compatible. S3 containers are expected to contain only lowercase letters, numbers, periods and dashes. The Swift API is less restrictive; container names can start with any character and contain any pattern. The container name cannot contain a slash (/) character because this character delimits the container and object name. The creation wizard will verify the proposed container name, a warning message will be displayed if a non-s3 compatible name was chosen. This restriction can be overridden by checking the Override S3 Naming Rules option.
Storage Policy - the target storage policy for the container creation
Encrypted - select if the container should be encrypted.
Object Lock - selecting object lock will prevent the deletion or modification of any object prior to its retention period expiration. For more information on Object Lock, see Object Lock Containers.
Container logging - selecting container logging generates audit logs for all object operations.
Target Container - from the dropdown, select the target container for storing the audit logs.
Important
The target container requires write permissions.
Apply logging prefix - define a prefix to simplify locating and identifying log objects.
For more information, see Container logging.
Click Create. The new container will be displayed in the Containers pane.
Delete Containers¶
To remove a container, open the console, go to the containers pane, select the container to be deleted and click Delete. The system will prompt you for deletion confirmation.
Delete a non-empty container¶
Object-Lock enabled container which contains objects cannot be deleted from the console. In order to delete it all contents (including object versions) should be deleted. Empty container can be deleted from the console.
Container without object-lock enabled can be deleted directly from the console, once the operation is confirmed by the user, the system will empty the container’s data and will delete the container itself.
Note
The delete operation of a container may take a while to complete. During the delete operation all new writes to the container will fail. Once the operation completes, the container will no longer be listed in the console.
Caution
After deletion confirmation, the container with all its content will be deleted. This operation is permanent and the data can not be recovered.
Adding folders¶
By definition, containers are flat and there is no hierarchy structure for storing the objects. However, since many users are accustomed to the folders tree concept of file systems, Object Storage Console gives you an option to simulate a hierarchical structure within the Object Storage Containers.
To create a folder, open the console, select a container in the containers pane, navigate to the hierarchy level where you want to create the new folder, and click Add Folder. Give it a name and click Submit.
Navigation within the container’s folders tree is done in a way similar to the common user experience of file systems explorer. By double clicking a folder, you enter it and see its content (objects and sub-folders). By double clicking the .. at the top of the objects pane, you navigate one level up to the parent Folder. The path indicator above the objects pane always show you current position in the tree.
Removing folders¶
To remove a folder, navigate to its parent folder, select the folder to be removed and click Delete from the toolbar above the folder pane.
After confirmation, the folder with all its content will be deleted.
Details Pane¶
The details pane at the bottom of the console screen includes the following tabs:
Properties - displays read only properties of selected container, folder, or object.
HTTP Headers - display, edit, add, or delete the HTTP headers used in the object storage operations.
Permissions - for information on assigning permissions to containers, see Setting Container Permissions (Account Admin).
Quotas - quotas allow placement of limits on size (in GiB) per container (Console view) or per account (Account view). For more information on quotas, see Account Quota Management.
Versioning - if enabled, versioning supports storing multiple versions of an object in the same container thus allowing recovery from unintended actions or failures. Note that once versioning is enabled for a container it cannot be disabled.
Versions - displays versions for all folders and objects within a container. Note this tab is only available for containers for which versioning is enabled.
Container Logging - display, enable, edit or disable container logging settings. If container logging is enabled, audit logs are generated in the selected Target Container for all object operations. Apply logging prefix to simplify locating and identifying log objects. For more information, see Create Containers and Container logging.
Event Log - displays log of events related to selected container.
Versioned Container¶
Zadara’s Object Storage supports container versioning which is the ability to maintain multiple versions of an object instance in the same container.
Object Storage PUT operation which normally would replace (overwrite) an existing object would yield an additional a new object version. The versioning mechanism intends to protect against unintended deletion.
Versioning is a pre-requisite for enabling Object Lock Containers.
“Versioned” container would add a version ID attribute to each object and a “Latest” flag to the most recent object’s version which is retrieved by default.
S3 API allows the user to list, get or delete a specific version while using its version ID.
Enable Versioning¶
Post creating a new bucket, navigate to the south pane of the console and open the versioning tab.
Deleting an object¶
Deleting an object in a container with versioning enabled would yield a new 0-byte version of the same object with a a DeleteMarker tombstone. The DeleteMarker version will list the object as deleted however, all previous versions are retained.
In the following example, we have a container with two versions of the same object:
$ aws s3api list-object-versions --endpoint-url=$ENDPOINT --profile=$PROFILE \
--bucket $BUCKET --output yaml
Versions:
- ETag: '"49e5c77426e2e3f5b635f7965f0020e3"'
IsLatest: true
Key: my-data.csv
LastModified: '2023-04-11T21:07:45.053000+00:00'
Owner:
DisplayName: <truncated>
ID: <truncated>
Size: 22032
StorageClass: STANDARD
VersionId: '1681247265.05321'
- ETag: '"49e5c77426e2e3f5b635f7965f0020e3"'
IsLatest: false
Key: my-data.csv
LastModified: '2023-04-11T21:05:02.306000+00:00'
Owner:
DisplayName: <truncated>
ID: <truncated>
Size: 22032
StorageClass: STANDARD
VersionId: '1681247102.30656'
Deleting the object without specifying a version ID would remove the object from the bucket listing while retaining the existing versions.
Delete the object:
$ aws s3api delete-object --endpoint-url=$ENDPOINT --profile=$PROFILE --bucket $BUCKET --key my-data.csv
Listing the container’s object would output an empty list.
$ aws s3api list-objects --endpoint-url=$ENDPOINT --profile=$PROFILE --bucket $BUCKET --output yaml
While listing the object’s versions:
$ aws s3api list-object-versions --endpoint-url=$ENDPOINT --profile=$PROFILE --bucket $BUCKET --output yaml
DeleteMarkers:
- IsLatest: true
Key: my-data.csv
LastModified: '2023-04-11T21:14:41.045000+00:00'
Owner:
DisplayName: <truncated>
ID: <truncated>
VersionId: '1681247681.04512'
Versions:
- ETag: '"49e5c77426e2e3f5b635f7965f0020e3"'
IsLatest: false
Key: my-data.csv
LastModified: '2023-04-11T21:07:45.053000+00:00'
Owner:
DisplayName: <truncated>
ID: <truncated>
Size: 22032
StorageClass: STANDARD
VersionId: '1681247265.05321'
- ETag: '"49e5c77426e2e3f5b635f7965f0020e3"'
IsLatest: false
Key: my-data.csv
LastModified: '2023-04-11T21:05:02.306000+00:00'
Owner:
DisplayName: <truncated>
ID: <truncated>
Size: 22032
StorageClass: STANDARD
VersionId: '1681247102.30656'
Object Lock Containers¶
Zadara Object Storage Immutability ensures data integrity by stopping stored objects from being deleted or overwritten during a specific retention timeframe. When an object is locked in compliance mode, its retention mode can’t be changed, and its retention period can’t be shortened. Immutability ensures object version integrity and availability throughout the defined retention period.
This feature can be leveraged directly from the S3 Compatible backup software (i.e. Veeam Backup and Replication) to ensure the integrity and availability of the backup as required. A configuration guide for Veeam Backup & Replication can be found in Zadara’s Knowledge-Base portal.
S3 Object Lock¶
The NextGen Object Storage uses the S3 Object Lock feature (Compliance Mode) in order to set a retention period to a given object and mark it as an immutable object. Deleting an object will be blocked until its retention period has expired. Object Lock should be enabled when creating a new container, directly from the management interface or by using AWS S3 Tools (CLI/SDK).
Note
Retention ensures that an object is WORM protected (cannot be deleted or overwritten) for a defined period of time that can be extended. It can be expressed in seconds, days or years.
Object Lock Retention is set at the object level. Retention period on an object version can be set either explicitly on the object level, or through a container default setting. Configuring a default container setting is supported via an S3 compatible client. A default lock configuration set at the container level does not apply retroactively to versions of objects created earlier. It only applies to objects that are created from the time that the configuration is applied. When a retention period is applied to an object version explicitly, you specify a Retain Until Date for the object version. The Retain Until Date setting is stored in the object version’s metadata and ensures the object version’s integrity until the retention period expires. A retention period can be extended by submitting a new lock request (put-object-retention).
Zadara’s Object Storage supports Object Lock Compliance mode, which is restrictive. It cannot be undone within the retention period. As a result, no user including the object storage administrator user, will be able to delete objects during their retention period.
Although the system’s s3api allows setting Object Lock Governance mode, it assumes the same restrictions as Compliance mode.
Object Lock cannot be enabled for existing containers.
Enable Object Lock from the management interface¶
Object Lock can be enabled for a new container during its creation. In order to create a new container with Object Lock:
Log in to the management interface.
Navigate to the Object Storage Console section.
In the upper options menu, click on the Add button.
Enter a new container name.
Check the “Object Lock” option.
Clicking Create to create the new container.
On creation, the Versioning feature will be automatically enabled for the new container.
Note
Automatic enabling of versioning for the new container could lead to additional storage consumption. Object Lock will prevent the deletion or modification of any object prior to its retention period expiry.
A container’s Object Lock property set to “true” identifies the container as being Object Lock enabled.
Enable Object Lock using the AWS S3 CLI¶
In the following examples, Object Lock is enabled using AWS Tools for PowerShell.
Currently, Object Lock can be enabled and reviewed only from the NextGen Object Storage S3 API interface.
Note
The following examples use PowerShell syntax. Equivalent API calls will achieve the same results using the language of your choice.
Enabling Object Lock¶
Object Lock should be enabled on the container level at creation time. Object versioning will be enabled automatically.
Make sure that the Object Storage credentials are set.
Define the NextGen Object Storage as an endpoint:
$ENDPOINT="https://vsa-0000000b-zadara-qa13.zadara.com"
Container creation
$BUCKET="immutable-container"
aws s3api --endpoint-url=$ENDPOINT create-bucket --bucket $BUCKET --object-lock-enabled-for-bucket
The expected result should be:
{
"Location": "/immutable-container"
}
Confirm that Object Lock was enabled for the newly created container
aws s3api --endpoint-url=$ENDPOINT get-object-lock-configuration --bucket $BUCKET
The expected result should be:
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled"
}
}
Upload a new object
$OBJECT="new-object-with-lock.log"
aws s3api --endpoint-url=$ENDPOINT put-object --bucket $BUCKET --key $OBJECT --body $OBJECT
#Response
{
"ETag": "\"c6125a47483a2823d993da3d31ba6a50\"",
"VersionId": "MzMxNjlmNzItOWQ3Ni00MWI0LTllOGYtZDQyN2RkMjRlN2Jk"
}
Set Object retention mode and date
aws s3api --endpoint-url=$ENDPOINT put-object-retention --bucket $BUCKET --key $OBJECT --retention Mode=COMPLIANCE,RetainUntilDate=2020-04-01
Retrieve Object Lock configuration
aws s3api --endpoint-url=$ENDPOINT get-object-retention --bucket $BUCKET --key $OBJECT
#Response
{
"Retention": {
"Mode": "COMPLIANCE",
"RetainUntilDate": "2020-04-01T00:00:00"
}
}
In this example, the object will remain locked until April 1st, 2020.
List an object’s versions and attempt to delete a specific version
List an object’s versions:
aws s3api --endpoint-url=$ENDPOINT list-object-versions --bucket $BUCKET --prefix $OBJECT
{
"Versions": [
{
"ETag": "%22c6125a47483a2823d993da3d31ba6a50%22",
"Size": 14871255,
"StorageClass": "STANDARD",
"Key": "new-object-with-lock.log",
"VersionId": "MzMxNjlmNzItOWQ3Ni00MWI0LTllOGYtZDQyN2RkMjRlN2Jk",
"IsLatest": true,
"LastModified": "2020-03-08T16:54:30.225Z",
"Owner": {
"DisplayName": "veeam:client",
"ID": "veeam:client"
}
}
]
}
Attempt to delete a specific version of an object:
aws s3api --endpoint-url=$ENDPOINT delete-object --bucket=$BUCKET --key=$OBJECT --version-id=$VERSION
An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied.
Configure container’s default retention values with object lock¶
Via the CLI, there is the option to configure an object lock and its default retention values at the bucket level, in a single command.
Note
This option is only available via the CLI.
Configure a bucket’s object lock with default retention values
aws s3api --profile $AWSUSER --endpoint-url=$ENDPOINT put-object-lock-configuration --bucket $BUCKET --object-lock-configuration '{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 50 }}}'
Retrieve the bucket’s object lock configuration:
aws s3api --profile $AWSUSER --endpoint-url=$ENDPOINT get-object-lock-configuration --bucket $BUCKET
# Response
{
"ObjectLockConfiguration": {
"ObjectLockEnabled": "Enabled",
"Rule": {
"DefaultRetention": {
"Mode": "COMPLIANCE",
"Days": 50
}
}
}
}
Container logging¶
Version: 23.09
The Container Logging Feature is designed to provide logging capabilities for objects stored in the object storage system. This feature will provide insights into object-level activities and audit trails, which will help them understand access patterns, usage statistics, and security incidents.
Logging can be enabled at container level, for the purpose of auditing and analyzing all operations on objects.
Logs are generated into a separate target container in the same account. A container that has logging enabled cannot be used as the target for any audit logs, neither its own logs, nor audit logs for any other container.
A target container containing audit logs can be used to collect logs for multiple containers that have logging enabled. Write permissions on the target container are required.
Enabling logging is activated either when creating a container, or in the Container Logging tab in the details pane for existing containers. See Create Containers and Details Pane.
The configuration includes an option to define a logging prefix, so that locating and identifying log objects is simplified.
In order to avoid additional workload on the Object Storage platform and to avoid any impact on client performance, the logs will be delivered once a day. The interval for sending the logs to the target container will be shortened in the Object Storage next release.
There is no extra charge for enabling Container logging. However, the logs that are stored in the system will increase storage consumption. The logs collected and stored can be deleted at any time.
Enabling Container Logging¶
Container Logging can be enabled/disabled at any time from the following interfaces:
From Zadara’s Object Storage Object Storage management console
While using S3 API (aws cli package or S3 compatible software that supports enabling S3 Server Logging)
Log record structure¶
In order to make use of existing tools and avoid the overhead when using the audit logs Zadara log structure is similar to the AWS S3 Server Logging option, while not all attributes can match due to the differences in the platform, logs that doesn’t have a Zadara Object Storage presence or are not currently implemented will have the value of - (dash).
Log filenames have the format:
<user defined prefix>-<ISO formatted log delivery date>
Example (no user defined prefix):
2023-12-02-22-07-22
The log record structure:
Field name |
Description |
|---|---|
Container Owner |
The canonical user ID of the source container’s owner. |
Container |
The name of the container that the request was processed against. Malformed requests will not appear in the log. |
Time |
The date and time that the request was received in UTC format. In |
Remote IP |
The requester’s IP address. Proxies and firewalls might obscure the actual IP address of the requesting machine. |
Requester ID |
The user ID of the request, and “-” for unauthenticated requests. |
Request ID |
A generated string that uniquely identifies the request. |
Operation |
The object operation type: GET/POST/PUT/DELETE/HEAD |
Key |
The request’s object name (key). For example, |
Request-URI |
The Request-URI part of the HTTP request message. |
HTTP status |
The numeric HTTP status code of the response. |
Error Code |
The error code, or “-” if no error occurred. |
Bytes Sent |
The number of response bytes sent, excluding HTTP protocol overhead, or “-” if zero. |
Object Size |
The object’s total size in bytes. |
Total Time |
The number of milliseconds that the request was in flight from the server’s perspective, measured from the time that the request is received to the time that the last byte of the response is sent. Measurements made from the client’s perspective might be longer because of network latency. |
User-Agent |
The value of the HTTP |
Version Id |
The version ID in the request, or “-” if the operation doesn’t take a “versionId” parameter. |
Host Id |
The VC that handled the request. |
Container Logging log format examples¶
Container logging example for HEAD request
- savihou-manual [23/Nov/2023:02:23:44 +0000] 172.23.224.102 7c6d15bdf0d74802a9c751dd5c55bfa2 txe6a729193f9e479f876cd-00655eb7b0 HEAD - "HEAD / HTTP/1.1" 200 - - - 0.0106 - - "APN/1.0 Veeam/1.0 Backup/12.0" - - - - - - - - -
Container logging example for GET request
- savihou-manual [23/Nov/2023:06:58:01 +0000] 172.23.224.102 7c6d15bdf0d74802a9c751dd5c55bfa2 tx39562231422843d99a776-00655ef7f9 GET - "GET /?delimiter=%2F&max-keys=1000&prefix= HTTP/1.1" 200 - 320 - 0.0216 - - "S3 Browser/11.4.5 (https://s3browser.com)" - - - - - - - - -
Container logging example for PUT request
- savihou-manual [23/Nov/2023:02:24:36 +0000] 172.23.224.102 7c6d15bdf0d74802a9c751dd5c55bfa2 tx8ebb2ddeb8fc4535826de-00655eb7e4 PUT 10k-objects-1/randobj-828 "PUT /10k-objects-1/randobj-828 HTTP/1.1" 200 - - 4096 0.0712 - - "aws-sdk-go/1.45.18 (go1.10.4; linux; amd64)" - - - - - - - - -
Large objects support¶
Zadara’s Object Storage has a 5GB limit on the size of a single uploaded object. However, leveraging segmentation a single object size is unlimited. Segments of the larger object are uploaded and a special manifest file is created that, when downloaded, sends all the segments concatenated as a single object. This also offers much greater upload speed with the possibility of parallel uploads of the segments.
Dynamic Large Object (DLO) is supported out-of-the-box.
The majority of object storage clients support multi-part upload and the allows the user to set the segment size.
Failed upload handling¶
In case the multipart upload doesn’t complete, the NextGen Object Storage will not assemble the object parts and will not create any object. The parts will remain stored in the Object Storage for a period of 15 days, this until the Object Storage segment tracker will cleanup automatically the orphan parts. In this case aborted/failed uploads incomplete parts will be considered as part of the account used capacity.