edit

S3-API Related Operations

Prior Knowledge

When choosing Amazon S3 API1 as LeoGateway's protocol, you need LeoFS' S3-API related operations including Endpoint, User, and Bucket operations.

You can access a LeoFS system whose LeoGateway's protocol is S3-API by following the flow.

  1. Create an endpoint to be able to access a LeoFS system's resources.
  2. Create a user to separately manage the user's buckets and objects.
  3. After creating a bucket, If you need to change ACL of a bucket, you can set it as an administrator of a LeoFS system. The current version, v1.3 of LeoFS does not support

Operations

Endpoint

You can continue to use a LeoFS system's DNS name to access its resources after you've set up an endpoint, which has a policy that controls the use of the endpoint to access the resources.

Create an endpoint, add-endpoint

Creates an endpoint so users can access a LeoFS system's resources.

1
2
3
4
5
$ leofs-adm add-endpoint <endpoint>

## Example
$ leofs-adm add-endpoint leo-project.net
OK

Remove an endpoint, delete-endpoint

Removes an endpoint if a LeoFS system's system does not need it.

1
2
3
4
5
$ leofs-adm delete-endpoint <endpoint>

## Example
$ leofs-adm delete-endpoint leo-project.net
OK

Retrieve a list of endpoints, get-endpoints

Retrieves a list of the endpoints if you need to know the existing endpoints.

1
2
3
4
5
6
7
8
9
$ leofs-adm get-endpoints

## Example
$ leofs-adm get-endpoints
endpoint         | created at
-----------------+---------------------------
leo-project.net  | 2017-04-07 10:07:31 +0900
localhost        | 2017-04-07 10:06:18 +0900
s3.amazonaws.com | 2017-04-07 10:06:18 +0900

User

To access a LeoFS system, its system's users need to create an account and retrieve AccessKeyID and SecretAccessKey.

Create a user, create-user

Creates a user's account so its user can access a LeoFS system.

1
2
3
4
5
6
$ leofs-adm create-user <user-id> <password>

## Example
$ leofs-adm create-user leofs-user-1 PASSWORD
access-key-id: b5f0413d45855fcc055e
secret-access-key: b74f9ae91dd0de81f71e19f8d455a7c081f34c57

Remove a user, delete-user

Removes a user account if a LeoFS system does not need it.

1
2
3
4
5
$ leofs-adm delete-user <user-id>

## Example
$ leofs-adm delete-user leofs-user-1
OK

Retrieve a list of users, get-users

Retrieves a list of the users if you need to know the existing users.

1
2
3
4
5
6
7
8
$ leofs-adm get-users

## Example
$ leofs-adm get-users
user_id      | role_id | access_key_id          | created_at
-------------+---------+------------------------+---------------------------
_test_leofs  | 9       | 05236                  | 2017-04-10 09:52:39 +0900
leofs-user-2 | 1       | b5f0413d45855fcc055e   | 2017-04-10 09:54:35 +0900

Change role of a user, update-user-role

Updates a user's role if you need to change the role of a LeoFS system. You can choose a role from Administrator or General user.

Role Id Description
1 Administrator
9 General User
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
$ leofs-adm update-user-role <user-id> <role-id>

## Example
$ leofs-adm update-user-role leofs-user-2 9
OK

## Confirm the latest list of the users
$ leofs-adm get-users
user_id      | role_id | access_key_id          | created_at
-------------+---------+------------------------+---------------------------
_test_leofs  | 9       | 05236                  | 2017-04-10 09:52:39 +0900
leofs-user   | 1       | 3c1d889cdd2088fb9482   | 2017-04-10 09:53:33 +0900
leofs-user-2 | 9       | b5f0413d45855fcc055e   | 2017-04-10 09:54:35 +0900

Bucket

To manage objects under a bucket, you need to create a bucket with leofs-adm or any client for Amazon S3 such as s3cmd2 and DragonDisk3.

Create a bucket, add-bucket

Creates a bucket so a LeoFS system' users can manage resources under its bucket.

1
2
3
4
5
$ leofs-adm add-bucket <bucket-name> <access-key-id>

## Example
$ leofs-adm add-bucket backup 05236
OK

Note: Creating bucket with the same name as one being removed

It's not possible to create bucket with the same name as one currently being removed. Deletion of all objects belonging to former bucket must complete before bucket with the same name can be created again.

Remove a bucket, delete-bucket

Removes a bucket if its user does not need it. The bucket name is removed right away, while objects from that bucket will be deleted in asynchronous fashion after that.

1
2
3
4
5
$ leofs-adm delete-bucket <bucket-name> <access-key-id>

## Example
$ leofs-adm delete-bucket backup 05236
OK

Note: How many delete-bucket requests can be proceeded at once

8 delete-bucket requests can be proceeded at once and if there are over 8 ones then exceeded ones are pushed on a queue and will be proceeded later once the number of processing delete-bucket go down under 8.

Retrieve a list of the ongoing delete-buckets, delete-bucket-stats

Retrieves a list of the ongoing delete-buckets if you need to know the progress of the ongoing delete-buckets.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ leofs-adm delete-bucket-stats <bucket-name>

## Example
$ leofs-adm delete-bucket-stats
- Bucket: test_2
node                         | state            | date
-----------------------------+------------------+-----------------------------
[email protected]          | pending          | 2017-07-27 09:20:47 +0900
[email protected]          | monitoring       | 2017-07-27 09:21:21 +0900
[email protected]          | monitoring       | 2017-07-27 09:21:21 +0900
[email protected]          | monitoring       | 2017-07-27 09:21:22 +0900

- Bucket: test_4
node                         | state            | date
-----------------------------+------------------+-----------------------------
[email protected]          | finished         | 2017-07-27 09:21:20 +0900
[email protected]          | finished         | 2017-07-27 09:21:18 +0900
[email protected]          | finished         | 2017-07-27 09:21:21 +0900
[email protected]          | monitoring       | 2017-07-27 09:21:10 +0900

## Example (No ongoing delete-buckets)
$ leofs-adm delete-bucket-stats
[ERROR] Delete-bucket's stats not found

The below table explains what each state means.

State Description
pending Waiting on the queue.
enqueuing Iterating all objects and enqueuing object identifiers into the queue if those belong to the bucket deleted.
monitoring Iterating all items in the queue and deleting objects one by one.
finished Finished delete-bucket and waiting for other storage nodes.

Note: How long the delete-bucket records exist

Once the state of every storage node transit into finished, the delete-bucket records get deleted within several tens of seconds.

Reset a delete-bucket-stats record, reset-delete-bucket-stats

Reset a delete-bucket-stats record if you face some trouble that inconsistency between manager nodes and storage nodes happened and only way to solve this problem is to reset(delete) a delete-bucket-stats record.

1
2
3
4
5
$ leofs-adm reset-delete-bucket-stats <bucket-name>

## Example
$ leofs-adm reset-delete-bucket-stats bucket
OK

Note: Removing inaccessible objects after reset-delete-bucket-stats

Since this operation aborts deletion of objects belonging to (already deleted) bucket, it might be a good idea to create and delete bucket with same name again after reset-delete-bucket-stats, to make sure that bucket deletion completes properly and no stale objects remain.

Retrieve a list of buckets, get-buckets

Retrieves a list of the buckets if you need to know the existing buckets.

1
2
3
4
5
6
7
$ leofs-adm get-buckets

## Example
$ leofs-adm get-buckets
cluster id   | bucket   | owner       | permissions      | redundancy method            | created at
-------------+----------+-------------+------------------+------------------------------+---------------------------
leofs_1      | test     | _test_leofs | Me(full_control) | copy, {n:1, w:1, r:1, d:1}   | 2017-04-10 10:57:29 +0900

Retrieve a bucket, get-bucket

Retrieves a bucket if you need to know its bucket's information.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
$ leofs-adm get-bucket <access-key-id>

## Example
$ leofs-adm get-bucket 05236
bucket   | permissions                            | created at
---------+----------------------------------------+---------------------------
backup   | Me(full_control), Everyone(read)       | 2017-04-10 10:39:01 +0900
docs     | Me(full_control), Everyone(read)       | 2017-04-10 10:39:25 +0900
logs     | Me(full_control), Everyone(read,write) | 2017-04-10 10:39:38 +0900
movie    | Me(full_control)                       | 2017-04-10 10:39:45 +0900

Change an owner of a bucket, chown-bucket

Updates an owner of a bucket if you need to change its owner to someone else.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
$ leofs-adm chown-bucket <bucket> <access-key-id>

## Example
$ leofs-adm chown-bucket test b5f0413d45855fcc055e
OK

## Confirm the latest list of the buckets
$ leofs-adm get-buckets
cluster id   | bucket   | owner        | permissions      | redundancy method            | created at
-------------+----------+--------------+------------------+------------------------------+---------------------------
leofs_1      | test     | leofs-user-2 | Me(full_control) | copy, {n:1, w:1, r:1, d:1}   | 2017-04-10 10:57:29 +0900

Update ACL of a bucket, update-acl

Updates ACL of a bucket if you need to change it.

1
2
3
4
5
6
7
8
9
$ leofs-adm update-acl <bucket> <access-key-id> <canned-ACL>

## Example
$ leofs-adm update-acl test 05236 private
ok
$ leofs-adm update-acl test 05236 public-read
ok
$ leofs-adm update-acl test 05236 public-read-write
ok
Canned ACL

When using S3-API, LeoFS supports a set of predefined grants, known as canned ACLs. Each canned ACL has a predefined a set of grantees and permissions. The following table lists the set of canned ACLs and the associated predefined grants.

Canned ACL Applies To Permissions Added To ACL
private Bucket and object [default] Owner gets FULL_CONTROL. No one else has access rights.
public-read Bucket and object Owner gets FULL_CONTROL. The AllUsers group gets READ access.
public-read-write Bucket and object Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. Granting this on a bucket is generally not recommended.