LeoFS Storage Operation¶
LeoFS Storage’s Cluster Operation¶
- LeoFS Storage operation commands are executed on LeoFS-Manager console OR the
leofs-admscript. - Refer LeoFS operation flow diagram
| Shell | Description |
|---|---|
| leofs-adm detach <storage-node> |
|
| leofs-adm suspend <storage-node> |
|
| leofs-adm resume <storage-node> |
|
| leofs-adm start |
|
| leofs-adm rebalance |
|
detach <storage-node>¶
Remove the storage node from the storage cluster¶
$ leofs-adm detach [email protected]
OK
$ leofs-adm rebalance
OK
Method of cancellation of detach-operation¶
$ leofs-adm detach [email protected]
OK
### Rollback this operation
$ leofs-adm rollback [email protected]
OK
Method of deletion of an attached node¶
## Confirm current status of ther storage cluster
$ leofs-adm status
[System Confiuration]
-----------------------------------+----------
Item | Value
-----------------------------------+----------
Basic/Consistency level
-----------------------------------+----------
system version | 1.2.12
cluster Id | leofs_1
DC Id | dc_1
Total replicas | 2
number of successes of R | 1
number of successes of W | 1
number of successes of D | 1
number of rack-awareness replicas | 0
ring size | 2^128
-----------------------------------+----------
Multi DC replication settings
-----------------------------------+----------
max number of joinable DCs | 2
number of replicas a DC | 1
-----------------------------------+----------
Manager RING hash
-----------------------------------+----------
current ring-hash | 0364701d
previous ring-hash | 0364701d
-----------------------------------+----------
[State of Node(s)]
-------+--------------------------+--------------+----------------+----------------+----------------------------
type | node | state | current ring | prev ring | updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
S | [email protected] | attached | | | 2015-09-09 11:17:49 +0900
S | [email protected] | running | 0364701d | 0364701d | 2015-09-09 11:13:31 +0900
S | [email protected] | running | 0364701d | 0364701d | 2015-09-09 11:09:15 +0900
G | [email protected] | running | 0364701d | 0364701d | 2015-09-09 10:53:19 +0900
-------+--------------------------+--------------+----------------+----------------+----------------------------
## Remove an attached node
$ leofs-adm detach [email protected]
OK
## Confirm the current status of the storage cluster, again
$ leofs-adm status
[System Confiuration]
-----------------------------------+----------
Item | Value
-----------------------------------+----------
Basic/Consistency level
-----------------------------------+----------
system version | 1.2.12
cluster Id | leofs_1
DC Id | dc_1
Total replicas | 2
number of successes of R | 1
number of successes of W | 1
number of successes of D | 1
number of rack-awareness replicas | 0
ring size | 2^128
-----------------------------------+----------
Multi DC replication settings
-----------------------------------+----------
max number of joinable DCs | 2
number of replicas a DC | 1
-----------------------------------+----------
Manager RING hash
-----------------------------------+----------
current ring-hash | 0364701d
previous ring-hash | 0364701d
-----------------------------------+----------
[State of Node(s)]
-------+--------------------------+--------------+----------------+----------------+----------------------------
type | node | state | current ring | prev ring | updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
S | [email protected] | running | 0364701d | 0364701d | 2015-09-09 11:13:31 +0900
S | [email protected] | running | 0364701d | 0364701d | 2015-09-09 11:09:15 +0900
G | [email protected] | running | 0364701d | 0364701d | 2015-09-09 10:53:19 +0900
suspend <storage-node>¶
- Suspend the storage node
- While suspending, it rejects any requests
- This command does NOT detach the node from the storage cluster
$ leofs-adm suspend [email protected]
OK
start¶
- Start LeoFS after distributing the RING from LeoFS Manager to LeoFS Storage and LeoFS Gateway
$ leofs-adm start
OK
rebalance¶
- Commit detached and attached node(s) to update the cluster and Ring(routing-table)
- Rebalance objects in the cluster which is based on the updated cluster topology
$ leofs-adm rebalance
OK
Storage MQ Operation¶
Since¶
LeoFS v1.2.2
Overview¶
LeoFS Storage MQ is controllable mechanism manually. We’ve published mq-suspend and mq-resume command in leofs-adm script.
In addition, LeoFS’s MQ mechanism is affected by the watchdog mechanism to reduce comsumption of message costs.
Description of the each MQ¶
| Id | Description |
|---|---|
| leo_delete_dir_queue | After executed leofs-adm delete-bucket, messages of deletion object is added into the queue. |
| leo_comp_meta_with_dc_queue | After executed leofs-adm recover-cluster, messages of comparison of metadata w/remote-node is added into the queue. |
| leo_sync_obj_with_dc_queue | After executed leofs-adm recover-cluster, messages of synchronization of objects w/remote-node is added into the queue. |
| leo_recovery_node_queue | After executed leofs-adm recover-node, messages of recovery objects of a node is added into the queue. |
| leo_async_deletion_queue | After executed leofs-adm delete-bucket OR deletion of a bucket OR deletion of an object, message of async deletion of objs is added into the queue. |
| leo_rebalance_queue | After executed leofs-adm rebalance, messages of rebalance is added in to the queue. |
| leo_sync_by_vnode_id_queue | After executed leofs-adm rebalance, messages of synchronization of virtual-nodes is added into the queue. |
| leo_per_object_queue | After executed leofs-adm rebalance OR leofs-adm recover-file OR leofs-adm recover-node
OR fixing inconsistent object(s) with the recovery data mechanism, messages of recover inconsistent objects is added into the queue. |
Commands¶
| Shell | Description |
|---|---|
| leofs-adm mq-stats <storage-node> |
|
| leofs-adm mq-suspend <storage-node> <mq-id> |
|
| leofs-adm mq-resume <storage-node> <mq-id> |
|
mq-stats <storage-node>¶
You can check tatuses of the message queues in the LeoFS’s storage node. Explanation of columns:
| Column | Description |
|---|---|
| state | A status of the MQ - [idling, running, suspending] |
| number of msgs | A number of messages in the queue |
| batch of msgs | A batch of messages of the MQ’s message-consumption |
| interval | An interval time between the batch processing |
$ ./leofs-adm mq-stats [email protected]
id | state | number of msgs | batch of msgs | interval | description
--------------------------------+-------------+----------------|----------------|----------------|-----------------------------------
leo_delete_dir_queue | idling | 0 | 1000 | 100 | delete directories
leo_comp_meta_with_dc_queue | idling | 0 | 1000 | 100 | compare metadata w/remote-node
leo_sync_obj_with_dc_queue | idling | 0 | 1000 | 100 | sync objs w/remote-node
leo_recovery_node_queue | idling | 0 | 1000 | 100 | recovery objs of node
leo_async_deletion_queue | idling | 0 | 1000 | 100 | async deletion of objs
leo_rebalance_queue | running | 2167 | 1400 | 10 | rebalance objs
leo_sync_by_vnode_id_queue | idling | 0 | 1000 | 100 | sync objs by vnode-id
leo_per_object_queue | idling | 0 | 1000 | 100 | recover inconsistent objs
mq-suspend <storage-node> <mq-id>¶
Note
When turning on the watchdog mechanism, this command is ignored.
$ ./leofs-adm mq-suspend [email protected] leo_delete_dir_queue
OK
mq-resume <storage-node> <mq-id>¶
Note
When turning on the watchdog mechanism, this command is ignored.
$ ./leofs-adm mq-resume [email protected] leo_delete_dir_queue
OK