Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Introduction

LeoFS is a highly scalable, fault-tolerant unstructured object storage system for the Web.

LeoFS provides High Cost Performance Ratio. It allows you to build LeoFS clusters using commodity hardware on top of the Linux operating system. LeoFS will provide very good performance even on commodity hardware. LeoFS will require a smaller cluster than other storage to achieve the same performance. LeoFS is also very easy to setup and to operate.

LeoFS provides High Reliability thanks to its great design on top of the Erlang/OTP capabilities. Erlang/OTP is known for being used in production systems for years with a solid nine nines (99.9999999%) availability, and LeoFS is no exception. A LeoFS system will stay up regardless of software errors or hardware failures happening inside the cluster.

LeoFS provides High Scalability. Adding and removing nodes is simple and quick, allowing you to react swiftly when your needs change. A LeoFS cluster can be thought as elastic storage that you can stretch as much and as often as you need.

LeoFS consists of 3 applications - LeoFS Storage, LeoFS Gateway and LeoFS Manager which depend on Erlang.

LeoFS Gateway handles http-request and http-response from any clients when using REST-API OR S3-API. Also, it is already built in the object-cache mechanism (memory and disk cache).

LeoFS Storage handles GET, PUT and DELETE objects as well as metadata. Also, it has replicator, recoverer and queueing mechanism in order to keep running a storage node and realise eventual consistency.

LeoFS Manager always monitors LeoFS Gateway and LeoFS Storage nodes. The main monitoring status are Node status and RING’s checksum in order to realise to keep high availability and keep data consistency.

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Introduction »

Goals

HIGH Reliability

Nine nines - Operating ratios is 99.9999999%

High Scalability

Build huge-cluster at low cost

HIGH Cost Performance

Fast - Over 10Gbps

A lower cost than other storages

Provide easy management and operation

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Introduction »

Milestones

v1.2 (DONE)

NFS Support (Beta)

Watchdog

Auto-compaction

v1.3 (WIP)

NFS Support (Stable)

Improve compatibility S3-APIs#6

AWS Signature v4 Support

Custom Metadata Support

v2.0

Improve performance of the list objects

Erasure Coding

Improve compatibility S3-APIs#7

Objects Expiration into the bucket

Versioning

Improve Web GUI Console (Option)

Published at 9th August, 2016

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Getting Started

Quick Start: All in one in a local node

Purpose

Install

Install LeoFS on CentOS-6.x

Install LeoFS on Ubuntu Server 12.04 LTS or Higher

Configuration

Modify “/etc/hosts”

Launch LeoFS’ managers and storage

Start the system

Start a LeoFS Gateway node

Confirm that the system is running

Get your S3 API Key from the LeoFS manager console

Clients

Using LeoFS

Wrap up

Quick Start: Building a cluster

Purpose

Case example

Install Erlang and LeoFS on each server

Configuration - Edit “leo_*.conf” on each server

Configuration - Consistency level

Order of server launch

Method of server launch

Start the system

Confirm that the system is running

Get your S3 API Key from the LeoFS manager console

Using LeoFS

Wrap up

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Getting Started »

Quick Start: All in one in a local node

Purpose

This section is a step by step guide to setting up LeoFS for the first time. By following this tutorial you can easily build a stand-alone LeoFS system.

Note

In this section, LeoFS Storage, LeoFS Gateway and LeoFS Manager are all installed on a single system, with no clustering.

Install

Download a LeoFS package from LeoFS website [https://leo-project.net/leofs/download.html]

Install LeoFS on CentOS-6.x

$ wget https://leo-project.net/leofs/packages/rpm/x86_64/leofs-{VERSION}.x86_64.rpm
$ sudo rpm -ivh leofs-{VERSION}.x86_64.rpm
$ ls -l /usr/local/leofs/
total 4
drwxr-xr-x 6 root   root   4096 Jun 20 15:37 {VERSION}
$ chown -R {USER}:{GROUP} /usr/local/leofs/{VERSION}

Install LeoFS on Ubuntu Server 12.04 LTS or Higher

$ wget https://leo-project.net/leofs/packages/ubuntu/x86_64/leofs_{VERSION}_amd64.deb
$ sudo dpkg -i leofs_{VERSION}_amd64.deb
$ ls -l /usr/local/leofs/
total 4
drwxr-xr-x 6 root   root   4096 Jun 20 15:37 {VERSION}
$ chown -R {USER}:{GROUP} /usr/local/leofs/{VERSION}

Configuration

Modify “/etc/hosts”

Add a domain for the LeoFS bucket in /etc/hosts

Bucket names must follow these rules

$ sudo vi /etc/hosts

## Replace {BUCKET_NAME} with the name of the bucket ##
127.0.0.1 localhost {BUCKET_NAME}.localhost

Launch LeoFS’ managers and storage

By default there is only one replica, you may want to configure the system.

Start master-manager, slave-manager

Start a storage node

$ cd /usr/local/leofs/{VERSION}
$ leo_manager_0/bin/leo_manager start
$ leo_manager_1/bin/leo_manager start
$ leo_storage/bin/leo_storage start

Start the system

Use the command start in the LeoFS manager console

$ leofs-adm start

Start a LeoFS Gateway node

$ leo_gateway/bin/leo_gateway start

Confirm that the system is running

Use the command status in the LeoFS manager console

$ leofs-adm status
[System config]
                System version : 1.0.0
                    Cluster Id : leofs_1
                         DC Id : dc_1
                Total replicas : 1
           # of successes of R : 1
           # of successes of W : 1
           # of successes of D : 1
 # of DC-awareness replicas    : 0
                     ring size : 2^128
             Current ring hash : 8cd79c31
                Prev ring hash : 8cd79c31
[Multi DC replication settings]
         max # of joinable DCs : 2
            # of replicas a DC : 1

[Node(s) state]
-------+--------------------------+--------------+----------------+----------------+----------------------------
 type  |           node           |    state     |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
  S    | storage_0@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:20 +0900
  G    | gateway_0@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:21 +0900

Get your S3 API Key from the LeoFS manager console

Use the command create-user in the LeoFS manager console

It takes the user name as its only argument

$ leofs-adm create-user <your_name>
access-key-id: 05dcba94333c7590a635
secret-access-key: c776574f3661579ceb91aa8788dfcac733b21b3a

Clients

You can now connect to LeoFS using any S3 client, including:

Ruby - ‘aws-sdk’

CUI - ‘s3cmd’

GUI - ‘Dragon Disk’

Using LeoFS

Use the command add-bucket in the LeoFS manager console

It takes the bucket name and access-key-id got in the previous section as its arguments

$ leofs-adm add-bucket <bucket> <access-key-id>
ok

Insert some data into LeoFS by using any S3 client as mentioned above

You can now get the data stored in LeoFS

$ curl https://localhost:8080/your_bucket_name/path/to/file
> {CONTENTS}

Note

From version 0.16.0, you need to set ACL settings of your bucket to public-read by using the command update-acl if you want to get the data stored in LeoFS via web browser.

Wrap up

You now know how to setup a stand-alone LeoFS system. Make sure to have a look at Quick Start -2 Cluster to learn how to setup a LeoFS cluster.

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Getting Started »

Quick Start: Building a cluster

Purpose

This tutorial teaches you how to easily build a LeoFS cluster. All steps will not be explained in detail, it is assumed you already know how to setup a stand-alone LeoFS system. This guide exists to help you get a cluster up and running quickly. We recommend that you read the LeoFS Installation, Configuration and Administration Guide to learn how to administer your LeoFS cluster. We hope that by reading this tutorial you will be able to get a cluster started as quickly as possible.

Case example

Manager

IP: 10.0.1.101, 10.0.1.102

Name: manager_0@10.0.1.101, manager_1@10.0.1.102

Gateway

IP: 10.0.1.103

Name: gateway_0@10.0.1.103

Storage

IP: 10.0.1.104 .. 10.0.1.106

Name: storage_0@10.0.1.104 .. storage_2@10.0.1.106

Install Erlang and LeoFS on each server

Install LeoFS

Configuration - Edit “leo_*.conf” on each server

File path: “$LEOFS_ROOT/package/leo_*/etc/leo_*.conf”

Precondition

nodename must be unique for each node in the LeoFS cluster

Edit Manager-master’s leo_manager.conf

## Name of the Node
nodename = manager_0@10.0.1.101
... omitted below

Edit Manager-slave’s leo_manager.conf

## Name of the Node
nodename = manager_1@10.0.1.102
... omitted below

Edit Gateway’s leo_gateway.conf

## Name of the Node
nodename = gateway_0@10.0.1.103
... omitted below

Edit Storage’s leo_storage.conf

## Name of the Node
nodename = storage_0@10.0.1.104
... omitted below

Configuration - Consistency level

Reference: The consistency level

Edit Manager’s leo_manager.conf

You only need to modify Manager-master for the consistency level.

“$LEOFS_ROOT/package/leo_manager_0/etc/leo_manager.conf”

## --------------------------------------------------------------------
## MANAGER - Consistency Level
##     * Only set its configurations to **Manager-master**
##     * See: https://leo-project.net/leofs/docs/configuration/configuration_1.html
## --------------------------------------------------------------------
## A number of replicas
consistency.num_of_replicas = 2

## A number of replicas needed for a successful WRITE operation
consistency.write = 1

## A number of replicas needed for a successful READ operation
consistency.read = 1

## A number of replicas needed for a successful DELETE operation
consistency.delete = 1

Order of server launch

Manager-master

Manager-slave

Storage nodes

Gateway(s)

Method of server launch

Shell script: “$LEOFS_ROOT/package/leo_*/bin/leo_*”

Launch Manager-master

$ $LEOFS_ROOT/package/leo_manager_0/bin/leo_manager start

Launch Manager-slave

$ $LEOFS_ROOT/package/leo_manager_1/bin/leo_manager start

Launch each Storage nodes

$ $LEOFS_ROOT/package/leo_storage/bin/leo_storage start

Launch each Gateway nodes

$ $LEOFS_ROOT/package/leo_gateway/bin/leo_gateway start

Start the system

Use the command start in the LeoFS manager console

$ leofs-adm start

Confirm that the system is running

Use the command status in the LeoFS manager console

$ leofs-adm status
[System config]
                System version : 1.0.0
                    Cluster Id : leofs_1
                         DC Id : dc_1
                Total replicas : 2
           # of successes of R : 1
           # of successes of W : 1
           # of successes of D : 1
 # of DC-awareness replicas    : 0
                     ring size : 2^128
             Current ring hash : 8cd79c31
                Prev ring hash : 8cd79c31
[Multi DC replication settings]
         max # of joinable DCs : 2
            # of replicas a DC : 1

[Node(s) state]
-------+--------------------------+--------------+----------------+----------------+----------------------------
 type  |           node           |    state     |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
  S    | storage_0@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:20 +0900
  S    | storage_1@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:20 +0900
  S    | storage_2@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:20 +0900
  G    | gateway_0@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:21 +0900

Get your S3 API Key from the LeoFS manager console

Use the command create-user in the LeoFS manager console

It takes the user name as its only argument

$ leofs-adm create-user <your-name>
access-key-id: 05dcba94333c7590a635
secret-access-key: c776574f3661579ceb91aa8788dfcac733b21b3a

Using LeoFS

Use the command add-bucket in the LeoFS manager console

It takes the bucket name and access-key-id got in the previous section as its arguments

$ leofs-adm add-bucket <bucket> <access-key-id>
ok

Insert some data into LeoFS by using any S3 client as mentioned above

You can now get the data stored in LeoFS

$ curl https://localhost:8080/your_bucket_name/path/to/file
> {CONTENTS}

Note

From version 0.16.0, you need to set ACL settings of your bucket to public-read by using the command update-acl if you want to get the data stored in LeoFS via web browser.

Wrap up

You now have a working LeoFS cluster. Make sure to have a look at LeoFS Installation, LeoFS Configuration and Administration Guide to learn more about setting up and managing your LeoFS cluster.

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Installation

Install Erlang

Prepare

Install required libraries using yum (CentOS 6.5/7)

Install required libraries using apt-get (Ubuntu Server 14.04 LTS or Higher)

Install “libatomic_ops” for Erlang 17.5 (both CentOS and Ubuntu)

Download “Erlang 17.5”

Build Erlang on CentOS 6.5/7

Build Erlang on Ubuntu/Debian

Confirm

XFS-related

Install required libraries for XFS with yum (CentOS 6.5/7)

Create an XFS Partition (Volume)

Start fdisk

Create partition

Confirm

Reboot

Format the partition

Modify the “/etc/fstab” file

Mount the partition

Confirm

Install LeoFS

File structure

Before running make

After running make

Building

Log Dir and Working Dir

Firewall Rules

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Installation »

Install Erlang

LeoFS development currently targets Debian 6, Ubuntu-Server 14.04 LTS or Higher and CentOS 6.5/7, but should work on most Linux platforms with the following software installed:

Erlang 17.5 [https://www.erlang.org/download_release/28]

Note

We recommend this installation method. Please follow the relevant instructions for your environment.

Prepare

Install required libraries using yum (CentOS 6.5/7)

$ sudo yum install gcc gcc-c++ glibc-devel make ncurses-devel openssl-devel autoconf \
                   libuuid-devel cmake check check-devel

Install required libraries using apt-get (Ubuntu Server 14.04 LTS or Higher)

$ sudo apt-get install build-essential libtool libncurses5-dev libssl-dev cmake check

Install “libatomic_ops” for Erlang 17.5 (both CentOS and Ubuntu)

$ wget https://www.ivmaisoft.com/_bin/atomic_ops/libatomic_ops-7.4.2.tar.gz
$ tar xzvf libatomic_ops-7.4.2.tar.gz
$ cd libatomic_ops-7.4.2
$ ./configure --prefix=/usr/local
$ make
$ sudo make install

Download “Erlang 17.5”

$ cd $WORK_DIR
$ wget https://www.erlang.org/download/otp_src_17.5.tar.gz

Build Erlang on CentOS 6.5/7

$ tar xzf otp_src_17.5.tar.gz
$ cd otp_src_17.5
$ CFLAGS="-DOPENSSL_NO_EC=1" \
  ./configure --prefix=/usr/local/erlang/17.5 \
              --enable-smp-support \
              --enable-m64-build \
              --enable-halfword-emulator \
              --enable-kernel-poll \
              --without-javac \
              --disable-native-libs \
              --disable-hipe \
              --disable-sctp \
              --enable-threads \
              --with-libatomic_ops=/usr/local
$ make
$ sudo make install

Build Erlang on Ubuntu/Debian

$ tar xzf otp_src_17.5.tar.gz
$ cd otp_src_17.5
$ ./configure --prefix=/usr/local/erlang/17.5 \
              --enable-smp-support \
              --enable-m64-build \
              --enable-halfword-emulator \
              --enable-kernel-poll \
              --without-javac \
              --disable-native-libs \
              --disable-hipe \
              --disable-sctp \
              --enable-threads \
              --with-libatomic_ops=/usr/local
$ make
$ sudo make install

Confirm

$ erl
Erlang/OTP 17 [erts-6.4] [source] [64-bit] [smp:8:8] [async-threads:10] [kernel-poll:false]

Eshell V6.4  (abort with ^G)
1>

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Installation »

XFS-related

Note

We highly recommend using an XFS partition, as it is the file system that shows the better results with LeoFS. This section describes the installation instructions related to XFS. If you are deploying LeoFS on a DEV environment, you do NOT need to perform this operation.

Install required libraries for XFS with yum (CentOS 6.5/7)

$ sudo yum --enablerepo=centosplus install kmod-xfs xfsprogs xfsprogs-devel

Create an XFS Partition (Volume)

Only the servers running LeoFS storage nodes will benefit from using XFS (unix local file system). XFS provides particularly efficient I/O for large files. LeoFS-Storage is implemented on top of files stored in a single file system created on top of the a few TB volume.

Start fdisk

$ sudo fdisk /dev/sda

The number of cylinders for this disk is set to 8908.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 73.2 GB, 73272393728 bytes
255 heads, 63 sectors/track, 8908 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        1951    15671376   83  Linux
/dev/sda2            1952        2472     4184932+  82  Linux swap / Solaris

Create partition

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (2473-8908, default 2473):[Enter]
Using default value 2473
Last cylinder or +size or +sizeM or +sizeK (2473-8908, default 8908):[Enter]
Using default value 8908

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or  resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

Confirm

$ sudo fdisk /dev/sda

The number of cylinders for this disk is set to 8908.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sda: 73.2 GB, 73272393728 bytes
255 heads, 63 sectors/track, 8908 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        1951    15671376   83  Linux
/dev/sda2            1952        2472     4184932+  82  Linux swap / Solaris
/dev/sda3            2473        8908    51697170   83  Linux

Reboot

$ sudo reboot

Format the partition

Reference(EN) [https://www.ibm.com/developerworks/linux/library/l-fs10/index.html]

Reference(JP) [https://www.ibm.com/developerworks/jp/linux/library/l-fs10/index.html]

$ sudo mkfs.xfs -d agcount=4 -l size=32m {TARGET_PARTITION}

Modify the “/etc/fstab” file

$ sudo vi /etc/fstab
/dev/sda3   /mnt/xfs   xfs   noatime,nodiratime,osyncisdsync 0 0

Mount the partition

$ sudo mkdir /mnt/xfs
$ sudo mount -a

Confirm

$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             15180256   2153492  12243196  15% /
tmpfs                  2025732         0   2025732   0% /dev/shm
/dev/sda3             51664400      4

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Installation »

Install LeoFS

This installation method is based on a source build, so if you do not have Erlang already installed, you need to first install Erlang. Also, building LeoFS from source requires Erlang R16B03-1 or 17.5.

File structure

Before running make

$ git clone https://github.com/leo-project/leofs.git

{LEOFS_SRC_DIR}
  |
  |--- LICENSE
  |--- Makefile
  |--- apps/
  |--- deps/
  |--- doc/
  |--- rebar
  |--- rebar.config
  `--- rel/
         |--- leo_gateway/
         |--- leo_manager/
         `--- leo_storage/

After running make

$ cd {LEOFS_SRC}/
$ make
$ make release

{LEOFS_SRC_DIR}
  |
  |--- LICENSE
  |--- Makefile
  |---- deps/
  |      |--- bear/
  |      |--- bitcask/
  |      |--- cowboy/
  |      |--- eleveldb/
  |      |--- folsom/
  |      |--- jiffy/
  |      |--- leo_backend_db/
  |      |--- leo_cache/
  |      |--- leo_commons/
  |      |--- leo_dcerl
  |      |--- leo_gateway/
  |      |--- leo_logger/
  |      |--- leo_manager/
  |      |--- leo_mcerl/
  |      |--- leo_mq/
  |      |--- leo_object_storage/
  |      |--- leo_ordning_reda/
  |      |--- leo_pod/
  |      |--- leo_redundant_manager/
  |      |--- leo_rpc/
  |      |--- leo_s3_libs/
  |      |--- leo_statistics/
  |      |--- leo_storage/
  |      |--- lz4/
  |      |--- meck/
  |      |--- proper/
  |      |--- ranch/
  |      |--- savanna_agent/
  |      `--- savanna_commons/
  |---- rebar
  |---- rebar.config
  `---- rel/
         |--- leo_gateway/
         |--- leo_manager/
         `--- leo_storage/

Building

$ cd leofs/
$ make
$ make release
$ cp -r package {LEOFS_DEPLOYED_DIR}
$ cd {LEOFS_DEPLOYED_DIR}/

[LeoFS deployed files layout]
{LEOFS_DEPLOYED_DIR}
        |--- leo_gateway/
        |        |--- bin/
        |        |--- erts-{VERSION}/
        |        |--- etc/
        |        |--- lib/
        |        |--- log/
        |        |--- releases/
        |        |--- snmp/
        |        `--- work/
        |--- leo_manager_0/
        |        |--- bin/
        |        |--- erts-{VERSION}/
        |        |--- etc/
        |        |--- lib/
        |        |--- log/
        |        |--- releases/
        |        |--- snmp/
        |        `--- work/
        |--- leo_manager_1/
        |        |--- bin/
        |        |--- erts-{VERSION}/
        |        |--- etc/
        |        |--- lib/
        |        |--- log/
        |        |--- releases/
        |        |--- snmp/
        |        `--- work/
        `--- leo_storage/
                 |--- bin/
                 |--- erts-{VERSION}/
                 |--- etc/
                 |--- lib/
                 |--- log/
                 |--- releases/
                 |--- snmp/
                 `--- work/

Log Dir and Working Dir

Directory | Description
log/
log/app/ | Application logs
log/ring/ | RING (routing-table for replication) dump files
log/sasl/ | SASL (Erlang system) Logs
work/
work/mnesia/ | System internal data stored into ‘Mnesia’
work/queue/ | Message queue data stored into ‘bitcask’

ref: Basho bitcask [https://github.com/basho/bitcask]

{LEOFS_DEPLOYED_DIR}
  |      `--- leo_storage/
  |               |--- bin/
  |               |--- erts-{VERSION}/
  |               |--- etc/
  |               |--- lib/
  |               |--- log/
  |               |     |--- app/
  |               |     |--- ring/
  |               |     `--- sasl/
  |               |--- releases/
  |               |--- snmp/
  |               `--- work/
  .                     |--- mnesia
  .                     `--- queue

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Installation »

Firewall Rules

In order for LeoFS to work correctly, it is necessary to set and check the firewall rules in your environment as follows:

Subsystem | Direction | Ports | Notes
LeoFS Manager-Master | Incoming | 10010/* | Manager console
LeoFS Manager-Master | Incoming | 4369/* | Erlang Port Mapper
LeoFS Manager-Master | Incoming | 4020/* | SNMP Listen Port
LeoFS Manager-Master | Outgoing | */4369 | Erlang Port Mapper
LeoFS Manager-Slave | Incoming | 10011/* | Manager console
LeoFS Manager-Slave | Incoming | 4369/* | Erlang Port Mapper
LeoFS Manager-Slave | Incoming | 4021/* | SNMP Listen Port
LeoFS Manager-Slave | Outgoing | */4369 | Erlang Port Mapper
LeoFS Storage | Incoming | 4369/* | Erlang Port Mapper
LeoFS Storage | Incoming | 4010/* | SNMP Listen Port
LeoFS Storage | Outgoing | */4369 | Erlang Port Mapper
LeoFS Gateway | Incoming | 8080/* | HTTP listen port
LeoFS Gateway | Incoming | 8443/* | HTTPS listen port
LeoFS Gateway | Incoming | 4369/* | Erlang Port Mapper
LeoFS Gateway | Incoming | 4000/* | SNMP Listen Port
LeoFS Gateway | Outgoing | */4369 | Erlang Port Mapper
ALL | Both | [1] | Erlang RPC to others

[1]
Port range can be specified by setting the kernel variables ‘inet_dist_listen_min’ AND ‘inet_dist_listen_max’

Example

%%% This forces Erlang to use only ports 9100--9105 for distributed Erlang traffic.
application:set_env(kernel, inet_dist_listen_min, 9100).
application:set_env(kernel, inet_dist_listen_max, 9105).

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Configuration

Configuration Fundamentals and LeoFS Manager

LeoFS Configuration

LeoFS Manager-Master

The Consistency Level

Configuration of the Manager-Master node

LeoFS Manager-Slave

LeoFS Storage

LeoFS Storage configuration

See Also

Erlang VM configuration

See Also

Configuration related to MQ

See Also

Configuration related to the auto-compaction

See Also

Configuration of eager check I/O scheduling for Erlang’s VM

See Also

LeoFS Gateway

LeoFS Gateway configuration

See Also

Erlang VM configuration

See Also

Configuration related to Disk Cache

Configuration of eager check I/O scheduling for Erlang’s VM

See Also

SNMPA Setup

LeoFS Manager

SNMPA Properties

SNMPA Items

Check the configuration

LeoFS Storage

SNMPA Properties

SNMPA Items

Check the configuration

LeoFS Gateway

SNMPA Properties

SNMPA Items

Check the configuration

Multi data center replication

Since

Purpose

Getting Started

Pre-requirement

LeoFS Manager Configuration

LeoFS Storage Configuration

LeoFS with NFS

Since

Purpose

Getting Started

Pre-requirement

Configuration

Start LeoFS as NFS Server with other dependent programs

Confirm that NFS works

Configuration

Limits

Watchdog

Since

Purpose

Getting Started

Pre-requirement

Configuration of LeoFS Gateway

Configuration of LeoFS Storage

Auto-compaction

Since

Purpose

Getting Started

Pre-requirement

Configuration of LeoFS Storage

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Configuration »

Configuration Fundamentals and LeoFS Manager

LeoFS Configuration

Each LeoFS node has one configuration files, leo_manager.conf, leo_storage.conf and leo_gateway.conf, which are located in the following directories:

Application | Location
Manager-Master | {LEOFS_HOME}/package/leo_manager_0/etc/leo_manager.conf
Manager-Slave | {LEOFS_HOME}/package/leo_manager_1/etc/leo_manager.conf
Storage | {LEOFS_HOME}/package/leo_storage/etc/leo_storage.conf
Gateway | {LEOFS_HOME}/package/leo_gateway/etc/leo_gateway.conf

The *.config file is used to set various attributes of the application. Also, it used to pass parameters to the Erlang node such as the name and cookie of the node.

LeoFS Manager-Master

The Consistency Level

Note

The consistency level is configured in this file. It should not be modified while the system is running.

Property | Description
n - consistency.num_of_replicas | # of replicas
r - consistency.read | # of replicas needed for a successful READ operation
w - consistency.write | # of replicas needed for a successful WRITE operation
d - consistency.delete | # of replicas needed for a successful DELETE operation
consistency.rack_aware_replicas | # of rack-aware replicas

A reference consistency level

Level | Configuration
Low | n = 3, r = 1, w = 1, d = 1
Middle | n = 3, [r = 1 | r = 2], w = 2, d = 2
High | n = 3, [r = 2 | r = 3], w = 3, d = 3

Example - File: {LEOFS_SRC}/package/manager_0/etc/leo_manager.conf:

## --------------------------------------------------------------------
## MANAGER - Consistency Level
##     * Only set its configurations to **Manager-master**
##     * See: https://leo-project.net/leofs/docs/configuration/configuration_1.html
## --------------------------------------------------------------------
## A number of replicas
consistency.num_of_replicas = 3

## A number of replicas needed for a successful WRITE operation
consistency.write = 2

## A number of replicas needed for a successful READ operation
consistency.read = 1

## A number of replicas needed for a successful DELETE operation
consistency.delete = 2

## A number of rack-aware replicas
consistency.rack_aware_replicas = 0

Configuration of the Manager-Master node

[leo_manager.conf]

The leo_manager.conf location: {LEOFS_HOME}/package/manager_0/etc/leo_manager.conf

Modification of the required items:

Property
Description

{SLAVE-IP}
Manager-Slave node’s IP-address

{SNMPA-DIR}

SNMPA configuration files directory

ref:{LEOFS_SRC}/apps/leo_manager/snmp/

[snmpa_manager_0|snmpa_manager_1|snmpa_manager_0]

## --------------------------------------------------------------------
## MANAGER
## --------------------------------------------------------------------
## Mode of Manager: [master, slave]
manager.mode = master

## Partner of manager's alias
manager.partner = manager_1@{SLAVE-IP}

## Manager-console accepatable port number
console.port.cui  = 10010
console.port.json = 10020

## Manager-console's number of acceptors
console.acceptors.cui = 3
console.acceptors.json = 16

## --------------------------------------------------------------------
## MANAGER - Consistency Level
##     * Only set its configurations to **Manager-master**
##     * See: https://leo-project.net/leofs/docs/configuration/configuration_1.html
## --------------------------------------------------------------------
## A number of replicas
consistency.num_of_replicas = 3

## A number of replicas needed for a successful WRITE operation
consistency.write = 2

## A number of replicas needed for a successful READ operation
consistency.read = 1

## A number of replicas needed for a successful DELETE operation
consistency.delete = 2

## A number of rack-aware replicas
consistency.rack_aware_replicas = 0

## --------------------------------------------------------------------
## MANAGER - Multi DataCenter Settings
## --------------------------------------------------------------------
## A number of replication targets
mdc_replication.max_targets = 2

## A number of replicas a DC
mdc_replication.num_of_replicas_a_dc = 1

## --------------------------------------------------------------------
## MANAGER - Mnesia
##     * Store the info storage-cluster and the info of gateway(s)
##     * Store the RING and the command histories
## --------------------------------------------------------------------
## Mnesia dir
mnesia.dir = ./work/mnesia/127.0.0.1

## The write threshold for transaction log dumps
## as the number of writes to the transaction log
mnesia.dump_log_write_threshold = 50000

## Controls how often disc_copies tables are dumped from memory
mnesia.dc_dump_limit = 40

## --------------------------------------------------------------------
## MANAGER - Log
## --------------------------------------------------------------------
## Log level: [0:debug, 1:info, 2:warn, 3:error]
log.log_level = 1

## Output log file(s) - Erlang's log
log.erlang = ./log/erlang

## Output log file(s) - app
log.app = ./log/app

## Output log file(s) - members of storage-cluster
log.member_dir = ./log/ring

## Output log file(s) - ring
log.ring_dir = ./log/ring

## --------------------------------------------------------------------
## MANAGER - Other Directories
## --------------------------------------------------------------------
## Directory of queue for monitoring "RING"
queue_dir = ./work/queue

## Directory of SNMP agent configuration
snmp_agent = {SNMPA-DIR}/snmp/snmpa_manager_0/LEO-MANAGER

[Erlang VM related properties]

Property | Description
{MASTER-IP} | Manager-Master IP
{SNMPA-DIR} | SNMPA configuration files directory

## Name of the leofs-gateway node
nodename = manager_0@{MASTER-IP}

## Cookie for distributed node communication.  All nodes in the same cluster
## should use the same cookie or they will not be able to communicate.
distributed_cookie = 401321b4

## Enable kernel poll
erlang.kernel_poll = true

## Number of async threads
erlang.asyc_threads = 32

## Increase number of concurrent ports/sockets
erlang.max_ports = 64000

## Set the location of crash dumps
erlang.crash_dump = ./log/erl_crash.dump

## Raise the ETS table limit
erlang.max_ets_tables = 256000

## Raise the default erlang process limit
process_limit = 1048576

## Path of SNMP-agent configuration
snmp_conf = {SNMPA-DIR}/snmp/snmpa_manager_0/leo_maanager_snmp

LeoFS Manager-Slave

Configuration of the Manager-Slave node

[leo_manager.conf]

The leo_manager.conf location: {LEOFS_HOME}/package/manager_1/etc/leo_manager.conf

Modification of the required items:

Property | Description
{MASTER-IP} | Manager-Master node’s IP-address
{SNMPA-DIR} | SNMPA configuration files directory

## --------------------------------------------------------------------
## MANAGER
## --------------------------------------------------------------------
## Mode of Manager: [master, slave]
manager.mode = slave

## Partner of manager's alias
manager.partner = manager_0@{MASTER-IP}

## Manager-console accepatable port number
console.port.cui  = 10011
console.port.json = 10021

## Manager-console's number of acceptors
console.acceptors.cui = 3
console.acceptors.json = 16

## --------------------------------------------------------------------
## MANAGER - Mnesia
##     * Store the info storage-cluster and the info of gateway(s)
##     * Store the RING and the command histories
## --------------------------------------------------------------------
## Mnesia dir
mnesia.dir = ./work/mnesia/127.0.0.1

## The write threshold for transaction log dumps
## as the number of writes to the transaction log
mnesia.dump_log_write_threshold = 50000

## Controls how often disc_copies tables are dumped from memory
mnesia.dc_dump_limit = 40

## --------------------------------------------------------------------
## MANAGER - Log
## --------------------------------------------------------------------
## Log level: [0:debug, 1:info, 2:warn, 3:error]
log.log_level = 1

## Output log file(s) - Erlang's log
log.erlang = ./log/erlang

## Output log file(s) - app
log.app = ./log/app

## Output log file(s) - members of storage-cluster
log.member_dir = ./log/ring

## Output log file(s) - ring
log.ring_dir = ./log/ring

## --------------------------------------------------------------------
## MANAGER - Other Directories
## --------------------------------------------------------------------
## Directory of queue for monitoring "RING"
queue_dir = ./work/queue

## Directory of SNMP agent configuration
snmp_agent = {SNMPA-DIR}/snmp/snmpa_manager_1/LEO-MANAGER

[Erlang VM related properties]

Property | Description
{SLAVE-IP} | Manager-Slave IP
{SNMPA-DIR} | SNMPA configuration files directory

## Name of the leofs-gateway node
nodename = manager_1@{SLAVE-IP}

## Cookie for distributed node communication.  All nodes in the same cluster
## should use the same cookie or they will not be able to communicate.
distributed_cookie = 401321b4

## Enable kernel poll
erlang.kernel_poll = true

## Number of async threads
erlang.asyc_threads = 32

## Increase number of concurrent ports/sockets
erlang.max_ports = 64000

## Set the location of crash dumps
erlang.crash_dump = ./log/erl_crash.dump

## Raise the ETS table limit
erlang.max_ets_tables = 256000

## Raise the default erlang process limit
process_limit = 1048576

## Path of SNMP-agent configuration
snmp_conf = {SNMPA-DIR}/snmp/snmpa_manager_1/leo_maanager_snmp

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Configuration »

LeoFS Storage

LeoFS Storage configuration

Location of configuration file:

<leofs-home-dir>/package/storage/etc/leo_storage.conf OR /usr/local/leofs/<version>/leo_storage/etc/leo_storage.conf

Note

You cannot modify the following Leo Storage configurations after launched a system(LeoFS) because the algorithm of retrieving object depends on their configurations, that means Leo Storage cannot handle objects.

NOT able to change items:

obj_containers.path

obj_containers.num_of_containers

obj_containers.metadata_storage

num_of_vnodes

Property
Default Value
Description

LeoFS Manager

managers
[manager_0@127.0.0.1, manager_1@127.0.0.1]
Node name of LeoFS Manager master and slave nodes

LeoFS Storage Basic Configuration

obj_containers.path
[./avs]
Not able to change items Directories of each object-containers

obj_containers.num_of_containers
[8]
Not able to change items A number of object-containers of each directory

obj_containers.metadata_storage
leveldb
Not able to change items Metadata Storage - [leveldb, bitcask], We recommend leveldb

num_of_vnodes
168
Not able to change items A number of virtual-nodes, which depends on the redundant-manager

object_storage.is_strict_check
false
Able to strictly check between checksum of a metadata and checksum of the object

Watchdog

rex(rpc)

watchdog.rex.interval
10
Watch interval for the rex*(rpc)* process in order to execute force GC when reaching the threshold value

watchdog.rex.threshold_mem_capacity
33554432
Threshold memory capacity of binary for rex*(rpc)*

CPU

watchdog.cpu.is_enabled
false
Enabled or disabled CPU-watchdog [true|false]

watchdog.cpu.interval
10
Watch interval(sec) for CPU-watchdog

watchdog.cpu.raised_error_times
5
An error is raised to subscribers when a number of errors reached this configuration.

watchdog.cpu.threshold_cpu_load_avg
5.0
Threshold CPU load avg for 1min/5min

watchdog.cpu.threshold_cpu_util
100
Threshold CPU load util (%)

DISK

watchdog.disk.is_enabled
false
Enabled or disabled disk-watchdog [true|false]

watchdog.disk.interval
10
Watch interval (sec) for Disk-watchdog

watchdog.disk.raised_error_times
5
An error is subscribers to clients when a number of errors reached this configuration.

watchdog.disk.threshold_disk_use
85
Threshold disk usage(capacity) (%) - leo_watchdog is using df command

watchdog.disk.threshold_disk_util
100
Threshold disk util (%) - leo_watchdog is using iostat command

watchdog.disk.threshold_disk_rkb
98304
Threshold disk read KB/sec

watchdog.disk.threshold_disk_wkb
98304
Threshold disk write KB/sec

watchdog.disk.target_devices
[]
Target devices for checking disk utilization

Autonomic Operation

autonomic_op.compaction.is_enabled
false

Is enabled auto-compaction? [true|false]

autonomic_op.compaction.parallel_procs
1

Number of parallel procs of data-compaction

autonomic_op.compaction.threshold_active_size_ratio
70

Warning ratio of active size (%)

NOT affects the auto-compaction by this configuration yet. We plan to support this configuration with v1.4.

autonomic_op.compaction.threshold_active_size_ratio
60

Threshold ratio of active size (%)

(Size of active objects / Size of total objects) * 100 (%)

When it was less than the ratio, LeoFS Storage executes data-compaction automatically

Data Compaction

compaction.limit_num_of_compaction_procs
4
Limit of a number of procs to execute data-compaction in parallel

Data Compaction - Interval between batch processings

compaction.waiting_time_regular
500
Regular value of compaction-proc waiting time/batch-proc(msec)

compaction.waiting_time_max
3000
Maximum value of compaction-proc waiting time/batch-proc(msec)

Data Compaction - Number of objects a batch processing

compaction.batch_procs_regular
1000
Regular number of objects a batch processing

compaction.batch_procs_max
5000
Maximum number of objects a batch processing

MQ

mq.backend_db = bitcask
bitcask
MQ backend storage: [bitcask]

mq.num_of_mq_procs
8
A number of mq-server’s processes

MQ - Number of messages a batch processing

mq.num_of_batch_process_max
3000
Maximum number of messages a bach processing

mq.num_of_batch_process_regular
1600
Regular number of messages a bach processing

MQ - Interval between batch processings

mq.interval_between_batch_procs_max
3000
Maximum value of interval beween batch-procs(msec)

mq.interval_between_batch_procs_regular
500
Regular value of interval between batch-procs(msec)

STORAGE - Replication/Recovery object(s)

replication.rack_awareness.rack_id
[]
Rack-id for the rack-awareness replica placement

replication.recovery.size_of_stacked_objs
33554432
Size of stacked objects (bytes)

replication.recovery.stacking_timeout
5
Stacking timeout (sec)

STORAGE - MDC Replication

mdc_replication.size_of_stacked_objs
33554432
Size of stacked objects (bytes)

mdc_replication.stacking_timeout
30
Stacking timeout (sec)

mdc_replication.req_timeout
30000
Request timeout (msec)

mdc_replication.stacking_procs
1
Number of stacking procecces

RPC for MDC-replication

rpc.server.acceptors
128
Number of RPC-server’s acceptors

rpc.server.listen_port
13077
RPC-Server’s listening port number

rpc.server.listen_timeout
30000
RPC-Server’s listening timeout (msec)

rpc.client.connection_pool_size
8
RPC-Client’s size of connection pool

rpc.client.connection_buffer_size
8
RPC-Client’s size of connection buffer

Log

log.log_level
1
Log level: [0:debug, 1:info, 2:warn, 3:error]

log.erlang
./log/erlang
Locatio of Erlang log-files

log.app
./log/app
Locatio of Gateway’s log-files

log.member_dir
./log/ring
Location of dump files of members of a storage cluster

log.ring_dir
./log/ring
Location of RING’s dump files

Others*

queue_dir
./work/queue
Directory of queue for monitoring “RING” and members of a storage cluster

snmp_agent
./snmp/snmpa_gateway_0/LEO-GATEWAY
Directory of SNMP agent configuration

See Also

LeoFS watchdog configuration

LeoFS auto-compaction configuration

Erlang VM configuration

Property | Default Value | Description
nodename | storage_0@127.0.0.1 | Node name of LeoFS Storage
distributed_cookie | 401321b4 | Cookie for distributed node communication
erlang.kernel_poll | true | Enable or disable kernel poll
erlang.async_threads | 32 | Number of async threads
erlang.max_ports | 64000 | Number of concurrent ports/sockets
erlang.crash_dump | ./log/erl_crash.dump | Location of crash dumps
erlang.max_ets_tables | 256000 | ETS table limit
erlang.smp | enable | Enable or disable SMP
erlang.schedulers.compaction_of_load | true | Erlang scheduler’s compaction of load
erlang.schedulers.utilization_balancing | false | Erlang scheduler’s balancing of load
erlang.distribution_buffer_size | 32768 | Sender-side network distribution buffer size (KB)
erlang.fullsweep_after | 0 | A non-negative integer which indicates how many times generational garbage collections can be done without forcing a fullsweep collection
erlang.secio | false | Enable or disable eager check I/O (Erlang 17.4/erts-6.3-, ref:OTP-12117)
process_limit | 1048576 | Default erlang process limit
snmp_conf | ./snmp/snmpa_storage_0/leo_storage_snmp | SNMPA configuration files directory

See Also

Eralng - erl

Configuration related to MQ

LeoFS’s MQ mechanism is affected by the watchdog mechanism in order to reduce costs of a message comsumption. MQ dinamically updates a num of batch processes and an interval (Figure: Number-of-batch-processes and interval).

Figure: Number-of-batch-processes and interval

As of Figure: Relationship of Watchdog and MQ, the watchdog automatically adjusts value of a num of batch processes between mq.num_of_batch_process_min and mq.num_of_batch_process_max, which is increased or decreased with mq.num_of_batch_process_step.

On the other hands value of an interval is adjusted between mq.interval_between_batch_procs_min and mq.interval_between_batch_procs_max, which is increased or decreased with mq.interval_between_batch_procs_step.

When the each value reached the min value, the MQ changes the status to suspending, after that the node’s processing costs is changed to low, the MQ updates the status to running, again.

Figure: Relationship between Watchdog and MQ

See Also

LeoFS Storage Operation

LeoFS Watchdog configuration

Configuration related to the auto-compaction

LeoFS’s auto-compaction mechanism is affected by the watchdog mechanism in order to reduce costs of a the processing. The auto-compaction dinamically updates a num of batch processes and an interval (Figure: Number-of-batch-processes and interval). The basic design of the relationship with the watchdog is similar to the MQ.

Figure: Number-of-batch-processes and interval

As of Figure: Relationship of Watchdog and Auto-compaction, the watchdog automatically adjusts value of a num of batch processes between compaction.batch_procs_min and compaction.batch_procs_max, which is increased or decreased with compaction.batch_procs_step.

On the other hands value of an interval is adjusted between compaction.waiting_time_min and compaction.waiting_time_max, which is increased or decreased with compaction.waiting_time_step.

When the each value reached the min value, the auto-compaction changes the status to suspending, after that the node’s processing costs is changed to low, the auto-compaction updates the status to running, again.

Figure: Relationship between Watchdog and Auto-compaction

See Also

LeoFS Storage Operation

LeoFS Auto-compaction configuration

Configuration of eager check I/O scheduling for Erlang’s VM

If you adopt LeoFS v1.2.7 later w/Erlang 17.5, we recommend you turn on erlang.secio. So you need to modify the configuration files and the schema files as follows.

Turn on erlang.secio at leo_storage.conf

erlang.secio = true

Uncomment every rows of erlang.secio at leo_storage.schema

{mapping,
  "erlang.secio",
  "vm_args.+secio",
  [
    {default, false}
  ]}.

See Also

Eralng - erl

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Configuration »

LeoFS Gateway

LeoFS Gateway configuration

Location of configuration file:

<leofs-home-dir>/package/leo_gateway/etc/leo_gateway.conf OR /usr/local/leofs/<version>/leo_gateway/etc/leo_gateway.conf

Property
Default Value
Description

LeoFS Manager

managers
[manager_0@127.0.0.1, manager_1@127.0.0.1]
Node name of LeoFS Manager master and slave nodes

LeoFS Gateway Basic Configuration

protocol
s3
Gateway Protocol - [s3 | rest | embed | nfs]

HTTP

http.port
8080
Port number the Gateway uses for HTTP connections

http.num_of_acceptors
128
Numbers of processes listening for connections

http.max_keepalive
4096
Maximum number of requests allowed in a single keep-alive session

http.layer_of_dirs
12
Maximum number of virtual directories

http.ssl_port
8443
Port number the Gateway uses for HTTPS connections

http.ssl_certfile
./etc/server_cert.pem
SSL Certificate file

http.ssl_keyfile
./etc/server_key.pem
SSL key file

http.headers_config_file
./etc/http_custom_header.conf
HTTP custom header configuration file path

http.timeout_for_header
5000
HTTP timeout for reading header (msec)

http.timeout_for_body
15000
HTTP timeout for reading body (msec)

Other

bucket_prop_sync_interval
300
Synchronized time of a bucket property (sec)

NFS-related configurations

nfs.mountd.port
22050
Mountd’s port number

nfs.mountd.acceptors
128
Mountd’s the number of acceptors

nfs.nfsd.port
2049
NFSd’s port number

nfs.nfsd.acceptors
128
NFSd’s the number of acceptors

Large-object processing configuration

large_object.max_chunked_objs
1000
Total number of chunked objects

large_object.chunked_obj_len
5242880
Length of a chunked object (Byte), This value must be >= large_object.reading_chunked_obj_len otherwise you may hit issue#531 (The last part of a large object can be broken with reading_chunked_obj_len > chunked_obj_len in leo_gateway.conf), With version >= 1.3.1, LeoFS check whether this constraint is violated or not when starting leo_gateway

large_object.threshold_of_chunk_len
5767168
Threshold of length of a chunked object (Byte)

large_object.reading_chunked_obj_len
5242880
Reading length of a chuncked object, This value must be <= large_object.chunked_obj_len otherwise you may hit issue#531 (The last part of a large object can be broken with reading_chunked_obj_len > chunked_obj_len in leo_gateway.conf), With version >= 1.3.1, LeoFS check whether this constraint is violated or not when starting leo_gateway

Cache configuration

cache.cache_workers
16
A number of cache workers at the cache-mechanism for load-balancing

cache.cache_ram_capacity
268435456

Memory cache capacity (Byte), Each cache-worker’s capacity for memory-cache is cache.cache_ram_capacity div cache.cache_workers , and needs to satisfy the following conditions:

(8 * 1024 * 1024) * ${cache.cache_workers} >= ${cache.cache_ram_capacity}

cache.cache_disc_capacity
524288000

Disk cache capacity (Byte), Each cache-worker’s capacity for disk-cache is cache.cache_disc_capacity div cache.cache_workers , and needs to satisfy the following conditions:

(8 * 1024 * 1024) * ${cache.cache_workers} >= ${cache.cache_disc_capacity}

cache.cache_disc_threshold_len
1048576
When the length of the object exceeds this value, stores the object on the local disk

cache.cache_disc_dir_data
./cache/data
Directory for the disk cache data

cache.cache_disc_dir_journal
./cache/journal
Directory for the disk cache journal

cache.cache_expire
300
Cache Expire in seconds

cache.cache_max_content_len
1048576
Cache Max Content Length in bytes

cache.cachable_content_type
[]
Cache Content Type(s)

cache.cachable_path_pattern
[]
Cache Path Pattern(s) (regular expression)

Watchdog

rex(rpc)

watchdog.rex.interval
5
Watch interval for the rex*(rpc)* process in order to execute force GC when reaching the threshold value

watchdog.rex.threshold_mem_capacity
33554432
Threshold memory capacity of binary for rex*(rpc)*

CPU

watchdog.cpu.is_enabled
false
Enabled or disabled CPU-watchdog [true|false]

watchdog.cpu.interval
5
Watch interval(sec) for CPU-watchdog

watchdog.cpu.raised_error_times
3
An error is raised to subscribers when a number of errors reached this configuration.

watchdog.cpu.threshold_cpu_load_avg
5.0
Threshold CPU load avg for 1min/5min

watchdog.cpu.threshold_cpu_util
100
Threshold CPU load util (%)

Erlang IO (Erlang’s VM Internal Traffic)

watchdog.io.is_enabled
false
Enabled or disabled io-watchdog [true|false]

watchdog.io.interval
1
Watch interval (sec) for Eralng-IO watchdog

watchdog.io.threshold_input_per_sec
134217728
Threshold input size/sec (byte)

watchdog.io.threshold_output_per_sec
134217728
Threshold output size/se (byte)

Timeout

timeout.level_1
5000
Timeout value when requesting put-operation to a storage (msec) - 0 to 65,535 bytes

timeout.level_2
7000
Timeout value when requesting put-operation to a storage (msec) - 65,535 to 131,071 bytes

timeout.level_3
10000
Timeout value when requesting put-operation to a storage (msec) - 131,072 to 524,287 bytes

timeout.level_4
20000
Timeout value when requesting put-operation to a storage (msec) - 524,288 to 1,048,576 bytes

timeout.level_5
30000
Timeout value when requesting put-operation to a storage (msec) - 1,048,576 bytes and over

timeout.get
30000
Timeout value when requesting get-operation to a storage (msec)

timeout.ls
30000
Timeout value when requesting a list-object to a storage (msec)

Log

log.log_level
1
Log level: [0:debug, 1:info, 2:warn, 3:error]

log.is_enable_access_log
false
Enable or disable access-log output [true, false]

log.erlang
./log/erlang
Locatio of Erlang log-files

log.app
./log/app
Locatio of Gateway’s log-files

log.member_dir
./log/ring
Location of dump files of members of a storage cluster

log.ring_dir
./log/ring
Location of RING’s dump files

Others*

queue_dir
./work/queue
Directory of queue for monitoring “RING” and members of a storage cluster

snmp_agent
./snmp/snmpa_gateway_0/LEO-GATEWAY
Directory of SNMP agent configuration

See Also

LeoFS watchdog configuration

issue#531 (The last part of a large object can be broken with reading_chunked_obj_len > chunked_obj_len in leo_gateway.conf)

Erlang VM configuration

Property | Default Value | Description
nodename | gateway_0@127.0.0.1 | Node name of LeoFS Storage
distributed_cookie | 401321b4 | Cookie for distributed node communication
erlang.kernel_poll | true | Enable or disable kernel poll
erlang.async_threads | 32 | Number of async threads
erlang.max_ports | 64000 | Number of concurrent ports/sockets
erlang.crash_dump | ./log/erl_crash.dump | Location of crash dumps
erlang.max_ets_tables | 256000 | ETS table limit
erlang.smp | enable | Enable or disable SMP
erlang.schedulers.compaction_of_load | true | Erlang scheduler’s compaction of load
erlang.schedulers.utilization_balancing | false | Erlang scheduler’s balancing of load
erlang.distribution_buffer_size | 32768 | Sender-side network distribution buffer size (KB)
erlang.fullsweep_after | 0 | A non-negative integer which indicates how many times generational garbage collections can be done without forcing a fullsweep collection
erlang.secio | false | Enable or disable eager check I/O (Erlang 17.4/erts-6.3-, ref:OTP-12117)
process_limit | 1048576 | Default erlang process limit
snmp_conf | ./snmp/snmpa_storage_0/leo_gateway_snmp | SNMPA configuration files directory

See Also

Eralng - erl

Configuration related to Disk Cache

A total number of directories to store cache files is equal to cache.cache_workers. A maximum size of a cacheable object per a directory has been determined by Maximum cacheable object size = cache.cache_disc_capacity / cache.cache_workers. If size of a requested object more than the maximum size, LeoFS Gateway avoids storing the object into the disk cache.

And also, when size of a requested object more than cache.cache_max_content_len, LeoFS Gateway similarly refuses storing the object into the disk cache.

Figure of a disk cache:

cache.cache_workers = 16

cache.cache_disc_capacity = 500MB

Configuration of eager check I/O scheduling for Erlang’s VM

If you adopt LeoFS v1.2.7 later w/Erlang 17.5, we recommend you turn on erlang.secio. So you need to modify the configuration files and the schema files as follows.

Turn on erlang.secio at leo_gateway.conf

erlang.secio = true

Uncomment every rows of erlang.secio at leo_storage.schema

{mapping,
  "erlang.secio",
  "vm_args.+secio",
  [
    {default, false}
  ]}.

See Also

Eralng - erl

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Configuration »

SNMPA Setup

Each LeoFS node provides a built in SNMP server which allows to connect external systems, such as Nagios and Zabbix. You can retrieve various statistics as follows:

LeoFS Manager

SNMPA Properties

Property | Value / Range
Port | 4020 .. 4022
Branch | 1.3.6.1.4.1.35450.15
snmpa_manager_0 | Port: 4020
snmpa_manager_1 | Port: 4021
snmpa_manager_2 | Port: 4022

SNMPA Items

Branch Number | Description
1 | Node name
1-min averages
2 | # of processes
3 | Total memory usage
4 | System memory usage
5 | Processes memory usage
6 | ETS memory usage
5-min averages
7 | # of processes
8 | Total memory usage
9 | System memory usage
10 | Processes memory usage
11 | ETS memory usage
allocated memmory
12 | Used/allocated memory for 1min
13 | Allocated memory for 1min
14 | Used/allocated memory for 5min
15 | Allocated memory for 5min

Check the configuration

$ snmpwalk -v 2c -c public 127.0.0.1:4020 .1.3.6.1.4.1.35450
SNMPv2-SMI::enterprises.35450.15.1.0 = STRING: "manager_0@127.0.0.1"
SNMPv2-SMI::enterprises.35450.15.2.0 = Gauge32: 123
SNMPv2-SMI::enterprises.35450.15.3.0 = Gauge32: 30289989
SNMPv2-SMI::enterprises.35450.15.4.0 = Gauge32: 24256857
SNMPv2-SMI::enterprises.35450.15.5.0 = Gauge32: 6033132
SNMPv2-SMI::enterprises.35450.15.6.0 = Gauge32: 1914017
SNMPv2-SMI::enterprises.35450.15.7.0 = Gauge32: 123
SNMPv2-SMI::enterprises.35450.15.8.0 = Gauge32: 30309552
SNMPv2-SMI::enterprises.35450.15.9.0 = Gauge32: 24278377
SNMPv2-SMI::enterprises.35450.15.10.0 = Gauge32: 6031175
SNMPv2-SMI::enterprises.35450.15.11.0 = Gauge32: 1935758
SNMPv2-SMI::enterprises.35450.15.12.0 = Gauge32: 75
SNMPv2-SMI::enterprises.35450.15.13.0 = Gauge32: 84635402
SNMPv2-SMI::enterprises.35450.15.14.0 = Gauge32: 78
SNMPv2-SMI::enterprises.35450.15.15.0 = Gauge32: 88735915

LeoFS Storage

SNMPA Properties

Property | Value / Range
Port | 4010 .. 4013
Branch | 1.3.6.1.4.1.35450
snmpa_storage_0 | Port: 4010
snmpa_storage_1 | Port: 4011
snmpa_storage_2 | Port: 4012
snmpa_storage_3 | Port: 4013

SNMPA Items

Branch Number | Description
1 | Node name
VM-related values (1min averages)
2 | # of processes
3 | Total memory usage
4 | System memory usage
5 | Processes memory usage
6 | ETS memory usage
VM-related values (5min averages)
7 | # of processes
8 | Total memory usage
9 | System memory usage
10 | Processes memory usage
11 | ETS memory usage
Request counter (for 1min)
12 | # of WRITEs
13 | # of READs
14 | # of DELETEs
Request counter (for 5min)
15 | # of WRITEs
16 | # of READs
17 | # of DELETEs
# of objects
18 | # of active objects
19 | Total objects
20 | Total size of active objects
21 | Total size
MQ-related properties
22 | # of replication messages
23 | # of sync-vnode messages
24 | # of rebalance messages
allocated memmory
31 | Used/allocated memory for 1min
32 | Allocated memory for 1min
33 | Used/allocated memory for 5min
34 | Allocated memory for 5min

Check the configuration

$ snmpwalk -v 2c -c public 127.0.0.1:4010 .1.3.6.1.4.1.35450
SNMPv2-SMI::enterprises.35450.34.1.0 = STRING: "storage_0@127.0.0.1"
SNMPv2-SMI::enterprises.35450.34.2.0 = Gauge32: 227
SNMPv2-SMI::enterprises.35450.34.3.0 = Gauge32: 33165164
SNMPv2-SMI::enterprises.35450.34.4.0 = Gauge32: 24504020
SNMPv2-SMI::enterprises.35450.34.5.0 = Gauge32: 8661144
SNMPv2-SMI::enterprises.35450.34.6.0 = Gauge32: 1952903
SNMPv2-SMI::enterprises.35450.34.7.0 = Gauge32: 227
SNMPv2-SMI::enterprises.35450.34.8.0 = Gauge32: 33379629
SNMPv2-SMI::enterprises.35450.34.9.0 = Gauge32: 24493694
SNMPv2-SMI::enterprises.35450.34.10.0 = Gauge32: 8885935
SNMPv2-SMI::enterprises.35450.34.11.0 = Gauge32: 1941680
SNMPv2-SMI::enterprises.35450.34.12.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.13.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.14.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.15.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.16.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.17.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.18.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.19.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.20.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.21.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.22.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.23.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.24.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.31.0 = Gauge32: 75
SNMPv2-SMI::enterprises.35450.34.32.0 = Gauge32: 84635402
SNMPv2-SMI::enterprises.35450.34.33.0 = Gauge32: 78
SNMPv2-SMI::enterprises.35450.34.34.0 = Gauge32: 88735915

LeoFS Gateway

SNMPA Properties

Item | Value / Range
Port | 4000 .. 4001
Branch | 1.3.6.1.4.1.35450
snmpa_gateway_0 | Port: 4000
snmpa_gateway_1 | Port: 4001

SNMPA Items

Branch Number | Description
1 | Node name
VM-related values (1-min averages)
2 | # of processes
3 | Total memory usage
4 | System memory usage
5 | Processes memory usage
6 | ETS memory usage
VM-related values (5-min averages)
7 | # of processes
8 | Total memory usage
9 | System memory usage
10 | Processes memory usage
11 | ETS memory usage
Request counter (for 1min)
12 | # of WRITEs
13 | # of READs
14 | # of DELETEs
Request counter (for 5min)
15 | # of WRITEs
16 | # of READs
17 | # of DELETEs
Cache-related properties
18 | Count of cache-hit
19 | Count of cache-miss
20 | Total of files (objects)
21 | Total cached size
allocated memmory
31 | Used/allocated memory for 1min
32 | Allocated memory for 1min
33 | Used/allocated memory for 5min
34 | Allocated memory for 5min

Check the configuration

$ snmpwalk -v 2c -c public 127.0.0.1:4000 .1.3.6.1.4.1.35450
SNMPv2-SMI::enterprises.35450.34.1.0 = STRING: "gateway_0@127.0.0.1"
SNMPv2-SMI::enterprises.35450.34.2.0 = Gauge32: 279
SNMPv2-SMI::enterprises.35450.34.3.0 = Gauge32: 45266128
SNMPv2-SMI::enterprises.35450.34.4.0 = Gauge32: 36653905
SNMPv2-SMI::enterprises.35450.34.5.0 = Gauge32: 8612223
SNMPv2-SMI::enterprises.35450.34.6.0 = Gauge32: 2276519
SNMPv2-SMI::enterprises.35450.34.7.0 = Gauge32: 279
SNMPv2-SMI::enterprises.35450.34.8.0 = Gauge32: 45157433
SNMPv2-SMI::enterprises.35450.34.9.0 = Gauge32: 36385227
SNMPv2-SMI::enterprises.35450.34.10.0 = Gauge32: 8772210
SNMPv2-SMI::enterprises.35450.34.11.0 = Gauge32: 2261105
SNMPv2-SMI::enterprises.35450.34.12.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.13.0 = Gauge32: 13
SNMPv2-SMI::enterprises.35450.34.14.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.15.0 = Gauge32: 3
SNMPv2-SMI::enterprises.35450.34.16.0 = Gauge32: 24
SNMPv2-SMI::enterprises.35450.34.17.0 = Gauge32: 0
SNMPv2-SMI::enterprises.35450.34.18.0 = Gauge32: 21
SNMPv2-SMI::enterprises.35450.34.19.0 = Gauge32: 39
SNMPv2-SMI::enterprises.35450.34.20.0 = Gauge32: 3
SNMPv2-SMI::enterprises.35450.34.21.0 = Gauge32: 565700
SNMPv2-SMI::enterprises.35450.34.31.0 = Gauge32: 75
SNMPv2-SMI::enterprises.35450.34.32.0 = Gauge32: 84635402
SNMPv2-SMI::enterprises.35450.34.33.0 = Gauge32: 78
SNMPv2-SMI::enterprises.35450.34.34.0 = Gauge32: 88735915

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Configuration »

Multi data center replication

Since

LeoFS v1.0.0

Purpose

This section is a step by step guide to setting up the multi data center replication.

Getting Started

Pre-requirement

Note

We have checked this mechanism with CentOS 6.5 and Ubuntu Server 14.04 LTS but we’re goinng to investigate other OS such as FreeBSD and SmartOS.

LeoFS Manager Configuration

Set leo_manager.conf - DC Id and Cluster Id

If you need to change DC’s Id and Cluster’s Id, set system.dc_id and cluster_id

## DC Id
system.dc_id = dc_N

## Cluster Id
system.cluster_id = leofs_N

Set RPC-related configuration

You need to set RPC-related configuration as follows to be able to communicate each cluster with LeoFS’s RPC. Regarding rpc.server.listen_port, which is used by the join-cluster command.

## RPC-Server's acceptors
rpc.server.acceptors = 16

## RPC-Server's listening port number
rpc.server.listen_port = 13075

## RPC-Server's listening timeout(second)
rpc.server.listen_timeout = 5000

## RPC-Client's size of connection pool
rpc.client.connection_pool_size = 16

## RPC-Client's size of connection buffer
rpc.client.connection_buffer_size = 16

LeoFS Storage Configuration

Set RPC-related configuration

You need to set RPC-related configuration as follows to be able to communicate each cluster with LeoFS’s RPC.

## RPC-Server's acceptors
## this value must be determinted by following logic
## rpc.server.acceptor need to be larger than
## rpc.client.connection_pool(buffer)_size * "# of storage nodes + # of manager nodes in your cluster"
## The default value is suitable for less than 16 nodes in a cluster
rpc.server.acceptors = 128

## RPC-Server's listening port number
rpc.server.listen_port = 13077

## RPC-Server's listening timeout(second)
rpc.server.listen_timeout = 30000

## RPC-Client's size of connection pool
rpc.client.connection_pool_size = 8

## RPC-Client's size of connection buffer
rpc.client.connection_buffer_size = 8

See Also: Multi Data Center Replication (1st phase)

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Configuration »

LeoFS with NFS

Since

LeoFS v1.1.0

Purpose

This section is a step by step guide to setting up LeoFS with NFS. By following this tutorial you can easily build a LeoFS system with NFS.

Getting Started

Pre-requirement

Note

We have checked this mechanism with CentOS 6.5 and Ubuntu Server 14.04 LTS but we’re goinng to investigate other OS such as FreeBSD and SmartOS.

Install NFS client on CentOS 6.5

$ sudo yum install nfs-utils

Install NFS client on Ubuntu Server 14.04 LTS

$ sudo apt-get install nfs-common

Configuration

Modify leo_gateway.conf

Set protocol to nfs

Set large_object.chunked_obj_len to 1048576 (ubuntu) / 65536 (FreeBSD)

Set large_object.threshold_of_chunk_len to 1048576 (ubuntu) / 65536 (FreeBSD)

Note

If you also want to access LeoFS via S3 intereface, you need to start another LeoFS Gateway with setting protocol to s3.

## Gateway protocol to use: [s3 | rest | embed | nfs]
protocol = nfs

## Length of a chunked object
large_object.chunked_obj_len = 1048576

## Threshold of length of a chunked object
large_object.threshold_of_chunk_len = 1048576

Start LeoFS as NFS Server with other dependent programs

Start LeoFS as usual

Start rpcbind

$ sudo service rpcbind start

Create a bucket and a token for NFS with leofs-adm

$ ./leofs-adm add-bucket test 05236
OK
$ ./leofs-adm get-buckets
cluster id   | bucket   | owner       | permissions      | created at
-------------+----------+-------------+------------------+---------------------------
leofs_1      | test     | _test_leofs | Me(full_control) | 2014-07-31 10:20:42 +0900

$ ./leofs-adm gen-nfs-mnt-key test 05236 127.0.0.1
bb5034f0c740148a346ed663ca0cf5157efb439f

Create a mount point and Mount

$ sudo mkdir /mnt/leofs
## for Linux - "sudo mount -t nfs -o nolock <host>:/<bucket>/<token> <dir>"
$ sudo mount -t nfs -o nolock 127.0.0.1:/test/05236/bb5034f0c740148a346ed663ca0cf5157efb439f /mnt/leofs
## for FreeBSD - "sudo mount -t nfs -o nolockd <host>:/<bucket>/<token> <dir>"
$ sudo mount -t nfs -o nolockd 127.0.0.1:/test/05236/bb5034f0c740148a346ed663ca0cf5157efb439f /mnt/leofs

Now you can operate the bucket test in LeoFS as a filesystem via /mnt/leofs.

Confirm that NFS works

Create a file

$ touch /mnt/leofs/newfile
$ ls -al /mnt/leofs

drwxrwxrwx. 0 root root 4096 Jul 31 10:09 2014 .
drwxr-xr-x. 6 root root 4096 Jul 11 12:38 2014 ..
-rw-rw-rw-  0 root root    0 Jul 31 10:25 2014 newfile

Modify a file

$ echo "hello world" > /mnt/leofs/newfile
$ cat /mnt/leofs/newfile

hello world

Copy a file

$ cp /mnt/leofs/newfile /mnt/leofs/newfile.copy
$ ls -al /mnt/leofs

drwxrwxrwx  0 root root 4096 Jul 31 10:09 2014 .
drwxr-xr-x. 6 root root 4096 Jul 11 12:38 2014 ..
-rw-rw-rw-  0 root root   12 Jul 31 10:29 2014 newfile
-rw-rw-rw-  0 root root   12 Jul 31 10:31 2014 newfile.copy

$ ./leofs-adm whereis test/newfile
-------+--------------------------+--------------------------------------+------------+--------------+----------------+----------------+----------------------------
 del?  |           node           |             ring address             |    size    |   checksum   |  # of chunks   |     clock      |             when
-------+--------------------------+--------------------------------------+------------+--------------+----------------+----------------+----------------------------
       | storage_0@127.0.0.1      | 22f3d93762d31abc5f5704f78edf1691     |        12B |   6f5902ac23 |              0 | 4ffe2d105f1f4  | 2014-07-31 10:29:01 +0900

$ ./leofs-adm whereis test/newfile.copy
-------+--------------------------+--------------------------------------+------------+--------------+----------------+----------------+----------------------------
 del?  |           node           |             ring address             |    size    |   checksum   |  # of chunks   |     clock      |             when
-------+--------------------------+--------------------------------------+------------+--------------+----------------+----------------+----------------------------
       | storage_0@127.0.0.1      | d02e1e52d93242d2dcdb98224421a1fb     |        12B |   6f5902ac23 |              0 | 4ffe2d20343a3  | 2014-07-31 10:31:17 +0900

Diff files

$ diff /mnt/leofs/newfile /mnt/leofs/newfile.copy

Remove a file

$ rm /mnt/leofs/newfile
$ ls -al /mnt/leofs

drwxrwxrwx  0 root root 4096 Jul 31 10:09 2014 .
drwxr-xr-x. 6 root root 4096 Jul 11 12:38 2014 ..
-rw-rw-rw-  0 root root   12 Jul 31 10:31 2014 newfile.copy

$ ./leofs-adm whereis test/newfile
-------+--------------------------+--------------------------------------+------------+--------------+----------------+----------------+----------------------------
 del?  |           node           |             ring address             |    size    |   checksum   |  # of chunks   |     clock      |             when
-------+--------------------------+--------------------------------------+------------+--------------+----------------+----------------+----------------------------
  *    | storage_0@127.0.0.1      | 22f3d93762d31abc5f5704f78edf1691     |         0B |   d41d8cd98f |              0 | 4ffe2e5d9cffe  | 2014-07-31 10:34:50 +0900

Create a directory

$ mkdir -p /mnt/leofs/1/2/3
$ ls -alR /mnt/leofs/1

/mnt/leofs/1:
drwxrwxrwx 0 root root 4096 Jul 31 19:37 2014 .
drwxrwxrwx 0 root root 4096 Jul 31 10:09 2014 ..
drwxrwxrwx 0 root root 4096 Jul 31 10:37 2014 2

/mnt/leofs/1/2:
drwxrwxrwx 0 root root 4096 Jul 31 19:37 2014 .
drwxrwxrwx 0 root root 4096 Jul 31 19:37 2014 ..
drwxrwxrwx 0 root root 4096 Jul 31 10:37 2014 3

/mnt/leofs/1/2/3:
drwxrwxrwx 0 root root 4096 Jul 31 19:37 2014 .
drwxrwxrwx 0 root root 4096 Jul 31 19:37 2014 ..

Remove files recursively

$ rm -rf /mnt/leofs/1/
$ ls -al /mnt/leofs

drwxrwxrwx  0 root root 4096 Jul 31 10:09 2014 .
drwxr-xr-x. 6 root root 4096 Jul 11 12:38 2014 ..
-rw-rw-rw-  0 root root   12 Jul 31 10:31 2014 leofs.copy

And other basic file/directory operations also should work except controlling owners/permissions/symbolic links/special files.

Configuration

You can change the port number of the NFS/Mount server and the number of acceptor processes at leo_gateway.conf.

Property | Description
nfs.port | Port number the NFS server use
nfs.num_of_acceptors | The number of acceptor processes listening for NFS server connection
mount.port | Port number the Mount server use
mount.num_of_acceptors | The number of acceptor processes listening for Mount server connection

Limits

Since LeoFS NFS implementation is still the beta version, there are some limitations. The details are described at LeoFS Limits [https://leo-project.net/leofs/docs/faq/faq_2.html#nfs-support]

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Configuration »

Watchdog

Since

LeoFS v1.2.2

Purpose

The watchdog mechanism monitors CPU, Erlang-IO, disk and a storage cluster status in order to keep running a node stably. Also, this mechanism communicates with the MQ mechanism and the auto-compaction/compaction mechanism. Their processings - comsumption of MQ’s messages and deletion of unnecessary objects are affected by this mechanism.

Getting Started

Pre-requirement

Note

We have supported this mechanism for CentOS 6.5/7.0, Ubuntu Server 14.04 LTS, FreeBSD and SmartOS. And also, disk monitoring is using the iostat command and df command, you need to install it before getting started the watchdog.

Configuration of LeoFS Gateway

Note

All default setting of the watchdogs are disabled. Before getting use this mechanism, you need to turn on them.

Modify leo_gateway.conf at the section of wathdog.

LeoFS Gateway’s watchdog properties:

Property | Default value | Description
rex (rpc-server)
watchdog.rex.interval | 5 | Watch interval (sec)
watchdog.rex.threshold_mem_capacity | 33554432 | Threshold memory capacity for binary (byte)
CPU
watchdog.cpu.is_enabled | false | Is cpu-watchdog enabled? [true|false]
watchdog.cpu.interval | 5 | Watch interval(sec)
watchdog.cpu.raised_error_times | 3 | An error is raised to subscribers when a number of errors reached this configuration.
watchdog.cpu.threshold_cpu_load_avg | 5.0 | Threshold CPU load avg for 1min/5min
watchdog.cpu.threshold_cpu_util | 100 | Threshold CPU load util (%)

Configuration of LeoFS Storage

Note

All default setting of the watchdogs are disabled. Before getting use this mechanism, you need to turn on them.

Modify leo_storage.conf at the section of watchdog.

LeoFS Storage’s watchdog properties:

Property | Default value | Description
rex (rpc-server)
watchdog.rex.interval | 10 | Watch interval (sec)
watchdog.rex.threshold_mem_capacity | 33554432 | Threshold memory capacity for binary (byte)
CPU
watchdog.cpu.is_enabled | false | Is cpu-watchdog enabled? [true|false]
watchdog.cpu.interval | 10 | Watch interval (sec)
watchdog.cpu.raised_error_times | 5 | An error is raised to subscribers when a number of errors reached this configuration.
watchdog.cpu.threshold_cpu_load_avg | 5.0 | Threshold CPU load avg for 1min/5min
watchdog.cpu.threshold_cpu_util | 100 | Threshold CPU load util (%)
DISK
watchdog.disk.is_enabled | false | Is disk-watchdog enabled? [true|false]
watchdog.disk.interval | 10 | Watch interval (sec)
watchdog.disk.raised_error_times | 5 | An error is raised to clients when a number of errors reached this configuration.
watchdog.disk.threshold_disk_use | 85 | Threshold disk usage(capacity) (%) - leo_watchdog is using df command
watchdog.disk.threshold_disk_util | 100 | Threshold disk util (%) - leo_watchdog is using iostat command
watchdog.disk.threshold_disk_rkb | 98304 | Threshold disk read KB/sec
watchdog.disk.threshold_disk_wkb | 98304 | Threshold disk write KB/sec
watchdog.disk.target_devices | [] | Target devices for checking disk utilization
Cluster
watchdog.cluster.is_enabled | false | Is cluster-watchdog enabled? [true|false]
watchdog.cluster.interval | 1 | Watch interval (sec)

See Also

LeoFS Storage configuration

LeoFS Auto-compaction configuration

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Configuration »

Auto-compaction

Since

LeoFS v1.2.2

Purpose

The compaction mechanism supports to remove unnecessary objects in a node of LeoFS Storage. And also, this mechanism does not block retrieving and putting object during data-compaction as much as possible.

Since LeoFS 1.2.2 it is possible to execute automatic data-compaction in order to reduce administration-costs and Keep running a storage cluster stably. You need to only modify the auto-compaction configuration.

Getting Started

Pre-requirement

Note

The auto-compaction mechanism communicates with the watchdog mechanism in order to restrict load of the processing and keep running the node. If you would like to use that, you need to set up the watchdog configuration.

Configuration of LeoFS Storage

Modify leo_storage.conf at the section of auto-compaction.

LeoFS Storage’s auto-compaction properties:

Property
Default value
Description

autonomic_op.compaction.is_enabled
false

Is enabled auto-compaction? [true|false]

autonomic_op.compaction.parallel_procs
1

Number of parallel procs of data-compaction

autonomic_op.compaction.interval
3600

Execution interval time between data-compaction (sec)

autonomic_op.compaction.warn_active_size_ratio
70

Warning ratio of active size (%)

NOT affects the auto-compaction by this configuration yet. We plan to support this configuration with v1.4.

autonomic_op.compaction.threshold_active_size_ratio
60

Threshold ratio of active size (%)

(Size of active objects / Size of total objects) * 100 (%)

When it becomes less than the ratio, LeoFS Storage executes data-compaction automatically

See Also

LeoFS Storage configuration

LeoFS Watchdog configuration

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Administration Guide

We explain core administrative tasks and practices that operators of LeoFS will want to consider.

leofs-adm Command Line

Launch and Operation Flow

Operation Flow Diagram

LeoFS launch order

Description of the Operations

LeoFS launch step by step

Start LeoFS Manager’s master

Start LeoFS Manager’s slave

Start LeoFS Storage

Connect LeoFS Manager with leofs-adm

The “start” command - Start LeoFS storage cluster

Confirm state of the every LeoFS storage

Start LeoFS Gateway

Confirm state of the every node

General Commands

status

status <node>

whereis <file-path>

LeoFS Storage Operation

LeoFS Storage’s Cluster Operation

detach <storage-node>

suspend <storage-node>

resume <storage-node>

start

rebalance

Storage MQ Operation

Since

Overview

mq-stats <storage-node>

mq-suspend <storage-node> <mq-id>

mq-resume <storage-node> <mq-id>

See Also

Recover Commands

recover-file <file-path>

recover-node <node>

recover-ring <node>

recover-cluster <cluster-id>

Data Compaction Commands

compact-start <storage-node> (all | <num-of-targets>) [<num-of-compaction-procs>]

compact-suspend <storage-node>

compact-resume <storage-node>

compact-status <storage-node>

diagnose-start <storage-node>

du <storage-node>

du detail <storage-node>

LeoFS Gateway Operation

purge-cache <file-path>

remove-gateway <gateway-node>

LeoFS Manager Maintenance

backup-mnesia <backup-filepath>

restore-mnesia <backup-filepath>

update-managers <manager-master> <manager-slave>

dump-ring (<manager-node>|<storage-node>|<gateway-node>)

S3-API Commands

S3-API Commands - User

create-user <user-id> <password>

delete-user <user-id>

get-users

update-user-role <user-id> <role-id>

S3-API Commands - Endpoint

add-endpoint <endpoint>

delete-endpoint <endpoint>

get-endpoints

S3-API Commands - Bucket

add-bucket <bcuket> <access-key-id>

delete-bucket <bucket> <access-key-id>

get-buckets

get-bucket <access-key-id>

chown-bucket <bucket> <access-key-id>

update-acl <bucket> <access-key-id>

Canned ACL

Administration of Multi Datacenter Replication

join-cluster <remote-manager-master>:<rpc-port> <remote-manager-slave>:<rpc-port>

remove-cluster <remote-manager-master>:<rpc-port> <remote-manager-slave>:<rpc-port>

cluster-status

System Maintenance

Upgrade your old version LeoFS to v1.3.1

Upgrade flow diagram

Takeover a part of confugurations

Adjust Every Path

Attach/Detach node into a Storage-cluster in operation

LeoFS Gateway Access-log Format

Since

Overview

Sample

Format

LeoFS Storage Data Diagnosis log Format

Since

Overview

Sample - List of files

Format - List of files

Sample - Report of a data diagnosis / data compaction

Format - Report of a data diagnosis / data compaction

Diagnosis Log Dir

Diagnosis Log Files

Recover a storage-node

Since

Overview

Workflow

See also

Administration of CDN Integration

Since

How to integrate LeoFS with CDN

How to use http_custom_header.conf

How to write http_custom_header.conf

Use cases

Appendix

Syntax for the expire field

List of verified CDN services

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Administration Guide »

leofs-adm Command Line

leofs-adm makes administrative operations of LeoFS very easy.

Shell
Description

General Commands

leofs-adm status [<node>]

Retrieve status of every node (default)

Retrieve status of the specified node

leofs-adm whereis <file-path>

Retrieve an assigned object by the file-path

LeoFS Storage Cluster Operation

leofs-adm detach <storage-node>

Remove the storage node in the LeoFS storage cluster

Current status: running | stop

leofs-adm suspend <storage-node>

Suspend a storage node for maintenance

This command does NOT detach the node from the storage cluster

While suspending, it rejects any requests

Current status: running

leofs-adm resume <storage-node>

Resume a storage node for finished maintenance

Current status: suspended | restarted

leofs-adm start

Start LeoFS after distributing the RING from LeoFS Manager to LeoFS Storage and LeoFS Gateway

leofs-adm rebalance

Commit detached and attached nodes to join the cluster

Rebalance objects in the cluster based on the updated cluster topology

LeoFS Storage MQ Operation [1.2.0-]

leofs-adm mq-stats <storage-node>

See the statuses of message queues used in LeoFS Storage

leofs-adm mq-suspend <storage-node> <mq-id>

Suspend a process consuming a message queue

Active message queues only can be suspended

While suspending, no messages are consumed

leofs-adm mq-resume <storage-node> <mq-id>

Resume a process consuming a message queue

Recover Commands

leofs-adm recover-file <file-path>

Recover an inconsistent object specified by the file-path

leofs-adm recover-node <storage-node>

Recover all inconsistent objects in the specified node

leofs-adm recover-ring <storage-node>

Recover rings of the specified node

leofs-adm recover-cluster <cluster-id>

v1.0.0- Recover all inconsistent objects in the specified cluster

Data Compaction Commands

leofs-adm compact-start <storage-node> (all|<num-of-targets>) [<num-of-compaction-proc>]

Remove unnecessary objects from the node

num-of-targets: It controls the number of containers in parallel

num-of-compaction-procs: It controls the number of procs to execute the compaction in parallel

leofs-adm compact-suspend <storage-node>

Suspend the compaction

leofs-adm compact-resume <storage-node>

Resume the compaction

leofs-adm compact-status <storage-node>

See the current compaction status

Compaction’s status: idle, running, suspend

leofs-adm diagnose-start <storage-node>

v1.1.5- Diagnose data of a target storage node

Disk Usage

leofs-adm du <storage-node>

See the current disk usages

leofs-adm du detail <storage-node>

See the current disk usages in the details

LeoFS Gateway Operation

leofs-adm purge-cache <file-path>

Remove the cache from each LeoFS gateway

leofs-adm remove-gateway <gateway-node>

Remove the LeoFS Gateway node, which is already stopped

LeoFS Manager Maintenance

leofs-adm backup-mnesia <backup-filepath>

Copy LeoFS’s Manager data to the filepath

leofs-adm restore-mnesia <backup-filepath>

Restore LeoFS’s Manager data from the backup file

leofs-adm update-managers <manager-master> <manager-slave>

Update LeoFS Manager nodes

Destribute the new LeoFS Manager nodes to LeoFS Storage and Gateway

leofs-adm dump-ring (<manager-node>|<storage-node>|<gateway-node>)

Dump the ring data to the local disk

S3-API Commands - User

leofs-adm create-user <user-id> <password>

Register the new user

Generate an S3 key pair (AccessKeyID and SecretAccessKey)

leofs-adm delete-user <user-id>

Remove the user

leofs-adm get-users

Retrieve the list of users

leofs-adm update-user-role

Update the user’s role

Currently, we are supporting two kinds of roles

1: General user, 9: Administrator

S3-API Commands - Endpoint

leofs-adm add-endpoint <endpoint>

Register a new S3 Endpoint

LeoFS’ domains are ruled by this rule

leofs-adm delete-endpoint <endpoint>

Remove the endpoint

leofs-adm get-endpoints

Retrieve the list of endpoints

S3-API Commands - Bucket

leofs-adm add-bucket <bucket> <access-key-id>

Create the new bucket

leofs-adm delete-bucket <bucket> <access-key-id>

Remove the bucket and all files stored in the bucket

leofs-adm get-buckets

Retrieve the list of the buckets registered

leofs-adm get-bucket <access-key-id>

Retrieve the list of the buckets owned by the specified user

leofs-adm chown-bucket <bucket> <access-key-id>

v0.16.5- Change the owner of the bucket

leofs-adm update-acl <bucket> <access-key-id> (private | public-read | public-read-write)

v0.16.0- Update the ACL (Access Control List) for the bucket

Available ACL list:

private (default) : No one except the owner has access rights

public-read : All users have READ access

public-read-write : All users have READ and WRITE access

Multi Data Center Operation

leofs-adm join-cluster <manager-master> <manager-slave>
1.0.0- Begin to communicate between the local cluster and the remote cluster

leofs-adm remove-cluster <manager-master> <manager-slave>
1.0.0- Terminate to communicate between the local cluster and the remote cluster

leofs-adm cluster-status
1.0.0- See the current state of cluster(s)

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Administration Guide »

Launch and Operation Flow

Operation Flow Diagram

The LeoFS operation flow is as follows:

The diagram only [https://leo-project.net/leofs/docs/_images/leofs-flow-diagram.jpg]

LeoFS launch order

LeoFS’s system launch process is very simple:

Description of the Operations

Order | Command/Script | Description
1 | $ bin/leofs_manager start | Start LeoFS Manager’s master
2 | $ bin/leofs_manager start | Start LeoFS Manager’s slave
3 | $ bin/leofs_storage start | Start LeoFS Storage
4 | $ leofs-adm start | Start LeoFS storage cluster
5 | $ leofs-adm status | Confirm state of the every LeoFS Storage
6 | $ bin/leofs_gateway start | Start LeoFS Gateway
7 | $ leofs-adm status | Confirm state of the every node - LeoFS Storaage and Gateway

LeoFS launch step by step

Start LeoFS Manager’s master

$ cd $LEOFS_DEPLOYED_DIR
$ manager_0/bin/leo_manager start

Start LeoFS Manager’s slave

$ cd $LEOFS_DEPLOYED_DIR
$ manager_1/bin/leo_manager start

Start LeoFS Storage

$ cd $LEOFS_DEPLOYED_DIR
$ leo_storage/bin/leo_storage start

Connect LeoFS Manager with leofs-adm

‘status’ command - Inspect LeoFS-cluster

$ leofs-adm status
[System config]
                System version : 1.0.0
                    Cluster Id : leofs_1
                         DC Id : dc_1
                Total replicas : 3
           # of successes of R : 1
           # of successes of W : 2
           # of successes of D : 2
 # of DC-awareness replicas    : 0
                     ring size : 2^128
             Current ring hash :
                Prev ring hash :
[Multi DC replication settings]
         max # of joinable DCs : 2
            # of replicas a DC : 1

[Node(s) state]
-------+--------------------------+--------------+----------------+----------------+----------------------------
 type  |           node           |    state     |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
  S    | storage_0@127.0.0.1      | attached     |                |                | 2014-04-03 11:28:20 +0900
  S    | storage_1@127.0.0.1      | attached     |                |                | 2014-04-03 11:28:20 +0900
  S    | storage_2@127.0.0.1      | attached     |                |                | 2014-04-03 11:28:20 +0900
  S    | storage_3@127.0.0.1      | attached     |                |                | 2014-04-03 11:28:20 +0900

The “start” command - Start LeoFS storage cluster

$ leofs-adm start
OK

Confirm state of the every LeoFS storage

$ leofs-adm status
[System config]
                System version : 1.0.0
                    Cluster Id : leofs_1
                         DC Id : dc_1
                Total replicas : 3
           # of successes of R : 1
           # of successes of W : 2
           # of successes of D : 2
 # of DC-awareness replicas    : 0
                     ring size : 2^128
             Current ring hash : 8cd79c31
                Prev ring hash : 8cd79c31
[Multi DC replication settings]
         max # of joinable DCs : 2
            # of replicas a DC : 1

[Node(s) state]
-------+--------------------------+--------------+----------------+----------------+----------------------------
 type  |           node           |    state     |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
  S    | storage_0@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:20 +0900
  S    | storage_1@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:20 +0900
  S    | storage_2@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:20 +0900
  S    | storage_3@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:20 +0900

Start LeoFS Gateway

$ cd $LEOFS_DEPLOYED_DIR/
$ gateway/bin/leo_gateway start

Confirm state of the every node

$ leofs-adm status
[System config]
                System version : 1.0.0
                    Cluster Id : leofs_1
                         DC Id : dc_1
                Total replicas : 3
           # of successes of R : 1
           # of successes of W : 2
           # of successes of D : 2
 # of DC-awareness replicas    : 0
                     ring size : 2^128
             Current ring hash : 8cd79c31
                Prev ring hash : 8cd79c31
[Multi DC replication settings]
         max # of joinable DCs : 2
            # of replicas a DC : 1

[Node(s) state]
-------+--------------------------+--------------+----------------+----------------+----------------------------
 type  |           node           |    state     |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
  S    | storage_0@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:20 +0900
  S    | storage_1@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:20 +0900
  S    | storage_2@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:20 +0900
  S    | storage_3@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:20 +0900
  G    | gateway_0@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:21 +0900
  G    | gateway_1@127.0.0.1      | running      | 8cd79c31       | 8cd79c31       | 2014-04-03 11:28:21 +0900

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Administration Guide »

General Commands

Shell
Description

leofs-adm status [<node>]

Retrieve status of every node (default)

Retrieve status of the specified node

leofs-adm whereis <file-path>
Retrieve an assigned object by the file-path

status

Retrieve status of every node (default)

-----------------------------------+----------
 Item                              | Value
-----------------------------------+----------
 Basic/Consistency level
-----------------------------------+----------
                    system version | 1.2.11
                        cluster Id | leofs_1
                             DC Id | dc_1
                    Total replicas | 2
          number of successes of R | 1
          number of successes of W | 1
          number of successes of D | 1
 number of rack-awareness replicas | 0
                         ring size | 2^128
-----------------------------------+----------
 Multi DC replication settings
-----------------------------------+----------
        max number of joinable DCs | 2
           number of replicas a DC | 1
-----------------------------------+----------
 Manager RING hash
-----------------------------------+----------
                 current ring-hash | 3923d007
                previous ring-hash | 3923d007
-----------------------------------+----------

 [State of Node(s)]
-------+--------------------------+--------------+----------------+----------------+----------------------------
 type  |           node           |    state     |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
  S    | storage_0@127.0.0.1      | running      | 3923d007       | 3923d007       | 2015-07-02 15:28:06 +0900
  S    | storage_1@127.0.0.1      | running      | 3923d007       | 3923d007       | 2015-07-02 15:28:06 +0900
  S    | storage_2@127.0.0.1      | running      | 3923d007       | 3923d007       | 2015-07-02 15:28:06 +0900
  S    | storage_3@127.0.0.1      | running      | 3923d007       | 3923d007       | 2015-07-02 15:28:06 +0900
-------+--------------------------+--------------+----------------+----------------+----------------------------

status <node>

Retrieve status of the specified node

$ leofs-adm status storage_0@127.0.0.1
--------------------------------------+--------------------------------------
                Item                  |                 Value
--------------------------------------+--------------------------------------
 Config-1: basic
--------------------------------------+--------------------------------------
                              version | 1.2.11
                     number of vnodes | 168
                    object containers | - path:[./avs], # of containers:8
                        log directory | ./log/erlang
--------------------------------------+--------------------------------------
 Config-2: watchdog
--------------------------------------+--------------------------------------
 [rex(rpc-proc)]                      |
                    check interval(s) | 5
               threshold mem capacity | 33554432
--------------------------------------+--------------------------------------
 [cpu]                                |
                     enabled/disabled | disabled
                    check interval(s) | 5
               threshold cpu load avg | 5.0
                threshold cpu util(%) | 100
--------------------------------------+--------------------------------------
 [disk]                               |
                     enabled/disalbed | disabled
                    check interval(s) | 5
                threshold disk use(%) | 100
               threshold disk util(%) | 90
                    threshold rkb(kb) | 131072
                    threshold wkb(kb) | 131072
--------------------------------------+--------------------------------------
 Config-3: message-queue
--------------------------------------+--------------------------------------
                   number of procs/mq | 8
        number of batch-procs of msgs | max:1000, regular:400
   interval between batch-procs (ms)  | max:3000, regular:500
--------------------------------------+--------------------------------------
 Config-4: autonomic operation
--------------------------------------+--------------------------------------
 [auto-compaction]                    |
                     enabled/disabled | disabled
        warning active size ratio (%) | 70
      threshold active size ratio (%) | 60
             number of parallel procs | 1
                        exec interval | 3600
--------------------------------------+--------------------------------------
 Config-5: data-compaction
--------------------------------------+--------------------------------------
  limit of number of compaction procs | 4
        number of batch-procs of objs | max:500, regular:100
   interval between batch-procs (ms)  | max:1000, regular:300
--------------------------------------+--------------------------------------
 Status-1: RING hash
--------------------------------------+--------------------------------------
                    current ring hash | 3923d007
                   previous ring hash | 3923d007
--------------------------------------+--------------------------------------
 Status-2: Erlang VM
--------------------------------------+--------------------------------------
                           vm version | 6.4
                      total mem usage | 41476672
                     system mem usage | 23647720
                      procs mem usage | 17850112
                        ets mem usage | 5844976
                                procs | 364/1048576
                          kernel_poll | true
                     thread_pool_size | 32
--------------------------------------+--------------------------------------
 Status-3: Number of messages in MQ
--------------------------------------+--------------------------------------
                 replication messages | 0
                  vnode-sync messages | 0
                   rebalance messages | 0
--------------------------------------+--------------------------------------

$ leofs-adm status gateway_0@127.0.0.1
-------------------------------+------------------
             Item              |       Value
-------------------------------+------------------
 Config-1: basic
--------------------------------------------------
 [basic]
-------------------------------+------------------
                       version | 1.2.11
                using protocol | s3
                 log directory | ./log/erlang
-------------------------------+------------------
 [http server related for rest/s3 api]
-------------------------------+------------------
                listening port | 8080
            listening ssl port | 8443
           number of acceptors | 128
-------------------------------+------------------
 [cache-related]
-------------------------------+------------------
       http cache [true/false] | false
       number of cache_workers | 16
                  cache expire | 300
         cache max content len | 1048576
            ram cache capacity | 0
        disk cache capacity    | 0
        disk cache threshold   | 1048576
        disk cache data dir    | ./cache/data
        disk cache journal dir | ./cache/journal
-------------------------------+------------------
 [large object related]
-------------------------------+------------------
      max number of chunk objs | 1000
           chunk object length | 5242880
             max object length | 5242880000
     reading  chunk obj length | 5242880
     threshold of chunk length | 5767168
-------------------------------+------------------
 Config-2: watchdog
-------------------------------+------------------
 [rex(rpc-proc)]               |
             watch interval(s) | 5
        threshold mem capacity | 33554432
-------------------------------+------------------
 [cpu]                         |
              eanbled/disabled | disabled
             check interval(s) | 5
        threshold cpu load avg | 5.0
         threshold cpu util(%) | 100
-------------------------------+------------------
 [io]                          |
              enabled/disabled | disabled
             check interval(s) | 1
        threshold input size/s | 134217728
       threshold output size/s | 134217728
-------------------------------+------------------
 Status-1: RING hash
-------------------------------+------------------
             current ring hash | 3923d007
            previous ring hash | 3923d007
-------------------------------+------------------
 Status-2: Erlang VM
-------------------------------+------------------
                    vm version | 6.4
               total mem usage | 69980072
              system mem usage | 52410432
               procs mem usage | 17588560
                 ets mem usage | 6891272
                         procs | 439/1048576
                   kernel poll | true
              thread pool size | 32
-------------------------------+------------------

whereis <file-path>

Retrieve an assigned object by the file-path Paths used by whereis are ruled by this rule

$ leofs-adm whereis leo/fast/storage.key
-----------------------------------------------------------------------------------------------------------------------
 del? node                 ring address    size   # of chunks  checksum    vclock            when
-----------------------------------------------------------------------------------------------------------------------
      storage_1@127.0.0.1  207643840133    35409  0             4116193149  1332407492290951  2012-06-29 14:23:31 +0900
      storage_0@127.0.0.1  207643840133    35409  0             4116193149  1332407492290951  2012-06-29 14:23:31 +0900

If you want to retrieve an object whose file-path contains spaces, Enclose the file-path with double quotation.

$ leofs-adm whereis "leo/fast/storage with space.key"
-----------------------------------------------------------------------------------------------------------------------
 del? node                 ring address    size   # of chunks  checksum    vclock            when
-----------------------------------------------------------------------------------------------------------------------
      storage_1@127.0.0.1  207643840133    35409  0             4116193149  1332407492290951  2012-06-29 14:23:31 +0900
      storage_0@127.0.0.1  207643840133    35409  0             4116193149  1332407492290951  2012-06-29 14:23:31 +0900

If you want to retrieve a chunk object which is part of a large object, Append \n and the chunk number to the file-path.

$ leofs-adm whereis leo/fast/storage.key\n1
-----------------------------------------------------------------------------------------------------------------------
 del? node                 ring address    size   # of chunks  checksum    vclock            when
-----------------------------------------------------------------------------------------------------------------------
      storage_1@127.0.0.1  207643840133    5120K  0             4116193149  1332407492290951  2012-06-29 14:23:31 +0900
      storage_0@127.0.0.1  207643840133    5120K  0             4116193149  1332407492290951  2012-06-29 14:23:31 +0900

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Administration Guide »

LeoFS Storage Operation

LeoFS Storage’s Cluster Operation

LeoFS Storage operation commands are executed on LeoFS-Manager console OR the leofs-adm script.

Refer LeoFS operation flow diagram

Shell
Description

leofs-adm detach <storage-node>

Remove the storage node from the LeoFS storage cluster

Current status: running | stop

leofs-adm suspend <storage-node>

Suspend a storage node for maintenance

This command is NOT similar to the detach command, just only to suspend the node.

While suspending, it rejects any requests

Current status: running

leofs-adm resume <storage-node>

Resume a storage node until a finished maintenance

Current status: suspended | restarted

leofs-adm start

Start LeoFS after distributing the RING from LeoFS Manager to LeoFS Storage and Gateway

leofs-adm rebalance

Commit detached and attached nodes to update the cluster and Ring(routing-table)

Rebalance objects in the cluster which is based on the updated cluster topology

detach <storage-node>

Remove the storage node from the storage cluster

$ leofs-adm detach storage_0@127.0.0.1
OK
$ leofs-adm rebalance
OK

Method of cancellation of detach-operation

$ leofs-adm detach storage_0@127.0.0.1
OK
### Rollback this operation
$ leofs-adm rollback storage_0@127.0.0.1
OK

Method of deletion of an attached node

## Confirm current status of ther storage cluster
$ leofs-adm status
 [System Confiuration]
-----------------------------------+----------
 Item                              | Value
-----------------------------------+----------
 Basic/Consistency level
-----------------------------------+----------
                    system version | 1.2.12
                        cluster Id | leofs_1
                             DC Id | dc_1
                    Total replicas | 2
          number of successes of R | 1
          number of successes of W | 1
          number of successes of D | 1
 number of rack-awareness replicas | 0
                         ring size | 2^128
-----------------------------------+----------
 Multi DC replication settings
-----------------------------------+----------
        max number of joinable DCs | 2
           number of replicas a DC | 1
-----------------------------------+----------
 Manager RING hash
-----------------------------------+----------
                 current ring-hash | 0364701d
                previous ring-hash | 0364701d
-----------------------------------+----------

 [State of Node(s)]
-------+--------------------------+--------------+----------------+----------------+----------------------------
 type  |           node           |    state     |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
  S    | storage_0@127.0.0.1      | attached     |                |                | 2015-09-09 11:17:49 +0900
  S    | storage_1@127.0.0.1      | running      | 0364701d       | 0364701d       | 2015-09-09 11:13:31 +0900
  S    | storage_2@127.0.0.1      | running      | 0364701d       | 0364701d       | 2015-09-09 11:09:15 +0900
  G    | gateway_0@127.0.0.1      | running      | 0364701d       | 0364701d       | 2015-09-09 10:53:19 +0900
-------+--------------------------+--------------+----------------+----------------+----------------------------

## Remove an attached node
$ leofs-adm detach storage_0@127.0.0.1
OK

## Confirm the current status of the storage cluster, again
$ leofs-adm status
 [System Confiuration]
-----------------------------------+----------
 Item                              | Value
-----------------------------------+----------
 Basic/Consistency level
-----------------------------------+----------
                    system version | 1.2.12
                        cluster Id | leofs_1
                             DC Id | dc_1
                    Total replicas | 2
          number of successes of R | 1
          number of successes of W | 1
          number of successes of D | 1
 number of rack-awareness replicas | 0
                         ring size | 2^128
-----------------------------------+----------
 Multi DC replication settings
-----------------------------------+----------
        max number of joinable DCs | 2
           number of replicas a DC | 1
-----------------------------------+----------
 Manager RING hash
-----------------------------------+----------
                 current ring-hash | 0364701d
                previous ring-hash | 0364701d
-----------------------------------+----------

 [State of Node(s)]
-------+--------------------------+--------------+----------------+----------------+----------------------------
 type  |           node           |    state     |  current ring  |   prev ring    |          updated at
-------+--------------------------+--------------+----------------+----------------+----------------------------
  S    | storage_1@127.0.0.1      | running      | 0364701d       | 0364701d       | 2015-09-09 11:13:31 +0900
  S    | storage_2@127.0.0.1      | running      | 0364701d       | 0364701d       | 2015-09-09 11:09:15 +0900
  G    | gateway_0@127.0.0.1      | running      | 0364701d       | 0364701d       | 2015-09-09 10:53:19 +0900

suspend <storage-node>

Suspend the storage node

While suspending, it rejects any requests

This command does NOT detach the node from the storage cluster

$ leofs-adm suspend storage_0@127.0.0.1
OK

resume <storage-node>

Resume the storage node

$ leofs-adm resume storage_0@127.0.0.1
OK

start

Start LeoFS after distributing the RING from LeoFS Manager to LeoFS Storage and LeoFS Gateway

$ leofs-adm start
OK

rebalance

Commit detached and attached node(s) to update the cluster and Ring(routing-table)

Rebalance objects in the cluster which is based on the updated cluster topology

$ leofs-adm rebalance
OK

Storage MQ Operation

Since

LeoFS v1.2.2

Overview

LeoFS Storage MQ is controllable mechanism manually. We’ve published mq-suspend and mq-resume command in leofs-adm script. In addition, LeoFS’s MQ mechanism is affected by the watchdog mechanism to reduce comsumption of message costs.

Description of the each MQ

Id | Description
leo_delete_dir_queue | After executed leofs-adm delete-bucket, messages of deletion object is added into the queue.
leo_comp_meta_with_dc_queue | After executed leofs-adm recover-cluster, messages of comparison of metadata w/remote-node is added into the queue.
leo_sync_obj_with_dc_queue | After executed leofs-adm recover-cluster, messages of synchronization of objects w/remote-node is added into the queue.
leo_recovery_node_queue | After executed leofs-adm recover-node, messages of recovery objects of a node is added into the queue.
leo_async_deletion_queue | After executed leofs-adm delete-bucket OR deletion of a bucket OR deletion of an object, message of async deletion of objs is added into the queue.
leo_rebalance_queue | After executed leofs-adm rebalance, messages of rebalance is added in to the queue.
leo_sync_by_vnode_id_queue | After executed leofs-adm rebalance, messages of synchronization of virtual-nodes is added into the queue.
leo_per_object_queue | After executed leofs-adm rebalance OR leofs-adm recover-file OR leofs-adm recover-node OR fixing inconsistent object(s) with the recovery data mechanism, messages of recover inconsistent objects is added into the queue.

Commands

Shell
Description

leofs-adm mq-stats <storage-node>

See the statuses of message queues used in LeoFS Storage

leofs-adm mq-suspend <storage-node> <mq-id>

Suspend a process consuming a message queue

Active message queues only can be suspended

While suspending, no messages are consumed

leofs-adm mq-resume <storage-node> <mq-id>

Resume a process consuming a message queue

mq-stats <storage-node>

You can check tatuses of the message queues in the LeoFS’s storage node. Explanation of columns:

Column | Description
state | A status of the MQ - [idling, running, suspending]
number of msgs | A number of messages in the queue
batch of msgs | A batch of messages of the MQ’s message-consumption
interval | An interval time between the batch processing

$ ./leofs-adm mq-stats storage_0@127.0.0.1
              id                |    state    | number of msgs | batch of msgs  |    interval    |            description
--------------------------------+-------------+----------------|----------------|----------------|-----------------------------------
leo_delete_dir_queue            |   idling    | 0              | 1000           | 100            | delete directories
leo_comp_meta_with_dc_queue     |   idling    | 0              | 1000           | 100            | compare metadata w/remote-node
leo_sync_obj_with_dc_queue      |   idling    | 0              | 1000           | 100            | sync objs w/remote-node
leo_recovery_node_queue         |   idling    | 0              | 1000           | 100            | recovery objs of node
leo_async_deletion_queue        |   idling    | 0              | 1000           | 100            | async deletion of objs
leo_rebalance_queue             |   running   | 2167           | 1400           | 10             | rebalance objs
leo_sync_by_vnode_id_queue      |   idling    | 0              | 1000           | 100            | sync objs by vnode-id
leo_per_object_queue            |   idling    | 0              | 1000           | 100            | recover inconsistent objs

mq-suspend <storage-node> <mq-id>

Note

When turning on the watchdog mechanism, this command is ignored.

$ ./leofs-adm mq-suspend storage_0@127.0.0.1 leo_delete_dir_queue
OK

mq-resume <storage-node> <mq-id>

Note

When turning on the watchdog mechanism, this command is ignored.

$ ./leofs-adm mq-resume storage_0@127.0.0.1 leo_delete_dir_queue
OK

See Also

LeoFS Storage configuration

LeoFS Watchdog configuration

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Administration Guide »

Recover Commands

Shell | Description
leofs-adm recover-file <file-path> | Recover an inconsistent object specified by the file-path
leofs-adm recover-node <storage-node> | Recover all inconsistent objects in the specified node
leofs-adm recover-ring <storage-node> | Recover rings of the specified node
leofs-adm recover-cluster <cluster-id> | [v1.0.0-] Recover all inconsistent objects in the specified cluster

recover-file <file-path>

Recover an inconsistent object specified by the file-path

$ leofs-adm recover-file leo/fast/storage.key
OK

recover-node <node>

Recover all inconsistent objects in the specified node

$ leofs-adm recover-node storage_0@127.0.0.1
OK

recover-ring <node>

Recover rings of the specified node

$ leofs-adm recover-ring storage_0@127.0.0.1
OK

recover-cluster <cluster-id>

Recover all inconsistent objects in the specified cluster

$ leofs-adm recover-cluster cluster-1
OK

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Administration Guide »

Data Compaction Commands

Remove logical deleted objects and meta data and check the current disk usage

Shell
Description

Compaction Commands

leofs-adm compact-start <storage-node> (all|<num-of-targets>) [<num-of-compaction-proc>]

Remove unnecessary objects from the node

num-of-targets: It controls the number of containers in parallel

num-of-compaction-procs: It controls the number of procs to execute the compaction in parallel

leofs-adm compact-suspend <storage-node>

Suspend the compaction

leofs-adm compact-resume <storage-node>

Resume the compaction

leofs-adm compact-status <storage-node>

See the current compaction status

Compaction’s status: idle, running, suspend

leofs-adm diagnose-start <storage-node>

v1.1.5- Diagnose data of a target storage node

Disk Usage

leofs-adm du <storage-node>

See the current disk usages

leofs-adm du detail <storage-node>

See the current disk usages in the details

compact-start <storage-node> (all | <num-of-targets>) [<num-of-compaction-procs>]

Remove unnecessary objects from the node

num-of-targets: It controls the number of containers in parallel

num-of-compaction-procs: It controls the number of procs to execute the compaction in parallel

Note

Default <num-of-compation-procs> is ‘3’ - You can control the number of processes to execute compaction in parallel. It enables you to get maximum performance by setting an appropriate number corresponding to the number of cores.

## All compaction-targets will be executed with 3 concurrent processes
## (default concurrency is 3)
$ leofs-adm compact-start storage_0@127.0.0.1 all
OK

## Number of compaction-targets will be executed with 2 concurrent processes
$ leofs-adm compact-start storage_0@127.0.0.1 5 2
OK

compact-suspend <storage-node>

Suspend the compaction

$ leofs-adm compact-suspend storage_0@127.0.0.1
OK

compact-resume <storage-node>

Resume the compaction

$ leofs-adm compact-resume storage_0@127.0.0.1
OK

compact-status <storage-node>

See the current compaction status

Compaction’s status: idle, running and suspend

$ leofs-adm compact-status storage_0@127.0.0.1
        current status: running
 last compaction start: 2013-03-04 12:39:47 +0900
         total targets: 64
  # of pending targets: 5
  # of ongoing targets: 3
  # of out of targets : 56

diagnose-start <storage-node>

v1.1.5- Diagnose data of a target storage node

See also: LeoFS Storage data-diagnosis-log format

$ leofs-adm diagnose-start storage_0@127.0.0.1
OK

du <storage-node>

See the current disk usages

$ leofs-adm du storage_0@127.0.0.1
 active number of objects: 19968
  total number of objects: 39936
   active size of objects: 198256974.0
    total size of objects: 254725020.0
     ratio of active size: 77.83%
    last compaction start: 2013-03-04 12:39:47 +0900
      last compaction end: 2013-03-04 12:39:55 +0900

du detail <storage-node>

See the current disk usages in the details

$ leofs-adm du detail storage_0@127.0.0.1
[du(storage stats)]
                file path: /home/leofs/dev/leofs/package/leofs/storage/avs/object/0.avs
 active number of objects: 320
  total number of objects: 640
   active size of objects: 3206378.0
    total size of objects: 4082036.0
     ratio of active size: 78.55%
    last compaction start: 2013-03-04 12:39:47 +0900
      last compaction end: 2013-03-04 12:39:55 +0900
.
.
.
                file path: /home/leofs/dev/leofs/package/leofs/storage/avs/object/63.avs
 active number of objects: 293
  total number of objects: 586
   active size of objects: 2968909.0
    total size of objects: 3737690.0
     ratio of active size: 79.43%
    last compaction start: ____-__-__ __:__:__
      last compaction end: ____-__-__ __:__:__

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Administration Guide »

LeoFS Gateway Operation

Shell
Description

leofs-adm purge-cache <file-path>

Remove the cache from each LeoFS Gateway node

leofs-adm remove-gateway <gateway-node>

Remove the LeoFS Gateway node, which is already stopped

purge-cache <file-path>

Remove the cache from each gateway

$ leofs-adm purge-cache leo/fast/storage.key
OK

remove-gateway <gateway-node>

$ leofs-adm remove-gateway gateway_0@127.0.0.1
OK

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Administration Guide »

LeoFS Manager Maintenance

Shell
Description

leofs-adm backup-mnesia <backup-filepath>

Copy LeoFS’s Manager data to the filepath

leofs-adm restore-mnesia <backup-filepath>

Restore LeoFS’s Manager data from the backup file

leofs-adm update-managers <manager-master> <manager-slave>

Update LeoFS Manager nodes

Destribute the new LeoFS Manager nodes to LeoFS Storage and Gateway

leofs-adm dump-ring (<manager-node>|<storage-node>|<gateway-node>)

Dump the ring data to the local disk

backup-mnesia <backup-filepath>

Copy LeoFS’s Manager data to the file-path

$ leofs-adm backup-mnesia /path/to/file
OK

restore-mnesia <backup-filepath>

Restore LeoFS’s Manager data from the backup file

$ leofs-adm restore-mnesia /path/to/file
OK

update-managers <manager-master> <manager-slave>

Update LeoFS Manager nodes

Destribute the new LeoFS Manager nodes to LeoFS Storage and Gateway

$ leofs-adm update-managers manager_0@10.1.0.1 manager_1@10.1.0.2
OK

dump-ring (<manager-node>|<storage-node>|<gateway-node>)

Dump the ring data to the local disk

$ leofs-adm dump-ring storage_0@10.1.0.11
OK

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Administration Guide »

S3-API Commands

Shell
Description

S3-API Commands - User

leofs-adm create-user <user-id> <password>

Register the new user

Generate an S3 key pair (AccessKeyID and SecretAccessKey)

leofs-adm delete-user <user-id>

Remove the user

leofs-adm get-users

Retrieve the list of users

leofs-adm update-user-role

Update the user’s role

Currently, we are supporting two kinds of roles

1: General user, 9: Administrator

S3-API Commands - Endpoint

leofs-adm add-endpoint <endpoint>

Register a new S3 Endpoint

LeoFS’ domains are ruled by this rule

leofs-adm delete-endpoint <endpoint>

Remove the endpoint

leofs-adm get-endpoints

Retrieve the list of endpoints

S3-API Commands - Bucket

leofs-adm add-bucket <bucket> <access-key-id>

Create the new bucket

leofs-adm delete-bucket <bucket> <access-key-id>

Remove the bucket and all files stored in the bucket

leofs-adm get-buckets

Retrieve the list of the buckets registered

leofs-adm get-bucket <access-key-id>

Retrieve the list of the buckets owned by the specified user

leofs-adm chown-bucket <bucket> <access-key-id>

v0.16.5- Change the owner of the bucket

leofs-adm update-acl <bucket> <access-key-id> (private | public-read | public-read-write)

v0.16.0- Update the ACL (Access Control List) for the bucket

Available ACL list:

private (default) : No one except the owner has access rights

public-read : All users have READ access

public-read-write : All users have READ and WRITE access

S3-API Commands - User

create-user <user-id> <password>

Register the new user

Generate an S3 key pair (AccessKeyID and SecretAccessKey)

$ leofs-adm create-user test_account password
access-key-id: be8111173c8218aaf1c3
secret-access-key: 929b09f9b794832142c59218f2907cd1c35ac163

delete-user <user-id>

Remove the user

$ leofs-adm delete-user test
ok

get-users

Retrieve the list of users

$ leofs-adm get-users
user_id     | access_key_id          | created_at
------------+------------------------+---------------------------
_test_leofs | 05236                  | 2012-12-07 10:27:39 +0900
leo         | 39bbad4f3b837ed209fb   | 2012-12-07 10:27:39 +0900

update-user-role <user-id> <role-id>

Update the user’s role

Currently, we are supporting two kinds of roles

role-id:

1: General user

9: Administrator

$ leofs-adm update-user-role 05236 1
OK

S3-API Commands - Endpoint

add-endpoint <endpoint>

Register a new Endpoint

Note

LeoFS domains are ruled by this rule

$ leofs-adm add-endpoint leo-project.net
OK

delete-endpoint <endpoint>

Remove the endpoint

$ leofs-adm delete-endpoint leo-project.net
OK

get-endpoints

Retrieve the list of endpoints

$ leofs-adm get-endpoints
endpoint         | created at
-----------------+---------------------------
s3.amazonaws.com | 2012-09-12 14:09:52 +0900
localhost        | 2012-09-12 14:09:52 +0900
leofs.org        | 2012-09-12 14:09:52 +0900

S3-API Commands - Bucket

add-bucket <bcuket> <access-key-id>

Create the bucket

$ leofs-adm add-bucket backup 05236
OK

delete-bucket <bucket> <access-key-id>

Remove the bucket and all files stored in the bucket

$ leofs-adm delete-bucket backup 05236
OK

get-buckets

Retrieve the list of the buckets registered

$ leofs-adm get-buckets
cluster id   | bucket   | owner       | permissions                            | created at
-------------+----------+-------------+----------------------------------------+---------------------------
leofs_1      | backup   | _test_leofs | Me(full_control), Everyone(read)       | 2014-04-03 11:39:01 +0900
leofs_1      | docs     | _test_leofs | Me(full_control), Everyone(read)       | 2014-04-03 11:39:25 +0900
leofs_1      | logs     | _test_leofs | Me(full_control), Everyone(read,write) | 2014-04-03 11:39:38 +0900
leofs_1      | movie    | _test_leofs | Me(full_control)                       | 2014-04-03 11:39:45 +0900

get-bucket <access-key-id>

Retrieve the list of the buckets owned by the specified user

$ leofs-adm get-bucket 05236
bucket   | permissions                            | created at
---------+----------------------------------------+---------------------------
backup   | Me(full_control), Everyone(read)       | 2014-04-03 11:39:01 +0900
docs     | Me(full_control), Everyone(read)       | 2014-04-03 11:39:25 +0900
logs     | Me(full_control), Everyone(read,write) | 2014-04-03 11:39:38 +0900
movie    | Me(full_control)                       | 2014-04-03 11:39:45 +0900

chown-bucket <bucket> <access-key-id>

v0.16.5- Change the owner of the bucket

$ leofs-adm chown-bucket backup 47ad5ca9
OK

update-acl <bucket> <access-key-id>

v0.16.0- Update the ACL (Access Control List) for the bucket

Available ACL list:

private (default) : No one except the owner has access rights

public-read : All users have READ access

public-read-write : All users have READ and WRITE access

$ leofs-adm update-acl photo 05236 private
ok

$ leofs-adm update-acl photo 05236 public-read
ok

$ leofs-adm update-acl photo 05236 public-read-write
ok

Canned ACL

Note

When using S3-API, LeoFS supports a set of predefined grants, known as canned ACLs. Each canned ACL has a predefined a set of grantees and permissions. The following table lists the set of canned ACLs and the associated predefined grants.

Canned ACL | Applies to | Permissions added to ACL
private | Bucket and object | Owner gets FULL_CONTROL. No one else has access rights (default).
public-read | Bucket and object | Owner gets FULL_CONTROL. The AllUsers group gets READ access.
public-read-write | Bucket and object | Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. Granting this on a bucket is generally not recommended.

Reference:Access Control List (ACL) Overview [https://docs.aws.amazon.com/AmazonS3/latest/dev/ACLOverview.html]

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Administration Guide »

Administration of Multi Datacenter Replication

Shell | Description
leofs-adm join-cluster <manager-master> <manager-slave> | 1.0.0- Begin to communicate between the local cluster and the remote cluster
leofs-adm remove-cluster <manager-master> <manager-slave> | 1.0.0- Terminate to communicate between the local cluster and the remote cluster
leofs-adm cluster-status | 1.0.0- See the current state of cluster(s)

join-cluster <remote-manager-master>:<rpc-port> <remote-manager-slave>:<rpc-port>

Begin to communicate between the local cluster and the remote cluster

$ leofs-adm join-cluster manager_c2_0@10.1.2.1:13075 manager_c2_1@10.1.2.2:13075
OK

See Also: Configuration/Multi data center replication

remove-cluster <remote-manager-master>:<rpc-port> <remote-manager-slave>:<rpc-port>

Terminate to communicate between the local cluster and the remote cluster

$ leofs-adm remove-cluster manager_c2_0@10.1.2.1:13075 manager_c2_1@10.1.2.2:13075
OK

cluster-status

See the current state of cluster(s)

$ leofs-adm cluster-status
cluster id |   dc id    |    status    | # of storages  |          updated at
-----------+------------+--------------+----------------+-----------------------------
leofs_2    | dc_2       |   running    |              8 | 2014-03-21 19:17:45 +0900

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Administration Guide »

System Maintenance

Upgrade your old version LeoFS to v1.3.1

This section describes the way of replacement of old LeoFS to v1.3.1

Upgrade flow diagram

The diagram only [https://leo-project.net/leofs/docs/_images/leofs-flow-diagram.jpg]

Note

If you’re using LeoFS v1.0.0-pre1, v0.16 or v0.14, you need to take over the configuration of metadata-storage as follows because the default configuration is leveldb from v1.0.0-pre2. We provide the db-converter tool - from bitcask to leveldb which is leofs_utils/tools/b2l.

Takeover a part of confugurations

Item | Default value
leo_object_storage.metadata_storage | bitcask

Adjust Every Path

LeoFS Manager: [mnesia, log-dir and queue-dir]

.
.
.
## Mnesia dir
mnesia.dir = ./work/mnesia/{IP}
.
.
.
## Log level: [0:debug, 1:info, 2:warn, 3:error]
log.log_level = 1
## Output log file(s) - Erlang's log
log.erlang = ./log/erlang
## Output log file(s) - app
log.app = ./log/app
## Output log file(s) - members of storage-cluster
log.member_dir = ./log/ring
## Output log file(s) - ring
log.ring_dir = ./log/ring

## Directory of queue for monitoring "RING"
queue_dir = ./work/queue
## Directory of SNMP agent configuration
snmp_agent = ./snmp/snmpa_manager_0/LEO-MANAGER

LeoFS Storage: [obj_containers, log-dir and queue-dir]

## Object container
obj_containers.path = [./avs]
obj_containers.num_of_containers = [8]

## e.g. Case of plural pathes
## obj_containers.path = [/var/leofs/avs/1, /var/leofs/avs/2]
## obj_containers.num_of_containers = [32, 64]
.
.
.
## Log level: [0:debug, 1:info, 2:warn, 3:error]
log.log_level = 1
## Output log file(s) - Erlang's log
log.erlang = ./log/erlang
## Output log file(s) - app
log.app = ./log/app
## Output log file(s) - members of storage-cluster
log.member_dir = ./log/ring
## Output log file(s) - ring
log.ring_dir = ./log/ring

## Directory of queue for monitoring "RING"
queue_dir = ./work/queue
## Directory of SNMP agent configuration
snmp_agent = ./snmp/snmpa_storage_0/LEO-STORAGE

LeoFS Gateway: [SSL-related files, cache-related pathes, log-dir and queue-dir]

## SSL Certificate file
http.ssl_certfile = ./etc/server_cert.pem
## SSL key
http.ssl_keyfile  = ./etc/server_key.pem

## Directory for the disk cache data
cache.cache_disc_dir_data    = ./cache/data
## Directory for the disk cache journal
cache.cache_disc_dir_journal = ./cache/journal
.
.
.
## Log level: [0:debug, 1:info, 2:warn, 3:error]
log.log_level = 1
## Output log file(s) - Erlang's log
log.erlang = ./log/erlang
## Output log file(s) - app
log.app = ./log/app
## Output log file(s) - members of storage-cluster
log.member_dir = ./log/ring
## Output log file(s) - ring
log.ring_dir = ./log/ring

## Directory of queue for monitoring "RING"
queue_dir = ./work/queue
## Directory of SNMP agent configuration
snmp_agent = ./snmp/snmpa_gateway_0/LEO-GATEWAY

Attach/Detach node into a Storage-cluster in operation

This section describes the process of adding and removing nodes in a LeoFS Storage cluster.

Adding a storage node:

The node can be added to the cluster once it is running. You can use the rebalance command to request a join from the Manager.

Removing a storage node:

The node can be removed from the cluster when it is either running or stopped. You can use the detach command to remove the node.

After that, you need to execute the rebalance command in the Manager to actually remove the node from the storage cluster.

LeoFS Gateway Access-log Format

Since

LeoFS v1.0.0-pre3

Overview

LeoFS-Gateway is able to output access-log. If you would like to use this option, you can check and set LeoFS Gateway configuration.

Sample

--------+-------+--------------------+----------+-------+---------------------------------------+-----------------------+----------
Method  | Bucket| Path               |Child Num |  Size | Timestamp                             | Unixtime              | Response
--------+-------+--------------------+----------|-------+---------------------------------------+-----------------------+----------
[HEAD]   photo   photo/1              0          0       2013-10-18 13:28:56.148269 +0900        1381206536148320        500
[HEAD]   photo   photo/1              0          0       2013-10-18 13:28:56.465670 +0900        1381206536465735        404
[HEAD]   photo   photo/city/tokyo.png 0          0       2013-10-18 13:28:56.489234 +0900        1381206536489289        200
[GET]    photo   photo/1              0          1024    2013-10-18 13:28:56.518631 +0900        1381206536518693        500
[GET]    photo   photo/city/paris.png 0          2048    2013-10-18 13:28:56.550376 +0900        1381206536550444        404
[PUT]    logs    logs/leofs           1          5242880 2013-10-18 13:28:56.518631 +0900        1381206536518693        500
[PUT]    logs    logs/leofs           2          5242880 2013-10-18 13:28:56.518631 +0900        1381206536518693        500
[PUT]    logs    logs/leofs           3          5120    2013-10-18 13:28:56.518631 +0900        1381206536518693        500

Format

Note

The format of the access log is Tab Separated Values.

Column Number | Description
1 | Method: [HEAD|PUT|GET|DELETE]
2 | Bucket
3 | Filename (including path)
4 | Child number of a file
5 | File Size (byte)
6 | Timestamp with timezone
7 | Unixtime (including micro-second)
8 | Response (HTTP Status Code)

LeoFS Storage Data Diagnosis log Format

Since

LeoFS v1.1.5

Overview

LeoFS-Storage is able to diagnose the data. If you would like to use this option, you can check and set LeoFS Storage configuration.

See also: diagnose-start command

Sample - List of files

------+------------------------------------------+------------------------------------------------------------+-----------+------------+------------------+--------------------------+----
Offset| RING's address-id                        | Filename                                                   | Child num | File Size  | Unixtime         | Localtime                |del?
------+------------------------------------------+------------------------------------------------------------+-----------+------------+------------------+--------------------------+----
194     296754181484029444656944009564610621293   photo/leo_redundant_manager/Makefile                             0       2034        1413348050768344   2014-10-15 13:40:50 +0900   0
2400    185993533055981727582172380494809056426   photo/leo_redundant_manager/ebin/leo_redundant_manager.beam      0       24396       1413348050869454   2014-10-15 13:40:50 +0900   0
38446   53208912738248114804281793572563205919    photo/leo_rpc/.git/refs/remotes/origin/HEAD                      0       33          1413348057441546   2014-10-15 13:40:57 +0900   0
38658   57520977797167422772945547576980778561    photo/leo_rpc/ebin/leo_rpc_client_utils.beam                     0       2576        1413348057512261   2014-10-15 13:40:57 +0900   0
69506   187294034498591995039607573685274229706   photo/leo_backend_db/src/leo_backend_db_server.erl               0       13911       1413348068031188   2014-10-15 13:41:08 +0900   0
83603   316467020376888598364250682951088839795   photo/leo_backend_db/test/leo_backend_db_api_prop.erl            0       3507        1413348068052219   2014-10-15 13:41:08 +0900   1

Format - List of files

Note

The format of the data diagnosis log is Tab Separated Values.

Column Number | Description
1 | Offset of the AVS-file
2 | RING’s address id (routing-table)
3 | Filename
4 | Child number of a file
5 | File Size (byte)
6 | Timestamp - unixtime
7 | Timestamp - localtime
8 | Removed file?

Sample - Report of a data diagnosis / data compaction

{file_path,"/home/yosuke/dev/leo/test/leofs-1.1.5/package/leo_storage/avs/object/0.avs"}.
{avs_ver,<<"LeoFS AVS-2.4">>}.
{num_of_active_objs,9}.
{size_of_active_objs,52518}.
{total_num_of_objs,9}.
{total_size_of_objs,52518}.
{start_datetime,"2014-10-15 14:21:31 +0900"}.
{end_datetime,"2014-10-15 14:21:31 +0900"}.
{errors,[]}.
{duration,0}.
{result,success}.

Format - Report of a data diagnosis / data compaction

Note

The format of the data diagnosis/compaction report is Erlang Term.

Element | Description
file_path | AVS-file path
avs_ver | AVS-file version
num_of_active_obj | Number of active objects
size_of_active_obj | Summary of size of active objects
total_num_of_objs | Total number of objects
total_size_of_objs | Total size of objects
start_datetime | Data diagnosis/compaciton start datetime
end_datetime | Data diagnosis/compaciton end datetime
duration | Data diagnosis/compaciton duration time(second)
errors | Errors
result | Result of data diagnosis/compaciton: [success|failure]

Diagnosis Log Dir

Note

The data diagnosis log is assigned at LeoFS Storage’s data directory - LeoFS Storage configuration.

leo_storage_0/avs/log/
|-- [ 102 Oct 15 14:00]  leo_object_storage_0 -> /home/leofs/dev/leofs-1.1.5/package/leo_storage_0/avs/log/leo_object_storage_0.20141015.14.2
|-- [1.3K Oct 15 13:53]  leo_object_storage_0.20141015.13.1
|-- [1.3K Oct 15 13:56]  leo_object_storage_0.20141015.13.2
|-- [1.3K Oct 15 13:58]  leo_object_storage_0.20141015.13.3
|-- [1.3K Oct 15 14:00]  leo_object_storage_0.20141015.13.4
|-- [1.3K Oct 15 14:00]  leo_object_storage_0.20141015.14.1
|-- [   0 Oct 15 14:00]  leo_object_storage_0.20141015.14.2
|-- [ 376 Oct 15 13:45]  leo_object_storage_0.report.63580567518
|-- [ 374 Oct 15 13:50]  leo_object_storage_0.report.63580567830
|-- [ 362 Oct 15 13:53]  leo_object_storage_0.report.63580568032
|-- [ 362 Oct 15 13:56]  leo_object_storage_0.report.63580568197
|-- [ 362 Oct 15 13:58]  leo_object_storage_0.report.63580568300
|-- [ 362 Oct 15 14:00]  leo_object_storage_0.report.63580568416
|-- [ 362 Oct 15 14:00]  leo_object_storage_0.report.63580568445
.
.
.

Diagnosis Log Files

Log File | Description
leo_object_storage_<avs-num>.<date>.<hour>.<seq> | List of files
leo_object_storage_<avs-num>.report.<timestamp> | Report of a data diagnosis

Recover a storage-node

Since

LeoFS v1.0.0

Overview

Since disk(s) of a node collapsed, you’re able to recover the disk(s) with LeoFS recover-command.

Workflow

Suspend and Stop the storage-node

Suspend a target node with the suspend-command with leofs-adm command

Stop the node with $ path/to/leo_storage/bin/leo_storage stop

Confirm the cluster status with the status-command with leofs-adm command. You need to check the node whether state of which is suspend or not.

Change the disk(s) (rebuild the array when using RAID) and the configuration of the target node

Restart the node and Execute recover-node-command

Restart the node with $ path/to/leo_storage/bin/leo_storage start

Resume a target node with the resume-command with leofs-adm command

Execute the recover-node-command with leofs-adm command

Like this: $ leofs-adm recover-node <target-node>

Also, you’re able to check the progress of the recover with the mq-stats-command with leofs-adm command

Like this (exclude the target node):

$ leofs-adm mq-stats <storage-node-1>

...

$ leofs-adm mq-stats <storage-node-N>

See also

the suspend-command

the status-command

the resume-command

the recover-node-command

the mq-stats-command

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Administration Guide »

Administration of CDN Integration

Since

LeoFS v1.2.0

How to integrate LeoFS with CDN

There is nothing special to do for integrating LeoFS with CDN. Since almost CDN service providers take care of a CacheControl Header received from an origin to determine how long a file should be cached on their edge servers, so if you want to modify TTL according to URLs, you can do it by using http_custom_header.conf.

Samples of http_custom_header.conf

How to use http_custom_header.conf

Append the following line to your leo_gateway.conf

http.headers_config_file = ./etc/http_custom_header.conf

Arrange the http_custom_header.conf into the path specified at leo_gateway.conf

How to write http_custom_header.conf

The syntax is a subset of a Nginx configuration. You can use location contexts to specify TTL and add any headers by the path like this.

location bucket/static {
    expires    12h;
    add_header Cache-Control public;
    add_header X-OriginalHeader OriginalValue;
}

In this case, assuming that a CDN service already have been enabled and thare is a file at bucket/static/path_to_file, if a user browse that file via the CDN, the CDN will receive a response from the Origin(LeoFS) with customized http headers like this.

Cache-Control: public, max-age=43200;
X-OriginalHeader: OriginalValue;

As a result, the CDN cache that file for 12 hours and forward the response with the customized http headers to the user.

Use cases

Specify TTL by the bucket

location bucket1 {
    expires    1h;
    add_header Cache-Control public;
}
location bucket2 {
    expires    1d;
    add_header Cache-Control public;
}
location bucket3 {
    expires    1h30m;
    add_header Cache-Control private;
}

Appendix

Syntax for the expire field

LeoFS supports a part of measurement units which can be used in Nginx configuration. Following time intervals can be specified.

s: seconds

m: minutes

h: hours

d: days

List of verified CDN services

LeoFS development team tested the following CDN services with LeoFS. But other CDN services also should work. If you have some troubles, please let us know.

AWS CloudFront

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Architecture

We’re really focused on is High Availability, High Scalability and High Cost Performance Ratio because unstructured data such as image/photo, short movie, event-data, log and so on, they have been exponentially increasing day by day, so we needed to build a cloud storage with all them.

We really succeeded in designing and implementing LeoFS as simple as possible. LeoFS consists of 3 core components - LeoFS Gateway, LeoFS Storage and LeoFS Manager. The role of each component is clearly defined.

Also, what we carefully desined LeoFS is 3 things:

To keep running and No SPOF

To keep high-performance, regardless of the kind of data and amout data

To provide easy administration, we already provide LeoFS CUI and GUI console.

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Architecture »

LeoFS Gateway

LeoFS Gateway consists of the fast HTTP-Server - Cowboy, the API handler and the cache mecanism. It provides the REST-API and Amazon S3-API. You’re able to easily access LeoFS with S3-Clients such as s3cmd, DragonDisk, program languages - Erlang, Java, Ruby, Python, Go and so on.

A client requests an object or a bucket operation to LeoFS Gateway then LeoFS Gateway requests the message of operation to a storage-node.

A destination storage node is decided by the routing-table. It is called RING which is generated and provided at LeoFS Manager and which is based on consistent-hashing.

Also, LeoFS Gateway provides built-in support for the object-cache mechanism in order to realize Keeping high performance and reduction of traffic between LeoFS Gateway and LeoFS Storage.

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Architecture »

LeoFS Storage

Fundamentals

LeoFS Storage consists of object and the metadata storage. In addition, it includes replicator and repairer in order to realise Eventual consistency.

In case of a write operation, LeoFS Storage accepts a request from LeoFS Gateway then automatically replicate an object into the LeoFS Storage cluster. Finally, LeoFS Storage confirms whether a stored object satisfy the consistency rule or NOT.

On the other hand, in case of a read operation, LeoFS Gateway requests a LeoFS Storage node. Then the LeoFS Storage node retrieves an object from the local object-storage or the remote LeoFS Storage node. Finaly, the LeoFS Storage node respond an object to the gateway. Also, the LeoFS Storage node checks consistency of the object with the asynchronous processing.

If the LeoFS Storage node finds inconsistency of an object, it will be recovered with the backend process. The object eventually keep consistensy with their functions.

Data Structure

LeoFS’s object consists of 3 layers which are metadata, needle and object container.

The object storage manages and stores both an object and a metadata, which merges as a needle.

The metadata storage manages and stores attributes of an object which includes filename, size, checksum, and so on. And it depends of bitcask or leveldb.

The object container is a log structured file format.

This format is robust and high performance because effect of local file system is just a little part.

LeoFS Storage is necessary to GC - the compaction mechanism in order to remove unnecessary objects from the object container.

Large object support

LeoFS supports to handle a large size object since v0.12. The purpose of this function is 2 things:

1st one is to equalize disk usage of every LeoFS Storage node.

2nd one is to realize high I/O efficiency and high availability.

In case of a write operation, a large size object is divided to plural objects at LeoFS Gateway then they’re replicated into the LeoFS Storage cluster similarly to a small size object. And also, the default chunk size is 5 mega bytes, value of which is able to change a custom chunked object size.

On the other hand, In case of READ of a large object, first, LeoFS Gateway retrieves a metadata of a requested object from a client. Then if it is a large size object, LeoFS Gateway retrieves the chunked objects in order of the chunk object number from the LeoFS Storage cluster. Finally, LeoFS Gateway responds the objects to the client.

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Architecture »

LeoFS Manager

LeoFS Manager generates and manages a routing table, which is called RING and based on consistent hashing [https://en.wikipedia.org/wiki/Consistent_hashing].

LeoFS Manager always monitors every LeoFS Storage and LeoFS Gateway of status and RING in order to keep running LeoFS and consistency of a RING. And also, it distributes RING to LeoFS Storage and LeoFS Gateway.

In addition, LeoFS Manager provides LeoFS administration commands to be able to easily operate LeoFS. LeoFS administration commands already cover entire LeoFS functions.

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Usage

Testing without a load balancer

Example:

Endpoint: yourhost.com

bucket: test

Client(s)

You need to edit /etc/hosts in your client machine(s)

You can use any S3 client, like

Program Lang’s clients

s3cmd

DragonDisk

Servers configuration (only ip and name)

Manager

IP: 192.168.1.7, 192.168.1.8

Name: manager_0@192.168.1.7, manager_1@192.168.1.8

Gateway

IP: 192.168.1.3

Name: gateway_0@192.168.1.3

Storage

IP: 192.168.1.4 .. 192.168.1.6

Name: storage_0@192.168.1.4 .. storage_2@192.168.1.6

Commands to execute with leofs-adm

Set an endpoint

Create a user

$ leofs-adm add-endpoint <your-host-name>
OK
$ leofs-adm create-user <your-name>
access-key-id: 05dcba94333c7590a635
secret-access-key: c776574f3661579ceb91aa8788dfcac733b21b3a

Production/Staging with a load balancer

(under construction)

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

REST API

Configuration

Update LeoFS Gateway HTTP API in your LeoFS Gateway configuration

protocol = rest

Interface

Description of LeoFS’ behavior for each HTTP verb

HTTP Verb | LeoFS’ Behavior
PUT/POST | Insert an object into the storage cluster
GET | Retrieve an object from the storage cluster
DELETE | Remove an object from the storage cluster

URL format

URL format: https://${HOST}:8080/<file-path>

LeoFS will only use the <file-path> part of the URL to identify objects

You can check that an object exists in the cluster by using:

$ leofs-adm whereis <file-path>

Examples

POST/PUT

$ curl -X POST -H "Content-Type:image/png" \
          --data-binary @test.jpg https://hostname:8080/_test/_image/file.png

$ curl -X PUT -H "Content-Type:image/png" \
          --data-binary @test.jpg https://hostname:8080/_test/_image/file.png

GET

$ curl -X GET https://hostname:8080/_test/_image/file.png
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6019k    0 6019k    0     0   210M      0 --:--:-- --:--:-- --:--:--  217M

DELETE

$ curl -X DELETE https://hostname:8080/_test/_image/file.png

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Amazon S3 API and Interface

S3 API Implementation Status

Objects API

Buckets API

Amazon S3 Interface

Amazon S3 Official Documentation

How to determine the name of Bucket

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Amazon S3 API and Interface »

S3 API Implementation Status

Objects API

| Method | Status
1 | DELETE Object | Yes
2 | Delete Multiple Objects | No
3 | GET Object | Yes
4 | GET Object ACL | Plans to support with 1.4
5 | GET Object torrent | No
6 | HEAD Object | Yes
7 | POST Object | Yes
8 | PUT Object | Yes
9 | PUT Object ACL | Plans to support with 1.4
10 | PUT Object - Copy | Yes
11 | Initiate Multipart Upload | Yes
12 | Upload Part | Yes
13 | Upload Part - Copy | Yes
14 | Complete Multipart Upload | Yes
15 | Abort Multipart Upload | Yes
16 | List Parts | No

Buckets API

| Method | Status
1 | DELETE Bucket | Yes
2 | DELETE Bucket life cycle | No
3 | DELETE Bucket policy | No
4 | DELETE Bucket website | Plans to support with 1.4
5 | GET Bucket (List Objects) | Yes
6 | GET Bucket ACL | Yes
7 | GET Bucket life cycle | No
8 | GET Bucket policy | No
9 | GET Bucket location | No
10 | GET Bucket logging | No
11 | GET Bucket notification | No
12 | GET Bucket Object versions | No
13 | GET Bucket requestPayment | No
14 | GET Bucket versioning | No
15 | GET Bucket website | Plans to support with 1.4
16 | HEAD Bucket | Yes
17 | List Multipart Uploads | No
18 | PUT Bucket | Yes
19 | PUT Bucket ACL | Yes
20 | PUT Bucket life cycle | No
21 | PUT Bucket policy | No
22 | PUT Bucket logging | No
23 | PUT Bucket notification | No
24 | PUT Bucket requestPayment | No
24 | PUT Bucket versioning | No
25 | PUT Bucket website | Plans to support with 1.4

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Amazon S3 API and Interface »

Amazon S3 Interface

Amazon S3 Official Documentation

The official documentation

How to determine the name of Bucket

Virtual Hosting of Buckets

Values used for determining the name: * S3 uses the HTTP Host header or a Path segment in the HTTP request line. * How S3 determines what to use depends on the Domain name.

Patterns

1. https://s3.amazonaws.com/bucket/path_to_file

In this case, the name of the bucket is the first segment of the path.

bucket

Name | Value
Request line | GET /bucket/path_to_file HTTP/1.1
Host header | Host: s3.amazonaws.com

The argument of LeoFS’ whereis | purge-cache commands should be bucket/path_to_file.

2. https://www.example.com.s3.amazonaws.com/path_to_file

In this case, the name of the bucket is the part of the domain name before .s3.amazonaws.com.

www.example.com

Name | Value
Request line | GET /path_to_file HTTP/1.1
Host header | Host: www.example.com.s3.amazonaws.com

The argument of LeoFS’ whereis | purge commands should be www.example.com/path_to_file.

3. https://www.example.com/path_to_file

In this case, the name of bucket is equal to the FQDN.

www.example.com

Name | Value
Request line | GET /path_to_file HTTP/1.1
Host header | Host: www.example.com

The argument of LeoFS’ whereis | purge commands should be www.example.com/path_to_file.

4. Rules for Bucket Naming (v0.16.1-)

Note

We recommend as a best practice that you always use DNS-compliant bucket names regardless of the region in which you create the bucket.

Bucket names can be as long as between 3 and 255 characters.

Bucket names can contain lowercase letters, numbers, periods (.), dashes (-) and underscores (_).

Bucket names must not be formatted as an IP address (e.g., 192.168.5.4).

Bucket name cannot start and end with periods (.), dashes (-) and underscores (_).

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Amazon S3 Client Tutorials

Getting Your S3 Key

Getting Started with Ruby

Install AWS-SDK for Ruby

Example usage

Connecting to LeoFS

PUT an object into LeoFS

GET an object from LeoFS

DELETE an object from LeoFS

Get an object HEAD from LeoFS

Uploading to LeoFS using multipart

Getting Started with Java

Getting AWS SDK for Java

Example usage

Getting Started with PHP

Install AWS-SDK version 2 for PHP

php5-curl (Debian)

PEAR (Debian)

aws-sdk for PHP

Edit /etc/hosts

Example usage

Getting Started with Erlang

Getting “erlcloud”

Edit /etc/hosts

Example usage

Getting Started with Python

Install “boto”

setup.py

easy_install

Example usage

Getting Started with Node.js

Install “Knox”

Edit /etc/hosts

Example usage

Getting Started with Go

GoAMZ

Install GoAMZ

Example usage

PUT an object into LeoFS

GET an object from LeoFS

DELETE an object from LeoFS

Getting Started with S3FS-C

Install libs for S3FS-C into Ubuntu-12.04

Install “S3FS-C”

Modify “/etc/hosts”

Create a credential file for S3FS

Mount “LeoFS”

Connect to LeoFS using s3cmd

Getting s3cmd

Configuration

Commands

Limits

Connect to LeoFS using DragonDisk

Setting up LeoFS account details

Create a bucket

Manipulating files

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Amazon S3 Client Tutorials »

Getting Your S3 Key

You can get S3 API Keys from leofs-adm. The create-user command generates your s3 key.

$ leofs-adm create-user <user-id>
access-key-id: 05dcba94333c7590a635
secret-access-key: c776574f3661579ceb91aa8788dfcac733b21b3a

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Amazon S3 Client Tutorials »

Getting Started with Ruby

aws-sdk’s official documentation is here. [https://aws.amazon.com/sdkforruby/]

Install AWS-SDK for Ruby

$ gem install aws-sdk

Example usage

Connecting to LeoFS

Note

LeoFS’ domains are ruled by this rule. You need to configure the Endpoint, Port, AccessKeyId and SecretAccessKey like in this example:

## This code supports "aws-sdk v1.9.5"
require "aws-sdk"

Endpoint = "localhost"
Port = 8080
# set your s3 key
AccessKeyId = "YOUR_ACCESS_KEY_ID"
SecretAccessKey = "YOUR_SECRET_ACCESS_KEY"

class LeoFSHandler < AWS::Core::Http::NetHttpHandler
  def handle(request, response)
    request.port = ::Port
    super
  end
end

SP = AWS::Core::CredentialProviders::StaticProvider.new(
{
    :access_key_id     => AccessKeyId,
    :secret_access_key => SecretAccessKey
})

AWS.config(
  access_key_id: AccessKeyId,
  secret_access_key: SecretAccessKey,
  s3_endpoint: Endpoint,
  http_handler: LeoFSHandler.new,
  credential_provider: SP,
  s3_force_path_style: true,
  use_ssl: false
)

s3 = AWS::S3.new

PUT an object into LeoFS

# create bucket
s3.buckets.create("photo")

# get bucket
bucket = s3.buckets["photo"]

# create a new object
object = bucket.objects.create("image", "value")

# show objects in the bucket
bucket.objects.with_prefix("").each do |obj|
  p obj
end

# retrieve an object
object = bucket.objects["image"]

# insert an object
object.write(
  file: "/path/to/image.png",
  content_type: "png/image"
)

GET an object from LeoFS

image = object.read

DELETE an object from LeoFS

object.delete

Get an object HEAD from LeoFS

metadata = object.head
p metadata.to_hash

Uploading to LeoFS using multipart

## This code supports "aws-sdk v1.9.5"
require 'aws-sdk'

Endpoint = "leofs.org"
Port = 8080

## set your s3 key
AccessKeyId = "YOUR_ACCESS_KEY_ID"
SecretAccessKey = "YOUR_SECRET_ACCESS_KEY"

class LeoFSHandler < AWS::Core::Http::NetHttpHandler
  def handle(request, response)
    request.port = ::Port
    super
  end
end

SP = AWS::Core::CredentialProviders::StaticProvider.new(
{
    :access_key_id     => AccessKeyId,
    :secret_access_key => SecretAccessKey
})

AWS.config(
  :access_key_id => 'access-key-id',
  :secret_access_key => 'secret-access-key',
  s3_endpoint: Endpoint,
  http_handler: LeoFSHandler.new,
  credential_provider: SP,
  s3_force_path_style: true,
  use_ssl: false
)

file_path_for_multipart_upload = '/path/to/file'
bucket = AWS::S3.new.buckets['bucket-name']

open(file_path_for_multipart_upload) do |file|
  uploading_object = bucket.objects[File.basename(file.path)]
  uploading_object.multipart_upload do |upload|
    while !file.eof?
      upload.add_part(file.read 5242880) ## 5MB ##
      p('Aborted') if upload.aborted?
    end
  end
end

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Amazon S3 Client Tutorials »

Getting Started with Java

Getting AWS SDK for Java

URL: SDK for Java [https://aws.amazon.com/sdkforjava/]

Note

LeoFS domains are ruled by this rule. You need to set ‘Proxy Host’ and ‘Proxy Port’ with ClientConfiguration class.

Example usage

import java.io.BufferedReader;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.util.UUID;

import com.amazonaws.AmazonClientException;
import com.amazonaws.AmazonServiceException;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.ListObjectsRequest;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.Bucket;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;

public class LeoFSSample {
    public static void main(String[] args) throws IOException {
        /* ---------------------------------------------------------
         * You need to set 'Proxy host', 'Proxy port' and 'Protocol'
         * --------------------------------------------------------- */
        ClientConfiguration config = new ClientConfiguration();
        config.setProxyHost("localhost"); // LeoFS Gateway's Host
        config.setProxyPort(8080);        // LeoFS Gateway's Port
        config.withProtocol(Protocol.HTTP);

        final String accessKeyId = "YOUR_ACCESS_KEY_ID";
        final String secretAccessKey = "YOUR_SECRET_ACCESS_KEY";

        AWSCredentials credentials = new BasicAWSCredentials(accessKeyId, secretAccessKey);
        AmazonS3 s3 = new AmazonS3Client(credentials, config);

        final String bucketName = "test-bucket-" + UUID.randomUUID();
        final String key = "test-key";

        try {
            // Create a bucket
            s3.createBucket(bucketName);

            // Retrieve list of buckets
            for (Bucket bucket : s3.listBuckets()) {
                System.out.println("Bucket:" + bucket.getName());
            }

            // PUT an object into the LeoFS
            s3.putObject(new PutObjectRequest(bucketName, key, createFile()));

            // GET an object from the LeoFS
            S3Object object = s3.getObject(new GetObjectRequest(bucketName, key));
            dumpInputStream(object.getObjectContent());

            // Retrieve list of objects from the LeoFS
            ObjectListing objectListing =
                s3.listObjects(new ListObjectsRequest().withBucketName(bucketName));

            for (S3ObjectSummary objectSummary : objectListing.getObjectSummaries()) {
                System.out.println(objectSummary.getKey() +
                                   "Size:" + objectSummary.getSize());
            }

            // DELETE an object from the LeoFS
            s3.deleteObject(bucketName, key);

            // DELETE a bucket from the LeoFS
            s3.deleteBucket(bucketName);

        } catch (AmazonServiceException ase) {
            System.out.println(ase.getMessage());
            System.out.println(ase.getStatusCode());
        } catch (AmazonClientException ace) {
            System.out.println(ace.getMessage());
        }
    }

    private static File createFile() throws IOException {
        File file = File.createTempFile("leofs_test", ".txt");
        file.deleteOnExit();

        Writer writer = new OutputStreamWriter(new FileOutputStream(file));
        writer.write("Hello, world!\n");
        writer.close();

        return file;
    }

    private static void dumpInputStream(InputStream input) throws IOException {
        BufferedReader reader = new BufferedReader(new InputStreamReader(input));
        while (true) {
            String line = reader.readLine();
            if (line == null) break;
            System.out.println(line);
        }
    }
}

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Amazon S3 Client Tutorials »

Getting Started with PHP

Install AWS-SDK version 2 for PHP

php5-curl (Debian)

sudo apt-get install php5-curl

PEAR (Debian)

sudo apt-get install php-pear

aws-sdk for PHP

sudo pear channel-discover pear.amazonwebservices.com
sudo pear install aws/sdk

Edit /etc/hosts

Note

LeoFS domains are ruled by this rule.

127.0.0.1 s3.amazonaws.com
127.0.0.1 ${bucket_name}.s3.amazonaws.com # if you use create_bucket

Example usage

<?php
require "vendor/autoload.php";

use Aws\Common\Enum\Region;
use Aws\S3\S3Client;

$client = S3Client::factory(array(
  "key" => "YOUR ACCESS KEY ID",
  "secret" => "YOUR SECRET ACCESS KEY",
  "region" => Region::US_EAST_1,
  "scheme" => "http",
));

// list buckets
$buckets = $client->listBuckets()->toArray();

foreach($buckets as $bucket){
  print_r($bucket);
}
print("\n\n");

// create bucket
$result = $client->createBucket(array(
  "Bucket" => "test"
));

// PUT object
$client->putObject(array(
  "Bucket" => "test",
  "Key" => "key-test",
  "Body" => "Hello, world!"
));

// GET object
$object = $client->getObject(array(
  "Bucket" => "test",
  "Key" => "key-test"
));
print($object->get("Body"));
print("\n\n");

// HEAD object
$headers = $client->headObject(array(
  "Bucket" => "test",
  "Key" => "key-test"
));
print_r($headers->toArray());

// DELETE object
$client->deleteObject(array(
  "Bucket" => "test",
  "Key" => "key-test"
));
?>

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Amazon S3 Client Tutorials »

Getting Started with Erlang

Getting “erlcloud”

erlcloud is a Erlang interface to Amazon Web Services. You can use it for LeoFS too.

Repository [https://github.com/gleber/erlcloud]

Edit /etc/hosts

Note

LeoFS domains are ruled by this rule.

127.0.0.1 s3.amazonaws.com
127.0.0.1 ${bucket_name}.s3.amazonaws.com # if you use create_bucket

Example usage

%% Launch "erlcloud" and configuration
erlcloud:start(),
Conf_1 = erlcloud_s3:new("?YOUR_ACCESS_KEY_ID",
                       "?YOUR_SECRET_ACCESS_KEY",
                       "localhost",
                       8080),
Conf_2 = Conf_1#aws_config{s3_scheme = "https://"},

Bucket = "erlang",
Key = "test-key",
Val = "value",

try
    %% Create an bucket
    erlcloud_s3:create_bucket(Bucket, Conf_2),

    %% Retrieve list of buckets
    List = erlcloud_s3:list_buckets(Conf_2),
    io:format("[debug]buckets:~p~n", [List]),

    %% Put an object
    erlcloud_s3:put_object(Bucket, Key, Val, [], Conf_2),

    %% Retrieve list of objects
    Objs = erlcloud_s3:list_objects(Bucket, Conf_2),
    io:format("[debug]objects:~p~n", [Objs]),

    %% Retrieve an object
    Obj = erlcloud_s3:get_object(Bucket, Key, Conf_2),
    io:format("[debug]inserted object:~p~n", [Obj]),

    %% Retrieve metadata of an object
    Meta = erlcloud_s3:get_object_metadata(Bucket, Key, Conf_2),
    io:format("[debug]metadata:~p~n", [Meta]),

    %% Remove an object
    DeletedObj = erlcloud_s3:delete_object(Bucket, Key, Conf_2),
    io:format("[debug]deleted object:~p~n", [DeletedObj]),

    try
        %% Retrieve an object, again
        NotFoundObj = erlcloud_s3:get_object(Bucket, Key, Conf_2),
        io:format("[debug]not found object:~p~n", [NotFoundObj])
    catch
        error:{aws_error,{http_error,404,_,_}} ->
            io:format("[debug]404 not found object~n")
    end
after
    ok = erlcloud_s3:delete_bucket(Bucket, Conf_2)
end.

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Amazon S3 Client Tutorials »

Getting Started with Python

Boto is a Python interface to Amazon Web Services. You can use it for LeoFS too.

Repository [https://github.com/boto/boto]

Documentation [https://docs.pythonboto.org/en/latest/index.html]

Install “boto”

setup.py

git clone https://github.com/boto/boto.git; cd boto; sudo python setup.py install

easy_install

sudo easy_install boto

Example usage

Note

LeoFS domains are ruled by this rule.

#!/usr/bin/python
# coding: utf8

from boto.s3.connection import S3Connection, OrdinaryCallingFormat
from boto.s3.bucket import Bucket
from boto.s3.key import Key

AWS_ACCESS_KEY = "YOUR_ACCESS_KEY_ID"
AWS_SECRET_ACCESS_KEY = "YOUR_SECRET_ACCESS_KEY"

conn = S3Connection(AWS_ACCESS_KEY,
                    AWS_SECRET_ACCESS_KEY,
                    host = "example.com",
                    port = 8080,
                    calling_format = OrdinaryCallingFormat(),
                    is_secure = False
       )

# create bucket
bucket = conn.create_bucket("leofs-bucket")

# create object
s3_object = bucket.new_key("image_file")

# write
s3_object.set_contents_from_string("This is a text.")

# show buckets
for bucket in conn.get_all_buckets():
  print bucket

  # show S3Objects
  for obj in bucket.get_all_keys():
    print obj

  print

# get bucket
bucket = conn.get_bucket("leofs-bucket")
print bucket

# get S3Object
s3_object = bucket.get_key("image_file")
print s3_object

# read
print s3_object.read()

# write from file
#s3_object.set_contents_from_filename("filename")

# delete S3Object
s3_object.delete()

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Amazon S3 Client Tutorials »

Getting Started with Node.js

Install “Knox”

npm install knox

Edit /etc/hosts

Note

LeoFS domains are ruled by this rule.

127.0.0.1 ${bucket_name}.localhost

Example usage

Var knox = require("knox")

var client = knox.createClient({
  key: "YOUR ACCESS KEY ID",
  secret: "YOUR SECRET ACCESS KEY",
  bucket: "bucket",
  endpoint: "bucket.localhost", // ${bucket_name}.localhost
  port: 8080
});

// PUT object
var string = "Hello, world!";
client.put("key", {
  "Content-Length": string.length,
  "Content-Type": "application/json"
}).end(string);

// HEAD object
client.headFile("key", function(err, res) {
  console.log("Headers:\n", res.headers);
});

// GET object
client.getFile("key", function(err, res) {
  res.on('data', function(chunk){
    console.log(chunk.toString());
  });
});

// DELETE object
client.deleteFile("key", function(err, res) {
  console.log(res.statusCode);
});

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Amazon S3 Client Tutorials »

Getting Started with Go

GoAMZ

GoAMZ is one of Go packages to interact with the Amazon Web Service [https://aws.amazon.com/] .

Install GoAMZ

$ go get github.com/crowdmob/goamz/aws
$ go get github.com/crowdmob/goamz/s3

Example usage

PUT an object into LeoFS

This example tries to connect LeoFS and put an object.

package main

import (
    "fmt"

    "github.com/crowdmob/goamz/aws"
    "github.com/crowdmob/goamz/s3"
)

func main() {

    auth := aws.Auth{
        AccessKey: "YOUR_ACCESS_KEY",
        SecretKey: "YOUR_SECRET_KEY",
    }

    region := aws.Region{
        S3Endpoint: fmt.Sprintf("%s:%s", "YOUR_END_POINT_HOST", "YOUR_END_POINT_PORT"),
    }

    client := s3.New(auth, region)
    bucket := client.Bucket("YOUR_BUCKET_NAME")

    err := bucket.Put("YOUR_PATH", []byte("YOUR_CONTENTS"), "text/plain", "", s3.Options{})
    if err != nil {
        panic(err)
    }
}

Note

LeoFS’ domains are ruled by this rule. You need to configure the Endpoint, Port, AccessKeyId and SecretAccessKey

GET an object from LeoFS

This example tries to connect LeoFS and get an object.

package main

import (
    "fmt"
    "io/ioutil"

    "github.com/crowdmob/goamz/aws"
    "github.com/crowdmob/goamz/s3"
)

func main() {

    auth := aws.Auth{
        AccessKey: "YOUR_ACCESS_KEY",
        SecretKey: "YOUR_SECRET_KEY",
    }

    region := aws.Region{
        S3Endpoint: fmt.Sprintf("%s:%s", "YOUR_END_POINT_HOST", "YOUR_END_POINT_PORT"),
    }

    client := s3.New(auth, region)
    bucket := client.Bucket("YOUR_BUCKET_NAME")

    r, err := bucket.GetReader("YOUR_PATH")
    if err != nil {
        panic(err)
    }

    content, _ := ioutil.ReadAll(r)
    r.Close()
}

DELETE an object from LeoFS

This example tries to connect LeoFS and delete an object.

package main

import (
    "fmt"

    "github.com/crowdmob/goamz/aws"
    "github.com/crowdmob/goamz/s3"
)

func main() {

    auth := aws.Auth{
        AccessKey: "YOUR_ACCESS_KEY",
        SecretKey: "YOUR_SECRET_KEY",
    }

    region := aws.Region{
        S3Endpoint: fmt.Sprintf("%s:%s", "YOUR_END_POINT_HOST", "YOUR_END_POINT_PORT"),
    }

    client := s3.New(auth, region)
    bucket := client.Bucket("YOUR_BUCKET_NAME")

    err := bucket.Del("YOUR_PATH")
    if err != nil {
        panic(err)
    }
}

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Amazon S3 Client Tutorials »

Getting Started with S3FS-C

S3FS-C is a FUSE (File System in User Space) based file system backed by Amazon S3 storage buckets. Once mounted, S3 can be used just like it was a local file system.

Install libs for S3FS-C into Ubuntu-12.04

$ sudo apt-get install libfuse-dev libcurl4-openssl-dev fuse-utils

Install “S3FS-C”

$ git clone https://github.com/leo-project/s3fs-c.git
$ cd s3fs-c
$ ./configure
$ make
$ sudo make install

Modify “/etc/hosts”

Add a LeoFS domain in /etc/hosts

LeoFS’ domains are ruled by this rule

$ sudo vi /etc/hosts

## Add a LeoFS domain ##
127.0.0.1 localhost <bucket>.localhost

Create a credential file for S3FS

$ vi ~/.passwd-s3fs

## Set access-key and secret-key ##
<access-key-id>:<secret-key>

$ chmod 600 ~/.passwd-s3fs

Mount “LeoFS”

$ s3fs <bucket> <mount-point> -o url='https://<endpoint>:<port>'

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Amazon S3 Client Tutorials »

Connect to LeoFS using s3cmd

Getting s3cmd [https://sourceforge.net/projects/s3tools/files/]

Configuration

Note

LeoFS’ domains are ruled by this rule. You need to set ‘Endpoint’ and ‘Port’.

$ s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3
Access Key: ${ACCESS_KEY}
Secret Key: ${SECRET_ACCESS_KEY}

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: localhost
HTTP Proxy server port [3128]: 8080

New settings:
  Access Key: ${ACCESS_KEY}
  Secret Key: ${SECRET_ACCESS_KEY}
  Encryption password:
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name: ${ENDPOINT}
  HTTP Proxy server port: ${PORT}

Test access with supplied credentials? [Y/n]

Note

LeoFS’s Gateway respond a large size object with “chunked transfer encording - NOT put Content-Length on the HTTP header”. Unfortunately, “s3cmd” does not support that, yet.

Commands

Command
Support Status

1
Make bucket
s3cmd mb s3://BUCKET
Yes

2
Remove bucket
s3cmd rb s3://BUCKET
Yes

3
List objects or buckets
s3cmd ls [s3://BUCKET[/PREFIX]]
Yes

4
List all object in all buckets
s3cmd la
No

5
Put file into bucket
s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX]
Yes

6
Get file from bucket
s3cmd get s3://BUCKET/OBJECT LOCAL_FILE
Yes

7
Delete file from bucket
s3cmd del s3://BUCKET/OBJECT
Yes

8
Synchronize a directory tree to S3

s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX]

s3://BUCKET[/PREFIX] LOCAL_DIR

Yes

9
Disk usage by buckets
s3cmd du [s3://BUCKET[/PREFIX]]
No

10
Get various info about buckets or files
s3cmd info s3://BUCKET[/OBJECT]
No

11
Copy object
s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
Yes

12
Move object
s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
Yes

Limits

LeoFS only support Amazon S3 authentication v2. If you use s3cmd v1.5.x, you need to add one line on ~/.s3cfg as below.

signature_v2 = True

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Amazon S3 Client Tutorials »

Connect to LeoFS using DragonDisk

Note

LeoFS’ domains are ruled by this rule.

DragonDisk [https://www.dragondisk.com/] is a powerful file manager for Amazon S3 Compatible Storage.

Setting up LeoFS account details

To setup your LeoFS’ account, go to the menu File/Accounts.

If the details are valid, you can see that S3 has been added to the Root list.

Create a bucket

You need to create the bucket where the objects will be stored.

Buckets can only be manipulated using a unique, developer-assigned key.

Manipulating files

You can upload files into LeoFS, download files from LeoFS and do any other operations on them.

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

Benchmarking

Setting up basho_bench

Installation

Basho’s basho_bench’s repository [https://github.com/basho/basho_bench/]

Basho’s basho_bench’s documentation [https://docs.basho.com/riak/latest/cookbooks/Benchmarking]

Use the following commands to set up basho_bench.

$ git clone git://github.com/basho/basho_bench.git
$ git clone https://github.com/leo-project/leofs.git
$ cd basho_bench
$ cp -i ../leofs/test/src/*.erl src/
$ cp -i ../leofs/test/include/*.hrl include/
$ make all

Preparations before testing

Create a test bucket

Note

After starting LeoFS, you need to create a bucket on “manager-console” that will be used for testing. In this example, the bucket name is test. It is owned by the user _test_leofs that is already registered internally by LeoFS.

$ leofs-adm add-endpoint <gateway-ip-address>
OK

$ leofs-adm add-bucket test 05236
OK

$ leofs-adm get-buckets
bucket   | owner       | created at
---------+-------------+---------------------------
test     | _test_leofs | 2013-02-27 14:06:54 +0900

$ leofs-adm update-acl test 05236 public-read
OK

Configuration file for basho_bench

Examples

Some examples are included in the LeoFS repository at ${LEOFS_ROOT}/test/conf/leofs_*.config

Learn more about basho_bench’s configuration [https://docs.basho.com/riak/latest/cookbooks/Benchmarking/#Configuration]

{mode,      max}.
{duration,   10}.
{concurrent, 50}.

{driver, basho_bench_driver_leofs}.
{code_paths, ["deps/ibrowse"]}.

{http_raw_ips, ["${HOST_NAME_OF_LEOFS_GATEWAY}"]}.
{http_raw_port, 8080}.
{http_raw_path, "/test"}.
%% {http_raw_path, "/${BUCKET}"}.

{key_generator,   {partitioned_sequential_int, 1000000}}.
{value_generator, {fixed_bin, 16384}}. %% 16KB
{operations, [{put,1}]}.               %% PUT:100%
%%{operations, [{put,1}, {get, 4}]}.   %% PUT:20%, GET:80%

{check_integrity, false}.

Description

Key
Value

http_raw_ips
The LeoFS Gateway nodes we want to benchmark

http_raw_port
The port used by the LeoFS Gateway nodes

http_raw_path
URL path prefix. The first path segment MUST be a BUCKET name

check_integrity (default:false)

Check integrity of registered object - compare an original MD5 with a retrieved object’s MD5

(Only for developers)

Running basho_bench(1)

Note

In this example, LeoFS and basho_bench are installed locally.

The following commands can be used to run basho_bench.

### Loading 1M records of size 16KB
cd basho_bench
./basho_bench ../leofs/test/conf/leofs_16K_LOAD1M.config

Running basho_bench(2)

Note

In this example, LeoFS and basho_bench are installed on different hosts.

Configure the endpoint on LeoFS-Manager console

Allows basho_bench’s requests to reach ${HOST_NAME_OF_LEOFS_GATEWAY}.

$ leofs-adm add-endpoint <host-name-of-leofs-gateway>
OK

$ leofs-adm get-endpoints
endpoint                      | created at
------------------------------+---------------------------
localhost                     | 2013-03-01 00:14:04 +0000
s3.amazonaws.com              | 2013-03-01 00:14:04 +0000
${HOST_NAME_OF_LEOFS_GATEWAY} | 2013-03-01 00:14:04 +0000

Edit the benchmark’s configuration file

You need to modify the values for http_raw_ips and http_raw_port

{mode,      max}.
{duration,   10}.
{concurrent, 50}.

{driver, basho_bench_driver_leofs}.
{code_paths, ["deps/ibrowse"]}.

{http_raw_ips, ["${HOST_NAME_OF_LEOFS_GATEWAY}"]}. %% able to set plural nodes
{http_raw_port, ${PORT}}. %% default: 8080
{http_raw_path, "/test"}.
%% {http_raw_path, "/${BUCKET}"}.

{key_generator,   {partitioned_sequential_int, 1000000}}.
{value_generator, {fixed_bin, 16384}}. %% 16KB
{operations, [{put,1}]}.               %% PUT:100%
%%{operations, [{put,1}, {get, 4}]}.   %% PUT:20%, GET:80%

{check_integrity, false}.

Running basho_bench

### Loading 1M records each size is 16KB
cd basho_bench
./basho_bench ../leofs/test/conf/leofs_16K_LOAD1M.config

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

LeoCenter v0.4.7

LeoCenter is LeoFS’ Web console in your browser. You can use it to easily operate LeoFS.

Installation and configuration

Requirements

LeoFS v0.16.8 or higher

Install LeoCenter

LeoCenter’s Repository: https://github.com/leo-project/leo_center

Ruby 2.0 OR higher

Download URL: https://www.ruby-lang.org/en/downloads/

gem install bundler
git clone https://github.com/leo-project/leo_center.git
cd leo_center
bundle install

Create Admin User

Note

You need to create an administrator user from LeoFS-Manager’s console.

$ leofs-adm create-user leo_admin password
access-key-id: ab96d56258e0e9d3621a
secret-access-key: 5c3d9c188d3e4c4372e414dbd325da86ecaa8068

$ leofs-adm update-user-role leo_admin 9
OK

Configure

Note

You need to modify config.yml to allow LeoCenter to connect to LeoFS-Manager.

:managers:
  - "localhost:10020" # leofs manager - master node's host/port
  - "localhost:10021" # leofs manager - slave node's host/port
:credential:
  :access_key_id: ${YOUR_ACCESS_KEY_ID}
  :secret_access_key: ${YOUR_SECRET_ACCESS_KEY}
:session:
  :local:
    :secret: ""
    :expire_after: 300
:snmp: # list of data in Node Status SNMP Chart
  - ETS memory usage (1-min Averages)
  - Processes memory usage (1-min Averages)
  - System memory usage (1-min Averages)

Starting LeoCenter

Using Thin Web Server

Launch LeoCenter

$ thin start -a ${HOST} -p ${PORT}

Launch LeoCenter with SSL

## Caution: You need to prepare key and certification first
$ thin start --ssl --ssl-key-file "${path/to/key}/leofs.key" \
             --ssl-cert-file "${path/to/cert}/leofs.crt" \
             --ssl-verify \
             -a ${HOST}

Note

You are able to add setting in detail by these steps:

1. Rename thin.yml.sample to thin.yml

2. Edit thin.yml with check explanation of using “thin - -help”

3. Launch LeoCenter by “thin start -C thin.yml”

Features

Your Credentials

You can confirm your credentials by using the Security Credentials menu on the top right of the screen.

Bucket Status View

You can get an overview of the buckets that belong to you.

You can create new buckets.

Deletion of a bucket is planned to be supported from LeoCenter v0.4.2.

Node Status View

You can get an overview of the nodes in the cluster, and group them by type or status.

Group by type View:

Group by status View:

Administrators can use the Change Status button to suspend, resume or detach storage nodes.

Table - Changeable Status

Current status | Action available
running running | suspend, detach
suspend suspend | resume
restarted restarted | resume
stop stop | detach

Re balancing the storage cluster

Note

The Rebalance button only becomes active when the storage status is attached or detached.

Administration Tools

System Conf View

Overview of the configuration of LeoFS

Please see LeoFS’ system-configuration

Users View

Lists the registered users

You can create and delete users

You can change an user’s role using the Update Role button

Buckets View

Lists the registered buckets, per owner

You can create new buckets

Endpoints View

List of registered endpoints

You can create and delete endpoints

Milestones

0.2 (Dec 2012 - Feb 2013) - DONE

Administration tools

User management

Bucket management

Endpoint management

Node Status

Status/Operation

Bucket status

Belonging bucket-list

0.4 (Mar - Aug 2014)

User Group

Sharing LeoFS’ credential-keys in the group

User management in the group

0.6 (Sept 2014 - )

Link LeoQoS (LeoFS’s Quality of Service)

Bucket status

total of files

total used disk capacity

0.8

Log Search/Analysis (Option)

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

LeoFS Frequently Asked Questions

FAQ: LeoFS Fundamentals

What kind of storage is leofs?

What are typical uses for LeoFS?

What is benefit for LeoFS users?

Advantage

For business managers

For administrators

What is architecture of LeoFS?

Is there the roadmap of LeoFS?

What language is LeoFS written in?

What language can I use to work with LeoFS?

FAQ: LeoFS Limits

Features

Operations

NFS Support

FAQ: LeoFS Administration

Where can I get the packages for LeoFS?

How can I run my LeoFS cluster automatically?

How do multiple users login into LeoFS Manager’s console at the same time?

Answer 1:

Answer 2:

The result of the du command can be different with the actual disk-usage

When issueing the recover node command the LeoFS can get into high load

What should I do when Too many processes errors happen?

Why does starting a leo_storage using bitcask as metadata take too much time?

How do I set “a number of containers” at LeoFS Storage configuration?

In conclusiton:

leo_storage can not start due to “enif_send_failed on non smp vm”

Why is the speed of rebalance/recover command too slow?

How to run LeoFS on docker container?

Why does LeoFS’s SNMP I/F give me wrong values(0) instead of correct values?

When adding a new storage node, that node doesn’t appear with leofs-adm status Why?

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

LeoFS Frequently Asked Questions »

FAQ: LeoFS Fundamentals

What kind of storage is leofs?

LeoFS is a highly scalable, fault-tolerant object storage for the Web. Significantly, LeoFS supports huge amount and various kind unstructured data such as photo, movie, document, log data and so on.

Operationally, LeoFS features multi-master replication with automated failover and built-in horizontal scaling via Consistent hashing.

See Also:

LeoFS Overview

Slide - Scaling and High Performance Storage System: LeoFS

What are typical uses for LeoFS?

If you are searching a storage system that is able to store huge amount and various kind of files such as photo, movie, log data and so on, LeoFS is suitable for that.

This is because LeoFS is a highly available, distributed storage system. Also, LeoFS can be used to store a lot of data efficiently, safely, and inexpensively.

See Also:

LeoFS Overview

What is benefit for LeoFS users?

Advantage

LeoFS is supporting the following features:

S3-API Support

LeoFS is an Amazon S3 compatible storage system.

Switch to LeoFS to decrease your cost from more expensive public-cloud solutions.

Large Object Support

LeoFS can handle files with more than GB

Multi Data Center Replication

LeoFS is a highly scalable, fault-tolerant distributed file system without SPOF - Single Point Of Failure.

LeoFS’s cluster can be viewed as ONE-HUGE storage. It consists of a set of loosely connected nodes.

We can build a global scale storage system with easy operations

High Performance without SPOF - Single Point Of Failure

According to the original cache mechanism and sophisticated innternal architecture, LeoFS keeps high performance regardless of amount and kind of data without SPOF - Single Point Of Failure.

In near future, LeoFS is going to provide the powerful features with LeoFS v1.2, v1.4 and v2.0.

NFS support

LeoFS insight

OpenStack integration

For business managers

Storing confidential/sensitive data internally

Saving cost to use commodity servers

Increasing service level with speedy response

Expanding business globally

For administrators

Easy to install with the packages

Easy to operate with LeoCenter on GitHub

What is architecture of LeoFS?

We’ve been mainly focusing on High Availability, High Scalability and High Cost Performance Ratio since unstructured data such as images, movies and logs have been exponentially increasing day by day, and we needed to build a cloud storage that can handle all them.

LeoFS consists of 3 core components - LeoFS Gateway, LeoFS Storage and LeoFS Manager. The role of each component is clearly defined.

LeoFS Gateway handles http-requests and http-responses from clients when using REST-API OR S3-API. Also, it has the built-in object-cache system.

LeoFS Storage handles GET, PUT and DELETE, Also it has replicator and recoverer in order to keep running and consistency.

LeoFS Manager always monitors Gateway(s) and Storage(s). Manger monitors node-status and RING(logical routing-table) checksum to keep running and consistency.

Also, what we payed attention when we desined LeoFS are the following 3 things:

To keep always running and No SPOF - Single Point Of Failure

To keep high-performance, regardless of the kind and amount of data

To provide easy administration, we already provide LeoFS CUI and GUI console.

Is there the roadmap of LeoFS?

We’ve published LeoFS milestones on both of LeoFS on GitHub and LeoFS Manager. We may revise the milestones occasionally because there is a possibility to add new features or change priority of implementation. We’ll keep them always updated.

What language is LeoFS written in?

LeoFS is implemented in Erlang. Also, LeoCenter on GitHub as Web GUI Console is written in Ruby and JavaScript.

See Also:

LeoFS Overview

LeoFS on GitHub

LeoCenter on GitHub

What language can I use to work with LeoFS?

LeoFS clients exist for all of the most popular programming languages as S3-API client. See the latest list of clients for details.

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

LeoFS Frequently Asked Questions »

FAQ: LeoFS Limits

Features

LeoFS have covered almost major Amazon S3 REST API but not all.

If you use Amazon S3 multipart upload API, the size of a part of an object must be less than the size of a chunked object in LeoFS.

When using the Multi DC feature, we have supported up to 2 clusters with LeoFS v1.2.8 or lowers, but we will support over 3 clusters with LeoFS v1.4.

See Also:

Amazon S3 API and Interface

Configuration of Gateway nodes

Respond an incorrect MD5 of an large object

Multipart uploads of large files produces partially corrupted data when upload chunk size

Support over 3 clusters

Operations

When you upgrade your LeoFS, you can NOT change the metadata storage as KVS - bitcask or leveldb can be used in LeoFS - used by LeoFS Storage. We recommend users to replace bitcask with leveldb by using leofs_utils/tools/b2l.

See Also:

Configuration of LeoFS Storage

Upgrade your old version LeoFS to v1.2.7

leofs_utils/tools/b2l

NFS Support

NFS implemantation with LeoFS v1.1 is a subset of NFS v3. Lock manager protocol, Authentication, and Owner/Permission management are NOT covered.

The ls command may take too much time when the target directory have lots of child. We’re planning to provide better performance with LeoFS v.1.4.

If you use LeoFS with NFS, you should set the size of a chunked object in LeoFS to 1MB (1048576Bytes) - large_object.chunked_obj_len = 1048576 (leo_gateway.conf), otherwise the efficiency of disk utilization can be decreased.

What are the requirements to run LeoFS with NFS?

LeoFS Gateway(NFS Server)

We’ve supporeted the targets Debian 6, Ubuntu-Server 14.04 LTS or Higher and CentOS 6.5/7.0 as LeoFS does, but should work on most linux platforms. In addition, We’ve confirmed LeoFS with NFS works properly on the latest FreeBSD and SmartOS by using NFS Integration Test Tool.

NFS Client

Since we still have not implemented NFS Lock Manager, you can only use NFS Clients which support disabling lock option(the linux client might be the only option for now). After we will finish implementing NFS Lock Manager, You can use NFS Client on other xnix platforms as well.

See Also:

Configuration of Gateway nodes

NFS R/W transfer block size is limited up to 1MB

NFS Lock Manager Protocol

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

LeoFS Frequently Asked Questions »

FAQ: LeoFS Administration

Where can I get the packages for LeoFS?

You can get the packages by platform from LeoFS download page. LeoFS package for FreeBSD has been published at Fresh ports/database and for SmartOS has been published at LeoFS packages for SmartOS, which is made by Project FiFo.

How can I run my LeoFS cluster automatically?

Currently after restarting a storage node, you need to issue the resume command with leofs-adm script manually like this.

leofs-adm resume storage_0@127.0.0.1

We’re planing to provide LeoFS’s autonomous operation like auto-compaction, auto-rebalance and others with LeoFS v1.2, v1.4 and v2.0. Actually, auto-compaction which was already supported with v1.2.

How do multiple users login into LeoFS Manager’s console at the same time?

Answer 1:

Actually, there is no login status in LeoFS, but the number of listening tcp connections is limited by leo_manger.conf (default: console.acceptors.cui = 3). So while using default settings, 3 connections can be connected to a manager in parallel.

Answer 2:

Since LeoFS v1.1.0, LeoFS have the more powerful alternative option leofs-adm command. This command have not only same functionalities with the existing telnet way but also do NOT keep an tcp connection established for long time(connect only when issueing a command). So you do not need to worry about the number of tcp connections.

The result of the du command can be different with the actual disk-usage

In order to reduce system resource usage for calculating the result of the du command, LeoFS keep that information on memory and when stopping itself, LeoFS save those data into a local file. Then when restarting, LeoFS load those on memory.

So if LeoFS is stopped unintentionally like killing by OOM killer, those data can become inconsistent with actual usage.

If you get into this situation, you can recover those data by issueing the compact-start command to the node. after finishing the compaction, the result of the du command will be consitent with actual usage.

See Also:

du-command

When issueing the recover node command the LeoFS can get into high load

Since the recover-node command can lead to issue lots of disk I/O and consume network bandwidth, if you face to issue the recover-node to multiple nodes at once, LeoFS can get into high load and become unresponsive. So we recommend you execute the recover-node command to target nodes one by one.

If this solution could not work for you, you’re able to control how much recover-node consume system resources by changing the MQ-related parameters in leo_storage.conf

See Also:

recover-node command

leo_storage.conf

What should I do when Too many processes errors happen?

LeoFS usually try to keep the number of Erlang processes as minimum as possible, but there are some exceptions when doing something asynchronously.

Replicating an object to the non-primary assigned nodes

Retrying to replicate an object when the previous attempt failed

Given that LeoFS suffered from very high load AND there are some nodes downed for some reason, The number of Erlang processes gradually have increased and might have reached the sysmte limit.

We recommend users to set an appropriate value which depends on your workload to the +P option. Also if the +P option does NOT work for you, there are some possibilities that some external system resources like disk, network equipments have broken, Please check out the dmesg/syslog on your sysmtem.

See Also:

Erlang - +P

Why does starting a leo_storage using bitcask as metadata take too much time?

When starting a leo_storage with bitcask, since leo_storage always call the bitcask:merge operation, starting process may take too much time if leo_storage stored lots of objects. We recommend users to replace bitcask with leveldb by using Tool:Converting metadata from bitcask to leveldb.

How do I set “a number of containers” at LeoFS Storage configuration?

Objects/files are stored into LeoFS Storage containers which are log-structured files. So LeoFS has the data-compaction mechanism in order to remove unncessary objects/files from the object-containers of LeoFS Storage.

LeoFS’s performance is affected by the data-compaction. And also, LeoFS Storage temporally creates a new file of a object-container corresponding to the compaction target container, which means during the data-compaction needs disk space for the new file of object-container(s).

If write/update/delete operation is a lot, we recommend that the number of containers is 32 OR 64 because it’s possible to make effect of the data-compaction at a minimum as much as possible.

In conclusiton:

Read operation > Write operation:

# of containers = 8

A lot of Write/Update/Delete:

# of containers = 32 OR 64 (depends on the disk capacity and performance)

leo_storage can not start due to “enif_send_failed on non smp vm”

When starting leo_storage on a single core machine which crashes with an erl_nif error.

## Error log
enif_send: env==NULL on non-SMP VM
Aborted (core dumped)

In this case, you have faced with eleveldb's issue#120. You need to set a Erlang’s VM flag - -smp in your leo_storage configuration - leo_storage.conf as follows:

## leo_storage.conf
erlang.smp = enable

See also:

Erlang - erl_nif

LeoFS documentation - leo_storage configuration

leo_storage - leo_storage.conf

Why is the speed of rebalance/recover command too slow?

You must hit LeoFS's issue#359. Since this issue has been fixed with LeoFS v1.2.9, we’d recommend you upgrading to 1.2.9 or higher one, and Set an appropriate value for your environment to mq.num_of_mq_procs in your leo_storage.conf.

See also:

LeoFS documentation - leo_storage configuration

leo_storage - leo_storage.conf

How to run LeoFS on docker container?

This hack also should work for LeoFS.

Why does LeoFS’s SNMP I/F give me wrong values(0) instead of correct values?

You must hit LeoFS's issue#361. Since this issue has been fixed with LeoFS v1.2.9, we’d recommend you upgrading to 1.2.9 or higher one.

When adding a new storage node, that node doesn’t appear with leofs-adm status Why?

If you changed a WRONG node name before stopping the daemon, As a result, when a new daemon was starting, it failed to detect that the previous was still running and stop command did not work too. Since you can notice this kind of mistake in error.log with LeoFS v1.2.9, we’d recommend you upgrading to 1.2.9 or higher one.

See also:

Troubleshooting Adding a Storage Node

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

next |

previous |

LeoFS 1.3 documentation »

LeoFS Production Checklist

Production Checklist: LeoFS

What version should we use?

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

previous |

LeoFS 1.3 documentation »

LeoFS Production Checklist »

Production Checklist: LeoFS

What version should we use?

Use the latest stable one. With the version <= 1.3.0, LeoFS had a serious issue that may cause data-lost, so that use at least >= 1.3.1. Or in case you need to keep running LeoFS with older one for some reason, make sure that large_object.reading_chunked_obj_len <= large_object.chunked_obj_len in leo_gateway.conf. This setting prevent LeoFS from suffering Issue#531 (The last part of a large object can be broken with reading_chunked_obj_len > chunked_obj_len in leo_gateway.conf).

See Also:

Issue#531 (The last part of a large object can be broken with reading_chunked_obj_len > chunked_obj_len in leo_gateway.conf)

leo_gateway.conf

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.

Navigation

index

LeoFS 1.3 documentation »

Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | U | W | X

A

add-bucket-command

S3-API commands

add-endpoint-command

S3-API commands

Administration

LeoFS launch order

LeoFS launch step by step

Operation flow diagram

Amazon S3 Interface

Official Documents

Appendix

CDN Integration

Architecture

LeoFS Gateway

LeoFS Manager

LeoFS Storage

Attach/Detach node operation

System maintenance

Auto-compaction

Configuration

Getting Started

LeoFS Storage

Purpose

B

backup-mnesia-command

Manager maintenance

Benchmark

Benchmark, [1]

Build Erlang on CentOS

Install

Build Erlang on Ubuntu/Debian

Install

C

CDN Integration

Appendix

How to integrate LeoFS with CDN

How to use http_custom_header.conf

How to write http_custom_header.conf

Use cases

CentOS

Install

Installation

chown-bucket-command

S3-API commands

cluster-status-command

Multi Datacenter replication commands

compact-resume-command

Compaction commands

compact-start-command

Compaction commands

compact-status-command

Compaction commands

compact-suspend-command

Compaction commands

Compaction commands

compact-resume-command

compact-start-command

compact-status-command

compact-suspend-command

diagnose-start-command

du-command

du-detail-command

Configuration

Auto-compaction

Consistency level

LeoFS Gateway

LeoFS Manager-Master

LeoFS Manager-Slave

LeoFS Storage

LeoFS Use Case - Without load-balancer

LeoFS with NFS

Multi data center replication

NFS

Relationship of configuration files

Watchdog

Configuration file for basho_bench

Configuration file for basho_bench, [1]

Configuration of eager check I/O scheduling for Erlang's VM

LeoFS Gateway

LeoFS Storage

Configuration related to Disk Cache

LeoFS Gateway

Configuration related to MQ

LeoFS Storage

Configuration related to the auto-compaction

LeoFS Storage

Confirm Erlang

Install

Consistency level

Configuration

create-user-command

S3-API commands

D

delete-bucket-command

S3-API commands

delete-endpoint-command

S3-API commands, [1]

delete-user-command

S3-API commands

detach-command

Storage operation

detach-storage

diagnose-start-command

Compaction commands

Download Erlang

Install

du-command

Compaction commands

du-detail-command

Compaction commands

dump-ring-command

Manager maintenance

E

Examples

REST API

F

FAQ fundamentals

Is there the roadmap of LeoFS?

What are typical uses for LeoFS?

What is architecture of LeoFS?

What is benefit for LeoFS users?

What kind of storage is leofs?

What language can I use to work with LeoFS?

What language is LeoFS written in?

FAQ LeoFS administration

Features

FAQ LeoFS limits

Features

NFS support

Operations

Features

FAQ LeoFS administration

FAQ LeoFS limits

Firewall Rules

Installation

G

General commands

status-command

whereis-command

get-bucket-command

S3-API commands

get-buckets-command

S3-API commands

get-users-command

S3-API commands

Getting Started

Auto-compaction

Multi data center replication

NFS

Quick Start: All in one in a local node

Quick Start: Building a cluster

Watchdog

H

How to determine the name of Bucket

Installation

How to integrate LeoFS with CDN

CDN Integration

How to use http_custom_header.conf

CDN Integration

How to write http_custom_header.conf

CDN Integration

I

Install

Build Erlang on CentOS

Build Erlang on Ubuntu/Debian

CentOS

Confirm Erlang

Download Erlang

Ubuntu/Debian

libatomic_ops

Installation

CentOS

Firewall Rules

How to determine the name of Bucket

LeoFS

XFS, [1]

Interface

REST API

Is there the roadmap of LeoFS?

FAQ fundamentals

J

join-cluster-command

Multi Datacenter replication commands

L

LeoCenter

LeoFS

Installation

LeoFS Gatewat

SNMP

LeoFS Gateway

Architecture

Configuration

Configuration of eager check I/O scheduling for Erlang's VM

Configuration related to Disk Cache

Watchdog

LeoFS Gateway access-log format

System maintenance

LeoFS Gateway operation

purge-command

remove-command

LeoFS launch order

Administration

LeoFS launch step by step

Administration

LeoFS Manager

Architecture

SNMP

LeoFS Manager-Master

Configuration

LeoFS Manager-Slave

Configuration

LeoFS Storage

Architecture

Auto-compaction

Configuration

Configuration of eager check I/O scheduling for Erlang's VM

Configuration related to MQ

Configuration related to the auto-compaction

SNMP

Watchdog

LeoFS Storage cluster

LeoFS Storage operation

LeoFS Storage data-diagnosis-log format

System maintenance

LeoFS Storage MQ

LeoFS Storage operation

LeoFS Storage operation

LeoFS Storage MQ

LeoFS Storage cluster

LeoFS Use Case - Without load-balancer

Configuration

LeoFS with NFS

Configuration

libatomic_ops

Install

Limits

NFS

M

Manager maintenance

backup-mnesia-command

dump-ring-command

restore-mnesia-command

update-managers-command

Multi data center replication

Configuration

Getting Started

Multi Datacenter replication commands

cluster-status-command

join-cluster-command

remove-cluster-command

N

NFS

Configuration

Getting Started

Limits

Purpose, [1]

NFS support

FAQ LeoFS limits

O

Official Documents

Amazon S3 Interface

Operation flow diagram

Administration

Operations

FAQ LeoFS limits

P

Production Checklist about LeoFS

What version should we use?

purge-command

LeoFS Gateway operation

Purpose

Auto-compaction

NFS, [1]

Watchdog

Q

Quick Start: All in one in a local node

Getting Started

Quick Start: Building a cluster

Getting Started

R

rebalance-command

Storage operation

Recover a storage-node

System maintenance

Recover commands

recover-cluster-command

recover-file-command

recover-node-command

recover-ring-command

recover-cluster-command

Recover commands

recover-file-command

Recover commands

recover-node-command

Recover commands

recover-ring-command

Recover commands

Relationship of configuration files

Configuration

remove-cluster-command

Multi Datacenter replication commands

remove-command

LeoFS Gateway operation

REST API

Examples

Interface

restore-mnesia-command

Manager maintenance

resume-command

Storage operation

Run basho_bench

Run basho_bench, [1]

S

S3-API commands

add-bucket-command

add-endpoint-command

chown-bucket-command

create-user-command

delete-bucket-command

delete-endpoint-command, [1]

delete-user-command

get-bucket-command

get-buckets-command

get-users-command

update-acl-command

update-user-role-command

Set up basho_bench

Set up basho_bench, [1]

SNMP

LeoFS Gatewat

LeoFS Manager

LeoFS Storage

start-command

Storage operation

status-command

General commands

Storage operation

detach-command

rebalance-command

resume-command

start-command

suspend-command

suspend-command

Storage operation

System maintenance

Attach/Detach node operation

LeoFS Gateway access-log format

LeoFS Storage data-diagnosis-log format

Recover a storage-node

Upgrade old version to 1.3.1

U

Ubuntu/Debian

Install

update-acl-command

S3-API commands

update-managers-command

Manager maintenance

update-user-role-command

S3-API commands

Upgrade old version to 1.3.1

System maintenance

Use cases

CDN Integration

W

Watchdog

Configuration

Getting Started

LeoFS Gateway

LeoFS Storage

Purpose

What are typical uses for LeoFS?

FAQ fundamentals

What is architecture of LeoFS?

FAQ fundamentals

What is benefit for LeoFS users?

FAQ fundamentals

What kind of storage is leofs?

FAQ fundamentals

What language can I use to work with LeoFS?

FAQ fundamentals

What language is LeoFS written in?

FAQ fundamentals

What version should we use?

Production Checklist about LeoFS

whereis-command

General commands

X

XFS

Installation, [1]

© Copyright 2012-2016, Leo Project - leo-project.net/leofs/ - Sponsored by Rakuten, Inc. and Rakuten Institute of Technology. Created using Sphinx 1.3.1.