Getting Started

System Requirements

LeoFS development currently targets Debian 6, Ubuntu-Server 12.04 LTS|13.04 and CentOS 6.x, but should work on most Linux platforms with the following software:

LeoFS includes the following Erlang libraries:

We recommend that you use a 64bit system to be able to handle large files.

Getting LeoFS

Quick Start -1 All in one for Application Development

Purpose

This section is a step by step guide to setting up LeoFS for the first time. By following this tutorial you can easily build a stand-alone LeoFS system.

Note

In this section, LeoFS-Storage, LeoFS-Gateway and LeoFS-Manager are all installed on a single system, with no clustering.

1. Install

Erlang (CentOS, Ubuntu, Other Linux OS)

##
## 1. Install libatomic
##
$ wget http://www.hpl.hp.com/research/linux/atomic_ops/download/libatomic_ops-7.2d.tar.gz
$ tar xzvf libatomic_ops-7.2d.tar.gz
$ cd libatomic_ops-7.2d
$ ./configure --prefix=/usr/local
$ make
$ sudo make install

##
## 2. Install Erlang (R16B03-1)
##
$ wget http://www.erlang.org/download/otp_src_R16B03-1.tar.gz
$ tar xzf otp_src_R16B03-1.tar.gz
$ cd otp_src_R16B03-1
$ ./configure --prefix=/usr/local/erlang/R16B03-1 \
              --enable-smp-support \
              --enable-m64-build \
              --enable-halfword-emulator \
              --enable-kernel-poll \
              --without-javac \
              --disable-native-libs \
              --disable-hipe \
              --disable-sctp \
              --enable-threads \
              --with-libatomic_ops=/usr/local
$ make
$ sudo make install

##
## 3. Set PATH
##
$ vi ~/.profile
    ## append the follows:
    export ERL_HOME=/usr/local/erlang/R16B03-1
    export PATH=$PATH:$ERL_HOME/bin

$ source ~/.profile

LeoFS

$ git clone https://github.com/leo-project/leofs.git
$ cd leofs
$ make && make release

2. Configuration

Modify “/etc/hosts”

  • Add a domain for the LeoFS bucket in /etc/hosts
  • Bucket names must follow these rules
$ sudo vi /etc/hosts

## Replace ${BUCKET_NAME} with the name of the bucket ##
127.0.0.1 localhost ${BUCKET_NAME}.localhost

3. Launch LeoFS’ managers and storage

  • By default there is only one replica, you may want to configure the system.
  • Start master-manager, slave-manager
  • Start a storage node
$ cd $LEOFS_ROOT/package
$ leo_manager_0/bin/leo_manager start
$ leo_manager_1/bin/leo_manager start
$ leo_storage/bin/leo_storage start

4. Start the system

  • Use the command start in the LeoFS manager console
$ telnet 127.0.0.1 10010
> start

5. Start a LeoFS gateway node

$ leo_gateway/bin/leo_gateway start

6. Confirm that the system is running

  • Use the command status in the LeoFS manager console
$ telnet 127.0.0.1 10010
> status
status
[system config]
             version : 0.14.4
 # of replicas       : 1
 # of successes of R : 1
 # of successes of W : 1
 # of successes of D : 1
 # of DC-awareness replicas   : 0
 # of Rack-awareness replicas : 0
           ring size : 2^128
    ring hash (cur)  : 1428891014
    ring hash (prev) : 1428891014

[node(s) state]
------------------------------------------------------------------------------------------------
 node                        state       ring (cur)    ring (prev)   when
------------------------------------------------------------------------------------------------
 storage_0@127.0.0.1         running     1428891014    1428891014    2013-07-04 11:23:08 +0900
 gateway@127.0.0.1           running     1428891014    1428891014    2013-07-04 11:24:37 +0900

7. Get your S3 API Key from the LeoFS manager console

  • Use the command create-user in the LeoFS manager console
  • It takes the user name as its only argument
$ telnet 127.0.0.1 10010
> create-user ${YOUR_NAME}
access-key-id: 05dcba94333c7590a635
secret-access-key: c776574f3661579ceb91aa8788dfcac733b21b3a

8. Clients

9. Using LeoFS

  • Use the command add-bucket in the LeoFS manager console
  • It takes the bucket name and access-key-id got in the previous section as its arguments
$ telnet 127.0.0.1 10010
> add-bucket ${BUCKET_NAME} ${YOUR_ACCESS_KEY_ID}
ok
  • Insert some data into LeoFS by using any S3 client as mentioned above
  • You can now get the data stored in LeoFS
$ curl http://localhost:8080/your_bucket_name/path/to/file
> ${CONTENTS}

Note

From version 0.16.0, you need to set ACL settings of your bucket to public-read by using the command update-acl if you want to get the data stored in LeoFS via web browser.

Wrap up

You now know how to setup a stand-alone LeoFS system. Make sure to have a look at Quick Start -2 Cluster to learn how to setup a LeoFS cluster.

Quick Start -2 Cluster

Purpose

This tutorial teaches you how to easily build a LeoFS cluster. All steps will not be explained in detail, it is assumed you already know how to setup a stand-alone LeoFS system. This guide exists to help you get a cluster up and running quickly. We recommend that you read the LeoFS Installation, Configuration and Administration Guide to learn how to administer your LeoFS cluster. We hope that by reading this tutorial you will be able to get a cluster started as quickly as possible.

Case example

1. Install Erlang and LeoFS on each server

2. Configuration - Edit “vm.args” on each server

  • File path: “$LEOFS_ROOT/package/leo_*/etc/vm.args”

  • Precondition
    • -name must be unique for each node in the LeoFS cluster
  • Edit Manager-master’s vm.args

## Name of the node
-name manager_0@10.0.1.101
... omitted below
  • Edit Manager-slave’s vm.args
## Name of the node
-name manager_1@10.0.1.102
... omitted below
  • Edit Gateway’s vm.args
## Name of the node
-name gateway_0@10.0.1.103
... omitted below
  • Edit Storage’s vm.args
## Name of the node
-name storage_0@10.0.1.104
... omitted below

3. Configuration - Consistency level

  • Reference: The consistency level

  • Edit Manager’s app.config
    • You only need to modify Manager-master for the consistency level.
    • “$LEOFS_ROOT/package/leo_manager_0/etc/app.config”
[
    {leo_manager, [
               %% == System Ver ==
               {system_version, "0.14.4" },

               %% == System Configuration ==
               %% - Consistency Level
               {system, [{n, 2 },  %% number of replicated files is 2
                         {w, 1 },  %% number of of successes of write-operation is 1
                         {r, 1 },  %% number of of successes of read-operation is 1
                         {d, 1 },  %% number of of successes of delete-operation is 1
                         {bit_of_ring, 128} %% size of routing-table (RING)
                        ]},

4. Order of server launch

  • Manager-master
  • Manager-slave
  • Storage nodes
  • Gateway(s)

5. Method of server launch

  • Shell script: “$LEOFS_ROOT/package/leo_*/bin/leo_*”
  • Launch Manager-master
$ $LEOFS_ROOT/package/leo_manager_0/bin/leo_manager start
  • Launch Manager-slave
$ $LEOFS_ROOT/package/leo_manager_1/bin/leo_manager start
  • Launch each Storage nodes
$ $LEOFS_ROOT/package/leo_storage/bin/leo_storage start
  • Launch each Gateway nodes
$ $LEOFS_ROOT/package/leo_gateway/bin/leo_gateway start

6. Start the system

  • Use the command start in the LeoFS manager console
$ telnet 127.0.0.1 10010
> start

7. Confirm that the system is running

  • Use the command status in the LeoFS manager console
$ telnet 127.0.0.1 10010
> status
status
[system config]
             version : 0.14.4
 # of replicas       : 2
 # of successes of R : 1
 # of successes of W : 1
 # of successes of D : 1
 # of DC-awareness replicas   : 0
 # of Rack-awareness replicas : 0
           ring size : 2^128
    ring hash (cur)  : 1428891014
    ring hash (prev) : 1428891014

[node(s) state]
------------------------------------------------------------------------------------------------
 node                        state       ring (cur)    ring (prev)   when
------------------------------------------------------------------------------------------------
 storage_0@10.0.1.104        running     1428891014    1428891014    2013-07-04 11:23:08 +0900
 storage_1@10.0.1.105        running     1428891014    1428891014    2013-07-04 11:23:08 +0900
 storage_2@10.0.1.106        running     1428891014    1428891014    2013-07-04 11:23:08 +0900
 gateway_0@10.0.1.103        running     1428891014    1428891014    2013-07-04 11:24:37 +0900

8. Get your S3 API Key from the LeoFS manager console

  • Use the command create-user in the LeoFS manager console
  • It takes the user name as its only argument
$ telnet 127.0.0.1 10010
> create-user ${YOUR_NAME}
access-key-id: 05dcba94333c7590a635
secret-access-key: c776574f3661579ceb91aa8788dfcac733b21b3a

9. Using LeoFS

  • Use the command add-bucket in the LeoFS manager console
  • It takes the bucket name and access-key-id got in the previous section as its arguments
$ telnet 127.0.0.1 10010
> add-bucket ${BUCKET_NAME} ${YOUR_ACCESS_KEY_ID}
ok
  • Insert some data into LeoFS by using any S3 client as mentioned above
  • You can now get the data stored in LeoFS
$ curl http://localhost:8080/your_bucket_name/path/to/file
> ${CONTENTS}

Note

From version 0.16.0, you need to set ACL settings of your bucket to public-read by using the command update-acl if you want to get the data stored in LeoFS via web browser.

Wrap up

You now have a working LeoFS cluster. Make sure to have a look at LeoFS installation, LeoFS Configuration and Administration Guide to learn more about setting up and managing your LeoFS cluster.