

FAQ: LeoFS Clients¶

Name Resolution¶

There are two ways to access LeoGateway from LeoFS' Clients.
- Path Style Access
- Sub-Domain Style Access
If a LeoFS Client you use adopts the sub-domain style access, then the client need to resolve bucket.endpoint domain into the IP address of LeoGateway.
Related Links:
- A Comment on LeoFS' Issue #748, Connecting to bucket causes badarg error
- For Administrators / System Operations / S3 / Endpoint

User Defined Metadata¶

Although we support User Defined Metadata according to the original S3 spec, we have reserved every key which prefix is "x-amz-meta-leofs-" as system reserved ones. So please be careful not to use User Defined Metadata with the key prefixed "x-amz-meta-leofs-".

Related Links:
- User Defined Metadata

Uploading Objects¶

See Administrators / Settings / LeoGateway Settings - Uploading Objects

Has LeoFS supported the gzip compression on GET?¶

Yes since 1.4.3, you can request the gzip compression to LeoFS by adding AcceptEncoding: gzip into the HTTP request header. However it doesn't mean the gzip compression is always applied to any objects returned from LeoFS. it's applied only in case object.size >= 300 && object.size < large_object.chunked_obj_len in leo_gateway.conf.

Can't read files stored in LeoFS using s3fs¶

s3fs can make a part request in a different size on each part as described below.

1
2
3

[PUT]   test    test/test.parquet       1       5596007 2018-08-01 14:28:13.217763 +0900        1533101293217754 200     49
[PUT]   test    test/test.parquet       2       5246231 2018-08-01 14:28:13.378359 +0900        1533101293378348 200     34
[PUT]   test    test/test.parquet       3       553677  2018-08-01 14:28:13.400117 +0900        1533101293400082 200     7

(The fifth column denotes the size of a body LeoGateway receives.)

However LeoGateway expects the part size to be same among all parts except the last one so that LeoGateway may fail to calculate the position where to start reading when handling a GET with Range Request. As a result, LeoGateway may respond wrong data to clients and that's the case.

Workaround¶

There is a init param "default_block_size" which allow library users to specify the threshold over which an object will be sent through multipart upload so setting a enough large value to the param make multipart uploads never happen. The below code is an example to avoid multipart uploads with s3fs.

boto3_dict = {'endpoint_url':'http://localhost:8080'}
fs = s3fs.S3FileSystem(key= ACCESS_KEY, secret = SECRET_ACCESS_KEY,
                       default_block_size = 104857600,
                       client_kwargs=boto3_dict)

With the above code, you can read/write files which size are less than 100MB with s3fs.

Related Links:
- s3fs