S3RS

an S3 Rich Support client - Download executable, unzip and ready to go - rust edition 2018 - multi config (please put config files under ~/.config/s3rs) - interactive command line tool - easy to debug with http protocol - AWS4, AWS2 support - support http redirect for multi region of AWS S3 - support automatically multipart upload - support filters v0.2.8 - support format without protocol v0.2.9 - support command-line interface mode v0.3.0 - support s3 config file encryption function v0.4.0 - s3rs -c ~/.config/s3rs/aws2.toml -s 'the password you like' secret encrypt - run s3rs with -s option or use secret set <secret phrases or hexal literal> - support multipart download

How to use

shell mode

• execute s3rs will into shell mode, and execute help, you can see the commands you can use

  command mode

• use config file with full path

  • s3rs --config=/your/s3s/config/file ls

• use config file example.toml in ~/.config/s3rs with file name without extension

  • s3rs --config=example ls

Command List

Table 1: List commands will send single http request

| COMMAND | FUNCTION | CEPH | AWS | |----------------------------------------------------------------------|-------------------------------------------------------------|------|-----| | ls | list all buckets | O | O | | ls s3://bucket | list objects in the bucket | O | O | | ls s3://bucket/prefix | list objects match prefix in the bucket | O | O | | ll | list all objects details (storage class, modify time, etag) | O | O | | ll s3://bucket | list objects detail in the bucket | O | O | | ll s3://bucket/prefix | list objects match prefix detail in the bucket | O | O | | mb s3://bucket | create bucket | O | O | | rb s3://bucket | delete bucket | O | O | | put s3://bucket/object | upload the file | O | O | | put s3://bucket | upload the file (file name as object name) | O | O | | put test s3://bucket/object | upload a test file specific object name | O | O | | get s3://bucket/object file | download object | O | O | | get s3://bucket/object | download object in current folder | O | O | | cat s3://bucket/object | show the object content | O | O | | del s3://bucket/object [delete-marker:true] | delete the object (with flag) | O | O | | | delete-marker used in AWS | | O | | tag list s3://bucket/object | list tag(s) to the object | O | O | | tag ls s3://bucket/object | list tag(s) to the object | O | O | | tag add s3://bucket/object key1=value1 [key2=value2] ... | add tag(s) to the object | O | O | | tag put s3://bucket/object key1=value1 [key2=value2] ... | add tag(s) to the object | O | O | | tag del s3://bucket/object | remove tag(s) from the object | O | O | | tag rm s3://bucket/object | remove tag(s) from the object | O | O | | /uri?query | give the original url | O | O | | secret set | set up the secret to run time encrypt or decrypt the config | O | O | | secret set | set up the secret to run time encrypt or decrypt the config | O | O | | secret unset | clean up the secret in memory | O | O | | secret encrypt | encrypted and print current config on screen | O | O | | secret encrypt | encrypted and save current config to s3rs config folder | O | O | | secret show | show current plain config on screen | O | O | | secret show | save current plain config to s3rs config folder | O | O |

Table 2: List commands will send more than one http request

| HIGH LEVEL COMMAND | INTEGRATE FUNCTIONS | CEPH | AWS | |--------------------|------------------------------------------------------------------------------------------|------|-----| | la | list all objects | O | O | | info s3://bucket | acl(ceph, aws), location(ceph, aws), versioning(ceph, aws), uploads(ceph), version(ceph) | O | O |

Table 3: List commands only for CEPH with system keys

| COMMAND | FUNCTION | |---------------------|-----------------------| | usage s3://bucket | show the bucket usage |

Table 4: List commands only for CEPH with system keys

| SHELL SETTING COMMAND | FUNCTION | |-------------------------------|------------------------------------------| | s3_type [ceph/aws/aws4/aws2] | change setting for different S3 provider | | format [xml/json] | change the format client request | | log [trace/debug/info/erro] | change the log level | | | - Info : for Http header and body | | | - debug: for auth signature hash info | | | - trace: more detail about rust | | logout | logout and reselect user | | Ctrl + d | logout and reselect user |

Table 5: The default format of S3 type

| S3 TYPE | AUTH TYPE | FORMAT | URL STYLE | |---------|-----------|--------|----------------------| | ceph | aws4 | json | path-style | | aws | aws4 | xml | virtual-hosted–style |

Table 6: The talks about this project

| Talks | Link | |------------|--------------------------------------------------------| | COSCon'19 | Slides |

Install via Crate.io

Install rust tools rustup and cargo - curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Additional package for ubuntu - apt-get install libssl-dev pkg-config openssl

It is easy to install s3rs via cargo as following command. - cargo install s3rs

Set up the path for rust binnary - export PATH=$PATH:~/.cargo/bin

The config file will auto setup when binary first time launch - s3rs

Edit the config file at ~/.s3rs.toml - vi ~/.s3rs.toml

Develop

Install rust tools rustup and cargo - curl https://sh.rustup.rs -sSf | sh - git clone https://github.com/yanganto/s3rs.git - cargo test - cargo build - The executable binary will in ./target/debug/s3rs

Demo

• A short demo video
• v0.2.8 update video
• v0.2.9 update video
• v0.3.0 update video
• snapshot