A simple tool to gather data about an S3 bucket (or subsection thereof). Designed to be simple, and (sort-of) shell consumable. Metadata will be added as it becomes available, and S3 interaction is controlled by rusoto_s3.
You can install s3-meta from either this repository, or from Crates (once it's published):
```shell
$ cargo install s3-meta
$ cargo install --git https://github.com/whitfin/s3-meta.git ```
Credentials can be configured by following the instructions on the AWS Documentation, although examples will use environment variables for the sake of clarity.
You can retrieve metadata about a bucket using the following form (making sure your region matches your bucket AWS region):
shell
$ AWS_ACCESS_KEY_ID=MY_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=MY_SECRET_ACCESS_KEY \
AWS_DEFAULT_REGION=us-west-2 \
s3-meta my.bucket.name
If you want to retrieve metadata about a "subdirectory" of a bucket, you can provide a prefix using this format:
shell
$ AWS_ACCESS_KEY_ID=MY_ACCESS_KEY_ID \
AWS_SECRET_ACCESS_KEY=MY_SECRET_ACCESS_KEY \
AWS_DEFAULT_REGION=us-west-2 \
s3-meta my.bucket.name/my/directory/path
Don't forget to add a space to the start of your command if you're going to inline your credentials as above!
Output is pretty straightforward, and follows a relatively simple format which is easily extensible, and hopefully convenient in shell pipelines. There may be changes made to this format to make it easier to consume (spaces placed to make splitting easier, unformatted numbers, etc).
Below is an example based on a real S3 bucket (although the file names have been tweaked :)):
``` [general] totaltime=7s totalspace=1.94TB total_files=51,152
[filesize] averagefilesize=37.95MB averagefilebytes=37949529 largestfilesize=1.82GB largestfilebytes=1818900684 largestfilename=path/to/mylargestfile.txt.gz smallestfilesize=54B smallestfilebytes=54 smallestfilename=path/to/mysmallestfile.txt.gz smallestfile_others=12
[extensions] uniqueextensions=1 mostfrequent_extension=gz
[modification] earliestfiledate=2016-06-11T17:36:57.000Z earliestfilename=path/to/myearliestfile.txt.gz earliestfileothers=3 latestfiledate=2017-01-01T00:03:19.000Z latestfilename=path/to/mylatestfile.txt.gz ```
This sample is based on the initial builds of s3-meta. Depending on when you come to this tool, there may be more (or less) included in the output above.