Files - CDN Origin Server for Cloud Storage

Purpose

The purpose of this service is to be your CDN Origin Server for Cloud Storage for any JSON client including JavaScript front-ends like React with fetch.

Files supports aws s3, vultr, wasabi, yandex, and digital ocean.

Files is built to be simple and blazing fast with JWT verification, bucket management, and object management.

To use this service you need to have a running broker server.

Features

• Very performant with almost no CPU and memory usage
• Supports bucket and object management
• Supports cloud storage providers: aws s3, wasabi, yandex, and digital ocean
• Supports being an origin server for CDN77, Cloudflare, or other CDN providers
• Under 1000 lines of code
• Supports CORS
• Multi-tenant
• Supports JWT authentication
• Supports JWT caching with expiry checking to minimize verify API calls
• Supports SSL - full end-to-end encryption
• JSON API
• Auto-provision and renews SSL cert via LetsEncrypt or use your own SSL cert
• Uses user authorization scoping
• CLI Application - zn
• Built on broker

Use

1. create a user on broker with the following scopes - for full permissions files:full or granular permissions files:provider, files:get_object, files:put_object, files:delete_object, files:create_bucket, files:list_bucket, or files:delete_bucket
2. login to broker and get a JWT
3. attach the JWT as an Authorization: Bearer {token} to the following JSON API endpoints
4. Add your credentials for your provider(s) - the creds are just for the specific user that creates them

• Valid providers are aws, wasabi, yandex, vultr, and do
• Valid regions are:

| Provider | Region | |--- | --- | | wasabi | wa-us-east-1 | | wasabi | wa-us-east-2 | | wasabi | wa-us-west-1 | | wasabi | wa-eu-central-1 | | aws | us-east-1 | | aws | us-east-2 | | aws | us-west-1 | | aws | us-west-2 | | aws | ca-central-1 | | aws | ap-south-1 | | aws | ap-northeast-1 | | aws | ap-northeast-2 | | aws | ap-northeast-3 | | aws | cn-north-1 | | aws | cn-northwest-1 | | aws | eu-north-1 | | aws | eu-central-1 | | aws | eu-west-1 | | aws | eu-west-2 | | aws | eu-west-3 | | aws | me-south-1 | | aws | sa-east-1 | | do | nyc3 | | do | ams3 | | do | spg1 | | do | fra1 | | vultr | ewr1 | | yandex | ru-central1 |

Add/Update Provider

html POST /provider - authenticated endpoint (Authorization: Bearer {jwt}) example: json { "name": "aws", "access_key": "AKIAIOSFODNN7EXAMPLE", "secret_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" } will return: 200 or 500 or 400 or 401

Get Object

html POST /get - authenticated endpoint (Authorization: Bearer {jwt}) example: json { "object_path": "/test.pdf", "region": "wa-us-east-1", "bucket": "test" } will return: 200 or 500 or 400 or 401

200 - will return a file stream (application/octet-stream)

Put Object

html POST /put - authenticated endpoint (Authorization: Bearer {jwt}) example: json { "object_path": "/test.pdf", "region": "wa-us-east-1", "bucket": "test", "file": "dGhpc2lzYXN0cmluZw==" } - file is a base64 encoded binary file - this is doable with the FileReader class in the browser

will return: the provider's status code (e.g. 200, 400)

• note: this action will overwrite a file if the file is the same name

Delete Object

html POST /del - authenticated endpoint (Authorization: Bearer {jwt}) example: json { "object_path": "/test.pdf", "region": "wa-us-east-1", "bucket": "test" } will return: the provider's status code (e.g. 204, 409)

Create Bucket

html POST /create_bucket - authenticated endpoint (Authorization: Bearer {jwt}) example: json { "region": "wa-us-east-1", "bucket": "test" } will return: 200 with no body response or the provider's status code (e.g. 409) and the provider's error json { "error": "provider's response as an xml string" } - note: if the bucket already exists the request will return 200

Delete Bucket

html POST /delete_bucket - authenticated endpoint (Authorization: Bearer {jwt}) example: json { "region": "wa-us-east-1", "bucket": "test" } will return: the provider's status code (e.g. 204, 409) - note: all objects in the bucket have to be deleted before the bucket can be deleted (based on provider's rules)

List Bucket

html POST /list_bucket - authenticated endpoint (Authorization: Bearer {jwt}) example: json { "region": "wa-us-east-1", "bucket": "test", "prefix": "/", "delimiter": "/" } - delimiter is an optional field

will return: 200, 400, 500, 401

200 - will return an array of objects json { "objects": [{ "key": "/test.pdf", "e_tag": "33a64df551425fcc55e4d42a148795d9f25f89d4", "storage_class": "STANDARD", "size": 100, "last_modified": "2021-04-14T22:34:04.000Z" }] }

Health Check

html GET or HEAD / - public endpoint

will return: 200

Install

cargo install files

• the origin can be passed in as a flag - default *
• the port can be passed in as a flag - default 9999 - can only be set for unsecure connections
• the secure flag for https can be true or false - default false
• the auto_cert flag for an autorenewing LetsEncrypt SSL cert can be true or false - requires a resolvable domain - default true
• the key_path flag when auto_cert is false to set the SSL key path for your own cert - default certs/private_key.pem
• the cert_path flag when auto_cert is false to set the SSL cert path for your own cert - default certs/chain.pem
• the certs flag is the storage path of LetsEncrypt certs - default certs
• the db flag is the path where the embedded database will be saved - default db
• the domain flag is the domain name (e.g. api.broker.com) of the domain you want to register with LetsEncrypt - must be fully resolvable
• the broker flag is the broker domain/ip/port of the broker server - default http://localhost:8080
• production example: ./portal --secure="true" --domain="files.broker.com" --broker="https://broker.broker.com"

Service

There is an example systemctl service for Ubuntu called files.service in the code

TechStack

• Tide
• RocksDB

Inspiration

• Broker