HAL : Hyper Adaptive Learning

Rust based Cross-GPU Machine Learning.

Why Rust?

This project is for those that miss strongly typed compiled languages. Rust was chosen specifically because of this Furthermore, we can offer fine grained control of your operations. This means being able to grab dimensions of tensors at any stage, none of that unknown shape nonsense. We can also micro-control steps. An example of this working with each individual forward timesteps on say an LSTM. Usually these are controlled by inner loops [Theano/Tensorflow, etc].

Features

• Multi GPU [model based] support
• OpenCL + CUDA + Parallel CPU support
• LSTM's with internal RTRL [Work in Progress]
• RNN's [Work in Progress]
• Perceptrons, AutoEncoders, ConvNets[TODO]
• Optimizers: [SGD, Adam, AdaGrad[TODO]]
• Activations: [Linear, Sigmoid, Tanh, ReLU, LReLU, Softmax]
• Initializations: [Lecun Uniform, Glorot Normal, Glorot Uniform, Normal, Uniform]
• Data Gatherers: [SinSource, MNIST[In Progress], CIFAR10[TODO]]
• Loss Functions: [MSE, L2, Cross-Entropy]
• OpenGL based plotting and image loading, see here for more info
• Multi GPU [horizontal] support [TODO]

Requirements

• Rust 1.8 +
• arrayfire libs preinstalled (or you can compile manually by following instructions below) bash cargo run --example autoencoder

Installation

See here

Testing

HAL utilizes RUST's test framework to extensively test all of our modules. We employ gradient checking on individual functions as well as layers. Testing needs to run on one thread due to our device probing methods. Furthermore, graphics needs to be disabled for testing [glfw issue]. bash AF_DISABLE_GRAPHICS=1 RUST_TEST_THREADS=1 cargo test

If you would like to see the results of the test (as well as benchmarks) run: bash AF_DISABLE_GRAPHICS=1 RUST_TEST_THREADS=1 cargo test -- --nocapture

Credits

• Thanks to the arrayfire team for working with me to get the rust bindings up.
• Keras for inspiration as a lot of functions are similar to their implementation (minus the theano nonsense).
• Dr. Felix Gers for his insight into internal RTRL
• Dr. Sepp Hochreiter for advise on LSTM's