Neural Networks in Rust

Implementing Neural Networks in Rust from scratch + utils for data manipulation.

This was made for the purpose of my own learning. It is obviously not a production-ready library by any means.

Feel free to give feedback.

Include

Add this in your project's Cargo.toml file:

toml [dependencies] neural_networks_rust = "*"

Code example

Simple regression workflow example:

```rust // Loading a model specification from JSON let mut model = Model::fromjsonfile("mymodelspec.json");

// Applying a data pipeline on it according to its dataset specification let mut pipeline = Pipeline::basicsinglepass(); let (updateddatasetspec, data) = pipeline .add(AttachIds::new("id")) .run("./dataset", &model.dataset);

let model = model.withnewdataset(updateddatasetspec);

// Training it using k-fold cross validation // + extracting test & training metrics per folds & per epochs // + extracting all predictions made during final epoch let kfold = model.trainer.maybekfold().expect("We only do k-folds here!"); let (validationpreds, modeleval) = kfold .attachrealtimereporter(|report| println!("Perf report: {:#?}", report)) .run(&model, &data);

// Reverting the pipeline on the predictions & data to get interpretable values let validationpreds = pipeline.revertcolumnswise(&validationpreds); let data = pipeline.revertcolumnswise(&data);

// Joining the data and the predictions together let dataandpreds = data.innerjoin(&validationpreds, "id", "id", Some("pred"));

// Saving it all to disk dataandpreds.tofile("mymodelpreds.csv"); modeleval.tojsonfile("mymodelevals.json"); ```

You can then plot the results using a third-party crate like gnuplot (recommended), plotly or even plotters.

But first you would need to write or generate your model's specification.

Here is an example generating it with code (recommended):

```rust // Including all features from some CSV dataset let mut datasetspec = Dataset::fromcsv("kchousedata.csv"); datasetspec // Removing useless features for both the model & derived features .removefeatures(&["id", "zipcode", "sqftliving15", "sqftlot15"]) // Setting up the price as the "output" predicted feature .addoptto("price", Out) // Setting up the date format .addoptto("date", DateFormat("%Y%m%dT%H%M%S")) // Converting the date to a datetimestamp feature .addoptto("date", AddExtractedTimestamp) // Excluding the date from the model .addoptto("date", Not(&UsedInModel)) // Mapping yrrenovated to yrbuilt if = to 0 .addoptto( "yrrenovated", Mapped( MapSelector::Equal(0.0.into()), MapOp::ReplaceWith(MapValue::Feature("yrbuilt".tostring())), ), ) // Converting relevant features to their log10 .addopt(Log10.only(&["sqftliving", "sqftabove", "price"])) // Adding ^2 features of all input features // (including the added ones like the timestamp) .addopt(AddSquared.except(&["price", "date"]).incladdedfeatures()) // Filtering rows according to feature's outliers .addopt(FilterOutliers.except(&["date"]).incladdedfeatures()) // Normalizing everything .addopt(Normalized.except(&["date"]).incladdedfeatures());

// Creating our layers let hsize = datasetspec.infeaturesnames().len() + 1; let nh = 8;

let mut layers = vec![]; for i in 0..nh { layers.push(LayerSpec::fromoptions(&[ OutSize(hsize), Activation(ReLU), Optimizer(adam()), ])); } let finallayer = LayerSpec::fromoptions(&[ OutSize(1), Activation(Linear), Optimizer(adam()), ]);

// Putting it all together let model = Model::fromoptions(&[ Dataset(datasetspec), HiddenLayers(layers.asslice()), FinalLayer(finallayer), BatchSize(128), Trainer(Trainers::KFolds(8)), Epochs(300), ]);

// Saving it all model.tojsonfile("mymodelspec.json"); ```

Working features

Neural Networks

• Neural Network abstraction
• Layers
  • Shared abstraction
    • Forward/Backward passes
    • Handle (mini)batches
  • Dense layers
    • Weights optimizer/initializer (no regularization yet)
    • Biases optimizer/initializer (no regularization yet)
  • Activation layers
    • Linear
    • Hyperbolic Tangent
    • ReLU
    • Sigmoid
    • Tanh
  • Full layers (Dense + Activation)
    • Optional Dropout regularization
• Stochastic Gradient Descent (SGD)
  • Optional optimizers
    • Momentum
    • Adam
  • Learning rate scheduling
    • Constant
    • Piecewise constant
    • Inverse time decay
• Initializers
  • Zeros
  • Random uniform (signed & unsigned)
  • Glorot uniform
• Losses
  • MSE

Models utils

• Abstractions over training
  • KFolds + model benchmarking in a few LoC
• Model specification
  • Simple API to create as code
  • From/to JSON conversion
  • Specify training parameters (k-folds, epochs...)
  • Specify the dataset
    • Features in/out
    • Features preprocessing
• Model benchmarking
  • Handy utilities to store & compute training metrics

Preprocessing

• Layer-based data pipelines abstraction
  • Cached & revertible feature mapping/feature extraction layers
    • Normalize
    • Square
    • Log10 scale
    • Extract month from Date string
    • Extract unix timestamp from Date string
    • Limited row mapping (for advanced manipulations)
  • Row filtering layers
    • Filter outliers
  • Other layers
    • Attach IDs column

Data analysis

• Simple Polars DataFrame wrapper (DataTable)
  • Load & manipulate CSV data
  • Generate input/output vectors
    • Sample
    • Shuffle
    • Split
    • K-folds
• Vector<f64> statistics & manipulation utils
  • Stats (mean, std dev, correlation, quartiles...)
  • Normalization

Linear algebra

• Many backends for the Matrix type (toggled using compile-time cargo features)
  • Feature nalgebra (enabled by default)
    • Fast
    • CPU-bound
  • Feature linalg
    • 100% custom Vec-based
    • Slow
    • CPU-bound
  • Feature linalg-rayon
    • linalg parallelized with rayon
    • Way faster than linalg but slower than nalgebra
    • CPU-bound
  • Feature faer
    • WIP
    • Should be fast but for now it's on par with linalg-rayon
    • CPU-bound
• Switching from f32 (default) to f64-backed Scalar type with the f64 feature