Lamellar is an asynchronous tasking runtime for HPC systems developed in RUST
Lamellar is an investigation of the applicability of the Rust systems programming language for HPC as an alternative to C and C++, with a focus on PGAS approaches.
Lamellar provides several different communication patterns to distributed applications. First, Lamellar allows for sending and executing active messages on remote nodes in a distributed environments. The runtime supports two forms of active messages: The first method works with Stable rust and requires the user the register the active message by implementing a runtime exported trait (LamellarAM) and calling a procedural macro (#[lamellar::am]) on the implementation. The second method only works on nightly, but allows users to write serializable closures that are transfered and exectued by the runtime without registration It also exposes the concept of remote memory regions, i.e. allocations of memory that can read/written into by remote nodes.
Lamellar relies on network providers called Lamellae to perform the transfer of data throughout the system. Currently two such Lamellae exist, one used for single node development purposed ("local"), and another based on the Rust OpenFabrics Interface Transport Layer (ROFI) (https://github.com/pnnl/rofi)
rust
use lamellar::Backend;
fn main(){
let mut world = lamellar::LamellarWorldBuilder::new()
.with_lamellae( Default::default() ) //if "enable-rofi" feature is active default is rofi, otherwise default is local
//.with_lamellae( Backend::Rofi ) //explicity set the lamellae backend
//.with_lamellae( Backend::Local )
.build();
}
```rust use lamellar::{ActiveMessaging, LamellarAm};
struct HelloWorld { //the "input data" we are sending with our active message my_pe: usize, // "pe" is processing element == a node }
impl LamellarAM for HelloWorld { fn exec(&self) { println!( "Hello pe {:?} of {:?}, I'm pe {:?}", lamellar::currentpe, lamellar::numpes, self.my_pe ); } }
fn main(){ let mut world = lamellar::LamellarWorldBuilder::new().build(); let mype = world.mype(); let numpes = world.numpes(); let am = HelloWorld { mype: mype }; for pe in 0..numpes{ world.execampe(pe,am.clone()); // explicitly launch on each PE } world.waitall(); // wait for all active messages to finish world.barrier(); // synchronize with other pes let handle = world.execall(am.clone()); //also possible to execute on every PE with a single call handle.get(); //both execall and execampe return request handles that can be used to access any returned result } ```
Optional: Lamellar requires the following dependencies if wanting to run in a distributed HPC environment: the rofi lamellae is enabled by adding "enable-rofi" to features either in cargo.toml or the command line when building. i.e. cargo build --features enable-rofi
To enable support for serializable remote closures compile with the nightly compiler and specify the "nightly" feature i.e. cargo build --features nightly
At the time of release, Lamellar has been tested with the following external packages:
| GCC | CLANG | ROFI | OFI | IB VERBS | MPI | SLURM | |--------:|----------:|----------:|----------:|--------------:|--------------:|----------:| | 7.1.0 | 8.0.1 | 0.1.0 | 1.9.0 | 1.13 | mvapich2/2.3a | 17.02.7 |
The OFIDIR environment variable must be specified with the location of the OFI installation. The ROFIDIR environment variable must be specified with the location of the ROFI installation.
In the following, assume a root directory ${ROOT}
0. download Lamellar to ${ROOT}/lamellar-runtime
cd ${ROOT} && git clone https://github.com/pnnl/lamellar-runtime
download rofi-sys to ${ROOT}/rofi-sys -- or update Cargo.toml to point to the proper location
cd ${ROOT} && git clone https://github.com/pnnl/rofi-sys
Select Lamellae to use
In Cargo.toml add "enable-rofi" feature in wanting to use rofi, otherwise local lamellae will be used it may also be necessary to adjust the symmetric heap size (const MEMSIZE) in rofilamellae.rs on the available memory in your system
Compile Lamellar lib and test executable (feature flags can be passed to command line instead of specifying in cargo.toml)
cargo build (--release) (--features enable-rofi) (--features nightly)
executables located at ./target/debug(release)/test
cargo build --examples (--release) (--features enable-rofi) (--features nightly)
executables located at ./target/debug(release)/examples/
Note: we do an explicit build instead of `cargo run --examples` as they are intended to run in a distriubted envrionment (see TEST section below.)
The examples are designed to be run with on at least two compute nodes, but they will work on a single node using the "local" lamellae. Here is a simple proceedure to run the tests that assume a compute cluster and SLURM job manager. Please, refer to the job manager documentaiton for details on how to run command on different clusters. Lamellar grabs job information (size, distribution, etc.) from the jbo manager and runtime launcher (e.g., MPI, please refer to the BUILING REQUIREMENTS section for a list of tested software versions).
salloc -N 2 -p partition_name
mpiexec launcher.mpiexec -n 2 ./target/release/test
runs a simple acitve message based bandwidth test
mpiexec -n 2 ./target/release/examples/{example}
where <test> in {all_to_all, array_put, array_static, array, get_bw, put_bw, hello, return}.
or alternatively:
srun -N 2 -p partition_name -mpi=pmi2 ./target/release/examples/{example}
where <test> in {all_to_all, array_put, array_static, array, get_bw, put_bw, hello, return, dft_proxy}
Finally, the number of worker threads used within lamellar is controlled by setting and environment variable: LAMELLAR_THREADS
e.g. export LAMELLAR_THREADS=10
Note, if running on a single node, simple execute the binaries directly, no need to use mpiexec or srun.
Lamellar is still under development, thus not all intended features are yet implemented.
Ryan Friese - ryan.friese@pnnl.gov
Roberto Gioiosa - roberto.gioiosa@pnnl.gov
Mark Raugas - mark.raugas@pnnl.gov
This project is licensed under the BSD License - see the LICENSE.md file for details.
This work was supported by the High Performance Data Analytics (HPDA) Program at Pacific Northwest National Laboratory (PNNL), a multi-program DOE laboratory operated by Battelle.