Rust bindings for ONNX Runtime

Coverage Results GitHub Workflow Status Crates.io
Crates.io ONNX Runtime

ort is an (unofficial) ONNX Runtime 1.15 wrapper for Rust based on the now inactive onnxruntime-rs. ONNX Runtime accelerates ML inference on both CPU & GPU.

See the docs for more detailed information and the examples. If you have any questions, feel free to ask in the #πŸ’¬ο½œort-discussions and related channels in the pyke Discord server or in GitHub Discussions.

Feature comparison

| Feature comparison | πŸ“• ort | πŸ“— ors | πŸͺŸ onnxruntime-rs | |------------------------|-----------|-----------|----------------------| | Upstream version | v1.15.1 | v1.12.0 | v1.8 | | dlopen()? | βœ… | βœ… | ❌ | | Execution providers? | βœ… | ❌ | ❌ | | IOBinding? | βœ… | ❌ | ❌ | | String tensors? | βœ… | ❌ | ⚠️ input only | | Multiple output types? | βœ… | βœ… | ❌ | | Multiple input types? | βœ… | βœ… | ❌ | | In-memory session? | βœ… | βœ… | βœ… |

Cargo features

Note: For developers using ort in a library (if you are developing an app, you can skip this part), it is heavily recommended to use default-features = false to avoid bringing in unnecessary bloat. Cargo features are additive. Users of a library that requires ort with default features enabled will not be able to remove those features, and if the library isn't using them, it's just adding unnecessary bloat and inflating compile times. Instead, you should enable ort's default features in your dev dependencies only. Disabling default features will disable download-binaries, so you should instruct downstream users to include ort = { version = "...", features = [ "download-binaries" ] } in their dependencies if they need it.

How to get binaries

You can use either the 'traditional' way, involving a strategy, or the new (and preferred) way, using load-dynamic.

Strategies

There are 2 'strategies' for obtaining and linking ONNX Runtime binaries. The strategy can be set with the ORT_STRATEGY environment variable. - download (default): Downloads prebuilt ONNX Runtime from Microsoft. Only a few execution providers are available for download at the moment, namely CUDA and TensorRT. These binaries may collect telemetry. In the future, pyke may provide binaries with telemetry disabled and more execution providers available. - system: Links to ONNX Runtime binaries provided by the system or a path pointed to by the ORT_LIB_LOCATION environment variable. ort will automatically link to static or dynamic libraries depending on what is available in the ORT_LIB_LOCATION folder.

Execution providers

To use other execution providers, you must explicitly enable them via their Cargo features, listed below. Some EPs are not currently implemented due to a lack of hardware for testing; please open an issue if your desired EP has a ⚠️

Note that the download strategy only provides some execution providers, namely CUDA and TensorRT for Windows & Linux. You'll need to compile ONNX Runtime from source and use the system strategy to point to the compiled binaries to enable other execution providers.

Execution providers will attempt to be registered in the order they are passed, silently falling back to the CPU provider if none of the requested providers are available. If you must know whether an EP is available, you can use ExecutionProvider::cuda().is_available().

For prebuilt Microsoft binaries, you can enable the CUDA or TensorRT execution providers for Windows and Linux via the cuda and tensorrt Cargo features respectively. Microsoft does not provide prebuilt binaries for other execution providers, and thus enabling other EP features will fail when ORT_STRATEGY=download. To use other execution providers, you must build ONNX Runtime from source.

Projects using ort ❀️

open a PR to add your project here 🌟

FAQ

I'm using a non-CPU execution provider, but it's still using the CPU!

ort is designed to fail gracefully when an execution provider is not available. It logs failure events through tracing, thus you'll need a library that subscribes to tracing events to see the logs. The simplest way to do this is to use tracing-subscriber.

Add tracing-subscriber to your Cargo.toml: toml [dependencies] tracing-subscriber = { version = "0.3", features = [ "env-filter", "fmt" ] }

In your main function: rs fn main() { tracing_subscriber::fmt::init(); }

Set the environment variable RUST_LOG to ort=debug to see all debug messages from ort; this will look like: - Windows (PowerShell): $env:RUST_LOG = 'ort=debug'; cargo run - Windows (Command Prompt): use PowerShell ;) - macOS & Linux: RUST_LOG="ort=debug" cargo run

My app exits with "status code 0xc000007b" without logging anything!

You probably need to copy the ONNX Runtime DLLs to the same path as the executable. - If you are running a binary (cargo run), copy them to e.g. target/debug - If you are running an example (cargo run --example xyz), copy them to e.g. target/debug/examples - If you are running tests (cargo test), copy them to e.g. target/debug/deps

Alternatively, you can use the load-dynamic feature to avoid this.

"thread 'main' panicked at 'assertion failed: (left != right)"

Most of the time this is because Windows ships its own (typically older) version of ONNX Runtime. Make sure you've copied the ONNX Runtime DLLs to the same folder as the exe.

Shared library hell

If using shared libraries (as is the default with ORT_STRATEGY=download), you may need to make some changes to avoid issues with library paths and load orders, or preferably use the load-dynamic feature to avoid all of this.

Windows

Some versions of Windows come bundled with an older vesrion of onnxruntime.dll in the System32 folder, which will cause an assertion error at runtime: plaintext The given version [14] is not supported, only version 1 to 13 is supported in this build. thread 'main' panicked at 'assertion failed: `(left != right)` left: `0x0`, right: `0x0`', src\lib.rs:114:5 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The fix is to copy the ONNX Runtime DLLs into the same directory as the binary, since DLLS in the same folder as the main executable resolves before system DLLs. ort can automatically copy the DLLs to the Cargo target folder with the copy-dylibs feature, though this fix only works for binary Cargo targets (cargo run). When running tests/benchmarks/examples for the first time, you'll have to manually copy the target/debug/onnxruntime*.dll files to target/debug/deps/ for tests & benchmarks or target/debug/examples/ for examples.

Linux

Running a binary via cargo run should work without copy-dylibs. If you'd like to use the produced binaries outside of Cargo, you'll either have to copy libonnxruntime.so to a known lib location (e.g. /usr/lib) or enable rpath to load libraries from the same folder as the binary and place libonnxruntime.so alongside your binary.

In Cargo.toml: ```toml [profile.dev] rpath = true

[profile.release] rpath = true

do this for all profiles

```

In .cargo/config.toml: ```toml [target.x86_64-unknown-linux-gnu] rustflags = [ "-Clink-args=-Wl,-rpath,\$ORIGIN" ]

do this for all Linux targets as well

```

macOS

macOS has the same limitations as Linux. If enabling rpath, note that the rpath should point to @loader_path rather than $ORIGIN:

```toml

.cargo/config.toml

[target.x8664-apple-darwin] rustflags = [ "-Clink-args=-Wl,-rpath,@loaderpath" ] ```