pantry.rs

Rust library for the pantry llm runner.

Read the documentation at docs.rs

Usage

The API connects to your local pantry installation, so you need to download pantry first. Client library for the Pantry LLM API.

Example

It's strongly recommended that you use PantryClient and LLMSession, which are a higher level wrapper around PantryAPI.

``` rust let perms = UserPermissions { permsuperuser: false, permloadllm: false, permunloadllm: false, permdownloadllm: false, permsession: true, //this is for createsession AND promptsession permrequestdownload: true, permrequestload: true, permrequestunload: true, permviewllms: true, };

let pantry = PantryClient::register("my project name".into(), perms).await.unwrap();

// Pause here and use the UI to accept the permission request.

// The empty hashmap means we just use default parameters. // createsession just uses the best currently running LLM. use createsessionid or _flex for // more finegrained control let sess = pantry.createsession(HashMap::new()).await.unwrap();

let recv = ses.prompt_session("About me: ".into(), HashMap::new()).await.unwrap(); ```

If you aren't already running an LLM from the ui, you can use rust pantry.load_llm_flex(None, None).await.unwrap();

If you want to use your existing ggml infrastructure, you can get a bare model path

rust let (model, path) = pantry.bare_model_flex(None, None).await.unwrap();