As Rust is becoming a popular programming language for its performance and type safety, the desire to use it with still a maniac time-series database kdb+ is brewing. The aspiration is understandable since we know kdb+ is fast and its interface or a shared library should be fast as well. This interface was created to satisfy such a natural demand, furthermore, in a manner users do not feel any pain to use. The notrious esoteric function names of the q/kdb+ C API is not an interest of Rust developers.
"Give us a Rust interface!!"
Here is your choice.
This interface provides two features:
You can find the detail descriptions of each feature below.
As Rust was conceived to address type unsafety of C/C++, replacing C/C++ with Rust can happen if possible. This interface is purposed to be used as a Rust client of q/kdb+ process that sends a query and receives its response. Query to kdb+ is supported in two ways:
Compression/decompression of messages is also implemented following kdb+ implementation.
As for connect method, usually client interfaces of q/kdb+ do not provide a listener due to its protocol. However, sometimes Rust process is connecting to an upstream and q/kdb+ starts afterward or is restarted more frequently. Then providing a listener method is a natural direction and it was achieved here. Following ways are supported to connect to kdb+:
Furthermore, in order to improve inter-operatability some casting, getter and setter methods are provided.
This crate uses q-native or crate-specific environmental variables.
KDBPLUS_ACCOUNT_FILE: A file path to a credential file which an acceptor loads in order to manage access from a q client. This file contains a user name and SHA-1 hashed password in each line which are delimited by ':' without any space. For example, a file containing two credentials "mattew:oracle" and "reluctant:slowday" looks like this:
mattew:431364b6450fc47ccdbf6a2205dfdb1baeb79412 reluctant:d03f5cc1cdb11a77410ee34e26ca1102e67a893c
The hashed password can be generated with q using a function .Q.sha1:
q).Q.sha1 "slowday" 0xd03f5cc1cdb11a77410ee34e26ca1102e67a893c
KDBPLUS_TLS_KEY_FILE and KDBPLUS_TLS_KEY_FILE_SECRET: The pkcs12 file and its password which TLS acceptor uses.
QUDSPATH (optional): q-native environmental variable to define an astract namespace. This environmental variable is used by UDS acceptor too. The abstract nameapace will be @${QUDSPATH}/kx.[server process port] if this environmental variable is defined; otherwise it will be @/tmp/kx.[server process port].Notes:
KX_SSL_CERT_FILE and KX_SSL_KEY_FILE on q side to make q/kdb+ to work as a TLS server. For details, see the KX website.All types are expressed as K struct which is quite similar to the K struct of api module but its structure is optimized for IPC
usage and for the convenience to interact with. The table below shows the input types of each q type which is used to construct K object.
Note that the input type can be different from the inner type. For example, timestamp has an input type of chrono::DateTime<Utc> but
the inner type is i64 denoting an elapsed time in nanoseconds since 2000.01.01D00:00:00.
| q | Rust |
|------------------|---------------------------------------------------|
| bool | bool |
| GUID | [u8; 16] |
| byte | u8 |
| short | i16 |
| int | i32 |
| long | i64 |
| real | f32 |
| float | f64 |
| char | char |
| symbol | String |
| timestamp | chrono::DateTime<Utc> |
| month | chrono::Date<Utc> |
| date | chrono::Date<Utc> |
| datetime | chrono::DateTime<Utc> |
| timespan | chrono::Duration |
| minute | chrono::Duration |
| second | chrono::Duration |
| time | chrono::Duration |
| list | Vec<Item> (Item is a corrsponding type above) |
| compound list | Vec<K> |
| table | Vec<K> |
| dictionary | Vec<K> |
| null | () |
```rust use kdbplus::ipc::*;
async fn main() -> Result<()>{
// Connect to qprocess running on localhost:5000 via UDS let mut socket=QStream::connect(ConnectionMethod::UDS, "", 5000u16, "ideal:person").await?; println!("Connection type: {}", socket.getconnection_type());
// Set remote function with asynchronous message socket.sendasyncmessage(&"collatz:{[n] seq:enlist n; while[not n = 1; seq,: n:$[n mod 2; 1 + 3 * n; `long$n % 2]]; seq}").await?;
// Send a query synchronously let mut result=socket.sendsyncmessage(&"collatz[12]").await?; println!("collatz[12]: {}", result);
result=socket.sendsyncmessage(&"collatz[a]").await?;
println!("collatz[a]: {}", result);
// Send a functional query. let mut message=K::newcompoundlist(vec![K::newsymbol(String::from("collatz")), K::newlong(100)]); result=socket.sendsyncmessage(&message).await?; println!("collatz[100]: {}", result);
// Modify query to (`collatz; 20) message.pop().unwrap(); message.push(&K::newlong(20)).unwrap(); result=socket.sendsync_message(&message).await?; println!("collatz[20]: {}", result);
// Send a functional asynchronous query. message=K::newcompoundlist(vec![K::newstring(String::from("show"), qattribute::NONE), K::newsymbol(String::from("goodbye"))]); socket.sendasyncmessage(&message).await?;
socket.shutdown().await?;
Ok(()) } ```
```rust use kdbplus::ipc::*;
async fn main() -> Result<()>{
// Start listenening over TCP at the port 7000 with authentication enabled. let mut socket_tcp=QStream::accept(ConnectionMethod::TCP, "127.0.0.1", 7000).await?;
// Send a query with the socket. let greeting=sockettcp.sendsync_message(&"string `Hello").await?; println!("Greeting: {}", greeting);
socket_tcp.shutdown().await?;
Ok(()) } ```
Then q client can connect to this acceptor with the acceptor's host, port and the credential configured in KDBPLUS_ACCOUNT_FILE:
q
q)h:hopen `::7000:reluctant:slowday
Use kdbplus as a library name in Cargo.toml with "ipc" feature.
toml
[dependencies]
kdbplus={version="^0.3", features=["ipc"]}
Programming language q (kdb+ is a database written in q) is providing only C API but sometimes an external library provides Rust interface but not C/C++ interface. From the fame of its performance, Rust still should be feasible to build a shared library for kdb+. This library is provided to address such a natural demand (desire, if you will). Since there is no way for everyone but creating a wrapper like this to write a shared library for kdb+, it probably make sense for someone to provide the wrapper, and it was done here.
In order to avoid writing too large unsafe block which leads to poor optimization, most of native C API functions are provided with a wrapper funtion with a bit of ergonomic safety and with intuitive implementation as a trait method. The only exceptions are knk and k which are using elipsis (...) as its argument. These functions are provided under native namespace with the other C API functions.
Note: This library is purposed to be used to build a sared library; therefore some unrelated functions are removed. For example, connection functions to kdb+ like khpu are not included.
Use kdbplus as a library name in Cargo.toml with "api" feature.
toml
[dependencies]
kdbplus={version="^0.3", features=["api"]}
The examples of using C API wrapper are included in api_examples folder. The examples are mirroring the examples in the document of kdbplus::api module and the functions are also used for simple tests of the library. The test is conducted in the test.q under tests/ by loading the functions defined in a shared library built from the examples.
Here are some examples:
```rust use kdbplus::qtype; use kdbplus::api::; use kdbplus::api::native::;
pub extern "C" fn createsymbollist(: K) -> K{ unsafe{ let mut list=ktn(qtype::SYMBOLLIST as i32, 0); js(&mut list, ss(strtoS!("Abraham"))); js(&mut list, ss(strtoS!("Isaac"))); js(&mut list, ss(strtoS!("Jacob"))); js(&mut list, sn(strtoS!("Josephine"), 6)); list } }
pub extern "C" fn catchy(func: K, args: K) -> K{ unsafe{ let result=ee(dot(func, args)); if (result).qtype == qtype::ERROR{ println!("error: {}", S_to_str((result).value.symbol)); // Decrement reference count of the error object r0(result); KNULL } else{ result } } }
pub extern "C" fn dictionarylisttotable() -> K{
unsafe{
let dicts=knk(3);
let dictsslice=dicts.asmutslice::();
keysslice[0]=ss(strtoS!("a"));
keysslice[1]=ss(strtoS!("b"));
let values=ktn(qtype::INTLIST as i32, 2);
values.asmutslice::()[0..2].copyfromslice(&[i*10, i*100]);
dictsslice[i as usize]=xD(keys, values);
}
// Format list of dictionary as a table.
// ([] a: 0 10 20i; b: 0 100 200i)
k(0, strtoS!("{[dicts] -1 _ dicts, (::)}"), dicts, KNULL)
}
}
```
q can use these functions like this:
``q
q)summon:libcapiexamples 2: (create_symbol_list; 1)
q)summon[]
AbrahamIsaacJacobJoseph
q)AbrahamIsaacJacobJoseph ~ summon[]
q)catchy:libcapiexamples 2: (catchy; 2);
q)catchy[$; ("J"; "42")]
42
q)catchy[+; (1;a)]
error: type
q)behold: libc_api_examples 2: (dictionarylistto_table; 1);
q)behold[]
0 0
10 100
20 200
```
The examples below are written without unsafe code. You can see how comfortably breathing are the wrapped functions in the code.
```rust use kdbplus::qtype; use kdbplus::api::; use kdbplus::api::native::;
pub extern "C" fn createsymbollist2(: K) -> K{ let mut list=newlist(qtype::SYMBOLLIST, 0); list.pushsymbol("Abraham").unwrap(); list.pushsymbol("Isaac").unwrap(); list.pushsymbol("Jacob").unwrap(); list.pushsymboln("Josephine", 6).unwrap(); list }
fn nopanick(func: K, args: K) -> K{ let result=errortostring(apply(func, args)); if let Ok(error) = result.geterrorstring(){ println!("FYI: {}", error); // Decrement reference count of the error object which is no longer used. decrementreference_count(result); KNULL } else{ println!("success!"); result } }
pub extern "C" fn createtable2(: K) -> K{
// Build keys
let keys=newlist(qtype::SYMBOLLIST, 2);
let keysslice=keys.asmutslice::();
keysslice[0]=enumerate(strtoS!("time"));
keysslice[1]=enumeraten(strtoS!("temperatureandhumidity"), 11);
// Build values
let values=newlist(qtype::COMPOUNDLIST, 2);
let time=newlist(qtype::TIMESTAMPLIST, 3);
// 2003.10.10D02:24:19.167018272 2006.05.24D06:16:49.419710368 2008.08.12D23:12:24.018691392
time.asmutslice::
flip(new_dictionary(keys, values)) } ```
And q code is here:
``q
q)summon:libcapiexamples 2: (create_symbol_list2; 1)
q)summon[]
AbrahamIsaacJacobJoseph
q)chill:libcapiexamples 2: (no_panick; 2);
q)chill[$; ("J"; "42")]
success!
42
q)chill[+; (1;a)]
FYI: type
q)climatechange: libcapiexamples 2: (`createtable2; 1);
q)climate_change[]
2003.10.10D02:24:19.167018272 22.1
2006.05.24D06:16:49.419710368 24.7
2008.08.12D23:12:24.018691392 30.5
```
Test is conducted in two ways:
Before starting the test, you need to start a q process on the port 5000:
bash
kdbplus]$ q -p 5000
q)
Then fire the cargo test:
bash
kdbplus]$ cargo test
Note: Currently 20 tests fails for api examples in document. This is because the examples do not have main function by nature of api but still use #[macro_use].
Tests are conducted with tests/test.q by loading the example functions built in api_examples.
bash
kdbplus]$ cargo build
kdbplus]$ cp target/debug/libapi_examples.so tests/
kdbplus]$ cd tests
tests]$ q test.q
Initialized something, probably it is your mindset.
bool: true
bool: false
byte: 0xc4
GUID: 8c6b-8b-64-68-156084
short: 10
int: 42
int: 122
int: 7336
int: 723
int: 14240
int: 2056636
long: -109210
long: 43200123456789
long: -325389000000021
long: 0
real: 193810.31
float: -37017.09330000
float: 742.41927468
char: "k"
symbol: `locust
string: "gnat"
string: "grasshopper"
error: type
What do you see, son of man?: a basket of summer fruit
What do you see, son of man?: boiling pot, facing away from the north
symbol: `rust
success!
FYI: type
this is KNULL
Planet { name: "earth", population: 7500000000, water: true }
Planet { name: "earth", population: 7500000000, water: true }
おいしい！
おいしい！
おいしい！
おいしい！
おいしい！
おいしい！
おいしい！
おいしい！
おいしい！
おいしい！
"Collect the clutter of apples!"
test result: ok. 147 passed; 0 failed
q)What are the three largest elements?: `belief`love`hope
The document of this crate itself is on the crates.io page.
For details of C API itself, check the documents of KX website.