uSiem SQlite store

uSiem component that stores events in a sqlite database.

Real numbers

Log indexing is fast enough for most use cases, being around 5000 logs/sec in debug mode. A total of 1 million logs with 52 columns with an index per column has a size of 293.3 MB.

Indexing example

SELECT event_created, event_received,vendor, product, service, category,tenant,tags,origin,`host.hostname`, message, `source.ip`, `user.domain`, `user.name`, `event.outcome` FROM log_table ORDER BY event_created DESC LIMIT 10;

|eventcreated|eventreceived|vendor |product |service|category |tenant |tags|origin |host.hostname|message |source.ip |user.domain|user.name|event.outcome| |-------------|--------------|------------|--------------|-------|--------------|-------|----|-------|-------------|---------------------------------------------------------------------------------------------------------|-----------|-----------|---------|-------------| |1637535361471|1637535361471 |MagicDevices|MagicDevice001|sshd |Authentication|Default|{} |0.0.0.0|hostname1 |This is a log example ..............111111111111111111111111222222222222222222222223333333333333333333333|10.10.10.10|CNMS |cancamusa|FAIL | |1637535361471|1637535361471 |MagicDevices|MagicDevice001|sshd |Authentication|Default|{} |0.0.0.0|hostname1 |This is a log example ..............111111111111111111111111222222222222222222222223333333333333333333333|10.10.10.10|CNMS |cancamusa|FAIL | |1637535361471|1637535361471 |MagicDevices|MagicDevice001|sshd |Authentication|Default|{} |0.0.0.0|hostname1 |This is a log example ..............111111111111111111111111222222222222222222222223333333333333333333333|10.10.10.10|CNMS |cancamusa|FAIL | |1637535361471|1637535361471 |MagicDevices|MagicDevice001|sshd |Authentication|Default|{} |0.0.0.0|hostname1 |This is a log example ..............111111111111111111111111222222222222222222222223333333333333333333333|10.10.10.10|CNMS |cancamusa|FAIL | |1637535361471|1637535361471 |MagicDevices|MagicDevice001|sshd |Authentication|Default|{} |0.0.0.0|hostname1 |This is a log example ..............111111111111111111111111222222222222222222222223333333333333333333333|10.10.10.10|CNMS |cancamusa|FAIL | |1637535361471|1637535361471 |MagicDevices|MagicDevice001|sshd |Authentication|Default|{} |0.0.0.0|hostname1 |This is a log example ..............111111111111111111111111222222222222222222222223333333333333333333333|10.10.10.10|CNMS |cancamusa|FAIL | |1637535361471|1637535361471 |MagicDevices|MagicDevice001|sshd |Authentication|Default|{} |0.0.0.0|hostname1 |This is a log example ..............111111111111111111111111222222222222222222222223333333333333333333333|10.10.10.10|CNMS |cancamusa|FAIL | |1637535361470|1637535361470 |MagicDevices|MagicDevice001|sshd |Authentication|Default|{} |0.0.0.0|hostname1 |This is a log example ..............111111111111111111111111222222222222222222222223333333333333333333333|10.10.10.10|CNMS |cancamusa|FAIL | |1637535361470|1637535361470 |MagicDevices|MagicDevice001|sshd |Authentication|Default|{} |0.0.0.0|hostname1 |This is a log example ..............111111111111111111111111222222222222222222222223333333333333333333333|10.10.10.10|CNMS |cancamusa|FAIL | |1637535361470|1637535361470 |MagicDevices|MagicDevice001|sshd |Authentication|Default|{} |0.0.0.0|hostname1 |This is a log example ..............111111111111111111111111222222222222222222222223333333333333333333333|10.10.10.10|CNMS |cancamusa|FAIL |

Example indexing logs

```rust let mut comp = SqliteDatastore::new( getdefaultschema(), "./storagedb".tostring(), 20000, 5000, ); let localchan = comp.localchannel(); let (localchnllogsnd, localchnllogrcv) = crossbeamchannel::bounded(1000); comp.setlogchannel(localchnllogsnd.clone(), localchnllog_rcv.clone());

std::thread::spawn(move || comp.run());

for _ in 1..100000 { let mut log = SiemLog::new(String::from("This is a log example ..............111111111111111111111111222222222222222222222223333333333333333333333"), chrono::Utc::now().timestampmillis(), SiemIp::V4(0)); log.setcategory(Cow::Borrowed("Authentication")); log.setproduct(Cow::Borrowed("MagicDevice001")); log.settenant(Cow::Borrowed("Default")); log.setservice(Cow::Borrowed("sshd")); log.setvendor(Cow::Borrowed("MagicDevices")); log.setevent(SiemEvent::Auth(AuthEvent { hostname: Cow::Borrowed("hostname1"), outcome: LoginOutcome::FAIL, logintype: AuthLoginType::Remote(RemoteLogin { domain: Cow::Borrowed("CNMS"), sourceaddress: Cow::Borrowed("10.10.10.10"), username: Cow::Borrowed("cancamusa"), }), })); let _ = localchnllogsnd.send(log); } // Stop the component std::thread::sleep(std::time::Duration::fromsecs(10)); let _ = localchan.send(SiemMessage::Command( 1, 1, SiemFunctionCall::STOPCOMPONENT(Cow::Borrowed("Stop!!")), )); ```