Provides a read_utf8 function for all types implementing
BufRead, allowing to read text file without worrying about
loading huge files without newline delimiters.
Add this crate as a dependency in your Cargo.toml:
toml
[dependencies]
utf8-bufread = "0.1.4"
This will allow you to use the function read_utf8 on any object
implementing std::io::BufRead. This function essentially reads a
stream and push the data to a String:
```rust use utf8_bufread::BufRead; use std::io::BufReader;
fn main() { // Reader may be any type implementing io::BufRead // We'll just use a BufReader wrapping a slice for this // example let mut reader = BufReader::<&[u8]>::new("💖".asref()); // The string we'll use to store the text of the read file let mut text = String::new(); loop { // Loop until EOF match reader.readutf8(&mut text) { Ok(0) => break, // EOF Ok() => continue, Err(e) => panic!(e), // io::Error or Utf8Error } } asserteq!("💖", text.as_str()); } ```
A common issue encountered when using the standard Rust library to read large
files of text is that these may have extremely long lines or no newline
delimiters at all. This makes BufReader::read_line or
BufReader::lines load a large amount of data into memory, which may
not be desirable.
The function read_utf8, on the other hand, will only read up until the
reader's buffer is full.
If valid utf-8 codepoint is read it will always be pushed. If an invalid
or incomplete codepoint is read, the function will first push all the valid
bytes read and an InvalidData error will be returned on the
next call:
```rust use utf8_bufreader::BufRead; use std::io::{BufReader, ErrorKind};
fn main() { use utf8_bufread::BufRead; use std::io::{BufReader, ErrorKind};
// We give the buffer more than enough capacity to be // able to read all the bytes in one call let mut reader = BufReader::withcapacity( 16, [ // "foo\nbar" + some invalid bytes 0x66u8, 0x6f, 0x6f, 0xa, 0x62, 0x61, 0x72, 0x9f, 0x92, 0x96 ].asref(), ); let mut buf = String::new();
// On the first readutf8() call, we will read up to the // first byte of the invalid codepoint (ie "foo\nbar") let nread = reader .readutf8(&mut buf) .expect("We will get all the valid bytes"); asserteq!("foo\nbar", buf.asstr()); asserteq!(7, n_read);
// Then on the second call we will get the InvalidData // error caused by the Utf8Error error, as there is no // bytes forming valid codepoints left let readerr = reader.readutf8(&mut buf) .expecterr("We will get an error"); asserteq!(ErrorKind::InvalidData, readerr.kind()); asserteq!(7, buf.len()); // no byte appended to buf } ```
This crate is fairly new, and for now only provides the read_utf8 function,
with a rather simple implementation. In the near future these features should
be added:
read_utf8 (see
from_utf8_lossy &
from_utf8_unchecked).chars iterator from the buffer, and its lossy version.This also means it may have a pretty unstable API
I am also looking for a way for read_utf8 to return a &str instead of a
String, meaning the reader is borrowed until the returned reference goes out
of scope, so that I let the user choose if they want to clone the data or not.
For the moment, the read codepoints are always cloned into a new String.
Finally, I want to test and benchmark this crate.
Given I'm not the most experience developer at all, you are very welcome to submit push requests here
Utf8-BufRead is distributed under the terms of the Apache License 2.0, see the LICENSE file in the root directory of this repository.