html5gum is a WHATWG-compliant HTML tokenizer.
```rust use std::fmt::Write; use html5gum::{Tokenizer, Token};
let html = "
html5gum fully implements 13.2.5 of the WHATWG HTML
spec, i.e. is able to tokenize HTML documents and passes html5lib's tokenizer
test suite. Since it is just a tokenizer, this means:
html5gum does not implement charset
detection.
This implementation takes and returns bytes, but assumes UTF-8. It recovers
gracefully from invalid UTF-8.html5gum does not correct mis-nested
tags.html5gum does not recognize implicitly self-closing elements like
<img>, as a tokenizer it will simply emit a start token. It does however
emit a self-closing tag for <img .. />.html5gum doesn't implement the DOM, and unfortunately in the HTML spec,
constructing the DOM ("tree construction") influences how tokenization is
done. For an example of which problems this causes see this example
code.html5gum does not generally qualify as a browser-grade HTML parser as
per the WHATWG spec. This can change in the future, see issue
21.With those caveats in mind, html5gum can pretty much ~parse~ tokenize
anything that browsers can.
Emitter traitA distinguishing feature of html5gum is that you can bring your own token
datastructure and hook into token creation by implementing the Emitter trait.
This allows you to:
Rewrite all per-HTML-tag allocations to use a custom allocator or datastructure.
Efficiently filter out uninteresting categories data without ever allocating for it. For example if any plaintext between tokens is not of interest to you, you can implement the respective trait methods as noop and therefore avoid any overhead creating plaintext tokens.
See the custom_emitter example for how this
looks like in practice.
jetscii, and can be disabled via crate features (see Cargo.toml)html5gum was created out of a need to parse HTML tag soup efficiently. Previous options were to:
use quick-xml or
xmlparser with some hacks to make
either one not choke on bad HTML. For some (rather large) set of HTML input
this works well (particularly quick-xml can be configured to be very
lenient about parsing errors) and parsing speed is stellar. But neither can
parse all HTML.
For my own usecase html5gum is about 2x slower than quick-xml.
use html5ever's own
tokenizer
to avoid as much tree-building overhead as possible. This was functional but
had poor performance for my own usecase (10-15x slower than quick-xml).
use lol-html, which would probably
perform at least as well as html5gum, but comes with a closure-based API
that I didn't manage to get working for my usecase.
Why is this library called html5gum?
G.U.M: Giant Unreadable Match-statement
\
Licensed under the MIT license, see ./LICENSE.