This is jja, a command line utility to interact with various chess file formats. It is still in its early stages of development. The initial intention of the author was to convert their opening books which were saved with ChessBase's proprietary CTG format to the free and open PolyGlot format. Overtime they intend to add support for other chess file formats ( cbh, epd, pgn, si4, si5 and so on).
As of version 0.6.1, jja supports reading/querying:
binabkctgobk
(version 1 and 2, w\o,with text notes)expopening book files, whereas it supports writing/converting to:
opening book files.
As of version 0.5.0, jja supports exporting all the supported opening book
formats to PGN. To
use this functionality, specify an output file with pgn extension as an argument
to jja edit.
During opening book conversion, jja uses the information provided in various
input opening book formats to come up with a move weight which accompanies the
move in the PolyGlot opening file.
jja also writes some custom numbers in the learn field, such as
NAGs during ctg
conversion or priority during abk conversion. You may disable this custom
usage using --no-learn as it may confuse other software making use of this field.
Note, Arena, aka abk, opening book file
writing support is only supported from
ChessBase, aka ctg books. Use the command
line flags --author, --comment, --probability-priority, --probability-games,
--probability-win-percent to configure ABK
header metadata. Game statistics (minimum number of games/wins, win percentages for
both sides) are managed automatically by jja.
In-place editing for Arena opening books is
also possible using -i, --in-place=SUFFIX command line option. Conversion from
PolyGlot, aka bin, and
ChessMaster, aka obk opening books
to Arena, aka abk opening book files is
planned for a future release.
Since version 0.4.0, jja can make
PolyGlot books out of
PGN
files. This feature is similar to polyglot make-book with the following
differences:
--no-scale to disable.The filter expression string should contain filter conditions, which consist of a
tag name, a comparison operator, and a value. The following operators are supported:
- > (greater than)
- >= (greater than or equal to)
- < (less than)
- <= (less than or equal to)
- = (equal to)
- != (not equal to)
- =~ (regex match, case insensitive)
- !~ (negated regex match, case insensitive)
Filter conditions can be combined using the following logical operators:
- AND (logical AND)
- OR (logical OR)
Example:
--filter="Event =~ World AND White =~ Carlsen AND ( Result = 1-0 OR ECO = B33 )"
Supported tags are Event, Site, Date, UTCDate, Round, Black, White, Result, BlackElo, WhiteElo, BlackRatingDiff, WhiteRatingDiff, BlackTitle, WhiteTitle, ECO, Opening, TimeControl, Termination, and ScidFlags.
In addition to these are four special variables, namely, Player, Elo, Title, and RatingDiff. These variables may be used to match the relevant header from either one of the sides. E.g the filter:
--filter="Player =~ Carlsen"
is functionally equivalent to
--filter="( White =~ Carlsen OR Black =~ Carlsen )"
Note: The filtering is designed to be simple and fast. The tokens, including parantheses are split by whitespace. Quoting values is not allowed. For more sophisticated filtering needs, use pgn-extract.
Scid uses one character flags, DWBMENPTKQ!?U123456, for each field where:
- D - Deleted
- W - White opening
- B - Black opening
- M - Middlegame
- E - Endgame
- N - Novelty
- P - Pawn structure
- T - Tactics
- K - Kingside play
- Q - Queenside play
- ! - Brilliancy
- ? - Blunder
- U - User-defined
- 1..6 - Custom flags
It is ill-advised to rely on the order of the characters flags.
Use a regex match if/when you can.
--max-open-files=-1 to keep
files always open.4, are
aimed at speedy conversion with relatively moderate space usage. If you run out
of space during conversion, try to use an algorithm like
Zstd with an "ultra" level, ie. a level
greater or equal to 20.--filter=<expr>,
--max-ply=<ply>, --min-pieces=<piece-count>, --only-white, and
--only-black.To compile from source, use cargo install jja. This requires the Rust
Toolchain to be installed. In addition you are going to need
OpenSSL libraries on
UNIX systems. Moreover you need
liburing on Linux. If
you're on a Linux system older than
5.1 or you are unable to install
liburing for another reason, you may disable
the feature by building jja with cargo install jja --no-default-features.
As an alternative, release builds of jja are hosted on chesswob.org for 64-bit Linux and Windows. These versions are signed by GnuPG, using key D076A377FB27DE70. To install, acquire the latest version from chesswob.org, verify the checksum and the GnuPG signature:
$> export JJA_VERSION=0.5.0
$> export JJA_FLAVOUR=glibc
$> curl https://keybase.io/alip/pgp_keys.asc | gpg --import
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 13292 100 13292 0 0 13535 0 --:--:-- --:--:-- --:--:-- 26584
gpg: key D076A377FB27DE70: public key "Ali Polatel (Caissa AI) <alip@caissa.ai>" imported
gpg: Total number processed: 1
gpg: imported: 1
$> for f in jja-${JJA_VERSION}-${JJA_FLAVOUR}.bin{,.sha512sum,.sha512sum.asc}; do wget -q https://chesswob.org/jja/${f}; done
$> gpg --verify jja-${JJA_VERSION}.bin.sha512sum.asc jja-${JJA_VERSION}.bin.sha512sum
gpg: Signature made Sun Mar 19 20:52:41 2023 CET
gpg: using RSA key 5DF763560390A149AC6C14C7D076A377FB27DE70
gpg: Good signature from "Ali Polatel (Caissa AI) ...
$> sha512sum -c jja-${JJA_VERSION}.bin.sha512sum
jja: OK
$> sudo install -m755 jja-${JJA_VERSION}-${JJA_FLAVOUR}.bin /usr/local/bin
Finally you may download the builds of the latest git version via the
SourceHut build server. There're three flavours,
windows, linux-glibc, and linux-musl. Simply browse to the latest build and
download the artifact listed on the left. Note: these artifacts are kept for 90
days.
jja info to get brief information about the chess file.jja find to search for a position in a chess file.jja edit to edit opening book files and convert to
PolyGlot files or
Arena files.jja make to compile
PGN files into
PolyGlot opening books.jja dump to dump a PolyGlot
or BrainLearn file as a stream of
JSON arrays.jja restore to restore a PolyGlot
or BrainLearn file from a stream of
JSON arrays.jja merge to merge two PolyGlot
opening books.jja match to arrange book matches using random playouts.jja play to make random playouts, optionally using books.jja hash to calculate Zobrist hash of a given chess position.jja open to browse ECO
classification.jja quote to print a chess quote.jja determines the type of the file using its file extension. Files with the
extension .bin are considered PolyGlot
books. Files with the extension .ctg are considered
ChessBase books. Files with the extension
.abk are considered Arena books. Files
with extension .obk are considered
ChessMaster books. Files with
extension .exp are considered BrainLearn
experience files.
By default if the standard output is a
TTY, jja will display
information using fancy tables. Use --porcelain command line option to get the
output in CSV
(comma-separated values) format instead.
Thanks to Steinar H. Gunderson, for publishing the CTG
Specification,
and authoring the remoteglot tool:
The CTG probing code in jja is very
directly ported from their
C probing code,
and the specification has been an enormous help in clearing up various rough edges.
Thanks to Fabien Letouzey, the author of the original
PolyGlot software: The
PolyGlot probing, book making and merging
code in jja is mostly ported from their respective
C code. Thanks to
Michel Van den Bergh, the author of pg_utils, a collection of tools to
interact with PolyGlot opening books: The
PolyGlot book editing code of jja
uses many ideas and code excerpts from pg_utils. Thanks to Peter Österlund,
the author of DroidFish: The
ABK opening book interface code in jja
makes use of ideas and code excerpts from
DroidFish. Thanks to Jens
Nissen, the author of ChessX: The
CTG and
ABK probing codes in jja use ideas and
code excerpts from ChessX. Thanks to
LiChess, the best chess website on the planet. The quote
command of jja has a selection of quotes imported from the
LiChess codebase. Thanks to Shane Hudson, the author of
Scid vs. PC: The jja eco command uses the
ECO classification which has been done by
the Scid project. In addition, the
PolyGlot editing code of jja uses
ideas and code from Scid. Thanks to Marcus
Bufett, the author of
chess-tactics-cli: The
chessboard displaying code in PolyGlot and
ABK edit screens is borrowed from
chess-tactics-cli.
jja is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.
Hey you, out there beyond the wall,
Breaking bottles in the hall,
Can you help me?
Report bugs to jja's bug tracker at https://todo.sr.ht/~alip/jja/: 1. Always be polite, respectful, and kind: https://css-tricks.com/open-source-etiquette-guidebook/ 2. Keep your final change as small and neat as possible: https://tirania.org/blog/archive/2010/Dec-31.html 3. Attaching poems with the bug report encourages consideration tremendously.
I've started hacking this on International Women's Day 2023, a day to honor the achievements of women and advocate for their rights worldwide. As a person of Kurdish heritage, I am particularly moved by the slogan "Woman, Life, Freedom", which has become a symbol of resistance against oppression and a call for equality. In the spirit of free software and free speech, I strive to contribute to the creation of a more just and inclusive society, where every human being is granted the freedom to express themselves and pursue their dreams. I also honor the memory of Mahsa Amini, whose tragic death reminds us of the urgent need to fight for women's freedom and safety.
More on Wikipedia, WikiPedia::Woman,Life,Freedom
jja restore which incorrectly wrote BrainLearn entries in big-endian rather
than little-endian.jja find tree output.rust-embed crate from 6.7 to 6.8.num_cpus crate from 1.15 to 1.16.-S, --stockfish to generate Stockfish compatible Zobrist hashes.jja::stockfish, and new public function
jja::stockfish::stockfish_hash to generate Stockfish compatible Zobrist hashes.
Before calling this function jja::stokfish::zobrist::init function must be
called once to compute hashtables used in Zobrist hashing.jja::chess::{de,}serialize_chess to serialize/deserialize a
shakmaty::Chess instance to/from an array of 5 unsigned 64-bit numbers. jja
dump uses this functionality in PGN dumps.jja::pgnfilt::Operator and jja::pgnfilt::LogicalOperator implements Eq as
well as PartialEq now.restore command to accompany the dump command which restores JSON
serialized PolyGlot or BrainLearn file entries into the given output file.SIGPIPE handling on UNIX systems so that jja does not panic when the
output is piped to another program such as a pager.-z <HASH>, --hash=<HASH> to query PolyGlot opening books and
BrainLearn experience files by Zobrist hash.info,
dump, and find are able to handle files in BrainLearn experience file format
with the extension .exp.polyglotbook::PolyGlotBook::read_book_key is
used for that.dump command to dump the full contents of a PolyGlot opening book or a PGN
file. The dump format of the PolyGlot opening book is JSON, whereas for PGN files
this is CSV.polyglotbook::PolyglotBook::lookup_moves has been changed
to take a zobrist hash of a chess position as an argument rather than the
shakmaty::Chess position itself.--signed to print Zobrist hashes as signed decimal numbers.jja::file which exports utilities for binary file i/o.jja::polyglot::entry_{from,to}_file have been renamed to
jja::polyglot::bin_entry_{from,to}_file.XorShift random number
generator to randomly pick moves during book matches using the match command. This
algorithm is cryptographically insecure but is very fast.--version output.memmap.book member of PolyGlotBook is now a BufReader<File> rather than a File
which is a breaking change.jja::hash::zobrist_hash function mistakenly checked
for full legality.ctgbook::CtgTree which holds the new return value
of the functions CtgBook::extract_all, and CtgBook::extract_all2. The tree
element of CtgEntry which were used by these functions has also been dropped and
the functions have been implemented in a much more performant way using
considerably less memory. As a result, most ctg to abk/polyglot conversions are
almost double as fast.ctgbook::CtgEntry member uci's type has been changed from
String to shakmaty::uci::Uci, and the nags member has been renamed to nag
and its type has been changed from Option<String> to Option<Nag>.ctgbook::CtgEntry no longer has a comment member which is a
breaking change.ctg::colored_uci function now accepts a
shakmaty::uci::Uci rather than a UCI string. The order of the function arguments
is also changed.Ctg::Nag to abstract CTG NAG (Numeric Annotation Glyph) entries1 during PolyGlot
conversion to prevent skipping these entries. We plan to make this
user-configurable in the future.IsTerminal trait, and drop the dependency on is_terminal
crateto_le_bytes(), to_be_bytes(), and drop the dependency on
byteorder crate-e=<MODE>, --enpassant-mode=<MODE> to select en-passant mode when
to include the en-passant square in Zobrist hash calculation.polyglot.from_uci now expects a bool argument
to correctly encode castling positions by determining whether the king to move is
on their starting square.1.64 to 1.70 due to shakmaty bumpchrono cratePolyGlotBook has two new public functions: find_book_key, and
read_book_entry-x, --hex to print hash as hexadecimal rather than decimalpgn-reader crate from 0.24 to 0.25shakmaty crate from 0.25 to 0.26ctg opening booksCtgBook::num_entries has been renamed to total_positions.CtgBook::total_pages function accepts a reference to
self, rather than consuming self. The return type is usize now.close functions of CtgBook and PolyGlotBook,
improve CtgBook to close the cto file immediately after open, and keep a File,
rather than an Option<File> in CtgBook. The now unused path element of
CtgBook is also dropped.--help output1.64 as determined by cargo-msrv--rescale for PolyGlot books. When specified, edit will rescale
weights of all entries in the book, rather than a single entry. This is useful to
quickly correct/optimize PolyGlot books which were generated without weight scaling.i18n feature disabledtempfile crate from 3.5 to 3.6rust-embed crate from 6.6 to 6.7once_cell crate from 1.17 to 1.18ctrlc crate from 3.3 to 3.4-H, --hashcode argument to skip duplicate games based on the
HashCode PGN tag. PGN files may be tagged using pgn-extract --addhashcode--version, prefer git version over package version
for git buildsclap crate from 4.2 to 4.3ctrlc crate from 3.2 to 3.3rocksdb crate to 0.21.0 which bundles RocksDB-8.1.1#[deny(missing_docs)]pgn
extension to export an opening book to a PGN-l <max-ply>, --line=<max-ply> to display lines from the opening
book as a table of opening variations reverse-sorted by cumulative weight--min-score now accepts floating point values as argument rather than an
unsigned 64-bit integer--win-factor, --draw-factor, and --loss-factor to specify
respective factors during score calculation, the defaults, 2, 1, and 0
respectively, resembles the original polyglot tool--tree=<max-ply> no longer panics on broken pipe, so it's more convenient
to use with a pager-p, --min-pieces to specify the minimum number of pieces on the
board for a position to be included in the book, defaults to 8-c, --weight-cutoff to specify the minimum weight of entries to be
included in the bookavg and wavg to merge using average
weight or weighted average weight respectively; the weighted averages should be
specified for wavg using -w, --weight1, and -W, --weight2max, min, ours, and sum, default is
sum which adds together move weights, max picks the one with the maximum
weight, min picks up the one with the minimum weight, and ours always picks
the entries from the first book-i, --irreversible to prefer irreversible moves during random
playouts. Pawn moves, captures, moves that destroy castling rights, and moves that
cede en-passant are irreversible.-B <games>, --batch-size=<games> to determine write batch size in
number of games--no-default-features on installation.ScidFlags, a set of character flags used by the
Scid software, which may be used in filter
expressions--debug flag to print information on matching filter expressions--max-open-files=<int> to specify the maximum number of open files
per thread opened by the temporary RocksDB database.-T=<threads>, --threads=<threads> or the
JJA_NPROC environment variable to overridePlayer, Elo, Title, and RatingDiff
to match the relevant field from either colour--filter=<expression> to filter PGN games by headers. The filtering
is designed to be simple and fast. The tokens, including parantheses are split by
whitespace. Quoting variables is not allowed. See jja make --help for more
information on Filter Expressions-t <max-ply>, --tree=<max-ply> to display lines from the opening
book as a tree using the nice termtree crate--pgn
commandline option-0, --null--no-scale--compression={none,bzip2,lz4,lz4hc,snappy,zlib,zstd},--compression-level=<level>,
defaults to lz4, level 4-T, --threads
to override, works transparently with compressed PGN files (zstd, bzip2, gunzip, lz4)