A versatile tool for downloading the linked contents of entire subreddits fast and efficiently.
Note that this is not the tool for the job if you want to retrieve information about the individual reddit posts.
bypass reddit API limitation of 1000 posts
update local copy
banned subreddits work
fast (because of an asynchronous job queue, powered by the fantastic [tokio])
custom title formatting
(for the Rust programmers) #![forbid(unsafe_code)]
Prebuilt binaries can be found in the [Releases] tab.
If you want a locally compiled version, you can use cargo:
Install Rust according to https://rustup.rs. The default configuration is sufficient.
$ cargo install redditrip
The usage can also be acquired by running redditrip --help.
The base command is redditrip [SUBREDDITS]....
The following arguments can optionally be used:
--update/-u: Stop at the first already existing file.
--force/-f: Force downloads from unsupported domains by simpling writing whatever is on the page to disk.
--after <date>: Only download posts after this date.
--before <date>: Only download posts before this date.
--selfposts/-s: Download self posts as text files.
--exclude <domain>/-e <domain>: Prevents downloading from a domain.
--title <formatter>: Use a custom title format.
There are a couple of more advanced options described in the --help output.
It is recommended to use -q/--quiet to see only the individual errors.
You should also use a high --batch-size, and subsequently a high ulimit -n (open files) because every download job takes >= 1 open file descriptor.
Finally, if you expect to run into a lot of unsupported sites, which can directly be saved, use --force.
redditrip supports custom titles. To use this feature, a formatting string must be provided with --title <formatter>.
Placeholders are the field names enclosed in curly brackets. For example: --title "{id}-{author}_{title}".
The file extension is always appended to the title.
The available fields can be queried by running the program with --formatting-fields.
They correspond to data from the Pushshift API.
An rough overview can be seen on https://api.pushshift.io/reddit/search/submission?size=1, though not all fields can be present.
Characters which are not allowed in file names are replaced with _.
Because reddit titles alone can be longer than the maximum file name length on many systems, one should know about --max-file-name-length <length>,
which is used to truncate file names.
Not using the {id} placeholder can lead to file name collisions, thus it is advised to always include it somewhere in the formatter,
preferably at the front so it is not at the risk of being truncated.
The most useful placeholders are:
| Placeholder | Type | Purpose |
| :---------: | ---- | ------- |
| {id} | string | The post ID |
| {title} | string | The post title |
| {author} | string | The post author name |
| {created_utc} | integer | The UNIX timestamp when the post was created |
| {link_flair_text} | string | The text of the post flair |
| {author_flair_text} | string | The text of the author flair |
| {domain} | string | The domain of the link the post points to |
| {over_18} | bool | Whether the post is NSFW |
It's quite easy to compile the master branch yourself.
Make sure that Rust and cargo are installed like described above.
Then:
$ git clone https://github.com/Draphar/redditrip
$ cd redditrip
Then this is sufficient:
$ cargo build --release
# The binary is `target/release/redditrip`
If you, for whatever reason, want a highly optimized build:
$ cargo rustc --release -- -C lto -C codegen-units=1 -C target_cpu=native
$ strip target/release/redditrip
implement a map of already downloaded links, and symlink instead of redownloading
filter posts
Copyright 2020 Joshua Prieth
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.