txtpp

A simple-to-use general purpose preprocessor for text files, written in rust.

You can: - Include another file in the current file, much like C-style #include - Execute a command in the middle of the current file and include the output

You can use txtpp both as a command line tool, or as a library with the rust crate. txtpp is well tested with unit tests and integration tests.

The full API doc is available on docs.rs

Currently in Preview. Not all features are implemented and some examples may not work.

Installation

Install with cargo cargo install txtpp txtpp --help

Examples

Include a file

Say you have 2 files foo.txt.txtpp and bar.txt ``` (foo.txt.txtpp) (bar.txt)

1 |hello 1 |bar 2 |-TXTPP#include bar.txt 2 | 3 |world 4 | ```

Running txtpp foo.txt will produce foo.txt: ``` (foo.txt)

1 |hello 2 |bar 3 |world 4 | `` Ifbar.txt.txtppalso exists, it will be preprocessed first to producebar.txt`, and the result will be used.

Execute a command

Say you have the file foo.txt.txtpp: ``` (foo.txt.txtpp)

1 |hello
2 |-TXTPP#run cat foo.txt.txtpp
3 |world 4 | Running `txtpp foo.txt` will produce `foo.txt`: (foo.txt)

1 |hello 2 |hello 3 |-TXTPP#run cat foo.txt.txtpp 3 |world 4 |world 5 | ```

More examples can be found in the examples directory. These are also used as integration tests.

Table of Contents

• Feature Summary
• Directive Overview
  • Syntax
  • Execution
• Directive Specification
  • Include Directive
  • Run Directive
  • Temp Directive
  • Tag Directive
  • Write Directive
• Output Specification

Feature Summary

txtpp provides directives that you can use in the .txtpp files. A directive replaces itself with the output of the directive. The directives are all prefixed with TXTPP#: - include - Include the content of another file. - run - Run a command and include the output of the command. - temp - Store text into a temporary file next to the input file. - tag - Hold the output of the next directive until a tag is seen, and replace the tag with the output. - write - Write content to the output file. Can be used for escaping directives.

Directive Overview

Syntax

A directive is a single- or multi-line structure in the source file, that looks like this: ``` {WHITESPACES}{PREFIX1}TXTPP#{DIRECTIVE} {ARG1} {WHITESPACES}{PREFIX2}{ARG2} {WHITESPACES}{PREFIX2}{ARG3} ...

``` Explanation:

1. First line:
   • {WHITESPACES}: Any number of whitespace characters
   • {PREFIX1}: Non-empty text that does not start with a whitespace character, and does not include TXTPP#
     • The prefix must be non-empty for multi-line directives, otherwise you won't be able to terminate it.
   • TXTPP#: the prefix before the directive
   • {DIRECTIVE}: can be one of the directives
   • : (space) At least one space between the directive name and its input. This will be trimmed.
   • {ARG1}: argument as one string, until the end of the line. Both leading and trailing whitespaces (including the new line) will be trimmed.
2. Subsequent lines: Some directives are allowed to have more than one lines (see the specification for what they are)
   • {WHITESPACES}: this must match exactly with {WHITESPACES} in the first line
   • {PREFIX2}: this must be one of:
     • the same number of space characters (i.e ' ') as the length of {PREFIX1} in the first line
     • the exact same string as the {PREFIX1} in the first line
     • the same string as the {PREFIX1} in the first line without trailing whitespaces, followed by new line (i.e. arg is empty)
   • {ARG2}, {ARG3} ...: Add more arguments to the argument list. Note that {WHITESPACES} and {PREFIX2} are not included in the argument. Unlike the first line, leading whitespaces are not trimmed, but trailing whitespaces are still trimmed.
3. Ending: if a line does not match the format above, the directive ends. The ending line won't be part of this directive, but can be the start of the next directive.

For example, you can write the directive like ``` // TXTPP#run echo "hello world"

``` which will be treated like a comment in most languages to help with syntax highlighting.

The same example as a block comment /* TXTPP#run echo " hello world " -TXTPP# */ This will execute the command echo "hello world". The - in the last line in front of TXTPP# is needed to indicate that it's the start of a different directive.

Execution

The directives are executed immediately after they are parsed. They may produce an output to be included in the output file and/or have side effects such as creating a temporary file.

If the directive has output (like include and run), it will be formated as: - Every line in the output will be prepended with {WHITESPACES}, so that the indentation is consistent Directive: 1 | // TXTPP#run echo 1; echo 2 Output: 1 | 1 2 | 2 3 | - The line endings will be normalized to be the same as the output file. Whether the last line has a trailing newline is persisted from the output of the command/included file. If the output from the directive doesn't have a newline character in the end, the next line from the source file will be on the same line as the last line of the directive output. Directive: 1 | // TXTPP#run echo -n hello 2 |world 3 | Output: 1 | helloworld 2 |

Note that normally, you will not be able to connect a directive output to the previous line, since directives always start on its own line. However, you can use tag (see below) to achieve this. If there is currently an active tag directive that is listening for output, the output will be sent to the tag instead of the output file, without the indentation. and the directive will produce no output.

Directive Specification

This section contains detailed specification of each directive.

Include Directive

USAGE

This directive is used include the content of another file into the current file.

ARGUMENTS

Single-line only. The argument is FILE_PATH

BEHAVIOR

• If FILE_PATH is an absolute path, it will be used as is. Otherwise, it should be relative to the (directory of) the current file.
• If FILE_PATH does not end in .txtpp, and FILE_PATH.txtpp exists, FILE_PATH.txtpp will be preprocessed first to produce FILE_PATH, and the result will be used as the output. Note that you would still include FILE_PATH, not FILE_PATH.txtpp.

  EXAMPLE

TXTPP#include foo.txt

Run Directive

USAGE

This directive is used to run a command and include the output of the command into the current file.

ARGUMENTS

Can have more than one line. The arguments are joined with a single space in between to form the COMMAND

BEHAVIOR

• The COMMAND will be executed as a sub-process.
• Default shell selection:
  • Windows: the following is tried in order:
  • (PowerShell 7) pwsh -NonInteractive -NoProfile -Command COMMAND
  • (PowerShell 5/Windows PowerShell) powershell -NonInteractive -NoProfile -Command COMMAND
  • (Command Prompt) cmd /C COMMAND
  • Other OS: sh -c COMMAND
  • You can override this with the --shell option.
• The working directory of the sub-process will be the directory of the current file.
• The sub-process will inherit the environment variables of the main process, with additional environment variables:
  • TXTPP_FILE: the path to the current file being processed.
  • (that's the only environment variable for now)

    EXAMPLE

TXTPP#run echo "hello world"

CAVEATS

1. txtpp will not run as a subcommand to avoid processing loops. You shouldn't need to run txtpp inside txtpp anyway.

Empty directive

USAGE

Empty directive has the empty string as the name and does nothing. It can be used to remove lines from the input.

ARGUMENTS

Can have more than one line. All arguments to the empty directive will be ignored.

BEHAVIOR

Nothing

EXAMPLE

For example, you can use it to terminate a block comment javascript function hello() { // GENERATED CODE /* TXTPP#run ./codegen -arg -really_long_arg --really-really-long-option -TXTPP# */ }

If you have to put the end of the block comment in a new line, make sure to format it correctly so it is treated as part of the directive. javascript function hello() { // GENERATED CODE /* TXTPP#run ./codegen -arg -really_long_arg --really-really-long-option -TXTPP# -*/ } In both scenarios, the entire block comment /**/ will be replaced with the output from running ./codegen

Temp Directive

USAGE

This directive is used to create a temporary file.

ARGUMENTS

Must have at least 1 argument. The first argument specifies the FILE_PATH to save the output (relative to the current file). The rest of the arguments are joined by line endings to form the CONTENT, with a trailing line ending.

BEHAVIOR

• FILE_PATH is resolved the same way as the include directive
• FILE_PATH cannot end in .txtpp. It will cause an error.
  • This is to avoid undefined behavior as the preprocessor may or may not pick up the file generated by the temp directive.
• CONTENT will be saved to FILE_PATH
• FILE_PATH will not be deleted after processing, but will be deleted with clean.

  EXAMPLE

```javascript function getcities() { // GENERATED return [ // TXTPP#temp printcities.py // import csv // with open("cities.csv") as f: // reader = csv.reader(f) // for (name, population) in reader: // print(f"{{name: "{name}", population: "{population}"}},")

// TXTPP#run python print_cities.py

]; } ```

Tag Directive

USAGE

This directive is used to create a tag to store the next directive's output.

ARGUMENTS

Single-line only. The argument is TAG

BEHAVIOR

• The lifecycle of a tag is as follows:
  1. A tag is created with the tag directive and will listen for the next directive's output.
  2. If the next directive has no output (e.g. temp), the tag will continue to listen for the next directive's output.
  3. When there is a directive that has output, the output will be stored in the tag
  4. When there is a non-directive line that has the tag, the tag will be replaced with the stored output.
  5. After the tag is replaced, it will be deleted.
• There can only be one tag at a time to store the output of the next directive. However, multiple tags can be in the "stored" state.
• None of the tags can be prefix of another tag. For example, if you have a tag Foo, you cannot have another tag FooBar, vice versa.
• Multiple tags can be injected into the same line of the source file. However, if the output of a tag contains another tag, it will be not replaced. The tags are replaced from left to right. If the tags overlap, the leftmost will be replaced.
• Only the first occurrence of a tag will be replaced.
• Each tag must be deleted before another tag with the same name can be created.
• The newlines in the output will be replaced by the line endings of the current file. Whether the output has a trailing newline or not will not be changed.
  • Example: if the stored output has no trailing newline, the part after the tag will be on the same line as the last line in the stored output.
• If there are any unused tags in the end of the file, there will be an error.

EXAMPLE

In this example, we want the output to be exactly as is because of the <pre> tag. The output of the run directive will be put in the <pre> tag. ```html

PRE_CONTENT -->

The following is invalid because the tag is used before the output is stored. html

PRE_CONTENT -->

```

Write Directive

USAGE

This directive writes its arguments to the output file. It can be used to escape other directives.

ARGUMENTS

Can have more than one line. Each argument is one line in the output

BEHAVIOR

• The arguments will be written to the output file as is.
• Because the content is treated as output of a directive, tags won't be injected.

  EXAMPLE

1 |-TXTPP#write the line below will be written to the output file as is 2 |-TXTPP#run echo "hello world" 3 |stuff 4 | Output 1 |the line below will be written to the output file as is 2 |TXTPP#run echo "hello world"stuff 3 | (To put stuff on its own line, add an extra argument to the write directive)

Output Specification

This section specifies details of the output of the preprocessor.

Line endings

The output files and temporary output files will have consistent line ending with the input .txtpp files. If the input file has mixed line endings, the output file will have the same line endings as the first line in the input file.

If the input file does not have a line ending, the output file will have the same line ending as the operating system (i.e. \r\n on Windows, \n on Unix).

The output files will have a trailing newline unless --no-trailing-newline is specified. The flag will not affect the temporary output files, however. Whether a temporary file has a trailing newline depends on if the directive has an empty line in the end.