Open-source accounting for personal and professional finances

Performance boost 🚀

Performance took a hit after implementing the balance assertions feature. After parsing the .journal, we then need to sort transactions/postings by date and sum each account/commodity pair in order to check the balance assertions.

Looking at the code, there were some simple optimisations we could make. Namely, reducing allocations.

By changing .into_iter() to .iter() and borrowing values instead of .clone(), we clawed back some of that lost performance.

There are plenty of other opportunities to improve performance further in the future: experiment with parallel processing (probably using rayon), using more optimised data structures, and reducing yet more allocations.

Improvements

• 65% speed up!

Date Range Filters

You can now add date range filters before the report commands.

-b, --begin <DATE>  include postings/txns on or after this date
-e, --end <DATE>    include postings/txns before this date

For example, to see your income statement for the UK 22/23 tax year:

rledger -f 2023.journal -b 2022-04-06 -e 2023-04-06 incomestatement

Balance assertions and assignments

Postings can contain "balance assertions" and "balance assignments".

2023-05-04 opening balances
    assets:bank  = $500 ; <- this is an "assignment"
    equity:opening/closing balances           

2023-05-05
    assets:bank  $1000 = $1500 ; <- this is an "assertion"
    income:work  $-1000

Assertions

After reading a journal file, rledger will check all balance assertions and report an error if any of them fail. Balance assertions can protect you from e.g. inadvertently disrupting reconciled balances while cleaning up old entries.

Assignments

These are like balance assertions, but with no posting amount on the left side of the equals sign; instead it is calculated automatically so as to satisfy the assertion. This can be a convenience during data entry e.g. when setting opening balances.

Improvements

• Assertions provide powerful error checking when adding new transactions.
• Assignments are a convenient way to declare opening balances or asset valuations for things like stock or mutual fund investments.

Example assertion error:

More directive support

In the .journal file, you can declare "directives" that influence how the journal is processed. We have already implemented the include directive (for including other .journal files in the main journal).

This update adds the account and commodity directives, which are useful for error checking. For example, checking that the accounts and commodities in the journal are expected (protect against typos).

Improvements

• Add the commodity directive
• Add the account directive

At this point in time, they have no impact on journal parsing. But we will use them when we implement strict checking later on.

Currency support

The commodity for a given amount controls how the amount will be displayed. I introduced the rusty_money crate to handle the formatting.

Improvements

• Amounts with a currency iso code will be displayed in the expected format for that currency.
• For convenience, the following common currency symbols will be interpreted without the need for the iso code:

"$" => USD
"£" => GBP
"€" => EUR
"¥" => JPY

include support

Within a journal, you can now include other journal files:

include 2022.journal

2023-01-20 Client A | Invoice #1
    assets:receivables      $10,000.00
    revenue:clients:A      -$10,000.00

Improvements

• This feature allows you to break up large journal files and organise them how you like. A typical approach is to create yearly journal files e.g. 2022.journal, and then include all the years in a all-years.journal.
• File paths are relative to the currently opened journal

Balance Report

The balance command produces a foundational report for listing account balances, balance changes, values, value changes and more, during one time period or many. Generally it shows a table, with rows representing accounts, and columns representing periods.

The higher level reports such as the balance sheet and income statement are special cases of the balance report.

Improvements

• This change also brought a small performance boost to the parser.

Benchmarks

On an AMD Ryzen 9 5950X 16-Core Processor, I am getting 275,549 transactions per second 🚀.

By comparison, hledger is running at 13,982 per second.

rledger is currently ~19x faster than hledger 🤯⚡.

Run benchmarks using criterion with cargo bench. This will show how performance has changed between runs.

Ad hoc tests can be taken with hyperfine:

cargo build --release 
hyperfine './target/release/rledger -f benches/10000x1000x10.journal bs'

This approach is good for comparing performance with hledger (how the results above were found):

hyperfine -w 5 './target/release/rledger -f benches/10000x1000x10.journal bs' 'hledger -f benches/10000x1000x10.journal bs'

Integration testing

Since this is a plain text accounting CLI, we can make use of snapshot testing. The idea is that we run a CLI command given some input and compare the output with our expected result.

I had rolled my own testing framework using nom to parse the test files. However, a colleague of mine, Austin McBee, tipped me on to trycmd.

This crate provides a harness for test case files and asserting stdout/stderr are expected.

Improvements

• Unit tests can get verbose and snapshot testing concisely covers testing the CLI -> parsing journal -> producing a report or error.
• trycmd allows for tests to be written in markdown files. This makes tests also serve as documentation and reduces the number of test files.
• Works with pretty errors from miette

Reading from stdin

So far, rledger has only been able to read the journal from a file. But it can be convenient to read from stdin, allowing clean integration with other unix tools like cat, grep etc.

Use the argument -f - to read from stdin:

cat examples/example.journal | rledger -f - bs

Automatic account recognition

In plain text accounting, you can use any account names you like. However, you normally want use the traditional accounting categories: assets, liabilities, equity, revenues, expenses.

For more precise reporting, we can divide the top level accounts into more detailed subaccounts, by writing a full colon between account name parts:

assets
assets:bank
assets:bank:current
expenses
expenses:groceries

Improvements

• Account names that start with one of the above are automatically recognized
• The matching is case insensitive
• High-level reports, such as the balance sheet and income statement, are built correctly when these accounts are used.

Balance sheet and Income statement

Using the tabled crate to produce these pretty tables.

Improvements

• Add balance sheet report.
• Add income statement.
• When amounts are negative, colour them in red.
• When the CLI is run outside of a tty, the colour is not added. This allows for reports to be sent to another process, e.g. writing to a file, without the ANSI escape codes.

Experiments with query engines

Now that rledger can parse a basic journal, we need a way to query the transactions and postings. My instinct is to use SQL. I considered using SQLite but wanted to explore the data frame libraries in Rust.

I tried using pola.rs. This crate has a very flexible API that seemed ideal for my querying needs. In addition, polars would provide excellent performance.

However, after experimenting with calculations needed for producing the balance sheet, I decided to remove polars due to unbearably long compile times. I will revisit using the crate in the future in case there are ways to improve dev-cycle compile times. Perhaps once features are more stable, longer compile times will become more tolerable?

Next, I tried datafusion. I immediately ran into build/linking failures. Instead of debugging those, I discounted using datafusion as it needs an async runtime (tokio), which I don't plan to use just yet (perhaps a web UI in the future will need this though).

I then tried using duckdb. This provides a nice SQL interface and is fast at runtime. However, it also increases compile times.

So I finally settled on writing my own query engine. I started by using built-in data structures such as HashMap and BTreeMap. This worked well and was fast, but I hadn't found a nice abstraction. The implementation was also low-level.

The current solution uses the itertools crate, which provides high level iterator methods like group_by. This solution is both ergonomic, fast and seems to have a negligible impact on compile times 🚀.

Fancy errors

Using the miette crate to produce these pretty diagnostics. Coupled with the error tree from nom_supreme yields one of the nicest error handling experiences.

Improvements

• Parse errors (e.g. bad date, missing posting indent etc.) are reported to the user
• Error message shows the problematic file, line number and column.

Amount parsing

Amounts can contain commas, spaces, + and - signs, and commodity symbols. For example:

$ +1 000 001
-$ 500,000.50
$ -500,000.50
$ 500000.50

A single posting amount is also allowed to be left blank. rledger will infer the amount in order to make the transaction balance.

Improvements

• Successfully parse the different amount formats
• Allow a single amount to be left blank for balance inference

Add CLI

Added a CLI using the clap crate. Using the derive method to build the CLI. It's incredibly clean:

/// Fast, accurate, robust plain text accounting
#[derive(Parser, Debug)]
#[command(author, version, about, long_about = None)]
struct Args {
    /// Name of the entry journal file. For stdin, use -
    #[arg(short, long)]
    file: PathBuf,

    #[command(subcommand)]
    commands: Commands,
}

#[derive(Subcommand, Debug)]
enum Commands {
    /// Show assets, liabilities and net worth
    #[command(visible_alias = "bs")]
    Balancesheet,

    /// Show revenues and expenses
    #[command(visible_alias = "is")]
    Incomestatement,
}

This produces a simple hledger-inspired interface for querying your ledger:

Fast, accurate, robust plain text accounting

Usage: rledger --file <FILE> <COMMAND>

Commands:
  balancesheet     Show assets, liabilities and net worth [aliases: bs]
  incomestatement  Show revenues and expenses [aliases: is]
  help             Print this message or the help of the given subcommand(s)

Options:
  -f, --file <FILE>  Name of the entry journal file. For stdin, use -
  -h, --help         Print help
  -V, --version      Print version

Support for comments in the journal

The .journal format can contain line comments and multiple line comments. This commit adds support for all types.

# My comment 
2023.02.01 GOODWORKS CORP
    # There can be multiple comments
    ; A comment using a semicolon
    assets:bank:checking           $1000
    income:salary                 $-1000

Add nom_supreme

One issue faced when using the looping nom parsers (e.g. many0, separated_list etc.) can be that parsing errors within the loop aren't raised correctly. To fix this, I added nom_supreme and used collect_separated_terminated. How this solves the problem (from the docs):

By requiring a terminator, we can ensure that they don’t suffer from the normal folding parser problem of unconditionally returning success because a subparser failure is interpreted as the end of the loop. This ensures that potentially important errors aren’t thrown away.

Improvements

• Errors are surfaced correctly
• nom_supreme provides an ErrorTree, which can be used for detailed error handling of all the branches the parser takes.
• nom_supreme adds a trait that makes available many common nom parser combinators as postfix methods e.g. space0.precedes(tag("i")).

Commit v0.1.0

Initial commit! Working on parsing the journal format following the spec documented in the hledger manual.

I'm using nom combinators to implement the parser.

I assessed pest for implementing the parser, but I already had some experience with nom from previous projects.

In addition, nom produces native speeds for the parser.

Example journal:

2023-01-01 opening balances as of this date ; This is a transaction comment
    assets:bank:current                $500,100.23 
    assets:bank:savings                 $25000 
    assets:cash                          $100
    liabilities:creditcard               $-50
    equity:opening/closing balances  ; One amount can be left blank; $-525150.23 is inferred here.

2023.02.01 GOODWORKS CORP
    # There can be multiple comments
    ; A comment using a semicolon
    assets:bank:checking           $1000
    income:salary                 $-1000