The commitlog so far assumed that the latest segment is never compressed
and can be opened for writing (if it is intact).
However, restoring the entire commitlog from cold storage results in all
segments being compressed. Make it so the resumption logic reads the
metadata from the potentially compressed last segment, and starts a new
segment for writing if the latest one was indeed compressed.
# Expected complexity level and risk
1.5
# Testing
Added a test.
# Description of Changes
It'd be best to review this commit-by-commit, and using
[difftastic](https://difftastic.wilfred.me.uk) to easily tell when
changes are minor in terms of syntax but a line based diff doesn't show
that.
# Expected complexity level and risk
3 - edition2024 does bring changes to drop order, which could cause
issues with locks, but I looked through [all of the warnings that
weren't fixed
automatically](ba80f3fecd/warnings.html)
and couldn't find any issues.
# Testing
n/a; internal code change
Changes the commitlog (and durability) write API, such that the caller
decides how many transactions are in a single commit, and has to supply
the transaction offsets.
This simplifies commitlog-side buffering logic to essentially a
`BufWriter` (which, of course, we must not forget to flush). This will
help throughput, but offers less opportunity to retry failed writes.
This is probably a good thing, as disks can fail in erratic ways, and we
should rather crash and re-verify the commitlog (suffix) than continue
writing.
To that end, this patch liberally raises panics when there is a chance
that internal state could be "poisoned" by partial writes, which may be
debatable.
# Motivation
The main motivation is to avoid maintaining the transaction offset in
two places in such a way that they could diverge. As ordering commits is
the responsibility of the datastore, we make it authoritative on this
matter -- the commitlog will still check that offsets are contiguous,
and refuse to commit if that's not the case.
A secondary, related motivation is the following:
A "commit" is an atomic unit of storage, meaning that a torn (partial)
write of a commit will render the entire commit corrupt. There hasn't
been a compelling case where we would want this, and have always
configured the server to write exactly one transaction per commit.
The code to handle buffering of transactions is, however, rather
complex, as it tries hard to allow the caller to retry writes at commit
boundaries. An unfortunate consequence of this is that we'd flush to the
OS very often, leaving throughput performance on the table.
So, if there is a compelling case for batching multiple transactions in
a commit, it should be the datastore's responsibility.
# API and ABI breaking changes
Breaks internal APIs only.
# Expected complexity level and risk
5 - Mostly for the risk
# Testing
Existing tests.
When a new commitlog segment is created, allocate disk space for it up
to the maximum segment size. Also do this when resuming writes to an
existing segment, such that segments created without preallocation will
allocate as well when the database is opened.
Preallocation is gated behind the feature "fallocate", because it is not
always desirable to preallocate, e.g. for local `standalone` users.
The feature can only be enabled on Linux targets, because allocation is
done using the Linux-specific `fallocate(2)` system call.
Unlike `ftruncate(2)` or the portable `posix_fallocate(3)`,
`fallocate(2)`
supports allocating disk space without zeroing. This is currently
required, because the commitlog format does not handle padding bytes.
If not enough space can be allocated, the commitlog refuses writes. For
commitlogs that were created without preallocation, this means that the
commitlog cannot even be opened in this situation.
The local durability impl will crash if it detects that the commitlog is
unable to allocate enough space.
This means that a database will eventually crash and be unable to start
in
an out-of-space situation.
Allocated space is not included in the reported size of the commitlog.
Instead, allocated blocks are reported separately.
# Expected complexity level and risk
3 - Disk size monitoring may need to be adjusted.
# Testing
- [x] Adds a test that demonstrates the crash behavior of
[`spacetimedb_durability::Local`]
when there is insufficient space. The test performs I/O against a loop
device.
- [x] Modified the `repo::Memory` impl so that it can run out of space.
No test currently
utilizes this, but existing tests assuming infinite space still pass.
# Description of Changes
We've run into a problem on Maincloud caused by a database that was
writing a relatively small number of very large transactions. This was
accruing many commitlog segments consuming hundreds of gigabytes of
disk, but had not ever taken a snapshot, or compressed or archived any
data, as the database had not progressed past one million transactions.
With this PR, we take a snapshot every time the commitlog segment
rotates. We still also snapshot every million transactions.
One BitCraft database we looked at had 2.5 million transactions per
commitlog segment, meaning that this change will not meaningfully affect
the frequency of snapshots. The offending Maincloud database, however,
had only 50 transactions per segment!
# API and ABI breaking changes
N/a
# Expected complexity level and risk
3: Hastily made changes to finnicky code across several crates.
# Testing
I am unsure how to test these changes.
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
Defines the canonical commitlog payload, and how to encode / decode it.
Also exposes folds alongside iterators, which allows the common case of
replaying the commitlog onto a database to be further optimized (the
`Txdata` does not have to be constructed in this case). This
optimization is, however, left for a future patch.
