The commitlog so far assumed that the latest segment is never compressed
and can be opened for writing (if it is intact).
However, restoring the entire commitlog from cold storage results in all
segments being compressed. Make it so the resumption logic reads the
metadata from the potentially compressed last segment, and starts a new
segment for writing if the latest one was indeed compressed.
# Expected complexity level and risk
1.5
# Testing
Added a test.
# Description of Changes
It'd be best to review this commit-by-commit, and using
[difftastic](https://difftastic.wilfred.me.uk) to easily tell when
changes are minor in terms of syntax but a line based diff doesn't show
that.
# Expected complexity level and risk
3 - edition2024 does bring changes to drop order, which could cause
issues with locks, but I looked through [all of the warnings that
weren't fixed
automatically](ba80f3fecd/warnings.html)
and couldn't find any issues.
# Testing
n/a; internal code change
# Description of Changes
Index offset `truncate` methods returns `IndexError::KeyNotFound` when
asked to truncate on empty index offset file or the key in input is
smaller than the first entry.
This was causing `commitlog::reset_to` method to return error, and stuck
replicas in re-spawn loop.
# API and ABI breaking changes
NA
# Expected complexity level and risk
1
# Testing
Added new tests to cover edge scenerios.
- Extends `commit::Metadata` to include the checksum
- Extends `segment::Metadata` to include `Some(commit::Metadata)`
containing the last commit in the segment (if there is one)
- Changes `committed_meta` to:
- ignore empty segments at the end of the log
- try harder to provide useful metadata, even if only a prefix of the
latest segment is readable
This is allows to eliminate remaining `Commitlog::open` calls with the
purpose of querying the latest commit (offset). `Commitlog::open`
creates an empty segment if the tail of the log is corrupt, which is a
non-obvious side-effect that can be confusing when debugging.
It also allows to eliminate uses where the `commits_from` iterator is
used to find the latest full commit. The `Commits` iterator requires the
caller to handle the case of a corrupted commit at the end of the log,
by advancing the iterator once more after it has yielded an error in
order to check that it is exhausted, and then deciding whether to ignore
the error. This is easy to forget.
`committed_meta` now just does the right thing, preserving information
about tail corruption for when that's useful.
Changes the commitlog (and durability) write API, such that the caller
decides how many transactions are in a single commit, and has to supply
the transaction offsets.
This simplifies commitlog-side buffering logic to essentially a
`BufWriter` (which, of course, we must not forget to flush). This will
help throughput, but offers less opportunity to retry failed writes.
This is probably a good thing, as disks can fail in erratic ways, and we
should rather crash and re-verify the commitlog (suffix) than continue
writing.
To that end, this patch liberally raises panics when there is a chance
that internal state could be "poisoned" by partial writes, which may be
debatable.
# Motivation
The main motivation is to avoid maintaining the transaction offset in
two places in such a way that they could diverge. As ordering commits is
the responsibility of the datastore, we make it authoritative on this
matter -- the commitlog will still check that offsets are contiguous,
and refuse to commit if that's not the case.
A secondary, related motivation is the following:
A "commit" is an atomic unit of storage, meaning that a torn (partial)
write of a commit will render the entire commit corrupt. There hasn't
been a compelling case where we would want this, and have always
configured the server to write exactly one transaction per commit.
The code to handle buffering of transactions is, however, rather
complex, as it tries hard to allow the caller to retry writes at commit
boundaries. An unfortunate consequence of this is that we'd flush to the
OS very often, leaving throughput performance on the table.
So, if there is a compelling case for batching multiple transactions in
a commit, it should be the datastore's responsibility.
# API and ABI breaking changes
Breaks internal APIs only.
# Expected complexity level and risk
5 - Mostly for the risk
# Testing
Existing tests.
# Description of Changes
Standardizes the query builder API across all three language SDKs (Rust,
TypeScript, C#) for consistency.
**Rust:**
- Rename `Query` struct to `RawQuery`, make `Query` a trait with `fn
into_sql(self) -> String`
- All builder types (`Table`, `FromWhere`, `LeftSemiJoin`,
`RightSemiJoin`) implement `Query<T>` trait
- Views can return `-> impl Query<T>` instead of specifying exact
builder types
- The `#[view]` macro auto-detects `impl Query<T>` and rewrites to
`RawQuery<T>`
- Add `Not` variant to `BoolExpr` with `.not()` method
**TypeScript:**
- Add `ne()` to `ColumnExpression`
- Refactor `BooleanExpr` to `BoolExpr` class with chainable `.and()`,
`.or()`, `.not()` methods
- Make builders valid queries directly (`.build()` deprecated but still
works)
- Deprecate `from()` wrapper — use `tables.person.where(...)` directly
- Merge `query` export into `tables` so table refs are also query
builders
- Add subscription callback form: `subscribe(ctx =>
ctx.from.person.where(...))`
- Unify `useTable` with query builder syntax; deprecate `filter.ts`
**C#:**
- Add `Not()` method to `BoolExpr<TRow>`
- Add `IQuery<TRow>` interface implemented by all builder types
(`Table`, `FromWhere`, `LeftSemiJoin`, `RightSemiJoin`, `Query`)
- Add `ToSql()` to all builder types so `.Build()` is no longer required
- Update `AddQuery` to accept `IQuery<TRow>` instead of `Query<TRow>`
# API and ABI breaking changes
- Rust: `Query<T>` is now a trait (was a struct). The struct is renamed
to `RawQuery<T>`. This is a breaking change for any code that used
`Query<T>` as a type directly.
- TypeScript: `BooleanExpr` is now a `BoolExpr` class (was a
discriminated union type). The `query` export is deprecated in favor of
`tables`.
- C#: `AddQuery` now accepts `Func<QueryBuilder, IQuery<TRow>>` instead
of `Func<QueryBuilder, Query<TRow>>`. Existing `.Build()` calls still
work since `Query<TRow>` implements `IQuery<TRow>`.
# Expected complexity level and risk
3 — Changes touch multiple language SDKs and codegen, but each
individual change is straightforward. The Rust macro rewrite for `impl
Query<T>` detection is the most complex piece. All existing
`.build()`/`.Build()` calls continue to work.
# Testing
- [x] `cargo test -p spacetimedb-query-builder` — 16/16 tests pass
- [x] `cargo check -p spacetimedb` — clean, no warnings
- [x] `cargo check` on views-query, views-sql, views-basic,
views-trapped smoketest modules — all clean
- [x] `cargo test -p spacetimedb-codegen codegen_csharp` — snapshot
updated, passes
- [x] `npm test` (TypeScript) — 101/101 tests pass
- [x] C# QueryBuilder tests — new tests for `Not()`, `IQuery<T>`
interface
- [ ] CI passes
# Description of Changes
Just adds public accessors for some existing internal functionality, to
enable a change in another repo. Review starting from there.
# API and ABI breaking changes
These crates are marked unstable.
# Expected complexity level and risk
0.5
# Testing
See other PR.
---------
Co-authored-by: Kim Altintop <kim@eagain.io>
It is almost always wrong / undesirable to pack more than one
transaction in a commit, so adjust the default accordingly.
This also avoids surprises when using `#[serde(default)]` with nested
structs -- serde evaluates the default depth-first, so overriding a
single field in a nested struct will not consider any
`#[serde(default = "custom_default")]` annotations on the parent.
When a new commitlog segment is created, allocate disk space for it up
to the maximum segment size. Also do this when resuming writes to an
existing segment, such that segments created without preallocation will
allocate as well when the database is opened.
Preallocation is gated behind the feature "fallocate", because it is not
always desirable to preallocate, e.g. for local `standalone` users.
The feature can only be enabled on Linux targets, because allocation is
done using the Linux-specific `fallocate(2)` system call.
Unlike `ftruncate(2)` or the portable `posix_fallocate(3)`,
`fallocate(2)`
supports allocating disk space without zeroing. This is currently
required, because the commitlog format does not handle padding bytes.
If not enough space can be allocated, the commitlog refuses writes. For
commitlogs that were created without preallocation, this means that the
commitlog cannot even be opened in this situation.
The local durability impl will crash if it detects that the commitlog is
unable to allocate enough space.
This means that a database will eventually crash and be unable to start
in
an out-of-space situation.
Allocated space is not included in the reported size of the commitlog.
Instead, allocated blocks are reported separately.
# Expected complexity level and risk
3 - Disk size monitoring may need to be adjusted.
# Testing
- [x] Adds a test that demonstrates the crash behavior of
[`spacetimedb_durability::Local`]
when there is insufficient space. The test performs I/O against a loop
device.
- [x] Modified the `repo::Memory` impl so that it can run out of space.
No test currently
utilizes this, but existing tests assuming infinite space still pass.
The commitlog creates new segments atomically, returning EEXIST if the
segment already exists. This is to break a retry loop in case the
filesystem becomes unwritable.
This error did not contain any context about what does not exist, so
this patch adds some.
Also, an unhandled edge case has been discovered:
When opening an existing log, the commitlog will try to resume the last
segment for writing. If it finds a corrupt commit in that segment, it
won't resume, but instead create a new segment at the corrupt commit's
offset + 1.
However, if the first commit in the last segment is corrupted, the
offset will be that of the last segment -- trying to start a new segment
will thus fail with EEXIST.
Without additional recovery mechanisms, it is not obvious what to do in
this case: the segment could contain valid data after the initial
commit, so we certainly don't want to throw it away.
Instead, we now detect this case and return `InvalidData` with some
context.
# Expected complexity level and risk
1
# Testing
- [ ] A (regression) test is included
# Description of Changes
We've run into a problem on Maincloud caused by a database that was
writing a relatively small number of very large transactions. This was
accruing many commitlog segments consuming hundreds of gigabytes of
disk, but had not ever taken a snapshot, or compressed or archived any
data, as the database had not progressed past one million transactions.
With this PR, we take a snapshot every time the commitlog segment
rotates. We still also snapshot every million transactions.
One BitCraft database we looked at had 2.5 million transactions per
commitlog segment, meaning that this change will not meaningfully affect
the frequency of snapshots. The offending Maincloud database, however,
had only 50 transactions per segment!
# API and ABI breaking changes
N/a
# Expected complexity level and risk
3: Hastily made changes to finnicky code across several crates.
# Testing
I am unsure how to test these changes.
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
# Description of Changes
Necessary for pulling in rolldown.
# API and ABI breaking changes
None
# Expected complexity level and risk
1, with the caveat that this updates the Rust version and therefore
touches all the code.
# Testing
- [ ] Just the automated testing
# Description of Changes
We recently merged several repos together. This PR clarifies the license
terms for several subdirectories, as well as the relationship between
the licenses.
The licenses in our subdirectories have become symbolic links to
licenses in our toplevel `licenses` directory. For any particular
subdirectory's license file in the diff, you can click `... -> View
file` and then click on the text that says "Symbolic Link" on that page.
This will take you to the license file that it links to.
I have also updated the `tools/upgrade-version` script to update the
change date in the new `licenses/BSL.txt` file.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
None. Only changes to license files.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Adds methods and free-standing functions to allow folds to stop at an
upper
bound, by passing a range instead of only a start offset.
# Expected complexity level and risk
1
# Testing
