# Description of Changes
- Updated the Unreal SDK and generated Unreal bindings for the websocket
2.0 protocol/model
- Reworked DbConnectionBase to handle the updated message shapes
- Switched subscription handling over to new message types and
QuerySetId
- Updated reducer to ReducerResult, removal of callbacks, and set
reducer flags
- Added event table support
- Baked in multi-module support replacing [the old
PR](<https://github.com/clockworklabs/SpacetimeDB/pull/3417>)
- Added functionality to generate module support for multiple folders in
the Unreal project (add <module>.Build.cs, <module>.h, <module>.cpp)
using the --module-name
- Add new configuration option for spacetime generate to handle module
prefix
- Regenerated Unreal Blackholio/TestClient/QuickstartChat bindings
- Rebuilt Unreal Blackholio's consume entity to use event tables
- Updated migration documentation
- Updated the version bump tool to impact C++
# API and ABI breaking changes
- Unreal websocket/message handling updated to the new protocol
- Unreal generation now expects a real .uproject target and will stop
immediately if project
metadata is invalid instead of continuing past setup issues.
# Expected complexity level and risk
3 - A large set of changes to update the websocket/message handling
along with heavy codegen changes to handle multi-module support
# Testing
Test coverage of the Unreal SDK will need expansion in a future ticket
once our issues with flakiness on CI is resolved.
- [x] Updated Unreal Blackholio
- [x] Ran full Unreal SDK test suite
- [x] Built new test project using the new `--module-prefix`
- [x] Run through Unreal Blackholio (C++ and Blueprint)
- [x] Rebuilt Unreal Blackholio with multi-module, and duplicate
generated module testing side-by-side modules that would overlap
# Review Question(s)
- [x] Updates to `spacetime init` have made the tutorial a little
confusing with pathing for the Unreal Blackholio tutorial. To fix though
we'd have to update all the commands to be more explicit, or update the
tutorial `spacetime init` to use `--project-path .` to keep pathing
simpler, thoughts?
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Ryan <r.ekhoff@clockworklabs.io>
# Description of Changes
Same change set as
https://github.com/clockworklabs/SpacetimeDB/pull/4614, just targeting
master.
The return type of a query builder view is now a special SATS product
type `{ __query__: T }`. A view with this return type now has a primary
key if `T` has a primary key. This means that client codegen will
generate an `OnUpdate` callback for such views. It will also generate
query builder index bindings for the primary key column.
# API and ABI breaking changes
None. Old modules with query builder views still work, they just don't
have primary keys.
# Expected complexity level and risk
2
# Testing
Added equivalent tests to the ones that were added in #4573 and #4572.
# Description of Changes
Query builder views return a subset of rows from a physical table. If
that table has a primary key, then so should the view. What this means
concretely is that the view should expose the same api as the table,
specifically as it relates to the primary key column.
With that in mind, this patch commits the following changes:
1. Annotates `ViewDef` with a `primary_key`
2. Updates the return type of query builder views in the raw module def
to a special product type
3. Adds an index for the primary key on the view's backing table
4. Updates the query planner to use this index
5. Updates rust client codegen to generate `on_update` for such views
# API and ABI breaking changes
None
Old `impl Query` views compiled with an older version of SpacetimeDB
will continue to work as they did before - without a primary key.
# Expected complexity level and risk
3
# Testing
- [x] New rust sdk integration suite exercising `on_update` for PK views
and semijoin scenarios
- [x] Smoketests for PK views and semijoin scenarios
# Description of Changes
Followup to #3802; was causing issues in modules in private. Rust 2024
now requires `unsafe()` around certain attributes, but the `settings`
macro was added after I first opened that PR, so I didn't wrap it.
Additionally, `settings` wasn't tested in this repo, so it wasn't
caught.
# Expected complexity level and risk
1
# Testing
- [x] Added use of `#[spacetimedb::settings]` in `module-test`
# Description of Changes
It'd be best to review this commit-by-commit, and using
[difftastic](https://difftastic.wilfred.me.uk) to easily tell when
changes are minor in terms of syntax but a line based diff doesn't show
that.
# Expected complexity level and risk
3 - edition2024 does bring changes to drop order, which could cause
issues with locks, but I looked through [all of the warnings that
weren't fixed
automatically](ba80f3fecd/warnings.html)
and couldn't find any issues.
# Testing
n/a; internal code change
# Description of Changes
- Migrated the C++ module-definition assembly path to V10-first
internals:
- Added v10_builder and module_type_registration systems.
- Switched Module::__describe_module__ to serialize RawModuleDef with
V10 payload.
- Updated macro registration pipeline to register through V10
- Added explicit naming support across macro surface (*_NAMED variants
for reducer/procedure/
view and field/index macros).
- Reworked multi-column index macros (FIELD_MultiColumnIndex,
FIELD_MultiColumnIndex_NAMED) with
migration alias.
- Added SPACETIMEDB_SETTING_CASE_CONVERSION(...) to support case
conversion policy
- Error-path hardening by adding explicit constraint-registration error
tracking and preinit validation
- Codegen updates:
- Updated C++ moduledef regen to V10 builder types.
- Adjusted C++ codegen duplicate-variant wrapper generation to emit
proper product-type
wrappers.
- Test/harness updates:
- type-isolation-test runner now defaults to focused V10 regression
checks; --v9 runs broader
legacy/full suite.
- Added focused modules for positive/negative V10 checks:
- test_multicolumn_index_valid
- error_multicolumn_missing_field
- error_default_missing_field
- Re-enabled C++ paths in sdks/rust/tests/test.rs procedure/view/test
suites.
# API and ABI breaking changes
- Refactor of the underlying module definition
- New *_NAMED variant macros for explicit canonical naming
- FIELD_NamedMultiColumnIndex renamed to FIELD_MultiColumnIndex
# Expected complexity level and risk
3 - Large set of changes moving over to V10 with underlying changes to
make future updates a little easier
# Testing
- [x] Ran the type isolation test and expanded it
- [x] Ran the spacetimedb-sdk test framework to confirm no more drift
between C++ and other module languages
- [x] Ran Unreal test suite though not really applicable
- [x] New app creation with `spacetime init --template basic-cpp`
- [x] Ran describe module tests against Rust + C# matching with C++ on
the /modules/sdk-test* modules to find any possible mis-alignment
# Review
- [x] Another look at the new features with C++
- [x] Thoughts on *_NAMED macros, I couldn't come up with a better
solution with C++20
# Description of Changes
Update the Default casing policy to `snake_case` for `RawModuleDefV10`.
Messy PR contains changes at different places, so that CI can pass:
Here are the main changes as follows:
- `bindings-macro` & `bindings` crate: `name` macro in Indexes for
canonical name and supply it to `RawModuleDefV10` via `ExplicitNames`.
- `bindings-typescript`:
- Changes has been reviewed through this PR -
https://github.com/clockworklabs/SpacetimeDB/pull/4308.
- `binding-csharp`: a single line change to pass `sourceName` of index
instead of null.
- `codegen`:
- Changes has been merged from branch -
https://github.com/clockworklabs/SpacetimeDB/pull/4337.
- Except a fix in rust codegen to use canonical name in Query buillder
instead of accessor.
- `lib/db/raw_def`: Extends `RawDefModuleV10` structure to support case
conversion.
- `schema` crate:
- `validate/v9` - Nothing itself should change or changes in v9
validation logic but the file contains a `CoreValidator` which is shared
with `validate/v10`. No test have t be updated to `validate/v9` which
ensures we aren't regressing it.
- `validate/v10`: This is the main meat, look at the new tests added in
bottom to understand what it does.
- Rest of the files are either test updates or module bindings.
## Testing:
1. Extensive unit tests have been added to verify generated `ModuleDef`
is correct.
2. I have done some e2e testing to verify rust codegen with rust and
typescript modules.
3. I would have like to do more testing for other codegens , I am
continue doing .
I have removed `sql.py` smoketest, as that seems to be already migated
in new framework and was headache to update.
## Expected complexity level and risk
4, It could have side-effect which aren't easily visible.
- - -
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Noa <coolreader18@gmail.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
Implementation of #4295
Convert existing `Name` attribute to `Accessor` to support new Canonical
Case conversation of 2.0
# API and ABI breaking changes
Yes, in C# modules, we no longer use the attribute name `Name`, it
should now be `Accessor`
# Expected complexity level and risk
1
# Testing
- [X] Build and tested locally
- [X] Ran regression tests locally
## Description of Changes
This PR primarily affects the `bindings-macro` and `schema` crates to
review:
### Core changes
1. Replaces the `name` macro with `accessor` for **Tables, Views,
Procedures, and Reducers** in Rust modules.
2. Extends `RawModuleDefV10` with a new section for:
* case conversion policies
* explicit names
New sections are not validated in this PR so not functional.
3. Updates index behavior:
* Index names are now always **system-generated** for clients. Which
will be fixed in follow-up PR when we start validating RawModuleDef with
explicit names.
* The `accessor` name for an index is used only inside the module.
## Breaking changes (API/ABI)
1. **Rust modules**
* The `name` macro must be replaced with `accessor`.
2. **Client bindings (all languages)**
* Index names are now system-generated instead of using explicitly
provided names.
**Complexity:** 3
A follow-up PR will reintroduce explicit names with support for case
conversion.
---------
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
# Description of Changes
This PR implements support for the `spacetime.json` configuration file
that can be used to set up common `generate` and `publish` targets. An
example of `spacetime.json` could look like this:
```
{
"dev_run": "pnpm dev",
"generate": [
{ "out-dir": "./foobar", "module-path": "region-module", "language": "c-sharp" },
{ "out-dir": "./global", "module-path": "global-module", "language": "c-sharp" },
],
"publish": {
"database": "bitcraft",
"module-path": "spacetimedb",
"server": "local",
"children": [
{ "database": "region-1", "module-path": "region-module", server: "local" },
{ "database": "region-2", "module-path": "region-module", server: "local" }
]
}
}
```
With this config, running `spacetime generate` without any arguments
would generate bindings for two targets: `region-module` and
`global-module`. `spacetime publish` without any arguments would publish
three modules, starting from the parent: `bitcraft`, `region-1`, and
`region-2`. On top of that, the command `pnpm dev` would be executed
when using `spacetime dev`.
It is also possible to pass additional command line arguments when
calling the `publish` and `generate` commands, but there are certain
limitations. There is a special case when passing either a module path
to generate or a module name to publish. Doing that will filter out
entries in the config file that do not match. For example, running:
```
spacetime generate --project-path global-module
```
would only generate bindings for the second entry in the `generate`
list.
In a similar fashion, running:
```
spacetime publish region-1
```
would only publish the child database with the name `region-1`
Passing other existing arguments is also possible, but not all of the
arguments are available for multiple configs. For example, when running
`spacetime publish --server maincloud`, the publish command would be
applied to all of the modules listed in the config file, but the
`server` value from the command line arguments would take precedence.
Running with arguments like `--bin-path` would, however, would throw an
error as `--bin-path` makes sense only in a context of a specific
module, thus this wouldn't work: `spacetime publish --bin-path
spacetimedb/target/debug/bitcraft.wasm`. I will throw an error unless
there is only one entry to process, thus `spacetime publish --bin-path
spacetimedb/target/debug/bitcraft.wasm bitcraft` would work, as it
filters the publish targets to one entry.
# API and ABI breaking changes
None
# Expected complexity level and risk
3
The config file in itself is not overly complex, but when coupled with
the CLI it is somewhat tricky to get right. There are also some changes
that I had to make to how clap arguments are validated - because the
values can now come from both the config file and the clap config, we
can't use some of the built-in validations like `required`, or at least
I haven't found a clean way to do so.
# Testing
I've added some automated tests, but more tests and manual testing is
coming.
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: bradleyshep <148254416+bradleyshep@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
# Description of Changes
Implements the rest of the casing proposal.
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
## Summary
- Restricts `update()` across all three server-side SDKs (Rust, C#,
TypeScript) to only work on primary key columns, not all unique columns
- Calling `update()` on a non-PK unique column is semantically a
delete+insert — clients won't see it as a row update unless the primary
key stays the same
- Fresh re-implementation of #1862, extended to cover C# and TypeScript
bindings
### Design principle
Primary key is a constraint on columns, not a property of indexes. An
index is unique or not unique — whether `update()` is available is
derived from the column's attributes at the point of use, not stored on
the index.
This differs from #1862 which introduced a `Uniqueness` enum (`No |
Unique | PrimaryKey`) on the index itself. Instead, we keep `is_unique:
bool` and check the column metadata where needed.
### Rust changes
- Add `PrimaryKey` marker trait and `where Col: PrimaryKey` bound on
`UniqueColumn::update()`
- `marker_type()` receives `primary_key_column` and checks the column
directly — no `is_pk` field on the index
- `sdk-test` unique tables use `update_non_pk_by` (delete+insert)
instead of `update_by`
- Benchmarks that used `update()` on `#[unique]` columns changed to
either `#[primary_key]` or delete+insert
### C# changes
- Only emit `Update()` on `UniqueIndex` when the column has `PrimaryKey`
attrs
- Update `unique_*` test reducers in `sdk-test-cs` to use
`Delete()`+`Insert()` instead
### TypeScript changes
- `UniqueIndex` now has only `find` + `delete` (no `update`)
- `AllColumnsPrimaryKey` type-level helper checks column metadata from
the `TableDef`
- `Index` routing conditionally intersects `{ update() }` when columns
are PK
- Runtime only attaches `update` method for PK unique indexes
- `sdk-test-ts` unique tables use delete+insert instead of update
- Type-level test in `schema.test-d.ts` verifies `id.update()` compiles
(PK) and `name2.update` errors (non-PK unique)
## Test plan
- [x] C# codegen tests pass (`dotnet test
crates/bindings-csharp/Codegen.Tests`)
- [x] Rust SDK test module compiles
- [x] Benchmarks compile
- [x] TypeScript type checks pass (`npx tsc --noEmit`)
- [ ] CI passes
Closes#1862
# Description of Changes
Modules are like programs, databases are like processes. Your client
connects to a remote database, not a remote module.
<!-- Please describe your change, mention any related tickets, and so on
here. -->
# API and ABI breaking changes
Yep!
# Expected complexity level and risk
2? I made this change with find + replace, and it's possible that I
borked the edit, or missed some reference somehow. It seems unlikely,
however, that this change will have deeper implications we haven't
considered.
# Testing
N/a
# Description of Changes
Haven't changed `schema()` to accept an object yet, maybe that's for a
followup?
Now everything exported from the module must be exported from the
typescript module.
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
# Description of Changes
This adds C++ server bindings (/crate/bindings-cpp) to allow writing C++
20 modules.
- Emscripten WASM build system integration with CMake
- Macro-based code generation (SPACETIMEDB_TABLE, SPACETIMEDB_REDUCER,
etc)
- All SpacetimeDB types supported (primitives, Timestamp, Identity,
Uuid, etc)
- Product types via SPACETIMEDB_STRUCT
- Sum types via SPACETIMEDB_ENUM
- Constraints marked with FIELD* macros
# API and ABI breaking changes
None
# Expected complexity level and risk
2 - Doesn't heavily impact any other areas but is complex macro C++
structure to support a similar developer experience, did have a small
impact on init command
# Testing
- [x] modules/module-test-cpp - heavily tested every reducer
- [x] modules/benchmarks-cpp - tested through the standalone (~6x faster
than C#, ~6x slower than Rust)
- [x] modules/sdk-test-cpp
- [x] modules/sdk-test-procedure-cpp
- [x] modules/sdk-test-view-cpp
- [x] Wrote several test modules myself
- [x] Quickstart smoketest [Currently in progress]
- [ ] Write Blackholio C++ server module
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Ryan <r.ekhoff@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Same changes as https://github.com/clockworklabs/SpacetimeDB/pull/4101
but for `connection_id`.
# API and ABI breaking changes
API Breaking
# Expected complexity level and risk
1
# Testing
Pure refactor. Tests and docs have been updated, but no new tests added.
# Description of Changes
Makes the `sender` field on `ViewContext` private and exposes a
`sender()` method. Does the same for `ReducerContext` and
`ProcedureContext`.
The purpose of this change: So that the host can determine if/when a
view invokes or reads the `sender`.
Currently, because `sender` is a field, the host assumes that it is
always read. This means views must be materialized per client, even if
the view doesn't actually depend on `sender`, resulting in data
duplication.
The initial solution for this problem was `AnonymousViewContext` which
doesn't have a `sender` field. The better solution is to make `sender` a
method so that it can call into the host and record when it's actually
invoked.
Note, this patch only updates the module API, so the current
implementation does not change. `ViewContext` views are still duplicated
across clients. Changing this requires a new host syscall and for
`sender()` to invoke that syscall. This however is backwards compatible
and can be done anytime after the module APIs for the other languages
(C#, TypeScript, C++) are updated.
Also note that `ReducerContext` and `ProcedureContext` were updated
purely for consistency. There are currently no plans to track reads of
`sender` in these contexts.
# API and ABI breaking changes
Breaks the rust module api.
# Expected complexity level and risk
1
# Testing
N/A
# Description of Changes
Client Query builder for rust, as per proposal -
https://github.com/clockworklabs/SpacetimeDBPrivate/pull/2356.
1. Pach moves query builder to its separate crate, so that it can be
shared between module and sdk.
2. Implements `TypedSubscriptionBuilder` in `sdks/rust` as mentioned in
proposal
3. Modify codegen to extend types to support query builder as mentioned
in proposal
4. a test
# API and ABI breaking changes
NA, additive changes.
# Expected complexity level and risk
2
# Testing
Added a test.
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
We would like to move all of the templates to a central directory
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
---------
Co-authored-by: spacetimedb-bot <spacetimedb-bot@users.noreply.github.com>
# Description of Changes
Closes#3673
*NOTE*: C++ part will be in another PR
# Expected complexity level and risk
2
Adding a new type touch everywhere
# Testing
- [x] Adding smoke and unit test
---------
Signed-off-by: Ryan <r.ekhoff@clockworklabs.io>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
# Description of Changes
Closes
[#3290](https://github.com/clockworklabs/SpacetimeDB/issues/3290).
Adds a new "special" type to SATS, `UUID`, which is represented as the
product `{ __uuid__: u128 }`. Adds versions of this type to all of our
various languages' module bindings libraries and client SDKs, and
updates codegen to recognize it and output references to those named
library types. Adds methods for creating new UUIDs according to the V4
(all random) and V7 (timestamp, monotonic counter and random)
specifications.
# API and ABI breaking changes
We add a new type
# Expected complexity level and risk
2
it impacts all over the code
# Testing
- [x] Extends the Rust and Unreal SDK tests, and the associated
`module-test` modules in Rust, C# and TypeScript, with uses of UUIDs.
- [x] Extends the C# SDK regression tests with uses of UUIDs.
- [x] Extends the TypeScript test suite with tests with uses of UUIDs.
---------
Signed-off-by: Mario Montoya <mamcx@elmalabarista.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
This reapplies the patch from #3704, and fixes the issues that were
causing it to deadlock.
The reason it was deadlocking was that it allowed for the following
sequence of events:
* `SchedulerActor::handle_queued()` begins mutable tx
* `ModuleHost::disconnect_client()` submits call to `call_reducer(tx:
None)`
* scheduler submits call to `call_reducer(tx: Some)`
* `WasmModuleInstance::disconnect_client` now has to try to take tx
lock, but the scheduler's call_reducer already holds it and is behind it
in the queue
So, I moved most of the logic from `handle_queued` back to being
executed in the module worker thread, but kept the code in
`scheduler.rs` so that it can all be reasoned about locally.
Fixes#3645. Should I uncomment the implementation of
`ExportFunctionForScheduledTable for F: Procedure` now?
# Expected complexity level and risk
2 - there's a chance that this patch hasn't fully fixed the deadlock
issue from #3704, but I'm quite confident.
# Testing
- [x] Manually verified that deadlock no longer occurs - previously,
`while true; do python -m smoketests schedule_reducer -k
test_scheduled_table_subscription; done` would freeze up in only 2 or 3
iterations, but now it can run for 10 minutes without issues.
# Description of Changes
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/3729
I genuinely don't know what came over me.
# API and ABI breaking changes
None
# Expected complexity level and risk
1.5 very straightforward but not strictly trivial
# Testing
Adds automated integration tests (written in Rust and run with `cargo
test`, although note this comment from @matklad about integration tests
for the future
https://internals.rust-lang.org/t/running-test-crates-in-parallel/15639/2):
- [x] Can publish an updated module if no migration is required
- [x] Can publish an updated module if auto-migration is required (with
the yes-break flag true/false)
- [x] Cannot publish if a manual migration is required
- [x] Can publish if a manual migration is required but the user
specified `--delete-data`
- [x] Can publish if a manual migration is required by the user
specified `--delete-data=on-conflict`
- [x] No data deletion occurs if no migration is required and
`--delete-data=on-conflict` is specified
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Applies the same implicit filter for views to delta tables that we
already did for physical tables.
Removes a single condition `scan.delta.is_none()` from the already
existing rewrite rule.
# API and ABI breaking changes
None
# Expected complexity level and risk
0
# Testing
- [x] Rust sdk test
# Description of Changes
Fixes:
- [This
issue](https://discord.com/channels/1037340874172014652/1398209084699709492/1441556875647647816)
by exporting the `SubscriptionHandle` type with the `REMOTE_MODULE` type
applied.
- [This
issue](https://discord.com/channels/1037340874172014652/1398209084699709492/1441559246213746749)
by converting to `camelCase` for column names in code generation.
- Fixes an issue where `onMyReducer` callbacks were passing arguments as
variadic params, while the types indicated they would be passed as an
object. `onMyReducer((ctx, argA, argB, argC) => {})` vs
onMyReducer((ctx, { argA, argB, argC}) => {})`
- [Fixes an
issue](https://github.com/clockworklabs/SpacetimeDB/issues/3503#issuecomment-3566715928)
where the table type name was used instead of the table name in code
generation for constructing tables.
- [Fixes
issue](https://discord.com/channels/1037340874172014652/1398209084699709492/1441886030436499466)
with `ScheduleAt` being used in non-table types.
- [Fixes
issue](https://github.com/clockworklabs/SpacetimeDBPrivate/issues/2168)
where template projects do not use the correct lifecycle reducer setup
# API and ABI breaking changes
Adds a new export, and fixes casing in code genreation.
# Expected complexity level and risk
2
# Testing
- I have tested that the `SubscriptionHandle` is correctly exported
- I have tested that the constraint name is now output in `camelCase`
- I have tested that `onMyReducer` callbacks now return a single
argument
- I have tested that the table name (and view name) is now used instead
of the type name for code generation
- I have tested that the new lifecycle reducers correctly compile
This reverts commit b2e37e8008b07673f8de7b617f6c4408ce727a66.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Reverts #3704 which I'm pretty sure contains some sort of bug which is
causing the smoketests to hang.
# API and ABI breaking changes
None
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] CI passing again
# Description of Changes
Reworks how `SchedulerActor::handle_queued` works so that it first
determines the parameters of the call to a reducer or the parameters of
the call to the procedure. This also enables the removal of the special
case `call_scheduled_reducer`.
Fixes#3645.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
A test `schedule_procedure` is added.
---------
Co-authored-by: Noa <coolreader18@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
# Description of Changes
Rust SDK test suite for views
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
This patch only adds tests, it does not change functionality.
# Description of Changes
Provides a fetch-alike API on `ctx.http`. I guess it could just be
`ctx.fetch()` instead of `ctx.http.fetch()`, but I'm not sure if that's
a good idea.
# Expected complexity level and risk
2
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Need to verify that this actually works
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
# Description of Changes
Closes#3517 .
With this PR, procedures (at least, those defined in Rust modules) can
perform HTTP requests! This is performed through a new field on the
`ProcedureContext`, `http: HttpClient`, which has a method `send` for
sending an `http::Request`, as well as a convenience wrapper `get`.
Internally, these methods hit the `procedure_http_request` ABI call /
host function, which uses reqwest to perform an HTTP request. The
request is run with a user-configurable timeout which defaults and is
clamped to 500 ms.
Rather than exposing the HTTP stream to modules, we download the entire
response body immediately, within the same timeout.
I've added an example usage of `get` to `module-test` which performs a
request against `localhost:3000` to read its own schema/moduledef.
This PR also makes all procedure-related definitions in the Rust module
bindings library `#[cfg(feature = "unstable")]`, as per #3644 . The
rename of the `/v1/database/:name/procedure/:name` route is not included
in this PR, so this does not close#3644 .
Left as TODOs are:
- Metrics for recording request and response size.
- Improving performance by stashing a long-lived `reqwest::Client`
someplace.
Currently we build a new `Client` for each request.
- Improving performance (possibly) by passing the request-future to the
global tokio executor
rather than running it on the single-threaded database executor.
# API and ABI breaking changes
Adds new APIs, which are marked as unstable. Adds a new ABI, which is
not unstable in any meaningful way (we can't really do that). Marks
unreleased APIs as unstable. Does not affect any pre-existing
already-released APIs or ABIs.
# Expected complexity level and risk
3 or so: networking is scary, and even though we impose a timeout which
prevents these connections from being truly long-lived, they're still
potentially long-lived on the scale of Tokio futures. It's possible that
running them on the database core is problematic in some way, and so
what I've left as a performance TODO could actually be a
concurrency-correctness issue.
# Testing
- [x] Manually wrote and executed some procedures which make HTTP
requests.
- [x] Added two automated tests to the `sdk-test` suite,
`procedure::http_ok` and `procedure::http_err`, which make successful
and failing requests respectively, then return its result. A client then
makes some assertions about the result.
---------
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
Closes: #3658
With the work on Views we ran into trouble with the Return Type and
certain C# code layout where the types are defined inside the Module
class. This fixes the bug in our current approach but needs
consideration to possibly be changed entirely.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1 - Minor change
# Testing
- [x] Retested with the linked code with and without the namespace
- [x] Re-ran regression tests
---------
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
# Description of Changes
Adds `ProcedureContext::{with_tx, try_with_tx}`.
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/3515.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
An integration test `test_calling_with_tx` is added.
# Description of Changes
This PR is a very large change to the workings of the TypeScript SDK and
as such requires a higher bar of testing than other PRs. However, it
does several important things:
1. Unifies the API of the server and client so they not only have the
same API, but they actually implement it with the same TypeScript types.
This fixes several inconsistencies between them and fixes several small
bugs as well.
2. Closes https://github.com/clockworklabs/SpacetimeDB/issues/3365
3. Closes https://github.com/clockworklabs/SpacetimeDB/issues/3431
4. Closes https://github.com/clockworklabs/SpacetimeDB/issues/3435
5. Subsumes the work done in
https://github.com/clockworklabs/SpacetimeDB/pull/3447
6. Derives all type information on the client from a single
`RemoteModule` type which vastly cleans up the correctness of type
checking on the client and helped me to find several small bugs
It accomplishes this by changing code generation of TypeScript on the
client to code generation approximately what a developer would manually
write in their module. The ultimate goal would be to allow the developer
to use the types and functions that they define on in their module
directly on the client without needing to do any code generation at all,
provided they are using TypeScript on the server and client.
https://github.com/clockworklabs/SpacetimeDB/issues/3365 is resolved by
`.build()`ing the `DbConnection` inside a React `useEffect` rather than
doing it directly in line with the render of the provider. In order to
do that we needed to not expose the `DbConnection` directly to
developers by returning a different type from `useSpacetimeDB`.
`useSpacetimeDB` now returns a `ConnectionState` object which is stored
as React state and updates when any of the fields change. This change
also resolves https://github.com/clockworklabs/SpacetimeDB/issues/3431.
https://github.com/clockworklabs/SpacetimeDB/issues/3435 was the issue
that initially lead me down the rabbit hole of unifying the server and
the client because it was nearly impossible to track down all the
various type functions and how they connect to the values that we code
generate on the server. After several hours of attempting this, I
decided to clean up the types a bit to be more uniform.
Implementing the unification between the client and the server also
necessitated fully implemented parts of the API that were fully
implemented on the server, but were broken or missing on the client.
# API and ABI breaking changes
[Unification] -> Means that this is breaking behavior for the client
SDK, but that the new behavior is identical to the server's existing
behavior
## Breaking changes:
- Table accessor names and index accessor names are converted to
camelCase on the `ctx`, so `ctx.db.foo_bar` is now `ctx.db.fooBar`
- [Unification] On the client `my_table.iter()` returns
`IterableIterator` instead of an `Array`
- [Unification] `module_bindings` now export `TypeBuilder`s for all
types instead of a `type MyType` and object `MyType`, so instead of
using `MyType` as a type directly, you need to infer the type `MyType`
-> `Infer<typeof MyType>`.
- [Unification] We no longer generate and export `MyTypeVariants` for
sum types (these are now accessed by `Infer<typeof
MyType.variants.myVariant>`)
- [Unification] `MyType.getTypeScriptAlgebraicType()` has been replaced
with `MyType.algebraicType`
- `useSpacetimeDB()` no longer takes type parameters
- `useTable()` now takes a `TableDef` parameter and type params are
inferred
- `useTable()` now just returns an `Array` directly instead of a object
with `{ rows }`
- [Unification] `ctx.reducers.createPlayer(argA, argB)` ->
`ctx.reducers.createPlayer({ argA, argB })`
- [Unification] `ctx.reducers.onCreatePlayer(ctx, argA, argB)` ->
`ctx.reducers.onCreatePlayer(ctx, { argA, argB })`
- [Unification] `ctx.reducers.removeOnCreatePlayer(ctx, argA, argB)` ->
`ctx.reducers.removeOnCreatePlayer(ctx, { argA, argB })`
- [Unification] `myTable.count(): number` -> `myTable.count(): bigint`
## Additive changes:
- `Infer<>` now also does `InferTypeOfRow<>` if applicable
- Added a `useReducer()` React hook
- `module_bindings` now exports a `tables` object with references to all
the `TableDef`s
- `module_bindings` now exports a `reducers` object with references to
all the `ReducerDef`s
- Added a new `MyType.create('MyVariant', ...)` function in addition to
the `MyType.MyVariant(...)` constructors (this is private)
## Notable things that did not change:
- `MyType.serialize(writer: BinaryWriter, value: Infer<typeof MyType>)`
and `MyType.deserialize(reader: BinaryReader): Infer<typeof MyType>` are
still supported exactly as before.
- The `MyType.MyVariant(...)` constructor function on sum types is still
present, but implemented with the private `MyType.create('MyVariant',
...)`. We could choose to move away from this API later if we didn't
like the variants polluting the namespace
# Expected complexity level and risk
4 - This is a deep reaching an complex change for the SDK. For the
server, it is much less deep reaching since it reuses much of the same
machinery, although it does require thorough testing there as some of
the code was modified.
This change is fully localized to TypeScript and does not touch the host
(or other languages) at all, and therefore only impacts a beta aspect of
SpacetimeDB.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] Added regression test for
https://github.com/clockworklabs/SpacetimeDB/issues/3435
- [x] Manually tested `test-app` and `test-react-router-app`
- [ ] Add test cases for camelCase-ing
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
Updates C# bindings to allow module authors to define `View`s and
`AnonymousView`s using the pattern:
```
[SpacetimeDB.View]
public static ExampleData? GetExampleDataById(ViewContext ctx, uint id)
{
return ctx.Db.ExampleData.Id.Find(id);
}
[SpacetimeDB.View]
public static ExampleData? GetAnonymousExampleDataById(AnonymousViewContext ctx, uint id)
{
return ctx.Db.ExampleData.Id.Find(id);
}
```
During publishing of a C# module, the views data will be converted into
`RawViewDefV9`
# API and ABI breaking changes
No known breaking changes
# Expected complexity level and risk
2
# Testing
- [X] Locally tested locally adding the above sample pattern to the
`sdks\csharp\examples~\regression-tests\server` test and performing a
`dotnet build` and a `spacetime publish test`, both of which succeeded.
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
# Description of Changes
This commit adds support to the Rust client SDK for calling procedures.
Similar to reducers, each `DbContext` implementor has a `pub procedures:
RemoteProcedures` field, with methods provided by extension traits for
each procedure.
Unlike reducers, the provided methods are invoke and invoke-then.
Invoke-then takes a `FnOnce` callback to run
when the SDK is notified of the procedure's termination status, while
invoke ignores that notification.
No mechanism is provided for observing procedures invoked by other
clients.
Procedure callbacks are implemented by storing a map from `request_id`
to `ProcedureCallback`, with the callback closure internally knowing how
to deserialize the return value. It's mildly unfortunate to deserialize
within the callback instead of on the preprocess background task, but it
saves significant complexity.
This commit also adds a new sdk-test module, `sdk-test-procedures`, and
a new Rust test client, `procedure-client`.
Together, these are used in two tests of invoking and observing
procedures. I've left TODOs for other tests that we should write as we
implement additional procedure features.
I also had to fix a few minor bugs in the Rust codegen which were not
strictly related to procedures: we previously assumed that the sets of
reducers and of tables were non-empty, which led to wonky invalid
codegen on modules which did not define any reducers or which did not
define any tables.
I'm sneaking a change to the Nix flake into this PR as well - when
initially writing it I had included `cargoArtifacts` (the pre-built and
cached dependencies of our actual builds) in its `packages`, but that
was neither necessary or useful, and just made building the shell take a
long time after dependency changes.
# API and ABI breaking changes
Breaks the internal interface between the Rust client SDK and codegen,
so users will have to re-run `spacetime generate`.
# Expected complexity level and risk
2-ish? Pretty simple change to the Rust SDK overall.
# Testing
- [x] Added new automated integration tests exercising the new
functionality.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Currently need changes from both #3548 and #3527, and the latter is
merged into master but the former isn't rebased on top of it. so we stay
silly for the moment
I'd like to pull out my first commit into its own PR, but that's not
really possible until #3548 rebases onto master.
# Expected complexity level and risk
2 - pretty straightforward translation of the wasm/Rust view
implementation to typescript.
# Testing
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
# Description of Changes
Specifically generate client table handles for views.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Added a `#[view]` to `module-test` and generated new client snapshots
# Description of Changes
Adds `modules/benchmarks-ts`.
# API and ABI breaking changes
None
# Expected complexity level and risk
2?
# Testing
This is a test.
---------
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
This commit adds a macro attribute `#[procedure]` which applies to
functions, and various in-WASM machinery for defining, calling and
running procedures.
A simple example of a procedure, included in `modules/module-test`:
```rust
fn sleep_one_second(ctx: &mut ProcedureContext) {
let prev_time = ctx.timestamp;
let target = prev_time + Duration::from_secs(1);
ctx.sleep_until(target);
let new_time = ctx.timestamp;
let actual_delta = new_time.duration_since(prev_time).unwrap();
log::info!("Slept from {prev_time} to {new_time}, a total of {actual_delta:?}");
}
```
We intend eventually to make procedures be `async` functions (with the
trivial `now_or_never` executor from `future-util`), but I found that
making the types work for this was giving me a lot of trouble, and
decided to put it off in the interest of unblocking more parallelizable
work.
Host-side infrastructure for executing procedures is not included in
this commit. I have a prototype working, but cleaning it up for review
and merge will come a bit later.
One item of complexity in this PR is enabling scheduled tables to
specify either reducers or procedures, while still providing
compile-time diagnostics for ill-typed scheduled functions (as opposed
to publish-time). I had to rewrite the previous
`schedule_reducer_typecheck` into a more complex `schedule_typecheck`
with a trait `ExportFunctionForScheduledTable`, which takes a "tacit
trait parameter" encoding reducer-ness or procedure-ness, as described
in https://willcrichton.net/notes/defeating-coherence-rust/ .
The trait name `ExportFunctionForScheduledTable` is user-facing in the
sense that it will appear in compiler diagnostics in ill-typed modules.
As such, I am open to bikeshedding.
# API and ABI breaking changes
Adds a new user-facing API, which we intend to change before releasing.
# Expected complexity level and risk
2? Mostly pretty mechanical changes to macros and bindings.
# Testing
- [x] Added a procedure definition to `module-test`, saw that it
typechecks.
- [x] Executed same procedure definition using #3390 , the prototype
implementation this PR draws from.
# Description of Changes
Currently based on #3361
Implements most of the TS module API (not yet a function for type
aliases).
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Extremely basic module stuff works
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
---------
Signed-off-by: Noa <coolreader18@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@aol.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Move the global.json file to the repo root, which should make `dotnet`
resolve the correct SDK regardless of the directory it is run in.
Where a copy of the global.json file is needed, symlink it to the one at
the root.
Should fix "C#/Unity - Test Suite".
# Description of Changes
Extends log records with a new field, `function`, which stores the name
of the reducer being executed when the log was produced. I have chosen
to name this field `function` rather than `reducer` because we will soon
be adding procedures, which are not reducers but will also be valid
values for this field.
While making this change, I noticed inconsistent values for injected
logs. Previously, we injected logs in three places, with different
values for the record fields:
1. `SystemLogger` (used when publishing and updating) set `filename:
Some("spacetimedb")` and `target: None`.
2. `log_reducer_error` (used for reducer error returns) set `filename:
None` and `target: Some(reducer)`, with `reducer` being the name of the
reducer.
3. `ModuleHost::inject_logs` (used for calls to nonexistent reducers and
calls with ill-typed arguments) set `filename: Some("external")` and
`target: None`.
With this change, I have decided that injected logs universally have
`filename: Some("__spacetimedb__")` and `target:
Some("__spacetimedb__")`. I have chosen the double-underscore convention
for reserved names to avoid confusion should a user name a source file
or reducer `spacetimedb`. I am not terribly attached to using
`spacetimedb` here, and would be happy to change to some other string,
like `internal` or `system`.
I have further decided that `log_reducer_error` and `inject_logs` both
have access to a sensible non-`None` value for `function` and so should
set it to the name of the reducer. The `target` field is not used for
this. Note that in cases where a client attempts to call a non-existent
reducer, the `function` field of the log record will be set to that
client-supplied non-existent reducer name.
# API and ABI breaking changes
Changes our log record format. This is at least backwards-compatible (as
in, newer consumers can read older logs) because the new field is
optional. I am unsure if it is forwards-compatible (as in, older
consumers reading newer logs) because I don't know if `serde_json` will
ignore unknown fields or will error.
# Expected complexity level and risk
1
# Testing
- [x] Manually ran `quickstart-chat` (modified to log messages and name
changes) and saw sensible output.
# Description of Changes
PR introduces support for column-level default values via a new
`#[default(...)]` attribute.
It also validates, `default` macro is not used along with `primary_key`,
`unique` or `auto_inc`.
# API and ABI breaking changes
NA
# Expected complexity level and risk
2
# Testing
Start using macro in `module-test`.
---------
Co-authored-by: James Gilles <jameshgilles@gmail.com>
# Description of Changes
Uncommented out code from test, in order to have "delete row by index"
test of C# module code ran.
This is the implementation of a fix for issue
https://github.com/clockworklabs/SpacetimeDB/issues/2233
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [X] Ran `dotnet run` and `dotnet test`, both returned no errors.
# Description of Changes
Apparently, I missed several license files in #3002. I'm not sure what
method I was using to find them, but apparently it was insufficient.
**This replaces all empty `LICENSE` files with an explicit (symlink to)
BSL license, and all apache licenses with symlinks to the root apache
license.** This PR does not intentionally change any license terms, so
if you see one that changed, **it's a mistake**.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
```bash
$ find . -name '*LICENSE*' -type f | grep -v '\.meta$'
./crates/sqltest/standards/LICENSE # this one is an external library that we are not allowed to re-license
./LICENSE.txt # this is the root license
```
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
