# Introduction
In this PR we're migrating the serverless function service that was
using the SF repo directly to the v2 build and runner.
The whole serverless engine now deals with flat entities only
## Resolvers
Refactored the resolvers ( serverlessFunction, route, database and cron
trigger) :
- return types to `dto`
- Standardized the flat to dto transpilation within the resolvers
- Find and findMany passing by the cached data
## Services
Refactored the services ( serverlessFunction, route, database and cron
trigger) :
- return type to be `flat`
- always calling v2 and computing cache
## New additional caches
- application variables ( cf
https://github.com/twentyhq/core-team-issues/issues/2116 )
- serverless function layer
## What to test:
- CRUD ( database trigger ✅ , route trigger, cron trigger, serverless
function through workflows ✅ )
- Duplicating a workflow with a serverless function code node ✅
## Concerns
We need to implement the cron that will hard delete soft deleted s3
serverless functions, not in this PR though ( cf
https://github.com/twentyhq/twenty/pull/17285#discussion_r2709168570 and
https://github.com/twentyhq/core-team-issues/issues/2118 )
# Introduction
Followup of
https://github.com/twentyhq/twenty/pull/17001#pullrequestreview-3638508738
close https://github.com/twentyhq/core-team-issues/issues/1910
We've completely decom the `sync-metadata` in production. We're now then
removing its implementation in favor of the v2.
## TODO:
- [x] Remove sync-metadata implem and commands
- [x] Remove workspace decorators
- [x] Type each deprecated field to deprecated on their workspaceEntity
- [x] Remove the `workspace-sync-metadata` folder entirely
- [x] remove workspace migration
- [x] workspace migration removal migration
- [x] remove the `v2` references from workspace manager file names
- [x] remove the `v2` references from workspace manager modules
- [ ] Double check impact on translation file path updates
## Note
- Removed the gate logic
- Remains some service v2 naming, serverless needs to be migrated on v2
fully
- Removed workspaceMigration service app health consumption, making it
always returning up ( no more down ) cc @FelixMalfait ( quite obsolete
health check now, will require complete refactor once we introduce inter
app dependency etc )
Changes:
- as we store date in redis as serialized, let's make all flatEntity
dates as string. This requires changing FlatEntity types and making sure
that entity are converted to flatEntity and flatEntity to dtos
# Introduction
Refactored the api `validateBuildAndRunWorkspaceMigration` to be require
less configuration but to infer required args dynamically depending on
provided metadata maps to compare
## `inferDeletionFromMissingEntities`
Is not dynamically computed avoiding any miss configuration issue and
any missleading devxp
## Maps computation
Making only one call to redis to build both dependency and to be
compared entity maps. It does not matter to avoid passing a about to
compared flat entity maps to could also be a depedency, it's handled
directly in the builder setup optimistic cache logic
Please note that the flat maps used for the service input transpilation
might differ from the one that we will dynamically compute and inject in
the builder. Leading to do 2 redis calls but also race condition prone
validation error
We prefer that this occurs at the builder rather than at the runner
level as the pg instance is not cache and reflect the real state of a
given workspace
In a nutshell, there's a possible race condition between cache
invalidation and computation in both service input transpilers and
builder but we're totally ok with that
# Introduction
Closes https://github.com/twentyhq/core-team-issues/issues/1980
In this PR we migrate the agent from v1 to v2.
## New FlatRoleTargetByAgentIdMaps
Derivated the `flatRoleTargetMaps` to be building a
`flatRoleTargetByAgentIdMaps` to ease retrieving a roleId to associate
to an agent
## Coverage
Added strong coverage on both failing and successful CRU agents
operations
---------
Co-authored-by: Weiko <corentin@twenty.com>
## Context
We've recently introduced a new workspace cache service which now acts
as a cache access and local storage for all workspace related data,
deprecating the individual specific services.
- Better performance through multiple caching/fetching strategies
- Consistent data access patterns across the codebase
- Reduced redis queries through MGET/MSET/PIPELINE with multiple cache
keys
# Introduction
Cleaner and fewer scope version of
https://github.com/twentyhq/twenty/pull/15745 ( removed sync-metadata
hack through, too ambitious migration and upgrade )
Please note that this PR won't have any interaction with the existing
sync-metadata
Which mean that the sync metadata does not update the standard entities
applicationId and universalIdentifier, and it won't we will deprecate it
on favor of a workspace migration aka twenty-standard app installation
## API Metadata
Any operation going through the api metadata nows automatically scope
the related entity to the workspace custom application instance. (
optionally passing an applicationId to allow current hacky implem of app
sync service )
We need to either ignore the tests or remove the cli status check from
the blocking status badges for a PR to be merged
## New workspace
Already handled in previous
https://github.com/twentyhq/twenty/pull/15625, when a workspace is
created it gets created a twenty standard and custom workspace instance
All his views and permissions will be prefilled to the its twenty
standard app instance with a specific universalIdentifier
## New universalIdentifier
At the contrary as before with standardIds, universalIdentifier are
unique for a given workspace
This means that createdAt field of both object company and opportunity
will have a unique universalIdentifier whereas they share the same
standardId
## FlatApplication
Introduced the flatApplication and cache. Will migrate existing
`MetadataName` to be `SyncableMetadataName` in a following PR
## What's next
Next we will describe a twenty standard app configuration as json that
will be used to generate a workspace migration that will be run instead
of the sync metadata, in a nutshell we aim to deprecated the sync
metadata
So we can standardize any entity to have a non nullable applicationId
and universalIdentifier
## Upgrade command
Introduced an upgrade command that will create a custom workspace
instance for any workspace that do not have one in order to align with
the new behavior when creating a new workspace
# Introduction
This PR aims to deprecate having to manually handle optimistic side
effect foreign key addition in the whole v2 experience.
This PR implements the strong basis + builder refactor of the optimistic
computation of a given flat entity maps with its related flat entity
maps ( runner needs a small refactor on actions type definition first )
Flat entity maps updates through mutations are now only scoped to the
generic entity builder ( very isolated )
## What's next
- Refactor actions v2 type definition to gain grain over `metadataName`
and action operation ( `create` `delete` `update` ).
from `{type: 'create_view_field'}` to `{metadataName: 'view_field',
type: 'create' }`
- Use new optimistic tool computation tools
- Only invalidate impacted flat maps cache
## New tools
Strictly dynamically typed new flat entity maps tools
- `addFlatEntityToFlatEntityAndRelatedEntityMapsThroughMutationOrThrow`
-
`deleteFlatEntityFromFlatEntityAndRelatedEntityMapsThroughMutationOrThrow`
## Unit test
Adding basic unit testing coverage to introduced tools
## `FlatEntityValidationArgs`
From
```ts
export type FlatEntityValidationArgs<T extends AllMetadataName> = {
flatEntityToValidate: MetadataFlatEntity<T>;
optimisticFlatEntityMaps: MetadataFlatEntityMaps<T>;
mutableDependencyOptimisticFlatEntityMaps: MetadataValidationRelatedFlatEntityMaps<T>;
workspaceId: string;
remainingFlatEntityMapsToValidate: MetadataFlatEntityMaps<T>;
buildOptions: WorkspaceMigrationBuilderOptions;
};
```
To
```ts
export type FlatEntityValidationArgs<T extends AllMetadataName> = {
flatEntityToValidate: MetadataFlatEntity<T>;
optimisticFlatEntityMapsAndRelatedFlatEntityMaps: MetadataFlatEntityAndRelatedFlatEntityMapsForValidation<T>;
workspaceId: string;
remainingFlatEntityMapsToValidate: MetadataFlatEntityMaps<T>;
buildOptions: WorkspaceMigrationBuilderOptions;
};
```
## Context
- All flatEntity should extend SyncableEntity
- SyncableEntity should now have applicationId and application relation
- Fix syncApp deletion, should now properly use migration v2 to delete
syncable entities
# Introduction
### Summary
Implements side effect handling for `ViewGroup` and `ViewFilters` when
field metadata is updated in the v2 architecture. This ensures that
view-related records are properly maintained when enum field options are
modified, deleted, or created.
### Side effects
- **Side Effect System**: Added side effect handling for field metadata
updates that manages related view groups and view filters
- **Enum Field Updates**: When enum field options are modified, the
system now:
- **View Groups**: Creates new groups for added options, updates
existing groups for modified options, and deletes groups for removed
options
- **View Filters**: Updates filter values to reflect option changes and
removes filters that reference deleted options
### Enum runner fix
Update now works for both atomic enum and array enum ( multi select for
instance )
### Compute flat entity maps from to
Standardized this method usage across v2 services
Next step is to require dependencies dynamically
## Conclusion
closes https://github.com/twentyhq/core-team-issues/issues/1649
## Goal
- inferDeletionFromMissingEntities is now a map instead of a single
bool, allowing us to parameterise it based on the entity we want to
compare
- Adding index creation/update when field isUnique is set to true and
index deletion when isUnique is false (for now)
close https://github.com/twentyhq/core-team-issues/issues/1346
# Introduction
This PR introduces a huge type refactor that will leverage dynamic intra
entity optimistic flat maps update in the future and also a more
granular cache invalidation enhancing performances
close https://github.com/twentyhq/core-team-issues/issues/1717
close https://github.com/twentyhq/core-team-issues/issues/1716
close https://github.com/twentyhq/core-team-issues/issues/1643
## What's done
### Comparators centralization
Comparator is now done through global configuration as const for each
metadata names
Thanks to
Note:
Definition of standard is evolving, standard is now scoped to an app.
Meaning that a manifest should be able to update its own standards
objects but on other app standards ones ? Each synchronizable entities
will have a standardOverrides ?
## Typing refactor
### `AllFlatEntityTypesByMetadataName`
**Single source of truth for the complete type ecosystem**, mapping each
metadata name to its entity types, flat entities, and migration actions:
```typescript
export type AllFlatEntityTypesByMetadataName = {
fieldMetadata: {
actions: { created: CreateFieldAction; updated: UpdateFieldAction; deleted: DeleteFieldAction; };
flatEntity: FlatFieldMetadata;
entity: FieldMetadataEntity;
};
objectMetadata: { /* ... */ };
// ... all 10 metadata types
};
```
### `ALL_METADATA_NAME_MANY_TO_ONE_RELATIONS`
**Explicitly declares database relationships** between entities with
compile-time validation:
```typescript
export const ALL_METADATA_NAME_MANY_TO_ONE_RELATIONS = {
viewField: {
view: 'viewId',
fieldMetadata: 'fieldMetadataId',
},
cronTrigger: {
serverlessFunction: 'serverlessFunctionId',
},
// ... all relations
} as const satisfies MetadataNameAndRelations;
```
### `ALL_FLAT_ENTITY_CONFIGURATION`
**Centralizes comparison and serialization logic** for each metadata
type:
```typescript
export const ALL_FLAT_ENTITY_CONFIGURATION = {
fieldMetadata: {
propertiesToCompare: ['name', 'type', 'label', 'defaultValue', /* ... */],
propertiesToStringify: ['options', 'settings', 'defaultValue'],
},
objectMetadata: {
propertiesToCompare: ['nameSingular', 'namePlural', 'isActive', /* ... */],
propertiesToStringify: [],
},
// ... all metadata types
} as const satisfies AllFlatEntityConfiguration;
```
## Combined Impact
These three configurations work together to create a **strongly-typed,
centrally-managed metadata system**:
1. **`AllFlatEntityTypesByMetadataName`** defines *what exists*
2. **`ALL_METADATA_NAME_MANY_TO_ONE_RELATIONS`** defines *how they
relate*
3. **`ALL_FLAT_ENTITY_CONFIGURATION`** defines *how to compare and
serialize them*
**Result:** Builders and validators become thin wrappers around
type-safe, configuration-driven logic instead of containing scattered,
error-prone manual implementations.
## What's next
### StandardOverrides standardization
Every metadata entity can be a standard one for a workspace if it's an
installed app, which means it might not expose the whole entity api to
be editable through an import dynamically
The standard overrides logic should not be applied to Fields and Objects
but to every entities
At the moment we have a logic of `EDITABLE_PROPERTIES` through the api,
and also `STANDARD_OVERRIDEDABLE_PROPERTIES`
This should be configuration centered like `propertiesToCompare` and
`propertiesToStringify`.
Scoping this PR to two last for the moment. As update dispatch to
standardOverrides could be considered as a side effect prefer waiting to
start the side effect refactor
### Granular Optimistic deprecation
With this new grain at runtime we will be able to add a flat entity and
dispatch its addition to related flat maps, so we don't have to describe
an optimistic method for each flat entity operations
See `addFlatEntityToFlatEntityAndRelatedEntityMapsOrThrow`
Note: Still in wip and included in this PR but about to create a new one
to integrate these utils and remove existing methods
### ValidateBuildAndRun dynamic args typed defintion
We should restrain the devxp to send expected flat maps entity as at
least from to or dependency as we now have the grain both a type lvl and
runtime to do so
It should not be possible in the devxp to forgot adding the views to the
v2 builder when passing the view field anymore ( that would lead to
permanent validation error in view field integrity checks )
## Conclusion
Thanks for reading and reviewing !
Any suggestions are more than welcomed ! ( same as for questions too ! )
When starting the app on a fresh database reset the cache would be
filled with empty flat field metadata maps
Because the reset command hack through the repository directly in order
to create views and stuff
implemented an iso flush as the one existing initially
added it to a workspace deletion tambien
```
{
byId: {],
universalIdById: {}
}
```
# Introduction
Extracting the legacy fields build and dispatch out of the object one to
follow the generic flat entity build, also update caches entries for
object
## Main tasks
- flat field map cache
- flat field builder
- refactored the dispatch matrix to return flat entity maps instead of
flat entity arrays
- orchestrator aggregator
- removing legacy code
- making universal identifier aka standardId of standard field for
custom object deterministically dynamic
- perfs debug logs for v2
## TODO
- [x] Refactor the generic entity builder to be dependency flat maps in
order to main foreign keys list in flat parent
- [x] Refactor the flat object metadata to contain the array of related
fields and avoid costy find object fields
- [ ] Improve the create field handler to handle multiple field at once
- [ ] Refactor the dispatch to embbed the comparison
- [ ] Improve perf by extracting from elements out of existing
- [ ] Fix the labelIdentifierId validators on object before field
creation ( integ tests are in failing mode )
## Debug logs snippet
```ts
[EntityBuilder fieldMetadata] matrix computation: 0.027ms
[EntityBuilder fieldMetadata] creation validation: 0.001ms
[EntityBuilder fieldMetadata] deletion validation: 0.293ms
[EntityBuilder fieldMetadata] update validation: 0.006ms
[EntityBuilder fieldMetadata] entity processing: 0.363ms
[EntityBuilder fieldMetadata] validateAndBuild: 0.455ms
[EntityBuilder index] matrix computation: 0.005ms
[EntityBuilder index] creation validation: 0.001ms
[EntityBuilder index] deletion validation: 0.146ms
[EntityBuilder index] update validation: 0.004ms
[EntityBuilder index] entity processing: 0.199ms
[EntityBuilder index] validateAndBuild: 0.228ms
[Runner] Initial cache retrieval: 0.549ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_index executeForWorkspaceSchema: 11.665ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_index executeForMetadata: 12.864ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForWorkspaceSchema: 1.476ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForMetadata: 6.816ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForWorkspaceSchema: 0.062ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForMetadata: 0.889ms
[Runner] Transaction execution: 23.434ms
[Runner] Cache invalidation: 316.662ms
[Runner] Total execution: 340.767ms
```
As you can see cache invalidation is way to long, we could replace the
cache by the optimistic in the end
- add serverlessFunction schema in twenty-cli
- add trigger schema in twenty-cli
- update serverless function code save and get
- sync serverless function
## Context
Now adding serverless code sync within the migration v2 logic itself. To
do that we need to follow the
- Prepare flat input
- Build migration
- Run migration
steps where now the serverless function entity will store in DB and
cache a checksum of its code and the flat input will contain the code
with the checksum. Build will compare checksum and create an update
action containing the code if it has changed and the migration will now
run the corresponding services (instead of calling those in the parent
serverless service exposed in the API) allowing us to keep that logic
functional for other use cases such as import/export, twenty-cli and
twenty upgrades.
Paul Rastoin