# Introduction
Following recent `FlatObjectMetadataMaps` manipulation, transpilers
utils introduction in https://github.com/twentyhq/twenty/pull/13620
Refactored the field metadata service and validator to iterate over
`FlatObjectMetadataMaps` instead of `FlatObjectMetadata[]`.
The object metadata service v2 is shadow coding
In this PR
1. Fix delete and soft-delete repository methods for repositories where
permission checks are NOT bypassed: they need to have a selection of
columns to return by default. To match what we did for insert I set it
to `'*'` by default. But I feel this may be bug prone as developers will
not necessarily think to fill the right values in. Maybe we should
change the api to use selectable fields by default. @charlesBochet
2. Add field permission seeds and enable field permission feature flag
in dev
# Introduction
The cache is not accurately typed regarding the `fieldMetadataMaps` they
do not contain relations
In this way we:
- Created dedicated transpilation tools for cache manipulation
- Avoided recursivity in transpilation tools
- Fix and finalized the createMany fields metadata service v2
- Refactor the validator to handle validation :)
## What's next:
- Implem integration tests to make things run 🙃
- migrate existing settings valdiation
- Finish the create object metadata service
- Handle update input transpilation and validation
# Introduction
Following https://github.com/twentyhq/twenty/pull/13420
What has been done:
- `CreateFieldInput` transpilation to `FlatFieldMetadata`
- `FlatFieldMetadata` validator service
- A lof of transpilation utils from `input` to `flatObject` or
`flatField`
- Created dedicated v2 api metadata services
- Introducing `inferDeletionFromMissingObjectFieldIndex` in the builder,
to avoid diffing every object and field of the current workspace we
allow only generating create/update migration operations, usefull when
passing by the api metadata
## We still need to in another PR:
- Implement a strong unit test coverage and critical functions and
services
- Finalize flat field metadata validation exception for `options`
`defaultValue` `settings` and `relations`
- Finalize `flatObjectMetadata` validation and v2 service refactor
- Plug the new service when feature flag is enabled
# Introduction
In this PR we create basic transpilation methods and utils to handle
input to flat, entity to flat, object maps to flat. In order to
transpile everything into a common validation that will be implemented
in another PR
## FieldMetadataEntity typing
Added `never | null` to fields that should never be in order to ease
general abstracted method to pass null, as anw it's what is in the
database
## Todo
- ~~Create a feature flag~~
- Integration test for object creation through metadata api + pg col
introspection and snapshoting
Fixes https://github.com/twentyhq/twenty/issues/13398
This bug was introduced by https://github.com/twentyhq/twenty/pull/13215
The emails were empty in scenarios where the user wasn't authenticated
(reset passwords for instance) because in `hydrateGraphqlRequest` the
information about the locale was added only if the user was
authenticated `!this.isTokenPresent(request)`. So the locale was
undefined making ` i18n.activate(undefined) ` fail silently resulting in
an empty email.
# Why
If we have a Morph Relation, like :
Opportunity <-> Company & People
Let's say it's a MANY_TO_ONE on Opportunity side
Then we have two joinColumnNames looking like
- ownerPersonId
- ownerCompanyId
Let's say someone renames the obejct Person (assume we can even though
standard obejcts cannot be renames per say at the moment in the API)
We need to update the joinColumnName and create the associated
migrations
# Introduction
Created the runner metadata for the field and object
We should keep in mind that any runner handler will only iterate over an
atomic instance of entity ( object field index etc )
Never triggerring any side effect or whats over
In this way we will need to implement a deferred transaction, as for
when we create a relation we will inject to the first created field of
both the `relationTargetFieldMetadataId` deterministically computed
before its own creation
This would result in pg constraint brokage if not deferred
## Updates
- We decided gather create_fields under the create_object as they will
be building within the same sql query. This will ease both generation
and computation and avoid disassembling to reassemble afterwards
- Refactored FlatFieldMetadata to handle relation typing with flat
occurences
```ts
runCreateFieldSchemaMigration = async ({
action,
queryRunner,
}: WorkspaceMigrationActionRunnerArgs<CreateFieldAction>) => {
if (isFlatFieldMetadataEntityOfType(action.flatFieldMetadata, FieldMetadataType.RELATION)) {
action.flatFieldMetadata.flatRelationTargetObjectMetadata
}else {
action.flatFieldMetadata.flatRelationTargetObjectMetadata // tsc-error never
}
return;
};
```
## TODO
- ~~Discuss action signature with @Weiko in order to anticipate `schema`
runner needed grain~~
- ~~Refactor the object actions to contain picked `flatObjectMetadata`~~
- Implem index service
- Renamed `WorkflowActionAdapter` to `ToolExecutorWorkflowAction`
- Renamed `settingPermission` table to `permissionFlag` and `setting`
column to `flag`
- Decoupled the send email logic from workflows to tools
- Add new `Tools Permission` section in FE
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
# Introduction
Following https://github.com/twentyhq/twenty/pull/13264, this PR
introduces several `fieldMetadataEntity` typing enhancement suggestions.
Mainly any nullable field metadata entity properties are now either
nullable or defined.
Or never if field is dynamically required or not depending on the field
metadata type
This enhance DevX
## Standards
- field enum ( `MULTI_SELECT`, `SELECT`, `RATING` ) will never have
`options` set to `NULL` in db
- field `RELATION` or `MORH_RELATION` won't ever have its relation
fields set to `NULL` in db
- field of any type `settings`, even if possibly defined, can still be
`NULL` in db
- field of any type `defaultValue`, even if possibly defined, can still
be `NULL` in db
It coud be interesting to guard these standards by adding dedicated pg
constraints on each field
## TypesScript type tests
added coverage for each `settings`, `defaultValue`, and `options`
depending on the current `fieldMetadata`
Honestly I don' know if this typescript assertions test file is not
overkill, but regarding metadata staticness it might be very interesting
to have this guard
## Possible improvements
- We could type as `unknown` instead of "all" on `FieldMetadataType`
inferrance
- We still need to deprecate remaining duplicated entities such as
`Index/Field/MetadataInterface` etc not a huge refactor neither urgent
# Introduction
Following https://github.com/twentyhq/twenty/pull/13310
> After this PR merge will create a new one removing the type and
replacing it to ObjectMetadataEntity.
This is it !
# Introduction
Following `FieldMetadataInterface` deprecation in
https://github.com/twentyhq/twenty/pull/13264
As for the previous PR will rename and remove all the file in a
secondary PR to avoid conflicts and over loading this one
## Improvements
Removed optional properties from the `objectMetadataEntity` model and
added utils to retrieve test data
## Notes
By touching to `ObjectMetadataDTO` I would have expected a twenty-front
codegenerated types mutation, but it does not seem to be granular enough
to null/undefined coercion
# Introduction
Following https://github.com/twentyhq/twenty/pull/13264
> After this PR merge will create a new one removing the type and
replacing it to FieldMetadataEntity.
This is it !
# Introduction
From the moment replaced the FieldMetadataInterface definition to:
```ts
import { FieldMetadataType } from 'twenty-shared/types';
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
export type FieldMetadataInterface<
T extends FieldMetadataType = FieldMetadataType,
> = FieldMetadataEntity<T>;
```
After this PR merge will create a new one removing the type and
replacing it to `FieldMetadataEntity`.
Did not renamed it here to avoid conflicts on naming + type issues fixs
within the same PR
## Field metadata entity RELATION or MORPH
Relations fields cannot be null for those field metadata entity instance
anymore, but are never for the others see
`packages/twenty-server/src/engine/metadata-modules/field-metadata/types/field-metadata-entity-test.type.ts`
( introduced TypeScript tests )
## Concerns
- TS_VECTOR is the most at risk with the `generatedType` and
`asExpression` removal from interface
## What's next
- `FielMetadataInterface` removal and rename ( see introduction )
- Depcrecating `ObjectMetadataInterface`
- Refactor `FieldMetadataEntity` optional fiels to be nullable only
- TO DIG `never` occurences on settings, defaultValue etc
- Some interfaces will be replaced by the `FlatFieldMetadata` when
deprecating the current sync and comparators tools
In the BE we throw custom errors with precise error codes (e.g.
"LABEL_ALREADY_EXISTS") before catching them in filters and rethrowing
BaseGraphQLErrors (standard errors such as NotFoundError, UserInputError
etc.).
In the FE we were grouping sentries based on the error codes but we were
actually grouping by very broad codes such as "NOT_FOUND" or
"BAD_USER_INPUT", extracted from the BaseGraphQLErrors.
To fix that, we update the BaseGraphQLError constructor api to allow to
pass on the CustomError directly and retrieve from it the original code
and store it in existing property `subCode` that we will use in the FE
to send errors to sentry.
This new api also eases usage of `userFriendlyMessage` that is passed on
to the api response and therefore to the FE when CustomError is passed
on directly to the BaseGraphQLError constructor.
Fixes https://github.com/twentyhq/twenty/issues/12867
Issue:
when you have a variable `toto` which is: `Record<string, MyType>` and
you do toto['xxx'], this will be typed as `MyType` instead of `MyType |
undefined`
Solutions:
- activate `noUncheckedIndexedAccess` check in tsconfig, this is the
preferred solution but will take time to get there (this raises 600+
errors)
- use a Map: cf https://github.com/twentyhq/twenty/pull/13125/files
- set the type to Partial<Record<string, MyType>>. Drawback is that when
you do Object.values(toto), you'll get `Array<MyType | undefined>`.
Hence why we have to filter these behind
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/d0a0bfed-c441-4e53-84c2-2da98ccbcf50"
/>
In this PR
- introduction of fieldPermission entity
- addition of upsertFieldPermission in role resolver
- computing of permissions taking fieldPermission into account. In order
to limit what is stored in Redis we only store fields restrictions. For
instance for objectMetadata with id XXX with a restriction on field with
id YYY we store:
`"XXX":{"canRead":true,"canUpdate":false,"canSoftDelete":false,"canDestroy":false,"restrictedFields":{"YYY":{"canRead":false,"canUpdate":null}}}`
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
Currently, when a server query or mutation from the front-end fails, the
error message defined server-side is displayed in a snackbar in the
front-end.
These error messages usually contain technical details that don't belong
to the user interface, such as "ObjectMetadataCollection not found" or
"invalid ENUM value for ...".
**BE**
In addition to the original error message that is still needed (for the
request response, debugging, sentry monitoring etc.), we add a
`displayedErrorMessage` that will be used in the snackbars. It's only
relevant to add it for the messages that will reach the FE (ie. not in
jobs or in rest api for instance) and if it can help the user sort out /
fix things (ie. we do add displayedErrorMessage for "Cannot create
multiple draft versions for the same workflow" or "Cannot delete
[field], please update the label identifier field first", but not
"Object metadata does not exist"), even if in practice in the FE users
should not be able to perform an action that will not work (ie should
not be able to save creation of multiple draft versions of the same
workflows).
**FE**
To ease the usage we replaced enqueueSnackBar with enqueueErrorSnackBar
and enqueueSuccessSnackBar with an api that only requires to pass on the
error.
If no displayedErrorMessage is specified then the default error message
is `An error occured.`
Modifying the data-model can sometimes fail in the middle of your
operation, due to the way we handle both metadata update and schema
migration separately, a field can be created while the associated column
creation failed (same for object/table and such). This is also an issue
because WorkspaceMigrations are then stored as FAILED can never really
recovered by themselves so the schema is broken and we can't update the
models anymore.
This PR adds a executeMigrationFromPendingMigrationsWithinTransaction
method where we can (and must) pass a queryRunner executing a
transaction, which should come from the metadata services so that if
anything during metadata update OR schema update fails, it rolls back
everything (this also mean a workspaceMigration should never stay in a
failed state now).
This also fixes some issues with migration not running in the correct
order due to having the same timestamp and having to do some weird logic
to fix that.
This is a first step and fix before working on a much more reliable
solution in the upcoming weeks where we will refactor the way we
interact with the data model.
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
Better catching label input
- there were absolutely no check on label when creating the target field
while doing a relation : we crearted these checks here.
- We keep the label quite open to special char as discussed with Felix.
so mostly checking length of label.
- We check that label does not already exists on the targetted object
- making sure the Target fieldinput label is checked before we create
it. The previous checks are not enough since the label goes through
anoteher merthod before going in the database
- validate-metadata-name-is-camel-case.utils.ts : making sure we can use
this error message for metadata name and for target label
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Co-authored-by: prastoin <paul@twenty.com>
In this PR:
- add query hashKey to ObjectMetadataItems query graphql cache to avoid
caching outdated queries
- improve performance by removing ResolveField at FieldLevel and adding
this at resolver level
In this PR:
## Improve recompute metadata cache performance. We are aiming for
~100ms
Deleting relationMetadata table and FKs pointing on it
Fetching indexMetadata and indexFieldMetadata in a separate query as
typeorm is suboptimizing
## Remove caching lock
As recomputing the metadata cache is lighter, we try to stop preventing
multiple concurrent computations. This also simplifies interfaces
## Introduce self recovery mecanisms to recompute cache automatically if
corrupted
Aka getFreshObjectMetadataMaps
## custom object resolver performance improvement: 1sec to 200ms
Double check queries and indexes used while creating a custom object
Remove the queries to db to use the cached objectMetadataMap
## reduce objectMetadataMaps to 500kb
<img width="222" alt="image"
src="https://github.com/user-attachments/assets/2370dc80-49b6-4b63-8d5e-30c5ebdaa062"
/>
We used to stored 3 fieldMetadataMaps (byId, byName, byJoinColumnName).
While this is great for devXP, this is not great for performances.
Using the same mecanisme as for objectMetadataMap: we only keep byIdMap
and introduce two otherMaps to idByName, idByJoinColumnName to make the
bridge
## Add dataloader on IndexMetadata (aka indexMetadataList in the API)
## Improve field resolver performances too
## Deprecate ClientConfig
We were using a global ValidationPipe in main.ts. This is an issue as
@Controllers should return HttpExecption and @Resolvers should return
GraphqlErrors
Removing the global pipe and creating a ResolverValidationPipe able to
generate GraphqlError. We also need to handle the exception in a filter
to avoid nest to think it's unhandled and make it flow to logs
Next step:
- it would be nice to have both @UsePipes(ResolverValidationPipe) +
@UseFilters(GraphqlValidationExceptionFilter) come together. This should
be possible if we create a @GraphQLResolver annotation
- Fix an issue where custom object were seeded with 2 views, and with
the wrong icon
- ACME becomes YCombinator
- Allow 2 workspaces to have different metadata seeded
- Add many seeds for messages
- Add many seeds for calendar events
- Randomize createdBy for person and companies
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Update the default set of system fields for custom objects, to ensure
position is not nullabel and has a default value to 0
Steps to reproduce :
create a custom object,
send a POST request with body ```{position:null}```
the record should be created
After the change,
an error will be thrown
<img width="754" alt="Screenshot 2025-06-13 at 17 16 56"
src="https://github.com/user-attachments/assets/d40931f7-16cc-4b68-8dbb-deb0fa292be5"
/>
# What
Fully deprecate old relations because we have one bug tied to it and it
make the codebase complex
# How I've made this PR:
1. remove metadata datasource (we only keep 'core') => this was causing
extra complexity in the refactor + flaky reset
2. merge dev and demo datasets => as I needed to update the tests which
is very painful, I don't want to do it twice
3. remove all code tied to RELATION_METADATA /
relation-metadata.resolver, or anything tied to the old relation system
4. Remove ONE_TO_ONE and MANY_TO_MANY that are not supported
5. fix impacts on the different areas : see functional testing below
# Functional testing
## Functional testing from the front-end:
1. Database Reset ✅
2. Sign In ✅
3. Workspace sign-up ✅
5. Browsing table / kanban / show ✅
6. Assigning a record in a one to many / in a many to one ✅
7. Deleting a record involved in a relation ✅ => broken but not tied to
this PR
8. "Add new" from relation picker ✅ => broken but not tied to this PR
9. Creating a Task / Note, Updating a Task / Note relations, Deleting a
Task / Note (from table, show page, right drawer) ✅ => broken but not
tied to this PR
10. creating a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
11. updating a relation from settings should not be possible ✅
12. deleting a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
13. Make sure timeline activity still work (relation were involved
there), espacially with Task / Note => to be double checked ✅ => Cannot
convert undefined or null to object
14. Workspace deletion / User deletion ✅
15. CSV Import should keep working ✅
16. Permissions: I have tested without permissions V2 as it's still hard
to test v2 work and it's not in prod yet ✅
17. Workflows global test ✅
## From the API:
1. Review open-api documentation (REST) ✅
2. Make sure REST Api are still able to fetch relations ==> won't do, we
have a coupling Get/Update/Create there, this requires refactoring
3. Make sure REST Api is still able to update / remove relation => won't
do same
## Automated tests
1. lint + typescript ✅
2. front unit tests: ✅
3. server unit tests 2 ✅
4. front stories: ✅
5. server integration: ✅
6. chromatic check : expected 0
7. e2e check : expected no more that current failures
## Remove // Todos
1. All are captured by functional tests above, nothing additional to do
## (Un)related regressions
1. Table loading state is not working anymore, we see the empty state
before table content
2. Filtering by Creator Tim Ap return empty results
3. Not possible to add Tasks / Notes / Files from show page
# Result
## New seeds that can be easily extended
<img width="1920" alt="image"
src="https://github.com/user-attachments/assets/d290d130-2a5f-44e6-b419-7e42a89eec4b"
/>
## -5k lines of code
## No more 'metadata' dataSource (we only have 'core)
## No more relationMetadata (I haven't drop the table yet it's not
referenced in the code anymore)
## We are ready to fix the 6 months lag between current API results and
our mocked tests
## No more bug on relation creation / deletion
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
# Introduction
Added a no-explicit-any rule to the twenty-server, not applicable to
tests and integration tests folder
Related to https://github.com/twentyhq/core-team-issues/issues/975
Discussed with Charles
## In case of conflicts
Until this is approved I won't rebased and handle conflict, just need to
drop two latest commits and re run the scripts etc
## Legacy
We decided not to handle the existing lint error occurrences and
programmatically ignored them through a disable next line rule comment
## Open question
We might wanna activate the
[no-explicit-any](https://typescript-eslint.io/rules/no-explicit-any/)
`ignoreRestArgs` for our use case ?
```
ignoreRestArgs?: boolean;
```
---------
Co-authored-by: etiennejouan <jouan.etienne@gmail.com>
This PR attemps at improving sentry grouping and filtering by
- Using the exceptionCode as the fingerprint when the error is a
customException. For this to work in this PR we are now throwing
customExceptions instead of internalServerError deprived of their code.
They will still be converted to Internal server errors when sent back as
response
- Filtering 4xx issues where it was missing (for emailVerification
because errors were not handled, for invalid captcha and billing errors
because they are httpErrors and not graphqlErrors)
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
This is a first PR to remove old relation logic
Next steps:
- remove relationMetadata from cache
- remove relationMetadata table content and structure
- refactor relationDefinition to leverage field.settings instead
In this PR:
- this should fix the sync metadata for new relation system
This goes with the recent PR:
https://github.com/twentyhq/twenty/pull/11725
What we want:
- ONE_TO_MANY relations should have no joinColumn and no onDelete
- MANY_TO_ONE should have both
leave intact the `input` argument to avoid side effects on the parent
caller
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
In this PR:
- Remove deactivated objects from ActivityTargetInlineCell record picker
- Prevent users to deactivate createdAt, updatedAt, deletedAt fields on
any objects
Still left:
- write unit tests on the assert utils
- write integration tests on field metadata service
- prevent users to deactivate createdAt, updatedAt, deletedAt on FE
Félix Malfait