# Introduction
From the moment replaced the FieldMetadataInterface definition to:
```ts
import { FieldMetadataType } from 'twenty-shared/types';
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
export type FieldMetadataInterface<
T extends FieldMetadataType = FieldMetadataType,
> = FieldMetadataEntity<T>;
```
After this PR merge will create a new one removing the type and
replacing it to `FieldMetadataEntity`.
Did not renamed it here to avoid conflicts on naming + type issues fixs
within the same PR
## Field metadata entity RELATION or MORPH
Relations fields cannot be null for those field metadata entity instance
anymore, but are never for the others see
`packages/twenty-server/src/engine/metadata-modules/field-metadata/types/field-metadata-entity-test.type.ts`
( introduced TypeScript tests )
## Concerns
- TS_VECTOR is the most at risk with the `generatedType` and
`asExpression` removal from interface
## What's next
- `FielMetadataInterface` removal and rename ( see introduction )
- Depcrecating `ObjectMetadataInterface`
- Refactor `FieldMetadataEntity` optional fiels to be nullable only
- TO DIG `never` occurences on settings, defaultValue etc
- Some interfaces will be replaced by the `FlatFieldMetadata` when
deprecating the current sync and comparators tools
Context:
Users are complaining to see their workspace in a language they don't
know. This behavior is transient, happens on data model update and
disappear on refresh
I've check the cache for users that got the issue and did not spot any
weird language
==> I think we somehow fallback the the request header locale. I feel we
should always use the userWorkspace.locale, request locale should not be
used in BE in my opinion except for unauthenticated endpoints. I'm also
adding logs to understand the locale issue
In this PR:
rename user.workspaces into user.userWorkspaces which is more correct
improve / simplify LOCALES typing
This PR does not produce any functional changes for our users. It
prepares the branches for workflows by:
- decommissioning `output` and `context` fields or `workflowRun` records
and use newly created `state` field from front-end and back-end
- use `stepStatus` computed by `back-end` in `front-end`
- add utils and types in `twenty-shared/workflow` (not completed, a
follow-up is scheduled
https://github.com/twentyhq/core-team-issues/issues/1211)
- add concurrency to `workflowQueue` message queue to avoid weird branch
execution when using forms in workflow branches
- add a WithLock decorator for better dev experience of
`CacheLockService.withLock` usage
Here is an example of such a workflow running (front branch display is
not yet done that's why it looks ugly) ->
https://discord.com/channels/1130383047699738754/1258024460238192691/1392897615171158098
This PR does not produce any functional change
First step of the workflow branch feature
- add gather `workflowRun.output` and `workflowRun.context` into one
column `workflowRun.runContext`
- add a command to fill `runContext` from `output` and `context` in
existing records
- maintain `runContext` up to date during workflow runs
- new status `ENQUEUED` added. With a command to backfill
- counter in cache per workspace, managed by a new service
[workflow-run-queue.workspace-service.ts](https://github.com/twentyhq/twenty/compare/tt-improve-workflow-run-queueing?expand=1#diff-1e2de2a48cd482a3bd7e8dedf1150a19d0b200afbd9282181a24ecddddb56927)
- cron added that will run every minute to look for not started
workflows
Here is the new flow:
- When executing a workflow, we check if the queue is not full. If not,
run is created as `ENQUEUED` and the run workflow job is triggered as
usual. If full, create the run as NOT_STARTED and do not trigger the job
- Cron will look for NOT_STARTED workflows and queue some if there is
some place again
- Only MANUAL and Form submit skip the queue limit
In this PR:
## Improve recompute metadata cache performance. We are aiming for
~100ms
Deleting relationMetadata table and FKs pointing on it
Fetching indexMetadata and indexFieldMetadata in a separate query as
typeorm is suboptimizing
## Remove caching lock
As recomputing the metadata cache is lighter, we try to stop preventing
multiple concurrent computations. This also simplifies interfaces
## Introduce self recovery mecanisms to recompute cache automatically if
corrupted
Aka getFreshObjectMetadataMaps
## custom object resolver performance improvement: 1sec to 200ms
Double check queries and indexes used while creating a custom object
Remove the queries to db to use the cached objectMetadataMap
## reduce objectMetadataMaps to 500kb
<img width="222" alt="image"
src="https://github.com/user-attachments/assets/2370dc80-49b6-4b63-8d5e-30c5ebdaa062"
/>
We used to stored 3 fieldMetadataMaps (byId, byName, byJoinColumnName).
While this is great for devXP, this is not great for performances.
Using the same mecanisme as for objectMetadataMap: we only keep byIdMap
and introduce two otherMaps to idByName, idByJoinColumnName to make the
bridge
## Add dataloader on IndexMetadata (aka indexMetadataList in the API)
## Improve field resolver performances too
## Deprecate ClientConfig
- Fix an issue where custom object were seeded with 2 views, and with
the wrong icon
- ACME becomes YCombinator
- Allow 2 workspaces to have different metadata seeded
- Add many seeds for messages
- Add many seeds for calendar events
- Randomize createdBy for person and companies
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
In this PR
1. fix workflow step creation by adding forgotten
`shouldBypassPermissionChecks` in WorkflowVersionStepWorkspaceService
2. clarify the rule for twentyORMGlobalManager: do not add unnecessary
`shouldBypassPermissionChecks` for system objects (there are no
object-records permission checks on system objects, they are dealt with
at resolver level)
# What
Fully deprecate old relations because we have one bug tied to it and it
make the codebase complex
# How I've made this PR:
1. remove metadata datasource (we only keep 'core') => this was causing
extra complexity in the refactor + flaky reset
2. merge dev and demo datasets => as I needed to update the tests which
is very painful, I don't want to do it twice
3. remove all code tied to RELATION_METADATA /
relation-metadata.resolver, or anything tied to the old relation system
4. Remove ONE_TO_ONE and MANY_TO_MANY that are not supported
5. fix impacts on the different areas : see functional testing below
# Functional testing
## Functional testing from the front-end:
1. Database Reset ✅
2. Sign In ✅
3. Workspace sign-up ✅
5. Browsing table / kanban / show ✅
6. Assigning a record in a one to many / in a many to one ✅
7. Deleting a record involved in a relation ✅ => broken but not tied to
this PR
8. "Add new" from relation picker ✅ => broken but not tied to this PR
9. Creating a Task / Note, Updating a Task / Note relations, Deleting a
Task / Note (from table, show page, right drawer) ✅ => broken but not
tied to this PR
10. creating a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
11. updating a relation from settings should not be possible ✅
12. deleting a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
13. Make sure timeline activity still work (relation were involved
there), espacially with Task / Note => to be double checked ✅ => Cannot
convert undefined or null to object
14. Workspace deletion / User deletion ✅
15. CSV Import should keep working ✅
16. Permissions: I have tested without permissions V2 as it's still hard
to test v2 work and it's not in prod yet ✅
17. Workflows global test ✅
## From the API:
1. Review open-api documentation (REST) ✅
2. Make sure REST Api are still able to fetch relations ==> won't do, we
have a coupling Get/Update/Create there, this requires refactoring
3. Make sure REST Api is still able to update / remove relation => won't
do same
## Automated tests
1. lint + typescript ✅
2. front unit tests: ✅
3. server unit tests 2 ✅
4. front stories: ✅
5. server integration: ✅
6. chromatic check : expected 0
7. e2e check : expected no more that current failures
## Remove // Todos
1. All are captured by functional tests above, nothing additional to do
## (Un)related regressions
1. Table loading state is not working anymore, we see the empty state
before table content
2. Filtering by Creator Tim Ap return empty results
3. Not possible to add Tasks / Notes / Files from show page
# Result
## New seeds that can be easily extended
<img width="1920" alt="image"
src="https://github.com/user-attachments/assets/d290d130-2a5f-44e6-b419-7e42a89eec4b"
/>
## -5k lines of code
## No more 'metadata' dataSource (we only have 'core)
## No more relationMetadata (I haven't drop the table yet it's not
referenced in the code anymore)
## We are ready to fix the 6 months lag between current API results and
our mocked tests
## No more bug on relation creation / deletion
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
BlocknoteJS requires an ESM module where our server is CJS, this forced
us to pin the server-util version, which led us to force the resolution
of several packages, leading to bugs downstream.
From Node 22.12 Node supports requiring ESM modules (available from Node
22.0 with a flag). So I upgrade the module.
I picked Node 22 and not Node 23 or Node 24 because 22 is the LTS and we
don't plan to change node versions frequently.
If you remain on Node 18, things should still mostly work, except if you
edit a Rich Text field.
I also starting changing the default runtime for Serverless Functions
which isn't directly related. This means new serverless functions will
be created on Node 22, but we will still need another PR to migrate
existing serverless functions before September (end of support by AWS).
(In this PR I also remove the upgrade commands from 0.43 since they rely
on Blocknote and I didn't want to have to deal with this)
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
I am seeing an issue where this migrations fails because the
`metadata._typeorm_migrations` table does not exist.
```pgsql
copy _typeorm_migrations from metadata to core
query failed: SELECT * FROM metadata._typeorm_migrations ORDER BY id ASC
error: error: relation "metadata._typeorm_migrations" does not exist
[Nest] 430 - 06/01/2025, 10:22:35 PM ERROR [CopyTypeormMigrationsCommand] Failed to copy migrations: relation "metadata._typeorm_migrations" does not exist
[Nest] 430 - 06/01/2025, 10:22:35 PM ERROR [CopyTypeormMigrationsCommand] undefined
[Nest] 430 - 06/01/2025, 10:22:35 PM ERROR [DatabaseMigrationService] Error running database migrations:
[Nest] 430 - 06/01/2025, 10:22:35 PM ERROR [DatabaseMigrationService] QueryFailedError: relation "metadata._typeorm_migrations" does not exist
[Nest] 430 - 06/01/2025, 10:22:35 PM ERROR [UpgradeCommand] Command failed
[Nest] 430 - 06/01/2025, 10:22:35 PM ERROR [UpgradeCommand] undefined
[Nest] 430 - 06/01/2025, 10:22:35 PM LOG [UpgradeCommand] Command completed!
[Nest] 430 - 06/01/2025, 10:22:35 PM ERROR [QueryFailedError] relation "metadata._typeorm_migrations" does not exist
```
I _think_ this table is not meant to exist anymore - which means that
anyone who is onboarding into the project will run into an issue unless
we handle the case where the table doesn't exist.
We need to handle both the existing case and the non existing case to
support people who _do_ have metadata._typeorm_migrations` to migrate.
Closes https://github.com/twentyhq/core-team-issues/issues/748
In the frame of the work on permissions we
- remove all raw queries possible to use repositories instead
- forbid usage workspaceDataSource.executeRawQueries()
- restrict usage of workspaceDataSource.query() to force developers to
pass on shouldBypassPermissionChecks to use it.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
### Solution
> After discussion with charles & weiko, we chose the long term
solution.
>
> Fix FE to request checkUserExists resolver with lowercased emails
> Add a decorator on User (and AppToken for invitation), to lowercase
email at user (appToken) creation. ⚠️ It works for TypeOrm .save method
only (there is no user email update in codebase, but in future it
could..)
> Add email lowercasing logic in external auth controller
> Fix FE to request sendInvitations resolver with lowercased emails
> Add migration command to lowercase all existing user emails and
invitation emails
> For other BE resolvers, we let them permissive. For example, if you
made a request on CheckUserExists resolver with uppercased email, you
will not found any user. We will not transform input before checking for
existence.
[link to comment
](https://github.com/twentyhq/twenty/pull/12130#discussion_r2098062093)
### Test 🚧
- sign-in and up from main subdomain and workspace sub domain > Google
Auth (lowercased email) ✔️ | Microsoft Auth (uppercased email ✔️ &
lowercased email) | LoginPassword (uppercased email ✔️& lowercased
email✔️)
- invite flow with uppercased and lowercased ✔️
- migration command + sign-in ( former uppercased microsoft email ✔️) /
sign-up ( former uppercased invited email ✔️)
closes https://github.com/twentyhq/private-issues/issues/278, closes
https://github.com/twentyhq/private-issues/issues/275, closes
https://github.com/twentyhq/private-issues/issues/279
- fix missing createBy injection in api createOne and createMany
endpoints
- add a command to fix null default value for createdBySource in
production entities
- tested on `1747159401197/` dump extract of production db without issue
# Introduction
Added a no-explicit-any rule to the twenty-server, not applicable to
tests and integration tests folder
Related to https://github.com/twentyhq/core-team-issues/issues/975
Discussed with Charles
## In case of conflicts
Until this is approved I won't rebased and handle conflict, just need to
drop two latest commits and re run the scripts etc
## Legacy
We decided not to handle the existing lint error occurrences and
programmatically ignored them through a disable next line rule comment
## Open question
We might wanna activate the
[no-explicit-any](https://typescript-eslint.io/rules/no-explicit-any/)
`ignoreRestArgs` for our use case ?
```
ignoreRestArgs?: boolean;
```
---------
Co-authored-by: etiennejouan <jouan.etienne@gmail.com>
In this PR:
- Set the default position for the DONE option of the task's status
option to `2` instead of `1`, which was the same as `IN_PROGRESS`
option's position.
- Write a command to prevent position duplicates in the database for the
task's status field.
What I've checked before setting this PR as ready to be reviewed:
- De-duplicating the position solves the issue and it's possible to edit
the field (solves the related issue)
- The upgrade command de-duplicates the position for each workspace.
There are no more DONE options with `position=2`. I ran the upgrade
command on the `database-snapshot-manager` dataset.
- Suspended workspaces aren't fixed
---
To test the script:
```ts
const scannedPositions = new Set();
let biggestPosition = -1;
// Sort options by position for consistent processing
const sortedOptions = [
{ name: 'a', position: 2 },
{ name: 'b', position: 1 },
{ name: 'c', position: 1 },
{ name: 'd', position: 2 },
].sort((a, b) => a.position - b.position);
for (const option of sortedOptions) {
if (scannedPositions.has(option.position)) {
option.position = biggestPosition + 1;
}
biggestPosition = Math.max(biggestPosition, option.position);
scannedPositions.add(option.position);
}
console.log('Sorted options:', sortedOptions);
```
Closes https://github.com/twentyhq/twenty/issues/11790
This is a first PR to remove old relation logic
Next steps:
- remove relationMetadata from cache
- remove relationMetadata table content and structure
- refactor relationDefinition to leverage field.settings instead
Weiko