Commit Graph

311 Commits

Author SHA1 Message Date
Weiko bd8ed03990 Add TTL eviction to local data cache (#16510)
We keep different versions of our cache to avoid race conditions but we
never evict stale data. This PR should fix that
2025-12-12 10:14:46 +01:00
Weiko 083a038f8a Deprecate workspace datasoure (#16507) 2025-12-11 18:27:49 +01:00
Guillim b83dc3aaff TimelineActivity migration to morph (#15652)
# TimelineActivity migration to morph


- Creates `timelineActivities2` relations on Company, Dashboard, Note,
Opportunity, Person, Task, Workflow, WorkflowRun, and WorkflowVersion
entities with proper metadata and cascade delete behavior.

It was required to create standard fields as well since the
mapObjectMetadataByUniqueIdentifier needs it. otherwise the fields won't
be considered

- Feature Flag `IS_TIMELINE_ACTIVITY_MIGRATED` necessary to have the two
states in parallel. It is used as a stamp once the migration has been
run

- Migration is done using the coreDataSource. Why ?
even though is unsafe to use, the first implementation of the migration
took forever on each workspace. See [this
commit](https://github.com/twentyhq/twenty/pull/15652/commits/477011e8d7d4c580f79ba7ec4a8fb002a3ec86b2)

The plan for this complex migration is as follows :

![plan](https://github.com/user-attachments/assets/51c63ea6-fb0d-40b4-b99d-3e0b35a204e6)
Note: we will need to rename fields in the release 1.12 (there is no
easy way to do all this in one release)
2025-12-11 14:28:26 +01:00
Weiko 9bd8f94b3a Refactor global datasource part 3 (#16447)
## Context
Following https://github.com/twentyhq/twenty/pull/16399
Now using the new global orm manager everywhere and returning a
GlobalDatasource/WorkspaceDatasource based on a feature flag.
This means we now need to wrap all our ORM calls within
executeInWorkspaceContext callback (at least for now) so the global
datasource can dynamically hydrate its context via the new store (the
global datasource does not store anything related to workspaces as it is
now a unique singleton). If feature flag is off it still uses local data
stored in the workspace datasource.
2025-12-10 17:17:33 +01:00
Paul Rastoin 7b98804ec1 Fix flat field metadata date type (#16445)
# Introduction
Following https://github.com/twentyhq/twenty/pull/16420
2025-12-09 19:53:36 +01:00
Charles Bochet 83fc434c5d Fix workspace invite onboarding loop (#16444)
- The issue is that the listener is not triggered if they are no change
to the workspaceMember (which happens when name is already filled
through GoogleSSO)

Fixes https://github.com/twentyhq/twenty/issues/16440
2025-12-09 19:18:17 +01:00
Charles Bochet a18203934c Fix flat entity maps date serialization (#16420)
Changes:
- as we store date in redis as serialized, let's make all flatEntity
dates as string. This requires changing FlatEntity types and making sure
that entity are converted to flatEntity and flatEntity to dtos
2025-12-09 16:41:50 +01:00
Paul Rastoin 4996f3dd28 Finalize twenty standard app as workspace migration object and fields (#16353)
# Introduction
Related to https://github.com/twentyhq/core-team-issues/issues/1995
In this PR we're fixing the remaining object/fields validation errors
resulting from standard objects and fields now passing a validation that
wasn't when using the sync metadata

## Key Changes
- **Field naming**: Renamed `iCalUID` to `iCalUid` for consistent
camelCase convention across calendar events
- **Enum standardization**: Uppercased enum values for message channels
(email→EMAIL), message participants (from→FROM, to→TO, cc→CC, bcc→BCC),
and message direction (incoming→INCOMING, outgoing→OUTGOING)
- **Label simplification**: Removed example values from workspace member
number format labels for cleaner UI
- **Migration infrastructure**: Added `isSystemBuild` flag throughout
field metadata service pipeline to allow system-level updates of
standard fields that bypass normal restrictions

## Migrating the existing data
We've created an upgrade command that will identify using the existing
object and field standard id field that needs to be updated, even though
the sync metadata still in usage could have fix them ( and the goal is
to deprecate it by the end of the sprint )
We will call the updateOneField for each of them, we're passing by the
field service in order to battle test what are going to be the temporary
way to handle standard migrations when we will start deprecating the
sync metadata but haven't still refactored the v2 workspace migration to
be workspace agnostic

## Twenty eng migration
Tested the whole migration + upgrade on twenty eng
Here are generated workspace migration
Records are handled natively gracefully too

### ICalUid
```json
{
  "status": "success",
  "workspaceMigration": {
    "relatedFlatEntityMapsKeys": [
      "flatFieldMetadataMaps",
      "flatIndexMaps",
      "flatViewFilterMaps",
      "flatViewGroupMaps",
      "flatViewMaps",
      "flatViewFieldMaps",
      "flatObjectMetadataMaps"
    ],
    "actions": [
      {
        "type": "update_field",
        "fieldMetadataId": "",
        "objectMetadataId": "",
        "updates": [
          {
            "from": "iCalUID",
            "to": "iCalUid",
            "property": "name"
          }
        ]
      }
    ],
    "workspaceId": ""
  }
}
```

### Incoming Outgoing
None as already caps in database somehow
```json
{
  "status": "success",
  "workspaceMigration": {
    "relatedFlatEntityMapsKeys": [
      "flatFieldMetadataMaps",
      "flatIndexMaps",
      "flatViewFilterMaps",
      "flatViewGroupMaps",
      "flatViewMaps",
      "flatViewFieldMaps",
      "flatObjectMetadataMaps"
    ],
    "actions": [],
    "workspaceId": ""
  }
}
```

### EMAIL
```json
{
  "status": "success",
  "workspaceMigration": {
    "relatedFlatEntityMapsKeys": [
      "flatFieldMetadataMaps",
      "flatIndexMaps",
      "flatViewFilterMaps",
      "flatViewGroupMaps",
      "flatViewMaps",
      "flatViewFieldMaps",
      "flatObjectMetadataMaps"
    ],
    "actions": [
      {
        "type": "update_field",
        "fieldMetadataId": "",
        "objectMetadataId": "",
        "updates": [
          {
            "from": "'email'",
            "to": "'EMAIL'",
            "property": "defaultValue"
          },
          {
            "from": [
              {
                "color": "green",
                "id": "",
                "label": "Email",
                "position": 0,
                "value": "email"
              },
              {
                "color": "blue",
                "id": "",
                "label": "SMS",
                "position": 1,
                "value": "sms"
              }
            ],
            "to": [
              {
                "color": "green",
                "id": "",
                "label": "Email",
                "position": 0,
                "value": "EMAIL"
              },
              {
                "color": "blue",
                "id": "",
                "label": "SMS",
                "position": 1,
                "value": "SMS"
              }
            ],
            "property": "options"
          }
        ]
      }
    ],
    "workspaceId": "e"
  }
}
```

### MessageParticipantRole
```json
{
  "status": "success",
  "workspaceMigration": {
    "relatedFlatEntityMapsKeys": [
      "flatFieldMetadataMaps",
      "flatIndexMaps",
      "flatViewFilterMaps",
      "flatViewGroupMaps",
      "flatViewMaps",
      "flatViewFieldMaps",
      "flatObjectMetadataMaps"
    ],
    "actions": [
      {
        "type": "update_field",
        "fieldMetadataId": "",
        "objectMetadataId": "",
        "updates": [
          {
            "from": "'from'",
            "to": "'FROM'",
            "property": "defaultValue"
          },
          {
            "from": [
              {
                "color": "green",
                "id": "",
                "label": "From",
                "position": 0,
                "value": "from"
              },
              {
                "color": "blue",
                "id": "",
                "label": "To",
                "position": 1,
                "value": "to"
              },
              {
                "color": "orange",
                "id": "",
                "label": "Cc",
                "position": 2,
                "value": "cc"
              },
              {
                "color": "red",
                "id": "",
                "label": "Bcc",
                "position": 3,
                "value": "bcc"
              }
            ],
            "to": [
              {
                "color": "green",
                "id": "",
                "label": "From",
                "position": 0,
                "value": "FROM"
              },
              {
                "color": "blue",
                "id": "",
                "label": "To",
                "position": 1,
                "value": "TO"
              },
              {
                "color": "orange",
                "id": "",
                "label": "Cc",
                "position": 2,
                "value": "CC"
              },
              {
                "color": "red",
                "id": "",
                "label": "Bcc",
                "position": 3,
                "value": "BCC"
              }
            ],
            "property": "options"
          }
        ]
      }
    ],
    "workspaceId": ""
  }
}
```

### Workspace member number format labels
```json
{
  "status": "success",
  "workspaceMigration": {
    "relatedFlatEntityMapsKeys": [
      "flatFieldMetadataMaps",
      "flatIndexMaps",
      "flatViewFilterMaps",
      "flatViewGroupMaps",
      "flatViewMaps",
      "flatViewFieldMaps",
      "flatObjectMetadataMaps"
    ],
    "actions": [
      {
        "type": "update_field",
        "fieldMetadataId": "",
        "objectMetadataId": "",
        "updates": [
          {
            "from": [
              {
                "color": "turquoise",
                "id": "",
                "label": "System",
                "position": 0,
                "value": "SYSTEM"
              },
              {
                "color": "blue",
                "id": "",
                "label": "Commas and dot (1,234.56)",
                "position": 1,
                "value": "COMMAS_AND_DOT"
              },
              {
                "color": "green",
                "id": "",
                "label": "Spaces and comma (1 234,56)",
                "position": 2,
                "value": "SPACES_AND_COMMA"
              },
              {
                "color": "orange",
                "id": "",
                "label": "Dots and comma (1.234,56)",
                "position": 3,
                "value": "DOTS_AND_COMMA"
              },
              {
                "color": "purple",
                "id": "",
                "label": "Apostrophe and dot (1'234.56)",
                "position": 4,
                "value": "APOSTROPHE_AND_DOT"
              }
            ],
            "to": [
              {
                "color": "turquoise",
                "id": "",
                "label": "System",
                "position": 0,
                "value": "SYSTEM"
              },
              {
                "color": "blue",
                "id": "",
                "label": "Commas and dot",
                "position": 1,
                "value": "COMMAS_AND_DOT"
              },
              {
                "color": "green",
                "id": "",
                "label": "Spaces and comma",
                "position": 2,
                "value": "SPACES_AND_COMMA"
              },
              {
                "color": "orange",
                "id": "",
                "label": "Dots and comma",
                "position": 3,
                "value": "DOTS_AND_COMMA"
              },
              {
                "color": "purple",
                "id": "",
                "label": "Apostrophe and dot",
                "position": 4,
                "value": "APOSTROPHE_AND_DOT"
              }
            ],
            "property": "options"
          }
        ]
      }
    ],
    "workspaceId": ""
  }
}
```
2025-12-09 14:09:48 +01:00
Etienne be91d870f9 Fix - revert standard sync logic on index + fix command (#16421) 2025-12-09 12:54:01 +00:00
Etienne 07cfaa78ef Fix unique standard field (#16371)
Fixes https://github.com/twentyhq/twenty/issues/15925

- update field metadata update logic
- uniformize the way index are named
- command to migrate v1-named unique index
- add integration testing

---------

Co-authored-by: prastoin <paul@twenty.com>
2025-12-08 18:05:28 +00:00
Weiko 859004f4fc Refactor global datasource part 2 (#16399)
## Context
Deprecating TwentyORMManager in favor of TwentyORMGlobalManager
(temporarily, as this will simplify the ultimate goal to later replace
all usages with the new TwentyORMGlobalManagerV2 which will have a
similar signature)
This means this PR had to refactor a bit of code to pass down the
workspaceId when not available directly as it is now a requirement,
meaning we also deprecated scopedWorkspaceContextFactory to have a less
obscure way to fetch the workspaceId and have something more
declarative.

Step 3 will be to update TwentyORMGlobalManager to use a featureFlag
toggling and use the new GlobalWorkspaceOrmManager internally using the
new cache service

Step 4 will be to remove the feature flag and pg_pool patch
2025-12-08 17:05:00 +01:00
Charles Bochet 2790d5dd93 Message fixes (#16389)
In this PR:
- add more logs to troubleshoot issues in production
- use messageChannelSyncStatus in messageSync service instead of making
a manual update
2025-12-08 14:56:43 +01:00
Marie 7ce22d5c7e breaking (soft) - Migrate viewGroup.fieldMetadataId -> view.mainGroupByFieldMetadataId (2/3) (#16277)
Should be merged once https://github.com/twentyhq/twenty/pull/16206 has
been released + command run to prod

In this PR
- Remove usage of viewGroup.fieldMetadataId, both in BE and FE states. 
- But we still need to properly populate it until we fully remove
viewGroup.fieldMetadataId from db and ORM entity (upcoming 3rd PR out of
3). fieldMetadataId was removed from CoreViewGroup type and
CreateViewGroupInput and is determined BE-side based on the associated
view's mainGroupByFieldMetadataId. **I expect this means a downtime on
viewGroup creation, until both FE and BE are deployed and cache is
flushed.** This seems acceptable to me as it only regards viewGroup
creation.
    - this information is replaced by view.mainGroupByFieldMetadataID
- Handle view group creation, update and deletion in the BE as a
side-effect of a view creation, update or deletion. Optimistic effects
are still used
- Add validation at view creation or update regarding
mainGroupByFieldMetadata

Left to do in 3rd PR
- Remove viewGroup.fieldMetadataId from db and ORM entity
- Restore feature allowing to update an existing grouped view's group by
field (already OK on BE side but need to rebuild FE optimistic)
2025-12-05 14:00:03 +00:00
Weiko 3d95d6ca00 Add local only cache to cache service and cache typeorm entity metadata (#16287)
## Problem

buildEntityMetadatas in GlobalWorkspaceOrmManager is computationally
expensive and was running on every executeInWorkspaceContext call. This
method uses
TypeORM's EntitySchemaTransformer and EntityMetadataBuilder to build
metadata for all workspace entities (30-50+ objects with many fields
each).

The resulting EntityMetadata[] is not serialisable which means it cannot
be cached in Redis because they contain:
- Circular references
- Functions/methods
- References to the DataSource instance

## Solution

Extended the workspace cache system to support local-only caching, then
created a cache provider for entityMetadatas.

## Implementation details
Updated @WorkspaceCache decorator (workspace-cache.decorator.ts)
- Added localOnly?: boolean option to skip Redis storage for
non-serializable data
Created WorkspaceEntityMetadatasCacheService
- Computes entity metadatas from DB to avoid race condition, this is
acceptable
Simplified GlobalWorkspaceOrmManager
- Now fetches entityMetadatas from cache instead of rebuilding on every
call
Updated Workspace migration runner - the only entry point where metadata
can change
- Now invalidate the new 'entityMetadata' local cache when
shouldIncrementMetadataGraphqlSchemaVersion is true (== field/object
mutations)
2025-12-03 19:50:40 +01:00
Paul Rastoin a829ae62e2 Remove IS_WORKSPACE_MIGRATION_V2_ENABLED feature flag (#16280)
# Introduction
closes https://github.com/twentyhq/core-team-issues/issues/1911
2025-12-03 15:47:08 +00:00
Marie 77409b6eb2 [Requires "warm" cache flush (no immediate downtime before flush)] Migrate viewGroup.fieldMetadataId -> view.mainGroupByFieldMetadataId (1/3) (#16206)
In this PR (1/3)
- introduce view.mainGroupByFieldMetadataId as the new reference
determining which fieldMetadataId is used in a grouped view, in order to
deprecate viewGroup.fieldMetadataId which creates inconsistencies.
view.mainGroupByFieldMetadataId is now filled at every view creation,
though not in use yet.
- Introduce a command to backfill view.mainGroupByFieldMetadataId for
existing views + delete all viewGroup.fieldMetadataId with a
fieldMetadataId that is not view.mainGroupByFieldMetadataId. (It should
concern 37 active workspaces)
- Temporarily disable the option to change a grouped view's
fieldMetadataId as for now it creates inconsistencies. This feature can
be reintroduced when we have done the full migration.

In a next PR
- (2/3) use view.mainGroupByFieldMetadataId instead of
viewGroup.fieldMetadataId. In FE we may keep viewGroup.fieldMetadataId
as a state (TBD). View groups will now be created / deleted as a side
effect of view's mainGroupByFieldMetadataId update.
- (3/3) remove viewGroup.fieldMetadataId

---------

Co-authored-by: Charles Bochet <charles@twenty.com>
2025-12-02 16:31:07 +01:00
Weiko 13e283fc3a Rename roleTargets -> roleTarget (#16247) 2025-12-02 14:39:54 +01:00
Etienne 68c429a54a Null equivalence - remove feature flag (#16222) 2025-12-01 19:06:24 +01:00
Weiko 1eb2e44058 Refactor workspace cache service (#16208)
## Context
We've recently introduced a new workspace cache service which now acts
as a cache access and local storage for all workspace related data,
deprecating the individual specific services.
- Better performance through multiple caching/fetching strategies
- Consistent data access patterns across the codebase
- Reduced redis queries through MGET/MSET/PIPELINE with multiple cache
keys
2025-12-01 17:08:21 +01:00
neo773 79e2602790 Remove IS_MESSAGE_FOLDER_CONTROL_ENABLED feature flag (#16183)
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
2025-12-01 16:29:09 +01:00
Abdullah. ee08060798 Improve deactivated objects & fields behaviors. (#16090)
Closes [1918](https://github.com/twentyhq/core-team-issues/issues/1918).

- For the first point in the issue, we just show the deactivated entries
along with the deactivated text.

---

- For the second point, we show a banner and control the
enabled/disabled state of save button depending on whether we're
allowing the user to create table with the typed name.
- For example, we do not want to allow the user to create a table with
reserved name, so we disable the save button without showing a banner.
- Similarly, we do not want the user to create a table with a name that
already exists in the database. In this case, we show a banner and we
also disable the save button.
- Finally, we do not want to allow the user to create a table where
singular and plural name are the same. Therefore, we disable the save
button for names like `works`.

---

- For the third point, if we add the delete button, it logically means
that we allow the user to delete a custom object/field even it has not
been deactivated yet, so did that.
- Upon deleting the object/field, if we wait for the metadata to refetch
before we navigate, this is what we see because the path does not exist
any longer after deletion and we're waiting for refetch on the path
until we navigate away.


https://github.com/user-attachments/assets/dbe0569c-db88-4285-851f-22551b1ca81e

- To avoid this page from appearing, I replaced awaiting refetch to not
awaiting refetch and redirecting while the refetch happens in the
background.
- Therefore, when we delete something, there is a slight delay for when
it is actually cleared out from the list, but the Not Found view does
not appear on the screen.


https://github.com/user-attachments/assets/47f49579-ce51-4d6a-b857-72046247bb4b

- I tried optimistically removing the object/field from the metadata,
but it leads to some issues (crashes the app) and I have not been able
to find a solution for it yet.
- Therefore, instead of getting stuck at perfection and blocking myself,
I stopped getting into the issue further and created this PR by ensuring
that the desired functionality works.


<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> Display deactivated objects/fields by default, add delete actions with
confirmation, and unify metadata name computation (auto-suffix reserved
keywords) across front/back with conflict checks in object creation.
> 
> - **Frontend (Settings/Data Model)**:
> - **Visibility/UX**: Show `Deactivated` labels for objects/fields;
filters default to include inactive (`showDeactivated`/`showInactive`
true); replace field action dropdown with chevron link.
> - **Delete flows**: Add delete buttons for custom objects/fields with
confirmation modals and background refetch to avoid Not Found flashes.
> - **Creation/Edit validation**: Add name conflict detection banner in
`SettingsDataModelObjectAboutForm` and disable Save on conflicts;
simplify `metadataLabelSchema` to use computed name; form fields
validate on change and sync API names.
> - **Shared (twenty-shared/metadata)**:
> - Add `computeMetadataNameFromLabel` util (slugify+camelCase) and
`RESERVED_METADATA_NAME_KEYWORDS`; auto-append `Custom` to reserved
names; export constants/utilities.
> - **Backend**:
> - Migrate to shared `computeMetadataNameFromLabel`; update validators
to use shared reserved keywords with new messages; allow deletion of
active custom fields/objects (keep standard guards); adjust
services/decorators accordingly.
> - **Tests/Stories**:
> - Update unit/integration snapshots for new reserved-name messages and
behaviors; add missing i18n/router decorators in stories.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
5b126155606f6dbc8f7f91e2192cffb7bd2ebd2c. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
2025-12-01 15:04:36 +00:00
Weiko 470888a23a Fix missing metadata version in legacy datasource (#16173) 2025-11-28 11:54:54 +00:00
Charles Bochet eb362c6d5f Update workspace entities to make all TEXT nullable (#16144)
Follow up on #15926

---------

Co-authored-by: Etienne <45695613+etiennejouan@users.noreply.github.com>
Co-authored-by: guillim <guigloo@msn.com>
2025-11-28 10:50:38 +01:00
Weiko 9620a4b0ba Optimize EntityMetadata caching in GlobalWorkspaceDataSource (#16146)
## Context
EntityMetadata was being rebuilt from scratch on every
findMetadata()/getMetadata() call (~20 times per request). This involved
running EntitySchemaTransformer.transform() and
EntityMetadataBuilder.build() repeatedly, causing unnecessary CPU
overhead.

## Implementation
Cache entityMetadatas in ORMWorkspaceContext: Build EntityMetadata once
during workspace context initialization instead of on every metadata
lookup
Remove redundant entitySchemas caching: Since flatMetadata is already
cached, the additional Redis cache for entitySchemaOptions was
unnecessary overhead
Remove WorkspaceEntitiesStorage: Replaced with direct lookup from
FlatObjectMetadataMap
Simplify getObjectMetadataFromEntityTarget: Now only accepts string
targets, using flat metadata maps directly

Also:
Removed unused injections in some services
2025-11-28 10:09:52 +01:00
Weiko da626f70b7 optimize buildFieldMapsFromFlatObjectMetadata usages (#16132)
This newly introduced util can be a bit expensive especially when done
recursively.
This PR improves that


File | Pattern Fixed | Impact
-- | -- | --
format-result.util.ts | Recursive array/object processing | N array
items → 1 call
format-data.util.ts | Recursive array processing | N array items → 1
call
process-nested-relations-v2.helper.ts | Duplicate call in call chain | 2
calls → 1 call
common-result-getters.service.ts | Per-record processing in array | N
records → 1 call
compute-relation-connect-query-configs.util.ts | Nested loop (entities ×
connect fields) | N×M calls → 1 call
2025-11-27 15:01:51 +01:00
Weiko 1607aebcc6 Deprecate object metadata maps in favor of flat entities (#16080)
## Context
Deprecating the old objectMetadataMap type in favour of split flat
entities to match with our new caching.
In the long run, trying to achieve:
- Better performance through caching
- Consistent data access patterns across the codebase
- Reduced database queries

Now that everything is based on flat entities, which are cached, we can
finish the refactoring of workspace context cache which should already
improve performances.
Then the last step will be to consume that new cache in the new global
datasource to get rid of the many workspace datasources stored in the
server
2025-11-27 13:43:34 +01:00
Etienne 2028a8f9be Null equivalence - Fix (#16050)
@charlesBochet 
In workflow codebase, NULL (instead of empty object) is expected on
workflow version object step field, when a new workflow is created for
example.
(packages/twenty-front/src/modules/workflow/workflow-diagram/utils/generateWorkflowDiagram.ts
- 42)
This case is not isolated and it creates many issues.

We decided to format NULL value to equivalent (empty string for text
field, empty object for raw_json) but it seems it complicates the dev x.

To unlock @Devessier I prefer revert the logic, the time we discuss how
to solve this cases.
2025-11-25 14:07:32 +01:00
Félix Malfait e7ebf51e50 Replace agent handoff system with planning-based router (#16003)
## Overview

This PR replaces the dynamic agent handoff system with a more
predictable planning-based router that decides upfront how to handle
multi-agent coordination.

## Major Changes

### 🔄 Architecture Shift: Handoffs → Planning

**Removed:**
- `AgentHandoffEntity` and handoff tracking system
- `AgentHandoffService` and `AgentHandoffExecutorService`
- Dynamic agent-to-agent transfers during execution
- Handoff tool generation and description templates

**Added:**
- `AiRouterService` with two strategies: `simple` (single agent) and
`planned` (multi-agent)
- `AgentPlanExecutorService` for executing multi-step plans
- Plan validation (cycle detection, dependency resolution)
- `UnifiedRouterResult` type with discriminated union

### 🤖 New Standard Agents

Added two new specialized agents:
- **Researcher Agent**: Web search, fact-finding, competitive
intelligence
- **Code Agent**: TypeScript function generation for serverless
workflows

### 🏗️ Router Refactoring (Latest)

Split router responsibilities into focused services:
- `AiRouterStrategyDeciderService`: Decides simple vs planned strategy
- `AiRouterPlanGeneratorService`: Generates and validates execution
plans
- `AiRouterService`: Coordinates between services (reduced from 426→275
lines)

### ⚙️ Configuration Improvements

- Added `outputStrategy` to agent definitions (`direct` vs `synthesize`)
- Removed hardcoded special cases for workflow-builder
- Added `plannerModel` field to workspace entity
- Increased `MAX_STEPS` from 10 to 25 for complex workflows

### 📝 Agent Prompt Refinements

Significantly simplified prompts for better clarity:
- Workflow Builder: 51→36 lines
- Helper: 49→28 lines
- Data Manipulator: Enhanced with sorting guidance

### 🔍 Enhanced Debugging

- Plan reasoning and step count in data message parts
- Router debug info with token usage tracking
- Better logging throughout execution pipeline

## Benefits

1. **Simpler Mental Model**: Router decides upfront vs dynamic transfers
2. **Better Predictability**: Users see the plan before execution
3. **Cleaner Architecture**: SRP with focused services
4. **Configuration Over Code**: Agent behavior via config, not hardcoded
logic
5. **Plan Validation**: Catches invalid dependencies and cycles

## Migration Notes

- Database migration removes `agentHandoff` table
- Adds `plannerModel` column to workspace table
- No API breaking changes (agent endpoints unchanged)

## Testing

- Integration tests updated to remove handoff dependencies
- Agent tool test utilities simplified
- Plan validation covered by new logic

## Next Steps (Future PRs)

- Parallel execution of independent plan steps
- Dynamic re-planning based on results
- Plan caching for common routing patterns
- Error recovery strategies in plan executor
2025-11-25 12:10:14 +01:00
Marie 46d1ea6505 [groupBy] groupBy relation fields (#15951)
Example query
Here person has 
- a N - 1 relationship with company
- a N - 1 morph relationship with pet or company

<img width="862" height="374" alt="image"
src="https://github.com/user-attachments/assets/59bc9b82-c943-43de-ad82-d3393b76904b"
/>
<img width="415" height="629" alt="image"
src="https://github.com/user-attachments/assets/9a3176bc-99cd-4983-8611-68ca3a2cf527"
/>

truncated response
<img width="299" height="447" alt="image"
src="https://github.com/user-attachments/assets/45af0322-9e66-4eae-8353-6c0dda487bbe"
/>

We don't allow grouping by relations of relations.

Left to do
- rest api
- tests on permissions
2025-11-25 09:03:16 +00:00
Weiko 04562b11fb Migrate metadata cache (#16030)
## Context
Deprecating legacy ObjectMetadata from cache in favor of flat entities.
Introducing utils to build byName/byNameSingular/byNamePlural in
isolated cases

## Next
- I had to introduce a util to build from flat to legacy
objectMetadataMaps, we should instead use flat maps directly when needed
(datasource, schema generation, etc)
- Deprecate metadata version in the cache
- Use the new cache strategy for flat entities with permissions and
feature flags and inject in the global datasource context
2025-11-24 19:50:47 +01:00
Etienne 208c0857ee common api - null equivalence (#15926)
closes https://github.com/twentyhq/core-team-issues/issues/1629

To do before requesting review : 
- filter update

Migration to come in an other PR


Strat : 
1/  Null transformation

- [x] Transform NULL equivalent value to NULL in field validation in
common api - pre-query - with feature flag
- [ ] Same logic in ORM (Not done, complex to handle feature flag here)
- [x] Transform NULL value to equivalent in data formatting in ORM -
post-query

2/ Migration (in other PR) for fieldMetadata not nullable with default
defaultValue (empty string, ...)

- [ ] Remove NOT NULL db constraint
- [ ] Update record value to NULL
- [ ] Update field metadata : isNullable:true
- [ ] Update uniqueIndex whereClause (also for standard uniqueIndex) 
- [ ] Activate feature flag

3/ Update metadata creation

- [x] No more default default value
- [x] Update standard field nullability
- [x] Remove index default whereClause for standard field

4/ Update filter

- [x] When filtering on NULL or empty string, be sure all records are
returned (the one with NULL + the one with "")

5/ Test

- [ ] Strat. to do
2025-11-24 18:57:47 +01:00
Charles Bochet 30e504628c Fix ORM event mixing records in batch updates (#15985)
While investigating the issue a customer was facing, I discovered that
before records and after records could be in different order, making the
orm event and timeline activity engine mix records
2025-11-22 13:16:04 +01:00
Weiko a21d58f43d Add entitySchemas to workspace context and cache (#15966) 2025-11-21 15:07:46 +01:00
Weiko 0a2d42e79f Implement workspace cache storage (#15962)
## Context
Implementing a single service managing all the cache scoped to a
workspace, this will be dynamically injected as a WorkspaceContext in
the app during a request lifetime (ingested by the future global
datasource for example).

Usage:

```typescript
this.globalWorkspaceOrmManager.executeInWorkspaceContext(
        authContext,
        async () => {
           // Everything here will have access to a workspaceContext, containing all the cache data + a ready to use datasource with workspace scoped metadata, permissions, feature flags, etc...
           // Internally will call loadWorkspaceContext
        }
```
Note: executeInWorkspaceContext will probably be owned by a higher level
service later and not only the ORM.

```typescript
  private async loadWorkspaceContext(
    authContext: WorkspaceAuthContext,
  ): Promise<WorkspaceContext> {
    const workspaceId = authContext.workspace.id;

    const cache =
      await this.workspaceContextCacheService.get<WorkspaceContextData>(
        workspaceId,
        [
          'objectMetadataMaps',
          'metadataVersion',
          'featureFlagsMap',
          'permissionsPerRoleId',
        ],
      );

    return {
      authContext,
      objectMetadataMaps: cache.objectMetadataMaps,
      metadataVersion: cache.metadataVersion,
      featureFlagsMap: cache.featureFlagsMap,
      permissionsPerRoleId: cache.permissionsPerRoleId,
    };
  }
  ```
  
  The cache retrieval strategy is as followed:
```
- Check if there is an ongoing promise fetching data from the cache =>
return the promise.
- Check in the local cache entry if lastCheckedAt has expired. If not,
return as it is without querying redis.
- Check in redis the cache entry hash and compare with local cache entry
hash, if they are the same return the local cache entry data
- Check in redis the cache entry data, if it's there return it and store
it into the local cache entry data and update local cache entry hash. If
it's not there recompute the data by querying the DB and update both
redis and local cache
```
2025-11-21 12:44:24 +01:00
martmull 3190ca5b9e 1858 extensibility create relation metadata decorator in thwenty sdkapplications (#15907)
as title

First PR 
I will update the twenty-cli in another PR
2025-11-20 10:22:42 +01:00
Paul Rastoin a39efeb1ab [BREAKING_CHANGE/GRAPHQL/OBJECT_METADATA_CREATE_ONE] Remove object/fields/view-fields v1 implementation (#15823)
# Introduction
Remove the v2 feature flag for view-field field-metadata and
object-metadata metadata entities

## Some details
- Disabled nestjs-query for object metadata creation and explicitly
calling it
- removed all v1 integration tests files

## Remarks
Not remove v2 referencing in both filenaming right now will handle that
globally later

## Breaking change
Due to object metadata resolver createOne standardization had to rename
the input from `CreateObjectInput` to `CreateOneObjectInput`
2025-11-17 10:55:19 +01:00
Paul Rastoin 48031a7ce2 Runner v2 on cascade delete_field/object + refactor workspace deletion side effect (#15830)
# Introduction
While removing the v1 https://github.com/twentyhq/twenty/pull/15823 I've
encountered the method `objectMetadataService.deleteObjectsMetadata`
that I didn't wanted to migrate as it is and if it's not challenging its
existence legitimacy.

## Motivations
In a nutshell on a workspace deletion object metadata and field metadata
cascading deletion is correclty handled
But that's not the case for all of a workspaces entities ( roles,
workspaceMigrations ) I suspect that we did not defined the foreignKey
explicitly through `typeorm`

## Battle testing v2
I still decided to give a try to a complex operation in the v2 such as a
workspace all object metadata deletion.
Spoiler it failed due to object being interdependent between them and
not being topologically sorted ( morph relation can introduce circular
dep anw ).
Note: Even after removing the delete field on delete object aggregator
we end up with an equivalent circular dep error which an object and its
field metadata identifier connection.

## Elegant `DEFERRED` and `DEFERRABLE` foreign keys
The most safe, low level solution would be to make all field relations
deferrable and start the runner transaction as deferred. But this
requires a quite invasive migration of existing FK

## On cascade solution
I've opted for the quick fix, on object or field deletion spread
cacasde.
It's pretty safe as the builder priorly validates the deletion and and
its related entities integrity
Only for both field and object deletion action types in v2 runner

## Integration coverage
Added a test that will scan a new workspace database core schema tables
and expect now result after workspace deletion through its only user
deletion
2025-11-16 22:03:50 +01:00
martmull f13c36234f Fix workspace-member-id missing from database events (#15819)
As title
2025-11-14 14:03:49 +00:00
Weiko 4dd6edb35a Fix authContext missing in ALS WorkspaceContext for global datasource (#15810)
Fixes https://github.com/twentyhq/twenty/issues/15809

## Context
We recently introduced a global datasource now consuming workspace
context from ALS store however the authContext was missing (only the
workspaceId was there) which "broke" event emission, now missing the
workspaceMemberId

This PR adds the missing authContext so we can access from anywhere in
the datasource. We are still passing it as a parameters on repository
level for legacy but in theory we should be able to remove it from
everywhere and consume the context

<img width="929" height="253" alt="Screenshot 2025-11-13 at 18 58 16"
src="https://github.com/user-attachments/assets/3e04f264-95e6-4831-94f3-fc01603f19bd"
/>
2025-11-14 10:59:15 +01:00
Weiko 998365457e Improve infer deletion from entities (#15807)
## Context
inferDeletionFromEntities only accepts a set of keys for each entities
that needs deletion. This could be error-prone if tmr we want to add a
new side effect and forget to add the entity when in practice you want
to delete all entities that are in the fromToAllFlatEntityMaps (this is
the case for applications for example)
2025-11-13 18:18:27 +01:00
Guillim 0389fcf00d removing feature flag IS_MORPH_RELATION_ENABLED (#15783)
releasing the morph feature for all workspaces
2025-11-13 12:03:25 +01:00
martmull 9880f192a5 Move composite types to twenty-shared (#15741) 2025-11-10 22:11:52 +01:00
Weiko 02fda92a93 Global workspace datasource poc (#15744)
## Context
This PR introduces a Global Workspace DataSource that consolidates
workspace-specific database access through a single TypeORM DataSource
instance with AsyncLocalStorage-based context management instead of N
workspace datasources.

- Created GlobalWorkspaceDataSource extending TypeORM's DataSource to
manage multiple workspaces with entity metadata caching (1-hour TTL)
- Implemented AsyncLocalStorage for workspace context propagation
(WorkspaceContextForStorage) containing workspace ID, metadata,
permissions, and feature flags
- Modified query execution flow to wrap operations in workspace context
via GlobalWorkspaceOrmManager.executeInWorkspaceContext()
- Added schema name to entity schemas for proper multi-tenant database
separation

Next: 
- use the new global workspace datasource everywhere and deprecate
workspace datasource factory
- improve metadata caching using a short TTL to avoid multiple calls to
redis
- Leverage the new WorkspaceContextALS and put it higher in the request
hierarchy to have access to permission, metadata and featureflag
everywhere --- build it manually for commands --- find a way to
propagate it in jobs?
- Remove PG_POOL patch once we have a unique datasource and increase
global datasource pool size

## Implementation
Why ALS:

1. Automatic Per-Request Isolation

With schema-based multi-tenancy, each workspace has its own PostgreSQL
schema
(e.g. workspace_20202020-1c25-4d02-bf25-6aeccf7ea419).
The critical challenge is ensuring that concurrent requests from
different tenants don't interfere with each other.
```typescript
// Request A (Workspace 1) and Request B (Workspace 2) executing concurrently

// Without ALS: Race condition — they'd share the same global state!
// With ALS: Each request has isolated context ✓
```
ALS automatically isolates context per async execution chain, so:
- Request from Tenant A → ALS stores workspaceId: "tenant-a" → Queries
hit workspace_tenant_a schema
- Request from Tenant B → ALS stores workspaceId: "tenant-b" → Queries
hit workspace_tenant_b schema

 No interference, even when executing simultaneously on the same
Node.js event loop.

2. No Manual Context Passing

Before ALS, you'd need to pass workspaceId through every function call:
```typescript
//  Without ALS - Context threading nightmare
getRepository(workspaceId, entity)
  → createEntityManager(workspaceId)
    → getMetadata(workspaceId, target)
      → findInCache(workspaceId, cacheKey)
```
With ALS:
```typescript
//  With ALS - Clean, implicit context
getRepository(entity)          // Reads workspaceId from ALS
  → createEntityManager()      // Reads workspaceId from ALS
    → getMetadata(target)      // Reads workspaceId from ALS
      → findInCache(cacheKey)  // Reads workspaceId from ALS
```
example
```typescript
override findMetadata(target: EntityTarget<ObjectLiteral>): EntityMetadata | undefined {
  const context = getWorkspaceContext(); // 👈 Automatically gets the right workspace!
  const { workspaceId, metadataVersion } = context;
  const cacheKey = `${workspaceId}-${metadataVersion}`;
  // ... returns metadata for THIS workspace's schema
}
```

3. Async Chain Propagation

Node.js operations are heavily async. ALS automatically propagates
context through:

- async/await chains
- Promise chains
- Callbacks

```typescript
executeInWorkspaceContext(workspaceId, async () => {
  await prepareContext();           // Has context ✓
  const results = await run();      // Has context ✓
  await enrichResults();            // Has context ✓

  // Even nested async operations maintain context!
  await Promise.all([
    saveToCache(),   // Has context ✓
    emitEvent(),     // Has context ✓
    logMetrics(),    // Has context ✓
  ]);
});
```
5. Schema-Specific Metadata Caching

The implementation caches entity metadata per workspace + version:
```typescript
// Cache key format: "workspaceId-metadataVersion"
const cacheKey = `${workspaceId}-${metadataVersion}`;
```
Why this matters with schemas:

- Each workspace has different table structures (custom fields, objects)
- EntitySchema includes schema: "workspace_xxx" property
- Each cached metadata points to the correct schema

ALS ensures getWorkspaceContext() returns the right workspaceId,
so you always get the correct schema's metadata from cache.

6. Single DataSource for All Tenants

The key change here:
```typescript
//  Old approach: One DataSource per tenant
const dataSourceTenantA = new DataSource({ schema: 'workspace_a' });
const dataSourceTenantB = new DataSource({ schema: 'workspace_b' });
// Problem: Hundreds of DB connection pools!
```
```typescript
//  New approach: One shared DataSource + ALS context
const globalDataSource = new GlobalWorkspaceDataSource();

// ALS determines which schema to use at runtime
```
When you call:
```typescript
globalDataSource.getRepository('person');
```
It internally does:
```typescript
const context = getWorkspaceContext();  // Gets current tenant from ALS
const metadata = this.findMetadata('person'); // Finds metadata for THIS tenant's schema
// EntityMetadata includes: schema: "workspace_20202020-1c25..."
// TypeORM automatically queries: SELECT * FROM "workspace_20202020-1c25...".person
```
7. Request Lifecycle Example

```typescript
// 1. GraphQL request arrives: "query people { ... }"
// 2. Middleware extracts authContext.workspace.id = "tenant-a"
// 3. Query runner wraps execution in ALS:
executeInWorkspaceContext("tenant-a", async () => {
  // 4. Everything inside has access to workspace context:
  const repo = getRepository('person');      // ALS → tenant-a
  const metadata = getMetadata('person');    // ALS → tenant-a → cache["tenant-a-v5"]

  // 5. TypeORM builds query with correct schema:
  // SELECT * FROM "workspace_tenant_a"."person" WHERE ...

  // 6. Even nested calls work:
  await saveAuditLog();  // ALS → tenant-a → correct audit schema
  await emitWebhook();   // ALS → tenant-a → correct tenant webhook
});
// 7. Request completes, ALS context automatically cleaned up
```

8. Safety & Error Prevention

```typescript
// If you forget to set context:
const context = getWorkspaceContext();
//  Throws: "Workspace context not set..."
// Fails fast rather than querying wrong schema!

// Can't accidentally query wrong tenant:
// Context is immutable within execution scope
```
2025-11-10 17:32:49 +01:00
Guillim 2274a937bd first step (#15687)
TODO : 

-> for test integreation for this connect 
-> add validation in common API

Fixes https://github.com/twentyhq/core-team-issues/issues/1278
2025-11-07 13:49:57 +00:00
Marie 4ce93aee52 Fix user deletion flows (#15614)
**Before**
- any user with workpace_members permission was able to remove a user
from their workspace. This triggered the deletion of workspaceMember +
of userWorkspace, but did not delete the user (even if they had no
workspace left) nor the roleTarget (acts as junction between role and
userWorkspace) which was left with a userWorkspaceId pointing to
nothing. This is because roleTarget points to userWorkspaceId but the
foreign key constraint was not implemented
- any user could delete their own account. This triggered the deletion
of all their workspaceMembers, but not of their userWorkspace nor their
user nor the roleTarget --> we have orphaned userWorkspace, not
technically but product wise - a userWorkspace without a workspaceMember
does not make sense

So the problems are
- we have some roleTargets pointing to non-existing userWorkspaceId
(which caused https://github.com/twentyhq/twenty/issues/14608 )
- we have userWorkspaces that should not exist and that have no
workspaceMember counterpart
- it is not possible for a user to leave a workspace by themselves, they
can only leave all workspaces at once, except if they are being removed
from the workspace by another user

**Now**
- if a user has multiple workspaces, they are given the possibility to
leave one workspace while remaining in the others (we show two buttons:
Leave workspace and Delete account buttons). if a user has just one
workspace, they only see Delete account
- when a user leaves a workspace, we delete their workspaceMember,
userWorkspace and roleTarget. If they don't belong to any other
workspace we also soft-delete their user
- soft-deleted users get hard deleted after 30 days thanks to a cron
- we have two commands to clean the orphans roleTarget and userWorkspace
(TODO: query db to see how many must be run)

**Next**
- once the commands have been run, we can implement and introduce the
foreign key constraint on roleTarget


Fixes https://github.com/twentyhq/twenty/issues/14608
2025-11-06 18:29:12 +00:00
martmull abde3c04ac 1630 extensibility twenty cli ability to create edit and delete fields (#15501)
As title

- adds decorators in twenty-sdk
- update twenty-cli load-manifest to it gets @FieldMetadata infos +
testing
- update twenty-server so it CRUD fields properly, using
universalIdentifier
- Fix UI so we can update managed objects records
- move FieldMetadata items from twenty-server to twenty-shared
2025-11-05 17:50:06 +01:00
Raphaël Bosi 05a28a8ec2 Create a feature flag for dashboards v2 (#15601)
Prevent users from creating v2 chart types via the api.
Only created unit tests and not integration tests (since it's not that
important, and the v2 will be released soon), but tested via the api
playground.
2025-11-04 16:29:08 +01:00
Thomas Trompette d1330a4a8c Improve workflow performances (#15528)
- on draft creation, do not fetch the full version. Avoid the fetch of
steps and trigger
- on activation, we were performing 8 queries/mutations synchronously +
2 additional for automated triggers. I refacto the call it it gets
reduced to 4 queries/mutations + 2 additional for automated triggers
2025-11-03 17:23:01 +01:00
Charles Bochet 20576f2ee7 Add Dashboard in Lab, make Calendar available out of lab (#15560)
As per title!
2025-11-03 15:40:26 +01:00
Guillim 53bb534af8 MORPH for relationDecorator (#15420)
# Adding MORPH support  to the relationDecorator

Extends the @WorkspaceRelation decorator to support morph relations by
adding isMorphRelation and morphId options

The implementation enforces type safety through TypeScript discriminated
unions, ensuring morphId is required when isMorphRelation: true, and
includes runtime validation that throws a descriptive error if morphId
is missing.

The morph relation metadata is properly stored in metadataArgsStorage
and integrates seamlessly with the existing StandardFieldRelationFactory
to convert decorated fields into FieldMetadataType.MORPH_RELATION
entities with appropriate database constraints.



To test :
in person entity, add
```ts
  @WorkspaceRelation({
    standardId: PERSON_STANDARD_FIELD_IDS.testRelatedRecordCompany,
    type: RelationType.MANY_TO_ONE,
    label: msg`Test Related Record of Company`,
    description: msg`Test relation to company`,
    icon: 'IconLink',
    inverseSideTarget: () => CompanyWorkspaceEntity,
    inverseSideFieldKey: 'testRelatedPeople',
    onDelete: RelationOnDeleteAction.CASCADE,
    isMorphRelation: true,
    morphId: PERSON_STANDARD_FIELD_IDS.testRelatedRecordMorphId,
  })
  @WorkspaceIsNullable()
  @WorkspaceIsSystem()
  testRelatedRecordCompany: Relation<CompanyWorkspaceEntity> | null;

  @WorkspaceJoinColumn('testRelatedRecordCompany')
  testRelatedRecordCompanyId: string | null;

  @WorkspaceRelation({
    standardId: PERSON_STANDARD_FIELD_IDS.testRelatedRecordOpportunity,
    type: RelationType.MANY_TO_ONE,
    label: msg`Test Related Record of Opportunity`,
    description: msg`Test relation to opportunity`,
    icon: 'IconLink',
    inverseSideTarget: () => OpportunityWorkspaceEntity,
    inverseSideFieldKey: 'testRelatedPeople',
    onDelete: RelationOnDeleteAction.CASCADE,
    isMorphRelation: true,
    morphId: PERSON_STANDARD_FIELD_IDS.testRelatedRecordMorphId,
  })
  @WorkspaceIsNullable()
  @WorkspaceIsSystem()
  testRelatedRecordOpportunity: Relation<OpportunityWorkspaceEntity> | null;

  @WorkspaceJoinColumn('testRelatedRecordOpportunity')
  testRelatedRecordOpportunityId: string | null;
```

in opportunity,
```ts
@WorkspaceRelation({
    standardId: OPPORTUNITY_STANDARD_FIELD_IDS.testRelatedPeople,
    type: RelationType.ONE_TO_MANY,
    label: msg`Test Related People`,
    description: msg`People linked to the opportunity via Test Related Record`,
    icon: 'IconUsers',
    inverseSideTarget: () => PersonWorkspaceEntity,
    inverseSideFieldKey: 'testRelatedRecordOpportunity',
    onDelete: RelationOnDeleteAction.SET_NULL,
  })
  @WorkspaceIsNullable()
  @WorkspaceIsSystem()
  testRelatedPeople: Relation<PersonWorkspaceEntity[]>;
  ```

in company,
```ts
  @WorkspaceRelation({
    standardId: COMPANY_STANDARD_FIELD_IDS.testRelatedPeople,
    type: RelationType.ONE_TO_MANY,
    label: msg`Test Related People`,
description: msg`People linked to the company via Test Related Record`,
    icon: 'IconUsers',
    inverseSideTarget: () => PersonWorkspaceEntity,
    inverseSideFieldKey: 'testRelatedRecordCompany',
    onDelete: RelationOnDeleteAction.SET_NULL,
  })
  @WorkspaceIsNullable()
  @WorkspaceIsSystem()
  testRelatedPeople: Relation<PersonWorkspaceEntity[]>;
```

In constants/standard-field-ids.ts, add the standard fields ids
```ts
// COMPANY_STANDARD_FIELD_IDS
testRelatedPeople: '20202020-7a90-47e9-b31f-916a716fd212',

// OPPORTUNITY_STANDARD_FIELD_IDS
testRelatedPeople: '20202020-7a90-47e9-b31f-916a716fd213',

// PERSON_STANDARD_FIELD_IDS
testRelatedRecordOpportunity: '20202020-5a90-47e9-b31f-916a716fd211',
testRelatedRecordCompany: '20202020-1eb2-4298-910b-66b015b36d72',
testRelatedRecordMorphId: '20202020-6ccf-48e3-bb92-bc9961bc011e',
```
2025-11-02 13:31:48 +01:00