- add to step output schema the information that field is a composite
sub field
- from output schema, when selecting the variable, copy all info
required to stepFilter
- from stepFilter, when selecting a value, display a special component
for composites
We will have floating steps in our workflow with the branch design.
Currently, a step without parent is considered linked to the trigger. We
need to distinguish the 2 cases. Thus this PR:
- add `nextStepIds` to workflowVersion.trigger
- create a command to migrate existing triggers
- Renamed `WorkflowActionAdapter` to `ToolExecutorWorkflowAction`
- Renamed `settingPermission` table to `permissionFlag` and `setting`
column to `flag`
- Decoupled the send email logic from workflows to tools
- Add new `Tools Permission` section in FE
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
This PR adds the ability to save an any field filter to a view.
It adds a new `anyFieldFilterValue` on both core view and workspace view
entities.
It also introduces the necessary utils that mimic the logic that manages
the save of record filters and record sorts on views.
In this PR, behind a feature flag, we add a permission layer check based
on the read permission.
It is done by computing a map of an object's fields, where keys are the
column names and values the fieldMetadata id, making them comparable to
the restricted fields ids list stored in the permission cache.
For mutations (create, update, delete, destroy), we need to check the
read permission on the returned field, as they may differ from the
updated field. The write field permission will be tackled in a different
PR.
## Problem
After migrating webhooks and API keys from workspace to core level, REST
API endpoints were still creating entities in workspace schema
(`workspace_*`) instead of core schema, causing webhooks to not fire.
## Solution
- Added dedicated REST controllers for webhooks (`/rest/webhooks`) and
API keys (`/rest/apiKeys`)
- Updated dynamic controller to block workspace-gated entities from
being processed
- Fixed OpenAPI documentation to exclude these endpoints from playground
- Ensured return formats match GraphQL resolvers exactly
## Testing
✅ All endpoints tested with provided auth token - webhooks and API keys
now correctly stored in `core` schema
- add fieldMetadataId to step output schema
- use it to display FormFieldInput in Filter input
- few fixes for a few fields
Next step:
- Handle composite fields
- Design review
## Problem
Previously, the newly created "message list fetch process" left the
message channel in an "ONGOING" status indefinitely.
## Why
Before only "fullSync" messageChannel were concerned by this method.
What happened it sometimes the providers gave empty messageExternalIds
during the initial fetch. Happens for a newly created email account on
gmail or microsoft (not sure which one)
Now that we gather both of sync methods (partial and full) we cannot use
this anly longer. Because after the first sync, if no new messages
arrived in the last 5 minutes, there will be none, so it was consiedered
as emptyMailbox (which is wrong)
## Solution
Removed this logic from the messageChannel since now we will rely on
each messageFolder (messageList) to handle the logic of going for a full
or not sync.
## Question
We might see some bugs in case some newly created email account without
messageList in case the provider does not give a nextSyncCursor at the
messageList level. Not easy to test
## Bonus
We also cleant a useless service method called getCursor that was not
used anywhere.
---------
Co-authored-by: prastoin <paul@twenty.com>
## Bug description
All old messages were deleted after the first partial sync because the
diff between the existing messages and the messages returned from the
fetch was done considering that it was always a full sync (ie, that we
always retrieve the full list of message ids). But in a partial sync,
only the new messages appear in the list.
This bug was introduced by https://github.com/twentyhq/twenty/pull/13302
when trying to merge the logic between the full sync and the partial
sync.
## Fix
The fix is to adapt the behavior to the type of sync, by looking at the
presence of the sync cursor.
---------
Co-authored-by: Guillim <guillim@users.noreply.github.com>
## Context
We would like to start leveraging more messageFolders during messageSync
which would allow users to select folders they want to sync on their
mailbox. MessageFolder is an abstraction that means folder for
Microsoft, labels for Gmail, not supported for IMAP.
## What
1) We do not aim to build a FE now but we would like to take it into a
consideration if they have been modified through API. So **out of
scope**
2) MessageFolders will be synced regularly from Gmail, Microsoft. This
is currently partially done and improving it is **out of scope**
3) Change: we were having two synchronization mechanism so far:
FULL_SYNC (first time, or when no cursor is present) and PARTIAL_SYNC
(when we have a cursor, we can fetch the diff since the last pull). This
Full vs Partial mode was a high level concept. We now think it's an
driver implementation detail (and can be inferred from the existence of
a cursor). Why making this change now: as we are trying to pull several
folders, a folder could need a full sync if it had no cursor, and
another a partial if it has one. It does not make sense anymore to have
this full vs partial difference at MessageChannel level
4) Once we are sure this work, we can start syncing different folders on
Gmail side (the case for the user it to be able to fetch Promotion
label)
## Implementation strategy
1) Re-use PartialMessageList implementation / API and rename it to
MessageList at it's more complete
2) re-use driver level fullMessageList methods and call them
messageListWithoutCursor
3) make sure that these method are folder specific
## Tests
### Gmail
- Fresh fetch (without cursor): OK
- Message import: OK
- Additional fetch (with messageChannel cursor): OK
### Microsoft
- Fresh fetch (without cursor): OK
- Message import: OK
- Additional fetch (with messageChannel cursor): OK
### Imap
- Fresh fetch (without cursor): OK
- Message import: OK
- Additional fetch (with messageChannel cursor): OK
# Introduction
Following https://github.com/twentyhq/twenty/pull/13310
> After this PR merge will create a new one removing the type and
replacing it to ObjectMetadataEntity.
This is it !
# Introduction
Following `FieldMetadataInterface` deprecation in
https://github.com/twentyhq/twenty/pull/13264
As for the previous PR will rename and remove all the file in a
secondary PR to avoid conflicts and over loading this one
## Improvements
Removed optional properties from the `objectMetadataEntity` model and
added utils to retrieve test data
## Notes
By touching to `ObjectMetadataDTO` I would have expected a twenty-front
codegenerated types mutation, but it does not seem to be granular enough
to null/undefined coercion
# Introduction
From the moment replaced the FieldMetadataInterface definition to:
```ts
import { FieldMetadataType } from 'twenty-shared/types';
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
export type FieldMetadataInterface<
T extends FieldMetadataType = FieldMetadataType,
> = FieldMetadataEntity<T>;
```
After this PR merge will create a new one removing the type and
replacing it to `FieldMetadataEntity`.
Did not renamed it here to avoid conflicts on naming + type issues fixs
within the same PR
## Field metadata entity RELATION or MORPH
Relations fields cannot be null for those field metadata entity instance
anymore, but are never for the others see
`packages/twenty-server/src/engine/metadata-modules/field-metadata/types/field-metadata-entity-test.type.ts`
( introduced TypeScript tests )
## Concerns
- TS_VECTOR is the most at risk with the `generatedType` and
`asExpression` removal from interface
## What's next
- `FielMetadataInterface` removal and rename ( see introduction )
- Depcrecating `ObjectMetadataInterface`
- Refactor `FieldMetadataEntity` optional fiels to be nullable only
- TO DIG `never` occurences on settings, defaultValue etc
- Some interfaces will be replaced by the `FlatFieldMetadata` when
deprecating the current sync and comparators tools
Fixes: https://github.com/twentyhq/twenty/issues/13110
I'm deprecating note.body and task.body to remove confusion between body
and bodyV2
What will be left but should be done later to avoid breaking changes:
- re-add a body field in the graphql API only that points to the same
bodyV2 field in SQL (need to be handled in fields and filter for note
and task)
- (wait some time)
- remove bodyV2 field
- Added a generic HTTP request tool, allowing agents and workflows to
make HTTP requests to external APIs with configurable method, headers,
and body.
- Decoupled HTTP request workflow nodes from workflow-specific types and
factories, introducing a generic tool interface.
- Updated agent system prompts to include explicit guidance for the HTTP
request tool, including when and how to use it, and how to communicate
limitations.
### Demo
https://github.com/user-attachments/assets/129bc445-a277-4a19-95ab-09f890f8f051
## Context
Add an eventEmitter instance to twenty datasources so we can emit DB
events.
Add input and output formatting to twenty orm (formatData, formatResult)
Those 2 elements simplified existing logic when we interact with the
ORM, input will be formatted by the ORM so we can directly use
field-like structure instead of column-like. The output will be
formatted, for builder queries it will be in `result.generatedMaps`
where `result.raw` preserves the previous column-like structure.
Important change: We now have an authContext that we can pass when we
get a repository, this will be used for the different events emitted in
the ORM. We also removed the caching for repositories as it was not
scaling well and not necessary imho
Note: An upcoming PR should handle the onDelete: cascade behavior where
we send DESTROY events in cascade when there is an onDelete: CASCADE on
the FK.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
This PR does not produce any functional changes for our users. It
prepares the branches for workflows by:
- decommissioning `output` and `context` fields or `workflowRun` records
and use newly created `state` field from front-end and back-end
- use `stepStatus` computed by `back-end` in `front-end`
- add utils and types in `twenty-shared/workflow` (not completed, a
follow-up is scheduled
https://github.com/twentyhq/core-team-issues/issues/1211)
- add concurrency to `workflowQueue` message queue to avoid weird branch
execution when using forms in workflow branches
- add a WithLock decorator for better dev experience of
`CacheLockService.withLock` usage
Here is an example of such a workflow running (front branch display is
not yet done that's why it looks ugly) ->
https://discord.com/channels/1130383047699738754/1258024460238192691/1392897615171158098
Instead of initializing model at start time we do it at run time to be
able to swap model provider more easily.
Also introduce a third driver for openai-compatible providers, which
among other allows for local models with Ollama
Step operand will more or less be the same as view filter operand.
This PR:
- moves `ViewFilterOperand` to twenty-shared
- use it as step operand
- check what operand should be available based on the selected field
type in filter action
- rewrite the function that evaluates filters so it uses
ViewFilterOperand instead
ViewFilterOperand may be renamed in a future PR.
Fixes https://github.com/twentyhq/twenty/issues/12867
Issue:
when you have a variable `toto` which is: `Record<string, MyType>` and
you do toto['xxx'], this will be typed as `MyType` instead of `MyType |
undefined`
Solutions:
- activate `noUncheckedIndexedAccess` check in tsconfig, this is the
preferred solution but will take time to get there (this raises 600+
errors)
- use a Map: cf https://github.com/twentyhq/twenty/pull/13125/files
- set the type to Partial<Record<string, MyType>>. Drawback is that when
you do Object.values(toto), you'll get `Array<MyType | undefined>`.
Hence why we have to filter these behind
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/d0a0bfed-c441-4e53-84c2-2da98ccbcf50"
/>
- update publishOneServerlessFunction so it does not break the workflow
if failing
- update upgrade documentation to update `docker-compose.yml` when
mutated
Fixes https://github.com/twentyhq/twenty/issues/13058
I rebuilt the advanced filters used in views and workflow search for a
specific filter step.
Components structure remains the same, using `stepFilterGroups` and
`stepFilters`. But those filters are directly sent to backend.
Also re-using the same kind of states we use for advanced filters to
share the current filters used. And a context to share what's coming
from workflow props (function to update step settings and readonly)
⚠️ this PR only focusses on the content of the step. There is still a
lot to do on the filter icon behavior in the workflow
https://github.com/user-attachments/assets/8a6a76f0-11fa-444a-82b9-71fc96b18af4
In this PR
- introduction of fieldPermission entity
- addition of upsertFieldPermission in role resolver
- computing of permissions taking fieldPermission into account. In order
to limit what is stored in Redis we only store fields restrictions. For
instance for objectMetadata with id XXX with a restriction on field with
id YYY we store:
`"XXX":{"canRead":true,"canUpdate":false,"canSoftDelete":false,"canDestroy":false,"restrictedFields":{"YYY":{"canRead":false,"canUpdate":null}}}`
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
This PR is purely technical, it does produces any functional change to
the user
- add Lock mecanism to run steps concurrently
- update `workflow-executor.workspace-service.ts` to handle multi branch
workflow execution
- stop passing `context` through steps, it causes race condition issue
- refactor a little bit
- simplify `workflow-run.workspace-service.ts` to prepare `output` and
`context` removal
- move workflowRun status computing from `run-workflow.job.ts` to
`workflow-executor.workspace-service.ts`
## NOTA BENE
When a code step depends of 2 parents like in this config (see image
below)
If the form is submitted before the "Code - 2s" step succeed, the branch
merge "Form" step is launched twice.
- once because form is submission Succeed resumes the workflow in an
asynchronous job
- the second time is when the asynchronous job is launched when "Code -
2s" is succeeded
- the merge "Form" step makes the workflow waiting for response to
trigger the resume in another job
- during that time, the first resume job is launched, running the merge
"Form" step again
This issue only occurs with branch workflows. It will be solved by
checking if the currentStepToExecute is already in a SUCCESS state or
not
<img width="505" alt="image"
src="https://github.com/user-attachments/assets/b73839a1-16fe-45e1-a0d9-3efa26ab4f8b"
/>
In this PR, I'm fixing a bug introduced in recent performance work on
the cache.
Bug context: https://github.com/twentyhq/twenty/issues/12865
Related PR opened by a contributor:
https://github.com/twentyhq/twenty/pull/13003
## Root cause
We cache all objectMetadataItems at graphql level : see
`useCachedMetadata` hook:
- instead of going through the regular resolvers, we direlcty load data
from the cache. However this data must be localized regarding labels and
descriptions
In a precedent refactoring, we introduced the notion of locale in the
cache key. However, the user locale was not properly taken into account
as we did not have the information in this hook.
## Fix
1. **Introduce locale in userWorkspace entity**. The locale is stored on
workspaceMember in each postgres workspaceSchema (workspace_xxx) which
is the alter ego of userWorkspace in postgres core schema. Note that we
can't store it in user as a user can be part of multiple workspaces (the
locale already there must be seen as a default for this user), and we
cannot rely on workspaceMember as we would need to query the
workspaceSchema in the authentication layer which we want to avoid for
performance reasons.
2. During request hydration from token (containing the userWorkspaceId),
we fetch the userWorkspace and store it in the Request (this impact both
AuthContext and Request interface)
3. Leverage userWorkspace.locale in the useCachedMetadata hook
## Additional notes
There is no need to change the way we store and retrieve the
object-metadata-maps object itself which is different from the graphql
layer cache. object-metadadata-maps are not localized
Currently, when a server query or mutation from the front-end fails, the
error message defined server-side is displayed in a snackbar in the
front-end.
These error messages usually contain technical details that don't belong
to the user interface, such as "ObjectMetadataCollection not found" or
"invalid ENUM value for ...".
**BE**
In addition to the original error message that is still needed (for the
request response, debugging, sentry monitoring etc.), we add a
`displayedErrorMessage` that will be used in the snackbars. It's only
relevant to add it for the messages that will reach the FE (ie. not in
jobs or in rest api for instance) and if it can help the user sort out /
fix things (ie. we do add displayedErrorMessage for "Cannot create
multiple draft versions for the same workflow" or "Cannot delete
[field], please update the label identifier field first", but not
"Object metadata does not exist"), even if in practice in the FE users
should not be able to perform an action that will not work (ie should
not be able to save creation of multiple draft versions of the same
workflows).
**FE**
To ease the usage we replaced enqueueSnackBar with enqueueErrorSnackBar
and enqueueSuccessSnackBar with an api that only requires to pass on the
error.
If no displayedErrorMessage is specified then the default error message
is `An error occured.`
When we use a record field in a form, record relations are displayed as
available variables in following step.
But those are actually empty at execution.
When choosing the record in the form and submitting, we enrich the
record id with the full record before starting the workflow again. But
we were not adding the relations to that enrichment.
This PR does not produce any functional change
First step of the workflow branch feature
- add gather `workflowRun.output` and `workflowRun.context` into one
column `workflowRun.runContext`
- add a command to fill `runContext` from `output` and `context` in
existing records
- maintain `runContext` up to date during workflow runs
# Context
We had an error saying "Unknown error importing calendar events for
[...]: Access token is undefined or empty. Please provide a valid token.
For more help -
https://github.com/microsoftgraph/msgraph-sdk-javascript/blob/dev/docs/CustomAuthenticationProvider.md
"
Reason is that the access token method for microsoft is a bit different
than the one from google. And in microsoft case, we want to check the
access token in the authProvider in case it fails. Currently it was not
catched, so it broke services above that counted on the accesstoken to
be valid.
That ended in UNKNOWN failure for our calendar event fetch service.
# Solution
This PR should solve the issue since :
1. forcing the method to break if accesstoken renewal fails
2. logs will help to know what kind of errors will be sent in case we
need to tackle this issue again
3. we now throw TEMPORARY error instead of unknown, allowing 3
getClientConfig failure before it is definitive
Why so many changes while it should have been simple :
The root cause is the `authProvider` from
`'@microsoft/microsoft-graph-client'` npm package. It does not throw a
custom error, and we cannot catch it on calling `Client.init`. Errors
only occurs when the client from
```
const client = this.microsoftOAuth2ClientManagerService.getOAuth2Client(refreshtoken)
```
is used, as in `client.api('/messages').get().catch(err => [...])`
So we need to go in every call using the client and catch errors, and
rethrow whenver we need as a newly created message type
`MessageImportDriverExceptionCode.CLIENT_NOT_AVAILABLE`
We discussed 1. and 2. with @bosiraphael already
I added 3. to make our system more robust without waiting for more
failures
# Related
Fixes : https://github.com/twentyhq/twenty/issues/12880
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Previous logic was using the previous step output and filtering items
that were passing filters.
What we actually want is:
- send filters, right operand being always a step output key, left
operand being either a key, either a value
- resolve those filter variables
- apply the filters to decide whether the condition is passed or not
Fixes: #12722
The problem is that there is no TS_VECTOR field in workflow objects.
Thus, I have added this field to three objects: workflow,
workflowVersions, and workflowRuns.
---------
Co-authored-by: Thomas Trompette <thomas.trompette@sfr.fr>
Thomas Trompette