Commit Graph

34 Commits

Author SHA1 Message Date
Paul Rastoin 442f8dbe3c [QRQC_2] No implicitAny in twenty-server (#12075)
# Introduction
Following https://github.com/twentyhq/twenty/pull/12068
Related with https://github.com/twentyhq/core-team-issues/issues/975

We're enabling `noImplicitAny` handled few use case manually, added a
`ts-expect-error` to the others, we should plan to handle them in the
future
2025-05-15 18:23:22 +02:00
Félix Malfait 7b78b64bca Update clickhouse tables (#11905)
Following a discussion with @Bonapara - changing the base tables
2025-05-07 09:39:18 +02:00
Félix Malfait 49b7f5255f Update what is being audit logged (#11833)
No need to audit log workflow runs as it's already a form of audit log.
Add more audit log for other objects
Rename MessagingTelemetry to MessagingMonitoring
Merge Analytics and Audit in one (Audit)

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-05-04 14:35:41 +02:00
Antoine Moreaux 587281a541 feat(analytics): add clickhouse (#11174) 2025-04-16 16:33:10 +00:00
martmull 1794b3a611 Remove build lambda at creation (#11531)
Remove useless build call (done at first execution)
2025-04-11 14:40:00 +02:00
nitin bd3ec6d5e3 rename core-module environment to twenty-config (#11445)
closes https://github.com/twentyhq/core-team-issues/issues/759
2025-04-09 14:11:26 +02:00
Paul Rastoin 9ad8287dbc [REFACTOR] twenty-shared multi barrel and CJS/ESM build with preconstruct (#11083)
# Introduction

In this PR we've migrated `twenty-shared` from a `vite` app
[libary-mode](https://vite.dev/guide/build#library-mode) to a
[preconstruct](https://preconstruct.tools/) "atomic" application ( in
the future would like to introduce preconstruct to handle of all our
atomic dependencies such as `twenty-emails` `twenty-ui` etc it will be
integrated at the monorepo's root directly, would be to invasive in the
first, starting incremental via `twenty-shared`)

For more information regarding the motivations please refer to nor:
- https://github.com/twentyhq/core-team-issues/issues/587
-
https://github.com/twentyhq/core-team-issues/issues/281#issuecomment-2630949682

close https://github.com/twentyhq/core-team-issues/issues/589
close https://github.com/twentyhq/core-team-issues/issues/590

## How to test
In order to ease the review this PR will ship all the codegen at the
very end, the actual meaning full diff is `+2,411 −114`
In order to migrate existing dependent packages to `twenty-shared` multi
barrel new arch you need to run in local:
```sh
yarn tsx packages/twenty-shared/scripts/migrateFromSingleToMultiBarrelImport.ts && \
npx nx run-many -t lint --fix -p twenty-front twenty-ui twenty-server twenty-emails twenty-shared twenty-zapier
```
Note that `migrateFromSingleToMultiBarrelImport` is idempotent, it's atm
included in the PR but should not be merged. ( such as codegen will be
added before merging this script will be removed )

## Misc
- related opened issue preconstruct
https://github.com/preconstruct/preconstruct/issues/617

## Closed related PR
- https://github.com/twentyhq/twenty/pull/11028
- https://github.com/twentyhq/twenty/pull/10993
- https://github.com/twentyhq/twenty/pull/10960

## Upcoming enhancement: ( in others dedicated PRs )
- 1/ refactor generate barrel to export atomic module instead of `*`
- 2/ generate barrel own package with several files and tests
- 3/ Migration twenty-ui the same way
- 4/ Use `preconstruct` at monorepo global level

## Conclusion
As always any suggestions are welcomed !
2025-03-22 19:16:06 +01:00
martmull 927b8c717e Poc lambda deployment duration (#10340)
closes https://github.com/twentyhq/core-team-issues/issues/436

## Acheivements
Improve aws lambda deployment time from ~10/15 secs to less that 1 sec

## Done
- migrate with the new code executor architecture for local and lambda
drivers
- support old and new executor architecture to avoid breaking changes
- first run is long, next runs are quick even if code step is updated

## Demo using `lambda` driver
### Before


https://github.com/user-attachments/assets/7f7664b4-658f-4689-8949-ea2c31131252


### After



https://github.com/user-attachments/assets/d486c8e2-f8f8-4dbd-a801-c9901e440b29
2025-02-20 10:49:57 +01:00
martmull b66289c44c replace log by throw (#10167)
as title
2025-02-12 16:31:41 +00:00
Paul Rastoin 7fd89678b7 [CHORE] Avoid isDefined duplicated reference, move it to twenty-shared (#9967)
# Introduction
Avoid having multiple `isDefined` definition across our pacakges
Also avoid importing `isDefined` from `twenty-ui` which exposes a huge
barrel for a such little util function

## In a nutshell
Removed own `isDefined.ts` definition from `twenty-ui` `twenty-front`
and `twenty-server` to move it to `twenty-shared`.
Updated imports for each packages, and added explicit dependencies to
`twenty-shared` if not already in place

Related PR https://github.com/twentyhq/twenty/pull/9941
2025-02-01 12:10:10 +01:00
martmull ae62789159 Serverless function follow up (#9924)
- remove asynchronous serverless function build
- build serverless function synchronously instead on activate workflow
or execute
- add a loader on workflow code step test tab test button
- add a new `ServerlessFunctionSyncStatus` `BUILDING`
- add a new route to build a serverless function draft version 
- delay artificially execution to avoid UI flashing



https://github.com/user-attachments/assets/8d958d9a-ef41-4261-999e-6ea374191e33
2025-01-31 17:12:42 +01:00
martmull ed7c48e12a Fix use as draft (#9718)
- remove delete serverless function when archiving workflow version
- update copy serverless function to reset serverless function to old
version
- remove createNewWorkflowVersion and use createDraftFromWorkflowVersion
- fix step update issue and optimistic rendering when generate draft
from active version
2025-01-21 15:44:52 +01:00
martmull f8f9bb2b78 Serverless function timeout concerns (#9689)
closes https://github.com/twentyhq/core-team-issues/issues/242
- unify timeout behavior between local and lambda
- add timeout in serverless entity
- set timeout default to 300s (5min)
2025-01-17 13:49:02 +00:00
martmull 5783d68d62 Remove serverless functions on version archivation (#9535)
Fixes https://github.com/twentyhq/core-team-issues/issues/52
- contrary to title, we do not remove serverless functions on workflow
version archivation because serverless fucntion might be used in another
workflow version
- we fix the serverless funciton version displayed in the code step
- we allow test function version in step display right drawer
- we delete serverless function only when serverless function has no
published version
2025-01-13 13:09:57 +00:00
martmull 0c75b244ba Remove serverless functions on soft delete (#9438)
Delete workflow versions serverless functions when soft delete workflow
2025-01-07 15:51:34 +00:00
Baptiste Devessier 17bf2b6173 Move the workflow draft version overriding to the backend (#9328)
- In the `formatFieldMetadataValue` function, allow people to call
TypeORM's `save()` method with unserialized JSON data.
- Create an `overrideWorkflowDraftVersion` mutation that takes a
workflow id and the id of the workflow version to use as the new draft
  - If no draft exists yet, create one
  - If a draft already exists, deactivate its serverless functions
- Duplicate every step. For serverless function steps, it includes
duplicating the functions
  - Save the data of the step in DB
- Call the `overrideWorkflowDraftVersion` mutation in the old workflow
header and in the new Cmd+K actions
- I chose to not update the Apollo cache manually as the information of
the new draft are going to be automatically fetched once the user lands
on the workflow's show page. Note that we redirect the user to this page
after overriding the draft version.
2025-01-06 14:56:09 +01:00
martmull 5dfcc413cf 9024 workflow test serverless function follow up (#9066)
-  Fix Tablist style
- Fix dropdown style (wrong grey background)
- Update dropdown variable when no outputSchema is available 



https://github.com/user-attachments/assets/56698fe8-8dd3-404a-b2b2-f1eca6f5fa28
2024-12-17 10:35:38 +01:00
martmull 2b3b073570 8725 workflow avoid serverless function autosave errors (#8916)
See issue #8725 
- Build function asynchronously using a job
- prevent useless builds
- run promises simultaneously

Todo:
- fix outputSchema computing
2024-12-06 10:13:12 +00:00
martmull d0ff1ffd5f 8723 workflow add editor in serverless function code step (#8805)
- create a serverless function when creating a new workflow code step
- add code editor in workflow code step
- move workflowVersion steps management from frontend to backend
  - add a custom resolver for workflow-version management
  - fix optimistic rendering on frontend
- fix css
- delete serverless function when deleting workflow code step

TODO
- Don't update serverlessFunction if no code change
- Factorize what can be between crud trigger and crud step
- Publish serverless version when activating workflow
- delete serverless functions when deleting workflow or workflowVersion
- fix optimistic rendering for code updates
- Unify CRUD types

<img width="1279" alt="image"
src="https://github.com/user-attachments/assets/3d97ee9f-4b96-4abc-9d36-5c0280058be4">
2024-12-03 08:41:13 +00:00
martmull 39373b4a28 8643 fix sentry error (#8644)
- fixes missing data in event payload when adding a new workspaceMember
- add strong typing to database event emitters
2024-11-21 16:09:36 +00:00
Thomas Trompette be8141ce5e Infer function input in workflow step (#8308)
- add `inputSchema` column in serverless function. This is an array of
parameters, with their name and type
- on serverless function id update, get the `inputSchema` + store empty
settings in step
- from step settings, build the form 

TODO in next PR:
- use field type to decide what kind of form should be printed
- have a strategy to handle object as input



https://github.com/user-attachments/assets/ed96f919-24b5-4baf-a051-31f76f45e575
2024-11-05 14:57:06 +01:00
Ana Sofia Marin Alexandre 29bd4e5f2d track serverless functions executions (#7963)
Solves: 
https://github.com/twentyhq/private-issues/issues/74

**TLDR**
When a serverless function is executed, the result is send to tinybird
event data source.

**In order to test:**

1. Set ANALYTICS_ENABLED to true
2. Put your TINYBIRD_INGEST_TOKEN from twenty_event_playground in your
.env file
3.  Don't forget to run the worker
4. Create your serverless function and run it
5. The event should be logged on the event datasource in
twenty_event_playground

**What is the structure of the payload of a serverless function event?**

Here are two examples of the payload:

`{"duration":37,"status":"SUCCESS","functionId":"a9fd87c0-af86-4e17-be3a-a6d3d961678a","functionName":"testingFunction"}`

`

{"duration":34,"status":"ERROR","errorType":"ReferenceError","functionId":"a9fd87c0-af86-4e17-be3a-a6d3d961678a","functionName":"testingFunction"}`

**Possible improvments**

- Change the status(str) to success(bool) 
- Enrich data in the payload
2024-10-22 17:51:51 +02:00
martmull e767f16dbe 7415 serverless functions update environment variables in a dedicated tab in settings functions not a env file (#7939)
![image](https://github.com/user-attachments/assets/0ef9551d-d867-479e-9a76-faee6930bc0a)

![image](https://github.com/user-attachments/assets/a7aac417-4dd8-401f-8d5b-5b72f31710f6)

![image](https://github.com/user-attachments/assets/16c98e52-a2db-4ed3-b5d2-77745b4d2918)

![image](https://github.com/user-attachments/assets/847d23d6-8a58-4d8f-aff1-4f8a81862964)
2024-10-22 14:51:03 +02:00
Thomas Trompette f338d01b4f Build code introspection service (#7760)
Starting to use ts-morph to retrieve function parameters
2024-10-17 15:08:42 +02:00
martmull 62fe1d0e88 6653 serverless functions store and use environment variables in serverless function scripts (#7390)
![image](https://github.com/user-attachments/assets/a15bd4c1-3db4-4466-b748-06bdf3874354)

![image](https://github.com/user-attachments/assets/71242dfb-956b-43ed-9704-87cb0dfbc98d)
2024-10-03 13:56:17 +02:00
martmull 3190f4a87b 6658 workflows add a first twenty piece email sender (#6965) 2024-09-12 11:00:25 +02:00
martmull 7e03419c16 Serverless function improvements (#6769)
- add layer for lambda execution
- add layer for local execution
- add package resolve for the monaco editor
- add route to get installed package for serverless functions
- add layer versioning
2024-09-02 15:25:20 +02:00
Thomas Trompette 9f69383aa2 Add function execution throttler (#6742)
Add throttler service to limit the number of function execution
2024-08-27 18:56:47 +02:00
martmull 873a4c1bd1 Fix serverless save when name empty (#6720)
- fix serverless function error on save when name empty
- remove useless unique constraint on serverless function names
2024-08-23 17:34:31 +02:00
martmull 6f9aa1e870 6654 serverless functions add a deploy button disable deploy when autosave (#6715)
- improvements on serverless function behavior (autosave performances,
deploy on execution only)
- add versioning to serverless functions
- add a publish endpoint to create a new version of a serverless
function
  - add deploy and reset to lastVersion button in the settings section:
<img width="736" alt="image"
src="https://github.com/user-attachments/assets/2001f8d2-07a4-4f79-84dd-ec74b6f301d3">
2024-08-23 12:06:03 +02:00
Félix Malfait 4157a67bf8 Fix permissions for serverless functions (#6555)
Fixes #6525

(@martmull fyi it was not related to AWS but linked to the fact that we
recently enforced passing a token to access files)
2024-08-06 15:31:22 +02:00
martmull 6b4c79ff0d Add workflow runner (#6458)
- add workflow runner module
- add an endpoint to trigger a workflow via api
- improve error handling for serverless functions

## Testing
- create 2 serverless functions
- create a workflow
- create this workflow Version
```
{
  "type": "MANUAL",
  "input": {"b": "bb"},
  "nextAction": {
    "name": "step_1",
    "displayName": "Code",
    "type": "CODE",
    "valid": true,
    "settings": {
      "serverlessFunctionId": "Serverless function 1 Id",
      "errorHandlingOptions": {
        "retryOnFailure": {
          "value": false
        },
        "continueOnFailure": {
          "value": false
        }
      }
    },
    "nextAction": {
      "name": "step_1",
      "displayName": "Code",
      "type": "CODE",
      "valid": true,
      "settings": {
        "serverlessFunctionId": "Serverless function 1 Id",
        "errorHandlingOptions": {
          "retryOnFailure": {
            "value": false
          },
          "continueOnFailure": {
            "value": false
          }
        }
      },
      "nextAction": {
        "name": "step_1",
        "displayName": "Code",
        "type": "CODE",
        "valid": true,
        "settings": {
          "serverlessFunctionId": "Serverless function 2 Id",
          "errorHandlingOptions": {
            "retryOnFailure": {
              "value": false
            },
            "continueOnFailure": {
              "value": false
            }
          }
        }
      }
    }
  }
}

`
``
- call 
```
mutation Trigger {
  triggerWorkflow(workflowVersionId: "WORKFLOW_VERSION_ID") {
    result
  }
}
```
- try when errors are injected in serverless function
2024-07-31 12:48:33 +02:00
martmull 00fea17920 Serverless function UI (#6388)
https://www.figma.com/design/xt8O9mFeLl46C5InWwoMrN/Twenty?node-id=36235-120877

Did not do the file manager part. A Function is defined using one unique
file at the moment

Feature protected by featureFlag `IS_FUNCTION_SETTINGS_ENABLED`

## Demo


https://github.com/user-attachments/assets/0acb8291-47b4-4521-a6fa-a88b9198609b
2024-07-29 13:03:09 +02:00
martmull 47ddc7be83 6181 workflows create a custom code executor (#6235)
Closes #6181

## Testing
- download Altair graphql dev tool https://altairgraphql.dev/#download
- create a file locally `test.ts` containing:
```
export const handler = async (event: object, context: object) => {
  return { test: 'toto', data: event['data'] };
}
```
- play those requests in Altair:
mutation UpsertFunction($file: Upload!) {
  upsertFunction(name: "toto", file: $file)
}

mutation ExecFunction {
  executeFunction(name:"toto", payload: {data: "titi"})
}
- it will run the local driver, add those env variable to test with
lambda driver
```
CUSTOM_CODE_ENGINE_DRIVER_TYPE=lambda
LAMBDA_REGION=eu-west-2
LAMBDA_ROLE=<ASK_ME>
```
2024-07-17 17:53:01 +02:00