## Introduction
https://github.com/twentyhq/twenty/pull/14363
In this PR we're creating builder and runner for index metadata
Removing its previous integration within object builder
## Next:
- implem index impact on field and object metadata api
- Add integration testing coverage add integration test on index
- Implement uniqueness toggle in field metadata api
- add integration test on uniqueness
related to https://github.com/twentyhq/core-team-issues/issues/1344
# Introduction
Initially this PR was about introducing integration coverage for view
field v2 tests
But feature itself wasn't finished, so ended up in a TDD style
devlopment
## The new orchestrator
Orchestrator will organize each flat entity diffing inferring either
validation errors or actions sequentially.
A new builder has been introduced when extended requires to define flat
entity validation utils and action generation for `delete` `update`
`create` operations. It's highly typed using generics
## View field integration testing
Introduce coverage on failing basic tests cases for view field
operations, successfull create is also covered. We could still add more
coverage, will be done later in following PRs
## Remaining tasks for upcoming PRs:
- [ ] rename builder methods names to add generate action in their names
- [x] refactor view validation to handle field and view uniqueness index
combination
- [ ] implem strict update validator on view and view field
- [ ] dynamic cache invalidation post run
- [ ] Add coverage to successful and failing view field operations
- [ ] error formatting summary computation
- [ ] extract types out of the workspace entity builder
## Some vision
Currently flat field metadata maps is nested in the flat object metadata
maps, making its build nested too. ( not extending the new generic
builder )
We will refactor this part for both to be stored extra flat such as the
other existing flat entities.
It will require a small refactor to the generic builder that will not
only have impact on the currently built flat entity optimistic cache but
to an other one ( for example when building a create object action it
will also have an impact flat field maps )
Fixes, depending on the contextual current operation in order to
retrieve relation target and all morph related fields to be delete:
- DELETE Field,
- DELETE Object,
- UPDATE Field, that affects all others related fields
we need to handle both:
- morph relation field deletion ( implies every related morph and target
deletion )
- target RELATION only the pair that could contain a morph or not we
don't care
close https://github.com/twentyhq/core-team-issues/issues/1411
# Introduction
When udpating a flat object metadata name we need to search for RELATION
field that has a MORPH RELATION target flat field metadata with
MANY_TO_ONE relationType as its settings are binded to the updated flat
object metadata name
In the best of the world we would remove this complexity to be computed
at runtime only and deprecate implemented logic here
close https://github.com/twentyhq/core-team-issues/issues/1412
## Introduction
Binding morph relation validation to relation validation code flow
Please note that ALL create field will go through basic atomic
validation, which involve name availability and so on
close https://github.com/twentyhq/core-team-issues/issues/1407
# Introduction
From `morphRelationCreationPayload` to `flatFieldMetadatas`
close https://github.com/twentyhq/core-team-issues/issues/1406
## Unit testing
I'm not a fan of covering things this way, where we could have some
strong integration testing tests in the first place
But I wanted to freeze the optimistic path computation, also covered the
exception because it was cheap, doesn't mean I won't cover them through
integration tests later anw
- Enable lingui/no-single-variables-to-translate and all other
recommended Lingui rules
- Fix single variable translation patterns (t`${variable}` → variable)
- Fix expression-in-message violations by extracting variables
- Fix t-call-in-function violations by moving translations inside
functions
- Update ESLint configs to use linguiPlugin.configs['flat/recommended']
- Clean up unused imports and improve translation patterns
# Introduction
Migrating and improving performances of field enum integrations tests
success and failing tests cases to be using the new v2 api
## Discovered issue
When deleting an object in v1 it will leave related enums until the
object is re-created
Something not done anymore within the create in v2 but in the delete
operation
We should implem an upgrade command to remove such relicas
## Bugs
- Update/create default value multi select runner wrong sql query -> FIX
- Update default value multi select regression, we should allow option
without an id to be inserted -> FIX
- default value compare dynamic json stringify convertion or not in
compare tools for object and fields
## Context
- Adding ts-vector generatedType/asExpression as TS_VECTOR settings
- Using those settings to setup properly tsVector searchVector column
through the new migration runner
- Fix enum creation/suppression
Note: regarding the new tsVector, we should implement a command to
update existing fields
TODO:
- TS_VECTOR search vector column update (note: should be properly
updated whenever the object labelIdentifier is updated or a new TEXT
field is added to the object to follow the current logic)
- relation type fields and columns are not implemented yet
- index migrations
# Introduction
Moving validation directly in the builder that has the perfect
granularity to do it.
When importing we won't have to infer and dispatch on the operation
nature ( update delete create ) and validate accordingly
## Objects
Only migrated object validation for the moment even though create object
involves a validate flat field metadata creation call too
## TODO
- improve `otherFlatObjectMetadataMapsToValidate` naming too vague
## Next
- handle fields validation within fields actions build
- Unit test coverage validation issue on builder and validate
- integration test plugging with new feature flag
## Manual tested
- Update
- Delete
- Create TODO
# Introduction
- Implementing the delete one for the new object metadata service v2.
- Handling relation fields and fields in the first place to finally
remove the object
- puting back updatedAt and createdAt in flat metadatas
- duplicate criteria addition for flat object
- removing datasource id from object metadata dto
# Introduction
Introducing v2 object metadata service create one handler v2, not
overkilling the flatObjectMetadata validation for the moment that will
require sequential validation for the import in order to handle
relations and morph relations
# Introduction
- ~~returning deleted field on deleteOneField~~ postponed for later
cannot build dto from flat atm
- refactor createFieldInput transpilation to handle multiple errors
instead of throwing
- renamed extract function
# Introduction
- Migrated the `deleteOneField` handler to new workspace migration v2
style
- Refactored the build to expect `flatObjectMetadataMaps` that he
instantly translate at the be beginning
Commented `ObjectMetadataServiceV2` as it's still not implemented
neither used
# Introduction
The workspace migration runner v2 now computes the next
flatObjectMetadataMaps post current ws migration action has been run
So the following one have an up to date informations
## Still TODO
~~Remove any contextual information from workspace migration v2 and
consume the optimistic cache to retrieve them~~
## Next steps
- Create flat-object-metadata-maps testing toolbox and flat-object/field
testing toolbox
- Implement strong coverage on every created utils/transpiler and
applyWorkspaceMigrationAction method
# Introduction
- Retrieveing flat-object-metadata-maps for input transpilation, we
decided not to introduce map merge for remaining validation operation
for the moment
- Refactored the field metadata service v2 to sequentially
optimistically render on each iteration
# Introduction
- deferrable constraint on foreignKey definition
- fix relation creation input transpilation to flat field metadata
- fix validation to allow contextual validation on not already created
field metadata ( usefull for relation, and will be necessary for the
import )
# Introduction
The cache is not accurately typed regarding the `fieldMetadataMaps` they
do not contain relations
In this way we:
- Created dedicated transpilation tools for cache manipulation
- Avoided recursivity in transpilation tools
- Fix and finalized the createMany fields metadata service v2
- Refactor the validator to handle validation :)
## What's next:
- Implem integration tests to make things run 🙃
- migrate existing settings valdiation
- Finish the create object metadata service
- Handle update input transpilation and validation
# Introduction
- FlatFieldMetadataType validators hashmap
- Do not fail fast on validation but aggregate errors
- Implemented `enum` validation
- Plugged the new v2 dynamic call in the field metadata service v2
## What's next:
- Implem integration tests to make things run 🙃
- migrate existing settings valdiation
- Finish the create object metadata service
- Handle update input transpilation and validation
## Open question
Should we implement, not covered validation ?, adding strictness now or
never.
This will be required by the import anw
## Discovered issue with cache
Currently the cache is not accurately typed, `fieldsById` map are not
storing any relations.
Which means the current transpilation tools are hitting undefined at
runtime
In the best of the world we will refactor the cache to be storing
`FlatObjectMetadata` and `FlatFieldMetadata` so we don't even have to
transpile them for validation and so on
But it would require to refactor the loaders that returns the cache to
the front on hit as FieldMetadataEntity, so we might land on a lighter
solution to rather add a new `getExistingFlatCache` that handles the
transpilation itself
About to do that in an other PR to be discussed with Coco
# Introduction
Following https://github.com/twentyhq/twenty/pull/13420
What has been done:
- `CreateFieldInput` transpilation to `FlatFieldMetadata`
- `FlatFieldMetadata` validator service
- A lof of transpilation utils from `input` to `flatObject` or
`flatField`
- Created dedicated v2 api metadata services
- Introducing `inferDeletionFromMissingObjectFieldIndex` in the builder,
to avoid diffing every object and field of the current workspace we
allow only generating create/update migration operations, usefull when
passing by the api metadata
## We still need to in another PR:
- Implement a strong unit test coverage and critical functions and
services
- Finalize flat field metadata validation exception for `options`
`defaultValue` `settings` and `relations`
- Finalize `flatObjectMetadata` validation and v2 service refactor
- Plug the new service when feature flag is enabled
# Introduction
In this PR we create basic transpilation methods and utils to handle
input to flat, entity to flat, object maps to flat. In order to
transpile everything into a common validation that will be implemented
in another PR
## FieldMetadataEntity typing
Added `never | null` to fields that should never be in order to ease
general abstracted method to pass null, as anw it's what is in the
database
## Todo
- ~~Create a feature flag~~
- Integration test for object creation through metadata api + pg col
introspection and snapshoting
Paul Rastoin