[HACKTOBERFEST] Add rollup engine app with UI-driven configuration (#15482)
## Summary
- add packages/twenty-apps/rollup-engine: a parameterised rollup engine
that ships a default Opportunity → Company
aggregation.
- declare runtime config in package.json (TWENTY_API_KEY, optional
TWENTY_API_BASE_URL, and ROLLUP_ENGINE_CONFIG) so
app configuration lives entirely in Settings → Apps → [App] →
Configuration.
- document the workflow in README.md: deploy via twenty app sync,
populate env vars in the UI, use the Test panel with
a ready JSON payload, and reference troubleshooting tips.
- adjust the function entry point to a named async export and add
fallback logic for blank base URLs, matching the
UI’s env behaviour.
- prune legacy templates and examples so
config/templates/opportunity-to-company.json is the single copy/paste
starting point.
## UI / UX impact
After syncing the app:
- the Configuration screen shows the three env keys with helpful
descriptions (API key, optional base URL, JSON
override),
- the built-in Test your function panel works immediately
- and the default JSON config is available from
config/templates/opportunity-to-company.json for users who need to
customise rollups.
## Testing
- Out-of-the-box deploy to the hosted workspace (Opportunity update) ✓
- “Test your function” with the default config ✓
- Override example: point debugOpportunityCount at a scratch field via
ROLLUP_ENGINE_CONFIG ✓
- Optional: local smoke test (yarn install && yarn smoke) still passes
This commit is contained in:
+942
File diff suppressed because one or more lines are too long
@@ -0,0 +1,107 @@
|
||||
# Rollup engine
|
||||
|
||||
General-purpose rollup engine for Twenty workspaces. The bundle ships a
|
||||
serverless function that materialises aggregations from a child object onto a
|
||||
parent object. The default configuration targets Opportunity → Company rollups.
|
||||
|
||||
## Requirements
|
||||
- `twenty-cli` `npm install -g twenty-cli`
|
||||
- an API key with access to your workspace. Generate one at
|
||||
`https://twenty.com/settings/api-webhooks`.
|
||||
- Node 20+ (only required if you plan to run the optional smoke test locally).
|
||||
|
||||
## Metadata prerequisites
|
||||
- Opportunity is a standard object, no extra provisioning required.
|
||||
- Add these fields to the `company` object (API names shown):
|
||||
- `totalPipelineAmount` (Currency)
|
||||
- `totalOpportunityCount` (Number)
|
||||
- `wonPipelineAmount` (Currency)
|
||||
- `wonOpportunityCount` (Number)
|
||||
- `openPipelineAmount` (Currency)
|
||||
- `openOpportunityCount` (Number)
|
||||
- `lastOpportunityCloseDate` (Date)
|
||||
- To script the setup, export `TWENTY_API_KEY` (and `TWENTY_METADATA_BASE_URL` if you are not targeting `http://localhost:3000/rest/metadata`) and run `yarn setup:metadata`. The helper script calls the Metadata API to ensure each field exists on the Company object, skipping anything already provisioned.
|
||||
|
||||
Create the fields before syncing so PATCH requests succeed.
|
||||
|
||||
## Quick start
|
||||
|
||||
1. **Deploy the app**
|
||||
```bash
|
||||
twenty auth login
|
||||
cd rollup-engine
|
||||
twenty app sync
|
||||
```
|
||||
|
||||
2. **Configure environment variables**
|
||||
- Open **Settings → Apps → Rollup engine → Configuration**.
|
||||
- Provide values for the manifest-defined keys:
|
||||
- `TWENTY_API_KEY` (required secret; create it in Twenty → Settings → API & Webhooks).
|
||||
- `TWENTY_API_BASE_URL` (optional; leave blank to use `https://app.twenty.com/rest`).
|
||||
- `ROLLUP_ENGINE_CONFIG` (optional JSON override; leave blank to use the baked-in config).
|
||||
- Save the configuration. Changes are applied immediately—no redeploy required.
|
||||
|
||||
3. **Verify via the Test panel**
|
||||
- Still on the app page, select **Test**.
|
||||
- Paste a JSON payload with the company ID you want to validate:
|
||||
|
||||
```json
|
||||
{
|
||||
"trigger": { "type": "databaseEvent", "eventName": "opportunity.updated" },
|
||||
"record": { "companyId": "YOUR_COMPANY_ID" },
|
||||
"opportunity": { "companyId": "YOUR_COMPANY_ID" }
|
||||
}
|
||||
```
|
||||
|
||||
- Click **Run**. You should see a success summary and the company’s rollup fields updating.
|
||||
|
||||
4. **Trigger live rollups**
|
||||
- Any Opportunity create/update/delete (or the nightly cron) now feeds into the Company metrics.
|
||||
|
||||
## Runtime configuration
|
||||
- `serverlessFunctions/calculaterollups/src/rollupConfig.ts` holds the baked-in defaults.
|
||||
Override them by setting `ROLLUP_ENGINE_CONFIG` (aliases: `ROLLUPS_CONFIG`,
|
||||
`CALCULATE_ROLLUPS_CONFIG`) to a JSON array in the Configuration UI.
|
||||
- `config/templates/opportunity-to-company.json` matches the default rollups shipped with the bundle.
|
||||
Copy it, tweak the object/field names you need, and paste the edited JSON into `ROLLUP_ENGINE_CONFIG`.
|
||||
- `filters[].dynamicValue` currently supports `"startOfYear"` (UTC midnight on the first day of the
|
||||
current calendar year).
|
||||
|
||||
### Default Opportunity → Company rollups
|
||||
| Parent field | Suggested type | Description |
|
||||
| --- | --- | --- |
|
||||
| `totalPipelineAmount` | Currency (`amountMicros`, `currencyCode`) | Sum of positive-valued Opportunities linked to the Company. |
|
||||
| `totalOpportunityCount` | Number | Count of Opportunities linked to the Company. |
|
||||
| `wonPipelineAmount` | Currency (`amountMicros`, `currencyCode`) | Sum of Opportunities where `stage === "CUSTOMER"`. |
|
||||
| `wonOpportunityCount` | Number | Count of Opportunities where `stage === "CUSTOMER"`. |
|
||||
| `openPipelineAmount` | Currency (`amountMicros`, `currencyCode`) | Sum of Opportunities where `stage !== "CUSTOMER"`. |
|
||||
| `openOpportunityCount` | Number | Count of Opportunities where `stage !== "CUSTOMER"`. |
|
||||
| `lastOpportunityCloseDate` | Date | Most recent `closeDate` (ISO date). |
|
||||
|
||||
Assumptions:
|
||||
- Opportunities expose `companyId`, `amount.amountMicros`, `amount.currencyCode`,
|
||||
`stage`, and `closeDate`. Adjust filters or fields as needed via the JSON
|
||||
config.
|
||||
- Amount totals are stored in micros (integers) to align with Twenty’s composite
|
||||
currency fields.
|
||||
|
||||
## Optional local smoke test
|
||||
Run this if you want to exercise the aggregation logic without calling the live API.
|
||||
|
||||
```bash
|
||||
cd rollup-engine
|
||||
yarn install
|
||||
yarn smoke
|
||||
```
|
||||
|
||||
The smoke script replaces `fetch` with an in-memory mock, executes the rollup function using sample
|
||||
Opportunity data, and asserts the PATCH payload that would be sent to Twenty.
|
||||
|
||||
## Troubleshooting
|
||||
- **`Test your function` reports `mainFile.main is not a function`:** delete the app
|
||||
(`twenty app delete`) and sync it again (`twenty app sync`) to clear any stale bundle.
|
||||
- **`status: "error", message: "Unable to parse rollup configuration override"`:** confirm that
|
||||
`ROLLUP_ENGINE_CONFIG` contains valid JSON or leave it blank to fall back to the baked-in defaults.
|
||||
- **No company updates after configuring everything:** double-check the Company field API names,
|
||||
confirm the Opportunity records have `amount.amountMicros > 0`, and use the Test panel to inspect
|
||||
the summary details/logs.
|
||||
@@ -0,0 +1,79 @@
|
||||
[
|
||||
{
|
||||
"parentObject": "company",
|
||||
"childObject": "opportunity",
|
||||
"relationField": "companyId",
|
||||
"childFilters": [
|
||||
{
|
||||
"field": "amount.amountMicros",
|
||||
"operator": "gt",
|
||||
"value": 0
|
||||
}
|
||||
],
|
||||
"aggregations": [
|
||||
{
|
||||
"type": "SUM",
|
||||
"childField": "amount.amountMicros",
|
||||
"parentField": "totalPipelineAmount",
|
||||
"currencyField": "amount.currencyCode"
|
||||
},
|
||||
{
|
||||
"type": "COUNT",
|
||||
"parentField": "totalOpportunityCount"
|
||||
},
|
||||
{
|
||||
"type": "SUM",
|
||||
"childField": "amount.amountMicros",
|
||||
"parentField": "wonPipelineAmount",
|
||||
"currencyField": "amount.currencyCode",
|
||||
"filters": [
|
||||
{
|
||||
"field": "stage",
|
||||
"operator": "equals",
|
||||
"value": "CUSTOMER"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "COUNT",
|
||||
"parentField": "wonOpportunityCount",
|
||||
"filters": [
|
||||
{
|
||||
"field": "stage",
|
||||
"operator": "equals",
|
||||
"value": "CUSTOMER"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "SUM",
|
||||
"childField": "amount.amountMicros",
|
||||
"parentField": "openPipelineAmount",
|
||||
"currencyField": "amount.currencyCode",
|
||||
"filters": [
|
||||
{
|
||||
"field": "stage",
|
||||
"operator": "notEquals",
|
||||
"value": "CUSTOMER"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "COUNT",
|
||||
"parentField": "openOpportunityCount",
|
||||
"filters": [
|
||||
{
|
||||
"field": "stage",
|
||||
"operator": "notEquals",
|
||||
"value": "CUSTOMER"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "MAX",
|
||||
"childField": "closeDate",
|
||||
"parentField": "lastOpportunityCloseDate"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"version": "0.0.1",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": "^24.5.0",
|
||||
"npm": "please-use-yarn",
|
||||
"yarn": ">=4.0.2"
|
||||
},
|
||||
"packageManager": "yarn@4.9.2",
|
||||
"$schema": "https://raw.githubusercontent.com/twentyhq/twenty/main/packages/twenty-cli/schemas/appManifest.schema.json",
|
||||
"universalIdentifier": "2b308f8c-6ff8-4838-9880-aa0271dfd8d8",
|
||||
"name": "Rollup engine",
|
||||
"description": "Rollup engine",
|
||||
"env": {
|
||||
"TWENTY_API_KEY": {
|
||||
"isSecret": true,
|
||||
"value": "",
|
||||
"description": "Workspace API key used by the rollup engine to call the Twenty REST API."
|
||||
},
|
||||
"TWENTY_API_BASE_URL": {
|
||||
"isSecret": false,
|
||||
"value": "",
|
||||
"description": "Optional override for the REST base URL (defaults to https://app.twenty.com/rest)."
|
||||
},
|
||||
"ROLLUP_ENGINE_CONFIG": {
|
||||
"isSecret": false,
|
||||
"value": "",
|
||||
"description": "Optional JSON override for rollup definitions. Leave blank to use the baked-in defaults."
|
||||
}
|
||||
},
|
||||
"scripts": {
|
||||
"smoke": "node --import tsx scripts/run-rollup-smoke.ts",
|
||||
"setup:metadata": "node scripts/setup-company-rollup-fields.mjs"
|
||||
},
|
||||
"devDependencies": {
|
||||
"dotenv": "^16.4.5",
|
||||
"tsx": "^4.20.6"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,154 @@
|
||||
import assert from 'node:assert/strict';
|
||||
|
||||
import { main as runRollups } from '../serverlessFunctions/calculaterollups/src/index.ts';
|
||||
|
||||
type Json = Record<string, unknown>;
|
||||
|
||||
interface RequestLogEntry {
|
||||
url: string;
|
||||
method: string;
|
||||
body?: unknown;
|
||||
}
|
||||
|
||||
const now = new Date();
|
||||
const currentYear = now.getUTCFullYear();
|
||||
const previousYear = currentYear - 1;
|
||||
|
||||
const mockOpportunities = [
|
||||
{
|
||||
id: 'opp-1',
|
||||
companyId: 'company-1',
|
||||
amount: { amountMicros: 150_000_000, currencyCode: 'USD' },
|
||||
stage: 'CUSTOMER',
|
||||
closeDate: `${currentYear}-01-10T12:00:00.000Z`,
|
||||
},
|
||||
{
|
||||
id: 'opp-2',
|
||||
companyId: 'company-1',
|
||||
amount: { amountMicros: 90_000_000, currencyCode: 'USD' },
|
||||
stage: 'PROPOSAL',
|
||||
closeDate: `${currentYear}-03-05T18:00:00.000Z`,
|
||||
},
|
||||
{
|
||||
id: 'opp-3',
|
||||
companyId: 'company-1',
|
||||
amount: { amountMicros: 60_000_000, currencyCode: 'USD' },
|
||||
stage: 'SCREENING',
|
||||
closeDate: `${previousYear}-12-15T09:00:00.000Z`,
|
||||
},
|
||||
];
|
||||
|
||||
const requestLog: RequestLogEntry[] = [];
|
||||
const updatePayloads: Array<{ id: string; payload: Json }> = [];
|
||||
|
||||
const jsonResponse = (data: Json | Json[]) =>
|
||||
new Response(JSON.stringify(data), {
|
||||
status: 200,
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
});
|
||||
|
||||
global.fetch = async (
|
||||
rawUrl: string | URL,
|
||||
init?: { method?: string; body?: unknown },
|
||||
): Promise<any> => {
|
||||
const url = typeof rawUrl === 'string' ? new URL(rawUrl) : rawUrl;
|
||||
const method = (init?.method ?? 'GET').toUpperCase();
|
||||
requestLog.push({
|
||||
url: url.toString(),
|
||||
method,
|
||||
body: init?.body ? safeParse(init.body) : undefined,
|
||||
});
|
||||
|
||||
if (url.pathname.endsWith('/opportunities') && method === 'GET') {
|
||||
const companyId = url.searchParams.get('filter[companyId]');
|
||||
const items = companyId
|
||||
? mockOpportunities.filter((opportunity) => opportunity.companyId === companyId)
|
||||
: mockOpportunities;
|
||||
return jsonResponse({
|
||||
data: {
|
||||
opportunities: items,
|
||||
},
|
||||
pageInfo: {
|
||||
hasNextPage: false,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
if (url.pathname.includes('/companies/') && method === 'PATCH') {
|
||||
const id = url.pathname.split('/').pop() ?? 'unknown';
|
||||
const payload = safeParse(init?.body) ?? {};
|
||||
updatePayloads.push({ id, payload });
|
||||
return jsonResponse({ data: { companies: [{ id, ...payload }] } });
|
||||
}
|
||||
|
||||
throw new Error(`Unhandled request in mock fetch: ${method} ${url.toString()}`);
|
||||
};
|
||||
|
||||
function safeParse(body: unknown) {
|
||||
if (!body) {
|
||||
return undefined;
|
||||
}
|
||||
if (typeof body === 'string') {
|
||||
try {
|
||||
return JSON.parse(body) as Json;
|
||||
} catch (error) {
|
||||
console.warn('Failed to parse request body', error);
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
process.env.TWENTY_API_KEY = 'mock-api-key';
|
||||
process.env.TWENTY_API_BASE_URL = 'https://mock.twenty/api';
|
||||
|
||||
const params = {
|
||||
trigger: { type: 'databaseEvent' },
|
||||
record: { companyId: 'company-1' },
|
||||
opportunity: { companyId: 'company-1' },
|
||||
};
|
||||
|
||||
const result = await runRollups(params);
|
||||
|
||||
assert.equal(result.status, 'ok', 'rollup execution should succeed');
|
||||
|
||||
const expectedPayload = {
|
||||
totalPipelineAmount: { amountMicros: 300_000_000, currencyCode: 'USD' },
|
||||
totalOpportunityCount: 3,
|
||||
wonPipelineAmount: { amountMicros: 150_000_000, currencyCode: 'USD' },
|
||||
wonOpportunityCount: 1,
|
||||
openPipelineAmount: { amountMicros: 150_000_000, currencyCode: 'USD' },
|
||||
openOpportunityCount: 2,
|
||||
lastOpportunityCloseDate: `${currentYear.toString().padStart(4, '0')}-03-05`,
|
||||
};
|
||||
|
||||
const companyUpdate = updatePayloads.find((payload) => payload.id === 'company-1');
|
||||
assert(companyUpdate, 'expected a PATCH payload for company-1');
|
||||
assert.deepStrictEqual(companyUpdate.payload, expectedPayload);
|
||||
|
||||
const opportunitiesRequest = requestLog.find(
|
||||
(entry) => entry.method === 'GET' && entry.url.includes('/opportunities'),
|
||||
);
|
||||
assert(opportunitiesRequest, 'expected at least one GET request for /opportunities');
|
||||
const filterParams = new URL(opportunitiesRequest.url).searchParams.getAll('filter');
|
||||
assert.deepStrictEqual(
|
||||
filterParams,
|
||||
['companyId[eq]:"company-1"'],
|
||||
'expected opportunity filter to use the new syntax',
|
||||
);
|
||||
|
||||
console.log('--- Rollup execution summary ---');
|
||||
console.dir(result, { depth: null });
|
||||
|
||||
console.log('\n--- PATCH payloads sent to companies ---');
|
||||
console.dir(updatePayloads, { depth: null });
|
||||
|
||||
console.log('\n--- Requests made ---');
|
||||
console.dir(requestLog, { depth: null });
|
||||
}
|
||||
|
||||
main().catch((error) => {
|
||||
console.error('Smoke test failed', error);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
@@ -0,0 +1,138 @@
|
||||
import { config as loadEnv } from 'dotenv';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = path.dirname(__filename);
|
||||
|
||||
const envCandidates = [
|
||||
process.env.TWENTY_ENV_PATH ? path.resolve(process.env.TWENTY_ENV_PATH) : null,
|
||||
path.resolve(__dirname, '..', '.env'),
|
||||
path.resolve(__dirname, '..', '..', '..', '..', '..', '.env'),
|
||||
].filter(Boolean);
|
||||
|
||||
envCandidates.forEach((candidate) => {
|
||||
if (!candidate) {
|
||||
return;
|
||||
}
|
||||
const result = loadEnv({ path: candidate });
|
||||
if (!result.error && process.env.TWENTY_API_KEY) {
|
||||
return;
|
||||
}
|
||||
});
|
||||
|
||||
const METADATA_BASE_URL = (
|
||||
process.env.TWENTY_METADATA_BASE_URL ??
|
||||
(process.env.TWENTY_API_BASE_URL
|
||||
? `${process.env.TWENTY_API_BASE_URL.replace(/\/+$/, '')}/metadata`
|
||||
: null) ??
|
||||
'http://localhost:3000/rest/metadata'
|
||||
).replace(/\/+$/, '');
|
||||
|
||||
const API_KEY = process.env.TWENTY_API_KEY;
|
||||
|
||||
if (!API_KEY) {
|
||||
console.error('TWENTY_API_KEY is required in the environment to call the Metadata API.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const COMPANY_ROLLUP_FIELDS = [
|
||||
{ name: 'totalPipelineAmount', label: 'Total Pipeline Amount', type: 'CURRENCY' },
|
||||
{ name: 'totalOpportunityCount', label: 'Total Opportunity Count', type: 'NUMBER' },
|
||||
{ name: 'wonPipelineAmount', label: 'Won Pipeline Amount', type: 'CURRENCY' },
|
||||
{ name: 'wonOpportunityCount', label: 'Won Opportunity Count', type: 'NUMBER' },
|
||||
{ name: 'openPipelineAmount', label: 'Open Pipeline Amount', type: 'CURRENCY' },
|
||||
{ name: 'openOpportunityCount', label: 'Open Opportunity Count', type: 'NUMBER' },
|
||||
{ name: 'lastOpportunityCloseDate', label: 'Last Opportunity Close Date', type: 'DATE' },
|
||||
];
|
||||
|
||||
const metadataRequest = async (method, endpoint, body) => {
|
||||
const url = `${METADATA_BASE_URL}${endpoint}`;
|
||||
const headers = {
|
||||
Authorization: `Bearer ${API_KEY}`,
|
||||
};
|
||||
const init = { method, headers };
|
||||
if (body !== undefined) {
|
||||
headers['Content-Type'] = 'application/json';
|
||||
init.body = JSON.stringify(body);
|
||||
}
|
||||
|
||||
const response = await fetch(url, init);
|
||||
const text = await response.text();
|
||||
let parsed = {};
|
||||
if (text) {
|
||||
try {
|
||||
parsed = JSON.parse(text);
|
||||
} catch (error) {
|
||||
throw new Error(`Failed to parse JSON response from ${url}: ${text}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const messages = parsed?.messages || parsed?.message;
|
||||
const error = new Error(
|
||||
`Metadata API ${method} ${endpoint} failed: ${messages || response.statusText}`,
|
||||
);
|
||||
error.response = parsed;
|
||||
throw error;
|
||||
}
|
||||
|
||||
return parsed;
|
||||
};
|
||||
|
||||
const findObjectId = async (nameSingular) => {
|
||||
const response = await metadataRequest('GET', '/objects');
|
||||
const objects =
|
||||
(response?.data && Array.isArray(response.data.objects) ? response.data.objects : []) || [];
|
||||
const match = objects.find((entry) => entry.nameSingular === nameSingular);
|
||||
if (!match) {
|
||||
throw new Error(`Unable to find object with nameSingular="${nameSingular}" via Metadata API`);
|
||||
}
|
||||
return match.id;
|
||||
};
|
||||
|
||||
const createField = async (objectMetadataId, field) => {
|
||||
try {
|
||||
const payload = {
|
||||
objectMetadataId,
|
||||
name: field.name,
|
||||
label: field.label,
|
||||
type: field.type,
|
||||
};
|
||||
await metadataRequest('POST', '/fields', payload);
|
||||
console.log(`✔ Created ${field.name}`);
|
||||
} catch (error) {
|
||||
const messages = error.response?.messages;
|
||||
const messageList = Array.isArray(messages)
|
||||
? messages
|
||||
: typeof messages === 'string'
|
||||
? [messages]
|
||||
: [];
|
||||
if (messageList.some((msg) => msg.includes('already exists'))) {
|
||||
console.log(`→ ${field.name} already exists; skipping`);
|
||||
return;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
};
|
||||
|
||||
const main = async () => {
|
||||
console.log(`Using metadata base URL: ${METADATA_BASE_URL}`);
|
||||
const companyObjectId = await findObjectId('company');
|
||||
|
||||
for (const field of COMPANY_ROLLUP_FIELDS) {
|
||||
// Metadata API prefers camelCase names; script assumes labels may contain spaces.
|
||||
await createField(companyObjectId, field);
|
||||
}
|
||||
|
||||
console.log('Company rollup fields ensured.');
|
||||
};
|
||||
|
||||
main().catch((error) => {
|
||||
console.error('Failed to provision rollup fields via Metadata API.');
|
||||
console.error(error instanceof Error ? error.message : error);
|
||||
if (error instanceof Error && error.stack) {
|
||||
console.error(error.stack);
|
||||
}
|
||||
process.exit(1);
|
||||
});
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"$schema": "https://raw.githubusercontent.com/twentyhq/twenty/main/packages/twenty-cli/schemas/serverlessFunction.schema.json",
|
||||
"universalIdentifier": "c3ec36c8-5b1d-421f-9172-a9e035ab9c18",
|
||||
"name": "calculaterollups",
|
||||
"triggers": [
|
||||
{
|
||||
"universalIdentifier": "eec8aaf2-b0cc-47fd-b522-8d4aa5fe4bd3",
|
||||
"type": "databaseEvent",
|
||||
"eventName": "opportunity.*"
|
||||
},
|
||||
{
|
||||
"universalIdentifier": "a3fea230-1121-44a6-b395-5811c3031f8e",
|
||||
"type": "cron",
|
||||
"schedule": " 0 2 * * *"
|
||||
},
|
||||
{
|
||||
"universalIdentifier": "d33b0fe4-4b2b-45c0-aa2f-e617fdbba484",
|
||||
"type": "route",
|
||||
"path": "/recalculate-all",
|
||||
"httpMethod": "POST",
|
||||
"isAuthRequired": true
|
||||
}
|
||||
]
|
||||
}
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
import { applyFilters, getNestedValue, toComparableNumber } from './filtering';
|
||||
import type { ChildRecord, RollupDefinition } from './types';
|
||||
|
||||
const roundForSum = (value: number) =>
|
||||
Number.isInteger(value) ? value : Math.round(value * 100) / 100;
|
||||
|
||||
export const computeAggregations = (
|
||||
definition: RollupDefinition,
|
||||
records: ChildRecord[],
|
||||
now: Date,
|
||||
) => {
|
||||
const baseFiltered = applyFilters(records, definition.childFilters, now);
|
||||
const result: Record<string, number | string | null | Record<string, unknown>> = {};
|
||||
|
||||
definition.aggregations.forEach((aggregation) => {
|
||||
const scopedRecords = applyFilters(baseFiltered, aggregation.filters, now);
|
||||
|
||||
switch (aggregation.type) {
|
||||
case 'COUNT':
|
||||
result[aggregation.parentField] = scopedRecords.length;
|
||||
break;
|
||||
case 'SUM': {
|
||||
if (!aggregation.childField) {
|
||||
throw new Error('SUM aggregation requires childField');
|
||||
}
|
||||
const total = scopedRecords.reduce(
|
||||
(accumulator, record) => {
|
||||
const rawValue = getNestedValue(record, aggregation.childField!);
|
||||
const currencyRaw =
|
||||
aggregation.currencyField !== undefined
|
||||
? getNestedValue(record, aggregation.currencyField)
|
||||
: undefined;
|
||||
const numeric =
|
||||
typeof rawValue === 'number'
|
||||
? rawValue
|
||||
: typeof rawValue === 'string'
|
||||
? Number(rawValue)
|
||||
: NaN;
|
||||
if (Number.isNaN(numeric)) {
|
||||
return accumulator;
|
||||
}
|
||||
return {
|
||||
amount: accumulator.amount + numeric,
|
||||
currency:
|
||||
typeof currencyRaw === 'string' && currencyRaw.trim().length > 0
|
||||
? currencyRaw
|
||||
: accumulator.currency,
|
||||
};
|
||||
},
|
||||
{ amount: 0, currency: undefined as string | undefined },
|
||||
);
|
||||
result[aggregation.parentField] = {
|
||||
amountMicros: Math.round(roundForSum(total.amount)),
|
||||
currencyCode: total.currency ?? '',
|
||||
};
|
||||
break;
|
||||
}
|
||||
case 'AVG': {
|
||||
if (!aggregation.childField) {
|
||||
throw new Error('AVG aggregation requires childField');
|
||||
}
|
||||
const { total, count } = scopedRecords.reduce(
|
||||
(accumulator, record) => {
|
||||
const rawValue = getNestedValue(record, aggregation.childField!);
|
||||
const numeric =
|
||||
typeof rawValue === 'number'
|
||||
? rawValue
|
||||
: typeof rawValue === 'string'
|
||||
? Number(rawValue)
|
||||
: NaN;
|
||||
if (Number.isNaN(numeric)) {
|
||||
return accumulator;
|
||||
}
|
||||
return {
|
||||
total: accumulator.total + numeric,
|
||||
count: accumulator.count + 1,
|
||||
};
|
||||
},
|
||||
{ total: 0, count: 0 },
|
||||
);
|
||||
result[aggregation.parentField] = count === 0 ? null : roundForSum(total / count);
|
||||
break;
|
||||
}
|
||||
case 'MAX':
|
||||
case 'MIN': {
|
||||
if (!aggregation.childField) {
|
||||
throw new Error(`${aggregation.type} aggregation requires childField`);
|
||||
}
|
||||
const direction = aggregation.type === 'MAX' ? 1 : -1;
|
||||
let chosen: { raw: unknown; comparable: number | null } | null = null;
|
||||
scopedRecords.forEach((record) => {
|
||||
const rawValue = getNestedValue(record, aggregation.childField!);
|
||||
const comparable = toComparableNumber(rawValue);
|
||||
if (comparable === null) {
|
||||
return;
|
||||
}
|
||||
if (
|
||||
chosen === null ||
|
||||
(chosen.comparable !== null && direction * (comparable - chosen.comparable) > 0)
|
||||
) {
|
||||
chosen = { raw: rawValue, comparable };
|
||||
}
|
||||
});
|
||||
|
||||
if (chosen === null) {
|
||||
result[aggregation.parentField] = null;
|
||||
} else if (typeof chosen.raw === 'number') {
|
||||
result[aggregation.parentField] = chosen.raw;
|
||||
} else if (chosen.raw instanceof Date) {
|
||||
result[aggregation.parentField] = chosen.raw.toISOString().slice(0, 10);
|
||||
} else if (chosen.raw === null || chosen.raw === undefined) {
|
||||
result[aggregation.parentField] = null;
|
||||
} else {
|
||||
const rawString = String(chosen.raw);
|
||||
const parsed = Date.parse(rawString);
|
||||
result[aggregation.parentField] = Number.isNaN(parsed)
|
||||
? rawString
|
||||
: new Date(parsed).toISOString().slice(0, 10);
|
||||
}
|
||||
break;
|
||||
}
|
||||
default:
|
||||
break;
|
||||
}
|
||||
});
|
||||
|
||||
return result;
|
||||
};
|
||||
+347
@@ -0,0 +1,347 @@
|
||||
import { getNestedValue } from './filtering';
|
||||
import type { ChildRecord, RollupDefinition } from './types';
|
||||
|
||||
const RESOURCE_PLURALS: Record<string, string> = {
|
||||
person: 'people',
|
||||
gift: 'gifts',
|
||||
company: 'companies',
|
||||
opportunity: 'opportunities',
|
||||
};
|
||||
|
||||
const RETRIABLE_STATUS = new Set([429, 500, 502, 503, 504]);
|
||||
const DEFAULT_PAGE_SIZE = 200;
|
||||
const MAX_RETRIES = 3;
|
||||
const RETRY_BASE_DELAY_MS = 300;
|
||||
|
||||
type QueryParamPrimitive = string | number | boolean;
|
||||
type QueryParamValue = QueryParamPrimitive | QueryParamPrimitive[];
|
||||
|
||||
const serializeFilterExpressions = (
|
||||
filters: Record<string, QueryParamValue>,
|
||||
): string[] => {
|
||||
const expressions: string[] = [];
|
||||
|
||||
Object.entries(filters).forEach(([field, rawValue]) => {
|
||||
if (rawValue === undefined || rawValue === null) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (Array.isArray(rawValue)) {
|
||||
if (rawValue.length === 0) {
|
||||
return;
|
||||
}
|
||||
const serializedValues = rawValue.map((entry) => JSON.stringify(entry)).join(',');
|
||||
expressions.push(`${field}[in]:${serializedValues}`);
|
||||
return;
|
||||
}
|
||||
|
||||
expressions.push(`${field}[eq]:${JSON.stringify(rawValue)}`);
|
||||
});
|
||||
|
||||
return expressions;
|
||||
};
|
||||
|
||||
interface RequestOptions {
|
||||
method?: 'GET' | 'POST' | 'PATCH' | 'PUT' | 'DELETE';
|
||||
query?: Record<string, QueryParamValue>;
|
||||
body?: unknown;
|
||||
headers?: Record<string, string>;
|
||||
}
|
||||
|
||||
interface ListPageOptions {
|
||||
filter?: Record<string, QueryParamValue>;
|
||||
orderBy?: Record<string, 'asc' | 'desc'>;
|
||||
cursor?: string;
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
interface ListPage<T> {
|
||||
items: T[];
|
||||
hasNextPage: boolean;
|
||||
nextCursor?: string;
|
||||
}
|
||||
|
||||
const sleep = (ms: number) =>
|
||||
new Promise((resolve) => {
|
||||
setTimeout(resolve, ms);
|
||||
});
|
||||
|
||||
const getResourceName = (objectName: string) => RESOURCE_PLURALS[objectName] ?? `${objectName}s`;
|
||||
|
||||
export class TwentyClient {
|
||||
private readonly baseUrl: string;
|
||||
|
||||
constructor(
|
||||
private readonly apiKey: string,
|
||||
baseUrl: string,
|
||||
) {
|
||||
this.baseUrl = baseUrl.replace(/\/+$/, '');
|
||||
}
|
||||
|
||||
private async fetchWithRetry(url: string, init: RequestInit, attempt = 1): Promise<Response> {
|
||||
const response = await fetch(url, init);
|
||||
if (response.ok) {
|
||||
return response;
|
||||
}
|
||||
|
||||
if (RETRIABLE_STATUS.has(response.status) && attempt < MAX_RETRIES) {
|
||||
const delayMs = RETRY_BASE_DELAY_MS * attempt;
|
||||
await sleep(delayMs);
|
||||
return this.fetchWithRetry(url, init, attempt + 1);
|
||||
}
|
||||
|
||||
const errorBody = await response.text();
|
||||
throw new Error(`Request failed (${response.status}): ${errorBody || 'no body returned'}`);
|
||||
}
|
||||
|
||||
private appendQueryParams(url: URL, params: Record<string, QueryParamValue>) {
|
||||
Object.entries(params).forEach(([key, value]) => {
|
||||
if (value === undefined || value === null) {
|
||||
return;
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
value.forEach((entry) => {
|
||||
url.searchParams.append(key, String(entry));
|
||||
});
|
||||
return;
|
||||
}
|
||||
url.searchParams.set(key, String(value));
|
||||
});
|
||||
}
|
||||
|
||||
private async request(resourcePath: string, options: RequestOptions = {}) {
|
||||
const { method = 'GET', query, body, headers = {} } = options;
|
||||
|
||||
const url = resourcePath.startsWith('http')
|
||||
? new URL(resourcePath)
|
||||
: new URL(`${this.baseUrl}/${resourcePath.replace(/^\/+/, '')}`);
|
||||
|
||||
if (query) {
|
||||
this.appendQueryParams(url, query);
|
||||
}
|
||||
|
||||
let serializedBody: string | undefined;
|
||||
|
||||
if (body !== undefined) {
|
||||
serializedBody = typeof body === 'string' ? body : JSON.stringify(body);
|
||||
}
|
||||
|
||||
const mergedHeaders: Record<string, string> = {
|
||||
Authorization: `Bearer ${this.apiKey}`,
|
||||
...headers,
|
||||
};
|
||||
|
||||
if (serializedBody && !mergedHeaders['Content-Type']) {
|
||||
mergedHeaders['Content-Type'] = 'application/json';
|
||||
}
|
||||
|
||||
const response = await this.fetchWithRetry(url.toString(), {
|
||||
method,
|
||||
headers: mergedHeaders,
|
||||
body: serializedBody,
|
||||
});
|
||||
|
||||
if (response.status === 204) {
|
||||
return {};
|
||||
}
|
||||
|
||||
const text = await response.text();
|
||||
if (!text) {
|
||||
return {};
|
||||
}
|
||||
|
||||
try {
|
||||
return JSON.parse(text) as unknown;
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`Failed to parse JSON response from ${url.toString()}: ${(error as Error).message}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
private extractRecords(body: unknown, resource: string): ChildRecord[] {
|
||||
if (typeof body !== 'object' || body === null) {
|
||||
return [];
|
||||
}
|
||||
const data = typeof (body as Record<string, unknown>).data === 'object'
|
||||
? ((body as Record<string, unknown>).data as Record<string, unknown>)
|
||||
: undefined;
|
||||
|
||||
const direct = data && Array.isArray(data[resource]) ? data[resource] : undefined;
|
||||
if (Array.isArray(direct)) {
|
||||
return direct as ChildRecord[];
|
||||
}
|
||||
|
||||
const singular = resource.endsWith('s') ? resource.slice(0, -1) : resource;
|
||||
const singularMatch = data && Array.isArray(data[singular]) ? data[singular] : undefined;
|
||||
if (Array.isArray(singularMatch)) {
|
||||
return singularMatch as ChildRecord[];
|
||||
}
|
||||
|
||||
const capitalized = resource.charAt(0).toUpperCase() + resource.slice(1);
|
||||
const findManyKey = `findMany${capitalized}`;
|
||||
const findMany = data && Array.isArray(data[findManyKey]) ? data[findManyKey] : undefined;
|
||||
if (Array.isArray(findMany)) {
|
||||
return findMany as ChildRecord[];
|
||||
}
|
||||
|
||||
return [];
|
||||
}
|
||||
|
||||
private extractPageInfo(body: unknown) {
|
||||
if (typeof body !== 'object' || body === null) {
|
||||
return undefined;
|
||||
}
|
||||
if (typeof (body as Record<string, unknown>).pageInfo === 'object') {
|
||||
return (body as Record<string, unknown>).pageInfo as Record<string, unknown>;
|
||||
}
|
||||
if (
|
||||
typeof (body as Record<string, unknown>).data === 'object' &&
|
||||
typeof ((body as Record<string, unknown>).data as Record<string, unknown>).pageInfo === 'object'
|
||||
) {
|
||||
return ((body as Record<string, unknown>).data as Record<string, unknown>)
|
||||
.pageInfo as Record<string, unknown>;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
private async listRecordsPage(
|
||||
resource: string,
|
||||
options: ListPageOptions = {},
|
||||
): Promise<ListPage<ChildRecord>> {
|
||||
const query: Record<string, QueryParamValue> = {};
|
||||
const limit = options.limit ?? DEFAULT_PAGE_SIZE;
|
||||
query.limit = String(limit);
|
||||
|
||||
if (options.cursor) {
|
||||
query.starting_after = options.cursor;
|
||||
}
|
||||
|
||||
if (options.filter) {
|
||||
const filterExpressions = serializeFilterExpressions(options.filter);
|
||||
if (filterExpressions.length > 0) {
|
||||
query.filter = filterExpressions;
|
||||
}
|
||||
}
|
||||
|
||||
if (options.orderBy) {
|
||||
Object.entries(options.orderBy).forEach(([field, direction]) => {
|
||||
query[`order_by[${field}]`] = direction;
|
||||
});
|
||||
}
|
||||
|
||||
const response = await this.request(resource, {
|
||||
method: 'GET',
|
||||
query,
|
||||
});
|
||||
|
||||
const items = this.extractRecords(response, resource);
|
||||
const pageInfo = this.extractPageInfo(response);
|
||||
const hasNextPage =
|
||||
!!pageInfo &&
|
||||
(Boolean(pageInfo.hasNextPage) ||
|
||||
Boolean(pageInfo.endCursor) ||
|
||||
Boolean(pageInfo.nextCursor));
|
||||
const nextCursor =
|
||||
(pageInfo && typeof pageInfo.endCursor === 'string' && pageInfo.endCursor) ||
|
||||
(pageInfo && typeof pageInfo.nextCursor === 'string' && pageInfo.nextCursor) ||
|
||||
undefined;
|
||||
|
||||
return {
|
||||
items,
|
||||
hasNextPage,
|
||||
nextCursor,
|
||||
};
|
||||
}
|
||||
|
||||
async listAllRecords(objectName: string, options: ListPageOptions = {}): Promise<ChildRecord[]> {
|
||||
const resource = getResourceName(objectName);
|
||||
const aggregated: ChildRecord[] = [];
|
||||
let cursor: string | undefined;
|
||||
let hasNext = true;
|
||||
|
||||
while (hasNext) {
|
||||
const page = await this.listRecordsPage(resource, {
|
||||
...options,
|
||||
cursor,
|
||||
});
|
||||
aggregated.push(...page.items);
|
||||
hasNext = page.hasNextPage && Boolean(page.nextCursor);
|
||||
cursor = hasNext ? page.nextCursor : undefined;
|
||||
}
|
||||
|
||||
return aggregated;
|
||||
}
|
||||
|
||||
async updateObject(objectName: string, id: string, payload: Record<string, unknown>) {
|
||||
const resource = getResourceName(objectName);
|
||||
await this.request(`${resource}/${id}`, {
|
||||
method: 'PATCH',
|
||||
body: payload,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
export const ensureMapHasTargets = (
|
||||
container: Map<string, ChildRecord[]>,
|
||||
targetIds: Set<string> | undefined,
|
||||
) => {
|
||||
if (!targetIds) {
|
||||
return;
|
||||
}
|
||||
targetIds.forEach((id) => {
|
||||
if (!container.has(id)) {
|
||||
container.set(id, []);
|
||||
}
|
||||
});
|
||||
};
|
||||
|
||||
export const buildChildRecordIndex = async (
|
||||
definition: RollupDefinition,
|
||||
client: TwentyClient,
|
||||
parentIds: Set<string> | undefined,
|
||||
): Promise<Map<string, ChildRecord[]>> => {
|
||||
if (parentIds && parentIds.size > 0) {
|
||||
const parentIdList = Array.from(parentIds);
|
||||
const result = new Map<string, ChildRecord[]>();
|
||||
const concurrency = 5;
|
||||
|
||||
for (let index = 0; index < parentIdList.length; index += concurrency) {
|
||||
const slice = parentIdList.slice(index, index + concurrency);
|
||||
const batch = await Promise.all(
|
||||
slice.map(async (parentId) => {
|
||||
const records = await client.listAllRecords(definition.childObject, {
|
||||
filter: {
|
||||
[definition.relationField]: parentId,
|
||||
},
|
||||
});
|
||||
const filtered = records.filter((record) => {
|
||||
const relationValue = getNestedValue(record, definition.relationField);
|
||||
return typeof relationValue === 'string' && relationValue === parentId;
|
||||
});
|
||||
return { parentId, records: filtered };
|
||||
}),
|
||||
);
|
||||
batch.forEach(({ parentId, records }) => {
|
||||
result.set(parentId, records);
|
||||
});
|
||||
}
|
||||
|
||||
ensureMapHasTargets(result, parentIds);
|
||||
return result;
|
||||
}
|
||||
|
||||
const allRecords = await client.listAllRecords(definition.childObject);
|
||||
const grouped = new Map<string, ChildRecord[]>();
|
||||
allRecords.forEach((record) => {
|
||||
const relationValue = getNestedValue(record, definition.relationField);
|
||||
if (typeof relationValue !== 'string' || relationValue.trim().length === 0) {
|
||||
return;
|
||||
}
|
||||
if (!grouped.has(relationValue)) {
|
||||
grouped.set(relationValue, []);
|
||||
}
|
||||
grouped.get(relationValue)!.push(record);
|
||||
});
|
||||
return grouped;
|
||||
};
|
||||
+174
@@ -0,0 +1,174 @@
|
||||
import { defaultRollupConfig } from './rollupConfig';
|
||||
import type {
|
||||
AggregationConfig,
|
||||
FilterConfig,
|
||||
RollupConfig,
|
||||
RollupDefinition,
|
||||
} from './types';
|
||||
|
||||
const operatorSet = new Set<FilterConfig['operator']>([
|
||||
'equals',
|
||||
'notEquals',
|
||||
'in',
|
||||
'notIn',
|
||||
'gt',
|
||||
'gte',
|
||||
'lt',
|
||||
'lte',
|
||||
]);
|
||||
|
||||
const aggregationTypes = new Set<AggregationConfig['type']>([
|
||||
'SUM',
|
||||
'COUNT',
|
||||
'MAX',
|
||||
'MIN',
|
||||
'AVG',
|
||||
]);
|
||||
|
||||
const isObject = (value: unknown): value is Record<string, unknown> =>
|
||||
typeof value === 'object' && value !== null;
|
||||
|
||||
export const validateRollupConfig = (config: unknown): asserts config is RollupConfig => {
|
||||
if (!Array.isArray(config)) {
|
||||
throw new Error('Rollup configuration must contain an array of rollup definitions');
|
||||
}
|
||||
|
||||
config.forEach((definition, definitionIndex) => {
|
||||
if (!isObject(definition)) {
|
||||
throw new Error(`Definition at index ${definitionIndex} must be an object`);
|
||||
}
|
||||
|
||||
const { parentObject, childObject, relationField, childFilters, aggregations } =
|
||||
definition as Partial<RollupDefinition>;
|
||||
|
||||
if (typeof parentObject !== 'string' || parentObject.trim().length === 0) {
|
||||
throw new Error(`Definition ${definitionIndex} missing parentObject`);
|
||||
}
|
||||
if (typeof childObject !== 'string' || childObject.trim().length === 0) {
|
||||
throw new Error(`Definition ${definitionIndex} missing childObject`);
|
||||
}
|
||||
if (typeof relationField !== 'string' || relationField.trim().length === 0) {
|
||||
throw new Error(`Definition ${definitionIndex} missing relationField`);
|
||||
}
|
||||
if (!Array.isArray(aggregations) || aggregations.length === 0) {
|
||||
throw new Error(`Definition ${definitionIndex} must declare at least one aggregation`);
|
||||
}
|
||||
|
||||
const filtersToValidate = [
|
||||
...(Array.isArray(childFilters) ? childFilters : []),
|
||||
...aggregations.flatMap((aggregation, aggregationIndex) => {
|
||||
if (!isObject(aggregation)) {
|
||||
throw new Error(
|
||||
`Aggregation ${aggregationIndex} in definition ${definitionIndex} must be an object`,
|
||||
);
|
||||
}
|
||||
|
||||
const { type, parentField, childField, filters } = aggregation as AggregationConfig;
|
||||
|
||||
if (!aggregationTypes.has(type)) {
|
||||
throw new Error(
|
||||
`Aggregation ${aggregationIndex} in definition ${definitionIndex} has unsupported type ${type}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (typeof parentField !== 'string' || parentField.trim().length === 0) {
|
||||
throw new Error(
|
||||
`Aggregation ${aggregationIndex} in definition ${definitionIndex} missing parentField`,
|
||||
);
|
||||
}
|
||||
|
||||
if (type !== 'COUNT' && (typeof childField !== 'string' || childField.length === 0)) {
|
||||
throw new Error(
|
||||
`Aggregation ${aggregationIndex} in definition ${definitionIndex} with type ${type} requires childField`,
|
||||
);
|
||||
}
|
||||
|
||||
if (filters && !Array.isArray(filters)) {
|
||||
throw new Error(
|
||||
`Aggregation ${aggregationIndex} in definition ${definitionIndex} has invalid filters shape`,
|
||||
);
|
||||
}
|
||||
|
||||
return filters ?? [];
|
||||
}),
|
||||
];
|
||||
|
||||
filtersToValidate.forEach((filter, filterIndex) => {
|
||||
if (!isObject(filter)) {
|
||||
throw new Error(
|
||||
`Filter ${filterIndex} in definition ${definitionIndex} must be an object`,
|
||||
);
|
||||
}
|
||||
|
||||
const { field, operator } = filter as FilterConfig;
|
||||
|
||||
if (typeof field !== 'string' || field.trim().length === 0) {
|
||||
throw new Error(
|
||||
`Filter ${filterIndex} in definition ${definitionIndex} missing field`,
|
||||
);
|
||||
}
|
||||
|
||||
if (!operatorSet.has(operator)) {
|
||||
throw new Error(
|
||||
`Filter ${filterIndex} in definition ${definitionIndex} has unsupported operator ${operator}`,
|
||||
);
|
||||
}
|
||||
});
|
||||
});
|
||||
};
|
||||
|
||||
const collectValuesByKey = (
|
||||
value: unknown,
|
||||
key: string,
|
||||
result: Set<string>,
|
||||
): void => {
|
||||
if (Array.isArray(value)) {
|
||||
value.forEach((entry) => collectValuesByKey(entry, key, result));
|
||||
return;
|
||||
}
|
||||
|
||||
if (!isObject(value)) {
|
||||
return;
|
||||
}
|
||||
|
||||
Object.entries(value).forEach(([currentKey, entryValue]) => {
|
||||
if (currentKey === key && typeof entryValue === 'string' && entryValue.trim().length > 0) {
|
||||
result.add(entryValue);
|
||||
return;
|
||||
}
|
||||
|
||||
collectValuesByKey(entryValue, key, result);
|
||||
});
|
||||
};
|
||||
|
||||
export const extractRelationValues = (
|
||||
params: unknown,
|
||||
relationField: string,
|
||||
): Set<string> => {
|
||||
const values = new Set<string>();
|
||||
collectValuesByKey(params, relationField, values);
|
||||
return values;
|
||||
};
|
||||
|
||||
export const resolveRollupConfig = (): RollupConfig => {
|
||||
const override =
|
||||
process.env.ROLLUP_ENGINE_CONFIG ??
|
||||
process.env.ROLLUPS_CONFIG ??
|
||||
process.env.CALCULATE_ROLLUPS_CONFIG;
|
||||
|
||||
if (override) {
|
||||
try {
|
||||
const parsed = JSON.parse(override) as unknown;
|
||||
validateRollupConfig(parsed);
|
||||
return parsed;
|
||||
} catch (error) {
|
||||
const reason =
|
||||
error instanceof Error ? error.message : typeof error === 'string' ? error : 'Unknown error';
|
||||
throw new Error(`Unable to parse rollup configuration override (reason: ${reason})`);
|
||||
}
|
||||
}
|
||||
|
||||
const config = defaultRollupConfig;
|
||||
validateRollupConfig(config);
|
||||
return config;
|
||||
};
|
||||
+149
@@ -0,0 +1,149 @@
|
||||
import type { ChildRecord, DynamicValue, FilterConfig } from './types';
|
||||
|
||||
const isObject = (value: unknown): value is Record<string, unknown> =>
|
||||
typeof value === 'object' && value !== null;
|
||||
|
||||
const resolveDynamicValue = (dynamicValue: DynamicValue, now: Date) => {
|
||||
switch (dynamicValue) {
|
||||
case 'startOfYear': {
|
||||
const utcStart = new Date(Date.UTC(now.getUTCFullYear(), 0, 1, 0, 0, 0, 0));
|
||||
return utcStart.toISOString();
|
||||
}
|
||||
default:
|
||||
throw new Error(`Unsupported dynamicValue ${dynamicValue}`);
|
||||
}
|
||||
};
|
||||
|
||||
const normalizeForEquality = (value: unknown) => {
|
||||
if (value === null || value === undefined) {
|
||||
return null;
|
||||
}
|
||||
if (typeof value === 'number' || typeof value === 'boolean') {
|
||||
return value;
|
||||
}
|
||||
if (typeof value === 'string') {
|
||||
const numeric = Number(value);
|
||||
if (!Number.isNaN(numeric)) {
|
||||
return numeric;
|
||||
}
|
||||
return value;
|
||||
}
|
||||
if (value instanceof Date) {
|
||||
return value.toISOString();
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
export const toComparableNumber = (value: unknown): number | null => {
|
||||
if (typeof value === 'number') {
|
||||
return value;
|
||||
}
|
||||
if (typeof value === 'string') {
|
||||
const numeric = Number(value);
|
||||
if (!Number.isNaN(numeric)) {
|
||||
return numeric;
|
||||
}
|
||||
const timestamp = Date.parse(value);
|
||||
if (!Number.isNaN(timestamp)) {
|
||||
return timestamp;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
if (value instanceof Date) {
|
||||
return value.getTime();
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
const compareValues = (left: unknown, right: unknown): number | null => {
|
||||
const leftComparable = toComparableNumber(left);
|
||||
const rightComparable = toComparableNumber(right);
|
||||
if (leftComparable === null || rightComparable === null) {
|
||||
return null;
|
||||
}
|
||||
return leftComparable - rightComparable;
|
||||
};
|
||||
|
||||
const evaluateFilter = (rawValue: unknown, filter: FilterConfig, now: Date) => {
|
||||
const comparisonSource =
|
||||
filter.dynamicValue !== undefined
|
||||
? resolveDynamicValue(filter.dynamicValue, now)
|
||||
: filter.value;
|
||||
|
||||
switch (filter.operator) {
|
||||
case 'equals': {
|
||||
const left = normalizeForEquality(rawValue);
|
||||
const right = normalizeForEquality(comparisonSource);
|
||||
return left !== null && right !== null ? left === right : rawValue === comparisonSource;
|
||||
}
|
||||
case 'notEquals': {
|
||||
const left = normalizeForEquality(rawValue);
|
||||
const right = normalizeForEquality(comparisonSource);
|
||||
return left !== null && right !== null ? left !== right : rawValue !== comparisonSource;
|
||||
}
|
||||
case 'in': {
|
||||
if (!Array.isArray(comparisonSource)) {
|
||||
return false;
|
||||
}
|
||||
const left = normalizeForEquality(rawValue);
|
||||
if (left === null) {
|
||||
return false;
|
||||
}
|
||||
return comparisonSource.map(normalizeForEquality).some((candidate) => candidate === left);
|
||||
}
|
||||
case 'notIn': {
|
||||
if (!Array.isArray(comparisonSource)) {
|
||||
return true;
|
||||
}
|
||||
const left = normalizeForEquality(rawValue);
|
||||
if (left === null) {
|
||||
return true;
|
||||
}
|
||||
return !comparisonSource.map(normalizeForEquality).some((candidate) => candidate === left);
|
||||
}
|
||||
case 'gt': {
|
||||
const comparison = compareValues(rawValue, comparisonSource);
|
||||
return comparison !== null ? comparison > 0 : false;
|
||||
}
|
||||
case 'gte': {
|
||||
const comparison = compareValues(rawValue, comparisonSource);
|
||||
return comparison !== null ? comparison >= 0 : false;
|
||||
}
|
||||
case 'lt': {
|
||||
const comparison = compareValues(rawValue, comparisonSource);
|
||||
return comparison !== null ? comparison < 0 : false;
|
||||
}
|
||||
case 'lte': {
|
||||
const comparison = compareValues(rawValue, comparisonSource);
|
||||
return comparison !== null ? comparison <= 0 : false;
|
||||
}
|
||||
default:
|
||||
return true;
|
||||
}
|
||||
};
|
||||
|
||||
export const getNestedValue = (record: ChildRecord, pathExpression: string): unknown => {
|
||||
if (!pathExpression.includes('.')) {
|
||||
return record[pathExpression];
|
||||
}
|
||||
|
||||
return pathExpression.split('.').reduce<unknown>((accumulator, key) => {
|
||||
if (!isObject(accumulator)) {
|
||||
return undefined;
|
||||
}
|
||||
return accumulator[key];
|
||||
}, record);
|
||||
};
|
||||
|
||||
export const applyFilters = (
|
||||
records: ChildRecord[],
|
||||
filters: FilterConfig[] | undefined,
|
||||
now: Date,
|
||||
) => {
|
||||
if (!filters || filters.length === 0) {
|
||||
return records;
|
||||
}
|
||||
return records.filter((record) =>
|
||||
filters.every((filter) => evaluateFilter(getNestedValue(record, filter.field), filter, now)),
|
||||
);
|
||||
};
|
||||
@@ -0,0 +1,207 @@
|
||||
import { computeAggregations } from './aggregations';
|
||||
import { buildChildRecordIndex, TwentyClient } from './client';
|
||||
import { extractRelationValues, resolveRollupConfig } from './config';
|
||||
import { getNestedValue } from './filtering';
|
||||
import type { ExecutionSummaryItem, RollupDefinition } from './types';
|
||||
|
||||
const isObject = (value: unknown): value is Record<string, unknown> =>
|
||||
typeof value === 'object' && value !== null;
|
||||
|
||||
const sanitizePayload = (payload: Record<string, unknown>) =>
|
||||
Object.fromEntries(
|
||||
Object.entries(payload).filter(([, value]) => {
|
||||
if (value === undefined) {
|
||||
return false;
|
||||
}
|
||||
if (typeof value === 'number' && !Number.isFinite(value)) {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}),
|
||||
);
|
||||
|
||||
const determineFullRebuild = (params: unknown) => {
|
||||
if (!isObject(params)) {
|
||||
return false;
|
||||
}
|
||||
if (params.recalculateAll === true || params.fullRebuild === true) {
|
||||
return true;
|
||||
}
|
||||
if (isObject(params.trigger) && params.trigger.type === 'cron') {
|
||||
return true;
|
||||
}
|
||||
if (params.trigger === 'cron') {
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
};
|
||||
|
||||
const getApiCredentials = () => {
|
||||
const apiKey =
|
||||
process.env.TWENTY_API_KEY ??
|
||||
process.env.TWENTY_API_TOKEN ??
|
||||
process.env.API_KEY;
|
||||
|
||||
if (!apiKey) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const baseUrlRaw =
|
||||
process.env.TWENTY_API_BASE_URL ??
|
||||
process.env.TWENTY_REST_BASE_URL ??
|
||||
process.env.TWENTY_API_URL ??
|
||||
'';
|
||||
|
||||
const baseUrl =
|
||||
typeof baseUrlRaw === 'string' && baseUrlRaw.trim().length > 0
|
||||
? baseUrlRaw
|
||||
: 'https://app.twenty.com/rest';
|
||||
|
||||
return { apiKey, baseUrl };
|
||||
};
|
||||
|
||||
const formatSummary = (summaries: ExecutionSummaryItem[], durationMs: number) => ({
|
||||
status: 'ok',
|
||||
tookMs: durationMs,
|
||||
totals: {
|
||||
processed: summaries.reduce((accumulator, item) => accumulator + item.processed, 0),
|
||||
updated: summaries.reduce((accumulator, item) => accumulator + item.updated, 0),
|
||||
},
|
||||
details: summaries,
|
||||
});
|
||||
|
||||
export async function main(params: unknown): Promise<object> {
|
||||
const start = Date.now();
|
||||
try {
|
||||
const config = resolveRollupConfig();
|
||||
if (config.length === 0) {
|
||||
return { status: 'noop', reason: 'No rollup definitions configured' };
|
||||
}
|
||||
|
||||
const fullRebuild = determineFullRebuild(params);
|
||||
const relationCache = new Map<string, Set<string>>();
|
||||
|
||||
config.forEach((definition) => {
|
||||
if (!relationCache.has(definition.relationField)) {
|
||||
relationCache.set(
|
||||
definition.relationField,
|
||||
extractRelationValues(params, definition.relationField),
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
const credentials = getApiCredentials();
|
||||
if (!credentials) {
|
||||
console.warn('[rollup] skipping execution because TWENTY_API_KEY is not set');
|
||||
return { status: 'noop', reason: 'TWENTY_API_KEY not configured' };
|
||||
}
|
||||
|
||||
const { apiKey, baseUrl } = credentials;
|
||||
const client = new TwentyClient(apiKey, baseUrl);
|
||||
const now = new Date();
|
||||
const summaries: ExecutionSummaryItem[] = [];
|
||||
|
||||
for (const definition of config) {
|
||||
const targetIds = fullRebuild ? undefined : relationCache.get(definition.relationField);
|
||||
|
||||
if (!fullRebuild && (!targetIds || targetIds.size === 0)) {
|
||||
summaries.push({
|
||||
parentObject: definition.parentObject,
|
||||
processed: 0,
|
||||
updated: 0,
|
||||
relationField: definition.relationField,
|
||||
mode: 'targeted',
|
||||
skipped: `No ${definition.relationField} values found in payload`,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
const childIndex = await buildChildRecordIndex(definition, client, targetIds);
|
||||
|
||||
const updates: Array<{
|
||||
id: string;
|
||||
payload: Record<string, unknown>;
|
||||
context: { relationId: string };
|
||||
}> = [];
|
||||
childIndex.forEach((records, parentId) => {
|
||||
const recordIds = records
|
||||
.map((record) => getNestedValue(record, 'id'))
|
||||
.filter((value): value is string => typeof value === 'string');
|
||||
console.info(
|
||||
`[rollup] relation ${parentId} includes ${recordIds.length} ${definition.childObject}(s): ${recordIds.join(', ')}`,
|
||||
);
|
||||
const aggregates = computeAggregations(definition, records, now);
|
||||
const payload = sanitizePayload(aggregates);
|
||||
if (Object.keys(payload).length === 0) {
|
||||
return;
|
||||
}
|
||||
console.info(
|
||||
`[rollup] computed aggregates for ${definition.parentObject} ${parentId}: ${JSON.stringify(payload)}`,
|
||||
);
|
||||
updates.push({ id: parentId, payload, context: { relationId: parentId } });
|
||||
});
|
||||
|
||||
let updatedCount = 0;
|
||||
for (const update of updates) {
|
||||
try {
|
||||
await client.updateObject(definition.parentObject, update.id, update.payload);
|
||||
updatedCount += 1;
|
||||
console.info(
|
||||
`[rollup] updated ${definition.parentObject} ${update.id} (relation ${update.context.relationId})`,
|
||||
);
|
||||
} catch (error) {
|
||||
const reason =
|
||||
error instanceof Error
|
||||
? error.message || error.toString()
|
||||
: typeof error === 'string'
|
||||
? error
|
||||
: 'Unknown error';
|
||||
console.warn(
|
||||
`[rollup] failed to update ${definition.parentObject} ${update.id} (relation ${update.context.relationId}): ${reason}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
summaries.push({
|
||||
parentObject: definition.parentObject,
|
||||
processed: childIndex.size,
|
||||
updated: updatedCount,
|
||||
relationField: definition.relationField,
|
||||
mode: fullRebuild ? 'full-rebuild' : 'targeted',
|
||||
});
|
||||
}
|
||||
|
||||
const totalProcessed = summaries.reduce(
|
||||
(accumulator, item) => accumulator + item.processed,
|
||||
0,
|
||||
);
|
||||
|
||||
if (!fullRebuild && totalProcessed === 0) {
|
||||
return {
|
||||
status: 'noop',
|
||||
reason: 'No matching relation ids found in payload',
|
||||
};
|
||||
}
|
||||
|
||||
const duration = Date.now() - start;
|
||||
console.info(
|
||||
`[rollup] completed mode=${fullRebuild ? 'full-rebuild' : 'targeted'} processed=${totalProcessed} durationMs=${duration}`,
|
||||
);
|
||||
return formatSummary(summaries, duration);
|
||||
} catch (error) {
|
||||
const serializedError =
|
||||
error instanceof Error
|
||||
? `${error.name}: ${error.message}${error.stack ? `\n${error.stack}` : ''}`
|
||||
: JSON.stringify(error);
|
||||
console.error('[rollup] execution failed', serializedError);
|
||||
return {
|
||||
status: 'error',
|
||||
message:
|
||||
error instanceof Error
|
||||
? error.message || 'Unknown error'
|
||||
: typeof error === 'string'
|
||||
? error
|
||||
: 'Unknown error',
|
||||
};
|
||||
}
|
||||
}
|
||||
+81
@@ -0,0 +1,81 @@
|
||||
import type { RollupConfig } from './types';
|
||||
|
||||
export const defaultRollupConfig: RollupConfig = [
|
||||
{
|
||||
parentObject: 'company',
|
||||
childObject: 'opportunity',
|
||||
relationField: 'companyId',
|
||||
childFilters: [
|
||||
{
|
||||
field: 'amount.amountMicros',
|
||||
operator: 'gt',
|
||||
value: 0,
|
||||
},
|
||||
],
|
||||
aggregations: [
|
||||
{
|
||||
type: 'SUM',
|
||||
childField: 'amount.amountMicros',
|
||||
parentField: 'totalPipelineAmount',
|
||||
currencyField: 'amount.currencyCode',
|
||||
},
|
||||
{
|
||||
type: 'COUNT',
|
||||
parentField: 'totalOpportunityCount',
|
||||
},
|
||||
{
|
||||
type: 'SUM',
|
||||
childField: 'amount.amountMicros',
|
||||
parentField: 'wonPipelineAmount',
|
||||
currencyField: 'amount.currencyCode',
|
||||
filters: [
|
||||
{
|
||||
field: 'stage',
|
||||
operator: 'equals',
|
||||
value: 'CUSTOMER',
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
type: 'COUNT',
|
||||
parentField: 'wonOpportunityCount',
|
||||
filters: [
|
||||
{
|
||||
field: 'stage',
|
||||
operator: 'equals',
|
||||
value: 'CUSTOMER',
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
type: 'SUM',
|
||||
childField: 'amount.amountMicros',
|
||||
parentField: 'openPipelineAmount',
|
||||
currencyField: 'amount.currencyCode',
|
||||
filters: [
|
||||
{
|
||||
field: 'stage',
|
||||
operator: 'notEquals',
|
||||
value: 'CUSTOMER',
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
type: 'COUNT',
|
||||
parentField: 'openOpportunityCount',
|
||||
filters: [
|
||||
{
|
||||
field: 'stage',
|
||||
operator: 'notEquals',
|
||||
value: 'CUSTOMER',
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
type: 'MAX',
|
||||
childField: 'closeDate',
|
||||
parentField: 'lastOpportunityCloseDate',
|
||||
},
|
||||
],
|
||||
},
|
||||
];
|
||||
@@ -0,0 +1,49 @@
|
||||
export type AggregationType = 'SUM' | 'COUNT' | 'MAX' | 'MIN' | 'AVG';
|
||||
|
||||
export type Operator =
|
||||
| 'equals'
|
||||
| 'notEquals'
|
||||
| 'in'
|
||||
| 'notIn'
|
||||
| 'gt'
|
||||
| 'gte'
|
||||
| 'lt'
|
||||
| 'lte';
|
||||
|
||||
export type DynamicValue = 'startOfYear';
|
||||
|
||||
export interface FilterConfig {
|
||||
field: string;
|
||||
operator: Operator;
|
||||
value?: string | number | boolean | Array<string | number | boolean>;
|
||||
dynamicValue?: DynamicValue;
|
||||
}
|
||||
|
||||
export interface AggregationConfig {
|
||||
type: AggregationType;
|
||||
parentField: string;
|
||||
childField?: string;
|
||||
currencyField?: string;
|
||||
filters?: FilterConfig[];
|
||||
}
|
||||
|
||||
export interface RollupDefinition {
|
||||
parentObject: string;
|
||||
childObject: string;
|
||||
relationField: string;
|
||||
childFilters?: FilterConfig[];
|
||||
aggregations: AggregationConfig[];
|
||||
}
|
||||
|
||||
export type RollupConfig = RollupDefinition[];
|
||||
|
||||
export type ChildRecord = Record<string, unknown>;
|
||||
|
||||
export interface ExecutionSummaryItem {
|
||||
parentObject: string;
|
||||
processed: number;
|
||||
updated: number;
|
||||
mode: 'full-rebuild' | 'targeted';
|
||||
relationField: string;
|
||||
skipped?: string;
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user