Make AI chat streaming render cost independent of message length (#23831)
Follow-up to #23573: chip-heavy answers are long by design, and each stream flush re-ran `protectChatReferencesForMarkdown` and `marked.lexer` over the whole message, so render cost grew quadratically with message length. This makes the per-flush cost proportional to the appended text instead, and offsets the new code by removing dead AI chat code. ## Streaming render - **Incremental block splitting.** `getMarkdownBlocksIncrementally` reuses blocks that can no longer change and re-tokenizes only the trailing ones. Two trailing blocks stay unstable, not one: a loose list followed by a blank line still merges with a later item (`- a\n\n` + `- b` is one list token). Uses `Lexer.blockTokens` instead of `marked.lexer` since only block raws are needed and the full lexer also runs the inline tokenizer. Simulated stream over a 22 KB chip-heavy message (120 chars/flush, matching the 100 ms flush throttle): 191 ms → 3.7 ms cumulative. The test suite pins char-by-char equivalence against full `marked.lexer` output across loose lists, unclosed fences, setext headings, tables, CRLF and chip markers. - **Per-block reference protection.** `protectChatReferencesForMarkdown` moved behind the existing block memo, so settled blocks never re-run reference parsing during a stream. - **Anchored open pattern.** `(?<!\[)\[\[+` anchors marker matching to the start of a bracket run. The greedy `+` from #23798 backtracked at every position inside a run, once per alternative: 429 ms → ~1 ms on a 10 KB bracket-run input. A run start always yields the same match, so no valid marker is lost. Also an `includes('[[')` bail-out in `findChatReferences`, which runs on every text node of the streaming block. ## Chip lookups `fieldMetadataItemByIdSelector` did `objectMetadataItems.find(obj => obj.fields.some(...))` per chip — O(workspace fields) each time the agent's tool calls trigger a metadata refetch mid-chat. The by-id and by-name map selectors mostly already existed with almost no consumers; this wires `fieldMetadataItemByIdSelector`, `objectMetadataItemFamilySelector` and `viewFromViewIdFamilySelector` to them (adding the missing `objectMetadataItemsByIdMapSelector` and `viewsByIdMapSelector`) and adds `areEqual` so unchanged lookups keep referential stability. ## Offscreen messages Settled messages (everything except the streaming last one) get `content-visibility: auto`, so long threads skip layout and paint for messages scrolled out of view. `contain-intrinsic-size: auto` keeps remembered heights, so scroll positions stay accurate once a message has been painted. ## Removed `ReasoningSummaryDisplay`, `agentChatMessagesComponentState`, `CHAT_THREADS_PAGE_SIZE`, `AgentResponseFormat` and `getFieldIcon` had no consumers. `TextWithChatReferences` and `protectChatReferencesForMarkdown` shared a duplicated segment-slicing loop, now in `getChatReferenceSegments`, and the nine identical per-tag markdown component entries collapse into `createChatReferenceElement`. The branch lands at +354/−329 including the new test suite; production code is net negative. Incidental: `marked` added to jest's `transformIgnorePatterns` allowlist (ESM-only, previously imported by no test). --- _Generated by [Claude Code](https://claude.ai/code/session_01MN8FVc63J4SJQHXWwzUwkh)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/23831?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
This commit is contained in:
Félix Malfait
@@ -27,8 +27,8 @@ const jestConfig = {
|
||||
testEnvironmentOptions: {},
|
||||
|
||||
transformIgnorePatterns: [
|
||||
'/node_modules/(?!(twenty-ui|apollo-upload-client|extract-files|is-plain-obj|@preact/signals-core)/.*)',
|
||||
'../../node_modules/(?!(twenty-ui|apollo-upload-client|extract-files|is-plain-obj|@preact/signals-core)/.*)',
|
||||
'/node_modules/(?!(twenty-ui|apollo-upload-client|extract-files|is-plain-obj|@preact/signals-core|marked)/.*)',
|
||||
'../../node_modules/(?!(twenty-ui|apollo-upload-client|extract-files|is-plain-obj|@preact/signals-core|marked)/.*)',
|
||||
'../../twenty-ui/',
|
||||
],
|
||||
transform: {
|
||||
|
||||
@@ -1,6 +1,18 @@
|
||||
import { AiChatMessage } from '@/ai/components/AiChatMessage';
|
||||
import { agentChatNonLastMessageIdsComponentSelector } from '@/ai/states/selectors/agentChatNonLastMessageIdsComponentSelector';
|
||||
import { useAtomComponentSelectorValue } from '@/ui/utilities/state/jotai/hooks/useAtomComponentSelectorValue';
|
||||
import { styled } from '@linaria/react';
|
||||
|
||||
// Settled messages never change, so let the browser skip layout and paint for
|
||||
// the ones scrolled out of view. `auto` in contain-intrinsic-size keeps the
|
||||
// last rendered height once a message has been painted; the estimate only
|
||||
// applies to never-painted messages, where scroll anchoring absorbs the
|
||||
// correction. 160px approximates a typical assistant message so the scrollbar
|
||||
// stays close to truth on long resumed threads.
|
||||
const StyledSettledMessage = styled.div`
|
||||
contain-intrinsic-size: auto 160px;
|
||||
content-visibility: auto;
|
||||
`;
|
||||
|
||||
export const AiChatNonLastMessageIdsList = () => {
|
||||
const agentChatNonLastMessageIds = useAtomComponentSelectorValue(
|
||||
@@ -8,6 +20,8 @@ export const AiChatNonLastMessageIdsList = () => {
|
||||
);
|
||||
|
||||
return agentChatNonLastMessageIds.map((messageId) => (
|
||||
<AiChatMessage key={messageId} messageId={messageId} />
|
||||
<StyledSettledMessage key={messageId}>
|
||||
<AiChatMessage messageId={messageId} />
|
||||
</StyledSettledMessage>
|
||||
));
|
||||
};
|
||||
|
||||
@@ -7,8 +7,9 @@ import {
|
||||
} from '@/ai/components/LazyMarkdownRendererStyledComponents';
|
||||
import { MarkdownCodeBlock } from '@/ai/components/MarkdownCodeBlock';
|
||||
import { TextWithChatReferences } from '@/ai/components/TextWithChatReferences';
|
||||
import { EMPTY_MARKDOWN_BLOCK_SPLIT_CACHE } from '@/ai/constants/EmptyMarkdownBlockSplitCache';
|
||||
import { getMarkdownBlocksIncrementally } from '@/ai/utils/getMarkdownBlocksIncrementally';
|
||||
import { protectChatReferencesForMarkdown } from '@/ai/utils/protectChatReferencesForMarkdown';
|
||||
import { marked } from 'marked';
|
||||
import {
|
||||
cloneElement,
|
||||
isValidElement,
|
||||
@@ -16,7 +17,7 @@ import {
|
||||
memo,
|
||||
Suspense,
|
||||
useContext,
|
||||
useMemo,
|
||||
useRef,
|
||||
} from 'react';
|
||||
import Skeleton, { SkeletonTheme } from 'react-loading-skeleton';
|
||||
import { getSafeUrl, isDefined } from 'twenty-shared/utils';
|
||||
@@ -48,6 +49,12 @@ const processChildrenForChatReferences = (
|
||||
return children;
|
||||
};
|
||||
|
||||
const createChatReferenceElement =
|
||||
(Element: React.ElementType) =>
|
||||
({ children }: { children?: React.ReactNode }) => (
|
||||
<Element>{processChildrenForChatReferences(children)}</Element>
|
||||
);
|
||||
|
||||
// react-markdown uses each entry as the JSX element type, so rebuilding this map
|
||||
// per render would remount every node on every streamed chunk.
|
||||
const MARKDOWN_COMPONENTS = {
|
||||
@@ -56,38 +63,16 @@ const MARKDOWN_COMPONENTS = {
|
||||
<table>{children}</table>
|
||||
</StyledTableScrollContainer>
|
||||
),
|
||||
p: ({ children }: { children?: React.ReactNode }) => (
|
||||
<StyledParagraph>
|
||||
{processChildrenForChatReferences(children)}
|
||||
</StyledParagraph>
|
||||
),
|
||||
td: ({ children }: { children?: React.ReactNode }) => (
|
||||
<td>{processChildrenForChatReferences(children)}</td>
|
||||
),
|
||||
th: ({ children }: { children?: React.ReactNode }) => (
|
||||
<th>{processChildrenForChatReferences(children)}</th>
|
||||
),
|
||||
li: ({ children }: { children?: React.ReactNode }) => (
|
||||
<li>{processChildrenForChatReferences(children)}</li>
|
||||
),
|
||||
h1: ({ children }: { children?: React.ReactNode }) => (
|
||||
<h1>{processChildrenForChatReferences(children)}</h1>
|
||||
),
|
||||
h2: ({ children }: { children?: React.ReactNode }) => (
|
||||
<h2>{processChildrenForChatReferences(children)}</h2>
|
||||
),
|
||||
h3: ({ children }: { children?: React.ReactNode }) => (
|
||||
<h3>{processChildrenForChatReferences(children)}</h3>
|
||||
),
|
||||
h4: ({ children }: { children?: React.ReactNode }) => (
|
||||
<h4>{processChildrenForChatReferences(children)}</h4>
|
||||
),
|
||||
h5: ({ children }: { children?: React.ReactNode }) => (
|
||||
<h5>{processChildrenForChatReferences(children)}</h5>
|
||||
),
|
||||
h6: ({ children }: { children?: React.ReactNode }) => (
|
||||
<h6>{processChildrenForChatReferences(children)}</h6>
|
||||
),
|
||||
p: createChatReferenceElement(StyledParagraph),
|
||||
td: createChatReferenceElement('td'),
|
||||
th: createChatReferenceElement('th'),
|
||||
li: createChatReferenceElement('li'),
|
||||
h1: createChatReferenceElement('h1'),
|
||||
h2: createChatReferenceElement('h2'),
|
||||
h3: createChatReferenceElement('h3'),
|
||||
h4: createChatReferenceElement('h4'),
|
||||
h5: createChatReferenceElement('h5'),
|
||||
h6: createChatReferenceElement('h6'),
|
||||
a: ({
|
||||
children,
|
||||
href,
|
||||
@@ -170,23 +155,29 @@ const LoadingSkeleton = () => {
|
||||
);
|
||||
};
|
||||
|
||||
// Protecting per block behind the memo means only the streaming tail blocks
|
||||
// pay the reference-parsing cost on each flush; settled blocks never re-run it.
|
||||
const MemoizedMarkdownBlock = memo(
|
||||
({ blockText }: { blockText: string }) => (
|
||||
<MarkdownRenderer>{blockText}</MarkdownRenderer>
|
||||
<MarkdownRenderer>
|
||||
{protectChatReferencesForMarkdown(blockText)}
|
||||
</MarkdownRenderer>
|
||||
),
|
||||
(previousProps, nextProps) => previousProps.blockText === nextProps.blockText,
|
||||
);
|
||||
|
||||
export const LazyMarkdownRenderer = ({ text }: { text: string }) => {
|
||||
const protectedText = useMemo(
|
||||
() => protectChatReferencesForMarkdown(text),
|
||||
[text],
|
||||
);
|
||||
// Not state: the blocks are a pure function of `text`, the ref only caches
|
||||
// the previous split so streaming appends skip re-tokenizing settled blocks.
|
||||
// oxlint-disable-next-line twenty/no-state-useref
|
||||
const blockSplitCacheRef = useRef(EMPTY_MARKDOWN_BLOCK_SPLIT_CACHE);
|
||||
|
||||
const markdownBlocks = useMemo(
|
||||
() => marked.lexer(protectedText).map((token) => token.raw),
|
||||
[protectedText],
|
||||
);
|
||||
const { blocks: markdownBlocks, cache } = getMarkdownBlocksIncrementally({
|
||||
text,
|
||||
cache: blockSplitCacheRef.current,
|
||||
});
|
||||
|
||||
blockSplitCacheRef.current = cache;
|
||||
|
||||
return (
|
||||
<StyledMarkdownContainer
|
||||
|
||||
@@ -1,117 +0,0 @@
|
||||
import { styled } from '@linaria/react';
|
||||
import { useContext, useState } from 'react';
|
||||
|
||||
import { IconBrain, IconChevronDown, IconChevronUp } from 'twenty-ui/icon';
|
||||
import { AnimatedExpandableContainer } from 'twenty-ui/layout';
|
||||
import { ThemeContext, themeCssVariables } from 'twenty-ui/theme-constants';
|
||||
|
||||
import { ShimmeringText } from '@/ai/components/ShimmeringText';
|
||||
import { t } from '@lingui/core/macro';
|
||||
|
||||
const StyledContainer = styled.div`
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: ${themeCssVariables.spacing[2]};
|
||||
margin-bottom: ${themeCssVariables.spacing[2]};
|
||||
`;
|
||||
|
||||
const StyledThinkingText = styled.div`
|
||||
color: ${themeCssVariables.font.color.tertiary};
|
||||
font-size: ${themeCssVariables.font.size.md};
|
||||
font-weight: ${themeCssVariables.font.weight.medium};
|
||||
`;
|
||||
|
||||
const StyledReasoningContainer = styled.div`
|
||||
background: ${themeCssVariables.background.transparent.lighter};
|
||||
border: 1px solid ${themeCssVariables.border.color.light};
|
||||
border-radius: ${themeCssVariables.border.radius.sm};
|
||||
padding: ${themeCssVariables.spacing[3]};
|
||||
`;
|
||||
|
||||
const StyledReasoningText = styled.div`
|
||||
color: ${themeCssVariables.font.color.secondary};
|
||||
font-size: ${themeCssVariables.font.size.sm};
|
||||
line-height: ${themeCssVariables.text.lineHeight.lg};
|
||||
white-space: pre-wrap;
|
||||
`;
|
||||
|
||||
const StyledToggleButton = styled.div`
|
||||
align-items: center;
|
||||
background: none;
|
||||
border: none;
|
||||
color: ${themeCssVariables.font.color.tertiary};
|
||||
cursor: pointer;
|
||||
display: flex;
|
||||
font-size: ${themeCssVariables.font.size.md};
|
||||
font-weight: ${themeCssVariables.font.weight.medium};
|
||||
gap: ${themeCssVariables.spacing[1]};
|
||||
padding: ${themeCssVariables.spacing[1]} 0;
|
||||
transition: color calc(${themeCssVariables.animation.duration.normal} * 1s);
|
||||
|
||||
&:hover {
|
||||
color: ${themeCssVariables.font.color.secondary};
|
||||
}
|
||||
`;
|
||||
|
||||
const StyledIconContainer = styled.div`
|
||||
align-items: center;
|
||||
display: flex;
|
||||
gap: ${themeCssVariables.spacing[1]};
|
||||
`;
|
||||
|
||||
export const ReasoningSummaryDisplay = ({
|
||||
content,
|
||||
isThinking = false,
|
||||
}: {
|
||||
content: string;
|
||||
isThinking?: boolean;
|
||||
}) => {
|
||||
const { theme } = useContext(ThemeContext);
|
||||
const [isExpanded, setIsExpanded] = useState(false);
|
||||
|
||||
const hasContent = content.trim().length > 0;
|
||||
|
||||
if (!hasContent) {
|
||||
return null;
|
||||
}
|
||||
|
||||
return (
|
||||
<StyledContainer>
|
||||
{isThinking && (
|
||||
<>
|
||||
<ShimmeringText>
|
||||
<StyledIconContainer>
|
||||
<IconBrain size={theme.icon.size.sm} />
|
||||
<StyledThinkingText>{t`Thinking...`}</StyledThinkingText>
|
||||
</StyledIconContainer>
|
||||
</ShimmeringText>
|
||||
<StyledReasoningContainer>
|
||||
<StyledReasoningText>{content}</StyledReasoningText>
|
||||
</StyledReasoningContainer>
|
||||
</>
|
||||
)}
|
||||
|
||||
{hasContent && !isThinking && (
|
||||
<>
|
||||
<StyledToggleButton onClick={() => setIsExpanded(!isExpanded)}>
|
||||
<StyledIconContainer>
|
||||
<IconBrain size={theme.icon.size.sm} />
|
||||
<span>{t`Finished thinking`}</span>
|
||||
</StyledIconContainer>
|
||||
{isExpanded ? (
|
||||
<IconChevronUp size={theme.icon.size.sm} />
|
||||
) : (
|
||||
<IconChevronDown size={theme.icon.size.sm} />
|
||||
)}
|
||||
</StyledToggleButton>
|
||||
|
||||
<AnimatedExpandableContainer isExpanded={isExpanded}>
|
||||
<StyledReasoningContainer>
|
||||
<StyledReasoningText>{content}</StyledReasoningText>
|
||||
</StyledReasoningContainer>
|
||||
</AnimatedExpandableContainer>
|
||||
</>
|
||||
)}
|
||||
</StyledContainer>
|
||||
);
|
||||
};
|
||||
@@ -1,6 +1,5 @@
|
||||
import { ChatReferenceChip } from '@/ai/components/ChatReferenceChip';
|
||||
import { findChatReferences } from '@/ai/utils/findChatReferences';
|
||||
import { type ReactNode } from 'react';
|
||||
import { getChatReferenceSegments } from '@/ai/utils/getChatReferenceSegments';
|
||||
|
||||
type TextWithChatReferencesProps = {
|
||||
text: string;
|
||||
@@ -9,30 +8,15 @@ type TextWithChatReferencesProps = {
|
||||
export const TextWithChatReferences = ({
|
||||
text,
|
||||
}: TextWithChatReferencesProps) => {
|
||||
const references = findChatReferences(text);
|
||||
|
||||
if (references.length === 0) {
|
||||
return <>{text}</>;
|
||||
}
|
||||
|
||||
const parts: ReactNode[] = [];
|
||||
let lastIndex = 0;
|
||||
|
||||
for (const reference of references) {
|
||||
if (reference.index > lastIndex) {
|
||||
parts.push(text.slice(lastIndex, reference.index));
|
||||
}
|
||||
|
||||
parts.push(
|
||||
<ChatReferenceChip key={reference.index} reference={reference} />,
|
||||
);
|
||||
|
||||
lastIndex = reference.index + reference.fullMatch.length;
|
||||
}
|
||||
|
||||
if (lastIndex < text.length) {
|
||||
parts.push(text.slice(lastIndex));
|
||||
}
|
||||
|
||||
return <>{parts}</>;
|
||||
return (
|
||||
<>
|
||||
{getChatReferenceSegments(text).map((segment) =>
|
||||
typeof segment === 'string' ? (
|
||||
segment
|
||||
) : (
|
||||
<ChatReferenceChip key={segment.index} reference={segment} />
|
||||
),
|
||||
)}
|
||||
</>
|
||||
);
|
||||
};
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
import { ChatReferenceChipDisplay } from '@/ai/components/ChatReferenceChipDisplay';
|
||||
import { flatObjectMetadataItemsSelector } from '@/object-metadata/states/flatObjectMetadataItemsSelector';
|
||||
import { objectMetadataItemsByIdMapSelector } from '@/object-metadata/states/objectMetadataItemsByIdMapSelector';
|
||||
import { useAtomStateValue } from '@/ui/utilities/state/jotai/hooks/useAtomStateValue';
|
||||
import { useViewById } from '@/views/hooks/useViewById';
|
||||
import { AppPath } from 'twenty-shared/types';
|
||||
import { findById, getAppPath, isDefined } from 'twenty-shared/utils';
|
||||
import { getAppPath, isDefined } from 'twenty-shared/utils';
|
||||
import { useIcons } from 'twenty-ui/icon';
|
||||
import { useTheme } from 'twenty-ui/theme-constants';
|
||||
|
||||
@@ -17,12 +17,12 @@ export const ViewLink = ({ viewId, displayName }: ViewLinkProps) => {
|
||||
const { getIcon } = useIcons();
|
||||
|
||||
const { view } = useViewById(viewId);
|
||||
const flatObjectMetadataItems = useAtomStateValue(
|
||||
flatObjectMetadataItemsSelector,
|
||||
const objectMetadataItemsByIdMap = useAtomStateValue(
|
||||
objectMetadataItemsByIdMapSelector,
|
||||
);
|
||||
|
||||
const objectMetadataItem = isDefined(view)
|
||||
? flatObjectMetadataItems.find(findById(view.objectMetadataId))
|
||||
? objectMetadataItemsByIdMap.get(view.objectMetadataId)
|
||||
: undefined;
|
||||
|
||||
if (!isDefined(view) || !isDefined(objectMetadataItem)) {
|
||||
|
||||
@@ -1 +1,5 @@
|
||||
export const CHAT_REFERENCE_OPEN_PATTERN = '\\[\\[+';
|
||||
// The lookbehind anchors the match to the start of a bracket run. Without it,
|
||||
// every position inside a long run is a candidate start and the greedy + makes
|
||||
// the scan quadratic in the run length; a run start always yields the same
|
||||
// match, so no valid marker is lost.
|
||||
export const CHAT_REFERENCE_OPEN_PATTERN = '(?<!\\[)\\[\\[+';
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
export const CHAT_THREADS_PAGE_SIZE = 20;
|
||||
@@ -0,0 +1,8 @@
|
||||
import { type MarkdownBlockSplitCache } from '@/ai/types/MarkdownBlockSplitCache';
|
||||
|
||||
export const EMPTY_MARKDOWN_BLOCK_SPLIT_CACHE: MarkdownBlockSplitCache = {
|
||||
text: '',
|
||||
blocks: [],
|
||||
stablePrefix: '',
|
||||
stableBlocks: [],
|
||||
};
|
||||
@@ -1,11 +0,0 @@
|
||||
import { AgentChatComponentInstanceContext } from '@/ai/contexts/AgentChatComponentInstanceContext';
|
||||
import { createAtomComponentState } from '@/ui/utilities/state/jotai/utils/createAtomComponentState';
|
||||
import { type ExtendedUIMessage } from 'twenty-shared/ai';
|
||||
|
||||
export const agentChatMessagesComponentState = createAtomComponentState<
|
||||
ExtendedUIMessage[]
|
||||
>({
|
||||
key: 'agentChatMessagesComponentState',
|
||||
defaultValue: [],
|
||||
componentInstanceContext: AgentChatComponentInstanceContext,
|
||||
});
|
||||
@@ -1,8 +0,0 @@
|
||||
import { type BaseOutputSchemaV2 } from 'twenty-shared/workflow';
|
||||
|
||||
export type AgentResponseFormat =
|
||||
| { type: 'text' }
|
||||
| {
|
||||
type: 'json';
|
||||
schema: BaseOutputSchemaV2;
|
||||
};
|
||||
@@ -0,0 +1,6 @@
|
||||
export type MarkdownBlockSplitCache = {
|
||||
text: string;
|
||||
blocks: string[];
|
||||
stablePrefix: string;
|
||||
stableBlocks: string[];
|
||||
};
|
||||
@@ -1,40 +0,0 @@
|
||||
import { getFieldIcon } from '@/ai/utils/getFieldIcon';
|
||||
|
||||
describe('getFieldIcon', () => {
|
||||
describe('supported field types', () => {
|
||||
it('should return IconAbc for string field type', () => {
|
||||
expect(getFieldIcon('string')).toBe('IconAbc');
|
||||
});
|
||||
it('should return IconText for number field type', () => {
|
||||
expect(getFieldIcon('number')).toBe('IconText');
|
||||
});
|
||||
it('should return IconCheckbox for boolean field type', () => {
|
||||
expect(getFieldIcon('boolean')).toBe('IconCheckbox');
|
||||
});
|
||||
});
|
||||
|
||||
describe('unsupported and edge cases', () => {
|
||||
it('should return IconQuestionMark for an unsupported field type', () => {
|
||||
expect(getFieldIcon('totally-unknown-type' as any)).toBe(
|
||||
'IconQuestionMark',
|
||||
);
|
||||
});
|
||||
it('should return IconQuestionMark for undefined', () => {
|
||||
expect(getFieldIcon(undefined)).toBe('IconQuestionMark');
|
||||
});
|
||||
it('should return IconQuestionMark for null', () => {
|
||||
expect(getFieldIcon(null as any)).toBe('IconQuestionMark');
|
||||
});
|
||||
it('should return IconQuestionMark for empty string', () => {
|
||||
expect(getFieldIcon('' as any)).toBe('IconQuestionMark');
|
||||
});
|
||||
});
|
||||
|
||||
describe('consistency', () => {
|
||||
it('should return the same icon for the same field type', () => {
|
||||
const result1 = getFieldIcon('string');
|
||||
const result2 = getFieldIcon('string');
|
||||
expect(result1).toBe(result2);
|
||||
});
|
||||
});
|
||||
});
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
import { marked } from 'marked';
|
||||
|
||||
import { EMPTY_MARKDOWN_BLOCK_SPLIT_CACHE } from '@/ai/constants/EmptyMarkdownBlockSplitCache';
|
||||
import { type MarkdownBlockSplitCache } from '@/ai/types/MarkdownBlockSplitCache';
|
||||
import { getMarkdownBlocksIncrementally } from '@/ai/utils/getMarkdownBlocksIncrementally';
|
||||
|
||||
const lexerBlocks = (text: string): string[] =>
|
||||
marked.lexer(text).map((token) => token.raw);
|
||||
|
||||
const streamThrough = (text: string, chunkSize: number): string[] => {
|
||||
let cache: MarkdownBlockSplitCache = EMPTY_MARKDOWN_BLOCK_SPLIT_CACHE;
|
||||
let blocks: string[] = [];
|
||||
|
||||
for (let end = chunkSize; end < text.length + chunkSize; end += chunkSize) {
|
||||
const result = getMarkdownBlocksIncrementally({
|
||||
text: text.slice(0, Math.min(end, text.length)),
|
||||
cache,
|
||||
});
|
||||
|
||||
cache = result.cache;
|
||||
blocks = result.blocks;
|
||||
}
|
||||
|
||||
return blocks;
|
||||
};
|
||||
|
||||
const FIXTURES: Record<string, string> = {
|
||||
paragraphs: 'First paragraph.\n\nSecond paragraph.\n\nThird paragraph.',
|
||||
looseList: '- a\n\n- b\n\n- c\n\nplain paragraph after list',
|
||||
listWithLooseContinuation:
|
||||
'- item one\n\n continued loose content\n\n- item two',
|
||||
nestedList: '1. first\n - sub a\n\n - sub b\n2. second',
|
||||
table: '| a | b |\n|---|---|\n| 1 | 2 |\n| 3 | 4 |\n\nafter table',
|
||||
codeFence: 'before\n\n```js\nconst x = 1;\n\nconst y = 2;\n```\n\nafter',
|
||||
unclosedCodeFence: 'before\n\n```js\nconst x = 1;\n\nstill code',
|
||||
setextHeading: 'Title\n=====\n\nBody text\n\nSub\n-----\n\nmore',
|
||||
mixedBlocks:
|
||||
'# H1\n\ntext\n\n## H2\n\n- list\n- items\n\n> quote\n> more quote\n\nend',
|
||||
blockquotes: '> a\n\n> b\n\ntext',
|
||||
chatReferences:
|
||||
'Check [[field:12345678-1234-5678-abcd-123456789012:Annual Revenue[[/field]] and\n\n- [[view:12345678-1234-5678-abcd-123456789012:All[[/view]]\n\n| [[object:company:Companies[[/object]] | x |\n|---|---|\n| a | b |',
|
||||
windowsLineEndings: 'line one\r\n\r\n- a\r\n\r\n- b\r\n\r\nend',
|
||||
};
|
||||
|
||||
describe('getMarkdownBlocksIncrementally', () => {
|
||||
it.each(Object.entries(FIXTURES))(
|
||||
'should match a full marked.lexer split when streamed char by char for %s',
|
||||
(_name, text) => {
|
||||
let cache: MarkdownBlockSplitCache = EMPTY_MARKDOWN_BLOCK_SPLIT_CACHE;
|
||||
|
||||
for (let end = 1; end <= text.length; end++) {
|
||||
const prefix = text.slice(0, end);
|
||||
const result = getMarkdownBlocksIncrementally({ text: prefix, cache });
|
||||
|
||||
cache = result.cache;
|
||||
|
||||
expect(result.blocks).toEqual(lexerBlocks(prefix));
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it.each(Object.entries(FIXTURES))(
|
||||
'should match a full marked.lexer split when streamed in chunks for %s',
|
||||
(_name, text) => {
|
||||
for (const chunkSize of [3, 7, 20]) {
|
||||
expect(streamThrough(text, chunkSize)).toEqual(lexerBlocks(text));
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it('should return the cached blocks when the text is unchanged', () => {
|
||||
const text = 'A paragraph.\n\nAnother paragraph.';
|
||||
const firstResult = getMarkdownBlocksIncrementally({
|
||||
text,
|
||||
cache: EMPTY_MARKDOWN_BLOCK_SPLIT_CACHE,
|
||||
});
|
||||
const secondResult = getMarkdownBlocksIncrementally({
|
||||
text,
|
||||
cache: firstResult.cache,
|
||||
});
|
||||
|
||||
expect(secondResult.blocks).toBe(firstResult.blocks);
|
||||
expect(secondResult.cache).toBe(firstResult.cache);
|
||||
});
|
||||
|
||||
it('should recover with a full split when the text is not an append', () => {
|
||||
const firstResult = getMarkdownBlocksIncrementally({
|
||||
text: 'First paragraph.\n\nSecond paragraph.\n\nThird one.',
|
||||
cache: EMPTY_MARKDOWN_BLOCK_SPLIT_CACHE,
|
||||
});
|
||||
|
||||
const replacedText = 'Completely different text.\n\nWith new blocks.';
|
||||
const secondResult = getMarkdownBlocksIncrementally({
|
||||
text: replacedText,
|
||||
cache: firstResult.cache,
|
||||
});
|
||||
|
||||
expect(secondResult.blocks).toEqual(lexerBlocks(replacedText));
|
||||
});
|
||||
|
||||
it('should handle an empty text', () => {
|
||||
const result = getMarkdownBlocksIncrementally({
|
||||
text: '',
|
||||
cache: EMPTY_MARKDOWN_BLOCK_SPLIT_CACHE,
|
||||
});
|
||||
|
||||
expect(result.blocks).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -7,6 +7,10 @@ import { getSurplusCloseBracketLength } from '@/ai/utils/getSurplusCloseBracketL
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
export const findChatReferences = (text: string): ChatReferenceMatch[] => {
|
||||
if (!text.includes('[[')) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const starts: ChatReferenceStart[] = [];
|
||||
|
||||
CHAT_REFERENCE_START_REGEX.lastIndex = 0;
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
import { type ChatReferenceMatch } from '@/ai/types/ChatReferenceMatch';
|
||||
import { findChatReferences } from '@/ai/utils/findChatReferences';
|
||||
|
||||
export const getChatReferenceSegments = (
|
||||
text: string,
|
||||
): Array<string | ChatReferenceMatch> => {
|
||||
const references = findChatReferences(text);
|
||||
const segments: Array<string | ChatReferenceMatch> = [];
|
||||
let lastIndex = 0;
|
||||
|
||||
for (const reference of references) {
|
||||
if (reference.index > lastIndex) {
|
||||
segments.push(text.slice(lastIndex, reference.index));
|
||||
}
|
||||
|
||||
segments.push(reference);
|
||||
lastIndex = reference.index + reference.fullMatch.length;
|
||||
}
|
||||
|
||||
if (lastIndex < text.length) {
|
||||
segments.push(text.slice(lastIndex));
|
||||
}
|
||||
|
||||
return segments;
|
||||
};
|
||||
@@ -1,14 +0,0 @@
|
||||
import { type AgentResponseFieldType } from 'twenty-shared/ai';
|
||||
|
||||
export const getFieldIcon = (fieldType?: AgentResponseFieldType): string => {
|
||||
switch (fieldType) {
|
||||
case 'string':
|
||||
return 'IconAbc';
|
||||
case 'number':
|
||||
return 'IconText';
|
||||
case 'boolean':
|
||||
return 'IconCheckbox';
|
||||
default:
|
||||
return 'IconQuestionMark';
|
||||
}
|
||||
};
|
||||
@@ -0,0 +1,73 @@
|
||||
import { type MarkdownBlockSplitCache } from '@/ai/types/MarkdownBlockSplitCache';
|
||||
import { Lexer } from 'marked';
|
||||
|
||||
// Appended text can reopen the last block, and a list followed by a blank-line
|
||||
// token merges with a later item ("- a\n\n" + "- b" is one loose list), so the
|
||||
// two trailing blocks must be re-tokenized on every flush. Anything before them
|
||||
// was terminated by content that is still present, so it can never change.
|
||||
const UNSTABLE_TRAILING_BLOCK_COUNT = 2;
|
||||
|
||||
// marked.lexer normalizes line endings before tokenizing; blockTokens does not,
|
||||
// so normalize here to keep raw offsets aligned with the sliced text.
|
||||
const normalizeLineEndings = (text: string): string =>
|
||||
text.replace(/\r\n|\r/g, '\n');
|
||||
|
||||
const splitIntoBlocks = (text: string): string[] =>
|
||||
new Lexer().blockTokens(text, []).map((token) => token.raw);
|
||||
|
||||
// A streamed message only grows, so instead of re-tokenizing the whole message
|
||||
// on every flush, reuse the blocks that can no longer change and re-tokenize
|
||||
// only the unstable tail, keeping each flush O(appended text) instead of
|
||||
// O(whole message). Any non-append change misses the stablePrefix guard and
|
||||
// falls back to a full split.
|
||||
export const getMarkdownBlocksIncrementally = ({
|
||||
text,
|
||||
cache,
|
||||
}: {
|
||||
text: string;
|
||||
cache: MarkdownBlockSplitCache;
|
||||
}): { blocks: string[]; cache: MarkdownBlockSplitCache } => {
|
||||
const normalizedText = normalizeLineEndings(text);
|
||||
|
||||
if (normalizedText === cache.text) {
|
||||
return { blocks: cache.blocks, cache };
|
||||
}
|
||||
|
||||
const canReuseStableBlocks =
|
||||
cache.stablePrefix.length > 0 &&
|
||||
normalizedText.startsWith(cache.stablePrefix);
|
||||
|
||||
const stableBlocks = canReuseStableBlocks ? cache.stableBlocks : [];
|
||||
const stablePrefix = canReuseStableBlocks ? cache.stablePrefix : '';
|
||||
|
||||
const tailBlocks = splitIntoBlocks(normalizedText.slice(stablePrefix.length));
|
||||
|
||||
const blocks = [...stableBlocks, ...tailBlocks];
|
||||
const nextStableBlocks = blocks.slice(
|
||||
0,
|
||||
Math.max(0, blocks.length - UNSTABLE_TRAILING_BLOCK_COUNT),
|
||||
);
|
||||
// Append newly stabilized raws instead of re-joining the whole prefix on
|
||||
// every flush. The stable set shrinks when the tail collapses into fewer
|
||||
// than two blocks; then trim the raws that fell out of it.
|
||||
const nextStablePrefix =
|
||||
nextStableBlocks.length >= stableBlocks.length
|
||||
? stablePrefix + nextStableBlocks.slice(stableBlocks.length).join('')
|
||||
: stablePrefix.slice(
|
||||
0,
|
||||
stablePrefix.length -
|
||||
stableBlocks
|
||||
.slice(nextStableBlocks.length)
|
||||
.reduce((removedLength, raw) => removedLength + raw.length, 0),
|
||||
);
|
||||
|
||||
return {
|
||||
blocks,
|
||||
cache: {
|
||||
text: normalizedText,
|
||||
blocks,
|
||||
stablePrefix: nextStablePrefix,
|
||||
stableBlocks: nextStableBlocks,
|
||||
},
|
||||
};
|
||||
};
|
||||
@@ -1,30 +1,22 @@
|
||||
import { escapeMarkdownForChatReference } from '@/ai/utils/escapeMarkdownForChatReference';
|
||||
import { findChatReferences } from '@/ai/utils/findChatReferences';
|
||||
import { formatChatReference } from '@/ai/utils/formatChatReference';
|
||||
import { getChatReferenceSegments } from '@/ai/utils/getChatReferenceSegments';
|
||||
|
||||
export const protectChatReferencesForMarkdown = (text: string): string => {
|
||||
const references = findChatReferences(text);
|
||||
const segments = getChatReferenceSegments(text);
|
||||
|
||||
if (references.length === 0) {
|
||||
if (segments.length === 1 && typeof segments[0] === 'string') {
|
||||
return text;
|
||||
}
|
||||
|
||||
const parts: string[] = [];
|
||||
let lastIndex = 0;
|
||||
|
||||
for (const reference of references) {
|
||||
parts.push(text.slice(lastIndex, reference.index));
|
||||
parts.push(
|
||||
formatChatReference({
|
||||
...reference,
|
||||
displayName: escapeMarkdownForChatReference(reference.displayName),
|
||||
}),
|
||||
);
|
||||
|
||||
lastIndex = reference.index + reference.fullMatch.length;
|
||||
}
|
||||
|
||||
parts.push(text.slice(lastIndex));
|
||||
|
||||
return parts.join('');
|
||||
return segments
|
||||
.map((segment) =>
|
||||
typeof segment === 'string'
|
||||
? segment
|
||||
: formatChatReference({
|
||||
...segment,
|
||||
displayName: escapeMarkdownForChatReference(segment.displayName),
|
||||
}),
|
||||
)
|
||||
.join('');
|
||||
};
|
||||
|
||||
+16
-16
@@ -1,32 +1,29 @@
|
||||
import { flattenedFieldMetadataItemsSelector } from '@/object-metadata/states/flattenedFieldMetadataItemsSelector';
|
||||
import { objectMetadataItemsWithFieldsSelector } from '@/object-metadata/states/objectMetadataItemsWithFieldsSelector';
|
||||
import { fieldMetadataItemByIdMapSelector } from '@/object-metadata/states/fieldMetadataItemByIdMapSelector';
|
||||
import { objectMetadataItemsByIdMapSelector } from '@/object-metadata/states/objectMetadataItemsByIdMapSelector';
|
||||
import { createAtomFamilySelector } from '@/ui/utilities/state/jotai/utils/createAtomFamilySelector';
|
||||
import { findById, isDefined } from 'twenty-shared/utils';
|
||||
import { isDefined } from 'twenty-shared/utils';
|
||||
|
||||
export const fieldMetadataItemByIdSelector = createAtomFamilySelector({
|
||||
key: 'fieldMetadataItemByIdSelector',
|
||||
get:
|
||||
({ fieldMetadataItemId }: { fieldMetadataItemId: string }) =>
|
||||
({ get }) => {
|
||||
const objectMetadataItems = get(objectMetadataItemsWithFieldsSelector);
|
||||
const flattenedFieldMetadataItems = get(
|
||||
flattenedFieldMetadataItemsSelector,
|
||||
const foundFieldMetadataItem = get(fieldMetadataItemByIdMapSelector).get(
|
||||
fieldMetadataItemId,
|
||||
);
|
||||
|
||||
const foundObjectMetadataItem = objectMetadataItems.find(
|
||||
(objectMetadataItem) =>
|
||||
objectMetadataItem.fields.some(findById(fieldMetadataItemId)),
|
||||
);
|
||||
|
||||
if (!isDefined(foundObjectMetadataItem)) {
|
||||
if (
|
||||
!isDefined(foundFieldMetadataItem) ||
|
||||
!isDefined(foundFieldMetadataItem.objectMetadataId)
|
||||
) {
|
||||
return {};
|
||||
}
|
||||
|
||||
const foundFieldMetadataItem = flattenedFieldMetadataItems.find(
|
||||
findById(fieldMetadataItemId),
|
||||
);
|
||||
const foundObjectMetadataItem = get(
|
||||
objectMetadataItemsByIdMapSelector,
|
||||
).get(foundFieldMetadataItem.objectMetadataId);
|
||||
|
||||
if (!isDefined(foundFieldMetadataItem)) {
|
||||
if (!isDefined(foundObjectMetadataItem)) {
|
||||
return {};
|
||||
}
|
||||
|
||||
@@ -35,4 +32,7 @@ export const fieldMetadataItemByIdSelector = createAtomFamilySelector({
|
||||
foundObjectMetadataItem,
|
||||
};
|
||||
},
|
||||
areEqual: (previous, next) =>
|
||||
previous.foundFieldMetadataItem === next.foundFieldMetadataItem &&
|
||||
previous.foundObjectMetadataItem === next.foundObjectMetadataItem,
|
||||
});
|
||||
|
||||
+7
-11
@@ -1,4 +1,5 @@
|
||||
import { objectMetadataItemsWithFieldsSelector } from '@/object-metadata/states/objectMetadataItemsWithFieldsSelector';
|
||||
import { objectMetadataItemsByNamePluralMapSelector } from '@/object-metadata/states/objectMetadataItemsByNamePluralMapSelector';
|
||||
import { objectMetadataItemsByNameSingularMapSelector } from '@/object-metadata/states/objectMetadataItemsByNameSingularMapSelector';
|
||||
import { type EnrichedObjectMetadataItem } from '@/object-metadata/types/EnrichedObjectMetadataItem';
|
||||
import { createAtomFamilySelector } from '@/ui/utilities/state/jotai/utils/createAtomFamilySelector';
|
||||
|
||||
@@ -15,23 +16,18 @@ export const objectMetadataItemFamilySelector = createAtomFamilySelector<
|
||||
get:
|
||||
({ objectNameType, objectName }: ObjectMetadataItemSelector) =>
|
||||
({ get }) => {
|
||||
const objectMetadataItems = get(objectMetadataItemsWithFieldsSelector);
|
||||
|
||||
if (objectNameType === 'singular') {
|
||||
return (
|
||||
objectMetadataItems.find(
|
||||
(objectMetadataItem) =>
|
||||
objectMetadataItem.nameSingular === objectName,
|
||||
) ?? null
|
||||
get(objectMetadataItemsByNameSingularMapSelector).get(objectName) ??
|
||||
null
|
||||
);
|
||||
} else if (objectNameType === 'plural') {
|
||||
return (
|
||||
objectMetadataItems.find(
|
||||
(objectMetadataItem) =>
|
||||
objectMetadataItem.namePlural === objectName,
|
||||
) ?? null
|
||||
get(objectMetadataItemsByNamePluralMapSelector).get(objectName) ??
|
||||
null
|
||||
);
|
||||
}
|
||||
return null;
|
||||
},
|
||||
areEqual: (previous, next) => previous === next,
|
||||
});
|
||||
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
import { objectMetadataItemsWithFieldsSelector } from '@/object-metadata/states/objectMetadataItemsWithFieldsSelector';
|
||||
import { type EnrichedObjectMetadataItem } from '@/object-metadata/types/EnrichedObjectMetadataItem';
|
||||
import { createAtomSelector } from '@/ui/utilities/state/jotai/utils/createAtomSelector';
|
||||
|
||||
export const objectMetadataItemsByIdMapSelector = createAtomSelector<
|
||||
Map<string, EnrichedObjectMetadataItem>
|
||||
>({
|
||||
key: 'objectMetadataItemsByIdMapSelector',
|
||||
get: ({ get }) => {
|
||||
const objectMetadataItems = get(objectMetadataItemsWithFieldsSelector);
|
||||
|
||||
return new Map(
|
||||
objectMetadataItems.map((objectMetadataItem) => [
|
||||
objectMetadataItem.id,
|
||||
objectMetadataItem,
|
||||
]),
|
||||
);
|
||||
},
|
||||
});
|
||||
+3
-3
@@ -1,5 +1,5 @@
|
||||
import { createAtomFamilySelector } from '@/ui/utilities/state/jotai/utils/createAtomFamilySelector';
|
||||
import { viewsSelector } from '@/views/states/selectors/viewsSelector';
|
||||
import { viewsByIdMapSelector } from '@/views/states/selectors/viewsByIdMapSelector';
|
||||
import { type View } from '@/views/types/View';
|
||||
|
||||
export const viewFromViewIdFamilySelector = createAtomFamilySelector<
|
||||
@@ -10,7 +10,7 @@ export const viewFromViewIdFamilySelector = createAtomFamilySelector<
|
||||
get:
|
||||
({ viewId }) =>
|
||||
({ get }) => {
|
||||
const views = get(viewsSelector);
|
||||
return views?.find((view) => view.id === viewId);
|
||||
return get(viewsByIdMapSelector).get(viewId);
|
||||
},
|
||||
areEqual: (previous, next) => previous === next,
|
||||
});
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
import { createAtomSelector } from '@/ui/utilities/state/jotai/utils/createAtomSelector';
|
||||
import { viewsSelector } from '@/views/states/selectors/viewsSelector';
|
||||
import { type ViewWithRelations } from '@/views/types/ViewWithRelations';
|
||||
|
||||
export const viewsByIdMapSelector = createAtomSelector<
|
||||
Map<string, ViewWithRelations>
|
||||
>({
|
||||
key: 'viewsByIdMapSelector',
|
||||
get: ({ get }) => new Map(get(viewsSelector).map((view) => [view.id, view])),
|
||||
});
|
||||
Reference in New Issue
Block a user