Skip to content

Commit

Permalink
feat(issue summary) Only show possible cause if confident and novel (#…
Browse files Browse the repository at this point in the history
…84349)

## Background

`possible_cause_confidence` increases when there's less speculation
(which correlates w/ slightly more accurate causes)

`possible_cause_novelty` increases when there's more novelty / less
redundancy wrt `whats_wrong`

thresholds led to 50% of possible causes getting dropped on our autofix
sentry issues ([notebook
here](https://github.com/getsentry/data-analysis/blob/main/autofix/issue_summary/issue_summary_confidence.ipynb))

## Backend changes

corresponding backend change to sentry: #84346

corresponding backend change to seer:
getsentry/seer#1788

without the sentry backend change (there are no `data.scores`), default
to current behavior: always show possible cause


![before](https://github.com/user-attachments/assets/9f632a94-690b-4602-bdea-cbd2b95824b0)

with the change, only show it if both scores are greater than the
threshold


![after](https://github.com/user-attachments/assets/0b1e84af-cc0b-4d0a-8106-58cc615ea602)

(note: this example is actually considered novel and confident-enough. I
hardcoded the threshold to test the behavior)
  • Loading branch information
kddubey authored Jan 31, 2025
1 parent 1f0d5ad commit e84cc72
Show file tree
Hide file tree
Showing 3 changed files with 63 additions and 10 deletions.
41 changes: 39 additions & 2 deletions static/app/components/group/groupSummary.spec.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,22 @@ describe('GroupSummary', function () {
trace: 'Test trace',
possibleCause: 'Test possible cause',
headline: 'Test headline',
scores: {
possibleCauseConfidence: 0.9,
possibleCauseNovelty: 0.8,
},
};

const mockSummaryDataWithLowScores = {
groupId: '1',
whatsWrong: 'Test whats wrong',
trace: 'Test trace',
possibleCause: 'Test possible cause',
headline: 'Test headline',
scores: {
possibleCauseConfidence: 0.5,
possibleCauseNovelty: 0.0,
},
};

beforeEach(() => {
Expand Down Expand Up @@ -62,6 +78,27 @@ describe('GroupSummary', function () {
expect(screen.getByText('Test possible cause')).toBeInTheDocument();
});

it('renders the summary without possible cause', async function () {
MockApiClient.addMockResponse({
url: `/organizations/${mockProject.organization.slug}/issues/${mockGroup.id}/summarize/`,
method: 'POST',
body: mockSummaryDataWithLowScores,
});

render(<GroupSummary event={mockEvent} group={mockGroup} project={mockProject} />, {
organization,
});

await waitFor(() => {
expect(screen.getByText("What's wrong")).toBeInTheDocument();
});
expect(screen.getByText('Test whats wrong')).toBeInTheDocument();
expect(screen.getByText('In the trace')).toBeInTheDocument();
expect(screen.getByText('Test trace')).toBeInTheDocument();
expect(screen.queryByText('Possible cause')).not.toBeInTheDocument();
expect(screen.queryByText('Test possible cause')).not.toBeInTheDocument();
});

it('shows loading state', function () {
MockApiClient.addMockResponse({
url: `/organizations/${mockProject.organization.slug}/issues/${mockGroup.id}/summarize/`,
Expand All @@ -73,8 +110,8 @@ describe('GroupSummary', function () {
organization,
});

// Should show loading placeholders
expect(screen.getAllByTestId('loading-placeholder')).toHaveLength(2);
// Should show loading placeholders. Currently we load the whatsWrong section
expect(screen.getAllByTestId('loading-placeholder')).toHaveLength(1);
});

it('shows error state', async function () {
Expand Down
30 changes: 23 additions & 7 deletions static/app/components/group/groupSummary.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,18 @@ import {useFeedbackForm} from 'sentry/utils/useFeedbackForm';
import useOrganization from 'sentry/utils/useOrganization';
import {useAiConfig} from 'sentry/views/issueDetails/streamline/hooks/useAiConfig';

const POSSIBLE_CAUSE_CONFIDENCE_THRESHOLD = 0.468;
const POSSIBLE_CAUSE_NOVELTY_THRESHOLD = 0.419;

interface GroupSummaryData {
groupId: string;
headline: string;
eventId?: string | null;
possibleCause?: string | null;
scores?: {
possibleCauseConfidence: number;
possibleCauseNovelty: number;
} | null;
trace?: string | null;
whatsWrong?: string | null;
}
Expand Down Expand Up @@ -152,6 +159,11 @@ export function GroupSummary({
: []),
];

const shouldShowPossibleCause =
!data?.scores ||
(data.scores.possibleCauseConfidence >= POSSIBLE_CAUSE_CONFIDENCE_THRESHOLD &&
data.scores.possibleCauseNovelty >= POSSIBLE_CAUSE_NOVELTY_THRESHOLD);

const insightCards = [
{
id: 'whats_wrong',
Expand All @@ -167,13 +179,17 @@ export function GroupSummary({
icon: <IconSpan size="sm" />,
showWhenLoading: false,
},
{
id: 'possible_cause',
title: t('Possible cause'),
insight: data?.possibleCause,
icon: <IconFocus size="sm" />,
showWhenLoading: true,
},
...(shouldShowPossibleCause
? [
{
id: 'possible_cause',
title: t('Possible cause'),
insight: data?.possibleCause,
icon: <IconFocus size="sm" />,
showWhenLoading: false,
},
]
: []),
];

return (
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ describe('SolutionsSection', () => {
);

expect(screen.getByText('Solutions Hub')).toBeInTheDocument();
expect(screen.getAllByTestId('loading-placeholder')).toHaveLength(3);
expect(screen.getAllByTestId('loading-placeholder')).toHaveLength(2); // whatsWrong and Open Autofix
});

it('renders summary when AI features are enabled and data is available', async () => {
Expand Down

0 comments on commit e84cc72

Please sign in to comment.