Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize dedup to avoid oom #568

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Optimize dedup to avoid oom #568

wants to merge 1 commit into from

Conversation

coolderli
Copy link

  • distinct() will storage the data, and the downstream will read from the shuffle. We do not need the cache any more.
  • Use the count() to instead the collect() to avoid the drive OOM.

@HYLcool HYLcool added enhancement New feature or request good first issue Good for newcomers dj:dist issues/PRs about distributed data processing dj:efficiency regarding to efficiency issues and enhancements dj:tools issues/PRs about specific tools labels Feb 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chenyushuo chenyushuo Awaiting requested review from chenyushuo

@pan-x-c pan-x-c Awaiting requested review from pan-x-c

@garyzhang99 garyzhang99 Awaiting requested review from garyzhang99

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
dj:dist issues/PRs about distributed data processing dj:efficiency regarding to efficiency issues and enhancements dj:tools issues/PRs about specific tools enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@coolderli @HYLcool