Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmenter is memory hungry #100

Open
bitsofbits opened this issue Oct 11, 2019 · 0 comments
Open

Segmenter is memory hungry #100

bitsofbits opened this issue Oct 11, 2019 · 0 comments

Comments

@bitsofbits
Copy link
Contributor

The segmenter is unnecessarily memory hungry. This became an issue during the rewrite to improve the core of the segmenter, which generated a lot more noise segments, and in turn caused crashes due to insufficient instance memory. Fix should be straightforward and consists of two parts.

  • Filter out noise segments.
  • Use Cogrouping of segments and messages rather than passing segments in as a side input
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant