Skip to content

Issues: huggingface/datatrove

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

60 Open 88 Closed
60 Open 88 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

[error] too many open files in MinhashDedupBuckets process
#335 opened Jan 30, 2025 by parkwonjae
2
MinhashBuildIndex fails with StopIteration when initializing priority queue
#333 opened Jan 30, 2025 by nelson-liu
1
1
Bug: Default Adapter assumes type of metadata column in source data
#328 opened Jan 25, 2025 by amangup
How to choose the best timeout value in extractors?
#326 opened Jan 22, 2025 by jordane95
2
JsonlWriter failed when trying to write Timestamp object
#325 opened Jan 21, 2025 by jgcb00
1
Unexpected behavior when using sentence_dedup with split_sentences=True
#324 opened Jan 18, 2025 by ftgreat
FastText: newer model
#321 opened Jan 13, 2025 by BramVanroy
Missing main function in check_dataset.py
#315 opened Dec 18, 2024 by hadim
1
How do I assess whether the accuracy before and after dedup
#312 opened Dec 10, 2024 by prozzzzzz
1
Embedding-based decontamination
#310 opened Dec 9, 2024 by jordane95
1
Bug in HuggingfaceDatasetReader in streaming mode
#308 opened Nov 29, 2024 by habanoz
2
How to read text files
#305 opened Nov 21, 2024 by srinjoym-cerebras
buffer of a multiple of 68 bytes Minhash bug
#301 opened Nov 12, 2024 by mcleish7
Explanation of what exactly the various dedup strategies do
#299 opened Oct 22, 2024 by aditya-hari
Unexpected performance degradation behavior in minhash deduplication stage 2
#298 opened Oct 17, 2024 by Maghoumi
2
Wrong stats in multi-node local executor
#297 opened Oct 15, 2024 by jordane95
1
Does fineweb.py perform Element and paragraph level deduplication?
#295 opened Oct 9, 2024 by silverriver
SLURM cannot achieve cross-node parallelism
#292 opened Sep 25, 2024 by ShayDuane
Multi-node parallelism on slurm clusters
#290 opened Sep 22, 2024 by shizhediao
6
Naming Gopher's "max_non_alpha_words_ratio"
#289 opened Sep 21, 2024 by BramVanroy
[FEATURE] CC Filter
#283 opened Sep 4, 2024 by BramVanroy
Filter very slow
#270 opened Aug 19, 2024 by hiennm15
6
example for decontamination
#269 opened Aug 19, 2024 by jordane95
How to look into the processed data?
#266 opened Aug 16, 2024 by shizhediao
3
Incorrect Job ID Extraction on Clusters with Custom Slurm Output
#265 opened Aug 12, 2024 by StephenRebelSSC
2
ProTip! Mix and match filters to narrow down what you’re looking for.