make pycld2 and fasttext libraries optional
pycld2
fasttext
replace langid.py library with py3langid
langid.py
py3langid
update github workflows and include Python 3.12 tests
OpusRead interface using moses format (requires opustools >= 1.6.2)
OpusRead
moses
opustools >= 1.6.2
fix score method in SentenceEmbeddingFilter (https://github.com/Helsinki-NLP/OpusFilter/pull/71)
SentenceEmbeddingFilter
fix filter and filterfalse methods in SentenceEmbeddingFilter
opusfilter-autogen script for automatic filter config generation
opusfilter-autogen
score_direction, accept_threshold, and reject_threshold properties for filters
score_direction
accept_threshold
reject_threshold
refactor code and move auxiliary methods to opusfilter.util
update varikn installation instructions (installable from PyPI)
Python 3.6 support
catch NotImplementedError from beautifulsoup 4.11.2
catch ParserRejectedMarkup from beautifulsoup 4.12.0
add slice missing from the enabled steps
slice
improve documentation
import slow libraries only when needed
change RepetitionFilter to use single score for consistency with the threshold
RepetitionFilter
allow float thresholds for AverageWordLengthFilter
AverageWordLengthFilter
remove unnecessary code from RegExpSub
RegExpSub
add missing document file
map_space_to option for Jieba and MeCab tokenizers to preserve existing space characters in input
map_space_to
parallel processing options for filter, score, and preprocess steps
re-organize documentation and support building it with sphinx
catch TypeError exceptions from BeautifulSoup in HtmlTagFilter
an option to write filter scores to a file with opusfilter-test
opusfilter-test
new filters: AlphabetRatioFilter, RegExpFilter, SimilarityFilter, SentenceEmbeddingFilter
AlphabetRatioFilter
RegExpFilter
SimilarityFilter
subword segmentation support for the n-gram language models and language model filters
allow per-language parameters for LengthFilter, LengthRatioFilter, LongWordFilter, and AverageWordLengthFilter
fix documentation for train_aligment parameters
train_aligment
fix bug in classifier training without development set
new OpusFilterRuntimeError exception for having e.g. empty training data
option to save scores from the training data when creating word aligment priors
LMClassifierFilter for classification based on n-gram language models
add workdir attribute to the FilterABC base class and change that the filters should use it for any file parameters
workdir
FilterABC
support for Chinese word segmentation using jieba as a tokenizer (https://github.com/Helsinki-NLP/OpusFilter/pull/27)
jieba
fix wrong keyword argument name in opusfilter-duplicates
move “How to contribute” to docs/CONTRIBUTING.md
fix setuptools requirement (https://github.com/Helsinki-NLP/OpusFilter/issues/21)
fix version requirement for pandas (>=1.0.0)
replace PyYAML with ruamel.yaml
support for variables in the YAML configuration (https://github.com/Helsinki-NLP/OpusFilter/pull/13)
support to fasttext based for language detection (https://github.com/Helsinki-NLP/OpusFilter/pull/20)
extend to n-lingual parallel data instead of just bilingual data
switch tokenizer to fast-mosestokenizer
fast-mosestokenizer
new commands: opusfilter-diagram, opusfilter-duplicates, opusfilter-test
opusfilter-diagram
opusfilter-duplicates
new filters: LongestCommonSubstringFilter, AverageWordLengthFilter
LongestCommonSubstringFilter
github CI workflow for flake8 and unittests
behaviour of simple filters on empty segments
improved logging, documentation, and project files
prevent UnboundLocalError for empty output after filter
UnboundLocalError
First tagged version.
Make a pull request to the develop branch instead of master.
develop
master
The code should support at least Python versions from 3.8 to 3.11.
The code should support at least Python versions from 3.8 to 3.12.
Please follow PEP 8. Exception: The maximum line length is 127 characters instead of 79.
Especially for new features, please include test cases for unit testing.
pytest
Especially for larger contributions, consider using a code analysis tool like Pylint. Install it e.g. via pip, run pylint opusfilter/ in the project root and fix diff --git a/_sources/CHANGELOG.md.txt b/_sources/CHANGELOG.md.txt index 836436c..4a496b5 100644 --- a/_sources/CHANGELOG.md.txt +++ b/_sources/CHANGELOG.md.txt @@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [3.2.0] - 2024-08-14 + +### Changed + +- make `pycld2` and `fasttext` libraries optional +- replace `langid.py` library with `py3langid` +- update github workflows and include Python 3.12 tests + +### Fixed + +- `OpusRead` interface using `moses` format (requires `opustools >= 1.6.2`) + ## [3.1.0] - 2024-06-05 ### Added @@ -204,7 +216,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 First tagged version. -[Unreleased]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.1.0...develop +[Unreleased]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.2.0...develop +[3.2.0]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.1.0...3.2.0 [3.1.0]: https://github.com/Helsinki-NLP/OpusFilter/compare/3.0.0...3.1.0 [3.0.0]: https://github.com/Helsinki-NLP/OpusFilter/compare/2.6.0...3.0.0 [2.6.0]: https://github.com/Helsinki-NLP/OpusFilter/compare/2.5.1...2.6.0 diff --git a/_sources/CONTRIBUTING.md.txt b/_sources/CONTRIBUTING.md.txt index 506eb41..9a0de65 100644 --- a/_sources/CONTRIBUTING.md.txt +++ b/_sources/CONTRIBUTING.md.txt @@ -5,7 +5,7 @@ issues page. We are also happy to consider pull requests. There are a few rules for pull requests: * Make a pull request to the `develop` branch instead of `master`. -* The code should support at least Python versions from 3.8 to 3.11. +* The code should support at least Python versions from 3.8 to 3.12. * Please follow [PEP 8](https://www.python.org/dev/peps/pep-0008/). Exception: The maximum line length is 127 characters instead of 79. * Especially for new features, please include test cases for unit testing. @@ -20,7 +20,7 @@ skips the respective tests if not.) GitHub workflows defined in the project run automatically `flake8` checks and unit testing with `pytest` using Python 3.8, 3.9, 3.10, -and 3.11. +3.11, and 3.12. Especially for larger contributions, consider using a code analysis tool like [Pylint](https://github.com/PyCQA/pylint). Install it diff --git a/_sources/filters/script_and_language_identification_filters.md.txt b/_sources/filters/script_and_language_identification_filters.md.txt index 961898e..cf1a665 100644 --- a/_sources/filters/script_and_language_identification_filters.md.txt +++ b/_sources/filters/script_and_language_identification_filters.md.txt @@ -35,7 +35,7 @@ Filter segments based on their language identification confidence scores. Parameters: * `languages`: expected languages (ISO639 language codes) for the segments -* `id_method`: language indentification method (`langid` for using the `langid` library, `cld2` for using the `cld2` library, or `fasttext` for using a `fasttext` model; the default is `langid`) +* `id_method`: language indentification method (`langid`, `lingua`, `cld2`, `fasttext`; default `langid`) * `thresholds`: minimum identification confidence score for the segments (a single float or a list of floats per language) * `fasttext_model_path`: path for a `fasttext` model (required only for the `fasttext` method; default `null`) * `langid_languages`: limit detection to a list of possible languages (valid only for the `langid` method; default `null`) @@ -44,7 +44,15 @@ Parameters: Returned scores are the language identification confidence scores from a given identification method for the segments. The scores range from 0 to 1. In filtering, all values have to be greater than the minimum thresholds. Negative threshold can be used to skip filtering for a language. -See [langid.py](https://github.com/saffsd/langid.py) and -[pycld2](https://github.com/aboSamoor/pycld2) for the method-specific -options. A pretrained `fasttext` model can be downloaded from -[fasttext.cc/docs/en/language-identification.html](https://fasttext.cc/docs/en/language-identification.html). +Currently the following identification methods are supported: + +* `langid` (default) :cite:`lui-baldwin-2012-langid` + * See https://github.com/adbar/py3langid +* `lingua` + * See https://github.com/pemistahl/lingua-py +* `cld2` + * See https://github.com/CLD2Owners/cld2 + * Requires [installing optional libraries](../installation.md). +* `fasttext` :cite:`joulin-etal-2016-fasttext` and :cite:`joulin-etal-2017-bag` + * A pretrained model can be downloaded from [fasttext.cc/docs/en/language-identification.html](https://fasttext.cc/docs/en/language-identification.html). + * Requires [installing optional libraries](../installation.md). diff --git a/_sources/functions/downloading_and_selecting_data.md.txt b/_sources/functions/downloading_and_selecting_data.md.txt index cad15ae..95d35a0 100644 --- a/_sources/functions/downloading_and_selecting_data.md.txt +++ b/_sources/functions/downloading_and_selecting_data.md.txt @@ -11,11 +11,16 @@ Parameters: * `source_language`: language code for the source language * `target_language`: language code for the target language * `release`: version of the corpus in OPUS -* `preprocessing`: `raw` for untokenized and `xml` for tokenized segments +* `preprocessing`: `moses` or `raw` for untokenized and `xml` for tokenized segments * `src_output`: output file for source language * `tgt_output`: output file for target language * `suppress_prompts`: `false` (default) prompts user to confirm before download, `true` to download without prompting +The `moses` preprocessing type (available with `OpusTools` version +1.6.2 and above) is recommended for those corpora for which it +exists. The output is equivalent to `raw`, but in some cases it can +significantly reduce the amount of data downloaded in the process. + ## concatenate Concatenate two or more text files. diff --git a/_sources/installation.md.txt b/_sources/installation.md.txt index 92e86b3..c14bee3 100644 --- a/_sources/installation.md.txt +++ b/_sources/installation.md.txt @@ -12,20 +12,18 @@ Install from source: Note that all required libraries are not available to install via PyPI on Windows OS. On Linux and MacOS, it should work directly for Python -versions from 3.8 to 3.11. +versions from 3.8 to 3.12. ## Required libraries * beautifulsoup4 * opus-fast-mosestokenizer -* fasttext * graphviz -* langid +* py3langid * matplotlib * morfessor * OpusTools * pandas -* pycld2 * rapidfuzz * ruamel.yaml * regex @@ -41,24 +39,42 @@ See `setup.py` for possible version requirements. ## Optional libraries and tools +### FastText and PyCLD2 language identification + +The language identification libraries currently supported out-of-the-box +are [py3langid](https://github.com/adbar/py3langid) and +[lingua](https://github.com/pemistahl/lingua-py). The support for for +[PyCLD2](https://github.com/aboSamoor/pycld2) and +[FastText models](https://fasttext.cc/docs/en/language-identification.html) +have been changed to optional due to the lack of support especially +for newer Python versions. + +The PyCLD2 support can be installed automatically with pip by +including the extras `[pycld2]` or `[all]` (e.g. +`pip install opusfilter[pycld2]`). + +The support for FastText models can be installed automatically with +pip by including the extras `[fasttext]` or `[all]` (e.g. +`pip install opusfilter[fasttext]`). + ### Jieba and MeCab word segmentation For Chinese tokenization (word segmentation), you can use the [jieba](https://github.com/fxsjy/jieba) library. It can be installed automatically with pip by including the extras `[jieba]` or `[all]` -(e.g. `pip install opusfilter[all]`). +(e.g. `pip install opusfilter[jieba]`). For Japanese tokenization (word segmentation), you can use the [MeCab](https://github.com/SamuraiT/mecab-python3) library. It can be installed automatically with pip by including the extras `[mecab]` or `[all]` -(e.g. `pip install opusfilter[all]`). +(e.g. `pip install opusfilter[mecab]`). ### LASER sentence embeddings For using sentence embeddings filters, you need to install `laserembeddings` (https://github.com/yannvgn/laserembeddings). It can be installed automatically with pip by including the extras `[laser]` -or `[all]` (e.g. `pip install opusfilter[all]`). The package will also +or `[all]` (e.g. `pip install opusfilter[laser]`). The package will also require a number of additional libraries, including PyTorch, jieba, and MeCab. Note that you need also to download the prebuild models with `python -m laserembeddings download-models`. @@ -68,12 +84,12 @@ with `python -m laserembeddings download-models`. For using n-gram language model filters, you need to install the Python wrapper for VariKN (https://github.com/vsiivola/variKN). It can be installed automatically with pip by including the extras `[varikn]` -or `[all]` (e.g. `pip install opusfilter[all]`). +or `[all]` (e.g. `pip install opusfilter[varikn]`). ### Eflomal word alignment For using word alignment filters, you need to install elfomal (https://github.com/robertostling/eflomal). It can be installed automatically with pip by including the extras `[eflomal]` or `[all]` -(e.g. `pip install opusfilter[all]`). Note that you will need `Cython` +(e.g. `pip install opusfilter[eflomal]`). Note that you will need `Cython` for the installation. diff --git a/_static/documentation_options.js b/_static/documentation_options.js index 6ba7fc6..db3470d 100644 --- a/_static/documentation_options.js +++ b/_static/documentation_options.js @@ -1,6 +1,6 @@ var DOCUMENTATION_OPTIONS = { URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'), - VERSION: '3.1.0', + VERSION: '3.2.0', LANGUAGE: 'en', COLLAPSE_INDEX: false, BUILDER: 'html', diff --git a/automatic_configuration.html b/automatic_configuration.html index 7a0f801..515813f 100644 --- a/automatic_configuration.html +++ b/automatic_configuration.html @@ -4,7 +4,7 @@ -
pip
pylint opusfilter/
languages: expected languages (ISO639 language codes) for the segments
languages
id_method: language indentification method (langid for using the langid library, cld2 for using the cld2 library, or fasttext for using a fasttext model; the default is langid)
id_method
langid
cld2
id_method: language indentification method (langid, lingua, cld2, fasttext; default langid)
lingua
thresholds: minimum identification confidence score for the segments (a single float or a list of floats per language)
thresholds
fasttext_model_path: path for a fasttext model (required only for the fasttext method; default null)
fasttext_model_path
null
langid_languages: limit detection to a list of possible languages (valid only for the langid method; default null)
langid_languages
high
low
Returned scores are the language identification confidence scores from a given identification method for the segments. The scores range from 0 to 1. In filtering, all values have to be greater than the minimum thresholds. Negative threshold can be used to skip filtering for a language.
See langid.py and -pycld2 for the method-specific -options. A pretrained fasttext model can be downloaded from -fasttext.cc/docs/en/language-identification.html.
Currently the following identification methods are supported:
langid (default) :cite:lui-baldwin-2012-langid
lui-baldwin-2012-langid
See https://github.com/adbar/py3langid
See https://github.com/pemistahl/lingua-py
See https://github.com/CLD2Owners/cld2
Requires installing optional libraries.
fasttext :cite:joulin-etal-2016-fasttext and :cite:joulin-etal-2017-bag
joulin-etal-2016-fasttext
joulin-etal-2017-bag
A pretrained model can be downloaded from fasttext.cc/docs/en/language-identification.html.
target_language: language code for the target language
target_language
release: version of the corpus in OPUS
release
preprocessing: raw for untokenized and xml for tokenized segments
preprocessing
raw
xml
preprocessing: moses or raw for untokenized and xml for tokenized segments
src_output: output file for source language
src_output
tgt_output: output file for target language
tgt_output
suppress_prompts: false (default) prompts user to confirm before download, true to download without prompting
suppress_prompts
false
true
The moses preprocessing type (available with OpusTools version +1.6.2 and above) is recommended for those corpora for which it +exists. The output is equivalent to raw, but in some cases it can +significantly reduce the amount of data downloaded in the process.
OpusTools
beautifulsoup4
opus-fast-mosestokenizer
graphviz
matplotlib
morfessor
pandas
rapidfuzz
ruamel.yaml
regex
The language identification libraries currently supported out-of-the-box +are py3langid and +lingua. The support for for +PyCLD2 and +FastText models +have been changed to optional due to the lack of support especially +for newer Python versions.
The PyCLD2 support can be installed automatically with pip by +including the extras [pycld2] or [all] (e.g. +pip install opusfilter[pycld2]).
[pycld2]
[all]
pip install opusfilter[pycld2]
The support for FastText models can be installed automatically with +pip by including the extras [fasttext] or [all] (e.g. +pip install opusfilter[fasttext]).
[fasttext]
pip install opusfilter[fasttext]
For Chinese tokenization (word segmentation), you can use the jieba library. It can be installed automatically with pip by including the extras [jieba] or [all] -(e.g. pip install opusfilter[all]).
[jieba]
pip install opusfilter[all]
pip install opusfilter[jieba]
For Japanese tokenization (word segmentation), you can use the MeCab library. It can be installed automatically with pip by including the extras [mecab] or [all] -(e.g. pip install opusfilter[all]).
[mecab]
pip install opusfilter[mecab]
For using sentence embeddings filters, you need to install laserembeddings (https://github.com/yannvgn/laserembeddings). It can be installed automatically with pip by including the extras [laser] -or [all] (e.g. pip install opusfilter[all]). The package will also +or [all] (e.g. pip install opusfilter[laser]). The package will also require a number of additional libraries, including PyTorch, jieba, and MeCab. Note that you need also to download the prebuild models with python -m laserembeddings download-models.
laserembeddings
[laser]
pip install opusfilter[laser]
python -m laserembeddings download-models
[varikn]
pip install opusfilter[varikn]
For using word alignment filters, you need to install elfomal (https://github.com/robertostling/eflomal). It can be installed automatically with pip by including the extras [eflomal] or [all] -(e.g. pip install opusfilter[all]). Note that you will need Cython +(e.g. pip install opusfilter[eflomal]). Note that you will need Cython for the installation.
[eflomal]
Cython
pip install opusfilter[eflomal]