Mypy fixes, and improvements to the InteractiveParser API

What's Changed

• Add test cases for tree templates by @plannigan in #1096
• 🖊 Fix Typo: plural "options" instead of singular "option" by @hf-kklein in #1101
• PEP 8: Minor Code Style Improvements by @hf-kklein in #1102
• Add Code Style Section to Contribution Guide by @hf-kklein in #1107
• Fix MyPy Warnings in lark/tools/init.py by @hf-kklein in #1100
• rename n to child when iterating over children by @hf-kklein in #1110
• specify ignored mypy error by using type: ignore[error] in lark/tree.py and lark/utils.py by @hf-kklein in #1099
• Add py.typed to package_data of lark package by @hf-kklein in #1109
• InteractiveParser: Added iter_parse() method, for easier instrumentation by @erezsh in #1111

New Contributors

• @hf-kklein made their first contribution in #1101

Full Changelog: 1.1.0...1.1.1

Version 1.1.0

• Better support for typing and mypy. Includes generic tree typing (Thanks @plannigan!)

• Improvements to python.lark (walrus operator, slashes in function params, and more). Now parses the entire Python 3.10 lib successfully

• Bugfixes:

  • Transformer.__default__ not called in tree-less LALR mode (Issue #1029)
  • v_args failed to apply to class under standalone parser (Issue #1059)
  • maybe_placeholders incorrectly accumulated params when it encountered the | operator (Issue #1078)

Version 1.0! Big release with breaking changes

Over the last few years, Lark has grown to become a comprehensive toolkit for parsing structured text.

Today, I'm happy to announce the long anticipated version 1.0 of Lark, marking the API as stable.

We've made quite a few breaking changes, in order to achieve congruous API with as little "gotchas" as possible. Upgrading to version 1.0 might require a few changes to your project.

Breaking changes

• Dropped Python 2 support! Lark now only supports Python 3.6 and up.

• Install lark using pip install lark (instead of lark-parser ).

• maybe_placeholders is now True by default.

• Renamed TraditionalLexer to BasicLexer, and 'standard' lexer option to 'basic'.

• Default priority is now 0, for both terminals and rules (used to be 1 for terminals).

• Discard mechanism is now done by returning Discard, instead of raising it as an exception.

• use_accepts in UnexpectedInput.match_examples() is now True by default.

• v_args(meta=True) now gives meta as the first argument. i.e. (meta, children).

Improvments

• Better type annotations
• Support for terminal priorities for dynamic Earley
• Python3 grammar is now officially supported, and can be used via %import python (...)
• New experimental feature: Tree Templates
• Various bugfixes

Acknowledgements

Many thanks to all our contributors and donors, who made this release possible. Special thanks goes to -

• @MegaIng, for innumerous features, bugfixes, and code-reviews.
• @chanicpanic, for his immense and continual contributions to the Earley parser, and for helping with the v1.0 effort.
• @erezsh, for being myself.

0.12.0 - Several improvements + Announcements

Announcements

• This is likely to be the last major release that supports Python 2 !

We are now working on a Python3.6+ only v1.0 branch, which will soon become the default. See the work in progress: #925

• We also have a new online IDE! Check it out here: https://lark-parser.github.io/ide

• Lark can now generate standalone Javascript parsers! Check it out here: https://github.com/lark-parser/Lark.js (still in beta)

Changes

• Using rule repeat (~ syntax) is now much much faster for large numbers, thanks to @MegaIng

• Bugfix for the propagate_positions option. Added option value propagate_positions='ignore_ws'.

• Fixed reconstructor for when keep_all_tokens=True

• Added merge_transformers (Thanks Robin!)

• Many minor bugfixes, and improvements to code and docs

InteractiveParser & Better Cache

Cache

• Lark now tracks changes in imported grammars (%import), and updates the cache if necessary
• Added support for atomicwrites, for multiprocess caching and crash recovery

InteractiveParser

• Now an official interface (renamed from Puppet)
• Added Lark.parse_interactive() for starting the parser in interactive mode

Other

• Added ast_utils, to assist in tranforming lark.Tree into a customized AST.

• Better docs

• Bugfixes

Notification: Support for Python 2 is ending

In the near future, Lark will drop support for Python 2. We will continue to develop for Python 3.6+ only, which will simplify the code and ease development.

Old releases (including this one) will still work, of course, and should be stable enough to accompany the remaining Python 2 users into the sunset.

If you have any objections, feel free to voice them here: #874

Thanks for everyone who helped make Lark better!

Better grammar re-use, and tons of improvements

New Features:

• Better grammar re-use with the %override and %extend statements, which allow to rewrite and extend imported rules and tokens, similarly to class inheritance. (See this example: https://github.com/lark-parser/lark/blob/master/examples/advanced/extend_python.py)

Improvements

• Indenter now throws DedentError instead of AssertionError

• Improved the Python3 grammar, now works with reconstructor. (See this example: https://github.com/lark-parser/lark/blob/master/examples/advanced/reconstruct_python.py)

• Lots of refactoring for a better tomorrow.

• rule/terminals names can now be in unicode. (thanks @julienmalard)

• Better errors.

• Better type hints.

• lark.lark is now part of the standard library.

• Earley:

  • Now works with match_examples()
  • Now supports a custom lexer
  • Better handling of ignored terminals
  • Faster forest visiting, and a few edge-case bugfixes (thanks @chanicpanic)

Other

• Lark now accepts funding as a member of Github Sponsors! See here: https://github.com/sponsors/lark-parser

Small bugfix in the Earley parser

0.11.1

Version bump to 0.11.1

Many bugfixes and improvements

• LALR parser

  • The LALR parser now supports priority in rules, as a way to resolve collision errors

  • Improvements to the standalone tool, including more command-line options, like optional compression for the json data.

  • Improvements to the puppet error handling interface

  • Better error reporting on LALR collisions

• Bugfixes in Earley

Misc

• Added support for syntax highlighting in Atom

• Fixes and improvements for the cache option. cache=True now uses a temporary directory instead of working directory.

• Lark can now be imported directly from a zip (See: ed5c8ec)

• Added more terminals to the grammar library (available for %import).

• Nearley tools now supports case insensitive strings

• Deprecated some interfaces

• Improvements to docs, stubs, and various bugfixes

Thanks to @MegaIng for helping with Lark's maintenance, and to @ldbo, @chanicpanic, @michael-k, @ThatXliner and everyone else for their help and contributions.

New SPPF interface, new docs, and a long list of improvements

• Complete overhaul of documentation. Now using sphinx to generate API docs from docstrings. (commit 0664cbd. Thank you @chsasank !)

  • Many improvements and additions to documentation

• New and friendlier Earley SPPF interface! (commit 555b268. Thank you @chanicpanic !)

  • Added the ambiguity='forest' option. Added ForestTransformer and TreeForestTranformer.

  • Various Bugfixes to improve the handling of ambiguous results.

  • Read the docs here: https://lark-parser.readthedocs.io/en/latest/forest.html

• New Vim syntax highlighting for Lark (https://github.com/lark-parser/vim-lark-syntax Thank you @omega16 !)

• Lark now loads faster from cache (commit 7dc0017)

• Terminals can now be composed of regexps and strings with different flags, if using Python 3.6+ (commit e6fc3c9)

• Added support for parsing byte-strings, with the use_bytes flag (commit 9ee8428).

• UnexpectedToken exception now has the accepts attribute, which contains a list of terminals that would be accepted by the parser instead (in addition to the expects attribute, which is guided by the lexer and may include terminals that won't be accepted by the parser) (commit a7bcd0b)

• Allow multiline regexes with the x flag (commit 9923987)

• Lark no longer uses the default logger. Instead uses lark.LOGGER. (commit 7010f96)

• Lark now notifies on unused terminals/rules through logging.debug.

• Standalone generator now creates smaller files (without comments and docstrings). Also undergone various fixes. (commit bf2d9bf)

• Wheel distribution due to (somewhat) popular demand.

• Lots of small bugfixes and improvements!

Many thanks to @MegaIng for his continued work on many of these new features and fixes, and to everyone else who contributed to Lark and helped make it even better.

Better regexps, and new error handling

• Added error handling to LALR!

  • on_error option to Lark.parse().
    Read here: https://lark-parser.readthedocs.io/en/latest/classes/#larkparse
  • Parser now comes with a puppet for advanced error handling.
    Read here: https://lark-parser.readthedocs.io/en/latest/classes/#parserpuppet

• Support for better regexps with the regex module, when using Lark(..., regex=True)
  Read here: https://lark-parser.readthedocs.io/en/latest/classes/#using-unicode-character-classes-with-regex