Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add guidance on handling comments in languages with significant indentation #1502

Merged
merged 6 commits into from
Jan 5, 2025
Merged

Add guidance on handling comments in languages with significant indentation #1502

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
eb74454
add guidance on handling comments
nchammas Jan 2, 2025
264d6e9
add blurb on building docs
nchammas Jan 3, 2025
df66301
fix markup now that i can see the html
nchammas Jan 3, 2025
20494d2
ignore strange file that appeared after running `make html`
nchammas Jan 3, 2025
1b7a2dd
significant whitespace -> significant indentation
nchammas Jan 3, 2025
c3b83f0
fix sphinx config
nchammas Jan 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,4 @@ tags
/build
docs/_build
docs/examples
docs/sg_execution_times.rst
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dunno if this shows up for you too after running make html, but I assume it's meant to be ignored.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think I get this. Not sure why you are.

Copy link
Contributor Author

@nchammas nchammas Jan 3, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could it be the version of Sphinx you are running? This is what I have:

$ pip list | grep -i sphinx
Sphinx                        7.3.7
sphinx-gallery                0.18.0
sphinx-markdown-tables        0.0.17
sphinx-rtd-theme              3.0.2
sphinxcontrib-applehelp       2.0.0
sphinxcontrib-devhelp         2.0.0
sphinxcontrib-htmlhelp        2.1.0
sphinxcontrib-jquery          4.1
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          2.0.0
sphinxcontrib-serializinghtml 2.0.0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dev requirements are open ended, so if you haven't updated them in a while you are likely running a much older version of Sphinx than I am (since I just installed them today).

If you're courageous, I would try pip install -U -r docs/requirements.txt and then see if you get the same behavior that I am seeing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you're right, it gets generated by sphinx-gallery. I think it's fine to just ignore it.

2 changes: 1 addition & 1 deletion docs/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
language = 'en'

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
Expand Down
12 changes: 12 additions & 0 deletions docs/how_to_develop.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,3 +65,15 @@ Another way to run the tests is using setup.py:
```bash
python setup.py test
```

## Building the Documentation

To build the documentation:

```sh
cd docs/
pip install -r requirements.txt
make html
```

To review the result, open the built HTML files under `_build/html/` in your browser.
11 changes: 11 additions & 0 deletions docs/recipes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ Prints out:

*Note: We don't have to return a token, because comments are ignored*


## CollapseAmbiguities

Parsing ambiguous texts with earley and `ambiguity='explicit'` produces a single tree with `_ambig` nodes to mark where the ambiguity occurred.
Expand Down Expand Up @@ -193,3 +194,13 @@ def parse_with_progress(parser: Lark, text: str, start=None):
```

Keep in mind that this implementation relies on the `InteractiveParser` and, therefore, only works with the `LALR(1)` parser, and not `Earley`.


## Parsing a Language with Significant Indentation

If your grammar needs to support significant indentation (e.g. Python, YAML), you will need to use
the `Indenter` class. Take a look at the [indented tree example][indent] as well as the
[Python grammar][python] for inspiration.

[indent]: examples/indented_tree.html
[python]: https://github.com/lark-parser/lark/blob/master/lark/grammars/python.lark
27 changes: 17 additions & 10 deletions examples/indented_tree.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,28 +3,34 @@
===================

A demonstration of parsing indentation (“whitespace significant” language)
and the usage of the Indenter class.
and the usage of the ``Indenter`` class.

Since indentation is context-sensitive, a postlex stage is introduced to
manufacture INDENT/DEDENT tokens.
manufacture ``INDENT``/``DEDENT`` tokens.

It is crucial for the indenter that the NL_type matches
the spaces (and tabs) after the newline.
It is crucial for the indenter that the ``NL_type`` matches the spaces (and
tabs) after the newline.

If your whitespace-significant grammar supports comments, then ``NL_type``
must match those comments too. Otherwise, comments that appear in the middle
of a line will `confuse Lark`_.

.. _`confuse Lark`: https://github.com/lark-parser/lark/issues/863
"""
from lark import Lark
from lark.indenter import Indenter

tree_grammar = r"""
?start: _NL* tree

tree: NAME _NL [_INDENT tree+ _DEDENT]

%import common.CNAME -> NAME
%import common.WS_INLINE
%declare _INDENT _DEDENT
%import common.SH_COMMENT
%ignore WS_INLINE
%ignore SH_COMMENT
%declare _INDENT _DEDENT

_NL: /(\r?\n[\t ]*)+/
?start: _NL* tree
tree: NAME _NL [_INDENT tree+ _DEDENT]
_NL: (/\r?\n[\t ]*/ | SH_COMMENT)+
"""

class TreeIndenter(Indenter):
nchammas marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -39,6 +45,7 @@ class TreeIndenter(Indenter):

test_tree = """
a
# check this comment out
b
c
d
Expand Down
Loading