-
Notifications
You must be signed in to change notification settings - Fork 63
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
parser rewrite #47
parser rewrite #47
Conversation
while !parser.eof() { | ||
match is_at_stmt_start(parser) { | ||
Some(stmt) => { | ||
statement(parser, stmt); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
custom parsers can be added here later, and statement
would just be the fallback.
.collect() | ||
} | ||
|
||
fn custom_handlers(node: &Node) -> TokenStream { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is the only manual node-by-node implementation required for the parser.
- make whitespace lexer consume consecutive tokens as one - merge String, Integer etc Nodes into their parent - due to the aforementioned change, we wont have never-visited leaf nodes anymore and can skip the child check when closing leaf nodes - instead of searching for a token in the entire token range, we now only search for it in the next n+m non-whitespace tokens where n is the number of properties and m the number of tokens with just a single character (e.g. "(")
What kind of change does this PR introduce?
complete rewrite of the parser for increased
Highlights
scan()
method with a very simple custom lexer that extracts whitespaces.(...)
) are not tested against these conditions.