Skip to content

Commit

Permalink
update according to mbt file
Browse files Browse the repository at this point in the history
  • Loading branch information
yqyq-w committed May 20, 2024
1 parent 2f27093 commit b985a44
Show file tree
Hide file tree
Showing 3 changed files with 30 additions and 27 deletions.
54 changes: 28 additions & 26 deletions course11/course_en.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ headingDivider: 1
- Input is string/byte stream; and output is token stream
- Example:`"12 +678"` -> `[ Value(12), Plus, Value(678) ]`
- Typically done by applications of finite state machines

- Usually defined in a DSL and then automatically generates the program
- Lexical rules of arithmetic expressions
```abnf
Expand Down Expand Up @@ -93,10 +94,10 @@ headingDivider: 1
fn pchar(predicate : (Char) -> Bool) -> Lexer[Char] {
Lexer(fn(input) {
if input.length() > 0 && predicate(input[0]) {
Some((input[0], input.to_bytes().sub_string(1, input.length() - 1)))
Some((input[0], input.to_bytes().sub_string(2, input.length() * 2 - 2)))
} else {
None
} }) }
} },) }
```
- For example:
```moonbit
Expand Down Expand Up @@ -136,10 +137,10 @@ let whitespace : Lexer[Char] = pchar(fn{ ch => ch == ' ' })
```moonbit
fn map[I, O](self : Lexer[I], f : (I) -> O) -> Lexer[O] {
Lexer(fn(input) {
// Non-empty value v is in Some(v), empty value None is directly returned
// Non-empty value v is in Some(v), empty value None is directly returned
let (value, rest) = self.parse(input)?
Some((f(value), rest))
}) }
},) }
```
- Parse the operators and parentheses, and map them to corresponding enum values.
```moonbit expr
Expand All @@ -162,17 +163,17 @@ fn and[V1, V2](self : Lexer[V1], parser2 : Lexer[V2]) -> Lexer[(V1, V2)] {
let (value, rest) = self.parse(input)?
let (value2, rest2) = parser2.parse(rest)?
Some(((value, value2), rest2))
}) }
},) }
```

- Parse `a`, and if it fails, parse `b`.
```moonbit
fn or[Value](self : Lexer[Value], parser2 : Lexer[Value]) -> Lexer[Value] {
Lexer(fn (input) {
Lexer(fn(input) {
match self.parse(input) {
None => parser2.parse(input)
Some(_) as result => result
} }) }
} },) }
```

# Parser Combinator
Expand All @@ -188,19 +189,19 @@ fn reverse_list[X](list : List[X]) -> List[X] {
go(Nil, list)
}
fn many[Value](self: Lexer[Value]) -> Lexer[List[Value]] {
fn many[Value](self : Lexer[Value]) -> Lexer[List[Value]] {
Lexer(fn(input) {
let mut rest = input
let mut cumul = List::Nil
while true {
match self.parse(rest) {
None => break
Some((value, new_rest)) => {
rest = new_rest
cumul = Cons(value, cumul) // Parsing succeeds, add the content
} } }
Some((reverse_list(cumul), rest)) // ⚠️List is a stack, reverse it for the correct order
}) }
let mut rest = input
let mut cumul = List::Nil
while true {
match self.parse(rest) {
None => break
Some((value, new_rest)) => {
rest = new_rest
cumul = Cons(value, cumul) // Parsing succeeds, add the content
} } }
Some((reverse_list(cumul), rest)) // ⚠️List is a stack, reverse it for the correct order
},) }
```

# Lexical Analysis
Expand All @@ -215,14 +216,15 @@ fn fold_left_list[A, B](list : List[A], f : (B, A) -> B, b : B) -> B {
// Convert characters to integers via encoding
let zero: Lexer[Int] =
pchar(fn{ ch => ch == '0' }).map(fn{ _ => 0 })
pchar(fn { ch => ch == '0' }).map(fn { _ => 0 })
let one_to_nine: Lexer[Int] =
pchar(fn{ ch => ch.to_int() >= 0x31 && ch.to_int() <= 0x39 }).map(fn { ch => ch.to_int() - 0x30 })
pchar(fn { ch => ch.to_int() >= 0x31 && ch.to_int() <= 0x39 },).map(fn { ch => ch.to_int() - 0x30 })
let zero_to_nine: Lexer[Int] =
pchar(fn{ ch => ch.to_int() >= 0x30 && ch.to_int() <= 0x39 }).map(fn { ch => ch.to_int() - 0x30 })
pchar(fn { ch => ch.to_int() >= 0x30 && ch.to_int() <= 0x39 },).map(fn { ch => ch.to_int() - 0x30 })
// number = %x30 / (%x31-39) *(%x30-39)
let value: Lexer[Token] =
zero.or(
one_to_nine.and(zero_to_nine.many()) // (Int, List[Int])
.map(fn{ // 1 2 3 -> 1 * 100 + 2 * 10 + 3
Expand All @@ -237,10 +239,10 @@ let value: Lexer[Token] =
- There may exist whitespaces between tokens

```moonbit
let tokens: Lexer[List[Token]] =
number.or(symbol).and(whitespace.many())
.map(fn { (symbols, _) => symbols }) // Ignore whitespaces
.many()
let tokens : Lexer[List[Token]] =
value.or(symbol).and(whitespace.many())
.map(fn { (symbols, _) => symbols },) // Ignore whitespaces
.many()
fn init {
debug(tokens.parse("-10123-+-523 103 ( 5) ) "))
Expand Down
1 change: 1 addition & 0 deletions course11/lec11.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ headingDivider: 1
- 输出:单词流
- 例如:`"12 +678"` -> `[ Value(12), Plus, Value(678) ]`
- 通常可以通过有限状态自动机完成

- 一般用领域特定语言定义后,由软件自动生成程序
- 算术表达式的词法定义
```abnf
Expand Down
2 changes: 1 addition & 1 deletion course11/lec11_script.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# 现代编程思想:案例:语法解析器

大家好,欢迎来到由IDEA研究院基础软件中心为大家带来的现代编程思想公开课。经过了十节课的学习,相信大家已经对月兔语言有了初步的了解,也可以尝试挑战一些更为复杂的程序。我们将为大家展示一些有趣的案例。这届课我们展示的是语法解析器
大家好,欢迎来到由IDEA研究院基础软件中心为大家带来的现代编程思想公开课。经过了十节课的学习,相信大家已经对月兔语言有了初步的了解,也可以尝试挑战一些更为复杂的程序。我们将为大家展示一些有趣的案例。这节课我们展示的是语法解析器

世界上有着种类繁多的语言,以及编程语言,和各类其他的符号语言等等。我们今天以自然数的四则运算为例。对于一个字符串,首先我们要做的就是分解为单词的列表。例如`"(1+ 5) * 7 / 2"`,我们就可以根据词法分解为左括号、1、加号、5、右括号、乘号、7、除号、2。显然,1和括号、加号之间虽然没有空格,应当被拆分为三个单词,这样的规则就是词法。这个步骤也就是词法分析。
在有了单词流之后,我们要根据语法将它组合为一个抽象语法树。例如,首先应该有1和5的和,再由这个和与7进行乘法,最后,再由这个乘积与2进行除法,而不是1加上5与7的乘积,因为这不符合语法。这个步骤就是语法分析。
Expand Down

0 comments on commit b985a44

Please sign in to comment.