我正在尝试使用lark实现一个记录定义DSL。它基于缩进,这使得事情变得有点复杂。
Lark是一个很棒的工具,但我面临一些困难。
这是我正在实现的DSL的片段:
record Order :
"""Order record documentation
should have arbitrary size"""
field1 Int
field2 Datetime:
"""Attributes should also have
multiline documentation"""
field3 String "inline documentation also works"
这里使用的语法是:
?start: (_NEWLINE | redorddef)*
simple_type: NAME
multiline_doc: MULTILINE_STRING _NEWLINE
inline_doc: INLINE_STRING
?element_doc: ":" _NEWLINE _INDENT multiline_doc _DEDENT | inline_doc
attribute_name: NAME
attribute_simple_type: attribute_name simple_type [element_doc] _NEWLINE
attributes: attribute_simple_type+
_recordbody: _NEWLINE _INDENT [multiline_doc] attributes _DEDENT
redorddef: "record" NAME ":" _recordbody
MULTILINE_STRING: /"""([^"\\]*(\\.[^"\\]*)*)"""/
INLINE_STRING: /"([^"\\]*(\\.[^"\\]*)*)"/
_WS_INLINE: (" "|/\t/)+
COMMENT: /#[^\n]*/
_NEWLINE: ( /\r?\n[\t ]*/ | COMMENT )+
%import common.CNAME -> NAME
%import common.INT
%ignore /[\t \f]+/ // WS
%ignore /\\[\t \f]*\r?\n/ // LINE_CONT
%ignore COMMENT
%declare _INDENT _DEDENT
它可以很好地处理多行字符串文档的记录定义,可以很好地处理内联属性定义,但对于属性多行字符串文档则无法正常工作。
我用来执行的代码是这样的:
import sys
import pprint
from pathlib import Path
from lark import Lark, UnexpectedInput
from lark.indenter import Indenter
scheman_data_works = '''
record Order :
"""Order record documentation
should have arbitrary size"""
field1 Int
# field2 Datetime:
# """Attributes should also have
# multiline documentation"""
field3 String "inline documentation also works"
'''
scheman_data_wrong = '''
record Order :
"""Order record documentation
should have arbitrary size"""
field1 Int
field2 Datetime:
"""Attributes should also have
multiline documentation"""
field3 String "inline documentation also works"
'''
grammar = r'''
?start: (_NEWLINE | redorddef)*
simple_type: NAME
multiline_doc: MULTILINE_STRING _NEWLINE
inline_doc: INLINE_STRING
?element_doc: ":" _NEWLINE _INDENT multiline_doc _DEDENT | inline_doc
attribute_name: NAME
attribute_simple_type: attribute_name simple_type [element_doc] _NEWLINE
attributes: attribute_simple_type+
_recordbody: _NEWLINE _INDENT [multiline_doc] attributes _DEDENT
redorddef: "record" NAME ":" _recordbody
MULTILINE_STRING: /"""([^"\\]*(\\.[^"\\]*)*)"""/
INLINE_STRING: /"([^"\\]*(\\.[^"\\]*)*)"/
_WS_INLINE: (" "|/\t/)+
COMMENT: /#[^\n]*/
_NEWLINE: ( /\r?\n[\t ]*/ | COMMENT )+
%import common.CNAME -> NAME
%import common.INT
%ignore /[\t \f]+/ // WS
%ignore /\\[\t \f]*\r?\n/ // LINE_CONT
%ignore COMMENT
%declare _INDENT _DEDENT
'''
class SchemanIndenter(Indenter):
NL_type = '_NEWLINE'
OPEN_PAREN_types = ['LPAR', 'LSQB', 'LBRACE']
CLOSE_PAREN_types = ['RPAR', 'RSQB', 'RBRACE']
INDENT_type = '_INDENT'
DEDENT_type = '_DEDENT'
tab_len = 4
scheman_parser = Lark(grammar, parser='lalr', postlex=SchemanIndenter())
print(scheman_parser.parse(scheman_data_works).pretty())
print("\n\n")
print(scheman_parser.parse(scheman_data_wrong).pretty())
结果是:
redorddef
Order
multiline_doc """Order record documentation
should have arbitrary size"""
attributes
attribute_simple_type
attribute_name field1
simple_type Int
attribute_simple_type
attribute_name field3
simple_type String
inline_doc "inline documentation also works"
Traceback (most recent call last):
File "schema_parser.py", line 83, in <module>
print(scheman_parser.parse(scheman_data_wrong).pretty())
File "/Users/branquif/Dropbox/swf_projects/schema-manager/.venv/lib/python3.7/site-packages/lark/lark.py", line 228, in parse
return self.parser.parse(text)
File "/Users/branquif/Dropbox/swf_projects/schema-manager/.venv/lib/python3.7/site-packages/lark/parser_frontends.py", line 38, in parse
return self.parser.parse(token_stream, *[sps] if sps is not NotImplemented else [])
File "/Users/branquif/Dropbox/swf_projects/schema-manager/.venv/lib/python3.7/site-packages/lark/parsers/lalr_parser.py", line 68, in parse
for token in stream:
File "/Users/branquif/Dropbox/swf_projects/schema-manager/.venv/lib/python3.7/site-packages/lark/indenter.py", line 31, in process
for token in stream:
File "/Users/branquif/Dropbox/swf_projects/schema-manager/.venv/lib/python3.7/site-packages/lark/lexer.py", line 319, in lex
for x in l.lex(stream, self.root_lexer.newline_types, self.root_lexer.ignore_types):
File "/Users/branquif/Dropbox/swf_projects/schema-manager/.venv/lib/python3.7/site-packages/lark/lexer.py", line 167, in lex
raise UnexpectedCharacters(stream, line_ctr.char_pos, line_ctr.line, line_ctr.column, state=self.state)
lark.exceptions.UnexpectedCharacters: No terminal defined for 'f' at line 11 col 2
field3 String "inline documentation also
^
我理解缩进语法更为复杂,lark似乎能够简化这一过程,但我在这里找不到错误。
另外,我也尝试了pyparsing,但在相同的情况下没有成功,而且由于可能需要大量代码,转移到PLY会很困难。
field2 DateTime
后面应该加上:
吗? - PaulMcG