PLY - 返回多个标记

Question

据我所知，解析Python源代码的技术是：

• 当当前行的缩进级别低于上一行时，产生DEDENT。如果它正在关闭多个INDENTs，则会产生多个DEDENTs。
• 当到达输入的结尾时，如果存在未闭合的INDENT，则产生DEDENT(s)。

现在，使用PLY：

• 如何从t_definition返回多个标记？
• 如何制作一个在EOF到达时调用的t_definition？简单的 \Z 不起作用--PLY抱怨它匹配了空字符串。

- Ron

Answer 1

据我所知，PLY没有实现推送解析器接口，这是您使用bison最容易解决此问题的方式。然而，非常容易注入自己的词法分析器包装器，它可以处理缩进标记队列。

一个最小的词法分析器实现需要实现一个返回具有"type"和"value"属性的对象的"token()"方法。(如果您的解析器使用它，那么您也需要，但我不会在这里担心这个问题。)

现在，假设底层(PLY生成的)词法分析器产生"NEWLINE"令牌，其值是换行符后面前导空格的长度。如果某些行不参与INDENT/DEDENT算法，则应该对这些行抑制"NEWLINE"；我们在这里不考虑这种情况。一个简单的示例词法分析器函数(仅适用于空格而不是制表符)可能是：

# This function doesn't handle tabs. Beware!
def t_NEWLINE(self, t):
  r'\n(?:\s*(?:[#].*)?\n)*\s*'
  t.value = len(t.value) - 1 - t.value.rfind('\n')
  return t

现在我们需要用一个包装器来处理缩进，将由PLY生成的词法分析器进行包装：

# WARNING:
# This code hasn't been tested much and it also may be inefficient
# and/or inexact. It doesn't do python-style tab handling. Etc. etc.

from collections import namedtuple, deque

# These are the tokens. We only generate one of each here. If
# we used lineno or didn't trust the parser to not mess with the
# token, we could generate a new one each time.
IndentToken = namedtuple('Token', 'type value')
dedent = IndentToken('DEDENT', None)
indent = IndentToken('INDENT', None)
newline= IndentToken('NEWLINE', None)

class IndentWrapper(object):

  def __init__(self, lexer):
    """Create a new wrapper given the lexer which is being wrapped"""
    self.lexer = lexer
    self.indent_stack = [0]
    # A queue is overkill for this case, but it's simple.
    self.token_queue = deque()
    # This is just in case the ply-generated lexer cannot be called again
    # after it returns None.
    self.eof_reached = False

  def token(self):
    """Return the next token, or None if end of input has been reached"""
    # Do we have any queued tokens?
    if self.token_queue:
      return self.token_queue.popleft()
    # Are we done?
    if self.eof_reached:
      return None
    # Get a token
    t = self.lexer.token()
    if t is None:
      # At end of input, we might need to send some dedents
      self.eof_reached = True
      if len(self.indent_stack) > 1:
        t = dedent
        for i in range(len(self.indent_stack) - 1):
          self.token_queue.append(dedent)
        self.indent_stack = [0]
    elif t.type == "NEWLINE":
      # The NEWLINE token includes the amount of leading whitespace.
      # Fabricate indent or dedents as/if necessary and queue them.
      if t.value > self.indent_stack[-1]:
        self.indent_stack.append(t.value)
        self.token_queue.append(indent)
      else:
        while t.value < self.indent_stack[-1]:
          self.indent_stack.pop()
          self.token_queue.append(dedent)
        if t.value != self.indent_stack[-1]:
          raise IndentError # Or however you indicate errors
    return t

- rici