使用nbconvert运行包含内联Markdown的Jupyter笔记本。

9
我有一个 Jupyter 笔记本,其中包含类似于以下内容的 Markdown 单元格中的 Python 变量:
代码单元格:
x = 10

标记单元格:

The value of x is {{x}}.

这段文字的意思是:“IPython-notebook-extension Python Markdown”可以让我在笔记本中按下shift-enter键执行markdown单元格后动态显示这些变量。
“markdown单元格:”
The value of x is 10.

我想以编程方式执行笔记本中的所有单元格,并使用类似以下方式将它们保存到新笔记本中:
import nbformat
from nbconvert.preprocessors import ExecutePreprocessor

with open('report.ipynb') as f:
    nb = nbformat.read(f, as_version=4)
        ep = ExecutePreprocessor(timeout=600, kernel_name='python3')
        ep.preprocess(nb, {})
with open('report_executed.ipynb', 'wt') as f:
    nbformat.write(nb, f)

这将执行代码单元格,但不会执行标记单元格。它们仍然看起来像这样:
The value of x is {{x}}.

我认为问题在于笔记本没有受到信任。有没有办法告诉ExecutePreprocessor去信任这个笔记本?还有其他的方法可以编程执行一个包含Python变量的Markdown单元格的笔记本吗?

1
将变量放入Markdown单元格的笔记本扩展程序可能不会影响使用ExecutePreprocessor运行笔记本。 - Thomas K
3个回答

7
执行预处理器 仅查看代码单元格,因此您的标记单元格完全未受影响。要进行标记处理,您需要Python Markdown预处理器,正如您所述。
不幸的是,Python Markdown预处理器系统仅在实时笔记本中执行代码,它通过修改涉及呈现单元格的javascript来完成。修改将执行代码片段的结果存储在单元格元数据中。 PyMarkdownPreprocessor类(在pre_pymarkdown.py中)旨在与nbconvert一起使用,nbconvert操作的是先在实时笔记本设置中呈现过的笔记本。它处理标记单元格,将{{}}模式替换为元数据中存储的值。
在您的情况下,您没有实时笔记本元数据。我曾经遇到过类似的问题,并通过编写自己的执行预处理器来解决它,该预处理器还包括处理markdown单元格的逻辑:
from nbconvert.preprocessors import ExecutePreprocessor, Preprocessor
import nbformat, nbconvert
from textwrap import dedent

class ExecuteCodeMarkdownPreprocessor(ExecutePreprocessor):

    def __init__(self, **kw):
        self.sections = {'default': True} # maps section ID to true or false
        self.EmptyCell = nbformat.v4.nbbase.new_raw_cell("")

        return super().__init__(**kw)

    def preprocess_cell(self, cell, resources, cell_index):
        """
        Executes a single code cell. See base.py for details.
        To execute all cells see :meth:`preprocess`.
        """

        if cell.cell_type not in ['code','markdown']:
            return cell, resources

        if cell.cell_type == 'code':
            # Do code stuff
            return self.preprocess_code_cell(cell, resources, cell_index)

        elif cell.cell_type == 'markdown':
            # Do markdown stuff
            return self.preprocess_markdown_cell(cell, resources, cell_index)
        else:
            # Don't do anything
            return cell, resources

    def preprocess_code_cell(self, cell, resources, cell_index):
        ''' Process code cell.
        '''
        outputs = self.run_cell(cell)
        cell.outputs = outputs

        if not self.allow_errors:
            for out in outputs:
                if out.output_type == 'error':
                    pattern = u"""\
                        An error occurred while executing the following cell:
                        ------------------
                        {cell.source}
                        ------------------
                        {out.ename}: {out.evalue}
                        """
                    msg = dedent(pattern).format(out=out, cell=cell)
                    raise nbconvert.preprocessors.execute.CellExecutionError(msg)

        return cell, resources

    def preprocess_markdown_cell(self, cell, resources, cell_index):
        # Find and execute snippets of code
        cell['metadata']['variables'] = {}
        for m in re.finditer("{{(.*?)}}", cell.source):
            # Execute code
            fakecell = nbformat.v4.nbbase.new_code_cell(m.group(1))
            fakecell, resources = self.preprocess_code_cell(fakecell, resources, cell_index)

            # Output found in cell.outputs
            # Put output in cell['metadata']['variables']
            for output in fakecell.outputs:
                html = self.convert_output_to_html(output)
                if html is not None:
                    cell['metadata']['variables'][fakecell.source] = html
                    break
        return cell, resources

    def convert_output_to_html(self, output):
        '''Convert IOpub output to HTML

        See https://github.com/ipython-contrib/IPython-notebook-extensions/blob/master/nbextensions/usability/python-markdown/main.js
        '''
        if output['output_type'] == 'error':
            text = '**' + output.ename + '**: ' + output.evalue; 
            return text
        elif output.output_type == 'execute_result' or output.output_type == 'display_data':
            data = output.data
            if 'text/latex' in data:
                html = data['text/latex']
                return html
            elif 'image/svg+xml' in data:
                # Not supported
                #var svg = ul['image/svg+xml'];
                #/* embed SVG in an <img> tag, still get eaten by sanitizer... */
                #svg = btoa(svg);
                #html = '<img src="data:image/svg+xml;base64,' + svg + '"/>';
                return None
            elif 'image/jpeg' in data:
                jpeg = data['image/jpeg']
                html = '<img src="data:image/jpeg;base64,' + jpeg + '"/>'
                return html
            elif 'image/png' in data:
                png = data['image/png']
                html = '<img src="data:image/png;base64,' + png + '"/>'
                return html
            elif 'text/markdown' in data:
                text = data['text/markdown']
                return text
            elif 'text/html' in data:
                html = data['text/html']
                return html
            elif 'text/plain' in data:
                text = data['text/plain']
                # Strip <p> and </p> tags
                # Strip quotes
                # html.match(/<p>([\s\S]*?)<\/p>/)[1]
                text = re.sub(r'<p>([\s\S]*?)<\/p>', r'\1', text)
                text = re.sub(r"'([\s\S]*?)'",r'\1', text)
                return text
            else:
            # Some tag we don't support
                return None
        else:
            return None

您可以使用类似于您发布的代码的逻辑处理笔记本电脑:
import nbformat
from nbconvert.preprocessors import ExecutePreprocessor
import ExecuteCodeMarkdownPreprocessor # from wherever you put it
import PyMarkdownPreprocessor # from pre_pymarkdown.py

with open('report.ipynb') as f:
    nb = nbformat.read(f, as_version=4)
    ep = ExecuteCodeMarkdownPreprocessor(timeout=600, kernel_name='python3')
    ep.preprocess(nb, {})
    pymk = PyMarkdownPreprocessor()
    pymk.preprocess(nb, {})

with open('report_executed.ipynb', 'wt') as f:
    nbformat.write(nb, f)

请注意,通过包含Python Markdown预处理,您得到的笔记本文件将不再在Markdown单元格中具有 {{}} 语法-Markdown将具有静态内容。如果结果笔记本的接收者更改代码并再次执行,则Markdown将不会更新。但是,如果您导出为不同格式(例如HTML),则确实需要使用静态内容替换 {{}} 语法。

3

更新2021年06月29日

由于nbconvert使用nbclient调用的更改(https://github.com/jupyter/nbconvert/commit/e7bf8350435a66cc50faf29ff12df492be5d7f57#diff-bee04d71b1dfc0202a0239b1513fd81d983edc339a9734ca4f4813276feed032),这篇文章需要更新。由于run_cell不再可用,因此需要修改代码和标记单元格处理方式。以下代码可以正常运行:

import nbformat
from nbconvert.preprocessors import ExecutePreprocessor
from nbconvert.preprocessors import execute
import re

# Taken from:
# https://dev59.com/b5Tfa4cB1Zd3GeqPVMAd modified to avoid using superseeded run_cell calls.

class ExecuteCodeMarkdownPreprocessor(ExecutePreprocessor):

    def __init__(self, **kw):
        self.sections = {'default': True} # maps section ID to true or false
        self.EmptyCell = nbformat.v4.nbbase.new_raw_cell("")

        super().__init__(**kw)

    def preprocess_cell(self, cell, resources, cell_index, store_history=True):
        """
        Executes a single code cell. See base.py for details.
        To execute all cells see :meth:`preprocess`.
        """

        if cell.cell_type not in ['code', 'markdown']:
            return cell, resources

        if cell.cell_type == 'code':
            # Do code stuff
            return self.preprocess_code_cell(cell, resources, cell_index, store_history)

        elif cell.cell_type == 'markdown':
            # Do markdown stuff
            cell, resources = self.preprocess_markdown_cell(cell, resources, cell_index, store_history)
            return cell, resources
        else:
            # Don't do anything
            return cell, resources

    def preprocess_code_cell(self, cell, resources, cell_index, store_history):
        """ Process code cell. Follow preprocess_cell from ExecutePreprocessor
        """
        self._check_assign_resources(resources)
        cell = self.execute_cell(cell, cell_index, store_history=True)
        return cell, self.resources

    def preprocess_markdown_cell(self, cell, resources, cell_index, store_history):
        # Find and execute snippets of code
        cell['metadata']['variables'] = {}
        for m in re.finditer("{{(.*?)}}", cell.source):
            # Execute code
            self.nb.cells.append(nbformat.v4.nbbase.new_code_cell(m.group(1)))
            fakecell, resources = self.preprocess_code_cell(self.nb.cells[-1], resources, len(self.nb.cells)-1, store_history)
            self.nb.cells.pop()
            # Output found in cell.outputs
            # Put output in cell['metadata']['variables']
            for output in fakecell.outputs:
                html = self.convert_output_to_html(output)
                if html is not None:
                    cell['metadata']['variables'][fakecell.source] = html
                    break
        return cell, resources

    def convert_output_to_html(self, output):
        """Convert IOpub output to HTML

        See https://github.com/ipython-contrib/IPython-notebook-extensions/blob/master/nbextensions/usability/python-markdown/main.js
        """
        if output['output_type'] == 'error':
            text = '**' + output.ename + '**: ' + output.evalue
            return text
        elif output.output_type == 'execute_result' or output.output_type == 'display_data':
            data = output.data
            if 'text/latex' in data:
                html = data['text/latex']
                return html
            elif 'image/svg+xml' in data:
                # Not supported
                #var svg = ul['image/svg+xml'];
                #/* embed SVG in an <img> tag, still get eaten by sanitizer... */
                #svg = btoa(svg);
                #html = '<img src="data:image/svg+xml;base64,' + svg + '"/>';
                return None
            elif 'image/jpeg' in data:
                jpeg = data['image/jpeg']
                html = '<img src="data:image/jpeg;base64,' + jpeg + '"/>'
                return html
            elif 'image/png' in data:
                png = data['image/png']
                html = '<img src="data:image/png;base64,' + png + '"/>'
                return html
            elif 'text/markdown' in data:
                text = data['text/markdown']
                return text
            elif 'text/html' in data:
                html = data['text/html']
                return html
            elif 'text/plain' in data:
                text = data['text/plain']
                # Strip <p> and </p> tags
                # Strip quotes
                # html.match(/<p>([\s\S]*?)<\/p>/)[1]
                text = re.sub(r'<p>([\s\S]*?)<\/p>', r'\1', text)
                text = re.sub(r"'([\s\S]*?)'",r'\1', text)
                return text
            else:
            # Some tag we don't support
                return None
        else:
            return None

使用方法保持不变。


1

更新于2020年7月8日

@gordon-bean提供的答案对我来说是救命稻草。在我放弃之前最后一轮谷歌搜索中,我发现了这个答案,所以在继续之前,我想说声谢谢!

然而,在原始答案四年多之后,jupyter / nbconvert经历了一些变化,提供的代码需要更新。因此,这里是更新后的代码:

from nbconvert.preprocessors import ExecutePreprocessor
from nbconvert.preprocessors import execute
import nbformat
import re


# Taken from:
# https://dev59.com/b5Tfa4cB1Zd3GeqPVMAd
class ExecuteCodeMarkdownPreprocessor(ExecutePreprocessor):

    def __init__(self, **kw):
        self.sections = {'default': True} # maps section ID to true or false
        self.EmptyCell = nbformat.v4.nbbase.new_raw_cell("")

        super().__init__(**kw)

    def preprocess_cell(self, cell, resources, cell_index, store_history=True):
        """
        Executes a single code cell. See base.py for details.
        To execute all cells see :meth:`preprocess`.
        """

        if cell.cell_type not in ['code', 'markdown']:
            return cell, resources

        if cell.cell_type == 'code':
            # Do code stuff
            return self.preprocess_code_cell(cell, resources, cell_index, store_history)

        elif cell.cell_type == 'markdown':
            # Do markdown stuff
            return self.preprocess_markdown_cell(cell, resources, cell_index, store_history)
        else:
            # Don't do anything
            return cell, resources

    def preprocess_code_cell(self, cell, resources, cell_index, store_history):
        """ Process code cell.
        """
        # outputs = self.run_cell(cell)
        reply, outputs = self.run_cell(cell, cell_index, store_history)

        cell.outputs = outputs

        cell_allows_errors = (self.allow_errors or "raises-exception"
                              in cell.metadata.get("tags", []))

        if self.force_raise_errors or not cell_allows_errors:
            for out in cell.outputs:
                if out.output_type == 'error':
                    raise execute.CellExecutionError.from_cell_and_msg(cell, out)
            if (reply is not None) and reply['content']['status'] == 'error':
                raise execute.CellExecutionError.from_cell_and_msg(cell, reply['content'])

        return cell, resources

    def preprocess_markdown_cell(self, cell, resources, cell_index, store_history):
        # Find and execute snippets of code
        cell['metadata']['variables'] = {}
        for m in re.finditer("{{(.*?)}}", cell.source):
            # Execute code
            fakecell = nbformat.v4.nbbase.new_code_cell(m.group(1))
            fakecell, resources = self.preprocess_code_cell(fakecell, resources, cell_index, store_history)

            # Output found in cell.outputs
            # Put output in cell['metadata']['variables']
            for output in fakecell.outputs:
                html = self.convert_output_to_html(output)
                if html is not None:
                    cell['metadata']['variables'][fakecell.source] = html
                    break
        return cell, resources

    def convert_output_to_html(self, output):
        """Convert IOpub output to HTML

        See https://github.com/ipython-contrib/IPython-notebook-extensions/blob/master/nbextensions/usability/python-markdown/main.js
        """
        if output['output_type'] == 'error':
            text = '**' + output.ename + '**: ' + output.evalue
            return text
        elif output.output_type == 'execute_result' or output.output_type == 'display_data':
            data = output.data
            if 'text/latex' in data:
                html = data['text/latex']
                return html
            elif 'image/svg+xml' in data:
                # Not supported
                #var svg = ul['image/svg+xml'];
                #/* embed SVG in an <img> tag, still get eaten by sanitizer... */
                #svg = btoa(svg);
                #html = '<img src="data:image/svg+xml;base64,' + svg + '"/>';
                return None
            elif 'image/jpeg' in data:
                jpeg = data['image/jpeg']
                html = '<img src="data:image/jpeg;base64,' + jpeg + '"/>'
                return html
            elif 'image/png' in data:
                png = data['image/png']
                html = '<img src="data:image/png;base64,' + png + '"/>'
                return html
            elif 'text/markdown' in data:
                text = data['text/markdown']
                return text
            elif 'text/html' in data:
                html = data['text/html']
                return html
            elif 'text/plain' in data:
                text = data['text/plain']
                # Strip <p> and </p> tags
                # Strip quotes
                # html.match(/<p>([\s\S]*?)<\/p>/)[1]
                text = re.sub(r'<p>([\s\S]*?)<\/p>', r'\1', text)
                text = re.sub(r"'([\s\S]*?)'",r'\1', text)
                return text
            else:
            # Some tag we don't support
                return None
        else:
            return None

这段代码的使用方法与Gordon Bean所报告的完全相同。


网页内容由stack overflow 提供, 点击上面的
可以查看英文原文,
原文链接