如何对tf.data.Dataset进行性能分析？

Question

我正在尝试了解在我的input_fn中与tf.data.Dataset相关的瓶颈问题，因此我想使用tf.profiler，但它只显示迭代器操作。如何让分析工具输出我的Dataset管道中相关的操作呢？

示例

dataset = input_fn()
iterator = dataset.make_one_shot_iterator()
minibatch = iterator.get_next()
run_metadata = tf.RunMetadata()
with tf.Session() as session:
    features, labels = session.run(minibatch, 
                                   options=tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE),
                                   run_metadata=run_metadata)

tf.profiler.advise(tf.get_default_graph(), run_metadata)

输出：

checkers {
  key: "AcceleratorUtilizationChecker"
  value {
  }
}
checkers {
  key: "ExpensiveOperationChecker"
  value {
    reports: "top 1 operation type: IteratorGetNext, cpu: 79.89sec, accelerator: 0us, total: 79.89sec (99.96%)\ntop 2 operation type: OneShotIterator, cpu: 27.92ms, accelerator: 0us, total: 27.92ms (0.03%)\ntop 3 operation type: _retval_IteratorGetNext_3_3, cpu: 57us, accelerator: 0us, total: 57us (0.00%)"
    reports: "top 1 graph node: IteratorGetNext, cpu: 79.89sec, accelerator: 0us, total: 79.89sec\ntop 2 graph node: OneShotIterator, cpu: 27.92ms, accelerator: 0us, total: 27.92ms"
    reports: "<ipython-input-2-c5f67ba0356f>:49:<module>, cpu: 79.89sec, accelerator: 0us, total: 79.89sec\n<ipython-input-2-c5f67ba0356f>:48:<module>, cpu: 27.92ms, accelerator: 0us, total: 27.92ms"
  }
}
checkers {
  key: "OperationChecker"
  value {
  }
}

- Carl Thomé

---

嗨，你找到数据集为什么不显示在个人资料中的原因了吗？ - peteyuan

我怀疑你需要在第一行dataset = input_fn()之前加上tf.RunMetadata()。 - John Jiang

使用nvidia-profiler。 - Ishant Mrinal

你好，上面的示例代码使用的是哪个版本的tensorflow？ - aka.ecc

Answer 1

看起来 tf.data 的性能分析还没有被实现。它似乎是在版本 1.14 中添加的。以下是代码片段：

Original Answer翻译成"最初的回答"

import tensorflow as tf

dataset = tf.data.Dataset.range(100)
dataset = dataset.shuffle(30)
dataset = dataset.repeat()

iterator = dataset.make_one_shot_iterator()
minibatch = iterator.get_next()
run_metadata = tf.RunMetadata()
options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
with tf.Session() as session:
    session.run(minibatch, options=options, run_metadata=run_metadata)

tf.profiler.advise(tf.get_default_graph(), run_metadata)

输出：

Parsing Inputs...

ExpensiveOperationChecker:
top 1 operation type: OneShotIterator, cpu: 3.01ms, accelerator: 0us, total: 3.01ms (87.19%)
top 2 operation type: IteratorGetNext, cpu: 440us, accelerator: 0us, total: 440us (12.75%)
top 3 operation type: _retval_IteratorGetNext_0_0, cpu: 2us, accelerator: 0us, total: 2us (0.06%)
top 1 graph node: OneShotIterator, cpu: 3.01ms, accelerator: 0us, total: 3.01ms
top 2 graph node: IteratorGetNext, cpu: 440us, accelerator: 0us, total: 440us
test.py:7:<module>, cpu: 3.01ms, accelerator: 0us, total: 3.01ms

OperationChecker:

AcceleratorUtilizationChecker:

- McAngus