如何修改Tensorflow Sequence2Sequence模型以实现双向LSTM而非单向LSTM？

Question

如何修改Tensorflow Sequence2Sequence模型以实现双向LSTM而非单向LSTM？

tensorflownlplstmsequence-to-sequenceattention-model

3

请参考以下帖子了解问题的背景： TensorFlow的embedding_attention_seq2seq方法是否默认实现双向RNN编码器？我正在使用相同的模型，并想用双向层替换单向LSTM层。我意识到我必须使用static_bidirectional_rnn而不是static_rnn，但由于张量形状不匹配而出现错误。

我替换了以下行：

encoder_outputs, encoder_state = core_rnn.static_rnn(encoder_cell, encoder_inputs, dtype=dtype)

使用下面的代码：

encoder_outputs, encoder_state_fw, encoder_state_bw = core_rnn.static_bidirectional_rnn(encoder_cell, encoder_cell, encoder_inputs, dtype=dtype)

这个错误信息如下：

InvalidArgumentError (参见上面的 traceback)：不兼容的形状： [32,5,1,256] 和 [16,1,1,256] [[Node: gradients/model_with_buckets/embedding_attention_seq2seq/embedding_attention_decoder/attention_decoder/Attention_0/add_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"](gradients/model_with_buckets/embedding_attention_seq2seq/embedding_attention_decoder/attention_decoder/Attention_0/add_grad/Shape, gradients/model_with_buckets/embedding_attention_seq2seq/embedding_attention_decoder/attention_decoder/Attention_0/add_grad/Shape_1)]]

我知道这两种方法的输出是不同的，但我不知道如何修改注意力代码以使其包含这点。我该如何将前向和后向状态都发送到注意力模块中 - 我需要将隐藏状态连接起来吗？

- Leena Shekhar

1个回答

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- Lerner Zhang · Accepted Answer

我从错误信息中发现两个张量的批处理大小不匹配，一个是32，另一个是16。我认为这是因为双向rnn的输出列表是单向rnn的两倍大小。在接下来的代码中，您没有相应地进行调整。

如何将前向和后向状态都发送到注意力模块-我是否需要连接两个隐藏状态？

您可以参考以下代码：

  def _reduce_states(self, fw_st, bw_st):
    """Add to the graph a linear layer to reduce the encoder's final FW and BW state into a single initial state for the decoder. This is needed because the encoder is bidirectional but the decoder is not.
    Args:
      fw_st: LSTMStateTuple with hidden_dim units.
      bw_st: LSTMStateTuple with hidden_dim units.
    Returns:
      state: LSTMStateTuple with hidden_dim units.
    """
    hidden_dim = self._hps.hidden_dim
    with tf.variable_scope('reduce_final_st'):

      # Define weights and biases to reduce the cell and reduce the state
      w_reduce_c = tf.get_variable('w_reduce_c', [hidden_dim * 2, hidden_dim], dtype=tf.float32, initializer=self.trunc_norm_init)
      w_reduce_h = tf.get_variable('w_reduce_h', [hidden_dim * 2, hidden_dim], dtype=tf.float32, initializer=self.trunc_norm_init)
      bias_reduce_c = tf.get_variable('bias_reduce_c', [hidden_dim], dtype=tf.float32, initializer=self.trunc_norm_init)
      bias_reduce_h = tf.get_variable('bias_reduce_h', [hidden_dim], dtype=tf.float32, initializer=self.trunc_norm_init)

      # Apply linear layer
      old_c = tf.concat(axis=1, values=[fw_st.c, bw_st.c]) # Concatenation of fw and bw cell
      old_h = tf.concat(axis=1, values=[fw_st.h, bw_st.h]) # Concatenation of fw and bw state
      new_c = tf.nn.relu(tf.matmul(old_c, w_reduce_c) + bias_reduce_c) # Get new cell from old cell
      new_h = tf.nn.relu(tf.matmul(old_h, w_reduce_h) + bias_reduce_h) # Get new state from old state
return tf.contrib.rnn.LSTMStateTuple(new_c, new_h) # Return new cell and state