我认为我掌握了DropOut的基础知识以及在实现中使用TensorFlow API的方法(链接)。但是在tf.nn.dropout
中与dropout概率相关联的归一化似乎不是DropConnect的一部分。这是正确的吗?如果是,那么归一化是否会造成任何“损害”,或者我是否可以简单地将tf.nn.dropout
应用于我的权重以实现DropConnect?
我认为我掌握了DropOut的基础知识以及在实现中使用TensorFlow API的方法(链接)。但是在tf.nn.dropout
中与dropout概率相关联的归一化似乎不是DropConnect的一部分。这是正确的吗?如果是,那么归一化是否会造成任何“损害”,或者我是否可以简单地将tf.nn.dropout
应用于我的权重以实现DropConnect?
是的,您可以使用tf.nn.dropout来执行DropConnect,只需使用tf.nn.dropout来包装您的权重矩阵,而不是后矩阵乘法。然后,您可以通过乘以dropout来撤消权重更改。
dropConnect = tf.nn.dropout( m1, keep_prob ) * keep_prob
这是一个使用drop connect计算XOR函数的代码示例。我还注释掉了执行dropout的代码,您可以用它来比较输出结果。
### imports
import tensorflow as tf
### constant data
x = [[0.,0.],[1.,1.],[1.,0.],[0.,1.]]
y_ = [[1.,0.],[1.,0.],[0.,1.],[0.,1.]]
### induction
# Layer 0 = the x2 inputs
x0 = tf.constant( x , dtype=tf.float32 )
y0 = tf.constant( y_ , dtype=tf.float32 )
keep_prob = tf.placeholder( dtype=tf.float32 )
# Layer 1 = the 2x12 hidden sigmoid
m1 = tf.Variable( tf.random_uniform( [2,12] , minval=0.1 , maxval=0.9 , dtype=tf.float32 ))
b1 = tf.Variable( tf.random_uniform( [12] , minval=0.1 , maxval=0.9 , dtype=tf.float32 ))
########## DROP CONNECT
# - use this to preform "DropConnect" flavor of dropout
dropConnect = tf.nn.dropout( m1, keep_prob ) * keep_prob
h1 = tf.sigmoid( tf.matmul( x0, dropConnect ) + b1 )
########## DROP OUT
# - uncomment this to use "regular" dropout
#h1 = tf.nn.dropout( tf.sigmoid( tf.matmul( x0,m1 ) + b1 ) , keep_prob )
# Layer 2 = the 12x2 softmax output
m2 = tf.Variable( tf.random_uniform( [12,2] , minval=0.1 , maxval=0.9 , dtype=tf.float32 ))
b2 = tf.Variable( tf.random_uniform( [2] , minval=0.1 , maxval=0.9 , dtype=tf.float32 ))
y_out = tf.nn.softmax( tf.matmul( h1,m2 ) + b2 )
# loss : sum of the squares of y0 - y_out
loss = tf.reduce_sum( tf.square( y0 - y_out ) )
# training step : discovered learning rate of 1e-2 through experimentation
train = tf.train.AdamOptimizer(1e-2).minimize(loss)
### training
# run 5000 times using all the X and Y
# print out the loss and any other interesting info
with tf.Session() as sess:
sess.run( tf.initialize_all_variables() )
print "\nloss"
for step in range(5000) :
sess.run(train,feed_dict={keep_prob:0.5})
if (step + 1) % 100 == 0 :
print sess.run(loss,feed_dict={keep_prob:1.})
results = sess.run([m1,b1,m2,b2,y_out,loss],feed_dict={keep_prob:1.})
labels = "m1,b1,m2,b2,y_out,loss".split(",")
for label,result in zip(*(labels,results)) :
print ""
print label
print result
print ""
这两种变体都能够将输入正确地分离成正确的输出。
y_out
[[ 7.05891490e-01 2.94108540e-01]
[ 9.99605477e-01 3.94574134e-04]
[ 4.99370173e-02 9.50062990e-01]
[ 4.39682379e-02 9.56031740e-01]]
在这里,您可以看到dropConnect输出的结果:正确分类为真、真、假、假。
tf.nn.dropout(b1, keep_prob)*keep_prob
? - bytesinflight