You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape

[GabrielLin]

When I train to epoch 3, the following errors have been shown. Could you help? thanks.

===========master initModelParams=================
fail to restore model params from /data1/asr/savemodels/ds2_0311
Epoch 0: 100%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 61/61 [05:26<00:00, 2.39s/it, loss=90.9, refresh=0, step=61]
Epoch 1: 100%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 61/61 [06:32<00:00, 6.07s/it, loss=249, refresh=0, step=122]
Epoch 2: 100%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 61/61 [06:09<00:00, 5.94s/it, loss=256, refresh=0, step=183]
Epoch 3: 100%|??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 61/61 [06:06<00:00, 6.32s/it, loss=269, refresh=0, step=244]
06-11 15:49:23 Epoch:3,begin validation, batch_num:66
2018-06-11 15:49:23.917666: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.917668: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.918618: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.918721: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.918824: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.918889: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.918947: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919004: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919060: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919127: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919185: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919246: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919303: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919359: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919411: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919467: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919520: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919574: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919628: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919680: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919735: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919788: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919843: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919899: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.919951: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.920006: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.920058: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.920117: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.920177: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.920230: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.920506: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.920581: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.920640: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.920698: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.920755: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.920810: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.920866: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.920921: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.920977: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.921029: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.921085: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.921147: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.921202: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.921258: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.921574: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.921655: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
2018-06-11 15:49:23.921717: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
Traceback (most recent call last):
File "run_train.py", line 66, in
tr.train()
File "/data1/myname/nlp/end2endASR/trainer/trainer.py", line 163, in train
self.train_single()
File "/data1/myname/nlp/end2endASR/trainer/trainer.py", line 180, in train_single
self.run_train_epoch(sess,epoch+self.args.initial_epoch,self.args.batch_size)
File "/data1/myname/nlp/end2endASR/trainer/trainer.py", line 269, in run_train_epoch
decode_cost,decode_cer, decode_wer = self.run_validate_decode_epoch(sess, epoch, batch_size)
File "/data1/myname/nlp/end2endASR/trainer/trainer.py", line 328, in run_validate_decode_epoch
feed_dict=feed_dict)
File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
[[Node: bidrnn4/bw/bw/All/_1767 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_259_bidrnn4/bw/bw/All", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/cpu:0"]]

Caused by op u'target_seq_length', defined at:
File "run_train.py", line 59, in
tr.build_model()
File "/data1/myname/nlp/end2endASR/utils/misc.py", line 78, in wrapper
result = func(*args, **kwargs)
File "/data1/myname/nlp/end2endASR/trainer/trainer.py", line 83, in build_model
self.classifier = model_factory(args=self.args, server=self.server, device=self.device)
File "/data1/myname/nlp/end2endASR/model/model_factory.py", line 35, in model_factory
return ds2_classifier.DeepSpeech2(args, name or args.model, server=server, device=device)
File "/data1/myname/nlp/end2endASR/model/ds2_classifier.py", line 134, in init
self.build_graph()
File "/data1/myname/nlp/end2endASR/model/ds2_classifier.py", line 208, in build_graph
self.add_input_layer()
File "/data1/myname/nlp/end2endASR/model/ds2_classifier.py", line 152, in add_input_layer
name='target_seq_length')
File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 1548, in placeholder
return gen_array_ops._placeholder(dtype=dtype, shape=shape, name=name)
File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2094, in _placeholder
name=name)
File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/opt/anaconda2/envs/tf1p3py27/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'target_seq_length' with dtype int32 and shape [32]
[[Node: target_seq_length = Placeholderdtype=DT_INT32, shape=[32], _device="/job:localhost/replica:0/task:0/gpu:0"]]
[[Node: bidrnn4/bw/bw/All/_1767 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_259_bidrnn4/bw/bw/All", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/cpu:0"]]

[poor1017]

Hi, I encounter the same issue, have you found the solution? Thank you.

[GabrielLin]

@poor1017 Not yet, I hope you can share your solution if you find it. Thanks.

[poor1017]

I think you used ds2 model, so you need add two lines:

@@ -134,6 +134,9 @@ class Trainer(object):
feed_dict[self.classifier.targetVals] = target_vals
feed_dict[self.classifier.targetShape] = target_shape
feed_dict[self.classifier.input_seq_length] = seqlens

•        label_lengths = batch['label_lengths']
  

•        feed_dict[self.classifier.target_seq_length] = label_lengths
  
     elif input_type == 'dense':
         inputx = batch['x']
  

[cdyangbo]

this bug may be fix soon! in ds2 model don't need this parameters, just omit it .