第三章训练报错 #71
Comments
r="", dtype=DT_FLOAT, shape=[1,1,2048,6], shared_name="", _device="/device:GPU:0"]()]] |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
WARNING:tensorflow:From train_image_classifier.py:398: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
WARNING:tensorflow:From /usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/losses/losses_impl.py:731: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.
See tf.nn.softmax_cross_entropy_with_logits_v2.
INFO:tensorflow:Summary name /clone_loss is illegal; using clone_loss instead.
INFO:tensorflow:Fine-tuning from satellite/pretrained/inception_v3.ckpt
WARNING:tensorflow:From /usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/learning.py:736: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-03-11 13:30:05.541266: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-03-11 13:30:05.566633: E tensorflow/stream_executor/cuda/cuda_driver.cc:406] failed call to cuInit: CUDA_ERROR_UNKNOWN
2019-03-11 13:30:05.566714: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:145] kernel driver does not appear to be running on this host (aile02): /proc/driver/nvidia/version does not exist
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, Cannot assign a device for operation 'InceptionV3/AuxLogits/Conv2d_2b_1x1/biases/RMSProp_1': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0 ]. Make sure the device specification refers to a valid device.
[[Node: InceptionV3/AuxLogits/Conv2d_2b_1x1/biases/RMSProp_1 = VariableV2_class=["loc:@InceptionV3/AuxLogits/Conv2d_2b_1x1/biases"], container="", dtype=DT_FLOAT, shape=[6], shared_name="", _device="/device:GPU:0"]]
Caused by op 'InceptionV3/AuxLogits/Conv2d_2b_1x1/biases/RMSProp_1', defined at:
File "train_image_classifier.py", line 573, in
tf.app.run()
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "train_image_classifier.py", line 540, in main
global_step=global_step)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 520, in apply_gradients
self._create_slots([_get_variable_for(v) for v in var_list])
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/rmsprop.py", line 115, in _create_slots
self._zeros_slot(v, "momentum", self._name)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 910, in _zeros_slot
named_slots[_var_key(var)] = slot_creator.create_zeros_slot(var, op_name)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/slot_creator.py", line 174, in create_zeros_slot
colocate_with_primary=colocate_with_primary)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/slot_creator.py", line 148, in create_slot_with_initializer
dtype)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/slot_creator.py", line 67, in _create_slot_var
validate_shape=validate_shape)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1297, in get_variable
constraint=constraint)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1093, in get_variable
constraint=constraint)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 439, in get_variable
constraint=constraint)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 408, in _true_getter
use_resource=use_resource, constraint=constraint)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 800, in _get_single_variable
use_resource=use_resource)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2157, in variable
use_resource=use_resource)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2147, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2130, in default_variable_creator
constraint=constraint)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 233, in init
constraint=constraint)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 333, in _init_from_args
name=name)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 134, in variable_op_v2
shared_name=shared_name)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 1043, in _variable_v2
shared_name=shared_name, name=name)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3271, in create_op
op_def=op_def)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1650, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'InceptionV3/AuxLogits/Conv2d_2b_1x1/biases/RMSProp_1': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0 ]. Make sure the device specification refers to a valid device.
[[Node: InceptionV3/AuxLogits/Conv2d_2b_1x1/biases/RMSProp_1 = VariableV2_class=["loc:@InceptionV3/AuxLogits/Conv2d_2b_1x1/biases"], container="", dtype=DT_FLOAT, shape=[6], shared_name="", _device="/device:GPU:0"]]
Traceback (most recent call last):
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1361, in _do_call
return fn(*args)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1332, in _run_fn
self._extend_graph()
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1392, in _extend_graph
graph_def.SerializeToString(), status)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation 'InceptionV3/AuxLogits/Conv2d_2b_1x1/biases/RMSProp_1': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0 ]. Make sure the device specification refers to a valid device.
[[Node: InceptionV3/AuxLogits/Conv2d_2b_1x1/biases/RMSProp_1 = VariableV2_class=["loc:@InceptionV3/AuxLogits/Conv2d_2b_1x1/biases"], container="", dtype=DT_FLOAT, shape=[6], shared_name="", _device="/device:GPU:0"]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train_image_classifier.py", line 573, in
tf.app.run()
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "train_image_classifier.py", line 569, in main
sync_optimizer=optimizer if FLAGS.sync_replicas else None)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 746, in train
master, start_standard_services=False, config=session_config) as sess:
File "/usr/local/anaconda3/lib/python3.6/contextlib.py", line 81, in enter
return next(self.gen)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 1000, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 828, in stop
ignore_live_threads=ignore_live_threads)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/usr/local/anaconda3/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 989, in managed_session
start_standard_services=start_standard_services)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/supervisor.py", line 726, in prepare_or_wait_for_session
init_feed_dict=self._init_feed_dict, init_fn=self._init_fn)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/session_manager.py", line 281, in prepare_session
sess.run(init_op, feed_dict=init_feed_dict)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1137, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1355, in _do_run
options, run_metadata)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1374, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation 'InceptionV3/AuxLogits/Conv2d_2b_1x1/biases/RMSProp_1': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0 ]. Make sure the device specification refers to a valid device.
[[Node: InceptionV3/AuxLogits/Conv2d_2b_1x1/biases/RMSProp_1 = VariableV2_class=["loc:@InceptionV3/AuxLogits/Conv2d_2b_1x1/biases"], container="", dtype=DT_FLOAT, shape=[6], shared_name="", _device="/device:GPU:0"]]
Caused by op 'InceptionV3/AuxLogits/Conv2d_2b_1x1/biases/RMSProp_1', defined at:
File "train_image_classifier.py", line 573, in
tf.app.run()
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "train_image_classifier.py", line 540, in main
global_step=global_step)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 520, in apply_gradients
self._create_slots([_get_variable_for(v) for v in var_list])
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/rmsprop.py", line 115, in _create_slots
self._zeros_slot(v, "momentum", self._name)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 910, in _zeros_slot
named_slots[_var_key(var)] = slot_creator.create_zeros_slot(var, op_name)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/slot_creator.py", line 174, in create_zeros_slot
colocate_with_primary=colocate_with_primary)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/slot_creator.py", line 148, in create_slot_with_initializer
dtype)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/slot_creator.py", line 67, in _create_slot_var
validate_shape=validate_shape)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1297, in get_variable
constraint=constraint)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 1093, in get_variable
constraint=constraint)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 439, in get_variable
constraint=constraint)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 408, in _true_getter
use_resource=use_resource, constraint=constraint)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 800, in _get_single_variable
use_resource=use_resource)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2157, in variable
use_resource=use_resource)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2147, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2130, in default_variable_creator
constraint=constraint)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 233, in init
constraint=constraint)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 333, in _init_from_args
name=name)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 134, in variable_op_v2
shared_name=shared_name)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 1043, in _variable_v2
shared_name=shared_name, name=name)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3271, in create_op
op_def=op_def)
File "/usr/local/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1650, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'InceptionV3/AuxLogits/Conv2d_2b_1x1/biases/RMSProp_1': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0 ]. Make sure the device specification refers to a valid device.
[[Node: InceptionV3/AuxLogits/Conv2d_2b_1x1/biases/RMSProp_1 = VariableV2_class=["loc:@InceptionV3/AuxLogits/Conv2d_2b_1x1/biases"], container="", dtype=DT_FLOAT, shape=[6], shared_name="", _device="/device:GPU:0"]]
The text was updated successfully, but these errors were encountered: