Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errors in attempt at reproducing micro search #26

Open
ahundt opened this issue Apr 19, 2018 · 3 comments
Open

Errors in attempt at reproducing micro search #26

ahundt opened this issue Apr 19, 2018 · 3 comments

Comments

@ahundt
Copy link

ahundt commented Apr 19, 2018

I tried running micro search on TF 1.7 and it made quite a bit of progress, up to 150 epochs, but then it failed out as follows:

[1 2 1 1 1 3 0 2 2 0 1 1 1 1 1 4 1 4 1 4]
val_acc=0.7750
--------------------------------------------------------------------------------
[0 0 1 0 0 4 0 1 0 4 1 1 1 4 0 1 0 1 5 2]
[0 1 1 0 1 1 1 0 1 2 1 3 1 0 3 3 1 0 2 4]
val_acc=0.6813
--------------------------------------------------------------------------------
[0 1 1 0 0 0 0 0 0 0 1 1 4 0 0 0 0 0 1 1]
[1 0 1 2 1 1 1 1 1 0 1 3 3 0 2 0 1 0 1 1]
val_acc=0.7312
--------------------------------------------------------------------------------
[0 1 0 4 0 0 0 2 1 0 1 3 1 0 3 0 1 1 1 1]
[1 0 1 0 1 1 1 1 1 4 1 1 1 1 1 0 3 4 1 4]
val_acc=0.7188
--------------------------------------------------------------------------------
[0 0 0 2 1 0 1 0 1 4 0 3 0 1 1 0 0 1 4 2]
[0 4 1 1 1 4 1 1 1 1 1 0 1 0 1 2 1 1 1 2]
val_acc=0.7250
--------------------------------------------------------------------------------
Epoch 150: Eval
Eval at 42300
valid_accuracy: 0.6946
Eval at 42300
test_accuracy: 0.6842
Exception in thread QueueRunnerThread-dummy_queue-sync_token_q_EnqueueMany:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ahundt/.local/lib/python2.7/site-packages/tensorflow/python/training/queue_runner_impl.py", line 268, in _run
    coord.request_stop(e)
  File "/home/ahundt/.local/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 213, in request_stop
    six.reraise(*sys.exc_info())
  File "/home/ahundt/.local/lib/python2.7/site-packages/tensorflow/python/training/queue_runner_impl.py", line 252, in _run
    enqueue_callable()
  File "/home/ahundt/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1249, in _single_operation_run
    self._call_tf_sessionrun(None, {}, [], target_list, None)
  File "/home/ahundt/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1420, in _call_tf_sessionrun
    status, run_metadata)
  File "/home/ahundt/.local/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
CancelledError: TakeGrad operation was cancelled
         [[Node: sync_replicas/AccumulatorTakeGradient = AccumulatorTakeGradient[_class=["loc:@sync_replicas/conditional_accumulator"], dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](sync_replicas/conditional_accumulator, sync_replicas/AccumulatorTakeGradient/num_required)]]
         [[Node: sync_replicas/AccumulatorTakeGradient_2/_16859 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_93_sync_replicas/AccumulatorTakeGradient_2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

I didn't take any steps to cancel it like hitting ctrl+c so I'm not sure why this is occurring.

@ahundt ahundt changed the title Errors in micro search Errors in attempt at reproducing micro search Apr 19, 2018
@MattVil
Copy link

MattVil commented Jul 11, 2018

did you solve this issue ? I have the same Error a the end of my search

@ahundt
Copy link
Author

ahundt commented Jul 11, 2018

I think so, look at the pull request I made

@ahundt
Copy link
Author

ahundt commented Jul 21, 2018

#29

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants