Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

What is this file: bn_17.sn_360.epoch_9.ckpt.data-00000-of-00001 #84

Open
Arkadiy-Garber opened this issue Mar 23, 2023 · 2 comments
Open

What is this file: bn_17.sn_360.epoch_9.ckpt.data-00000-of-00001 #84

Arkadiy-Garber opened this issue Mar 23, 2023 · 2 comments

Comments

@Arkadiy-Garber
Copy link

Arkadiy-Garber commented Mar 23, 2023 •
edited
Loading

I am downloading the trained models, but I am not getting any files ending with .ckpt.

Here are the contents of one of the directories:

bn_17.sn_360.epoch_9.ckpt.data-00000-of-00001 

bn_17.sn_360.epoch_9.ckpt.index 

bn_17.sn_360.epoch_9.ckpt.meta 

checkpoint

Which of these files should i provide to the --model_path command?

Traceback (most recent call last):
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 255, in _fast5s_q_to_pred_str_q
    saver.restore(sess, model_path)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1276, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.DataLossError: Unable to open table file /data/MAB/bin/deepsignal/models/model.CpG.R9.4_1D.human_hx1.bn17.sn360.v0.1.7+/bn_17.sn_360.epoch_9.ckpt.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
	 [[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]

Caused by op 'save/RestoreV2', defined at:
  File "/data/MAB/miniconda3/envs/deepsignalenv/bin/deepsignal", line 33, in <module>
    sys.exit(load_entry_point('deepsignal==0.2.0', 'console_scripts', 'deepsignal')())
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 424, in main
    args.func(args)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 88, in main_call_mods
    f5_args)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 448, in call_mods
    is_rnn, is_base, is_cnn)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 326, in _call_mods_from_fast5s_cpu
    p.start()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
    code = process_obj._bootstrap()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 254, in _fast5s_q_to_pred_str_q
    saver = tf.train.Saver()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 832, in __init__
    self.build()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 844, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 881, in _build
    build_save=build_save, build_restore=build_restore)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 513, in _build_internal
    restore_sequentially, reshape)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps
    restore_sequentially)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2
    name=name)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

DataLossError (see above for traceback): Unable to open table file /data/MAB/bin/deepsignal/models/model.CpG.R9.4_1D.human_hx1.bn17.sn360.v0.1.7+/bn_17.sn_360.epoch_9.ckpt.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
	 [[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]

write process 1316634 ending
0 of 110340 fast5 files failed..
call_mods costs 29.43 seconds..
@PengNi
Copy link
Collaborator

PengNi commented Mar 24, 2023

@Arkadiy-Garber , hi, just use the prefix /path/to/bn_17.sn_360.epoch_9.ckpt.

@Arkadiy-Garber
Copy link
Author

Thanks, that seemed to worked to fix the previous issue. However, I am not getting the following output at then end of the stdout (with the command):

deepsignal call_mods --input_path single_barcode12/ --model_path /data/MAB/bin/deepsignal/models/model.GATC.R9_2D.tem.puc19.bn17.sn360/bn_17.sn_360.epoch_5.ckpt --result_file single_barcode12.gatc.call_mods.tsv --corrected_group RawGenomeCorrected_001 --nproc 10 --is_gpu no

/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/sklearn/utils/validation.py:37: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  LARGE_SPARSE_SUPPORTED = LooseVersion(scipy_version) >= '0.14.0'
# ===============================================
## parameters: 
input_path:
	single_barcode12/
f5_batch_num:
	50
model_path:
	/data/MAB/bin/deepsignal/models/model.GATC.R9_2D.tem.puc19.bn17.sn360/bn_17.sn_360.epoch_5.ckpt
is_cnn:
	yes
is_rnn:
	yes
is_base:
	yes
kmer_len:
	17
cent_signals_len:
	360
batch_size:
	512
learning_rate:
	0.001
class_num:
	2
result_file:
	single_barcode12.gatc.call_mods.tsv
recursively:
	yes
corrected_group:
	RawGenomeCorrected_001
basecall_subgroup:
	BaseCalled_template
is_dna:
	yes
normalize_method:
	mad
motifs:
	CG
mod_loc:
	0
positions:
	None
reference_path:
	None
nproc:
	1
is_gpu:
	no
# ===============================================
110340 fast5 files in total..
parse the motifs string..
read genome reference file..
read position file if it is not None..
call_mods process 1352721 starts
write process 1352722 starts
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:49: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:66: MultiRNNCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.StackedRNNCells, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:71: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.Bidirectional(keras.layers.RNN(cell))`, which is equivalent to this API
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/rnn.py:443: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.RNN(cell)`, which is equivalent to this API
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/rnn.py:626: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/rnn_cell_impl.py:1259: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:185: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:191: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:235: average_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.average_pooling2d instead.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:77: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dense instead.
2023-03-24 19:35:20.760064: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2023-03-24 19:35:20.786036: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz
2023-03-24 19:35:20.790056: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55c23ba56f60 executing computations on platform Host. Devices:
2023-03-24 19:35:20.790083: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2023-03-24 19:35:23.436685: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
Process Process-1:
Traceback (most recent call last):
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[{{node save/RestoreV2}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1276, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]

Caused by op 'save/RestoreV2', defined at:
  File "/data/MAB/miniconda3/envs/deepsignalenv/bin/deepsignal", line 33, in <module>
    sys.exit(load_entry_point('deepsignal==0.2.0', 'console_scripts', 'deepsignal')())
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 424, in main
    args.func(args)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 88, in main_call_mods
    f5_args)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 448, in call_mods
    is_rnn, is_base, is_cnn)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 326, in _call_mods_from_fast5s_cpu
    p.start()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
    code = process_obj._bootstrap()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 254, in _fast5s_q_to_pred_str_q
    saver = tf.train.Saver()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 832, in __init__
    self.build()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 844, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 881, in _build
    build_save=build_save, build_restore=build_restore)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 513, in _build_internal
    restore_sequentially, reshape)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps
    restore_sequentially)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2
    name=name)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1286, in restore
    names_to_keys = object_graph_key_mapping(save_path)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1591, in object_graph_key_mapping
    checkpointable.OBJECT_GRAPH_PROTO_KEY)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 370, in get_tensor
    status)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 255, in _fast5s_q_to_pred_str_q
    saver.restore(sess, model_path)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1292, in restore
    err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]

Caused by op 'save/RestoreV2', defined at:
  File "/data/MAB/miniconda3/envs/deepsignalenv/bin/deepsignal", line 33, in <module>
    sys.exit(load_entry_point('deepsignal==0.2.0', 'console_scripts', 'deepsignal')())
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 424, in main
    args.func(args)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 88, in main_call_mods
    f5_args)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 448, in call_mods
    is_rnn, is_base, is_cnn)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 326, in _call_mods_from_fast5s_cpu
    p.start()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
    code = process_obj._bootstrap()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 254, in _fast5s_q_to_pred_str_q
    saver = tf.train.Saver()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 832, in __init__
    self.build()
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 844, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 881, in _build
    build_save=build_save, build_restore=build_restore)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 513, in _build_internal
    restore_sequentially, reshape)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps
    restore_sequentially)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2
    name=name)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]

write process 1352722 ending
0 of 110340 fast5 files failed..
call_mods costs 26.37 seconds..

The results file (single_barcode12.gatc.call_mods.tsv) that is produced is empty.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@PengNi @Arkadiy-Garber