Failure mode of pipeline: file writing

[laldoroty]

Traceback (most recent call last):
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/util.py", line 810, in _free_space_check
    yield
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 722, in _writeto
    self._writeto_internal(fileobj, inplace, copy)
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 728, in _writeto_internal
    data_offset, data_size = self._writedata(fileobj)
                             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 659, in _writedata
    size += self._writedata_internal(fileobj)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/hdu/image.py", line 674, in _writedata_internal
    fileobj.writearray(output)
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/file.py", line 432, in writearray
    _array_to_file(array, self._file)
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/util.py", line 616, in _array_to_file
    return write(arr, outfile)
           ^^^^^^^^^^^^^^^^^^^
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/util.py", line 591, in <lambda>
    write = lambda a, f: a.tofile(f)
                         ^^^^^^^^^^^
OSError: problem writing element 619520 to file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/util.py", line 810, in _free_space_check
    yield
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1043, in writeto
    hdu._writeto(hdulist._file)
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 721, in _writeto
    with _free_space_check(self, dirname):
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/util.py", line 830, in _free_space_check
    raise OSError(error_message + str(exc))
OSError: problem writing element 619520 to file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/global/cfs/cdirs/m4385/users/lauren/phrosty/phrosty/pipeline.py", line 559, in <module>
    main()
  File "/global/cfs/cdirs/m4385/users/lauren/phrosty/phrosty/pipeline.py", line 555, in main
    pipeline( args.through_step )
  File "/global/cfs/cdirs/m4385/users/lauren/phrosty/phrosty/pipeline.py", line 486, in __call__
    fits.writeto( savepath, cp.asnumpy( decorimg ).T, header=hdr, overwrite=True )
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/convenience.py", line 464, in writeto
    hdu.writeto(
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 412, in writeto
    hdulist.writeto(name, output_verify, overwrite=overwrite, checksum=checksum)
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1040, in writeto
    with _free_space_check(self, dirname=dirname):
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/global/homes/l/laldorot/.conda/envs/diff/lib/python3.11/site-packages/astropy/io/fits/util.py", line 830, in _free_space_check
    raise OSError(error_message + str(exc))
OSError: problem writing element 619520 to file

[laldoroty]

Note that this doesn't fail like this for a while. It runs for some time and then crashes.

Here's one that failed like this:

sn="20000104" 
ra="8.167274983711236" 
dec="-43.08698868442529" 
band="F184"