Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tutorial notebooks from simex_notebooks #163

Open
1 of 2 tasks
n01r opened this issue Apr 18, 2018 · 25 comments
Open
1 of 2 tasks

Tutorial notebooks from simex_notebooks #163

n01r opened this issue Apr 18, 2018 · 25 comments
Assignees
@n01r

Comments

@n01r
Copy link
Contributor

n01r commented Apr 18, 2018

Run and report errors:
@n01r n01r self-assigned this Apr 18, 2018
@n01r n01r mentioned this issue Apr 18, 2018
Open
3 tasks
@n01r
Copy link
Contributor Author

n01r commented Apr 18, 2018

The xstal_diffraction.ipynb notebook works on Maxwell @ DESY

@n01r
Copy link
Contributor Author

n01r commented Apr 18, 2018

Running the start_to_end_demo.ipynb yields errors.

The photon source file is not available but a test file can be downloaded from https://zenodo.org/record/888853/files/s2e_tutorial_example_source_input.h5

Then the notebook runs fine until the photon matter interaction

photon_matter_interactor.backengine(); photon_matter_interactor.saveH5()

Previous module:  simex_notebooks
  NOT: data
  NOT: history
info
misc
params
version
['arrEhor', 'arrEver']
[('arrEhor', <HDF5 dataset "arrEhor": shape (42, 84, 258, 2), type "<f4">), ('arrEver', <HDF5 dataset "arrEver": shape (42, 84, 258, 2), type "<f4">)]
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-27-37c843a80bc5> in <module>()
----> 1 photon_matter_interactor.backengine(); photon_matter_interactor.saveH5()

/data/netapp/s2e/simex/lib/python3.4/SimEx/Calculators/XMDYNDemoPhotonMatterInteractor.py in backengine(self)
    187             pmi_demo.f_init_random()
    188             pmi_demo.f_save_info()
--> 189             pmi_demo.f_load_pulse( pmi_demo.g_s2e['prop_out'] )
    190 
    191             # Get file extension.

/data/netapp/s2e/simex/lib/python3.4/SimEx/Calculators/XMDYNDemoPhotonMatterInteractor.py in f_load_pulse(self, a_prop_out)
    501         sel_x = self.g_s2e['pulse']['nx'] // 2 ;
    502         sel_y = self.g_s2e['pulse']['ny'] // 2 ;
--> 503         sel_pixV = self.g_s2e['pulse']['arrEver'] [sel_x,sel_y,:,:]
    504         sel_pixH = self.g_s2e['pulse']['arrEhor'] [sel_x,sel_y,:,:]
    505         dt = ( self.g_s2e['pulse']['sliceMax'] - self.g_s2e['pulse']['sliceMin'] ) / ( self.g_s2e['pulse']['nSlices'] * 1.0 )

IndexError: index 42 is out of bounds for axis 0 with size 42

Seems to be that there is something wrong with the slicing order.

ping @CFGrote

@n01r n01r mentioned this issue Apr 19, 2018
Merged
@n01r
Copy link
Contributor Author

n01r commented Apr 19, 2018

After #166 is merged, we can upload a new example notebook that uses the new example FEL source file.

@rshopa
Copy link

rshopa commented Apr 5, 2019

Multiple errors on running start_to_end_demo.ipynb. I am not sure whither it is the consequence of bad install (see an issue related to Polish XFEL users) or not.

If run manually line by line and initially it looks ok, at least by the line

In [4]: source_analysis.plotTotalPower()

Plotting total power.
Pulse energy 0.00017 J
# image here is ok

Next line yields segmentation fault (core dumped) error on bash and a message The kernel appears to have died. It will restart automatically. in the notebook:

source_analysis.plotTotalPower(spectrum=True)

The same error appears for prop_analysis.plotTotalPower(spectrum=True) later.

Next, an error if trying to plot

In [5]: source_analysis.plotIntensityMap()

Plotting intensity map.
R-space
Error in callback <function install_repl_displayhook.<locals>.post_execute at 0x7fd772f9cbf8> (for post_execute):
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/pyplot.py in post_execute()
    107             def post_execute():
    108                 if matplotlib.is_interactive():
--> 109                     draw_all()
    110 
    111             # IPython >= 2

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/_pylab_helpers.py in draw_all(cls, force)
    130         for f_mgr in cls.get_all_fig_managers():
    131             if force or f_mgr.canvas.figure.stale:
--> 132                 f_mgr.canvas.draw_idle()
    133 
    134 atexit.register(Gcf.destroy_all)

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/backend_bases.py in draw_idle(self, *args, **kwargs)
   1897         if not self._is_idle_drawing:
   1898             with self._idle_draw_cntx():
-> 1899                 self.draw(*args, **kwargs)
   1900 
   1901     def draw_cursor(self, event):

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py in draw(self)
    400         toolbar = self.toolbar
    401         try:
--> 402             self.figure.draw(self.renderer)
    403             # A GUI class may be need to update a window using this draw, so
    404             # don't forget to call the superclass.

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     48                 renderer.start_filter()
     49 
---> 50             return draw(artist, renderer, *args, **kwargs)
     51         finally:
     52             if artist.get_agg_filter() is not None:

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/figure.py in draw(self, renderer)
   1647 
   1648             mimage._draw_list_compositing_images(
-> 1649                 renderer, self, artists, self.suppressComposite)
   1650 
   1651             renderer.close_group('figure')

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/image.py in _draw_list_compositing_images(renderer, parent, artists, suppress_composite)
    136     if not_composite or not has_images:
    137         for a in artists:
--> 138             a.draw(renderer)
    139     else:
    140         # Composite any adjacent images together

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     48                 renderer.start_filter()
     49 
---> 50             return draw(artist, renderer, *args, **kwargs)
     51         finally:
     52             if artist.get_agg_filter() is not None:

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/axes/_base.py in draw(self, renderer, inframe)
   2573             self.apply_aspect(pos)
   2574         else:
-> 2575             self.apply_aspect()
   2576 
   2577         artists = self.get_children()

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/axes/_base.py in apply_aspect(self, position)
   1561         # Not sure whether we need this check:
   1562         if shared_x and shared_y:
-> 1563             raise RuntimeError("adjustable='datalim' is not allowed when both"
   1564                                " axes are shared.")
   1565 

RuntimeError: adjustable='datalim' is not allowed when both axes are shared.

<Figure size 432x288 with 0 Axes>
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/miniconda3/envs/xfel/lib/python3.6/site-packages/IPython/core/formatters.py in __call__(self, obj)
    339                 pass
    340             else:
--> 341                 return printer(obj)
    342             # Finally look for special method names
    343             method = get_real_method(obj, self.print_method)

~/miniconda3/envs/xfel/lib/python3.6/site-packages/IPython/core/pylabtools.py in <lambda>(fig)
    242 
    243     if 'png' in formats:
--> 244         png_formatter.for_type(Figure, lambda fig: print_figure(fig, 'png', **kwargs))
    245     if 'retina' in formats or 'png2x' in formats:
    246         png_formatter.for_type(Figure, lambda fig: retina_figure(fig, **kwargs))

~/miniconda3/envs/xfel/lib/python3.6/site-packages/IPython/core/pylabtools.py in print_figure(fig, fmt, bbox_inches, **kwargs)
    126 
    127     bytes_io = BytesIO()
--> 128     fig.canvas.print_figure(bytes_io, **kw)
    129     data = bytes_io.getvalue()
    130     if fmt == 'svg':

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/backend_bases.py in print_figure(self, filename, dpi, facecolor, edgecolor, orientation, format, bbox_inches, **kwargs)
   2047                         orientation=orientation,
   2048                         dryrun=True,
-> 2049                         **kwargs)
   2050                     renderer = self.figure._cachedRenderer
   2051                     bbox_artists = kwargs.pop("bbox_extra_artists", None)

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py in print_png(self, filename_or_obj, *args, **kwargs)
    508 
    509         """
--> 510         FigureCanvasAgg.draw(self)
    511         renderer = self.get_renderer()
    512 

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py in draw(self)
    400         toolbar = self.toolbar
    401         try:
--> 402             self.figure.draw(self.renderer)
    403             # A GUI class may be need to update a window using this draw, so
    404             # don't forget to call the superclass.

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     48                 renderer.start_filter()
     49 
---> 50             return draw(artist, renderer, *args, **kwargs)
     51         finally:
     52             if artist.get_agg_filter() is not None:

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/figure.py in draw(self, renderer)
   1647 
   1648             mimage._draw_list_compositing_images(
-> 1649                 renderer, self, artists, self.suppressComposite)
   1650 
   1651             renderer.close_group('figure')

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/image.py in _draw_list_compositing_images(renderer, parent, artists, suppress_composite)
    136     if not_composite or not has_images:
    137         for a in artists:
--> 138             a.draw(renderer)
    139     else:
    140         # Composite any adjacent images together

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     48                 renderer.start_filter()
     49 
---> 50             return draw(artist, renderer, *args, **kwargs)
     51         finally:
     52             if artist.get_agg_filter() is not None:

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/axes/_base.py in draw(self, renderer, inframe)
   2573             self.apply_aspect(pos)
   2574         else:
-> 2575             self.apply_aspect()
   2576 
   2577         artists = self.get_children()

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/axes/_base.py in apply_aspect(self, position)
   1561         # Not sure whether we need this check:
   1562         if shared_x and shared_y:
-> 1563             raise RuntimeError("adjustable='datalim' is not allowed when both"
   1564                                " axes are shared.")
   1565 

RuntimeError: adjustable='datalim' is not allowed when both axes are shared.

<Figure size 1000x1000 with 3 Axes>

Propagator apparently does not create .h5 file:

In [6]: propagator = XFELPhotonPropagator(parameters=propagation_parameters,
                                  input_path='FELsource_out_0000001.h5',
                                  output_path='prop_out.h5')
        propagator.backengine()
In [7]: prop_analysis=XFELPhotonAnalysis(input_path='prop_out.h5')

 Start initialization.

 Loading wavefront from /mnt/home/rshopa/SIMEX/simex_platform/Playground/prop_out.h5.
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-26-878ea2f958c6> in <module>
----> 1 prop_analysis=XFELPhotonAnalysis(input_path='/mnt/home/rshopa/SIMEX/simex_platform/Playground/prop_out.h5')

~/SIMEX/simex_platform/lib/python3.6/SimEx/Analysis/XFELPhotonAnalysis.py in __init__(self, input_path)
     46 
     47         print("\n Loading wavefront from %s." % (self.input_path))
---> 48         wavefront.load_hdf5(self.input_path)
     49         print(" ... done.")
     50 

~/SIMEX/simex_platform/lib/python3.6/wpg/wavefront.py in load_hdf5(self, file_name)
    182         :type  file_name: string
    183         """
--> 184         self._update_from_dict(utils.load_dict_slash_hdf5(file_name))
    185 
    186     def get_intensity(self, slice_number=None, polarization=None):

~/SIMEX/simex_platform/lib/python3.6/wpg/utils.py in load_dict_slash_hdf5(hdf5_file_name)
     91             out_dict.update({name.encode(): obj.value})
     92 
---> 93     with h5py.File(hdf5_file_name, 'r') as h5_file:
     94         h5_file.visititems(add_item)
     95 

~/miniconda3/envs/xfel/lib/python3.6/site-packages/h5py/_hl/files.py in __init__(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, **kwds)
    392                 fid = make_fid(name, mode, userblock_size,
    393                                fapl, fcpl=make_fcpl(track_order=track_order),
--> 394                                swmr=swmr)
    395 
    396             if swmr_support:

~/miniconda3/envs/xfel/lib/python3.6/site-packages/h5py/_hl/files.py in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
    168         if swmr and swmr_support:
    169             flags |= h5f.ACC_SWMR_READ
--> 170         fid = h5f.open(name, flags, fapl=fapl)
    171     elif mode == 'r+':
    172         fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/h5f.pyx in h5py.h5f.open()

OSError: Unable to open file (unable to open file: name = '/mnt/home/rshopa/SIMEX/simex_platform/Playground/prop_out.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

As an alternative, I loaded it from tests and tried to visualise:

In [11]: prop_analysis=XFELPhotonAnalysis(input_path='***/simex_platform/Tests/python/unittest/TestFiles/prop_out/prop_out_0000001.h5')

 Start initialization.

 Loading wavefront from ***/simex_platform/Tests/python/unittest/TestFiles/prop_out/prop_out_0000001.h5.
 ... done.

 Getting intensities.
 ... done.
 Data dimensions =  (16, 16, 550)

 Masking NANs.
 ... done.

In [12]: prop_analysis.plotTotalPower()

Plotting total power.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-28-32cf5cebdb4d> in <module>
----> 1 prop_analysis.plotTotalPower()

~/SIMEX/simex_platform/lib/python3.6/SimEx/Analysis/XFELPhotonAnalysis.py in plotTotalPower(self, spectrum)
    246         # Get meaningful slices.
    247         aw = [a[0] for a in numpy.argwhere(int0 > int0max*0.01)]
--> 248         int0_mean = int0[min(aw):max(aw)]  # meaningful range of pulse
    249         dSlice = (mesh.sliceMax - mesh.sliceMin)/(mesh.nSlices - 1)
    250         xs = numpy.arange(mesh.nSlices)*dSlice+ mesh.sliceMin

ValueError: min() arg is an empty sequence

<Figure size 432x288 with 0 Axes>

So, it looks like there is a major problem with dependencies, paths to libraries or regarding the installation itself.

@CFGrote
Copy link
Collaborator

CFGrote commented Apr 9, 2019

your prop_out_*.h5 file has suspiciously few transverse grid points. how did you generate it? most probably, the propagation failed and all entries are 0 or NaN, hence min() and max() fail as well.

@rshopa
Copy link

rshopa commented Apr 11, 2019

I installed git-lfs and loaded .h5 file by git checkout (apparently no more errors while testing). Yet the problems remain. Maybe it's better to solve them one by one.

I follow the example from the notebook, omitting # Cleanup previous run. Why the notebook breaks on the line source_analysis.plotTotalPower(spectrum=True)?

In [1]: # Import all SimEx modules
        import SimEx
        from SimEx import *
        source_analysis = XFELPhotonAnalysis(input_path="FELsource_out_0000001.h5")
 Start initialization.

 Loading wavefront from FELsource_out_0000001.h5.
 ... done.

 Getting intensities.
 ... done.
 Data dimensions =  (104, 104, 651)

 Masking NANs.
 ... done.

In [2]: source_analysis.plotTotalPower()

Plotting total power.
Pulse energy 0.00017 J
# image here is ok just as in the example!

In [3]: source_analysis.plotTotalPower(spectrum=True)
...

Here, a message appears and notebook is down: The kernel appears to have died. It will restart automatically. Without jupyter, in mere bash, it's segmentation fault (core dumped) error. The same happens for prop_analysis.plotTotalPower(spectrum=True)

@CFGrote
Copy link
Collaborator

CFGrote commented Apr 11, 2019 via email

@rshopa
Copy link

rshopa commented Apr 11, 2019 •
edited
Loading

[UPD]

Thank you, Carsten, I'll try: have no options for python3 on cluster except anaconda.
Apparently, the solution you mentioned would remove MKL. Apparently, it AFFECTS the rest of the framework.

$ conda install nomkl numpy scipy scikit-learn numexpr
$ conda remove mkl mkl-service

$ python Test.py -v 2>&1 | tee Test.log
...
testBackengine (SimExTest.Calculators.FEFFPhotonMatterInteractorTest.FEFFPhotonMatterInteractorTest)
Test the backengine execution. ... feff85L: error while loading shared libraries: libmkl_gf_lp64.so: cannot open shared object file: No such file or directory
ERROR
... # and so on...

The error in the notebook for source_analysis.plotTotalPower(spectrum=True) disappeared, but the very next line returns an error:

In [7]: source_analysis.plotIntensityMap()
Plotting intensity map.
R-space
Error in callback <function install_repl_displayhook.<locals>.post_execute at 0x7f685894abf8> (for post_execute):
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/pyplot.py in post_execute()
    107             def post_execute():
    108                 if matplotlib.is_interactive():
--> 109                     draw_all()
    110 
    111             # IPython >= 2

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/_pylab_helpers.py in draw_all(cls, force)
    130         for f_mgr in cls.get_all_fig_managers():
    131             if force or f_mgr.canvas.figure.stale:
--> 132                 f_mgr.canvas.draw_idle()
    133 
    134 atexit.register(Gcf.destroy_all)

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/backend_bases.py in draw_idle(self, *args, **kwargs)
   1897         if not self._is_idle_drawing:
   1898             with self._idle_draw_cntx():
-> 1899                 self.draw(*args, **kwargs)
   1900 
   1901     def draw_cursor(self, event):

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py in draw(self)
    400         toolbar = self.toolbar
    401         try:
--> 402             self.figure.draw(self.renderer)
    403             # A GUI class may be need to update a window using this draw, so
    404             # don't forget to call the superclass.

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     48                 renderer.start_filter()
     49 
---> 50             return draw(artist, renderer, *args, **kwargs)
     51         finally:
     52             if artist.get_agg_filter() is not None:

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/figure.py in draw(self, renderer)
   1647 
   1648             mimage._draw_list_compositing_images(
-> 1649                 renderer, self, artists, self.suppressComposite)
   1650 
   1651             renderer.close_group('figure')

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/image.py in _draw_list_compositing_images(renderer, parent, artists, suppress_composite)
    136     if not_composite or not has_images:
    137         for a in artists:
--> 138             a.draw(renderer)
    139     else:
    140         # Composite any adjacent images together

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     48                 renderer.start_filter()
     49 
---> 50             return draw(artist, renderer, *args, **kwargs)
     51         finally:
     52             if artist.get_agg_filter() is not None:

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/axes/_base.py in draw(self, renderer, inframe)
   2573             self.apply_aspect(pos)
   2574         else:
-> 2575             self.apply_aspect()
   2576 
   2577         artists = self.get_children()

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/axes/_base.py in apply_aspect(self, position)
   1561         # Not sure whether we need this check:
   1562         if shared_x and shared_y:
-> 1563             raise RuntimeError("adjustable='datalim' is not allowed when both"
   1564                                " axes are shared.")
   1565 

RuntimeError: adjustable='datalim' is not allowed when both axes are shared.

<Figure size 432x288 with 0 Axes>
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/miniconda3/envs/xfel/lib/python3.6/site-packages/IPython/core/formatters.py in __call__(self, obj)
    339                 pass
    340             else:
--> 341                 return printer(obj)
    342             # Finally look for special method names
    343             method = get_real_method(obj, self.print_method)

~/miniconda3/envs/xfel/lib/python3.6/site-packages/IPython/core/pylabtools.py in <lambda>(fig)
    242 
    243     if 'png' in formats:
--> 244         png_formatter.for_type(Figure, lambda fig: print_figure(fig, 'png', **kwargs))
    245     if 'retina' in formats or 'png2x' in formats:
    246         png_formatter.for_type(Figure, lambda fig: retina_figure(fig, **kwargs))

~/miniconda3/envs/xfel/lib/python3.6/site-packages/IPython/core/pylabtools.py in print_figure(fig, fmt, bbox_inches, **kwargs)
    126 
    127     bytes_io = BytesIO()
--> 128     fig.canvas.print_figure(bytes_io, **kw)
    129     data = bytes_io.getvalue()
    130     if fmt == 'svg':

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/backend_bases.py in print_figure(self, filename, dpi, facecolor, edgecolor, orientation, format, bbox_inches, **kwargs)
   2047                         orientation=orientation,
   2048                         dryrun=True,
-> 2049                         **kwargs)
   2050                     renderer = self.figure._cachedRenderer
   2051                     bbox_artists = kwargs.pop("bbox_extra_artists", None)

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py in print_png(self, filename_or_obj, *args, **kwargs)
    508 
    509         """
--> 510         FigureCanvasAgg.draw(self)
    511         renderer = self.get_renderer()
    512 

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py in draw(self)
    400         toolbar = self.toolbar
    401         try:
--> 402             self.figure.draw(self.renderer)
    403             # A GUI class may be need to update a window using this draw, so
    404             # don't forget to call the superclass.

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     48                 renderer.start_filter()
     49 
---> 50             return draw(artist, renderer, *args, **kwargs)
     51         finally:
     52             if artist.get_agg_filter() is not None:

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/figure.py in draw(self, renderer)
   1647 
   1648             mimage._draw_list_compositing_images(
-> 1649                 renderer, self, artists, self.suppressComposite)
   1650 
   1651             renderer.close_group('figure')

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/image.py in _draw_list_compositing_images(renderer, parent, artists, suppress_composite)
    136     if not_composite or not has_images:
    137         for a in artists:
--> 138             a.draw(renderer)
    139     else:
    140         # Composite any adjacent images together

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/artist.py in draw_wrapper(artist, renderer, *args, **kwargs)
     48                 renderer.start_filter()
     49 
---> 50             return draw(artist, renderer, *args, **kwargs)
     51         finally:
     52             if artist.get_agg_filter() is not None:

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/axes/_base.py in draw(self, renderer, inframe)
   2573             self.apply_aspect(pos)
   2574         else:
-> 2575             self.apply_aspect()
   2576 
   2577         artists = self.get_children()

~/miniconda3/envs/xfel/lib/python3.6/site-packages/matplotlib/axes/_base.py in apply_aspect(self, position)
   1561         # Not sure whether we need this check:
   1562         if shared_x and shared_y:
-> 1563             raise RuntimeError("adjustable='datalim' is not allowed when both"
   1564                                " axes are shared.")
   1565 

RuntimeError: adjustable='datalim' is not allowed when both axes are shared.

<Figure size 1000x1000 with 3 Axes>
​

@rshopa
Copy link

rshopa commented May 7, 2019

I ran the example several times both on cluster and on laptop (docker container) and noticed that the impulse /Tests/python/unittest/TestFiles/FELsource_out/FELsource_out_0000001.h5 differs from the one presented here. For instance, Data dimensions = (21, 21, 550) is much smaller (as was for prop_out_0000001.h5 mentioned above).

Could it be the cause of the error on source_analysis.plotIntensityMap()? Apparently .h5 files in /Tests/python/unittest/TestFiles/ are somewhat deprecated.

image
image

@CFGrote
Copy link
Collaborator

CFGrote commented May 7, 2019 via email

@CFGrote
Copy link
Collaborator

CFGrote commented May 7, 2019

hi roman,
looks like i will have to recreate my simex environment on the desy cluster, it was removed during my absence from XFEL. that will take bit of time, hope that is not too big a setback for you.

best regards, carsten

@rshopa
Copy link

rshopa commented May 10, 2019 •
edited
Loading

Hi Carsten,
Did you mean that .h5 files from Tests are corrupted (by "simex environment on the desy cluster")? Or it concerns the whole repository?

I ask because there are errors in multiple examples from wiki, e.g. the very first example from a year-old workshop:
image

@CFGrote
Copy link
Collaborator

CFGrote commented May 10, 2019 via email

@CFGrote
Copy link
Collaborator

CFGrote commented Jun 13, 2019

hi roman,
it took me a while, but now i have an up-to-date simex installation on the desy cluster, again. the notebook https://github.com/eucall-software/simex_notebooks/blob/master/start_to_end_demo.ipynb goes through without problems and produces reasonable results (diffraction patterns). i used this file as FELsource output:
https://desycloud.desy.de/index.php/s/oq76Zfs9P6KiqFd

can you try again?

@rshopa
Copy link

rshopa commented Jun 14, 2019

Hi. Carsten.
Thanks a lot. I'll check this as soon as possible.

@rshopa
Copy link

rshopa commented Jun 18, 2019

Hi, Carsten.
Apparently there's a major problem with the installation - none of the reported problems disappeared.

The line from the notebook In [9]: source_analysis.plotIntensityMap() still doesn't work. An error occurs which ends as RuntimeError: adjustable='datalim' is not allowed when both axes are shared (I mentioned this in the very first comment).

The further error is as previously: propagator.backengine() returns 255 and no prop_out.h5 file is created:

In [11]: propagation_parameters = WavePropagatorParameters(beamline=exfel_spb_kb_beamline)
In [12]: propagator = XFELPhotonPropagator(parameters=propagation_parameters,
                                           input_path='5keV_9fs_2015_slice12_fromYoon2016.h5',
                                           output_path='prop_out.h5')
In [13]: propagator.backengine()
Out [13]: 255
...

@CFGrote
Copy link
Collaborator

CFGrote commented Jun 18, 2019 via email

@rshopa
Copy link

rshopa commented Jun 19, 2019

Hi Carsten.

Here is the description of the environment I use on CIŚ cluster. :

System: Scientific Linux CERN SLC release 6.8 (Carbon)
Memory (free).......: 46.18GB / 47.10GB (98 %)

$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                24
On-line CPU(s) list:   0-23
Thread(s) per core:    1
Core(s) per socket:    12
Socket(s):             2
NUMA node(s):          4
Vendor ID:             AuthenticAMD
CPU family:            16
Model:                 9
Model name:            AMD Opteron(tm) Processor 6174
Stepping:              1
CPU MHz:               2199.930
BogoMIPS:              4400.08
Virtualization:        AMD-V
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K
L3 cache:              5118K
NUMA node0 CPU(s):     0,2,4,6,8,10
NUMA node1 CPU(s):     12,14,16,18,20,22
NUMA node2 CPU(s):     13,15,17,19,21,23
NUMA node3 CPU(s):     1,3,5,7,9,11

Python:

Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34) 
[GCC 7.3.0] on linux

Since I use anaconda, I removed mkl, as you advised (unit tests would fail):

$ conda install nomkl numpy scipy scikit-learn numexpr
$ conda remove mkl mkl-service

gcc (I might have used higher version for the compilation):

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/mnt/opt/tools/slc6/gcc/4.6.3/libexec/gcc/x86_64-unknown-linux-gnu/4.6.3/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: ../gcc-4.6.3/configure --enable-languages=c,c++,fortran --prefix=/mnt/opt/tools/slc6/gcc/4.6.3
Thread model: posix
gcc version 4.6.3 (GCC)

Here is the list of other modules loaded on cluster:

hdf5/1.8.17.patch1
blas/3.5.0-x86_64-gcc46
bzip2/1.0.6
cuda/4.2.9
lapack/3.5.0-x86_64-gcc46
boost/1.64.0-x86_64-gcc71
binutils/2.22
fftw/3.3.4-x86_64-gcc482
intel-composer/2016.2.181 # for mkl, but I turned it off

Full list of conda modules:

$ conda list
# packages in environment at /mnt/home/rshopa/miniconda3/envs/xfel:
#
# Name                    Version                   Build  Channel
alabaster                 0.7.12                   py36_0  
asn1crypto                0.24.0                   py36_0  
attrs                     19.1.0                   py36_1  
babel                     2.6.0                    py36_0  
backcall                  0.1.0                    py36_0  
biopython                 1.72             py36h04863e7_0  
blas                      2.4                    openblas    conda-forge
bleach                    3.1.0                    py36_0  
breathe                   4.12.0                     py_0    conda-forge
ca-certificates           2019.5.15                     0  
certifi                   2019.3.9                 py36_0  
cffi                      1.12.2           py36h2e261b9_1  
chardet                   3.0.4                    py36_1  
cryptography              2.6.1            py36h1ba5d50_0  
cycler                    0.10.0                   py36_0  
cython                    0.29.6           py36he6710b0_0  
dbus                      1.13.6               h746ee38_0  
decorator                 4.4.0                      py_0  
defusedxml                0.5.0                    py36_1  
dill                      0.2.9                    py36_0  
docutils                  0.14                     py36_0  
entrypoints               0.3                      py36_0  
expat                     2.2.6                he6710b0_0  
fabio                     0.8.0           py36h3010b51_1000    conda-forge
fontconfig                2.13.0               h9420a91_0  
freetype                  2.9.1                h8a8886c_1  
glib                      2.56.2               hd408876_0  
gmp                       6.1.2                h6c8ec71_1  
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb453b48_1  
h5py                      2.9.0            py36h7918eee_0  
hdf5                      1.10.4               hb1b8bf9_0  
icu                       58.2                 h9c2bf20_1  
idna                      2.8                      py36_0  
imagesize                 1.1.0                    py36_0  
intel-openmp              2019.1                      144  
ipykernel                 5.1.0            py36h39e3cac_0  
ipython                   7.3.0            py36h39e3cac_0  
ipython_genutils          0.2.0                    py36_0  
ipywidgets                7.4.2                    py36_0  
jedi                      0.13.3                   py36_0  
jinja2                    2.10                     py36_0  
joblib                    0.13.2                   py36_0  
jpeg                      9b                   h024ee3a_2  
jsonschema                3.0.1                    py36_0  
jupyter                   1.0.0                    py36_7  
jupyter_client            5.2.4                    py36_0  
jupyter_console           6.0.0                    py36_0  
jupyter_core              4.4.0                    py36_0  
kiwisolver                1.0.1            py36hf484d3e_0  
libblas                   3.8.0                4_openblas    conda-forge
libcblas                  3.8.0                4_openblas    conda-forge
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 8.2.0                hdf63c60_1  
libgfortran-ng            7.3.0                hdf63c60_0  
liblapack                 3.8.0                4_openblas    conda-forge
liblapacke                3.8.0                4_openblas    conda-forge
libopenblas               0.3.3                h5a2b251_3  
libpng                    1.6.36               hbc83047_0  
libsodium                 1.0.16               h1bed415_0  
libstdcxx-ng              8.2.0                hdf63c60_1  
libtiff                   4.0.10               h2733197_2  
libuuid                   1.0.3                h1bed415_2  
libxcb                    1.13                 h1bed415_1  
libxml2                   2.9.9                he19cac6_0  
libxslt                   1.1.33               h7d1a2b0_0  
llvmlite                  0.28.0           py36hd408876_0  
lxml                      4.3.2            py36hefd8a0e_0  
mako                      1.0.7                    py36_0  
markupsafe                1.1.1            py36h7b6447c_0  
matplotlib                3.0.3            py36h5429711_0  
mistune                   0.8.4            py36h7b6447c_0  
mpi4py                    2.0.0                    py36_2  
mpich2                    1.4.1p1                       0  
nbconvert                 5.4.1                    py36_3  
nbformat                  4.4.0                    py36_0  
ncurses                   6.1                  he6710b0_1  
nomkl                     3.0                           0  
notebook                  5.7.8                    py36_0  
numba                     0.43.0           py36h962f231_0  
numexpr                   2.6.9            py36h2ffa06c_0  
numpy                     1.16.4           py36h99e49ec_0  
numpy-base                1.16.4           py36h2f8d375_0  
nvidia-ml-py3             7.352.0                    py_0    fastai
olefile                   0.46                     py36_0  
openblas                  0.3.5             h9ac9557_1001    conda-forge
openssl                   1.1.1c               h7b6447c_1  
packaging                 19.0                     py36_0  
pandoc                    2.2.3.2                       0  
pandocfilters             1.4.2                    py36_1  
parso                     0.3.4                    py36_0  
pcre                      8.43                 he6710b0_0  
periodictable             1.5.0                      py_1    conda-forge
pexpect                   4.6.0                    py36_0  
pickleshare               0.7.5                    py36_0  
pillow                    5.4.1            py36h34e0f95_0  
pint                      0.9                      py36_2    conda-forge
pip                       19.0.3                   py36_0  
prometheus_client         0.6.0                    py36_0  
prompt_toolkit            2.0.9                    py36_0  
ptyprocess                0.6.0                    py36_0  
py3nvml                   0.2.3                     <pip>
pycparser                 2.19                     py36_0  
pyfai                     0.17.0          py36hf2d7682_1001    conda-forge
pyfftw                    0.11.1          py36h3010b51_1001    conda-forge
pygments                  2.3.1                    py36_0  
pyopenssl                 19.0.0                   py36_0  
pyparsing                 2.3.1                    py36_0  
pyqt                      5.9.2            py36h05f1152_2  
pyrsistent                0.14.11          py36h7b6447c_0  
pysocks                   1.6.8                    py36_0  
python                    3.6.8                h0371630_0  
python-dateutil           2.8.0                    py36_0  
pytz                      2018.9                   py36_0  
pyzmq                     18.0.0           py36he6710b0_0  
qt                        5.9.7                h5867ecd_1  
qtconsole                 4.4.3                    py36_0  
readline                  7.0                  h7b6447c_5  
requests                  2.21.0                   py36_0  
scikit-learn              0.21.2           py36h22eb022_0  
scipy                     1.2.1            py36he2b7bc3_0  
send2trash                1.5.0                    py36_0  
setuptools                40.8.0                   py36_0  
silx                      0.10.0           py36h637b7d7_0    conda-forge
sip                       4.19.8           py36hf484d3e_0  
six                       1.12.0                   py36_0  
snowballstemmer           1.2.1                    py36_0  
sphinx                    1.8.5                    py36_0  
sphinxcontrib             1.0                      py36_1  
sphinxcontrib-websupport  1.1.0                    py36_1  
sqlalchemy                1.3.1            py36h7b6447c_0  
sqlite                    3.27.2               h7b6447c_0  
terminado                 0.8.1                    py36_1  
testpath                  0.4.2                    py36_0  
tk                        8.6.8                hbc83047_0  
tornado                   6.0.1            py36h7b6447c_0  
traitlets                 4.3.2                    py36_0  
urllib3                   1.24.1                   py36_0  
wcwidth                   0.1.7                    py36_0  
webencodings              0.5.1                    py36_1  
wheel                     0.33.1                   py36_0  
widgetsnbextension        3.4.2                    py36_0  
xmltodict                 0.12.0                    <pip>
xz                        5.2.4                h14c3975_4  
zeromq                    4.3.1                he6710b0_3  
zlib                      1.2.11               h7b6447c_3  
zstd                      1.3.7                h0b5b093_0

@CFGrote
Copy link
Collaborator

CFGrote commented Jun 19, 2019 via email

@CFGrote
Copy link
Collaborator

CFGrote commented Jun 19, 2019 via email

@CFGrote
Copy link
Collaborator

CFGrote commented Jun 19, 2019 via email

@rshopa
Copy link

rshopa commented Jun 20, 2019

Hi Carsten,
Thank you for the hints.
Probably I have to discuss the latest issue with our admins.

It still seems odd for me that memory issues cause error at propagator.backengine() stage.

@CFGrote
Copy link
Collaborator

CFGrote commented Jun 20, 2019 via email

@rshopa
Copy link

rshopa commented Jul 8, 2019 •
edited
Loading

Hi Carsten,
The problem with .backengine() functions was related to open MPI (apparently I forgot to add the library to conda environment). It disappears after I loaded the module.
The example appears to run ok (haven't updated to v0.4.1 to test the issue with matplotlib), but with little difference in images. Since I yet have to dive into the physics, I don't know whether it is consistent or not. Here's some screenshots:
image
image

Thank you again for the help and for the enormous patience!

I hope that the examples from wiki would work, as well.

@CFGrote
Copy link
Collaborator

CFGrote commented Jul 8, 2019 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@n01r n01r

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@n01r @CFGrote @rshopa