-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OpenMP: kernel died every time #117
Comments
Please send us example of your script.
Do you use default python binaries or some like Anaconda installers?
…On Mon, Oct 9, 2017 at 9:40 AM, Yajun Tong ***@***.***> wrote:
when the *.ipynb run , the kernel is died.
when the *.py run, it crash with "segmentation fault".
and the srw example can run without any error.
The OS is Ubuntu 16.04,
python version is 3.6,
thank you!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#117>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/AA6Pt9aTRUCZ7OYNl3DLS4uUj8wPplMaks5sqb_zgaJpZM4PyDab>
.
|
the Python is from Anaconda. and I just run the Tutorials example. |
Hi, this is known issue with anaconda numpy MKL optimizations and SRW core.
Please remove mkl optimizations and try run tutorials.
conda install nomkl numpy scipy scikit-learn numexpr
conda remove mkl mkl-service
…On Sat, Oct 14, 2017 at 4:36 PM, Yajun Tong ***@***.***> wrote:
the Python is from Anaconda. and I just run the Tutorials example.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#117 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AA6Pt7bxV_6qMpa8vb3bHfcypqVgAorEks5ssLjcgaJpZM4PyDab>
.
|
Thank you very much! I install nomkl packeges and it works now. |
I installed the newest master today following installation instructions, in particular the I attach the core dump:
|
Same observation when running under plain python 3.7 without anaconda. |
with the wpg_cryst_branch installation, I got the following error message
after the 3d S1_SPB_CRL_simplified.ipynb cell (where wpg and srw modules
are imported)
It worked properly in December...
…---------------------------------------------------------------------------ImportError
Traceback (most recent call
last)<ipython-input-4-3fa6b3b9fcd5> in <module> 11 12 ---> 13
from wpg import Wavefront, Beamline 14 from wpg.optical_elements
import Aperture, Drift, CRL, Empty, Use_PP 15 from wpg.generators
import build_gauss_wavefront
/gpfs/exfel/data/user/lsamoylv/public/wpg_cryst_branch/wpg/__init__.py
in <module> 25 26 # fix segmentation fault using fftw from
numpy mkl---> 27 from .srw import srwlpy 28 29 # Create
aliases for simple importing
ImportError: cannot import name 'srwlpy' from 'wpg.srw'
(../../../wpg/srw/__init__.py)
On Fri, Jan 24, 2020 at 5:43 AM Carsten Fortmann-Grote < ***@***.***> wrote:
Same observation when running under plain python 3.7 without anaconda.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=AAOFOKOZA47ZA2SEN4XZ3ELQ7JWXLA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJZWSAI#issuecomment-577988865>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAOFOKLUDKSFX5KBQ2C3XX3Q7JWXLANCNFSM4D6IG2NQ>
.
|
Could you please print output of "conda list" output from your working
conda environment?
AB
…On Mon, Jan 27, 2020 at 7:11 PM samoylv ***@***.***> wrote:
with the wpg_cryst_branch installation, I got the following error message
after the 3d S1_SPB_CRL_simplified.ipynb cell (where wpg and srw modules
are imported)
It worked properly in December...
---------------------------------------------------------------------------ImportError
Traceback (most recent call
last)<ipython-input-4-3fa6b3b9fcd5> in <module> 11 12 ---> 13
from wpg import Wavefront, Beamline 14 from wpg.optical_elements
import Aperture, Drift, CRL, Empty, Use_PP 15 from wpg.generators
import build_gauss_wavefront
/gpfs/exfel/data/user/lsamoylv/public/wpg_cryst_branch/wpg/__init__.py
in <module> 25 26 # fix segmentation fault using fftw from
numpy mkl---> 27 from .srw import srwlpy 28 29 # Create
aliases for simple importing
ImportError: cannot import name 'srwlpy' from 'wpg.srw'
(../../../wpg/srw/__init__.py)
On Fri, Jan 24, 2020 at 5:43 AM Carsten Fortmann-Grote <
***@***.***> wrote:
> Same observation when running under plain python 3.7 without anaconda.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <
#117?email_source=notifications&email_token=AAOFOKOZA47ZA2SEN4XZ3ELQ7JWXLA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJZWSAI#issuecomment-577988865
>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AAOFOKLUDKSFX5KBQ2C3XX3Q7JWXLANCNFSM4D6IG2NQ
>
> .
>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=AAHI7N24LII5KB2A2IUNZKLQ74BUFA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKACDCA#issuecomment-578822536>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAHI7N7FDH2L6VDKW3CTLC3Q74BUFANCNFSM4D6IG2NQ>
.
|
# packages in environment at /gpfs/exfel/data/user/buzmakov/conda_env/wpg3:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
alabaster 0.7.12 py37_0
asn1crypto 1.2.0 py37_0
attrs 19.3.0 py_0
autopep8 1.4.4 py_0
babel 2.7.0 py_0
backcall 0.1.0 py37_0
blas 1.0 openblas
bleach 3.1.0 py37_0
ca-certificates 2019.11.27 0
certifi 2019.11.28 py37_0
cffi 1.13.2 py37h2e261b9_0
chardet 3.0.4 py37_1003
cryptography 2.3.1 py37hc365091_0
cycler 0.10.0 py37_0
dbus 1.13.12 h746ee38_0
decorator 4.4.1 py_0
defusedxml 0.6.0 py_0
docutils 0.15.2 py37_0
entrypoints 0.3 py37_0
expat 2.2.6 he6710b0_0
fontconfig 2.13.0 h9420a91_0
freetype 2.9.1 h8a8886c_1
git 2.19.1 pl526h7fee0ce_0
glib 2.63.1 h5a9c865_0
gmp 6.1.2 h6c8ec71_1
gprof2dot 2017.9.19 py_0 conda-forge
gst-plugins-base 1.14.0 hbbd80ab_1
gstreamer 1.14.0 hb453b48_1
h5py 2.9.0 py37h7918eee_0
hdf5 1.10.4 hb1b8bf9_0
icu 58.2 h9c2bf20_1
idna 2.8 py37_0
imagesize 1.1.0 py37_0
importlib_metadata 1.2.0 py37_0
ipykernel 5.1.3 py37h39e3cac_0
ipython 7.10.1 py37h39e3cac_0
ipython_genutils 0.2.0 py37_0
ipywidgets 7.5.1 py_0
jedi 0.15.1 py37_0
jinja2 2.10.3 py_0
jpeg 9b h024ee3a_2
jsonschema 3.2.0 py37_0
jupyter 1.0.0 py37_7
jupyter_client 5.3.4 py37_0
jupyter_console 6.0.0 py37_0
jupyter_core 4.6.1 py37_0
kiwisolver 1.1.0 py37he6710b0_0
libcurl 7.61.1 heec0ca6_0
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 hd88cf55_4
libgcc 7.2.0 h69d50b8_2
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
libopenblas 0.3.6 h5a2b251_2
libpng 1.6.37 hbc83047_0
libsodium 1.0.16 h1bed415_0
libssh2 1.8.0 h9cfc8f7_4
libstdcxx-ng 9.1.0 hdf63c60_0
libtiff 4.1.0 h2733197_0
libuuid 1.0.3 h1bed415_2
libxcb 1.13 h1bed415_1
libxml2 2.9.9 hea5a465_1
markupsafe 1.1.1 py37h7b6447c_0
matplotlib 3.1.1 py37h5429711_0
mistune 0.8.4 py37h7b6447c_0
mock 3.0.5 py37_0
more-itertools 7.2.0 py37_0
nbconvert 5.6.1 py37_0
nbformat 4.4.0 py37_0
ncurses 6.1 he6710b0_1
nomkl 3.0 0
notebook 6.0.2 py37_0
numpy 1.17.4 py37hd5be1e1_0
numpy-base 1.17.4 py37h2f8d375_0
olefile 0.46 py37_0
openssl 1.0.2t h7b6447c_1
packaging 19.2 py_0
pandoc 2.2.3.2 0
pandocfilters 1.4.2 py37_1
parso 0.5.1 py_0
pcre 8.43 he6710b0_0
perl 5.26.2 h14c3975_0
pexpect 4.7.0 py37_0
pickleshare 0.7.5 py37_0
pillow 6.2.1 py37h34e0f95_0
pip 19.3.1 py37_0
pluggy 0.13.1 py37_0
prometheus_client 0.7.1 py_0
prompt_toolkit 2.0.10 py_0
ptyprocess 0.6.0 py37_0
py 1.8.0 py37_0
pycodestyle 2.5.0 py37_0
pycparser 2.19 py37_0
pygments 2.5.2 py_0
pyopenssl 19.0.0 py37_0
pyparsing 2.4.5 py_0
pyqt 5.9.2 py37h05f1152_2
pyrsistent 0.15.6 py37h7b6447c_0
pysocks 1.7.1 py37_0
pytest 5.3.1 py37_0
pytest-profiling 1.7.0 py_1 conda-forge
python 3.7.0 h6e4f718_3
python-dateutil 2.8.1 py_0
pytz 2019.3 py_0
pyzmq 18.1.0 py37he6710b0_0
qt 5.9.6 h8703b6f_2
qtconsole 4.6.0 py_0
readline 7.0 h7b6447c_5
requests 2.22.0 py37_1
scipy 1.3.2 py37he2b7bc3_0
send2trash 1.5.0 py37_0
setuptools 42.0.2 py37_0
sip 4.19.8 py37hf484d3e_0
six 1.13.0 py37_0
snowballstemmer 2.0.0 py_0
sphinx 2.2.2 py_0
sphinxcontrib-applehelp 1.0.1 py_0
sphinxcontrib-devhelp 1.0.1 py_0
sphinxcontrib-htmlhelp 1.0.2 py_0
sphinxcontrib-jsmath 1.0.1 py_0
sphinxcontrib-qthelp 1.0.2 py_0
sphinxcontrib-serializinghtml 1.1.3 py_0
sqlite 3.30.1 h7b6447c_0
terminado 0.8.3 py37_0
testpath 0.4.4 py_0
tk 8.6.8 hbc83047_0
tornado 6.0.3 py37h7b6447c_0
traitlets 4.3.3 py37_0
urllib3 1.25.7 py37_0
wcwidth 0.1.7 py37_0
webencodings 0.5.1 py37_1
wheel 0.33.6 py37_0
widgetsnbextension 3.5.1 py37_0
xz 5.2.4 h14c3975_4
zeromq 4.3.1 he6710b0_3
zipp 0.6.0 py_0
zlib 1.2.11 h7b6447c_3
zstd 1.3.7 h0b5b093_0
Note: you may need to restart the kernel to use updated packages.
On Mon, Jan 27, 2020 at 5:22 PM Alexey Buzmakov <[email protected]>
wrote:
… Could you please print output of "conda list" output from your working
conda environment?
AB
On Mon, Jan 27, 2020 at 7:11 PM samoylv ***@***.***> wrote:
> with the wpg_cryst_branch installation, I got the following error message
> after the 3d S1_SPB_CRL_simplified.ipynb cell (where wpg and srw modules
> are imported)
> It worked properly in December...
>
>
>
---------------------------------------------------------------------------ImportError
> Traceback (most recent call
> last)<ipython-input-4-3fa6b3b9fcd5> in <module> 11 12 ---> 13
> from wpg import Wavefront, Beamline 14 from wpg.optical_elements
> import Aperture, Drift, CRL, Empty, Use_PP 15 from wpg.generators
> import build_gauss_wavefront
> /gpfs/exfel/data/user/lsamoylv/public/wpg_cryst_branch/wpg/__init__.py
> in <module> 25 26 # fix segmentation fault using fftw from
> numpy mkl---> 27 from .srw import srwlpy 28 29 # Create
> aliases for simple importing
> ImportError: cannot import name 'srwlpy' from 'wpg.srw'
> (../../../wpg/srw/__init__.py)
>
>
> On Fri, Jan 24, 2020 at 5:43 AM Carsten Fortmann-Grote <
> ***@***.***> wrote:
>
> > Same observation when running under plain python 3.7 without anaconda.
> >
> > —
> > You are receiving this because you are subscribed to this thread.
> > Reply to this email directly, view it on GitHub
> > <
>
#117?email_source=notifications&email_token=AAOFOKOZA47ZA2SEN4XZ3ELQ7JWXLA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJZWSAI#issuecomment-577988865
> >,
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/AAOFOKLUDKSFX5KBQ2C3XX3Q7JWXLANCNFSM4D6IG2NQ
> >
> > .
> >
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <
#117?email_source=notifications&email_token=AAHI7N24LII5KB2A2IUNZKLQ74BUFA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKACDCA#issuecomment-578822536
>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AAHI7N7FDH2L6VDKW3CTLC3Q74BUFANCNFSM4D6IG2NQ
>
> .
>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=AAOFOKKRC5OVJK57DIHBSHLQ74C3RA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKADK5Q#issuecomment-578827638>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAOFOKJDAL5IZI2ZCAUVPPTQ74C3RANCNFSM4D6IG2NQ>
.
|
Here's my output from |
and these are all packages installed in the plain python (no conda) setup where I get the same kernel dying issue: |
but that's a different issue, right? I had the same (see #128) and it got fixed after re-installing everything from scratch. |
Carsten,
I should debug this issue with gdb.
AB
…On Tue, Jan 28, 2020 at 7:33 AM Carsten Fortmann-Grote < ***@***.***> wrote:
with the wpg_cryst_branch installation, I got the following error message
after the 3d S1_SPB_CRL_simplified.ipynb cell (where wpg and srw modules
are imported) It worked properly in December...t
but that's a different issue, right? I had the same (see #128
<#128>) and it got fixed after
re-installing everything from scratch.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=AAHI7N4PMI3RKYHGE2YTNCLQ76YQNA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKB7VHY#issuecomment-579074719>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAHI7N3GC7NRUIRD7O2G6YLQ76YQNANCNFSM4D6IG2NQ>
.
|
good luck ;)
i attempted this once with very unrepreducible results. looked sike a
memory leak to me which kicks in in more or less random fashion.
c
|
i rebuilt wpg on a debian10 linux machine, no problem with latest develop commit. conda environment with nomkl package. so probably some centos problem, maybe compiler? |
UPDATE: still in debuging |
This issue appear only on maxwell server. |
currently i believe that it is not possible to use wpg on maxwell. not
even older versions, as indicated above.
|
Investigation shows that problem in OpenMP inside SRW. |
Seem to be fixed in 'develop' branch (turn of OpenMP by default. See README.md for details). please run following commands: 'git pull; make clean; make' |
but that's not a solution! if openmp is not functional, many
applications become unfeasible. plus, oleg et al just submitted a paper
for SPIE where they advertise openMP.
best, carsten
…On 2/16/20 4:45 PM, Alexey Buzmakov wrote:
Seem to be fixed in 'develop' branch (turn of OpenMP by default. See
README.md for details).
I add S1 beamline as the test for future degradations.
please run following commands: 'git pull; make clean; make'
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=ADWCQRZ2MWWCOA67F5JAKW3RDFNQXA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL4KOFQ#issuecomment-586721046>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADWCQR6LY2JVZGPFP4LYV43RDFNQXANCNFSM4D6IG2NQ>.
|
Liuba,
Can we ask Serguey Yakubov to debug openmp build on maxwell?
пн, 17 февр. 2020 г., 10:53 Carsten Fortmann-Grote <[email protected]
…:
but that's not a solution! if openmp is not functional, many
applications become unfeasible. plus, oleg et al just submitted a paper
for SPIE where they advertise openMP.
best, carsten
On 2/16/20 4:45 PM, Alexey Buzmakov wrote:
> Seem to be fixed in 'develop' branch (turn of OpenMP by default. See
> README.md for details).
> I add S1 beamline as the test for future degradations.
>
> please run following commands: 'git pull; make clean; make'
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <
#117?email_source=notifications&email_token=ADWCQRZ2MWWCOA67F5JAKW3RDFNQXA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL4KOFQ#issuecomment-586721046
>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/ADWCQR6LY2JVZGPFP4LYV43RDFNQXANCNFSM4D6IG2NQ
>.
>
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=AAHI7NYSQ3NKVFKLUT4UAHLRDI67RA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5MNHY#issuecomment-586860191>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAHI7N63KEOI7CRVO5ACC73RDI67RANCNFSM4D6IG2NQ>
.
|
yes, we can! he's also an author on that paper...
…On 2/17/20 9:28 AM, Alexey Buzmakov wrote:
Liuba,
Can we ask Serguey Yakubov to debug openmp build on maxwell?
пн, 17 февр. 2020 г., 10:53 Carsten Fortmann-Grote ***@***.***
>:
> but that's not a solution! if openmp is not functional, many
> applications become unfeasible. plus, oleg et al just submitted a paper
> for SPIE where they advertise openMP.
>
> best, carsten
>
> On 2/16/20 4:45 PM, Alexey Buzmakov wrote:
> > Seem to be fixed in 'develop' branch (turn of OpenMP by default. See
> > README.md for details).
> > I add S1 beamline as the test for future degradations.
> >
> > please run following commands: 'git pull; make clean; make'
> >
> > —
> > You are receiving this because you were mentioned.
> > Reply to this email directly, view it on GitHub
> > <
>
#117?email_source=notifications&email_token=ADWCQRZ2MWWCOA67F5JAKW3RDFNQXA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL4KOFQ#issuecomment-586721046
> >,
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/ADWCQR6LY2JVZGPFP4LYV43RDFNQXANCNFSM4D6IG2NQ
> >.
> >
>
> —
> You are receiving this because you were assigned.
> Reply to this email directly, view it on GitHub
>
<#117?email_source=notifications&email_token=AAHI7NYSQ3NKVFKLUT4UAHLRDI67RA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5MNHY#issuecomment-586860191>,
> or unsubscribe
>
<https://github.com/notifications/unsubscribe-auth/AAHI7N63KEOI7CRVO5ACC73RDI67RANCNFSM4D6IG2NQ>
> .
>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=ADWCQR2VHOQE6UZW4T6YCJDRDJDDLA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5PPPY#issuecomment-586872767>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADWCQR2YVV355V7QVKJ2X2LRDJDDLANCNFSM4D6IG2NQ>.
|
BTW, beamlines (SRW containers) with Drifts only can be run smoothly with
openMP as well (Drift has only one numerical variable, propagation
distance). With any other propagator, kernel died
On Mon, Feb 17, 2020 at 9:32 AM Carsten Fortmann-Grote <
[email protected]> wrote:
… yes, we can! he's also an author on that paper...
On 2/17/20 9:28 AM, Alexey Buzmakov wrote:
> Liuba,
> Can we ask Serguey Yakubov to debug openmp build on maxwell?
>
>
> пн, 17 февр. 2020 г., 10:53 Carsten Fortmann-Grote <
***@***.***
>>:
>
>> but that's not a solution! if openmp is not functional, many
>> applications become unfeasible. plus, oleg et al just submitted a paper
>> for SPIE where they advertise openMP.
>>
>> best, carsten
>>
>> On 2/16/20 4:45 PM, Alexey Buzmakov wrote:
>> > Seem to be fixed in 'develop' branch (turn of OpenMP by default. See
>> > README.md for details).
>> > I add S1 beamline as the test for future degradations.
>> >
>> > please run following commands: 'git pull; make clean; make'
>> >
>> > —
>> > You are receiving this because you were mentioned.
>> > Reply to this email directly, view it on GitHub
>> > <
>>
>
#117?email_source=notifications&email_token=ADWCQRZ2MWWCOA67F5JAKW3RDFNQXA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL4KOFQ#issuecomment-586721046
>> >,
>> > or unsubscribe
>> > <
>>
>
https://github.com/notifications/unsubscribe-auth/ADWCQR6LY2JVZGPFP4LYV43RDFNQXANCNFSM4D6IG2NQ
>> >.
>> >
>>
>> —
>> You are receiving this because you were assigned.
>> Reply to this email directly, view it on GitHub
>>
> <
#117?email_source=notifications&email_token=AAHI7NYSQ3NKVFKLUT4UAHLRDI67RA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5MNHY#issuecomment-586860191
>,
>> or unsubscribe
>>
> <
https://github.com/notifications/unsubscribe-auth/AAHI7N63KEOI7CRVO5ACC73RDI67RANCNFSM4D6IG2NQ
>
>> .
>>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <
#117?email_source=notifications&email_token=ADWCQR2VHOQE6UZW4T6YCJDRDJDDLA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5PPPY#issuecomment-586872767
>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/ADWCQR2YVV355V7QVKJ2X2LRDJDDLANCNFSM4D6IG2NQ
>.
>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=AAOFOKOEN6FCXUDXGKMSAVDRDJDRFA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5P2BA#issuecomment-586874116>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAOFOKOP3R5I6CQLIFNZXP3RDJDRFANCNFSM4D6IG2NQ>
.
|
It is seems that error happens in free() function in srw aperture creation.
пн, 17 февр. 2020 г., 13:14 samoylv <[email protected]>:
… BTW, beamlines (SRW containers) with Drifts only can be run smoothly with
openMP as well (Drift has only one numerical variable, propagation
distance). With any other propagator, kernel died
On Mon, Feb 17, 2020 at 9:32 AM Carsten Fortmann-Grote <
***@***.***> wrote:
> yes, we can! he's also an author on that paper...
>
> On 2/17/20 9:28 AM, Alexey Buzmakov wrote:
> > Liuba,
> > Can we ask Serguey Yakubov to debug openmp build on maxwell?
> >
> >
> > пн, 17 февр. 2020 г., 10:53 Carsten Fortmann-Grote <
> ***@***.***
> >>:
> >
> >> but that's not a solution! if openmp is not functional, many
> >> applications become unfeasible. plus, oleg et al just submitted a
paper
> >> for SPIE where they advertise openMP.
> >>
> >> best, carsten
> >>
> >> On 2/16/20 4:45 PM, Alexey Buzmakov wrote:
> >> > Seem to be fixed in 'develop' branch (turn of OpenMP by default. See
> >> > README.md for details).
> >> > I add S1 beamline as the test for future degradations.
> >> >
> >> > please run following commands: 'git pull; make clean; make'
> >> >
> >> > —
> >> > You are receiving this because you were mentioned.
> >> > Reply to this email directly, view it on GitHub
> >> > <
> >>
> >
>
#117?email_source=notifications&email_token=ADWCQRZ2MWWCOA67F5JAKW3RDFNQXA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL4KOFQ#issuecomment-586721046
> >> >,
> >> > or unsubscribe
> >> > <
> >>
> >
>
https://github.com/notifications/unsubscribe-auth/ADWCQR6LY2JVZGPFP4LYV43RDFNQXANCNFSM4D6IG2NQ
> >> >.
> >> >
> >>
> >> —
> >> You are receiving this because you were assigned.
> >> Reply to this email directly, view it on GitHub
> >>
> > <
>
#117?email_source=notifications&email_token=AAHI7NYSQ3NKVFKLUT4UAHLRDI67RA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5MNHY#issuecomment-586860191
> >,
> >> or unsubscribe
> >>
> > <
>
https://github.com/notifications/unsubscribe-auth/AAHI7N63KEOI7CRVO5ACC73RDI67RANCNFSM4D6IG2NQ
> >
> >> .
> >>
> >
> > —
> > You are receiving this because you were mentioned.
> > Reply to this email directly, view it on GitHub
> > <
>
#117?email_source=notifications&email_token=ADWCQR2VHOQE6UZW4T6YCJDRDJDDLA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5PPPY#issuecomment-586872767
> >,
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/ADWCQR2YVV355V7QVKJ2X2LRDJDDLANCNFSM4D6IG2NQ
> >.
> >
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <
#117?email_source=notifications&email_token=AAOFOKOEN6FCXUDXGKMSAVDRDJDRFA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5P2BA#issuecomment-586874116
>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AAOFOKOP3R5I6CQLIFNZXP3RDJDRFANCNFSM4D6IG2NQ
>
> .
>
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=AAHI7NZ2QPHHXJVIPHDDVG3RDJPRBA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL52M2Q#issuecomment-586917482>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAHI7NY7BZQWGXLMR45T4UDRDJPRBANCNFSM4D6IG2NQ>
.
|
Hi Carsten, could you ask Sergei?
On Mon, 17 Feb 2020, 09:32 Carsten Fortmann-Grote, <[email protected]>
wrote:
… yes, we can! he's also an author on that paper...
On 2/17/20 9:28 AM, Alexey Buzmakov wrote:
> Liuba,
> Can we ask Serguey Yakubov to debug openmp build on maxwell?
>
>
> пн, 17 февр. 2020 г., 10:53 Carsten Fortmann-Grote <
***@***.***
>>:
>
>> but that's not a solution! if openmp is not functional, many
>> applications become unfeasible. plus, oleg et al just submitted a paper
>> for SPIE where they advertise openMP.
>>
>> best, carsten
>>
>> On 2/16/20 4:45 PM, Alexey Buzmakov wrote:
>> > Seem to be fixed in 'develop' branch (turn of OpenMP by default. See
>> > README.md for details).
>> > I add S1 beamline as the test for future degradations.
>> >
>> > please run following commands: 'git pull; make clean; make'
>> >
>> > —
>> > You are receiving this because you were mentioned.
>> > Reply to this email directly, view it on GitHub
>> > <
>>
>
#117?email_source=notifications&email_token=ADWCQRZ2MWWCOA67F5JAKW3RDFNQXA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL4KOFQ#issuecomment-586721046
>> >,
>> > or unsubscribe
>> > <
>>
>
https://github.com/notifications/unsubscribe-auth/ADWCQR6LY2JVZGPFP4LYV43RDFNQXANCNFSM4D6IG2NQ
>> >.
>> >
>>
>> —
>> You are receiving this because you were assigned.
>> Reply to this email directly, view it on GitHub
>>
> <
#117?email_source=notifications&email_token=AAHI7NYSQ3NKVFKLUT4UAHLRDI67RA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5MNHY#issuecomment-586860191
>,
>> or unsubscribe
>>
> <
https://github.com/notifications/unsubscribe-auth/AAHI7N63KEOI7CRVO5ACC73RDI67RANCNFSM4D6IG2NQ
>
>> .
>>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <
#117?email_source=notifications&email_token=ADWCQR2VHOQE6UZW4T6YCJDRDJDDLA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5PPPY#issuecomment-586872767
>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/ADWCQR2YVV355V7QVKJ2X2LRDJDDLANCNFSM4D6IG2NQ
>.
>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=AAOFOKOEN6FCXUDXGKMSAVDRDJDRFA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5P2BA#issuecomment-586874116>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAOFOKOP3R5I6CQLIFNZXP3RDJDRFANCNFSM4D6IG2NQ>
.
|
done (email)
…On 2/17/20 9:57 PM, samoylv wrote:
Hi Carsten, could you ask Sergei?
On Mon, 17 Feb 2020, 09:32 Carsten Fortmann-Grote,
***@***.***>
wrote:
> yes, we can! he's also an author on that paper...
>
> On 2/17/20 9:28 AM, Alexey Buzmakov wrote:
> > Liuba,
> > Can we ask Serguey Yakubov to debug openmp build on maxwell?
> >
> >
> > пн, 17 февр. 2020 г., 10:53 Carsten Fortmann-Grote <
> ***@***.***
> >>:
> >
> >> but that's not a solution! if openmp is not functional, many
> >> applications become unfeasible. plus, oleg et al just submitted a paper
> >> for SPIE where they advertise openMP.
> >>
> >> best, carsten
> >>
> >> On 2/16/20 4:45 PM, Alexey Buzmakov wrote:
> >> > Seem to be fixed in 'develop' branch (turn of OpenMP by default. See
> >> > README.md for details).
> >> > I add S1 beamline as the test for future degradations.
> >> >
> >> > please run following commands: 'git pull; make clean; make'
> >> >
> >> > —
> >> > You are receiving this because you were mentioned.
> >> > Reply to this email directly, view it on GitHub
> >> > <
> >>
> >
>
#117?email_source=notifications&email_token=ADWCQRZ2MWWCOA67F5JAKW3RDFNQXA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL4KOFQ#issuecomment-586721046
> >> >,
> >> > or unsubscribe
> >> > <
> >>
> >
>
https://github.com/notifications/unsubscribe-auth/ADWCQR6LY2JVZGPFP4LYV43RDFNQXANCNFSM4D6IG2NQ
> >> >.
> >> >
> >>
> >> —
> >> You are receiving this because you were assigned.
> >> Reply to this email directly, view it on GitHub
> >>
> > <
>
#117?email_source=notifications&email_token=AAHI7NYSQ3NKVFKLUT4UAHLRDI67RA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5MNHY#issuecomment-586860191
> >,
> >> or unsubscribe
> >>
> > <
>
https://github.com/notifications/unsubscribe-auth/AAHI7N63KEOI7CRVO5ACC73RDI67RANCNFSM4D6IG2NQ
> >
> >> .
> >>
> >
> > —
> > You are receiving this because you were mentioned.
> > Reply to this email directly, view it on GitHub
> > <
>
#117?email_source=notifications&email_token=ADWCQR2VHOQE6UZW4T6YCJDRDJDDLA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5PPPY#issuecomment-586872767
> >,
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/ADWCQR2YVV355V7QVKJ2X2LRDJDDLANCNFSM4D6IG2NQ
> >.
> >
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
>
<#117?email_source=notifications&email_token=AAOFOKOEN6FCXUDXGKMSAVDRDJDRFA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL5P2BA#issuecomment-586874116>,
> or unsubscribe
>
<https://github.com/notifications/unsubscribe-auth/AAOFOKOP3R5I6CQLIFNZXP3RDJDRFANCNFSM4D6IG2NQ>
> .
>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=ADWCQR4CL5J3ZII3VZCHSJLRDL22TA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL7UM7Q#issuecomment-587155070>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADWCQRZT3JR5BIB2VUXXFRTRDL22TANCNFSM4D6IG2NQ>.
|
Hi! |
I can provide a wpg python example @CFGrote : I ran this example at maxwell with anaconda3/5.2 loaded and the following conda_env to resolve wpg/srw imports properly /gpfs/exfel/data/user/buzmakov/conda_env/wpg3 |
ok, I just installed wpg, set PYTHONPATH to it and tried your script with standard python3 on maxwell. Can see the crash. If I set OMP_NUM_THREADS to 1 - no crash. I'll have a look. |
thanks guys!
|
thank you!
…On Tue, Feb 18, 2020 at 11:47 AM Sergey Yakubov ***@***.***> wrote:
ok, I just installed wpg, set PYTHONPATH to it and tried your script with
standard python3 on maxwell.
Can see the crash. If I set OMP_NUM_THREADS to 1 - no crash.
I'll have a look.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=AAOFOKLDE5HMKULLQSKI4XLRDO4D3A5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMBPZRA#issuecomment-587398340>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAOFOKNCPJVOJTXJ2FPKOC3RDO4D3ANCNFSM4D6IG2NQ>
.
|
ups! I can correct it. But where? |
@buzmakov : what do you think? |
Sergey,
we use unmodified SRW from Oleg,
I see 2 simple possibilities:
1) If you create pull request to original SRW, I'll switch to your branch,
until Oleg will not fix it
2) If you send me corrected file, I'll create patch and will apply it
during SRW build.
…On Tue, Feb 18, 2020 at 3:20 PM Sergey Yakubov ***@***.***> wrote:
ups!
there is a memory leak (even two, e.g.
https://github.com/ochubar/SRW/blob/5d3d4cff1e64f6c6587a5731fb94efdb4612c06e/cpp/src/core/sroptel2.cpp#L188).
I used results[threadNum] (with thread number) where results were allocated
with pRadAccessData->ne.
So, if number of threads are large than ne, we have problem. Which was the
case on Maxwell where one can have up to 80 threads.
I can correct it. But where?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=AAHI7NZATI2WZRAIXN2XEUTRDPHCHA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMBYP4Q#issuecomment-587433970>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAHI7NZ7UC3ASEADOZNCEJ3RDPHCHANCNFSM4D6IG2NQ>
.
|
Pull request to original SRW would be better solution...
On Tue, Feb 18, 2020 at 1:26 PM Alexey Buzmakov <[email protected]>
wrote:
… Sergey,
we use unmodified SRW from Oleg,
I see 2 simple possibilities:
1) If you create pull request to original SRW, I'll switch to your branch,
until Oleg will not fix it
2) If you send me corrected file, I'll create patch and will apply it
during SRW build.
On Tue, Feb 18, 2020 at 3:20 PM Sergey Yakubov ***@***.***>
wrote:
> ups!
> there is a memory leak (even two, e.g.
>
>
https://github.com/ochubar/SRW/blob/5d3d4cff1e64f6c6587a5731fb94efdb4612c06e/cpp/src/core/sroptel2.cpp#L188
).
> I used results[threadNum] (with thread number) where results were
allocated
> with pRadAccessData->ne.
> So, if number of threads are large than ne, we have problem. Which was
the
> case on Maxwell where one can have up to 80 threads.
>
> I can correct it. But where?
>
> —
> You are receiving this because you were assigned.
> Reply to this email directly, view it on GitHub
> <
#117?email_source=notifications&email_token=AAHI7NZATI2WZRAIXN2XEUTRDPHCHA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMBYP4Q#issuecomment-587433970
>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AAHI7NZ7UC3ASEADOZNCEJ3RDPHCHANCNFSM4D6IG2NQ
>
> .
>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=AAOFOKLMAYQCAFPQTCEBIHLRDPHZDA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMBZACA#issuecomment-587436040>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAOFOKIUB6XS6MLJBPKQBG3RDPHZDANCNFSM4D6IG2NQ>
.
|
I put the code here: would be nice to run some tests before creating pull request. |
Thnx!
I'll test it on our test cases including real beamlines and push it to
develop branch, if all will be ok.
…On Tue, Feb 18, 2020 at 4:26 PM Sergey Yakubov ***@***.***> wrote:
I put the code here:
https://github.com/SergeyYakubov/SRW/tree/openmp_memoryfix
would be nice to run some tests before creating pull request.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=AAHI7N33QH6AG35HE5UYZJLRDPOWZA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMB6RZY#issuecomment-587458791>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAHI7N3M6B7D5M46FWFFSNLRDPOWZANCNFSM4D6IG2NQ>
.
|
@SergeyYakubov @samoylv @CFGrote I add new SRW to WPG develop branch ant turn on OpenMP by default. |
cool. @Sergey, thanks for the quick fix!
carsten
…On 2/19/20 1:11 PM, Alexey Buzmakov wrote:
@SergeyYakubov <https://github.com/SergeyYakubov> @samoylv
<https://github.com/samoylv> @CFGrote <https://github.com/CFGrote> I add
new SRW to WPG develop branch ant turn on OpenMP by default.
All my tests passed on Maxwell.
Please test in your configurations.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=ADWCQR6JNJHGAVQKBHN3LQ3RDUOWLA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMHRZII#issuecomment-588192929>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADWCQR5CLXEEUGTWJ53476DRDUOWLANCNFSM4D6IG2NQ>.
|
sure, after you bought me with the authorship in this publication ... :) |
exactly :D that's how it works
…On 2/19/20 2:19 PM, Sergey Yakubov wrote:
sure, after you bought me with the authorship in this publication ... :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#117?email_source=notifications&email_token=ADWCQRY3TA27GMAQOHMFCZLRDUWVXA5CNFSM4D6IG2N2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMHZT7I#issuecomment-588225021>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADWCQR477PFUVPDCQO4SAK3RDUWVXANCNFSM4D6IG2NQ>.
|
@SergeyYakubov @samoylv Maybe time to push OpenMP patch to Oleg SRW repository is come? |
Agreed
…On Fri, 15 May 2020, 19:12 Alexey Buzmakov, ***@***.***> wrote:
@SergeyYakubov <https://github.com/SergeyYakubov> @samoylv
<https://github.com/samoylv> Maybe time to push OpenMP patch to Oleg SRW
repository is come?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#117 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAOFOKMTEOHSGVW4MK426MDRRVZXRANCNFSM4D6IG2NQ>
.
|
Just that you be aware how Oleg deals with OpenMP changes - he keeps it separated from his non-OpenMP code, i.e. if (non-openmp) {} else {open-mp code} instead of merging and having only one code. Three years ago (or when was it) the both versions produced exactly same results (up to the last digit), I check it thoroughly on the examples available. But since then he has made some changes in non-openmp version and did not touch the open-mp one. As a consequence, the results produced by two version a slightly different now. |
@CFGrote: what do you think? |
should merge openmp into mainline and carefully fix all conflicts. this
means someone has to set aside some time for it. @samoylv, is this
something that your student could take care of?
carsten
…On 5/16/20 9:28 AM, samoylv wrote:
@CFGrote <https://github.com/CFGrote>: what do you think?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#117 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADWCQRYHER67WKL3FWYL6ADRRY6CXANCNFSM4D6IG2NQ>.
|
Hi @SergeyYakubov There is a switch in Makefile: I think that is the source of the slight differences. Is it possible to use OpenMP code ith FFTW3? |
Hi! This is not only this switch, there are also switches in the code, e.g. I unfortunately have no time to deal with it. Anyway, would be good to agree with Oleg to merge the codes correctly once and forever. Otherwise with the next change in "non-openmp" part it can diverge again. Or one can of course leave it as it is and periodically check/update "openmp" part. |
when the *.ipynb run , the kernel is died.
when the *.py run, it crash with "segmentation fault".
and the srw example can run without any error.
The OS is Ubuntu 16.04,
python version is 3.6,
thank you!
The text was updated successfully, but these errors were encountered: