texinfo documentation for sage

[mantepse]

texinfo is the native documentation format to emacs. sphinx can output texinfo. we should use that.

CC: [email protected] @hivert

Component: documentation

Keywords: Sphinx

Author: Martin Rubey

Branch/Commit: u/mantepse/texinfo_documentation_for_sage @ b3dac9d

Issue created by migration from https://trac.sagemath.org/ticket/21734

[mantepse]

Branch: u/mantepse/texinfo_documentation_for_sage

[mantepse]

comment:2

see also https://groups.google.com/forum/#!topic/sage-devel/MjhzbpqnLXs

applying this patch and doing ./sage -docbuild reference texinfo leaves a .texi file and a Makefile in each directory below SAGEROOT/local/share/doc/sage/texinfo/en/reference. Issuing make in each directory then creates a beautiful .info file, which can be visited in emacs using C-u C-h i filename.

Currently I do not know how to make sage make the .info files itself, and I do not know how sage-shell-mode could display them.

---

New commits:

556e8d3

create texi files

[mantepse]

Commit: 556e8d3

[mantepse]

Author: Martin Rubey

[mantepse]

comment:4

I am having some problems with the texinfo-hyperlinks. For example, in

 sage: ASM = AlternatingSignMatrices(3)

the (3) is interpreted as a link to a footnote. The corresponding bit of the texinfo source looks OK, though:

@geindex from_contre_tableau() (sage.combinat.alternating_sign_matrix.AlternatingSignMatrices method)
@anchor{sage/combinat/alternating_sign_matrix sage combinat alternating_sign_matrix AlternatingSignMatrices from_contre_tableau}@anchor{227}
@deffn {Method} from_contre_tableau (comps)

Return an alternating sign matrix from a contre-tableau.

EXAMPLES:

@example
sage: ASM = AlternatingSignMatrices(3)
sage: ASM.from_contre_tableau([[1, 2, 3], [1, 2], [1]])
[0 0 1]
[0 1 0]
[1 0 0]
sage: ASM.from_contre_tableau([[1, 2, 3], [2, 3], [3]])
[1 0 0]
[0 1 0]
[0 0 1]
@end example
@end deffn

[mantepse]

comment:5

Replying to @mantepse:

I am having some problems with the texinfo-hyperlinks. For example, in

 sage: ASM = AlternatingSignMatrices(3)

the (3) is interpreted as a link to a footnote. The corresponding bit of the texinfo source looks OK, though:

in the standalone info program, this problem does not appear. There are some hints in the Sphinx FAQ http://www.sphinx-doc.org/en/stable/faq.html#texinfo-info but I do not see why they would apply.

[sagetrac-stakemori]

comment:6

Replying to @mantepse:

Thank you for the patch. I attended a conference this week and will try the patch when I go back home.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

66f2f43

also build info files

[sagetrac-git]

Changed commit from 556e8d3 to 66f2f43

[sagetrac-git]

Changed commit from 66f2f43 to 7012f33

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

7012f33

give the info file the same name as the directory

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

4294647

simplify (images are still not copied)

[sagetrac-git]

Changed commit from 7012f33 to 4294647

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

db003fa

more sensible values in texinfo_documents

[sagetrac-git]

Changed commit from 4294647 to db003fa

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

6774f4f	remove line breaks in rubrics titles
491ad5e	repair removal of newlines

[sagetrac-git]

Changed commit from db003fa to 491ad5e

[mantepse]

comment:12

Florent, if you could help with the problem reported at https://groups.google.com/forum/#!topic/sphinx-users/-8cHf1ZD5S4 that would be great. This seems very much related to #9128!

[mantepse]

Changed keywords from none to Sphinx

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

c73c232

not sure, but I think this needs to set texinfo_documents, too

[sagetrac-git]

Changed commit from 491ad5e to c73c232

[mantepse]

comment:14

Dear Sho,

unfortunately, I still have no clue how to enable the crossreferences between the individual manuals. I asked on the sphinx user group https://groups.google.com/forum/#!topic/sphinx-users/-8cHf1ZD5S4 and also on the sphinx github page sphinx-doc/sphinx#3089, but have not got any response so far.

meanwhile, I discovered SAGEROOT/src/doc/common/conf.py, which contains the call app.connect('missing-reference', find_sage_dangling_links).

I suspect that this (I mean find_sage_dangling_links) is where I should put the proper handling for texinfo crossreferences, but I'm not sure yet...

[sagetrac-stakemori]

comment:15

Replying to @mantepse:

It is already convenient to view Sage document via info. Thank you very much. Concerning with sphinx, I cannot help you, sorry.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

b3dac9d

Merge branch 'develop' of git://trac.sagemath.org/sage into t/21734/texinfo_documentation_for_sage

[sagetrac-git]

Changed commit from c73c232 to b3dac9d

[jhpalmieri]

comment:17

The reference manual doesn't build for me (html, not texinfo) without this change:

diff --git a/src/doc/en/reference/conf.py b/src/doc/en/reference/conf.py
index 5c7bbe95ec..22961d90ae 100644
--- a/src/doc/en/reference/conf.py
+++ b/src/doc/en/reference/conf.py
@@ -67,7 +67,7 @@ latex_elements['preamble'] += r'''
 # @end direntry
 
 texinfo_documents = [
-('index', name, title, u'The Sage Development Team', name, title, 'SageMath')
+('index', name, project, u'The Sage Development Team', name, project, 'SageMath')
 ]
 
 #Ignore all .rst in the _sage subdirectory

or something similar: title is not defined in reference/conf.py.

[jhpalmieri]

comment:18

You could also consider adding a Makefile target:

diff --git a/build/make/deps b/build/make/deps
index 371a56a767..65ad3fdc07 100644
--- a/build/make/deps
+++ b/build/make/deps
@@ -247,6 +247,8 @@ doc-src-clean:
 doc-output-clean:
        rm -rf "$(SAGE_SHARE)/doc/sage"
 
+doc-texinfo: $(DOC_DEPENDENCIES)
+       $(AM_V_at)cd ../.. && sage-logger -p './sage --docbuild all texinfo $(SAGE_DOCBUILD_OPTS)' logs/doctexinfo.log
 
 ###############################################################################
 # Cleaning up

(After doing this, make doc-texinfo fails for me when building the developer's guide, by the way.) Then someone who is better at make than I am could possibly add a target to run make in all of the subdirectories of local/share/doc/sage/texinfo. It could be a bit of a mess, but we could instead set $infodir to point to local/share/info and then run make install-info instead of make. The problem is that texinfo/de/tutorial/sagetutorial.texi and texinfo/en/tutorial/sagetutorial.texi will produce .info files of the same name. Does info handle subdirectories well? Should we have local/share/info/de/ or local/share/info/sage/de/, etc.? The Sage reference manual will also consist of lots of separate .info files; how should those be handled?

[mantepse]

comment:19

Thank you for looking into this!

I'm afraid, however, that there is a shostopper: I couldn't get the links between the individual texinfo files to work. I filed sphinx-doc/sphinx#3089 but it was closed. The advice was: "Please integrate your project into single one."

It may well be that I misunderstood the whole thing - but as far as I could tell at the time, the sage documentation consists of many subprojects, one for each folder below src/sage. If that's not the case, that would be great and I'd be happy to learn more about it!

[aikrahguzar]

@mantepse a texinfo manual would be great and I would like to help but I have little familiarity with texinfo, sphinx and sage's docbuilder. I would like to try a working manual but my chances of succeeding are dim.

I think the problems are:

1. Links between different manuals produced by sage don't work.
2. Links to sage classes with a manual don't work. I think this might be an info documentation? E.g. even in Emacs I don't now how to refer to the location of a function's description rather than the node containing that description.
3. What to do about math in Sage manuals? It seems to be included verbatim in the manual. Looking at the texinfo documentation, it seems like math is handled when producing a pdf but not when using Emacs or standalone info viewer.

I try your work and extend it but it will probably take a lot of time. Thanks for the foundation.

[aikrahguzar]

@mantepse maybe the suggestion to integrate the project into a single one is worth considering. Not as a change to the sage project but for the more restricted purpose of building an info manual.

With this diff

@@ -75,6 +75,8 @@ multidocs_subdoc_list = sorted([x for x in os.listdir(ref_src)
 
 # List of directories, relative to source directory, that shouldn't be
 # searched for source files.
-exclude_patterns += multidocs_subdoc_list + [
-    'sage', 'options'
-    ]
+# exclude_patterns += multidocs_subdoc_list + [
+#     'sage', 'options'
+#     ]
+
+exclude_patterns = ['_sage']

to src/doc/en/reference/conf.py, I am able to build a usable manual for the reference by running

./sage -sh -c "sphinx-build -M info ./src/doc/en/reference ../info"

from the directory with sage repo. This command actually fails after producing the texi due to errors like

@menu reference to nonexistent node `b7'

and I have to run from the directory containing the texi

makeinfo --no-split --force -o 'referencemanual.info' 'referencemanual.texi'

to produce the info file.

A lot of cross references don't work but I am not sure why, since they are defined as @anchor in the texi file and in the info file I can see them in lines like,

Ref: anchor_id^?a_large_number

Do you have any idea? Might be due to the fact that there are multiple anchors for the same location in the texi? I messes up the navigation pretty badly but for me at least the manual is usable with Info-goto-node.

[aikrahguzar]

The cross references actually work. Here is what I did: I used open the info file then M-x Info-on-current-buffer RET. At this stage cross references don't work, however they start working after reverting the buffer. I don't know why.

Another thing, in addition to conf.py above, src/doc/en/reference/index.rst also needs to be modified,

@@ -183,3 +183,9 @@ Indices and Tables
 
 .. math::
    :nowrap:
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   */index

[mantepse]

I find it very cool that you are obviously making progress! Do you think you could prepare a pull request at some point so that it is easy to follow your work?

I use emacs and sage-shell-mode (and nothing else), so I'd be thrilled to see this working!

[aikrahguzar]

I find it very cool that you are obviously making progress! Do you think you could prepare a pull request at some point so that it is easy to follow your work?

I don't if there is a path for this hack to be upstreamed and I still don't understand how the sage build process works. Maybe conditionally excluding directories depending on the format will be acceptable for conf.py but I think that toctree in the index.rst is going to be problematic. However I don't understand sphinx enough to say anything with confidence.

I use emacs and sage-shell-mode (and nothing else), so I'd be thrilled to see this working!

Me too, though I feel I can get by with the hack I have in place now and build the documentation myself. Anyhow here is the gzipped manual I was able to obtain. You can download it and M-x Info-on-current-buffer to see how it looks. It has (I think) everything in the English section. Some things are not ideal (like too many Indices and Tables sections in (sage.info) Indices and Tables) but for me it is an improvement over the html one.

sage.info.gz

[mantepse]

I find it very cool that you are obviously making progress! Do you think you could prepare a pull request at some point so that it is easy to follow your work?

I don't if there is a path for this hack to be upstreamed and I still don't understand how the sage build process works. Maybe conditionally excluding directories depending on the format will be acceptable for conf.py but I think that toctree in the index.rst is going to be problematic. However I don't understand sphinx enough to say anything with confidence.

Even if it's not complete, it would be cool if you could create a pull request that contains the necessary modifications (if there are any), and a step by step list how to recreate sage.info.gz. It is quite possible that other people can continue from there.

[aikrahguzar]

@mantepse I created a branch https://github.com/aikrahguzar/sage/tree/info-manual, I don't want to create a draft pr since it is based off a branch which is not merged yet but which I need to build on my computer.

To build the info manual,

# Generate src/doc/en/reference/repl/options.txt
cd src/doc/
make doc-src
cd -
# Generate inventory
./sage --docbuild reference inventory
# Generate texi (replace target by the directory where build artifacts should go.)
./sage -sh -c "sphinx-build -b texinfo ./src/doc/en target"
# Generate info
makeinfo --no-split --force -o target/sage.info target/sage.texi

The --force argument can be removed by adding texinfo_domain_indices = False to the src/doc/en/conf.py but that removes the python module index from the manual which I think is worse.

I am mostly satisfied with the state here. This setup allows for generating an info manual without messing with the regular build. However if someone wants to continue, here are some thing to figure out:

1. Anyway to remove the --force? I think the problem is with the generation of texi which includes lines like,

   * sage.algebras.affine_nil_temperley_lieb: 0. 
   

   and the problem that 0. on the right. Other nodes don't any such numbers.
2. Is there a way to get rid of nodes like (sage) Indices and Tables<2>? They are empty or refer to html files and aren't needed.
3. The top node can be better organized if done manually.

One problem I would like to solve is the math rendering. But that probably requires changes in texinfo so unlikely in the short term.

[mantepse]

Hi @aikrahguzar, thanks for sharing! I created a local branch that only contains your two commits with the modifications for texinfo. If you don't mind, I would turn it into a pull request.

Did you have a look at my (ancient) modifications, i.e., u/mantepse/texinfo_documentation_for_sage? Have they become unnecessary? I do not remember what I did and what it was good for, unfortunately.

Browsing through the info file your procedure produces, it seems that the structure of the documentation could be improved quite a bit. There are also a few issues with the typesetting - the current doc in sage-emacs-mode does a good job there (concerning colors and stripping unwanted information).

Also, I really would like to be able to use

sage: LazySymmetricFunctions?

to jump to the correct node in the info manual.

[aikrahguzar]

Hi @aikrahguzar, thanks for sharing! I created a local branch that only contains your two commits with the modifications for texinfo. If you don't mind, I would turn it into a pull request.

Please go ahead.

Did you have a look at my (ancient) modifications, i.e., u/mantepse/texinfo_documentation_for_sage? Have they become unnecessary? I do not remember what I did and what it was good for, unfortunately.

My guess is they are not needed. I don't understand them well enough but I think they were trying to use the sage docbuild system for building info manuals. This bypasses that and uses sphinx-build directly. There were others which addressed a couple of issues mentioned in sphinx-doc/sphinx#3089 but those have been addressed in sphinx.

Browsing through the info file your procedure produces, it seems that the structure of the documentation could be improved quite a bit.

Agreed, I think some of the issues could be addressed by a handwritten src/doc/en/index.rst. That might be good enough to get rid of the Indices and Tables entries which I think are the biggest issues.

There are also a few issues with the typesetting - the current doc in sage-emacs-mode does a good job there (concerning colors and stripping unwanted information).

The only typesetting problem I see is how the latex math code is copied verbatim. I sent en email to the info help mailing list about it.

Coloring Class, Method, sage prompts etc should be possible with a custom jit-lock-function. I might try my hand at that and will let you know if I do.

But even the Emacs manual doesn't do much coloring and I find the look alright there.

Also, I really would like to be able to use

sage: LazySymmetricFunctions?

to jump to the correct node in the info manual.

I think this is very doable. Use a custom comint-input-sender to capture the input ending with ?. Compute the fully qualified name of the identifier and then Info-goto-node("(sage)NODE-NAME") where NODE-NAME is computed from the fully qualified name.

[aikrahguzar]

There are also a few issues with the typesetting - the current doc in sage-emacs-mode does a good job there (concerning colors and stripping unwanted information).

I took a stab at the fontification here https://codeberg.org/rahguzar/sage-mode/src/commit/361e11ded046d5ffbb1e4eb89b7590c44dcf5e49/sage.el#L528-L618

It can be trivially adapted to sage-shell-mode by replacing one call to sage-mode with the one provided by sage-shell-mode and changing the variable names etc to fit the convention there.

I also asked on the mailing list for gnu-info help and we might get the math parts of the manual as typeset images at some point https://lists.gnu.org/archive/html/help-texinfo/2024-08/msg00003.html

[9IPOCJIABbI4]

@aikrahguzar thanks a lot for your patches! Having documentation for sage right in the Emacs is extremely useful!

[aikrahguzar]

@mantepse texinfo people have now implemented an option to generate images for math https://lists.gnu.org/archive/html/bug-texinfo/2024-12/msg00061.html

It is a work in progress and it will take me a few days to try it out but if you can please try it out. It should help with making the manual more readable.

[mantepse]

@aikrahguzar, I am very interested in trying (I use emacs exclusively, almost as operating system) , but I did not look at this for a very long time. Am I right that I should first build the documentation with #38457?

[mantepse]

./sage --docbuild reference inventory

fails for me, but seems to be unnecessary after make doc.

So, I can reproduce the info file, but it doesn't quite seem to work: I get No such node or anchor: together with a number when following a link.

[aikrahguzar]

./sage --docbuild reference inventory

fails for me, but seems to be unnecessary after make doc.

It is unnecessary and I got the same errors when building sage 10.5 with the patches. ./sage --docbuild reference inventory used to work when I was messing about with it.

We only need the inventory so make (what I did) or make doc (like you) is enough.

So, I can reproduce the info file, but it doesn't quite seem to work: I get No such node or anchor: together with a number when following a link.

This is why --force is necessary to actually generate the manual. In my experience the missing anchors don't cause a problem.

If you want I can attach the compressed manual I obtained here.

[mantepse]

But I used '--force'

[aikrahguzar]

But I used '--force'

If you are using Info-on-current-buffer you are probably running into https://lists.gnu.org/archive/html/bug-gnu-emacs/2024-07/msg01340.html

It is fixed on Emacs 30 but for Emacs 29 (or lower) you can just revert the buffer and the links will start working.

[mantepse]

Wow, excellent!

It would be great to integrate this into sage! Perhaps so that make doc-info creates the texinfo installation and also tells me what to do so that emacs (and perhaps info) can find it.

Apart from that, every second node in the Listing of Reference Manual is Indices and Tables: Indices and Tables, as follows. They look useless - can / should they be removed?

2.153 Indices and Tables
========================

   * Index(1)

   * Module Index(2)

   * Search Page(3)



   ---------- Footnotes ----------

   (1) ../genindex.html

   (2) ../py-modindex.html

   (3) ../search.html

Finally, what would be absolutely wonderful for me, is integration with sage-shell-mode. I tried the sage-info-mode, it looks mostly OK with some minor glitches. Most importantly, this would include jumping to the correct node from the sage buffer when appending a single question mark to an object (like sage: Permutations?), and enabling C-c C-d to send doctests from info to the current sage buffer.

[aikrahguzar]

Wow, excellent!

It would be great to integrate this into sage! Perhaps so that make doc-info creates the texinfo installation and also tells me what to do so that emacs (and perhaps info) can find it.

make doc-info would be nice but I am probably not the person to do it. I have no knowledge of make apart from running make when a readme tells me to.

Making emacs finding the info file is not so hard, you can either put the info file into one of the directories in Info-directory-list or else add the directory containing the file to Info-directory-list e.g. I have this in my init.el

(push "~/.local/state/emacs/info/" Info-directory-list)

Apart from that, every second node in the Listing of Reference Manual is Indices and Tables: Indices and Tables, as follows. They look useless - can / should they be removed?

They are useless and as far as I can remember it is possible to remove them by adding manual index.rst files (e.g. the one in src/doc/en/index.rst). However, I don't understand the syntax and sphinx well enough so I gave up as soon as I got a manual I can use. These entries can also be hidden by hacks in Emacs (easier than messing with sphinx for me).

Finally, what would be absolutely wonderful for me, is integration with sage-shell-mode. I tried the sage-info-mode, it looks mostly OK with some minor glitches. Most importantly, this would include jumping to the correct node from the sage buffer when appending a single question mark to an object (like sage: Permutations?), and enabling C-c C-d to send doctests from info to the current sage buffer.

The closest thing I have to this is this-function. If you invoke it when the point is on a symbol, it will try to find the corresponding info node. It can be adapted to do what you want using a custom comint-input-sender.

I have no idea about doctests.

BTW: have you tried the images for math in info yet?