Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-546: Migrate Sphinx documentation #553

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/workflows/rc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -412,6 +412,39 @@ jobs:
with:
name: release-docs
path: docs.tar.gz
docs:
name: Docs
needs:
- source
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
with:
cache: 'pip'
- name: Download source archive
uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
with:
name: release-source
- name: Extract source archive
run: |
tar -xf apache-arrow-java-*.tar.gz --strip-components=1
- name: Build
run: |
cd docs
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
make html
- name: Compress into single artifact to keep directory structure
run: tar -cvzf docs.tar.gz -C docs/build html
- name: Upload artifacts
uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4.6.0
with:
name: release-sphinx-docs
path: docs.tar.gz
verify:
name: Verify
needs:
Expand Down Expand Up @@ -452,6 +485,7 @@ jobs:
name: Upload
if: github.ref_type == 'tag'
needs:
- docs
- verify
runs-on: ubuntu-latest
permissions:
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
/dev/release/apache-rat-0.16.1.jar
/dev/release/filtered_rat.txt
/dev/release/rat.xml
/docs/build/
CMakeCache.txt
CMakeFiles/
Makefile
Expand Down
1 change: 1 addition & 0 deletions dev/release/rat_exclude_files.txt
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,4 @@

.gitmodules
dataset/src/test/resources/data/student.csv
docs/Makefile
20 changes: 20 additions & 0 deletions docs/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#

# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?= -W
SPHINXBUILD ?= sphinx-build
SOURCEDIR = source
BUILDDIR = build

# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

.PHONY: help Makefile

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
28 changes: 28 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
<!---
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Documentation

Build with Sphinx.

```
cd docs
pip install -r requirements.txt
make html
```
28 changes: 28 additions & 0 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

furo==2024.8.6
myst-parser==4.0.0
Sphinx==8.1.3
sphinx-autobuild==2024.10.3
sphinx-basic-ng==1.0.0b2
sphinxcontrib-applehelp==2.0.0
sphinxcontrib-devhelp==2.0.0
sphinxcontrib-htmlhelp==2.1.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==2.0.0
sphinxcontrib-serializinghtml==2.0.0
Empty file added docs/source/_static/.gitignore
Empty file.
92 changes: 92 additions & 0 deletions docs/source/algorithm.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements. See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership. The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License. You may obtain a copy of the License at

.. http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing,
.. software distributed under the License is distributed on an
.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
.. KIND, either express or implied. See the License for the
.. specific language governing permissions and limitations
.. under the License.

Java Algorithms
===============

Arrow's Java library provides algorithms for some commonly-used
functionalities. The algorithms are provided in the ``org.apache.arrow.algorithm``
package of the ``algorithm`` module.

Comparing Vector Elements
-------------------------

Comparing vector elements is the basic for many algorithms. Vector
elements can be compared in one of the two ways:

1. **Equality comparison**: there are two possible results for this type of comparisons: ``equal`` and ``unequal``.
Currently, this type of comparison is supported through the ``org.apache.arrow.vector.compare.VectorValueEqualizer``
interface.

2. **Ordering comparison**: there are three possible results for this type of comparisons: ``less than``, ``equal to``
and ``greater than``. This comparison is supported by the abstract class ``org.apache.arrow.algorithm.sort.VectorValueComparator``.

We provide default implementations to compare vector elements. However, users can also define ways
for customized comparisons.

Vector Element Search
---------------------

A search algorithm tries to find a particular value in a vector. When successful, a vector index is
returned; otherwise, a ``-1`` is returned. The following search algorithms are provided:

1. **Linear search**: this algorithm simply traverses the vector from the beginning, until a match is
found, or the end of the vector is reached. So it takes ``O(n)`` time, where ``n`` is the number of elements
in the vector. This algorithm is implemented in ``org.apache.arrow.algorithm.search.VectorSearcher#linearSearch``.

2. **Binary search**: this represents a more efficient search algorithm, as it runs in ``O(log(n))`` time.
However, it is only applicable to sorted vectors. To get a sorted vector,
one can use one of our sorting algorithms, which will be discussed in the next section. This algorithm
is implemented in ``org.apache.arrow.algorithm.search.VectorSearcher#binarySearch``.

3. **Parallel search**: when the vector is large, it takes a long time to traverse the elements to search
for a value. To make this process faster, one can split the vector into multiple partitions, and perform the
search for each partition in parallel. This is supported by ``org.apache.arrow.algorithm.search.ParallelSearcher``.

4. **Range search**: for many scenarios, there can be multiple matching values in the vector.
If the vector is sorted, the matching values reside in a contiguous region in the vector. The
range search algorithm tries to find the upper/lower bound of the region in ``O(log(n))`` time.
An implementation is provided in ``org.apache.arrow.algorithm.search.VectorRangeSearcher``.

Vector Sorting
--------------

Given a vector, a sorting algorithm turns it into a sorted one. The sorting criteria must
be specified by some ordering comparison operation. The sorting algorithms can be
classified into the following categories:

1. **In-place sorter**: an in-place sorter performs the sorting by manipulating the original
vector, without creating any new vector. So it just returns the original vector after the sorting operations.
Currently, we have ``org.apache.arrow.algorithm.sort.FixedWidthInPlaceVectorSorter`` for in-place
sorting in ``O(nlog(n))`` time. As the name suggests, it only supports fixed width vectors.

2. **Out-of-place sorter**: an out-of-place sorter does not mutate the original vector. Instead,
it copies vector elements to a new vector in sorted order, and returns the new vector.
We have ``org.apache.arrow.algorithm.sort.FixedWidthInPlaceVectorSorter.FixedWidthOutOfPlaceVectorSorter``
and ``org.apache.arrow.algorithm.sort.FixedWidthInPlaceVectorSorter.VariableWidthOutOfPlaceVectorSorter``
for fixed width and variable width vectors, respectively. Both algorithms run in ``O(nlog(n))`` time.

3. **Index sorter**: this sorter does not actually sort the vector. Instead, it returns an integer
vector, which correspond to indices of vector elements in sorted order. With the index vector, one can
easily construct a sorted vector. In addition, some other tasks can be easily achieved, like finding the ``k`` th
smallest value in the vector. Index sorting is supported by ``org.apache.arrow.algorithm.sort.IndexSorter``,
which runs in ``O(nlog(n))`` time. It is applicable to vectors of any type.

Other Algorithms
----------------

Other algorithms include vector deduplication, dictionary encoding, etc., in the ``algorithm`` module.
Loading
Loading