diff --git a/docs/tutorials/import_catalogs.ipynb b/docs/tutorials/import_catalogs.ipynb index 1485574c..4fe5b377 100644 --- a/docs/tutorials/import_catalogs.ipynb +++ b/docs/tutorials/import_catalogs.ipynb @@ -9,7 +9,10 @@ "source": [ "# Importing catalogs to HiPSCat format\n", "\n", - "This notebook presents two ways of importing catalogs to HiPSCat format. The first uses the `lsdb.from_dataframe()` method, which is helpful to load smaller catalogs from a single dataframe, while the second uses the `hipscat-import pipeline`." + "This notebook presents two modes of importing catalogs to HiPSCat format:\n", + "\n", + "1. `lsdb.from_dataframe()` method: helpful to load smaller catalogs from a single dataframe. data should have fewer than 1-2 million rows and the pandas dataframe should be less than 1-2G in-memory. if your data is larger, the format is complicated, you need more flexibility, or you notice any performance issues when importing with this mode, use the next mode.\n", + "2. `hipscat-import` package: for large datasets (1G - 100s of TB). this is a purpose-built map-reduce pipeline for creating hipscat catalogs from various datasets. in this notebook, we use a very basic dataset and basic import options. please see [the full package documentation](https://hipscat-import.readthedocs.io/) if you need to do anything more complicated." ] }, { diff --git a/docs/tutorials/margins.ipynb b/docs/tutorials/margins.ipynb index 77f5ef33..71cc3d9a 100644 --- a/docs/tutorials/margins.ipynb +++ b/docs/tutorials/margins.ipynb @@ -30,20 +30,6 @@ "To load a margin cache, we first load the margin catalog the same as other HiPSCat catalogs with a `lsdb.read_hipscat` call." ] }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "ExecuteTime": { - "end_time": "2024-03-26T20:32:10.418696Z", - "start_time": "2024-03-26T20:32:08.688730Z" - } - }, - "outputs": [], - "source": [ - "!pip install aiohttp --quiet # for reading from http" - ] - }, { "cell_type": "code", "execution_count": null, diff --git a/docs/tutorials/quickstart.ipynb b/docs/tutorials/quickstart.ipynb index b85e30d6..9caf3b22 100644 --- a/docs/tutorials/quickstart.ipynb +++ b/docs/tutorials/quickstart.ipynb @@ -76,24 +76,8 @@ "id": "3b889e0e", "metadata": {}, "source": [ - "Next, read the catalog we want to crossmatch with. Because we are downloading it from a web source we need to install an additional package (__aiohttp__). If your catalog happens to be present in local storage you can call `read_hipscat` directly." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "fcca5a0b", - "metadata": {}, - "outputs": [], - "source": [ - "!pip install aiohttp --quiet" - ] - }, - { - "cell_type": "markdown", - "id": "611cd2f6", - "metadata": {}, - "source": [ + "Next, read the catalog we want to crossmatch with. If your catalog happens to be present in local storage you can call `read_hipscat` on your local path.\n", + "\n", "In this tutorial we will read a small 1 degree cone region of Gaia DR3, one that should contain our objects. While LSDB typically reads into memory only the minimal amount of data it needs for our workflow, manually providing it with spatial information helps it identify which files to search for on disk." ] }, @@ -106,7 +90,7 @@ "source": [ "from lsdb.core.search import ConeSearch\n", "\n", - "gaia_path = \"https://epyc.astro.washington.edu/~lincc-frameworks/hipscat_surveys/gaia_dr3/gaia\"\n", + "gaia_path = \"https://data.lsdb.io/unstable/gaia_dr3/gaia\"\n", "\n", "gaia = lsdb.read_hipscat(gaia_path, search_filter=ConeSearch(ra=180, dec=10, radius_arcsec=0.6 * 3600))\n", "gaia" diff --git a/docs/tutorials/working_with_large_catalogs.ipynb b/docs/tutorials/working_with_large_catalogs.ipynb index 15aeb687..06c6ac56 100644 --- a/docs/tutorials/working_with_large_catalogs.ipynb +++ b/docs/tutorials/working_with_large_catalogs.ipynb @@ -29,26 +29,6 @@ "import lsdb" ] }, - { - "cell_type": "markdown", - "id": "20aa14ea", - "metadata": {}, - "source": [ - "## Installing dependencies\n", - "\n", - "To load catalogs from the web using the HTTP protocol we'll need to install __aiohttp__." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "d4b7cef3", - "metadata": {}, - "outputs": [], - "source": [ - "!pip install aiohttp --quiet" - ] - }, { "cell_type": "markdown", "id": "c2606cc9-648a-49ac-bbe6-17dfe1f9309f", @@ -102,7 +82,7 @@ "metadata": {}, "outputs": [], "source": [ - "surveys_path = \"https://epyc.astro.washington.edu/~lincc-frameworks/other_degree_surveys\"" + "surveys_path = \"https://data.lsdb.io/unstable/\"" ] }, { @@ -112,7 +92,7 @@ "metadata": {}, "outputs": [], "source": [ - "ztf_object_path = f\"{surveys_path}/ztf/ztf_object\"\n", + "ztf_object_path = f\"{surveys_path}/ztf/ztf_dr14\"\n", "ztf_object = lsdb.read_hipscat(ztf_object_path, columns=[\"ps1_objid\", \"ra\", \"dec\"])\n", "ztf_object" ] @@ -403,6 +383,11 @@ } ], "metadata": { + "kernelspec": { + "display_name": "hipscatenv", + "language": "python", + "name": "python3" + }, "language_info": { "codemirror_mode": { "name": "ipython", @@ -413,7 +398,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.13" + "version": "3.10.14" } }, "nbformat": 4,