polars学习-06 Lazy （延迟、惰性）、eager（即时、实时） API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d396ab30",
   "metadata": {},
   "source": [
    "# 小编环境"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "65bfa10f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-20T09:27:02.424769Z",
     "start_time": "2024-06-20T09:27:00.830650Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python 版本： 3.11.9 \n",
      "polars 版本： 0.20.22\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "\n",
    "print('python 版本：',sys.version.split('|')[0])\n",
    "#python 版本： 3.11.9 \n",
    "\n",
    "import polars as pl\n",
    "\n",
    "print(\"polars 版本：\",pl.__version__)\n",
    "#polars 版本： 0.20.22"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3509a3e",
   "metadata": {},
   "source": [
    "# Lazy / Eager API 区别"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "081e0539",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-19T12:23:53.745221Z",
     "start_time": "2024-06-19T12:23:53.737958Z"
    }
   },
   "source": [
    "- Eager API：\n",
    "实时进行计算，每一步操作都会进行计算，类似pandas那样，每操作一步都会进行计算，得到这一步的结果，所见即所得，如果没有明确指定或者调用特定的方法之外，polars 基本都是使用该模式\n",
    "    \n",
    "\n",
    "- Lazy API：\n",
    "推迟进行计算，把所有的操作步骤先记下来，Query plan（查询计划），等到需要结果时，才统一进行计算，polars 会对这些计算步骤自动进行优化，提升性能\n",
    "    - pl.scan_csv 等 pl.scan_ 函数\n",
    "    - 调用DataFrame 的 .lazy 方法，转换为 Lazy 模式"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1cc81d9",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-19T12:23:53.745221Z",
     "start_time": "2024-06-19T12:23:53.737958Z"
    }
   },
   "source": [
    "# Eager API 数据处理案例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ebf9e7c2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-20T09:27:02.440588Z",
     "start_time": "2024-06-20T09:27:02.428782Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (3, 2)\n",
      "┌────────────┬─────────────┐\n",
      "│ Species    ┆ Sepal.Width │\n",
      "│ ---        ┆ ---         │\n",
      "│ str        ┆ f64         │\n",
      "╞════════════╪═════════════╡\n",
      "│ versicolor ┆ 2.804255    │\n",
      "│ setosa     ┆ 3.713636    │\n",
      "│ virginica  ┆ 2.983673    │\n",
      "└────────────┴─────────────┘\n"
     ]
    }
   ],
   "source": [
    "df = pl.read_csv(\"./data/iris.csv\")\n",
    "df_small = df.filter(pl.col(\"Sepal.Length\") > 5)\n",
    "df_agg = df_small.group_by(\"Species\").agg(pl.col(\"Sepal.Width\").mean())\n",
    "print(df_agg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9efa0897",
   "metadata": {},
   "source": [
    "# Lazy API 数据处理案例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8aff779e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-20T09:27:02.450353Z",
     "start_time": "2024-06-20T09:27:02.444604Z"
    }
   },
   "outputs": [],
   "source": [
    "q = (\n",
    "    pl.scan_csv(\"./data/iris.csv\")\n",
    "    .filter(pl.col(\"Sepal.Length\") > 5)\n",
    "    .group_by(\"Species\")\n",
    "    .agg(pl.col(\"Sepal.Width\").mean())\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "10c9fd5c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-20T09:27:02.462912Z",
     "start_time": "2024-06-20T09:27:02.452363Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "polars.lazyframe.frame.LazyFrame"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(q)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "21d0b999",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-20T09:27:02.760477Z",
     "start_time": "2024-06-20T09:27:02.465926Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h4>NAIVE QUERY PLAN</h4><p>run <b>LazyFrame.show_graph()</b> to see the optimized version</p><?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.50.0 (0)\r\n",
       " -->\r\n",
       "<!-- Title: polars_query Pages: 1 -->\r\n",
       "<svg width=\"246pt\" height=\"222pt\"\r\n",
       " viewBox=\"0.00 0.00 246.00 222.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 218)\">\r\n",
       "<title>polars_query</title>\r\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-218 242,-218 242,4 -4,4\"/>\r\n",
       "<!-- AGG [col(&quot;Sepal.Width&quot;).mean()]\r\n",
       "BY\r\n",
       "[col(&quot;Species&quot;)] [(0, 0)] [(0, 0)] -->\r\n",
       "<g id=\"node1\" class=\"node\">\r\n",
       "<title>AGG [col(&quot;Sepal.Width&quot;).mean()]\r\n",
       "BY\r\n",
       "[col(&quot;Species&quot;)] [(0, 0)] [(0, 0)]</title>\r\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"222,-214 16,-214 16,-161 222,-161 222,-214\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"119\" y=\"-198.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">AGG [col(&quot;Sepal.Width&quot;).mean()]</text>\r\n",
       "<text text-anchor=\"middle\" x=\"119\" y=\"-183.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">BY</text>\r\n",
       "<text text-anchor=\"middle\" x=\"119\" y=\"-168.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">[col(&quot;Species&quot;)] [(0, 0)]</text>\r\n",
       "</g>\r\n",
       "<!-- FILTER BY (col(&quot;Sepal.Length&quot;)) &gt; (... [(0, 1)] -->\r\n",
       "<g id=\"node2\" class=\"node\">\r\n",
       "<title>FILTER BY (col(&quot;Sepal.Length&quot;)) &gt; (... [(0, 1)]</title>\r\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"238,-125 0,-125 0,-89 238,-89 238,-125\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"119\" y=\"-103.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">FILTER BY (col(&quot;Sepal.Length&quot;)) &gt; (...</text>\r\n",
       "</g>\r\n",
       "<!-- AGG [col(&quot;Sepal.Width&quot;).mean()]\r\n",
       "BY\r\n",
       "[col(&quot;Species&quot;)] [(0, 0)] [(0, 0)]&#45;&#45;FILTER BY (col(&quot;Sepal.Length&quot;)) &gt; (... [(0, 1)] -->\r\n",
       "<g id=\"edge1\" class=\"edge\">\r\n",
       "<title>AGG [col(&quot;Sepal.Width&quot;).mean()]\r\n",
       "BY\r\n",
       "[col(&quot;Species&quot;)] [(0, 0)] [(0, 0)]&#45;&#45;FILTER BY (col(&quot;Sepal.Length&quot;)) &gt; (... [(0, 1)]</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M119,-160.8C119,-149.17 119,-135.74 119,-125.33\"/>\r\n",
       "</g>\r\n",
       "<!-- Csv SCAN ./data/iris.csv;\r\n",
       "π */?;\r\n",
       "σ &#45; [(0, 2)] -->\r\n",
       "<g id=\"node3\" class=\"node\">\r\n",
       "<title>Csv SCAN ./data/iris.csv;\r\n",
       "π */?;\r\n",
       "σ &#45; [(0, 2)]</title>\r\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"198,-53 40,-53 40,0 198,0 198,-53\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"119\" y=\"-37.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Csv SCAN ./data/iris.csv;</text>\r\n",
       "<text text-anchor=\"middle\" x=\"119\" y=\"-22.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">π */?;</text>\r\n",
       "<text text-anchor=\"middle\" x=\"119\" y=\"-7.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">σ &#45;</text>\r\n",
       "</g>\r\n",
       "<!-- FILTER BY (col(&quot;Sepal.Length&quot;)) &gt; (... [(0, 1)]&#45;&#45;Csv SCAN ./data/iris.csv;\r\n",
       "π */?;\r\n",
       "σ &#45; [(0, 2)] -->\r\n",
       "<g id=\"edge2\" class=\"edge\">\r\n",
       "<title>FILTER BY (col(&quot;Sepal.Length&quot;)) &gt; (... [(0, 1)]&#45;&#45;Csv SCAN ./data/iris.csv;\r\n",
       "π */?;\r\n",
       "σ &#45; [(0, 2)]</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M119,-88.97C119,-78.51 119,-64.9 119,-53.16\"/>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<LazyFrame at 0x25DE673DB90>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "50a4030c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-20T09:27:02.771224Z",
     "start_time": "2024-06-20T09:27:02.762489Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (3, 2)\n",
      "┌────────────┬─────────────┐\n",
      "│ Species    ┆ Sepal.Width │\n",
      "│ ---        ┆ ---         │\n",
      "│ str        ┆ f64         │\n",
      "╞════════════╪═════════════╡\n",
      "│ versicolor ┆ 2.804255    │\n",
      "│ virginica  ┆ 2.983673    │\n",
      "│ setosa     ┆ 3.713636    │\n",
      "└────────────┴─────────────┘\n"
     ]
    }
   ],
   "source": [
    "df = q.collect()\n",
    "\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e07badc",
   "metadata": {},
   "source": [
    "# 使用建议"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f45897e8",
   "metadata": {},
   "source": [
    "- 如果你是在进行探索性分析，想知道中间的每个步骤数据情况，那么可以使用 Eager 模式\n",
    "- 如果想得到最终的计算结果，那么可以使用 Lazy 模式，让polars对中间的计算进行优化，提升数据处理效率\n",
    "\n",
    "注：在大部分情况下，Eager API 背后其实调用的是 Lazy API，这样Eager 模式也可以利用上查询优化\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b69efe8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.11",
   "language": "python",
   "name": "python3.11"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}