Merge pull request #730 from Josverl/stdlib_rename_cpython_stubs

Improve verity of micropython-stdlib-stubs

.github/workflows/test_stub_quality.yml

@@ -1,10 +1,13 @@
 name: test_stub_quality
-on: [push, pull_request, workflow_dispatch]
+on: [ pull_request, workflow_dispatch] # Not on push to avoid wasting 15 mins CPU 
 
 env:
   # Setting an environment variable with the value of a configuration variable
   SNIPPET_SCORE: ${{ vars.SNIPPET_SCORE }}
   GH_TOKEN_VARS: ${{ secrets.GH_TOKEN_VARS }}
+  # fix:  DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs
+  JUPYTER_PLATFORM_DIRS: "1"
+
 
 jobs:
   test_snippets:
@@ -40,14 +43,15 @@ jobs:
         run: stubber clone
 
 
-      - name: update the stubs and test the snippets (not pushed) 
+      - name: update the stubs (not pushed) 
+        continue-on-error: true
+        run: |
+          pwsh -file ./update-stubs.ps1
+
+      - name: Test the snippets 
         continue-on-error: true
         run: |
-          pwsh -file ./update-stubs.ps1 
           pytest -m 'snippets' --cache-clear --junitxml=./results/results.xml
-        env:
-          JUPYTER_PLATFORM_DIRS: "1"
-          # fix:  DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs
 
       - name: Testspace push test content
         run: |

README.md

@@ -4,7 +4,8 @@
 [![Star on GitHub](https://img.shields.io/github/stars/josverl/micropython-stubs.svg?style=social)](https://github.com/josverl/micropython-stubs/stargazers)
 [![All Contributors](https://img.shields.io/badge/all_contributors-19-green.svg?style=flat-square)](#Contributions)
 [![Black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/python/black "Black badge")
-
+[![Checked with pyright](https://microsoft.github.io/pyright/img/pyright_badge.svg)](https://microsoft.github.io/pyright/) 
+[![Checked with pyright](https://github.com/Josverl/micropython-stubs/actions/workflows/test_stub_quality.yml/badge.svg?branch=main)](https://microsoft.github.io/pyright/) 
 
 <img src="docs/img/colorstubs.jpg"
      alt="pencil stubs"

publish/micropython-stdlib-stubs/creation.md

@@ -0,0 +1,58 @@
+This is a short description of the steps taken to create the stubs for the micropython-stdlib-stubs distribution.
+There may be some steps and details missing as I have not documented the process as I went along.
+
+1. copy typeshed/stdlib (from pyright fallback)
+    ( I used a different version than listed here, but the process should be the same )
+    - ...\.vscode\extensions\ms-python.vscode-pylance-2023.12.1\dist\typeshed-fallback\stdlib
+    - to ...\micropython-stdlib-stubs\stdlib
+
+2. Keep only the following **sub folders** / packages 
+    - [ ] `_typeshed`
+    - [ ] `asyncio`
+    - [ ] `collections`
+    - [ ] `os`
+
+    - remove all other folders (28-ish folders)
+
+3. Rename the stubs that are used internally, but are not part of micropython
+    with two leading underscores ( __ ) 
+    NOTE: rename them in vscode to that the files that reference them get automagically refactored 
+    - [ ] `codecs.pyi`         --> `__codecs.pyi`
+    - [ ] `contextlib.pyi`     --> `__contextlib.pyi`
+    - [ ] `contextvars.pyi`    --> `__contextvars.pyi`
+    - [ ] `dataclasses.pyi`    --> `__dataclasses.pyi`
+    - [ ] `decimal.pyi`        --> `__decimal.pyi`
+    - [ ] `enum.pyi`           --> `__enum.pyi`
+    - [ ] `fractions.pyi`      --> `__fractions.pyi`
+    - [ ] `functools.pyi`      --> `__functools.pyi`
+    - [ ] `numbers.pyi`        --> `__numbers.pyi`
+    - [ ] `queue.pyi`          --> `__queue.pyi`
+    - [ ] `selectors.pyi`      --> `__selectors.pyi`
+    - [ ] `sre_compile.pyi`    --> `__sre_compile.pyi`
+    - [ ] `sre_constants.pyi`  --> `__sre_constants.pyi`
+    - [ ] `sre_parse.pyi`      --> `__sre_parse.pyi`
+
+4. Remove all stubs and stub-only packages that are not in micropython
+    Keep only the following: 
+    - [ ] All `__<package>.pyi` from step 3 
+    - [ ] `__future__.pyi`
+    - [ ] `_ast.pyi`
+    - [ ] `_codecs.pyi`
+    - [ ] `_collections_abc.pyi`
+    - [ ] `_decimal.pyi`
+    - [ ] `abc.pyi`
+    - [ ] `builtins.pyi`
+    - [ ] `io.pyi`
+    - [ ] `re.pyi`
+    - [ ] `socket.pyi`
+    - [ ] `sys.pyi`
+    - [ ] `types.pyi`
+    - [ ] `typing_extensions.pyi`
+    - [ ] `typing.pyi`
+
+
+
+
+
+
+

publish/micropython-stdlib-stubs/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "micropython-stdlib-stubs"
-version = "1.0.0"
+version = "1.0.1a1"
 description = "Micropython stdlib is a reduced and augmented copy of typeshed's stdlib for use by MicroPython stub packages"
 authors = ["josverl <[email protected]>"]
 license = "MIT"

publish/micropython-stdlib-stubs/stdlib/sys.pyi

@@ -349,21 +349,3 @@ if sys.version_info < (3, 8):
 def set_int_max_str_digits(maxdigits: int) -> None: ...
 def get_int_max_str_digits() -> int: ...
 
-# MicroPython specific functions
-# Copyright (c) 2023 Jos Verlinde
-
-from typing import Optional
-from _typeshed import Incomplete
-def atexit(func:Optional[Callable[[],Any]]) -> Optional[Callable[[],Any]]: 
-    """\
-    Register func to be called upon termination. func must be a callable that takes no arguments, 
-    or None to disable the call. The atexit function will return the previous value set by this function, 
-    which is initially None.
-
-    Ports: Unix, Windows
-    """
-    ...
-
-def print_exception(exc, file=sys.stdout, /):
-    """Print exception with a traceback to a file-like object file (or sys.stdout by default)."""
-    ...

requirements-test.txt

@@ -2,7 +2,7 @@
 micropython-stubber
 
 # for quality reporting
-pyright
+pyright==1.1.339
 pytest
 fasteners 
 python-dotenv

tests/quality_tests/.vscode/launch.json

@@ -4,14 +4,28 @@
     // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
     "version": "0.2.0",
     "configurations": [
+        // {
+        //     "name": "debug build-stubs",
+        //     "type": "PowerShell",
+        //     "request": "launch",
+        //     "script": "${workspaceFolder}/build-stubs.ps1",
+        //     "args": [
+        //         "-do_build"
+        //     ]
+        // },
         {
-            "name": "debug build-stubs",
-            "type": "PowerShell",
+            "name": "Debug pytest tests",
+            "purpose": [
+                "debug-test"
+            ],
+            "type": "python",
             "request": "launch",
-            "script": "${workspaceFolder}/build-stubs.ps1",
-            "args": [
-                "-do_build"
-            ]
+            "console": "integratedTerminal",
+            "justMyCode": false,
+            "stopOnEntry": false,
+            // false : Avoid debugpy trying to debug micropython
+            // but may cause issue with pytest-cov coverage reporting 
+            "subProcess": false,
         }
     ]
 }

tests/quality_tests/.vscode/settings.json

@@ -1,9 +1,6 @@
 {
     "python.analysis.diagnosticMode": "workspace",
     "python.analysis.typeCheckingMode": "basic",
-    "python.testing.pytestArgs": [
-        "."
-    ],
     "python.testing.unittestEnabled": false,
     "python.testing.pytestEnabled": true
 }

tests/quality_tests/conftest.py

@@ -145,13 +145,21 @@ def install_stubs(
     if stub_source == "pypi":
         # Add version
         cmd = f"pip install micropython-{portboard}-stubs=={version.lower().lstrip('v')}.* --target {tsc_path} --no-user"
+    elif stub_source == "pypi-pre":
+        # Add version and --pre
+        cmd = f"pip install micropython-{portboard}-stubs=={version.lower().lstrip('v')}.* --pre --target {tsc_path} --no-user"
     else:
-        foldername = f"micropython-{flatversion}-{portboard}-stubs"
+        # local source and --pre to pull in a pre-release version of stdlib
+        if version == "-": 
+            # stdlib has no version in publish/path
+            foldername = f"micropython-{portboard}-stubs"
+        else:
+            foldername = f"micropython-{flatversion}-{portboard}-stubs"
         # stubsource = pytestconfig.inipath.parent / f"repos/micropython-stubs/publish/{foldername}"
         stubsource = pytestconfig.inipath.parent / f"publish/{foldername}"
         if not stubsource.exists():
             pytest.skip(f"Could not find stubs for {portboard} {version} at {stubsource}")
-        cmd = f"pip install {stubsource} --target {tsc_path} --no-user"
+        cmd = f"pip install {stubsource} --pre --target {tsc_path} --no-user"
 
     try:
         subprocess.run(cmd, shell=True, check=True, capture_output=True, text=True)
@@ -212,6 +220,7 @@ def copy_type_stubs(
             if typings_path.exists():
                 shutil.rmtree(typings_path, ignore_errors=True)
             shutil.copytree(type_stub_cache_path, typings_path)
+            print(f" - copied to {typings_path}")
 
 
 def pytest_terminal_summary(terminalreporter, exitstatus, config: pytest.Config):
@@ -220,7 +229,7 @@ def pytest_terminal_summary(terminalreporter, exitstatus, config: pytest.Config)
         stats[status] = snipcount(terminalreporter, status)
     # simple straigth forward scoring
     stats["snippet_score"] = int(stats["passed"] - stats["failed"])
-    if stats["snippet_score"] >= 0:
+    if stats["snippet_score"] > 0:
         # Write stats to file
         (config.rootpath / "results").mkdir(exist_ok=True)
         with open(config.rootpath / "results" / "snippet_score.json", "w") as f:

tests/quality_tests/feat_stdlib_only/check_bytes.py

@@ -0,0 +1,5 @@
+PING_MSG = b"ping"
+channel = 5
+
+check = PING_MSG + b"x"
+check = PING_MSG + bytes([channel])  # type: ignore #TODO Operator "+" not supported for types "Literal[b"ping"]" and "bytes"

tests/quality_tests/feat_stdlib_only/check_collections/check_namedtuple.py

@@ -0,0 +1,12 @@
+from collections import namedtuple
+
+MyTuple = namedtuple("MyTuple", ("id", "name"))
+
+t1 = MyTuple(1, "foo")  # type: ignore # TODO: collections.namedtuple is not callable
+t2 = MyTuple(2, "bar")  # type: ignore
+print(t1.name)
+assert t2.name == t2[1]
+
+print(type(MyTuple))
+print(type(t1))
+print(type(t2))

tests/quality_tests/feat_stdlib_only/check_collections/check_ordereddict.py

@@ -0,0 +1,12 @@
+from collections import OrderedDict
+
+# To make benefit of ordered keys, OrderedDict should be initialized
+# from sequence of (key, value) pairs.
+d = OrderedDict([("z", 1), ("a", 2)])
+# More items can be added as usual
+
+#TODO add to ordered dict : https://github.com/Josverl/micropython-stubber/issues/333
+d["w"] = 5  # type: ignore 
+d["b"] = 3  # type: ignore
+for k, v in d.items():
+    print(k, v)

tests/quality_tests/feat_stdlib_only/check_hasable.py

@@ -0,0 +1,10 @@
+"""Validate that a type is hashable.
+ref: https://github.com/Josverl/micropython-stubs/issues/723
+"""
+
+i = 0
+d = {i: "a"}
+
+type_text = "int"
+if type_text in {"int", "float", "str", "bool", "tuple", "list", "dict"}:
+    order = 1

tests/quality_tests/feat_stdlib_only/check_io.py

@@ -0,0 +1,16 @@
+import io
+from typing import IO, Any, Optional
+
+alloc_size = 512
+
+
+buffer_1 = io.StringIO("hello world") 
+buffer_2 = io.BytesIO(b"some initial binary data: \x00\x01") # type: ignore # TODO 
+
+stream = open("file")
+
+buffer_3 = io.BufferedWriter(stream, 8)  # type: ignore # TODO stdlib.io "TextIOWrapper" is incompatible with "RawIOBase"
+print(buffer_3.write(bytearray(16)))  # type: ignore # TODO  stdlib.io "bytearray" is incompatible with protocol "ReadableBuffer"
+
+
+stream.close()

tests/quality_tests/feat_stdlib_only/check_os/check_files.py

@@ -0,0 +1,60 @@
+# runs on MicroPython (and Python)
+# get file and folder information and return this as JSON
+# params : folder , traverse subdirectory , output format, gethash
+# intended to allow simple processing of files
+# [email protected]
+import json
+import logging
+import os
+from typing import Any, Dict, List, Optional, Tuple, Union
+
+import ubinascii  # type: ignore
+import uhashlib  # type: ignore
+
+logging.basicConfig(level=logging.DEBUG)
+log = logging.getLogger(__name__)
+
+
+def listdir(path=".", sub=False, JSON=True, gethash=False):
+    # Lists the file information of a folder
+    li :List[dict]= []  # type: List[dict]
+    if path == ".":  # Get current folder name
+        path = os.getcwd()
+    files = os.listdir(path)
+    for file in files:
+        # get size of each file
+        info = {"Path": path, "Name": file, "Size": 0}
+        if path[-1] == "/":
+            full = "%s%s" % (path, file)
+        else:
+            full = "%s/%s" % (path, file)
+        log.debug("os.stat({})".format(full))
+        subdir = []
+        try:
+            stat = os.stat(full) # type: ignore
+            if stat[0] & 0x4000:  # stat.S_IFDIR
+                info["Type"] = "dir"
+                # recurse folder(s)
+                if sub == True:
+                    log.debug("Folder :{}".format(full))
+                    subdir = listdir(path=full, sub=True, JSON=False, gethash=gethash)
+            else:
+                info["Size"] = stat[6]
+                info["Type"] = "file"
+                if gethash:
+                    with open(full, "rb") as f:
+                        h = uhashlib.sha256(f.read())
+                        info["Hash"] = ubinascii.hexlify(h.digest())
+        except OSError as e:
+            log.error("error:{} processing file:{}".format(e, full))
+            info["OSError"] = e.args[0]
+            info["Type"] = "OSError"
+        info["Fullname"] = full
+        li.append(info)
+        # recurse folder(s)
+        if sub == True:
+            li = li + subdir
+    if JSON == True:
+        return json.dumps(li)
+    else:
+        return li

tests/quality_tests/feat_stdlib_only/check_re/check-Match.py

@@ -0,0 +1,9 @@
+from typing_extensions import assert_type
+
+import re
+Substring ='.*Python'
+String1 = "MicroPython"
+m =re.match(Substring, String1)
+
+assert m is not None
+assert_type(m, re.Match[str])

tests/quality_tests/feat_stdlib_only/check_re/check_compile.py

@@ -0,0 +1,15 @@
+import re
+from typing import Any, List, Union
+
+from typing_extensions import assert_type
+
+# As re doesn't support escapes itself, use of r"" strings is not
+# recommended.
+regex = re.compile("[\r\n]")
+
+result = regex.split("line1\rline2\nline3\r\n")
+
+assert_type(result, List[Union[str, Any]])
+
+# Result:
+# ['line1', 'line2', 'line3', '', '']