Try to simplify NApps requirements/dev.txt for trusted transitive dependencies from kytos-ng org

[viniarck]

Currently, NApps are also pinning transitive dependencies from kytos in their requirements/dev.txt, which is typically what's generally recommended for Python projects, but in this case specifically kytos has already pinned on its requirements/run.txt so it's guaranteed that NApps will have it, and we have ownership of kytos so we can trust it on NApps unlike other third-parties dependencies.

If we manage to simplify this, then wen regenerating core dependencies, it'll only require a single PR as opposed to dozens in cases when it doesn't completely break compatibility (which tends to be most the vast majority of the cases when minor upstream updates or dependencies are bumped), which will drastically improve our team work flow.

[viniarck]

We ended up with a stricter pinning approach some time ago, but that was wen even third parties weren't strictly pinned.

Also, this opens the possibility for kytos to also provide a [test] set of dependencies on setup.py mode, which NApps can opt to install simplifying which testing libs should be installed as well, NApps already rely on core testing utilities so that'd be helpful too.

[viniarck]

After doing some research, with pip-compile, it's possible to exclude packages and transitive packages from the compiled dependency tree. In summary, we can exclude trustworthy transitive kytos dependency while still getting them pinned since kytos package ensures this. So whenever a core dependency changes you don't need to regenerate requirements for NApps, which currently would require dozens of PRs for Kytos-ng org at least. This also means that if core upgrades a new major development breaking change, then NApps might break on master, but we have control over this situation so it won't catch us by surprise.

We can use a concept of --unsafe-package=xyz to leave it unpinned, the name unsafe at first might be alarming but indeed what it does is just get ignored, the unsafe list is a concept of pip-compile. We can have a script getting kytos pinned dependencies and pass them as --unsafe-package=xyz. Here's an example, of a NApp dev.txt that has kytos and requests as dependencies:

-e git+https://github.com/kytos-ng/kytos.git#egg=kytos
requests

#
# This file is autogenerated by pip-compile with Python 3.9
# by the following command:
#
#    pip-compile --output-file=requirements/dev33.txt --unsafe-package=asgiref --unsafe-package=asttokens --unsafe-package=backcall --unsafe-package=bidict --unsafe-package=blinker --unsafe-package=certifi --unsafe-package=click --unsafe-package=decorator --unsafe-package=dnspython --unsafe-package=docutils --unsafe-package=elastic-apm --unsafe-package=email-validator --unsafe-package=executing --unsafe-package=flask --unsafe-package=flask-cors --unsafe-package=flask-socketio --unsafe-package=idna --unsafe-package=importlib-metadata --unsafe-package=ipython --unsafe-package=itsdangerous --unsafe-package=janus --unsafe-package=jedi --unsafe-package=jinja2 --unsafe-package=lockfile --unsafe-package=markupsafe --unsafe-package=matplotlib-inline --unsafe-package=parso --unsafe-package=pexpect --unsafe-package=pickleshare --unsafe-package=prompt-toolkit --unsafe-package=ptyprocess --unsafe-package=pure-eval --unsafe-package=pydantic --unsafe-package=pygments --unsafe-package=pyjwt --unsafe-package=pymongo --unsafe-package=python-daemon --unsafe-package=python-engineio --unsafe-package=python-openflow --unsafe-package=python-socketio --unsafe-package=six --unsafe-package=stack-data --unsafe-package=tenacity --unsafe-package=traitlets --unsafe-package=typing-extensions --unsafe-package=urllib3 --unsafe-package=watchdog --unsafe-package=wcwidth --unsafe-package=werkzeug --unsafe-package=zipp requirements/dev.in
#
charset-normalizer==3.1.0
    # via requests
requests==2.28.2
    # via -r requirements/dev.in
-e git+https://github.com/kytos-ng/kytos.git#egg=kytos
    # via -r requirements/dev.in

Notice that during installation kytos transitive dependencies installed ultimately were the ones that kytos pinned:

❯ pip3 install -r requirements/dev33.txt --no-cache
Obtaining kytos from git+https://github.com/kytos-ng/kytos.git#egg=kytos (from -r requirements/dev33.txt (line 11))
  Cloning https://github.com/kytos-ng/kytos.git to ./.direnv/python-3.9.16/src/kytos
  Running command git clone --filter=blob:none --quiet https://github.com/kytos-ng/kytos.git /home/viniarck/repos/pip-t/.direnv/python-3.9.16/src/kytos
  Resolved https://github.com/kytos-ng/kytos.git to commit ad4b118f773f7bf806818aea0a8b232b9ae7535a
  Preparing metadata (setup.py) ... done
Collecting charset-normalizer==3.1.0
  Downloading charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 199.2/199.2 KB 3.1 MB/s eta 0:00:00
Collecting requests==2.28.2
  Downloading requests-2.28.2-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 KB 41.1 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
  Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 KB 26.9 MB/s eta 0:00:00
Collecting idna<4,>=2.5
  Downloading idna-3.4-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 KB 353.4 MB/s eta 0:00:00
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.9/140.9 KB 57.4 MB/s eta 0:00:00
Collecting asgiref==3.5.2
  Downloading asgiref-3.5.2-py3-none-any.whl (22 kB)
Collecting asttokens==2.0.8
  Downloading asttokens-2.0.8-py2.py3-none-any.whl (23 kB)
Collecting backcall==0.1.0
  Downloading backcall-0.1.0.zip (11 kB)
  Preparing metadata (setup.py) ... done
Collecting bidict==0.22.0
  Downloading bidict-0.22.0-py3-none-any.whl (36 kB)
Collecting blinker==1.4
  Downloading blinker-1.4.tar.gz (111 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 111.5/111.5 KB 366.2 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting certifi>=2017.4.17
  Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 149.2/149.2 KB 80.3 MB/s eta 0:00:00
Collecting click==8.1.3
  Downloading click-8.1.3-py3-none-any.whl (96 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.6/96.6 KB 30.6 MB/s eta 0:00:00
Collecting decorator==4.4.2
  Downloading decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting dnspython==2.2.1
  Downloading dnspython-2.2.1-py3-none-any.whl (269 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 269.1/269.1 KB 39.0 MB/s eta 0:00:00
Collecting docutils==0.19
  Downloading docutils-0.19-py3-none-any.whl (570 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 570.5/570.5 KB 28.5 MB/s eta 0:00:00
Collecting elastic-apm[flask]==6.9.1
  Downloading elastic_apm-6.9.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (376 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 376.8/376.8 KB 30.8 MB/s eta 0:00:00
Collecting email-validator==1.3.0
  Downloading email_validator-1.3.0-py2.py3-none-any.whl (22 kB)
Collecting executing==1.0.0
  Downloading executing-1.0.0-py2.py3-none-any.whl (16 kB)
Collecting flask[async]==2.1.3
  Downloading Flask-2.1.3-py3-none-any.whl (95 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.6/95.6 KB 93.6 MB/s eta 0:00:00
Collecting flask-cors==3.0.10
  Downloading Flask_Cors-3.0.10-py2.py3-none-any.whl (14 kB)
Collecting flask-socketio==5.2.0
  Downloading Flask_SocketIO-5.2.0-py3-none-any.whl (17 kB)
Collecting idna<4,>=2.5
  Downloading idna-3.3-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 KB 323.3 MB/s eta 0:00:00
Collecting importlib-metadata==4.12.0
  Downloading importlib_metadata-4.12.0-py3-none-any.whl (21 kB)
Collecting ipython==8.1.1
  Downloading ipython-8.1.1-py3-none-any.whl (750 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 750.3/750.3 KB 26.8 MB/s eta 0:00:00
Collecting itsdangerous==2.1.2
  Downloading itsdangerous-2.1.2-py3-none-any.whl (15 kB)
Collecting janus==1.0.0
  Downloading janus-1.0.0-py3-none-any.whl (6.9 kB)
Collecting jedi==0.16.0
  Downloading jedi-0.16.0-py2.py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 26.6 MB/s eta 0:00:00
Collecting jinja2==3.1.2
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 KB 215.5 MB/s eta 0:00:00
Collecting lockfile==0.12.2
  Downloading lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
Collecting markupsafe==2.1.1
  Downloading MarkupSafe-2.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting matplotlib-inline==0.1.6
  Downloading matplotlib_inline-0.1.6-py3-none-any.whl (9.4 kB)
Collecting parso==0.6.2
  Downloading parso-0.6.2-py2.py3-none-any.whl (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.9/97.9 KB 355.9 MB/s eta 0:00:00
Collecting pexpect==4.8.0
  Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.0/59.0 KB 262.1 MB/s eta 0:00:00
Collecting pickleshare==0.7.5
  Downloading pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting prompt-toolkit==3.0.5
  Downloading prompt_toolkit-3.0.5-py3-none-any.whl (351 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 351.4/351.4 KB 36.5 MB/s eta 0:00:00
Collecting ptyprocess==0.6.0
  Downloading ptyprocess-0.6.0-py2.py3-none-any.whl (39 kB)
Collecting pure-eval==0.2.2
  Downloading pure_eval-0.2.2-py3-none-any.whl (11 kB)
Collecting pydantic==1.9.0
  Downloading pydantic-1.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.2/12.2 MB 24.4 MB/s eta 0:00:00
Collecting pygments==2.13.0
  Downloading Pygments-2.13.0-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 28.0 MB/s eta 0:00:00
Collecting pyjwt==2.4.0
  Downloading PyJWT-2.4.0-py3-none-any.whl (18 kB)
Collecting pymongo==4.1.0
  Downloading pymongo-4.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (469 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 469.9/469.9 KB 30.7 MB/s eta 0:00:00
Collecting python-daemon==2.3.1
  Downloading python_daemon-2.3.1-py2.py3-none-any.whl (34 kB)
Collecting python-engineio==4.3.4
  Downloading python_engineio-4.3.4-py3-none-any.whl (52 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.9/52.9 KB 420.8 MB/s eta 0:00:00
Collecting python-openflow
  Downloading python_openflow-2021.1-py3-none-any.whl (198 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 198.6/198.6 KB 38.5 MB/s eta 0:00:00
Collecting python-socketio==5.7.1
  Downloading python_socketio-5.7.1-py3-none-any.whl (56 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.6/56.6 KB 309.8 MB/s eta 0:00:00
Collecting six==1.16.0
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting stack-data==0.5.0
  Downloading stack_data-0.5.0-py3-none-any.whl (24 kB)
Collecting tenacity==8.0.1
  Downloading tenacity-8.0.1-py3-none-any.whl (24 kB)
Collecting traitlets==5.3.0
  Downloading traitlets-5.3.0-py3-none-any.whl (106 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 106.8/106.8 KB 138.9 MB/s eta 0:00:00
Collecting typing-extensions>=4.0.1
  Downloading typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 138.8/138.8 KB 270.9 MB/s eta 0:00:00
Collecting watchdog==2.1.9
  Downloading watchdog-2.1.9-py3-none-manylinux2014_x86_64.whl (78 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.4/78.4 KB 524.5 MB/s eta 0:00:00
Collecting wcwidth==0.1.9
  Downloading wcwidth-0.1.9-py2.py3-none-any.whl (19 kB)
Collecting werkzeug==2.0.3
  Downloading Werkzeug-2.0.3-py3-none-any.whl (289 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 289.2/289.2 KB 31.3 MB/s eta 0:00:00
Collecting zipp==3.8.1
  Downloading zipp-3.8.1-py3-none-any.whl (5.6 kB)
Collecting Flask>=0.9
  Downloading Flask-2.2.3-py3-none-any.whl (101 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 101.8/101.8 KB 60.0 MB/s eta 0:00:00
Requirement already satisfied: setuptools>=18.5 in ./.direnv/python-3.9.16/lib/python3.9/site-packages (from ipython==8.1.1->kytos->-r requirements/dev33.txt (line 11)) (58.1.0)
WARNING: The candidate selected for download or install is a yanked version: 'python-daemon' candidate (version 2.3.1 at https://files.pythonhosted.org/packages/aa/b0/bc79d8ff019c2583d83
9e0143b1f91eafd4cfe92f86fb9d378a515dfb612/python_daemon-2.3.1-py2.py3-none-any.whl#sha256=4e3bf67784c78aaa55ec001a2f832b464a54c5f9c89c11b311e2416a8c247431 (from https://pypi.org/simple/p
ython-daemon/))
Reason for being yanked: Incorrectly declares Python 2 support
Using legacy 'setup.py install' for backcall, since package 'wheel' is not installed.
Using legacy 'setup.py install' for blinker, since package 'wheel' is not installed.
Installing collected packages: wcwidth, python-openflow, pure-eval, ptyprocess, pickleshare, lockfile, executing, certifi, blinker, backcall, zipp, werkzeug, watchdog, urllib3, typing-ex
tensions, traitlets, tenacity, six, python-engineio, pymongo, pyjwt, pygments, prompt-toolkit, pexpect, parso, markupsafe, itsdangerous, idna, docutils, dnspython, decorator, click, char
set-normalizer, bidict, asgiref, requests, python-socketio, python-daemon, pydantic, matplotlib-inline, jinja2, jedi, janus, importlib-metadata, email-validator, elastic-apm, asttokens, 
stack-data, flask, ipython, flask-socketio, flask-cors, kytos
  Running setup.py install for blinker ... done
  Running setup.py install for backcall ... done
  Running setup.py develop for kytos
Successfully installed asgiref-3.5.2 asttokens-2.0.8 backcall-0.1.0 bidict-0.22.0 blinker-1.4 certifi-2021.10.8 charset-normalizer-3.1.0 click-8.1.3 decorator-4.4.2 dnspython-2.2.1 docut
ils-0.19 elastic-apm-6.9.1 email-validator-1.3.0 executing-1.0.0 flask-2.1.3 flask-cors-3.0.10 flask-socketio-5.2.0 idna-3.3 importlib-metadata-4.12.0 ipython-8.1.1 itsdangerous-2.1.2 ja
nus-1.0.0 jedi-0.16.0 jinja2-3.1.2 kytos-2022.3.1 lockfile-0.12.2 markupsafe-2.1.1 matplotlib-inline-0.1.6 parso-0.6.2 pexpect-4.8.0 pickleshare-0.7.5 prompt-toolkit-3.0.5 ptyprocess-0.6
.0 pure-eval-0.2.2 pydantic-1.9.0 pygments-2.13.0 pyjwt-2.4.0 pymongo-4.1.0 python-daemon-2.3.1 python-engineio-4.3.4 python-openflow-2021.1 python-socketio-5.7.1 requests-2.28.2 six-1.1
6.0 stack-data-0.5.0 tenacity-8.0.1 traitlets-5.3.0 typing-extensions-4.5.0 urllib3-1.26.7 watchdog-2.1.9 wcwidth-0.1.9 werkzeug-2.0.3 zipp-3.8.1
WARNING: You are using pip version 22.0.4; however, version 23.0.1 is available.
You should consider upgrading via the '/home/viniarck/repos/pip-t/.direnv/python-3.9.16/bin/python3.9 -m pip install --upgrade pip' command.

~/repos/pip-t master*  18s
❯ pipf G idna
idna==3.3

~/repos/pip-t master* 
❯ pipf G certifi
certifi==2021.10.8

[viniarck]

Another alternative for dev requirements it would be to rely on requirements/dev.in it would still benefit from the pinned dependencies from requirements/run.txt and also transitive pinned dependencies from kytos. This would be even easier to maintain instead of --unsafe-package=xyz, although --unsafe-package=xyz is convenient too.

[viniarck]

Related NApps PRs have landed.