PyInstaller can't find Django migration files in monorepo architecture

[shiwa-sh]

I am trying to build a Django project using PyInstaller, but I'm encountering an issue related to migration files when working with a monorepo structure.

Build Command:

pyinstaller --paths="/home/shiva/PycharmProjects/dara" apps/backend/gateway/manage.py

INSTALLED_APPS in settings:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'apps.backend.gateway.api',
]

Error Message:

File "<frozen posixpath>", line 181, in dirname
TypeError: expected str, bytes or os.PathLike object, not NoneType

This error occurs when PyInstaller attempts to locate migration files for the api app. It seems like PyInstaller can't find the migration files in my monorepo structure.
Context

I am using a monorepo where the project root is /home/shiva/PycharmProjects/dara.
The Django app is located at apps/backend/gateway/api.
The error only happens when PyInstaller looks for the migration files during the build process.

Any help would be greatly appreciated!

[rokm]

Does it help if you add an (empty) __init__.py to the directory where your migration files are?

[shiwa-sh]

27804 INFO: Collecting Django migration scripts.
- module -- >  django.conf.app_template.migrations
- module -- >  django.contrib.admin.migrations
- module -- >  django.contrib.auth.migrations
- module -- >  django.contrib.contenttypes.migrations
- module -- >  django.contrib.flatpages.migrations
- module -- >  django.contrib.redirects.migrations
- module -- >  django.contrib.sessions.migrations
- module -- >  django.contrib.sites.migrations
- module -- >  django.contrib.messages.migrations
- module -- >  django_filters.migrations
- module -- >  django.contrib.admin.migrations
- module -- >  rest_framework_simplejwt.migrations
- module -- >  allauth.migrations
- module -- >  rest_framework.authtoken.migrations
- module -- >  allauth.account.migrations
- module -- >  django.contrib.contenttypes.migrations
- module -- >  drf_spectacular.migrations
- module -- >  rest_framework.migrations
- module -- >  django.contrib.sessions.migrations
- module -- >  django.contrib.auth.migrations
- module -- >  django.contrib.staticfiles.migrations
- module -- >  apps.backend.gateway.api.migrations
Traceback (most recent call last):
  File "/home/shiva/PycharmProjects/dara/.venv/bin/pyinstaller", line 8, in <module>
    sys.exit(_console_script_run())
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/__main__.py", line 231, in _console_script_run
    run()
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/__main__.py", line 215, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/__main__.py", line 70, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/building/build_main.py", line 1216, in main
    build(specfile, distpath, workpath, clean_build)
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/building/build_main.py", line 1156, in build
    exec(code, spec_namespace)
  File "/home/shiva/PycharmProjects/dara/manage.spec", line 4, in <module>
    a = Analysis(
        ^^^^^^^^^
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/building/build_main.py", line 556, in __init__
    self.__postinit__()
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/building/datastruct.py", line 184, in __postinit__
    self.assemble()
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/building/build_main.py", line 696, in assemble
    program_scripts.append(self.graph.add_script(script))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/depend/analysis.py", line 268, in add_script
    self._top_script_node = super().add_script(pathname)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 1160, in add_script
    self._process_imports(n)
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2591, in _process_imports
    target_modules = self._safe_import_hook(*import_info, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/depend/analysis.py", line 437, in _safe_import_hook
    ret_modules = super()._safe_import_hook(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2048, in _safe_import_hook
    target_modules = self.import_hook(
                     ^^^^^^^^^^^^^^^^^
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 1232, in import_hook
    target_package, target_module_partname = self._find_head_package(
                                             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 1411, in _find_head_package
    target_package = self._safe_import_module(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/depend/analysis.py", line 501, in _safe_import_module
    return super()._safe_import_module(module_basename, module_name, parent_package)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 1792, in _safe_import_module
    self._process_imports(n)
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/lib/modulegraph/modulegraph.py", line 2591, in _process_imports
    target_modules = self._safe_import_hook(*import_info, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/depend/analysis.py", line 368, in _safe_import_hook
    excluded_imports = self._find_all_excluded_imports(source_module.identifier)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/depend/analysis.py", line 356, in _find_all_excluded_imports
    excluded_imports.update(module_hook.excludedimports)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/depend/imphook.py", line 316, in __getattr__
    self._load_hook_module()
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/depend/imphook.py", line 383, in _load_hook_module
    self._hook_module = importlib_load_source(self.hook_module_name, self.hook_filename)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/compat.py", line 583, in importlib_load_source
    mod_loader.exec_module(mod)
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/shiva/PycharmProjects/dara/.venv/lib/python3.12/site-packages/PyInstaller/hooks/hook-django.py", line 74, in <module>
    mod_dir = os.path.dirname(hooks.get_module_file_attribute(mod_name))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 181, in dirname
TypeError: expected str, bytes or os.PathLike object, not NoneType

[rokm]

Hmm, so I think that means that we cannot handle the case when apps (= mod_name) is a namespace package. Does apps have an __init__.py file?

[shiwa-sh]

The build process completed successfully after adding an __init__.py file to the apps directory. However, when I try to run the file created by PyInstaller with the following command:
./manage runserver --noreload
I encounter this error:

Traceback (most recent call last):
  File "manage.py", line 14, in <module>
  File "manage.py", line 10, in main
  File "django/core/management/__init__.py", line 442, in execute_from_command_line
  File "django/core/management/__init__.py", line 382, in execute
  File "django/conf/__init__.py", line 89, in __getattr__
  File "django/conf/__init__.py", line 76, in _setup
  File "django/conf/__init__.py", line 190, in __init__
  File "importlib/__init__.py", line 90, in import_module
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "PyInstaller/loader/pyimod02_importers.py", line 419, in exec_module
  File "config/settings.py", line 9, in <module>
  File "environs/__init__.py", line 443, in read_env
  File "dotenv/main.py", line 262, in _walk_to_root
OSError: Starting path not found
[41895] Failed to execute script 'manage' due to unhandled exception!

Here is the relevant part of settings.py where the error occurs:

from environs import Env

env = Env()
env.read_env()

[rokm]

Try passing the path to the .env file to read_env (e.g., anchor it to __file__ attribute). You will also need to ensure that .env file is collected, if you have not already done so.

If no path is given, environs try to infer the path to caller via its co_filename (see here). This will not work in frozen application, because co_filename is anonymized into relative path (config/settings.py), and so it tries to access config directory in the current working directory, which does not exist.

[shiwa-sh]

Thank you for your help! The issue has been resolved. I really appreciate your quick response and support.