feat: Create OEP-0065 Composable Micro-frontends

oeps/architectural-decisions/oep-0065-composable-micro-frontends.rst

@@ -0,0 +1,160 @@
+==================================
+OEP-65: Composable Micro-frontends
+==================================
+
+.. list-table::
+   :widths: 25 75
+
+   * - OEP
+     - :doc:`OEP-65 <oep-0065-composable-micro-frontends>`
+   * - Title
+     - Composable Micro-frontends
+   * - Last Modified
+     - 2023-12-05
+   * - Authors
+     - 
+       * Adolfo R. Brandes <[email protected]>
+       * Pedro Martello <[email protected]>
+   * - Arbiter
+     - Adam Stankiewicz <[email protected]>
+   * - Status
+     - Draft
+   * - Type
+     - Architecture
+   * - Created
+     - 2022-11-09
+   * - Review Period
+     - 2023-12-05 - 2023-12-19
+   * - Resolution
+     - TBD
+   * - References
+     - TBD
+
+
+***********
+Terminology
+***********
+
+* *MFE*: a micro-frontend
+* *SPA*: single-page application
+
+
+********
+Abstract
+********
+
+The decision to gradually implement a micro-frontend architecture for Open edX was a largely successful attempt at
+avoiding some limitations of the monolithic ``edx-platform`` codebase.  In particular, it allowed the project to
+integrate more and better features by leveraging newer frontend technology, all much faster than would've been possible
+otherwise.  At the same time, because features encapsulated in MFEs are very loosely coupled to each other, this
+architecture eliminates most integration bottlenecks, allowing individual teams working on any given feature to be a lot
+more productive.
+
+However, roughly 3 years into this gradual re-platforming, several drawbacks to this choice of complete MFE independence
+have been identified.  This OEP proposes a middle-ground based on current industry best-practices, where a minor
+increase in integration costs is traded for large gains in user experience, customizability, ease of deployment, and
+development efficiency.
+
+
+**********
+Motivation
+**********
+
+When moving to the micro-frontend architecture, a choice was made to have *completely* independent MFEs.  Each a
+separate SPA, one could go as far as employing a totally different technology stack for each one.  While the goal of making
+development teams be more independent, and thus at least theoretically more productive, was achieved, the price to pay
+was not insignificant.  The following are examples of the impact this has had on the different classes of users of the
+platform:
+
+  * Learners: each MFE uses whatever paradigm (from design language to version of Paragon) was in vogue during its
+    development, leading to noticeable differences in UX and performance; this is made progressively worse if the MFE is
+    not actively maintained
+
+  * Operators: as any given MFE can implement common features differently, or since MFEs can lag behind others in
+    adoption of new versions of Paragon, more customization work is necessary to guarantee a unified experience
+
+  * Maintainers: the job of keeping all repositories up to date is multiplied by the number of MFEs and their individual
+    dependency graphs; while this comes with the territory of maintaining a large project, the fact security fixes to
+    common libraries such as React or Paragon have to be considered individually per repository, for example, makes the
+    job that much harder
+
+  * Developers: Jumping from one MFE to another is now a tricky proposition; one has to challenge all assumptions: what
+    is the version of React being used?  Of Paragon?  Does this MFE use the header component or not?  This makes Open
+    edX frontend development, particularly taken as whole, much more difficult, negating some of the benefits of using a
+    common technology stack in the first place
+
+That's not all, though.  Letting the browser load a slightly similar but ultimately different set of packages for every
+page begs for optimization.  Why make the user download 10 different versions of `react-dom` if this can be avoided?
+Can we go further and avoid most page-loads in the first place?
+
+Surely these are issues that the industry has already faced, and possibly solved.  It is thus a good time to pick
+between existing solutions - specified below - weighing both the development cost of implementation and future recurring
+ones against their benefits.
+
+
+*************
+Specification
+*************
+
+We propose a compromise between the monolithic codebase and full MFE independence: the Open edX frontend will be unified
+into just a couple of single-page applications - initially just the LMS and CMS - each of which will be implemented via
+a so-called "shell application".  The shell application, in turn, will render multiple MFEs as sub-components of itself,
+routing between them dynamically and loading common dependencies only once.  This will require MFEs to be re-factored to
+conform to the shell application's interfaces, including, for instance, the list of dependencies that they will now
+inherit.
+
+The chosen technology is `Piral <https://github.com/smapiot/piral>`_, and a reference implementation can be found in
+the `frontend-app-shell <https://github.com/openedx/frontend-app-shell>`_ repository.  For the purposes of this OEP, the
+latter should be taken as the canonical specification for how Piral should be used with Open edX, but the following
+guiding principles should hold true:
+
+  * MFEs should not reimplement common features such as headers, footers, and navigation, and should instead rely on the
+    shell to provide them
+
+  * Libraries elected to be common to all MFEs such as React and Paragon should be declared as peer dependencies in the
+    MFE, and provided by the app shell
+
+  * MFEs should be able to provide "hints" so that external page elements can configure themselves in context (for
+    example, adding links to a header dropdown), but the MFEs need not be aware of how those hints are implemented
+
+
+***********************
+Implementation Strategy
+***********************
+
+It is suggested that the conversion be undertaken gradually, where each release sees a few more MFEs being pulled into the shell
+application.
+
+
+*********************
+Rejected alternatives
+*********************
+
+At time of writing these were commonly recommended strategies for integration of MFEs into single-page apps:
+
+* `Run-time integration via JavaScript <https://martinfowler.com/articles/micro-frontends.html#Run-timeIntegrationViaJavascript>`_
+* `Run-time integration via Web Components <https://martinfowler.com/articles/micro-frontends.html#Run-timeIntegrationViaWebComponents>`_
+* `Webpack Module Federation <https://webpack.js.org/concepts/module-federation/>`_
+* `single-spa <https://single-spa.js.org/>`_
+
+They were rejected in favor of Piral as per the following checklist:
+
+  * Cost of conversion: how easy is it to convert existing MFEs to the new paradigm?
+  * Cost of integration: compare the level of MFE independence.
+  * Hosting of build artifacts: is it possible to host build artifacts separately on object storage or CDN?
+  * Lazy loading: is it possible to load MFEs and other components lazily?
+  * Efficiency of development: consider the level of boiler-plate code.
+  * User impact: compare the benefits and losses from the point of view of the browser.
+  * Maturity: how reliable is the codebase?  Is it maintained actively?  Can it be relied upon to exist in 5 years?
+  * Project health: is the project active (measure commit activity)?  How active is the community?
+
+
+**************
+Change History
+**************
+
+2023-12-05
+==========
+
+* Document created
+* `Pull request #XXX <https://github.com/openedx/open-edx-proposals/pull/XXX>`_

oeps/architectural-decisions/oep-0065-modular-micro-frontend-domains.rst

@@ -0,0 +1,228 @@
+======================================
+OEP-65: Modular Micro-frontend Domains
+======================================
+
+.. list-table::
+   :widths: 25 75
+
+   * - OEP
+     - :doc:`OEP-65 <oep-0065-modular-micro-frontend-domains>`
+   * - Title
+     - Modular Micro-frontend Domains
+   * - Last Modified
+     - 2023-07-20
+   * - Authors
+     - 
+       * Adolfo R. Brandes <[email protected]>
+       * Pedro Martello <[email protected]>
+   * - Arbiter
+     - Adam Stankiewicz <[email protected]>
+   * - Status
+     - Draft
+   * - Type
+     - Architecture
+   * - Created
+     - 2022-11-09
+   * - Review Period
+     - 2023-07-31 - 2023-09-01
+   * - Resolution
+     - TBD
+   * - References
+     - TBD
+
+
+***********
+Terminology
+***********
+
+* *MFE*: a micro-frontend
+* *SPA*: single-page application
+* *DDD*: domain-driven development
+
+
+********
+Abstract
+********
+
+The decision to gradually implement a micro-frontend architecture for Open edX was a largely successful attempt at avoiding some limitations of the monolithic ``edx-platform`` codebase.  In particular, it allowed the project to integrate more and better features by leveraging newer frontend technology, all much faster than would've been possible otherwise.  At the same time, because features encapsulated in MFEs are very loosely coupled to each other, this architecture eliminates most integration bottlenecks, allowing individual teams working on any given feature to be a lot more *productive*.
+
+However, roughly 3 years into this gradual re-platforming, several drawbacks to the choice of complete MFE independence have been identified.  This OEP proposes a middle-ground based on current micro-frontend best-practices, where a minor increase in integration costs is traded for large gains in user experience, customizability, ease of deployment, and development efficiency.
+
+
+**********
+Motivation
+**********
+
+When moving to the micro-frontend architecture, a choice was made to have *completely* independent MFEs, each hosted as a separate SPA, each with its own deployment, separate URL, and potentially different technology stack.  While the goal of developing platform features more efficiently was achieved, the price to pay was not insignificant.  What follows is a list of issues that were either introduced or exacerbated by this decision that the authors intend to address via this OEP, along with acceptance criteria that any proposed solution would have to satisfy.
+
+-------------------
+1. UX Inconsistency
+-------------------
+
+While ultimately a product (as opposed to architectural) question, the fact each MFE is developed independently, often reimplementing common components such as headers, made it much easier for individual teams to veer off from the design language implemented elsewhere in the platform.  The LMS header is the more egregious - and unfortunately, common - example: MFEs can not only use different versions of ``frontend-compoment-header``, which can also lead to minor visual changes, they can reimplement the header component entirely.
+
+This has impacted:
+
+  * UI being designed differently, resulting in poorer UX across boundaries
+  * UI being *implemented* differently, making tooling, development, and branding that much harder than before
+  * Each MFE can use whatever paradigm was in vogue during its development, leading to differences (some minor, some major) in UX;
+  * This makes it inordinately difficult to theme components that are not common to all MFEs, such as headers and footers.
+  * Operators: as any given MFE can implement the LMS header differently, it's harder to implement a common experience
+  * Learners: unless operators engange in heavy custom theming, the learner will be presented with potentially different
+
+Acceptance criteria
+-------------------
+
+  * Individual micro-frontends should not have to reimplement - either via importing a module or writing a local component - common domain features such as headers, footers, navigation, and others
+  * MFEs should be able to provide "hints" so that external page elements can configure themselves in context (for example, adding links to a header dropdown), but the MFEs need not be aware of how those hints are implemented
+
+--------------------------------
+2. Regression in customizability
+--------------------------------
+
+The move to MFEs abandoned "Comprehensive Theming" by design: the same level of customizability is now possible by forking individual repositories.  However, the fact that each MFE uses potentially different versions of libraries - including Paragon - makes it 
+
+  * To achieve similar customizability to the legacy codebase, MFEs need to be forked, with all the maintenance cost that that entails;
+  * The fact each MFE is fully independent means that it is harder to implement a consistent plugin interface between all of them.
+
+Acceptance criteria
+-------------------
+
+--------------------------------------------
+Changes that affect multiple MFEs are costly
+--------------------------------------------
+
+  * Development effort is multiplied by the number of existing MFEs, as each change needs to be duplicated;
+  * Because no dependencies are shared, Webpack build time of any potentially shared library is multiplied by the number of existing MFEs.
+
+Acceptance criteria
+-------------------
+
+------------------------------------
+Inefficient use of network bandwidth
+------------------------------------
+
+  * Inability to route (e.g., with react-router) between MFEs, necessitating a full refresh each time;
+  * Dependencies have to be downloaded multiple times by the browser.
+
+Acceptance criteria
+-------------------
+
+It is thus a good time to consider these practices - specified below - weighing both the development cost of implementation and future recurring ones against their benefits.
+
+
+*************
+Specification
+*************
+
+We propose a compromise between the monolithic codebase and full MFE independence based on Domain Driven Development: the Open edX frontend will be split into formal top-level domains, including but not limited to LMS and CMS, each of which will be implemented as a Container SPA.  The container, in turn, will render multiple partly-independent MFEs as sub-components.  The latter will be re-engineered to conform to the hosting domain's interfaces.
+
+This is to be a full re-platforming, to be undertaken as a single long-term project prior to release.  It is in opposition to the currently in-progress gradual conversion of single pages to fully independent MFEs that can be linked to from the pre-existing monolithic framework; the opposite, injecting legacy pages into a domain SPA as components, would be unjustifiable.  The difficulty of working with the legacy codebase is the main reason why it was decided to move to MFEs in the first place!
+
+On the other hand, the current gradual conversion to fully independent MFEs need not be interrupted; the domain-based re-platforming proposed here can be executed in parallel.  Why?  Converting an already fully independent MFE to a domain-hosted one is much less costly than the initial corresponding extraction from the monolith.  And conversely, it is conceivable that a particular monolithic page, after extraction directly into a domain-hosted MFE, could then be cheaply backported *the other way*, into a fully-independent one for earlier release while the re-platforming project is incomplete.
+
+As this OEP is developed, we will further investigate:
+
+#. Integration strategy
+#. Domain SPAs and their interfaces
+#. Shared dependencies
+#. MFEs as components
+
+Discovery
+=========
+
+* Are there any further top-level architectural elements to consider?
+* What is the relationship of this project to the Modular Learning effort?
+
+  * Both are long-term projects, but beyond that, is there anything else that could be coordinated between them?
+  * Should this project target the modular codebase exclusively, ignoring ``edx-platform``?  (Relevant if the APIs are incompatible between them.)
+
+Integration strategy
+====================
+
+To be defined.
+
+Discovery
+---------
+
+At time of writing these appear to commonly recommended strategies for integration of MFEs into modular container apps:
+
+* `Run-time integration via JavaScript <https://martinfowler.com/articles/micro-frontends.html#Run-timeIntegrationViaJavascript>`_
+* `Run-time integration via Web Components <https://martinfowler.com/articles/micro-frontends.html#Run-timeIntegrationViaWebComponents>`_
+* `Webpack Module Federation <https://webpack.js.org/concepts/module-federation/>`_
+* `single-spa <https://single-spa.js.org/>`_
+* `Piral <https://github.com/smapiot/piral>`_
+
+With these in mind:
+
+* What are the pros and cons of each integration option, in particular in terms of:
+
+  * Cost of conversion: how easy is it to convert existing MFEs to the new paradigm?
+  * Cost of integration: compare the level of MFE independence.
+  * Hosting of build artifacts: is it possible to host build artifacts separately on object storage or CDN?
+  * Lazy loading: is it possible to load MFEs and other components lazily?
+  * Efficiency of development: consider the level of boiler-plate code.
+  * User impact: compare the benefits and losses from the point of view of the browser.
+  * Maturity: how reliable is the codebase?  Is it maintained actively?  Can it be relied upon to exist in 5 years?
+  * Project health: is the project active (measure commit activity)?  How active is the community?
+
+* Recommend a single choice between the integration options, or suggest another, better alternative.
+
+Domain SPAs and their interfaces
+================================
+
+To be defined.
+
+Discovery
+---------
+
+* What UX elements should go into the domain's containing app? The header? Footer?
+* Define further container SPA and interface characteristics.
+
+Shared dependencies
+===================
+
+To be defined.
+
+Discovery
+---------
+
+* What dependencies should be shared between all MFEs?  (React, Redux, ...?)
+* Should we use lazy loading, and if so, how?
+
+
+MFEs as components
+==================
+
+To be defined.
+
+Discovery
+---------
+
+* Define how an MFE will connect to the containing SPA's interfaces.
+* Can an MFE be used across domains?
+
+
+************************
+Reference Implementation
+************************
+
+To be defined.
+
+
+*********************
+Rejected Alternatives
+*********************
+
+To be defined.
+
+
+**************
+Change History
+**************
+
+2022-11-09
+==========
+
+* Document created
+* `Pull request #XXX <https://github.com/openedx/open-edx-proposals/pull/XXX>`_