Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC 59: Roadmap for new StreamField development #59

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+46 −0
Open

RFC 59: Roadmap for new StreamField development #59

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions text/xxx-streamfield-roadmap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# RFC x: Roadmap for new StreamField development

* RFC: x
* Author: Matthew Westcott
* Created: 2020-08-28
* Last Modified: 2020-08-28

## Abstract

This RFC presents a roadmap for future development of StreamField, based on surveys, interviews and general feedback from Wagtail developers.

## Specification

Based on the Wagtail developer survey and follow-up interviews, we've found that developers have the following feature requests for StreamField:

### UX and performance improvements to StreamField within the Wagtail admin

These include a number of features already implemented in third-party code - the ability to duplicate blocks (currently implemented in react-streamfield) and to split rich text blocks into multiple blocks (currently implemented as a customisation by NYPR).

Features such as these are currently blocked from being added to Wagtail core as they rely on undocumented properties of the client-side code which cannot be generalised to all block types, or cannot be guaranteed to remain stable in future Wagtail releases.

To illustrate: the rich client-side UX as seen in react-streamfield is built upon the ability to perform fundamental operations such as: "create a new instance of a StreamField block, populated with a given value". However, a key principle of StreamField is that any Django form field may be used as a block - and Django form fields do not guarantee anything about the client-side behaviour of a field (except to the extent that it needs to be able to produce an HTML form submission that the Django-side component can make sense of). Assumptions that might seem safe, such as a Django form field corresponding to a single HTML input element, cannot be relied upon - and without that, there is no fully robust way to implement those fundamental client-side operations.

(StreamField itself is an extreme example of this assumption being broken - it presents itself to other Django-side code as one form field, but may consist of many HTML form elements. This in itself is not a showstopper for react-streamfield, since there's no real prospect of a developer trying to embed StreamField inside a block inside StreamField. However, it does imply that any future development of a similar complexity to StreamField will not be usable within react-streamfield; and conversely, any future development that attempts to make the same assumptions as react-streamfield is liable to fail when presented with a StreamField. In other words: the "eat your own dogfood" test is a meaningful one to apply here!)

This limitation also causes performance issues in the current implementation of StreamField - since blocks populated with a given value cannot be created client-side, displaying the edit view for an existing page requires these to be rendered up-front on the server, often generating a lot of redundant HTML.
Copy link
Member

@thibaudcolas thibaudcolas Sep 14, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’s not clear to me whether the work proposed below will address these, but might just be me failing to see it. Does the work below include serializing the whole StreamField’s value as a JSON structure server-side, for it to then be rendered client-side with the Django form fields adapters? I think the below implies that, but worth confirming.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on our discussion, the answer is "yes" :)

  • Serialising server-side, injecting extra data as needed (for example, de-referencing page ids to edit/live URLs + title)
  • Adapters then know how to render the serialised data into a form client-side
  • There are also further opportunities for optimisation (which might also be possible currently, but would make more sense as part of this type of rework) – having a global (page-level) registry of adapters/field templates, rather than having this repeated at different depths in the StreamField tree structure.


**Proposed development:** Define and implement a common interface for accessing and manipulating Django form fields on the front end. This will most likely take the form of a set of Javascript 'adapter' objects, one per field type, implementing operations on that field type such as "create a new instance of this field populated with the given value", and "extract the JSON-serialisable value from an instance of this field". Developers implementing new field types for use within StreamField will need to provide such an adapter for that field; for conventional form fields consisting of a single HTML input, this will be a straightforward and well-documented task.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there fields that are supported in the current StreamField implementation, and would no longer work over this paradigm? For example file uploads?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Side question – are there any examples of APIs similar to this in the Django world (say Saleor?), or in other frameworks? I don’t have the best sense of likely pitfalls.

Copy link
Contributor

@zerolab zerolab Sep 15, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/mirumee/saleor-dashboard is probably the best place to check how Saleor does it. The Dashboard is the store admin with various forms.

But we should probably contact the team and get some input from them

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the case of file upload fields, the setState method would have no effect, as browsers don't let you populate them from HTML or Javascript. However, the end result of that is the same as it would be for any other implementation: the field would be left empty on page load when editing an existing page, and on duplicating a block.

In theory you could embed the file data into JSON as a base-64 encoded string, but it's probably worth thinking about a less hacky solution... in the ordinary page-edit view, it probably makes most sense to stick to the Django-ish approach where a StreamField named body would generate a distinct form field named body-file-1 to be picked up by the server-side code. For auto-saving or a DRF JSON API, you'd probably want to come up with something different, so it's best that the JS API doesn't force you down one path. Maybe an additional method that returns the file as a File or FileList object for the calling code to use as it sees fit, then...

I can't think of any other field types that would need special treatment - if there's any way to interface with a field through JS, then it ought to be possible to write that code inside a Telepath adapter object, so I don't think this pattern constrains us in any way.


### Easier ways to manipulate StreamField data within Python code

Wagtail does not provide any formal APIs for manipulating StreamField data in place. The officially recommended way to update StreamField data is to construct a new data structure replicating the previous value but with the desired changes applied, and assign that back to the StreamField; for complex nested structures, this is difficult and in certain edge cases, impossible. Workarounds include building the StreamField value in JSON format (which is a less human-friendly representation and adds an extra data-conversion round trip) or accessing StreamField internals (such as the StreamValue.stream_data property) directly, which is often poorly understood (leading to logic errors, especially when previewing) and liable to change across Wagtail releases.

**Proposed development:** Extend the existing StreamValue class (currently designed to be an immutable value object) with list-like methods that allow it to be modified in-place. Initial work on this has been started, at: https://github.com/wagtail/wagtail/pull/2886

### Ability to build your own StreamField-like editing interfaces outside of the Wagtail admin and have those interoperate with Wagtail's StreamField

Developers frequently request the ability to use StreamField on a site front-end, to allow users to contribute to a site without having a Wagtail editor account. Given the amount of work involved in making StreamField work in the relatively controlled environment of the Wagtail admin, and ensuring that all possible combinations of elements are styled appropriately, we do not feel that it's currently feasible to build a StreamField component for site front-ends with the full generality of the Wagtail admin's StreamField. However, a special-purpose StreamField interface built for one specific page type with a fixed set of blocks can take various shortcuts: it does not need to interoperate with Django's form framework, will most likely not need to handle arbitrary nesting of blocks and has fewer styling combinations to test.

As this would be site-specific bespoke code, written in the site implementer's preferred front-end framework, Wagtail itself would not provide any tools for building this; however, Wagtail does need to provide a way for the resulting data to be submitted back so that a page can be created.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea of a REST API more than a reusable StreamField component implementation – but it feels like it wouldn’t be too much work to provide both? (a React implementation at least). It’s good design for a StreamField component to have explicit, direct dependencies, and be renderable outside of Wagtail when those dependencies are met.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's probably an 80/20 rule at work here - 80% of StreamField users only use 20% of the functionality, but it's never the same 20%.

I certainly wouldn't be against a full-featured standalone StreamField implementation (and having one would improve the code quality of Wagtail a lot - the "better chuck _editor_js.html into your template if you want to do anything useful" thing bothers me immensely...) - but I think that building it would be a much bigger job than it looks at first glance. Taking something like ImageChooserBlock - there's the immediate job of untangling the dependencies of image_chooser_modal, modal_workflow and so on, but there's also the more general issue of what it implies to give a front-end user access to the image library, and whether different projects need different approaches for how to lock permissions down... at that point, it feels more pragmatic to say "if you're part of the 20% that actually need an image chooser in your bespoke front-end StreamField UI, then go ahead and build it with whatever business rules make sense to you".

I also get the sense that the CSS would be as much of a nightmare as the Javascript side, but you have a better perspective on that than I do :-)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this would be site-specific bespoke code, written in the site implementer's preferred front-end framework, Wagtail itself would not provide any tools for building this; however, Wagtail does need to provide a way for the resulting data to be submitted back so that a page can be created.

for this, it can't be done by any built-in modifications/implementations, as you already said (preferred front-end framework) and also what block types the site builders decide to make them available for site visitors, therefore, the best way wagtail community and core team can offer, is to provide few use case samples, and let the site-builders dig deeper and innovate as the develop their projects.


**Proposed development:** Implement a REST API endpoint to allow creating pages from posted JSON data, where any StreamFields are also represented in JSON format. (It is expected that JSON will be a more convenient format for front-end developers to work with, rather than the HTML form submission currently used by the Wagtail admin.)
Copy link
Contributor

@kaedroho kaedroho Sep 21, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also allow this to be used for other snippets and modeladmin as well?

Copy link
Contributor Author

@gasman gasman Sep 21, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, why not... presumably implementors would register an API endpoint for each distinct model, just as they currently do for the read API, so effectively all that means is that we need to provide a base viewset class that doesn't have the page publishing workflow hardcoded into it.


## Further development

Items 1 and 3 combined could potentially serve as the basis for an auto-save feature in the Wagtail page editor: the Javascript adapter system (item 1) would provide a way to extract the contents of the active edit view as JSON, and a background AJAX request could then post this JSON to a new REST API endpoint (item 3) that saves the data to a draft revision.
Copy link
Member

@thibaudcolas thibaudcolas Sep 14, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’d suggest fleshing out this section further, to sanity-check that the developments above do indeed allow for those features – feels like this is our chance to get the foundations right so we get to build those exciting features, that will make Wagtail feel more modern.

Specific features that could be interesting to validate:

  • Auto-save of page edits in the admin as outlined here
  • Inline editing of pages (directly on the site’s pages, outside of the admin). I think the save mechanism is similar to auto-save(?), but it might make us think differently on how to package the client-side APIs and UI components.
  • Instant preview in the CMS – I think this is already buildable with the current implementations, but worth checking whether any of the above makes it easier – for example a "render preview" REST API.
  • Collaborative editing – auto-save problems taken further, making sure the page data is also serialisable in a format that’s collaboration-friendly ("diffs over websockets") like CRDTs or operational transforms.
  • StreamField copy-paste – Clipboard-serialisable StreamField content for cross-pages (or cross-sites!) copy-paste.

Copy link
Contributor

@zerolab zerolab Sep 15, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://wagtailcms.slack.com/archives/C014L7KJH3N/p1599936875057300 has a demo of "Inline editing of pages (directly on the site’s pages, outside of the admin)". https://gitlab.com/kocherga/core/-/commit/ef1638ff0ede9305462a9096bd486e53ae6fbf73 for the code implementing that via GraphQL. It is quite specific to the project, but showw appetite for it