Telepath - the next evolution of StreamField
Adding the missing link to provide richer client-side behaviour in StreamField and beyond
I think it's fair to say that StreamField has been one of the most successful and popular things we've built within Wagtail - it's a design that's stood the test of time. Nothing in software is ever really complete, though... as developers become more confident using it, it's being used in ever more complex and demanding applications that stretch StreamField's performance and UI usability to the limit. It's not unknown for page content to run to tens or even hundreds of blocks - and in the classic Django form paradigm, that adds up to a lot of HTML being rendered server-side.
Faced with this problem, it's natural to look at shifting this work to the browser side, building on client-side frameworks such as React to turn StreamField's JSON data into a rich editing interface. The react-streamfield project born out of NoriPyt's Wagtail's First Hatch crowdfunder has led the way on this, and many of the UI improvements introduced there have already been adopted into Wagtail itself as part of the 2.7 release.
However, up to now one thing has stood in the way of Wagtail embracing the react-streamfield model in full: backwards compatibility. Wagtail site creators have invested much effort into building custom StreamField blocks from Django form widgets, and we don't intend to turn our backs on that and make everyone rewrite them all in React. This means that any rich client-side implementation of StreamField that creates and populates FieldBlocks on-the-fly will need to interoperate with widgets written in the classic Django way. react-streamfield handles this by making some reasonable assumptions about the client-side behaviour of form widgets - for example, there will be an HTML input element matching the widget's name, and writing to its '.value' property will update it.
Unfortunately, these assumptions break down for more complex widgets, such as Wagtail's page and image choosers. Since Django has never really cared about the front-end representation of form fields, beyond the fact that they contribute some sort of data to a form submission that we can extract later, widgets have a lot of leeway to render themselves however they see fit, and frequently make use of inline JavaScript code and multiple input elements. As a result, there's no standardised way for front-end code to manipulate and extract data from form fields. It's possible to work around this by adding special cases for individual form fields, or rework those form fields to fit StreamField's requirements, but going down that route presents maintainability issues. StreamField surely won't be the only feature of this level of complexity with these kinds of requirements - we wouldn't want to be in a situation a few years down the line where building a new form widget involves juggling the demands of half a dozen different Wagtail components.
We think it's time to tackle that missing piece of the Django ecosystem, and provide a standardised way for form widgets to make their front-end workings available to JavaScript code. That requires some way of passing configuration data from Python to JavaScript, and with that in mind, I've put together a library with the working title of Telepath - representing the notion of controlling things from a distance. The idea is that every widget instance on the Python side will have a JavaScript counterpart, implementing a handful of basic operations such as creating new copies of a form element, extracting a value from it, and writing a value back - the fundamental building blocks for implementing StreamField in React or any other client-side framework.
The applications of this go well beyond StreamField. Being able to retrieve data from forms without going through a conventional HTML form submission will allow us to submit that data over a JSON API - something we believe will be a key foundation of future developments such as auto-saving and collaborative editing. For now, though, we're keen to explore how the Telepath model can be extended further into the design of StreamField. One of StreamField's great strengths is the ability to compose complex blocks from StructBlocks, ListBlocks and StreamBlocks - since these complex blocks follow the same internal Python-side APIs as basic FieldBlocks, this process can be repeated to any level of nesting. With Telepath brought into the picture, we can bring this thinking to the client side - just as basic form fields will receive a JavaScript API, so will these container blocks, opening up possibilities for inserting and manipulating sub-blocks from user code. And since the API will be vanilla JavaScript, there's no issue of being locked into a particular client-side framework.
At the latest Wagtail Space conference, NYPR demonstrated an in-house StreamField extension allowing editors to split a rich text block at a paragraph break into multiple blocks - something that's currently only possible through some tricky intercepting of StreamField and Draftail internals. A JavaScript API will mean that StreamField becomes a full-fledged platform for innovations like these. Since StreamField's launch we've seen Wagtail site builders pushing its content editing potential far further than we ever imagined - now it's time to make that happen for the user experience too.