ideas@sullice.com

Why is decoupled Drupal so hard?

Building decoupled applications is… hard.

Why?

That question gnawed at me. It kept me awake, staring at my ceiling. I’ve spent years working on Drupal to support decoupled use cases, but my own early experiences and the stories I hear at conferences and from colleagues all indicate that building a decoupled application involves many trade-offs and complications. How can I reconcile those things? Have I been wrong about decoupled Drupal? Am I chasing a mirage?

I’ve learned that it’s all too easy for decoupled projects to quickly became bloated with opaque state management, tedious request management, unweildy prop drilling, confused access logic, and more. Validated through DrupalCon presentations, Decoupled Dev Days talks, Slack conversations, and chatter around the now proverbial watercooler, my experience isn’t unique. Countless decoupled projects are mired in technical debt. Why?

The problem: mishandled mechanics.

What are mechanics? A project’s mechanics are its access controls, its menu hierarchy and depth, its authentication mechanism, its state management, its routing, etc.

I don’t hear it quite as often as I did a few years ago, but it is often said that “decoupling is the separation of content and presentation”. You may have heard a similar sentiment stated as: “just give me the data and let me do the rest.” Usually, “the rest” is a stand-in for presentation information that Drupal shouldn’t care about. Rarely does “the rest” encompass the full scope of work being hand-waved away.

These refrains reinforce the notion that, to build the platonic ideal of a decoupled application, developers must architect their projects so that the back end has no control over a application’s presentation (layout, color, font size, button placement, etc.) and the front end has no content (pages, menu titles, button text, etc.).

That’s true. But it’s not the whole truth. That pithy refrain: “decoupling is the separation of content and presentation” says nothing about mechanics. All projects have content, presentation, and mechanics.

I’ll wager that 8 times out of 10, a decoupled project gone wrong has mishandled mechanics. It will have a front end bloated by custom code and dependencies all working around the fact that mechanics don’t belong there. Mechanics should be driven by the back end.

It’s not the front end developer’s fault.

The hard path is the easy path.

The hard path is the easy path?

Yeah. Good tools make hard work easy and bad tools make hard work harder. I once was trying to loosen a bolt. It was too tight to take off with my fingers. Looking for some leverage, I looked in my toolbag and found pair of pliers. I ended up with a rounded bolt that I had to cut off. What I needed was a wrench… but my wrench was in the garage. The hard path was the easy path.

Drupal makes it very difficult to architect a decoupled project driven by the back end. To do so, you need to rebuild or heavily customize the back-end API. That’s because Drupal’s JSON:API handles only two mechanics: access and pagination. That is, it omits inaccessible content and provides next and prev collection links.

If you want Drupal to control routing in a decoupled context, it can’t. Authentication state? Nope. Contextual links? Sorry! Forms? stares

This functional vacuum begs to be filled. It’s a natural next step to refill it with front end code. After all, the front end is already replacing the back end presentation layer. It even seems like that’s the way you’re supposed to do it. react-router handles routing and it’s very popular. redux and vuex can handle state and they’re widely used. It’s JavaScript after all! There’s an ocean of libraries claiming to solve your problems with just one more import.

Some front end developers don’t use libraries. They have a toolbag. One toolbag is named Ember, another Gatsby—Next, Nuxt, etc. And Drupal itself lures developers down the hard path with bolt-rounding “features”.

Perhaps this will be a familiar conundrum: A front end developer is looking at a wireframe with a Log in button. The developer looks but doesn’t find a wireframe with a Log out button but—surely—that’s implied… “Wait!” The developer’s mind pauses. “How will the front end ‘know’ which button state to display?”

Developer Googles: “how to get authentication state Drupal 9”

Enter stage left: the first spaghetti noodle.

What follows might lead to a custom fetch request to the /jsonapi endpoint. Apparently, that response contains JSON and, in it, a link under /meta/links/me. If that link is present then the user is authenticated; if it is not then the user is anonymous.¹ Great! Right?

Unfortunately, now the front end route must send and wait for an extra HTTP request. And the front end must poll that same endpoint in case the user logged in or out in another browser tab (you forgot about that, didn’t ya?). Moreover, since that fetch isn’t synchronized with the primary request, the front end must either hide the Log in link until it receives the second response or deal with the chance that button flashes Log in before updating to Log out (yuck!).

Perhaps the developer should look to NPM for a helpful library?

Developer Googles: “how to handle asynchronous requests React”

Enter stage right: the second spaghetti noodle.

These little conundrums keep compounding and the plate of code spaghetti keeps growing.

Yet, the back end always knows whether every request is authenticated or not. Why doesn’t Drupal put this information on every response so that front end developers don’t need the extra request?

Mechanics belong on the back end.

Here’s another short story: A front end developer built a decoupled web app with a filtered content listing. In a server-rendered Drupal application, the page would have been a View. However, this application is using Drupal’s JSON:API. Apparently, “Collections are JSON:APIs API-First replacement for … Views.”² and the way the content listing was built uses a custom-built filter query parameter.³

A few months after the application launch, the developer’s client calls and requests that certain content be filtered out. The developer diligently added the new filter parameter, produced a new browser bundle, and deployed that updated JavaScript app file to production. Fast-forward to a phone call with the client. Their customers are still seeing the extra content; hasn’t the feature been deployed?

Developer Googles: “how to invalidate cached JS file in the browser”

Enter from above: a spaghetti-code meatball.

Why doesn’t Drupal provide a static URL for the content listing whose filters are managed on the server so the front end isn’t entangled by this business logic in the first place?

Mechanics belong on the back end.

What’s the alternative? Progressive decoupling? For now, yes. Forever? I don’t think so.

The promise of rendering HTML with JavaScript, a language designed for manipulating the DOM and native to the web browser, is too great. JavaScript tooling comes with no shortage of online help and almost every new web developer begins with HTML, CSS, and JS. Learning a server language like PHP or Python to do your templating is no longer necessary. This means that even if you’re building a completely static site, it’s easier to code templates with JavaScript rendering frameworks than it is with any other templating engine.

When the day that you must add a moderately complex interactive flair to a webpage eventually arrives, modern JS rendering frameworks can do that with ease. PHP cannot.

The future of HTML is rendered by JavaScript.⁴ For Drupal, this means decoupling is imperative.⁵

If decoupling is imperative and decoupled applications are hard? How do we make decoupled Drupal easy? And if not entirely easy from start to finish, how can we at least create a shallow ramp that invites developers unfamiliar with the decoupled stack to produce valuable work at each step of their ascent?

Drupal has a real opportunity to become the best back end for JavaScript rendered applications. Not only does it have rich and extendable data-modelling capabilities, it already has all the underlying mechanics of a web application figured out. Whether it’s file uploads, session management, editorial workflows, or emails, Drupal has the foundations. However, these application mechanics are not easy to integrate into a decoupled architecture today.

What could be changed?

Here are some examples:

1. Instead of placing a magical link on the /jsonapi endpoint, it could provide an authenticate link on every anonymous response and a logout link on every authenticated one. This would eliminate the required, secondary and asynchronous request.
2. Menu links could be retrievable over JSON:API and conditionally attached to every response in a way similar to how a menu block is conditionally placed via Drupal’s block layout screen.
3. Canonical entity URLs could return JSON:API responses so that front ends do not have to map URL aliases to JSON:API URL equivalents.
4. Drupal could expose Views (or a simpler Views alternative) over JSON:API so that collection URLs don’t need to be hardcoded or compiled on the front end.⁶

These ideas need to be refined and implemented, of course, but I believe they would get Drupal to the point where 80% of its use cases are handled out-of-the-box, even in a fully decoupled architecture. I also believe that these features would make it easier for existing non-Drupal developers to build a decoupled application with Drupal than it would be to use any other headless CMS.

The intent of each proposal is to reduce the complexity of front end applications by moving mechanical complexity to the back end. Taken together, they permit front end applications to consume individual HTTP responses that each contain the information needed to render a single page/route/screen in its entirety. This means that front ends, and front end developers, will be able focus on presentation and interaction without having to worry about the additional complexity that comes with a decoupled application today.

Practically, the lifecycle a conventional React or Vue application might look something like this:

1. The JS application is loaded and mounted to a root DOM element.
2. The JS application inspects the browser URL and requests that same URL from the back end with an Accept header containing the JSON:API media type.
3. Drupal will take that request and return the appropriate JSON:API response.
4. That response data can be directly passed as a prop into the top-level component of the JS application.
5. Elements of that response data will be passed as props to subcomponents (or provided via React Hooks or Vue’s Composition API).
6. Subcomponents like the menu will be able read this response data to get the menu tree. Since the menu tree was attached to the primary data, the menu tree data can include an active boolean to indicate the active menu item⁷.
7. Another subcomponent rendering authentication buttons can also observe the response data and conditionally render a Log in/out button. Since every response will include this data, there will be no need to poll the /jsonapi endpoint or handle a secondary request.
8. Finally, when a user follows a link, all that needs to be done on the front end is to push the href of the link into the browser’s History API and to request a new primary JSON:API document. Since Drupal can respond to aliased URLs and return JSON:API responses, a client side router to handle the navigation isn’t required. Instead of using URL path patterns to choose components, the JS application can observe the type of the response data and react⁸ to it by rendering a different component, if necessary.
9. If the user bookmarks or shares the new URL, nothing needs to be done, even without a client-side router. A browser request for that URL will return the index.html which loads and mounts the JS application and the lifecycle returns to step #2.

As you can see, the role of the front end in this example is only to parse and beautifully render JSON-encoded data–not to reimplement application mechanics that were once exclusively back-end concerns.

This reduced complexity would not only be a welcome simplification for new developers, but also developers maintaining multiple front ends for different devices or use cases. Three simple clients are easier to maintain than three complicated ones, even if they share common functionality.

I think we need a new refrain: Decoupling is the separation of content and mechanics, from presentation. Drupal already handles the content, now it needs to handle the mechanics.

My friends Peter Weber and Ben Mullins were incredibly helpful with this post. Thank you!