MoVE beyond MVC
Since our 1.0 release, Wagn has steadily evolved from an app to a platform – a unique one, in fact. We believe it may be the first instance of a new architecture that will bring real data organizing power (integration, customization, evolution) to many who've never before had access to it.
For now, we're calling the new architecture MoVE (Model - View - Event). What follows is a first, informal attempt to start a conversation about the new architecture and its potential.
the genesis of MoVE
Model-View-Controller, or "MVC", has deservedly gained great traction among software developers in recent years. As the theoretical underpinning for highly successful platforms like Ruby-on-Rails, MVC has provided a wonderful framework for creating web applications with conventional relational database models. But the limitations of relational database models are also the limitations of MVC. Most notably: relational databases and MVC systems tend to lead to socially awkward data that doesn't play well with others.
MVC tends to encourage the creation of more and more models. Whenever you have a new data structure to deal with it, the most natural MVC solution is to create a new model. Creating a new model, in turn, propels developers to create new controllers and new views. The practice is so thoroughly embraced, in fact, that Rails generators automate the entire process.
Creating new things is vital; the question is how much each new creation must start from scratch. In MVC, it's quite a lot. Each new structure knows nothing about the others beyond what the creator explicitly tells them. In Rails terms, when you generate another model, it has no associations with other models until you specify them. Rails has excellent tools for managing the vocabulary of each model, but that doesn't remove the fact that each model means a new vocabulary to manage. It takes a ton of work to integrate models with each other, much less with external software. This complexity proliferation problem permeates relational databases and software built upon them.
The more Wagn committed to the "everything is a card" principle, the more we struggled with this approach. We wanted new capabilities and new kinds of structures, but (a) we wanted them all to have all the power, simplicity, and addressability of cards, and (b) we wanted their creation and configuration to be the domain of the web user, not just the developer. But to try to accomplish these things within MVC meant overloading card models, card controllers, and card views. If we weren't proliferating models, MVC couldn't help us.
Increasingly, MVC didn't fit. We slowly began shedding Rails structures and replacing them with our own. We built a custom views system based on inheritable format classes. We began unifying all controller code under a single controller to support an extremely simple RESTful web API and moving almost all our controller logic into events. We moved away from Rails' ActiveRecord queries in favor of our native CQL queries andstarted using "set modules" in place of Rails models. After time, we realized this was not just a non-vanilla app; we were moving inexorably towards a new architecture with its own design values and principles.
Model
The central difference between MVC and MoVE: MVC adds structure by adding models; MoVE has just one model and adds structure by dividing it. If you like, you might interpret the "o" in the acronym as short for "one", as in Model (one) - View - Event. All other differences flow from there.
If you're familiar with Wagn, you may have encountered sets of cards. A set can be as general as "all cards", as specific as a single card, or somewhere in between. We're referring to sets when we say MoVE divides one model. A set is, in this context, a way of specifying which instances of the model are affected by a given configuration, or rule. In fact, you could certainly make the case that SVE is the better name for the architecture. Sets are different from models in some significant ways:
- Each card is a member of multiple sets and can be affected by configuration to any of them. A model instance is, well, just an instance of that model.
- Wagn begins by adding capabilities to all cards and then subdivides as necessary. MVC works the other way.
- New sets typically involve no new low-level data structure, nor any of the headaches of the resultant increase in complexity. An MVC model typically means a new relational database table.
Note: It's true that Rails has an ActiveModel abstraction that may be used with non-relational databases. But this is basically a tool to make non-relational databases act as if they were relational, which means adding all the same structural problems on top of a database model that doesn't inherently have them.
Both via the web and via code, Wagn creators add power to Wagn by extending sets of cards with rules. Wagneers create card-based rules by connecting settings to sets of cards. Developers create code-based rules by connecting views, events, and model methods to sets of cards.
In the early days of Wagn, we followed common MVC patterns and created new Rails models for cards, revisions, cardtypes, users, roles, roles_users, etc. To interact with all these models, we created custom controllers and views in standard MVC fashion. But as Wagn has evolved, we've seen tremendous benefit from getting rid of all that and moving functionality back into cards. In fact, we've deleted most of the old tables and are working to delete more.
Those familiar with Wagn code may justifiably that Wagn actually still has multiple rails models (cards, card_revisions, card_references, etc). This is true: the Wagn platform manages these structures, a Wagn app does not. This distinction matters. Wagn itself (the platform) isn't a MoVE app; it supports their creation. The same goes for Rails: the Rails platform is not an MVC app; it supports them.
The delineation between app and platform will grow much clearer when we package Wagn as a gem, at which point the core Wagn code will be tidily packaged away and only examined by folks wanting to dig deep. At that point Wagn site creator's experience will be in many ways more like a Rails site creator's experience is now. Rails developers install Rails and start working with MVC structures. Similarly, you'll install Wagn and start working with MoVE structures (though much more of this will be done via web interface). Generally speaking, Wagneers and Wagn module developers shouldn't have to give a thought to the underlying handling of revisions and references, which are really just low-level support structures for cards.
Views
Views in MoVE play the same role as views in MVC, but there are some key improvements:
- It's extremely easy to reuse the same view with different sets of cards
- Views are built with Format classes that follow a traditional inheritance structure, which creates incredibly valuable patterns for rendering data in different formats
- Views are not married to controllers, making them much easier to reuse in contexts other than web requests (eg emails, data export, etc)
- There is no clear line between "views" and "partials" (parts of views). Like many things in MoVE, views are fractal, and views can include views that include views; each view should stand on its own.
#1 above was the key impetus for shedding Rails views. We kept wanting to reuse, reuse, reuse on a smaller and smaller scale. This pattern is not impossible in Rails, but it requires cheating further and further away from their core structures with partials, helpers, and custom libraries.
Events
As the only letter that doesn't appear in MVC, the "E" certainly requires an explanation. As does the question of what happened to the controllers!
It's easiest to begin here with the extreme simplicity of a MoVE web API. There are four, and only four, actions that are direct analogs of MVC's controller actions (and, in fact, are still implemented that way in Wagn). These are:
- Create (HTTP POST)
- Read (HTTP GET)
- Update (HTTP PUT), and
- Delete (HTTP DELETE)
By Wagn 2.0, every Wagn request will be one of the above four actions on a card (there are currently exceptions being refactored to follow this principle). That's the entire basis of the web api; any other information needed is in CGI parameters. By design we will strongly discourage developers from adding any others. If you're not to add any actions, what need is there of controllers?
Any Rails developer is likely to respond that we need controllers for, well, all the logic that is typically found in MVC controllers. After all, that's where most of the action is.
In MoVE, the action is in "events". Events, like views, are rules, and as such they are triggered by actions – or other events – on a given set of cards. For example, you might set an event to happen "whenever someone creates a registration card" or "after someone comments on my profile." Validations, permissions checks, emails, data triggers, etc, are all examples of events.
Events actually harmonize two very different MVC patterns: controller actions and model callbacks. Many MVC developers consider callbacks "hidden" or "hacky", and indeed they often play controller-like roles despite being located in the model.
Some key improvements that events offer over controller actions:
- events can be as broadly or narrowly triggered as needed
- events can easily be shared among actions (eg, on create or update)
- events follow MoVE's adherence to fractal patterns. events can trigger events that trigger events.
- like views, events are not married to controllers and so are much easier to use with backend integrations.
Not badmouthing!
Just in case it's not clear: we love the Rails community. We never would have been able to create what we've created without them, and we're still highly dependent upon their code.
As the name suggests, MoVE owes much to MVC, just as Wagn owes much to Rails. We're currently using MoVE rather than SVE to name the architecture in part to make a more vigorous bow to its heritage. MoVE embraces a very similar separation of concerns to MVC, and indeed we consider MoVE an evolutionary step forward. But it's a very valuable step!
Todos for theorists
This post is intentionally (and necessarily?) a little casual about the dividing line between Wagn and MoVE. MoVE did not arise as a theory first; it's an abstraction from Wagn. It's not entirely clear yet whether, for example, Wagn's compound names, types, and structure patterns should be considered part of MoVE or Wagn idiosyncracies. To be sure, they evolved together to support each other in a coherent whole, but, with a typical Wagn nod to emergent structure, perhaps it's best to let the community of creators figure out together which of Wagn's parts are necessary for a MoVE hole.
This theoretical work is not just academic. MoVE can bring deep integration where it's never been before. It can help independent data managers collaborate as deeply and effectively as departments in a corporation, helping to empower small players to compete with the big guys. It can promote transparency by giving nuanced set-based permissioning control (so data managers can share all that they are willing and nothing else) and by ending the need for custom web APIs. It can change the way we work together and how we organize the data that increasingly shapes (not just reflects) the world around us.
Here we pose more questions than answers. Welcome to the conversation!
Thanks for this wonderful explanation of Wagn' architecture, it has been a great deal of fun working on it during this period of major refactoring. I find that each of Wagn' 'idiosyncracies' is essential to the whole pattern that makes MoVE possible. Sets are not possible without compound names and types. Maybe the formatting is its own independent dimension, but it is also so essential to the overall design.
Let's not forget a nod to Wiki and link/inclusion notations that are core to the content and reference models. Wagn makes wiki content into a platform, and that is quite something. MoVE brings it all together in a coherent pattern.
Re Sets, I *think* I agree. I mean, our current Set Patterns are almost all based on types and compound names, so Wagn sets completely depend upon them. But I was trying to suggest the possibility, at least, that in a different context sets might be based on something else, or at least some very different pattern of typing / naming. There's a lot of room in Set Theory beyond the tiny piece of it we've explored, and I think the notion of having hierarchies in set patterns (rather than in the sets themselves, which would be more like traditional MVC) is a pattern that may be more broadly explored while still largely fitting within MoVE.
Re wikis, I certainly agree that we should do more to acknowledge our debt and our heritage there. Even more than the notation (which is just superficial markup and may largely become invisible when we build link and inclusion editors), we have embraced the patterns of marrying the user / creator space, of managing broken/missing references, and of tracking / exposing histories.
Not sure I get all of this, but what I do? Great stuff. I'm also interested in the inquiry about different sets, especially because I have a strong hunch we will want to add more ways of defining sets over time.
Couple of questions about this:
'Events, like views, are rules, and as such they are triggered by actions – or other events – on a given set of cards. For example, you might set an event to happen "whenever someone creates a registration card" or "after someone comments on my profile."'
How are views (or events) rules?
What makes an action different from other kinds of events?
Re models, sets, and types & compound names, does all this mean that in a way you get a new set (=~ a new model) every time you add a type, or use a card as a right for the first time?
Are we okay to spread the word about this? There are a few people I'd like to invite to take a look at it.
How are views/events rules? This is not made clear on the rules card, but it was introduced as a concept in the wagn rules blog post a long while back. The notion is that a rule is a configuration applied to a Set. Card rules are "settings" applied to sets. Code rules are model methods, views, and events applied to a set. I will try to update the rules card to reflect this (without making the code piece too prominent; it's not the most approachable stuff for new wagneers).
What makes an action different from other kinds of events? It might help to consider an action a "root" event. Actions are tied to HTTP methods and are part of the web api. Other events are not. Via the web api, at least, it's always an action that gets the "chain of events" started. So you can trigger the "create" action via an HTTP POST, but you can't directly trigger, say, "pull_from_trash" (an event that gets called during create actions).
Re more ways of defining sets: no doubt. fwiw, the discussion with Gerry was less about whether we'd expand set patterns and more about whether there might someday be a MoVE app that did not follow Wagn's patterns of types and compound names, despite the fact that they're currently the basis of all our sets.
Re the "new model" question, the way I'm using the word, the answer is "no". There's only one model. It's true that every time you create a new card or even mention a new name you're conceptually opening up a new set space, but in practical terms Wagn is intentionally lazy about generating new set representations for any set that has no rules (code or card) attached to it. Concretely, when I create an "apple" card, that means there is now conceptually an "apple+*self" set (not to mention an "apple+*self+*self" set and so forth), but if nobody has done anything to deem that set special in some way, Wagn doesn't spend any time pondering its settiniess.
Re publicity: yes, it's published now. @wagneer tweeted about it at some point, I think. But I'd love for it to get more attention!
I'm returning here as this become potentially relevant to my day job where I work on an enormously complex Rails MVC CMS. I am painfully aware of just how much complexity came along because of exactly what you wrote in the introductory paragraphs above.
I think I wasn't clear in my first comment, it isn't that specifically Wagn structures are necessary, but that some regular structure is needed. In other words, you have to have a segmented name system, but it doesn't have to be SmartNames, and you probably want types, maybe an even more complex type system. You need both Sets and Patterns, but not the ones Wagn has chosen specifically. I agree completely with your comments that there is a lot to explore in "set theory". You hit the important point in the post, you need ways to divide the model space.
Also remember that we get the need to name each model object from wiki heritage, but we add a structured namespace by a somewhat serendipitous route. It started as a way to join tags, and became a way to group data (object and attributes, the later sets and settings). The hierarchical relationships of the namespace is essential to both how rules are named, and how contextual names work in content rules (*structure) to create analogs of MVC type records. The fact that objects all have names also maps nicely onto the RESTful api where we simply map the URI path onto a (one) model name, and map HTTP actions to model actions (CRUD) and everything just fits comfortably.
The situation is you already have a large and complex MVC application, and you want to make it MoVE. Where do you start? You're not going to be able to get rid of all the complex models, controllers and views all at once. Maybe first attach MoVE to models you already have. I think implementing namespaces will be important, so maybe I attach a model as a namespace: model_space/model_key/model_attribute_or_association to reference data in the existing models, and be able to overlay card based content on top of such a namespace.
A design question that comes up for me in thinking this through is about just how rule search will proceed. I suspect I'll want to have rules in the model_space namespace that hide rules in the root namespace, but I won't want to have to load all the rules necessary just to access and view core/standard patterns into each namespace.
As a general comment on the architecture, I'm getting a bit less comfortable with the name MoVE, because it actually doesn't map that neatly to how we organize things. There is one model (card), and events are really just special model methods. Any card can have many formats, and views are just special format methods. All of this is organized into sets. The acronym "MoVE" somehow seems to cut across that space in a weird way. What about MoFoS? Models and Formats organized into Sets ;)
It's definitely tough at this point to figure out where the line is between the general principles of the architecture and the specific implementation decisions of Wagn, given that Wagn's the only implementation. Is it possible there could be a MoVE application without segmented names? Or with multiple types? The reference handling is clearly central to Wagn. Is it central to MoVE? It's largely an academic argument; we definitely want to make it clear how important these things are to Wagn and explain our rationale for those choices. I'm thinking I'll generally keep the architectural framing a bit more open, but I suppose Wagn's design decisions will sort of implicitly be held as "best practice" until someone comes up with something better.
The use case you describe is definitely very valuable. We should open up a separate discussion about "moving to MoVE" somewhere, but I will offer these first thoughts:
1. a card can often be a thin wrapper around external data.
2. the "extend" phase is designed for connecting cards to the outside world (emails, apis, etc)
3. I agree that namespaces will be key here
4. If I had 20 rails models and I wanted to start moving them into cards, I'd start looking at them one at a time and seeing which ones I could pick off first.
Not having attempted this before, I'm not really clear on where the trickiest parts will be, but I would certainly guess that some patterns would emerge quickly...