Due out this Friday (March 26, 2015), Wagn 1.15 includes several major improvements.
The first and most obvious is support for Bootstrap. Whether you're creating a new "deck" of Wagn cards or updating an old one, you will now be able choose from one of 16 pre-installed professional Bootstrap themes. And, because Bootstrap has become quite popular, you can readily find thousands of more themes on the web (not all of them free, of course!) that you could apply to your Wagn site. Bootstrap is known for its "mobile-first" approach, so not only will these themes make your site look better, they'll help ensure your Wagn site is responsive to many different kinds of displays. Please be sure to pay close attention to Wagn 1 15+upgrade notes when you update, especially if you have heavily customized your site's layout/styling.
The second major improvement is to "following" cards. It is now possible to follow any Set of cards: all cards, all User cards, all +discussions, etc. Moreover, you can also restrict those sets to only cards that you have created and/or edited.
The third one is a bit more subtle, but it gets us closer to a key functionality that many have requested: the capacity to install Wagn as an extension to an existing Rails application. To the Rails geeks, this means "Wagn as an Engine". The major improvement in this direction is that we've rearchitected Wagn to make this possible: all the card handling is in a separate "card" gem, the core routing has been abstracted into a separate engine within the "wagn" gem (soon to be renamed to "decko"), and the support for adding engines to rails sites will be housed in a new "decko-rails" gem.
But the revision system had some key flaws. For example, Wagn didn't track name or type changes, and it didn't group together edits to multiple cards that took place as part of one submission. It was also very difficult to navigate changes for structured cards; you had to look through one field card at a time.
Wagn 1.14 fixes all of those issues and adds a much more compelling interface to boot. Now you can...
- review and revert changes to a card's name and/or type in addition to content.
- browse through lists of acts on a card, each of which might include actions on multiple cards. You can then expand actions to see more detail.
- load changes to content-heavy pages much more quickly.
- view all changes to included cards as part of the same revision list.
In parallel, we began an overhaul of the system for following cards (receiving notifications for changes). Now you can:
- see the actual content changed in the email (emails will include diffs in Wagn 1.16)
- view and edit the list of cards that you're following by editing your card name+*following.
- customize follower notification email
In fact, not just follower emails but all Wagn emails are now handled in cards and fully customizable, including:
- verification email, which are sent out to verify emails used for Sign up.
- signup alert email, which are sent to notify interested users of a new signup.
- password reset email, which are sent out when someone has lost a password and clicks "RESET PASSWORD" on the Sign in page.
A bit more on the geeky side, it's also worth noting that Wagn 1.14 marks a major milestone for Wagn in that it's the first release in which all actions can be accomplished via our RESTful Web API. That is to say that every server action you'll ever take on Wagn, from signing up to signing in to opening a card to following it to anything else will involve a simple CRUD (create, read, update, or delete) operation on a card.
no more activity on Wagn?
Wagn helps savvy web users do things previously only developers could do. With its new skinning system and key improvements to handling of inclusions, Wagn 1.12 takes this pattern even further. But it also empowers developers in vital new ways with its new events API and RESTful Web API.
All of these are bringing us much closer to Decko 1.0, which is focused on two key goals:
- discoverability and usability for readers, editors, and Wagneers
- extensibility for developers
The roadmap to 2.0 now includes just one more minor release (Wagn 1.14) and a final round of polish. Part of that polish will include adding new skins. (We're hoping you'll help with that part!) The new skinning system brings many improvements:
- dropdowns: new wagneers will simply be able to edit their *style rules and choose from a list of skins. (though, of course, we have to create those new skins first!)
- site areas: Different cards can have different style rules. So, for example, you can use different skinning for different cardtypes.
- stylesheet cards: Skin developers have a much improved interface that includes syntax highlighting, SCSS support, and stylesheets organized into cards for easy reuse.
- control: You also have complete control over the CSS on your site (whereas much of it was previously hard-coded). Some stylesheet cards do pull their CSS from code, but you can choose not to use those cards.
- speed: all style rules are automatically compiled, concatenated, compressed, and cached automatically upon update. When combined with the more efficient HTML, this should make page and editor rendering considerably faster.
That last improvement ("cached immediately upon update") is just one example of the "events" that are possible now with our new events API. Almost all of Wagn's core behaviors have now been refactored into events and make use of this new API. There is some early documentation of the API at present, but there is much more to be done there. In the meantime, those interested may find it most useful to explore the events defined in built-in modules.
Events are central to the MoVE architecture, which we've been working to articulate more clearly. The recent Blog post outlines the architecture in more detail, but the short of it is that we're exploring the space of handling data atoms in increasingly fractal ways. Cards can be built out of cards, views built out of views, and, now, events out of events.
The current release also makes inclusion handling more fractal. Old inclusion syntax let you configure display details for the included card. The new syntax lets you specify those details for the included card as well as cards included by the included card, cards included by cards included by the included card, etc. We've wandered into wonky land, clearly, but know that this simple improvement adds very little syntax yet adds great power for making pretty lists :)
If upgrading to Wagn 1.12, make sure you refer to the upgrade notes. We've done a lot of work to make this a smooth upgrade experience (including migrating the old "*css" card into a new custom skin for you), but the upgrade may take a tweak or two if you've done a lot of styling work on your Wagn.
Thank you, so much, for all the work you have done on this. Have any more organizations adopted Wagn?
It would be great if cldstr offered a directory somewhere of all of the Wagns (that are okay being known).
We don't have great stats on this, since github doesn't track clones / downloads. I would guess there are somewhere around 200 active wagn installations now.
We will attempt to bring Wagn to a much wider audience with the 2.0 release, which will bring significant discoverability improvements and a ready-for-primetime api.
Since our 1.0 release, Wagn has steadily evolved from an app to a platform – a unique one, in fact. We believe it may be the first instance of a new architecture that will bring real data organizing power (integration, customization, evolution) to many who've never before had access to it.
For now, we're calling the new architecture MoVE (Model - View - Event). What follows is a first, informal attempt to start a conversation about the new architecture and its potential.
the genesis of MoVE
Model-View-Controller, or "MVC", has deservedly gained great traction among software developers in recent years. As the theoretical underpinning for highly successful platforms like Ruby-on-Rails, MVC has provided a wonderful framework for creating web applications with conventional relational database models. But the limitations of relational database models are also the limitations of MVC. Most notably: relational databases and MVC systems tend to lead to socially awkward data that doesn't play well with others.
MVC tends to encourage the creation of more and more models. Whenever you have a new data structure to deal with it, the most natural MVC solution is to create a new model. Creating a new model, in turn, propels developers to create new controllers and new views. The practice is so thoroughly embraced, in fact, that Rails generators automate the entire process.
Creating new things is vital; the question is how much each new creation must start from scratch. In MVC, it's quite a lot. Each new structure knows nothing about the others beyond what the creator explicitly tells them. In Rails terms, when you generate another model, it has no associations with other models until you specify them. Rails has excellent tools for managing the vocabulary of each model, but that doesn't remove the fact that each model means a new vocabulary to manage. It takes a ton of work to integrate models with each other, much less with external software. This complexity proliferation problem permeates relational databases and software built upon them.
The more Wagn committed to the "everything is a card" principle, the more we struggled with this approach. We wanted new capabilities and new kinds of structures, but (a) we wanted them all to have all the power, simplicity, and addressability of cards, and (b) we wanted their creation and configuration to be the domain of the web user, not just the developer. But to try to accomplish these things within MVC meant overloading card models, card controllers, and card views. If we weren't proliferating models, MVC couldn't help us.
Increasingly, MVC didn't fit. We slowly began shedding Rails structures and replacing them with our own. We built a custom views system based on inheritable format classes. We began unifying all controller code under a single controller to support an extremely simple RESTful Web API and moving almost all our controller logic into events. We moved away from Rails' ActiveRecord queries in favor of our native WQL queries andstarted using "set modules" in place of Rails models. After time, we realized this was not just a non-vanilla app; we were moving inexorably towards a new architecture with its own design values and principles.
The central difference between MVC and MoVE: MVC adds structure by adding models; MoVE has just one model and adds structure by dividing it. If you like, you might interpret the "o" in the acronym as short for "one", as in Model (one) - View - Event. All other differences flow from there.
If you're familiar with Wagn, you may have encountered Set of cards. A set can be as general as "all cards", as specific as a single card, or somewhere in between. We're referring to sets when we say MoVE divides one model. A set is, in this context, a way of specifying which instances of the model are affected by a given configuration, or rule. In fact, you could certainly make the case that SVE is the better name for the architecture. Sets are different from models in some significant ways:
- Each card is a member of multiple sets and can be affected by configuration to any of them. A model instance is, well, just an instance of that model.
- Wagn begins by adding capabilities to all cards and then subdivides as necessary. MVC works the other way.
- New sets typically involve no new low-level data structure, nor any of the headaches of the resultant increase in complexity. An MVC model typically means a new relational database table.
Note: It's true that Rails has an ActiveModel abstraction that may be used with non-relational databases. But this is basically a tool to make non-relational databases act as if they were relational, which means adding all the same structural problems on top of a database model that doesn't inherently have them.
Both via the web and via code, Wagn creators add power to Wagn by extending sets of cards with rules. Wagneers create card-based rules by connecting Setting to Set of cards. Developers create code-based rules by connecting views, events, and model methods to sets of cards.
In the early days of Wagn, we followed common MVC patterns and created new Rails models for cards, revisions, cardtypes, users, roles, roles_users, etc. To interact with all these models, we created custom controllers and views in standard MVC fashion. But as Wagn has evolved, we've seen tremendous benefit from getting rid of all that and moving functionality back into cards. In fact, we've deleted most of the old tables and are working to delete more.
Those familiar with Wagn code may justifiably that Wagn actually still has multiple rails models (cards, card_revisions, card_references, etc). This is true: the Wagn platform manages these structures, a Wagn app does not. This distinction matters. Wagn itself (the platform) isn't a MoVE app; it supports their creation. The same goes for Rails: the Rails platform is not an MVC app; it supports them.
The delineation between app and platform will grow much clearer when we package Wagn as a gem, at which point the core Wagn code will be tidily packaged away and only examined by folks wanting to dig deep. At that point Wagn site creator's experience will be in many ways more like a Rails site creator's experience is now. Rails developers install Rails and start working with MVC structures. Similarly, you'll install Wagn and start working with MoVE structures (though much more of this will be done via web interface). Generally speaking, Wagneers and Wagn module developers shouldn't have to give a thought to the underlying handling of revisions and references, which are really just low-level support structures for cards.
Views in MoVE play the same role as views in MVC, but there are some key improvements:
- It's extremely easy to reuse the same view with different sets of cards
- Views are built with Format classes that follow a traditional inheritance structure, which creates incredibly valuable patterns for rendering data in different formats
- Views are not married to controllers, making them much easier to reuse in contexts other than web requests (eg emails, data export, etc)
- There is no clear line between "views" and "partials" (parts of views). Like many things in MoVE, views are fractal, and views can include views that include views; each view should stand on its own.
#1 above was the key impetus for shedding Rails views. We kept wanting to reuse, reuse, reuse on a smaller and smaller scale. This pattern is not impossible in Rails, but it requires cheating further and further away from their core structures with partials, helpers, and custom libraries.
As the only letter that doesn't appear in MVC, the "E" certainly requires an explanation. As does the question of what happened to the controllers!
It's easiest to begin here with the extreme simplicity of a MoVE web API. There are four, and only four, actions that are direct analogs of MVC's controller actions (and, in fact, are still implemented that way in Wagn). These are:
- Create (HTTP POST)
- Read (HTTP GET)
- Update (HTTP PUT), and
- Delete (HTTP DELETE)
By Wagn 2.0, every Wagn request will be one of the above four actions on a card (there are currently exceptions being refactored to follow this principle). That's the entire basis of the web api; any other information needed is in CGI parameters. By design we will strongly discourage developers from adding any others. If you're not to add any actions, what need is there of controllers?
Any Rails developer is likely to respond that we need controllers for, well, all the logic that is typically found in MVC controllers. After all, that's where most of the action is.
In MoVE, the action is in "events". Events, like views, are rules, and as such they are triggered by actions – or other events – on a given set of cards. For example, you might set an event to happen "whenever someone creates a registration card" or "after someone comments on my profile." Validations, permissions checks, emails, data triggers, etc, are all examples of events.
Events actually harmonize two very different MVC patterns: controller actions and model callbacks. Many MVC developers consider callbacks "hidden" or "hacky", and indeed they often play controller-like roles despite being located in the model.
Some key improvements that events offer over controller actions:
- events can be as broadly or narrowly triggered as needed
- events can easily be shared among actions (eg, on create or update)
- events follow MoVE's adherence to fractal patterns. events can trigger events that trigger events.
- like views, events are not married to controllers and so are much easier to use with backend integrations.
Just in case it's not clear: we love the Rails community. We never would have been able to create what we've created without them, and we're still highly dependent upon their code.
As the name suggests, MoVE owes much to MVC, just as Wagn owes much to Rails. We're currently using MoVE rather than SVE to name the architecture in part to make a more vigorous bow to its heritage. MoVE embraces a very similar separation of concerns to MVC, and indeed we consider MoVE an evolutionary step forward. But it's a very valuable step!
Todos for theorists
This post is intentionally (and necessarily?) a little casual about the dividing line between Wagn and MoVE. MoVE did not arise as a theory first; it's an abstraction from Wagn. It's not entirely clear yet whether, for example, Wagn's compound names, types, and formatting patterns should be considered part of MoVE or Wagn idiosyncracies. To be sure, they evolved together to support each other in a coherent whole, but, with a typical Wagn nod to emergent structure, perhaps it's best to let the community of creators figure out together which of Wagn's parts are necessary for a MoVE hole.
This theoretical work is not just academic. MoVE can bring deep integration where it's never been before. It can help independent data managers collaborate as deeply and effectively as departments in a corporation, helping to empower small players to compete with the big guys. It can promote transparency by giving nuanced set-based permissioning control (so data managers can share all that they are willing and nothing else) and by ending the need for custom web APIs. It can change the way we work together and how we organize the data that increasingly shapes (not just reflects) the world around us.
Here we pose more questions than answers. Welcome to the conversation!
Thanks for this wonderful explanation of Wagn' architecture, it has been a great deal of fun working on it during this period of major refactoring. I find that each of Wagn' 'idiosyncracies' is essential to the whole pattern that makes MoVE possible. Sets are not possible without compound names and types. Maybe the formatting is its own independent dimension, but it is also so essential to the overall design.
Let's not forget a nod to Wiki and link/inclusion notations that are core to the content and reference models. Wagn makes wiki content into a platform, and that is quite something. MoVE brings it all together in a coherent pattern.
Re Sets, I *think* I agree. I mean, our current Set Patterns are almost all based on types and compound names, so Wagn sets completely depend upon them. But I was trying to suggest the possibility, at least, that in a different context sets might be based on something else, or at least some very different pattern of typing / naming. There's a lot of room in Set Theory beyond the tiny piece of it we've explored, and I think the notion of having hierarchies in set patterns (rather than in the sets themselves, which would be more like traditional MVC) is a pattern that may be more broadly explored while still largely fitting within MoVE.
Re wikis, I certainly agree that we should do more to acknowledge our debt and our heritage there. Even more than the notation (which is just superficial markup and may largely become invisible when we build link and inclusion editors), we have embraced the patterns of marrying the user / creator space, of managing broken/missing references, and of tracking / exposing histories.
Not sure I get all of this, but what I do? Great stuff. I'm also interested in the inquiry about different sets, especially because I have a strong hunch we will want to add more ways of defining sets over time.
Couple of questions about this:
'Events, like views, are rules, and as such they are triggered by actions – or other events – on a given set of cards. For example, you might set an event to happen "whenever someone creates a registration card" or "after someone comments on my profile."'
How are views (or events) rules?
What makes an action different from other kinds of events?
Re models, sets, and types & compound names, does all this mean that in a way you get a new set (=~ a new model) every time you add a type, or use a card as a right for the first time?
Are we okay to spread the word about this? There are a few people I'd like to invite to take a look at it.
How are views/events rules? This is not made clear on the rules card, but it was introduced as a concept in the Wagn Rules blog post a long while back. The notion is that a rule is a configuration applied to a Set. Card rules are "settings" applied to sets. Code rules are model methods, views, and events applied to a set. I will try to update the rules card to reflect this (without making the code piece too prominent; it's not the most approachable stuff for new wagneers).
What makes an action different from other kinds of events? It might help to consider an action a "root" event. Actions are tied to HTTP methods and are part of the web api. Other events are not. Via the web api, at least, it's always an action that gets the "chain of events" started. So you can trigger the "create" action via an HTTP POST, but you can't directly trigger, say, "pull_from_trash" (an event that gets called during create actions).
Re more ways of defining sets: no doubt. fwiw, the discussion with Gerry was less about whether we'd expand set patterns and more about whether there might someday be a MoVE app that did not follow Wagn's patterns of types and compound names, despite the fact that they're currently the basis of all our sets.
Re the "new model" question, the way I'm using the word, the answer is "no". There's only one model. It's true that every time you create a new card or even mention a new name you're conceptually opening up a new set space, but in practical terms Wagn is intentionally lazy about generating new set representations for any set that has no rules (code or card) attached to it. Concretely, when I create an "apple" card, that means there is now conceptually an "apple+*self" set (not to mention an "apple+*self+*self" set and so forth), but if nobody has done anything to deem that set special in some way, Wagn doesn't spend any time pondering its settiniess.
Re publicity: yes, it's published now. @wagneer tweeted about it at some point, I think. But I'd love for it to get more attention!
I'm returning here as this become potentially relevant to my day job where I work on an enormously complex Rails MVC CMS. I am painfully aware of just how much complexity came along because of exactly what you wrote in the introductory paragraphs above.
I think I wasn't clear in my first comment, it isn't that specifically Wagn structures are necessary, but that some regular structure is needed. In other words, you have to have a segmented name system, but it doesn't have to be SmartNames, and you probably want types, maybe an even more complex type system. You need both Sets and Patterns, but not the ones Wagn has chosen specifically. I agree completely with your comments that there is a lot to explore in "set theory". You hit the important point in the post, you need ways to divide the model space.
Also remember that we get the need to name each model object from wiki heritage, but we add a structured namespace by a somewhat serendipitous route. It started as a way to join tags, and became a way to group data (object and attributes, the later sets and settings). The hierarchical relationships of the namespace is essential to both how rules are named, and how contextual names work in content rules (*structure) to create analogs of MVC type records. The fact that objects all have names also maps nicely onto the RESTful api where we simply map the URI path onto a (one) model name, and map HTTP actions to model actions (CRUD) and everything just fits comfortably.
The situation is you already have a large and complex MVC application, and you want to make it MoVE. Where do you start? You're not going to be able to get rid of all the complex models, controllers and views all at once. Maybe first attach MoVE to models you already have. I think implementing namespaces will be important, so maybe I attach a model as a namespace: model_space/model_key/model_attribute_or_association to reference data in the existing models, and be able to overlay card based content on top of such a namespace.
A design question that comes up for me in thinking this through is about just how rule search will proceed. I suspect I'll want to have rules in the model_space namespace that hide rules in the root namespace, but I won't want to have to load all the rules necessary just to access and view core/standard patterns into each namespace.
As a general comment on the architecture, I'm getting a bit less comfortable with the name MoVE, because it actually doesn't map that neatly to how we organize things. There is one model (card), and events are really just special model methods. Any card can have many formats, and views are just special format methods. All of this is organized into sets. The acronym "MoVE" somehow seems to cut across that space in a weird way. What about MoFoS? Models and Formats organized into Sets ;)
It's definitely tough at this point to figure out where the line is between the general principles of the architecture and the specific implementation decisions of Wagn, given that Wagn's the only implementation. Is it possible there could be a MoVE application without segmented names? Or with multiple types? The reference handling is clearly central to Wagn. Is it central to MoVE? It's largely an academic argument; we definitely want to make it clear how important these things are to Wagn and explain our rationale for those choices. I'm thinking I'll generally keep the architectural framing a bit more open, but I suppose Wagn's design decisions will sort of implicitly be held as "best practice" until someone comes up with something better.
The use case you describe is definitely very valuable. We should open up a separate discussion about "moving to MoVE" somewhere, but I will offer these first thoughts:
1. a card can often be a thin wrapper around external data.
2. the "extend" phase is designed for connecting cards to the outside world (emails, apis, etc)
3. I agree that namespaces will be key here
4. If I had 20 rails models and I wanted to start moving them into cards, I'd start looking at them one at a time and seeing which ones I could pick off first.
Not having attempted this before, I'm not really clear on where the trickiest parts will be, but I would certainly guess that some patterns would emerge quickly...
Even today, most web content can fairly be called either bloggy or wikish.
Tweets, Facebook posts, Reddit news items, discussion forums, and (gasp) even traditional news sites qualify as bloggy: Each piece is timely and static; there is one clear date of publication, and you only edit to correct egregious errors.
On the opposite end of the data-shelf-life spectrum is a timeless (no single publication date), dynamic wiki page. Obvious wikish sites include pretty much anything whose name contains "iki", "pedia", or both. Less obvious: the vast majority of company websites are in this camp, whether they use wiki software or not. The content is intended to be authoritative and evolving, part of the great pedia that is the web.
Savvy web users know these two worlds are generally kept separate. Software websites refer folks to "their wiki" for documentation. Corporate sites refer users to "their blog" for the latest news. They may be under the domain, but the blog and wiki are clearly different neighborhoods. Is this segregation wise?
There are several reasons for the split, some old, some new, some complex, some confused. Some folks think "wiki" always means "anyone can edit". Some think blogs are just for opinion pieces. Many more fall into segregation by habitually approaching their web challenges piecemeal: we need a blog; what's the best tool for blogs?
More thoughtful web creators will point out real challenges to blending the two. To mix individual and collective creations, you need elegant management. To mix one-off announcements and thoughtful edits you need elegant interface. To organize timely and timeless data, you need elegant organization.
All true, but suppose we meet those challenges. What would the payoff be?
Well for starters, a unified site means a single signin account; separate signins are a nuisance, a barrier to entry, and a source of data confusion. It also means a single search, so you don't have to look multiple places any time you can't remember where a bit of information was entered. It means a better user experience for visitors, who don't feel like they have to learn their way around multiple realms. It means simpler, easier-to-manage references from blog to wiki and vice versa. It means simple data reuse – duplication is the bane of data managers everywhere.
The payoff, in sum, is integration. Better integration means better visitor experience, better creator/maintainer experience, and, ultimately, better data.
Ethan McCutchen is a creator of Wagn, a tool for team-driven websites. Learn more at Wagn.org
bloggy or wikish
Wagn 1.10 introduces some significant interface changes.
Most notable is the new menu, which opens as a dropdown from a gear icon in the upper right corner. This new menu reduces visual noise considerably and replaces the functionality of the old header, the old footer, and several submenus. It also helps separate out the "wagneer" functionality (under "advanced") in order to keep things simpler for casual users.
Also cleaned up: forms. By getting rid of details other than the fields being edited, we were able to make forms far less noisy. Name and type fields now align vertically with content fields, and old edit submenus are obviated.
As part of the cleanup, CSS is considerably simplified, and behavior on mobile devices is incrementally improved. Both will receive considerably more attention in upcoming releases as we roll out new support for skins and mobile-friendliness.
If you've been following Wagn, you know how excited we are about Wagn 2.0: how it will complete our transition from a nifty app to a powerful platform... how it will be easy to discover, use, and extend...
That said,Wagn 1.9 takes some big steps.
One key step is that we've considerably increased the number of cards that can be renamed without creating problems by giving them an additional "codename" identifier to be used when referenced in the code. This is important for Wagn because we give so much power to Wagneers, who are essentially coding by manipulating card content. Our new, more robust approach allows for powerful relationships between code and content, so that we can continue to give Wagneers more and more new capacities.
1.9 is also slimmer. By moving data into cards, we've gone from having seven core data tables to having just four. (We'll be down to three in 1.10). This has considerably simplified our codebase without sacrificing any functionality.
Perhaps most importantly, it gets us much closer to the 2.0 apis (both REST and pack) that we think will make Wagn the platform of the future.
See the Release card for tickets and update details.
Warning: this is going to get geeky. It's an exploration of current and future server optimization for hosts supporting multiple Wagn websites. Noone who does simple wagn installation should have to worry about such things.
The new pack/modules system is pushing us to reconsider our hosting architecture. Without packs it was relatively easy for multiple Wagn sites to share rails processes. On our old servers, all sites shared all the same code but used different databases (or, technically, different PostgreSQL schemas). With packs, each site can use different chunks of code, so the old way won't work anymore. Moreover, the new Cloudstore deployment framework, which gives us dramatically simplified installation, maintenance, migration, and scaling, has pushed us to have greater independence of each site's data and management.
In this article I'm going to describe a bit of what we've done so far in our own hosting set up and invite the Wagn community into a discussion of where we should go. If you're reading this because you want to speed up your own hosting setup, be sure to see the Wagn in production card, which outlines some concrete steps you can take to make your Wagn site run faster.
This discussion will center around a conventional Apache2/Passenger stack. There are many choices for hosting rails sites, and to be clear, we did not come to our current arrangement by testing them all. In fact, it is quite likely that there are some potential performance gains to be had by using the likes of nginx or thin. Our rationale for using Apache2/Passenger at this point is that:
- Apache2 is currently the webserver supported by Cloudstore, which, as mentioned, buys a tons for the Wagn community.
- Apache2/Passenger is a dominant rails hosting setup that we need to support strongly
- The performance challenges within Wagn are greater than those in the server setup, so it makes sense to prioritize server simplicity until those challenges are addressed.
A little background for those who see the word "spawning" and think of salmon.
When someone loads a webpage, the request is handled by a "process" on a webserver that is sometimes called a "listener" or a "handler". When there aren't enough such processes to handle the requests coming in, the server "spawns" a new one. Spawning can take a long time when starting from scratch. It entails loading up all of Ruby on Rails, all of Wagn, and all the packs. On our current production server, for example, it takes about 5 seconds to load our full environment, and that's before the server has starting to consider the specifics of the request. Obviously, we don't want folks to wait over 5 seconds for webpages.
The other option is, of course, not loading from scratch. It is much preferable that your spawning be largely "pre-loaded". Doing this well is key to optimizing our site performance.
Our current setup is very inefficient in this regard whenever a site has not been visited in a while. Basically, extra processes for any given site can spawned very quickly after the first one is spawned, but processes for separate Wagn sites make no use of the fact that they share tons of code with other Wagn sites. So Wagn site A doesn't make any use of the loading of Wagn site B.
This is happening because Passenger's default spawn method loads an entire application as one beast. When a request comes for a website with no current processes, it reloads Rails, Wagn, and the site's packs, even if Rails and Wagn are currently in use for several other sites. Since each website can have different packs and different environments, they are treated as if they weren't related at all.
We're currently trying to use Passenger's "smart" spawning, which will mean the Rails framework can be pre-spawned for all the Wagn sites. Rails has a lot of code – much more than Wagn – so this should be a big gain. With luck, we'll have this working within the week.
In the long term we could do even better and make sure that everything but the packs themselves are pre-spawned. This would mean some rearchitecting of our code load order to make sure packs and site-specific data are loaded entirely separately from the core. And it may well mean developing an Apache module (mod_wagn) to handle the nuances.
If you're using a webserver besides Apache, please note that this could be an area that needs some work. That's because Wagn uses an apache module called "xsendfile" to give the file-handling responsibilities back to Apache after it has checked permissions. This frees up Wagn to move on to the next request. It should be possible to do something analogous in other webservers, but this is not a case that we've developed for. If you are willing to give this some attention, we will try to support you in the endeavor!
Once we've plucked all this low-hanging fruit, we'll likely be back to optimizing Wagn's internals. It's clear that there are many opportunities in there, but we'll need to do more profiling/benchmarking before it's obvious where to pluck next. Here are some hunches.
Our nested permissions-checking has tended to thwart simple page caching, so to date we have focused on honing our card cache. As we grow, we will want to dig deeper and find more and more opportunities for speed. For example, when a page nests no restricted cards, we should be able to cache the whole thing. The challenge moving forward is to figure out how all that should work! For example, Gerry Gleason has proposed that Wagn inclusions might be handled as nested Rack requests.
As our Rules system matures, there will be caching opportunities there, and I suspect the views system will only offer more. Trickier, potentially, is WQL caching, but that may grow more important as more and more sites feature more and more dynamically queried content.
Finally, some of this may be pushed to the Wagneering level. One could imagine Snapshot cards that allow wagneers to take manual snapshots of a result and save it as card content, for example.
Wagn 1.0 was a pioneering web app. Wagn 2.0 will be a pioneering web app and web platform.
The current release (Wagn 1.8) gets us considerably closer. To name a few highlights:
- uses Rails 3 - the latest and greatest.
- supports Ruby 1.9 (1.8.7 still works)
- new file and image handling is more secure, flexible, and re-usable
- new settings editor is much crisper and easier to understand
- Almost 50 tickets resolved: new features, bug fixes, performance enhancements, etc.
This release includes more resolved tickets and more commits than any prior Wagn release, and yet the codebase is actually smaller than ever.
With every new release, Wagn adds not just more power, but also more harmony. Wagn 1.7 brings new power and harmony to our permissions system.
- Easier management. Our old permissions table stored separate permissions settings for every single card. Since our rules system allows for much more elegant expression of broad patterns, it is now far easier to see and manage your permissions. As an example, imagine you have 1000 user cards that can only be seen by "anyone signed in", and you want to change them so they can be seen by "anyone". In the old system, this would require going to the options tab of 1000 cards. Now this just requires one edit.
- Better interface. Because permissions are now rules, all the new improvements to our rules interface will make it easier to create and edit permissions configurations.
- More flexible assignments. Before, permissions could only be assigned to Roles, or groups of users. Permissions can now be assigned to any combination of Roles and Users that you might want.
- Searching. Since permissions are now in cards, you can now access them through the Wagn Query Language (WQL).
- Simpler API. Now you can create and edit permissions rules through the standard card API.
- Customizability. Because permissions are now managed through cards, you can tweak their editors and other views with our packs API.
As of Wagn 1.6, developers can extend Wagn without forking its code by creating a "pack".
"Packs" are ways to extend Wagn's functionality. In the long term, we envision that a "pack" may refer to several things:
- "modularized" code that can easily be added or subtracted
- a "tradeable" group of configuration cards
- a combination of the above
We'd like people to be able to use Wagn in languages other than English. Some of the text you see in Wagn's interface is in the code, and we are beginning to document that so that we can move cue text to editable cards. But a surprising amount of Wagn's interface is already in cards. New Wagns currently draw all of their default cards from a Wagn called "en" (for English), and it will be fairly straightforward to add an option for new Wagns to draw their default cards from a different Wagn.
So the idea is to set up a Wagn for each new language, with a list of translatable cards, and instructions on what exactly needs translating. That list will look something like this:
New Wagns now come with close to 300 cards by default, and this number is only going to go up as Wagn's power increases and functionality is moved from code into the cards themselves. We want it to be easy for Wagneer to get a sense of what's built in to new Wagns, and what all of these cards do. We're also preparing the way for translating Wagn into other languages (more on that in a later blog entry). So, I've written up a list of all of the cards that come with new Wagns, briefly explaining them and linking to deeper documentation:
There's space at the end of that card for any feedback you'd like to give on how you find it useful, or any suggestions you have for improvements (you can also just edit the card directly of course — I'll be watching it).
The headline isn't bragging; we have a new Wagn term: "Rules".
Even if you're just getting started with Wagn, you probably have some understanding of "cards". You understand that by "everything is a card", we mean that almost every unique chunk of information on Wagn has a common structure, whether it's text, an image, a user, a search, a setting, a number, etc.
If you've gotten serious about exploring Wagn, you also know that Set are groups of cards you can configure with Setting. The idea is that you can apply Settings like layouts, templates, captchas, help text, etc, very broadly or very narrowly, depending on which Set you apply them to.
When you configure a specific Set with a specific Setting, you get a "Rule." So, for example, if you want to change the permission for who can view user cards, you might go through the Options tab for a user card and change the *read permission for "All User cards". In doing so, you will be prompted to created a rule card named "User+*type+*read". Note: this pattern isn't new, but the term "Rule" is. You don't have to remember long card names to use rules, but if you take the time to understand one or two, you'll have a deeper sense of how Wagn works.
Like all compound names, our new rule card -- "User+*type+*read" -- has a left side and a right side, with the divider being the rightmost "+" sign. So the left in this case is "User+*type", and the right is "*read". Like all rules, the left side is a Set, and the right side is a Setting. In short: Set + Setting = Rule. Specifically, this rule applies the "*read" setting (which determines who can read a card) to the "User+*type" set (all user cards).
You may have noticed that many cardtypes or individual cards have some sort of special appearance or behavior that is not fully configured in cards. Image cards have special editors, Account Requests have special links to approve/deny, the recent changes card has special date groupings, the "*navbox" card shows the search bar, etc. All of those special configurations are accomplished through code rules that extend the relevant set of cards.
What's more - if you've got a little coding skill, you can develop new code rules yourself using Wagn's new Pack API. Even if not, you can now hire someone to extend Wagn to meet your personalized needs quickly and affordably. And as Wagn progresses, it will get easier and easier to build and share these "packs" of cards. If you're interested in tracking the progress of the new API, check out (and "watch") Packs+api.
Development toward Wagn 2.0 has centered around "modules" -- easy ways for coders to extend Wagn's functionality without having to learn their way around all of Wagn's codebase.
Now we're putting out an API "proposal" to our community. If you're someone who might someday be interested in writing a Ruby module here or there, have a look and let us know what you think.
Blessed vs. PossibleThe initial API release will be somewhat limited in scope, because many aspects of Wagn's architecture are...
Yay! Gratitude for your persistence!
--Patrick Gibbs.....Fri Nov 05 14:08:03 -0700 2010
Wagn 1.4 is here, and it fleshes out a very cool concept: cross-patterned content.
We all see a lot of patterned content every day: calendars, weather forecasts, stock reports, etc. But usually these patterns don't cross-pollinate. In Wagn, they can.
If you've been using Wagn for a while, you can understand the updates in 1.4 as mating our Setting / Set system with our formatting system. In short, Wagn lets users organize information into cards and apply Setting (configuration options) to Set (groups) of cards. The sets may be as broad as all cards or as narrow as a single card. We're now introducing two new formattings settings: *default and *content, which respectively determine cards' initial content (formerly "soft" format) and content template (formerly "hard" format).
The key thing to understand is that in Wagn everything is organized into cards, and every card's content can now be arranged with patterns that may be very broad or very narrow. And if the card is a complex whole built of other card parts, this same breadth of pattern can apply to every part. It's a very powerful concept.
Example: Wagn vs. relational database
To see how this all works and is different from a normal "relational" database, consider a system tracking information about pieces of furniture. Imagine that you want fields for measurements, type (chair, sofa, table), number of pieces, materials, comments, etc. The typical relational database model would be to make a furniture table with all of these fields. If you happened to have another table (say "artwork") with lots of the same fields (measurements, number of pieces, materials, comments, etc.), then those fields would not actually be related in any way to the furniture fields — separate table, separate universe.
Now let's consider how this would go in Wagn. Similarly, you might create a card type called "Furniture". Then you would include cards for those same fields in the furniture form card. (Check out all the new ways to navigate to form cards.)
But in Wagn, the configuration of each of those fields (or subcards) could be as general or as specific as you like. You could set it up so a subcard named "number of pieces" would be a number card regardless of the type of card it's on. You could have different default values for measurements depending upon what you're measuring (furniture or artwork). You could override any of this formatting for a specific field on a specific card. And you could even override all of this formatting for an exceptional piece of furniture.
This is why I referred to Wagn's power as cross-patterned content. All different parts of the content you're organizing can follow patterns — or diverge from them — as appropriate.
And it's just one more advantage of our approach of building everything out of powerfully simple building blocks. Cards make all kinds of new integrations possible.
More in Wagn 1.4
Of course, there's lots more in Wagn 1.4. Wagn 1.0 was about flexible ways of organizing information: a cutting-edge wiki. Wagn 2.0 will be a full-fledged whole-system-building platform: true to it's wiki roots, but more powerful, modular, and intuitive. Wagn 1.4 is a big step in that direction.
Additional enhancements include:
- a flexible email system for configuring emails to be sent when you add a card
- briefer and more powerful syntax for referring to card parts (contextual names)
- a "raw" views for displaying unprocessed card contents — show me the markup!
- an "array" views for using Ruby cards to perform calculations on search results
- performance optimizations for faster page loads
- fixes to some old, hated bugs. Ever double-click in a comments box and lose the comment? Never again!