Packs+solution
Our architecture needs to focus on three primary Wagn "participant types". Open to see a list of what they need from the architecture.
- simplicity in finding and evaluating/screening modules
- simple installation, generally without interacting directly with the filesystem.
- simple removal of modules
- automated handling of dependencies
- good starting points -- bare bones not too bare
- manageable amounts of modules to install
- access to good help if/when any of the above comes up short
- Create simplified deployment packages for "non-admin" installs
- Create complex deployment architectures for specialized application
- Manage change in each class of deployment
- Manage change from networks of developers (internal and external)
- Create releases, Bagns
- Create distributed architectures to support development, testing and collaboration across multiple wagn spaces.
- Add support for other information sources important to my organization
- Manage Wagn resources (storage, memory/cpu, network)
- small chunks, easy to get your head around
- simple, clear api
- possibility to add value without tons of work
- access to feedback
- minimal barriers to entry
Clearly there is potential for conflicting needs. Developers want lots of "small parts", Wagneers want a few, intuitive big ones. The architecture below is aimed at addressing both:
Core
The core is basically what you have when you have no packs. Gems are modular in that rails apps can choose to use them or not. Packs are modular in that wagns can choose to use them or not. Note that with no packs there are no views, so although you could do basic data operations with the Wagn core, you couldn't load a web page. Presumably the first failure with a standard GET request would be that the renderer would look for a "show" view and fail to find it.
Names / Cards
Every module must have a card, and a unique name on that wagn. (Though it need not be universally unique). If the existing name is in use, it should be renameable on install. Note: this is not yet the case; needs design. It may be that this is not in place until the namespaces are.
Trees / Dependence
We must have a way to track what other modules any given module depends upon. Installing the module should automatically prompt installation of its dependents. With this representation, any modules representation can be tested by installing on a core wagn. If it doesn't work, something's missing.
These trees are key to meeting the needs of Wagneers and Developers simultaneously. If our modules interface helps Wagneers to find modules on the basis of high-level functionality, and the installation handles all the lower level stuff, then we've met the "big parts" need via well-assembled "small parts".
Conflicts
Conflicts should be kept to a minimum by banning direct reference to card names in any module, but ultimately they are unavoidable. Like dependencies, however, they should be trackable and identified at point of installation.
Bundles
James said (of the nuisances of MediaWiki's modules):
As a result of these problems bundle releases have sprung up that include a set of modules already installed
I actually think a key part of the solution may be embracing something like bundles/distros from the start.
People installing Wagns will have very different needs / expectations. Some might mostly want it for traditional wiki functions, others for blogging, and others for more specialized wagn functions that may have been created for, say, energy audits or investment research or bug tracking or whatever else. It would be nice to offer bundles to help these folks.
Separating dependency from defaults
Defaults need to be robust and low-hassle (for wagneers). These are the kinds of considerations James has been highlighting. New installations (bags) will clearly need to have lots of "important", maybe even "vital" modules -- the kinds of things nobody would enjoy interacting with Wagn without, but they won't be "core" in the sense that they're not the root of the dependency tree.
Module dependency needs to be clean and explicit (for developers). This is an architectural / design decision we'll need to make: can the core depend on modules?
Wagn actually already has some infrastructure for reference tracking, and I'm thinking we should piggy back on and refine that (core) structure for tracking dependence references, which will be much cleaner and more complete if it goes all the way down to the bare no-modules core. (A module with no references to other modules must function and pass all tests with nothing but the bare core). There's a cool way in which modules as cards and dependencies as references means our data referential integrity and our modular dependency tree become integrated systems. Dig the wagniness there!
I'm not sure I understand why modules need a card. Maybe for the "wagneered" part of it, but you have to leave cardnaming up to that app designer then, don't you? If they have a card, what is the cardtype?
--Gerry Gleason.....Fri Oct 16 03:57:58 -0700 2009
I want modules to be a card for references/dependency tracking, revision history, linkability, etc. And that's how I see WADL as requiring a module -- add the module card to the list. I want it for standardizing module interface -- we'll want ways to turn them on and off. I've mostly assumed there would be a Module cardtype (which could plausibly inherit from Cardtype?)
--Ethan McCutchen.....Fri Oct 16 10:10:15 -0700 2009
Sorry, I'm trying to focus on what needs core and ruby module support. I agree that would be good, but I don't think that is part of the core. There needs to be some interface from the user/admin level stuff and how it loads modules and such, but we cannot even count on the services to automatically load dependencies will be available on any given system. We could still support rake tasks pretty easily, and cards could even be fetched to tell the rake tasks what the dependency parts would be. But the point is we want to split this feature into the parts that can be wagneered and the parts that are needed to support the system/code changes in typical admin contexts.
--Gerry Gleason.....Fri Oct 16 15:08:45 -0700 2009
In fact, you can start wagneering some of it to find out what you might want automation and code support for.
--Gerry Gleason.....Fri Oct 16 15:10:28 -0700 2009
Is there a reason these things always go in the root of the namespace? I mean, why not have System+ or maybe *system+, or *modules+ . Is there a wagn principle that argues against such practices?
--Gerry Gleason.....Fri Oct 16 15:13:31 -0700 2009
what do you mean by "these things"? Nothing inherently awful about Systems+. Could also do naming conventions. Some qualifiers:
1. Let's say I have a module called "Mapman" that plugs into some mapping software. If I name it System+mapman, that will automatically create a MapMan card. So it's messier, and it ties up the root anyway. System+ is not really a new namespace.
2. Plus cards can't be tags. So things like *options, for example, need to be simple (nonplus) cards.
--Ethan McCutchen.....Mon Oct 19 16:26:44 -0700 2009
Why does the tag card have to exist. I don't see how it is actually related to the plus card. It really only has a contextual meaning in the context of a '+' The '+' operator is not commutative, I see why the trunk needs to exist, but not the tag. Doesn't mean you can use it, but if it is an empty card, I don't think you should create it.
--Gerry Gleason.....Mon Oct 19 18:22:45 -0700 2009
OK. Some interesting stuff in here. My approach is to learn more about what the architecture reveals to the "central participant type" - the Wagnerian Operator which seems to have been sort of skipped over in this discourse. I intuit that the relationship between naming conventions and namespaces become obvious and intuitive so that the Humans can meet the code not half-way but closer to it. I hope to find a way to get my "hand" in play here.
--~~~~
--cquin.....Mon Mar 07 11:36:49 -0800 2011
Did we just replace 'bags' with 'packs' in vocabulary? Should probably edit above to fix that if we have.
I think I understood that, Charley. Operations concerns are not completely invisible. My professional work has been a lot more about deploying that developing, and as you know I host some Wagns. This is very much a now and future concern. Most wagns are now deployed by wagn.org, but things are in the works that may change that a lot, and operations will be a much bigger part of the development roadmap.
--Gerry Gleason.....Sat Mar 12 03:59:08 -0800 2011
Blessed vs. Possible
That doesn't mean we'll prevent you from using unblessed API; it means we're not going to break our backs to support anything unblessed. So if you use unblessed code, we celebrate your bravery, but know that future releases may break your modules without much warning or apology.
+API+best practices
An area to record helpful habits for private modules as well as conventions to facilitate adoption and compatibility of public modules.
We need more examples of what this might be, for example use of * in rightnames. In general managing the Wagn namespace, etc. What else is in here that we have already identified?
Original intro:
A ton of this vocabulary is still in flux, even at the highest level. Are these Extensions? Modules? Are they all Modules but Extensions are ones that are not part of the core? What's our terminology for distinguishing between mini-applications built solely with cards vs. those that involve code?
...and nobody is commited to that "WEAPI" term.
I'm happy to bring more folks into those discussions if interested, but I don't really want to have them here. For present purposes, I'm using Modules and Extensions pretty much interchangeably, and I'm specifically talking about extending wagn's capabilities through code.
With the implementation of this blueprint comes a bunch of things that we want to have clear vocabulary, which may include changing the meaning of terms that were already in use. None of this is fully settled yet.
0. Group of cards (exchanged between/among Wagns
Hand
1. Any chunk of code (and cards) generated that use the functionality implemented from this Blueprint.
just the code
Proposed: Module
Related terms — private |whatever| for code you develop for use on a Wagn you're hosting, and public |whatever| for code you develop with the idea that others might use them, and which may be distributed via wagn.org
2. Any such code components that are built in to new Wagn installs by default (and without which Wagn may not even function).
Proposed: Core module
√3. Any such code and card content that are not built into new Wagns by default.
Code, cards and
Proposed: Add-on, app, pack,
could be module, vendor or gem
4. The "extra" functionality & data already connected to some groups of cards (specifically: Cardtypes, Roles, cards with accounts, Files, and Images). These will be implemented in the way described in this Blueprint eventually, but will most likely be around for at least some time after this Blueprint is implemented.
Currently: Extension
May be helpful to look at this list of different kinds of chunks that might get developed:
Kinds of modules
It's helpful in this to consider some specific modules people might develop, and what kinds of things people might want to do in modules generally. Even better to flesh them out as user stories and tag them "Packs".
- A module enabling adding custom tabs (at the top level, not just in the Related tab) -- now this will add/modify the cascading menus on the 'gear' icon. Includes full customization of the theme/UI or the gear. This is both changing the look/feel and changing the contents of these controls.
- A module exposing content from an external database or web service in Wagn cards. (Note that databases with a web API can already be accessed via HTML cards, and even if you want to massage the HTML they return, you can do that via Ruby or Script cards.). For example, Meta-currency flows associated with some or all cards.
- Adding cardtypes:
- an improved Image cardtype which exposes image card metadata from their files, such as +*make and +*model of the camera that took a photo.
- a Contacts cardtype (possibly involving several cardtypes) managing relationships among people, organizations, addresses, phone numbers, etc., possibly integration with LDAP and other systems, etc.
- Table cardtype
- The Cardtype and/or *right sets used to access external data as above.
- Adding views:
- display a Pointer or Search (of Numbers) as a bar chart
- (add more)
Keep in mind that this initial API is not aimed at making all of these easy, and obviously many of them will involve non-blessed methods.
Thoughts?
Hoping that having a clear vocabulary might make our exploration more accessible to others.
--John Abbe.....Wed Feb 23 08:10:17 -0800 2011
Extension isn't going to be changed that much, the change is really more implementation than functional. We don't really want API users to even learn the current meaning because that will be gone by the time they see the API.
I'm now more confused between "Core Modules" and "??? (new)?. I think what we intend for the former is pretty close to what you have there. It is stuff like Renderers that you need one or more of. RichHtmlRenderer will be like that. Hard to call it a module, though as it isn't clear it can even be swapped out, but in principle it can. For example when we have XML, the XmlRenderer will likely be core. It isn't that Wagn doesn't work without it, and it could even be optional, but it is really deeply embedded in Wagn. The main thing may be release synchronization much as you have with Rails where there are many modules at the same version number, upgraded at the same time.
Card Modules for stuff like: Permissions (probably also Core as are many of these), Roles, Users (and accounts). Here we also expect many user contributed modules as well.
--Gerry Gleason.....Wed Feb 23 09:52:15 -0800 2011
re "Core Modules" and "???" - the former is any module in the core, the latter probably is fuzzy and not useful to distinguish from modules in general. I thought maybe "Card module" for "???" because I was thinking of modules that add custom data to a Cardtype, but accounts (a current extension) obviously breaks that definition. It could be Set module, but yeah, I think it's too broad/fuzzy to be useful, so I'm removing it. Just for reference, it was:
??? (new) — this will be the new term for the added functionality that used to be called an "extension". Existing code will be refactored to use the module architecture. This currently includes extra functionality for account cards and Cardtype cards (which will become "core modules"), and it will now become much easier for developers to add their own (which will be "extensions" in the new sense, unless they are made part of the core Wagn installation).
--John Abbe.....Wed Feb 23 14:06:54 -0800 2011
"Extension" seems to have lost all semantic content here. If you aren't making reference to what feature or component that is being extended, or there is some part of the feature that logically extends the base, then the word is redundant and confusing.
--Gerry Gleason.....Wed Feb 23 14:59:25 -0800 2011
My understanding (which could of course be wrong) is that we'd like to stop using "extension" for the extra stuff now associated with cards-with-accounts, Cardtypes and Roles. To do that, we need a new term for whatever that is, which will only be in use until we implement that functionality with modules. After that we'll instead talk about the cards-with-accounts Module, Cardtype Module, and Role Module.
The reason for this shift is so that we can start using the word "extension" for modules that are not part of the core.
--John Abbe.....Wed Feb 23 15:05:13 -0800 2011
More exactly, that implementation feature is going away. It isn't clear how quickly, so there is potential for confusion during the overlap period. I don't like generalizing it in that way, as I think modifiers on Module gets at that better.
In my view, if we keep using that word at all as blessed vocabulary, that it should have a meaning connected to some core feature that is getting extended. What we had would be "Card extensions" and we may have some remnants of that for a while. What I'm saying is that it isn't a noun on its own, it has to modify another noun.
--Gerry Gleason.....Wed Feb 23 15:13:42 -0800 2011
To clarify an idea (not advocate for it), it's possible to thing of "extension" as extending Wagn, not a particular structure or feature.
So one notion was that wagneers would end up rarely using the word "module" but instead always using the word "extension".
Back in the advocacy space, I really think there should be just one word that most Wagneers would encounter for all of this before they venture into actual module development (as in, if you're only installing and uninstalling, not writing anything). "Plugins" already has a specific rails meaning that we may or may not want to attach to. "Add-ons" is another option.
The other thing I'd like us to get clear on is how/where these words overlap with non-code structures in wagn.
It might be helpful for us to use this card to make one list things we want to name, another list of possible terms, and then propose ideas for mapping them. Maybe?
--Ethan McCutchen.....Wed Feb 23 17:03:35 -0800 2011
I've reworked the section up top along the lines that Ethan suggested, and am thinking maybe it's good to not even think about terms yet, but just see if we all think that those are the four things we want terms for? Anyone seeing a better way to slice the pie, more/fewer/different categories?
--John Abbe.....Wed Feb 23 22:16:06 -0800 2011
I realize I'm a little fuzzy on 4. Does it only apply to the existing code for the five groups listed there? (I'm using "group" instead of "set" because we can't (yet at least) define the set of cards with accounts :-P.) That is, is it true that no chunk of code developed as described in this Blueprint will fall into category 4?
--John Abbe.....Wed Feb 23 22:23:54 -0800 2011
This is looking really good now. We may be able to clear a lot of this discussions. Maybe leaving behind the few items that are not completely resolved.
Ethan, right-on for the "what to call it on first encounter" (canonically). Add-on or AddOn (inflected for context) works for me.
--Gerry Gleason.....Thu Feb 24 05:04:22 -0800 2011
I started this in hopes of resolving the vocabulary very soon, thinking that having it be consistent on Packs+api will make it more accessible to others (and us for that matter). Does that seem worthwhile?
If so, maybe a call with us plus Lew?
--John Abbe.....Thu Feb 24 14:42:51 -0800 2011
Should the modules be implemented as Ruby Gems, we may get some assitance with dependency checking and installation from Y. Katz's Gem Bundler. See Rails Bundling and Gem Bundler is the future.
+gem bundler+example
I use Wagn 1.5.2 with Passenger 3.0.2 and Bundler 1.0.9. Bundler should really be the way to roll (out) the Wagn. It makes setup so much simpler. Here is a short How To:
To checkout 1.5.2 run:
git clone https://github.com/wagn/wagn.git
cd wagn
git checkout -b v1.5.2
git submodule update --init
Then create the proper Gemfile and Gemfile.lock files in the wagn directory :
wget https://gist.github.com/raw/832674/985fc97d7f5ec04373815c28319cd87c5544d70e/Gemfile
wget https://gist.github.com/raw/832682/e63ef5773a5f8a21654bd044dee955876729c5a3/Gemfile.lock
and run
bundle install --deployment
This installs all required gems into the vendor/bundle directory.
The --deployment switch ensures that only gems managed by Bundler are used when running as Passenger or with bundle exec.
E.g. to run rake tasks under that Bundle use
bundle exec rake <task>
Do not forget to set up config/database.yml and config/wagn.rb.
To run Wagn with WEBrick (in development environment):
bundle exec ruby script/server
Best of all: Passenger uses your bundle automatically.
- How do folks find/explore existing packs?
- What meta data do we gather about packs?
- How do we decide which packs come with default wagns?