Mod+API+discussion

Awesome, Ethan!

--John Abbe.....Tue Feb 22 03:02:42 -0800 2011

Presumably, one function the API offers is the ability to store/retrieve/update data about a card beyond what the minimum card functionality requires. This is necessary to move Cardtype and account data into the card table, yes? Andt would be part of Model, and therefore part of the +setup section?

In any case, what does it look like?

This functionality should imho make it easy for developers to expose such additional data as cards. For example, an updated Image Cardtype might pull metadata from the image file (make and model of camera, etc.) and make it available as +*make, +*model, etc.

This will result in many new names in the Wagn namespace being claimed. So it would be good to include in Module documentation a clear philosophy/guidelines on choosing names (including when they should be Star Cards), and to Wagneer a space here on wagn.org where people can propose and get feedback on card names they're introducing before releasing a new non-core Module.

--John Abbe.....Wed Feb 23 07:45:22 -0800 2011

--John Abbe.....Wed Feb 23 09:52:51 -0800 2011

I think the API already gives them the ability to define such "attributes" and you are correct that we need to define the naming conventions in use. I can define *make (and similarly *model) something like this:

view(:raw, :type=>'Image', :right=>'*make' do Image[card.id].make end

The module would define the "Image" class (typically an ActiveRecord model class, but it could be anything) which would work classes like User and Role (particularly after the codename refactor is in) where you can access the extension model with card_id as an index (cached or not) and access its attributes.

--Gerry Gleason.....Wed Feb 23 10:07:09 -0800 2011

Hehe, we may end up having something like that Gerry, but Image[card.id].make is all new API that isn't related to the current proposal. I *think* I get what you mean, and I trust that since it comes from you there's value in the idea and am interested in exploring it elsewhere. But I think we should start by showing the power of the API proposed above, not proposing new stuff.

John, I would nitpickilly say that the API is *not* about offering any new way to store cards per se -- no new underlying structures. Instead, it gives you the ability to add more levels of meaning to higher level structures (through actions) and to offer new ways to view them (through views).

Let's take the example of accounts. The way they currently work is that (a) there is a separate users table, (b) a separate accounts controller, (c) separate accounts views, and (d) a bunch of other custom stuff to join all that stuff with cards.

In the new model, you would *setup* a system of storing all that account information in cards. The setup portion will not be about adding any new functionality, low-level structure, or views -- it's just basically automated wagneering intended for getting the cards in place.

Most of the magic for this use case will come in the Actions section, where you will add all sorts of conditions around adding the relevant cards. Eg upon creating (event) account cards (set), you have to have a valid password (validations), you have to send out a notification (triggers), etc.

The same goes for your camera example. When you update (event) image cards (set), you automatically add this and that information to *make and *model.

Good point about best practices. Lets add a best practices card here.

One "and" (not a "but"): a large percentage of modules created -- probably the majority -- will be for people trying to get some damn thing working on their own wagn. We may want to make a distinction between the kind of best practices that are intended to ease folks' development practices vs the kinds of extra steps needed to make a module more broadly useful. Or, since these are unlikely to be easily separated by a bright line, we should be as clear as we can about the rationale for every convention.

--Ethan McCutchen.....Wed Feb 23 11:01:52 -0800 2011

By the way, John Abbe added a Packs+vocabulary discussion to work through some of the vocabulary conversations that we need to have around modules.

--Ethan McCutchen.....Wed Feb 23 11:50:27 -0800 2011

No, it isn't part of the API, it would be part of the support added by the module creating the extension. The rest of Wagn shouldn't know anything about it. I get what you are saying about handling the card based stuff in the API, but we will have core modules with this sort of thing and we may never refactor those out. The principle is that if a module does this sort of thing, 1) it must be card_id based, and 2) the rest of Wagn shouldn't know about it or need to.

--Gerry Gleason.....Wed Feb 23 12:51:22 -0800 2011

We're definitely planning on factoring out the old roles, users, and cardtype extensions? Are we only talking about files and images? If so, it's probably worth getting rid of them, too. I don't even think those tables are really used by wagn in any meaningful way. It looks to me like they're just there to keep has_attachment happy. I think the data is pretty much ignored.

None of this invalidates what you're saying at all, and we may want to have conversations about this somewhere in wagn to help solve people's issues when they go this route. But I don't want to do anything to promote the spread of this kind of approach until we've seen more evidence of its need/value.

--Ethan McCutchen.....Wed Feb 23 12:59:46 -0800 2011

So...will we not be blessing actions that add custom content to new cardtypes? What will be blessed in the actions section?

--John Abbe.....Wed Feb 23 13:35:12 -0800 2011

Is that in response to my previous comment? I'm talking about not adding db tables. So far the only thing we know will be blessed is #define_action.

--Ethan McCutchen.....Wed Feb 23 13:39:58 -0800 2011

Yes.

And one thing someone might do with an action is adding custom data. We definitely wouldn't bless doing that by adding a table, but how about if they're just adding a (dunno the terminology) row? field? to the one big card table?

--John Abbe.....Wed Feb 23 13:51:03 -0800 2011

Nope, not blessing any api for that. Nothing on the SQL level, but whatever you want on the card level. Value transformations, CRUD operations on other cards, validations... all with whatever logic is needed.

--Ethan McCutchen.....Wed Feb 23 13:56:39 -0800 2011

Ah, they'll have to recode when we allow non-SQL backends, that's why not to bless it, right?

Are validations just about determining if _user has permission to do whatever action?

--John Abbe.....Wed Feb 23 14:10:27 -0800 2011

Yes, that is part of it. Another way of looking at this is not as whether this sort of thing is "blessed" or not, but just that the standard Wagn components, such as WQL, won't know about it.

There is a subtle point here about the way Wagn handles content that is produced at the view level. I don't think it will be easy for example, to search the values of this sort of content. It doesn't exist at the model level, just the view level. It begs the question of how WQL could even see the values of this sort of field.

I think the conclusion we are reaching is that while my point that module developers may want to do this sort of thing, it will be completely outside of the Wagn space. If we really wanted to support this, we would have to support ways for these fields to be included in searches.

Ethan's point about how most of the extension types do just go away is well taken. The one problem area we will have to address is Users and accounts. Note that even now including email in searches (out of the User extension) is problematic. If we can make even this core function use the more strict API with "everything in the cards", I think it will be a big win, it just isn't clear yet when it will be worthwhile to tackle that.

--Gerry Gleason.....Wed Feb 23 14:49:24 -0800 2011

John, by validations I mostly meant content validations. valid numbers, valid emails, valid passwords, etc. It gets relational when we get into required fields, so there's a bunch of design work there, but this is whereabouts the magic will happen.

Gerry: right on, brother. Only quibble would be with the "if" in the "when and if" we move account info into cards :)

--Ethan McCutchen.....Wed Feb 23 19:04:28 -0800 2011

Thx re validations.

make exposed external data available to WQL

--John Abbe.....Wed Feb 23 21:49:35 -0800 2011

(moved all the following from the +actions discussion)

One critical part is what will be passed to the block. Let's dig a little bit around, the C and U; in RichHTML (slot based standard Wagn UI), typically this will be a post to a 'new' window or 'edit' slot would generate create or update events depending on whether the card existed to start with. The inputs at the "thin wrapper" level are CGI parameters from new and edit forms as rendered by the RichHtmlRenderer (well it will be by the time any of this is done).

With XML we would have similar structure. We want to implement a RESTful interface to the same CRUD action layer. Here the wrapper is actually thinner than with RichHTML. We want to implement a POST that is Create (if new) or Update (if exists) of a set of cards as extracted from an XML representation of those cards (as presented by the XmlRenderer for the GET operation on the same URL (proto://site/card.xml).

I think something like the Xml subsystem with Rendering and controller wrappers is worthy of being some sort of "Core Extension". In the lingo, also, "Card Extension" should be what will become of the current use of "extension" in the code. This will be a core facility of the API, whereas the above mentioned "Core Extension" is part of the modular core rather than a API class extension. Someday we may be able to take contributions at that level, but for now you have to join the core team to do it ;) --Gerry Gleason.....Tue Feb 22 13:38:00 -0800 2011

My thought was that there really wouldn't be anything different on the action level for different formats, just on the display (view) level. I would think that the impact on the card data shouldn't be different because it had been CRUDded from a different format, should it? Can you give an example of when you'd want different behavior for different formats (beyond the view rendering)?

I do see that getting there means thinning up the wrapper dramatically - right now there's a lot of RichHtml-specific logic in the controllers. I'd figured we move almost all of that elsewhere.

If I understand what you're saying about card extensions, you mean that we'll be able to extend cards with other cards (without extra db tables), right? I'm guessing it will be more helpful if we create a more distinct vocabulary for that phenomenon and leave "extension" to mean custom code.

If you're thinking that we'll need different custom code to accomplish what you're calling "card extensions", what do you think that looks like?

--Ethan McCutchen.....Tue Feb 22 16:25:30 -0800 2011

On card extensions, the extension must join any supplementary data on card_id, and all that is considered part of the card, whether or not it is stored in the card content. Think of File and Image as the model. Well, those don't include cards in this sense, and yes I want to include that.

Now, cards have an extension, but in this model the added cardtype is an extension, or an extended type. It can still carry data external to the card contents if it needs to.

On XML and the controllers, the problem is that HTML and CGI controllers are never really RESTful, they have to fake it. The ideal action API would map HTTP based REST protocol directly to the CRUD actions, but as it is we don't have that.

Let me illustrate the core of the problem. Part of REST is the idea that the object has a single URL, so with HTML and CGI we have:

/card/action/cardname.html ( or action can be in cgi params as ?action=, or whatever detailed proto you have)

We want:

http GET /card/cardname.format => read

http POST /card/cardname.format => create or update

http POST /card => create (name in data)

http DELETE => delete

http GET /card => read all cards

If you are going to put that sort of thing into the base code, then I'm good. What I was thinking is that this sort of thing is what we can call a "core extension". Something necessary to support certain kinds of protocol routing. This doesn't actually involve the action api so much as it is about the connecting of routes and service protocols to this api.

It is right on to focus the API on CRUD, then with that clearly defined in the core action declarations that support Wagn, we will be in a much better position to do what we need to in this connecting layer. It is actually the HTML and CGI handling that requires all the warts and hacks in the controller wrappers, the REST interfaces are simpler.

--Gerry Gleason.....Wed Feb 23 04:41:18 -0800 2011

Oh, you are actually still talking about extending data outside of the current database tables. I've been thinking that we don't even want to do anything to support that in Wagn 2.0. Granted, there may eventually be a need, but for now, why bother? Is this driven by a user story?

In any case, I want to punt on that and dig into the Web API. In fact, this should probably be a separate section, but for now...

YES, this is almost exactly what I have in mind, with one exception -- if everything's a card, why mention it?

It basically boils down to the following common cases

GET /cardname

POST /cardname

DELETE /cardname

with the following variants:

1) .format to specify response format

2) view args (currently "view", "item", "size". I guess things like "_keyword" for the search falls in this category, too?)

3) action args (not sure what those are yet, but I assume there'll be something)

4) model args for POSTS (:name, :type, :content)

5) name can be in args, too

We might also make it so that cardname can also be a number (id) or start with a colon (codename). If we go that route, we need to restrict cardnames (eg, no number-only names) to make that work. Might be better to have a number prefix, but not "#" -- that's too loaded in urls.

I don't think we have to add any special handling for "read all cards" -- we can manage that with search cards, no?

And that's it!! Pretty friggin thin, right?

Thanks for advancing this part of the conversation!

--Ethan McCutchen.....Wed Feb 23 11:30:53 -0800 2011

hmm, how do you do /new/Cardtype? I guess it's just this: GET /?type=Cardtype

--Ethan McCutchen.....Wed Feb 23 11:35:59 -0800 2011

... and /card/edit/cardname = GET /cardname?view=edit

--Ethan McCutchen.....Wed Feb 23 11:38:46 -0800 2011

"new" is a "POST", and type would be in the data for XML, for Html, it can be in the path or CGI-args.

The XML might be something like <card name="foobar" type="footype"> some xhtmlish content here < /card>

as data posted to the "collection" URL, or you can post it to the name. The only difference is that the first should be create and potentially fails if it exists. The point to take away, though is how different the request data is for these REST actions than with CGI parameters. The issue may be that like with the Renderers, we have to look up an action handler from the format.

One thing I'm not really getting, though. How do you expect to handle the complexity of routing for these very different protocols with a single simple wrapper? I can see pushing those details into Rack middleware modules, but that will require some more work in this area. What do you see? Can you outline what would be in the wrapper in psuedo code?

--Gerry Gleason.....Wed Feb 23 12:38:59 -0800 2011

new isn't a POST -- it's a GET request for a form. This follows the rails convention: new and edit are GETs, create and update are POSTs.

I wouldn't think you'd have format-specific handling for posts at all (except for the eventual response). HTTP is HTTP, regardless of what you get back, right?

--Ethan McCutchen.....Wed Feb 23 12:45:50 -0800 2011

Ok, right. Then REST maybe doesn't have a "new". That's really one of the big warts on REST vs. HTML. In a way, what you are asking for in that case is the "schema" for the card, that is what subcards it will want you to post together to create a new card of a given type. You don't really "render for edit" here, the REST API client has to know somehow what the shape of the data it posts should be. I don't think REST really specifies that, schemas and meta-data are outside its scope. You "get" the data, then "post" it back modified.

I think I see where you are going with new => GET. So new(:name => name, :type => type) would give the default content (*default or *content settings) for a new record. In XML, this would give something I can modify and post back. Makes sense. Just like when you do fetch_or_new internally.

--Gerry Gleason.....Wed Feb 23 13:02:15 -0800 2011

Back to your original comment, Gerry: do you think there will be a Handler class that roughly parallels the Renderer class (I feel like you brought that up elsewhere). And if so, mightn't it be the case that we may not end up passing anything to the define_action block but rather attaching it to the object? Or, at least, we'd have that option, right? It's certainly helpful for inheritance in the renderer case. We don't have anything as clear as formats to determine an Action class hierarchy for us yet, but I wouldn't be surprised if we ended up going there eventually for some reason...

--Ethan McCutchen.....Wed Feb 23 19:00:11 -0800 2011

suggest moving the REST conversation to RESTful Web API

--Ethan McCutchen.....Thu Feb 24 11:20:45 -0800 2011

Now I'm not sure about that. The messy part that we aren't really talking about is the part that I have tentatively proposed using Rack for, getting from the incoming request to the handler. I'm thinking that is where all the argument handling will go. The idea is that however the request comes in, there will be middleware modules that parse the incoming form of the data (i.e. CGI parameters for single and multi-edit, etc., and parsing the XML data for an XML PUT/POST) and put them in a canonical for handling and rendering.

This suggests that the per format special handling for controllers is before it gets to our API. What I am proposing is that the method for extending Wagn in this area would be through optional or replacible middleware modules.

--Gerry Gleason.....Thu Feb 24 19:50:44 -0800 2011

Based on the call last night, I think we may need to do a little bit of reset here. I added a page to gather and form more thoughts about how Rack relates to this, and I'm now getting how this is distinct from RESTful Web API.

What we need to go into here is how that page is connected to this one, and that's where I think we want to use Rack's simple protocols. What should happen is that "RESTful Wagn API" will be an Rack app implementing this, but this module won't service any requests, it will call one or more Rack service applications.

What I have been confusing with that other task is this one: to define the Rack::Wagn service interface. At this point all traces of the request being about web protocols will be gone, it will be pure CRUD operations on cards or card collections. This is what I think we need to base our action handlers on.

Note that other web apps and frameworks might be combined in this RESTful routing layer (or above it) as well whether or not they are serviced in Rack::Wagn. I think you can even do something like throw an exception from a Rack::Wagn handler and have another application fill in the content for just that part of the content. That part may be the inside of a slot (i.e. the :naked level with all the wrappings intact around it).

--Gerry Gleason.....Thu Mar 03 08:31:36 -0800 2011

I think time defining the Rack::Wagn service interface would be well spent, but I'm not clear on what it means to base our action handlers on that.

Maybe it would help to get a touch more concrete. I would think at the service interface level, you would be making model calls like:

Card.create 'fooname', :content=>'barluga whale'

... and that the actual "events" in the Action Pattern api -- define_action( event, set ) &block -- would be things that happen within those model calls (like #create).

So what does it mean to base the handlers on the service interface? Is there exception handling going on in that layer? How does that look?

--Ethan McCutchen.....Thu Mar 03 09:34:00 -0800 2011

Per Rack, the service interface would be AppClass.call(env), then we define what handlers expect to find in env (i.e. a card and an action, maybe a bit more). Then we define handlers per action, and the actions would be CRUD plus if we need it.

The handling of exceptions is then a critical question, or more precisely, what exceptions can the handlers throw to get different behaviors from the upper layers. Also, I think callers of these handlers can use return codes as well (a Rack app just returns a triple of [status, [headers], [content]]), status being the http return code.

--Gerry Gleason.....Thu Mar 03 13:08:25 -0800 2011

ok, but when we say "define_action( event, set)", the "event" part is still lower level than anything the service interface knows about, right? Because, for example, ActiveRecord callbacks include before_validation, after_validation, etc. Our thinking has been that, at a minimum, our events would cover all those callbacks, and they're of necessity at a lower level than the service interface (which, by the way, will really only ever know about the outermost cards, whereas the events may involve inner cards).

My thinking is that the service interface level does pretty few things (though, I agree, they're critical): It handles issues in finding or accessing the card, and it handles exceptions thrown by the model level. But I wouldn't think that the Module API would really be communicating explicitly in the lingo of the service interface, it would be at the level of the model API, no?

--Ethan McCutchen.....Thu Mar 03 13:36:02 -0800 2011