Jun 19, 2013

MailChimp’s API, V2

New MailChimp—AKA: Chinchillais here. (OK, so maybe only one other person calls it chinchilla.) Coincidentally (a coincidence since this has nothing to do with that), we’ve also released a totally re-architected version 2 of our API. The version 1.x series has proven plenty useful for years, but it’s old and was built before most modern principles of API design really caught on. Our major goals while revamping the API were:

The first thing to keep in mind while reading this is that, as you continually say to yourself, "Duh, why didn’t you do that before?" remember that API functionality typically needs to be backwards compatible, and stay stable across major versions. In other words, consider the effect some of these changes would have on existing code if we just changed stuff. Which is to say: none of this applies to v1 of the API! All right. Onward!

(Alternately: tl;dr: RTFM!)

More RESTful

We’re by no means REST purists, so we’ve implemented the principles that we like the most and eschewed the ones we kind of hate (like "proper" verbs). First and foremost, proper HTTP response codes. In the version 1.x series, any call to a valid API endpoint/method would return a 200/OK (unless we really had broken something good [thankfully that rarely has ever happened]), even if an error actually occurred. Now, any time we’re returning an error, you’re going to get a 500 response code and the response body will contain the details of the error. Many folks will recognize this makes error handling even easier since you don’t have to parse a response body to figure out what happened even though you’ll probably eventually want to. For anyone who had noticed/relied on our X-MailChimp-API-Error-Code (that contains the error code that’s in the response body), fret not—it’s still there, too. One exception to this rule is that the various batch methods will always be returning a 200 with error codes inside them—not really anything we could do there. Anywho, then, as you’d expect, calls that work without error will return you a nice 200 response and you can deal with whatever they send back.

Next up, the URL structures are vastly different. Version 1 documentation always split methods up into various categories (List Related, Campaign Related, etc.) and used funky camelCasedMethodNames (that weren’t always even consistent—yes, we’re looking at you, xxxAIM methods [more on those later, too!]) and generally required you to always pass said method name via the "method" GET parameter (please don’t use XML-RPC). So silly. Things have been moved around, renamed, namespaced, and generally made to make more sense. GET parameters are no more (we’re begrudgingly going to support them because POSTing is "hard"), but would generally rather they not be used. Because POSTing really isn’t hard. The URLs also match some of the app’s URL schemes a bit more closely, which we hope proves a tad helpful. Some examples, you say? Sure:

v1.3 v2.0
/1.3/?method=listSubscribe /2.0/lists/subscribe
/1.3/?method=campaignCreate /2.0/campaigns/create
/1.3/?method=ping /2.0/helper/ping
/1.3/?method=campaignsClickStats /2.0/reports/clicks
/1.3/?method=campaignEmailStatsAIM /2.0/reports/member-activity

If you’ve worked with the Mandrill API, that should seem eerily familiar. Good! Lastly, and this really has nothing to do with REST, but API v2 is HTTPS-only.

Standardization and extensibility

I like big words. That really just means be less dumb and give ourselves room to expand so that less dumb has to be done in the future. Version 1 of the API has (jankily) supported POSTing JSON objects in lieu of using serialized HTTP parameters for quite a while. Why jankily? Because you had to urlencode them (don’t ask). That was unnecessary, and version 2 removes that. JSON objects, of course, must still be POSTed (but you’re POSTing any way, right? RIGHT?). Serialized HTTP parameters are certainly still supported as input parameters as well (and you should POST them!). JSON as an output format was even more janky because we weren’t paying attention when it was implemented. How so? The JSON standard—and thus, most libraries that implement it—does not like scalar values (makes sense, they aren’t objects or collections), such as the tons of places v1 would just return a boolean or string value. Of course, PHP on the other hand, turns out to be totally cool with encoding/decoding a pure scalar value (it doesn’t, it just passes them around), and that’s what we failed to pay attention to. Whoops. That’s fixed—every return value is either a struct or an array.

Which segues nicely into this: A longtime pain point has been that languages of the programming variety don’t call the same/similar data structures the same things. We often tended to just go with calling everything arrays the way PHP does. It might be an array or it might be an associative array (hashmap, struct, dictionary, tuple, and on and on and on), and that confused/frustrated folks, especially when they didn’t look at raw data being returned (no one does that). There’s only so much that can be done about that, but we’re trying really hard in the documentation to consistently call things in these manners when describing input/output values:

  • Arrays will be pure arrays that just contain a bunch of items (no keys!).
  • Structs will be key=>value pairs (associative arrays, dicts, hashmaps, whatever).
  • And, of course, arrays may contain structs and structs may contain arrays.

Next up, we’ve moved to using more nested input parameters where possible. For one example, where "start," "limit," "sort_field," and "sort_dir" previously had been listed as separate parameters, we’ve often collapsed them into an "opts" struct. The point there is to make huge method signatures a bit smaller (which really just matters in API wrapper-land) and make it significantly easier to drop new options in without affecting signatures and the such. In other words: we’re lazy.

Finally, we started working on getting you more data in a more standard fashion (more on this in a minute). What that means is where we may have just returned a single email address alongside report data, we’re now going to dump out a full member profile record (this is basically what you get from the v1.3 listMemberInfo() call). Ditto for campaigns and places that may have just returned a campaign ID instead of a full set of campaign data (for example, searchCampaigns()).

Clean up time!

This is partly just an extension of everything already said. However! As mentioned above, HTTPS will be required—no more HTTP-only support. The ridiculous API Key "security related" methods that never should have been added in the first place will not be making an appearance. If you ever wanted to get, say, the members who opened a campaign, you would have found yourself using the oddly name campaignOpenedAIM() method. Wait. AIM? What in the world is that? Well, that is the very definition of old, old, old cruft. Many moons ago, A.I.M. stood for "Actionable Intelligence and Metrics," and was actually a paid add-on. Then we and our servers grew up some, renamed that to the more sensible, less-obtuse "Subscriber Activity," and made the data free. As it should be. Because it is yours. That’s my long-winded way of saying that, in v2, you’ll now be calling the more reasonably named /2.0/reports/opened method. This is pretty minor, but if you’ve ever left off basic required parameters you’d be presented with a -99/XML_RPC2_Exception error code. That’s now going to be a more logical ValidationError/-100. The actual error message text being returned will still generally be the same and explain issues in detail as would be expected.

Match the app + improvements

One of the most amusing things is the number of times people believe that the API is some magical thing that lets you do all sorts of crazy stuff that our service doesn’t support or can end-run how things work. Ha. Nope. Our API—just like most any you’d run into—is generally meant for automating what you can do inside the app. But you knew that. V1 mostly does match what you can do in the app, but over the years things change and can fall out of line a bit. It’s also infinitely easier to drop a UI change on folks since there’s little-to-no need for considering backwards compatibility—and that happens.

Not so much with the API. But with a sparkly new one, we can kind of do whatever we want. So, first up was to match the various "data tables" folks are used to seeing in the app. Using our new /reports/opened call as an example, where it used to simply return an email address and the number of opens, it’s been expanded to return the number of opens and that entire member profile data structure previously mentioned as well as allowing the same sorting as the app. Ditto for the other "subscriber activity" report data plus the lists/members call (that’s been a pain point for a while). However, due to returning WAY more data along with increased load from the potential heavy lifting of sorting, those methods are also now going to be limited to a max of 100 records (where they previously were at 15,000). Part of the hope with that, as well, is that people will stop trying to use them for doing huge syncs since we have our lovely Export API for that.

Next, you may remember that we launched ACL/multi-user accounts/permissions/whatever support back in v8. Since then, however, only logins with the admin/owner permissions actually had API access. Version 2 changes that in a number of ways. First, where our OAuth2 implementation would not allow logins with viewer/editor/manager roles to "log in," that restriction is being removed. API Keys/tokens issued will now be tied to those logins (so we know what permission level they have), and API methods have been limited to appropriate roles based on how they work in the app. Which is to say: a viewer has access to reports/opened , but most certainly not to lists/members or campaigns/create. Second, there are a number of new users/* methods that will allow admins and owners to manage the logins with access to an account via the API should someone want to do that. So, whee.

Another pain point that required some significant changes was how our "unique" IDs worked for accessing/changing/etc. list member data. Basically, it’s always been unique to an email address, not to your list member, so they were useless for scenarios where you wanted something truly unique to a member that wouldn’t change (like when changing email addresses). We’ve always had something that would work for that, but it’s now clearly exposed, and the majority of places an email address or unique ID would previously have passed will now accept that. You’ll also find we’ve explicitly defined a standard struct for passing that data. The last big win here was redoing how merge_vars are passed in lists/batch-subscribe and lists/subscribe. Essentially what you’ll find is that lists/batch-subscribe takes an incredibly more sane data structure for defining those things and that it, list/subscribe, and list/update-member get these nifty modifications:

  • GROUPINGS will now be passed in a pretty array/struct setup—no more crazy, potentially escaped comma delimited strings. That’s a general change we’re aiming for across the API.
  • MC_NOTES will now allow passing multiple notes to add/change/delete/etc. rather than a single one. Previously multiples could be done in one call via listBatchSubscribe, but this makes that slightly easier.

There you have it: a pretty comprehensive overview of how and why we’ve re-architected the API and what to expect. API v2 is now fully available, and we’ll start posting detailed change logs for methods and such soon. As with any code changes this massive and completely new, we’ve slapped a nice little beta type tag on it and will not immediately commit to anything changing for a few weeks as people poke and prod it and yell at us about things we missed or screwed up. That said, it’s certainly been going through tons of internal testing, so we’re not anticipating too much of that, though we’d asinine to believe it won’t happen. If you’d like to follow along, subscribe to the API Announcement Group, the API Support Group, or keep an eye on our @MailChimp_API twitter account for updates and announcements about v2.

And in case you missed it, RTFM!