EmmaTech

  • Emma Home
  • Emma Blog
  • Job Openings
  • RSS

Don’t fork, factor

Robert Church Robert Church 6 Mar 2012 objects programming refactoring 2 Comments

There comes a time in the life­cy­cle of any sig­nif­i­cant soft­ware project when devel­op­ers will be tempted to fork the sys­tem into two dif­fer­ent prod­ucts. You might, for instance, find your­self with a cus­tomer whose require­ments seem at odds with the direc­tion of your prod­uct. You might be fac­ing some seem­ingly unre­solv­able incom­pat­i­bil­ity of sys­tem components.

At Emma, we’re in the process of rolling out a new API that will allow our cus­tomers to man­age their data in cool new ways. As part of this process, we’re migrat­ing users to a new data­base schema and adapt­ing our soft­ware to talk to the new API. In decid­ing how quickly we can move our sys­tems to sup­port both API users and those who still use our legacy data­base, we’ve faced this fork­ing ques­tion a num­ber of times.

Forking can look allur­ing. A sys­tem that only has to sup­port one kind of user sure looks sim­pler than one that has to fig­ure out what kind of user it’s work­ing for and then do the right thing. Failure to fork with­out good fac­tor­ing can result in code with ‘if’ state­ments strewn all over the place, each of them adding com­plex­ity and con­fu­sion for maintainers.

On the other hand, fork­ing pro­duces two sys­tems, both of which need to be main­tained. You’ll prob­a­bly find your­self mak­ing the same change in more than one place. It also com­pli­cates deploy­ment, and you may have to work out schemes to direct users to the right system.

Adding fea­tures to our inter­nal user admin tool to sup­port API users gave us an oppor­tu­nity to try the approach of fac­tor­ing in code that would work with legacy accounts and new accounts at the same time. We were pleased to find that we could achieve that by mak­ing a small num­ber of low risk changes to the admin tool, adding a layer of indi­rec­tion. The code actu­ally looks bet­ter than it did before the change.

Our sup­port team uses the admin tool to, among other things, restore deleted mem­bers to mail­ing lists. The old imple­men­ta­tion of this con­nects directly to the data­base and issues some SQL queries to move some records around. The new way is to issue an HTTP request to the REST API.

The code in the admin tool isn’t super nice-looking. Still, it’s served us well for many years, and we wanted to sup­port the new API with­out chang­ing the older code extensively.

So, we have some old code that looks about like this:


   class Controller(…):
        def undelete_members(account_id, user, member_ids):
            # 50 or so lines of code to execute some SQL
            # queries inside of the controller method.
            # Here be dragons.
            …

We use a sim­ple fac­tory method so that the behav­ior can be changed on a per-account basis and then move the oper­a­tions into a pair of classes that abstract the oper­a­tions on the members.


    class Controller(…):
        def undelete_members(account_id, user, member_ids):
            impl = get_backend(account_id, user)
            impl.undelete_members(member_ids)

        def get_backend(account_id, user):
            # Just the one 'if' statement
           backend_class = APIBackend \
               if is_an_api_account(account_id, user) \
               else LegacyBackend

           return backend_class(account_id, user)

    class APIBackend(object):

        def undelete_members(self, member_ids):
            res = self.call_api(
                self.account_id,
                '/members/undelete',
                method='POST',
                data=member_ids)

       class LegacyBackend(object):

           def undelete_members(self, member_ids):
               # The same 50 or so lines of code to execute
               # some SQL queries, moved out of the
               # controller. Let sleeping dragons lie.
               …

One of the inter­est­ing things about this is that, in addi­tion to wedg­ing some nice new API code into the admin tool, the legacy code gets bet­ter too. While it’s mostly untouched, so we don’t have to worry too much about intro­duc­ing sub­tle bugs, we now have a struc­ture with greater sep­a­ra­tion of con­cerns. The part of the code that deals with the incom­ing HTTP request is sep­a­rated from the part of the code that man­ages the mem­ber data. Also, in LegacyBackend, you see some­thing that looks kind of like a prim­i­tive object model. Better still, the APIBackend class looks like the begin­nings of some­thing that could be used as a client that sim­pli­fies access to the REST API in our other sys­tems, or for end users who want to man­age their own accounts in Python scripts. Everybody wins.

The moral of the story is that, with a lit­tle bit of basic old fash­ioned object inge­nu­ity, you might be able to stretch your system’s behav­ior a lot fur­ther than you think, sav­ing your­self from sup­port­ing and man­ag­ing two dif­fer­ent prod­ucts, while mak­ing your code more flex­i­ble and more maintainable.

A new platform for Emma

Matt Thackston Matt Thackston 15 Nov 2011 api 3 Comments

We’ve been shar­ing details about our API for a while now here on Emma Tech, and we’re excited to announce that we’re ready to roll it out for cus­tomers. Read all about its release — and the new Emma plat­form — on the Emma blog. If you have any ques­tions, please let us know.

Using IIFEs in JavaScript to control variable scope

Because creating too many global variables is iffy

Josh Mock Josh Mock 11 Oct 2011 javascript 3 Comments

On a recent project, a few of us have been chal­lenged to take our JavaScript skills to a new level. We’ve even learned a few new tricks from a group of out­side devel­op­ers that we’re work­ing with who spe­cial­ize in JavaScript.

One such trick is using immediately-invoked func­tion expres­sions (IIFEs), which are often used to take advan­tage of JavaScript’s scope rules to fun­nel in cer­tain global vari­ables and avoid cre­at­ing too many of your own globals.

The basic con­cept of an IIFE (lov­ingly pro­nounced “iffy,” occa­sion­ally referred to as a self-executing or self-invoked anony­mous func­tion) is to wrap a por­tion of JavaScript in an anony­mous func­tion that gets called immediately:

(function () {
     // your code here
})();

Wrapping the func­tion in paren­the­ses turns it into an expres­sion rather than a dec­la­ra­tion. The empty set of paren­the­ses that fol­lows tells the inter­preter to run that expres­sion as a func­tion. It is a more com­pact and anony­mous equiv­a­lent to the following:

var foo = (function () {
     // your code here
});
foo();

Take con­trol of your globals

Writing code like this might seem redun­dant until you give it more sub­stance, of course. Let’s use an exam­ple I recently came across in a web­site I was work­ing on at home.

This web­site, unfor­tu­nately, has depen­den­cies on both the jQuery and Prototype libraries. Both libraries use $ as a global vari­able; Prototype as a short­ened way to access document.getElementById(), and jQuery for, well, pretty much every­thing it does. Because of this, the value of $ changes depend­ing on which library is included last. In this case, Prototype was the last-included library, so $ belongs to Prototype.

Now if I want to use jQuery’s func­tions, instead of using $( ... ), I have to use jQuery( ... ). This is a lit­tle annoy­ing, what with jQuery being the more widely-used library that more devel­op­ers are famil­iar with. To fix this, I wrapped my code in an IIFE and passed it the jQuery global variable:

(function ($) {
     $(document).ready(function () {
          // my code
     });
)(jQuery);

This means that, no mat­ter what the global vari­able $ means out­side my IIFE, within it I’m explic­itly telling it to run jQuery func­tions. I could just as eas­ily assign jQuery to any other unused vari­able and it would have the same effect:

(function (p) {
     p(document).ready(function () {
          // my code
     });
)(jQuery);

Keep your vari­ables to yourself

Aside from fun­nel­ing in global vari­ables to what­ever val­ues I want, I can also use IIFEs to pre­vent vari­ables from being added to the global scope.

For exam­ple, you might run the fol­low­ing code on a web page out­side of any function:

var x = "hello";
alert(x); // alert dialog with the text "hello"

By doing this, x is now a global vari­able. If x was assigned else­where before this script was called, it’s now reas­signed to the value "hello". Not always ideal. To avoid this with­out hav­ing to name and call a new func­tion, wrap your code in an IIFE. This keeps your vari­ables within a smaller scope and still runs the lines immediately:

(function() {
     var x = "hello";
     alert(x); // alert dialog with the text "hello"
})();

Now, x is only assigned to the value "hello" for the dura­tion of the IIFE. If x was assigned glob­ally before the IIFE, it would keep that value after the IIFE runs:

var x = "foo";

(function() {
     var x = "hello";
     alert(x); // alert dialog with the text "hello"
})();

alert(x); // alert dialog with the text "foo"

As you’ve seen, the IIFE con­ven­tion can be a great stan­dard prac­tice when writ­ing your own libraries and plu­g­ins, when extend­ing JavaScript where you may not be aware of every con­text in which cer­tain vari­able names are used and to pre­vent too much data from creep­ing its way into the global scope.

Making a match: Salesforce and Emma

A few notes on our upcoming Salesforce integration

Alex Ezell Alex Ezell 29 Sep 2011 api integration salesforce 10 Comments

Salesforce is clearly a jug­ger­naut. Businesses from the small­est retail bak­ery to Fortune 500 behe­moths are using it to track cus­tomers, sales and a thou­sand other things. It’s long been a request from Emma’s cus­tomers to pro­vide a sim­ple two-way sync with Salesforce. And while we’ve been plan­ning to build this inte­gra­tion for some time, it’s just recently that we’ve made major head­way on this work.

Salesforce inte­gra­tion is one of the first major projects we’re build­ing using our new API (out­side of our own appli­ca­tion, of course). Not one sin­gle piece of data is being moved from Emma to Salesforce or back that doesn’t use a call that’s pub­licly acces­si­ble via our new API. This is eat­ing our own dog food in the same way build­ing our appli­ca­tion on top of our API is. We’ve learned a few things about our API and Salesforce so far, and I’d like to share the progress with you.

Salesforce is complicated

There are vary­ing lev­els of fea­tures depend­ing on what plan an orga­ni­za­tion uses. As such, there are lim­its and con­straints at every turn regard­ing API usage, object and field access, user access and much more. It becomes a byzan­tine and gru­el­ing task to fig­ure out not how but if you can do what the Salesforce doc­u­men­ta­tion says you can do. That com­plex­ity also yields great power and flex­i­bil­ity; once you deter­mine if you can do some­thing, fig­ur­ing out how is yet another chal­lenge. It’s one model, but not one that Emma will likely mir­ror. Instead, we’ve opted for the “all you can eat” sort of model with our API. If you have an Emma account, then you can use any and all of the API.

Webhooks are easy

Webhooks, which we ini­tially added as a bit of a fluke, are now the lynch­pin of keep­ing the two sys­tems synced. Anytime an event occurs — delet­ing a mem­ber, adding a mem­ber, send­ing a mail­ing, open­ing a mail­ing or click­ing a link — web­hooks are used to let Salesforce know that it should update its data. In this way, we don’t have to use a sched­uled job or man­ual update process to get mem­ber data and response data into Salesforce in a timely man­ner. Webhooks have negated the need to build a mid­dle­ware sys­tem that would have col­lected data from the two sys­tems and then shuf­fled it around where it needed to be.

Documentation makes the med­i­cine go down

The doc­u­men­ta­tion largely works. We’ve uncov­ered a few incon­sis­ten­cies here and there, but using Sphinx and a shared con­ven­tion of doc­strings for every pub­lic method has kept the doc­u­men­ta­tion cor­rect and com­pre­hen­sive. By and large, the folks work­ing on the Salesforce appli­ca­tion have been able to use it to answer most of their ques­tions about what the API can do. This is a sig­nif­i­cant dis­cov­ery as we strive to make the doc­u­men­ta­tion as clear and as help­ful as pos­si­ble. This is some­thing we’ve come to appre­ci­ate about projects like Django and Flask, and while we might not emu­late them exactly, we’re look­ing to share the same spirit. We have some work to do with regards to sam­ple client code, more exam­ples of typ­i­cal work­flows and what kinds of things might go wrong or be con­fus­ing. We are also work­ing on ways to keep the pub­lished doc­u­men­ta­tion more in sync with the auto-generated documentation.

Where’s the beef?

So, what are we build­ing? If you’ve read this far, you’re prob­a­bly inter­ested to know what exactly our Salesforce inte­gra­tion will do.

Here’s a quick overview:

After installing the Emma appli­ca­tion from the AppExchange, the user will pro­vide their Emma API cre­den­tials and ver­ify those details. To start, there will be a quick wiz­ard type of process to per­form an ini­tial sync of the groups and mem­bers from Emma which you might wish to pull into Salesforce. You can also cre­ate new Emma groups using the results of reports you’ve gen­er­ated in Salesforce. After that, any time you make updates in either sys­tem, those changes will be mir­rored in the other system.

As for response data, the opens, clicks and shares that Emma tracks will be auto­mat­i­cally updated in Salesforce for every mem­ber that has been synced with Emma. There are a few places it will be vis­i­ble, but the most obvi­ous will be in the Contact History sec­tion of each contact’s record. You’ll be able to use this Emma response data in reports and dash­boards just as you would any other piece of data in Salesforce. We hope that after the ini­tial sync and with the excep­tion of cre­at­ing new groups from reports, you won’t really have to worry about the Emma sync-up. You’ll just be able to use the response data gen­er­ated by your Emma mail­ings and know that all your mem­ber data is synced between the sys­tems. Like they say on those infomer­cials, “Set it and for­get it.”

A sneak peek at an upcoming feature

Get ready for Social Posting, plus how we're structuring the Emma teams to get projects done

Donna Smith Donna Smith 15 Sep 2011 api Facebook github OAuth Python Redis Social Posting Twitter UX 0 Comments

Re-structuring project teams

As we pre­pare for a fall/winter release sched­ule that’s very ambi­tious, we’ve been try­ing some­thing new on the tech­nol­ogy team at Emma. Rather than work­ing on one project at a time and break­ing it into indi­vid­ual tasks, we’ve split our­selves into small cross-departmental teams, and we’re work­ing on sev­eral projects all at once. It sounds fairly straight­for­ward, but it’s a depar­ture from what we’ve done in the past — and it’s improv­ing our speed and communication.

Most of these small teams include one or two devel­op­ers, one or two UXers, one prod­uct owner and one project man­ager. We’ve kept the teams small so each indi­vid­ual is involved in the plan­ning process and dis­cov­ery phase. Turns out, this is a very good thing. It’s helped us set fea­si­ble goals, and it’s given us a true sense of own­er­ship of our projects. A few of the teams, mine included, even lugged their mon­i­tors and lap­tops up to the third floor of the Nashville office so they could sit together while col­lab­o­rat­ing. (Typically, devel­op­ers and UXers are on sep­a­rate floors.)

A sneak peek at a com­ing feature

Let’s get to the good stuff. You want to know which fea­tures are com­ing up, right?

The project I’ve been work­ing on for the past few months is Social Posting — what we’re call­ing SPX in-house (the ‘X’ is for EXTREME!) — and it allows users to share their mail­ing on their social net­works, like Twitter and Facebook, right after they’ve sent it out to their email audi­ence. Quite sim­ply, you’re get­ting your cam­paign in front of your email audi­ence, Twitter fol­low­ers and Facebook fans all at once — and right within Emma.

Let’s take a peek, cour­tesy of Social Posting’s res­i­dent UXer, Trey. Once you’ve sent your cam­paign, you’ll see the option to post it socially. Just add a com­ment in the dia­logue box, and choose when to share.

Post to Facebook

Post to Twitter

How we did it

In order to build this, we needed to use the Facebook and Twitter APIs, and we needed per­mis­sion to access users’ Facebook and Twitter accounts. For this, we used OAuth. Specifically, the Twitter doc­u­men­ta­tion rec­om­mended that we use Brosner’s python-oauth2. This inter­face does most of the heavy lift­ing that’s needed when using OAuth to authen­ti­cate with Twitter.

Once we fin­ished with the OAuth dance for both Facebook and Twitter, we were given an access token that we stored for later use. That token will allow us to post cam­paigns to social net­works from Emma on the user’s behalf, with­out hav­ing to leave Emma or open a new browser tab to Facebook or Twitter.

Next, we needed to fig­ure out how to shorten the cam­paign web­view links so that our cus­tomers will be able share the URL plus more text, like the sub­ject line and a com­ment, on Twitter. All within 140 char­ac­ters or less, of course. We thought about rolling our own URL short­ener or using a third party short­ener API like goo.gl or bit.ly. We also knew about Twitter’s short­ener (t.co), but we couldn’t find much doc­u­men­ta­tion for API use. Turns out they hadn’t rolled it out to the API users yet. Then, as if on cue, Twitter rolled out the URL short­ener to the API right when we needed it. All we had to do was turn on the link short­ener in our app set­tings or send the right para­me­ter with each API call.

But, we weren’t done yet. We encoun­tered another hur­dle: how would we allow users to sched­ule Social Posting? We wanted to give our forward-thinking cus­tomers a way to post cam­paigns to their social net­works at 11:00 pm or 2:00 am — or when­ever they wanted — with­out hav­ing to turn on their com­puter. Okay, maybe 2:00 am is a lit­tle unrea­son­able, but it’s not unrea­son­able to think that some­one would want to sched­ule in advance, say, if they’re going on vaca­tion next week.

To do this, we set up a Redis queue cou­pled with Tantu to pub­lish these posts at the sched­uled times. Tantu is a Python work queue that is fed by Redis, and we wrote it in-house. It’s pretty cool because it lets you have con­cur­rent threaded work­ers so it can han­dle your workload.

We’re pretty excited about unveil­ing Social Posting to our cus­tomers. Stay tuned for more news here and over on the main Emma blog, and let us know if you have any questions.

Usability testing at Emma

How we plan, execute and learn from usability tests

Geoff Alday Geoff Alday 9 Sep 2011 usability testing UX 0 Comments

There’s a lot of talk these days about being a user expe­ri­ence designer — what it means, who does it, who doesn’t do it, how to break into the field. This kind of dia­logue is illu­mi­nat­ing at times — con­tentious at oth­ers — and, hon­estly, I’m not a big fan of talk­ing about it like that (I think it can get quite silly actu­ally). But I do like talk­ing about the dif­fer­ent things you typ­i­cally find within the user expe­ri­ence field and how to apply them to your work.

Usability test­ing is one of the most inter­est­ing aspects of our field. It’s some­thing we incor­po­rate in our work at Emma, but haven’t shared much about until now. In the com­ing months, we’ll be talk­ing more about usabil­ity test­ing here and over on the main Emma blog.

Today, I’ll share how we run our usabil­ity tests at Emma. We don’t have a fancy lab or any­thing (not that there’s any­thing wrong with that), yet we have a fairly estab­lished process around test­ing new prod­uct fea­tures. It’s a mix­ture of some typ­i­cal usabil­ity test­ing pro­to­col and some guerilla tech­niques (we’re such rebels).

Have some­thing to test

It may be a bit obvi­ous to men­tion this, but we need some­thing inter­ac­tive to test before we get started. This can be a pro­to­type, func­tional front-end code or even an early devel­op­ment ver­sion of a fea­ture. We pre­fer to show the new fea­ture in con­text. But if we don’t have that, we can fake it with a lit­tle dig­i­tal cam­ou­flage. We’ll take a screen­shot of the UI where the new fea­ture will be imple­mented, clear out every­thing we don’t need in Photoshop, and use that as a back­ground image for the pro­to­type. Not every­thing on the screen works, but at least it looks closer to the final product.

Know what you want to test

There are usu­ally spe­cific tasks par­tic­i­pants need to be able to accom­plish with a new fea­ture. We doc­u­ment these tasks — from very small tasks to more com­plex ones — and write non-leading ques­tions to see if par­tic­i­pants can accom­plish them. We also write a few sen­tences that describe the con­text of use and how they might get to the fea­ture if it’s part of a larger work­flow. (Again, con­text is so important!)

Prep the test­ing environment

We make sure the envi­ron­ment we’re test­ing in has every­thing we need to eas­ily reset after each ses­sion. If it’s a pro­to­type or front-end code, that means we refresh the browser after each ses­sion. If it’s a devel­op­ment ver­sion, we might need to clear out data on the server that was cre­ated dur­ing the pre­vi­ous ses­sion. We want to make sure each par­tic­i­pant starts with the same expe­ri­ence. It lends struc­ture to the test — and helps us get reli­able results.

Record each session

We uti­lize Silverback to cap­ture video and audio. In short, we want to see and hear how the par­tic­i­pant inter­acts with the fea­ture. If you haven’t used Silverback before, it’s pretty straight­for­ward. We load up the fea­ture we’re test­ing, start Silverback, cre­ate a new project, cre­ate a new ses­sion, cen­ter our mug for the cam­era, and hit the space bar (this will make sense if you watch the demo). We hold usabil­ity tests in a quiet meet­ing room at our office, but the participant’s work­space would also serve well. If those aren’t options, just find a spot some­where that’s easy to access and distraction-free. After all, this is a guerilla usabil­ity test. No mercy.

Find par­tic­i­pants

So, you might won­der, how do we find our vic­tims, er, par­tic­i­pants? We pre­fer to test with at least five peo­ple, and we don’t worry too much about recruit­ing the exact demographic/audience for each fea­ture. It’s best to select par­tic­i­pants of vary­ing ages, gen­ders and tech­ni­cal exper­tise. (You might call this guerilla sam­pling.) After all, we’re look­ing for qual­i­ta­tive infor­ma­tion here, and it mat­ters more that par­tic­i­pants are will­ing to dig in and give us hon­est feed­back. Once we select the par­tic­i­pants, we sched­ule meet­ing times.

Running the test

When our par­tic­i­pant arrives, we intro­duce the usabil­ity test­ing process: what we will be doing and that we’ll be record­ing the ses­sion (there’s a dis­clo­sure doc­u­ment and all). Then, we read a state­ment of informed con­sent that out­lines their rights as a par­tic­i­pant of the test. To be sure, we’re test­ing the inter­face and not them; there are no right or wrong answers. We encour­age par­tic­i­pants to talk out loud about what they’re think­ing dur­ing the test to help us bet­ter under­stand the moti­va­tions for their inter­ac­tions with the fea­ture. Next, we explain the con­text for the fea­ture to help them under­stand how they arrived at the screen we’re showing.

When they’re ready, we’re ready to roll — it’s time to hit the space­bar to start record­ing. We go through the task-related ques­tions and give par­tic­i­pants time to play around and com­plete the tasks. This can take any­where from 15 min­utes to almost an hour, but usu­ally no more, and at the end of the tasks, we ask some more gen­eral ques­tions about the inter­face. We want to know their impres­sions about how it works and looks, and we encour­age them to share any other thoughts and ask ques­tions. Finally, we con­clude the test.

Rinse. Repeat.

The results

After all the ses­sions are com­pleted, it’s time to see if any usabil­ity issues emerged (they always do). We export movies of each par­tic­i­pant and watch them one at a time while tak­ing copi­ous notes. While we usu­ally notice some fairly obvi­ous issues dur­ing the ses­sions, it isn’t until we watch the recorded ses­sions that we notice the more sub­tle ones.

Then, we trans­fer the notes about the issues that emerged into a spread­sheet (the AK-47 of usabil­ity test­ing). It helps us quan­tify the sim­i­lar­i­ties and dif­fer­ences between the participants. From there, we pri­or­i­tize and design solu­tions where appro­pri­ate. Finally, we put all of our find­ings into a usabil­ity report to share with project stake­hold­ers (along with shar­ing some of the “best of” por­tions of the ses­sions videos). It’s fun — even sur­pris­ing — for stake­hold­ers to see how par­tic­i­pants use the fea­tures we’ve designed.

The usabil­ity test­ing pro­to­col we use is based mostly on con­cepts from the excel­lent book, Observing the User Experience: A Practitioner’s Guide to User Research. A heavy, yet highly rec­om­mended read.

I’d love to hear how you plan and imple­ment usabil­ity tests at your orga­ni­za­tion. Feel free to share in the com­ments below, or ask any ques­tions about usabil­ity test­ing at Emma. We’ll answer your ques­tions in upcom­ing posts.

Viva la revolution!

Adding webhooks to your application

It's easier than you might think -- and lets your users have a bit of control

Pete Keen Pete Keen 30 Aug 2011 api 0 Comments

We’ve been work­ing hard over the last few months to release a new ver­sion of Emma, one built on top of an entirely new plat­form that will help our cus­tomers get the most out of their sub­scrip­tion. One of the things we recently added was a suite of web­hooks, which will allow Emma to tell API users when an event occurs with­out them hav­ing to poll us continually.

For those of you who don’t know what a web­hook is, it’s basi­cally a jazzed up HTTP POST that a web appli­ca­tion sends to a URL that a user has reg­is­tered. Several of the pop­u­lar code and project host­ing sites allow users to set up web­hooks that get called when some­one makes a com­mit. The Emma web­hooks are not much dif­fer­ent, but we poten­tially have a much larger suite of events a user will be able to reg­is­ter for.

It turns out that adding web­hooks to your appli­ca­tion is actu­ally pretty triv­ial if you have these bits already:

  • User/API authen­ti­ca­tion
  • An asyn­cro­nous pro­cess­ing sys­tem like Resque or Celery

The bare min­i­mum you need to pro­vide to users is an API they can use to reg­is­ter a new web­hook. These are just your nor­mal every day CRUD APIs that oper­ate on a really sim­ple model that, in SQLAlchemy, would look like this:

class Webhook(Base):
    __tablename__ = 'webhooks'

    webhook_id = Column(BigInteger, primary_key=True)
    user_id = Column(BigInteger, nullable=False,
        ForeignKey('users.user_id'))
    url = Column(Text, nullable=False)

So now you’ve got a bunch of web­hooks. Let’s say our web­hook will fire when we send an email. In a real sys­tem you’d want to be more dis­crim­i­nat­ing, pass­ing around user_id and only fir­ing off the appro­pri­ate hooks.

Where you define your back­ground tasks, you’d do some­thing like this (using Celery):

import requests
import json
from celery import task

@task
def send_webhook(to, contents):
    webhooks = session.query(Webhook)

    for webhook in webhooks:
        data = {'user_id': webhook.user_id,
            'to': to, 'body': contents}
        requests.post(webhook.url,
            data=json.dumps(data))

@task
def send_an_email(to, contents):
    (from, to, msg) = build_email_message(
            to, contents)
    smtplib.SMTP().sendmail(from, to, msg)
    send_webhook.delay(to,
            contents) # fire off the webhook soon

This sam­ple also uses the excel­lent Requests library, which wraps all of the urllib and urllib2 mad­ness in an extremely easy to use library. I highly rec­om­mend it.

Now, our lit­tle appli­ca­tion will send off web­hooks when­ever we send an email. How do our users know that it’s a) from us, and b) for them? Let’s add some HMAC-MD5 signing:

from base64 import b64encode
import hashlib
import hmac

@task
def send_webhook(to, contents):
    webhooks = session.query(Webhook)

    for webhook in webhooks:
        data = json.dumps({
            'user_id': webhook.user_id,
            'to': to,
            'body': contents})

        user = session.query(User).\
            get(webhook.user_id)
        digestor = hmac.new(
            bytes(user.api_key),
            bytes(data),
            hashlib.md5)
        digest = b64encode(digestor.digest())

        requests.post(webhook.url, data,
            headers={"Webhook-Hmac-MD5": digest})

Users can authen­ti­cate the mes­sage by sign­ing the POST request body they get from you using the same method and com­par­ing it to the Webhook-Hmac-MD5 header.

So, web­hooks are pretty cool. They let users do fun, unex­pected things with your app with­out much effort on your part. Using Requests and your favorite back­ground task sys­tem, they’re also almost triv­ial to add to an exist­ing sys­tem. If you’ve got expe­ri­ence with web­hooks, I’d love to hear about it. Comment below to get the dia­logue going.

Yay-PI!

Join us in making a better Emma experience for everyone

Hernán Ciudad Hernán Ciudad 11 Aug 2011 api 4 Comments

Last Wednesday, our new API offi­cially entered pri­vate beta, swing­ing the doors open to early adopters. To pre­pare for the big day, we tech elves were busy get­ting the API to a state where, while still not bug-free, it would be func­tional. And now our adven­tur­ous beta testers are help­ing us iden­tify remain­ing bugs and per­for­mance bottlenecks.

On top of the work to get the inter­face part of our API locked down (view the doc­u­men­ta­tion), we have also been focus­ing on con­vert­ing the appli­ca­tion to use this new API. We are, to use a less-than-appetizing phrase, eat­ing our own dog food. We’ve stan­dard­ized the way data is han­dled by the sys­tem, so that when we build a new fea­ture for the app, we auto­mat­i­cally get an API ver­sion of the fea­ture that we can open to the pub­lic. To those who have used our legacy web ser­vices offer­ing, and have had to work with the lim­ited func­tion­al­ity it pro­vides, you under­stand how impor­tant it is to ensure that all users of Emma, be they via the API or the UI, have access to the most com­plete set of tools we offer. This also means the API calls our cus­tomers have access to are the same calls my cowork­ers and I must develop against. So, if a call doesn’t return cor­rect data, or isn’t per­for­mant, we will feel the headaches just as you do. And noth­ing moti­vates a devel­oper to improve their code faster than mak­ing them actu­ally use it.

I hear some of you ask­ing, “I only use the web inter­face and don’t really care about APIs or build­ing my own Emma at home, so what’s it in for me?” Quite a bit, actu­ally. However, if we’ve done our job right, not much that you’ll see. Not ini­tially, at least. Redesigning the archi­tec­ture has taught us a lot about the chal­lenges of the exist­ing sys­tem, and the new sys­tem avoids those alto­gether. From the way mem­bers and groups are stored to the way emails are sent and tracked, the new sys­tems focuses on sim­plic­ity and exten­si­bil­ity. So, as we start to roll out the new Emma to all of our cus­tomers, you’ll go on using Emma just as you always do, only notic­ing that the app is more respon­sive, and that your mail­ings are deliv­ered even faster. The real fun will hap­pen later as our new sys­tem design will facil­i­tate the addi­tion of new fea­tures, some­thing the cur­rent design makes more dif­fi­cult to accom­plish at the pace we’d like.

So far we’ve only invited about 25 peo­ple into the pri­vate beta, but it is already yield­ing some fan­tas­tic feed­back, uncov­er­ing bugs and giv­ing us a slightly dif­fer­ent point of view on what some of our power users need in order to inte­grate Emma into their own appli­ca­tion. If you’re inter­ested in join­ing the pri­vate beta group and test dri­ving the new API, or the updated Emma built on top of it, please visit the beta tester signup form.

PyOhio was a blast

PHP to Python conversions, Python on an Atari and more

Alex Ezell Alex Ezell 5 Aug 2011 conferences Python 0 Comments

With all the mod­ern means of con­nect­ing peo­ple, I find that a lit­tle face time still goes a very long way. Software devel­op­ers are often labeled as not being ter­ri­bly social, but the giant secret is that we love hang­ing out with each other. To that end, I trav­eled last week to Columbus, OH to attend and speak at PyOhio.

PyOhio is a regional gath­er­ing of Python devel­op­ers and those inter­ested in Python. This year there were nearly 200 atten­dees over the two-day con­fer­ence. I was hon­ored to be cho­sen as a speaker and gave a pre­sen­ta­tion that prob­a­bly seems famil­iar to our reg­u­lar blog read­ers: PHP to Python With No Regrets. A video is forth­com­ing. Update: the video is now online.

We are not alone

The entire con­fer­ence was chock full of qual­ity. However, there were sev­eral ses­sions dur­ing the con­fer­ence that piqued my inter­est. Chief among them was Taavi Burns of FreshBooks who pre­sented on their move from a PHP appli­ca­tion to an inter­nal Python-based API. It was fas­ci­nat­ing to hear that they had some of the exact same rea­sons for mak­ing the switch. Additionally, it turns out we solved some sim­i­lar prob­lems in sim­i­lar ways. This is a great val­i­da­tion for the grand plans we’re cook­ing up here at Emma. Inspired by con­ver­sa­tions with Taavi, I’m work­ing on some­thing with Freshbooks for the Emma app that I hope to be able to talk about more at a later date.

Python on an Atari

Somewhere on the spec­trum from fun to crazy was Jeffrey Armstrong’s pre­sen­ta­tion about bring­ing Python to the Atari. He delin­eated a path to get­ting it work­ing that would have stopped me in my tracks mul­ti­ple times. It’s mostly a hobby pur­suit, but the problem-solving skills on dis­play were impres­sive. The ded­i­ca­tion is envi­able as well. In the end, he was able to cre­ate and draw win­dows on the Atari OS using Python. It was cool to see it all come together.

Why is httplib so painful?

If you’ve never seen Brandon Craig Rhodes give a pre­sen­ta­tion about Python, I implore you to do so at the very next oppor­tu­nity. Brandon is one of the authors of Foundations of Python Network Programming and really knows Python inside and out. On top of his expert knowl­edge, he is an accom­plished speaker with great energy and style that makes the often com­pli­cated con­tent seem easy. He gave sev­eral pre­sen­ta­tions at PyOhio and the con­fer­ence is the bet­ter for it. In par­tic­u­lar, his expla­na­tion of how httplib and urllib2 work and where they might fall down was a wel­come salve to those of us that have had to work with those modules.

Data analy­sis for the acci­den­tal programmer

The last ses­sion I’d like to call out is the Clark C. Evans’s pre­sen­ta­tion about the data­base report­ing toolkit he has cre­ated call HTSQL. He calls it a toolkit for “the acci­den­tal pro­gram­mer” as it allows some­one to ask com­pli­cated busi­ness ques­tions of the data in a data­base with­out learn­ing SQL. I was espe­cially impressed with his team’s ded­i­ca­tion to mak­ing the gen­er­ated SQL queries be per­for­mant. Often, when some­one is try­ing to sim­plify SQL, they do so at the expense of per­for­mance. That doesn’t seem to be the case with HTSQL. I love the idea of giv­ing tech­ni­cal sup­port peo­ple and data ana­lysts a tool that gets them closer to the data­base, but also insu­lates them from the vagaries of SQL. Having tons of easy ways to out­put the results in XML, JSON, HTML or oth­ers is icing on the cake.

Let’s party

On Saturday night, Emma hosted a happy hour at Eddie George’s Bar and Grill, and we got to hang out with a bunch of the folks from the con­fer­ence. It was a great time to catch up about our favorite ses­sions and talk about all of the cool stuff peo­ple are work­ing on using Python. Despite the stereo­type devel­op­ers typ­i­cally get, it’s always fun when a bunch of us get together to talk shop.

PyOhio is impec­ca­bly over­seen by Eric Floehr of Intellovations and an army of fan­tas­tic vol­un­teers. It was an honor to be a pre­sen­ter, and I look for­ward to going back next year. The con­fer­ence attracted folks from as far away from Columbus as Chicago, Nashville, Indianapolis, Philadelphia and Atlanta. If you’re remotely close to Columbus, I’d rec­om­mend attend­ing next year.

Update: The video of the talk is now online here.

The challenges of implementation

Just because you're done coding doesn't mean the project's over

Hernán Ciudad Hernán Ciudad 20 Jul 2011 0 Comments

So, you’ve put the fin­ish­ing touches on your lat­est awe­some appli­ca­tion or fea­ture (as much as code is ever really done) and you’re ready to release it to the wild. Implementation is the fun part, right? The last step in a long process, where you drop your code in a pub­lic place and do the big reveal to your cus­tomers, after which they cheer and shower you with acco­lades. Also, there is cake.

As much as I hope you’re able to achieve cak­ery with the low­est pos­si­ble amount of pain, more often than not imple­men­ta­tion never goes as smoothly as you would like. That genius code that passes all of its unit tests and runs flaw­lessly in the devel­op­ment and QA envi­ron­ments typ­i­cally expe­ri­ences at least a few bumps when you deploy it to production.

We have been work­ing on some pretty sig­nif­i­cant behind-the-scenes changes to Emma (still in early beta), designed to improve per­for­mance and scal­a­bil­ity. Without giv­ing away too much of the secret sauce, the new plat­form is a big change to our archi­tec­ture, made up of mul­ti­ple appli­ca­tions, each built fol­low­ing the prin­ci­ples of test-driven devel­op­ment (dis­cussed in more detail by Kevin). Having a bat­tery of auto­mated unit tests to develop against has helped us ensure that each fea­ture works as expected with­out break­ing oth­ers in the process.

As we reached the end of the pri­mary devel­op­ment phase, it came time to release the var­i­ous appli­ca­tions to their new homes, a task under­taken by me and the infra­struc­ture team. During this step, we expe­ri­enced sev­eral chal­lenges; even though we are very con­fi­dent in the qual­ity of our code and the new appli­ca­tion archi­tec­ture, there are some improve­ments we can still make to our imple­men­ta­tion process.

Challenges

Here are some of the issues we expe­ri­enced dur­ing deploy­ment of the code to production:

Architectural dif­fer­ences

This was a big one. During devel­op­ment, the appli­ca­tions, their depen­den­cies and the data­bases all lived on a sin­gle machine, typ­i­cally the developer’s lap­top. In the QA envi­ron­ment, we gave the data­bases their own machine, but all the appli­ca­tions and depen­den­cies still lived on a sin­gle server. In pro­duc­tion, we break it down to even more machines, each spe­cially built by the infra­struc­ture team to han­dle a spec­i­fied task. Even though we knew this was ulti­mately the goal — and had built in the abil­ity to con­fig­ure the servers to point to each other — there were still a few areas where an appli­ca­tion went look­ing on local­host and got really angry when nobody was lis­ten­ing on the expected port.

Troubleshooting, log­ging and monitoring

Troubleshooting in devel­op­ment is easy when you can redi­rect every sin­gle log and error mes­sage to std­out, but that’s not really an option in pro­duc­tion. In addi­tion, our infra­struc­ture team likes to have log mon­i­tor­ing on all machines. This required us to revisit our log­ging strat­egy, to make sure we pro­vided the infra­struc­ture team with the infor­ma­tion they needed in the for­mat in which they needed it, while still giv­ing the devel­op­ers access to impor­tant trou­bleshoot­ing infor­ma­tion. Not that any­thing ever goes wrong in production.

Minor envi­ron­men­tal differences

Differences in assump­tions about instal­la­tion direc­to­ries led to a few instances where code was deployed to the wrong direc­tory or user, or a direc­tory was miss­ing from the PATH. This caused eas­ily fix­able, yet frus­trat­ing and time-consuming, issues to troubleshoot.

Missing func­tion­al­ity

The unit tests did such a great job set­ting up test envi­ron­ments, that it wasn’t until we moved to pro­duc­tion that I noticed a cou­ple of com­pletely absent pieces of code, namely a script we use to read logs and a deploy script for an appli­ca­tion that is a depen­dency of the pri­mary appli­ca­tion, but also lives on its own. Fortunately, it was easy to build a pro­duc­tion ver­sion of the script based on the test ver­sion, as well as cre­ate a new deploy script. However, it added more time to the imple­men­ta­tion phase that we hadn’t planned for.

How to avoid them in the future

None of the issues we ran into dur­ing roll­out were major flaws in the code or even our process. However, they led to delays in mak­ing the new code avail­able to our early testers (in this case, my co-workers) and could be some­what avoided in the future with a few enhance­ments to the process.

Budget time to han­dle the unexpected

More often than not, you will run into issues you just don’t expect. Even though you can’t cre­ate a detailed plan to han­dle those issues, you can still bud­get the time to get them resolved, help­ing to ensure the project meets its dead­lines. We have recently tweaked our project man­age­ment guide­lines so that roll­out starts near the begin­ning of a project, run­ning con­cur­rently with the rest of the project phases. The ini­tial roll­out efforts might be lit­tle more than deploy­ing the basic shell of an appli­ca­tion to a test envi­ron­ment. However, the ear­lier you start test­ing dif­fer­ent deploy­ment sce­nar­ios, the sooner you will be able to per­fect the process and rec­og­nize any fail­ure points or holes in your appli­ca­tion architecture.

Get your code run­ning in an envi­ron­ment that mir­rors your pro­duc­tion environment

As men­tioned before, your appli­ca­tion might score 100% with your auto­mated test suite and work per­fectly in a test envi­ron­ment, but until you put that appli­ca­tion in the envi­ron­ment where it will be run­ning when your cus­tomers access it, you haven’t really tested it. The best way to do this is to put it into your pro­duc­tion envi­ron­ment; how­ever, that is often not going to be an option. Either pro­duc­tion will be busy serv­ing up the cur­rent ver­sion of your appli­ca­tion to your cus­tomers, or the envi­ron­ment will still be under con­struc­tion, either await­ing new hard­ware or the resources to build it. This is where the magic of vir­tual machines comes in. If the ulti­mate goal is to have the com­po­nent parts of your appli­ca­tion run­ning on sev­eral machines, a sin­gle server divvied up into mul­ti­ple vir­tual machines will give you a much more accu­rate test than just run­ning every­thing on one phys­i­cal machine. Performance won’t be the best, but, for the pur­poses of this exer­cise, the point is to make sure all of the sep­a­rate com­po­nents can talk to each and func­tion as expected when the archi­tec­ture is ini­tially sketched out on a white­board. VMWare is a pop­u­lar tool around Emma. However, if you’re look­ing for some­thing a bit more open source and auto­mated, VirtualBox and Vagrant can be used to eas­ily build and re-build whole vir­tual envi­ron­ments in which to run your application.

Become friends with your infra­struc­ture team

Our infra­struc­ture team at Emma is made up of some great peo­ple, so I would want to be friends with them regard­less of my self­ish tech­ni­cal needs. However, on a purely pro­fes­sional level, we’re mak­ing sure to involve them as early as pos­si­ble on all future projects to ensure we are get­ting their insight into the best prac­tices for appli­ca­tion archi­tec­ture, as well as hav­ing them inform us of any­thing they might need in order to prop­erly mon­i­tor the appli­ca­tion or build out the hard­ware (vir­tual or oth­er­wise). We are also coor­di­nat­ing sched­ules dur­ing crit­i­cal points in the roll­out process so that we can call on each other to assist in resolv­ing any issues that pop up with­out hav­ing to deal with lag due to wait­ing for a response.

The issues I listed above are chal­leng­ing, but not over­whelm­ing, and I believe we’ll see some improve­ments in our process with the efforts we’ve under­taken to min­i­mize their impact on our abil­ity to build and deploy cool new fea­tures. A lit­tle patience and plan­ning can go a long way to mak­ing our next round of fea­ture releases even smoother than this one. Does any­body else have exam­ples of unex­pected issues they expe­ri­enced while try­ing to deploy a new code­base for the first time?

Next Page »

About Emma Tech

Emma's technology group brings you updates and shenanigans related to all things engineering.

Emma Tech on Twitter

    Follow Emma Tech »
    Help wanted

    • Popular Tags

      Python12 api7 UX5 conferences4 postgres4 workflow4 time4 javascript3 PHP3 jQuery3 tools3 editors2 travel2 server maintenance2 Git2 maintenance windows2 Haml1 Frank1 Ruby1 CSS1 PyCon1 office1 Sass1 downtime1 post‑mortems1 cgit1 books1 Trac1 collaboration1 community1 Twitter1 Facebook1 OAuth1 coding1 cool sites1 Redis1 github1 objects programming refactoring1 integration1 salesforce1 usability testing1 Social Posting1 music1 productivity1 bugs1 TextExpander1 san francisco1 Convore1 Vim1 releases1 legacy data1 HTML1 reading1 Django1 PgCon1 testing1 TDD1

    Emma is a member of the Email Sender & Provider Coalition and the Messaging Anti-Abuse Working Group.

    Copyright © 2003 - 2012 Emma.
    All rights reserved.

    • Get Emma's Newsletter
    • Visit the Emma Blog
    • @emmaemailtech on Twitter
    • @emmaemail on Twitter
    • Emma on Facebook

    Emma's email marketing makes communicating simple and stylish.
    Inquire now for more details.