Last week was a big one for our dev team. We released a large chunk of code in the long-standing PHP to Python conversion process. With that release, most (probably greater than 95%) of the screens that our users see are now generated through our Python application. We believe that these new Python-powered pages are faster to render and respond to user action. Certainly, they are a strong foundation on which we will begin to innovate shortly.
The release last week was unfortunately not the smoothest that we’ve ever had. As developers, we strive to make all of our releases painless for our users, our support team and ourselves. However, we missed a few key bits last week, and I think it’s instructive to look at one of them through the lens of the PHP to Python project.
One of the major issues affected our users’ ability to send emails. Clearly, in our business, keeping folks from sending emails effectively is a serious concern. Of course, the team jumped in to fix it as quickly as possible.
We’ve been around for a while
What’s interesting to look at is that the problem really related back to the data stored in our system. Some of our accounts have been with us for quite a long time, and they may have data that’s up to eight years old in some cases. That data might have been written to the database by one codebase, processed two years ago by a version of the PHP codebase, and we’re attempting to display it via a third codebase written in Python. To put that into perspective, we have data in our database that is older than Facebook.
Language differences aside, dealing with legacy data processed over time by varying systems is bound to be an issue. In this particular case, moving from a weakly typed language like PHP to a strongly typed language like Python exposed some of the warts in our legacy data. This was the primary issue that affected our users’ ability to send. The Python code came across some data that should have been a timestamp data type (and was such in many accounts) but was instead a string in a few accounts. While PHP might have just ignored this difference, Python is much less forgiving, which resulted in our users encountering an error. A similar bug occurred around the signup form selection part of campaign send as well. In that case, the differences in how each language handles the concept of a null was at fault.
Couldn’t you test for this?
These types of bugs are hard to test for with functional tests because any test data we are creating is more than likely going to have the correct data type. It’s only when we put the code into production and legacy data flows through that our oversight becomes apparent. Unit testing is more likely to catch this kind of bug, but we aren’t quite to the point where our unit tests are covering as much of the Python code that generates the UI as we might like.
Will it get better?
Our new API code, which is becoming the platform on which an even newer version of the app is being built, is chock-full of hundreds of unit tests that are helping us to avoid this very situation. I’m sure we’ll have a post soon about the success that a test-driven style has generated for us with the development of our API platform.
While these errors are problematic, fixing them in the Python code is truly progress. We are not only making the code more resilient, but we’re also fixing years of workarounds and reliances on quirks of PHP that contributed to this data inconsistency. It’s sometimes painful for us — and unfortunately so for our customers — but this is progress. Hopefully, you’ll start to see the progress that’s happening behind-the-scenes bubbling up to the surface very soon.