Metric Driven Design (AKA: Applying the Scientific Method)
There has been a lot of discussion lately about metrics, metric driven design, data visualization, machine learning, etc. These topics converge in interesting ways and I wanted to draw attention to some good talks in this space (and of course rant a little).
The first one is a talk about managing complex systems based on metrics given by Coda Hale of Yammer. This talk is interesting for a few reasons. The first has less to do with the content than it does with something that always bothers me about the state of the tech industry. This is a fantastic talk; funny, articulate, and with concepts laid out that every Dev, DevOps, and QA person should know about. Concepts that I have seen first hand are completely foreign to a lot of folks that should know better. The talk is free. The talk looks like it is being given it in a tiny conference room to an audience likely lured in by the promise of free pizza. The youtube video has 814 views. I’ll leave it at that.
I’ll highlight a few of the points Coda makes that I think are especially important. We (folks who write code) get paid for what that code does in production, not for writing the code itself. This is similar to the quality topic - it is the product that counts, not the work that goes in to it. So, how do we know if what we put in production is working as it should?
Complex systems are, well, complex. Even in simple cases you’ll likely need to instrument and measure a lot of different things. You need to be smart about what you measure and how you measure it. The idea here is that the map is not the territory but the better you make the map the easier your life will be.
A misleading map is worse than no map at all and like it or not we all have maps or mental models of the systems we work with so I’d argue that there really isn’t a “no map” scenario. More than I’d like to admit (or remember) I’ve spent long days and nights working to tune or fix a component that was misbehaving only to find that there was something else in the chain that was really causing the problem. I didn’t have a good enough map.
I’ll definitely be checking out the metrics framework they put out, if for no other reason than to irritate people with jokes about Scala.
The second place I’ve been hearing a lot of discussion around this come up is with Metric Driven Design. To be honest, the first time I heard about this I thought, “Uh, Yeah, so you’re applying the scientific method to application design.” It sounds pretty obvious. And it is. And surprisingly, it isn’t terribly intuitive to many people.
Stephanie Kaiser at Wooga has an interview and a talk where she discusses how they make Monster World easy to use and super addicting. And unsurprisingly, they use A/B testing and metrics driven design to achieve this. The metrics they choose are a little bit different than what Coda talks about and the approaches are a bit more touchy-feely but the fundamental mechanisms are the same:
Formulate a question: How can I increase user engagement?
Create a hypothesis: Adding a magic power game mechanic will increase user engagement.
Make a Prediction: We’ll increase user engagement (and probably define the key metrics to track that engagement).
Test the hypothesis: Deploy magic powers game mechanic, monitor metrics (A/B, with and without)
Analyze: Yes. (Oh, and do it a lot, and do it in real-time).
Different problems, different fields, same messages. Measure everything you can, design experiments, and use rigor, even about seemingly fuzzy topics. This doesn’t mean you remove the intuition and creativity from the design, it means that when you come up with your next innovation you should test it with real metrics.
Guess what, that new game mechanic you just came up with? That patch to the geolocation service you just made? The decision to hang the shirts side-on instead of facing the customers at the MegaMart? Those decisions are all going to be tested whether you watch or not, and those decisions will be tested by users and potential customers. Do you even know what to watch for or measure? Do you just wait for someone to call at three in the morning when everything falls over? Am I being rhetorical?
There is a lot of good work going on in this space and I think its worth paying attention to.
Science has made us gods even before we are worthy of being men.
Who Owns Quality Anyways?
In a product team the conventional wisdom is that quality is the responsibility of the entire team. Even in complex multi-team development projects an independent test team should be responsible for testing the hardest and most complex parts with the bulk of the quality responsibility falling to the product team. If you couple concept this with the idea that you want small teams (the two pizza rule) then where does this leave the role of a traditional black box or exploratory tester? In short, learning to automate.
Automation over Exploration
In my opinion this is the way the world is heading. Agile methodologies just don’t work very well with long iterations, off cycle testing, and having bugs crop up an iteration (or more) after the story has been delivered. From a Lean perspective, traditional approaches to QA do not ensure quality at the source.
Burn down charts, Kanban boards, and all the other nifty ways of tracking progress fall apart when you start having bugs show up in code that was “finished” two iterations back. This can end with comical approaches to iterative development where you might have say, five development iterations and then three or more testing iterations to try and find all the bugs your team just delivered. Couple this with the fact that manual testing approaches generally stink and you get a pretty ugly picture.
Bottom line, you have to get away from manual testing, even as a safety net (because it isn’t one). For folks that insist that some things can not be tested in an automated fashion - I’d contend that they likely can’t be tested in a reliable fashion manually either.
Well, if everybody is responsible for quality, who is responsible for writing tests? The person delivering the story. And I’m not just talking about unit tests. You’ll certainly have the largest proportion of unit tests, but most stories I’ve seen cannot be encapsulated cleanly in to unit or even integration tests. So, that means that when you deliver a new widget on the screen, you’ll also need to have functional tests that demonstrates that the new widget does what the story says it does. So the developer is responsible for testing, and it’s likely the story owner/creator is responsible for validating that the tests provided indeed exercise the new functionality correctly.
I think there is more to it than this though. I think there is something to be gained by having a member of the team more focused on testing than other responsibilities but I’m not advocating for full time tester role(s) on the product team, especially if that testing is going to be manual.
So I’ve got a typical product team, let’s say 8 members. In a product team we’ll have overlapping skills and capabilities with different team members pitching in based on their skills, desires, and the teams needs. This isn’t a place for hyper-specialization but likely you’ll have a couple folks better at gathering requirements, 4 or 5 better at developing and database work and other sundry tasks. This isn’t an exhaustive list of responsibilities - folks on the team will be acting as ScrumMaster, Architect, Designer, Sys Admin, or Staff Psychologist as needed.
That doesn’t leave a whole lot of folks to focus exclusively on testing - certainly not enough to go with something like a semi-traditional 1-3 on-shore off-shore split for manual testing. So what to do? Well, I’d take one of my remaining two, wave a wand and create an Automation Czar. While still a normal part of a product team this person would be focused on ensuring that the application development and deployment pipelines are as automated as possible (Continuous Delivery) and that the system can prove that it works at any point in time, all the way up through deployment to production and post-go live. This is certainly not just an Ops role and probably not even just DevOps (although that is closer).
So that leaves one person who is focused primarily on testing. I’ll reiterate that everyone is responsible for the quality of the product. This means that everyone is proving that their code works, their stories are testable, and that the Continuous Integration and Delivery pipelines prove this, well, continuously.
So what is our last team member doing? I’d say a little development, a very minimal amount of exploratory (manual) testing, and spending the lion’s share of their time automating more elaborate test suites than are being put together by the rest of the team. Testing scenarios that were not thought of and generally being devious. They’ll also have an eye towards scalability, integration, and other non-functional requirements. This is pretty close to what Google is calling a “Software Engineer in Test" and by their own admission it’s a pretty hard role to fill.
I’d start off by reading up on BDD and TDD and see what practices you can adopt right off the bat. Getting to where tests are not viewed as optional but are actually considered to be part of story delivery is critical. Once you are there you’ll be well on the way - Mountain Goat Software has some thoughts on this process that are pretty good but most of it boils down to “Start doing it, then do it better.”
I’ll expound on this a little bit though. Let’s say you’ve gotten your testing off the ground, CI and CD are doing well, but you don’t have an Engineer in Test lying around and you can’t seem to find one to hire? Then you’ll probably have to grow the role. A good way to get this moving is to only fix tests, not bug reports.
This means that even if your teams tester can’t use a more code focused functional testing framework (I’m a fan of Geb and Spock) they’ll still need to provide an automated test demonstrating the defect. A simple way to go after this for a web application is to use Selenium or JMeter to record the bug. With just a little training this can then be folded in to your CI framework and voila, instant test automation.
“But wait,” you say, “now I’ve got a mindless, brittle test in my CI system that is going to break the build as soon as someone updates a pixel on the screen!” Yes, there is that - and I would have your tester role busily learning the tools needed to remedy that (hint: I’d steer clear of record-and-script style tools like Quick Test Pro and LoadRunner for the same reasons as Selenium IDE when you are looking at testing large amounts of functionality). In the mean time, if the build breaks because of the test, it’s time to refactor the test in to a more durable form - a broken test is a broken test regardless of the reason.
I like this approach because it emphasizes consistency in how defects are found, communicated, and fixed - via test. It also allows an incremental adoption while getting much of the benefits that a full blown Engineer in Test on the team would bring. Over time the manual testing falls away entirely, the skill sets build, and the product keeps getting better.
The fewer the facts, the stronger the opinion.
worldshap.in - Visualization For the Masses?
I love the overall concept and think there should be more of this type of work going on. With all of the censuses, health and wellness studies, and other big clumps of data lying around there doesn’t seem to be a lot of progress being made (at least that I’m aware of) to make it globally accessible. Yes, most of these datasets are available for download relatively easily. Sure, lots of studies come out in journals. It seems to me that with modern tools and technologies we can throw the doors open wider and allow global access. Sure you won’t get deep statistical analysis but there is a lot to be learned from even cursory exploration of data for yourself. Sites like this are a good start.
I do get a little hung up on the choice of visualizations though. The implementation of the polar chart is done well and I’m encouraged to see what folks are doing with JS and HTML5 but it feels a bit forced. The chart works OK for up to around 3 countries, but as the numbers increase it becomes very hard to read and the legend off on the side doesn’t really help.
Beneath the fold there are some more traditional bar charts that I think do a better job of describing the information than the polar chart itself. I’m not trying to knock the work that is presented here, it works well for comparing a couple of countries, I just feel like there would be more effective ways of comparing the information and they’re on to something with the other views.
The biggest plus here is encouraging a wide audience to examine and explore information about their world. The more we enable this kind of ability the more informed we can all be. Perhaps an open-source data visualization project? Wolfram|Alpha? - I’m still not sure what to make of that. Maybe such a beast already exists but I haven’t found it yet.