"I will utterly reject harm and mischief."
These words, taken from the Hippocratic oath, are ones that I think application development and delivery professionals should consider carefully as we watch the latest example of "Software eating the world" gone wrong. In this case the software algorithms in the "defeat device" that Bosch created for VW defeated emissions testing for millions of diesel cars. Now, 7 years later, VW is setting aside $7.3 billion to remediate the result. But this is just the latest example of developer complicity in creating algorithms of questionable quality. Consider:
- Facebook's manipulation of users' news feeds. In 2014 Facebook revealed that it had manipulated the news feeds of over half a million randomly selected users to change the number of positive and negative posts they saw. It was part of a psychological study to examine how emotions can be spread on social media.
- Apple's "life threatening" maps. When released in 2012, Apple's new Maps app was so poor that warnings were issued in some countries. In this case, employees were quickly fired, and problems were addressed – after the fact.
If you've ever written code professionally or run software projects you know about the time pressures, unreasonable deadlines, and the pressure to ship stuff before it's ready. While I think shipping "low quality" code isn't as egregious as shipping code that's main purpose is cheating, both can have devastating consequences to customers and our companies. We need to take a step back from the race to ship the next minimum viable product and ask ourselves?
- Are we shipping code that is in our customer's best interests? Does it create corporate profit through a win-win relationship with customers? Is it designed to create long-term customer loyalty? Does it make it easier for customers and employees to engage?
- Is it of acceptable quality? Have we closed out all high priority defects? Are the corner cases truly rare and not likely to occur in the real world? Can we adequately measure our customers' success?
- Is the code we've written something we're proud to own? If another developer that we respected looked at what's written, would we be proud to own it? Would we be willing to explain it's value to our parents, spouse, kids, or clergyman?
If the anwser to any of those questions is "no", then consider your obligation to speak up. And if neccessary, walk out.
Application development managers should watch the course of the VW investigation closely and take the following steps in thier own shops:
- Add ethical checks to the software development process. Good software should be both technically good and morally good. Governance regimens like CMMI and ITIL address the former, but you should ensure the latter.
- Is there a reporting process for bad coding practices? You probably already have a reporting and remediation process for security vulnerabilites and inadvertant open source usage. Consider adding ethically questionable or privacy challenged code and algorithms to that process.
- Make sure "doing the right thing" is part of developers' career development. As a developer it's easy to get lost in the rush of problem solving and trying to get the 1s and 0s to sucessfully compile. Make sure that developers, especially senior ones, reflexively ask whether an algorithm is both technically good and morally virtuous. Otherwise we're not much better than the hackers we try distinguish ourselves from.
Got a story where you've had to put your principles before your LOCs? Hold up a release train? Look for a new job? Feel free to share… we need to support each other in doing the right thing, especially as the code we write begins to directly manipulate the minds of and matter around our customers.
I'll take the oath above… will you?