How an ad-hoc team of outside coders and troubleshooters revived HealthCare.gov (Steven Brill/TIME)
[Via Techmeme]
Steven Brill / TIME:
How an ad-hoc team of outside coders and troubleshooters revived HealthCare.gov — Obama’s Trauma Team — How an unlikely group of high-tech wizards revived Obama’s troubled HealthCare.gov website — more than two weeks after the launch of HealthCare.gov—White House chief …[More]
A very interesting read. We sometimes miss the unselfish efforts of Americans and their need to step forward to offer themselves to help. Thankfully, the desperate politicians allowed them to this time.
As much fun as seeing the engineers in Apollo 13 work the problem.
What started as an ad hoc group assembled in the middle of October produced by December 24 a website functioning beyond their expectations.
And every one of them was as proud of this as anything they had ever done — whether it was starting a company or helping run political campaigns. No sociopaths here
A key paragraph is from page two, after describing just how bad things were on Oct. 17, including the possibility it would be scrapped:
Carney tried to fend off the inquisition, but he had little to work with. Pressed repeatedly on when the site would be fixed, the best he could say was that “they are making improvements every day.”
“They” were, in fact, not making improvements, except by chance, much as you or I might reboot or otherwise play with a laptop to see if some shot in the dark somehow fixes a snafu.
Yet barely six weeks later, HealthCare.gov not only had not been scrapped, it was working well and on its way to working even better.
This is the story of a team of unknown–except in elite technology circles–coders and troubleshooters who dropped what they were doing in various enterprises across the country and came together in mid-October to save the website. In about a tenth of the time that a crew of usual-suspect, Washington contractors had spent over $300 million building a site that didn’t work, this ad hoc team rescued it and, arguably, Obama’s chance at a health-reform legacy.
One-tenth the time and for a lot less money. That is what committed ad hoc communities can accomplish using the tools being created in this modern age.
Taking the form that would be recognizable to anyone who has read about the Manhattan Project, it leveraged the bottom-up, pragmatic strengths of engineers without imposing serious top-down, authoritarian restrictions from above.
It just worked so fast with so few people, because the tools we have allow that to happen.
It shows the failure of our top-down, authoritarian approach to deal with some of complex problems facing us. A decade ago this could not have been fixed, harming millions of people, besides being a political defeat.
Now we have had a group of just plain Americans rapidly come together to fix the problem, demonstrating a degree of adaptability and resilience that will become more important as we progress into the future.
One thing learned:
But one lesson of the fall and rise of HealthCare.gov has to be that the practice of awarding high-tech, high-stakes contracts to companies whose primary skill seems to be getting those contracts rather than delivering on them has to change.
Will it be remembered? Industrial Age companies look for cost-plus approaches, allowing them to make a profit no matter what. Information Age approaches use a different set of criteria, ones that can be observed here.
The Industrial Age approach took months and millions to produce something that made them a profit but did not work. Information Age approaches fixed the problems in days and for much less cost.
By creating ad hoc, highly networked communities of experts each focussed on win-win solutions to help the community, not zero-sum ones to protect their career.
Using people who stepped forward to volunteer their best abilities. All wanted to make it work, not looking at making a profit.
Just read about the road trip they made, passing around a iPhone conferencing in people from across America, to get an idea of just how bad this all was. Because these were not politicos who were working out ways to save a political career but actually trained engineers and scientists.
Working the problem.
Usually, politicians only allow these guys in at the end of the process and keep them on a short leash. Richard Feynman demonstrated what can happen when a scientist is let loose.
The first thing they did was exactly the same thing Richard Feynman did when he was involved with the Challenger committee – talk directly with the engineers doing the work.
Here, the engineers were embarrassed and wanted to fix things. No talk of scrapping.
As one of them said, “If you can get the managers out of the way, the engineers will want to solve things.”
Managing an Information Age project requires a very different ethos than an Industrial Age one.
And that was key. If the people responsible for writing the original code do not want to help, then it is better to just start over. Having the enthusiastic help of the original engineers made the solutions possible.
This approach also meant getting help from anyone who could help, not fighting turf wars.The White House crew simply connected and pulled in people like Mike Abbott, who had saved Twitter’s website when it was out of control, possible on track to destroy the value of the company.
One man, in the middle of raising money for his startup, dropped that effort to join.
They had people who could make tens of thousands dollars a day from their consulting business step forward to help. And even here they had to find a way around the top down constraints:
As for Dickerson, Burt and the others who arrived for what they thought was a few days only to stay eight to 10 weeks, they were told that government regulations did not allow them, even though they offered, to be volunteers if they worked for any sustained period. So they were put on the payroll of contractor QSSI as hourly workers, making what Dickerson says was “a fraction” of his Google pay.
Don’t want to be abusive of volunteer workers? I can agree with that. Simple fix. Pay them by the hour. It did not stop their work at all.
As Dickerson said, “It’s just a website. We’re not going to the moon.” Although, the health of millions could be put at risk if they failed to correct the problems.
They started fixing problems at the site within hours of arriving at the headquarters. And they provided this awesome insight that has to be remembered:
What were the tech problems? Were they beyond repair? Nothing I saw was beyond repair. Yes, it was messed up. Software wasn’t built to talk to other software, stuff like that. A lot of that,” Abbott continues, “was because they had made the most basic mistake you can ever make. The government is not used to shipping products to consumers. You never open a service like this to everyone at once. You open it in small concentric circles and expand”–such as one state first, then a few more–“so you can watch it, fix it and scale it.”
Will this be remembered?
This ad hoc model is not leaderless. In fact, the problems had arisen because the standard contractor approach had produced no organization in a leadership role. Just a bunch of nameless people around a table.
The ad hoc team quickly identified a leader and away they went, beginning work within hours. Over the first weekend, they had made simple changes that increased the efficiency 4-fold and stopped any talk of scrapping.
Nice to see that the newer model — bottom up, decentralized — simply worked around the damage created by the centralized, authoritarians. (It is actually an old model but new technology allows it to be used against tremendously complicated and extensive problems.)
There would still be problems, many of them outside of their control, but they were on a path to fix the structural problems of the site.
It is a model that we will really need to make better use of. It was used here because the politicians were desperate. Their top-down, cost-plus approach to use contractors may work for Industrial Age processes but are not optimal for modern, Information Age needs.
The authoritarian need for control of all aspects of the process simply do not mesh well with the adaptive strengths of these dispersed approaches when dealing with complex systems.
Simply read the rules of the twice daily standups — which demonstrably saved the site — to get an idea of the differences.
Authoritarians not welcome at all. In fact, the one true button-down authoritarian involved — a business executive, not an IT guy — was actually smart enough to let the coders do their job.
It was a true ad-hoc team focussed on pragmatic solutions that fixed things.
Authoritarians are present in both parties. It appears to be a necessary trait for some. While some have discussed the ability of Obama’s administration to leverage the new ad hoc tools — as he did in his Presidential elections — the administration also demonstrated a failure to understand the lesson, falling into authoritarian mode when under pressure, as every President has.
Indeed, the key mistake made by President Obama and his team–who never publicized the arrival of Burt and other campaign coders in October the way they touted the role of the data-analytics marketing team last summer–is that they had turned only to the campaign’s marketing whiz kids instead of the technologists who enabled them.
Listening to the managers and not the engineers. Sounds like NASA.
Obama’s authoritarian tendencies simply allowed the geeks who made such advances for his campaign to disappear, settling for the same standard group of insiders seen in every previous administration.
But at least some in his administration did keep their names of the tech engineers on its contact lists. And it did have people who knew when to call in the cleaners. It was smart enough to make the best decision rather than simply work to save its own butt — as most bureaucracies do when they fail.
At least it was smart enough to call them back in. (Gotta thank the CTO, Todd Park.) And it saved the website, thus providing insurance relief to millions.
Will they remember the lessons?
A Man With A Ph.D. 