Rejected titles included:
As a developer tool, CircleCI is uniquely positioned with a talented customer base. Our customers are smart. They are infinitely creative, and sometimes surprise us in ways we did not design our system to support. This is both a blessing and a curse. On the curse end of the spectrum, the long tail of mind-bogglingly inventive edge cases can transform simple tasks into Sisyphean Hero’s Journeys. As we work through requests such as: “please refactor the codebase to use Golang’s implementation of git instead of the current OS-specific blend,” I find myself reflecting on this comic:
Original art from xkcd
Despite complications, the benefits of customer feedback far outweigh the costs. Our customers’ observations are detailed. Their requests are informed by their own experience of working with product specifications and building applications. We are mindful of the advantage their expertise brings and strive to collect their input during the launch cycle. Just in case I had any doubts around the value of our customer-focused launch process, our recent Windows release drove home its importance in ways I never anticipated.
Incorporating customer feedback into our launch cycle
At CircleCI, new products go through a series of launch phases that encourage customer input and scrutiny to help us fine-tune the product as we go. We’ve employed these phases successfully to launch orbs, Windows (more below!), and soon our new GPU resource class.
Our launch story begins once we’ve completed a functional proof of concept. From there, the process goes through these six stages:
Internal launch: dogfood. We are our own first customers, always. We continue to use our own product throughout its lifetime.
Preview launch. After we are satisfied that our preliminary Well Architected and Functionally Limited (WAFL) product can provide value, we open access to a limited set of self-selecting customers that have generally expressed interest in the new feature. We expose preview documentation to this group.
Bleeding edge improvements (cycle). As the limited customer group raises issues we address them, make improvements, and repeat. They’re on the bleeding edge of our product, and while it’s useful enough for them to realize value, there are rough edges.
Silent launch. When the roughest edges of our WAFL have been removed, we quietly open up use of the product to our entire customer base. The customers who discover the new feature tend to be actively seeking its functionality. These intrepid explorers are motivated to get it working for their specific use case, and often share strong opinions regarding the roadblocks they encounter while getting to their elusive first green build. As we make roughest-edge improvements, the severity of incoming issues declines; this is when a more traditional soft launch (the preview release of a product to a limited audience) would begin.
Rough edge improvements (cycle). As this larger customer group raises issues, we see the difficulties that highly-motivated users experience and we address those first. Successive waves of customer input help us whittle down pain points to a finite short-list of lingering “bumpy ride” experiences, leaving the product ready for general launch. We discuss any remaining issues (often UX concerns) with our Support team in preparation for the launch.
General availability launch! Game day. We clutch our pagers for comfort and review the prioritized issues lists one final time. We have a clear idea of what areas may cause disappointment, what will spark joy, and where we may need to triage scaling concerns.
Throughout this process, we gain incredibly valuable, pointed feedback from early adopters. On occasion, they’ve even debugged issues for us. Most importantly, their feedback offers nuanced direction, providing us with confidence in our final product fit. Our Windows launch exemplifies these benefits.
Customer feedback for the Win(dows)
When we started building support for Windows in January 2019, our five-person team quickly sprinted to a proof of concept. Around other keeping-the-lights-on work, we made our way to a WAFL and promptly ran into our first problem. In order to open up our Windows preview launch, we needed a way to support capacity demands on this new resource and, at the time, we did not have automatic resource scaling in place. Without autoscaling, we were faced with three options, we could either:
Overprovision compute (keep virtual machines warm to spare customers from excessive VM startup times), at a high monetary cost to CircleCI.
Marginally provision compute and tie SLA-breaches to our on-call rotation, at a high cost to developers, or
Not provision compute, and let customers suffer slow VM-provisioning times.
We ultimately decided to overscale our resources to provide the best user experience and the lowest maintenance overhead for our engineers.
With our preview launch unblocked and running in the background, we waded into stabilizing our compute systems. Five engineers joined our team, and the ten of us set about grokking the best path forward in the new (to us) systems. We managed our Mac fleet and image release cycle, began to improve fault tolerance with our cloud providers by building a circuit breaker, researched queuing theory and started to implement autoscaling; new feature development for Windows remained largely on the back burner.
It wasn’t entirely forgotten. While the rest of us worked on improving our systems, a single engineer from the original team chipped away at our incoming issues from preview users. And Windows usage grew. After a few months, we silently removed opt-in gating. By the end of the summer we saw a regular, cyclical build cycle emerging on our resource usage dashboard, an indicator that feature traction had been achieved.
And then, our Windows GA launch date was announced and our timeline went from months to weeks: it would take place in 25 days.
As engineering manager, I laid out a plan for how we would succeed, and the silent launch became my rock. We’d been in silent launch for months now. We knew the lingering paint points our early adopters felt and we knew the functionality they appreciated.
It is difficult to articulate just what a relief it was to myself and to the team to review the results of our silent launch, and be able to quickly and confidently gain a clear view of where we were and what outstanding work we needed to wrap up in the weeks leading up to the launch. Despite our team’s many new responsibilities, shifting areas of focus, and the fast introduction of a launch date, when GA rolled around we were confident in the strengths and limitations of our offering. From a product-completeness perspective, we were already in a good place.
On launch day, we clutched our pagers with sweaty palms. We never needed them. Our circuit breaker resolved a provider outage, the freshly-completed autoscaler scaled, and my blood pressure normalized.
Conclusion
Even the best-planned feature releases are daunting and involve a certain element of hope. While hope is certainly one launch tactic, I do not recommend it as a strategy. To improve both our team’s confidence and our product, we incorporate customer feedback early and often in the development cycle through a series of increasingly public launches. You can, too.
Because of our customer-focused strategy, we were able to launch Windows confident that:
We had thoroughly prod-tested our code.
We had fixed the most pressing issues early in the development process.
We had achieved feature traction.
We are incredibly grateful for the feedback that our users provide, and we always welcome comments and suggestions.
Want to contribute? Share your ideas with us here.