As you engage external developers who work with your APIs, it is often early in their product lifecycle that they discover, evaluate and integrate your services.
The good times, if you will.
As with many other aspects of life, developers will make a longer term, and more sustainable, judgement of your company and its offering based on how you interact with them throughout the lifecycle of your relationship. This period can often include some not-so-good times when you need to communicate bad news about the status of your system, upcoming changes in your APIs and eventual deprecation of APIs. In this chapter, I’ll talk about the best way to ensure that developers dependent on your services are kept informed, content and stick with you through the good times, and the bad times.
The separate disciplines of Developer Relations and Developer Marketing were defined and differentiated up front in the book’s introduction and while this book clearly focuses on Developer Marketing, this subject of this chapter probably overlaps with Developer Relations more than any other. What we have learned at Visa is that there is no better way to build credibility with developers than to be there when they need help. Not just help with documentation or domain expertise or evaluating your service. But also help when they are in the midst of a crisis, a crisis that your software may or may not have been responsible for. If you can be just as forthcoming with responses, explanations, tools, at these times, when they need you, as you are with emails, blog posts, beta programs when you need them, then you will have something that all platforms, all businesses, crave; a customer for life.
Introduction
Throughout this book we have talked about developers’ fundamental issue with marketing and the fact that it is all too often a one-way conversation. The scenarios described in this chapter can represent embarrassing or mission critical situations for your company, but they also present opportunities to have substantial and meaningful two-way conversations with your developer community.
I will reinforce the key point to remember when building and managing any commercially available API; your customers rely on this API to run their business. As a global payment platform provider, at Visa we are more acutely aware of this fact than many API providers. In this chapter I hope to pass on Visa’s experience and learning, to help ensure your communication with developers is great in the good times and really great in the bad times.
I have broken up the narrative of the full lifecycle of any service or API and identify areas where we can always do better if we think about the situation from the developer’s perspective. I cover key touch points where the Developer Marketing team can lend a hand throughout a service lifecycle: when it is introduced, when it is generally available in production and when it is retired. The common theme is that each of these touch points must be a two-way conversation, otherwise other developer marketing efforts could be futile. If you do a good job in these areas, it will help generate authenticity and goodwill that no marketing budget in the world can buy.
Visa’s developer ecosystem
Visa is a world leader in digital payments. Our mission is to connect the world through the most innovative, reliable and secure payment network—enabling individuals, businesses and economies to thrive. The Visa developer ecosystem is a broad and diverse community of developers working on applications and solutions for issuing banks, acquirers, merchants and technology partners. Visa developer platforms and brands enable use cases as simple as charging a credit card, with the Cybersource API or as complex as building a global peer-to-peer payment application with the Visa Direct API.
I have been with Visa for eight years, spending about half of that time as a Java and .NET developer on the Authorize.net and Cybersource platforms. As the rapidly changing payments landscape moved from primarily a retail eCommerce feature to a core component of almost every application and service, it became clear that APIs and Developer Experience needed to be managed as a first-class product portfolio. I moved into product management to help lead this effort for Authorize.net and then Cybersource. Our teams pioneered many efforts within Visa, which today are taken for granted in most developer programs, such as open source SDKs on GitHub, API status pages and community forum post mortems.
To help developers build easier, faster, more secure ways to power commerce, Visa Developer offers direct access to a growing number of APIs, tools and support resources, which had previously only been available to select financial institutions. Through these efforts and through acquisitions such as Cybersource and Cardinal Commerce, a Visa developer can be a small business merchant, a global enterprise ecommerce company, an issuing bank or a disruptive fintech startup.
While other developer marketing efforts covered in this book may be segmented and targeted for different segments and personas, most all of the content in this chapter is, uniquely, universal to all Visa developers, and, indeed, to most developers, period. Introducing new APIs, retiring old APIs, dealing with availability or performance issues, these are fundamental challenges that all platforms face, regardless of their business domain.
Communicating future changes
New services and features are exciting to introduce, launch and promote. However, developer products, unlike end-user products, cannot be adopted without development effort on the customer’s side. With end-user products, a human being can interpret the changes, intuitively adapt to the new behavior and even choose whether to adopt a new feature or not. As much as software is taking over the world, a computer can still not do any of these simple things. When new developer/programmatic features are introduced, you are responsible for documenting every minute detail of the change, providing steps and guides to adapt to the change from older versions. Most fundamentally, you need to convince developers that the change is worth integrating.
New features
Contrary to what many marketers think, developers are not always falling over themselves to try new features. When a new API or new feature is introduced, developers want very clear communication as to why this new API/feature is easier to integrate, more performant and why it is worth their time and effort to adopt. Certain developer communities embrace rapid change because it’s already part of their technology stack, e.g. JavaScript. In contrast, developers on crucial mission-critical Enterprise stacks can be much slower to adopt anything new. Beta programs, early access opportunities and limited feature releases should clearly communicate the expectations for both sides of the agreement. What is supported, what changes can be expected, what the upgrade path will be, and so forth.
In addition to backward compatibility, one thing that cannot be overlooked is parity with legacy APIs. That is, what is NOT in the new API. Regardless of your justification for removing an existing feature, somewhere out there, that feature could be the cornerstone of someone’s business.
At Cybersource, Visa’s global merchant services platform, new REST APIs have been introduced with a completely new Developer Center which explicitly defines the APIs available. An additional design feature of the Developer Center was to route all legacy gaps, questions about features, and comparison with older versions through a single Upgrade Guide, which reiterates the evolutionary path of the integration technology before linking the developer back to the legacy docs. In this way developers can educate themselves about future roadmap plans rather than being forced down those paths prematurely. I’ll discuss this guide later.
What about the old version?
One of the biggest mistakes that API providers make is expecting/believing that people will move off old technologies simply because you have released a fantastic new version. The world doesn’t work like that; many people still have desktop computers, listen to CDs and use Internet Explorer, after all!
New features should be consumed on the developer’s terms not yours. Even without a hard migration, such as a forced upgrade, if developers/customers have to spend the time and money to upgrade, you should make sure that it’s for a very good reason. The onus is on you to demonstrate the business value of any new API or integration method. Developers may be motivated to move to REST APIs because of a more efficient JSON message format or broader tool support but when a new REST API is introduced you should also highlight additional business benefits, such as new API methods or enhanced data access.
There are times when a forced upgrade is unavoidable. Developers understand why these scenarios occur and they judge your service based on how the process is managed with respect to THEIR software and THEIR business. We saw this most recently when the payments industry made the decision to disable older, insecure communications protocols and force customers to upgrade to the latest version of TLS. The value of doing this was clear and uncontested, however merchants still had to spend significant resources to ensure their own software and all their dependent systems were compatible.
Effective communications cannot stop at emails and marketing campaigns but rather should include tools and platforms: sandbox systems, version checkers, sample code and practical advice, even beyond the realms of your own services (e.g. compatibility of base platforms such as .NET, Java, etc). Clear timelines are vital and timeline extensions are often necessary; they may be undesirable but are often unavoidable if large clients are impacted. Well managed brown-outs are also an effective way for developers to be included in testing efforts and to alert customers should other communications prove ineffective. Cybersource recently removed some old IP addresses through carefully planned, well communicated, short periods of downtime before the final decommissioning, allowing us to minimize the impact to our developer customers.
Any forced migration, such as the TLS upgrade mentioned above, is clearly a last resort. However, even when the benefits of newer APIs are obvious and you have provided tools to simplify migration, developers still need to be shepherded to the new and away from the old. While it is never a good idea to remove documentation or completely cut off support, simple low-impact methods are often overlooked. For example, older integration methods should be deemphasized in all forms of marketing and support material, those methods should be unavailable for new customers to use, with advance notice, new features should not be added to the older APIs, and so on. Sunsetting a developer feature is an order of magnitude more effective when the entire business is coordinated around that objective and follows a consistent strategy.
Operational changes
Shorter term, smaller, future activities can also be key touchpoints for you to interact with your developers and provide an opportunity to build trust, or lose it. Planned downtime should be an oxymoron in today’s Platform-as-a-Service environment, however there may be situations where it is unavoidable. In addition to sensible scheduling based on analysis, developers should always be included in any communications or alerting of these occurrences. For example, announcements should be made in advance through developer dashboards, forums and email distributions in addition to customer and business channels, such as account managers, application alerting or notifications.
Even when downtime is not part of scheduled maintenance, any planned or known activity where a change in behavior may result, should be communicated through developer channels. Something as simple as a tweet like: “As part of ongoing efforts to enhance our sandbox, test transactions will now return error code 234 for transactions which fail AVS checks” will help build trust and goodwill.
An additional note on sandbox systems. A common bugbear for developers is when changes hit production systems (or are even announced for production systems) without being implemented on the pre-production (sandbox) systems ahead of time. It is always worth restating that sandbox communications should be just as important to your developer marketing team as production communications. A good rule of thumb is to make sandbox changes available for the entire “reasonable advance warning” period, otherwise the advance notification is essentially useless to developers, as they cannot act on the call to action. We recently had to extend the production date for a change due to the fact that we had been delayed getting the change to sandbox. Business pressure to maintain the production date, albeit with a compressed sandbox period, must be balanced against adequate testing time for developers.
Experimental features
At Cybersource we have seen one area of future communications become more important as we innovate rapidly, with an ever-stronger focus on developer experience. A critical step we have incorporated into the product life cycle is a validation stage, shortly after ideation and before we ramp up the creation stage of building the product. For our developer products this validation stage may involve API specification reviews, usability studies and sometimes alpha/limited release sandbox builds.
Candidates to help with future product validation could be “MVP” developers, developers from partner companies as well as “blind test” developers who can often be recruited through usability test agencies. These developers are often key influencers in their technology community and can be clear advocates for your platform. Transparency is critical and objectives must be clearly defined internally across the broadest range of stakeholders to ensure these relationships grow and thrive. Commonly, companies ignore feedback, expect technical reviewers to provide quality assurance and miscommunicate the production readiness of their software.
The future should be a two-way conversation
At Authorize.net, one successful way we have found to make sure developers feel invested in enhancements to the product is to make sure they are part of the process. When we launched the Ideas forum on our developer community we were committed to including at least 2 or 3 of these requested features every quarter of our product lifecycle. We let the community know when features were in the final stages of consideration and worked with the ideas of supporters (up voters) directly while the feature was in development, to ensure we delivered the correct functionality as they understood it. By demonstrating that submitted ideas were under consideration, in development and eventually released, we encouraged developers to engage, submit ideas and make the program (and ultimately the product) successful.
Communicating current events
Other chapters have talked about developer events, developer programs and communicating with developers in the context of the present. The focus for developer marketing is almost always on the good messages; the great APIs, the interactive documentation, the comprehensive sample code. That’s the proactive, planned, predictable version of the current state of play.
This section deals with the burning questions that developers may have on a day to day basis, over and above general developer support, which I will not cover here. How and when these reactive events are handled is a key factor in their overall developer experience.
API status
Communicate effectively about status. That is, the status of your APIs, the status of their issues, the status of any feature requests. These can be some of the most difficult conversations to resource, to manage and to make successful. At Authorize.net we implemented a status page so that customers could check on the status of our service and be updated on the progress of ongoing issues. Some lessons we learned early on were:
- Try to update the status page as soon as you are alerted on an issue. We have found it’s much better to be over communicative than have developers determine there is an issue and make alarmist posts on social media before your site is updated.
- Use automated alerting of API slowdown/downtime where possible. There can be understandable reluctance from marketing and corporate relations to alert without manual intervention. However with careful consideration of event type, alert threshold and status message, automated alerting can save your developers hours of troubleshooting analysis.
- Regular updates reassure your customers and developers that you are prioritizing the issue. We have found that sentiment can quickly degrade when developers are called in the middle of the night to manage and mitigate issues, which turn out to be your service, but see no evidence that you are addressing those issues.
API status via API
A common requirement from large-scale API consumers is to provide a “ping” request, or a status API method which will return the status of your API. What seems like a simple and reasonable feature can turn out to be much more difficult, particularly if there are multiple downstream parties. So while you can easily determine and return that your system is up and taking requests, your customers may find the API is effectively down because a third-party backend is returning a system error. So my advice here would be to determine what is practically possible and feasible, then clearly communicate limitations. Remember, that while your developer audience may understand the nuance here, the business they work with may not. There will always be times when the only way to determine if an API is up and working correctly is to call the API itself.
Sandbox status
When we initially thought about improving service status and transparency, our operations team came up with a plan for 24/7 production support and week day, US business hours, sandbox support. Everyone thought this was a fantastic plan until we got the very loud and clear feedback that, for developers, comprehensive 24/7 monitoring and alerting of the sandbox environment was often more important than production. This makes total sense when you consider that the bulk of developers’ time is spent interacting with your pre-production systems, not production systems. Developers will often be working evenings and weekends coming up to a release, even during a normal development period engineers’ work schedules can vary significantly, not to mention software developers are working all across the globe. Your sandbox should be fully operational, monitored and supported 24/7.
API performance
More and more companies are pushing the boundaries of transparency with real-time performance metrics. Benefits of this include increasing customer trust, winning prospective customers, and deflecting possible inbound support requests if customers can self-diagnose issues. A simple graphing of API response times allows developers and customers to see any spikes at a glance and correlate their own monitoring and alerting systems.
API issues
Similar to real time API response times and performance metrics, developers will evaluate your commitment to them by considering how issues are solicited, gathered, tracked and resolved. In the past this may have been through a proprietary ticketing system but with so many companies providing source code, SDKs and samples on GitHub, handling of issues is available for all to see. While there can certainly be a management overhead in adding an additional issue-tracking system to your process, it is critical that dedicated resources are committed to every repository you host. A well-maintained project is a valuable asset to your developer program and community, cultivating relationships which can benefit recruitment, evangelism and industry recognition.
API changes
An API Change Log can be thought of as release notes for your API. It should be updated when API changes are deployed to production and as with everything else in this section it should also be updated when sandbox changes are released. In the context of past events, this simple log of changes, organized by date and API version, means that developers can reconcile any changes in features or responses with observed impacts or side-effects on their own system.
It’s worth noting that the Change Log is not the place to publicize a breaking change for the first time (see Communicating Future Changes and also note the title of this chapter!). For substantial changes, links to new documentation, blog posts or tutorials should be included. If properly maintained and organized, this current event communication mechanism develops into a trusted record for communicating historic changes, which is the focus of my next topic.
Communicating past issues
While supporting a business critical service such as payments, one important thing we have learned about developers is that they understand and accept that issues will arise, latent bugs will surface, cables will get severed, human error can occur. After all, these same developers are responsible for a system of their own, which may be more mission critical to their customers than our payments component. What developers cannot accept is a lack of transparency or a proliferation of excuses when these things happen. They want the cold hard facts, the root cause analysis, the mitigation steps put in place, etc. and they want it in a timely manner. In addition, when technology is superseded, deprecated or retired, developers want to know if, where and for how long, they can get support for that technology.
Historical incident & performance logs
When something does interrupt or delay your service, details of the duration and impact should be clear, concise and simple to find on your Developer Center or status page. Similar to the Change Log, this feature could save a developer hours of troubleshooting on their own system if they can satisfy themselves that the incident was monitored, logged and addressed. Services like StatusPage can be used to offer industry experience and best practices, effectively becoming a de facto standard for incident communication. A log entry for an hour-long performance degradation generally includes an initial entry to confirm the issue, a further update to describe progress and estimated duration with a final update to confirm resolution. For significant material outages, for example a hard down situation, the standard incident logs may not be enough and that’s when a more detailed post mortem may be required.
Post mortems
The level of integrity and transparency with which service outages are handled can often be the difference between a developer leaving your platform or staying. While the appetite for full transparency can vary widely between organizations, often for valid reasons, developer marketing and developer experience groups should always be advocating for the highest possible level of detail, in both the root cause analysis and future mitigation plan. A comprehensive post mortem should have an issue summary, a detailed description of the impact, a root cause analysis and a remediation plan. While contractual obligations (Service Level Agreements) and discretionary compensation (for example, service/support credits) vary across business segments, a sincere apology is universal and always appreciated by developers. Regardless of the level of disclosure, recognition of the incident and its impact on your customers is non-negotiable.
Legacy documentation & support
An API is like a puppy; it’s for life, not just for Christmas. As soon as an API is in the public domain and customers, even just one customer, builds on top of that API, you are obliged to document and support it, until the end of its natural life. We generally draw a distinction between alternative/older APIs, deemphasized APIs, deprecated APIs and sunset APIs. For each of these groupings there is a different level of support and documentation that any developer can justifiably expect. While we cannot claim to get that level right all of the time here’s how we generally break it down:
Alternative/older APIs
This classification can often be as subtle as the message format or the transport mechanism. If you do have alternative APIs then they should have parity with newer versions and features should be delivered via both at the same time. These “flavors” should always have the same level of support and documentation. The only exception may be where certain styles of documentation simply may not be available or make sense for a particular flavor.
Deemphasized APIs
Often this grouping may not seem necessary, but we have seen the usability benefits of streamlining the pathways to newer APIs, meaning that less used/more legacy APIs may inevitably become deemphasized. The level of support and documentation should not be significantly less than the first grouping, but when your ultimate goal is to deprecate and sunset an API, this phase of making it less accessible can be effective in reducing uptake by new customers, which therefore starts reducing support costs. Of course, as I mentioned earlier, you need the support of the entire business to exclude anything you deemphasize from marketing and sales materials.
Deprecated APIs
This is generally the first point where you will make public announcements that an API is on its way out. At this point there may be decisions to change support levels, move documentation, mark up legacy pages, and so on. Critically there is no public, definitive timeline or sunset plan at this stage, since customers must feel assured that the API is not going away yet (usually not any time in the coming three years).
One important thing to note here is that the term “Deprecated” is one that is almost always understood by developers but hardly ever understood by anyone else. The term was introduced widely to software developers with early versions of Java and has been co-opted as part of software versioning nomenclature. But you cannot rely on the designation alone being sufficient. Support teams must be ready to answer calls and questions on what it means, the timelines for sun-setting, how developer customers should plan their move to a replacement API, and more.
Another point that is commonly left dangling is non-developer marketing material that still references deprecated APIs as if they were current. Deprecating any API or service must be part of a broader, thoughtful, planned campaign.
Sunset APIs
When an API is sunset it is officially on a path to End-Of-Life, which ultimately means at some date in the future, software using that API will cease to function correctly. This is a significant and material decision for a company to make and can often require executive approval along with corporate relations and legal advice.
While that statement was meant to be impactful it should not mean that product management or marketing shirk from making the decision when the time has naturally come to move customers off an older API. By the same token product management should never be coerced into sun-setting an API simply because engineering or operations are tired of supporting it. Remember that cute puppy, everyone said they would feed and walk and pick up after, right?
Establish guidelines and, when the time does come to say goodbye to an API, the most important thing is to decide upon the timeline, and in particular, to communicate and stick to it. The timeline must be annotated with clear goals and objectives from all stakeholders; regular metrics on declining usage, results of marketing campaigns to move customers, direct contact with any large client users, and so on.
Taking a product to end-of-life is a discipline that should be owned by product management. However, developer marketing and developer relations have a critical role to play. Like everything in this chapter, sun-setting an API becomes an opportunity to differentiate your developer experience and to add authenticity and legitimacy to everything else you do.
Evolution/upgrade guide
An Upgrade Guide, or as we’ve begun to call it at Cybersource, an Evolution Guide, becomes an essential resource for both internal and external teams to understand how they can begin to take advantage of the fantastic new developer features while plotting a clear and considered path from their current integration. We have found that rather than sprinkle FAQs or advice or migration tips around your developer documentation, you should funnel developers to a single resource where they can feel like they’re being briefed on the what and the why (while also being expertly advised on the how). Of course, options should be available but it’s important to have a strong point of view on a sensible path for upgrade, on timelines and on cut offs/hard deadlines if you have them. You think about this resource as closing the loop, referring back to the beginning of this chapter when we talked about communicating future changes, this page will lay the groundwork with your developer audience, for how, when and why those future changes are being delivered.
In conclusion
Our biggest lessons on this journey with the Visa developer community have been:
- Think about migration first rather than last. It’s simply too easy to get caught up in beta release, MVP, pilot program and before you know it those things have rolled into General Availability, a full rollout and no easy upgrade path for your existing developers. From day one write down your Developer Marketing KPIs and make the first one “#users left on legacy = 0”. That way you focus on ease of migration from the first ideation, architecture and design.
- Transparency trumps everything. Many times throughout this book we have highlighted developer’s natural aversion to marketing. This distaste is rooted in developers’ instinct to understand, look under the hood, get to the bottom of everything. The effectiveness of your Developer Marketing will be directly proportional to your capacity to have an honest, authentic and transparent, two-way conversation with your developer and partner community.
- Tools, tools, tools. While clearly overlapping into the discipline of Developer Product Engineering, I have called it out here for that precise reason; the messaging/marketing scenarios in this chapter can almost always be made more effective, authentic and acceptable, with simple web-based tools to aid and inform developers. Tools such as response code lookups, compatibility checkers, migration wizards, etc.
Most importantly, remember the number one rule: Don’t break their software, ever.
What’s next
Despite the lessons learned and the collective experience of over half a million developers, we are still striving to improve in all the areas described in this chapter. One of our brands may do a fantastic job of real time issue response while still needing to improve on post mortems. Another platform may have expertly managed beta programs but an immature end-of-life capability. One of our biggest goals over the next few years is to drive towards consistency in Developer Marketing and Developer Experience across every platform, making the Visa Developer program a consistent experience for our developers regardless of which they are using.
This article was originally published in SlashData’s “Developer Marketing: Essential Guide” Book.