Does evaluating your Developer Experience seem like an insurmountable journey?
All great journeys start with a map, and the developer journey for your product is no exception. That’s why so many DevRel practitioners have adopted our developer journey map, as it is a tool that enables you to think holistically about your journey and to visualize it as you design, test, and evolve it.
One area where you can really leverage the developer journey map is by conducting a Developer Experience audit – or DX audit.
A DX Audit is a pragmatic, holistic evaluation of your developer journey, as seen through the lens of your target developer audience.
In this blog, we dive into our version of a Developer Experience Audit. We call them DEFT (Developer Experience Friction Tool) Reports. You’ll learn the different components of a DEFT, how we create them, and other useful information for you to track your own developer journey.
This blog is laid out as follows:
DEFT Reports 101: What DEFTs contain.
Pre-DEFT Process: What’s needed to build a DEFT.
Conducting the DX Audit: How we conduct a DEFT.
Building a DEFT report: What a DEFT can uncover for you.
DEFT Reports 101
DEFT reports are the output from our DX audits. They contain comprehensive findings and recommendations specific to your developer journey. The reports are achieved by our team taking on the persona of your target developer audience and auditing your developer product and/or service, documenting the findings as they go.
The report includes technical and marketing findings. Specific points of friction are highlighted, along with strategic and tactical recommendations. You’ll find the DEFT Report to be insight-packed and a valuable resource for all stakeholders across your organization.
DEFT reports are organized into four sections:
Strategic Overview: Summary of the project with strategic themes for improvement and opportunity and an overall score.
Recommendations and Summary of Key Findings: A list of the highest priority findings, each with a recommended course of action to rectify.
Key Observations: Summary of all findings with their consequences and recommendations.
Developer Audit Logs: Raw, detailed developer audit logs listing the steps the developers experienced and the findings they identified during the audit.
Generally, sections 1 and 2 provide valuable insight for leaders and decision makers, while sections 3 and 4 are useful for internal developers and other implementation stakeholders who can take action on the recommendations.
Note the DEFT reports we produce for clients are often in the 100 to 150-page range.
Pre-DEFT Process
Before we start a DEFT audit, there are three things we work with you to identify:
Purpose
Goals
Segmentation and Personas
Purpose
The first thing we establish is the purpose of your DEFT audit. This varies based on your situation. Below are some examples from clients on why they wanted to conduct a DEFT audit:
Our organization’s DX is failing to convert leads into paying customers and we want to understand how our DX is helping or hindering.
We know our DX isn’t optimal, but we need concrete “proof” to present to our higher-ups to secure more budget to improve it.
We need a prioritized list of recommendations to build a strategic and tactical plan.
We think our DX is great but have never verified that assumption.
We’re unsure if our product’s proposition, messaging, and/or learning resources resonate with our target developer audience.
Our direct competitor’s DX is great, and we want to identify how ours can be great too.
We want an independent, expert perspective on our DX to take it to the next level.
Goals
Once you’ve established your purpose, the next step is establishing your goals. Below are several common goals that our clients seek to fulfill through our DEFT audits.
Identify specific DX friction points.
Identify KPIs and metrics by which to measure DX over time. This can range from quantitative metrics (e.g., CPC and time spent on your home page), to qualitative metrics (e.g., those gathered through a survey).
Understand the path developers take across your touch points.
Uncover other problems contributing to poor DX (e.g., failure to align products across acquired companies).
Segmentation and Personas
The final prerequisite is a solid understanding of your target developer audience. Your organization should have its segmentation and personas identified. Still, it’s important to ensure all your stakeholders are aligned with them, as your target audience underpins all assumptions about how the DEFT audit is conducted.
If you don’t yet have a fully-formed developer segmentation, review our Developer Segmentation Canvas for guidance. Alternatively, book a meeting with us to discuss how we can help you create your own.
From there, we identify specific developer personas on which to conduct the audit. We typically build our DEFT audits using two or three personas. Each persona has its starting point and assumptions. For instance, a person’s experience level is a crucial factor in conducting our tests. We also recommend the persona of a first-time user to gain the perspective of someone unaware of you and your product. Additional personas can then test against increasing familiarity levels of your company, product, and the problem space to represent more sophisticated buyers.
Conducting the DEFT Audit
A good DX audit must be built from the ground up. We walk in the shoes of your target developer personas and experience what you have to offer through their eyes.
Our developers (using their designated persona) typically start with a Google search to discover the product. Based on the persona’s attributes, the accuracy and specificity of search terms will vary greatly. From there, we see where the journey takes them.
If they can Discover your organization’s website from those searches, they follow your messaging and signposting in the mission to find resources to help Evaluate and Learn more about your product offer.
And yes, our developers do get hands-on with your technology. Our DEFT audits are not just desk-based research exercises. If all goes well, the journey should enable the developer to Build a proof of concept like a ‘hello world’ style app or similar foundational tasks like encoding a video, moving cryptocurrency from one account to another, or invoking an API for a task.
Note: the words in bold above relate to the stages of the Developer Journey Map. Understanding the developer journey from Discover to Scale, is fundamental to getting your developer journey right.
It is noted if the journey fails (i.e., our developers can’t achieve Hello World without excessive effort or additional assistance). However, our developers will push on with their audit to uncover additional findings and recommendations to improve your DX.
Building the DEFT Report
Once the audit is complete, we build the report bottom up. We start by extracting the key developer findings from the audit to build Section 3 – Summary of Key Findings. This is a critical phase, as the thought process of identifying consequences for each finding is where the recommendations are born. Below is an example of how this looks:
We use the following color coding scheme throughout all sections to help highlight the severity of issues:
Purple: Critical issue that blocks progress.
Red: Major issue that needs to be addressed.
Amber: Minor issue.
Green: Good/well done.
We also generate and include bespoke developer journey maps for each developer persona to visually show how our developers navigated your touch points across each stage of their journey.
Upon an in-depth review of the details of the Key Finding, we identify and summarize the priorities to include in Section 2 – Recommendations & Summary. The recommendations are identified for each stage of the journey.
Finally, we complete Section 1 – the Strategic Overview. Here, we restate the project purpose and goals, list the test personas and assumptions, and assign an overall DX score.
In addition, we summarize key themes of areas of opportunity, identify low-hanging fruit, show examples of best practices, and recommend the next steps. This may also include a preliminary implementation plan and schedule.
Clients have found this report structure effective as it provides the right level of detail in the right order and considers the various stakeholders who may receive the report.
Conclusion
You can use our approach to conduct your own DX audit. Alternatively, benefit from an independent, expert, perspective by using our Developer Experience Audit service – book a meeting with us to get started.
Either way, conducting a DX audit is a valuable and worthwhile exercise to conduct regularly or when new features or products are launched. A DX audit can uncover tactical and strategic issues and opportunities to enhance and make your developer journey the best experience for your target developer audience.