The 37 Dimension API-as-Product Assessment Framework

We’ve developed an API-as-Product Assessment Framework that we’re using to assess public APIs. We’re sharing this framework because you will likely find it to be a useful tool for understanding your own API as a product or set of products.

The API-as-Product Assessment Framework has evolved through a combination of years of experience working with others’ APIs and developing APIs ourselves as well as a recent high-level market scan, and more in-depth analysis of a handful of APIs (e.g. Xero and the bank NAB). It looks at an API from multiple perspectives, like through the lens of product management, and not just the technical implementation.

This framework is purposefully trying to move beyond slideware from vendors to a practical understanding of what makes a great API, grounded in real-world data. It’s also trying to bridge the gap between technical best practice and product best practice to create a holistic view of your API.

The framework has been designed so that you can take it and assess how great your organisation’s API is (or not) in a more thoughtful and considered manner.

Overview

The API-as-Product Assessment Framework is broken down into the following key areas:

1. Onboarding – the ease and speed with which you can get started using the API
2. Documentation – the information available to help you understand the API 
3. API Design – the technical design of the API itself
4. API as Product – how the API fits with product disciplines
5. Community – how users are engaged together around the API and documentation
6. Other – anything that doesn’t fit into the above, or its own key area, but is deemed important.

Key Area 1: Onboarding

Onboarding is usually the first area that any user of your API will touch. They will need to get onboarded to access documentation, run test queries and begin developing against your services.

At best, a great onboarding process can be a source of growth for an API and a business. In many cases, it isn’t this extreme, but a smooth onboarding will assist with productivity and reduce support costs. At worst, a poor onboarding experience will mean developers choose other providers or create friction in your relationship with them.

Registration

This is the part of onboarding where you get an account and/or access.

Getting Started Help

The guidance and assistance provided to make first time use straightforward.

Getting to Your First Query

This is a measure of how quickly a developer can run their first query.

Time from registration has been included because for some APIs it is the registration process that requires background checks and other activities.

Getting to Production

This is a measure of how quickly a developer can get a query to the API running against the APIs production data/systems.

Key Area 2: Documentation

After someone has onboarded, they are going to need to start working with your API, which is where the documentation comes in. Strong documentation will mean little to no human interaction is required. Poor documentation will result in high support costs and developer frustration.

Key Area 3: API Design 

This key area looks at the technical implementation and design of the API itself.

Key Area 4: As Product

Great APIs are developed as products.

Key Area 5: Community

Great APIs have engaged communities. Engaged communities assist each other with solutions and help drive the API forward in the best way possible. Community members will often become evangelists that help promote the API to others.

Key Area 6: Other

This area is reserved for capturing any other information that is relevant while assessing that hasn’t been captured elsewhere. 

Over time, as we develop this assessment framework, we expect to merge common-themes from this area into the other key areas or new key areas.

In doing the assessments we have done so far, this section has been a useful placeholder.

Notes on Using the Framework

Why are some evaluations able to be left as blank? When doing the assessment of some criteria (like time to register) the evaluator can leave it as blank. Leaving as blank might be necessary if the criteria is too hard to evaluate in a commercially reasonable timeframe.

Download the Template

---