Increasingly, there’s an off-the-shelf solution for every use-case and, at first glance, they all look pretty impressive – but that’s often because their investment is channeled at marketing gloss. This makes the job of selecting third party software and software assessment a difficult one. The goal of course, is to leverage existing solutions rather than reinventing the wheel but how can you judge quality? We find there’s no substitute for rolling up your sleeves and testing a vendor’s APIs, before signing any contracts. The process will also reveal how good their technical support is, and how good a fit the vendor is for your project.
Acquisition – you need users, but you really need the right users.
Before you even start looking at the vendors operating in a particular space, you should first ask yourself whether the feature set you’re focussed on really needs third party software in the first place. If you’re in the business of software development, then it may be that your own team could custom develop what you need fairly easily. Often vendors offer a broad suite of features for any given product, and it may be that you’re only really interested in one or two components. Ask your development team how long these components would take to develop yourself – are they really so complex?
Also remember that using a third party tool has its downsides. The software has not been written for your specific needs, so the team will have to accept compromises. Things may not work exactly as you want and existing solutions are often so feature rich that management interfaces can be complex from a UX perspective. A custom solution with more limited menus is often easier for your users to get onboard with.
The time to leverage existing solutions is where the cost and effort to develop something massively outweighs the monthly cost for an existing service. For example, there is such a wide range of features that users expect of a content management system (CMS) that to develop these from scratch would cost your client way more than integrating with a system like Contentful or Sitecore.
Leveraging an existing service also makes sense where a vendor offers domain expertise that would take years to develop yourself. Payment providers and messaging services for example take care of a huge amount of complex integration work, requiring agreements with multiple other third party providers (eg. banks, phone networks).
Short Listing Vendors
Generally the first stage of a vendor assessment is to identify a short list of vendors operating in the space you’re focussed on, to take to a second phase involving a more in-depth exploration of their products. Try to get a broad range of company types into your list – make sure one of the more established players are considered, but also try to find some smaller companies or startups. Though their client list may not be as impressive and their feature set will almost certainly be narrower, they may be cheaper and their solutions may be fresher with less legacy code.
The initial search will typically start in Google and you’ll spend hours reading vendor home pages. It can be difficult to determine from a software company’s website exactly what services they provide, and in particular what their core business and strengths are. Many vendors employ a scatter gun approach to the product features they advertise, for a combination of SEO and upselling purposes. Understanding what makes a company unique and what part of their offering they do really well is often hard to determine just from their home page. As early as you can, it’s worth arranging an introductory meeting.
Get the vendor to demo their product. Ensure you have technical and business or strategic representation on both sides at the interview. Be prepared with a set of questions which you should ask each vendor consistently (as far as possible). You may find producing a scoring matrix is a useful way to get you thinking about the key requirements of the system you’re looking for, and help you to structure your interview. Include both functional and non-functional requirements.
Gauging Company Fit
It’s important that the vendor you choose is a good “fit” for your business. On the one hand, you don’t necessarily want to be a vendor’s biggest client, as they may not have the team or infrastructure in place to cope with the demands of your product’s users. On the other hand, you may not want your project to be insignificant to a vendor, as they are likely to be less interested in providing support to get you started, let alone developing the new features or bug fixes you may need.
When you ask about features that aren’t currently supported, is their reaction one of interest or defensiveness? Ask about the size (and location) of their development team and the cadence of their release cycle. What’s the process for feeding into their development roadmap?
You can often tell how much of a priority your project is to a vendor by the seniority of the people you meet at your interview. You’re unlikely to get the CTO of Stripe in your round of payment provider interviews, of course, but you should expect a well informed cross discipline team to join your session, and this should certainly include a senior technical contact. If you’re speaking with a smaller vendor or a start up, you should expect to be given access to C level contacts and not just a sales rep.
In one interview with a big name provider of a trading platform, our team was only able to speak with one sharp suited sales rep who asked virtually no questions about our project. We were told that their eye-watering prices were non-negotiable due to their brand reputation, before he admitted they didn’t currently have any clients using the product we were enquiring about. This was a bad combination and, clearly, not a great match for our startup project.
Getting Your Hands Dirty with Software Assessment
Reading websites and interviewing the vendor team will only get you so far in your goal to acquire a thorough understanding of the features and quality of a vendor’s service. Once you’ve got a shortlist of up to half a dozen companies, it’s time to get hands on with each vendor’s product.
The process of testing a product will tell you a lot about each provider. How willing are they to give you a sandbox to do some API testing? Having to sign an NDA is to be expected. If the vendor requires you or the client to sign up for a contract before they’ll even spin up a test environment, alarm bells should be ringing.
How easy is their documentation to use? Are the endpoints you’ll be using easy to find and described thoroughly and accurately? Extra points if they use a standard format such as Open API, but not if it’s just a dump of every endpoint across all of their products. The best documentation is both thorough and structured – ideally presented to you in a guide tailored to your use case.
You will get stuck and you will have questions – this isn’t necessarily a problem, and it’s a chance to test the responsiveness of their support team. Reach out to your technical contact or, better still, use their regular support channels. Was your issue resolved quickly? Were their support staff approachable and friendly?
Focus on testing a sample of key endpoints, or prototyping your core user journeys. You shouldn’t expect to have to spend more than a few hours per vendor before you feel comfortable that you understand how their product works, and that it does what they say it does. If it takes much longer, it may be a sign that the vendor’s product or its documentation is immature and this is likely to frustrate you further down the line. First impressions count.
Software Assessment: Decision Time
The chances are, you’ll run out of time before you’ve filled out every cell on your carefully crafted vendor assessment matrix with the comprehensive scoring system and stress tested every product feature. But don’t beat yourself up. The initial research, vendor interviews and API testing are all just paths on the journey to your decision – each one an opportunity for your candidates to impress you within a finite timebox – ideally just a week or two from start to finish.
If you ended up spending longer chatting to the support team of one vendor, or your prototyping got further against one vendor’s API, or you had a much more in depth discussion around a whiteboard during one vendor interview – don’t feel guilt over inconsistency – these are the indicators that you’ve found your preferred vendor, and are already at the start of a great collaboration.
