Special type of buyer's remorse
We often have the feeling after buying something that it was not the right choice. There is a phenomenon called buyer's remorse over an expensive purchase, but it's not what we are talking about here. What I mean is that you have bought an expensive bicycle and somehow missed the fact that you most often go for a walk with a one year old son - who is too small to ride a bike. Well that's the way it is...
To no surprise, many people do such a think quite often with small purchase, like smartphone/tablet apps or candies in a cake shop. You have a certain need that need fulfillment, then you came across an add or description which sounds just right so you buy it. It might even pass you simple quick check just before you pay for it.
I had this happen to me recently when I bought an webcam, I wanted to use with my laptop. It worked quite well in Windows environment, however what I learned after buying was useless in any other operating system. I tried to use it under Ubuntu, without luck.
Take advises from other with a grain of salt
Such a scenario can often happen regarding choosing the right development tools. People often take advice from others who were using the tool (often in their private projects or some other constrained environment). When it comes to testing such a tool the
proof-of-concept seems to be just fine. Piloting the project succeeds. The the troubles arise during deployment. Many configuration problems, trying to stretch the current process to include the new tool, missing required metrics, too many false positives... and so on, and so on... I think these issues are quite prosaic.
We end up with a bitter pill to swallow when we agree that the selection process was unsatisfactory.
Most proofs-of-concept that can be seen are really just basic comparison of different tools. Then someone who work in isolation or at least it looks like he is, have an idea about what the needed tool or tools should do and so a good old features checklist is created. At times a single vendor often helps in creating such a bake-off.
Different products are put on such a list just looking at their features listed on some wiki or documentation (not often outdated?). What is missed here is the holistic view of what such a tool can offer within the whole process it is going to be put into. How is it going to fit? What are the output from such tool? Is it going to be easily integrated? Can it be easily replaced? It the tool easy to be adjusted to the needs of the environment it is place into?
Put anything new into current working environment perspective
In addition, this technique tries to take into account the biggest costs and most likely obstacles to success. In order to select the right tool for the job it is crucial to take into account the way in which you work, your team works and your process is described.
Let's assume that your team work in Eclipse on a daily basis and you have selected a tool without proper
Eclipse backup - you are forcing your teammates to spend time using a second tool, wasting time switching between two windows, having to configure another application to meet their needs.
Furthermore when they are given results, they are not placed in developers' work-environment - the
Integrated development environment. It has to be further changed in order to be useful. Such small struggles tend to bracket together over time and people, carrying a significant burden withing the team.
For example, some time ago people got inspired with the idea of batch checking for static analysis. What was most important was the ability to perform software analysis without actually executing the code (so called static code analysis).
Contriving techniques like pattern-based analysis - which were used by SCA (static code analysis) where patterns of wither good or bad code are used as rules or checkes and a comparison is made between them and the source code - no even compiling and executing it. That gave a way to pottentially easy find metrics, common errors, check code style, ensure there are no coding rules policy violations... and show the results to the developers to act as quickly as they can to fix the problems early in the releace cycle.
What is important here is to make sure developers are not given the results on e-mails or just some external tool, but as close to their daily working environment as possible - so make sure IDE support is a crucial feature on your list.