How difficult is it to test the solution (especially in an automated manner)? If the solution is an appliance (such as Microsoft Word), it needs to work. Initial selection and subsequent upgrades may rely more on manual, exploratory testing. However, as we begin to introduce customizations and configurations, the importance of being able to setup automated testing (along with other development features) grows.
Is it clear who is responsible for testing the functionalities, customizations/configurations, and integrations with your internal systems? Which testing activities are the responsibility of the vendor? What kind of test results do you get of the vendor?
Do you have a system in place for managing requirements, tests and bugs? Can it be integrated with vendor’s tool?
Are there manual or automated regression test sets available on existing and new functional flows, customizations/configurations, and integrations? How frequently are they executed?
Who is responsible for your internal testing? Do you have a team of dedicated (automation) testers?
Which test environments do you use to run your tests and who configures them? Are they integrated with the vendor’s testing environments?
How simple is it to create or access test data? etc…
