The 4 basic phases of a software evaluation project

Selecting the proper software solution that will satisfy both business and technical requirements can be a daunting task for anyone who has never gone through this process.  Vendors with expertise in a certain domain (ie: CRM, ERP, LMS, etc.) will typically boast a “proprietary” methodology that can be used to help in identifying the best software solution for a business problem.  However, these “propriety” methodologies tend to utilize criteria that skew the advantage towards the vendor’s product.  I have been engaged by clients in various software evaluation projects to deliver a non-biased recommendation.  I have found that there are 4 basic phases that are common no matter the business solution is being evaluated.   

   

The 4 basic phases

The following example below is a 4 phase approach utilized by PTCI for selecting specifically a LMS (Learning Management System) solution.  One can substitute the “type” of solution being evaluated depending on the particular initiative.   In this example, “learning management”  can be replaced with another solution such as “customer relationship management” and “LMS” can be replaced with “CRM“ (Customer Relationship Management).

 Phase 1: The Discover Phase

This phase will evaluate the responses from the LMS vendors. Utilizing a scoring system developed by PTCI and with input from the client, the responses to the RFI is evaluated and a score is associated with each response from the vendors.  Based on the score, the vendor list will be trimmed to the top 3 or 4 vendors.  The top 3 or 4 vendors will be invited “on-site” for a 1 to 2 hour demonstration of their product. This will provide a chance for all interested users to get a first hand look at each LMS application.  Based on feedback from users, the list may be trimmed down to the top 2 vendors.

 As part of this phase, test case scenarios will be generated by PTCI that will be used during the on-site demonstration.  The test case scenarios will be given to vendors in advance of the on-site demonstration.  Vendors will be asked to demonstrate product capability based on the custom test case scenarios.  The test cases will demonstrate how closely the “out-of-the-box” LMS application can meet the specific user requirements. 

 Deliverables for this phase are:

1.  Test case scenarios – based on “use cases” and functionality matrix documented in the Discover Phase.
2.  Updated Functionality Requirements Matrix – matrix will be updated using a scoring system developed by PTCI and client.
3.  Identification of top 2 LMS vendors
4. Gap analysis and documentation for the top 2 LMS vendors.  This will include changes to workflow or user interface
5. Project Plan – the plan will updated if necessary

Phase 2: The Evaluate Phase

This phase will evaluate the responses from the LMS vendors. Utilizing a scoring system developed by PTCI and with input from the client, the responses to the RFI is evaluated and a score is associated with each response from the vendors.  Based on the score, the vendor list will be trimmed to the top 3 or 4 vendors.  The top 3 or 4 vendors will be invited “on-site” for a 1 to 2 hour demonstration of their product. This will provide a chance for all interested users to get a first hand look at each LMS application.  Based on feedback from users, the list may be trimmed down to the top 2 vendors.

 As part of this phase, test case scenarios will be generated by PTCI that will be used during the on-site demonstration.  The test case scenarios will be given to vendors in advance of the on-site demonstration.  Vendors will be asked to demonstrate product capability based on the custom test case scenarios.  The test cases will demonstrate how closely the “out-of-the-box” LMS application can meet the specific user requirements. 

Deliverables for this phase are:

1.  Test case scenarios – based on “use cases” and functionality matrix documented in the Discover Phase.
2.  Updated Functionality Requirements Matrix – matrix will be updated using a scoring system developed by PTCI and client.
3. Identification of top 2 LMS vendors
4. Gap analysis and documentation for the top 2 LMS vendors.  This will include changes to workflow or user interface
5. Project Plan – the plan will updated if necessary

 Phase 3: The Select Phase

In the event that the LMS is to be installed and supported internally (vs. ASP), it is  recommend that vendors agree to install their product on-site for a 1-week trial period.  This provides the opportunity to:

1. Determine product installation issues
2. Compatibility with organization’s technical architecture
3. Additional opportunity for interested users to “demo” the product

Documents that were generated from the prior phases will be utilized to provide the client with the recommendation for the appropriate LMS product that meets the highest percentage of the requirements.  Based on our findings, PTCI will make a recommendation for the LMS suited for the client’s specific requirements.   If required,

PTCI will also develop a “Build vs. Buy” Comparison document in the event the client is interested in a custom solution. 

 Deliverables for this phase are:

1. Recommendation of the Learning Management System
2. LMS Vendor Evaluation and Selection Executive Summary
3. Build vs. Buy Comparison (if required)
4. Implementation Cost Estimate (if required)
5. Project Plan –  the plan will updated if necessary

 Phase 4: The Implement Phase

 Once the client has selected the vendor, PTCI will develop implementation detail design necessary to insure that the base employee data is converted and ongoing data interfaces to/from the LMS system is properly documented.  PTCI will work closely with the selected vendor to ensure that conversion and interfaces work seamlessly with the LMS package.  An implementation project plan will be developed to ensure that the appropriate tasks and milestones are identified and can be easily tracked.  PTCI can also provide the technical resources to work on the conversion and system interfaces to/from the LMS application.

Deliverables for this phase are:

1. Detailed conversion design and plan
2. Detailed system interface design
3. Updated Cost Estimate (if required)
4. Additional technical resources (if required)
5. Project Plan –  Detailed Implementation work plan with identified tasks and milestones.

 This basic 4 phase methodology can be used with flexibility in mind.  With a little thought, the format of the deliverables for each phase can easily be created as either an Excel or a Word document.  Hopefully, this information can provide a basis of understanding the basic phases of a system selection process.  At the very least the 4 basic phases discussed here can be used to compare with other “proprietary” vendor evaluation methodology.

Advertisement