Evaluating Software

Seems like a good idea. A feature a find many products lack is a good clearly laid PERT of network logic view. To me this is a key feature when creating and building plans form scratch. Without mentioning product names one major vendor makes his money by providing a graphical add on service to virtually all, if not all current planning software packages out there in the market place. Perhaps the details of graphical reporting could be split to take into account: network view, barcharts, resources, wbs/ obs structure drawing etc. Usability/ configurability, etc could all be scored

I feel sure that other Planning Planeters could add to this initial stab of mine.

Just what level of detail do we "collectively" want to go to? Who would value that level of detail?

May also I suggest we go back to the vendors that have approached PP and ask them what sort of feedback they require.

This is a normal up-front PM activity - define the customers requirements.

Great idea.

Lets see what the software vendors feel they offer, and if there are differences between what we want/ need as Planners. We could work on both approaches independantly and then correlate the two and identify any differences, if any.

[quote]Just what level of detail do we "collectively" want to go to? Who would value that level of detail? [/quote]

Good question Guy!

IMO, detail is going to have an exponential relationship with the number of questions/factors to be asked/reported. Each software package has its own strengths and weaknesses. What about niche products?

Bernard Ertl
InterPlan Systems Inc. - Project Management Software, Project Planning Software

[quote]We been approached by several software vendors who have asked if we can find out more about how a planner values their software, easy of use, shortfalls and functionality etc.[/quote]

1. Planner valuation of software (overall) - This could easily be a generic 1-10 opinion rating question by itself.
   
2. Ease of use - Same as above
   
3. Shortfalls - Free-form commentary?
   
4. Functionality - If we use the question/rating format, the detail question comes into play. The free-form commentary format may be more appropriate.
   
5. Excellence - Why not point out strengths as well? Free form comentary as with shortfalls

Some consideration should be given to the date that a review is posted to match up with a particular version of the software. Vendors may improve their systems based upon commentary and release new versions. How will this system maintain relevancy? Clear the database upon new releases?

Bernard Ertl
InterPlan Systems Inc. - Project Management Software, Project Planning Software

OK, Bernard has outlined a marking / review template. Im sure that is under control.

Next, what is the process PP will employ to carry out reviews ?
Will we spotlight one product per month ? Two ?
How do we decide what to review ?
Who reviews and how many PP reviewers do we use ?
How do we recruit them ?

I suggest we do a trial run. James, please shortlist 3 or 4 of the interested parties so we can decide which one to trial. We should then approach the vendor with our suggested process and results (Bernards template) and get agreement. Once the vendor is happy, we can start.

If it is a review of software existing in the market, we can invite PP users (open invite) to give ther evaluations (based on Bernards template). Give PPers 2 weeks to reply from invite. PP moderators then need to collate (and perhaps edit) the reviews, before we post a consolidated review on PP.

If it is a review of new software, perhaps the vendor can issue, say, 10 trial versions for PP evaluation.

Remember, this is a service PP is providing. PP should get some benefit (freebies from the vendors, to be given away as prizes), and should look at its disclaimers.

We have a guy called Winston, from Microsoft who wants to see what the "masses" think of their MS Project suite. His response to our request for some questions was that he did not mind what we asked (hmmm?), but was concerned that the process could...

(a) Offer comparrative feedback such that their strengths could be measured against another competitors.

(b) Offer some means of allowing specific, personal feedback from people in the field.

Bernards template is great. Things to consider...

Set of Questions (marks out of ten) - this will allow the some quantitive analysis and feedback, giving the potential to rate one package / tool against another.

Free Form Commentary - this is good as it will offer the chance to put our specific concerns, complaints or suggestions across.

If we have a survey with a lot of textual (textual!??) answers, will this make it difficult to form a comparrison?

Our man at Microsoft (Winston, who does not think any free software will be on the cards) and would prefer a survey in the form of a set of questions (marks 5 or 10). He also wishes to leave the survey running over time and said it should be open to all the site planners.

Do we then go for...

A user free-form text box plus marks out of ten for (a) overall (b) ease of use (c) functionality etc.

OR

A user free-form text box plus marks out of ten for (a) overall (b) WBS functionality (c) network plotting functionaity (d) Text reports etc etc

Appologies for only creating more questions. It would probably be best to run with the MS Project initially, as it is well-known. However, how to we show any results in a comparative format if we do not run the survey for several packages from day one?

IMO, some background on the poll participants is necessary to qualify their votes. The Planning Planet membership is spread out across a range of industries and project management is handled differently in each. Ill propose the following comparative analysis questions (feel free to add, remove or amend suggestions as you wish):

• Set of Questions (marks out of ten)
  
  
  
  • Participant background
    
    
    
    • What industry are you in? [Multiple Choice - Engineering, Construction, IT, Rail, Petrochem/Refining, etc.]
      
    • How many years of experience do you have with this software? [Multiple Choice - Limited, 1, 2, 3-5, 6-10, 11+]
      
    • What is your current role? [Multiple Choice - Owner/Employee, Consultant, Affiliate/Reseller]
    
    
  • Software Review [all ?s = ranges/ratings 1-10 with a N/A option]
    
    
    
    • Ease of use
      
      
      
      • Software (user interface)?
        
      • System (to manage a project)?
      
      
    • Documentation
      
      
      
      • Printed?
        
      • Electronic (help system)?
      
      
    • Stand Alone Project Functionality
      
      
      
      • Collaboration?
        
      • Planning?
        
      • Cost Estimating?
        
      • Scheduling?
        
      • Resource Management?
        
      • Progress Tracking?
        
      • Earned Value Analysis?
        
      • Reporting?
        
      • Data Sharing (Import/Export facilities)?
      
      
    • Enterprise (Multiple) Project Functionality
      
      
      
      • Collaboration?
        
      • Planning?
        
      • Cost Estimating?
        
      • Scheduling?
        
      • Resource Management?
        
      • Progress Tracking?
        
      • Earned Value Analysis?
        
      • Reporting?
        
      • Data Sharing (Import/Export facilities)?
      
      
    • Vendor Support?
      
    • Overall Value (for the cost)?
      
    
    
  
  

I would prefer to limit participation in the polling to PP members and to limit voting to one time (per product) per member. If new versions are released, a new poll should be added for that release (ie. treat each version of a program as a separate product for the purposes of polling).

All 1-10 votes (excluding any N/A answers) could be averaged to display an overall review rating.

[quote]Free Form Commentary - this is good as it will offer the chance to put our specific concerns, complaints or suggestions across.

If we have a survey with a lot of textual (textual!??) answers, will this make it difficult to form a comparison?[/quote]

Perhaps just one free form box will be sufficient for the participant to address any points which the comparative analysis questions do not address adequately.

Bernard Ertl
InterPlan Systems Inc. - Project Management Software, Project Planning Software

You have a small problem here. Do I understand that we are seeking the opinion of only those people who have already bought the software and not of people who have experience in all of the soft wares being reviewed?

Don’t you think that someone who bought a particular piece of software would be biased toward that software, even without trying the other brand? Otherwise, why did they purchase it? In addition, they may think that ‘everybody does it this way’ because of their limited experience and not know that a superior feature exists in another package?

Perhaps each rating should be accompanied by a list of software that that person has used. Then their comments can at least be taken in context.

Ron, the intention with my suggested questions was to allow members to review individual software as they like. Some software will naturally have more reviews posted as I perceive the Planning Planets membership to be skewed in favor of Primavera users.

I tried to account for the bias you mention by including a question covering years of experience with the software being reviewed.

There really isnt any way to remove all bias from an open (and anonymous) survey.

BTW, Ive catalogued about two dozen different Windows based project management systems in my Project Management Software Directory. There are also many web based systems. Are we going to include them all??

Bernard Ertl
InterPlan Systems Inc. - Project Management Software, Project Planning Software

That is a very complete list of software on your site! Might I suggest for our purposes, that we also break-out the software by major version where appropriate?

MS Project 98 is far different from MSP 2000 and that is far different from the MSP Enterprise version available today. Lumping these together would confuse, rather than clarify the responses.

Wot no Artemis?

Nope, no Artemis - they have not replied to us yet.

However, we will still include them, and they cant say we did not offer / request their assistance!

We have had a good response from many suppliers.

It is now only a case of finding the time to formulate a list of questions for us all to answer - thereby generating the results.

We have also been asked to allow the software people the option to reqview our set of questions before we go live with the survey.

We have had several emails suggesting that the survey should offer the voter the choice of which packages he / she wishes to review. Following on from these, options would be provided in order to grade / score each (comparatively) against each other.

Sounds like an interesting thought.

Anyone of you PPers with to volunteer to write the questions using input from the other members (who are interested) ?

James,

A step back. Lets plan what we are to do:

Given that you have collated all the responses from vendors, can you define the objectives of the exercise ? When we have defined the objectives we can begin to structure the survey / questions set to give the required output. Can we also agree some tasks and deadlines for the process before we issue the survey / questions set to the vendors ?

A salutary lesson, which Im sure we are all aware of - dont charge off and start doing the work without adequate planning !

Hi there,

Objectives of the Exercise
To develop and investigate the differences (strengths and weaknesses) of various Project Planning software packages. The primary aim being to be able to compare the packages wherever possible.

Tasks and Deadlines
We ideally need someone in the PP group to take it on themselves to draft a set of questions. The simplest thing, in order to be able to compare these packages, is to offer a picklist of set answers wherever possible.

In my humble opinion it would be good to define tasks and deadlines once one of the PP member-group has come forward to take the lead to collate answers/opinions and write out useful questions - perhaps from some forum QandA etc.

Some consideration needs to be given to the fact that some packages have specialist objectives - so we should endeavour to avoid penalising them for not offering all of the usual common planning-tool functions etc.

What do you think?

PP Team,

Good news! I was beginning to wonder if this project was dead.

I have already offered some input that may help form a base for the study in my post dated 28 July in this thread. I am willing to help where I can.

Bernard Ertl
InterPlan Systems Inc. - Project Management Software, Project Planning Software

Bernard,

I have added to your question set (from this thread) and sent it to you and James by email. I do not have an appropriate site to post it to - grateful if you (or James) would.

We can then get it refined, before checking that it will give the desired results.

Mark

Great job Mark! I have posted the Word document to our server for all to review: Proposed Planning Planet Survey Anyone who is interested should feel free to offer comments regarding the survey either by posting here in this thread or e-mailing me (bernarde at my company domain), Mark (through the PP member link) or the PP team. Please do not e-mail us with your answers / completed survey at this time! We are seeking comments to refine the survey questions. Please help ensure that this survey will be a useful tool for everyone to compare the relative strengths and weaknesses of available commercial project planning and project management software. Bernard Ertl InterPlan Systems Inc. - Project Management Software, Project Planning Software

Thanks Bernard,

May I now suggest we set a two week limit for comment. By close on Thursday 25 Sep 03 ?

Following that we can offer to the vendors for comment before starting to run surveys.

Mark

Yes, two weeks is more than reasonable for everyone who is interested to have chance to offer their input.

Folks, opportunity is knocking. If you want a tool to compare software packages, act now to ensure you get something you can use.

Bernard Ertl
InterPlan Systems Inc. - Project Management Software, Project Planning Software

James, can you also put an alert on the homepage that we are requesting comment on this survey. It may prompt more people to respond. Thanks.

Mark

Gents,

I have changed the main front page as suggested. Lets see what help we get. I have been in Hong Kong on business, I will review your most-impressive question sheet!

Is it too late to add a vendor to the evaluation list?
If not, I propose we add Concerto from www.realization.com.

Concerto adds the dimension of managing uncertainty and appears to have a very good track record in terms of increasing project througput.

Does this argue for adding an effectiveness component to the evaluation criteria?

Would adding a line item for "Risk/Uncertainty Management" to the Project Functionality categories cover it?

Perhaps the PP team can contact Realization to see if they would like to participate in the survey?

Bernard Ertl
InterPlan Systems Inc. - Project Management Software, Project Planning Software

"Risk/Uncertainty management" sounds like a good category to add.
My suggestion was also to meant to include measuring how effective the introduction or use of PM software was to the implementing organization. Can this be measured? Is there a common metric across organizations? A measure of total project days per employee per year. Measuring the total departmental project completion rate per employee over time would allow you to see the effectiveness of the PM system.
This may be a moot point however as I doubt most organizations would have any historical information on this measure.

Dear All,

It appear to me that the task of comparing these software packages is becoming complicated. In fairness, we want to make sure we are comparing apple to apple and not car to a truck.

Maybe, we could make the comparison in two parts. The first part comparing the basic functions of the planning softwares and the second part comparing the specialist functions. We need to categorize the the planning softwares in the second part.

Just an idea.

Regards,

Nestor

Nestor,

Have you reviewed the proposed survey? Can you offer specific recommendations in line with your thoughts? Please post a sample of what you propose.

Bernard Ertl
InterPlan Systems Inc. - Project Management Software, Project Planning Software

Nestor,
I agreed with you, lets talk and share our information based on systematic regulations and after that make a suitable conclusion .

I had presumed that all the packages we were listing were PLANNING packages. If, as Nestor points out, we widen to non-planning packages then we cant compare.

If we stick to planning packages only (for now), does the survey give enough depth ? We may wish to break out under "functionality" to include progress spotlights, EVA, resource loading, filtering, etc., etc.. Thaoughts on this are possibly what the vendors may be after.

Can you provide a clear definition of PLANNING package to avoid confusion?

IMHO, the current survey as proposed is adequate to allow comparing of all manner of project planning, scheduling and management tools. The 1-5 rating system with the inclusion of a N/A option should allow direct comparisons of any feature via a database search on an apples to apples basis. The key is to ensure that the survey is asking for all the important features (and at an appropriate level of definition to be meaningful).

In reviewing the survey again, I think the software reviewed question should be a pick list instead of a fill in the blank to ensure proper categorization of the results.

Also, I would like to suggest the following changes

• progress tracking - break out the different possible measures of percent complete (time, effort and cost) as sub-items
• data sharing - break out import and export as sub-items
• planning - add sub-items for more definition:
  • pre-defined coding structures
  • user defined coding structures
  • organization (work order / WBS / etc.)
  • data re-use facilities (templates / database / estimating engine)
• scheduling - add sub-items for more definition:
  • critical path scheduling
    • speed of calculations
    • PDM interface
    • ADM interface
  • critical chain scheduling
    • speed of calculations
    • PDM interface
    • ADM interface (are there any ADM style CC systems?)
  • calendar based scheduling (no logic network support)
• cost estimating - add sub-items for more definition:
  • detail
  • summary
• reporting - add sub-items for more definition:
  • pre-defined (hard coded) reports
  • ad-hoc (user defined) reporting
  • presentation
  • readability
  • useability

If sub-items are defined, perhaps the parent item should not be a user defined field - it should be calculated as an average of the sub-item responses.

Also, all listed applications in the survey should include a version number (ie. P3e/c vs. 3.5 or whatever) as survey results may change across different versions of the same product.

Hopefully this will spark some additional ideas...

Bernard Ertl
InterPlan Systems Inc. - Project Management Software, Project Planning Software

This sounds very good. May I suggest we add a Collaboration section (e.g. Project portal, Exchange or Lotus Notes integration...)?
It should include some measure of ease of use and navigability (up stream task, down stream tasks)
Also some sense of the roll-up facility (costs, timelines. Finally portfolio management would seem to be an important feature, particularly for multi-project matrix organizations.

Jacques, good ideas. Can you lay them out in a more specific fashion? Is the following what you had in mind (for the 1-5 rating system with N/A options)?

Collaboration

• (Proprietary) Project portal
• Microsoft Exchange integration
• Lotus Notes integration
• Ease of use
• Navigability
  • Up stream task
  • Downstream task
• Roll-Up Facilities
  • costs / accounting
  • timelines
  • Portfolio management

This is not my area of expertise, so the above may not make sense - please offer suggestions.

Bernard Ertl
InterPlan Systems Inc. - Project Management Software, Project Planning Software

Thats pretty good for someone out of his area of expertise. Heres my suggestion:

Collaboration engine:
Project Portal (proprietary)
MS Outlook
Lotus Notes

Navigability:
Click to view upstream tasks
Click to view downstream tasks
Click to view tasks assigned to same resource
Click to view all tasks controlled by task manager

Roll-Up facilities:
Costs/Accounting
WBS levels

Portfolio Management:
Ability to rank projects by priority
Ability to perform "what-if" analysis impact on all projects in the portfolio
Ability to synchronize task priority with project portfolio priority

Effectiveness:
The use of this tool has increased project throughput (more projects per year without increasing resources)
The use of this tool has eased the PM administrative burden
The use of this tool has improved project delivery (on-time, on budget, in scope)

One of the problems that I foresee is that respondants may not know how to evaluate some of these (e.g. project throughput)

JAS

Thanks Jaques,

When we get around to restructure the survey, based in feedback herein, should we consider:

Ease of Use
Usefullness

as the two main headings ?

Ease of use then splits into:

Input
Process
Output

containing all the information we have gathered to date.

Usefullness can then follow Jaquess suggestions, either on a personal level (a planner may feel it makes him / her more effective) or a corporate level (either with or without data to back it up).

Mark, Jaques, et. al.,

Ease of Use is already included in the survey with two subtopics (though they need to be indented for clarity). Do most people think that Ease of Use (and/or usefulness) should be a subitem of every listed feature instead?

Bernard Ertl
InterPlan Systems Inc. - Project Management Software, Project Planning Software

So there are three things we want to measure then:
1) Functionality
2) Ease of use
3) Usefulness/effectiveness

Certainly 1 and 2 above could be measured for each feature. For item 3 it may be more productive to assess the entire PM package or major components of the package (e.g. resource leveling, portal...) to determine relevance.

Also a clear definition of usefulness and/or effectiveness will be required. What may be useful for an individual is not always effective for the organization. We should only measure one or the other and ensure that the respondents understand the meaning of the measure.

Team,

I would love to be involve in the survey.

Cheers

Alex

Jacques,

on effectiveness we could either give a "tick all that apply" list of statements, eg:

has improved my effectiveness as a planner
has improved my departments throughput
has improved my companies profitability
etc

or
give a 1-5 and n/a grading option against a similar series of statements.

Alex,

thanks. Any comment on the survey structure so far ?

All,

we may also want to consider a best feature / worst feature type-in box.

Mark

Mark,

That looks like a good solution for measuring effectiveness.

btw, Ive contacted Realizations, the company that markets Concerto. They seem interested in participating.

Regards,

JAS

In order to maintain the original aim of this survey - to be able to compare different systems against each other - I would prefer to maintain the 1-5 & N/A rating system.

While measuring effectiveness is a great idea, the latest proposed format really does not offer any meaningful basis for comparing two (or more) different systems. IMHO, there are too many external variables that affect a planners/companys implementation. I think the focus should remain on the capabilities / features of the software.

Perhaps we can expand upon the functionality survey question and break it down into subitems:

• flexibility
• usefulness
• robustness
• etc.

Alex (and everyone), we are currently generating the format for the survey. We invite you to share your thoughts. Please offer specific suggestions.

Bernard Ertl
InterPlan Systems Inc. - Project Management Software, Project Planning Software

Bernard,

I think you are correct in your assessment that we are getting into a subjective area. Since I initially brought up the issue of effectiveness I am now willing to remove it, if no one else minds. This would allow us to focus on comparing package features.

Is there a new version of the Proposed Planning Survey as well as a list of the packages to be evaluated?

JAS

OK folks, tomorrow is the deadline for suggestions. Please take a minute to review the proposed survey (Proposed Planning Planet Survey) and offer your input before it is too late.

Bernard Ertl
InterPlan Systems Inc. - Project Management Software, Project Planning Software

Looks good. Here are my comments:

Under "What software do you also have experience with"
PLEASE ADD:
- Concerto - Realization Technologies, Inc.

Under "Stand Alone Project Functionality" and "Enterprise Project Functionality"
PLEASE ADD:

- Navigability:
Drill through to upstream tasks
Drill through to downstream tasks
View tasks assigned to same resource
View all tasks controlled by task manager

- Portfolio Management:
Projects by priority ranking mechanism
Task priority ranking mechanism
Ability to perform "what-if" analysis impact on all projects in the portfolio

- Critical Chain Buffer Management

Please let me know if you need clarification.

JAS

Hell, thats what we needed, a doctoral research student ! The size of that survey !

Seriously, the posted survey is about the use of PM software and ours is about comparative functionality of Planning software. But there may be things in there that could help our survey.

Lets review it when we close comments, i.e. today.

Mark

I took a quick look at Abdullah Bani Ali`s Project Management Software Effectiveness Survey and it appears to have a slightly different focus than what we are aiming for here.

I think we could probably incorporate some aspects of

• Section I - Frequent of Use (sic)
• Section I - Extent of Use

Bernard Ertl
InterPlan Systems Inc. - Project Management Software, Project Planning Software

I disagree. IMHO the survey is not too long.

Ive also noted that it is lacking the suggestion from Bernards 16 Sep 2003 at 08:36 post. These would be a valuable addition.

JAS

Jacques,

Mark was referring to Abdullah Bani Ali`s survey (linked in my previous post).

Also, the PP survey (linked previously) has not been updated to incorporate any of the suggestions mentioned here yet. We will be working on that shortly.

Bernard Ertl
InterPlan Systems Inc. - Project Management Software, Project Planning Software

Never mind!

Thanks Bernard.

JAS