As part of the effort to quickly address performance problems, the U hired Andersen Consulting for a three-week engagement to provide the following information:

• Provide an independent assessment of the performance problems of the PeopleSoft implementation. In common practice, what is our responsibility to fix? What is a software developer's responsibility?
• Assess the technical infrastructure's capacity (database, network, CPU, servers, etc.) to support the current PeopleSoft implementation. Have we configured everything correctly to date?
• Examine how performance issues have been addressed in the past and provide feedback on streamlining these processes.
• Recommend the fastest, most cost-effective way to meet the current performance demands and prepare for future performance needs.
• Address performance uncertainties surrounding payroll implementation.

Summaries tend to oversimplify complex issues, so please note that both sponsors and staff were very impressed with the quality of the consultants and their work.

The consultants were from Andersen's PeopleSoft Performance Group and have worked on performance methodologies for several PeopleSoft products. They know both the product and its problems intimately.

FINDINGS

WHAT PROBLEMS ARE OUR RESPONSIBILITY TO FIX? WHICH ARE PEOPLESOFT'S?
The common practice is:
The customer is expected to "tune" the database to perform to its needs by getting rid of excess SQL [unclogging the database], indexing huge tables [so the application doesn't scan thousands of names to retrieve one], and other database maintenance tasks.

The customer is also expected to tune processes which they have modified to address unique organizational needs.

The company developing the application is expected to fix and tune those processes that don't work as delivered in the "vanilla", unmodified state. The consultants said many of our problems are the result of bugs and broken processes and that the developer is expected to address those (and has, by sending "patches" or fixes").

They pointed out, however, that because of pressing business needs, most customers don't wait for the developer to deliver a fix, but correct the problem on their own. The U is following this course, since bills have to be calculated and printed and financial aid disbursed.

THE HARDWARE AND OPERATING SYSTEM ARE NOT TAXED AND ARE GENERALLY VERY WELL-CONFIGURED.
Oracle database, network, servers, CPU are performing efficiently. OIT has the most robust monitoring tools they had seen. The database is about at 40% of its capacity. They pointed our areas which could be "fixed" but these would provide very modest performance gains.

LARGEST PERFORMANCE GAINS WILL COME THROUGH APPLICATION TUNING.
The consultants were not surprised by our problems. PeopleSoft developed its product for use with many kinds of databases, so the student/human resource product was not tested on the Oracle database that the U and other Big Ten Schools are using. Because it wasn't written specifically for our database, the code is not as efficient as it could be. In addition, the student system is a "young" system.

The consultants observed student processes during the first week of the semester when it was very busy, noted problem processes and worked hands-on with staff on methodologies that resulted in greatly improved performance in the test database. These methodologies involve both panel and batch processes. They can also be used to address human resource performance problems before the payroll module is implemented.

RECOMMENDATIONS FOR NEXT STEPS:

• Create dedicated performance team with mechanism for user input.
• Tune online/batch/reports to reflect business priorities.
• Prevent future issues through design/code reviews and performance testing during development.

UNCERTAINTY ABOUT PAYROLL PERFORMANCE
Few other organizations have implemented the combined student/human resources product. The U is on the"bleeding edge." While the stand-alone payroll product has been on the market for several years, the integrated student/human resources product is new. So it's hard to predict the effect of the payroll implementation on the shared system.

Payroll will add 400+ users to the system, but it's not clear how big the workload is. We need to measure anticipated workload.

Since may of the methodologies used in improving performance of student services are "portable," they can be used to tune HR payroll before implementation. Fixes in
Campus Community (the "name and address book" of the database, will also improve HR processes).

ANDERSEN CONSULTING KEY OBSERVATIONS:

Infrastructure

• All servers examined have available capacity at almost all times. Observed situations where server utilization was excessive were traced to inefficient code.
• Database wait conditions were primarily associated with application code inefficiencies.
• No messaging queues or unusual behavior was observed on the app servers.

Performance tuning

• Developers have been hard pressed to meet functional delivery deadlines. As performance testing is generally one of the last implementation steps, it's usually sacrificed to meet schedule requirement. This isn't unique to the U. Y2K and semesters affected the ability to focus resources on performance tuning.
• Need to apply consistent performance reviews and release standards and undertake a coordinated performance testing strategy in a production-like environment.
• Hands on case studies indicate the U is not facing a hopeless situation. Many current problems can be solved in an iterative one-by-one fashion

HOW WE'RE FOLLOWING UP

• Project staff is preparing detailed plans to execute the Report's recommendations.
• Project managers are creating a dedicated Performance Team, all of whom are permanent University employees. It's important that knowledge transfer take place.
• We are asking Andersen Consulting to return one of the consultants for a short engagement to continue to work with staff on hands-on performance methodologies.
• We are considering another short contract with Andersen to more fully investigate aspects of the payroll implementation (e.g., workload, data entry complexity, and performance methodologies).

IF YOU HAVE FURTHER QUESTIONS about technical infrastructure, contact Tim Fitzpatrick (612) 624-6522 or at t-fitz@umn.edu or Nick Choban at (612) 626-7201 or at n-chob@tc.umn.edu. If you have questions about the application performance, contact Mark Powell at (612) 625-8598 or m-powe@tc.umn.edu.

We will continue to communicate further developments to you.