The Importance of Stress Testing in ERP Implementations
Posted on: January 29, 2015 | By: Tim Lovely | QAD Distribution, Professional Services, Microsoft Dynamics Manufacturing, QAD Financials, Microsoft Dynamics CRM, Microsoft Dynamics AX/365, Microsoft Dynamics GP, ERP Selection, QAD Manufacturing, QAD Business Process
One definition of the word stress is: “pressure or tension exerted on a material object”. Stress Testing, often called Performance Testing, is focused on the transactional stress on an ERP system to validate the systems readiness for a Go-Live in a true Production environment.
As part of any ERP implementation, the core steps within a methodology consist of the following basics:
• Project Team Training
• Process Design
• Procedure Development
• Development and Unit Testing
• Data Conversion
• User Acceptance Testing
• End User Training
• Cutover Planning and Go-Live
• Cutover Support
The list above can be viewed as over-simplified in the face of normally complex ERP implementations. However, Stress Testing, also known as performance testing, is often overlooked. The tasks above tend to be done in isolated environments with a small subset of actual production users and transactional volume. Poor performance does not manifest itself in these portions of the project for this reason. Project Team Training, Process Design, and Procedure (work Instruction) Development are usually done in an initial environment by the Project Team, with a limited set of users. Development and Unit Testing, if required, is often done in a stand-alone Development environment used only by the development team. Depending on approach, Data Conversion testing, User Acceptance Testing and End User Training may be done with additional load and users, but still will not represent Production Volumes and scenarios.
It is critical in any ERP implementation, especially ones where multiple sites and users are involved, to schedule, coordinate, and execute a thorough Stress Test.
The items listed below are critical to the planning of the Stress Test:
• Documented Stress Test Plan – A Stress Test which simply requests users to enter transactions in their ERP system will not be effective. Planning and coordination is required. A documented time frame, perhaps over two to four hours, is required. Additionally, the plan should document what each specific user will be doing, including type of transaction and equipment used for the transaction (ie PC, Laptop, Wired or Wireless, Scanner, etc…). Expected results should also be noted in the plan. Often it is wise to plan for failure, to make sure users understand what can reasonably go wrong in a normal ERP environment. Examples could include expected record locking or slower, yet acceptable, performance during peak transactional times. The documented plan should be repeatable in case multiple iterations are required.
• Timing, Meeting, and Communication – A set web meeting with dial in is recommended for all users involved. This allows an open line of communication when issues are encountered, allowing a facilitator or Stress Test manager to openly inquire about an issue, confirm as much detail as possible for eventual resolution, and allow a screen share with the users if required. Issues should be logged centrally with as much detail as possible. Priorities should be given that reflect, at a minimum, whether or not the issue must be resolved prior to Go-Live.
• Production Server – There are often significant differences between the performance obtained from servers used in Training or Development, for example, and the eventual Production Server. Any Stress Testing should be done on a server that closely mirrors the eventual Production Server to mirror true performance after go-live.
• Realistic Load Volumes – Although difficult, the Stress Test should include realistic transactional volumes that closely mirror higher volume times anticipated at cutover. Examples of this would include multiple users across multiple sites doing PO Receiving transactions, Truck Loading transactions, and Production Reporting transactions at the same time over the course of the Stress Test. After all, this is what will happen in a Live environment. Often due to lack of resources available to accomplish this, automated transactions coordinated and monitored by an IT group can be mixed with actual user entry of transactions to support the Stress Test.
• Network and Infrastructure Planning – Prior to the Stress Test, the internal IT team needs to validate that the network used in the Production environment is the same as that used in the Stress Testing. If hosted in a Cloud by an ERP vendor, the network lines of communication should be validated and tested prior to the Stress Test. Any wireless networks used or wireless access points used for RF Bar Code data Collection should be performance tested prior to the actual Stress Test. These tasks can typically be completed working with an internal IT Infrastructure team or Third Party vendor that specializes in these areas.
• Third Party Systems Integration Points – Integration performance testing with third party systems is often overlooked. Examples to consider are MES interfaces to load production reporting into the ERP system, bank file data loads for bank reconciliation, or loads to/from other customer or supplier facing systems such as EDI or portals. These transactions represent additional load and data which could affect other processing performance or result in unexpected record locking if not managed.
• Transaction Coordination and Perspective – As part of the documented plan, consideration should be given to test realistic scenarios based on when actual processes will be run. Certain processes, such as a full Regenerative MRP, are usually not run during the course of normal day time processing. They are often run once per week or once per day at night, when users are more limited or not on the system. In this example, the Stress Test may not include a full MRP Regen at the same time as other users are mirroring standard day time transactional volume.
• Specific Scenario Planning – There are often key scenarios or transactional processes that must run concurrently in a Production environment. These scenarios should be documented in the plan and coordinated as part of the testing. An example of this may be to make sure that two users can load a truck at the same time against the same picklist or shipping record. To confirm this is viable, specific data set up may be required to support this testing and two more users may be required to utilize their data as part of the testing
The items below are issues that the Stress Test team should watch out for as part of the testing:
• Record Locking – Normal record locking is expected in an ERP system. This may include a system message to a user that the record they are trying to update is being updated by another user. Most ERP systems release the lock once the first user is done with the record. These expected locks ensure transactional integrity and are coded into the ERP system as part of the initial development. Unexpected record locking often occurs due to poor configuration of control files or custom developed programs which have not been scoped or developed properly. Each record lock found during the Stress Test should be documented and categorized into groupings discussed above.
• Speed of Transactions – As part of the plan, expectations on transactional speed should be set. This is often difficult to do for each transaction, but as a general rule Shop Floor Transaction prompt response time should be second or sub-second with transactional commit slightly longer, depending on the complexity of the transaction involved. As an example, if a user is testing a picking transaction scanning skids from storage racks, they cannot be expected to wait multiple seconds after each scan.
• Response Time of Reports/Browses/Inquiries – Reporting response times often vary, depending on the amount of data queried and returned. Reasonable response times should be expected which allow the business to operate and obtain data required on a daily, weekly and monthly basis. It can be reasonable to expect that larger month end report may take longer than inquiries used on a daily basis for each shift’s production.
Although it is difficult to catch and plan for all performance issues prior to cutover, the Stress Test approach above will help an organization know what to expect at cutover and go into the endeavor with confidence that transactional volumes and scenarios have been accounted for, leading to a more successful Go-Live.
About Logan Consulting
Logan Consulting is a leading consulting firm with strengths in strategy, project management, business process design, ERP and CRM implementation, recruitment and training. Since 1992, our business process-based delivery techniques and tools have helped our clients build a solid business process and information technology foundation to support their business. Our clients count on us for objective, unbiased analysis, recommendations and project work. To learn more and hear what our customers are saying about us, please visit our website (www.logan-consulting.com).
Logan Consulting is headquartered in Chicago, IL with global services reach.