Millennium Medic

OnTrack Panther Supports Millennium 2012.x Code

By Chris Lanaman

David Hallberg’s recent blogs on preparing for your Millennium^® site’s upgrade to 2012.x code have coincided with several conversations with clients who use OnTrack™ Panther™ to manage and monitor system performance. They want to know: “Does Panther support Millennium 2012.x code?” And: “Will OnTrack audits still point out issues as I test the new code and then move it to prod?” I want to assure you that all Softek OnTrack products and services support this version of code.

For those of you not familiar with Softek, our software and services give you a high level of visibility into your EMR system along with actionable evaluations and alerts developed and serviced by a team who collectively had years of experience at Cerner and various Millennium hospitals.

As we enter a new year, we have the same great news we have had since Cerner developed Millennium version 7.6: Softek is working to provide your site an objective view into the performance and reliability of your Millennium system.

The following table lists the Millennium code versions supported by OnTrack Panther.

Millennium Version
7.6
7.7
7.8
2001.01
2001.x
2003.01
2003.02
2005.01
2005.02
2007.02
2007.03-2007.17
2007.18.x
2007.19.x
2010.01.x
2010.02.x
2012.01.x
Choose and Book
eBookings

Prognosis: Third-party support that gives your organization intuitive management of your Millennium EMR system can be the foundation for a smooth upgrade to 2012.x code.

Citrix Load Balancing in a Complex Environment

By David Hallberg

A recent client’s Millennium^® performance issues had me stumped. The organization had a normal two-application and two-database node configuration. It was using very little CPU and had virtually no paging on these nodes. Queuing was pretty good from an MQ and proprietary middleware perspective. There was a lot of interface queuing, but the organization was on an older version of Oracle and we were not going to be able to address this area of their queuing until a database rebuild. So how could I help them right then?

I turned my attention to the Citrix servers — they had more than 100 of them — to check the load balance. The backend gave me little to go on, so I had to look deeper to see how the Citrix servers were connecting with Millennium. In this CPOE shop, Discern Notification alerts clinicians to potential problems with their order deliveries, so I chose to evaluate the Citrix load balancing in this component of Millennium.

When I did, I found that the method of Citrix load balancing being used was causing inconsistent response times for clinicians. Fast application and database nodes could not fix this performance problem. Rather, the team had to use the Citrix load balancing tool on Application User Load to more evenly distribute the core applications of, for instance, PowerChart^® and FirstNet^®. The better application distribution resulted in more consistent performance for their clinicians.

If you want to investigate whether you have a load balancing issue between Citrix and the application nodes, I’ll take you through the steps to parse your data and then evaluate the results. Start with one of the following select statements (either from CCL or SQL*Plus):

            CCL> rdb select * from EKS_NOTIFY_DEST go

            SQLPLUS> select * from EKS_NOTIFY_DEST ;

Your output will look something like this (for space reasons, I have eliminated three columns on the right and reduced my example to two rows of data):

LOCATION	TCP_IP_ADDR	TCP_IP _PORT	PERSON _ID	UPDT_DT_TM	UPDT_ID	UPDT _TASK
54099	192.168.1.62!RLL9989: PROD:DEARMAN	2048	11199999	2011/11/15 01:54:54	11199999	3071000
1555555	192.168.1.61!DAVID. HALLBERG:PROD:DWH9999	2048	8999999	2011/11/04 07:33:07	8999999	3071000

The LOCATION column identifies the device location, set either on the fat client or via the WTS location tool. The TCP_IP_PORT is the port the client is listening on. The PERSON_ID is a unique number for the user name in Millennium. The UPDT_DT_TM shows the last time there was any communication between the application nodes and this client. UPDT_ID is a repeat of the PERSON_ID. UPDT_TASK is the task number for the process that stuck the row in the database or changed it. This number should be 3071000, which is the Discern Notification. Not shown here are UPDT_CNT; UPDT_APPLCTX, the application context number for the application updating this table; and APP_VERSION_NBR, the version of Discern Notification that the client is using. The version generally will be 2, unless you are on some very old code.

The TCP_IP_ADDR column is the key to making sense of this gibberish. It is composed of four elements: the TCP/IP address of the Citrix server or fat client connecting to Millennium, the Microsoft user name, the Millennium domain the connection is for, and the Millennium user name. The next step is to parse the TCP_IP_ADDR column into these four elements.

Import the table into Excel to use the Text to Columns function, and insert three columns between TCP_IP_ADDR and TCP_IP_PORT.

To parse the fat clients or older Citrix servers:

1. Sort the TCP_IP_ADDR column so any cells with a “$” separating the TCP/IP address (first element) from the Microsoft user name (second element) are at the top of the list. Highlight these cells only.
2. From the Excel Data tab, select the Text to Column function, select Delimited and click Next.
3. In the next box, uncheck Tab, select Other, type $ in the open cell and click Finish.

To parse the rest of the Citrix servers:

1. Highlight the other cells in the TCP_IP_ADDR column, which should all have a “!” between the TCP/IP address and Microsoft user name.
2. With the Text to Column function, select Delimited and click Next.
3. In the next box, uncheck Tab, select Other, type ! in the open cell and click Finish.

The first part of the parsing is complete. You now have a list of all the TCP/IP addresses that have connected to the application nodes for Discern Notification. Next you want to see all of the Microsoft user names, Millennium domains and Millennium user names that have connected to the application nodes for Discern Notification.

1. In row 1 column C, which should be blank, type “Microsoft Username:Millennium Domain:Millennium Username”.
2. Highlight all the rows in column C, which contains the remaining three data elements.
3. From the Excel Data tab, select the Text to Column function, select Delimited and click Next.
4. In the next box, uncheck Tab, select Other, type : in the open cell and click Finish.

The next step in getting usable information is to build a Pivot Table (steps are based on Excel 2010) to see the TCP/IP addresses that have the most Discern Notifications:

1. From the Insert tab, select Pivot Table and verify the entire spreadsheet is highlighted. The dialog box should read, “Table1.” Select OK.
2. From the Pivot Table Field List, left-click and drag the TCP_IP_ADDR field to the Row Labels box.
3. Left-click and drag the TCP_IP_ADDR field to the Values box. The Values box will then read, “Count of TCP….” The Table to the left will have two columns.
4. In cell A3, titled Row Labels, click the down arrow and select More Sort Options.
5. Select Descending (Z to A) by:, click the down arrow to change the value to Count of TCP_IP_ADDR and select OK.

The resulting table will list the Discern Notifications for each TCP/IP address, as shown in this example:

Row Labels	Count of TCP_IP_ADDR
192.168.1.213	469
192.168.1.55	426
192.168.1.54	67
192.168.1.57	48
192.168.1.137	43
192.168.1.125	36
192.168.1.79	35
192.168.1.122	34
192.168.1.132	31
192.168.1.107	30

If your servers are properly balanced, the count should be fairly similar across all the Citrix servers. If you have fat clients, they will have a smaller count. In my example, the server ending in .213 handles 15.63 times more connections than the .107 server. Even if we exclude the first two rows, the .54 server is doing 2.23 times more work than the .107 server. Once this imbalance is corrected, clinician response times will be more consistent.

Prognosis: A little research might uncover Citrix load balancing issues that are causing response time variability or other performance problems for your clinicians.

An Abnormal Definition of Normal

By David Hallberg

Thanks to all of you who stopped by last week during the Cerner Health Conference. We had some great talks about your Millennium^® performance expectations, although I was discouraged to find that for a number of you your expectations were exceedingly low. For example:

• Some of you claimed, “You cannot fix the performance of the application.”
• Others told me, “We are remote hosted and have been assured that we get the best performance that anyone can. When the applications are slow, that’s just the way it is.”
• One client showed me their Lights On Network data for the number of transactions that took over 9 seconds, and I almost fell off my chair. Though the percentage was low, the actual number of transactions was enormous. “That’s way too many,” I said. “It’s the best we can get,” he replied.
• Still another client was told they needed to spend more than $1 million in hardware upgrades to fix their problem.

If you’ve been coaxed into believing any of these statements, let me tell you what I told them: Slow performance is not normal. OK performance is not OK. Don’t believe that you are the fastest Millennium client and there is nothing more for you to do. Your clinicians should not have to wait for 9 seconds for a transaction to go through. Many problems can be fixed without any additional hardware. I know. I have seen Millennium perform extremely well over the past 14 years.

When you are equipped with just a little knowledge of where you have bottlenecks in your system and take the appropriate corrective action, you can see immediate improvements. These improvements do not need to take years and do not require a new version of code, a new operating system or new version of Oracle. You can improve what you have. In the case of the client told to spend $1 million on hardware, we were able to work with the existing hardware and make a couple of uptime changes. Lights On showed that transaction speed for the clinicians had increased and variability had decreased.

Readers of this blog know that I regularly encourage you to trust your clinicians and believe your own eyes rather than blindly accepting performance excuses from the company that wrote the software and is now managing the environment. If the responsiveness of the system is slow, you should be able to see the first level of issues immediately. In a matter of hours or a few days — not months or years — you can determine the initial action steps.

I need to emphasize the words “initial steps.” There is no single silver bullet that will slay your performance dragon; you’ll need a magazine full of lead bullets. The beast is immune to silver but is susceptible to a lead overdose, to an ongoing maintenance regimen that accounts for your continually evolving EMR system. Every new package or service pack you install has the potential to change the processing of the entire system. Make sure you have the right people using the right tools to evaluate what is happening, to identify performance issues and to take action to correct the problems.

Prognosis: If your current support structure is not helping you help your clinicians deliver care more effectively, it’s time to find one that will.

Easy Tuning of Oracle Sequences Improves Throughput

By David Hallberg

In more than 13 years of helping clients tune their Cerner Millennium^® systems, I have only recently turned my analysis to Oracle sequences. I have the IT team for a PowerChart client to thank for my newfound understanding of how sequence numbers can increase the variability in system response times…and drive clinicians crazy.

After engaging me to diagnose their PowerChart performance problems, the team corrected several MQ queuing issues and other straightforward problems, but clinicians were still experiencing slowdowns during their peak usage times. The culprit turned out to be the Oracle sequence numbers.

Oracle sequences give an identifying number to certain elements in the database. Millennium has about 964 separate sequences, Oracle’s System and Sys have about 20 and 66 sequences, respectively, DBSNMP has two, and CMS_DBA and Panther_Read have one each. Oracle generates these sequences when new data or rows are added to the database. Sequences become a performance problem when you have a lot of new data that needs sequencing but the tables aren’t configured to adequately handle the load. Making the change is surprisingly simple.

Here’s what I discovered for my client. I took a 1-minute and a 30-minute sampling of the Oracle sequence numbers being generated by PowerChart. The following chart shows the results of the 1-minute interval: 

SEQUENCE_OWNER	SEQUENCE_NAME	CACHE_SIZE	Sequences Generated
V500	CHARGE_EVENT_SEQ	2,000	8,000
V500	OCF_SEQ	2,000	6,000
V500	CLINICAL_EVENT_SEQ	2,000	4,000
SYS	IDGEN1$	20	2,000
V500	PATHNET_RECOVERY_SEQ	2,000	2,000
V500	PERSON_SEQ	2,000	2,000
V500	LONG_DATA_SEQ	2,000	2,000
V500	CPMAPP_APPLCTX	20	1,320
V500	CPMAPP_APP_CTX_ID	20	960
V500	CHART_DIST_LOG_SEQ	20	40
V500	ENTITY_LOCK_SEQ	20	20
PANTHER_READ	AUD_ORA_ERROR_SEQ	0	7
SYS	ORA_TQ_BASE$	0	2

The key columns are Cache_Size compared to Sequences Generated. The Cache_Size shows how many sequence numbers can be generated before updating the high water number, which is the highest sequence number ever created for that sequence name. It takes time for Oracle to update this number, so you don’t want the system to have to do it too often. Increasing the Cache_Size decreases the number of times the high water value needs to be updated and, therefore, improves the throughput for the tables that use that sequence number.

Let’s look at an example in the chart. The Sequence_Name of Charge_Event_Seq generated 8,000 sequences in 1 minute. Because its Cache_Size was only 2,000, Oracle had to update the high water value four times during that minute. The Sequence_Name of Idgen1$ generated far fewer sequences, only 2,000, but its Cache_Size of 20 meant that Oracle had to update its high water value 100 times in this same minute. You can reduce this overhead associated with sequences — and improve the throughput for their corresponding tables — by increasing the Cache_Size to match or, preferably, exceed the number of sequences generated during your time frame. I recommended my client set Charge_Event_Seq to a Cache_Size of 9,000 and Idgen1$ to 2,000.

The change can be done in uptime without taking any action on the Millennium application nodes and without doing anything else on the Oracle nodes. The change is accomplished via SQL*Plus with this command:

       Alter sequence charge_event_seq cache 8000;

In CCL, use the following command:

       rdb alter sequence charge_event_seq cache 8000 go

The following chart shows the results from my 30-minute interval, which are a little more staggering in terms of the impact that high-water-value generation has on overall throughput. It also shows that only 35 sequences — about 3 percent of the more than 1,050 — had increases in their sequence numbers during this time (32 of 964 for V500 or Millennium and 2 of 66 for Sys).

SEQUENCE_OWNER	SEQUENCE_NAME	CACHE_SIZE	Sequences Generated
V500	CHARGE_EVENT_SEQ	2,000	106,000
V500	OCF_SEQ	2,000	76,000
V500	CLINICAL_EVENT_SEQ	2,000	48,000
SYS	IDGEN1$	20	40,000
V500	CHART_VIEW_SEQ	20	24,260
V500	CPMAPP_APPLCTX	20	22,660
V500	CQM_TRIGGER_ID_SEQ	2,000	18,000
V500	CPMAPP_APP_CTX_ID	20	16,160
V500	CQM_QUEUE_ID_SEQ	2,000	16,000
V500	ORDER_SEQ	2,000	14,000
V500	PATHNET_RECOVERY_SEQ	2,000	14,000
V500	CARENET_SEQ	2,000	8,000
V500	PATHNET_SEQ	2,000	6,000
V500	PERSON_SEQ	2,000	6,000
V500	DM_CLINICAL_SEQ	2,000	4,000
V500	EKS_DLG_EVENT_SEQ	2,000	4,000
V500	ENCOUNTER_SEQ	2,000	4,000
V500	SCH_CONVERSATION_SEQ	2,000	4,000
V500	AAR_REPORT_SEQ	2,000	2,000
V500	CHART_REQUEST_EVENT_SEQ	2,000	2,000
V500	CHART_SEQ	2,000	2,000
V500	ENCOUNTER_ONLY_SEQ	2,000	2,000
V500	ENTITY_LOCKKEYID	2,000	2,000
V500	ENTITY_RELTN_SEQ	2,000	2,000
V500	ESI_LOG_SEQ	2,000	2,000
V500	MEDADMIN_SEQ	2,000	2,000
V500	NETTING_SEQ	2,000	2,000
V500	OMF_SEQ	2,000	2,000
V500	PHARM_FILL_HX_SEQ	2,000	2,000
V500	REFERENCE_SEQ	2,000	2,000
V500	RESULT_SET_SEQ	2,000	2,000
V500	ENTITY_LOCK_SEQ	20	400
V500	CHART_DIST_LOG_SEQ	20	300
PANTHER_READ	AUD_ORA_ERROR_SEQ	0	125
SYS	ORA_TQ_BASE$	0	4

So I recommend only changing sequence numbers that are being used a lot. In this case, I recommended:

• Charge_Event_Seq and OCF_Seq have Cache_Size set to 10,000
• All V500 sequences with more than 2,000 sequences generated have Cache_Size set to 8,000
• All V500 sequences that match their Cache_Size be left at 2,000, at least initially
• All V500 sequences with a Cache_Size less than 2,000 be increased to 2,000
• Sys Idgen1$ have Cache_Size set to 10,000
• Sys Ora_Tq_Base$ have Cache_Size set to 20
• Panther_Read have Cache_Size set to 2,000

It is possible to make the Cache_Size too large, so my recommendations are fairly conservative. Still, the client will have much less variability in Millennium response times after implementing these straightforward fixes.

Prognosis: Keeping your Oracle sequence Cache_Size properly set is a simple, uptime tuning procedure that decreases performance-robbing events and variability in your Millennium environment.

New Code and Lights On Network Compliance Tabs

By David Hallberg

Several Millennium^® users I’ve talked with since the first of the year have had problems in production after taking new service packages. All had the same question for me: “How could we have anticipated the problems we now have in production?” Because some of the problems involved a new solution being brought up, I had a question for them: “What is your compliance percentage on the System Compliance and Solution Compliance tabs of the Lights On Network?” Either they didn’t know what I was talking about or they only had the tabs enabled for production. In their ignorance, they were missing out on key data from Cerner and its clients that could have helped them prevent the issues they were experiencing.

New code undergoes several quality checks before Cerner releases it to its client base. However, internal testing can never be rigorous enough to anticipate the varied and complicated production environment. So several years ago the Lights On Network was set up to allow clients to see several aspects of their configuration. The two most useful, from my perspective, are the System Compliance and Solution Compliance tabs.

The System Compliance has four dials: Known Errors, Oracle Configuration, OS Configuration and HNAM Configuration.

• Known Errors shows quality issues, sometimes called CIRB, as well as CernerWorks Process Improvement (CWXPI) corrections. Essentially, when one or more clients have a problem in a production environment or a potential for a problem in production, notifications are sent out to alert all clients of potential issues (CIRB). The CWXPI items are problems found at sites that Cerner manages or hosts, and to improve all Millennium environments, Cerner publishes how its teams corrected these issues.
• Oracle Configuration shows clients what several basic Oracle configurations should be when they first go live. There may be reasons your configuration differs from the recommendations. These reasons or differences can be documented in the Inactivate selection from this screen, an option available on all four of these dials.
• OS Configuration shows what basic Operating System configurations should be. It’s a very good baseline. If you deviate from it, you need to be certain why.
• HNAM Configuration shows more CWXPI items, basic logging settings and some solution/application configurations. Generally, this tab focuses on technical aspects of the Millennium environments.

The application side of the fence is found in the Solution Compliance tab. This tab shows topics such as what Ops Jobs (Millennium batch scheduling) need to be run and recommendations on how to configure Ops Jobs if they are not being run. The tab provides feedback on a subset of the Millennium purges. These purges delete temporary data that is not considered part of the permanent patient record. It helps control the size of the Oracle database and can improve or stabilize performance.

Prognosis: Once you and your organization are aware of the quality checks available on the System Compliance and Solution Compliance tabs of Cerner’s Lights On Network, you can put processes in place to leverage this service so that you reduce problems in your Millennium domains and improve the effectiveness of your infrastructure.

Why Auditing Your EMR Is Essential

By Chris Lanaman

I want to expand on the comments I made in my last blog on the need for third-party auditing of your EMR system. I’m including an excerpt from the white paper “Why Auditing Your EMR Is Essential” that our Softek team wrote and distributed last week at HIMSS. The message struck a chord with the people we talked with — both self- and remote-hosted organizations— so I’m confident it will resound with you as well.  

When patients are admitted to your hospital, they count on reliable, competent care:  caregivers who are board certified, care delivery that has passed the standard of the Joint Commission, lab procedures certified for quality by the College of American Pathologists, certified dieticians who oversee patients’ dietary needs and ethical business and financial practices. In each of these areas, facilities and individuals submit to regular, independent evaluation to make sure their practices follow generally accepted industry standards and, when necessary, implement changes to bring them back into compliance.

One area overlooked in these evaluations is the operation of the EMR system itself. With more and more functionality demanded from these systems, they continue to grow in complexity. Yet no independent evaluation program is required to ensure that the EMR is performing as expected. And if it’s not, the consequences could be devastating for patients, caregivers and the facility itself.

Think of the changes you continually introduce to your configuration. Besides the customizing you do for your own unique uses, vendors are supplying a steady stream of system updates and upgrades. A technology upgrade can create myriad downstream impacts on how the EMR will behave. Every time you add a new layer, a new process, a new handoff point, you increase the likelihood of delivery failure. Each failure poses the risk of losing patient information or financial data or extending wait times for clinicians trying to input or access that information.

If your EMR system is hosted outside your own organization, you might think independent auditing doesn’t apply. After all, one of the primary reasons you sought out a knowledgeable vendor was to oversee system performance. You wanted the assurance that comes from having competent partners hosting your system. Yet how do you know that the hosting partner is catching all the performance problems that arise? Because so many performance and stability issues arise from errors with application settings rather than from technical infrastructure, who is really responsible?

Your hospital IT team knows when performance problems develop — you deal with the complaints. With the visibility that comes through a system evaluation, you can more confidently partner with your service provider to not only address system issues but also set higher standards for performance and quality of patient care.

As healthcare focuses more and more on evidence-based medicine, the IT delivery vehicle for that medicine should be evaluated in the same way: using real data from your own system, gathered by an organization independent from the vendor or your own organization, and clearly presented so that you can understand where you are and what adjustments need to be done to accomplish your goal of optimal healthcare.

Prognosis: You can learn even more about how to get an evaluation of your EMR by reading our white paper, “Why Auditing Your EMR Is Essential.” Click here to receive it.

Betrayed by an Old Friend

By David Hallberg

I had a client complain to me recently that I was reporting a false positive, an alert that was triggered even though there was, in fact, no problem or issue with his Millennium^® EMR system. I decided to figure out why there was a discrepancy between my measurements and the client’s quality-assurance tool, which happened to be sysmon.

Let me begin by saying that I love sysmon. Before sysmon was developed in the late 1990s, Millennium had so many tools or utilities that provided a single view of a piece of information that it was difficult to keep track of all of them. At one point there were more than 200 build or monitoring tools even though Millennium only had about 40 actual application executables that were used in various configurations for its 60 plus solutions. Multi-node clients kept asking their Millennium middleware team to figure out where performance bottlenecks were occurring. Believe me, this sleuthing was extremely labor-intensive. Users had to have mon_ss (a Millennium monitoring tool used to watch for proprietary middleware queuing) and joumon (a Millennium monitoring tool used to watch for persistent queuing in the proprietary middleware) up in a different window for each application node. Then they would have to try to show someone else when one or more of these screens had a “blip” or negative performance indicator. It was painful on the good days.

So those managing Millennium were understandably ecstatic when Cerner developed a single utility that told what was happening across Millennium’s persistent and non-persistent queuing mechanisms across all application nodes, in a single screen. Sysmon also provided the date and time when something bad happened.

Sysmon proved to be reliable, and I became a proselytizer for this great monitoring tool. While I was at Cerner, I taught many clients how to use sysmon and its various switches. I also used it to tune a four-node client with ConnectionBias (a load-balancing mechanism for Millennium), moving the transaction workload from one application node to another. I was able to show enough value with this one tool that some clients actually chose not to purchase BMC Patrol (later called BMC Performance Manager and now called BMC ProactiveNet Performance Management). Sysmon had some limitations — it was really difficult and unreliable to pipe or move the output to a file for additional processing, and it had no mechanism to redirect its output to a common data format like CSV (comma-separated values), tab-delimited or regular flat files — yet it was the most accurate view of a multi-node application configuration.

So why was my data conflicting with sysmon? It appears that with 2007.02 some of the newer Millennium services do not register with the old middleware in a way that allows sysmon to see the queuing. With the client I referred to earlier, I found that the SenSage add-on called P2Sentinel did not publish its queuing in a way that sysmon could see it. The toolset I used, however, did see the problem.

That’s not the only problem. Another client reported that one of the Java services introduced with 2007.02 was crashing several hundred times a day. I ran sysmon to see if it would show these crashes, and it did not. I am not exaggerating to say that I was crushed. Sysmon was the one tool I absolutely trusted to let me know in a timely manner when Millennium was failing, typically 15-45 minutes before BMC. Now I find it is not reliable!

If, as it appears, the newer programs will not share or publish their information like the old programs, Millennium clients now must find other tools. None of the options, however, shows them this information in a single view. Sadly, we all have to go back to running multiple windows for each application node when we are using command-line tools.

A source of truth that you can trust and believe in is difficult to find. It is even more difficult to see it no longer be a source of truth.

Determine if Millennium’s Size Is Just Right

By Chris Lanaman

Like Goldilocks trying out chairs at the cottage of the Three Bears, every healthcare organization is looking to land on a system that is sized “just right.” When you first had Cerner Millennium^® installed, smart people thought through what kind of hardware and how much of it would need to be deployed in order to meet your organization’s needs. Most likely they also considered the projected growth of the organization over the next five to 10 years. Buying too much would add complexity for your users, loss of control in managing the system and, of course, wasted cost and time. No one wanted Millennium to be like Papa Bear’s too-big chair.

I would suggest that a bigger problem exists when the Millennium system is not sized big enough for the demands of your organization. Any one of the following symptoms might indicate that your workload has outgrown your system: lower than expected productivity of outcomes, the loss of business revenue because of turning away patients or poor response time for end-users.

I recommend starting with the following five steps if you are concerned that your organization has outgrown its current Millennium size.

1. Data, data and more data. You must have data in order to see if your organization is suffering from a sizing issue. This data is gathered by your team of data-miners (IT analysts) or through an application that is generating reports for your system, such as Cerner’s PowerInsight or PowerVision. Use this information to see if there has been a drop in activity, a workflow slowdown or loss of productivity.
2. Patient management cycle. Once you have obtained measurements for your organization’s workflow, you will need to evaluate where you are in your patient cycle — is this the time of year that you have increased patient traffic or are these your slower months? Your metrics should also include a staffing model to meet patient demand. Understanding your cycle will keep you from over-sizing your system if you evaluate during the high point of the year or under-sizing if you are in a valley part of your cycle. These measurements will identify the revenue cycle expectations for your organization.
3. Mergers and acquisitions. When your organization merges with another healthcare organization, great pains are taken to try and unify the new organization under a common vision. The impact of the merger on the EMR system’s workload, however, might be overlooked. The IT analyst needs visibility into the new organization’s past data to understand how workloads might increase.
4. Market share. Does your organization understand the footprint it has in your market? When trying to understand growth potential, you need an understanding of your opportunities and who the suppliers in the market are. Theoretically, one can say a set percentage of business equals a set amount of hardware needed, but this is an educated guess at best and should never be the rule of thumb.
5. Is your Millennium system healthy? As I alluded to earlier, when your system is not sized correctly, you will see an increase in performance issues. If your end-users are telling you, “The system seems slower than it used to be” or “It seems to take longer for me to do my job in the system,” the first thing on your mind should be the health — i.e., the efficiency — of your system. Before joining the Softek team, I used Softek Panther for a number of years to uncover trouble spots. It let me know when Millennium stepped out of line. Additionally, Softek’s Queuing Performance Spreadsheet (QPS) allows you to see where within Millennium your user transactions are getting clogged, and the Messaging Performance Audit (MPA) gives your system an overall checkup. The audit lets you see if you have build issues in Millennium, if transactions are going where they’re supposed to go, if your servers are configured correctly and when your servers are being over-tasked. If you have other monitoring tools you recommend, please let me know. Whatever you use, it’s important to check your system thoroughly before buying more hardware. Hardware purchases should always be considered the last resort for resolving performance problems. 

In a blog, I only have space to give you the Reader’s Digest Condensed version for scoping your organization’s size and determining how that affects Millennium’s performance. But addressing these key points will help keep your organization healthy and your users happy.

Prognosis: Evaluating all of the different factors that make up your organization is the first step in understanding if your Millennium system is sized “just right.”

It’s Just a Fax, Right? Part 2

Keeping relationships strong with PCPs

By Chris Lanaman

My last blog showed why doctors’ offices have a strong reliance on receiving test results and patient information through faxes — which are sent through Millennium’s Remote Report Distribution (RRD)  servers — and why it’s important for hospitals to make sure their faxes go out on schedule. Today we’ll look at another strategic importance of faxing, one that will help your healthcare organization distinguish itself from others.

In his novel 1984, George Orwell wrote the following ominous statement related to information control: “Who controls the past controls the future. Who controls the present controls the past.” I want to present a more practical paraphrase of the quote: “Whoever controls the old technology (like faxes) will be able to drive the introduction of the new technology (like interfaces).” What do I mean? The more relationships that your organization has established and is actively sending faxes to gives you the power to drive conversations about joint ventures in connecting electronic medical record (EMR) systems through interfaces.

Although many doctors’ offices do not have any form of EMR system, the number of “office size” EMRs has grown in the last five years and will continue to grow. As part of this conversion, I have found that larger doctors’ offices are happy to say good-bye to the piles of faxes that can often be difficult to manage. They still want notifications but are ready to move to a “paperless” environment.

When these offices start asking you how their number of faxes can be reduced, they are giving you the opportunity to strengthen your alliance with their office. If you can successfully interface your system with theirs, you stand a good chance of merging your organization with theirs to increase your share of the healthcare pie. In any case, the result is the same: Faxes go digital.

You might be thinking, “Great, no more RRD problems.” Au contraire, I suggest you keep your RRD servers well-tuned because faxes will be around for quite some time. Not all the information that offices request can be sent digitally. Plus, not all primary care physicians (PCPs) have embraced the paperless mindset. As long as a patient’s PCP wants to keep a paper patient record in the office, you will need to keep your RRD servers spitting out faxes. The volume of faxes that your organization spits out every day is a barometer for how much work it has in front of it to encourage doctors to move to an EMR system. The tighter your relationships, the more likely your fellow healthcare providers will listen to your vision for moving forward and working together into the digital future.

Prognosis: “Who controls the past controls the future.” If your organization has complete control of its RRD servers and the faxes they send, you will maintain and grow your partnerships with doctors’ offices well into the future.

Next: How to maximize your RRD investment so you have complete control over how faxing is conducted at your healthcare organization.

It’s Just a Fax, Right? Part 1

Doctors Depend on this Old-School Technology

By Chris Lanaman

In today’s need for fast data transfers through the Internet, Wi-Fi or 4G methods, many people feel that faxing is “so old school.” I don’t even think today’s eighth-graders can tell you what a fax is. Throughout the medical community, however, faxing is still an instrumental form of passing patient information. I also believe that healthcare organizations value faxing for its strategic importance in driving business to their facilities over their competitors. Over the next three blogs, I will explain why doctors’ offices consider faxes their lifeline to business, how faxes help healthcare providers maintain their relationships with doctors and how to ensure your information gets where it needs to go when it needs to get there.

Let me set the proper context. Cerner calls its module for faxing Remote Report Distribution or RRD. It is one of the members of the system group called Clinical Reporting. I contend that management should require at least one full-time employee (analyst) to support faxing. Even if the Millennium^® system is configured exactly as designed, RRD faxing can still create large problems for healthcare organizations (more on that in a later blog).

Many doctors’ offices do not have the equipment needed to perform complex clinical tests, so patients are referred to places that have the equipment, which is often the nearby hospital. Both doctors and patients want to get the results as quickly as possible, and hospitals look to their EMR system to be able to provide the timely results through faxes. In addition to faxing radiology findings and lab results, hospitals also use faxing to report on consults, send patient summaries of hospital visits, distribute notification letters to doctors and even send patient appointment reminders. Doctors’ offices rely on these faxes on a daily basis.

I have talked with many doctors’ offices on how faxes affect their processes for seeing patients. Here’s a typical scenario: When a fax containing a patient’s test results comes in, the office staff is responsible for delivering the information to the doctor for his or her review and for setting up a follow-up appointment for the patient to hear the results and begin necessary treatment. From a business perspective, these faxes are helping doctors generate new office visits. Some of the offices I have spoken to say this process accounts for up to 40 percent of their office visits. Without the faxes being sent to these doctors’ offices, physicians would be relying on results being mailed to them or the patients themselves calling for follow-up visits once they received the results from the hospital — neither of which would be acceptable to the referring physician.

So how do organizations leverage faxing to doctors’ offices? Some have contracts with doctors’ offices that include time frames for how soon results will be sent out, such as within 24 to 48 hours of the completion of the test. Additionally, organizations will offer to schedule the faxes to arrive during the nighttime hours or at a time the office deems appropriate. The office just needs to make sure its fax machine is on. All of these commitments can be configured through Millennium’s RRD system. Some of the hospitals I have worked for have processed anywhere from 400 to 3,500 faxes within a 24-hour period. If each fax represents just a $10 co-payment, this could be a lot of money to doctors. Because doctors rely on faxes to deliver quality patient care and drive more business, you can be sure that they won’t want to work with facilities that can’t deliver faxes in a timely and reliable way.

Prognosis: Hospitals can leverage a dependable faxing service to become the provider of choice for referring physicians.

Next: Why faxing is the gateway for healthcare organizations to strengthen relationships with primary care physicians and ultimately promote their vision for practicing healthcare.

Let’s Put Clothes on the Emperor

 By David Hallberg

When clients are looking for improved Millennium® performance – which they almost always are – our conversations often go like this:

Client: “Can you please help me parse and manage the RTMS timer files?”

Me: “Certainly. But I thought we wanted to improve Millennium’s performance for your clinicians and business staff. The RTMS timers will do nothing to assist in resolving performance for the point of care people and the business people utilizing Millennium.”

Client (with great indignation): “What are you talking about? The RTMS timers are THE performance metric for Millennium.”

With a name that stands for Real Time Monitoring System, you would assume that statement to be true. Its truth, however, depends on how you define the word performance. To clinicians and the business staff, performance relates to how quickly they can get their job accomplished. They are concerned with Millennium’s speed. The data center staff, especially those at hosting vendors, define performance as uptime. They are concerned with the amount of time that the backend systems are available. Some may boast that they offer a performance guarantee of 99.8%. Such assurances mean nothing to clinicians, who assume that the system should always be up. Their standard is a system that runs, in the words of a friend of mine, at blink speed.  

And while RTMS timers once did measure speed, Millennium architecture changes have altered the meaning of the data the timers deliver. The result is that end-users might complain that the system is slow but the metrics look fine inside the data center. A bit of history will help explain the discrepancies.

Millennium did not originally have RTMS timers in its code. Instead, the middleware had a setting to allow the Citrix server or fat client to log all of the transactions being sent from that device to the application node. In the days when the applications and database resided on the same node, this log was a great metric. However, the setting had some drawbacks. There would be an observable performance degradation in Millennium applications with this logging turned on (degradation of 10 percent or more was not unheard of), and the approach did not show any grouping of applications and functions. So it was a great tool for the middleware team but not so helpful for the application engineering teams.

A grouping feature was eventually deployed. Initially, there was a high degree of linking between the functions being measured between the Citrix server or fat client and the application node, which proved helpful in seeing backend performance issues. Over time, though, we have seen that Windows tab changes and other functions actually only measure how fast the Citrix server or fat client hardware is running rather than how long it takes the application and database nodes to process a transaction for the end-user. I call this “RTMS timer death.” You now have lots of functions being timed, but you can no longer be assured that the function being measured is actually providing a roundtrip view of the transaction time.

Several other changes are also obscuring the quality of the RTMS timer data.

1.    A greater and greater reliance on asynchronous work leads the interactive user to believe the system is more responsive, but it is in fact a bit of a performance cheat. For instance, I may do a function in PowerChart in which a request is sent to CPM Script and CPM Script responds when the function is complete. In fact, even though CPM Script is done with the function, CPM Script actually sent the request to CPM Process and CPM Process exploded my request into numerous other requests that it then handed to other Millennium executables.

2.    If the Citrix or fat client application (e.g., PowerChart) crashes or is X’d out, you will see a DBAPI_Exception event for the Millennium application executables that were processing the requests from this client. The Data or Record information starts with “DbDestroyTimer: invalid object handle: xxx,” where xxx is the handle being used to update the RTMS timer data. What does this mean? It’s incomplete data. It’s not clear that each of these events represents an RTMS timer that started but never finished, nor is it intuitive that there is a relevant impact, much less intervention, to consider.

3.    Two files in $cer_mgr – slareporting.cerner and timer.cerner – control whether a timer is active, what functions are used, if the timer meets the hit/miss count, and the timer’s average or mean time. Though it wasn’t originally the case, the CRM Timer information is now configured in the timer.cerner file. In the slareporting.cerner file, the 54 application timers are defined. In the timer.cerner file, the 1,022 functions are defined. This would seem pretty good – until you remember that there are more than 35 million lines of code for Millennium, more than 4GB of Citrix and fat client files, and more than 5,000 Oracle tables. In the slareporting.cerner file, by definition, you have to have each of the 54 applications run 500 times in two hours in order for the data to be counted. You also have to have each application run at least 10,000 times a month or the data does not count.

Let me use PowerChart to explain the potential for problems. Let’s assume that it’s 7 a.m. and physicians are starting to round, nurses are changing shifts and many other clinicians are getting down to their day’s work. In all, 499 clinicians log into PowerChart from 7-9 a.m. The IT department gets reports from these users that Millennium is slow. IT checks the slareporting.cerner file and sees nothing counted for the day’s performance. Why? The slareporting.cerner file must have a minimum of 500 functions of an application – like logging into PowerChart – in a two-hour window, or they do not count. Over the course of a month, if your site did 9,999 PowerChart logins, it would be stated that you did not do enough work to be statistically significant. For the purposes of a Lights On Network™ evalution, this data would not be considered substantial enough for valid comparison. Therefore, if you logged a Service Request stating you are having performance issues, you are likely to be told that based on the data you do not have any issues.

The initial intent of the RTMS timers was right on the money: to create a mechanism to allow Millennium sites to look at a clinical function and understand where the issue is. However, the execution and changes over time have made the timers unusable. Every client who I have worked with in processing their RTMS data has logged Service Requests and been told either 1) there is no problem or 2) a code level update will make the problem go away.

This approach to analytics showcases the different definitions of performance that I mentioned earlier. The clinical and business staffs are telling you that Millennium is slow and unpredictable, but the data shows no performance issues. This conundrum with the impenetrability of the data leads many organizations to conclude that the environment is unmanageable and they need to outsource the support for Millennium Citrix servers, application nodes, Oracle databases, etc. After all, the outsource vendors guarantee “performance” to be 99.8 percent.

I’ve already discussed how meaningless that statistic is to everyone outside the data center. A 99.8 percent uptime from the clinicians’ perspective makes no claims to speed and probably means that the .2 percent downtime will occur when they need the system the most. Here’s one more caveat: The statistic refers only to unplanned downtime. The two to four hours of scheduled downtime for weekly preventative maintenance are not included in the number. I have been in healthcare IT since 1989. I have been employed by four different hospital systems, have been an HIT consultant and have worked for Cerner, and I have never been allowed to have that much downtime in a month, let alone a week.

Nor was I allowed to ignore the clinicians’ desire for blink speed. When I did performance benchmarking at a facility running a different clinical system, we had the system tuned so we could do the screen changes in 0.6 seconds. Seriously. On one occasion we had a drive problem that spiked the time to 0.9 seconds. The screen flips were 0.3 seconds slower, about the time it takes to blink your eyes, and our help desk phones were on fire from the outraged clinicians complaining of performance problems.

So what can you do to help resolve your performance problems? The first step is probably the hardest. Stop believing the propaganda that your system is fine. Anyone telling you that you don’t have a performance issue that can be tuned is like the “tailor” telling the emperor that his new clothes are exquisite. They’re trying to convince you of something your eyes and brain know is not true. Listen instead to the people who interactively use your system every day. The clinicians and business staff will be the first to notice a difference in performance. The next step is trying to understand what you can do to monitor the performance of all your Millennium systems. The last step is to determine what staff and budget changes you need to make to address the issues.

There is a lot you can do with the tools you already have. You’ll find lots of tips in previous blog postings. I also encourage you to respond to this blog with your own questions and insights. I welcome a lively dialogue.

Security Changes Create Queuing Issues

By David Hallberg

Cerner recently made security changes to Millennium’s Java services and several older servers that might have disabled your system’s ability to automatically replay failed transactions. Unless corrected, caregivers might be losing important clinical information.

The changes involve Millennium’s personnel (prsnl) table, the listing of users who have permission to access applications and modify tables that contain clinical information. Previously, the username “system” was not typically included in this table because this username was created for those doing low-level executables with no need to access clinical applications or view clinical data. It was used to access failed transactions for SCP Entry IDs 105 and 108 (ORM Explode Cont Order and ORM Explode Cont Order Batch) and to replay them. A few months ago Cerner sent out a Millennium Flash announcement saying, essentially, that  failed transactions could only be replayed if the username was listed in the prsnl table and, to that end, the “system” username was now included in the prsnl table.

By forcing transactions to be replayed or re-queued by a user listed in this table, Millennium is plugging a gap in its security model. However, contrary to the announcement, most clients don’t have “system” listed in the prsnl table. If you’re one of them, you might find that transactions are failing to be replayed and that some might not post the first time.

“System” is one of three Cerner default usernames. The other two — “cerner” and  “systemoe” — are already in the prsnl table, where they need to be. But having this listing is only the first step toward ensuring proper replaying of failed transactions. I worked with two clients in June who had numerous interactive users all properly listed in the prsnl table but whose transactions still were not re-queuing. We discovered that some (hopefully all) Java servers have an extra level of security. Users must be granted access to application 3202004 (java.exe). I suspect that this requirement applies to SCP Entry IDs 105 and 108 as well.

Unfortunately, this access change does not seem to have been widely documented or deployed, so you might be losing some information as you’re waiting on others to put the information into Millennium. Far from being your fault, you just were caught in the switch to the new application security.

What do I recommend? Place the following usernames in the prsnl table and give them access to at least the 3202004 application: system, systemoe, cerner and panthercycler (if you’re a Softek client). Then cycle the CPM Application Authorization servers, SCP Entry IDs 105 and 108, and all of the Java servers (generally SCP Entry IDs 350-374).

Prognosis: Correcting the username listings and Java application access required by the new Millennium security model will allow your failed transactions to replay and will result in improved throughput of clinical transactions. To truly improve system reliability, however, you’ll need to address the underlying problem of why the transactions failed in the first place.

Help Your MQ Become PDQ

By David Hallberg

For those of you trying to get the most out of your MQ system for Millennium®, a few simple adjustments can deliver big benefits.

WebsphereMQ is an IBM middleware used extensively by Millennium. It has two major roles and one minor role.

1.    MQ’s most important use is for messages that have to get processed even if an individual application process or an application node crashes. This security measure means the data has to be written to log files and data files on disk drives, making it the slowest type of transaction for MQ.

2.    The 2007.x and 2010.x versions of Millennium, assuming MQ Shared Services (MQSS) is enabled, use MQ for communication from a client, like Citrix, to the application node. This queuing is non-persistent, meaning if the Citrix client or application node crashes, then the data is lost.

3.    Several years ago the Foundations team worked with the Foreign System Interface team to allow interfaces to use MQ.

For the first and third uses of MQ, the mqs.ini file defines how the logs are configured. Proper tuning of this configuration can have a tremendous impact on throughput. For MQ 6, several of the older limits have been increased. For HP-UX, AIX and Linux, you can now define up to 510 LogPrimaryFiles and up to 509 LogSecondaryFiles, with a maximum combined total of 511 LogPrimaryFiles and LogSecondaryFiles. This is a significant change from MQ 5.3, where the maximums were 62 and 61, respectively, with a maximum of 63 combined.

Setting these log files to their maximums makes the MQ log filesystems much larger, but it also allows for much greater concurrency. This concurrency allows certain functions — such as adding, updating, deleting and reading information — to run at the same time. It also makes it more likely that long-running batch jobs will complete instead of looping because the log files have wrapped.

Next we’ll look at the setting of my favorite mqs.ini entry, LogWriteIntegrity, which controls how many times a transaction is written to a log file. The default value is TripleWrite, meaning every update has to be written to three log files before the next step in the processing can occur. Why all the redundancy? To be fair to IBM, I believe they recognize that they have no control over the hardware and operating systems MQ is installed on. If your hardware server has no disk redundancy, IBM is giving you a reasonable chance to recover data if your node crashes or your drive dies.

Redundancy, I’m glad to report, is not an issue with Millennium production systems. Even old AIX systems have mirroring, that is, two drives with the identical data on it. If one drive dies, the data is available on the other. On current SANs, a writeback cache typically has a battery backup or a mirroring of the data on different memory arrays. The SANs are normally configured with RAID 5 drives, so the loss of a single drive does not mean any lost data. Thanks to this redundancy, you can change the TripleWrite default of the LogWriteIntegrity entry to SingleWrite. You should see improved MQ performance without risking any of your data.

As far as I know, the Cerner Millennium System Settings (CMSS) have not been updated in this area to help Millennium organizations achieve higher throughput from their investment. If any of you have current access to the CMSS for your operating system and the document is updated to include this, please let me know. I would enjoy hearing that the system tuning is keeping up with newer technology.

One last note, the LogWriteIntegrity parameter is valid on MQ 5.3 as well. You can get more information from IBM’s WebsphereMQ System Administration Guide for Version 6.0 (SC34-6584-00) and WebSphere MQ System Administration Guide for Version 5.3 (SC34-6068-02). Please keep in mind that I recommend you use normal change control processes for this tuning and then migrate any changes into your production environment.

Prognosis: A little MQ tuning goes a long way. You may be surprised at the benefits you see when you increase old log file limits and remove unnecessary redundancies.

 

Fragmentation’s Biggest Rift? The Generation Gap

By David Hallberg

My last blog covered deleting old files from your Millennium® application nodes. This week I’ll discuss the second step for keeping Millennium's speed consistent over time when you’re tuning filesystems and directories: defragmentation. Just like users disagree over the need to purge unneeded files, they also disagree over the need to defragment files.

I’ve had some interesting conversations over this issue this past year. People weaned on mainframes and older operating systems universally say, "Defragment!" Those who grew up on Linux, HP-UX or AIX virtually to a person say, "Defragmentation is only needed for the old and slower operating systems." With the latter group, I have to remember to stay calm and remind myself that their ideas probably come from the websites and authors who claim that, except with Windows, fragmentation is a thing of the past. But I also remember the documentation from the vendors who wrote the operating systems and the supporting filesystems. Plus I remember my work on the Advanced Technology Group at Cerner and what we benchmarked on AIX, VMS and Windows. Only then do I begin the conversation.

Filesystem and directory fragmentation occurs on all of the operating systems supporting Millennium®: VMS, AIX, HP-UX, Linux and Windows. There are generally two types of fragmentation: file fragmentation and space fragmentation.

In the first type, a file, say dic.dat, becomes too large to fit into a single disk block, so it has to be split into two or more blocks. If the dic.dat file is then modified, it may need a third or fourth block to fit within. It’s likely that code upgrades or custom CCL changes will have added more files to the filesystem where dic.dat is located, and they will have been saved to the blocks right next to the dic.dat's first two blocks. That means the third and fourth blocks for dic.dat will not be able to be written right next to (or contiguous with) the first two blocks. Now the dic.dat file is fragmented. With the file split into two chucks, the operating system has to perform two reads to completely read the dic.dat. When the disk drive or RAID set has to do more reads (disk I/O's), performance suffers.

Space fragmentation occurs when you delete a file, typically because it was a temporary file. Once deleted, the space it originally occupied leaves a gap on the filesystem. Many deleted files create greater fragmentation, making it impossible to write a file contiguously.

The filesystems/directories I discussed in my last post ($CCLSCRATCH, $CCLUSERDIR, $cer_temp, $cer_fifo, $cer_usock, $cer_log, /var, /tmp) are all designed to hold temporary files, such as print files or logging files, that will be deleted as soon as they are no longer needed, e.g., daily or possibly hourly.

The downstream impact of this design is that all current operating systems need to have both types of fragmentation corrected. Vendors provide tools for this defragmentation, including defragfs for AIX, fsadm for HP-UX, and fsadm for CAMM (Millennium's PACS) and the Vxfs filesystem that it uses. VMS has the DFU tool on the freeware CDs that has to be installed before it can be used. Also for VMS, HP has a DFO tool and Diskeeper offers Diskeeper for VMS, which has been around since 1986. You’ll have to pay for these latter two tools, but it might be worth investing in one of them if you’re trying to keep your VMS system. For Windows, you can use the built-in tool called Disk Defragmenter; a third-party tool like Diskeeper 2010 and its various versions or PerfectDisk 11 and its various versions; or one of the plethora of freeware/shareware tools, my favorite being Defraggler from Piriform. I encourage you to spend an hour or so looking through the numerous white papers, blogs and forums discussing these options.

My first team in Cerner's Advanced Technology Group was tasked with successfully completing client conversions no matter what technical hurdles we faced. One of our biggest obstacles was severe fragmentation of both files and free space. We had such significant file and space fragmentation at several sites that we added a drastic step to our normal recommendations. In addition to deleting lots of files daily and defragmenting the filesystem at least once per week, the clients had to set up a schedule to point the filesystem, say $cer_print, to an entirely new filesystem every three months. Miraculously, their systems became much more stable.

Why did it help to move the filesystem? Each filesystem had an index that kept track of where each file started, the size of the file, where it ended, etc. As thousands of files were created and deleted every day, the operating system had to slow down in order to keep track of them. The periodic refresh of the Inode or indexf.sys (on VMS) brought performance back to a better level. With today’s improved drive technologies, this type of severe fragmentation is less likely, but it still needs to be reviewed. Some of you might find it necessary to recreate your filesystem on another set of disks and benefit from having contiguous files and free space.

Prognosis: Using defragmentation tools on a regular basis to reduce filesystem fragmentation on all operating systems will increase Millennium’s speed, consistency and reliability.

Deleting Old Files

By David Hallberg

Today’s topic is an issue over which Millennium  users disagree: whether the number of files in a directory or filesystem impacts the Operating System (OS). Many of you say no, but I believe it does. Having too many files in a directory or filesystem with no scheduled purging of those filesystems will degrade your system’s performance.

Those who state that the number of files has no impact on the OS begin with a great assumption: that Millennium was written by the OS vendor and that it is compiled for the OS version it is running on. Those versions include 64-bit for VMS, AIX and Linux and 48-bit for HP-UX. Unfortunately, the Millennium application is written with the 32-bit compilers for each of these platforms. Under ideal situations, the OS may not care about how many files exist, so long as there is free disk space, but Millennium does. The number of files affects CCL report generation to either your Millennium session or a printer and definitely affects the primary Millennium print servers, such as DCP Output (SCP 106), System Print Server (Node Specific) (SCP 127), Print Services Cache Manager Server (SCP 128) and System Print Server (SCP 129).

Let me give you an example. The SysSrvPrint error message from the servers indicates that the Millennium executable cannot find the file in the filesystem. This event can happen because 1) the file is on another node, 2) the file was purged before it was printed, or 3) it took too long for the executable to look up the file. That third failure reason is easy to prevent. If you delete files from the $cer_print and $CCLSCRATCH directories every four to six hours, you should be in good shape.

You can count how many files are in the $cer_print directory or filesystem using this command (the last character is a lowercase L):

ls $cer_print | wc -l

Likewise, to see a count of the files in the $CCLSCRATCH directory, type the command:

ls $CCLSCRATCH | wc -l

The goal for optimal performance within all Millennium solutions is to have less than 5,000 files in any of these directories at any point in time. Due to the number of facilities and solutions you have and the Millennium version you use, you might not always be able to achieve this level. But I encourage you to keep working towards it.

The crontab job scheduler in the $cer_proc directory comes with a UNIX Korn shell (ksh) named clean_wh_env.ksh that deletes unnecessary files. The ksh will probably have to be tweaked to fit your specific site. This takes a little getting used to if you are unfamiliar with UNIX scripting but will greatly improve Millennium’s speed and reliability. Beginning from the directory count you ran from the commands listed above, identify the directory that contains the greatest amount of files and then modify the clean_wh_env.ksh file either 1) to be run more frequently from crontab or 2) to delete additional file patterns or extensions.

The ksh should also be used to clean up the following additional filesystems: $cer_temp, $cer_log, $cer_usock, $cer_fifo, $cer_lock and $CCLUSERDIR. Access the file count in these directories with the command listed above. For these directories, deletions scheduled to run once a day should be enough.

Prognosis: Using your Operating System’s scheduling service to routinely delete unneeded files from your filesystems will increase Millennium’s speed and reliability.

Help Citrix Handle Unexpected Login Spikes

By David Hallberg

Recently I was working with a client who had experienced an unexpected power outage. Once the power returned, the PCs had to be rebooted. The client uses Citrix to access Millennium, and the rebooting overwhelmed the Citrix Zone Data Collectors. The overload delayed system access for everyone trying to get back into Millennium® — even though the data center stayed up during the outage.

Was the overload inevitable? Or would it have been possible to tune the site’s five-year-old hardware to better recover from or live through abnormal spikes in activity?

These questions are important because many sites use Citrix’s XenApp servers to deploy Millennium. Citrix — how most users refer to all of the various Citrix components — allows a small set of servers to present Millennium applications, which individual PCs then access by connecting to the Citrix servers. The Zone Data Collector (ZDC) is used by Citrix to evenly balance the PC connections across all of the XenApp servers. Without the ZDC, the first Citrix server in the group or zone would get all of the connections, which would quickly overwhelm the server and cause other issues.

As my client discovered, even with the ZDC, abnormal events can overwhelm the server. So I want to suggest some basic tuning steps to help you balance the ZDC loads and ward off future overloads. We’ll discuss the data store (often SQL Server) tuning and registry changes that can make the data store and ZDCs able to handle sudden increases in volume.

I would suggest you review your Citrix configuration even if it was set up by an “expert.” Often the experts are skilled in following a specific set of requirements to set up an environment or domain and show you how to do it next time. Then they leave. The more knowledge you have about your environment and its configuration, the better you are positioned to address potential issues. Ask yourself, “If I had to accommodate a doubling of connections or clients in less than 60 minutes, where would I have problems?” After reading through the steps I recommend, you can create a list of what tuning needs to be done and start the change control process to put these changes into production. That way, the next time you have a power outage at your organization, the recovery time will be shorter and less painful for you and your Millennium users.

For details on the Citrix tuning process, click here.

Prognosis: By moving your data store to an optimized disk partition and tweaking some registry settings, you will find that Citrix can become much more responsive, even when the workload more than doubles.

Message Offers Quick Fix for ECO Server

By David Hallberg

In my auditing of Millennium® systems over the past decade, I have seen many meaningless and vague error messages. So I was intrigued by the message I found recently while performing a system audit for a client. The interesting thing about the message was that it actually told us how to fix the problem. I applaud the Millennium developer who was kind enough to think of how to help clients with this issue.

The message appeared when I was reviewing some fast-wrapping message log files. Several messages appeared in the ECO (Explode Continuing Orders) server. The ECO server is used to “explode” orders that have multiple schedule times. For instance, if a clinician orders a daily x-ray for a patient, the information is exploded initially on the ECO server: ORM Explode Cont Order (SCP Entry ID 105). Once the first day’s x-ray is ordered, the order is exploded by a second ECO server for batch jobs: ORM Explode Cont Order 001 (SCP Entry ID 108).

One ECO server message in particular grabbed my attention. Even though it was only an audit-level message, it included the way to resolve the issue. It said:

The number of orders to explode for order_id: 123456789 is > than NUM_ORDERS (50). Only 50 instances will be exploded. To increase the limit, run dcputil_set_config_prefs in CCL and then cycle this server.

I learned that I had more orders to explode than the system would allow. But I also learned how to fix the problem. I had to:

1.    Go to CCL.

2.    Run the script dcputil_set_config_prefs.

3.    Change the NUM_ORDERS from 50 to something larger.

4.    Cycle the ORM Explode Cont Order (SCP Entry ID 105) servers on all of the application nodes and then cycle any clones of this server, such as ORM Explode Cont Order 001 (SCP Entry ID 108).

What about your system? If the LogLevel of SCP Entry ID 105 and any clones of it is 2 or higher, I would encourage you to look for Event ExplodeOrders. If you have a message similar to the one above, you will have a quick fix for your environment. Once you execute it, the clinicians will think that Millennium has magically started working and they can stop working around it or, rather, working around the fact that the orders that were supposed to be there were not.

Prognosis: Interesting error messages can actually be helpful. If you take their advice, you’ll help ensure that Millennium delivers all the information the clinicians need.

 

Preparing for an Upgrade?

By David Hallberg

We have seen so many Millennium® clients have problems after upgrading to 2007.18, 2007.19 or 2010.01 that I feel compelled to address this issue. If you have not yet upgraded, this blog will link you to specific recommendations to help minimize the known risks to your production environment. It’s best to execute them before the upgrade. If you have already upgraded, you may want to review these items and see how many of them you can do to improve your current stability.

In general, I’ve ranked my 14 recommendations in order of importance. The majority of them are MQ and message log issues. They apply whether you are on AIX, HP-UX or VMS (although VMS applies only through the 2007.19 upgrade since Cerner doesn’t support this platform on 2010.01 and above). Click here to read the specifics on correcting the following 14 issues:

1.    The hidden MQ queue.

2.    Java server stress.

3.    Citrix crashes.

4.    Exception queue transactions.

5.    Message log disk I/O increases

6.    Confusing MQ log files.

7.    Extra message log writing.

8.    Too much extraneous logging.

9.    MQ information loss.

10.  Missing or incorrect security settings for a file.

11.  MQ running in mixed mode.

12.  Request class routing needs.

13.  Wrapping issues with message logs.

14.  QCP server instances.

Prognosis: By performing a handful of critical tasks before an upgrade, I believe you’ll have a much shorter troubleshooting time during the upgrade process.

If You Rewrite the Rules…

By David Hallberg

How I love rules. By following certain clear principles, you can be reasonably sure of the outcomes. If you wash your hands regularly, then you’ll spread fewer diseases. If you listen carefully, then you’ll be better able to address a person’s concerns. In the IT world, it’s called the if-then-else logic loop. But I’m the first to admit that rules have limitations. I’ll leave you to figure out their limitations in your personal life, but this week I’ll help you determine if some of the rules you have set up in Millennium® are a bit broken.

Millennium has a wonderful subsystem called Discern EKS, which allows users to identify special events that will automatically trigger other events. For instance, users can set a rule that whenever a patient’s white blood cell count is very high, say 15,000 to 30,000, an alert suggesting a blood culture will be generated. Each department determines the rules it needs based on its standard operating procedures, works with the appropriate teams in the organization to set them up and outlines its response when the rules are triggered.

It sounds straightforward enough, and most organizations don’t give much thought to the rules if they are running in less than a second or two. Run time alone, however, does not tell the whole story. Organizations can have hundreds or thousands of rules, some of which fire thousands or tens of thousands of times a day. They can present a huge drag on the Millennium system, and these tuning issues need to be addressed.

For example, if a rule takes one second to run – which would be considered a reasonably fast rule – but is triggered 1,000 times in one day, that one rule is using up 1,000 seconds – or over 16 minutes – of application CPU time. Rules firing more than once per minute – more than 1,440 times a day – need closer scrutiny. I would recommend that you look at that rule’s trigger mechanism and verify it really should be firing that frequently.

Rules that trigger once per minute are not uncommon. I’ve worked with clients that have rules triggering more than once per second – that’s 86,400 times a day. Now consider that many sites have hundreds or thousands of these rules. Even if all of them are so-called fast rules, you can still see where you might have Millennium performance issues.

To determine if you’re having issues with a rule or set of rules, we’ll look at the Oracle table EKS_MODULE_AUDIT. This table shows when rules are triggered and how long they take to fire, along with a lot more useful information. To learn how to access the EKS audit table and analyze its numbers, click here.

Prognosis: An analysis of the time Millennium spends each day running rules will show you whether any rules need to be rewritten to help the system run faster.

Shared Service Proxies in an MQSS World

By David Hallberg

Millennium®, since version 2007.02, has tried to reduce its proprietary middleware footprint through the use of IBM’s WebSphereMQ for client communications to the application nodes. This is often referred to as MQSS for WebSphereMQ Shared Services. It provides an enormous leap forward in throughput and performance for most Millennium users but has not lessened the need for Shared Service Proxies (SSPs), as was expected.

Millennium’s initial design specification for its proprietary middleware was 125 messages per second, which was pretty good when it was introduced 15 years ago. Plus it often saw higher throughput thanks to newer compilers, faster CPUs and some other changes. WebSphereMQ on version 5.3 for this sort of communication (often referred to as non-persistent queuing), however, was benchmarked at more than 3,200 transactions per second, and 6.2 is benchmarked at more than 3,800 tps, where each transaction is 2 kB in size.

The diagram above gives a simple overview of how shared services work with IBM MQ. Starting at the left of the diagram, we have a client application that is connected to both a request and a reply queue. These queues are used to transport the client request to the application servers and the reply back to the client application.

If you are an AIX or HP-UX client, I recommend that you enable MQSS as quickly as possible to help your throughput. I wish I could say the same for VMS clients. Unfortunately, VMS experiences capacity issues with MQSS. Its TCP/IP stack and BG devices can exhaust their limit of 32,768 ports, at which point they no longer take new connections. Cerner now recommends that all VMS clients stop using MQSS, except for the Java services, which only know how to communicate via MQSS.

While all of this speed is great for clinicians and business staff using Millennium, there is a hitch. Updated Millennium configuration standards state that when MQSS is enabled, the number of SSPs (SCP Entry ID 1) should be reduced to four and that all dedicated SSPs should be disabled. Based on the client data I’ve seen, numerous clients who have gone live with MQSS with only four SSPs have experienced varying degrees of queuing, backlogs or slowdowns in the SSPs. I recently worked with a client who had more than 82 hours of queuing for the CPM Script Batch 001 service.

Their queuing was typical of many others: It was constant, not centered around normal usage patterns of one peak in the late morning and another in the early afternoon. The only way to mitigate this backlog is to increase the number of SSPs or create a dedicated SSP, stop all of those servers on the node and then restart them. This action will move the service to the new generic  SSP or dedicated SSP and should stop the queuing. (To learn more about SSPs, read previous blogs on “SSP’s Path for Rapid Transactions” and “Setting up a Dedicated SSP.”)

This is a very surprising discovery. The Millennium switch to a commercial middleware like WebSphereMQ was supposed to eliminate this queuing. Millennium support might attribute the problem to the SSPs consuming resources that could be freed up for other processing. Although technically true, the impact of the SSPs is relatively small. An SSP starts with less than 16 MBs of RAM and can grow to about 128 MB of RAM. If I ran 16 SSPs that all grew to 128 MB (they often do not grow that large), I would be using the same amount of memory as any one of the Java servers (SCP Entry ID 351 through 370). Since the SSPs do not handle client communication — they don’t talk with PowerChart, FirstNet, PowerChart Office, etc. — they consume very little CPU. These processes do some disk I/O, but compared to CPM Script, SCP Entry ID 51 or the Java servers, it is such a small amount of I/O that it is really just background noise. The resource consumption, therefore, is too small compared to the rest of the Millennium application executables to have any significant impact to the resources of the node, lpar or vpar.

I still recommend that all non-VMS users switch to MQSS as soon as you can. The throughput gains are impressive. I just ask you not to reduce the SSPs to four. Although I don’t know your specific configuration needs, I would suggest you start somewhere in the range of 8-20. Eight was the old minimum recommended value for SSPs in non-production domains, and I have always recommended 20 as a base for any production domain.

Although VMS clients can’t benefit from MQSS, I do urge you to upgrade your system to make it as stable as possible and to ensure that you are able to use all 32,768 ports. Clients on VMS version 8.3 should upgrade to the appropriate VMS 8.3 with TCP/IP Services version 5.6 ECO 5 or later.

Prognosis: Even with MQSS’s welcome increases in throughput, SSPs still have a role in Millennium. Keeping them properly tuned will help prevent queuing in Millennium's proprietary middleware.

Chart Servers — Part 1

Chart Request Table Dishes up a Feast of Data

By David Hallberg

Today I want to begin a short series on chart servers by introducing you to a table that many Millennium® users are unaware of: the Chart Request Table. This table dishes up a lot of information about how your chart servers are performing — particularly how long it takes the servers to process chart prints — so it’s important to know how to access the information, understand what it’s telling you and correct performance issues.

With all the other issues you need to manage in a day, you might be asking, “Why does it matter how long the chart print takes?” In a nutshell, chart prints are used to treat patients. Printing delays could mean a clinician does not have the latest information on the patient being treated. Waits of more than an hour for a larger chart to print are not uncommon. And a delay in printing one chart slows down all the other chart requests queued up behind it. With Expedite and Ad Hoc chart requests in particular, this lack of information has the potential to compromise patient care and it opens the organization to liability concerns.

I have found an enormous spread among clients on how long it takes to print charts. Some clients can process a page in 1 second, and others take more than 2 minutes to generate a single page. A 2-page chart that requires 2 minutes to process each page stays in the chart server for 4 minutes before being sent to the print server and finally the printer. At this rate, a 50-page chart takes the chart server 100 minutes — or more than an hour and a half — to complete the generation of a single chart request. That same chart, if sent to a well-tuned chart server processing at 1 page a second, would be sent to the print server in less than 1 minute.

I understand that your time, money and staff are limited. But access and analysis of chart server performance details can be done in minutes, and a skilled technician can correct slow servers in a few hours. It starts by accessing the Chart Request Table and exporting the data to a spreadsheet for analysis. Using this information to then reset your slower chart servers to match the fastest one will dramatically improve the clinically perceived speed of Millennium, reduce end-user frustration and quite possibly improve the quality of the delivered care. Click here for the step-by-step process.

And then keep your eye out for upcoming blogs, in which we’ll:

1.    Examine the implication of other pertinent data contained in the Chart Request Table — in particular, the servers’ print quantity.

2.    Dig deeper into Windows tuning techniques of the chart server that will reap even larger rewards for your clinicians.

Prognosis: Knowing which chart server or group of chart servers is fastest and getting all of the chart servers to perform at the same level can improve clinical end-users’ acceptance of Millennium and potentially some patient outcomes.

A Healthy Dose of Doubt

By David Hallberg

 “…and the only people I fear are those who never have doubts.”

– Billy Joel, “Shades of Grey”

One of the most frustrating discoveries I’ve made about Millennium’s technical infrastructure is that tuning is never done. If I think I’ve gotten all of the performance out of Millennium that I can, I need to step back and re-evaluate my metrics…or just wait a little while for other issues to arise.

Another important realization, one that I learned a long time ago, is that my knowledge and experience is not the pinnacle of knowledge and experience. No matter how hard I try I can never know all there is to know on a given topic. There is always someone else who has dealt with a similar performance/stability/capacity issue and knows more about it than I do. When working with clients, I prefer a collaborative approach that comes with a little doubt that the problem solving will ever be done.  Performance is an iterative process that has to be monitored and adjusted through time, not just at a single point in time.

Millennium is a large, complex suite of applications with many inter-relationships and impacts that are not easily visible. Millennium requires DNS servers, networks, Citrix servers, Windows servers, Oracle, SQL Server (both for Citrix and some applications), MySQL, Oracle with raw logical volumes, Oracle with the Veritas File System, Oracle with ASM, AIX, HP-UX, VMS, thin clients, WebSphere application servers, WebSphereMQ, Apache web servers, load balancers, Java, Win32, ActiveX components. The list goes on and on. Plus, the Millennium environment is never static. There is always another project, another service package to install, another site to bring into production. Every change in the environment means changes in Millennium and affects how stable Millennium is and how well it performs.

All of these variables should help keep us humble. When I was in consulting and managing Millennium directly, I knew that if a client followed all of my recommendations, we could typically expect a 90- to 180-day period of stability. After that, change would happen. Ultimately, the earlier tuning would help us adapt more easily to these changes, but there would always be a pain period after a change in Millennium. The tuning was never done.

I work very hard to understand Millennium and share this knowledge with Millennium clients. In the process, I learn from each client I interact with. I’ve recently seen two extremes around this point: One organization benefited from this collaborative approach and from having a little doubt that they had done everything possible to tune Millennium, and another was certain they had done all the tuning they could.

The first client said that they just had the “normal” Millennium performance issues but asked me to evaluate the system and suggest ways to improve it. The Messaging Performance Audit revealed several straightforward, easy-to-resolve issues as well as others that required new Millennium front-end code. Here was a person willing to admit some doubt that his team was doing all it could to improve the hospital’s electronic medical record system. After implementing some of the recommendations, they saw quantifiable benefits.

On the flip side, a second client was experiencing significant performance issues in production after introducing key changes to their environment. The system basically came to a crawl every day from 9-11 a.m. and 1-3 p.m. Analysis revealed disk I/O issues. They had not implemented the baseline Cerner Millennium System Settings (CMSS). They had not tuned the operating system’s disk buffering memory regions. Response times from their internal SCSI drives were taking 20 milliseconds instead of three to eight. The Oracle database showed some large waits for disk writes to complete.

When I presented my findings and recommendations, the lead technical person dismissed them. There was no doubt in his mind that his team had done all they could to address the problems and that I was completely wrong. He was beyond adamant. His team continued to address the escalating problem simply by logging Millennium Service Requests until clinicians revolted. Now the client is paying Cerner and other outside consultants to come in and save the day. No doubt proper tuning would have cost a whole lot less and, more importantly, would not have impacted the clinicians’ ability to deliver patient care.

Prognosis: Being willing to doubt we have done everything possible to make Millennium as fast as it can be opens us to new ways of providing clinicians a more reliable, consistent EMR system.

What Is Measured Improves

By David Hallberg

I just came across the following quote by famed management consultant Peter Drucker: “What’s measured improves.” It occurred to me that this was the crux of some recent conversations I have had with Millennium clients. I keep being asked why Millennium’s message logs are important. Why would a CIO, CTO, CMIO, director, or project manager care about an application’s log files?

Because what is measured improves.

Millennium has more than 25 distinct solution areas with more than 60 distinct, sellable solutions or applications. How do you know if you have high quality in the delivery of these solutions? How do you know if the deployment of the technologies that support the solutions is working as designed? Look at the message logs.

The IT department is often charged with quality and performance improvement as well as patient safety. So how do you detect and categorize these quality and performance problems in Millennium? Look at the message logs.

I realize that this simple answer gets a bit more complicated by the fact that each application node can have between 283 and 538 log files. How can anyone figure out what to do with all of these files? Most people see the number and simply give up before they start. I admit it is a daunting task, especially if you do not have system management tools to help. However, the following steps will help you sort through the hundreds of files to identify the message logs that point to quality and performance issues with your system.

Start by looking at log files that are wrapping in less than 10 days. Those wrapping in less than one day pop to the top of the log list. Each log file contains the date/time stamp of the oldest and newest record and the number of times the log file has wrapped. To avoid running out of disk space on the application nodes, Millennium sets by default the log files to 4,096 records. The 4,097th record overwrites the first record. This means that if a log file is wrapping, you are losing information you may need to determine if you are having a problem.

You can get to the information you need using one of the following options: the message controls in Panther or Olympus, the ls -l command on AIX and HP-UX, or the dir command on VMS. Panther and Olympus will give you wrap times. If you don’t have these tools, you will need to derive them. Perform the ls -l or dir command of the $cer_log directory, wait one day, and do the command again. Save your output, import it into Excel, and look for the files that changed. Now you know which files are wrapping quickly.

Next look for Errors and Warnings (the Level column will show either 0 for Error or 1 for Warning). They indicate that all or part of a transaction was not put into the database. Since it is not in the database, the database tools cannot see it; these events are generally only visible in the log files. If you have a log file wrapping quickly with the Level being greater than 2, you need to change the loglevel in SCP and cycle the server.

Do not accept the Service Request (SR) response you will get when you log this to Cerner. They will tell you to put the Event into a file called messages.suppress and cycle the server. This action will stop the messages from being written to the log files, but it will not stop the problem. It’s a little like turning up the radio when your car starts making a strange sound. You might not hear the noise, but you haven’t eliminated the problem. Likewise, suppressing messages will declutter the log files, but you are only hiding errors or events, not fixing them, and ultimately putting your clinicians and organization at risk. (See our earlier blog titled “When No News Is Bad News.”)

Once your staff has gathered the information, you need to create and execute an action plan to resolve the issues. Unfortunately, this problem solving is an iterative process. By the time the first round of issues is resolved, you probably will have completed more build in Production and had more Millennium code installed. So the process starts again. I have found that if you start this process in the non-Production domains, you will generally be able to resolve the issues before they get to Production. Then the Production message log audits become much easier and faster.

Once you create this quality loop, clinician and business office complaints will diminish, response times will become more consistent, and the general acceptance of the Millennium system will be higher because people will trust the information it holds.

Prognosis: By using message log audits to measure Millennium’s performance, you uncover problem areas and improve the system. Peter Drucker would not be surprised.

Remedies for RHOs — Part 6

The OS Path for Rapid Transactions

By David Hallberg

The first five installments in our series on Millennium^® performance issues for RHOs walked you through the necessary steps for evaluating connectivity issues as well as queuing problems involving Websphere MQ, shared service proxies, interfaces and Oracle. This week I’ll finish up with yet another area where queuing or waits can be a problem: the operating system (OS).

The OS runs the hardware for Millennium and is often referred to as the backend or backend servers. OS capacity issues can cause Millennium to perform slowly or, worse, sporadically. A sporadic manner could mean that at 7 a.m. Millennium is working fine, but from 9-11:30 a.m. it’s slow.

Before detailing helpful OS diagnostics, I want to answer the larger question, “Why do I need to bother with all this diagnosis? I’ve got an outsourcer to troubleshoot for me.” Experience has taught me that any time you outsource or out-task work, you should audit it. That way, if your clinicians and business staff are complaining about performance, you will have the information you need to have a data-driven conversation with your outsourcer.

The generic complaint of “Millennium is too slow” will generally be met with deaf ears. A comment such as “I have a Run Queue on all application nodes exceeding 4x my physical CPUs” is something your outsourcer can track down and help you correct. So I urge you to maintain technical knowledge in-house or find a trusted consultant who can assist in gathering and interpreting key data concerning your Millennium operations.

So what can you do to determine if your application nodes have performance or capacity issues? Click here to read more.

Prognosis: Monitoring OS capacity and taking action to reduce or eliminate issues as soon as they start — instead of days, weeks or months later — will improve Millennium's consistency.

Easy Script Analysis – Part 5

Automating Data Collection

By Rob Dearden

For the past few weeks, David Fletcher and I have covered how to manually collect script performance information and create spreadsheets for analysis. We end the topic this week with an explanation and example of how to automate this collection process.

Our goal is to create reliable data collection in a timely manner without impacting the system significantly. To that end, we need to address two key concerns:

1.    How often should scripts run in a day, and how many days should they run per month?

2.    How big of an impact will the script analysis create on the system, since the analysis is done in Oracle?

Frequency

How often you should run scripts varies, but we have some guidelines to help you determine the answer. You do not want to take snapshots too often, since you could degrade the performance of the database. I would suggest taking snaps an hour or more apart at first. Taking a snapshot every hour is going to be noticeable since it collects all of the data held in the GV$SQL and GV$SQLTEXT views. You might consider taking snapshots every two hours, but you want to ensure script data is not being flushed from the aforementioned views. Taking more snaps to start will allow you to perform analysis of several snapshots to determine the correct frequency. My example below is built with snapshots every two hours, with several “at –f” commands commented out to reduce the number of snapshots taken.

The frequency for days per month, like snaps per day, can start higher for initial analysis and then taper off.  Even though it will have a higher impact, this greater frequency helps create a baseline. Depending on your site, this start could be every day for one week of the month (which will help account for workload changes according to the day), or two or three weekdays in a month.

Impact

The cost associated with script collection and analysis will be available in your data, so be sure to review the cost of DM2_SQLAREA_GATHER and DM2_SQLAREA_ANALYZE scripts in your output files. Make sure to schedule snapshots around each DM2_SQLAREA_ANALYZE command execution, even if you run all of them at a low usage time of night (my example later runs all of them at once).

AIX Auto Collection

To see a detailed example from a previous AIX site I worked with, click here.

Prognosis: Automation will create consistent data collection for more reliable script analysis.