Millennium Medic

A Time to Use Messages.Suppress

By David Hallberg

If you are a regular reader, you know that over the last couple of years I have strongly advocated minimizing the use of Cerner Millennium’s messages.suppress file. Functionality that was intended to prevent extraneous events from cluttering up your message logs has been horribly misused by Millennium system administrators and the application support and application development staff. As a result, vital events are not being sent to the message logs, events that need appropriate troubleshooting and longer term problem resolution. (See “When No News Is Bad News” and “File Can Hide Serious Issues.”)

In recent months, however, I have seen some improvements in Millennium’s base tuning and configuration when a particular event is suppressed, so today I have an unexpected recommendation.

The improvement involves an event called SEC_seclibAudit, which is an Audit-level event telling you that debug is not enabled for the security components. A couple of years ago, only CPM Code Cache Manager seemed to log this event. With the 2010.x code, however, SEC_seclibAudit appears to be written by all of the Millennium servers. It generates two rows in the Data column of the message log each time it occurs. The first row says:

[SecDebugLog::Instance] Creating debug logging singleton.

The second row says:

[SecDebugLog::CheckConfiguration] Debug logging not enabled for all security components, componentLogicalName (SecDebugLogsecrtl)

This is a good setting, but you don’t need to be told about it with every executable. It would be better if you could assume the setting is off unless you took action to turn it on. (To enable debug logging, set the LogLevel of a server to 4 or set the environment variable or logical of CPM _GLOBAL_LOGGING or any one of a number of other logging increases that are available to individual Millennium servers.) This message has become so prevalent, logging on all servers, it has become noise. One client, who cycles 19,000 servers each week, gets 38,000 messages every week for this one event.

To stop the noise, I have to make a recommendation I rarely make: Put this event in the $cer_mgr directory’s messages.suppress file. Then cycle all of your multi-instance servers to make the change take effect.

Prognosis: Using the messages.suppress file appropriately stops extraneous logging and keeps your message logs clearer for real problems.

Easy Tuning of Oracle Sequences Improves Throughput

By David Hallberg

In more than 13 years of helping clients tune their Cerner Millennium^® systems, I have only recently turned my analysis to Oracle sequences. I have the IT team for a PowerChart client to thank for my newfound understanding of how sequence numbers can increase the variability in system response times…and drive clinicians crazy.

After engaging me to diagnose their PowerChart performance problems, the team corrected several MQ queuing issues and other straightforward problems, but clinicians were still experiencing slowdowns during their peak usage times. The culprit turned out to be the Oracle sequence numbers.

Oracle sequences give an identifying number to certain elements in the database. Millennium has about 964 separate sequences, Oracle’s System and Sys have about 20 and 66 sequences, respectively, DBSNMP has two, and CMS_DBA and Panther_Read have one each. Oracle generates these sequences when new data or rows are added to the database. Sequences become a performance problem when you have a lot of new data that needs sequencing but the tables aren’t configured to adequately handle the load. Making the change is surprisingly simple.

Here’s what I discovered for my client. I took a 1-minute and a 30-minute sampling of the Oracle sequence numbers being generated by PowerChart. The following chart shows the results of the 1-minute interval: 

SEQUENCE_OWNER	SEQUENCE_NAME	CACHE_SIZE	Sequences Generated
V500	CHARGE_EVENT_SEQ	2,000	8,000
V500	OCF_SEQ	2,000	6,000
V500	CLINICAL_EVENT_SEQ	2,000	4,000
SYS	IDGEN1$	20	2,000
V500	PATHNET_RECOVERY_SEQ	2,000	2,000
V500	PERSON_SEQ	2,000	2,000
V500	LONG_DATA_SEQ	2,000	2,000
V500	CPMAPP_APPLCTX	20	1,320
V500	CPMAPP_APP_CTX_ID	20	960
V500	CHART_DIST_LOG_SEQ	20	40
V500	ENTITY_LOCK_SEQ	20	20
PANTHER_READ	AUD_ORA_ERROR_SEQ	0	7
SYS	ORA_TQ_BASE$	0	2

The key columns are Cache_Size compared to Sequences Generated. The Cache_Size shows how many sequence numbers can be generated before updating the high water number, which is the highest sequence number ever created for that sequence name. It takes time for Oracle to update this number, so you don’t want the system to have to do it too often. Increasing the Cache_Size decreases the number of times the high water value needs to be updated and, therefore, improves the throughput for the tables that use that sequence number.

Let’s look at an example in the chart. The Sequence_Name of Charge_Event_Seq generated 8,000 sequences in 1 minute. Because its Cache_Size was only 2,000, Oracle had to update the high water value four times during that minute. The Sequence_Name of Idgen1$ generated far fewer sequences, only 2,000, but its Cache_Size of 20 meant that Oracle had to update its high water value 100 times in this same minute. You can reduce this overhead associated with sequences — and improve the throughput for their corresponding tables — by increasing the Cache_Size to match or, preferably, exceed the number of sequences generated during your time frame. I recommended my client set Charge_Event_Seq to a Cache_Size of 9,000 and Idgen1$ to 2,000.

The change can be done in uptime without taking any action on the Millennium application nodes and without doing anything else on the Oracle nodes. The change is accomplished via SQL*Plus with this command:

       Alter sequence charge_event_seq cache 8000;

In CCL, use the following command:

       rdb alter sequence charge_event_seq cache 8000 go

The following chart shows the results from my 30-minute interval, which are a little more staggering in terms of the impact that high-water-value generation has on overall throughput. It also shows that only 35 sequences — about 3 percent of the more than 1,050 — had increases in their sequence numbers during this time (32 of 964 for V500 or Millennium and 2 of 66 for Sys).

SEQUENCE_OWNER	SEQUENCE_NAME	CACHE_SIZE	Sequences Generated
V500	CHARGE_EVENT_SEQ	2,000	106,000
V500	OCF_SEQ	2,000	76,000
V500	CLINICAL_EVENT_SEQ	2,000	48,000
SYS	IDGEN1$	20	40,000
V500	CHART_VIEW_SEQ	20	24,260
V500	CPMAPP_APPLCTX	20	22,660
V500	CQM_TRIGGER_ID_SEQ	2,000	18,000
V500	CPMAPP_APP_CTX_ID	20	16,160
V500	CQM_QUEUE_ID_SEQ	2,000	16,000
V500	ORDER_SEQ	2,000	14,000
V500	PATHNET_RECOVERY_SEQ	2,000	14,000
V500	CARENET_SEQ	2,000	8,000
V500	PATHNET_SEQ	2,000	6,000
V500	PERSON_SEQ	2,000	6,000
V500	DM_CLINICAL_SEQ	2,000	4,000
V500	EKS_DLG_EVENT_SEQ	2,000	4,000
V500	ENCOUNTER_SEQ	2,000	4,000
V500	SCH_CONVERSATION_SEQ	2,000	4,000
V500	AAR_REPORT_SEQ	2,000	2,000
V500	CHART_REQUEST_EVENT_SEQ	2,000	2,000
V500	CHART_SEQ	2,000	2,000
V500	ENCOUNTER_ONLY_SEQ	2,000	2,000
V500	ENTITY_LOCKKEYID	2,000	2,000
V500	ENTITY_RELTN_SEQ	2,000	2,000
V500	ESI_LOG_SEQ	2,000	2,000
V500	MEDADMIN_SEQ	2,000	2,000
V500	NETTING_SEQ	2,000	2,000
V500	OMF_SEQ	2,000	2,000
V500	PHARM_FILL_HX_SEQ	2,000	2,000
V500	REFERENCE_SEQ	2,000	2,000
V500	RESULT_SET_SEQ	2,000	2,000
V500	ENTITY_LOCK_SEQ	20	400
V500	CHART_DIST_LOG_SEQ	20	300
PANTHER_READ	AUD_ORA_ERROR_SEQ	0	125
SYS	ORA_TQ_BASE$	0	4

So I recommend only changing sequence numbers that are being used a lot. In this case, I recommended:

• Charge_Event_Seq and OCF_Seq have Cache_Size set to 10,000
• All V500 sequences with more than 2,000 sequences generated have Cache_Size set to 8,000
• All V500 sequences that match their Cache_Size be left at 2,000, at least initially
• All V500 sequences with a Cache_Size less than 2,000 be increased to 2,000
• Sys Idgen1$ have Cache_Size set to 10,000
• Sys Ora_Tq_Base$ have Cache_Size set to 20
• Panther_Read have Cache_Size set to 2,000

It is possible to make the Cache_Size too large, so my recommendations are fairly conservative. Still, the client will have much less variability in Millennium response times after implementing these straightforward fixes.

Prognosis: Keeping your Oracle sequence Cache_Size properly set is a simple, uptime tuning procedure that decreases performance-robbing events and variability in your Millennium environment.

Setting Proper LogLevels Is as Easy as 0-1-2-3-4

By David Hallberg

I am often asked about proper log level settings (LogLevel) for Millennium^® servers. The setting determines what types of messages are written into a server’s message log files, which provide critical information to support teams for correcting application problems and some technical issues. If you set the LogLevel too low, you might miss important information. If you set it too high, you might be flooded with extraneous messages (and overlook the ones you need to address).

LogLevel can be set to one of the following values:

• 0 = Error (ERROR on Java servers)
• 1 = Warning (WARN on Java)
• 2 = Audit (AUDIT on Java)
• 3 = Information (INFORM on Java)
• 4 = Debug (DEBUG on Java)

The settings are cumulative, meaning that servers will write messages from that level as well as all lower levels. For example, a 2 setting would include all Audit (2), Warning (1) and Error (0) messages. You can view LogLevels and change their settings in OnTrack Panther™, SCPView or Olympus. Any time you change a LogLevel you need to cycle the server in order for the change to take effect.

In general, the lower the number, the greater the severity of the messages sent to the log files. Unfortunately, many Audit events are actually failed messages or transactions, which is one reason why a too-low LogLevel might result in lost information…and why many people have questions about what the settings should be. That wasn’t always the case.

In 1997, when most Millennium clients were running on VMS, disk saturation concerns meant that LogLevel was always set to 0/ERROR. When you had two or more application nodes with hundreds of servers pumping log information on the same disk drive, the drive — and everything else — would slow down. AIX, with its SSA drives, had problems as well. The drives shared the same connections as the database. If the executables were using up the connection from the CPU to the disk, then the Oracle database could not get the disk time it needed to operate as well as it could.

Disk saturation is not as much of an issue now that most clients are on 2007.02 or greater, many VMS clients have migrated to some flavor of Unix and the AIX clients are off of SSA drives. Yet logging can still be a problem, even when you are running on a well-tuned storage area network. Today, logging usually manifests itself as a specific queuing problem on a specific executable. If you set servers to LogLevel 0/ERROR, you might miss information that is important to use in finding and correcting problems in your environments. I recommend you no longer use this setting.

I also recommend that no server be set to LogLevel 3/INFORM or 4/DEBUG unless you have an active point asking for debug information. Short of that, these settings cause too many disk writes. I was working with a client recently who had the EKS servers in debug, and the message logs were wrapping in less than 15 minutes. This much logging slowed down all of the EKS rules processing but provided no benefit to the client or its Millennium support team.

So what should you set your LogLevels to?

Monitoring servers — such as 27, 29, 33, 66, 76, 97 and 98 — should be at least a LogLevel 2/AUDIT. The Audit level provides a great deal of useful information when you are working through issues or looking for problems before they actually become outages.

I recommend a LogLevel of 1/WARN for the servers listed below. As a general rule, if your executable is not listed, I recommend you set the LogLevel to 2/AUDIT. You might find you need to make some specific tuning or tweaking to these settings, but they provide a great starting point.

SCP Entry ID		SCP Description
72		CPM Notify Server
73		Time Repository Server
75		User Services
80		CPM Script Cache
86		CPM Expedite
92		OMF PowerVision Document Server
127		System Print Server (Node Specific)
130		PathNet Resource Routing Server
131		SCS Netting Server
132		SCS Collections Server
133		PathNet Result Order Posting Server
134		PathNet Sync Result Order Posting Server
134		PathNet Synchronous Result Order Posting
140		Pathnet - AP Case Retrieval Server
141		Pathnet - AP Automatic Diagnostic Coder
144		Pathnet - AP Coding Cache Manager
205		Clinical Event Server For ESI
209		Clinical Event Server - Batch
221		Search Server
231		MDI Result Server
235		MDI Order Server
237		BMDI Result Server
239		DI Inbound Client
252		FSI ESO-CQM Server
275		FSI ESO Notify Server
280		FSI ESO Server Load #1
281		FSI ESO Server Load #2
282		FSI ESO Server Load #3
283		FSI ESO Server Load #4
284		FSI ESO Server Load #5
285		FSI ESO Server Batch #1
286		FSI ESO Server Batch #2
290		FSI ESO Query Extract Server
291		FSI Batch CQM Server
335		PM Trans Async
340		PM Query
369		Chart Coding Services
375		KIA Event Server
425		FRN Tracking View Server
455		Health Expectation Server

Prognosis: Keeping your LogLevels properly set will ensure that your IT team is notified of the issues it needs to address without being overwhelmed with meaningless messages.

It’s Just a Fax, Right? Part 3

Troubleshooting your RRD application

By Chris Lanaman

For the past two blogs, we have looked at the important role faxes play in relaying critical information from the hospital to doctors’ offices and how a reliable fax service can improve relationships and business with these offices. Today we turn our attention to making sure Millennium’s Remote Report Distribution (RRD) servers, which send the faxes, are operating at peak performance.

It’s not always an easy task — which is why in my first blog on this subject I recommended that management dedicate at least one full-time analyst to support faxing. Gone are the days that you walk over to the fax machine, set your forms on the paper feeder, type in the phone number, hit a button and away it goes. Now many processes typically occur before a document reaches the RRD queue and is ready to be made into the fax and sent on to the primary care physician.

When training new analysts over the past few years, I often taught them the Cerner RRD application first, before graduating them to the Clinical Reporting applications. This method might seem backward, but I found it to be the most effective. With most things in the IT world, the best way to learn how to fix things is to start at the outcome and then work your way backward. If your end-users are saying they didn’t get the fax, you check the RRD server.

RRD technology has not changed much in recent years. Because of this, RRD servers can quickly become overwhelmed when a hefty number of faxes are pushed through them. If you’re experience faxing delays or failures, I recommend you check the following:

1. Operating system. Imagine sitting at your computer all day and faxing a Word document over and over, all day and night, through the computer’s modem. At some point, you would become exhausted and would need a break to regroup and refresh. Similarly, the RRD application needs to be refreshed. Small things like clearing the memory (cache) and straightening (defragging) the files are often forgotten but can help speed up the process.
2. Fax volume. It’s important to know the number of faxes you are sending each day. If you’re sending more than your RRD servers can deliver in a timely manner, you might need to beef up your RRD servers. Remember, as I stated in an earlier blog, delays in fax deliveries can be very damaging to your organization’s relationships with referring physicians.
3. RRD services. Make sure the RRD services are running on your RRD servers. Without these running, you simply have a desktop computer not doing anything.
4. Phone lines. Another area that often gets overlooked is whether the phone lines are still active. Troubleshooting might be as simple as making sure you have a dial tone on the lines going into your RRD servers.
5. Modems. Check to see if your modem is hung and not releasing your transmissions. If you use Softek Panther, make sure the RRD: Stalled sensor is set and then check for any Error messages. These messages might indicate a problem with your modem, not your RRD system.
6. Obsolescence. Sometimes customers have fax machines that are so old that the Smithsonian is looking to add them to its collection. Ask your customers for the make and model of their fax machine, and then check to see if your current RRD application level supports that model. Keep in mind that if you are not using internal modems on the RRD server, and instead have serial modems attached to the RRD server, those serial modems might be the issue as well. Sometimes a simple reboot fixes the problem.

As you can see, Cerner's RRD application around faxing can get hung up in a lot of areas. That’s one reason why tracking faxes for a healthcare organization is a full-time job. The next time someone tells you, “It’s just a fax,” remember you have your organization’s reputation – as well as a lot of money - riding on the fax getting where it needs to go.

Prognosis: Troubleshooting Millennium’s RRD application is much easier when a full-time member of the IT team can own the maintenance and support of this important faxing service.

 

Let’s Put Clothes on the Emperor

 By David Hallberg

When clients are looking for improved Millennium® performance – which they almost always are – our conversations often go like this:

Client: “Can you please help me parse and manage the RTMS timer files?”

Me: “Certainly. But I thought we wanted to improve Millennium’s performance for your clinicians and business staff. The RTMS timers will do nothing to assist in resolving performance for the point of care people and the business people utilizing Millennium.”

Client (with great indignation): “What are you talking about? The RTMS timers are THE performance metric for Millennium.”

With a name that stands for Real Time Monitoring System, you would assume that statement to be true. Its truth, however, depends on how you define the word performance. To clinicians and the business staff, performance relates to how quickly they can get their job accomplished. They are concerned with Millennium’s speed. The data center staff, especially those at hosting vendors, define performance as uptime. They are concerned with the amount of time that the backend systems are available. Some may boast that they offer a performance guarantee of 99.8%. Such assurances mean nothing to clinicians, who assume that the system should always be up. Their standard is a system that runs, in the words of a friend of mine, at blink speed.  

And while RTMS timers once did measure speed, Millennium architecture changes have altered the meaning of the data the timers deliver. The result is that end-users might complain that the system is slow but the metrics look fine inside the data center. A bit of history will help explain the discrepancies.

Millennium did not originally have RTMS timers in its code. Instead, the middleware had a setting to allow the Citrix server or fat client to log all of the transactions being sent from that device to the application node. In the days when the applications and database resided on the same node, this log was a great metric. However, the setting had some drawbacks. There would be an observable performance degradation in Millennium applications with this logging turned on (degradation of 10 percent or more was not unheard of), and the approach did not show any grouping of applications and functions. So it was a great tool for the middleware team but not so helpful for the application engineering teams.

A grouping feature was eventually deployed. Initially, there was a high degree of linking between the functions being measured between the Citrix server or fat client and the application node, which proved helpful in seeing backend performance issues. Over time, though, we have seen that Windows tab changes and other functions actually only measure how fast the Citrix server or fat client hardware is running rather than how long it takes the application and database nodes to process a transaction for the end-user. I call this “RTMS timer death.” You now have lots of functions being timed, but you can no longer be assured that the function being measured is actually providing a roundtrip view of the transaction time.

Several other changes are also obscuring the quality of the RTMS timer data.

1.    A greater and greater reliance on asynchronous work leads the interactive user to believe the system is more responsive, but it is in fact a bit of a performance cheat. For instance, I may do a function in PowerChart in which a request is sent to CPM Script and CPM Script responds when the function is complete. In fact, even though CPM Script is done with the function, CPM Script actually sent the request to CPM Process and CPM Process exploded my request into numerous other requests that it then handed to other Millennium executables.

2.    If the Citrix or fat client application (e.g., PowerChart) crashes or is X’d out, you will see a DBAPI_Exception event for the Millennium application executables that were processing the requests from this client. The Data or Record information starts with “DbDestroyTimer: invalid object handle: xxx,” where xxx is the handle being used to update the RTMS timer data. What does this mean? It’s incomplete data. It’s not clear that each of these events represents an RTMS timer that started but never finished, nor is it intuitive that there is a relevant impact, much less intervention, to consider.

3.    Two files in $cer_mgr – slareporting.cerner and timer.cerner – control whether a timer is active, what functions are used, if the timer meets the hit/miss count, and the timer’s average or mean time. Though it wasn’t originally the case, the CRM Timer information is now configured in the timer.cerner file. In the slareporting.cerner file, the 54 application timers are defined. In the timer.cerner file, the 1,022 functions are defined. This would seem pretty good – until you remember that there are more than 35 million lines of code for Millennium, more than 4GB of Citrix and fat client files, and more than 5,000 Oracle tables. In the slareporting.cerner file, by definition, you have to have each of the 54 applications run 500 times in two hours in order for the data to be counted. You also have to have each application run at least 10,000 times a month or the data does not count.

Let me use PowerChart to explain the potential for problems. Let’s assume that it’s 7 a.m. and physicians are starting to round, nurses are changing shifts and many other clinicians are getting down to their day’s work. In all, 499 clinicians log into PowerChart from 7-9 a.m. The IT department gets reports from these users that Millennium is slow. IT checks the slareporting.cerner file and sees nothing counted for the day’s performance. Why? The slareporting.cerner file must have a minimum of 500 functions of an application – like logging into PowerChart – in a two-hour window, or they do not count. Over the course of a month, if your site did 9,999 PowerChart logins, it would be stated that you did not do enough work to be statistically significant. For the purposes of a Lights On Network™ evalution, this data would not be considered substantial enough for valid comparison. Therefore, if you logged a Service Request stating you are having performance issues, you are likely to be told that based on the data you do not have any issues.

The initial intent of the RTMS timers was right on the money: to create a mechanism to allow Millennium sites to look at a clinical function and understand where the issue is. However, the execution and changes over time have made the timers unusable. Every client who I have worked with in processing their RTMS data has logged Service Requests and been told either 1) there is no problem or 2) a code level update will make the problem go away.

This approach to analytics showcases the different definitions of performance that I mentioned earlier. The clinical and business staffs are telling you that Millennium is slow and unpredictable, but the data shows no performance issues. This conundrum with the impenetrability of the data leads many organizations to conclude that the environment is unmanageable and they need to outsource the support for Millennium Citrix servers, application nodes, Oracle databases, etc. After all, the outsource vendors guarantee “performance” to be 99.8 percent.

I’ve already discussed how meaningless that statistic is to everyone outside the data center. A 99.8 percent uptime from the clinicians’ perspective makes no claims to speed and probably means that the .2 percent downtime will occur when they need the system the most. Here’s one more caveat: The statistic refers only to unplanned downtime. The two to four hours of scheduled downtime for weekly preventative maintenance are not included in the number. I have been in healthcare IT since 1989. I have been employed by four different hospital systems, have been an HIT consultant and have worked for Cerner, and I have never been allowed to have that much downtime in a month, let alone a week.

Nor was I allowed to ignore the clinicians’ desire for blink speed. When I did performance benchmarking at a facility running a different clinical system, we had the system tuned so we could do the screen changes in 0.6 seconds. Seriously. On one occasion we had a drive problem that spiked the time to 0.9 seconds. The screen flips were 0.3 seconds slower, about the time it takes to blink your eyes, and our help desk phones were on fire from the outraged clinicians complaining of performance problems.

So what can you do to help resolve your performance problems? The first step is probably the hardest. Stop believing the propaganda that your system is fine. Anyone telling you that you don’t have a performance issue that can be tuned is like the “tailor” telling the emperor that his new clothes are exquisite. They’re trying to convince you of something your eyes and brain know is not true. Listen instead to the people who interactively use your system every day. The clinicians and business staff will be the first to notice a difference in performance. The next step is trying to understand what you can do to monitor the performance of all your Millennium systems. The last step is to determine what staff and budget changes you need to make to address the issues.

There is a lot you can do with the tools you already have. You’ll find lots of tips in previous blog postings. I also encourage you to respond to this blog with your own questions and insights. I welcome a lively dialogue.

Help Your MQ Become PDQ

By David Hallberg

For those of you trying to get the most out of your MQ system for Millennium®, a few simple adjustments can deliver big benefits.

WebsphereMQ is an IBM middleware used extensively by Millennium. It has two major roles and one minor role.

1.    MQ’s most important use is for messages that have to get processed even if an individual application process or an application node crashes. This security measure means the data has to be written to log files and data files on disk drives, making it the slowest type of transaction for MQ.

2.    The 2007.x and 2010.x versions of Millennium, assuming MQ Shared Services (MQSS) is enabled, use MQ for communication from a client, like Citrix, to the application node. This queuing is non-persistent, meaning if the Citrix client or application node crashes, then the data is lost.

3.    Several years ago the Foundations team worked with the Foreign System Interface team to allow interfaces to use MQ.

For the first and third uses of MQ, the mqs.ini file defines how the logs are configured. Proper tuning of this configuration can have a tremendous impact on throughput. For MQ 6, several of the older limits have been increased. For HP-UX, AIX and Linux, you can now define up to 510 LogPrimaryFiles and up to 509 LogSecondaryFiles, with a maximum combined total of 511 LogPrimaryFiles and LogSecondaryFiles. This is a significant change from MQ 5.3, where the maximums were 62 and 61, respectively, with a maximum of 63 combined.

Setting these log files to their maximums makes the MQ log filesystems much larger, but it also allows for much greater concurrency. This concurrency allows certain functions — such as adding, updating, deleting and reading information — to run at the same time. It also makes it more likely that long-running batch jobs will complete instead of looping because the log files have wrapped.

Next we’ll look at the setting of my favorite mqs.ini entry, LogWriteIntegrity, which controls how many times a transaction is written to a log file. The default value is TripleWrite, meaning every update has to be written to three log files before the next step in the processing can occur. Why all the redundancy? To be fair to IBM, I believe they recognize that they have no control over the hardware and operating systems MQ is installed on. If your hardware server has no disk redundancy, IBM is giving you a reasonable chance to recover data if your node crashes or your drive dies.

Redundancy, I’m glad to report, is not an issue with Millennium production systems. Even old AIX systems have mirroring, that is, two drives with the identical data on it. If one drive dies, the data is available on the other. On current SANs, a writeback cache typically has a battery backup or a mirroring of the data on different memory arrays. The SANs are normally configured with RAID 5 drives, so the loss of a single drive does not mean any lost data. Thanks to this redundancy, you can change the TripleWrite default of the LogWriteIntegrity entry to SingleWrite. You should see improved MQ performance without risking any of your data.

As far as I know, the Cerner Millennium System Settings (CMSS) have not been updated in this area to help Millennium organizations achieve higher throughput from their investment. If any of you have current access to the CMSS for your operating system and the document is updated to include this, please let me know. I would enjoy hearing that the system tuning is keeping up with newer technology.

One last note, the LogWriteIntegrity parameter is valid on MQ 5.3 as well. You can get more information from IBM’s WebsphereMQ System Administration Guide for Version 6.0 (SC34-6584-00) and WebSphere MQ System Administration Guide for Version 5.3 (SC34-6068-02). Please keep in mind that I recommend you use normal change control processes for this tuning and then migrate any changes into your production environment.

Prognosis: A little MQ tuning goes a long way. You may be surprised at the benefits you see when you increase old log file limits and remove unnecessary redundancies.

 

Fragmentation’s Biggest Rift? The Generation Gap

By David Hallberg

My last blog covered deleting old files from your Millennium® application nodes. This week I’ll discuss the second step for keeping Millennium's speed consistent over time when you’re tuning filesystems and directories: defragmentation. Just like users disagree over the need to purge unneeded files, they also disagree over the need to defragment files.

I’ve had some interesting conversations over this issue this past year. People weaned on mainframes and older operating systems universally say, "Defragment!" Those who grew up on Linux, HP-UX or AIX virtually to a person say, "Defragmentation is only needed for the old and slower operating systems." With the latter group, I have to remember to stay calm and remind myself that their ideas probably come from the websites and authors who claim that, except with Windows, fragmentation is a thing of the past. But I also remember the documentation from the vendors who wrote the operating systems and the supporting filesystems. Plus I remember my work on the Advanced Technology Group at Cerner and what we benchmarked on AIX, VMS and Windows. Only then do I begin the conversation.

Filesystem and directory fragmentation occurs on all of the operating systems supporting Millennium®: VMS, AIX, HP-UX, Linux and Windows. There are generally two types of fragmentation: file fragmentation and space fragmentation.

In the first type, a file, say dic.dat, becomes too large to fit into a single disk block, so it has to be split into two or more blocks. If the dic.dat file is then modified, it may need a third or fourth block to fit within. It’s likely that code upgrades or custom CCL changes will have added more files to the filesystem where dic.dat is located, and they will have been saved to the blocks right next to the dic.dat's first two blocks. That means the third and fourth blocks for dic.dat will not be able to be written right next to (or contiguous with) the first two blocks. Now the dic.dat file is fragmented. With the file split into two chucks, the operating system has to perform two reads to completely read the dic.dat. When the disk drive or RAID set has to do more reads (disk I/O's), performance suffers.

Space fragmentation occurs when you delete a file, typically because it was a temporary file. Once deleted, the space it originally occupied leaves a gap on the filesystem. Many deleted files create greater fragmentation, making it impossible to write a file contiguously.

The filesystems/directories I discussed in my last post ($CCLSCRATCH, $CCLUSERDIR, $cer_temp, $cer_fifo, $cer_usock, $cer_log, /var, /tmp) are all designed to hold temporary files, such as print files or logging files, that will be deleted as soon as they are no longer needed, e.g., daily or possibly hourly.

The downstream impact of this design is that all current operating systems need to have both types of fragmentation corrected. Vendors provide tools for this defragmentation, including defragfs for AIX, fsadm for HP-UX, and fsadm for CAMM (Millennium's PACS) and the Vxfs filesystem that it uses. VMS has the DFU tool on the freeware CDs that has to be installed before it can be used. Also for VMS, HP has a DFO tool and Diskeeper offers Diskeeper for VMS, which has been around since 1986. You’ll have to pay for these latter two tools, but it might be worth investing in one of them if you’re trying to keep your VMS system. For Windows, you can use the built-in tool called Disk Defragmenter; a third-party tool like Diskeeper 2010 and its various versions or PerfectDisk 11 and its various versions; or one of the plethora of freeware/shareware tools, my favorite being Defraggler from Piriform. I encourage you to spend an hour or so looking through the numerous white papers, blogs and forums discussing these options.

My first team in Cerner's Advanced Technology Group was tasked with successfully completing client conversions no matter what technical hurdles we faced. One of our biggest obstacles was severe fragmentation of both files and free space. We had such significant file and space fragmentation at several sites that we added a drastic step to our normal recommendations. In addition to deleting lots of files daily and defragmenting the filesystem at least once per week, the clients had to set up a schedule to point the filesystem, say $cer_print, to an entirely new filesystem every three months. Miraculously, their systems became much more stable.

Why did it help to move the filesystem? Each filesystem had an index that kept track of where each file started, the size of the file, where it ended, etc. As thousands of files were created and deleted every day, the operating system had to slow down in order to keep track of them. The periodic refresh of the Inode or indexf.sys (on VMS) brought performance back to a better level. With today’s improved drive technologies, this type of severe fragmentation is less likely, but it still needs to be reviewed. Some of you might find it necessary to recreate your filesystem on another set of disks and benefit from having contiguous files and free space.

Prognosis: Using defragmentation tools on a regular basis to reduce filesystem fragmentation on all operating systems will increase Millennium’s speed, consistency and reliability.

Deleting Old Files

By David Hallberg

Today’s topic is an issue over which Millennium  users disagree: whether the number of files in a directory or filesystem impacts the Operating System (OS). Many of you say no, but I believe it does. Having too many files in a directory or filesystem with no scheduled purging of those filesystems will degrade your system’s performance.

Those who state that the number of files has no impact on the OS begin with a great assumption: that Millennium was written by the OS vendor and that it is compiled for the OS version it is running on. Those versions include 64-bit for VMS, AIX and Linux and 48-bit for HP-UX. Unfortunately, the Millennium application is written with the 32-bit compilers for each of these platforms. Under ideal situations, the OS may not care about how many files exist, so long as there is free disk space, but Millennium does. The number of files affects CCL report generation to either your Millennium session or a printer and definitely affects the primary Millennium print servers, such as DCP Output (SCP 106), System Print Server (Node Specific) (SCP 127), Print Services Cache Manager Server (SCP 128) and System Print Server (SCP 129).

Let me give you an example. The SysSrvPrint error message from the servers indicates that the Millennium executable cannot find the file in the filesystem. This event can happen because 1) the file is on another node, 2) the file was purged before it was printed, or 3) it took too long for the executable to look up the file. That third failure reason is easy to prevent. If you delete files from the $cer_print and $CCLSCRATCH directories every four to six hours, you should be in good shape.

You can count how many files are in the $cer_print directory or filesystem using this command (the last character is a lowercase L):

ls $cer_print | wc -l

Likewise, to see a count of the files in the $CCLSCRATCH directory, type the command:

ls $CCLSCRATCH | wc -l

The goal for optimal performance within all Millennium solutions is to have less than 5,000 files in any of these directories at any point in time. Due to the number of facilities and solutions you have and the Millennium version you use, you might not always be able to achieve this level. But I encourage you to keep working towards it.

The crontab job scheduler in the $cer_proc directory comes with a UNIX Korn shell (ksh) named clean_wh_env.ksh that deletes unnecessary files. The ksh will probably have to be tweaked to fit your specific site. This takes a little getting used to if you are unfamiliar with UNIX scripting but will greatly improve Millennium’s speed and reliability. Beginning from the directory count you ran from the commands listed above, identify the directory that contains the greatest amount of files and then modify the clean_wh_env.ksh file either 1) to be run more frequently from crontab or 2) to delete additional file patterns or extensions.

The ksh should also be used to clean up the following additional filesystems: $cer_temp, $cer_log, $cer_usock, $cer_fifo, $cer_lock and $CCLUSERDIR. Access the file count in these directories with the command listed above. For these directories, deletions scheduled to run once a day should be enough.

Prognosis: Using your Operating System’s scheduling service to routinely delete unneeded files from your filesystems will increase Millennium’s speed and reliability.

Help Citrix Handle Unexpected Login Spikes

By David Hallberg

Recently I was working with a client who had experienced an unexpected power outage. Once the power returned, the PCs had to be rebooted. The client uses Citrix to access Millennium, and the rebooting overwhelmed the Citrix Zone Data Collectors. The overload delayed system access for everyone trying to get back into Millennium® — even though the data center stayed up during the outage.

Was the overload inevitable? Or would it have been possible to tune the site’s five-year-old hardware to better recover from or live through abnormal spikes in activity?

These questions are important because many sites use Citrix’s XenApp servers to deploy Millennium. Citrix — how most users refer to all of the various Citrix components — allows a small set of servers to present Millennium applications, which individual PCs then access by connecting to the Citrix servers. The Zone Data Collector (ZDC) is used by Citrix to evenly balance the PC connections across all of the XenApp servers. Without the ZDC, the first Citrix server in the group or zone would get all of the connections, which would quickly overwhelm the server and cause other issues.

As my client discovered, even with the ZDC, abnormal events can overwhelm the server. So I want to suggest some basic tuning steps to help you balance the ZDC loads and ward off future overloads. We’ll discuss the data store (often SQL Server) tuning and registry changes that can make the data store and ZDCs able to handle sudden increases in volume.

I would suggest you review your Citrix configuration even if it was set up by an “expert.” Often the experts are skilled in following a specific set of requirements to set up an environment or domain and show you how to do it next time. Then they leave. The more knowledge you have about your environment and its configuration, the better you are positioned to address potential issues. Ask yourself, “If I had to accommodate a doubling of connections or clients in less than 60 minutes, where would I have problems?” After reading through the steps I recommend, you can create a list of what tuning needs to be done and start the change control process to put these changes into production. That way, the next time you have a power outage at your organization, the recovery time will be shorter and less painful for you and your Millennium users.

For details on the Citrix tuning process, click here.

Prognosis: By moving your data store to an optimized disk partition and tweaking some registry settings, you will find that Citrix can become much more responsive, even when the workload more than doubles.

If You Rewrite the Rules…

By David Hallberg

How I love rules. By following certain clear principles, you can be reasonably sure of the outcomes. If you wash your hands regularly, then you’ll spread fewer diseases. If you listen carefully, then you’ll be better able to address a person’s concerns. In the IT world, it’s called the if-then-else logic loop. But I’m the first to admit that rules have limitations. I’ll leave you to figure out their limitations in your personal life, but this week I’ll help you determine if some of the rules you have set up in Millennium® are a bit broken.

Millennium has a wonderful subsystem called Discern EKS, which allows users to identify special events that will automatically trigger other events. For instance, users can set a rule that whenever a patient’s white blood cell count is very high, say 15,000 to 30,000, an alert suggesting a blood culture will be generated. Each department determines the rules it needs based on its standard operating procedures, works with the appropriate teams in the organization to set them up and outlines its response when the rules are triggered.

It sounds straightforward enough, and most organizations don’t give much thought to the rules if they are running in less than a second or two. Run time alone, however, does not tell the whole story. Organizations can have hundreds or thousands of rules, some of which fire thousands or tens of thousands of times a day. They can present a huge drag on the Millennium system, and these tuning issues need to be addressed.

For example, if a rule takes one second to run – which would be considered a reasonably fast rule – but is triggered 1,000 times in one day, that one rule is using up 1,000 seconds – or over 16 minutes – of application CPU time. Rules firing more than once per minute – more than 1,440 times a day – need closer scrutiny. I would recommend that you look at that rule’s trigger mechanism and verify it really should be firing that frequently.

Rules that trigger once per minute are not uncommon. I’ve worked with clients that have rules triggering more than once per second – that’s 86,400 times a day. Now consider that many sites have hundreds or thousands of these rules. Even if all of them are so-called fast rules, you can still see where you might have Millennium performance issues.

To determine if you’re having issues with a rule or set of rules, we’ll look at the Oracle table EKS_MODULE_AUDIT. This table shows when rules are triggered and how long they take to fire, along with a lot more useful information. To learn how to access the EKS audit table and analyze its numbers, click here.

Prognosis: An analysis of the time Millennium spends each day running rules will show you whether any rules need to be rewritten to help the system run faster.

Shared Service Proxies in an MQSS World

By David Hallberg

Millennium®, since version 2007.02, has tried to reduce its proprietary middleware footprint through the use of IBM’s WebSphereMQ for client communications to the application nodes. This is often referred to as MQSS for WebSphereMQ Shared Services. It provides an enormous leap forward in throughput and performance for most Millennium users but has not lessened the need for Shared Service Proxies (SSPs), as was expected.

Millennium’s initial design specification for its proprietary middleware was 125 messages per second, which was pretty good when it was introduced 15 years ago. Plus it often saw higher throughput thanks to newer compilers, faster CPUs and some other changes. WebSphereMQ on version 5.3 for this sort of communication (often referred to as non-persistent queuing), however, was benchmarked at more than 3,200 transactions per second, and 6.2 is benchmarked at more than 3,800 tps, where each transaction is 2 kB in size.

The diagram above gives a simple overview of how shared services work with IBM MQ. Starting at the left of the diagram, we have a client application that is connected to both a request and a reply queue. These queues are used to transport the client request to the application servers and the reply back to the client application.

If you are an AIX or HP-UX client, I recommend that you enable MQSS as quickly as possible to help your throughput. I wish I could say the same for VMS clients. Unfortunately, VMS experiences capacity issues with MQSS. Its TCP/IP stack and BG devices can exhaust their limit of 32,768 ports, at which point they no longer take new connections. Cerner now recommends that all VMS clients stop using MQSS, except for the Java services, which only know how to communicate via MQSS.

While all of this speed is great for clinicians and business staff using Millennium, there is a hitch. Updated Millennium configuration standards state that when MQSS is enabled, the number of SSPs (SCP Entry ID 1) should be reduced to four and that all dedicated SSPs should be disabled. Based on the client data I’ve seen, numerous clients who have gone live with MQSS with only four SSPs have experienced varying degrees of queuing, backlogs or slowdowns in the SSPs. I recently worked with a client who had more than 82 hours of queuing for the CPM Script Batch 001 service.

Their queuing was typical of many others: It was constant, not centered around normal usage patterns of one peak in the late morning and another in the early afternoon. The only way to mitigate this backlog is to increase the number of SSPs or create a dedicated SSP, stop all of those servers on the node and then restart them. This action will move the service to the new generic  SSP or dedicated SSP and should stop the queuing. (To learn more about SSPs, read previous blogs on “SSP’s Path for Rapid Transactions” and “Setting up a Dedicated SSP.”)

This is a very surprising discovery. The Millennium switch to a commercial middleware like WebSphereMQ was supposed to eliminate this queuing. Millennium support might attribute the problem to the SSPs consuming resources that could be freed up for other processing. Although technically true, the impact of the SSPs is relatively small. An SSP starts with less than 16 MBs of RAM and can grow to about 128 MB of RAM. If I ran 16 SSPs that all grew to 128 MB (they often do not grow that large), I would be using the same amount of memory as any one of the Java servers (SCP Entry ID 351 through 370). Since the SSPs do not handle client communication — they don’t talk with PowerChart, FirstNet, PowerChart Office, etc. — they consume very little CPU. These processes do some disk I/O, but compared to CPM Script, SCP Entry ID 51 or the Java servers, it is such a small amount of I/O that it is really just background noise. The resource consumption, therefore, is too small compared to the rest of the Millennium application executables to have any significant impact to the resources of the node, lpar or vpar.

I still recommend that all non-VMS users switch to MQSS as soon as you can. The throughput gains are impressive. I just ask you not to reduce the SSPs to four. Although I don’t know your specific configuration needs, I would suggest you start somewhere in the range of 8-20. Eight was the old minimum recommended value for SSPs in non-production domains, and I have always recommended 20 as a base for any production domain.

Although VMS clients can’t benefit from MQSS, I do urge you to upgrade your system to make it as stable as possible and to ensure that you are able to use all 32,768 ports. Clients on VMS version 8.3 should upgrade to the appropriate VMS 8.3 with TCP/IP Services version 5.6 ECO 5 or later.

Prognosis: Even with MQSS’s welcome increases in throughput, SSPs still have a role in Millennium. Keeping them properly tuned will help prevent queuing in Millennium's proprietary middleware.

Chart Servers — Part 4

Tuning Windows OS

By David Hallberg

This week we move on to our third and final step in tuning your Millennium chart servers: tuning the Windows operating system (OS) on each server. In step 1, you determined which of your chart servers was the fastest and copied that server’s configuration to the other servers. In step 2, you tuned your Oracle database tables. Once you complete this last part of our holistic tuning approach, you can expect the process time of your chart servers to be less than half a second per page.

Tuning of the OS running chart servers should start with Cerner. A document on Cerner Millennium System Settings (CMSS) for Windows is available from Cerner Support. The document has about 15 pages and contains a couple of key items that relate to our topic: 1) Pay attention to the effect that anti-virus scanning software has on the solution, and 2) be wary of the recommendation to run a defragmentation tool at least monthly. I recommend that you defragment the chart servers every day.

Many freeware and shareware solutions are available to defragment your servers. I enjoy using Defraggler from Piriform (www.piriform.com/defraggler). It has a native 64-bit client but runs on a 32-bit OS as well. Diskeeper from www.diskeeper.com is another excellent solution if you have the operational budget for it. For any of you running VMS, Diskeeper has great VMS defragmentation tools as well as I/O analysis tools.

A lot of people are documenting how they tune Windows. Doing a search on Google for “XP performance tweaks” returns 8.5 million results, “2003 performance tweaks” returns 2.6 million, and “2000 performance tweaks” returns nearly 4.7 million. Of the many sites that have excellent information on tuning the OS, I have a couple that I keep going back to:

1.    PC Pitstop (www.pcpitstop.com) will run an audit on your server or PC and show you where you have potential performance and configuration issues. If you want to have the tool do the registry changes for you, there is a fee.

2.    SpeedGuide.net (www.speedguide.net/index.php) has a lot of topics. If you choose the Broadband option at the top left of the page and then select Registry Tweaks, you will find a wealth of information on tuning the Microsoft TCP/IP stack for optimal performance.

3.    Microsoft has excellent documentation at www.microsoft.com. Once there, search for “XP performance tuning” or “2003 performance tuning.” You will have to read through a lot of documents to determine which pieces you want to apply to your OS. That’s why I prefer PC Pitstop and SpeedGuide.net. I have found that by using them, I can quickly get the most from my OS.

My final piece of advice is to turn off the following two services, no matter what version of Windows you’re running:

1.    ComputerBrowser was designed for Windows 95 PCs to act as the WINS server for DOS and Windows PCs. It is very chatty and actually quite a security hole. There is no need for this to run on any chart server.

2.    Themes was designed to make the chart server have a graphically rich (pretty) view of the OS. I want chart servers to be fast, consistent and reliable. I do not need them to be pretty.

Prognosis: With apologies to the ′60s rock band The Byrds, I would encourage you “tune, tune, tune, there is a season, tune, tune, tune, and a time for every purpose….” The season is now and the purpose is to dramatically improve the overall responsiveness of the chart prints for clinicians.

Chart Servers — Part 3

Tuning Oracle’s Database Tables

By David Hallberg

In my first blog a few weeks ago about Millennium’s chart servers, I talked about the need to determine which of your chart servers is the fastest (that is, which server prints each page of your organization’s charts in the shortest period of time) and how to make a fairly simple adjustment to speed up the slower servers. By copying the configuration of the fastest server’s operating system to the other servers, the processing time from our sample site increased an average of more than 1 second per page over its five chart servers. So why don’t I move on to a new topic? Why would we do anything else to the chart servers after getting them to perform in the same manner?

As I’ve touched on in the past, a frequently misunderstood concept for tuning Millennium is that making one change will cure a problem forever (see “A Healthy Dose of Doubt”). A second misunderstanding is that good change control dictates that you only change one thing in Millennium at a time. These sound very good on the surface. After all, how often are we told to start small when making our New Year’s resolutions? If we want to lose weight, we will have better long-term success by choosing one change that we can maintain rather than a bunch of quick fixes.

With Millennium, however, things aren’t that simple. The Oracle schema (a fancy name for the way the data is held in the database) alone has more than 300 tablespaces, 4,400 tables and 11,000 indices. A system can have anywhere from a handful (say, five) of chart servers to more than 20 defined and functional chart servers. Changing one row per chart table per change request window under a system of every-other-week change control means that chart changes could take a year to complete. Why would you want your organization to suffer for that long?

I’m suggesting that you can address most of your chart performance issues in three change windows. I’ve already covered the first — making all of the chart servers have the same baseline Windows and Word image. Today I’ll help you tune some of the Oracle database tables affecting chart performance (click here for the details), and next we’ll look at ways to tune your Windows operating system just a little more.

Prognosis: A comprehensive tuning approach will increase the performance of your chart servers (much to the delight of clinicians and other chart users) and will give you a solid base from which you can monitor for future throughput issues.

Chart Servers — Part 2

Chart Prints Can Raise Reams of Issues

By David Hallberg

In my last blog, I showed how analysis and simple tuning of the Millennium® chart servers at one site saved nearly 7 hours in print times over the course of a week. I gathered those times from Millennium’s Chart Request Table, which also yields information about the size of the charts sent to each server. You might be surprised at what you can learn from those numbers.

Below is the Excel PivotTable we built from a week’s worth of requests sent to a client’s chart servers.

Chart Server Name	Count of SERVER_ NAME	Sum of PROCESS_ TIME	Sum of TOTAL_ PAGES	Pages per Request	Time per Page
DWHMOCHART03	728.00	4,216.97	1,315.00	1.81	3.21
DWHMOCHART04	3,982.00	19,332.78	6,390.00	1.60	3.03
DWHMOCHART05	1,068.00	5,409.00	1,961.00	1.84	2.76
DWHMOCHART06	2,363.00	12,296.50	3,747.00	1.59	3.28
DWHMOCHART07	1,081.00	23,780.52	1,857.00	1.72	12.81
DWHMOCHART08	1,892.00	9,179.29	3,433.00	1.81	2.67
(blank)		0.00	0.00	#DIV/0!	#DIV/0!
Grand Total	11,114.00	74,215.08	18,703.00	1.68	3.97
			2,671.86	<<Pages per Day
			5.34	<<Reams per Day

Let’s look at the Total Pages and Pages per Chart Request columns.

Dividing the number in the Grand Total cell for Total Pages by 7 gives you the total number of pages printed through the chart servers in a day. If you take that number and divide it by 500 (the number of sheets in a ream of paper), you get the number of reams of paper you are printing JUST for chart printouts. The site above is printing more than 5 reams of paper a day. I had one client generating more than 7,000 pages — more than 14 reams of paper — every day.

Most sites I work with are astonished to realize the amount of paper they are using. But it helps explain why nursing units and medical record/health information management departments have seen an increase in operational costs as the use of EMRs expands. Toner is used up more rapidly, printers may be pushed beyond their duty cycle (the maximum pages they can print per month and still be in warranty), or the amount of RAM or memory on the printers may have to be increased.

Knowing paper usage patterns for chart prints may be used for planning and acquisition of properly sized printers for both clinical and business areas. At one Millennium site I worked with, I recommended that all printers tied to Millennium have 128 MB of RAM. For Medical Records/Information Management, I recommended a duty cycle of at least 50,000 pages per month. The site represented in the table above is printing more than 80,000 pages per month. This data helps to explain why inkjet printers generally will not work well in a Millennium environment. Neither will printers with paper trays of less than 250 pages.

Prognosis: It is crucial for areas using Millennium chart printing to understand how much they spend in support of Millennium each fiscal year. Using the Chart Request Table to analyze paper usage for chart printing will help organizations look realistically at printer and paper needs…and budget accordingly.

Next: Tuning Oracle database tables to improve chart server performance.

 

Chart Servers — Part 1

Chart Request Table Dishes up a Feast of Data

By David Hallberg

Today I want to begin a short series on chart servers by introducing you to a table that many Millennium® users are unaware of: the Chart Request Table. This table dishes up a lot of information about how your chart servers are performing — particularly how long it takes the servers to process chart prints — so it’s important to know how to access the information, understand what it’s telling you and correct performance issues.

With all the other issues you need to manage in a day, you might be asking, “Why does it matter how long the chart print takes?” In a nutshell, chart prints are used to treat patients. Printing delays could mean a clinician does not have the latest information on the patient being treated. Waits of more than an hour for a larger chart to print are not uncommon. And a delay in printing one chart slows down all the other chart requests queued up behind it. With Expedite and Ad Hoc chart requests in particular, this lack of information has the potential to compromise patient care and it opens the organization to liability concerns.

I have found an enormous spread among clients on how long it takes to print charts. Some clients can process a page in 1 second, and others take more than 2 minutes to generate a single page. A 2-page chart that requires 2 minutes to process each page stays in the chart server for 4 minutes before being sent to the print server and finally the printer. At this rate, a 50-page chart takes the chart server 100 minutes — or more than an hour and a half — to complete the generation of a single chart request. That same chart, if sent to a well-tuned chart server processing at 1 page a second, would be sent to the print server in less than 1 minute.

I understand that your time, money and staff are limited. But access and analysis of chart server performance details can be done in minutes, and a skilled technician can correct slow servers in a few hours. It starts by accessing the Chart Request Table and exporting the data to a spreadsheet for analysis. Using this information to then reset your slower chart servers to match the fastest one will dramatically improve the clinically perceived speed of Millennium, reduce end-user frustration and quite possibly improve the quality of the delivered care. Click here for the step-by-step process.

And then keep your eye out for upcoming blogs, in which we’ll:

1.    Examine the implication of other pertinent data contained in the Chart Request Table — in particular, the servers’ print quantity.

2.    Dig deeper into Windows tuning techniques of the chart server that will reap even larger rewards for your clinicians.

Prognosis: Knowing which chart server or group of chart servers is fastest and getting all of the chart servers to perform at the same level can improve clinical end-users’ acceptance of Millennium and potentially some patient outcomes.

A Healthy Dose of Doubt

By David Hallberg

 “…and the only people I fear are those who never have doubts.”

– Billy Joel, “Shades of Grey”

One of the most frustrating discoveries I’ve made about Millennium’s technical infrastructure is that tuning is never done. If I think I’ve gotten all of the performance out of Millennium that I can, I need to step back and re-evaluate my metrics…or just wait a little while for other issues to arise.

Another important realization, one that I learned a long time ago, is that my knowledge and experience is not the pinnacle of knowledge and experience. No matter how hard I try I can never know all there is to know on a given topic. There is always someone else who has dealt with a similar performance/stability/capacity issue and knows more about it than I do. When working with clients, I prefer a collaborative approach that comes with a little doubt that the problem solving will ever be done.  Performance is an iterative process that has to be monitored and adjusted through time, not just at a single point in time.

Millennium is a large, complex suite of applications with many inter-relationships and impacts that are not easily visible. Millennium requires DNS servers, networks, Citrix servers, Windows servers, Oracle, SQL Server (both for Citrix and some applications), MySQL, Oracle with raw logical volumes, Oracle with the Veritas File System, Oracle with ASM, AIX, HP-UX, VMS, thin clients, WebSphere application servers, WebSphereMQ, Apache web servers, load balancers, Java, Win32, ActiveX components. The list goes on and on. Plus, the Millennium environment is never static. There is always another project, another service package to install, another site to bring into production. Every change in the environment means changes in Millennium and affects how stable Millennium is and how well it performs.

All of these variables should help keep us humble. When I was in consulting and managing Millennium directly, I knew that if a client followed all of my recommendations, we could typically expect a 90- to 180-day period of stability. After that, change would happen. Ultimately, the earlier tuning would help us adapt more easily to these changes, but there would always be a pain period after a change in Millennium. The tuning was never done.

I work very hard to understand Millennium and share this knowledge with Millennium clients. In the process, I learn from each client I interact with. I’ve recently seen two extremes around this point: One organization benefited from this collaborative approach and from having a little doubt that they had done everything possible to tune Millennium, and another was certain they had done all the tuning they could.

The first client said that they just had the “normal” Millennium performance issues but asked me to evaluate the system and suggest ways to improve it. The Messaging Performance Audit revealed several straightforward, easy-to-resolve issues as well as others that required new Millennium front-end code. Here was a person willing to admit some doubt that his team was doing all it could to improve the hospital’s electronic medical record system. After implementing some of the recommendations, they saw quantifiable benefits.

On the flip side, a second client was experiencing significant performance issues in production after introducing key changes to their environment. The system basically came to a crawl every day from 9-11 a.m. and 1-3 p.m. Analysis revealed disk I/O issues. They had not implemented the baseline Cerner Millennium System Settings (CMSS). They had not tuned the operating system’s disk buffering memory regions. Response times from their internal SCSI drives were taking 20 milliseconds instead of three to eight. The Oracle database showed some large waits for disk writes to complete.

When I presented my findings and recommendations, the lead technical person dismissed them. There was no doubt in his mind that his team had done all they could to address the problems and that I was completely wrong. He was beyond adamant. His team continued to address the escalating problem simply by logging Millennium Service Requests until clinicians revolted. Now the client is paying Cerner and other outside consultants to come in and save the day. No doubt proper tuning would have cost a whole lot less and, more importantly, would not have impacted the clinicians’ ability to deliver patient care.

Prognosis: Being willing to doubt we have done everything possible to make Millennium as fast as it can be opens us to new ways of providing clinicians a more reliable, consistent EMR system.

From Clueless to Queueless

By David Hallberg

I have been working with Millennium® in various capacities since November 1997. Year after year I hear the following statement, and each time it stuns me: “We will always have a backlog. There is nothing we can do to prevent queuing.” I have heard these words from the engineering team lead up to the vice president on the vendor side and from the first-level IT person through the CIO on the client side — including two clients in the past few weeks. I keep shaking my head when I replay the conversations.

So let me be very, very clear: There is no baseline with Millennium that includes transaction backlogs or queues. You should never tell the clinicians or business staff using Millennium — or yourself, for that matter — that “waiting is normal” or “well, that’s just the way the system is.” If queuing truly is the norm for your system, it’s telling you it needs a serious tune-up. I have gathered and analyzed performance data from more than 100 Millennium clients, and I can assure you that I always find ways to reduce a user’s waiting time.

I understand that getting to some of the performance information needed to make the right tuning decisions can be difficult. It’s easiest with the right monitoring tools (such as Softek Panther), but with a bit more effort, you also can use Millennium’s mon_ss and quemon along with Oracle, Citrix and network monitoring tools to uncover the data.

Every layer of the n-tier cient/server architecture was built to be tuned, which leads many people to conclude that tuning is too complex and yields too little benefit — hence their belief that queuing is inevitable. They are right on one point: Tuning can be complex. With the mix of Java, Visual C, Visual Basic, Dcom, ActiveX, C++ and C code on each tier, you will not find a silver bullet. Changing a single setting on any of the tiers will rarely increase the speed of the entire system. Invariably, you have to follow a complete process of data gathering, data analysis, action-plan development, execution and post-change data gathering. Executing one change at a time because someone thinks it reduces risk or hopes it will be that silver bullet is counterproductive to your organization.

The only viable alternative is to tune all the areas that impact Millennium performance. I realize it’s a daunting task. Besides the code types I mentioned earlier, Millennium has many technologies as well — the client PC or thin station, file shares, Citrix or Terminal Server, the main application tier (where the SCP servers run), the database tier, Chart, RRD, Multum, WAS (Choose and Book, XR Reporting, Powerchart Outreach), Linux, Oracle with raw LVs, Oracle with regular filesystem for CAMM, MySql, SQLServer, VMS, AIX, HP-UX, GoldenGate, SonicESB (MDBus), backups, network, storage. And that’s not even everything. So how do you know what to do?

Start by gathering as much information as you can. Then you will know what you know and can begin to take action. Typically, you will at least have nmon for AIX and Linux, T4 for VMS, and Glance or other Unix tools/data for HP-UX. Ask yourself the following questions: Do I have an Ethernet port I am not using? Do I have SCSI cards I am not using? Do I have fiber channel cards I am not using? Do I have memory or CPUs on the lpar/vpar/npar that are not activated or simply not allocated? I’m always amazed when clients with performance issues claim, “There is nothing we can do,” yet they have unused hardware sitting on their data center floor.

If you are being told you have to live with backlogs, then get another opinion. If you believe you have tuned all you can and still are waiting, find someone who has a history of tuning large, complex systems for performance and throughput, and see if they agree. If you are being told you need to buy more hardware to fix your performance problems, get another opinion. The majority of tuning can be done without buying anything new. It only requires the proper sequence of keystrokes.

Prognosis: Once IT managers, CTOs, CIOs and CFOs stop embracing the common belief that queuing is an inevitable part of the Millennium environment, they can begin to address the tuning steps that will reduce user waiting times and improve the patient experience.

 

Remedies for RHOs — Part 6

The OS Path for Rapid Transactions

By David Hallberg

The first five installments in our series on Millennium^® performance issues for RHOs walked you through the necessary steps for evaluating connectivity issues as well as queuing problems involving Websphere MQ, shared service proxies, interfaces and Oracle. This week I’ll finish up with yet another area where queuing or waits can be a problem: the operating system (OS).

The OS runs the hardware for Millennium and is often referred to as the backend or backend servers. OS capacity issues can cause Millennium to perform slowly or, worse, sporadically. A sporadic manner could mean that at 7 a.m. Millennium is working fine, but from 9-11:30 a.m. it’s slow.

Before detailing helpful OS diagnostics, I want to answer the larger question, “Why do I need to bother with all this diagnosis? I’ve got an outsourcer to troubleshoot for me.” Experience has taught me that any time you outsource or out-task work, you should audit it. That way, if your clinicians and business staff are complaining about performance, you will have the information you need to have a data-driven conversation with your outsourcer.

The generic complaint of “Millennium is too slow” will generally be met with deaf ears. A comment such as “I have a Run Queue on all application nodes exceeding 4x my physical CPUs” is something your outsourcer can track down and help you correct. So I urge you to maintain technical knowledge in-house or find a trusted consultant who can assist in gathering and interpreting key data concerning your Millennium operations.

So what can you do to determine if your application nodes have performance or capacity issues? Click here to read more.

Prognosis: Monitoring OS capacity and taking action to reduce or eliminate issues as soon as they start — instead of days, weeks or months later — will improve Millennium's consistency.

The Ghost of Standards Past

By David Hallberg

As Halloween approaches, I’ll look at a second tuning standard for AIX that has taken on the status of an urban legend. I stumbled on both last week during a client visit. Like last week’s phantom belief that you need to have AIX disk I/O pacing enabled to make Oracle run faster, this week’s myth involves Millennium® users running on IBM’s AIX operating system. Unlike last week, however, today’s myth can be traced directly back to me.

While working at Cerner in 1997, I stumbled on a tuning parameter Millennium was not using: the logical volume manager buffer count, or lvm_bufcnt. The parameter was designed to provide greater I/O throughput on raw devices, which for Millennium deployments applied to the nodes where the Oracle database was running. I began a campaign to get this parameter tuned.

During this time, Millennium ran with both the application and database on the same node and was only a 32-bit system. The 32-bit architecture severely limited the amount of memory on these nodes, yet lvm_bufcnt needed memory to buffer disk reads and writes. As a compromise, we determined that setting the lvm_bufcnt to 20 would provide better Oracle throughput without consuming too much memory.

Now that Millennium is a 64-bit system and its application nodes and database nodes are split, the tuning standard for this parameter is very different. But, at least for the client I visited last week, the setting has remained the same. What should the new parameters be?

· For the application nodes, the setting should be 9, which is the default. Because there are no raw logical volumes (LVs) mounted on the application nodes, the higher setting of 20 is simply a waste of memory.

· For the Oracle nodes, the setting should be 64, which is the maximum value. These databases benefit from having the highest amount of memory buffers for performing reads and writes to the raw LVs.

Many urban myths have a kernel of truth in them. In this case, the truth that existed years ago under very different operational conditions still haunts us. It seems like a perfect time for an exorcism.

Prognosis: Millennium users running on AIX can set the LVM buffer count on the application and database nodes for greater I/O throughput.

 

Past Standards Still Haunt Us

By David Hallberg

A client visit earlier this week brought my attention to tuning standards established years ago for AIX nodes that are still being circulated. Like urban legends, the standards no longer have evidence to support them, but they live on. October seems like a good time to debunk these phantom beliefs, so I’ll be putting my next installment on issues facing RHO clients on hold for a bit longer.

The legend I’ll address this week claims that you need to have AIX disk I/O pacing enabled to make Oracle run faster.

When Millennium® was first created, AIX (IBM’s Unix) was only 32-bit, and Millennium was deployed with the application node and the database node existing on the same node or lpar. Back then, like today, the various AIX nodes had to be able to share resources for failover. For instance, if you had two Millennium application nodes running WebsphereMQ (MQ) and one node crashed, you needed them to share the log and data file systems for MQ. Why? If node 1 crashed, the other node would be able to complete its transactions.

Way back in the 1990s, Serial Storage Architecture (SSA) drives and HACMP (High Availability Cluster Multiprocessing) were used to share data between nodes. One artifact of this technology was that AIX could send more disk I/O messages to the shared SSA drives than the drives could handle. To minimize this risk, the minpout and maxpout throttles on disk I/O were used to control the disk I/O rate. Maxpout set a limit on the number of pending I/Os (typically 33) for any logical volume (LV) or file. The minpout was set lower (typically from 20-24) and would allow the drive to read and write to the file again once the number of messages fell back to that level.

This disk I/O pacing was a workable solution for SSA drives running on a 32-bit system, but it does not translate to today’s AIX client using storage area networks (SANs) on a 64-bit system. The pacing creates an artificial bottleneck and prevents you from getting the most out of your disk subsystem. It should be disabled.

Unfortunately, the disk I/O pacing caused so many issues when it was not set, the myth persists that you need to keep it set. If you are using SAN storage — as most of you are — this simply is not true and never has been true. SAN fabrics have other pacing mechanisms you should use instead of the AIX OS-level disk pacing to achieve optimal throughput. To disable the disk I/O pacing, you can run the following two commands as root on all of your AIX nodes (please note that these changes take effect immediately):

chdev -l sys0 -a minpout=0

chdev -l sys0 -a maxpout=0

Prognosis: To advance toward optimal throughput, discard the outdated tuning standards of AIX disk I/O pacing in favor of SAN pacing mechanisms.

Next week: Illogical tuning of the logical volumes.

From Triage to Treatment: Configure a Dedicated SSP

By David Hallberg

We all know the top three priorities realtors repeat to new home buyers: “location, location, location.” Those of you working with your hospital’s IT infrastructure might similarly be able to sum up the top three desires of clinicians and administrators with one critical word: “speed, speed, speed.”

Last week I began our triage into systems operating “slow, slower, slowest.” We examined Millennium to identify the services that are backing up and causing the greatest slowdown for clinical and business users. Then I outlined the first treatment of opening up more instances to alleviate the backlog on problem servers. Unfortunately, this basic treatment will not cure every backlogged queue. So this week we will need a more radical intervention. We will examine the Shared Service Proxies (SSPs) associated with the backlogged instances and the creation process for dedicated SSPs.

The SSP is Millennium’s proprietary middleware. It’s the pipe that connects the PC or Citrix servers to Millennium’s back end. By increasing the middleware throughput, you will eliminate your current bottleneck and take users in the production domain one step closer to consistently reliable Millennium responsiveness. Best of all, you can make these corrections through your normal change control processes without any third-party support.

To continue to the step-by-step details for determining the SSP associated with your slowdown and creating a dedicated SSP, click here.

Triage 1: Diagnose System Slowdowns

By David Hallberg

During my years working with and managing hospital IT infrastructure, the three most common calls to the help desk were: “The system is slow,” “Millennium is slow” and “Millennium is too slow to use.”  I’m guessing these are quite possibly the hue and cry for your Millennium installations as well. And the clamor is often accompanied by a director, vice president, nurse manager or chief resident rushing into your work area and demanding you to tell them what you’re doing about it.

In crises like these, it’s important to keep your cool and conduct your triage. I use the word triage because you have to determine if there is an actual problem and, if there is, who will get treated first. What can you do to make the environment fast enough to be usable? System performance diagnosis is a large topic and one I will be addressing in detail over the next two weeks.

Today I’ll take Millennium users through the steps necessary to identify which services are backing up and causing the greatest suffering among clinical and business users. Softek Panther users will be able to view Millennium’s service queues and interface operations through a simple click-and-drag process of setting up a monitoring screen on their desktop. Non-Panther users will need to use Millennium point tools to gain access to their information.

After learning how to view and interpret Millennium’s vital signs, you’ll be able to begin treatment, which in many cases is as easy as starting more instances on the problem server. This first line of treatment might be all that’s needed to produce the reliable system responsiveness that the clinicians and administrators breathing down your neck are demanding.

To continue to the step-by-step details for diagnosing and treating Millennium slowdowns, click here.