Millennium Medic

An Abnormal Definition of Normal

By David Hallberg

Thanks to all of you who stopped by last week during the Cerner Health Conference. We had some great talks about your Millennium^® performance expectations, although I was discouraged to find that for a number of you your expectations were exceedingly low. For example:

• Some of you claimed, “You cannot fix the performance of the application.”
• Others told me, “We are remote hosted and have been assured that we get the best performance that anyone can. When the applications are slow, that’s just the way it is.”
• One client showed me their Lights On Network data for the number of transactions that took over 9 seconds, and I almost fell off my chair. Though the percentage was low, the actual number of transactions was enormous. “That’s way too many,” I said. “It’s the best we can get,” he replied.
• Still another client was told they needed to spend more than $1 million in hardware upgrades to fix their problem.

If you’ve been coaxed into believing any of these statements, let me tell you what I told them: Slow performance is not normal. OK performance is not OK. Don’t believe that you are the fastest Millennium client and there is nothing more for you to do. Your clinicians should not have to wait for 9 seconds for a transaction to go through. Many problems can be fixed without any additional hardware. I know. I have seen Millennium perform extremely well over the past 14 years.

When you are equipped with just a little knowledge of where you have bottlenecks in your system and take the appropriate corrective action, you can see immediate improvements. These improvements do not need to take years and do not require a new version of code, a new operating system or new version of Oracle. You can improve what you have. In the case of the client told to spend $1 million on hardware, we were able to work with the existing hardware and make a couple of uptime changes. Lights On showed that transaction speed for the clinicians had increased and variability had decreased.

Readers of this blog know that I regularly encourage you to trust your clinicians and believe your own eyes rather than blindly accepting performance excuses from the company that wrote the software and is now managing the environment. If the responsiveness of the system is slow, you should be able to see the first level of issues immediately. In a matter of hours or a few days — not months or years — you can determine the initial action steps.

I need to emphasize the words “initial steps.” There is no single silver bullet that will slay your performance dragon; you’ll need a magazine full of lead bullets. The beast is immune to silver but is susceptible to a lead overdose, to an ongoing maintenance regimen that accounts for your continually evolving EMR system. Every new package or service pack you install has the potential to change the processing of the entire system. Make sure you have the right people using the right tools to evaluate what is happening, to identify performance issues and to take action to correct the problems.

Prognosis: If your current support structure is not helping you help your clinicians deliver care more effectively, it’s time to find one that will.

Improving Interface Failovers

By David Hallberg

Today I'll tackle a topic new to this blog: successful failover. Your organization, like many that I talk with, might have brought in a very highly paid consultant to set up failover for your Cerner Millennium® system. Yet, when you need to fail one application node to another to perform maintenance, the interfaces don't fail over correctly. Let me give you a few ideas that might help — and other than a change in your thinking, my ideas won't cost you a thing.

Cerner has extended some functionality in Open Engine's (OEN) Oracle tables that makes failover less painful for its clients and their Millennium support team. Users can set up the system so that the base Millennium failover scripts automatically start the interfaces on the failover node. (In the OEN_PERSONALITY table, where the column Name is PRODUCTION, the flag would be set to Y.)

I know I'm in danger of losing the OEN team here, but keep breathing and hear me out. Typically, the OEN team believes that nothing should be automatically failed over because you could have synchronization issues when more than one registration system sends Millennium ADT transactions. Even single ADT systems, where other systems send Millennium orders, order statuses, transcriptions, etc., can have synchronization issues. These issues can result in an order coming through before the patient has been added to Millennium. I don't dispute your concern. However, I absolutely do not agree that none of the interface servers can be automatically failed over.  

ESI Servers and Outbound Interfaces

Two types of servers, in particular, can be set to fail over automatically with no synchronization risk for incoming interfaces. They are:

1. The ESI servers, or ESI COMM servers. These are not SCP Entry ID 250 and its clones but the servers that take the Inbound client transactions and route them to SCP Entry ID 250 (CPM ESI) and its clones as well as SCP Entry ID 335 (PM Trans Async).
2. The Outbound interfaces, those that send transactions to systems outside of Millennium.

Finding these interface servers can be tricky — Millennium OpenView and Olympus make you go through several screens and tabs — so I suggest you start an interactive CCL session and run the following query to find the ESI COMM servers:

Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PROTOCOL' and op.value ='ESI')
order by pi.SCP_EID asc
GO

The resulting five columns will show you the ESI COMM servers, which can all have their PRODUCTION flag safely set to Y. They will only process transactions after you have started the Inbound COMM server that feeds these servers' transactions. That gives you one less set of servers that have to be started for the Inbound interfaces. I would suggest that the primary ADT Inbound interface be enabled for automatic failover. If during a controlled outage, you do not want the primary ADT interface to be up and running first after the failover, you can easily change the PRODUCTION flag back to N.

The interfaces configured to send transactions to systems outside of Millennium can be found by running this CCL command:

Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PACKESO' and op.value > '0')
order by pi.SCP_EID asc
GO

As with the ESI COMM servers, you can safely set the PRODUCTION flag to Y. Now as your clinicians are doing their normal work, the transactions configured to leave Millennium will not get hung up in Millennium for any length of time.

You might want to take one extra safety measure if you have Outbound pharmacy orders. SCP Entry ID 625 (Medication Dispense and Dose Server) and ID 626 (Medication Fill/Report Server) have a property called FsiCqmDowntimePref that can be added and configured. Set this property to 1 so that Outbound pharmacy transactions can be recovered if SCP Entry ID 252, the FSI ESO-CQM server, is down for any reason.

Additional Servers for Failover Node

Another factor that can impede interface failovers is if any of the following servers are not already running on the failover node:

• SCP Entry ID 250, the CPM ESI servers, along with any clones or copies. If no interface server is running on the failover node, then nothing should be sending these servers any transactions.
• SCP Entry ID 107, ORM Order Write Synch for FSI.
• SCP Entry ID 252, the FSI ESO-CQM server.
• SCP Entry ID 205, the Clinical Event server for ESI, and any of its clones or copies.

The cerner.vars file should be reviewed for the single-instance servers. The file should include at least the following servers: SCP Entry ID 240, 243, 245, 255, 270, 275, 276, 277, 280, 281, 282, 283, 284, 285, 291 and 292. I excluded MDI servers from this list because I am only reviewing interfaces and they involve a great number of variables, which I'll talk about later.

Keeping TCP/IP Ports Free

The last area I want to cover involves the TCP/IP ports. Interfaces will not fail over properly if the TCP/IP ports that you have configured the interfaces to communicate with are in use. How can the ports be in use? Through the normal cycling of servers, ports that initially were free can become used over time. You can prevent this by accessing the file named "/etc/services" and adding the incoming ports for the interfaces and even the MDI servers. For example, telnet is a service that runs on a port, normally 23. So the entry for this would look like:

telnet                23/tcp

If you want to reserve your Inbound interface and MDI ports, all you have to do is add the ports and a name to this file. I suggest you use the SCP Description (minus spaces) for the name along with the SCP Entry ID or OEN PID for clarity, as shown below:

telnet                23/tcp
ADTin                10046/tcp
ORUin               11001/tcp

Once added, these ports will not be used for anything else.

To create a list of the TCP ports you need to put in the "/etc/services" file, use the following command in CCL:

Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PORT' and op.value > '0')
order by pi.SCP_EID asc
GO

Dealing with MDIs

For MDIs, you have to run the command enumlog –all on the two nodes and verify that all of the MDI configurations exist on both nodes and match exactly on both nodes. Because typographical errors are extremely painful to find, I recommend you output the data to Excel and use the EXACT function to compare cells. It will quickly find any typographical errors.

If you want the MDIs to fail over automatically, you can use the following command for CCL to find them and then use OEN_PROC_VIEW GO to modify their PRODUCTION flag to Y:

Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PROTOCOL' and op.value ='CUSTOM')
order by pi.SCP_EID asc
GO

I did not include the MDI servers in the cerner.vars list because too many variables exist for these. Which MDI servers need to be in the file and not able to run on the failover node depend on the code level you are on, the Millennium Distribution Packages you have taken and your volumes. If you have MDIs that are not in the new CareAware or iBus framework, log a Service Request to verify if your code level can have the MDI servers running on multiple application nodes or if you can only run a single instance on a single node. Review the server guide for the MDI servers, looking for the section titled "Considerations for Multiple-Node and Multiple-Instance Usage." This will let you know if you can have more than a single instance and if you can run the server on multiple nodes. I want to emphasize that if you have an MDI server, like SCP Entry ID 230 (MDI Genlab Result Client) and you can run multiple instances on multiple nodes, then on the failover node the server will take very little CPU or memory after it is started. It will idle until the failover event.

Prognosis: A little preparation work can dramatically ease interface and MDI failovers and make your night, weekend and holiday events much shorter and quieter.

Setting Proper LogLevels Is as Easy as 0-1-2-3-4

By David Hallberg

I am often asked about proper log level settings (LogLevel) for Millennium^® servers. The setting determines what types of messages are written into a server’s message log files, which provide critical information to support teams for correcting application problems and some technical issues. If you set the LogLevel too low, you might miss important information. If you set it too high, you might be flooded with extraneous messages (and overlook the ones you need to address).

LogLevel can be set to one of the following values:

• 0 = Error (ERROR on Java servers)
• 1 = Warning (WARN on Java)
• 2 = Audit (AUDIT on Java)
• 3 = Information (INFORM on Java)
• 4 = Debug (DEBUG on Java)

The settings are cumulative, meaning that servers will write messages from that level as well as all lower levels. For example, a 2 setting would include all Audit (2), Warning (1) and Error (0) messages. You can view LogLevels and change their settings in OnTrack Panther™, SCPView or Olympus. Any time you change a LogLevel you need to cycle the server in order for the change to take effect.

In general, the lower the number, the greater the severity of the messages sent to the log files. Unfortunately, many Audit events are actually failed messages or transactions, which is one reason why a too-low LogLevel might result in lost information…and why many people have questions about what the settings should be. That wasn’t always the case.

In 1997, when most Millennium clients were running on VMS, disk saturation concerns meant that LogLevel was always set to 0/ERROR. When you had two or more application nodes with hundreds of servers pumping log information on the same disk drive, the drive — and everything else — would slow down. AIX, with its SSA drives, had problems as well. The drives shared the same connections as the database. If the executables were using up the connection from the CPU to the disk, then the Oracle database could not get the disk time it needed to operate as well as it could.

Disk saturation is not as much of an issue now that most clients are on 2007.02 or greater, many VMS clients have migrated to some flavor of Unix and the AIX clients are off of SSA drives. Yet logging can still be a problem, even when you are running on a well-tuned storage area network. Today, logging usually manifests itself as a specific queuing problem on a specific executable. If you set servers to LogLevel 0/ERROR, you might miss information that is important to use in finding and correcting problems in your environments. I recommend you no longer use this setting.

I also recommend that no server be set to LogLevel 3/INFORM or 4/DEBUG unless you have an active point asking for debug information. Short of that, these settings cause too many disk writes. I was working with a client recently who had the EKS servers in debug, and the message logs were wrapping in less than 15 minutes. This much logging slowed down all of the EKS rules processing but provided no benefit to the client or its Millennium support team.

So what should you set your LogLevels to?

Monitoring servers — such as 27, 29, 33, 66, 76, 97 and 98 — should be at least a LogLevel 2/AUDIT. The Audit level provides a great deal of useful information when you are working through issues or looking for problems before they actually become outages.

I recommend a LogLevel of 1/WARN for the servers listed below. As a general rule, if your executable is not listed, I recommend you set the LogLevel to 2/AUDIT. You might find you need to make some specific tuning or tweaking to these settings, but they provide a great starting point.

SCP Entry ID		SCP Description
72		CPM Notify Server
73		Time Repository Server
75		User Services
80		CPM Script Cache
86		CPM Expedite
92		OMF PowerVision Document Server
127		System Print Server (Node Specific)
130		PathNet Resource Routing Server
131		SCS Netting Server
132		SCS Collections Server
133		PathNet Result Order Posting Server
134		PathNet Sync Result Order Posting Server
134		PathNet Synchronous Result Order Posting
140		Pathnet - AP Case Retrieval Server
141		Pathnet - AP Automatic Diagnostic Coder
144		Pathnet - AP Coding Cache Manager
205		Clinical Event Server For ESI
209		Clinical Event Server - Batch
221		Search Server
231		MDI Result Server
235		MDI Order Server
237		BMDI Result Server
239		DI Inbound Client
252		FSI ESO-CQM Server
275		FSI ESO Notify Server
280		FSI ESO Server Load #1
281		FSI ESO Server Load #2
282		FSI ESO Server Load #3
283		FSI ESO Server Load #4
284		FSI ESO Server Load #5
285		FSI ESO Server Batch #1
286		FSI ESO Server Batch #2
290		FSI ESO Query Extract Server
291		FSI Batch CQM Server
335		PM Trans Async
340		PM Query
369		Chart Coding Services
375		KIA Event Server
425		FRN Tracking View Server
455		Health Expectation Server

Prognosis: Keeping your LogLevels properly set will ensure that your IT team is notified of the issues it needs to address without being overwhelmed with meaningless messages.

Why Auditing Your EMR Is Essential

By Chris Lanaman

I want to expand on the comments I made in my last blog on the need for third-party auditing of your EMR system. I’m including an excerpt from the white paper “Why Auditing Your EMR Is Essential” that our Softek team wrote and distributed last week at HIMSS. The message struck a chord with the people we talked with — both self- and remote-hosted organizations— so I’m confident it will resound with you as well.  

When patients are admitted to your hospital, they count on reliable, competent care:  caregivers who are board certified, care delivery that has passed the standard of the Joint Commission, lab procedures certified for quality by the College of American Pathologists, certified dieticians who oversee patients’ dietary needs and ethical business and financial practices. In each of these areas, facilities and individuals submit to regular, independent evaluation to make sure their practices follow generally accepted industry standards and, when necessary, implement changes to bring them back into compliance.

One area overlooked in these evaluations is the operation of the EMR system itself. With more and more functionality demanded from these systems, they continue to grow in complexity. Yet no independent evaluation program is required to ensure that the EMR is performing as expected. And if it’s not, the consequences could be devastating for patients, caregivers and the facility itself.

Think of the changes you continually introduce to your configuration. Besides the customizing you do for your own unique uses, vendors are supplying a steady stream of system updates and upgrades. A technology upgrade can create myriad downstream impacts on how the EMR will behave. Every time you add a new layer, a new process, a new handoff point, you increase the likelihood of delivery failure. Each failure poses the risk of losing patient information or financial data or extending wait times for clinicians trying to input or access that information.

If your EMR system is hosted outside your own organization, you might think independent auditing doesn’t apply. After all, one of the primary reasons you sought out a knowledgeable vendor was to oversee system performance. You wanted the assurance that comes from having competent partners hosting your system. Yet how do you know that the hosting partner is catching all the performance problems that arise? Because so many performance and stability issues arise from errors with application settings rather than from technical infrastructure, who is really responsible?

Your hospital IT team knows when performance problems develop — you deal with the complaints. With the visibility that comes through a system evaluation, you can more confidently partner with your service provider to not only address system issues but also set higher standards for performance and quality of patient care.

As healthcare focuses more and more on evidence-based medicine, the IT delivery vehicle for that medicine should be evaluated in the same way: using real data from your own system, gathered by an organization independent from the vendor or your own organization, and clearly presented so that you can understand where you are and what adjustments need to be done to accomplish your goal of optimal healthcare.

Prognosis: You can learn even more about how to get an evaluation of your EMR by reading our white paper, “Why Auditing Your EMR Is Essential.” Click here to receive it.

Learn about EMR Auditing at HIMSS

By Chris Lanaman

I’m heading off to the HIMSS11 conference in a couple days to mingle with some of the nearly 30,000 healthcare IT professionals descending on the Orlando Convention Center. While some might get sidetracked by Mickey and his friends, you’ll find me every day of the show with my Softek colleagues in booth 3221. If you’re going to be there, I’d love for you to stop by and meet with us.

I think our message at this year’s show is worth hearing: Every hospital would profit from an independent audit of their EMR system. Just as their care delivery must meet certain objective standards set by the Joint Commission and their financial operations undergo regular auditing, hospitals can use EMR auditing to see if their system is efficiently handling transactions and properly interfacing with its myriad subsystems. Where EMR performance is lagging, adjustments can be made — and patient care and safety improved.

I found an educational session that will make the same case. Dr. Dean Sittig, a professor at the University of Texas Health Science Center is speaking Tuesday morning on “Inspecting EHRs to Ensure Safe Use” in which he will “describe how to implement an annual EHR inspection program … to ensure that their clinical systems are functioning as designed and are being used safely and effectively.” 

Since joining Softek last fall as Director of Millennium Consulting Services, I’ve been involved with a number of such evaluations. Many clients have been surprised at our findings. We commonly find lost charges, queuing backlogs, bottlenecks throughout the system and even clinical information that fails to make it to the patient record. Following our report and recommendations, one client reduced wait times across the organization by hundreds of hours a month. That makes for a lot of happy clinicians and support staff.

Prognosis: A peak-performing EMR system is vital for reaching your goal of optimal healthcare delivery.

MQ Maintenance for Millennium

By David Hallberg

I am frequently asked about maintenance of the Cerner Millennium® system’s MQ queues, specifically what to do with the Admin.Reply, Audit.Exception, Exception and Z_Exception queues. It’s an excellent question, and the answer is not as clear-cut as you might believe. Each of these queues has different reasons for filling up or keeping transactions and different impacts if they do fill up.

Let’s look at the purpose of each of these queues.

The Admin.Reply queue is used to ensure that appropriate communication lines exist for a server starting a transaction. Millennium executables check with MQ to see if the needed queues exist on the node. So, for example, if an SSREQ/SSREP or RDM queue is needed, the Admin.Reply queue either finds the needed queues or creates them. At that point, the transaction is supposed to be deleted from the queue. The problem with this queue is that various Millennium versions and executables do not always delete the transaction correctly and the queue fills up. Once the queue is full, no new Millennium servers can be started from the Server Control Panel (SCP), command line, Panther or Olympus. So any automated cycle-server jobs as well as any interactive server cycling will fail.

This problem can very, very quickly cause significant performance and stability issues in your domain. I helped one client whose Admin.Reply queue, which had a Max_Depth of 15,000 rows, filled up in less than 10 minutes. The client cycled servers between midnight and 7 a.m., but because this queue was full, all of the cycle jobs failed and caused performance issues.

So how do you prevent a similar scenario? By the time you see a transaction in this queue, the work the transaction was doing is complete. These transactions do not contain any clinical or financial information, so it’s safe to delete them. I recommend that you check this queue on each application node at least once a week and delete any transactions you find.

The Audit.Exception queue is connected with the queues CPMSRVAUDIT, CPMSRVAUDITBATCH, CPMSRVAUDITOUT, SSREP.SRVAUDITMGR and SSREQ.SRVAUDITMGR. It is designed to capture transactions that could not be sent to P2Sentinel, which can happen for any variety of reasons. (Be aware that P2Sentinel is the third-party product used to capture information and could cause HIPAA violations or show inappropriate use of Millennium by your organization’s staff.)

Begin your troubleshooting of this queue by verifying that P2Sentinel was up fully and able to take transactions. If P2Sentinel can take transactions, then you should be able to replay these transactions. If you do not replay or recover these transactions, there will be gaps in the auditing information that you need to be aware of. It is possible to have incorrectly formed transactions, which you might have to delete. Before you do, I suggest that you engage your IT management, Security and whoever is tasked with overseeing HIPAA compliance. Your organization might need to complete additional documentation prior to any deletions.

The Exception queue is designed to hold transactions that could not be put on an MQ queue. How does that make any sense? Here’s the basis of MQ: A process or executable has a transaction to hand off to another process or executable, and application programmers want to make certain these transactions make it from one process to another. So the executable that wants to hand off a transaction puts that transaction into a queue. From there, the receiving executable retrieves the transaction and processes it. Still confused? Suppose the transaction is my electric bill. The electric company wants me to get the bill (transaction), which it delivers to my mailbox (MQ queue), where I go and get the bill (transaction). If for some reason the postal carrier cannot put my bill into my mailbox, the bill will go to the infamous dead-letter office (Exception queue). Someone at the central post office will then do additional processing to send me my bill.

In the case of Millennium, the bill is actually a clinical transaction, so you can’t ignore it. Any transactions sent to the Exception queue have clinical or financial data that was supposed to be processed and acted upon but never made it to queue it was originally sent to. It’s important to recover this information, which you generally can do simply by replaying the transaction to the appropriate queue. This is a little like UPS leaving a message on your door stating they need a signature before leaving a package, and since you were not home, they will try again tomorrow. The next day you stay home and get the package.

A replay generally works with Exception queue transactions because the failure reasons are often transient. If a replay does not work, you will want to log an SR to get additional assistance in either processing or deleting the transactions. As with the Audit.Exception queue, Exception queue transactions generally include clinical or financial data, so deleting them could create gaps in patient records. I would suggest that you engage your IT management, Security and whoever is tasked with overseeing HIPAA compliance before deleting these transactions. You might need to complete additional documentation prior to any deletions.

The Z_Exception queue is my favorite. It is used by the Queue Control Panel (QCP) when the front-end QCPView.exe or Quemon.exe or any other Millennium executable cannot communicate with the QCP. Since the QCP does not handle any clinical or financial data, I would suggest deleting the transactions in this queue. If the Z_Exception queue fills up, the QCP will not work normally and you will not be able to use QCPView.exe or Quemon.exe.

Prognosis: A little housekeeping with the Millennium Exception queues will help make your EMR environment more stable.

Determine if Millennium’s Size Is Just Right

By Chris Lanaman

Like Goldilocks trying out chairs at the cottage of the Three Bears, every healthcare organization is looking to land on a system that is sized “just right.” When you first had Cerner Millennium^® installed, smart people thought through what kind of hardware and how much of it would need to be deployed in order to meet your organization’s needs. Most likely they also considered the projected growth of the organization over the next five to 10 years. Buying too much would add complexity for your users, loss of control in managing the system and, of course, wasted cost and time. No one wanted Millennium to be like Papa Bear’s too-big chair.

I would suggest that a bigger problem exists when the Millennium system is not sized big enough for the demands of your organization. Any one of the following symptoms might indicate that your workload has outgrown your system: lower than expected productivity of outcomes, the loss of business revenue because of turning away patients or poor response time for end-users.

I recommend starting with the following five steps if you are concerned that your organization has outgrown its current Millennium size.

1. Data, data and more data. You must have data in order to see if your organization is suffering from a sizing issue. This data is gathered by your team of data-miners (IT analysts) or through an application that is generating reports for your system, such as Cerner’s PowerInsight or PowerVision. Use this information to see if there has been a drop in activity, a workflow slowdown or loss of productivity.
2. Patient management cycle. Once you have obtained measurements for your organization’s workflow, you will need to evaluate where you are in your patient cycle — is this the time of year that you have increased patient traffic or are these your slower months? Your metrics should also include a staffing model to meet patient demand. Understanding your cycle will keep you from over-sizing your system if you evaluate during the high point of the year or under-sizing if you are in a valley part of your cycle. These measurements will identify the revenue cycle expectations for your organization.
3. Mergers and acquisitions. When your organization merges with another healthcare organization, great pains are taken to try and unify the new organization under a common vision. The impact of the merger on the EMR system’s workload, however, might be overlooked. The IT analyst needs visibility into the new organization’s past data to understand how workloads might increase.
4. Market share. Does your organization understand the footprint it has in your market? When trying to understand growth potential, you need an understanding of your opportunities and who the suppliers in the market are. Theoretically, one can say a set percentage of business equals a set amount of hardware needed, but this is an educated guess at best and should never be the rule of thumb.
5. Is your Millennium system healthy? As I alluded to earlier, when your system is not sized correctly, you will see an increase in performance issues. If your end-users are telling you, “The system seems slower than it used to be” or “It seems to take longer for me to do my job in the system,” the first thing on your mind should be the health — i.e., the efficiency — of your system. Before joining the Softek team, I used Softek Panther for a number of years to uncover trouble spots. It let me know when Millennium stepped out of line. Additionally, Softek’s Queuing Performance Spreadsheet (QPS) allows you to see where within Millennium your user transactions are getting clogged, and the Messaging Performance Audit (MPA) gives your system an overall checkup. The audit lets you see if you have build issues in Millennium, if transactions are going where they’re supposed to go, if your servers are configured correctly and when your servers are being over-tasked. If you have other monitoring tools you recommend, please let me know. Whatever you use, it’s important to check your system thoroughly before buying more hardware. Hardware purchases should always be considered the last resort for resolving performance problems. 

In a blog, I only have space to give you the Reader’s Digest Condensed version for scoping your organization’s size and determining how that affects Millennium’s performance. But addressing these key points will help keep your organization healthy and your users happy.

Prognosis: Evaluating all of the different factors that make up your organization is the first step in understanding if your Millennium system is sized “just right.”

Fragmentation’s Biggest Rift? The Generation Gap

By David Hallberg

My last blog covered deleting old files from your Millennium® application nodes. This week I’ll discuss the second step for keeping Millennium's speed consistent over time when you’re tuning filesystems and directories: defragmentation. Just like users disagree over the need to purge unneeded files, they also disagree over the need to defragment files.

I’ve had some interesting conversations over this issue this past year. People weaned on mainframes and older operating systems universally say, "Defragment!" Those who grew up on Linux, HP-UX or AIX virtually to a person say, "Defragmentation is only needed for the old and slower operating systems." With the latter group, I have to remember to stay calm and remind myself that their ideas probably come from the websites and authors who claim that, except with Windows, fragmentation is a thing of the past. But I also remember the documentation from the vendors who wrote the operating systems and the supporting filesystems. Plus I remember my work on the Advanced Technology Group at Cerner and what we benchmarked on AIX, VMS and Windows. Only then do I begin the conversation.

Filesystem and directory fragmentation occurs on all of the operating systems supporting Millennium®: VMS, AIX, HP-UX, Linux and Windows. There are generally two types of fragmentation: file fragmentation and space fragmentation.

In the first type, a file, say dic.dat, becomes too large to fit into a single disk block, so it has to be split into two or more blocks. If the dic.dat file is then modified, it may need a third or fourth block to fit within. It’s likely that code upgrades or custom CCL changes will have added more files to the filesystem where dic.dat is located, and they will have been saved to the blocks right next to the dic.dat's first two blocks. That means the third and fourth blocks for dic.dat will not be able to be written right next to (or contiguous with) the first two blocks. Now the dic.dat file is fragmented. With the file split into two chucks, the operating system has to perform two reads to completely read the dic.dat. When the disk drive or RAID set has to do more reads (disk I/O's), performance suffers.

Space fragmentation occurs when you delete a file, typically because it was a temporary file. Once deleted, the space it originally occupied leaves a gap on the filesystem. Many deleted files create greater fragmentation, making it impossible to write a file contiguously.

The filesystems/directories I discussed in my last post ($CCLSCRATCH, $CCLUSERDIR, $cer_temp, $cer_fifo, $cer_usock, $cer_log, /var, /tmp) are all designed to hold temporary files, such as print files or logging files, that will be deleted as soon as they are no longer needed, e.g., daily or possibly hourly.

The downstream impact of this design is that all current operating systems need to have both types of fragmentation corrected. Vendors provide tools for this defragmentation, including defragfs for AIX, fsadm for HP-UX, and fsadm for CAMM (Millennium's PACS) and the Vxfs filesystem that it uses. VMS has the DFU tool on the freeware CDs that has to be installed before it can be used. Also for VMS, HP has a DFO tool and Diskeeper offers Diskeeper for VMS, which has been around since 1986. You’ll have to pay for these latter two tools, but it might be worth investing in one of them if you’re trying to keep your VMS system. For Windows, you can use the built-in tool called Disk Defragmenter; a third-party tool like Diskeeper 2010 and its various versions or PerfectDisk 11 and its various versions; or one of the plethora of freeware/shareware tools, my favorite being Defraggler from Piriform. I encourage you to spend an hour or so looking through the numerous white papers, blogs and forums discussing these options.

My first team in Cerner's Advanced Technology Group was tasked with successfully completing client conversions no matter what technical hurdles we faced. One of our biggest obstacles was severe fragmentation of both files and free space. We had such significant file and space fragmentation at several sites that we added a drastic step to our normal recommendations. In addition to deleting lots of files daily and defragmenting the filesystem at least once per week, the clients had to set up a schedule to point the filesystem, say $cer_print, to an entirely new filesystem every three months. Miraculously, their systems became much more stable.

Why did it help to move the filesystem? Each filesystem had an index that kept track of where each file started, the size of the file, where it ended, etc. As thousands of files were created and deleted every day, the operating system had to slow down in order to keep track of them. The periodic refresh of the Inode or indexf.sys (on VMS) brought performance back to a better level. With today’s improved drive technologies, this type of severe fragmentation is less likely, but it still needs to be reviewed. Some of you might find it necessary to recreate your filesystem on another set of disks and benefit from having contiguous files and free space.

Prognosis: Using defragmentation tools on a regular basis to reduce filesystem fragmentation on all operating systems will increase Millennium’s speed, consistency and reliability.

Deleting Old Files

By David Hallberg

Today’s topic is an issue over which Millennium  users disagree: whether the number of files in a directory or filesystem impacts the Operating System (OS). Many of you say no, but I believe it does. Having too many files in a directory or filesystem with no scheduled purging of those filesystems will degrade your system’s performance.

Those who state that the number of files has no impact on the OS begin with a great assumption: that Millennium was written by the OS vendor and that it is compiled for the OS version it is running on. Those versions include 64-bit for VMS, AIX and Linux and 48-bit for HP-UX. Unfortunately, the Millennium application is written with the 32-bit compilers for each of these platforms. Under ideal situations, the OS may not care about how many files exist, so long as there is free disk space, but Millennium does. The number of files affects CCL report generation to either your Millennium session or a printer and definitely affects the primary Millennium print servers, such as DCP Output (SCP 106), System Print Server (Node Specific) (SCP 127), Print Services Cache Manager Server (SCP 128) and System Print Server (SCP 129).

Let me give you an example. The SysSrvPrint error message from the servers indicates that the Millennium executable cannot find the file in the filesystem. This event can happen because 1) the file is on another node, 2) the file was purged before it was printed, or 3) it took too long for the executable to look up the file. That third failure reason is easy to prevent. If you delete files from the $cer_print and $CCLSCRATCH directories every four to six hours, you should be in good shape.

You can count how many files are in the $cer_print directory or filesystem using this command (the last character is a lowercase L):

ls $cer_print | wc -l

Likewise, to see a count of the files in the $CCLSCRATCH directory, type the command:

ls $CCLSCRATCH | wc -l

The goal for optimal performance within all Millennium solutions is to have less than 5,000 files in any of these directories at any point in time. Due to the number of facilities and solutions you have and the Millennium version you use, you might not always be able to achieve this level. But I encourage you to keep working towards it.

The crontab job scheduler in the $cer_proc directory comes with a UNIX Korn shell (ksh) named clean_wh_env.ksh that deletes unnecessary files. The ksh will probably have to be tweaked to fit your specific site. This takes a little getting used to if you are unfamiliar with UNIX scripting but will greatly improve Millennium’s speed and reliability. Beginning from the directory count you ran from the commands listed above, identify the directory that contains the greatest amount of files and then modify the clean_wh_env.ksh file either 1) to be run more frequently from crontab or 2) to delete additional file patterns or extensions.

The ksh should also be used to clean up the following additional filesystems: $cer_temp, $cer_log, $cer_usock, $cer_fifo, $cer_lock and $CCLUSERDIR. Access the file count in these directories with the command listed above. For these directories, deletions scheduled to run once a day should be enough.

Prognosis: Using your Operating System’s scheduling service to routinely delete unneeded files from your filesystems will increase Millennium’s speed and reliability.

Help Citrix Handle Unexpected Login Spikes

By David Hallberg

Recently I was working with a client who had experienced an unexpected power outage. Once the power returned, the PCs had to be rebooted. The client uses Citrix to access Millennium, and the rebooting overwhelmed the Citrix Zone Data Collectors. The overload delayed system access for everyone trying to get back into Millennium® — even though the data center stayed up during the outage.

Was the overload inevitable? Or would it have been possible to tune the site’s five-year-old hardware to better recover from or live through abnormal spikes in activity?

These questions are important because many sites use Citrix’s XenApp servers to deploy Millennium. Citrix — how most users refer to all of the various Citrix components — allows a small set of servers to present Millennium applications, which individual PCs then access by connecting to the Citrix servers. The Zone Data Collector (ZDC) is used by Citrix to evenly balance the PC connections across all of the XenApp servers. Without the ZDC, the first Citrix server in the group or zone would get all of the connections, which would quickly overwhelm the server and cause other issues.

As my client discovered, even with the ZDC, abnormal events can overwhelm the server. So I want to suggest some basic tuning steps to help you balance the ZDC loads and ward off future overloads. We’ll discuss the data store (often SQL Server) tuning and registry changes that can make the data store and ZDCs able to handle sudden increases in volume.

I would suggest you review your Citrix configuration even if it was set up by an “expert.” Often the experts are skilled in following a specific set of requirements to set up an environment or domain and show you how to do it next time. Then they leave. The more knowledge you have about your environment and its configuration, the better you are positioned to address potential issues. Ask yourself, “If I had to accommodate a doubling of connections or clients in less than 60 minutes, where would I have problems?” After reading through the steps I recommend, you can create a list of what tuning needs to be done and start the change control process to put these changes into production. That way, the next time you have a power outage at your organization, the recovery time will be shorter and less painful for you and your Millennium users.

For details on the Citrix tuning process, click here.

Prognosis: By moving your data store to an optimized disk partition and tweaking some registry settings, you will find that Citrix can become much more responsive, even when the workload more than doubles.

If You Rewrite the Rules…

By David Hallberg

How I love rules. By following certain clear principles, you can be reasonably sure of the outcomes. If you wash your hands regularly, then you’ll spread fewer diseases. If you listen carefully, then you’ll be better able to address a person’s concerns. In the IT world, it’s called the if-then-else logic loop. But I’m the first to admit that rules have limitations. I’ll leave you to figure out their limitations in your personal life, but this week I’ll help you determine if some of the rules you have set up in Millennium® are a bit broken.

Millennium has a wonderful subsystem called Discern EKS, which allows users to identify special events that will automatically trigger other events. For instance, users can set a rule that whenever a patient’s white blood cell count is very high, say 15,000 to 30,000, an alert suggesting a blood culture will be generated. Each department determines the rules it needs based on its standard operating procedures, works with the appropriate teams in the organization to set them up and outlines its response when the rules are triggered.

It sounds straightforward enough, and most organizations don’t give much thought to the rules if they are running in less than a second or two. Run time alone, however, does not tell the whole story. Organizations can have hundreds or thousands of rules, some of which fire thousands or tens of thousands of times a day. They can present a huge drag on the Millennium system, and these tuning issues need to be addressed.

For example, if a rule takes one second to run – which would be considered a reasonably fast rule – but is triggered 1,000 times in one day, that one rule is using up 1,000 seconds – or over 16 minutes – of application CPU time. Rules firing more than once per minute – more than 1,440 times a day – need closer scrutiny. I would recommend that you look at that rule’s trigger mechanism and verify it really should be firing that frequently.

Rules that trigger once per minute are not uncommon. I’ve worked with clients that have rules triggering more than once per second – that’s 86,400 times a day. Now consider that many sites have hundreds or thousands of these rules. Even if all of them are so-called fast rules, you can still see where you might have Millennium performance issues.

To determine if you’re having issues with a rule or set of rules, we’ll look at the Oracle table EKS_MODULE_AUDIT. This table shows when rules are triggered and how long they take to fire, along with a lot more useful information. To learn how to access the EKS audit table and analyze its numbers, click here.

Prognosis: An analysis of the time Millennium spends each day running rules will show you whether any rules need to be rewritten to help the system run faster.

Chart Servers — Part 4

Tuning Windows OS

By David Hallberg

This week we move on to our third and final step in tuning your Millennium chart servers: tuning the Windows operating system (OS) on each server. In step 1, you determined which of your chart servers was the fastest and copied that server’s configuration to the other servers. In step 2, you tuned your Oracle database tables. Once you complete this last part of our holistic tuning approach, you can expect the process time of your chart servers to be less than half a second per page.

Tuning of the OS running chart servers should start with Cerner. A document on Cerner Millennium System Settings (CMSS) for Windows is available from Cerner Support. The document has about 15 pages and contains a couple of key items that relate to our topic: 1) Pay attention to the effect that anti-virus scanning software has on the solution, and 2) be wary of the recommendation to run a defragmentation tool at least monthly. I recommend that you defragment the chart servers every day.

Many freeware and shareware solutions are available to defragment your servers. I enjoy using Defraggler from Piriform (www.piriform.com/defraggler). It has a native 64-bit client but runs on a 32-bit OS as well. Diskeeper from www.diskeeper.com is another excellent solution if you have the operational budget for it. For any of you running VMS, Diskeeper has great VMS defragmentation tools as well as I/O analysis tools.

A lot of people are documenting how they tune Windows. Doing a search on Google for “XP performance tweaks” returns 8.5 million results, “2003 performance tweaks” returns 2.6 million, and “2000 performance tweaks” returns nearly 4.7 million. Of the many sites that have excellent information on tuning the OS, I have a couple that I keep going back to:

1.    PC Pitstop (www.pcpitstop.com) will run an audit on your server or PC and show you where you have potential performance and configuration issues. If you want to have the tool do the registry changes for you, there is a fee.

2.    SpeedGuide.net (www.speedguide.net/index.php) has a lot of topics. If you choose the Broadband option at the top left of the page and then select Registry Tweaks, you will find a wealth of information on tuning the Microsoft TCP/IP stack for optimal performance.

3.    Microsoft has excellent documentation at www.microsoft.com. Once there, search for “XP performance tuning” or “2003 performance tuning.” You will have to read through a lot of documents to determine which pieces you want to apply to your OS. That’s why I prefer PC Pitstop and SpeedGuide.net. I have found that by using them, I can quickly get the most from my OS.

My final piece of advice is to turn off the following two services, no matter what version of Windows you’re running:

1.    ComputerBrowser was designed for Windows 95 PCs to act as the WINS server for DOS and Windows PCs. It is very chatty and actually quite a security hole. There is no need for this to run on any chart server.

2.    Themes was designed to make the chart server have a graphically rich (pretty) view of the OS. I want chart servers to be fast, consistent and reliable. I do not need them to be pretty.

Prognosis: With apologies to the ′60s rock band The Byrds, I would encourage you “tune, tune, tune, there is a season, tune, tune, tune, and a time for every purpose….” The season is now and the purpose is to dramatically improve the overall responsiveness of the chart prints for clinicians.

Chart Servers — Part 3

Tuning Oracle’s Database Tables

By David Hallberg

In my first blog a few weeks ago about Millennium’s chart servers, I talked about the need to determine which of your chart servers is the fastest (that is, which server prints each page of your organization’s charts in the shortest period of time) and how to make a fairly simple adjustment to speed up the slower servers. By copying the configuration of the fastest server’s operating system to the other servers, the processing time from our sample site increased an average of more than 1 second per page over its five chart servers. So why don’t I move on to a new topic? Why would we do anything else to the chart servers after getting them to perform in the same manner?

As I’ve touched on in the past, a frequently misunderstood concept for tuning Millennium is that making one change will cure a problem forever (see “A Healthy Dose of Doubt”). A second misunderstanding is that good change control dictates that you only change one thing in Millennium at a time. These sound very good on the surface. After all, how often are we told to start small when making our New Year’s resolutions? If we want to lose weight, we will have better long-term success by choosing one change that we can maintain rather than a bunch of quick fixes.

With Millennium, however, things aren’t that simple. The Oracle schema (a fancy name for the way the data is held in the database) alone has more than 300 tablespaces, 4,400 tables and 11,000 indices. A system can have anywhere from a handful (say, five) of chart servers to more than 20 defined and functional chart servers. Changing one row per chart table per change request window under a system of every-other-week change control means that chart changes could take a year to complete. Why would you want your organization to suffer for that long?

I’m suggesting that you can address most of your chart performance issues in three change windows. I’ve already covered the first — making all of the chart servers have the same baseline Windows and Word image. Today I’ll help you tune some of the Oracle database tables affecting chart performance (click here for the details), and next we’ll look at ways to tune your Windows operating system just a little more.

Prognosis: A comprehensive tuning approach will increase the performance of your chart servers (much to the delight of clinicians and other chart users) and will give you a solid base from which you can monitor for future throughput issues.

Chart Servers — Part 2

Chart Prints Can Raise Reams of Issues

By David Hallberg

In my last blog, I showed how analysis and simple tuning of the Millennium® chart servers at one site saved nearly 7 hours in print times over the course of a week. I gathered those times from Millennium’s Chart Request Table, which also yields information about the size of the charts sent to each server. You might be surprised at what you can learn from those numbers.

Below is the Excel PivotTable we built from a week’s worth of requests sent to a client’s chart servers.

Chart Server Name	Count of SERVER_ NAME	Sum of PROCESS_ TIME	Sum of TOTAL_ PAGES	Pages per Request	Time per Page
DWHMOCHART03	728.00	4,216.97	1,315.00	1.81	3.21
DWHMOCHART04	3,982.00	19,332.78	6,390.00	1.60	3.03
DWHMOCHART05	1,068.00	5,409.00	1,961.00	1.84	2.76
DWHMOCHART06	2,363.00	12,296.50	3,747.00	1.59	3.28
DWHMOCHART07	1,081.00	23,780.52	1,857.00	1.72	12.81
DWHMOCHART08	1,892.00	9,179.29	3,433.00	1.81	2.67
(blank)		0.00	0.00	#DIV/0!	#DIV/0!
Grand Total	11,114.00	74,215.08	18,703.00	1.68	3.97
			2,671.86	<<Pages per Day
			5.34	<<Reams per Day

Let’s look at the Total Pages and Pages per Chart Request columns.

Dividing the number in the Grand Total cell for Total Pages by 7 gives you the total number of pages printed through the chart servers in a day. If you take that number and divide it by 500 (the number of sheets in a ream of paper), you get the number of reams of paper you are printing JUST for chart printouts. The site above is printing more than 5 reams of paper a day. I had one client generating more than 7,000 pages — more than 14 reams of paper — every day.

Most sites I work with are astonished to realize the amount of paper they are using. But it helps explain why nursing units and medical record/health information management departments have seen an increase in operational costs as the use of EMRs expands. Toner is used up more rapidly, printers may be pushed beyond their duty cycle (the maximum pages they can print per month and still be in warranty), or the amount of RAM or memory on the printers may have to be increased.

Knowing paper usage patterns for chart prints may be used for planning and acquisition of properly sized printers for both clinical and business areas. At one Millennium site I worked with, I recommended that all printers tied to Millennium have 128 MB of RAM. For Medical Records/Information Management, I recommended a duty cycle of at least 50,000 pages per month. The site represented in the table above is printing more than 80,000 pages per month. This data helps to explain why inkjet printers generally will not work well in a Millennium environment. Neither will printers with paper trays of less than 250 pages.

Prognosis: It is crucial for areas using Millennium chart printing to understand how much they spend in support of Millennium each fiscal year. Using the Chart Request Table to analyze paper usage for chart printing will help organizations look realistically at printer and paper needs…and budget accordingly.

Next: Tuning Oracle database tables to improve chart server performance.

 

Chart Servers — Part 1

Chart Request Table Dishes up a Feast of Data

By David Hallberg

Today I want to begin a short series on chart servers by introducing you to a table that many Millennium® users are unaware of: the Chart Request Table. This table dishes up a lot of information about how your chart servers are performing — particularly how long it takes the servers to process chart prints — so it’s important to know how to access the information, understand what it’s telling you and correct performance issues.

With all the other issues you need to manage in a day, you might be asking, “Why does it matter how long the chart print takes?” In a nutshell, chart prints are used to treat patients. Printing delays could mean a clinician does not have the latest information on the patient being treated. Waits of more than an hour for a larger chart to print are not uncommon. And a delay in printing one chart slows down all the other chart requests queued up behind it. With Expedite and Ad Hoc chart requests in particular, this lack of information has the potential to compromise patient care and it opens the organization to liability concerns.

I have found an enormous spread among clients on how long it takes to print charts. Some clients can process a page in 1 second, and others take more than 2 minutes to generate a single page. A 2-page chart that requires 2 minutes to process each page stays in the chart server for 4 minutes before being sent to the print server and finally the printer. At this rate, a 50-page chart takes the chart server 100 minutes — or more than an hour and a half — to complete the generation of a single chart request. That same chart, if sent to a well-tuned chart server processing at 1 page a second, would be sent to the print server in less than 1 minute.

I understand that your time, money and staff are limited. But access and analysis of chart server performance details can be done in minutes, and a skilled technician can correct slow servers in a few hours. It starts by accessing the Chart Request Table and exporting the data to a spreadsheet for analysis. Using this information to then reset your slower chart servers to match the fastest one will dramatically improve the clinically perceived speed of Millennium, reduce end-user frustration and quite possibly improve the quality of the delivered care. Click here for the step-by-step process.

And then keep your eye out for upcoming blogs, in which we’ll:

1.    Examine the implication of other pertinent data contained in the Chart Request Table — in particular, the servers’ print quantity.

2.    Dig deeper into Windows tuning techniques of the chart server that will reap even larger rewards for your clinicians.

Prognosis: Knowing which chart server or group of chart servers is fastest and getting all of the chart servers to perform at the same level can improve clinical end-users’ acceptance of Millennium and potentially some patient outcomes.

Oracle Tables Expose Hidden Issues

By David Hallberg

In working with a couple of clients, I discovered a large number of Millennium® Oracle tables that were used for logging or errors (like a message log) — more than 90 in the 2007.18 and above schema. I find this attribute perplexing. If you wanted to put all of the error and logging information in the database, why create the message logs, .err and .out files? On a practical level, that means analysis of application or stability issues requires users not only to gather data from the message logs but also to run a count of these 90+ Oracle tables to determine which tables have a lot of activity and then run a retrieval to get the information from those Oracle tables. No wonder many don’t bother looking for issues proactively — the task is simply too daunting.

Despite the magnitude of the task, I believe it’s important to do all you can to keep Millennium running as error-free as possible. And one way to accomplish this goal is to monitor error messages and logs and resolve the issues they show you. Last month I wrote about how to find and correct quickly wrapping message logs ( “What Is Measured Improves” ). Today I’ll list the queries you can run to see if any of these 90+ Oracle tables are active on your system — if, for instance, they include more than 10 rows of information, or whatever amount your team sets as its threshold. To see the list, click here.

This analysis is related to the previous blog on message logs and will use a similar methodology for reviewing the Oracle tables. You want to view activity in the tables over time. Run the queries I have listed and gather the row counts, and then run the queries again in a day or week or month and see which ones have grown.

Due to the fact that the tables are based on the Millennium solutions you have configured for your own site, I cannot provide specific guidance on what to look for and how to address an issue. I can, however, provide a little insight into a few of the tables:

· CPMPROCESS_ERROR is important to anyone still on 2001.x or earlier code. This is the Oracle table that the CPM Process server and any clones of CPM Process use when they cannot complete a transaction.

· For the CQM_MDRESULTS_LOG, CQM_MICRESULTS_LOG and CQM_OENTXLOG_QUE tables, you want to make certain that all of the rows are only as old as your purge criterion. For instance, if your purge criterion is seven days, then there should not be any records in these tables more than eight days old. If there are, you need to manually purge them and log a service request (SR) with Cerner to get a purge script that works correctly.

· CHART_SERV_LOG is important to monitor to verify that the chart servers are not sending too much logging information to the database.

· The DCP_ERROR_LOG and ORM_ERROR_LOG are for DCP and order errors. They are often in the message logs, but their inclusion here helps consolidate several of these messages from all of the application nodes into a single queriable location.

· If you are a Millennium Profit client, the CSI_LOG and CSI_ERROR_LOG tables are important to monitor for interfaced charges and their errors.

· The GL_TRANS_LOG for general ledger events and PFT_*_LOG tables for general Profit issues should be monitored and issues rectified as quickly as possible to minimize cash flow and reimbursement issues.

As I recommended in the previous blog, your staff will need to log SRs with Cerner to figure out how to correct your errors and then create and execute an action plan.

Prognosis: Maintaining a list of Oracle tables and knowing their rate of change or growth can be time-consuming, but the work you do up front to catch issues early will mitigate problems for the clinicians and support staff in the future.

A Healthy Dose of Doubt

By David Hallberg

 “…and the only people I fear are those who never have doubts.”

– Billy Joel, “Shades of Grey”

One of the most frustrating discoveries I’ve made about Millennium’s technical infrastructure is that tuning is never done. If I think I’ve gotten all of the performance out of Millennium that I can, I need to step back and re-evaluate my metrics…or just wait a little while for other issues to arise.

Another important realization, one that I learned a long time ago, is that my knowledge and experience is not the pinnacle of knowledge and experience. No matter how hard I try I can never know all there is to know on a given topic. There is always someone else who has dealt with a similar performance/stability/capacity issue and knows more about it than I do. When working with clients, I prefer a collaborative approach that comes with a little doubt that the problem solving will ever be done.  Performance is an iterative process that has to be monitored and adjusted through time, not just at a single point in time.

Millennium is a large, complex suite of applications with many inter-relationships and impacts that are not easily visible. Millennium requires DNS servers, networks, Citrix servers, Windows servers, Oracle, SQL Server (both for Citrix and some applications), MySQL, Oracle with raw logical volumes, Oracle with the Veritas File System, Oracle with ASM, AIX, HP-UX, VMS, thin clients, WebSphere application servers, WebSphereMQ, Apache web servers, load balancers, Java, Win32, ActiveX components. The list goes on and on. Plus, the Millennium environment is never static. There is always another project, another service package to install, another site to bring into production. Every change in the environment means changes in Millennium and affects how stable Millennium is and how well it performs.

All of these variables should help keep us humble. When I was in consulting and managing Millennium directly, I knew that if a client followed all of my recommendations, we could typically expect a 90- to 180-day period of stability. After that, change would happen. Ultimately, the earlier tuning would help us adapt more easily to these changes, but there would always be a pain period after a change in Millennium. The tuning was never done.

I work very hard to understand Millennium and share this knowledge with Millennium clients. In the process, I learn from each client I interact with. I’ve recently seen two extremes around this point: One organization benefited from this collaborative approach and from having a little doubt that they had done everything possible to tune Millennium, and another was certain they had done all the tuning they could.

The first client said that they just had the “normal” Millennium performance issues but asked me to evaluate the system and suggest ways to improve it. The Messaging Performance Audit revealed several straightforward, easy-to-resolve issues as well as others that required new Millennium front-end code. Here was a person willing to admit some doubt that his team was doing all it could to improve the hospital’s electronic medical record system. After implementing some of the recommendations, they saw quantifiable benefits.

On the flip side, a second client was experiencing significant performance issues in production after introducing key changes to their environment. The system basically came to a crawl every day from 9-11 a.m. and 1-3 p.m. Analysis revealed disk I/O issues. They had not implemented the baseline Cerner Millennium System Settings (CMSS). They had not tuned the operating system’s disk buffering memory regions. Response times from their internal SCSI drives were taking 20 milliseconds instead of three to eight. The Oracle database showed some large waits for disk writes to complete.

When I presented my findings and recommendations, the lead technical person dismissed them. There was no doubt in his mind that his team had done all they could to address the problems and that I was completely wrong. He was beyond adamant. His team continued to address the escalating problem simply by logging Millennium Service Requests until clinicians revolted. Now the client is paying Cerner and other outside consultants to come in and save the day. No doubt proper tuning would have cost a whole lot less and, more importantly, would not have impacted the clinicians’ ability to deliver patient care.

Prognosis: Being willing to doubt we have done everything possible to make Millennium as fast as it can be opens us to new ways of providing clinicians a more reliable, consistent EMR system.

Past Standards Still Haunt Us

By David Hallberg

A client visit earlier this week brought my attention to tuning standards established years ago for AIX nodes that are still being circulated. Like urban legends, the standards no longer have evidence to support them, but they live on. October seems like a good time to debunk these phantom beliefs, so I’ll be putting my next installment on issues facing RHO clients on hold for a bit longer.

The legend I’ll address this week claims that you need to have AIX disk I/O pacing enabled to make Oracle run faster.

When Millennium® was first created, AIX (IBM’s Unix) was only 32-bit, and Millennium was deployed with the application node and the database node existing on the same node or lpar. Back then, like today, the various AIX nodes had to be able to share resources for failover. For instance, if you had two Millennium application nodes running WebsphereMQ (MQ) and one node crashed, you needed them to share the log and data file systems for MQ. Why? If node 1 crashed, the other node would be able to complete its transactions.

Way back in the 1990s, Serial Storage Architecture (SSA) drives and HACMP (High Availability Cluster Multiprocessing) were used to share data between nodes. One artifact of this technology was that AIX could send more disk I/O messages to the shared SSA drives than the drives could handle. To minimize this risk, the minpout and maxpout throttles on disk I/O were used to control the disk I/O rate. Maxpout set a limit on the number of pending I/Os (typically 33) for any logical volume (LV) or file. The minpout was set lower (typically from 20-24) and would allow the drive to read and write to the file again once the number of messages fell back to that level.

This disk I/O pacing was a workable solution for SSA drives running on a 32-bit system, but it does not translate to today’s AIX client using storage area networks (SANs) on a 64-bit system. The pacing creates an artificial bottleneck and prevents you from getting the most out of your disk subsystem. It should be disabled.

Unfortunately, the disk I/O pacing caused so many issues when it was not set, the myth persists that you need to keep it set. If you are using SAN storage — as most of you are — this simply is not true and never has been true. SAN fabrics have other pacing mechanisms you should use instead of the AIX OS-level disk pacing to achieve optimal throughput. To disable the disk I/O pacing, you can run the following two commands as root on all of your AIX nodes (please note that these changes take effect immediately):

chdev -l sys0 -a minpout=0

chdev -l sys0 -a maxpout=0

Prognosis: To advance toward optimal throughput, discard the outdated tuning standards of AIX disk I/O pacing in favor of SAN pacing mechanisms.

Next week: Illogical tuning of the logical volumes.

Easy Script Analysis – Part 5

Automating Data Collection

By Rob Dearden

For the past few weeks, David Fletcher and I have covered how to manually collect script performance information and create spreadsheets for analysis. We end the topic this week with an explanation and example of how to automate this collection process.

Our goal is to create reliable data collection in a timely manner without impacting the system significantly. To that end, we need to address two key concerns:

1.    How often should scripts run in a day, and how many days should they run per month?

2.    How big of an impact will the script analysis create on the system, since the analysis is done in Oracle?

Frequency

How often you should run scripts varies, but we have some guidelines to help you determine the answer. You do not want to take snapshots too often, since you could degrade the performance of the database. I would suggest taking snaps an hour or more apart at first. Taking a snapshot every hour is going to be noticeable since it collects all of the data held in the GV$SQL and GV$SQLTEXT views. You might consider taking snapshots every two hours, but you want to ensure script data is not being flushed from the aforementioned views. Taking more snaps to start will allow you to perform analysis of several snapshots to determine the correct frequency. My example below is built with snapshots every two hours, with several “at –f” commands commented out to reduce the number of snapshots taken.

The frequency for days per month, like snaps per day, can start higher for initial analysis and then taper off.  Even though it will have a higher impact, this greater frequency helps create a baseline. Depending on your site, this start could be every day for one week of the month (which will help account for workload changes according to the day), or two or three weekdays in a month.

Impact

The cost associated with script collection and analysis will be available in your data, so be sure to review the cost of DM2_SQLAREA_GATHER and DM2_SQLAREA_ANALYZE scripts in your output files. Make sure to schedule snapshots around each DM2_SQLAREA_ANALYZE command execution, even if you run all of them at a low usage time of night (my example later runs all of them at once).

AIX Auto Collection

To see a detailed example from a previous AIX site I worked with, click here.

Prognosis: Automation will create consistent data collection for more reliable script analysis.

Easy Script Analysis – Part 4

The Cost Basis of Scripts

By David Fletcher

For the past few weeks, Rob has covered how to collect script performance information and create spreadsheets for analysis. This week I want to take a step back and talk about the formula used to calculate the cost basis of scripts. The equations presented in Parts 2 and 3 of this blog were:

(BUFFER_GETS * DISK_READS + BUFFER_GETS * CPU_TIME + DISK_READS * CPU_TIME) / EXECUTIONS

and

(BUFFER_GETS * DISK_READS + BUFFER_GETS * CPU_TIME + DISK_READS * CPU_TIME) * EXECUTIONS

The basis for the equations is to take each of the metrics and multiply them by each of the other metrics, thus creating a “cost vector.” After further consideration (and a reminder from the Pythagorean theorem), however, we determined that a more proper form of the cost basis for scripts should be:

                          

where BG is BUFFER_GETS, DR is DISK_READS, and CT is CPU_TIME.

Because there are three factors in our equation, the result is the length of the hypotenuse in a three-dimensional space and can be considered the magnitude of the Cost Basis vector. Figure 1 depicts a visual representation of the Cost Basis vector.

Figure 1 — Cost Basis Vector

Additionally, because one buffer get is not equivalent to one disk read or one microsecond of CPU time, we can include a “cost factor” in the equation, resulting in the following:

       

where BG_f  is the cost factor for BUFFER_GETS, DR_f  is the cost factor for DISK_READS, and CT_f  is the cost factor for CPU_TIME.

Determining the appropriate cost factors is dependent on the hardware configuration at your site, but if you relate everything back to CPU_TIME (CT_f = 1), all you need to do is determine the average time (measured in microseconds) consumed by the other operations. For both disk reads and buffer gets, two factors influence performance: seek time and transfer rate.

1.    For disk I/O, the transfer rate becomes inconsequential when compared to seek time, so for a good general starting point, use the following equation to determine the cost factor for disk reads:

 

2.    For buffer gets, it is the seek time that becomes inconsequential when compared to the transfer rate, so use the following equation to determine the cost factor for buffer gets:

            

You will notice that the cost factor for disk reads is much larger than the cost factor for buffer gets. This is to be expected because it takes much longer to get data from disk than from memory.

In Excel, if you insert a row at the top of the spreadsheet and put the cost factor values in the first row above the appropriate columns, the Cost Basis equation will look something like this:

=Sqrt(($J$1*J3)^2 + ($K$1*K3)^2 + ($P$1*P3)^2)

I realize that this blog entry has a lot of mathematical equations, so if you have any questions or want to discuss these equations further, please feel free to add a comment below.

Prognosis: An accurate cost basis for each script can be calculated.

Next week: How to automate script analysis.

Easy Script Analysis – Part 3

Scripts that Have a High Impact

Due to the Number of Times They Run

By Rob Dearden and David Fletcher

For the past two weeks we have been talking about script analysis — first, how to collect the data from your Cerner Millennium® system and, then, how to create a spreadsheet to analyze the scripts that have a high impact each time they run. This week we are going to find the scripts that have a high impact on your system due to the number of times they run. At first glance, you may wonder why you can’t just sort by the EXECUTIONS column to find the ones that have the highest impact. All scripts will have a larger impact with more executions, but we must consider all of the metrics that impact script performance to determine their impact.

So, picking up from where we left off last week, open your analysis spreadsheet (the .CSV file that was generated by the dm2_sqlarea_analyze command) and follow these steps:

1.    Insert a column next to the Cost_Per_Execution column we created last week and enter High_Executions into the first row of this new column.

2.    In the second row of the High_Executions column, enter the formula to match this equation: (BUFFER_GETS * DISK_READS + BUFFER_GETS * CPU_TIME + DISK_READS * CPU_TIME) * EXECUTIONS 

Note: Your equation will look something like this:

            =(J2 * K2 + J2 * P2 + K2 * P2) * M2
If at your site you feel that one of the metrics is more costly, you can modify the equation to reflect that.

3.    Copy this equation to every other row in this column where there is corresponding analysis data.

4.    You may need to remove and reapply the filter for the spreadsheet. In Excel 2007, go to the Data ribbon and click the Filter button. After you have turned on the filter feature, you should see a button with a down arrow in the first row of every column.

5.    Click the down arrow in the High_Executions column and then click Sort Largest to Smallest.

You should now have a spreadsheet where the scripts with the highest impact and a large number of executions are at the top. If you have any questions about the High_Executions equation, please post your comment at the end of the blog.

So what do you do now that you’ve identified a script that has a high impact? Once again, it depends why the score is high: Does it have a high number of BUFFER_GETS, DISK_READS, CPU_TIME, or EXECUTIONS — or some combination of these — that are an order of magnitude above other scripts? The same procedure applies for BUFFER_GETS, DISK_READS, and CPU_TIME as we covered last week. For EXECUTIONS that appear to be an order of magnitude above other scripts, however, you will want to investigate the workflow associated with the application or applications that call that script. Your goal will be to identify if the workflow can be modified so that the script isn’t called so frequently.

Prognosis: Analysis of data from Millennium can identify and address scripts that have a high impact due to the number of times they run.

Next week: A look at the formula we used to calculate the baseline cost of scripts.

Easy Script Analysis – Part 2

Scripts that Have a High Impact Each Time They Run

By Rob Dearden

Last week we discussed the importance of analyzing scripts and how to execute CCL commands to gather snapshots and produce an analysis file. This week we’re going to talk about what is and isn’t important in the analysis file, and what to look at to determine which scripts are causing performance issues on your system. Specifically, we are going to help you identify scripts that have a high impact each time they run.

As you may recall, the CCL command to create a snapshot is:

ccl> dm2_sqlarea_gather ‘<snapshot filename>’ go8

and the CCL command to create an analysis of the two snapshots is:

ccl> dm2_sqlarea_analyze ‘<snap1>’,’<snap2>’,’<snap_analysis_file>’ go8

When you open the analysis file in Excel (remember to use a .CSV extension when specifying file names), you may be overwhelmed by the amount of information that is presented. So your first step is to identify which columns are important for analysis and what those columns represent. They are:

        ·         SCRIPT_OBJECT      Script that contains the SQL statement that was executed.

        ·         TYPE                      Column contains either the word TOTAL or the word DETAIL. For analysis purposes we are only interested in the DETAIL records, so we will filter out the TOTAL rows.

        ·         SCORE                    Calculation based on the following formula: SCORE = BUFFER_GETS + (200 x DISK_READS) + (200 x EXECUTIONS). While the ranking this produces is debatable, the intent is to produce an overall view of system impact, rather than the impact of individual statistics.

        ·         BUFFER_GETS         Total number of memory reads that the SQL command performed.

        ·         DISK_READS            Number of disk reads that the statement made.

        ·         EXECUTIONS            Number of times the statement was executed.

        ·         CPU_TIME               Total amount of CPU time used by the statement.

        ·         SQL                         SQL statement.

Now, to identify scripts that have a high impact each time they run, you must consider all of the metrics that give indications of script performance – not just CPU_TIME or SCORE. What we are looking for are SQL statements that have a high impact (whether that is measured in CPU_TIME, DISK_READS, or BUFFER_GETS) along with a low number of executions. To identify these scripts, follow these steps:

1.    Insert a column after the SCORE column and enter Cost_Per_Execution into the first row of this new column.

2.    In the second row of the Cost_Per_Execution column, enter the formula to match this equation: (BUFFER_GETS * DISK_READS+ BUFFER_GETS * CPU_TIME + DISK_READS * CPU_TIME)/EXECUTIONS

Your equation will look similar to this: =(I2 * J2 + I2 * P2 + J2 * P2)/L2
Note: Change 0 in Executions to 0.01 or delete (prevents division by zero), and delete rows with “N/A” for CPU_TIME.

If at your site you feel that one of the metrics is more costly, you can modify the equation to reflect that.

3.    Copy this equation to every other row in this column where there is corresponding analysis data.

4.    Filter the data in the spreadsheet. In Excel 2007, go to the Data ribbon and click the Filter button. After you have turned on the filter feature, you should see a button with a down arrow in the first row of every column.

5.    Click the down arrow in the TYPE column, uncheck TOTAL, and then click OK.

6.    Click the down arrow in the Cost_Per_Execution column, and then click Sort Largest to Smallest.

You should now have a spreadsheet where the scripts with the highest impact are at the top. If you have any questions about the Cost_Per_Execution equation, post a comment at the end of the blog.

So what do you do now that you’ve identified a script that has a high impact every time it is run? The answer is, it depends upon why the score is high: Does it have a high number of BUFFER_GETS, DISK_READS, CPU_TIME, or some combination of these? Here is a starting point based on the metric causing the high score:

1.    A high number of BUFFER_GETS is an indication that the script has to sift through a large number of rows (for example, joining to a table on an un-indexed column). To address this, you could either create appropriate indexes or have the script modified to take advantages of existing indexes. This could also indicate the table has historical data, and purges could reduce the impact of the script.

2.    A high number of DISK_READS is an indication that the data requested by the script is not in the Oracle SGA, which can be resolved in various ways, including (but not limited to) pinning the table in the SGA and/or increasing your SGA size. This solution does not take into account the system resources at your site; discussing options with your Oracle DBA and system administrator is recommended.

3.    A high CPU_TIME is an indication that Oracle has to process a large amount of data. To identify this in a script, check for functions within the script – for example, “WHERE (END_DT_TM – BEG_DT_TM) > :1”.  Functions cause Oracle to perform full table scans and use the CPU to calculate the results of the functions. To address this, evaluate whether the script’s conditions are necessary and as efficient as possible, or if the script can be modified to not include functions within the script.

Once appropriate indexes are in place, the SGA is sized appropriately, and the scripts are as efficient as possible, scheduling them at appropriate times or modifying end-user workflows can alleviate negative end-user impact.

Prognosis: Data analysis can identify and address scripts that have a high impact every time they are run.

Next week: Analyzing scripts that create a high impact due to the number of times they run.

 

Easy Script Analysis – Part 1

By Rob Dearden

During my time as a system administrator, I would regularly analyze script performance in order to discover what items were impacting overall system performance. Our team would need to determine not only what to analyze, but how to do it. Using tools that the software supplier was familiar with helped to ensure everyone was speaking a common language. That way, any differences in interpretations would at least be based on common data points.

For Millennium®, script analysis includes determining how scripts for the application, custom queries, and reports impact the Oracle database. Millennium has tools that help to create snapshots of the Oracle processing as well as a weighted scale for script analysis. The scoring system accounts for the larger impact that, for example, disk reads/writes and executions have on a system compared to the lesser impact of reading/writing to memory. Oracle’s tracking of CPU processing should closely resemble the weighted result produced by the score.

The following example shows you how to run snapshots and then compare the data. The files created will be saved in your ccluserdir by default. If you do not specify a file extension for the snapshot file or the analysis file, the default extension will be .DAT; however, you can specify a file extension, and we recommend using .CSV. For simplicity, the filenames in our example are generic, but it is beneficial to include the domain name and a date/time stamp in each file created to avoid confusion.

The format of the snapshot command is:

ccl> dm2_sqlarea_gather ‘<snapshot filename>’8

Two snapshots are required to perform an analysis. The format of the analyze command is:

ccl> dm2_sqlarea_analyze ‘<snap1>’,’<snap2>’,’<snap_analysis_file>’ go8

The analyze command will give you a .DAT file (unless you specify a .CSV extension for the snap_analysis_file name) containing the following columns (note: your Millennium code level may impact columns): SCRIPT_OBJECT, TYPE, START_DATE, END_DATE, TIME_IN_MINUTES, PERCENT, SCORE, BUFFER_GETS, DISK_READS, ROWS_PROCESSED, EXECUTIONS, BUFFER_RATIO, DISK_RATIO, EXECUTIONS_PER_MIN, CPU_TIME, MAX_SHARABLE_MEM, OUTLINE_CATEGORY, OPTIMIZER_MODE, HASH, FIRST_LOAD_TIME, SQL.

If you are proficient in script analysis, feel free to utilize these commands. (Because the analyze command is comparing elements that are common between the two snapshot files, we recommend using snapshots that are no more than eight hours apart.) For the less proficient, I will guide you through script analysis over the next few weeks. My topics will include:

·         Analyzing scripts that have a high impact each time they run.

·         Analyzing scripts that create a high impact due to the number of  times they run.

·         Automating the collection of script analysis data.

Prognosis: The execution of scripts is necessary for Millennium to function. Analyzing their execution will allow you to correct any negative impacts on overall system performance, thus creating a more consistent end-user experience.