Millennium Medic

MQ Assumptions Might Send the Wrong Message

By David Hallberg

Although not the official motto of the U.S. Postal Service, the following ancient words by the Greek philosopher Herodotus have formed my expectation for mail delivery — and quite possible yours:

Neither snow, nor rain, nor heat, nor gloom of night, stays these couriers from the swift completion of their appointed rounds.

What if my assumption is wrong? What if my letter doesn’t make it to its destination? In recent years, a couple of postal carriers have been convicted of stealing packages. The people who mailed those packages assumed their couriers would swiftly complete their appointed rounds. They operated on a false assumption. Although the vast majority of mail arrives without a hitch, theirs didn’t.

So it is with Millennium’s delivery system. Be careful to assume that all transactions are going through.

One of my recent clients had been told that Millennium cannot lose transactions because it uses IBM’s WebsphereMQ for communication. That’s true to an extent: If a transaction is put into a persistent queue (Millennium RDM queue), and if the receiving MQ queue exists and does not have InhibitPut enabled, the data in that transaction will make it. That’s two “ifs,” and here’s another one: If the filesystems or disk drives that the MQ uses for logs fills up, MQ on that node will stop. This means no messages are getting from their source to their destination.

What about non-persistent queues (SSREP)? They have limitations too: The transaction will go through if the requesting process is connected to the SSREP queue, if the Shared Service Queue Administrator (SCP Entry ID 36) does not have InhibitPut enabled, if the request processed by the server (like CPM Script) completes its work successfully, if the SSREP queue is not full, and if the Exception queue is not full. Whew! Any of these issues could cause MQ to fail to deliver something or to deliver what is called a null return, a return that contains no data.

This might sound to you like a transient issue with a specific request or server that only happens on occasion. Unfortunately for my client, this particular problem caused more than one production outage. How could one request cause a production outage? The problem started to spiral when the Java server that made the request to CPM Script did not get a response back. The following data shows what happened next with the Java server:

Data	Event	Location	Level	Source
{{binding::SSREP.CPMSCRIPT}}{{status::1544}}{{descrip:: There was no message that met the selection criteria}}	QUE_Get	appnode1/ 21215/380	1	srvjavadriver
The request 115458 returned with a status of 1.	CRM_Perform Status	appnode1/ 21215/380	2	srvjavadriver
[3200285_2012_0110_150541_742_CST - Exception occurred during transaction [msvc_svr_get_medication_administrations (3200285)] called by [BOBSURUNCLE]: DataAccessException: com.cerner.msvc.security.internal. user.data.access.proxy.UserDAOProxy at com.cerner.d		appnode1/ 21215/380	0	com.cerner.server. clinicalevent

The generic Java driver sees that the request from the Java service clinical_event never received a response from CPM Script (illustrated by the first row of data: “… There was no message that met the selection criteria”). Since the request from the user BOBSURUNCLE did not get the data needed from CPM Script, the transaction “msvc_svr_get_medication_administrations” failed. I would hope that the clinician was told about the problem and resubmitted the request. But I do not know if that happened. I do know that the client had their production environment hang for a while. Once they called support, they were told to cycle all of the CPM Script servers. After cycling, everything started working again.

So I urge you to be careful with your assumptions. Sometimes the mail carrier fails to deliver your package or delivers a package with nothing in it. Sometimes WebsphereMQ fails to complete a transaction or sends it to the wrong place or completes it without the right data. The impact might be small, affecting just one clinician, or it can be large, affecting the entire organization. Keep an eye on the message log event “QUE_Get.” It will be a very good indicator that something is going wrong, especially with any of the Java services.

Prognosis: Having a reliable messaging system is important; having messages properly handled by specific executables is priceless.

Serving up Answers at HIMSS

By David Hallberg

Buried in the alphabet soup of this year’s HIMSS sessions — EHRs, HIEs, ACOs, CDS, ICD-10, MU — is the reminder that expectations for HIT (healthcare information technology) continue to skyrocket and, with it, the number of EMR options you have to implement. Improved patient safety, affordable care, access to information and accurate recordkeeping all rely on the EMR system.

Is your system ready to meet these growing regulatory and public demands? How do you keep it running reliably and securely as new software and systems are integrated? Will you be at HIMSS later this month to get some answers?

If so, we’d love for you to let us know what you hope to accomplish at HIMSS. What sessions will you attend? What people do you want to talk with? What vendors interest you? You can start a conversation by commenting on this post.

If your hospital is running Cerner Millennium^®, we also invite you to come see us while you’re on the exhibit floor. We Millennium Medics will be in the Softek Solutions booth, #4056, ready to talk with you about any Millennium performance questions you may have. For the past three years, this blog has walked you through steps to troubleshoot common system issues and has sung a familiar tune: “Tune, Tune, Tune.” We have years of experience working with the Cerner system and are the geeks who love to talk about that kind of thing. Please come see us.

OnTrack Panther Supports Millennium 2012.x Code

By Chris Lanaman

David Hallberg’s recent blogs on preparing for your Millennium^® site’s upgrade to 2012.x code have coincided with several conversations with clients who use OnTrack™ Panther™ to manage and monitor system performance. They want to know: “Does Panther support Millennium 2012.x code?” And: “Will OnTrack audits still point out issues as I test the new code and then move it to prod?” I want to assure you that all Softek OnTrack products and services support this version of code.

For those of you not familiar with Softek, our software and services give you a high level of visibility into your EMR system along with actionable evaluations and alerts developed and serviced by a team who collectively had years of experience at Cerner and various Millennium hospitals.

As we enter a new year, we have the same great news we have had since Cerner developed Millennium version 7.6: Softek is working to provide your site an objective view into the performance and reliability of your Millennium system.

The following table lists the Millennium code versions supported by OnTrack Panther.

Millennium Version
7.6
7.7
7.8
2001.01
2001.x
2003.01
2003.02
2005.01
2005.02
2007.02
2007.03-2007.17
2007.18.x
2007.19.x
2010.01.x
2010.02.x
2012.01.x
Choose and Book
eBookings

Prognosis: Third-party support that gives your organization intuitive management of your Millennium EMR system can be the foundation for a smooth upgrade to 2012.x code.

Citrix Load Balancing in a Complex Environment

By David Hallberg

A recent client’s Millennium^® performance issues had me stumped. The organization had a normal two-application and two-database node configuration. It was using very little CPU and had virtually no paging on these nodes. Queuing was pretty good from an MQ and proprietary middleware perspective. There was a lot of interface queuing, but the organization was on an older version of Oracle and we were not going to be able to address this area of their queuing until a database rebuild. So how could I help them right then?

I turned my attention to the Citrix servers — they had more than 100 of them — to check the load balance. The backend gave me little to go on, so I had to look deeper to see how the Citrix servers were connecting with Millennium. In this CPOE shop, Discern Notification alerts clinicians to potential problems with their order deliveries, so I chose to evaluate the Citrix load balancing in this component of Millennium.

When I did, I found that the method of Citrix load balancing being used was causing inconsistent response times for clinicians. Fast application and database nodes could not fix this performance problem. Rather, the team had to use the Citrix load balancing tool on Application User Load to more evenly distribute the core applications of, for instance, PowerChart^® and FirstNet^®. The better application distribution resulted in more consistent performance for their clinicians.

If you want to investigate whether you have a load balancing issue between Citrix and the application nodes, I’ll take you through the steps to parse your data and then evaluate the results. Start with one of the following select statements (either from CCL or SQL*Plus):

            CCL> rdb select * from EKS_NOTIFY_DEST go

            SQLPLUS> select * from EKS_NOTIFY_DEST ;

Your output will look something like this (for space reasons, I have eliminated three columns on the right and reduced my example to two rows of data):

LOCATION	TCP_IP_ADDR	TCP_IP _PORT	PERSON _ID	UPDT_DT_TM	UPDT_ID	UPDT _TASK
54099	192.168.1.62!RLL9989: PROD:DEARMAN	2048	11199999	2011/11/15 01:54:54	11199999	3071000
1555555	192.168.1.61!DAVID. HALLBERG:PROD:DWH9999	2048	8999999	2011/11/04 07:33:07	8999999	3071000

The LOCATION column identifies the device location, set either on the fat client or via the WTS location tool. The TCP_IP_PORT is the port the client is listening on. The PERSON_ID is a unique number for the user name in Millennium. The UPDT_DT_TM shows the last time there was any communication between the application nodes and this client. UPDT_ID is a repeat of the PERSON_ID. UPDT_TASK is the task number for the process that stuck the row in the database or changed it. This number should be 3071000, which is the Discern Notification. Not shown here are UPDT_CNT; UPDT_APPLCTX, the application context number for the application updating this table; and APP_VERSION_NBR, the version of Discern Notification that the client is using. The version generally will be 2, unless you are on some very old code.

The TCP_IP_ADDR column is the key to making sense of this gibberish. It is composed of four elements: the TCP/IP address of the Citrix server or fat client connecting to Millennium, the Microsoft user name, the Millennium domain the connection is for, and the Millennium user name. The next step is to parse the TCP_IP_ADDR column into these four elements.

Import the table into Excel to use the Text to Columns function, and insert three columns between TCP_IP_ADDR and TCP_IP_PORT.

To parse the fat clients or older Citrix servers:

1. Sort the TCP_IP_ADDR column so any cells with a “$” separating the TCP/IP address (first element) from the Microsoft user name (second element) are at the top of the list. Highlight these cells only.
2. From the Excel Data tab, select the Text to Column function, select Delimited and click Next.
3. In the next box, uncheck Tab, select Other, type $ in the open cell and click Finish.

To parse the rest of the Citrix servers:

1. Highlight the other cells in the TCP_IP_ADDR column, which should all have a “!” between the TCP/IP address and Microsoft user name.
2. With the Text to Column function, select Delimited and click Next.
3. In the next box, uncheck Tab, select Other, type ! in the open cell and click Finish.

The first part of the parsing is complete. You now have a list of all the TCP/IP addresses that have connected to the application nodes for Discern Notification. Next you want to see all of the Microsoft user names, Millennium domains and Millennium user names that have connected to the application nodes for Discern Notification.

1. In row 1 column C, which should be blank, type “Microsoft Username:Millennium Domain:Millennium Username”.
2. Highlight all the rows in column C, which contains the remaining three data elements.
3. From the Excel Data tab, select the Text to Column function, select Delimited and click Next.
4. In the next box, uncheck Tab, select Other, type : in the open cell and click Finish.

The next step in getting usable information is to build a Pivot Table (steps are based on Excel 2010) to see the TCP/IP addresses that have the most Discern Notifications:

1. From the Insert tab, select Pivot Table and verify the entire spreadsheet is highlighted. The dialog box should read, “Table1.” Select OK.
2. From the Pivot Table Field List, left-click and drag the TCP_IP_ADDR field to the Row Labels box.
3. Left-click and drag the TCP_IP_ADDR field to the Values box. The Values box will then read, “Count of TCP….” The Table to the left will have two columns.
4. In cell A3, titled Row Labels, click the down arrow and select More Sort Options.
5. Select Descending (Z to A) by:, click the down arrow to change the value to Count of TCP_IP_ADDR and select OK.

The resulting table will list the Discern Notifications for each TCP/IP address, as shown in this example:

Row Labels	Count of TCP_IP_ADDR
192.168.1.213	469
192.168.1.55	426
192.168.1.54	67
192.168.1.57	48
192.168.1.137	43
192.168.1.125	36
192.168.1.79	35
192.168.1.122	34
192.168.1.132	31
192.168.1.107	30

If your servers are properly balanced, the count should be fairly similar across all the Citrix servers. If you have fat clients, they will have a smaller count. In my example, the server ending in .213 handles 15.63 times more connections than the .107 server. Even if we exclude the first two rows, the .54 server is doing 2.23 times more work than the .107 server. Once this imbalance is corrected, clinician response times will be more consistent.

Prognosis: A little research might uncover Citrix load balancing issues that are causing response time variability or other performance problems for your clinicians.

Preparing for Your 2012 Upgrade, Part 2

By David Hallberg

Today, in the second half of my recommendations for ensuring a smooth transition to Millennium version 2012.x, I’ll look at Windows and Cerner’s back-end application nodes. (To see my thoughts on Shared Service Proxy settings and WebSphere MQ, see Part 1.) I recommend you check your settings in the following areas of your 2012.x domain:

• Microsoft added the User Account Control to Windows to protect computers from running unwanted programs, but this feature can cause unwanted interruptions in running Millennium applications. If you are using Windows Server 2008 SP1 32-bit or Windows Server 2008 R2 with Virtual Machines (VMs) running Windows Server 2003 R2 SP2 or Server 2008 SP1 32-bit, make sure the User Account Control is disabled. Two documents explain how: “Turn User Account Control On or Off” from Windows and “User Account Control Step-by-Step Guide” from Microsoft TechNet.
• DNS devolution includes improved security in Windows Server 2008, Vista and Windows 7. Make sure you have it set. Documents to explain how are on uCern at “Windows Vista Minimal Configuration Guide, Install Windows 2008” or “Microsoft Windows 7 and Windows Vista Configuration.” Microsoft offers “Post Installation Behavior … after you install the DNSA update.”
• In the past six months, I have run across more and more sites that have incorrect settings for permissions and owners of files and directories. You can correct the settings by running the secure_cerner_500 executable located in the $cer_exe filesystem. Run it as root, and provide the correct inputs. One caution about this executable: If you are running more than one domain on the same server, lpar, npar or vpar, you have to remove the sticky bit on the files in $cer_fifo, $cer_lock and $cer_usock. If you only have one domain per server, lpar, npar or vpar, this will not be an issue.
• Defrag the $cer_config filesystem to improve throughput for the security and transaction database servers.
• Use the bcheck tool to rebuild the dic.idx file on each application node. If bcheck is not in your $cer_exe filesystem, look in $cer_ocd and find the file named bcheck.dat. Rename the file to bcheck, and put it in $cer_exe. You can search uCern or log a Service Request to get Cerner’s recommended flags to use on this. You might also be able to use the -? flag to get a list of the switches that can be used when you create the file.
• Finally, you want to check the dedicated Shared Service Proxies (SSPs) for your security servers to make sure they won’t be overloaded with connections. At a minimum, you need an SSP for each instance of the Security Slave processes. If you want to have the Security Master service on its own SSP, add one more instance than is needed for the Security Slaves. This means you will have an SSP on the application nodes with no connections, but that’s okay. The SSP only takes 3MB of memory. If you have 2 Security Slaves running with your Security Master and 3 Security Slaves on your other application nodes, you will probably have to increase to at least 5 Security Slaves running with the Security Master and 6 Security Slaves everywhere else. This means you’ll need at least 6 dedicated SSPs for the Security Slaves; if you want a dedicated SSP for the Security Master service, you’ll need 7 dedicated SSPs. Remember to start the dedicated SSPs prior to starting additional Security Slaves in order to get the desired load balancing. Your goal is to keep the maximum connections for each Security Slave under 300. You can check the maximum connections from the Server Control Panel. If you want the steps for how to do it, send me a comment, and I’ll send them to you.

Prognosis: If you prepare for conversion by looking at and correcting your infrastructure and key items in your 2012.x domain, you will have a much quieter upgrade.

Preparing for Your 2012 Upgrade, Part 1

By David Hallberg

Readers have been asking about the changes coming with Millennium^® version 2012.x. This major code upgrade is the first Millennium release that supports Linux as a database, and all Cerner clients are being encouraged to support it. When it hits prod early next year, it will contain a number of upgrades to help users file the reports needed for Meaningful Use reimbursements. Over my next two postings, I’ll share what I know about the release from a technical perspective.

Let me start with a warning. Do not follow Cerner’s recommendation to reduce the number of Shared Service Proxies (SCP Entry ID 1) to between 1 and 4. If you do, you will encounter several sporadic performance issues with Millennium that will be extremely difficult to track down. I recommend that you run 10 to 20 SSPs so that you have more consistent response times from all of the solutions. I realize these processes are not used for communicating with the clients or Citrix servers, but they are doing work — without taking up much space. Since each instance takes about 3.3MB of memory, 20 instances would consume only 66MB of memory, a trivial amount for today’s typical application node that has at least 32GB of memory.

The Millennium upgrade process begins by ensuring that your infrastructure is sound. The rest of today’s post will look at IBM WebSphere MQ, and my next post will make recommendations for Windows before moving on to Millennium’s back-end application nodes.

I recommend you upgrade to WebSphere MQ version 7. The links below will take you to IBM documents that give MQ benchmarks for various operating systems. You want to review the sections on Performance and Capacity Limits and Tuning Recommendation. Then choose the largest value between Millennium’s baseline documents and these documents.

AIX: “MP6N: WebSphere MQ for AIX V7.0 — Performance Evaluations”

HP-UX: “MP6O: WebSphere MQ for HP-UX V7.0 — Performance Evaluations”

Linux: “MPL5: WebSphere MQ for Linux (Intel) V7.0 — Performance Evaluations”

Before going live, make sure you correct any errors resulting from the MQ upgrade. Errors will be shown on the Lights On Network under the System Compliance tab. Look for Known Errors Non-Compliance rows that cover MQ.

If you decide to use MQ version 6, you can reference the following documents for your platform. Again, look at the Performance and Capacity Limits and Tuning Recommendation sections, and choose the largest value between Millennium’s baseline documents and these documents.

AIX: “MP6K: WebSphere MQ for AIX V6.0 — Performance Evaluations”

HP-UX: “MP6L: WebSphere MQ for HP-UX V6.0 — Performance Evaluations”

Linux: “MPL3: WebSphere MQ for Linux (Intel) V6.0 —Performance Evaluations”

Due to the increased number of ports being used by MQ on the front end, I suggest you review Microsoft Support article 196271 (previously published under Q196271). Based on Microsoft’s recommendation, I suggest increasing the MaxUserPort to 65534. You can use a tool called TCP Optimizer. Select the link to the new TCP Optimizer version 3.0.8. You want this executable to be run in XP SP2 mode, and you will have to be a local administrator for all the appropriate keys to be modified. Assuming you are running on Windows Server 2008, any version, here’s the steps to follow: Right-click on the downloaded executable, select Troubleshoot Compatibility, select Try Recommended Settings, select Start the program, slide the slider bar to 100+Mbps, select Modify All Network Adapters, select Optimal, select Apply Changes, and select OK. Finally, reboot your computer.

Keep an eye on the MQ statistics for Enqueue and Dequeue counts. If any Java services are in the top 30, you should consider splitting them. The split process would be similar to the process I described in the “Steps to Ensuring a First-Class Upgrade” blog (under step 2, Java Server Stress).

Prognosis: Doing a little research and establishing a solid infrastructure is the first step to ensuring a smooth conversion.

Next: Proper settings for Windows, permissions and ownerships, security slaves and more.

An Abnormal Definition of Normal

By David Hallberg

Thanks to all of you who stopped by last week during the Cerner Health Conference. We had some great talks about your Millennium^® performance expectations, although I was discouraged to find that for a number of you your expectations were exceedingly low. For example:

• Some of you claimed, “You cannot fix the performance of the application.”
• Others told me, “We are remote hosted and have been assured that we get the best performance that anyone can. When the applications are slow, that’s just the way it is.”
• One client showed me their Lights On Network data for the number of transactions that took over 9 seconds, and I almost fell off my chair. Though the percentage was low, the actual number of transactions was enormous. “That’s way too many,” I said. “It’s the best we can get,” he replied.
• Still another client was told they needed to spend more than $1 million in hardware upgrades to fix their problem.

If you’ve been coaxed into believing any of these statements, let me tell you what I told them: Slow performance is not normal. OK performance is not OK. Don’t believe that you are the fastest Millennium client and there is nothing more for you to do. Your clinicians should not have to wait for 9 seconds for a transaction to go through. Many problems can be fixed without any additional hardware. I know. I have seen Millennium perform extremely well over the past 14 years.

When you are equipped with just a little knowledge of where you have bottlenecks in your system and take the appropriate corrective action, you can see immediate improvements. These improvements do not need to take years and do not require a new version of code, a new operating system or new version of Oracle. You can improve what you have. In the case of the client told to spend $1 million on hardware, we were able to work with the existing hardware and make a couple of uptime changes. Lights On showed that transaction speed for the clinicians had increased and variability had decreased.

Readers of this blog know that I regularly encourage you to trust your clinicians and believe your own eyes rather than blindly accepting performance excuses from the company that wrote the software and is now managing the environment. If the responsiveness of the system is slow, you should be able to see the first level of issues immediately. In a matter of hours or a few days — not months or years — you can determine the initial action steps.

I need to emphasize the words “initial steps.” There is no single silver bullet that will slay your performance dragon; you’ll need a magazine full of lead bullets. The beast is immune to silver but is susceptible to a lead overdose, to an ongoing maintenance regimen that accounts for your continually evolving EMR system. Every new package or service pack you install has the potential to change the processing of the entire system. Make sure you have the right people using the right tools to evaluate what is happening, to identify performance issues and to take action to correct the problems.

Prognosis: If your current support structure is not helping you help your clinicians deliver care more effectively, it’s time to find one that will.