Millennium Medic

MQ Assumptions Might Send the Wrong Message

By David Hallberg

Although not the official motto of the U.S. Postal Service, the following ancient words by the Greek philosopher Herodotus have formed my expectation for mail delivery — and quite possible yours:

Neither snow, nor rain, nor heat, nor gloom of night, stays these couriers from the swift completion of their appointed rounds.

What if my assumption is wrong? What if my letter doesn’t make it to its destination? In recent years, a couple of postal carriers have been convicted of stealing packages. The people who mailed those packages assumed their couriers would swiftly complete their appointed rounds. They operated on a false assumption. Although the vast majority of mail arrives without a hitch, theirs didn’t.

So it is with Millennium’s delivery system. Be careful to assume that all transactions are going through.

One of my recent clients had been told that Millennium cannot lose transactions because it uses IBM’s WebsphereMQ for communication. That’s true to an extent: If a transaction is put into a persistent queue (Millennium RDM queue), and if the receiving MQ queue exists and does not have InhibitPut enabled, the data in that transaction will make it. That’s two “ifs,” and here’s another one: If the filesystems or disk drives that the MQ uses for logs fills up, MQ on that node will stop. This means no messages are getting from their source to their destination.

What about non-persistent queues (SSREP)? They have limitations too: The transaction will go through if the requesting process is connected to the SSREP queue, if the Shared Service Queue Administrator (SCP Entry ID 36) does not have InhibitPut enabled, if the request processed by the server (like CPM Script) completes its work successfully, if the SSREP queue is not full, and if the Exception queue is not full. Whew! Any of these issues could cause MQ to fail to deliver something or to deliver what is called a null return, a return that contains no data.

This might sound to you like a transient issue with a specific request or server that only happens on occasion. Unfortunately for my client, this particular problem caused more than one production outage. How could one request cause a production outage? The problem started to spiral when the Java server that made the request to CPM Script did not get a response back. The following data shows what happened next with the Java server:

Data	Event	Location	Level	Source
{{binding::SSREP.CPMSCRIPT}}{{status::1544}}{{descrip:: There was no message that met the selection criteria}}	QUE_Get	appnode1/ 21215/380	1	srvjavadriver
The request 115458 returned with a status of 1.	CRM_Perform Status	appnode1/ 21215/380	2	srvjavadriver
[3200285_2012_0110_150541_742_CST - Exception occurred during transaction [msvc_svr_get_medication_administrations (3200285)] called by [BOBSURUNCLE]: DataAccessException: com.cerner.msvc.security.internal. user.data.access.proxy.UserDAOProxy at com.cerner.d		appnode1/ 21215/380	0	com.cerner.server. clinicalevent

The generic Java driver sees that the request from the Java service clinical_event never received a response from CPM Script (illustrated by the first row of data: “… There was no message that met the selection criteria”). Since the request from the user BOBSURUNCLE did not get the data needed from CPM Script, the transaction “msvc_svr_get_medication_administrations” failed. I would hope that the clinician was told about the problem and resubmitted the request. But I do not know if that happened. I do know that the client had their production environment hang for a while. Once they called support, they were told to cycle all of the CPM Script servers. After cycling, everything started working again.

So I urge you to be careful with your assumptions. Sometimes the mail carrier fails to deliver your package or delivers a package with nothing in it. Sometimes WebsphereMQ fails to complete a transaction or sends it to the wrong place or completes it without the right data. The impact might be small, affecting just one clinician, or it can be large, affecting the entire organization. Keep an eye on the message log event “QUE_Get.” It will be a very good indicator that something is going wrong, especially with any of the Java services.

Prognosis: Having a reliable messaging system is important; having messages properly handled by specific executables is priceless.

Serving up Answers at HIMSS

By David Hallberg

Buried in the alphabet soup of this year’s HIMSS sessions — EHRs, HIEs, ACOs, CDS, ICD-10, MU — is the reminder that expectations for HIT (healthcare information technology) continue to skyrocket and, with it, the number of EMR options you have to implement. Improved patient safety, affordable care, access to information and accurate recordkeeping all rely on the EMR system.

Is your system ready to meet these growing regulatory and public demands? How do you keep it running reliably and securely as new software and systems are integrated? Will you be at HIMSS later this month to get some answers?

If so, we’d love for you to let us know what you hope to accomplish at HIMSS. What sessions will you attend? What people do you want to talk with? What vendors interest you? You can start a conversation by commenting on this post.

If your hospital is running Cerner Millennium^®, we also invite you to come see us while you’re on the exhibit floor. We Millennium Medics will be in the Softek Solutions booth, #4056, ready to talk with you about any Millennium performance questions you may have. For the past three years, this blog has walked you through steps to troubleshoot common system issues and has sung a familiar tune: “Tune, Tune, Tune.” We have years of experience working with the Cerner system and are the geeks who love to talk about that kind of thing. Please come see us.