Millennium Medic

MQ Assumptions Might Send the Wrong Message

By David Hallberg

Although not the official motto of the U.S. Postal Service, the following ancient words by the Greek philosopher Herodotus have formed my expectation for mail delivery — and quite possible yours:

Neither snow, nor rain, nor heat, nor gloom of night, stays these couriers from the swift completion of their appointed rounds.

What if my assumption is wrong? What if my letter doesn’t make it to its destination? In recent years, a couple of postal carriers have been convicted of stealing packages. The people who mailed those packages assumed their couriers would swiftly complete their appointed rounds. They operated on a false assumption. Although the vast majority of mail arrives without a hitch, theirs didn’t.

So it is with Millennium’s delivery system. Be careful to assume that all transactions are going through.

One of my recent clients had been told that Millennium cannot lose transactions because it uses IBM’s WebsphereMQ for communication. That’s true to an extent: If a transaction is put into a persistent queue (Millennium RDM queue), and if the receiving MQ queue exists and does not have InhibitPut enabled, the data in that transaction will make it. That’s two “ifs,” and here’s another one: If the filesystems or disk drives that the MQ uses for logs fills up, MQ on that node will stop. This means no messages are getting from their source to their destination.

What about non-persistent queues (SSREP)? They have limitations too: The transaction will go through if the requesting process is connected to the SSREP queue, if the Shared Service Queue Administrator (SCP Entry ID 36) does not have InhibitPut enabled, if the request processed by the server (like CPM Script) completes its work successfully, if the SSREP queue is not full, and if the Exception queue is not full. Whew! Any of these issues could cause MQ to fail to deliver something or to deliver what is called a null return, a return that contains no data.

This might sound to you like a transient issue with a specific request or server that only happens on occasion. Unfortunately for my client, this particular problem caused more than one production outage. How could one request cause a production outage? The problem started to spiral when the Java server that made the request to CPM Script did not get a response back. The following data shows what happened next with the Java server:

Data	Event	Location	Level	Source
{{binding::SSREP.CPMSCRIPT}}{{status::1544}}{{descrip:: There was no message that met the selection criteria}}	QUE_Get	appnode1/ 21215/380	1	srvjavadriver
The request 115458 returned with a status of 1.	CRM_Perform Status	appnode1/ 21215/380	2	srvjavadriver
[3200285_2012_0110_150541_742_CST - Exception occurred during transaction [msvc_svr_get_medication_administrations (3200285)] called by [BOBSURUNCLE]: DataAccessException: com.cerner.msvc.security.internal. user.data.access.proxy.UserDAOProxy at com.cerner.d		appnode1/ 21215/380	0	com.cerner.server. clinicalevent

The generic Java driver sees that the request from the Java service clinical_event never received a response from CPM Script (illustrated by the first row of data: “… There was no message that met the selection criteria”). Since the request from the user BOBSURUNCLE did not get the data needed from CPM Script, the transaction “msvc_svr_get_medication_administrations” failed. I would hope that the clinician was told about the problem and resubmitted the request. But I do not know if that happened. I do know that the client had their production environment hang for a while. Once they called support, they were told to cycle all of the CPM Script servers. After cycling, everything started working again.

So I urge you to be careful with your assumptions. Sometimes the mail carrier fails to deliver your package or delivers a package with nothing in it. Sometimes WebsphereMQ fails to complete a transaction or sends it to the wrong place or completes it without the right data. The impact might be small, affecting just one clinician, or it can be large, affecting the entire organization. Keep an eye on the message log event “QUE_Get.” It will be a very good indicator that something is going wrong, especially with any of the Java services.

Prognosis: Having a reliable messaging system is important; having messages properly handled by specific executables is priceless.

ESI Logs Detail Lost Transactions

By David Hallberg

During the past several years, I have seen ESI (External Systems Inbound) logs detailing problems with the inbound interfaces for Millennium^® systems. Unfortunately, people often don’t believe the data I show them. If there were interface issues, I am told, lots of people would know. I wish that were true. When an interface or Millennium first went live, the interfaces might have been error-free, but errors invariably are introduced over the years following changes on the sending system and in Millennium. Rather than deny their existence, users should be working to identify errors and warnings and determine how to correct these issues.

An interface is a group of executables that communicates with another system, like an interface engine or a registration system. These executables allow the systems to communicate without a person reading data from one system and retyping that data into another. The data can be registration or demographic information, such as the patient name, medical record number, date of birth, next of kin, etc. Or it may be clinical information, like a laboratory or radiology order or information from pharmacy dispensing machines. If an error occurs with one of these transactions, the information might not get to its destination, which can lead to sub-optimal clinical care and sub-optimal financial health for the organization. A check of the ESI message logs will help you identify these errors.

Before telling you how to do just that, here’s a quick cheat sheet of the HL7 acronyms of the transaction types (msh_msg_type) you’re likely to encounter:

DFT — Detail Financial Transaction

ORM — Order or pharmacy order

ORU — Order status update, new order or point-of-care observation

SIU — Schedule Information Unsolicited

ADT — Admissions, Discharges and Transfers (demographics)

MDM —  Medical Document Management transaction (e.g., transcribed results)

A complete list of the transaction or message types that compose the HL7 standards can be downloaded from http://www.hl7.org/implement/standards/index.cfm for a nominal fee.

The first step to diagnosing ESI issues is to gather your data. Create two files, one for ESI warnings and one for ESI errors, and run the following queries:

For warnings from CCL:

Rdb SELECT msh_msg_type, msh_msg_trig, msh_sending_app, error_text from ESI_LOG where error_stat = 'ESI_STAT_WARNING' go

For warnings from SQL:

SELECT msh_msg_type, msh_msg_trig, msh_sending_app, error_text from ESI_LOG where error_stat = 'ESI_STAT_WARNING’;

For errors from CCL:

Rdb SELECT msh_msg_type, msh_msg_trig, msh_sending_app, error_text from ESI_LOG where error_stat = 'ESI_STAT_FAILURE' go

For errors from SQL:

SELECT msh_msg_type, msh_msg_trig, msh_sending_app, error_text from ESI_LOG where error_stat = 'ESI_STAT_FAILURE’;

The data will be returned in four columns, which can be pulled into Excel or any spreadsheet or database software and sorted. Create pivot tables that show what message types are having issues. It might be obvious that errors represent failures, but be aware that warnings are often transaction failures as well. For example, a warning of msh_msg_type of DFT with an msh_msg_trig of P03 will have a message such as “Invalid Service Resource Code,” “No rows exist on bill_item_modifier table for this CDM” or “No Matching Debit Charge.” Each represents a financial transaction that did not update the financial record, which means your organization was not compensated for the care you provided.

ESI warnings also result if an appropriate alias has not been built. The ESI server will not understand where to put the data and will send a warning to the message log, but nobody will know that the transaction failed unless someone is checking the log.

ESI failures are often build or configuration issues, such as a missing alias for a code set or missing priority codes, specimen type codes or detailed task assays. The interface might also be sending addendums to a transcribed report that is not in Millennium. No matter what the reason, none of the transactions updated the clinical record, and your clinicians are treating their patients without the most up-to-date information.

Once you have the number and type of events and understand their importance, you can prioritize how to analyze and resolve them.

Prognosis: A quick review of ESI logs can provide great insight into where Millennium might not be fully updating the clinical and financial database. Setting up the processes to review and resolve these events as quickly as your change control allows will provide great benefit to your patients.



A Time to Use Messages.Suppress

By David Hallberg

If you are a regular reader, you know that over the last couple of years I have strongly advocated minimizing the use of Cerner Millennium’s messages.suppress file. Functionality that was intended to prevent extraneous events from cluttering up your message logs has been horribly misused by Millennium system administrators and the application support and application development staff. As a result, vital events are not being sent to the message logs, events that need appropriate troubleshooting and longer term problem resolution. (See “When No News Is Bad News” and “File Can Hide Serious Issues.”)

In recent months, however, I have seen some improvements in Millennium’s base tuning and configuration when a particular event is suppressed, so today I have an unexpected recommendation.

The improvement involves an event called SEC_seclibAudit, which is an Audit-level event telling you that debug is not enabled for the security components. A couple of years ago, only CPM Code Cache Manager seemed to log this event. With the 2010.x code, however, SEC_seclibAudit appears to be written by all of the Millennium servers. It generates two rows in the Data column of the message log each time it occurs. The first row says:

[SecDebugLog::Instance] Creating debug logging singleton.

The second row says:

[SecDebugLog::CheckConfiguration] Debug logging not enabled for all security components, componentLogicalName (SecDebugLogsecrtl)

This is a good setting, but you don’t need to be told about it with every executable. It would be better if you could assume the setting is off unless you took action to turn it on. (To enable debug logging, set the LogLevel of a server to 4 or set the environment variable or logical of CPM _GLOBAL_LOGGING or any one of a number of other logging increases that are available to individual Millennium servers.) This message has become so prevalent, logging on all servers, it has become noise. One client, who cycles 19,000 servers each week, gets 38,000 messages every week for this one event.

To stop the noise, I have to make a recommendation I rarely make: Put this event in the $cer_mgr directory’s messages.suppress file. Then cycle all of your multi-instance servers to make the change take effect.

Prognosis: Using the messages.suppress file appropriately stops extraneous logging and keeps your message logs clearer for real problems.

Make Sure Servers Write Events to Message Logs

By David Hallberg

A client wanting to determine who had cycled a server in Millennium’s Server Control Panel (SCP) at a certain time went to the SCP’s message log cmb_0033 for the answer. When they opened the file, what did they find? Absolutely nothing. No entries or rows had been written to the message log since their last code upgrade. All of the stability issues and change control events that should have been logged were lost. The problem was much bigger than not knowing who had cycled a server. The organization had no way of knowing what else was wrong or how to resolve those issues.

We checked out three parts of the system to track down the problem.

1. Permissions settings in the $cer_log directory. The owners of the files in the $cer_log directory had been set to root for the User and root for the Group. That setting works great much of the time, but not for the following message logs: cmb_0000.mlg, cmb_0033.mlg, cmb_0057.mlg, cmb_0066.mlg and cmb_0078.mlg. For these, the User setting should be root and the Group setting should be your domain user name (for instance, d_prod or d_pxxx, with the xxx being your Millennium client number if you are remote hosted). Then the permissions on the file can be -rw-rw----, which allows the user of root as well as anyone in the group of d_prod to read and write to these log files. If you set these permissions properly, you will have lots of additional information about performance and stability issues in your Millennium domain.
2. SCP settings. SCP servers should have a User Name set to root and a Group Name set to your domain user (for example, d_prod). You also need to set the SCP Property of LogLevel to 2 on the following servers: 0 (Service Manager), 33 (Server Control Panel), 66 (CPM HNAM Agent) and 78 (CPM Script Install). The LogLevel for SCP Entry ID 57 (CPM Script Batch Priv) should be set to 1, unless you are actively troubleshooting an issue with this server.
3. The messages.suppress file in the $cer_mgr directory. If you cat messages.suppress, you will see all of the messages that are not being written to the message log. You are missing significant information if any rows in this file start with the following characters: CMB_, HNAM_, SCP_, InstallOCD, OCDPURGE and Script.

Prognosis: If you check and correct your message log settings before and after an upgrade, you will have much more insight into your Millennium domain and will be better equipped to troubleshoot stability problems and monitor change control.

Messages.suppress File Can Hide Serious Issues

By David Hallberg

Historically, when I was brought into situations where a problem within the EMR system was difficult to track down, I used the message logs quite extensively. They helped me understand what was happening and where the troubleshooting should begin, and they explained any application-specific problems. This method worked very effectively until Millennium^® 2007.18 TP4, when a new file in the $cer_mgr directory — the messages.suppress file — started shipping with some troubling exclusions, namely ScriptError and  ScriptErrMsg events.

The messages.suppress file stops Millennium from writing events to the message logs. This file was designed to stop creating message log events when an engineer accidentally sent an event to clients with an incorrectly set LogLevel. It’s a legitimate place for extraneous events that would otherwise clutter your log files and potentially overwrite messages that need attention. For example, the event CRM_ClientTimeZone writes the current time zone to the message log. Since domains rarely change time zones, the event is seldom of benefit when troubleshooting an issue. Yet it can cause a lot of application node writes, since every transaction going through the application server with this set has to write a message log row into the system message log as well as the specific server message log. To reduce the application node’s disk I/O and performance variability, it’s good to have CRM_ClientTimeZone in the messages.suppress file and exclude these events from the message logs.

If you hide the wrong events, however, you miss out on important information. A problem, failure or issue that should be written to the message log goes undocumented, and your clinical or technical support team has no record to assist them in identifying and resolving issues.

So it makes no sense to me that with the initial TP4, ScriptError and ScriptErrMsg events were included in the messages.suppress exclusions and that the file has since expanded to include ScriptErrors-Fatal and ScriptErrors-NonFatal events. You need visibility into these events. They indicate that an application or Oracle has a problem (often a code or build issue) and needs help to fix it.

What might the problem be? ScriptError events include blood bank transfusion script errors, MDI download problems, scripts failing to run (such as those getting PharmNet frequencies) and Oracle deadlock issues. These events all have a negative outcome to the clinicians using Millennium and the business office needing to bill accurately for services delivered.

ScriptErrors-Fatal events include memory problems that prevent clinical information from being updated and from being viewed by the clinicians who need this information.

ScriptErrors-NonFatal events show a wide range of issues. Expert Knowledge System (EKS) compile errors, Visual Explorer errors, SurgiNet documentation errors, getting orders asynchronously for PowerChart, Lights On Network data-gathering script errors and sequence number problems are all events that can degrade Millennium performance or even cause a failure. These errors must be logged so that problems can be seen and corrected.

Prognosis: Keeping your messages.suppress file properly set will ensure that the Millennium message logs contain the information that your support team needs to identify and resolve issues in a timely manner.

Setting Proper LogLevels Is as Easy as 0-1-2-3-4

By David Hallberg

I am often asked about proper log level settings (LogLevel) for Millennium^® servers. The setting determines what types of messages are written into a server’s message log files, which provide critical information to support teams for correcting application problems and some technical issues. If you set the LogLevel too low, you might miss important information. If you set it too high, you might be flooded with extraneous messages (and overlook the ones you need to address).

LogLevel can be set to one of the following values:

• 0 = Error (ERROR on Java servers)
• 1 = Warning (WARN on Java)
• 2 = Audit (AUDIT on Java)
• 3 = Information (INFORM on Java)
• 4 = Debug (DEBUG on Java)

The settings are cumulative, meaning that servers will write messages from that level as well as all lower levels. For example, a 2 setting would include all Audit (2), Warning (1) and Error (0) messages. You can view LogLevels and change their settings in OnTrack Panther™, SCPView or Olympus. Any time you change a LogLevel you need to cycle the server in order for the change to take effect.

In general, the lower the number, the greater the severity of the messages sent to the log files. Unfortunately, many Audit events are actually failed messages or transactions, which is one reason why a too-low LogLevel might result in lost information…and why many people have questions about what the settings should be. That wasn’t always the case.

In 1997, when most Millennium clients were running on VMS, disk saturation concerns meant that LogLevel was always set to 0/ERROR. When you had two or more application nodes with hundreds of servers pumping log information on the same disk drive, the drive — and everything else — would slow down. AIX, with its SSA drives, had problems as well. The drives shared the same connections as the database. If the executables were using up the connection from the CPU to the disk, then the Oracle database could not get the disk time it needed to operate as well as it could.

Disk saturation is not as much of an issue now that most clients are on 2007.02 or greater, many VMS clients have migrated to some flavor of Unix and the AIX clients are off of SSA drives. Yet logging can still be a problem, even when you are running on a well-tuned storage area network. Today, logging usually manifests itself as a specific queuing problem on a specific executable. If you set servers to LogLevel 0/ERROR, you might miss information that is important to use in finding and correcting problems in your environments. I recommend you no longer use this setting.

I also recommend that no server be set to LogLevel 3/INFORM or 4/DEBUG unless you have an active point asking for debug information. Short of that, these settings cause too many disk writes. I was working with a client recently who had the EKS servers in debug, and the message logs were wrapping in less than 15 minutes. This much logging slowed down all of the EKS rules processing but provided no benefit to the client or its Millennium support team.

So what should you set your LogLevels to?

Monitoring servers — such as 27, 29, 33, 66, 76, 97 and 98 — should be at least a LogLevel 2/AUDIT. The Audit level provides a great deal of useful information when you are working through issues or looking for problems before they actually become outages.

I recommend a LogLevel of 1/WARN for the servers listed below. As a general rule, if your executable is not listed, I recommend you set the LogLevel to 2/AUDIT. You might find you need to make some specific tuning or tweaking to these settings, but they provide a great starting point.

SCP Entry ID		SCP Description
72		CPM Notify Server
73		Time Repository Server
75		User Services
80		CPM Script Cache
86		CPM Expedite
92		OMF PowerVision Document Server
127		System Print Server (Node Specific)
130		PathNet Resource Routing Server
131		SCS Netting Server
132		SCS Collections Server
133		PathNet Result Order Posting Server
134		PathNet Sync Result Order Posting Server
134		PathNet Synchronous Result Order Posting
140		Pathnet - AP Case Retrieval Server
141		Pathnet - AP Automatic Diagnostic Coder
144		Pathnet - AP Coding Cache Manager
205		Clinical Event Server For ESI
209		Clinical Event Server - Batch
221		Search Server
231		MDI Result Server
235		MDI Order Server
237		BMDI Result Server
239		DI Inbound Client
252		FSI ESO-CQM Server
275		FSI ESO Notify Server
280		FSI ESO Server Load #1
281		FSI ESO Server Load #2
282		FSI ESO Server Load #3
283		FSI ESO Server Load #4
284		FSI ESO Server Load #5
285		FSI ESO Server Batch #1
286		FSI ESO Server Batch #2
290		FSI ESO Query Extract Server
291		FSI Batch CQM Server
335		PM Trans Async
340		PM Query
369		Chart Coding Services
375		KIA Event Server
425		FRN Tracking View Server
455		Health Expectation Server

Prognosis: Keeping your LogLevels properly set will ensure that your IT team is notified of the issues it needs to address without being overwhelmed with meaningless messages.

MQ Maintenance for Millennium

By David Hallberg

I am frequently asked about maintenance of the Cerner Millennium® system’s MQ queues, specifically what to do with the Admin.Reply, Audit.Exception, Exception and Z_Exception queues. It’s an excellent question, and the answer is not as clear-cut as you might believe. Each of these queues has different reasons for filling up or keeping transactions and different impacts if they do fill up.

Let’s look at the purpose of each of these queues.

The Admin.Reply queue is used to ensure that appropriate communication lines exist for a server starting a transaction. Millennium executables check with MQ to see if the needed queues exist on the node. So, for example, if an SSREQ/SSREP or RDM queue is needed, the Admin.Reply queue either finds the needed queues or creates them. At that point, the transaction is supposed to be deleted from the queue. The problem with this queue is that various Millennium versions and executables do not always delete the transaction correctly and the queue fills up. Once the queue is full, no new Millennium servers can be started from the Server Control Panel (SCP), command line, Panther or Olympus. So any automated cycle-server jobs as well as any interactive server cycling will fail.

This problem can very, very quickly cause significant performance and stability issues in your domain. I helped one client whose Admin.Reply queue, which had a Max_Depth of 15,000 rows, filled up in less than 10 minutes. The client cycled servers between midnight and 7 a.m., but because this queue was full, all of the cycle jobs failed and caused performance issues.

So how do you prevent a similar scenario? By the time you see a transaction in this queue, the work the transaction was doing is complete. These transactions do not contain any clinical or financial information, so it’s safe to delete them. I recommend that you check this queue on each application node at least once a week and delete any transactions you find.

The Audit.Exception queue is connected with the queues CPMSRVAUDIT, CPMSRVAUDITBATCH, CPMSRVAUDITOUT, SSREP.SRVAUDITMGR and SSREQ.SRVAUDITMGR. It is designed to capture transactions that could not be sent to P2Sentinel, which can happen for any variety of reasons. (Be aware that P2Sentinel is the third-party product used to capture information and could cause HIPAA violations or show inappropriate use of Millennium by your organization’s staff.)

Begin your troubleshooting of this queue by verifying that P2Sentinel was up fully and able to take transactions. If P2Sentinel can take transactions, then you should be able to replay these transactions. If you do not replay or recover these transactions, there will be gaps in the auditing information that you need to be aware of. It is possible to have incorrectly formed transactions, which you might have to delete. Before you do, I suggest that you engage your IT management, Security and whoever is tasked with overseeing HIPAA compliance. Your organization might need to complete additional documentation prior to any deletions.

The Exception queue is designed to hold transactions that could not be put on an MQ queue. How does that make any sense? Here’s the basis of MQ: A process or executable has a transaction to hand off to another process or executable, and application programmers want to make certain these transactions make it from one process to another. So the executable that wants to hand off a transaction puts that transaction into a queue. From there, the receiving executable retrieves the transaction and processes it. Still confused? Suppose the transaction is my electric bill. The electric company wants me to get the bill (transaction), which it delivers to my mailbox (MQ queue), where I go and get the bill (transaction). If for some reason the postal carrier cannot put my bill into my mailbox, the bill will go to the infamous dead-letter office (Exception queue). Someone at the central post office will then do additional processing to send me my bill.

In the case of Millennium, the bill is actually a clinical transaction, so you can’t ignore it. Any transactions sent to the Exception queue have clinical or financial data that was supposed to be processed and acted upon but never made it to queue it was originally sent to. It’s important to recover this information, which you generally can do simply by replaying the transaction to the appropriate queue. This is a little like UPS leaving a message on your door stating they need a signature before leaving a package, and since you were not home, they will try again tomorrow. The next day you stay home and get the package.

A replay generally works with Exception queue transactions because the failure reasons are often transient. If a replay does not work, you will want to log an SR to get additional assistance in either processing or deleting the transactions. As with the Audit.Exception queue, Exception queue transactions generally include clinical or financial data, so deleting them could create gaps in patient records. I would suggest that you engage your IT management, Security and whoever is tasked with overseeing HIPAA compliance before deleting these transactions. You might need to complete additional documentation prior to any deletions.

The Z_Exception queue is my favorite. It is used by the Queue Control Panel (QCP) when the front-end QCPView.exe or Quemon.exe or any other Millennium executable cannot communicate with the QCP. Since the QCP does not handle any clinical or financial data, I would suggest deleting the transactions in this queue. If the Z_Exception queue fills up, the QCP will not work normally and you will not be able to use QCPView.exe or Quemon.exe.

Prognosis: A little housekeeping with the Millennium Exception queues will help make your EMR environment more stable.

Putting the Profit in ProFit

By David Hallberg

I’ve written previously about lost charges and how to examine Millennium’s message logs to find and correct billing issues. Today I want to speak specifically to clients who use ProFit^® Patient Accounting, Millennium’s billing solution. Those wanting to maximize their revenue will find a wealth of information in the message logs on the ProFit servers.

One of ProFit’s components is a third-party subsystem called Sablotron, an XML toolkit that was used to make ProFit easier to support across operating systems (http://freshmeat.net/projects/sablotron). Because it is XML based, Sablotron is independent of the operating system and works the same on AIX, HP-UX, Linux and VMS. This is great from an engineering support perspective. But what does it mean to a healthcare organization?

ProFit has two primary servers: ProFit Transaction Server (SCP Entry ID 475) and ProFit Billing Server (SCP Entry ID 476). Generally, an organization can only control the memory size and loglevel of these asynchronous processing servers, and interactive users typically do not know when these servers (executables) have had any errors or warnings. Someone would have to go digging for these issues to become aware of them. I recommend you get out your shovels.

I recently ran across two events on the Billing Server of a ProFit client. To correct the first issue, we needed to log a service request to get help in correcting the build. For the second, we needed to log a service request to get new code.

The first event was the Sablotron Message, which indicated that the subsystem could not properly handle some data in a billing transaction. In a single four-hour time frame, my client had 4,002 of these events. The Data component of these records had a consistent beginning section that looked like this:

{{msgtype::warning}}{{code::93}}{{module::Sablotron}}

It was followed by one of three ending components.

1.  The most numerous Sablotron Message ending was:

{{URI::arg:/_parsed_}}{{line::544 }}{{node::element '<xsl:for-each>'}}{{msg::unsupported language 'en'}}{{Routine::sablotronErrorHandler::log}}

You’re probably thinking, “This looks like someone sat down at the keyboard after drinking a bit too much.” But the message actually illustrates how many XML files are laid out or formatted. XML is just another way to show data, and the message tells us that in line 544 of the server, there was a language code of “en” that did not have a translation (‘<xsl:for-each>’) and the server didn’t know what to do with it. So the server wrote the message to its message log.

2.  The second most numerous Sablotron Message ending was:

{{URI::arg:/_parsed_}}{{line:: 497}}{{node::element '<xsl:for-each>'}}{{msg::unsupported language 'en'}}{{Routine::sablotronErrorHandler::log}}

This looks like the same drinker was at the keyboard. The message is the same as the previous one, except that the unsupported language code of “en” was in line 497.

3.  The third Sablotron Message ending was:

{{module::Sablotron}}{{msg::unsupported language 'en'}}{{Routine::sablotronErrorHandler::log}}

This message appears to be a higher level failure, but the end result is the same. The server had a language code of “en” that it could not translate, so it wrote the message to its message log.

The second event my ProFit client experienced with the Billing Server was SRV_LookupFailure, which told us in the Data that the bill_item_id was missing from the transaction. This would generally not be a problem, except that in this case the user name was SYSTEM and, therefore, no interactive user was made aware of these failures. They were only in the message log.

Not all the people I work with think these messages are a big deal — they see errors in Millennium as normal. I disagree. The client I’ve referenced here had, on a single node, 3,038 of the line-544 Sablotron Message events and 904 of the line-497 Sablotron Message events in two hours as well as 60 higher level Sablotron Message events and 92 SRV_LookupFailure events in one hour. How could this be considered normal? If an executable is having a problem, you want to correct it, whether the issue is in the build, the code or the use of the code.

Prognosis: Little events can add up to big revenue losses. Looking for these events and resolving the code or workflow issues will maximize your revenue.

Another Upgrade Issue: The TDB

By David Hallberg

Although I have blogged before on issues that might be overlooked during a Millennium^® upgrade, a recent discussion with a client revealed another issue: one with the transaction database (TDB). A common oversight in this area might need a simple fix.

The TDB is a flat file found on each Millennium application node that tells the executables (application servers) what transactions they can process and if the transactions are Request/Reply (RR), Reliable Delivery Message (RDM) or Remote Procedure Call (RPC). Let me start with some background on these three types:

· With RR transactions, the clinical information remains intact even if the transactions are lost or discarded. For instance, using CPM Script to retrieve or read data from the database is an RR transaction. If the transaction is lost, the person asking for the information just has to ask for the information again.

· An RDM transaction is like an order. If the client crashes or the application executable or application node fails, there has to be a mechanism in place to recover or replay that transaction when the system is fully functional again.

· The RPC transaction was added with Millennium 2007.x. It allows the Java servers and Java client applications to process transactions originating from both the local hardware and remote calls. It was the first extension to the Common Request Model in quite some time.

Until recently, the TDB was a connection-based service. PowerChart or the front-end application connected directly to the TDB’s port. The TDB is now a shared service. The TDB server communicates with Citrix servers and fat clients over a network port that it allocates when it starts up. Once the communication port is selected or opened, the TDB server tells the Millennium Service Manager on the application node which port it will be listening to for incoming transactions (requests). The Citrix servers and fat clients must be on the same code level as the application node so that when PowerChart or any front-end Millennium application starts up, it will connect to the Service Manager and be told what port to use to connect with the TDB.

I have found that sometimes during an upgrade, the fat clients or outlying Citrix servers are overlooked. In the case of the TDB, fat clients and Citrix servers that aren’t upgraded might be connecting to the wrong port. If a connection fails because the TDB was not listening on that port, the Service Manager logs a message stating that event CMB_ServiceUnavail1Net occurred. The service name (such as TDB_SYSTEM@PROD) is logged, the Citrix server or fat client’s TCP/IP address is included, and the port the client was trying to communicate with is logged. You can see these messages by looking in the cmb_0000 message log and search for the event CMB_ServiceUnavail1Net.

An example would look something like this:

Domain	Name	Node	Record	Cont.	Date
prod	cmb_0000	Hallberg	{{service_name::TDB_SYSTEM@PROD}} {{tcp_address::10.10.10.10}}{{tcp_port::1240}}	N	8/17/2010 6:53

Event	Level	Location	No.	Source	Sourceref	User
CMB_ServiceUnavail1Net	1	hallberg/ 20200cce/8	1022	cb_service_mngr	cb_service_ mngr.cpp:5635	n/a - [d_prod]

The message under the Record column indicates that the Citrix server or fat client whose TCP/IP address is 10.10.10.10 is trying to communicate with the TDB on port 1240. Because this happened after a Millennium upgrade, the client just has to use tracert or traceroute (depending on your operating system) to find out how far the problem server or client is from the application node, i.e., how many network router hops you have to make. The Citrix server or fat client needs to be scanned for components after the Millennium client upgrade in order to be in sync with the newer application node code. This is the most common problem and is simple to correct.

Prognosis: Reviewing message logs after an upgrade can highlight issues that make Millennium appear unusable or unstable to the clinicians whose Millennium servers or fat clients have not been upgraded.

 

Get a Charge out of Failed Events

By David Hallberg

In 1997, when I first began performing technical audits on Millennium® infrastructures, the biggest concern was to keep the Millennium middleware functional and to tune the Oracle kernel. My job was to make sure the interactive user transactions made it to the Millennium application nodes and the database was able to keep up with the workload. These days, however, the MQ generally is working well and the dependency on the Millennium middleware is greatly reduced, so there’s a lot less to tweak in those areas. Likewise, Oracle nodes/lpar/vpars have at least 64 GB of RAM and large system global areas (SGAs) and provide much higher throughput for the Millennium solutions.

Unfortunately, as these issues have improved over the past 12 years, other key problems have surfaced. One trend that I’ve found in the last two years is an increase in lost charges. Analysis of Oracle tables and Millennium log files show that clients are losing charges at two points: in the orders/activity to charge mapping and at inbound interfaces.

Most clients don’t believe me when I highlight their billing mistakes. I find they get rather defensive instead of eager for me to show them how to fix their problems, some of which can be done fairly easily.

If you want to investigate whether your system is losing charges — and your organization is losing revenue — start at the message logs for CS Charging Async and CS Charging Sync. You want to look for IncompleteEvent, AddBillCodeType, AddItem_Failed, CheckCharge, CheckChargeModForType, DBAPI_FatalScriptErr, OffsetCharges and ScriptError. These events represent the problems I have seen most recently. All but DBAPI_FatalScriptErr and ScriptError are build or configuration issues that typically need only a little research to resolve.

The DBAPI issues are often Oracle space or connectivity issues that should be resolved with your database administrator. If the DBAPI errors are constraint violations or lock issues, they are probably code issues that require you to take updated Millennium service packages. The ScriptError events are almost always code issues, also requiring you to take updated Millennium service packages.

Your investigation might show a staggering number of ESI errors and warnings. The ESI issues are often from the Omnicell or Pyxis interface, which could indicate you are losing pharmacy charges. Even if it’s only 1-2% of all pharmacy charges, your organization could be losing thousands of dollars in revenue.

From the ESI logs, the failures are often from a “null person_id.” That means Millennium received a transaction but it couldn’t apply the charge or event because the patient was not identified. Other failures could include a “null meddef” or “null manufactur id.” Both of these are build issues in which Millennium is receiving incomplete or mismatched information and therefore cannot update the database. ESI warnings often include aliasing issues, script issues, invalid catalog_cd, action_type_cd, order_id or synonym_id. The problem also could be as simple as missing data or an incorrect date/time format.

The ESI log typically contains only the last seven days of data. Failures, therefore, are purged and the information lost unless you are looking at these issues at least once a week.

All of these items are straightforward to correct — once you know about them. The key is to be willing to believe the data in your log files, use it to identify the failed charges and have a process in place to correct the problem.  

Prognosis: Addressing the build issues associated with the charge errors in your message logs and ESI log will translate into increased revenue for your organization. 

Security Changes Create Queuing Issues

By David Hallberg

Cerner recently made security changes to Millennium’s Java services and several older servers that might have disabled your system’s ability to automatically replay failed transactions. Unless corrected, caregivers might be losing important clinical information.

The changes involve Millennium’s personnel (prsnl) table, the listing of users who have permission to access applications and modify tables that contain clinical information. Previously, the username “system” was not typically included in this table because this username was created for those doing low-level executables with no need to access clinical applications or view clinical data. It was used to access failed transactions for SCP Entry IDs 105 and 108 (ORM Explode Cont Order and ORM Explode Cont Order Batch) and to replay them. A few months ago Cerner sent out a Millennium Flash announcement saying, essentially, that  failed transactions could only be replayed if the username was listed in the prsnl table and, to that end, the “system” username was now included in the prsnl table.

By forcing transactions to be replayed or re-queued by a user listed in this table, Millennium is plugging a gap in its security model. However, contrary to the announcement, most clients don’t have “system” listed in the prsnl table. If you’re one of them, you might find that transactions are failing to be replayed and that some might not post the first time.

“System” is one of three Cerner default usernames. The other two — “cerner” and  “systemoe” — are already in the prsnl table, where they need to be. But having this listing is only the first step toward ensuring proper replaying of failed transactions. I worked with two clients in June who had numerous interactive users all properly listed in the prsnl table but whose transactions still were not re-queuing. We discovered that some (hopefully all) Java servers have an extra level of security. Users must be granted access to application 3202004 (java.exe). I suspect that this requirement applies to SCP Entry IDs 105 and 108 as well.

Unfortunately, this access change does not seem to have been widely documented or deployed, so you might be losing some information as you’re waiting on others to put the information into Millennium. Far from being your fault, you just were caught in the switch to the new application security.

What do I recommend? Place the following usernames in the prsnl table and give them access to at least the 3202004 application: system, systemoe, cerner and panthercycler (if you’re a Softek client). Then cycle the CPM Application Authorization servers, SCP Entry IDs 105 and 108, and all of the Java servers (generally SCP Entry IDs 350-374).

Prognosis: Correcting the username listings and Java application access required by the new Millennium security model will allow your failed transactions to replay and will result in improved throughput of clinical transactions. To truly improve system reliability, however, you’ll need to address the underlying problem of why the transactions failed in the first place.

Help Your MQ Become PDQ

By David Hallberg

For those of you trying to get the most out of your MQ system for Millennium®, a few simple adjustments can deliver big benefits.

WebsphereMQ is an IBM middleware used extensively by Millennium. It has two major roles and one minor role.

1.    MQ’s most important use is for messages that have to get processed even if an individual application process or an application node crashes. This security measure means the data has to be written to log files and data files on disk drives, making it the slowest type of transaction for MQ.

2.    The 2007.x and 2010.x versions of Millennium, assuming MQ Shared Services (MQSS) is enabled, use MQ for communication from a client, like Citrix, to the application node. This queuing is non-persistent, meaning if the Citrix client or application node crashes, then the data is lost.

3.    Several years ago the Foundations team worked with the Foreign System Interface team to allow interfaces to use MQ.

For the first and third uses of MQ, the mqs.ini file defines how the logs are configured. Proper tuning of this configuration can have a tremendous impact on throughput. For MQ 6, several of the older limits have been increased. For HP-UX, AIX and Linux, you can now define up to 510 LogPrimaryFiles and up to 509 LogSecondaryFiles, with a maximum combined total of 511 LogPrimaryFiles and LogSecondaryFiles. This is a significant change from MQ 5.3, where the maximums were 62 and 61, respectively, with a maximum of 63 combined.

Setting these log files to their maximums makes the MQ log filesystems much larger, but it also allows for much greater concurrency. This concurrency allows certain functions — such as adding, updating, deleting and reading information — to run at the same time. It also makes it more likely that long-running batch jobs will complete instead of looping because the log files have wrapped.

Next we’ll look at the setting of my favorite mqs.ini entry, LogWriteIntegrity, which controls how many times a transaction is written to a log file. The default value is TripleWrite, meaning every update has to be written to three log files before the next step in the processing can occur. Why all the redundancy? To be fair to IBM, I believe they recognize that they have no control over the hardware and operating systems MQ is installed on. If your hardware server has no disk redundancy, IBM is giving you a reasonable chance to recover data if your node crashes or your drive dies.

Redundancy, I’m glad to report, is not an issue with Millennium production systems. Even old AIX systems have mirroring, that is, two drives with the identical data on it. If one drive dies, the data is available on the other. On current SANs, a writeback cache typically has a battery backup or a mirroring of the data on different memory arrays. The SANs are normally configured with RAID 5 drives, so the loss of a single drive does not mean any lost data. Thanks to this redundancy, you can change the TripleWrite default of the LogWriteIntegrity entry to SingleWrite. You should see improved MQ performance without risking any of your data.

As far as I know, the Cerner Millennium System Settings (CMSS) have not been updated in this area to help Millennium organizations achieve higher throughput from their investment. If any of you have current access to the CMSS for your operating system and the document is updated to include this, please let me know. I would enjoy hearing that the system tuning is keeping up with newer technology.

One last note, the LogWriteIntegrity parameter is valid on MQ 5.3 as well. You can get more information from IBM’s WebsphereMQ System Administration Guide for Version 6.0 (SC34-6584-00) and WebSphere MQ System Administration Guide for Version 5.3 (SC34-6068-02). Please keep in mind that I recommend you use normal change control processes for this tuning and then migrate any changes into your production environment.

Prognosis: A little MQ tuning goes a long way. You may be surprised at the benefits you see when you increase old log file limits and remove unnecessary redundancies.

 

Message Offers Quick Fix for ECO Server

By David Hallberg

In my auditing of Millennium® systems over the past decade, I have seen many meaningless and vague error messages. So I was intrigued by the message I found recently while performing a system audit for a client. The interesting thing about the message was that it actually told us how to fix the problem. I applaud the Millennium developer who was kind enough to think of how to help clients with this issue.

The message appeared when I was reviewing some fast-wrapping message log files. Several messages appeared in the ECO (Explode Continuing Orders) server. The ECO server is used to “explode” orders that have multiple schedule times. For instance, if a clinician orders a daily x-ray for a patient, the information is exploded initially on the ECO server: ORM Explode Cont Order (SCP Entry ID 105). Once the first day’s x-ray is ordered, the order is exploded by a second ECO server for batch jobs: ORM Explode Cont Order 001 (SCP Entry ID 108).

One ECO server message in particular grabbed my attention. Even though it was only an audit-level message, it included the way to resolve the issue. It said:

The number of orders to explode for order_id: 123456789 is > than NUM_ORDERS (50). Only 50 instances will be exploded. To increase the limit, run dcputil_set_config_prefs in CCL and then cycle this server.

I learned that I had more orders to explode than the system would allow. But I also learned how to fix the problem. I had to:

1.    Go to CCL.

2.    Run the script dcputil_set_config_prefs.

3.    Change the NUM_ORDERS from 50 to something larger.

4.    Cycle the ORM Explode Cont Order (SCP Entry ID 105) servers on all of the application nodes and then cycle any clones of this server, such as ORM Explode Cont Order 001 (SCP Entry ID 108).

What about your system? If the LogLevel of SCP Entry ID 105 and any clones of it is 2 or higher, I would encourage you to look for Event ExplodeOrders. If you have a message similar to the one above, you will have a quick fix for your environment. Once you execute it, the clinicians will think that Millennium has magically started working and they can stop working around it or, rather, working around the fact that the orders that were supposed to be there were not.

Prognosis: Interesting error messages can actually be helpful. If you take their advice, you’ll help ensure that Millennium delivers all the information the clinicians need.

 

Preparing for an Upgrade?

By David Hallberg

We have seen so many Millennium® clients have problems after upgrading to 2007.18, 2007.19 or 2010.01 that I feel compelled to address this issue. If you have not yet upgraded, this blog will link you to specific recommendations to help minimize the known risks to your production environment. It’s best to execute them before the upgrade. If you have already upgraded, you may want to review these items and see how many of them you can do to improve your current stability.

In general, I’ve ranked my 14 recommendations in order of importance. The majority of them are MQ and message log issues. They apply whether you are on AIX, HP-UX or VMS (although VMS applies only through the 2007.19 upgrade since Cerner doesn’t support this platform on 2010.01 and above). Click here to read the specifics on correcting the following 14 issues:

1.    The hidden MQ queue.

2.    Java server stress.

3.    Citrix crashes.

4.    Exception queue transactions.

5.    Message log disk I/O increases

6.    Confusing MQ log files.

7.    Extra message log writing.

8.    Too much extraneous logging.

9.    MQ information loss.

10.  Missing or incorrect security settings for a file.

11.  MQ running in mixed mode.

12.  Request class routing needs.

13.  Wrapping issues with message logs.

14.  QCP server instances.

Prognosis: By performing a handful of critical tasks before an upgrade, I believe you’ll have a much shorter troubleshooting time during the upgrade process.

Shared Service Proxies in an MQSS World

By David Hallberg

Millennium®, since version 2007.02, has tried to reduce its proprietary middleware footprint through the use of IBM’s WebSphereMQ for client communications to the application nodes. This is often referred to as MQSS for WebSphereMQ Shared Services. It provides an enormous leap forward in throughput and performance for most Millennium users but has not lessened the need for Shared Service Proxies (SSPs), as was expected.

Millennium’s initial design specification for its proprietary middleware was 125 messages per second, which was pretty good when it was introduced 15 years ago. Plus it often saw higher throughput thanks to newer compilers, faster CPUs and some other changes. WebSphereMQ on version 5.3 for this sort of communication (often referred to as non-persistent queuing), however, was benchmarked at more than 3,200 transactions per second, and 6.2 is benchmarked at more than 3,800 tps, where each transaction is 2 kB in size.

The diagram above gives a simple overview of how shared services work with IBM MQ. Starting at the left of the diagram, we have a client application that is connected to both a request and a reply queue. These queues are used to transport the client request to the application servers and the reply back to the client application.

If you are an AIX or HP-UX client, I recommend that you enable MQSS as quickly as possible to help your throughput. I wish I could say the same for VMS clients. Unfortunately, VMS experiences capacity issues with MQSS. Its TCP/IP stack and BG devices can exhaust their limit of 32,768 ports, at which point they no longer take new connections. Cerner now recommends that all VMS clients stop using MQSS, except for the Java services, which only know how to communicate via MQSS.

While all of this speed is great for clinicians and business staff using Millennium, there is a hitch. Updated Millennium configuration standards state that when MQSS is enabled, the number of SSPs (SCP Entry ID 1) should be reduced to four and that all dedicated SSPs should be disabled. Based on the client data I’ve seen, numerous clients who have gone live with MQSS with only four SSPs have experienced varying degrees of queuing, backlogs or slowdowns in the SSPs. I recently worked with a client who had more than 82 hours of queuing for the CPM Script Batch 001 service.

Their queuing was typical of many others: It was constant, not centered around normal usage patterns of one peak in the late morning and another in the early afternoon. The only way to mitigate this backlog is to increase the number of SSPs or create a dedicated SSP, stop all of those servers on the node and then restart them. This action will move the service to the new generic  SSP or dedicated SSP and should stop the queuing. (To learn more about SSPs, read previous blogs on “SSP’s Path for Rapid Transactions” and “Setting up a Dedicated SSP.”)

This is a very surprising discovery. The Millennium switch to a commercial middleware like WebSphereMQ was supposed to eliminate this queuing. Millennium support might attribute the problem to the SSPs consuming resources that could be freed up for other processing. Although technically true, the impact of the SSPs is relatively small. An SSP starts with less than 16 MBs of RAM and can grow to about 128 MB of RAM. If I ran 16 SSPs that all grew to 128 MB (they often do not grow that large), I would be using the same amount of memory as any one of the Java servers (SCP Entry ID 351 through 370). Since the SSPs do not handle client communication — they don’t talk with PowerChart, FirstNet, PowerChart Office, etc. — they consume very little CPU. These processes do some disk I/O, but compared to CPM Script, SCP Entry ID 51 or the Java servers, it is such a small amount of I/O that it is really just background noise. The resource consumption, therefore, is too small compared to the rest of the Millennium application executables to have any significant impact to the resources of the node, lpar or vpar.

I still recommend that all non-VMS users switch to MQSS as soon as you can. The throughput gains are impressive. I just ask you not to reduce the SSPs to four. Although I don’t know your specific configuration needs, I would suggest you start somewhere in the range of 8-20. Eight was the old minimum recommended value for SSPs in non-production domains, and I have always recommended 20 as a base for any production domain.

Although VMS clients can’t benefit from MQSS, I do urge you to upgrade your system to make it as stable as possible and to ensure that you are able to use all 32,768 ports. Clients on VMS version 8.3 should upgrade to the appropriate VMS 8.3 with TCP/IP Services version 5.6 ECO 5 or later.

Prognosis: Even with MQSS’s welcome increases in throughput, SSPs still have a role in Millennium. Keeping them properly tuned will help prevent queuing in Millennium's proprietary middleware.

Oracle Tables Expose Hidden Issues

By David Hallberg

In working with a couple of clients, I discovered a large number of Millennium® Oracle tables that were used for logging or errors (like a message log) — more than 90 in the 2007.18 and above schema. I find this attribute perplexing. If you wanted to put all of the error and logging information in the database, why create the message logs, .err and .out files? On a practical level, that means analysis of application or stability issues requires users not only to gather data from the message logs but also to run a count of these 90+ Oracle tables to determine which tables have a lot of activity and then run a retrieval to get the information from those Oracle tables. No wonder many don’t bother looking for issues proactively — the task is simply too daunting.

Despite the magnitude of the task, I believe it’s important to do all you can to keep Millennium running as error-free as possible. And one way to accomplish this goal is to monitor error messages and logs and resolve the issues they show you. Last month I wrote about how to find and correct quickly wrapping message logs ( “What Is Measured Improves” ). Today I’ll list the queries you can run to see if any of these 90+ Oracle tables are active on your system — if, for instance, they include more than 10 rows of information, or whatever amount your team sets as its threshold. To see the list, click here.

This analysis is related to the previous blog on message logs and will use a similar methodology for reviewing the Oracle tables. You want to view activity in the tables over time. Run the queries I have listed and gather the row counts, and then run the queries again in a day or week or month and see which ones have grown.

Due to the fact that the tables are based on the Millennium solutions you have configured for your own site, I cannot provide specific guidance on what to look for and how to address an issue. I can, however, provide a little insight into a few of the tables:

· CPMPROCESS_ERROR is important to anyone still on 2001.x or earlier code. This is the Oracle table that the CPM Process server and any clones of CPM Process use when they cannot complete a transaction.

· For the CQM_MDRESULTS_LOG, CQM_MICRESULTS_LOG and CQM_OENTXLOG_QUE tables, you want to make certain that all of the rows are only as old as your purge criterion. For instance, if your purge criterion is seven days, then there should not be any records in these tables more than eight days old. If there are, you need to manually purge them and log a service request (SR) with Cerner to get a purge script that works correctly.

· CHART_SERV_LOG is important to monitor to verify that the chart servers are not sending too much logging information to the database.

· The DCP_ERROR_LOG and ORM_ERROR_LOG are for DCP and order errors. They are often in the message logs, but their inclusion here helps consolidate several of these messages from all of the application nodes into a single queriable location.

· If you are a Millennium Profit client, the CSI_LOG and CSI_ERROR_LOG tables are important to monitor for interfaced charges and their errors.

· The GL_TRANS_LOG for general ledger events and PFT_*_LOG tables for general Profit issues should be monitored and issues rectified as quickly as possible to minimize cash flow and reimbursement issues.

As I recommended in the previous blog, your staff will need to log SRs with Cerner to figure out how to correct your errors and then create and execute an action plan.

Prognosis: Maintaining a list of Oracle tables and knowing their rate of change or growth can be time-consuming, but the work you do up front to catch issues early will mitigate problems for the clinicians and support staff in the future.

What Is Measured Improves

By David Hallberg

I just came across the following quote by famed management consultant Peter Drucker: “What’s measured improves.” It occurred to me that this was the crux of some recent conversations I have had with Millennium clients. I keep being asked why Millennium’s message logs are important. Why would a CIO, CTO, CMIO, director, or project manager care about an application’s log files?

Because what is measured improves.

Millennium has more than 25 distinct solution areas with more than 60 distinct, sellable solutions or applications. How do you know if you have high quality in the delivery of these solutions? How do you know if the deployment of the technologies that support the solutions is working as designed? Look at the message logs.

The IT department is often charged with quality and performance improvement as well as patient safety. So how do you detect and categorize these quality and performance problems in Millennium? Look at the message logs.

I realize that this simple answer gets a bit more complicated by the fact that each application node can have between 283 and 538 log files. How can anyone figure out what to do with all of these files? Most people see the number and simply give up before they start. I admit it is a daunting task, especially if you do not have system management tools to help. However, the following steps will help you sort through the hundreds of files to identify the message logs that point to quality and performance issues with your system.

Start by looking at log files that are wrapping in less than 10 days. Those wrapping in less than one day pop to the top of the log list. Each log file contains the date/time stamp of the oldest and newest record and the number of times the log file has wrapped. To avoid running out of disk space on the application nodes, Millennium sets by default the log files to 4,096 records. The 4,097th record overwrites the first record. This means that if a log file is wrapping, you are losing information you may need to determine if you are having a problem.

You can get to the information you need using one of the following options: the message controls in Panther or Olympus, the ls -l command on AIX and HP-UX, or the dir command on VMS. Panther and Olympus will give you wrap times. If you don’t have these tools, you will need to derive them. Perform the ls -l or dir command of the $cer_log directory, wait one day, and do the command again. Save your output, import it into Excel, and look for the files that changed. Now you know which files are wrapping quickly.

Next look for Errors and Warnings (the Level column will show either 0 for Error or 1 for Warning). They indicate that all or part of a transaction was not put into the database. Since it is not in the database, the database tools cannot see it; these events are generally only visible in the log files. If you have a log file wrapping quickly with the Level being greater than 2, you need to change the loglevel in SCP and cycle the server.

Do not accept the Service Request (SR) response you will get when you log this to Cerner. They will tell you to put the Event into a file called messages.suppress and cycle the server. This action will stop the messages from being written to the log files, but it will not stop the problem. It’s a little like turning up the radio when your car starts making a strange sound. You might not hear the noise, but you haven’t eliminated the problem. Likewise, suppressing messages will declutter the log files, but you are only hiding errors or events, not fixing them, and ultimately putting your clinicians and organization at risk. (See our earlier blog titled “When No News Is Bad News.”)

Once your staff has gathered the information, you need to create and execute an action plan to resolve the issues. Unfortunately, this problem solving is an iterative process. By the time the first round of issues is resolved, you probably will have completed more build in Production and had more Millennium code installed. So the process starts again. I have found that if you start this process in the non-Production domains, you will generally be able to resolve the issues before they get to Production. Then the Production message log audits become much easier and faster.

Once you create this quality loop, clinician and business office complaints will diminish, response times will become more consistent, and the general acceptance of the Millennium system will be higher because people will trust the information it holds.

Prognosis: By using message log audits to measure Millennium’s performance, you uncover problem areas and improve the system. Peter Drucker would not be surprised.

Troubleshooting the MQ Queues

By David Hallberg

This week I’m going to cover a topic near and dear to my heart: MQ queuing. If your Millennium® system is sluggish, the MQ is an important area to investigate. Backlogs here indicate transactions are taking too long to process.

MQ has one of two uses within Millennium. The first is to be a storage location for messages carrying critical clinical or financial data that must be processed. Called Reliable Delivery Model (RDM) messages and stored in the RDM queues, they include orders, charges and other information that must not be lost if something happens to the node, network or other infrastructure component. Essentially, these queues take a transaction from a client’s Citrix server or PC and write the message to disk; then a Millennium application server (executable) picks up the message and processes it. When the processing is complete, the message is removed from the queue. The RDM queues are listed in Panther under the Queues control as CERN.<SCP SERVICE NAME>.

The second use of MQ is for non-persistent queuing, or request/reply (RR) queues. (In Millennium versions prior to 2007.18, the Shared Service Proxies handled this function, unless the MQ Shared Services feature was manually enabled.) CPM Script messages are an example of non-persistent messages. Unlike RDM messages, RR messages are not written to disk. If the application node crashes or becomes unreachable, RR transactions will simply be deleted from the queue. These queues are configured in pairs as CERN.SSREQ and CERN.SSREP queues. The SSREQ queue is where the Citrix server or PC places its messages. The SSREP queue is where the Citrix server or PC gets the data it requested.

Transactions backlogged in the RDM or CERN.SSREQ queues and waiting to be processed cannot be viewed by users. Lowering this queuing will speed up Millennium’s response time to clinicians and other users.

Let’s look briefly at the four areas that can have queues: RDM, RR for SSREQ, RR for SSREP and CERN.EXCEPTION.

1.    To mitigate a backlog of messages in an RDM queue — CERN.CPMPROCESS, for instance — you need to add instances of CPM Process (SCP 55) to the node experiencing the issue. Assuming you are not out of CPU, memory/paging space or disk I/O, this action will increase the throughput of the queue. (This solution does not apply to single instance servers.)

2.    To mitigate a backlog of messages in the RR for SSREQ — CERN.SSREQ.CPMSCRIPTASYNC, for instance — you need to add more instances of CPM Script Async (SCP 54) to the node experiencing the issue. Assuming you are not out of CPU, memory/paging space or disk I/O, this action will increase the throughput of the queue. (This solution does not apply for single instance servers.)

3.    To mitigate a backlog of messages in the RR for SSREP — CERN.SSREP.CPMSCRIPT, for instance — you will need to do additional troubleshooting. These backlogs indicate the Citrix server, PC or other Millennium application executable is no longer running to retrieve its requested data from the queue. This is typically an indication that the Citrix servers or PCs are experiencing crashes in applications like PowerChart and FirstNet. Please follow the documented process from Cerner to identify and resolve these issues.

4.    The CERN.EXCEPTION queue contains a list of transactions that could not be processed by Millennium. The information in these transactions, therefore, is not usable in Millennium applications. Messages go to this queue for one of the following reasons:

·   The destination queue is full.

·   The destination queue does not exist.

·   Message puts have been inhibited on the destination queue (an MQ queue configuration property).

·   The sender is not authorized to use the destination queue.

·   The message is too large for the destination queue.

·   The message contains a duplicate message sequence number.

·   The destination queue is too busy to accept the message.

When the issue is caused by a Millennium server that is too busy, you can often resolve the issue simply by requeuing the message after a few seconds. If requeuing does not resolve the issue, you will need to troubleshoot further to determine the appropriate resolution path.

Prognosis: Backlogs in MQ queues indicate a throughput or speed problem that can typically be resolved by increasing the application servers (executables) processing those messages. Resolving these issues is one step toward ensuring a reliable, fast-running Millennium domain.

When No News Is Bad News

By David Hallberg

In a recent evaluation of a hospital’s Millennium system, I was surprised at the small number of entries in some of the facility’s message log files. While most of you would salivate at the thought of an empty log – and the promise it held for a relaxing lunch break – I was intrigued and wondered what they had done to the system to correct the longer list of errors and warnings that typically appeared in the log files. My intrigue turned to concern when I discovered that nothing had been done.

As I dug further into this hospital’s system, I discovered the problem: Errors were still occurring, but they were being suppressed instead of logged. Had this suppression continued, the hospital ran the real risk of compromising patient care.

The culprit was Millennium’s messages.suppress file, which stops Millennium from writing out these messages. The file is a legitimate place to define a list of unactionable informational messages that would otherwise clutter your log files and slow down your system. In this instance, however, entries that potentially contained critical information relating to system operation and patient care – such as APPAUTH, ESO and ScriptErrors – were included on the suppression list and disappeared from the system. Without these error messages, there was no record of a problem and therefore no way to research an issue and correct it.

I strongly encourage you to obtain the messages.suppress file ($cer_mgr/messages.suppress or CER_MGR:MESSAGES.SUPPRESS) from the back end of your system and examine it closely. If there are any events defined in the file, you need to determine who put them there and why. When entries are either added or removed from the messages.suppress file, you must cycle all of the appropriate server processes to make the change take effect. If, upon investigation of your messages.suppress file, you have questions about what an entry means, I will be more than happy to discuss it with you.

The bottom line – you don’t want to suppress errors. You want to fix them and ensure that the system does not hamper your clinicians from providing the best care they can deliver. You also don’t want to open your organization to significant liability issues without any mechanism for defense.

Prognosis: We caught this hospital’s suppression problem before any patients were affected. We hope you can say the same.