Millennium Medic

MQ Assumptions Might Send the Wrong Message

By David Hallberg

Although not the official motto of the U.S. Postal Service, the following ancient words by the Greek philosopher Herodotus have formed my expectation for mail delivery — and quite possible yours:

Neither snow, nor rain, nor heat, nor gloom of night, stays these couriers from the swift completion of their appointed rounds.

What if my assumption is wrong? What if my letter doesn’t make it to its destination? In recent years, a couple of postal carriers have been convicted of stealing packages. The people who mailed those packages assumed their couriers would swiftly complete their appointed rounds. They operated on a false assumption. Although the vast majority of mail arrives without a hitch, theirs didn’t.

So it is with Millennium’s delivery system. Be careful to assume that all transactions are going through.

One of my recent clients had been told that Millennium cannot lose transactions because it uses IBM’s WebsphereMQ for communication. That’s true to an extent: If a transaction is put into a persistent queue (Millennium RDM queue), and if the receiving MQ queue exists and does not have InhibitPut enabled, the data in that transaction will make it. That’s two “ifs,” and here’s another one: If the filesystems or disk drives that the MQ uses for logs fills up, MQ on that node will stop. This means no messages are getting from their source to their destination.

What about non-persistent queues (SSREP)? They have limitations too: The transaction will go through if the requesting process is connected to the SSREP queue, if the Shared Service Queue Administrator (SCP Entry ID 36) does not have InhibitPut enabled, if the request processed by the server (like CPM Script) completes its work successfully, if the SSREP queue is not full, and if the Exception queue is not full. Whew! Any of these issues could cause MQ to fail to deliver something or to deliver what is called a null return, a return that contains no data.

This might sound to you like a transient issue with a specific request or server that only happens on occasion. Unfortunately for my client, this particular problem caused more than one production outage. How could one request cause a production outage? The problem started to spiral when the Java server that made the request to CPM Script did not get a response back. The following data shows what happened next with the Java server:

Data	Event	Location	Level	Source
{{binding::SSREP.CPMSCRIPT}}{{status::1544}}{{descrip:: There was no message that met the selection criteria}}	QUE_Get	appnode1/ 21215/380	1	srvjavadriver
The request 115458 returned with a status of 1.	CRM_Perform Status	appnode1/ 21215/380	2	srvjavadriver
[3200285_2012_0110_150541_742_CST - Exception occurred during transaction [msvc_svr_get_medication_administrations (3200285)] called by [BOBSURUNCLE]: DataAccessException: com.cerner.msvc.security.internal. user.data.access.proxy.UserDAOProxy at com.cerner.d		appnode1/ 21215/380	0	com.cerner.server. clinicalevent

The generic Java driver sees that the request from the Java service clinical_event never received a response from CPM Script (illustrated by the first row of data: “… There was no message that met the selection criteria”). Since the request from the user BOBSURUNCLE did not get the data needed from CPM Script, the transaction “msvc_svr_get_medication_administrations” failed. I would hope that the clinician was told about the problem and resubmitted the request. But I do not know if that happened. I do know that the client had their production environment hang for a while. Once they called support, they were told to cycle all of the CPM Script servers. After cycling, everything started working again.

So I urge you to be careful with your assumptions. Sometimes the mail carrier fails to deliver your package or delivers a package with nothing in it. Sometimes WebsphereMQ fails to complete a transaction or sends it to the wrong place or completes it without the right data. The impact might be small, affecting just one clinician, or it can be large, affecting the entire organization. Keep an eye on the message log event “QUE_Get.” It will be a very good indicator that something is going wrong, especially with any of the Java services.

Prognosis: Having a reliable messaging system is important; having messages properly handled by specific executables is priceless.

OnTrack Panther Supports Millennium 2012.x Code

By Chris Lanaman

David Hallberg’s recent blogs on preparing for your Millennium^® site’s upgrade to 2012.x code have coincided with several conversations with clients who use OnTrack™ Panther™ to manage and monitor system performance. They want to know: “Does Panther support Millennium 2012.x code?” And: “Will OnTrack audits still point out issues as I test the new code and then move it to prod?” I want to assure you that all Softek OnTrack products and services support this version of code.

For those of you not familiar with Softek, our software and services give you a high level of visibility into your EMR system along with actionable evaluations and alerts developed and serviced by a team who collectively had years of experience at Cerner and various Millennium hospitals.

As we enter a new year, we have the same great news we have had since Cerner developed Millennium version 7.6: Softek is working to provide your site an objective view into the performance and reliability of your Millennium system.

The following table lists the Millennium code versions supported by OnTrack Panther.

Millennium Version
7.6
7.7
7.8
2001.01
2001.x
2003.01
2003.02
2005.01
2005.02
2007.02
2007.03-2007.17
2007.18.x
2007.19.x
2010.01.x
2010.02.x
2012.01.x
Choose and Book
eBookings

Prognosis: Third-party support that gives your organization intuitive management of your Millennium EMR system can be the foundation for a smooth upgrade to 2012.x code.

Citrix Load Balancing in a Complex Environment

By David Hallberg

A recent client’s Millennium^® performance issues had me stumped. The organization had a normal two-application and two-database node configuration. It was using very little CPU and had virtually no paging on these nodes. Queuing was pretty good from an MQ and proprietary middleware perspective. There was a lot of interface queuing, but the organization was on an older version of Oracle and we were not going to be able to address this area of their queuing until a database rebuild. So how could I help them right then?

I turned my attention to the Citrix servers — they had more than 100 of them — to check the load balance. The backend gave me little to go on, so I had to look deeper to see how the Citrix servers were connecting with Millennium. In this CPOE shop, Discern Notification alerts clinicians to potential problems with their order deliveries, so I chose to evaluate the Citrix load balancing in this component of Millennium.

When I did, I found that the method of Citrix load balancing being used was causing inconsistent response times for clinicians. Fast application and database nodes could not fix this performance problem. Rather, the team had to use the Citrix load balancing tool on Application User Load to more evenly distribute the core applications of, for instance, PowerChart^® and FirstNet^®. The better application distribution resulted in more consistent performance for their clinicians.

If you want to investigate whether you have a load balancing issue between Citrix and the application nodes, I’ll take you through the steps to parse your data and then evaluate the results. Start with one of the following select statements (either from CCL or SQL*Plus):

            CCL> rdb select * from EKS_NOTIFY_DEST go

            SQLPLUS> select * from EKS_NOTIFY_DEST ;

Your output will look something like this (for space reasons, I have eliminated three columns on the right and reduced my example to two rows of data):

LOCATION	TCP_IP_ADDR	TCP_IP _PORT	PERSON _ID	UPDT_DT_TM	UPDT_ID	UPDT _TASK
54099	192.168.1.62!RLL9989: PROD:DEARMAN	2048	11199999	2011/11/15 01:54:54	11199999	3071000
1555555	192.168.1.61!DAVID. HALLBERG:PROD:DWH9999	2048	8999999	2011/11/04 07:33:07	8999999	3071000

The LOCATION column identifies the device location, set either on the fat client or via the WTS location tool. The TCP_IP_PORT is the port the client is listening on. The PERSON_ID is a unique number for the user name in Millennium. The UPDT_DT_TM shows the last time there was any communication between the application nodes and this client. UPDT_ID is a repeat of the PERSON_ID. UPDT_TASK is the task number for the process that stuck the row in the database or changed it. This number should be 3071000, which is the Discern Notification. Not shown here are UPDT_CNT; UPDT_APPLCTX, the application context number for the application updating this table; and APP_VERSION_NBR, the version of Discern Notification that the client is using. The version generally will be 2, unless you are on some very old code.

The TCP_IP_ADDR column is the key to making sense of this gibberish. It is composed of four elements: the TCP/IP address of the Citrix server or fat client connecting to Millennium, the Microsoft user name, the Millennium domain the connection is for, and the Millennium user name. The next step is to parse the TCP_IP_ADDR column into these four elements.

Import the table into Excel to use the Text to Columns function, and insert three columns between TCP_IP_ADDR and TCP_IP_PORT.

To parse the fat clients or older Citrix servers:

1. Sort the TCP_IP_ADDR column so any cells with a “$” separating the TCP/IP address (first element) from the Microsoft user name (second element) are at the top of the list. Highlight these cells only.
2. From the Excel Data tab, select the Text to Column function, select Delimited and click Next.
3. In the next box, uncheck Tab, select Other, type $ in the open cell and click Finish.

To parse the rest of the Citrix servers:

1. Highlight the other cells in the TCP_IP_ADDR column, which should all have a “!” between the TCP/IP address and Microsoft user name.
2. With the Text to Column function, select Delimited and click Next.
3. In the next box, uncheck Tab, select Other, type ! in the open cell and click Finish.

The first part of the parsing is complete. You now have a list of all the TCP/IP addresses that have connected to the application nodes for Discern Notification. Next you want to see all of the Microsoft user names, Millennium domains and Millennium user names that have connected to the application nodes for Discern Notification.

1. In row 1 column C, which should be blank, type “Microsoft Username:Millennium Domain:Millennium Username”.
2. Highlight all the rows in column C, which contains the remaining three data elements.
3. From the Excel Data tab, select the Text to Column function, select Delimited and click Next.
4. In the next box, uncheck Tab, select Other, type : in the open cell and click Finish.

The next step in getting usable information is to build a Pivot Table (steps are based on Excel 2010) to see the TCP/IP addresses that have the most Discern Notifications:

1. From the Insert tab, select Pivot Table and verify the entire spreadsheet is highlighted. The dialog box should read, “Table1.” Select OK.
2. From the Pivot Table Field List, left-click and drag the TCP_IP_ADDR field to the Row Labels box.
3. Left-click and drag the TCP_IP_ADDR field to the Values box. The Values box will then read, “Count of TCP….” The Table to the left will have two columns.
4. In cell A3, titled Row Labels, click the down arrow and select More Sort Options.
5. Select Descending (Z to A) by:, click the down arrow to change the value to Count of TCP_IP_ADDR and select OK.

The resulting table will list the Discern Notifications for each TCP/IP address, as shown in this example:

Row Labels	Count of TCP_IP_ADDR
192.168.1.213	469
192.168.1.55	426
192.168.1.54	67
192.168.1.57	48
192.168.1.137	43
192.168.1.125	36
192.168.1.79	35
192.168.1.122	34
192.168.1.132	31
192.168.1.107	30

If your servers are properly balanced, the count should be fairly similar across all the Citrix servers. If you have fat clients, they will have a smaller count. In my example, the server ending in .213 handles 15.63 times more connections than the .107 server. Even if we exclude the first two rows, the .54 server is doing 2.23 times more work than the .107 server. Once this imbalance is corrected, clinician response times will be more consistent.

Prognosis: A little research might uncover Citrix load balancing issues that are causing response time variability or other performance problems for your clinicians.

ESI Logs Detail Lost Transactions

By David Hallberg

During the past several years, I have seen ESI (External Systems Inbound) logs detailing problems with the inbound interfaces for Millennium^® systems. Unfortunately, people often don’t believe the data I show them. If there were interface issues, I am told, lots of people would know. I wish that were true. When an interface or Millennium first went live, the interfaces might have been error-free, but errors invariably are introduced over the years following changes on the sending system and in Millennium. Rather than deny their existence, users should be working to identify errors and warnings and determine how to correct these issues.

An interface is a group of executables that communicates with another system, like an interface engine or a registration system. These executables allow the systems to communicate without a person reading data from one system and retyping that data into another. The data can be registration or demographic information, such as the patient name, medical record number, date of birth, next of kin, etc. Or it may be clinical information, like a laboratory or radiology order or information from pharmacy dispensing machines. If an error occurs with one of these transactions, the information might not get to its destination, which can lead to sub-optimal clinical care and sub-optimal financial health for the organization. A check of the ESI message logs will help you identify these errors.

Before telling you how to do just that, here’s a quick cheat sheet of the HL7 acronyms of the transaction types (msh_msg_type) you’re likely to encounter:

DFT — Detail Financial Transaction

ORM — Order or pharmacy order

ORU — Order status update, new order or point-of-care observation

SIU — Schedule Information Unsolicited

ADT — Admissions, Discharges and Transfers (demographics)

MDM —  Medical Document Management transaction (e.g., transcribed results)

A complete list of the transaction or message types that compose the HL7 standards can be downloaded from http://www.hl7.org/implement/standards/index.cfm for a nominal fee.

The first step to diagnosing ESI issues is to gather your data. Create two files, one for ESI warnings and one for ESI errors, and run the following queries:

For warnings from CCL:

Rdb SELECT msh_msg_type, msh_msg_trig, msh_sending_app, error_text from ESI_LOG where error_stat = 'ESI_STAT_WARNING' go

For warnings from SQL:

SELECT msh_msg_type, msh_msg_trig, msh_sending_app, error_text from ESI_LOG where error_stat = 'ESI_STAT_WARNING’;

For errors from CCL:

Rdb SELECT msh_msg_type, msh_msg_trig, msh_sending_app, error_text from ESI_LOG where error_stat = 'ESI_STAT_FAILURE' go

For errors from SQL:

SELECT msh_msg_type, msh_msg_trig, msh_sending_app, error_text from ESI_LOG where error_stat = 'ESI_STAT_FAILURE’;

The data will be returned in four columns, which can be pulled into Excel or any spreadsheet or database software and sorted. Create pivot tables that show what message types are having issues. It might be obvious that errors represent failures, but be aware that warnings are often transaction failures as well. For example, a warning of msh_msg_type of DFT with an msh_msg_trig of P03 will have a message such as “Invalid Service Resource Code,” “No rows exist on bill_item_modifier table for this CDM” or “No Matching Debit Charge.” Each represents a financial transaction that did not update the financial record, which means your organization was not compensated for the care you provided.

ESI warnings also result if an appropriate alias has not been built. The ESI server will not understand where to put the data and will send a warning to the message log, but nobody will know that the transaction failed unless someone is checking the log.

ESI failures are often build or configuration issues, such as a missing alias for a code set or missing priority codes, specimen type codes or detailed task assays. The interface might also be sending addendums to a transcribed report that is not in Millennium. No matter what the reason, none of the transactions updated the clinical record, and your clinicians are treating their patients without the most up-to-date information.

Once you have the number and type of events and understand their importance, you can prioritize how to analyze and resolve them.

Prognosis: A quick review of ESI logs can provide great insight into where Millennium might not be fully updating the clinical and financial database. Setting up the processes to review and resolve these events as quickly as your change control allows will provide great benefit to your patients.



Continuing to Improve Failover: Security Master Configuration

By David Hallberg

After our last post, “Improving Interface Failovers,” a few people wondered why some Cerner Millennium^® servers — most often Ops Job (Millennium batch process), interface and MDI servers — would not start during the failover. Because this situation can have disastrous consequences, it’s an issue worth addressing.

Imagine the following scenario: It’s 3 a.m. on Memorial Day, and your primary application node crashes. The failover scripts run so that users can log into Millennium, but none of the Ops Jobs are running. That means the nursing staff is missing care plans for its shift change, new patients and their lab results are not visible in Millennium, and the Emergency Department director is calling the CIO, COO and CEO complaining that his staff has to go on diversion because they cannot get the patients into Millennium. A storm just ruined your holiday plans.

Had you been aware of a few simple steps that would make your servers fail over successfully, your day could have been clear sailing.

Why didn’t the Ops Job servers start? If the server on a specific Server Control Panel (SCP) Entry ID has not been started on this node before, the Security Master server has to be running to create the service with the correct permissions. If no Security Master server is running after the failover process completes, any of the servers or services that have not gotten to a running state on the failover node prior to the failover cannot be started.

A little analysis will explain why this scenario happens so frequently. Way back when, about 90 days before you first brought Millennium up, consultants from the vendor swooped in to set up failover. They ran through a couple of tests and showed you how, should the primary node go down, you could still log into Millennium. They flew back home, and you turned your attention back to what seemed like more critical work that had to be done before go-live.

Now, many years later, you have added and deleted Ops Jobs, your old registration system has been upgraded, the inbound feed changed names to allow testing into Millennium production and to minimize the cutover time, you added about a dozen MDI servers, you moved on to a different Millennium client, or you were promoted, and your project staff is scattered around various clinical areas. Go-live and pre-go-live are remembered for the number of service packs you had to install and test and the huge amount of training you had to do. Who recalls the little detail about the need for the Security Master to be running to allow a new Millennium server to start on a node?

What, in fact, is the Security Master? For organizations using LDAP to access and maintain their user directory, the Security Master server updates the sec_user.dat file with local accounts, for example SYSTEM, SYSTEMOE and CERNER. For those not using LDAP, the Security Master also is used to add, remove, modify and delete all user IDs. So when a new person joins your organization and you run the tool HNAUser, a new row is added to sec_user.dat as well as some Oracle tables. Additionally, the Security Master creates a service key for the Millennium processes as they start up. These service keys only exist in the sec_user.dat file; they are not kept in LDAP. This key is used to make sure the processes running on the application nodes are legitimate processes. During normal operations, no one really cares about the security services. They quietly do their work when you stop and start Millennium servers on the application nodes, and everyone — especially your security manager — is happy. Until a failover.

Let’s go back to our Memorial Day crisis, where your family headed to the lake without you. You, like most Millennium clients, have Node 1 of SCP Entry ID 32 configured as the Security Master and Node 2 as the Security Slave. With this common configuration, however, you do not fail over the Security Master, so all of the Ops Jobs you added after go-live, along with new interfaces and MDI servers, didn’t seem to start. Yet nothing shows in any of the message logs. You called your vendor’s Immediate Response Center, but the response has not been immediate enough. After calling you back in 15 to 45 minutes to verify you are still having the problem, IRC might take four more hours to contact you again. In the meantime, the ED director is calling for your resignation and the ICU, CCU and NNICU staff are looking for a throat to choke.

How could you have prevented the chaos and been fishing for something other than answers? The single biggest thing you can do is NOT put a Security Slave on SCP Entry ID 32. Use any other SCP Entry ID — 31 and 19 are often open. After you have the Security Slave configured to run on a different SCP Entry ID, copy the Security Master from the primary node to the secondary nodes. Verify that the Instance is set to 0, which will ensure that SCP Entry ID 32 does not start up and that CPM HNAM Agent does not start it. Finally, put 32 in the cerner.vars file as a single-instance server. That way, when the failover scripts run, you will be able to start the Ops Jobs, interfaces and MDIs as needed to minimize your downtime.

Here’s one last detail for all the system administrators who like such specifics: Create a chron job to automatically copy the $cer_config’s security.jnl file from the application node where the Security Master is running to the other application nodes. Then verify that the permissions are “-rw-rw-----” and the owner and group (assuming your domain name is prod) are both d_prod.

Prognosis: A little preparation can dramatically improve the failover process and make your night, weekend and holiday events much shorter and quieter.

Improving Interface Failovers

By David Hallberg

Today I'll tackle a topic new to this blog: successful failover. Your organization, like many that I talk with, might have brought in a very highly paid consultant to set up failover for your Cerner Millennium® system. Yet, when you need to fail one application node to another to perform maintenance, the interfaces don't fail over correctly. Let me give you a few ideas that might help — and other than a change in your thinking, my ideas won't cost you a thing.

Cerner has extended some functionality in Open Engine's (OEN) Oracle tables that makes failover less painful for its clients and their Millennium support team. Users can set up the system so that the base Millennium failover scripts automatically start the interfaces on the failover node. (In the OEN_PERSONALITY table, where the column Name is PRODUCTION, the flag would be set to Y.)

I know I'm in danger of losing the OEN team here, but keep breathing and hear me out. Typically, the OEN team believes that nothing should be automatically failed over because you could have synchronization issues when more than one registration system sends Millennium ADT transactions. Even single ADT systems, where other systems send Millennium orders, order statuses, transcriptions, etc., can have synchronization issues. These issues can result in an order coming through before the patient has been added to Millennium. I don't dispute your concern. However, I absolutely do not agree that none of the interface servers can be automatically failed over.  

ESI Servers and Outbound Interfaces

Two types of servers, in particular, can be set to fail over automatically with no synchronization risk for incoming interfaces. They are:

1. The ESI servers, or ESI COMM servers. These are not SCP Entry ID 250 and its clones but the servers that take the Inbound client transactions and route them to SCP Entry ID 250 (CPM ESI) and its clones as well as SCP Entry ID 335 (PM Trans Async).
2. The Outbound interfaces, those that send transactions to systems outside of Millennium.

Finding these interface servers can be tricky — Millennium OpenView and Olympus make you go through several screens and tabs — so I suggest you start an interactive CCL session and run the following query to find the ESI COMM servers:

Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PROTOCOL' and op.value ='ESI')
order by pi.SCP_EID asc
GO

The resulting five columns will show you the ESI COMM servers, which can all have their PRODUCTION flag safely set to Y. They will only process transactions after you have started the Inbound COMM server that feeds these servers' transactions. That gives you one less set of servers that have to be started for the Inbound interfaces. I would suggest that the primary ADT Inbound interface be enabled for automatic failover. If during a controlled outage, you do not want the primary ADT interface to be up and running first after the failover, you can easily change the PRODUCTION flag back to N.

The interfaces configured to send transactions to systems outside of Millennium can be found by running this CCL command:

Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PACKESO' and op.value > '0')
order by pi.SCP_EID asc
GO

As with the ESI COMM servers, you can safely set the PRODUCTION flag to Y. Now as your clinicians are doing their normal work, the transactions configured to leave Millennium will not get hung up in Millennium for any length of time.

You might want to take one extra safety measure if you have Outbound pharmacy orders. SCP Entry ID 625 (Medication Dispense and Dose Server) and ID 626 (Medication Fill/Report Server) have a property called FsiCqmDowntimePref that can be added and configured. Set this property to 1 so that Outbound pharmacy transactions can be recovered if SCP Entry ID 252, the FSI ESO-CQM server, is down for any reason.

Additional Servers for Failover Node

Another factor that can impede interface failovers is if any of the following servers are not already running on the failover node:

• SCP Entry ID 250, the CPM ESI servers, along with any clones or copies. If no interface server is running on the failover node, then nothing should be sending these servers any transactions.
• SCP Entry ID 107, ORM Order Write Synch for FSI.
• SCP Entry ID 252, the FSI ESO-CQM server.
• SCP Entry ID 205, the Clinical Event server for ESI, and any of its clones or copies.

The cerner.vars file should be reviewed for the single-instance servers. The file should include at least the following servers: SCP Entry ID 240, 243, 245, 255, 270, 275, 276, 277, 280, 281, 282, 283, 284, 285, 291 and 292. I excluded MDI servers from this list because I am only reviewing interfaces and they involve a great number of variables, which I'll talk about later.

Keeping TCP/IP Ports Free

The last area I want to cover involves the TCP/IP ports. Interfaces will not fail over properly if the TCP/IP ports that you have configured the interfaces to communicate with are in use. How can the ports be in use? Through the normal cycling of servers, ports that initially were free can become used over time. You can prevent this by accessing the file named "/etc/services" and adding the incoming ports for the interfaces and even the MDI servers. For example, telnet is a service that runs on a port, normally 23. So the entry for this would look like:

telnet                23/tcp

If you want to reserve your Inbound interface and MDI ports, all you have to do is add the ports and a name to this file. I suggest you use the SCP Description (minus spaces) for the name along with the SCP Entry ID or OEN PID for clarity, as shown below:

telnet                23/tcp
ADTin                10046/tcp
ORUin               11001/tcp

Once added, these ports will not be used for anything else.

To create a list of the TCP ports you need to put in the "/etc/services" file, use the following command in CCL:

Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PORT' and op.value > '0')
order by pi.SCP_EID asc
GO

Dealing with MDIs

For MDIs, you have to run the command enumlog –all on the two nodes and verify that all of the MDI configurations exist on both nodes and match exactly on both nodes. Because typographical errors are extremely painful to find, I recommend you output the data to Excel and use the EXACT function to compare cells. It will quickly find any typographical errors.

If you want the MDIs to fail over automatically, you can use the following command for CCL to find them and then use OEN_PROC_VIEW GO to modify their PRODUCTION flag to Y:

Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PROTOCOL' and op.value ='CUSTOM')
order by pi.SCP_EID asc
GO

I did not include the MDI servers in the cerner.vars list because too many variables exist for these. Which MDI servers need to be in the file and not able to run on the failover node depend on the code level you are on, the Millennium Distribution Packages you have taken and your volumes. If you have MDIs that are not in the new CareAware or iBus framework, log a Service Request to verify if your code level can have the MDI servers running on multiple application nodes or if you can only run a single instance on a single node. Review the server guide for the MDI servers, looking for the section titled "Considerations for Multiple-Node and Multiple-Instance Usage." This will let you know if you can have more than a single instance and if you can run the server on multiple nodes. I want to emphasize that if you have an MDI server, like SCP Entry ID 230 (MDI Genlab Result Client) and you can run multiple instances on multiple nodes, then on the failover node the server will take very little CPU or memory after it is started. It will idle until the failover event.

Prognosis: A little preparation work can dramatically ease interface and MDI failovers and make your night, weekend and holiday events much shorter and quieter.

Use Oracle Tables to Kill Blocking Sessions

By David Hallberg

Cerner Millennium^® clients I talk with have lots of questions about Oracle. Today I’ll talk about how you can improve system throughput with the information in Oracle’s v$session table or gv$session table for the production database or any database in an Oracle Real Application Cluster (RAC). This Oracle table keeps track of all of the connections (or sessions) on the database. The table shows where the connection came from, the Millennium application node where executables like CPM Script run, the status of the session, the logon time, the blocking status (when another session is preventing or blocking the work from being completed), and the event or description of what the session is doing.

Why is this information important? This one table can tell you about any long-running script — either the cause of the problem or whether the script is just taking the resources it needs to accomplish its work. Problem scripts mean clinicians and business staff are not able to place orders or see results, so these issues need to be addressed.

Let me begin with a couple of cautions. First, with more than 70 columns, the table is quite wide, which can make viewing the data in a telnet, ssh, SQL*Plus session or CCL session quite difficult. I strongly recommend using Excel or another spreadsheet to save the query output and evaluate the data. Additionally, only use this set of queries if you are having a problem getting data into or out of the database. It is completely normal to have transient or sporadic blocked sessions. They provide a safety mechanism for Oracle to maintain data integrity.

The script analysis begins by finding the blocked sessions. Please keep in mind as you execute the following commands that SQL*Plus requires a semicolon (;) at the end of each command and CCL requires the characters rdb and a space at the front of the command as well as a space and go at the end. For the first command, type:

select * from gv$session where blocking_session is not null

If nothing is returned, then there are no blocked sessions. If you do get rows of data back, then you have found the sessions that are compromising Millennium’s performance. To manage the data, you can reduce the number of columns to only retrieve the entries you need. Modify the query as follows:

select inst_id, sid, serial#, osuser, process, machine, terminal, program, type, logon_time, current_queue_duration, blocking_session_status, blocking_session, event, wait_time, seconds_in_wait from gv$session where blocking_session is not null

Once the data is returned, you can use the sid column entry to find the problem processes. Type the following query:

select * from gv$session where sid = <sid number>

From the data returned, you can use the next command to kill the offending session. You need the ALTER SYSTEM privilege for Oracle to perform this function as well as the sid, serial # and inst_id values from the query. Execute the following:

alter system kill session ‘<sid>, <serial#,@<inst_id>’ IMMEDIATE

After you run the command, the session will either be completely removed or will be marked as SNIPED until the Millennium executable or server tries to use the session again. A SNIPED status will not inhibit Millennium process, so either result puts you in much better shape.

Prognosis: Using Oracle’s v$session table can allow you to quickly find problem processes that are blocking other processes and keeping them from completing their work. Unfinished processes mean users are not able to place orders or see results.

A Time to Use Messages.Suppress

By David Hallberg

If you are a regular reader, you know that over the last couple of years I have strongly advocated minimizing the use of Cerner Millennium’s messages.suppress file. Functionality that was intended to prevent extraneous events from cluttering up your message logs has been horribly misused by Millennium system administrators and the application support and application development staff. As a result, vital events are not being sent to the message logs, events that need appropriate troubleshooting and longer term problem resolution. (See “When No News Is Bad News” and “File Can Hide Serious Issues.”)

In recent months, however, I have seen some improvements in Millennium’s base tuning and configuration when a particular event is suppressed, so today I have an unexpected recommendation.

The improvement involves an event called SEC_seclibAudit, which is an Audit-level event telling you that debug is not enabled for the security components. A couple of years ago, only CPM Code Cache Manager seemed to log this event. With the 2010.x code, however, SEC_seclibAudit appears to be written by all of the Millennium servers. It generates two rows in the Data column of the message log each time it occurs. The first row says:

[SecDebugLog::Instance] Creating debug logging singleton.

The second row says:

[SecDebugLog::CheckConfiguration] Debug logging not enabled for all security components, componentLogicalName (SecDebugLogsecrtl)

This is a good setting, but you don’t need to be told about it with every executable. It would be better if you could assume the setting is off unless you took action to turn it on. (To enable debug logging, set the LogLevel of a server to 4 or set the environment variable or logical of CPM _GLOBAL_LOGGING or any one of a number of other logging increases that are available to individual Millennium servers.) This message has become so prevalent, logging on all servers, it has become noise. One client, who cycles 19,000 servers each week, gets 38,000 messages every week for this one event.

To stop the noise, I have to make a recommendation I rarely make: Put this event in the $cer_mgr directory’s messages.suppress file. Then cycle all of your multi-instance servers to make the change take effect.

Prognosis: Using the messages.suppress file appropriately stops extraneous logging and keeps your message logs clearer for real problems.

Make Sure Servers Write Events to Message Logs

By David Hallberg

A client wanting to determine who had cycled a server in Millennium’s Server Control Panel (SCP) at a certain time went to the SCP’s message log cmb_0033 for the answer. When they opened the file, what did they find? Absolutely nothing. No entries or rows had been written to the message log since their last code upgrade. All of the stability issues and change control events that should have been logged were lost. The problem was much bigger than not knowing who had cycled a server. The organization had no way of knowing what else was wrong or how to resolve those issues.

We checked out three parts of the system to track down the problem.

1. Permissions settings in the $cer_log directory. The owners of the files in the $cer_log directory had been set to root for the User and root for the Group. That setting works great much of the time, but not for the following message logs: cmb_0000.mlg, cmb_0033.mlg, cmb_0057.mlg, cmb_0066.mlg and cmb_0078.mlg. For these, the User setting should be root and the Group setting should be your domain user name (for instance, d_prod or d_pxxx, with the xxx being your Millennium client number if you are remote hosted). Then the permissions on the file can be -rw-rw----, which allows the user of root as well as anyone in the group of d_prod to read and write to these log files. If you set these permissions properly, you will have lots of additional information about performance and stability issues in your Millennium domain.
2. SCP settings. SCP servers should have a User Name set to root and a Group Name set to your domain user (for example, d_prod). You also need to set the SCP Property of LogLevel to 2 on the following servers: 0 (Service Manager), 33 (Server Control Panel), 66 (CPM HNAM Agent) and 78 (CPM Script Install). The LogLevel for SCP Entry ID 57 (CPM Script Batch Priv) should be set to 1, unless you are actively troubleshooting an issue with this server.
3. The messages.suppress file in the $cer_mgr directory. If you cat messages.suppress, you will see all of the messages that are not being written to the message log. You are missing significant information if any rows in this file start with the following characters: CMB_, HNAM_, SCP_, InstallOCD, OCDPURGE and Script.

Prognosis: If you check and correct your message log settings before and after an upgrade, you will have much more insight into your Millennium domain and will be better equipped to troubleshoot stability problems and monitor change control.

Easy Tuning of Oracle Sequences Improves Throughput

By David Hallberg

In more than 13 years of helping clients tune their Cerner Millennium^® systems, I have only recently turned my analysis to Oracle sequences. I have the IT team for a PowerChart client to thank for my newfound understanding of how sequence numbers can increase the variability in system response times…and drive clinicians crazy.

After engaging me to diagnose their PowerChart performance problems, the team corrected several MQ queuing issues and other straightforward problems, but clinicians were still experiencing slowdowns during their peak usage times. The culprit turned out to be the Oracle sequence numbers.

Oracle sequences give an identifying number to certain elements in the database. Millennium has about 964 separate sequences, Oracle’s System and Sys have about 20 and 66 sequences, respectively, DBSNMP has two, and CMS_DBA and Panther_Read have one each. Oracle generates these sequences when new data or rows are added to the database. Sequences become a performance problem when you have a lot of new data that needs sequencing but the tables aren’t configured to adequately handle the load. Making the change is surprisingly simple.

Here’s what I discovered for my client. I took a 1-minute and a 30-minute sampling of the Oracle sequence numbers being generated by PowerChart. The following chart shows the results of the 1-minute interval: 

SEQUENCE_OWNER	SEQUENCE_NAME	CACHE_SIZE	Sequences Generated
V500	CHARGE_EVENT_SEQ	2,000	8,000
V500	OCF_SEQ	2,000	6,000
V500	CLINICAL_EVENT_SEQ	2,000	4,000
SYS	IDGEN1$	20	2,000
V500	PATHNET_RECOVERY_SEQ	2,000	2,000
V500	PERSON_SEQ	2,000	2,000
V500	LONG_DATA_SEQ	2,000	2,000
V500	CPMAPP_APPLCTX	20	1,320
V500	CPMAPP_APP_CTX_ID	20	960
V500	CHART_DIST_LOG_SEQ	20	40
V500	ENTITY_LOCK_SEQ	20	20
PANTHER_READ	AUD_ORA_ERROR_SEQ	0	7
SYS	ORA_TQ_BASE$	0	2

The key columns are Cache_Size compared to Sequences Generated. The Cache_Size shows how many sequence numbers can be generated before updating the high water number, which is the highest sequence number ever created for that sequence name. It takes time for Oracle to update this number, so you don’t want the system to have to do it too often. Increasing the Cache_Size decreases the number of times the high water value needs to be updated and, therefore, improves the throughput for the tables that use that sequence number.

Let’s look at an example in the chart. The Sequence_Name of Charge_Event_Seq generated 8,000 sequences in 1 minute. Because its Cache_Size was only 2,000, Oracle had to update the high water value four times during that minute. The Sequence_Name of Idgen1$ generated far fewer sequences, only 2,000, but its Cache_Size of 20 meant that Oracle had to update its high water value 100 times in this same minute. You can reduce this overhead associated with sequences — and improve the throughput for their corresponding tables — by increasing the Cache_Size to match or, preferably, exceed the number of sequences generated during your time frame. I recommended my client set Charge_Event_Seq to a Cache_Size of 9,000 and Idgen1$ to 2,000.

The change can be done in uptime without taking any action on the Millennium application nodes and without doing anything else on the Oracle nodes. The change is accomplished via SQL*Plus with this command:

       Alter sequence charge_event_seq cache 8000;

In CCL, use the following command:

       rdb alter sequence charge_event_seq cache 8000 go

The following chart shows the results from my 30-minute interval, which are a little more staggering in terms of the impact that high-water-value generation has on overall throughput. It also shows that only 35 sequences — about 3 percent of the more than 1,050 — had increases in their sequence numbers during this time (32 of 964 for V500 or Millennium and 2 of 66 for Sys).

SEQUENCE_OWNER	SEQUENCE_NAME	CACHE_SIZE	Sequences Generated
V500	CHARGE_EVENT_SEQ	2,000	106,000
V500	OCF_SEQ	2,000	76,000
V500	CLINICAL_EVENT_SEQ	2,000	48,000
SYS	IDGEN1$	20	40,000
V500	CHART_VIEW_SEQ	20	24,260
V500	CPMAPP_APPLCTX	20	22,660
V500	CQM_TRIGGER_ID_SEQ	2,000	18,000
V500	CPMAPP_APP_CTX_ID	20	16,160
V500	CQM_QUEUE_ID_SEQ	2,000	16,000
V500	ORDER_SEQ	2,000	14,000
V500	PATHNET_RECOVERY_SEQ	2,000	14,000
V500	CARENET_SEQ	2,000	8,000
V500	PATHNET_SEQ	2,000	6,000
V500	PERSON_SEQ	2,000	6,000
V500	DM_CLINICAL_SEQ	2,000	4,000
V500	EKS_DLG_EVENT_SEQ	2,000	4,000
V500	ENCOUNTER_SEQ	2,000	4,000
V500	SCH_CONVERSATION_SEQ	2,000	4,000
V500	AAR_REPORT_SEQ	2,000	2,000
V500	CHART_REQUEST_EVENT_SEQ	2,000	2,000
V500	CHART_SEQ	2,000	2,000
V500	ENCOUNTER_ONLY_SEQ	2,000	2,000
V500	ENTITY_LOCKKEYID	2,000	2,000
V500	ENTITY_RELTN_SEQ	2,000	2,000
V500	ESI_LOG_SEQ	2,000	2,000
V500	MEDADMIN_SEQ	2,000	2,000
V500	NETTING_SEQ	2,000	2,000
V500	OMF_SEQ	2,000	2,000
V500	PHARM_FILL_HX_SEQ	2,000	2,000
V500	REFERENCE_SEQ	2,000	2,000
V500	RESULT_SET_SEQ	2,000	2,000
V500	ENTITY_LOCK_SEQ	20	400
V500	CHART_DIST_LOG_SEQ	20	300
PANTHER_READ	AUD_ORA_ERROR_SEQ	0	125
SYS	ORA_TQ_BASE$	0	4

So I recommend only changing sequence numbers that are being used a lot. In this case, I recommended:

• Charge_Event_Seq and OCF_Seq have Cache_Size set to 10,000
• All V500 sequences with more than 2,000 sequences generated have Cache_Size set to 8,000
• All V500 sequences that match their Cache_Size be left at 2,000, at least initially
• All V500 sequences with a Cache_Size less than 2,000 be increased to 2,000
• Sys Idgen1$ have Cache_Size set to 10,000
• Sys Ora_Tq_Base$ have Cache_Size set to 20
• Panther_Read have Cache_Size set to 2,000

It is possible to make the Cache_Size too large, so my recommendations are fairly conservative. Still, the client will have much less variability in Millennium response times after implementing these straightforward fixes.

Prognosis: Keeping your Oracle sequence Cache_Size properly set is a simple, uptime tuning procedure that decreases performance-robbing events and variability in your Millennium environment.

Messages.suppress File Can Hide Serious Issues

By David Hallberg

Historically, when I was brought into situations where a problem within the EMR system was difficult to track down, I used the message logs quite extensively. They helped me understand what was happening and where the troubleshooting should begin, and they explained any application-specific problems. This method worked very effectively until Millennium^® 2007.18 TP4, when a new file in the $cer_mgr directory — the messages.suppress file — started shipping with some troubling exclusions, namely ScriptError and  ScriptErrMsg events.

The messages.suppress file stops Millennium from writing events to the message logs. This file was designed to stop creating message log events when an engineer accidentally sent an event to clients with an incorrectly set LogLevel. It’s a legitimate place for extraneous events that would otherwise clutter your log files and potentially overwrite messages that need attention. For example, the event CRM_ClientTimeZone writes the current time zone to the message log. Since domains rarely change time zones, the event is seldom of benefit when troubleshooting an issue. Yet it can cause a lot of application node writes, since every transaction going through the application server with this set has to write a message log row into the system message log as well as the specific server message log. To reduce the application node’s disk I/O and performance variability, it’s good to have CRM_ClientTimeZone in the messages.suppress file and exclude these events from the message logs.

If you hide the wrong events, however, you miss out on important information. A problem, failure or issue that should be written to the message log goes undocumented, and your clinical or technical support team has no record to assist them in identifying and resolving issues.

So it makes no sense to me that with the initial TP4, ScriptError and ScriptErrMsg events were included in the messages.suppress exclusions and that the file has since expanded to include ScriptErrors-Fatal and ScriptErrors-NonFatal events. You need visibility into these events. They indicate that an application or Oracle has a problem (often a code or build issue) and needs help to fix it.

What might the problem be? ScriptError events include blood bank transfusion script errors, MDI download problems, scripts failing to run (such as those getting PharmNet frequencies) and Oracle deadlock issues. These events all have a negative outcome to the clinicians using Millennium and the business office needing to bill accurately for services delivered.

ScriptErrors-Fatal events include memory problems that prevent clinical information from being updated and from being viewed by the clinicians who need this information.

ScriptErrors-NonFatal events show a wide range of issues. Expert Knowledge System (EKS) compile errors, Visual Explorer errors, SurgiNet documentation errors, getting orders asynchronously for PowerChart, Lights On Network data-gathering script errors and sequence number problems are all events that can degrade Millennium performance or even cause a failure. These errors must be logged so that problems can be seen and corrected.

Prognosis: Keeping your messages.suppress file properly set will ensure that the Millennium message logs contain the information that your support team needs to identify and resolve issues in a timely manner.

New Code and Lights On Network Compliance Tabs

By David Hallberg

Several Millennium^® users I’ve talked with since the first of the year have had problems in production after taking new service packages. All had the same question for me: “How could we have anticipated the problems we now have in production?” Because some of the problems involved a new solution being brought up, I had a question for them: “What is your compliance percentage on the System Compliance and Solution Compliance tabs of the Lights On Network?” Either they didn’t know what I was talking about or they only had the tabs enabled for production. In their ignorance, they were missing out on key data from Cerner and its clients that could have helped them prevent the issues they were experiencing.

New code undergoes several quality checks before Cerner releases it to its client base. However, internal testing can never be rigorous enough to anticipate the varied and complicated production environment. So several years ago the Lights On Network was set up to allow clients to see several aspects of their configuration. The two most useful, from my perspective, are the System Compliance and Solution Compliance tabs.

The System Compliance has four dials: Known Errors, Oracle Configuration, OS Configuration and HNAM Configuration.

• Known Errors shows quality issues, sometimes called CIRB, as well as CernerWorks Process Improvement (CWXPI) corrections. Essentially, when one or more clients have a problem in a production environment or a potential for a problem in production, notifications are sent out to alert all clients of potential issues (CIRB). The CWXPI items are problems found at sites that Cerner manages or hosts, and to improve all Millennium environments, Cerner publishes how its teams corrected these issues.
• Oracle Configuration shows clients what several basic Oracle configurations should be when they first go live. There may be reasons your configuration differs from the recommendations. These reasons or differences can be documented in the Inactivate selection from this screen, an option available on all four of these dials.
• OS Configuration shows what basic Operating System configurations should be. It’s a very good baseline. If you deviate from it, you need to be certain why.
• HNAM Configuration shows more CWXPI items, basic logging settings and some solution/application configurations. Generally, this tab focuses on technical aspects of the Millennium environments.

The application side of the fence is found in the Solution Compliance tab. This tab shows topics such as what Ops Jobs (Millennium batch scheduling) need to be run and recommendations on how to configure Ops Jobs if they are not being run. The tab provides feedback on a subset of the Millennium purges. These purges delete temporary data that is not considered part of the permanent patient record. It helps control the size of the Oracle database and can improve or stabilize performance.

Prognosis: Once you and your organization are aware of the quality checks available on the System Compliance and Solution Compliance tabs of Cerner’s Lights On Network, you can put processes in place to leverage this service so that you reduce problems in your Millennium domains and improve the effectiveness of your infrastructure.

Betrayed by an Old Friend

By David Hallberg

I had a client complain to me recently that I was reporting a false positive, an alert that was triggered even though there was, in fact, no problem or issue with his Millennium^® EMR system. I decided to figure out why there was a discrepancy between my measurements and the client’s quality-assurance tool, which happened to be sysmon.

Let me begin by saying that I love sysmon. Before sysmon was developed in the late 1990s, Millennium had so many tools or utilities that provided a single view of a piece of information that it was difficult to keep track of all of them. At one point there were more than 200 build or monitoring tools even though Millennium only had about 40 actual application executables that were used in various configurations for its 60 plus solutions. Multi-node clients kept asking their Millennium middleware team to figure out where performance bottlenecks were occurring. Believe me, this sleuthing was extremely labor-intensive. Users had to have mon_ss (a Millennium monitoring tool used to watch for proprietary middleware queuing) and joumon (a Millennium monitoring tool used to watch for persistent queuing in the proprietary middleware) up in a different window for each application node. Then they would have to try to show someone else when one or more of these screens had a “blip” or negative performance indicator. It was painful on the good days.

So those managing Millennium were understandably ecstatic when Cerner developed a single utility that told what was happening across Millennium’s persistent and non-persistent queuing mechanisms across all application nodes, in a single screen. Sysmon also provided the date and time when something bad happened.

Sysmon proved to be reliable, and I became a proselytizer for this great monitoring tool. While I was at Cerner, I taught many clients how to use sysmon and its various switches. I also used it to tune a four-node client with ConnectionBias (a load-balancing mechanism for Millennium), moving the transaction workload from one application node to another. I was able to show enough value with this one tool that some clients actually chose not to purchase BMC Patrol (later called BMC Performance Manager and now called BMC ProactiveNet Performance Management). Sysmon had some limitations — it was really difficult and unreliable to pipe or move the output to a file for additional processing, and it had no mechanism to redirect its output to a common data format like CSV (comma-separated values), tab-delimited or regular flat files — yet it was the most accurate view of a multi-node application configuration.

So why was my data conflicting with sysmon? It appears that with 2007.02 some of the newer Millennium services do not register with the old middleware in a way that allows sysmon to see the queuing. With the client I referred to earlier, I found that the SenSage add-on called P2Sentinel did not publish its queuing in a way that sysmon could see it. The toolset I used, however, did see the problem.

That’s not the only problem. Another client reported that one of the Java services introduced with 2007.02 was crashing several hundred times a day. I ran sysmon to see if it would show these crashes, and it did not. I am not exaggerating to say that I was crushed. Sysmon was the one tool I absolutely trusted to let me know in a timely manner when Millennium was failing, typically 15-45 minutes before BMC. Now I find it is not reliable!

If, as it appears, the newer programs will not share or publish their information like the old programs, Millennium clients now must find other tools. None of the options, however, shows them this information in a single view. Sadly, we all have to go back to running multiple windows for each application node when we are using command-line tools.

A source of truth that you can trust and believe in is difficult to find. It is even more difficult to see it no longer be a source of truth.

Putting the Profit in ProFit

By David Hallberg

I’ve written previously about lost charges and how to examine Millennium’s message logs to find and correct billing issues. Today I want to speak specifically to clients who use ProFit^® Patient Accounting, Millennium’s billing solution. Those wanting to maximize their revenue will find a wealth of information in the message logs on the ProFit servers.

One of ProFit’s components is a third-party subsystem called Sablotron, an XML toolkit that was used to make ProFit easier to support across operating systems (http://freshmeat.net/projects/sablotron). Because it is XML based, Sablotron is independent of the operating system and works the same on AIX, HP-UX, Linux and VMS. This is great from an engineering support perspective. But what does it mean to a healthcare organization?

ProFit has two primary servers: ProFit Transaction Server (SCP Entry ID 475) and ProFit Billing Server (SCP Entry ID 476). Generally, an organization can only control the memory size and loglevel of these asynchronous processing servers, and interactive users typically do not know when these servers (executables) have had any errors or warnings. Someone would have to go digging for these issues to become aware of them. I recommend you get out your shovels.

I recently ran across two events on the Billing Server of a ProFit client. To correct the first issue, we needed to log a service request to get help in correcting the build. For the second, we needed to log a service request to get new code.

The first event was the Sablotron Message, which indicated that the subsystem could not properly handle some data in a billing transaction. In a single four-hour time frame, my client had 4,002 of these events. The Data component of these records had a consistent beginning section that looked like this:

{{msgtype::warning}}{{code::93}}{{module::Sablotron}}

It was followed by one of three ending components.

1.  The most numerous Sablotron Message ending was:

{{URI::arg:/_parsed_}}{{line::544 }}{{node::element '<xsl:for-each>'}}{{msg::unsupported language 'en'}}{{Routine::sablotronErrorHandler::log}}

You’re probably thinking, “This looks like someone sat down at the keyboard after drinking a bit too much.” But the message actually illustrates how many XML files are laid out or formatted. XML is just another way to show data, and the message tells us that in line 544 of the server, there was a language code of “en” that did not have a translation (‘<xsl:for-each>’) and the server didn’t know what to do with it. So the server wrote the message to its message log.

2.  The second most numerous Sablotron Message ending was:

{{URI::arg:/_parsed_}}{{line:: 497}}{{node::element '<xsl:for-each>'}}{{msg::unsupported language 'en'}}{{Routine::sablotronErrorHandler::log}}

This looks like the same drinker was at the keyboard. The message is the same as the previous one, except that the unsupported language code of “en” was in line 497.

3.  The third Sablotron Message ending was:

{{module::Sablotron}}{{msg::unsupported language 'en'}}{{Routine::sablotronErrorHandler::log}}

This message appears to be a higher level failure, but the end result is the same. The server had a language code of “en” that it could not translate, so it wrote the message to its message log.

The second event my ProFit client experienced with the Billing Server was SRV_LookupFailure, which told us in the Data that the bill_item_id was missing from the transaction. This would generally not be a problem, except that in this case the user name was SYSTEM and, therefore, no interactive user was made aware of these failures. They were only in the message log.

Not all the people I work with think these messages are a big deal — they see errors in Millennium as normal. I disagree. The client I’ve referenced here had, on a single node, 3,038 of the line-544 Sablotron Message events and 904 of the line-497 Sablotron Message events in two hours as well as 60 higher level Sablotron Message events and 92 SRV_LookupFailure events in one hour. How could this be considered normal? If an executable is having a problem, you want to correct it, whether the issue is in the build, the code or the use of the code.

Prognosis: Little events can add up to big revenue losses. Looking for these events and resolving the code or workflow issues will maximize your revenue.

It’s Just a Fax, Right? Part 3

Troubleshooting your RRD application

By Chris Lanaman

For the past two blogs, we have looked at the important role faxes play in relaying critical information from the hospital to doctors’ offices and how a reliable fax service can improve relationships and business with these offices. Today we turn our attention to making sure Millennium’s Remote Report Distribution (RRD) servers, which send the faxes, are operating at peak performance.

It’s not always an easy task — which is why in my first blog on this subject I recommended that management dedicate at least one full-time analyst to support faxing. Gone are the days that you walk over to the fax machine, set your forms on the paper feeder, type in the phone number, hit a button and away it goes. Now many processes typically occur before a document reaches the RRD queue and is ready to be made into the fax and sent on to the primary care physician.

When training new analysts over the past few years, I often taught them the Cerner RRD application first, before graduating them to the Clinical Reporting applications. This method might seem backward, but I found it to be the most effective. With most things in the IT world, the best way to learn how to fix things is to start at the outcome and then work your way backward. If your end-users are saying they didn’t get the fax, you check the RRD server.

RRD technology has not changed much in recent years. Because of this, RRD servers can quickly become overwhelmed when a hefty number of faxes are pushed through them. If you’re experience faxing delays or failures, I recommend you check the following:

1. Operating system. Imagine sitting at your computer all day and faxing a Word document over and over, all day and night, through the computer’s modem. At some point, you would become exhausted and would need a break to regroup and refresh. Similarly, the RRD application needs to be refreshed. Small things like clearing the memory (cache) and straightening (defragging) the files are often forgotten but can help speed up the process.
2. Fax volume. It’s important to know the number of faxes you are sending each day. If you’re sending more than your RRD servers can deliver in a timely manner, you might need to beef up your RRD servers. Remember, as I stated in an earlier blog, delays in fax deliveries can be very damaging to your organization’s relationships with referring physicians.
3. RRD services. Make sure the RRD services are running on your RRD servers. Without these running, you simply have a desktop computer not doing anything.
4. Phone lines. Another area that often gets overlooked is whether the phone lines are still active. Troubleshooting might be as simple as making sure you have a dial tone on the lines going into your RRD servers.
5. Modems. Check to see if your modem is hung and not releasing your transmissions. If you use Softek Panther, make sure the RRD: Stalled sensor is set and then check for any Error messages. These messages might indicate a problem with your modem, not your RRD system.
6. Obsolescence. Sometimes customers have fax machines that are so old that the Smithsonian is looking to add them to its collection. Ask your customers for the make and model of their fax machine, and then check to see if your current RRD application level supports that model. Keep in mind that if you are not using internal modems on the RRD server, and instead have serial modems attached to the RRD server, those serial modems might be the issue as well. Sometimes a simple reboot fixes the problem.

As you can see, Cerner's RRD application around faxing can get hung up in a lot of areas. That’s one reason why tracking faxes for a healthcare organization is a full-time job. The next time someone tells you, “It’s just a fax,” remember you have your organization’s reputation – as well as a lot of money - riding on the fax getting where it needs to go.

Prognosis: Troubleshooting Millennium’s RRD application is much easier when a full-time member of the IT team can own the maintenance and support of this important faxing service.

 

Keys to Unlocking a Server

By David Hallberg

Occasionally I have clients who are unable to start new instances of Millennium^® application servers or cycle their servers. Let’s walk through the steps to troubleshoot this problem.

The first place to look is the ADMIN.REPLY queue. If the queue is full, no new Millennium servers can be started.

The next place to check is the operating system’s semaphores that Millennium is using. The executable called semdump, which is used on the Unix platforms, AIX, HP-UX and Linux, shows you on each application node if there are any locks on any of the domain semaphores. Access the data by typing the command:

./semdump –v

Group [x] semaphores

Assuming you have a LIBPATH statement where $cer_exe is or in the directory of $cer_exe, you will see a table that looks something like the following (columns have been removed for space reasons):

Key	ID	VAL	Waiters	PID	Name
00000009	35	1	0125/0000	1492	ipc-key-table
abcd0009	273	90011	0000/0000	32768	MLGcmb_0000
efab0009	37	90011	0000/0000	1492	Group
cdef0009	282	90011	0000/0000	19025	code_cache
abc10009	76	90011	0000/0000	32768	MLGsystem
12340009	284	90011	0000/0000	19025	MLGempi_cachertl
56780009	283	1	0000/0000	19025	g_code_cache
9abc0009	279	90011	0000/0000	19025	MLGmaillog
def00009	280	90011	0000/0000	19025	code_cache_stats
a1230009	36	90011	0000/0000	7711	logical-table-index
b1230009	99085	90011	0000/0000	80133	sps_table
c1230009	302	1	0000/0000	80133	g_sps_table
d1230009	281	1	0000/0000	19025	g_code_cache_stat

Look at the Waiters column, which contains four digits followed by a slash (/) and four more digits. The first row in the table, Entry ID 35 IPC Key Table, indicates a persistent lock on that resource. In this case, it stopped the ability of new Millennium servers. It can also indicate why an application is hanging or frozen.

To break the lock, execute the following command:

ps –ef | grep 1492

The command could return more than a single row or line. You want to look for the second column being an exact match to the PID number. Information on server 1492 from the example above appears below:

 d_cert 1492  1 0 75:19:36 ?  0:15  cer_exe/cpm_srvcachemanager CERT * * 1200

 i_cert 415500 25210  1 11:17:24 pts/td  0:00 grep 1492

If you still have access to SCPVIEW from Citrix or a fat client or access to Servers from Panther, you want to stop that server gracefully. If you cannot do it from these tools, you need to have root access and perform a kill on that PID. (Please do not do a kill -9 as this can cause other problems on the application node.)

The last thing to do is to restart the server that had to be stopped so that you are running the appropriate number of instances for your configuration.

Prognosis: When you are unable to start a new server or cycle an existing server, a few simple commands can find and resolve the problem.

Let’s Put Clothes on the Emperor

 By David Hallberg

When clients are looking for improved Millennium® performance – which they almost always are – our conversations often go like this:

Client: “Can you please help me parse and manage the RTMS timer files?”

Me: “Certainly. But I thought we wanted to improve Millennium’s performance for your clinicians and business staff. The RTMS timers will do nothing to assist in resolving performance for the point of care people and the business people utilizing Millennium.”

Client (with great indignation): “What are you talking about? The RTMS timers are THE performance metric for Millennium.”

With a name that stands for Real Time Monitoring System, you would assume that statement to be true. Its truth, however, depends on how you define the word performance. To clinicians and the business staff, performance relates to how quickly they can get their job accomplished. They are concerned with Millennium’s speed. The data center staff, especially those at hosting vendors, define performance as uptime. They are concerned with the amount of time that the backend systems are available. Some may boast that they offer a performance guarantee of 99.8%. Such assurances mean nothing to clinicians, who assume that the system should always be up. Their standard is a system that runs, in the words of a friend of mine, at blink speed.  

And while RTMS timers once did measure speed, Millennium architecture changes have altered the meaning of the data the timers deliver. The result is that end-users might complain that the system is slow but the metrics look fine inside the data center. A bit of history will help explain the discrepancies.

Millennium did not originally have RTMS timers in its code. Instead, the middleware had a setting to allow the Citrix server or fat client to log all of the transactions being sent from that device to the application node. In the days when the applications and database resided on the same node, this log was a great metric. However, the setting had some drawbacks. There would be an observable performance degradation in Millennium applications with this logging turned on (degradation of 10 percent or more was not unheard of), and the approach did not show any grouping of applications and functions. So it was a great tool for the middleware team but not so helpful for the application engineering teams.

A grouping feature was eventually deployed. Initially, there was a high degree of linking between the functions being measured between the Citrix server or fat client and the application node, which proved helpful in seeing backend performance issues. Over time, though, we have seen that Windows tab changes and other functions actually only measure how fast the Citrix server or fat client hardware is running rather than how long it takes the application and database nodes to process a transaction for the end-user. I call this “RTMS timer death.” You now have lots of functions being timed, but you can no longer be assured that the function being measured is actually providing a roundtrip view of the transaction time.

Several other changes are also obscuring the quality of the RTMS timer data.

1.    A greater and greater reliance on asynchronous work leads the interactive user to believe the system is more responsive, but it is in fact a bit of a performance cheat. For instance, I may do a function in PowerChart in which a request is sent to CPM Script and CPM Script responds when the function is complete. In fact, even though CPM Script is done with the function, CPM Script actually sent the request to CPM Process and CPM Process exploded my request into numerous other requests that it then handed to other Millennium executables.

2.    If the Citrix or fat client application (e.g., PowerChart) crashes or is X’d out, you will see a DBAPI_Exception event for the Millennium application executables that were processing the requests from this client. The Data or Record information starts with “DbDestroyTimer: invalid object handle: xxx,” where xxx is the handle being used to update the RTMS timer data. What does this mean? It’s incomplete data. It’s not clear that each of these events represents an RTMS timer that started but never finished, nor is it intuitive that there is a relevant impact, much less intervention, to consider.

3.    Two files in $cer_mgr – slareporting.cerner and timer.cerner – control whether a timer is active, what functions are used, if the timer meets the hit/miss count, and the timer’s average or mean time. Though it wasn’t originally the case, the CRM Timer information is now configured in the timer.cerner file. In the slareporting.cerner file, the 54 application timers are defined. In the timer.cerner file, the 1,022 functions are defined. This would seem pretty good – until you remember that there are more than 35 million lines of code for Millennium, more than 4GB of Citrix and fat client files, and more than 5,000 Oracle tables. In the slareporting.cerner file, by definition, you have to have each of the 54 applications run 500 times in two hours in order for the data to be counted. You also have to have each application run at least 10,000 times a month or the data does not count.

Let me use PowerChart to explain the potential for problems. Let’s assume that it’s 7 a.m. and physicians are starting to round, nurses are changing shifts and many other clinicians are getting down to their day’s work. In all, 499 clinicians log into PowerChart from 7-9 a.m. The IT department gets reports from these users that Millennium is slow. IT checks the slareporting.cerner file and sees nothing counted for the day’s performance. Why? The slareporting.cerner file must have a minimum of 500 functions of an application – like logging into PowerChart – in a two-hour window, or they do not count. Over the course of a month, if your site did 9,999 PowerChart logins, it would be stated that you did not do enough work to be statistically significant. For the purposes of a Lights On Network™ evalution, this data would not be considered substantial enough for valid comparison. Therefore, if you logged a Service Request stating you are having performance issues, you are likely to be told that based on the data you do not have any issues.

The initial intent of the RTMS timers was right on the money: to create a mechanism to allow Millennium sites to look at a clinical function and understand where the issue is. However, the execution and changes over time have made the timers unusable. Every client who I have worked with in processing their RTMS data has logged Service Requests and been told either 1) there is no problem or 2) a code level update will make the problem go away.

This approach to analytics showcases the different definitions of performance that I mentioned earlier. The clinical and business staffs are telling you that Millennium is slow and unpredictable, but the data shows no performance issues. This conundrum with the impenetrability of the data leads many organizations to conclude that the environment is unmanageable and they need to outsource the support for Millennium Citrix servers, application nodes, Oracle databases, etc. After all, the outsource vendors guarantee “performance” to be 99.8 percent.

I’ve already discussed how meaningless that statistic is to everyone outside the data center. A 99.8 percent uptime from the clinicians’ perspective makes no claims to speed and probably means that the .2 percent downtime will occur when they need the system the most. Here’s one more caveat: The statistic refers only to unplanned downtime. The two to four hours of scheduled downtime for weekly preventative maintenance are not included in the number. I have been in healthcare IT since 1989. I have been employed by four different hospital systems, have been an HIT consultant and have worked for Cerner, and I have never been allowed to have that much downtime in a month, let alone a week.

Nor was I allowed to ignore the clinicians’ desire for blink speed. When I did performance benchmarking at a facility running a different clinical system, we had the system tuned so we could do the screen changes in 0.6 seconds. Seriously. On one occasion we had a drive problem that spiked the time to 0.9 seconds. The screen flips were 0.3 seconds slower, about the time it takes to blink your eyes, and our help desk phones were on fire from the outraged clinicians complaining of performance problems.

So what can you do to help resolve your performance problems? The first step is probably the hardest. Stop believing the propaganda that your system is fine. Anyone telling you that you don’t have a performance issue that can be tuned is like the “tailor” telling the emperor that his new clothes are exquisite. They’re trying to convince you of something your eyes and brain know is not true. Listen instead to the people who interactively use your system every day. The clinicians and business staff will be the first to notice a difference in performance. The next step is trying to understand what you can do to monitor the performance of all your Millennium systems. The last step is to determine what staff and budget changes you need to make to address the issues.

There is a lot you can do with the tools you already have. You’ll find lots of tips in previous blog postings. I also encourage you to respond to this blog with your own questions and insights. I welcome a lively dialogue.

Another Upgrade Issue: The TDB

By David Hallberg

Although I have blogged before on issues that might be overlooked during a Millennium^® upgrade, a recent discussion with a client revealed another issue: one with the transaction database (TDB). A common oversight in this area might need a simple fix.

The TDB is a flat file found on each Millennium application node that tells the executables (application servers) what transactions they can process and if the transactions are Request/Reply (RR), Reliable Delivery Message (RDM) or Remote Procedure Call (RPC). Let me start with some background on these three types:

· With RR transactions, the clinical information remains intact even if the transactions are lost or discarded. For instance, using CPM Script to retrieve or read data from the database is an RR transaction. If the transaction is lost, the person asking for the information just has to ask for the information again.

· An RDM transaction is like an order. If the client crashes or the application executable or application node fails, there has to be a mechanism in place to recover or replay that transaction when the system is fully functional again.

· The RPC transaction was added with Millennium 2007.x. It allows the Java servers and Java client applications to process transactions originating from both the local hardware and remote calls. It was the first extension to the Common Request Model in quite some time.

Until recently, the TDB was a connection-based service. PowerChart or the front-end application connected directly to the TDB’s port. The TDB is now a shared service. The TDB server communicates with Citrix servers and fat clients over a network port that it allocates when it starts up. Once the communication port is selected or opened, the TDB server tells the Millennium Service Manager on the application node which port it will be listening to for incoming transactions (requests). The Citrix servers and fat clients must be on the same code level as the application node so that when PowerChart or any front-end Millennium application starts up, it will connect to the Service Manager and be told what port to use to connect with the TDB.

I have found that sometimes during an upgrade, the fat clients or outlying Citrix servers are overlooked. In the case of the TDB, fat clients and Citrix servers that aren’t upgraded might be connecting to the wrong port. If a connection fails because the TDB was not listening on that port, the Service Manager logs a message stating that event CMB_ServiceUnavail1Net occurred. The service name (such as TDB_SYSTEM@PROD) is logged, the Citrix server or fat client’s TCP/IP address is included, and the port the client was trying to communicate with is logged. You can see these messages by looking in the cmb_0000 message log and search for the event CMB_ServiceUnavail1Net.

An example would look something like this:

Domain	Name	Node	Record	Cont.	Date
prod	cmb_0000	Hallberg	{{service_name::TDB_SYSTEM@PROD}} {{tcp_address::10.10.10.10}}{{tcp_port::1240}}	N	8/17/2010 6:53

Event	Level	Location	No.	Source	Sourceref	User
CMB_ServiceUnavail1Net	1	hallberg/ 20200cce/8	1022	cb_service_mngr	cb_service_ mngr.cpp:5635	n/a - [d_prod]

The message under the Record column indicates that the Citrix server or fat client whose TCP/IP address is 10.10.10.10 is trying to communicate with the TDB on port 1240. Because this happened after a Millennium upgrade, the client just has to use tracert or traceroute (depending on your operating system) to find out how far the problem server or client is from the application node, i.e., how many network router hops you have to make. The Citrix server or fat client needs to be scanned for components after the Millennium client upgrade in order to be in sync with the newer application node code. This is the most common problem and is simple to correct.

Prognosis: Reviewing message logs after an upgrade can highlight issues that make Millennium appear unusable or unstable to the clinicians whose Millennium servers or fat clients have not been upgraded.

 

Get a Charge out of Failed Events

By David Hallberg

In 1997, when I first began performing technical audits on Millennium® infrastructures, the biggest concern was to keep the Millennium middleware functional and to tune the Oracle kernel. My job was to make sure the interactive user transactions made it to the Millennium application nodes and the database was able to keep up with the workload. These days, however, the MQ generally is working well and the dependency on the Millennium middleware is greatly reduced, so there’s a lot less to tweak in those areas. Likewise, Oracle nodes/lpar/vpars have at least 64 GB of RAM and large system global areas (SGAs) and provide much higher throughput for the Millennium solutions.

Unfortunately, as these issues have improved over the past 12 years, other key problems have surfaced. One trend that I’ve found in the last two years is an increase in lost charges. Analysis of Oracle tables and Millennium log files show that clients are losing charges at two points: in the orders/activity to charge mapping and at inbound interfaces.

Most clients don’t believe me when I highlight their billing mistakes. I find they get rather defensive instead of eager for me to show them how to fix their problems, some of which can be done fairly easily.

If you want to investigate whether your system is losing charges — and your organization is losing revenue — start at the message logs for CS Charging Async and CS Charging Sync. You want to look for IncompleteEvent, AddBillCodeType, AddItem_Failed, CheckCharge, CheckChargeModForType, DBAPI_FatalScriptErr, OffsetCharges and ScriptError. These events represent the problems I have seen most recently. All but DBAPI_FatalScriptErr and ScriptError are build or configuration issues that typically need only a little research to resolve.

The DBAPI issues are often Oracle space or connectivity issues that should be resolved with your database administrator. If the DBAPI errors are constraint violations or lock issues, they are probably code issues that require you to take updated Millennium service packages. The ScriptError events are almost always code issues, also requiring you to take updated Millennium service packages.

Your investigation might show a staggering number of ESI errors and warnings. The ESI issues are often from the Omnicell or Pyxis interface, which could indicate you are losing pharmacy charges. Even if it’s only 1-2% of all pharmacy charges, your organization could be losing thousands of dollars in revenue.

From the ESI logs, the failures are often from a “null person_id.” That means Millennium received a transaction but it couldn’t apply the charge or event because the patient was not identified. Other failures could include a “null meddef” or “null manufactur id.” Both of these are build issues in which Millennium is receiving incomplete or mismatched information and therefore cannot update the database. ESI warnings often include aliasing issues, script issues, invalid catalog_cd, action_type_cd, order_id or synonym_id. The problem also could be as simple as missing data or an incorrect date/time format.

The ESI log typically contains only the last seven days of data. Failures, therefore, are purged and the information lost unless you are looking at these issues at least once a week.

All of these items are straightforward to correct — once you know about them. The key is to be willing to believe the data in your log files, use it to identify the failed charges and have a process in place to correct the problem.  

Prognosis: Addressing the build issues associated with the charge errors in your message logs and ESI log will translate into increased revenue for your organization. 

Security Changes Create Queuing Issues

By David Hallberg

Cerner recently made security changes to Millennium’s Java services and several older servers that might have disabled your system’s ability to automatically replay failed transactions. Unless corrected, caregivers might be losing important clinical information.

The changes involve Millennium’s personnel (prsnl) table, the listing of users who have permission to access applications and modify tables that contain clinical information. Previously, the username “system” was not typically included in this table because this username was created for those doing low-level executables with no need to access clinical applications or view clinical data. It was used to access failed transactions for SCP Entry IDs 105 and 108 (ORM Explode Cont Order and ORM Explode Cont Order Batch) and to replay them. A few months ago Cerner sent out a Millennium Flash announcement saying, essentially, that  failed transactions could only be replayed if the username was listed in the prsnl table and, to that end, the “system” username was now included in the prsnl table.

By forcing transactions to be replayed or re-queued by a user listed in this table, Millennium is plugging a gap in its security model. However, contrary to the announcement, most clients don’t have “system” listed in the prsnl table. If you’re one of them, you might find that transactions are failing to be replayed and that some might not post the first time.

“System” is one of three Cerner default usernames. The other two — “cerner” and  “systemoe” — are already in the prsnl table, where they need to be. But having this listing is only the first step toward ensuring proper replaying of failed transactions. I worked with two clients in June who had numerous interactive users all properly listed in the prsnl table but whose transactions still were not re-queuing. We discovered that some (hopefully all) Java servers have an extra level of security. Users must be granted access to application 3202004 (java.exe). I suspect that this requirement applies to SCP Entry IDs 105 and 108 as well.

Unfortunately, this access change does not seem to have been widely documented or deployed, so you might be losing some information as you’re waiting on others to put the information into Millennium. Far from being your fault, you just were caught in the switch to the new application security.

What do I recommend? Place the following usernames in the prsnl table and give them access to at least the 3202004 application: system, systemoe, cerner and panthercycler (if you’re a Softek client). Then cycle the CPM Application Authorization servers, SCP Entry IDs 105 and 108, and all of the Java servers (generally SCP Entry IDs 350-374).

Prognosis: Correcting the username listings and Java application access required by the new Millennium security model will allow your failed transactions to replay and will result in improved throughput of clinical transactions. To truly improve system reliability, however, you’ll need to address the underlying problem of why the transactions failed in the first place.

Fragmentation’s Biggest Rift? The Generation Gap

By David Hallberg

My last blog covered deleting old files from your Millennium® application nodes. This week I’ll discuss the second step for keeping Millennium's speed consistent over time when you’re tuning filesystems and directories: defragmentation. Just like users disagree over the need to purge unneeded files, they also disagree over the need to defragment files.

I’ve had some interesting conversations over this issue this past year. People weaned on mainframes and older operating systems universally say, "Defragment!" Those who grew up on Linux, HP-UX or AIX virtually to a person say, "Defragmentation is only needed for the old and slower operating systems." With the latter group, I have to remember to stay calm and remind myself that their ideas probably come from the websites and authors who claim that, except with Windows, fragmentation is a thing of the past. But I also remember the documentation from the vendors who wrote the operating systems and the supporting filesystems. Plus I remember my work on the Advanced Technology Group at Cerner and what we benchmarked on AIX, VMS and Windows. Only then do I begin the conversation.

Filesystem and directory fragmentation occurs on all of the operating systems supporting Millennium®: VMS, AIX, HP-UX, Linux and Windows. There are generally two types of fragmentation: file fragmentation and space fragmentation.

In the first type, a file, say dic.dat, becomes too large to fit into a single disk block, so it has to be split into two or more blocks. If the dic.dat file is then modified, it may need a third or fourth block to fit within. It’s likely that code upgrades or custom CCL changes will have added more files to the filesystem where dic.dat is located, and they will have been saved to the blocks right next to the dic.dat's first two blocks. That means the third and fourth blocks for dic.dat will not be able to be written right next to (or contiguous with) the first two blocks. Now the dic.dat file is fragmented. With the file split into two chucks, the operating system has to perform two reads to completely read the dic.dat. When the disk drive or RAID set has to do more reads (disk I/O's), performance suffers.

Space fragmentation occurs when you delete a file, typically because it was a temporary file. Once deleted, the space it originally occupied leaves a gap on the filesystem. Many deleted files create greater fragmentation, making it impossible to write a file contiguously.

The filesystems/directories I discussed in my last post ($CCLSCRATCH, $CCLUSERDIR, $cer_temp, $cer_fifo, $cer_usock, $cer_log, /var, /tmp) are all designed to hold temporary files, such as print files or logging files, that will be deleted as soon as they are no longer needed, e.g., daily or possibly hourly.

The downstream impact of this design is that all current operating systems need to have both types of fragmentation corrected. Vendors provide tools for this defragmentation, including defragfs for AIX, fsadm for HP-UX, and fsadm for CAMM (Millennium's PACS) and the Vxfs filesystem that it uses. VMS has the DFU tool on the freeware CDs that has to be installed before it can be used. Also for VMS, HP has a DFO tool and Diskeeper offers Diskeeper for VMS, which has been around since 1986. You’ll have to pay for these latter two tools, but it might be worth investing in one of them if you’re trying to keep your VMS system. For Windows, you can use the built-in tool called Disk Defragmenter; a third-party tool like Diskeeper 2010 and its various versions or PerfectDisk 11 and its various versions; or one of the plethora of freeware/shareware tools, my favorite being Defraggler from Piriform. I encourage you to spend an hour or so looking through the numerous white papers, blogs and forums discussing these options.

My first team in Cerner's Advanced Technology Group was tasked with successfully completing client conversions no matter what technical hurdles we faced. One of our biggest obstacles was severe fragmentation of both files and free space. We had such significant file and space fragmentation at several sites that we added a drastic step to our normal recommendations. In addition to deleting lots of files daily and defragmenting the filesystem at least once per week, the clients had to set up a schedule to point the filesystem, say $cer_print, to an entirely new filesystem every three months. Miraculously, their systems became much more stable.

Why did it help to move the filesystem? Each filesystem had an index that kept track of where each file started, the size of the file, where it ended, etc. As thousands of files were created and deleted every day, the operating system had to slow down in order to keep track of them. The periodic refresh of the Inode or indexf.sys (on VMS) brought performance back to a better level. With today’s improved drive technologies, this type of severe fragmentation is less likely, but it still needs to be reviewed. Some of you might find it necessary to recreate your filesystem on another set of disks and benefit from having contiguous files and free space.

Prognosis: Using defragmentation tools on a regular basis to reduce filesystem fragmentation on all operating systems will increase Millennium’s speed, consistency and reliability.

Message Offers Quick Fix for ECO Server

By David Hallberg

In my auditing of Millennium® systems over the past decade, I have seen many meaningless and vague error messages. So I was intrigued by the message I found recently while performing a system audit for a client. The interesting thing about the message was that it actually told us how to fix the problem. I applaud the Millennium developer who was kind enough to think of how to help clients with this issue.

The message appeared when I was reviewing some fast-wrapping message log files. Several messages appeared in the ECO (Explode Continuing Orders) server. The ECO server is used to “explode” orders that have multiple schedule times. For instance, if a clinician orders a daily x-ray for a patient, the information is exploded initially on the ECO server: ORM Explode Cont Order (SCP Entry ID 105). Once the first day’s x-ray is ordered, the order is exploded by a second ECO server for batch jobs: ORM Explode Cont Order 001 (SCP Entry ID 108).

One ECO server message in particular grabbed my attention. Even though it was only an audit-level message, it included the way to resolve the issue. It said:

The number of orders to explode for order_id: 123456789 is > than NUM_ORDERS (50). Only 50 instances will be exploded. To increase the limit, run dcputil_set_config_prefs in CCL and then cycle this server.

I learned that I had more orders to explode than the system would allow. But I also learned how to fix the problem. I had to:

1.    Go to CCL.

2.    Run the script dcputil_set_config_prefs.

3.    Change the NUM_ORDERS from 50 to something larger.

4.    Cycle the ORM Explode Cont Order (SCP Entry ID 105) servers on all of the application nodes and then cycle any clones of this server, such as ORM Explode Cont Order 001 (SCP Entry ID 108).

What about your system? If the LogLevel of SCP Entry ID 105 and any clones of it is 2 or higher, I would encourage you to look for Event ExplodeOrders. If you have a message similar to the one above, you will have a quick fix for your environment. Once you execute it, the clinicians will think that Millennium has magically started working and they can stop working around it or, rather, working around the fact that the orders that were supposed to be there were not.

Prognosis: Interesting error messages can actually be helpful. If you take their advice, you’ll help ensure that Millennium delivers all the information the clinicians need.

 

Preparing for an Upgrade?

By David Hallberg

We have seen so many Millennium® clients have problems after upgrading to 2007.18, 2007.19 or 2010.01 that I feel compelled to address this issue. If you have not yet upgraded, this blog will link you to specific recommendations to help minimize the known risks to your production environment. It’s best to execute them before the upgrade. If you have already upgraded, you may want to review these items and see how many of them you can do to improve your current stability.

In general, I’ve ranked my 14 recommendations in order of importance. The majority of them are MQ and message log issues. They apply whether you are on AIX, HP-UX or VMS (although VMS applies only through the 2007.19 upgrade since Cerner doesn’t support this platform on 2010.01 and above). Click here to read the specifics on correcting the following 14 issues:

1.    The hidden MQ queue.

2.    Java server stress.

3.    Citrix crashes.

4.    Exception queue transactions.

5.    Message log disk I/O increases

6.    Confusing MQ log files.

7.    Extra message log writing.

8.    Too much extraneous logging.

9.    MQ information loss.

10.  Missing or incorrect security settings for a file.

11.  MQ running in mixed mode.

12.  Request class routing needs.

13.  Wrapping issues with message logs.

14.  QCP server instances.

Prognosis: By performing a handful of critical tasks before an upgrade, I believe you’ll have a much shorter troubleshooting time during the upgrade process.

If You Rewrite the Rules…

By David Hallberg

How I love rules. By following certain clear principles, you can be reasonably sure of the outcomes. If you wash your hands regularly, then you’ll spread fewer diseases. If you listen carefully, then you’ll be better able to address a person’s concerns. In the IT world, it’s called the if-then-else logic loop. But I’m the first to admit that rules have limitations. I’ll leave you to figure out their limitations in your personal life, but this week I’ll help you determine if some of the rules you have set up in Millennium® are a bit broken.

Millennium has a wonderful subsystem called Discern EKS, which allows users to identify special events that will automatically trigger other events. For instance, users can set a rule that whenever a patient’s white blood cell count is very high, say 15,000 to 30,000, an alert suggesting a blood culture will be generated. Each department determines the rules it needs based on its standard operating procedures, works with the appropriate teams in the organization to set them up and outlines its response when the rules are triggered.

It sounds straightforward enough, and most organizations don’t give much thought to the rules if they are running in less than a second or two. Run time alone, however, does not tell the whole story. Organizations can have hundreds or thousands of rules, some of which fire thousands or tens of thousands of times a day. They can present a huge drag on the Millennium system, and these tuning issues need to be addressed.

For example, if a rule takes one second to run – which would be considered a reasonably fast rule – but is triggered 1,000 times in one day, that one rule is using up 1,000 seconds – or over 16 minutes – of application CPU time. Rules firing more than once per minute – more than 1,440 times a day – need closer scrutiny. I would recommend that you look at that rule’s trigger mechanism and verify it really should be firing that frequently.

Rules that trigger once per minute are not uncommon. I’ve worked with clients that have rules triggering more than once per second – that’s 86,400 times a day. Now consider that many sites have hundreds or thousands of these rules. Even if all of them are so-called fast rules, you can still see where you might have Millennium performance issues.

To determine if you’re having issues with a rule or set of rules, we’ll look at the Oracle table EKS_MODULE_AUDIT. This table shows when rules are triggered and how long they take to fire, along with a lot more useful information. To learn how to access the EKS audit table and analyze its numbers, click here.

Prognosis: An analysis of the time Millennium spends each day running rules will show you whether any rules need to be rewritten to help the system run faster.

Chart Servers — Part 4

Tuning Windows OS

By David Hallberg

This week we move on to our third and final step in tuning your Millennium chart servers: tuning the Windows operating system (OS) on each server. In step 1, you determined which of your chart servers was the fastest and copied that server’s configuration to the other servers. In step 2, you tuned your Oracle database tables. Once you complete this last part of our holistic tuning approach, you can expect the process time of your chart servers to be less than half a second per page.

Tuning of the OS running chart servers should start with Cerner. A document on Cerner Millennium System Settings (CMSS) for Windows is available from Cerner Support. The document has about 15 pages and contains a couple of key items that relate to our topic: 1) Pay attention to the effect that anti-virus scanning software has on the solution, and 2) be wary of the recommendation to run a defragmentation tool at least monthly. I recommend that you defragment the chart servers every day.

Many freeware and shareware solutions are available to defragment your servers. I enjoy using Defraggler from Piriform (www.piriform.com/defraggler). It has a native 64-bit client but runs on a 32-bit OS as well. Diskeeper from www.diskeeper.com is another excellent solution if you have the operational budget for it. For any of you running VMS, Diskeeper has great VMS defragmentation tools as well as I/O analysis tools.

A lot of people are documenting how they tune Windows. Doing a search on Google for “XP performance tweaks” returns 8.5 million results, “2003 performance tweaks” returns 2.6 million, and “2000 performance tweaks” returns nearly 4.7 million. Of the many sites that have excellent information on tuning the OS, I have a couple that I keep going back to:

1.    PC Pitstop (www.pcpitstop.com) will run an audit on your server or PC and show you where you have potential performance and configuration issues. If you want to have the tool do the registry changes for you, there is a fee.

2.    SpeedGuide.net (www.speedguide.net/index.php) has a lot of topics. If you choose the Broadband option at the top left of the page and then select Registry Tweaks, you will find a wealth of information on tuning the Microsoft TCP/IP stack for optimal performance.

3.    Microsoft has excellent documentation at www.microsoft.com. Once there, search for “XP performance tuning” or “2003 performance tuning.” You will have to read through a lot of documents to determine which pieces you want to apply to your OS. That’s why I prefer PC Pitstop and SpeedGuide.net. I have found that by using them, I can quickly get the most from my OS.

My final piece of advice is to turn off the following two services, no matter what version of Windows you’re running:

1.    ComputerBrowser was designed for Windows 95 PCs to act as the WINS server for DOS and Windows PCs. It is very chatty and actually quite a security hole. There is no need for this to run on any chart server.

2.    Themes was designed to make the chart server have a graphically rich (pretty) view of the OS. I want chart servers to be fast, consistent and reliable. I do not need them to be pretty.

Prognosis: With apologies to the ′60s rock band The Byrds, I would encourage you “tune, tune, tune, there is a season, tune, tune, tune, and a time for every purpose….” The season is now and the purpose is to dramatically improve the overall responsiveness of the chart prints for clinicians.