Millennium Medic

OnTrack Panther Supports Millennium 2012.x Code

By Chris Lanaman

David Hallberg’s recent blogs on preparing for your Millennium^® site’s upgrade to 2012.x code have coincided with several conversations with clients who use OnTrack™ Panther™ to manage and monitor system performance. They want to know: “Does Panther support Millennium 2012.x code?” And: “Will OnTrack audits still point out issues as I test the new code and then move it to prod?” I want to assure you that all Softek OnTrack products and services support this version of code.

For those of you not familiar with Softek, our software and services give you a high level of visibility into your EMR system along with actionable evaluations and alerts developed and serviced by a team who collectively had years of experience at Cerner and various Millennium hospitals.

As we enter a new year, we have the same great news we have had since Cerner developed Millennium version 7.6: Softek is working to provide your site an objective view into the performance and reliability of your Millennium system.

The following table lists the Millennium code versions supported by OnTrack Panther.

Millennium Version
7.6
7.7
7.8
2001.01
2001.x
2003.01
2003.02
2005.01
2005.02
2007.02
2007.03-2007.17
2007.18.x
2007.19.x
2010.01.x
2010.02.x
2012.01.x
Choose and Book
eBookings

Prognosis: Third-party support that gives your organization intuitive management of your Millennium EMR system can be the foundation for a smooth upgrade to 2012.x code.

Have You Talked with the Clinicians Today?

By David Hallberg

How do you begin to manage an EMR system as complex as Cerner Millennium^® — with a back end containing more than 35 million lines of code from Cerner along with Linux, AIX, HP-UX, VMS, Oracle, WebSphere MQ, Sonic ESB, MySQL, Java, storage area networks (SANs), networks, clustering software, network connections between Oracle instances, Citrix, Windows servers and desktops; with a front end that has more than 4 GB of space in its code warehouse; and with countless application builds?

I recommend you start your work by stepping away from your computer screens and talking to the people who are using the system to care for patients. Are they frustrated with response times? System availability? The number of clicks to admit or discharge a patient? Don’t dismiss their complaints with the assumption that “clinicians just don’t understand computers.” They are highly skilled people who understand the most complex system on the planet: human life. You might be surprised what you can learn when you take the time to really listen. Or shadow them during their rounds.

When I worked at Cerner, I brought my stopwatch with me to one hospital to join the CMIO and director of nursing informatics on the floor. It took more than 15 seconds to place one order in PowerChart^®. That’s an eternity when you’re with a patient and still have the rest of your rounds to complete. When I took the data back to the engineering lead responsible for the code, she emphatically told me, “Fifteen seconds is fast enough. They (the clinicians) just do not understand computers.” No wonder the clinical staff was frustrated. The system was hampering their ability to provide patient care, but their concerns weren’t given any consideration by the people capable of addressing the problems.

So I encourage you to gather the staff’s list of concerns, criticisms and objections. If they feel they’re not normally heard, no matter how bad their experiences, don’t be surprised if they’re distrustful at first and reluctant to talk. Keep at it. Move through the obligatory moans and groans to get to their true desires.

Now that the easy work is done, begin to rebuild their faith in the system by evaluating each complaint against each need and determine how to address them. You can use Millennium’s greatest asset, its flexibility, to configure it for very specific purposes. If the issues outweigh your staff’s ability to correct them — we’re back to that complexity I mentioned earlier — don’t be too proud to seek outside help. (If you’ll be at the Cerner Health Conference next month in Kansas City, I’d be happy to listen to your system concerns and talk through some options.) Then you can begin the journey of reconfiguring this multifaceted solution set into the system your clinicians actually enjoy using rather than avoid using or simply tolerate.

Striking up a Conversation

One of the strengths of an online community is its collective knowledge — the collaboration and innovation of many that can accomplish far more than any one individual or organization. It’s an open-source culture that has expanded way beyond the software community.

At a conference in Kansas City last week, fusion-physicist-turned-farmer Marcin Jakubowski told about Open Source Ecology, a network of farmers, engineers and supporters designing and posting do-it-yourself schematics for what they see as the world’s 50 most important machines. At a 30-acre site in northwest Missouri, workers have prototyped the first eight modular machines in this collaborative Global Village Construction Set. They envision their designs being used to “lower the barriers for entry into farming, building and manufacturing” around the globe. Another conference speaker, crisis mapper Patrick Meier, talked about the open-source Ushahidi mapping software and legion of worldwide volunteers that have helped governments and the United Nations pinpoint particular people and areas devastated in such recent and ongoing crisis areas as Haiti, Libya and Somalia.

Our humble blog can’t boast such expansive life-saving work, but we’re glad to be a part of an online community working together to improve the world of healthcare IT (which actually can save lives). For the past several years, we’ve addressed many of your concerns with our practical advice for maintaining and troubleshooting your complex Cerner Millennium® EMR system. We’ve even compiled a number of these into in-depth papers on such topics as message log monitoring, tips for remote-hosted organizations and steps to minimize risks during upgrades. These and more are available on our website.

To help us continue to cover the issues affecting your organization, we’d love to hear from you. Where specifically are your pain points with your EMR? In what internal or external areas do you need more understanding? What creative solutions has your IT team found? We will endeavor to address whatever ideas you suggest.

Just click the Comment link below to get the conversation started.

Continuing to Improve Failover: Security Master Configuration

By David Hallberg

After our last post, “Improving Interface Failovers,” a few people wondered why some Cerner Millennium^® servers — most often Ops Job (Millennium batch process), interface and MDI servers — would not start during the failover. Because this situation can have disastrous consequences, it’s an issue worth addressing.

Imagine the following scenario: It’s 3 a.m. on Memorial Day, and your primary application node crashes. The failover scripts run so that users can log into Millennium, but none of the Ops Jobs are running. That means the nursing staff is missing care plans for its shift change, new patients and their lab results are not visible in Millennium, and the Emergency Department director is calling the CIO, COO and CEO complaining that his staff has to go on diversion because they cannot get the patients into Millennium. A storm just ruined your holiday plans.

Had you been aware of a few simple steps that would make your servers fail over successfully, your day could have been clear sailing.

Why didn’t the Ops Job servers start? If the server on a specific Server Control Panel (SCP) Entry ID has not been started on this node before, the Security Master server has to be running to create the service with the correct permissions. If no Security Master server is running after the failover process completes, any of the servers or services that have not gotten to a running state on the failover node prior to the failover cannot be started.

A little analysis will explain why this scenario happens so frequently. Way back when, about 90 days before you first brought Millennium up, consultants from the vendor swooped in to set up failover. They ran through a couple of tests and showed you how, should the primary node go down, you could still log into Millennium. They flew back home, and you turned your attention back to what seemed like more critical work that had to be done before go-live.

Now, many years later, you have added and deleted Ops Jobs, your old registration system has been upgraded, the inbound feed changed names to allow testing into Millennium production and to minimize the cutover time, you added about a dozen MDI servers, you moved on to a different Millennium client, or you were promoted, and your project staff is scattered around various clinical areas. Go-live and pre-go-live are remembered for the number of service packs you had to install and test and the huge amount of training you had to do. Who recalls the little detail about the need for the Security Master to be running to allow a new Millennium server to start on a node?

What, in fact, is the Security Master? For organizations using LDAP to access and maintain their user directory, the Security Master server updates the sec_user.dat file with local accounts, for example SYSTEM, SYSTEMOE and CERNER. For those not using LDAP, the Security Master also is used to add, remove, modify and delete all user IDs. So when a new person joins your organization and you run the tool HNAUser, a new row is added to sec_user.dat as well as some Oracle tables. Additionally, the Security Master creates a service key for the Millennium processes as they start up. These service keys only exist in the sec_user.dat file; they are not kept in LDAP. This key is used to make sure the processes running on the application nodes are legitimate processes. During normal operations, no one really cares about the security services. They quietly do their work when you stop and start Millennium servers on the application nodes, and everyone — especially your security manager — is happy. Until a failover.

Let’s go back to our Memorial Day crisis, where your family headed to the lake without you. You, like most Millennium clients, have Node 1 of SCP Entry ID 32 configured as the Security Master and Node 2 as the Security Slave. With this common configuration, however, you do not fail over the Security Master, so all of the Ops Jobs you added after go-live, along with new interfaces and MDI servers, didn’t seem to start. Yet nothing shows in any of the message logs. You called your vendor’s Immediate Response Center, but the response has not been immediate enough. After calling you back in 15 to 45 minutes to verify you are still having the problem, IRC might take four more hours to contact you again. In the meantime, the ED director is calling for your resignation and the ICU, CCU and NNICU staff are looking for a throat to choke.

How could you have prevented the chaos and been fishing for something other than answers? The single biggest thing you can do is NOT put a Security Slave on SCP Entry ID 32. Use any other SCP Entry ID — 31 and 19 are often open. After you have the Security Slave configured to run on a different SCP Entry ID, copy the Security Master from the primary node to the secondary nodes. Verify that the Instance is set to 0, which will ensure that SCP Entry ID 32 does not start up and that CPM HNAM Agent does not start it. Finally, put 32 in the cerner.vars file as a single-instance server. That way, when the failover scripts run, you will be able to start the Ops Jobs, interfaces and MDIs as needed to minimize your downtime.

Here’s one last detail for all the system administrators who like such specifics: Create a chron job to automatically copy the $cer_config’s security.jnl file from the application node where the Security Master is running to the other application nodes. Then verify that the permissions are “-rw-rw-----” and the owner and group (assuming your domain name is prod) are both d_prod.

Prognosis: A little preparation can dramatically improve the failover process and make your night, weekend and holiday events much shorter and quieter.

Make Sure Servers Write Events to Message Logs

By David Hallberg

A client wanting to determine who had cycled a server in Millennium’s Server Control Panel (SCP) at a certain time went to the SCP’s message log cmb_0033 for the answer. When they opened the file, what did they find? Absolutely nothing. No entries or rows had been written to the message log since their last code upgrade. All of the stability issues and change control events that should have been logged were lost. The problem was much bigger than not knowing who had cycled a server. The organization had no way of knowing what else was wrong or how to resolve those issues.

We checked out three parts of the system to track down the problem.

1. Permissions settings in the $cer_log directory. The owners of the files in the $cer_log directory had been set to root for the User and root for the Group. That setting works great much of the time, but not for the following message logs: cmb_0000.mlg, cmb_0033.mlg, cmb_0057.mlg, cmb_0066.mlg and cmb_0078.mlg. For these, the User setting should be root and the Group setting should be your domain user name (for instance, d_prod or d_pxxx, with the xxx being your Millennium client number if you are remote hosted). Then the permissions on the file can be -rw-rw----, which allows the user of root as well as anyone in the group of d_prod to read and write to these log files. If you set these permissions properly, you will have lots of additional information about performance and stability issues in your Millennium domain.
2. SCP settings. SCP servers should have a User Name set to root and a Group Name set to your domain user (for example, d_prod). You also need to set the SCP Property of LogLevel to 2 on the following servers: 0 (Service Manager), 33 (Server Control Panel), 66 (CPM HNAM Agent) and 78 (CPM Script Install). The LogLevel for SCP Entry ID 57 (CPM Script Batch Priv) should be set to 1, unless you are actively troubleshooting an issue with this server.
3. The messages.suppress file in the $cer_mgr directory. If you cat messages.suppress, you will see all of the messages that are not being written to the message log. You are missing significant information if any rows in this file start with the following characters: CMB_, HNAM_, SCP_, InstallOCD, OCDPURGE and Script.

Prognosis: If you check and correct your message log settings before and after an upgrade, you will have much more insight into your Millennium domain and will be better equipped to troubleshoot stability problems and monitor change control.

Easy Tuning of Oracle Sequences Improves Throughput

By David Hallberg

In more than 13 years of helping clients tune their Cerner Millennium^® systems, I have only recently turned my analysis to Oracle sequences. I have the IT team for a PowerChart client to thank for my newfound understanding of how sequence numbers can increase the variability in system response times…and drive clinicians crazy.

After engaging me to diagnose their PowerChart performance problems, the team corrected several MQ queuing issues and other straightforward problems, but clinicians were still experiencing slowdowns during their peak usage times. The culprit turned out to be the Oracle sequence numbers.

Oracle sequences give an identifying number to certain elements in the database. Millennium has about 964 separate sequences, Oracle’s System and Sys have about 20 and 66 sequences, respectively, DBSNMP has two, and CMS_DBA and Panther_Read have one each. Oracle generates these sequences when new data or rows are added to the database. Sequences become a performance problem when you have a lot of new data that needs sequencing but the tables aren’t configured to adequately handle the load. Making the change is surprisingly simple.

Here’s what I discovered for my client. I took a 1-minute and a 30-minute sampling of the Oracle sequence numbers being generated by PowerChart. The following chart shows the results of the 1-minute interval: 

SEQUENCE_OWNER	SEQUENCE_NAME	CACHE_SIZE	Sequences Generated
V500	CHARGE_EVENT_SEQ	2,000	8,000
V500	OCF_SEQ	2,000	6,000
V500	CLINICAL_EVENT_SEQ	2,000	4,000
SYS	IDGEN1$	20	2,000
V500	PATHNET_RECOVERY_SEQ	2,000	2,000
V500	PERSON_SEQ	2,000	2,000
V500	LONG_DATA_SEQ	2,000	2,000
V500	CPMAPP_APPLCTX	20	1,320
V500	CPMAPP_APP_CTX_ID	20	960
V500	CHART_DIST_LOG_SEQ	20	40
V500	ENTITY_LOCK_SEQ	20	20
PANTHER_READ	AUD_ORA_ERROR_SEQ	0	7
SYS	ORA_TQ_BASE$	0	2

The key columns are Cache_Size compared to Sequences Generated. The Cache_Size shows how many sequence numbers can be generated before updating the high water number, which is the highest sequence number ever created for that sequence name. It takes time for Oracle to update this number, so you don’t want the system to have to do it too often. Increasing the Cache_Size decreases the number of times the high water value needs to be updated and, therefore, improves the throughput for the tables that use that sequence number.

Let’s look at an example in the chart. The Sequence_Name of Charge_Event_Seq generated 8,000 sequences in 1 minute. Because its Cache_Size was only 2,000, Oracle had to update the high water value four times during that minute. The Sequence_Name of Idgen1$ generated far fewer sequences, only 2,000, but its Cache_Size of 20 meant that Oracle had to update its high water value 100 times in this same minute. You can reduce this overhead associated with sequences — and improve the throughput for their corresponding tables — by increasing the Cache_Size to match or, preferably, exceed the number of sequences generated during your time frame. I recommended my client set Charge_Event_Seq to a Cache_Size of 9,000 and Idgen1$ to 2,000.

The change can be done in uptime without taking any action on the Millennium application nodes and without doing anything else on the Oracle nodes. The change is accomplished via SQL*Plus with this command:

       Alter sequence charge_event_seq cache 8000;

In CCL, use the following command:

       rdb alter sequence charge_event_seq cache 8000 go

The following chart shows the results from my 30-minute interval, which are a little more staggering in terms of the impact that high-water-value generation has on overall throughput. It also shows that only 35 sequences — about 3 percent of the more than 1,050 — had increases in their sequence numbers during this time (32 of 964 for V500 or Millennium and 2 of 66 for Sys).

SEQUENCE_OWNER	SEQUENCE_NAME	CACHE_SIZE	Sequences Generated
V500	CHARGE_EVENT_SEQ	2,000	106,000
V500	OCF_SEQ	2,000	76,000
V500	CLINICAL_EVENT_SEQ	2,000	48,000
SYS	IDGEN1$	20	40,000
V500	CHART_VIEW_SEQ	20	24,260
V500	CPMAPP_APPLCTX	20	22,660
V500	CQM_TRIGGER_ID_SEQ	2,000	18,000
V500	CPMAPP_APP_CTX_ID	20	16,160
V500	CQM_QUEUE_ID_SEQ	2,000	16,000
V500	ORDER_SEQ	2,000	14,000
V500	PATHNET_RECOVERY_SEQ	2,000	14,000
V500	CARENET_SEQ	2,000	8,000
V500	PATHNET_SEQ	2,000	6,000
V500	PERSON_SEQ	2,000	6,000
V500	DM_CLINICAL_SEQ	2,000	4,000
V500	EKS_DLG_EVENT_SEQ	2,000	4,000
V500	ENCOUNTER_SEQ	2,000	4,000
V500	SCH_CONVERSATION_SEQ	2,000	4,000
V500	AAR_REPORT_SEQ	2,000	2,000
V500	CHART_REQUEST_EVENT_SEQ	2,000	2,000
V500	CHART_SEQ	2,000	2,000
V500	ENCOUNTER_ONLY_SEQ	2,000	2,000
V500	ENTITY_LOCKKEYID	2,000	2,000
V500	ENTITY_RELTN_SEQ	2,000	2,000
V500	ESI_LOG_SEQ	2,000	2,000
V500	MEDADMIN_SEQ	2,000	2,000
V500	NETTING_SEQ	2,000	2,000
V500	OMF_SEQ	2,000	2,000
V500	PHARM_FILL_HX_SEQ	2,000	2,000
V500	REFERENCE_SEQ	2,000	2,000
V500	RESULT_SET_SEQ	2,000	2,000
V500	ENTITY_LOCK_SEQ	20	400
V500	CHART_DIST_LOG_SEQ	20	300
PANTHER_READ	AUD_ORA_ERROR_SEQ	0	125
SYS	ORA_TQ_BASE$	0	4

So I recommend only changing sequence numbers that are being used a lot. In this case, I recommended:

• Charge_Event_Seq and OCF_Seq have Cache_Size set to 10,000
• All V500 sequences with more than 2,000 sequences generated have Cache_Size set to 8,000
• All V500 sequences that match their Cache_Size be left at 2,000, at least initially
• All V500 sequences with a Cache_Size less than 2,000 be increased to 2,000
• Sys Idgen1$ have Cache_Size set to 10,000
• Sys Ora_Tq_Base$ have Cache_Size set to 20
• Panther_Read have Cache_Size set to 2,000

It is possible to make the Cache_Size too large, so my recommendations are fairly conservative. Still, the client will have much less variability in Millennium response times after implementing these straightforward fixes.

Prognosis: Keeping your Oracle sequence Cache_Size properly set is a simple, uptime tuning procedure that decreases performance-robbing events and variability in your Millennium environment.

Messages.suppress File Can Hide Serious Issues

By David Hallberg

Historically, when I was brought into situations where a problem within the EMR system was difficult to track down, I used the message logs quite extensively. They helped me understand what was happening and where the troubleshooting should begin, and they explained any application-specific problems. This method worked very effectively until Millennium^® 2007.18 TP4, when a new file in the $cer_mgr directory — the messages.suppress file — started shipping with some troubling exclusions, namely ScriptError and  ScriptErrMsg events.

The messages.suppress file stops Millennium from writing events to the message logs. This file was designed to stop creating message log events when an engineer accidentally sent an event to clients with an incorrectly set LogLevel. It’s a legitimate place for extraneous events that would otherwise clutter your log files and potentially overwrite messages that need attention. For example, the event CRM_ClientTimeZone writes the current time zone to the message log. Since domains rarely change time zones, the event is seldom of benefit when troubleshooting an issue. Yet it can cause a lot of application node writes, since every transaction going through the application server with this set has to write a message log row into the system message log as well as the specific server message log. To reduce the application node’s disk I/O and performance variability, it’s good to have CRM_ClientTimeZone in the messages.suppress file and exclude these events from the message logs.

If you hide the wrong events, however, you miss out on important information. A problem, failure or issue that should be written to the message log goes undocumented, and your clinical or technical support team has no record to assist them in identifying and resolving issues.

So it makes no sense to me that with the initial TP4, ScriptError and ScriptErrMsg events were included in the messages.suppress exclusions and that the file has since expanded to include ScriptErrors-Fatal and ScriptErrors-NonFatal events. You need visibility into these events. They indicate that an application or Oracle has a problem (often a code or build issue) and needs help to fix it.

What might the problem be? ScriptError events include blood bank transfusion script errors, MDI download problems, scripts failing to run (such as those getting PharmNet frequencies) and Oracle deadlock issues. These events all have a negative outcome to the clinicians using Millennium and the business office needing to bill accurately for services delivered.

ScriptErrors-Fatal events include memory problems that prevent clinical information from being updated and from being viewed by the clinicians who need this information.

ScriptErrors-NonFatal events show a wide range of issues. Expert Knowledge System (EKS) compile errors, Visual Explorer errors, SurgiNet documentation errors, getting orders asynchronously for PowerChart, Lights On Network data-gathering script errors and sequence number problems are all events that can degrade Millennium performance or even cause a failure. These errors must be logged so that problems can be seen and corrected.

Prognosis: Keeping your messages.suppress file properly set will ensure that the Millennium message logs contain the information that your support team needs to identify and resolve issues in a timely manner.

Why Auditing Your EMR Is Essential

By Chris Lanaman

I want to expand on the comments I made in my last blog on the need for third-party auditing of your EMR system. I’m including an excerpt from the white paper “Why Auditing Your EMR Is Essential” that our Softek team wrote and distributed last week at HIMSS. The message struck a chord with the people we talked with — both self- and remote-hosted organizations— so I’m confident it will resound with you as well.  

When patients are admitted to your hospital, they count on reliable, competent care:  caregivers who are board certified, care delivery that has passed the standard of the Joint Commission, lab procedures certified for quality by the College of American Pathologists, certified dieticians who oversee patients’ dietary needs and ethical business and financial practices. In each of these areas, facilities and individuals submit to regular, independent evaluation to make sure their practices follow generally accepted industry standards and, when necessary, implement changes to bring them back into compliance.

One area overlooked in these evaluations is the operation of the EMR system itself. With more and more functionality demanded from these systems, they continue to grow in complexity. Yet no independent evaluation program is required to ensure that the EMR is performing as expected. And if it’s not, the consequences could be devastating for patients, caregivers and the facility itself.

Think of the changes you continually introduce to your configuration. Besides the customizing you do for your own unique uses, vendors are supplying a steady stream of system updates and upgrades. A technology upgrade can create myriad downstream impacts on how the EMR will behave. Every time you add a new layer, a new process, a new handoff point, you increase the likelihood of delivery failure. Each failure poses the risk of losing patient information or financial data or extending wait times for clinicians trying to input or access that information.

If your EMR system is hosted outside your own organization, you might think independent auditing doesn’t apply. After all, one of the primary reasons you sought out a knowledgeable vendor was to oversee system performance. You wanted the assurance that comes from having competent partners hosting your system. Yet how do you know that the hosting partner is catching all the performance problems that arise? Because so many performance and stability issues arise from errors with application settings rather than from technical infrastructure, who is really responsible?

Your hospital IT team knows when performance problems develop — you deal with the complaints. With the visibility that comes through a system evaluation, you can more confidently partner with your service provider to not only address system issues but also set higher standards for performance and quality of patient care.

As healthcare focuses more and more on evidence-based medicine, the IT delivery vehicle for that medicine should be evaluated in the same way: using real data from your own system, gathered by an organization independent from the vendor or your own organization, and clearly presented so that you can understand where you are and what adjustments need to be done to accomplish your goal of optimal healthcare.

Prognosis: You can learn even more about how to get an evaluation of your EMR by reading our white paper, “Why Auditing Your EMR Is Essential.” Click here to receive it.

Betrayed by an Old Friend

By David Hallberg

I had a client complain to me recently that I was reporting a false positive, an alert that was triggered even though there was, in fact, no problem or issue with his Millennium^® EMR system. I decided to figure out why there was a discrepancy between my measurements and the client’s quality-assurance tool, which happened to be sysmon.

Let me begin by saying that I love sysmon. Before sysmon was developed in the late 1990s, Millennium had so many tools or utilities that provided a single view of a piece of information that it was difficult to keep track of all of them. At one point there were more than 200 build or monitoring tools even though Millennium only had about 40 actual application executables that were used in various configurations for its 60 plus solutions. Multi-node clients kept asking their Millennium middleware team to figure out where performance bottlenecks were occurring. Believe me, this sleuthing was extremely labor-intensive. Users had to have mon_ss (a Millennium monitoring tool used to watch for proprietary middleware queuing) and joumon (a Millennium monitoring tool used to watch for persistent queuing in the proprietary middleware) up in a different window for each application node. Then they would have to try to show someone else when one or more of these screens had a “blip” or negative performance indicator. It was painful on the good days.

So those managing Millennium were understandably ecstatic when Cerner developed a single utility that told what was happening across Millennium’s persistent and non-persistent queuing mechanisms across all application nodes, in a single screen. Sysmon also provided the date and time when something bad happened.

Sysmon proved to be reliable, and I became a proselytizer for this great monitoring tool. While I was at Cerner, I taught many clients how to use sysmon and its various switches. I also used it to tune a four-node client with ConnectionBias (a load-balancing mechanism for Millennium), moving the transaction workload from one application node to another. I was able to show enough value with this one tool that some clients actually chose not to purchase BMC Patrol (later called BMC Performance Manager and now called BMC ProactiveNet Performance Management). Sysmon had some limitations — it was really difficult and unreliable to pipe or move the output to a file for additional processing, and it had no mechanism to redirect its output to a common data format like CSV (comma-separated values), tab-delimited or regular flat files — yet it was the most accurate view of a multi-node application configuration.

So why was my data conflicting with sysmon? It appears that with 2007.02 some of the newer Millennium services do not register with the old middleware in a way that allows sysmon to see the queuing. With the client I referred to earlier, I found that the SenSage add-on called P2Sentinel did not publish its queuing in a way that sysmon could see it. The toolset I used, however, did see the problem.

That’s not the only problem. Another client reported that one of the Java services introduced with 2007.02 was crashing several hundred times a day. I ran sysmon to see if it would show these crashes, and it did not. I am not exaggerating to say that I was crushed. Sysmon was the one tool I absolutely trusted to let me know in a timely manner when Millennium was failing, typically 15-45 minutes before BMC. Now I find it is not reliable!

If, as it appears, the newer programs will not share or publish their information like the old programs, Millennium clients now must find other tools. None of the options, however, shows them this information in a single view. Sadly, we all have to go back to running multiple windows for each application node when we are using command-line tools.

A source of truth that you can trust and believe in is difficult to find. It is even more difficult to see it no longer be a source of truth.

Partners for Success in 2011

By Chris Lanaman

With the start of a new year, I’m wondering if you’re feeling a bit like Bill Murray’s weatherman character in the movie Groundhog Day, in which he eerily wakes day after day, week after week to the sound of Sonny and Cher and the realization that he is about to relive the same day over yet one more time. Rather than dreading the challenges and aggravations that you suspect are coming when you arrive at work today, we’re hoping instead that you might see today as a chance to erase the mistakes of yesterday and make a fresh start. To that end, our team of bloggers is invigorated to introduce you to some new and meaningful tips to improve your Millennium® system’s performance, along with information on where healthcare IT is going in 2011. Dave Hallberg and I have been busy already this year with several client visits. When our travels slow down, we’ll be writing about some of the issues that have turned up at these sites. With each blog we hope to peak your interest and spark exceptional conversations that will lead your organization to greater levels of success. As always, if there are items you would like to see discussed, let us know what they are. We are always willing to address your most pressing issues. As the Director of Millennium Consulting Services for Softek, I look forward to sharing knowledge with you and your organizations.

Determine if Millennium’s Size Is Just Right

By Chris Lanaman

Like Goldilocks trying out chairs at the cottage of the Three Bears, every healthcare organization is looking to land on a system that is sized “just right.” When you first had Cerner Millennium^® installed, smart people thought through what kind of hardware and how much of it would need to be deployed in order to meet your organization’s needs. Most likely they also considered the projected growth of the organization over the next five to 10 years. Buying too much would add complexity for your users, loss of control in managing the system and, of course, wasted cost and time. No one wanted Millennium to be like Papa Bear’s too-big chair.

I would suggest that a bigger problem exists when the Millennium system is not sized big enough for the demands of your organization. Any one of the following symptoms might indicate that your workload has outgrown your system: lower than expected productivity of outcomes, the loss of business revenue because of turning away patients or poor response time for end-users.

I recommend starting with the following five steps if you are concerned that your organization has outgrown its current Millennium size.

1. Data, data and more data. You must have data in order to see if your organization is suffering from a sizing issue. This data is gathered by your team of data-miners (IT analysts) or through an application that is generating reports for your system, such as Cerner’s PowerInsight or PowerVision. Use this information to see if there has been a drop in activity, a workflow slowdown or loss of productivity.
2. Patient management cycle. Once you have obtained measurements for your organization’s workflow, you will need to evaluate where you are in your patient cycle — is this the time of year that you have increased patient traffic or are these your slower months? Your metrics should also include a staffing model to meet patient demand. Understanding your cycle will keep you from over-sizing your system if you evaluate during the high point of the year or under-sizing if you are in a valley part of your cycle. These measurements will identify the revenue cycle expectations for your organization.
3. Mergers and acquisitions. When your organization merges with another healthcare organization, great pains are taken to try and unify the new organization under a common vision. The impact of the merger on the EMR system’s workload, however, might be overlooked. The IT analyst needs visibility into the new organization’s past data to understand how workloads might increase.
4. Market share. Does your organization understand the footprint it has in your market? When trying to understand growth potential, you need an understanding of your opportunities and who the suppliers in the market are. Theoretically, one can say a set percentage of business equals a set amount of hardware needed, but this is an educated guess at best and should never be the rule of thumb.
5. Is your Millennium system healthy? As I alluded to earlier, when your system is not sized correctly, you will see an increase in performance issues. If your end-users are telling you, “The system seems slower than it used to be” or “It seems to take longer for me to do my job in the system,” the first thing on your mind should be the health — i.e., the efficiency — of your system. Before joining the Softek team, I used Softek Panther for a number of years to uncover trouble spots. It let me know when Millennium stepped out of line. Additionally, Softek’s Queuing Performance Spreadsheet (QPS) allows you to see where within Millennium your user transactions are getting clogged, and the Messaging Performance Audit (MPA) gives your system an overall checkup. The audit lets you see if you have build issues in Millennium, if transactions are going where they’re supposed to go, if your servers are configured correctly and when your servers are being over-tasked. If you have other monitoring tools you recommend, please let me know. Whatever you use, it’s important to check your system thoroughly before buying more hardware. Hardware purchases should always be considered the last resort for resolving performance problems. 

In a blog, I only have space to give you the Reader’s Digest Condensed version for scoping your organization’s size and determining how that affects Millennium’s performance. But addressing these key points will help keep your organization healthy and your users happy.

Prognosis: Evaluating all of the different factors that make up your organization is the first step in understanding if your Millennium system is sized “just right.”

Putting the Profit in ProFit

By David Hallberg

I’ve written previously about lost charges and how to examine Millennium’s message logs to find and correct billing issues. Today I want to speak specifically to clients who use ProFit^® Patient Accounting, Millennium’s billing solution. Those wanting to maximize their revenue will find a wealth of information in the message logs on the ProFit servers.

One of ProFit’s components is a third-party subsystem called Sablotron, an XML toolkit that was used to make ProFit easier to support across operating systems (http://freshmeat.net/projects/sablotron). Because it is XML based, Sablotron is independent of the operating system and works the same on AIX, HP-UX, Linux and VMS. This is great from an engineering support perspective. But what does it mean to a healthcare organization?

ProFit has two primary servers: ProFit Transaction Server (SCP Entry ID 475) and ProFit Billing Server (SCP Entry ID 476). Generally, an organization can only control the memory size and loglevel of these asynchronous processing servers, and interactive users typically do not know when these servers (executables) have had any errors or warnings. Someone would have to go digging for these issues to become aware of them. I recommend you get out your shovels.

I recently ran across two events on the Billing Server of a ProFit client. To correct the first issue, we needed to log a service request to get help in correcting the build. For the second, we needed to log a service request to get new code.

The first event was the Sablotron Message, which indicated that the subsystem could not properly handle some data in a billing transaction. In a single four-hour time frame, my client had 4,002 of these events. The Data component of these records had a consistent beginning section that looked like this:

{{msgtype::warning}}{{code::93}}{{module::Sablotron}}

It was followed by one of three ending components.

1.  The most numerous Sablotron Message ending was:

{{URI::arg:/_parsed_}}{{line::544 }}{{node::element '<xsl:for-each>'}}{{msg::unsupported language 'en'}}{{Routine::sablotronErrorHandler::log}}

You’re probably thinking, “This looks like someone sat down at the keyboard after drinking a bit too much.” But the message actually illustrates how many XML files are laid out or formatted. XML is just another way to show data, and the message tells us that in line 544 of the server, there was a language code of “en” that did not have a translation (‘<xsl:for-each>’) and the server didn’t know what to do with it. So the server wrote the message to its message log.

2.  The second most numerous Sablotron Message ending was:

{{URI::arg:/_parsed_}}{{line:: 497}}{{node::element '<xsl:for-each>'}}{{msg::unsupported language 'en'}}{{Routine::sablotronErrorHandler::log}}

This looks like the same drinker was at the keyboard. The message is the same as the previous one, except that the unsupported language code of “en” was in line 497.

3.  The third Sablotron Message ending was:

{{module::Sablotron}}{{msg::unsupported language 'en'}}{{Routine::sablotronErrorHandler::log}}

This message appears to be a higher level failure, but the end result is the same. The server had a language code of “en” that it could not translate, so it wrote the message to its message log.

The second event my ProFit client experienced with the Billing Server was SRV_LookupFailure, which told us in the Data that the bill_item_id was missing from the transaction. This would generally not be a problem, except that in this case the user name was SYSTEM and, therefore, no interactive user was made aware of these failures. They were only in the message log.

Not all the people I work with think these messages are a big deal — they see errors in Millennium as normal. I disagree. The client I’ve referenced here had, on a single node, 3,038 of the line-544 Sablotron Message events and 904 of the line-497 Sablotron Message events in two hours as well as 60 higher level Sablotron Message events and 92 SRV_LookupFailure events in one hour. How could this be considered normal? If an executable is having a problem, you want to correct it, whether the issue is in the build, the code or the use of the code.

Prognosis: Little events can add up to big revenue losses. Looking for these events and resolving the code or workflow issues will maximize your revenue.

It’s Just a Fax, Right? Part 3

Troubleshooting your RRD application

By Chris Lanaman

For the past two blogs, we have looked at the important role faxes play in relaying critical information from the hospital to doctors’ offices and how a reliable fax service can improve relationships and business with these offices. Today we turn our attention to making sure Millennium’s Remote Report Distribution (RRD) servers, which send the faxes, are operating at peak performance.

It’s not always an easy task — which is why in my first blog on this subject I recommended that management dedicate at least one full-time analyst to support faxing. Gone are the days that you walk over to the fax machine, set your forms on the paper feeder, type in the phone number, hit a button and away it goes. Now many processes typically occur before a document reaches the RRD queue and is ready to be made into the fax and sent on to the primary care physician.

When training new analysts over the past few years, I often taught them the Cerner RRD application first, before graduating them to the Clinical Reporting applications. This method might seem backward, but I found it to be the most effective. With most things in the IT world, the best way to learn how to fix things is to start at the outcome and then work your way backward. If your end-users are saying they didn’t get the fax, you check the RRD server.

RRD technology has not changed much in recent years. Because of this, RRD servers can quickly become overwhelmed when a hefty number of faxes are pushed through them. If you’re experience faxing delays or failures, I recommend you check the following:

1. Operating system. Imagine sitting at your computer all day and faxing a Word document over and over, all day and night, through the computer’s modem. At some point, you would become exhausted and would need a break to regroup and refresh. Similarly, the RRD application needs to be refreshed. Small things like clearing the memory (cache) and straightening (defragging) the files are often forgotten but can help speed up the process.
2. Fax volume. It’s important to know the number of faxes you are sending each day. If you’re sending more than your RRD servers can deliver in a timely manner, you might need to beef up your RRD servers. Remember, as I stated in an earlier blog, delays in fax deliveries can be very damaging to your organization’s relationships with referring physicians.
3. RRD services. Make sure the RRD services are running on your RRD servers. Without these running, you simply have a desktop computer not doing anything.
4. Phone lines. Another area that often gets overlooked is whether the phone lines are still active. Troubleshooting might be as simple as making sure you have a dial tone on the lines going into your RRD servers.
5. Modems. Check to see if your modem is hung and not releasing your transmissions. If you use Softek Panther, make sure the RRD: Stalled sensor is set and then check for any Error messages. These messages might indicate a problem with your modem, not your RRD system.
6. Obsolescence. Sometimes customers have fax machines that are so old that the Smithsonian is looking to add them to its collection. Ask your customers for the make and model of their fax machine, and then check to see if your current RRD application level supports that model. Keep in mind that if you are not using internal modems on the RRD server, and instead have serial modems attached to the RRD server, those serial modems might be the issue as well. Sometimes a simple reboot fixes the problem.

As you can see, Cerner's RRD application around faxing can get hung up in a lot of areas. That’s one reason why tracking faxes for a healthcare organization is a full-time job. The next time someone tells you, “It’s just a fax,” remember you have your organization’s reputation – as well as a lot of money - riding on the fax getting where it needs to go.

Prognosis: Troubleshooting Millennium’s RRD application is much easier when a full-time member of the IT team can own the maintenance and support of this important faxing service.

 

New Blogger on the Block

By Chris Lanaman

I’m excited to join the distinguished Softek team of Millennium Medics. For the past 17 months, the bloggers have been offering real-world answers for common performance issues with the Cerner Millennium^® system, the kind of straightforward actions those of us at client sites could readily implement. I should know. Until joining Softek at the end of the summer, I served as the Cerner implementation manager at Carondelet Health Network in Tucson. Over the past two years, I led the team that moved Carondelet from 2007.12 Cerner code to 2007.17 and then 2007.19. We also moved from the W32 version of Clinical Reporting to the new Clinical Reporting XR and installed the PowerChart^® Local Access backup system and PowerInsight^® report generating system. Before that I spent six years as an analyst at Cerner.

With my various roles and the titles they carried — at Softek I am now Director of Millennium Consulting Services — I have always had one primary responsibility: to survey the big picture and figure out how to put the pieces together to provide end-users a system they can actually use or, even more, a system they want to use. I believe that IT is an important tool in advancing healthcare and bringing order to the informational chaos that plagues the industry. All too often, however, clinicians resist adding technology to their workday. They want to care for patients, not spend time waiting for a computer to finish a command.

I can sympathize with their frustration, so I am committed to bringing a level of predictability to their everyday workflow. Once they begin electronic charting or ordering, we had better make sure the system performs as promised.

What can I bring to the Millennium discussion? Certainly not the in-depth system tuning and troubleshooting that David Hallberg has been serving up to system engineers and database administrators. My expertise lies on the application support side of the system. I will be able to address concerns from the clinicians’ point of view, to bridge the gap between the clinician and the system analyst. I hope my ideas will help your hospital’s IT and Support teams better understand each other and work together.

I’ll start off later this week with an exploration of the importance of faxes in the life of the primary care physician. Stay tuned.

Let’s Put Clothes on the Emperor

 By David Hallberg

When clients are looking for improved Millennium® performance – which they almost always are – our conversations often go like this:

Client: “Can you please help me parse and manage the RTMS timer files?”

Me: “Certainly. But I thought we wanted to improve Millennium’s performance for your clinicians and business staff. The RTMS timers will do nothing to assist in resolving performance for the point of care people and the business people utilizing Millennium.”

Client (with great indignation): “What are you talking about? The RTMS timers are THE performance metric for Millennium.”

With a name that stands for Real Time Monitoring System, you would assume that statement to be true. Its truth, however, depends on how you define the word performance. To clinicians and the business staff, performance relates to how quickly they can get their job accomplished. They are concerned with Millennium’s speed. The data center staff, especially those at hosting vendors, define performance as uptime. They are concerned with the amount of time that the backend systems are available. Some may boast that they offer a performance guarantee of 99.8%. Such assurances mean nothing to clinicians, who assume that the system should always be up. Their standard is a system that runs, in the words of a friend of mine, at blink speed.  

And while RTMS timers once did measure speed, Millennium architecture changes have altered the meaning of the data the timers deliver. The result is that end-users might complain that the system is slow but the metrics look fine inside the data center. A bit of history will help explain the discrepancies.

Millennium did not originally have RTMS timers in its code. Instead, the middleware had a setting to allow the Citrix server or fat client to log all of the transactions being sent from that device to the application node. In the days when the applications and database resided on the same node, this log was a great metric. However, the setting had some drawbacks. There would be an observable performance degradation in Millennium applications with this logging turned on (degradation of 10 percent or more was not unheard of), and the approach did not show any grouping of applications and functions. So it was a great tool for the middleware team but not so helpful for the application engineering teams.

A grouping feature was eventually deployed. Initially, there was a high degree of linking between the functions being measured between the Citrix server or fat client and the application node, which proved helpful in seeing backend performance issues. Over time, though, we have seen that Windows tab changes and other functions actually only measure how fast the Citrix server or fat client hardware is running rather than how long it takes the application and database nodes to process a transaction for the end-user. I call this “RTMS timer death.” You now have lots of functions being timed, but you can no longer be assured that the function being measured is actually providing a roundtrip view of the transaction time.

Several other changes are also obscuring the quality of the RTMS timer data.

1.    A greater and greater reliance on asynchronous work leads the interactive user to believe the system is more responsive, but it is in fact a bit of a performance cheat. For instance, I may do a function in PowerChart in which a request is sent to CPM Script and CPM Script responds when the function is complete. In fact, even though CPM Script is done with the function, CPM Script actually sent the request to CPM Process and CPM Process exploded my request into numerous other requests that it then handed to other Millennium executables.

2.    If the Citrix or fat client application (e.g., PowerChart) crashes or is X’d out, you will see a DBAPI_Exception event for the Millennium application executables that were processing the requests from this client. The Data or Record information starts with “DbDestroyTimer: invalid object handle: xxx,” where xxx is the handle being used to update the RTMS timer data. What does this mean? It’s incomplete data. It’s not clear that each of these events represents an RTMS timer that started but never finished, nor is it intuitive that there is a relevant impact, much less intervention, to consider.

3.    Two files in $cer_mgr – slareporting.cerner and timer.cerner – control whether a timer is active, what functions are used, if the timer meets the hit/miss count, and the timer’s average or mean time. Though it wasn’t originally the case, the CRM Timer information is now configured in the timer.cerner file. In the slareporting.cerner file, the 54 application timers are defined. In the timer.cerner file, the 1,022 functions are defined. This would seem pretty good – until you remember that there are more than 35 million lines of code for Millennium, more than 4GB of Citrix and fat client files, and more than 5,000 Oracle tables. In the slareporting.cerner file, by definition, you have to have each of the 54 applications run 500 times in two hours in order for the data to be counted. You also have to have each application run at least 10,000 times a month or the data does not count.

Let me use PowerChart to explain the potential for problems. Let’s assume that it’s 7 a.m. and physicians are starting to round, nurses are changing shifts and many other clinicians are getting down to their day’s work. In all, 499 clinicians log into PowerChart from 7-9 a.m. The IT department gets reports from these users that Millennium is slow. IT checks the slareporting.cerner file and sees nothing counted for the day’s performance. Why? The slareporting.cerner file must have a minimum of 500 functions of an application – like logging into PowerChart – in a two-hour window, or they do not count. Over the course of a month, if your site did 9,999 PowerChart logins, it would be stated that you did not do enough work to be statistically significant. For the purposes of a Lights On Network™ evalution, this data would not be considered substantial enough for valid comparison. Therefore, if you logged a Service Request stating you are having performance issues, you are likely to be told that based on the data you do not have any issues.

The initial intent of the RTMS timers was right on the money: to create a mechanism to allow Millennium sites to look at a clinical function and understand where the issue is. However, the execution and changes over time have made the timers unusable. Every client who I have worked with in processing their RTMS data has logged Service Requests and been told either 1) there is no problem or 2) a code level update will make the problem go away.

This approach to analytics showcases the different definitions of performance that I mentioned earlier. The clinical and business staffs are telling you that Millennium is slow and unpredictable, but the data shows no performance issues. This conundrum with the impenetrability of the data leads many organizations to conclude that the environment is unmanageable and they need to outsource the support for Millennium Citrix servers, application nodes, Oracle databases, etc. After all, the outsource vendors guarantee “performance” to be 99.8 percent.

I’ve already discussed how meaningless that statistic is to everyone outside the data center. A 99.8 percent uptime from the clinicians’ perspective makes no claims to speed and probably means that the .2 percent downtime will occur when they need the system the most. Here’s one more caveat: The statistic refers only to unplanned downtime. The two to four hours of scheduled downtime for weekly preventative maintenance are not included in the number. I have been in healthcare IT since 1989. I have been employed by four different hospital systems, have been an HIT consultant and have worked for Cerner, and I have never been allowed to have that much downtime in a month, let alone a week.

Nor was I allowed to ignore the clinicians’ desire for blink speed. When I did performance benchmarking at a facility running a different clinical system, we had the system tuned so we could do the screen changes in 0.6 seconds. Seriously. On one occasion we had a drive problem that spiked the time to 0.9 seconds. The screen flips were 0.3 seconds slower, about the time it takes to blink your eyes, and our help desk phones were on fire from the outraged clinicians complaining of performance problems.

So what can you do to help resolve your performance problems? The first step is probably the hardest. Stop believing the propaganda that your system is fine. Anyone telling you that you don’t have a performance issue that can be tuned is like the “tailor” telling the emperor that his new clothes are exquisite. They’re trying to convince you of something your eyes and brain know is not true. Listen instead to the people who interactively use your system every day. The clinicians and business staff will be the first to notice a difference in performance. The next step is trying to understand what you can do to monitor the performance of all your Millennium systems. The last step is to determine what staff and budget changes you need to make to address the issues.

There is a lot you can do with the tools you already have. You’ll find lots of tips in previous blog postings. I also encourage you to respond to this blog with your own questions and insights. I welcome a lively dialogue.

If You Rewrite the Rules…

By David Hallberg

How I love rules. By following certain clear principles, you can be reasonably sure of the outcomes. If you wash your hands regularly, then you’ll spread fewer diseases. If you listen carefully, then you’ll be better able to address a person’s concerns. In the IT world, it’s called the if-then-else logic loop. But I’m the first to admit that rules have limitations. I’ll leave you to figure out their limitations in your personal life, but this week I’ll help you determine if some of the rules you have set up in Millennium® are a bit broken.

Millennium has a wonderful subsystem called Discern EKS, which allows users to identify special events that will automatically trigger other events. For instance, users can set a rule that whenever a patient’s white blood cell count is very high, say 15,000 to 30,000, an alert suggesting a blood culture will be generated. Each department determines the rules it needs based on its standard operating procedures, works with the appropriate teams in the organization to set them up and outlines its response when the rules are triggered.

It sounds straightforward enough, and most organizations don’t give much thought to the rules if they are running in less than a second or two. Run time alone, however, does not tell the whole story. Organizations can have hundreds or thousands of rules, some of which fire thousands or tens of thousands of times a day. They can present a huge drag on the Millennium system, and these tuning issues need to be addressed.

For example, if a rule takes one second to run – which would be considered a reasonably fast rule – but is triggered 1,000 times in one day, that one rule is using up 1,000 seconds – or over 16 minutes – of application CPU time. Rules firing more than once per minute – more than 1,440 times a day – need closer scrutiny. I would recommend that you look at that rule’s trigger mechanism and verify it really should be firing that frequently.

Rules that trigger once per minute are not uncommon. I’ve worked with clients that have rules triggering more than once per second – that’s 86,400 times a day. Now consider that many sites have hundreds or thousands of these rules. Even if all of them are so-called fast rules, you can still see where you might have Millennium performance issues.

To determine if you’re having issues with a rule or set of rules, we’ll look at the Oracle table EKS_MODULE_AUDIT. This table shows when rules are triggered and how long they take to fire, along with a lot more useful information. To learn how to access the EKS audit table and analyze its numbers, click here.

Prognosis: An analysis of the time Millennium spends each day running rules will show you whether any rules need to be rewritten to help the system run faster.

Oracle Tables Expose Hidden Issues

By David Hallberg

In working with a couple of clients, I discovered a large number of Millennium® Oracle tables that were used for logging or errors (like a message log) — more than 90 in the 2007.18 and above schema. I find this attribute perplexing. If you wanted to put all of the error and logging information in the database, why create the message logs, .err and .out files? On a practical level, that means analysis of application or stability issues requires users not only to gather data from the message logs but also to run a count of these 90+ Oracle tables to determine which tables have a lot of activity and then run a retrieval to get the information from those Oracle tables. No wonder many don’t bother looking for issues proactively — the task is simply too daunting.

Despite the magnitude of the task, I believe it’s important to do all you can to keep Millennium running as error-free as possible. And one way to accomplish this goal is to monitor error messages and logs and resolve the issues they show you. Last month I wrote about how to find and correct quickly wrapping message logs ( “What Is Measured Improves” ). Today I’ll list the queries you can run to see if any of these 90+ Oracle tables are active on your system — if, for instance, they include more than 10 rows of information, or whatever amount your team sets as its threshold. To see the list, click here.

This analysis is related to the previous blog on message logs and will use a similar methodology for reviewing the Oracle tables. You want to view activity in the tables over time. Run the queries I have listed and gather the row counts, and then run the queries again in a day or week or month and see which ones have grown.

Due to the fact that the tables are based on the Millennium solutions you have configured for your own site, I cannot provide specific guidance on what to look for and how to address an issue. I can, however, provide a little insight into a few of the tables:

· CPMPROCESS_ERROR is important to anyone still on 2001.x or earlier code. This is the Oracle table that the CPM Process server and any clones of CPM Process use when they cannot complete a transaction.

· For the CQM_MDRESULTS_LOG, CQM_MICRESULTS_LOG and CQM_OENTXLOG_QUE tables, you want to make certain that all of the rows are only as old as your purge criterion. For instance, if your purge criterion is seven days, then there should not be any records in these tables more than eight days old. If there are, you need to manually purge them and log a service request (SR) with Cerner to get a purge script that works correctly.

· CHART_SERV_LOG is important to monitor to verify that the chart servers are not sending too much logging information to the database.

· The DCP_ERROR_LOG and ORM_ERROR_LOG are for DCP and order errors. They are often in the message logs, but their inclusion here helps consolidate several of these messages from all of the application nodes into a single queriable location.

· If you are a Millennium Profit client, the CSI_LOG and CSI_ERROR_LOG tables are important to monitor for interfaced charges and their errors.

· The GL_TRANS_LOG for general ledger events and PFT_*_LOG tables for general Profit issues should be monitored and issues rectified as quickly as possible to minimize cash flow and reimbursement issues.

As I recommended in the previous blog, your staff will need to log SRs with Cerner to figure out how to correct your errors and then create and execute an action plan.

Prognosis: Maintaining a list of Oracle tables and knowing their rate of change or growth can be time-consuming, but the work you do up front to catch issues early will mitigate problems for the clinicians and support staff in the future.

A Shovelful of Refresh Reality

By David Hallberg

I was shoveling my driveway this past weekend with three 9-year-old helpers. They sort of shoveled but mainly just loved making snow piles and jumping on them. With high hopes, my son asked me if the three of them could shovel the neighbors’ driveways and get paid for it. Sure, I said, as I outlined the process: Neighbors would not pay him until they could see the concrete in the driveway; we needed to finish our own driveway first; working very hard with our two shovels, our driveway would take at least two more hours. In no time at all, my realistic estimate had shattered his dream of making lots of easy money.  

That scene reminded me of a time at a previous job when every week the technical team I was leading was blamed by the Millennium project manager for missed deadlines. To figure out the problem, we arranged a meeting between the application and technology teams. The meeting revealed a number of communication and procedural issues. Among them, the tech team stated that it only needed one day to refresh a domain. This estimate seemed really optimistic to me, so I dug a bit deeper into the steps required for the refresh. I discovered that this time frame was based on the following assumptions:

1.    The tech team would receive at least a seven-day notice that a domain refresh was needed. That would provide ample time for a full backup of production and for our off-site tape storage vendor to bring the tapes back without a rush charge.

2.    The tech team would have enough space in the non-production domains’ storage area network (SAN) to restore production.

3.    Once we had completed the database restore and cer_exe copy on the back end and run a scan for components or reinstall on Citrix, we would be done.

So if the application gave us a seven-day warning before the project needed a domain refreshed and if we had a full backup and if we had enough disk space on the SAN and if we had a clean reinstall or clean scan for components and if the tech team worked 24 hours a day seven days a week, the domain refresh could be done in one day. I still shake my head at the memory of this conversation. The reality was that few of these things ever happened and none of them ever happened all together.

1.    We never had seven days’ notice.

2.    On several occasions we paid for our off-site tapes to be brought back (yes, I took the beating for that cost).

3.    Only once did we have enough SAN disk space. Generally, we would have to stop the restore several times to add more.

4.    The scan for components rarely was completed without issue.

After bringing reality into the domain-refresh process, I modified the project estimate from one day for the tech team to hand over a domain to three to five business days. Did this stop the finger-pointing? Quite the opposite, the application team went crazy. They now blamed the tech team for not delivering domains. I tried to appease them: “Well, if you can tell me how many refreshes and the exact dates you need them, I can modify the estimate.” Oddly enough, no one — not even the project manager — could tell us the number of refreshes they needed, let alone the dates.

I find it ironic that my 9-year-old son had the same unrealistic optimism for completing a task as the 20- to 50-year-olds on my tech team. They also shared the same look of discouragement when I tempered their optimism with the facts. But they eventually reaped the benefits. At work, the project team accepted the new estimates, the tech team was able to put in relatively normal 10- to 12-hour days during a refresh, and we began meeting our deadlines. At home, my son jumped in the snow until he couldn’t feel his toes and then happily warmed up with a cup of cocoa.

Prognosis:  Work is no place for a snow job. A realistic assessment of procedures and the time required to accomplish them will keep teams working well together — and meeting project deadlines.

A Healthy Dose of Doubt

By David Hallberg

 “…and the only people I fear are those who never have doubts.”

– Billy Joel, “Shades of Grey”

One of the most frustrating discoveries I’ve made about Millennium’s technical infrastructure is that tuning is never done. If I think I’ve gotten all of the performance out of Millennium that I can, I need to step back and re-evaluate my metrics…or just wait a little while for other issues to arise.

Another important realization, one that I learned a long time ago, is that my knowledge and experience is not the pinnacle of knowledge and experience. No matter how hard I try I can never know all there is to know on a given topic. There is always someone else who has dealt with a similar performance/stability/capacity issue and knows more about it than I do. When working with clients, I prefer a collaborative approach that comes with a little doubt that the problem solving will ever be done.  Performance is an iterative process that has to be monitored and adjusted through time, not just at a single point in time.

Millennium is a large, complex suite of applications with many inter-relationships and impacts that are not easily visible. Millennium requires DNS servers, networks, Citrix servers, Windows servers, Oracle, SQL Server (both for Citrix and some applications), MySQL, Oracle with raw logical volumes, Oracle with the Veritas File System, Oracle with ASM, AIX, HP-UX, VMS, thin clients, WebSphere application servers, WebSphereMQ, Apache web servers, load balancers, Java, Win32, ActiveX components. The list goes on and on. Plus, the Millennium environment is never static. There is always another project, another service package to install, another site to bring into production. Every change in the environment means changes in Millennium and affects how stable Millennium is and how well it performs.

All of these variables should help keep us humble. When I was in consulting and managing Millennium directly, I knew that if a client followed all of my recommendations, we could typically expect a 90- to 180-day period of stability. After that, change would happen. Ultimately, the earlier tuning would help us adapt more easily to these changes, but there would always be a pain period after a change in Millennium. The tuning was never done.

I work very hard to understand Millennium and share this knowledge with Millennium clients. In the process, I learn from each client I interact with. I’ve recently seen two extremes around this point: One organization benefited from this collaborative approach and from having a little doubt that they had done everything possible to tune Millennium, and another was certain they had done all the tuning they could.

The first client said that they just had the “normal” Millennium performance issues but asked me to evaluate the system and suggest ways to improve it. The Messaging Performance Audit revealed several straightforward, easy-to-resolve issues as well as others that required new Millennium front-end code. Here was a person willing to admit some doubt that his team was doing all it could to improve the hospital’s electronic medical record system. After implementing some of the recommendations, they saw quantifiable benefits.

On the flip side, a second client was experiencing significant performance issues in production after introducing key changes to their environment. The system basically came to a crawl every day from 9-11 a.m. and 1-3 p.m. Analysis revealed disk I/O issues. They had not implemented the baseline Cerner Millennium System Settings (CMSS). They had not tuned the operating system’s disk buffering memory regions. Response times from their internal SCSI drives were taking 20 milliseconds instead of three to eight. The Oracle database showed some large waits for disk writes to complete.

When I presented my findings and recommendations, the lead technical person dismissed them. There was no doubt in his mind that his team had done all they could to address the problems and that I was completely wrong. He was beyond adamant. His team continued to address the escalating problem simply by logging Millennium Service Requests until clinicians revolted. Now the client is paying Cerner and other outside consultants to come in and save the day. No doubt proper tuning would have cost a whole lot less and, more importantly, would not have impacted the clinicians’ ability to deliver patient care.

Prognosis: Being willing to doubt we have done everything possible to make Millennium as fast as it can be opens us to new ways of providing clinicians a more reliable, consistent EMR system.

The Mystery of the Changing Citrix Registry Key

By Matt Tran

At a site I recently talked with – and where I had once worked – the technical team stumbled upon an issue where specific Cerner registry keys were being changed by an application on the Citrix servers. Users were reporting that they would log into Millennium®, but nothing would happen following the login splash screen. The technical team would identify which server the users were logging onto and begin its troubleshooting. Analysts were able to identify that the registry values were not the default values from their image. Once they corrected the registry values, the applications worked fine. With no knowledge as to why the registry values had changed, however, analysts couldn’t resolve the issue. They just waited for another user to call in the problem and then wait while they corrected it.

Back when I worked at this site, we had a similar issue. So I wrote a script that monitored for registry changes to specific keys and then emailed/paged our on-call analyst when these changes occurred. This notification allowed our analysts to quickly log onto the server and correct the registry keys before our end users were impacted. Additionally, the notification provided a log of all these occurrences and allowed us to capture more information.

This very basic script allows you to specify the registry keys you wish to monitor and the email address where notifications will be sent. If you are interested in using this script, please send your request to millennium.medic@softekinc.com with Registry Listener Script in the subject line. We will be happy to share it with you. If you have encountered or resolved this issue in the past, please comment below.

Prognosis: Early detection and quick response to registry changes can reduce user frustration and calls to the help desk.

In Lean Times, Lean on Creative Solutions

By David Hallberg

Last year’s economic tailspin is old news, but its downstream impact on the healthcare environment is very much today’s headline. Staff are being cut, operational budgets are being slashed, and capital budgets are disappearing more rapidly than you can cut a P.O. How can you still manage a complex client/server architecture in this environment?

With patience, planning and a large dose of creativity.

I’ve been in healthcare IT for well over 17 years and have seen my share of budget cuts. This latest round, while severe, is just that: the latest round. And it’s a reminder that our work in healthcare IT is a marathon. To finish strong you need to focus on the basics: determining who the key people are in your organization and trusting their skill and creativity to pull you out of your nose dive.

When I started in healthcare, our capital limit was $300. Staff reductions and capital expenditure cutbacks were the norm. When a PC died or was needed, my team had to order pieces and parts and assemble them. If a printer died, we had to cannibalize other dead printers to build one that worked. We knew what it meant to “reduce, reuse, and recycle.” And it seems like these current times offer the perfect challenge for a staff to relearn these three R’s.

Start by determining what resources are available. Do you have a bunch of old PCs, switches, routers, servers, and old SANS that were too slow for the clinical application or PACS system? Instead of discarding all of this hardware, you may be able to refurbish it. You’ll be trading capital costs for labor, but capital expenditures are usually harder to get approved. You are much more likely to win a battle for increased overtime required to keep a department’s systems running.

During lean times a number of years ago, I started evaluating the cpu usage on some application servers(hardware) and found that while they were running on some very powerful equipment, they were hardly using any cpu. So I began moving our older servers to be the application servers for these applications. I also invested in creating an inventory of all of the hardware in our data centers and storage areas. I discovered several racks of gear that were still powered up and running but doing nothing. Their applications had moved to newer hardware or a different data center years before, but no one had bothered to turn them off. This utilization pattern has been one of the major reasons virtualization has been so successful.

One of the drawbacks to reusing older hardware is the work needed to get it to a functional state before redeploying. You have to upgrade BIOS’s, device drivers, perhaps hard drives, microcode, etc. Next, you have to determine which hardware is underutilized and which is running hot. Then the jigsaw puzzle begins as you take all of these pieces and determine the best fit.

Here’s where you need to rely on key members of your staff. It takes a lot of energy to collate all of the data and create a plan of action…and a lot of patience as you work your way through the multiple managers/directors who must sign off on testing  before the migrations can take place. After the migrations, you need to keep track of performance and utilization to ensure the migration has not degraded clinical or business performance and to ensure the capacity of the hardware is not saturated or bottlenecking in new places.

Prognosis: Discovering and redeploying old, underused equipment may seem like a daunting task, but it might be a realistic solution for a budget-strapped IT department staffed by creative problem solvers.

Who Zooms Out to See the Big Picture?

By Matt Tran

It seems that many hospitals tend to fall into a scenario where their support teams are very focused on their own particular discipline. This focus works well in theory because it creates experts in each area. A team assigned to PowerChart, for example, can maximize the abilities of the application quickly with deep knowledge. In reality, however, too much IT specialization can create an environment where many responsibilities fall through the cracks.

Cerner Millennium is a highly integrated system that weaves together many applications. A focused approach falls short when one support analyst is unaware of what other analysts are doing in the system and nobody is responsible for overall coordination. For example, this tunnel vision can hamper the efficient running of batch scheduled jobs, also known as Ops Jobs. If each application team were to own its own jobs, the overall scheduling and support of the jobs become lost.

Very recently, I talked with an organization that had too many Ops Jobs running concurrently but had no idea who should address the problem. This issue had a high impact on the performance and functionality in the Millennium applications and was affecting the clinician experience.

How does your organization manage these application tools that spread across multiple teams? Do you have an individual that maintains Ops Jobs? You can use the Comments link below to begin a dialog.