Millennium Medic

X Marks Our Spot to Meet with You at CHC11

By Chris Lanaman

If you’re one of the chosen few from your hospital coming to Kansas City for the Cerner Health Conference, you probably have a long to-do list: speakers, education sessions, exhibits, networking, a party or two. But I’m hoping you’ll strike out on your own at some point to visit with the team that’s been bringing you the Millennium Medic the past several years. Lucky Strike Lanes & Lounge just a few blocks east of the CHC activities will be our spot throughout the conference. Why do we think you’ll want to add us to your already busy agenda?

• We’re having a party, and you’re invited — Monday, Oct. 10^th, from 7 p.m. to midnight at Lucky Strike, 1370 Grand Blvd. We might be a bit biased, but we think it’ll be one of the best of the week. It’s Softek’s annual Client Appreciation Party, but we’re welcoming non-clients as well for an evening of food, games, music, bowling and lots of prizes (we’re talking cool electronics). Your CHC badge is your ticket to get in, or you can sign up ahead of time and get more details at the Softek website.
• Our Millennium experts are sharing their expertise. Where this blog is best used to address common system issues, a face-to-face meeting would let us answer your specific questions. Stop by Lucky Strike any time from Monday morning to Wednesday noon — we’ll even be there Tuesday night overlooking the Big Bash in the Power & Light District. Come talk with us and enjoy a free meal and bowling. If you prefer to set up an appointment, call Erin at 913.981.5242.
• We know more about your system than you might think. Softek conducts independent audits of the Millennium production environment, and we can show you what we’ve discovered in every audit we’ve ever run, findings that have huge clinical and financial implications.
• We’ve got our head in the clouds. Softek OnTrack’s new cloud access is part of our commitment to opening access to Millennium users to more information about their system. We’d love to tell you about it: Millennium whitepapers and user training at OnTrack University, user forums, mobile applications for system management and other resources.

The CHC will be a busy week for education, association and recreation. We hope to enjoy all three with you.

Going to Kansas City for the Cerner Health Conference?

Many of you will be in Kansas City in a few weeks for the Cerner Health Conference. We’ll be on hand as well from our “headquarters” at Lucky Strike Lanes & Lounge in the Kansas City Power & Light District a few blocks east of Bartle Hall and the CHC. We’d love to talk with you in person about your Millennium® system and will have more details in the next blog concerning our activities for the week.

Today, however, we want you to get excited about the city we call home. Here’s a list of 10 places you might not have thought to visit while you’re here…but are definitely worth considering. Our potpourri includes places that appeal to all sorts of interests: history, art, architecture, sports, music, barbeque (you are in KC, after all) and even potpourri.

1. The story of the sinking of the steamboat Arabia in the Missouri River in 1856 is just as fascinating as its discovery in 1987 buried 45-feet deep in a cornfield half a mile from the river. The Arabia Steamboat Museum in the River Market documents it all and displays the china, food, guns and clothing preserved in the silt. Outside the museum, explore the Farmer’s Market and stroll the Riverfront Heritage Trail to Berkley Riverfront Park.
2. The Money Museum inside the Federal Reserve Bank of Kansas City gives you a view of the automated vault and sends you home with a bag of cash (sadly, it’s shredded). Across the street from the Fed, you can appreciate the irony in the 65-foot-high sculpture of cargo containers at Memorial Park. Sculptor John Salvest’s installation spells out its title: “IOU/USA.”
3. The National World War I Museum is housed in the Liberty Memorial across the lawn from the bank. Its 55,000 items include everything from common soldier gear to rare artifacts. The “Man & Machine” exhibit shows German technology never before displayed in this country.
4. The National Archives west of the Liberty Memorial takes you back to an earlier war. The “Lee and Grant” exhibit contains artifacts and writings of two great Civil War generals.
5. The newly opened Kauffman Center for the Performing Arts south of Bartle Hall owes its impressive front facade to the 27 cables supporting the glass and steel structure. Tours are available, and multimedia artist Laurie Anderson performs “Transitory Life” at 7 p.m. on Oct. 9. For another KC architectural gem, visit the Bloch Building at the Nelson-Atkins Museum of Art. View 40 bronze sculptures by Auguste Rodin (of "The Thinker" fame) throughout the lobby.
6. Local entrepreneur Mary Carol Garrity, owner of the Nell Hill’s retail home stores, will speak at the downtown Kansas City Public Library at 6:30 p.m. on Oct. 11. Also at the library is an exhibit of original illustrations by Missourian Thomas Hart Benton for some of Mark Twain’s signature works.
7. KC’s most famous hometown brewery, the Boulevard Brewing Co., gives free tours (including tastings). Reservations are required. If you prefer the taste of coffee, visit the Roasterie Kansas City down the road from the brewery.
8. Celebrate KC’s rich music and food tradition at the annual Rhythm and Ribs festival Oct. 8 at the American Jazz Museum in KC's 18^th and Vine district. The daylong festival features top jazz and blues performers and local food vendors. Also at 18^th and Vine is the Negro Leagues Baseball Museum.
9. If you prefer a faster pace, visit the College Basketball Experience inside the Sprint Center. Besides learning all you want about the game, you’ll find lots of areas for play.
10. It’s race weekend at the Kansas Speedway just a short interstate drive west of the metro area. Oct. 8 is the NASCAR Nationwide Series’ Kansas Lottery 300, and Oct. 9 is the NASCAR Sprint Cup Series’ Hollywood Casino 400. Next to the track is the Legends outlet mall, several major retailers, land for the soon-to-be-built Cerner office buildings and the Livestrong Sporting Park. Major League Soccer’s Sporting KC plays Chivas Guadalajara at Livestrong on Oct. 12. 

You might be coming to the CHC to learn and spend time with colleagues, but we hope you also take time to enjoy a few of the places and activities unique to KC. 

Have You Talked with the Clinicians Today?

By David Hallberg

How do you begin to manage an EMR system as complex as Cerner Millennium^® — with a back end containing more than 35 million lines of code from Cerner along with Linux, AIX, HP-UX, VMS, Oracle, WebSphere MQ, Sonic ESB, MySQL, Java, storage area networks (SANs), networks, clustering software, network connections between Oracle instances, Citrix, Windows servers and desktops; with a front end that has more than 4 GB of space in its code warehouse; and with countless application builds?

I recommend you start your work by stepping away from your computer screens and talking to the people who are using the system to care for patients. Are they frustrated with response times? System availability? The number of clicks to admit or discharge a patient? Don’t dismiss their complaints with the assumption that “clinicians just don’t understand computers.” They are highly skilled people who understand the most complex system on the planet: human life. You might be surprised what you can learn when you take the time to really listen. Or shadow them during their rounds.

When I worked at Cerner, I brought my stopwatch with me to one hospital to join the CMIO and director of nursing informatics on the floor. It took more than 15 seconds to place one order in PowerChart^®. That’s an eternity when you’re with a patient and still have the rest of your rounds to complete. When I took the data back to the engineering lead responsible for the code, she emphatically told me, “Fifteen seconds is fast enough. They (the clinicians) just do not understand computers.” No wonder the clinical staff was frustrated. The system was hampering their ability to provide patient care, but their concerns weren’t given any consideration by the people capable of addressing the problems.

So I encourage you to gather the staff’s list of concerns, criticisms and objections. If they feel they’re not normally heard, no matter how bad their experiences, don’t be surprised if they’re distrustful at first and reluctant to talk. Keep at it. Move through the obligatory moans and groans to get to their true desires.

Now that the easy work is done, begin to rebuild their faith in the system by evaluating each complaint against each need and determine how to address them. You can use Millennium’s greatest asset, its flexibility, to configure it for very specific purposes. If the issues outweigh your staff’s ability to correct them — we’re back to that complexity I mentioned earlier — don’t be too proud to seek outside help. (If you’ll be at the Cerner Health Conference next month in Kansas City, I’d be happy to listen to your system concerns and talk through some options.) Then you can begin the journey of reconfiguring this multifaceted solution set into the system your clinicians actually enjoy using rather than avoid using or simply tolerate.

Striking up a Conversation

One of the strengths of an online community is its collective knowledge — the collaboration and innovation of many that can accomplish far more than any one individual or organization. It’s an open-source culture that has expanded way beyond the software community.

At a conference in Kansas City last week, fusion-physicist-turned-farmer Marcin Jakubowski told about Open Source Ecology, a network of farmers, engineers and supporters designing and posting do-it-yourself schematics for what they see as the world’s 50 most important machines. At a 30-acre site in northwest Missouri, workers have prototyped the first eight modular machines in this collaborative Global Village Construction Set. They envision their designs being used to “lower the barriers for entry into farming, building and manufacturing” around the globe. Another conference speaker, crisis mapper Patrick Meier, talked about the open-source Ushahidi mapping software and legion of worldwide volunteers that have helped governments and the United Nations pinpoint particular people and areas devastated in such recent and ongoing crisis areas as Haiti, Libya and Somalia.

Our humble blog can’t boast such expansive life-saving work, but we’re glad to be a part of an online community working together to improve the world of healthcare IT (which actually can save lives). For the past several years, we’ve addressed many of your concerns with our practical advice for maintaining and troubleshooting your complex Cerner Millennium® EMR system. We’ve even compiled a number of these into in-depth papers on such topics as message log monitoring, tips for remote-hosted organizations and steps to minimize risks during upgrades. These and more are available on our website.

To help us continue to cover the issues affecting your organization, we’d love to hear from you. Where specifically are your pain points with your EMR? In what internal or external areas do you need more understanding? What creative solutions has your IT team found? We will endeavor to address whatever ideas you suggest.

Just click the Comment link below to get the conversation started.

ESI Logs Detail Lost Transactions

By David Hallberg

During the past several years, I have seen ESI (External Systems Inbound) logs detailing problems with the inbound interfaces for Millennium^® systems. Unfortunately, people often don’t believe the data I show them. If there were interface issues, I am told, lots of people would know. I wish that were true. When an interface or Millennium first went live, the interfaces might have been error-free, but errors invariably are introduced over the years following changes on the sending system and in Millennium. Rather than deny their existence, users should be working to identify errors and warnings and determine how to correct these issues.

An interface is a group of executables that communicates with another system, like an interface engine or a registration system. These executables allow the systems to communicate without a person reading data from one system and retyping that data into another. The data can be registration or demographic information, such as the patient name, medical record number, date of birth, next of kin, etc. Or it may be clinical information, like a laboratory or radiology order or information from pharmacy dispensing machines. If an error occurs with one of these transactions, the information might not get to its destination, which can lead to sub-optimal clinical care and sub-optimal financial health for the organization. A check of the ESI message logs will help you identify these errors.

Before telling you how to do just that, here’s a quick cheat sheet of the HL7 acronyms of the transaction types (msh_msg_type) you’re likely to encounter:

DFT — Detail Financial Transaction

ORM — Order or pharmacy order

ORU — Order status update, new order or point-of-care observation

SIU — Schedule Information Unsolicited

ADT — Admissions, Discharges and Transfers (demographics)

MDM —  Medical Document Management transaction (e.g., transcribed results)

A complete list of the transaction or message types that compose the HL7 standards can be downloaded from http://www.hl7.org/implement/standards/index.cfm for a nominal fee.

The first step to diagnosing ESI issues is to gather your data. Create two files, one for ESI warnings and one for ESI errors, and run the following queries:

For warnings from CCL:

Rdb SELECT msh_msg_type, msh_msg_trig, msh_sending_app, error_text from ESI_LOG where error_stat = 'ESI_STAT_WARNING' go

For warnings from SQL:

SELECT msh_msg_type, msh_msg_trig, msh_sending_app, error_text from ESI_LOG where error_stat = 'ESI_STAT_WARNING’;

For errors from CCL:

Rdb SELECT msh_msg_type, msh_msg_trig, msh_sending_app, error_text from ESI_LOG where error_stat = 'ESI_STAT_FAILURE' go

For errors from SQL:

SELECT msh_msg_type, msh_msg_trig, msh_sending_app, error_text from ESI_LOG where error_stat = 'ESI_STAT_FAILURE’;

The data will be returned in four columns, which can be pulled into Excel or any spreadsheet or database software and sorted. Create pivot tables that show what message types are having issues. It might be obvious that errors represent failures, but be aware that warnings are often transaction failures as well. For example, a warning of msh_msg_type of DFT with an msh_msg_trig of P03 will have a message such as “Invalid Service Resource Code,” “No rows exist on bill_item_modifier table for this CDM” or “No Matching Debit Charge.” Each represents a financial transaction that did not update the financial record, which means your organization was not compensated for the care you provided.

ESI warnings also result if an appropriate alias has not been built. The ESI server will not understand where to put the data and will send a warning to the message log, but nobody will know that the transaction failed unless someone is checking the log.

ESI failures are often build or configuration issues, such as a missing alias for a code set or missing priority codes, specimen type codes or detailed task assays. The interface might also be sending addendums to a transcribed report that is not in Millennium. No matter what the reason, none of the transactions updated the clinical record, and your clinicians are treating their patients without the most up-to-date information.

Once you have the number and type of events and understand their importance, you can prioritize how to analyze and resolve them.

Prognosis: A quick review of ESI logs can provide great insight into where Millennium might not be fully updating the clinical and financial database. Setting up the processes to review and resolve these events as quickly as your change control allows will provide great benefit to your patients.



Continuing to Improve Failover: Security Master Configuration

By David Hallberg

After our last post, “Improving Interface Failovers,” a few people wondered why some Cerner Millennium^® servers — most often Ops Job (Millennium batch process), interface and MDI servers — would not start during the failover. Because this situation can have disastrous consequences, it’s an issue worth addressing.

Imagine the following scenario: It’s 3 a.m. on Memorial Day, and your primary application node crashes. The failover scripts run so that users can log into Millennium, but none of the Ops Jobs are running. That means the nursing staff is missing care plans for its shift change, new patients and their lab results are not visible in Millennium, and the Emergency Department director is calling the CIO, COO and CEO complaining that his staff has to go on diversion because they cannot get the patients into Millennium. A storm just ruined your holiday plans.

Had you been aware of a few simple steps that would make your servers fail over successfully, your day could have been clear sailing.

Why didn’t the Ops Job servers start? If the server on a specific Server Control Panel (SCP) Entry ID has not been started on this node before, the Security Master server has to be running to create the service with the correct permissions. If no Security Master server is running after the failover process completes, any of the servers or services that have not gotten to a running state on the failover node prior to the failover cannot be started.

A little analysis will explain why this scenario happens so frequently. Way back when, about 90 days before you first brought Millennium up, consultants from the vendor swooped in to set up failover. They ran through a couple of tests and showed you how, should the primary node go down, you could still log into Millennium. They flew back home, and you turned your attention back to what seemed like more critical work that had to be done before go-live.

Now, many years later, you have added and deleted Ops Jobs, your old registration system has been upgraded, the inbound feed changed names to allow testing into Millennium production and to minimize the cutover time, you added about a dozen MDI servers, you moved on to a different Millennium client, or you were promoted, and your project staff is scattered around various clinical areas. Go-live and pre-go-live are remembered for the number of service packs you had to install and test and the huge amount of training you had to do. Who recalls the little detail about the need for the Security Master to be running to allow a new Millennium server to start on a node?

What, in fact, is the Security Master? For organizations using LDAP to access and maintain their user directory, the Security Master server updates the sec_user.dat file with local accounts, for example SYSTEM, SYSTEMOE and CERNER. For those not using LDAP, the Security Master also is used to add, remove, modify and delete all user IDs. So when a new person joins your organization and you run the tool HNAUser, a new row is added to sec_user.dat as well as some Oracle tables. Additionally, the Security Master creates a service key for the Millennium processes as they start up. These service keys only exist in the sec_user.dat file; they are not kept in LDAP. This key is used to make sure the processes running on the application nodes are legitimate processes. During normal operations, no one really cares about the security services. They quietly do their work when you stop and start Millennium servers on the application nodes, and everyone — especially your security manager — is happy. Until a failover.

Let’s go back to our Memorial Day crisis, where your family headed to the lake without you. You, like most Millennium clients, have Node 1 of SCP Entry ID 32 configured as the Security Master and Node 2 as the Security Slave. With this common configuration, however, you do not fail over the Security Master, so all of the Ops Jobs you added after go-live, along with new interfaces and MDI servers, didn’t seem to start. Yet nothing shows in any of the message logs. You called your vendor’s Immediate Response Center, but the response has not been immediate enough. After calling you back in 15 to 45 minutes to verify you are still having the problem, IRC might take four more hours to contact you again. In the meantime, the ED director is calling for your resignation and the ICU, CCU and NNICU staff are looking for a throat to choke.

How could you have prevented the chaos and been fishing for something other than answers? The single biggest thing you can do is NOT put a Security Slave on SCP Entry ID 32. Use any other SCP Entry ID — 31 and 19 are often open. After you have the Security Slave configured to run on a different SCP Entry ID, copy the Security Master from the primary node to the secondary nodes. Verify that the Instance is set to 0, which will ensure that SCP Entry ID 32 does not start up and that CPM HNAM Agent does not start it. Finally, put 32 in the cerner.vars file as a single-instance server. That way, when the failover scripts run, you will be able to start the Ops Jobs, interfaces and MDIs as needed to minimize your downtime.

Here’s one last detail for all the system administrators who like such specifics: Create a chron job to automatically copy the $cer_config’s security.jnl file from the application node where the Security Master is running to the other application nodes. Then verify that the permissions are “-rw-rw-----” and the owner and group (assuming your domain name is prod) are both d_prod.

Prognosis: A little preparation can dramatically improve the failover process and make your night, weekend and holiday events much shorter and quieter.

Improving Interface Failovers

By David Hallberg

Today I'll tackle a topic new to this blog: successful failover. Your organization, like many that I talk with, might have brought in a very highly paid consultant to set up failover for your Cerner Millennium® system. Yet, when you need to fail one application node to another to perform maintenance, the interfaces don't fail over correctly. Let me give you a few ideas that might help — and other than a change in your thinking, my ideas won't cost you a thing.

Cerner has extended some functionality in Open Engine's (OEN) Oracle tables that makes failover less painful for its clients and their Millennium support team. Users can set up the system so that the base Millennium failover scripts automatically start the interfaces on the failover node. (In the OEN_PERSONALITY table, where the column Name is PRODUCTION, the flag would be set to Y.)

I know I'm in danger of losing the OEN team here, but keep breathing and hear me out. Typically, the OEN team believes that nothing should be automatically failed over because you could have synchronization issues when more than one registration system sends Millennium ADT transactions. Even single ADT systems, where other systems send Millennium orders, order statuses, transcriptions, etc., can have synchronization issues. These issues can result in an order coming through before the patient has been added to Millennium. I don't dispute your concern. However, I absolutely do not agree that none of the interface servers can be automatically failed over.  

ESI Servers and Outbound Interfaces

Two types of servers, in particular, can be set to fail over automatically with no synchronization risk for incoming interfaces. They are:

1. The ESI servers, or ESI COMM servers. These are not SCP Entry ID 250 and its clones but the servers that take the Inbound client transactions and route them to SCP Entry ID 250 (CPM ESI) and its clones as well as SCP Entry ID 335 (PM Trans Async).
2. The Outbound interfaces, those that send transactions to systems outside of Millennium.

Finding these interface servers can be tricky — Millennium OpenView and Olympus make you go through several screens and tabs — so I suggest you start an interactive CCL session and run the following query to find the ESI COMM servers:

Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PROTOCOL' and op.value ='ESI')
order by pi.SCP_EID asc
GO

The resulting five columns will show you the ESI COMM servers, which can all have their PRODUCTION flag safely set to Y. They will only process transactions after you have started the Inbound COMM server that feeds these servers' transactions. That gives you one less set of servers that have to be started for the Inbound interfaces. I would suggest that the primary ADT Inbound interface be enabled for automatic failover. If during a controlled outage, you do not want the primary ADT interface to be up and running first after the failover, you can easily change the PRODUCTION flag back to N.

The interfaces configured to send transactions to systems outside of Millennium can be found by running this CCL command:

Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PACKESO' and op.value > '0')
order by pi.SCP_EID asc
GO

As with the ESI COMM servers, you can safely set the PRODUCTION flag to Y. Now as your clinicians are doing their normal work, the transactions configured to leave Millennium will not get hung up in Millennium for any length of time.

You might want to take one extra safety measure if you have Outbound pharmacy orders. SCP Entry ID 625 (Medication Dispense and Dose Server) and ID 626 (Medication Fill/Report Server) have a property called FsiCqmDowntimePref that can be added and configured. Set this property to 1 so that Outbound pharmacy transactions can be recovered if SCP Entry ID 252, the FSI ESO-CQM server, is down for any reason.

Additional Servers for Failover Node

Another factor that can impede interface failovers is if any of the following servers are not already running on the failover node:

• SCP Entry ID 250, the CPM ESI servers, along with any clones or copies. If no interface server is running on the failover node, then nothing should be sending these servers any transactions.
• SCP Entry ID 107, ORM Order Write Synch for FSI.
• SCP Entry ID 252, the FSI ESO-CQM server.
• SCP Entry ID 205, the Clinical Event server for ESI, and any of its clones or copies.

The cerner.vars file should be reviewed for the single-instance servers. The file should include at least the following servers: SCP Entry ID 240, 243, 245, 255, 270, 275, 276, 277, 280, 281, 282, 283, 284, 285, 291 and 292. I excluded MDI servers from this list because I am only reviewing interfaces and they involve a great number of variables, which I'll talk about later.

Keeping TCP/IP Ports Free

The last area I want to cover involves the TCP/IP ports. Interfaces will not fail over properly if the TCP/IP ports that you have configured the interfaces to communicate with are in use. How can the ports be in use? Through the normal cycling of servers, ports that initially were free can become used over time. You can prevent this by accessing the file named "/etc/services" and adding the incoming ports for the interfaces and even the MDI servers. For example, telnet is a service that runs on a port, normally 23. So the entry for this would look like:

telnet                23/tcp

If you want to reserve your Inbound interface and MDI ports, all you have to do is add the ports and a name to this file. I suggest you use the SCP Description (minus spaces) for the name along with the SCP Entry ID or OEN PID for clarity, as shown below:

telnet                23/tcp
ADTin                10046/tcp
ORUin               11001/tcp

Once added, these ports will not be used for anything else.

To create a list of the TCP ports you need to put in the "/etc/services" file, use the following command in CCL:

Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PORT' and op.value > '0')
order by pi.SCP_EID asc
GO

Dealing with MDIs

For MDIs, you have to run the command enumlog –all on the two nodes and verify that all of the MDI configurations exist on both nodes and match exactly on both nodes. Because typographical errors are extremely painful to find, I recommend you output the data to Excel and use the EXACT function to compare cells. It will quickly find any typographical errors.

If you want the MDIs to fail over automatically, you can use the following command for CCL to find them and then use OEN_PROC_VIEW GO to modify their PRODUCTION flag to Y:

Rdb select op.interfaceid, pi.SCP_EID, pi.proc_name, op.name, op.value
from oen_personality op, oen_procinfo pi
where op.interfaceid=pi.interfaceid
and (op.name = 'PROTOCOL' and op.value ='CUSTOM')
order by pi.SCP_EID asc
GO

I did not include the MDI servers in the cerner.vars list because too many variables exist for these. Which MDI servers need to be in the file and not able to run on the failover node depend on the code level you are on, the Millennium Distribution Packages you have taken and your volumes. If you have MDIs that are not in the new CareAware or iBus framework, log a Service Request to verify if your code level can have the MDI servers running on multiple application nodes or if you can only run a single instance on a single node. Review the server guide for the MDI servers, looking for the section titled "Considerations for Multiple-Node and Multiple-Instance Usage." This will let you know if you can have more than a single instance and if you can run the server on multiple nodes. I want to emphasize that if you have an MDI server, like SCP Entry ID 230 (MDI Genlab Result Client) and you can run multiple instances on multiple nodes, then on the failover node the server will take very little CPU or memory after it is started. It will idle until the failover event.

Prognosis: A little preparation work can dramatically ease interface and MDI failovers and make your night, weekend and holiday events much shorter and quieter.

Use Oracle Tables to Kill Blocking Sessions

By David Hallberg

Cerner Millennium^® clients I talk with have lots of questions about Oracle. Today I’ll talk about how you can improve system throughput with the information in Oracle’s v$session table or gv$session table for the production database or any database in an Oracle Real Application Cluster (RAC). This Oracle table keeps track of all of the connections (or sessions) on the database. The table shows where the connection came from, the Millennium application node where executables like CPM Script run, the status of the session, the logon time, the blocking status (when another session is preventing or blocking the work from being completed), and the event or description of what the session is doing.

Why is this information important? This one table can tell you about any long-running script — either the cause of the problem or whether the script is just taking the resources it needs to accomplish its work. Problem scripts mean clinicians and business staff are not able to place orders or see results, so these issues need to be addressed.

Let me begin with a couple of cautions. First, with more than 70 columns, the table is quite wide, which can make viewing the data in a telnet, ssh, SQL*Plus session or CCL session quite difficult. I strongly recommend using Excel or another spreadsheet to save the query output and evaluate the data. Additionally, only use this set of queries if you are having a problem getting data into or out of the database. It is completely normal to have transient or sporadic blocked sessions. They provide a safety mechanism for Oracle to maintain data integrity.

The script analysis begins by finding the blocked sessions. Please keep in mind as you execute the following commands that SQL*Plus requires a semicolon (;) at the end of each command and CCL requires the characters rdb and a space at the front of the command as well as a space and go at the end. For the first command, type:

select * from gv$session where blocking_session is not null

If nothing is returned, then there are no blocked sessions. If you do get rows of data back, then you have found the sessions that are compromising Millennium’s performance. To manage the data, you can reduce the number of columns to only retrieve the entries you need. Modify the query as follows:

select inst_id, sid, serial#, osuser, process, machine, terminal, program, type, logon_time, current_queue_duration, blocking_session_status, blocking_session, event, wait_time, seconds_in_wait from gv$session where blocking_session is not null

Once the data is returned, you can use the sid column entry to find the problem processes. Type the following query:

select * from gv$session where sid = <sid number>

From the data returned, you can use the next command to kill the offending session. You need the ALTER SYSTEM privilege for Oracle to perform this function as well as the sid, serial # and inst_id values from the query. Execute the following:

alter system kill session ‘<sid>, <serial#,@<inst_id>’ IMMEDIATE

After you run the command, the session will either be completely removed or will be marked as SNIPED until the Millennium executable or server tries to use the session again. A SNIPED status will not inhibit Millennium process, so either result puts you in much better shape.

Prognosis: Using Oracle’s v$session table can allow you to quickly find problem processes that are blocking other processes and keeping them from completing their work. Unfinished processes mean users are not able to place orders or see results.

A Time to Use Messages.Suppress

By David Hallberg

If you are a regular reader, you know that over the last couple of years I have strongly advocated minimizing the use of Cerner Millennium’s messages.suppress file. Functionality that was intended to prevent extraneous events from cluttering up your message logs has been horribly misused by Millennium system administrators and the application support and application development staff. As a result, vital events are not being sent to the message logs, events that need appropriate troubleshooting and longer term problem resolution. (See “When No News Is Bad News” and “File Can Hide Serious Issues.”)

In recent months, however, I have seen some improvements in Millennium’s base tuning and configuration when a particular event is suppressed, so today I have an unexpected recommendation.

The improvement involves an event called SEC_seclibAudit, which is an Audit-level event telling you that debug is not enabled for the security components. A couple of years ago, only CPM Code Cache Manager seemed to log this event. With the 2010.x code, however, SEC_seclibAudit appears to be written by all of the Millennium servers. It generates two rows in the Data column of the message log each time it occurs. The first row says:

[SecDebugLog::Instance] Creating debug logging singleton.

The second row says:

[SecDebugLog::CheckConfiguration] Debug logging not enabled for all security components, componentLogicalName (SecDebugLogsecrtl)

This is a good setting, but you don’t need to be told about it with every executable. It would be better if you could assume the setting is off unless you took action to turn it on. (To enable debug logging, set the LogLevel of a server to 4 or set the environment variable or logical of CPM _GLOBAL_LOGGING or any one of a number of other logging increases that are available to individual Millennium servers.) This message has become so prevalent, logging on all servers, it has become noise. One client, who cycles 19,000 servers each week, gets 38,000 messages every week for this one event.

To stop the noise, I have to make a recommendation I rarely make: Put this event in the $cer_mgr directory’s messages.suppress file. Then cycle all of your multi-instance servers to make the change take effect.

Prognosis: Using the messages.suppress file appropriately stops extraneous logging and keeps your message logs clearer for real problems.

Make Sure Servers Write Events to Message Logs

By David Hallberg

A client wanting to determine who had cycled a server in Millennium’s Server Control Panel (SCP) at a certain time went to the SCP’s message log cmb_0033 for the answer. When they opened the file, what did they find? Absolutely nothing. No entries or rows had been written to the message log since their last code upgrade. All of the stability issues and change control events that should have been logged were lost. The problem was much bigger than not knowing who had cycled a server. The organization had no way of knowing what else was wrong or how to resolve those issues.

We checked out three parts of the system to track down the problem.

1. Permissions settings in the $cer_log directory. The owners of the files in the $cer_log directory had been set to root for the User and root for the Group. That setting works great much of the time, but not for the following message logs: cmb_0000.mlg, cmb_0033.mlg, cmb_0057.mlg, cmb_0066.mlg and cmb_0078.mlg. For these, the User setting should be root and the Group setting should be your domain user name (for instance, d_prod or d_pxxx, with the xxx being your Millennium client number if you are remote hosted). Then the permissions on the file can be -rw-rw----, which allows the user of root as well as anyone in the group of d_prod to read and write to these log files. If you set these permissions properly, you will have lots of additional information about performance and stability issues in your Millennium domain.
2. SCP settings. SCP servers should have a User Name set to root and a Group Name set to your domain user (for example, d_prod). You also need to set the SCP Property of LogLevel to 2 on the following servers: 0 (Service Manager), 33 (Server Control Panel), 66 (CPM HNAM Agent) and 78 (CPM Script Install). The LogLevel for SCP Entry ID 57 (CPM Script Batch Priv) should be set to 1, unless you are actively troubleshooting an issue with this server.
3. The messages.suppress file in the $cer_mgr directory. If you cat messages.suppress, you will see all of the messages that are not being written to the message log. You are missing significant information if any rows in this file start with the following characters: CMB_, HNAM_, SCP_, InstallOCD, OCDPURGE and Script.

Prognosis: If you check and correct your message log settings before and after an upgrade, you will have much more insight into your Millennium domain and will be better equipped to troubleshoot stability problems and monitor change control.

Easy Tuning of Oracle Sequences Improves Throughput

By David Hallberg

In more than 13 years of helping clients tune their Cerner Millennium^® systems, I have only recently turned my analysis to Oracle sequences. I have the IT team for a PowerChart client to thank for my newfound understanding of how sequence numbers can increase the variability in system response times…and drive clinicians crazy.

After engaging me to diagnose their PowerChart performance problems, the team corrected several MQ queuing issues and other straightforward problems, but clinicians were still experiencing slowdowns during their peak usage times. The culprit turned out to be the Oracle sequence numbers.

Oracle sequences give an identifying number to certain elements in the database. Millennium has about 964 separate sequences, Oracle’s System and Sys have about 20 and 66 sequences, respectively, DBSNMP has two, and CMS_DBA and Panther_Read have one each. Oracle generates these sequences when new data or rows are added to the database. Sequences become a performance problem when you have a lot of new data that needs sequencing but the tables aren’t configured to adequately handle the load. Making the change is surprisingly simple.

Here’s what I discovered for my client. I took a 1-minute and a 30-minute sampling of the Oracle sequence numbers being generated by PowerChart. The following chart shows the results of the 1-minute interval: 

SEQUENCE_OWNER	SEQUENCE_NAME	CACHE_SIZE	Sequences Generated
V500	CHARGE_EVENT_SEQ	2,000	8,000
V500	OCF_SEQ	2,000	6,000
V500	CLINICAL_EVENT_SEQ	2,000	4,000
SYS	IDGEN1$	20	2,000
V500	PATHNET_RECOVERY_SEQ	2,000	2,000
V500	PERSON_SEQ	2,000	2,000
V500	LONG_DATA_SEQ	2,000	2,000
V500	CPMAPP_APPLCTX	20	1,320
V500	CPMAPP_APP_CTX_ID	20	960
V500	CHART_DIST_LOG_SEQ	20	40
V500	ENTITY_LOCK_SEQ	20	20
PANTHER_READ	AUD_ORA_ERROR_SEQ	0	7
SYS	ORA_TQ_BASE$	0	2

The key columns are Cache_Size compared to Sequences Generated. The Cache_Size shows how many sequence numbers can be generated before updating the high water number, which is the highest sequence number ever created for that sequence name. It takes time for Oracle to update this number, so you don’t want the system to have to do it too often. Increasing the Cache_Size decreases the number of times the high water value needs to be updated and, therefore, improves the throughput for the tables that use that sequence number.

Let’s look at an example in the chart. The Sequence_Name of Charge_Event_Seq generated 8,000 sequences in 1 minute. Because its Cache_Size was only 2,000, Oracle had to update the high water value four times during that minute. The Sequence_Name of Idgen1$ generated far fewer sequences, only 2,000, but its Cache_Size of 20 meant that Oracle had to update its high water value 100 times in this same minute. You can reduce this overhead associated with sequences — and improve the throughput for their corresponding tables — by increasing the Cache_Size to match or, preferably, exceed the number of sequences generated during your time frame. I recommended my client set Charge_Event_Seq to a Cache_Size of 9,000 and Idgen1$ to 2,000.

The change can be done in uptime without taking any action on the Millennium application nodes and without doing anything else on the Oracle nodes. The change is accomplished via SQL*Plus with this command:

       Alter sequence charge_event_seq cache 8000;

In CCL, use the following command:

       rdb alter sequence charge_event_seq cache 8000 go

The following chart shows the results from my 30-minute interval, which are a little more staggering in terms of the impact that high-water-value generation has on overall throughput. It also shows that only 35 sequences — about 3 percent of the more than 1,050 — had increases in their sequence numbers during this time (32 of 964 for V500 or Millennium and 2 of 66 for Sys).

SEQUENCE_OWNER	SEQUENCE_NAME	CACHE_SIZE	Sequences Generated
V500	CHARGE_EVENT_SEQ	2,000	106,000
V500	OCF_SEQ	2,000	76,000
V500	CLINICAL_EVENT_SEQ	2,000	48,000
SYS	IDGEN1$	20	40,000
V500	CHART_VIEW_SEQ	20	24,260
V500	CPMAPP_APPLCTX	20	22,660
V500	CQM_TRIGGER_ID_SEQ	2,000	18,000
V500	CPMAPP_APP_CTX_ID	20	16,160
V500	CQM_QUEUE_ID_SEQ	2,000	16,000
V500	ORDER_SEQ	2,000	14,000
V500	PATHNET_RECOVERY_SEQ	2,000	14,000
V500	CARENET_SEQ	2,000	8,000
V500	PATHNET_SEQ	2,000	6,000
V500	PERSON_SEQ	2,000	6,000
V500	DM_CLINICAL_SEQ	2,000	4,000
V500	EKS_DLG_EVENT_SEQ	2,000	4,000
V500	ENCOUNTER_SEQ	2,000	4,000
V500	SCH_CONVERSATION_SEQ	2,000	4,000
V500	AAR_REPORT_SEQ	2,000	2,000
V500	CHART_REQUEST_EVENT_SEQ	2,000	2,000
V500	CHART_SEQ	2,000	2,000
V500	ENCOUNTER_ONLY_SEQ	2,000	2,000
V500	ENTITY_LOCKKEYID	2,000	2,000
V500	ENTITY_RELTN_SEQ	2,000	2,000
V500	ESI_LOG_SEQ	2,000	2,000
V500	MEDADMIN_SEQ	2,000	2,000
V500	NETTING_SEQ	2,000	2,000
V500	OMF_SEQ	2,000	2,000
V500	PHARM_FILL_HX_SEQ	2,000	2,000
V500	REFERENCE_SEQ	2,000	2,000
V500	RESULT_SET_SEQ	2,000	2,000
V500	ENTITY_LOCK_SEQ	20	400
V500	CHART_DIST_LOG_SEQ	20	300
PANTHER_READ	AUD_ORA_ERROR_SEQ	0	125
SYS	ORA_TQ_BASE$	0	4

So I recommend only changing sequence numbers that are being used a lot. In this case, I recommended:

• Charge_Event_Seq and OCF_Seq have Cache_Size set to 10,000
• All V500 sequences with more than 2,000 sequences generated have Cache_Size set to 8,000
• All V500 sequences that match their Cache_Size be left at 2,000, at least initially
• All V500 sequences with a Cache_Size less than 2,000 be increased to 2,000
• Sys Idgen1$ have Cache_Size set to 10,000
• Sys Ora_Tq_Base$ have Cache_Size set to 20
• Panther_Read have Cache_Size set to 2,000

It is possible to make the Cache_Size too large, so my recommendations are fairly conservative. Still, the client will have much less variability in Millennium response times after implementing these straightforward fixes.

Prognosis: Keeping your Oracle sequence Cache_Size properly set is a simple, uptime tuning procedure that decreases performance-robbing events and variability in your Millennium environment.

Messages.suppress File Can Hide Serious Issues

By David Hallberg

Historically, when I was brought into situations where a problem within the EMR system was difficult to track down, I used the message logs quite extensively. They helped me understand what was happening and where the troubleshooting should begin, and they explained any application-specific problems. This method worked very effectively until Millennium^® 2007.18 TP4, when a new file in the $cer_mgr directory — the messages.suppress file — started shipping with some troubling exclusions, namely ScriptError and  ScriptErrMsg events.

The messages.suppress file stops Millennium from writing events to the message logs. This file was designed to stop creating message log events when an engineer accidentally sent an event to clients with an incorrectly set LogLevel. It’s a legitimate place for extraneous events that would otherwise clutter your log files and potentially overwrite messages that need attention. For example, the event CRM_ClientTimeZone writes the current time zone to the message log. Since domains rarely change time zones, the event is seldom of benefit when troubleshooting an issue. Yet it can cause a lot of application node writes, since every transaction going through the application server with this set has to write a message log row into the system message log as well as the specific server message log. To reduce the application node’s disk I/O and performance variability, it’s good to have CRM_ClientTimeZone in the messages.suppress file and exclude these events from the message logs.

If you hide the wrong events, however, you miss out on important information. A problem, failure or issue that should be written to the message log goes undocumented, and your clinical or technical support team has no record to assist them in identifying and resolving issues.

So it makes no sense to me that with the initial TP4, ScriptError and ScriptErrMsg events were included in the messages.suppress exclusions and that the file has since expanded to include ScriptErrors-Fatal and ScriptErrors-NonFatal events. You need visibility into these events. They indicate that an application or Oracle has a problem (often a code or build issue) and needs help to fix it.

What might the problem be? ScriptError events include blood bank transfusion script errors, MDI download problems, scripts failing to run (such as those getting PharmNet frequencies) and Oracle deadlock issues. These events all have a negative outcome to the clinicians using Millennium and the business office needing to bill accurately for services delivered.

ScriptErrors-Fatal events include memory problems that prevent clinical information from being updated and from being viewed by the clinicians who need this information.

ScriptErrors-NonFatal events show a wide range of issues. Expert Knowledge System (EKS) compile errors, Visual Explorer errors, SurgiNet documentation errors, getting orders asynchronously for PowerChart, Lights On Network data-gathering script errors and sequence number problems are all events that can degrade Millennium performance or even cause a failure. These errors must be logged so that problems can be seen and corrected.

Prognosis: Keeping your messages.suppress file properly set will ensure that the Millennium message logs contain the information that your support team needs to identify and resolve issues in a timely manner.

Setting Proper LogLevels Is as Easy as 0-1-2-3-4

By David Hallberg

I am often asked about proper log level settings (LogLevel) for Millennium^® servers. The setting determines what types of messages are written into a server’s message log files, which provide critical information to support teams for correcting application problems and some technical issues. If you set the LogLevel too low, you might miss important information. If you set it too high, you might be flooded with extraneous messages (and overlook the ones you need to address).

LogLevel can be set to one of the following values:

• 0 = Error (ERROR on Java servers)
• 1 = Warning (WARN on Java)
• 2 = Audit (AUDIT on Java)
• 3 = Information (INFORM on Java)
• 4 = Debug (DEBUG on Java)

The settings are cumulative, meaning that servers will write messages from that level as well as all lower levels. For example, a 2 setting would include all Audit (2), Warning (1) and Error (0) messages. You can view LogLevels and change their settings in OnTrack Panther™, SCPView or Olympus. Any time you change a LogLevel you need to cycle the server in order for the change to take effect.

In general, the lower the number, the greater the severity of the messages sent to the log files. Unfortunately, many Audit events are actually failed messages or transactions, which is one reason why a too-low LogLevel might result in lost information…and why many people have questions about what the settings should be. That wasn’t always the case.

In 1997, when most Millennium clients were running on VMS, disk saturation concerns meant that LogLevel was always set to 0/ERROR. When you had two or more application nodes with hundreds of servers pumping log information on the same disk drive, the drive — and everything else — would slow down. AIX, with its SSA drives, had problems as well. The drives shared the same connections as the database. If the executables were using up the connection from the CPU to the disk, then the Oracle database could not get the disk time it needed to operate as well as it could.

Disk saturation is not as much of an issue now that most clients are on 2007.02 or greater, many VMS clients have migrated to some flavor of Unix and the AIX clients are off of SSA drives. Yet logging can still be a problem, even when you are running on a well-tuned storage area network. Today, logging usually manifests itself as a specific queuing problem on a specific executable. If you set servers to LogLevel 0/ERROR, you might miss information that is important to use in finding and correcting problems in your environments. I recommend you no longer use this setting.

I also recommend that no server be set to LogLevel 3/INFORM or 4/DEBUG unless you have an active point asking for debug information. Short of that, these settings cause too many disk writes. I was working with a client recently who had the EKS servers in debug, and the message logs were wrapping in less than 15 minutes. This much logging slowed down all of the EKS rules processing but provided no benefit to the client or its Millennium support team.

So what should you set your LogLevels to?

Monitoring servers — such as 27, 29, 33, 66, 76, 97 and 98 — should be at least a LogLevel 2/AUDIT. The Audit level provides a great deal of useful information when you are working through issues or looking for problems before they actually become outages.

I recommend a LogLevel of 1/WARN for the servers listed below. As a general rule, if your executable is not listed, I recommend you set the LogLevel to 2/AUDIT. You might find you need to make some specific tuning or tweaking to these settings, but they provide a great starting point.

SCP Entry ID		SCP Description
72		CPM Notify Server
73		Time Repository Server
75		User Services
80		CPM Script Cache
86		CPM Expedite
92		OMF PowerVision Document Server
127		System Print Server (Node Specific)
130		PathNet Resource Routing Server
131		SCS Netting Server
132		SCS Collections Server
133		PathNet Result Order Posting Server
134		PathNet Sync Result Order Posting Server
134		PathNet Synchronous Result Order Posting
140		Pathnet - AP Case Retrieval Server
141		Pathnet - AP Automatic Diagnostic Coder
144		Pathnet - AP Coding Cache Manager
205		Clinical Event Server For ESI
209		Clinical Event Server - Batch
221		Search Server
231		MDI Result Server
235		MDI Order Server
237		BMDI Result Server
239		DI Inbound Client
252		FSI ESO-CQM Server
275		FSI ESO Notify Server
280		FSI ESO Server Load #1
281		FSI ESO Server Load #2
282		FSI ESO Server Load #3
283		FSI ESO Server Load #4
284		FSI ESO Server Load #5
285		FSI ESO Server Batch #1
286		FSI ESO Server Batch #2
290		FSI ESO Query Extract Server
291		FSI Batch CQM Server
335		PM Trans Async
340		PM Query
369		Chart Coding Services
375		KIA Event Server
425		FRN Tracking View Server
455		Health Expectation Server

Prognosis: Keeping your LogLevels properly set will ensure that your IT team is notified of the issues it needs to address without being overwhelmed with meaningless messages.

New Code and Lights On Network Compliance Tabs

By David Hallberg

Several Millennium^® users I’ve talked with since the first of the year have had problems in production after taking new service packages. All had the same question for me: “How could we have anticipated the problems we now have in production?” Because some of the problems involved a new solution being brought up, I had a question for them: “What is your compliance percentage on the System Compliance and Solution Compliance tabs of the Lights On Network?” Either they didn’t know what I was talking about or they only had the tabs enabled for production. In their ignorance, they were missing out on key data from Cerner and its clients that could have helped them prevent the issues they were experiencing.

New code undergoes several quality checks before Cerner releases it to its client base. However, internal testing can never be rigorous enough to anticipate the varied and complicated production environment. So several years ago the Lights On Network was set up to allow clients to see several aspects of their configuration. The two most useful, from my perspective, are the System Compliance and Solution Compliance tabs.

The System Compliance has four dials: Known Errors, Oracle Configuration, OS Configuration and HNAM Configuration.

• Known Errors shows quality issues, sometimes called CIRB, as well as CernerWorks Process Improvement (CWXPI) corrections. Essentially, when one or more clients have a problem in a production environment or a potential for a problem in production, notifications are sent out to alert all clients of potential issues (CIRB). The CWXPI items are problems found at sites that Cerner manages or hosts, and to improve all Millennium environments, Cerner publishes how its teams corrected these issues.
• Oracle Configuration shows clients what several basic Oracle configurations should be when they first go live. There may be reasons your configuration differs from the recommendations. These reasons or differences can be documented in the Inactivate selection from this screen, an option available on all four of these dials.
• OS Configuration shows what basic Operating System configurations should be. It’s a very good baseline. If you deviate from it, you need to be certain why.
• HNAM Configuration shows more CWXPI items, basic logging settings and some solution/application configurations. Generally, this tab focuses on technical aspects of the Millennium environments.

The application side of the fence is found in the Solution Compliance tab. This tab shows topics such as what Ops Jobs (Millennium batch scheduling) need to be run and recommendations on how to configure Ops Jobs if they are not being run. The tab provides feedback on a subset of the Millennium purges. These purges delete temporary data that is not considered part of the permanent patient record. It helps control the size of the Oracle database and can improve or stabilize performance.

Prognosis: Once you and your organization are aware of the quality checks available on the System Compliance and Solution Compliance tabs of Cerner’s Lights On Network, you can put processes in place to leverage this service so that you reduce problems in your Millennium domains and improve the effectiveness of your infrastructure.

A Green Light for Your Application’s Journey

By Chris Lanaman

St. Patrick’s Day parades, the NCAA tournament. It looks like there will be lots of mad marching today. Whether you’re wearing the green of Ireland or the red/blue/green/gold/purple of your favorite college basketball team, I want to focus today on a journey that, although complex, should be much more predictable: the route that data needs to travel through Millennium^® from the moment of entry to execution.

Every application that makes up the Cerner system — FirstNet^®, PowerChart^®, etc. — has its own pathway to getting information and placing it in your database. Each application is comprised of many different functions, which cascade to complete each task. For example, when placing an order for an aspirin, the clinician’s final step of clicking the “sign” button starts several functions within Millennium. The system validates the patient, finds the code for aspirin, executes checks to see if the patient is allergic to the aspirin, validates that the person ordering the aspirin is authorized to order it, moves the tasks associated with administrating the aspirin, adds the order to the pick list for pharmacy or the Pyxis^® system, drops a charge and, finally, updates the patient record in the database.

This process is repeated over and over for all of the different workflow processes that clinicians carry out. If your system is finely tuned, these steps happen in fractions of seconds. Unfortunately, there are many places where even the simplest action like ordering an aspirin can go wrong. The good news is that there are ways of checking those pathways to see where the bottlenecks are and then setting controls to prevent them from recurring.

One of the best ways to understand the bottlenecks in Millennium is to imagine the checkout line at the grocery store. If only one lane is open, shoppers might have to wait a long time to finish their purchase. However, if managers open another lane, they can process shoppers twice as fast. The same concept applies to Millennium. If the lines get long (queues fill up) where data is processing, you can increase the number of lanes (instances) that are processing your information. The result will be less waiting for clinicians.

With such a myriad of functions to complete in order to execute even the simplest of orders, it’s not surprising that the system slows down at times. What’s surprising is that some organizations equate slow with normal and do nothing to unclog the bottlenecks. It doesn’t have to be that way. Keeping your system tuned will help it deliver the performance clinicians expect.

Prognosis: You can learn how to identify which services are backing up and how to start more instances on the problem servers by reading our previous postings: “Triage 1: Diagnose System Slowdowns” and “From Triage to Treatment: Configure a Dedicated SSP.”

Why Auditing Your EMR Is Essential

By Chris Lanaman

I want to expand on the comments I made in my last blog on the need for third-party auditing of your EMR system. I’m including an excerpt from the white paper “Why Auditing Your EMR Is Essential” that our Softek team wrote and distributed last week at HIMSS. The message struck a chord with the people we talked with — both self- and remote-hosted organizations— so I’m confident it will resound with you as well.  

When patients are admitted to your hospital, they count on reliable, competent care:  caregivers who are board certified, care delivery that has passed the standard of the Joint Commission, lab procedures certified for quality by the College of American Pathologists, certified dieticians who oversee patients’ dietary needs and ethical business and financial practices. In each of these areas, facilities and individuals submit to regular, independent evaluation to make sure their practices follow generally accepted industry standards and, when necessary, implement changes to bring them back into compliance.

One area overlooked in these evaluations is the operation of the EMR system itself. With more and more functionality demanded from these systems, they continue to grow in complexity. Yet no independent evaluation program is required to ensure that the EMR is performing as expected. And if it’s not, the consequences could be devastating for patients, caregivers and the facility itself.

Think of the changes you continually introduce to your configuration. Besides the customizing you do for your own unique uses, vendors are supplying a steady stream of system updates and upgrades. A technology upgrade can create myriad downstream impacts on how the EMR will behave. Every time you add a new layer, a new process, a new handoff point, you increase the likelihood of delivery failure. Each failure poses the risk of losing patient information or financial data or extending wait times for clinicians trying to input or access that information.

If your EMR system is hosted outside your own organization, you might think independent auditing doesn’t apply. After all, one of the primary reasons you sought out a knowledgeable vendor was to oversee system performance. You wanted the assurance that comes from having competent partners hosting your system. Yet how do you know that the hosting partner is catching all the performance problems that arise? Because so many performance and stability issues arise from errors with application settings rather than from technical infrastructure, who is really responsible?

Your hospital IT team knows when performance problems develop — you deal with the complaints. With the visibility that comes through a system evaluation, you can more confidently partner with your service provider to not only address system issues but also set higher standards for performance and quality of patient care.

As healthcare focuses more and more on evidence-based medicine, the IT delivery vehicle for that medicine should be evaluated in the same way: using real data from your own system, gathered by an organization independent from the vendor or your own organization, and clearly presented so that you can understand where you are and what adjustments need to be done to accomplish your goal of optimal healthcare.

Prognosis: You can learn even more about how to get an evaluation of your EMR by reading our white paper, “Why Auditing Your EMR Is Essential.” Click here to receive it.

Learn about EMR Auditing at HIMSS

By Chris Lanaman

I’m heading off to the HIMSS11 conference in a couple days to mingle with some of the nearly 30,000 healthcare IT professionals descending on the Orlando Convention Center. While some might get sidetracked by Mickey and his friends, you’ll find me every day of the show with my Softek colleagues in booth 3221. If you’re going to be there, I’d love for you to stop by and meet with us.

I think our message at this year’s show is worth hearing: Every hospital would profit from an independent audit of their EMR system. Just as their care delivery must meet certain objective standards set by the Joint Commission and their financial operations undergo regular auditing, hospitals can use EMR auditing to see if their system is efficiently handling transactions and properly interfacing with its myriad subsystems. Where EMR performance is lagging, adjustments can be made — and patient care and safety improved.

I found an educational session that will make the same case. Dr. Dean Sittig, a professor at the University of Texas Health Science Center is speaking Tuesday morning on “Inspecting EHRs to Ensure Safe Use” in which he will “describe how to implement an annual EHR inspection program … to ensure that their clinical systems are functioning as designed and are being used safely and effectively.” 

Since joining Softek last fall as Director of Millennium Consulting Services, I’ve been involved with a number of such evaluations. Many clients have been surprised at our findings. We commonly find lost charges, queuing backlogs, bottlenecks throughout the system and even clinical information that fails to make it to the patient record. Following our report and recommendations, one client reduced wait times across the organization by hundreds of hours a month. That makes for a lot of happy clinicians and support staff.

Prognosis: A peak-performing EMR system is vital for reaching your goal of optimal healthcare delivery.

MQ Maintenance for Millennium

By David Hallberg

I am frequently asked about maintenance of the Cerner Millennium® system’s MQ queues, specifically what to do with the Admin.Reply, Audit.Exception, Exception and Z_Exception queues. It’s an excellent question, and the answer is not as clear-cut as you might believe. Each of these queues has different reasons for filling up or keeping transactions and different impacts if they do fill up.

Let’s look at the purpose of each of these queues.

The Admin.Reply queue is used to ensure that appropriate communication lines exist for a server starting a transaction. Millennium executables check with MQ to see if the needed queues exist on the node. So, for example, if an SSREQ/SSREP or RDM queue is needed, the Admin.Reply queue either finds the needed queues or creates them. At that point, the transaction is supposed to be deleted from the queue. The problem with this queue is that various Millennium versions and executables do not always delete the transaction correctly and the queue fills up. Once the queue is full, no new Millennium servers can be started from the Server Control Panel (SCP), command line, Panther or Olympus. So any automated cycle-server jobs as well as any interactive server cycling will fail.

This problem can very, very quickly cause significant performance and stability issues in your domain. I helped one client whose Admin.Reply queue, which had a Max_Depth of 15,000 rows, filled up in less than 10 minutes. The client cycled servers between midnight and 7 a.m., but because this queue was full, all of the cycle jobs failed and caused performance issues.

So how do you prevent a similar scenario? By the time you see a transaction in this queue, the work the transaction was doing is complete. These transactions do not contain any clinical or financial information, so it’s safe to delete them. I recommend that you check this queue on each application node at least once a week and delete any transactions you find.

The Audit.Exception queue is connected with the queues CPMSRVAUDIT, CPMSRVAUDITBATCH, CPMSRVAUDITOUT, SSREP.SRVAUDITMGR and SSREQ.SRVAUDITMGR. It is designed to capture transactions that could not be sent to P2Sentinel, which can happen for any variety of reasons. (Be aware that P2Sentinel is the third-party product used to capture information and could cause HIPAA violations or show inappropriate use of Millennium by your organization’s staff.)

Begin your troubleshooting of this queue by verifying that P2Sentinel was up fully and able to take transactions. If P2Sentinel can take transactions, then you should be able to replay these transactions. If you do not replay or recover these transactions, there will be gaps in the auditing information that you need to be aware of. It is possible to have incorrectly formed transactions, which you might have to delete. Before you do, I suggest that you engage your IT management, Security and whoever is tasked with overseeing HIPAA compliance. Your organization might need to complete additional documentation prior to any deletions.

The Exception queue is designed to hold transactions that could not be put on an MQ queue. How does that make any sense? Here’s the basis of MQ: A process or executable has a transaction to hand off to another process or executable, and application programmers want to make certain these transactions make it from one process to another. So the executable that wants to hand off a transaction puts that transaction into a queue. From there, the receiving executable retrieves the transaction and processes it. Still confused? Suppose the transaction is my electric bill. The electric company wants me to get the bill (transaction), which it delivers to my mailbox (MQ queue), where I go and get the bill (transaction). If for some reason the postal carrier cannot put my bill into my mailbox, the bill will go to the infamous dead-letter office (Exception queue). Someone at the central post office will then do additional processing to send me my bill.

In the case of Millennium, the bill is actually a clinical transaction, so you can’t ignore it. Any transactions sent to the Exception queue have clinical or financial data that was supposed to be processed and acted upon but never made it to queue it was originally sent to. It’s important to recover this information, which you generally can do simply by replaying the transaction to the appropriate queue. This is a little like UPS leaving a message on your door stating they need a signature before leaving a package, and since you were not home, they will try again tomorrow. The next day you stay home and get the package.

A replay generally works with Exception queue transactions because the failure reasons are often transient. If a replay does not work, you will want to log an SR to get additional assistance in either processing or deleting the transactions. As with the Audit.Exception queue, Exception queue transactions generally include clinical or financial data, so deleting them could create gaps in patient records. I would suggest that you engage your IT management, Security and whoever is tasked with overseeing HIPAA compliance before deleting these transactions. You might need to complete additional documentation prior to any deletions.

The Z_Exception queue is my favorite. It is used by the Queue Control Panel (QCP) when the front-end QCPView.exe or Quemon.exe or any other Millennium executable cannot communicate with the QCP. Since the QCP does not handle any clinical or financial data, I would suggest deleting the transactions in this queue. If the Z_Exception queue fills up, the QCP will not work normally and you will not be able to use QCPView.exe or Quemon.exe.

Prognosis: A little housekeeping with the Millennium Exception queues will help make your EMR environment more stable.

Betrayed by an Old Friend

By David Hallberg

I had a client complain to me recently that I was reporting a false positive, an alert that was triggered even though there was, in fact, no problem or issue with his Millennium^® EMR system. I decided to figure out why there was a discrepancy between my measurements and the client’s quality-assurance tool, which happened to be sysmon.

Let me begin by saying that I love sysmon. Before sysmon was developed in the late 1990s, Millennium had so many tools or utilities that provided a single view of a piece of information that it was difficult to keep track of all of them. At one point there were more than 200 build or monitoring tools even though Millennium only had about 40 actual application executables that were used in various configurations for its 60 plus solutions. Multi-node clients kept asking their Millennium middleware team to figure out where performance bottlenecks were occurring. Believe me, this sleuthing was extremely labor-intensive. Users had to have mon_ss (a Millennium monitoring tool used to watch for proprietary middleware queuing) and joumon (a Millennium monitoring tool used to watch for persistent queuing in the proprietary middleware) up in a different window for each application node. Then they would have to try to show someone else when one or more of these screens had a “blip” or negative performance indicator. It was painful on the good days.

So those managing Millennium were understandably ecstatic when Cerner developed a single utility that told what was happening across Millennium’s persistent and non-persistent queuing mechanisms across all application nodes, in a single screen. Sysmon also provided the date and time when something bad happened.

Sysmon proved to be reliable, and I became a proselytizer for this great monitoring tool. While I was at Cerner, I taught many clients how to use sysmon and its various switches. I also used it to tune a four-node client with ConnectionBias (a load-balancing mechanism for Millennium), moving the transaction workload from one application node to another. I was able to show enough value with this one tool that some clients actually chose not to purchase BMC Patrol (later called BMC Performance Manager and now called BMC ProactiveNet Performance Management). Sysmon had some limitations — it was really difficult and unreliable to pipe or move the output to a file for additional processing, and it had no mechanism to redirect its output to a common data format like CSV (comma-separated values), tab-delimited or regular flat files — yet it was the most accurate view of a multi-node application configuration.

So why was my data conflicting with sysmon? It appears that with 2007.02 some of the newer Millennium services do not register with the old middleware in a way that allows sysmon to see the queuing. With the client I referred to earlier, I found that the SenSage add-on called P2Sentinel did not publish its queuing in a way that sysmon could see it. The toolset I used, however, did see the problem.

That’s not the only problem. Another client reported that one of the Java services introduced with 2007.02 was crashing several hundred times a day. I ran sysmon to see if it would show these crashes, and it did not. I am not exaggerating to say that I was crushed. Sysmon was the one tool I absolutely trusted to let me know in a timely manner when Millennium was failing, typically 15-45 minutes before BMC. Now I find it is not reliable!

If, as it appears, the newer programs will not share or publish their information like the old programs, Millennium clients now must find other tools. None of the options, however, shows them this information in a single view. Sadly, we all have to go back to running multiple windows for each application node when we are using command-line tools.

A source of truth that you can trust and believe in is difficult to find. It is even more difficult to see it no longer be a source of truth.

Partners for Success in 2011

By Chris Lanaman

With the start of a new year, I’m wondering if you’re feeling a bit like Bill Murray’s weatherman character in the movie Groundhog Day, in which he eerily wakes day after day, week after week to the sound of Sonny and Cher and the realization that he is about to relive the same day over yet one more time. Rather than dreading the challenges and aggravations that you suspect are coming when you arrive at work today, we’re hoping instead that you might see today as a chance to erase the mistakes of yesterday and make a fresh start. To that end, our team of bloggers is invigorated to introduce you to some new and meaningful tips to improve your Millennium® system’s performance, along with information on where healthcare IT is going in 2011. Dave Hallberg and I have been busy already this year with several client visits. When our travels slow down, we’ll be writing about some of the issues that have turned up at these sites. With each blog we hope to peak your interest and spark exceptional conversations that will lead your organization to greater levels of success. As always, if there are items you would like to see discussed, let us know what they are. We are always willing to address your most pressing issues. As the Director of Millennium Consulting Services for Softek, I look forward to sharing knowledge with you and your organizations.

Determine if Millennium’s Size Is Just Right

By Chris Lanaman

Like Goldilocks trying out chairs at the cottage of the Three Bears, every healthcare organization is looking to land on a system that is sized “just right.” When you first had Cerner Millennium^® installed, smart people thought through what kind of hardware and how much of it would need to be deployed in order to meet your organization’s needs. Most likely they also considered the projected growth of the organization over the next five to 10 years. Buying too much would add complexity for your users, loss of control in managing the system and, of course, wasted cost and time. No one wanted Millennium to be like Papa Bear’s too-big chair.

I would suggest that a bigger problem exists when the Millennium system is not sized big enough for the demands of your organization. Any one of the following symptoms might indicate that your workload has outgrown your system: lower than expected productivity of outcomes, the loss of business revenue because of turning away patients or poor response time for end-users.

I recommend starting with the following five steps if you are concerned that your organization has outgrown its current Millennium size.

1. Data, data and more data. You must have data in order to see if your organization is suffering from a sizing issue. This data is gathered by your team of data-miners (IT analysts) or through an application that is generating reports for your system, such as Cerner’s PowerInsight or PowerVision. Use this information to see if there has been a drop in activity, a workflow slowdown or loss of productivity.
2. Patient management cycle. Once you have obtained measurements for your organization’s workflow, you will need to evaluate where you are in your patient cycle — is this the time of year that you have increased patient traffic or are these your slower months? Your metrics should also include a staffing model to meet patient demand. Understanding your cycle will keep you from over-sizing your system if you evaluate during the high point of the year or under-sizing if you are in a valley part of your cycle. These measurements will identify the revenue cycle expectations for your organization.
3. Mergers and acquisitions. When your organization merges with another healthcare organization, great pains are taken to try and unify the new organization under a common vision. The impact of the merger on the EMR system’s workload, however, might be overlooked. The IT analyst needs visibility into the new organization’s past data to understand how workloads might increase.
4. Market share. Does your organization understand the footprint it has in your market? When trying to understand growth potential, you need an understanding of your opportunities and who the suppliers in the market are. Theoretically, one can say a set percentage of business equals a set amount of hardware needed, but this is an educated guess at best and should never be the rule of thumb.
5. Is your Millennium system healthy? As I alluded to earlier, when your system is not sized correctly, you will see an increase in performance issues. If your end-users are telling you, “The system seems slower than it used to be” or “It seems to take longer for me to do my job in the system,” the first thing on your mind should be the health — i.e., the efficiency — of your system. Before joining the Softek team, I used Softek Panther for a number of years to uncover trouble spots. It let me know when Millennium stepped out of line. Additionally, Softek’s Queuing Performance Spreadsheet (QPS) allows you to see where within Millennium your user transactions are getting clogged, and the Messaging Performance Audit (MPA) gives your system an overall checkup. The audit lets you see if you have build issues in Millennium, if transactions are going where they’re supposed to go, if your servers are configured correctly and when your servers are being over-tasked. If you have other monitoring tools you recommend, please let me know. Whatever you use, it’s important to check your system thoroughly before buying more hardware. Hardware purchases should always be considered the last resort for resolving performance problems. 

In a blog, I only have space to give you the Reader’s Digest Condensed version for scoping your organization’s size and determining how that affects Millennium’s performance. But addressing these key points will help keep your organization healthy and your users happy.

Prognosis: Evaluating all of the different factors that make up your organization is the first step in understanding if your Millennium system is sized “just right.”

Putting the Profit in ProFit

By David Hallberg

I’ve written previously about lost charges and how to examine Millennium’s message logs to find and correct billing issues. Today I want to speak specifically to clients who use ProFit^® Patient Accounting, Millennium’s billing solution. Those wanting to maximize their revenue will find a wealth of information in the message logs on the ProFit servers.

One of ProFit’s components is a third-party subsystem called Sablotron, an XML toolkit that was used to make ProFit easier to support across operating systems (http://freshmeat.net/projects/sablotron). Because it is XML based, Sablotron is independent of the operating system and works the same on AIX, HP-UX, Linux and VMS. This is great from an engineering support perspective. But what does it mean to a healthcare organization?

ProFit has two primary servers: ProFit Transaction Server (SCP Entry ID 475) and ProFit Billing Server (SCP Entry ID 476). Generally, an organization can only control the memory size and loglevel of these asynchronous processing servers, and interactive users typically do not know when these servers (executables) have had any errors or warnings. Someone would have to go digging for these issues to become aware of them. I recommend you get out your shovels.

I recently ran across two events on the Billing Server of a ProFit client. To correct the first issue, we needed to log a service request to get help in correcting the build. For the second, we needed to log a service request to get new code.

The first event was the Sablotron Message, which indicated that the subsystem could not properly handle some data in a billing transaction. In a single four-hour time frame, my client had 4,002 of these events. The Data component of these records had a consistent beginning section that looked like this:

{{msgtype::warning}}{{code::93}}{{module::Sablotron}}

It was followed by one of three ending components.

1.  The most numerous Sablotron Message ending was:

{{URI::arg:/_parsed_}}{{line::544 }}{{node::element '<xsl:for-each>'}}{{msg::unsupported language 'en'}}{{Routine::sablotronErrorHandler::log}}

You’re probably thinking, “This looks like someone sat down at the keyboard after drinking a bit too much.” But the message actually illustrates how many XML files are laid out or formatted. XML is just another way to show data, and the message tells us that in line 544 of the server, there was a language code of “en” that did not have a translation (‘<xsl:for-each>’) and the server didn’t know what to do with it. So the server wrote the message to its message log.

2.  The second most numerous Sablotron Message ending was:

{{URI::arg:/_parsed_}}{{line:: 497}}{{node::element '<xsl:for-each>'}}{{msg::unsupported language 'en'}}{{Routine::sablotronErrorHandler::log}}

This looks like the same drinker was at the keyboard. The message is the same as the previous one, except that the unsupported language code of “en” was in line 497.

3.  The third Sablotron Message ending was:

{{module::Sablotron}}{{msg::unsupported language 'en'}}{{Routine::sablotronErrorHandler::log}}

This message appears to be a higher level failure, but the end result is the same. The server had a language code of “en” that it could not translate, so it wrote the message to its message log.

The second event my ProFit client experienced with the Billing Server was SRV_LookupFailure, which told us in the Data that the bill_item_id was missing from the transaction. This would generally not be a problem, except that in this case the user name was SYSTEM and, therefore, no interactive user was made aware of these failures. They were only in the message log.

Not all the people I work with think these messages are a big deal — they see errors in Millennium as normal. I disagree. The client I’ve referenced here had, on a single node, 3,038 of the line-544 Sablotron Message events and 904 of the line-497 Sablotron Message events in two hours as well as 60 higher level Sablotron Message events and 92 SRV_LookupFailure events in one hour. How could this be considered normal? If an executable is having a problem, you want to correct it, whether the issue is in the build, the code or the use of the code.

Prognosis: Little events can add up to big revenue losses. Looking for these events and resolving the code or workflow issues will maximize your revenue.

It’s Just a Fax, Right? Part 3

Troubleshooting your RRD application

By Chris Lanaman

For the past two blogs, we have looked at the important role faxes play in relaying critical information from the hospital to doctors’ offices and how a reliable fax service can improve relationships and business with these offices. Today we turn our attention to making sure Millennium’s Remote Report Distribution (RRD) servers, which send the faxes, are operating at peak performance.

It’s not always an easy task — which is why in my first blog on this subject I recommended that management dedicate at least one full-time analyst to support faxing. Gone are the days that you walk over to the fax machine, set your forms on the paper feeder, type in the phone number, hit a button and away it goes. Now many processes typically occur before a document reaches the RRD queue and is ready to be made into the fax and sent on to the primary care physician.

When training new analysts over the past few years, I often taught them the Cerner RRD application first, before graduating them to the Clinical Reporting applications. This method might seem backward, but I found it to be the most effective. With most things in the IT world, the best way to learn how to fix things is to start at the outcome and then work your way backward. If your end-users are saying they didn’t get the fax, you check the RRD server.

RRD technology has not changed much in recent years. Because of this, RRD servers can quickly become overwhelmed when a hefty number of faxes are pushed through them. If you’re experience faxing delays or failures, I recommend you check the following:

1. Operating system. Imagine sitting at your computer all day and faxing a Word document over and over, all day and night, through the computer’s modem. At some point, you would become exhausted and would need a break to regroup and refresh. Similarly, the RRD application needs to be refreshed. Small things like clearing the memory (cache) and straightening (defragging) the files are often forgotten but can help speed up the process.
2. Fax volume. It’s important to know the number of faxes you are sending each day. If you’re sending more than your RRD servers can deliver in a timely manner, you might need to beef up your RRD servers. Remember, as I stated in an earlier blog, delays in fax deliveries can be very damaging to your organization’s relationships with referring physicians.
3. RRD services. Make sure the RRD services are running on your RRD servers. Without these running, you simply have a desktop computer not doing anything.
4. Phone lines. Another area that often gets overlooked is whether the phone lines are still active. Troubleshooting might be as simple as making sure you have a dial tone on the lines going into your RRD servers.
5. Modems. Check to see if your modem is hung and not releasing your transmissions. If you use Softek Panther, make sure the RRD: Stalled sensor is set and then check for any Error messages. These messages might indicate a problem with your modem, not your RRD system.
6. Obsolescence. Sometimes customers have fax machines that are so old that the Smithsonian is looking to add them to its collection. Ask your customers for the make and model of their fax machine, and then check to see if your current RRD application level supports that model. Keep in mind that if you are not using internal modems on the RRD server, and instead have serial modems attached to the RRD server, those serial modems might be the issue as well. Sometimes a simple reboot fixes the problem.

As you can see, Cerner's RRD application around faxing can get hung up in a lot of areas. That’s one reason why tracking faxes for a healthcare organization is a full-time job. The next time someone tells you, “It’s just a fax,” remember you have your organization’s reputation – as well as a lot of money - riding on the fax getting where it needs to go.

Prognosis: Troubleshooting Millennium’s RRD application is much easier when a full-time member of the IT team can own the maintenance and support of this important faxing service.

 

It’s Just a Fax, Right? Part 2

Keeping relationships strong with PCPs

By Chris Lanaman

My last blog showed why doctors’ offices have a strong reliance on receiving test results and patient information through faxes — which are sent through Millennium’s Remote Report Distribution (RRD)  servers — and why it’s important for hospitals to make sure their faxes go out on schedule. Today we’ll look at another strategic importance of faxing, one that will help your healthcare organization distinguish itself from others.

In his novel 1984, George Orwell wrote the following ominous statement related to information control: “Who controls the past controls the future. Who controls the present controls the past.” I want to present a more practical paraphrase of the quote: “Whoever controls the old technology (like faxes) will be able to drive the introduction of the new technology (like interfaces).” What do I mean? The more relationships that your organization has established and is actively sending faxes to gives you the power to drive conversations about joint ventures in connecting electronic medical record (EMR) systems through interfaces.

Although many doctors’ offices do not have any form of EMR system, the number of “office size” EMRs has grown in the last five years and will continue to grow. As part of this conversion, I have found that larger doctors’ offices are happy to say good-bye to the piles of faxes that can often be difficult to manage. They still want notifications but are ready to move to a “paperless” environment.

When these offices start asking you how their number of faxes can be reduced, they are giving you the opportunity to strengthen your alliance with their office. If you can successfully interface your system with theirs, you stand a good chance of merging your organization with theirs to increase your share of the healthcare pie. In any case, the result is the same: Faxes go digital.

You might be thinking, “Great, no more RRD problems.” Au contraire, I suggest you keep your RRD servers well-tuned because faxes will be around for quite some time. Not all the information that offices request can be sent digitally. Plus, not all primary care physicians (PCPs) have embraced the paperless mindset. As long as a patient’s PCP wants to keep a paper patient record in the office, you will need to keep your RRD servers spitting out faxes. The volume of faxes that your organization spits out every day is a barometer for how much work it has in front of it to encourage doctors to move to an EMR system. The tighter your relationships, the more likely your fellow healthcare providers will listen to your vision for moving forward and working together into the digital future.

Prognosis: “Who controls the past controls the future.” If your organization has complete control of its RRD servers and the faxes they send, you will maintain and grow your partnerships with doctors’ offices well into the future.

Next: How to maximize your RRD investment so you have complete control over how faxing is conducted at your healthcare organization.

It’s Just a Fax, Right? Part 1

Doctors Depend on this Old-School Technology

By Chris Lanaman

In today’s need for fast data transfers through the Internet, Wi-Fi or 4G methods, many people feel that faxing is “so old school.” I don’t even think today’s eighth-graders can tell you what a fax is. Throughout the medical community, however, faxing is still an instrumental form of passing patient information. I also believe that healthcare organizations value faxing for its strategic importance in driving business to their facilities over their competitors. Over the next three blogs, I will explain why doctors’ offices consider faxes their lifeline to business, how faxes help healthcare providers maintain their relationships with doctors and how to ensure your information gets where it needs to go when it needs to get there.

Let me set the proper context. Cerner calls its module for faxing Remote Report Distribution or RRD. It is one of the members of the system group called Clinical Reporting. I contend that management should require at least one full-time employee (analyst) to support faxing. Even if the Millennium^® system is configured exactly as designed, RRD faxing can still create large problems for healthcare organizations (more on that in a later blog).

Many doctors’ offices do not have the equipment needed to perform complex clinical tests, so patients are referred to places that have the equipment, which is often the nearby hospital. Both doctors and patients want to get the results as quickly as possible, and hospitals look to their EMR system to be able to provide the timely results through faxes. In addition to faxing radiology findings and lab results, hospitals also use faxing to report on consults, send patient summaries of hospital visits, distribute notification letters to doctors and even send patient appointment reminders. Doctors’ offices rely on these faxes on a daily basis.

I have talked with many doctors’ offices on how faxes affect their processes for seeing patients. Here’s a typical scenario: When a fax containing a patient’s test results comes in, the office staff is responsible for delivering the information to the doctor for his or her review and for setting up a follow-up appointment for the patient to hear the results and begin necessary treatment. From a business perspective, these faxes are helping doctors generate new office visits. Some of the offices I have spoken to say this process accounts for up to 40 percent of their office visits. Without the faxes being sent to these doctors’ offices, physicians would be relying on results being mailed to them or the patients themselves calling for follow-up visits once they received the results from the hospital — neither of which would be acceptable to the referring physician.

So how do organizations leverage faxing to doctors’ offices? Some have contracts with doctors’ offices that include time frames for how soon results will be sent out, such as within 24 to 48 hours of the completion of the test. Additionally, organizations will offer to schedule the faxes to arrive during the nighttime hours or at a time the office deems appropriate. The office just needs to make sure its fax machine is on. All of these commitments can be configured through Millennium’s RRD system. Some of the hospitals I have worked for have processed anywhere from 400 to 3,500 faxes within a 24-hour period. If each fax represents just a $10 co-payment, this could be a lot of money to doctors. Because doctors rely on faxes to deliver quality patient care and drive more business, you can be sure that they won’t want to work with facilities that can’t deliver faxes in a timely and reliable way.

Prognosis: Hospitals can leverage a dependable faxing service to become the provider of choice for referring physicians.

Next: Why faxing is the gateway for healthcare organizations to strengthen relationships with primary care physicians and ultimately promote their vision for practicing healthcare.