Millennium Medic

OnTrack Panther Supports Millennium 2012.x Code

By Chris Lanaman

David Hallberg’s recent blogs on preparing for your Millennium^® site’s upgrade to 2012.x code have coincided with several conversations with clients who use OnTrack™ Panther™ to manage and monitor system performance. They want to know: “Does Panther support Millennium 2012.x code?” And: “Will OnTrack audits still point out issues as I test the new code and then move it to prod?” I want to assure you that all Softek OnTrack products and services support this version of code.

For those of you not familiar with Softek, our software and services give you a high level of visibility into your EMR system along with actionable evaluations and alerts developed and serviced by a team who collectively had years of experience at Cerner and various Millennium hospitals.

As we enter a new year, we have the same great news we have had since Cerner developed Millennium version 7.6: Softek is working to provide your site an objective view into the performance and reliability of your Millennium system.

The following table lists the Millennium code versions supported by OnTrack Panther.

Millennium Version
7.6
7.7
7.8
2001.01
2001.x
2003.01
2003.02
2005.01
2005.02
2007.02
2007.03-2007.17
2007.18.x
2007.19.x
2010.01.x
2010.02.x
2012.01.x
Choose and Book
eBookings

Prognosis: Third-party support that gives your organization intuitive management of your Millennium EMR system can be the foundation for a smooth upgrade to 2012.x code.

Preparing for Your 2012 Upgrade, Part 2

By David Hallberg

Today, in the second half of my recommendations for ensuring a smooth transition to Millennium version 2012.x, I’ll look at Windows and Cerner’s back-end application nodes. (To see my thoughts on Shared Service Proxy settings and WebSphere MQ, see Part 1.) I recommend you check your settings in the following areas of your 2012.x domain:

• Microsoft added the User Account Control to Windows to protect computers from running unwanted programs, but this feature can cause unwanted interruptions in running Millennium applications. If you are using Windows Server 2008 SP1 32-bit or Windows Server 2008 R2 with Virtual Machines (VMs) running Windows Server 2003 R2 SP2 or Server 2008 SP1 32-bit, make sure the User Account Control is disabled. Two documents explain how: “Turn User Account Control On or Off” from Windows and “User Account Control Step-by-Step Guide” from Microsoft TechNet.
• DNS devolution includes improved security in Windows Server 2008, Vista and Windows 7. Make sure you have it set. Documents to explain how are on uCern at “Windows Vista Minimal Configuration Guide, Install Windows 2008” or “Microsoft Windows 7 and Windows Vista Configuration.” Microsoft offers “Post Installation Behavior … after you install the DNSA update.”
• In the past six months, I have run across more and more sites that have incorrect settings for permissions and owners of files and directories. You can correct the settings by running the secure_cerner_500 executable located in the $cer_exe filesystem. Run it as root, and provide the correct inputs. One caution about this executable: If you are running more than one domain on the same server, lpar, npar or vpar, you have to remove the sticky bit on the files in $cer_fifo, $cer_lock and $cer_usock. If you only have one domain per server, lpar, npar or vpar, this will not be an issue.
• Defrag the $cer_config filesystem to improve throughput for the security and transaction database servers.
• Use the bcheck tool to rebuild the dic.idx file on each application node. If bcheck is not in your $cer_exe filesystem, look in $cer_ocd and find the file named bcheck.dat. Rename the file to bcheck, and put it in $cer_exe. You can search uCern or log a Service Request to get Cerner’s recommended flags to use on this. You might also be able to use the -? flag to get a list of the switches that can be used when you create the file.
• Finally, you want to check the dedicated Shared Service Proxies (SSPs) for your security servers to make sure they won’t be overloaded with connections. At a minimum, you need an SSP for each instance of the Security Slave processes. If you want to have the Security Master service on its own SSP, add one more instance than is needed for the Security Slaves. This means you will have an SSP on the application nodes with no connections, but that’s okay. The SSP only takes 3MB of memory. If you have 2 Security Slaves running with your Security Master and 3 Security Slaves on your other application nodes, you will probably have to increase to at least 5 Security Slaves running with the Security Master and 6 Security Slaves everywhere else. This means you’ll need at least 6 dedicated SSPs for the Security Slaves; if you want a dedicated SSP for the Security Master service, you’ll need 7 dedicated SSPs. Remember to start the dedicated SSPs prior to starting additional Security Slaves in order to get the desired load balancing. Your goal is to keep the maximum connections for each Security Slave under 300. You can check the maximum connections from the Server Control Panel. If you want the steps for how to do it, send me a comment, and I’ll send them to you.

Prognosis: If you prepare for conversion by looking at and correcting your infrastructure and key items in your 2012.x domain, you will have a much quieter upgrade.

Preparing for Your 2012 Upgrade, Part 1

By David Hallberg

Readers have been asking about the changes coming with Millennium^® version 2012.x. This major code upgrade is the first Millennium release that supports Linux as a database, and all Cerner clients are being encouraged to support it. When it hits prod early next year, it will contain a number of upgrades to help users file the reports needed for Meaningful Use reimbursements. Over my next two postings, I’ll share what I know about the release from a technical perspective.

Let me start with a warning. Do not follow Cerner’s recommendation to reduce the number of Shared Service Proxies (SCP Entry ID 1) to between 1 and 4. If you do, you will encounter several sporadic performance issues with Millennium that will be extremely difficult to track down. I recommend that you run 10 to 20 SSPs so that you have more consistent response times from all of the solutions. I realize these processes are not used for communicating with the clients or Citrix servers, but they are doing work — without taking up much space. Since each instance takes about 3.3MB of memory, 20 instances would consume only 66MB of memory, a trivial amount for today’s typical application node that has at least 32GB of memory.

The Millennium upgrade process begins by ensuring that your infrastructure is sound. The rest of today’s post will look at IBM WebSphere MQ, and my next post will make recommendations for Windows before moving on to Millennium’s back-end application nodes.

I recommend you upgrade to WebSphere MQ version 7. The links below will take you to IBM documents that give MQ benchmarks for various operating systems. You want to review the sections on Performance and Capacity Limits and Tuning Recommendation. Then choose the largest value between Millennium’s baseline documents and these documents.

AIX: “MP6N: WebSphere MQ for AIX V7.0 — Performance Evaluations”

HP-UX: “MP6O: WebSphere MQ for HP-UX V7.0 — Performance Evaluations”

Linux: “MPL5: WebSphere MQ for Linux (Intel) V7.0 — Performance Evaluations”

Before going live, make sure you correct any errors resulting from the MQ upgrade. Errors will be shown on the Lights On Network under the System Compliance tab. Look for Known Errors Non-Compliance rows that cover MQ.

If you decide to use MQ version 6, you can reference the following documents for your platform. Again, look at the Performance and Capacity Limits and Tuning Recommendation sections, and choose the largest value between Millennium’s baseline documents and these documents.

AIX: “MP6K: WebSphere MQ for AIX V6.0 — Performance Evaluations”

HP-UX: “MP6L: WebSphere MQ for HP-UX V6.0 — Performance Evaluations”

Linux: “MPL3: WebSphere MQ for Linux (Intel) V6.0 —Performance Evaluations”

Due to the increased number of ports being used by MQ on the front end, I suggest you review Microsoft Support article 196271 (previously published under Q196271). Based on Microsoft’s recommendation, I suggest increasing the MaxUserPort to 65534. You can use a tool called TCP Optimizer. Select the link to the new TCP Optimizer version 3.0.8. You want this executable to be run in XP SP2 mode, and you will have to be a local administrator for all the appropriate keys to be modified. Assuming you are running on Windows Server 2008, any version, here’s the steps to follow: Right-click on the downloaded executable, select Troubleshoot Compatibility, select Try Recommended Settings, select Start the program, slide the slider bar to 100+Mbps, select Modify All Network Adapters, select Optimal, select Apply Changes, and select OK. Finally, reboot your computer.

Keep an eye on the MQ statistics for Enqueue and Dequeue counts. If any Java services are in the top 30, you should consider splitting them. The split process would be similar to the process I described in the “Steps to Ensuring a First-Class Upgrade” blog (under step 2, Java Server Stress).

Prognosis: Doing a little research and establishing a solid infrastructure is the first step to ensuring a smooth conversion.

Next: Proper settings for Windows, permissions and ownerships, security slaves and more.

Have You Talked with the Clinicians Today?

By David Hallberg

How do you begin to manage an EMR system as complex as Cerner Millennium^® — with a back end containing more than 35 million lines of code from Cerner along with Linux, AIX, HP-UX, VMS, Oracle, WebSphere MQ, Sonic ESB, MySQL, Java, storage area networks (SANs), networks, clustering software, network connections between Oracle instances, Citrix, Windows servers and desktops; with a front end that has more than 4 GB of space in its code warehouse; and with countless application builds?

I recommend you start your work by stepping away from your computer screens and talking to the people who are using the system to care for patients. Are they frustrated with response times? System availability? The number of clicks to admit or discharge a patient? Don’t dismiss their complaints with the assumption that “clinicians just don’t understand computers.” They are highly skilled people who understand the most complex system on the planet: human life. You might be surprised what you can learn when you take the time to really listen. Or shadow them during their rounds.

When I worked at Cerner, I brought my stopwatch with me to one hospital to join the CMIO and director of nursing informatics on the floor. It took more than 15 seconds to place one order in PowerChart^®. That’s an eternity when you’re with a patient and still have the rest of your rounds to complete. When I took the data back to the engineering lead responsible for the code, she emphatically told me, “Fifteen seconds is fast enough. They (the clinicians) just do not understand computers.” No wonder the clinical staff was frustrated. The system was hampering their ability to provide patient care, but their concerns weren’t given any consideration by the people capable of addressing the problems.

So I encourage you to gather the staff’s list of concerns, criticisms and objections. If they feel they’re not normally heard, no matter how bad their experiences, don’t be surprised if they’re distrustful at first and reluctant to talk. Keep at it. Move through the obligatory moans and groans to get to their true desires.

Now that the easy work is done, begin to rebuild their faith in the system by evaluating each complaint against each need and determine how to address them. You can use Millennium’s greatest asset, its flexibility, to configure it for very specific purposes. If the issues outweigh your staff’s ability to correct them — we’re back to that complexity I mentioned earlier — don’t be too proud to seek outside help. (If you’ll be at the Cerner Health Conference next month in Kansas City, I’d be happy to listen to your system concerns and talk through some options.) Then you can begin the journey of reconfiguring this multifaceted solution set into the system your clinicians actually enjoy using rather than avoid using or simply tolerate.

Continuing to Improve Failover: Security Master Configuration

By David Hallberg

After our last post, “Improving Interface Failovers,” a few people wondered why some Cerner Millennium^® servers — most often Ops Job (Millennium batch process), interface and MDI servers — would not start during the failover. Because this situation can have disastrous consequences, it’s an issue worth addressing.

Imagine the following scenario: It’s 3 a.m. on Memorial Day, and your primary application node crashes. The failover scripts run so that users can log into Millennium, but none of the Ops Jobs are running. That means the nursing staff is missing care plans for its shift change, new patients and their lab results are not visible in Millennium, and the Emergency Department director is calling the CIO, COO and CEO complaining that his staff has to go on diversion because they cannot get the patients into Millennium. A storm just ruined your holiday plans.

Had you been aware of a few simple steps that would make your servers fail over successfully, your day could have been clear sailing.

Why didn’t the Ops Job servers start? If the server on a specific Server Control Panel (SCP) Entry ID has not been started on this node before, the Security Master server has to be running to create the service with the correct permissions. If no Security Master server is running after the failover process completes, any of the servers or services that have not gotten to a running state on the failover node prior to the failover cannot be started.

A little analysis will explain why this scenario happens so frequently. Way back when, about 90 days before you first brought Millennium up, consultants from the vendor swooped in to set up failover. They ran through a couple of tests and showed you how, should the primary node go down, you could still log into Millennium. They flew back home, and you turned your attention back to what seemed like more critical work that had to be done before go-live.

Now, many years later, you have added and deleted Ops Jobs, your old registration system has been upgraded, the inbound feed changed names to allow testing into Millennium production and to minimize the cutover time, you added about a dozen MDI servers, you moved on to a different Millennium client, or you were promoted, and your project staff is scattered around various clinical areas. Go-live and pre-go-live are remembered for the number of service packs you had to install and test and the huge amount of training you had to do. Who recalls the little detail about the need for the Security Master to be running to allow a new Millennium server to start on a node?

What, in fact, is the Security Master? For organizations using LDAP to access and maintain their user directory, the Security Master server updates the sec_user.dat file with local accounts, for example SYSTEM, SYSTEMOE and CERNER. For those not using LDAP, the Security Master also is used to add, remove, modify and delete all user IDs. So when a new person joins your organization and you run the tool HNAUser, a new row is added to sec_user.dat as well as some Oracle tables. Additionally, the Security Master creates a service key for the Millennium processes as they start up. These service keys only exist in the sec_user.dat file; they are not kept in LDAP. This key is used to make sure the processes running on the application nodes are legitimate processes. During normal operations, no one really cares about the security services. They quietly do their work when you stop and start Millennium servers on the application nodes, and everyone — especially your security manager — is happy. Until a failover.

Let’s go back to our Memorial Day crisis, where your family headed to the lake without you. You, like most Millennium clients, have Node 1 of SCP Entry ID 32 configured as the Security Master and Node 2 as the Security Slave. With this common configuration, however, you do not fail over the Security Master, so all of the Ops Jobs you added after go-live, along with new interfaces and MDI servers, didn’t seem to start. Yet nothing shows in any of the message logs. You called your vendor’s Immediate Response Center, but the response has not been immediate enough. After calling you back in 15 to 45 minutes to verify you are still having the problem, IRC might take four more hours to contact you again. In the meantime, the ED director is calling for your resignation and the ICU, CCU and NNICU staff are looking for a throat to choke.

How could you have prevented the chaos and been fishing for something other than answers? The single biggest thing you can do is NOT put a Security Slave on SCP Entry ID 32. Use any other SCP Entry ID — 31 and 19 are often open. After you have the Security Slave configured to run on a different SCP Entry ID, copy the Security Master from the primary node to the secondary nodes. Verify that the Instance is set to 0, which will ensure that SCP Entry ID 32 does not start up and that CPM HNAM Agent does not start it. Finally, put 32 in the cerner.vars file as a single-instance server. That way, when the failover scripts run, you will be able to start the Ops Jobs, interfaces and MDIs as needed to minimize your downtime.

Here’s one last detail for all the system administrators who like such specifics: Create a chron job to automatically copy the $cer_config’s security.jnl file from the application node where the Security Master is running to the other application nodes. Then verify that the permissions are “-rw-rw-----” and the owner and group (assuming your domain name is prod) are both d_prod.

Prognosis: A little preparation can dramatically improve the failover process and make your night, weekend and holiday events much shorter and quieter.

A Time to Use Messages.Suppress

By David Hallberg

If you are a regular reader, you know that over the last couple of years I have strongly advocated minimizing the use of Cerner Millennium’s messages.suppress file. Functionality that was intended to prevent extraneous events from cluttering up your message logs has been horribly misused by Millennium system administrators and the application support and application development staff. As a result, vital events are not being sent to the message logs, events that need appropriate troubleshooting and longer term problem resolution. (See “When No News Is Bad News” and “File Can Hide Serious Issues.”)

In recent months, however, I have seen some improvements in Millennium’s base tuning and configuration when a particular event is suppressed, so today I have an unexpected recommendation.

The improvement involves an event called SEC_seclibAudit, which is an Audit-level event telling you that debug is not enabled for the security components. A couple of years ago, only CPM Code Cache Manager seemed to log this event. With the 2010.x code, however, SEC_seclibAudit appears to be written by all of the Millennium servers. It generates two rows in the Data column of the message log each time it occurs. The first row says:

[SecDebugLog::Instance] Creating debug logging singleton.

The second row says:

[SecDebugLog::CheckConfiguration] Debug logging not enabled for all security components, componentLogicalName (SecDebugLogsecrtl)

This is a good setting, but you don’t need to be told about it with every executable. It would be better if you could assume the setting is off unless you took action to turn it on. (To enable debug logging, set the LogLevel of a server to 4 or set the environment variable or logical of CPM _GLOBAL_LOGGING or any one of a number of other logging increases that are available to individual Millennium servers.) This message has become so prevalent, logging on all servers, it has become noise. One client, who cycles 19,000 servers each week, gets 38,000 messages every week for this one event.

To stop the noise, I have to make a recommendation I rarely make: Put this event in the $cer_mgr directory’s messages.suppress file. Then cycle all of your multi-instance servers to make the change take effect.

Prognosis: Using the messages.suppress file appropriately stops extraneous logging and keeps your message logs clearer for real problems.

New Code and Lights On Network Compliance Tabs

By David Hallberg

Several Millennium^® users I’ve talked with since the first of the year have had problems in production after taking new service packages. All had the same question for me: “How could we have anticipated the problems we now have in production?” Because some of the problems involved a new solution being brought up, I had a question for them: “What is your compliance percentage on the System Compliance and Solution Compliance tabs of the Lights On Network?” Either they didn’t know what I was talking about or they only had the tabs enabled for production. In their ignorance, they were missing out on key data from Cerner and its clients that could have helped them prevent the issues they were experiencing.

New code undergoes several quality checks before Cerner releases it to its client base. However, internal testing can never be rigorous enough to anticipate the varied and complicated production environment. So several years ago the Lights On Network was set up to allow clients to see several aspects of their configuration. The two most useful, from my perspective, are the System Compliance and Solution Compliance tabs.

The System Compliance has four dials: Known Errors, Oracle Configuration, OS Configuration and HNAM Configuration.

• Known Errors shows quality issues, sometimes called CIRB, as well as CernerWorks Process Improvement (CWXPI) corrections. Essentially, when one or more clients have a problem in a production environment or a potential for a problem in production, notifications are sent out to alert all clients of potential issues (CIRB). The CWXPI items are problems found at sites that Cerner manages or hosts, and to improve all Millennium environments, Cerner publishes how its teams corrected these issues.
• Oracle Configuration shows clients what several basic Oracle configurations should be when they first go live. There may be reasons your configuration differs from the recommendations. These reasons or differences can be documented in the Inactivate selection from this screen, an option available on all four of these dials.
• OS Configuration shows what basic Operating System configurations should be. It’s a very good baseline. If you deviate from it, you need to be certain why.
• HNAM Configuration shows more CWXPI items, basic logging settings and some solution/application configurations. Generally, this tab focuses on technical aspects of the Millennium environments.

The application side of the fence is found in the Solution Compliance tab. This tab shows topics such as what Ops Jobs (Millennium batch scheduling) need to be run and recommendations on how to configure Ops Jobs if they are not being run. The tab provides feedback on a subset of the Millennium purges. These purges delete temporary data that is not considered part of the permanent patient record. It helps control the size of the Oracle database and can improve or stabilize performance.

Prognosis: Once you and your organization are aware of the quality checks available on the System Compliance and Solution Compliance tabs of Cerner’s Lights On Network, you can put processes in place to leverage this service so that you reduce problems in your Millennium domains and improve the effectiveness of your infrastructure.

Why Auditing Your EMR Is Essential

By Chris Lanaman

I want to expand on the comments I made in my last blog on the need for third-party auditing of your EMR system. I’m including an excerpt from the white paper “Why Auditing Your EMR Is Essential” that our Softek team wrote and distributed last week at HIMSS. The message struck a chord with the people we talked with — both self- and remote-hosted organizations— so I’m confident it will resound with you as well.  

When patients are admitted to your hospital, they count on reliable, competent care:  caregivers who are board certified, care delivery that has passed the standard of the Joint Commission, lab procedures certified for quality by the College of American Pathologists, certified dieticians who oversee patients’ dietary needs and ethical business and financial practices. In each of these areas, facilities and individuals submit to regular, independent evaluation to make sure their practices follow generally accepted industry standards and, when necessary, implement changes to bring them back into compliance.

One area overlooked in these evaluations is the operation of the EMR system itself. With more and more functionality demanded from these systems, they continue to grow in complexity. Yet no independent evaluation program is required to ensure that the EMR is performing as expected. And if it’s not, the consequences could be devastating for patients, caregivers and the facility itself.

Think of the changes you continually introduce to your configuration. Besides the customizing you do for your own unique uses, vendors are supplying a steady stream of system updates and upgrades. A technology upgrade can create myriad downstream impacts on how the EMR will behave. Every time you add a new layer, a new process, a new handoff point, you increase the likelihood of delivery failure. Each failure poses the risk of losing patient information or financial data or extending wait times for clinicians trying to input or access that information.

If your EMR system is hosted outside your own organization, you might think independent auditing doesn’t apply. After all, one of the primary reasons you sought out a knowledgeable vendor was to oversee system performance. You wanted the assurance that comes from having competent partners hosting your system. Yet how do you know that the hosting partner is catching all the performance problems that arise? Because so many performance and stability issues arise from errors with application settings rather than from technical infrastructure, who is really responsible?

Your hospital IT team knows when performance problems develop — you deal with the complaints. With the visibility that comes through a system evaluation, you can more confidently partner with your service provider to not only address system issues but also set higher standards for performance and quality of patient care.

As healthcare focuses more and more on evidence-based medicine, the IT delivery vehicle for that medicine should be evaluated in the same way: using real data from your own system, gathered by an organization independent from the vendor or your own organization, and clearly presented so that you can understand where you are and what adjustments need to be done to accomplish your goal of optimal healthcare.

Prognosis: You can learn even more about how to get an evaluation of your EMR by reading our white paper, “Why Auditing Your EMR Is Essential.” Click here to receive it.

Betrayed by an Old Friend

By David Hallberg

I had a client complain to me recently that I was reporting a false positive, an alert that was triggered even though there was, in fact, no problem or issue with his Millennium^® EMR system. I decided to figure out why there was a discrepancy between my measurements and the client’s quality-assurance tool, which happened to be sysmon.

Let me begin by saying that I love sysmon. Before sysmon was developed in the late 1990s, Millennium had so many tools or utilities that provided a single view of a piece of information that it was difficult to keep track of all of them. At one point there were more than 200 build or monitoring tools even though Millennium only had about 40 actual application executables that were used in various configurations for its 60 plus solutions. Multi-node clients kept asking their Millennium middleware team to figure out where performance bottlenecks were occurring. Believe me, this sleuthing was extremely labor-intensive. Users had to have mon_ss (a Millennium monitoring tool used to watch for proprietary middleware queuing) and joumon (a Millennium monitoring tool used to watch for persistent queuing in the proprietary middleware) up in a different window for each application node. Then they would have to try to show someone else when one or more of these screens had a “blip” or negative performance indicator. It was painful on the good days.

So those managing Millennium were understandably ecstatic when Cerner developed a single utility that told what was happening across Millennium’s persistent and non-persistent queuing mechanisms across all application nodes, in a single screen. Sysmon also provided the date and time when something bad happened.

Sysmon proved to be reliable, and I became a proselytizer for this great monitoring tool. While I was at Cerner, I taught many clients how to use sysmon and its various switches. I also used it to tune a four-node client with ConnectionBias (a load-balancing mechanism for Millennium), moving the transaction workload from one application node to another. I was able to show enough value with this one tool that some clients actually chose not to purchase BMC Patrol (later called BMC Performance Manager and now called BMC ProactiveNet Performance Management). Sysmon had some limitations — it was really difficult and unreliable to pipe or move the output to a file for additional processing, and it had no mechanism to redirect its output to a common data format like CSV (comma-separated values), tab-delimited or regular flat files — yet it was the most accurate view of a multi-node application configuration.

So why was my data conflicting with sysmon? It appears that with 2007.02 some of the newer Millennium services do not register with the old middleware in a way that allows sysmon to see the queuing. With the client I referred to earlier, I found that the SenSage add-on called P2Sentinel did not publish its queuing in a way that sysmon could see it. The toolset I used, however, did see the problem.

That’s not the only problem. Another client reported that one of the Java services introduced with 2007.02 was crashing several hundred times a day. I ran sysmon to see if it would show these crashes, and it did not. I am not exaggerating to say that I was crushed. Sysmon was the one tool I absolutely trusted to let me know in a timely manner when Millennium was failing, typically 15-45 minutes before BMC. Now I find it is not reliable!

If, as it appears, the newer programs will not share or publish their information like the old programs, Millennium clients now must find other tools. None of the options, however, shows them this information in a single view. Sadly, we all have to go back to running multiple windows for each application node when we are using command-line tools.

A source of truth that you can trust and believe in is difficult to find. It is even more difficult to see it no longer be a source of truth.

Determine if Millennium’s Size Is Just Right

By Chris Lanaman

Like Goldilocks trying out chairs at the cottage of the Three Bears, every healthcare organization is looking to land on a system that is sized “just right.” When you first had Cerner Millennium^® installed, smart people thought through what kind of hardware and how much of it would need to be deployed in order to meet your organization’s needs. Most likely they also considered the projected growth of the organization over the next five to 10 years. Buying too much would add complexity for your users, loss of control in managing the system and, of course, wasted cost and time. No one wanted Millennium to be like Papa Bear’s too-big chair.

I would suggest that a bigger problem exists when the Millennium system is not sized big enough for the demands of your organization. Any one of the following symptoms might indicate that your workload has outgrown your system: lower than expected productivity of outcomes, the loss of business revenue because of turning away patients or poor response time for end-users.

I recommend starting with the following five steps if you are concerned that your organization has outgrown its current Millennium size.

1. Data, data and more data. You must have data in order to see if your organization is suffering from a sizing issue. This data is gathered by your team of data-miners (IT analysts) or through an application that is generating reports for your system, such as Cerner’s PowerInsight or PowerVision. Use this information to see if there has been a drop in activity, a workflow slowdown or loss of productivity.
2. Patient management cycle. Once you have obtained measurements for your organization’s workflow, you will need to evaluate where you are in your patient cycle — is this the time of year that you have increased patient traffic or are these your slower months? Your metrics should also include a staffing model to meet patient demand. Understanding your cycle will keep you from over-sizing your system if you evaluate during the high point of the year or under-sizing if you are in a valley part of your cycle. These measurements will identify the revenue cycle expectations for your organization.
3. Mergers and acquisitions. When your organization merges with another healthcare organization, great pains are taken to try and unify the new organization under a common vision. The impact of the merger on the EMR system’s workload, however, might be overlooked. The IT analyst needs visibility into the new organization’s past data to understand how workloads might increase.
4. Market share. Does your organization understand the footprint it has in your market? When trying to understand growth potential, you need an understanding of your opportunities and who the suppliers in the market are. Theoretically, one can say a set percentage of business equals a set amount of hardware needed, but this is an educated guess at best and should never be the rule of thumb.
5. Is your Millennium system healthy? As I alluded to earlier, when your system is not sized correctly, you will see an increase in performance issues. If your end-users are telling you, “The system seems slower than it used to be” or “It seems to take longer for me to do my job in the system,” the first thing on your mind should be the health — i.e., the efficiency — of your system. Before joining the Softek team, I used Softek Panther for a number of years to uncover trouble spots. It let me know when Millennium stepped out of line. Additionally, Softek’s Queuing Performance Spreadsheet (QPS) allows you to see where within Millennium your user transactions are getting clogged, and the Messaging Performance Audit (MPA) gives your system an overall checkup. The audit lets you see if you have build issues in Millennium, if transactions are going where they’re supposed to go, if your servers are configured correctly and when your servers are being over-tasked. If you have other monitoring tools you recommend, please let me know. Whatever you use, it’s important to check your system thoroughly before buying more hardware. Hardware purchases should always be considered the last resort for resolving performance problems. 

In a blog, I only have space to give you the Reader’s Digest Condensed version for scoping your organization’s size and determining how that affects Millennium’s performance. But addressing these key points will help keep your organization healthy and your users happy.

Prognosis: Evaluating all of the different factors that make up your organization is the first step in understanding if your Millennium system is sized “just right.”

Putting the Profit in ProFit

By David Hallberg

I’ve written previously about lost charges and how to examine Millennium’s message logs to find and correct billing issues. Today I want to speak specifically to clients who use ProFit^® Patient Accounting, Millennium’s billing solution. Those wanting to maximize their revenue will find a wealth of information in the message logs on the ProFit servers.

One of ProFit’s components is a third-party subsystem called Sablotron, an XML toolkit that was used to make ProFit easier to support across operating systems (http://freshmeat.net/projects/sablotron). Because it is XML based, Sablotron is independent of the operating system and works the same on AIX, HP-UX, Linux and VMS. This is great from an engineering support perspective. But what does it mean to a healthcare organization?

ProFit has two primary servers: ProFit Transaction Server (SCP Entry ID 475) and ProFit Billing Server (SCP Entry ID 476). Generally, an organization can only control the memory size and loglevel of these asynchronous processing servers, and interactive users typically do not know when these servers (executables) have had any errors or warnings. Someone would have to go digging for these issues to become aware of them. I recommend you get out your shovels.

I recently ran across two events on the Billing Server of a ProFit client. To correct the first issue, we needed to log a service request to get help in correcting the build. For the second, we needed to log a service request to get new code.

The first event was the Sablotron Message, which indicated that the subsystem could not properly handle some data in a billing transaction. In a single four-hour time frame, my client had 4,002 of these events. The Data component of these records had a consistent beginning section that looked like this:

{{msgtype::warning}}{{code::93}}{{module::Sablotron}}

It was followed by one of three ending components.

1.  The most numerous Sablotron Message ending was:

{{URI::arg:/_parsed_}}{{line::544 }}{{node::element '<xsl:for-each>'}}{{msg::unsupported language 'en'}}{{Routine::sablotronErrorHandler::log}}

You’re probably thinking, “This looks like someone sat down at the keyboard after drinking a bit too much.” But the message actually illustrates how many XML files are laid out or formatted. XML is just another way to show data, and the message tells us that in line 544 of the server, there was a language code of “en” that did not have a translation (‘<xsl:for-each>’) and the server didn’t know what to do with it. So the server wrote the message to its message log.

2.  The second most numerous Sablotron Message ending was:

{{URI::arg:/_parsed_}}{{line:: 497}}{{node::element '<xsl:for-each>'}}{{msg::unsupported language 'en'}}{{Routine::sablotronErrorHandler::log}}

This looks like the same drinker was at the keyboard. The message is the same as the previous one, except that the unsupported language code of “en” was in line 497.

3.  The third Sablotron Message ending was:

{{module::Sablotron}}{{msg::unsupported language 'en'}}{{Routine::sablotronErrorHandler::log}}

This message appears to be a higher level failure, but the end result is the same. The server had a language code of “en” that it could not translate, so it wrote the message to its message log.

The second event my ProFit client experienced with the Billing Server was SRV_LookupFailure, which told us in the Data that the bill_item_id was missing from the transaction. This would generally not be a problem, except that in this case the user name was SYSTEM and, therefore, no interactive user was made aware of these failures. They were only in the message log.

Not all the people I work with think these messages are a big deal — they see errors in Millennium as normal. I disagree. The client I’ve referenced here had, on a single node, 3,038 of the line-544 Sablotron Message events and 904 of the line-497 Sablotron Message events in two hours as well as 60 higher level Sablotron Message events and 92 SRV_LookupFailure events in one hour. How could this be considered normal? If an executable is having a problem, you want to correct it, whether the issue is in the build, the code or the use of the code.

Prognosis: Little events can add up to big revenue losses. Looking for these events and resolving the code or workflow issues will maximize your revenue.

It’s Just a Fax, Right? Part 2

Keeping relationships strong with PCPs

By Chris Lanaman

My last blog showed why doctors’ offices have a strong reliance on receiving test results and patient information through faxes — which are sent through Millennium’s Remote Report Distribution (RRD)  servers — and why it’s important for hospitals to make sure their faxes go out on schedule. Today we’ll look at another strategic importance of faxing, one that will help your healthcare organization distinguish itself from others.

In his novel 1984, George Orwell wrote the following ominous statement related to information control: “Who controls the past controls the future. Who controls the present controls the past.” I want to present a more practical paraphrase of the quote: “Whoever controls the old technology (like faxes) will be able to drive the introduction of the new technology (like interfaces).” What do I mean? The more relationships that your organization has established and is actively sending faxes to gives you the power to drive conversations about joint ventures in connecting electronic medical record (EMR) systems through interfaces.

Although many doctors’ offices do not have any form of EMR system, the number of “office size” EMRs has grown in the last five years and will continue to grow. As part of this conversion, I have found that larger doctors’ offices are happy to say good-bye to the piles of faxes that can often be difficult to manage. They still want notifications but are ready to move to a “paperless” environment.

When these offices start asking you how their number of faxes can be reduced, they are giving you the opportunity to strengthen your alliance with their office. If you can successfully interface your system with theirs, you stand a good chance of merging your organization with theirs to increase your share of the healthcare pie. In any case, the result is the same: Faxes go digital.

You might be thinking, “Great, no more RRD problems.” Au contraire, I suggest you keep your RRD servers well-tuned because faxes will be around for quite some time. Not all the information that offices request can be sent digitally. Plus, not all primary care physicians (PCPs) have embraced the paperless mindset. As long as a patient’s PCP wants to keep a paper patient record in the office, you will need to keep your RRD servers spitting out faxes. The volume of faxes that your organization spits out every day is a barometer for how much work it has in front of it to encourage doctors to move to an EMR system. The tighter your relationships, the more likely your fellow healthcare providers will listen to your vision for moving forward and working together into the digital future.

Prognosis: “Who controls the past controls the future.” If your organization has complete control of its RRD servers and the faxes they send, you will maintain and grow your partnerships with doctors’ offices well into the future.

Next: How to maximize your RRD investment so you have complete control over how faxing is conducted at your healthcare organization.

It’s Just a Fax, Right? Part 1

Doctors Depend on this Old-School Technology

By Chris Lanaman

In today’s need for fast data transfers through the Internet, Wi-Fi or 4G methods, many people feel that faxing is “so old school.” I don’t even think today’s eighth-graders can tell you what a fax is. Throughout the medical community, however, faxing is still an instrumental form of passing patient information. I also believe that healthcare organizations value faxing for its strategic importance in driving business to their facilities over their competitors. Over the next three blogs, I will explain why doctors’ offices consider faxes their lifeline to business, how faxes help healthcare providers maintain their relationships with doctors and how to ensure your information gets where it needs to go when it needs to get there.

Let me set the proper context. Cerner calls its module for faxing Remote Report Distribution or RRD. It is one of the members of the system group called Clinical Reporting. I contend that management should require at least one full-time employee (analyst) to support faxing. Even if the Millennium^® system is configured exactly as designed, RRD faxing can still create large problems for healthcare organizations (more on that in a later blog).

Many doctors’ offices do not have the equipment needed to perform complex clinical tests, so patients are referred to places that have the equipment, which is often the nearby hospital. Both doctors and patients want to get the results as quickly as possible, and hospitals look to their EMR system to be able to provide the timely results through faxes. In addition to faxing radiology findings and lab results, hospitals also use faxing to report on consults, send patient summaries of hospital visits, distribute notification letters to doctors and even send patient appointment reminders. Doctors’ offices rely on these faxes on a daily basis.

I have talked with many doctors’ offices on how faxes affect their processes for seeing patients. Here’s a typical scenario: When a fax containing a patient’s test results comes in, the office staff is responsible for delivering the information to the doctor for his or her review and for setting up a follow-up appointment for the patient to hear the results and begin necessary treatment. From a business perspective, these faxes are helping doctors generate new office visits. Some of the offices I have spoken to say this process accounts for up to 40 percent of their office visits. Without the faxes being sent to these doctors’ offices, physicians would be relying on results being mailed to them or the patients themselves calling for follow-up visits once they received the results from the hospital — neither of which would be acceptable to the referring physician.

So how do organizations leverage faxing to doctors’ offices? Some have contracts with doctors’ offices that include time frames for how soon results will be sent out, such as within 24 to 48 hours of the completion of the test. Additionally, organizations will offer to schedule the faxes to arrive during the nighttime hours or at a time the office deems appropriate. The office just needs to make sure its fax machine is on. All of these commitments can be configured through Millennium’s RRD system. Some of the hospitals I have worked for have processed anywhere from 400 to 3,500 faxes within a 24-hour period. If each fax represents just a $10 co-payment, this could be a lot of money to doctors. Because doctors rely on faxes to deliver quality patient care and drive more business, you can be sure that they won’t want to work with facilities that can’t deliver faxes in a timely and reliable way.

Prognosis: Hospitals can leverage a dependable faxing service to become the provider of choice for referring physicians.

Next: Why faxing is the gateway for healthcare organizations to strengthen relationships with primary care physicians and ultimately promote their vision for practicing healthcare.

Another Upgrade Issue: The TDB

By David Hallberg

Although I have blogged before on issues that might be overlooked during a Millennium^® upgrade, a recent discussion with a client revealed another issue: one with the transaction database (TDB). A common oversight in this area might need a simple fix.

The TDB is a flat file found on each Millennium application node that tells the executables (application servers) what transactions they can process and if the transactions are Request/Reply (RR), Reliable Delivery Message (RDM) or Remote Procedure Call (RPC). Let me start with some background on these three types:

· With RR transactions, the clinical information remains intact even if the transactions are lost or discarded. For instance, using CPM Script to retrieve or read data from the database is an RR transaction. If the transaction is lost, the person asking for the information just has to ask for the information again.

· An RDM transaction is like an order. If the client crashes or the application executable or application node fails, there has to be a mechanism in place to recover or replay that transaction when the system is fully functional again.

· The RPC transaction was added with Millennium 2007.x. It allows the Java servers and Java client applications to process transactions originating from both the local hardware and remote calls. It was the first extension to the Common Request Model in quite some time.

Until recently, the TDB was a connection-based service. PowerChart or the front-end application connected directly to the TDB’s port. The TDB is now a shared service. The TDB server communicates with Citrix servers and fat clients over a network port that it allocates when it starts up. Once the communication port is selected or opened, the TDB server tells the Millennium Service Manager on the application node which port it will be listening to for incoming transactions (requests). The Citrix servers and fat clients must be on the same code level as the application node so that when PowerChart or any front-end Millennium application starts up, it will connect to the Service Manager and be told what port to use to connect with the TDB.

I have found that sometimes during an upgrade, the fat clients or outlying Citrix servers are overlooked. In the case of the TDB, fat clients and Citrix servers that aren’t upgraded might be connecting to the wrong port. If a connection fails because the TDB was not listening on that port, the Service Manager logs a message stating that event CMB_ServiceUnavail1Net occurred. The service name (such as TDB_SYSTEM@PROD) is logged, the Citrix server or fat client’s TCP/IP address is included, and the port the client was trying to communicate with is logged. You can see these messages by looking in the cmb_0000 message log and search for the event CMB_ServiceUnavail1Net.

An example would look something like this:

Domain	Name	Node	Record	Cont.	Date
prod	cmb_0000	Hallberg	{{service_name::TDB_SYSTEM@PROD}} {{tcp_address::10.10.10.10}}{{tcp_port::1240}}	N	8/17/2010 6:53

Event	Level	Location	No.	Source	Sourceref	User
CMB_ServiceUnavail1Net	1	hallberg/ 20200cce/8	1022	cb_service_mngr	cb_service_ mngr.cpp:5635	n/a - [d_prod]

The message under the Record column indicates that the Citrix server or fat client whose TCP/IP address is 10.10.10.10 is trying to communicate with the TDB on port 1240. Because this happened after a Millennium upgrade, the client just has to use tracert or traceroute (depending on your operating system) to find out how far the problem server or client is from the application node, i.e., how many network router hops you have to make. The Citrix server or fat client needs to be scanned for components after the Millennium client upgrade in order to be in sync with the newer application node code. This is the most common problem and is simple to correct.

Prognosis: Reviewing message logs after an upgrade can highlight issues that make Millennium appear unusable or unstable to the clinicians whose Millennium servers or fat clients have not been upgraded.

 

Get a Charge out of Failed Events

By David Hallberg

In 1997, when I first began performing technical audits on Millennium® infrastructures, the biggest concern was to keep the Millennium middleware functional and to tune the Oracle kernel. My job was to make sure the interactive user transactions made it to the Millennium application nodes and the database was able to keep up with the workload. These days, however, the MQ generally is working well and the dependency on the Millennium middleware is greatly reduced, so there’s a lot less to tweak in those areas. Likewise, Oracle nodes/lpar/vpars have at least 64 GB of RAM and large system global areas (SGAs) and provide much higher throughput for the Millennium solutions.

Unfortunately, as these issues have improved over the past 12 years, other key problems have surfaced. One trend that I’ve found in the last two years is an increase in lost charges. Analysis of Oracle tables and Millennium log files show that clients are losing charges at two points: in the orders/activity to charge mapping and at inbound interfaces.

Most clients don’t believe me when I highlight their billing mistakes. I find they get rather defensive instead of eager for me to show them how to fix their problems, some of which can be done fairly easily.

If you want to investigate whether your system is losing charges — and your organization is losing revenue — start at the message logs for CS Charging Async and CS Charging Sync. You want to look for IncompleteEvent, AddBillCodeType, AddItem_Failed, CheckCharge, CheckChargeModForType, DBAPI_FatalScriptErr, OffsetCharges and ScriptError. These events represent the problems I have seen most recently. All but DBAPI_FatalScriptErr and ScriptError are build or configuration issues that typically need only a little research to resolve.

The DBAPI issues are often Oracle space or connectivity issues that should be resolved with your database administrator. If the DBAPI errors are constraint violations or lock issues, they are probably code issues that require you to take updated Millennium service packages. The ScriptError events are almost always code issues, also requiring you to take updated Millennium service packages.

Your investigation might show a staggering number of ESI errors and warnings. The ESI issues are often from the Omnicell or Pyxis interface, which could indicate you are losing pharmacy charges. Even if it’s only 1-2% of all pharmacy charges, your organization could be losing thousands of dollars in revenue.

From the ESI logs, the failures are often from a “null person_id.” That means Millennium received a transaction but it couldn’t apply the charge or event because the patient was not identified. Other failures could include a “null meddef” or “null manufactur id.” Both of these are build issues in which Millennium is receiving incomplete or mismatched information and therefore cannot update the database. ESI warnings often include aliasing issues, script issues, invalid catalog_cd, action_type_cd, order_id or synonym_id. The problem also could be as simple as missing data or an incorrect date/time format.

The ESI log typically contains only the last seven days of data. Failures, therefore, are purged and the information lost unless you are looking at these issues at least once a week.

All of these items are straightforward to correct — once you know about them. The key is to be willing to believe the data in your log files, use it to identify the failed charges and have a process in place to correct the problem.  

Prognosis: Addressing the build issues associated with the charge errors in your message logs and ESI log will translate into increased revenue for your organization. 

Preparing for an Upgrade?

By David Hallberg

We have seen so many Millennium® clients have problems after upgrading to 2007.18, 2007.19 or 2010.01 that I feel compelled to address this issue. If you have not yet upgraded, this blog will link you to specific recommendations to help minimize the known risks to your production environment. It’s best to execute them before the upgrade. If you have already upgraded, you may want to review these items and see how many of them you can do to improve your current stability.

In general, I’ve ranked my 14 recommendations in order of importance. The majority of them are MQ and message log issues. They apply whether you are on AIX, HP-UX or VMS (although VMS applies only through the 2007.19 upgrade since Cerner doesn’t support this platform on 2010.01 and above). Click here to read the specifics on correcting the following 14 issues:

1.    The hidden MQ queue.

2.    Java server stress.

3.    Citrix crashes.

4.    Exception queue transactions.

5.    Message log disk I/O increases

6.    Confusing MQ log files.

7.    Extra message log writing.

8.    Too much extraneous logging.

9.    MQ information loss.

10.  Missing or incorrect security settings for a file.

11.  MQ running in mixed mode.

12.  Request class routing needs.

13.  Wrapping issues with message logs.

14.  QCP server instances.

Prognosis: By performing a handful of critical tasks before an upgrade, I believe you’ll have a much shorter troubleshooting time during the upgrade process.

Shared Service Proxies in an MQSS World

By David Hallberg

Millennium®, since version 2007.02, has tried to reduce its proprietary middleware footprint through the use of IBM’s WebSphereMQ for client communications to the application nodes. This is often referred to as MQSS for WebSphereMQ Shared Services. It provides an enormous leap forward in throughput and performance for most Millennium users but has not lessened the need for Shared Service Proxies (SSPs), as was expected.

Millennium’s initial design specification for its proprietary middleware was 125 messages per second, which was pretty good when it was introduced 15 years ago. Plus it often saw higher throughput thanks to newer compilers, faster CPUs and some other changes. WebSphereMQ on version 5.3 for this sort of communication (often referred to as non-persistent queuing), however, was benchmarked at more than 3,200 transactions per second, and 6.2 is benchmarked at more than 3,800 tps, where each transaction is 2 kB in size.

The diagram above gives a simple overview of how shared services work with IBM MQ. Starting at the left of the diagram, we have a client application that is connected to both a request and a reply queue. These queues are used to transport the client request to the application servers and the reply back to the client application.

If you are an AIX or HP-UX client, I recommend that you enable MQSS as quickly as possible to help your throughput. I wish I could say the same for VMS clients. Unfortunately, VMS experiences capacity issues with MQSS. Its TCP/IP stack and BG devices can exhaust their limit of 32,768 ports, at which point they no longer take new connections. Cerner now recommends that all VMS clients stop using MQSS, except for the Java services, which only know how to communicate via MQSS.

While all of this speed is great for clinicians and business staff using Millennium, there is a hitch. Updated Millennium configuration standards state that when MQSS is enabled, the number of SSPs (SCP Entry ID 1) should be reduced to four and that all dedicated SSPs should be disabled. Based on the client data I’ve seen, numerous clients who have gone live with MQSS with only four SSPs have experienced varying degrees of queuing, backlogs or slowdowns in the SSPs. I recently worked with a client who had more than 82 hours of queuing for the CPM Script Batch 001 service.

Their queuing was typical of many others: It was constant, not centered around normal usage patterns of one peak in the late morning and another in the early afternoon. The only way to mitigate this backlog is to increase the number of SSPs or create a dedicated SSP, stop all of those servers on the node and then restart them. This action will move the service to the new generic  SSP or dedicated SSP and should stop the queuing. (To learn more about SSPs, read previous blogs on “SSP’s Path for Rapid Transactions” and “Setting up a Dedicated SSP.”)

This is a very surprising discovery. The Millennium switch to a commercial middleware like WebSphereMQ was supposed to eliminate this queuing. Millennium support might attribute the problem to the SSPs consuming resources that could be freed up for other processing. Although technically true, the impact of the SSPs is relatively small. An SSP starts with less than 16 MBs of RAM and can grow to about 128 MB of RAM. If I ran 16 SSPs that all grew to 128 MB (they often do not grow that large), I would be using the same amount of memory as any one of the Java servers (SCP Entry ID 351 through 370). Since the SSPs do not handle client communication — they don’t talk with PowerChart, FirstNet, PowerChart Office, etc. — they consume very little CPU. These processes do some disk I/O, but compared to CPM Script, SCP Entry ID 51 or the Java servers, it is such a small amount of I/O that it is really just background noise. The resource consumption, therefore, is too small compared to the rest of the Millennium application executables to have any significant impact to the resources of the node, lpar or vpar.

I still recommend that all non-VMS users switch to MQSS as soon as you can. The throughput gains are impressive. I just ask you not to reduce the SSPs to four. Although I don’t know your specific configuration needs, I would suggest you start somewhere in the range of 8-20. Eight was the old minimum recommended value for SSPs in non-production domains, and I have always recommended 20 as a base for any production domain.

Although VMS clients can’t benefit from MQSS, I do urge you to upgrade your system to make it as stable as possible and to ensure that you are able to use all 32,768 ports. Clients on VMS version 8.3 should upgrade to the appropriate VMS 8.3 with TCP/IP Services version 5.6 ECO 5 or later.

Prognosis: Even with MQSS’s welcome increases in throughput, SSPs still have a role in Millennium. Keeping them properly tuned will help prevent queuing in Millennium's proprietary middleware.

A Shovelful of Refresh Reality

By David Hallberg

I was shoveling my driveway this past weekend with three 9-year-old helpers. They sort of shoveled but mainly just loved making snow piles and jumping on them. With high hopes, my son asked me if the three of them could shovel the neighbors’ driveways and get paid for it. Sure, I said, as I outlined the process: Neighbors would not pay him until they could see the concrete in the driveway; we needed to finish our own driveway first; working very hard with our two shovels, our driveway would take at least two more hours. In no time at all, my realistic estimate had shattered his dream of making lots of easy money.  

That scene reminded me of a time at a previous job when every week the technical team I was leading was blamed by the Millennium project manager for missed deadlines. To figure out the problem, we arranged a meeting between the application and technology teams. The meeting revealed a number of communication and procedural issues. Among them, the tech team stated that it only needed one day to refresh a domain. This estimate seemed really optimistic to me, so I dug a bit deeper into the steps required for the refresh. I discovered that this time frame was based on the following assumptions:

1.    The tech team would receive at least a seven-day notice that a domain refresh was needed. That would provide ample time for a full backup of production and for our off-site tape storage vendor to bring the tapes back without a rush charge.

2.    The tech team would have enough space in the non-production domains’ storage area network (SAN) to restore production.

3.    Once we had completed the database restore and cer_exe copy on the back end and run a scan for components or reinstall on Citrix, we would be done.

So if the application gave us a seven-day warning before the project needed a domain refreshed and if we had a full backup and if we had enough disk space on the SAN and if we had a clean reinstall or clean scan for components and if the tech team worked 24 hours a day seven days a week, the domain refresh could be done in one day. I still shake my head at the memory of this conversation. The reality was that few of these things ever happened and none of them ever happened all together.

1.    We never had seven days’ notice.

2.    On several occasions we paid for our off-site tapes to be brought back (yes, I took the beating for that cost).

3.    Only once did we have enough SAN disk space. Generally, we would have to stop the restore several times to add more.

4.    The scan for components rarely was completed without issue.

After bringing reality into the domain-refresh process, I modified the project estimate from one day for the tech team to hand over a domain to three to five business days. Did this stop the finger-pointing? Quite the opposite, the application team went crazy. They now blamed the tech team for not delivering domains. I tried to appease them: “Well, if you can tell me how many refreshes and the exact dates you need them, I can modify the estimate.” Oddly enough, no one — not even the project manager — could tell us the number of refreshes they needed, let alone the dates.

I find it ironic that my 9-year-old son had the same unrealistic optimism for completing a task as the 20- to 50-year-olds on my tech team. They also shared the same look of discouragement when I tempered their optimism with the facts. But they eventually reaped the benefits. At work, the project team accepted the new estimates, the tech team was able to put in relatively normal 10- to 12-hour days during a refresh, and we began meeting our deadlines. At home, my son jumped in the snow until he couldn’t feel his toes and then happily warmed up with a cup of cocoa.

Prognosis:  Work is no place for a snow job. A realistic assessment of procedures and the time required to accomplish them will keep teams working well together — and meeting project deadlines.

Purges Help Minimize Database Growth

By David Hallberg

Millennium® uses the Oracle database as its main repository of information, which includes data that you may only want to keep for a period of time. This period may be one day or 365 days, but what is important is to delete the data as soon as you can to keep your database as trim as you can.

Millennium has many purges to delete rows from application tables that are not designed for long-term storage. Generally, a 2007.18 client has 147 active purge templates. Without all of them running, you are going to experience a couple of side effects. The first is general database growth. You want to control growth where you can so that backups run as fast as possible and, if you have to recover the database, recovery times are shortened. A second side effect also relates to database size. Depending on the way the applications are written and the indexing on the tables being purged, if you get too large you can have stability and performance issues within the applications. Generally, the Open Engine and old MDI applications are the most susceptible.

I realize that my advice to configure all of the purges for your Millennium version goes against what many of you believe or have been told by consultants — that you only want purges for the applications you are using. Here’s why I disagree. The system experiences no degradation when it runs a purge for an unused application. If you try to purge a table that has no data in it, the purge script looks for qualifying rows and says, “No rows match," and then the process moves on to the next purge. There is no harm to the system and no performance impact to users. If, however, you decide at some point to implement that application, the purge will already be set up, saving a step in the activation process.

You should set up all of the purges to run at least once each day. After any Millennium upgrade, look for new purges to configure for optimal performance of Millennium and to make database recovery as rapid as possible.

To learn how to configure and run purges, click here.

Prognosis: Purging unused data daily helps keep the system as trim and spry as possible.

Finding Fault with Default Settings

By Mike Zimmerman

One of the critical management features of an enterprise system is configuration. Rather than settling for suggested default settings from the vendor, users need to configure their system based on goals specific to their organization.

First, let me define my terms. In this instance, I use the word configuration to refer to Millennium® system settings such as database connection strings, log levels, the number of brokers in a queuing system, etc. I am not referring to areas such as user preferences or reference data. 

Traditionally, configuration settings that allow for default settings often require a system administrator to define the value before the system can function properly. A newer trend in system management is to define an overall configuration profile for the system that can be used to drive more intelligent default values without input from an administrator. The profile is created from high-level organizational goals or priorities. For example, an administrator could create a profile with the following system priorities: 1) availability, 2) performance and 3) security. The configuration system would use these priorities to optimize the default value for such areas as where to use SSL connections or what log level defaults are most appropriate for a given layer in the stack.

Knowing your priorities is important because changing a configuration setting to optimize for one goal oftentimes can detract from another goal. For example, you may determine that you want the system to perform as fast as possible, but set a higher goal to never lose any data; therefore, the logging level should be set higher than it would be for a system that doesn’t prioritize data loss over performance.

Configuration profiles are gaining momentum in the industry, but probably won’t appear in Millennium for several years. Even so, I would recommend that you take time now to determine the priorities for your system. Is high availability the most important thing for you? Then review the potential choke points in the architecture and make sure you have backup and failover strategies for these layers. Is performance most important? Then make sure your log levels aren’t set to DEBUG.

In addition to defining an overall set of system priorities to drive your configuration settings, you need to define a change control process for all configuration updates. Proper procedure is vital because most configuration settings do not track history, making it impossible to undo a change or even track down who made the change. Keep in mind that configuration settings are stored in the Millennium directory, the system registry, .cerner files, .ini files, the Windows registry and in database tables.

Prognosis: Understanding your organization’s goals and priorities is the first step in determining optimum configuration settings for your Millennium system.