Millennium Medic

OnTrack Panther Supports Millennium 2012.x Code

By Chris Lanaman

David Hallberg’s recent blogs on preparing for your Millennium^® site’s upgrade to 2012.x code have coincided with several conversations with clients who use OnTrack™ Panther™ to manage and monitor system performance. They want to know: “Does Panther support Millennium 2012.x code?” And: “Will OnTrack audits still point out issues as I test the new code and then move it to prod?” I want to assure you that all Softek OnTrack products and services support this version of code.

For those of you not familiar with Softek, our software and services give you a high level of visibility into your EMR system along with actionable evaluations and alerts developed and serviced by a team who collectively had years of experience at Cerner and various Millennium hospitals.

As we enter a new year, we have the same great news we have had since Cerner developed Millennium version 7.6: Softek is working to provide your site an objective view into the performance and reliability of your Millennium system.

The following table lists the Millennium code versions supported by OnTrack Panther.

Millennium Version
7.6
7.7
7.8
2001.01
2001.x
2003.01
2003.02
2005.01
2005.02
2007.02
2007.03-2007.17
2007.18.x
2007.19.x
2010.01.x
2010.02.x
2012.01.x
Choose and Book
eBookings

Prognosis: Third-party support that gives your organization intuitive management of your Millennium EMR system can be the foundation for a smooth upgrade to 2012.x code.

Citrix Load Balancing in a Complex Environment

By David Hallberg

A recent client’s Millennium^® performance issues had me stumped. The organization had a normal two-application and two-database node configuration. It was using very little CPU and had virtually no paging on these nodes. Queuing was pretty good from an MQ and proprietary middleware perspective. There was a lot of interface queuing, but the organization was on an older version of Oracle and we were not going to be able to address this area of their queuing until a database rebuild. So how could I help them right then?

I turned my attention to the Citrix servers — they had more than 100 of them — to check the load balance. The backend gave me little to go on, so I had to look deeper to see how the Citrix servers were connecting with Millennium. In this CPOE shop, Discern Notification alerts clinicians to potential problems with their order deliveries, so I chose to evaluate the Citrix load balancing in this component of Millennium.

When I did, I found that the method of Citrix load balancing being used was causing inconsistent response times for clinicians. Fast application and database nodes could not fix this performance problem. Rather, the team had to use the Citrix load balancing tool on Application User Load to more evenly distribute the core applications of, for instance, PowerChart^® and FirstNet^®. The better application distribution resulted in more consistent performance for their clinicians.

If you want to investigate whether you have a load balancing issue between Citrix and the application nodes, I’ll take you through the steps to parse your data and then evaluate the results. Start with one of the following select statements (either from CCL or SQL*Plus):

            CCL> rdb select * from EKS_NOTIFY_DEST go

            SQLPLUS> select * from EKS_NOTIFY_DEST ;

Your output will look something like this (for space reasons, I have eliminated three columns on the right and reduced my example to two rows of data):

LOCATION	TCP_IP_ADDR	TCP_IP _PORT	PERSON _ID	UPDT_DT_TM	UPDT_ID	UPDT _TASK
54099	192.168.1.62!RLL9989: PROD:DEARMAN	2048	11199999	2011/11/15 01:54:54	11199999	3071000
1555555	192.168.1.61!DAVID. HALLBERG:PROD:DWH9999	2048	8999999	2011/11/04 07:33:07	8999999	3071000

The LOCATION column identifies the device location, set either on the fat client or via the WTS location tool. The TCP_IP_PORT is the port the client is listening on. The PERSON_ID is a unique number for the user name in Millennium. The UPDT_DT_TM shows the last time there was any communication between the application nodes and this client. UPDT_ID is a repeat of the PERSON_ID. UPDT_TASK is the task number for the process that stuck the row in the database or changed it. This number should be 3071000, which is the Discern Notification. Not shown here are UPDT_CNT; UPDT_APPLCTX, the application context number for the application updating this table; and APP_VERSION_NBR, the version of Discern Notification that the client is using. The version generally will be 2, unless you are on some very old code.

The TCP_IP_ADDR column is the key to making sense of this gibberish. It is composed of four elements: the TCP/IP address of the Citrix server or fat client connecting to Millennium, the Microsoft user name, the Millennium domain the connection is for, and the Millennium user name. The next step is to parse the TCP_IP_ADDR column into these four elements.

Import the table into Excel to use the Text to Columns function, and insert three columns between TCP_IP_ADDR and TCP_IP_PORT.

To parse the fat clients or older Citrix servers:

1. Sort the TCP_IP_ADDR column so any cells with a “$” separating the TCP/IP address (first element) from the Microsoft user name (second element) are at the top of the list. Highlight these cells only.
2. From the Excel Data tab, select the Text to Column function, select Delimited and click Next.
3. In the next box, uncheck Tab, select Other, type $ in the open cell and click Finish.

To parse the rest of the Citrix servers:

1. Highlight the other cells in the TCP_IP_ADDR column, which should all have a “!” between the TCP/IP address and Microsoft user name.
2. With the Text to Column function, select Delimited and click Next.
3. In the next box, uncheck Tab, select Other, type ! in the open cell and click Finish.

The first part of the parsing is complete. You now have a list of all the TCP/IP addresses that have connected to the application nodes for Discern Notification. Next you want to see all of the Microsoft user names, Millennium domains and Millennium user names that have connected to the application nodes for Discern Notification.

1. In row 1 column C, which should be blank, type “Microsoft Username:Millennium Domain:Millennium Username”.
2. Highlight all the rows in column C, which contains the remaining three data elements.
3. From the Excel Data tab, select the Text to Column function, select Delimited and click Next.
4. In the next box, uncheck Tab, select Other, type : in the open cell and click Finish.

The next step in getting usable information is to build a Pivot Table (steps are based on Excel 2010) to see the TCP/IP addresses that have the most Discern Notifications:

1. From the Insert tab, select Pivot Table and verify the entire spreadsheet is highlighted. The dialog box should read, “Table1.” Select OK.
2. From the Pivot Table Field List, left-click and drag the TCP_IP_ADDR field to the Row Labels box.
3. Left-click and drag the TCP_IP_ADDR field to the Values box. The Values box will then read, “Count of TCP….” The Table to the left will have two columns.
4. In cell A3, titled Row Labels, click the down arrow and select More Sort Options.
5. Select Descending (Z to A) by:, click the down arrow to change the value to Count of TCP_IP_ADDR and select OK.

The resulting table will list the Discern Notifications for each TCP/IP address, as shown in this example:

Row Labels	Count of TCP_IP_ADDR
192.168.1.213	469
192.168.1.55	426
192.168.1.54	67
192.168.1.57	48
192.168.1.137	43
192.168.1.125	36
192.168.1.79	35
192.168.1.122	34
192.168.1.132	31
192.168.1.107	30

If your servers are properly balanced, the count should be fairly similar across all the Citrix servers. If you have fat clients, they will have a smaller count. In my example, the server ending in .213 handles 15.63 times more connections than the .107 server. Even if we exclude the first two rows, the .54 server is doing 2.23 times more work than the .107 server. Once this imbalance is corrected, clinician response times will be more consistent.

Prognosis: A little research might uncover Citrix load balancing issues that are causing response time variability or other performance problems for your clinicians.

Preparing for Your 2012 Upgrade, Part 2

By David Hallberg

Today, in the second half of my recommendations for ensuring a smooth transition to Millennium version 2012.x, I’ll look at Windows and Cerner’s back-end application nodes. (To see my thoughts on Shared Service Proxy settings and WebSphere MQ, see Part 1.) I recommend you check your settings in the following areas of your 2012.x domain:

• Microsoft added the User Account Control to Windows to protect computers from running unwanted programs, but this feature can cause unwanted interruptions in running Millennium applications. If you are using Windows Server 2008 SP1 32-bit or Windows Server 2008 R2 with Virtual Machines (VMs) running Windows Server 2003 R2 SP2 or Server 2008 SP1 32-bit, make sure the User Account Control is disabled. Two documents explain how: “Turn User Account Control On or Off” from Windows and “User Account Control Step-by-Step Guide” from Microsoft TechNet.
• DNS devolution includes improved security in Windows Server 2008, Vista and Windows 7. Make sure you have it set. Documents to explain how are on uCern at “Windows Vista Minimal Configuration Guide, Install Windows 2008” or “Microsoft Windows 7 and Windows Vista Configuration.” Microsoft offers “Post Installation Behavior … after you install the DNSA update.”
• In the past six months, I have run across more and more sites that have incorrect settings for permissions and owners of files and directories. You can correct the settings by running the secure_cerner_500 executable located in the $cer_exe filesystem. Run it as root, and provide the correct inputs. One caution about this executable: If you are running more than one domain on the same server, lpar, npar or vpar, you have to remove the sticky bit on the files in $cer_fifo, $cer_lock and $cer_usock. If you only have one domain per server, lpar, npar or vpar, this will not be an issue.
• Defrag the $cer_config filesystem to improve throughput for the security and transaction database servers.
• Use the bcheck tool to rebuild the dic.idx file on each application node. If bcheck is not in your $cer_exe filesystem, look in $cer_ocd and find the file named bcheck.dat. Rename the file to bcheck, and put it in $cer_exe. You can search uCern or log a Service Request to get Cerner’s recommended flags to use on this. You might also be able to use the -? flag to get a list of the switches that can be used when you create the file.
• Finally, you want to check the dedicated Shared Service Proxies (SSPs) for your security servers to make sure they won’t be overloaded with connections. At a minimum, you need an SSP for each instance of the Security Slave processes. If you want to have the Security Master service on its own SSP, add one more instance than is needed for the Security Slaves. This means you will have an SSP on the application nodes with no connections, but that’s okay. The SSP only takes 3MB of memory. If you have 2 Security Slaves running with your Security Master and 3 Security Slaves on your other application nodes, you will probably have to increase to at least 5 Security Slaves running with the Security Master and 6 Security Slaves everywhere else. This means you’ll need at least 6 dedicated SSPs for the Security Slaves; if you want a dedicated SSP for the Security Master service, you’ll need 7 dedicated SSPs. Remember to start the dedicated SSPs prior to starting additional Security Slaves in order to get the desired load balancing. Your goal is to keep the maximum connections for each Security Slave under 300. You can check the maximum connections from the Server Control Panel. If you want the steps for how to do it, send me a comment, and I’ll send them to you.

Prognosis: If you prepare for conversion by looking at and correcting your infrastructure and key items in your 2012.x domain, you will have a much quieter upgrade.

Preparing for Your 2012 Upgrade, Part 1

By David Hallberg

Readers have been asking about the changes coming with Millennium^® version 2012.x. This major code upgrade is the first Millennium release that supports Linux as a database, and all Cerner clients are being encouraged to support it. When it hits prod early next year, it will contain a number of upgrades to help users file the reports needed for Meaningful Use reimbursements. Over my next two postings, I’ll share what I know about the release from a technical perspective.

Let me start with a warning. Do not follow Cerner’s recommendation to reduce the number of Shared Service Proxies (SCP Entry ID 1) to between 1 and 4. If you do, you will encounter several sporadic performance issues with Millennium that will be extremely difficult to track down. I recommend that you run 10 to 20 SSPs so that you have more consistent response times from all of the solutions. I realize these processes are not used for communicating with the clients or Citrix servers, but they are doing work — without taking up much space. Since each instance takes about 3.3MB of memory, 20 instances would consume only 66MB of memory, a trivial amount for today’s typical application node that has at least 32GB of memory.

The Millennium upgrade process begins by ensuring that your infrastructure is sound. The rest of today’s post will look at IBM WebSphere MQ, and my next post will make recommendations for Windows before moving on to Millennium’s back-end application nodes.

I recommend you upgrade to WebSphere MQ version 7. The links below will take you to IBM documents that give MQ benchmarks for various operating systems. You want to review the sections on Performance and Capacity Limits and Tuning Recommendation. Then choose the largest value between Millennium’s baseline documents and these documents.

AIX: “MP6N: WebSphere MQ for AIX V7.0 — Performance Evaluations”

HP-UX: “MP6O: WebSphere MQ for HP-UX V7.0 — Performance Evaluations”

Linux: “MPL5: WebSphere MQ for Linux (Intel) V7.0 — Performance Evaluations”

Before going live, make sure you correct any errors resulting from the MQ upgrade. Errors will be shown on the Lights On Network under the System Compliance tab. Look for Known Errors Non-Compliance rows that cover MQ.

If you decide to use MQ version 6, you can reference the following documents for your platform. Again, look at the Performance and Capacity Limits and Tuning Recommendation sections, and choose the largest value between Millennium’s baseline documents and these documents.

AIX: “MP6K: WebSphere MQ for AIX V6.0 — Performance Evaluations”

HP-UX: “MP6L: WebSphere MQ for HP-UX V6.0 — Performance Evaluations”

Linux: “MPL3: WebSphere MQ for Linux (Intel) V6.0 —Performance Evaluations”

Due to the increased number of ports being used by MQ on the front end, I suggest you review Microsoft Support article 196271 (previously published under Q196271). Based on Microsoft’s recommendation, I suggest increasing the MaxUserPort to 65534. You can use a tool called TCP Optimizer. Select the link to the new TCP Optimizer version 3.0.8. You want this executable to be run in XP SP2 mode, and you will have to be a local administrator for all the appropriate keys to be modified. Assuming you are running on Windows Server 2008, any version, here’s the steps to follow: Right-click on the downloaded executable, select Troubleshoot Compatibility, select Try Recommended Settings, select Start the program, slide the slider bar to 100+Mbps, select Modify All Network Adapters, select Optimal, select Apply Changes, and select OK. Finally, reboot your computer.

Keep an eye on the MQ statistics for Enqueue and Dequeue counts. If any Java services are in the top 30, you should consider splitting them. The split process would be similar to the process I described in the “Steps to Ensuring a First-Class Upgrade” blog (under step 2, Java Server Stress).

Prognosis: Doing a little research and establishing a solid infrastructure is the first step to ensuring a smooth conversion.

Next: Proper settings for Windows, permissions and ownerships, security slaves and more.

An Abnormal Definition of Normal

By David Hallberg

Thanks to all of you who stopped by last week during the Cerner Health Conference. We had some great talks about your Millennium^® performance expectations, although I was discouraged to find that for a number of you your expectations were exceedingly low. For example:

• Some of you claimed, “You cannot fix the performance of the application.”
• Others told me, “We are remote hosted and have been assured that we get the best performance that anyone can. When the applications are slow, that’s just the way it is.”
• One client showed me their Lights On Network data for the number of transactions that took over 9 seconds, and I almost fell off my chair. Though the percentage was low, the actual number of transactions was enormous. “That’s way too many,” I said. “It’s the best we can get,” he replied.
• Still another client was told they needed to spend more than $1 million in hardware upgrades to fix their problem.

If you’ve been coaxed into believing any of these statements, let me tell you what I told them: Slow performance is not normal. OK performance is not OK. Don’t believe that you are the fastest Millennium client and there is nothing more for you to do. Your clinicians should not have to wait for 9 seconds for a transaction to go through. Many problems can be fixed without any additional hardware. I know. I have seen Millennium perform extremely well over the past 14 years.

When you are equipped with just a little knowledge of where you have bottlenecks in your system and take the appropriate corrective action, you can see immediate improvements. These improvements do not need to take years and do not require a new version of code, a new operating system or new version of Oracle. You can improve what you have. In the case of the client told to spend $1 million on hardware, we were able to work with the existing hardware and make a couple of uptime changes. Lights On showed that transaction speed for the clinicians had increased and variability had decreased.

Readers of this blog know that I regularly encourage you to trust your clinicians and believe your own eyes rather than blindly accepting performance excuses from the company that wrote the software and is now managing the environment. If the responsiveness of the system is slow, you should be able to see the first level of issues immediately. In a matter of hours or a few days — not months or years — you can determine the initial action steps.

I need to emphasize the words “initial steps.” There is no single silver bullet that will slay your performance dragon; you’ll need a magazine full of lead bullets. The beast is immune to silver but is susceptible to a lead overdose, to an ongoing maintenance regimen that accounts for your continually evolving EMR system. Every new package or service pack you install has the potential to change the processing of the entire system. Make sure you have the right people using the right tools to evaluate what is happening, to identify performance issues and to take action to correct the problems.

Prognosis: If your current support structure is not helping you help your clinicians deliver care more effectively, it’s time to find one that will.

X Marks Our Spot to Meet with You at CHC11

By Chris Lanaman

 
If you’re one of the chosen few from your hospital coming to Kansas City for the Cerner Health Conference, you probably have a long to-do list: speakers, education sessions, exhibits, networking, a party or two. But I’m hoping you’ll strike out on your own at some point to visit with the team that’s been bringing you the Millennium Medic the past several years. Lucky Strike Lanes & Lounge just a few blocks east of the CHC activities will be our spot throughout the conference. Why do we think you’ll want to add us to your already busy agenda?

• We’re having a party, and you’re invited — Monday, Oct. 10^th, from 7 p.m. to midnight at Lucky Strike, 1370 Grand Blvd. We might be a bit biased, but we think it’ll be one of the best of the week. It’s Softek’s annual Client Appreciation Party, but we’re welcoming non-clients as well for an evening of food, games, music, bowling and lots of prizes (we’re talking cool electronics). Your CHC badge is your ticket to get in, or you can sign up ahead of time and get more details at the Softek website.
• Our Millennium experts are sharing their expertise. Where this blog is best used to address common system issues, a face-to-face meeting would let us answer your specific questions. Stop by Lucky Strike any time from Monday morning to Wednesday noon — we’ll even be there Tuesday night overlooking the Big Bash in the Power & Light District. Come talk with us and enjoy a free meal and bowling. If you prefer to set up an appointment, call Erin at 913.981.5242.
• We know more about your system than you might think. Softek conducts independent audits of the Millennium production environment, and we can show you what we’ve discovered in every audit we’ve ever run, findings that have huge clinical and financial implications.
• We’ve got our head in the clouds. Softek OnTrack’s new cloud access is part of our commitment to opening access to Millennium users to more information about their system. We’d love to tell you about it: Millennium whitepapers and user training at OnTrack University, user forums, mobile applications for system management and other resources. 

The CHC will be a busy week for education, association and recreation. We hope to enjoy all three with you.

Going to Kansas City for the Cerner Health Conference?

Many of you will be in Kansas City in a few weeks for the Cerner Health Conference. We’ll be on hand as well from our “headquarters” at Lucky Strike Lanes & Lounge in the Kansas City Power & Light District a few blocks east of Bartle Hall and the CHC. We’d love to talk with you in person about your Millennium® system and will have more details in the next blog concerning our activities for the week.

Today, however, we want you to get excited about the city we call home. Here’s a list of 10 places you might not have thought to visit while you’re here…but are definitely worth considering. Our potpourri includes places that appeal to all sorts of interests: history, art, architecture, sports, music, barbeque (you are in KC, after all) and even potpourri.

1. The story of the sinking of the steamboat Arabia in the Missouri River in 1856 is just as fascinating as its discovery in 1987 buried 45-feet deep in a cornfield half a mile from the river. The Arabia Steamboat Museum in the River Market documents it all and displays the china, food, guns and clothing preserved in the silt. Outside the museum, explore the Farmer’s Market and stroll the Riverfront Heritage Trail to Berkley Riverfront Park.
2. The Money Museum inside the Federal Reserve Bank of Kansas City gives you a view of the automated vault and sends you home with a bag of cash (sadly, it’s shredded). Across the street from the Fed, you can appreciate the irony in the 65-foot-high sculpture of cargo containers at Memorial Park. Sculptor John Salvest’s installation spells out its title: “IOU/USA.”
3. The National World War I Museum is housed in the Liberty Memorial across the lawn from the bank. Its 55,000 items include everything from common soldier gear to rare artifacts. The “Man & Machine” exhibit shows German technology never before displayed in this country.
4. The National Archives west of the Liberty Memorial takes you back to an earlier war. The “Lee and Grant” exhibit contains artifacts and writings of two great Civil War generals.
5. The newly opened Kauffman Center for the Performing Arts south of Bartle Hall owes its impressive front facade to the 27 cables supporting the glass and steel structure. Tours are available, and multimedia artist Laurie Anderson performs “Transitory Life” at 7 p.m. on Oct. 9. For another KC architectural gem, visit the Bloch Building at the Nelson-Atkins Museum of Art. View 40 bronze sculptures by Auguste Rodin (of "The Thinker" fame) throughout the lobby.
6. Local entrepreneur Mary Carol Garrity, owner of the Nell Hill’s retail home stores, will speak at the downtown Kansas City Public Library at 6:30 p.m. on Oct. 11. Also at the library is an exhibit of original illustrations by Missourian Thomas Hart Benton for some of Mark Twain’s signature works.
7. KC’s most famous hometown brewery, the Boulevard Brewing Co., gives free tours (including tastings). Reservations are required. If you prefer the taste of coffee, visit the Roasterie Kansas City down the road from the brewery.
8. Celebrate KC’s rich music and food tradition at the annual Rhythm and Ribs festival Oct. 8 at the American Jazz Museum in KC's 18^th and Vine district. The daylong festival features top jazz and blues performers and local food vendors. Also at 18^th and Vine is the Negro Leagues Baseball Museum.
9. If you prefer a faster pace, visit the College Basketball Experience inside the Sprint Center. Besides learning all you want about the game, you’ll find lots of areas for play.
10. It’s race weekend at the Kansas Speedway just a short interstate drive west of the metro area. Oct. 8 is the NASCAR Nationwide Series’ Kansas Lottery 300, and Oct. 9 is the NASCAR Sprint Cup Series’ Hollywood Casino 400. Next to the track is the Legends outlet mall, several major retailers, land for the soon-to-be-built Cerner office buildings and the Livestrong Sporting Park. Major League Soccer’s Sporting KC plays Chivas Guadalajara at Livestrong on Oct. 12. 

You might be coming to the CHC to learn and spend time with colleagues, but we hope you also take time to enjoy a few of the places and activities unique to KC.