Millennium Medic

Preparing for Your 2012 Upgrade, Part 2

By David Hallberg

Today, in the second half of my recommendations for ensuring a smooth transition to Millennium version 2012.x, I’ll look at Windows and Cerner’s back-end application nodes. (To see my thoughts on Shared Service Proxy settings and WebSphere MQ, see Part 1.) I recommend you check your settings in the following areas of your 2012.x domain:

• Microsoft added the User Account Control to Windows to protect computers from running unwanted programs, but this feature can cause unwanted interruptions in running Millennium applications. If you are using Windows Server 2008 SP1 32-bit or Windows Server 2008 R2 with Virtual Machines (VMs) running Windows Server 2003 R2 SP2 or Server 2008 SP1 32-bit, make sure the User Account Control is disabled. Two documents explain how: “Turn User Account Control On or Off” from Windows and “User Account Control Step-by-Step Guide” from Microsoft TechNet.
• DNS devolution includes improved security in Windows Server 2008, Vista and Windows 7. Make sure you have it set. Documents to explain how are on uCern at “Windows Vista Minimal Configuration Guide, Install Windows 2008” or “Microsoft Windows 7 and Windows Vista Configuration.” Microsoft offers “Post Installation Behavior … after you install the DNSA update.”
• In the past six months, I have run across more and more sites that have incorrect settings for permissions and owners of files and directories. You can correct the settings by running the secure_cerner_500 executable located in the $cer_exe filesystem. Run it as root, and provide the correct inputs. One caution about this executable: If you are running more than one domain on the same server, lpar, npar or vpar, you have to remove the sticky bit on the files in $cer_fifo, $cer_lock and $cer_usock. If you only have one domain per server, lpar, npar or vpar, this will not be an issue.
• Defrag the $cer_config filesystem to improve throughput for the security and transaction database servers.
• Use the bcheck tool to rebuild the dic.idx file on each application node. If bcheck is not in your $cer_exe filesystem, look in $cer_ocd and find the file named bcheck.dat. Rename the file to bcheck, and put it in $cer_exe. You can search uCern or log a Service Request to get Cerner’s recommended flags to use on this. You might also be able to use the -? flag to get a list of the switches that can be used when you create the file.
• Finally, you want to check the dedicated Shared Service Proxies (SSPs) for your security servers to make sure they won’t be overloaded with connections. At a minimum, you need an SSP for each instance of the Security Slave processes. If you want to have the Security Master service on its own SSP, add one more instance than is needed for the Security Slaves. This means you will have an SSP on the application nodes with no connections, but that’s okay. The SSP only takes 3MB of memory. If you have 2 Security Slaves running with your Security Master and 3 Security Slaves on your other application nodes, you will probably have to increase to at least 5 Security Slaves running with the Security Master and 6 Security Slaves everywhere else. This means you’ll need at least 6 dedicated SSPs for the Security Slaves; if you want a dedicated SSP for the Security Master service, you’ll need 7 dedicated SSPs. Remember to start the dedicated SSPs prior to starting additional Security Slaves in order to get the desired load balancing. Your goal is to keep the maximum connections for each Security Slave under 300. You can check the maximum connections from the Server Control Panel. If you want the steps for how to do it, send me a comment, and I’ll send them to you.

Prognosis: If you prepare for conversion by looking at and correcting your infrastructure and key items in your 2012.x domain, you will have a much quieter upgrade.

Preparing for Your 2012 Upgrade, Part 1

By David Hallberg

Readers have been asking about the changes coming with Millennium^® version 2012.x. This major code upgrade is the first Millennium release that supports Linux as a database, and all Cerner clients are being encouraged to support it. When it hits prod early next year, it will contain a number of upgrades to help users file the reports needed for Meaningful Use reimbursements. Over my next two postings, I’ll share what I know about the release from a technical perspective.

Let me start with a warning. Do not follow Cerner’s recommendation to reduce the number of Shared Service Proxies (SCP Entry ID 1) to between 1 and 4. If you do, you will encounter several sporadic performance issues with Millennium that will be extremely difficult to track down. I recommend that you run 10 to 20 SSPs so that you have more consistent response times from all of the solutions. I realize these processes are not used for communicating with the clients or Citrix servers, but they are doing work — without taking up much space. Since each instance takes about 3.3MB of memory, 20 instances would consume only 66MB of memory, a trivial amount for today’s typical application node that has at least 32GB of memory.

The Millennium upgrade process begins by ensuring that your infrastructure is sound. The rest of today’s post will look at IBM WebSphere MQ, and my next post will make recommendations for Windows before moving on to Millennium’s back-end application nodes.

I recommend you upgrade to WebSphere MQ version 7. The links below will take you to IBM documents that give MQ benchmarks for various operating systems. You want to review the sections on Performance and Capacity Limits and Tuning Recommendation. Then choose the largest value between Millennium’s baseline documents and these documents.

AIX: “MP6N: WebSphere MQ for AIX V7.0 — Performance Evaluations”

HP-UX: “MP6O: WebSphere MQ for HP-UX V7.0 — Performance Evaluations”

Linux: “MPL5: WebSphere MQ for Linux (Intel) V7.0 — Performance Evaluations”

Before going live, make sure you correct any errors resulting from the MQ upgrade. Errors will be shown on the Lights On Network under the System Compliance tab. Look for Known Errors Non-Compliance rows that cover MQ.

If you decide to use MQ version 6, you can reference the following documents for your platform. Again, look at the Performance and Capacity Limits and Tuning Recommendation sections, and choose the largest value between Millennium’s baseline documents and these documents.

AIX: “MP6K: WebSphere MQ for AIX V6.0 — Performance Evaluations”

HP-UX: “MP6L: WebSphere MQ for HP-UX V6.0 — Performance Evaluations”

Linux: “MPL3: WebSphere MQ for Linux (Intel) V6.0 —Performance Evaluations”

Due to the increased number of ports being used by MQ on the front end, I suggest you review Microsoft Support article 196271 (previously published under Q196271). Based on Microsoft’s recommendation, I suggest increasing the MaxUserPort to 65534. You can use a tool called TCP Optimizer. Select the link to the new TCP Optimizer version 3.0.8. You want this executable to be run in XP SP2 mode, and you will have to be a local administrator for all the appropriate keys to be modified. Assuming you are running on Windows Server 2008, any version, here’s the steps to follow: Right-click on the downloaded executable, select Troubleshoot Compatibility, select Try Recommended Settings, select Start the program, slide the slider bar to 100+Mbps, select Modify All Network Adapters, select Optimal, select Apply Changes, and select OK. Finally, reboot your computer.

Keep an eye on the MQ statistics for Enqueue and Dequeue counts. If any Java services are in the top 30, you should consider splitting them. The split process would be similar to the process I described in the “Steps to Ensuring a First-Class Upgrade” blog (under step 2, Java Server Stress).

Prognosis: Doing a little research and establishing a solid infrastructure is the first step to ensuring a smooth conversion.

Next: Proper settings for Windows, permissions and ownerships, security slaves and more.