Case Study
Akron Children's Hospital Enhances Digital Experience
The Problem
At the time of this publishing, hospital operations included 359 beds, 900 Cisco WLAN access points (APs), with over 800 Wi-Fi connected clients running critical applications such as IV-pumps, VoIP voice calls, Computers on Wheels, laptops, tablets and PDAs, access to electronic medical records and billing systems. Not to mention guest access from patients and visitors. The number of wireless devices used for medical and patient care purposes was expected to triple within two years and continue growing rapidly in the future.
The hospital’s WLAN was not considered reliable for hospital communications, because there was frequent degradation of quality and connectivity. “Our goal is to make wireless serve medical applications at all times. This is a special challenge in an IT network using lightly-regulated WLAN frequencies where your network operation is continuously impacted by the number of devices and traffic patterns in hospital and guest networks; different device types and driver versions, and WLAN configuration changes” recounted Russ Johnson, Network Manager at Akron Children’s Hospital. “If you don’t know how service is delivered to end users, you don’t know enough. We did not have this visibility and were suffering.”
The Solution
7SIGNAL’s Wireless Network Monitoring platform was initially deployed in the Neonatal Intensive Care Unit (NICU), the Emergency Department (ED) and one remote clinical facility. This gave hospital IT staff the ability to fully understand how the WLAN network was serving their needs and meeting their SLA’s from the user’s perspective. With this new visibility, IT now had the means to improve the hospital network to better meet medical grade wireless requirements, and support mission critical applications and wireless medical devices.
After three days, initial analysis of the data, revealed the following performance issues:
• Low success rates for end user association
• Long end-to-end delays (round trip times)
• High retransmission rates and high packet loss
• Significant drops in voice quality during the evening
• Low throughput during the day at the remote clinic
Seeing the first round of quantified performance deficiencies, Brian Kuner, Director of Network Infrastructure, said “This data correlates well with our experiences. Based on feedback from our end users, we felt our network had these issues, but had no way to quantify and pinpoint them, until now.”
After a detailed analysis of the data, the following were identified as the main reasons for the degraded performance observed:
• Excessive RF reflections and out-of-target-area propagation, due to antenna type and position
• Co-channel interference caused by non-optimal channel and AP power plan
• High percentage of air-time wasted by control traffic due to non-optimal radio parameters
• LAN/WAN routing differences between sub networks (SSID’s)
• Lack of link capacity in one layer 2 core router
• Incorrectly configured router at a remote site
"Based on our feedback... we felt our network had these issues, but had no way to pinpoint or quantify them, until now."
The Results
Russ Johnson and his team from Akron Children’s Hospital implemented corrective actions on these areas to improve network performance and end user experience.
The initial optimization phase delivered the following impact:
• Throughput increased by up to 110%
• Packet losses reduced by 80%
• Retransmissions reduced over 50%
• End-to-end network delays reduced by 35%
• Client’s ability to receive access point beacons increased by 20%
After Sapphire was deployed in the NICU, ED and remote clinical locations, and with the upcoming EPIC electronic medical record deployment, Tom Ogg, CIO determined that the improved network performance implemented in the three areas of the hospital were still insufficient for the planned rapid growth in device count, new applications and traffic volume growth. The decision was made to install Sapphire in all critical clinical areas in the hospital and five remote medical facilities, and upgrade their wireless network to support 802.11n.
The upgraded 7SIGNAL configuration at Akron Children’s included 60 Sapphire Eye sensors installed prior to the 802.11n network upgrade. The hospital IT team wanted to minimize risks during the network upgrade process and be able to start optimizing the new network immediately.
They had expected a marked performance improvement with the 802.11n upgrade to Cisco AP 1262’s, and while performance of some SSID’s did improve as expected, performance actually degraded in some cases. This was attributed to multiple issues. Rapidly growing retransmission rates and wide use of low data rates codecs were identified from the data produced by 7SIGNAL.
Over two months following the 802.11n upgrade, 600 new “Computers on Wheels” equipped with HP laptops which used Intel 6025 Wi-Fi chipsets were brought online on the main hospital patient care network which consisted of about 500 Cisco 1262 access points. As a result, retransmissions increased rapidly, degrading network performance and lowering available capacity by approximately 40%. This Sapphire Eye chart shows what occurred as the “Computers on Wheels” were deployed.
Using the data provided by Sapphire Eye, 7SIGNAL engineers along with HP technical support, Intel TAC, Intel WLAN specialists, and the Akron Children’s Hospital IT team, identified three main causes of the network issues. They included:
• Intel 6205 Wi-Fi chipsets generated unnecessary “QoS null data frames”, reducing capacity
• Cisco AP 1262 rate control moved to higher data rate too rapidly, causing retransmission rates to increase significantly
• “Computers on Wheels” carts significantly attenuated the RF signals
HP, Intel, and Cisco were not able to offer improvements within the required timeframe. T ogether, 7signal and the Akron Children’s Hospital team, based on the Sapphire network analytics, determined that replacing laptop internal radios with external Wi-Fi radios based on a different chipset would eliminate the first two issues. This was confirmed by first implementing changes to a smaller test area, then to an entire floor, and continuously validating the impact from Sapphire data throughout the entire WLAN coverage area.
Hospital wireless networks with 1000’s of access points are very complex. Operating medical devices and patient care equipment in such an environment without sufficient data for decision making, causes network management to be very difficult, and may carry significant risk to patient care.
Tom Ogg, CIO said, “I definitely did not have an easy feeling when I saw the performance dashboard and data of rapidly increasing retransmission in the network, as we were close to going live with EPIC”. The Sapphire data enabled the IT team to assess risk of insufficient WLAN capacity during the EPIC roll out phase. Based on live data, they determined the network was still able to carry the necessary load, so decision was made to proceed with the roll out. At the same time, they crafted a plan to neutralize “Computer-on-Wheels” issues and move forward.
“Sapphire not only helped us to recognize issues that could have jeopardized our EPIC roll out, it enabled us to validate corrective actions before end users were impacted. This was a perfect example of risk management, change control and fact-based decision making which are crucial when deploying Wi-Fi for mission-critical medical applications.” noted Tom Ogg.
"Based on our feedback... we felt our network had these issues, but had no way to pinpoint or quantify them, until now."