Blog
Have you ever studied card magic and sleight of hand? If you have, you’ll quickly learn it has a lot to do with angles and speed. When an “angle sensitive” trick is performed well, you are left scratching your head in bewilderment. But when viewed from a different angle you can see how was done.
Unfortunately, Wi-Fi troubleshooting is a lot like watching card magic. That’s because most Wi-Fi network monitoring software and troubleshooting tools only give you a narrow field of view, which obscures what’s really happening.
In this 2-part blog, we examine 9 “angle-sensitive” Wi-Fi and network problems that our customers were able to isolate and resolve thanks to the 360-degree view provided by the 7SIGNAL Wireless Network Monitoring platform.
Hospital discovers “silent” 5 GHz radio bug on Cisco 3500 and 3700 series APs
These days Wi-Fi is so pervasive in hospitals that it is no longer regarded as an extension to the infrastructure, it IS the infrastructure! That means anything short of five nines availability should set alarm bells ringing because, depending on location, it could be a patient safety issue.
In recent years a plethora of new patient tracking and monitoring devices has proliferated in hospitals – virtually anything you can hook up to a patient is now wireless. Combine that with a massive surge in deployment of workstations on wheels, and you have a serious problem on your hands if an AP goes off line, and no one even knows about it.
This was the case at several facilities of a hospital network based in California which is among the top ten largest health systems in the US. Thanks to 7SIGNAL, they could see the 5 GHz radio on certain Cisco 3500 and 3700 series APs was going to sleep and not accepting client connections, essentially creating a coverage hole. Yet, from Cisco’s management console the AP was “online” and everything appeared normal.
Consequently, clients entering the affected area were unable to successfully roam to the SSID served by the silent 5 GHz radio. The clients would remain connected to a more distant radio at a much lower data rate. But since they were still getting beacons from the nearby SSID, they repeatedly tried to roam in vain, to the unresponsive radio. This activity wasted client resources and increased airtime utilization, slowing down the affected clients even more. The only solution was to power off the AP and reboot it.
Even if those APs had been serving the same SSID on the 2.4 GHz band, it would still a problem. In such a case a client switching bands would naturally drop its network connection. Most EMR applications and wireless medical devices are latency sensitive, and cannot maintain session after a dropped connection, resulting in session data loss and needing to login again. This is one reason why it is generally preferred to not serve the same SSID on both bands in mission critical environments such as hospitals.
NBA stadium detects misconfigured 80 MHz channel on game day
Sports stadiums and arenas are some of the most difficult venues to provide good Wi-Fi services in. The thick concrete walls, large open space and seating play havoc with signals. But that’s not all, human bodies can make a big difference too. So the RF characteristics of an empty stadium and a full one, are completely different. Thank goodness for 5 GHz, for without it, serving 15 to 20 thousand phone-toting basketball fans would be impossible. Even so, it is very difficult to do well.
Obviously a well-designed channel plan in a stadium is crucial, and it is quite sensitive to small changes. The knack is to strike the right balance between the different extremes in attendance level. And it is not something you want to tinker with once it is stable, unless you have the tools to accurately measure the impact of your changes, both before and after conditions change from one extreme to another. Good performance management tools can make a big difference in striking the right balance.
Many stadium networks are operated by Service Providers. As is the case in this incident at a famous NBA stadium just hours before an NBA Playoff game. A network engineer accidentally configured the 5 GHz radio on an access point to use an 80 MHz wide channel, instead of a standard 20 MHz channel, with devastating consequences for client performance on that access point and those around it. As you can see from the chart below showing channel utilization going through the roof at 6:00 pm when the configuration mistake was made.
Even in a highly compartmentalized office setting the use cases for 80 MHz are few and far between, because 80 MHz is pretty-much useless for client access, since most mobile devices cannot do channel bonding above 40 MHz anyway. In a high density, open stadium where the goal is to minimize cell size and maximize spectrum reuse, 80 MHz is a kiss of death for client capacity and performance in the 80 MHz cell and all the adjacent cells.
Luckily this misconfiguration was identified by network engineers when the 7SIGNAL system generated alerts due to excessive channel utilization. This was then verified from access point info elements and packet capture. And the erroneous configuration was fixed in the nick of time just before the game started, avoiding embarrassment and irate fans.
New York media giant detects and eliminates bluetooth interference in offices
A leading New York financial news and information service found they were getting poor Wi-Fi performance in their offices on the 2.4 GHz band. The cause turned out to be interference from a large number of classic Bluetooth and BLE headsets, mice and keyboards.
Everyone knows the 2.4 GHz band is crowded, and adding lots of Bluetooth devices doesn’t help. The problem is classic Bluetooth devices frequency-hop across the whole band, so it is impossible to avoid them. BLE coexistence with Wi-Fi is better, but not completely interference free. BLE radios uses less energy, go to sleep when not connected, and transmit advertisements only in Wi-Fi’s guard bands located at 2402Mhz, 2426Mhz and 2480Mhz (shown in orange in the image below), which Wi-Fi does not use to transmit data.
So, when BLE and Bluetooth devices are in close proximity to Wi-Fi and the Bluetooth signal is less than 10 dB lower than the Wi-Fi signal then interference can be expected. In reality that means when you have Bluetooth / BLE and Wi-Fi devices side by side at the edge of a Wi-Fi cell, Wi-Fi performance will likely be degraded.
Actually, this is quite a common issue in open-plan offices. But most WLAN teams simply ignore it, because they have no practical way to measure its impact on the Wi-Fi experience. With 7SIGNAL, it is easy to see the effect of Bluetooth on the 2.4 GHz band, raising the noise level and increasing channel utilization across the entire 2.4 GHz band.
To see the difference, they disabled approximately half of the devices Bluetooth devices in one area and immediately saw ~300% improved download throughput and ~100% lift in upload performance. Some will rightly argue, “so what? We prefer to use 5 GHz anyway” and that is mostly true, however very few enterprises have turned off 2.4 GHz completely.
Large UK-based media and telecom conglomerate finds cellular booster blocking Wi-Fi
With so many people using their cellphone for business even if they are not sales execs… businesses can no longer shrug their shoulders and pretend poor cellular coverage on the premises is not their problem. It IS their problem. So what can businesses do to fix it? There are two basic approaches:
Without getting into the pros and cons of DAS (distributed antenna systems) over RANs (radio area networks), the first approach involves installing some kind of antenna system to distribute a local cellular signal inside the building, in conjunction with a gateway that connects back to the MSO over a broadband connection - essentially it’s like having a private cell tower on the premises, with its own backhaul to the MSO. Depending on the size of the building and footprint required, we might be talking about a network of antennas… much like a network if Wi-Fi access points, or just a single node (A.K.A. a Pico cell or Femto cell).
A simpler approach involves using one or more cell signal boosters. These work in a very different way. Instead of having a backchannel to the MSO, a signal booster simply amplifies the weak signal so local devices can hear it. It can do this because it has better receive antennas than those in a typical cell phone. Likewise, it amplifies the signal received from local phones and retransmits to the cell tower.
This is all good, except that some solutions utilize the 5 GHz U-NII frequency bands for internal system communications between system components. Since this overlaps with Wi-Fi, it can impact Wi-Fi performance. This may come as a surprise to many readers, because there is a common myth that the 5GHz band is immune to interference from non-Wi-Fi devices, because nothing else uses these frequencies. Not so.
In this strange case, the customer received an automated alert from the 7SIGNAL Wireless Network Monitoring platform triggered by the Beacon Availability KPI falling from 100% to zero. This indicated that access point beacons were no longer being detected. The consequence of this is that nearby clients could not detect the presence of the affected APs, and therefore has nowhere to roam to. Notably, the WLAN Controller that APs were connected was oblivious of any problem.
Looking at the problem from another angle, one can clearly see a dramatic increase in the noise floor on certain channels, this suggests that something was blasting the spectrum across so many channels that not even beacons could cut through the noise.
Again, so-called auto-tuning radio resource management features enabled on all the customer’s APs, turned out to be useless. Instead of detecting the problem, and switching the radio to a different channel, they failed to kick-in and switch the radio away from the noisy channels.
7SIGNAL sensors use an array of six 60-degree directional antennas with spectrum analysis. This enables easy correlation between signal detection and the actual location of the interference source. Seeing as how the customers WLAN equipment failed on two counts to detect and mitigate the problem, without 7SIGNAL they might have spent many hours dealing with support calls and groping around with spectrum analyzers to detect and isolate the problem. Instead, 7SIGNAL identified a problem automatically and informed engineers precisely where to look for the culprit.
You can read a more detailed account here: https://7signal.com/indoor-cell-phone-signal-booster-destroys-5ghz-band/
In part 2 of this blog we’ll look at five network infrastructure issues exposed by 7SIGNAL