Vodafone HQ Visit: Network management, handset testing and a look to the future

It’s been a long time since I last visited Vodafone’s Newbury HQ to talk about the introduction of GSM, and how mobile data might one day become a thing.

We’re talking about a time when the company worked out of multiple offices dotted around town.

Now the new(ish) campus has transformed the way the company operates. Not even the CEO has his own office here, with all staff encouraged to hot desk and collaborate in small teams.

I spent a day there to see what makes the network tick.

With 5,500 staff and just 3,500 desks, staff are actively encouraged to move around, work from home, and book meeting rooms sparingly. There’s no chance of falling asleep in a meeting you feel will never end, because the next people wanting the room can come in and kick you out.

There’s a distinct lack of power sockets to stop people getting too attached to whatever desk they grab, and at the end of the day everything needs to be left empty, or it will be removed by the cleaners.

Many of the desks I saw were salvaged from previous offices. According to Vodafone, there was no need to spend money on new desks and more need to spend it on comfortable, ergonomic, seats instead.

All staff, whatever their position, are given the same phone and laptop computer. No favouritism here, as everyone has a valuable job to do.

However, my visit wasn’t just to see how people can enjoy a working environment that resembles taking a table in a nearby Starbucks to become your office for the day (oh, they have a Starbucks too), but rather to see how the company manages its network when things go wrong, approves new handsets before release, and supports devices already on sale.

Plus a chance to talk a bit about what the future holds for Vodafone’s network.

And if all that wasn’t enough, I also got a glimpse at some impressive new products being showcased by Vodafone Xone, a new place to test out new ideas with a limited number of customers before the products go into full-scale production.

I’ll be writing about that in a future feature in the coming days.

Network Operations Centre

Like many of the things I saw on my trip, taking photos was often restricted.

None more so than when inside the Network Operations Centre (NOC), the only building where most Vodafone staff members are not allowed access.

To reflect their ‘mission critical’ status, all staff working inside the NOC wear a unique black uniform, rather than the more casual and laid-back attire for other staff.

You can probably imagine what a control centre looks like though.

As a guide, think of a large open room, with lots of big monitors on walls, and people sitting in front of a number of screens.

Something like this, almost…

Most pleasing was hearing a Star Trek style siren go off when something needed attention. Yes, seriously, they have that.

After the tragic events in Paris recently, it was here that Vodafone was able to make it possible for customers roaming in France to make calls and use data for free to contact concerned friends and family.

The weather is also monitored here, as storms and high winds (and associated power failures) are a major problem.

Operatives also check CCTV at numerous sites to look out for theft and vandalism that could also knock out service.

Various alerts will flag up seemingly trivial incidents, like for example the failure of a single air conditioner (there are two, for dual redundancy) at sites that won’t immediately affect the customer, but still requires urgent attention so as not to become a problem when the second one dies.

It’s in the NOC where plans are made on providing enhanced coverage for large events, such as Glastonbury, or making sure the network can cope with increased demand at fixed venues, like sporting grounds.

The NOC operates 24/7 and has to deal with predictable spikes, like the still not insignificant number of texts sent just after midnight at New Year.

All Vodafone users travelling abroad are also monitored here.

Integration is Key

The other big task, for a network currently spending around £2 million a day on upgrading its network, is making sure that as new sites go live, they don’t cause localised issues.

Sometimes capacity is held back to allow a site to cope with increased usage on a temporary basis, such as when a nearby site may go down due to a fault, or through routine maintenance.

Vodafone’s huge network upgrade work is being carried out jointly with Telefonica (O2) where the mobile radio side is being shared (backhaul, billing etc is still run independently), and coverage can be enhanced by sharing sites where necessary.

Given the many planning issues that, at worst, can prevent new sites being constructed at all, or at best can delay them by as long as 18 months, network sharing makes sense and mimics the same approach employed by EE and Three.

Currently, Vodafone has been concentrating its upgrade efforts in the west, while Telefonica has been doing the east. In London, there’s a more complicated split, and the use of lots of different kit that needs to be tested thoroughly (more on that later on).

The War Room

While not situated in a bunker, to one side of the NOC building is a room known as the war room, where people can potentially spend days (working shifts) dealing with major events as they unfold, and making vital decisions – all of which is recorded and sent to the relevant regulators for any future investigations or inquiries.

On the day of my visit, the war room was in use by people seeking to make sure the network was able to cope with increased demand on ‘Red Thursday’, which started one day before Black Friday, where the company had further deals on PAYG phones, contract phones and special tariffs created just for the weekend.

In the past, it has been used to manage somewhat more serious issues like having to deal with being ordered to shut down its operations in Egypt, under orders from the Government there during the anti-(Hosni) Mubarak protests in 2011.

At other times, you can expect to see the room filled with people discussing how to get parts of the network back up and running after a major storm, or heavy flooding.

Even things like managing new activations on the network are as important as keeping the network running, like being ready to cope with a large number of activations at Christmas, including people activating their prepay accounts on Christmas Day.

Mobile Testing

Just as important as keeping the network in check, is making sure that every phone sold by Vodafone works as intended.

It’s a process that often begins many months before a phone is even announced by the manufacturer, let alone announced as being sold by Vodafone.

Testing is done both in Newbury and also in a central testing facility in Germany, as much of the portfolio for Vodafone is planned globally. However, there are sometimes handsets only for local markets, such as Vodafone Italy, where handsets aren’t subsidised and price plays a bigger part in a customers purchasing decision.

Each phone goes through hundreds of hours of testing to detect bugs, possible stability issues, or battery problems. In that time, there will be a number of different versions of firmware, from the earliest prototype software, right through to the final retail build.

Its own apps, and partner apps that Vodafone actively promotes, like Now TV or Spotify (these can be used for free on certain tariffs) must also be tested thoroughly.

The My Vodafone app, showing how many minutes, texts, and data, is left is now pre-installed on new phones. The purpose of that is to reduce simple calls to customer services to find out information that can be readily accessed in seconds.

The Net Perform app allows each user to send valuable data on network performance in any area they visit, which could help make improvements and detect blackspots, while network speed tests can be performed without using a single byte of your data allowance.

If a handset doesn’t meet Vodafone’s stringent set of rules, it will ask for changes to be made. In a worst case scenario, failing to fix issues could lead to that phone being rejected for sale, or at the very least delayed.

Occasionally, Vodafone finds issues that are significant enough to result in an update that goes out to every handset, even the non-Vodafone branded variants.

Even Vodafone’s own branded phones go through the same strict processes.

Some manufacturers are better at responding to such requests than others, and suffice to say the long-established manufacturers understand more about the strict requirements Vodafone has, than some of the newer entrants into the market – particularly new manufacturers based in China – where it can be a bit more of a surprise.

Besides software, there are many health and safety issues, like making sure the handset conforms to strict regulations on batteries, SAR ratings (emissions) and recycling. Then there’s Bluetooth and Wi-Fi certifications to comply with too.

Vodafone also carries out tests that go beyond the standard industry requirements. For example, instead of carrying out a EU-standard 1 metre drop test, Vodafone tests from a height of 1.5 metres. What Vodafone doesn’t do is test things like waterproofing, as promoted by some manufacturers (e.g. Sony Xperia models).

Finally, I thought I’d ask about how Vodafone deals with new Apple products ahead of release. I didn’t expect much detail, and was simply told that all work with Apple is done under a strict Non-Disclosure Agreement. Well, I tried!

Ongoing Handset Support

No network operator in this day and age can ignore the issue of software updates, especially when it comes to Android users.

Every phone ranged by Vodafone has a plan for at least 18 months of support, which includes regular updates for bug fixes, Vodafone’s own apps, as well as planning how to manage the big OS updates.

Vodafone is able to schedule OS updates with manufacturers quicker than in the past, thanks to Google’s new approach of releasing each new OS to developers months in advance of its release to consumers.

It means the lead times for getting a new build developed and tested is greatly reduced, although some manufacturers are still faster than others.

It seemed clear that there’s still work to be done in this area, but nobody could accuse Vodafone of not being aware of it, or wanting to improve things as much as possible.

Network Testing

After checking out the NOC and learning a bit more about handset testing, we ventured over to another building where the really technical work is carried out.

Hidden away on an industrial estate, we arrived in style in a converted VW Camper van that I’d love to have driven home.

After clearing security, we ventured inside to see the various test labs that include a number of Faraday cages that allow a range of tests to be carried out without in a totally radio-free environment.

Feeling a little like stepping into a nuclear bunker, every Faraday cage at Vodafone’s testing centre allows staff to isolate themselves from the outside world and feed in only the radio signals they wish to test.

It might be to test out handing off from 2G to 3G, or 3G to Wi-Fi calling, or make calls on 4G, or it might be to test handsets out to make sure their internal antennas work optimally on every frequency that Vodafone uses.

It’s here too that Vodafone tests the ‘boring’ (but absolutely vital) things like how to make sure the network can work with a myriad of different equipment that has been sourced and installed since Vodafone’s inception in the early 1980s, and has gone from analogue to 2G (GSM), 3G and now 4G.

Behind the Faraday cages we saw a huge room kitted out with rows upon rows of tall cabinets, each fitted with every bit of network infrastructure equipment that Vodafone uses all around the country.

The noise was overwhelming from all the fans keeping everything cool, and it was amazing to imagine how much time and effort was required to connect everything together, especially when all the cabling runs under the floor.

There was equipment from Huawei, Ericsson, Nokia Siemens and many others, including equipment from different generations. As with most technology, the newer cabinets contained equipment that replaced that of multiple cabinets before it.

Before anything can be installed on the live network, it is tested here.

The Future

Vodafone is beginning to refarm more of its 900MHz spectrum, currently used primarily for 2G, to 3G. With the accompanying backhaul upgrades that go with every site update, this will greatly increase data speeds for users, as well as giving HD Voice to more people.

According to ‘Dr Rob’ (real name Robert Matthews), Vodafone’s resident network expert, having that 900MHz spectrum allows Vodafone more freedom with its 800MHz 4G network.

This is because some other networks have to reduce the 800MHz 4G power levels to remain within existing 2G (1800MHz) and 3G (2100MHz) coverage footprints. Lower frequencies can travel further, and penetrate buildings more successfully.

Until voice is routinely carried over 4G, all networks must ensure there’s always the ability to make a call (in particular emergency calls) wherever there’s phone service available.

In the case of Three, which only has a 2100MHz 3G service alongside its 1800MHz 4G service, the recently activated 800MHz 4G spectrum is heavily restricted because it spread well beyond the confines of its existing network. Only users owning a compatible handset with Three firmware installed get access to it, which is quite limiting.

The reason? Because it requires Voice over LTE (VoLTE) technology, which currently needs special settings that can’t be manually added to any device.

Only the iPhone, regardless of where you obtained it, receives a standard set of carrier updates that means anyone can use VoLTE once it’s enabled. Ironically, Android Lollipop and above has the ability to allow over the air updates for VoLTE, but as yet no network has utilised the feature.

VoLTE will eventually remove the need for networks to restrict 4G coverage, but Vodafone doesn’t appear to be in a rush to launch and wants to get things working perfectly instead.

Like EE, Vodafone needs to be sure that VoLTE won’t impact on its dropped call rate. Some Three users have complained of dropped calls, and increased battery drain.

But, I can at least confirm Vodafone is at least working on VoLTE and taking it seriously, because I saw it with my own eyes. In fact, I went one step further than that and actually got to make a test VoLTE call from a Samsung Galaxy S6 to a Galaxy S6 edge.

As you might have guessed, all this was done in one of the Faraday cages on its test network.

The same tests are also being carried out on the iPhone 6s and 6s plus, confirming that these will be the first devices Vodafone will support when VoLTE does launch, presumably early next year.

My Thoughts

In the past, I’ve expressed concerns about Vodafone’s slow, almost lazy, roll out of 3G.

Compared to the likes of Three and T-Mobile (at the time), Vodafone seemed to misjudge the huge increase in data usage and got caught out hugely, with many users unable to access data at all.

Even where 3G existed, it was often far slower than rival offerings, due in part to antiquated backhaul links that couldn’t cope with heavy usage, and a lack of foresight in upgrading sites to support faster 3G technologies like HSPA and DC-HSDPA before 4G came along.

Since it began rolling out 4G two years ago, the upgraded areas have been transformed beyond all recognition. 4G upgraded sites bring upgraded 2G and 3G too, and Vodafone’s plan to run more fibre to sites will only further improve things.

Many people can now enjoy speeds in excess of 100Mbps, which is a huge leap.

The policy of working on a town-by-town basis does mean that some people are yet to see the full benefits of the network upgrade, but they will come. And when they do, the difference is staggering.

With a good chunk of 2600MHz 4G spectrum, and plans to switch much of its 900MHz 2G spectrum over to 3G usage, I came away confident that Vodafone is effectively building an all-new network that’s fit for the future, and one that will easily rival a certain other network that benefitted from a rather large head start!

• Did you like this feature? Please share your comments below.
• Also, please consider clicking on the ad below to show your support. If you can’t see it, please consider whitelisting this site from your ad blocker!