Part 3: SD-WAN Operations

Troubleshooting / Service Assurance

With top tier SD-WAN vendors, you should expect next-generation trouble shooting capability. The embedded tools in SD-WAN management portals are vastly superior to MPLS tools. VeloCloud’s Orchestrator provides excellent link diagnostics (including throughput, jitter, latency and packet loss), connectivity and routing, and application performance. Voice MOS reporting is also a key feature for voice trouble shooting, which Macquarie tracks for every voice call. This user experience reporting with granular resolution down to ‘per user and per site’ application is a real Star.

Site level traffic destination and volumes

Top applications for any device

Velocloud’s Orchestrator makes this sophisticated diagnostic information simple to use, yet incredibly powerful – catering for both customer engineers and administrators, as well as Macquarie’s NOC engineers to triage, identify root cause of problems and remediate rapidly.

When trouble shooting, Macquarie NOC engineers enable real-time reporting via the Orchestrator to poll the SDWAN edge every 2 seconds. This provides valuable insights such as:

• Live throughput for the site and for each access in use
• Jitter, packet loss and latency statistics
• Behaviour of traffic flow at a protocol level
• Load balancing and packet replication in action

 

 

Network Management

Management thinker Peter Drucker is quoted as saying that “you can’t manage what you can’t measure.” This statement is never truer for an SD-WAN deployment where you are using software to define your network. You should ensure your SD-WAN Management Portal has at least the following feature checklist:

• Sophisticated network connectivity and routing capabilities
• Application-level network performance report
• Link access diagnostics including jitter, packet loss and MOS
• Site policy administration, paste and click new site configuration
• Graphic reports
• Applications reports per user
• User read/write management control
• SYSLOG event access

With top tier SD-WAN vendors, you should expect next-generation trouble shooting capability. The embedded tools in SD-WAN management portals are vastly superior to MPLS tools. VeloCloud’s Orchestrator provides excellent link diagnostics (including throughput, jitter, latency and packet loss), connectivity and routing, and application performance. Voice MOS reporting is also a key feature for voice trouble shooting, which Macquarie tracks for every voice call. This user experience reporting with granular resolution down to ‘per user and per site’ application is a real Star.

 

GUI display for sophisticated, but easy to use network management.

Cloud

With the megatrends of cloud and hybrid cloud, it is important to consider extending the software-defined network boundary to include virtual instances of an SD-WAN edge device in the cloud or DC. So, if you are using or are planning to use public cloud, Macquarie’s Cloud or your own DC’s, you can stretch the network boundary to securely include your cloud resources by deploying the SD-WAN virtual edge on VMs.

The virtual edge will run on standard VMs hosted by Azure, AWS and GCP as well as Macquarie’s Cloud. All the benefits of cloud workload flexibility are retained, but you get better security, control and visibility of your network allowing any cloud environment, public or private, to be an extension of the WAN. The single SD-WAN Orchestrator window provides visibility of connectivity, routing, bandwidth and performance of your whole network.

 

Profile Management

Setting up profiles for groups of sites drives standardisation and consistency across the network. The added bonus is that it makes our lives easier while deploying new sites or changes! With Macquarie’s SD-WAN solution, a new site can be configured and integrated into the network in minutes. Similarly, application tuning can be done rapidly across the network.

One of the Stars of Macquarie’s VeloCloud by VMware solution is how we set up customer profiles on-site type basis. For example, a customer may have four profiles set up – DC, HQ, State Office & Branch. A certain set of rules, e.g. blocking Netflix, increases voice prioritisation while throttling Windows updates can be done for all four profiles and rapidly deployed across the whole network. In contrast, YouTube is not a priority for most locations but the State Offices have marketing staff who rely on this for work purposes. No problem, YouTube can be throttled in the DC, HQ and Branch profiles and the remaining sites using the State Office profile continue using YouTube without restrictions.

 

Packet steering to remediate link performance across multiple accesses.

Ticketing

Wouldn’t it be great to proactively share every network event with customers? No! This was a Scars and Stars story. With the extensive event and performance tracking of VeloCloud’s SD-WAN, we turned on auto-ticketing of the Orchestrater’s SYSLOGs into Macquarie’s OSS tools which in turn created tickets for every SD-WAN network event. This was a definitely a Scar. We ended up generating a five-fold increase in customer tickets and swamping our customer held desk with reams of unnecessary alerts (fluctuations in 4G signal strength, variations in jitter, packet loss etc.) that did not impact our customer experience or SLGs at all. Well, we could turn off SD-WAN auto-ticketing, but that would mask some genuine events and that we absolutely wanted to proactively notify customers about. The solution was challenging, and so we kicked off an extensive OSS development project to filter out certain events and correlate other events. This project has now been successfully completed, and auto-ticketing runs successfully. We definitely wore some Scars here, but we also made it a Star.

 

4G back-up – it’s awesome, but watch the costs

All access services are subject to performance degradation or outages – even fibre. So, dual access redundancy is an ideal but often expensive design. Most customers in our business segment have redundant accesses for their DC and HQ. But few can afford or justify redundant access for all of their branch sites. 4G ethernet modems have evolved to be a cost-effective alternative. The SD-WAN edge device has USB ports to attach a dongle or ethernet modem. The SD-WAN Orchestrator detects if there is primary link degradation or break, then switches from primary to 4G back-up. This solution comes with the expected 4G data costs from carriers – typically you choose between 2GB to 50GB data plans and of course, the price increases correspondingly. For branch sites say less than 100Mb/s, 4G service is a common back-up solution.

So, when 4G kicks in as a back-up, how do we manage the risk of the primary link being down for many hours and blowing out the 4G mobile costs?

• 4G alerts have been implemented in our auto-ticketing systems, which alerts customers when their primary access is offline.
• Selecting the right 4G data plan is a key conversation with customers and we can recommend the appropriately sized bucket data plans that will encompass all sites.

4G backup capability is a real Star, but if we get the mobile data plans wrong, it can lead to a few Scars.

 

4G Ethernet Modem with external aerial

The concept is straightforward – let’s drop in a 4G dongle as a back-up to those branch edge devices providing a low cost and higher availability solution. If the primary links fails or performance drops away, the Orchestrator has the intelligence to switch from primary to 4G backup. After deploying and operating the network for a while, it became evident that some customer sites had 4G performance issues. In many instances, the SD-WAN edge device is deep inside some computer room in the basement or back office, with inherently attenuated 4G signal strength. After a worldwide search, we implemented a new improved solution using a 4G Ethernet Modem with external aerial to replace the dongle. The device has rabbit ears antenna for enhanced signal providing better throughput and a stable connection to the network. The modem can be run up to 50m from the edge device, allowing us to place the 4G antenna elsewhere in the customer site. This was a real game changer for those sites with signal strength issues – A Scar story that turned into a genuine Star.

 

Voice Boost

Voice – ask any NOC engineer, is one of the most sensitive customer applications. Loss of voice or call quality degradation is a high-priority ticket. Hence, our ability to implement ‘Voice Boost’ is very effective to not only reduce the number of voice tickets logged with our support team, but it provides end users with good quality voice calls even when there are underlying performance issues in the network. VeloCloud’s SD-WAN actively monitors network statistics like jitter, latency and packet loss to automatically implement jitter buffering, FEC and packet steering to ensure the best possible voice performance. This is a unique feature to the VeloCloud technology and how we implement it – a real Star.

 

Conclusion

In conclusion, Macquarie Telecom’s experience over more than three years running Australia’s largest SD-WAN deployments has given us great knowledge in designing optimised, high performance and cost-effective customer SD-WAN solutions for our customers.

This experience in running networks has given as some Scars, like bespoke design for each customer network. But our learnings have also generated some wonderful Stars such as over-the-top solutions, and customer policy templating. Not to mention Macquarie Telecom’s selection of VeloCloud by VMware, as one of the best SD-WAN vendors in the world, as validated by Gartner.

More importantly, we have been able to provide great service to our customers. Macquarie’s industry-leading +81 Net Promotor Score by our customers over the last six months tells the story. Follow the link to our SD-WAN reference customer, Transit Systems.