This post is a little war story about a time I helped out a friend of mine. My favourite movie for 2017 was John Wick Chapter 2. Not quite as good as the first movie but still awesome. In one of the scenes Laurence Fishburn shouts “Somebody, please! get this man a gun” Just prior to Christmas a friend of mine asked if I was available to assist with an issue he was working on with one of his customers networks. They were experiencing severe problems with applications affecting almost the entire Data Centre. As it happened I’d finished work for Christmas so had some time to take my sword to the fight. Trouble shooting was via an online conference session with an onsite engineers laptop tethered to a mobile phone. Brilliant.
Troubleshooting
I requested CLI access to troubleshoot as connecting to any of the devices using Web UI was failing. The windows jump box they provided me didn’t have any terminal software installed. Due to AD permissions we couldn’t even load a command prompt. The customer had no out of band management, amazing. All the tooling resided on servers connected to the network via production switches on the same network that was currently on its knee’s. Eventually one of the managers on the conference call suddenly shouted “Will somebody please get this engineer a CLI“
This made me chuckle (microphone muted of course) and reminded me of the fore mentioned scene from the film. Anyway, they eventually provided me access to a Linux VM, a short while later after some tcpdump action the issue was identified to be MTU. It turned out that just prior to the issues a core firewall had been replaced. The DC was running jumbo frames on almost all the devices including the firewall. Jumbo frames needed to be enabled on the replacement firewall in addition to the restored configuration.
Issue Resolved
Jumbo frames enabled, firewall rebooted and boom! everything started working. MTU mismatches on the network affect applications that use crypto, especially TLS.
Closing Thoughts
As the SDN apocalypse moves closer lets hope that out of band tooling is not forgotten.
