Connection Going Offline Every 10 Minutes

  • 1
  • Problem
  • Updated 2 years ago
  • (Edited)
Greetings all.

I've encountered an interesting issue that I've never seen before. It started yesterday morning at around 5am when our fiber connection when down for exactly one second. Normally I wouldn't care about such a brief interruption, but it happened again 10 minutes later and kept happening every 10 minutes until about 5am this morning (almost exactly 24hrs).

Below is a truncated snippet of yesterday and today's connection history. As you can see, my gateway logged a "disconnect" event every 10 minutes that ended about 6 hours ago:

> Main Uplink log of day(s) Oct 16-17, 2015.
> Oct 16 05:06:57 uplink[main] Uplink 'main' status: 'OFFLINE'
> Oct 16 05:06:59 uplink[main] Uplink 'main' status: 'ONLINE'
> Oct 16 05:17:00 uplink[main] Uplink 'main' status: 'OFFLINE'
> Oct 16 05:17:01 uplink[main] Uplink 'main' status: 'ONLINE'
> Oct 16 05:27:02 uplink[main] Uplink 'main' status: 'OFFLINE'
> Oct 16 05:27:03 uplink[main] Uplink 'main' status: 'ONLINE'

< this went on all day >

> Oct 17 04:41:43 uplink[main] Uplink 'main' status: 'OFFLINE'
> Oct 17 04:41:44 uplink[main] Uplink 'main' status: 'ONLINE'
> Oct 17 04:51:45 uplink[main] Uplink 'main' status: 'OFFLINE'
> Oct 17 04:51:46 uplink[main] Uplink 'main' status: 'ONLINE'
> Oct 17 05:01:47 uplink[main] Uplink 'main' status: 'OFFLINE'
> Oct 17 05:01:48 uplink[main] Uplink 'main' status: 'ONLINE'

Those of you familiar with MTS's fiber service can probably see I'm not running a typical setup. In fact I'm running a custom gateway with MTS's router set to "bridge mode" and all of its garbage features disabled so that I can control things better with my own technology. That, and my gateway gives me much greater insight into what's happening on the network than MTS's router does.

So here's my question: What's going on here?

If the MTS router were malfunctioning and rebooting every 10 minutes then it would explain the disconnect, but it would also mean that every outage would have lasted up to a minute instead of just a second (because it takes a while for the router to reboot). That... and my gateway would have logged a level 1 link-loss, which never happened. So this seems unlikely.

Another option is the fiber modem. It is possible there was some kind of malfunction that could cause the outage, but why only for one second and why every ten minutes? And, perhaps the most perplexing is why for exactly 24 hours? This seems unlikely because technology rarely fixes itself.

My last explanation might be that MTS's fiber service was either under attack or undergoing some kind of automated maintenance since a one second outage every 10 minutes for exactly 24hrs is unlikely to be the work of a human in a truck somewhere.

Thoughts? Can anyone from MTS weigh in?

- Sven
Photo of Sven Thomas

Sven Thomas

  • 132 Points 100 badge 2x thumb
  • perplexed, but content

Posted 3 years ago

  • 1
Photo of JG

JG, Champion

  • 4,556 Points 4k badge 2x thumb
I've never seen it, but maybe there was an issue with the Alcatel ONT unit or the power supply/battery backup for the ONT. But, like you said, technology rarely fixes itself.


Maybe your light level coming in is right on the fringe of being acceptable? Maybe there was an issue at the hub in your neighbourhood and the optical splitter you're on went bad, and if everyone on that splitter was having this issue, tech support may have sent a tech to replace the splitter?


Last thought is maybe they were doing something software related on the back end?


Would be interested to know what it was, in case we ever come across it in the field.
(Edited)
Photo of Sven Thomas

Sven Thomas

  • 132 Points 100 badge 2x thumb

You might not be far off.

Though my wife made a pretty astute observation when I laid out the nature of the outage.  She said, "sounds like someone was testing something."

Now there's a thought...

Photo of Jason Z

Jason Z, Official Rep

  • 16,946 Points 10k badge 2x thumb
Hi Sven,

I have received your info and have sent it on to our Engineering/Operations folks to have a look.  Given your service is working fine we will look at it and get back to you within a week or so.  If you see the issue again let us know.

Thanks again for being an MTS customer - we do appreciate it!!
Photo of Sven Thomas

Sven Thomas

  • 132 Points 100 badge 2x thumb
Thanks Jason!
Photo of Jason Z

Jason Z, Official Rep

  • 16,946 Points 10k badge 2x thumb
Sven - send us your adress and any account info to community@mts.ca so we can investigate further.

Thanks for your post!
Photo of Jason Z

Jason Z, Official Rep

  • 16,946 Points 10k badge 2x thumb
Hi Sven,

I heard back from the technical team - we can see that right at the start of the issues you noted we upgraded the Residential Gateway (RG) to your house.  There is a chance that the router you are providing had some conflicts with the firmware that were eventually resolved (did you restart something?).

 At any rate you are correct your setup isn't exactly standard - the technical folks did mention that our router was in DMZ mode not bridge mode but perhaps that is not relevant - as we haven't heard from you I suspect all is continuing to work fine.

Thanks for the question
Photo of Sven Thomas

Sven Thomas

  • 132 Points 100 badge 2x thumb
Thanks so much for the update Jason - I was just thinking about this the other day and wondering if anyone had a chance to look into it.

A firmware update to the upstream router certainly represents a credible explanation, though in my 20 years in dealing with networks I've never seen exactly this behaviour. Still, the router isn't mine and I'm only familiar with it insofar as I can configure it to get out of my way; so in all fairness I'm not really in a position to say with authority whether or not it was the cause of the issue. If MTS' engineers believe it was an anomaly caused by a firmware update then I'm inclined to trust their assessment.

As a final note in case they're interested (along with anyone else following this thread), I'm running the Endian Community Edition firewall on an old PC. From there I run a backbone to a managed switch where it gets distributed to both the wired and wireless clients. We first noticed the outage because Endian audibly beeps when the main interface changes states with varying tones for outages at different ISO levels. We have, as you suspect, been running very well since the outage so my investigation really is one out of curiosity. :)

http://www.endian.com/community/overv...

Anyway... thank you for your time and be sure to extend my thanks to your engineers for their efforts.

- Sven
Photo of Graham

Graham, RF Engineering

  • 6,902 Points 5k badge 2x thumb
I vote this the most erudite response yet :)  Thanks for the correspondence Sven.
Photo of Sven Thomas

Sven Thomas

  • 132 Points 100 badge 2x thumb

I actually lol'd upon reading this.  Graham, you made my day.  :)