Outage Update (12/10/20: 10am)

As of this morning, most BTES IPTV cable services have been restored. While the total number of set top boxes impacted due to the IPTV cable outage was approximately 9,000 – this equates to about 4,800 customers.

“As of 9am, this number is down to approximately 425 set top boxes, which equated to about 200 customers,” said BTES CEO Mike Browder. “If your cable service through your set top box has not been restored, please unplug the power cord from your box, wait two minutes, and plug it back in. If this does not restore your cable services after 15 minutes, please email us at customerservice@btes.net.”

The outages began on Monday evening.

Browder continued, “We made some changes in our system in preparation for a 100 Gigabit backbone network upgrade that happened beginning at 12:30am Tuesday morning. This upgrade was not an IPTV cable system upgrade but rather an upgrade to our entire backbone network. For days leading up to this upgrade, we conducted testing and made changes to the system with the expectation of minimizing potential customer down time. However, the changes we made to the system on Monday afternoon in preparation for the upgrade responded differently than it did during our testing which resulted in many IPTV cable customers experiencing an outage.”

BTES is continuing to research and working towards eliminating the situation that caused this to happen. Any additional updates will be posted on the BTES Facebook page (@BTESupdates) and on the BTES website at www.btes.net/news.

 

Outage Update (12/9/2020: 5pm)

Good evening. There has been a lot of information on Facebook and other places from us and others so we want to provide a bit more clarity on what happened, what we are working on, and some things that we’ve seen work in some cases for our customers.

We made some changes in our system in preparation for a 100 Gigabit backbone network upgrade that happened beginning at 12:30am Tuesday morning. This upgrade was not an IPTV cable system upgrade but rather an upgrade to our entire fiber optic backbone network.

We recognize that some of our customers started experiencing issues with their services Monday evening. The changes we made on Monday afternoon (which, as a side note, our testing indicated that those changes would not impact our customers’ cable, Internet, and telephone services) is what actually caused the issues – not the upgrade. We tested these changes prior to going live with them but our system responded differently than it did during our testing which resulted in the outages.

Our team immediately began working to resolve the issues – and are continuing to work to restore services. The upgrade did not start until 12:30am. At that time, the upgrade started and successfully completed.

Here are some things that we’ve seen work in some cases. While these may not work in all cases and it may not immediately bring services back up for you, it is something to try. With the additional information we now have, here are a few more direct actions you can take to potentially speed up the restoration process.

We know we told you not to reboot before, but because the bottleneck has been reduced to a manageable level, if you see a red X or the BTES logo on your TV screen – go ahead and unplug the power cord from your box, wait a couple of minutes, and plug it back in.

If you see a message to restart your system, go ahead and choose the option to restart.

If you can hear sound and see the message to restart your system, then on your remote, press set top box and then press okay to select don’t restart.

If your screen says “your television signal has been lost” – we need to do some further checking. Please send us a message at customerservice@btes.net so we can walk through a few steps.

If you bypassed your set top box, you can reconnect your coax cable to your set top box at a time that is convenient for you. This will start the reconnection process for that box.

Our team has been working to answer calls, messages, and emails and respond to comments as quickly as we can. We can’t tell you enough how much we appreciate the kind words and messages that have been sent to us. We are also very appreciative of those customers that have worked with us throughout this process to provide feedback on what they are seeing on their screens and being patient as we walked through steps and tested scenarios with them.

We are so sorry this happened, and we are working to ensure this doesn’t happen again in the future.

Press Release - BTES IPTV Cable Outage Information

BTES IPTV cable customers experienced an outage this week.

BTES CEO Mike Browder explained, “Our plan was to upgrade our core backbone network to 100 Gigabits to provide our customers with even better service and keep up with increasing customer demands. We conduct system upgrades regularly and followed similar processes for this upgrade. We planned for and started this upgrade at 12:30am on Tuesday morning (12/8).”

The upgrade to the 100 Gigabit backbone was successful.

“For days leading up to this upgrade, we conducted testing and made changes to the system with the expectation of minimizing potential customer down time,” said Browder. “However, the changes we made to the system on Monday afternoon in preparation for the upgrade responded differently than it did during our testing which resulted in many IPTV cable customers experiencing an outage.”

Of BTES’ 17,000 Internet customers, less than 200 customers also experienced a disruption in their Internet service. A very small number of customers experienced a telephone outage. Most Internet service was restored in less than an hour and telephone service came back on throughout the day on Tuesday.

The IPTV cable outage included nearly 9,000 of BTES’ IPTV set top boxes and most attempted to reconnect to the system at the same time and, in this situation, the system bottlenecked, and very few boxes were progressing in the reconnection process.

Browder further explained, “We compare this situation to what one may experience if they go to their favorite store on Black Friday. A lot of people arrive at the same time and try to get into one place all at once, at the same time to go after the same items. Just like fighting to get in the door, the set top boxes were fighting to get their information so that they can come back online. We had nearly 9,000 set top boxes attempt to reconnect to our system at the same time.”

BTES has been working since the outage began to slow down the “crowd” trying to get into the system and make it so that a few boxes could come back at a time instead of almost 9,000 at once.

“We have never had a situation when nearly 9,000 set top boxes tried to reconnect to our system all at the same time,” said Browder.

BTES is seeing success with this process and customers’ set top boxes are slowly coming back online.

“Throughout the day on Tuesday, some customers were not able to reach us regarding their service interruptions,” explained Browder. “Most of our phones at BTES were out on Tuesday and we apologize that we were not able to answer more calls. We communicated updates as they became available on Facebook and our website and our team continues to respond to calls, emails, and messages as quickly as we can.”

BTES is continuing to research and working towards eliminating the situation that caused this to happen. Any additional updates will be posted on the BTES Facebook page (@BTESupdates) and on the BTES website at www.btes.net/news

IPTV OUTAGE UPDATE:

 

8am (12/9) - Our team made a lot of progress last night. We have received confirmation that more set top boxes are coming back up and staying up.

If you are prompted with an on-screen message to restart the system, go ahead with this step. Please only restart your set top box if you see this on-screen message. If you do not have this message on your screen, just leave your box as is at this time. If you restart your set top box without this message, this will put your box at the end of the line and may take much longer for it to come back up. We will provide further instruction soon.

Once you see the BTES logo on your screen, please just leave the box as is for the time being. This will help speed up the process and not overload the system.

We will have another update soon with more details and will provide additional direction, if necessary. Please keep watching Facebook and our website for updates.

 

 

10:00pm - Based on the current situation, we recommend that our customers do not disconnect the power from their set top boxes. (Normally - and even last night - we asked our customers to unplug their set top box, wait a couple of minutes, and plug it back in. However, in the current situation, we recommend not disconnecting the power from your set top box.)

 

 

9:30pm - We are not planning to do a roll back so Internet services should not be impacted. We are continuing to work to restore IPTV cable services. We are seeing indications that services may be coming up slowly and some set top boxes are being restored although it may not be all boxes in the home. Some customers are reporting that their set top boxes have come back on and stayed on. Our team has been and will continue working with outside vendors to receive additional help and support and we will continue to work to restore services. We will provide another update tomorrow morning as soon as we have more information available.

 

3:30pm - We understand that some customers have not been able to reach us regarding their service interruptions. Most of our phones at BTES have been out today and we apologize that we have not been able to answer more calls. 

We are continuing to work on the issues that our customers are experiencing with IPTV cable and some phone services. All Internet service should be restored. Please check to see if your Internet service is working. If you are still experiencing an Internet outage, please send us a private message through Facebook or send an email to customerservice@btes.net with your protected account information including your account name, service address, and best contact phone number to reach you so we can troubleshoot further. 

As we continue to work to restore IPTV cable, some customers are watching channels 2-82 by bypassing their wired set-top box. To do this, unplug the coax cable from the box and plug it directly into the television. Some TVs may require additional steps such as a channel scan for this to work.

If services are not able to be fully restored, we will plan to roll back from the upgrade tonight. The roll back will require most BTES fiber services, including Internet, to be unavailable for a brief period of time. If this becomes necessary, more details will be provided in a future update.

We understand your frustration. Credit amounts will be evaluated and applied after services are restored. Our main priority at this time is to restore services as quickly as possible.

We apologize for this inconvenience as it was our intention to upgrade our system to provide better service and meet the increasing Internet demands of our customers. We appreciate the grace that has been extended to us as we work through these issues and will provide further information as it becomes available.

 

 

10:20am - As we continue to work to restore services, some customers are watching channels 2-82 by bypassing their set-top box by unplugging the coax cable from the box and plugging it directly into the television. Some TVs may require additional steps such as a channel scan for this to work. We are continuing to work as quickly as possible to restore IPTV cable services and are providing updates as they become available on Facebook and our website at btes.net/news

 

 

9:45amAll Internet service should be restored. If you are still experiencing an Internet outage, please send us a private message through Facebook with your protected account information (account name, service address, and best contact phone number) so we can troubleshoot further.

 

 

9:15am - First, let us say that we are incredibly sorry. We realize that not having your BTES services is an inconvenience and we want to apologize for any issues that our customers are experiencing. Our team has been working throughout the night and are continuing to work today to restore services to our customers. Last night, our plan was to upgrade our core backbone network to 100 Gigabits to provide our customers with even better service and keep up with increasing customer demands. For days leading up to this upgrade, we’ve been making minor changes to our system without issue. We are still working to troubleshoot what happened last night and are working with our national support help desk. We will provide updates as they become available.

 

 

6:47am - Many of our IPTV cable customers may be experiencing a cable service outage. We apologize for any inconvenience. Our team is working to restore service as quickly as possible. We will provide more updates as they are available.