Network Status News & Information

We will be conducting scheduled maintenance on Friday, Mar 22nd, 2024, from 6:00 a.m. to 12:00 p.m. (EDT) Eastern Daylight Time.

During this scheduled maintenance event, we will be upgrading the underlying host hardware for the VPS hypervisor cluster. This upgrade is part of our commitment to further enhance the performance, reliability, and security of our physical hosting infrastructure.

Maintenance Details

• Start Date: March 22nd, 2024
• Start / End Time: 0600 - 1200 (EDT)
• Maintenance Duration: 6 hours
• Expected Interruption: 1 hour
• Affected Services: All Virtual Private Servers (VPS/VMs) and VM management access

Impact on Services

Virtual servers are expected to be unavailable for up to 1 hour during this maintenance window while hardware migration is completed.

Updated information regarding this maintenance window will be posted to https://maintenance.thirdhosting.com 

Action Required

No action is necessary.  All servers will be shutdown automatically at the start of their migration and restarted post migration.

We appreciate your patience and understanding as we continue to work to improve our services and server reliability.  If you have any questions or concerns regarding this maintenance action, please contact our Support Department.

Best Regards,
Third Generation Internet Services
support@thirdhosting.com 

0600 EDT: Scheduled maintenance has begun. 
0620 EDT: Hardware migration has started
0650 EDT: All Virtual Machines are now offline and being migrated to the new cluster.
0800 EDT: All VMs have been successfully migrated on the cluster.

We will be conducting scheduled maintenance on Friday, Feb 9th, 2024, from 6:00 a.m. to 10:00 a.m. (EST) Eastern Standard Time.

During this scheduled maintenance event, we will be upgrading the underlying host hardware for the cloudwalker.thirdhosting.com cPanel server. This upgrade is part of our commitment to further enhance the performance, reliability, and security of our physical hosting infrastructure.

Maintenance Details

• Start Date: February 9th, 2024
• Start / End Time: 0600 - 1000 (EST)
• Expected Duration: 4 hours
• Affected Services: All accounts and services on cloudwalker

Impact on Services

All accounts and services hosted on cloudwalker.thirdhosting.com will be unavailable during this scheduled period.

All services and background processes will be shutdown at the start of the maintenance and will be restarted once the server is back online.

Updated information regarding this maintenance window will be posted to https://maintenance.thirdhosting.com 

Action Required

It is recommended that MU* customers shutdown their background game processes prior to the maintenance event to ensure proper shutdown of their databases and files.  No further action is necessary.

We appreciate your patience and understanding as we continue to work to improve our services and server reliability.  If you have any questions or concerns regarding this maintenance action, please contact our Support Department.

Best Regards,
Third Generation Internet Services
support@thirdhosting.com 

Updates

0600 EST: Scheduled maintenance has begun. Shutdown of cloudwalker.thirdhosting.com in progress.
0605 EST: cloudwalker.thirdhosting.com has been successfully shutdown. Migration to new hardware has begun.
0645 EST: Migration has reached 25% complete.
0720 EST: Migration has reached 50% complete.
0815 EST: Migration has reached 75% complete.
0920 EST: Migration has reached 100% complete. The system has been powered back online.
0925 EST: We'll work over the next 30 minutes to restore previously running background services.
1000 EST: Scheduled maintenance has been completed.  All services are online.

We will be conducting scheduled maintenance on Friday, Feb 2nd, 2024, from 6:00 a.m. to 10:00 a.m. (EST) Eastern Standard Time.

During this scheduled maintenance event, we will be upgrading the underlying host hardware for the stonebreaker.thirdhosting.com cPanel server. This upgrade is part of our commitment to further enhance the performance, reliability, and security of our physical hosting infrastructure.

Maintenance Details

• Start Date: February 2nd, 2024
• Start / End Time: 0600 - 1000 (EST)
• Expected Duration: 4 hours
• Affected Services: All accounts and services on stonebreaker

Impact on Services

All accounts and services hosted on stonebreaker.thirdhosting.com will be unavailable during this scheduled period.

All services and background processes will be shutdown at the start of the maintenance and will be restarted once the server is back online.

Updated information regarding this maintenance window will be posted to https://maintenance.thirdhosting.com 

Action Required

It is recommended that MU* customers shutdown their background game processes prior to the maintenance event to ensure proper shutdown of their databases and files.  No further action is necessary.

We appreciate your patience and understanding as we continue to work to improve our services and server reliability.  If you have any questions or concerns regarding this maintenance action, please contact our Support Department.

Best Regards,
Third Generation Internet Services
support@thirdhosting.com 

Updates

0600 EST: Scheduled maintenance has begun. Shutdown of stonebreaker.thirdhosting.com in progress.
0610 EST: The stonebreaker server is now offline, all processes have been shutdown.
0615 EST: Hardware migration has started
0800 EST: The migration has reached its 50% threshold
0945 EST: The maintenance window will be extended approximately 1 additional hour while services are restored.
1015 EST: Connectivity to stonebreaker has been restored. We will restore background processes over the next 30 minutes
1045 EST: Scheduled Maintenance has been completed.

Systems on disk array vps01 are not responding and unable to gracefully shutdown processes due to device errors.

We're attempting a cold reboot of the system which will affected systems cloudwalker and stonebreaker

All services were impacted while the OVH network was unavailable.

http://travaux.ovh.net/?do=details&id=53798& 

There is a scheduled maintenance event for server fireforge.thirdhosting.com on Tuesday, November 10th, from 2:00 AM to 6:00 EST.

As a reminder, all accounts on the fireforge server will be migrated to their new server cloudwalker.thirdhosting.com (192.99.187.232).  Although this maintenance window is scheduled for up to 4 hours, each individual account migration is expected to take approximately 20 minutes.  During this migration event, background processes will be stopped on the former server and restarted on their new server upon account migration.  DNS information for all domains will be automatically updated to point to the new IP address of 192.99.187.232.  Any external domains not hosted under our DNS system will need to be manually adjusted.  Please refer to previous email notices in regard to the account migration for additional details.

Sincerely,
Third Generation

There is a scheduled maintenance event for server swiftblade.thirdhosting.com on Monday, November 9th, from 2:00 AM to 6:00 EST.

As a reminder, all accounts on the swiftblade server will be migrated to their new server stonebreaker.thirdhosting.com (144.217.205.176).  Although this maintenance window is scheduled for up to 4 hours, each individual account migration is expected to take approximately 20 minutes.  During this migration event, background processes will be stopped on the former server and restarted on their new server upon account migration.  DNS information for all domains will be automatically updated to point to the new IP address of 144.217.205.176.  Any external domains not hosted under our DNS system will need to be manually adjusted.  Please refer to previous email notices in regard to the account migration for additional details.

Sincerely,
Third Generation

In an effort to improve our network services we will be scheduling a maintenance event from Wednesday, May 29th 2019 from 2:00 a.m. to Thursday, May 30th at 10:00 a.m.Mountain Standard Time (MST).  During this maintenance window, our network provider will be conducting maintenance on their Global Private Network infrastructure to implement routing changes to the network.  Throughout this maintenance window, you may encounter brief service disruptions of less than one minute while traffic is re-converging.

We appreciate your patience and understanding as we continue to improve our network and welcome any feedback you may have.  If you have any questions or concerns regarding this maintenance schedule, please contact our Support Department at any time.

Sincerely,
Third Generation Internet Services

Our monitoring systems have reported a network outage for the virtual stack at the Phoenix datacenter.  This stack currently hosts the Fireforge and Swiftblade servers.  The outage was reported at 15:05 MST.

We are currently investigating the issue to determine the cause and recovery time.  Additional information will be posted to this notice as it is made available.

We appreciate your patience and understanding as we work to identify and resolve the issue.

2018-06-21 15:05 MST - Outage event reported.

2018-06-21 15:15 MST – We will be restarting hardware shortly as the issue does not appear to be related to the data center or network services.

2018-06-21 15:35 MST - The hardware node has failed to respond to a system restart and further troubleshooting is required at this time to determine the extent of the issue.

2018-06-21 16:10 MST - On-site hardware technicians are currently reviewing the server's boot state to determine if the issue is software service or hardware related.

2018-06-21 19:30 MST - Data center technicians have determined the issue to be the network switch in our local rack and the issue is infact network hardware related.   The data center is currently preparing to replace the hardware at 20:00 MST.  Unfortunately, our server was restarted during the troubleshooting process so we will have to restart any background game services that was previously running once network connectivity has been restored.

2018-06-21 20:00 MST - Replacement of the front-end network switch has begun. ETR is approximiately 1 hour while network connections are migrated and configurations made.  We will restart previously running background services once network connectivity has been restored.

2018-06-21 22:00 MST - Replacement of the network switch was successful.  Connectivity to services has been restored and all previously running background processes have been restarted on customer behalfs.

We have received reports of intermittent latency and lag that is currently affecting services on the swiftblade server.

Upon investigating the issue, it has been determined that there is a routing issue for the IP address 66.85.147.93.

We are currently working with our network providers to resolve the issue and will update this log as information is provided.

For customers experiencing packet loss and "lag spikes" when connecting to their game ports, it is recommended to use an alternate IP address such as 66.85.147.92 to connect to their respective port.  This IP is a secondary IP associated with the server and currently unaffected by the routing issue.

Thank you for your patience as we work to resolve this matter.

Regards,
Third Generation

-- 2018/03/19 10:00 PM MST - NetworkOPS team at data center has been notified and is currently investigating the issue.  Please use secondary IP address (66.85.147.92) to bypass latency issues at this time.

-- 2018/03/20  5:00 AM MST - NetworkOPS has reported that the IP address was subject to a 12Gbps DDoS and has been routed through a DDoS protection network adding to the additional latency.

-- 2018/03/20  8:30 AM MST - The route on the IP address has been restored and intermittent latency appears to have been resolved at this time.

-- 2018/03/20 10:30 AM MST - With the IP route restored, DDoS attacks have resumed heavily against the server.  We're looking at alternative solutions.  With the DDoS proteciton route off, the issue fully impacts all connected severs to the hardware.

-- 2018/03/20 11:05 AM MST - We're null routing all IP addresses and ports at this time to regain administrative access over the server.  We'll begin opening IPs and ports one at a time to isolate how the DoS is occurring.

-- 2018/03/20 12:10 PM MST - Access to services has been restored and DDoS mitigations are being restored.  If further events occur, all ports and IP's will be temporarily firewalled to an exclusive whitelist based upon "game connectivity" for the day.  This should maintain transparent access while curbing the attack behavior.  However, this also means we may not have your IP address in the whitelist at the time of event if you are just connecting for the evening.  Please feel free to send us an email or ticket with your IP to be added to the whitelist if you are experiencing any connectivity issues.

-- 2018/03/20 5:00 PM MST - Server IP's 66.85.147.93 and 66.85.147.85 continue to receive intermittent DDoS traffic.  We're currently tuning our firewall and blocking network CIDRs as they occur.

-- 2018/03/20 8:30 PM MST - Our data center provider has placed the 66.85.147.93 IP address on the DDoS scrubbing network again, this results in additional latency being noticable for shell environments and game ports.  For swiftblade customers, please feel free to use the IP address 66.85.147.92 to connect to your shell or game ports.  This is an alternate IP address assigned to the server system and is routed around the DDoS protection network.  At this time, DDoS traffic spikes appear to be mostly mitigated as the protection network and our firewalls work together, but traffic on .93 will appear slow as the network continues to mitigate.

-- 2018/03/21 2:30 PM MST - At this time, intermittent packet loss is still on going and may be experienced as additional DDoS continues to occur.  These events are generally infrequent and services have been restored.  As we continue to mitigate the threat, it is believed the source is familiar with our services.  Due to these circumstances we will be limiting our communication on mitigation methods.  A follow up will be posted when the activity ceases.  Until then, we appreciate everyones patience in the matter as we continue to make adjustments.

-- 2018/03/27 12:05 AM MST - After a week, events still continue to occur intermittently throughout specific hours of the day but have been drastically reduced.  These events can lead to some additional latency as the firewall drops malicious packets prior to the DDoS network route engaging.  We appreciate everyone's patience.  If your site is majorly impacted by events, we're offering migration to one of our other servers on a space available basis.

-- 2018/03/28 12:10 AM MST - As of this evening, DDoS traffic has increased in considerable magnitude.  This has resulted in the null routing of the specific IP addresses under load.  While this appears to be effective, some additional latency will be felt while those IP's are removed from our route and broadcast domain.

-- 2018/03/28 4:10 PM MST - As a result of the the increased DDoS traffic, we have null routed previous IP addresses associated with the server (Ending in .93 and .92).  These IP addresses are no longer associated with any account as of 03/21.  If you're experiencing issues connecting to a game port by IP address, please use the DNS domain name to resolve the address.   The IP routes will cease to function in a few hours as they're cleared from the routers.

-- 2018/05/10 - At this time no DDoS events have been detected for several days.  Service has been fully restored to all null-routed IP addresses that were affected by this event.  Firewall configurations have been returned to a normal state.

In an effort to improve our customer support services and web presence, we will be scheduling an extended maintenance window, Monday, March 5th 2018 from 7:00 a.m. through Tuesday, March 6th 2018 to 10:00 p.m. Mountain Standard Time (MST).

During this maintenance schedule, we will be upgrading our company’s web servers and mail servers for the thirdhosting.com domain in order to align with our commitment to provide the best support services we can deliver.  This maintenance event will not impact customer services directly and will not affect any active services on customer nodes.  Throughout this maintenance event, shortcut URLs such as maintenance.thirdhosting.com and vpsmanage.thirdhosting.com may be intermittently unavailable while the site redirection configs are migrated.  Customers can continue to manage their services through our Client Support Area without impact.  During migration of the primary MX Mail Servers for the thirdhosting.com domain, support and sales ticket imports will be delayed upwards of an hour as the MX DNS records are adjusted.  It is recommended to login to the Client Support Area and submit a support ticket directly via the support form if assistance is required during this maintenance event.

We appreciate your patience and understanding as we continue to work to improve our support systems and welcome any feedback you may have.  If you have any questions or concerns regarding this maintenance schedule, please contact our Support Department at any time.

Sincerely,

Third Generation Internet Services

In an effort to improve our customer support services we will be scheduling a maintenance event for Monday, June 26th 2017 from 5:00 a.m. to 1:00 p.m. Mountain Standard Time (MST).

Through an extensive review of our company earlier this year and to align with our commitment to provide the best support services we can deliver to our customers, we will be taking efforts to upgrade our customer billing panel, support systems and ticketing systems with the latest offering provided by our software vendor.  This upgrade will require the migration of our billing site to our latest software build and new IP where we can deliver our service in the full confidence of our latest in a high-performance and secure software platform.

This maintenance window is currently an extended schedule with an estimated timeframe of 8 hours, but maintenance may extend further due to geographical DNS replications.  During this 8 hour maintenance window, our ticketing, invoicing and billing systems will currently be unavailable.  For immediate support assistance during this event, customers are recommended to email support services directly at support at thirdhosting.com.  This maintenance action will not affect any active services on customer nodes and no customer outage is expected.  We appreciate your patience and understanding as we continue to work to improve our support systems and welcome any feedback you may have.  If you have any questions or concerns regarding this maintenance schedule, please contact our Support Department at any time.

Sincerely,
Third Generation Internet Services

06-26-2017 11:45 MST - Migration of our billing software and support site has been completed.  This maintenance window is being extended however while we review new configuration options supplied by our vendor as well reviewing possible upgrades to theme support for the client area.  No major interruptions to support services are expected.  At this time we're also reviewing an issue with the order form not functioning due to out of date theme controls.

Datacenter engineers will be performing network maintenance on Thursday, June 22nd from 5:00 to 6:00 a.m. Eastern Daylight Time (EDT) which may affect public connectivity for a brief duration.  During this one-hour maintenance window, engineers will be performing scheduled maintenance to upgrade site routers.  Due to the software issues of the June 6th network outage, to improve services network engineers will be replacing routers at BHS-3.  This maintenance may result in a brief network disruption or periods of increased latency as traffic is rerouted.  Our provider has made all necessary adjustments to minimize any impact to service.  This maintenance window only affects services located at the OVH data center and our East Coast based services.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

In preparation of an upcoming account migration, technicians will be performing database maintenance on the Goldmoon cPanel web server Monday, June 19th 2017 from 8:00 to 10:00 a.m. Mountain Standard Time (MST).  During this maintenance window, system administrators will be restarting the MySQL service on the Goldmoon server and launching the service in a maintenance mode to review the integrity and consistency of customer databases that are currently using InnoDB tables.  Throughout this maintenance window services dependent upon the databases may be temporarily unavailable while system administrators rebuild the InnoDB tables and log cache to improve the reliability and consistency of the service.  Although this maintenance window has been scheduled for 2 hours, the databases located on the Goldmoon server are minimal and the expected interruption is anticipated at approximately 15 minutes while database caches are dumped and re-imported.

We appreciate your patience and understanding as we work to improve our service reliability.  If you have any questions or concerns regarding our maintenance schedule, please feel free to contact Customer Support at any time.

Sincerely,
Third Generation Internet Services

6-19-2017 8:00: This maintenance is now in progress.  Technicians are now reviewing databases and checking the integrity of tables.
6-19-2017 9:40: This maintenance window is now closed.  All maintenance has been successfully completed.  All database issues have been identified and repaired.

Our next scheduled maintenance will take place on Monday, May 22nd, 2017 from 1:00 a.m. to 2:00 a.m. (EDT) Eastern Daylight Time.

This maintenance will affect VPS customers located at the BHS datacenter and customers of the Goldmoon hosting server.  During this scheduled maintenance event we will be checking the performance, reliability and current state of the system hardware due to a suspected raid array degradation.  To accomplish this we will be migrating customer servers to a new upgraded hardware node.  A service interruption of up to 20 minutes is anticipated throughout this 1 hour maintenance window as we live migrate customer systems and hosting servers to the new hardware.  A brief interruption of up to twenty minutes is anticipated as we relocate existing IPs and re-establish connectivity.

There is no further action is necessary for customers for this event.  All customer data has already been synced in preparation of the relocation and customer services will be migrated in a live state.

We appreciate your patience and understanding as we continue to work to improve our services and server reliability.  If you have any questions or concerns regarding this maintenance action, please contact our Support Department.

Best Regards,
Third Generation Internet Services
support@thirdhosting.com

2017-05-22 01:05 EDT - This scheduled maintenance event is now in progress.  Further updates will be noted here.
2017-05-22 01:55 EDT - This maintenance window is being extended to 3:00 am EDT.
2017-05-22 02:35 EDT - Scheduled server moves were successful with the exception of a live migration for customers of server Goldmoon.  We're currently in the process of restarting all background game processes manually.
2017-05-22 03:00 EDT - This maintenance has been completed.  All game processes on goldmoon have been restarted.

Services Affected:  Riverwind, Nightshade, VPS Cluster Control Panel and VPS Services at Phoenix.

We are currently investigating issues across multiple virtual servers with service failure and database crash.

These services affect the srv04ssphzaz hardware node as well as Riverwind and Nightshade hosting servers.

We apologize for the inconvience and appreciate everyone's patience as we work to restore services.
Email for support has been routed off to a backup MX host in the mean time as we work to reply to support requests.
We will restore any background processes that were running prior to issue.
Service credits will be issues for the outage, a follow-up email will be released once outage event has been resolved.

2017-05-15 20:50 MST - An emergency reboot of the hardware node is necessary.  All services on across the node are experiencing issues and the database problems are belived to be due to disk cache issues.
2017-05-15 21:19 MST - The server has failed to restart from reboot and we're currently investigating the cause.  At this time no ETR.
2017-05-16 01:30 MST - We're currently reviewing options of migration of customer data to new hardware.  The server is experiencing hardware issues at this time.
2017-05-16 04:00 MST - New hardware is being procured and setup.  Data backup off of the previous server has begun to be used for restoration.
2017-06-16 06:50 MST - Data restoration is at approximiately 25%.  Recovering the system's data from the raid disks has yielded in large amount of 'recovery' files that has the potential for issues.  As we're restoring servers, we are manually comparing customer data for missing or corrupted files.  Due to the data corruption, the last snapshot backup from earlier in the day was corrupted/missing data and we'll compare/restore against the previous daily.  This may yield is previously deleted files being restored.  These files can be safely deleted again.  This may be necessary to verify that customers have all necessary files for operation.
2017-06-16 11:45 MST - Data restoration still in progress. 
2017-06-16 12:30 MST - Unfortunately, after making a detailed review of the file system we're finding widespead file corruption randomly throughout files in 100,000s of inodes.  Most of the corruption does fortunately reside in operating system files and there is only sparse corruption throughout customer profiles.  After reviewing the data and options, we're unfortunately making the decision to restore the last known good backup from May 13.  We'll then work through each individual customer profile to try and determine if we can restore the live data to their profile.  This process will take significantly longer due to the manual checking process, but we believe it will yield a more consistent result.  
2017-06-16 14:20 MST - We have begun restoring services to customer VPS services.  At this time, the VPS Control Panel, HN cluster and customer VMs are back online.  Data was restored from the last backup from yesterday and all servers are publically responding to pings. A copy of your data prior to the raid crash is available upon request.  Due to the block corruption that we're seeing and the extensity of the corruption, it is recommended to check the file prior to use.
2017-06-16 14:30 MST - We are currently in the process of restoring services to the Nightshade and Riverwind web and game servers.  Container services have been brought online while we work to restore data. Soft ETA of about 6 to 8 hours as we work to relcoate recovery data from our other data center.
2017-06-16 18:30 MST - All DNS servers part of our DNS clusters for VPS and Hosting services that resided on the node have been restored at this time.  All records have been sync successfully.
2017-06-16 20:00 MST - To clarify, services hosted on the Riverwind and Nightshade servers are still down at this time.  All other services have been recovered, but data restoration is still in progress for these two servers.  Due to having to make a separate pull from the backup servers the restoration time of these servers has been extended.  ETA for restoration for all services on these server is 8 hours.  All game processes, websites and databases will be started and verified once these servers come online.
2017-06-16 21:30 MST - Services for our web hosting servers have been offline for 24 hours.  We are hard at work still restoring services.
2017-06-17 05:45 MST - Data has been restored to the Riverwind server.  In the process of bringing the server online we're unfortunately encountering an error with MySQL services and InnoDB database corruption still.  The database issue appears to unrelated and will need further restoration.  At this time, we are not unlocking customer services until fully restored.
2017-06-17 12:55 MST - Services for the Nightshade server have been fully restored.  All recent game databases that could be located have been started.  Database services for Nightshade have been recovered.
2017-06-17 13:00 MST - We're still working to restore the MySQL databases on the Riverwind server and currently at about 80% restoration.  Meanwhile all game databases that could be located have been started in preparation of returning the server to production ready status.  The server is not available for access at this time.
2017-06-17 15:30 MST - The MySQL database server has been restored on Riverwind.  All web services and service processes have been restored.  Server access has been re-granted.  If you have a game process and we were able to determine the correct folder, it has been started.  Any customer needing assistance is encouraged to submit a support request at this time.  

STATUS: All servers and services are back online.

Thank you everyone for your patience.  We will be following up with a detailed email in regards to the issues, how we plan to mitigate and improve in the future as well as making service credits as per our SLA/TOS in the following day.  We still have a bit of cleanup work to do and tuning for the new hardware.  These changes should have no/minimal impact during the process.  Any customer needing assistance at this time is encouraged to submit a support request.  A current copy of customer data off previous hardware is available upon request.  Given that the data was throughly corrupted, we have opted to not grant this data to the customer unless upon request.  The silent file corruption issue requires manual review of each file and difficult to detect issues with encoded or binary files.  For this reason we do not recommend attempting to restore this data in mass.  This data can be used however to recover file changes, profiles, or work that may have occured between May 14th and 15th.

Datacenter engineers will be performing network maintenance on Thursday, February 16th from 12:00 to 8:00 a.m. Mountain Standard Time (MST) which may affect public connectivity for a brief duration.  During this eight-hour maintenance window, engineers will be performing scheduled maintenance on the backened network in our Phoenix location.  The maintenance may result in up to three brief network disruptions for up to 5 minutes each.  This outage will only affect services located at the Phoenix NAP data center.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

System administrators will be performing emergency scheduled maintenance on Friday, August 5th from 2:00 a.m. to 6:00 a.m. (EDT).

Our monitoring systems have reported a hardware failure on node srv03 and the system is currently operating on redundant hardware in a degraded status.  Due to the service life of the hardware, our system administrators will be migrating customer services to a new srv05 hardware node.  This 4 hour maintenance event will result in a service interruption while IP fail-over schemes are migrated.  A brief service interruption of up to 1 hour for network connectivity is anticipated throughout this 4 hour maintenance window.  Throughout this event every effort will be made to migrate operational servers and services in an online status.  All system services will be suspended briefly and resumed from snapshot in their operational state.  Updates for this maintenance action will be posted to this notice at http://maintenance.thirdhosting.com.

We appreciate your patience and understanding as we continue to work to provide a reliable and fault tolerant service environment.  If you have any questions or concerns regarding this maintenance action, please feel free to contact our Support Department.

Best Regards,
Third Generation Internet Services

Update: 8/4/2016 @ 7:30 p.m. :: Premigration of data has been started.  No impact to customer performance or service is expected at this time.

Update: 8/5/2016 @ 3:15 a.m. :: System migration has begun.  Customer services will loose network connectivity briefly while the IP fail-over scheme is migrated.

Update: 8/5/2016 @ 3:50 a.m. :: All systems have been successfully migrated at this time.

As we continue to update our systems to provide our customers with the latest software and support, we will be scheduling an extended maintenance window for our support site on Thursday, July 7th 2016 from 17:00 to Friday, July 8th 2016 till 02:00 MST.

Systems administrators will be upgrading our primary web site, customer billing panel, support system and ticketing system with the latest offering provided by our vendor.  This latest change will require migration of our primary website to a new server and IP.  This maintenance window is currently scheduled with an estimated timeframe of 6 hours but may be extended as well as affected by geographical DNS replication.  For immediate support assistance during this window customers are recommended to email the support email address directly at support at thirdhosting.com.

This maintenance action will not affect any active services or processing on customer nodes and no customer outage is expected.  We appreciate your patience and understanding as we work to improve our customer support systems and welcome any feedback our customers may have.  If you have any questions or concerns regarding this maintenance schedule, please contact our Support Department at any time.

Sincerely,
Third Generation Internet Services

Customers originating on the East Coast who utilize ISPs with Level3-based backend routing (Comcast and Century Link) are currently experiencing up to a 50% packet loss reaching services at the Phoenix Datacenter. 

This outage affects the eastern United States regions for Level3 based customers.  Customers performing trace rouets will notice a significant packet loss at bar2.phoenix1.level.net.

Customers currently affected by their ISP can use a Proxy, VPN or other service to route around their ISP's routing issues.  US Based Open Proxy List: http://www.xroxy.com/proxy---US-ssl.htm

Datacenter engineers have been notified and are currently working with the ISP to mitigate the routing issue.

Sincerely,
Third Generation

Customers using Cox Communications ISP are reporting connectivity issues to services located at the Phoenix Datacenter.  Datacenter engineers have been notified but the issue originates with the customer's ISP routing.  

The outage affects the southwestern United States regions.  Customers performing trace routes will notice their last hop within Cox Communications as 68.105.31.41 before being routed to geographical routes outside of the United States.  The last hops of the trace route will show a flapping typically between the 202.97.50.74 and 202.97.50.73 addresses resulting in a failure to route service.

These addresses are external to our services, https://whois.arin.net/rest/net/NET-68-105-30-0-1/pft?s=68.105.31.41 and https://whois.arin.net/rest/net/NET-202-0-0-0-1/pft?s=202.97.50.77

Customers currently affected by their ISP can use a Proxy, VPN or other service to route around their ISP's routing issues.  US Based Open Proxy List: http://www.xroxy.com/proxy---US-ssl.htm

Regards,
Third Generation Internet Services

Update: Routing issues are reported to be between Cox and China Telecom.

Update: 3/23/2016 @ 16:00 MST - Emerngecy maintenance wil be conducted to re-route BGP traffic from China Telecom to Level3 @ 1 AM MST 3/24/2016.  This maintenance will result in a brief disconnect for customers of Cox Cable Communications ISP.

Technicians will be performing software upgrades on Saturday, February 20th 2016 from 9:00 to 11:00 p.m. Mountain Standard Time (MST) which may affect public connectivity for a brief duration for IMAP and POP3 services. During this two-hour maintenance window we will be migrating mail server support from Courier to Dovecot in support of end-of-life announcements for WHM 54.0 and later versions. This maintenance may result in a brief service interruption for IMAP and POP3 services only for up to 5 minutes. This outage only affects services located on cPanel production servers.

We appreciate your patience and understanding as we continue to work to improve our software baseline and security posture. If you have any questions, concerns or feedback regarding our maintenance schedule, please feel free to contact Customer Support at any time.

Sincerely,
Third Generation Internet Services

Update 23:05 MST - Mail Server migration has been completed and SSL update completed successfully.  Please report any issues if your mail clients or mobile clients are experiencing any disruptions.

Datacenter engineers will be performing network maintenance on Wednesday, February 10th from 1:00 to 5:00 a.m. Mountain Standard Time (MST) which may affect public connectivity for a brief duration.  During this four-hour maintenance window engineers will be upgrading firmware on the access routers.  The maintenance may result in a brief network disruption for up to 5 minutes.  This outage will only affect services located at the Phoenix NAP data center.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Our system administrators will be performing an emergency maintenance action for the srv04ssphxaz.thirdhosting.com node on Monday, January 25th, 2016 from 7:00 p.m. to 8:00 p.m. (MST) Mountain Standard Time. This is a short notice emergency scheduled maintenance action to determine the hardware's integrity.

We're currently investigating multiple reports of intermittent packet loss for inbound and outbound traffic. Network boundary traffic has been ruled out at this time. System administrators will be performing a system restart and hardware testing to isolate the cause to determine if hardware replacement will be necessary.  The srv04ssphxaz.thirdhosting.com reboot affects all underlining customer VE servers as well as cPanel services for riverwind.thirdhosting.com and nightshade.thirdhosting.com. While this maintenance action is scheduled for approximately 1 hour, a brief service interruption of approximately 15 minutes is expected while services restart. We highly encourage customers operating full time processes and databases to safely shutdown their processes prior to the scheduled server event.  If assistance is required to execute any full-time processes after the restart please submit a support ticket at any time and we'll be glad to assist.

We appreciate your patience and understanding as we work to verify the hardware state and the condition of our network devices.  If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Update 1/25/2016 20:10 - Resetting of the network adapter and network device was not successful and we're currently reviewing our options with our provider. 

Update 1/25/2016 22:45 - Our provider will be disconnecting our public network interface for physical hardware replacement.  This will result in a brief disruption for up to 5 minutes.  We apologies for this inconvenience. 

Update 1/26/2016 11:00 - We are still experiencing a 6% packet loss for inbound and outbound traffic and continue to work with out provider for resolution.  If any assistance is required to startup processes please feel free to request assistance.

Update 1/26/2016 15:30 - Our provider has made adjustments to our network route which has appeared to have stabilized our traffic inbound and outbound.  Packet replies are currently responding at 100% and the previously noticeable lag and latency is no longer evident.  We will continue to monitor the situation to verify the resolution is permanent.  At this time all services are fully operational.  Customers experiencing issues connecting to their services or starting game processes please feel free to request assistance.

A network outage to the BHS data center resulting in large amounts of packet loss and latency have been reported by our monitoring systems.

The BHS data center is currently home to the srvnode03 VPS VM cluster and goldmoon mud/web services.  For any connectivity issues please refer to the updates on this notice for up-to-date status.

At this time our service provider has reported a network outage affecting several WAN links between Newark and Beauharnois, Montreal.  Remaining providers continue to service the uplink, however remaining circuits are being saturated due to failover.

UPDATE: (From our provider) The cut is located 108 km from Montreal (fiber that passes through the north of the lake), our technicians follow the path from BHS to find the cause of the fault (probably a fiber cut at approx. 9km from the datacentre).

UPDATE: Supplier is dispatching a team to evaluate the damage.  ETA expected soon.

UPDATE: Supplier is working to repair the damage.  In the mean time our provider is implementing 3 new eastern network routes that were scheduled in the upcomming days early.  2 out of 3 routes have been provisioned by the provider and supplier and network saturation is balancing.

UPDATE: As of 15:00 MST our monitoring systems have reported all systems back online.  Packet loss has returned to 0% however states of intermittent connectivity are being reported in short bursts as our data center provider works to establish additional uplink connectivity and the supplier resolve primary issues.

UPDATE: (From our provider) The repair of the cable by the western route is complete. All cutted links are now up (bhs <> chicago, bhs <> montreal, internal network to Europe and vrack). Traffic rerouted from peering at the Bell / Videotron was standardized to back to normal performance.  The new fiber to the new Montreal East road had to be put into production the same week. Bad timing. Although not planned and carried out entirely in the urgency of the situation, its production has allowed us to avoid several hours of extra failures and to permanently enhance the capacity to Newark. In the coming days we will migrate some other links to this link to secure the capacity to Chicago, Montreal and inter-data center network to Europe. We are waiting soon a 3rd other road from BHS to Newark wich will allow us to more securise the connectivity of BHS DC. We present again our apologies for this incident.

At this time the network has been stabled and all services fully restored.

We like to thank all customers experiencing issues for their patience.  If any questions or customers still experiencing issues please feel free to contact support.

This outage will be credited to affected customers as per our SLA: http://www.thirdhosting.com/third_generation/legal/service_level_agreement/

Thank You,
Third Generation Internet Services

Datacenter engineers will be performing network maintenance on Wednesday, September 23rd from 1:00 to 5:00 a.m. Mountain Standard Time (MST) which may affect public connectivity for a brief duration.  During this four-hour maintenance window engineers will be upgrading routing codes.  The maintenance may result in a brief network disruption for up to 10 minutes.  This outage will only affect services located at the Phoenix NAP data center.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Datacenter engineers will be performing network maintenance on Thursday, August 20th from 1:00 to 5:00 a.m. Mountain Standard Time (MST) which may affect public connectivity for a brief duration.  During this four-hour maintenance window engineers will be migrating our uplink interface to a new more robust network infrastructure device.  The maintenance may result in a brief network disruption for up to 10 minutes.  This outage will only affect services located at the Phoenix NAP data center.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Packet loss and higher than normal packet latency has been reported for servers in the Phoenix NAP Datacenter.  This includes hardware nodes srv04 and servers, Riverwind and Nightshade.

Network technicians are currently working to narrow the issue however the issue does not appear to be persistent and is midly intermittent at this time.  MTR is reporting less than a 0.005% packet loss.

Sincerely,
Third Generation

We have received several reports of intermittent connectivity for several various customers from different ISPs connecting to services at the Phoenix NAP datacenter, particularly the Riverwind, Nightshade servers and some VPS services hosted on srv04.  At this time we have been informed by our provider that they are currently mitigating a large scale DDoS attack that is currently targeting multiple US based data centers and external carrier routing services.  Services are currently up and operational, however they are experiencing periods of intermittent packet loss affecting particular ISPs as external carrier services are currently affected.  We will post any updates on the situation as we receive them.

As always, we thank our customers for their patience as our data center provider works to mitigate this issue and provide us the best service possible.

Best Regards,

Third Generation Internet Services

Update: 3/29/2015 2:25 PM MST - Networking teams are actively working on routing around the external carriers with issues, and attempt to create secondary mitigation functionality for these network attacks. Update will be sent as new information comes available.

Update: 3/29/2015 4:45 PM MST - Data center has reported that the second level of mitigation was put into place.  After an hour and half of monitoring our network connectivity issues appear to have subsided.

Datacenter engineers will be performing emergency maintenance on Tuesday, January 13th from 12:00 to 3:00 a.m. Mountain Standard Time (MST) which may affect public connectivity for a brief duration.  During this three-hour maintenance window, engineers will be restoring interface connectivity to redudant edge devices.  The maintenance may result in a brief network disruption for up to 5 minutes.  This outage will only affect services located at the Phoenix NAP data center.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Our next scheduled maintenance will take place on Monday, November 24th, 2014 from 2:00 a.m. to 4:00 a.m. (MST) Mountain Standard Time.

During this scheduled maintenance event we will be applying required software updates, making file system performance improvements and restarting the hardware node.  A service interruption of up to 1 hour is anticipated throughout this 2 hour maintenance window as upgrades and testing are completed.

Due to the circumstances of this maintenance event it will be necessary to restart all services.  We ask for customers that run background services such as game servers, daemons and other processes terminate these services prior to the maintenance event to ensure their data integrity.  There is no further action necessary for web hosting customers.

We appreciate your patience and understanding as we continue to work to improve our services and server performances.  If you have any questions or concerns regarding this maintenance action, please contact our Support Department.

Best Regards,
Third Generation Internet Services
support@thirdhosting.com

Update: Maintenance has been completed successfully.

Network support for BHS1 data center has been notified of ongoing network issues.

Currently services hosted in the BHS1 datacenter are experiencing a 10% packet loss and route flapping.  Network technicians are currently reviewing the issue.

Datacenter engineers will be performing maintenance on Thursday, September 25th from 2:00 to 5:00 a.m. Mountain Standard Time (MST) which may affect public connectivity for a brief duration.  During this three-hour maintenance window, engineers will be performing table and code upgrades for core routing devices.  The upgrade may result in a brief network disruption for up to 10 minutes.  This outage will only affect services located at the Phoenix NAP data center.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Our next scheduled maintenance will take place on Friday, August 8th, 2014 from 4:00 a.m. to 6:00 a.m. (MST) Mountain Standard Time.

During this scheduled maintenance event we will be applying required software and kernel updates and restarting the hardware node to comply with our quarterly maintenance schedule.  A service interruption of up to 1 hour is anticipated throughout this 2 hour maintenance window as upgrades and testing are completed.

Due to the circumstances of this maintenance event it will be necessary to shut down all services.  We ask for customers that run background services such as game servers, daemons and other processes terminate these services prior to the maintenance event to ensure data integrity.  There is no further action necessary for web hosting customers.

We appreciate your patience and understanding as we continue to work to improve our services and server performances.  If you have any questions or concerns regarding this maintenance action, please feel free to contact our Support Department.

Best Regards,
Third Generation Internet Services
support@thirdhosting.com

Update: This maintenance has been cancelled and will be rescheduled at a later time.

Our system administrators will be performing emergency scheduled maintenance on Sunday, June 29th from 12:00 a.m. to 6:00 a.m. (MST) Mountain Standard Time.

This 6 hour maintenance event may result in a brief service interruption for web services and database services only.  A service interruption of up to 5 - 10 minutes may be anticipated throughout this 6 hour maintenance window as software upgrades and testing is completed.  During this schedule, server software for cPanel server systems will be upgraded with the latest production versions available to improve their performance, security and stability.  Software migrations will occur within the environment transparently; however a brief interruption for service restarts is anticipated.

We appreciate your patience and understanding as we continue to work to improve our services and server performances.  If you have any questions or concerns regarding this maintenance action, please feel free to contact our Support Department.

Best Regards,
Third Generation Internet Services

Update 6-29-2014:  Software upgrades have been completed.  All cPanel shared environments have been upgraded to the latest production releases. MySQL 5.6, PHP 5.4, Apache 2.4.  Note to game customers, if you encounter an issue with your startup script or executing your codebase's binary due to libmysqlclient.so.16 - please perform a clean compile of your source code to regenerate the binary and then restart your codebase. 

Datacenter engineers will be performing maintenance on Monday, June 2nd from 1:00 to 6:00 a.m. Mountain Standard Time (MST) and Tuesday, June 3rd from 1:00 to 6:00 a.m. Mountain Standard Time (MST) which may affect public connectivity for a brief duration.  During this five-hour maintenance window, engineers will be upgrading hardware and migrating up-link ports.  The upgrade may result in a brief network disruption for up to 10 minutes.  This outage will only affect services located at the Phoenix NAP data center.

Datacenter engineers will be taking all necessary precautions to mitigate any prolonged connectivity issues during the maintenance windows.

Monday, June 2, 2014 , from 1:00 a.m. to 6:00 a.m. Mountain Standard Time (MST)
Tuesday, June 3, 2014, from 1:00 a.m. to 6:00 a.m. Mountain Standard Time (MST)

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Datacenter engineers will be performing maintenance on Thursday, May 15th from 1:00 to 5:00 a.m. Mountain Standard Time (MST) which may affect public connectivity for a brief duration.  During this four-hour maintenance window, engineers will be performing routing changes for network devices to support future network growth.  The upgrade may result in a brief network disruption for up to 10 minutes.  This outage will only affect services located at the Phoenix NAP data center.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Our system administrators will be performing scheduled maintenance on Saturday, February 22nd from 2:00 a.m. to 6:00 a.m. (MST) Mountain Standard Time.

This 4 hour maintenance event will result in a brief service interruption as system administrators will be performing necessary hardware upgrades.  A service interruption of up to 1 hour is anticipated throughout this 4 hour maintenance window as hardware upgrades and testing are completed.  During this schedule servers will be upgraded with additional system resources to improve their future performance and stability.

Due to the circumstances of the maintenance event it will be necessary to shut down all services.  We ask that customers that run background services such as game servers, daemons and other processes terminate these services prior to the maintenance event to ensure data integrity.  There is no further action necessary for web hosting customers.

We appreciate your patience and understanding as we continue to work to improve our services and server performances.  If you have any questions or concerns regarding this maintenance action, please feel free to contact our Support Department.

Best Regards,
Third Generation Internet Services
support@thirdhosting.com

UPDATE: MAINTENANCE FOR RIVERWIND HAS BEEN EXTENDED - ALL OTHER SERVERS ARE BACK ONLINE AT THIS TIME.

Update: We are currently waiting action from our data center provider.  Migration for the riverwind server is complete but experienced complications due to numerous disk errors.  These disks were replaced during the hardware upgrade, however we need to confirm integrity of all files that were transferred before restarting the riverwind node.  Estimate for full operation is 7:30 - 8:00 am MST.

Update: Waiting on hardware reboot request from data center.

Update: File verification in progress.  Thank you everyone for your patience.

Update:  What we have determined is a few directories have corrupted sectors causing the server to lockup.  We're having difficulty moving the data without agitating the issue further.  The data is being migrated to a new hardware node and we should have everything over and operational shortly.

Update: Feb-22 1:30 PM MST - Thank you everyone again for your patience.  The Riverwind server is back online and operational.  The system drives on the hardware node were experiencing corrupted and unreadable sectors which was resulting in system halts.  The directories had to be scanned and user data recovered and migrated to the new hardware node.  We had to patch the missing directories and files with yesterday's system snapshots.  Actual user home directories do not appear to have been affected by these unreadable sectors and user data should be current and consistent.  One of the primary goals of this maintenance was to upgrade and migrate to new hardware to prevent future stability issues.  The disks began to receive early warnings from SMART earlier in the week.  Unfortunately the system directories were damaged more than expected resulting in numerous issues.  The server is back online and homed with new resources and hardware.

Our system administrators will be performing scheduled maintenance on Saturday, February 22nd from 2:00 a.m. to 6:00 a.m. (MST) Mountain Standard Time.

This 4 hour maintenance event will result in a brief service interruption as system administrators will be performing necessary hardware upgrades.  A service interruption of up to 1 hour is anticipated throughout this 4 hour maintenance window as hardware upgrades and testing are completed.  During this schedule servers will be upgraded with additional system resources to improve their future performance and stability.

Due to the circumstances of the maintenance event it will be necessary to shut down all services.  We ask that customers that run background services such as game servers, daemons and other processes terminate these services prior to the maintenance event to ensure data integrity.  There is no further action necessary for web hosting customers.

We appreciate your patience and understanding as we continue to work to improve our services and server performances.  If you have any questions or concerns regarding this maintenance action, please feel free to contact our Support Department.

Best Regards,
Third Generation Internet Services
support@thirdhosting.com

Update: Maintenance progress is complete.

We will be migrating the primary public-shared address for riverwind.thirdhosting.com server on Jan 12, 2014 through Jan 29.

The primary IP 66.85.154.235 will change to 66.85.147.85 in preparation of future hardware maintenance and upgrades.  The new IP address will be added to the server and the former IP will be retained for the time being.  The former IP will remain active until the completion of a future hardware upgrade to be announced.  Customer accounts currently associated with this public IP will slowly be migrated to the new shared primary and their DNS records will be transparently updated on our DNS cluster.  Customers will be notified prior to any hardware upgrades so that they may update any external DNS standalone A records to the new IP address before the former IP is deactivated.  At this time there is no action needed by the customer and no outage of connectivity or loss of service expected.

Sincerely,
Third Generation

Update: This maintenance has been extended till Jan 29, 2014 to compensate for the IP migration tools and geographic IP replication.  The server's primary IP has been updated however additional changes are necessary to verify transparent migration for software services.

Datacenter engineers will be performing a planned scheduled maintenance on Thursday, January 23rd from 1:00 a.m. to 4:00 a.m. (MST) Mountain Standard Time.  During this three-hour maintenance window a brief service interruption is anticipated for 10-15 minutes while engineers perform a code upgrade to the core routers.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability.  If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department.

Regards,
Third Generation Internet Services

Latency issues have been reported for Hardware Node - srv03ovhmoca.  This hardware node supports the VPS cluster in OVH-MO-CA and also houses the Goldmoon web server. 

The latency issues are currently being reviewed and the issue has been narrowed to a disk array issue that administrators are currently working to resolve.

Sincerely,
Third Generation

Update:  The repair process to the partition with issue has started.  You may experience additional latency while this process runs in idle IO.

Datacenter engineers will be performing planned maintenance on Thursday, January 9th from 1:00 to 3:00 a.m. Mountain Standard Time (MST) which may affect public connectivity for a brief duration.  During this two-hour maintenance window, engineers will be performing a software upgrade to the core routers.  During this maintenance schedule you may experience intermittent connectivity while service is performed.  However, no extended network disruptions are anticipated.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Datacenter engineers will be performing maintenance on Tuesday, August 20th from 1:00 to 5:00 a.m. Mountain Standard Time (MST) which may affect public connectivity for a brief duration.  During this four-hour maintenance window, engineers will be performing code upgrades for network devices to support network growth.  The upgrade may result in a brief network disruption for up to 15 minutes.  This outage will only affect services located at the Phoenix NAP datacenter.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Datacenter engineers will be performing maintenance on Thursday, April 11th from 1:00 to 4:00 a.m. Mountain Standard Time (MST) which may affect public connectivity for a brief duration.  During this three-hour maintenance window, engineers will be replacing hardware in one of the routing device modules.  The hardware replacement may result in a brief network disruption for up to 15 minutes.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Datacenter engineers will be performing maintenance on Thursday, March 21st from 1:00 to 5:00 a.m. Mountain Standard Time (MST) which may affect public connectivity for a brief duration.  During this four-hour maintenance window, engineers will be replacing hardware in one of the routing device modules.  The hardware replacement may result in a brief network disruption for up to 15 minutes.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Datacenter engineers will be performing maintenance on Friday and Saturday, March 1 and 2 from 2:00 to 4:00 a.m. Mountain Standard Time (MST) on both days which may affect public connectivity for a brief duration.  During this two-hour maintenance window, a patch will be applied to top-of-rack switches which will require a restart of the rack switch.  The restart of the switch will result in a 5 to 10 minute outage during the reboot event.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Scheduled Network Maintenance: 01/31/2013, 1AM to 4AM MST.

On Thursday, January 31st our network provider will be conducting a planned maintenance starting at 01:00 a.m. Mountain Standard Time (MST) and concluding at 4:00 a.m. MST.  During this three-hour window network engineers will be addressing a detected hardware issue within one of the core routers.  While this upgrade is occurring, a brief network disruption is anticipated.  A network disruption of approximately 45 minutes is anticipated however precautions to mitigate any prolonged disruptions have been taken.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems and maintain stability. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Scheduled (Backend) Network Maintenance 11/28/2012, 1AM to 5AM MST

A planned maintenance action for our datacenter's network will be taking place on Wednesday, November 28, 2012 from 1:00 a.m. Mountain Standard Time (MST) to 5:00 a.m. MST. During this 4 hour window engineers will be adding capacity to the Core Backend network infrastructure, and no configuration changes are needed on your end. Note that while this maintenance does not affect Internet traffic, we do anticipate a brief outage on the backend network of no longer than 15 minutes.

The Network Engineering Team will be taking all necessary precautions to mitigate any prolonged disruptions, and if you have any questions, please feel free to contact support.

Dear Valued Customers,

Out next scheduled maintenance will take place on Wednesday, November 7, 2012 starting at 11:00 p.m. Mountain Standard Time (MST) and will conclude November 8, 2012 at 2:00 a.m.

During this maintenance action we will be restarting the hardware node to comply with our quarterly update schedule and as well conducting a needed file system repair for one of the disk partitions.  While this maintenance window is scheduled for three hours, we are anticipating no longer than 90 minutes of scheduled downtime.  Throughout this maintenance window connectivity to your public services will be unavailable.

We appreciate your patience and understanding as we continue to work to improve our systems.  If you have any questions or concerns please feel free to contact our Support Department at any time.

Sincerely,
Third Generation Internet Services

UPDATE: The riverwind server is currently back online. All services have been restarted.

The hardware node which riverwind.thirdhosting.com currently resides on went down at 03:05 MST, local techicians are currently reviewing the situation.

Start Time: 11/01/2012 03:05 MST
End Time: 11/01/2012 05:45 MST

Third Generation SLA - http://www.thirdhosting.com/third_generation/legal/service_level_agreement/

Scheduled Network Maintenance: 06/13/2012, 1AM to 4AM MST.

On Wednesday, June 13th our provider's Network Engineers will be conducting a planned maintenance starting at 01:00 a.m. Mountain Standard Time (MST) and concluding at 4:00 a.m. MST.  During this three-hour window engineers will be upgrading access routers to provide for greater capacity and stability.  While this upgrade is occuring, a brief network disruption is anticipated.  A network disruption of approximately 15 minutes is anticipated however precautions to mitigate any prolonged disruptions have been taken.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

On Thursday, April 26 we will be conducting a planned maintenance starting at 11:00 p.m. Mountain Standard Time (MST) and concluding at 1:00 a.m. MST on April 27. During this two-hour window, we will be upgrading code on one of our core routers to accommodate for growing capacity needs, per our ongoing efforts to provide the best network possible. This maintenance should not have any significant impact on our network, but it is possible you may experience some intermittent connectivity during the maintenance. However, our networking team will be taking a number of steps to mitigate any prolonged disruptions.

Scheduled Server and Network Maintenance 02/15/2012, 10PM to 11PM MST

Dear Valued Customers,

Our next scheduled maintenance will take place on Wednesday, February 15, 2012 starting at 10:00 p.m. Mountain Standard Time (MST). This maintenance action will conclude our long awaited network upgrade and will require approximately 30 minutes to 1 hour to complete.

During this maintenance action our provider will be concluding a large-scale network upgrade. This upgrade is a major step to bring to you the best network performance possible and will provide a fully redundant and scalable network for future growth.  Throughout this maintenance window, connectivity to your services will be unavailable as our servers will be powered down and relocated to a new server cabinet.  Our support services, main site, DNS and email routing services however will continue to be provided through redundant operation. The relocation of our servers to a new equipment rack is necessary to maintain our shared VLAN capabilities between servers.

We appreciate your patience and understanding as we continue to work to improve our systems. If you have any questions or concerns regarding this schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Below is an insert of the technical details regarding the upgrade from our provider, for those who are interested.

It is with great excitement that we share some additional news about our upcoming network upgrades. As you’re aware, we have been working diligently over the last few months to bring you a state-of-the-art, bullet proof network. And we’re now coming up on the last mile.

As mentioned in our previous newsletter, we gave our network team a goal of designing a bullet-proof network from top to bottom, and we can confidently say they accomplished that goal! We just have one last piece to finish up before we can consider this project 100% complete. Replacing the top-of -rack switches in each and every cabinet.

Every single layer of the network is getting replaced. The Edge and Core networks each have a redundant a pair of top-of-the-line carrier class Brocade XMRs. These routing monsters support up to 2 billion packets per second with a non-blocking capacity of 3.2 Tbps. We are also deploying multiple redundant pairs of 10Gbit aggregation switches and brand new top-of-rack switches (brocade FCXs). Each rack will be redundantly connected to the aggregation network by 2x10Gbit connections.

Once the migration is complete, you will no longer have to question the stability of your service with us, and can rest easy knowing your data will be supported by a bullet proof, state-of-the-art network.  As always, our NOC team is here to answer any questions you have 24x7x365.

Dear Valued Customers,

Our next scheduled maintenance will take place on Sunday December 18, 2011 starting at 11:00 p.m. Mountain Standard Time (MST). The projected completion time is 2:00 a.m. MST on December 19, 2011. Please note that during this maintenance window you may see intermittent connectivity issues until the maintenance is complete.

During this maintenance action our provider is in the process of upgrading and replacing their core routing devices to new and better hardware. This migration is another step to provide you with the best network performance possible and increase our network scalability, and redundancy. During the next few weeks you will be receiving maintenance notifications as they are slowly migrating to these new core routers. The migration process has been broken into smaller pieces to ensure they focus on only a few clients per maintenance to minimize any possible downtime.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

RESCHEDULED!

Dear Valued Customers,

Our datacenter engineers will be scheduling network maintenance for Thursday, December 11th 2011. The schedule will occur from 11:00PM to 2:00AM MST.

During this maintenance action engineers will be bringing online additional circuits from their existing carriers into the bandwidth blend for increased capacity and redundancy. They will also be replacing a primary core router with newer upgraded hardware to improve our network performance and connectivity reliability. During these maintenance windows you may notice some intermittent connectivity and packet loss while the router tables converge on all edge routers.

We appreciate your patience and understanding as the network engineers continue to work to improve our network systems. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

We will be scheduling a maintenance window for the riverwind server on Saturday, December 3rd 2011 from 1AM to 5AM CST.

After carefully considering the growth of our company over the past year and the requirements to continue to deliver our customers with a reliable environment and cost effective services, we have elected to migrate our hardware and services to the Phoenix NAP Datacenter.  We feel this change will provide the most benefit to our clients as well as our company in regards of more powerful hardware, a faster and more reliable networking environment and increased growth potential for the future.

During this maintenance schedule the riverwind server will be migrated to its respective new hardware node.  Due to the nature of this migration and the relocation to our new Datacenter, the server will be assigned new IP addresses.  This maintenance action will NOT require any special action on your part. Details for your new IP information can be located below.

We appreciate your patience and understanding as we continue to work to improve our network systems and support. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Customer Support Department at any time.

Sincerely,
Third Generation Internet Services

Your account’s new IP address on December 3rd: 66.85.154.235

Please note, if you have a domain name that is currently in use with our services that is registered with an external Registrar and is NOT currently assigned to our Name Servers (NS1.THIRDHOSTING.COM and NS2.THIRDHOSTING.COM), please update your DNS information on December 3rd to the above IP address replacing the old IP address of 208.43.220.232. Thank You.

Dear Customers,

As we continue to update our systems to provide our customers with industry's best software and support, we will be scheduling an extended maintenance window on Saturday, August 20th 2011 from 20:00 to Sunday, August 28th 2011 till 23:59.

Third Generation System Administrators will be upgrading our primary website, support site and billing services over an extended period. This maintenance action will not affect any running services or processes and no customer outage is expected.

We appreciate your patience and understanding as we work to improve our support systems and welcome any feedback our customers may have. If you have any questions or concerns regarding this maintenance schedule, please contact our Support Department at any time.

Start Time: 8/20/2011 20:00 CST
End Time: 8/28/2011 23:59 CST

* NON SERVICE IMPACT *

Target Services: Primary website, customer support sites and billing services.

Thank You,
Third Generation Support

We will be scheduling a maintenance reboot for srv01sldaltx.thirdhosting.com and riverwind.thirdhosting.com which will affect all web services, game services as well as all underlining VPS customers and control panel accesses. This maintenance window is scheduled for Friday, July 8th 2011 from 01:50 to 02:00 CST.

Start Time: 01:50 CST
End Time: 02:00 CST

We appreciate your patience and understand at this time and apologize in advance for any inconvience this maintenance window may cause.

Thank You,
Third Generation Internet Services Support

Network monitoring systems reported no connectivity to hardware node srv01sldaltx.thirdhosting.com @ 21:55 2011/06/27.

Network downtime investigated and reported. Server connectivity impacted all VE nodes for srv01sldaltx and riverwind

Server srv01sldaltx, riverwind and all VE nodes and services restored @ 22:15 2011/06/27.

Third Generation Service Level Agreement (SLA) Policy: http://www.thirdhosting.com/policy-sla.php

Our datacenter provider will be scheduling a maintenance window on Thursday, June 16th 2011 from 12AM to 5AM CDT.

System Engineers will be conducting maintenance on the aggregate switches on the DAL01 network. During this maintenance window, Engineers will be upgrading the firmware on these devices to increase security and stability.

The following services will be impacted during this maintenance:

- Public network connectivity for the DAL01 facility
- Public / Private connectivity to other SoftLayer facilities from the DAL01 facility

While this window is scheduled for 5 hours, actual down time should vary by customer, and should be much less than 5 hours. Individual upgrades and reboots are expected to take less than 10 minutes per affected device.

During this maintenance, Network Engineers will make every effort to ensure that impact to customer traffic is minimal; however, customers will see packet-loss, and increased latency.

We appreciate your patience and understanding as we work to improve our network systems and welcome any feedback you may have. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Technical Support Department at any time.

Due to several inconsitent service/software failures we will be scheduling an emergency maintenance reboot for srv01sldaltx.thirdhosting.com which affects all underlining VPS services as well as CPanel services for riverwind.thirdhosting.com customers. This maintenance window is scheduled for Monday, June 14th 2010 from 02:50 to 02:55 CST.

Start Time: 02:50 CST
End Time: 02:55 CST

Due to the nature of the restart, all processes will be shutdown. Please restart your processes once the reboot has been completed.

We appreciate your patience and understand at this time and apologize in advance for any inconvience this maintenance window may cause.

Thank You,
Third Generation Internet Services Support

As we continue to update our server systems to provide our customers with the industry's best software and support, we will be re-scheduling our maintenance window for Saturday, January 30th 2010 from 1AM to 5AM CST. We will be re-scheduling our maintenance window due to a large number of upgrades and patches that have not been tested for stable environment operation.

Third Generation's system administrators will be upgrading CPanel and WHM web services as well as firewall, security and spam-filtering technology services. We will also be updating our support site at this time as well. This upgrade will affect all web services, email services and FTP services on the riverwind.thirdhosting.com server. This maintenance will NOT affect any running game services or background processes.

While the maintenance window is scheduled for four hours, we are anticipating no longer than 120 minutes of scheduled downtime. As this maintenance only affects our public web services and email services, customers are not required to do anything before this maintenance window.

We appreciate your patience and understanding as we work to improve our network systems and welcome any feedback you may have. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Technical Support Department at any time.

Our datacenter provider will be scheduling an emergency maintenance window on Sunday, January 17th 2010 from 1AM to 2AM CST.

Start Time: 01:00 CST
End Time: 02:00 CST

System Engineers will be addressing an identified hardware issue on an aggregated switch which provides connectivity for multiple servers hosted by Third Generation. To prevent an unplanned outage resulting in significant downtime, Engineers will be replacing the upstream front end aggregate switch during a scheduled maintenance window. During this maintenance, customer servers will not be reachable on the public network. Private connectivity will NOT be effected by this maintenance (iSCSI, DNS, NAS, etc). While the maintenance window is set for 1 hour, we expect no longer than 15-20 minutes of downtime on the public network.

We appreciate your patience during this work and welcome any feedback.

Thank You,
Third Generation Support

Our datacenter provider will be scheduling an emergency maintenance window on Monday, November 30th 2009 from 2AM to 3AM CST.

Start Time: 02:00 CST (08:00 UTC)
End Time: 03:00 CST (09:00 UTC)

System Engineers will be addressing identified security vulnerabilities and will be upgrading and rebooting public network routers across the Dallas datacenter facility. Periodically throughout this maintenance, customers may notice a complete loss of connectivity to all public services. This maintenance schedule will affect all of our public-based services such as web services, game services and email services. While the upgrade duration is scheduled for 1 hour, we only expect between 15 - 20 minutes of downtime as the routers reload and fully converge.

We appreciate your patience and understanding as we work to improve our network systems and welcome any feedback you may have. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Technical Support Department at any time.

As we continue to update our server systems to provide our customers with the industry's best software and support, we will be scheduling a maintenance window on Sunday, October 18th 2009 from 1AM to 4AM CDT.

Third Generation's system administrators will be upgrading CPanel and WHM web services, email services, PHP user services and bandwidth tracking services. This upgrade will affect all web services, email services and possibly FTP services on the riverwind.thirdhosting.com server. This maintenance will not affect any running game services or processes.

While the maintenance window is scheduled for three hours, we are anticipating no longer than 90 minutes of scheduled downtime. As this maintenance only affects our public web services and email services, customers are not required to do anything before this maintenance window.

We appreciate your patience and understanding as we work to improve our network systems and welcome any feedback you may have. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Technical Support Department at any time.

Network monitoring systems reported no connectivity to hardware node srv01sldaltx.thirdhosting.com @ 23:33 2009/06/02 CDT.

Network downtime investigated and determined as an internal denial of service attack. Threat application and account removed.

Server connectivity impacted all VE nodes for srv01sldaltx.

Server srv01sldaltx and all VE nodes and services restored @ 01:21 2009/06/03 CDT.

Third Generation Service Level Agreement (SLA) Policy: http://www.thirdhosting.com/policy-sla.php

Network monitoring systems reported no connectivity to hardware node srv01sldaltx.thirdhosting.com @ 05:30 2009/03/21.

Network downtime investigated and reported. Server connectivity impacted all VE nodes for srv01sldaltx.

Server srv01sldaltx and all VE nodes and services restored @ 06:08 2009/03/21.

Third Generation Service Level Agreement (SLA) Policy: http://www.thirdhosting.com/policy-sla.php

As we continue to update our server systems to provide our customers with the industry’s best software and support, we will be scheduling a maintenance window on Monday, March 23rd 2009 from 1AM to 3AM CDT.

Third Generation’s system administrators will be upgrading all operating system kernels with our latest virtualization technology. These upgrades and modifications will require the restart of the srv01sldaltx.thirdhosting.com hardware-node affecting all riverwind.thirdhosting.com customers.

While the maintenance window is scheduled for two hours, we are anticipating no longer than 30 minutes of downtime. Customers are advised to shutdown any scripts or processes prior to maintenance as a sudden loss in connectivity could potentially cause file corruption or loss of data.

We appreciate your patience and understanding as we work to improve our network systems and welcome any feedback you may have. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Technical Support Department at any time.

As we continue to update our server systems to provide our customers with the industry's best software and support, we will be scheduling a maintenance window on Tuesday, March 24th 2009 from 1AM to 4AM CDT.

Third Generation's system administrators will be upgrading Cpanel and WHM web services as well as spam-filtering technology services. This upgrade will affect all web services, email services and FTP services on the riverwind.thirdhosting.com server. This maintenance will not affect any running game services or processes.

While the maintenance window is scheduled for three hours, we are anticipating no longer than 60 minutes of scheduled downtime. As this maintenance only affects our public web services and email services, customers are not required to do anything before this maintenance window.

We appreciate your patience and understanding as we work to improve our network systems and welcome any feedback you may have. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Technical Support Department at any time.

Our datacenter provider will be scheduling a maintenance window on Saturday, March 28th 2009 from 3AM to 5AM CDT.

System Engineers will be upgrading the IOS and rebooting backend/private network routers across the Dallas network facility. Periodically throughout this maintenance, customers will not have access to our remote backup services. This maintenance schedule will not affect our public-based services such as web services and will be transparent to all customers.

We appreciate your patience and understanding as we work to improve our network systems and welcome any feedback you may have. If you have any questions or concerns regarding this maintenance schedule, please feel free to contact our Technical Support Department at any time.