User avatar
By Cosmic Andy (Staff)
#9714
We have had an outage with our primary router this evening. As a short term solution i've put a fix in place to get the world backup and running.

Please log out of the backup world and log back into WA.

Some parts of WA are still not working such as the radio stream, web client & painters. These will be sorted on Sunday.

The world will need to go down a few times on Sunday to make restore normal service but this will be for a few short times no more than 5 to 10 minutes each.

I'll be IW later on Sunday to answer any questions and will issue something extra to make up for the downtime today.

Very sorry for issue and will see you all IW soon.
#10146
I would like to apologise for the outage to WA. Around 1:35 AM another DOS attack forced the router to over load and fail. At this point our server monitoring provider (NodePing) will send an SMS & Push notification to myself and Dimitri. For some unknown reason the notifications did not arrive and we were not aware of the outage until arriving to the office.

We have now configured 3rd party voice monitoring from site24x7.com & SMS momitoring from uptimerobot.com so in the event of an outage we have 3 providers who will notify us with the most important being from site24x7.com which will call us within 60 seconds of an outage. We have configured our phones to favourite this number allowing notification 24/7 regardless of the phones volume setting and enabled consistent notification which will keep calling until the call is answered by one of us.

Regarding the main issue of the router crashing during DOS. I've purchased a WatchGuard Firebox M370 Firewall to replace our custom pfsense firewall.

We will have this setup in the next 10 days.

In Q1 2020 we will work on the automatic failover from one network provider to another while still running WorldsAway (rather than switch to Summerland). This would benefit us when the network provider is down and not during an attack which would switch us over to Summerland.

Once again sorry for the downtime.