Basilisk extended downtime 08/09/2017

[Vlada]

Basilisk extended downtime

Posted August 9th 2017
The SWGEmu Staff

Basilisk extended downtime


While doing a routine backup before the merge one of the HDD's died, unfortunately that was the second one we lost in a matter of weeks. Right now TheAnswer and the rest of development staff are discussing the options and trying to figure out how best to try to recover all Basilisks data. Of course, with two HDD's gone, there is a strong possibility that what they recover is too corrupt and unusable or they may not even manage to recover all the necessary data to restore Basilisk in its original state before merging it with Publish 9 code and we will be forced to do a full database wipe. Of course its still early to give any predictions but one thing is for sure, we are not at a point where we can definitely say that we will be able to move forward one way or the other.


We apologize for the extended downtime that is probably going to last for days.


No matter the the outcome, we thank you for your dedication and your continuous support. Of course we will keep you posted as things progress. As soon as we have any info, you will have it too.

Update 6:

Alright folks, here's our current status. From what we can tell, our database is in good enough shape to proceed with another Publish 9 merge attempt. There's still a lot of potential for corruption or database issues to crop up that we haven't seen, so we're going to be taking some extra precautions in case those issues do pop up. Because of this, when the server is taken down, expect an extended downtime before it comes back up with the new publish.

Our current schedule is looking like Basilisk will be taken down approximately 12 hours from now. That timeframe may change between now and then. Once it's taken down, we'll start the process of backing up the database again. With the size of the database, this alone will take a fair amount of time, so we're not going to give an ETA on when it will be back up. Once Basilisk does come up, the server's new navmesh system will begin building meshes for all static and player cities as well as POI's, which will also take some time. Expect the server to remain locked for a while after it comes up.

TLDR: When we take the server down, expect the update process to take a while to complete, especially with the issues we ran into over the past few days. We'll get the server up and running with Pub9 as quickly as we can.

Update 5:

Its a 48 hour test, so its ongoing. But, no bad news so far.

UPDATE 4:

Alright folks, update time. We've managed a recovery of the database and we've got everything copied over to our new SSD's. We're letting Bas run for the weekend as a stress test on the database to ensure there is no corruption that didn't pop up initially. This means that for this weekend, Basilisk is going to remain on Publish 8. We're looking to give the Publish 9 merge another shot on Monday, assuming all goes well over the next 48 hours. We'll update regarding Pub9 closer to Monday when we know for sure.

In the meantime, continue on Basilisk as you normally would. We'll be keeping an eye out on our side for any potential stability issues.

Keep in mind that we're still doing integrity checks on the database, if we run into any database corruption between now and then it's very likely that any progress you make this weekend will be lost when we revert to a backup. This is not guaranteed to occur, but be aware that it is definitely a possibility.

UPDATE 3:

We finished uploading 1.1TB of data and now we're going though it.

UPDATE 2:

To clarify a bit, Basilisk was up but still in the process of validating the integrity of most of the database objects. The server is currently running through our backup HDD's, which is causing the server's loading to take a lot longer than it normally would on SSD's. We shut down the login server because any actions triggered on Bas will delay the database check, and it's going to take long enough as it is.

The good news is, the server didn't explode using the recovered database. There's no current bad news, but as of right now, that's the only good news. We still have to verify that all of the objects in the database are actually valid and load properly, and it will take time. The possibility of a wipe still remains if we find too much corrupted data. Time will tell, but in the meantime you'll see Basilisk go online and offline multiple times, but you won't be able to access it. Those still connected will not be able to reconnect when they disconnect, and in the interest of getting this testing done as quickly as possible, they should disconnect on their own anyway.

UPDATE:

About the loading status: We are attempting to load a test db instance to check for data integrity. This should give us an indication of whether we'll need to wipe or not. It's loading on the HDDs so it's slow and will take time. Thanks for the continued patience.

~The SWGEMu Staff