Results 1 to 15 of 828

Thread: Basilisk extended downtime 08/09/2017

Threaded View

  1. #1
    The Vlada Vlada's Avatar
    Join Date
    Mar 2007
    The Vlada

    Basilisk extended downtime 08/09/2017

    Basilisk extended downtime

    Posted August 9th 2017
    The SWGEmu Staff

    Basilisk extended downtime

    While doing a routine backup before the merge one of the HDD's died, unfortunately that was the second one we lost in a matter of weeks. Right now TheAnswer and the rest of development staff are discussing the options and trying to figure out how best to try to recover all Basilisks data. Of course, with two HDD's gone, there is a strong possibility that what they recover is too corrupt and unusable or they may not even manage to recover all the necessary data to restore Basilisk in its original state before merging it with Publish 9 code and we will be forced to do a full database wipe. Of course its still early to give any predictions but one thing is for sure, we are not at a point where we can definitely say that we will be able to move forward one way or the other.

    We apologize for the extended downtime that is probably going to last for days.

    No matter the the outcome, we thank you for your dedication and your continuous support. Of course we will keep you posted as things progress. As soon as we have any info, you will have it too.

    Update 6:

    Quote Originally Posted by Miztah View Post
    Alright folks, here's our current status. From what we can tell, our database is in good enough shape to proceed with another Publish 9 merge attempt. There's still a lot of potential for corruption or database issues to crop up that we haven't seen, so we're going to be taking some extra precautions in case those issues do pop up. Because of this, when the server is taken down, expect an extended downtime before it comes back up with the new publish.

    Quote Originally Posted by Miztah View Post
    Our current schedule is looking like Basilisk will be taken down approximately 12 hours from now. That timeframe may change between now and then. Once it's taken down, we'll start the process of backing up the database again. With the size of the database, this alone will take a fair amount of time, so we're not going to give an ETA on when it will be back up. Once Basilisk does come up, the server's new navmesh system will begin building meshes for all static and player cities as well as POI's, which will also take some time. Expect the server to remain locked for a while after it comes up.

    TLDR: When we take the server down, expect the update process to take a while to complete, especially with the issues we ran into over the past few days. We'll get the server up and running with Pub9 as quickly as we can.

    Update 5:

    Quote Originally Posted by TheAnswer
    Its a 48 hour test, so its ongoing. But, no bad news so far.
    UPDATE 4:

    Quote Originally Posted by Miztah View Post
    Alright folks, update time. We've managed a recovery of the database and we've got everything copied over to our new SSD's. We're letting Bas run for the weekend as a stress test on the database to ensure there is no corruption that didn't pop up initially. This means that for this weekend, Basilisk is going to remain on Publish 8. We're looking to give the Publish 9 merge another shot on Monday, assuming all goes well over the next 48 hours. We'll update regarding Pub9 closer to Monday when we know for sure.

    In the meantime, continue on Basilisk as you normally would. We'll be keeping an eye out on our side for any potential stability issues.

    Keep in mind that we're still doing integrity checks on the database, if we run into any database corruption between now and then it's very likely that any progress you make this weekend will be lost when we revert to a backup. This is not guaranteed to occur, but be aware that it is definitely a possibility.

    UPDATE 3:

    Quote Originally Posted by TheAnswer
    We finished uploading 1.1TB of data and now we're going though it.
    UPDATE 2:

    Quote Originally Posted by Miztah View Post
    To clarify a bit, Basilisk was up but still in the process of validating the integrity of most of the database objects. The server is currently running through our backup HDD's, which is causing the server's loading to take a lot longer than it normally would on SSD's. We shut down the login server because any actions triggered on Bas will delay the database check, and it's going to take long enough as it is.
    Quote Originally Posted by Miztah View Post
    The good news is, the server didn't explode using the recovered database. There's no current bad news, but as of right now, that's the only good news. We still have to verify that all of the objects in the database are actually valid and load properly, and it will take time. The possibility of a wipe still remains if we find too much corrupted data. Time will tell, but in the meantime you'll see Basilisk go online and offline multiple times, but you won't be able to access it. Those still connected will not be able to reconnect when they disconnect, and in the interest of getting this testing done as quickly as possible, they should disconnect on their own anyway.


    Quote Originally Posted by Ivojedi View Post
    About the loading status: We are attempting to load a test db instance to check for data integrity. This should give us an indication of whether we'll need to wipe or not. It's loading on the HDDs so it's slow and will take time. Thanks for the continued patience.

    ~The SWGEMu Staff
    Last edited by Lolindir; 08-14-2017 at 01:18 PM.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts