Basilisk extended downtime 08/09/2017

[ahdinko]

To me, a wipe would be preferable to getting the data recovered!

[papilove]

lol @ wanting a wipe , only way i'd be down for a wipe if they cant recover the data but if they can we have to keep going till suncrusher is ready

[nee2earth]

more from discord:

Here's the actual quote...

To clarify a bit, Basilisk is up but still in the process of validating the integrity of most of the database objects. The server is currently running through our backup HDD's, which is causing the server's loading to take a lot longer than it normally would on SSD's. We shut down the login server because any actions triggered on Bas will delay the database check, and it's going to take long enough as it is.

The good news is, the server didn't explode using the recovered database. There's no current bad news, but as of right now, that's the only good news. We still have to verify that all of the objects in the database are actually valid and load properly, and it will take time. The possibility of a wipe still remains if we find too much corrupted data. Time will tell, but in the meantime you'll see Basilisk go online and offline multiple times, but you won't be able to access it. Those still connected will not be able to reconnect when they disconnect, and in the interest of getting this testing done as quickly as possible, they should disconnect on their own anyway.

...from this other thread in Basilisk section.

------

Login server should be UP now btw, for anyone wanting to test anything on TCNova during the interim.

[risenangel]

more from discord:


RisenAngel - Today at 4:25 PM
To clarify a bit, Basilisk is up but still in the process of validating the integrity of most of the database objects. The server is currently running through our backup HDD's, which is causing the server's loading to take a lot longer than it normally would on SSD's. We shut down the login server because any actions triggered on Bas will delay the database check, and it's going to take long enough as it is.

The good news is, the server didn't explode using the recovered database. There's no current bad news, but as of right now, that's the only good news. We still have to verify that all of the objects in the database are actually valid and load properly, and it will take time. The possibility of a wipe still remains if we find too much corrupted data. Time will tell, but in the meantime you'll see Basilisk go online and offline multiple times, but you won't be able to access it. Those still connected will not be able to reconnect when they disconnect, and in the interest of getting this testing done as quickly as possible, they should disconnect on their own anyway.

that was from Miztah...

edit: oh hi nee. see you said that already.

[Hexiss]

Would have been better to wipe it now instead of Suncrusher wipe later. Its one of the reasons im not doing the Jedi grind just yet.

[sugarlizard]

Agreed, also hi sugar! former Scylla Imperial resident of Fort Yukon

(assuming youre the original )

It's me

[Virtul]

+1 for wipe

[Vlada]

Would have been better to wipe it now instead of Suncrusher wipe later. Its one of the reasons im not doing the Jedi grind just yet.

No, no it wouldn't. It would be the exact opposite of better. It would be not better.

[wren]

Not to support the conspiracy folks or people saying they should have planned better. They planned fine, imho.

I've worked on products and file systems and helped design and build appliances and have shipped and tracked many millions hard drives in a previous work life building storage clusters. Actually the majority of drive failures are trivial to detect and are easily recognizable if you monitor the SMART stats. Most people do not do this. Most IT guys do not do this. Most IT guys look at meaningless metrics or metrics that mean something completely different than what they thought they meant.

As far as spinning disks go, these guys (https://www.backblaze.com/blog/hard-...stats-q1-2016/) with their measly 65k drives mirror quite closely the metrics we kept for failures in our shipped appliances. That old Intel drive group (now known as HGST, and owned by WD who makes crappy drives) still makes the best spinning disk drives in the industry.

SSD failures are different, and far more catastrophic when they fail. Spinners are typically quite easy to recover data in failure, but it depends on the failure. There were only about .001% of our failed drives in that previous life that we could not recover the data.

Here are your 5 key SMART metrics to monitor for spinning disks: http://www.computerworld.com/article...e-failure.html
For SSD's you look at different SMART stats, here is a good place to start with these: https://www.brentozar.com/archive/20...d-performance/

Its very simple and there are many ways to create an alert when certain thresholds are hit and you know you should replace the drive soon.

Now you know, knowledge is power, use it wisely.

One other comment a few mentioned using S3 or something similar. I think S3 would be a horrendous choice. Apparently you never pay the bills for that or ran a high transaction service using S3 as your storage without some form of caching in between S3 and the server itself or the clients accessing that storage.

AWS nickles and dimes you for many things that you wouldn't expect and most other cloud providers do not charge for. There are ways around that but it involves some creative architectural changes to your infrastructure or service design. 50 a month for 2TB, lol. I ran a test a while ago that cost 20k dollars for 1 month and only used 2 TB of S3 and about 10TB of bandwidth over that month. The don't just charge you for space and bandwidth. They also charge for the number of GETS over a certain threshold and a few other things if you hit those limits. You need to understand your storage workflow before using S3 otherwise you may get an unexpected bill.

[birdmanyo]

Wipe it.. wipe it good

[PanGalactic]

I don't understand people who cling to the current db as if it were some kind of universal truth. This project is a "work in progress", there have been several wipes in the past and we KNOW that there will be wipes in the future. RELAX, it's all just zeros and ones...

[zazano]

keep your facts and science to yourself nerd

[Ryjarob]

Something something ... wipe . Something something .... COMPLETE

[Buuz Ound]

Thanks for everyone's hard work 👍

[PanGalactic]

As somebody who wrote his first application on a Univac 1108 in 1973 (stored on a drum, not a disk) I recommend everybody here read post# 384 by Wren to get an idea of how modern storage hardware works.