Vendors disappearing and possible Basilisk Rollback

[stilgarfett]

The disappearing cause of the old NPCs(vendors/shuttles/sps/etc..) has been identified and should stop happening from this restart on.

The items will be recovered some way or another, we are looking into how to do it as painless as possible.

This is great news! Any plan for what we should do about missing trainers? The slots are still consumed so I can't place new.

[MaelDiablo]

We are not out off the woods yet. The shuttleport in Fanfar Dantooine disappeared. I mean the structure is still there, but you can't pick it as a destination on the travel terminal.

[TheAnswer]

We are not out off the woods yet. The shuttleport in Fanfar Dantooine disappeared. I mean the structure is still there, but you can't pick it as a destination on the travel terminal.

The issue was fixed during the last restart, shuttles/trainers/vendors that disappeared before that are not in the world, except static shuttles/sps, those are easy to regenerate.
We are working on solutions to bring the items back.

I'll post more in detail info tomorrow about what the issue was and how we will proceed from here.

[Aekzo]

The issue was fixed during the last restart, shuttles/trainers/vendors that disappeared before that are not in the world. We are working on solutions to bring the items back.

I'll post more in detail info tomorrow about what the issue was and how we will proceed from here.

Assumed as much; thank you again.

[ipalla]

Hey TA.. just adding in more info that may help you.
MISSING VENDOR ISSUE
All missing vendors in RiverView are pre-publish 6 vendors, mostly twileks.
Five of ten vendors placed by one merchant were missing. We did a troubleshooting test..
We remade four vendors.. placed 300 credits on them.. and none on another.
We DID NOT receive notification of vendor limit maxed out.
No email notifications for vendor working, not working due to no maintenance or no items on vendors.
We since removed the four test vendors.

SHUTTLEPORT NOT WORKING
Riverview simply re-deeded the shuttleport(s) and seems to work fine now.

Thanks for the opportunity to contribute and hope this helps narrow down the issue for you. Get some Sleep and you have the rest of 2015 to fix this.. lol

[shilo]

Scurb was able to bring back my vendors, but they are empty. I checked the galaxy wide vendor search, and none of my items are listed.

[stilgarfett]

I pulled and placed the shuttle in LIVE to Rebel again and it is now working.

[TheAnswer]

Scurb was able to bring back my vendors, but they are empty. I checked the galaxy wide vendor search, and none of my items are listed.

Dannuic added a quick patch to reinsert the vendor into zone, the problem is that the auction data(vendor data) was not readded because it needs separate function calls and that was not included in the patch. If you didnt touch the vendor and did anything else the vendor data should reappear on restart...anyhow let me try a post mortem on the issue.

There was an attribute added to all AI called despawnOnNoPlayerInRange during pub6(?). What this attribute does is despawn the creature in question when there are no players in range. The first issue was that all old creatures that didnt have it got this attribute defaulted to true, this is why you see only old NPCs bugging out(shuttles are considered NPCs aswell because SOE decided so).

Now comes the interesting part. The counter of in range players was bugged, so the despawning actually never triggered untill couple of restarts ago, this is due some changes in the order of objects being loaded on server startup. The function to decrease the in range player count of AI started being called more times than it should and it started triggering the despawning all these old creatures. This of course could never have happened on Nova because it doesnt have old NPCs.

The despawning from world of these creatures triggered a chain of events that made the issue escalate into a bigger mess.

The creatures despawning from world means the creatures got saved to the database without their planet/building info of where they are spawned, so that is a problem but the creatures were still in the database... or so we thought.

When only the bazaars existed, the code to delete the items and bazaar metadata only ran on server startup so thats the first thing I disabled before doing any restarts. At that point I also did the rollbacks to try and recover as much vendors as possible to their previous state but without going too far to avoid other potential issues.

Here comes one thing I missed that made the issue worse. When kyle added vendors to the system he also added the maintanence task, and, for some reason or another he added a piece of code that specifically deleted any vendor that was not in the world when this task triggered, so that meant all the vendors that were just despawned now were being deleted one by one when their maintanence task hit, this also deleted the vendor metatadata that you see in the bazaar listings or vendor listings. So even after the rollbacks we werent safe, the vendors were being deleted and we didnt know why untill I found the cause and disabled it in the last restart. Fortunately this specific code deletes the vendors but doesnt delete the items that they contain, so the items still remain in the database but without a vendor.

Now we are in state where we still have a bunch of vendors in the database with no position info, those are "easy" to recover, we can spawn them all in a common place and let their owners retrieve the items. But we also have completely missing vendors that got wiped when their maintanence task hit.

That problem is going to be solved in a different manner. First thing I did when noticing the whole disappearing issue was doing a backup of the entire thing, so even if the current basilisk database is missing those vendors, the data itself is in the backup and I am in the process of downloading that data locally(400GB...it will take a day or two).

Due to how BerkeleyDB works, it stores log files of all the modifications, this allows for catastrophic recoveries + rollbacks. But since we dont have infinite storage we only mantain the equivalent of the last 24 hours or so of this log.

So when I get this data downloaded what I will do is, frst, do the most possible aggressive rollback to find the point before this mess happened, it should be there because we killed the server before the 24h limit of the log files.

Then I am going to generate a list of vendors that are missing from current basilisk data. From this data I can also generate a list of the items that they contain and the owner of these items. With this list I will then be able to identify all the current missing items that got orphaned due to the maintanence task deleting them from current basilisk database.

I will be able to, for the vendors that completely got wiped from current basilisk, create special containers and reinsert their items and give them back to the vendor owners, this seems like the fastest task and one that involves less hacks and consequently less potential issues down the road.

That is just one example of what we can do, theres other options of course since the data is still in the backups, but we are leaning towards that for now.

I will focus on Shuttles/trainers after all this vendor mess. Shuttles can be redeeded to function so they are not a prioirty, I have yet to check the trainer code to see how to recover them easly, but it should not be a big issue either.

So as you can see, it has been a pretty interesting couple of days and we have lots of work still to be done.

Thank you for your patience and happy holidays!

[leeta]

Thanks for the heads up The Answer, thanks to you and the team for all of your hard work and happy holidays.

[diclarkio]

Excellent detailed answer. I was lucky, being so new I never lost anything... but I know those who did will appreciate all the efforts.

[dfink]

Thanks TA for this Great work! Incredible :-)

[oddwan]

/hi5! Stunned

[cubor]

Thanks for the detailed explanation TA.

We already knew that you guys will handle that one way or another.

Coffees on me

[shilo]

If you didnt touch the vendor and did anything else the vendor data should reappear on restart...

Thank you so much TA!!! I appreciate everything you, and the team are doing! I did however, put an item up for sale on those re-spawned vendors just to initiate them so they did not disappear on me. Since that involved touching them, will the items not appear now?

Thanks!

[Vlada]

Now you all know why he is called TheAnswer.