Quote Originally Posted by shilo View Post
Scurb was able to bring back my vendors, but they are empty. I checked the galaxy wide vendor search, and none of my items are listed.
Dannuic added a quick patch to reinsert the vendor into zone, the problem is that the auction data(vendor data) was not readded because it needs separate function calls and that was not included in the patch. If you didnt touch the vendor and did anything else the vendor data should reappear on restart...anyhow let me try a post mortem on the issue.

There was an attribute added to all AI called despawnOnNoPlayerInRange during pub6(?). What this attribute does is despawn the creature in question when there are no players in range. The first issue was that all old creatures that didnt have it got this attribute defaulted to true, this is why you see only old NPCs bugging out(shuttles are considered NPCs aswell because SOE decided so).

Now comes the interesting part. The counter of in range players was bugged, so the despawning actually never triggered untill couple of restarts ago, this is due some changes in the order of objects being loaded on server startup. The function to decrease the in range player count of AI started being called more times than it should and it started triggering the despawning all these old creatures. This of course could never have happened on Nova because it doesnt have old NPCs.

The despawning from world of these creatures triggered a chain of events that made the issue escalate into a bigger mess.

The creatures despawning from world means the creatures got saved to the database without their planet/building info of where they are spawned, so that is a problem but the creatures were still in the database... or so we thought.

When only the bazaars existed, the code to delete the items and bazaar metadata only ran on server startup so thats the first thing I disabled before doing any restarts. At that point I also did the rollbacks to try and recover as much vendors as possible to their previous state but without going too far to avoid other potential issues.

Here comes one thing I missed that made the issue worse. When kyle added vendors to the system he also added the maintanence task, and, for some reason or another he added a piece of code that specifically deleted any vendor that was not in the world when this task triggered, so that meant all the vendors that were just despawned now were being deleted one by one when their maintanence task hit, this also deleted the vendor metatadata that you see in the bazaar listings or vendor listings. So even after the rollbacks we werent safe, the vendors were being deleted and we didnt know why untill I found the cause and disabled it in the last restart. Fortunately this specific code deletes the vendors but doesnt delete the items that they contain, so the items still remain in the database but without a vendor.

Now we are in state where we still have a bunch of vendors in the database with no position info, those are "easy" to recover, we can spawn them all in a common place and let their owners retrieve the items. But we also have completely missing vendors that got wiped when their maintanence task hit.

That problem is going to be solved in a different manner. First thing I did when noticing the whole disappearing issue was doing a backup of the entire thing, so even if the current basilisk database is missing those vendors, the data itself is in the backup and I am in the process of downloading that data locally(400GB...it will take a day or two).

Due to how BerkeleyDB works, it stores log files of all the modifications, this allows for catastrophic recoveries + rollbacks. But since we dont have infinite storage we only mantain the equivalent of the last 24 hours or so of this log.

So when I get this data downloaded what I will do is, frst, do the most possible aggressive rollback to find the point before this mess happened, it should be there because we killed the server before the 24h limit of the log files.

Then I am going to generate a list of vendors that are missing from current basilisk data. From this data I can also generate a list of the items that they contain and the owner of these items. With this list I will then be able to identify all the current missing items that got orphaned due to the maintanence task deleting them from current basilisk database.

I will be able to, for the vendors that completely got wiped from current basilisk, create special containers and reinsert their items and give them back to the vendor owners, this seems like the fastest task and one that involves less hacks and consequently less potential issues down the road.

That is just one example of what we can do, theres other options of course since the data is still in the backups, but we are leaning towards that for now.

I will focus on Shuttles/trainers after all this vendor mess. Shuttles can be redeeded to function so they are not a prioirty, I have yet to check the trainer code to see how to recover them easly, but it should not be a big issue either.

So as you can see, it has been a pretty interesting couple of days and we have lots of work still to be done.

Thank you for your patience and happy holidays!