• Vlada

    by Published on 12-24-2018 10:11 PM   



    gif by Evgenii Barankin
    by Published on 10-14-2018 09:06 AM   

    Basilisk Maintenance - October 19th 2018

    Date: 10/14/2018
    The SWGEmu Staff




    Basilisk Maintenance - October 19th 2018


    Hello there fellow SWGEmuers,

    It is with heavy hearts that we have to announce that on October 19th, around 5AM Eastern Time (you can find your local time here: https://goo.gl/LNYS6X) Basilisk will be taken down for much needed maintenance. Downtime will (roughly) last 36 hours. Yes, yes, we know its a long time and that it is the weekend, but that's the only time our developers have enough spare time needed to finish the whole process.


    To understand just why its gonna take this long, here are some of the things we are hoping to achieve:
    1. Rebuild databases (especially sceneobjects.db at 575 Gigs)
    2. Purge mail.db
    3. Cleanup ghosted vendor items (we commented out deletes a while back due to vendor issues)
    4. Migrate to a container like The Pit, Nova, TC-Prime
    5. Stretch goal: Automated builds like Nova, The Pit etc.
    Why exactly is all this needed?
    1. Donations have plummeted in the last few months, so if we plan to keep on going for a foreseeable time we need to consolidate our services and reduce our expenditures. (Of course that shouldn't stop you from visiting our Donations page, every little bit helps us keep our services up and running)
    2. Less disk usage leading to quicker clones to TC-Prime and smaller backup storage requirements
    3. Easier start/stop by more of the developer team
    4. More portable setup (can move Basilisk around like we move Nova, The Pit etc.)
    5. Clean setup that doesn't require special knowledge to build and maintain Basilisk
    6. Ability to enable more developers to contribute (core developer team) directly to Basilisk code (Automated Jenkins build, push, restarts)
    So, make plans, spend time with your family or friends, or just go outside for a change. I've heard its lovely there this time of the year. Thank you for your understanding and your continuous support.


    Oh yeah, TC Nova and TC Prime will be online for the duration, so you don't really have to waste your time on that RL thing.


    ~The SWGEmu team
    by Published on 12-05-2017 11:37 AM   

    Infrastructure overhaul of 2017

    December 2017
    lordkator




    Infrastructure overhaul of 2017

    As many of the community are aware on August 9, 2017 Basilisk experienced an extended unplanned outage due to disk issues on the server. As that process unfolded TheAnswer promised the community we would share what happened and what we planned to do about it.

    TL;DR (Summary)

    We lost disks on the original Basilisk server causing us to do manual work to restore the game server databases, this work resulted in the restoration of the server on August 16, 2017.

    As of December 4, 2017 all services have been moved to a new environment that is much more robust, has considerably more resources and is designed for us to handle hardware outages much more easily in the future. Our new environment includes redundant servers, faster internet access, more CPU power, more RAM and improved storage redundancy and speed.

    The completion of this migration provides a stable platform for the community for many years into the future.

    What happened?

    When the original server for Basilisk was deployed in 2006 it was a state of the art machine and had experienced a number of hardware upgrades over the years.

    The system was configured with multiple Solid State Disks (SSD) setup to keep two on-line live copies of the data (RAID 1 mirror). The goal of such a setup is that when one disk fails we can replace it before data is lost and rebuild the copies. This setup also provides higher read rates because the system can ask both disks for different data at the same time.

    The week before this incident one of the mirrored disks failed, our hosting provider failed to notify us and meanwhile we did not get the emails from the sever alerting us to disk issues. In a sad twist of fate the second disk that was mirroring this failed drive also started to fail a week later. The odds of two disks failing within such a short timeframe are fairly rare.

    Because of the size of Basilisk's database (over 600 Gigs) we were doing backups on an ad-hoc basis to the Nova server's disks. This meant any restore would lose significant amounts of player progress. With this in mind TheAnswer worked with low-level filesystem debugging tools to extract the database files from the failing drive. This was a painful, slow process that required many iterations to get the data back to a usable state. Much of it was manual and each step could take many hours to run before the results are known and decisions are made on the next step. After many sleepless nights TheAnswer was able to get Basilisk back online on August 16.


    How do we avoid this in the future?


    In response to this event we took inventory of all our services and did an analysis of our current setup. As you can imagine after 10 plus years the project had accumulated many services and servers to run our community. This setup was very difficult to maintain due to the many dependencies between various services and the underlying software, operating systems and hardware.

    After debating various paths forward the team decided it was time to overhaul our infrastructure. We decided to re-build from scratch on new bare metal servers from packet.net and utilize an open-source technology called kubernetes to manage the services as individual movable containers. We would deploy our servers on top of ZFS storage pools which would allow us to have modern data safety and management tools.

    Deploying on packet.net gives us an incredible amount of flexibility, rather than opening tickets and asking for new machines or emailing back and forth we can just launch new resources using the packet.net API. In addition to that we have "reserved" three servers that allow us to run our infrastructure and provide on-line ready to run redundancy for our services.

    Containerizing our services and using kubernetes to manage them allows us the ability to quickly re-schedule services on other hardware if we lose a node or it becomes overloaded with work. The industry is rapidly turning to support kubernetes (originally a Google technology) and by standardizing on this system we can leverage other providers if needed in the future or quickly expand our footprint in any of packet.net's datacenters.

    By utilizing ZFS for our storage system we are able to make instantaneous snapshots of the data underlying a service. We setup these storage volumes using very high speed non-volatile memory (PCIe NVMe) and join those in redundant and high-speed configurations (RAID 10). For most services we were able to deploy packet.net's block store volumes. These are high-speed (PCIe NVMe) network attached volumes that allow us to quickly move between servers if a server crashes or becomes overloaded with work.

    This combination of our hosting, containerization and storage strategy provides us with many options that were not available to us before the overhaul. This investment should power the project's needs for many years to come and will make it easier for the team to manage existing services and provide new and exciting capabilities to the community in the future.

    We expect the short term financial impact to be a bit higher as the services transition and overlap plus bandwidth to copy files into the new environment. Over time we predict the costs to be about the same as our previous setup with easily a 10x increase in capacity and capabilities.

    Status?


    As of December 2, 2017, all services have been moved to the new infrastructure. Basilisk has been happily running on the new hardware since September 24, 2017 and Nova followed not long after that. We have moved everything from forums, support site, jenkins, gerrit and various other servers to the new infrastructure.

    We have daily snapshot backups that are pushed to external block store volumes so we can lose a host completely and have worst case one day of lost progression. Meanwhile we have deployed database logging on Basilisk and Nova so that every transaction is saved in storage in a way that we can roll-forward a database crash if needed by re-playing the changes that happend since the prior copy of the database.

    We maintain daily snapshots for a week both locally on the server's disks and remotely on blockstore volumes attached over the network.


    What's it look like?


    Here is a simplified diagram of our current environment for your viewing pleasure:



    Random Stats
    • Migrated 16 distinct services
    • Over 3 Terabytes of data migrated
    • 3,009,329 lines of PHP
    • 18,640,175 lines of C++
    • 102,682,949 lines of Lua
    • 12 copies of sceneobjects.db in various folders
    • 3 people actually read this far in this post.
    Next Steps

    We've started creating alert bots that send messages to a channel the staff can monitor for issues so they can help escalate as needed.

    We will be adding more alerts, testing some deep storage solutions (AWS S3 Glacier and the like) and adding more tools so other members of the staff can help with various tasks w/o being Unix admin experts.


    ~lordkator DevOps Engineer
    by Published on 11-17-2017 12:04 AM   

    Q. What is SWGEmu?


    A. SWGEmu is an acronym/abbreviation for Star Wars Galaxies Emulator.
    Star ...
    by Published on 03-07-2017 10:14 PM   

    REMINDER: RL Monetary Trades in-game are a bannable offense

    March 7, 2017
    The SWGEmu Staff




    REMINDER: RL Monetary Trades in-game are a bannable offense


    Lets start off with a quote from our SWGEmu Test Center & Basilisk Rules & Policies

    Quote Originally Posted by Vlada View Post



    1.7. On SWGEmu Servers, all accounts and characters used for financial gain will be deleted/banned. It is forbidden to use your account for:
    • 1.7.1. Soliciting real life monetary trades.
    • 1.7.2. Purchasing of in-game credits or items for real life currency.
    • 1.7.3. Exchange of in-game items/credits for real life items.
    • 1.7.4. Exchange of in-game items/credits for items/currency from other games.
    Basilisk and TC: Nova are temporary test servers, which means everything you have on them, every character, item or credit will be eventually wiped when SWGEmu project is complete. SWGEmu project is free, and everything in game is free and available to anyone willing to put in time and some effort to obtain it by just playing.

    Now that we have that out of the way, lets reiterate. Buying or selling of in-game items and/or credits is against our rules and is a bannable offense. Credits transactions are logged and if caught all those involved will be permanently banned from any and all of our services. SWGEmu does not in any way support nor does it condone any real life monetary trades for in-game items and/or credits.

    Anyone who has purchased credits or in-game items, even if their account has not been banned, should ask for a chargeback immediately and report credit/item seller to paypal or their bank as a scammer.

    Furthermore we will be instituting a "Snitch" program, for those caught or just willing to help us and themselves and get a fresh start on SWGEmu servers. If you bought credits/items and you got caught, you will be offered an option to start fresh with a new account if you cooperate and help us get rid of those breaking our rules and ruining the emulator for everyone.


    ~The SWGEmu Staff
    by Published on 05-17-2016 07:13 AM

    SWGEmu Recruitment

    SWGEmu Quality Assurance
    Community Tester Program



    Quality Assurance team - Community Testers Program (non-staff) recruitement

    Community Tester APPLICATION
    • Name of applicant (last name optional):
    • Age of applicant:
    • Email address:
    • Forum, Mantis, and IRC nickname:
    • Time zone:
    • Times you are most often available:


    SWG Knowledge/Experience:

    • When did you start playing SOEs retail SWG? (dates or publish specific):
    • What primary profession(s) did you play Pre-CU?
    • What other profession(s) have you tried Pre-CU?
    • What other experience do you have from Pre-CU?
    • How long have you been following this project?

    QA/Testing Experience:

    • Do you have any previous/current experience in testing/QA? If so, please explain:
    • Situational Assessment: A player comes to you in-game or on IRC/forums with a possible new bug. What is your immediate course of action?
    • Fill out a bug report. You may either link a report you submitted on Mantis previously or make up a new one using the form below.
    Title:
    Category:
    Reproducibility:
    Severity:
    Priority:
    Report Details (feel free to include screenshot(s) if applicable):

    • Please add anything else you would like, to support your application:



    Please send reply to application@swgemu.com


    ~ The SWGEmu QA team
    Page 1 of 3 1 2 3 LastLast