11/01/2022 SWGEmu and Service Provider Change

[lordkator]

With the shutdown of bas now pretty much confirmed...

Can we get some insight into what the donations would now actually be funding?

specially with the data that has been shown, the server is not even hitting close to a 1Gbit of bandwidth let alone 10Gbit.

I've been away with, well, you know, Real Life(tm), had to move, big changes at work, need to pay my personal bills, etc..

The picture shared by Lolindir was from the ISP's network report in MB, not bits (we pay by megabytes per month), not Megabits.

Also, internet speed is only one of the many network criteria; latency, loss, jitter, and availability are important.

Basilisk consumes 600Gb+ of online disk and 64G of RAM, but now Finalizer actually outstrips it in RAM and CPU and is starting to build up quite an online disk footprint.

This is part of why we're optimizing Basilisk out of the environment, it's wasting resources and supporting a handful of players.

Also, it's yet another server we have to support, keep track of, handle crashes, and deal with support questions, and all by the 100% volunteer staff.

As stated by Lolindir, the environment runs many services, Basilisk is just one of them. You can see the current setup here.

I think I can remove one of the three servers by pushing all its services onto the secondary server. It'll add some lag to the forums but that's not super important, it will also slow down builds which is annoying but that's life.

And donations not only pay for the servers but our infra costs like github org, off-site backups (90 days = 6+ Terabytes), domain name registrations, on and on. And we had a great deal for the servers and network from packet.net.

However, those days are past, and we have to move, and no, we're not moving to a data center provider who's datacenters burn up because they pack their servers too tightly, nor are we moving to an EU-based location.

The project is a US-based legal entity and does not have the resources to deal with laws in other countries, regulations, and other challenges such as currency or filing taxes in other countries.

The live environment is not just a single simple server, remember we run a CI/CD pipeline (Jenkins, Gerrit, Docker), login server, Finalizer, multiple test servers (Nova, TC-Prime), global registration, bug reporting, forums, archives, CSR/Support tooling, support ticketing system, Eye of Sauron for anti-cheat analytics and alerting, primary and replica databases to support all these, monitoring and alerting. In total, there are 38 production containers and 3 VM's running on the three servers.

Finalizer has seen peaks of 2,500 logins a day (1,725 online at a time), and at those times, we're not only burning network bandwidth, but we're also burning lots of RAM, and CPU is often right at the edge even with 24 cores and 48 HT's.

The reality is we've gotten away with a 55%+ discount on our servers for years now, and the donations barely cover that these days.

We will do our best to optimize the services we support to keep the project going, but over time if donations go down, we will have to shut down more services and combine others all onto a smaller footprint, and people will just have to deal with the latency, and availability issues.

Oh, and don't forget all of us are 100% volunteers, we run all this in our spare time, and the last thing we need is more randomness injected by running this all in a closet somewhere.

PS: This move is already costing me 20 hours a week in prep; when it's done, it'll be 160+ hours of my life gone; please be careful about waving hands and saying things are easy, they are not, and I know first hand.

[Praxi34]

I've been away with, well, you know, Real Life(tm), had to move, big changes at work, need to pay my personal bills, etc..

The picture shared by Lolindir was from the ISP's network report in MB, not bits (we pay by megabytes per month), not Megabits.

Also, internet speed is only one of the many network criteria; latency, loss, jitter, and availability are important.

Basilisk consumes 600Gb+ of online disk and 64G of RAM, but now Finalizer actually outstrips it in RAM and CPU and is starting to build up quite an online disk footprint.

This is part of why we're optimizing Basilisk out of the environment, it's wasting resources and supporting a handful of players.

Also, it's yet another server we have to support, keep track of, handle crashes, and deal with support questions, and all by the 100% volunteer staff.

As stated by Lolindir, the environment runs many services, Basilisk is just one of them. You can see the current setup here.

I think I can remove one of the three servers by pushing all its services onto the secondary server. It'll add some lag to the forums but that's not super important, it will also slow down builds which is annoying but that's life.

And donations not only pay for the servers but our infra costs like github org, off-site backups (90 days = 6+ Terabytes), domain name registrations, on and on. And we had a great deal for the servers and network from packet.net.

However, those days are past, and we have to move, and no, we're not moving to a data center provider who's datacenters burn up because they pack their servers too tightly, nor are we moving to an EU-based location.

The project is a US-based legal entity and does not have the resources to deal with laws in other countries, regulations, and other challenges such as currency or filing taxes in other countries.

The live environment is not just a single simple server, remember we run a CI/CD pipeline (Jenkins, Gerrit, Docker), login server, Finalizer, multiple test servers (Nova, TC-Prime), global registration, bug reporting, forums, archives, CSR/Support tooling, support ticketing system, Eye of Sauron for anti-cheat analytics and alerting, primary and replica databases to support all these, monitoring and alerting. In total, there are 38 production containers and 3 VM's running on the three servers.

Finalizer has seen peaks of 2,500 logins a day (1,725 online at a time), and at those times, we're not only burning network bandwidth, but we're also burning lots of RAM, and CPU is often right at the edge even with 24 cores and 48 HT's.

The reality is we've gotten away with a 55%+ discount on our servers for years now, and the donations barely cover that these days.

We will do our best to optimize the services we support to keep the project going, but over time if donations go down, we will have to shut down more services and combine others all onto a smaller footprint, and people will just have to deal with the latency, and availability issues.

Oh, and don't forget all of us are 100% volunteers, we run all this in our spare time, and the last thing we need is more randomness injected by running this all in a closet somewhere.

PS: This move is already costing me 20 hours a week in prep; when it's done, it'll be 160+ hours of my life gone; please be careful about waving hands and saying things are easy, they are not, and I know first hand.

Brilliant work LK. A great explanation to add to your already big workload....We appreciate you and the team massively! Hopefully the donations stay up and we dont have to downsize!! Especially with what's going on in the world financially, streamlining and getting rid of the 'non-essentials' is good project management!

[neopixie]

PS: This move is already costing me 20 hours a week in prep; when it's done, it'll be 160+ hours of my life gone; please be careful about waving hands and saying things are easy, they are not, and I know first hand.

No one here said it was easy?

I, among others pretty do this on the daily as a real life(tm) job and understand its can be a royal pain in the breasticales. Its more a curiousity question how you're hitting such high server usage and bandwidth (although id maybe get your ISP to display as MB and not Mb) when a certain other 'place in a galaxy far far away' is able to run 'double' the pop with a fracture of the cost.

[pugguh]

I've been away with, well, you know, Real Life(tm), had to move, big changes at work, need to pay my personal bills, etc..

The picture shared by Lolindir was from the ISP's network report in MB, not bits (we pay by megabytes per month), not Megabits.

Also, internet speed is only one of the many network criteria; latency, loss, jitter, and availability are important.

Basilisk consumes 600Gb+ of online disk and 64G of RAM, but now Finalizer actually outstrips it in RAM and CPU and is starting to build up quite an online disk footprint.

This is part of why we're optimizing Basilisk out of the environment, it's wasting resources and supporting a handful of players.

Also, it's yet another server we have to support, keep track of, handle crashes, and deal with support questions, and all by the 100% volunteer staff.

As stated by Lolindir, the environment runs many services, Basilisk is just one of them. You can see the current setup here.

I think I can remove one of the three servers by pushing all its services onto the secondary server. It'll add some lag to the forums but that's not super important, it will also slow down builds which is annoying but that's life.

And donations not only pay for the servers but our infra costs like github org, off-site backups (90 days = 6+ Terabytes), domain name registrations, on and on. And we had a great deal for the servers and network from packet.net.

However, those days are past, and we have to move, and no, we're not moving to a data center provider who's datacenters burn up because they pack their servers too tightly, nor are we moving to an EU-based location.

The project is a US-based legal entity and does not have the resources to deal with laws in other countries, regulations, and other challenges such as currency or filing taxes in other countries.

The live environment is not just a single simple server, remember we run a CI/CD pipeline (Jenkins, Gerrit, Docker), login server, Finalizer, multiple test servers (Nova, TC-Prime), global registration, bug reporting, forums, archives, CSR/Support tooling, support ticketing system, Eye of Sauron for anti-cheat analytics and alerting, primary and replica databases to support all these, monitoring and alerting. In total, there are 38 production containers and 3 VM's running on the three servers.

Finalizer has seen peaks of 2,500 logins a day (1,725 online at a time), and at those times, we're not only burning network bandwidth, but we're also burning lots of RAM, and CPU is often right at the edge even with 24 cores and 48 HT's.

The reality is we've gotten away with a 55%+ discount on our servers for years now, and the donations barely cover that these days.

We will do our best to optimize the services we support to keep the project going, but over time if donations go down, we will have to shut down more services and combine others all onto a smaller footprint, and people will just have to deal with the latency, and availability issues.

Oh, and don't forget all of us are 100% volunteers, we run all this in our spare time, and the last thing we need is more randomness injected by running this all in a closet somewhere.

PS: This move is already costing me 20 hours a week in prep; when it's done, it'll be 160+ hours of my life gone; please be careful about waving hands and saying things are easy, they are not, and I know first hand.

Thanks for the update LK.