AffectedArc07

Lack of TG bloat so we can actually sustain >60 players without running out of memory. For context, DD caps out at 1.8GB of RAM

Vox In all seriousness, the dorms pool controller. It has no use unless someone emags it once per millennia, and used to cause horrific turf init issues

I have had both and I like both, they are nice pets The former

I am fucking tired of this being thrown around as a buzzphrase on the github to the point where I will actively call people out who throw this out.

I like off-station content because it makes stuff feel less packed on station I really like the idea of wizard because of the chaos, and absolutely hate revs I think they should be both Yes

Heads of Staff and Maintainers share the same authority/weight in making codebase decisions, with a few exceptions: Heads have no vote on fix or refactor PRs Heads can object on design solely, maintainers can object on design and implementation Heads cannot veto a PR singlehandedly, nor can maintainers unless: The PR has copyright/legal/license issues The PR is going to cause horrific issues to the server down the road due to implementation The server host has no decision in pull requests, apart from an implementation veto regarding technical matters. They cannot veto on design principles. Examples include tooling that runs server side, extra libraries for the server, data storage methods, etc etc The three arms of the staff team (Heads - Host - Maints) tend to govern themselves and subordinates, however the heads have input in hiring/firing in each arm, and can be seen as the general overseers

I powerpointed it up and tweaked it to how it runs, as far as I know

@MxNimbus433I was unaware of the threat level thing, after reading up, this seems fine. Objection withdrawn.

Not in favour Wizard + Nukies + Cult in the same round? No thanks

This project has been dead for multiple years

@Sirryan2002 should be fixed now

@Shadeykins and @Sirryan2002 are now wiki managers

High pop has always been a dislike for me for to performance, lack of job slots and a low RP level due to so much chaos, but everyone else seems to enjoy it so I’m not going to be a dictator and change it. Code priorities shifted recently and as of now my priority is implementing 2FA for BYOND accounts on the server.

I would love more advanced ingame scripting/nerd shit to do because I found that really entertaining, but sadly its too performance intensive to evaluate it all at runtime and manage, so its unlikely we will ever get something like that again.

I honestly dont know. We have a lot of species variety and the karma jobs are nice, but as well as this, its nice being a standout server on the hub that isnt just "oh hey were yet another cookie-cutter TG downstream"

USD because spark is US so its easier for him to convert

Were not doing this forever. Give me $50,000, no more offers

Conclusion post This is just a tiny glimpse into the hell of hosting, and why it takes so many hours and why its so draining on people. This entire thread only covers some of my issues, and doesnt even go into VPN Hell Thread-pinning to avoid resource conflicts Juggling multiple IPs DNS Webserver mail sending hell DNS Did I mention DNS? So yeah, this is why the git has been a bit slow. Theres been a lot to deal with, and an entire mix of IRL hell too. But hey, atleast things are where they are now.

Chapter 10 (BONUS CHAPTER) - Remotely managing a server where you dont have a KVM/IPMI console So as I mentioned in chapter 2, IPMI is super nice as you can do full hardware level access without being there. However, the secondary server I bought did not have this as standard. Let me give some context. As well as the primary server in NYC, we needed another server to handle being an offsite backup incase the NYC datacentre burns to the floor or theres a natural disaster or whatever. Because of location (and pricing), I got a backup server in germany. Germany was great since its almost on the opposite side of the world to NYC, and the way I see it, if there is a natural disaster that affects both NYC and germany at the same time, paradise isnt my priority, finding a nuclear bunker is. Anyways, the server had the following: An i7-2600 CPU 16GB RAM 2x3TB HDDs 1 Gigabit symmetrical internet This was plenty for backup resources, and for only £25 a month it was great, however, this is where the problems begin. There is no IPMI, only a "rescue system", which is where the real chaos begins. To install the backup server OS, I had to: Boot into the rescue system Get an ISO of the backup server Make a VM in the rescue system that uses the real HDD as a virtual drive Open the VM window over VNC so I could actually use it and install stuff Close the rescue system and restart the host OS, praying that it comes back up .... and it didnt come back up. Now, the provider does allow you to get a KVM console, but there are some catches: You only get a 3 hour lease on it Theres no virtual media or hardware control, just a keyboard, mouse and monitor The compression and input latency is horrific To give you an idea of what I mean on the third point, have you ever used a keyboard with a full second of latency on keyup and keydown? If you held a key for a split second, it would type 5 of that letter at the other end, which makes entering stuff you cant see (like passwords) very unfun. Oh, and in regards to the compression, this is a windows login screen However, in a weird turn of events, this was the rescue system with ludicrous compression It kinda has an old abandoned terminal aesthetic to it that I actually kinda like. After an hour of screwing around with horrific latency, I had the system actually boot properly and have its external IP set, which meant I could finally configure the backup server to backup the main server.

Chapter 9 - The woes of getting logs into a nice viewer without killing the CPU So, if youre an admin you already know this, but admins have a fancy webtool thingy to look at logs (Courtesy of Denghis and his wizardry). However, you need some way to get the logs into that (In real time keep in mind). At first I ran the importer on the gameserver, but that caused servere resource issues, which I wasnt having because BYOND needs to be the most performant thing running on that machine. To try and combat this, I made a shared folder for the logs and mapped it on the logging server. I thought this would be the solution, but the log importer doesnt like network shares and kept throwing errors out, crippling the import process. A lot of tinkering and tuning later, we finally had the log importer running on the gamserver again without nuking the entire CPU. This made admins happy because they could use log webtools again.

Chapter 8 - Why emulation sucks and paravirtualisation is the future (EXTRA EXTRA: BYOND sucks) So. The gameserver was now running on the new hardware. Everything was migrated, and it was all going nicely, apart from the gameserver having some spurious init time rises. After a little investigation, I discovered that all of the things that increased were related to operations on the /icon type. Now this is where the "BYOND Sucks" part comes in. Every time you do an operation on a /icon, its written to the HDD as cache. Don't ask me why BYOND doesnt use RAM, because that would make a ton more sense. Situations like roundstart have a lot of icon operations at once, which means the HDD gets absolutely thrashed. Spikes here are from mid MC initialize, with SSatoms being the first and SSair being the second (Both have lots of icon operations) Now heres where emulation and paravirtualisation tie into this. When you are adding hardware to a virtual machine, you have 2 choices, you can either emulate it, or partavirtualise it (Commonly referred to as VirtIO). Emulation is a lot slower as it is having to pretend it is an actual device (Such as an intel e1000 network card), but since it is faking real hardware, it is much more compatible with other systems. Emulation also requires device emulation to live on a specific thread, and that thread jumps around a lot. In a worst case scenario, the SATA emulation process ends up on the same thread as BYOND, which makes for a really bad time) VirtIO on the other hand just says to the VM "Ok youre a VM, heres a direct pipe to the storage, go use it". This has much better overhead (I am talking literally half the time for IO operations), not to mention higher throughput (VirtIO network cards operate at 10 gigabits per second) VirtIO drives are natively available on Linux, however since the game was running windows, they werent available by default. To make matters more complicated, windows does not like it when you change your HDD interface (IE: going from SATA -> SCSI). The network card was relatively easy, just install drivers, reboot, change NIC, reset IP, done. The HDD however, I had to modify the boot image to preload the VirtIO drivers, then change the boot ID to make it accept, and it was just generally pain all around, not to mention being on time constraints because "AA THE SERVER IS DOWN WHAT ARE YOU DOING". But hey, it got sorted, and as an added benefit of paravirtualisation, I could tell the HDD IO process to live on its own thread outside of everything else, preventing resource bottlenecks. This had major performance improvements and reduced a lot of roundstart inconsistencies, and just made the server more performant overall.

Chapter 7 - How to migrate with minimal downtime The hardest part of any migration, is well, the migration itself. Moving services from A to B is hard enough on its own, combine that with the stress of downtime and people yelling "HEY AA WHEN IS THE SERVER BACK" doesn't help matters, but there were some ways to save time. The main part of the game server is the DB itself. That is the one thing thats always active and has to be moved as one block, so the way around that is to stop the main server, snapshot the DB, move it over, and reimport. This itself took time, but that gave me time to do other things, such as prepare webservices (forum/wiki) and DNS for the new boxes. Migration time is also important. Stats show that our lowpop time is about 10AM UTC Which means about 11AM on a Saturday morning. Wonderful. What a wonderful time to be awake.

Chapter 6 - Getting the game running on windows I was wanting to avoid hosting on windows for many reasons, but most importantly, windows update. God I despise windows update. Whats this? You're running a production server and you have active connections doing shit? Too bad, update time. The VM itself had slept a few times before I switched to it over, which meant I had a lot of policy tweaking to do, to make sure that the windows VM never shut down under any circumstances to keep the game running. It took some doing, but I got there in the end. The major disadvantage with windows is lack of paravirtualisation (More in chapter 8), this was a slight performance loss at first (It got much worse later), but I bit the bullet and deemed it fine enough to continue working, especially since the windows performance tests were proving fruitful. Players werent whining and people were starting to say it was better than before. Things were starting to look up. The only major hurdle was moving stuff such as game logs (runtime_summary from endround) from a windows host to a linux host, but that was easily overcome with SMB shares. For anyone curious, this is how minimal the windows install on the gameserver is

Chapter 5 - The first performance test, AKA byond sucks on linux My original plan was to run BYOND on linux to get maximum performance and the ability to use paravirtualisation (More in chapter 8). I had set everything up and was ready to go, so I closed the main server and funnelled everyone over to the test one. Good god it was horrible. Linux DD (DreamDaemon) does not play nice with our code at all. Movement stutters every 5 seconds and the server just chugs to play on. Something about it just becomes super unbearable to deal with and the entire experience was appauling, with a pretty compelling "This is way worse" vote. Armed with this information, it was time to go back to the drawing board.

Chapter 4 - Moving everything to a new platform With some testing done, I decided to migrate everything to proxmox VE. This carried 2 advantages: It uses the KVM hypervisor which allows for a lot more tuning than Hyper-V, not to mention overhead so low you can run GTA V with no performance loss (Or 7 entire gaming VMs off of one PC) It would actually allow the CPU to friggin boost However, this meant I had to go through the hell of chapter 2 again. It took 3 attempts, but I finally had it done after a load of screaming. Upon being landed at a terminal, the very first thing I did was check clocks, and I was so happy to finally see this From here, things started to look up, but there was still a long way to go, and many (unknown as of yet to me) problems, but hey, the CPU was turboing and I could start to make VMs again, and do more performance testing and optimisations.