AffectedArc07
-
Posts
879 -
Joined
-
Last visited
-
Days Won
62
Posts posted by AffectedArc07
-
-
Chapter 7 - How to migrate with minimal downtime
The hardest part of any migration, is well, the migration itself. Moving services from A to B is hard enough on its own, combine that with the stress of downtime and people yelling "HEY AA WHEN IS THE SERVER BACK" doesn't help matters, but there were some ways to save time.
The main part of the game server is the DB itself. That is the one thing thats always active and has to be moved as one block, so the way around that is to stop the main server, snapshot the DB, move it over, and reimport. This itself took time, but that gave me time to do other things, such as prepare webservices (forum/wiki) and DNS for the new boxes.
Migration time is also important. Stats show that our lowpop time is about 10AM UTC
Which means about 11AM on a Saturday morning. Wonderful. What a wonderful time to be awake.
-
Chapter 6 - Getting the game running on windows
I was wanting to avoid hosting on windows for many reasons, but most importantly, windows update. God I despise windows update. Whats this? You're running a production server and you have active connections doing shit? Too bad, update time. The VM itself had slept a few times before I switched to it over, which meant I had a lot of policy tweaking to do, to make sure that the windows VM never shut down under any circumstances to keep the game running. It took some doing, but I got there in the end.
The major disadvantage with windows is lack of paravirtualisation (More in chapter 8), this was a slight performance loss at first (It got much worse later), but I bit the bullet and deemed it fine enough to continue working, especially since the windows performance tests were proving fruitful. Players werent whining and people were starting to say it was better than before. Things were starting to look up. The only major hurdle was moving stuff such as game logs (runtime_summary from endround) from a windows host to a linux host, but that was easily overcome with SMB shares.
For anyone curious, this is how minimal the windows install on the gameserver is
-
1
-
-
Chapter 5 - The first performance test, AKA byond sucks on linux
My original plan was to run BYOND on linux to get maximum performance and the ability to use paravirtualisation (More in chapter 8). I had set everything up and was ready to go, so I closed the main server and funnelled everyone over to the test one.
Good god it was horrible. Linux DD (DreamDaemon) does not play nice with our code at all. Movement stutters every 5 seconds and the server just chugs to play on. Something about it just becomes super unbearable to deal with and the entire experience was appauling, with a pretty compelling "This is way worse" vote. Armed with this information, it was time to go back to the drawing board.
-
Chapter 4 - Moving everything to a new platform
With some testing done, I decided to migrate everything to proxmox VE. This carried 2 advantages:
- It uses the KVM hypervisor which allows for a lot more tuning than Hyper-V, not to mention overhead so low you can run GTA V with no performance loss (Or 7 entire gaming VMs off of one PC)
- It would actually allow the CPU to friggin boost
However, this meant I had to go through the hell of chapter 2 again. It took 3 attempts, but I finally had it done after a load of screaming. Upon being landed at a terminal, the very first thing I did was check clocks, and I was so happy to finally see this
From here, things started to look up, but there was still a long way to go, and many (unknown as of yet to me) problems, but hey, the CPU was turboing and I could start to make VMs again, and do more performance testing and optimisations.
-
Chapter 3 - The story of how windows (and/or asrock) sucks
So I mentioned before how BYOND needs single threaded speed to be usable at all. This is achieved using a high end CPU, and having a high clock speed. Our previous server had an intel 9900K CPU which regularly turbo boosted to 4.9Ghz. This was very impressive and ran the server very well. In my research, I found that the RYZEN 5900X outperforms the 9900K in single thread performance by roughly 20%, so I expected it to all be going well. Now heres where the real issue begins.
Windows (Hyper-V server, windows server 2016, and windows server 2019) does not have the CPU microcode to properly boost this CPU, which meant our new, faster CPU was now being hardcapped at a lower speed, making it slower than the previous CPU.
After about 2 weeks of debugging and testing, I discovered that it was down to windows and unfixable, which meant I had to use a new platform for everything. Enter Proxmox VE.
-
1
-
-
Chapter 2 - The pain of an OS install when your ISO is 6 gigabytes and moves at 3 megabytes a second
With the server obtained, I was now able to get to work. Most servers have a thing called IPMI, which is basically a web GUI for anything you'd need physical access for, including but not limited to:
- Hardware status such as fan speeds and temperatures
- Remote media mounting to install an operating system without a USB stick
- Power on/off control
-
A remote console with mouse, keyboard and video
- This operates outside of an OS remote desktop, meaning you can use it even when the OS is locked up, and before the OS has loaded, so you can do things such as change BIOS settings and more
Now, to actually install the OS on here, I just had to put it on the virtual media drive and boot the server to that. Simple enough right? Oh lord...
Asrock's IPMI is pretty damn flawed, and the virtual media has a max speed of 3 megabytes a second, and the ISO for the OS was about 6 gigabytes. So I had to sit here and babysit the OS install to make sure it installed properly. Oh, and it also has to be reset if your connection drops for even a second. I spent a full 2 hours just getting the OS installed, and the mentors can tell you this involved a lot of screaming.
However, after the screaming, I had my OS (Microsoft hyper-v server) installed, and I was ready to start work on things. I made my VMs, did the base security setup, and was ready to start performance testing, however I was about to run into a major issue.
-
1
-
Chapter 1 - Obtaining the damn server
Actually obtaining the server is no easy feat, as there are 2 key factors that make it difficult to do:
- Needs very high fast single-thread performance
- Needs to be in NYC for the high-speed connection between US <-> EU
These two factors combined make the server excellent for SS13 hosting, but also horrifically difficult to buy. This kind of server is also perfect for stock trading due to very fast execution speed, and since its in NYC, it has sub-millisecond latency to the NYSE, allowing you to have trade bots run at phenomenal speeds. It took about 2 weeks of constant scouring to actually secure one in time. We even had an incident where we were in process of buying one, and it went out of stock while we were putting in payment. That is how hard-to-get these things are. Oh, and since its just my luck, on the day the server was available, I was ill and bedridden to the point where I couldn't even use a PC, so I had to spam poor neca from my phone like "OI! OI! OI! OI! OI! BUY THE SERVER". But hey, we ended up with one, that I declared "beefy enough".
The end result server has:
- A RYZEN 5900X CPU
- 128GB of RAM
- A 2TB NVMe SSD
- A 10 gigabit symmetrical outbound internet connect
and is of course, in NYC so we get the good connection.
After the purchasing hell, I was expecting stuff to be nowhere near as bad as this, but oh boy was this only the beginning.
-
WARNING: This entire story is just complete nerd shit so prepare for technohell
Expect more chapters as time goes on, but this is basically the chronicles of why you never want to become host
Chapter listing
- Obtaining the damn server
- The pain of an OS install when your ISO is 6 gigabytes and moves at 3 megabytes a second
- The story of how windows (and/or asrock) sucks
- Moving everything to a new platform
- The first performance test, AKA byond sucks on linux
- Getting the game running on windows
- How to migrate with minimal downtime
- Why emulation sucks and paravirtualisation is the future (EXTRA EXTRA: BYOND sucks)
- The woes of getting logs into a nice viewer without killing the CPU
- BONUS CHAPTER - Remotely managing a server where you dont have a KVM/IPMI console
-
6
-
1
-
2
-
15 minutes ago, Gaty said:
Do you have any other hobbies other than being green?
General sysadminnery stuff, but para is probably my primary hobby
6 minutes ago, zwan1990 said:With support going out for internet explorer do you think it will affect paradise and if so how so?
IE Webview (the thing that BYOND runs on) isn't being phased out as so many apps run on it. The IE frontend however, will be going.
-
1
-
-
1 hour ago, Ziiro said:
rebase when
When you code itDont actually someone tried that once and didnt ask anyone and it got absolutely annihilated
-
1
-
-
31 minutes ago, Spacemanspark said:
give me 5$
Give me $500
-
2 hours ago, CapnKitty said:
who is better, Kiwi or Derek? also can I keep them both?
Kiwi is softer and much more of a huggable cat, meanwhile derek is an attention whore who scratches and sings to you at 3AM.
No you’re not having my cats.
-
1
-
1
-
-
28 minutes ago, Woje said:
What's your opinion on the current number of active staff tasked with reviewing and approving PRs when contrasted with the number of contributors, and if you think there is a problem, do you see a way to solve it?
The number of staff is low but a lot of us are exceedingly busy. A lot of the PRs are waiting on head approval at the moment, with me waiting for that before I review them. Some other PRs I wont review because I feel like its outside my skillset. I wouldnt really call it a problem because we are going along fine, stuff is still being merged, its just a little slower (especially since I been dealing with hosting stuff)
-
1
-
-
58 minutes ago, LiberatedWaveMan said:
Worst/most cursed PR you've ever had to review?
Sorry for the late response, I dug through my entire PR stack, and I have narrowed it down to 2
Cursed by design: https://github.com/ParadiseSS13/Paradise/pull/15539
Cursed by complexity: https://github.com/ParadiseSS13/Paradise/pull/15331
-
1
-
1
-
-
5 hours ago, Vargh said:
Moths when?
WYCI
5 hours ago, Skysorrow said:Nuggets, Tenders, or Fries? (Chicken or otherwise)
No real preference
-
1 minute ago, dearmochi said:
How do you like your bacon cooked?
Until it starts to become slightly crispy
-
22 minutes ago, Sirryan2002 said:
Would you prefer to port features or sprites from other servers or for Paradise to develop more of its own original/unique(albeit harder) gamemodes, features, looks, etc. Obviously it may be a combination of both but which of the two do you think we need to do more of?
Sprites are a mixed bag and I have no real opinion on them. Feature-wise, I want us to have OC to stand out from the rest of the hub because I dont want us turning into "TG clone number 72". I think we need more OC in general but I dont know where to start with that
23 minutes ago, Sirryan2002 said:And on top of that, of the various features we have on Paradise what game avenues do you think needs or could use a lot more development/work?
I want more variety in station goals and other modes so we have more things to do and stuff wont feel as samey, but perhaps I am just jaded
-
2
-
-
Just now, Shadeykins said:
moths?
I dont have a problem with them but youll need to convince the rest of leadership
-
1
-
2
-
3
-
-
30 minutes ago, Frank said:
What is your favorite beer?
I dont drink beer
-
1 hour ago, Charliminator said:
If you could give one piece of advice to all contributers, what would it be?
Ask heads and maints if a feature you are thinking about is actually wanted before putting large amounts of time and effort into it just to be closed. This seems like a no brainer but you would be surprised at how often people do things without asking, then get upset they wasted their time.
-
1
-
-
19 minutes ago, SabreML said:
How's your coffee addiction going, if that's not too personal a question?
Theres a bit of a story of events here.
About a month ago from today, I became ill (not with covid), which meant I had to be careful with what I ate to avoid throwing up. Leading up to this point, I had been having about 5 coffees a day. Going from that daily intake to complete cold turkey, and an illness, had me bed-ridden for 3 days straight where I could not even stand up. Sidenote during this point, the perfect server for hosting came up (the one we are using right now), but I couldn't even get out of bed, so I had to guide neca through purchasing it from my phone while sounding absolutely dead inside.
I have recovered since and I stayed off of coffee for 2 weeks, however now I am having about a cup or 2 a day, but no way in hell am I going back to where I was before. Being bed-ridden for 3 days and unable to even stand up was not a joyful experience in the slightest, and I would not recommend anyone go through what I did.
Sustaining on 5 a day was fine while it was happening, but I knew at some point, I would end up having a harsh break from it, and boy did I have one.
-
1
-
-
40 minutes ago, Denghis said:
What is a mechanic or feature another server has, (that Paradise does not) that you really enjoy?
Clarification, it does not need to be a feature that would fit Paradise at all, but more of in the context of that server.I honestly really like TG nanites. It is a super cool feature (even though theres no way its coming here). The weird and wonderful things you can do with them makes for a really in depth and advanced system thats a lot of fun to tinker round with.
35 minutes ago, Esenno said:Ok. I'll bite. What was with you at that time? I wasn't around for that but come on. Who on the team hasn't skimmed the notes/bans?
I was just not a mature individual at the time and was prone to being a complete and utter idiot. I had troubles in the past with holding grudges, but those issues are gone now and I am reformed. If I was still bad, well, I wouldn't be where I am now.
21 minutes ago, Trololiver112 said:pineapple on pizza?
No.
13 minutes ago, S34N said:Straying into design discussion with this one, but how do you personally feel about the fact that a lot of balance decisions are made by people who don't really play actively, or "bridge hobo" when they do log in?
This is a very mixed bag and not sure if I can give a coherent answer, but I tend to trust other people's balance views over my own (I rarely vote on balance changes unless its super obvious that its overpowered and awful).
14 minutes ago, S34N said:What would your thoughts be on a system not too dissimilar to EVE online's CSM?
We did have a design team eons ago but it is unlikely that it will ever return, and I dont feel qualified to make such a decision.
15 minutes ago, S34N said:Also, Apple or Android?
May come as a shock, but I use an iphone. The OS just feels nicer and less clunky for me, backups are easier to manage, the rest of my family has iphones which means I don't have to buy apps twice, not to mention use imessage to message them over wifi, and in general I just prefer the iOS user experience to the android one.
PC wise, windows for life. I used linux for a bit but it turned out to be an awful experience, and in regards to mac OS, you could not pay me to use that abomination.
-
Update: Gifs now rescale properly and the above issue is fixed. I needed to make the wiki code work with en_GB instead of en_US because I installed the UK language because I use a UK keyboard and I am not having my @ sign be in a stupid place.
-
3
-
-
1 minute ago, Qwertytoforty said:
Why did you want to host an AMAA?
Because I assume people would have weird questions to ask especially given 2017-era AA.
That and this is a good way to make it look like I am not at a total disconnect with the community
2021 - A space (hosting) odyssey
in Stories of NSS Cyberiad
Posted
Chapter 8 - Why emulation sucks and paravirtualisation is the future (EXTRA EXTRA: BYOND sucks)
So. The gameserver was now running on the new hardware. Everything was migrated, and it was all going nicely, apart from the gameserver having some spurious init time rises.
After a little investigation, I discovered that all of the things that increased were related to operations on the /icon type. Now this is where the "BYOND Sucks" part comes in.
Every time you do an operation on a /icon, its written to the HDD as cache. Don't ask me why BYOND doesnt use RAM, because that would make a ton more sense. Situations like roundstart have a lot of icon operations at once, which means the HDD gets absolutely thrashed.
Spikes here are from mid MC initialize, with SSatoms being the first and SSair being the second (Both have lots of icon operations)
Now heres where emulation and paravirtualisation tie into this. When you are adding hardware to a virtual machine, you have 2 choices, you can either emulate it, or partavirtualise it (Commonly referred to as VirtIO).
Emulation is a lot slower as it is having to pretend it is an actual device (Such as an intel e1000 network card), but since it is faking real hardware, it is much more compatible with other systems. Emulation also requires device emulation to live on a specific thread, and that thread jumps around a lot. In a worst case scenario, the SATA emulation process ends up on the same thread as BYOND, which makes for a really bad time)
VirtIO on the other hand just says to the VM "Ok youre a VM, heres a direct pipe to the storage, go use it". This has much better overhead (I am talking literally half the time for IO operations), not to mention higher throughput (VirtIO network cards operate at 10 gigabits per second)
VirtIO drives are natively available on Linux, however since the game was running windows, they werent available by default. To make matters more complicated, windows does not like it when you change your HDD interface (IE: going from SATA -> SCSI). The network card was relatively easy, just install drivers, reboot, change NIC, reset IP, done.
The HDD however, I had to modify the boot image to preload the VirtIO drivers, then change the boot ID to make it accept, and it was just generally pain all around, not to mention being on time constraints because "AA THE SERVER IS DOWN WHAT ARE YOU DOING". But hey, it got sorted, and as an added benefit of paravirtualisation, I could tell the HDD IO process to live on its own thread outside of everything else, preventing resource bottlenecks.
This had major performance improvements and reduced a lot of roundstart inconsistencies, and just made the server more performant overall.