Everything started with decino wanting a special thing to do for his 40k subscriber youtube event, since i had a bit of background using linux servers i immediately started testing some DOOM multiplayer ports with the guys on decino's patreon discord server.

We decided to use Odamex as the multiplayer port since it was easy to tweak some of the variables that were important in that time, like collision, making the spawned weapons stay on the floor after picking, keep keycards, etc...

Then, the public testing. He called it decino's stress test #1. I was nervous before starting the stream, i was expecting everything except the amount of people that entered (and tried to). Everything went well until more than 80 people started to play, then it lagged like hell (haha), so i limited the max amount of players to 50ish and the server ran almost flawless.

DOOR STUCK! was being shout by decino, as an horde of people tried to open a door that anyone can trigger. Up and down, hell unleashed. It really was a miracle that it finally opened and everyone could play just fine.

We played DOOM II, Plutonia, Ultimate DOOM and Deathmatch. Oh crap what an insane deathmatch, from telefragging to massive multikills with the rocket launcher and BFG.

---

Second round; we managed to get some attention from the ODAMEX team, i provided them some stats and they did black magic, a simple modification to the code and a lot of CPU processing was cut from the main thread.

This fix was prosimising and we hoped we could manage +150 players without hiccups. The day came and testing began. Goal achieved.

We were hitting apprx. 180 players and there was again, CPU was the bottleneck. I had the idea to upgrade the VM hosted on Google Cloud -> Twice the CPU cores, 4x the RAM, blah blah. Then i realized i was an idiot and learned something; DOOM was released before multithreading was a thing and so, every upgrade was worthless. But anyway, the testing was a real success and we were happy with 150 players.

Holy $%#t, we realized that we most surely "broke the record" (there's no official record, but i don't care) for the most players on any Coop or Deathmatch DOOM match, and i was part of the staff. AMAZING!

Google Cloud invoice arrived!, i calculated the VM costs including the temporary upgrades and it was affordable. Until i didn't noticed something, important... We had a lot of outgoing traffic due to the amount of players, and the cost per gig of Google Cloud was outrageous.

After that, i decided to move out to DigitalOcean and it was a miracle. They have servers in Amsterdam, their network infrastructure is solid, simply and elegant admin UI, CPU Optimized VM's (some last gen CPU's, which was the only upgrade that can be done to improve the server's performance), and most importantly, their cost per EXCESS traffic for each VM is 1/10th of Google Cloud's.

---

DigitalOcean's VM debut was amazing, the third stress test ran smooth as we always wanted. And even better, the cost of these events was really low.

---

Nice! now the final private attempt with a little surprise; Experimental stuff

What's more fun than adding custom stuff to improve the gameplay mechanics of DOOM, a very smart ODAMEX dev volunteered to make us these unholy things. It was the same wizard that made the 150 players possible with a 4 line fix, spectatives were high and he delivered. Only Berserk, door and elevator fixes among other stuff. It worked fine with the discord guys.

---

We were Ready! The 40k Event

The same server configuration with some tweaks to allow the new gamemodes to run, some scripts to make the wad changes easier, polish stuff, new odamex binaries, everything was in order. I even asked decino if he could make some kind of lobby while the players joined the server. He agreed.

Ok, i was as nervous as the first time, he started the stream and i opened the server. EVERYTHING WENT WRONG. Lag appeared with only 20 players, stutters like hell, i panicked and decino too. I looked up to the server stats and what i saw was a nightmare;

The amount of traffic that was incoming to the server was more than 12 times the usual, 1.51Gbps!. Something was very wrong, i immediately thought it was a DDoS attack and reported it to decino and to the odamex dev's (just to be sure it wasn't odamex client by itself).

I tried to fix the issue changing some netrate variables but it didn't worked at all. 1 hour passed, still not fixed. Then 2 and 3 hours, i was stressed as heck, and then i realized, maybe it's the odamex server or client binaries. I downgraded the server to the older version (without the experimental stuff) and BAM!. Magic... Everything went back to normal, i told decino and the 40k event started as it should. Lesson: DO NOT USE EXPERIMENTAL CODE IN PRODUCTION!

After some hours, the stream was done and after that horrible time, the aftertaste was kinda bitter, so i immediately logged off and fell asleep. We didn't talked about the event for like 2 weeks and there was when we realized we got cocky with the new experimental features.

We had learned a valuable lesson and personally, it was a truly unique experience to be part of this event, so i'm immensely grateful to the people involved in this.