9 ms·

Monitoring my Minecraft server with OpenTelemetry and Prometheus

94 points by mmanciop 1y ago

My kids demand SLOs stricter than Moon exploration technology, so I had to monitor our family’s Minecraft server Minecraft server like a pro. As luck would have it, I am one.

Calliope1 1y ago
[flagged]
- deleted 1y ago
  [deleted]
- Yasuraka 1y ago
  Why are you doing this?
  - mmanciop 1y ago
    Because I enjoy observability and monitoring a LOT, and because my kids nag me to hell and back when our home IT infrastructure is having a bad day.
    - Yasuraka 1y ago
      I was asking some LLM spammer
- acedTrex 1y ago
  Hi gippity
deleted 1y ago
[deleted]
strogonoff 1y ago
Monitoring and metric collection makes a lot of sense when you run a production system, or a personal but critical system. Promoting a telemetry solution when it comes to a hobby server, which you host for yourself and which can’t bankrupt you by running up a massive AWS bill, doesn’t seem to make much sense when simply bottling it up in Docker and being able to restart or recreate at will is enough (mount volumes for logs and persistent data, back it up, and you’re good). With games like Minecraft in particular there’s value in being able to have multiple servers with different worlds, perhaps different mods, etc. If you decide not to have more servers because they are snowflakes you do not have time to set up monitoring for then you rob yourself and your players of the opportunity to have more fun. Furthermore, containerizing it allows you to upgrade as new game versions come out quickly by simply spinning up a new container with your preexisting world as a test, and you get you basic system resource usage monitoring built-in. What I think could be a more interesting exercise is a dashboard for friends or family that allows to manage the lifetime and configuration of their respective containers.
- dpe82 1y ago
  Have you never just built something for fun?
  - strogonoff 1y ago
    I have built a panel like the one I mentioned for fun with friends! The goal of my comment was to highlight opportunities for more fun and less what seems like toil. Furthermore, this is an article about a telemetry solution posted on a site of that telemetry solution. They make money from this.
    - dewey 1y ago
      One persons toil is another persons fun.
      - strogonoff 1y ago
        And sometimes a person is paid to pretend toil is fun. We are talking about spending hours setting up telemetry instead of playing a game.
        mmanciop 1y ago
        I swear I had a lot of fun setting doing the setup. I am also a massive observability nerd, so YMMV :-)
        strogonoff 1y ago
        I believe you! Just due to your affiliation I wanted to highlight to any newbie SREs in the audience that perhaps there is a better way. I still think my approach is better, but we can do things differently.
        mmanciop 1y ago
        Indeed if there were “official” container images out there, I might have instead run the server on Google Cloud Run or AWS AppRunner, without having to take care of the Linux underneath. Or an Amazon ECS task. I don’t have a Kubernetes cluster, but I will at some point make a version of this blog to run it on K8s :-)
doabell 1y ago
> I am a man of simple tastes, and running the “vanilla” Minecraft server as a Systemd unit on a Linux VM in the cloud Minecraft is famously under-optimized and needy in terms of CPU frequency. If running a vanilla (no server mods) version, then using something optimized, like PaperMC is a better idea for datacenter VMs. (Until you need to dupe sand or something.) The other route is installing a bunch of optimization mods - some really do help.
- strogonoff 1y ago
  From my understanding, Paper and the like are good for Minecraft servers focused around specific mini-games (rather than freedorm building), and are the only sensible choice for servers with many people (or not that many people, but really underpowered hardware). However, they may be a problem if players are sensitive to possible non-vanilla behaviour (as you mentioned, and it’s not limited to cheaty duping). Thankfully, spinning up a server with a selection of performance mods is very easy these days. Various tricks like pre-generating chunks in advance also help.
  - treyd 1y ago
    It's kinda nuts. The upstream mojang server binary starts to groan if you have >4-5 players on the same server doing stuff. They've really been dropping the ball on optimization in recent years. Paper is good enough for anyone but very technical players pushing to the limits of redstone tick timing logic, entity behavior, chunk loading mechanics, etc. These don't matter even for advanced players doing normal things.
    - mmanciop 1y ago
      I actually had to splurge got 2 VCPUs on Digital Ocean to avoid "skipping ticks" and it does sound pretty nuts to me. We play max 3 players. I would expect the server with such a load to be able to run on a slightly tuned up toaster.
      - skrtskrt 1y ago
Lirael 1y ago
[dead]
darknavi 1y ago
> Microsoft made things confusing by adding the Bedrock server, which reportedly uses a combination of C, C# and Java, No C# in Bedrock. No Java unless you're talking about the Android versions. Very little C. It's mostly C++.
- mmanciop 1y ago
  Thanks for setting me straight :-) I updated the article to reflect that.
cpburns2009 1y ago
> The minecraft-prometheus-exporter ... which uses Fabric, another way to run Minecraft servers with mods. Like Bukkit, Fabric was not an option for me. Forge and its recent fork Neoforge are supported too.

94 points by mmanciop 1y ago

My kids demand SLOs stricter than Moon exploration technology, so I had to monitor our family’s Minecraft server Minecraft server like a pro. As luck would have it, I am one.

Calliope1 1y ago

[flagged]

deleted 1y ago
[deleted]
Yasuraka 1y ago
Why are you doing this?
- mmanciop 1y ago
  Because I enjoy observability and monitoring a LOT, and because my kids nag me to hell and back when our home IT infrastructure is having a bad day.
  - Yasuraka 1y ago
    I was asking some LLM spammer
acedTrex 1y ago
Hi gippity

deleted 1y ago

[deleted]

strogonoff 1y ago

Monitoring and metric collection makes a lot of sense when you run a production system, or a personal but critical system. Promoting a telemetry solution when it comes to a hobby server, which you host for yourself and which can’t bankrupt you by running up a massive AWS bill, doesn’t seem to make much sense when simply bottling it up in Docker and being able to restart or recreate at will is enough (mount volumes for logs and persistent data, back it up, and you’re good). With games like Minecraft in particular there’s value in being able to have multiple servers with different worlds, perhaps different mods, etc. If you decide not to have more servers because they are snowflakes you do not have time to set up monitoring for then you rob yourself and your players of the opportunity to have more fun. Furthermore, containerizing it allows you to upgrade as new game versions come out quickly by simply spinning up a new container with your preexisting world as a test, and you get you basic system resource usage monitoring built-in. What I think could be a more interesting exercise is a dashboard for friends or family that allows to manage the lifetime and configuration of their respective containers.

dpe82 1y ago
Have you never just built something for fun?
- strogonoff 1y ago
  I have built a panel like the one I mentioned for fun with friends! The goal of my comment was to highlight opportunities for more fun and less what seems like toil. Furthermore, this is an article about a telemetry solution posted on a site of that telemetry solution. They make money from this.
  - dewey 1y ago
    One persons toil is another persons fun.
    - strogonoff 1y ago
      And sometimes a person is paid to pretend toil is fun. We are talking about spending hours setting up telemetry instead of playing a game.
      - mmanciop 1y ago
        I swear I had a lot of fun setting doing the setup. I am also a massive observability nerd, so YMMV :-)
        strogonoff 1y ago
        I believe you! Just due to your affiliation I wanted to highlight to any newbie SREs in the audience that perhaps there is a better way. I still think my approach is better, but we can do things differently.
        mmanciop 1y ago
        Indeed if there were “official” container images out there, I might have instead run the server on Google Cloud Run or AWS AppRunner, without having to take care of the Linux underneath. Or an Amazon ECS task. I don’t have a Kubernetes cluster, but I will at some point make a version of this blog to run it on K8s :-)

doabell 1y ago

> I am a man of simple tastes, and running the “vanilla” Minecraft server as a Systemd unit on a Linux VM in the cloud Minecraft is famously under-optimized and needy in terms of CPU frequency. If running a vanilla (no server mods) version, then using something optimized, like PaperMC is a better idea for datacenter VMs. (Until you need to dupe sand or something.) The other route is installing a bunch of optimization mods - some really do help.

strogonoff 1y ago
From my understanding, Paper and the like are good for Minecraft servers focused around specific mini-games (rather than freedorm building), and are the only sensible choice for servers with many people (or not that many people, but really underpowered hardware). However, they may be a problem if players are sensitive to possible non-vanilla behaviour (as you mentioned, and it’s not limited to cheaty duping). Thankfully, spinning up a server with a selection of performance mods is very easy these days. Various tricks like pre-generating chunks in advance also help.
- treyd 1y ago
  It's kinda nuts. The upstream mojang server binary starts to groan if you have >4-5 players on the same server doing stuff. They've really been dropping the ball on optimization in recent years. Paper is good enough for anyone but very technical players pushing to the limits of redstone tick timing logic, entity behavior, chunk loading mechanics, etc. These don't matter even for advanced players doing normal things.
  - mmanciop 1y ago
    I actually had to splurge got 2 VCPUs on Digital Ocean to avoid "skipping ticks" and it does sound pretty nuts to me. We play max 3 players. I would expect the server with such a load to be able to run on a slightly tuned up toaster.
    - skrtskrt 1y ago

Lirael 1y ago

[dead]

darknavi 1y ago

> Microsoft made things confusing by adding the Bedrock server, which reportedly uses a combination of C, C# and Java, No C# in Bedrock. No Java unless you're talking about the Android versions. Very little C. It's mostly C++.

mmanciop 1y ago
Thanks for setting me straight :-) I updated the article to reflect that.

cpburns2009 1y ago

> The minecraft-prometheus-exporter ... which uses Fabric, another way to run Minecraft servers with mods. Like Bukkit, Fabric was not an option for me. Forge and its recent fork Neoforge are supported too.