Optimizing the Go side of things to get allocations down, its pretty memory hungry during backfill, loooots of stuff going on.
Setting GOMEMLIMIT=6GiB and GOGC=80 for now
Setting GOMEMLIMIT=6GiB and GOGC=80 for now
March 22, 2025 at 12:43 AM
Optimizing the Go side of things to get allocations down, its pretty memory hungry during backfill, loooots of stuff going on.
Setting GOMEMLIMIT=6GiB and GOGC=80 for now
Setting GOMEMLIMIT=6GiB and GOGC=80 for now
PSA: runtime/debug.SetCrashOutput + GOMEMLIMIT can lead to OOM kills. This was fun to debug :).
github.com/golang/go/is...
github.com/golang/go/is...
runtime/debug.SetCrashOutput example interacts badly with GOMEMLIMIT · Issue #73490 · golang/go
We recently noticed some OOM kills, and after a fun debugging adventure involving core dumps, we concluded that the issue was caused by our usage of the SetCrashOutput API which mirrors the example...
github.com
April 24, 2025 at 11:04 AM
PSA: runtime/debug.SetCrashOutput + GOMEMLIMIT can lead to OOM kills. This was fun to debug :).
github.com/golang/go/is...
github.com/golang/go/is...
zenn.dev/reality_tech/arti...
OOMKill の犯人は CPU だった：GOGC と GOMEMLIMIT を実測で設計し直す
OOMKill の犯人は CPU だった：GOGC と GOMEMLIMIT を実測で設計し直す
OOMKill の犯人は CPU だった：GOGC と GOMEMLIMIT を実測で設計し直す
zenn.dev
September 30, 2026 at 8:10 PM
zenn.dev/reality_tech/arti...
OOMKill の犯人は CPU だった：GOGC と GOMEMLIMIT を実測で設計し直す
OOMKill の犯人は CPU だった：GOGC と GOMEMLIMIT を実測で設計し直す
Caddy will now automatically adapt to the available memory. This improves performance of Caddy but also of Mercure and FrankenPHP, especially when using containers (Docker, Kubernetes, LXC…).
feat: automatically set GOMEMLIMIT by dunglas · Pull Request #6809 · caddyserver/caddy
Setting the Go runtime memory limit can improve significantly the performance of Caddy (and FrankenPHP and Mercure, according to our tests), especially on low-resources machines and containers. Thi...
github.com
January 27, 2025 at 5:02 PM
Caddy will now automatically adapt to the available memory. This improves performance of Caddy but also of Mercure and FrankenPHP, especially when using containers (Docker, Kubernetes, LXC…).
Why Set GOMEMLIMIT for Go Apps in Containers/Kubernetes Pods? 🔽
- Linux containers isolate network and fs but share host CPU cores/RAM.
- cgroup memory limits aren’t considered by the Go runtime.
- The Go GC assumes the host’s total memory.
- Ignoring cgroup limits risks terminations by OOM Killer.
- Linux containers isolate network and fs but share host CPU cores/RAM.
- cgroup memory limits aren’t considered by the Go runtime.
- The Go GC assumes the host’s total memory.
- Ignoring cgroup limits risks terminations by OOM Killer.
December 22, 2024 at 5:55 PM
Why Set GOMEMLIMIT for Go Apps in Containers/Kubernetes Pods? 🔽
- Linux containers isolate network and fs but share host CPU cores/RAM.
- cgroup memory limits aren’t considered by the Go runtime.
- The Go GC assumes the host’s total memory.
- Ignoring cgroup limits risks terminations by OOM Killer.
- Linux containers isolate network and fs but share host CPU cores/RAM.
- cgroup memory limits aren’t considered by the Go runtime.
- The Go GC assumes the host’s total memory.
- Ignoring cgroup limits risks terminations by OOM Killer.
Mercure 0.18.3 is out! It is now compile with Go 1.24.
Release v0.18.3 · dunglas/mercure
What's Changed feat: automatically set GOMEMLIMIT by @dunglas in #1010 fix: metrics with Caddy 2.9 by @IndraGunawan in #1007 chore: compile with Go 1.24 by @dunglas in #1008 docs: updated docs for...
github.com
February 17, 2025 at 4:27 PM
Mercure 0.18.3 is out! It is now compile with Go 1.24.
If it runs in a container, the thing that finally cured my GOGC abuse: GOGC=off plus GOMEMLIMIT at ~85-90% of the cgroup limit. GC then runs when the heap nears the limit instead of at every 2x growth. It's a soft cap though, so leave headroom for stacks/cgo or the kernel still OOMKills you.
September 30, 2026 at 9:34 AM
If it runs in a container, the thing that finally cured my GOGC abuse: GOGC=off plus GOMEMLIMIT at ~85-90% of the cgroup limit. GC then runs when the heap nears the limit instead of at every 2x growth. It's a soft cap though, so leave headroom for stacks/cgo or the kernel still OOMKills you.
Can we do GOMEMLIMIT too?
April 7, 2025 at 12:32 PM
Can we do GOMEMLIMIT too?
go 1.19から導入されたGOMEMLIMIT環境変数をcgroupsのmemory limitに応じていい感じに調整してくれるライブラリ。コンテナ内でもいい感じのGOMEMLIMITを設定してくれるってことだろう。わいわい。
GitHub - KimMachineGun/automemlimit: Automatically set GOMEMLIMIT to match Linux cgroups(7) memory limit.
Automatically set GOMEMLIMIT to match Linux cgroups(7) memory limit. - KimMachineGun/automemlimit
github.com
April 20, 2024 at 1:37 PM
go 1.19から導入されたGOMEMLIMIT環境変数をcgroupsのmemory limitに応じていい感じに調整してくれるライブラリ。コンテナ内でもいい感じのGOMEMLIMITを設定してくれるってことだろう。わいわい。
Funny
- name: GOMEMLIMIT
valueFrom:
resourceFieldRef:
resource: limits.memory
misses in most go charts/descriptor...just the promise to fail if used
#kubernetes #golang
- name: GOMEMLIMIT
valueFrom:
resourceFieldRef:
resource: limits.memory
misses in most go charts/descriptor...just the promise to fail if used
#kubernetes #golang
July 18, 2025 at 4:14 PM
Funny
- name: GOMEMLIMIT
valueFrom:
resourceFieldRef:
resource: limits.memory
misses in most go charts/descriptor...just the promise to fail if used
#kubernetes #golang
- name: GOMEMLIMIT
valueFrom:
resourceFieldRef:
resource: limits.memory
misses in most go charts/descriptor...just the promise to fail if used
#kubernetes #golang
🗣️ Emin Laletović explains how to address memory issues in Go apps on Kubernetes using `GOMEMLIMIT`, improving container memory management
Watch: https://ku.bz/7fnF-tJ8R
Watch: https://ku.bz/7fnF-tJ8R
November 21, 2025 at 2:37 PM
🗣️ Emin Laletović explains how to address memory issues in Go apps on Kubernetes using `GOMEMLIMIT`, improving container memory management
Watch: https://ku.bz/7fnF-tJ8R
Watch: https://ku.bz/7fnF-tJ8R
We've done this before so we don't have to pull in another dependency.
```
env:
- name: GOMEMLIMIT
valueFrom:
resourceFieldRef:
resource: limits.memory
- name: GOMAXPROCS
valueFrom:
resourceFieldRef:
resource: limits.cpu
```
```
env:
- name: GOMEMLIMIT
valueFrom:
resourceFieldRef:
resource: limits.memory
- name: GOMAXPROCS
valueFrom:
resourceFieldRef:
resource: limits.cpu
```
May 1, 2025 at 1:14 PM
We've done this before so we don't have to pull in another dependency.
```
env:
- name: GOMEMLIMIT
valueFrom:
resourceFieldRef:
resource: limits.memory
- name: GOMAXPROCS
valueFrom:
resourceFieldRef:
resource: limits.cpu
```
```
env:
- name: GOMEMLIMIT
valueFrom:
resourceFieldRef:
resource: limits.memory
- name: GOMAXPROCS
valueFrom:
resourceFieldRef:
resource: limits.cpu
```
This guide shows how to right-size Kubernetes pod resources (CPU, memory, ephemeral storage) using real Prometheus metrics, Go runtime env tuning (GOMAXPROCS, GOMEMLIMIT), and node-level capacity planning
➜ https://ku.bz/Mn_1bDnwh
➜ https://ku.bz/Mn_1bDnwh
May 19, 2025 at 3:11 PM
This guide shows how to right-size Kubernetes pod resources (CPU, memory, ephemeral storage) using real Prometheus metrics, Go runtime env tuning (GOMAXPROCS, GOMEMLIMIT), and node-level capacity planning
➜ https://ku.bz/Mn_1bDnwh
➜ https://ku.bz/Mn_1bDnwh
This guide shows how to right-size Kubernetes pod resources (CPU, memory, ephemeral storage) using real Prometheus metrics, Go runtime env tuning (GOMAXPROCS, GOMEMLIMIT), and node-level capacity planning
➤ https://ku.bz/Mn_1bDnwh
➤ https://ku.bz/Mn_1bDnwh
June 19, 2025 at 3:31 PM
This guide shows how to right-size Kubernetes pod resources (CPU, memory, ephemeral storage) using real Prometheus metrics, Go runtime env tuning (GOMAXPROCS, GOMEMLIMIT), and node-level capacity planning
➤ https://ku.bz/Mn_1bDnwh
➤ https://ku.bz/Mn_1bDnwh
automemlimit is a Go library that auto-sets `GOMEMLIMIT` to 90% of the Linux cgroup memory limit
It ensures Go GC stays container-aware and optionally falls back to system memory if cgroups aren’t available
➜ https://ku.bz/cCfD2QbXw
It ensures Go GC stays container-aware and optionally falls back to system memory if cgroups aren’t available
➜ https://ku.bz/cCfD2QbXw
December 30, 2025 at 7:06 PM
automemlimit is a Go library that auto-sets `GOMEMLIMIT` to 90% of the Linux cgroup memory limit
It ensures Go GC stays container-aware and optionally falls back to system memory if cgroups aren’t available
➜ https://ku.bz/cCfD2QbXw
It ensures Go GC stays container-aware and optionally falls back to system memory if cgroups aren’t available
➜ https://ku.bz/cCfD2QbXw
[Перевод] Как Go 1.24 и Swiss Tables вернули нам 200 гигабайт памяти Команда Go for Devs подготовила перевод статьи о том, ка...
#Go #map #SwissTables #производительность #оптимизация #память #GOMEMLIMIT #Kubernetes #профилирование #datadog
Origin | Interest | Match
#Go #map #SwissTables #производительность #оптимизация #память #GOMEMLIMIT #Kubernetes #профилирование #datadog
Origin | Interest | Match
September 22, 2025 at 12:14 PM
[Перевод] Как Go 1.24 и Swiss Tables вернули нам 200 гигабайт памяти Команда Go for Devs подготовила перевод статьи о том, ка...
#Go #map #SwissTables #производительность #оптимизация #память #GOMEMLIMIT #Kubernetes #профилирование #datadog
Origin | Interest | Match
#Go #map #SwissTables #производительность #оптимизация #память #GOMEMLIMIT #Kubernetes #профилирование #datadog
Origin | Interest | Match
[Перевод] Как Go 1.24 и Swiss Tables вернули нам 200 гигабайт памяти Команда Go for Devs подготовила перевод статьи о том, ка...
#Go #map #SwissTables #производительность #оптимизация #память #GOMEMLIMIT #Kubernetes #профилирование #datadog
Origin | Interest | Match
#Go #map #SwissTables #производительность #оптимизация #память #GOMEMLIMIT #Kubernetes #профилирование #datadog
Origin | Interest | Match
September 22, 2025 at 9:43 AM
[Перевод] Как Go 1.24 и Swiss Tables вернули нам 200 гигабайт памяти Команда Go for Devs подготовила перевод статьи о том, ка...
#Go #map #SwissTables #производительность #оптимизация #память #GOMEMLIMIT #Kubernetes #профилирование #datadog
Origin | Interest | Match
#Go #map #SwissTables #производительность #оптимизация #память #GOMEMLIMIT #Kubernetes #профилирование #datadog
Origin | Interest | Match
Check out this realistic DevOps challenge I prepared to illustrate the impact of a missing GOMEMLIMIT setting on a Go Kubernetes deployment with a strict memory limit.
labs.iximiuz.com/challenges/f...
labs.iximiuz.com/challenges/f...
Diagnose and Resolve Stability Issues in a Go Application Deployment | Challenge
This challenge focuses on debugging memory usage issues in a Go application deployed in a Kubernetes cluster.
The goal is to ensure the application can handle moderate traffic without crashing,
even w...
labs.iximiuz.com
December 22, 2024 at 6:00 PM
Check out this realistic DevOps challenge I prepared to illustrate the impact of a missing GOMEMLIMIT setting on a Go Kubernetes deployment with a strict memory limit.
labs.iximiuz.com/challenges/f...
labs.iximiuz.com/challenges/f...
A #telemetry spike can push the #OpenTelemetry Collector over the edge.
When buffers swell, #OOM means data loss - right when #observability matters most.
Learn how the Memory Limiter Processor applies backpressure (incl. #GOMEMLIMIT).
Full guide: dash0.link/otel-memory-...
When buffers swell, #OOM means data loss - right when #observability matters most.
Learn how the Memory Limiter Processor applies backpressure (incl. #GOMEMLIMIT).
Full guide: dash0.link/otel-memory-...
Mastering the OpenTelemetry Memory Limiter Processor · Dash0
Prevent data loss and OutofMemory errors in your OpenTelemetry Collector by understanding the essential Memory Limiter Processor configurations and behaviors
dash0.link
January 5, 2026 at 1:01 PM
A #telemetry spike can push the #OpenTelemetry Collector over the edge.
When buffers swell, #OOM means data loss - right when #observability matters most.
Learn how the Memory Limiter Processor applies backpressure (incl. #GOMEMLIMIT).
Full guide: dash0.link/otel-memory-...
When buffers swell, #OOM means data loss - right when #observability matters most.
Learn how the Memory Limiter Processor applies backpressure (incl. #GOMEMLIMIT).
Full guide: dash0.link/otel-memory-...
OOMKill の犯人は CPU だった：GOGC と GOMEMLIMIT を実測で設計し直す
https://zenn.dev/reality_tech/articles/f6305331bccee0
https://zenn.dev/reality_tech/articles/f6305331bccee0
OOMKill の犯人は CPU だった：GOGC と GOMEMLIMIT を実測で設計し直す
zenn.dev
September 2, 2026 at 1:51 AM
OOMKill の犯人は CPU だった：GOGC と GOMEMLIMIT を実測で設計し直す
https://zenn.dev/reality_tech/articles/f6305331bccee0
https://zenn.dev/reality_tech/articles/f6305331bccee0
Good framing. The follow-up gotcha: the runtime has to actually see the cgroup limit. JVM: -XX:MaxRAMPercentage=75 instead of a fixed -Xmx (heap is not the only memory). Go: GOMEMLIMIT at ~90% of the limit so GC gets aggressive before the kernel does. Both turn 137s into GC pauses.
September 1, 2026 at 9:33 AM
Good framing. The follow-up gotcha: the runtime has to actually see the cgroup limit. JVM: -XX:MaxRAMPercentage=75 instead of a fixed -Xmx (heap is not the only memory). Go: GOMEMLIMIT at ~90% of the limit so GC gets aggressive before the kernel does. Both turn 137s into GC pauses.
medium.com/@yalovoy/wha...
Most Go GC tuning advice focuses on stop-the-world pauses that last 200μs. The real p99 killer is mark assist -where your request-serving goroutines get conscripted into GC work.
#golang
#garbage-collection
#performance
#backend-engineering
#latency
Most Go GC tuning advice focuses on stop-the-world pauses that last 200μs. The real p99 killer is mark assist -where your request-serving goroutines get conscripted into GC work.
#golang
#garbage-collection
#performance
#backend-engineering
#latency
What Happens When Go’s Garbage Collector Pauses Your Latency-Sensitive Service
How default GOGC=100 creates p99 latency spikes, why most benchmarks miss it, and why GOMEMLIMIT might be the correct answer you haven’t…
medium.com
July 18, 2026 at 5:59 AM
medium.com/@yalovoy/wha...
Most Go GC tuning advice focuses on stop-the-world pauses that last 200μs. The real p99 killer is mark assist -where your request-serving goroutines get conscripted into GC work.
#golang
#garbage-collection
#performance
#backend-engineering
#latency
Most Go GC tuning advice focuses on stop-the-world pauses that last 200μs. The real p99 killer is mark assist -where your request-serving goroutines get conscripted into GC work.
#golang
#garbage-collection
#performance
#backend-engineering
#latency
automemlimit is a Go library that auto-sets `GOMEMLIMIT` to 90% of the Linux cgroup memory limit
It ensures Go GC stays container-aware and optionally falls back to system memory if cgroups aren’t available
➤ https://ku.bz/cCfD2QbXw
It ensures Go GC stays container-aware and optionally falls back to system memory if cgroups aren’t available
➤ https://ku.bz/cCfD2QbXw
January 30, 2026 at 7:41 PM
automemlimit is a Go library that auto-sets `GOMEMLIMIT` to 90% of the Linux cgroup memory limit
It ensures Go GC stays container-aware and optionally falls back to system memory if cgroups aren’t available
➤ https://ku.bz/cCfD2QbXw
It ensures Go GC stays container-aware and optionally falls back to system memory if cgroups aren’t available
➤ https://ku.bz/cCfD2QbXw
🐝New Cilium Contributor🐝
The newest Cilium contributor is Donnie McMahan
They helped add default divisor for GOMEMLIMIT to satisfy ArgoCD diff in Cilium
https://github.com/cilium/cilium/pull/30635/
The newest Cilium contributor is Donnie McMahan
They helped add default divisor for GOMEMLIMIT to satisfy ArgoCD diff in Cilium
https://github.com/cilium/cilium/pull/30635/
helm: Add default `divisor` for `GOMEMLIMIT` by jdmcmahan · Pull Request #30635 · cilium/cilium
Please ensure your pull request adheres to the following guidelines:
For first time contributors, read Submitting a pull request
All code is covered by unit and/or runtime tests where feasible.
...
github.com
November 7, 2024 at 8:12 PM
🐝New Cilium Contributor🐝
The newest Cilium contributor is Donnie McMahan
They helped add default divisor for GOMEMLIMIT to satisfy ArgoCD diff in Cilium
https://github.com/cilium/cilium/pull/30635/
The newest Cilium contributor is Donnie McMahan
They helped add default divisor for GOMEMLIMIT to satisfy ArgoCD diff in Cilium
https://github.com/cilium/cilium/pull/30635/