Why Telegram Bot Notifications Die: IPv6, DNS, and a 400 That Took a Day
_Originally published on my blog — read it there with the full context, code and updates._
I run an automated monitoring pipeline that watches Carousell for newly-listed
items and pings me on Telegram when something fresh appears. It is one container
on my Synology NAS, driven through Portainer, with a single Python process as
`PID 1` on a Docker bridge network. One day it stopped pinging. The container was healthy, the scraper was still archiving
listings into NocoDB — but every notification died with the same three
characters in the log:
telegram sendPhoto failed: 400 ... nginx/1.30.1
That string, `nginx/1.30.1`, became the villain of a day-long debugging
session. This is the story of how a deceptively small bug turned out to be
three separate problems stacked on top of each other — and how I'd find each
one faster next time.
## The problem
The bot had two jobs: **collect** listings, and **notify** me. Collection
worked. Notification didn't. The log said `400 Bad Request` and pointed at a
server identifying itself as `nginx/1.30.1`.
My first assumption was textbook: **something in the network is intercepting
the request.** `nginx/1.30.1` is not a header I associated with Telegram's
API. A 400 with an HTML body (Telegram returns JSON even on errors) smelled
like a local reverse proxy or a VPN gateway rejecting traffic before it ever
left the machine.
Assumptions like that are cheap. They're also wrong a lot. Let me walk
through what I actually found.
## What I tried, and why each thing failed
### Round 1: it must be DNS / IPv6
I checked `getMe` from inside the container. It worked. Then I checked
`sendMessage`. It failed. Same token, same container, one method up, one down.
The giveaway was in how the two endpoints resolved:
# inside the container
$ getent hosts api.telegram.org
2001:67c:4e8:f004::9 api.telegram.org
The container resolved `api.telegram.org` to **IPv6 only** — a single `AAAA`
record and no `A` record. The container ran on a Docker bridge network with
**no IPv6 connectivity**. So every request that needed `api.telegram.org` was
trying to reach an IPv6 address it had no route to.
The fix looked obvious: pin the IPv4 address with `extra_hosts`. I did, and
the DNS answer became exactly what I expected:
149.154.167.220 api.telegram.org
And the notifications **still failed.** Same 400. My proud fix did nothing.
### Round 2: I picked the wrong IP
Here's a mistake worth remembering: I hard-coded `149.154.167.220` from
memory. When I finally queried a public resolver, the real `A` record for
`api.telegram.org` was different:
149.154.166.110 api.telegram.org # what DNS actually says
149.154.167.220 api.telegram.org # what I hard-coded
`167.220` is _inside_ Telegram's IP range (`149.154.160.0/20`), so it's not
"wrong" in the sense of being somebody else's server — but it's not the
active endpoint, and bot-API traffic to it behaved unpredictably. I switched
to `166.110`. It still failed. So the IP wasn't the whole story either.
### Round 3: the same command, two different outcomes
This was the moment the whole thing got weird. Inside one container, at
almost the same second:
* the **long-running monitor process** (`PID 1`) → `sendPhoto` failed, 400
* a **fresh`docker exec` process** running identical code → `sendPhoto` succeeded, 200
Same container. Same code. Same payload. Same second.
I ruled out token, chat_id, emoji in the caption, Unicode quotes, image
format, environment variables, and the image URL itself — I download-tested
the photo and it was a perfectly valid JPEG. Nothing in the code differed
between the two paths.
When you hit a contradiction like that — "identical inputs, opposite
outputs" — the answer is usually that the inputs _aren't_ identical. In my `docker exec` test, I had tested against a lightweight sample image; but the PID 1 loop was processing real scraped listing photos where binary byte sequences happened to break multipart boundary parsing. The binary payloads were not the same. But
finding the difference took one more frame change.
### Round 4: `nginx/1.30.1` is Telegram
I re-read the raw 400 response instead of assuming. The response was an HTML
"400 Bad Request" page, and the `Server` header was `nginx/1.30.1`.
**That nginx is Telegram's own edge.** Telegram's API sits behind nginx
(version 1.30.1), and when a request is malformed _before_ it reaches their
application layer, nginx itself answers with a plain 400 page — no JSON, no
friendly error code.
That reframed everything. The request wasn't being intercepted. It was
**reaching Telegram, and Telegram's nginx was rejecting it as malformed.**
And the malformed thing was the one part I hadn't scrutinized: the
`sendPhoto` multipart body.
## The fix
My original helper assembled the multipart body by treating headers and binary payloads as a list of "lines" and joining them with `b"\r\n".join(body_lines)` — the kind of "obviously fine" shortcut that ships a framing bug:
# BROKEN: pre-terminated chunks or line-split binary data re-joined with b"\r\n"
body = b"\r\n".join(body_lines)
It fails in two ways, and neither is `bytes.join()` rewriting the JPEG — `b"\r\n".join()` only inserts the delimiter _between_ list elements and never touches bytes inside an element. The damage happens upstream. First, if the payload ever passes through line-oriented handling such as `.splitlines()`, every lone `0x0D` or `0x0A` byte inside the compressed image stream is treated as a line break and rewritten to `\r\n`, which mutates the JPEG _and_ desynchronises the `Content-Length` that was computed from the original file. Second, if any chunk in `body_lines` already carries its own trailing `\r\n`, the join adds a second one, so the body reads `--boundary\r\n\r\nContent-Disposition: ...` — and in MIME framing a blank line terminates the part headers, so that part becomes an anonymous body whose first line is the `Content-Disposition` header. Telegram's edge `nginx/1.30.1` rejects that as malformed with an HTML 400 long before the Bot API parses anything.
That also closes the Round 3 contradiction. The `docker exec` one-liner I typed by hand sent a minimal two-field payload from the freshly edited script, while the long-running `PID 1` process was still executing the old in-memory framing path against a real marketplace JPEG whose bytes were long enough to hit the doubled-`\r\n` case — same function, different input, one 200 and one 400.
The correct approach is to frame every part explicitly and concatenate the byte chunks verbatim with `b"".join(parts)`:
boundary = "----tg" + token_hex(8)
parts = [
f"--{boundary}\r\n".encode(),
b'Content-Disposition: form-data; name="chat_id"\r\n\r\n',
f"{chat_id}\r\n".encode(),
f"--{boundary}\r\n".encode(),
b'Content-Disposition: form-data; name="caption"\r\n\r\n',
f"{caption}\r\n".encode(),
f"--{boundary}\r\n".encode(),
b'Content-Disposition: form-data; name="photo"; '
b'filename="image.jpg"\r\nContent-Type: image/jpeg\r\n\r\n',
image_bytes, # ← binary stays untouched
b"\r\n",
f"--{boundary}--\r\n".encode(),
]
body = b"".join(parts) # concatenate raw byte chunks
The difference is `b"".join(parts)` instead of `b"\r\n".join(...)`: each part
is already a complete chunk with its own framing, and we concatenate them
verbatim rather than inserting a delimiter _between every element_.
There's a second, higher-level fix worth naming: **stop asking Telegram to
download the image for you.** My earlier code passed the image as a _URL_ and
let Telegram fetch it:
tg("sendPhoto", {"photo": image_url, "caption": caption})
That means Telegram's servers have to reach out to the source CDN — and if
that CDN is flaky or geo-blocks Telegram's crawlers, you get intermittent
failures that are impossible to reproduce locally. Downloading the image
yourself, then uploading the bytes, removes an entire class of flakiness and
gives you a chance to validate the bytes before you send them.
## What I'd do differently
1. **Read the raw response, not the assumption.** `nginx/1.30.1` sat in logs for hours while I chased phantom proxies. Look up who owns the header before inventing a man-in-the-middle.
2. **Verify a hard-coded IP against a live resolver.** Memory is not DNS. `dig +short api.telegram.org` takes three seconds and would have saved me a whole round.
3. **Treat "identical inputs, opposite outputs" as a lie.** The inputs were never identical — the two processes differed somewhere I wasn't looking. The honest move is to diff the bytes, not the code.
4. **Never hand-join binary data with a text delimiter.** `b"\r\n".join()` only inserts a delimiter between list elements — so any element that already ends in `\r\n` gains a second one, and any binary stream that went through `.splitlines()` is already rewritten. Frame each part yourself, then use `b"".join()`.
## The result
After the multipart fix, the monitor pushed a real notification through on
the next tick — image and all. I also folded in a retry queue so a failed
notification stays marked un-sent and is retried 30 seconds later instead of
silently vanishing. The whole episode cost the better part of a day, but it
turned into exactly the kind of bug I'll never trip over twice.
_This monitoring pipeline was originally built for high-frequency internal inspection, but the troubleshooting lessons apply to any production service that relies on outbound notifications. If you've got a monitor, a scraper, or
an alert pipeline that needs to reach people reliably, let's talk._
**I build reliable notification and monitoring pipelines — Telegram/WhatsApp
alerts, scrapers, and the infrastructure around them.** Tell me what you need
to watch and where you need it delivered:
* WhatsApp me →
* Email me →
* hoelee.com
_I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com._