PixelSmash: FFmpeg flaw turns a video file into Jellyfin RCE

A new flaw in FFmpeg, the video decoding library quietly doing the heavy lifting behind a huge chunk of internet media tooling, can be triggered without anyone actually watching a video. Scanning a folder for thumbnails is enough. Researchers at JFrog have named it PixelSmash. It carries CVE-2026-8461 and a severity score of 8.8.

What the bug actually does

The fault sits in FFmpeg's MagicYUV decoder. The frame allocator and the decoder disagree about how tall the chroma planes should be, and that mismatch produces a one-row heap buffer overflow when a crafted AVI, MKV or MOV file is processed. Processed, not played. Opening a directory in a file manager, running an automated ingest job, or letting a file system watcher notice a new download is all it takes to set things off.

The Jellyfin demo

JFrog's Yuval Moravchick walked the bug all the way to full remote code execution against Jellyfin 10.11.9, the second most popular self-hosted media server after Plex. The choreography is unnervingly clean. A booby-trapped MagicYUV file appears in the library. Jellyfin runs ffprobe on it automatically to pull metadata. The overflow hijacks AVBuffer.free, points it at system(), and arbitrary commands run as the jellyfin service user.

The torrent path is tidier still. Seed the file, wait for someone to aim a download client at their media folder, and the real-time watcher delivers the payload without anyone touching anything.

The catch, and why it still matters

Full RCE needs ASLR disabled, or a separate info-disclosure bug chained in to defeat it. That's a meaningful guardrail on a default Linux box. The unwelcome news is that even without RCE, you have reliable denial of service against a long list of software that uses FFmpeg under the hood:

Kodi
OBS Studio
PhotoPrism
Nextcloud
The thumbnail generators in GNOME, KDE and XFCE

Slack, Discord, Telegram and WhatsApp all lean on FFmpeg for server-side previews and may also be exposed, though JFrog did not test them. Plex sidesteps the whole thing because it ships a stripped-down FFmpeg with most decoders, including MagicYUV, disabled.

Patches and posture

FFmpeg fixed the issue in version 8.1.2, released on 17 June. Jellyfin has updated its bundled build. PhotoPrism is adding a format blocklist. Nextcloud, somewhat awkwardly, told HackerOne the bug was not theirs to fix and bounced it back upstream.

If you run any of the affected software, particularly a self-hosted media server pointed at a directory that anyone else can write to, this is a good week to check what version of FFmpeg is sitting underneath it. The interesting part of PixelSmash is not the overflow itself. It is how little user action stands between a file existing and code running. Every modern desktop and media stack is wired to look at files the moment they appear, and that helpful behaviour is exactly what the attacker is counting on.