avformat/rtpdec_latm: avoid integer overflow in LATM length parsing

latm_parse_packet() accumulated attacker-controlled AU length bytes in
a signed int and later checked data->pos + cur_len against data->len.
That addition could overflow, allowing malformed packets to bypass the
bounds check and drive memcpy() far past the end of the LATM buffer.

Reject length-byte accumulation that would exceed the remaining packet
size, and compare cur_len against the remaining buffer space using
subtraction so the bounds check cannot overflow.

Fixes: DFVULN-610

*Vulnerability reported by Zhenpeng (Leo) Lin at depthfirst*
*Patch validated by Zheng Yu at depthfirst*

(cherry picked from commit 664d44a825)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

libavformat/rtpdec_latm.c

@@ -72,11 +72,15 @@ static int latm_parse_packet(AVFormatContext *ctx, PayloadContext *data,
     cur_len = 0;
     while (data->pos < data->len) {
         uint8_t val = data->buf[data->pos++];
+        if (val > data->len - cur_len) {
+            av_log(ctx, AV_LOG_ERROR, "Malformed LATM packet\n");
+            return AVERROR_INVALIDDATA;
+        }
         cur_len += val;
         if (val != 0xff)
             break;
     }
-    if (data->pos + cur_len > data->len) {
+    if (cur_len > data->len - data->pos) {
         av_log(ctx, AV_LOG_ERROR, "Malformed LATM packet\n");
         return AVERROR(EIO);
     }