This fixes cases where the shifted number is 64, but we shifted non-
zero numbers away in the shift. The change makes behaviour consistent
with libvpx.
Also disable the mmx/iwht optimization when the bitexact flag is set.
With synthetically coded coefficients (i.e. these that lead to a
residual well outside the [-255,255] range), our optimizations will
overflow. It doesn't make sense to fix the overflows, since they can
only occur on synthetic input, not on real fwht-generated input. Thus,
add a bitexact flag that disables this optimization.
the max value of the lookup in expanded form is:
(((1 << 7) - 1) << 1) - 65 + 1 + 64 = 254
add one entry of padding to inv_map_table[] to prevent out of bounds
access with non-conforming / fuzzed bitstreams
Signed-off-by: James Zern <jzern@google.com>
Reviewed-by: "Ronald S. Bultje" <rsbultje@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Otherwise the check 'tile_size < size' treats a negative size as
unsigned, causing the check to pass. This subsequently leads to
segmentation faults.
This was originally fixed as part of Libav commit 72ca83, so the
original author is one of the following developers:
Anton Khirnov <anton@khirnov.net>
Diego Biurrun <diego@biurrun.de>
Luca Barbato <lu_zero@gentoo.org>
Martin Storsjö <martin@martin.st>
Reviewed-by: Ronald S. Bultje <rsbultje@gmail.com>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
* rbultje/vp9-bugfixes:
vp9: match another find_ref_mvs() bug in libvpx.
vp9: fix scaled motion vector clipping for sub8x8 blocks.
vp9: improve signbias check.
vp9: don't allow compound references if error_resilience is enabled.
vp9: clamp segmented lflvl before applying ref/mode deltas.
vp9: reset loopfilter mode/ref deltas on keyframe.
vp9: fix crash when playing back 440/440 content with width%64<56.
vp9: extend loopfilter workaround for vp9 h/v mix-up to work for 422.
vp9: clip motion vectors in the same way as libvpx does.
vp9: set skip flag if the block had no coded coefficients.
vp9: apply mv scaling workaround only when subsampling is enabled.
vp9: read all 4x4 blocks in sub8x8 blocks individually with scalability.
vp9: fix segmentation map referencing upon framesize change.
vp9: disable more pmulhrsw optimizations in idct16/32.
vp9: disable all pmulhrsw in 8/16 iadst x86 optimizations.
Merged-by: Michael Niedermayer <michaelni@gmx.at>
If we find a second non-sub8x8 motion vector for a non-first sub8x8
block, and the clamped value is identical to the first non-sub8x8
motion vector, then the resulting nearmv motion vector is forced to
zero.
libvpx (probably accidentally) clears the bits if error_res is set,
along with keyframe/intraonly. This probably wasn't the intention
(since it's local data), but it's behaviour we have to copy...
Makes no real difference, but maybe scares coverity less (CID1297578)
Reviewed-by: "Ronald S. Bultje" <rsbultje@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
The practical effect of this is that the scaling will wrongly not be
applied to the interpolation edge (the 3/4 constants in this patch).
In other words, we clip to the pre-scaling interpolation, even though
these should be clipped post-scaling. The resulting out-of-frame MVs
are thus automatically clipped within the visible portion of the frame,
which is probably not the intention, but is unfortunately what libvpx
does, so we need to copy that behaviour.
This reproduces libvpx behaviour. It seems like it originally only
targeted loopfilter behaviour, but this unfortunately effects following
block contexting and thus directs bitstream sync.
Ronald S. Bultje