com.xaymar.www/_guides/lost-knowledge-and-secrets.html

---
layout: default
title: "Lost Knowledge & Secrets"
comments: true
date: "2022-01-13"
---

{% capture content %}When knowledge becomes common knowledge, the curse of time starts affecting it and eventually it may be lost. This is a page dedicated to such knowledge I possess or rediscovered through encountering the same problems. I can't guarantee that someone may hold a patent on the knowledge recorded here.{% endcapture %}{% include blocks/paragraph.liquid content=content %}



{% capture terminology_block %}
{% capture list %}
<li>
	<strong>Bits per Pixel (bpp, BpP)</strong><br>
	The number of bits that are used for a single pixel in total.
</li>
<li>
	<strong>Bits per Channel (bpc, BpC)</strong><br>
	The number of bits that represent a single channel, such as Red, Green, Blue or Alpha. Usually describes a buffer that will be aligned with 2^n sizes.
</li>
<li>
	<strong>Bits per Pixel (bpp, BpP)</strong><br>
	The number of bits that are used for a single pixel in total.
</li>
<li>
	<strong>Bits per Channel (bpc, BpC)</strong><br>
	The number of bits that represent a single channel, such as Red, Green, Blue or Alpha. Usually describes a buffer that will be aligned with 2^n sizes.
</li>
<li>
	<strong>Dithering</strong><br>
	A technique used to create a percieved or actual additional color by taking advantage of human or physical faults. Very common in Games and Software written during the CRT days, where color reproduction was based on physical properties.
	</li>
<li>
	<strong>Frames per Second (fps, FpS)</strong><br>
	The number of frames displayed in a single second at a constant pace.
</li>
<li>
	<strong>SDR</strong><br>
	Often used interchangeably with a precision of 8 bits per channel or less. Usually paired with a color space unable to provide most of the human vision color spectrum.
</li>
<li>
	<strong>HDR</strong><br>
	Often used interchangeably with a precision of 10 bits per channel or more. Usually paired with a color space able to cover 95% or more of the human vision color spectrum.
</li>
<li>
	<strong>FakeHDR</strong><br>
	An ancient technique that abused the last channel in a 8/8/8/8 texture as a multiplier.
</li>
<li>
	<strong>A-B-C-D</strong>, <strong>A/B/C/D</strong><br>
	A combined texture format where each of the A, B, C, and D elements specify the bits for a specific channel, usually in the order RGBA or BGRA.
</li>
{% endcapture %}{% include blocks/list.liquid content=list %}
{% endcapture %}{% include blocks/details.liquid level=3 title="Terminology" content=terminology_block %}

{% capture color_space_block %}

{% include blocks/heading.liquid level=2 content="Is lossless conversion between YUV and RGB possible?" %}
{% capture content %}It depends on the matrix used to transfer to and from YUV, as well as the number of bits per channel. To be on the safe side, it should be assumed that converting to and from YUV will result in slight quality loss, which may be acceptable to you, but will result in accumulating error.{% endcapture %}{% include blocks/paragraph.liquid content=content %}

{% endcapture %}{% include blocks/details.liquid level=1 title="Color Space Handling" content=color_space_block %}

{% capture block_hosting %}

{% capture block_hosting_trunas %}

{% capture block_hosting_trunas_recordsize %}{% capture content %}The Record Size of a storage dataset in TrueNAS and FreeNAS matters a lot. It can be the difference between being able to transfer 2 GiB in a second, and taking multiple minutes to do so. While documentation is a bit sparse and fractured, through testing I was able to find out that the best record size is the default it has already selected. It offers the best mixture of performance for large and small writes, especially on faster storage.{% endcapture %}{% include blocks/paragraph.liquid content=content %}

{% capture content %}My tests went through many record sizes, halving and doubling from 128K outwards. Every halving resulted in large read/writes getting significantly slower, but small read/writes getting slightly faster. Every doubling resulting in large read/write getting slightly faster, but small read/writes getting significantly slower. The best value ended up being 128K, which managed to keep most of the NVMe performance as is (~80MiB/s in RND4K, ~2.5GiB/s in RND1M).{% endcapture %}{% include blocks/paragraph.liquid content=content %}{% endcapture %}{% include blocks/details.liquid level=3 title="Ideal Record Size" content=block_hosting_trunas_recordsize %}

{% endcapture %}{% include blocks/details.liquid level=2 title="TrueNAS & FreeNAS" content=block_hosting_trunas %}

{% endcapture %}{% include blocks/details.liquid level=1 title="Hosting" content=block_hosting %}


{% capture particle_block %}

{% include blocks/heading.liquid level=2 content="Calculate maximum Particle count ahead of time" %}
{% capture content %}Many modern Particle simulation engines rely on user input for their maximum particle count, despite the maximum particle count being something that can be calculated ahead of time. All you need is the highest possible spawn rate (as particles per second), the minimum decay rate (as decay value per second), and the maximum life time (in seconds). Once you have these variables the following formulae will result in the maximum particle count:{% endcapture %}{% include blocks/paragraph.liquid content=content %}

{% capture particle_max_count_code %}particles = min(maxSpawnRate / minDecayRate, maxLifeTime * maxSpawnRate){% endcapture %}{% include blocks/code.liquid content=particle_max_count_code %}

{% endcapture %}{% include blocks/details.liquid level=1 title="Particle Simulation" content=particle_block %}

{% capture rendering_block %}

{% include blocks/heading.liquid level=2 content="Is it possible to avoid Color Banding?" %}
{% capture content %}Not while we work with finite space, but we can make it nearly invisible to a Human-like observer through Dithering. Dithering requires a higher precision input, so a 10/10/10/X texture for intermediate and 8/8/8/X texture for output would already improve quality. Most game engines opt for 16/16/16/X or 10/10/10/X for their internal render buffer, but many do not opt to dither to the target buffer.{% endcapture %}{% include blocks/paragraph.liquid content=content %}

{% include blocks/heading.liquid level=2 content="Could Dithering provide higher quality?" %}
{% capture content %}Yes with a Human-like observer (perceptual Quality), otherwise no. We can abuse the poor quality of Human vision through spatial and temporal Dithering, thus pretending to have extra precision that doesn't exist. Many older games and consoles abused this to draw more colors than they actually could render on screen. Often it is comined with sub-pixel rendering to provide further quality improvements.{% endcapture %}{% include blocks/paragraph.liquid content=content %}

{% capture content %}With modern technology, this trick became much easier than before, to the point that Displays have this built-in now: FRC. After you hit a refresh rate of roughly 40 Hz, you can fake an extra bit of precision every additional 40 Hz. So a 120 Hz monitor with only 8 bits per channel could fake 10 bits per channel to a human-like observer, and they would very likely not notice it.{% endcapture %}{% include blocks/paragraph.liquid content=content %}

{% include blocks/heading.liquid level=2 content="Can you store HDR in 8/8/8/8-bit?" %}
{% capture content %}No, but some game engines did abuse this format for faking HDR. The 4th channel would contain a multiplier for the first three channels, however this came at a cost of precision and often reduced overall quality. As GPUs got more advanced, we now have a better HDR format when space is limited without any of the downsides: 10/10/10/2.{% endcapture %}{% include blocks/paragraph.liquid content=content %}

{% endcapture %}{% include blocks/details.liquid level=1 title="Rendering" content=rendering_block %}