Long-term Memoryhttps://blog.dest-unreach.be/2024-01-03T00:00:00+01:00A collection of note-to-self'sPodman on ZFS2024-01-03T00:00:00+01:002024-01-03T00:00:00+01:00Niobostag:blog.dest-unreach.be,2024-01-03:/2024/01/03/podman-on-zfs/<p>When running rootless <a href="https://podman.io/">podman</a> on a ZFS mount, it defaults to the <code>vfs</code> storage driver.
And while this works, it causes a huge waste of disk space by duplicating files for every layer.
Using <code>overlay</code> solves this when running rootfull, but this is <a href="https://github.com/openzfs/zfs/issues/8648">currently not supported</a> on ZFS.
It should …</p><p>When running rootless <a href="https://podman.io/">podman</a> on a ZFS mount, it defaults to the <code>vfs</code> storage driver.
And while this works, it causes a huge waste of disk space by duplicating files for every layer.
Using <code>overlay</code> solves this when running rootfull, but this is <a href="https://github.com/openzfs/zfs/issues/8648">currently not supported</a> on ZFS.
It should arrive with ZFS 2.2, but that will probably only be available in <a href="https://www.debian.org/releases/">Debian</a> 13 Trixie somewhere in 2025.
However, we should have the same functionality with the FUSE-variant <code>fuse-overlay</code>, with an additional performance penalty for the userspace roundtrip.</p>
<p>For example, the current <a href="https://www.home-assistant.io/">Home Assistant</a> image in <code>vfs</code> mode takes up:</p>
<div class="highlight"><pre><span></span><code>$ podman unshare rm -rf .local/share/containers
$ podman system reset
<span class="o">[</span>...<span class="o">]</span>
$ <span class="nb">time</span> podman pull ghcr.io/home-assistant/home-assistant:2023.12.4
Trying to pull ghcr.io/home-assistant/home-assistant:2023.12.4...
Getting image <span class="nb">source</span> signatures
Copying blob 2651e446c136 <span class="k">done</span>
<span class="o">[</span>...<span class="o">]</span>
Copying config 003e99ae19 <span class="k">done</span>
Writing manifest to image destination
Storing signatures
003e99ae19b0c68a9cbb0c28e68cf843e12513287d1945c1b0b3d1e0eb246973
real 16m1.174s
user 0m58.612s
sys 1m2.829s
$ du -chs .local/share/containers
19G .local/share/containers
</code></pre></div>
<p>Switching to the <code>overlay</code> storage driver is a huge win, both in time and in storage space:</p>
<div class="highlight"><pre><span></span><code>$ cat .config/containers/storage.conf
<span class="o">[</span>storage<span class="o">]</span>
<span class="nv">driver</span><span class="o">=</span><span class="s2">"overlay"</span>
<span class="o">[</span>storage.options.overlay<span class="o">]</span>
<span class="nv">mount_program</span> <span class="o">=</span> <span class="s2">"/usr/bin/fuse-overlayfs"</span>
<span class="c1">#mountopt = "noacl" # see below</span>
$ podman unshare rm -rf .local/share/containers
$ podman system reset
<span class="o">[</span>...<span class="o">]</span>
$ <span class="nb">time</span> podman pull ghcr.io/home-assistant/home-assistant:2023.12.4
Trying to pull ghcr.io/home-assistant/home-assistant:2023.12.4...
Getting image <span class="nb">source</span> signatures
Copying blob 2651e446c136 <span class="k">done</span>
<span class="o">[</span>...<span class="o">]</span>
Copying config 003e99ae19 <span class="k">done</span>
Writing manifest to image destination
Storing signatures
003e99ae19b0c68a9cbb0c28e68cf843e12513287d1945c1b0b3d1e0eb246973
real 1m41.778s
user 0m33.646s
sys 0m9.357s
$ du -chs .local/share/containers
<span class="m">2</span>.4G .local/share/containers
</code></pre></div>
<h2 id="acl-hickup"><a class="toclink" href="#acl-hickup">ACL hickup</a></h2>
<p>But this doesn't seem to work for all images.
The <a href="https://grafana.com/grafana/">Grafana</a> image fails with a rather unspecific <code>Operation not supported</code>:</p>
<div class="highlight"><pre><span></span><code>$ podman run -it --rm docker.io/grafana/grafana-oss:10.0.10
Trying to pull docker.io/grafana/grafana-oss:10.0.10...
Getting image <span class="nb">source</span> signatures
Copying blob bf463c6d6fd9 <span class="k">done</span>
<span class="o">[</span>...<span class="o">]</span>
Copying config e5b83aa02b <span class="k">done</span>
Writing manifest to image destination
Storing signatures
<span class="o">{</span><span class="s2">"msg"</span>:<span class="s2">"exec container process `/run.sh`: Operation not supported"</span>,<span class="s2">"level"</span>:<span class="s2">"error"</span>,<span class="s2">"time"</span>:<span class="s2">"2024-01-03T08:12:03.063449Z"</span><span class="o">}</span>
</code></pre></div>
<p>After some looking around, I found <a href="https://github.com/containers/podman/issues/11213">these</a> <a href="https://github.com/containers/fuse-overlayfs/issues/367">two</a> GitHub issue that seems very similar.
The problem described there was that the underlying ZFS file system had ACLs disabled, and the container tried to use them.
And indeed, I have:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># zfs get aclmode /tank/podman</span>
NAME PROPERTY VALUE SOURCE
tank/podman aclmode discard default
</code></pre></div>
<p>Adding the <code>noacl</code> mount option to <code>storage.conf</code> seems to solve that:</p>
<div class="highlight"><pre><span></span><code>$ cat .config/containers/storage.conf
<span class="o">[</span>storage<span class="o">]</span>
<span class="nv">driver</span><span class="o">=</span><span class="s2">"overlay"</span>
<span class="o">[</span>storage.options.overlay<span class="o">]</span>
<span class="nv">mount_program</span> <span class="o">=</span> <span class="s2">"/usr/bin/fuse-overlayfs"</span>
<span class="nv">mountopt</span> <span class="o">=</span> <span class="s2">"noacl"</span>
</code></pre></div>
<h2 id="system-wide"><a class="toclink" href="#system-wide">System-wide</a></h2>
<p>Next, I wanted to configure this system-wide so all users would benefit from this.
I thought it would be as simple as moving the config file from <code>~/.config/containers/storage.conf</code> to <code>/etc/containers/storage.conf</code>, but that does not seem the case:</p>
<div class="highlight"><pre><span></span><code>$ podman run -it --rm docker.io/grafana/grafana-oss:10.0.10
Error: <span class="s1">'overlay'</span> is not supported over zfs, a mount_program is required: backing file system is unsupported <span class="k">for</span> this graph driver
</code></pre></div>
<p>Podman complains about the missing <code>mount_program</code> option, even though it is configured.
I haven't figured out why yet... To be continued</p>
<h2 id="building-images"><a class="toclink" href="#building-images">Building images</a></h2>
<p>I also had some issues when building container images using <code>podman build</code>.
They resulted in an "Operation Not Supported" error when creating a layer.</p>
<p>I haven't figured out what exactly went wrong, but using <code>buildah</code> directly does work...</p>Configuring WLED to wipe-on the LEDs on startup2023-11-22T00:00:00+01:002023-11-22T00:00:00+01:00Niobostag:blog.dest-unreach.be,2023-11-22:/2023/11/22/WLED_wipe_on/<p><a href="https://kno.wled.ge/">WLED</a> is an open source project for controlling LED-strips.
I wanted my LED strip to "wipe on" when powering on, similar to the first half of the "Wipe" effect, but staying on after that.
These are my notes.</p>
<p>Searching around for similar idea's, I found a <a href="https://kno.wled.ge/advanced/custom-features/">usermod</a> for a <a href="https://github.com/Aircoookie/WLED/tree/main/usermods/Animated_Staircase">stairway …</a></p><p><a href="https://kno.wled.ge/">WLED</a> is an open source project for controlling LED-strips.
I wanted my LED strip to "wipe on" when powering on, similar to the first half of the "Wipe" effect, but staying on after that.
These are my notes.</p>
<p>Searching around for similar idea's, I found a <a href="https://kno.wled.ge/advanced/custom-features/">usermod</a> for a <a href="https://github.com/Aircoookie/WLED/tree/main/usermods/Animated_Staircase">stairway</a>.
That module did much more than I needed, but it got me started.
In the end, it looked like my desired effect was possible using <a href="https://kno.wled.ge/features/presets/">Presets and Playlist</a>.</p>
<p>I created two presets, a "wipe" and a "solid", and chained these together with a playlist.
The playlist has a single entry, the "wipe" preset, and is configured to <em>not</em> repeat, and end on the "solid" preset.
Getting the timing right was a bit of a journey: I needed to convert "effect speed" into "seconds to halfway".
Luckily, open source code makes things easy.
<a href="https://github.com/Aircoookie/WLED/blob/main/wled00/FX.cpp#L163">The source code</a> shows the used formula:
<code>cycleTime = 750 + (255 - SEGMENT.speed)*150</code>.
This means that a speed of 253 gives a cycleTime of 1050ms.
So by setting the playlist duration of the wipe preset to 0.5 seconds, I got the desired result.</p>
<p>Another thing to fix is that I wanted the wipe animation to always start at the beginning.
This can be fixed by including <code>"tb": 0</code> in the <a href="https://kno.wled.ge/interfaces/json-api/">preset JSON</a>.</p>
<p>Note that you want do <em>disable</em> the <code>Turn LEDs on after power up/reset</code>.
When you leave this on, the LEDs first go to on, and only then start the animation.</p>
<p>For reference, here is the relevant part of my <code>preset.json</code>:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"2"</span><span class="p">:{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"on"</span><span class="p">:</span><span class="kc">true</span><span class="p">,</span><span class="nt">"bri"</span><span class="p">:</span><span class="mi">255</span><span class="p">,</span><span class="nt">"transition"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"tb"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"mainseg"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"seg"</span><span class="p">:[</span><span class="w"></span>
<span class="w"> </span><span class="p">{</span><span class="nt">"id"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"start"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">271</span><span class="p">,</span><span class="nt">"grp"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"spc"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"of"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"on"</span><span class="p">:</span><span class="kc">true</span><span class="p">,</span><span class="nt">"frz"</span><span class="p">:</span><span class="kc">false</span><span class="p">,</span><span class="nt">"bri"</span><span class="p">:</span><span class="mi">255</span><span class="p">,</span><span class="nt">"cct"</span><span class="p">:</span><span class="mi">127</span><span class="p">,</span><span class="nt">"set"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"col"</span><span class="p">:[[</span><span class="mi">100</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">152</span><span class="p">,</span><span class="mi">255</span><span class="p">],[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">],[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">]],</span><span class="nt">"fx"</span><span class="p">:</span><span class="mi">3</span><span class="p">,</span><span class="nt">"sx"</span><span class="p">:</span><span class="mi">253</span><span class="p">,</span><span class="nt">"ix"</span><span class="p">:</span><span class="mi">128</span><span class="p">,</span><span class="nt">"pal"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"c1"</span><span class="p">:</span><span class="mi">128</span><span class="p">,</span><span class="nt">"c2"</span><span class="p">:</span><span class="mi">128</span><span class="p">,</span><span class="nt">"c3"</span><span class="p">:</span><span class="mi">16</span><span class="p">,</span><span class="nt">"sel"</span><span class="p">:</span><span class="kc">true</span><span class="p">,</span><span class="nt">"rev"</span><span class="p">:</span><span class="kc">false</span><span class="p">,</span><span class="nt">"mi"</span><span class="p">:</span><span class="kc">false</span><span class="p">,</span><span class="nt">"o1"</span><span class="p">:</span><span class="kc">false</span><span class="p">,</span><span class="nt">"o2"</span><span class="p">:</span><span class="kc">false</span><span class="p">,</span><span class="nt">"o3"</span><span class="p">:</span><span class="kc">false</span><span class="p">,</span><span class="nt">"si"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"m12"</span><span class="p">:</span><span class="mi">0</span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">],</span><span class="w"></span>
<span class="w"> </span><span class="nt">"n"</span><span class="p">:</span><span class="s2">"wipe"</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="nt">"3"</span><span class="p">:{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"on"</span><span class="p">:</span><span class="kc">true</span><span class="p">,</span><span class="nt">"bri"</span><span class="p">:</span><span class="mi">255</span><span class="p">,</span><span class="nt">"transition"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"mainseg"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"seg"</span><span class="p">:[</span><span class="w"></span>
<span class="w"> </span><span class="p">{</span><span class="nt">"id"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"start"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">271</span><span class="p">,</span><span class="nt">"grp"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"spc"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"of"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"on"</span><span class="p">:</span><span class="kc">true</span><span class="p">,</span><span class="nt">"frz"</span><span class="p">:</span><span class="kc">false</span><span class="p">,</span><span class="nt">"bri"</span><span class="p">:</span><span class="mi">255</span><span class="p">,</span><span class="nt">"cct"</span><span class="p">:</span><span class="mi">127</span><span class="p">,</span><span class="nt">"set"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"col"</span><span class="p">:[[</span><span class="mi">100</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">152</span><span class="p">,</span><span class="mi">255</span><span class="p">],[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">],[</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span><span class="mi">0</span><span class="p">]],</span><span class="nt">"fx"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"sx"</span><span class="p">:</span><span class="mi">16</span><span class="p">,</span><span class="nt">"ix"</span><span class="p">:</span><span class="mi">108</span><span class="p">,</span><span class="nt">"pal"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"c1"</span><span class="p">:</span><span class="mi">128</span><span class="p">,</span><span class="nt">"c2"</span><span class="p">:</span><span class="mi">128</span><span class="p">,</span><span class="nt">"c3"</span><span class="p">:</span><span class="mi">16</span><span class="p">,</span><span class="nt">"sel"</span><span class="p">:</span><span class="kc">true</span><span class="p">,</span><span class="nt">"rev"</span><span class="p">:</span><span class="kc">false</span><span class="p">,</span><span class="nt">"mi"</span><span class="p">:</span><span class="kc">false</span><span class="p">,</span><span class="nt">"o1"</span><span class="p">:</span><span class="kc">false</span><span class="p">,</span><span class="nt">"o2"</span><span class="p">:</span><span class="kc">false</span><span class="p">,</span><span class="nt">"o3"</span><span class="p">:</span><span class="kc">false</span><span class="p">,</span><span class="nt">"si"</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="nt">"m12"</span><span class="p">:</span><span class="mi">0</span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">},{</span><span class="nt">"stop"</span><span class="p">:</span><span class="mi">0</span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">],</span><span class="w"></span>
<span class="w"> </span><span class="nt">"n"</span><span class="p">:</span><span class="s2">"default on"</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="nt">"1"</span><span class="p">:{</span><span class="w"></span>
<span class="w"> </span><span class="nt">"playlist"</span><span class="p">:{</span><span class="nt">"ps"</span><span class="p">:[</span><span class="mi">2</span><span class="p">],</span><span class="nt">"dur"</span><span class="p">:[</span><span class="mi">5</span><span class="p">],</span><span class="nt">"transition"</span><span class="p">:[</span><span class="mi">0</span><span class="p">],</span><span class="nt">"repeat"</span><span class="p">:</span><span class="mi">1</span><span class="p">,</span><span class="nt">"end"</span><span class="p">:</span><span class="mi">3</span><span class="p">,</span><span class="nt">"r"</span><span class="p">:</span><span class="mi">0</span><span class="p">},</span><span class="nt">"on"</span><span class="p">:</span><span class="kc">true</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="nt">"n"</span><span class="p">:</span><span class="s2">"power on animation"</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>Specs of the SK6812 LED strip2023-08-31T00:00:00+02:002023-08-31T00:00:00+02:00Niobostag:blog.dest-unreach.be,2023-08-31:/2023/08/31/sk6812/<p>I was considering a SK6812 strip for a project of mine, but I found it very hard to find information online about the power consumption.
So I ordered a test strip to do the measurements myself.
The SK6812 is an individually addressable LED strip.
I got the RGBW variant, where …</p><p>I was considering a SK6812 strip for a project of mine, but I found it very hard to find information online about the power consumption.
So I ordered a test strip to do the measurements myself.
The SK6812 is an individually addressable LED strip.
I got the RGBW variant, where each dot has a Red, Green, Blue and White LED that can be controlled.
The protocol used is similar to the WS2811/WS2812/WS2813 series, but reading 32 bits per LED instead of 24 bits.
This is what I found.</p>
<ul>
<li>The supply voltage is 5V</li>
<li>A 5m, 300 LED strip consumes about 70mA in idle condition, or 0.23mA/LED</li>
<li>My ESP32-based controller requires another 150mA, or a bit more when it's actively refreshing the LEDs</li>
<li>For the LED current, I measured the current for 1, 2, 5, 10, 20 and 40 LEDs, and determined the slope.
The Red LED averages at about 7.98mA/LED at full brightness; Green is at 8.11mA/LED; Blue is 7.98mA/LED; White is 16.11mA/LED</li>
<li>100% RGB gives 23.06mA/LED, 100% RGBW gives 38.9mA/LED</li>
</ul>
<p>With these currents and only 5V to start from, voltage drop is a significant problem.
Based on some measurements, I estimate the resistance between 2 LEDs to be around 6.2mΩ.
The voltage drop increases approximately with the square of the number of LEDs, so this problem grows quickly!</p>
<p>I measured what minimum voltage is needed to avoid fading.
To do that, I folded the strip to put the first & last LED next to each other and turned them on full brightness.
Next, I pulsed the middle LEDs and watched for a brightness difference between the first LED (getting the full supply voltage)
and last LED (getting the full voltage drop because of the middle LEDs).
I increased the number of pulsing LEDs until I could see the last LED pulse as well.
I was slightly surprised to find that I could drop down to 3.4V without noticing any dimming; I was expecting a higher threshold.
When the voltage dropped to 3.37V (1 more pulsing LED in the middle), the fading was just noticeable.</p>
<p>So for my 5m 60LED/m strip, this means I need power injected for every 1.9m.
Since you can power from either end, I can stretch this up to 3.8m before I need to inject power in the middle of a run.
Note that these numbers assume you can inject 5V at that point.
If the 5V power supply is not located right next to the strip, you should account for the voltage drop in the supply cables as well.
To sidestep that problem, I'm thinking of using a 12V (or 24V) power supply, and put a 12V to 5V buck converter right next to the strip.</p>
<h2 id="color-output"><a class="toclink" href="#color-output">Color output</a></h2>
<p>I measured the LEDs color with the Opple Light Master 3:</p>
<ul>
<li>Red (x, y)=(0.6532, 0.2984)</li>
<li>Green (x, y)=(0.1637, 0.5288)</li>
<li>Blue (x, y)=(0.1609, 0.0709)</li>
<li>White (x, y)=(0.4181, 0.4259), CCT=3500K, ∆uv=0.019, so slightly greenish</li>
<li>100% RGBW (x, y)=(0.3407, 0.3217), CCT=5100K, ∆uv=-0.014, so slightly purplish</li>
<li>100%W + 55%R + 57%B corrects for the greenish tint and yields (x, y)=(0.4072, 0.3951), CCT=3500K, ∆uv=0.001</li>
</ul>Kerbal Space Program Grand Tour2023-07-05T00:00:00+02:002023-07-05T00:00:00+02:00Niobostag:blog.dest-unreach.be,2023-07-05:/2023/07/05/kerbal_grand_tour/<p>I wanted a challenge, so I decided to try a "Grand Tour" of the Kerbolar system in <a href="https://www.kerbalspaceprogram.com/">Kerbal Space Program</a>.
A "Grand Tour" is a single mission where you visit all bodies in the Kerbolar system.
The definition of "visit" varies from fly-by to manned landing, depending on how challenging …</p><p>I wanted a challenge, so I decided to try a "Grand Tour" of the Kerbolar system in <a href="https://www.kerbalspaceprogram.com/">Kerbal Space Program</a>.
A "Grand Tour" is a single mission where you visit all bodies in the Kerbolar system.
The definition of "visit" varies from fly-by to manned landing, depending on how challenging you want things to be.</p>
<p>I decided to try the "manned landing"-variant in a fully reusable craft,
except for Eve (and Jool and Kerbol, since you can't land there).</p>
<p>Going around the entire Kerbol system requires about 20km/s of ∆v.
That's a lot.
So I decided to equip my ship with an <a href="https://wiki.kerbalspaceprogram.com/wiki/Convert-O-Tron_250">ISRU</a> and mine extra fuel along the way.</p>
<h2 id="eve-moho"><a class="toclink" href="#eve-moho">Eve – Moho</a></h2>
<p>The Eve–Moho hop is the most challenging hop ∆v-wise.
A fairly good transfer window still <a href="https://alexmoon.github.io/ksp/#/Eve/100/Moho/100/false/optimal/false/3/200">requires</a>
3480m/s from Low Eve Orbit to Low Moho Orbit, plus an additional ~1000m/s for landing.
By departing from Gilly and diving into Eve's gravity well to maximize the Oberth effect,
I was able to reduce this to <a href="/static/ksp-tools/moon-departure.html#t=Moho&t0l=34644673.71225&t0h=35292885.412250005&dtl=1081884.375&dth=1837884.375&f=Eve&o=%7B%22sma%22%3A31499999.99999989%2C%22e%22%3A0.55%2C%22argp%22%3A0.17453292519943295%2C%22inc%22%3A0.20943951023931953%2C%22lon_an%22%3A1.3962634015954636%2C%22ma0%22%3A0.9%7D&apo=266817">around 2600m/s</a>,
depending on the chosen transfer window.</p>
<p>This particular transfer requires a low-TWR burn from Gilly to dive into Eve of about 530m/s.
At Eve periapsis, we need to do an escape burn of 930m/s, at reasonably high TWR,
since we lose Oberth effect fast when this burn takes too long.
The capture burn is 870m/s, also at high TWR.
Finally we need to circularize the orbits around Moho with an additional 320m/s, but that can be low-TWR and split between orbits.
Landing can start out at low TWR, but should end up well above 2.7m/s² (Moho's surface gravity).</p>
<h2 id="tylo"><a class="toclink" href="#tylo">Tylo</a></h2>
<p>Landing on Tylo is hard, with a beefy 2300m/s of speed to bleed off and no atmosphere to help with the breaking.
Tylo also has a surface gravity of 7.85m/s², so to do some actual breaking,
we're looking at at least 10m/s² (TWR=1.3) of thrust, but preferably 16m/s² (TWR=2) or more.</p>
<p>Getting to Tylo is easiest from Pol, where it's easy to refuel.
The trip <a href="https://alexmoon.github.io/ksp/#/Pol/20/Tylo/20/false/optimal/false/1/1">needs</a> approximately 1000m/s of ∆v:
The Pol escape burn of 215m/s can be done at low TWR, since the low gravity of Pol doesn't yield much Oberth-effect gains.
Arriving at Tylo, we only need about 42m/s at a periapsis of 20kmAGL to remain in orbits around Tylo.
Circularizing needs another 800m/s, but we can split that out over multiple orbits.</p>
<h2 id="the-ship"><a class="toclink" href="#the-ship">The ship</a></h2>
<p>With the above constraints in mind, I set out to design a spaceship.
I started with a spaceplane design, but couldn't get enough ∆v packed into it.
I guess the "dead mass" of the gears, wings and control surfaces adds up.
In addition, I needed an extra set of downward pointing engines for landing on bodies without atmosphere,
adding to the dead mass.
So I ended up with this Single Stage To Orbit (SSTO) rocket:</p>
<p><img alt="My Single Stage To Orbit Rocket" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/sstor.jpg"></p>
<p>The rocket consists of a <a href="https://wiki.kerbalspaceprogram.com/wiki/Mk1-3_Command_Pod">Mk1-3 Command Pod</a>, topped with a <a href="https://wiki.kerbalspaceprogram.com/wiki/Clamp-O-Tron_Shielded_Docking_Port">Shielded Docking Port</a>.
Underneath is the <a href="https://wiki.kerbalspaceprogram.com/wiki/Convert-O-Tron_250">Convert-O-Tron</a> and a <a href="https://wiki.kerbalspaceprogram.com/wiki/Mk3_to_2.5m_Adapter">2.5m to Mk3 Adapter</a>.
On the adapter are 4 <a href="https://wiki.kerbalspaceprogram.com/wiki/A.I.R.B.R.A.K.E.S">Airbrakes</a> to assist in orienting the rocket engines-first during atmospheric re-entry,
as well as 4 <a href="https://wiki.kerbalspaceprogram.com/wiki/Vernor_Engine">Vernor RCS engines</a> to keep the rocket upright at touchdown.
The 8 <a href="https://wiki.kerbalspaceprogram.com/wiki/Mk2-R_Radial-Mount_Parachute">parachutes</a> are attached here as well.
Next up is the <a href="https://wiki.kerbalspaceprogram.com/wiki/Mk3_Cargo_Bay_CRG-25">short Mk3 cargo bay</a> with an <a href="https://wiki.kerbalspaceprogram.com/wiki/Radial_Holding_Tank">ore tank</a>,
the <a href="https://wiki.kerbalspaceprogram.com/wiki/M4435_Narrow-Band_Scanner">Narrow-band Scanner</a> to pinpoint landing locations,
as well as a <a href="https://wiki.kerbalspaceprogram.com/wiki/Communotron_88-88">comm dish</a> to be able to access KerbNet.
I also packed a bit of <a href="https://wiki.kerbalspaceprogram.com/wiki/Advanced_Reaction_Wheel_Module,_Large">extra torque</a> to manoeuvre this thing a bit better.
Next are fuel tanks: A <a href="https://wiki.kerbalspaceprogram.com/wiki/Mk3_Rocket_Fuel_Fuselage">Mk3 Rocket Fuel</a>, a <a href="https://wiki.kerbalspaceprogram.com/wiki/Mk3_Liquid_Fuel_Fuselage">Mk3 Liquid Fuel</a> and 4 <a href="https://wiki.kerbalspaceprogram.com/wiki/Rockomax_Jumbo-64_Fuel_Tank">Jumbo-64</a> tanks.
The central column has the <a href="https://wiki.kerbalspaceprogram.com/wiki/LV-N_%22Nerv%22_Atomic_Rocket_Motor">NERV</a> engine, the 4 side tanks are fitted with a <a href="https://wiki.kerbalspaceprogram.com/wiki/S3_KS-25_%22Vector%22_Liquid_Fuel_Engine">Vector</a> engine.
At the bottom I added a <a href="https://wiki.kerbalspaceprogram.com/wiki/%27Drill-O-Matic_Junior%27_Mining_Excavator">Drill-O-Matic Junior</a> on a <a href="https://wiki.kerbalspaceprogram.com/wiki/1P2_Hydraulic_Cylinder">piston</a> to be able to reach the ground.
Power is generated by a <a href="https://wiki.kerbalspaceprogram.com/wiki/PB-NUK_Radioisotope_Thermoelectric_Generator">PB-NUK</a> for normal operations,
4 <a href="https://wiki.kerbalspaceprogram.com/wiki/Gigantor_XL_Solar_Array">Gigantors</a> and a <a href="https://wiki.kerbalspaceprogram.com/wiki/Fuel_Cell_Array">Fuel Cell Array</a> during mining.
Cooling is provided by 4 <a href="https://wiki.kerbalspaceprogram.com/wiki/Thermal_Control_System_(small)">small Thermal Control Systems</a>.</p>
<p>It has <a href="/static/ksp-tools/multiengine.html#b=[{"dv"%3A530%2C"engines"%3A[{"number"%3A1%2C"type"%3A"LV-N+\"Nerv\""}]}%2C{"dv"%3A930%2C"engines"%3A[{"number"%3A"4"%2C"type"%3A"S3+KS-25+\"Vector\""}%2C{"number"%3A1%2C"type"%3A"LV-N+\"Nerv\""}]}%2C{"dv"%3A870%2C"engines"%3A[{"number"%3A"4"%2C"type"%3A"S3+KS-25+\"Vector\""}%2C{"number"%3A1%2C"type"%3A"LV-N+\"Nerv\""}]}%2C{"dv"%3A320%2C"engines"%3A[{"number"%3A1%2C"type"%3A"LV-N+\"Nerv\""}]}%2C{"dv"%3A1000%2C"engines"%3A[{"number"%3A"4"%2C"type"%3A"S3+KS-25+\"Vector\""}%2C{"number"%3A1%2C"type"%3A"LV-N+\"Nerv\""}]}]&m0=57.507&f={"lf"%3A19895%2C"ox"%3A18205%2C"air"%3A0%2C"mono"%3A0%2C"sf"%3A0%2C"xe"%3A0%2C"el"%3A0%2C"ore"%3A0}">enough fuel</a>
to do the Eve–Moho leg described above, with 1000m/s to spare.
The Pol–Tylo leg also has <a href="/static/ksp-tools/multiengine.html#b=%5B%7B"dv"%3A212%2C"engines"%3A%5B%7B"number"%3A1%2C"type"%3A"LV-N+%5C"Nerv%5C""%7D%5D%7D%2C%7B"dv"%3A42%2C"engines"%3A%5B%7B"number"%3A1%2C"type"%3A"LV-N+%5C"Nerv%5C""%7D%5D%7D%2C%7B"dv"%3A800%2C"engines"%3A%5B%7B"number"%3A1%2C"type"%3A"LV-N+%5C"Nerv%5C""%7D%5D%7D%2C%7B"dv"%3A3803.411453341944%2C"engines"%3A%5B%7B"number"%3A1%2C"type"%3A"LV-N+%5C"Nerv%5C""%7D%2C%7B"number"%3A"4"%2C"type"%3A"S3+KS-25+%5C"Vector%5C""%7D%5D%7D%5D&m0=57.507&f=%7B"lf"%3A19895%2C"ox"%3A18205%2C"air"%3A0%2C"mono"%3A0%2C"sf"%3A0%2C"xe"%3A0%2C"el"%3A0%2C"ore"%3A0%7D">enough fuel</a>,
and leaves us with 3700m/s for the 2300m/s landing.
When using all engines, we can start our landing burn at 18.7m/s² (TWR=2.4), increasing to 70m/s² (TWR=8.9) when empty.
When fully re-fueled, we still get 16.4m/s² (TWR=2.1) for takeoff.</p>
<h2 id="the-trip"><a class="toclink" href="#the-trip">The Trip</a></h2>
<p>My trip around the solar system obviously started at the KSC on Kerbin. From there I went to:</p>
<p>Minmus
<img alt="Minmus" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/02-minmus.jpg"></p>
<p>Mun
<img alt="Mun" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/03-mun.jpg"></p>
<p>Back to Minmus to refuel for the trip to Gilly
<img alt="Gilly" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/05-gilly.jpg"></p>
<p>Moho
<img alt="Moho" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/06-moho.jpg"></p>
<p>Gilly again
<img alt="Gilly" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/07-gilly.jpg"></p>
<p>Ike
<img alt="Ike" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/08-ike.jpg"></p>
<p>Duna
<img alt="Duna" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/09-duna.jpg"></p>
<p>Ike again
<img alt="Ike" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/10-ike.jpg"></p>
<p>Dres
<img alt="Dres" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/11-dres.jpg"></p>
<p>Pol
<img alt="Pol" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/12-pol.jpg"></p>
<p>Tylo
<img alt="Tylo" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/13-tylo.jpg"></p>
<p>Bop
<img alt="Bop" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/14-bop.jpg"></p>
<p>Val
<img alt="Vall" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/15-vall.jpg"></p>
<p>Laythe
<img alt="Laythe" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/16-laythe.jpg"></p>
<p>Back to Pol for a refuel and a really long wait for a transfer window to Eeloo
<img alt="Eeloo" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/18-eeloo.jpg"></p>
<p>Then straight to Minmus, which was on the edge of the ∆v capabilities of the ship.
<img alt="Minmus" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/19-minmus.jpg"></p>
<p>And finally back home on Kerbin, after about 45 years of travel.
<img alt="Kerbin" src="https://blog.dest-unreach.be/2023/07/05/kerbal_grand_tour/kerbal_grand_tour/20-kerbin.jpg"></p>Flight Simulator bush trips2023-06-12T00:00:00+02:002023-06-12T00:00:00+02:00Niobostag:blog.dest-unreach.be,2023-06-12:/2023/06/12/Flight_Simulator_bush_trips/<p><a href="https://www.flightsimulator.com/">Microsoft Flight Simulator</a> provides "bush trips": a set of VFR flights across an area with nice scenery.
When flying these bush trips, you have to rely on the description only, as no navigation aids are available.
To me, as a non-native speaker, the descriptions aren't always very clear.</p>
<p>So I …</p><p><a href="https://www.flightsimulator.com/">Microsoft Flight Simulator</a> provides "bush trips": a set of VFR flights across an area with nice scenery.
When flying these bush trips, you have to rely on the description only, as no navigation aids are available.
To me, as a non-native speaker, the descriptions aren't always very clear.</p>
<p>So I decided to prepare each flight a bit better by having a SkyVector map ready.
This allows me to cross-reference landmarks to reconcile my current position on the map.</p>
<h2 id="alaska-bush-trip-unalaska-to-kulik-lake"><a class="toclink" href="#alaska-bush-trip-unalaska-to-kulik-lake">Alaska bush trip: Unalaska to Kulik Lake</a></h2>
<p><a href="https://flight.fandom.com/wiki/Microsoft_Flight_Simulator_(2020)/Bush_Trips/Unalaska_to_Kulik_Lake">This trip</a> takes you along the Alaska islands in an <a href="https://flight.fandom.com/wiki/CubCrafters_XCub">XCub</a>.
These hops are based on <a href="https://earth.google.com/earth/d/1EmGqZbVZddD55HBgh6jnLFQXjMrHRNCq?usp=sharing">the Google Earth path</a> by <a href="https://flight.fandom.com/wiki/User:Thadius856">Thadius856</a>.</p>
<ol>
<li><a href="https://skyvector.com/?ll=54.04396935000848,-166.05598068158972&chart=301&zoom=1&fpl=%20PADU%205354N16627W%205353N16621W%205355N16617W%205358N16612W%205359N16606W%205404N16555W%205410N16550W%20PAUT">Tom Madsen (Dutch Harbor) Airport, PADU – Akutan Airport, PAUT</a></li>
<li><a href="https://skyvector.com/?ll=54.32533384540808,-165.17710876380963&chart=301&zoom=1&fpl=%20PAUT%205408N16531W%205405N16523W%205404N16517W%205405N16511W%205407N16506W%205408N16500W%205409N16455W%205411N16451W%205413N16448W%205424N16445W%205425N16451W%205429N16453W%20PACS">Akutan Airport, PAUT – Cape Sarichef Airport, PACS</a></li>
<li><a href="https://skyvector.com/?ll=54.715340034077826,-164.12626647858707&chart=301&zoom=1&fpl=%20PACS%205440N16442W%205439N16429W%205442N16422W%205443N16417W%205443N16357W%205442N16347W%205438N16346W%205437N16335W%205445N16320W%20PAKF">Cape Sarichef Airport, PACS – False Pass Airport, PAKF</a></li>
<li><a href="https://skyvector.com/?ll=55.027294000506814,-163.06670379541475&chart=301&zoom=1&fpl=%20PAKF%205452N16320W%205456N16318W%205459N16318W%205507N16318W%205511N16305W%205511N16252W%20PACD">False Pass Airport, PAKF – Cold Bay Airport, PACD</a></li>
<li><a href="https://skyvector.com/?ll=55.56079654164309,-161.9431457509102&chart=301&zoom=3&fpl=%20PACD%205516N16241W%205515N16231W%205507N16221W%205507N16214W%205510N16205W%205514N16157W%205519N16203W%205520N16208W%205523N16208W%205527N16207W%205539N16219W%205554N16142W%205557N16126W%20PAOU">Cold Bay Airport, PACD – Nelson Lagoon Airport, PAOU</a></li>
<li><a href="https://skyvector.com/?ll=55.827927387283424,-160.1605224597821&chart=301&zoom=2&fpl=%20PAOU%205601N16101W%205600N16052W%205556N16049W%205553N16048W%205543N16040W%205542N16035W%205540N16025W%205538N16017W%205540N16008W%205549N15951W%205551N15941W%205559N15930W%205601N15919W%20PAPE">Nelson Lagoon Airport, PAOU – Perryville Airport, PAPE</a></li>
<li><a href="https://skyvector.com/?ll=56.081500612374825,-158.9692840563866&chart=301&zoom=1&fpl=%20PAPE%205555N15903W%205601N15851W%205606N15856W%205609N15850W%20A79">Perryville Airport, PAPE – Chignik Lake Airport, A79</a></li>
<li><a href="https://skyvector.com/?ll=56.608993969879464,-158.44812011592404&chart=301&zoom=2&fpl=%20A79%205621N15830W%205629N15807W%205634N15811W%205637N15815W%205641N15818W%205648N15841W%20PAPH">Chignik Lake Airport, A79 – Port Heiden Airport, PAPH</a></li>
<li><a href="https://skyvector.com/?ll=57.27102472608631,-158.102600096371&chart=301&zoom=2&fpl=%20PAPH%205719N15810W%205734N15741W%20PAPN">Port Heiden Airport, PAPH – Pilot Point Airport, PAPN</a></li>
<li><a href="https://skyvector.com/?ll=57.88108206549591,-157.05541992052366&chart=301&zoom=2&fpl=%20PAPN%205734N15723W%205734N15700W%205736N15650W%205743N15636W%205751N15632W%205802N15651W%205812N15716W%20PAII">Pilot Point Airport, PAPN – Egegik Airport, PAII</a></li>
<li><a href="https://skyvector.com/?ll=58.44658300489315,-157.09213256701034&chart=301&zoom=1&fpl=%20PAII%205817N15732W%205842N15705W%20PAKN">Egegik Airport, PAII – King Salmon Airport, PAKN</a></li>
<li><a href="https://skyvector.com/?ll=58.831373006884284,-155.87223815775369&chart=301&zoom=1&fpl=%20PAKN%205841N15626W%205843N15614W%205842N15609W%205845N15602W%205846N15554W%205847N15540W%205853N15540W%205859N15532W%20PAKL">King Salmon Airport, PAKN – Kulik Lake Airport, PAKL</a></li>
</ol>In Search for a new Monitoring Solution2023-01-07T00:00:00+01:002023-01-07T00:00:00+01:00Niobostag:blog.dest-unreach.be,2023-01-07:/2023/01/07/monitoring_solution/<p>I'm currently running all of my monitoring through <a href="https://prometheus.io/">Prometheus</a>, backed with <a href="https://www.influxdata.com/products/">InfluxDB</a> for the long-term storage of selected metrics.
But recently, I've been limited in functionality when creating graphs in <a href="https://grafana.com/oss/">Grafana</a>.
And although I like Prometheus, it looks like my use-case is outside the domain of Prometheus.</p>
<h2 id="use-case"><a class="toclink" href="#use-case">Use Case</a></h2>
<p>I …</p><p>I'm currently running all of my monitoring through <a href="https://prometheus.io/">Prometheus</a>, backed with <a href="https://www.influxdata.com/products/">InfluxDB</a> for the long-term storage of selected metrics.
But recently, I've been limited in functionality when creating graphs in <a href="https://grafana.com/oss/">Grafana</a>.
And although I like Prometheus, it looks like my use-case is outside the domain of Prometheus.</p>
<h2 id="use-case"><a class="toclink" href="#use-case">Use Case</a></h2>
<p>I want to monitor my home's energy consumption throughout the year.
The metrics I have available are:</p>
<ul>
<li>
<p>An energy <a href="https://prometheus.io/docs/concepts/metric_types/#counter">counter</a> for gas, updated every few minutes.
Since these are counters, they may reset to 0 at any time, and this should be accounted for when graphing.</p>
<p><img alt="Graph of the input metric with annotations" src="https://blog.dest-unreach.be/2023/01/07/monitoring_solution/monitoring/input.png"></p>
</li>
<li>
<p>The energy price. This metric is not known real-time, and is very coarse: 1 datapoint per day.
Depending on the contract, it may be known up to 3 months in advance, or only 1 month after the consumption.</p>
<p><img alt="Graph of the energy price" src="https://blog.dest-unreach.be/2023/01/07/monitoring_solution/monitoring/price.png"></p>
</li>
</ul>
<p>I would like to create the following graphs:</p>
<p><img alt="Desired output graph" src="https://blog.dest-unreach.be/2023/01/07/monitoring_solution/monitoring/result.png"></p>
<p>To make this graph, I need several features from the time series database:</p>
<ul>
<li>Handle counter resets correctly.</li>
<li>Calculate back-and-forth between the cumulative counter and the derived rate:
To convert from energy to price, the current unit price needs to be applied to the marginal usage in the corresponding time period.
You can't just multiply the counter value with the price as that would create a discontinuity.</li>
<li>Be able to draw cumulative/running sums. I.e. start at zero at the left edge.</li>
</ul>
<h2 id="prometheus"><a class="toclink" href="#prometheus">Prometheus</a></h2>
<p>Since Prometheus is my current monitoring solution, I first tried to get it working with Prometheus.
I covered <a href="https://blog.dest-unreach.be/2020/08/16/cumulative-graphs-prometheus/">the cumulative part</a> of the problem before.
That solution can't handle counter resets, but the biggest hurdle is incorporating the price:
it <a href="https://stackoverflow.com/questions/57112964/simple-cumulative-increase-in-prometheus">seems</a>
that there is no way to have Prometheus calculate a cumulative sum.</p>
<h2 id="influxdb"><a class="toclink" href="#influxdb">InfluxDB</a></h2>
<p>Since I'm already using InfluxDB as long-term storage anyway, the next logical step was to skip Prometheus and interact directly with InfluxDB.
I'm fairly new to the Flux language, but the query below seems to give the results I want:</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="s2">"date"</span>
<span class="n">earlyStartTime</span> <span class="o">=</span> <span class="n">date</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="n">d</span><span class="p">:</span> <span class="mi">1</span><span class="n">d</span><span class="p">,</span> <span class="n">from</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">timeRangeStart</span><span class="p">)</span>
<span class="o">//</span> <span class="n">Make</span> <span class="n">sure</span> <span class="n">we</span> <span class="n">have</span> <span class="n">the</span> <span class="n">previous</span> <span class="n">gas_price</span> <span class="ow">in</span> <span class="nb">range</span><span class="o">.</span>
<span class="o">//</span> <span class="n">gas_price</span> <span class="ow">is</span> <span class="n">update</span> <span class="p">(</span><span class="n">at</span> <span class="n">least</span><span class="p">)</span> <span class="n">once</span> <span class="n">a</span> <span class="n">day</span>
<span class="n">from</span><span class="p">(</span><span class="n">bucket</span><span class="p">:</span> <span class="s2">"default"</span><span class="p">)</span>
<span class="o">|></span> <span class="nb">range</span><span class="p">(</span><span class="n">start</span><span class="p">:</span> <span class="n">earlyStartTime</span><span class="p">,</span> <span class="n">stop</span><span class="p">:</span><span class="n">v</span><span class="o">.</span><span class="n">timeRangeStop</span><span class="p">)</span>
<span class="o">|></span> <span class="nb">filter</span><span class="p">(</span><span class="n">fn</span><span class="p">:</span> <span class="p">(</span><span class="n">r</span><span class="p">)</span> <span class="o">=></span>
<span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">_measurement</span> <span class="o">==</span> <span class="s2">"gas"</span> <span class="ow">and</span> <span class="n">r</span><span class="o">.</span><span class="n">_field</span> <span class="o">==</span> <span class="s2">"volume_m3"</span><span class="p">)</span> <span class="ow">or</span>
<span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">_measurement</span> <span class="o">==</span> <span class="s2">"gas_price"</span> <span class="ow">and</span> <span class="n">r</span><span class="o">.</span><span class="n">_field</span> <span class="o">==</span> <span class="s2">"price_EUR_per_m3"</span><span class="p">)</span>
<span class="p">)</span>
<span class="o">|></span> <span class="n">group</span><span class="p">()</span> <span class="o">//</span> <span class="n">join</span> <span class="n">series</span> <span class="n">into</span> <span class="n">a</span> <span class="n">single</span> <span class="n">table</span> <span class="k">for</span> <span class="n">pivot</span>
<span class="o">|></span> <span class="n">pivot</span><span class="p">(</span><span class="n">rowKey</span><span class="p">:</span> <span class="p">[</span><span class="s2">"_time"</span><span class="p">],</span> <span class="n">columnKey</span><span class="p">:</span> <span class="p">[</span><span class="s2">"_measurement"</span><span class="p">],</span> <span class="n">valueColumn</span><span class="p">:</span> <span class="s2">"_value"</span><span class="p">)</span>
<span class="o">|></span> <span class="n">sort</span><span class="p">(</span><span class="n">columns</span><span class="p">:</span> <span class="p">[</span><span class="s2">"_time"</span><span class="p">])</span>
<span class="o">|></span> <span class="n">fill</span><span class="p">(</span><span class="n">usePrevious</span><span class="p">:</span> <span class="n">true</span><span class="p">,</span> <span class="n">column</span><span class="p">:</span> <span class="s2">"gas_price"</span><span class="p">)</span> <span class="o">//</span> <span class="n">fill</span> <span class="ow">in</span> <span class="n">missing</span> <span class="n">gas_price</span>
<span class="o">|></span> <span class="n">difference</span><span class="p">(</span><span class="n">nonNegative</span><span class="p">:</span> <span class="n">true</span><span class="p">,</span> <span class="n">columns</span><span class="p">:</span> <span class="p">[</span><span class="s2">"gas"</span><span class="p">])</span> <span class="o">//</span> <span class="n">convert</span> <span class="n">to</span> <span class="n">per</span><span class="o">-</span><span class="n">sample</span> <span class="n">difference</span><span class="p">,</span> <span class="n">nonNegative</span> <span class="n">detects</span> <span class="n">resets</span>
<span class="o">|></span> <span class="nb">range</span><span class="p">(</span><span class="n">start</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">timeRangeStart</span><span class="p">,</span> <span class="n">stop</span><span class="p">:</span><span class="n">v</span><span class="o">.</span><span class="n">timeRangeStop</span><span class="p">)</span> <span class="o">//</span> <span class="n">crop</span> <span class="n">to</span> <span class="n">actual</span> <span class="n">requested</span> <span class="nb">range</span> <span class="n">after</span> <span class="n">fill</span> <span class="p">(</span><span class="n">to</span> <span class="n">have</span> <span class="n">price</span> <span class="n">info</span><span class="p">)</span>
<span class="o">//</span> <span class="ow">and</span> <span class="n">difference</span> <span class="p">(</span><span class="n">to</span> <span class="n">have</span> <span class="n">data</span> <span class="k">for</span> <span class="n">the</span> <span class="n">first</span> <span class="n">point</span><span class="p">)</span>
<span class="o">|></span> <span class="nb">map</span><span class="p">(</span><span class="n">fn</span><span class="p">:</span> <span class="p">(</span><span class="n">r</span><span class="p">)</span> <span class="o">=></span> <span class="p">({</span><span class="n">_time</span><span class="p">:</span> <span class="n">r</span><span class="o">.</span><span class="n">_time</span><span class="p">,</span> <span class="n">_value</span><span class="p">:</span> <span class="n">r</span><span class="o">.</span><span class="n">gas</span> <span class="o">*</span> <span class="n">r</span><span class="o">.</span><span class="n">gas_price</span><span class="p">}))</span>
<span class="o">|></span> <span class="n">aggregateWindow</span><span class="p">(</span><span class="n">every</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">windowPeriod</span><span class="p">,</span> <span class="n">fn</span><span class="p">:</span> <span class="nb">sum</span><span class="p">,</span> <span class="n">createEmpty</span><span class="p">:</span> <span class="n">false</span><span class="p">)</span> <span class="o">//</span> <span class="n">We</span> <span class="n">no</span> <span class="n">longer</span> <span class="n">need</span> <span class="n">dense</span> <span class="n">data</span> <span class="n">at</span> <span class="n">this</span> <span class="n">point</span>
<span class="o">|></span> <span class="n">cumulativeSum</span><span class="p">()</span>
</code></pre></div>Rootless podman on Debian Bullseye2023-01-03T00:00:00+01:002023-01-03T00:00:00+01:00Niobostag:blog.dest-unreach.be,2023-01-03:/2023/01/03/rootless_podman_on_debian_bullseye/<p>My home server already uses LXC-containers.
I use them to keep services isolated, so that I can upgrade my mail server without causing issues to my <a href="https://nextcloud.com/">Nextcloud</a> setup.
But these containers contain a full OS inside them.
And while this works, it's a bit of work to upgrade a handful …</p><p>My home server already uses LXC-containers.
I use them to keep services isolated, so that I can upgrade my mail server without causing issues to my <a href="https://nextcloud.com/">Nextcloud</a> setup.
But these containers contain a full OS inside them.
And while this works, it's a bit of work to upgrade a handful of containers every few months.
Docker-style containers should be a better fit for this single-service-per-container setup.</p>
<p>After reading up a bit, I decided to go for <a href="https://podman.io/">podman</a>.
I was mostly convinced by the ability to run rootless, but it seems Docker is getting that as well.</p>
<h2 id="rootless-networking"><a class="toclink" href="#rootless-networking">Rootless networking</a></h2>
<p>In my LXC setup, every container gets its own <code>veth</code>-based network interface.
On the host, these <code>veth</code>-interfaces are bridged together with the physical network card.
That way, every container gets its own MAC and IP(v6) address.</p>
<p>But since we're running as a normal user, podman can't create a veth-pair.
It was very interesting to learn (how podman solved this)<a href="https://mcastelino.medium.com/slirp4netns-how-does-it-work-5c0bd31200ce">slirp4netns</a> with some <code>netns</code> and <code>tap</code> magic!</p>
<p>The result isn't exactly equivalent to what I had on the LXC-side:
the containers do have their own IP, but it's not exposed to the outside world.
I found (a mailing list post describing how to roll my own)<a href="https://lists.podman.io/archives/list/podman@lists.podman.io/thread/W6MCYO6RY5YFRTSUDAOEZA7SC2EFXRZE/">manual-netns</a>, but I haven't figured out yet how to combine this with DHCP...</p>
<h2 id="compose"><a class="toclink" href="#compose">Compose</a></h2>
<p>Podman has its own <code>docker-compose</code> equivalent.
Unsurprisingly, it's called <code>podman-compose</code>, but is otherwise very similar.</p>
<p>I didn't find a ready-made way to auto-start a service,
so I rolled my own systemd service which I put in <code>~/.config/systemd/user/podman-compose.service</code>:</p>
<div class="highlight"><pre><span></span><code><span class="k">[Unit]</span><span class="w"></span>
<span class="na">Description</span><span class="o">=</span><span class="s">Rootless pod (podman-compose)</span><span class="w"></span>
<span class="k">[Service]</span><span class="w"></span>
<span class="na">Type</span><span class="o">=</span><span class="s">oneshot</span><span class="w"></span>
<span class="na">RemainAfterExit</span><span class="o">=</span><span class="s">true</span><span class="w"></span>
<span class="na">WorkingDirectory</span><span class="o">=</span><span class="s">%h</span><span class="w"></span>
<span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/local/bin/podman-compose up -d --remove-orphans</span><span class="w"></span>
<span class="na">ExecStop</span><span class="o">=</span><span class="s">/usr/local/bin/podman-compose down</span><span class="w"></span>
<span class="k">[Install]</span><span class="w"></span>
<span class="na">WantedBy</span><span class="o">=</span><span class="s">default.target</span><span class="w"></span>
</code></pre></div>
<p>This will run <code>compose.yaml</code> from the home directory of the user.</p>The Unifi Protect G4 Doorbell2022-06-12T00:00:00+02:002022-06-12T00:00:00+02:00Niobostag:blog.dest-unreach.be,2022-06-12:/2022/06/12/unifi-g4-doorbell/<p>I've been dragging my feet for years to get a video doorbell.
I wanted a doorbell that "just works" as a doorbell, even if the WiFi is down,
and I didn't want my video footage to be stored in the cloud,
but only accessible when on the local network,
either …</p><p>I've been dragging my feet for years to get a video doorbell.
I wanted a doorbell that "just works" as a doorbell, even if the WiFi is down,
and I didn't want my video footage to be stored in the cloud,
but only accessible when on the local network,
either directly, or via VPN.
Apparently, these are fairly high requirements nowadays...
I settled for the <a href="https://store.ui.com/products/uvc-g4-doorbell">Unifi Protect G4 Doorbell</a>.
I wanted to get the Pro-variant, but that one still hasn't left Early Access.</p>
<p>There were some issues, though.
The first one is that this doorbell works on 16V AC, and requires the chime to also work with that.
While 16V may be the standard in the US, here where I live, most chimes are 6V.
They'll probably work with 16V, but not for a very long time.</p>
<h2 id="the-chime"><a class="toclink" href="#the-chime">The Chime</a></h2>
<p>You don't <em>need</em> to connect a chime.
You can go all in digital and only use your phone to notify you someone is at the door.
But if you do, you need to connect a special white chime-box adapter in series with the doorbell:</p>
<p><img alt="Chime box adapter" src="https://blog.dest-unreach.be/2022/06/12/unifi-g4-doorbell/unifi-g4-doorbell/chime-box-adapter.jpg"></p>
<p>Presumably, this box would output 16V AC when the door bell button is pressed.
So I needed to find out what exactly is being output from this box, and how I can convert it to the correct voltage.</p>
<p>Contrary to my expectations, there was some output present while the system was in idle:</p>
<p><img alt="Idle state output" src="https://blog.dest-unreach.be/2022/06/12/unifi-g4-doorbell/unifi-g4-doorbell/idle.png"></p>
<p>The signal is non-sinusoidal, but 50Hz periodic and about 5.5Vpeak-to-peak.
When I pressed the button, nothing happened.</p>
<p>In the Doorbell's settings, you can choose which chime is attached.
By default, it's configured to "None".
The other settings are "Mechanical" and "Digital".
When selecting Digital, you can choose a duration between 1 and 10 seconds.</p>
<p>So I choose "Mechanical", and pushed the button again.
Now something did happen: the output of the chime box adaptor jumped up to 21Vrms AC and stayed there for many seconds,
but the Doorbell shut down because it lost power. It recovered after 30seconds or so.</p>
<p><img alt="Mechanical chime, unloaded" src="https://blog.dest-unreach.be/2022/06/12/unifi-g4-doorbell/unifi-g4-doorbell/chime, unloaded.png"></p>
<p>It looks like the chime box adapter outputs the full output of the transformer to the chime connections,
leaving no power for the doorbell itself.
I'm assuming that the voltage is supposed to be shared between the chime and the doorbell during this period,
but since I have no chime connected (yet), all voltage falls over my oscilloscope and nothing remains for the camera.</p>
<p>So I decided to load the output with a (second) transformer to step down the voltage from 21Vrms AC to something in the neighbourhood of 6~10V.
This seems to have done the trick:</p>
<p><img alt="Mechanical chime, loaded with inductor" src="https://blog.dest-unreach.be/2022/06/12/unifi-g4-doorbell/unifi-g4-doorbell/mechanical chime, loaded with inductor.png"></p>
<p>The chime box output full voltage for around 430ms, and resets to its idle state,
leaving the camera enough power to keep on working.</p>Reverse engineering the Mitsubishi heat-pump WiFi adapter2022-06-04T00:00:00+02:002022-06-04T00:00:00+02:00Niobostag:blog.dest-unreach.be,2022-06-04:/2022/06/04/mitsubishi-wifi-adapter/<p>Recently we had a heat pump (A/C, air conditioner, AirCon) installed.
The indoor units came with the usual IR remote,
but they also had a built-in WiFi module along with a smartphone app to control them from anywhere.
I'm not a big fan of connecting home appliances to clouds …</p><p>Recently we had a heat pump (A/C, air conditioner, AirCon) installed.
The indoor units came with the usual IR remote,
but they also had a built-in WiFi module along with a smartphone app to control them from anywhere.
I'm not a big fan of connecting home appliances to clouds for a variety of reasons:</p>
<ul>
<li>
<p>Running a cloud service requires a continuous stream of money to keep the service running.
So either you will have to pay some sort of subscription, or the service will (eventually) stop working.
There is also the possibility that the manufacturer is very altruistic, but that isn't very likely.</p>
</li>
<li>
<p>If my appliance is connected to the internet, the manufacturer can choose to alter its functionality.
<a href="https://thenextweb.com/news/update-brainwashes-microwaves-thinking-theyre-steam-ovens">Or make it stop working entirely</a>.</p>
</li>
<li>
<p>Connecting things to the cloud makes them more vulnerable to attacks.
Right now, an attacker needs to be within infrared range of my unit (a few meters) to control it.
Once the device phones home to its cloud, an attacker can potentially control thousands of units from their couch.</p>
</li>
</ul>
<p>Don't get me wrong. I love when appliances offer some sort of API to control them externally.
And I see why a cloud-service is a nice addition for a lot of people.
But I prefer local control.</p>
<h2 id="the-mitsubishi-wifi-adapter"><a class="toclink" href="#the-mitsubishi-wifi-adapter">The Mitsubishi WiFi adapter</a></h2>
<p>My heat pump came with a MAC-577IF2-E adapter built-in.
So I set out to try and locally control the unit via WiFi.
If you're in a hurry, I'll save you some time: <em>I didn't succeed</em>.
But here is what I did find out, in the hope it can be useful for other people.</p>
<p>I started by looking around to interesting projects:
I found the <a href="https://github.com/ncaunt/meldec">meldec</a> GitHub project that can decode the messages sent to MELcloud.
Another related project I found was <a href="https://github.com/SwiCago/HeatPump">HeatPump</a>,
but that focuses on communicating directly with the unit over its serial interface.</p>
<h2 id="first-discoveries"><a class="toclink" href="#first-discoveries">First discoveries</a></h2>
<p>For setup, I switched the adapter into Access Point mode.
After associating with it using the SSID & WPA-PSK printed on the box,
I could connect to a web-server to configure the network settings.
The page was fairly simple:</p>
<p><img alt="network setup screen" src="https://blog.dest-unreach.be/2022/06/04/mitsubishi-wifi-adapter/mitsubishi/network.png"></p>
<p>Once the details were filled in, it connected to my home WiFi network just fine.
The MAC address of the unit stats with <code>70:61:BE</code>, which is assigned to <code>Wistron Neweb Corporation</code>.
After requesting an IP via DHCP, it started to phone home, of course.
First thing it did was a DNS-lookup for <code>production.receiver.melcloud.com</code>.</p>
<p>The first request to that domain is a plain-text HTTP POST-request to <code>/synchro</code>:</p>
<div class="highlight"><pre><span></span><code>POST /synchro HTTP/1.1
Host: production.receiver.melcloud.com:80
Accept: */*
User-Agent: MAC-577IF-E
Pragma: no-cache
Content-type: text/plain; charset=UTF-8
Content-Length: 68
<?xml version="1.0" encoding="UTF-8"?><LSV><SYNCHRO></SYNCHRO></LSV>
</code></pre></div>
<p>The response contained the current datetime:</p>
<div class="highlight"><pre><span></span><code>HTTP/1.0 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Date: Sat, 4 Jun 2022 09:47:47 GMT
Server: Microsoft-IIS/10.0
X-Robots-Tag: noindex, nofollow, noimageindex
<?xml version="1.0" encoding="UTF-8"?><CSV><SYNCHRO><DATE>2022/06/04 09:47:47</DATE></SYNCHRO></CSV>
</code></pre></div>
<p>So far so good.
The next connection, however, was an encrypted HTTPS request to the same domain.
<code>production.receiver.melcloud.com</code> presents an HTTPS certificate that is signed by <code>MELCloud Root Authority</code>,
and thus not trusted by browsers (by default).
I tried to redirect this request to my own server, presenting a look-alike certificate chain.
But the device refused with an <code>Unknown CA</code> error:
it didn't like my own spoofed CA and wanted to see the actual Mitsubishi CA.
Normally, I consider this a good thing for security, but in this particular case it's a bummer...</p>
<h2 id="inbound"><a class="toclink" href="#inbound">Inbound</a></h2>
<p>There also seems to be a webserver listening on the device itself on TCP port 80.
But according to <a href="https://nmap.org/">nmap</a>, there is nothing else listening on TCP (all other ports are connection-refused).
Note that a normal nmap-scan crashes the web server, so you won't see it as open on a consecutive scan.</p>
<p>But the web server seems to be locked down: requests to <code>/</code> require authentication, which I don't have/know:</p>
<div class="highlight"><pre><span></span><code>> GET / HTTP/1.1
> Host: IP-address-of-unit-omitted-for-privacy
> User-Agent: curl/7.79.1
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Content-Type: text/plain
< Content-Length: 22
< WWW-Authenticate: Basic realm="generic"
<
< Authorization Required
</code></pre></div>
<p>Some URLs do work without authentication: <code>/license</code>, <code>/common.css</code>, <code>/javaScript.js</code> work.
Others return a 404 instead of a 401.</p>
<h2 id="digging-deeper"><a class="toclink" href="#digging-deeper">Digging deeper</a></h2>
<p>Someone <a href="https://github.com/ambiot/amb1_sdk/blob/master/doc/UM0034%20Realtek%20Ameba-1%20memory%20layout.pdf">posted the firmware</a> to GitHub. Time to brush off my reverse engineering skills.</p>
<p>Based on a simple <code>strings</code> of the firmware, I found <code>C:\sdk-ameba-v4.0c\component\common\mbed\targets\hal\rtl8195a\gpio_api.c</code>.
So it looks like the hardware may be an <a href="https://www.amebaiot.com/en/control-mcu/">Ameba 1</a>.
This houses an ARM Cortex M3 CPU clocked at 166MHz with 2.5MB of SRAM and built-in WiFi (2.4GHz only).
I used <a href="https://github.com/radareorg/radare2">radare2</a> to disassemble the binary,
and kept the <a href="https://users.ece.utexas.edu/~valvano/EE345M/CortexM3InstructionSet.pdf">ARM Cortex M3 Instruction Set Reference</a> nearby.
It looks like the SDK for this SoC is available <a href="https://github.com/ambiot/amb1_sdk">on GitHub</a>.</p>
<p>The Flash layout section of <a href="https://github.com/ambiot/amb1_sdk/blob/master/doc/UM0034%20Realtek%20Ameba-1%20memory%20layout.pdf">the memory layout PDF from the SDK</a> contains some information on the boot loader process.
From the description, it seems that an image starts with a 4-byte length, a 4-byte address, and 8 bytes of <code>0xffffffffffffffff</code>.
The Memory Mapping section of <a href="https://file.elecfans.com/web1/M00/BA/E6/o4YBAF6ijEqAQpDBACLQAF2ggPY084.pdf">the RTL8195A data sheet</a> tells what address is mapped to what hardware.
Based on that info, the firmware contains two sections:</p>
<ul>
<li>A first section of 0x4aefc = 306_940 bytes, to be loaded at 0x10006000, which is in SRAM</li>
<li>A second section of 0x10cea4 = 1_101_476 bytes, to be loaded at 0x30000000, which is in SDRAM</li>
<li>The final 4 bytes of the file. Maybe a CRC of some sort?</li>
</ul>
<p>While scrolling through the <code>strings</code> output, the function at <code>0x3001de60</code> caught my eye:
It's the only place to reference the string <code>401 Unauthorized</code>.
The only place where this function is referenced, is around <code>0x30028984</code>.
This section loads the string <code>Authorization</code>, and calls a function (I'm guessing to find this header).
Next, it checks if the returned value (?) starts with <code>Basic</code>, and passes the rest of the value to <code>0x30028820</code>.</p>
<p><code>0x30028820</code> seems to be the "verify authorization"-function.
It seems to iterate through a linked list of entries, checking the path (?).
If the entry matches, the base64-auth-value is compared to another attribute of the entry.
If it matches, the "return a 401"-part is skipped.
Interestingly, the comparison is done by first <code>memcmp()</code>ing, and later verifying that <code>strlen()</code> is equal.
The actual <code>memcpy()</code> implementation is in ROM, but it's possible this is vulnerable to a timing side channel
(although doing timing side channels over WiFi is hard...).</p>
<p>Unfortunately, this function just checks the auth-value against a pre-computed list.
So I still need to find the code that generates this list...
The list is loaded from <code>*(*(0x30029130)+0x18)</code>.
<code>0x30029130</code> contains <code>0x1004f134</code>, but it may be overwritten by the time we get here.
<code>0x1004f134+0x18</code> = <code>0x1004f14c</code> contains <code>0x00000000</code> at bootup.
This will definitely be overwritten, since otherwise no authentication will work (the linked list has 0 entries).
And I know that <code>/config</code> can be accessed with username <code>user</code> and the Key-password printed on the device.</p>
<h2 id="trying-another-approach"><a class="toclink" href="#trying-another-approach">Trying another approach</a></h2>
<p><code>0x300696bc</code> looks interesting: it seems to contain a list of paths & usernames.
The entries seem to have the odd size of 0xe7=231 bytes.</p>
<table>
<thead>
<tr>
<th>Path \ user</th>
<th>user</th>
<th>root</th>
<th>suser</th>
<th>admin</th>
</tr>
</thead>
<tbody>
<tr>
<td>/network</td>
<td></td>
<td>x</td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>/unitinfo</td>
<td></td>
<td>x</td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>/service</td>
<td></td>
<td>x</td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>/server</td>
<td></td>
<td></td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>/update</td>
<td></td>
<td></td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>/updateFinish</td>
<td></td>
<td></td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>/updateFail</td>
<td></td>
<td></td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>/default</td>
<td></td>
<td></td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>/analyze</td>
<td></td>
<td></td>
<td></td>
<td>x</td>
</tr>
<tr>
<td>/apinfo</td>
<td></td>
<td>x</td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>/config</td>
<td>x</td>
<td>x</td>
<td>x</td>
<td>x</td>
</tr>
<tr>
<td>/smart</td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>/adapter_image2.gif</td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>/javaScript.js</td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>/common.css</td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>/license</td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>/</td>
<td></td>
<td>x</td>
<td>x</td>
<td>x</td>
</tr>
</tbody>
</table>
<p>Based on <code>/license</code>, I'm assuming "no user listed" means "unauthenticated".</p>Backups2022-05-04T00:00:00+02:002022-05-04T00:00:00+02:00Niobostag:blog.dest-unreach.be,2022-05-04:/2022/05/04/backup/<p>Recently I was rethinking my off-site backup strategy.
Ever since Amazon AWS launched their <a href="https://aws.amazon.com/blogs/aws/new-amazon-s3-storage-class-glacier-deep-archive/">Glacier Deep Archive</a>
storage tier, it looked really interesting for backup use-cases.
For starters, it's super cheap to keep your bytes around:
$1.01376/TiB/month in most regions <a href="https://aws.amazon.com/s3/pricing/">as of this writing</a>.
I haven't found …</p><p>Recently I was rethinking my off-site backup strategy.
Ever since Amazon AWS launched their <a href="https://aws.amazon.com/blogs/aws/new-amazon-s3-storage-class-glacier-deep-archive/">Glacier Deep Archive</a>
storage tier, it looked really interesting for backup use-cases.
For starters, it's super cheap to keep your bytes around:
$1.01376/TiB/month in most regions <a href="https://aws.amazon.com/s3/pricing/">as of this writing</a>.
I haven't found any other storage proposition this cheap.
Of course there is some fine print, but it matches my backup use-case:</p>
<ul>
<li>you are charged for at least 180 days of storage, even if you delete the objects before that</li>
<li>you are charged for some overhead on top of the stored objects.
Especially for smaller objects, this can add up significantly</li>
</ul>
<p>But all in all, it still was very attractive.</p>
<h2 id="requirements"><a class="toclink" href="#requirements">Requirements</a></h2>
<p>Based on discussions I had, it seems I have a fairly beefy list of requirements:</p>
<ul>
<li>
<p>Bringing the backup up-to-date should be cheap.
This means I don't want to update the full content every single time.
I prefer to only upload what has changed (rsync-style). </p>
</li>
<li>
<p>Keeping the backup around should be cheap.</p>
</li>
<li>
<p>Restoring the backup may be expensive, either in time, money or both.
Since this is not my first backup, I intend to never need it.
So paying several hundred euro/dollar to get my most valuable files back is fine.</p>
</li>
<li>
<p>Restoring should be "easy".
Since I'll be using this backup in a disaster recovery mode,
I prefer to be able to restore crucial files without the need for any particular software,
just relying on standard tools.</p>
</li>
<li>
<p>Backups need to be client-side encrypted, preferably with auditable/trusted tools, independent of the backup tool.</p>
</li>
</ul>
<p>I looked around for existing projects and/or products, but didn't find one that checked all my boxes.
Especially the "client-side encrypted" seemed like a high bar.
So I decided to roll my own.</p>
<h2 id="design"><a class="toclink" href="#design">Design</a></h2>
<p>S3 natively supports versioning of objects.
By default, you see the latest version of an object.
Or, if the last version is a "delete marker", the object is hidden by default.
But you can request previous versions explicitly if you need them.
In addition, I use <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html">S3 Lifecycle management</a>
to remove old versions after a configured amount of time. </p>
<p>This setup ensures the "easy restore"-requirement:
The S3 web console gives me access to the most recent backup content,
but allows access to previous versions if needed.</p>
<p>The client-side encryption is a bit more challenging:
the backup-tool needs to figure out if a particular file has changed since the previous backup or not.
You can't just encrypt the file again, and compare that, since encrypting the same file twice is not guaranteed to give the same result.
My solution is to add metadata to the S3-object: the size of the plaintext file, and a hash of the plaintext file.
The size is included because it's very cheap to check: if a file's size has changed, the file has changed and needs uploading.
The hash is more expensive to calculate, but will spot changes even when the file size is the same.
I know there is an astronomically small chance that a changed file will not result in a changed hash, but I'm taking my chances.</p>
<p>To make things more efficient, I also included a client-side cache of the S3 content.
Listing the objects in an S3 bucket is fairly efficient, but getting the metadata requires a HEAD-call per object.
At 71 seconds per 1000 calls, which is what I practically get, this takes way to long for a 200k-object backup.
Not to mention the monetary cost of doing these calls every single time.
But since this data is cached locally, care should be taken to never modify the bucket directly.
This would cause the cache to be incorrect, and may result in bad backups.</p>
<h2 id="small-files"><a class="toclink" href="#small-files">Small files</a></h2>
<p>One of the things I want to backup are my git repositories.
Git has the habit of creating lots of small files.
And while the above design supports small files, it gets slow and expensive:
Uploading ten 1-byte files takes way longer than uploading a single 10-byte file.
And since there is an additional 40kB overhead per object (some of it billed at standard S3 pricing),
a single-byte file is (relatively speaking) expensive.</p>
<p>So I wanted to ZIP together smaller files and upload the ZIP instead.
This is where it gets tricky: you want both very small ZIPs and very large ZIPs to cover contradicting needs.
On one hand, you want the ZIPs to be as large as possible, to maximize the efficiency-gains.
On the other hand, you want the ZIPs to be as small as possible, since a change to a single file in the ZIP requires the upload of the complete new ZIP.</p>
<p>From a restore point of view, you also want the ZIPs to be "logical".
Given a filename and a point in time, I want to be able to cherry-pick that file from the backup with as little overhead as possible.
I want to either find the file itself in the backup, or should be able to tell fairly easy which ZIP contains the given file.</p>
<p>I went through several variants of grouping-logic.
My current implementation takes a configurable threshold as input.
Files that are larger than the threshold are stored to S3 directly.
Smaller files are grouped based on their filenames, where the algorithm tries to find the longest prefix that will result in a ZIP of at least the required size. So given the following files and a threshold of 1 MiB:</p>
<div class="highlight"><pre><span></span><code><span class="nf">large-file</span><span class="w"> </span><span class="mi">5</span><span class="w"> </span><span class="no">MiB</span><span class="w"></span>
<span class="na">.git</span><span class="err">/</span><span class="w"></span>
<span class="w"> </span><span class="nf">a</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="no">KiB</span><span class="w"></span>
<span class="w"> </span><span class="nf">b</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="no">KiB</span><span class="w"></span>
<span class="w"> </span><span class="nf">c</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="no">KiB</span><span class="w"></span>
<span class="w"> </span><span class="nf">d</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="no">MiB</span><span class="w"></span>
</code></pre></div>
<p>It will pass through <code>large-file</code> and <code>.git/d</code>, but will ZIP the other files in the <code>.git/</code> folder together as <code>.git/*~.zip</code>.
To find a file for restore, I know that the file will be either directly visible on S3, or be in first ZIP-file I encounter by removing characters from the end of the filename I'm looking for.</p>Surviving Mars2022-04-11T00:00:00+02:002022-04-11T00:00:00+02:00Niobostag:blog.dest-unreach.be,2022-04-11:/2022/04/11/surviving_mars/<p>I discovered another game that gives me an excuse to write silly calculator modules: <a href="https://www.paradoxinteractive.com/games/surviving-mars/about">Surviving Mars</a>. Some random notes and a few tools I wrote, or interesting tools I found online:</p>
<ul>
<li><a href="/static/surviving-mars-tools/colonists.html">Population calculator</a></li>
<li><a href="/static/surviving-mars/tools/electricity.html">Electricity calculator (work in progress)</a></li>
</ul>Kerbal Space Program2021-08-25T00:00:00+02:002021-08-25T00:00:00+02:00Niobostag:blog.dest-unreach.be,2021-08-25:/2021/08/25/kerbal_space_program/<p>I am usually late to proverbial parties, but I think I’ve outdone myself by discovering the computer game <a href="https://www.kerbalspaceprogram.com/">Kerbal Space Program</a> roughly 10 years after it first became playable. It’s the kind of game that triggers my “I’ll optimize this by writing a script”, so here are …</p><p>I am usually late to proverbial parties, but I think I’ve outdone myself by discovering the computer game <a href="https://www.kerbalspaceprogram.com/">Kerbal Space Program</a> roughly 10 years after it first became playable. It’s the kind of game that triggers my “I’ll optimize this by writing a script”, so here are a few of the tools I wrote, or interesting tools I found online:</p>
<ul>
<li><a href="https://forum.kerbalspaceprogram.com/index.php?/topic/87463-173-community-delta-v-map-27/page/10/#comment-2581323">∆v chart by Kowgan</a>
(<a href="https://blog.dest-unreach.be/2021/08/25/kerbal_space_program/ksp/dv-chart.png">local copy</a>)</li>
<li><a href="https://alexmoon.github.io/ksp/">Alex Moon's Launch Window Planner</a></li>
<li><a href="https://krafpy.github.io/KSP-MGA-Planner/">A multi-gravity assist planner</a></li>
<li><a href="https://kerbal-transfer-illustrator.netlify.app/">Another planner</a></li>
<li><a href="https://meyerweb.com/eric/ksp/resonant-orbits/">Eric Meyer's Resonant Orbit Calculator</a></li>
<li><a href="https://www.jacktex.eu/software/ksp_parachutes.php">Parachute planner by JackTex</a></li>
<li><a href="/static/ksp-tools/electricity.html">Electricity generation & storage</a></li>
<li><a href="/static/ksp-tools/engines.html">Engine selection</a></li>
<li><a href="/static/ksp-tools/commnet-link-budget.html">CommNet link budget</a></li>
<li><a href="/static/ksp-tools/commnet-line-of-sight.html">CommNet line of sight</a></li>
<li><a href="/static/ksp-tools/mining.html">Mining</a></li>
<li><a href="/static/ksp-tools/orbits.html">Orbit transfer</a></li>
<li><a href="/static/ksp-tools/moon-departure.html">Interplanetary departure from a moon</a></li>
<li><a href="/static/ksp-tools/planner.html">Mission planner</a> (work in progress)</li>
<li><a href="/static/ksp-tools/multiengine.html">Multi-engine planner</a></li>
</ul>
<p>And some random notes:</p>
<ul>
<li>
<p>When launching from KSC to rendez-vous with something in a circular orbits at 100km altitude:
For vertical launches with "my usual ascend profile",
launch when the target is 26º before the KSC position (viewable as Phase Angle).
For spaceplane launches (with <a href="https://wiki.kerbalspaceprogram.com/wiki/CR-7_R.A.P.I.E.R._Engine">RAPIERs</a> and 10º pitch up all the way),
launch when the target is 16º before the KSC.</p>
</li>
<li>
<p>When starting from a 100km circular orbits around Kerbin, a deorbit burn down to Pe=40km will touch down 152º later. Craft consists of:</p>
<ul>
<li><a href="https://wiki.kerbalspaceprogram.com/wiki/Mk1-3_Command_Pod">Mk3-1 Command module</a></li>
<li><a href="https://wiki.kerbalspaceprogram.com/wiki/RC-L01_Remote_Guidance_Unit">RC-L01</a> (for storing science and remote control of probes)</li>
<li><a href="https://wiki.kerbalspaceprogram.com/wiki/Heat_Shield_(2.5m)">Large Heath Shield</a></li>
<li><a href="https://wiki.kerbalspaceprogram.com/wiki/Mk16-XL_Parachute">Mk16-XL Parachute</a></li>
<li>2 × <a href="https://wiki.kerbalspaceprogram.com/wiki/Mk2-R_Radial-Mount_Parachute">Mk2-R Radial-Mount Parachute</a></li>
</ul>
</li>
<li>
<p>For interplanetary travel, you need ~930m/s of Δv to leave Kerbin <a href="https://wiki.kerbalspaceprogram.com/wiki/Sphere_of_influence">SoI</a> starting from a 100kmAGL orbits,
and an additional 100~1000m/s to reach other planets.
Design for a high enough TWR to do this in a reasonable time.
Multi-impulse departures are possible:
as long as the apoapsis is < 8.9MmASL (9.5Mm),
the Mun will not alter your orbits.
The below table tries to keep burns <2 minutes until we cross the orbits of the Mun.</p>
</li>
</ul>
<table>
<thead>
<tr>
<th>TWR [G<sub>Kerbin</sub>]</th>
<th>Burns for escape</th>
<th>Burns for 2100m/s</th>
</tr>
</thead>
<tbody>
<tr>
<td>1.79</td>
<td>53s</td>
<td>120s</td>
</tr>
<tr>
<td>1.5</td>
<td>63s</td>
<td>56s + 87s</td>
</tr>
<tr>
<td>1.0</td>
<td>95s</td>
<td>84s + 130s</td>
</tr>
<tr>
<td>0.75</td>
<td>111s + 15s</td>
<td>111s + 174s</td>
</tr>
<tr>
<td>0.696</td>
<td>120s + 16s</td>
<td>120s + 186s</td>
</tr>
<tr>
<td>0.5</td>
<td>120s + 47s + 23s</td>
<td>120s + 47s + 261s</td>
</tr>
<tr>
<td>0.25</td>
<td>2×120s + 94s + 45s</td>
<td>2×120s + 94s + 522s</td>
</tr>
<tr>
<td>0.1</td>
<td>6×120s + 115s + 113s</td>
<td>6×120s + 115s + 1305s</td>
</tr>
</tbody>
</table>
<ul>
<li>
<p>Most engines produce a constant thrust while on.
Its fuel consumption scales with the time the engine is on.
But the energy scales with the distance the force is applied.
This means that running an engine at a higher speed, yields more kinetic energy for the same amount of fuel.
This is known as the <a href="https://en.wikipedia.org/wiki/Oberth_effect">Oberth effect</a>.</p>
<p>When starting out at Minmus, this means that it could be beneficial to first spend some ∆v to dive into Kerbin,
and burn at periapsis for the escape trajectory.
Escaping from Kerbin from a Minmus orbits requires at least ~115m/s.
The boundary seems to be 274.1m/s: for ∆v's above that, it's optimal to
first burn 228.1m/s to put our Kerbin periapsis at 70km (just above the atmosphere),
and burn the rest of our ∆v there.
For a 500m/s total ∆v, this boosts our hyperbolic excess velocity from 670.1m/s (when expended at Minmus orbits)
to 1294.8m/s when diving first and spending the remaining ∆v near Kerbin.
This is the case for all trajectories to other planets.</p>
<p><img alt="Hyperbolic Excess Velocity after burn(s) from Minmus Orbit" src="https://blog.dest-unreach.be/2021/08/25/kerbal_space_program/ksp/hev-minmus.png"></p>
</li>
<li>
<p>The orbits of Minmus has an inclination of 6°.
Ideally, we want to launch in this orbital plane to avoid plane-change maneuvers.
Launching into a 6° inclined orbits can be done from launch sites with a latitude of less than 6°, i.e. the Kerbal Space Center.
The Longitude of the ascending node of Minmus’s orbits is 78°.
So we can either launch slightly northbound when KSC passes 78° right ascension,
or slightly southwards when KSC passes 258°.
In-game, it’s not easy to read out the current right ascension.
We can, however see the Longitude of the Ascending Node (LAN) of our current orbits in the second orbital info pane:</p>
<p><img alt="Orbital pane 2" src="https://blog.dest-unreach.be/2021/08/25/kerbal_space_program/ksp/orbit-2.png"></p>
<p>Since KSC is in the southern hemisphere, our current position is 90° before our ascending node.
In other words: launch slightly south (97°) when LAN reads 348°, or slightly north (83°) when LAN reads 168°.
You need to eyeball these directions, especially at the beginning of the launch, when mostly vertical, where the Heading-indicator is very sensitive.
I do my gravity turns to the right.
In that case, you can align the 90° line on the navball with the longer tick mark of the G-meter for 83° departures.
97° departures should aim for the letter G in the G-meter.</p>
<p><img alt="Navball aimed for 83° departure" src="https://blog.dest-unreach.be/2021/08/25/kerbal_space_program/ksp/navball-83E.png">
<img alt="Navball aimed for 83° departure" src="https://blog.dest-unreach.be/2021/08/25/kerbal_space_program/ksp/navball-97E.png"></p>
</li>
<li>
<p>Launch windows for the first few years:</p>
</li>
</ul>
<table>
<thead>
<tr>
<th>Departure Date</th>
<th>Arrival Date</th>
<th>Destination</th>
<th>∆v (m/s)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Y1 D98</td>
<td>Y1 D64</td>
<td>Moho</td>
<td>5165</td>
</tr>
<tr>
<td>Y1 D196</td>
<td>Y4 D138</td>
<td>Jool</td>
<td>5043</td>
</tr>
<tr>
<td>Y1 D231</td>
<td>Y2 D75</td>
<td>Duna</td>
<td>1678</td>
</tr>
<tr>
<td>Y1 D242</td>
<td>Y6 D227</td>
<td>Eeloo</td>
<td>3635</td>
</tr>
<tr>
<td>Y1 D269</td>
<td>Y1 D405</td>
<td>Moho</td>
<td>5031</td>
</tr>
<tr>
<td>Y1 D390</td>
<td>Y3 D93</td>
<td>Dres</td>
<td>3526</td>
</tr>
<tr>
<td>Y2 D86</td>
<td>Y2 D243</td>
<td>Moho</td>
<td>4833</td>
</tr>
<tr>
<td>Y2 D160</td>
<td>Y2 D357</td>
<td>Eve</td>
<td>2741</td>
</tr>
<tr>
<td>Y2 D242</td>
<td>Y5 D304</td>
<td>Jool</td>
<td>4978</td>
</tr>
<tr>
<td>Y2 D245</td>
<td>Y2 D386</td>
<td>Moho</td>
<td>5242</td>
</tr>
<tr>
<td>Y2 D257</td>
<td>Y7 D20</td>
<td><strong>Eeloo</strong></td>
<td><strong>3556</strong></td>
</tr>
<tr>
<td>Y2 D393</td>
<td>Y3 D86</td>
<td>Moho</td>
<td>5748</td>
</tr>
<tr>
<td>Y3 D83</td>
<td>Y3 D225</td>
<td>Moho</td>
<td>4481</td>
</tr>
<tr>
<td>Y3 D120</td>
<td>Y4 D283</td>
<td><strong>Dres</strong></td>
<td><strong>3072</strong></td>
</tr>
<tr>
<td>Y3 D270</td>
<td>Y7 D63</td>
<td>Eeloo</td>
<td>3644</td>
</tr>
<tr>
<td>Y3 D275</td>
<td>Y6 D216</td>
<td><strong>Jool</strong></td>
<td><strong>4902</strong></td>
</tr>
<tr>
<td>Y3 D305</td>
<td>Y4 D160</td>
<td><strong>Duna</strong></td>
<td><strong>1645</strong></td>
</tr>
<tr>
<td>Y3 D366</td>
<td>Y4 D63</td>
<td>Moho</td>
<td>5695</td>
</tr>
<tr>
<td>Y3 D401</td>
<td>Y4 D162</td>
<td><strong>Eve</strong></td>
<td><strong>2560</strong></td>
</tr>
<tr>
<td>Y4 D84</td>
<td>Y4 D207</td>
<td><strong>Moho</strong></td>
<td><strong>4179</strong></td>
</tr>
<tr>
<td>Y4 D226</td>
<td>Y6 D108</td>
<td>Dres</td>
<td>3348</td>
</tr>
<tr>
<td>Y4 D289</td>
<td>Y5 D40</td>
<td>Moho</td>
<td>5383</td>
</tr>
<tr>
<td>Y4 D294</td>
<td>Y7 D365</td>
<td>Eeloo</td>
<td>3943</td>
</tr>
<tr>
<td>Y4 D306</td>
<td>Y7 D159</td>
<td>Jool</td>
<td>4976</td>
</tr>
</tbody>
</table>Monitoring network bandwidth of individual hosts from the router2020-11-28T00:00:00+01:002020-11-28T00:00:00+01:00Niobostag:blog.dest-unreach.be,2020-11-28:/2020/11/28/monitoring_hosts_from_a_router/<p>I live in a part of the world where broadband internet subscriptions come with a monthly data cap.
The datacap is relatively large (750GB/month), but streaming 4k content will get you there!
So I wanted to keep taps on my data usage.
My router is the ideal place to …</p><p>I live in a part of the world where broadband internet subscriptions come with a monthly data cap.
The datacap is relatively large (750GB/month), but streaming 4k content will get you there!
So I wanted to keep taps on my data usage.
My router is the ideal place to monitor this: all traffic needs to pass through there anyway.
And since it runs <a href="https://openwrt.org/">OpenWRT</a>, I can easily add the component I need.</p>
<h1 id="interface-level-statistics"><a class="toclink" href="#interface-level-statistics">Interface-level statistics</a></h1>
<p>The first step is relatively easy.
The Linux kernel already keeps a counter for each network interface that counts the number of bytes it has sent & received.
These counters are exposed through the <code>ifconfig</code> command, but also through the <code>/proc/net/dev</code> "file".</p>
<p>I'm using <a href="https://prometheus.io/">Prometheus</a> as my monitoring tool, so I wrote a simple shell-script to expose these numbers as prometheus metrics.
I should note that I'm not very fluent in Prometheus metric naming, so I might have picked something "wrong".</p>
<div class="highlight"><pre><span></span><code><span class="nv">NAME</span><span class="o">=</span><span class="s2">"wan"</span>
<span class="nv">STATS</span><span class="o">=</span><span class="s2">"</span><span class="k">$(</span>grep pppoe /proc/net/dev <span class="k">)</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"network_bytes_total{interface=\"</span><span class="nv">$NAME</span><span class="s2">\",direction=\"rx\"} </span><span class="k">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$STATS</span><span class="s2">"</span> <span class="p">|</span> awk <span class="s1">'{print $2}'</span><span class="k">)</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"network_packets_total{interface=\"</span><span class="nv">$NAME</span><span class="s2">\",direction=\"rx\"} </span><span class="k">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$STATS</span><span class="s2">"</span> <span class="p">|</span> awk <span class="s1">'{print $3}'</span><span class="k">)</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"network_bytes_total{interface=\"</span><span class="nv">$NAME</span><span class="s2">\",direction=\"tx\"} </span><span class="k">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$STATS</span><span class="s2">"</span> <span class="p">|</span> awk <span class="s1">'{print $10}'</span><span class="k">)</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"network_packets_total{interface=\"</span><span class="nv">$NAME</span><span class="s2">\",direction=\"tx\"} </span><span class="k">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$STATS</span><span class="s2">"</span> <span class="p">|</span> awk <span class="s1">'{print $11}'</span><span class="k">)</span><span class="s2">"</span>
</code></pre></div>
<p>To expose these metrics to Prometheus, I converted the above snippet into a CGI-script.
This sounds more complicated than it is.
I just added these lines to the top, and linked in into <code>/www/cgi-bin/metrics</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nb">echo</span> <span class="s2">"Content-Type: text/plain; version=0.0.4"</span>
<span class="nb">echo</span> <span class="s2">""</span>
</code></pre></div>
<h1 id="host-level-statistics"><a class="toclink" href="#host-level-statistics">Host-level statistics</a></h1>
<p>Adding metrics for individual hosts turned out to be a lot more complicated than I anticipated.
In the good old days, one could simply use an <a href="https://en.wikipedia.org/wiki/Iptables">iptables</a>-table that matched all 254 IP-addresses of the subnet, and call it a day.
But we don't live in the good old days anymore, I live in a time where I have native IPv6 connectivity!</p>
<p>IPv6 complicated this idea a lot.
Not only is it unfeasable to create a table of all 18446744073709551616 IPv6 addresses in the subnet,
<a href="https://tools.ietf.org/html/rfc3041">IPv6 Privacy Extensions</a> means that there is no way to combine the different addresses for a particular host...
The only identifier that is usable in a <a href="https://en.wikipedia.org/wiki/Dual-stack#Dual-stack_IP_implementation">Dual-Stack</a> environment with <a href="https://tools.ietf.org/html/rfc3041">IPv6 Privacy Extensions</a> is the Ethernet <a href="https://en.wikipedia.org/wiki/MAC_address">MAC address</a> of the host.</p>
<p>Once I realised I should use the MAC addresses, the rest seemed simple:</p>
<div class="highlight"><pre><span></span><code>iptables -N accounting
iptables -A FORWARD -j accounting
iptables -A accounting -m mac --mac-source <span class="m">00</span>:00:5E:01:02:03 -j RETURN
iptables -A accounting -m mac --mac-destination <span class="m">00</span>:00:5E:01:02:03 -j RETURN
</code></pre></div>
<p>Only, that doesn't work...
There is no <code>--mac-destination</code> filter.
The reason is pretty simple: we are still in IP-land, so the kernel does not know yet what MAC-address it will sent the frame to.
You could get around this by adding a complete <a href="https://ebtables.netfilter.org/">ebtables</a> setup, but that seems like the "wrong" solution to track IP-level counters.</p>
<p>The solution is rather complicated.
And I couldn't have done it without help from <a href="https://superuser.com/questions/977997/mark-packets-with-iptables-by-destination-mac-address/1228588#1228588">this SuperUser answer</a>.
Use the connection tracking logic to carry the MAC-information over to the return packets:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># CONNMARK only works in the mangle table, so we'll do the accounting there</span>
iptables -t mangle -N accounting_out
iptables -t mangle -N accounting_in
iptables -t mangle -A FORWARD -j CONNMARK --restore-mark <span class="c1"># Restore the saved mark from the CONNTRACK table into the IP-level MARK</span>
iptables -t mangle -A FORWARD -i pppoe -j account_in <span class="c1"># do accounting based on the MARK</span>
iptables -t mangle -A FORWARD -o pppoe -j account_out <span class="c1"># do accounting based on MAC, and set MARK</span>
iptables -t mangle -A FORWARD -j CONNMARK --save-mark <span class="c1"># save IP-level MARK into CONNTRACK table</span>
<span class="c1"># Assign unique integer IDs to each host:</span>
iptables -t mangle -A account_out -m mac --mac-source <span class="m">00</span>:00:5E:01:02:03 -m comment --comment <span class="s2">"desktop"</span> -j MARK --set-mark <span class="m">1</span>
iptables -t mangle -A account_in -m mark --mark <span class="m">1</span> -m comment --comment <span class="s2">"desktop"</span> -j RETURN
iptables -t mangle -A account_out -m mac --mac-source <span class="m">00</span>:00:5E:04:05:06 -m comment --comment <span class="s2">"desktop"</span> -j MARK --set-mark <span class="m">2</span>
iptables -t mangle -A account_in -m mark --mark <span class="m">2</span> -m comment --comment <span class="s2">"desktop"</span> -j RETURN
<span class="c1"># repeat for ip6tables</span>
</code></pre></div>
<p>Some things that I figured out while trying to get this to work:</p>
<ul>
<li>
<p>There is a CONNMARK target that allows you to directly save the mark to the conntrack-table.
This would allow you to skip the <code>--save-mark</code> rule.
This works in <code>iptables</code>, but does not work in <code>ip6tables</code></p>
</li>
<li>
<p>restoring a CONNMARK mark to the IP-layer only works in the <code>mangle</code> table.</p>
</li>
</ul>
<p>Exposing these counters to prometheus isn't very difficult, but the resulting script is rather long and ugly:</p>
<div class="highlight"><pre><span></span><code>iptables -t mangle -vnL account_in --exact <span class="p">|</span> sed -n <span class="s1">'s%\s*\([0-9]\+\)\s\+\([0-9]\+\)\s\+.*/\* \(.*\) \*/.*%host_packets_total{ipv="4",direction="in",host="\3"} \1\nhost_bytes_total{ipv="4",direction="in",host="\3"} \2%p'</span>
iptables -t mangle -vnL account_out --exact <span class="p">|</span> sed -n <span class="s1">'s%\s*\([0-9]\+\)\s\+\([0-9]\+\)\s\+.*/\* \(.*\) \*/.*%host_packets_total{ipv="4",direction="out",host="\3"} \1\nhost_bytes_total{ipv="4",direction="out",host="\3"} \2%p'</span>
ip6tables -t mangle -vnL account_in --exact <span class="p">|</span> sed -n <span class="s1">'s%\s*\([0-9]\+\)\s\+\([0-9]\+\)\s\+.*/\* \(.*\) \*/.*%host_packets_total{ipv="6",direction="in",host="\3"} \1\nhost_bytes_total{ipv="6",direction="in",host="\3"} \2%p'</span>
ip6tables -t mangle -vnL account_out --exact <span class="p">|</span> sed -n <span class="s1">'s%\s*\([0-9]\+\)\s\+\([0-9]\+\)\s\+.*/\* \(.*\) \*/.*%host_packets_total{ipv="6",direction="out",host="\3"} \1\nhost_bytes_total{ipv="6",direction="out",host="\3"} \2%p'</span>
</code></pre></div>Cumulative graphs in Prometheus2020-08-16T00:00:00+02:002020-08-16T00:00:00+02:00Niobostag:blog.dest-unreach.be,2020-08-16:/2020/08/16/cumulative-graphs-prometheus/<p>I've been experimenting a bit with <a href="https://prometheus.io/">Prometheus</a> and <a href="https://grafana.com/">Grafana</a>.
It allows you to make beautiful graphs from the metric data
using relatively simple expressions.
But one of the graphs that I had in mind,
turned out to be more difficult.
These are my notes.</p>
<p>I have a metric that counts …</p><p>I've been experimenting a bit with <a href="https://prometheus.io/">Prometheus</a> and <a href="https://grafana.com/">Grafana</a>.
It allows you to make beautiful graphs from the metric data
using relatively simple expressions.
But one of the graphs that I had in mind,
turned out to be more difficult.
These are my notes.</p>
<p>I have a metric that counts the electrical energy my solar panels have produced: <code>energy_out_watthour_total</code>.
(There is some debate whether <code>watthour</code> is "right". Some people argue that it should be Joules, but that is not the point of this post.)
Since this is a counter (<code>_total</code> suffix), Prometheus does all the magic to compensate if this counter would reset to zero.
This also means that the absolute value of the counter is irrelevant, only the changes are meaningful.</p>
<p>If you are only interested in the changes, <code>rate(metric)</code> will give you exactly that:
it will calculate the rate of change (for this case, this will yield the average power in Watt between every two datapoints).
But I wanted a cumulative view: how much energy have I produced over this day, this week, ...
For a given day, this give a <a href="https://en.wikipedia.org/wiki/Sigmoid_function">sigmoid</a>-like curve:
(I have removed the actual values for privacy)</p>
<p><img alt="Cumulative energy" src="https://blog.dest-unreach.be/2020/08/16/cumulative-graphs-prometheus/prometheus/cum-1d.png"></p>
<p>The corresponding power-graph looks like this:</p>
<p><img alt="Power" src="https://blog.dest-unreach.be/2020/08/16/cumulative-graphs-prometheus/prometheus/power-1d.png"></p>
<p>Both graphs show the same information, and for a single day, they don't differ much.
But when comparing e.g. today to yesterday, I find a cumulative view much easier to read:
are we ahead/behind "schedule" compared to yesterday? Especially when the power is very spikey, I find cumulative graphs much easier to interpret.
Take a look for yourself:</p>
<p><img alt="Cumulative energy vs yesterday" src="https://blog.dest-unreach.be/2020/08/16/cumulative-graphs-prometheus/prometheus/cum-2d.png"></p>
<p><img alt="Power vs yesterday" src="https://blog.dest-unreach.be/2020/08/16/cumulative-graphs-prometheus/prometheus/power-2d.png"></p>
<h1 id="promql"><a class="toclink" href="#promql">PromQL</a></h1>
<p>Next up: getting this idea of a cumulative graph translated into a Grafana dashboard.
The basic idea is not too hard. I want to plot <code>energy_out_watthour_total - ${energy_out_watthour_total_at_the_beginning_of_this_graph}</code>.
Turns out that last part isn't so easy:
There is no easy way to have Grafana query Prometheus for the value at a particular point in time.
Or, at least I didn't find that way.</p>
<p>So I reached out to the #prometheus IRC channel on FreeNode. With the help of <code>SuperQ</code> and <code>roidelapluie</code>, we figured out some tools to help us:
Grafana exposes a few <a href="https://grafana.com/docs/grafana/latest/variables/variable-types/global-variables/">variables</a> that you can use inside your query.
Especially the <code>$__from</code> variable looks interesting for this case.</p>
<p>Next, we figured out a few ways that didn't work:</p>
<ul>
<li>
<p><code>metric_total - metric_total offset (time() - $__from)</code> doesn't work, since offset requires a constant offset, not something dynamic</p>
</li>
<li>
<p><code>scalar(metric_total and on() time() == vector(1597579200))</code> doesn't work for multiple reasons:
you'd need to get the timestamp exactly right, or use a range that covers exactly 1 datapoint.
And, since prometheus only evaluates the metrics within the time window that is it about to graph,
this may yield no points at all if hte timestamp is outside that range.</p>
</li>
</ul>
<p>What does work is this construction:</p>
<div class="highlight"><pre><span></span><code>metric_total
- max_over_time(
(
metric_total and on() vector(time()) <span class="err"><</span> <span class="nv">$__from</span>/1000
)[<span class="cp">${</span><span class="n">__range_s</span><span class="cp">}</span>s:<span class="nv">$__interval</span>]
)
</code></pre></div>
<p>The inner <code>and</code> construction gives the metric up until the start of the graph.
The range vector selection allows us to gather up data from before the start of the graph.
The actual values are less important, as long as it scoops up at least one datapoint.
The <code>max_over_time()</code> takes the highest value it sees. Since this is a counter,
this effectively means taking the most recent value.</p>
<p>There is an edge case when a reset occurs within this window. We can solve that by using</p>
<div class="highlight"><pre><span></span><code>metric_total
- avg_over_time(
(
metric_total and on() vector(time()) >= <span class="nv">$__from</span>/1000 <span class="err"><</span> <span class="nv">$__from</span>/1000+<span class="nv">$__interval_ms</span>/1000
)[<span class="cp">${</span><span class="n">__range_s</span><span class="cp">}</span>s:<span class="nv">$__interval</span>]
)
</code></pre></div>
<p>This narrows the time-frame that is considered, and uses <code>avg_over_time()</code> to average out the remaining points.
Hopefully, there will only one point.
The downside of this expression is that it doesn't work for my solar production:
during the night, the invertor goes into a sleep-state, and I can't get any metrics from it.
This means that, depending on the actual time-window of the graph, the start-time may have no data, failing the query.</p>
<h1 id="grafana"><a class="toclink" href="#grafana">Grafana</a></h1>
<p>I've struggled a bit on the finishing touches in Grafana, so I'll note them down here as well.
Since solar panels only work during daylight, it makes no sense to waste space on the night hours.
I found that plotting from <code>now/d+5h</code> to <code>now/d-90m</code> works best for my location.
This plots the current day from 05:00 until 22:30 (local/browser time).</p>Half-duplex UART (RS485) on ESP322020-04-26T00:00:00+02:002020-04-26T00:00:00+02:00Niobostag:blog.dest-unreach.be,2020-04-26:/2020/04/26/uart-half-duplex-esp32/<p>I'm a happy owner of a few <a href="https://en.wikipedia.org/wiki/ESP32">ESP32 microcontrollers</a>. These are similar to the Arduino, but have WiFi & Bluetooth built in, which makes them very versatile. These chips come with <a href="https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/peripherals/index.html">a lot of peripherals</a>, including three <a href="https://en.wikipedia.org/wiki/UART">UART</a>s. These allow offloading the actual bit-banging of the pins to dedicated hardware …</p><p>I'm a happy owner of a few <a href="https://en.wikipedia.org/wiki/ESP32">ESP32 microcontrollers</a>. These are similar to the Arduino, but have WiFi & Bluetooth built in, which makes them very versatile. These chips come with <a href="https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/peripherals/index.html">a lot of peripherals</a>, including three <a href="https://en.wikipedia.org/wiki/UART">UART</a>s. These allow offloading the actual bit-banging of the pins to dedicated hardware, freeing up the main processor for more useful work.</p>
<p>Normally, UARTs work in full-duplex mode: they can send and receive simultaneously over separate wires. (Obviously, you can operate it in simplex mode as well, by only sending or receiving.) Half-duplex means that the device can both send and receive, but not simultaneously. This post is about getting this half-duplex mode to work, which was more tricky than it should have been.</p>
<p>Half-duplex is used by <a href="https://en.wikipedia.org/wiki/RS-485">RS-485</a>, but also <a href="https://en.wikipedia.org/wiki/Open_Collector">open-drain</a> (or open-collector in <a href="https://en.wikipedia.org/wiki/Bipolar_junction_transistor">bipolar</a> terminology) buses are inherently half-duplex. The ESP's documentation refers to its half-duplex provisions by the RS-485 name.</p>
<div class="toc">
<ul>
<li><a href="#challenges">Challenges</a></li>
<li><a href="#test-setup">Test setup</a><ul>
<li><a href="#full-duplex-mode">Full duplex mode</a></li>
<li><a href="#rs485-modes">RS485 modes</a><ul>
<li><a href="#uart_mode_rs485_half_duplex">UART_MODE_RS485_HALF_DUPLEX</a></li>
<li><a href="#uart_mode_rs485_collision_detect">UART_MODE_RS485_COLLISION_DETECT</a></li>
<li><a href="#uart_mode_rs485_app_ctrl">UART_MODE_RS485_APP_CTRL</a></li>
</ul>
</li>
<li><a href="#custom-rs485-modes">Custom RS485 modes</a></li>
</ul>
</li>
<li><a href="#conclusion">Conclusion</a></li>
</ul>
</div>
<h2 id="challenges"><a class="toclink" href="#challenges">Challenges</a></h2>
<p>Half-duplex works just like full duplex when only one device is transmitting at a time. So for simple request-response protocols with only a single pair of endpoints, this usuall doesn't matter. It becomes more challenging when multiple devices may initiate a conversation:</p>
<p>Before starting a transmission, the device should first check if the bus is free. I.e. if there is no-one else transmitting at that time. If the bus is busy, the device should hold off its transmission until it is free.</p>
<p>Even during a transmission, the device should monitor the bus to see if it can see its own transmission correctly. Ideally, it should be able to detect a "collision", and retransmit in that case.</p>
<h2 id="test-setup"><a class="toclink" href="#test-setup">Test setup</a></h2>
<p>To perform the test, I tested a single ESP32 against itself: UART2 is the device under test, UART1 is used as a reference, and kept in full-duplex mode. The half-duplex magic is done by abusing the second core the ESP32 offers: I looped back the Tx and Rx pins to GPIO pins and logically ANDed them:</p>
<div class="highlight"><pre><span></span><code><span class="n">gpio_set_direction</span><span class="p">(</span><span class="n">GPIO_NUM_27</span><span class="p">,</span><span class="w"> </span><span class="n">GPIO_MODE_INPUT</span><span class="p">);</span><span class="w"> </span><span class="c1">// connected to UART1 Tx</span>
<span class="n">gpio_set_direction</span><span class="p">(</span><span class="n">GPIO_NUM_26</span><span class="p">,</span><span class="w"> </span><span class="n">GPIO_MODE_INPUT</span><span class="p">);</span><span class="w"> </span><span class="c1">// connected to UART2 Tx</span>
<span class="n">gpio_set_direction</span><span class="p">(</span><span class="n">GPIO_NUM_25</span><span class="p">,</span><span class="w"> </span><span class="n">GPIO_MODE_OUTPUT</span><span class="p">);</span><span class="w"> </span><span class="c1">// connected to UART1 & UART2 Rx</span>
<span class="k">for</span><span class="p">(;;)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">level1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">gpio_get_level</span><span class="p">(</span><span class="n">GPIO_NUM_27</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">level2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">gpio_get_level</span><span class="p">(</span><span class="n">GPIO_NUM_26</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="c1">// idle level is 1</span>
<span class="w"> </span><span class="c1">// if either Tx asserts 0, output 0</span>
<span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">level</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">level1</span><span class="w"> </span><span class="o">&</span><span class="w"> </span><span class="n">level2</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">gpio_set_level</span><span class="p">(</span><span class="n">GPIO_NUM_25</span><span class="p">,</span><span class="w"> </span><span class="n">level</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>The test consists of 4 parts (<a href="https://blog.dest-unreach.be/2020/04/26/uart-half-duplex-esp32/rs485/half_duplex_test.c">source code</a>):</p>
<ol>
<li>Receive only</li>
<li>Transmit only</li>
<li>Transmit while bus is busy</li>
<li>Transmit with collision</li>
</ol>
<p>The first two parts are trivial, but are included for completeness. The third test should reveal if the device waits for the bus to become available before starting a transmission. The forth test will always result in corruption, but the sender should report this.</p>
<h3 id="full-duplex-mode"><a class="toclink" href="#full-duplex-mode">Full duplex mode</a></h3>
<p>As a reference test, I set up UART2 similar to UART1:</p>
<div class="highlight"><pre><span></span><code><span class="n">uart_config_t</span><span class="w"> </span><span class="n">uart_config</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">baud_rate</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">9600</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">data_bits</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">UART_DATA_8_BITS</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">parity</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">UART_PARITY_DISABLE</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">stop_bits</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">UART_STOP_BITS_1</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">flow_ctrl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">UART_HW_FLOWCTRL_DISABLE</span><span class="p">,</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
<span class="n">ESP_ERROR_CHECK</span><span class="p">(</span><span class="n">uart_param_config</span><span class="p">(</span><span class="n">UART_NUM_2</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">uart_config</span><span class="p">));</span><span class="w"></span>
<span class="n">ESP_ERROR_CHECK</span><span class="p">(</span><span class="n">uart_set_pin</span><span class="p">(</span><span class="n">UART_NUM_2</span><span class="p">,</span><span class="w"> </span><span class="n">GPIO_NUM_16</span><span class="p">,</span><span class="w"> </span><span class="n">GPIO_NUM_17</span><span class="p">,</span><span class="w"> </span><span class="n">UART_PIN_NO_CHANGE</span><span class="p">,</span><span class="w"> </span><span class="n">UART_PIN_NO_CHANGE</span><span class="p">));</span><span class="w"></span>
<span class="n">ESP_ERROR_CHECK</span><span class="p">(</span><span class="n">uart_driver_install</span><span class="p">(</span><span class="n">UART_NUM_2</span><span class="p">,</span><span class="w"> </span><span class="mi">1024</span><span class="p">,</span><span class="w"> </span><span class="mi">1024</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">));</span><span class="w"></span>
</code></pre></div>
<p>As expected, part 1 & 2 returned the expected data; During test 3, UART2 failed to wait until the bus was free (as expected), which garbled the resulting output: (0) is UART1's output (reference), (1) is UART2's Tx (DUT), (2) is the resulting Rx signal given to both UARTs.</p>
<p><img alt="transmit with busy bus, full duplex" src="https://blog.dest-unreach.be/2020/04/26/uart-half-duplex-esp32/rs485/fdx-busy-tx.png"></p>
<p>Instead of receiving the "CCCC" (sent by UART1) or the "dd" (sent by UART2), garbage was received:</p>
<div class="highlight"><pre><span></span><code>03 80 a8 e8 |....|
</code></pre></div>
<p>Test 4 failed in the expected way as well. UART2 was transmitting "eeee", while UART1 sent "F" simultaniously, colliding in the first byte.</p>
<p><img alt="collision while transmitting, full duplex" src="https://blog.dest-unreach.be/2020/04/26/uart-half-duplex-esp32/rs485/fdx-col.png"></p>
<div class="highlight"><pre><span></span><code>04 65 65 65 |.eee|
</code></pre></div>
<h3 id="rs485-modes"><a class="toclink" href="#rs485-modes">RS485 modes</a></h3>
<p>The ESP's documentation is very sparse on its half-duplex capabilities. The <a href="https://www.espressif.com/sites/default/files/documentation/esp32_technical_reference_manual_en.pdf">technical reference manual</a> mentions the "RS485 mode configuration" register, but no further explanation is given. The <a href="https://docs.espressif.com/projects/esp-idf/en/latest/esp32/">ESP-IDF</a> programming guide goes in to a bit more detail, describing the <a href="https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/peripherals/uart.html#interface-connection-options">different modes</a> to connect to a 485 bus. Its API provides three modes that look interesting: UART_MODE_RS485_HALF_DUPLEX, UART_MODE_RS485_COLLISION_DETECT and UART_MODE_RS485_APP_CTRL. </p>
<h4 id="uart_mode_rs485_half_duplex"><a class="toclink" href="#uart_mode_rs485_half_duplex">UART_MODE_RS485_HALF_DUPLEX</a></h4>
<p>In RS485_HALF_DUPLEX mode, contrary to my expectations, nothing happened: test 1 & 2 still passed, test 3 & 4 still failed in a very similar way.</p>
<p>Test 3 failed to wait for the bus to free up, and returned garbage data:</p>
<p><img alt="transmit with busy bus, RS485_HALF_DUPLEX" src="https://blog.dest-unreach.be/2020/04/26/uart-half-duplex-esp32/rs485/rs485hdx-busy-tx.png"></p>
<div class="highlight"><pre><span></span><code>03 80 a8 e8 |....|
</code></pre></div>
<p>Test 4 got corrupted as well.</p>
<p><img alt="collision while transmitting, RS485_HALF_DUPLEX" src="https://blog.dest-unreach.be/2020/04/26/uart-half-duplex-esp32/rs485/rs485hdx-col.png"></p>
<div class="highlight"><pre><span></span><code>04 65 65 65 |.eee|
</code></pre></div>
<p>In this mode, you can use the <code>uart_get_collision_flag()</code> function to see if there was a collision. It returned <code>false</code> for test 1 (as expected), but <code>true</code> for tests 2, 3 and 4 (only 4 was expected). </p>
<h4 id="uart_mode_rs485_collision_detect"><a class="toclink" href="#uart_mode_rs485_collision_detect">UART_MODE_RS485_COLLISION_DETECT</a></h4>
<p>This setting paniced. I haven't figured out why (yet).</p>
<h4 id="uart_mode_rs485_app_ctrl"><a class="toclink" href="#uart_mode_rs485_app_ctrl">UART_MODE_RS485_APP_CTRL</a></h4>
<p>RS485_APP_CTRL mode was indistinguishable from HALF_DUPLEX in my tests:</p>
<p>Test 3 failed to wait for the bus to free up, and returned garbage data:</p>
<p><img alt="transmit with busy bus, RS485_APP_CTRL" src="https://blog.dest-unreach.be/2020/04/26/uart-half-duplex-esp32/rs485/rs485appctl-busy-tx.png"></p>
<div class="highlight"><pre><span></span><code>03 80 a8 e8 |....|
</code></pre></div>
<p>Test 4 got corrupted as well.</p>
<p><img alt="collision while transmitting, RS485_APP_CTL" src="https://blog.dest-unreach.be/2020/04/26/uart-half-duplex-esp32/rs485/rs485appctl-col.png"></p>
<div class="highlight"><pre><span></span><code>04 65 65 65 |.eee|
</code></pre></div>
<p><code>uart_get_collision_flag()</code> returned <code>false</code>, <code>true</code>, <code>true</code>, <code>true</code>.</p>
<h3 id="custom-rs485-modes"><a class="toclink" href="#custom-rs485-modes">Custom RS485 modes</a></h3>
<p>From the tests above, it is clear that neither of the provided modes perform the desired functionality. Time to dig in deeper.</p>
<p>The ESP-IDF SDK is nothing more than a set of provided library functions that abstract away the register manipulations. <a href="https://github.com/espressif/esp-idf/blob/143d26aa49df524e10fb8e41a71d12e731b9b71d/components/driver/uart.c#L1480">uart_set_mode()</a> for example manipulates the <code>UART_RS485_CONF_REG</code> register (accessed as <code>UART[uart_num]->rs485_conf</code> in ESP-IDF). The code shows that all three modes activate RS485 mode (<code>UART[uart_num]->rs485_conf.en = 1</code>), but they differ in the settings of the other flags.</p>
<p>What surprised me, is that all three modes set
<code>UART_RS485RXBY_TX_EN</code>/<code>rx_busy_tx_en</code> flag: <code>enable the RS-485 transmitter to send data, when the RS-485 receiver line is busy</code>. As soon as that was turned off, things started looking better: UART2 waited for the bus to free up before transmitting its "dd" message, which resulted in the correct sequence being received. The collision flag was still wrong (it indicated collisions in test 2 & 3 as well).</p>
<p><img alt="transmit with busy bus, rx_busy_tx_dis" src="https://blog.dest-unreach.be/2020/04/26/uart-half-duplex-esp32/rs485/rxbusy-busy-tx.png"></p>
<div class="highlight"><pre><span></span><code>43 43 43 43 64 64 |CCCCdd|
</code></pre></div>
<p>Another flag that is available is the <code>UART_RS485TX_RX_EN</code>/<code>tx_rx_en</code>, which will <code>enable the transmitter’s output signal loop back to the receiver’s input signal</code>. The <code>APP_CTRL</code> mode (which I used as a starting point) leaves this flag off. I expected this to suppress receiving my own transmissions, but instead, it returned a <code>0x00</code>-byte for every byte transmitted. So let's try setting this to <code>1</code>.</p>
<p>This seems to be the magic combination: Tx waits until the bus is free, and the collision flag yields the expected result (only <code>true</code> for test 4). It only introduces a small problem:</p>
<div class="highlight"><pre><span></span><code>E (1398) uart: uart_get_collision_flag(1568): wrong mode
</code></pre></div>
<p>The <code>uart_get_collision_flag()</code> reads out the collision state as reported by the SDK. And the SDK doesn't <a href="https://github.com/espressif/esp-idf/blob/143d26aa49df524e10fb8e41a71d12e731b9b71d/components/driver/uart.c#L997-L1007">track</a> the collisions when in <code>APP_CTRL</code>. But nothing is stopping us to do that ourselves: The hardware sets the <code>UART_RS485_CLASH_INT_RAW</code> bit in the <code>UART_INT_RAW_REG</code> register. This can trigger an actual interrupt if enabled, but we are only interested in finding out <em>if</em> a clash happened, not exactly <em>when</em> it happened. So we can simply read out this register, and clear it before the next transmission:</p>
<div class="highlight"><pre><span></span><code><span class="n">UART2</span><span class="p">.</span><span class="n">int_clr</span><span class="p">.</span><span class="n">rs485_clash</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"> </span><span class="c1">// clear bit</span>
<span class="n">uart_write_bytes</span><span class="p">(</span><span class="n">UART_NUM_2</span><span class="p">,</span><span class="w"> </span><span class="s">"whatever"</span><span class="p">,</span><span class="w"> </span><span class="mi">8</span><span class="p">);</span><span class="w"></span>
<span class="kt">bool</span><span class="w"> </span><span class="n">collision_happened</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">UART2</span><span class="p">.</span><span class="n">int_raw</span><span class="p">.</span><span class="n">rs485_clash</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<h2 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h2>
<p>For my purpose, I wanted the transmision to be kept back if the bus is busy. I also wanted to have accurate collision reporting. The following code sets this up for me:</p>
<div class="highlight"><pre><span></span><code><span class="n">uart_config_t</span><span class="w"> </span><span class="n">uart_config</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">baud_rate</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">9600</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">data_bits</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">UART_DATA_8_BITS</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">parity</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">UART_PARITY_DISABLE</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">stop_bits</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">UART_STOP_BITS_1</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">flow_ctrl</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">UART_HW_FLOWCTRL_DISABLE</span><span class="p">,</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
<span class="n">ESP_ERROR_CHECK</span><span class="p">(</span><span class="n">uart_param_config</span><span class="p">(</span><span class="n">UART_NUM_2</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">uart_config</span><span class="p">));</span><span class="w"></span>
<span class="n">ESP_ERROR_CHECK</span><span class="p">(</span><span class="n">uart_set_pin</span><span class="p">(</span><span class="n">UART_NUM_2</span><span class="p">,</span><span class="w"> </span><span class="n">GPIO_NUM_16</span><span class="p">,</span><span class="w"> </span><span class="n">GPIO_NUM_17</span><span class="p">,</span><span class="w"> </span><span class="n">UART_PIN_NO_CHANGE</span><span class="p">,</span><span class="w"> </span><span class="n">UART_PIN_NO_CHANGE</span><span class="p">));</span><span class="w"></span>
<span class="n">ESP_ERROR_CHECK</span><span class="p">(</span><span class="n">uart_driver_install</span><span class="p">(</span><span class="n">UART_NUM_2</span><span class="p">,</span><span class="w"> </span><span class="mi">1024</span><span class="p">,</span><span class="w"> </span><span class="mi">1024</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="nb">NULL</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">));</span><span class="w"></span>
<span class="n">uart_set_mode</span><span class="p">(</span><span class="n">UART_NUM_2</span><span class="p">,</span><span class="w"> </span><span class="n">UART_MODE_RS485_APP_CTRL</span><span class="p">);</span><span class="w"></span>
<span class="n">UART2</span><span class="p">.</span><span class="n">rs485_conf</span><span class="p">.</span><span class="n">rx_busy_tx_en</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="c1">// don't send while receiving => collision avoidance</span>
<span class="n">UART2</span><span class="p">.</span><span class="n">rs485_conf</span><span class="p">.</span><span class="n">tx_rx_en</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"> </span><span class="c1">// loopback (1), so collision detection works</span>
<span class="kt">bool</span><span class="w"> </span><span class="nf">did_collision_happen_since_last_check</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="n">ret</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">UART2</span><span class="p">.</span><span class="n">int_raw</span><span class="p">.</span><span class="n">rs485_clash</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">UART2</span><span class="p">.</span><span class="n">int_clr</span><span class="p">.</span><span class="n">rs485_clash</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"> </span><span class="c1">// clear bit</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">ret</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>Raspberry pi 4 as broadband router2020-02-21T00:00:00+01:002020-02-21T00:00:00+01:00Niobostag:blog.dest-unreach.be,2020-02-21:/2020/02/21/raspberry-pi-4-router/<p><a href="https://2007.blog.dest-unreach.be/2014/03/05/raspberry-pi-as-broadband-router/">A few years ago</a>, I already considered using a Raspberry Pi as a broadband router. Back then, its first generation wasn't fast enough. This changed with the 4th generation of the Raspberry pi.</p>
<p>The test setup was similar to the setup I use previously:</p>
<ul>
<li>Two beefy laptops on the ends …</li></ul><p><a href="https://2007.blog.dest-unreach.be/2014/03/05/raspberry-pi-as-broadband-router/">A few years ago</a>, I already considered using a Raspberry Pi as a broadband router. Back then, its first generation wasn't fast enough. This changed with the 4th generation of the Raspberry pi.</p>
<p>The test setup was similar to the setup I use previously:</p>
<ul>
<li>Two beefy laptops on the ends</li>
<li>A VLAN-capable switch to connect everything</li>
<li>The Raspberry pi "<a href="https://en.wikipedia.org/wiki/One-armed_router">on a stick</a>" to route traffic from one VLAN to the other</li>
</ul>
<p>I was happy with the results. Ping latency was 0.834 ±0.075 ms.</p>
<div class="highlight"><pre><span></span><code>$ iperf -c 192.168.23.231 -p 5001 -t 60 -i 5
------------------------------------------------------------
Client connecting to 192.168.23.231, TCP port 5001
TCP window size: 289 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.10.20 port 60887 connected with 192.168.23.231 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 5.0 sec 510 MBytes 855 Mbits/sec
[ 4] 5.0-10.0 sec 514 MBytes 862 Mbits/sec
[ 4] 10.0-15.0 sec 513 MBytes 860 Mbits/sec
[ 4] 15.0-20.0 sec 512 MBytes 860 Mbits/sec
[ 4] 20.0-25.0 sec 513 MBytes 860 Mbits/sec
[ 4] 25.0-30.0 sec 514 MBytes 862 Mbits/sec
[ 4] 30.0-35.0 sec 516 MBytes 866 Mbits/sec
[ 4] 35.0-40.0 sec 515 MBytes 864 Mbits/sec
[ 4] 40.0-45.0 sec 514 MBytes 863 Mbits/sec
[ 4] 45.0-50.0 sec 506 MBytes 849 Mbits/sec
[ 4] 50.0-55.0 sec 498 MBytes 835 Mbits/sec
[ 4] 55.0-60.0 sec 506 MBytes 849 Mbits/sec
[ 4] 0.0-60.0 sec 5.99 GBytes 857 Mbits/sec
</code></pre></div>
<p>The bidirectional flows performed worse, as expected. But I had expected a bit better results:</p>
<div class="highlight"><pre><span></span><code>$ iperf -c 192.168.23.231 -d -p 5001 -t 60 -i 5
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: -1.00 Byte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.23.231, TCP port 5001
TCP window size: 273 KByte (default)
------------------------------------------------------------
[ 5] local 192.168.10.20 port 60891 connected with 192.168.23.231 port 5001
[ 6] local 192.168.10.20 port 5001 connected with 192.168.23.231 port 36352
[ ID] Interval Transfer Bandwidth
[ 5] 0.0- 5.0 sec 82.2 MBytes 138 Mbits/sec
[ 6] 0.0- 5.0 sec 187 MBytes 313 Mbits/sec
[ 5] 5.0-10.0 sec 71.4 MBytes 120 Mbits/sec
[ 6] 5.0-10.0 sec 163 MBytes 273 Mbits/sec
[ 5] 10.0-15.0 sec 75.8 MBytes 127 Mbits/sec
[ 6] 10.0-15.0 sec 177 MBytes 296 Mbits/sec
[ 6] 15.0-20.0 sec 177 MBytes 297 Mbits/sec
[ 5] 15.0-20.0 sec 79.6 MBytes 134 Mbits/sec
[ 5] 20.0-25.0 sec 66.0 MBytes 111 Mbits/sec
[ 6] 20.0-25.0 sec 187 MBytes 313 Mbits/sec
[ 5] 25.0-30.0 sec 70.6 MBytes 118 Mbits/sec
[ 6] 25.0-30.0 sec 200 MBytes 335 Mbits/sec
[ 6] 30.0-35.0 sec 174 MBytes 291 Mbits/sec
[ 5] 30.0-35.0 sec 75.2 MBytes 126 Mbits/sec
[ 5] 35.0-40.0 sec 74.0 MBytes 124 Mbits/sec
[ 6] 35.0-40.0 sec 185 MBytes 310 Mbits/sec
[ 5] 40.0-45.0 sec 65.2 MBytes 109 Mbits/sec
[ 6] 40.0-45.0 sec 171 MBytes 287 Mbits/sec
[ 5] 45.0-50.0 sec 67.0 MBytes 112 Mbits/sec
[ 6] 45.0-50.0 sec 173 MBytes 291 Mbits/sec
[ 5] 50.0-55.0 sec 65.0 MBytes 109 Mbits/sec
[ 6] 50.0-55.0 sec 163 MBytes 273 Mbits/sec
[ 5] 55.0-60.0 sec 69.2 MBytes 116 Mbits/sec
[ 5] 0.0-60.0 sec 861 MBytes 120 Mbits/sec
[ 6] 55.0-60.0 sec 166 MBytes 279 Mbits/sec
[ 6] 0.0-60.0 sec 2.07 GBytes 297 Mbits/sec
[SUM] 0.0-60.0 sec 2.25 GBytes 323 Mbits/sec
</code></pre></div>Reading CPU temperature and Fan speeds from the command line in macOs 10.14 Mojave2019-09-27T00:00:00+02:002019-09-27T00:00:00+02:00Niobostag:blog.dest-unreach.be,2019-09-27:/2019/09/27/macos_temp_fan_cli/<p>I was looking to read out the current temperature and fan speed of my MacBook Pro running macOS 10.14 Mojave. I prefered build-in tools over shady apps. After some searching, I found the <code>powermetrics</code> tool (which needs <code>sudo</code> privileges). By default, it outputs lots of details every 5 seconds …</p><p>I was looking to read out the current temperature and fan speed of my MacBook Pro running macOS 10.14 Mojave. I prefered build-in tools over shady apps. After some searching, I found the <code>powermetrics</code> tool (which needs <code>sudo</code> privileges). By default, it outputs lots of details every 5 seconds:</p>
<div class="highlight"><pre><span></span><code>Machine model: MacBookPro15,1
SMC version: Unknown
EFI version: 220.99.0
OS version: 18G103
Boot arguments:
Boot time: Fri Sep 27 13:23:38 2019
*** Sampled system activity (Fri Sep 27 14:05:02 2019 +0200) (5015.24ms elapsed) ***
*** Running tasks ***
Name ID CPU ms/s User% Deadlines (<2 ms, 2-5 ms) Wakeups (Intr, Pkg idle)
kernel_task 0 28.82 0.00 0.00 0.00 673.63 212.91
WindowServer 183 67.37 61.86 29.70 10.57 46.45 15.35
CalendarAgent 321 32.20 82.65 0.00 0.00 0.20 0.20
[...]
ALL_TASKS -2 766.12 80.97 101.09 14.16 1125.97 384.23
**** Battery and backlight usage ****
Backlight level: 791 (range 0-1024)
Keyboard Backlight level: 0 (off 0 on range 32-512)
**** Network activity ****
out: 30.71 packets/s, 11892.96 bytes/s
in: 31.30 packets/s, 18962.61 bytes/s
**** Disk activity ****
read: 1.00 ops/s 4.08 KBytes/s
write: 7.58 ops/s 78.40 KBytes/s
**** Interrupt distribution ****
CPU 0:
Vector 0x46(SMC): 12.96 interrupts/sec
Vector 0x72(IGPU): 17.95 interrupts/sec
Vector 0x74(GFX0): 418.52 interrupts/sec
Vector 0x75(HDAU): 3.79 interrupts/sec
Vector 0x76(XHC1): 143.36 interrupts/sec
Vector 0x79(ANS2): 6.58 interrupts/sec
Vector 0x86(ARPT): 7.58 interrupts/sec
Vector 0x8c(IOBC): 0.80 interrupts/sec
Vector 0xdd(TMR): 195.40 interrupts/sec
Vector 0xde(IPI): 112.86 interrupts/sec
Vector 0xdf(PMI): 0.20 interrupts/sec
CPU 1:
Vector 0xdd(TMR): 1.20 interrupts/sec
Vector 0xde(IPI): 35.09 interrupts/sec
[...]
CPU 15:
Vector 0xdd(TMR): 0.60 interrupts/sec
Vector 0xde(IPI): 4.39 interrupts/sec
**** Processor usage ****
Intel energy model derived package power (CPUs+GT+SA): 9.62W
LLC flushed residency: 44.5%
System Average frequency as fraction of nominal: 118.43% (2723.96 Mhz)
Package 0 C-state residency: 47.56% (C2: 47.56% C3: 0.00% C6: 0.00% C7: 0.00% C8: 0.00% C9: 0.00% C10: 0.00% )
Performance Limited Due to:
CPU LIMIT TURBO_ATTENUATION
CPU LIMIT SPARE_IA_14
CPU/GPU Overlap: 1.38%
Cores Active: 49.10%
GPU Active: 1.51%
Avg Num of Cores Active: 0.83
Core 0 C-state residency: 74.42% (C3: 3.57% C6: 0.00% C7: 70.85% )
CPU 0 duty cycles/s: active/idle [< 16 us: 618.91/176.66] [< 32 us: 148.35/20.34] [< 64 us: 119.44/226.11] [< 128 us: 138.18/270.18] [< 256 us: 155.53/166.69] [< 512 us: 67.20/161.31] [< 1024 us: 40.08/87.73] [< 2048 us: 21.33/90.32] [< 4096 us: 5.78/65.40] [< 8192 us: 1.00/51.24] [< 16384 us: 1.00/1.20] [< 32768 us: 0.20/0.00]
CPU Average frequency as fraction of nominal: 103.26% (2375.02 Mhz)
[...]
CPU 15 duty cycles/s: active/idle [< 16 us: 57.82/4.98] [< 32 us: 0.40/4.39] [< 64 us: 0.60/5.58] [< 128 us: 0.20/5.58] [< 256 us: 0.60/5.98] [< 512 us: 0.20/6.18] [< 1024 us: 0.20/5.78] [< 2048 us: 0.20/1.99] [< 4096 us: 0.40/2.19] [< 8192 us: 0.00/4.59] [< 16384 us: 0.00/3.79] [< 32768 us: 0.00/2.79]
CPU Average frequency as fraction of nominal: 113.15% (2602.45 Mhz)
**** GPU usage ****
GPU 0 name IntelIG
GPU 0 C-state residency: 98.66% (0.08%, 98.58%)
GPU 0 P-state residency: 1200MHz: 0.00%, 1150MHz: 0.00%, 1100MHz: 0.00%, 1050MHz: 0.00%, 1000MHz: 0.00%, 950MHz: 0.00%, 900MHz: 0.00%, 850MHz: 0.00%, 800MHz: 0.00%, 750MHz: 0.00%, 700MHz: 0.00%, 650MHz: 0.00%, 600MHz: 0.00%, 550MHz: 0.00%, 500MHz: 0.00%, 450MHz: 0.00%, 400MHz: 0.00%, 350MHz: 0.62%
GPU 0 average active frequency as fraction of nominal (350.00Mhz): 100.00% (350.00Mhz)
GPU 0 HW average active frequency : 0.00
GPU 0 GPU Busy : 1.34%
GPU 0 DC6 Residency : 0.00%
GPU 0 [PSR] GPU + TCON are Off : 0.00%
GPU 0 [PSR] Only GPU is On : 100.00%
GPU 0 [PSR] Only TCON is On : 0.00%
GPU 0 [PSR] GPU + TCON are On : 0.00%
GPU 0 [PSR] StateMachine Bypass : 100.00%
GPU 0 [PSR] StateMachine FIFO : 0.00%
GPU 0 [PSR] StateMachine Others : 0.00%
GPU 0 DPB Strong On : 0.00%
GPU 0 DPB Weak On : 0.00%
GPU 0 PPFM on : 0.00%
GPU 0 Throttle High Priority(%): 0
GPU 0 Throttle NormalHi Priority(%): 0
GPU 0 Throttle Normal Priority(%): 0
GPU 0 Throttle Low Priority(%): 0
GPU 0 Slice switch : 0 (0.00/second)
GPU 0 DC6 Exit Reason - Flip: 0 (0.00/second)
GPU 0 DC6 Exit Reason - Register: 0 (0.00/second)
GPU 0 DC6 Exit Reason - Gamma: 0 (0.00/second)
GPU 0 DC6 Exit Reason - Interrupt: 0 (0.00/second)
GPU 0 DC6 Exit Reason - Cursor: 0 (0.00/second)
GPU 0 DC6 Exit Reason - Render: 0 (0.00/second)
GPU 0 FB Test Case 0
**** SMC sensors ****
CPU Thermal level: 19
GPU Thermal level: 0
IO Thermal level: 0
Fan: 2001.43 rpm
CPU die temperature: 65.77 C
GPU die temperature: 58.00 C
CPU Plimit: 0.00
GPU Plimit: 0.00
Number of prochots: 0
</code></pre></div>
<p>For scripted execution, you can limit the output in several ways. If you are only interested in the current temperature and fan speed, <code>sudo powermetrics -n 1 -i 1 --samplers smc</code> returns these without 5 seconds of waiting.</p>404to302 Lambda@Edge2019-09-22T00:00:00+02:002019-09-22T00:00:00+02:00Niobostag:blog.dest-unreach.be,2019-09-22:/2019/09/22/404to302_lae/<p>I discovered the <a href="https://4042302.org/">404to302</a> concept after <a href="https://twitter.com/aral/status/949966935344762880">a tweet</a> by <a href="https://ar.al/">Aral Balkan</a>, and immediately liked the idea. It solves a hard problem (keeping permalinks alive and working) in a fairly easy way: simply archive the old site, and avoid re-using the same permalink again. If you permalinks contain a date of …</p><p>I discovered the <a href="https://4042302.org/">404to302</a> concept after <a href="https://twitter.com/aral/status/949966935344762880">a tweet</a> by <a href="https://ar.al/">Aral Balkan</a>, and immediately liked the idea. It solves a hard problem (keeping permalinks alive and working) in a fairly easy way: simply archive the old site, and avoid re-using the same permalink again. If you permalinks contain a date of some sort, this is very easy.</p>
<p>Since I've moved to host my static site on <a href="https://aws.amazon.com/s3/">Amazon S3</a>, I needed to add this functionality myself. Luckily, <a href="https://aws.amazon.com/cloudfront/">Amazon CloudFront</a> allows you to run custom code when handling a request: <a href="https://aws.amazon.com/lambda/edge/">Lambda@Edge</a>. For this 404to302 concept, I hooked in to the "origin response" phase, inspecting every response for possible 404's. If the response is a 404, it looks up if a fallback URL is configured on the CloudFront distribution for this request, and returns a 302 to that location if present.</p>
<p><a href="https://github.com/niobos/404to302-lae">The code</a> is available on GitHub.</p>Switching to a static website2019-09-21T00:00:00+02:002019-09-21T00:00:00+02:00Niobostag:blog.dest-unreach.be,2019-09-21:/2019/09/21/switching-to-static-website/<p>After over 12 years of using WordPress, I switched to a static site generator. Here's the long story.</p><p>After over 12 years of using <a href="https://wordpress.org/">WordPress</a>, I got tired of keeping up-to-date with all the patches. Wordpress itself was fairly easy to keep up-to-date thanks to the auto-updater. But it doesn't stop there: PHP, MySQL, Apache, ...</p>
<p>So I reconsidered what I used my blog for. It's mostly a way for me to document my experiments, hoping that the information would be useful for someone. Often, that someone is me, a few years down the line. While I did appreciate some useful comments now and then, the fight against comment-spam was tiring as well. So I'm not too sad that I'd loose that functionality.</p>
<p>Which brings me to picking a static site generator. Unlike the "dynamic" CMSes, there are a lot more players in the "static site generation" market. I explored different scenario's:</p>
<p>At first, I wanted to import the entire archive of my WordPress blog into the new tool. That turns out to be close to impossible. There <a href="https://import.jekyllrb.com/docs/wordpress/">are</a> <a href="https://docs.getpelican.com/en/3.6.3/importer.html">tools</a> that claim to do so, but it never worked out: Images being linked to the wrong URL, layout shifting all around, ...</p>
<p>I was about to admit defeat when I discovered the <a href="https://4042302.org/">404to302</a> idea. The idea is as simple as it is briliant: modify the webserver to convert a 404 (File not found) into a 302 (redirect) to the previous version of the site. This way, you no longer need to import all previous content, but all old (perma)links simply keep on working (albeit with an additional redirect step)! This removed one big item from my requirement list: being able to import the previous content.</p>
<p>I ended up picking <a href="https://getpelican.com/">Pelican</a>. It's slightly less popular compared to <a href="https://jekyllrb.com/">Jekyll</a> and <a href="https://gohugo.io/">Hugo</a>, but it's written in a language that I have experience in: Python. I initially tried to get Jekyll to work, but failed to get all required Gems installed on my system.</p>Receiving NOAA images with RTL-SDR2019-08-25T00:00:00+02:002019-08-25T00:00:00+02:00Niobostag:blog.dest-unreach.be,2019-08-25:/2019/08/25/receiving-noaa/<p>During summer brake, I set out for a tech adventure together with my son. We wanted to do weather predictions ourselves. Or at least pretend to do so.</p>
<p>What sparked the idea, was a <a href="https://twitter.com/rtlsdrblog/status/1163702625402904576">beautiful tweet from rtl-sdr.com</a>. Receiving GOES satellites was a bit of a high bar for …</p><p>During summer brake, I set out for a tech adventure together with my son. We wanted to do weather predictions ourselves. Or at least pretend to do so.</p>
<p>What sparked the idea, was a <a href="https://twitter.com/rtlsdrblog/status/1163702625402904576">beautiful tweet from rtl-sdr.com</a>. Receiving GOES satellites was a bit of a high bar for a first attempt, so I started out with receiving NOAA images. Contrary to the GOES satellites, NOAA's are in Low Earth Orbit and have an analogue downlink at 137MHz, which does not require a dish and amplifier to receive.</p>
<p>I followed the guide I found in the <a href="https://noaa-apt.mbernardi.com.ar/guide.html">NOAA_APT</a> GitHub repo. We started out by building a very simple V-dipole antenna out of some leftover 1.5mm² electrical wiring. A V-dipole is not the best antenna to receive a circular polarized signal, but let's first see if we get this to work. I connected the two dipole wires to a short piece of coax to connect it to the <a href="https://en.wikipedia.org/wiki/Belling-Lee_connector">Belling-Lee</a> connector that fits my <a href="https://www.rtl-sdr.com/about-rtl-sdr/">RTL-SDR</a> dongle.</p>
<p><img alt="our V-dipole antenna" src="https://blog.dest-unreach.be/2019/08/25/receiving-noaa/noaa/antenna.jpg"></p>
<p>There used to be a lot of NOAA-satellites active, but as of this writing, only three satellites remain in service: NOAA-15, NOAA-18 and NOAA-19. These satellites are in polar orbits around the earth, completing one revolution every 102 minutes. But since the earth is rotating underneath them, they fly over different parts of the planet every time. Figuring out when the satellite will be overhead requires a degree in orbital mechanics. Or the use of some software, such as <a href="http://gpredict.oz9aec.net/">gpredict</a>. It will calculate where the satellites are at this moment (or an arbitrary moment), and tell you when the next pass will be at your location (or any location, for that matter). It also plots the "ground track" of the satellites, which shows that the satellite visits different parts of the planet on every orbits.</p>
<p><img alt="ground track of NOAA-18" src="https://blog.dest-unreach.be/2019/08/25/receiving-noaa/noaa/NOAA-18-ground.png"></p>
<p>To increase my odds, I picked a pass with very high elevation. The higher the satellite is above the horizon, the stronger its signal can be received. And since I had no idea what to expect, I could use every advantage available. gpredict predicted a pass with 85º elevation for today, so I made sure I was ready to receive "something": Antenna oriented for a North-South trajectory, flat on a plastic table in our garden. Things go quick: in 15 minutes, the satellite rises above the horizon, flies overhead, and disappears on the other horizon.</p>
<p><img alt="polar plot of satellite position" src="https://blog.dest-unreach.be/2019/08/25/receiving-noaa/noaa/NOAA-18-polar.png"></p>
<p>I had <a href="http://gqrx.dk/">GQRX</a> open, tuned at 137.872500MHz, 300kHz bandwidth. I figured that I'd want the signal a bit away from DC to avoid trouble demodulating. NOAA-18 transmits at 137.9125MHz, with an RF bandwidth of 34kHz, which gives me 23kHz room above DC to accommodate any callibration issues and doppler shift. I enabled the FM demodulator, but all I heard was noise. But the waterfall already showed some promise.</p>
<p><img alt="screenshot of GQRX during early acquisition" src="https://blog.dest-unreach.be/2019/08/25/receiving-noaa/noaa/gqrx-early.png"></p>
<p>A bit of patience allowed the satellite to climb higher above the horizon, and that yielded a stronger signal. I could hear the expected 2.4kHz beeping and "tic-toc" sound of the sync signals.</p>
<p><img alt="screenshot of GQRX at peak" src="https://blog.dest-unreach.be/2019/08/25/receiving-noaa/noaa/gqrx-full.png"></p>
<p>The next step was a bit of a disappointment. I wanted to demodulate the FM signal sitting at ~40kHz to baseband using a command line tool. But I didn't find any tool to do this. So I took the easy way out and simply replayed the recording in GQRX, recording the demodulated output to a WAV-file. One note if you intend to use <a href="https://wxtoimgrestored.xyz/">WXtoImg</a>: GQRX records in 16bit 48kHz, while WXtoImg requires 11kHz. Luckily, <a href="https://noaa-apt.mbernardi.com.ar/guide.html">NOAA_APT</a> is happy to accept pretty much anything.</p>
<p>The result was more impressive that I expected. It was a relatively cloudless day, so you could clearly see the coastlines of Denmark and Great Britain, especially on the Infrared channel (right image). Based on the <a href="https://noaa-apt.mbernardi.com.ar/how-it-works.html#about-apt-images">description of the APT signal</a>, the image contains one visible spectrum image (channel 1, 580nm - 680nm, left), and an infrared image (channel 4, 10.30µm - 11.30µm, right). </p>
<p><img alt="noaa-18 image" src="https://blog.dest-unreach.be/2019/08/25/receiving-noaa/noaa/output.png"></p>