Long-term Memory

Controlling SMA inverters via ModBus

2026-05-07T00:00:00+02:00

Harnessing the power of solar energy has become an important step toward a sustainable and energy-efficient home. An key component are solar inverters and home battery systems, which work together to convert sunlight into usable electricity and store excess energy for later use. Controlling these systems effectively can maximize energy savings, but can also optimize the cost/profit when combined with a dynamic energy contract.

Most vendors provide some form of energy management system with their home batteries. It typically uses solar surplus power to charge the battery, and uses battery power to cover periods with solar deficit. But what if you want to something more fancy?

I have a system from SMA. Their Energy Management System (EMS) is called the Home Manager, and it works fairly well, but also has some downsides:

[+] It does basic PV surplus to battery and PV deficit from battery
[+] It supports "peak shaving" to optimize around the "capacity tariff" (a tariff in Flanders that is related to your peak power demand as opposed to energy consumption).
[-] Management of your settings is only possible through the SMA cloud via the internet, and it takes about a minute for changes to be applied.
[-] The peak shaving algorithm is suboptimal. The peaks are calculated as total energy per 15 minutes. This means that you can go over your peak during the last 5 minutes if you were under in the first 10 minutes. The HM does not use this opportunity.

Luckily, you can also ditch the Home Manager, and control the inverters yourself via Modbus. Here is what I found out about that.

Sunny Boy

My first inverter is a Sunny Boy. SMA publishes the Modbus registers in the Downloads section, so that made things a lot easier. One thing to note is that most writeable parameters have a "Warning against cyclical writing". Since these parameters are committed to Flash every time you change them, doeing so often will wear out the storage of the inverter.

There are four registers that seem interesting:

40016, Normalized active power limitation by PV system control, Inverter.WModCfg.WCtlComCfg.WNom
40023, Normalized active power limitation by PV system control, Inverter.WModCfg.WCtlComCfg.WNomPrc
40149, Active power setpoint, Inverter.WModCfg.WCtlComCfg.WSpt
44425, Active power setpoint, Inverter.WModCfg.WCtlComCfg.WSpt

On my machine, both "Active power setpoint" registers didn't seem to do anything. Writing to 40023 however does work. Here is a Python snippet to curtail to 1% production:

from pyModbusTCP.client import ModbusClient

host = "192.0.2.1"
unit_id = 3

def reg_encode(value: int | float, size: int = 16, fix: int = 0, signed: bool = True) -> list[int]:
   """
   Encode a value into a list of integers to write to sequential registers.
   """
    v = int(value * (10**fix))
    b = v.to_bytes(length=size//8, byteorder='big', signed=signed)
    words = [int.from_bytes(b[i:i+2], byteorder='big', signed=False) for i in range(0, len(b), 2)]
    return words

def reg_decode(regs: list[int], fix: int = 0, signed: bool = True) -> int | float:
   """
   Decode a series of registers into the corresponding value.
   """
    b = b''.join([reg.to_bytes(length=2, byteorder='big', signed=False) for reg in regs])
    v = int.from_bytes(b, byteorder='big', signed=signed)
    value = v / (10**fix)
    return value   

c = ModbusClient(host=host, port=502, unit_id=unit_id, auto_open=True)

grid_out = reg_decode(c.read_holding_registers(30775, 2))
print("Current AC power: {grid_out}")

curtail_percent = 1.0
c.write_multiple_registers(40023, reg_encode(curtail_percent, size=16, fix=2, signed=False))

Note that the response isn't immediate. This means care should be taken to avoid oscillating.

But if you give the inverter a couple of seconds to stabilize (3 seconds for the graph below), the control is surprisingly linear. The dips starting at 25% are clouds ruining my test.

And it doesn't seem to be possible to modulate entirely down to 0%:

Sunny Boy Smart Energy

The second machine is a hybrid invertor, the Sunny Boy Smart Energy (SBSE). Besides solar panels, it also has a battery attached for energy storage. This means that I want to have control over battery charging and discharging, as well as PV curtailment. Here also, you can find the Modbus registers in the downloads section.

It took a bit of experimenting, but the following registers seem to work for curtailing and battery management:

41431 and 41433: Maximum/Minimum active power, Setpoint.PlantControl.Inverter.WModCfg.WCtlComCfg.WSptMax and ...WSptMin
41467 and 41469: Max/Min setpoint value for active power of storage, Setpoint.PlantControl.Bat.WCtlCom.WSptMax and ...WSptMin

Both are marked as "Device Control Object", so it should be safe to write to those often. The second pair of registers (41467 and 41469) seem to control the generation power of the battery. So positive values mean discharging, negative values mean changing. When being controlled by the Home Manager, I can see that the WSptMin stays at -15000 W, while the WSptMax is constantly modified to match the current situation. When it's sunny it goes down to -1000 W (charging the battery), when a cloud takes the sunlight away, it goes up to +500 W (discharging the battery).

The following Python snipped will set your battery to charge at 500 W. Note to disconnect your Home Manager, since it will overwrite this setting immediately. Also note that the inverter may have a timeout and fallback configured, so it may stop doing what you sent after some time.

# ... Same boilerplate as above ...

bat_ch = reg_decode(c.read_holding_registers(31393, 2), signed=False, fix=0)
bat_dis = reg_decode(c.read_holding_registers(31395, 2), signed=False, fix=0)
bat_max = reg_decode(c.read_holding_registers(41467, 2), signed=True, fix=0)
bat_min = reg_decode(c.read_holding_registers(41469, 2), signed=True, fix=0)
print(f"Battery power: {bat_ch} W ch, {bat_dis} W disch = {bat_dis - bat_ch} W from battery; setpoint [{bat_min}, {bat_max}]")

storage_power = [-15000, -500]
c.write_multiple_registers(41467, [
   *reg_encode(storage_power[1], signed=True, fix=0, size=32),
   *reg_encode(storage_power[0], signed=True, fix=0, size=32),
])

The other pair of registers (41431 and 41433) control the output of the whole inverter towards the grid. Here also, positive values mean energy is being pushed towards the grid, negative values means the inverter is consuming power. Typically, the Home Manager leaves these at -5000 W and 5000 W for my 5kW unit, although I've seen it put the WSptMax lower.

Since the settings are ranges, the inverter typically does "the right thing". For example, with 700W of solar, and a set maximum of 500W grid export, the inverter will put excess PV into the battery. Note that these control changes happen slowly, over about 5 seconds.

Requesting the battery to discharge at 1.5kW, while limiting the grid output to 1kW causes the inverter to reduce the battery discharge and keep within the 1kW grid limit. This is expected, since the output from the battery is still within the configured limits of -15kW and +1.5kW. Here the control slope is limited to about 23W/s.

We can also give conflicting instructions. Say you command "discharge the battery between -15kW and -2kW" (i.e. charge at 2kW), and you limit "grid output between -500W and +5kW" (i.e. max 500W consumption). If there is not enough solar power, the inverter will initially follow the battery constraint, but after about 30 seconds gently fall back to get within the grid constraints.

Let's say you are on a Dynamic Tariff and there are highly negative prices around midday. You may want to discharge the battery now (while you are still getting paid for putting energy on the grid), so that you can recharge it later from grid power when the prices are negative (and thus you are getting paid to consume energy). To drain your battery now, you want to command "discharge between +5kW and +5kW". But there is no direct control over PV power, so the actual discharge rate is limited by the 5kW grid output minus the PV production. So if you really want to discharge your battery, you need to walk up to the inverter and disconnect the PV arrays.

Around midday that same hypothetical day, you want to configure for maximum grid usage. Also here, since there is no direct control over PV generation, you are limited. But since the battery can charge up to 10kW, the limit will be the energy storage capacity of the battery, and not the grid power: I tested "discharge between -15kW and -6kW" (i.e. charge) along with "grid export between -5kW and -4.5kW" (i.e. import). This resulted in a 6kW charge, but only a 2.5kW grid import (+3.5kW from solar). So the inverter ignored the grid control, but this still allows you to pull in maximum 5kW from the grid as long as your battery has place to put the energy.

Replacing Lightroom

2025-02-01T00:00:00+01:00

We've been using Adobe Lightroom for the last 12 years to manage our family pictures. I have enjoyed using it, but lately am getting more and more annoyed with Adobe's practices. First there was the push to the cloud with the CC-series of their products. And although you can still run Lightroom "Classic", I can't shake the feeling that Adobe isn't going to stop pulling me to their Cloud. Next was the uproar of Adobe using my photo's to train their AI. And although this turned out to be false, here again, I feel like they secretly would want to. Then there's the absurd cancellation fees story. And finally, €18/month isn't really expensive, but I'm not getting €18 worth of value out of it. We manage our holiday pictures, a couple of times a year; we're not doing wedding photography 5 times a week. So I set out to look for an alternative, starting with with the hard part: figuring out what we actually want and need.

One of the nice things about Lightroom, is that it integrates the "management" and the "editing" function into one tool, letting you seamlessly switch between developing and searching. And although there are cases where quickly switching between these two is useful, for example when creating a physical photo-album, most of the time I'm in one mode exclusively. At the end of an event, I usually import the media and groom them. This includes selecting the best picture from a burst, cropping down, adjusting the exposure. When we have lots of pictures, e.g. from a longer holiday, I use the 1–5 stars to extract the highlights without making our family sit through a 350-picture slide-show.

One thing that Lightroom (Classic) lacking in, was accessibility of the library. Since it's a stand alone desktop app, you can only access your pictures on that single machine. And while this was perfectly acceptable in 2012, nowadays, I would like to access my complete photo library from my smartphone while on the go, without actually having to have all pictures on my phone. Privacy and control are very important to me, so I don't want to put my pictures in some cloud. But I do want to have my pictures exposed via a web-interface and optionally an app. This comes with the security risk that I have to secure this web access myself, but I believe I have the needed expertise to do so.

Historically, our family pictures are stored as files in a directory tree, one directory per event, organized by year/month. Really old events have only JPEGs. Around 2012 we got a DSLR, so we switched to using RAW-files. Since we've been using Lightroom, we usually didn't bother exporting JPEGs, and just kept working with the RAW files. Looking back, this may not be the best way to do things: all edits we've made (crops, exposure adjustments, ...) as well as the selections (stacks, stars) are stored by Lightroom and are not (easily) transferable to whatever software we're migrating to. So I intend to change our workflow to export JPEGs from every event and use these instead, while keeping the RAW files tucked a level deeper.

The library management software I'm looking for should:

Be self-hosted
Have a web-interface to view photos & movies.
Support the existing library of pictures, as-is, on disk, preferably read-only. This means support for JPEG/JFIF, HEIC/HEIF and Canon CR2 and CR3 raw files, as well as MPEG-TS and MP4 video's
Be able to handle a significant library size. We currently have over 160k media items, and well over 2TB in data
Have a faceted search tool to find pictures
AI and/or Machine Learning for face detection, recognition and grouping

Nice to haves are:

Ability to share/export an album
Have some form of user management, with the ability to share media between users
AI and/or Machine Learning to extract descriptive metadata of a picture to enable "Smart search"
The ability to show pictures on a map, based on the embedded geotagging. Bonus points if it can handle the format that Lightroom uses to put geotags in XMPs and JPEGs

For the editing part, I don't think we're power users. What we need is:

Compatibility with our DSLRs (Canon)
Lossless editing. This will probably be software-specific
Cropping & rotating
Exposure, contrast and white balance adjustment; Ideally picture-wide as well as localized

Others are nice to have, but not really hard requirements:

Minor touch-ups, mostly spot removal
Lens correction, vignetting correction
HDR
Panorama stitching

The contenders

For the library management I tried both PhotoPrism and Immich. I tried to import my library and tried to do some simulated activities such as find a particular photo in the library, and show an album to an (imagined) family member. And although PhotoPrism didn't lack in any way, I liked Immich better. I'm only using the External Library mode for now to access my historic photos. There's also Nextcloud Memories that seems similar, but I haven't tried that yet. Check out Meichthys's comparison for a full feature comparison.

For editing, I'm currently trying Luminar Neo and DarkTable.

Shapez.io machines

2024-08-31T00:00:00+02:00

Shapez.io is a factory simulation game where you need to design a factory to produce a given shape. The needed shapes increase in difficulty in the first 25 levels, but stay constant after that. Obviously, some factory designs are reusable, so I documented my factory-parts here.

IF you want to use any of these designs, you may find the mod blueprint strings useful. It allows you to copy-paste buildings from/to the Shapez game as strings. Also check out the wiki for other designs that I didn't figure out on my own.

Podman on ZFS

2024-01-03T00:00:00+01:00

When running rootless podman on a ZFS mount, it defaults to the vfs storage driver. And while this works, it causes a huge waste of disk space by duplicating files for every layer. Using overlay solves this when running rootfull, but this is currently not supported on ZFS. It should arrive with ZFS 2.2, but that will probably only be available in Debian 13 Trixie somewhere in 2025. However, we should have the same functionality with the FUSE-variant fuse-overlay, with an additional performance penalty for the userspace roundtrip.

For example, the current Home Assistant image in vfs mode takes up:

$ podman unshare rm -rf .local/share/containers

$ podman system reset
[...]

$ time podman pull ghcr.io/home-assistant/home-assistant:2023.12.4
Trying to pull ghcr.io/home-assistant/home-assistant:2023.12.4...
Getting image source signatures
Copying blob 2651e446c136 done
[...]
Copying config 003e99ae19 done
Writing manifest to image destination
Storing signatures
003e99ae19b0c68a9cbb0c28e68cf843e12513287d1945c1b0b3d1e0eb246973

real    16m1.174s
user    0m58.612s
sys     1m2.829s

$ du -chs .local/share/containers
19G     .local/share/containers

Switching to the overlay storage driver is a huge win, both in time and in storage space:

$ cat .config/containers/storage.conf
[storage]
driver="overlay"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"
#mountopt = "noacl"  # see below

$ podman unshare rm -rf .local/share/containers

$ podman system reset
[...]

$ time podman pull ghcr.io/home-assistant/home-assistant:2023.12.4
Trying to pull ghcr.io/home-assistant/home-assistant:2023.12.4...
Getting image source signatures
Copying blob 2651e446c136 done
[...]
Copying config 003e99ae19 done
Writing manifest to image destination
Storing signatures
003e99ae19b0c68a9cbb0c28e68cf843e12513287d1945c1b0b3d1e0eb246973

real    1m41.778s
user    0m33.646s
sys     0m9.357s
$ du -chs .local/share/containers
2.4G    .local/share/containers

ACL hickup

But this doesn't seem to work for all images. The Grafana image fails with a rather unspecific Operation not supported:

$ podman run -it --rm docker.io/grafana/grafana-oss:10.0.10
Trying to pull docker.io/grafana/grafana-oss:10.0.10...
Getting image source signatures
Copying blob bf463c6d6fd9 done
[...]
Copying config e5b83aa02b done
Writing manifest to image destination
Storing signatures
{"msg":"exec container process `/run.sh`: Operation not supported","level":"error","time":"2024-01-03T08:12:03.063449Z"}

After some looking around, I found these two GitHub issue that seems very similar. The problem described there was that the underlying ZFS file system had ACLs disabled, and the container tried to use them. And indeed, I have:

# zfs get aclmode /tank/podman
NAME         PROPERTY  VALUE        SOURCE
tank/podman  aclmode   discard      default

Adding the noacl mount option to storage.conf seems to solve that:

$ cat .config/containers/storage.conf
[storage]
driver="overlay"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"
mountopt = "noacl"

System-wide

Next, I wanted to configure this system-wide so all users would benefit from this. I thought it would be as simple as moving the config file from ~/.config/containers/storage.conf to /etc/containers/storage.conf, but that does not seem the case:

$ podman run -it --rm docker.io/grafana/grafana-oss:10.0.10
Error: 'overlay' is not supported over zfs, a mount_program is required: backing file system is unsupported for this graph driver

Podman complains about the missing mount_program option, even though it is configured. I haven't figured out why yet... To be continued

Building images

I also had some issues when building container images using podman build. They resulted in an "Operation Not Supported" error when creating a layer.

I haven't figured out what exactly went wrong, but using buildah directly does work...

Configuring WLED to wipe-on the LEDs on startup

2023-11-22T00:00:00+01:00

WLED is an open source project for controlling LED-strips. I wanted my LED strip to "wipe on" when powering on, similar to the first half of the "Wipe" effect, but staying on after that. These are my notes.

Searching around for similar idea's, I found a usermod for a stairway. That module did much more than I needed, but it got me started. In the end, it looked like my desired effect was possible using Presets and Playlist.

I created two presets, a "wipe" and a "solid", and chained these together with a playlist. The playlist has a single entry, the "wipe" preset, and is configured to not repeat, and end on the "solid" preset. Getting the timing right was a bit of a journey: I needed to convert "effect speed" into "seconds to halfway". Luckily, open source code makes things easy. The source code shows the used formula: cycleTime = 750 + (255 - SEGMENT.speed)*150. This means that a speed of 253 gives a cycleTime of 1050ms. So by setting the playlist duration of the wipe preset to 0.5 seconds, I got the desired result.

Another thing to fix is that I wanted the wipe animation to always start at the beginning. This can be fixed by including "tb": 0 in the preset JSON.

Note that you want do disable the Turn LEDs on after power up/reset. When you leave this on, the LEDs first go to on, and only then start the animation.

For reference, here is the relevant part of my preset.json:

{
  "2":{
    "on":true,"bri":255,"transition":0,"tb":0,"mainseg":0,
    "seg":[
      {"id":0,"start":0,"stop":271,"grp":1,"spc":0,"of":0,"on":true,"frz":false,"bri":255,"cct":127,"set":0,"col":[[100,0,152,255],[0,0,0,0],[0,0,0,0]],"fx":3,"sx":253,"ix":128,"pal":0,"c1":128,"c2":128,"c3":16,"sel":true,"rev":false,"mi":false,"o1":false,"o2":false,"o3":false,"si":0,"m12":0},
      {"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0}
    ],
    "n":"wipe"
  },
  "3":{
    "on":true,"bri":255,"transition":0,"mainseg":0,
    "seg":[
      {"id":0,"start":0,"stop":271,"grp":1,"spc":0,"of":0,"on":true,"frz":false,"bri":255,"cct":127,"set":0,"col":[[100,0,152,255],[0,0,0,0],[0,0,0,0]],"fx":0,"sx":16,"ix":108,"pal":0,"c1":128,"c2":128,"c3":16,"sel":true,"rev":false,"mi":false,"o1":false,"o2":false,"o3":false,"si":0,"m12":0},
      {"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0},{"stop":0}
    ],
    "n":"default on"
  },
  "1":{
    "playlist":{"ps":[2],"dur":[5],"transition":[0],"repeat":1,"end":3,"r":0},"on":true,
    "n":"power on animation"
  }
}

Specs of the SK6812 LED strip

2023-08-31T00:00:00+02:00

I was considering a SK6812 strip for a project of mine, but I found it very hard to find information online about the power consumption. So I ordered a test strip to do the measurements myself. The SK6812 is an individually addressable LED strip. I got the RGBW variant, where each dot has a Red, Green, Blue and White LED that can be controlled. The protocol used is similar to the WS2811/WS2812/WS2813 series, but reading 32 bits per LED instead of 24 bits. This is what I found.

The supply voltage is 5V
A 5m, 300 LED strip consumes about 70mA in idle condition, or 0.23mA/LED
My ESP32-based controller requires another 150mA, or a bit more when it's actively refreshing the LEDs
For the LED current, I measured the current for 1, 2, 5, 10, 20 and 40 LEDs, and determined the slope. The Red LED averages at about 7.98mA/LED at full brightness; Green is at 8.11mA/LED; Blue is 7.98mA/LED; White is 16.11mA/LED
100% RGB gives 23.06mA/LED, 100% RGBW gives 38.9mA/LED

With these currents and only 5V to start from, voltage drop is a significant problem. Based on some measurements, I estimate the resistance between 2 LEDs to be around 6.2mΩ. The voltage drop increases approximately with the square of the number of LEDs, so this problem grows quickly!

I measured what minimum voltage is needed to avoid fading. To do that, I folded the strip to put the first & last LED next to each other and turned them on full brightness. Next, I pulsed the middle LEDs and watched for a brightness difference between the first LED (getting the full supply voltage) and last LED (getting the full voltage drop because of the middle LEDs). I increased the number of pulsing LEDs until I could see the last LED pulse as well. I was slightly surprised to find that I could drop down to 3.4V without noticing any dimming; I was expecting a higher threshold. When the voltage dropped to 3.37V (1 more pulsing LED in the middle), the fading was just noticeable.

So for my 5m 60LED/m strip, this means I need power injected for every 1.9m. Since you can power from either end, I can stretch this up to 3.8m before I need to inject power in the middle of a run. Note that these numbers assume you can inject 5V at that point. If the 5V power supply is not located right next to the strip, you should account for the voltage drop in the supply cables as well. To sidestep that problem, I'm thinking of using a 12V (or 24V) power supply, and put a 12V to 5V buck converter right next to the strip.

Color output

I measured the LEDs color with the Opple Light Master 3:

Red (x, y)=(0.6532, 0.2984)
Green (x, y)=(0.1637, 0.5288)
Blue (x, y)=(0.1609, 0.0709)
White (x, y)=(0.4181, 0.4259), CCT=3500K, ∆uv=0.019, so slightly greenish
100% RGBW (x, y)=(0.3407, 0.3217), CCT=5100K, ∆uv=-0.014, so slightly purplish
100%W + 55%R + 57%B corrects for the greenish tint and yields (x, y)=(0.4072, 0.3951), CCT=3500K, ∆uv=0.001

Kerbal Space Program Grand Tour

2023-07-05T00:00:00+02:00

I wanted a challenge, so I decided to try a "Grand Tour" of the Kerbolar system in Kerbal Space Program. A "Grand Tour" is a single mission where you visit all bodies in the Kerbolar system. The definition of "visit" varies from fly-by to manned landing, depending on how challenging you want things to be.

I decided to try the "manned landing"-variant in a fully reusable craft, except for Eve (and Jool and Kerbol, since you can't land there).

Going around the entire Kerbol system requires about 20km/s of ∆v. That's a lot. So I decided to equip my ship with an ISRU and mine extra fuel along the way.

Eve – Moho

The Eve–Moho hop is the most challenging hop ∆v-wise. A fairly good transfer window still requires 3480m/s from Low Eve Orbit to Low Moho Orbit, plus an additional ~1000m/s for landing. By departing from Gilly and diving into Eve's gravity well to maximize the Oberth effect, I was able to reduce this to around 2600m/s, depending on the chosen transfer window.

This particular transfer requires a low-TWR burn from Gilly to dive into Eve of about 530m/s. At Eve periapsis, we need to do an escape burn of 930m/s, at reasonably high TWR, since we lose Oberth effect fast when this burn takes too long. The capture burn is 870m/s, also at high TWR. Finally we need to circularize the orbits around Moho with an additional 320m/s, but that can be low-TWR and split between orbits. Landing can start out at low TWR, but should end up well above 2.7m/s² (Moho's surface gravity).

Tylo

Landing on Tylo is hard, with a beefy 2300m/s of speed to bleed off and no atmosphere to help with the breaking. Tylo also has a surface gravity of 7.85m/s², so to do some actual breaking, we're looking at at least 10m/s² (TWR=1.3) of thrust, but preferably 16m/s² (TWR=2) or more.

Getting to Tylo is easiest from Pol, where it's easy to refuel. The trip needs approximately 1000m/s of ∆v: The Pol escape burn of 215m/s can be done at low TWR, since the low gravity of Pol doesn't yield much Oberth-effect gains. Arriving at Tylo, we only need about 42m/s at a periapsis of 20kmAGL to remain in orbits around Tylo. Circularizing needs another 800m/s, but we can split that out over multiple orbits.

The ship

With the above constraints in mind, I set out to design a spaceship. I started with a spaceplane design, but couldn't get enough ∆v packed into it. I guess the "dead mass" of the gears, wings and control surfaces adds up. In addition, I needed an extra set of downward pointing engines for landing on bodies without atmosphere, adding to the dead mass. So I ended up with this Single Stage To Orbit (SSTO) rocket:

The rocket consists of a Mk1-3 Command Pod, topped with a Shielded Docking Port. Underneath is the Convert-O-Tron and a 2.5m to Mk3 Adapter. On the adapter are 4 Airbrakes to assist in orienting the rocket engines-first during atmospheric re-entry, as well as 4 Vernor RCS engines to keep the rocket upright at touchdown. The 8 parachutes are attached here as well. Next up is the short Mk3 cargo bay with an ore tank, the Narrow-band Scanner to pinpoint landing locations, as well as a comm dish to be able to access KerbNet. I also packed a bit of extra torque to manoeuvre this thing a bit better. Next are fuel tanks: A Mk3 Rocket Fuel, a Mk3 Liquid Fuel and 4 Jumbo-64 tanks. The central column has the NERV engine, the 4 side tanks are fitted with a Vector engine. At the bottom I added a Drill-O-Matic Junior on a piston to be able to reach the ground. Power is generated by a PB-NUK for normal operations, 4 Gigantors and a Fuel Cell Array during mining. Cooling is provided by 4 small Thermal Control Systems.

It has enough fuel to do the Eve–Moho leg described above, with 1000m/s to spare. The Pol–Tylo leg also has enough fuel, and leaves us with 3700m/s for the 2300m/s landing. When using all engines, we can start our landing burn at 18.7m/s² (TWR=2.4), increasing to 70m/s² (TWR=8.9) when empty. When fully re-fueled, we still get 16.4m/s² (TWR=2.1) for takeoff.

The Trip

My trip around the solar system obviously started at the KSC on Kerbin. From there I went to:

Minmus

Mun

Back to Minmus to refuel for the trip to Gilly

Moho

Gilly again

Ike

Duna

Ike again

Dres

Pol

Tylo

Bop

Val

Laythe

Back to Pol for a refuel and a really long wait for a transfer window to Eeloo

Then straight to Minmus, which was on the edge of the ∆v capabilities of the ship.

And finally back home on Kerbin, after about 45 years of travel.

Flight Simulator bush trips

2023-06-12T00:00:00+02:00

Microsoft Flight Simulator provides "bush trips": a set of VFR flights across an area with nice scenery. When flying these bush trips, you have to rely on the description only, as no navigation aids are available. To me, as a non-native speaker, the descriptions aren't always very clear.

So I decided to prepare each flight a bit better by having a SkyVector map ready. This allows me to cross-reference landmarks to reconcile my current position on the map.

Alaska bush trip: Unalaska to Kulik Lake

This trip takes you along the Alaska islands in an XCub. These hops are based on the Google Earth path by Thadius856.

In Search for a new Monitoring Solution

2023-01-07T00:00:00+01:00

I'm currently running all of my monitoring through Prometheus, backed with InfluxDB for the long-term storage of selected metrics. But recently, I've been limited in functionality when creating graphs in Grafana. And although I like Prometheus, it looks like my use-case is outside the domain of Prometheus.

Use Case

I want to monitor my home's energy consumption throughout the year. The metrics I have available are:

An energy counter for gas, updated every few minutes. Since these are counters, they may reset to 0 at any time, and this should be accounted for when graphing.
The energy price. This metric is not known real-time, and is very coarse: 1 datapoint per day. Depending on the contract, it may be known up to 3 months in advance, or only 1 month after the consumption.

I would like to create the following graphs:

To make this graph, I need several features from the time series database:

Handle counter resets correctly.
Calculate back-and-forth between the cumulative counter and the derived rate: To convert from energy to price, the current unit price needs to be applied to the marginal usage in the corresponding time period. You can't just multiply the counter value with the price as that would create a discontinuity.
Be able to draw cumulative/running sums. I.e. start at zero at the left edge.

Prometheus

Since Prometheus is my current monitoring solution, I first tried to get it working with Prometheus. I covered the cumulative part of the problem before. That solution can't handle counter resets, but the biggest hurdle is incorporating the price: it seems that there is no way to have Prometheus calculate a cumulative sum.

InfluxDB

Since I'm already using InfluxDB as long-term storage anyway, the next logical step was to skip Prometheus and interact directly with InfluxDB. I'm fairly new to the Flux language, but the query below seems to give the results I want:

import "date"

earlyStartTime = date.sub(d: 1d, from: v.timeRangeStart)
// Make sure we have the previous gas_price in range.
// gas_price is update (at least) once a day

from(bucket: "default")
  |> range(start: earlyStartTime, stop:v.timeRangeStop)
  |> filter(fn: (r) =>
    (r._measurement == "gas" and r._field == "volume_m3") or
    (r._measurement == "gas_price" and r._field == "price_EUR_per_m3")
  )
  |> group()  // join series into a single table for pivot
  |> pivot(rowKey: ["_time"], columnKey: ["_measurement"], valueColumn: "_value")
  |> sort(columns: ["_time"])
  |> fill(usePrevious: true, column: "gas_price")  // fill in missing gas_price
  |> difference(nonNegative: true, columns: ["gas"])  // convert to per-sample difference, nonNegative detects resets
  |> range(start: v.timeRangeStart, stop:v.timeRangeStop)  // crop to actual requested range after fill (to have price info)
                                                           // and difference (to have data for the first point)
  |> map(fn: (r) => ({_time: r._time, _value: r.gas * r.gas_price}))
  |> aggregateWindow(every: v.windowPeriod, fn: sum, createEmpty: false)  // We no longer need dense data at this point
  |> cumulativeSum()

Rootless podman on Debian Bullseye

2023-01-03T00:00:00+01:00

My home server already uses LXC-containers. I use them to keep services isolated, so that I can upgrade my mail server without causing issues to my Nextcloud setup. But these containers contain a full OS inside them. And while this works, it's a bit of work to upgrade a handful of containers every few months. Docker-style containers should be a better fit for this single-service-per-container setup.

After reading up a bit, I decided to go for podman. I was mostly convinced by the ability to run rootless, but it seems Docker is getting that as well.

Rootless networking

In my LXC setup, every container gets its own veth-based network interface. On the host, these veth-interfaces are bridged together with the physical network card. That way, every container gets its own MAC and IP(v6) address.

But since we're running as a normal user, podman can't create a veth-pair. It was very interesting to learn (how podman solved this)slirp4netns with some netns and tap magic!

The result isn't exactly equivalent to what I had on the LXC-side: the containers do have their own IP, but it's not exposed to the outside world. I found (a mailing list post describing how to roll my own)manual-netns, but I haven't figured out yet how to combine this with DHCP...

Compose

Podman has its own docker-compose equivalent. Unsurprisingly, it's called podman-compose, but is otherwise very similar.

I didn't find a ready-made way to auto-start a service, so I rolled my own systemd service which I put in ~/.config/systemd/user/podman-compose.service:

[Unit]
Description=Rootless pod (podman-compose)

[Service]
Type=oneshot
RemainAfterExit=true
WorkingDirectory=%h
ExecStart=/usr/local/bin/podman-compose up -d --remove-orphans
ExecStop=/usr/local/bin/podman-compose down

[Install]
WantedBy=default.target

This will run compose.yaml from the home directory of the user.

The Unifi Protect G4 Doorbell

2022-06-12T00:00:00+02:00

I've been dragging my feet for years to get a video doorbell. I wanted a doorbell that "just works" as a doorbell, even if the WiFi is down, and I didn't want my video footage to be stored in the cloud, but only accessible when on the local network, either directly, or via VPN. Apparently, these are fairly high requirements nowadays... I settled for the Unifi Protect G4 Doorbell. I wanted to get the Pro-variant, but that one still hasn't left Early Access.

There were some issues, though. The first one is that this doorbell works on 16V AC, and requires the chime to also work with that. While 16V may be the standard in the US, here where I live, most chimes are 6V. They'll probably work with 16V, but not for a very long time.

The Chime

You don't need to connect a chime. You can go all in digital and only use your phone to notify you someone is at the door. But if you do, you need to connect a special white chime-box adapter in series with the doorbell:

Presumably, this box would output 16V AC when the door bell button is pressed. So I needed to find out what exactly is being output from this box, and how I can convert it to the correct voltage.

Contrary to my expectations, there was some output present while the system was in idle:

The signal is non-sinusoidal, but 50Hz periodic and about 5.5Vpeak-to-peak. When I pressed the button, nothing happened.

In the Doorbell's settings, you can choose which chime is attached. By default, it's configured to "None". The other settings are "Mechanical" and "Digital". When selecting Digital, you can choose a duration between 1 and 10 seconds.

So I choose "Mechanical", and pushed the button again. Now something did happen: the output of the chime box adaptor jumped up to 21Vrms AC and stayed there for many seconds, but the Doorbell shut down because it lost power. It recovered after 30seconds or so.

It looks like the chime box adapter outputs the full output of the transformer to the chime connections, leaving no power for the doorbell itself. I'm assuming that the voltage is supposed to be shared between the chime and the doorbell during this period, but since I have no chime connected (yet), all voltage falls over my oscilloscope and nothing remains for the camera.

So I decided to load the output with a (second) transformer to step down the voltage from 21Vrms AC to something in the neighbourhood of 6~10V. This seems to have done the trick:

The chime box output full voltage for around 430ms, and resets to its idle state, leaving the camera enough power to keep on working.

Reverse engineering the Mitsubishi heat-pump WiFi adapter

2022-06-04T00:00:00+02:00

Recently we had a heat pump (A/C, air conditioner, AirCon) installed. The indoor units came with the usual IR remote, but they also had a built-in WiFi module along with a smartphone app to control them from anywhere. I'm not a big fan of connecting home appliances to clouds for a variety of reasons:

Running a cloud service requires a continuous stream of money to keep the service running. So either you will have to pay some sort of subscription, or the service will (eventually) stop working. There is also the possibility that the manufacturer is very altruistic, but that isn't very likely.
If my appliance is connected to the internet, the manufacturer can choose to alter its functionality. Or make it stop working entirely.
Connecting things to the cloud makes them more vulnerable to attacks. Right now, an attacker needs to be within infrared range of my unit (a few meters) to control it. Once the device phones home to its cloud, an attacker can potentially control thousands of units from their couch.

Don't get me wrong. I love when appliances offer some sort of API to control them externally. And I see why a cloud-service is a nice addition for a lot of people. But I prefer local control.

The Mitsubishi WiFi adapter

My heat pump came with a MAC-577IF2-E adapter built-in. So I set out to try and locally control the unit via WiFi. If you're in a hurry, I'll save you some time: I didn't succeed. But here is what I did find out, in the hope it can be useful for other people.

I started by looking around to interesting projects: I found the meldec GitHub project that can decode the messages sent to MELcloud. Another related project I found was HeatPump, but that focuses on communicating directly with the unit over its serial interface.

Update 2025-07-28: Ash Hopkins did some really amazing work on this and figured it all out.

First discoveries

For setup, I switched the adapter into Access Point mode. After associating with it using the SSID & WPA-PSK printed on the box, I could connect to a web-server to configure the network settings. The page was fairly simple:

Once the details were filled in, it connected to my home WiFi network just fine. The MAC address of the unit stats with 70:61:BE, which is assigned to Wistron Neweb Corporation. After requesting an IP via DHCP, it started to phone home, of course. First thing it did was a DNS-lookup for production.receiver.melcloud.com.

The first request to that domain is a plain-text HTTP POST-request to /synchro:

POST /synchro HTTP/1.1
Host: production.receiver.melcloud.com:80
Accept: */*
User-Agent: MAC-577IF-E
Pragma:  no-cache
Content-type:  text/plain; charset=UTF-8
Content-Length: 68

<?xml version="1.0" encoding="UTF-8"?><LSV><SYNCHRO></SYNCHRO></LSV>

The response contained the current datetime:

HTTP/1.0 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Date: Sat, 4 Jun 2022 09:47:47 GMT
Server: Microsoft-IIS/10.0
X-Robots-Tag: noindex, nofollow, noimageindex

<?xml version="1.0" encoding="UTF-8"?><CSV><SYNCHRO><DATE>2022/06/04 09:47:47</DATE></SYNCHRO></CSV>

So far so good. The next connection, however, was an encrypted HTTPS request to the same domain. production.receiver.melcloud.com presents an HTTPS certificate that is signed by MELCloud Root Authority, and thus not trusted by browsers (by default). I tried to redirect this request to my own server, presenting a look-alike certificate chain. But the device refused with an Unknown CA error: it didn't like my own spoofed CA and wanted to see the actual Mitsubishi CA. Normally, I consider this a good thing for security, but in this particular case it's a bummer...

Inbound

There also seems to be a webserver listening on the device itself on TCP port 80. But according to nmap, there is nothing else listening on TCP (all other ports are connection-refused). Note that a normal nmap-scan crashes the web server, so you won't see it as open on a consecutive scan.

But the web server seems to be locked down: requests to / require authentication, which I don't have/know:

> GET / HTTP/1.1
> Host: IP-address-of-unit-omitted-for-privacy
> User-Agent: curl/7.79.1
> Accept: */*
>

< HTTP/1.1 401 Unauthorized
< Content-Type: text/plain
< Content-Length: 22
< WWW-Authenticate: Basic realm="generic"
<
< Authorization Required

Some URLs do work without authentication: /license, /common.css, /javaScript.js work. Others return a 404 instead of a 401.

Digging deeper

Someone posted the firmware to GitHub. Time to brush off my reverse engineering skills.

Based on a simple strings of the firmware, I found C:\sdk-ameba-v4.0c\component\common\mbed\targets\hal\rtl8195a\gpio_api.c. So it looks like the hardware may be an Ameba 1. This houses an ARM Cortex M3 CPU clocked at 166MHz with 2.5MB of SRAM and built-in WiFi (2.4GHz only). I used radare2 to disassemble the binary, and kept the ARM Cortex M3 Instruction Set Reference nearby. It looks like the SDK for this SoC is available on GitHub.

The Flash layout section of the memory layout PDF from the SDK contains some information on the boot loader process. From the description, it seems that an image starts with a 4-byte length, a 4-byte address, and 8 bytes of 0xffffffffffffffff. The Memory Mapping section of the RTL8195A data sheet tells what address is mapped to what hardware. Based on that info, the firmware contains two sections:

A first section of 0x4aefc = 306_940 bytes, to be loaded at 0x10006000, which is in SRAM
A second section of 0x10cea4 = 1_101_476 bytes, to be loaded at 0x30000000, which is in SDRAM
The final 4 bytes of the file. Maybe a CRC of some sort?

While scrolling through the strings output, the function at 0x3001de60 caught my eye: It's the only place to reference the string 401 Unauthorized. The only place where this function is referenced, is around 0x30028984. This section loads the string Authorization, and calls a function (I'm guessing to find this header). Next, it checks if the returned value (?) starts with Basic, and passes the rest of the value to 0x30028820.

0x30028820 seems to be the "verify authorization"-function. It seems to iterate through a linked list of entries, checking the path (?). If the entry matches, the base64-auth-value is compared to another attribute of the entry. If it matches, the "return a 401"-part is skipped. Interestingly, the comparison is done by first memcmp()ing, and later verifying that strlen() is equal. The actual memcpy() implementation is in ROM, but it's possible this is vulnerable to a timing side channel (although doing timing side channels over WiFi is hard...).

Unfortunately, this function just checks the auth-value against a pre-computed list. So I still need to find the code that generates this list... The list is loaded from *(*(0x30029130)+0x18). 0x30029130 contains 0x1004f134, but it may be overwritten by the time we get here. 0x1004f134+0x18 = 0x1004f14c contains 0x00000000 at bootup. This will definitely be overwritten, since otherwise no authentication will work (the linked list has 0 entries). And I know that /config can be accessed with username user and the Key-password printed on the device.

Trying another approach

0x300696bc looks interesting: it seems to contain a list of paths & usernames. The entries seem to have the odd size of 0xe7=231 bytes.

Path \ user	user	root	suser	admin
/network		x	x	x
/unitinfo		x	x	x
/service		x	x	x
/server			x	x
/update			x	x
/updateFinish			x	x
/updateFail			x	x
/default			x	x
/analyze				x
/apinfo		x	x	x
/config	x	x	x	x
/smart
/adapter_image2.gif
/javaScript.js
/common.css
/license
/		x	x	x

Based on /license, I'm assuming "no user listed" means "unauthenticated".

Backups

2022-05-04T00:00:00+02:00

Recently I was rethinking my off-site backup strategy. Ever since Amazon AWS launched their Glacier Deep Archive storage tier, it looked really interesting for backup use-cases. For starters, it's super cheap to keep your bytes around: $1.01376/TiB/month in most regions as of this writing. I haven't found any other storage proposition this cheap. Of course there is some fine print, but it matches my backup use-case:

you are charged for at least 180 days of storage, even if you delete the objects before that
you are charged for some overhead on top of the stored objects. Especially for smaller objects, this can add up significantly

But all in all, it still was very attractive.

Requirements

Based on discussions I had, it seems I have a fairly beefy list of requirements:

Bringing the backup up-to-date should be cheap. This means I don't want to update the full content every single time. I prefer to only upload what has changed (rsync-style).
Keeping the backup around should be cheap.
Restoring the backup may be expensive, either in time, money or both. Since this is not my first backup, I intend to never need it. So paying several hundred euro/dollar to get my most valuable files back is fine.
Restoring should be "easy". Since I'll be using this backup in a disaster recovery mode, I prefer to be able to restore crucial files without the need for any particular software, just relying on standard tools.
Backups need to be client-side encrypted, preferably with auditable/trusted tools, independent of the backup tool.

I looked around for existing projects and/or products, but didn't find one that checked all my boxes. Especially the "client-side encrypted" seemed like a high bar. So I decided to roll my own.

Design

S3 natively supports versioning of objects. By default, you see the latest version of an object. Or, if the last version is a "delete marker", the object is hidden by default. But you can request previous versions explicitly if you need them. In addition, I use S3 Lifecycle management to remove old versions after a configured amount of time.

This setup ensures the "easy restore"-requirement: The S3 web console gives me access to the most recent backup content, but allows access to previous versions if needed.

The client-side encryption is a bit more challenging: the backup-tool needs to figure out if a particular file has changed since the previous backup or not. You can't just encrypt the file again, and compare that, since encrypting the same file twice is not guaranteed to give the same result. My solution is to add metadata to the S3-object: the size of the plaintext file, and a hash of the plaintext file. The size is included because it's very cheap to check: if a file's size has changed, the file has changed and needs uploading. The hash is more expensive to calculate, but will spot changes even when the file size is the same. I know there is an astronomically small chance that a changed file will not result in a changed hash, but I'm taking my chances.

To make things more efficient, I also included a client-side cache of the S3 content. Listing the objects in an S3 bucket is fairly efficient, but getting the metadata requires a HEAD-call per object. At 71 seconds per 1000 calls, which is what I practically get, this takes way to long for a 200k-object backup. Not to mention the monetary cost of doing these calls every single time. But since this data is cached locally, care should be taken to never modify the bucket directly. This would cause the cache to be incorrect, and may result in bad backups.

Small files

One of the things I want to backup are my git repositories. Git has the habit of creating lots of small files. And while the above design supports small files, it gets slow and expensive: Uploading ten 1-byte files takes way longer than uploading a single 10-byte file. And since there is an additional 40kB overhead per object (some of it billed at standard S3 pricing), a single-byte file is (relatively speaking) expensive.

So I wanted to ZIP together smaller files and upload the ZIP instead. This is where it gets tricky: you want both very small ZIPs and very large ZIPs to cover contradicting needs. On one hand, you want the ZIPs to be as large as possible, to maximize the efficiency-gains. On the other hand, you want the ZIPs to be as small as possible, since a change to a single file in the ZIP requires the upload of the complete new ZIP.

From a restore point of view, you also want the ZIPs to be "logical". Given a filename and a point in time, I want to be able to cherry-pick that file from the backup with as little overhead as possible. I want to either find the file itself in the backup, or should be able to tell fairly easy which ZIP contains the given file.

I went through several variants of grouping-logic. My current implementation takes a configurable threshold as input. Files that are larger than the threshold are stored to S3 directly. Smaller files are grouped based on their filenames, where the algorithm tries to find the longest prefix that will result in a ZIP of at least the required size. So given the following files and a threshold of 1 MiB:

large-file        5 MiB
.git/
  a               1 KiB
  b               1 KiB
  c               1 KiB
  d               4 MiB

It will pass through large-file and .git/d, but will ZIP the other files in the .git/ folder together as .git/*~.zip. To find a file for restore, I know that the file will be either directly visible on S3, or be in first ZIP-file I encounter by removing characters from the end of the filename I'm looking for.

Surviving Mars

2022-04-11T00:00:00+02:00

I discovered another game that gives me an excuse to write silly calculator modules: Surviving Mars. Some random notes and a few tools I wrote, or interesting tools I found online:

Kerbal Space Program

2021-08-25T00:00:00+02:00

I am usually late to proverbial parties, but I think I’ve outdone myself by discovering the computer game Kerbal Space Program roughly 10 years after it first became playable. It’s the kind of game that triggers my “I’ll optimize this by writing a script”, so here are a few of the tools I wrote, or interesting tools I found online:

And some random notes:

When launching from KSC to rendez-vous with something in a circular orbits at 100km altitude: For vertical launches with "my usual ascend profile", launch when the target is 26º before the KSC position (viewable as Phase Angle). For spaceplane launches (with RAPIERs and 10º pitch up all the way), launch when the target is 16º before the KSC.
When starting from a 100km circular orbits around Kerbin, a deorbit burn down to Pe=40km will touch down 152º later. Craft consists of:
- Mk3-1 Command module
- RC-L01 (for storing science and remote control of probes)
- Large Heath Shield
- Mk16-XL Parachute
- 2 × Mk2-R Radial-Mount Parachute
For interplanetary travel, you need ~930m/s of Δv to leave Kerbin SoI starting from a 100kmAGL orbits, and an additional 100~1000m/s to reach other planets. Design for a high enough TWR to do this in a reasonable time. Multi-impulse departures are possible: as long as the apoapsis is < 8.9MmASL (9.5Mm), the Mun will not alter your orbits. The below table tries to keep burns <2 minutes until we cross the orbits of the Mun.

TWR [G_Kerbin]	Burns for escape	Burns for 2100m/s
1.79	53s	120s
1.5	63s	56s + 87s
1.0	95s	84s + 130s
0.75	111s + 15s	111s + 174s
0.696	120s + 16s	120s + 186s
0.5	120s + 47s + 23s	120s + 47s + 261s
0.25	2×120s + 94s + 45s	2×120s + 94s + 522s
0.1	6×120s + 115s + 113s	6×120s + 115s + 1305s

Most engines produce a constant thrust while on. Its fuel consumption scales with the time the engine is on. But the energy scales with the distance the force is applied. This means that running an engine at a higher speed, yields more kinetic energy for the same amount of fuel. This is known as the Oberth effect.

When starting out at Minmus, this means that it could be beneficial to first spend some ∆v to dive into Kerbin, and burn at periapsis for the escape trajectory. Escaping from Kerbin from a Minmus orbits requires at least ~115m/s. The boundary seems to be 274.1m/s: for ∆v's above that, it's optimal to first burn 228.1m/s to put our Kerbin periapsis at 70km (just above the atmosphere), and burn the rest of our ∆v there. For a 500m/s total ∆v, this boosts our hyperbolic excess velocity from 670.1m/s (when expended at Minmus orbits) to 1294.8m/s when diving first and spending the remaining ∆v near Kerbin. This is the case for all trajectories to other planets.
The orbits of Minmus has an inclination of 6°. Ideally, we want to launch in this orbital plane to avoid plane-change maneuvers. Launching into a 6° inclined orbits can be done from launch sites with a latitude of less than 6°, i.e. the Kerbal Space Center. The Longitude of the ascending node of Minmus’s orbits is 78°. So we can either launch slightly northbound when KSC passes 78° right ascension, or slightly southwards when KSC passes 258°. In-game, it’s not easy to read out the current right ascension. We can, however see the Longitude of the Ascending Node (LAN) of our current orbits in the second orbital info pane:

Since KSC is in the southern hemisphere, our current position is 90° before our ascending node. In other words: launch slightly south (97°) when LAN reads 348°, or slightly north (83°) when LAN reads 168°. You need to eyeball these directions, especially at the beginning of the launch, when mostly vertical, where the Heading-indicator is very sensitive. I do my gravity turns to the right. In that case, you can align the 90° line on the navball with the longer tick mark of the G-meter for 83° departures. 97° departures should aim for the letter G in the G-meter.
Launch windows for the first few years:

Departure Date	Arrival Date	Destination	∆v (m/s)
Y1 D98	Y1 D64	Moho	5165
Y1 D196	Y4 D138	Jool	5043
Y1 D231	Y2 D75	Duna	1678
Y1 D242	Y6 D227	Eeloo	3635
Y1 D269	Y1 D405	Moho	5031
Y1 D390	Y3 D93	Dres	3526
Y2 D86	Y2 D243	Moho	4833
Y2 D160	Y2 D357	Eve	2741
Y2 D242	Y5 D304	Jool	4978
Y2 D245	Y2 D386	Moho	5242
Y2 D257	Y7 D20	Eeloo	3556
Y2 D393	Y3 D86	Moho	5748
Y3 D83	Y3 D225	Moho	4481
Y3 D120	Y4 D283	Dres	3072
Y3 D270	Y7 D63	Eeloo	3644
Y3 D275	Y6 D216	Jool	4902
Y3 D305	Y4 D160	Duna	1645
Y3 D366	Y4 D63	Moho	5695
Y3 D401	Y4 D162	Eve	2560
Y4 D84	Y4 D207	Moho	4179
Y4 D226	Y6 D108	Dres	3348
Y4 D289	Y5 D40	Moho	5383
Y4 D294	Y7 D365	Eeloo	3943
Y4 D306	Y7 D159	Jool	4976

Monitoring network bandwidth of individual hosts from the router

2020-11-28T00:00:00+01:00

I live in a part of the world where broadband internet subscriptions come with a monthly data cap. The datacap is relatively large (750GB/month), but streaming 4k content will get you there! So I wanted to keep taps on my data usage. My router is the ideal place to monitor this: all traffic needs to pass through there anyway. And since it runs OpenWRT, I can easily add the component I need.

Interface-level statistics

The first step is relatively easy. The Linux kernel already keeps a counter for each network interface that counts the number of bytes it has sent & received. These counters are exposed through the ifconfig command, but also through the /proc/net/dev "file".

I'm using Prometheus as my monitoring tool, so I wrote a simple shell-script to expose these numbers as prometheus metrics. I should note that I'm not very fluent in Prometheus metric naming, so I might have picked something "wrong".

NAME="wan"
STATS="$(grep pppoe /proc/net/dev )"
echo "network_bytes_total{interface=\"$NAME\",direction=\"rx\"} $(echo "$STATS" | awk '{print $2}')"
echo "network_packets_total{interface=\"$NAME\",direction=\"rx\"} $(echo "$STATS" | awk '{print $3}')"
echo "network_bytes_total{interface=\"$NAME\",direction=\"tx\"} $(echo "$STATS" | awk '{print $10}')"
echo "network_packets_total{interface=\"$NAME\",direction=\"tx\"} $(echo "$STATS" | awk '{print $11}')"

To expose these metrics to Prometheus, I converted the above snippet into a CGI-script. This sounds more complicated than it is. I just added these lines to the top, and linked in into /www/cgi-bin/metrics.

echo "Content-Type: text/plain; version=0.0.4"
echo ""

Host-level statistics

Adding metrics for individual hosts turned out to be a lot more complicated than I anticipated. In the good old days, one could simply use an iptables-table that matched all 254 IP-addresses of the subnet, and call it a day. But we don't live in the good old days anymore, I live in a time where I have native IPv6 connectivity!

IPv6 complicated this idea a lot. Not only is it unfeasable to create a table of all 18446744073709551616 IPv6 addresses in the subnet, IPv6 Privacy Extensions means that there is no way to combine the different addresses for a particular host... The only identifier that is usable in a Dual-Stack environment with IPv6 Privacy Extensions is the Ethernet MAC address of the host.

Once I realised I should use the MAC addresses, the rest seemed simple:

iptables -N accounting
iptables -A FORWARD -j accounting
iptables -A accounting -m mac --mac-source 00:00:5E:01:02:03 -j RETURN
iptables -A accounting -m mac --mac-destination 00:00:5E:01:02:03 -j RETURN

Only, that doesn't work... There is no --mac-destination filter. The reason is pretty simple: we are still in IP-land, so the kernel does not know yet what MAC-address it will sent the frame to. You could get around this by adding a complete ebtables setup, but that seems like the "wrong" solution to track IP-level counters.

The solution is rather complicated. And I couldn't have done it without help from this SuperUser answer. Use the connection tracking logic to carry the MAC-information over to the return packets:

# CONNMARK only works in the mangle table, so we'll do the accounting there
iptables -t mangle -N accounting_out
iptables -t mangle -N accounting_in

iptables -t mangle -A FORWARD -j CONNMARK --restore-mark  # Restore the saved mark from the CONNTRACK table into the IP-level MARK
iptables -t mangle -A FORWARD -i pppoe -j account_in      # do accounting based on the MARK
iptables -t mangle -A FORWARD -o pppoe -j account_out     # do accounting based on MAC, and set MARK
iptables -t mangle -A FORWARD -j CONNMARK --save-mark     # save IP-level MARK into CONNTRACK table

# Assign unique integer IDs to each host:
iptables -t mangle -A account_out -m mac --mac-source 00:00:5E:01:02:03 -m comment --comment "desktop" -j MARK --set-mark 1
iptables -t mangle -A account_in -m mark --mark 1 -m comment --comment "desktop" -j RETURN

iptables -t mangle -A account_out -m mac --mac-source 00:00:5E:04:05:06 -m comment --comment "desktop" -j MARK --set-mark 2
iptables -t mangle -A account_in -m mark --mark 2 -m comment --comment "desktop" -j RETURN

# repeat for ip6tables

Some things that I figured out while trying to get this to work:

There is a CONNMARK target that allows you to directly save the mark to the conntrack-table. This would allow you to skip the --save-mark rule. This works in iptables, but does not work in ip6tables
restoring a CONNMARK mark to the IP-layer only works in the mangle table.

Exposing these counters to prometheus isn't very difficult, but the resulting script is rather long and ugly:

iptables -t mangle -vnL account_in --exact | sed -n 's%\s*\([0-9]\+\)\s\+\([0-9]\+\)\s\+.*/\* \(.*\) \*/.*%host_packets_total{ipv="4",direction="in",host="\3"} \1\nhost_bytes_total{ipv="4",direction="in",host="\3"} \2%p'
iptables -t mangle -vnL account_out --exact | sed -n 's%\s*\([0-9]\+\)\s\+\([0-9]\+\)\s\+.*/\* \(.*\) \*/.*%host_packets_total{ipv="4",direction="out",host="\3"} \1\nhost_bytes_total{ipv="4",direction="out",host="\3"} \2%p'
ip6tables -t mangle -vnL account_in --exact | sed -n 's%\s*\([0-9]\+\)\s\+\([0-9]\+\)\s\+.*/\* \(.*\) \*/.*%host_packets_total{ipv="6",direction="in",host="\3"} \1\nhost_bytes_total{ipv="6",direction="in",host="\3"} \2%p'
ip6tables -t mangle -vnL account_out --exact | sed -n 's%\s*\([0-9]\+\)\s\+\([0-9]\+\)\s\+.*/\* \(.*\) \*/.*%host_packets_total{ipv="6",direction="out",host="\3"} \1\nhost_bytes_total{ipv="6",direction="out",host="\3"} \2%p'

Cumulative graphs in Prometheus

2020-08-16T00:00:00+02:00

I've been experimenting a bit with Prometheus and Grafana. It allows you to make beautiful graphs from the metric data using relatively simple expressions. But one of the graphs that I had in mind, turned out to be more difficult. These are my notes.

I have a metric that counts the electrical energy my solar panels have produced: energy_out_watthour_total. (There is some debate whether watthour is "right". Some people argue that it should be Joules, but that is not the point of this post.) Since this is a counter (_total suffix), Prometheus does all the magic to compensate if this counter would reset to zero. This also means that the absolute value of the counter is irrelevant, only the changes are meaningful.

If you are only interested in the changes, rate(metric) will give you exactly that: it will calculate the rate of change (for this case, this will yield the average power in Watt between every two datapoints). But I wanted a cumulative view: how much energy have I produced over this day, this week, ... For a given day, this give a sigmoid-like curve: (I have removed the actual values for privacy)

The corresponding power-graph looks like this:

Both graphs show the same information, and for a single day, they don't differ much. But when comparing e.g. today to yesterday, I find a cumulative view much easier to read: are we ahead/behind "schedule" compared to yesterday? Especially when the power is very spikey, I find cumulative graphs much easier to interpret. Take a look for yourself:

PromQL

Next up: getting this idea of a cumulative graph translated into a Grafana dashboard. The basic idea is not too hard. I want to plot energy_out_watthour_total - ${energy_out_watthour_total_at_the_beginning_of_this_graph}. Turns out that last part isn't so easy: There is no easy way to have Grafana query Prometheus for the value at a particular point in time. Or, at least I didn't find that way.

So I reached out to the #prometheus IRC channel on FreeNode. With the help of SuperQ and roidelapluie, we figured out some tools to help us: Grafana exposes a few variables that you can use inside your query. Especially the $__from variable looks interesting for this case.

Next, we figured out a few ways that didn't work:

metric_total - metric_total offset (time() - $__from) doesn't work, since offset requires a constant offset, not something dynamic
scalar(metric_total and on() time() == vector(1597579200)) doesn't work for multiple reasons: you'd need to get the timestamp exactly right, or use a range that covers exactly 1 datapoint. And, since prometheus only evaluates the metrics within the time window that is it about to graph, this may yield no points at all if hte timestamp is outside that range.

What does work is this construction:

metric_total
- max_over_time(
  (
    metric_total and on() vector(time()) < $__from/1000
  )[${__range_s}s:$__interval]
)

The inner and construction gives the metric up until the start of the graph. The range vector selection allows us to gather up data from before the start of the graph. The actual values are less important, as long as it scoops up at least one datapoint. The max_over_time() takes the highest value it sees. Since this is a counter, this effectively means taking the most recent value.

There is an edge case when a reset occurs within this window. We can solve that by using

metric_total
- avg_over_time(
    (
      metric_total and on() vector(time()) >= $__from/1000 < $__from/1000+$__interval_ms/1000
    )[${__range_s}s:$__interval]
  )

This narrows the time-frame that is considered, and uses avg_over_time() to average out the remaining points. Hopefully, there will only one point. The downside of this expression is that it doesn't work for my solar production: during the night, the invertor goes into a sleep-state, and I can't get any metrics from it. This means that, depending on the actual time-window of the graph, the start-time may have no data, failing the query.

Grafana

I've struggled a bit on the finishing touches in Grafana, so I'll note them down here as well. Since solar panels only work during daylight, it makes no sense to waste space on the night hours. I found that plotting from now/d+5h to now/d-90m works best for my location. This plots the current day from 05:00 until 22:30 (local/browser time).

Half-duplex UART (RS485) on ESP32

2020-04-26T00:00:00+02:00

I'm a happy owner of a few ESP32 microcontrollers. These are similar to the Arduino, but have WiFi & Bluetooth built in, which makes them very versatile. These chips come with a lot of peripherals, including three UARTs. These allow offloading the actual bit-banging of the pins to dedicated hardware, freeing up the main processor for more useful work.

Normally, UARTs work in full-duplex mode: they can send and receive simultaneously over separate wires. (Obviously, you can operate it in simplex mode as well, by only sending or receiving.) Half-duplex means that the device can both send and receive, but not simultaneously. This post is about getting this half-duplex mode to work, which was more tricky than it should have been.

Half-duplex is used by RS-485, but also open-drain (or open-collector in bipolar terminology) buses are inherently half-duplex. The ESP's documentation refers to its half-duplex provisions by the RS-485 name.

Challenges
Test setup
Conclusion

Challenges

Half-duplex works just like full duplex when only one device is transmitting at a time. So for simple request-response protocols with only a single pair of endpoints, this usuall doesn't matter. It becomes more challenging when multiple devices may initiate a conversation:

Before starting a transmission, the device should first check if the bus is free. I.e. if there is no-one else transmitting at that time. If the bus is busy, the device should hold off its transmission until it is free.

Even during a transmission, the device should monitor the bus to see if it can see its own transmission correctly. Ideally, it should be able to detect a "collision", and retransmit in that case.

Test setup

To perform the test, I tested a single ESP32 against itself: UART2 is the device under test, UART1 is used as a reference, and kept in full-duplex mode. The half-duplex magic is done by abusing the second core the ESP32 offers: I looped back the Tx and Rx pins to GPIO pins and logically ANDed them:

gpio_set_direction(GPIO_NUM_27, GPIO_MODE_INPUT);  // connected to UART1 Tx
gpio_set_direction(GPIO_NUM_26, GPIO_MODE_INPUT);  // connected to UART2 Tx
gpio_set_direction(GPIO_NUM_25, GPIO_MODE_OUTPUT);  // connected to UART1 & UART2 Rx
for(;;) {
    int level1 = gpio_get_level(GPIO_NUM_27);
    int level2 = gpio_get_level(GPIO_NUM_26);
    // idle level is 1
    // if either Tx asserts 0, output 0
    int level = level1 & level2;
    gpio_set_level(GPIO_NUM_25, level);
}

The test consists of 4 parts (source code):

Receive only
Transmit only
Transmit while bus is busy
Transmit with collision

The first two parts are trivial, but are included for completeness. The third test should reveal if the device waits for the bus to become available before starting a transmission. The forth test will always result in corruption, but the sender should report this.

Full duplex mode

As a reference test, I set up UART2 similar to UART1:

uart_config_t uart_config = {
    .baud_rate = 9600,
    .data_bits = UART_DATA_8_BITS,
    .parity    = UART_PARITY_DISABLE,
    .stop_bits = UART_STOP_BITS_1,
    .flow_ctrl = UART_HW_FLOWCTRL_DISABLE,
};
ESP_ERROR_CHECK(uart_param_config(UART_NUM_2, &uart_config));
ESP_ERROR_CHECK(uart_set_pin(UART_NUM_2, GPIO_NUM_16, GPIO_NUM_17, UART_PIN_NO_CHANGE, UART_PIN_NO_CHANGE));
ESP_ERROR_CHECK(uart_driver_install(UART_NUM_2, 1024, 1024, 0, NULL, 0));

As expected, part 1 & 2 returned the expected data; During test 3, UART2 failed to wait until the bus was free (as expected), which garbled the resulting output: (0) is UART1's output (reference), (1) is UART2's Tx (DUT), (2) is the resulting Rx signal given to both UARTs.

Instead of receiving the "CCCC" (sent by UART1) or the "dd" (sent by UART2), garbage was received:

03 80 a8 e8                                       |....|

Test 4 failed in the expected way as well. UART2 was transmitting "eeee", while UART1 sent "F" simultaniously, colliding in the first byte.

04 65 65 65                                       |.eee|

RS485 modes

The ESP's documentation is very sparse on its half-duplex capabilities. The technical reference manual mentions the "RS485 mode configuration" register, but no further explanation is given. The ESP-IDF programming guide goes in to a bit more detail, describing the different modes to connect to a 485 bus. Its API provides three modes that look interesting: UART_MODE_RS485_HALF_DUPLEX, UART_MODE_RS485_COLLISION_DETECT and UART_MODE_RS485_APP_CTRL.

UART_MODE_RS485_HALF_DUPLEX

In RS485_HALF_DUPLEX mode, contrary to my expectations, nothing happened: test 1 & 2 still passed, test 3 & 4 still failed in a very similar way.

Test 3 failed to wait for the bus to free up, and returned garbage data:

03 80 a8 e8                                       |....|

Test 4 got corrupted as well.

04 65 65 65                                       |.eee|

In this mode, you can use the uart_get_collision_flag() function to see if there was a collision. It returned false for test 1 (as expected), but true for tests 2, 3 and 4 (only 4 was expected).

UART_MODE_RS485_COLLISION_DETECT

This setting paniced. I haven't figured out why (yet).

UART_MODE_RS485_APP_CTRL

RS485_APP_CTRL mode was indistinguishable from HALF_DUPLEX in my tests:

Test 3 failed to wait for the bus to free up, and returned garbage data:

03 80 a8 e8                                       |....|

Test 4 got corrupted as well.

04 65 65 65                                       |.eee|

uart_get_collision_flag() returned false, true, true, true.

Custom RS485 modes

From the tests above, it is clear that neither of the provided modes perform the desired functionality. Time to dig in deeper.

The ESP-IDF SDK is nothing more than a set of provided library functions that abstract away the register manipulations. uart_set_mode() for example manipulates the UART_RS485_CONF_REG register (accessed as UART[uart_num]->rs485_conf in ESP-IDF). The code shows that all three modes activate RS485 mode (UART[uart_num]->rs485_conf.en = 1), but they differ in the settings of the other flags.

What surprised me, is that all three modes set UART_RS485RXBY_TX_EN/rx_busy_tx_en flag: enable the RS-485 transmitter to send data, when the RS-485 receiver line is busy. As soon as that was turned off, things started looking better: UART2 waited for the bus to free up before transmitting its "dd" message, which resulted in the correct sequence being received. The collision flag was still wrong (it indicated collisions in test 2 & 3 as well).

43 43 43 43 64 64                                 |CCCCdd|

Another flag that is available is the UART_RS485TX_RX_EN/tx_rx_en, which will enable the transmitter’s output signal loop back to the receiver’s input signal. The APP_CTRL mode (which I used as a starting point) leaves this flag off. I expected this to suppress receiving my own transmissions, but instead, it returned a 0x00-byte for every byte transmitted. So let's try setting this to 1.

This seems to be the magic combination: Tx waits until the bus is free, and the collision flag yields the expected result (only true for test 4). It only introduces a small problem:

E (1398) uart: uart_get_collision_flag(1568): wrong mode

The uart_get_collision_flag() reads out the collision state as reported by the SDK. And the SDK doesn't track the collisions when in APP_CTRL. But nothing is stopping us to do that ourselves: The hardware sets the UART_RS485_CLASH_INT_RAW bit in the UART_INT_RAW_REG register. This can trigger an actual interrupt if enabled, but we are only interested in finding out if a clash happened, not exactly when it happened. So we can simply read out this register, and clear it before the next transmission:

UART2.int_clr.rs485_clash = 1;  // clear bit
uart_write_bytes(UART_NUM_2, "whatever", 8);
bool collision_happened = UART2.int_raw.rs485_clash;

Conclusion

For my purpose, I wanted the transmision to be kept back if the bus is busy. I also wanted to have accurate collision reporting. The following code sets this up for me:

uart_config_t uart_config = {
    .baud_rate = 9600,
    .data_bits = UART_DATA_8_BITS,
    .parity    = UART_PARITY_DISABLE,
    .stop_bits = UART_STOP_BITS_1,
    .flow_ctrl = UART_HW_FLOWCTRL_DISABLE,
};
ESP_ERROR_CHECK(uart_param_config(UART_NUM_2, &uart_config));
ESP_ERROR_CHECK(uart_set_pin(UART_NUM_2, GPIO_NUM_16, GPIO_NUM_17, UART_PIN_NO_CHANGE, UART_PIN_NO_CHANGE));
ESP_ERROR_CHECK(uart_driver_install(UART_NUM_2, 1024, 1024, 0, NULL, 0));

uart_set_mode(UART_NUM_2, UART_MODE_RS485_APP_CTRL);
UART2.rs485_conf.rx_busy_tx_en = 0;  // don't send while receiving => collision avoidance
UART2.rs485_conf.tx_rx_en = 1;  // loopback (1), so collision detection works

bool did_collision_happen_since_last_check() {
    bool ret = UART2.int_raw.rs485_clash;
    UART2.int_clr.rs485_clash = 1;  // clear bit
    return ret;
}

Raspberry pi 4 as broadband router

2020-02-21T00:00:00+01:00

A few years ago, I already considered using a Raspberry Pi as a broadband router. Back then, its first generation wasn't fast enough. This changed with the 4th generation of the Raspberry pi.

The test setup was similar to the setup I use previously:

Two beefy laptops on the ends
A VLAN-capable switch to connect everything
The Raspberry pi "on a stick" to route traffic from one VLAN to the other

I was happy with the results. Ping latency was 0.834 ±0.075 ms.

$ iperf -c 192.168.23.231 -p 5001 -t 60 -i 5
------------------------------------------------------------
Client connecting to 192.168.23.231, TCP port 5001
TCP window size:  289 KByte (default)
------------------------------------------------------------
[  4] local 192.168.10.20 port 60887 connected with 192.168.23.231 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 5.0 sec   510 MBytes   855 Mbits/sec
[  4]  5.0-10.0 sec   514 MBytes   862 Mbits/sec
[  4] 10.0-15.0 sec   513 MBytes   860 Mbits/sec
[  4] 15.0-20.0 sec   512 MBytes   860 Mbits/sec
[  4] 20.0-25.0 sec   513 MBytes   860 Mbits/sec
[  4] 25.0-30.0 sec   514 MBytes   862 Mbits/sec
[  4] 30.0-35.0 sec   516 MBytes   866 Mbits/sec
[  4] 35.0-40.0 sec   515 MBytes   864 Mbits/sec
[  4] 40.0-45.0 sec   514 MBytes   863 Mbits/sec
[  4] 45.0-50.0 sec   506 MBytes   849 Mbits/sec
[  4] 50.0-55.0 sec   498 MBytes   835 Mbits/sec
[  4] 55.0-60.0 sec   506 MBytes   849 Mbits/sec
[  4]  0.0-60.0 sec  5.99 GBytes   857 Mbits/sec

The bidirectional flows performed worse, as expected. But I had expected a bit better results:

$ iperf -c 192.168.23.231 -d -p 5001 -t 60 -i 5
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: -1.00 Byte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.23.231, TCP port 5001
TCP window size:  273 KByte (default)
------------------------------------------------------------
[  5] local 192.168.10.20 port 60891 connected with 192.168.23.231 port 5001
[  6] local 192.168.10.20 port 5001 connected with 192.168.23.231 port 36352
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0- 5.0 sec  82.2 MBytes   138 Mbits/sec
[  6]  0.0- 5.0 sec   187 MBytes   313 Mbits/sec
[  5]  5.0-10.0 sec  71.4 MBytes   120 Mbits/sec
[  6]  5.0-10.0 sec   163 MBytes   273 Mbits/sec
[  5] 10.0-15.0 sec  75.8 MBytes   127 Mbits/sec
[  6] 10.0-15.0 sec   177 MBytes   296 Mbits/sec
[  6] 15.0-20.0 sec   177 MBytes   297 Mbits/sec
[  5] 15.0-20.0 sec  79.6 MBytes   134 Mbits/sec
[  5] 20.0-25.0 sec  66.0 MBytes   111 Mbits/sec
[  6] 20.0-25.0 sec   187 MBytes   313 Mbits/sec
[  5] 25.0-30.0 sec  70.6 MBytes   118 Mbits/sec
[  6] 25.0-30.0 sec   200 MBytes   335 Mbits/sec
[  6] 30.0-35.0 sec   174 MBytes   291 Mbits/sec
[  5] 30.0-35.0 sec  75.2 MBytes   126 Mbits/sec
[  5] 35.0-40.0 sec  74.0 MBytes   124 Mbits/sec
[  6] 35.0-40.0 sec   185 MBytes   310 Mbits/sec
[  5] 40.0-45.0 sec  65.2 MBytes   109 Mbits/sec
[  6] 40.0-45.0 sec   171 MBytes   287 Mbits/sec
[  5] 45.0-50.0 sec  67.0 MBytes   112 Mbits/sec
[  6] 45.0-50.0 sec   173 MBytes   291 Mbits/sec
[  5] 50.0-55.0 sec  65.0 MBytes   109 Mbits/sec
[  6] 50.0-55.0 sec   163 MBytes   273 Mbits/sec
[  5] 55.0-60.0 sec  69.2 MBytes   116 Mbits/sec
[  5]  0.0-60.0 sec   861 MBytes   120 Mbits/sec
[  6] 55.0-60.0 sec   166 MBytes   279 Mbits/sec
[  6]  0.0-60.0 sec  2.07 GBytes   297 Mbits/sec
[SUM]  0.0-60.0 sec  2.25 GBytes   323 Mbits/sec

Reading CPU temperature and Fan speeds from the command line in macOs 10.14 Mojave

2019-09-27T00:00:00+02:00

I was looking to read out the current temperature and fan speed of my MacBook Pro running macOS 10.14 Mojave. I prefered build-in tools over shady apps. After some searching, I found the powermetrics tool (which needs sudo privileges). By default, it outputs lots of details every 5 seconds:

Machine model: MacBookPro15,1
SMC version: Unknown
EFI version: 220.99.0
OS version: 18G103
Boot arguments:
Boot time: Fri Sep 27 13:23:38 2019



*** Sampled system activity (Fri Sep 27 14:05:02 2019 +0200) (5015.24ms elapsed) ***

*** Running tasks ***

Name                               ID     CPU ms/s  User%  Deadlines (<2 ms, 2-5 ms)  Wakeups (Intr, Pkg idle)
kernel_task                        0      28.82     0.00   0.00    0.00               673.63  212.91
WindowServer                       183    67.37     61.86  29.70   10.57              46.45   15.35
CalendarAgent                      321    32.20     82.65  0.00    0.00               0.20    0.20
[...]
ALL_TASKS                          -2     766.12    80.97  101.09  14.16              1125.97 384.23


**** Battery and backlight usage ****

Backlight level: 791 (range 0-1024)
Keyboard Backlight level: 0 (off 0 on range 32-512)


**** Network activity ****

out: 30.71 packets/s, 11892.96 bytes/s
in:  31.30 packets/s, 18962.61 bytes/s


**** Disk activity ****

read: 1.00 ops/s 4.08 KBytes/s
write: 7.58 ops/s 78.40 KBytes/s

****  Interrupt distribution ****

CPU 0:
    Vector 0x46(SMC): 12.96 interrupts/sec
    Vector 0x72(IGPU): 17.95 interrupts/sec
    Vector 0x74(GFX0): 418.52 interrupts/sec
    Vector 0x75(HDAU): 3.79 interrupts/sec
    Vector 0x76(XHC1): 143.36 interrupts/sec
    Vector 0x79(ANS2): 6.58 interrupts/sec
    Vector 0x86(ARPT): 7.58 interrupts/sec
    Vector 0x8c(IOBC): 0.80 interrupts/sec
    Vector 0xdd(TMR): 195.40 interrupts/sec
    Vector 0xde(IPI): 112.86 interrupts/sec
    Vector 0xdf(PMI): 0.20 interrupts/sec
CPU 1:
    Vector 0xdd(TMR): 1.20 interrupts/sec
    Vector 0xde(IPI): 35.09 interrupts/sec
[...]
CPU 15:
    Vector 0xdd(TMR): 0.60 interrupts/sec
    Vector 0xde(IPI): 4.39 interrupts/sec



**** Processor usage ****

Intel energy model derived package power (CPUs+GT+SA): 9.62W

LLC flushed residency: 44.5%

System Average frequency as fraction of nominal: 118.43% (2723.96 Mhz)
Package 0 C-state residency: 47.56% (C2: 47.56% C3: 0.00% C6: 0.00% C7: 0.00% C8: 0.00% C9: 0.00% C10: 0.00% )

Performance Limited Due to:
CPU LIMIT TURBO_ATTENUATION
CPU LIMIT SPARE_IA_14
CPU/GPU Overlap: 1.38%
Cores Active: 49.10%
GPU Active: 1.51%
Avg Num of Cores Active: 0.83

Core 0 C-state residency: 74.42% (C3: 3.57% C6: 0.00% C7: 70.85% )

CPU 0 duty cycles/s: active/idle [< 16 us: 618.91/176.66] [< 32 us: 148.35/20.34] [< 64 us: 119.44/226.11] [< 128 us: 138.18/270.18] [< 256 us: 155.53/166.69] [< 512 us: 67.20/161.31] [< 1024 us: 40.08/87.73] [< 2048 us: 21.33/90.32] [< 4096 us: 5.78/65.40] [< 8192 us: 1.00/51.24] [< 16384 us: 1.00/1.20] [< 32768 us: 0.20/0.00]
CPU Average frequency as fraction of nominal: 103.26% (2375.02 Mhz)

[...]

CPU 15 duty cycles/s: active/idle [< 16 us: 57.82/4.98] [< 32 us: 0.40/4.39] [< 64 us: 0.60/5.58] [< 128 us: 0.20/5.58] [< 256 us: 0.60/5.98] [< 512 us: 0.20/6.18] [< 1024 us: 0.20/5.78] [< 2048 us: 0.20/1.99] [< 4096 us: 0.40/2.19] [< 8192 us: 0.00/4.59] [< 16384 us: 0.00/3.79] [< 32768 us: 0.00/2.79]
CPU Average frequency as fraction of nominal: 113.15% (2602.45 Mhz)

**** GPU usage ****

GPU 0 name IntelIG
GPU 0 C-state residency: 98.66% (0.08%, 98.58%)
GPU 0 P-state residency: 1200MHz: 0.00%, 1150MHz: 0.00%, 1100MHz: 0.00%, 1050MHz: 0.00%, 1000MHz: 0.00%, 950MHz: 0.00%, 900MHz: 0.00%, 850MHz: 0.00%, 800MHz: 0.00%, 750MHz: 0.00%, 700MHz: 0.00%, 650MHz: 0.00%, 600MHz: 0.00%, 550MHz: 0.00%, 500MHz: 0.00%, 450MHz: 0.00%, 400MHz: 0.00%, 350MHz: 0.62%
GPU 0 average active frequency as fraction of nominal (350.00Mhz): 100.00% (350.00Mhz)
GPU 0 HW average active frequency   : 0.00
GPU 0 GPU Busy                      : 1.34%
GPU 0 DC6 Residency                 : 0.00%
GPU 0 [PSR] GPU + TCON are Off      : 0.00%
GPU 0 [PSR] Only GPU is On          : 100.00%
GPU 0 [PSR] Only TCON is On         : 0.00%
GPU 0 [PSR] GPU + TCON are On       : 0.00%
GPU 0 [PSR] StateMachine Bypass     : 100.00%
GPU 0 [PSR] StateMachine FIFO       : 0.00%
GPU 0 [PSR] StateMachine Others     : 0.00%
GPU 0 DPB Strong On                 : 0.00%
GPU 0 DPB Weak On                   : 0.00%
GPU 0 PPFM on                       : 0.00%
GPU 0 Throttle High Priority(%): 0
GPU 0 Throttle NormalHi Priority(%): 0
GPU 0 Throttle Normal Priority(%): 0
GPU 0 Throttle Low Priority(%): 0
GPU 0 Slice switch                  : 0 (0.00/second)
GPU 0 DC6 Exit Reason - Flip: 0 (0.00/second)
GPU 0 DC6 Exit Reason - Register: 0 (0.00/second)
GPU 0 DC6 Exit Reason - Gamma: 0 (0.00/second)
GPU 0 DC6 Exit Reason - Interrupt: 0 (0.00/second)
GPU 0 DC6 Exit Reason - Cursor: 0 (0.00/second)
GPU 0 DC6 Exit Reason - Render: 0 (0.00/second)
GPU 0 FB Test Case 0


**** SMC sensors ****

CPU Thermal level: 19
GPU Thermal level: 0
IO Thermal level: 0
Fan: 2001.43 rpm
CPU die temperature: 65.77 C
GPU die temperature: 58.00 C
CPU Plimit: 0.00
GPU Plimit: 0.00
Number of prochots: 0

For scripted execution, you can limit the output in several ways. If you are only interested in the current temperature and fan speed, sudo powermetrics -n 1 -i 1 --samplers smc returns these without 5 seconds of waiting.

404to302 Lambda@Edge

2019-09-22T00:00:00+02:00

I discovered the 404to302 concept after a tweet by Aral Balkan, and immediately liked the idea. It solves a hard problem (keeping permalinks alive and working) in a fairly easy way: simply archive the old site, and avoid re-using the same permalink again. If you permalinks contain a date of some sort, this is very easy.

Since I've moved to host my static site on Amazon S3, I needed to add this functionality myself. Luckily, Amazon CloudFront allows you to run custom code when handling a request: Lambda@Edge. For this 404to302 concept, I hooked in to the "origin response" phase, inspecting every response for possible 404's. If the response is a 404, it looks up if a fallback URL is configured on the CloudFront distribution for this request, and returns a 302 to that location if present.

The code is available on GitHub.

Switching to a static website

2019-09-21T00:00:00+02:00

After over 12 years of using WordPress, I got tired of keeping up-to-date with all the patches. Wordpress itself was fairly easy to keep up-to-date thanks to the auto-updater. But it doesn't stop there: PHP, MySQL, Apache, ...

So I reconsidered what I used my blog for. It's mostly a way for me to document my experiments, hoping that the information would be useful for someone. Often, that someone is me, a few years down the line. While I did appreciate some useful comments now and then, the fight against comment-spam was tiring as well. So I'm not too sad that I'd loose that functionality.

Which brings me to picking a static site generator. Unlike the "dynamic" CMSes, there are a lot more players in the "static site generation" market. I explored different scenario's:

At first, I wanted to import the entire archive of my WordPress blog into the new tool. That turns out to be close to impossible. There are tools that claim to do so, but it never worked out: Images being linked to the wrong URL, layout shifting all around, ...

I was about to admit defeat when I discovered the 404to302 idea. The idea is as simple as it is briliant: modify the webserver to convert a 404 (File not found) into a 302 (redirect) to the previous version of the site. This way, you no longer need to import all previous content, but all old (perma)links simply keep on working (albeit with an additional redirect step)! This removed one big item from my requirement list: being able to import the previous content.

I ended up picking Pelican. It's slightly less popular compared to Jekyll and Hugo, but it's written in a language that I have experience in: Python. I initially tried to get Jekyll to work, but failed to get all required Gems installed on my system.

Receiving NOAA images with RTL-SDR

2019-08-25T00:00:00+02:00

During summer brake, I set out for a tech adventure together with my son. We wanted to do weather predictions ourselves. Or at least pretend to do so.

What sparked the idea, was a beautiful tweet from rtl-sdr.com. Receiving GOES satellites was a bit of a high bar for a first attempt, so I started out with receiving NOAA images. Contrary to the GOES satellites, NOAA's are in Low Earth Orbit and have an analogue downlink at 137MHz, which does not require a dish and amplifier to receive.

I followed the guide I found in the NOAA_APT GitHub repo. We started out by building a very simple V-dipole antenna out of some leftover 1.5mm² electrical wiring. A V-dipole is not the best antenna to receive a circular polarized signal, but let's first see if we get this to work. I connected the two dipole wires to a short piece of coax to connect it to the Belling-Lee connector that fits my RTL-SDR dongle.

There used to be a lot of NOAA-satellites active, but as of this writing, only three satellites remain in service: NOAA-15, NOAA-18 and NOAA-19. These satellites are in polar orbits around the earth, completing one revolution every 102 minutes. But since the earth is rotating underneath them, they fly over different parts of the planet every time. Figuring out when the satellite will be overhead requires a degree in orbital mechanics. Or the use of some software, such as gpredict. It will calculate where the satellites are at this moment (or an arbitrary moment), and tell you when the next pass will be at your location (or any location, for that matter). It also plots the "ground track" of the satellites, which shows that the satellite visits different parts of the planet on every orbits.

To increase my odds, I picked a pass with very high elevation. The higher the satellite is above the horizon, the stronger its signal can be received. And since I had no idea what to expect, I could use every advantage available. gpredict predicted a pass with 85º elevation for today, so I made sure I was ready to receive "something": Antenna oriented for a North-South trajectory, flat on a plastic table in our garden. Things go quick: in 15 minutes, the satellite rises above the horizon, flies overhead, and disappears on the other horizon.

I had GQRX open, tuned at 137.872500MHz, 300kHz bandwidth. I figured that I'd want the signal a bit away from DC to avoid trouble demodulating. NOAA-18 transmits at 137.9125MHz, with an RF bandwidth of 34kHz, which gives me 23kHz room above DC to accommodate any callibration issues and doppler shift. I enabled the FM demodulator, but all I heard was noise. But the waterfall already showed some promise.

A bit of patience allowed the satellite to climb higher above the horizon, and that yielded a stronger signal. I could hear the expected 2.4kHz beeping and "tic-toc" sound of the sync signals.

The next step was a bit of a disappointment. I wanted to demodulate the FM signal sitting at ~40kHz to baseband using a command line tool. But I didn't find any tool to do this. So I took the easy way out and simply replayed the recording in GQRX, recording the demodulated output to a WAV-file. One note if you intend to use WXtoImg: GQRX records in 16bit 48kHz, while WXtoImg requires 11kHz. Luckily, NOAA_APT is happy to accept pretty much anything.

The result was more impressive that I expected. It was a relatively cloudless day, so you could clearly see the coastlines of Denmark and Great Britain, especially on the Infrared channel (right image). Based on the description of the APT signal, the image contains one visible spectrum image (channel 1, 580nm - 680nm, left), and an infrared image (channel 4, 10.30µm - 11.30µm, right).