/ or using full disk encryption. The only requirement was to map the disks to regular QEMU devices, such as -hda /dev/sda arguments. I don’t remember exactly when I first did that, but there are references to this process in their community forums dating back to at least 2020. I might have seen someone mention it in a blog post a few years earlier.
Since I usually opt for their cheaper line of servers, from Kimsufi or SoYouStart, I’m used to old hardware. The server I’m currently replacing, for instance, is an Intel Xeon E3-1245 V2 from 2012 (!) with regular SATA SSDs, where the BIOS boots from the MBR. What happens is that the same custom OS installation procedure doesn’t work for newer servers, which use UEFI and are powered by NVMe disks. I tried multiple combinations of:
-drive file=/dev/nvme0n1 and -device nvme.-bios /usr/share/ovmf/OVMF.fd, using the OVMF firmware.And probably a few other combinations I don’t recall. What’s important is that none of them worked. Every time I checked the boot logs via KVM/IPMI, I got an error like: rEFInd - Chain on hard drive failed. Next. I kept wondering what could be missing, but more important than doing the installation exactly as I wanted was achieving the final goal. I didn’t want to install an unsupported OS. I only wanted a Debian installation with full disk encryption, which isn’t supported by the OVH web-based OS installer. That’s when I started looking into how to encrypt an existing Linux installation.
There are guides like Encrypt an existing Debian 12 system with LUKS, which aren’t exactly wrong but are overly complicated and contain unnecessary steps. Every time I see a complicated guide, I wonder how to simplify the process. Starting from that and with the always-on-point instructions from the Arch Linux Wiki, I was able to encrypt an existing Debian installation.
The process goes like this:
Do a regular Debian 13 (Trixie) installation using the OVH web installer. The most important part is to keep /boot separate from other partitions and on RAID-1 if you’re using RAID. It will also create the ESP partition separately. The goal is to never need to touch these two partitions.
After the installation is finished, log in to the new system and install the required packages to boot it. The dropbear-initramfs package is what allows us to unlock the encrypted root partition via SSH. For it to work, you need to set its own authorized_keys file, since it won’t have access to anything on disk before unlocking.
$ sudo apt install cryptsetup-initramfs dropbear-initramfs
(...)
$ sudo vim /etc/dropbear/initramfs/authorized_keys
Configure the server to boot in rescue mode and restart it. After logging in again, check the current partitions to identify the data partition to be encrypted.
# lsblk | grep -v nbd
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme1n1 259:0 0 419.2G 0 disk
├─nvme1n1p1 259:1 0 511M 0 part
├─nvme1n1p2 259:2 0 1G 0 part
│ └─md2 9:2 0 1022M 0 raid1
└─nvme1n1p3 259:3 0 417.7G 0 part
└─md3 9:3 0 835.1G 0 raid0
nvme0n1 259:4 0 419.2G 0 disk
├─nvme0n1p1 259:5 0 511M 0 part
├─nvme0n1p2 259:6 0 1G 0 part
│ └─md2 9:2 0 1022M 0 raid1
├─nvme0n1p3 259:7 0 417.7G 0 part
│ └─md3 9:3 0 835.1G 0 raid0
└─nvme0n1p4 259:8 0 2M 0 part
In this case, we are interested in /dev/md3, the root partition on top of RAID-0. Check the filesystem for errors so that other tools don’t complain about it later.
# e2fsck -f /dev/md3
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
root: 31979/54730752 files (1.1% non-contiguous), 3972359/218920960 blocks
Now comes a very important part: shrink the filesystem but not the whole partition. The goal is to leave space for the LUKS header to be created at the end of the partition.
# resize2fs -p -M /dev/md3
resize2fs 1.47.0 (5-Feb-2023)
Resizing the filesystem on /dev/md3 to 957980 (4k) blocks.
Begin pass 2 (max = 512733)
Relocating blocks XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Begin pass 3 (max = 6681)
Scanning inode table XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Begin pass 4 (max = 3388)
Updating inode references XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
The filesystem on /dev/md3 is now 957980 (4k) blocks long.
Next, perform the actual encryption. This is also where you set the passphrase to unlock it. This will take time, depending on the disk speed and partition size, as the process rewrites everything, including unused/free space. In this case, it took a little over 25 minutes to encrypt the 835GB partition.
# cryptsetup reencrypt --encrypt --reduce-device-size 32M /dev/md3
WARNING!
========
This will overwrite data on LUKS2-temp-03ea47d5-415b-4622-9510-5b7340d6c557.new irrevocably.
Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for LUKS2-temp-03ea47d5-415b-4622-9510-5b7340d6c557.new:
Verify passphrase:
Finished, time 25m20s, 835 GiB written, speed 562.6 MiB/s
When encryption finishes, open it as a regular device and expand the filesystem to fill the partition again. This will use all available space minus what was reserved for the LUKS header.
# cryptsetup open /dev/md3 md3_crypt
Enter passphrase for /dev/md3:
# resize2fs /dev/mapper/md3_crypt
resize2fs 1.47.0 (5-Feb-2023)
Resizing the filesystem on /dev/mapper/md3_crypt to 218916864 (4k) blocks.
The filesystem on /dev/mapper/md3_crypt is now 218916864 (4k) blocks long.
Mount the device and update both /etc/crypttab and /etc/fstab. The former should refer to the filesystem UUID, but the latter can just point to the /dev/mapper/md3_crypt device, since it will use the name defined in crypttab.
# mount /dev/mapper/md3_crypt /mnt/
# blkid | grep /dev/md3
/dev/md3: UUID="a9c05676-6aa5-4a7b-acc9-a5eca5de4fed" TYPE="crypto_LUKS"
# cat /mnt/etc/crypttab
# <target name> <source device> <key file> <options>
md3_crypt UUID=a9c05676-6aa5-4a7b-acc9-a5eca5de4fed none luks,discard
# cat /mnt/etc/fstab
/dev/mapper/md3_crypt / ext4 defaults 0 1
UUID=5bc9cb1c-6607-429d-9219-3675ee773b12 /boot ext4 defaults 0 0
LABEL=EFI_SYSPART /boot/efi vfat defaults 0 1
The last step is to bind-mount the system directories, plus the actual /boot, and update the initramfs again in the chroot. This is required for two reasons:
authorized_keys file defined earlier in this process.crypttab file, which tells it what to unlock during boot.# mount --bind /dev/ /mnt/dev/
# mount --bind /proc/ /mnt/proc/
# mount --bind /sys/ /mnt/sys/
# mount /dev/md2 /mnt/boot/
# chroot /mnt/
# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-6.12.43+deb13-amd64
After that, exit the chroot, reboot, and log in via SSH as root once your machine is online and responding to pings. Then, after running cryptroot-unlock and entering the proper passphrase, the machine will mount the encrypted device and proceed with the boot.
To unlock root partition, and maybe others like swap, run `cryptroot-unlock`.
BusyBox v1.37.0 (Debian 1:1.37.0-6+b3) built-in shell (ash)
Enter 'help' for a list of built-in commands.
~ # cryptroot-unlock
Please unlock disk md3_crypt:
cryptsetup: md3_crypt set up successfully
Finally, the machine should now be online and running with full disk encryption, except for /boot and the ESP partition.
$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme1n1 259:0 0 419.2G 0 disk
├─nvme1n1p1 259:1 0 511M 0 part /boot/efi
├─nvme1n1p2 259:2 0 1G 0 part
│ └─md2 9:2 0 1022M 0 raid1 /boot
└─nvme1n1p3 259:3 0 417.7G 0 part
└─md3 9:3 0 835.1G 0 raid0
└─md3_crypt 253:0 0 835.1G 0 crypt /
nvme0n1 259:4 0 419.2G 0 disk
├─nvme0n1p1 259:5 0 511M 0 part
├─nvme0n1p2 259:6 0 1G 0 part
│ └─md2 9:2 0 1022M 0 raid1 /boot
├─nvme0n1p3 259:7 0 417.7G 0 part
│ └─md3 9:3 0 835.1G 0 raid0
│ └─md3_crypt 253:0 0 835.1G 0 crypt /
└─nvme0n1p4 259:8 0 2M 0 part
The process may be a bit more involved than installing the OS from scratch using built-in encryption options. Still, it’s not too complex, provided you don’t skip any steps. It’s definitely better than running a machine outside of your physical reach that stores everything in plain text.
]]>To be honest, the process was a bit daunting. Some articles suggest that there’s no hope unless you switch to a browser supported by a company that doesn’t rely on ads - something that doesn’t really exist. In situations like this, I tend to try the simplest, most straightforward solution first before diving into more complex, over-the-top options. In this case, I decided to switch to the Manifest V3-based extension from the same developers: uBlock Origin Lite.
The experience was actually better than I expected. Even in “Basic” mode, which doesn’t require permission to read or change data on all sites, it worked quite well with the default filters. I did notice some empty ad boxes, as the page layouts aren’t reworked, but that’s something I’m already used to when visiting sites on mobile with AdGuard (doing the block via DNS). No need to change to the “Optimal” mode for now.
In fact, enabling the “Overlay Notices” and “Other Annoyances” filter lists made the experience even more pleasant than before. There are no more “please disable your ad-block” or “please donate to this site using Google” overlays to disrupt my browsing. So while the transition from V2 to V3 may have been a hassle for extension developers, as an end-user, I can’t say I’m unhappy with it. I’m still experiencing the (mostly) ad-free internet I was used to.
]]>I’ve been a Dropbox user for nearly 15 years. It really simplified my approach to backups for personal stuff: a cloud-synced folder on my laptop where I put everything that’s not already on a cloud service. Accidentally deleted something? I can just go to their web app and restore it. The problem is that I don’t want it offering a read-write version of this folder on every device I have. Sometimes I just need a temporary folder to drop a screenshot from my Windows gaming machine so I can access it from my phone.
Resilio Sync (previously known as BitTorrent Sync) is what I was using for that. It has a few problems, including being super slow even after configuring everything possible to bypass its relay servers (spoiler: it doesn’t). Plus, it doesn’t have cloud-backed storage, so I ran an instance of it on a server to have an always-on copy of the files there. Not exactly a drop-in replacement for Dropbox, but it was still useful until I realized I was completely unhappy with its performance.
Things were reaching a point where I was considering self-hosting Nextcloud (fork of the original ownCloud) just for its file-syncing feature or even writing my own cloud-folder synchronization tool backed by S3-compatible storage. That’s when it clicked: I realized I don’t need real-time syncing for the simple use case of easily sharing single files from a computer to my phone. I just needed a way to access a Cloudflare R2 bucket from a mobile app.
After looking around, asking ChatGPT and Perplexity, I settled on S3 Files. I can upload a file from Windows using WinSCP or from a macOS/Linux terminal using s3cmd. Each machine uses a fine-grained access key I can revoke if needed. The S3 API is ubiquitous. I just needed a mobile app to access it when I’m away from my computer. It offered me the ease, speed, availability, and robustness of the cloud, which are miles ahead compared to self-hosting anything.
]]>But it doesn’t need to always be like that. Not every piece of advice or knowledge I’d like to share needs to be in a long blog article format. Sometimes I discover something useful, either through my own exploration or based on someone else’s experience, and I share it with close friends with a small comment of a few sentences. This happens on Slack workspaces, WhatsApp groups or even in direct messages. In the end, I thought: what if I write this for the wider internet and share the link with the same people, instead of writing directly to them?
Based on the concept of “blogmarks”, where small blog posts are used to share links that, in a distant past, would be bookmarked to del.icio.us, I started Myhro Notes. There’s usually one or two short paragraphs adding context, explaining why the link is interesting. They are listed by date, like a blog, but only the title is visible on the home page. The posts themselves will eventually be available on search engines. I expect my future self to be one of its users looking for content posted there.
]]>kubectl.
But dealing with multiple runtimes or packages is just one piece of the equation in the grand schema of cleanly handling dependencies. Sometimes you have to worry about the versions of the external services a project makes use of, like a database or cache system. This is also a solved problem since Docker came into the picture over 10 years ago. I remember that the Docker Compose mantra when it launched (still called Fig) was: “no more installing Postgres on your laptop!”. This is just a bit more complicated when you don’t use the original Docker implementation, but another container management system, like Podman.
Podman offers several advantages over Docker. It can run containers without requiring root access; it doesn’t depend on a service daemon running all the time; and it doesn’t require a paid subscription depending on the usage. It’s a simpler tool, with an almost 1:1 compatible UI overall. The main difference is that it doesn’t seamlessly handle containers with --restart flags. I mean, of course it does restart containers when their processes are interrupted, but they won’t be brought up after a host reboot - which tends to happen from time to time in a workstation.
When looking into how to solve this problem, I realised that the podman generate command can create systemd service unit files. So instead of tinkering about how to integrate the two tools, figuring the file syntax and functionality, it’s possible to just create a new systemd service of the desired container as if it were any other program/process. And the best part is that we can still do that without root, thanks to systemd user services.
$ podman create --name redis -p 6379:6379 docker.io/redis:7
Trying to pull docker.io/library/redis:7...
(...)
$ mkdir -p ~/.config/systemd/user/
$ podman generate systemd --name redis > ~/.config/systemd/user/redis.service
To increase the service’s reliability, it’s preferable to drop the PIDFile line from the configuration file. It typically looks like:
PIDFile=/run/user/1000/containers/vfs-containers/c1b1c3e5dba5368c29ada52a638378e5fec74e1a62e913919528b9c3846c14bb/userdata/conmon.pid
This ensures that even if the container is recreated, like when updating its image, systemd won’t be referencing its older ID, as it will only care about its name. This can be done programmatically with sed:
$ sed -i '/PIDFile/d' ~/.config/systemd/user/redis.service
The generated file should be similar to:
# container-redis.service
# autogenerated by Podman 4.3.1
# Sat Dec 23 17:18:01 -03 2023
[Unit]
Description=Podman container-redis.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/run/user/1000/containers
[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStart=/usr/bin/podman start redis
ExecStop=/usr/bin/podman stop \
-t 10 redis
ExecStopPost=/usr/bin/podman stop \
-t 10 redis
Type=forking
[Install]
WantedBy=default.target
The final step consists in starting the service and enabling it to launch on boot:
$ systemctl --user start redis
$ systemctl --user enable redis
Created symlink /home/myhro/.config/systemd/user/default.target.wants/redis.service → /home/myhro/.config/systemd/user/redis.service.
$ systemctl --user status redis
● redis.service - Podman container-redis.service
Loaded: loaded (/home/myhro/.config/systemd/user/redis.service; enabled; preset: enabled)
Active: active (running) since Sat 2023-12-23 17:25:40 -03; 13s ago
Docs: man:podman-generate-systemd(1)
Tasks: 17 (limit: 37077)
Memory: 11.1M
CPU: 60ms
CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/redis.service
├─46851 /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -e 3 -r 4 --netns-type=path /run/user/1000/netns/netns-102ff957-157c-adcb-bd4a-45e7e0d2a50f tap0
├─46853 rootlessport
├─46859 rootlessport-child
└─46869 /usr/bin/conmon --api-version 1 -c c1b1c3e5dba5368c29ada52a638378e5fec74e1a62e913919528b9c3846c14bb -u c1b1c3e5dba5368c29ada52a638378e5fec74e1a62e913919528b9c3846c14bb -r /usr/bin/crun -b /home/myhro/.local/share/(...)
Dec 23 17:25:40 leptok redis[46869]: 1:C 23 Dec 2023 20:25:40.071 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
Dec 23 17:25:40 leptok redis[46869]: 1:M 23 Dec 2023 20:25:40.072 * monotonic clock: POSIX clock_gettime
Dec 23 17:25:40 leptok redis[46869]: 1:M 23 Dec 2023 20:25:40.072 * Running mode=standalone, port=6379.
Dec 23 17:25:40 leptok redis[46869]: 1:M 23 Dec 2023 20:25:40.072 * Server initialized
Dec 23 17:25:40 leptok redis[46869]: 1:M 23 Dec 2023 20:25:40.072 * Loading RDB produced by version 7.2.3
Dec 23 17:25:40 leptok redis[46869]: 1:M 23 Dec 2023 20:25:40.072 * RDB age 18 seconds
Dec 23 17:25:40 leptok redis[46869]: 1:M 23 Dec 2023 20:25:40.072 * RDB memory usage when created 0.83 Mb
Dec 23 17:25:40 leptok redis[46869]: 1:M 23 Dec 2023 20:25:40.072 * Done loading RDB, keys loaded: 0, keys expired: 0.
Dec 23 17:25:40 leptok redis[46869]: 1:M 23 Dec 2023 20:25:40.072 * DB loaded from disk: 0.000 seconds
Dec 23 17:25:40 leptok redis[46869]: 1:M 23 Dec 2023 20:25:40.072 * Ready to accept connections tcp
In summary, I quite liked how easy it was to leverage the strengths of both the Podman and systemd in their integration. Being able to do that in a rootless way is definitely a huge plus. Before doing that, I always believed that managing Linux services was a root-only thing. But now that I think about it, I realize that when Docker was the only game in town, managing containers also required elevated privileges. I’m glad that we are moving away from this idea, piece by piece.
]]>Forget about endless searching for the perfect router and scrolling through an eternity of reviews and specifications! We have created a single affordable home access point that has all the features you might need for years to come.
A highly configurable router, with gigabit wired networking and dual-band 2.4 and 5 GHz Wi-Fi at an affordable price (abroad, where it costs US$ 99, not here where it costs R$ 800)? It seemed like exactly what I was looking for.
After spending 3 hours configuring the wired router, I thought to myself: “Ah, now that I know how MikroTik works, it’ll be easy. 15 or 20 minutes and everything will be working.” What a mistake. The interface is literally the same with an additional Wireless option in the menu, but even setting a password on the unprotected Wi-Fi network was challenging. I really scratched my head trying to understand how things worked and spent another 2 hours configuring it in the way I wanted.
Configuring the 5 GHz Wi-Fi transmission, in particular, was quite difficult. It has a “radar detection” system to use higher frequencies (5.5-5.6 GHz) that takes literally 10 minutes (!) on each boot to decide which one to use, time in which the wireless network remains unavailable during this process. To avoid frustration, I manually chose a lower frequency option (5.1-5.2 GHz).
After everything was configured, I noticed that the 5 GHz Wi-Fi signal was weaker in other rooms than it used to be with my TP-Link router. Weak enough for iOS to automatically fall back to 4G. Along with the weaker signal came a drop in connection speed. On my MacBook, it fell from 400 to 200 Mbps, and on my iPhone from 200 to 100 Mbps, both measured at Fast.com in other rooms, compared to the TP-Link router I intended to replace. Although that would be sufficient bandwidth to cover most of my use cases, it seemed unacceptable to downgrade the speed I was used to, given the price and quality I expected from the device.
The solution was to go back to a setup identical to the wired MikroTik: connecting the TP-Link router to the new MikroTik and using only the Wi-Fi from the former. In router mode, the speed loss was the same. In access point mode, I achieved the same speed as before when connecting the TP-Link directly to the modem or the wired MikroTik.
It wasn’t a very wise decision to buy a more expensive model because of Wi-Fi and ultimately not use it, but the experience was valuable. It still solves my Dual WAN support issue, albeit in a less than ideal way, and I could return the borrowed MikroTik. I couldn’t exactly pinpoint why its 5 GHz network was so much slower than the TP-Link, but I’ve encountered similar situations caused by software (as the same has happened to me with DD-WRT) in a not-so distant past. It’s not what I expected from MikroTik, a device whose software is precisely its selling point, but who knows. Today, if I were to set up the same system without having the TP-Link router available, I would get a simpler wired MikroTik and connect a Unifi AP to it. It would be the best of both worlds and the cost would be virtually the same.
]]>Terminal).192.168.88.1) was associated with. This made it completely lose connectivity and there was no way to access its web UI ever again. Had to reset it to restore access after that.Quick Set offers very few options for people who want to configure the router as quick as possible. And the same time, it does too much magic under the hood, resulting in possible headaches in the future. I don’t recommend using it.WebFig, its web GUI, I scratched my head to understand how to access this interface after connecting through a Wi-Fi router, which I had to use given it only offers wired connections.
192.168.0.0/24 range, I wasn’t able to access the router at 192.168.88.1, even though the Wi-Fi router could reach it. I was only able to access it after manually adding the 192.168.0.0/16 range as allowed to the admin user. When using Quick Set this isn’t needed, as this configuration is done without ever informing the user.WebFig interface, like changing the MAC address of the ethernet ports directly. At the same time the Quick Set does this (maybe via CLI in the background), suggesting that this is indeed possible. A workaround for that is to create a single-port bridge and change the MAC address of this virtual interface.It’s not a router that I would recommend for the faint of heart nor people who are not ready to face a few frustrations. Even I, being someone used to configure routers even before I was 15 years old, scratched my head to understand how a few things work and spent at least 3 hours to leave it as close as possible from what I wanted. Even though, I wasn’t able to configure the Dual-WAN option with a stand-by connection that is automatically activated when the main one goes down. Via the web UI this didn’t work right and via CLI it looked like too much of a hassle.
In the end I was able to manage both connections manually, accessing the WebFig and deactivating one while re-activating the other. This is still better than physically switching the cable from one modem to the other, and still having to access the configuration page to switch between DHCP and PPPoE in the Wi-Fi router solely WAN port.
I’m considering trying out a MikroTik Wi-Fi router, given that having fewer devices involved might simplify the setup. Having one less device connected to the uninterruptible power supply will also probably improve its autonomy.
]]>Get usage report button in the project/organization billing page. The only problem is that the generated report is a CSV file, shifting the responsibility of filtering and visualizing data to the user. While it’s true that most of the users of this report are used to deal with CSV files, be them developers or accountants experts in handling spreadsheets, this is definitely not the most user-friendly way of offering insights into billing data.
When facing this issue, at first I thought about using harelba/q to query the CSV files directly in the command line. The problem is that q isn’t that straightforward to install, as apparently it is not available via apt nor pip, nor one is able to easily change the data once it’s imported, like in a regular database. In the first time I resorted to create a database on PostgreSQL and import the CSV file into it, but after that I never remember the CSV import syntax and it still requires a daemon running just for that. I kept thinking that there should be a simpler way: what if I use SQLite for that?
In order to not have to CAST() each TEXT column whenever working with dates or numbers, the following schema.sql can be used:
CREATE TABLE billing (
date DATE,
product TEXT,
repository TEXT,
quantity NUMERIC,
unity TEXT,
price NUMERIC,
workflow TEXT,
notes TEXT
);
After that, it’s possible to import the CSV file with the sqlite3 CLI tool. The --skip 1 argument to the .import command is needed to avoid importing the CSV header as data, given that SQLite considers it to be a regular row when the table already exists:
$ sqlite3 github.db
SQLite version 3.36.0 2021-06-18 18:58:49
Enter ".help" for usage hints.
sqlite> .read schema.sql
sqlite> .mode csv
sqlite> .import --skip 1 c2860a05_2021-12-23_01.csv billing
sqlite> SELECT COUNT(*) FROM billing;
1834
Now it’s easy to dig into the billing data. In order to have a better presentation, .mode column can be enabled to both show the column names and align their output. We can, for instance, find out which workflows consumed most minutes in the last week and their respective repositories:
sqlite> .mode column
sqlite> SELECT date, repository, workflow, quantity FROM billing WHERE date > date('now', '-7 days') AND product = 'actions' ORDER BY quantity DESC LIMIT 5;
date repository workflow quantity
---------- ----------------- ------------------------------------ --------
2021-12-21 contoso/api .github/workflows/main.yml 392
2021-12-18 contoso/terraform .github/workflows/staging-images.yml 361
2021-12-22 contoso/api .github/workflows/main.yml 226
2021-12-21 contoso/api .github/workflows/qa.yml 185
2021-12-20 contoso/api .github/workflows/main.yml 140
Another important example of the data that can be fetched is the cost per repository in the last week, summing the cost of all their workflows. An UPDATE statement is required to apply a small data fix, given that the CSV contains a dollar sign $ in the rows of the price column that needs to be dropped:
sqlite> UPDATE billing SET price = REPLACE(price, '$', '');
sqlite> SELECT repository, SUM(quantity) * price AS amount FROM billing WHERE date > date('now', '-7 days') AND product = 'actions' GROUP BY repository;
repository amount
------------------ ------
contoso/api 11.68
contoso/public-web 0.128
contoso/status 1.184
contoso/terraform 2.92
contoso/webapp 0.6
Not intuitive as a web page where one can just click around to filter and sort a report, but definitely doable. As a side note, one cool aspect of SQLite is that it doesn’t require a file database do be used. If started as sqlite3, with no arguments, all of it’s storage needs are handled entirely in memory. This makes it even more interesting for data exploration cases like these, offering all of its queries capabilities without ever persisting data to disk.
iptables command wasn’t available. While it’s known for at least 5 years that this was going to happen, it still took me some time to let the idea of its deprecation sink and actually digest the situation. I scratched my head a bit wondering if the day I would be obliged to learn how to use nftables had finally came.
While looking for some guidance on what are the best practices to manage firewall rules these days, I found the article “What to expect in Debian 11 Bullseye for nftables/iptables”, which explains the situation in a straightforward way. The article ends up suggesting that firewalld is supposed to be the default firewall rules wrapper/manager - something that is news to me. I never met the author while actively working on Debian, but I do know he’s the maintainer of multiple firewall-related packages in the distribution and also works on the netfilter project itself. Based on these credentials, I took the advice knowing it came from someone who knows what they are doing.
A fun fact is that the iptables package is actually a dependency for firewalld on Debian Bullseye. This should not be the case on future releases. After installing it, I went for the simplest goal ever: block all incoming connections while allowing SSH (and preferably Mosh, if possible). Before doing any changes, I tried to familiarize myself with the basic commands. I won’t repeat what multiple other sources say, so I suggest this Digital Ocean article that explains firewalld concepts, like zones and rules persistency.
In summary, what one needs to understand is that there are multiple “zones” within firewalld. Each one can have different sets of rules. In order to simplify the setup, I checked what was the default zone, added the network interface adapter to it and defined the needed rules there. No need for further granularity in this use case. Here, the default zone is the one named public:
$ sudo firewall-cmd --get-default-zone
public
$ sudo firewall-cmd --list-all
public
target: default
icmp-block-inversion: no
interfaces:
sources:
services: dhcpv6-client ssh
ports:
protocols:
forward: no
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
Knowing that, it was quite simple to associate the internet-connected network interface to it and update the list of allowed services. dhcpv6-client is going to be removed because this machine isn’t on an IPv6-enabled network:
$ sudo firewall-cmd --change-interface eth0
success
$ sudo firewall-cmd --add-service mosh
success
$ sudo firewall-cmd --remove-service dhcpv6-client
success
It’s important to execute sudo firewall-cmd --runtime-to-permanent after confirming the rules where defined as expected, otherwise they would be lost on service/machine restarts:
$ sudo firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources:
services: mosh ssh
ports:
protocols:
forward: no
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
$ sudo firewall-cmd --runtime-to-permanent
success
A side effect of the target: default setting is that it REJECTs packets by default, instead of DROPing them. This basically informs the client that any connections were actively rejected instead of silently dropping the packets - the latter which might be preferable. It’s confusing why it’s called default instead of REJECT, and also not clear if it’s actually possible to change the default behavior. In any case, it’s possible to explicitly change it:
$ sudo firewall-cmd --set-target DROP --permanent
success
$ sudo firewall-cmd --reload
success
The --set-target option requires the --permanent flag, but it doesn’t apply the changes instantly, requiring them to be reloaded.
An implication of dropping everything is that ICMP packets are blocked as well, preventing the machine from answering ping requests. The way this can be configured is a bit confusing, given that the logic is flipped. There’s a need to enable icmp-block-inversion and add (which in practice would be removing it) an ICMP block for echo-request:
$ sudo firewall-cmd --add-icmp-block-inversion
success
$ sudo firewall-cmd --add-icmp-block echo-request
success
The result will look like this, always remembering to persist the changes:
$ sudo firewall-cmd --list-all
public (active)
target: DROP
icmp-block-inversion: yes
interfaces: eth0
sources:
services: mosh ssh
ports:
protocols:
forward: no
masquerade: no
forward-ports:
source-ports:
icmp-blocks: echo-request
rich rules:
$ sudo firewall-cmd --runtime-to-permanent
success
For someone who hadn’t used firewalld before, I can say it was OK to use it in this simple use case. There was no need to learn the syntax for nft commands nor the one for nftables rules and it worked quite well in the end. The process of unblocking ICMP ping requests is a bit cumbersome with the flipped logic, and could have been made simpler, but it’s still doable. All-in-all I’m happy with the solution and will look forward how to use it, for instance, in a non-interactive way with Ansible.
The first step is to decide which type is more suitable for the metric to be exported. The Prometheus documentation gives a nice explanation about the four types (Counter, Gauge, Histogram and Summary) offered. What’s important to understand is that they are basically a metric name (like job_queue_size), possibly associated with labels (like {type="email"}) that will have a numeric value associated with it (like 10). When scraped, these will be associated with the collection time, which makes it possible, for instance, to later plot these values in a graph. Different types of metrics will offer different facilities to collect the data.
Next, there’s a need to decide when metrics will be observed. The short answer is “synchronously, at collection time”. The application shouldn’t worry about observing metrics in the background and give the last collected values when scraped. The scrape request itself should trigger the metrics observation - it doesn’t matter if this process isn’t instant. The long answer is that it depends, as when monitoring events, like HTTP requests or jobs processed in a queue, metrics will be observed at event time to be later collected when scraped.
The following example will illustrate how metrics can be observed at event time:
package main
import (
"io"
"log"
"net/http"
"github.com/gorilla/mux"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
var httpRequestsTotal = prometheus.NewCounter(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "Total number of HTTP requests",
ConstLabels: prometheus.Labels{"server": "api"},
},
)
func HealthCheck(w http.ResponseWriter, r *http.Request) {
httpRequestsTotal.Inc()
w.WriteHeader(http.StatusOK)
io.WriteString(w, "OK")
}
func main() {
prometheus.MustRegister(httpRequestsTotal)
r := mux.NewRouter()
r.HandleFunc("/healthcheck", HealthCheck)
r.Handle("/metrics", promhttp.Handler())
addr := ":8080"
srv := &http.Server{
Addr: addr,
Handler: r,
}
log.Print("Starting server at ", addr)
log.Fatal(srv.ListenAndServe())
}
There’s a single Counter metric called http_requests_total (the “total” suffix is a naming convention) with a constant label {server="api"}. The HealthCheck() HTTP handler itself will call the Inc() method responsible for incrementing this counter, but in a real-life application that would preferable be done in a HTTP middleware. It’s important to not forget to register the metrics variable within the prometheus library itself, otherwise it won’t show up in the collection.
Let’s see how they work using the xh HTTPie Rust clone:
$ xh localhost:8080/metrics | grep http_requests_total
# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{server="api"} 0
$ xh localhost:8080/healthcheck
HTTP/1.1 200 OK
content-length: 2
content-type: text/plain; charset=utf-8
date: Sat, 14 Aug 2021 12:26:03 GMT
OK
$ xh localhost:8080/metrics | grep http_requests_total
# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{server="api"} 1
This is cool, but as the metric relies on constant labels, the measurement isn’t that granular. With a small modification we can use dynamic labels to store this counter per route and HTTP method:
diff --git a/main.go b/main.go
index 5d6079a..53249b1 100644
--- a/main.go
+++ b/main.go
@@ -10,16 +10,17 @@ import (
"github.com/prometheus/client_golang/prometheus/promhttp"
)
-var httpRequestsTotal = prometheus.NewCounter(
+var httpRequestsTotal = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "Total number of HTTP requests",
ConstLabels: prometheus.Labels{"server": "api"},
},
+ []string{"route", "method"},
)
func HealthCheck(w http.ResponseWriter, r *http.Request) {
- httpRequestsTotal.Inc()
+ httpRequestsTotal.WithLabelValues("/healthcheck", r.Method).Inc()
w.WriteHeader(http.StatusOK)
io.WriteString(w, "OK")
}
Again, in a real-life application it’s better to let the route be auto-discovered in runtime instead of hard-coding its value within the handler. The result will look like:
$ xh localhost:8080/metrics | grep http_requests_total
# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{route="/healthcheck",method="GET",server="api"} 1
The key here is to understand that the counter vector doesn’t that mean multiple values will be stored in the same metric. What it does is to use different label values to create a multi-dimensional metric, where each label combination is an element of the vector.
]]>