Debian is a trademark of Software in the Public Interest, Inc. This site is operated independently in the spirit of point three of the Debian Social Contract which tells us We will not hide problems.


June 12, 2024

hackergotchi for Matthew Garrett

Matthew Garrett

SSH agent extensions as an arbitrary RPC mechanism

A while back, I wrote about using the SSH agent protocol to satisfy WebAuthn requests. The main problem with this approach is that it required starting the SSH agent with a special argument and also involved being a little too friendly with the implementation - things worked because I could provide an arbitrary public key and the implementation never validated that, but it would be legitimate for it to start doing so and then break everything. And it also only worked for keys stored on tokens that ssh supports - there was no way to extend this to other keystores on the client (such as the Secure Enclave on Macs, or TPM-backed keys on PCs). I wanted a better solution.

It turns out that it was far easier than I expected. The ssh agent protocol is documented here, and the interesting part is the extension support extension mechanism. Basically, you can declare an extension and then just tunnel whatever you want over it. As before, my goto was the go ssh agent package which conveniently implements both the client and server side of this. Implementing the local agent is trivial - look up SSH_AUTH_SOCK, connect to it, create a new agent client that can communicate with that by calling NewClient, and then implement the ExtendedAgent interface, create a new socket, and call ServeAgent against that. Most of the ExtendedAgent functions should simply call through to the original agent, with the exception of Extension(). Just add a case statement against extensionType, define some reasonably namespaced extension, and you're done.

Now you need to use this agent. You probably don't want to use this for arbitrary hosts (agent forwarding should only be enabled for remote systems you trust, not arbitrary machines you connect to - if you enabled agent forwarding for github and github got compromised, github would be able to use any private keys loaded into your agent, and you probably don't want that). So the right approach is to add a Host entry to the ssh config with a ForwardAgent stanza pointing at the socket you created in your new agent. This way the configured subset of remote hosts will automatically talk to this new custom agent, while forwarding for anything else will still be at the user's discretion.

For the remote end things are even easier. Look up SSH_AUTH_SOCK and call NewClient as before, and then simply call client.Extension(). Whatever you stick in the contents argument will simply end up being received at the client end. You now have a communication channel between a the remote system and the local client, and what you do with that is up to you. I'm using it to allow a remote system to obtain auth tokens from Okta and forward WebAuthn challenges that can either be satisfied via a local WebAuthn token or by passing the query off to Mac TouchID, but there's fundamentally no constraints whatsoever on what can be done here.

(If you want to do this on Windows and still have everything work with existing clients you'll need to take this into account - Windows didn't really do Unix sockets until recently so everything there is awful)

comment count unavailable comments

12 June, 2024 02:57AM

June 10, 2024

hackergotchi for Daniel Pocock

Daniel Pocock

Thanking the voters of Midlands-North-West, Ireland

Saturday's blog about the European election predicted that after the character attacks on my family and I, especially the attacks in the week of the election, I would probably get the lowest number of votes.

The returning officer announced the counts for the first preference votes at approximately 10:30pm tonight and confirmed that my prediction was correct and I am no longer in contention for a seat.

I want to thank the 524 voters who gave me their first preference and all those people who gave me their second or subsequent preferences.

I want to thank the other candidates and especially emphasize that while we have some differences, we also found numerous things in common, including a passion for Ireland on par with a passion for politics.

Senator Chambers and I have both suffered the problem of stalkers coming to our homes. In my case, I resigned from mentoring in Google Summer of Code in August 2018 and some of the Debian people developed a pathological obsession with denouncing my family and I to employers, clients and now the voters of Ireland.

I want to thank the journalists who have had to work extra hard to remember the names and faces of 27 candidates, a record for this constituency.

I want to thank the election officials who have to carry those massive ballot papers around in the count center. Candidates and our supporters have been looking on in amazement and hoping nobody gets a paper-cut at our expense.

Lessons and achievements from the election campaign

As a consequence of my run for the parliament, various people tried to inquire why these rogue elements of Debian began denouncing my family and I in 2018. Nobody has ever been able to point to any document from that era to prove how and why this gangstalking phenomena was born out of the Debian cult. Rather, the people around that group keep trying to create documents and secret victim drama retrospectively to avoid the inconvenient truth: attacking the family of a volunteer at a time of grief.

Before my nomination, these cowards had been slinking around behind my back telling potential employers, past employers, clients, personal friends and business partners that I have committed some secret crime that nobody can clearly define. By putting myself through the public election process, I drew these cowards out into the open where they repeated their character attacks in public.

Other people are now leaving that group. Nobody wants to be their next victim. By helping people to avoid this group, I am helping people to avoid going further down a path that leads into the Debian suicide cluster.

At the same time, their financial disclosures have revealed the legal fees of over $120,000 to retrospectively make up reasons why they attacked my family at the time my father died. Yet a volunteer drowned at the annual conference because there were insufficient funds to fully cover the cost of the kayak excursion and he couldn't contribute his own money.

Election regulations in Ireland set a maximum limit of EUR 230,000 for a campaign budget. The sum of money spent by rogue elements of Debian on this anti-campaign, $120,000, is approximately half the authorised amount for a real campaign.

Moreover, this $120,000 was laundered through a bank account operated by Software in the Public Interest, Inc in the United States. Therefore, it is foreign influence against an Irish election candidate.

Most voluntary groups only want to give thanks to contributors who have given decades of service. Voluntary groups that intrude into every other part of our lives and seek to sabotage our ability to pursue social and career opportunities outside the group are incredibly unhealthy and can only be thought of as cults.

Most significantly, more people have started coming forward to give me information about their bad experiences with the Debian harassment culture and similar groups in the world of free, open source software development.

General notes about the elections in Europe and Ireland's local elections

Before the elections, the media had predicted resentment of the current Irish government would benefit rival parties and independents.

In reality, many voters who traditionally voted for the main parties Fine Gael and Fianna Fail simply voted the same as before. Therefore, apart from some extra votes for the far right, those predictions were largely wrong.

People have subsequently tried to explain this vote as an expression of enthusiasm for the current coalition government.

I don't feel this is a correct interpretation of the results. I feel that the behavior of the voters could be explained in terms of war-time voting, whereby voters are choosing the status quo. Choosing the status quo does not imply an enormous enthusiasm for the status quo, it implies voters do not want to risk trying other alternatives at that particular point in time. Similar results occurred in the past, for example Roosevelt's victory in 1944 and George W Bush's victory in 2004 during the Iraq crisis. While the actual wars are not on Irish territory, the presence of refugees and constant news coverage of both Ukraine and Middle East conflict have contributed to the mindset of a war-time election.

Final remarks

The real result of this election won't be known until 2029 when we look back and see what the elected candidates actually achieved for Ireland. Although nobody has achieved a quota so far, I wish the winners well from the moment their identities are confirmed.

I want to emphasize how thrilled I am for the 524 people who chose to go down to the polling station and put a 1 next to my name. Voting is not mandatory in Ireland so each and every one of you could have simply stayed at home. I understand that many of you are following my blogs or you liked something that I said in one of the public meetings or radio broadcasts.

Many people have approached me in person or by email and confided in me about the challenges in your lives and your hopes for the future. The confidence that you had in me to listen and potentially help is a greater honor than holding any political office.

10 June, 2024 10:30PM

Jacob Appelbaum wanted SFTP package uploads to Debian

Jacob Appelbaum is the security researcher who exposed the foreign power that was monitoring the communications of German Chancellor Angela Merkel.

People got mad about his discoveries and used social media to whip up a mob against him.

I previously analyzed the case and proved that Appelbaum was being falsely accused. These games undermine the credibility of women who want to come forward with genuine complaints about certain men in Debian.

I've also demonstrated in a range of cases that the Debian people don't take the Debian Social Contract seriously. This really brings their integrity into question. Anything goes with this group.

Appelbaum had participated in several discussions in the debian-private secret cubby house that is gradually being published online.

Here he talks about using SFTP for package uploads. This is interesting because we know there have been many attempts to subvert the security of SSH, which underlies SFTP. The mysterious Edward Brocklesby affair is one example where the SSH2 package maintainer was secretly expelled.

Subject: sftp package upload (was Re: How many DDs work for NSA anyway?)
Date: Mon, 21 Apr 2014 10:08:14 -0400
From: James McCoy <>

On Mon, Apr 21, 2014 at 01:29:13PM +0000, Jacob Appelbaum wrote:
> I agree with you here too. I'd also argue that it would be nice to
> have sftp uploads.

We have that already.  See
which is supported in dput(-ng) by using the ssh-upload host alias.

GPG Key: 4096R/331BA3DB 2011-12-05 James McCoy <>

Please respect the privacy of this mailing list. Some posts may be declassified
3 years after posting as per

Archive: file://

To UNSUBSCRIBE, use the web form at <>.

10 June, 2024 08:00AM

June 09, 2024

Integrity Fail & Debian Social Contract

Various "Debian" branded insults and character attacks have appeared during the recent European Elections.

To weigh the relevance of Internet community gossip, we need to fact check.

Fact 1: the original Debian Developers adopted the Debian Social Contract (version 1.0) on 5 July 1997.

Fact 2: the original version of the contract and the subsequent version 1.1 and 1.2 both include the promise We won't hide problems. This has always been point 3 in the Debian Social Contract (DSC).

Fact 3: the Shaya Potter affair occured in 1998 and it was totally hidden in the debian-private secret cubby-house. Within a year of adopting the Debian Social Contract, people were already violating it.

Fact 4: the mysterious Edward Brocklesby affair occurrd in early 2000 and it was totally hidden in the debian-private secret cubby-house. Within three years of adopting the Debian Social Contract, the most serious security incidents involving the SSH2 package maintainer were being swept under the carpet.

Fact 5: the email from Andreas Tille tells us that the Debian Social Contract is not being respected.

Subject: We are hiding problems
Date: Thu, 17 Mar 2005 08:16:09 +0100 (CET)
From: Andreas Tille <>
To: Debian Private List <>


I do not want to spread other flame wars to debian-private.
   [ debian-private is the secret cubby-house that has been leaked ]
So please do
not missunderstand this as an intent to discuss the "dropping some archs from
etch" thread here.  I just want to point out that we either have to reword
our social contract item 3:

    We Won't Hide Problems
    We Won't Hide Bugs

or we should provide more relevant information.  I will not argue that we
really have to open all problems.  I want to present a neutral point of view
here.  But the current wording of the social contract does not reflect the
current situation - at least to my knowledge of English language.

Some facts we hide:

[ ... snip ... ]

Fact 6: the rumors about Jacob Appelbaum were proven to be lies when I published the internal discussions from the debian-private secret cubby house.

Fact 7: the financial disclosures published on the SPI web site show that at least $120,000 has been spent paying lawyers to create lies. If you are telling the truth you don't need to pay so much money to a lawyer. You only pay $120,000 if you need help making lies look credible.

Fact 8: Andreas Tille, who wrote that email about the Debian Social Contract being a sham, was subsequently elected as the Debian Project Leader on 20 April 2024, the birthday of Adolf Hitler.

09 June, 2024 11:30PM

European Parliament election count 2024 day 2

As noted in my previous blog, I suspect I am going to get the least number of votes in this campaign but nonetheless, I got up on Sunday monring to travel to Castlebar where all the European Parliament ballot papers have been brought together for counting.

I took a few more photos today.

Due to the recent acts of harassment against people who participate in public life, there is a high level of security to protect us. We have been given passes to go in and out. Garda have been stationed at all the entrances to protect the VIPs and the election officials.

Garda, Castlebar


Daniel Pocock, Castlebar

Luke Ming Flanagan, one of the sitting members, was being interviewed. I took a picture.

Luke Ming Flanagan, Castlebar, count

The reporter questioned Ming about the celebrity candidate Nina Carberry:

"And not only that, hiding a candidate during a campaign works if Nina Carberry gets elected – because I have never ever in my life met Nina Carberry.”

Mr Flanagan said it was a “pity” that the candidates likely to be elected do not have more experience in the European Parliament.

The district is bigger than the three smallest countries in the EU and it is possible that their paths didn't cross for some other reason. I met Miss Carberry once at the Chambers of Commerce hustings in Meath.

Meath Chamber of Commerce, Navan, Nina Carberry, Lisa Chambers, Peadar Tóibín, Pauline O'Reilly, Daniel Pocock

Some of the reporters have asked me about my own political experience. The story of FSFE Fellowship electing me on the anniversary of the Easter Rising has become just as well known as the story of the FSFE (German non-profit) hacking the constitution to prohibit further elections.

While the candidates engaged in discussion, the election officials have a huge task counting over one million ballot papers. As there are a record 27 candidates, the ballot paper has never been so big. Procedures for handling these huge ballot papers are still evolving, as reported in the news.

Some of the voters write notes or draw cartoons on their ballot papers.

Nina Carberry, Pauline O'Reilly, Daniel Pocock

You need two hands to handle them all:

European Election, Midlands-North-West, ballot papers, Castlebar

After they fill up the pigeon holes, the ballot papers are moved to these tables for checking and they are organized into bundles of fifty.

European Election, Midlands-North-West, ballot papers, Castlebar


European Election, Midlands-North-West, ballot papers, Castlebar

The ballot papers are enormous. Some people started using shopping trolleys to move them around.

European Election, Midlands-North-West, ballot papers, Castlebar

To comply with regulations, these have been photographed from a distance where they are not readable.

Daniel Pocock, votes, European Election, Midlands-North-West, ballot papers, Castlebar

09 June, 2024 08:00PM

June 08, 2024

European Parliament election count 2024 day 1

The European Union countries conduct elections for the European Parliament between 6 and 9 June 2024.

The media will not begin broadcasting any results until all the countries finish their voting. Each country has chosen different days for voting. Some of the countries are still voting on the last day, Sunday, 9 June and we will not see official results before the last vote is cast anywhere in Europe.

Ireland voted on Friday, 7 June and opened the ballot boxes on Saturday, 8 June. The officials began counting the ballot papers for the local council elections and some of those results have already been announced today.

The ballot papers for the European Union were sorted into bundles and secured for transport to a central location. For the region of Midlands-North-West, the army will transport all the ballot papers to Castlebar in County Mayo tonight. On Sunday morning, they will begin sorting and counting those ballot papers.

As there were 27 candidates for the region, the voters can number every candidate and there are 1.8 million registered voters, it could take a week to do all the counting and preference transfers. Five seats are being elected. The first seat may be known on Monday or Tuesday while the fifth seat may not be known until the end of the week.

After all the years of harassment from rogue elements of Debian, the people who began attacking my family and I around the time my father died, including the defamation and insults distributed on 6 June, the day before voting, I personally feel that I am going to end up with the smallest number of votes or maybe no votes at all and I will probably be the first candidate eliminated from counting.

Evidence has confirmed that there is a Debian suicide cluster. The people who impose themselves upon the careers of their co-authors and our families like this appear to revel in causing pain and grief. When predicted suicides occur in such a toxic culture, can we just call these Debianists murderers?

Daniel Pocock, Ireland, Galway, count, Midlands-North-West


Ireland, Galway, count, Midlands-North-West

I began my campaign with a visit to Ali Farren of the brand new 100% Redress Party. They are not competing in the European election, they are only competing in the local elections. Mr Farren has achieved brilliant results today, outperforming all the existing Irish political parties in his region.

The defective concrete blocks in these houses remind me of the defective legal protection insurance in the Swiss JuristGate affair.

Congratulations Mr Farren.

Daniel Pocock, Ali Farren

Highland Radio published a photo of Mr Farren in their studio. Please consider the incredible task this man has ahead of him. The smile might not last long.

Ali Farren, Highland Radio

08 June, 2024 10:30PM

Thorsten Alteholz

My Debian Activities in May 2024

FTP master

This month I accepted 347 and rejected 49 packages. The overall number of packages that got accepted was 348.

Debian LTS

This was my hundred-nineteenth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.

During my allocated time I uploaded or worked on:

  • [#1070154] bullseye-pu: qtbase-opensource-src/5.15.2+dfsg-9+deb11u1 package upload
  • [#1064550] bullseye-pu: libjwt 1.10.2-1+deb11u1 has been marked for accept
  • [#1067544] bullseye-pu: libmicrohttpd 0.9.72-2+deb11u1 has been marked for accept

I also continued to work on tiff and last but not least did a week of FD and attended the monthly LTS/ELTS meeting.

Unfortunately I used lots of time to debug an issue with nghttp2. Please see my odyssey below.

Debian ELTS

This month was the seventieth ELTS month. During my allocated time I uploaded:

  • [ELA-1104-1-1]nghttp2 security update for one CVEs to fix an DoS resulting from bad handling of CONTINUATION frames in Stretch

For some tests I installed the new nghttp2 package on my Stretch VM and started the daemon. Unfortunately I got an unexpected error from getaddrinfo() about ai_socktype not supported. The daemon was configured to listen on lo, the device was available, but the error remained. I was pretty sure that my patch was not the reason for this and indeed the unpatched version showed this error as well. I didn’t want to release an untested package, so nghttp2 had to start at least! Therefore I built a minimal example to reproduce the issue. getaddrinfo() failed for hints.ai_socktype=SOCK_STREAM and a numerical IP address. Having no hints at all or “localhost” instead of “” made the error disappear (as a remark: “localhost” resolves to, the ipv6 variant is “ip6-localhost”). I could see that in nghttp2 as well. Configuring it with “localhost” let the error vanish but the daemon still exited due to other reasons. After some time of debugging, I added another network interface to my VM and configured it with a dummy IPv4 address. Voila, everything worked as expected. According to Wikipedia, IPv6 was ratified as standard in 2017 and Stretch was also released in 2017. No wonder that a IPv6-only-VM had problems back then and these problems survived to the present.

I also continued to work on an update for tiff in Jessie and Stretch, did a week of FD and attended the LTS/ELTS meeting.

Debian Printing

This month I uploaded new upstream or bugfix versions of:

This work is generously funded by Freexian!

Debian Astro

This month I uploaded a new upstream or bugfix version of:

Debian IoT

This month I uploaded new upstream or bugfix versions of:

Debian Mobcom

Due to more and more problems with time_t, I removed osmo-iuh and all dependencies from armel, armhf and i386, sorry. If there is really anybody using this software on 32-bit architectures don’t hesitate to get in touch.

It is official now, the GSoC student working on the Mobcom packages is Nathan Doris. He already finished the hardest part of the job and I could upload the latest version of libosmocore. I really enjoy working with him and look forward to a pleasant SoC :-).


This month I uploaded new upstream or bugfix versions of:

Did I already mention that I love lists with topics I can work on. I print out such lists and enjoy checking off one after the other. End of May Helmut told me that I am a bit lazy and gave me such a list with all my packages that have one or the other issue with /usr-move. Most of the uploads above are packages on that list and I could check off a lot :-).

08 June, 2024 05:58PM by alteholz

Reproducible Builds

Reproducible Builds in May 2024

Welcome to the May 2024 report from the Reproducible Builds project! In these reports, we try to outline what we have been up to over the past month and highlight news items in software supply-chain security more broadly. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.

Table of contents:

  1. A peek into build provenance for Homebrew
  2. Distribution news
  3. Mailing list news
  4. Miscellaneous news
  5. Two new academic papers
  6. diffoscope
  7. Website updates
  8. Upstream patches
  9. Reproducibility testing framework

A peek into build provenance for Homebrew

Joe Sweeney and William Woodruff on the Trail of Bits blog wrote an extensive post about build provenance for Homebrew, the third-party package manager for MacOS. Their post details how each “bottle” (i.e. each release):

[…] built by Homebrew will come with a cryptographically verifiable statement binding the bottle’s content to the specific workflow and other build-time metadata that produced it. […] In effect, this injects greater transparency into the Homebrew build process, and diminishes the threat posed by a compromised or malicious insider by making it impossible to trick ordinary users into installing non-CI-built bottles.

The post also briefly touches on future work, including work on source provenance:

Homebrew’s formulae already hash-pin their source artifacts, but we can go a step further and additionally assert that source artifacts are produced by the repository (or other signing identity) that’s latent in their URL or otherwise embedded into the formula specification.

Distribution news

In Debian this month, Johannes Schauer Marin Rodrigues (aka josch) noticed that the Debian binary package bash version 5.2.15-2+b3 was “uploaded to the archive twice. Once to bookworm and once to sid but with differing content.” This is problem for reproducible builds in Debian due its assumption that the package name, version and architecture triplet is unique. However, josch highlighted that

This example with bash is especially problematic since bash is Essential:yes, so there will now be a large portion of .buildinfo files where it is not possible to figure out with which of the two differing bash packages the sources were compiled.

In response to this, Holger Levsen performed an analysis of all .buildinfo files and found that this needs almost 1,500 binNMUs to fix the fallout from this bug.

Elsewhere in Debian, Vagrant Cascadian posted about a Non-Maintainer Upload (NMU) sprint to take place during early June, and it was announced that there is now a #debian-snapshot IRC channel on OFTC to discuss the creation of a new source code archiving service to, perhaps, replace Lastly, 11 reviews of Debian packages were added, 15 were updated and 48 were removed this month adding to our extensive knowledge about identified issues. A number of issue types have been updated by Chris Lamb as well. [][]

Elsewhere in the world of distributions, deep within a larger announcement from Colin Percival about the release of version 14.1-BETA2, it was mentioned that the FreeBSD kernels are now built reproducibly.

In Fedora, however, the change proposal mentioned in our report for April 2024 was approved, so, per the ReproduciblePackageBuilds wiki page, the add-determinism tool is now running in new builds for Fedora 41 (‘rawhide’). The add-determinism tool is a Rust program which, as its name suggests, adds determinism to files that are given as input by “attempting to standardize metadata contained in binary or source files to ensure consistency and clamping to $SOURCE_DATE_EPOCH in all instances”. This is essentially the Fedora version of Debian’s strip-nondeterminism. However, strip-nondeterminism is written in Perl, and Fedora did not want to pull Perl in the buildroot for every package. The add-determinism tool eliminates many causes of non-determinism and work is ongoing to continue the scope of packages it can operate on.

Mailing list news

On our mailing list this month, regular contributor kpcyrd wrote to the list with an update on their source code indexing project, The project, which was launched last month in response to the XZ Utils backdoor, now contains and indexes almost 250,000 unique source code archives. In their post, kpcyrd gives an example of its intended purpose, noting that it shown that whilst “there seems to be consensus about [the] source code for zsh 5.9” in various Linux distributions, it “does not align with the contents of the zsh Git repository”.

Holger Levsen also posted to the list with a ‘pre-announcement’ of sorts for the 2024 Reproducible Builds summit. In particular:

[Whilst] the dates and location are not fixed yet, however if you don’ help us with finding a suitable location soon, it is very likely that we’ll meet again in Hamburg in the 2nd half of September 2024 […].

Lastly, Frederic-Emmanuel Picca wrote to the list asking for help understanding the “non-reproducible status of the Debian silx package” and received replies from both Vagrant Cascadian and Chris Lamb.

Miscellaneous news

strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. This month strip-nondeterminism version 1.14.0-1 was uploaded to Debian unstable by Chris Lamb chiefly to incorporate a change from Alex Muntada to avoid a dependency on Sub::Override to perform monkey-patching and break circular dependencies related to debhelper []. Elsewhere in our tooling, Jelle van der Waa modified reprotest because the pipes module will be removed in Python version 3.13 [].

It was also noticed that a new blog post by Daniel Stenberg detailing “How to verify a Curl release” mentions the SOURCE_DATE_EPOCH environment variable. This is because:

The [curl] release tools document also contains another key component: the exact time stamp at which the release was done – using integer second resolution. In order to generate a correct tarball clone, you need to also generate the new version using the old version’s timestamp. Because the modification date of all files in the produced tarball will be set to this timestamp.

Furthermore, Fay Stegerman filed a bug against the Signal messenger app for Android to report that their ‘reproducible’ builds cannot, in fact, be reproduced. However, Fay is quick to note that she has:

… found zero evidence of any kind of compromise. Some differences are yet unexplained but everything I found seems to be benign. I am disappointed that Reproducible Builds have been broken for months but I have zero reason to doubt Signal’s security in any way.

Lastly, it was observed that there was a concise and diagrammatic overview of “supply chain threats” on the SLSA website.

Two new academic papers

Two new scholarly papers were published this month.

Firstly, Mathieu Acher, Benoît Combemale, Georges Aaron Randrianaina and Jean-Marc Jézéquel of University of Rennes on Embracing Deep Variability For Reproducibility & Replicability. The authors describe their approach as follows:

In this short [vision] paper we delve into the application of software engineering techniques, specifically variability management, to systematically identify and explicit points of variability that may give rise to reproducibility issues (e.g., language, libraries, compiler, virtual machine, OS, environment variables, etc.). The primary objectives are: i) gaining insights into the variability layers and their possible interactions, ii) capturing and documenting configurations for the sake of reproducibility, and iii) exploring diverse configurations to replicate, and hence validate and ensure the robustness of results. By adopting these methodologies, we aim to address the complexities associated with reproducibility and replicability in modern software systems and environments, facilitating a more comprehensive and nuanced perspective on these critical aspects.

(A PDF of this article is available.)

Secondly, Ludovic Courtès, Timothy Sample, Simon Tournier and Stefano Zacchiroli have collaborated to publish a paper on Source Code Archiving to the Rescue of Reproducible Deployment. Their paper was motivated because:

The ability to verify research results and to experiment with methodologies are core tenets of science. As research results are increasingly the outcome of computational processes, software plays a central role. GNU Guix is a software deployment tool that supports reproducible software deployment, making it a foundation for computational research workflows. To achieve reproducibility, we must first ensure the source code of software packages Guix deploys remains available.

(A PDF of this article is also available.)


diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes such as uploading versions 266, 267, 268 and 269 to Debian, making the following changes:

  • New features:

    • Use xz --list to supplement output when comparing .xz archives; essential when metadata differs. (#1069329)
    • Include xz --verbose --verbose (ie. double) output. (#1069329)
    • Strip the first line from the xz --list output. []
    • Only include xz --list --verbose output if the xz has no other differences. []
    • Actually append the xz --list after the container differences, as it simplifies a lot. []
  • Testing improvements:

    • Allow Debian testing to fail right now. []
    • Drop apktool from Build-Depends; we can still test APK functionality via autopkgtests. (#1071410)
    • Add a versioned dependency for at least version 5.4.5 for the xz tests as they fail under (at least) version 5.2.8. (#374)
    • Fix tests for 7zip 24.05. [][]
    • Fix all tests after additon of xz --list. [][]
  • Misc:

    • Update copyright years. []

In addition, James Addison fixed an issue where the HTML output showed only the first difference in a file, while the text output shows all differences [][][], Sergei Trofimovich amended the 7zip version test for older 7z versions that include the string “[64]“ [][] and Vagrant Cascadian relaxed the versioned dependency to allow version 5.4.1 for the xz tests [] and proposed updates to guix for versions 267, 268 and pushed version 269 to Guix. Furthermore, Eli Schwartz updated the website in order to explain how to install diffoscope on Gentoo [].

Website updates

There were a number of improvements made to our website this month, including Chris Lamb making the “print” CSS stylesheet nicer []. Fay Stegerman made a number of updates to the page about the SOURCE_DATE_EPOCH environment variable [][][] and Holger Levsen added some of their presentations to the “Resources” page. Furthermore, IOhannes zmölnig stipulated support for SOURCE_DATE_EPOCH in clang version 16.0.0+ [], Jan Zerebecki expanded the “Formal definition” page and fixed a number of typos on the “Buy-in” page [] and Simon Josefsson fixed the link to Trisquel GNU/Linux on the “Projects” page [].

Upstream patches

This month, we wrote a number of patches to fix specific reproducibility issues, including:

Reproducibility testing framework

The Reproducible Builds project operates a comprehensive testing framework running primarily at in order to check packages and other artifacts for reproducibility. In May, a number of changes were made by Holger Levsen:

  • Debian-related changes:

    • Enable the rebuilder-snapshot API on osuosl4. []
    • Schedule the i386 architecture a bit more often. []
    • Adapt to the new way of running our build services. []
    • Add 8 more workers for the i386 architecture. []
    • Update configuration now that the infom07 and infom08 nodes have been reinstalled as “real” i386 systems. []
    • Make diffoscope timeouts more visible on the #debian-reproducible-changes IRC channel. []
    • Mark the cbxi4a-armhf node as down. [][]
    • Only install the hdmi2usb-mode-switch package only on Debian bookworm and earlier [] and only install the haskell-platform package on Debian bullseye [].
  • Misc:

    • Install the ntpdate utility as we need it later. []
    • Document the progress on the i386 architecture nodes at Infomaniak. []
    • Drop an outdated and unnoticed notice. []
    • Add live_setup_schroot to the list of so-called “zombie” jobs. []

In addition, Mattia Rizzolo reinstalled the infom07 and infom08 nodes [] and Vagrant Cascadian marked the cbxi4a node as online [].

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

08 June, 2024 10:30AM

June 07, 2024


D-Day propaganda & Debian statements on volunteers

6 June 2024 was the 80th anniversary of the D-Day landings (Operation Overlord) from 6 June 1944. D-Day was the beginning of the end for the German fascists.

Hitler's propaganda department had to work really hard to ensure the German soldiers continued fighting and the German citizens continue going to work in their factories every day. In other words, even if Hitler couldn't defend the territory, he had to defend the hearts and minds of those under the Nazi spell.

What did German news reports show the public on the day of the D-Day landings and the subsequent days? According to reports, they reused old stock footage from battles where Germans were successful:

German propaganda delighted in showing images of immense gun emplacements, guarded by grim Aryan-looking soldiers straight out of central casting. In fact, though, German newsreels often showed the same shot again and again: the Lindemann Battery at Cap Gris Nez on the coast, with its three 406mm guns.

Coincidentally, the countries of the European Union are conducting elections for the European Parliament between 6 and 9 June 2024. There are concerns that fascist groups will use these elections to recover some of the ground that was lost by Hitler eighty years ago this week.

Oddly, the rise of fascism in Europe has been facillitated by social media technology created by American companies such as Twitter and Facebook.

Social media algorithms put each individual user into a silo where they see the views of people who agree with them. The algorithms take people to extremes of the left or extremes of the right and snuff out moderate center viewpoints.

The well known Debian Developer Daniel Pocock is a candidate in the European elections. The new Debian Project Leader is a German, Andreas Tille, who was elected on 20 April, the birthday of Adolf Hitler.

The Debian constitution describes an autocracy. The constitution tells us that the Debian Project Leader is personally responsible for decisions and communications. Therefore, we can hold Herr Tille responsible for the publication of insults and defamation.

Under the leadership of Herr Tille, we see the same tactics as those used by the propaganda department in the Third Reich. Just as the Nazis re-used the same stock footage of German soldiers defending their positions, Debian infrastructure is now being used for repeat attacks on a single volunteer, Daniel Pocock.

What is the objective of these defamation and insult attacks on volunteers? Do they want volunteers/victims to commit suicide, as in the previous members of the Debian Suicide Cluster?

07 June, 2024 03:00PM

June 06, 2024

Paul Wise

FLOSS Activities May 2024


This month I didn't have any particular focus. I just worked on issues in my info bubble.




  • Debian BTS usertags: changes for the month


  • Debian wiki: approve accounts


  • Respond to queries from Debian users and contributors on the mailing lists and IRC


All work was done on a volunteer basis.

06 June, 2024 04:28AM

June 05, 2024

hackergotchi for Alberto García

Alberto García

More ways to install software in SteamOS: Distrobox and Nix


In my previous post I talked about how to use systemd-sysext to add software to the Steam Deck without modifying the root filesystem. In this post I will give a brief overview of two additional methods.


distrobox is a tool that uses containers to create a mutable environment on top of your OS.

Distrobox running in SteamOS

With distrobox you can open a terminal with your favorite Linux distro inside, with full access to the package manager and the ability to install additional software. Containers created by distrobox are integrated with the system so apps running inside have normal access to the user’s home directory and the Wayland/X11 session.

Since these containers are not stored in the root filesystem they can survive an OS update and continue to work fine. For this reason they are particularly suited to systems with an immutable root filesystem such as Silverblue, Endless OS or SteamOS.

Starting from SteamOS 3.5 the system comes with distrobox (and podman) preinstalled and it can be used right out of the box without having to do any previous setup.

For example, in order to create a Debian bookworm container simply open a terminal and run this:

$ distrobox create -i debian:bookworm debbox

Here debian:bookworm is the image that this container is created from (debian is the name and bookworm is the tag, see the list of supported tags here) and debbox is the name that is given to this new container.

Once the container is created you can enter it:

$ distrobox enter debbox

Or from the ‘Debian’ entry in the desktop menu -> Lost & Found.

Once inside the container you can run your Debian commands normally:

$ sudo apt update
$ sudo apt install vim-gtk3


Nix is a package manager for Linux and other Unix-like systems. It has the property that it can be installed alongside the official package manager of any distribution, allowing the user to add software without affecting the rest of the system.

Nix running in SteamOS

Nix installs everything under the /nix directory, and packages are made available to the user through a new entry in the PATH and a ~/.nix-profile symlink stored in the home directory.

Nix is more things, including the basis of the NixOS operating system. Explaning Nix in more detail is beyond the scope of this blog post, but for SteamOS users these are perhaps its most interesting properties:

  • Nix is self-contained: all packages and their dependencies are installed under /nix.
  • Unlike software installed with pacman, Nix survives OS updates.
  • Unlike podman / distrobox, Nix does not create any containers. All packages have normal access to the rest of the system, just like native SteamOS packages.
  • Nix has a very large collection of packages, here is a search engine:

The only thing that Nix needs from SteamOS is help to set up the /nix directory so its contents are not stored in the root filesystem. This is already happening starting from SteamOS 3.5 so you can install Nix right away in single-user mode:

$ sudo chown deck:deck /nix
$ wget
$ sh ./install --no-daemon

This installs Nix and adds a line to ~/.bash_profile to set up the necessary environment variables. After that you can log in again and start using it. Here’s a very simple example (refer to the official documentation for more details):

# Install and run Midnight Commander
$ nix-env -iA
$ mc

# List installed packages
$ nix-env -q

# Uninstall Midnight Commander
$ nix-env -e mc-4.8.31

What we have seen so far is how to install Nix in single-user mode, which is the simplest one and probably good enough for a single-user machine like the Steam Deck. The Nix project however recommends a multi-user installation, see here for the reasons.

Unfortunately the official multi-user installer does not work out of the box on the Steam Deck yet, but if you want to go the multi-user way you can use the Determinate Systems installer:


Distrobox and Nix are useful tools and they give SteamOS users the ability to add additional software to the system without having to modify the base operating system.

While for graphical applications the recommended way to install third-party software is still Flatpak, Distrobox and Nix give the user additional flexibility and are particularly useful for installing command-line utilities and other system tools.

05 June, 2024 03:53PM by berto

June 04, 2024

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

ulid 0.4.0 on CRAN: Extended to Milliseconds

A new version of the ulid package is now on CRAN. The packages provides ‘universally (unique) lexicographically (sortable) identifiers’ – see the spec at GitHub for details on those – which offer sorting which uuids lack. The R package provides access via the standard C++ library, had been put together by Bob Rudis and is now maintained by me.

Mark Heckmann noticed that a ulid round trip of generating and unmarshalling swallowed subsecond informationm and posted on a well-known site I no longer go to. Duncan Murdoch was kind enough to open an issue to make me aware, and in it included the nice minimally complete verifiable example by Mark.

It turns out that this issue was known, documented upstream in two issues and fixed in fork by the authors of those issues, Chris Bove. It replaces time_t as the value of record (constrained at the second resolution) with a proper std::chrono object which offers milliseconds (and much more, yay Modern C++). So I switched the two main files of library to his, and updated the wrapper code to interface from POSIXct to std::chrono object. And with that we are in business. The original example of five ulids create 100 millisecond part, then unmarshalled and here printed as a data.table as data.frame by default truncates to seconds:

> library(ulid)
> gen_ulid <- \(sleep) replicate(5, {Sys.sleep(sleep); generate()})
> u <- gen_ulid(.1)
> df <- unmarshal(u)
> data.table::data.table(df)
                        ts              rnd
                    <POSc>           <char>
1: 2024-05-30 16:38:28.588 CSQAJBPNX75R0G5A
2: 2024-05-30 16:38:28.688 XZX0TREDHD6PC1YR
3: 2024-05-30 16:38:28.789 0YK9GKZVTED27QMK
4: 2024-05-30 16:38:28.890 SC3M3G6KGPH7S50S
5: 2024-05-30 16:38:28.990 TSKCBWJ3TEKCPBY0

We updated the documentation accordingly, and added some new tests as well. The NEWS entry for this release follows.

Changes in version 0.4.0 (2024-06-03)

  • Switch two functions to fork by Chris Bove using std::chrono instead of time_t for consistent millisecond resolution (#3 fixing #2)

  • Updated documentation showing consistent millisecond functionality

  • Added unit tests for millisecond functionality

Courtesy of my CRANberries, there is also a diffstat report for this release. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

04 June, 2024 11:43PM

hackergotchi for Jonathan Dowland

Jonathan Dowland

Quake (soundtrack)

I haven't done that much crate digging recently, but I did stick this on last week: Trent Reznor's soundtrack for Quake, originally released (within the game) in 1996, and finally issued for the first time independently in 2020.

Quake LP cover and inner covers

Quake LP cover and inner covers

I picked it up the Nine Inch Nails gig in Cornwall, 2022.

An interesting factoid about the original release was the CD was mastered with the little-known pre-emphasis flag set to "on". This was relatively unusual at the time (1996) that it was never clear whether it was deliberate or not. CD ripping back then usually used an analog audio path from the CD-ROM drive to the PC sound card, and the CD-ROM would apply the necessary pre-emphasis. Therefore, ripping software didn't need to deal with it, and so most of it (then and now) doesn't, even though the path had long since changed to a purely-digital extraction. Thus, the various copies of the soundtrack circulating may or may not have had pre-emphasis correction applied, and if they did, it may or may not have been required to hear the soundtrack as it was intended.

I spent a bit of time a few years ago, before the reissue, trying to determine what was "correct". There is certainly an audible difference with pre-emphasis applied (or not), but it wasn't clear which was the intended experience. The reissue should have cleared this up once and for all, but I haven't gone back to check what the outcome was.

04 June, 2024 10:17AM

June 03, 2024

hackergotchi for Norbert Preining

Norbert Preining

Arch Linux: Fix gcr-agent interfering with ssh/gpg-agent

I finally found the reason why my ssh authentication socket provided by gpg-agent wasn’t used by programs started from the desktop environment (KDE in my case, but this is irrelevant). The problem is, not surprisingly, a bug in the Arch package of gcr-4.

I did configure the env variable SSH_AUTH_SOCK to use gpg-agent via ~/.config/environment.d/80_gpg_agent.conf containing


but despite this, all programs started from the DE (like start menu, task bar, etc) did have the wrong setting. This resulted in the ssh passphrase being asked multiple times.

Basically what happened is that the Arch packaging of gcr (Gnome Crypto ???) started enabling by default the gcr-ssh-agent.socket, which – in the infinite wisdom of Gnome whoohaa people – forcibly updated the environment

ExecStartPost=-/usr/bin/systemctl --user set-environment SSH_AUTH_SOCK=%t/gcr/ssh

(no comment about this masterpiece, never heard about gpg-agent etc …).

The bug in the Arch package was fixed 2 months ago in this commit, but unfortunately, Arch does not provide clean-up of incorrectly configured systems. That means in my case that the incorrect link did remain there even after fixed packages got installed.

Sometimes I long for the Debian way requiring messed up configuration to be fixed by the package maintainers …

So, for those who are still seeing this bug, the following steps are necessary:

systemctl --user stop gcr-ssh-agent.service
systemctl --user stop gcr-ssh-agent.socket
systemctl --user disable gcr-ssh-agent.service
systemclt --user disable gcr-ssh-agent.socket

and most importantly

systemctl --global disable gcr-ssh-agent.socket

After that and a reboot, the SSH_AUTH_SOCK env variable should point to the correct place, and one should only get asked once for the passphrase.

03 June, 2024 12:42PM by Norbert Preining

June 02, 2024

hackergotchi for Colin Watson

Colin Watson

Free software activity in May 2024

My Debian contributions this month were all sponsored by Freexian.

The bulk of my Debian time this month went towards trying to haul more Python packages up to current versions, but I got a few other bits and pieces done as well.

  • I did a little work on improving debbugs’ autopkgtest status.
  • openssh:
    • I fixed an OpenSSL version mismatch error in openssh-ssh1.
    • I finally tracked down a baffling CI issue in openssh, unblocking several contributed merge requests that I’d been sitting on until I could get CI to pass for them. (Special thanks to Andreas Hasenack; GSS-API integration tests will make my life much easier.)
    • I removed the user_readenv=1 option from openssh’s PAM configuration, and did some work on release notes to document this change for affected users.
    • I started work on the first stage of my plan to split out GSS-API key exchange support to separate packages.
  • Python team:
    • I upgraded bitstruct, flufl.enum, flufl.testing, gunicorn, langtable, psycopg3, pygresql, pylint-flask, python-click-didyoumean, python-gssapi, python-httplib2, python-json-log-formatter, python-persistent, python-pgspecial, python-pyld, python-repoze.tm2, python-serializable, python-tenacity, python-typing-extensions, python-unidiff, responses, shortuuid (including an upstream packaging tweak), sqlparse, vulture, zc.lockfile, and zope.interface to new upstream versions.
    • I cherry-picked an upstream PR to fix a pytest 8 incompatibility in ipywidgets.
  • I decided that fixing my old troffcvt package to support groff 1.23.0 wasn’t worth the time investment, and filed a removal request instead.
  • I NMUed bidentd and linuxtv-dvb-apps to declare Architecture: linux-any (and in the latter case also to fix a build failure due to 64-bit time), and worked with the buildd team to remove several of the other remaining entries from Packages-arch-specific.

You can support my work directly via Liberapay.

02 June, 2024 10:53AM by Colin Watson

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

SIMD detection of nested quotes

I recently spent some time thinking about the problem of detecting quoted strings using SIMD, and I've come annoyingly close without actually having a practical solution; yet, I thought I would share my half-solution because I think it's fairly interesting in its own right.

To give a brief recap, here's an idealized version of the problem:

  • You have 16 (or whatever) ASCII bytes in a single SIMD register (SSE2, NEON, etc.).
  • Strings are delineated with "double quotes" or 'single quotes'.
  • Your task is, to efficiently as possible, make a value that is all-ones for each character within quotes and all-zeros everywhere else. (We don't care about what the mask says about the quotes themselves; that's easy to fix up afterwards anyway.)

The use, of course, is to be able to mask out special characters within strings. In my case, we were looking for the first right brace (}) in a byte stream, except that one wouldn't count those that were within quoted strings.

I'm going to recap first how to do this with only one kind of quotes; it's widely known, but I think it's useful to think about why it works. (There's also a variant based on carryless multiplication, which is now often efficient because CPUs implement them for cryptographic purposes, but I'm not going to go into it.) We'll then try to make an analogous solution that supports the harder case where quote markers can be within other quotes and themselves ignored. (For instance, in the string "quo'ted" not quoted "'", the “not quoted” part is between single quotes, but nevertheless, it should not be considered as quoted, because the single quote itself was quoted and thus ignored.)

If you only have one kind of quote (say, "), then the question for any given character is simply whether an odd number of quotes was before it. " turns on quoting, " turns off quoting again. This is computed most easily by comparing each character (in parallel) with ", and then XOR-ing the booleans as we go. This is called a prefix sum (just with XOR instead of add), and I'm going to write it out in full for eight bytes:

r7 = x7 ^ x6 ^ x5 ^ x4 ^ x3 ^ x2 ^ x1 ^ x0;
r6 =      x6 ^ x5 ^ x4 ^ x3 ^ x2 ^ x1 ^ x0;
r5 =           x5 ^ x4 ^ x3 ^ x2 ^ x1 ^ x0;
r4 =                x4 ^ x3 ^ x2 ^ x1 ^ x0;
r3 =                     x3 ^ x2 ^ x1 ^ x0;
r2 =                          x2 ^ x1 ^ x0;
r1 =                               x1 ^ x0;
r0 =                                    x0;

So x0..x7 is whether each byte was a quote or not, and r0..r7 is whether we consider that byte quoted or not. (In the full problem, we'll need to consider whether we started our byte group in a quoted state or not, so technically, this only says whether each byte should invert the initial state or not. But for XOR, this is easy; just XOR in the initial value somewhere, either into x0 or into all of the r0..r7 as a post-processing step.)

For scalar code, there's a very simple and efficient way to compute this:

r0 = x0;
r1 = r0 ^ x1;
r2 = r1 ^ x2;
r3 = r2 ^ x3;
r4 = r3 ^ x4;
r5 = r4 ^ x5;
r6 = r5 ^ x6;
r7 = r6 ^ x7;

However, this doesn't really work well for SIMD; we get one long dependency chain, with zero parallelism. We want to use those fancy ALUs for something. So instead, we can look at the expression and regroup it a bit:

r7 = (x7 ^ x6) ^ (x5 ^ x4) ^ (x3 ^ x2) ^ (x1 ^ x0);
r6 =       (x6 ^ x5) ^ (x4 ^ x3) ^ (x2 ^ x1) ^ x0;
r5 =             (x5 ^ x4) ^ (x3 ^ x2) ^ (x1 ^ x0);
r4 =                   (x4 ^ x3) ^ (x2 ^ x1) ^ x0;
r3 =                         (x3 ^ x2) ^ (x1 ^ x0);
r2 =                               (x2 ^ x1) ^ x0;
r1 =                                     (x1 ^ x0);
r0 =                                           x0;

This hints that making some temporaries would be a useful step. Let's compute all of those pairs that we just created:

y7 = x7 ^ x6;
y6 = x6 ^ x5;
y5 = x5 ^ x4;
y4 = x4 ^ x3;
y3 = x3 ^ x2;
y2 = x2 ^ x1;
y1 = x1 ^ x0;
y0 = x0;

This is very efficient to do in basically any SIMD instruction set; just shift the x0..x7 vector by 8 bits, and then do a XOR between the two vectors. (In some instruction sets, you may not have full 128-bit shifts available. It is fairly easy to work around at a slight loss in efficiency, so let's not bother ourselves with it.)

This allows us to rewrite our expression as:

r7 = y7 ^ y5 ^ y3 ^ y1;
r6 = y6 ^ y4 ^ y2 ^ y0;
r5 =      y5 ^ y3 ^ y1;
r4 =      y4 ^ y2 ^ y0;
r3 =           y3 ^ y1;
r2 =           y2 ^ y0;
r1 =                y1;
r0 =                y0;

Hopefully, after staring a bit at this, you can see where we're going even though it's slightly different from before. We can make new parentheses and new temporaries and rewrite again:

z7 = y7 ^ y5;
z6 = y6 ^ y4;
z5 = y5 ^ y3;
z4 = y4 ^ y2;
z3 = y3 ^ y1;
z2 = y2 ^ y0;
z1 = y1;
z0 = y0;

r7 = z7 ^ z3;
r6 = z6 ^ z2;
r5 = z5 ^ z1;
r4 = z4 ^ z0;
r3 = z3;
r2 = z2;
r1 = z1;
r0 = z0;

This is exactly the same, just shift by 16 bits instead of 8. And the last step is similarly easy.

Now, it's useful to thinking about what properties of our XOR operation we actually needed. Notably, we didn't need the property that XOR-ing something twice cancels out. And we didn't need that a ^ b = b ^ a (commutativity). Pretty much the only thing we needed was that our operation was associative ((a ^ b) ^ c = a ^ (b ^ c)) and that we could combine our elements before we knew the quoted state at the point in time, i.e. our values could be looked upon at an action on the state, and not manipulating the state directly until at later.

Now let's look at the more complicated example where we don't have just quoted/non-quoted, but three states for any byte. Let's number them:

  • 0: We are not within quotes.
  • 1: We are within double quotes.
  • 2: We are within single quotes.

For the purposes of the final answer, we don't really care about the difference between 1 and 2, but for figuring out the effect of any given input character, we definitely need to understand it. Let's look at a trivial character; call it c. It is not a quote, so what does it do in each of our states?

  • 0: Not within quotes, so we stay within quotes (stay in 0).
  • 1: In double quotes, so we are still within double quotes (stay in 1).
  • 2: In single quotes, so we are still within single quotes (stay in 2).

OK, so 0 → 0, 1 → 1, 2 → 2. Let's call that 012 for convenience of notation. What about the double-quote character "?

  • 0: Not within quotes, but after this, we are (move to 1).
  • 1: Within double quotes, so we're ending that (move to 0).
  • 2: Within single quotes, so the character is to be ignored (move to 2).

So double quotes gave rise to the mapping 0 → 1, 1 → 0, 2 → 2, or 102 for short. And we'll similarly find out that a single quote character ' will give the mapping 0 → 2, 1 → 1, 2 → 0, or 210 for short.

Now, the crucial insight comes: This is exactly like our XOR case! We can combine these without knowing what state we are in; just keep following the effect for all three states. For instance, the character sequence "x will trivially become 102 just like the double quote itself did. "x" will trivially become 012 (no effect). And we have some more interesting examples like "'; if we follow both maps in turn, we'll have 0 → 1 → 1, 1 → 0 → 2, 2 → 2 → 0. So this combination of signs will give is 120, or written out:

  • If we started with no quotes (0), we are now within double quotes (1).
  • If we started in double quotes (1), we are now in single quotes (2).
  • If we started in single quotes (2), we are now not in quotes (0).

Can we always do this? Yes! Group theory tells us that this is the symmetric group S₃. And groups are always closed (always return a new element of the group), and always associative. We know that S₃ has 3! = 6 elements, and it's not hard to construct examples of how to get into all 6 of them. We can label them 012 021 102 120 201 210, or we can give them easier-for-a-computer names like 0 1 2 3 4 5.

So that's the general battle plan. For each character, find out which state permutation it belongs to (one out of three). Then run the same cascade as before, just with the group product instead of XOR, given by the following table (trivially calculated by just following the states for each case; note that it is not symmetric, i.e., our group is nonabelian, but remember, we never required commutativity):

    | 012 021 102 120 210 201
012 | 012 021 102 120 210 201
021 | 021 012 201 210 120 102
102 | 102 120 012 021 201 210
120 | 120 102 210 201 021 012
210 | 210 201 120 102 012 021
201 | 201 210 021 012 102 120

or, equivalently with shorter names:

  | 0 1 2 3 4 5 
0 | 0 1 2 3 4 5 
1 | 1 0 4 5 2 3 
2 | 2 3 0 1 5 4 
3 | 3 2 5 4 0 1 
4 | 4 5 1 0 3 2 
5 | 5 4 3 2 1 0 

and then finally combine with the status from the previous case. If we're in the 012 or 021 state (0 or 1), we're not within quotes for that character; otherwise, we are.

Now, of course, here's the problem: It's not obvious how to do this group product effieciently without a table. If we had only 16 elements, we could have used PSHUFB, but we have 36. When looking at the numeric variant, there are some values we could special-case (e.g. the entire first row and column), but e.g. the last row is only nearly trivial to calculate but not quite. So we'd be at an annoying amount of table lookups to get this to work.

So, that's where I stand. Faster than linear in theory, but probably less efficient in practice unless you had absolutely huge vectors and/or unusually efficient gather instructions. (I guess for an FPGA solution, this method wouldn't be so bad?) I didn't really bother trying to write up the actual code; I added some code to detect the difficult cases (they happen if any character thinks it's both within single quotes and double quotes at the same time) and error out if it were found, and then mostly called it a day. I guess the most efficient solution, if you really need to handle this case branch-free, would be a character-by-character one, but based on something like Sheng.

Edit: I noticed that there is actually a little respite; the group product table in this formulation is vertically antisymmetric (rows 0/5, 1/4 and 2/3 form pairs where one is 5-x of the other; some are also reverses of other rows, but that's perhaps less interesting), which makes it possible to do only two shuffles and then a lot of comparing and blending and fixup. I implemented it and it actually works, but is definitely too slow to be useful in practice. It seems there's also an AVX512 instruction that actually gives a 64-entry LUT, which would be really nice if it's fast, and well, if AVX512 actually existed everywhere. :-)

02 June, 2024 10:10AM

Jacob Adams

What to Do When You Forget Your Root Password

Forgetting your root password would initially seem like a problem requiring a full re-install, one that you can’t easily recover from without wiping everything away.

Forgetting your user password can of course be solved by changing it as root, as in the following, which changes the password for user jacob:

# passwd jacob

but only the root user can change their own password, so you need to somehow get root access in order to do so.

Changing Root’s Password with Sudo

This one is probably obvious, but if you have a user with the ability to use sudo, then you can change root’s password without access to the root account by running:

$ sudo passwd

which will reset the password for the root account without requiring the existing password.

Boot Directly to a Shell

Getting root access to any Linux machine you have physical access to is surprisingly simple. You can just boot the machine directly into a root shell without any access control, i.e. passwords.

Why You Should Always Encrypt Your Storage1

To boot directly to a shell you need to append the following text to the kernel command line:


(You could use pretty much any program here, but you’re putting your system into a weird state doing this, and so I’d recommend the simplest approach.)


GRUB will allow you to edit boot parameters on startup using the e key. You’ll then be presented with a editor2 that you can use to change the kernel command line by appending to the linux line.

E.g. If your editor looks like this:

        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod ext2
        search --no-floppy --fs-uuid --set=root abcd1234-5678-0910-1112-abcd12345678
        echo    'Loading Linux 6.1.0-21-amd64 ...'
        linux   /boot/vmlinuz-6.1.0-21-amd64 root=UUID=abcd1234-5678-0910-1112-abcd12345678 ro  quiet
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd.img-6.1.0-21-amd64

Then you would add init=/bin/sh like so:

        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod ext2
        search --no-floppy --fs-uuid --set=root abcd1234-5678-0910-1112-abcd12345678
        echo    'Loading Linux 6.1.0-21-amd64 ...'
        linux   /boot/vmlinuz-6.1.0-21-amd64 root=UUID=abcd1234-5678-0910-1112-abcd12345678 ro  quiet init=/bin/sh
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd.img-6.1.0-21-amd64

Once you’ve edited it you can start your machine with Ctrl+x, as you can see from the prompt text under the editor.

Raspberry Pi cmdline.txt

On a Raspberry Pi you’ll want to append the above to only line of the cmdline.txt file on the boot partition of the SD card. This is the first partition of the disk, the one that is FAT32.

You’ll need to do this on another machine, since if you had root access to edit cmdline.txt you could also just change your password.

As it is a FAT32 partition on an SD card, it should be editable on any other machine that supports SD cards.

E.g. If your cmdline.txt looks like this

console=serial0,115200 console=tty1 root=PARTUUID=fb33757d-02 rootfstype=ext4 rootwait quiet

Then you would add init=/bin/sh like so:

console=serial0,115200 console=tty1 root=PARTUUID=fb33757d-02 rootfstype=ext4 rootwait quiet init=/bin/sh

Mount Read / Write

Since you’re replacing the init process of the machine with a shell, no other processes will be running.

Also, your root filesystem will be mounted read-only, as init is expected to remount it read-write as needed.

# mount -o remount,rw /

Change Root Password

Once you’ve remounted the root filesystem, all that’s needed is to run the passwd command.

# passwd

Since you’re running the command as root you won’t need to provide your existing password, and will only need to type a new password twice.

Now of course you simply need to remember that password in order to ensure you don’t need to do this again.

Reboot Safely

You now cannot follow the standard reboot process here, as you’re only running one process.

Therefore it’s important to put your root filesystem back into read-only before powering off your machine:

# mount -o remount,ro /

Once you’ve done that you just need to hold down the power button until the machine completely powers off or pull the plug.

And then you’re done! Boot the computer again and you’ll have everything working as normal, with a root password you remember.

  1. Not that this is the only reason, anyone with physical access to your machine could also boot it into another operating system they control, or just remove your storage device and put it into another computer, or probably other things I’m not thinking of now. You should always encrypt your devices. 

  2. The editor uses emacs-like keybindings. The manual includes a list of all the options available. 

02 June, 2024 12:00AM

June 01, 2024

hackergotchi for Guido Günther

Guido Günther

Free Software Activities May 2024

A short status update of what happened on my side last month. A broken gcovr in Debian triggered a bit of busy work but 0.39.0 came out nicely nevertheless. We also reduced build time quiet a bit in phosh and phoc.

If you want to support my work see donations.

01 June, 2024 05:17PM

Ian Jackson

What your vote is worth - a back of the envelope calculation

tl;dr: Your vote really counts!

Each vote in a UK General Election is worth maybe £100,000 - to you and all your fellow citizens taken together. If you really care about the welfare of everyone affected by actions of the UK government, then it’s worth that to you too.


It seems a common perception that one vote, in amongst all those millions, doesn’t really matter. So maybe it’s not worth voting. But, voting is (largely) what determines what the government does - and the government is big. It’s as big as all the people.

If you are the kind of person who cares about what happens to everyone in your polity and indeed everyone its actions affect, then even your one vote is very important indeed.

A method for back of the envelope calculation

It would be nice to give a quantitative estimate. Many things in our society are measured in money, so let’s try taking a stab at calculating the money value of your vote.

The argument I’m going to make is this: the government (by which I include the legislature), which is selected by our votes, decides how to spend the national budget.

So, basically, I’m going to divide the budget, by the electorate.

UK Parliament

UK Parliamentary elections decide not only the House of Commons, but, through that, the government. The upper house, the House of Lords, has very limited influence. So I think it’s fair to regard the Parliamentary election as, simply, controlling that budget.

Being lazy, I’m going to use Wikipedia data. We have the size of the electorate, for 2019, 47.6 million. But your influence isn’t shared with the whole electorate, only with the other people who also vote. Turnout in 2019 was 67.3%. The 2019 budget isn’t listed but I’ll just average the 2018 and March 2020 figures £842bn and £873bn, so £857 billion. (Strictly speaking I should add up the budgets for the period of the Parliament, but that seems like a lot of effort.)

There’s a discrepancy in the timescale we need to account for. Your vote influences the budgets for several years, depending how long it is until the next election. Taking Wikipedia’s list of elections this century there’ve been 7 in 24 years. So that’s an average of about 3.4y.

So, multiplying it through, we have (£857b * (24 / 7)) / (47.6M * 67.3%), giving a guess at the value of your UK General Election vote:


European Parliament

2022 budget for the European Union (Wikipedia again) was €170.6 bn.

The last election, in 2019, had a turnout of 198,352,638. Each EU Parliament lasts 5 years.

The Parliament, however, shares responsibility for the budget with the European Council, which is controlled, ultimately, by national governments. We have to pick a numerical value for the Parliament’s share of the influence. Over the past years the Parliament has gradually been more willing to exercise its powers in this area. I’m going to arbitrarily call its share 50%.

The calculation, then, is €170.6 bn * 5 * 50% / 198M, giving a guess at the value of your EU Parliamentary Election vote:


This much smaller figure reflects simply that the EU doesn’t spend very much money, for a polity of its size. (Those stories in the British press giving the impression that the EU is massively wasteful are, simply, lies.)

The interaction of this calculation with the Council’s share of the influence, and with national budgets, is a bit of a question, but given the much smaller amounts involved, it doesn’t seem worth thinking about that too hard.

Only if you care about other people as much as yourself!

All of this is only true for you if you value and want to help everyone in your society. That includes immigrants, women, unemployed people, disabled people, people who are much poorer or richer than you, etc.

If you think about it in purely personal terms, your vote is hardly worth anything - because while the effect of your vote, overall, is very large, that effect is shared by everyone in your polity. So if you only care about yourself, voting is a total waste of time. The more selfish and xenophobic and racist and so on you are - caring only about people like yourself - the less your vote is worth.

This is why voting is rightly seen as a civic duty. I just spent £30 to courier my EP vote to Den Haag. That only makes sense because I’m very willing to spend that £30 to try to improve the spending of the €2000 or so that’s my share of the EU budget.

This is a very rough analysis

These calculations neglect a lot of very important things: politics isn’t just about the allocation of resources. It’s also about values, and bad politics can seriously harm people.

Arguably many of those effects of your vote, are much more important than just how the budget is set and spent.

It would be interesting to see an attempt at a similar analysis but for taking into account life and death questions like hate crime, traffic violence, healthcare, refugees’ welfare, and so on. I’m not sure how to approach that. Maybe some real social scientists have done so? References welcome.

Also, even on its own terms, this analysis is very rough and ready. We haven’t modelled the ability of the government to change its tax rates; perhaps we should be multiplying GDP (or some other better measure) by 90% percentile total tax rate amongst “countries like this one”. The amount of influence that can be wielded by one vote is probably nonlinear in the size of the political faction, but IDK in which direction. In unfair voting systems like the UK’s, some people’s votes are worth much more than others. In a very marginal constituency, which is a target seat, your vote might be worth tens of millions. In a safe seat, it might “only” be worth a few thousand. And in practical terms you don’t get to choose precisely the policies you want; you have to pick a party, which is sometimes very much a question of the lesser evil.

So, there is much I haven’t modelled. But the key point stands:


Although your vote is diluted by everyone else’s votes, together, we control the government, which affects us all. So if you care about the whole of society, the big numbers in the divisor, and the numerator, cancel out.

You can think of your vote as controlling one citizen’s worth of government activity.

edited 2024-06-01 09:40 Z to fix a grammar botch

comment count unavailable comments

01 June, 2024 09:40AM

Russ Allbery

Review: I Shall Wear Midnight

Review: I Shall Wear Midnight, by Terry Pratchett

Series: Discworld #38
Publisher: Harper
Copyright: 2010
Printing: 2011
ISBN: 0-06-143306-3
Format: Trade paperback
Pages: 355

I Shall Wear Midnight is the 38th Discworld novel and the 4th Tiffany Aching novel. This is not a good place to start reading.

Tiffany has finished her training and has returned to her home on the Chalk, taking up her duties as the local witch. There are a lot of those, because there's a lot that needs doing. In some cases, such as taking away the pain of the old Duke, they involve things that require magic and that only Tiffany can do. In many other cases, other people could pick up some of the work, but they lack Tiffany's sense of duty and willingness to pay attention.

The people of the Chalk have always been a bit suspicious of witches, in part because the job was done for so long by Tiffany's grandmother and no one thought she was a witch. (She was a witch.) Of late, however, that suspicion seems to be getting worse. It comes to a head when Tiffany is accused of theft and worse by the old Duke's maid, a woman with very fixed ideas about the evils of witches. Tiffany has to sort out what's going on and clear herself, all while navigating her now-awkward relationship with the Duke's son Roland, his unimpressive fiancee, and his spectacularly annoying aunt.

Ah, this is the stuff. This is exactly the Tiffany Aching novel that I have been hoping Pratchett would write. It's pure, snarky competence porn from start to finish.

"I'm a witch. It's what we do. When it's nobody else's business, it's my business."

One of the things that I adore about this series is how well Pratchett shows the different ways in which one can be a witch. Granny Weatherwax out-thinks everyone and nudges (or shoves) people in the right direction, but her natural tendency is to be icy and a bit frightening. Nanny Ogg is that person you can't help but talk to, who may seem happy-go-lucky and hedonistic but who can effortlessly change the mood of a room. And Tiffany is stubborn duty and blunt practicality, which fits the daughter of shepherds. In previous books, we've watched Tiffany as a student, learning the practicalities of being a witch. This is the book where she realizes how much she knows and how much easier the world is to navigate when she's in her own territory.

There is a wonderful scene, late in this book, where Pratchett shows Nanny Ogg at her best, doing the kinds of things that only Nanny Ogg can do. Both Tiffany and the reader are in awe.

I should have learned this, she thought. I wanted to learn fire, and pain, but I should have learned people.

And it's true that Nanny Ogg can do things that Tiffany can't. But what makes this book so great is that it shows how Tiffany's personality and her training come together with her knowledge of the Chalk. She may not know people, in general, but she knows her neighbors and how they think. She doesn't manage them the way that Nanny Ogg would; she's better at solving different kinds of problems, in different ways. But they're the right ways, and the right problems, for her home.

This is another Discworld novel with a forgettable villain that's more of a malevolent force of nature than a character in its own right. It's also another Discworld novel where Pratchett externalizes a human tendency into a malevolent force that can possess people. I have mixed feelings about this narrative approach. That externalization of evil into (in essence) demons has been repeatedly used to squirm out of responsibility and excuse atrocities, and it neatly avoids having to wrestle with the hard questions of prejudice and injustice and why apparently good people do awful things.

I think some of those weaknesses persist even in Pratchett's hands, but I think what he was attempting with that approach in this book is to show how almost no one is immune to nastier ideas that spread through society. Rather than using the externalization of evil as an excuse, he's using it as a warning. With enough exposure to those ideas, they start sounding tempting and partly credible even to people who would never have embraced them earlier. Pratchett also does a good job capturing the way prejudice can start from thoughtless actions that have more to do with the specific circumstances of someone's life than any coherent strategy.

Still, the one major complaint I have about this book is that the externalization of evil is an inaccurate portrayal of the world, and this catches up with Pratchett at the ending. Postulating an external malevolent force reduces evil to something that can be puzzled out and decisively defeated, thus resolving the problem. Sadly, this is not how humans actually work.

I'll forgive that structural flaw, though, because the rest of this book is so good. It's rare that a plot twist in a Discworld novel surprises me — twisty plots are not Pratchett's strength — but this one did. I will not spoil the surprise, but one of the characters is not quite who they seem to be, and Tiffany's reactions once she figures that out are one of my favorite parts of this book. Pratchett is making a point about assumptions, observation, and the importance of being willing to change one's mind about someone when you know more, and I thought it was very well done.

But, most of all, I enjoyed reading about Tiffany being calm, competent, determined, and capable. There's also a bit of an unexpected romance plot that's one of my favorite types: the person who notices that you're doing a lot of work and quietly steps in and starts helping while paying attention to what's needed and not taking over. And it's full of the sort of pithy moral wisdom that makes Discworld such a delight to read.

"There have been times, lately, when I dearly wished that I could change the past. Well, I can't, but I can change the present, so that when it becomes the past it will turn out to be a past worth having."

This was just what I wanted. Highly recommended.

Followed by Snuff in publication order. The next (and last, sadly) Tiffany Aching book is The Shepherd's Crown.

Rating: 9 out of 10

01 June, 2024 04:29AM

Russell Coker

hackergotchi for Junichi Uekawa

Junichi Uekawa

June already.

June already. Thinking about what to do in Debconf.

01 June, 2024 12:56AM by Junichi Uekawa

May 31, 2024

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppArmadillo on CRAN: Upstream Bugfix

armadillo image

Armadillo is a powerful and expressive C++ template library for linear algebra and scientific computing. It aims towards a good balance between speed and ease of use, has a syntax deliberately close to Matlab, and is useful for algorithm development directly in C++, or quick conversion of research code into production environments. RcppArmadillo integrates this library with the R environment and language–and is widely used by (currently) 1151 other packages on CRAN, downloaded 34.6 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint / vignette) by Conrad and myself has been cited 584 times according to Google Scholar.

Conrad released a new upstream bugfix yesterday (to improve views of sparse matrices). We uploaded it yesterday too but it once agfain took a day for the hard-working CRAN maintainers to concur that the two NOTEs from reverse-dependency checking over 1100 packages were in a fact false positves. And so it appeared on CRAN earlier today. We also increased the versioned dependency on Rcpp to match the use of optional entry-point headers Rcpp/Light, Rcpp/Lighter and Rcpp/Lightest. No other changes were made.

The set of changes since the last CRAN release follows.

Changes in RcppArmadillo version (2024-05-30)

  • Upgraded to Armadillo release 12.8.4 (Cortisol Injector)

    • Faster handling of sparse submatrix views
  • Update versioned Depends on Rcpp to 1.0.8 or later to match use of Light/Lighter/Lightest headers.

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

31 May, 2024 09:57PM

hackergotchi for Bits from Debian

Bits from Debian

New Debian Developers and Maintainers (March and April 2024)

The following contributors got their Debian Developer accounts in the last two months:

  • Patrick Winnertz (winnie)
  • Fabian Gruenbichler (fabiang)

The following contributors were added as Debian Maintainers in the last two months:

  • Juri Grabowski
  • Tobias Heider
  • Jean Charles Delépine
  • Guilherme Puida Moreira
  • Antoine Le Gonidec
  • Arthur Barbosa Diniz


31 May, 2024 04:00PM by Jean-Pierre Giraud

May 30, 2024

hackergotchi for Matthew Palmer

Matthew Palmer

GitHub's Missing Tab

Visit any GitHub project page, and the first thing you see is something that looks like this:

screenshot of the GitHub repository page, showing the Code, Issues, and Pull Requests tabs

“Code”, that’s fairly innocuous, and it’s what we came here for. The “Issues” and “Pull Requests” tabs, with their count of open issues, might give us some sense of “how active” the project is, or perhaps “how maintained”. Useful information for the casual visitor, undoubtedly.

However, there’s another user community that visits this page on the regular, and these same tabs mean something very different to them.

I’m talking about the maintainers (or, more commonly, maintainer, singular). When they see those tabs, all they see is work. The “Code” tab is irrelevant to them – they already have the code, and know it possibly better than they know their significant other(s) (if any). “Issues” and “Pull Requests” are just things that have to be done.

I know for myself, at least, that it is demoralising to look at a repository page and see nothing but work. I’d be surprised if it didn’t contribute in some small way to maintainers just noping the fudge out.

A Modest Proposal

So, here’s my thought. What if instead of the repo tabs looking like the above, they instead looked like this:

modified screenshot of the GitHub repository page, showing a new Kudos tab, with a smiley face icon, between the Code and Issues tabs

My conception of this is that it would, essentially, be a kind of “yearbook”, that people who used and liked the software could scribble their thoughts on. With some fairly straightforward affordances elsewhere to encourage its use, it could be a powerful way to show maintainers that they are, in fact, valued and appreciated.

There are a number of software packages I’ve used recently, that I’d really like to say a general “thanks, this is awesome!” to. However, I’m not about to make the Issues tab look even scarier by creating an “issue” to say thanks, and digging up an email address is often surprisingly difficult, and wouldn’t be a public show of my gratitude, which I believe is a valuable part of the interaction.

You Can’t Pay Your Rent With Kudos

Absolutely you cannot. A means of expressing appreciation in no way replaces the pressing need to figure out a way to allow open source developers to pay their rent. Conversely, however, the need to pay open source developers doesn’t remove the need to also show those people that their work is appreciated and valued by many people around the world.

Anyway, who knows a senior exec at GitHub? I’ve got an idea I’d like to run past them…

30 May, 2024 12:00AM by Matt Palmer (

May 28, 2024

Russell Coker

Creating a Micro Users’ Group

Fosdem had a great lecture Building an Open Source Community One Friend at a Time [1]. I recommend that everyone who is involved in the FOSS community watches this lecture to get some ideas.

For some time I’ve been periodically inviting a few friends to visit for lunch, chat about Linux, maybe do some coding, and watch some anime between coding. It seems that I have accidentally created a micro users’ group.

LUGs were really big in the mid to late 90s and still quite vibrant in the early 2000’s. But they seem to have decreased in popularity even before Covid19 and since Covid19 a lot of people have stopped attending large meetings to avoid health risks. I think that a large part of the decline of users’ groups has been due to the success of YouTube. Being able to choose from thousands of hours of lectures about computers on YouTube is a disincentive to spending the time and effort needed to attend a meeting with content that’s probably not your first choice of topic. Attending a formal meeting where someone you don’t know has arranged a lecture might not have a topic that’s really interesting to you. Having lunch with a couple of friends and watching a YouTube video that one of your friends assures you is really good is something more people will find interesting.

In recent times homeschooling [2] has become more widely known. The same factors that allow learning about computers at home also make homeschooling easier. The difference between the traditional LUG model of having everyone meet at a fixed time for a lecture and a micro LUG of a small group of people having an informal meeting is similar to the difference between traditional schools and homeschooling.

I encourage everyone to create their own micro LUG. All you have to do is choose a suitable time and place and invite some people who are interested. Have a BBQ in a park if the weather is good, meet at a cafe or restaurant, or invite people to visit you for lunch on a weekend.

28 May, 2024 10:08PM by etbe

May 27, 2024

Thomas Koch

Thank you Republika Srpska

Posted on May 27, 2024
Tags: debian, peace

In 2011 I went to my first DebConf and was very overwhelmed not only by DebConf, Debian and all the interesting sessions. I was also overwhelmed by the hospitality of the Republika Srpska. Our venue was the cultural center “Banski Dvor” in Banja Luka, “the most important cultural center of the city of Banja Luka and the Serb Republic”1.

We were accommodated across the street of the venue in the four star “Hotel Bosna”. The DebConf dinner was in a government building.

I learned, that in addition to the sponsorship of the venue and accommodation, the financial sponsorship was gracious enough that there was money left for the following DebConfs.

I remember this now, because the United Nations General Assembly voted on a resolution that might have very severe implications in the coming weeks for the Republika Srpska.

The matter is extremely tricky, complex and not a topic for Planet Debian.

I think it is the least I can do to thank my former hosts, to get myself informed about what is going on there.


27 May, 2024 12:00AM

May 26, 2024

Russell Coker


USB-A is the original socket for USB at the PC end. There are 2 variants of it, the first is for USB 1.1 to USB 2 and the second is for USB 3 which adds extra pins in a plug and socket compatible manner – you can plug a USB-A device into a USB-A socket without worrying about the speeds of each end as long as you don’t need USB 3 speeds.

The differences between USB-A and USB-C are:

  1. USB-C has the same form factor as Thunderbolt and the Thunderbolt protocol can run over it if both ends support it.
  2. USB-C generally supports higher power modes for charging (like 130W for Dell laptops, monitors, and plugpacks) but there’s no technical reason why USB-A couldn’t do it. You can buy chargers that do 60W over USB-A which could power one of our laptops via a USB-A to USB-C cable. So high power USB-A is theoretically possible but generally you won’t see it.
  3. USB-C has “DisplayPort alternate mode” which means using some of the wires for DisplayPort.
  4. USB-C is more likely to support the highest speeds than USB-A sockets for “super speed” etc. This is not a difference in the standards just a choice made by manufacturers.

While USB-C tends to support higher power delivery modes in actual implementations for connecting to a PC the PC end seems to only support lower power modes regardless of port. I think it would be really good if workstations could connect to monitors via USB-C and provide power, DisplayPort, and keyboard, mouse, etc over the same connection. But unfortunately the PC and monitor ends don’t appear to support such things.

If you don’t need any of those benefits in the list above (IE you are using USB for almost anything we do other than connecting a laptop to a dock/monitor/charger) then USB-A will do the job just as well as USB-C. The choice of which type to use should be based on price and which ports are available, EG My laptop has 2*USB-C ports and 2*USB-A so given that one USB-C port is almost always used for the monitor or for charging I don’t really want to use USB-C for anything else to avoid running out of ports.

When buying USB devices you can’t always predict which systems you will need to connect them to. Currently there are a lot of systems without USB-C that are working well and have no need to be replaced. I haven’t yet seen a system where the majority of ports are USB-C but that will probably happen in the next few years. Maybe in 2027 there will be PCs on sale with only two USB-A sockets forcing people who don’t want to use a USB hub to save both of them for keyboard and mouse. Currently USB-C keyboards and mice are available on AliExpress but they are expensive and I haven’t seen them in Australian stores. Most computer users don’t wear out keyboards or mice so a lot of USB-A keyboard and mice will be in service for a long time. As an aside there are still many PCs with PS/2 keyboard and mouse ports in service so these things don’t go away for a long time.

There is one corner case where USB-C is convenient which is when you want to connect a mass storage device for system recovery or emergency backup, want a high speed, and don’t want to spend time figuring out which of the ports are “super speed” (which can be difficult at the back of a PC with poor lighting). With USB-C you can expect a speed of at least 5Gbit/s and don’t have to worry about accidentally connecting to a USB 2 port as is the situation with USB-A.

For my own use the only times that I prefer USB-C over USB-A are for devices to connect to phones. Eventually I’ll get a laptop that only has USB-C ports and this will change, but even then adaptors are possible.

For someone who doesn’t know the details of how things works it’s not unreasonable to just buy the newest stuff and assume it’s better as it usually is. But hopefully blog posts like this can help people make more informed decisions.

26 May, 2024 10:31PM by etbe

May 25, 2024

hackergotchi for Steve Kemp

Steve Kemp

The CP/M emulator is working well

In my recent posts I've talked about implementing BDOS and BIOS syscalls for my cp/m emulator. I've now implemented enough of the calls that I can run many of the standard binaries:

  • The Aztech C Compiler
  • Microsoft BASIC
  • Turbo Pascal
  • Wordstar
  • etc

Of course I've not implemented all the syscalls, so the emulation isn't 100% perfect and many binaries won't run. But I sent myself on a detour by implementing extra syscalls, custom syscalls.

Traditionally CP/M systems are "rebooted" by pressing Ctrl-C at the CCP prompt. I thought that was something I'd press by accident so I implemented the restart behaviour only when the user pressed Ctrl-C twice in a row. But then I added a custom syscall that lets you change hte value:

The Ctrl-C count is currently set to 2
A>ctrlc 1
The Ctrl-C count is currently set to 1

So you can now change the value at runtime. Similarly there is support for switching CCP at runtime, and even changing the default output-device from ADM-3A to ANSI, or vice-versa. It's kinda neat to make these kind of extensions, and happily the traditional BIOS has two syscalls reserved for custom use so I just used one of those.

I've added support for testing whether a binary is running under my emulator, or not, using a custom syscall. So I can run:

This binary is running under cpmulator:

cpmulator unreleased

On another emulator I see this:

Illegal BIOS call 31
No, this binary is not running under cpmulator.

Anyway I'm happy with the current state of things, and I fixed a couple of bugs which means I now have support for SUBMIT.COM which is a real time-saver.

25 May, 2024 12:00PM

hackergotchi for Gunnar Wolf

Gunnar Wolf

How computers make books • from graphics rendering, search algorithms, and functional programming to indexing and typesetting

This post is a review for Computing Reviews for How computers make books • from graphics rendering, search algorithms, and functional programming to indexing and typesetting , a book published in Manning

If we look at the age-old process of creating books, how many different areas can a computer help us with? And how can each of them be used to teach computer science (CS) fundamentals to a nontechnical audience? This is the premise of John Whitington’s enticing book and the result is quite amazing.

The book immediately drew my attention when looking at the titles available for review. After all, my initiation into computing as a kid was learning the LaTeX typesetting system while my father worked on his first book on scientific language and typography [1]. Whitington picks 11 different technical aspects of book production, from how dots of ink are transferred to a white page and how they are made into controllable, recognizable shapes, all the way to forming beautiful typefaces and the nuances of properly addressing white-space to present aesthetically pleasing paragraphs, building it all into specific formats aimed at different ends.

But if we dig beyond just the chapter titles, we will find a very interesting book on CS that, without ever using technical language or notation, presents aspects as varied as anti-aliasing, vector and raster images, character sets such as ASCII and Unicode, an introduction to programming, input methods for different writing systems, efficient encoding (compression) methods, both for text and images, lossless and lossy, and recursion and dithering methods. To my absolute surprise, while the author thankfully spared the reader the syntax usually associated with LISP-related languages, the programming examples clearly stem from the LISP school, presenting solutions based on tail recursion. Of course, it is no match for Donald Knuth’s classic book on this same topic [2], but could very well be a primer for readers to approach it.

The book is light and easy to read, and keeps a very informal, nontechnical tone throughout. My only complaint relates to reading it in PDF format; the topic of this book, and the care with which the images were provided by the author, warrant high resolution. The included images are not only decorative but an integral part of the book. Maybe this is specific to my review copy, but all of the raster images were in very low resolution.

This book is quite different from what readers may usually expect, as it introduces several significant topics in the field. CS professors will enjoy it, of course, but also readers with a humanities background, students new to the field, or even those who are just interested in learning a bit more.


  1. Sánchez y Gándara, A.; Magariños Lamas, F.; Wolf, K. B., Manual de lenguaje y tipografía científica en castellano. Trillas, Mexico City, Mexico, 1986,

  2. Knuth, D. E. Digital typographyCSLI Lecture Notes: CSLI Lecture Notes. CSLI Publications, Stanford, CA, 1999,

25 May, 2024 12:11AM

May 24, 2024

Julian Andres Klode

Observations in Debian dependency solving

In my previous blog, I explored The New APT 3.0 solver. Since then I have been at work in the test suite making tests pass and fixing some bugs.

You see for all intents and purposes, the new solver is a very stupid naive DPLL SAT solver (it just so happens we don’t actually have any pure literals in there). We can control it in a bunch of ways:

  1. We can mark packages as “install” or “reject”
  2. We can order actions/clauses. When backtracking the action that came later will be the first we try to backtrack on
  3. We can order the choices of a dependency - we try them left to right.

This is about all that we really want to do, we can’t go if we reach a conflict, say “oh but this conflict was introduced by that upgrade, and it seems more important, so let’s not backtrack on the upgrade request but on this dependency instead.”.

This forces us to think about lowering the dependency problem into this form, such that not only do we get formally correct solutions, but also semantically correct ones. This is nice because we can apply a systematic way to approach the issue rather than introducing ad-hoc rules in the old solver which had a “which of these packages should I flip the opposite way to break the conflict” kind of thinking.

Now our test suite has a whole bunch of these semantics encoded in it, and I’m going to share some problems and ideas for how to solve them. I can’t wait to fix these and the error reporting and then turn it on in Ubuntu and later Debian (the defaults change is a post-trixie change, let’s be honest).

apt upgrade is hard

The apt upgrade commands implements a safe version of dist-upgrade that essentially calculates the dist-upgrade, and then undoes anything that would cause a package to be removed, but it (unlike its apt-get counterpart) allows the solver to install new packages.

Now, consider the following package is installed:

X Depends: A (= 1) | B

An upgrade from A=1 to A=2 is available. What should happen?

The classic solver would choose to remove X in a dist-upgrade, and then upgrade A, so it’s answer is quite clear: Keep back the upgrade of A.

The new solver however sees two possible solutions:

  1. Install B to satisfy X Depends A (= 1) | B.
  2. Keep back the upgrade of A

Which one does it pick? This depends on the order in which it sees the upgrade action for A and the dependency, as it will backjump chronologically. So

  1. If it gets to the dependency first, it marks A=1 for install to satisfy A (= 1). Then it gets to the upgrade request, which is just A Depends A (= 2) | A (= 1) and sees it is satisfied already and is content.

  2. If it gets to the upgrade request first, it marks A=2 for install to satisfy A (= 2). Then later it gets to X Depends: A (= 1) | B, sees that A (= 1) is not satisfiable, and picks B.

We have two ways to approach this issue:

  1. We always order upgrade requests last, so they will be kept back in case of conflicting dependencies
  2. We require that, for apt upgrade a currently satisfied dependency must be satisfied by currently installed packages, hence eliminating B as a choice.

Recommends are hard too

See if you have a X Recommends: A (= 1) and a new version of A, A (= 2), the solver currently will silently break the Recommends in some cases.

But let’s explore what the behavior of a X Recommends: A (= 1) in combination with an available upgrade of A (= 2) should be. We could say the rule should be:

  • An upgrade should keep back A instead of breaking the Recommends
  • A dist-upgrade should either keep back A or remove X (if it is obsolete)

This essentially leaves us the same choices as for the previous problem, but with an interesting twist. We can change the ordering (and we already did), but we could also introduce a new rule, “promotions”:

A Recommends in an installed package, or an upgrade to that installed package, where the Recommends existed in the installed version, that is currently satisfied, must continue to be satisfied, that is, it effectively is promoted to a Depends.

This neatly solves the problem for us. We will never break Recommends that are satisfied.

Likewise, we already have a Recommends demotion rule:

A Recommends in an installed package, or an upgrade to that installed package, where the Recommends existed in the installed version, that is currently unsatisfied, will not be further evaluated (it is treated like a Suggests is in the default configuration).

Whether we should be allowed to break Suggests with our decisions or not (the old autoremover did not, for instance) is a different decision. Should we promote currently satisified Suggests to Depends as well? Should we follow currently satisified Suggests so the solver sees them and doesn’t autoremove them, but treat them as optional?

tightening of versioned dependencies

Another case of versioned dependencies with alternatives that has complex behavior is something like

X Depends: A (>= 2) | B
X Recommends: A (>= 2) | B

In both cases, installing X should upgrade an A < 2 in favour of installing B. But a naive SAT solver might not. If your request to keep A installed is encoded as A (= 1) | A (= 2), then it first picks A (= 1). When it sees the Depends/Recommends it will switch to B.

We can solve this again as in the previous example by ordering the “keep A installed” requests after any dependencies. Notably, we will enqueue the common dependencies of all A versions first before selecting a version of A, so something may select a version for us.

version narrowing instead of version choosing

A different approach to dealing with the issue of version selection is to not select a version until the very last moment. So instead of selecting a version to satisfy A (>= 2) we instead translate

Depends: A (>= 2)

into two rules:

  1. The package selection rule:

     Depends: A

    This ensures that any version of A is installed (i.e. it adds a version choice clause, A (= 1) | A (= 2) in an example with two versions for A.

  2. The version narrowing rule:

     Conflicts: A (<< 2)

    This outright would reject a choice of A (= 1).

So now we have 3 kinds of clauses:

  1. package selection
  2. version narrowing
  3. version selection

If we process them in that order, we should surely be able to find the solution that best matches the semantics of our Debian dependency model, i.e. selecting earlier choices in a dependency before later choices in the face of version restrictions.

This still leaves one issue: What if our maintainer did not use Depends: A (>= 2) | B but e.g. Depends: A (= 3) | B | A (= 2). He’d expect us to fall back to B if A (= 3) is not installable, and not to B. But we’d like to enqueue A and reject all choices other than 3 and 2. I think it’s fair to say: “Don’t do that, then” here.

Implementing strict pinning correctly

APT knows a single candidate version per package, this makes the solver relatively deterministic: It will only ever pick the candidate, or an installed version. This also happens to significantly reduce the search space which is good - less backtracking. An uptodate system will only ever have one version per package that can be installed, so we never actually have to choose versions.

But of course, APT allows you to specify a non-candidate version of a package to install, for example:

apt install foo/oracular-proposed

The way this works is that the core component of the previous solver, which is the pkgDepCache maintains what essentially amounts to an overlay of the policy that you could see with apt-cache policy.

The solver currently however validates allowed version choices against the policy directly, and hence finds these versions are not allowed and craps out. This is an interesting problem because the solver should not be dependent on the pkgDepCache as the pkgDepCache initialization (Building dependency tree...) accounts for about half of the runtime of APT (until the Y/n prompt) and I’d really like to get rid of it.

But currently the frontend does go via the pkgDepCache. It marks the packages in there, building up what you could call a transaction, and then we translate it to the new solver, and once it is done, it translates the result back into the pkgDepCache.

The current implementation of “allowed version” is implemented by reducing the search space, i.e. every dependency, we outright ignore any non-allowed versions. So if you have a version 3 of A that is ignored a Depends: A would be translated into A (= 2) | A (= 1).

However this has two disadvantages. (1) It means if we show you why A could not be installed, you don’t even see A (= 3) in the list of choices and (2) you would need to keep the pkgDepCache around for the temporary overrides.

So instead of actually enforcing the allowed version rule by filtering, a more reasonable model is that we apply the allowed version rule by just marking every other version as not allowed when discovering the package in the from depcache translation layer. This doesn’t really increase the search space either but it solves both our problem of making overrides work and giving you a reasonable error message that lists all versions of A.

pulling up common dependencies to minimize backtracking cost

One of the common issues we have is that when we have a dependency group

`A | B | C | D`

we try them in order, and if one fails, we undo everything it did, and move on to the next one. However, this isn’t perhaps the best choice of operation.

I explained before that one thing we do is queue the common dependencies of a package (i.e. dependencies shared in all versions) when marking a package for install, but we don’t do this here: We have already lowered the representation of the dependency group into a list of versions, so we’d need to extract the package back out of it.

This can of course be done, but there may be a more interesting solution to the problem, in that we simply enqueue all the common dependencies. That is, we add n backtracking levels for n possible solutions:

  1. We enqueue the common dependencies of all possible solutions deps(A)&deps(B)&deps(C)&deps(D)
  2. We decide (adding a decision level) not to install D right now and enqueue deps(A)&deps(B)&deps(C)
  3. We decide (adding a decision level) not to install C right now and enqueue deps(A)&deps(B)
  4. We decide (adding a decision level) not to install B right now and enqueue A

Now if we need to backtrack from our choice of A we hopefully still have a lot of common dependencies queued that we do not need to redo. While we have more backtracking levels, each backtracking level would be significantly cheaper, especially if you have cheap backtracking (which admittedly we do not have, yet anyway).

The caveat though is: It may be pretty expensive to find the common dependencies. We need to iterate over all dependency groups of A and see if they are in B, C, and D, so we have a complexity of roughly

#A * (#B+#C+#D)

Each dependency group we need to check i.e. is X|Y in B meanwhile has linear cost: We need to compare the memory content of two pointer arrays containing the list of possible versions that solve the dependency group. This means that X|Y and Y|X are different dependencies of course, but that is to be expected – they are. But any dependency of the same order will have the same memory layout.

So really the cost is roughly N^4. This isn’t nice.

You can apply various heuristics here on how to improve that, or you can even apply binary logic:

  1. Enqueue common dependencies of A|B|C|D
  2. Move into the left half, enqueue of A|B
  3. Again divide and conquer and select A.

This has a significant advantage in long lists of choices, and also in the common case, where the first solution should be the right one.

Or again, if you enqueue the package and a version restriction instead, you already get the common dependencies enqueued for the chosen package at least.

24 May, 2024 08:57AM

May 22, 2024

hackergotchi for Evgeni Golov

Evgeni Golov

Upgrading CentOS Stream 8 to CentOS Stream 9 using Leapp

Warning to the Planet Debian readers: the following post might shock you, if you're used to Debian's smooth upgrades using only the package manager.


Contrary to distributions like Debian and Fedora, RHEL can't be upgraded using the package manager alone.

Instead there is a tool called Leapp that takes care of orchestrating the update and also includes a set of checks whether a system can be upgraded at all. Have a look at the RHEL documentation about upgrading if you want more details on the process itself.

You might have noticed that the title of this post says "CentOS Stream" but here I am talking about RHEL. This is mostly because Leapp was originally written with RHEL in mind.

Upgrading CentOS 7 to EL8

When people started pondering upgrading their CentOS 7 installations, AlmaLinux started the ELevate project to allow upgrading CentOS 7 to CentOS Stream 8 but also to AlmaLinux 8, Rocky 8 or Oracle Linux 8.

ELevate was essentially Leapp with patches to allow working on CentOS, which has different package signature keys, different OS release versioning, etc.

Sadly these patches were never merged back into Leapp.

Making Leapp work with CentOS Stream 8 (and other distributions)

At some point I noticed that things weren't moving and EL8 to EL9 upgrades were coming closer (and I had my own systems that I wanted to be able to upgrade in place).

Annoyed-Evgeni-Development is best development? Not sure, but it produced a set of patches that allowed some movement:

However, this is not yet the end of the story. At least convert dot-less CentOS versions to X.999 is open, and another followup would be needed if we go that route. But I don't expect this to be merged soon, as the patch is technically wrong - yet it makes things mostly work.

The big problem here is that CentOS Stream doesn't have X.Y versioning, just X as it's a constant stream with no point releases. Leapp however relies on X.Y versioning to know which package changes it needs to perform. Pretending CentOS Stream 8 is "RHEL" 8.999 works if you assume that Stream is always ahead of RHEL.

This is however a CentOS only problem. I still need to properly test that, but I'd expect things to work fine with upstream Leapp on AlmaLinux/Rocky if you feed it the right signature and repository data.

Actually upgrading CentOS Stream 8 to CentOS Stream 9 using Leapp

Like I've already teased in my HPE rant, I've actually used that code to upgrade to CentOS Stream 9. I've also used it to upgrade a server at home that's responsible for running important containers like Home Assistant and UniFi. So it's absolutely battle tested and production grade! It's also hungry for kittens.

As mentioned above, you can't just use upstream Leapp, but I have a Copr: evgeni/leapp.

# dnf copr enable evgeni/leapp
# dnf install leapp leapp-upgrade-el8toel9

Apart from the software, we'll also need to tell it which repositories to use for the upgrade.

# vim /etc/leapp/files/leapp_upgrade_repositories.repo
name=CentOS Stream $releasever - BaseOS

name=CentOS Stream $releasever - AppStream

Depending on the setup and installed packages, more repositories might be needed. Just make sure that the $stream substitution is not used as Leapp doesn't override that and you'd end up with CentOS Stream 8 repos again.

Once all that is in place, we can call leapp preupgrade and let it analyze the system.

Ideally, the output will look like this:

# leapp preupgrade
                      REPORT OVERVIEW                       

Reports summary:
    Errors:                      0
    Inhibitors:                  0
    HIGH severity reports:       0
    MEDIUM severity reports:     0
    LOW severity reports:        3
    INFO severity reports:       3

Before continuing consult the full report:
    A report has been generated at /var/log/leapp/leapp-report.json
    A report has been generated at /var/log/leapp/leapp-report.txt

                   END OF REPORT OVERVIEW                   

But trust me, it won't ;-)

As mentioned above, Leapp analyzes the system before the upgrade. Some checks can completely inhibit the upgrade, while others will just be logged as "you better should have a look".

Firewalld Configuration AllowZoneDrifting Is Unsupported

EL7 and EL8 shipped with AllowZoneDrifting=yes, but since EL9 this is not supported anymore. As this can potentially break the networking of the system, the upgrade gets inhibited.

Newest installed kernel not in use

Admit it, you also don't reboot into every new kernel available! Well, Leapp won't let that pass and inhibits the upgrade.

Cannot perform the VDO check of block devices

In EL8 there are two ways to manage VDO: using the dedicated vdo tool and via LVM. If your system uses LVM (it should!) but not VDO, you probably don't have the vdo package installed. But then Leapp can't check if your LVM devices really aren't VDO without the vdo tooling and will inhibit the upgrade. So you gotta install vdo for it to find out that you don't use VDO…

LUKS encrypted partition detected

Yeah. Sorry. Using LUKS? Straight into the inhibit corner!

But hey, if you don't use LUKS for / you can probably get away by deleting the inhibitwhenluks actor. That worked for me, but remember the kittens!

Really upgrading CentOS Stream 8 to CentOS Stream 9 using Leapp

The headings are getting silly, huh?

Anyway, once leapp preupgrade is happy and doesn't throw any inhibitors anymore, the actual (real?) upgrade can be done by calling leapp upgrade.

This will download all necessary packages and create an intermediate initramfs that contains all the things needed for the upgrade and ask you to reboot.

Once booted, the upgrade itself takes somewhere between 5 and 10 minutes. Then another minute or 5 to relabel your disks with the new SELinux policy.

And three reboots (into the upgrade initramfs, into SELinux relabel, into real OS) of a ProLiant DL325 - 5 minutes each? 😿

And then for good measure another one, to flip SELinux from permissive to enforcing.

Are we done yet? Nope.

There are a few post-upgrade tasks you get to do yourself. Yes, the switching of SELinux back to enforcing is one of them. Please don't forget it.

Using the system after the upgrade

A customer once said "We're not running those systems for the sake of running systems, but for the sake of running some application ontop of them". This is very true.

libvirt doesn't support Spice/QXL

In EL9, support for Spice/QXL was dropped, so if you try to boot a VM using it, libvirt will nicely error out with

Error starting domain: unsupported configuration: domain configuration does not support video model 'qxl'

Interestingly, because multiple parts of the VM are invalid, you can't edit it in virt-manager (at least the one in Fedora 39) as removing/fixing one part requires applying the new configuration which is still invalid.

So virsh edit <vm> it is!

Look for entries like

    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
    <audio id='1' type='spice'/>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    <redirdev bus='usb' type='spicevmc'> 
      <address type='usb' bus='0' port='2'/> 
    <redirdev bus='usb' type='spicevmc'> 
      <address type='usb' bus='0' port='3'/> 

and either just delete the or (better) replace them with VNC/cirrus

    <graphics type='vnc' port='-1' autoport='yes'>
      <listen type='address'/>
    <audio id='1' type='none'/>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>

Podman needs re-login to private registries

One of the machines I've updated runs Podman and pulls containers from GitHub which are marked as private. To do so, I have a personal access token that I've used to login to After the CentOS Stream 9 upgrade (which included an upgrade to Podman 5), pulls stopped working with authentication/permission errors. No idea what exactly happened, but a simple podman login fixed this issue quickly.

$ echo ghp_token | podman login -u <user> --password-stdin

shim has an el8 tag

One of the documented post-upgrade tasks is to verify that no EL8 packages are installed, and to remove those if there are any.

However, when you do this, you'll notice that the shim-x64 package has an EL8 version: shim-x64-15-15.el8_2.x86_64.

That's because the same build is used in both CentOS Stream 8 and CentOS Stream 9. Confusing, but should really not be uninstalled if you want the machine to boot ;-)

Are we done yet?

Yes! That's it. Enjoy your CentOS Stream 9!

22 May, 2024 07:19PM by evgeni

May 20, 2024

Russell Coker

Respect and Children

I attended the school Yarra Valley Grammer (then Yarra Valley Anglican School which I will refer to as “YV”) and completed year 12 in 1990. The school is currently in the news for a spreadsheet some boys made rating girls where “unrapeable” was one of the ratings. The school’s PR team are now making claims like “Respect for each other is in the DNA of this school”. I’d like to know when this DNA change allegedly occurred because respect definitely wasn’t in the school DNA in 1990! Before I go any further I have to note that if the school threatens legal action against me for this post it will be clear evidence that they don’t believe in respect. The actions of that school have wronged me, several of my friends, many people who aren’t friends but who I wish they hadn’t had to suffer and I hadn’t had to witness it, and presumably countless others that I didn’t witness. If they have any decency they would not consider legal action but I have learned that as an institution they have no decency so I have to note that they should read the Wikipedia page about the Streisand Effect [1] and keep it in mind before deciding on a course of action.

I think it is possible to create a school where most kids enjoy being there and enjoy learning, where hardly any students find it a negative experience and almost no-one finds it traumatic. But it is not possible to do that with the way schools tend to be run.

When I was at high school there was a general culture that minor sex crimes committed by boys against boys weren’t a problem, this probably applied to all high schools. Things like ripping a boy’s pants off (known as “dakking”) were considered a big joke. If you accept that ripping the pants off an unwilling boy is a good thing (as was the case when I was at school) then that leads to thinking that describing girls as “unrapeable” is acceptable. The Wikipedia page for “Pantsing” [2] has a reference for this issue being raised as a serious problem by the British Secretary of State for Education and Skills Alan Johnson in 2007. So this has continued to be a widespread problem around the world. Has YV become better than other schools in dealing with it or is Dakking and Wedgies as well accepted now as it was when I attended? There is talk about schools preparing kids for the workforce, but grabbing someone’s underpants without consent will result in instant dismissal from almost all employment. There should be more tolerance for making mistakes at school than at work, but they shouldn’t tolerate what would be serious crimes in other contexts. For work environments there have been significant changes to what is accepted, so it doesn’t seem unreasonable to expect that schools can have a similar change in culture.

One would hope that spending 6 years wondering who’s going to grab your underpants next would teach boys the importance of consent and some sympathy for victims of other forms of sexual assault. But that doesn’t seem to happen, apparently it’s often the opposite.

When I was young Autism wasn’t diagnosed for anyone who was capable of having a normal life. Teachers noticed that I wasn’t like other kids, some were nice, but some encouraged other boys to attack me as a form of corporal punishment by proxy – not a punishment for doing anything wrong (detentions were adequate for that) but for being different. The lesson kids will take from that sort of thing is that if you are in a position of power you can mistreat other people and get away with it. There was a girl in my year level at YV who would probably be diagnosed as Autistic by today’s standards, the way I witnessed her being treated was considerably worse than what was described in the recent news reports – but it is quite likely that worse things have been done recently which haven’t made the news yet. If this issue is declared to be over after 4 boys were expelled then I’ll count that as evidence of a cover-up. These things don’t happen in a vacuum, there’s a culture that permits and encourages it.

The word “respect” has different meanings, it can mean “treat a superior as the master” or “treat someone as a human being”. The phrase “if you treat me with respect I’ll treat you with respect” usually means “if you treat me as the boss then I’ll treat you as a human being”. The distinction is very important when discussing respect in schools. If teachers are considered the ultimate bosses whose behaviour can never be questioned then many boys won’t need much help from Andrew Tate in developing the belief that they should be the boss of girls in the same way. Do any schools have a process for having students review teachers? Does YV have an ombudsman to take reports of misbehaving teachers in the way that corporations typically have an ombudsman to take reports about bad managers? Any time you have people whose behaviour is beyond scrutiny or oversight you will inevitably have bad people apply for jobs, then bad things will happen and it will create a culture of bad behaviour. If teachers can treat kids badly then kids will treat other kids badly, and this generally ends with girls being treated badly by boys.

My experience at YV was that kids barely had the status of people. It seemed that the school operated more as a caretaker of the property of parents than as an organisation that cares for people. The current YV website has a Whistleblower policy [3] that has only one occurrence of the word “student” and that is about issues that endanger the health or safety of students. Students are the people most vulnerable to reprisal for complaining and not being listed as an “eligible whistleblower” shows their status. The web site also has a flowchart for complaints and grievances [4] which doesn’t describe any policy for a complaint to be initiated by a student. One would hope that parents would advocate for their children but that often isn’t the case. When discussing the possibility of boys being bullied at school with parents I’ve had them say things like “my son wouldn’t be so weak that he would be bullied”, no boy will tell his parents about being bullied if that’s their attitude! I imagine that there are similar but different issues of parents victim-blaming when their daughter is bullied (presumably substituting immoral for weak) but don’t have direct knowledge of the topic. The experience of many kids is being disrespected by their parents, the school system, and often siblings too. A school can’t solve all the world’s problems but can ideally be a refuge for kids who have problems at home.

When I was at school the culture in the country and the school was homophobic. One teacher when discussing issues such as how students could tell him if they had psychological problems and no-one else to talk to said some things like “the Village People make really good music” which was the only time any teacher said anything like “It’s OK to be gay” (the Village People were the gayest pop group at the time). A lot of the bullying at school had a sexual component to it. In addition to the wedgies and dakking (which while not happening often was something you had to constantly be aware of) I routinely avoided PE classes where a shower was necessary because of a thug who hung around by the showers and looked hungrily at my penis, I don’t know if he had a particular liking to mine or if he stared at everyone that way. Flashing and perving was quite common in change rooms. Presumably as such boy-boy sexual misbehaviour was so accepted that led to boys mistreating girls.

I currently work for a company that is active in telling it’s employees about the possibility of free psychological assistance. Any employee can phone a psychologist to discuss problems (whether or not they are work related) free of charge and without their manager or colleagues knowing. The company is billed and is only given a breakdown of the number of people who used the service and roughly what the issue was (work stress, family, friends, grief, etc). When something noteworthy happens employees are given reminders about this such as “if you need help after seeing a homeless man try to steal a laptop from the office then feel free to call the assistance program”. Do schools offer something similar? With the school fees paid to a school like YV they should be able to afford plenty of psychologist time. Every day I was at YV I saw something considerably worse than laptop theft, most days something was done to me.

The problems with schools are part of larger problems with society. About half of the adults in Australia still support the Liberal party in spite of their support of Christian Porter, Cardinal Pell, and Bruce Lehrmann. It’s not logical to expect such parents to discourage their sons from mistreating girls or to encourage their daughters to complain when they are mistreated. The Anglican church has recently changed it’s policy to suggesting that victims of sexual abuse can contact the police instead of or in addition to the church, previously they had encouraged victims to only contact the church which facilitated cover-ups. One would hope that schools associated with the Anglican church have also changed their practices towards such things.

I approve of the “respect is in our DNA” concept, it’s like Google’s former slogan of “Don’t be evil” which is something that they can be bound to.

Here’s a list of questions that could be asked of schools (not just YV but all schools) by journalists when reporting on such things:

  1. Do you have a policy of not trying to silence past students who have been treated badly?
  2. Do you take all sexual assaults seriously including wedgies and dakking?
  3. Do you take all violence at school seriously? Even if there’s no blood? Even if the victim says they don’t want to make an issue of it?
  4. What are your procedures to deal with misbehaviour from teachers? Do the students all know how to file complaints? Do they know that they can file a complaint if they aren’t the victim?
  5. Does the school have policies against homophobia and transphobia and are they enforced?
  6. Does the school offer free psychological assistance to students and staff who need it? NB This only applies to private schools like YV that have huge amounts of money, public schools can’t afford that.
  7. Are serious incidents investigated by people who are independent of the school and who don’t have a vested interest in keeping things quiet?
  8. Do you encourage students to seek external help from organisations like the ones on the resources list of the Grace Tame Foundation [5]? Having your own list of recommended external organisations would be good too.

Counter Arguments

I’ve had practice debating such things, here’s some responses to common counter arguments.

  • Teachers are nice people how dare you criticise them. Teachers like any other large group of people includes good and bad people. The issue is how well the good people are supported in doing good things, how much effort is spent on tracking down and removing the bad people, and how much effort is spent training people to be the best version of themselves. Also my father worked as a teacher so I really don’t think that all teachers are bad.
  • Teachers are overworked and underpaid and you shouldn’t criticise them. When a school has 25 students in a class whose parents each pay $30,000 per annum the school can afford to pay as much as is necessary. Arguments about teachers being overworked and underpaid are a criticism of the organisation of private school and of government priorities for public schools not a counter argument to criticisms of the way schools operate.
  • When I went to school no bad things happened. Did you go to YV? If not then your experience isn’t relevant to this post.
  • I was a prefect at YV and didn’t see any bad things, if you saw bad things you should have reported it to me. I was not aware of any prefect who had a history of opposing bullying in previous years, I can think of some who had a history of encouraging it. Prefects were selected on the basis of supporting the system so anyone who would be expected to try to change things would have been rejected.
  • Children will make false and frivolous claims so we should ignore most of what they say, therefore complaints should only come from parents. Children have considerably less ability to lie than adults and the senior teachers are much better at detecting lies than most people. Sorting out accurate claims from false ones shouldn’t be difficult but if you reject all criticism as false claims then you will definitely miss reports of bad things and allow problems to continue.
  • I had a hard time at school and I turned out fine. If having bad things done to you doesn’t make you want to protect others from the same things then you didn’t turn out fine at all.
  • Kids need to toughen up to survive the real world. The “real world” that I live in doesn’t involve much violence at all, even having someone raise their voice at work is uncommon. Of the situations where being “tough” due to my experience at YV has been useful almost all of them involve me choosing to help someone I don’t know in a dangerous situation while other men pretend that they didn’t even notice it. The real solution is to create a world with less violence and a large part of that involves improving schools.


I don’t think that YV is necessarily worse than other schools, although I’m sure that representatives of other private schools are now working to assure parents of students and prospective students that they are.

I don’t think that all the people who were employed as teachers there when I attended were bad people, some of them were nice people who were competent teachers. But a few good people can’t turn around a bad system. I will note that when I attended all the sports teachers were decent people, it was the only department I could say such things about. But sports involves situations that can lead to a bad result, issues started at other times and places can lead to violence or harassment in PE classes regardless of how good the teachers are.

Teachers who know that there are problems need to be able to raise issues with the administration. When a teacher quits teaching to join the clergy and another teacher describes it as “a loss for the clergy but a gain for YV” it raises the question of why the bad teacher in question couldn’t have been encouraged to leave earlier.

A significant portion of the population will do whatever is permitted. If you say “no teacher would ever bully a student so we don’t need to look out for that” then some teacher will do exactly that.

I hope that this will lead to changes both in YV and in other schools. But if they declare this issue as resolved after expelling 4 students then something similar or worse will happen again. At least now students know that when this sort of thing happens they can send evidence to journalists to get some action.

20 May, 2024 10:27AM by etbe

May 18, 2024

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

Perfy perf

I don't like slow software. So I use profilers to make software faster. What I like even less, is slow profilers! And perf is sometimes slow for completely unavoidable reasons; to resolve source line information (needed primarily for figuring out inlining, at least in the default settings), you need to go ask libbfd. But libbfd comes from binutils, and binutils is GPLv3. And perf is part of the Linux kernel, which famously is GPLv2. So if you build perf against libbfd, the result is… nondistributable. Distros cannot ship them. Not Spiderman pointing at Spiderman, but Stallman pointing at Stallman. perf has to resort to calling out to addr2line over a pipe, which sometimes works well and sometimes… well, not. A couple of years ago, I suggested an improvement here that got me a small amount of attention, but it still isn't a really reliable way to do things.

But over the last 20 years, some other group has been busy making compilers and linkers and disassemblers and low-level binary stuff. And they were pretty careful to make their stuff GPLv2-compatible. So I give you… perf using libllvm for source line lookup (and disassembling).

Hoping for a constructive review process and that I can reach the 6.11 merge window :-)

18 May, 2024 09:45PM

hackergotchi for Sean Whitton

Sean Whitton


Ian suggested I share the highly involved build process for my doctoral dissertation, which I submitted for examination earlier this year. Beyond just compiling a PDF from Markdown and LaTeX sources, there are just two, simple-seeming goals: produce a PDF that passes PDF/A validation, for long term archival, and replace the second page with a scanned copy of the page after it was signed by the examiners. Achieving these two things reproducibly turned out to require a lot of complexity.

First we build dissertation1.tex out of a number of LaTeX and Markdown files, and a Pandoc metadata.yml, using Pandoc in a Debian sid chroot. I had to do the latter because I needed a more recent Pandoc than was available in Debian stable at the time, and didn’t dare upgrade anything else. Indeed, after switching to the newer Pandoc, I carefully diff’d dissertation1.tex to ensure nothing other than what I needed had changed.

dissertation1.tex: preamble.tex \
 citeproc-preamble.tex \
 committee.tex \
 acknowledgements.tex \
 dedication.tex \
 contents.tex \
 abbreviations.tex \
 abstract.tex \
 metadata.yaml \
 template.latex \
 philos.csl \
 philos.bib \
    schroot -c melete-sid -- pandoc -s -N -C -H preamble.tex \
        --template=template.latex -B committee.tex \
        -B acknowledgements.tex -B dedication.tex \
        -B contents.tex -B abbreviations.tex -B abstract.tex \ \
        citeproc-preamble.tex metadata.yaml -o $@

With hindsight, I think that I should have eschewed Pandoc in favour of plain LaTeX for a project as large as this was. Pandoc is good for journal submissions, where one is responsible for the content but not really the presentation. However, one typesets one’s own dissertation, without anyone else’s help. I decided to commit dissertation1.tex to git, because Pandoc’s LaTeX generation is not too stable.

We then compile a first PDF. My Makefile comments say that pdfx.sty requires this particular xelatex invocation. pdfx.sty is supposed to make the PDF satisfy the PDF/A-2B long term archival standard … but dissertation1.pdf doesn’t actually pass PDF/A validation. We instead rely on GhostScript to produce a valid PDF/A-2B, at the final step. But we have to include pdfx.sty at this stage to ensure that the hyperlinks in the PDF are PDF/A-compatible – without pdfx.sty, GhostScript rejects hyperref’s links.

dissertation1.pdf: \
 dissertation1.tex dissertation1.xmpdata committee_watermark.png
    xelatex -shell-escape -output-driver="xdvipdfmx -z 0" $<
    xelatex -shell-escape -output-driver="xdvipdfmx -z 0" $<
    xelatex -shell-escape -output-driver="xdvipdfmx -z 0" $<

As I said, the second page of the PDF needs to be replaced with a scanned version of the page after it was signed by the examiners. The usual tool to stitch PDFs together is pdftk. But pdftk loses the PDF’s metadata. For the true, static metadata like the title, author and keywords, it would be no problem to add them back. But the metadata that’s lost includes the PDF’s table of contents, which PDF readers display in a sidebar, with clickable links to chapters, and the sections within those. This information is not static because each time any of the source Markdown and LaTeX files change, there is the potential for the table of contents to change. So we have to extract all the metadata from dissertation1.pdf and save it to one side, before we stitch in the scanned page. We also have to hack the metadata to ensure that the second page will have the correct orientation.

SED = /^PageMediaNumber: 2$$/ { n; s/0/90/; n; s/612 792/792 612/ }
KEYWORDS = virtue ethics, virtue, happiness, eudaimonism, good lives, final ends
dissertation1_meta.txt: dissertation1.pdf
    printf "InfoBegin\nInfoKey: Keywords\nInfoValue: %s\n%s\n" \
        "${KEYWORDS}" "$$(pdftk $^ dump_data)" \
        | sed "${SED}" >$@

Now we can stitch in the signed page, and then put the metadata back. You can’t do this in one invocation of pdftk, so far as I could see.

dissertation1_stitched_updated.pdf: \
 dissertation1_stitched.pdf dissertation1_meta.txt
    pdftk dissertation1_stitched.pdf \
        update_info dissertation1_meta.txt output $@

dissertation1_stitched.pdf: dissertation1.pdf
    pdftk A=$^ \
        B=$$HOME/annex/philos/Dissertation/committee_signed.pdf \
        cat A1 B1 A3-end output $@

Finally, we use GhostScript to reprocess the PDF into two valid PDF/A-2Bs, one optimised for the web. This requires supplying a colour profile, a postscript file, a whole sequence of GhostScript options, and some raw postscript on the command line, which gives the PDF reader some display hints.

    -sColorConversionStrategy=UseDeviceIndependentColor \
    -dEmbedAllFonts=true -dPrinted=false -dPDFA=2 \
    -dPDFACompatibilityPolicy=1 -dDetectDuplicateImages \
    -dPDFSETTINGS=/printer -sOutputFile=$@
GS_OPTS2 = dissertation1_stitched_updated.pdf \
    -c "[ /PageMode /UseOutlines \
        /Page 1 /View [/XYZ null null 1] \
        /PageLayout /SinglePage /DOCVIEW pdfmark"

all: Whitton_dissert_web.pdf Whitton_dissert_gradcol.pdf
Whitton_dissert_gradcol.pdf: \ dissertation1_stitched_updated.pdf srgb.icc
    gs ${GS_OPTS1} ${GS_OPTS2}
Whitton_dissert_web.pdf: \ dissertation1_stitched_updated.pdf srgb.icc
    gs ${GS_OPTS1} -dFastWebView=true ${GS_OPTS2}

And here’s, based on a sample in the GhostScript docs:

% Define an ICC profile :
/ICCProfile (srgb.icc)

[/_objdef {icc_PDFA} /type /stream /OBJ pdfmark

/N 3
>> /PUT pdfmark
[{icc_PDFA} ICCProfile (r) file /PUT pdfmark

% Define the output intent dictionary :
[/_objdef {OutputIntent_PDFA} /type /dict /OBJ pdfmark
[{OutputIntent_PDFA} <<
  /Type /OutputIntent               % Must be so (the standard requires).
  /S /GTS_PDFA1                     % Must be so (the standard requires).
  /DestOutputProfile {icc_PDFA}     % Must be so (see above).
  /OutputConditionIdentifier (sRGB)
>> /PUT pdfmark
[{Catalog} <</OutputIntents [ {OutputIntent_PDFA} ]>> /PUT pdfmark


Some comments on Hacker News

18 May, 2024 08:28AM

Russell Coker

Kogan 5120*2160 40″ Monitor

I’ve just got a new Kogan 5120*2160 40″ curved monitor. It cost $599 including shipping etc which is much cheaper than the Dell monitor with similar specs selling for about $2500. For monitors with better than 4K resolution (by which I don’t mean 5K*1440) this is the cheapest option. The nearest competitors are the 27″ monitors that do 5120*2880 from Apple and some companies copying Apple’s specs. While 5120*2880 is a significantly better resolution than what I got it’s probably not going to help me at 27″ size.

I’ve had a Dell 32″ 4K monitor since the 1st of July 2022 [1]. It is a really good monitor and I had no complaints at all about it. It was clearer than the Samsung 27″ 4K monitor I used before it and I’m not sure how much of that is due to better display technology (the Samsung was from 2017) and how much was due to larger size. But larger size was definitely a significant factor.

I briefly owned a Phillips 43″ 4K monitor [2] and determined that a 43″ flat screen was definitely too big. At the time I thought that about 35″ would have been ideal but after a couple of years using a flat 32″ screen I think that 32″ is about the upper limit for a flat screen. This is the first curved monitor I’ve used but I’m already thinking that maybe 40″ is too big for a 21:9 aspect ratio even with a curved screen. Maybe if it was 4:4 or even 16:9 that would be ok. Otherwise the ideal for a curved screen for me would be something between about 36″ and 38″. Also 43″ is awkward to move around my desk. But this is still quite close to ideal.

The first system I tested this on was a work laptop, a Dell Latitude 7400 2in1. On the Dell dock that did 4K resolution and on a HDMI cable it did 1440p which was a disappointment as that laptop has talked to many 4K monitors at native resolution on the HDMI port with the same cable. This isn’t an impossible problem, as I work in the IT department I can just go through all the laptops in the store room until I find one that supports it. But the 2in1 is a very nice laptop, so I might even just keep using it in 4K resolution when WFH. The laptop in question is deemed an “executive” laptop so I have to wait another 2 years for the executives to get new laptops before I can get a newer 2in1.

On my regular desktop I had the problem of the display going off for a few seconds every minute or so and also occasionally giving a white flicker. That was using 5120*2160 with a DisplayPort switch as described in the blog post about the Dell 32″ monitor. When I ran it in 4K resolution with the DisplayPort switch from my desktop it was fine. I then used the DisplayPort cable that came with the monitor directly connecting the video card to the display and it was fine at 5120*2160 with 75Hz.

The monitor has the joystick thing that seems to have become some sort of standard for controlling modern monitors. It’s annoying that pressing it in powers it off. I think there should be a separate button for that. Also the UI in general made me wonder if one of the vendors of expensive monitors had paid whoever designed it to make the UI suck.

The monitor had a single dead pixel in the center of the screen about 1/4 the way down from the top when I started writing this post. Now it’s gone away which is a concern as I don’t know which pixels might have problems next or if the number of stuck pixels will increase. Also it would be good if there was a “dark mode” for the WordPress editor. I use dark mode wherever possible so I didn’t notice the dead pixel for several hours until I started writing this blog post.

I watched a movie on Netflix and it took the entire screen area, I don’t know if they are storing movies in 64:27 ratio or if the clipped the top and bottom, it was probably clipped but still looked OK. The monitor has different screen modes which make it look different, I can’t see much benefit to the different modes. The “standard” mode is what I usually use and it’s brighter and the “movie” mode seems OK for the one movie I’ve watched so far.

In other news BenQ has just announced a 3840*2560 28″ monitor specifically designed for programming [3]. This is the first time I’ve heard of a monitor with 3:2 ratio with modern resolution, we still aren’t at the 4:3 type ratio that we were used to when 640*480 was high resolution but it’s a definite step in the right direction. It’s also the only time I recall ever seeing a monitor advertised as being designed for programming. In the 80s there were home computers advertised as being computers for kids to program, but at that time it was either TV sets for monitors or monitors sold with computers. It was only after the IBM PC compatible market took off that having a choice of different monitors for one computer was a thing. In recent years monitors advertised as being for office use (meaning bright and expensive) have become common as are monitors designed for gamer use (meaning high refresh rate). But BenQ seems to be the first to advertise a monitor for the purpose of programming. They have a “desktop partition” feature (which could be software or hardware – the article doesn’t make it clear) to give some of the benefits of a tiled window manager to people who use OSs that don’t support such things. The BenQ monitor is a bit small for my taste, I don’t know if my vision is good enough to take advantage of 3840*2560 in a 28″ monitor nowadays. I think at least 32″ would be better. Google seems to be really into buying good monitors for their programmers, if every Google programmer got one of those BenQ monitors then that would be enough sales to make it worth-while for them.

I had hoped that we would have 6K monitors become affordable this year and 8K become less expensive than most cars. Maybe that won’t happen and we will instead have a wider range of products like the ultra wide monitor I just bought and the BenQ programmer’s monitor. If so I don’t think that will be a bad result.

Now the question is whether I can use this monitor for 2 years before finding something else that makes me want to upgrade. I can afford to spend the equivalent of a bit under $1/day on monitor upgrades.

18 May, 2024 04:24AM by etbe

James Morrison

Goodbye Firefox

 I've been on Chromebooks for a while.  However, since I had to recently try a Mac, I figured it was time to give Firefox a try again.  After two weeks of trying, I've given up.  At least for myself, I figured I'd write down the reasons I've given up.


  • There is no way from the tab context menu to move a tab between windows.  I typically try to keep no more than 3 windows open at a time.  Ideally one, but maybe a second.  Without the ability to through the context menu to move a tab, I need a very large screen (not a laptop screen) to move tabs between windows.
  • I couldn't find a way to take a URL and turn it into a custom search.  This really is a critical feature as it allows me to use short names to access specific searches.  E.g. search code (cs), show a calendar (c), etc.

18 May, 2024 12:09AM by Jim (

May 16, 2024

John Goerzen

Review of Reputable, Functional, and Secure Email Service

I last reviewed email services in 2019. That review focused a lot of attention on privacy. At the time, I selected as my provider, and have been using them for these 5 years since. However, both their service and their support have gone significantly downhill since, so it is time for me to look at other options.

Here I am focusing strongly on email. Some of the providers mentioned here provide other services (IM, video calls, groupware, etc.), and to the extent they do, I am ignoring them.

What Matters in 2024

I want to start off by acknowledging that what you need in email probably depends on your circumstances and the country in which you live. For me, I begin by naming that the largest threat most of us face isn’t from state actors but from criminals: hackers, ransomware gangs, etc. It is important to take as many steps as possible to secure one’s account against that. Privacy and security are both part of the mix. I still value privacy but I am acknowledging, as Migadu does, that “Email as we know it and encryption are incompatible.” Although some of these services strongly protect parts of the conversation, the reality is that most people will be emailing people using plain old email services which don’t. For stronger security, something like Signal would be needed. (I wrote about Signal in 2021 also.)

Interestingly, OpenPGP support seems to be something of a standard feature in the providers I reviewed by this point. All or almost all of them provide integration with browser-based encryption as well as server-side encryption if you prefer that.

Although can automatically PGP-encrypt every message that arrives in plaintext, for general use, this is unwieldy; there isn’t good tooling for searching mailboxes where every message is encrypted, etc. So I never enabled that feature at Mailbox. I still value security and privacy, but a pragmatic approach addresses the most pressing threats first.

My criteria

The basic requirements for an email service include:

  1. Ability to use my own domains
  2. Strong privacy policy
  3. Ability for me to use my own IMAP and SMTP clients on both desktop and mobile
  4. It must be extremely reliable
  5. It must not be free
  6. It must have excellent support for those rare occasions when it is needed
  7. Support for basic aliases

Why do I say it must not be free? Because if someone is providing a service with the quality I’m talking about here, and not charging for it, it implies something is fishy: either they are unscrupulous, are financially unstable, or the product is something else like ads. I am not aware of any provider that matches the other criteria with a free account anyhow. These providers range from about $30 to $90 per year, so cheaper than a Netflix subscription.

Immediately, this rules out several options:

  • Proton doesn’t let me use my own clients on mobile (their bridge is desktop-only)
  • Tuta also doesn’t let me use my own clients
  • Posteo doesn’t let me use my own domain
  • lacks a strong privacy policy, and its policy has numerous causes for concern (for instance, “If you repeatedly send email to invalid/unroutable recipients, they may be published on our GitHub”)

I will have a bit more to say about a couple of these providers below.

There are some additional criteria that are strongly desired but not absolutely required:

  1. Ability to set individual access passwords for every device/app
  2. Support for two-factor authentication (2FA/TFA/TOTP) for web-based access
  3. Support for basics in filtering: ability to filter on envelope recipient (so if I get BCC’d, I can still filter), and ability to execute more than one action on filter match (eg, deliver to two folders, or deliver to a folder and forward to someone else)

IMAP and SMTP don’t really support 2FA, so by setting individual passwords for every device, you can at least limit the blast radius and cut off a specific device if something is (or might be) compromised.

The candidates

I considered these providers: Startmail, Mailfence, Runbox, Fastmail, Kolab,, and Migadu. I’ll review each, and highlight the pricing of the plan I would most likely use. Each provider offers multiple plans; some may be more expensive and some may be cheaper than the one I reviewed. I included a link to each provider’s full pricing information so you can compare for your needs.

I set up trials with each of these (except, with which I already had a paid account). It so happend that I had actual questions for support for each one, which gave me an opportunity to see how support responded. I did not fabricate questions, and would not have contacted support if I didn’t have real ones. (This means that I asked different questions of each provider, because they were the REAL questions I had.) I’ll jump to the spoiler right now: I eventually chose Migadu, with Fastmail and Mailfence as close seconds.

I looked for providers myself, and also solicited recommendations in a Mastodon thread.

I begin with Mailbox, as it was my top choice in 2019 and the incumbent.

Until this year, I had been quite happy with it. I had cause to reach their support less than once a year on average, and each time they replied the same day or next day. Now, however, they are failing on reliability and on support.

Their spam filter has become overly aggressive. It has blocked quite a bit of legitimate mail. When contacting their support about a prior issue earlier this year, they initially took 4 days to reply, and then 6 days to reply after that. Ouch. They had me disable some spam settings.

It didn’t really help. I continue to lose mail. I don’t know how much, because they block a lot of it before it even hits the spam folder. One of my friends texted to say mail was dropping. I raised a new ticket with mailbox, which took them 5 days to reply to. Their reply was unhelpful. “As the Internet is not a static system, unforeseen events can always occur.” Well yes, that’s true, and I get it, false positives exist with email. But this was from an ISP’s mail system with an address that had been established for years, and it was part of a larger pattern of rejecting quite a bit of legit mail. And every interaction with them recently hasn’t resulted in them actually doing anything to resolve anything. It’s just a paragraph or two of reply that does nothing and helps nothing.

When I complained that it took 5 days to reply, they said “We have not been able to reply sooner as we are currently experiencing a high volume of customer enquiries.” Even though their SLA for my account is a not-great “48 business hour” turnaround, they still missed it and their reason is “we’re busy.” I finally asked what RBL had caught the blocked email, since when I checked, the sender wasn’t on any RBL. Mailbox’s reply: they only keep their logs for 7 days, so next time I should contact them within 7 days. Which, of course, I DID; it was them that kept delaying. Ugh! It’s like they’ve become a cable company.

Even worse is how they have been blocking mail from GrapheneOS’s discussion form. See their thread about it. In short, Graphene’s mail server has a clean reputation and Mailbox has no problem with it. But because one of Graphene’s IPv6 webservers has an IPv6 allocation of a size Mailbox doesn’t like, they drop mail. It’s ridiculous, and Mailbox was dismissive of this well-known and well-regarded Open Source project. So if the likes of GrapheneOS can’t get good faith effort to deliver their mail, what chance does an individual like me have?

I’m sorry, but I’m literally paying you to deliver email for me and provide good support. If you can’t do either of those, you don’t get to push that problem down onto me. Hire appropriate staff.

On the technical side, they support aliases, my own clients, and have a reasonable privacy policy. Their 2FA support exists for the web interface (though weirdly not the support site), though it is somewhat weird. They do not support app passwords.

A somewhat unique feature is the domain. If you try to receive mail at that address, will block it unless it uses TLS. Same for sending. This isn’t E2EE, but it does at least require things not be in plaintext for the last hop to Mailbox.

Verdict: not recommended due to poor reliability and support.

Mailbox.Org summary:

  • Website:
  • Reliability: iffy due to over-aggressive spam filtering
  • Support: Poor; takes 4-6 days for a reply and replies are unhelpful
  • Individual access passwords: No
  • 2FA: Yes, but with a PIN instead of a password as the other factor
  • Filtering: Full SIEVE feature set and GUI editor
  • Spam settings: greylisting on/off, reject some/all spam, etc. But they’re insufficient to address Mailbox’s overzealousness, which support says I cannot workaround within the interface.
  • Server storage location: Germany
  • Plan as reviewed: standard [pricing link]
    • Cost per year: EUR 30 (about $33)
    • Mail storage included: 10GB
    • Limits on send/receive volume: none
    • Aliases: 50 on your domain name, 25 on
    • Additional mailboxes: Available; each one at the same fee as the primary mailbox


I really wanted to like Startmail. Its “vault” is an interesting idea and should contribute to the security and privacy of an account. They clearly care about privacy.

It falls down in filtering. They have no way to filter on envelope recipient (BCC or similar). Their support confirmed this to me and that’s a showstopper.

Startmail support was also as slow as Mailbox, taking 5 days to respond to me.

Two showstoppers right there.

Verdict: Not recommended due to slow support responsiveness and weak filtering.

Startmail summary:

  • Website:
  • Reliability: Seems to be fine
  • Support: Mediocre; Took 5 days for a reply, but the reply was helpful
  • Individual app access passwords: Yes
  • 2FA: Yes
  • Filtering: Poor; cannot filter on envelope recipient, and can’t build filters with multiple actions
  • Spam settings: None
  • Server storage location: The Netherlands
  • Plan as reviewed: Custom domain (trial was Personal), [pricing link]
    • Cost per year: $70
    • Mail storage included: 20GB
    • Limits on send/receive volume: none
    • Aliases: unlimited, with lots of features: can set expiration, etc.
    • Additional mailboxes: not available


Kolab Now is mainly positioned as a full groupware service, but they do have a email-only option which I investigated. There isn’t much documentation about it compared to other providers, and also not much in the way of settings. You can turn greylisting on or off. And…. that’s it.

It has a full suite of filtering options. They set an X-Envelope-To header which you can use with the arbitrary header match to do the right thing even for BCC situations. Filters can have multiple conditions and multiple actions. It is SIEVE-based and you can download your SIEVE definitions.

If you enable 2FA, you disable IMAP and SMTP; not great.

Verdict: Not an impressive enough email featureset to justify going with it.

Kolab Now summary:

  • Website:
  • Reliability: Seems to be fine
  • Support: Fine responsiveness (next day)
  • Invidiaul app passwords: no
  • 2FA: Yes, but if you enable it, they disable IMAP and SMTP
  • Filtering: Excellent
  • Spam settings: Only greylisting on/off
  • Server storage location: Switzerland; they have lots of details on their setup
  • Plan as reviewed: “Just email” [pricing link]
    • Cost per year: CHF 60, about $66
    • Mail storage included: 5GB
    • Limitations on send/receive volume: None
    • Aliases: Yes. Not sure if there are limits.
    • Additional mailboxes: Yes if you set up a group account. “Flexible pricing based on user count” is not documented anywhere I could find.


Mailfence is another option, somewhat similar to Startmail but without the unique vault. I had some questions about filters, and support was quite responsive, responding in a couple of hours.

Some of their copy on their website is a bit misleading, but support clarified when I asked them. They do not offer encryption at rest (like most of the entries here).

Mailfence’s filtering system is the kind I’d like to see. It allows multiple conditions and multiple actions for each rule, and has some unique actions as well (notify by SMS or XMPP). Support says that “Recipients” matches envelope recipients. However, one ommission is that I can’t match on arbitrary headers; only the canned list of headers they provide.

They have only two spam settings:

  • spam filter on/off
  • whitelist

Given some recent complaints about their spam filter being overly aggressive, I find this lack of control somewhat concerning. (However, I discount complaints about people begging for more features in free accounts; free won’t provide the kind of service I’m looking for with any provider.) There are generally just very few settings for email as well.

Verdict: Response and helpful support, filtering has the right structure but lacks arbitrary header match. Could be a good option.

Mailfence summary:

  • Website:
  • Reliability: Seems to be fine
  • Support: Excellent responsiveness and helpful replies (after some initial confusion about my question of greylisting)
  • Individual app access passwords: No. You can set a per-service password (eg, an IMAP password), but those will be shared with all devices speaking that protocol.
  • 2FA: Yes
  • Filtering: Good; only misses the ability to filter on arbitrary headers
  • Spam settings: Very few
  • Server storage location: Belgium
  • Plan as reviewed: Entry [pricing link]
    • Cost per year: $42
    • Mail storage included: 10GB, with a maximum of 50,000 messages
    • Limits on send/receive volume: none
    • Aliases: 50. Aliases can’t be deleted once created (there may be an exeption to this for aliases on your own domain rather than
    • Additional mailboxes: Their page on this is a bit confusing, and the pricing page lacks the information promised. It looks like you can pay the same $42/year for additional mailboxes, with a limit of up to 2 additional paid mailboxes and 2 additional free mailboxes tied to the account.


This one came recommended in a Mastodon thread. I had some questions about it, and support response was fantastic – I heard from two people that were co-founders of the company! Even within hours, on a weekend. Incredible! This kind of response was only surpassed by Migadu.

I initially wrote to Runbox with questions about the incoming and outgoing message limits, which I hadn’t seen elsewhere, as well as the bandwidth limit. They said the bandwidth limit is no longer enforced on paid accounts. The incoming and outgoing limits are enforced, and all email (even spam) counts towards the limit. Notably the outgoing limit is per recipient, so if you send 10 messages to your 50-recipient family group, that’s the limit. However, they also indicated a willingness to reset the limit if something happens. Unfortunately, hitting the limit results in a hard bounce (SMTP 5xx) rather than a temporary failure (SMTP 4xx) so it can result in lost mail. This means I’d be worried about some attack or other weirdness causing me to lose mail.

Their filter is a pain point. Here are the challenges:

  • You can’t directly match on a BCC recipient. Support advised to use a “headers” match, which will search for something anywhere in the headers. This works and is probably “good enough” since this data is in the Received: headers, but it is a little more imprecise.
  • They only have a “contains”, not an “equals” operator. So, for instance, a pattern searching for “” would also match “”. Support advised to put the email address in angle brackets to avoid this. That will work… mostly. Angle brackets aren’t always required in headers.
  • There is no way to have multiple actions on the filter (there is just no way to file an incoming message into two folders). This was the ultimate showstopper for me.

Support advised they are planning to upgrade the filter system in the future, but these are the limitations today.

Verdict: A good option if you don’t need much from the filtering system. Lots of privacy emphasis.

Runbox summary:

  • Website:
  • Reliability: Seems to be fine, except returning 5xx codes if per-day limits are exceeded
  • Support: Excellent responsiveness and replies from founders
  • Individual app passwords: Yes
  • 2FA: Yes
  • Filtering: Poor
  • Spam settings: Very few
  • Server storage location: Norway
  • Plan as reviewed: Mini [pricing link]
    • Cost per year: $35
    • Mail storage included: 10GB
    • Limited on send/receive volume: Receive 5000 messages/day, Send 500 recipients/day
    • Aliases: 100 on; unlimited on your own domain
    • Additional mailboxes: $15/yr each, also with 10GB non-shared storage per mailbox


Fastmail came recommended to me by a friend I’ve known for decades.

Here’s the thing about Fastmail, compared to all the services listed above: It all just works. Everything. Filtering, spam prevention, it is all there, all feature-complete, and all just does the right thing as you’d hope. Their filtering system has a canned dropdown for “To/Cc/Bcc”, it supports multiple conditions and multiple actions, and just does the right thing. (Delivering to multiple folders is a little cumbersome but possible.) It has a particularly strong feature set around administering multiple accounts, including things like whether users can prevent admins from reading their mail.

The not-so-great part of the picture is around privacy. Fastmail is based in Australia, where the government has extensive power around spying on data, even to the point of forcing companies to add wiretap capabilities. Fastmail’s privacy policy states user data may be held in Australia, USA, India, and Netherlands. By default, they share data with unidentified “spam companies”, though you can disable this in settings. On the other hand, they do make a good effort towards privacy.

I contacted support with some questions and got back a helpful response in three hours. However, one of the questions was about in which countries my particular data would be stored, and the support response said they would have to get back to me on that. It’s been several days and no word back.

Verdict: A featureful option that “just works”, with a lot of features for managing family accounts and the like, but lacking in the privacy area.

Fastmail summary:

  • Website:
  • Reliability: Seems to be fine
  • Support: Good response time on most questions; dropped the ball on one tha trequired research
  • Individual app access passwords: Yes
  • 2FA: Yes
  • Filtering: Excellent
  • Spam settings: Can set filter aggressiveness, decide whether to share spam data with “spam-fighting companies”, configure how to handle backscatter spam, and evaluate the personal learning filter.
  • Server storage locations: Australia, USA, India, and The Netherlands. Legal jurisdiction is Australia.
  • Plan as reviewed: Individual [pricing link]
    • Cost per year: $60
    • Mail storage included: 50GB
    • Limits on send/receive volume: 300/hour
    • Aliases: Unlimited from what I can see
    • Additional mailboxes: No; requires a different plan for that


Migadu was a service I’d never heard of, but came recommended to me on Mastodon.

I listed Migadu last because it is a class of its own compared to all the other options. Every other service is basically a webmail interface with a few extra settings tacked on.

Migadu has a full-featured email admin console in addition. By that I mean you can:

  • View usage graphs (incoming, outgoing, storage) over time
  • Manage DNS (if you want Migadu to run your nameservers)
  • Manage multiple domains, and cross-domain relationships with mailboxes
  • View a limited set of logs
  • Configure accounts, reset their passwords if needed/authorized, etc.
  • Configure email address rewrite rules with wildcards and so forth

Basically, if you were the sort of person that ran your own mail servers back in the day, here is Migadu giving you most of that functionality. Effectively you have a web interface to do all the useful stuff, and they handle the boring and annoying bits. This is a really attractive model.

Migadu support has been fantastic. They are quick to respond, and went above and beyond. I pointed out that their X-Envelope-To header, which is needed for filtering by BCC, wasn’t being added on emails I sent myself. They replied 5 hours later indicating they had added the feature to add X-Envelope-To even for internal mails! Wow! I am impressed.

With Migadu, you buy a pool of resources: storage space and incoming/outgoing traffic. What you do within that pool is up to you. You can set up users (“mailboxes”), aliases, domains, whatever you like. It all just shares the pool. You can restrict users further so that an individual user has access to only a subset of the pool resources.

I was initially concerned about Migadu’s daily send/receive message count limits, but in visiting with support and reading the documentation, what really comes out is that Migadu is a service with a personal touch. Hitting the incoming traffic limit will cause a SMTP temporary fail (4xx) response so you won’t lose legit mail – and support will work with you if it’s a problem for legit uses. In other words, restrictions are “soft” and they are interpreted reasonably.

One interesting thing about Migadu is that they do not offer accounts under their domain. That is, you MUST bring your own domain. That’s pretty easy and cheap, of course. It also puts you in a position of power, because it is easy to migrate email from one provider to another if you own the domain.

Filtering is done via SIEVE. There is a GUI editor which lets you accomplish most things, though it has an odd blind spot where you can’t file a message into multiple folders. However, you can edit a SIEVE ruleset directly and you get the full SIEVE featureset, which is extensive (and does support filing a message into multiple folders). I note that the SIEVE :envelope match doesn’t work, but Migadu adds an X-Envelope-To header which is just as good.

I particularly love a company that tells you all the reasons you might not want to use them. Migadu’s pro/con list is an honest drawbacks list (of course, their homepage highlights all the features!).

Verdict: Fantastically powerful, excellent support, and good privacy. I chose this one.

Migadu summary:

  • Website:
  • Reliability: Excellent
  • Support: Fantastic. Good response times and they added a feature (or fixed a bug?) a few hours after I requested it.
  • Individual access passwords: Yes. Create “identities” to support them.
  • 2FA: Yes, on both the admin interface and the webmail interface
  • Filtering: Excellent, based on SIEVE. GUI editor doesn’t support multiple actions when filing into a folder, but full SIEVE functionality is exposed.
  • Spam settings:
    • On the domain level, filter aggressiveness, Greylisting on/off, black and white lists
    • On the mailbox level, filter aggressiveness, black and whitelists, action to take with spam; compatible with filters.
  • Server storage location: France; legal jurisdiction Switzerland
  • Plan as reviewed: mini [pricing link]
    • Cost per year: $90
    • Mail storage included: 30GB (“soft” quota)
    • Limits on send/receive volume: 1000 messgaes in/day, 100 messages out/day (“soft” quotas)
    • Aliases: Unlimited on an unlimited number of domains
    • Additional mailboxes: Unlimited and free; uses pooled quotas, but individual quotas can be set


Here are a few others that I didn’t think worthy of getting a trial:

  • mxroute was recommended by several. Lots of concerning things in their policy, such as:
    • if you repeatedly send mail to unroutable recipients, they may publish the addresses on Github
    • they will terminate your account if they think you are “rude” or want to contest a charge
    • they reserve the right to cancel your service at any time for any (or no) reason.
  • Proton keeps coming up, and I will not consider it so long as I am locked into their client on mobile.
  • Skiff comes up sometimes, but they were acquired by Notion.
  • Disroot comes up; this discussion highlights a number of reasons why I avoid them. Their Terms of Service (ToS) is inconsistent with a general-purpose email account (I guess for targeting nonprofits and activists, that could make sense). Particularly laughable is that they claim to be friends of Open Source, but then would take down your account if you upload “copyrighted” material. News flash: in order for an Open Source license to be meaningful, the underlying work is copyrighted. It is perfectly legal to upload copyrighted material when you wrote it or have the license to do so!


There are a lot of good options for email hosting today, and in particular I appreciate the excellent personal support from companies like Migadu and Runbox. Support small businesses!

16 May, 2024 05:42PM by John Goerzen

May 15, 2024

hackergotchi for Evgeni Golov

Evgeni Golov

Using HPONCFG on CentOS Stream 9 with OpenSSL 3.2

Today I've updated an HPE ProLiant DL325 G10 from CentOS Stream 8 to CentOS Stream 9 (details on that to follow) and realized that hponcfg was broken afterwards.

As I do not have a support contract with HPE, I couldn't just yell at them in private, so I am doing this in public now ;-)

# hponcfg
HPE Lights-Out Online Configuration utility
Version 5.6.0 Date 11/30/2020 (c) 2005,2020 Hewlett Packard Enterprise Development LP
Error: Unable to locate SSL library.
       Install latest SSL library to use HPONCFG.

Welp, what the heck?

But wait, 5.6.0 from 2020 looks old, let's update this first!

hponcfg is part of the "Management Component Pack" (at least if you're not running RHEL or SLES where you get it via the "Service Pack for ProLiant" which requires a support contract) and can be downloaded from the Software Delivery Repository.

The Software Delivery Repository tells you to configure it in /etc/yum.repos.d/mcp.repo as

name=Management Component Pack

gpgcheck=0? Suuure! Plain HTTP? Suuure!

But it gets better! When you look at (you have to substitute dist with your distribution!) you'll see that there is no 9 folder and thus no packages for CentOS (Stream) 9. There are however folders for Oracle, Rocky and Alma. Phew. Let's take one of these!

name=Management Component Pack

dnf upgrade hponcfg updates it to hponcfg-6.0.0-0.x86_64 and:

# hponcfg
HPE Lights-Out Online Configuration utility
Version 6.0.0 Date 10/30/2022 (c) 2005,2022 Hewlett Packard Enterprise Development LP
Error: Unable to locate SSL library.
       Install latest SSL library to use HPONCFG.


ldd doesn't show hponcfg being linked to libssl, do they dlopen() at runtime and fucked something up? ltrace to the rescue!

# ltrace hponcfg

popen("strings /bin/openssl | grep 'Ope"..., "r")            = 0x621700
fgets("OpenSSL 3.2.1 30 Jan 2024\n", 256, 0x621700)          = 0x7ffd870e2e10
strstr("OpenSSL 3.2.1 30 Jan 2024\n", "OpenSSL 3.0")         = nil


They run strings /bin/openssl |grep 'OpenSSL' and compare the result with "OpenSSL 3.0"?!

Sure, OpenSSL 3.2 in EL9 is rather fresh and didn't hit RHEL/Oracle/Alma/Rocky yet, but surely there are better ways to check for a compatible version of OpenSSL than THIS?!

Anyway, I am not going to downgrade my OpenSSL. Neither will I patch it to pretend to be 3.0.

But I can patch the hponcfg binary!

# vim /sbin/hponcfg
<go to line 146>
<replace 3.0 with 3.2>

Yes, I used vim. Yes, it works. No, I won't guarantee this won't kill a kitten somewhere.

# ./hponcfg
HPE Lights-Out Online Configuration utility
Version 6.0.0 Date 10/30/2022 (c) 2005,2022 Hewlett Packard Enterprise Development LP
Firmware Revision = 2.44 Device type = iLO 5 Driver name = hpilo

  hponcfg  -?
  hponcfg  -h
  hponcfg  -m minFw
  hponcfg  -r [-m minFw] [-u username] [-p password]
  hponcfg  -b [-m minFw] [-u username] [-p password]
  hponcfg  [-a] -w filename [-m minFw] [-u username] [-p password]
  hponcfg  -g [-m minFw] [-u username] [-p password]
  hponcfg  -f filename [-l filename] [-s namevaluepair] [-v] [-m minFw] [-u username] [-p password]
  hponcfg  -i [-l filename] [-s namevaluepair] [-v] [-m minFw] [-u username] [-p password]

  -h,  --help           Display this message
  -?                    Display this message
  -r,  --reset          Reset the Management Processor to factory defaults
  -b,  --reboot         Reboot Management Processor without changing any setting
  -f,  --file           Get/Set Management Processor configuration from "filename"
  -i,  --input          Get/Set Management Processor configuration from the XML input
                        received through the standard input stream.
  -w,  --writeconfig    Write the Management Processor configuration to "filename"
  -a,  --all            Capture complete Management Processor configuration to the file.
                        This should be used along with '-w' option
  -l,  --log            Log replies to "filename"
  -v,  --xmlverbose     Display all the responses from Management Processor
  -s,  --substitute     Substitute variables present in input config file
                        with values specified in "namevaluepairs"
  -g,  --get_hostinfo   Get the Host information
  -m,  --minfwlevel     Minimum firmware level
  -u,  --username       iLO Username
  -p,  --password       iLO Password

For comparison, here is the diff --text output:

# diff -u --text /sbin/hponcfg ./hponcfg
--- /sbin/hponcfg   2022-08-02 01:07:55.000000000 +0000
+++ ./hponcfg   2024-05-15 09:06:54.373121233 +0000
@@ -143,7 +143,7 @@
 helpget_hostinforesetwriteconfigallfileinputlogminfwlevelxmlverbosesubstitutetimeoutdbgverbosityrebootusernamepasswordlibpath%Ah*Ag7Ar=AwIAaMAfRAiXAl\AmgAvrAs}At�Ad�Ab�Au�Ap�Azhgrbaw:f:il:m:vs:t:d:z:u:p:tmpXMLinputFile%2d.xmlw+Error: Syntax Error - Invalid options present.
 =O@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@�M@�M@aQ@�M@aQ@�N@�M@�N@�P@aQ@aQ@�M@�M@aQ@aQ@LN@aQ@�M@�O@�M@�M@�M@�M@aQ@aQ@�M@<!----><LOGINUSER_LOGINPASSWORD<LOGIN USER_LOGIN="%s" PASSWORD="%s"ERROR: LOGIN tag is missing.
 >ERROR: LOGIN end tag is missing.
-strings  | grep 'OpenSSL 1' | grep 'OpenSSL 3'OpenSSL 1.0OpenSSL 1.1OpenSSL 3.0which openssl 2>&1/usr/bin/opensslOpenSSL location - %s
+strings  | grep 'OpenSSL 1' | grep 'OpenSSL 3'OpenSSL 1.0OpenSSL 1.1OpenSSL 3.2which openssl 2>&1/usr/bin/opensslOpenSSL location - %s
 Current version %s

 No response from command.

Pretty sure it won't apply like this with patch, but you get the idea.

And yes, double-giggles for the fact that the error message says "Install latest SSL library to use HPONCFG" and the issues is because I have the latest SSL library installed…

15 May, 2024 09:14AM by evgeni

May 14, 2024

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RApiSerialize 0.1.3 on CRAN: Skipping XDR

A new bug fix release 0.1.3 of RApiSerialize got onto CRAN earlier today. This is the first release in well over a year, and permits the skip the XDR serialization format which is needed when transfering between big- and little-endian machines. But it comes at a certain run-time cost one can avoid on the (much more common) little-endian machines. This is a new option, and the old behavior is the default. Those who want to can now skip the step.

The RApiSerialize package is used by both my RcppRedis as well as by Travers excellent qs package. We also addressed the recent nag by the CRAN concerning ‘NO_REMAP’.

Changes in version 0.1.3 (2024-05-13)

  • Add an xdr argument to disable XDR for an approx. threefold speed increase (Travers Ching and Dirk in #6)

  • Use R_NO_REMAP and Rf_* prefix for API calls

  • Minor continuous integration updates

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More details are at the RApiSerialize page; code, issue tickets etc at the GitHub repositoryrapiserializerepo.

If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

14 May, 2024 11:28PM

hackergotchi for Evgeni Golov

Evgeni Golov

Using Packit to build RPMs for projects that depend on or vendor your code

I am a huge fan of Packit as it allows us to provide RPMs to our users and testers directly from a pull-request, thus massively tightening the feedback loop and involving people who otherwise might not be able to apply the changes (for whatever reason) and "quickly test" something out. It's also a great way to validate that a change actually builds in a production environment, where no unnecessary development and test dependencies are installed.

You can also run tests of the built packages on Testing Farm and automate pushing releases into Fedora/CentOS Stream, but this is neither a (plain) Packit advertisement post, nor is that functionality that I can talk about with a certain level of experience.

Adam recently asked why we don't have Packit builds for our our Puppet modules and my first answer was: "well, puppet-* doesn't produce a thing we ship directly, so nobody dared to do it".

My second answer was that I had blogged how to test a Puppet module PR with Packit, but I totally agree that the process was a tad cumbersome and could be improved.

Now some madman did it and we all get to hear his story! ;-)

What is the problem anyway?

The Foreman Installer is a bit of Ruby code1 that provides a CLI to puppet apply based on a set of Puppet modules. As the Puppet modules can also be used outside the installer and have their own lifecycle, they live in separate git repositories and their releases get uploaded to the Puppet Forge. Users however do not want to (and should not have to) install the modules themselves.

So we have to ship the modules inside the foreman-installer package. Packaging 25 modules for two packaging systems (we support Enterprise Linux and Debian/Ubuntu) seems like a lot of work. Especially if you consider that the main foreman-installer package would need to be rebuilt after each module change as it contains generated files based on the modules which are too expensive to generate at runtime.

So we can ship the modules inside the foreman-installer source release, thus vendoring those modules into the installer release.

To do so we use librarian-puppet with a Puppetfile and either a Puppetfile.lock for stable releases or by letting librarian-puppet fetch latest for nightly snapshots.

This works beautifully for changes that land in the development and release branches of our repositories - regardless if it's foreman-installer.git or any of the puppet-*.git ones. It also works nicely for pull-requests against foreman-installer.git.

But because the puppet-* repositories do not map to packages, we assumed it wouldn't work well for pull-requests against those.

How can we solve this?

Well, the "obvious" solution is to build the foreman-installer package via Packit also for pull-requests against the puppet-* repositories. However, as usual, the devil is in the details.

Packit by default clones the repository of the pull-request and tries to create a source tarball from that using git archive. As this might be too simple for many projects, one can define a custom create-archive action that runs after the pull-request has been cloned and produces the tarball instead. We already use that in the Packit configuration for foreman-installer to run the pkg:generate_source rake target which executes librarian-puppet for us.

But now the pull-request is against one of the Puppet modules, so Packit will clone that, not the installer.

We gotta clone foreman-installer on our own. And then point librarian-puppet at the pull-request. Fun.

Cloning is relatively simple, call git clone -- sorry Packit/Copr infrastructure.

But the Puppet module pull-request? One can use :git => '' in the Puppetfile to fetch a git repository. In fact, that's what we already do for our nightly snapshots. It also supports :ref => 'some_branch_or_tag_name', if the remote HEAD is not what you want.

My brain first went "I know this! GitHub has this magic refs/pull/1/head and refs/pull/1/merge refs you can checkout to get the contents of the pull-request without bothering to add a remote for the source of the pull-request". Well, this requires to know the ID of the pull-request and Packit does not expose that in the environment variables available during create-archive.

Wait, but we already have a checkout. Can we just say :git => '../.git'? Cloning a .git folder is totally possible after all.

[Librarian]     --> fatal: repository '../.git' does not exist
Could not checkout ../.git: fatal: repository '../.git' does not exist

Seems librarian disagrees. Damn. (Yes, I checked, the path exists.)

💡 does it maybe just not like relative paths?! Yepp, using an absolute path absolutely works!

For some reason it ends up checking out the default HEAD of the "real" (GitHub) remote, not of ../. Luckily this can be fixed by explicitly passing :ref => 'origin/HEAD', which resolves to the branch Packit created for the pull-request.

Now we just need to put all of that together and remember to execute all commands from inside the foreman-installer checkout as that is where all our vendoring recipes etc live.

Putting it all together

Let's look at the diff between the packit.yaml for foreman-installer and the one I've proposed for puppet-pulpcore:

--- a/foreman-installer/.packit.yaml    2024-05-14 21:45:26.545260798 +0200
+++ b/puppet-pulpcore/.packit.yaml  2024-05-14 21:44:47.834162418 +0200
@@ -18,13 +18,15 @@
     - "wget -O foreman-installer.spec"
+    - "git clone"
+    - "sed -i '/theforeman.pulpcore/ s@:git.*@:git => \"#{__dir__}/../.git\", :ref => \"origin/HEAD\"@' foreman-installer/Puppetfile"
-    - "sed 's/-develop//' VERSION"
+    - "sed 's/-develop//' foreman-installer/VERSION"
-    - bundle config set --local path vendor/bundle
-    - bundle config set --local without development:test
-    - bundle install
-    - bundle exec rake pkg:generate_source
+    - bash -c "cd foreman-installer && bundle config set --local path vendor/bundle"
+    - bash -c "cd foreman-installer && bundle config set --local without development:test"
+    - bash -c "cd foreman-installer && bundle install"
+    - bash -c "cd foreman-installer && bundle exec rake pkg:generate_source"
  1. It clones foreman-installer (in post-upstream-clone, as that felt more natural after some thinking)
  2. It adjusts the Puppetfile to use #{__dir__}/../.git as the Git repository, abusing the fact that a Puppetfile is really just a Ruby script (sorry Ben!) and knows the __dir__ it lives in
  3. It fetches the version from the foreman-installer checkout, so it's sort-of reasonable
  4. It performs all building inside the foreman-installer checkout

Can this be used in other scenarios?

I hope so! Vendoring is not unheard of. And testing your "consumers" (dependents? naming is hard) is good style anyway!

  1. three Ruby modules in a trench coat, so to say ↩

14 May, 2024 08:12PM by evgeni

Julian Andres Klode

The new APT 3.0 solver

APT 2.9.3 introduces the first iteration of the new solver codenamed solver3, and now available with the –solver 3.0 option. The new solver works fundamentally different from the old one.

How does it work?

Solver3 is a fully backtracking dependency solving algorithm that defers choices to as late as possible. It starts with an empty set of packages, then adds the manually installed packages, and then installs packages automatically as necessary to satisfy the dependencies.

Deferring the choices is implemented multiple ways:

First, all install requests recursively mark dependencies with a single solution for install, and any packages that are being rejected due to conflicts or user requests will cause their reverse dependencies to be transitively marked as rejected, provided their or group cannot be solved by a different package.

Second, any dependency with more than one choice is pushed to a priority queue that is ordered by the number of possible solutions, such that we resolve a|b before a|b|c.

Not just by the number of solutions, though. One important point to note is that optional dependencies, that is, Recommends, are always sorting after mandatory dependencies. Do note on that: Recommended packages do not “nest” in backtracking - dependencies of a Recommended package themselves are not optional, so they will have to be resolved before the next Recommended package is seen in the queue.

Another important step in deferring choices is extracting the common dependencies of a package across its version and then installing them before we even decide which of its versions we want to install - one of the dependencies might cycle back to a specific version after all.

Decisions about package levels are recorded at a certain decision level, if we reach a conflict we backtrack to the previous decision level, mark the decision we made (install X) in the inverse (DO NOT INSTALL X), reset all the state all decisions made at the higher level, and restore any dependencies that are no longer resolved to the work queue.

Comparison to SAT solver design.

If you have studied SAT solver design, you’ll find that essentially this is a DPLL solver without pure literal elimination. A pure literal eliminitation phase would not work for a package manager: First negative pure literals (packages that everything conflicts with) do not exist, and positive pure literals (packages nothing conflicts with) we do not want to mark for install - we want to install as little as possible (well subject, to policy).

As part of the solving phase, we also construct an implication graph, albeit a partial one: The first package installing another package is marked as the reason (A -> B), the same thing for conflicts (not A -> not B).

Once we have added the ability to have multiple parents in the implication graph, it stands to reason that we can also implement the much more advanced method of conflict-driven clause learning; where we do not jump back to the previous decision level but exactly to the decision level that caused the conflict. This would massively speed up backtracking.

What changes can you expect in behavior?

The most striking difference to the classic APT solver is that solver3 always keeps manually installed packages around, it never offers to remove them. We will relax that in a future iteration so that it can replace packages with new ones, that is, if your package is no longer available in the repository (obsolete), but there is one that Conflicts+Replaces+Provides it, solver3 will be allowed to install that and remove the other.

Implementing that policy is rather trivial: We just need to queue obsolete | replacement as a dependency to solve, rather than mark the obsolete package for install.

Another critical difference is the change in the autoremove behavior: The new solver currently only knows the strongest dependency chain to each package, and hence it will not keep around any packages that are only reachable via weaker chains. A common example is when gcc-<version> packages accumulate on your system over the years. They all have Provides: c-compiler and the libtool Depends: gcc | c-compiler is enough to keep them around.

New features

The new option --no-strict-pinning instructs the solver to consider all versions of a package and not just the candidate version. For example, you could use apt install foo=2.0 --no-strict-pinning to install version 2.0 of foo and upgrade - or downgrade - packages as needed to satisfy foo=2.0 dependencies. This mostly comes in handy in use cases involving Debian experimental or the Ubuntu proposed pockets, where you want to install a package from there, but try to satisfy from the normal release as much as possible.

The implication graph building allows us to implement an apt why command, that while not as nicely detailed as aptitude, at least tells you the exact reason why a package is installed. It will only show the strongest dependency chain at first of course, since that is what we record.

What is left to do?

At the moment, error information is not stored across backtracking in any way, but we generally will want to show you the first conflict we reach as it is the most natural one; or all conflicts. Currently you get the last conflict which may not be particularly useful.

Likewise, errors currently are just rendered as implication graphs of the form [not] A -> [not] B -> ..., and we need to put in some work to present those nicely.

The test suite is not passing yet, I haven’t really started working on it. A challenge is that most packages in the test suite are manually installed as they are mocked, and the solver now doesn’t remove those.

We plan to implement the replacement logic such that foo can be replaced by foo2 Conflicts/Replaces/Provides foo without needing to be automatically installed.

Improving the backtracking to be non-chronological conflict-driven clause learning would vastly enhance our backtracking performance. Not that it seems to be an issue right now in my limited testing (mostly noble 64-bit-time_t upgrades). A lot of that complexity you have normally is not there because the manually installed packages and resulting unit propagation (single-solution Depends/Reverse-Depends for Conflicts) already ground us fairly far in what changes we can actually make.

Once all the stuff has landed, we need to start rolling it out and gather feedback. On Ubuntu I’d like automated feedback on regressions (running solver3 in parallel, checking if result is worse and then submitting an error to the error tracker), on Debian this could just be a role email address to send solver dumps to.

At the same time, we can also incrementally start rolling this out. Like phased updates in Ubuntu, we can also roll out the new solver as the default to 10%, 20%, 50% of users before going to the full 100%. This will allow us to capture regressions early and fix them.

14 May, 2024 11:26AM

hackergotchi for Matthew Palmer

Matthew Palmer

"Is This Project Still Maintained?"

If you wander around a lot of open source repositories on the likes of GitHub, you’ll invariably stumble over repos that have an issue (or more than one!) with a title like the above. Sometimes sitting open and unloved, often with a comment or two from the maintainer and a bunch of “I’ll help out!” followups that never seemed to pan out. Very rarely, you’ll find one that has been closed, with a happy ending.

These issues always fascinate me, because they say a lot about what it means to “maintain” an open source project, the nature of succession (particularly in a post-Jia Tan world), and the expectations of users and the impedence mismatch between maintainers, contributors, and users. I’ve also recently been thinking about pre-empting this sort of issue, and opening my own issue that answers the question before it’s even asked.

Why These Issues Are Created

As both a producer and consumer of open source software, I completely understand the reasons someone might want to know whether a project is abandoned. It’s comforting to be able to believe that there’s someone “on the other end of the line”, and that if you have a problem, you can ask for help with a non-zero chance of someone answering you. There’s also a better chance that, if the maintainer is still interested in the software, that compatibility issues and at least show-stopper bugs might get fixed for you.

But often there’s more at play. There is a delusion that “maintained” open source software comes with entitlements – an expectation that your questions, bug reports, and feature requests will be attended to in some fashion.

This comes about, I think, in part because there are a lot of open source projects that are energetically supported, where generous volunteers do answer questions, fix reported bugs, and implement things that they don’t personally need, but which random Internet strangers ask for. If you’ve had that kind of user experience, it’s not surprising that you might start to expect it from all open source projects.

Of course, these wonders of cooperative collaboration are the exception, rather than the rule. In many (most?) cases, there is little practical difference between most projects that are “maintained” and those that are formally declared “unmaintained”. The contributors (or, most often, contributor – singular) are unlikely to have the time or inclination to respond to your questions in a timely and effective manner. If you find a problem with the software, you’re going to be paddling your own canoe, even if the maintainer swears that they’re still “maintaining” it.

A Thought Appears

With this in mind, I’ve been considering how to get ahead of the problem and answer the question for the software projects I’ve put out in the world. Nothing I’ve built has anything like what you’d call a “community”; most have never seen an external PR, or even an issue. The last commit date on them might be years ago.

By most measures, almost all of my repos look “unmaintained”. Yet, they don’t feel unmaintained to me. I’m still using the code, sometimes as often as every day, and if something broke for me, I’d fix it. Anyone who needs the functionality I’ve developed can use the code, and be pretty confident that it’ll do what it says in the README.

I’m considering creating an issue in all my repos, titled “Is This Project Still Maintained?”, pinning it to the issues list, and pasting in something I’m starting to think of as “The Open Source Maintainer’s Manifesto”.

It goes something like this:

Is This Project Still Maintained?

Yes. Maybe. Actually, perhaps no. Well, really, it depends on what you mean by “maintained”.

I wrote the software in this repo for my own benefit – to solve the problems I had, when I had them. While I could have kept the software to myself, I instead released it publicly, under the terms of an open licence, with the hope that it might be useful to others, but with no guarantees of any kind. Thanks to the generosity of others, it costs me literally nothing for you to use, modify, and redistribute this project, so have at it!

OK, Whatever. What About Maintenance?

In one sense, this software is “maintained”, and always will be. I fix the bugs that annoy me, I upgrade dependencies when not doing so causes me problems, and I add features that I need. To the degree that any on-going development is happening, it’s because I want that development to happen.

However, if “maintained” to you means responses to questions, bug fixes, upgrades, or new features, you may be somewhat disappointed. That’s not “maintenance”, that’s “support”, and if you expect support, you’ll probably want to have a “support contract”, where we come to an agreement where you pay me money, and I help you with the things you need help with.

That Doesn’t Sound Fair!

If it makes you feel better, there are several things you are entitled to:

  1. The ability to use, study, modify, and redistribute the contents of this repository, under the terms stated in the applicable licence(s).

  2. That any interactions you may have with myself, other contributors, and anyone else in this project’s spaces will be in line with the published Code of Conduct, and any transgressions of the Code of Conduct will be dealt with appropriately.

  3. … actually, that’s it.

Things that you are not entitled to include an answer to your question, a fix for your bug, an implementation of your feature request, or a merge (or even review) of your pull request. Sometimes I may respond, either immediately or at some time long afterwards. You may luck out, and I’ll think “hmm, yeah, that’s an interesting thing” and I’ll work on it, but if I do that in any particular instance, it does not create an entitlement that I will continue to do so, or that I will ever do so again in the future.

But… I’ve Found a Huge and Terrible Bug!

You have my full and complete sympathy. It’s reasonable to assume that I haven’t come across the same bug, or at least that it doesn’t bother me, otherwise I’d have fixed it for myself.

Feel free to report it, if only to warn other people that there is a huge bug they might need to avoid (possibly by not using the software at all). Well-written bug reports are great contributions, and I appreciate the effort you’ve put in, but the work that you’ve done on your bug report still doesn’t create any entitlement on me to fix it.

If you really want that bug fixed, the source is available, and the licence gives you the right to modify it as you see fit. I encourage you to dig in and fix the bug. If you don’t have the necessary skills to do so yourself, you can get someone else to fix it – everyone has the same entitlements to use, study, modify, and redistribute as you do.

You may also decide to pay me for a support contract, and get the bug fixed that way. That gets the bug fixed for everyone, and gives you the bonus warm fuzzies of contributing to the digital commons, which is always nice.

But… My PR is a Gift!

If you take the time and effort to make a PR, you’re doing good work and I commend you for it. However, that doesn’t mean I’ll necessarily merge it into this repository, or even work with you to get it into a state suitable for merging.

A PR is what is often called a “gift of work”. I’ll have to make sure that, at the very least, it doesn’t make anything actively worse. That includes introducing bugs, or causing maintenance headaches in the future (which includes my getting irrationally angry at indenting, because I’m like that). Properly reviewing a PR takes me at least as much time as it would take me to write it from scratch, in almost all cases.

So, if your PR languishes, it might not be that it’s bad, or that the project is (dum dum dummmm!) “unmaintained”, but just that I don’t accept this particular gift of work at this particular time.

Don’t forget that the terms of licence include permission to redistribute modified versions of the code I’ve released. If you think your PR is all that and a bag of potato chips, fork away! I won’t be offended if you decide to release a permanent fork of this software, as long as you comply with the terms of the licence(s) involved.

(Note that I do not undertake support contracts solely to review and merge PRs; that reeks a little too much of “pay to play” for my liking)

Gee, You Sound Like an Asshole

I prefer to think of myself as “forthright” and “plain-speaking”, but that brings to mind that third thing you’re entitled to: your opinion.

I’ve written this out because I feel like clarifying the reality we’re living in, in the hope that it prevents misunderstandings. If what I’ve written makes you not want to use the software I’ve written, that’s fine – you’ve probably avoided future disappointment.

Opinions Sought

What do you think? Too harsh? Too wishy-washy? Comment away!

14 May, 2024 12:00AM by Matt Palmer (

May 12, 2024

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

Wikimedia jumps on the AI bandwagon

I've been resisting the Wikipedia ads about “we don't run ads, give us money“ for over a decade now (mostly since WMF already has tons of cash and use very little on it to actually improve Wikipedia), and now they are jumping on the AI/LLM hype. It does not help.

(Of course you should pick your charities yourself. I've donated to Signal Foundation in the past even though I think they could run things somewhat cheaper if they didn't insist on all-cloud, and is basically, for better or for worse, the collective memory of the Internet by now, at least unless they get bankrupted by the ongoing lawsuit from some overly silly book lending. Not to mention that in a day and age where there's a certain Eurovision-participating country and another banned-from-Eurovision country both causing tons of civilian suffering and casualities these days, perhaps there are non-tech charities that are also important.)

12 May, 2024 04:40PM

Elana Hashman

I am very sick

I have not been able to walk since February 18, 2023.

When people ask me how I'm doing, this is the first thing that comes to mind. "Well, you know, the usual, but also I still can't walk," I think to myself.

If I dream at night, I often see myself walking or running. In conversation, if I talk about going somewhere, I'll imagine walking there. Even though it's been over a year, I remember walking to the bus, riding to see my friends, going out for brunch, cooking community dinners.

But these days, I can't manage going anywhere except by car, and I can't do the driving, and I can't dis/assemble and load my chair. When I'm resting in bed and follow a guided meditation, I might be asked to imagine walking up a staircase, step by step. Sometimes, I do. Other times, I imagine taking a little elevator in my chair, or wheeling up ramps.

I feel like there is little I can say that can express the extent of what this illness has taken from me, but it's worth trying. To an able-bodied person, seeing me in a power wheelchair is usually "enough." One of my acquaintances cried when they last saw me in person. But frankly, I love my wheelchair. I am not "wheelchair-bound"—I am bed-bound, and the wheelchair gets me out of bed. My chair hasn't taken anything from me.


In October of 2022, I was diagnosed with myalgic encephalomyelitis.

Scientists and doctors don't really know what myalgic encephalomyelitis (ME) is. Diseases like it have been described for over 200 years.1 It primarily affects women between the ages of 10-39, and the primary symptom is "post-exertional malaise" or PEM: debilitating, disproportionate fatigue following activity, often delayed by 24-72 hours and not relieved by sleep. That fatigue has earned the illness the misleading name of "Chronic Fatigue Syndrome" or CFS, as though we're all just very tired all the time. But tired people respond to exercise positively. People with ME/CFS do not.2

Given the dearth of research and complete lack of on-label treatments, you may think this illness is at least rare, but it is actually quite common: in the United States, an estimated 836k-2.5m people3 have ME/CFS. It is frequently misdiagnosed, and it is estimated that as many as 90% of cases are missed,4 due to mild or moderate symptoms that mimic other diseases. Furthermore, over half of Long COVID cases likely meet the diagnostic criteria for ME,5 so these numbers have increased greatly in recent years. That is, ME is at least as common as rheumatoid arthritis,6 another delightful illness I have. But while any doctor knows what rheumatoid arthritis is, not enough7 have heard of "myalgic encephalomylitis."

Despite a high frequency and disease burden, post-viral associated conditions (PASCs) such as ME have been neglected for medical funding for decades.8 Indeed, many people, including medical care workers, find it hard to believe that after the acute phase of illness, severe symptoms can persist. PASCs such as ME and Long COVID defy the typical narrative around common illnesses. I was always told that if I got sick, I should expect to rest for a bit, maybe take some medications, and a week or two later, I'd get better, right? But I never got better.

These are complex, multi-system diseases that do not neatly fit into the Western medical system's specializations. I have seen nearly every specialty because ME/CFS affects nearly every system of the body: cardiology, nephrology, pulmonology, neurology, opthalmology, and, many, many more. You'd think they'd hand out frequent flyer cards, or a medical passport with fun stamps, but nope. Just hundreds of pages of medical records. And when I don't fit neatly into one particular specialist's box, then I'm sent back to my primary care doctor to regroup while we try to troubleshoot my latest concerning symptoms. "Sorry, can't help you. Not my department."

With little available medical expertise, a lot of my disease management has been self-directed in partnership with primary care. I've read hundreds of articles, papers, publications, CME material normally reserved for doctors. It's truly out of necessity, and I'm certain I would be much worse off if I lacked the skills and connections to do this; there are so few ME/CFS experts in the US that there isn't one in my state or any adjacent state.9 So I've done a lot of my own work, much of it while barely being able to read. (A text-to-speech service is a real lifesaver.) To facilitate managing my illness, I've built a mental model of how my particular flavour of ME/CFS works based on the available research I've been able to read and how I respond to treatments. Here is my best attempt to explain it:

  • After a severe (non-COVID) infection, an ongoing interaction between my immune system and my metabolism have stopped my body from being able to do aerobic respiration.10
  • I don't know why or how, but my mitochondria don't work properly anymore.11
  • This means that if I use too much energy, my body isn't able to make enough energy to catch up, and I have severe symptoms over the next few days as my body tries to manage the consequences.
  • Those symptoms aren't limited to fatigue: I've developed flu-like symptoms and even fevers, limbs so heavy they felt paralyzed, tachycardia in response to even the slightest activity.

The best way I have learned to manage this is to prevent myself from doing activities where I will exceed that aerobic threshold by wearing a heartrate monitor,12 but the amount of activity that permits in my current state of health is laughably restrictive. Most days I'm unable to spend more than one to two hours out of bed.

Over time, this has meant worsening from a persistent feeling of tiredness all the time and difficulty commuting into an office or sitting at a desk, to being unable to sit at a desk for an entire workday even while working from home and avoiding physically intense chores or exercise without really understanding why, to being unable to leave my apartment for days at a time, and finally, being unable to stand for more than a minute or two or walk.

But it's not merely that I can't walk. Many folks in wheelchairs are able to live excellent lives with adaptive technology. The problem is that I am so fatigued, any activity can destroy my remaining quality of life. In my worst moments, I've been unable to read, move my arms or legs, or speak aloud. Every single one of my limbs burned, as though I had caught fire. Food sat in my stomach for hours, undigested, while my stomach seemingly lacked the energy to do its job. I currently rely on family and friends for full-time caretaking, plus a paid home health aide, as I am unable to prep meals, shower, or leave the house independently. This assistance has helped me slowly improve from my poorest levels of function.

While I am doing better than I was at my worst, I've had to give up essentially all of my hobbies with physical components. These include singing, cooking, baking, taking care of my houseplants, cross-stitching, painting, and so on. Doing any of these result in post-exertional malaise so I've had to stop; this reduction of activity to prevent worsening the illness is referred to as "pacing." I've also had to cut back essentially all of my volunteering and work in open source; I am only cleared by my doctor to work 15h/wk (from bed) as of writing.


CW: severe illness, death, and suicide (skip this section)

The difficulty of living with a chronic illness is that there's no light at the end of the tunnel. Some diseases have a clear treatment path: you take the medications, you complete the procedures, you hit all the milestones, and then you're done, perhaps with some long-term maintenance work. But with ME, there isn't really an end in sight. The median duration of illness reported in one 1997 study was over 6 years, with some patients reporting 20 years of symptoms.13 While a small number of patients spontaneously recover, and many improve, the vast majority of patients are unable to regain their baseline function.14

My greatest fear since losing the ability to walk is getting worse still. Because, while I already require assistance with nearly every activity of daily living, there is still room for decline. The prognosis for extremely ill patients is dismal, and many require feeding tubes and daily nursing care. This may lead to life-threatening malnutrition;15 a number of these extremely severe patients have died, either due to medical neglect or suicide.16 Extremely severe patients cannot tolerate light, sound, touch, or cognitive exertion,17 and often spend most of their time lying flat in a darkened room with ear muffs or an eye mask.18

This is all to say, my prognosis is not great.

But while I recognize that the odds aren't exactly in my favour, I am also damn stubborn. (A friend once cheerfully described me as "stubbornly optimistic!") I only get one shot at life, and I do not want to spend the entirety of it barely able to perceive what's going on around me. So while my prognosis is uncertain, there's lots of evidence that I can improve somewhat,19 and there's also lots of evidence that I can live 20+ years with this disease. It's a bitter pill to swallow, but it also means I might have the gift of time—something that not all my friends with severe complex illnesses have had.

I feel like I owe it to myself to do the best I can to improve; to try to help others in a similar situation; and to enjoy the time that I have. I already feel like my life has been moving in slow motion for the past 4 years—there's no need to add more suffering. Finding joy, as much as I can, every day, is essential to keep up my strength for this marathon. Even if it takes 20 years to find a cure, I am convinced that the standard of care is going to improve. All the research and advocacy that's been happening over the past decade is plenty to feel hopeful about.20 Hope is a discipline,21 and I try to remind myself of this on the hardest days.


I'm not entirely sure why I decided to write this. Certainly, today is International ME/CFS Awareness Day, and I'm hoping this post will raise awareness in spaces that aren't often thinking about chronic illnesses. But I think there is also a part of me that wants to share, reach out in some way to the people I've lost contact with while I've been treading water, managing the day to day of my illness. I experience this profound sense of loss, especially when I think back to the life I had before. Everyone hits limitations in what they can do and accomplish, but there is so little I can do with the time and energy that I have. And yet, I understand even this precious little could still be less. So I pace myself.

Perhaps I can inspire you to take action on behalf of those of us too fatigued to do the advocacy we need and deserve. Should you donate to a charity or advocacy organization supporting ME/CFS research? In the US, there are many excellent organizations, such as ME Action, the Open Medicine Foundation, SolveME, the Bateman Horne Center, and the Workwell Foundation. I am also happy to match any donations through the end of May 2024 if you send me your receipts. But charitable giving only goes so far, and I think this problem deserves the backing of more powerful organizations.

Proportionate government funding and support is desperately needed. It's critical for us to push governments22 to provide the funding required for research that will make an impact on patients' lives now. Many organizers are running campaigns around the world, advocating for this investment. There is a natural partnership between ME advocacy and Long COVID advocacy, for example, and we have an opportunity to make a great difference to many people by pushing for research and resources inclusive of all PASCs. Some examples I'm aware of include:

But outside of collective organizing, there are a lot of sick individuals out there that need help, too. Please, don't forget about us. We need you to visit us, care for us, be our confidantes, show up as friends. There are a lot of people who are very sick out here and need your care.

I'm one of them.

12 May, 2024 02:00PM by Elana Hashman

May 11, 2024

Sven Hoexter

xdg and mime types - stuff I would've loved to know a week ago

Learned a few things about xdg and mimetype registration in the last week that could be helpful to have condensed in a single place.

No Need to Ship a Mailcap Mime File

If you already ship a .desktop file (that is what ends up in /usr/share/applications/) which has a MimeType declared, there is no need to also ship a mailcap file (that is what ends up in /usr/lib/mime/packages/). Some triggers will do the conversion work for you. See also Debian Policy 4.9.

Reverse DNS Naming Convention for .desktop Files

Seems to be a closely guarded secret, maybe mainly known inside the Gnome world, but it's in the spec. Also not very widely known inside Debian if I look at my local system as not very representative sample.

Your hicolor Theme App Icon can be a Mime Type Icon as Well

In case you didn't know the hicolor icon theme is the default fallback theme. Many of us already install application icons e.g. in /usr/share/icons/hicolor/48x48/apps/ which is used in conjunction with the Icon field in the .desktop file to locate the application icon. Now the next step, and there it seems quite of few us miss out, is to create a symlink to also provide a mime type icon, so it's displayed in graphical file managers for the application data files. The schema here is simple: Take the MimeType e.g. application/x-vymand replace the / with a - and use that as file name in e.g. /usr/share/icons/hicolor/48x48/mimetypes/. In the vym case that is /usr/share/icons/hicolor/48x48/mimetypes/application-x-vym.png. If you have one use a scalable .svg file instead of .png.

This seems to be an area where Debian lacks a bit of tooling to automatically convert application icons to all the different sizes and install it in all the appropriate places. What is already there is a trigger to run gtk-update-icon-cache when you install new icons into one of the icon theme folder so they're picked up.

No Priority or Order in .desktop Files

Likely something that hapens on all my fresh installations: Libreoffice is installed and xdg-open starts to open pdf files with Libreoffice instead of evince. Now I've to figure out again to run xdg-mime default org.gnome.Evince.desktop application/pdf to change that (at least for my user). Background here is that the desktop file spec explicitly mandates "Priority for applications is handled external to the .desktop files.". That's why we got in addition to all of that mimeapps.list files. And now, after running the xdg-mime command from above, we've a ~/.config/mimeapps.list defining

[Default Applications]

Debian as whole seems to be not very keen on shipping something like a sensible default mimeapps.list outside of desktop environment specific ones. A quick search gave me just

$ apt-file search mimeapps.list
cinnamon-desktop-data: /usr/share/applications/x-cinnamon-mimeapps.list
gdm3: /usr/share/gdm/greeter/applications/mimeapps.list
gnome-session-common: /usr/share/applications/gnome-mimeapps.list
plasma-workspace: /usr/share/applications/kde-mimeapps.list
sxmo-utils: /usr/share/applications/mimeapps.list
sxmo-utils: /usr/share/sxmo/xdg/mimeapps.list

While it's a bit anoying to run into that pdf vs Libreoffice thing every now and then, it's maybe better to not have long controversial threads about default pdf viewer, like the ones we already had about the default MTA choices. ;) And while we're at it: everyone using Libreoffice should give a virtual hug to rene@ for taming that beast since 2010 and before.

11 May, 2024 04:57PM

May 10, 2024

Reproducible Builds

Reproducible Builds in April 2024

Welcome to the April 2024 report from the Reproducible Builds project! In our reports, we attempt to outline what we have been up to over the past month, as well as mentioning some of the important things happening more generally in software supply-chain security. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.

Table of contents:

  1. New backseat-signed tool to validate distributions’ source inputs
  2. ‘NixOS is not reproducible’
  3. Certificate vulnerabilities in F-Droid’s fdroidserver
  4. Website updates
  5. ‘Reproducible Builds and Insights from an Independent Verifier for Arch Linux’
  6. libntlm now releasing ‘minimal source-only tarballs’
  7. Distribution work
  8. Mailing list news
  9. diffoscope
  10. Upstream patches
  11. reprotest
  12. Reproducibility testing framework

New backseat-signed tool to validate distributions’ source inputs

kpcyrd announced a new tool called backseat-signed, after:

I figured out a somewhat straight-forward way to check if a given git archive output is cryptographically claimed to be the source input of a given binary package in either Arch Linux or Debian (or both).

Elaborating more in their announcement post, kpcyrd writes:

I believe this to be the “reproducible source tarball” thing some people have been asking about. As explained in the README, I believe reproducing autotools-generated tarballs isn’t worth everybody’s time and instead a distribution that claims to build from source should operate on VCS snapshots instead of tarballs with 25k lines of pre-generated shell-script.

Indeed, many distributions’ packages already build from VCS snapshots, and this trend is likely to accelerate in response to the xz incident. The announcement led to a lengthy discussion on our mailing list, as well as shorter followup thread from kpcyrd about bootstrapping Autotools projects.

‘NixOS is not reproducible’

Morten Linderud posted an post on his blog this month, provocatively titled, “NixOS is not reproducible”. Although quickly admitting that his title is indeed “clickbait”, Morten goes on to clarify the precise guarantees and promises that NixOS provides its users.

Later in the most, Morten mentions that he was motivated to write the post because:

I have heavily invested my free-time on this topic since 2017, and met some of the accomplishments we have had with “Doesn’t NixOS solve this?” for just as long… and I thought it would be of peoples interest to clarify[.]

Certificate vulnerabilities in F-Droid’s fdroidserver

In early April, Fay Stegerman announced a certificate pinning bypass vulnerability and Proof of Concept (PoC) in the F-Droid fdroidserver tools for “managing builds, indexes, updates, and deployments for F-Droid repositories” to the oss-security mailing list.

We observed that embedding a v1 (JAR) signature file in an APK with minSdk >= 24 will be ignored by Android/apksigner, which only checks v2/v3 in that case. However, since fdroidserver checks v1 first, regardless of minSdk, and does not verify the signature, it will accept a “fake” certificate and see an incorrect certificate fingerprint. […] We also realised that the above mentioned discrepancy between apksigner and androguard (which fdroidserver uses to extract the v2/v3 certificates) can be abused here as well. […]

Later on in the month, Fay followed up with a second post detailing a third vulnerability and a script that could be used to scan for potentially affected .apk files and mentioned that, whilst upstream had acknowledged the vulnerability, they had not yet applied any ameliorating fixes.

Website updates

There were a number of improvements made to our website this month, including Chris Lamb updating the archive page to recommend -X and unzipping with TZ=UTC [] and adding Maven, Gradle, JDK and Groovy examples to the SOURCE_DATE_EPOCH page []. In addition Jan Zerebecki added a new /contribute/opensuse/ page [] and Sertonix fixed the automatic RSS feed detection [][].

Reproducible Builds and Insights from an Independent Verifier for Arch Linux

Joshua Drexel, Esther Hänggi and Iyán Méndez Veiga of the School of Computer Science and Information Technology, Hochschule Luzern (HSLU) in Switzerland published a paper this month entitled Reproducible Builds and Insights from an Independent Verifier for Arch Linux. The paper establishes the context as follows:

Supply chain attacks have emerged as a prominent cybersecurity threat in recent years. Reproducible and bootstrappable builds have the potential to reduce such attacks significantly. In combination with independent, exhaustive and periodic source code audits, these measures can effectively eradicate compromises in the building process. In this paper we introduce both concepts, we analyze the achievements over the last ten years and explain the remaining challenges.

What is more, the paper aims to:

… contribute to the reproducible builds effort by setting up a rebuilder and verifier instance to test the reproducibility of Arch Linux packages. Using the results from this instance, we uncover an unnoticed and security-relevant packaging issue affecting 16 packages related to Certbot […].

A PDF of the paper is available.

libntlm now releasing ‘minimal source-only tarballs’

Simon Josefsson wrote on his blog this month that, going forward, the libntlm project will now be releasing what they call “minimal source-only tarballs”:

The XZUtils incident illustrate that tarballs with files that are not included in the git archive offer an opportunity to disguise malicious backdoors. [The] risk of hiding malware is not the only motivation to publish signed minimal source-only tarballs. With pre-generated content in tarballs, there is a risk that GNU/Linux distributions [ship] generated files coming from the tarball into the binary *.deb or *.rpm package file. Typically the person packaging the upstream project never realized that some installed artifacts was not re-built[.]

Simon’s post goes into further details how this was achieved, and describes some potential caveats and counters some expected responses as well. A shorter version can be found in the announcement for the 1.8 release of libntlm.

Distribution work

In Debian this month, Helmut Grohne filed a bug suggesting the removal of dh-buildinfo, a tool to generate and distribute .buildinfo-like files within binary packages. Note that this is distinct from the .buildinfo generation performed by dpkg-genbuildinfo. By contrast, the entirely optional dh-buildinfo generated a debian/buildinfo file that would be shipped within binary packages as /usr/share/doc/package/buildinfo_$arch.gz.

Adrian Bunk recently asked about including source hashes in Debian’s .buildinfo files, which prompted Guillem Jover to refresh some old patches to dpkg to make this possible, which revealed some quirks Vagrant Cascadian discovered when testing.

In addition, 21 reviews of Debian packages were added, 22 were updated and 16 were removed this month adding to our knowledge about identified issues. A number issue types have been added, such as new random_temporary_filenames_embedded_by_mesonpy and timestamps_added_by_librime toolchain issues.

In openSUSE, it was announced that their Factory distribution enabled bit-by-bit reproducible builds for almost all parts of the package. Previously, more parts needed to be ignored when comparing package files, but now only the signature needs to be deleted.

In addition, Bernhard M. Wiedemann published theunreproduciblepackage as a proper .rpm package which it allows to better test tools intended to debug reproducibility. Furthermore, it was announced that Bernhard’s work on a 100% reproducible openSUSE-based distribution will be funded by NLnet. He also posted another monthly report for his reproducibility work in openSUSE.

In GNU Guix, Janneke Nieuwenhuizen submitted a patch set for creating a reproducible source tarball for Guix. That is to say, ensuring that make dist is reproducible when run from Git. []

Lastly, in Fedora, a new wiki page was created to propose a change to the distribution. Titled “Changes/ReproduciblePackageBuilds”, the page summarises itself as a proposal whereby “A post-build cleanup is integrated into the RPM build process so that common causes of build irreproducibility in packages are removed, making most of Fedora packages reproducible.”

Mailing list news

On our mailing list this month:

  • Continuing a thread started in March 2024 about the Arch Linux minimal container now being 100% reproducible, John Gilmore followed up with a post about the practical and philosophical distinctions of local vs. remote storage of the various artifacts needed to build packages.

  • Chris Lamb asked the list which conferences readers are attending these days: “After peak Covid and other industry-wide changes, conferences are no longer the ‘must attend’ events they previously were… especially in the area of software supply-chain security. In rough, practical terms, it seems harder to justify conference travel today than it did in mid-2019.” The thread generated a number of responses which would be of interest to anyone planning travel in Q3 and Q4 of 2024.

  • James Addison wrote to the list about a “quirk” in Git related to its core.autocrlf functionality, thus helpfully passing on a “slightly off-topic and perhaps not of direct relevance to anyone on the list today” note that might still be “the kind of issue that is useful to be aware of if-and-when puzzling over unexpected git content / checksum issues (situations that I do expect people on this list encounter from time-to-time)”.


diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes such as uploading versions 263, 264 and 265 to Debian and made the following additional changes:

  • Don’t crash on invalid .zip files, even if we encounter their ‘badness’ halfway through the file and not at the time of their initial opening. []
  • Prevent odt2txt tests from always being skipped due to an (impossibly) new version requirement. []
  • Avoid parens-in-parens in test ‘skipping’ messages. []
  • Ensure that tests with >=-style version constraints actually print the tool name. []

In addition, Fay Stegerman fixed a crash when there are (invalid) duplicate entries in .zip which was originally reported in Debian bug #1068705). [] Fay also added a user-visible ‘note’ to a diff when there are duplicate entries in ZIP files []. Lastly, Vagrant Cascadian added an external tool pointer for the zipdetails tool under GNU Guix [] and proposed updates to diffoscope in Guix as well [] which were merged as [264] [265], fixed a regression in test coverage and increased verbosity of the test suite[].

Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:


reprotest is our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences. This month, reprotest version 0.7.27 was uploaded to Debian unstable) by Vagrant Cascadian who made the following additional changes:

  • Enable specific number of CPUs using --vary=num_cpus.cpus=X. []
  • Consistently use 398 days for time variation, rather than choosing randomly each time. []
  • Disable builds of arch:any packages. []
  • Update the description for the build_path.path option in README.rst. []
  • Update escape sequences for compatibility with Python 3.12. (#1068853). []
  • Remove the generic ‘upstream’ signing-key [] and update the packages’ signing key with the currently active team members [].
  • Update the packaging Standards-Version to 4.7.0. []

In addition, Holger Levsen fixed some spelling errors detected by the spellintian tool [] and Vagrant Cascadian updated reprotest in GNU Guix to 0.7.27.

Reproducibility testing framework

The Reproducible Builds project operates a comprehensive testing framework running primarily at in order to check packages and other artifacts for reproducibility.

In April, an enormous number of changes were made by Holger Levsen:

  • Debian-related changes:

    • Adjust for changed internal IP addresses at Codethink. []
    • Automatically cleanup failed diffoscope user services if there are too many failures. [][]
    • Configure two new nodes at [][]
    • Schedule Debian experimental even less. [][]
  • Breakage detection:

    • Exclude currently building packages from breakage detection. []
    • Be more noisy if diffoscope crashes. []
    • Health check: provide clickable URLs in jenkins job log for failed pkg builds due to diffoscope crashes. []
    • Limit graph to about the last 100 days of breakages only. []
    • Fix all found files with bad permissions. []
    • Prepare dealing with diffoscope timeouts. []
    • Detect more cases of failure to debootstrap base system. []
    • Include timestamps of failed job runs. []
  • Documentation updates:

    • Document how to access arm64 nodes at Codethink. []
    • Document how to use []
    • Drop notes about long stalled LeMaker HiKey960 boards sponsored by HPE and hosted at ETH. []
    • Mention osuosl4 and osuosl5 and explain their usage. []
    • Mention that some packages are built differently. [][]
    • Improve language in a comment. []
    • Add more notes how to query resource usage from []
  • Node maintenance:

    • Add ionos4 and ionos14 to THANKS. [][][][][]
    • Deprecate Squid on ionos1 and ionos10. []
    • Drop obsolete script to powercycle arm64 architecture nodes. []
    • Update system_health_check for new proxy nodes. []
  • Misc changes:

    • Make the script more robust. [][]
    • Update my SSH public key. []

In addition, Mattia Rizzolo added some new host details. []

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

10 May, 2024 10:05AM

May 09, 2024

Vincent Sanders

Bee to the blossom, moth to the flame; Each to his passion; what's in a name?

I like the sentiment of Helen Hunt Jackson in that quote and it generally applies double for computer system names. However I like to think when I named the first NetSurf VM host server phoenix fourteen years ago I captured the nature of its continuous cycle of replacement.

Image of the fourth phoenix server
We have been very fortunate to receive a donated server to replace the previous every few years and the very generous folks at Collabora continue to provide hosting for it.

Recently I replaced the server for the third time. We once again were given a replacement by Huw Jones in the form of a SuperServer 6017R-TDAF system with dual Intel Xeon Ivy Bridge E5-2680v2 processors. There were even rack rails!

The project bought some NVMe drives and an adaptor cards and I attempted to arrange to swap out the server in January.

The old phoenixiii server being replaced
Here we come to the slight disadvantage of an informal arrangement where access to the system depends upon a busy third party. Unfortunately it took until May to arrange access (I must thank Vivek again for coming in on a Saturday to do this)

In the intervening time, once I realised access was going to become increasingly difficult, I decided to obtain as good a system as I could manage to reduce requirements for future access. 

I turned to eBay and acquired a slightly more modern SuperServer with dual Intel Xeon Haswell E5-2680v3 processors which required purchase of 64G of new memory (Haswell is a DDR4 platform). 

I had wanted to use Broadwell processors but this exceeded my budget and would only be a 10% performance uplift (The chassis, motherboard and memory cost £180 and another £50 for processors was just too much, maybe next time)

graph of cpu mark improvements in the phoenix servers over time
While making the decision on the processor selection I made a quick chart of previous processing capabilities (based on a passmark comparison) of phoenix servers and was startled to discover I needed a logarithmic vertical axis. Multi core performance of processors has improved at a startling rate in the last decade.

When the original replacement was donated I checked where the performance was limited and noticed it was mainly in disc access which is what prompted the upgrade to NVMe (2 gigabytes a second peek read throughput) which moved the bottleneck to the processors where, even with the upgrades, it remains.

I do not really know if there is a conclusion here beyond noting NetSurf is very fortunate as a project to have some generous benefactors both for donating hardware and hosting for which I know all the developers are grateful.

Now I just need to go and migrate a huge bunch of virtual machines and associated sysadmin to make use of these generous donations.

09 May, 2024 09:24PM by Vincent Sanders (

Thorsten Alteholz

My Debian Activities in April 2024

FTP master

This month I accepted 386 and rejected 39 packages. The overall number of packages that got accepted was 386.

I also added lots of +moreinfo tags to some RM bugs. Is it that hard to check the reverse dependencies on your own?

Debian LTS

This was my hundred-eighteenth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.

During my allocated time I uploaded:

  • [DLA 3781-1] libgd2 security update for three CVEs to fix out-of-bounds reads or NULL pointer derefence
    • [DLA 3784-1] libcaca security update for two CVEs to fix heap buffer overflows
      • [DLA 3805-1] qtbase-opensource-src security update for seven CVEs to fix buffer overflows, infinite loops or application crashs due to processing of crafted input files. When trying to compile the fixed version, I got an error that there is no .compare() for QByteArray available. Yes, indeed, QByteArray::compare, which was used in a patch, was introduced only in Qt 6.0. So I had to backport that to Buster as well. It is astonishing that such a basic function was not needed before.
      • [#1070153] bookworm-pu: qtbase-opensource-src/5.15.8+dfsg-11+deb12u2 to fix two CVEs
      • [#1070154] bullseye-pu: qtbase-opensource-src/5.15.2+dfsg-9+deb11u1 to fix ten CVEs
      • [#1064550] uploaded libjwt
      • [#1067544] uploaded libmicrohttpd

      I also continued to work on tiff and last but not least did a week of FD and attended the monthly LTS/ELTS meeting.

      Debian ELTS

      This month was the sixty-ninth ELTS month. During my allocated time I uploaded:

      • [ELA-1069-1]libgd2 security update for three CVEs to fix out-of-bounds reads or NULL pointer derefence in Jessie and Stretch
      • [ELA-1070-1]libcaca security update for two CVEs to fix heap buffer overflows in Jessie and Stretch
      • [ELA-1083-1]qtbase-opensource-src security update for five CVEs to fix buffer overflows, infinite loops or application crashs due to processing of crafted input files in Stretch

      I also continued to work on an update for tiff in Jessie and Stretch, did a week of FD and attended the LTS/ELTS meeting.

      Debian Printing

      This month I uploaded new upstream or bugfix versions of:

      In preparation for cups3 I introduced a new package:

      This work is generously funded by Freexian!

      Debian Astro

      This month I uploaded a new upstream or bugfix version of:

      Debian IoT

      This month I uploaded new upstream or bugfix versions of:

      Debian Mobcom

      This month I uploaded new upstream or bugfix versions of:

      I have done these uploads in preparation for my GSoC student, who will be officially announced in May.


      I am sorry for people still using 32bit computers, but from my point of view these are dying architectures. So if there are any problems with builds on those architectures, I no longer try to fix them but file RM bugs. Patches are welcome, but I am no longer willing to spend any time for this.

      This month I uploaded new upstream or bugfix versions of:

      This month I even found some time to introduce new packages:

      • mailio a cross platform C++ library for email support
      • pksc11-proxy a proxy for the PKCS11-library

09 May, 2024 11:44AM by alteholz

hackergotchi for Gunnar Wolf

Gunnar Wolf

Hacks, leaks, and revelations • The art of analyzing hacked and leaked data

This post is a review for Computing Reviews for Hacks, leaks and revelations • The art of analyzing hacked and leaked data , a book published in No Starch Press

Imagine you’ve come across a trove of files documenting a serious deed and you feel the need to “blow the whistle.” Or maybe you are an investigative journalist and this whistleblower trusts you and wants to give you said data. Or maybe you are a technical person, trusted by said journalist to help them do things right–not only to help them avoid being exposed while leaking the information, but also to assist them in analyzing the contents of the dataset. This book will be a great aid for all of the above tasks.

The author, Micah Lee, is both a journalist and a computer security engineer. The book is written entirely from his experience handling important datasets, and is organized in a very logical and sound way. Lee organized the 14 chapters in five parts. The first part–the most vital to transmitting the book’s message, in my opinion–begins by talking about the care that must be taken when handling a sensitive dataset: how to store it, how to communicate it to others, sometimes even what to redact (exclude) so the information retains its strength but does not endanger others (or yourself). The first two chapters introduce several tools for encrypting information and keeping communication anonymous, not getting too deep into details and keeping it aimed at a mostly nontechnical audience.

Something that really sets this book apart from others like it is that Lee’s aim is not only to tell stories about the “hacks and leaks” he has worked with, or to present the technical details on how he analyzed them, but to teach readers how to do the work. From Part 2 onward the book adopts a tutorial style, teaching the reader numerous tools for obtaining and digging information out of huge and very timely datasets. Lee guides the reader through various data breaches, all of them leaked within the last five years: BlueLeaks, Oath Keepers email dumps, Heritage Foundation, Parler, Epik, and Cadence Health. He guides us through a tutorial on using the command line (mostly targeted at Linux, but considering MacOS and Windows as well), running Docker containers, learning the basics of Python, parsing and filtering structured data, writing small web applications for getting at the right bits of data, and working with structured query language (SQL) databases.

The book does an excellent job of fulfilling its very ambitious aims, and this is even more impressive given the wide range of professional profiles it is written for; that being said, I do have a couple critiques. First, the book is ideologically loaded: the datasets all exhibit the alt-right movement that has gained strength in the last decade. Lee takes the reader through many instances of COVID deniers, rioters for Donald Trump during the January 2021 attempted coup, attacks against Black Lives Matter activists, and other extremism research; thus this book could alienate right-wing researchers, who might also be involved in handling important whistleblowing cases.

Second, given the breadth of the topic and my 30-plus years of programming experience, I was very interested in the first part of each chapter but less so in the tutorial part. I suppose a journalist reading through the same text might find the sections about the importance of data handling and source protection to be similarly introductory. This is unavoidable, of course, given the nature of this work. However, while Micah Lee is an excellent example of a journalist with the appropriate technical know-how to process the types of material he presents as examples, expecting any one person to become a professional in both fields is asking too much.

All in all, this book is excellent. The writing style is informal and easy to read, the examples are engaging, and the analysis is very good. It will certainly teach you something, no matter your background, and it might very well complement your professional skills.

09 May, 2024 04:24AM

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppArmadillo on CRAN: Upstream Bugfix

armadillo image

Armadillo is a powerful and expressive C++ template library for linear algebra and scientific computing. It aims towards a good balance between speed and ease of use, has a syntax deliberately close to Matlab, and is useful for algorithm development directly in C++, or quick conversion of research code into production environments. RcppArmadillo integrates this library with the R environment and language–and is widely used by (currently) 1144 other packages on CRAN, downloaded 34.2 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint / vignette) by Conrad and myself has been cited 583 times according to Google Scholar.

Conrad released a new upstream bugfix yesterday (for a corner case with fftw3). We uploaded it yesterday too but it took a day for the hard-working CRAN maintainers to concur that the one (!) NOTE from reverse-dependency checking over 1100 packages was in a fact a false positve. And so it appeared on CRAN (very) early this morning. We also made a change removing a long-redundant setter for C++11 mode via the plugin. No other changes were made.

The set of changes since the last CRAN release follows.

Changes in RcppArmadillo version (2024-05-07)

  • Upgraded to Armadillo release 12.8.3 (Cortisol Injector)

    • Fix issue in fft() and fft2() in multi-threaded contexts with FFTW3 enabled
  • No longer set C++11 for the Rcpp plugin as this standard has been the default by R for very long time now.

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

09 May, 2024 01:54AM

May 07, 2024

John Goerzen

Photographic comparison: Is the Kobo Libra Colour display worse than the Kobo Libra 2?

I’ve been using E Ink-based ereaders for quite a number of years now. I’ve had my Kobo Libra 2 for a few years, and was looking forward to the Kobo Libra Colour — the first color E Ink display in a mainstream ereader line.

I found the display to be a mixed bag; contrast seemed a lot worse on B&W images, and the device “backlight” (it’s not technically a “back” light) seemed to cause a particular contrast reduction in dark mode. I went searching for information on this. I found a lot of videos on “Kobo Libra 2 vs Libra Colour” and so forth, but they were all pretty much useless. These were the mistakes they made:

  • Being videos. Photos would show the differences in better detail.
  • Shooting videos with cameras with automatic light levels. Since the thing we’re trying to evaluate here is how much darker the Kobo Libra Colour screen is than the Kobo Libra screen, having a camera that automatically adjusts for brighter or darker images defeats the purpose. Cell phone cameras (still and video) all do this by default and I saw evidence of it in all the videos.
  • Placing the two devices side-by-side instead of in identical locations for subsequent shots. This led to different shadows on each device (because OF COURSE the people shooting videos had to have their phone and head between the light source and the device), again preventing a good comparison.

So I dug out my Canon DSLR, tripod, and set up shots. Every shot here is set at ISO 100. Every shot in the same setting has the same exposure settings, which I document. The one thing I forgot to shut off was automatic white balance; you can notice it is active if you look closely at the backgrounds, but WB isn’t really relevant to this comparison anyhow.

Because there has also been a lot of concern about how well fine B&W details will show up on the Kobo Libra Colour screen, I shot all photos using a PDF test image from the open source hplip package ( converted to PDF). This also rules out font differences between the devices. I ensured a full screen refresh before each shot.

This is all because color E Ink is effectively a filter called Kaleido over the B&W layer. This causes dimming and some other visual effects.

You can click on any image here to see a full-resolution view. The full-size images are the exact JPEG coming from the camera, with only two modifications: 1) metadata has been redacted for privacy reasons, and 2) some images were losslessly rotated after the shoot.

OK, onwards!

Outdoors, bright sun, shot from directly overhead

Bright sun is ideal lighting for an E Ink display. They need no lighting at all in this scenario, and in fact, if you turn on their internal display light, it will probably not be very noticeable. Of course, this is in contrast to phone LCD screens, for which bright sunlight is the worst.

Scene: Morning sunlight reaching the ereaders at an angle. The angle was sufficient so that no shadows were cast by the camera or tripod.

Device light: Off on both

Exposure: 1/160, f16, ISO 100

You can see how much darker the Libra Colour is here. Though in these bright conditions, it is still plenty bright. There may actually be situations in which the Libra 2 is too bright in direct sunlight, requiring a person to squint or whatnot.

Looking at the radial lines, it is a bit difficult to tell because the difference in brightness, but I don’t see a hugely obvious reduction in quality in the Libra 2. Later I have a shot where I try to match brightness, and we’ll check it out again there.

Outdoors, shade, shot from directly overhead

For the next shot, I set the ereaders in shade, but still well-lit with the diffuse sunlight from all around.

The first two have both device lights off. For the third, I set the device light on the Kobo Colour to 100%, full cool shade, to try to see how close I could get it to the Libra 2 brightness. (Sorry it looks like I forgot to close the toolbar on the Colour for this set, but it doesn’t modify the important bits of the underlying image.)

Device light: Initially off on both

Exposure: 1/60, f6.4, ISO 100

Here you can see the light on the Libra Colour was nearly able to match the brightness on the Libra 2.

Indoors, room lit with overhead and window light, device light off

We continue to move into dimmer light with this next shot.

Device light: Off on both

Exposure: 1/4, f5, ISO 100

Indoors, room lit with overhead and window light, device light on

Now we have the first head-to-head with the device light on. I set the Libra 2 to my favorite warmth setting, found a brightness that looked good, and then tried my best to match those settings on the Libra Colour. My camera’s light meter aided in matching brightness.

Device light: On (Libra 2 at 40%, Libra Colour at 59%)

Exposure: 1/8, f5, ISO 100

(Apparently I am terrible at remembering to dismiss menus, sigh.)

Indoors, dark room, dark mode, at an angle

The Kobo Libra Colour surprised me with its dark mode. When viewed at an oblique angle, the screen gets pretty washed out. I maintained the same brightness settings here as I did above. It is much more noticeable when the brightness is set down to my preferred nighttime level (4%), or with a more significant angle.

Since you can’t see my tags, the order of the photos here will be: Libra 2 (standard orientation), Colour (standard orientation), Colour (turned around.

Device light: On (as above)

Exposure: 1/4, f5.6, ISO 100

Notice how I said I maintained the same brightness settings as before, and yet the Libra Colour looks brighter than the Libra 2 here, whereas it looked the same in the prior non-dark mode photos. Here’s why. I set the exposure of each set of shots based on camera metering. As we have seen from the light-off photos, the brightness of a white pixel is a lot less on a Libra Colour than on the Libra 2. However, it is likely that the brightness of a black pixel is about that same. Therefore, contrast on the Libra Colour is lower than on the Libra 2. The traditional shot is majority white pixels, so to make the Libra Colour brightness match that of the Libra 2, I had to crank up the brightness on the Libra Colour to compensate for the darker “white” background. With me so far?

Now with the inverted image, you can see what that does. It doesn’t just raise the brightness of the white pixels, but it also raises the brightness of the black pixels. This is expected because we didn’t raise contrast, only brightness.

Also, in the last image, you can see it is brighter to the right. Again, other conditions that are more difficult to photograph make that much more pronounced. Viewing the Libra Colour from one side (but not the other), in dark mode, with the light on, produces noticeably worse contrast on one side.


This isn’t a slam dunk. Let’s walk through this:

I don’t think there is any noticeable loss of detail on the Libra Colour. The radial lines appeared as well defined on it as on the Libra 2. Oddly, with the backlight, some striations were apparent in the gray gradient test, but I wouldn’t be using an E Ink device for clear photographic reproduction anyhow.

If you read mostly black and white: If you had been using a Kobo Libra Colour and were handed a Libra 2, you would go, “Wow! What an upgrade! The screen is so much brighter!” There’s little reason to get a Libra Colour. The Libra 2 might be hard to find these days, but the new Clara BW (with a 6″ instead of the 7″ screen on the Libra series) might be just the thing for you. The Libra 2 is at home in any lighting, from direct sun to pitch black, and has all the usual E Ink benefits (eg, battery life measured in weeks) and drawbacks (slower refresh rate) that we’re all used to.

If you are interested in photographic color reproduction mostly indoors: Consider a small tablet. The Libra Colour’s 4096 colors are going to appear washed out compared to what you’re used to on a LCD screen.

If you are interested in color content indoors and out: The Libra Colour might be a good fit. It could work well for things where superb color rendition isn’t essential — for instance, news stories (the Pocket integration or Calibre’s news feature could be nice there), comics, etc.

In a moderately-lit indoor room, it looks like the Libra Colour’s light can lead it to results that approach Libra 2 quality. So if most of your reading is in those conditions, perhaps the Libra Colour is right for you.

As a final aside, I wrote in this article about the Kobo devices. I switched from Kindles to Kobos a couple of years ago due to the greater openness of the Kobo devices (you can add things like Nickel Menu and KOReader to them, and they have built-in support for more useful formats), their featureset, and their cost. The top-of-the-line Kindle devices will have a screen very similar if not identical to the Libra 2, so you can very easily consider this to be a comparison between the Oasis and the Libra Colour as well.

07 May, 2024 10:16PM by John Goerzen

hackergotchi for Thomas Lange

Thomas Lange

Removing tens of thousands of web pages

In January I've removed tens of thousands of web pages on Have you noticed it?

In the past

From 1997 onwards, we had web pages for security announcements. We had to manually prepare a .data and a .wml file which then generated a web page for each security announcement (DSA or DLA). We have listed the 6 most recent messages in a short list that was created from these files. Most of the work that went into the Debian web pages was creating these files.

Our search engine often listed the pages with security announcements instead of a more relevant web page for a particular topic.


At DebConf Kosovo (2022) I started with a proof of concept and wrote a script, that generates this list without using the .data/.wml files in the Git repository, but instead reading the primary sources of security information[1]. This new list now includes links to the security tracker and the email of the announcement.

Following web pages and scripts were also using these .data and .wml files:

  • OVAL files
  • RSS feeds for security announcements (and LTS)
  • Apache config file for mapping URLs from dsa-NNN to YEAR/dsa-NNN
  • A huge list of crossreferences between DSA and CVE numbers

Before I could remove all the security web pages, I had to adjust the scripts, that create the above information.

When I looked at the OVAL files and the apache logs of our web server, I saw that more than 99% of the web traffic was generated by these XML files (134TB of 135TB total in two weeks). They were not compressed and were around 50MB in size. With the help of Carsten Schönert we managed to modify the python scripts that generate this OVAL file without using the .data/.wml files and now we only provide bzip2 compressed XML files[2].

The RSS feeds are created by the new Perl script which reads the DSA/DLA list the security tracker and determines the URL of the email of all entries. This script also generates the list of the most recent DSA/DLA entries. Currently we show the last 350 entries which covers more than the last year and includes links to the announcement email and the security tracker.

The huge list of crossreferences is not needed any more, since the mapping of CVE to DSA is already included in the DSA list[3] of the security tracker.

The amount of translations of the DSA/DLA was very different. French translations were almost all done, but all other languages did translations for a couple of months or years only. E.g. in 2022, Italian had 2 translations, Russian 15, Danish 212, French and English each 279. But from 2023 on only French translations were made. By generating the list of DSA/DLA we lost the ability to translate these web pages, but since these announcements are made of simple, identical sentences it is easy to use an automatic translation service if needed.

Now the translation statistics of all web pages are more accurate. Instead of 12200 pages that need to be translated (including all these old DSA/DLA) there are now only 2500 pages to translate[4]. Languages that had a lot of old translations of DSA/DLA lost some percentage but languages that are doing translations of newer web pages won in the statistics of how many pages are translated. Examples:


German (de)   3501  28.5%
Italian (it)  1005   8.2%
Danish (da)   6336  51.7%


German (de)   1486  59.0%
Italian (it)   909  36.1%
Danish (da)    982  39.0%

Cleanup of all the security web pages

Finally in January, I could remove all web pages of the security announcements in one git commit[5]. Using several git rm -rf commands this commit removed 54335 files, including around 9650 DSA/DLA data files, 44189 wml files, nearly 500 Makefiles.


No more manual work is needed for the security team and we now have direct links from a DSA-NNN/DLA-NNN to the email in our mailing list archive. This was not possible before. The search results became more accurate.

But we still host a lot of other old content on the Debian web pages which may be removed in the future.






07 May, 2024 10:08AM

May 04, 2024

Sven Hoexter

vym - view your mind

Had a need for a mindmapping application and found view your mind in the archive. Works but the version is a bit rusty. Sadly my Debian packaging skills are a bit rusty as well, especially when it comes to bigger GUI applications. Thus I spent a good chunk of yesterday afternoon to rip out cdbs and package the last source release on github which is right now 2.9.22 (the release branch already has 2.9.27, still sorting that out).

Git repository and a amd64 build of the current state. It still deserves some additional love, e.g. creating a -common package for arch indep content.

Proposed a few changes upstream:

Also pinged pollux@ who uploaded vym up to 2019 if he'd be fine if I pick it up. If someone else is interested, I'm also fine to put it up on salsa in the general "Debian" group for shared maintenance. I guess I will use it in the future, but time is still a scarce resource for all of us.

04 May, 2024 03:03PM

hackergotchi for Steve Kemp

Steve Kemp

The CP/M emulator development continues

In my previous post I introduced a toy CP/M Emulator I'd been working on.

At the time it was capable of running the Infocom text-based adventure games, so I thought it was done. Of course I also wanted to run Microsoft's original BASIC and it turned out that was a challenge because the coding of their interpreter didn't use the standard CP/M entry-point for making syscalls (call 0x0005).

Instead of calling 0x0005 to invoke the BDOS/BIOS functions the BASIC interpreter used the single-byte CALL instructions which are available on the Z80 processor. There are a bunch of these instructions:

  • RST 00
  • RST 08
  • RST 10
  • RST 18
  • RST 20
  • RST 28
  • RST 30
  • RST 38

Each of those instructions is equivalent to a call instruction with a fixed offset, "call 0x0010", "call 0x0020", etc. I had to rework the emulator to cope with this approach, which causes repetition but nothing too surprising. The end result is that now my emulator can run Microsoft Basic, Tasty Basic, and some more programs.

Things work but a couple of the syscalls are of the form "Return true if there is a pending keystroke", or "wait until there is keyboard input present and return the first character". I have some busy-loops which peg the CPU, which sucks but works. On the downside running the code on a MacOS machine has some weird issues with repeated keys and similar. So I need to look into fixing that for my own sense of peace.

I put together a little repository of binaries for playing with though, and that's been helpful. My emulator has a special flag which treats sub-directories as "Drives". So A: points to A/, B: points to B/, etc. That makes distributing and working with things easy!

04 May, 2024 12:00PM

May 03, 2024

hackergotchi for Colin Watson

Colin Watson

Playing with rich

One of the things I do as a side project for Freexian is to work on various bits of business automation: accounting tools, programs to help contributors report their hours, invoicing, that kind of thing. While it’s not quite my usual beat, this makes quite a good side project as the tools involved are mostly rather sensible and easy to deal with (Python, git, ledger, that sort of thing) and it’s the kind of thing where I can dip into it for a day or so a week and feel like I’m making useful contributions. The logic can be quite complex, but there’s very little friction in the tools themselves.

A recent case where I did run into some friction in the tools was with some commands that need to present small amounts of tabular data on the terminal, using OSC 8 hyperlinks if the terminal supports them: think customer-related information with some links to issues. One of my colleagues had previously done this using a hack on top of texttable, which was perfectly fine as far as it went. However, now I wanted to be able to add multiple links in a single table cell in some cases, and that was really going to stretch the limits of that approach: working out the width of the displayed text in the cell was going to take an annoying amount of bookkeeping.

I started looking around to see whether any other approaches might be easier, without too much effort (remember that “a day or so a week” bit above). ansiwrap looked somewhat promising, but it isn’t currently packaged in Debian, and it would have still left me with the problem of figuring out how to integrate it into texttable, which looked like it would be quite complicated. Then I remembered that I’d heard good things about rich, and thought I’d take a look.

rich turned out to be exactly what I wanted. Instead of something like this based on the texttable hack above:

import shutil
from pyxian.texttable import UrlTable

termsize = shutil.get_terminal_size((80, 25))
table = UrlTable(max_width=termsize.columns)
table.add_row([(issue_url, f"#{issue_id}")]

… now I can do this instead:

import rich
from rich import box
from rich.table import Table

table = Table(box=box.SIMPLE)

While this is a little shorter, the real bonus is that I can now just put multiple [link] tags in a single string, and it all just works. No ceremony. In fact, once the relevant bits of code passed type-checking (since the real code is a bit more complex than the samples above), it worked first time. It’s a pleasure to work with a library like that.

It looks like I’ve only barely scratched the surface of rich, but I expect I’ll reach for it more often now.

03 May, 2024 03:09PM by Colin Watson

May 02, 2024

hackergotchi for Sergio Talens-Oliag

Sergio Talens-Oliag

The Freaky Wall (Part 2: Initial Installation)

For the Freaky Wall I have installed a Debian GNU/kFreeBSD system using the installer with ZFS support announced on:

I used the mini.iso found on:

the 12th of October of 2010; as I had some problems and reported them to Robert is possible that the current image solves part of them.

Installation plan

I did a standard installation on a machine with two hard disks, but only used the first one from the installer.

The plan was to use ZFS with RAID-1, but current versions of grub do not support booting from a ZFS + RAID file system, so I had to use the same technique used for Linux for a long time; three partitions: a swap partition, a small /boot partition and a big partition for /; / and /boot were formated to use ZFS.

First reboot

After the installation the system failed to boot because of a bug when building the /boot/grub/grub.cfg (some paths were missing a //@ prefix); to be able to boot Iwe edited the config on the grub prompt and later fixed the file:

    --- grub.cfg.orig       2010-10-13 16:40:39.000000000 +0200
    +++ grub.cfg    2010-10-13 18:38:47.535436766 +0200
    @@ -64,7 +64,7 @@
            set root='(hd0,1)'
            search --no-floppy --fs-uuid --set a371979bb836d1fe
            echo                    'Loading kernel of FreeBSD 8.1-1-amd64 ...'
    -       kfreebsd                /kfreebsd-8.1-1-amd64.gz
    +       kfreebsd                //@/kfreebsd-8.1-1-amd64.gz
            insmod part_msdos
            insmod zfs
            set root='(hd0,3)'
    @@ -75,7 +75,7 @@
            insmod zfs
            set root='(hd0,1)'
            search --no-floppy --fs-uuid --set a371979bb836d1fe
    -       kfreebsd_module         /zfs/zpool.cache type=/boot/zfs/zpool.cache
    +       kfreebsd_module         //@/zfs/zpool.cache type=/boot/zfs/zpool.cache
            set kFreeBSD.vfs.root.mountfrom=zfs:dkfbf1-ad4s3
            set kFreeBSD.vfs.root.mountfrom.options=rw

I haven't tested the installer since that day, but I believe that the current ZFS installer was fixed by Robert to deal with that problem.

Once the system was booted I had to fix a couple of things:

  • The keyboard configuration was wrong, but it was easy to fix the Debian Way:

    dpkg-reconfigure kbdcontrol
  • The /boot partition was mounted on /target/boot, as that was what was recorded on the ZFS file system; to fix it I executed the following commands:

     # zfs set mountpoint=/     dkfbf1-ad4s3
     # zfs set mountpoint=/boot dkfbf1-ad4s1

    Where dkfbf1-ad4s3 is the root file system and dkfbf1-ad4s1 is the original /boot.

    I reported that to Robert also and I believe it is fixed on the ZFS installer now.

Adjusting ZFS to do RAID-1

On the second disk I created the same partitions as the ones on the first disk using parted; the final result was:

    # parted -l     
    Model: ST3250620NS/3BKS (ide)
    Disk /dev/ad6: 250GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos

    Number  Start   End     Size    Type     File system  Flags
     1      1049kB  256MB   255MB   primary
     2      256MB   4256MB  4000MB  primary
     3      4256MB  250GB   246GB   primary

    Model: ST3250620NS/3BKS (ide)
    Disk /dev/ad4: 250GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos

    Number  Start   End     Size    Type     File system     Flags
      1      1049kB  256MB   255MB   primary
      2      256MB   4256MB  4000MB  primary  linux-swap(v1)
      3      4256MB  250GB   246GB   primary

To use the second partition of both disks as swap I added the following to /etc/fstab:

    /dev/ad4s2      none            swap    sw              0       0
    /dev/ad6s2      none            swap    sw              0       0

To configure the mirroring for the root file system I did the following:

    zpool attach dkfbf1-ad4s3 ad4s3 ad6s3

As the /boot can't work as a replica I adjusted it to make two copies of every file:

    zfs   set copies=2     dkfbf1-ad4s1

Leaving the second disk copy alone, although my plan is to configure it to hold a copy of the /boot partition synchronized with rsync each night.

After all those changes the system didn't boot, as the grub-pc generates a buggy /boot/grub/grub.cfg; the problem is on the /etc/grub.d/10_kfreebsd section:

    ### BEGIN /etc/grub.d/10_kfreebsd ###
    menuentry 'Debian GNU/kFreeBSD, with kFreeBSD 8.1-1-amd64' --class debian \
      --class gnu-kfreebsd --class gnu --class os {
          insmod part_msdos
          insmod zfs
          set root='(hd0,1)'
          search --no-floppy --fs-uuid --set a371979bb836d1fe
          echo 'Loading kernel of FreeBSD 8.1-1-amd64 ...'
          kfreebsd /kfreebsd-8.1-1-amd64.gz
          set kFreeBSD.vfs.root.mountfrom=unknown:/dev/ad4s3
          set kFreeBSD.vfs.root.mountfrom.options=rw
    ### END /etc/grub.d/10_kfreebsd ###

To fix it there has to be a copy of the modules for ZFS on the boot partition (in my case I moved the /lib/modules directory to /boot and created a link on the root partition to the new directory):

    cd /boot
    mkdir lib
    mv /lib/modules lib
    cd /lib
    ln -s ../boot/lib/modules

And instead of fixing the /etc/grub.d/10_kfreebsd code I wrote a new script (/etc/grub.d/09_zfs_kfreebsd) that creates the right config for my current configuration on the grub.cfg file:

    . ${libdir}/grub/grub-mkconfig_lib

    prepare_boot_cache="$(prepare_grub_to_access_device ${GRUB_DEVICE_BOOT} | sed -e "s/^/\t/")"
    kfreebsd_versions="$(ls /lib/modules/)"
    zfs_root_device="$(zfs list | awk '/\/$/ { print $1 }' | head -1)"

    for kversion in $kfreebsd_versions; do
      cat << EOF
    # Entry when using ZFS (we have issues with /etc/grub.d/10_kfreebsd)
    menuentry 'Debian GNU/kFreeBSD, with kFreeBSD $kversion and ZFS' --class debian --class gnu-kfreebsd --class gnu --class os {
          echo                    'Loading kernel of FreeBSD $kversion ...'
          kfreebsd                //@/kfreebsd-$kversion.gz
          kfreebsd_module_elf     //@/lib/modules/$kversion/opensolaris.ko
          kfreebsd_module_elf     //@/lib/modules/$kversion/zfs.ko
          kfreebsd_module         //@/zfs/zpool.cache type=/boot/zfs/zpool.cache
          set kFreeBSD.vfs.root.mountfrom=zfs:$zfs_root_device
          set kFreeBSD.vfs.root.mountfrom.options=rw

I solved the problem this way to have a working solution that does not break with squeeze upgrades, assuming that a future grub-pc package will deal well with my config and I'll be able to remove this script, but I guess I'll have to install it from backports.

The entry generated by the script when called from update-grub will be similar to:

    ### BEGIN /etc/grub.d/09_zfs-kfreebsd ###
    # Entry when using ZFS (we have issues with /etc/grub.d/10_kfreebsd)
    menuentry 'Debian GNU/kFreeBSD, with kFreeBSD 8.1-1-amd64 @ ITI' --class debian --class gnu-kfreebsd --class gnu --class os {
          insmod part_msdos
          insmod zfs
          set root='(hd0,1)'
          search --no-floppy --fs-uuid --set a371979bb836d1fe
          echo                    'Loading kernel of FreeBSD 8.1-1-amd64 ...'
          kfreebsd                //@/kfreebsd-8.1-1-amd64.gz
          kfreebsd_module_elf     //@/lib/modules/8.1-1-amd64/opensolaris.ko
          kfreebsd_module_elf     //@/lib/modules/8.1-1-amd64/zfs.ko
          kfreebsd_module         //@/zfs/zpool.cache type=/boot/zfs/zpool.cache
          set kFreeBSD.vfs.root.mountfrom=zfs:dkfbf1-ad4s3
          set kFreeBSD.vfs.root.mountfrom.options=rw
    ### END /etc/grub.d/10_iti-kfreebsd ###

And after rebooting the machine with this new configuration the system boots OK.

On my next post I'll continue explaining how to compile a kernel that supports the use of the OpenBSD Packet Filter and related technologies (CARP, pflog, etc.).

02 May, 2024 05:11PM


I haven't blogged for a long time, but I've decided that I'm going to try to write again, at least about technical stuff.

My plan was to blog about the projects I've been working on lately, the main one being the setup of the latest version of Kolab with the systems we already have at work, but I'll do that on the next days.

Today I'm just going to make a list of the tools I use on a daily basis and my plans to start using additional ones in the near future.

Shells, Terminals and Text Editors

I do almost all my work on Z Shell sessions running inside tmux; for terminal emulation I use gnome-terminal on X, VX ConnectBot on Android systems and iTerm2 on Mac OS X.

For text editing I've been using Vim for a long time (even on Mobile devices) and while I'm aware I don't know half of the things it can do, what I know is good enough for my day to day needs.

In the past I also used Emacs as a programming editor and my main tool to write HTML, SGML and XML, but since I haven't really needed an IDE for a long time and I mainly use Lightweight Markup Languages I haven't used it for a long time (I briefly tried to use Org mode, but for some reason I ended up leaving it).

Documentation formats and tools

Since a long time ago I've been an advocate of Lightweight Markup Languages; I started to use LaTeX and Lout, then moved to SGML/XML formats (LinuxDoc and DocBook) and finally moved to plain text based formats.

I started using Wiki formats (parsewiki) and soon moved to reStructuredText; I also use other markup languages like Markdown (for this blog, aka ikiwiki) and tried MultiMarkdown to replace reStructuredText for general use, but as I never liked Markdown syntax I didn't liked an extended version of it.

While I've been using ReStructuredText for a long time, I recently found Asciidoctor and the Asciidoc format and I guess I'll be using it instead of rst whenever I can (I still need to try the slide backends and conversions to ODT, but if that works I guess I'll write all my new documents using Asciidoc).

Programming languages

I'm not a developer, but I read and patch a lot of free software code written on a lot of different programming languages (I wouldn't be able to write whole programs on most of them, but thanks to Stack Overflow I'm usually able to fix what I need).

Anyway, I'm able to program in some languages; I write a lot of shell scripts and I go for Python and C when I need something more complicated.

On the near future I plan to read about javascript programming and nodejs (I'll probably need it at work) and I already started looking at Haskell (I guess it was time to learn about functional programming and after reading about it, it looks like haskell is the way to go for me).

Version Control

For a long time I've been a Subversion user, at least for my own projects, but seems that everything has moved to git now and I finally started to use it (I even opened a github account) and plan to move all my personal subversion repositories at home and at work to git, including the move of all my debian packages from svn-buildpackage to git-buildpackage.

Further Reading

With the previous plans in mind, I've started reading a couple of interesting books:

Now I just need to get enough time to finish reading them ... ;)

02 May, 2024 05:11PM

The FreakyWall (Part 3: Packages)

In this post I'll describe the changes made to the kernel and some of the Squeeze packages for the Freaky Wall.

The plan is to submit whishlist bugs to the BTS on the hope of having all what is needed for this project available on Debian after the Squeeze release, as my feeling is that a freeze is not the right time to push this changes... ;)

I'm giving access here to all the changes made to the source packages, but if anyone wants the binary packages (amd64 only) send me an email and I'll give you the URL of an apt repository that contains all the modified packages (it's the one at work, that contains other modified packages) or, if there is interest, I can put them on


To be able to build the firewall we need a kFreeBSD kernel with some options not compiled on the version distributed with Debian.

To compile the kernel I've followed the procedure described on the following debian-bsd mailing list post:

Basically I've done the following:

    apt-get build-dep kfreebsd-8
    apt-get source kfreebsd-8
    cd kfreebsd-8-8.1
    cat >> debian/arch/amd64/amd64.config << EOF

    # Add pflog, pfsync, ALTQ and CARP support
    # ----------------------------------------

    device      pf
    device      pflog
    device      pfsync

    options         ALTQ
    options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
    options         ALTQ_RED        # Random Early Detection (RED)
    options         ALTQ_RIO        # RED In/Out
    options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
    options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
    options         ALTQ_NOPCC      # Required for SMP build

    device      carp
    vi debian/changelog 
    dpkg-buildpackage -B -uc

Once the package was built I installed the new kernel package and rebooted the machine.


To be able to use some utilities related to pf I have built patched versions of three packages:

  • freebsd-utils: I have included pflogd and ftp-proxy on the package and have added some options to allow the use of additional interface types with ifconfig (carp, pfsync, lagg, bridges, ...).

    There were a lot of changes needed:

    1. The pflogd and ftp-proxy Makefiles are missing from the original tarball, I modified the get-orig-source of the debian/rules, but I build my packages against the original tarball, with the extra files included directly on the package .diff.gz.

    2. The pflogd daemon needs a _pflogd user and group and a /var/run/_pflogd directory, so I've added the directory and the creation of the user and group to the package post-install.

    3. The ftp-proxy daemon uses the proxy user when dropping privileges and I've modified the code to make it use the /var/run/ftp-proxy directory for the chroot.

    4. Some C header files that are not available on -dev packages were missing and I put them on the packages' debian directory. I've created a debian/include directory and moved there the original debian/net directory and added the headers debian/include/net/if_bridgevar.h and debian/include/net/if_lagg.h to add bridging support to ifconfig and the debian/include/pcap-config.h and debian/include/pcap-int.h libpcap private headers to be able to compile the pflogd binary.

    As I'm not familiar with the way people changes code for kFreeBSD some of the patches are a little bit dirty, but at least things work; besides, probably I should also have had to include init.d scripts for pf, pflogd and ftp-proxy, but I have not done it at the package level yet, as what I got was enough to work with the tools.

    The debdiff against the freebsd-utils-8.1-2 source package is available here or here.

  • libpcap: A test had to be removed in order to be able to support pflog on the library; the debdiff against the libpcap-1.1.1-2 package is available here or here.

  • tcpdump: The package also had to be modified to include the code to print the pflogd entries on the pcap file; the debdiff against tcpdump_4.1.1-1 is available here or here.

On the next post I'll describe how I've configured the system, the network interfaces and the different utilities patched and compiled on this post.

02 May, 2024 05:11PM