Debian is a trademark of Software in the Public Interest, Inc. This site is operated independently in the spirit of point three of the Debian Social Contract which tells us We will not hide problems.

Feeds

May 20, 2024

hackergotchi for Daniel Pocock

Daniel Pocock

Veteran of successful Upfield Railway campaign backs Galway-Sligo Western Railway Corridor

The Upfield railway line in Melbourne, Australia was condemned to extinction three times by two successive Governments. Fortunately, due to the most vigorous pressure from the community, the Government backed down each time.

Nonetheless, as every other railway in Melbourne was upgraded to electronic signalling and automatic boom gates, the Upfield line continued to operate with human gatekeepers who sit in a little hut and come out to move the gates. In other words, residents of the inner north were left with the feeling that the railway was on life support, on the verge of obsolescence.

By the mid-1990s, the trains were no longer operating any later than 7pm while other lines had train service up to midnight. People felt it was only a matter of time before the trains would stop for good.

In 1997, the Government announced they would make a temporary six month closure of the railway so they could construct a freeway over the top of it. The community was suspicious that the six month temporary closure would become permanent.

I personally became involved in the Upgrade Upfield Coordinating Committee, which was the main activist group in the region, some time during 1996 or 1997 at this critical juncture in the Upfield railway saga.

There was no social media in the 1990s. We had to go out and talk to people in the trains, we spoke to people on the trams and busses and we organized a series of small but tactical protests that appeared on the evening news.

The Government kept their word to reopen the railway at the end of the freeway construction works. Patronage on the line has been steadily increasing.

In 2016, Melbourne newspaper The Age published a report about the legacy of our efforts, Not the end of the line: How people power saved the Upfield rail line. Nik Dow, pictured, is the person who recruited me to the campaign.

"We held the tram up for maybe two minutes trying to get wheelchairs and surfboards and bicycles on it," Mr Dow says."We gave out leaflets and let them go on their way and then we'd do the next tram. We might be there for half an hour, just getting the word out."

Everybody in the district began to see how stupid it would be to replace the trains with light-rail.

On 17 May, I attended the West=on=track forum to discuss the Galway-Sligo railway. The trains were stopped in the 1960s but the tracks are still there. Approximately five hundred people were in the room. There is solid community support for bringing back passenger trains on this railway.

Several leading candidates for the European parliament elections were present. As far as I can tell, I was the only candidate who is bringing real experience of a previous campaign to save a railway from extinction. Actions speak louder than words. I hope I can bring the spirit of success on the Upfield railway to the communities around the Galway-Sligo railway.

There are rival groups pushing to replace the Galway-Sligo line with a Greenway. Greenway is a term that has been created to sugarcoat the idea of Irish Rail ceding its title to the land.

When Melbourne constructed the Eastern Freeway in the 1970s, the Government kept space in the middle for railway tracks. Nobody ever built the train there. Today it is a greenway. In the gridlock that exists at rush hour, drivers sitting in their cars can look out the window and watch the grass growing.

If you want an expert from the Irish diaspora to bring the spirit of success to the Galway-Sligo rail project, please give your most vigorous support to my campaign for European parliament.

Eastern Freeway, Melbourne

Upfield line, 1998

20 May, 2024 12:00PM

Choosing an RSS feed reader

If you want to know about new articles on my blog without having to check social media then the best solution is an RSS feed reader.

You can get feed readers that work through a web site and you can get feed readers that are installed as apps in your phone.

The browser on your desktop or laptop can follow RSS feeds too.

Mozilla has published an article with suggestions about how to choose a feed reader.

Here is the feed URL for my blog. Please copy and paste it into your feed reader:

https://danielpocock.com/feed.xml

Here is the same link in a form you can click:

https://danielpocock.com/feed.xml

20 May, 2024 12:00PM

Can an election campaign succeed without social media accounts?

If an election campaign can't succeed without a social media account then democracy is already dead. Long live artificial intelligence. The algorithms will herd us around like sheep. Those are the same algorithms that co-opted British schoolgirl Molly Russell to commit suicide. Where are the algorithms taking your vote and is it a place you would want to go?

After I started publishing blogs about the Debian suicide cluster, whereby a number of tech workers have taken their own lives, people have noticed that some social media networks are graylisting links to my blog and refusing to let people share these factual articles. Coincidentally, many of the social media companies and Google are built with software created by the victims. The volunteers who took their own lives worked on this software, without payment, while owners and managers in Silicon Valley became very rich.

None of my blogs contain any details about suicide methods, personal details about the victims or any other inappropriate content. The blogs examine the workload and the social context around these deaths. Elon Musk promised he was going to make Twitter completely free of censorship. One of the first things that Musk did after acquiring Twitter was to ban the very journalists who published critical commentary about him. We can see that over $120,000 of money that may come from Google was used to stop me publishing critical commentary about the suicide cluster on sites related to Debian.

Social media companies were very outspoken about censoring the former US President Donald Trump. In most cases, social media networks do not tell the community when they are censoring certain people and web sites. It is all done secretly using graylists. If you share a link to a blog about inconvenient subjects, it will look like your post is successful but your followers will not see that post in the aggregated feeds from all the people they follow. They will only see your more controversial posts when they come to your personal page on the platform. This eliminates the possibility that certain topics can achieve a viral spread on social media.

Nonetheless, the world worked before the creation of social media. If you really want an expert from the Irish diaspora to work for you in this district, please consider some of the following actions that are more powerful than a share on social media:

  • Print some copies of my brochure and distribute it in your your workplace, school, shopping centers, public transport and anywhere else
  • Organize a meeting to review candidate profiles with your neighbours and friends
  • Send me an email invitation about any hustings or town hall events on any topics in your community
  • Use an RSS feed reader to receive updates from my blog
  • Block out time in your calendar for voting on 7 June. Create a reminder.
  • Agree to meet your friends or neighbors on 7 June and vote together

If you do want to share my campaign on social media then please feel free to do so. I'm not telling anybody else to stop using social media. But if you really want change, you need to go the extra mile to have discussions about this campaign.

Have a look at the words of Victoria's former transport minister Alan Brown. He notes that the Government felt the pressure of action in the community. They abandoned plans to close our railway line. We can bring that power to save the Galway-Sligo railway from total destruction. Governments are not afraid of social media. A like on social media doesn't have anywhere near the same power as a queue at the ballot box. If you want to make change, you need to get your friends out to vote and replace the people in charge.

20 May, 2024 12:00PM

Russell Coker

Respect and Children

I attended the school Yarra Valley Grammer (then Yarra Valley Anglican School which I will refer to as “YV”) and completed year 12 in 1990. The school is currently in the news for a spreadsheet some boys made rating girls where “unrapeable” was one of the ratings. The school’s PR team are now making claims like “Respect for each other is in the DNA of this school”. I’d like to know when this DNA change allegedly occurred because respect definitely wasn’t in the school DNA in 1990! Before I go any further I have to note that if the school threatens legal action against me for this post it will be clear evidence that they don’t believe in respect. The actions of that school have wronged me, several of my friends, many people who aren’t friends but who I wish they hadn’t had to suffer and I hadn’t had to witness it, and presumably countless others that I didn’t witness. If they have any decency they would not consider legal action but I have learned that as an institution they have no decency so I have to note that they should read the Wikipedia page about the Streisand Effect [1] and keep it in mind before deciding on a course of action.

I think it is possible to create a school where most kids enjoy being there and enjoy learning, where hardly any students find it a negative experience and almost no-one finds it traumatic. But it is not possible to do that with the way schools tend to be run.

When I was at high school there was a general culture that minor sex crimes committed by boys against boys weren’t a problem, this probably applied to all high schools. Things like ripping a boy’s pants off (known as “dakking”) were considered a big joke. If you accept that ripping the pants off an unwilling boy is a good thing (as was the case when I was at school) then that leads to thinking that describing girls as “unrapeable” is acceptable. The Wikipedia page for “Pantsing” [2] has a reference for this issue being raised as a serious problem by the British Secretary of State for Education and Skills Alan Johnson in 2007. So this has continued to be a widespread problem around the world. Has YV become better than other schools in dealing with it or is Dakking and Wedgies as well accepted now as it was when I attended? There is talk about schools preparing kids for the workforce, but grabbing someone’s underpants without consent will result in instant dismissal from almost all employment. There should be more tolerance for making mistakes at school than at work, but they shouldn’t tolerate what would be serious crimes in other contexts. For work environments there have been significant changes to what is accepted, so it doesn’t seem unreasonable to expect that schools can have a similar change in culture.

One would hope that spending 6 years wondering who’s going to grab your underpants next would teach boys the importance of consent and some sympathy for victims of other forms of sexual assault. But that doesn’t seem to happen, apparently it’s often the opposite.

When I was young Autism wasn’t diagnosed for anyone who was capable of having a normal life. Teachers noticed that I wasn’t like other kids, some were nice, but some encouraged other boys to attack me as a form of corporal punishment by proxy – not a punishment for doing anything wrong (detentions were adequate for that) but for being different. The lesson kids will take from that sort of thing is that if you are in a position of power you can mistreat other people and get away with it. There was a girl in my year level at YV who would probably be diagnosed as Autistic by today’s standards, the way I witnessed her being treated was considerably worse than what was described in the recent news reports – but it is quite likely that worse things have been done recently which haven’t made the news yet. If this issue is declared to be over after 4 boys were expelled then I’ll count that as evidence of a cover-up. These things don’t happen in a vacuum, there’s a culture that permits and encourages it.

The word “respect” has different meanings, it can mean “treat a superior as the master” or “treat someone as a human being”. The phrase “if you treat me with respect I’ll treat you with respect” usually means “if you treat me as the boss then I’ll treat you as a human being”. The distinction is very important when discussing respect in schools. If teachers are considered the ultimate bosses whose behaviour can never be questioned then many boys won’t need much help from Andrew Tate in developing the belief that they should be the boss of girls in the same way. Do any schools have a process for having students review teachers? Does YV have an ombudsman to take reports of misbehaving teachers in the way that corporations typically have an ombudsman to take reports about bad managers? Any time you have people whose behaviour is beyond scrutiny or oversight you will inevitably have bad people apply for jobs, then bad things will happen and it will create a culture of bad behaviour. If teachers can treat kids badly then kids will treat other kids badly, and this generally ends with girls being treated badly by boys.

My experience at YV was that kids barely had the status of people. It seemed that the school operated more as a caretaker of the property of parents than as an organisation that cares for people. The current YV website has a Whistleblower policy [3] that has only one occurrence of the word “student” and that is about issues that endanger the health or safety of students. Students are the people most vulnerable to reprisal for complaining and not being listed as an “eligible whistleblower” shows their status. The web site also has a flowchart for complaints and grievances [4] which doesn’t describe any policy for a complaint to be initiated by a student. One would hope that parents would advocate for their children but that often isn’t the case. When discussing the possibility of boys being bullied at school with parents I’ve had them say things like “my son wouldn’t be so weak that he would be bullied”, no boy will tell his parents about being bullied if that’s their attitude! I imagine that there are similar but different issues of parents victim-blaming when their daughter is bullied (presumably substituting immoral for weak) but don’t have direct knowledge of the topic. The experience of many kids is being disrespected by their parents, the school system, and often siblings too. A school can’t solve all the world’s problems but can ideally be a refuge for kids who have problems at home.

When I was at school the culture in the country and the school was homophobic. One teacher when discussing issues such as how students could tell him if they had psychological problems and no-one else to talk to said some things like “the Village People make really good music” which was the only time any teacher said anything like “It’s OK to be gay” (the Village People were the gayest pop group at the time). A lot of the bullying at school had a sexual component to it. In addition to the wedgies and dakking (which while not happening often was something you had to constantly be aware of) I routinely avoided PE classes where a shower was necessary because of a thug who hung around by the showers and looked hungrily at my penis, I don’t know if he had a particular liking to mine or if he stared at everyone that way. Flashing and perving was quite common in change rooms. Presumably as such boy-boy sexual misbehaviour was so accepted that led to boys mistreating girls.

I currently work for a company that is active in telling it’s employees about the possibility of free psychological assistance. Any employee can phone a psychologist to discuss problems (whether or not they are work related) free of charge and without their manager or colleagues knowing. The company is billed and is only given a breakdown of the number of people who used the service and roughly what the issue was (work stress, family, friends, grief, etc). When something noteworthy happens employees are given reminders about this such as “if you need help after seeing a homeless man try to steal a laptop from the office then feel free to call the assistance program”. Do schools offer something similar? With the school fees paid to a school like YV they should be able to afford plenty of psychologist time. Every day I was at YV I saw something considerably worse than laptop theft, most days something was done to me.

The problems with schools are part of larger problems with society. About half of the adults in Australia still support the Liberal party in spite of their support of Christian Porter, Cardinal Pell, and Bruce Lehrmann. It’s not logical to expect such parents to discourage their sons from mistreating girls or to encourage their daughters to complain when they are mistreated. The Anglican church has recently changed it’s policy to suggesting that victims of sexual abuse can contact the police instead of or in addition to the church, previously they had encouraged victims to only contact the church which facilitated cover-ups. One would hope that schools associated with the Anglican church have also changed their practices towards such things.

I approve of the “respect is in our DNA” concept, it’s like Google’s former slogan of “Don’t be evil” which is something that they can be bound to.

Here’s a list of questions that could be asked of schools (not just YV but all schools) by journalists when reporting on such things:

  1. Do you have a policy of not trying to silence past students who have been treated badly?
  2. Do you take all sexual assaults seriously including wedgies and dakking?
  3. Do you take all violence at school seriously? Even if there’s no blood? Even if the victim says they don’t want to make an issue of it?
  4. What are your procedures to deal with misbehaviour from teachers? Do the students all know how to file complaints? Do they know that they can file a complaint if they aren’t the victim?
  5. Does the school have policies against homophobia and transphobia and are they enforced?
  6. Does the school offer free psychological assistance to students and staff who need it? NB This only applies to private schools like YV that have huge amounts of money, public schools can’t afford that.
  7. Are serious incidents investigated by people who are independent of the school and who don’t have a vested interest in keeping things quiet?
  8. Do you encourage students to seek external help from organisations like the ones on the resources list of the Grace Tame Foundation [5]? Having your own list of recommended external organisations would be good too.

Counter Arguments

I’ve had practice debating such things, here’s some responses to common counter arguments.

  • Teachers are nice people how dare you criticise them. Teachers like any other large group of people includes good and bad people. The issue is how well the good people are supported in doing good things, how much effort is spent on tracking down and removing the bad people, and how much effort is spent training people to be the best version of themselves. Also my father worked as a teacher so I really don’t think that all teachers are bad.
  • Teachers are overworked and underpaid and you shouldn’t criticise them. When a school has 25 students in a class whose parents each pay $30,000 per annum the school can afford to pay as much as is necessary. Arguments about teachers being overworked and underpaid are a criticism of the organisation of private school and of government priorities for public schools not a counter argument to criticisms of the way schools operate.
  • When I went to school no bad things happened. Did you go to YV? If not then your experience isn’t relevant to this post.
  • I was a prefect at YV and didn’t see any bad things, if you saw bad things you should have reported it to me. I was not aware of any prefect who had a history of opposing bullying in previous years, I can think of some who had a history of encouraging it. Prefects were selected on the basis of supporting the system so anyone who would be expected to try to change things would have been rejected.
  • Children will make false and frivolous claims so we should ignore most of what they say, therefore complaints should only come from parents. Children have considerably less ability to lie than adults and the senior teachers are much better at detecting lies than most people. Sorting out accurate claims from false ones shouldn’t be difficult but if you reject all criticism as false claims then you will definitely miss reports of bad things and allow problems to continue.
  • I had a hard time at school and I turned out fine. If having bad things done to you doesn’t make you want to protect others from the same things then you didn’t turn out fine at all.
  • Kids need to toughen up to survive the real world. The “real world” that I live in doesn’t involve much violence at all, even having someone raise their voice at work is uncommon. Of the situations where being “tough” due to my experience at YV has been useful almost all of them involve me choosing to help someone I don’t know in a dangerous situation while other men pretend that they didn’t even notice it. The real solution is to create a world with less violence and a large part of that involves improving schools.

Conclusion

I don’t think that YV is necessarily worse than other schools, although I’m sure that representatives of other private schools are now working to assure parents of students and prospective students that they are.

I don’t think that all the people who were employed as teachers there when I attended were bad people, some of them were nice people who were competent teachers. But a few good people can’t turn around a bad system. I will note that when I attended all the sports teachers were decent people, it was the only department I could say such things about. But sports involves situations that can lead to a bad result, issues started at other times and places can lead to violence or harassment in PE classes regardless of how good the teachers are.

Teachers who know that there are problems need to be able to raise issues with the administration. When a teacher quits teaching to join the clergy and another teacher describes it as “a loss for the clergy but a gain for YV” it raises the question of why the bad teacher in question couldn’t have been encouraged to leave earlier.

A significant portion of the population will do whatever is permitted. If you say “no teacher would ever bully a student so we don’t need to look out for that” then some teacher will do exactly that.

I hope that this will lead to changes both in YV and in other schools. But if they declare this issue as resolved after expelling 4 students then something similar or worse will happen again. At least now students know that when this sort of thing happens they can send evidence to journalists to get some action.

20 May, 2024 10:27AM by etbe

hackergotchi for Daniel Pocock

Daniel Pocock

Fact check: relation to Julian Assange, founded Wikileaks at University of Melbourne and Arjen Kamphuis

In a previous blog about my campaign for election to the European parliament, I tried to fact check the discussions about the possibility that I might be related to Senator David Pocock, the former rugby star.

Another popular theme with online mobs around the open source software community is the possibility of a connection with Julian Assange and Wikileaks.

The reason this has come up is that Julian and I were both in Melbourne at around the same time. Or were we?

Given the proximity, I've always been very conscious of what happened to Julian and the related themes about freedom of expression and privacy for government employees. Ireland was recently ranked in the top ten countries for press freedom. If people feel this freedom is important then we can't take it for granted. We need to be constantly vigilant about people who spread lies and censor the truth.

Various companies have spent over $120,000 trying to censor my blogs about the Debian suicide cluster. In exchange, they obtained an order to censor some web sites. The WIPO lawyers found "critical commentary" on my web sites and they developed a rash at the thought of somebody putting the truth into writing. The WIPO censorship order was dated 3 May, that is World Press Freedom day.

On 7 December 2010, Melbourne newspaper The Age published a report with the title World leaders would love the key to this Melbourne PO box... but WikiLeaks won't have it for much longer. They include a photo of post office box 4080 at the University of Melbourne. The article states:

Australia Post insists its sudden decision to close the University of Melbourne post office has nothing to do with the fact that box 4080 is the Australian address for submissions to the whistleblower website.

In other words, the university has averted the risk of a US drone stroke by agreeing to voluntarily demolish the post office. Oddly enough, I can neither confirm nor deny whether the US drones are controlled by software developed by researchers at the University of Melbourne.

The demolition was captured on video, a lot like the execution of the Reuters journalists.

Fact checking, we can see that my first business operated from PO Box 4313. That is about a meter away from the Wikileaks post office box. Here is the letterhead from an old invoice:

Skysoft, letterhead

The Wikipedia page about University of Melbourne tells us there are over 50,000 students. This is a large institution and it is big enough that not everybody will meet everybody else.

University of Melbourne provides a service where people can verify if somebody really graduated. This is the link to the service. Putting in my name and the year 2002, anybody can confirm that I graduated in 2002.

The Wikipedia page for Julian Assange tells us he enrolled in 2003.

Where was I in 2003 when Julian arrived at Melbourne? I have the best alibi. I was in London.

In 2007, I was employed by Reuters in Paris, France at the same time that unarmed Reuters journalists in Iraq were shot by a US helicopter. These extrajudicial killings were captured on video. The leak of the video, dubbed Collateral murder, was the point when Wikileaks became a household name, especially in the White House.

Daniel Pocock, Trafalgar Square

Arjen Kamphuis collaboration with the open source communities

Arjen Kamphuis is notable for his collaboration with the UK Centre for Investigative Journalism and his work on the book Information Security for Journalists and for his subsequent disappearance. It is not clear if Arjen was very close to Wikileaks or just one of many people with a similar interest in the security of journalists.

Much later, in 2017, I traveled to Tirana, Albania for the OSCAL conference. I made substantial discoveries about poor behaviour by certain people that looks a lot like low-level people trafficking.

Coincidentally, Arjen Kamphuis was invited to the same conference. Arjen and I met for the first time when looking for the speakers dinner.

Arjen and I collaborated to provide workshops for people visiting the Tirana cryptoparty later in 2017.

Looking at the photos from these events, it is clear that there are representatives from a number of well known open source organizations and the companies behind them, including Red Hat, Mozilla and Wikipedia. We were all present at the same events with Arjen. It is not actually clear if Arjen was connected to Wikileaks or if that was just a rumor too.

Shortly before he disappeared, Arjen sent an email confirming that the source of the harassment in open source communities is Redon Skikuli, one of the founders of the Tirana hackerspace.

Subject: Re: Outreachy, Open Labs follow-up
Date: Mon, 23 Oct 2017 23:28:21 +0200
From: Arjen Kamphuis <arjen@pgknowledge.eu>
Organization: Pretty Good Knowledge
To: Daniel Pocock <daniel@pocock.pro>

Hi Daniel,

Signed up two mentors for Silva. Did not receive detailed info from
others to persue mentor-selection.

Been traveling like crazy (conference season) so did not have time to
look into other issues. Also don't want to get caught up in internal
politics but have noticed some strange events around the Dutch visit
I've organised and co-funded for next month. Will have discusson about
that with Openlabs and especially Rendon who seems to put himself in
positions of control in cases he should not.

Best,
Arjen

Many people have been spreading rumors about the scandals. The women sent multiple emails confirming that I was a witness and thanking me for my assistance.

Here is a group photo from the cryptoparty. Arjen Kamphuis is in the middle.

Daniel Pocock, Arjen Kamphuis

20 May, 2024 09:00AM

Peter Eckersley, Laura Smyth & the rushed closure of dial-up Internet in Australian universities

On 27 October 1998, one of Melbourne's leading newspapers featured a report about the demise of dial-up Internet for Australian university students.

Today, nobody misses the snail-like performance of their dial-up connection. Nonetheless, the events of that era are fascinating for a number of reasons.

The newspaper has chosen to publish a photo of Danny Michell, Laura Smyth and I together.

Peter Eckersley is quoted in the same article. Peter's death was a great tragedy for our industry and for humankind.

Fergus Vial and Michael Crozier are quoted in the article too.

At the time this article was printed, DSL broadband internet was not widely available in Australia. The dial-up services from the universities offered untimed Internet access for students in computer science, software engineering and some related departments like physics. The commercial ISPs were offering a timed dial-up service. The change from an untimed service to a time-limited service required students to think more carefully about planning their online time.

Within two or three years, the DSL broadband services were widely available. While we had to pay for this, DSL offers an untimed, always-on experience.

Therefore, a real concern that is not emphasized in this article is the poor manner in which the transition period was handled. The universities were eliminating their dial-in services a couple of years before any serious untimed DSL service became available. Another interesting piece of trivia is that Australia has untimed local telephone calls. Therefore, students would simply pay a flat-fee of twenty-five cents to dial the university and then stay online as long as possible.

The other key thing to note from this article is that it discusses the concerns about misuse or abuse of services provided to students. This discussion was taking place in the press at exactly the same time that Shaya Potter of Debian was using dial-up servers and Debian-owned servers for the purposes of download and sharing copies of computer games that had been obtained without respect for their software licenses.

Danny Michell, Laura Smyth, Daniel Pocock

Students count the cost of Net reforms

The Age, Tuesday 27 October 1998

At Monash University, free dial-in access to the Web was withdrawn this year and replaced with a choice of commercial providers, including OzEmail.

The preferred providers are usually the only way to get access to the university networks from outside but Monash uses a password system.

Dial-in access to Monash's internal network is free.

OzEmail now describes the student market as "potentially huge", with half a million cut-price customers.

Although after one year it has only 3000 customers at seven universities, including UTS in Sydney, the ISP expects a boom next year as more universities move to commercial provision for student access.

Why, just as the Web takes off, are these new restrictions coming in?

One reason is money. Megabytes cost, and extra students online multiplied by the increased size of files being sought has sent costs through the roof.

At Melbourne University alone, the cost of Internet traffic jumped from about $170,000 in 1996 to $530,000 in 1997.

This year, the forecast is for more than $900,000.

At Monash, the cost is about $1 million a year.

Universities link into the Internet via AARNET (the Australian Academic and Research Network), an asynchronous transfer mode-based national network that also connects CSIRO and three other research organizations. AARNET's traffic rates are also sky-rocketing.

Web forms are already in use in the engineering faculty, where some students have had accounts suspending and had to provide written explanations before they regain Internet access.

The University of Melbourne technology action group spokesman, Daniel Pocock, said that collectively, the moves were symptomatic of changes in academia generally.

Where universities once existed as key community contributors, he said students now feared that their main aims were financial.

Abuses occured, but generally students not abuse their limited resources, he said.

In an article to be published in the student newspaper Farrago this week, two student representatives on university technology policy committees, Peter Eckersley and Michael Crozier, attack the planned changes.

They say the university is "going overboard" in spending on things such as multimedia projects while student IT needs are not adequately supported.

Vice-chancellor in charge of IT at Melbourne, Iain Morrison said that equity was always a consideration in IT decisions, and that students would always have "at least one access opportunity".

At Monash, students hope a current review of Internet access will at least reverse the trend that started when they were funnelled off to commercial ISPs at the start of this year.

Student welfare officer, Fergus Vial, who works at Monash's student union, said students formed action groups and held meetings at the start of this year when they discovered dial-in access had been withdrawn, but "at the end of the day, people felt that the battle had already been lost".

"In a university environment, students are basically powerless."

Dial-in access is a privilege that some students have never had, but those who did resent having to switch to commercial ISPs, again citing equity issues. AARNET general manager George McLaughlin said that when AARNET left Telstra and took a feed from Optus in 1997, part of the agreement was that universities would not act commercially.

He did not believe that had caused the trend to commercial provision.

But the National Union of Students said what the big institutions were doing to save money and restrict use, smaller and regional ones would soon copy.

NUS welfare officer for Victoria, Danny Michell, called it a "dangerous trend", and said it would be an important subject at this year's NUS national conference.

For companies such as OzEmail and Connect.com, students paying for Web access represent a huge commercial opportunity - even at OzEmail's cheapest student rate of 43 cents an hour.

But for students, it is "taxing people for using a library", Michell says.

20 May, 2024 02:30AM

May 18, 2024

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

Perfy perf

I don't like slow software. So I use profilers to make software faster. What I like even less, is slow profilers! And perf is sometimes slow for completely unavoidable reasons; to resolve source line information (needed primarily for figuring out inlining, at least in the default settings), you need to go ask libbfd. But libbfd comes from binutils, and binutils is GPLv3. And perf is part of the Linux kernel, which famously is GPLv2. So if you build perf against libbfd, the result is… nondistributable. Distros cannot ship them. Not Spiderman pointing at Spiderman, but Stallman pointing at Stallman. perf has to resort to calling out to addr2line over a pipe, which sometimes works well and sometimes… well, not. A couple of years ago, I suggested an improvement here that got me a small amount of attention, but it still isn't a really reliable way to do things.

But over the last 20 years, some other group has been busy making compilers and linkers and disassemblers and low-level binary stuff. And they were pretty careful to make their stuff GPLv2-compatible. So I give you… perf using libllvm for source line lookup (and disassembling).

Hoping for a constructive review process and that I can reach the 6.11 merge window :-)

18 May, 2024 09:00PM

hackergotchi for Sean Whitton

Sean Whitton

dissertmake

Ian suggested I share the highly involved build process for my doctoral dissertation, which I submitted for examination earlier this year. Beyond just compiling a PDF from Markdown and LaTeX sources, there are just two, simple-seeming goals: produce a PDF that passes PDF/A validation, for long term archival, and replace the second page with a scanned copy of the page after it was signed by the examiners. Achieving these two things reproducibly turned out to require a lot of complexity.

First we build dissertation1.tex out of a number of LaTeX and Markdown files, and a Pandoc metadata.yml, using Pandoc in a Debian sid chroot. I had to do the latter because I needed a more recent Pandoc than was available in Debian stable at the time, and didn’t dare upgrade anything else. Indeed, after switching to the newer Pandoc, I carefully diff’d dissertation1.tex to ensure nothing other than what I needed had changed.

dissertation1.tex: preamble.tex \
 citeproc-preamble.tex \
 committee.tex \
 acknowledgements.tex \
 dedication.tex \
 contents.tex \
 abbreviations.tex \
 abstract.tex \
 metadata.yaml \
 template.latex \
 philos.csl \
 philos.bib \
 ch1.md ch1_appA.md ch2.md ch3.md ch3_appB.md ch4.md ch5.md
    schroot -c melete-sid -- pandoc -s -N -C -H preamble.tex \
        --template=template.latex -B committee.tex \
        -B acknowledgements.tex -B dedication.tex \
        -B contents.tex -B abbreviations.tex -B abstract.tex \
        ch1.md ch1_appA.md ch2.md ch3.md ch3_appB.md ch4.md ch5.md \
        citeproc-preamble.tex metadata.yaml -o $@

With hindsight, I think that I should have eschewed Pandoc in favour of plain LaTeX for a project as large as this was. Pandoc is good for journal submissions, where one is responsible for the content but not really the presentation. However, one typesets one’s own dissertation, without anyone else’s help. I decided to commit dissertation1.tex to git, because Pandoc’s LaTeX generation is not too stable.

We then compile a first PDF. My Makefile comments say that pdfx.sty requires this particular xelatex invocation. pdfx.sty is supposed to make the PDF satisfy the PDF/A-2B long term archival standard … but dissertation1.pdf doesn’t actually pass PDF/A validation. We instead rely on GhostScript to produce a valid PDF/A-2B, at the final step. But we have to include pdfx.sty at this stage to ensure that the hyperlinks in the PDF are PDF/A-compatible – without pdfx.sty, GhostScript rejects hyperref’s links.

dissertation1.pdf: \
 dissertation1.tex dissertation1.xmpdata committee_watermark.png
    xelatex -shell-escape -output-driver="xdvipdfmx -z 0" $<
    xelatex -shell-escape -output-driver="xdvipdfmx -z 0" $<
    xelatex -shell-escape -output-driver="xdvipdfmx -z 0" $<

As I said, the second page of the PDF needs to be replaced with a scanned version of the page after it was signed by the examiners. The usual tool to stitch PDFs together is pdftk. But pdftk loses the PDF’s metadata. For the true, static metadata like the title, author and keywords, it would be no problem to add them back. But the metadata that’s lost includes the PDF’s table of contents, which PDF readers display in a sidebar, with clickable links to chapters, and the sections within those. This information is not static because each time any of the source Markdown and LaTeX files change, there is the potential for the table of contents to change. So we have to extract all the metadata from dissertation1.pdf and save it to one side, before we stitch in the scanned page. We also have to hack the metadata to ensure that the second page will have the correct orientation.

SED = /^PageMediaNumber: 2$$/ { n; s/0/90/; n; s/612 792/792 612/ }
KEYWORDS = virtue ethics, virtue, happiness, eudaimonism, good lives, final ends
dissertation1_meta.txt: dissertation1.pdf
    printf "InfoBegin\nInfoKey: Keywords\nInfoValue: %s\n%s\n" \
        "${KEYWORDS}" "$$(pdftk $^ dump_data)" \
        | sed "${SED}" >$@

Now we can stitch in the signed page, and then put the metadata back. You can’t do this in one invocation of pdftk, so far as I could see.

dissertation1_stitched_updated.pdf: \
 dissertation1_stitched.pdf dissertation1_meta.txt
    pdftk dissertation1_stitched.pdf \
        update_info dissertation1_meta.txt output $@

dissertation1_stitched.pdf: dissertation1.pdf
    pdftk A=$^ \
        B=$$HOME/annex/philos/Dissertation/committee_signed.pdf \
        cat A1 B1 A3-end output $@

Finally, we use GhostScript to reprocess the PDF into two valid PDF/A-2Bs, one optimised for the web. This requires supplying a colour profile, a PDFA_def.ps postscript file, a whole sequence of GhostScript options, and some raw postscript on the command line, which gives the PDF reader some display hints.

GS_OPTS1 = -sDEVICE=pdfwrite -dBATCH -dNOPAUSE -dNOSAFER \
    -sColorConversionStrategy=UseDeviceIndependentColor \
    -dEmbedAllFonts=true -dPrinted=false -dPDFA=2 \
    -dPDFACompatibilityPolicy=1 -dDetectDuplicateImages \
    -dPDFSETTINGS=/printer -sOutputFile=$@
GS_OPTS2 = PDFA_def.ps dissertation1_stitched_updated.pdf \
    -c "[ /PageMode /UseOutlines \
        /Page 1 /View [/XYZ null null 1] \
        /PageLayout /SinglePage /DOCVIEW pdfmark"

all: Whitton_dissert_web.pdf Whitton_dissert_gradcol.pdf
Whitton_dissert_gradcol.pdf: \
 PDFA_def.ps dissertation1_stitched_updated.pdf srgb.icc
    gs ${GS_OPTS1} ${GS_OPTS2}
Whitton_dissert_web.pdf: \
 PDFA_def.ps dissertation1_stitched_updated.pdf srgb.icc
    gs ${GS_OPTS1} -dFastWebView=true ${GS_OPTS2}

And here’s PDFA_def.ps, based on a sample in the GhostScript docs:

% Define an ICC profile :
/ICCProfile (srgb.icc)
def

[/_objdef {icc_PDFA} /type /stream /OBJ pdfmark

[{icc_PDFA}
<<
/N 3
>> /PUT pdfmark
[{icc_PDFA} ICCProfile (r) file /PUT pdfmark

% Define the output intent dictionary :
[/_objdef {OutputIntent_PDFA} /type /dict /OBJ pdfmark
[{OutputIntent_PDFA} <<
  /Type /OutputIntent               % Must be so (the standard requires).
  /S /GTS_PDFA1                     % Must be so (the standard requires).
  /DestOutputProfile {icc_PDFA}     % Must be so (see above).
  /OutputConditionIdentifier (sRGB)
>> /PUT pdfmark
[{Catalog} <</OutputIntents [ {OutputIntent_PDFA} ]>> /PUT pdfmark

Phew!

Some comments on Hacker News

18 May, 2024 08:28AM

Russell Coker

Kogan 5120*2160 40″ Monitor

I’ve just got a new Kogan 5120*2160 40″ curved monitor. It cost $599 including shipping etc which is much cheaper than the Dell monitor with similar specs selling for about $2500. For monitors with better than 4K resolution (by which I don’t mean 5K*1440) this is the cheapest option. The nearest competitors are the 27″ monitors that do 5120*2880 from Apple and some companies copying Apple’s specs. While 5120*2880 is a significantly better resolution than what I got it’s probably not going to help me at 27″ size.

I’ve had a Dell 32″ 4K monitor since the 1st of July 2022 [1]. It is a really good monitor and I had no complaints at all about it. It was clearer than the Samsung 27″ 4K monitor I used before it and I’m not sure how much of that is due to better display technology (the Samsung was from 2017) and how much was due to larger size. But larger size was definitely a significant factor.

I briefly owned a Phillips 43″ 4K monitor [2] and determined that a 43″ flat screen was definitely too big. At the time I thought that about 35″ would have been ideal but after a couple of years using a flat 32″ screen I think that 32″ is about the upper limit for a flat screen. This is the first curved monitor I’ve used but I’m already thinking that maybe 40″ is too big for a 21:9 aspect ratio even with a curved screen. Maybe if it was 4:4 or even 16:9 that would be ok. Otherwise the ideal for a curved screen for me would be something between about 36″ and 38″. Also 43″ is awkward to move around my desk. But this is still quite close to ideal.

The first system I tested this on was a work laptop, a Dell Latitude 7400 2in1. On the Dell dock that did 4K resolution and on a HDMI cable it did 1440p which was a disappointment as that laptop has talked to many 4K monitors at native resolution on the HDMI port with the same cable. This isn’t an impossible problem, as I work in the IT department I can just go through all the laptops in the store room until I find one that supports it. But the 2in1 is a very nice laptop, so I might even just keep using it in 4K resolution when WFH. The laptop in question is deemed an “executive” laptop so I have to wait another 2 years for the executives to get new laptops before I can get a newer 2in1.

On my regular desktop I had the problem of the display going off for a few seconds every minute or so and also occasionally giving a white flicker. That was using 5120*2160 with a DisplayPort switch as described in the blog post about the Dell 32″ monitor. When I ran it in 4K resolution with the DisplayPort switch from my desktop it was fine. I then used the DisplayPort cable that came with the monitor directly connecting the video card to the display and it was fine at 5120*2160 with 75Hz.

The monitor has the joystick thing that seems to have become some sort of standard for controlling modern monitors. It’s annoying that pressing it in powers it off. I think there should be a separate button for that. Also the UI in general made me wonder if one of the vendors of expensive monitors had paid whoever designed it to make the UI suck.

The monitor had a single dead pixel in the center of the screen about 1/4 the way down from the top when I started writing this post. Now it’s gone away which is a concern as I don’t know which pixels might have problems next or if the number of stuck pixels will increase. Also it would be good if there was a “dark mode” for the WordPress editor. I use dark mode wherever possible so I didn’t notice the dead pixel for several hours until I started writing this blog post.

I watched a movie on Netflix and it took the entire screen area, I don’t know if they are storing movies in 64:27 ratio or if the clipped the top and bottom, it was probably clipped but still looked OK. The monitor has different screen modes which make it look different, I can’t see much benefit to the different modes. The “standard” mode is what I usually use and it’s brighter and the “movie” mode seems OK for the one movie I’ve watched so far.

In other news BenQ has just announced a 3840*2560 28″ monitor specifically designed for programming [3]. This is the first time I’ve heard of a monitor with 3:2 ratio with modern resolution, we still aren’t at the 4:3 type ratio that we were used to when 640*480 was high resolution but it’s a definite step in the right direction. It’s also the only time I recall ever seeing a monitor advertised as being designed for programming. In the 80s there were home computers advertised as being computers for kids to program, but at that time it was either TV sets for monitors or monitors sold with computers. It was only after the IBM PC compatible market took off that having a choice of different monitors for one computer was a thing. In recent years monitors advertised as being for office use (meaning bright and expensive) have become common as are monitors designed for gamer use (meaning high refresh rate). But BenQ seems to be the first to advertise a monitor for the purpose of programming. They have a “desktop partition” feature (which could be software or hardware – the article doesn’t make it clear) to give some of the benefits of a tiled window manager to people who use OSs that don’t support such things. The BenQ monitor is a bit small for my taste, I don’t know if my vision is good enough to take advantage of 3840*2560 in a 28″ monitor nowadays. I think at least 32″ would be better. Google seems to be really into buying good monitors for their programmers, if every Google programmer got one of those BenQ monitors then that would be enough sales to make it worth-while for them.

I had hoped that we would have 6K monitors become affordable this year and 8K become less expensive than most cars. Maybe that won’t happen and we will instead have a wider range of products like the ultra wide monitor I just bought and the BenQ programmer’s monitor. If so I don’t think that will be a bad result.

Now the question is whether I can use this monitor for 2 years before finding something else that makes me want to upgrade. I can afford to spend the equivalent of a bit under $1/day on monitor upgrades.

18 May, 2024 04:24AM by etbe

James Morrison

Goodbye Firefox

 I've been on Chromebooks for a while.  However, since I had to recently try a Mac, I figured it was time to give Firefox a try again.  After two weeks of trying, I've given up.  At least for myself, I figured I'd write down the reasons I've given up.

Reasons:

  • There is no way from the tab context menu to move a tab between windows.  I typically try to keep no more than 3 windows open at a time.  Ideally one, but maybe a second.  Without the ability to through the context menu to move a tab, I need a very large screen (not a laptop screen) to move tabs between windows.
  • I couldn't find a way to take a URL and turn it into a custom search.  This really is a critical feature as it allows me to use short names to access specific searches.  E.g. search code (cs), show a calendar (c), etc.

18 May, 2024 12:09AM by Jim (noreply@blogger.com)

May 16, 2024

John Goerzen

Review of Reputable, Functional, and Secure Email Service

I last reviewed email services in 2019. That review focused a lot of attention on privacy. At the time, I selected mailbox.org as my provider, and have been using them for these 5 years since. However, both their service and their support have gone significantly downhill since, so it is time for me to look at other options.

Here I am focusing strongly on email. Some of the providers mentioned here provide other services (IM, video calls, groupware, etc.), and to the extent they do, I am ignoring them.

What Matters in 2024

I want to start off by acknowledging that what you need in email probably depends on your circumstances and the country in which you live. For me, I begin by naming that the largest threat most of us face isn’t from state actors but from criminals: hackers, ransomware gangs, etc. It is important to take as many steps as possible to secure one’s account against that. Privacy and security are both part of the mix. I still value privacy but I am acknowledging, as Migadu does, that “Email as we know it and encryption are incompatible.” Although some of these services strongly protect parts of the conversation, the reality is that most people will be emailing people using plain old email services which don’t. For stronger security, something like Signal would be needed. (I wrote about Signal in 2021 also.)

Interestingly, OpenPGP support seems to be something of a standard feature in the providers I reviewed by this point. All or almost all of them provide integration with browser-based encryption as well as server-side encryption if you prefer that.

Although mailbox.org can automatically PGP-encrypt every message that arrives in plaintext, for general use, this is unwieldy; there isn’t good tooling for searching mailboxes where every message is encrypted, etc. So I never enabled that feature at Mailbox. I still value security and privacy, but a pragmatic approach addresses the most pressing threats first.

My criteria

The basic requirements for an email service include:

  1. Ability to use my own domains
  2. Strong privacy policy
  3. Ability for me to use my own IMAP and SMTP clients on both desktop and mobile
  4. It must be extremely reliable
  5. It must not be free
  6. It must have excellent support for those rare occasions when it is needed
  7. Support for basic aliases

Why do I say it must not be free? Because if someone is providing a service with the quality I’m talking about here, and not charging for it, it implies something is fishy: either they are unscrupulous, are financially unstable, or the product is something else like ads. I am not aware of any provider that matches the other criteria with a free account anyhow. These providers range from about $30 to $90 per year, so cheaper than a Netflix subscription.

Immediately, this rules out several options:

  • Proton doesn’t let me use my own clients on mobile (their bridge is desktop-only)
  • Tuta also doesn’t let me use my own clients
  • Posteo doesn’t let me use my own domain
  • mxroute.com lacks a strong privacy policy, and its policy has numerous causes for concern (for instance, “If you repeatedly send email to invalid/unroutable recipients, they may be published on our GitHub”)

I will have a bit more to say about a couple of these providers below.

There are some additional criteria that are strongly desired but not absolutely required:

  1. Ability to set individual access passwords for every device/app
  2. Support for two-factor authentication (2FA/TFA/TOTP) for web-based access
  3. Support for basics in filtering: ability to filter on envelope recipient (so if I get BCC’d, I can still filter), and ability to execute more than one action on filter match (eg, deliver to two folders, or deliver to a folder and forward to someone else)

IMAP and SMTP don’t really support 2FA, so by setting individual passwords for every device, you can at least limit the blast radius and cut off a specific device if something is (or might be) compromised.

The candidates

I considered these providers: Startmail, Mailfence, Runbox, Fastmail, Kolab, Mailbox.org, and Migadu. I’ll review each, and highlight the pricing of the plan I would most likely use. Each provider offers multiple plans; some may be more expensive and some may be cheaper than the one I reviewed. I included a link to each provider’s full pricing information so you can compare for your needs.

I set up trials with each of these (except Mailbox.org, with which I already had a paid account). It so happend that I had actual questions for support for each one, which gave me an opportunity to see how support responded. I did not fabricate questions, and would not have contacted support if I didn’t have real ones. (This means that I asked different questions of each provider, because they were the REAL questions I had.) I’ll jump to the spoiler right now: I eventually chose Migadu, with Fastmail and Mailfence as close seconds.

I looked for providers myself, and also solicited recommendations in a Mastodon thread.

Mailbox.org

I begin with Mailbox, as it was my top choice in 2019 and the incumbent.

Until this year, I had been quite happy with it. I had cause to reach their support less than once a year on average, and each time they replied the same day or next day. Now, however, they are failing on reliability and on support.

Their spam filter has become overly aggressive. It has blocked quite a bit of legitimate mail. When contacting their support about a prior issue earlier this year, they initially took 4 days to reply, and then 6 days to reply after that. Ouch. They had me disable some spam settings.

It didn’t really help. I continue to lose mail. I don’t know how much, because they block a lot of it before it even hits the spam folder. One of my friends texted to say mail was dropping. I raised a new ticket with mailbox, which took them 5 days to reply to. Their reply was unhelpful. “As the Internet is not a static system, unforeseen events can always occur.” Well yes, that’s true, and I get it, false positives exist with email. But this was from an ISP’s mail system with an address that had been established for years, and it was part of a larger pattern of rejecting quite a bit of legit mail. And every interaction with them recently hasn’t resulted in them actually doing anything to resolve anything. It’s just a paragraph or two of reply that does nothing and helps nothing.

When I complained that it took 5 days to reply, they said “We have not been able to reply sooner as we are currently experiencing a high volume of customer enquiries.” Even though their SLA for my account is a not-great “48 business hour” turnaround, they still missed it and their reason is “we’re busy.” I finally asked what RBL had caught the blocked email, since when I checked, the sender wasn’t on any RBL. Mailbox’s reply: they only keep their logs for 7 days, so next time I should contact them within 7 days. Which, of course, I DID; it was them that kept delaying. Ugh! It’s like they’ve become a cable company.

Even worse is how they have been blocking mail from GrapheneOS’s discussion form. See their thread about it. In short, Graphene’s mail server has a clean reputation and Mailbox has no problem with it. But because one of Graphene’s IPv6 webservers has an IPv6 allocation of a size Mailbox doesn’t like, they drop mail. It’s ridiculous, and Mailbox was dismissive of this well-known and well-regarded Open Source project. So if the likes of GrapheneOS can’t get good faith effort to deliver their mail, what chance does an individual like me have?

I’m sorry, but I’m literally paying you to deliver email for me and provide good support. If you can’t do either of those, you don’t get to push that problem down onto me. Hire appropriate staff.

On the technical side, they support aliases, my own clients, and have a reasonable privacy policy. Their 2FA support exists for the web interface (though weirdly not the support site), though it is somewhat weird. They do not support app passwords.

A somewhat unique feature is the @secure.mailbox.org domain. If you try to receive mail at that address, mailbox.org will block it unless it uses TLS. Same for sending. This isn’t E2EE, but it does at least require things not be in plaintext for the last hop to Mailbox.

Verdict: not recommended due to poor reliability and support.

Mailbox.Org summary:

  • Website: https://mailbox.org/en/
  • Reliability: iffy due to over-aggressive spam filtering
  • Support: Poor; takes 4-6 days for a reply and replies are unhelpful
  • Individual access passwords: No
  • 2FA: Yes, but with a PIN instead of a password as the other factor
  • Filtering: Full SIEVE feature set and GUI editor
  • Spam settings: greylisting on/off, reject some/all spam, etc. But they’re insufficient to address Mailbox’s overzealousness, which support says I cannot workaround within the interface.
  • Server storage location: Germany
  • Plan as reviewed: standard [pricing link]
    • Cost per year: EUR 30 (about $33)
    • Mail storage included: 10GB
    • Limits on send/receive volume: none
    • Aliases: 50 on your domain name, 25 on mailbox.org
    • Additional mailboxes: Available; each one at the same fee as the primary mailbox

Startmail

I really wanted to like Startmail. Its “vault” is an interesting idea and should contribute to the security and privacy of an account. They clearly care about privacy.

It falls down in filtering. They have no way to filter on envelope recipient (BCC or similar). Their support confirmed this to me and that’s a showstopper.

Startmail support was also as slow as Mailbox, taking 5 days to respond to me.

Two showstoppers right there.

Verdict: Not recommended due to slow support responsiveness and weak filtering.

Startmail summary:

  • Website: https://www.startmail.com/
  • Reliability: Seems to be fine
  • Support: Mediocre; Took 5 days for a reply, but the reply was helpful
  • Individual app access passwords: Yes
  • 2FA: Yes
  • Filtering: Poor; cannot filter on envelope recipient, and can’t build filters with multiple actions
  • Spam settings: None
  • Server storage location: The Netherlands
  • Plan as reviewed: Custom domain (trial was Personal), [pricing link]
    • Cost per year: $70
    • Mail storage included: 20GB
    • Limits on send/receive volume: none
    • Aliases: unlimited, with lots of features: can set expiration, etc.
    • Additional mailboxes: not available

Kolab

Kolab Now is mainly positioned as a full groupware service, but they do have a email-only option which I investigated. There isn’t much documentation about it compared to other providers, and also not much in the way of settings. You can turn greylisting on or off. And…. that’s it.

It has a full suite of filtering options. They set an X-Envelope-To header which you can use with the arbitrary header match to do the right thing even for BCC situations. Filters can have multiple conditions and multiple actions. It is SIEVE-based and you can download your SIEVE definitions.

If you enable 2FA, you disable IMAP and SMTP; not great.

Verdict: Not an impressive enough email featureset to justify going with it.

Kolab Now summary:

  • Website: https://kolabnow.com/
  • Reliability: Seems to be fine
  • Support: Fine responsiveness (next day)
  • Invidiaul app passwords: no
  • 2FA: Yes, but if you enable it, they disable IMAP and SMTP
  • Filtering: Excellent
  • Spam settings: Only greylisting on/off
  • Server storage location: Switzerland; they have lots of details on their setup
  • Plan as reviewed: “Just email” [pricing link]
    • Cost per year: CHF 60, about $66
    • Mail storage included: 5GB
    • Limitations on send/receive volume: None
    • Aliases: Yes. Not sure if there are limits.
    • Additional mailboxes: Yes if you set up a group account. “Flexible pricing based on user count” is not documented anywhere I could find.

Mailfence

Mailfence is another option, somewhat similar to Startmail but without the unique vault. I had some questions about filters, and support was quite responsive, responding in a couple of hours.

Some of their copy on their website is a bit misleading, but support clarified when I asked them. They do not offer encryption at rest (like most of the entries here).

Mailfence’s filtering system is the kind I’d like to see. It allows multiple conditions and multiple actions for each rule, and has some unique actions as well (notify by SMS or XMPP). Support says that “Recipients” matches envelope recipients. However, one ommission is that I can’t match on arbitrary headers; only the canned list of headers they provide.

They have only two spam settings:

  • spam filter on/off
  • whitelist

Given some recent complaints about their spam filter being overly aggressive, I find this lack of control somewhat concerning. (However, I discount complaints about people begging for more features in free accounts; free won’t provide the kind of service I’m looking for with any provider.) There are generally just very few settings for email as well.

Verdict: Response and helpful support, filtering has the right structure but lacks arbitrary header match. Could be a good option.

Mailfence summary:

  • Website: https://mailfence.com/
  • Reliability: Seems to be fine
  • Support: Excellent responsiveness and helpful replies (after some initial confusion about my question of greylisting)
  • Individual app access passwords: No. You can set a per-service password (eg, an IMAP password), but those will be shared with all devices speaking that protocol.
  • 2FA: Yes
  • Filtering: Good; only misses the ability to filter on arbitrary headers
  • Spam settings: Very few
  • Server storage location: Belgium
  • Plan as reviewed: Entry [pricing link]
    • Cost per year: $42
    • Mail storage included: 10GB, with a maximum of 50,000 messages
    • Limits on send/receive volume: none
    • Aliases: 50. Aliases can’t be deleted once created (there may be an exeption to this for aliases on your own domain rather than mailfence.com)
    • Additional mailboxes: Their page on this is a bit confusing, and the pricing page lacks the information promised. It looks like you can pay the same $42/year for additional mailboxes, with a limit of up to 2 additional paid mailboxes and 2 additional free mailboxes tied to the account.

Runbox

This one came recommended in a Mastodon thread. I had some questions about it, and support response was fantastic – I heard from two people that were co-founders of the company! Even within hours, on a weekend. Incredible! This kind of response was only surpassed by Migadu.

I initially wrote to Runbox with questions about the incoming and outgoing message limits, which I hadn’t seen elsewhere, as well as the bandwidth limit. They said the bandwidth limit is no longer enforced on paid accounts. The incoming and outgoing limits are enforced, and all email (even spam) counts towards the limit. Notably the outgoing limit is per recipient, so if you send 10 messages to your 50-recipient family group, that’s the limit. However, they also indicated a willingness to reset the limit if something happens. Unfortunately, hitting the limit results in a hard bounce (SMTP 5xx) rather than a temporary failure (SMTP 4xx) so it can result in lost mail. This means I’d be worried about some attack or other weirdness causing me to lose mail.

Their filter is a pain point. Here are the challenges:

  • You can’t directly match on a BCC recipient. Support advised to use a “headers” match, which will search for something anywhere in the headers. This works and is probably “good enough” since this data is in the Received: headers, but it is a little more imprecise.
  • They only have a “contains”, not an “equals” operator. So, for instance, a pattern searching for “test@example.com” would also match “newtest@example.com”. Support advised to put the email address in angle brackets to avoid this. That will work… mostly. Angle brackets aren’t always required in headers.
  • There is no way to have multiple actions on the filter (there is just no way to file an incoming message into two folders). This was the ultimate showstopper for me.

Support advised they are planning to upgrade the filter system in the future, but these are the limitations today.

Verdict: A good option if you don’t need much from the filtering system. Lots of privacy emphasis.

Runbox summary:

  • Website: https://runbox.com/
  • Reliability: Seems to be fine, except returning 5xx codes if per-day limits are exceeded
  • Support: Excellent responsiveness and replies from founders
  • Individual app passwords: Yes
  • 2FA: Yes
  • Filtering: Poor
  • Spam settings: Very few
  • Server storage location: Norway
  • Plan as reviewed: Mini [pricing link]
    • Cost per year: $35
    • Mail storage included: 10GB
    • Limited on send/receive volume: Receive 5000 messages/day, Send 500 recipients/day
    • Aliases: 100 on runbox.com; unlimited on your own domain
    • Additional mailboxes: $15/yr each, also with 10GB non-shared storage per mailbox

Fastmail

Fastmail came recommended to me by a friend I’ve known for decades.

Here’s the thing about Fastmail, compared to all the services listed above: It all just works. Everything. Filtering, spam prevention, it is all there, all feature-complete, and all just does the right thing as you’d hope. Their filtering system has a canned dropdown for “To/Cc/Bcc”, it supports multiple conditions and multiple actions, and just does the right thing. (Delivering to multiple folders is a little cumbersome but possible.) It has a particularly strong feature set around administering multiple accounts, including things like whether users can prevent admins from reading their mail.

The not-so-great part of the picture is around privacy. Fastmail is based in Australia, where the government has extensive power around spying on data, even to the point of forcing companies to add wiretap capabilities. Fastmail’s privacy policy states user data may be held in Australia, USA, India, and Netherlands. By default, they share data with unidentified “spam companies”, though you can disable this in settings. On the other hand, they do make a good effort towards privacy.

I contacted support with some questions and got back a helpful response in three hours. However, one of the questions was about in which countries my particular data would be stored, and the support response said they would have to get back to me on that. It’s been several days and no word back.

Verdict: A featureful option that “just works”, with a lot of features for managing family accounts and the like, but lacking in the privacy area.

Fastmail summary:

  • Website: https://www.fastmail.com/
  • Reliability: Seems to be fine
  • Support: Good response time on most questions; dropped the ball on one tha trequired research
  • Individual app access passwords: Yes
  • 2FA: Yes
  • Filtering: Excellent
  • Spam settings: Can set filter aggressiveness, decide whether to share spam data with “spam-fighting companies”, configure how to handle backscatter spam, and evaluate the personal learning filter.
  • Server storage locations: Australia, USA, India, and The Netherlands. Legal jurisdiction is Australia.
  • Plan as reviewed: Individual [pricing link]
    • Cost per year: $60
    • Mail storage included: 50GB
    • Limits on send/receive volume: 300/hour
    • Aliases: Unlimited from what I can see
    • Additional mailboxes: No; requires a different plan for that

Migadu

Migadu was a service I’d never heard of, but came recommended to me on Mastodon.

I listed Migadu last because it is a class of its own compared to all the other options. Every other service is basically a webmail interface with a few extra settings tacked on.

Migadu has a full-featured email admin console in addition. By that I mean you can:

  • View usage graphs (incoming, outgoing, storage) over time
  • Manage DNS (if you want Migadu to run your nameservers)
  • Manage multiple domains, and cross-domain relationships with mailboxes
  • View a limited set of logs
  • Configure accounts, reset their passwords if needed/authorized, etc.
  • Configure email address rewrite rules with wildcards and so forth

Basically, if you were the sort of person that ran your own mail servers back in the day, here is Migadu giving you most of that functionality. Effectively you have a web interface to do all the useful stuff, and they handle the boring and annoying bits. This is a really attractive model.

Migadu support has been fantastic. They are quick to respond, and went above and beyond. I pointed out that their X-Envelope-To header, which is needed for filtering by BCC, wasn’t being added on emails I sent myself. They replied 5 hours later indicating they had added the feature to add X-Envelope-To even for internal mails! Wow! I am impressed.

With Migadu, you buy a pool of resources: storage space and incoming/outgoing traffic. What you do within that pool is up to you. You can set up users (“mailboxes”), aliases, domains, whatever you like. It all just shares the pool. You can restrict users further so that an individual user has access to only a subset of the pool resources.

I was initially concerned about Migadu’s daily send/receive message count limits, but in visiting with support and reading the documentation, what really comes out is that Migadu is a service with a personal touch. Hitting the incoming traffic limit will cause a SMTP temporary fail (4xx) response so you won’t lose legit mail – and support will work with you if it’s a problem for legit uses. In other words, restrictions are “soft” and they are interpreted reasonably.

One interesting thing about Migadu is that they do not offer accounts under their domain. That is, you MUST bring your own domain. That’s pretty easy and cheap, of course. It also puts you in a position of power, because it is easy to migrate email from one provider to another if you own the domain.

Filtering is done via SIEVE. There is a GUI editor which lets you accomplish most things, though it has an odd blind spot where you can’t file a message into multiple folders. However, you can edit a SIEVE ruleset directly and you get the full SIEVE featureset, which is extensive (and does support filing a message into multiple folders). I note that the SIEVE :envelope match doesn’t work, but Migadu adds an X-Envelope-To header which is just as good.

I particularly love a company that tells you all the reasons you might not want to use them. Migadu’s pro/con list is an honest drawbacks list (of course, their homepage highlights all the features!).

Verdict: Fantastically powerful, excellent support, and good privacy. I chose this one.

Migadu summary:

  • Website: https://migadu.com/
  • Reliability: Excellent
  • Support: Fantastic. Good response times and they added a feature (or fixed a bug?) a few hours after I requested it.
  • Individual access passwords: Yes. Create “identities” to support them.
  • 2FA: Yes, on both the admin interface and the webmail interface
  • Filtering: Excellent, based on SIEVE. GUI editor doesn’t support multiple actions when filing into a folder, but full SIEVE functionality is exposed.
  • Spam settings:
    • On the domain level, filter aggressiveness, Greylisting on/off, black and white lists
    • On the mailbox level, filter aggressiveness, black and whitelists, action to take with spam; compatible with filters.
  • Server storage location: France; legal jurisdiction Switzerland
  • Plan as reviewed: mini [pricing link]
    • Cost per year: $90
    • Mail storage included: 30GB (“soft” quota)
    • Limits on send/receive volume: 1000 messgaes in/day, 100 messages out/day (“soft” quotas)
    • Aliases: Unlimited on an unlimited number of domains
    • Additional mailboxes: Unlimited and free; uses pooled quotas, but individual quotas can be set

Others

Here are a few others that I didn’t think worthy of getting a trial:

  • mxroute was recommended by several. Lots of concerning things in their policy, such as:
    • if you repeatedly send mail to unroutable recipients, they may publish the addresses on Github
    • they will terminate your account if they think you are “rude” or want to contest a charge
    • they reserve the right to cancel your service at any time for any (or no) reason.
  • Proton keeps coming up, and I will not consider it so long as I am locked into their client on mobile.
  • Skiff comes up sometimes, but they were acquired by Norton.
  • Disroot comes up; this discussion highlights a number of reasons why I avoid them. Their Terms of Service (ToS) is inconsistent with a general-purpose email account (I guess for targeting nonprofits and activists, that could make sense). Particularly laughable is that they claim to be friends of Open Source, but then would take down your account if you upload “copyrighted” material. News flash: in order for an Open Source license to be meaningful, the underlying work is copyrighted. It is perfectly legal to upload copyrighted material when you wrote it or have the license to do so!

Conclusions

There are a lot of good options for email hosting today, and in particular I appreciate the excellent personal support from companies like Migadu and Runbox. Support small businesses!

16 May, 2024 05:42PM by John Goerzen

May 15, 2024

hackergotchi for Evgeni Golov

Evgeni Golov

Using HPONCFG on CentOS Stream 9 with OpenSSL 3.2

Today I've updated an HPE ProLiant DL325 G10 from CentOS Stream 8 to CentOS Stream 9 (details on that to follow) and realized that hponcfg was broken afterwards.

As I do not have a support contract with HPE, I couldn't just yell at them in private, so I am doing this in public now ;-)

# hponcfg
HPE Lights-Out Online Configuration utility
Version 5.6.0 Date 11/30/2020 (c) 2005,2020 Hewlett Packard Enterprise Development LP
Error: Unable to locate SSL library.
       Install latest SSL library to use HPONCFG.

Welp, what the heck?

But wait, 5.6.0 from 2020 looks old, let's update this first!

hponcfg is part of the "Management Component Pack" (at least if you're not running RHEL or SLES where you get it via the "Service Pack for ProLiant" which requires a support contract) and can be downloaded from the Software Delivery Repository.

The Software Delivery Repository tells you to configure it in /etc/yum.repos.d/mcp.repo as

[mcp]
name=Management Component Pack
baseurl=http://downloads.linux.hpe.com/repo/mcp/dist/dist_ver/arch/project_ver
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/GPG-KEY-mcp

gpgcheck=0? Suuure! Plain HTTP? Suuure!

But it gets better! When you look at https://downloads.linux.hpe.com/repo/mcp/centos/ (you have to substitute dist with your distribution!) you'll see that there is no 9 folder and thus no packages for CentOS (Stream) 9. There are however folders for Oracle, Rocky and Alma. Phew. Let's take one of these!

[mcp]
name=Management Component Pack
baseurl=https://downloads.linux.hpe.com/repo/mcp/rocky/9/x86_64/current/
enabled=1
gpgcheck=1
gpgkey=https://downloads.linux.hpe.com/repo/mcp/GPG-KEY-mcp

dnf upgrade hponcfg updates it to hponcfg-6.0.0-0.x86_64 and:

# hponcfg
HPE Lights-Out Online Configuration utility
Version 6.0.0 Date 10/30/2022 (c) 2005,2022 Hewlett Packard Enterprise Development LP
Error: Unable to locate SSL library.
       Install latest SSL library to use HPONCFG.

Fuck.

ldd doesn't show hponcfg being linked to libssl, do they dlopen() at runtime and fucked something up? ltrace to the rescue!

# ltrace hponcfg

popen("strings /bin/openssl | grep 'Ope"..., "r")            = 0x621700
fgets("OpenSSL 3.2.1 30 Jan 2024\n", 256, 0x621700)          = 0x7ffd870e2e10
strstr("OpenSSL 3.2.1 30 Jan 2024\n", "OpenSSL 3.0")         = nil

WAT?

They run strings /bin/openssl |grep 'OpenSSL' and compare the result with "OpenSSL 3.0"?!

Sure, OpenSSL 3.2 in EL9 is rather fresh and didn't hit RHEL/Oracle/Alma/Rocky yet, but surely there are better ways to check for a compatible version of OpenSSL than THIS?!

Anyway, I am not going to downgrade my OpenSSL. Neither will I patch it to pretend to be 3.0.

But I can patch the hponcfg binary!

# vim /sbin/hponcfg
<go to line 146>
<replace 3.0 with 3.2>
:x

Yes, I used vim. Yes, it works. No, I won't guarantee this won't kill a kitten somewhere.

# ./hponcfg
HPE Lights-Out Online Configuration utility
Version 6.0.0 Date 10/30/2022 (c) 2005,2022 Hewlett Packard Enterprise Development LP
Firmware Revision = 2.44 Device type = iLO 5 Driver name = hpilo

USAGE:
  hponcfg  -?
  hponcfg  -h
  hponcfg  -m minFw
  hponcfg  -r [-m minFw] [-u username] [-p password]
  hponcfg  -b [-m minFw] [-u username] [-p password]
  hponcfg  [-a] -w filename [-m minFw] [-u username] [-p password]
  hponcfg  -g [-m minFw] [-u username] [-p password]
  hponcfg  -f filename [-l filename] [-s namevaluepair] [-v] [-m minFw] [-u username] [-p password]
  hponcfg  -i [-l filename] [-s namevaluepair] [-v] [-m minFw] [-u username] [-p password]

  -h,  --help           Display this message
  -?                    Display this message
  -r,  --reset          Reset the Management Processor to factory defaults
  -b,  --reboot         Reboot Management Processor without changing any setting
  -f,  --file           Get/Set Management Processor configuration from "filename"
  -i,  --input          Get/Set Management Processor configuration from the XML input
                        received through the standard input stream.
  -w,  --writeconfig    Write the Management Processor configuration to "filename"
  -a,  --all            Capture complete Management Processor configuration to the file.
                        This should be used along with '-w' option
  -l,  --log            Log replies to "filename"
  -v,  --xmlverbose     Display all the responses from Management Processor
  -s,  --substitute     Substitute variables present in input config file
                        with values specified in "namevaluepairs"
  -g,  --get_hostinfo   Get the Host information
  -m,  --minfwlevel     Minimum firmware level
  -u,  --username       iLO Username
  -p,  --password       iLO Password

For comparison, here is the diff --text output:

# diff -u --text /sbin/hponcfg ./hponcfg
--- /sbin/hponcfg   2022-08-02 01:07:55.000000000 +0000
+++ ./hponcfg   2024-05-15 09:06:54.373121233 +0000
@@ -143,7 +143,7 @@
 helpget_hostinforesetwriteconfigallfileinputlogminfwlevelxmlverbosesubstitutetimeoutdbgverbosityrebootusernamepasswordlibpath%Ah*Ag7Ar=AwIAaMAfRAiXAl\AmgAvrAs}At�Ad�Ab�Au�Ap�Azhgrbaw:f:il:m:vs:t:d:z:u:p:tmpXMLinputFile%2d.xmlw+Error: Syntax Error - Invalid options present.
 =O@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@aQ@�M@�M@aQ@�M@aQ@�N@�M@�N@�P@aQ@aQ@�M@�M@aQ@aQ@LN@aQ@�M@�O@�M@�M@�M@�M@aQ@aQ@�M@<!----><LOGINUSER_LOGINPASSWORD<LOGIN USER_LOGIN="%s" PASSWORD="%s"ERROR: LOGIN tag is missing.
 >ERROR: LOGIN end tag is missing.
-strings  | grep 'OpenSSL 1' | grep 'OpenSSL 3'OpenSSL 1.0OpenSSL 1.1OpenSSL 3.0which openssl 2>&1/usr/bin/opensslOpenSSL location - %s
+strings  | grep 'OpenSSL 1' | grep 'OpenSSL 3'OpenSSL 1.0OpenSSL 1.1OpenSSL 3.2which openssl 2>&1/usr/bin/opensslOpenSSL location - %s
 Current version %s

 No response from command.

Pretty sure it won't apply like this with patch, but you get the idea.

And yes, double-giggles for the fact that the error message says "Install latest SSL library to use HPONCFG" and the issues is because I have the latest SSL library installed…

15 May, 2024 09:14AM by evgeni

May 14, 2024

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RApiSerialize 0.1.3 on CRAN: Skipping XDR

A new bug fix release 0.1.3 of RApiSerialize got onto CRAN earlier today. This is the first release in well over a year, and permits the skip the XDR serialization format which is needed when transfering between big- and little-endian machines. But it comes at a certain run-time cost one can avoid on the (much more common) little-endian machines. This is a new option, and the old behavior is the default. Those who want to can now skip the step.

The RApiSerialize package is used by both my RcppRedis as well as by Travers excellent qs package. We also addressed the recent nag by the CRAN concerning ‘NO_REMAP’.

Changes in version 0.1.3 (2024-05-13)

  • Add an xdr argument to disable XDR for an approx. threefold speed increase (Travers Ching and Dirk in #6)

  • Use R_NO_REMAP and Rf_* prefix for API calls

  • Minor continuous integration updates

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More details are at the RApiSerialize page; code, issue tickets etc at the GitHub repositoryrapiserializerepo.

If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

14 May, 2024 11:28PM

hackergotchi for Evgeni Golov

Evgeni Golov

Using Packit to build RPMs for projects that depend on or vendor your code

I am a huge fan of Packit as it allows us to provide RPMs to our users and testers directly from a pull-request, thus massively tightening the feedback loop and involving people who otherwise might not be able to apply the changes (for whatever reason) and "quickly test" something out. It's also a great way to validate that a change actually builds in a production environment, where no unnecessary development and test dependencies are installed.

You can also run tests of the built packages on Testing Farm and automate pushing releases into Fedora/CentOS Stream, but this is neither a (plain) Packit advertisement post, nor is that functionality that I can talk about with a certain level of experience.

Adam recently asked why we don't have Packit builds for our our Puppet modules and my first answer was: "well, puppet-* doesn't produce a thing we ship directly, so nobody dared to do it".

My second answer was that I had blogged how to test a Puppet module PR with Packit, but I totally agree that the process was a tad cumbersome and could be improved.

Now some madman did it and we all get to hear his story! ;-)

What is the problem anyway?

The Foreman Installer is a bit of Ruby code1 that provides a CLI to puppet apply based on a set of Puppet modules. As the Puppet modules can also be used outside the installer and have their own lifecycle, they live in separate git repositories and their releases get uploaded to the Puppet Forge. Users however do not want to (and should not have to) install the modules themselves.

So we have to ship the modules inside the foreman-installer package. Packaging 25 modules for two packaging systems (we support Enterprise Linux and Debian/Ubuntu) seems like a lot of work. Especially if you consider that the main foreman-installer package would need to be rebuilt after each module change as it contains generated files based on the modules which are too expensive to generate at runtime.

So we can ship the modules inside the foreman-installer source release, thus vendoring those modules into the installer release.

To do so we use librarian-puppet with a Puppetfile and either a Puppetfile.lock for stable releases or by letting librarian-puppet fetch latest for nightly snapshots.

This works beautifully for changes that land in the development and release branches of our repositories - regardless if it's foreman-installer.git or any of the puppet-*.git ones. It also works nicely for pull-requests against foreman-installer.git.

But because the puppet-* repositories do not map to packages, we assumed it wouldn't work well for pull-requests against those.

How can we solve this?

Well, the "obvious" solution is to build the foreman-installer package via Packit also for pull-requests against the puppet-* repositories. However, as usual, the devil is in the details.

Packit by default clones the repository of the pull-request and tries to create a source tarball from that using git archive. As this might be too simple for many projects, one can define a custom create-archive action that runs after the pull-request has been cloned and produces the tarball instead. We already use that in the Packit configuration for foreman-installer to run the pkg:generate_source rake target which executes librarian-puppet for us.

But now the pull-request is against one of the Puppet modules, so Packit will clone that, not the installer.

We gotta clone foreman-installer on our own. And then point librarian-puppet at the pull-request. Fun.

Cloning is relatively simple, call git clone -- sorry Packit/Copr infrastructure.

But the Puppet module pull-request? One can use :git => 'https://git.example.com/repo.git' in the Puppetfile to fetch a git repository. In fact, that's what we already do for our nightly snapshots. It also supports :ref => 'some_branch_or_tag_name', if the remote HEAD is not what you want.

My brain first went "I know this! GitHub has this magic refs/pull/1/head and refs/pull/1/merge refs you can checkout to get the contents of the pull-request without bothering to add a remote for the source of the pull-request". Well, this requires to know the ID of the pull-request and Packit does not expose that in the environment variables available during create-archive.

Wait, but we already have a checkout. Can we just say :git => '../.git'? Cloning a .git folder is totally possible after all.

[Librarian]     --> fatal: repository '../.git' does not exist
Could not checkout ../.git: fatal: repository '../.git' does not exist

Seems librarian disagrees. Damn. (Yes, I checked, the path exists.)

💡 does it maybe just not like relative paths?! Yepp, using an absolute path absolutely works!

For some reason it ends up checking out the default HEAD of the "real" (GitHub) remote, not of ../. Luckily this can be fixed by explicitly passing :ref => 'origin/HEAD', which resolves to the branch Packit created for the pull-request.

Now we just need to put all of that together and remember to execute all commands from inside the foreman-installer checkout as that is where all our vendoring recipes etc live.

Putting it all together

Let's look at the diff between the packit.yaml for foreman-installer and the one I've proposed for puppet-pulpcore:

--- a/foreman-installer/.packit.yaml    2024-05-14 21:45:26.545260798 +0200
+++ b/puppet-pulpcore/.packit.yaml  2024-05-14 21:44:47.834162418 +0200
@@ -18,13 +18,15 @@
 actions:
   post-upstream-clone:
     - "wget https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/foreman-installer/foreman-installer.spec -O foreman-installer.spec"
+    - "git clone https://github.com/theforeman/foreman-installer"
+    - "sed -i '/theforeman.pulpcore/ s@:git.*@:git => \"#{__dir__}/../.git\", :ref => \"origin/HEAD\"@' foreman-installer/Puppetfile"
   get-current-version:
-    - "sed 's/-develop//' VERSION"
+    - "sed 's/-develop//' foreman-installer/VERSION"
   create-archive:
-    - bundle config set --local path vendor/bundle
-    - bundle config set --local without development:test
-    - bundle install
-    - bundle exec rake pkg:generate_source
+    - bash -c "cd foreman-installer && bundle config set --local path vendor/bundle"
+    - bash -c "cd foreman-installer && bundle config set --local without development:test"
+    - bash -c "cd foreman-installer && bundle install"
+    - bash -c "cd foreman-installer && bundle exec rake pkg:generate_source"
  1. It clones foreman-installer (in post-upstream-clone, as that felt more natural after some thinking)
  2. It adjusts the Puppetfile to use #{__dir__}/../.git as the Git repository, abusing the fact that a Puppetfile is really just a Ruby script (sorry Ben!) and knows the __dir__ it lives in
  3. It fetches the version from the foreman-installer checkout, so it's sort-of reasonable
  4. It performs all building inside the foreman-installer checkout

Can this be used in other scenarios?

I hope so! Vendoring is not unheard of. And testing your "consumers" (dependents? naming is hard) is good style anyway!


  1. three Ruby modules in a trench coat, so to say ↩

14 May, 2024 08:12PM by evgeni

Julian Andres Klode

The new APT 3.0 solver

APT 2.9.3 introduces the first iteration of the new solver codenamed solver3, and now available with the –solver 3.0 option. The new solver works fundamentally different from the old one.

How does it work?

Solver3 is a fully backtracking dependency solving algorithm that defers choices to as late as possible. It starts with an empty set of packages, then adds the manually installed packages, and then installs packages automatically as necessary to satisfy the dependencies.

Deferring the choices is implemented multiple ways:

First, all install requests recursively mark dependencies with a single solution for install, and any packages that are being rejected due to conflicts or user requests will cause their reverse dependencies to be transitively marked as rejected, provided their or group cannot be solved by a different package.

Second, any dependency with more than one choice is pushed to a priority queue that is ordered by the number of possible solutions, such that we resolve a|b before a|b|c.

Not just by the number of solutions, though. One important point to note is that optional dependencies, that is, Recommends, are always sorting after mandatory dependencies. Do note on that: Recommended packages do not “nest” in backtracking - dependencies of a Recommended package themselves are not optional, so they will have to be resolved before the next Recommended package is seen in the queue.

Another important step in deferring choices is extracting the common dependencies of a package across its version and then installing them before we even decide which of its versions we want to install - one of the dependencies might cycle back to a specific version after all.

Decisions about package levels are recorded at a certain decision level, if we reach a conflict we backtrack to the previous decision level, mark the decision we made (install X) in the inverse (DO NOT INSTALL X), reset all the state all decisions made at the higher level, and restore any dependencies that are no longer resolved to the work queue.

Comparison to SAT solver design.

If you have studied SAT solver design, you’ll find that essentially this is a DPLL solver without pure literal elimination. A pure literal eliminitation phase would not work for a package manager: First negative pure literals (packages that everything conflicts with) do not exist, and positive pure literals (packages nothing conflicts with) we do not want to mark for install - we want to install as little as possible (well subject, to policy).

As part of the solving phase, we also construct an implication graph, albeit a partial one: The first package installing another package is marked as the reason (A -> B), the same thing for conflicts (not A -> not B).

Once we have added the ability to have multiple parents in the implication graph, it stands to reason that we can also implement the much more advanced method of conflict-driven clause learning; where we do not jump back to the previous decision level but exactly to the decision level that caused the conflict. This would massively speed up backtracking.

What changes can you expect in behavior?

The most striking difference to the classic APT solver is that solver3 always keeps manually installed packages around, it never offers to remove them. We will relax that in a future iteration so that it can replace packages with new ones, that is, if your package is no longer available in the repository (obsolete), but there is one that Conflicts+Replaces+Provides it, solver3 will be allowed to install that and remove the other.

Implementing that policy is rather trivial: We just need to queue obsolete | replacement as a dependency to solve, rather than mark the obsolete package for install.

Another critical difference is the change in the autoremove behavior: The new solver currently only knows the strongest dependency chain to each package, and hence it will not keep around any packages that are only reachable via weaker chains. A common example is when gcc-<version> packages accumulate on your system over the years. They all have Provides: c-compiler and the libtool Depends: gcc | c-compiler is enough to keep them around.

New features

The new option --no-strict-pinning instructs the solver to consider all versions of a package and not just the candidate version. For example, you could use apt install foo=2.0 --no-strict-pinning to install version 2.0 of foo and upgrade - or downgrade - packages as needed to satisfy foo=2.0 dependencies. This mostly comes in handy in use cases involving Debian experimental or the Ubuntu proposed pockets, where you want to install a package from there, but try to satisfy from the normal release as much as possible.

The implication graph building allows us to implement an apt why command, that while not as nicely detailed as aptitude, at least tells you the exact reason why a package is installed. It will only show the strongest dependency chain at first of course, since that is what we record.

What is left to do?

At the moment, error information is not stored across backtracking in any way, but we generally will want to show you the first conflict we reach as it is the most natural one; or all conflicts. Currently you get the last conflict which may not be particularly useful.

Likewise, errors currently are just rendered as implication graphs of the form [not] A -> [not] B -> ..., and we need to put in some work to present those nicely.

The test suite is not passing yet, I haven’t really started working on it. A challenge is that most packages in the test suite are manually installed as they are mocked, and the solver now doesn’t remove those.

We plan to implement the replacement logic such that foo can be replaced by foo2 Conflicts/Replaces/Provides foo without needing to be automatically installed.

Improving the backtracking to be non-chronological conflict-driven clause learning would vastly enhance our backtracking performance. Not that it seems to be an issue right now in my limited testing (mostly noble 64-bit-time_t upgrades). A lot of that complexity you have normally is not there because the manually installed packages and resulting unit propagation (single-solution Depends/Reverse-Depends for Conflicts) already ground us fairly far in what changes we can actually make.

Once all the stuff has landed, we need to start rolling it out and gather feedback. On Ubuntu I’d like automated feedback on regressions (running solver3 in parallel, checking if result is worse and then submitting an error to the error tracker), on Debian this could just be a role email address to send solver dumps to.

At the same time, we can also incrementally start rolling this out. Like phased updates in Ubuntu, we can also roll out the new solver as the default to 10%, 20%, 50% of users before going to the full 100%. This will allow us to capture regressions early and fix them.

14 May, 2024 11:26AM

hackergotchi for Matthew Palmer

Matthew Palmer

"Is This Project Still Maintained?"

If you wander around a lot of open source repositories on the likes of GitHub, you’ll invariably stumble over repos that have an issue (or more than one!) with a title like the above. Sometimes sitting open and unloved, often with a comment or two from the maintainer and a bunch of “I’ll help out!” followups that never seemed to pan out. Very rarely, you’ll find one that has been closed, with a happy ending.

These issues always fascinate me, because they say a lot about what it means to “maintain” an open source project, the nature of succession (particularly in a post-Jia Tan world), and the expectations of users and the impedence mismatch between maintainers, contributors, and users. I’ve also recently been thinking about pre-empting this sort of issue, and opening my own issue that answers the question before it’s even asked.

Why These Issues Are Created

As both a producer and consumer of open source software, I completely understand the reasons someone might want to know whether a project is abandoned. It’s comforting to be able to believe that there’s someone “on the other end of the line”, and that if you have a problem, you can ask for help with a non-zero chance of someone answering you. There’s also a better chance that, if the maintainer is still interested in the software, that compatibility issues and at least show-stopper bugs might get fixed for you.

But often there’s more at play. There is a delusion that “maintained” open source software comes with entitlements – an expectation that your questions, bug reports, and feature requests will be attended to in some fashion.

This comes about, I think, in part because there are a lot of open source projects that are energetically supported, where generous volunteers do answer questions, fix reported bugs, and implement things that they don’t personally need, but which random Internet strangers ask for. If you’ve had that kind of user experience, it’s not surprising that you might start to expect it from all open source projects.

Of course, these wonders of cooperative collaboration are the exception, rather than the rule. In many (most?) cases, there is little practical difference between most projects that are “maintained” and those that are formally declared “unmaintained”. The contributors (or, most often, contributor – singular) are unlikely to have the time or inclination to respond to your questions in a timely and effective manner. If you find a problem with the software, you’re going to be paddling your own canoe, even if the maintainer swears that they’re still “maintaining” it.

A Thought Appears

With this in mind, I’ve been considering how to get ahead of the problem and answer the question for the software projects I’ve put out in the world. Nothing I’ve built has anything like what you’d call a “community”; most have never seen an external PR, or even an issue. The last commit date on them might be years ago.

By most measures, almost all of my repos look “unmaintained”. Yet, they don’t feel unmaintained to me. I’m still using the code, sometimes as often as every day, and if something broke for me, I’d fix it. Anyone who needs the functionality I’ve developed can use the code, and be pretty confident that it’ll do what it says in the README.

I’m considering creating an issue in all my repos, titled “Is This Project Still Maintained?”, pinning it to the issues list, and pasting in something I’m starting to think of as “The Open Source Maintainer’s Manifesto”.

It goes something like this:

Is This Project Still Maintained?

Yes. Maybe. Actually, perhaps no. Well, really, it depends on what you mean by “maintained”.

I wrote the software in this repo for my own benefit – to solve the problems I had, when I had them. While I could have kept the software to myself, I instead released it publicly, under the terms of an open licence, with the hope that it might be useful to others, but with no guarantees of any kind. Thanks to the generosity of others, it costs me literally nothing for you to use, modify, and redistribute this project, so have at it!

OK, Whatever. What About Maintenance?

In one sense, this software is “maintained”, and always will be. I fix the bugs that annoy me, I upgrade dependencies when not doing so causes me problems, and I add features that I need. To the degree that any on-going development is happening, it’s because I want that development to happen.

However, if “maintained” to you means responses to questions, bug fixes, upgrades, or new features, you may be somewhat disappointed. That’s not “maintenance”, that’s “support”, and if you expect support, you’ll probably want to have a “support contract”, where we come to an agreement where you pay me money, and I help you with the things you need help with.

That Doesn’t Sound Fair!

If it makes you feel better, there are several things you are entitled to:

  1. The ability to use, study, modify, and redistribute the contents of this repository, under the terms stated in the applicable licence(s).

  2. That any interactions you may have with myself, other contributors, and anyone else in this project’s spaces will be in line with the published Code of Conduct, and any transgressions of the Code of Conduct will be dealt with appropriately.

  3. … actually, that’s it.

Things that you are not entitled to include an answer to your question, a fix for your bug, an implementation of your feature request, or a merge (or even review) of your pull request. Sometimes I may respond, either immediately or at some time long afterwards. You may luck out, and I’ll think “hmm, yeah, that’s an interesting thing” and I’ll work on it, but if I do that in any particular instance, it does not create an entitlement that I will continue to do so, or that I will ever do so again in the future.

But… I’ve Found a Huge and Terrible Bug!

You have my full and complete sympathy. It’s reasonable to assume that I haven’t come across the same bug, or at least that it doesn’t bother me, otherwise I’d have fixed it for myself.

Feel free to report it, if only to warn other people that there is a huge bug they might need to avoid (possibly by not using the software at all). Well-written bug reports are great contributions, and I appreciate the effort you’ve put in, but the work that you’ve done on your bug report still doesn’t create any entitlement on me to fix it.

If you really want that bug fixed, the source is available, and the licence gives you the right to modify it as you see fit. I encourage you to dig in and fix the bug. If you don’t have the necessary skills to do so yourself, you can get someone else to fix it – everyone has the same entitlements to use, study, modify, and redistribute as you do.

You may also decide to pay me for a support contract, and get the bug fixed that way. That gets the bug fixed for everyone, and gives you the bonus warm fuzzies of contributing to the digital commons, which is always nice.

But… My PR is a Gift!

If you take the time and effort to make a PR, you’re doing good work and I commend you for it. However, that doesn’t mean I’ll necessarily merge it into this repository, or even work with you to get it into a state suitable for merging.

A PR is what is often called a “gift of work”. I’ll have to make sure that, at the very least, it doesn’t make anything actively worse. That includes introducing bugs, or causing maintenance headaches in the future (which includes my getting irrationally angry at indenting, because I’m like that). Properly reviewing a PR takes me at least as much time as it would take me to write it from scratch, in almost all cases.

So, if your PR languishes, it might not be that it’s bad, or that the project is (dum dum dummmm!) “unmaintained”, but just that I don’t accept this particular gift of work at this particular time.

Don’t forget that the terms of licence include permission to redistribute modified versions of the code I’ve released. If you think your PR is all that and a bag of potato chips, fork away! I won’t be offended if you decide to release a permanent fork of this software, as long as you comply with the terms of the licence(s) involved.

(Note that I do not undertake support contracts solely to review and merge PRs; that reeks a little too much of “pay to play” for my liking)

Gee, You Sound Like an Asshole

I prefer to think of myself as “forthright” and “plain-speaking”, but that brings to mind that third thing you’re entitled to: your opinion.

I’ve written this out because I feel like clarifying the reality we’re living in, in the hope that it prevents misunderstandings. If what I’ve written makes you not want to use the software I’ve written, that’s fine – you’ve probably avoided future disappointment.

Opinions Sought

What do you think? Too harsh? Too wishy-washy? Comment away!

14 May, 2024 12:00AM by Matt Palmer (mpalmer@hezmatt.org)

May 12, 2024

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

Wikimedia jumps on the AI bandwagon

I've been resisting the Wikipedia ads about “we don't run ads, give us money“ for over a decade now (mostly since WMF already has tons of cash and use very little on it to actually improve Wikipedia), and now they are jumping on the AI/LLM hype. It does not help.

(Of course you should pick your charities yourself. I've donated to Signal Foundation in the past even though I think they could run things somewhat cheaper if they didn't insist on all-cloud, and archive.org is basically, for better or for worse, the collective memory of the Internet by now, at least unless they get bankrupted by the ongoing lawsuit from some overly silly book lending. Not to mention that in a day and age where there's a certain Eurovision-participating country and another banned-from-Eurovision country both causing tons of civilian suffering and casualities these days, perhaps there are non-tech charities that are also important.)

12 May, 2024 04:40PM

Elana Hashman

I am very sick

I have not been able to walk since February 18, 2023.

When people ask me how I'm doing, this is the first thing that comes to mind. "Well, you know, the usual, but also I still can't walk," I think to myself.

If I dream at night, I often see myself walking or running. In conversation, if I talk about going somewhere, I'll imagine walking there. Even though it's been over a year, I remember walking to the bus, riding to see my friends, going out for brunch, cooking community dinners.

But these days, I can't manage going anywhere except by car, and I can't do the driving, and I can't dis/assemble and load my chair. When I'm resting in bed and follow a guided meditation, I might be asked to imagine walking up a staircase, step by step. Sometimes, I do. Other times, I imagine taking a little elevator in my chair, or wheeling up ramps.

I feel like there is little I can say that can express the extent of what this illness has taken from me, but it's worth trying. To an able-bodied person, seeing me in a power wheelchair is usually "enough." One of my acquaintances cried when they last saw me in person. But frankly, I love my wheelchair. I am not "wheelchair-bound"—I am bed-bound, and the wheelchair gets me out of bed. My chair hasn't taken anything from me.

***

In October of 2022, I was diagnosed with myalgic encephalomyelitis.

Scientists and doctors don't really know what myalgic encephalomyelitis (ME) is. Diseases like it have been described for over 200 years.1 It primarily affects women between the ages of 10-39, and the primary symptom is "post-exertional malaise" or PEM: debilitating, disproportionate fatigue following activity, often delayed by 24-72 hours and not relieved by sleep. That fatigue has earned the illness the misleading name of "Chronic Fatigue Syndrome" or CFS, as though we're all just very tired all the time. But tired people respond to exercise positively. People with ME/CFS do not.2

Given the dearth of research and complete lack of on-label treatments, you may think this illness is at least rare, but it is actually quite common: in the United States, an estimated 836k-2.5m people3 have ME/CFS. It is frequently misdiagnosed, and it is estimated that as many as 90% of cases are missed,4 due to mild or moderate symptoms that mimic other diseases. Furthermore, over half of Long COVID cases likely meet the diagnostic criteria for ME,5 so these numbers have increased greatly in recent years. That is, ME is at least as common as rheumatoid arthritis,6 another delightful illness I have. But while any doctor knows what rheumatoid arthritis is, not enough7 have heard of "myalgic encephalomylitis."

Despite a high frequency and disease burden, post-viral associated conditions (PASCs) such as ME have been neglected for medical funding for decades.8 Indeed, many people, including medical care workers, find it hard to believe that after the acute phase of illness, severe symptoms can persist. PASCs such as ME and Long COVID defy the typical narrative around common illnesses. I was always told that if I got sick, I should expect to rest for a bit, maybe take some medications, and a week or two later, I'd get better, right? But I never got better.

These are complex, multi-system diseases that do not neatly fit into the Western medical system's specializations. I have seen nearly every specialty because ME/CFS affects nearly every system of the body: cardiology, nephrology, pulmonology, neurology, opthalmology, and, many, many more. You'd think they'd hand out frequent flyer cards, or a medical passport with fun stamps, but nope. Just hundreds of pages of medical records. And when I don't fit neatly into one particular specialist's box, then I'm sent back to my primary care doctor to regroup while we try to troubleshoot my latest concerning symptoms. "Sorry, can't help you. Not my department."

With little available medical expertise, a lot of my disease management has been self-directed in partnership with primary care. I've read hundreds of articles, papers, publications, CME material normally reserved for doctors. It's truly out of necessity, and I'm certain I would be much worse off if I lacked the skills and connections to do this; there are so few ME/CFS experts in the US that there isn't one in my state or any adjacent state.9 So I've done a lot of my own work, much of it while barely being able to read. (A text-to-speech service is a real lifesaver.) To facilitate managing my illness, I've built a mental model of how my particular flavour of ME/CFS works based on the available research I've been able to read and how I respond to treatments. Here is my best attempt to explain it:

  • After a severe (non-COVID) infection, an ongoing interaction between my immune system and my metabolism have stopped my body from being able to do aerobic respiration.10
  • I don't know why or how, but my mitochondria don't work properly anymore.11
  • This means that if I use too much energy, my body isn't able to make enough energy to catch up, and I have severe symptoms over the next few days as my body tries to manage the consequences.
  • Those symptoms aren't limited to fatigue: I've developed flu-like symptoms and even fevers, limbs so heavy they felt paralyzed, tachycardia in response to even the slightest activity.

The best way I have learned to manage this is to prevent myself from doing activities where I will exceed that aerobic threshold by wearing a heartrate monitor,12 but the amount of activity that permits in my current state of health is laughably restrictive. Most days I'm unable to spend more than one to two hours out of bed.

Over time, this has meant worsening from a persistent feeling of tiredness all the time and difficulty commuting into an office or sitting at a desk, to being unable to sit at a desk for an entire workday even while working from home and avoiding physically intense chores or exercise without really understanding why, to being unable to leave my apartment for days at a time, and finally, being unable to stand for more than a minute or two or walk.

But it's not merely that I can't walk. Many folks in wheelchairs are able to live excellent lives with adaptive technology. The problem is that I am so fatigued, any activity can destroy my remaining quality of life. In my worst moments, I've been unable to read, move my arms or legs, or speak aloud. Every single one of my limbs burned, as though I had caught fire. Food sat in my stomach for hours, undigested, while my stomach seemingly lacked the energy to do its job. I currently rely on family and friends for full-time caretaking, plus a paid home health aide, as I am unable to prep meals, shower, or leave the house independently. This assistance has helped me slowly improve from my poorest levels of function.

While I am doing better than I was at my worst, I've had to give up essentially all of my hobbies with physical components. These include singing, cooking, baking, taking care of my houseplants, cross-stitching, painting, and so on. Doing any of these result in post-exertional malaise so I've had to stop; this reduction of activity to prevent worsening the illness is referred to as "pacing." I've also had to cut back essentially all of my volunteering and work in open source; I am only cleared by my doctor to work 15h/wk (from bed) as of writing.

***

CW: severe illness, death, and suicide (skip this section)

The difficulty of living with a chronic illness is that there's no light at the end of the tunnel. Some diseases have a clear treatment path: you take the medications, you complete the procedures, you hit all the milestones, and then you're done, perhaps with some long-term maintenance work. But with ME, there isn't really an end in sight. The median duration of illness reported in one 1997 study was over 6 years, with some patients reporting 20 years of symptoms.13 While a small number of patients spontaneously recover, and many improve, the vast majority of patients are unable to regain their baseline function.14

My greatest fear since losing the ability to walk is getting worse still. Because, while I already require assistance with nearly every activity of daily living, there is still room for decline. The prognosis for extremely ill patients is dismal, and many require feeding tubes and daily nursing care. This may lead to life-threatening malnutrition;15 a number of these extremely severe patients have died, either due to medical neglect or suicide.16 Extremely severe patients cannot tolerate light, sound, touch, or cognitive exertion,17 and often spend most of their time lying flat in a darkened room with ear muffs or an eye mask.18

This is all to say, my prognosis is not great.

But while I recognize that the odds aren't exactly in my favour, I am also damn stubborn. (A friend once cheerfully described me as "stubbornly optimistic!") I only get one shot at life, and I do not want to spend the entirety of it barely able to perceive what's going on around me. So while my prognosis is uncertain, there's lots of evidence that I can improve somewhat,19 and there's also lots of evidence that I can live 20+ years with this disease. It's a bitter pill to swallow, but it also means I might have the gift of time—something that not all my friends with severe complex illnesses have had.

I feel like I owe it to myself to do the best I can to improve; to try to help others in a similar situation; and to enjoy the time that I have. I already feel like my life has been moving in slow motion for the past 4 years—there's no need to add more suffering. Finding joy, as much as I can, every day, is essential to keep up my strength for this marathon. Even if it takes 20 years to find a cure, I am convinced that the standard of care is going to improve. All the research and advocacy that's been happening over the past decade is plenty to feel hopeful about.20 Hope is a discipline,21 and I try to remind myself of this on the hardest days.

***

I'm not entirely sure why I decided to write this. Certainly, today is International ME/CFS Awareness Day, and I'm hoping this post will raise awareness in spaces that aren't often thinking about chronic illnesses. But I think there is also a part of me that wants to share, reach out in some way to the people I've lost contact with while I've been treading water, managing the day to day of my illness. I experience this profound sense of loss, especially when I think back to the life I had before. Everyone hits limitations in what they can do and accomplish, but there is so little I can do with the time and energy that I have. And yet, I understand even this precious little could still be less. So I pace myself.

Perhaps I can inspire you to take action on behalf of those of us too fatigued to do the advocacy we need and deserve. Should you donate to a charity or advocacy organization supporting ME/CFS research? In the US, there are many excellent organizations, such as ME Action, the Open Medicine Foundation, SolveME, the Bateman Horne Center, and the Workwell Foundation. I am also happy to match any donations through the end of May 2024 if you send me your receipts. But charitable giving only goes so far, and I think this problem deserves the backing of more powerful organizations.

Proportionate government funding and support is desperately needed. It's critical for us to push governments22 to provide the funding required for research that will make an impact on patients' lives now. Many organizers are running campaigns around the world, advocating for this investment. There is a natural partnership between ME advocacy and Long COVID advocacy, for example, and we have an opportunity to make a great difference to many people by pushing for research and resources inclusive of all PASCs. Some examples I'm aware of include:

But outside of collective organizing, there are a lot of sick individuals out there that need help, too. Please, don't forget about us. We need you to visit us, care for us, be our confidantes, show up as friends. There are a lot of people who are very sick out here and need your care.

I'm one of them.

12 May, 2024 02:00PM by Elana Hashman

May 11, 2024

Sven Hoexter

xdg and mime types - stuff I would've loved to know a week ago

Learned a few things about xdg and mimetype registration in the last week that could be helpful to have condensed in a single place.

No Need to Ship a Mailcap Mime File

If you already ship a .desktop file (that is what ends up in /usr/share/applications/) which has a MimeType declared, there is no need to also ship a mailcap file (that is what ends up in /usr/lib/mime/packages/). Some triggers will do the conversion work for you. See also Debian Policy 4.9.

Reverse DNS Naming Convention for .desktop Files

Seems to be a closely guarded secret, maybe mainly known inside the Gnome world, but it's in the spec. Also not very widely known inside Debian if I look at my local system as not very representative sample.

Your hicolor Theme App Icon can be a Mime Type Icon as Well

In case you didn't know the hicolor icon theme is the default fallback theme. Many of us already install application icons e.g. in /usr/share/icons/hicolor/48x48/apps/ which is used in conjunction with the Icon field in the .desktop file to locate the application icon. Now the next step, and there it seems quite of few us miss out, is to create a symlink to also provide a mime type icon, so it's displayed in graphical file managers for the application data files. The schema here is simple: Take the MimeType e.g. application/x-vymand replace the / with a - and use that as file name in e.g. /usr/share/icons/hicolor/48x48/mimetypes/. In the vym case that is /usr/share/icons/hicolor/48x48/mimetypes/application-x-vym.png. If you have one use a scalable .svg file instead of .png.

This seems to be an area where Debian lacks a bit of tooling to automatically convert application icons to all the different sizes and install it in all the appropriate places. What is already there is a trigger to run gtk-update-icon-cache when you install new icons into one of the icon theme folder so they're picked up.

No Priority or Order in .desktop Files

Likely something that hapens on all my fresh installations: Libreoffice is installed and xdg-open starts to open pdf files with Libreoffice instead of evince. Now I've to figure out again to run xdg-mime default org.gnome.Evince.desktop application/pdf to change that (at least for my user). Background here is that the desktop file spec explicitly mandates "Priority for applications is handled external to the .desktop files.". That's why we got in addition to all of that mimeapps.list files. And now, after running the xdg-mime command from above, we've a ~/.config/mimeapps.list defining

[Default Applications]
application/pdf=org.gnome.Evince.desktop

Debian as whole seems to be not very keen on shipping something like a sensible default mimeapps.list outside of desktop environment specific ones. A quick search gave me just

$ apt-file search mimeapps.list
cinnamon-desktop-data: /usr/share/applications/x-cinnamon-mimeapps.list
gdm3: /usr/share/gdm/greeter/applications/mimeapps.list
gnome-session-common: /usr/share/applications/gnome-mimeapps.list
plasma-workspace: /usr/share/applications/kde-mimeapps.list
sxmo-utils: /usr/share/applications/mimeapps.list
sxmo-utils: /usr/share/sxmo/xdg/mimeapps.list

While it's a bit anoying to run into that pdf vs Libreoffice thing every now and then, it's maybe better to not have long controversial threads about default pdf viewer, like the ones we already had about the default MTA choices. ;) And while we're at it: everyone using Libreoffice should give a virtual hug to rene@ for taming that beast since 2010 and OpenOffice.org before.

11 May, 2024 04:57PM

May 10, 2024

Reproducible Builds

Reproducible Builds in April 2024

Welcome to the April 2024 report from the Reproducible Builds project! In our reports, we attempt to outline what we have been up to over the past month, as well as mentioning some of the important things happening more generally in software supply-chain security. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.

Table of contents:

  1. New backseat-signed tool to validate distributions’ source inputs
  2. ‘NixOS is not reproducible’
  3. Certificate vulnerabilities in F-Droid’s fdroidserver
  4. Website updates
  5. ‘Reproducible Builds and Insights from an Independent Verifier for Arch Linux’
  6. libntlm now releasing ‘minimal source-only tarballs’
  7. Distribution work
  8. Mailing list news
  9. diffoscope
  10. Upstream patches
  11. reprotest
  12. Reproducibility testing framework

New backseat-signed tool to validate distributions’ source inputs

kpcyrd announced a new tool called backseat-signed, after:

I figured out a somewhat straight-forward way to check if a given git archive output is cryptographically claimed to be the source input of a given binary package in either Arch Linux or Debian (or both).

Elaborating more in their announcement post, kpcyrd writes:

I believe this to be the “reproducible source tarball” thing some people have been asking about. As explained in the README, I believe reproducing autotools-generated tarballs isn’t worth everybody’s time and instead a distribution that claims to build from source should operate on VCS snapshots instead of tarballs with 25k lines of pre-generated shell-script.

Indeed, many distributions’ packages already build from VCS snapshots, and this trend is likely to accelerate in response to the xz incident. The announcement led to a lengthy discussion on our mailing list, as well as shorter followup thread from kpcyrd about bootstrapping Autotools projects.


‘NixOS is not reproducible’

Morten Linderud posted an post on his blog this month, provocatively titled, “NixOS is not reproducible”. Although quickly admitting that his title is indeed “clickbait”, Morten goes on to clarify the precise guarantees and promises that NixOS provides its users.

Later in the most, Morten mentions that he was motivated to write the post because:

I have heavily invested my free-time on this topic since 2017, and met some of the accomplishments we have had with “Doesn’t NixOS solve this?” for just as long… and I thought it would be of peoples interest to clarify[.]


Certificate vulnerabilities in F-Droid’s fdroidserver

In early April, Fay Stegerman announced a certificate pinning bypass vulnerability and Proof of Concept (PoC) in the F-Droid fdroidserver tools for “managing builds, indexes, updates, and deployments for F-Droid repositories” to the oss-security mailing list.

We observed that embedding a v1 (JAR) signature file in an APK with minSdk >= 24 will be ignored by Android/apksigner, which only checks v2/v3 in that case. However, since fdroidserver checks v1 first, regardless of minSdk, and does not verify the signature, it will accept a “fake” certificate and see an incorrect certificate fingerprint. […] We also realised that the above mentioned discrepancy between apksigner and androguard (which fdroidserver uses to extract the v2/v3 certificates) can be abused here as well. […]

Later on in the month, Fay followed up with a second post detailing a third vulnerability and a script that could be used to scan for potentially affected .apk files and mentioned that, whilst upstream had acknowledged the vulnerability, they had not yet applied any ameliorating fixes.


Website updates

There were a number of improvements made to our website this month, including Chris Lamb updating the archive page to recommend -X and unzipping with TZ=UTC [] and adding Maven, Gradle, JDK and Groovy examples to the SOURCE_DATE_EPOCH page []. In addition Jan Zerebecki added a new /contribute/opensuse/ page [] and Sertonix fixed the automatic RSS feed detection [][].


Reproducible Builds and Insights from an Independent Verifier for Arch Linux

Joshua Drexel, Esther Hänggi and Iyán Méndez Veiga of the School of Computer Science and Information Technology, Hochschule Luzern (HSLU) in Switzerland published a paper this month entitled Reproducible Builds and Insights from an Independent Verifier for Arch Linux. The paper establishes the context as follows:

Supply chain attacks have emerged as a prominent cybersecurity threat in recent years. Reproducible and bootstrappable builds have the potential to reduce such attacks significantly. In combination with independent, exhaustive and periodic source code audits, these measures can effectively eradicate compromises in the building process. In this paper we introduce both concepts, we analyze the achievements over the last ten years and explain the remaining challenges.

What is more, the paper aims to:

… contribute to the reproducible builds effort by setting up a rebuilder and verifier instance to test the reproducibility of Arch Linux packages. Using the results from this instance, we uncover an unnoticed and security-relevant packaging issue affecting 16 packages related to Certbot […].

A PDF of the paper is available.


libntlm now releasing ‘minimal source-only tarballs’

Simon Josefsson wrote on his blog this month that, going forward, the libntlm project will now be releasing what they call “minimal source-only tarballs”:

The XZUtils incident illustrate that tarballs with files that are not included in the git archive offer an opportunity to disguise malicious backdoors. [The] risk of hiding malware is not the only motivation to publish signed minimal source-only tarballs. With pre-generated content in tarballs, there is a risk that GNU/Linux distributions [ship] generated files coming from the tarball into the binary *.deb or *.rpm package file. Typically the person packaging the upstream project never realized that some installed artifacts was not re-built[.]

Simon’s post goes into further details how this was achieved, and describes some potential caveats and counters some expected responses as well. A shorter version can be found in the announcement for the 1.8 release of libntlm.


Distribution work

In Debian this month, Helmut Grohne filed a bug suggesting the removal of dh-buildinfo, a tool to generate and distribute .buildinfo-like files within binary packages. Note that this is distinct from the .buildinfo generation performed by dpkg-genbuildinfo. By contrast, the entirely optional dh-buildinfo generated a debian/buildinfo file that would be shipped within binary packages as /usr/share/doc/package/buildinfo_$arch.gz.

Adrian Bunk recently asked about including source hashes in Debian’s .buildinfo files, which prompted Guillem Jover to refresh some old patches to dpkg to make this possible, which revealed some quirks Vagrant Cascadian discovered when testing.

In addition, 21 reviews of Debian packages were added, 22 were updated and 16 were removed this month adding to our knowledge about identified issues. A number issue types have been added, such as new random_temporary_filenames_embedded_by_mesonpy and timestamps_added_by_librime toolchain issues.

In openSUSE, it was announced that their Factory distribution enabled bit-by-bit reproducible builds for almost all parts of the package. Previously, more parts needed to be ignored when comparing package files, but now only the signature needs to be deleted.

In addition, Bernhard M. Wiedemann published theunreproduciblepackage as a proper .rpm package which it allows to better test tools intended to debug reproducibility. Furthermore, it was announced that Bernhard’s work on a 100% reproducible openSUSE-based distribution will be funded by NLnet. He also posted another monthly report for his reproducibility work in openSUSE.

In GNU Guix, Janneke Nieuwenhuizen submitted a patch set for creating a reproducible source tarball for Guix. That is to say, ensuring that make dist is reproducible when run from Git. []

Lastly, in Fedora, a new wiki page was created to propose a change to the distribution. Titled “Changes/ReproduciblePackageBuilds”, the page summarises itself as a proposal whereby “A post-build cleanup is integrated into the RPM build process so that common causes of build irreproducibility in packages are removed, making most of Fedora packages reproducible.”


Mailing list news

On our mailing list this month:

  • Continuing a thread started in March 2024 about the Arch Linux minimal container now being 100% reproducible, John Gilmore followed up with a post about the practical and philosophical distinctions of local vs. remote storage of the various artifacts needed to build packages.

  • Chris Lamb asked the list which conferences readers are attending these days: “After peak Covid and other industry-wide changes, conferences are no longer the ‘must attend’ events they previously were… especially in the area of software supply-chain security. In rough, practical terms, it seems harder to justify conference travel today than it did in mid-2019.” The thread generated a number of responses which would be of interest to anyone planning travel in Q3 and Q4 of 2024.

  • James Addison wrote to the list about a “quirk” in Git related to its core.autocrlf functionality, thus helpfully passing on a “slightly off-topic and perhaps not of direct relevance to anyone on the list today” note that might still be “the kind of issue that is useful to be aware of if-and-when puzzling over unexpected git content / checksum issues (situations that I do expect people on this list encounter from time-to-time)”.


diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes such as uploading versions 263, 264 and 265 to Debian and made the following additional changes:

  • Don’t crash on invalid .zip files, even if we encounter their ‘badness’ halfway through the file and not at the time of their initial opening. []
  • Prevent odt2txt tests from always being skipped due to an (impossibly) new version requirement. []
  • Avoid parens-in-parens in test ‘skipping’ messages. []
  • Ensure that tests with >=-style version constraints actually print the tool name. []

In addition, Fay Stegerman fixed a crash when there are (invalid) duplicate entries in .zip which was originally reported in Debian bug #1068705). [] Fay also added a user-visible ‘note’ to a diff when there are duplicate entries in ZIP files []. Lastly, Vagrant Cascadian added an external tool pointer for the zipdetails tool under GNU Guix [] and proposed updates to diffoscope in Guix as well [] which were merged as [264] [265], fixed a regression in test coverage and increased verbosity of the test suite[].


Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:


reprotest

reprotest is our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences. This month, reprotest version 0.7.27 was uploaded to Debian unstable) by Vagrant Cascadian who made the following additional changes:

  • Enable specific number of CPUs using --vary=num_cpus.cpus=X. []
  • Consistently use 398 days for time variation, rather than choosing randomly each time. []
  • Disable builds of arch:any packages. []
  • Update the description for the build_path.path option in README.rst. []
  • Update escape sequences for compatibility with Python 3.12. (#1068853). []
  • Remove the generic ‘upstream’ signing-key [] and update the packages’ signing key with the currently active team members [].
  • Update the packaging Standards-Version to 4.7.0. []

In addition, Holger Levsen fixed some spelling errors detected by the spellintian tool [] and Vagrant Cascadian updated reprotest in GNU Guix to 0.7.27.


Reproducibility testing framework

The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility.

In April, an enormous number of changes were made by Holger Levsen:

  • Debian-related changes:

    • Adjust for changed internal IP addresses at Codethink. []
    • Automatically cleanup failed diffoscope user services if there are too many failures. [][]
    • Configure two new nodes at infomanik.cloud. [][]
    • Schedule Debian experimental even less. [][]
  • Breakage detection:

    • Exclude currently building packages from breakage detection. []
    • Be more noisy if diffoscope crashes. []
    • Health check: provide clickable URLs in jenkins job log for failed pkg builds due to diffoscope crashes. []
    • Limit graph to about the last 100 days of breakages only. []
    • Fix all found files with bad permissions. []
    • Prepare dealing with diffoscope timeouts. []
    • Detect more cases of failure to debootstrap base system. []
    • Include timestamps of failed job runs. []
  • Documentation updates:

    • Document how to access arm64 nodes at Codethink. []
    • Document how to use infomaniak.cloud. []
    • Drop notes about long stalled LeMaker HiKey960 boards sponsored by HPE and hosted at ETH. []
    • Mention osuosl4 and osuosl5 and explain their usage. []
    • Mention that some packages are built differently. [][]
    • Improve language in a comment. []
    • Add more notes how to query resource usage from infomaniak.cloud. []
  • Node maintenance:

    • Add ionos4 and ionos14 to THANKS. [][][][][]
    • Deprecate Squid on ionos1 and ionos10. []
    • Drop obsolete script to powercycle arm64 architecture nodes. []
    • Update system_health_check for new proxy nodes. []
  • Misc changes:

    • Make the update_jdn.sh script more robust. [][]
    • Update my SSH public key. []

In addition, Mattia Rizzolo added some new host details. []



If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

10 May, 2024 10:05AM

May 09, 2024

Vincent Sanders

Bee to the blossom, moth to the flame; Each to his passion; what's in a name?

I like the sentiment of Helen Hunt Jackson in that quote and it generally applies double for computer system names. However I like to think when I named the first NetSurf VM host server phoenix fourteen years ago I captured the nature of its continuous cycle of replacement.

Image of the fourth phoenix server
We have been very fortunate to receive a donated server to replace the previous every few years and the very generous folks at Collabora continue to provide hosting for it.

Recently I replaced the server for the third time. We once again were given a replacement by Huw Jones in the form of a SuperServer 6017R-TDAF system with dual Intel Xeon Ivy Bridge E5-2680v2 processors. There were even rack rails!

The project bought some NVMe drives and an adaptor cards and I attempted to arrange to swap out the server in January.

The old phoenixiii server being replaced
Here we come to the slight disadvantage of an informal arrangement where access to the system depends upon a busy third party. Unfortunately it took until May to arrange access (I must thank Vivek again for coming in on a Saturday to do this)

In the intervening time, once I realised access was going to become increasingly difficult, I decided to obtain as good a system as I could manage to reduce requirements for future access. 

I turned to eBay and acquired a slightly more modern SuperServer with dual Intel Xeon Haswell E5-2680v3 processors which required purchase of 64G of new memory (Haswell is a DDR4 platform). 

I had wanted to use Broadwell processors but this exceeded my budget and would only be a 10% performance uplift (The chassis, motherboard and memory cost £180 and another £50 for processors was just too much, maybe next time)

graph of cpu mark improvements in the phoenix servers over time
While making the decision on the processor selection I made a quick chart of previous processing capabilities (based on a passmark comparison) of phoenix servers and was startled to discover I needed a logarithmic vertical axis. Multi core performance of processors has improved at a startling rate in the last decade.

When the original replacement was donated I checked where the performance was limited and noticed it was mainly in disc access which is what prompted the upgrade to NVMe (2 gigabytes a second peek read throughput) which moved the bottleneck to the processors where, even with the upgrades, it remains.

I do not really know if there is a conclusion here beyond noting NetSurf is very fortunate as a project to have some generous benefactors both for donating hardware and hosting for which I know all the developers are grateful.

Now I just need to go and migrate a huge bunch of virtual machines and associated sysadmin to make use of these generous donations.

09 May, 2024 09:24PM by Vincent Sanders (noreply@blogger.com)

Thorsten Alteholz

My Debian Activities in April 2024

FTP master

This month I accepted 386 and rejected 39 packages. The overall number of packages that got accepted was 386.

I also added lots of +moreinfo tags to some RM bugs. Is it that hard to check the reverse dependencies on your own?

Debian LTS

This was my hundred-eighteenth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.

During my allocated time I uploaded:

  • [DLA 3781-1] libgd2 security update for three CVEs to fix out-of-bounds reads or NULL pointer derefence
    • [DLA 3784-1] libcaca security update for two CVEs to fix heap buffer overflows
      • [DLA 3805-1] qtbase-opensource-src security update for seven CVEs to fix buffer overflows, infinite loops or application crashs due to processing of crafted input files. When trying to compile the fixed version, I got an error that there is no .compare() for QByteArray available. Yes, indeed, QByteArray::compare, which was used in a patch, was introduced only in Qt 6.0. So I had to backport that to Buster as well. It is astonishing that such a basic function was not needed before.
      • [#1070153] bookworm-pu: qtbase-opensource-src/5.15.8+dfsg-11+deb12u2 to fix two CVEs
      • [#1070154] bullseye-pu: qtbase-opensource-src/5.15.2+dfsg-9+deb11u1 to fix ten CVEs
      • [#1064550] uploaded libjwt
      • [#1067544] uploaded libmicrohttpd

      I also continued to work on tiff and last but not least did a week of FD and attended the monthly LTS/ELTS meeting.

      Debian ELTS

      This month was the sixty-ninth ELTS month. During my allocated time I uploaded:

      • [ELA-1069-1]libgd2 security update for three CVEs to fix out-of-bounds reads or NULL pointer derefence in Jessie and Stretch
      • [ELA-1070-1]libcaca security update for two CVEs to fix heap buffer overflows in Jessie and Stretch
      • [ELA-1083-1]qtbase-opensource-src security update for five CVEs to fix buffer overflows, infinite loops or application crashs due to processing of crafted input files in Stretch

      I also continued to work on an update for tiff in Jessie and Stretch, did a week of FD and attended the LTS/ELTS meeting.

      Debian Printing

      This month I uploaded new upstream or bugfix versions of:

      In preparation for cups3 I introduced a new package:

      This work is generously funded by Freexian!

      Debian Astro

      This month I uploaded a new upstream or bugfix version of:

      Debian IoT

      This month I uploaded new upstream or bugfix versions of:

      Debian Mobcom

      This month I uploaded new upstream or bugfix versions of:

      I have done these uploads in preparation for my GSoC student, who will be officially announced in May.

      misc

      I am sorry for people still using 32bit computers, but from my point of view these are dying architectures. So if there are any problems with builds on those architectures, I no longer try to fix them but file RM bugs. Patches are welcome, but I am no longer willing to spend any time for this.

      This month I uploaded new upstream or bugfix versions of:

      This month I even found some time to introduce new packages:

      • mailio a cross platform C++ library for email support
      • pksc11-proxy a proxy for the PKCS11-library

09 May, 2024 11:44AM by alteholz

hackergotchi for Gunnar Wolf

Gunnar Wolf

Hacks, leaks, and revelations • The art of analyzing hacked and leaked data

This post is a review for Computing Reviews for Hacks, leaks and revelations • The art of analyzing hacked and leaked data , a book published in No Starch Press

Imagine you’ve come across a trove of files documenting a serious deed and you feel the need to “blow the whistle.” Or maybe you are an investigative journalist and this whistleblower trusts you and wants to give you said data. Or maybe you are a technical person, trusted by said journalist to help them do things right–not only to help them avoid being exposed while leaking the information, but also to assist them in analyzing the contents of the dataset. This book will be a great aid for all of the above tasks.

The author, Micah Lee, is both a journalist and a computer security engineer. The book is written entirely from his experience handling important datasets, and is organized in a very logical and sound way. Lee organized the 14 chapters in five parts. The first part–the most vital to transmitting the book’s message, in my opinion–begins by talking about the care that must be taken when handling a sensitive dataset: how to store it, how to communicate it to others, sometimes even what to redact (exclude) so the information retains its strength but does not endanger others (or yourself). The first two chapters introduce several tools for encrypting information and keeping communication anonymous, not getting too deep into details and keeping it aimed at a mostly nontechnical audience.

Something that really sets this book apart from others like it is that Lee’s aim is not only to tell stories about the “hacks and leaks” he has worked with, or to present the technical details on how he analyzed them, but to teach readers how to do the work. From Part 2 onward the book adopts a tutorial style, teaching the reader numerous tools for obtaining and digging information out of huge and very timely datasets. Lee guides the reader through various data breaches, all of them leaked within the last five years: BlueLeaks, Oath Keepers email dumps, Heritage Foundation, Parler, Epik, and Cadence Health. He guides us through a tutorial on using the command line (mostly targeted at Linux, but considering MacOS and Windows as well), running Docker containers, learning the basics of Python, parsing and filtering structured data, writing small web applications for getting at the right bits of data, and working with structured query language (SQL) databases.

The book does an excellent job of fulfilling its very ambitious aims, and this is even more impressive given the wide range of professional profiles it is written for; that being said, I do have a couple critiques. First, the book is ideologically loaded: the datasets all exhibit the alt-right movement that has gained strength in the last decade. Lee takes the reader through many instances of COVID deniers, rioters for Donald Trump during the January 2021 attempted coup, attacks against Black Lives Matter activists, and other extremism research; thus this book could alienate right-wing researchers, who might also be involved in handling important whistleblowing cases.

Second, given the breadth of the topic and my 30-plus years of programming experience, I was very interested in the first part of each chapter but less so in the tutorial part. I suppose a journalist reading through the same text might find the sections about the importance of data handling and source protection to be similarly introductory. This is unavoidable, of course, given the nature of this work. However, while Micah Lee is an excellent example of a journalist with the appropriate technical know-how to process the types of material he presents as examples, expecting any one person to become a professional in both fields is asking too much.

All in all, this book is excellent. The writing style is informal and easy to read, the examples are engaging, and the analysis is very good. It will certainly teach you something, no matter your background, and it might very well complement your professional skills.

09 May, 2024 04:24AM

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppArmadillo 0.12.8.3.0 on CRAN: Upstream Bugfix

armadillo image

Armadillo is a powerful and expressive C++ template library for linear algebra and scientific computing. It aims towards a good balance between speed and ease of use, has a syntax deliberately close to Matlab, and is useful for algorithm development directly in C++, or quick conversion of research code into production environments. RcppArmadillo integrates this library with the R environment and language–and is widely used by (currently) 1144 other packages on CRAN, downloaded 34.2 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint / vignette) by Conrad and myself has been cited 583 times according to Google Scholar.

Conrad released a new upstream bugfix yesterday (for a corner case with fftw3). We uploaded it yesterday too but it took a day for the hard-working CRAN maintainers to concur that the one (!) NOTE from reverse-dependency checking over 1100 packages was in a fact a false positve. And so it appeared on CRAN (very) early this morning. We also made a change removing a long-redundant setter for C++11 mode via the plugin. No other changes were made.

The set of changes since the last CRAN release follows.

Changes in RcppArmadillo version 0.12.8.3.0 (2024-05-07)

  • Upgraded to Armadillo release 12.8.3 (Cortisol Injector)

    • Fix issue in fft() and fft2() in multi-threaded contexts with FFTW3 enabled
  • No longer set C++11 for the Rcpp plugin as this standard has been the default by R for very long time now.

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.

If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

09 May, 2024 01:54AM

May 07, 2024

John Goerzen

Photographic comparison: Is the Kobo Libra Colour display worse than the Kobo Libra 2?

I’ve been using E Ink-based ereaders for quite a number of years now. I’ve had my Kobo Libra 2 for a few years, and was looking forward to the Kobo Libra Colour — the first color E Ink display in a mainstream ereader line.

I found the display to be a mixed bag; contrast seemed a lot worse on B&W images, and the device “backlight” (it’s not technically a “back” light) seemed to cause a particular contrast reduction in dark mode. I went searching for information on this. I found a lot of videos on “Kobo Libra 2 vs Libra Colour” and so forth, but they were all pretty much useless. These were the mistakes they made:

  • Being videos. Photos would show the differences in better detail.
  • Shooting videos with cameras with automatic light levels. Since the thing we’re trying to evaluate here is how much darker the Kobo Libra Colour screen is than the Kobo Libra screen, having a camera that automatically adjusts for brighter or darker images defeats the purpose. Cell phone cameras (still and video) all do this by default and I saw evidence of it in all the videos.
  • Placing the two devices side-by-side instead of in identical locations for subsequent shots. This led to different shadows on each device (because OF COURSE the people shooting videos had to have their phone and head between the light source and the device), again preventing a good comparison.

So I dug out my Canon DSLR, tripod, and set up shots. Every shot here is set at ISO 100. Every shot in the same setting has the same exposure settings, which I document. The one thing I forgot to shut off was automatic white balance; you can notice it is active if you look closely at the backgrounds, but WB isn’t really relevant to this comparison anyhow.

Because there has also been a lot of concern about how well fine B&W details will show up on the Kobo Libra Colour screen, I shot all photos using a PDF test image from the open source hplip package (testpage.ps.gz converted to PDF). This also rules out font differences between the devices. I ensured a full screen refresh before each shot.

This is all because color E Ink is effectively a filter called Kaleido over the B&W layer. This causes dimming and some other visual effects.

You can click on any image here to see a full-resolution view. The full-size images are the exact JPEG coming from the camera, with only two modifications: 1) metadata has been redacted for privacy reasons, and 2) some images were losslessly rotated after the shoot.

OK, onwards!

Outdoors, bright sun, shot from directly overhead

Bright sun is ideal lighting for an E Ink display. They need no lighting at all in this scenario, and in fact, if you turn on their internal display light, it will probably not be very noticeable. Of course, this is in contrast to phone LCD screens, for which bright sunlight is the worst.

Scene: Morning sunlight reaching the ereaders at an angle. The angle was sufficient so that no shadows were cast by the camera or tripod.

Device light: Off on both

Exposure: 1/160, f16, ISO 100

You can see how much darker the Libra Colour is here. Though in these bright conditions, it is still plenty bright. There may actually be situations in which the Libra 2 is too bright in direct sunlight, requiring a person to squint or whatnot.

Looking at the radial lines, it is a bit difficult to tell because the difference in brightness, but I don’t see a hugely obvious reduction in quality in the Libra 2. Later I have a shot where I try to match brightness, and we’ll check it out again there.

Outdoors, shade, shot from directly overhead

For the next shot, I set the ereaders in shade, but still well-lit with the diffuse sunlight from all around.

The first two have both device lights off. For the third, I set the device light on the Kobo Colour to 100%, full cool shade, to try to see how close I could get it to the Libra 2 brightness. (Sorry it looks like I forgot to close the toolbar on the Colour for this set, but it doesn’t modify the important bits of the underlying image.)

Device light: Initially off on both

Exposure: 1/60, f6.4, ISO 100

Here you can see the light on the Libra Colour was nearly able to match the brightness on the Libra 2.

Indoors, room lit with overhead and window light, device light off

We continue to move into dimmer light with this next shot.

Device light: Off on both

Exposure: 1/4, f5, ISO 100

Indoors, room lit with overhead and window light, device light on

Now we have the first head-to-head with the device light on. I set the Libra 2 to my favorite warmth setting, found a brightness that looked good, and then tried my best to match those settings on the Libra Colour. My camera’s light meter aided in matching brightness.

Device light: On (Libra 2 at 40%, Libra Colour at 59%)

Exposure: 1/8, f5, ISO 100

(Apparently I am terrible at remembering to dismiss menus, sigh.)

Indoors, dark room, dark mode, at an angle

The Kobo Libra Colour surprised me with its dark mode. When viewed at an oblique angle, the screen gets pretty washed out. I maintained the same brightness settings here as I did above. It is much more noticeable when the brightness is set down to my preferred nighttime level (4%), or with a more significant angle.

Since you can’t see my tags, the order of the photos here will be: Libra 2 (standard orientation), Colour (standard orientation), Colour (turned around.

Device light: On (as above)

Exposure: 1/4, f5.6, ISO 100

Notice how I said I maintained the same brightness settings as before, and yet the Libra Colour looks brighter than the Libra 2 here, whereas it looked the same in the prior non-dark mode photos. Here’s why. I set the exposure of each set of shots based on camera metering. As we have seen from the light-off photos, the brightness of a white pixel is a lot less on a Libra Colour than on the Libra 2. However, it is likely that the brightness of a black pixel is about that same. Therefore, contrast on the Libra Colour is lower than on the Libra 2. The traditional shot is majority white pixels, so to make the Libra Colour brightness match that of the Libra 2, I had to crank up the brightness on the Libra Colour to compensate for the darker “white” background. With me so far?

Now with the inverted image, you can see what that does. It doesn’t just raise the brightness of the white pixels, but it also raises the brightness of the black pixels. This is expected because we didn’t raise contrast, only brightness.

Also, in the last image, you can see it is brighter to the right. Again, other conditions that are more difficult to photograph make that much more pronounced. Viewing the Libra Colour from one side (but not the other), in dark mode, with the light on, produces noticeably worse contrast on one side.

Conclusions

This isn’t a slam dunk. Let’s walk through this:

I don’t think there is any noticeable loss of detail on the Libra Colour. The radial lines appeared as well defined on it as on the Libra 2. Oddly, with the backlight, some striations were apparent in the gray gradient test, but I wouldn’t be using an E Ink device for clear photographic reproduction anyhow.

If you read mostly black and white: If you had been using a Kobo Libra Colour and were handed a Libra 2, you would go, “Wow! What an upgrade! The screen is so much brighter!” There’s little reason to get a Libra Colour. The Libra 2 might be hard to find these days, but the new Clara BW (with a 6″ instead of the 7″ screen on the Libra series) might be just the thing for you. The Libra 2 is at home in any lighting, from direct sun to pitch black, and has all the usual E Ink benefits (eg, battery life measured in weeks) and drawbacks (slower refresh rate) that we’re all used to.

If you are interested in photographic color reproduction mostly indoors: Consider a small tablet. The Libra Colour’s 4096 colors are going to appear washed out compared to what you’re used to on a LCD screen.

If you are interested in color content indoors and out: The Libra Colour might be a good fit. It could work well for things where superb color rendition isn’t essential — for instance, news stories (the Pocket integration or Calibre’s news feature could be nice there), comics, etc.

In a moderately-lit indoor room, it looks like the Libra Colour’s light can lead it to results that approach Libra 2 quality. So if most of your reading is in those conditions, perhaps the Libra Colour is right for you.

As a final aside, I wrote in this article about the Kobo devices. I switched from Kindles to Kobos a couple of years ago due to the greater openness of the Kobo devices (you can add things like Nickel Menu and KOReader to them, and they have built-in support for more useful formats), their featureset, and their cost. The top-of-the-line Kindle devices will have a screen very similar if not identical to the Libra 2, so you can very easily consider this to be a comparison between the Oasis and the Libra Colour as well.

07 May, 2024 10:16PM by John Goerzen

hackergotchi for Thomas Lange

Thomas Lange

Removing tens of thousands of web pages

In January I've removed tens of thousands of web pages on www.debian.org. Have you noticed it?

In the past

From 1997 onwards, we had web pages for security announcements. We had to manually prepare a .data and a .wml file which then generated a web page for each security announcement (DSA or DLA). We have listed the 6 most recent messages in a short list that was created from these files. Most of the work that went into the Debian web pages was creating these files.

Our search engine often listed the pages with security announcements instead of a more relevant web page for a particular topic.

Preparation

At DebConf Kosovo (2022) I started with a proof of concept and wrote a script, that generates this list without using the .data/.wml files in the Git repository, but instead reading the primary sources of security information[1]. This new list now includes links to the security tracker and the email of the announcement.

Following web pages and scripts were also using these .data and .wml files:

  • OVAL files
  • RSS feeds for security announcements (and LTS)
  • Apache config file for mapping URLs from dsa-NNN to YEAR/dsa-NNN
  • A huge list of crossreferences between DSA and CVE numbers

Before I could remove all the security web pages, I had to adjust the scripts, that create the above information.

When I looked at the OVAL files and the apache logs of our web server, I saw that more than 99% of the web traffic was generated by these XML files (134TB of 135TB total in two weeks). They were not compressed and were around 50MB in size. With the help of Carsten Schönert we managed to modify the python scripts that generate this OVAL file without using the .data/.wml files and now we only provide bzip2 compressed XML files[2].

The RSS feeds are created by the new Perl script which reads the DSA/DLA list the security tracker and determines the URL of the email of all entries. This script also generates the list of the most recent DSA/DLA entries. Currently we show the last 350 entries which covers more than the last year and includes links to the announcement email and the security tracker.

The huge list of crossreferences is not needed any more, since the mapping of CVE to DSA is already included in the DSA list[3] of the security tracker.

The amount of translations of the DSA/DLA was very different. French translations were almost all done, but all other languages did translations for a couple of months or years only. E.g. in 2022, Italian had 2 translations, Russian 15, Danish 212, French and English each 279. But from 2023 on only French translations were made. By generating the list of DSA/DLA we lost the ability to translate these web pages, but since these announcements are made of simple, identical sentences it is easy to use an automatic translation service if needed.

Now the translation statistics of all web pages are more accurate. Instead of 12200 pages that need to be translated (including all these old DSA/DLA) there are now only 2500 pages to translate[4]. Languages that had a lot of old translations of DSA/DLA lost some percentage but languages that are doing translations of newer web pages won in the statistics of how many pages are translated. Examples:

Before

German (de)   3501  28.5%
Italian (it)  1005   8.2%
Danish (da)   6336  51.7%

After

German (de)   1486  59.0%
Italian (it)   909  36.1%
Danish (da)    982  39.0%

Cleanup of all the security web pages

Finally in January, I could remove all web pages of the security announcements in one git commit[5]. Using several git rm -rf commands this commit removed 54335 files, including around 9650 DSA/DLA data files, 44189 wml files, nearly 500 Makefiles.

Outcome

No more manual work is needed for the security team and we now have direct links from a DSA-NNN/DLA-NNN to the email in our mailing list archive. This was not possible before. The search results became more accurate.

But we still host a lot of other old content on the Debian web pages which may be removed in the future.

[1] https://www.debian.org/security/#infos

[2] https://www.debian.org/security/oval/

[3] https://salsa.debian.org/security-tracker-team/security-tracker/-/raw/master/data/DSA/list

[4] https://www.debian.org/devel/website/stats

[5] https://salsa.debian.org/webmaster-team/webwml/-/commit/2aa73ff15bfc4eb2afd85c

07 May, 2024 10:08AM

May 05, 2024

hackergotchi for Junichi Uekawa

Junichi Uekawa

Been very busy with real life.

Been very busy with real life. Hardly any time to get things done.

05 May, 2024 10:00AM by Junichi Uekawa

May 04, 2024

Sven Hoexter

vym - view your mind

Had a need for a mindmapping application and found view your mind in the archive. Works but the version is a bit rusty. Sadly my Debian packaging skills are a bit rusty as well, especially when it comes to bigger GUI applications. Thus I spent a good chunk of yesterday afternoon to rip out cdbs and package the last source release on github which is right now 2.9.22 (the release branch already has 2.9.27, still sorting that out).

Git repository and a amd64 build of the current state. It still deserves some additional love, e.g. creating a -common package for arch indep content.

Proposed a few changes upstream:

Also pinged pollux@ who uploaded vym up to 2019 if he'd be fine if I pick it up. If someone else is interested, I'm also fine to put it up on salsa in the general "Debian" group for shared maintenance. I guess I will use it in the future, but time is still a scarce resource for all of us.

04 May, 2024 03:03PM

hackergotchi for Steve Kemp

Steve Kemp

The CP/M emulator development continues

In my previous post I introduced a toy CP/M Emulator I'd been working on.

At the time it was capable of running the Infocom text-based adventure games, so I thought it was done. Of course I also wanted to run Microsoft's original BASIC and it turned out that was a challenge because the coding of their interpreter didn't use the standard CP/M entry-point for making syscalls (call 0x0005).

Instead of calling 0x0005 to invoke the BDOS/BIOS functions the BASIC interpreter used the single-byte CALL instructions which are available on the Z80 processor. There are a bunch of these instructions:

  • RST 00
  • RST 08
  • RST 10
  • RST 18
  • RST 20
  • RST 28
  • RST 30
  • RST 38

Each of those instructions is equivalent to a call instruction with a fixed offset, "call 0x0010", "call 0x0020", etc. I had to rework the emulator to cope with this approach, which causes repetition but nothing too surprising. The end result is that now my emulator can run Microsoft Basic, Tasty Basic, and some more programs.

Things work but a couple of the syscalls are of the form "Return true if there is a pending keystroke", or "wait until there is keyboard input present and return the first character". I have some busy-loops which peg the CPU, which sucks but works. On the downside running the code on a MacOS machine has some weird issues with repeated keys and similar. So I need to look into fixing that for my own sense of peace.

I put together a little repository of binaries for playing with though, and that's been helpful. My emulator has a special flag which treats sub-directories as "Drives". So A: points to A/, B: points to B/, etc. That makes distributing and working with things easy!

04 May, 2024 11:51AM

May 03, 2024

hackergotchi for Colin Watson

Colin Watson

Playing with rich

One of the things I do as a side project for Freexian is to work on various bits of business automation: accounting tools, programs to help contributors report their hours, invoicing, that kind of thing. While it’s not quite my usual beat, this makes quite a good side project as the tools involved are mostly rather sensible and easy to deal with (Python, git, ledger, that sort of thing) and it’s the kind of thing where I can dip into it for a day or so a week and feel like I’m making useful contributions. The logic can be quite complex, but there’s very little friction in the tools themselves.

A recent case where I did run into some friction in the tools was with some commands that need to present small amounts of tabular data on the terminal, using OSC 8 hyperlinks if the terminal supports them: think customer-related information with some links to issues. One of my colleagues had previously done this using a hack on top of texttable, which was perfectly fine as far as it went. However, now I wanted to be able to add multiple links in a single table cell in some cases, and that was really going to stretch the limits of that approach: working out the width of the displayed text in the cell was going to take an annoying amount of bookkeeping.

I started looking around to see whether any other approaches might be easier, without too much effort (remember that “a day or so a week” bit above). ansiwrap looked somewhat promising, but it isn’t currently packaged in Debian, and it would have still left me with the problem of figuring out how to integrate it into texttable, which looked like it would be quite complicated. Then I remembered that I’d heard good things about rich, and thought I’d take a look.

rich turned out to be exactly what I wanted. Instead of something like this based on the texttable hack above:

import shutil
from pyxian.texttable import UrlTable

termsize = shutil.get_terminal_size((80, 25))
table = UrlTable(max_width=termsize.columns)
table.set_deco(UrlTable.HEADER)
table.set_cols_align(["l"])
table.set_cols_dtype(["u"])
table.add_row(["Issue"])
table.add_row([(issue_url, f"#{issue_id}")]
print(table.draw())

… now I can do this instead:

import rich
from rich import box
from rich.table import Table

table = Table(box=box.SIMPLE)
table.add_column("Issue")
table.add_row(f"[link={issue_url}]#{issue_id}[/link]")
rich.print(table)

While this is a little shorter, the real bonus is that I can now just put multiple [link] tags in a single string, and it all just works. No ceremony. In fact, once the relevant bits of code passed type-checking (since the real code is a bit more complex than the samples above), it worked first time. It’s a pleasure to work with a library like that.

It looks like I’ve only barely scratched the surface of rich, but I expect I’ll reach for it more often now.

03 May, 2024 03:09PM by Colin Watson

May 02, 2024

hackergotchi for Sergio Talens-Oliag

Sergio Talens-Oliag

The Freaky Wall (Part 2: Initial Installation)

For the Freaky Wall I have installed a Debian GNU/kFreeBSD system using the installer with ZFS support announced on:

http://robertmh.wordpress.com/2010/09/06/debian-installer-with-zfs/

I used the mini.iso found on:

http://people.debian.org/~rmh/zfs/kfreebsd-amd64/monolithic/mini.iso

the 12th of October of 2010; as I had some problems and reported them to Robert is possible that the current image solves part of them.

Installation plan

I did a standard installation on a machine with two hard disks, but only used the first one from the installer.

The plan was to use ZFS with RAID-1, but current versions of grub do not support booting from a ZFS + RAID file system, so I had to use the same technique used for Linux for a long time; three partitions: a swap partition, a small /boot partition and a big partition for /; / and /boot were formated to use ZFS.

First reboot

After the installation the system failed to boot because of a bug when building the /boot/grub/grub.cfg (some paths were missing a //@ prefix); to be able to boot Iwe edited the config on the grub prompt and later fixed the file:

    --- grub.cfg.orig       2010-10-13 16:40:39.000000000 +0200
    +++ grub.cfg    2010-10-13 18:38:47.535436766 +0200
    @@ -64,7 +64,7 @@
            set root='(hd0,1)'
            search --no-floppy --fs-uuid --set a371979bb836d1fe
            echo                    'Loading kernel of FreeBSD 8.1-1-amd64 ...'
    -       kfreebsd                /kfreebsd-8.1-1-amd64.gz
    +       kfreebsd                //@/kfreebsd-8.1-1-amd64.gz
            insmod part_msdos
            insmod zfs
            set root='(hd0,3)'
    @@ -75,7 +75,7 @@
            insmod zfs
            set root='(hd0,1)'
            search --no-floppy --fs-uuid --set a371979bb836d1fe
    -       kfreebsd_module         /zfs/zpool.cache type=/boot/zfs/zpool.cache
    +       kfreebsd_module         //@/zfs/zpool.cache type=/boot/zfs/zpool.cache
            set kFreeBSD.vfs.root.mountfrom=zfs:dkfbf1-ad4s3
            set kFreeBSD.vfs.root.mountfrom.options=rw
     }

I haven't tested the installer since that day, but I believe that the current ZFS installer was fixed by Robert to deal with that problem.

Once the system was booted I had to fix a couple of things:

  • The keyboard configuration was wrong, but it was easy to fix the Debian Way:

    dpkg-reconfigure kbdcontrol
    
  • The /boot partition was mounted on /target/boot, as that was what was recorded on the ZFS file system; to fix it I executed the following commands:

     # zfs set mountpoint=/     dkfbf1-ad4s3
     # zfs set mountpoint=/boot dkfbf1-ad4s1
    

    Where dkfbf1-ad4s3 is the root file system and dkfbf1-ad4s1 is the original /boot.

    I reported that to Robert also and I believe it is fixed on the ZFS installer now.

Adjusting ZFS to do RAID-1

On the second disk I created the same partitions as the ones on the first disk using parted; the final result was:

    # parted -l     
    Model: ST3250620NS/3BKS (ide)
    Disk /dev/ad6: 250GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos

    Number  Start   End     Size    Type     File system  Flags
     1      1049kB  256MB   255MB   primary
     2      256MB   4256MB  4000MB  primary
     3      4256MB  250GB   246GB   primary

    Model: ST3250620NS/3BKS (ide)
    Disk /dev/ad4: 250GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos

    Number  Start   End     Size    Type     File system     Flags
      1      1049kB  256MB   255MB   primary
      2      256MB   4256MB  4000MB  primary  linux-swap(v1)
      3      4256MB  250GB   246GB   primary

To use the second partition of both disks as swap I added the following to /etc/fstab:

    /dev/ad4s2      none            swap    sw              0       0
    /dev/ad6s2      none            swap    sw              0       0

To configure the mirroring for the root file system I did the following:

    zpool attach dkfbf1-ad4s3 ad4s3 ad6s3

As the /boot can't work as a replica I adjusted it to make two copies of every file:

    zfs   set copies=2     dkfbf1-ad4s1

Leaving the second disk copy alone, although my plan is to configure it to hold a copy of the /boot partition synchronized with rsync each night.

After all those changes the system didn't boot, as the grub-pc generates a buggy /boot/grub/grub.cfg; the problem is on the /etc/grub.d/10_kfreebsd section:

    ### BEGIN /etc/grub.d/10_kfreebsd ###
    menuentry 'Debian GNU/kFreeBSD, with kFreeBSD 8.1-1-amd64' --class debian \
      --class gnu-kfreebsd --class gnu --class os {
          insmod part_msdos
          insmod zfs
          set root='(hd0,1)'
          search --no-floppy --fs-uuid --set a371979bb836d1fe
          echo 'Loading kernel of FreeBSD 8.1-1-amd64 ...'
          kfreebsd /kfreebsd-8.1-1-amd64.gz
          set kFreeBSD.vfs.root.mountfrom=unknown:/dev/ad4s3
          set kFreeBSD.vfs.root.mountfrom.options=rw
    }
    ### END /etc/grub.d/10_kfreebsd ###

To fix it there has to be a copy of the modules for ZFS on the boot partition (in my case I moved the /lib/modules directory to /boot and created a link on the root partition to the new directory):

    cd /boot
    mkdir lib
    mv /lib/modules lib
    cd /lib
    ln -s ../boot/lib/modules

And instead of fixing the /etc/grub.d/10_kfreebsd code I wrote a new script (/etc/grub.d/09_zfs_kfreebsd) that creates the right config for my current configuration on the grub.cfg file:

    #!/bin/sh
    prefix=/usr
    exec_prefix=${prefix}
    bindir=${exec_prefix}/bin
    libdir=${exec_prefix}/lib
    . ${libdir}/grub/grub-mkconfig_lib

    prepare_boot_cache="$(prepare_grub_to_access_device ${GRUB_DEVICE_BOOT} | sed -e "s/^/\t/")"
    kfreebsd_versions="$(ls /lib/modules/)"
    zfs_root_device="$(zfs list | awk '/\/$/ { print $1 }' | head -1)"

    for kversion in $kfreebsd_versions; do
      cat << EOF
    # Entry when using ZFS (we have issues with /etc/grub.d/10_kfreebsd)
    menuentry 'Debian GNU/kFreeBSD, with kFreeBSD $kversion and ZFS' --class debian --class gnu-kfreebsd --class gnu --class os {
    ${prepare_boot_cache}
          echo                    'Loading kernel of FreeBSD $kversion ...'
          kfreebsd                //@/kfreebsd-$kversion.gz
          kfreebsd_module_elf     //@/lib/modules/$kversion/opensolaris.ko
          kfreebsd_module_elf     //@/lib/modules/$kversion/zfs.ko
          kfreebsd_module         //@/zfs/zpool.cache type=/boot/zfs/zpool.cache
          set kFreeBSD.vfs.root.mountfrom=zfs:$zfs_root_device
          set kFreeBSD.vfs.root.mountfrom.options=rw
    }
    EOF
    done

I solved the problem this way to have a working solution that does not break with squeeze upgrades, assuming that a future grub-pc package will deal well with my config and I'll be able to remove this script, but I guess I'll have to install it from backports.

The entry generated by the script when called from update-grub will be similar to:

    ### BEGIN /etc/grub.d/09_zfs-kfreebsd ###
    # Entry when using ZFS (we have issues with /etc/grub.d/10_kfreebsd)
    menuentry 'Debian GNU/kFreeBSD, with kFreeBSD 8.1-1-amd64 @ ITI' --class debian --class gnu-kfreebsd --class gnu --class os {
          insmod part_msdos
          insmod zfs
          set root='(hd0,1)'
          search --no-floppy --fs-uuid --set a371979bb836d1fe
          echo                    'Loading kernel of FreeBSD 8.1-1-amd64 ...'
          kfreebsd                //@/kfreebsd-8.1-1-amd64.gz
          kfreebsd_module_elf     //@/lib/modules/8.1-1-amd64/opensolaris.ko
          kfreebsd_module_elf     //@/lib/modules/8.1-1-amd64/zfs.ko
          kfreebsd_module         //@/zfs/zpool.cache type=/boot/zfs/zpool.cache
          set kFreeBSD.vfs.root.mountfrom=zfs:dkfbf1-ad4s3
          set kFreeBSD.vfs.root.mountfrom.options=rw
    }
    ### END /etc/grub.d/10_iti-kfreebsd ###

And after rebooting the machine with this new configuration the system boots OK.

On my next post I'll continue explaining how to compile a kernel that supports the use of the OpenBSD Packet Filter and related technologies (CARP, pflog, etc.).

02 May, 2024 05:11PM

Retooling

I haven't blogged for a long time, but I've decided that I'm going to try to write again, at least about technical stuff.

My plan was to blog about the projects I've been working on lately, the main one being the setup of the latest version of Kolab with the systems we already have at work, but I'll do that on the next days.

Today I'm just going to make a list of the tools I use on a daily basis and my plans to start using additional ones in the near future.

Shells, Terminals and Text Editors

I do almost all my work on Z Shell sessions running inside tmux; for terminal emulation I use gnome-terminal on X, VX ConnectBot on Android systems and iTerm2 on Mac OS X.

For text editing I've been using Vim for a long time (even on Mobile devices) and while I'm aware I don't know half of the things it can do, what I know is good enough for my day to day needs.

In the past I also used Emacs as a programming editor and my main tool to write HTML, SGML and XML, but since I haven't really needed an IDE for a long time and I mainly use Lightweight Markup Languages I haven't used it for a long time (I briefly tried to use Org mode, but for some reason I ended up leaving it).

Documentation formats and tools

Since a long time ago I've been an advocate of Lightweight Markup Languages; I started to use LaTeX and Lout, then moved to SGML/XML formats (LinuxDoc and DocBook) and finally moved to plain text based formats.

I started using Wiki formats (parsewiki) and soon moved to reStructuredText; I also use other markup languages like Markdown (for this blog, aka ikiwiki) and tried MultiMarkdown to replace reStructuredText for general use, but as I never liked Markdown syntax I didn't liked an extended version of it.

While I've been using ReStructuredText for a long time, I recently found Asciidoctor and the Asciidoc format and I guess I'll be using it instead of rst whenever I can (I still need to try the slide backends and conversions to ODT, but if that works I guess I'll write all my new documents using Asciidoc).

Programming languages

I'm not a developer, but I read and patch a lot of free software code written on a lot of different programming languages (I wouldn't be able to write whole programs on most of them, but thanks to Stack Overflow I'm usually able to fix what I need).

Anyway, I'm able to program in some languages; I write a lot of shell scripts and I go for Python and C when I need something more complicated.

On the near future I plan to read about javascript programming and nodejs (I'll probably need it at work) and I already started looking at Haskell (I guess it was time to learn about functional programming and after reading about it, it looks like haskell is the way to go for me).

Version Control

For a long time I've been a Subversion user, at least for my own projects, but seems that everything has moved to git now and I finally started to use it (I even opened a github account) and plan to move all my personal subversion repositories at home and at work to git, including the move of all my debian packages from svn-buildpackage to git-buildpackage.

Further Reading

With the previous plans in mind, I've started reading a couple of interesting books:

Now I just need to get enough time to finish reading them ... ;)

02 May, 2024 05:11PM

The FreakyWall (Part 3: Packages)

In this post I'll describe the changes made to the kernel and some of the Squeeze packages for the Freaky Wall.

The plan is to submit whishlist bugs to the BTS on the hope of having all what is needed for this project available on Debian after the Squeeze release, as my feeling is that a freeze is not the right time to push this changes... ;)

I'm giving access here to all the changes made to the source packages, but if anyone wants the binary packages (amd64 only) send me an email and I'll give you the URL of an apt repository that contains all the modified packages (it's the one at work, that contains other modified packages) or, if there is interest, I can put them on people.debian.org.

Kernel

To be able to build the firewall we need a kFreeBSD kernel with some options not compiled on the version distributed with Debian.

To compile the kernel I've followed the procedure described on the following debian-bsd mailing list post:

http://lists.debian.org/debian-bsd/2010/09/msg00023.html

Basically I've done the following:

    apt-get build-dep kfreebsd-8
    apt-get source kfreebsd-8
    cd kfreebsd-8-8.1
    cat >> debian/arch/amd64/amd64.config << EOF

    # Add pflog, pfsync, ALTQ and CARP support
    # ----------------------------------------

    # http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/firewalls-pf.html
    device      pf
    device      pflog
    device      pfsync

    options         ALTQ
    options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
    options         ALTQ_RED        # Random Early Detection (RED)
    options         ALTQ_RIO        # RED In/Out
    options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
    options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
    options         ALTQ_NOPCC      # Required for SMP build

    # http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/carp.html
    device      carp
    EOF
    vi debian/changelog 
    dpkg-buildpackage -B -uc

Once the package was built I installed the new kernel package and rebooted the machine.

Utilities

To be able to use some utilities related to pf I have built patched versions of three packages:

  • freebsd-utils: I have included pflogd and ftp-proxy on the package and have added some options to allow the use of additional interface types with ifconfig (carp, pfsync, lagg, bridges, ...).

    There were a lot of changes needed:

    1. The pflogd and ftp-proxy Makefiles are missing from the original tarball, I modified the get-orig-source of the debian/rules, but I build my packages against the original tarball, with the extra files included directly on the package .diff.gz.

    2. The pflogd daemon needs a _pflogd user and group and a /var/run/_pflogd directory, so I've added the directory and the creation of the user and group to the package post-install.

    3. The ftp-proxy daemon uses the proxy user when dropping privileges and I've modified the code to make it use the /var/run/ftp-proxy directory for the chroot.

    4. Some C header files that are not available on -dev packages were missing and I put them on the packages' debian directory. I've created a debian/include directory and moved there the original debian/net directory and added the headers debian/include/net/if_bridgevar.h and debian/include/net/if_lagg.h to add bridging support to ifconfig and the debian/include/pcap-config.h and debian/include/pcap-int.h libpcap private headers to be able to compile the pflogd binary.

    As I'm not familiar with the way people changes code for kFreeBSD some of the patches are a little bit dirty, but at least things work; besides, probably I should also have had to include init.d scripts for pf, pflogd and ftp-proxy, but I have not done it at the package level yet, as what I got was enough to work with the tools.

    The debdiff against the freebsd-utils-8.1-2 source package is available here or here.

  • libpcap: A test had to be removed in order to be able to support pflog on the library; the debdiff against the libpcap-1.1.1-2 package is available here or here.

  • tcpdump: The package also had to be modified to include the code to print the pflogd entries on the pcap file; the debdiff against tcpdump_4.1.1-1 is available here or here.

On the next post I'll describe how I've configured the system, the network interfaces and the different utilities patched and compiled on this post.

02 May, 2024 05:11PM

Debian Signs @ DebConf 9

I have not blogged about until now, but I'm in Cáceres since last Thursday, where jordi and I came by car from Valencia to attend the DebConf 9.

I'm not doing any Debian work here, but I'm having a good time socializing meeting and talking with a lot of people and attending some of the talks and BOFs.

Yesterday we went to the Valle del Jerte for the Day Trip, where we did a short walk and got to Los Pilones, where we saw a lot of natural pools between the mountains where we swimmed (I loved a small cascade that was like a natural hydro massage system) and stayed there for a couple of hours until we had to go back to the bus to visit another natural pool on the village of Jerte, that time a big one built inside the river that was also very nice.

Anyway, what I wanted to say is that yesterday's Day Trip was another good example of how the Debian Project helps its users and developers; when we were walking back from the mountains to pick the bus we found Debian people on a cross road telling us what was the right way and after a little while we found a Debian Sign on the floor:

Day Trip Go

We followed the advice and we confirmed that it was a shortcut in our way down, obviously installed there by someone from the Debian Project.

02 May, 2024 05:11PM

hackergotchi for Rhonda D'Vine

Rhonda D'Vine

Puberty

I was musing about writing about this publicly. For the first time in all these years of writing pretty personal stuff about my feelings, my way of becoming more honest with myself and a more authentic person through that I was thinking about letting you in on this is a good idea.

You see, people have used information from my personal blog in the past, and tried to use it against me. Needless to say they failed with it, and it only showed their true face. So why does it feel different this time?

Thing is, I'm in the midst of my second puberty, and the hormones are kicking in in complete hardcore mode. And it doesn't help at all that there is trans antagonist crap from the past and also from the present popping up left and right at a pace and a concentrated amount that is hard to swallow on its own without the puberty.

Yes, I used to be able to take those things with a much more stable state. But every. Single. Of. These. Issues is draining all the energy out of myself. And even though I'm aware that I'm not the only one trying to fix all of those, even though for some spots I'm the only one doing the work, it's easier said than done that I don't have to fix the world, when the areas involved mean the world to me. Are areas that support me in so many ways. Are places that I need. And on top of that, the hormones are multiplying the energy drain of those.

So ... I know it's not that common. I know you are not used to a grown up person to go through puberty. But for god's sake. Don't make it harder than it has to be. I know it's hard to deal with a 46 year old teenager, so to say, I'm just trying to survive in this world of systematic oppression of trans people.

It would be nice to go for a week without having to cry your eyes out because another hostile event happened that directly affects your existence. The existence of trans lives aren't a matter of different opinions or different points of view, so don't treat it like that, if you want me to believe that you are a person able of empathy and basic respect.

Sidenote: Finishing to write this at this year's #36c3 is quite interesting because of the conference title: Resource Exhaution. Oh the irony.

/personal | permanent link | Comments: 14 | Flattr this

02 May, 2024 04:55PM by Rhonda

Paul Wise

FLOSS Activities April 2024

Focus

This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

Issues

Administration

  • Debian IRC: updated #debian-dpl access list for new DPL
  • Debian wiki: unblock IP addresses, approve accounts

Communication

Sponsors

The SWH work was sponsored. All other work was done on a volunteer basis.

02 May, 2024 08:16AM

hackergotchi for Bits from Debian

Bits from Debian

Bits from the DPL

Hi,

Keeping my promise for monthly bits, here's a quick snapshot of my first ten days as DPL.

Special thanks to Jonathan for an insightful introduction that left less room for questions. His introduction covered my first tasks like expense approval and CTTE member appointments thoroughly. Although I made a visible oversight by forgetting to exclude Simon McVittie from the list, whose term has ended , I'm committed to learning from this mistake. In future I'll prioritize thorough proofreading to ensure accuracy.

Part of my "work" was learning what channels I need to subscribe and adjust my .procmailrc and .muttrc took some time.

Recently I had my first press interview. I had to answer a couple of prepared questions for Business IT News. It seems journalists are always on the lookout for unique angles. When asked if humility is a new trait for DPLs, my response would be a resounding "No." In my experience, humility is a common quality among DPLs I've encountered, including Jonathan.

One of my top priorities is reaching out to all our dedicated and appointed teams, including those managing critical infrastructure. I've begun with the CTTE, Salsa Admins and Debian Snapshot. Everything appears to be in order with the CTTE team. I'm waiting for response from Salsa and Snapshot, which is fine given the recent contact.

I was pointed out to the fact that lintian is in an unfortunate state as Axel Beckert confirmed on the lintian maintainers list. It turns out that bug #1069745 of magics-python should not have been undetected for a long time if lintian bug #677078 would have been fixed. It seems obvious to me that lintian needs more work to fulfill its role as reliably policy checker to ensure our high level of packaging quality.

In any case thanks a lot to Axel who is doing his best but it seems urgent to me to find some more person-power for this task. Any volunteer to lend some helping hand in the lintian maintainers team?

On 2024-04-30 I gave my first talk "Bits from greenhorn DPL" online at MiniDebConf Brasil in Belo Horizonte. The Q&A afterwards stired some flavours of the question: "What can Debian Brasil do better?" My answer was always in a way: Given your great activity in now organising the fifth MiniDebConf you are doing pretty well and I have no additional hints for the moment.

Kind regards Andreas.

02 May, 2024 01:00AM by Andreas Tille

May 01, 2024

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppInt64 0.0.5 on CRAN: Minor Maintenance

The new-ish package RcppInt64 (announced last fall in this post, with three small updates following) arrived on CRAN yesterday as relase 0.0.5. RcppInt64 collects some of the previous conversions between 64-bit integer values in R and C++, and regroups them in a single package. It offers two interfaces: both a more standard as<>() converter from R values along with its companions wrap() to return to R, as well as more dedicated functions ‘from’ and ‘to’.

This release addresses an new nag from CRAN who no longer want us to use the ‘non-API’ header function SET_S4_OBJECT so a small change was made.

The brief NEWS entry follows:

Changes in version 0.0.5 (2024-04-30)

  • Minor refactoring of internal code to not rely on SET_S4_OBJECT.

Courtesy of my CRANberries, there is a diffstat report relative to the previous release. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

01 May, 2024 11:44PM

hackergotchi for Bits from Debian

Bits from Debian

Debian welcomes the 2024 GSOC contributors/students

GSoC logo

We are very excited to announce that Debian has selected seven contributors to work under mentorship on a variety of projects with us during the Google Summer of Code.

Here are the list of the projects, students, and details of the tasks to be performed.


Project: Android SDK Tools in Debian

  • Student: anuragxone

Deliverables of the project: Make the entire Android toolchain, Android Target Platform Framework, and SDK tools available in the Debian archives.


Project: Benchmarking Parallel Performance of Numerical MPI Packages

  • Student: Nikolaos

Deliverables of the project: Deliver an automated method for Debian maintainers to test selected numerical Debian packages for their parallel performance in clusters, in particular to catch performance regressions from updates, and to verify expected performance gains, such as Amdahl’s and Gufstafson’s law, from increased cluster resources.


Project: Debian MobCom

  • Student: Nathan D

Deliverables of the project: Update the outdated mobile packages and recreate aged packages due to new dependencies. Bring in more mobile communication tools by adding about 5 new packages.


Project: Improve support of the Rust coreutils in Debian

  • Student: Sreehari Prasad TM

Deliverables of the project: Make uutils behave more like GNU’s coreutils by improving compatibility with GNU coreutils test suit.


Project: Improve support of the Rust findutils in Debian

  • Student: hanbings

Deliverables of the project: A safer and more performant implementation of the GNU suite's xargs, find, locate and updatedb tools in rust.


Project: Expanding ROCm support within Debian and derivatives

  • Student: xuantengh

Deliverables of the project: Building, packaging, and uploading missing ROCm software into Debian repositories, starting with simple tools and progressing to high-level applications like PyTorch, with the final deliverables comprising a series of ROCm packages meeting community quality assurance standards.


Project: procps: Development of System Monitoring, Statistics and Information Tools in Rust

  • Student: Krysztal Huang

Deliverables of the project: Improve the usability of the entire Rust-based implementation of the procps utility on Linux.


Congratulations and welcome to all the contributors!

The Google Summer of Code program is possible in Debian thanks to the efforts of Debian Developers and Debian Contributors that dedicate part of their free time to mentor contributors and outreach tasks.

Join us and help extend Debian! You can follow the contributors' weekly reports on the debian-outreach mailing-list, chat with us on our IRC channel or reach out to the individual projects' team mailing lists.

01 May, 2024 09:56PM by Nilesh Patra

hackergotchi for Colin Watson

Colin Watson

Free software activity in April 2024

My Debian contributions this month were all sponsored by Freexian.

  • I’m trying to get back into bugs.debian.org administration, so I spent some time catching up on my owner@bugs.debian.org mailbox and answering a number of support requests there.
  • I fixed a regression I’d introduced last year where groff’s PDF output had invalid date headers, both upstream and in Debian.
  • I released man-db 2.12.1.
  • openssh:
    • I did a little more testing of Luca Boccassi’s modifications to upstream’s inline systemd notification patch.
    • I did an extensive review of some of the choices in Debian’s OpenSSH packaging, in light of last month’s xz-utils backdoor.
    • I fixed a build failure on ppc64el, forwarded upstream.
    • I proposed reducing shared library linkage in tcp-wrappers; its maintainer accepted this by disabling NIS support.
    • I applied a suggestion to improve ordering of systemd services in relation to nss-user-lookup.target.
  • I updated putty to 0.81.
  • Python team:
  • I did some inconclusive investigation of flaky tests in gcr4. More work is needed there.
  • I proposed a patch for a build failure in gyoto, both upstream and in Debian.

You can support my work directly via Liberapay.

01 May, 2024 11:34AM by Colin Watson

hackergotchi for Bits from Debian

Bits from Debian

Infomaniak First Platinum Sponsor of DebConf24

infomaniaklogo

We are pleased to announce that Infomaniak has committed to sponsor DebConf24 as a Platinum Sponsor.

Infomaniak is an independent cloud service provider recognised throughout Europe for its commitment to privacy, the local economy and the environment. Recording growth of 18% in 2023, the company is developing a suite of online collaborative tools and cloud hosting, streaming, marketing and events solutions.

Infomaniak uses exclusively renewable energy, builds its own data centers and develops its solutions in Switzerland at the heart of Europe, without relocating. The company powers the website of the Belgian radio and TV service (RTBF) and provides streaming for more than 3,000 TV and radio stations in Europe.

With this commitment as Platinum Sponsor, Infomaniak is contributing to the Debian annual Developers' conference, directly supporting the progress of Debian and Free Software. Infomaniak contributes to strengthen the community that collaborates on Debian projects from all around the world throughout all of the year.

Thank you very much, Infomaniak, for your support of DebConf24!

Become a sponsor too!

DebConf24 will take place from 28th July to 4th August 2024 in Busan, South Korea, and will be preceded by DebCamp, from 21st to 27th July 2024.

DebConf24 is accepting sponsors! Interested companies and organizations should contact the DebConf team through sponsors@debconf.org, or viisit the DebConf24 website at https://debconf24.debconf.org/sponsors/become-a-sponsor/.

01 May, 2024 10:08AM by Sahil Dhiman

hackergotchi for Guido Günther

Guido Günther

Free Software Activities April 2024

A short status update of what happened on my side last month. Maintenance and code review keep to be the top time sinks (in a positive way).

If you want to support my work see donations.

01 May, 2024 06:10AM

Russ Allbery

Review: To Each This World

Review: To Each This World, by Julie E. Czerneda

Publisher: DAW
Copyright: November 2022
ISBN: 0-7564-1543-8
Format: Kindle
Pages: 676

To Each This World is a standalone science fiction novel.

Henry m'Yama t'Nowak is the Arbiter of New Earth. This is somewhat akin to a president, but only in very specific ways. Henry's job is to deal with the Kmet.

New Earth was settled by a slower-than-light colony ship from old Earth, our Earth. It is, so far as they know, the last of humanity in the universe. Origin Earth fell silent hundreds of years previous, before the colonists even landed. New Earth is now a carefully and thoughtfully managed world where humans survived, thrived, and at one point sent out six slower-than-light colony ships of its own. All were feared lost after a rushed launch due to a solar storm.

As this story opens, a probe from one of those ships arrives.

This is cause for rejoicing, but there are two small problems. The first is that the culture of New Earth has changed drastically since the days when they launched the Halcyon colony ships. New Earth is now part of the Duality, a new alliance with aliens painstakingly negotiated after their portal appeared in orbit. The Kmet were peaceful, eager to form an alliance and offer new technology, although they struggled with concepts such as individuality and insisted on interacting only with the Arbiter. Their technological gifts and the apparent loss of the Halcyon colony ships refocused New Earth on safety and caution. This unexpected message is a somewhat tricky political problem, a reminder of the path not taken.

The other small problem is that the reaction of the Kmet to this message is... dramatic.

This book has several problems, but the most serious is that it is simply too long. If you have read any other Czerneda novels, you know that she tends towards sprawling world-building, but usually there are enough twists and turns in the plot to keep the story moving while the protagonists slowly puzzle out the scientific mysteries. To Each This World is not sufficiently twisty for 676 pages. I think you could have cut half the novel without losing any major plot points.

The interesting parts of this book, to me, were figuring out what's going on with the Kmet, some of the political tensions within the New Earth government, and understanding what Henry and Pilot Killian's story had to do with the apparently-unrelated but intriguing interludes following Beth Seeker in a strange place called Doublet. All that stuff is in here, but it's alongside a whole lot of Henry wrestling with lifeboat ethics in situations where he thinks he needs to lie to and manipulate people for their own good. We also get several extended tours of societies that, while vaguely interesting in a science fiction world-building way, have essentially nothing to do with the plot.

We also get a whole lot of Henry's eagerly helpful AI polymorph Flip. I wanted to like this character, and I occasionally managed, but I felt like there was a constant mismatch between, in hindsight, how Czerneda meant for me to see Flip and what I thought she was signaling while I was reading. I wanted Flip to either be a fascinatingly weird companion or to be directly relevant to the plot, but instead there were hundreds of pages of unnerving creepiness mixed with obsequiousness and emotional neediness, all of which I think I read more into than Czerneda had intended. The overall experience was more exhausting than fun.

The core of the plot is solid, and if you like SF novels built around world-building and scientific mysteries, there's a lot here to enjoy. I think Czerneda's Species Imperative series (starting with Survival) is a better execution of some of the same ideas, but I liked that series a lot and was willing to read another take on it. Czerneda is one of the SF writers who takes biology seriously and is willing to write very alien aliens, and that leads to a few satisfying twists. Also, Beth Seeker is a great character (I wish we'd seen more of her), and Killian, while a bit generic, is a serviceable protagonist when Czerneda needs someone to go poke things with a stick.

Henry... I'm not sure what I think of Henry, and your enjoyment of this book may depend on how much you click with him.

Henry is a diplomat and an extrovert. His greatest joy and talent is talking to people, navigating political situations, and negotiating. Science fiction is full of protagonists who should be this character, but they rarely are this character, probably because a lot of writers are introverts. I think Czerneda deserves real credit for making her charismatic politician sufficiently accurate that his thought processes occasionally felt alien. For me, Henry was easiest to appreciate when Killian was the viewpoint protagonist and I could look at him through someone else's eyes, but Henry's viewpoint mostly worked as well. There's a lot of competence porn enjoyment in watching him do his thing.

The problem for me is that I thought several of his actions were unforgivably unethical, but no one in the book who matters seems to agree. I can see why he reached those unethical decisions, but they were profound violations of consent. He directly lies to people because he thinks telling the truth would be too risky and not get them to do what he wants them to do, and Czerneda sets up the story to imply that he might be right.

This is not necessarily a bad choice in a novel, but the author has to do some work to bring me along, and Czerneda didn't do enough of that work. I kept wanting there to be some twist or sting or complication that forced Henry to come to terms with what he was doing, but it never happens. He has to pick between two moral principles that I consider rather finely balanced, if not tilted in the opposite direction that he does, and he treats one principle as inviolable and the other as mostly unimportant. The plans he makes on that basis work fine, and those on the other side of that decision are never heard from again. It left a bad taste in my mouth, particularly given how much of the book is built around Henry making tough, tricky decisions under pressure.

I don't know about this book. I have a lot of mixed feelings. Parts of it I quite enjoyed. Parts of it I mostly enjoyed but wish were much less dragged out. Parts of it frustrated or bored me. It's one of those books where the more I thought about it after reading it, the more the parts I disliked annoyed me.

If you like Czerneda's style of world-building and biology, and if you have more tolerance for Henry's decisions than I did, you may well like this, but read Species Imperative first. I should probably also warn that there is a lot of magical technology in this book that blatantly violates some core principles of physics. I have a high tolerance for that sort of thing, but if you don't, you're going to be grumbling.

Rating: 6 out of 10

01 May, 2024 03:39AM

hackergotchi for Matthew Palmer

Matthew Palmer

The Mediocre Programmer's Guide to Rust

Me: “Hi everyone, my name’s Matt, and I’m a mediocre programmer.”

Everyone: “Hi, Matt.”

Facilitator: “Are you an alcoholic, Matt?”

Me: “No, not since I stopped reading Twitter.”

Facilitator: “Then I think you’re in the wrong room.”

Yep, that’s my little secret – I’m a mediocre programmer. The definition of the word “hacker” I most closely align with is “someone who makes furniture with an axe”. I write simple, straightforward code because trying to understand complexity makes my head hurt.

Which is why I’ve always avoided the more “academic” languages, like OCaml, Haskell, Clojure, and so on. I know they’re good languages – people far smarter than me are building amazing things with them – but the time I hear the word “endofunctor”, I’ve lost all focus (and most of my will to live). My preferred languages are the ones that come with less intellectual overhead, like C, PHP, Python, and Ruby.

So it’s interesting that I’ve embraced Rust with significant vigour. It’s by far the most “complicated” language that I feel at least vaguely comfortable with using “in anger”. Part of that is that I’ve managed to assemble a set of principles that allow me to almost completely avoid arguing with Rust’s dreaded borrow checker, lifetimes, and all the rest of the dark, scary corners of the language. It’s also, I think, that Rust helps me to write better software, and I can feel it helping me (almost) all of the time.

In the spirit of helping my fellow mediocre programmers to embrace Rust, I present the principles I’ve assembled so far.

Neither a Borrower Nor a Lender Be

If you know anything about Rust, you probably know about the dreaded “borrow checker”. It’s the thing that makes sure you don’t have two pieces of code trying to modify the same data at the same time, or using a value when it’s no longer valid.

While Rust’s borrowing semantics allow excellent performance without compromising safety, for us mediocre programmers it gets very complicated, very quickly. So, the moment the compiler wants to start talking about “explicit lifetimes”, I shut it up by just using “owned” values instead.

It’s not that I never borrow anything; I have some situations that I know are “borrow-safe” for the mediocre programmer (I’ll cover those later). But any time I’m not sure how things will pan out, I’ll go straight for an owned value.

For example, if I need to store some text in a struct or enum, it’s going straight into a String. I’m not going to start thinking about lifetimes and &'a str; I’ll leave that for smarter people. Similarly, if I need a list of things, it’s a Vec<T> every time – no &'b [T] in my structs, thank you very much.

Attack of the Clones

Following on from the above, I’ve come to not be afraid of .clone(). I scatter them around my code like seeds in a field. Life’s too short to spend time trying to figure out who’s borrowing what from whom, if I can just give everyone their own thing.

There are warnings in the Rust book (and everywhere else) about how a clone can be “expensive”. While it’s true that, yes, making clones of data structures consumes CPU cycles and memory, it very rarely matters. CPU cycles are (usually) plentiful and RAM (usually) relatively cheap. Mediocre programmer mental effort is expensive, and not to be spent on premature optimisation. Also, if you’re coming from most any other modern language, Rust is already giving you so much more performance that you’re probably ending up ahead of the game, even if you .clone() everything in sight.

If, by some miracle, something I write gets so popular that the “expense” of all those spurious clones becomes a problem, it might make sense to pay someone much smarter than I to figure out how to make the program a zero-copy masterpiece of efficient code. Until then… clone early and clone often, I say!

Derive Macros are Powerful Magicks

If you start .clone()ing everywhere, pretty quickly you’ll be hit with this error:


error[E0599]: no method named `clone` found for struct `Foo` in the current scope

This is because not everything can be cloned, and so if you want your thing to be cloned, you need to implement the method yourself. Well… sort of.

One of the things that I find absolutely outstanding about Rust is the “derive macro”. These allow you to put a little marker on a struct or enum, and the compiler will write a bunch of code for you! Clone is one of the available so-called “derivable traits”, so you add #[derive(Clone)] to your structs, and poof! you can .clone() to your heart’s content.

But there are other things that are commonly useful, and so I’ve got a set of traits that basically all of my data structures derive:


#[derive(Clone, Debug, Default)]
struct Foo {
    // ...
}

Every time I write a struct or enum definition, that line #[derive(Clone, Debug, Default)] goes at the top.

The Debug trait allows you to print a “debug” representation of the data structure, either with the dbg!() macro, or via the {:?} format in the format!() macro (and anywhere else that takes a format string). Being able to say “what exactly is that?” comes in handy so often, not having a Debug implementation is like programming with one arm tied behind your Aeron.

Meanwhile, the Default trait lets you create an “empty” instance of your data structure, with all of the fields set to their own default values. This only works if all the fields themselves implement Default, but a lot of standard types do, so it’s rare that you’ll define a structure that can’t have an auto-derived Default. Enums are easily handled too, you just mark one variant as the default:


#[derive(Clone, Debug, Default)]
enum Bar {
    Something(String),
    SomethingElse(i32),
    #[default]   // <== mischief managed
    Nothing,
}

Borrowing is OK, Sometimes

While I previously said that I like and usually use owned values, there are a few situations where I know I can borrow without angering the borrow checker gods, and so I’m comfortable doing it.

The first is when I need to pass a value into a function that only needs to take a little look at the value to decide what to do. For example, if I want to know whether any values in a Vec<u32> are even, I could pass in a Vec, like this:


fn main() {
    let numbers = vec![0u32, 1, 2, 3, 4, 5];

    if has_evens(numbers) {
        println!("EVENS!");
    }
}

fn has_evens(numbers: Vec<u32>) -> bool {
    numbers.iter().any(|n| n % 2 == 0)
}

Howver, this gets ugly if I’m going to use numbers later, like this:


fn main() {
    let numbers = vec![0u32, 1, 2, 3, 4, 5];

    if has_evens(numbers) {
        println!("EVENS!");
    }

    // Compiler complains about "value borrowed here after move"
    println!("Sum: {}", numbers.iter().sum::<u32>());
}

fn has_evens(numbers: Vec<u32>) -> bool {
    numbers.iter().any(|n| n % 2 == 0)
}

Helpfully, the compiler will suggest I use my old standby, .clone(), to fix this problem. But I know that the borrow checker won’t have a problem with lending that Vec<u32> into has_evens() as a borrowed slice, &[u32], like this:


fn main() {
    let numbers = vec![0u32, 1, 2, 3, 4, 5];

    if has_evens(&numbers) {
        println!("EVENS!");
    }
}

fn has_evens(numbers: &[u32]) -> bool {
    numbers.iter().any(|n| n % 2 == 0)
}

The general rule I’ve got is that if I can take advantage of lifetime elision (a fancy term meaning “the compiler can figure it out”), I’m probably OK. In less fancy terms, as long as the compiler doesn’t tell me to put 'a anywhere, I’m in the green. On the other hand, the moment the compiler starts using the words “explicit lifetime”, I nope the heck out of there and start cloning everything in sight.

Another example of using lifetime elision is when I’m returning the value of a field from a struct or enum. In that case, I can usually get away with returning a borrowed value, knowing that the caller will probably just be taking a peek at that value, and throwing it away before the struct itself goes out of scope. For example:


struct Foo {
    id: u32,
    desc: String,
}

impl Foo {
    fn description(&self) -> &str {
        &self.desc
    }
}

Returning a reference from a function is practically always a mortal sin for mediocre programmers, but returning one from a struct method is often OK. In the rare case that the caller does want the reference I return to live for longer, they can always turn it into an owned value themselves, by calling .to_owned().

Avoid the String Tangle

Rust has a couple of different types for representing strings – String and &str being the ones you see most often. There are good reasons for this, however it complicates method signatures when you just want to take some sort of “bunch of text”, and don’t care so much about the messy details.

For example, let’s say we have a function that wants to see if the length of the string is even. Using the logic that since we’re just taking a peek at the value passed in, our function might take a string reference, &str, like this:


fn is_even_length(s: &str) -> bool {
    s.len() % 2 == 0
}

That seems to work fine, until someone wants to check a formatted string:


fn main() {
    // The compiler complains about "expected `&str`, found `String`"
    if is_even_length(format!("my string is {}", std::env::args().next().unwrap())) {
        println!("Even length string");
    }
}

Since format! returns an owned string, String, rather than a string reference, &str, we’ve got a problem. Of course, it’s straightforward to turn the String from format!() into a &str (just prefix it with an &). But as mediocre programmers, we can’t be expected to remember which sort of string all our functions take and add & wherever it’s needed, and having to fix everything when the compiler complains is tedious.

The converse can also happen: a method that wants an owned String, and we’ve got a &str (say, because we’re passing in a string literal, like "Hello, world!"). In this case, we need to use one of the plethora of available “turn this into a String” mechanisms (.to_string(), .to_owned(), String::from(), and probably a few others I’ve forgotten), on the value before we pass it in, which gets ugly real fast.

For these reasons, I never take a String or an &str as an argument. Instead, I use the Power of Traits to let callers pass in anything that is, or can be turned into, a string. Let us have some examples.

First off, if I would normally use &str as the type, I instead use impl AsRef<str>:


fn is_even_length(s: impl AsRef<str>) -> bool {
    s.as_ref().len() % 2 == 0
}

Note that I had to throw in an extra as_ref() call in there, but now I can call this with either a String or a &str and get an answer.

Now, if I want to be given a String (presumably because I plan on taking ownership of the value, say because I’m creating a new instance of a struct with it), I use impl Into<String> as my type:


struct Foo {
    id: u32,
    desc: String,
}

impl Foo {
    fn new(id: u32, desc: impl Into<String>) -> Self {
        Self { id, desc: desc.into() }
    }
}

We have to call .into() on our desc argument, which makes the struct building a bit uglier, but I’d argue that’s a small price to pay for being able to call both Foo::new(1, "this is a thing") and Foo::new(2, format!("This is a thing named {name}")) without caring what sort of string is involved.

Always Have an Error Enum

Rust’s error handing mechanism (Results… everywhere), along with the quality-of-life sugar surrounding it (like the short-circuit operator, ?), is a delightfully ergonomic approach to error handling. To make life easy for mediocre programmers, I recommend starting every project with an Error enum, that derives thiserror::Error, and using that in every method and function that returns a Result.

How you structure your Error type from there is less cut-and-dried, but typically I’ll create a separate enum variant for each type of error I want to have a different description. With thiserror, it’s easy to then attach those descriptions:


#[derive(Clone, Debug, thiserror::Error)]
enum Error {
    #[error("{0} caught fire")]
    Combustion(String),
    #[error("{0} exploded")]
    Explosion(String),
}

I also implement functions to create each error variant, because that allows me to do the Into<String> trick, and can sometimes come in handy when creating errors from other places with .map_err() (more on that later). For example, the impl for the above Error would probably be:


impl Error {
    fn combustion(desc: impl Into<String>) -> Self {
        Self::Combustion(desc.into())
    }

    fn explosion(desc: impl Into<String>) -> Self {
        Self::Explosion(desc.into())
    }
}

It’s a tedious bit of boilerplate, and you can use the thiserror-ext crate’s thiserror_ext::Construct derive macro to do the hard work for you, if you like. It, too, knows all about the Into<String> trick.

Banish map_err (well, mostly)

The newer mediocre programmer, who is just dipping their toe in the water of Rust, might write file handling code that looks like this:


fn read_u32_from_file(name: impl AsRef<str>) -> Result<u32, Error> {
    let mut f = File::open(name.as_ref())
        .map_err(|e| Error::FileOpenError(name.as_ref().to_string(), e))?;

    let mut buf = vec![0u8; 30];
    f.read(&mut buf)
        .map_err(|e| Error::ReadError(e))?;

    String::from_utf8(buf)
        .map_err(|e| Error::EncodingError(e))?
        .parse::<u32>()
        .map_err(|e| Error::ParseError(e))
}

This works great (or it probably does, I haven’t actually tested it), but there are a lot of .map_err() calls in there. They take up over half the function, in fact. With the power of the From trait and the magic of the ? operator, we can make this a lot tidier.

First off, assume we’ve written boilerplate error creation functions (or used thiserror_ext::Construct to do it for us)). That allows us to simplify the file handling portion of the function a bit:


fn read_u32_from_file(name: impl AsRef<str>) -> Result<u32, Error> {
    let mut f = File::open(name.as_ref())
        // We've dropped the `.to_string()` out of here...
        .map_err(|e| Error::file_open_error(name.as_ref(), e))?;

    let mut buf = vec![0u8; 30];
    f.read(&mut buf)
        // ... and the explicit parameter passing out of here
        .map_err(Error::read_error)?;

    // ...

If that latter .map_err() call looks weird, without the |e| and such, it’s passing a function-as-closure, which just saves on a few characters typing. Just because we’re mediocre, doesn’t mean we’re not also lazy.

Next, if we implement the From trait for the other two errors, we can make the string-handling lines significantly cleaner. First, the trait impl:


impl From<std::string::FromUtf8Error> for Error {
    fn from(e: std::string::FromUtf8Error) -> Self {
        Self::EncodingError(e)
    }
}

impl From<std::num::ParseIntError> for Error {
    fn from(e: std::num::ParseIntError) -> Self {
        Self::ParseError(e)
    }
}

(Again, this is boilerplate that can be autogenerated, this time by adding a #[from] tag to the variants you want a From impl on, and thiserror will take care of it for you)

In any event, no matter how you get the From impls, once you have them, the string-handling code becomes practically error-handling-free:


    Ok(
        String::from_utf8(buf)?
            .parse::<u32>()?
    )

The ? operator will automatically convert the error from the types returned from each method into the return error type, using From. The only tiny downside to this is that the ? at the end strips the Result, and so we’ve got to wrap the returned value in Ok() to turn it back into a Result for returning. But I think that’s a small price to pay for the removal of those .map_err() calls.

In many cases, my coding process involves just putting a ? after every call that returns a Result, and adding a new Error variant whenever the compiler complains about not being able to convert some new error type. It’s practically zero effort – outstanding outcome for the mediocre programmer.

Just Because You’re Mediocre, Doesn’t Mean You Can’t Get Better

To finish off, I’d like to point out that mediocrity doesn’t imply shoddy work, nor does it mean that you shouldn’t keep learning and improving your craft. One book that I’ve recently found extremely helpful is Effective Rust, by David Drysdale. The author has very kindly put it up to read online, but buying a (paper or ebook) copy would no doubt be appreciated.

The thing about this book, for me, is that it is very readable, even by us mediocre programmers. The sections are written in a way that really “clicked” with me. Some aspects of Rust that I’d had trouble understanding for a long time – such as lifetimes and the borrow checker, and particularly lifetime elision – actually made sense after I’d read the appropriate sections.

Finally, a Quick Beg

I’m currently subsisting on the kindness of strangers, so if you found something useful (or entertaining) in this post, why not buy me a refreshing beverage? It helps to know that people like what I’m doing, and helps keep me from having to sell my soul to a private equity firm.

01 May, 2024 12:00AM by Matt Palmer (mpalmer@hezmatt.org)

April 30, 2024

Russell Coker

April 28, 2024

USB PSUs

I just bought a new USB PSU from AliExpress [1]. I got this to reduce the clutter in my bedroom, I charge my laptop, PineTime, and a few phones at the same time and a single PSU with lots of ports makes it easier. Also I bought a couple of really short USB-C cables as it’s been proven by both real life tests and mathematical modelling that shorter cables get tangled less. This power supply is based on Gallium Nitride (GaN) [2] technology which makes it efficient and cool.

One thing I only learned about after that purchase is the new USB PPS standard (see the USB Wikipedia page for details [3]). The PPS (Programmable Power Supply) standard allows (quoting Wikipedia) “allowing a voltage range of 3.3 to 21 V in 20 mV steps, and a current specified in 50 mA steps, to facilitate constant-voltage and constant-current charging”. What this means in practice (when phones support it which for me will probably be 2029 or something) is that the phone could receive power exactly matching the voltage needed for the battery and not have any voltage conversion inside the phone. Phones are designed to stop charging at a certain temperature, this probably doesn’t concern people in places like Northern Europe but in Australia it can be an issue. Removing the heat dissipation from inefficiencies in voltage change circuitry means the phone will be cooler when charging and can charge at a higher rate.

There is a “Certified USB Fast Charger” logo for chargers which do this, but it seems that at the moment they just include “PPS” in the feature list. So I highly recommend that GaN and PPS be on your feature list for your next USB PSU, but failing that the 240W PSU I bought for $36 was a good deal.

28 April, 2024 10:02PM by etbe

hackergotchi for Evgeni Golov

Evgeni Golov

Running Ansible Molecule tests in parallel

Or "How I've halved the execution time of our tests by removing ten lines". Catchy, huh? Also not exactly true, but quite close. Enjoy!

Molecule?!

"Molecule project is designed to aid in the development and testing of Ansible roles."

No idea about the development part (I have vim and mkdir), but it's really good for integration testing. You can write different test scenarios where you define an environment (usually a container), a playbook for the execution and a playbook for verification. (And a lot more, but that's quite unimportant for now, so go read the docs if you want more details.)

If you ever used Beaker for Puppet integration testing, you'll feel right at home (once you've thrown away Ruby and DSLs and embraced YAML for everything).

I'd like to point out one thing, before we continue. Have another look at the quote above.

"Molecule project is designed to aid in the development and testing of Ansible roles."

That's right. The project was started in 2015 and was always about roles. There is nothing wrong about that, but given the Ansible world has moved on to collections (which can contain roles), you start facing challenges.

Challenges using Ansible Molecule in the Collections world

The biggest challenge didn't change since the last time I looked at the topic in 2020: running tests for multiple roles in a single repository ("monorepo") is tedious.

Well, guess what a collection is? Yepp, a repository with multiple roles in it.

It did get a bit better though. There is pytest-ansible now, which has integration for Molecule. This allows the execution of Molecule and even provides reasonable logging with something as short as:

% pytest --molecule roles/

That's much better than the shell script I used in 2020!

However, being able to execute tests is one thing. Being able to execute them fast is another one.

Given Molecule was initially designed with single roles in mind, it has switches to run all scenarios of a role (--all), but it has no way to run these in parallel. That's fine if you have one or two scenarios in your role repository. But what if you have 10 in your collection?

"No way?!" you say after quickly running molecule test --help, "But there is…"

% molecule test --help
Usage: molecule test [OPTIONS] [ANSIBLE_ARGS]...

  --parallel / --no-parallel      Enable or disable parallel mode. Default is disabled.

Yeah, that switch exists, but it only tells Molecule to place things in separate folders, you still need to parallelize yourself with GNU parallel or pytest.

And here our actual journey starts!

Running Ansible Molecule tests in parallel

To run Molecule via pytest in parallel, we can use pytest-xdist, which allows pytest to run the tests in multiple processes.

With that, our pytest call becomes something like this:

% MOLECULE_OPTS="--parallel" pytest --numprocesses auto --molecule roles/

What does that mean?

  • MOLECULE_OPTS passes random options to the Molecule call pytest does, and we need to add --parallel there.
  • --numprocesses auto tells pytest-xdist to create as many workers as you have CPUs and balance the work across those.

However, once we actually execute it, we see:

% MOLECULE_OPTS="--parallel" pytest --numprocesses auto --molecule roles/

WARNING  Driver podman does not provide a schema.
INFO     debian scenario test matrix: dependency, cleanup, destroy, syntax, create, prepare, converge, idempotence, side_effect, verify, cleanup, destroy
INFO     Performing prerun with role_name_check=0...
WARNING  Retrying execution failure 250 of: ansible-galaxy collection install -vvv --force ../..
ERROR    Command returned 250 code:

OSError: [Errno 39] Directory not empty: 'roles'

FileExistsError: [Errno 17] File exists: b'/home/user/namespace.collection/collections/ansible_collections/namespace/collection'

FileNotFoundError: [Errno 2] No such file or directory: b'/home/user/namespace.collection//collections/ansible_collections/namespace/collection/roles/my_role/molecule/debian/molecule.yml'

You might see other errors, other paths, etc, but they all will have one in common: they indicate that either files or directories are present, while the tool expects them not to be, or vice versa.

Ah yes, that fine smell of race conditions.

I'll spare you the wild-goose chase I went on when trying to find out what the heck was calling ansible-galaxy collection install here. Instead, I'll just point at the following line:

INFO     Performing prerun with role_name_check=0...

What is this "prerun" you ask? Well… "To help Ansible find used modules and roles, molecule will perform a prerun set of actions. These involve installing dependencies from requirements.yml specified at the project level, installing a standalone role or a collection."

Turns out, this step is not --parallel-safe (yet?).

Luckily, it can easily be disabled, for all our roles in the collection:

% mkdir -p .config/molecule
% echo 'prerun: false' >> .config/molecule/config.yml

This works perfectly, as long as you don't have any dependencies.

And we don't have any, right? We didn't define any in a molecule/collections.yml, our collection has none.

So let's push a PR with that and see what our CI thinks.

OSError: [Errno 39] Directory not empty: 'tests'

Huh?

FileExistsError: [Errno 17] File exists: b'remote.sh' -> b'/home/runner/work/namespace.collection/namespace.collection/collections/ansible_collections/ansible/posix/tests/utils/shippable/aix.sh'

What?

ansible_compat.errors.InvalidPrerequisiteError: Found collection at '/home/runner/work/namespace.collection/namespace.collection/collections/ansible_collections/ansible/posix' but missing MANIFEST.json, cannot get info.

Okay, okay, I get the idea… But why?

Well, our collection might not have any dependencies, BUT MOLECULE HAS! When using Docker containers, it uses community.docker, when using Podman containers.podman, etc…

So we have to install those before running Molecule, and everything should be fine. We even can use Molecule to do this!

$ molecule dependency --scenario <scenario>

And with that knowledge, the patch to enable parallel Molecule execution on GitHub Actions using pytest-xdist becomes:

diff --git a/.config/molecule/config.yml b/.config/molecule/config.yml
new file mode 100644
index 0000000..32ed66d
--- /dev/null
+++ b/.config/molecule/config.yml
@@ -0,0 +1 @@
+prerun: false
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 0f9da0d..df55a15 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -58,9 +58,13 @@ jobs:
       - name: Install Ansible
         run: pip install --upgrade https://github.com/ansible/ansible/archive/${{ matrix.ansible }}.tar.gz
       - name: Install dependencies
-        run: pip install molecule molecule-plugins pytest pytest-ansible
+        run: pip install molecule molecule-plugins pytest pytest-ansible pytest-xdist
+      - name: Install collection dependencies
+        run: cd roles/repository && molecule dependency -s suse
       - name: Run tests
-        run: pytest -vv --molecule roles/
+        run: pytest -vv --numprocesses auto --molecule roles/
+        env:
+          MOLECULE_OPTS: --parallel

   ansible-lint:
     runs-on: ubuntu-latest

But you promised us to delete ten lines, that's just a +7-2 patch!

Oh yeah, sorry, the +10-20 (so a net -10) is the foreman-operations-collection version of the patch, that also migrates from an ugly bash script to pytest-ansible.

And yes, that cuts down the execution from ~26 minutes to ~13 minutes.

In the collection I originally tested this with, it's a more moderate "from 8-9 minutes to 5-6 minutes", which is still good though :)

28 April, 2024 07:04PM by evgeni

Russell Coker

Galaxy Note 9 Droidian

Droidian Support for Note 9

Droidian only supported the version of this phone with the Exynos chipset. The GSM Arena specs page for the Note 9 shows that it’s the SM-N960F part number [1]. In Australia all Note 9 phones should have the Exynos but it doesn’t hurt to ask for the part number before buying.

The status of the Note9 in Droidian went from fully supported to totally unsupported in the time I was working on this blog post. Such a rapid change is disappointing, it would be good if they at least kept the old data online. It would also be good if they didn’t require a hash character in the URL for each phone which breaks the archive.org mirroring.

Installing Droidian

Firstly Power+VolumeDown will reboot in some situations where Power button on its own won’t. The Note 9 hardware keys are:

  • Power – Right side
  • Volume up/down – long button top of the left side
  • Bixby – key for Samsung assistant that’s below the volume on the left

The Droidian install document for the Galaxy Note 9 9 now deleted is a bit confusing and unclear. Here is the install process that worked for me.

  1. The doc says to start by installing “Android 10 (Q) stock firmware”, but apparently a version of Android 10 that’s already on the phone will do for that.
  2. Download the rescue.img file and the “Droidian’s image” files from the Droidian page and extract the “Droidian’s image” zip.
  3. Connect your phone to your workstation by USB, preferably USB 3 because it will take a few minutes to transfer the image at USB 2 speed. Install the Debian package adb on the workstation.
  4. To “Unlock the bootloader” you can apparently use a PC and the Samsung software but the unlock option in the Android settings gives the same result without proprietary software, here’s how to do it:
    1. Connect the phone to Wifi. Then in settings go to “Software update”, then click on “Download and install”. Refuse to install if it offers you a new version (the unlock menu item will never appear unless you do this, so you can’t unlock without Internet access).
    2. In settings go to “About phone”, then “Software information”, then tap on “Build number” repeatedly until “Developer mode” is enabled.
    3. In settings go to the new menu “Developer options” then turn on the “OEM unlocking” option, this does a factory reset of the phone.
  5. To flash the recovery.img you apparently use Odin on Windows. I used the heimdall-flash package on Debian. On your Linux workstation run the commands:
    adb reboot download
    heimdall flash --RECOVERY recovery.img

    Then press VOLUME-UP+BIXBY+POWER as soon as it reboots to get into the recovery image. If you don’t do it soon enough it will do a default Android boot which will wipe the recovery.img you installed and also do a factory reset which will disable “Developer mode” and you will need to go back to step 4.

  6. If the above step works correctly you will have a RECOVERY menu where the main menu has options “Reboot system now”, “Apply update”, “Factory reset”, and “Advanced” in a large font. If you failed to install recovery.img then you would get a similar menu but with a tiny font which is the Samsung recovery image which won’t work so reboot and try again.
  7. When at the main recovery menu select “Advanced” and then “Enter fastboot”. Note that this doesn’t run a different program or do anything obviously different, just gives a menu – that’s OK we want it at this menu.
  8. Run “./flash_all.sh” on your workstation.
  9. Then it should boot Droidian! This may take a bit of time.

First Tests

Battery

The battery and its charge and discharge rates are very important to me, it’s what made the PinePhonePro and Librem5 unusable as daily driver phones.

After running for about 100 minutes of which about 40 minutes were playing with various settings the phone was at 89% battery. The output of “upower -d” isn’t very accurate as it reported power use ranging from 0W to 25W! But this does suggest that the phone might last for 400 minutes of real use that’s not CPU intensive, such as reading email, document editing, and web browsing. I don’t think that 6.5 hours of doing such things non-stop without access to a power supply or portable battery is something I’m ever going to do. Samsung when advertising the phone claimed 17 hours of video playback which I don’t think I’m ever going to get – or want.

After running for 11 hours it was at 58% battery. Then after just over 21 hours of running it had 13% battery. Generally I don’t trust the upower output much but the fact that it ran for over 21 hours shows that its battery life is much better than the PinePhonePro and the Librem5. During that 21 hours I’ve had a ssh session open with the client set to send ssh keep-alive messages every minute. So it had to remain active. There is an option to suspend on Droidian but they recommend you don’t use it. There is no need for the “caffeine mode” that you have on Mobian. For comparison my previous tests suggested that when doing nothing a PinePhonePro might last for 30 hours on battery while the Liberem5 might only list 10 hours [2]. This test with Droidian was done with the phone within my reach for much of that time and subject to my desire to fiddle with new technology – so it wasn’t just sleeping all the time.

When charging from the USB port on my PC it went from 13% to 27% charge in half an hour and then after just over an hour it claimed to be at 33%. It ended up taking just over 7 hours to fully charge from empty that’s not great but not too bad for a PC USB port. This is the same USB port that my Librem5 couldn’t charge from. Also the discharge:charge ratio of 21:7 is better than I could get from the PinePhonePro with Caffeine mode enabled.

rndis0

The rndis0 interface used for IP over USB doesn’t work. Droidian bug #36 [3].

Other Hardware

The phone I bought for testing is the model with 6G of RAM and 128G of storage, has a minor screen crack and significant screen burn-in. It’s a good test system for $109. The screen burn-in is very obvious when running the default Android setup but when running the default Droidian GNOME setup set to the Dark theme (which is a significant power saving with an AMOLED screen) I can’t see it at all. Buying a cheap phone with screen burn-in is something I recommend.

The stylus doesn’t work, this isn’t listed on the Droidian web page. I’m not sure if I tested the stylus when the phone was running Android, I think I did.

D State Processes

I get a kernel panic early in the startup for unknown reasons and some D state kernel threads which may or may not be related to that. Droidian bug #37 [4].

Second Phone

The Phone

I ordered a second Note9 on ebay, it had been advertised at $240 for a month and the seller accepted my offer of $200. With postage that’s $215 for a Note9 in decent condition with 8G of RAM and 512G of storage. But Droidian dropped support for the Note9 before I got to install it. At the moment I’m not sure what I’ll do with this, maybe I’ll keep it on Android.

I also bought four phone cases for $16. I got spares because of the high price of postage relative to the case cost and the fact that they may be difficult to get in a few years.

The Tests

For the next phone my plan was to do more tests on Android before upgrading it to Debian. Here are the ones I can think of now, please suggest any others I should do.

  • Log output of “ps auxf” equivalent.
  • Make notes on what they are doing with SE Linux.
  • Test the stylus.
  • Test USB networking to my workstation and my laptop.
  • Make a copy of the dmesg output. Also look for D state processes and other signs of problems.

Droidian and Security

When I tell technical people about Droidian a common reaction is “great you can get a cheap powerful phone and have better security than Android”. This is wrong in several ways. Firstly Android has quite decent security. Android runs most things in containers and uses SE Linux. Droidian has the Debian approach for most software (IE it all runs under the same UID without any special protections) and the developers have no plans to use SE Linux. I’ve previously blogged about options for Sandboxing for Debian phone use, my blog post is NOT a solution to the problem but an analysis of the different potential ways of going about solving it [5].

The next issue is that Droidian has no way to update the kernel and the installation instructions often advise downgrading Android (running a less secure kernel) before the installation. The Android Generic Kernel Image project [6] addresses this by allowing a separation between drivers supplied by the hardware vendor and the kernel image supplied by Google. This also permits running the hardware vendor’s drivers with a GKI kernel released by Google after the hardware vendor dropped security support. But this only applies to Android 11 and later, so Android 10 devices (like the Note 9 image for Droidian) miss out on this.

28 April, 2024 11:40AM by etbe

April 27, 2024

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

qlcal 0.0.11 on CRAN: Calendar Updates

The eleventh release of the qlcal package arrivied at CRAN today.

qlcal delivers the calendaring parts of QuantLib. It is provided (for the R package) as a set of included files, so the package is self-contained and does not depend on an external QuantLib library (which can be demanding to build). qlcal covers over sixty country / market calendars and can compute holiday lists, its complement (i.e. business day lists) and much more. Examples are in the README at the repository, the package page, and course at the CRAN package page.

This releases synchronizes qlcal with the QuantLib release 1.34 and contains more updates to 2024 calendars.

Changes in version 0.0.11 (2024-04-27)

  • Synchronized with QuantLib 1.34

  • Calendar updates for Brazil, India, Singapore, South Africa, Thailand, United States

  • Minor continuous integration update

Courtesy of my CRANberries, there is a diffstat report for this release. See the project page and package documentation for more details, and more examples. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

27 April, 2024 09:58PM

April 26, 2024

RcppSpdlog 0.0.17 on CRAN: New Upstream

Version 0.0.17 of RcppSpdlog arrived on CRAN overnight following and has been uploaded to Debian. RcppSpdlog bundles spdlog, a wonderful header-only C++ logging library with all the bells and whistles you would want that was written by Gabi Melman, and also includes fmt by Victor Zverovich. You can learn more at the nice package documention site.

This releases updates the code to the version 1.14 of spdlog which was release yesterday.

The NEWS entry for this release follows.

Changes in RcppSpdlog version 0.0.17 (2024-04-25)

  • Minor continuous integration update

  • Upgraded to upstream release spdlog 1.14.0

Courtesy of my CRANberries, there is also a diffstat report. More detailed information is on the RcppSpdlog page, or the package documention site. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

26 April, 2024 09:16PM

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

Continued life with bcachefs

This post was supposed to be called “death with bcachefs”, but it sounded a bit too dramatic. :-) Evidently bcachefs-tools in Debian is finally getting an update (although in experimental), so that's good. Meanwhile, one of my multi-device filesystems died a horrible death, and since I had backups, I didn't ask for its fix to be prioritized—fsck still is unable to repair it and I don't use bcachefs on that machine anymore. But the other one still lives fairly happily.

Hanging around #bcachefs on IRC tells me that indeed, this thing is still quite experimental. Some of the killer features (like proper compression) don't perform as well as they should yet. Large rewrites are still happening. People are still reporting quite weird bugs that are being triaged and mostly fixed (although if you can't reproduce them, you're pretty much hosed). But it's a fun ride. Again: Have backups. They saved me. :-)

26 April, 2024 08:05PM

hackergotchi for Robert McQueen

Robert McQueen

Update from the GNOME board

It’s been around 6 months since the GNOME Foundation was joined by our new Executive Director, Holly Million, and the board and I wanted to update members on the Foundation’s current status and some exciting upcoming changes.

Finances

As you may be aware, the GNOME Foundation has operated at a deficit (nonprofit speak for a loss – ie spending more than we’ve been raising each year) for over three years, essentially running the Foundation on reserves from some substantial donations received 4-5 years ago. The Foundation has a reserves policy which specifies a minimum amount of money we have to keep in our accounts. This is so that if there is a significant interruption to our usual income, we can preserve our core operations while we work on new funding sources. We’ve now “hit the buffers” of this reserves policy, meaning the Board can’t approve any more deficit budgets – to keep spending at the same level we must increase our income.

One of the board’s top priorities in hiring Holly was therefore her experience in communications and fundraising, and building broader and more diverse support for our mission and work. Her goals since joining – as well as building her familiarity with the community and project – have been to set up better financial controls and reporting, develop a strategic plan, and start fundraising. You may have noticed the Foundation being more cautious with spending this year, because Holly prepared a break-even budget for the Board to approve in October, so that we can steady the ship while we prepare and launch our new fundraising initiatives.

Strategy & Fundraising

The biggest prerequisite for fundraising is a clear strategy – we need to explain what we’re doing and why it’s important, and use that to convince people to support our plans. I’m very pleased to report that Holly has been working hard on this and meeting with many stakeholders across the community, and has prepared a detailed and insightful five year strategic plan. The plan defines the areas where the Foundation will prioritise, develop and fund initiatives to support and grow the GNOME project and community. The board has approved a draft version of this plan, and over the coming weeks Holly and the Foundation team will be sharing this plan and running a consultation process to gather feedback input from GNOME foundation and community members.

In parallel, Holly has been working on a fundraising plan to stabilise the Foundation, growing our revenue and ability to deliver on these plans. We will be launching a variety of fundraising activities over the coming months, including a development fund for people to directly support GNOME development, working with professional grant writers and managers to apply for government and private foundation funding opportunities, and building better communications to explain the importance of our work to corporate and individual donors.

Board Development

Another observation that Holly had since joining was that we had, by general nonprofit standards, a very small board of just 7 directors. While we do have some committees which have (very much appreciated!) volunteers from outside the board, our officers are usually appointed from within the board, and many board members end up serving on multiple committees and wearing several hats. It also means the number of perspectives on the board is limited and less representative of the diverse contributors and users that make up the GNOME community.

Holly has been working with the board and the governance committee to reduce how much we ask from individual board members, and improve representation from the community within the Foundation’s governance. Firstly, the board has decided to increase its size from 7 to 9 members, effective from the upcoming elections this May & June, allowing more voices to be heard within the board discussions. After that, we’re going to be working on opening up the board to more participants, creating non-voting officer seats to represent certain regions or interests from across the community, and take part in committees and board meetings. These new non-voting roles are likely to be appointed with some kind of application process, and we’ll share details about these roles and how to be considered for them as we refine our plans over the coming year.

Elections

We’re really excited to develop and share these plans and increase the ways that people can get involved in shaping the Foundation’s strategy and how we raise and spend money to support and grow the GNOME community. This brings me to my final point, which is that we’re in the run up to the annual board elections which take place in the run up to GUADEC. Because of the expansion of the board, and four directors coming to the end of their terms, we’ll be electing 6 seats this election. It’s really important to Holly and the board that we use this opportunity to bring some new voices to the table, leading by example in growing and better representing our community.

Allan wrote in the past about what the board does and what’s expected from directors. As you can see we’re working hard on reducing what we ask from each individual board member by increasing the number of directors, and bringing additional members in to committees and non-voting roles. If you’re interested in seeing more diverse backgrounds and perspectives represented on the board, I would strongly encourage you consider standing for election and reach out to a board member to discuss their experience.

Thanks for reading! Until next time.

Best Wishes,
Rob
President, GNOME Foundation

Update 2024-04-27: It was suggested in the Discourse thread that I clarify the interaction between the break-even budget and the 1M EUR committed by the STF project. This money is received in the form of a contract for services rather than a grant to the Foundation, and must be spent on the development areas agreed during the planning and application process. It’s included within this year’s budget (October 23 – September 24) and is all expected to be spent during this fiscal year, so it doesn’t have an impact on the Foundation’s reserves position. The Foundation retains a small % fee to support its costs in connection with the project, including the new requirement to have our accounts externally audited at the end of the financial year. We are putting this money towards recruitment of an administrative assistant to improve financial and other operational support for the Foundation and community, including the STF project and future development initiatives.

(also posted to GNOME Discourse, please head there if you have any questions or comments)

26 April, 2024 10:39AM by ramcq

April 25, 2024

hackergotchi for Jonathan McDowell

Jonathan McDowell

Sorting out backup internet #3: failover

With local recursive DNS and a 5G modem in place the next thing was to work on some sort of automatic failover when the primary FTTP connection failed. My wife works from home too and I sometimes travel so I wanted to make sure things didn’t require me to be around to kick them into switch the link in use.

First, let’s talk about what I didn’t do. One choice to try and ensure as seamless a failover as possible would be to get a VM somewhere out there. I’d then run Wireguard tunnels over both the FTTP + 5G links to the VM, and run some sort of routing protocol (RIP, OSPF?) over the links. Set preferences such that the FTTP is preferred, NAT v4 to the VM IP, and choose somewhere that gave me a v6 range I could just use directly.

This has the advantage that I’m actively checking link quality to the outside work, rather than just to the next hop. It also means, if the failover detection is fast enough, that existing sessions stay up rather than needing re-established.

The downsides are increased complexity, adding another point of potential failure (the VM + provider), the impact on connection quality (even with a decent endpoint it’s an extra hop and latency), and finally the increased cost involved.

I can cope with having to reconnect my SSH sessions in the event of a failure, and I’d rather be sure I can make full use of the FTTP connection, so I didn’t go this route. I chose to rely on local link failure detection to provide the signal for failover, and a set of policy routing on top of that to make things a bit more seamless.

Local link failure turns out to be fairly easy. My FTTP is a PPPoE configuration, so in /etc/ppp/peers/aquiss I have:

lcp-echo-interval 1
lcp-echo-failure 5
lcp-echo-adaptive

Which gives me a failover of ~ 5s if the link goes down.

I’m operating the 5G modem in “bridge” rather than “router” mode, which means I get the actual IP from the 5G network via DHCP. The DHCP lease the modem hands out is under a minute, and in the event of a network failure it only hands out a 192.168.254.x IP to talk to its web interface. As the 5G modem is the last resort path I choose not to do anything special with this, but the information is at least there if I need it.

To allow both interfaces to be up and the FTTP to be preferred I’m simply using route metrics. For the PPP configuration that’s:

defaultroute-metric 100

and for the 5G modem I have:

iface sfp.31 inet dhcp
    metric 1000
    vlan-raw-device sfp

There’s a wrinkle in that pppd will not replace an existing default route, so I’ve created /etc/ppp/ip-up.d/default-route to ensure it’s added:

#!/bin/bash

[ "$PPP_IFACE" = "pppoe-wan" ] || exit 0

# Ensure we add a default route; pppd will not do so if we have
# a lower pref route out the 5G modem
ip route add default dev pppoe-wan metric 100 || true

Additionally, in /etc/dhcp/dhclient.conf I’ve disabled asking for any server details (DNS, NTP, etc) - I have internal setups for the servers I want, and don’t want to be trying to select things over the 5G link by default.

However, what I do want is to be able to access the 5G modem web interface and explicitly route some traffic out that link (e.g. so I can add it to my smokeping tests). For that I need some source based routing.

First step, add a 5g table to /etc/iproute2/rt_tables:

16  5g

Then I ended up with the following in /etc/dhcp/dhclient-exit-hooks.d/modem-interface-route, which is more complex than I’d like but seems to do what I want:

#!/bin/sh

case "$reason" in
    BOUND|RENEW|REBIND|REBOOT)
        # Check if we've actually changed IP address
        if [ -z "$old_ip_address" ] ||
           [ "$old_ip_address" != "$new_ip_address" ] ||
           [ "$reason" = "BOUND" ] || [ "$reason" = "REBOOT" ]; then
            if [ ! -z "$old_ip_address" ]; then
                ip rule del from $old_ip_address lookup 5g
            fi
            ip rule add from $new_ip_address lookup 5g

            ip route add default dev sfp.31 table 5g || true
            ip route add 192.168.254.1 dev sfp.31 2>/dev/null || true
        fi
    ;;

    EXPIRE)
        if [ ! -z "$old_ip_address" ]; then
            ip rule del from $old_ip_address lookup 5g
        fi
    ;;

    *)
    ;;
esac

What does all that aim to do? We want to ensure traffic directed to the 5G WAN address goes out the 5G modem, so I can SSH into it even when the main link is up. So we add a rule directing traffic from that IP to hit the 5g routing table, and a default route in that table which uses the 5G link. There’s no configuration for the FTTP connection in that table, so if the 5G link is down the traffic gets dropped, which is what we want. We also configure 192.168.254.1 to go out the link to the modem, as that’s where the web interface lives.

I also have a curl callout (curl --interface sfp.31 … to ensure it goes out the 5G link) after the routes are configured to set dynamic DNS with Mythic Beasts, which helps with knowing where to connect back to. I seem to see IP address changes on the 5G link every couple of days at least.

Additionally, I have an entry in the interfaces configuration carving out the top set of the netblock my smokeping server is in:

    up ip rule add from 192.0.2.224/27 lookup 5g

My smokeping /etc/smokeping/config.d/Probes file then looks like:

*** Probes ***

+ FPing

binary = /usr/bin/fping

++ FPingNormal

++ FPing5G

sourceaddress = 192.0.2.225

+ FPing6

binary = /usr/bin/fping

which allows me to use probe = FPing5G for targets to test them over the 5G link.

That mostly covers the functionality I want for a backup link. There’s one piece that isn’t quite solved, however, IPv6, which can wait for another post.

25 April, 2024 05:38PM

hackergotchi for Jonathan Dowland

Jonathan Dowland

Biosphere

I've been enjoying Biosphere as the soundtrack to my recent "concentrated work" spells.

Knives by Biosphere

I remember seeing their name on playlists of yester-year: axioms, bluemars1, and (still a going concern) soma.fm's drone zone.


  1. Bluemars lives on, at echoes of bluemars

25 April, 2024 03:15PM

Russ Allbery

Review: Nation

Review: Nation, by Terry Pratchett

Publisher: Harper
Copyright: 2008
Printing: 2009
ISBN: 0-06-143303-9
Format: Trade paperback
Pages: 369

Nation is a stand-alone young adult fantasy novel. It was published in the gap between Discworld novels Making Money and Unseen Academicals.

Nation starts with a plague. The Russian influenza has ravaged Britain, including the royal family. The next in line to the throne is off on a remote island and must be retrieved and crowned as soon as possible, or an obscure provision in Magna Carta will cause no end of trouble. The Cutty Wren is sent on this mission, carrying the Gentlemen of Last Resort.

Then comes the tsunami.

In the midst of fire raining from the sky and a wave like no one has ever seen, Captain Roberts tied himself to the wheel of the Sweet Judy and steered it as best he could, straight into an island. The sole survivor of the shipwreck: one Ermintrude Fanshaw, daughter of the governor of some British island possessions. Oh, and a parrot.

Mau was on the Boys' Island when the tsunami came, going through his rite of passage into manhood. He was to return to the Nation the next morning and receive his tattoos and his adult soul. He survived in a canoe. No one else in the Nation did.

Terry Pratchett considered Nation to be his best book. It is not his best book, at least in my opinion; it's firmly below the top tier of Discworld novels, let alone Night Watch. It is, however, an interesting and enjoyable book that tackles gods and religion with a sledgehammer rather than a knife.

It's also very, very dark and utterly depressing at the start, despite a few glimmers of Pratchett's humor. Mau is the main protagonist at first, and the book opens with everyone he cares about dying. This is the place where I thought Pratchett diverged the most from his Discworld style: in Discworld, I think most of that would have been off-screen, but here we follow Mau through the realization, the devastation, the disassociation, the burials at sea, the thoughts of suicide, and the complete upheaval of everything he thought he was or was about to become. I found the start of this book difficult to get through. The immediate transition into potentially tragic misunderstandings between Mau and Daphne (as Ermintrude names herself once there is no one to tell her not to) didn't help.

As I got farther into the book, though, I warmed to it. The best parts early on are Daphne's baffled but scientific attempts to understand Mau's culture and her place in it. More survivors arrive, and they start to assemble a community, anchored in large part by Mau's stubborn determination to do what's right even though he's lost all of his moorings. That community eventually re-establishes contact with the rest of the world and the opening plot about the British monarchy, but not before Daphne has been changed profoundly by being part of it.

I think Pratchett worked hard at keeping Mau's culture at the center of the story. It's notable that the community that reforms over the course of the book essentially follows the patterns of Mau's lost Nation and incorporates Daphne into it, rather than (as is so often the case) the other way around. The plot itself is fiercely anti-colonial in a way that mostly worked. Still, though, it's a quasi-Pacific-island culture written by a white British man, and I had some qualms.

Pratchett quite rightfully makes it clear in the afterward that this is an alternate world and Mau's culture is not a real Pacific island culture. However, that also means that its starkly gender-essentialist nature was a free choice, rather than one based on some specific culture, and I found that choice somewhat off-putting. The religious rituals are all gendered, the dwelling places are gendered, and one's entire life course in Mau's world seems based on binary classification as a man or a woman. Based on Pratchett's other books, I assume this was more an unfortunate default than a deliberate choice, but it's still a choice he could have avoided.

The end of this book wrestles directly with the relative worth of Mau's culture versus that of the British. I liked most of this, but the twists that Pratchett adds to avoid the colonialist results we saw in our world stumble partly into the trap of making Mau's culture valuable by British standards. (I'm being a bit vague here to avoid spoilers.) I think it is very hard to base this book on a different set of priorities and still bring the largely UK, US, and western European audience along, so I don't blame Pratchett for failing to do it, but I'm a bit sad that the world still revolved around a British axis.

This felt quite similar to Discworld to me in its overall sensibilities, but with the roles of moral philosophy and humor reversed. Discworld novels usually start with some larger-than-life characters and an absurd plot, and then the moral philosophy sneaks up behind you when you're not looking and hits you over the head. Nation starts with the moral philosophy: Mau wrestles with his gods and the problem of evil in a way that reminded me of Job, except with a far different pantheon and rather less tolerance for divine excuses on the part of the protagonist. It's the humor, instead, that sneaks up on you and makes you laugh when the plot is a bit too much. But the mix arrives at much the same place: the absurd hand-in-hand with the profound, and all seen from an angle that makes it a bit easier to understand.

I'm not sure I would recommend Nation as a good place to start with Pratchett. I felt like I benefited from having read a lot of Discworld to build up my willingness to trust where Pratchett was going. But it has the quality of writing of late Discworld without the (arguable) need to read 25 books to understand all of the backstory. Regardless, recommended, and you'll never hear Twinkle Twinkle Little Star in quite the same way again.

Rating: 8 out of 10

25 April, 2024 04:18AM

April 24, 2024

Nazi.Compare

Daniel Pocock elected on ANZAC Day and anniversary of Easter Rising (FSFE Fellowship)

24 April is the anniversary of the Easter Rising. That is the day that Irish republicans bravely rose up against foreign control. The Irish leaders were captured and killed.

25 April is ANZAC Day. It is the anniversary of various battles, most notable of which is Gallipoli.

The Gallipoli landings, like the Easter Rising, are notable for a significant loss of life.

ANZAC Day has evolved to commemorate not only those who lost their lives at Gallipoli but all the sacrifices made by those who serve in uniform for Australia and New Zealand.

When the FSFE Fellowship voted for the Irish-Australian Daniel Pocock in 2017, voting finished at 11:05 UTC on 24 April 2017. That is the 101st anniversary of the Easter rising.

FSFE announced Pocock's victory the following day, 25 April 2017, ANZAC Day, as we can see in the screenshot below.

FSFE headquarters are in Berlin, Germany and the majority of members appear to be Germans. After the community voted for an Irish-Australian, the management removed the elections from the FSFE constitution. There hasn't been another election since.

Simply mentioning the ANZACs is akin to standing on the shoulders of giants. Nonetheless, Pocock successfully discovered a lot of evidence of bad faith, corruption and deception by the FSFE management.

The Nazi.Compare site was created to continue the story

Daniel Pocock, FSFE, ANZAC Day

Read more of Daniel Pocock's blogs about visits to ANZAC sites around the world.

24 April, 2024 09:00PM

April 22, 2024

hackergotchi for Bits from Debian

Bits from Debian

Debian Project Leader Election 2024, Andreas Tille elected.

The voting period for the Debian Project Leader election has ended. Please join us in congratulating Andreas Tille as the new Debian Project Leader.

The new term for the project leader started on 2024-04-21.

369 of 1,010 Debian Developers voted using the Condorcet method.

More information about the results of the voting are available on the Debian Project Leader Elections 2024 page.

Many thanks all of our Developers for voting.

22 April, 2024 12:00PM by Donald Norwood

Vincent Fourmond

QSoas version 3.3 is out

Version 3.3 brings in new features, including reverse Laplace transforms and fits, pH fits, commands for picking points from a dataset, averaging points with the same X value, or perform singular value decomposition.

In addition to these new features, many previous commands were improved, like the addition of a bandcut filter in FFT filtering, better handling of the loading of files produced by QSoas itself, and a button to interrupt the processing of scripts.

There are a lot of other new features, improvements and so on, look for the full list there.

About QSoas


QSoas is a powerful open source data analysis program that focuses on flexibility and powerful fitting capacities. It is released under the GNU General Public License. It is described in Fourmond, Anal. Chem., 2016, 88 (10), pp 5050–5052. Current version is 3.3. You can download for free its source code or precompiled versions for MacOS and Windows there. Alternatively, you can clone from the GitHub repository.

22 April, 2024 10:50AM by Vincent Fourmond (noreply@blogger.com)

Russ Allbery

Review: The Stars, Like Dust

Review: The Stars, Like Dust, by Isaac Asimov

Series: Galactic Empire #2
Publisher: Fawcett Crest
Copyright: 1950, 1951
Printing: June 1972
Format: Mass market
Pages: 192

The Stars, Like Dust is usually listed as the first book in Asimov's lesser-known Galactic Empire Trilogy since it takes place before Pebble in the Sky. Pebble in the Sky was published first, though, so I count it as the second book. It is very early science fiction with a few mystery overtones.

Buying books produces about 5% of the pleasure of reading them while taking much less than 5% of the time. There was a time in my life when I thoroughly enjoyed methodically working through a used book store, list in hand, tracking down cheap copies to fill in holes in series. This means that I own a lot of books that I thought at some point that I would want to read but never got around to, often because, at the time, I was feeling completionist about some series or piece of world-building. From time to time, I get the urge to try to read some of them.

Sometimes this is a poor use of my time.

The Galactic Empire series is from Asimov's first science fiction period, after the Foundation series but contemporaneous with their collection into novels. They're set long, long before Foundation, but after humans have inhabited numerous star systems and Earth has become something of a backwater. That process is just starting in The Stars, Like Dust: Earth is still somewhere where an upper-class son might be sent for an education, but it has been devastated by nuclear wars and is well on its way to becoming an inward-looking relic on the edge of galactic society.

Biron Farrill is the son of the Lord Rancher of Widemos, a wealthy noble whose world is one of those conquered by the Tyranni. In many other SF novels, the Tyranni would be an alien race; here, it's a hierarchical and authoritarian human civilization. The book opens with Biron discovering a radiation bomb planted in his dorm room. Shortly after, he learns that his father had been arrested. One of his fellow students claims to be on Biron's side against the Tyranni and gives him false papers to travel to Rhodia, a wealthy world run by a Tyranni sycophant.

Like most books of this era, The Stars, Like Dust is a short novel full of plot twists. Unlike some of its contemporaries, it's not devoid of characterization, but I might have liked it better if it were. Biron behaves like an obnoxious teenager when he's not being an arrogant ass. There is a female character who does a few plot-relevant things and at no point is sexually assaulted, so I'll give Asimov that much, but the gender stereotypes are ironclad and there is an entire subplot focused on what I can only describe as seduction via petty jealousy.

The writing... well, let me quote a typical passage:

There was no way of telling when the threshold would be reached. Perhaps not for hours, and perhaps the next moment. Biron remained standing helplessly, flashlight held loosely in his damp hands. Half an hour before, the visiphone had awakened him, and he had been at peace then. Now he knew he was going to die.

Biron didn't want to die, but he was penned in hopelessly, and there was no place to hide.

Needless to say, Biron doesn't die. Even if your tolerance for pulp melodrama is high, 192 small-print pages of this sort of thing is wearying.

Like a lot of Asimov plots, The Stars, Like Dust has some of the shape of a mystery novel. Biron, with the aid of some newfound companions on Rhodia, learns of a secret rebellion against the Tyranni and attempts to track down its base to join them. There are false leads, disguised identities, clues that are difficult to interpret, and similar classic mystery trappings, all covered with a patina of early 1950s imaginary science. To me, it felt constructed and artificial in ways that made the strings Asimov was pulling obvious. I don't know if someone who likes mystery construction would feel differently about it.

The worst part of the plot thankfully doesn't come up much. We learn early in the story that Biron was on Earth to search for a long-lost document believed to be vital to defeating the Tyranni. The nature of that document is revealed on the final page, so I won't spoil it, but if you try to think of the stupidest possible document someone could have built this plot around, I suspect you will only need one guess. (In Asimov's defense, he blamed Galaxy editor H.L. Gold for persuading him to include this plot, and disavowed it a few years later.)

The Stars, Like Dust is one of the worst books I have ever read. The characters are overwrought, the politics are slapdash and build on broad stereotypes, the romantic subplot is dire and plays out mainly via Biron egregiously manipulating his petulant love interest, and the writing is annoying. Sometimes pulp fiction makes up for those common flaws through larger-than-life feats of daring, sweeping visions of future societies, and ever-escalating stakes. There is little to none of that here. Asimov instead provides tedious political maneuvering among a class of elitist bankers and land owners who consider themselves natural leaders. The only places where the power structures of this future government make sense are where Asimov blatantly steals them from either the Roman Empire or the Doge of Venice.

The one thing this book has going for it — the thing, apart from bloody-minded completionism, that kept me reading — is that the technology is hilariously weird in that way that only 1940s and 1950s science fiction can be. The characters have access to communication via some sort of interstellar telepathy (messages coded to a specific person's "brain waves") and can travel between stars through hyperspace jumps, but each jump is manually calculated by referring to the pilot's (paper!) volumes of the Standard Galactic Ephemeris. Communication between ships (via "etheric radio") requires manually aiming a radio beam at the area in space where one thinks the other ship is. It's an unintentionally entertaining combination of technology that now looks absurdly primitive and science that is so advanced and hand-waved that it's obviously made up.

I also have to give Asimov some points for using spherical coordinates. It's a small thing, but the coordinate systems in most SF novels and TV shows are obviously not fit for purpose.

I spent about a month and a half of this year barely reading, and while some of that is because I finally tackled a few projects I'd been putting off for years, a lot of it was because of this book. It was only 192 pages, and I'm still curious about the glue between Asimov's Foundation and Robot series, both of which I devoured as a teenager. But every time I picked it up to finally finish it and start another book, I made it about ten pages and then couldn't take any more. Learn from my error: don't try this at home, or at least give up if the same thing starts happening to you.

Followed by The Currents of Space.

Rating: 2 out of 10

22 April, 2024 02:22AM