koldfront

Dear Signal #community #free software #programming #vcs

Please provide a useful changelog in your signal-desktop Debian package.

The previous package broke the application, today apt(8) fetched a new package, and the complete changelog reads:

signal-desktop (1.27.2) whatever; urgency=medium
 
  * Package created with FPM.
 
 -- Open Whisper Systems   Fri, 06 Sep 2019 15:36:44 -0700

That is useless. The changelog should provide, at minimum, a summary of the changes between this version and the previous one.

Thank you.

⊗ (0 comments)

Hosts sending 2 or 6 NULL bytes #net

Recently, like the last month or so, my server has been receiving packets from various, seemingly arbitrary, hosts, containing either 2 or 6 NULL bytes.

They hit mostly port 22 (ssh), 53 (dns), 80 (http), 443 (https) and imaps (993). I only have a very limited number of ports open in the router, so they might be hitting more ports.

Looking at them with ngrep(8), it looks like this:

$ sudo ngrep -x -q '^\x00\x00*$'
interface: enp4s0 (192.168.1.0/255.255.255.0)
filter: ((ip || ip6) || (vlan && (ip || ip6)))
match: ^\x00\x00*$
 
T 88.208.41.159:42560 -> 192.168.1.101:22 [S] #32
  00 00                                                 ..
 
T 78.140.142.55:44230 -> 192.168.1.101:80 [S] #41
  00 00                                                 ..
 
T 78.140.141.247:47152 -> 192.168.1.101:53 [S] #46
  00 00                                                 ..
 
T 5.11.81.197:48256 -> 192.168.1.101:443 [S] #62
  00 00                                                 ..
 
T 54.36.150.116:41070 -> 192.168.1.101:443 [R] #404
  00 00 00 00 00 00                                     ......

I'm not quite sure to make of it. The sources seem to change, sometimes they're mostly from Japan, sometimes from China, sometimes from AWS, other times from various hosting companies.

Anyone know what this is?

⊗ (0 comments)

rdiff-backup backport to Debian 11 (buster) #debian

A new version of rdiff-backup, 1.3.3, has been added to Debian unstable.

When that is used to backup to a machine running Debian 11 (buster), you get errors, because it has version 1.2.8.

As a work-around, I have backported librsync-dev and rdiff-backup from unstable to Debian 11 (buster), packages are in my local repository. Use at your own peril.

⊗ (0 comments)

Making subprocess.Popen in Python 3 play nice with "elaborate" output #programming #python

docker-compose produces "elaborate" output when run, making use of carriage returns and ANSI CSI codes to move the cursor about.

A wrapper script written in Python 3 tried to handle the output - printing a '.' for each line usually, and actually printing the lines if given a --verbose option.

Unfortunately the output with --verbose got mangled, which really annoyed me. So I tried to find a solution.

Here is a small script that does some output, mimicking the stuff that docker-compose outputs. Note that we want the lines printed as they are produced, and not all at once at the end of the script.

command.py:

#!/usr/bin/python3
 
from time import sleep
 
print("1", flush=True)
sleep(1)
print("2", flush=True)
sleep(1)
print("3", flush=True)
sleep(1)
 
print("\x1b[2A\x1b[K\r", end="", flush=True)
sleep(1)
print("2 - let's go\r", end="", flush=True)
sleep(1)
print("2 - lookin' good\r", end="", flush=True)
sleep(1)
print("\x1b[1A\x1b[K\r", end="", flush=True)
sleep(1)
print("1 - abba\r", end="", flush=True)
sleep(1)
print("\x1b[2B\r", end="", flush=True)
sleep(1)
print("\r3 - flappa\r", end="", flush=True)
sleep(1)
print("\x1b[1A\x1b[K\r", end="", flush=True)
sleep(1)
print("2 - done\r", end="", flush=True)
sleep(1)
print("\x1b[1A\x1b[K\r", end="", flush=True)
sleep(1)
print("1 - done\r", end="", flush=True)
sleep(1)
print("\x1b[2B\r", end="", flush=True)
sleep(1)
print("3 - done\x1b[K\r", end="", flush=True)
sleep(1)
print("\nFinito", flush=True)

Try running it:

The original code wrapping it, was like this (slightly simplified):

runner_orig.py:

#!/usr/bin/python3
 
import subprocess
 
 
def run_command():
    p = subprocess.Popen("./command.py",
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT,
                         universal_newlines=True)  # this converts \r into \n #fail
 
    for line in iter(p.stdout.readline, ""):
        yield line, p.poll()
 
    yield "", p.wait()
 
 
for l, rc in run_command():
    print(l, end="", flush=True)

If you run this, you'll see how the output it mangled:

So why does this happen? Well, the first culprit is universal_newlines=True. That means that any combination of carriage return and/or line feed is interpreted as and converted to a line feed. Uh-oh, definitely not what we want, if the fancy output it to be reproduced as intended.

If set to False, the situation improves, carriage returns are no longer converted. Unfortunately the p.stdout.readline() function only interprets line feeds as end of line and not carriage returns, so all the fancy stuff piles up, and is only shown at the very end. Not what we want.

Looking at the documentation of open() reveals that it has an option called newline, which is used to control how universal newlines are handled, and that if set to '' it actually does what we want: carriage returns are not converted, and they are recognized and line endings.

Unfortunately the documentation of subprocess.Popen() only documents universal_newlines taking two values, True and False, and '' doesn't work. I tried.

Fortunately Klaus had a tip - if you use os.dup(), you can open the file handle again, and - hey - now I can give it the option to open() we need, newline=''.

Lo and behold, it works!

runner_fixed.py:

#!/usr/bin/python3
 
import subprocess
import os
 
def run_command():
    p = subprocess.Popen("./command.py",
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT,
                         universal_newlines=False)  # \r goes through
 
    nice_stdout = open(os.dup(p.stdout.fileno()), newline='')  # re-open to get \r recognized as new line
    for line in nice_stdout:
        yield line, p.poll()
 
    yield "", p.wait()
 
 
for l, rc in run_command():
    print(l, end="", flush=True)

It is kind of kludgy to have to do that, but it does work:

I wonder why subprocess.Popen() in Python 3 does not have the same options for handling newlines as open() has. It feels like open() has moved on (to the newline parameter), but subprocess.Popen() hasn't followed yet.

⊗ (0 comments)

SanDisk Ultra Flair 64GB USB flash drive #hardware

Useful consumer review: The SanDisk Ultra Flair 64GB USB flash drive I bought last summer survived less than 1 year of rattling around in my bag.

Basically it has been on a keyring with a Yubikey Neo the whole time, and it just stopped working. As in nothing shows up in syslog or dmesg, and the USB device gets very hot quickly when inserted into a USB-port.

I tried prying it apart, to see if something was shorting and that was why it didn't work, but even inserting just the "core" of the device, without the casing yields nothing registered by the computer and a very hot device.

While it worked, it worked fine - but less than a year until catastrophical failure, without any warning, I give it 1 star.

So, what should I get instead? The Samsung BAR Plus Titan 128GB seems to have nice specs (300 MB/s).

⊗ (0 comments)

Server upgraded to Debian 11 (buster) #debian

All in all it sort of ok.

Problems:

• Old Perl modules with iso-8859-1 encoding needed to be updated to utf-8 (recode latin1..utf8 WhatEver.pm)
• Haskell programs needed cabal configure and some packages installed again
• While upgrading PostgreSQL from 9.6 to 11, it wouldn't start due to an old, too short, snakeoil certificate
• radicale won't start - the comment in /etc/default/radicale says to run sudo update-rc.d radicale enable but that gives an error, and doesn't help.
  • Ah, data needed to be exported and config changed
  • Exported data could not be read, had to remove line feeds from Quoted-Printable encoded lines
  • And had to manually changes PHOTO;ENCODING=BASE64;JPEG: to PHOTO;ENCODING=b;TYPE=jpeg:, removing duplicates and empty lines, in every contact file, otherwise Python Exception
  • Having a calender in a single file is no longer supported, apparently, so I have to rework my entire setup for exporting my calenders to my phone, to export to a file per event. Argh
• Haven't gotten Hatta to work yet, some problem with the Mercurial Python API. Sorted with a virtual environment with an older mercurial package in it, thanks to the author of Hatta

⊗ (0 comments)

MIME support for inline images #email

I was about to send an image inline in an email using the venerable MIME standard, when I remembered that many clients don't handle that very well.

So I created a test email to see how they do.

The email is very simple - some text, an inline image, and some more text. Here is a copy, if you want to test your favourite client.

My email client shows it like this:

Here are the results:

Client	Shows image inline	Shows text after image	Screenshot
Gmail	❌ at bottom	✓	image
Office 365 Outlook	❌ at top	❌ hidden as attachment	image
Outlook iPhone	✓	❌ hidden as attachment	image
Protonmail	❌ as attachment at bottom	✓	image
Tutanota	❌ as attachment at top	❌ hidden as attachment	image
K9 Android	❌ as attachment at bottom	✓	image
Gnus	✓	✓	image
Thunderbird	✓	✓	image

So what I remembered was not wrong. Most email clients suck at MIME.

The Outlook app on the iPhone gets an extra half mark subtracted for showing an exceedingly ridiculous amount of vertical white space.

Which is not surprising, the relevant RFC is from 1996.

⊗ (0 comments)