koldfront

Making subprocess.Popen in Python 3 play nice with "elaborate" output #programming #python

🕢︎ - 2019-07-23

docker-compose produces "elaborate" output when run, making use of carriage returns and ANSI CSI codes to move the cursor about.

A wrapper script written in Python 3 tried to handle the output - printing a '.' for each line usually, and actually printing the lines if given a --verbose option.

Unfortunately the output with --verbose got mangled, which really annoyed me. So I tried to find a solution.

Here is a small script that does some output, mimicking the stuff that docker-compose outputs. Note that we want the lines printed as they are produced, and not all at once at the end of the script.

command.py:

#!/usr/bin/python3
 
from time import sleep
 
print("1", flush=True)
sleep(1)
print("2", flush=True)
sleep(1)
print("3", flush=True)
sleep(1)
 
print("\x1b[2A\x1b[K\r", end="", flush=True)
sleep(1)
print("2 - let's go\r", end="", flush=True)
sleep(1)
print("2 - lookin' good\r", end="", flush=True)
sleep(1)
print("\x1b[1A\x1b[K\r", end="", flush=True)
sleep(1)
print("1 - abba\r", end="", flush=True)
sleep(1)
print("\x1b[2B\r", end="", flush=True)
sleep(1)
print("\r3 - flappa\r", end="", flush=True)
sleep(1)
print("\x1b[1A\x1b[K\r", end="", flush=True)
sleep(1)
print("2 - done\r", end="", flush=True)
sleep(1)
print("\x1b[1A\x1b[K\r", end="", flush=True)
sleep(1)
print("1 - done\r", end="", flush=True)
sleep(1)
print("\x1b[2B\r", end="", flush=True)
sleep(1)
print("3 - done\x1b[K\r", end="", flush=True)
sleep(1)
print("\nFinito", flush=True)

Try running it:

The original code wrapping it, was like this (slightly simplified):

runner_orig.py:

#!/usr/bin/python3
 
import subprocess
 
 
def run_command():
    p = subprocess.Popen("./command.py",
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT,
                         universal_newlines=True)  # this converts \r into \n #fail
 
    for line in iter(p.stdout.readline, ""):
        yield line, p.poll()
 
    yield "", p.wait()
 
 
for l, rc in run_command():
    print(l, end="", flush=True)

If you run this, you'll see how the output it mangled:

So why does this happen? Well, the first culprit is universal_newlines=True. That means that any combination of carriage return and/or line feed is interpreted as and converted to a line feed. Uh-oh, definitely not what we want, if the fancy output it to be reproduced as intended.

If set to False, the situation improves, carriage returns are no longer converted. Unfortunately the p.stdout.readline() function only interprets line feeds as end of line and not carriage returns, so all the fancy stuff piles up, and is only shown at the very end. Not what we want.

Looking at the documentation of open() reveals that it has an option called newline, which is used to control how universal newlines are handled, and that if set to '' it actually does what we want: carriage returns are not converted, and they are recognized and line endings.

Unfortunately the documentation of subprocess.Popen() only documents universal_newlines taking two values, True and False, and '' doesn't work. I tried.

Fortunately Klaus had a tip - if you use os.dup(), you can open the file handle again, and - hey - now I can give it the option to open() we need, newline=''.

Lo and behold, it works!

runner_fixed.py:

#!/usr/bin/python3
 
import subprocess
import os
 
def run_command():
    p = subprocess.Popen("./command.py",
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT,
                         universal_newlines=False)  # \r goes through
 
    nice_stdout = open(os.dup(p.stdout.fileno()), newline='')  # re-open to get \r recognized as new line
    for line in nice_stdout:
        yield line, p.poll()
 
    yield "", p.wait()
 
 
for l, rc in run_command():
    print(l, end="", flush=True)

It is kind of kludgy to have to do that, but it does work:

I wonder why subprocess.Popen() in Python 3 does not have the same options for handling newlines as open() has. It feels like open() has moved on (to the newline parameter), but subprocess.Popen() hasn't followed yet.

IMHO, universal_newlines=True head up to text=True in Python 3.7, maybe...

Sincerely, Byung-Hee

- (황병희) 🕐︎ - 2019-12-30

+=

- 🕧︎ - 2021-04-14

+=

Add comment

To avoid spam many websites make you fill out a CAPTCHA, or log in via an account at a corporation such as Twitter, Facebook, Google or even Microsoft GitHub.

I have chosen to use a more old school method of spam prevention.

To post a comment here, you need to:

¹ Such as Thunderbird, Pan, slrn, tin or Gnus (part of Emacs).

Or, you can fill in this form:

+=