Making subprocess.Popen in Python 3 play nice with "elaborate" output #programming #python
docker-compose produces "elaborate" output when run, making use of carriage returns and ANSI CSI codes to move the cursor about.
A wrapper script written in Python 3 tried to handle the output - printing a '.' for each line usually, and actually printing the lines if given a --verbose
option.
Unfortunately the output with --verbose
got mangled, which really annoyed me. So I tried to find a solution.
Here is a small script that does some output, mimicking the stuff that docker-compose outputs. Note that we want the lines printed as they are produced, and not all at once at the end of the script.
#!/usr/bin/python3 from time import sleep print("1", flush=True) sleep(1) print("2", flush=True) sleep(1) print("3", flush=True) sleep(1) print("\x1b[2A\x1b[K\r", end="", flush=True) sleep(1) print("2 - let's go\r", end="", flush=True) sleep(1) print("2 - lookin' good\r", end="", flush=True) sleep(1) print("\x1b[1A\x1b[K\r", end="", flush=True) sleep(1) print("1 - abba\r", end="", flush=True) sleep(1) print("\x1b[2B\r", end="", flush=True) sleep(1) print("\r3 - flappa\r", end="", flush=True) sleep(1) print("\x1b[1A\x1b[K\r", end="", flush=True) sleep(1) print("2 - done\r", end="", flush=True) sleep(1) print("\x1b[1A\x1b[K\r", end="", flush=True) sleep(1) print("1 - done\r", end="", flush=True) sleep(1) print("\x1b[2B\r", end="", flush=True) sleep(1) print("3 - done\x1b[K\r", end="", flush=True) sleep(1) print("\nFinito", flush=True)
Try running it:
The original code wrapping it, was like this (slightly simplified):
#!/usr/bin/python3 import subprocess def run_command(): p = subprocess.Popen("./command.py", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True) # this converts \r into \n #fail for line in iter(p.stdout.readline, ""): yield line, p.poll() yield "", p.wait() for l, rc in run_command(): print(l, end="", flush=True)
If you run this, you'll see how the output it mangled:
So why does this happen? Well, the first culprit is universal_newlines=True
. That means that any combination of carriage return and/or line feed is interpreted as and converted to a line feed. Uh-oh, definitely not what we want, if the fancy output it to be reproduced as intended.
If set to False
, the situation improves, carriage returns are no longer converted. Unfortunately the p.stdout.readline()
function only interprets line feeds as end of line and not carriage returns, so all the fancy stuff piles up, and is only shown at the very end. Not what we want.
Looking at the documentation of open()
reveals that it has an option called newline
, which is used to control how universal newlines are handled, and that if set to ''
it actually does what we want: carriage returns are not converted, and they are recognized and line endings.
Unfortunately the documentation of subprocess.Popen()
only documents universal_newlines
taking two values, True
and False
, and ''
doesn't work. I tried.
Fortunately Klaus had a tip - if you use os.dup()
, you can open the file handle again, and - hey - now I can give it the option to open()
we need, newline=''
.
Lo and behold, it works!
#!/usr/bin/python3 import subprocess import os def run_command(): p = subprocess.Popen("./command.py", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=False) # \r goes through nice_stdout = open(os.dup(p.stdout.fileno()), newline='') # re-open to get \r recognized as new line for line in nice_stdout: yield line, p.poll() yield "", p.wait() for l, rc in run_command(): print(l, end="", flush=True)
It is kind of kludgy to have to do that, but it does work:
I wonder why subprocess.Popen()
in Python 3 does not have the same options for handling newlines as open()
has. It feels like open()
has moved on (to the newline
parameter), but subprocess.Popen()
hasn't followed yet.
IMHO, universal_newlines=True head up to text=True in Python 3.7, maybe...
Sincerely, Byung-Hee
- (황병희) 🕑︎ - 2019-12-30
- 🕜︎ - 2021-04-14