Why piping output of docker-compose exec to grep, breaks it? - php

Question

I'm running this command to run Drush which is basically a PHP CLI for Drupal, in the running container:
docker-compose -f ../docker-compose.test.yml exec php scripts/bin/vendor/drush.phar -r public_html status-report
The output if this command is fine, it's the list of status information about a specific Drupal instance in the container. I won't be pasting it here as it's long, and irrelevant.
Now let's filter this information by piping it into grep:
docker-compose -f ../docker-compose.test.yml exec php scripts/bin/vendor/drush.phar -r public_html status-report | grep -e Warning -e Error
The result is:
Cro Error L
Gra Warning P
HTT Error F
HTT Warning T
Dru Warning N
XML Error L
Which is wrong, it looks like it has been cut to pieces, and most of it is missing.
Now, if we will disable allocating of pseudo-tty by adding -T flag:
docker-compose -f ../docker-compose.test.yml exec -T php scripts/bin/vendor/drush.phar -r public_html status-report | grep -e Warning -e Error
The output is correct:
Cron maintenance Error Last run 3 weeks 1 day ago
Gravatar Warning Potential issues
HTTP request status Error Fails
HTTPRL - Non Warning This server does not handle hanging
Drupal core update Warning No update data available
XML sitemap Error Last attempted generation on Tue, 04/18/2017
Why is that?
Bonus question, which probably will be answered by the answer to the previous one: Are there any important side effects of using -T?
Docker version 18.06.1-ce, build e68fc7a215
docker-compose version 1.22.0
UPDATE #1:
To simplify things I saved the correct output of the whole scripts/bin/vendor/drush.phar -r public_html status-report into a file test.txt and tried:
docker-compose -f ../docker-compose.test.yml exec php cat test.txt | grep -e Warning -e Error
Interestingly the output is correct now with and witout -T, so it has to have something to do with Drush/php, although I'm still interested what can be a cause of this.
PHP 7.1.12 (cli) (built: Dec 1 2017 04:07:00) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies
with Zend OPcache v7.1.12, Copyright (c) 1999-2017, by Zend Technologies
with Xdebug v2.5.5, Copyright (c) 2002-2017, by Derick Rethans
Drush 8.1.17
UPDATE #2:
To isolate problem further I put all content in a PHP file, that is simply printing it, and after:
docker-compose -f ../docker-compose.test.yml exec php php php.php | grep -e Warning -e Error
I'm getting a correct output!
So it has to have something to do with how Drush is printing its messages, but I fail to see what it can be. That could be pretty interesting if we could figure this out.
UPDATE #3:
Ok guys, that's actual magic. The problem happens also with running drush without any commands, to list all available ones. The list of commands is broken when output is being piped, so this can be tested without actual Drupal instance.
Now I want to present you magic.
In drush, output for list of available commands in being generated in commands/core/help.drush.phpin function drush_core_help(). There is this call: drush_help_listing_print($command_categories); I looked into it. Inside is a call drush_print_table($rows, FALSE, array('name' => 20)); that is responsible for generating part of the output that's getting broken.
So inside of it, I decided to intercept the output, just before the last call to drush_print(), by adding simple file_put_contents('/var/www/html/data.txt', $output);
And now it's time for the absolutely magical part for me.
When I execute:
docker-compose -f ../docker-compose.test.yml exec php scripts/bin/vendor/drush/drush -r public_html
The last group of commands can be checked in this file, and in my case it's:
adminrole-update Update the administrator role permissions.
elysia-cron Run all cron tasks in all active modules for specified site using elysia cron system. This replaces the standard "core-cron" drush handler.
generate-redirects Create redirects.
libraries-download Download library files of registered libraries.
(ldl, lib-download)
libraries-list (lls, Show a list of registered libraries.
lib-list)
BUT, if I execute the same command, but the output will be piped or redirected, so for example:
docker-compose -f ../docker-compose.test.yml exec php scripts/bin/vendor/drush/drush -r public_html | cat
SOMETHING DIFFERENT WILL BE SAVED INTO A FILE:
adminrole-update U
p
d
a
t
e
t
h
e
a
d
m
i
n
i
s
t
r
a
t
o
r
r
(and the rest of the broken output)
So the fact of piping/redirecting of the output, influences execution of the command, before the pipe/redirection actually happens.
How is that even possible? O_o

Answer 1

It's not uncommon for a command-line program to change its output presentation based on whether its output is a terminal, or not. For example, ls by itself, with no options, displays files in a columnar format. When piped, the output changes to a list of one-file-per-line. You can see this in the source code for GNU ls:
case LS_LS:
/* This is for the 'ls' program. */
if (isatty (STDOUT_FILENO))
{
format = many_per_line;
set_quoting_style (NULL, shell_escape_quoting_style);
/* See description of qmark_funny_chars, above. */
qmark_funny_chars = true;
}
else
{
format = one_per_line;
qmark_funny_chars = false;
}
break;
You can emulate the behavior of ls | ... with the explicit argument ls -1, and this too is not uncommon: programs that implicitly change their output presentation often provide a way to explicitly engage that alternate presentation.
Support for this isn't just a convention: it's actually a requirement for ls in POSIX:
The default format shall be to list one entry per line to standard output; the exceptions are to terminals or when one of the -C, -m, or -x options is specified. If the output is to a terminal, the format is implementation-defined.
This all may seem magical: how does ls know it's got a pipe following it since it comes before the pipe? The answer is quite simple, really: the shell parses the whole command line, sets up the pipes, and then forks the respective programs with the input/output wired to pipes appropriately.
So, what part of the command is doing the alternate presentation? I suspect it's an interaction between the environment of your exec and the column width calculation in drush. On my local environment, drush help | ... doesn't produce any unusual results. You might try piping to (or through) cat -vet to discover any unusual characters in the output.
That said, regarding docker-compose specifically: based on this thread, you're not the only one who has encountered this or a similar issue. I've not trawled the docker source code, but - generally - not allocating a pseudo-tty will make the other end act like a non-interactive shell, which means things like your .bash_profile won't run and you won't be able to read stdin in the run command. This can give the appearance of things not working.
The thread linked above mentions a work around of this form:
docker exec -i $(docker-compose ...) < input-file
which seems reasonable given the meaning of -i, but it also seems rather convoluted for basic scripting.
The fact that -T makes it work for you suggests to me that you have something in your .bash_profile (or similar login-shell-specific start up file) that's changing certain values (maybe COLUMNS) or altering the values in such a way as to have the observed deleterious effect. You might try removing everything from those files, then adding them back to see if any particular one causes the issue.

Answer 2

I didn't read that very detailed question, but from glancing over it, I'd say the -T option to the exec subcommand is essential if you want to process stdout and stderr in the environment where you execute docker-compose.