Removing allowable whitespace in PHP using parser tokens - php

Question

I am trying to make a simple script that will remove all unnecessary whitespace from a PHP file/string.
I have been successful at parsing the string using tokens, but don't see a good method to remove "extra" whitespace.
For instance,
function test() { return TRUE; }
should be
function test(){return TRUE;}
and NOT
functiontest(){returnTRUE;}
You end up with the last version if you just remove the T_WHITESPACE token.
Is there something I am missing to remove whitespace, but keep spaces after things like "function " and "return ". Thank you!

Answer 1

$newSource = '';
foreach (token_get_all($source) as $i => $token) {
if (!is_array($token)) {
$newSource .= $token;
}
if ($token[0] == T_WHITESPACE) {
if ( isset($tokens[$i - 1]) && isset($tokens[$i + 1])
&& is_array($tokens[$i - 1]) && is_array($tokens[$i + 1])
&& isLabel($tokens[$i - 1][1]) && isLabel($tokens[$i + 1][1])
) {
$newSource .= ' ';
}
} else {
$newSource .= $token[1];
}
}
function isLabel($str) {
return preg_match('~^[a-zA-Z0-9_\x7f-\xff]+$~', $str);
}
Removing the whitespace is always allowed, apart from the case where there is a LABEL on both sides of it. I check that and either add nothing or a single space character.
There is just another special case I know about, there whitespace is of importance: T_END_HEREDOC must be followed by either ; or \n. Compacting or stripping the space here is not allowed. So, if that is of importance to you, you can simply add that ;)

Answer 2

Well, T_WHITESPACE can be spaces or newlines, etc. So one trivial approach would be to automatically replace all T_WHITESPACE instances with a new one that consists of exactly one space.
But for a smarter method, just go through the list of parser tokens, and figure out which ones should have a whitespace after it and which should not (something like this):
foreach ($tokens as $k => $val) {
if (is_array($val) && $val[0] == T_WHITESPACE) {
if (!is_array($tokens[$k - 1])) {
//remove this space
} else {
switch ($tokens[$k - 1][0]) {
case T_ABSTRACT:
case T_FUNCTION:
//.. other keeps here:
continue;
break;
default:
//remove the space
}
}
}
}
And one more note, don't do this for performance. If you're using an OPCODE Cache (Such as APC) you'll see no benefit for a lot of work. If you're not using one, why aren't you?

Answer 3

Your effort is futile.
php -w
Allows already to strip whitespace off scripts. It uses a bit more elaborate logic to remove whitespace from the token stream.
Here's the zend_strip() function as found in zend_highlight.c:
while ((token_type=lex_scan(&token TSRMLS_CC))) {
switch (token_type) {
case T_WHITESPACE:
if (!prev_space) {
zend_write(" ", sizeof(" ") - 1);
prev_space = 1;
}
/* lack of break; is intentional */
case T_COMMENT:
case T_DOC_COMMENT:
token.type = 0;
continue;
case T_END_HEREDOC:
zend_write(LANG_SCNG(yy_text), LANG_SCNG(yy_leng));
efree(token.value.str.val);
/* read the following character, either newline or ; */
if (lex_scan(&token TSRMLS_CC) != T_WHITESPACE) {
zend_write(LANG_SCNG(yy_text), LANG_SCNG(yy_leng));
}
zend_write("\n", sizeof("\n") - 1);
prev_space = 1;
token.type = 0;
continue;
default:
zend_write(LANG_SCNG(yy_text), LANG_SCNG(yy_leng));
break;
}
if (token.type == IS_STRING) {
switch (token_type) {
case T_OPEN_TAG:
case T_OPEN_TAG_WITH_ECHO:
case T_CLOSE_TAG:
case T_WHITESPACE:
case T_COMMENT:
case T_DOC_COMMENT:
break;
default:
efree(token.value.str.val);
break;
}
}
prev_space = token.type = 0;
}