Run Node.js script from PHP - output is truncated to 512 characters - php

We run node.js CLI script from PHP with Symfony Process.
The script always print whole response as JSON in one line.
The response is somehow truncated on 512 characters.
I only found that xdebug.var_display_max_data => 512 => 512 in php.ini but don't see how this is related.
Adapter > Symfony Process > node script.js
A) Test Node script
from terminal node script $ node user-update.js parameters returns full result in all cases - like 629 chars.
from Symfony Process node script response is truncated to 512 chars.
B) Test Symfony Process
$process = new Process($cmd);
try {
$process->mustRun();
$response = $process->getOutput();
} catch (ProcessFailedException $e) {
$response = $e->getMessage();
}
echo $response;
echo PHP_EOL;
echo strlen($response);
$cmd = 'node user-update.js parameters'; - truncated to 512.
$cmd = 'php -r \'for($i=0; $i<520; $i++){ echo "."; }\''; - does not truncate.
$cmd = 'cat long_one_line.txt'; - print full file. 1650 chars in one line.
C) Try with PHP shell functions
$response = shell_exec($cmd); // response is truncated to 512
system($cmd, $returnVal); // print directly to STDOut, truncated to 512
What could be the cause and solution?
node v7.6.0
PHP 7.1.2

I suspect your process is ending before the buffer can be read by PHP.
As a work-around you can add something like this:
// The `| cat` at the end of this line means we wait for
// cat's process to end instead of node's process.
$process = new Process('node user-update.js parameters | cat');

Related

Adding numbers and getting a percentage

I have a PHP script that runs a few du commands to extract the kB size of 3 specific directories. That works, and I have $dir1size as 521046477, dir2size as 521043911 and $dir3size as 67167152. These pop out fine (ie no dodgy characters in the strings) if I echo the results into a JSON format.
I then want to add them up to get $tot and divide by the disk size which is $disksize to get $pctused.
The code I have is...
$rtn = array();
# Get disk size
$cmd = "df | sed -n '/DataVolume/s/ \+/ /gp' | cut -d' ' -f 2";
exec($cmd, $disksize, $int);
$rtn["disk"]["kB"]["total"] = $disksize[0];
# Get dir1 size
$cmd = "du --apparent-size .. | sed -n 's/[\ ]*\.\.\/TNFrontCam1$/ TNFront/p' | cut -d' ' -f 1";
exec($cmd, $dir1size, $int);
$rtn["disk"]["kB"]["TNFront"] = $dir1size[0];
# 2 more blocks like that for the other 2 directories
# Do the calulations
$tot = $dir1size + $dir2size + $dir3size;
$rtn["disk"]["kB"]["used"] = $tot;
$pctused = ($tot / $disksize) * 100;
$rtn["disk"]["percent"]["used"] = $pctused;
$rtn["disk"]["percent"]["free"] = 100 - $pctused;
echo json_encode($rtn, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES | JSON_NUMERIC_CHECK);
When I run this get Fatal Error: Unsupported operand types in the line calculating the percentage used.
If I comment out the 3 percentage lines I get the JSON string but instead of showing me the sum of the 3, I just get $dir1size but in square brackets! (hence the fatal error when it tries to do the percent calcs).
I've also tried intval($dirxsize) but then I get a null in the JSON output.
(I should add I am running this on a WD NAS drive that I've SSHed into in the hope that I can report usage info to my Home Assistant)
Edit - When I use PHP Sandbox, the maths all works out fine!
The second parameter of the exec function is an array (one element for each line of output), so if the size is on the first line you have to access the element at index zero, like:
$disksize[0]
For every variable used as the second argument of exec:
$tot = $dir1size[0] + $dir2size[0] + $dir3size[0];
$rtn["disk"]["kB"]["used"] = $tot;
$pctused = ($tot / $disksize[0]) * 100;

Unicode error when loading a json_encoded PHP array from memcached

I need to transfer a PHP associative array to further processing using python. The python code however using pylibmc is unable to load the string from memcached, throwing this error:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe0 in position 32: invalid continuation byte
I wrote a little tester. The PHP code to create the memcached data:
<?php
$mc = new Memcached();
$mc->addServer('localhost', 11211);
$data = array();
for ( $i = 0; $i < 100; $i++) {
$index = "ti" . $i;
$data += [$index => "test string $i"];
}
$mc->delete('test');
$mc->add('test', json_encode($data), 60);
$reverse = $mc->get('test');
echo "$reverse\n"; // prints {"ti0":"test string 0" ...... "ti99":"test string 99"} as expected
$reverse_array = json_decode($reverse, true);
echo $reverse_array['ti10'] . "\n";
//prints 'test string 10' as expected
?>
so this works fine writing to memcached from PHP and reading it back.
On the python side, this is the code I use to read it in:
#!/usr/bin/python
import pylibmc
import json
mc = pylibmc.Client(["127.0.0.1"], binary=True, behaviors={"cas": True, "tcp_nodelay": True,"ketama": True})
temp = json.loads(mc.get("test"))
When running the python code, this is the output I get:
Traceback (most recent call last):
File "./mctest.py", line 7, in <module>
temp = json.loads(mc.get("test")))
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe0 in position 32: invalid continuation byte
If I create a non-associative array in PHP and share that through memcached, things work fine.
Two further options I've tried:
adding utf8_encode to make sure it's properly encoded:
$mc->add('test', utf8_encode(json_encode($data)), 60);
adding JSON_UNESCAPED_UNICODE to the json_encode function:
$mc->add('test', json_encode($data, JSON_UNESCAPED_UNICODE), 60);
both result in identical outcomes on the python side.
Bit at a loss here - any ideas welcome!
While trying to determine the encoding of the resulting string retrieved from memcached via pymemcache, it occurred to me that the string didn't look like any known encoding, I confirmed this using chardet as well as cchardet.
After some more digging at the PHP end, I discovered that the PHP memcached module adulterates the strings it saves to memcached by compressing the data!
Solution was to add this line to the /etc/php/7.2/cli/conf.d/25-memcached.ini file:
memcached.compression_threshold=9999999999
Now the data comes into python as it should!

php runs terminal app with system()/shell_exec()/exec() but only half of the output shows up

so I have a php script:
<?php
error_reporting(E_ALL);
//print_r($_GET);
$command = '../../build/program_name ' . $_GET['arg_id'];
$command .= ' -test '.$_GET['arg1'].' '.$_GET['arg2'];
//echo $command . "\n";
//system('pwd');
//system('ls -la ../../build');
//system('../../build/program_name 17 -test 125 1500 2>&1');
//passthru('../../build/program_name 17 -test 125 1500');
system('../../build/program_name 17 -test 125 1500');
//system($command);
//$data = exec($command);
//var_dump($data);
//echo $data;
//echo "\n";
?>
simple version:
<?php
error_reporting(E_ALL);
system('../../build/program_name 17 -test 125 1500');
?>
The output from this is:
argv[0] = ../../build/program_name
argv[1] = 17
argv[2] = -test
argv[3] = 125
argv[4] = 1500
argc = 5 so appears to be test request.
TEST(125, 1500) test requested.
If I run the command in the terminal however the output is:
argv[0] = ../../build/program_name
argv[1] = 17
argv[2] = -test
argv[3] = 125
argv[4] = 1500
argc = 5 so appears to be test request.
TEST(125, 1500) test requested.
{ "function": "test" , "inputs": [125.000000, 1500.000000], "output": 999.000000}
the last bit of output (the important part) isn't showing up when php runs the exact same command... I've printed the working directory and the command just to verify that I'm in fact running the command correctly, the results are consistently the same across exec, shell_exec, and system.... I'm just at a loss for what is happening here...
Edit: additional info about 'program_name' heres an extremely simplified version. its c++:
float arg1;
float arg2;
if(argc == 5){
arg1 = atof(argv[3]);
arg2 = atof(argv[4]);
} else {
cout << "please supply appropriate args" << endl;
return -1;
}
#ifdef DEBUG
cout << "TEST(" << arg1 << ", " << arg2 << ") test requested." << endl;
#endif
float output = test(arg1, arg2);
string jsonOut = "{ ";
jsonOut.append(
"\"function\": \"test\" , ").append(
"\"inputs\": [").append(to_string(arg1)).append(", ").append(to_string(arg2)).append("], ").append(
"\"output\": ").append(to_string(output)).append("}");
cout << jsonOut << endl;
return 0;
Use passthru() instead of system(). The difference is that passthru() passes the command's output through to PHP's output, while system() captures the output and returns the last line.
The other output you're seeing is presumably written to stderr, which system() doesn't capture.
Okay this is insane, but fixed it. the program 'program_name' was saving a little log file using fprintf(). php would ignore all output from the program after the first call to fprintf()... commenting out fprintf() and recompiling program_name fixed the problem and now php doesn't bail in the middle, apparently php is incompatible to calls to fprintf() on OS X???

Different Output Characters on VBScript and PHP in Windows Command Prompt [duplicate]

I'm trying to generate mail configurations and personalized signatures through a batch file that reads a list of users, a template, and creates a personalized output. That's done and works:
#ECHO OFF
SETLOCAL ENABLEEXTENSIONS
GOTO begin
:writesignature
cscript //NoLogo replacetext.vbs "[NAME]" %1 signature.html stdout | cscript //NoLogo replacetext.vbs "[JOB]" %3 stdin stdout | cscript //NoLogo replacetext.vbs "[EMAIL]" %2 stdin signature-%4.html
GOTO :end
:begin
FOR /F "tokens=1,2,3,4 delims=;" %%A IN ('TYPE people.lst') DO CALL :writesignature "%%A" "%%B" "%%C" %%D
:end
To do the text replacing, I created replacetext.vbs, that allows me to replace a string for oter, and can be piped if stdin and stdout are indicated as the source and target files:
CONST ForReading = 1
CONST ForWritting = 2
CONST ForAppending = 8
CONST OpenAsASCII = false
CONST OpenAsUnicode = true
CONST OpenAsDefault = -2
Const OverwriteIfExist = true
Const FailIfExist = false
Const CreateIfNotExist = true
Const FailIfNotExist = false
SET objFSO = CreateObject("Scripting.FileSystemObject")
SET objFILEINPUT = Wscript.StdIn
SET objFILEOUTPUT = Wscript.StdOut
IF (Wscript.Arguments.Count < 2) OR (Wscript.Arguments.Count > 4) THEN
Wscript.Echo "Not enought arguments"
Wscript.Echo "replacetext ""<original>"" ""<replacement>"" "
Wscript.Quit(1 MOD 255)
END IF
IF Wscript.Arguments.Count > 2 THEN
IF Wscript.Arguments(2) = "stdin" THEN
' Wscript.Echo "Input: StdIn"
ELSE
' Wscript.Echo "Input: " + Wscript.Arguments(2)
SET objFILEINPUT = objFSO.OpenTextFile(Wscript.Arguments(2), ForReading, OpenAsASCII)
END IF
IF Wscript.Arguments.Count = 4 THEN
IF Wscript.Arguments(3) = "stdout" THEN
' Wscript.Echo "Output: StdOut"
ELSE
' Wscript.Echo "Output: " + Wscript.Arguments(3)
IF objFSO.FileExists(Wscript.Arguments(3)) THEN
SET objFILEOUTPUT = objFSO.OpenTextFile(Wscript.Arguments(3), ForWritting, CreateIfNotExist, OpenAsASCII)
ELSE
SET objFILEOUTPUT = objFSO.CreateTextFile(Wscript.Arguments(3), OverwriteIfExist, OpenAsASCII)
END IF
END IF
END IF
END IF
strText = objFILEINPUT.ReadAll()
strNewText = Replace(strText, Wscript.Arguments(0), Wscript.Arguments(1))
objFILEOUTPUT.Write(strNewText)
objFILEOUTPUT.Close
objFILEINPUT.Close
Wscript.Quit(0 MOD 255)
The problem is that when I put non-ASCII characters in ANSI/Windows-1250 in the people.lst (Comunicación), while it works and reads them in console, showing them (not converting them) as OEM characters (Comunicaci¾n) when I write the output files, somehow it does convert them transparently, so the output file in Windows shows Comunicaci¾n instead of Comunicación.
After much debugging, I've localized the problem in ONLY the arguments (no automatic conversion on the template file).
How can I disable said transparent conversion, or convert back the input from ANSI to OEM so the conversion works as intended?
The problem is that the cmd.exe works with different code page than cscript.exe/wscript.exe. I have similiar problem in Poland, where cmd.exe works with codepage 852 (I believe this is for compatibility with older MS-DOS programs) and wscript.exe works in Windows' native codepage 1250.
To solve the problem, put the following line on the beginning of the batch file:
mode con cp select=1250

Problem reading files greater than 1GB with XMLReader

Is there a maximum file size the XMLReader can handle?
I'm trying to process an XML feed about 3GB large. There are certainly no PHP errors as the script runs fine and successfully loads to the database after it's been run.
The script also runs fine with smaller test feeds - 1GB and below. However, when processing larger feeds the script stops reading the XML File after about 1GB and continues running the rest of the script.
Has anybody experienced a similar problem? and if so how did you work around it?
Thanks in advance.
I had same kind of problem recently and I thought to share my experience.
It seems that problem is in the way PHP was compiled, whether it was compiled with support for 64bit file sizes/offsets or only with 32bit.
With 32bits you can only address 4GB of data. You can find a bit confusing but good explanation here: http://blog.mayflower.de/archives/131-Handling-large-files-without-PHP.html
I had to split my files with Perl utility xml_split which you can find here: http://search.cpan.org/~mirod/XML-Twig/tools/xml_split/xml_split
I used it to split my huge XML file into manageable chunks. The good thing about the tool is that it splits XML files over whole elements. Unfortunately its not very fast.
I needed to do this one time only and it suited my needs, but I wouldn't recommend it repetitive use. After splitting I used XMLReader on smaller files of about 1GB in size.
Splitting up the file will definitely help. Other things to try...
adjust the memory_limit variable in php.ini. http://php.net/manual/en/ini.core.php
rewrite your parser using SAX -- http://php.net/manual/en/book.xml.php . This is a stream-oriented parser that doesn't need to parse the whole tree. Much more memory-efficient but slightly harder to program.
Depending on your OS, there might also be a 2gb limit on the RAM chunk that you can allocate. Very possible if you're running on a 32-bit OS.
It should be noted that PHP in general has a max file size. PHP does not allow for unsigned integers, or long integers, meaning you're capped at 2^31 (or 2^63 for 64 bit systems) for integers. This is important because PHP uses an integer for the file pointer (your position in the file as you read through), meaning it cannot process a file larger than 2^31 bytes in size.
However, this should be more than 1 gigabyte. I ran into issues with two gigabytes (as expected, since 2^31 is roughly 2 billion).
I've run into a similar issue when parsing large documents. What I wound up doing is breaking the feed into smaller chunks using filesystem functions, then parsing those smaller chunks... So if you have a bunch of <record> tags that you are parsing, parse them out with string functions as a stream, and when you get a full record in the buffer, parse that using the xml functions... It sucks, but it works quite well (and is very memory efficient, since you only have at most 1 record in memory at any one time)...
Do you get any errors with
libxml_use_internal_errors(true);
libxml_clear_errors();
// your parser stuff here....
$r = new XMLReader(...);
// ....
foreach( libxml_get_errors() as $err ) {
printf(". %d %s\n", $err->code, $err->message);
}
when the parser stops prematurely?
Using WindowsXP, NTFS as filesystem and php 5.3.2 there was no problem with this test script
<?php
define('SOURCEPATH', 'd:/test.xml');
if ( 0 ) {
build();
}
else {
echo 'filesize: ', number_format(filesize(SOURCEPATH)), "\n";
timing('read');
}
function timing($fn) {
$start = new DateTime();
echo 'start: ', $start->format('Y-m-d H:i:s'), "\n";
$fn();
$end = new DateTime();
echo 'end: ', $start->format('Y-m-d H:i:s'), "\n";
echo 'diff: ', $end->diff($start)->format('%I:%S'), "\n";
}
function read() {
$cnt = 0;
$r = new XMLReader;
$r->open(SOURCEPATH);
while( $r->read() ) {
if ( XMLReader::ELEMENT === $r->nodeType ) {
if ( 0===++$cnt%500000 ) {
echo '.';
}
}
}
echo "\n#elements: ", $cnt, "\n";
}
function build() {
$fp = fopen(SOURCEPATH, 'wb');
$s = '<catalogue>';
//for($i = 0; $i < 500000; $i++) {
for($i = 0; $i < 60000000; $i++) {
$s .= sprintf('<item>%010d</item>', $i);
if ( 0===$i%100000 ) {
fwrite($fp, $s);
$s = '';
echo $i/100000, ' ';
}
}
$s .= '</catalogue>';
fwrite($fp, $s);
flush($fp);
fclose($fp);
}
output:
filesize: 1,380,000,023
start: 2010-08-07 09:43:31
........................................................................................................................
#elements: 60000001
end: 2010-08-07 09:43:31
diff: 07:31
(as you can see I screwed up the output of the end-time but I don't want to run this script another 7+ minutes ;-))
Does this also work on your system?
As a side-note: The corresponding C# test application took only 41 seconds instead of 7,5 minutes. And my slow harddrive might have been the/one limiting factor in this case.
filesize: 1.380.000.023
start: 2010-08-07 09:55:24
........................................................................................................................
#elements: 60000001
end: 2010-08-07 09:56:05
diff: 00:41
and the source:
using System;
using System.IO;
using System.Xml;
namespace ConsoleApplication1
{
class SOTest
{
delegate void Foo();
const string sourcepath = #"d:\test.xml";
static void timing(Foo bar)
{
DateTime dtStart = DateTime.Now;
System.Console.WriteLine("start: " + dtStart.ToString("yyyy-MM-dd HH:mm:ss"));
bar();
DateTime dtEnd = DateTime.Now;
System.Console.WriteLine("end: " + dtEnd.ToString("yyyy-MM-dd HH:mm:ss"));
TimeSpan s = dtEnd.Subtract(dtStart);
System.Console.WriteLine("diff: {0:00}:{1:00}", s.Minutes, s.Seconds);
}
static void readTest()
{
XmlTextReader reader = new XmlTextReader(sourcepath);
int cnt = 0;
while (reader.Read())
{
if (XmlNodeType.Element == reader.NodeType)
{
if (0 == ++cnt % 500000)
{
System.Console.Write('.');
}
}
}
System.Console.WriteLine("\n#elements: " + cnt + "\n");
}
static void Main()
{
FileInfo f = new FileInfo(sourcepath);
System.Console.WriteLine("filesize: {0:N0}", f.Length);
timing(readTest);
return;
}
}
}

Categories