I would to like to get contents from each url in a list using fread and Fibers where each stream does not need to wait a feof to run another fread in another url
My current code is the follow:
<?php
function getFiberFromStream($stream, $url): Fiber {
return new Fiber(function ($stream) use ($url): void {
while (!feof($stream)) {
echo "reading 100 bytes from $url".PHP_EOL;
$contents = fread($stream, 100);
Fiber::suspend($contents);
}
});
}
function getContents(array $urls): array {
$contents = [];
foreach ($urls as $key => $url) {
$stream = fopen($url, 'r');
stream_set_blocking($stream, false);
$fiber = getFiberFromStream($stream, $url);
$content = $fiber->start($stream);
while (!$fiber->isTerminated()) {
$content .= $fiber->resume();
}
fclose($stream);
$contents[$urls[$key]] = $content;
}
return $contents;
}
$urls = [
'https://www.google.com/',
'https://www.twitter.com',
'https://www.facebook.com'
];
var_dump(getContents($urls));
Unfortunatelly, the echo used in getFiberFromStream() are showing that this current code is waiting to get the entire content from a url to go to next one:
reading 100 bytes from https://www.google.com
reading 100 bytes from https://www.google.com
reading 100 bytes from https://www.google.com //finished
reading 100 bytes from https://www.twitter.com
reading 100 bytes from https://www.twitter.com
reading 100 bytes from https://www.twitter.com //finished
reading 100 bytes from https://www.facebook.com
[...]
I would like something like:
reading 100 bytes from https://www.google.com
reading 100 bytes from https://www.twitter.com
reading 100 bytes from https://www.facebook.com
reading 100 bytes from https://www.google.com
reading 100 bytes from https://www.twitter.com
reading 100 bytes from https://www.facebook.com
[...]
The behaviour you see is because you poll the current fiber till full completion before go onto next fiber.
Solution here is to start all fibers for all urls at once and only after that do poll them.
Try something like this:
function getContents(array $urls): array {
$contents = [];
$fibers = [];
// start them all up
foreach ($urls as $key => $url) {
$stream = fopen($url, 'r');
stream_set_blocking($stream, false);
$fiber = getFiberFromStream($stream, $url);
$content = $fiber->start($stream);
// save fiber context so we can process them later
$fibers[$key] = [$fiber, $content, $stream];
}
// now poll
$have_unterminated_fibers = true;
while ($have_unterminated_fibers) {
// first suppose we have no work to do
$have_unterminated_fibers = false;
// now loop over fibers to see if any is still working
foreach ($fibers as $key => $item) {
// fetch context
$fiber = $item[0];
$content = $item[1];
$stream = $item[2];
// don't do while till the end here,
// just process next chunk
if (!$fiber->isTerminated()) {
// yep, mark we still have some work left
$have_unterminated_fibers = true;
// update content in the context
$content .= $fiber->resume();
$fibers[$key][1] = $content;
} else {
if ($stream) {
fclose($stream);
// save result for return
$contents[$urls[$key]] = $content;
// mark stream as closed in context
// so it don't close twice
$fibers[$key][2] = null;
}
}
}
}
return $contents;
}
I am using the following code to pull a CSV file from a website that i do not have control over. and many of times i get the undefined index or headers already sent but all the data is there at the bottom. i want to write a script to open the file and remove all lines until it gets to the actual header line that should be in a csv.
the # of lines changes every time i pull it...
the current example has 49107 lines that i don't need before the part i want to parse.. This is a small part of the first 15 lines of code and about 20 lines of code before what i REALLY WANT from the file.
<pre class="cake-debug"><b>Notice</b> (8): Undefined index: name [<b>APP/controllers/loads_controller.php</b> line <b>327</b>]<div id="cakeErr1-trace" class="cake-stack-trace" style="display: none;">Code | Context<div id="cakeErr1-code" class="cake-code-dump" style="display: none;"><pre><code><span style="color: #000000"> $data[$i]['Load']['drop_date'] = date('m/d/Y' strtotime($value['Load']['drop']));</span></code>
<code><span style="color: #000000"> $data[$i]['Load']['pickup_city'] = $value['Pickup']['city'];</span></code>
"<span class=""code-highlight""><code><span style=""color: #000000""> $data[$i]['Load']['pickup_state'] = $value['Pickup']['State']['name'];</span></code></span></pre></div><pre id=""cakeErr1-context"" class=""cake-context"" style=""display: none;"">$order = ""Load.load_number ASC"""
"$fields = array("
" ""*"""
)
"$conditions = array("
" ""Load.active"" => true"
)
"$results = array("
" array("
" ""Load"" => array()"
" ""Pickup"" => array()"
" ""Destination"" => array()"
)
$result = array(
"Load" => array(
"name" => "ICE CREAM OR RELATED",
"load_number" => "8891517",
"trailer_type" => "R",
"phone_number1" => "800-555-8287",
"phone_number2" => "800-555-8287",
"pickup_date" => "03/09/2014",
"drop_date" => "03/09/2014",
"pickup_city" => "Indianapolis",
"pickup_state" => "Indiana",
"pickup_zipcode" => "46201",
"destination_city" => "London",
"destination_state" => "Kentucky",
"destination_zipcode" => "40741"
)
)
$fp=</pre><pre class="stack-trace">header - [internal], line ??
LoadsController::csv() - APP/controllers/loads_controller.php, line 360
Dispatcher::_invoke() - CORE/cake/dispatcher.php, line 204
Dispatcher::dispatch() - CORE/cake/dispatcher.php, line 170
[main] - APP/webroot/index.php, line 83</pre></div>
</pre>name,load_number,trailer_type,phone_number1,phone_number2,pickup_date,drop_date,pickup_city,pickup_state,pickup_zipcode,destination_city,destination_state,destination_zipcode
"FOOD OR KINDRED PROD",8831029,R,800-555-8287,800-555-8287,03/09/2014,03/10/2014,Aurora,Illinois,60504,"West Memphis",Arkansas,72301
"FOOD OR KINDRED PROD",8831031,R,800-555-8287,800-555-8287,03/12/2014,03/13/2014,Aurora,Illinois,60504,Ashley,Indiana,46705
This is how I would like the file to look after removing the lines that should not be there...
name,load_number,trailer_type,phone_number1,phone_number2,pickup_date,drop_date,pickup_city,pickup_state,pickup_zipcode,destination_city,destination_state,destination_zipcode
FOOD OR KINDRED PROD,8831029,R,800-555-8287,800-555-8287,3/9/2014,3/10/2014,Aurora,Illinois,60504,West Memphis,Arkansas,72301
FOOD OR KINDRED PROD,8831031,R,800-555-5555,800-555-5555,3/12/2014,3/13/2014,Aurora,Illinois,60504,Ashley,Indiana,46705
Currently i am using this code to get my CSV
set_time_limit (24 * 60 * 60);
// folder to save downloaded files to. must end with slash
$destination_folder = 'downloads/';
$url = 'http://www.somesite.com/loads/csv';
$newfname = $destination_folder . 'loads1.csv';
$file = fopen ($url, "rb");
if ($file) {
$newf = fopen ($newfname, "wb");
if ($newf)
while(!feof($file)) {
fwrite($newf, fread($file, 1024 * 8 ), 1024 * 8 );
}
}
if ($file) {
fclose($file);
}
if ($newf) {
fclose($newf);
}
and this Code to parse it
$selectfile1 = "https://www.somesite.com/downloads/loads1.csv";
// check mime type - application/octet-stream
$content = file($selectfile1);
$posted_content = array();
list($rownum, $row) = each($content);
$posted_content[0] = explode(",", $row);
array_push($posted_content[0], "ID");
$count = 0;
// iterate each row (1 post)
while (list($rownum, $row) = each($content))
{
$count++;
$cols = "ShipAfterDate, ShipBeforeDate, EquipmentID, LengthID, VendorCode, LoadCount, Rate, CargoDescription, Notes,Phone1, Phone2, PostDate,";
$vals = "";
// extract fields from row columns
$items = explode(",", $row);
list( $Description, $OrderNumber, $EquipmentCode, $Phone1, $Phone2, $ShipDate, $DeliveryDate, $OriginCity, $OriginState, $OriginZip, $DestinationCity, $DestinationState, $DestinationZip
) = $items;
array_push($posted_content, $items);
Check out 'fgetcsv' (PHP manual) which just returns false if there's a parse error or the actual CSV values if not. It might be not the fastest solution to unsuccessfully parse 50k lines, but I think it should work nevertheless
I am using stream_get_line to store some php output in a variable, while I'm running a telnet session via fsockopen.
However, my second server does not run PHP5, which is disabled the ability to use stream_get_line. Is there any alternative for PHP 4.3?
I heard that fgets is almost the same, but I don't seem to get it to work exactly like stream_get_line.
Code:
...
# opening connection
$fp = #fsockopen($ip, 23, $errno, $errstr, 8);
# loggin in
fputs($fp, "$user\r");
usleep(250000);
fputs($fp, "$password\r");
# getting information
fputs($fp, "show info\n");
usleep(250000);
fputs($fp, "show info 2\n");
usleep(250000);
fputs($fp, "show info 3\n");
usleep(250000);
fputs($fp, "show info 4\n");
usleep(250000);
fputs($fp, "?\n");
$content = stream_get_line($fp, 0, "?");
$contentvalues = array(
1 => substr($content, 130, 3),
2 => substr($content, 180, 3)
);
fclose($fp);
...
(I am storing specific parts of my output in the $contentvalues variable.)
From the docs:
This function is nearly identical to fgets() except in that it allows
end of line delimiters other than the standard \n, \r, and \r\n, and
does not return the delimiter itself.
From the comments:
when fgets reads some bytes from socket, where EOF is reached, it
returns bool(false) same as stream_get_line
BUT if remote client drops connection, and server script will try to
read some data with function fgets, function will return bool(false),
and stream_get_line will return string(0) ""
so you can detect remote client disconnection with stream_get_line,
and cannot with fgets
There's also some dithering about which function is faster, but it seems to be dependant on the version of PHP, the day of the week, and what the commenter had for dinner the previous night.
edit
Judging by your response to Steffen's answer you're hung up on the fact that fgets() does not take a third parameter as a delimiter. Applying a basic input loop and checking the string will get you there. Also, in Steffen's defense, you were never quite clear on in your question, stating only that it doesn't "work exactly like stream_get_line".
<?php
$delim = '?';
$buf = 4096;
$fp = #fsockopen($ip, 23, $errno, $errstr, 8);
// ... yadda yadda yadda ... //
$content = '';
while( $part = fgets($fp, $buf) ) {
$ind = strpos($part, $delim);
if( $ind !== false ) {
$content .= substr($part, 0, $ind);
break;
}
$content .= $part;
}
Also, even with stream_get_line() you should be using a loop to get the input as a length parameter or 0 does not mean "unlimited", but rather will use one of PHP's defaults which is 8192 bytes.
You can use fgets() (string fgets ( resource $handle [, int $length ] )) instead.
http://www.php.net/manual/de/function.fgets.php
A service is listening to port 1234 on 1xx.xxx.xx.xx
I'm using php fsockopen() to establish a tcp connection with the service! I've to send incoming data to the service and save the reply I got from the service in a file.
First time when I send data, the connection is established. The second time the code again tries to open the port and this time the service says, SP tries to connect. connection refused. An active SP connection exists. SP closed.
How do I overcome this problem?
#!/usr/local/php5/bin/php-cgi
<?php
//The Client
error_reporting(E_ALL);
$CONTENT = $_GET["DATA"]."";
echo urldecode($CONTENT);
$Handle = fopen("/xxx/xxx/xxx.txt", "a");
fwrite($Handle, $CONTENT);
fclose($Handle);
//$address = "1xx.xxx.xx.xx";
//$port = 1234;
/* Create a TCP/IP socket. */
$fp = fsockopen("tcp://1xx.xxx.xx.xx",1234 , $errno, $errdesc);
if ( ! $fp ) {
die ( "Couldn't connect to 1xx.xxx.xx.xx :\nError: $errno\nDesc: $errdesc\n" );
}
fputs ( $fp, $CONTENT );
while ( ! feof( $fp ) ) {
$output = fgets( $fp, 2048 );
}
fclose( $fp );
$Handle1 = fopen("/xxx/xxx/yyy.txt", "a");
fwrite($Handle1, $output);
fclose($Handle1);
?>
One more note:
You need to replace
while ( ! feof( $fp ) ) {
$output = fgets( $fp, 2048 );
}
with
$output=''
while ( ! feof( $fp ) ) {
$output .= fgets( $fp, 2048 );
}
Because $output = fgets( $fp, 2048 ); overwrites $output content in each iteration loop.
About your question - I think there is a some error in a service.
I'm working on the code below to allow HTTP user agents that cannot perform XSL transformations to view the resources on my server. I'm mystified because the result of transformToXML is false, but the result of libxml_get_errors() is an empty array. As you can see, the code outputs the LibXSLT version ID and I'm getting the problem on WinVista with version 1.1.24. Is libxml_get_errors() not the right function to get the errors from the XSLTProcessor object?
If you're interested in the XML documents, you can get them from http://bobberinteractive.com/index.xhtml and .../stylesheets/layout.xsl
<?php
//redirect browsers that can handle the source files.
if (strpos ( $_SERVER ['HTTP_ACCEPT'], 'application/xhtml+xml' )) {
header ( "HTTP/1.1 301 Moved Permanently" );
header ( "Location: http://" . $_SERVER ['SERVER_NAME'] . "/index.xhtml" );
header ( "Content-Type: text/text" );
echo "\nYour browser is capable of processing the <a href='/index.xhtml'> site contents on its own.";
die ();
}
//start by checking the template
$baseDir = dirname ( __FILE__ );
$xslDoc = new DOMDocument ();
if (! $xslDoc->load ( $baseDir . '/stylesheets/layout.xsl' )) {
header ( "HTTP/1.1 500 Server Error" );
header ( "Content-Type: text/plain" );
echo "\n Can't load " . $baseDir . '/stylesheets/layout.xsl';
die ();
}
//resolve the requested resource (browsers that need transformation will request the resource without the suffix)
$uri = $_SERVER ['REQUEST_URI'];
$len = strlen ( $uri );
if (1 >= $len || '/' == substr ( $uri, $len - 1 )) {
$fileName = $baseDir . "/index.xhtml"; // use 'default' document if pathname ends in '/'
} else {
$fileName = $baseDir . (1 load ( $fileName )) {
header ( "HTTP/1.1 500 Server Error" );
echo "\n Can't load " . $fileName;
die ();
}
// now start the XSL template processing
$proc = new XSLTProcessor ();
$proc->importStylesheet ( $xslDoc );
$doc = $proc->transformToXML ( $xmlDoc );
if (false === $doc) {
header ( "HTTP/1.1 500 Server Error" );
header ( "Content-Type: text/plain" );
echo "\n";
// HERE is where it gets strange: the value of $doc is false and libxml_get_errors returns 0 entries.
display_xml_errors ( libxml_get_errors() );
die ();
}
header ( "Content-Type: text/html" );
echo "\n";
echo $doc;
function display_xml_errors($errors) {
echo count ( $errors ) . " Error(s) from LibXSLT " . LIBXSLT_DOTTED_VERSION;
for($i = 0; $i level) {
case LIBXML_ERR_WARNING :
$return .= "Warning $error->code: ";
break;
case LIBXML_ERR_ERROR :
$return .= "Error $error->code: ";
break;
case LIBXML_ERR_FATAL :
$return .= "Fatal Error $error->code: ";
break;
}
$return .= trim ( $error->message ) . "\n Line: $error->line" . "\n Column: $error->column";
if ($error->file) {
$return .= "\n File: $error->file";
}
echo "$return\n\n--------------------------------------------\n\n";
}
}
I'm getting several errors when loading your XML or executing your XSL. Upgrade your error_reporting level
error_reporting(E_ALL | E_STRICT);
// or
error_reporting(-1); // overzealous, but works
Got me:
PHP Warning: DOMDocument::load(): Entity 'ndash' not defined in /tmp/index.xhtml, line: 8 in /tmp/test.php on line 4
PHP Warning: XSLTProcessor::importStylesheet(): compilation error: file /tmp/layout.xsl line 59 element a in /tmp/test.php on line 10
PHP Warning: XSLTProcessor::importStylesheet(): Attribute 'onclick': The content is expected to be a single text node when compiling an AVT. in /tmp/test.php on line 10
PHP Warning: XSLTProcessor::importStylesheet(): compilation error: file /tmp/layout.xsl line 185 element a in /tmp/test.php on line 10
PHP Warning: XSLTProcessor::importStylesheet(): Attribute 'onclick': The content is expected to be a single text node when compiling an AVT. in /tmp/test.php on line 10
PHP Warning: XSLTProcessor::transformToXml(): No stylesheet associated to this object in /tmp/test.php on line 12