interrupt a function by other function - php

i have a symfony job that have two function:
launch and stop.
My launch function will import contacts,for example 4 by 4 from database and send to all of them messages.
public function launchAction()
{
$offset = 0;
$limit = 4;
$sizeData /= $limit;
for( $i = 0; $i < $sizeData; $i++)
{
$contacts = $repository->getListByLimit($offset, $limit);
$sender->setContacts($contacts);
$sender->send();
$offset += $limit;
}
}
when i launched my launch function it will take for example 20 seconds to import and send the message to all contacts
but if i want to stop it,how can the stop function interrupt the launch function
public function stopAction()
{
}

i will not fully answer but give you two hints how it could work
1:
save a file with process id on launch()
on stop() you could check for existence and kill the process by id
2:
on launch() you can check for a specific db-entry in loop so it breaks if value is present
on stop you set this db entry

If your only purpose is to be able to stop the script, you don't need a full event loop implementation I think. You can listen to a local socket, and break when you receive data.
You could for example run this in launchAction
public function launchAction()
{
$offset = 0;
$limit = 4;
$sizeData /= $limit;
// Init IPC connection
$server = stream_socket_server("tcp://127.0.0.1:1337", $errno, $errorMessage);
if ($server === false) {
throw new UnexpectedValueException("Could not bind to socket: $errorMessage");
}
for( $i = 0; $i < $sizeData; $i++)
{
// Check our socket for data
$client = #stream_socket_accept($server);
if ($client) {
// Read sent data
$data = fread($client, 1024);
// Probably break
if ($data === 'whatever') {
break;
}
}
$contacts = $repository->getListByLimit($offset, $limit);
$sender->setContacts($contacts);
$sender->send();
$offset += $limit;
}
// Close socket after sending all messages
fclose($client);
}
And stopAction could hit the socket to terminate the connection like so:
public function stopAction()
{
$socket = stream_socket_client('tcp://127.0.0.1:1337');
fwrite($socket, 'whatever');
fclose($socket);
}
This should work if you run both functions on the same machine. Also note that PHP can only listen to sockets which are not occupied already. So you might need to change the port number. And in case you start a second process to send messages in parallel, the new one will not be able to bind to the same socket.
A great blogpost explaining some detail is https://www.christophh.net/2012/07/24/php-socket-programming/
If however you wish to start a long running process, I suggest you take a look at ReactPHP, which is an excellent event loop implementation that runs on several different setups. It also includes timers, and other useful libs.
You might want to take a look at this blogpost series, to get an idea https://blog.wyrihaximus.net/2015/01/reactphp-introduction/

Related

Connection errors in forked PHP processes

I have a PHP script that takes N documents from MongoDB, forks the process into K child PHP processes, each process does some things with each document and tries to update document's info (see the code below).
On my local environment (Docker) everything is cool, but on the server (no Docker there) sometimes during the loop strange things happen...
Randomly all forked processes can not connect to MongoDB. The updateOne command returns an error :
"Failed to send "update" command with database "databasename": Invalid reply from server. in /vendor/mongodb/mongodb/src/Operation/Update.php on line 158".
This happens to all processes at the same time only for one (or several) random loop iterations. When each process goes to another iteration (takes the next document) -- everything is ok again. I make 5 tries to write to MongoDB.
Each try is with delay +1 sec to the previous, so the first try makes immediately, if any exception is caught -- wait a second and try again, the next try will be in 2 seconds and so on. But this does not help, all these 5 tries are broken.
This is not mongoDB problem, it's log is empty and it even don't receive anything from PHP, when error happens.
Also I have admitted, the more simultaneous processes I run -- the more frequent errors occur.
Also it is not server resource problem, when error occurs, half of RAM (4 gig) is free and CPU is working for the half of it's power.
Maybe PHP has some configuration for this? Some memory limits or something...
I use PHP v 7.1.30
MongoDB v 3.2.16
PHP Package mongodb/mongodb v 1.1.2
<?php
$processesAmount = 5;
$documents = $this->mongoResource->getDocuments();
for ($processNumber = 0; $processNumber < $processesAmount; $processNumber++) {
// create child process
$pid = pcntl_fork();
// do not create new processes in child processes
if ($pid === 0) {
break;
}
if ($pid === -1) {
// some errors catching staff here...
}
else if ($pid === 0) {
// create new MongoDB connection
}
else {
// Protect against Zombie children
// main process waits before all child processes end
for ($i = 0; $i < $processesAmount; $i++) {
pcntl_wait($status);
}
return null;
}
// spread documents to each process without duplicates
for ($i = $processNumber; $i < count($documents); $i += $processesAmount) {
$newDocumentData = $this->doSomeStaffWithDocument($documents[$i]);
$this->mongoResource->updateDocument($documents[$i], $newDocumentData);
}
}
There could be many issues here, one being that all processes are sharing 1 DB connection and the first to connect is then disconnecting and killing the connection for them all. Check the second example in the docs here: https://www.php.net/manual/en/ref.pcntl.php
If that doesn't help, the way I read your code the "spreading" part is happening in every process, when it should be happening once. Shouldn't you be putting the "work" in the child section like below?
$processesAmount = 5;
$documents = $this->mongoResource->getDocuments();
$numDocs = count($documents);
$i = 0;
$children = [];
for ($processNumber = 0; $processNumber < $processesAmount; $processNumber++) {
// create child
$pid = pcntl_fork();
if ($pid === -1) {
// some errors catching staff here...
} else if ($pid) {
//parent
$children[] = $pid;
} else {
//child
while (!empty($documents) && $i <= $numDocs) {
$i += $processNumber;
$doc = $documents[$i] ?? null;
unset($documents[$i]);
$newDocumentData = $this->doSomeStaffWithDocument($doc);
$this->mongoResource->updateDocument($doc, $newDocumentData);
}
}
}
//protect against zombies and wait for parent
//children is always empty unless in parent
while (!empty($children)) {
foreach ($children as $key => $pid) {
$status = null;
$res = pcntl_waitpid($pid, $status, WNOHANG);
if ($res == -1 || $res > 0) { //if the process has already exited
unset($children[$key]);
}
}
}
}

PHP: Writing a lot of small files the fastest or/and most efficient way

Imagine that a campaign will have 10,000 to 30,000 files about 4kb each should be written to disk.
And, there will be a couple of campaigns running at the same time. 10 tops.
Currently, I'm going with the usual way: file_put_contents.
it gets the job done but in a slow way and its php process is taking 100% cpu usage all the way.
fopen, fwrite, fclose, well, the result is similar to file_put_contents.
I've tried some async io stuff such as php eio and swoole.
it's faster but it'll yield "too many open files" after some time.
php -r 'echo exec("ulimit -n");' the result is 800000.
Any help would be appreciated!
well, this is sort of embarrassing... you guys are correct, the bottleneck is how it generates the file content...
I am assuming that you cannot follow SomeDude's very good advice on using databases instead, and you already have performed what hardware tuning could be performed (e.g. increasing cache, increasing RAM to avoid swap thrashing, purchasing SSD drives).
I'd try and offload the file generation to a different process.
You could e.g. install Redis and store the file content into the keystore, which is very fast. Then, a different, parallel process could extract the data from the keystore, delete it, and write to a disk file.
This removes all disk I/O from the main PHP process, and lets you monitor the backlog (how many keypairs are still unflushed: ideally zero) and concentrate on the bottleneck in content generation. You'll possibly need some extra RAM.
On the other hand, this is not too different from writing to a RAM disk. You could also output data to a RAM disk, and it would be probably even faster:
# As root
mkdir /mnt/ramdisk
mount -t tmpfs -o size=512m tmpfs /mnt/ramdisk
mkdir /mnt/ramdisk/temp
mkdir /mnt/ramdisk/ready
# Change ownership and permissions as appropriate
and in PHP:
$fp = fopen("/mnt/ramdisk/temp/{$file}", "w");
fwrite($fp, $data);
fclose($fp);
rename("/mnt/ramdisk/temp/{$file}", "/mnt/ramdisk/ready/{$file}");
and then have a different process (crontab? Or continuously running daemon?) move files from the "ready" directory of the RAM disk to the disk, deleting then the RAM ready file.
File System
The time required to create a file depends on the number of files in the directory, with various dependency functions that themselves depend on the file system. ext4, ext3, zfs, btrfs etc. will exhibit different behaviour. Specifically, you might experience significant slowdowns if the number of files exceeds some quantity.
So you might want to try timing the creation of a large number of sample files in one directory, and see how this time grows with the growth of the number. Keep in mind that there will be a performance penalty for access to different directories, so using straight away a very large number of subdirectories is again not recommended.
<?php
$payload = str_repeat("Squeamish ossifrage. \n", 253);
$time = microtime(true);
for ($i = 0; $i < 10000; $i++) {
$fp = fopen("file-{$i}.txt", "w");
fwrite($fp, $payload);
fclose($fp);
}
$time = microtime(true) - $time;
for ($i = 0; $i < 10000; $i++) {
unlink("file-{$i}.txt");
}
print "Elapsed time: {$time} s\n";
Creation of 10000 files takes 0.42 seconds on my system, but creation of 100000 files (10x) takes 5.9 seconds, not 4.2. On the other hand, creating one eighth of those files in 8 separate directories (the best compromise I found) takes 6.1 seconds, so it's not worthwhile.
But suppose that creating 300000 files took 25 seconds instead of 17.7; dividing those files in ten directories might take 22 seconds, and make the directory split worthwhile.
Parallel processing: r strategy
TL;DR this doesn't work so well on my system, though your mileage may vary. If the operations to be done are lengthy (here they are not) and differently bound from the main process, then it can be advantageous to offload them each to a different thread, provided you don't spawn too many threads.
You will need pcntl functions installed.
$payload = str_repeat("Squeamish ossifrage. \n", 253);
$time = microtime(true);
for ($i = 0; $i < 100000; $i++) {
$pid = pcntl_fork();
switch ($pid) {
case 0:
// Parallel execution.
$fp = fopen("file-{$i}.txt", "w");
fwrite($fp, $payload);
fclose($fp);
exit();
case -1:
echo 'Could not fork Process.';
exit();
default:
break;
}
}
$time = microtime(true) - $time;
print "Elapsed time: {$time} s\n";
(The fancy name r strategy is taken from biology).
In this example, spawning times are catastrophic if compared to what each child needs to do. Therefore, overall processing time skyrockets. With more complex children things would go better, but you must be careful not to turn the script into a fork bomb.
One possibility, if possible, could be to divide the files to be created into, say, chunks of 10% each. Each child would then change its working directory with chdir(), and create its files in a different directory. This would negate the penalty for writing files in different subdirectories (each child writes in its current directory), while benefiting from writing less files. In this case, with very lightweight and I/O bound operations in the child, again the strategy isn't worthwhile (I get doubled execution time).
Parallel processing: K strategy
TL;DR this is more complex but works well... on my system. Your mileage may vary.
While r strategy involves lots of fire-and-forget threads, K strategy calls for a limited (possibly one) child which is nurtured carefully. Here we offload the creation of all the files to one parallel thread, and communicate with it via sockets.
$payload = str_repeat("Squeamish ossifrage. \n", 253);
$sockets = array();
$domain = (strtoupper(substr(PHP_OS, 0, 3)) == 'WIN' ? AF_INET : AF_UNIX);
if (socket_create_pair($domain, SOCK_STREAM, 0, $sockets) === false) {
echo "socket_create_pair failed. Reason: ".socket_strerror(socket_last_error());
}
$pid = pcntl_fork();
if ($pid == -1) {
echo 'Could not fork Process.';
} elseif ($pid) {
/*parent*/
socket_close($sockets[0]);
} else {
/*child*/
socket_close($sockets[1]);
for (;;) {
$cmd = trim(socket_read($sockets[0], 5, PHP_BINARY_READ));
if (false === $cmd) {
die("ERROR\n");
}
if ('QUIT' === $cmd) {
socket_write($sockets[0], "OK", 2);
socket_close($sockets[0]);
exit(0);
}
if ('FILE' === $cmd) {
$file = trim(socket_read($sockets[0], 20, PHP_BINARY_READ));
$len = trim(socket_read($sockets[0], 8, PHP_BINARY_READ));
$data = socket_read($sockets[0], $len, PHP_BINARY_READ);
$fp = fopen($file, "w");
fwrite($fp, $data);
fclose($fp);
continue;
}
die("UNKNOWN COMMAND: {$cmd}");
}
}
$time = microtime(true);
for ($i = 0; $i < 100000; $i++) {
socket_write($sockets[1], sprintf("FILE %20.20s%08.08s", "file-{$i}.txt", strlen($payload)));
socket_write($sockets[1], $payload, strlen($payload));
//$fp = fopen("file-{$i}.txt", "w");
//fwrite($fp, $payload);
//fclose($fp);
}
$time = microtime(true) - $time;
print "Elapsed time: {$time} s\n";
socket_write($sockets[1], "QUIT\n", 5);
$ok = socket_read($sockets[1], 2, PHP_BINARY_READ);
socket_close($sockets[1]);
THIS IS HUGELY DEPENDENT ON THE SYSTEM CONFIGURATION. For example on a mono-processor, mono-core, non-threading CPU, this is madness - you'll at least double the total runtime, but more likely it will go from three to ten times as slow.
So this is definitely not the way to pimp up something running on an old system.
On a modern multithreading CPU and supposing the main content creation loop is CPU bound, you may experience the reverse - the script might go ten times faster.
On my system, the "forking" solution above runs a bit less than three times faster. I expected more, but there you are.
Of course, whether the performance is worth the added complexity and maintenance, remains to be evaluated.
The bad news
While experimenting above, I came to the conclusion that file creation on a reasonably configured and performant machine in Linux is fast as hell, so not only it's difficult to squeeze more performances, but if you're experiencing slowness, it's very likely that it is not file related. Try detailing some more about how you create that content.
Having read your description, I understand you're writing many files that are each rather small. The way PHP usually works (at least in the Apache server), there is overhead for each filesystem access: a file pointer and buffer is opened and maintained for each file. Since there's no code samples to review here, it's hard to see where inefficiencies are.
However, using file_put_contents() for 300,000+ files appears to be slightly less efficient than using fopen() and fwrite() or fflush() directly, then fclose() when you're done. I'm saying that based on a benchmark done by a fellow in the comments of the PHP documentation for file_put_contents() at http://php.net/manual/en/function.file-put-contents.php#105421
Next, when dealing with such small file sizes, it sounds like there's a great opportunity to use a database instead of flat files (I'm sure you've got that before). A database, whether mySQL or PostgreSQL, is highly optimized for simultaneous access to many records, and can internally balance CPU workload in ways that filesystem access never can (and binary data in records is possible too). Unless you need access to real files directly from your server hard drives, a database can simulate many files by allowing PHP to return individual records as file data over the web (i.e., by using the header() function). Again, I'm assuming this PHP is running as a web interface on a server.Overall, what I am reading suggests that there may be an inefficiency somewhere else besides filesystem access. How is the file content generated? How does the operating system handle file access? Is there compression or encryption involved? Are these images or text data? Is the OS writing to one hard drive, a software RAID array, or some other layout? Those are some of the questions I can think of just glancing over your problem. Hopefully my answer helped. Cheers.
The main idea is to have less files.
Ex: 1,000 files can be added in 100 files, each containing 10 files - and parsed with explode and you will get 5x faster on write and 14x faster on read+parse
with file_put_contents and fwrite optimized, you will not get more than 1.x speed. This solution can be useful for read/write. Other solution may be mysql or other db.
On my computer to create 30k files with a small string it takes 96.38 seconds and to append 30k times same string in one file it takes 0.075 sec
I can offer you an unusual solution, when you can use it fewer times file_put_contents function. bellow this i show you a simple code to understand how it works.
$start = microtime(true);
$str = "Aaaaaaaaaaaaaaaaaaaaaaaaa";
if( !file_exists("test/") ) mkdir("test/");
foreach( range(1,1000) as $i ) {
file_put_contents("test/".$i.".txt",$str);
}
$end = microtime(true);
echo "elapsed_file_put_contents_1: ".substr(($end - $start),0,5)." sec\n";
$start = microtime(true);
$out = '';
foreach( range(1,1000) as $i ) {
$out .= $str;
}
file_put_contents("out.txt",$out);
$end = microtime(true);
echo "elapsed_file_put_contents_2: ".substr(($end - $start),0,5)." sec\n";
this is a full example with 1000 files and elapsed time
with 1000 files
writing file_put_contens: elapsed: 194.4 sec
writing file_put_contens APPNED :elapsed: 37.83 sec ( 5x faster )
............
reading file_put_contens elapsed: 2.401 sec
reading append elapsed: 0.170 sec ( 14x faster )
$start = microtime(true);
$allow_argvs = array("gen_all","gen_few","read_all","read_few");
$arg = isset($argv[1]) ? $argv[1] : die("php ".$argv[0]." gen_all ( ".implode(", ",$allow_argvs).")");
if( !in_array($arg,$allow_argvs) ) {
die("php ".$argv[0]." gen_all ( ".implode(", ",$allow_argvs).")");
}
if( $arg=='gen_all' ) {
$dir_campain_all_files = "campain_all_files/";
if( !file_exists($dir_campain_all_files) ) die("\nFolder ".$dir_campain_all_files." not exist!\n");
$exists_campaings = false;
foreach( range(1,10) as $i ) { if( file_exists($dir_campain_all_files.$i) ) { $exists_campaings = true; } }
if( $exists_campaings ) {
die("\nDelete manualy all subfolders from ".$dir_campain_all_files." !\n");
}
build_campain_dirs($dir_campain_all_files);
// foreach in campaigns
foreach( range(1,10) as $i ) {
$campain_dir = $dir_campain_all_files.$i."/";
$nr_of_files = 1000;
foreach( range(1,$nr_of_files) as $f ) {
$file_name = $f.".txt";
$data_file = generateRandomString(4*1024);
$dir_file_name = $campain_dir.$file_name;
file_put_contents($dir_file_name,$data_file);
}
echo "campaing #".$i." done! ( ".$nr_of_files." files writen ).\n";
}
}
if( $arg=='gen_few' ) {
$delim_file = "###FILE###";
$delim_contents = "###FILE###";
$dir_campain = "campain_few_files/";
if( !file_exists($dir_campain) ) die("\nFolder ".$dir_campain_all_files." not exist!\n");
$exists_campaings = false;
foreach( range(1,10) as $i ) { if( file_exists($dir_campain.$i) ) { $exists_campaings = true; } }
if( $exists_campaings ) {
die("\nDelete manualy all files from ".$dir_campain." !\n");
}
$amount = 100; // nr_of_files_to_append
$out = ''; // here will be appended
build_campain_dirs($dir_campain);
// foreach in campaigns
foreach( range(1,10) as $i ) {
$campain_dir = $dir_campain.$i."/";
$nr_of_files = 1000;
$cnt_few=1;
foreach( range(1,$nr_of_files) as $f ) {
$file_name = $f.".txt";
$data_file = generateRandomString(4*1024);
$my_file_and_data = $file_name.$delim_file.$data_file;
$out .= $my_file_and_data.$delim_contents;
// append in a new file
if( $f%$amount==0 ) {
$dir_file_name = $campain_dir.$cnt_few.".txt";
file_put_contents($dir_file_name,$out,FILE_APPEND);
$out = '';
$cnt_few++;
}
}
// append remaning files
if( !empty($out) ) {
$dir_file_name = $campain_dir.$cnt_few.".txt";
file_put_contents($dir_file_name,$out,FILE_APPEND);
$out = '';
}
echo "campaing #".$i." done! ( ".$nr_of_files." files writen ).\n";
}
}
if( $arg=='read_all' ) {
$dir_campain = "campain_all_files/";
$exists_campaings = false;
foreach( range(1,10) as $i ) {
if( file_exists($dir_campain.$i) ) {
$exists_campaings = true;
}
}
foreach( range(1,10) as $i ) {
$campain_dir = $dir_campain.$i."/";
$files = getFiles($campain_dir);
foreach( $files as $file ) {
$data = file_get_contents($file);
$substr = substr($data, 100, 5); // read 5 chars after char100
}
echo "campaing #".$i." done! ( ".count($files)." files readed ).\n";
}
}
if( $arg=='read_few' ) {
$dir_campain = "campain_few_files/";
$exists_campaings = false;
foreach( range(1,10) as $i ) {
if( file_exists($dir_campain.$i) ) {
$exists_campaings = true;
}
}
foreach( range(1,10) as $i ) {
$campain_dir = $dir_campain.$i."/";
$files = getFiles($campain_dir);
foreach( $files as $file ) {
$data_temp = file_get_contents($file);
$explode = explode("###FILE###",$data_temp);
//#mkdir("test/".$i);
foreach( $explode as $exp ) {
$temp_exp = explode("###FILE###",$exp);
if( count($temp_exp)==2 ) {
$file_name = $temp_exp[0];
$file_data = $temp_exp[1];
$substr = substr($file_data, 100, 5); // read 5 chars after char100
//file_put_contents("test/".$i."/".$file_name,$file_data); // test if files are recreated correctly
}
}
//echo $file." has ".strlen($data_temp)." chars!\n";
}
echo "campaing #".$i." done! ( ".count($files)." files readed ).\n";
}
}
$end = microtime(true);
echo "elapsed: ".substr(($end - $start),0,5)." sec\n";
echo "\n\nALL DONE!\n\n";
/*************** FUNCTIONS ******************/
function generateRandomString($length = 10) {
$characters = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
$charactersLength = strlen($characters);
$randomString = '';
for ($i = 0; $i < $length; $i++) {
$randomString .= $characters[rand(0, $charactersLength - 1)];
}
return $randomString;
}
function build_campain_dirs($dir_campain) {
foreach( range(1,10) as $i ) {
$dir = $dir_campain.$i;
if( !file_exists($dir) ) {
mkdir($dir);
}
}
}
function getFiles($dir) {
$arr = array();
if ($handle = opendir($dir)) {
while (false !== ($file = readdir($handle))) {
if ($file != "." && $file != "..") {
$arr[] = $dir.$file;
}
}
closedir($handle);
}
return $arr;
}

Symfony 1.4 functional test - reduce memory usage

I have a csv file that defines routes to test, and the expected status code each route should return.
I am working on a functional test that iterates over the csv file and makes a request to each route, then checks to see if the proper status code is returned.
$browser = new sfTestFunctional(new sfBrowser());
foreach ($routes as $route)
{
$browser->
get($route['path'])->
with('response')->begin()->
isStatusCode($route['code'])->
end()
;
print(memory_get_usage());
}
/*************** OUTPUT: *************************
ok 1 - status code is 200
97953280# get /first_path
ok 2 - status code is 200
109607536# get /second_path
ok 3 - status code is 403
119152936# get /third_path
ok 4 - status code is 200
130283760# get /fourth_path
ok 5 - status code is 200
140082888# get /fifth_path
...
/***************************************************/
This continues until I get an allowed memory exhausted error.
I have increased the amount of allowed memory, which temporarily solved the problem. That is not a permanent solution since more routes will be added to the csv file over time.
Is there a way to reduce the amount of memory this test is using?
I faced the same out of memory problem. I needed to crawl a very long list of URI (around 30K) to generate the HTML cache. Thanks to Marek, I tried to fork processes. There is still a little leak, but this is insignificant.
As an input, I had a text file with one line per URI. Of course you can easily adapt the following script with a CSV.
const NUMBER_OF_PROCESS = 4;
const SIZE_OF_GROUPS = 5;
require_once(dirname(__FILE__).'/../../config/ProjectConfiguration.class.php');
$configuration = ProjectConfiguration::getApplicationConfiguration('frontend', 'prod', false);
sfContext::createInstance($configuration);
$file = new SplFileObject(dirname(__FILE__).'/list-of-uri.txt');
while($file->valid())
{
$count = 0;
$uris = array();
while($file->valid() && $count < NUMBER_OF_PROCESS * SIZE_OF_GROUPS) {
$uris[] = trim($file->current());
$file->next();
$count++;
}
$urisGroups = array_chunk($uris, SIZE_OF_GROUPS);
$childs = array();
echo "---\t\t\t Forking ".sizeof($urisGroups)." process \t\t\t ---\n";
foreach($urisGroups as $uriGroup) {
$pid = pcntl_fork();
if($pid == -1)
die('Could not fork');
if(!$pid) {
$b = new sfBrowser();
foreach($uriGroup as $key => $uri) {
$starttime = microtime(true);
$b->get($uri);
$time = microtime(true) - $starttime;
echo 'Mem: '.memory_get_peak_usage().' - '.$time.'s - URI N°'.($key + 1).' PID '.getmypid().' - Status: '.$b->getResponse()->getStatusCode().' - URI: '.$uri."\n";
}
exit();
}
if($pid) {
$childs[] = $pid;
}
}
while(count($childs) > 0) {
foreach($childs as $key => $pid) {
$res = pcntl_waitpid($pid, $status, WNOHANG);
// If the process has already exited
if($res == -1 || $res > 0)
unset($childs[$key]);
}
sleep(1);
}
}
const NUMBER_OF_PROCESS is defining the number of parallel processes working (thus, you save time if you have a multi-core processor)
const NUMBER_OF_PROCESS is defining the number of URI that will be crawled by sfBrowser in each process. You can decrease it if you still have out of memory problems

Help to test if a proxy works using php

I have a list of proxies, need to use them in a php script I am writing.
How can I test that the proxy will work before I use it? is that possible? at the moment if the proxy doesn't work the script just dies after 30 seconds. Is there a quicker way to identify whether it will work or not?
perhaps create a socket connection? send something to it that will reveal whether the proxy is open?
thanks
you can use fsockopen for this, where you can specify a timeout.
A simple proxy list checker. You can check a list ip:port if that port is opened on that IP.
<?php
$fisier = file_get_contents('proxy_list.txt'); // Read the file with the proxy list
$linii = explode("\n", $fisier); // Get each proxy
$fisier = fopen("bune.txt", "a"); // Here we will write the good ones
for($i = 0; $i < count($linii) - 1; $i++) test($linii[$i]); // Test each proxy
function test($proxy)
{
global $fisier;
$splited = explode(':',$proxy); // Separate IP and port
if($con = #fsockopen($splited[0], $splited[1], $eroare, $eroare_str, 3))
{
fwrite($fisier, $proxy . "\n"); // Check if we can connect to that IP and port
print $proxy . '<br>'; // Show the proxy
fclose($con); // Close the socket handle
}
}
fclose($fisier); // Close the file
?>
you may also want to use set_time_limit so that you can run your script for longer.
Code taken from: http://www.php.net/manual/en/function.fsockopen.php#95605
You can use this function
function check($proxy=null)
{
$proxy= explode(':', $proxy);
$host = $proxy[0];
$port = $proxy[1];
$waitTimeoutInSeconds = 10;
$bool=false;
if($fp = #fsockopen($host,$port,$errCode,$errStr,$waitTimeoutInSeconds)){
$bool=true;
}
fclose($fp);
return $bool;
}

Explain basic PHP Socket server code

I'm trying to learn how to make a chat with a socket server.
I noticed everybody uses the same code (ripoff from the zend developer zone).
The problem is no one really explains how it works. Especially the cryptic code after while(true) { .
This would benefit many so i hope someone could take the time and explain the code in detail (DETAIL!).
You can find the code here
I'll answer it myselfe. I went over it line by line.. this is how it works (I'm only explaining the part in the while(true) loops.
1.
// Setup clients listen socket for reading
$read[0] = $sock;
for ($i = 0; $i < $max_clients; $i++) {
if (isset($client[$i]['sock']))
$read[$i + 1] = $client[$i]['sock'];
}
This asings freshly created connections to the $read array to be watched for incoming data.
// Set up a blocking call to socket_select()
if (socket_select($read, $write = NULL, $except = NULL, $tv_sec = 5) < 1)
continue;
Watches the $read array for new data (I'm still a bit unclear how this works)
/* if a new connection is being made add it to the client array */
if (in_array($sock, $read)) {
for ($i = 0; $i < $max_clients; $i++) {
if (empty($client[$i]['sock'])) {
$client[$i]['sock'] = socket_accept($sock);
echo "New client connected $i\r\n";
break;
}
elseif ($i == $max_clients - 1)
echo "Too many clients...\r\n";
}
}
Determines when a new connection is being made, than finds an empty spot in the $client array and add the socket.
This next part I'll split up for easier explanation.
for ($i = 0; $i < $max_clients; $i++) { // for each client
if (isset($client[$i]['sock'])) {
Loops through all the $client array but only works on the ones that actually have a connection.
if (in_array($client[$i]['sock'], $read)) {
$input = socket_read($client[$i]['sock'], 1024);
if ($input == null) {
echo "Client disconnecting $i\r\n";
// Zero length string meaning disconnected
unset($client[$i]);
} else {
echo "New input received $i\r\n";
// send it to the other clients
for ($j = 0; $j < $max_clients; $j++) {
if (isset($client[$j]['sock']) && $j != $i) {
echo "Writing '$input' to client $j\r\n";
socket_write($client[$j]['sock'], $input, strlen($input));
}
}
if ($input == 'exit') {
// requested disconnect
socket_close($client[$i]['sock']);
}
}
} else {
echo "Client disconnected $i\r\n";
// Close the socket
socket_close($client[$i]['sock']);
unset($client[$i]);
}
First it sees if there is still an active connection, if not it closes it.
If there is a connection it read the data, if there is noone this is code for a disconnect.
If there is data it passes it along to other clients (but itselfe).
That's ti. Hope I got it right.
PHP is not multithreaded, because of that you cant build good socket server.
Use Python instead.
http://docs.python.org/library/socketserver.html
http://www.rohitab.com/discuss/index.php?showtopic=24808

Categories