I am getting errors (too many connections) when I cURL from a parent to multiple children and execute mysqli queries
I have tried using persistent connections in just the parent and in the parent and children. When I cURL from the parent to 25 children as a test, roughly 20 will process and then I get a "too many connections" error. Logs confirm that not all the child processes are executing.
Here's the code for the parent:
<!-- parent start -->
$link = mysqli_connect('p:host', 'user', 'password', 'database');
$sql = mysqli_query($link, "SELECT * FROM t1");
$rowcount = mysqli_num_rows($sql);
if ($rowcount > '0') {
$mh = curl_multi_init();
$active = null;
}
while ($row = mysqli_fetch_assoc($sql)) {
$sql2 = mysqli_query($link, "SELECT id FROM t2 WHERE field1 = 'Waiting'");
$row2 = mysqli_fetch_assoc($sql2);
extract($row2);
// spawn child process
$ch = '$ch' . $id;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'loadtest.child.php');
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_USERAGENT, 'loadtest.parent.php');
curl_setopt($ch, CURLOPT_TIMEOUT, 1);
$curl_data = 'jid=' . $id;
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $curl_data);
curl_multi_add_handle($mh,$ch);
}
do {
$mrc = curl_multi_exec($mh, $active);
} while ($active > 0);
curl_multi_remove_handle($mh, $ch);
curl_multi_close($mh);
?>
<!-- parent end -->
And the child:
<!-- child start -->
$link = mysqli_connect('p:host', 'user', 'password', 'database');
$sql = mysqli_query($link, "SELECT * FROM t3 WHERE field1 ='" . $jid . "' LIMIT 1");
$row = mysqli_fetch_assoc($sql);
extract($row);
$sql2 = mysqli_query($link, "SELECT * FROM t4 WHERE field1='" . $jid . "' LIMIT 1");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'loadtest.script.php);
$result = curl_exec($ch);
curl_close($ch);
$sql_update = mysqli_query($link, "UPDATE t1 SET status = 'Processed' WHERE field1 = '$jid'");
<!-- child end -->
I have removed some code from both parent and child that writes to logs, and the reference in child to loadtest.script.php is to a placeholder that does nothing more than write one line to a text file.
Right now I am on a shared server and have a maximum of 240 connections that I cannot change. I will be going to a dedicated server in the near future.
Any suggestions on how to get the children to run on the same persistent connection and avoid the "too many connections" mysql error?
Also any other suggestions would be appreciated.
Thanks.
Related
I have a table where I want to run a PHP script during the maintenance.
I have the following code.
$sql = "SELECT `id`, `url` FROM `log_requests` WHERE `is_sent` = '0'";
$e = $conn->execute($sql);
My url array after fetching the table from DB.
Array
(
[0] => https://www.smsgatewaycenter.com/library/send_sms_2.php?UserName=username&Password=password&Type=Individual&To=9999999999&Mask=Senderid&Message=Hello+World+1
[1] => https://www.smsgatewaycenter.com/library/send_sms_2.php?UserName=username&Password=password&Type=Individual&To=9999999999&Mask=Senderid&Message=Hello+World+2
[2] => https://www.smsgatewaycenter.com/library/send_sms_2.php?UserName=username&Password=password&Type=Individual&To=9999999999&Mask=Senderid&Message=Hello+World+3
[3] => https://www.smsgatewaycenter.com/library/send_sms_2.php?UserName=username&Password=password&Type=Individual&To=9999999999&Mask=Senderid&Message=Hello+World+4
)
Then I run while loop
$ch = curl_init();
while($row = $e->FetchRow() ){
curl_setopt($ch, CURLOPT_URL, $row['url']);
curl_setopt($ch, CURLOPT_HEADER, 0);
$q = "UPDATE `log_requests` SET `is_sent` = '1' WHERE `id` = '".$row['id']."'";
$conn->execute($q);
}
curl_close($ch);
Issue is only one URL is executed using CURL and in my table only one row gets updated as 1 in is_sent column.
Why is it running single url in my while loop. What mistakes am I making here?
************************* EDIT ********************
Okay, now I came out with following and running successfully now. I just want to know whether am doing right or not.
$set = array();
while ($row = $executequery2->FetchRow()) {
$set[$row['url']][] = $row;
}
$curl_handles = array();
foreach($set as $url => $rows) {
$curl_handles[$url] = curl_init();
curl_setopt($curl_handles[$url], CURLOPT_URL, $url);
foreach($rows as $row) {
$query3 = "UPDATE `log_requests` SET `is_sent` = '1' WHERE `id` = '" . $row['id'] . "'";
$conn->Execute($query3);
}
}
$curl_multi_handle = curl_multi_init();
$i = 0;
$block = array();
foreach($curl_handles as $a_curl_handle) {
$i++;
curl_multi_add_handle($curl_multi_handle, $a_curl_handle);
$block[] = $a_curl_handle;
if (($i % 5 == 0) or ($i == count($curl_handles))) {
$running = NULL;
do {
$running_before = $running;
curl_multi_exec($curl_multi_handle, $running);
if ($running != $running_before) {
echo ("Waiting for $running sites to finish...<br />");
}
}
while ($running > 0);
foreach($block as $handle) {
$code = curl_getinfo($handle, CURLINFO_HTTP_CODE);
$curl_errno = curl_errno($handle);
$curl_error = curl_error($handle);
if ($curl_error) {
echo (" *** cURL error: ($curl_errno) $curl_error\n");
}
curl_multi_remove_handle($curl_multi_handle, $handle);
}
$block = array();
}
}
curl_multi_close($curl_multi_handle);
Updating the table column is the right way?
I had faced same issue and got this solution online. Just separate curl code from the loop and then try.
Something like this:
while($row = $e->FetchRow()) {
$result = call_curl($row['url']);
if($result === FALSE) {
die("Error");
}
else
{
$q = "UPDATE `log_requests` SET `is_sent` = '1' WHERE `id` = '".$row['id']."'";
}
}
function call_curl($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
$result = curl_exec ($ch);
curl_close($ch);
return $result;
}
Hope this helps you as well.
Hello there I've written some PHP that gets XML from a website, and will store it on a table on my database. It echo's Success, yet the table is blank? Heres my code:
<?php
$url ="http://rates.fxcm.com/RatesXML3";
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
$data = curl_exec ($ch);
curl_close($ch);
$xml = simplexml_load_string($data);
$con = mysql_connect("localhost","username","password"); //this does contain proper information, just ive hidden it with the following
mysql_select_db("database", $con) or die(mysql_error()); //same for database too
foreach ($xml -> Rate as $row){
$Symbol = $row -> Symbol;
$Bid = $row -> Bid;
$Ask = $row -> Ask;
//performing sql query
$sql = "INSERT INTO 'FXCM_Rates' ('Symbol', 'Bid', 'Ask')"
. "VALUES ('$Symbol', '$Bid', '$Ask')";
$result = mysql_query($sql);
if (!result) {
echo 'MySQL ERROR';
} else{
echo 'SUCCESS';
}
}
?>
Now when I check the PHP, it has echoed success for all 63 Rates, yet when I go to check the table, it is blank? I just find it odd that it has worked for everyone else, but not me :(
Thanks to anyone that can help fix my code / point out my error.
You have '$' missing in the end.
if (!$result) {
Also now that I've done some tests, your variables are objects which I'm not sure that you can INSERT in database like that.
foreach ($xml -> Rate as $row){
$Symbol = (string)$row -> Symbol;
$Bid = (string)$row -> Bid;
$Ask = (string)$row -> Ask;
}
This will return strings that might be easier to use for you query
mysql is deprecated in newer version of php, you should be using mysqli(improved)..
<?php
$url ="http://rates.fxcm.com/RatesXML3";
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
$data = curl_exec ($ch);
curl_close($ch);
$xml = simplexml_load_string($data);
$con = mysqli_connect("localhost","username","password","database") or die(mysqli_error($con));
foreach ($xml -> Rate as $row){
$Symbol = $row -> Symbol;
$Bid = $row -> Bid;
$Ask = $row -> Ask;
//performing sql query
$sql = "INSERT INTO 'FXCM_Rates' ('Symbol', 'Bid', 'Ask')"
. "VALUES ('$Symbol', '$Bid', '$Ask')";
$result = mysqli_query($con,$sql);
if (!$result) {
echo 'MySQL ERROR';
} else{
echo 'SUCCESS';
}
}
i have php script it keep downloading feeds from the internet.
but unfortunately it keep duplicate the image feed into the database.
part from the image, the script working fine on other feed like description,link, title.
php script
<?php
require 'database.php';
$url = "http://www.albaldnews.com/rss.php?cat=24";
$rss = simplexml_load_file($url);
if($rss)
{
echo '<h1>'.$rss->channel->title.'</h1>';
echo '<li>'.$rss->channel->pubDate.'</li>';
$items = $rss->channel->item;
foreach($items as $item)
{
$enclosure = $item->enclosure[0]['url'];
$ch = curl_init ("$enclosure");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER,1);
$rawdata=curl_exec ($ch);
curl_close ($ch);
$img=mysqli_real_escape_string($conn,$rawdata);
$query = "SELECT image from feedtable where link = '$img'";
$result= mysqli_query($conn, $query);
$num_rows = mysqli_num_rows($result);
if ($num_rows == 0)
{
$query1 = "INSERT INTO feedtable (image)VALUES ('$img')";
$result1= mysqli_query($conn, $query1);
echo "image added\n";
}
else
{
echo "duplicate entry\n";
}
}
}
?>
I am trying to see if keyWord already exists in table searchedWords. If it does, then the countr increases by one. If it does not exists in the table, then I used INSERT. The problem is, the keyWord being passed to the site is not stored in the DB. The other BIG problem is that the countr does not add at. Is it because of the if statement? Or is it the while loop?
<?php
date_default_timezone_set('Asia/Manila');
$today = date('m-d-Y');
echo $today;
$urltopost = "http://opac.usls.edu.ph/TLCScripts/interpac.dll?Search";
$datatopost = "FormId=0&Config=pac&LimitsId=0&StartIndex=0&SearchField=7&SearchType=1&ItemsPerPage=20&SearchData=$_POST[keyWord]";
$ch = curl_init ($urltopost);
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $datatopost);
curl_setopt ($ch, CURLOPT_HEADER, 0);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
$returndata = curl_exec ($ch);
echo $returndata;
$con=mysqli_connect("...","...","...","...")or die ('Error: ' . mysql_error());
$sql= "SELECT * FROM searchedWords";
$result= mysqli_query($con,$sql);
while($row=mysqli_fetch_array($result, MYSQLI_ASSOC))
{
if($row['keyWord']==$_POST[keyWord])
{
$upD="UPDATE searchedWords SET countr = countr + 1 WHERE keyWord = '".$row['keyWord']."'";
while (!mysqli_query($con,$upD))
{
die('Error: ' . mysqli_error($con));
}
}
else
{
$insertIn="INSERT INTO `searchedWords`( `keyWord`, `countr`) values ('$_POST[keyWord]',1)";
while (!mysqli_query($con,$insertIn))
{
die('Error: ' . mysqli_error($con));
}
}
}
?>
Thank you to those who can help me out.
Why are you making things complex... Try the simple way
$KeyWord = $_POST['keyWord']; //Do not forget to sanitize this for security
$sql= "SELECT * FROM searchedWords WHERE keyWord='$KeyWord'";
$result= mysqli_query($con,$sql);
$count= mysqli_num_rows($result);
if($count) {
$upD="UPDATE searchedWords SET countr = countr + 1 WHERE keyWord = '$keyWord'";
mysqli_query($con,$upD);
} else {
$insertIn="INSERT INTO `searchedWords`( `keyWord`, `countr`) values ('$keyWord',1)";
mysqli_query($con,$insertIn); }
It is really bad practice to run sql queries in the loop. Try to use such query to update all existing keywords:
$keyword = htmlspecialchars($_POST['keyWord']);
UPDATE searchedWords SET countr = countr + 1 WHERE keyWord = '".$keyword."'";
No more loops!
And to work more safely and efficient with database take a look on http://php.net/manual/en/pdo.prepared-statements.php
I have three filed in my mysql table they are: id, url, status
How to check all urls from the column url and write either 1 (available) or 0 (unavailable) to the status column?
To just check urls manualy in php w/o mysql I could use this:
<?php
function Visit($url){
$agent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL,$url );
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch,CURLOPT_VERBOSE,false);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
$page=curl_exec($ch);
//echo curl_error($ch);
$httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
if($httpcode >= 200 && $httpcode < 300){
return true;
}
else {
return false;
}
}
if(Visit("http://www.google.com")){//maybe make it a variable from a result of a mysql select, but how to process it one by one?
echo "Website OK"; //maybe somesql here to wtite '1'
}
else{
echo "Website DOWN";//maybe somesql here to wtite '0'
}
?>
This is bad in terms of performance as doing queries in a loop is bad design, you should instead build a batch update and run the query in the end, but I've not enough time now to elaborate that. This is just to get you started:
$sql = "SELECT id,url FROM mytable";
$res = mysql_query($sql) or die(mysql_error());
if($res)
{
while($row = mysql_fetch_assoc($res))
{
$status = visit($row['url']) ? '1' : '0';
$id = $row['id'];
$update = "UPDATE mytable SET status = $status WHERE id = $id";
$res = mysql_query($update) or die(mysql_error());
}
}
echo mysql_num_rows($result) ? '1' : '0';
You can use sql queries for example:
select id, url from urltable;
update urltable set status=1 where id=99;
So, how about use this code with your Visit function:
$link = mysql_connect('localhost', 'test', 'pppppp');
$db_selected = mysql_select_db('test', $link);
$query = "select id, url from urltable";
$result = mysql_query($query);
while ($row = mysql_fetch_assoc($result)) {
$visitid = $row['id'];
$visiturl = $row['url'];
$visitstatus = Visit($visiturl)? 1: 0;
$upquery = sprintf("update urltable set status=%d where id=%d", $visitstatus, $visitid);
$upresult = mysql_query($upquery);
}