Random 500 errors in a multi CURL request - php

I am trying to get some hal+json data from a web service through a curl_multi to fill in a Bootstrap Typeahead. Every time I run this code, some of my requests in the curl_multi will be 500, and some will return the data I need. The ones that are 500 are completely random; the next time I load the page different queries will be 500 instead (And I get no errors). Why does that keep happening?
<?php
$curlurl = 'https://service.domain.com/'.$_POST['customer-name'].'/contact?callback=?';
$cuurl = singleRequest($curlurl);
global $contacts_typeahead_data;
$contacts_typeahead_data = array();
$clinks = $cuurl['_links']['https://service.domain.com/rel/contacts'];
for ($i=0; $i < count($clinks); $i++) {
$data[] = 'https://service.domain.com'.$clinks[$i]['href'].'?callback=?';
};
print_r($data);
$datar = multiRequest($data);
print_r($datar);
foreach($datar as $c){
$cs = json_decode($c, true);?>
<option value="<?php echo $cs['id']; ?>"><?php echo $cs['id'].' ('.$cs['info'][0]['name'].')'; ?></option>
<?php $contacts_typeahead_data[] = $cs['id'].' ('.$cs['info'][0]['name'].')';
}?>
And here is the code to the singleRequest and multiRequest, which is based on http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/:
function multiRequest($data, $options = array()) {
// array of curl handles
$curly = array();
// data to be returned
$result = array();
// multi handle
$mh = curl_multi_init();
// loop through $data and create curl handles
// then add them to the multi-handle
foreach ($data as $id => $d) {
$curly[$id] = curl_init();
$url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d;
$username = base64_encode('username');
$password2 = 'password';
$auth_token = $username . $password2 . 'QQ==';
curl_setopt($curly[$id], CURLOPT_URL, $url);
curl_setopt($curly[$id], CURLOPT_HEADER, 0);
curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curly[$id], CURLOPT_USERPWD, "$username:$password2");
curl_setopt($curly[$id], CURLOPT_SSLVERSION,3);
curl_setopt($curly[$id], CURLOPT_HTTPHEADER, array(
'Accept: application/hal+json',
'Access-Control-Allow-Origin: *',
'Content-Type: application/hal+json',
'Authorization: Basic ' . $auth_token
));
curl_setopt($curly[$id], CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($curly[$id], CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($curly[$id], CURLOPT_TIMEOUT, 30);
curl_setopt($curly[$id], CURLOPT_PORT, 443);
curl_setopt($curly[$id], CURLOPT_REFERER, $_SERVER['HTTP_REFERER']);
curl_setopt($curly[$id], CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)');
curl_setopt($curly[$id], CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curly[$id], CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curly[$id], CURLOPT_COOKIE, 'cert_url=https://service.domain.com/cert/yagowyefayygwflagliwygelifyaigwepifgpai');
// post?
if (is_array($d)) {
if (!empty($d['post'])) {
curl_setopt($curly[$id], CURLOPT_POST, 1);
curl_setopt($curly[$id], CURLOPT_POSTFIELDS, $d['post']);
}
}
// extra options?
if (!empty($options)) {
curl_setopt_array($curly[$id], $options);
}
curl_multi_add_handle($mh, $curly[$id]);
}
// execute the handles
$running = null;
do {
curl_multi_exec($mh, $running);
} while($running > 0);
// get content and remove handles
foreach($curly as $id => $c) {
// cURL error number
$curl_errno = curl_errno($c);
// cURL error message
$curl_error = curl_error($c);
// output if there was an error
if ($curl_error) {
echo " *** cURL error: (".$curl_errno.") ".$curl_error;
} else {
$result[$id] = curl_multi_getcontent($c);
}
curl_multi_remove_handle($mh, $c);
}
// all done
curl_multi_close($mh);
return $result;
}
function singleRequest($curlurl){
$username = base64_encode('username');
$password2 = 'password';
$auth_token = $username . $password2 . 'QQ==';
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERPWD, "$username:$password2");
curl_setopt($ch, CURLOPT_SSLVERSION,3);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Accept: application/hal+json',
'Access-Control-Allow-Origin: *',
'Content-Type: application/hal+json',
'Authorization: Basic ' . $auth_token
));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_COOKIE, 'cert_url=https://service.domain.com/cert/yagowyefayygwflagliwygelifyaigwepifgpai');
$result = curl_exec($ch);
if($result === false){
echo 'Curl error: ' . curl_error($ch). '<br><br>';
print_r(error_get_last());
}
$response = json_decode($result, true);
curl_close($ch);
return $response;
}
Edit Added error check

Why does that keep happening?
The server you send the request to makes this happen. It is just that requests can fail.
Deal with it and design for failure.

Related

Submit a ASPX form with curl from php

The form can be fount here: https://services.smartree.com/mcdonalds/applicationform/WebForms/ApplyToJob.aspx
After I get __VIEWSTATE & __EVENTVALIDATION I start the submit process.
// Start Submit process
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt($ch, CURLOPT_COOKIEFILE, $ckfile);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_STDERR, $f);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
// Populate all POST fields
$postfields = array();
$postfields['__EVENTVALIDATION'] = urlencode($eventValidation);
$postfields['__VIEWSTATEENCRYPTED'] = urlencode('');
$postfields['__VIEWSTATEGENERATOR'] = urlencode($eventGenerator);
$postfields['ucApplyToJob$cmbSelectedJob']=urlencode($values['job_type']);
$postfields['ucApplyToJob$txtLastName'] = $values['lname'];
$postfields['ucApplyToJob$txtFirstName'] = $values['name'];
$p = "";
foreach ($postfields as $k => $v) {
$p .= $k . '=' . $v . '&';
}
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);
$ret = curl_exec($ch); // Get result after login page.
$err = curl_errno($ch);
$errmsg = curl_error($ch);
$header = curl_getinfo($ch);
curl_close($ch);
$header['errno'] = $err;
$header['errmsg'] = $errmsg;
$header['content'] = $ret;
print_r($header);
I have populated all required fields but nothing. Please help.

PHP Curl Multi post json data

I got stuck for 2 hours on this php curl multiple request
i wantta make post json data start from 1111(This is a start point and as a verificationCode) to 1121(end point 1111 + $process_count)
check this out guys :
<?php
$url = "https://api.mywebsite.com/myapp/customer/verification";
$mh = curl_multi_init();
$handles = array();
$process_count = 10;
for($c=1111;$c <= 1121;$c++){
$data_verification = array(
"phone" => "+6285643103039", // +6285643103039 9025
"verificationCode" => $c
);
$str_verification = json_encode($data_verification);
}
while ($process_count--)
{
$ch = curl_init($url);
$headers= array('Accept: application/json','Content-Type: application/json');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_TIMEOUT, 4000);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_POSTFIELDS,$str_verification);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
ob_start();
curl_multi_add_handle($mh, $ch);
$handles[] = $ch;
}
$running=null;
do
{
curl_multi_exec($mh, $running);
}
while ($running > 0);
for($i = 0; $i < count($handles); $i++)
{
$out = curl_multi_getcontent($handles[$i]);
echo "$i. ";
print $out . "\r\n";
echo "<br>";
curl_multi_remove_handle($mh, $handles[$i]);
}
curl_multi_close($mh);
?>
But
curl_setopt($ch, CURLOPT_POSTFIELDS,$str_verification); always given end point value 1121.
And doesn't looping
from 1111 to 1121.
Anyone can figure it out ? i'll glad for any help
You are doing a mistake in your first loop, you are erasing data every time into only one variable and not array
$str_verification = json_encode($data_verification);
Here is what I suggest you to do :
$str_verification = array();
for($c=1111;$c <= 1121;$c++){
$data_verification = array(
"phone" => "+6285643103039", // +6285643103039 9025
"verificationCode" => $c
);
$str_verification[] = json_encode($data_verification);
}
for ($i = 0; $i != 10; $i++)
{
$ch = curl_init($url);
$headers= array('Accept: application/json','Content-Type: application/json');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_TIMEOUT, 4000);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_POSTFIELDS,$str_verification[$i]);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
ob_start();
curl_multi_add_handle($mh, $ch);
$handles[] = $ch;
}

regarding php curl and curl command

Php curl returning blank page for some sites where curl command working fine.
for example : curl www.wikipedia.org generating output but php curl giving blank pages with <html> tags
$ch = curl_init(); // initialize curl with given url
//TBD: all setopt commands return true/false. Should be handled
//curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER["HTTP_USER_AGENT"]); // set useragent
$res = curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // write the response to a variable
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // follow redirects if any
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout); // max. seconds to execute
curl_setopt($ch, CURLOPT_FAILONERROR, 0); // stop when it encounters an error
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // iNetClean is web-crawling, no need to verify certificates
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_ENCODING, ""); // accept all encodings - identity, deflate, gzip
// now set the URL
curl_setopt($ch, CURLOPT_URL, $url);
$fp_out = fopen($html_file, 'w');
if (!$fp_out) {
if ($DEBUG) {
error_log("couldn't create " . $html_file);
}
} else {
if ($DEBUG) {
error_log("file created " . $html_file);
}
}
$fp_err = fopen($html_err_file, 'w');
if (!$fp_err) {
if ($DEBUG) {
error_log("couldn't create " . $html_err_file);
}
} else {
if ($DEBUG) {
error_log("error file created " . $html_err_file);
}
}
curl_setopt($ch, CURLOPT_FILE, $fp_out); //rawurlencode($url) . "txt"); // for debugging only
curl_setopt($ch, CURLOPT_STDERR, $fp_err);
$result = curl_exec($ch);
//0 size file is created if no data is downloaded or URL does not exist such as pron00.com. Hence added handler to such errors.
if (#filesize($html_file) > 0) {
//file exists and contain some data
} else {
return false;
}
if ($result == false) {
trigger_error(curl_error($ch));
if ($DEBUG) {
error_log("Curl_exec fail");
}
return false;
}
fclose($fp_out);
fclose($fp_err);
curl_close($ch);
return $result;
You can do this using below code:
<?php
$debug = 1;
$fb_page_url = "http://www.wikipedia.org";
$cookies = 'cookies.txt';
touch($cookies);
$uagent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/36.0.1985.125 Chrome/36.0.1985.125 Safari/537.36';
/**
Get __VIEWSTATE & __EVENTVALIDATION
*/
$ch = curl_init($fb_page_url);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookies);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $uagent);
$html = curl_exec($ch);
curl_close($ch);
preg_match('~<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="(.*?)" />~', $html, $viewstate);
preg_match('~<input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="(.*?)" />~', $html, $eventValidation);
$viewstate = $viewstate[1];
$eventValidation = $eventValidation[1];
/**
Start Fetching process
*/
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $fb_page_url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookies);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookies);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 9850);
curl_setopt($ch, CURLOPT_USERAGENT, $uagent);
// Collecting all POST fields
$postfields = array();
//$postfields['__EVENTTARGET'] = ""; //this is for further clicking any link
//$postfields['__EVENTARGUMENT'] = ""; //this is for further clicking any link
$postfields['__LASTFOCUS'] = "";
$postfields['__VIEWSTATE'] = $viewstate;
$postfields['__EVENTVALIDATION'] = $eventValidation;
$postfields['hidStates'] = "";
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);
$ret = curl_exec($ch); // Get result as fetched web page.
if ($debug) {
echo $ret;
}
curl_close($ch);
?>

Accessing Google Bookmarks server side with PHP

I used to access my Google Bookmarks, server side with this PHP code:
$curlObj = curl_init();
curl_setopt($curlObj, CURLOPT_URL, "https://www.google.com/bookmarks/?output=rss");
curl_setopt($curlObj, CURLOPT_USERPWD, "whatever#googlemail.com:mypassword");
curl_setopt ($curlObj, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec($curlObj);
echo $response;
curl_close($curlObj);
Formerly, with the code above, I would have seen an XML feed.
Now it shows "302 Your document has moved. Click Here".
The link takes me to a login page.
Any ideas?
Thanks.
such authorization is no longer working. you need auth via https://accounts.google.com/ServiceLogin and after get https://www.google.com/bookmarks/?output=rss
example:
<?
$USERNAME = 'aaa';
$PASSWORD = 'bbb';
$ch = curl_init();
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 120);
curl_setopt($ch, CURLOPT_TIMEOUT, 120);
curl_setopt($ch, CURLOPT_URL,
'https://accounts.google.com/ServiceLogin?hl=en&service=bookmarks&continue=http://www.google.com/bookmarks');
$data = curl_exec($ch);
$formFields = getFormFields($data); // my code to get form fields, ask if you want it
$formFields['Email'] = $USERNAME;
$formFields['Passwd'] = $PASSWORD;
unset($formFields['PersistentCookie']);
$post_string = '';
foreach($formFields as $key => $value) {
$post_string .= $key . '=' . urlencode($value) . '&';
}
$post_string = substr($post_string, 0, -1);
curl_setopt($ch, CURLOPT_URL, 'https://accounts.google.com/ServiceLoginAuth');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_string);
$result = curl_exec($ch);
$info = curl_getinfo($ch);
// echo var_dump($info);
if ($info['url']=="https://accounts.google.com/ServiceLoginAuth")
{
die("Login failed");
var_dump($result);
} else {
curl_setopt($ch, CURLOPT_URL, 'http://www.google.com/bookmarks/?output=rss');
curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_POSTFIELDS, null);
$result = curl_exec($ch);
var_dump($result);
}
function getFormFields($data)
{
if (preg_match('/(<form id=.?gaia_loginform.*?<\/form>)/is', $data, $matches)) {
$inputs = getInputs($matches[1]);
return $inputs;
} else {
die('didnt find login form');
}
}
function getInputs($form)
{
$inputs = array();
$elements = preg_match_all('/(<input[^>]+>)/is', $form, $matches);
if ($elements > 0) {
for($i = 0; $i < $elements; $i++) {
$el = preg_replace('/\s{2,}/', ' ', $matches[1][$i]);
if (preg_match('/name=(?:["\'])?([^"\'\s]*)/i', $el, $name)) {
$name = $name[1];
$value = '';
if (preg_match('/value=(?:["\'])?([^"\'\s]*)/i', $el, $value)) {
$value = $value[1];
}
$inputs[$name] = $value;
}
}
}
return $inputs;
}

PHP CURL post request and Error 417

function query($url, $pfields = 0, $cookie = 0)
{
curl_setopt($ch, CURLOPT_HEADER, 1);
if (!empty($pfields))
{
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $pfields);
}
if (!empty($cookie))
{
curl_setopt($ch, CURLOPT_COOKIE, $cookie);
}
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_ENCODING,'gzip');
if (!$login)
{
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
}
$content = curl_exec($ch);
return $content;
}
$cookie = 'sessionID=3864cab58412ec567b634db3c317898;OAGEO=RU%7C%7C%7C%7C%7C%7C%7C%7C%7C%7C;';
$p = '++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++';
$post = 'clientid=23&campaignid=52&bannerid=111&appendsave=1&appendtype=0&append=' . urlencode($p) . '&submitbutton=';
echo query('http://example.com/in.php', $post, $cookie);
This code is returned 417 error(
BUT $p is not usage urlencode but IS OK but +(plus) change for " "(space)
Sooooorry for my very bad english
Try adding this:
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Expect:'));

Categories