I use this function to make cURL requests:
function curl_request($options) //single custom cURL request.
{
$ch = curl_init();
$options[CURLOPT_FOLLOWLOCATION] = true;
$options[CURLOPT_COOKIEJAR] = 'cookies.txt';
$options[CURLOPT_COOKIEFILE] = 'cookies.txt';
$options[CURLINFO_HEADER_OUT] = true;
$options[CURLOPT_VERBOSE] = true;
$options[CURLOPT_RETURNTRANSFER] = true;
$options[CURLOPT_CONNECTTIMEOUT] = 5;
$options[CURLOPT_TIMEOUT] = 5;
curl_setopt_array($ch, $options);
$response = curl_exec($ch);
curl_close($ch);
return $response;
}
The script hangs sometimes, but not always, on the $response = curl_exec($ch) line. This occurs even when the PHP script is set with infinite timeout (on the client side, Firebug takes this as "Aborted"). There is nothing in the error log.. It just doesn't get past that line when it hangs.
What could be going on? Any suggestions?
The issue seems to have been the resources of the server. When I switched to a better web host with a higher bandwidth limit things worked fine.
Related
I have spent a couple of hours reading up on this but as so yet I find no clear solutions....I am using WAMP to run as my Local server. I have a successful API call set up to return data.
I would like to store that data locally, thus reducing the number of API call being made.
For simplicity I have created a cache.json file in the same folder as my PHP scripts and when I run the process I can see the file has been accessed as the time stamp updates.
But the file remains empty.
Based on research I suspect the issue may come down to a permission issue; I have gone through the folders and files and unchecked read only etc.
Appreciate if someone could validate that my code is correct and if it is hopefully point me in the the direction of a solution.
many thanks
<?php
$url = 'https://restcountries.eu/rest/v2/name/'. $_REQUEST['country'];
$cache = __DIR__."/cache.json"; // make this file in same dir
$force_refresh = true; // dev
$refresh = 60; // once an min (set short time frame for testing)
// cache json results so to not over-query (api restrictions)
if ($force_refresh || ((time() - filectime($cache)) > ($refresh) || 0 == filesize($cache))) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_URL,$url);
$result=curl_exec($ch);
curl_close($ch);
$decode = json_decode($result,true);
$handle = fopen($cache, 'w');// or die('no fopen');
$json_cache = $decode;
fwrite($handle, $json_cache);
fclose($handle);
}
} else {
$json_cache = file_get_contents($cache); //locally
}
echo json_encode($json_cache, JSON_UNESCAPED_UNICODE);
?>
I managed to solve this by using file_put_contents(), not being an expert I do not understand why this works and the code above doesn't, but maybe this helps someone else.
adjusted code:
<?php
$url = 'https://restcountries.eu/rest/v2/name/'. $_REQUEST['country'];
$cache = __DIR__."/cache.json"; // make this file in same dir
$force_refresh = false; // dev
$refresh = 60; // once an min (short time frame for testing)
// cache json results so to not over-query (api restrictions)
if ($force_refresh || ((time() - filectime($cache)) > ($refresh) || 0 == filesize($cache))) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_URL,$url);
$result=curl_exec($ch);
curl_close($ch);
$decode = json_decode($result,true);
$handle = fopen($cache, 'w');// or die('no fopen');
$json_cache = $result;
file_put_contents($cache, $json_cache);
} else {
$json_cache = file_get_contents($cache); //locally
$decode = json_decode($json_cache,true);
}
echo json_encode($decode, JSON_UNESCAPED_UNICODE);
?>
I have a limit of 25 requests/min from PUBGs official API. For some reason instead of it requesting twice for each search its using up 4 requests. I can't figure out why. I have checked that the code isn't running twice. Only once, but still it's requesting 4 times.
UPDATE:
I tried making a separate page and apparently there is a bug somewhere calling my function twice. Still don't know why but I'm now 99% sure it's not the function itself.
Code For My Request
function getProfile($profileName, $region, $seasonDate){
// Just check if there is an acctual user
if($profileName === null){
$data->error = "Player Not Found";
$data->noUser = true;
return $data;
}else{
$season = "division.bro.official.".$seasonDate;
/*
Get The UserID
*/
$ch = curl_init("https://api.pubg.com/shards/$region/players?filter[playerNames]=$profileName");
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Authorization: Bearer APIKEY',
'Accept: application/vnd.api+json'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$rawData = json_decode(curl_exec($ch), true);
$data->playerId = $rawData["data"][0]["id"];
curl_close($ch);
// Testing if user exists
if($rawData["errors"][0]["title"] === "Not Found"){
$data->noUser = true;
$data->error = "Player Not Found";
return $data;
}else{
/*
Get The acctual stats
*/
$ch = curl_init("https://api.pubg.com/shards/$region/players/$data->playerId/seasons/$season");
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Authorization: Bearer APIKEY',
'Accept: application/vnd.api+json'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$data->playerDataJSON = curl_exec($ch);
$data->playerData = json_decode($data->playerDataJSON, true);
curl_close($ch);
return $data;
}
}
}
This is how it's getting called
if (isset($_POST['search-username'])) {
$username = $_POST['search-username'];
header("Location: /profile/$username/pc-na/2018-01/overall/tpp");
die();
}
In The actual profile php
$data = getProfile($page_parts[1], $page_parts[2], $page_parts[3]);
absolutely sure it's only called once? set a lock on it. change it to
function getProfile($profileName, $region, $seasonDate){
static $once=true;
if($once!==true){
throw new \LogicException("tried to run getProfile() twice!");
}
$once=false;
Shortly after I figure out it's not the function I realized that the culprit was an empty script I was calling. I knew this script created an error which I didn't really care about since it was empty and I had no idea why it was creating the error. For some obscure reason this script created the error. I'll just make a lesson out of it to always fix the smallest errors.
I've also been seeing this behavior - a script with a single curl_exec() request gets called twice.
The strange thing is this was only happening when running on localhost (under a wampp installation) but when run from any other webserver was fine and it is just called once.
I never managed to debug it completely but it seems to be an issue with the local server so test elsewhere if you are seeing this.
I am making a website that will check if a website is working and live. I pass in the URL of the site I would like to check and the following code will check if the site is live and return the HTTP response code as well as true or false.
function urlExists($url=NULL)
{
if($url == NULL) return false;
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($ch);
$httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
if ($httpcode == 0) {
return array (false, $httpcode);
}
else if($httpcode < 400){
return array (true, $httpcode);
} else {
return array (false, $httpcode);
}
}
With one of the sites I am testing though I am getting the HTTP response code of 0 even though I know that the site is live and working.
The site is very slow as its a large site on a not very powerful server so response times can vary between 7 - 25 seconds.
Any help would be greatly appreciated.
Thanks,
Sam
Based on these two links:-
https://curl.haxx.se/libcurl/c/CURLOPT_TIMEOUT.html
And
https://curl.haxx.se/libcurl/c/CURLOPT_CONNECTTIMEOUT.html
First one is:- set maximum time the request is allowed to take
Second one is:- timeout for the connect phase
As you said that the Site URL you are hitting is taking 7-25 second for responding. meanwhile your CURL request is terminated and closed because of these two time settings.
Increase these two time settings in your code and it will work for you.
thanks.
I will offer 2 alternatives for you to compare - along with your curl() function, you will have 3 options to see which one is better/faster for you.
Option A (all php versions), requires fopen() to be activated:
if (!$fp = fopen($url, 'r'))
{
trigger_error("Unable to open URL ($url)", E_USER_ERROR);
}
$headers = stream_get_meta_data($fp);
fclose($fp);
$http_header_info = $headers['wrapper_data'][0];
$httpCode = (int)substr($http_header_info, 9, 3);
Option B (php5+):
$headers = get_headers($url, 1);
$http_header_info = $headers[0];
$httpCode = substr($http_header_info, 9, 3);
Also, if anyone has benchmarks on these 3 approaches, i am curious to see which is more appropriate (only for retrieving http response headers of course)
Code 0 returns often when used invalid URL syntax or host not found error.
You can also call curl_error($ch) function (http://php.net/manual/en/function.curl-error.php) to determine error details.
I'm trying to get Intuit CAD API to work from within a PHP application. I've read the docs using the recommended "SDK" (well, more like a demo app), search the web and tried multiple advice scattered here and there. Here's what I came up with:
// I also use log file for curl - which doesn't add any info
$curl_logfile = fopen('/tmp/curl_debug', 'w+');
$ch = curl_init();
$options = array();
$options[CURLOPT_VERBOSE] = true;
$options[CURLOPT_RETURNTRANSFER] = true;
$options[CURLOPT_TIMEOUT] = 360;
$options[CURLOPT_CERTINFO] = false;
$options[CURLOPT_SSL_VERIFYPEER] = false;
$options[CURLINFO_HEADER_OUT] = true;
$options[CURLOPT_STDERR] = $curl_logfile;
$options[CURLOPT_URL] = $signed_request['signed_url'];
curl_setopt_array($ch, $options);
$raw = curl_exec($ch);
if ($error_num = curl_errno($ch)) {
$error_desc = curl_error($ch);
}
// debug - get all info about the request just issues
$all_data = curl_getinfo($ch);
The code above return 400 as the return code (viewable using curl_getinfo()) and empty string in $raw.
How can I make my PHP app talk to Intuit CAD API?
Also, on the same note - some API method require transferring parameters, so it seem, as part of the path. For example: getInstitutionDetails (v1/institutions/INST-ID). Can I put those variables as parameters? Otherwise, I need to craft each request non-generically, according to the specific API method used :-(
Thanks!
(I'm scraping this stuff with the permission of the website in question, by the way).
Pretty simple web scraper, was working fine when I was loading all the links by hand, but when I've tried to load them in via JSON and variables (so I can do lots of scraping with the one script and make the process more modular by just adding more links to JSON) it runs on an infinite loop.
(Page has been loading for about 15 minutes now)
Here is my JSON. Only one store is in there for testing purposes but there is going to be about 15 more.
[
{
"store":"Incu Men",
"cat":"Accessories",
"general_cat":"Accessories",
"spec_cat":"accessories",
"url":"http://www.incuclothing.com/shop-men/accessories/",
"baseurl":"http://www.incuclothing.com",
"next_select":"a.next",
"prod_name_select":".infobox .fn",
"label_name_select":".infobox .brand",
"desc_select":".infobox .description",
"price_select":"#price",
"mainImg_select":"",
"more_imgs":".product-images",
"product_url":".hproduct .photo-link"
}
]
Here is the PHP scraper code:
<?php
//Set infinite time limit
set_time_limit (0);
// Include simple html dom
include('simple_html_dom.php');
// Defining the basic cURL function
function curl($url) {
$ch = curl_init();
// Initialising cURL
curl_setopt($ch, CURLOPT_URL, $url);
// Setting cURL's URL option with the $url variable passed into the function
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
// Setting cURL's option to return the webpage data
$data = curl_exec($ch);
// Executing the cURL request and assigning the returned data to the $data variable
curl_close($ch);
// Closing cURL
return $data;
// Returning the data from the function
}
function getLinks($catURL, $prodURL, $baseURL, $next_select) {
$urls = array();
while($catURL) {
echo "Indexing: $url" . PHP_EOL;
$html = str_get_html(curl($catURL));
foreach ($html->find($prodURL) as $el) {
$urls[] = $baseURL . $el->href;
}
$next = $html->find($next_select, 0);
$url = $next ? $baseURL . $next->href : null;
echo "Results: $next" . PHP_EOL;
}
return $urls;
}
$string = file_get_contents("jsonWorkers/incuMens.json");
$json_array = json_decode($string,true);
foreach ($json_array as $value){
$baseURL = $value['baseurl'];
$catURL = $value['url'];
$store = $value['store'];
$general_cat = $value['general_cat'];
$spec_cat = $value['spec_cat'];
$next_select = $value['next_select'];
$prod_name = $value['prod_name_select'];
$label_name = $value['label_name_select'];
$description = $value['desc_select'];
$price = $value['price_select'];
$prodURL = $value['product_url'];
if (!is_null($value['mainImg_select'])){
$mainImg = $value['mainImg_select'];
}
$more_imgs = $value['more_imgs'];
$allLinks = getLinks($catURL, $prodURL, $baseURL, $next_select);
}
?>
Any ideas why the script would be running infinitely and not returning anything/stopping/printing anything to screen? I'm just gonna let it run until it stops. When I was doing this by hand it would only take a minute or so, sometimes less, so I'm sure it's a problem with my variables/json but I can't for the life of me see what the issues lie.
Can anyone take a quick look and point me in the right direction?
There is a problem with your while($catURL) loop. What do you want to do ?
Moreover, you can force to display information on your browser with the flush() command.