this is an issue I've run into before using curl. We do a mass data import at the beginning of the day for and part of that is geocoding some addresses. We use google's API to do this, so a simple curl loop (should) work, at least that's what I thought.
Here are my two functions: Note that the properties variable contains about 100 entries. However, no matter how many times I refresh, the loop always stops calling the curl function after the 5th iteration. Note that the loop does not terminate, only the call to the curl function appears to be lost.
function geocode()
{
$service_url = 'https://maps.googleapis.com/maps/api/geocode/json?key=' . GAPI_KEY . '&address=';
$properties = $this->listing_model->get_non_geocoded();
if ($properties) {
foreach ($properties->result() as $property) {
$service_url .= urlencode($property->address . "," . $property->city . "," . $property->state . " " . $property->zip);
try {
$search = $this->curl($service_url);
} catch (Exception $e) {
var_dump($e);
}
}
}
}
function curl($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($ch);
curl_close($ch);
unset($ch);
return $result;
}
And of course the var dumps:
...previous (first) 3 entries...
string '14' (length=2)
object(stdClass)[116]
public 'lat' => float #######
public 'lng' => float #######
string '15' (length=2)
object(stdClass)[131]
public 'lat' => float #######
public 'lng' => float #######
string '16' (length=2)
null <-- ? Should be from curl
string '17' (length=2)
null <-- ? Should be from curl
string '18' (length=2)
null <-- ? Should be from curl
string '19' (length=2)
null <-- ? Should be from curl
According to the API's documentation found here: https://developers.google.com/maps/documentation/geocoding/, it looks like you can only do five calls a second with a total of 2500 requests per day.
So now, I think your question is how to properly handle the rate limit and it looks like there are some good ideas here: How to manage request limited API calls
(Keeping track of every request, sleeping between every set of five requests, etc.)
Best of luck!
EDIT: Sorry! Didn't see that you had it solved. Figured that since the limit on the API was five and your loop was failing every five iterations, there was a correlation.
Related
I have a script that is responsible for extracting names of people from an external web page by passing an ID as a parameter.
Note: The information provided by this external website is public access, everyone can check this data.
This is the code that I created:
function names($ids)
{
$url = 'https://www.exampledomain.com/es/query_data_example?name=&id='.$ids;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER,array("Accept-Lenguage: es-es,es"));
curl_setopt($ch, CURLOPT_TIMEOUT,10);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION,1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$html = curl_exec($ch);
$error = curl_error($ch);
curl_close($ch);
preg_match_all('/<tr class="odd"><td><a href="(.*?)">/',$html ,$matches);
if (count($matches[1] == 0))
{
$result = "";
}
else if(count($matches[1] == 1))
{
$result = $matches[1][0];
$result = str_replace('/es/person/','', $result);
$result = substr($result, 0,-12);
$result = str_replace('-', ' ', $result);
$result = ucwords($result);
}
return $result;
}
Note2: in the variable $ url I have placed an example url, it is not the real url. It's just an exact example of the original URL that I use in my code.
I make the call to the function, and I show the result with an echo:
$info = names('8476756848');
echo $info;
and everything is perfect, I extracted the name of the person to whom that id belongs.
The problem arises when I try to query that function within a for(or while) loop, since I have an array with many ids
$myids = ["2809475460", "2332318975", "2587100534", "2574144252", "2611639906", "2815870980", "0924497817", "2883119946", "2376743158", "2387362041", "2804754226", "2332833975", "258971534", "2574165252", "2619016306", "2887098054", "2449781007", "2008819946", "2763767158", "2399362041", "2832047546", "2331228975", "2965871534", "2574501252", "2809475460", "2332318975", "2587100534", "2574144252", "2611639906", "2815870980", "0924497817", "2883119946", "2376743158", "2387362041", "2804754226", "2332833975", "258971534", "2574165252", "2619016306", "2887098054", "2449781007", "2008819946", "2763767158", "2399362041", "2832047546", "2331228975", "2965871534", "2574501252", "2809475460", "2332318975", "2587100534", "2574144252", "2611639906", "2815870980", "0924497817", "2883119946", "2376743158", "2387362041", "2804754226", "2332833975", "258971534", "2574165252", "2619016306", "2887098054", "2449781007", "2008819946", "2763767158", "2399362041", "2832047546", "2331228975", "2965871534", "2574501252"];
//Note: These data are for example only, they are not the real ids.
$size = count($myids);
for ($i=0; $i < $size; $i++)
{
//sleep(20);
$data = names($myids[$i]);
echo "ID IS: " . $myids[$i] . "<br> THE NAME IS: " . $data . "<br><br>";
}
The result is something like this:
ID IS: 258971534
THE NAME IS:
ID IS: 2883119946
THE NAME IS:
and so on. I mean, it shows me the Ids but the names do not extract them from the names function.
It shows me the whole list of ids but in the case of the names it does not show me any, as if the function names does not work.
If I put only 3 ids in the array and run the for loop again, then it gives me the names of those 3 ids, because they are few. But when the array contains many ids, then the function already returns no names. It is as if the multiple requests do not accept them or limit them, I do not know.
I have placed the function set_time_limit (0) at the beginning of my php file; to avoid that I get the error of excess time of 30 seconds.
because I thought that was why the function was not working, but it did not work. Also try placing a sleep (20) inside the cycle, before calling the function names to see if it was that it was making many requests very quickly to said web page but it did not work either.
This script is already in production on a server that I have hired and I have this problem that prevents my script from working properly.
Note: There may be arrays with more than 2000 ids or I am even preparing a script that will read files .txt and .csv that will contain more than 10000 ids, which I will extract from each file and call the function name, and then those ids and the names will be saved in a table from a mysql database.
Someone will know why names are not extracted when there are many ids but when they are few for example 1 or 10 the function name does work?
I'm using softtouch API with WordPress, and posting data to API via curl.
But in response I am not able to send the big integer value in function. I'm not getting is that data type range issue or curl.
Below is my code:
//create reservation
$prod_items = array();
$single_item = array('product_uid'=>11449701010101);
$prod_items[] = $single_item;
$res_params = array(
'customer_id' => 1111,
'payment_type' => '',
'invoice_address_id' => 123,
'delivery_address_id' => 142,
'giftlist_id' => '',
'store_id' => '',
'items' => $prod_items
);
$res_url = $base_url . 'reservations';
$res_content = json_encode($res_params);
$res_curl = curl_init();
curl_setopt($res_curl, CURLOPT_HTTPHEADER, array('Authorization: ' . $authToken, 'Content-Type: application/json'));
curl_setopt($res_curl, CURLOPT_POST, true);
curl_setopt($res_curl, CURLOPT_POSTFIELDS, $res_content);
curl_setopt($res_curl, CURLOPT_URL, $res_url);
curl_setopt($res_curl, CURLOPT_RETURNTRANSFER, true);
$res_response = curl_exec($res_curl);
if ($res_response === FALSE)
die(curl_error($res_curl));
curl_close($res_curl);
$res_array = json_decode($res_response);
While sending data to the curl_exec() function it removes the last two digits of the product_uid, as I passed it 11449701010101 it send it as 114497010101.
Is that any integer range issue or something curl function issue?
tl:dr There would seem to be something else (in the script that processes the CURL request) that is truncating the product_uid. (?)
array('product_uid'=>11449701010101)
If you are on a 32 bit system (PHP compiled as 32 bit) then 11449701010101 does exceed the 32 bit integer range. However, in such cases, PHP silently converts the number to a float and nothing is "lost" in this instance.
json_encode($res_params);
The PHP function json_encode() converts the passed array into a string respresentation (JSON string). Nothing is lost. The 11449701010101 value is preserved.
While sending data to the curl_exec() function it removes the last two digits
The (POST) data being transmitted is an ordinary string, so nothing can be lost to the individual properties at this stage.
If the receiving script then decodes the transmitted JSON string, encodes it again and sends it back, the data is returned intact. The product_uid is a float, not an integer (as it is in the original data).
If you specifically force 11449701010101 to an integer (eg. (int)11449701010101) then you get -681801035 - the last 2 digits are not simply truncated. For the last 2 digits to be truncated there would seem to be some kind of string manipulation going on?
So, there would seem to be something else (not shown here) during the processing of this data that is perhaps truncating the value. (?)
I have a homemade web interface with several text and text area input fields. The primary text field are being sent to Parse in a php script via REST API. It works brilliant until I input approximately more than 3.000 characters in one of the textarea input fields.
When below approx. 3.000 characters, everything works fine. When above approx. 3.000 characters, the data gets sent to parse, and I can see it among the other push messages in my dashboard with a green check mark in "status", but the push notification is NOT being sent to any of the subscribed devices!
I'm using it for sending out news articles, so it is a must to be able to send whole articles above approx. 3.000 characters.
Is there any limitations in relation to the amount of characters to be sent or anything similar?
Does anyone know something that can help me or has any suggestions to how I can resolve this issue?!
The Parse parts of my script:
/*
* PUSHING THE ARTICLE NOTIFICATION TO DEVICES SUBSCRIBING TO SPECIFIC CHANNELS(S)
*/
$url = PUSH_URL;
$appId = PUSH_APP_ID;
$restKey = PUSH_REST_KEY;
$channels = array($channelArray);
$headers = array(
"Content-Type: application/json",
"X-Parse-Application-Id: " . $appId,
"X-Parse-REST-API-Key: " . $restKey
);
$objectData = json_encode(array(
"data" => array(
"id" => $id ,
"watchTitle" => stripslashes($watchTitle) ,
"title" => $textTitle ,
"alert" => $textSubtitle ,
"mainText" => $textMain ,
"imgUrl" => $imgUrl ,
"uri" => "http://watch.jgskytte.com/parse/article.php?p=" . $id ,
),
"channels" => $channels ,
));
$rest = curl_init($url);
curl_setopt($rest, CURLOPT_HTTPHEADER, $headers);
curl_setopt($rest, CURLOPT_POST, true);
curl_setopt($rest, CURLOPT_POSTFIELDS, $objectData);
curl_setopt($rest, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($rest);
echo $response;
curl_close($rest);
There's no explicit limit on string length, but Parse Objects are limited to 128k. Otherwise, you should use a Parse File.
This is probably occurring because you're hitting the maximum length limit for a URL. To get around this, you can use a POST request instead which will send the query string as part of the HTTP request body, which has no such length limit.
Since Parse's REST API expects this query to be sent as a GET request, you will need to pass an additional header to let Parse know to treat this POST as a GET:
X-HTTP-Method-Override: GET
I want to open HTTPS file with PHP but this page makes a redirection to another page so fopen function doesn't parse the page I want.
I have this code :
$url = 'myHTMLPageWithParameters';
$file = file($url);
// test
var_dump($file);
And result :
array (size=12)
0 => string '<html>
' (length=7)
1 => string '<head>
' (length=7)
2 => string '<script language="javascript">
' (length=31)
3 => string 'function setTop(){top.location="/index.htm"}
' (length=45)
4 => string '</script>
' (length=10)
5 => string '<title>...</title>
' (length=19)
6 => string '</head>
' (length=8)
7 => string '
' (length=1)
8 => string '<body onLoad="setTop()">
' (length=25)
9 => string '</body>
' (length=8)
10 => string '
' (length=1)
11 => string '</html>
' (length=8)
When I display 'myHTMLPageWithParameters' in a HTML browser, I see the correct page after redirection. I'm just looking for a way to capture HTML code of the second page (after the redirection). Thanks for any help
Possible duplicate of follow redirects with curl in php
In short: it's not doable in a reliable manner.
This is not a redirection done by the server, you are getting the page that you requested. Then, that page redirects to another, but using javascript. Javascript it's interpreted by the browser, not by php, curl or any other library.
The only way I can think of, it's by using regex to find location.href or location.top and then follow those redirects. But again, there are plenty ways to redirect a page, you can't expect to parse them all!
Check out this solution from another SO post:
Will PHPs fopen follow 301 redirects?
Another option would be to use curl instead of fopen, which has an option you can set telling it to follow redirects (CURLOPT_FOLLOWLOCATION).
You can use FOLLOW_LOCATION;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "myHTMLPageWithParameters");
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$html_response = curl_exec($ch);
// We get the content
$html = str_get_html($html_response);
// Get #result div for example
$content = $html->find('#result');
I'm currently running into a massive wall due to a problem i cannot seem to solve.
The problem is that, when you issue a payment through the Facebook payment platform (facebook javascript sdk), it sends data to your callback page, which should handle the payment on the background.
This all works decent, but there is 1 problem: The order ID that facebook uses is a 64bit ID, and my server is a 32bits server, thus it loses precision on the ID when it gets saved to a variable in the callback page. This ultimately results in not being able to get a proper order_ID in the end, because it cannot save the ID.
The issue has been described on several pages on this forum, for example:
facebook credit callback, order_id changes format changes after moving to a new server
and
PHP: Converting a 64bit integer to string
Yet, on both pages there is no solution to the problem, and i cannot seem to fix this myself.
I have tried to convert the json data that facebook sends to my callback page into string data instead of an array with integers (this happens in the basic code provided by facebook), but i just cannot get this to work.
Seeing that others have overcome this problem (without having to migrate everything to a 64bits server), i am wondering how.
Is anyone able to shine a light on this subject?
Edit:
I have tried converting to string, the standard facebook code that gets called to decode the json data (code provided by facebook):
$request = parse_signed_request($_POST['signed_request'], $app_secret);
This calls the function parse_signed_request, which does:
function parse_signed_request($signed_request, $secret) {
list($encoded_sig, $payload) = explode('.', $signed_request, 2);
$sig = base64_url_decode($encoded_sig);
$data = json_decode(base64_url_decode($payload), true);
if (strtoupper($data['algorithm']) !== 'HMAC-SHA256') {
error_log('Unknown algorithm. Expected HMAC-SHA256');
return null;
}
// check sig
$expected_sig = hash_hmac('sha256', $payload, $secret, $raw = true);
if ($sig !== $expected_sig) {
error_log('Bad Signed JSON signature!');
return null;
}
return $data;
}
This function decodes the encrypted json data from facebook (using the app secret) and should decode the json data to a PHP array.
That function uses the following function (the exact:
function base64_url_decode($input) {
return base64_decode(strtr($input, '-_', '+/'));
}
Now, the above code results in the order ID not being saved properly, and it loses its precision, resulting in an id like: 4.8567130814993E+14
I have tried to use the following function to somehow decode the json data into a string (so the 64bit integer ID does not lose its precision), but to no avail:
function largeint($rawjson) {
$rawjson = substr($rawjson, 1, -1);
$rawjson = explode(',' , $rawjson);
array_walk($rawjson, 'strfun');
$rawjson = implode(',', $rawjson);
$rawjson = '{' . $rawjson . '}';
$json = json_decode($rawjson);
return $json;
}
function strfun(&$entry, $key) {
$data = explode(':', $entry);
if (FALSE === strpos($data[1], '"')) {
$data[1] = '"' . $data[1] . '"';
$entry = implode(':', $data);
}
}
Edit (Eugenes answer):
If i were to try modifying the JSON data before i use json_decode to make it a php variable, i should be using the preg_replace function?
Below is an example of the initial JSON data that gets sent to the callback page to initiate the payment process.
Here you can already see what the problem is (this is after using json_decode, the id and other data lose their precision). The ID's are modified to not reflect any real data.
If you compare the buyer ID on the top and user id on the bottom, you can see precision is lost.
Array
(
[algorithm] => HMAC-SHA256
[credits] => Array
(
[buyer] => 1.0055555551318E+14
[receiver] => 1.0055555551318E+14
[order_id] => 5.2555555501665E+14
[order_info] => {"item_id":"77"}
[test_mode] => 1
)
[expires] => 1358456400
[issued_at] => 1358452270
[oauth_token] => AAAH4s2ZCCEMkBAPiGSNsmj98HNdTandalotmoredata
[user] => Array
(
[country] => nl
[locale] => nl_NL
[age] => Array
(
[min] => 21
)
)
[user_id] => 100555555513181
)
Edit #3:
I have tried the following to make all the integers in the JSON data seen as strings, but that results in an error from the facebook platform. It does however change the integers to a string, so i do not lose precision (too bad nothing else works now xD)
preg_replace('/([^\\\])":([0-9]{10,})(,|})/', '$1":"$2"$3', $a)
Which version of PHP are you running?
If you are running a version of PHP that supports the JSON "JSON_BIGINT_AS_STRING" option, that may be your answer. You may have to modify their library wherever json_decode is being used to add that option. See http://php.net/manual/en/function.json-decode.php
If your PHP version does not support JSON_BIGINT_AS_STRING, then your options are limited to:
The hacky option: Do some kind of regex operation on the JSON string as it comes back from the FB API and wrap that big ints in double-quotes, so that they decode as a string and not a big int.
The ideal option: Bite the bullet and migrate to a 64 bit environment. It will save you from a lot of headaches in the long run.