I'm creating a crawler to capture some public information.
However, it is returning:
Access Denied
You don't have permission to access "http://www.americanas.com.br/" on this server.
Using Postman to test a request, cURL works perfectly. I even got the code generated by Postman (as shown below), but when I use it directly on my PHP server, return the error informed above.
My cURL code:
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://www.americanas.com.br/",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"cache-control: no-cache",
"postman-token: 112ebf89-1bb7-aa7a-0645-cdeabcf96488"
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if($err) echo "cURL Error #:" . $err;
else echo $response;
exit();
I found that there are sites with more complex locks. In these cases, it is necessary to use more complete crawler solutions.
The one I'm using and is working is Proxycawl (https://proxycrawl.com/).
Your postman is querying https://www.americanas.com.br/ while from the error message we can suppose that in your crawler you are querying http://www.americanas.com.br/
Related
I have an auth.php file that should make a request to an API with some headers, data and stuff.
I tried Postman, and gave me a response almost immediately.
I copied the code (PHP > cURL) and tried it, and it would be waiting for MYPRIVATESITE.com for 30 seconds (I set the timeout to that), and then just cURL ERROR: TIMED OUT (or something like that).
What did I do wrong? It works with e.g. postman, so why not my website?
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://discordapp.com/api/v6/oauth2/token",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => "client_id=PRIVATEID&client_secret=PRIVATEKEY&grant_type=authorization_code&code=$code&redirect_uri=https%3A%2F%2Fkanebot.epizy.com%2Fauth.php&scope=identify%20guilds&undefined=",
CURLOPT_HTTPHEADER => array(
"Content-Type: application/x-www-form-urlencoded",
"cache-control: no-cache"
)
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}
Note: The PRIVATEKEY and PRIVATEID are there, I just remove them because I don't want anyone else to steal it. It's defined, and it worked (read up).
The $code is also defined.
You are missing the & operator in your POSTFIELDS between the client_secret and the grant_type
try to add the & and see if its working after (it will sure solve one of the problems you have)
client_id=PRIVATEID&client_secret=PRIVATEKEYgrant_type=authorization_code&code=$code&redirect_uri=https%3A%2F%2Fkanebot.epizy.com%2Fauth.php&scope=identify%20guilds&undefined=
UPS has recently raised their security standards. As a result, on one host in particular, I get a 403 when attempting to do a GET to their rates API:
"http://www.ups.com/using/services/rave/qcostcgi.cgi?accept_UPS_license_agreement=yes&10_action=4&13_product=GNDRES&14_origCountry=US&15_origPostal=98584&19_destPostal=33773&22_destCountry=US&23_weight=2.375&47_rate_chart=Regular+Daily+Pickup&48_container=00&49_residential=1"
(I'm doing the GET in PHP using cURL.) I notice that this host is using an older cURL (7.19.7) and and older NSS (NSS/3.27.1), but I have other hosts that are using these versions where the GET will work.
What can I do to track down the issue?
This works fine:
<?php
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "http://www.ups.com/using/services/rave/qcostcgi.cgi?accept_UPS_license_agreement=yes&10_action=4&13_product=GNDRES&14_origCountry=US&15_origPostal=98584&19_destPostal=33773&22_destCountry=US&23_weight=2.375&47_rate_chart=Regular+Daily+Pickup&48_container=00&49_residential=1",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_POSTFIELDS => "",
CURLOPT_HTTPHEADER => array(
"cache-control: no-cache"
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}
Response:
UPSOnLine4%1DM%98584%US%33773%US%108%3%130.41%0.00%130.41% 8:00 A.M.%
4%1DA%98584%US%33773%US%108%3%98.23%0.00%98.23%10:30 A.M.%
4%1DP%98584%US%33773%US%138%3%89.08%0.00%89.08%End of Day%
4%2DA%98584%US%33773%US%208%3%45.79%0.00%45.79%End of Day%
4%3DS%98584%US%33773%US%308%3%36.05%0.00%36.05%End of Day%
4%GND%98584%US%33773%US%008%3%16.86%0.00%16.86%End of Day%
I am using moodle rest plugins for mobile web-services and from this tutorial I have enabled rest API and other related services as per the documentation when I try to test the web service then I got following error
{"exception":"moodle_exception","errorcode":"noauthheader","message":"No Authorization header found in request sent to Moodle"}
I have the cross check the moodle token even try may new tokens but no luck
here is me test php code to check the web-services:
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "mydomain.com/webservice/restful/server.php/core_course_get_courses",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => '{"options": {"ids":[2]}}',
CURLOPT_HTTPHEADER => array(
"accept: application/json",
"authorization: {4838e6771cdbfd1eefbf37b3839587d9}",
"cache-control: no-cache",
"content-type: application/json",
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}
please let me know, what i am doing wronge
thanks
I am new to understanding the coding languages and have started to learn things using wordpress.
I am trying to integrate a payment gateway and could generate the access_token and refresh_token which i need to store the same into my database for making further api calls for payments and user data updation.
The following is my code to run to generate tokens.
Please help as to how to store the token values in database.
I have tried replicating the wordpress register code to insert values, but
could not succeed.
<?php
session_start();
include_once "page-signup.php";
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://test.instamojo.com/oauth2/token/",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS =>
"client_id=gC4z8rBV55SDiavZobpvZCcanwK3mbnY
&client_secret=dYFbCXaX9WUy1c3kkCXV8JRJkTxmLcxCXwncVKWlWsh8c0QOI
5Uz30PCzOieC879RT7PLsEwrRKZDvZXqYpF5fZiE2Z62z3dly7p7ZUbGHTmOWmBsh3
&grant_type=password
&username=$username
&password=$password",
CURLOPT_HTTPHEADER => array(
"Cache-Control: no-cache",
"Content-Type: application/x-www-form-urlencoded",
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
$json_decode=json_decode($response,true);
$refresh_token=$json_decode['refresh_token'];
$access_token=$json_decode['access_token'];
}
echo $response;
?>
Probably the best way to store and retrieve this info from wordpress db is to use the Options API: with only one command you can store, retrieve and delete the token.
Normally the wordpress options uses a normalized array but you can also store directly a json token or simply a string.
I have been using postman to explore a REST interface. When using Postman's code generation feature, regardless of which programming language I select, Postman will always add a postman-token attribute in the header. Why is it there?
See for example PHP Curl:
<?php
$curl = curl_init();
curl_setopt_array($curl, array(CURLOPT_URL => "https://myURL.com,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"authorization: Basic abcdefghijklmnop",
"cache-control: no-cache",
"postman-token: wt53gwg-e9bb-645d-g53d-e42f8765aut0"
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}
This is primarily used to bypass a bug in Chrome. If an XMLHttpRequest is pending and another request is sent with the same parameters then Chrome returns the same response for both of them. Sending a random token avoids this issue. This can also help you distinguish between request on the server side.
See docs/settings postman.