I'm trying to scrape data from google results and for this doing curl request which works fine for few searches but after that it starts showing "Our systems have detected unusual traffic from your" then tried using proxy but still getting the same however; the ip is now changed after uisng proxy but not the error.
Error Image
Here is the code i'm using;
$curl=curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_PROXY, 'ip');
curl_setopt($curl, CURLOPT_PROXYPORT, 'port');
curl_setopt($curl, CURLOPT_PROXYUSERPWD,'user:pass');
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($curl);
if($result === FALSE) {
die(curl_error($curl));
}
curl_close($curl);
Related
When I run the below code on server it only shows the blank page and suddenly stop further execution, I also checked the cUrl on server which is installed.
Here is my code.
$ftp_server = 'ftps://'.'server/Voorraadtonen link.csv';
$ch = curl_init(str_replace(" ","%20",$ftp_server));
curl_setopt($ch, CURLOPT_URL, $ftp_server);
curl_setopt($ch, CURLOPT_USERPWD,'username'.':'.'password');
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_FTP_SSL, CURLFTPSSL_TRY);
curl_setopt($ch, CURLOPT_FTPSSLAUTH, CURLFTPAUTH_TLS);
//curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_PORT, 990);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0');
curl_setopt($ch, CURLOPT_VERBOSE,true);
$output = curl_exec($ch);
$error_no = curl_errno($ch);
echo $output; exit;
Latest update!
You have more than 1 errors in your codes,
you are using FTPS in url which requires SSL verification, and its false in
your codes.
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
//Dont use try! you shouldnt use
curl_setopt($ch, CURLOPT_FTP_SSL, CURLFTPSSL_TRY);
They should be true : SSL doesnt support true so they should be like following on #dharman warn in another answer.
But turning ssl true will require another setup like cacert file
etc. lik so
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 1);
//and include cacert.pem
curl_setopt($ch, CURLOPT_CAINFO, dirname(__FILE__) . '/cacert.pem');
Download cacert file here : https://curl.haxx.se/docs/caextract.html
2.Your url is not a true url $ftp_server = 'ftps://'.'server/Voorraadtonen link.csv';, this url will get nothing, but it should return an error atleast in error_log file, as you said all errors reporting are enabled
3.Your code should look like this
$curl = curl_init();
$file = fopen("link.csv", 'w');
curl_setopt($curl, CURLOPT_URL, "ftp://ftp.site.com/link.csv");
//Make sure for correct url
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 1);
curl_setopt($curl, CURLOPT_CAINFO, dirname(__FILE__) . '/cacert.pem');
//Make sure for correct url
curl_setopt($curl, CURLOPT_FILE, $file);
curl_setopt($curl, CURLOPT_USERPWD, "$_FTP[username]:$_FTP[password]");
//Make sure for your ftp credentials
curl_setopt($curl, CURLOPT_TIMEOUT, 20); //20 seconds will be enough
curl_exec($curl);
echo curl_errno($ch);
echo curl_error($ch);
curl_close($curl);
fclose($file);
1 more thing left headers should not be required but in case its required.
curl_setopt($curl, CURLOPT_HEADER, false); //Or
curl_setopt($curl, CURLOPT_HEADER, true);
Now it should work without any problem
NOTE : Example code is a working example you can edit it to your requirements
UPDATE : After modification you said you did in your codes (Still not showing us), finaly we get an error. once again I am asking you to add modified code into your question.
Error_no 28 cURL error 28: Connection timed out
the cURL 28 error occurs when the cURL request isn’t completed in a certain amount of time.
This happens when the cURL timeout value is set too low or when a firewall is blocking the cURL request.
Another possibility is a security module, for example the Apache mod_security module.
To fix the cURL error 28 you can contact your hosting provider.
So basicaly!
Your server is blocking. your credentials not match to required
credentails. SSL is required by server, but you are not setting it up.
Your function runing max of your Server Memory Limits settings.
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, "ftp.site.com/link.csv");
//make sure your path to file is correct
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_USERPWD, "$_FTP[username]:$_FTP[password]");
//make sure your login credentials correct
curl_setopt($curl, CURLOPT_TIMEOUT, 500);
//Set timeout for connection
curl_exec($curl);
echo curl_errno($ch);
echo curl_error($ch);
//Get errors
curl_close($curl);
//Importand close curl connection.
I am using Converge API to transfer funds from Credit Cards to my merchant account in my system.
This is the API URL that I am using:
https://api.convergepay.com/VirtualMerchant/process.do
Sometimes I am having connectivity issues to the API and as a result, I am getting blank screens or similar errors.
Is there a way I can check if the API is available before I do the CCSALE transaction? This is a simple code I setup to try and achieve this task.
$url = 'https://api.convergepay.com/VirtualMerchant/process.do';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_VERBOSE, true);
$result = curl_exec($ch);
//$result will be false if the CURL request fails.
curl_close($ch);
Try getting the error info and then redirect the user. Use the code below to get a detailed response from CURL
$url = 'https://api.convergepay.com/VirtualMerchant/process.do';
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_VERBOSE, true);
//Tell cURL that it should only spend 10 seconds
//trying to connect to the URL.
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 10);
//A given cURL operation should only take
//30 seconds max.
curl_setopt($curl, CURLOPT_TIMEOUT, 30);
$httpCode = curl_getinfo($curl , CURLINFO_HTTP_CODE);
$response = curl_exec($curl);
if ($response === false)
$response = curl_error($curl);
echo stripslashes($response);
curl_close($curl);
I should connect to a webservice that use Basic Authntication and GET method to fetch a json string. when I enter url in browser I get correct result but when I try it using cUrl I get error bellow:
Unknown SSL protocol error in connection to www.example.com:443
here is code:
$url = "https://www.example.com/api/Voucher/VerifyDiscountCode?discountCode=test&cellPhoneNumber=123456&otp=false";
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_TIMEOUT, 120);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_HEADER, 0);
//curl_setopt($curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4);
curl_setopt($curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_USERPWD, "test:test");
$data = curl_exec($curl);
if($data === false) $data = curl_error($curl);
$status_code = curl_getinfo($curl, CURLINFO_HTTP_CODE);
curl_close($curl);
echo $data;
note: it is funny that this code works in my local pc but not in server! I am sure it is not firewall thing. not sure it is from php configuration or what?
I have an issue trying to implement Neteller's REST API. I have created my app in the merchant account (I don't have an SSL certificate in my site), I have a static IP for my web site and it is added to Neteller's allowed IP list, but for some reason I always receive a null response back from them.
This is my code:
$username = 'XXXXXXXXXXXXXXXX';
$password = 'XXxxxxxXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
$curl = curl_init();
curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($curl, CURLOPT_URL, "https://api.neteller.com/v1/oauth2/token?grant_type=client_credentials");
curl_setopt($curl, CURLOPT_USERPWD, "$username:$password");
curl_setopt($curl, CURLOPT_HTTPHEADER, array("Content-Type:application/json", "Cache-Control:no-cache"));
curl_setopt($curl, CURLOPT_POSTFIELDS, array("scope"=>"default"));
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
$serverOutput = curl_exec($curl);
$info = curl_getinfo($curl);
echo $serverOutput; // null response
Can anyone help me to solve this issue?
If you are receiving a null response to the curl_exec() method, this could indicate a problem establishing the HTTP connection.
try adding:
print_r(curl_errno($curl));
This will return the curl error code, which you can then check against this list.
Most probably it is a problem establishing the SSL connection - because the certificate issuing authority is not trusted by your cUrl installation. You can fix this by updating the trusted certificate authorities list. More info here.
$url = 'https://api.stackexchange.com/2.2/answers/'.$id.'?order=desc&sort=activity&site=stackoverflow&filter=!-*f(6t*ZdXeu&key=MY_KEY';
gzdecode(file_get_contents ($url)) ;
this caused me problems today when I played with stackoverflow APIs
alternative approach - just use CURL, it does decompression automatically
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_TIMEOUT, 400);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true); // Follow redirects, need this if the url changes
curl_setopt($curl, CURLOPT_MAXREDIRS, 5); //if http server gives redirection responce
curl_setopt($curl, CURLOPT_ENCODING, "gzip"); // the page encoding
$data = curl_exec($curl); // data already decompressed
curl_close($curl);