If/else for https connection through proxy - php

I am using this code to check a proxy server, before doing anything else. When the connection is ok, I have a file_get_contents line to connect to a website.
$host = '123.45.678.90';
$port = 80;
$waitTimeoutInSeconds = 1;
if($fp = fsockopen($host,$port,$errCode,$errStr,$waitTimeoutInSeconds)){
$aContext = array(
'http' => array(
'proxy' => 'tcp://123.45.678.90:80',
'request_fulluri' => true,
),
);
$cxContext = stream_context_create($aContext);
// connect to website using a proxy server
$file_content = file_get_contents('https://www.anything.com', False, $cxContext);
} else {
// It didn't work
}
fclose($fp);
But though the proxy connection was succesfull, I see this: Warning Cannot connect to HTTPS server through proxy? Is there any chance to have an if/else statement, that allows me to do something if there is no warning and stop doing something if there is a warning? I have looked and googled a lot, but haven't found anything I could try.
Thanks for your help!

try to use curl instead of file_get_contents and it will work
<?php
$url = 'https://www.google.com';
// to check your proxy
// $url = 'http://whatismyipaddress.com/';
$proxy = '50.115.194.97:8080';
// create curl resource
$ch = curl_init();
// set options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // read more about HTTPS http://stackoverflow.com/questions/31162706/how-to-scrape-a-ssl-or-https- url/31164409#31164409
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
// $output contains the output string
$output = curl_exec($ch);
// close curl resource to free up system resources
curl_close($ch);
echo $output;
?>
from PHPhil

if($fp = fsockopen($host,$port,$errCode,$errStr,$waitTimeoutInSeconds) doesn't make sense, if this proxy is working, you can simply use:
$aContext = array(
'http' => array(
'proxy' => 'tcp://123.45.678.90:80',
'request_fulluri' => true,
),
);
$cxContext = stream_context_create($aContext);
$file_content = file_get_contents('https://google.com', False, $cxContext);
print $file_content;

Related

Send request with user ip when scraping data in php

I am stuck in a problem I have a url which has a geo location restriction like it can only be viewed from europe or USA. My location is Asia. I want to extract all href's from the url.
However I am using curl but the problem is that it send server ip address and I want the request to be made with user ip address inorder to track a user which links he has visited. If you can guide me how to send request with user ip address and without using curl I'll be grateful.
Following is the source code. The url which I am accesing is:
http&colon;//partnerads.ysm.yahoo.com/ypa/?ct=2&c=000000809&u=http%3A%2F%2Ftrouve.autocult.fr%2F_test.php%3Fq%3Dtarif%2520skoda%2520superb%2520combi&r=&w=1&tv=&tt=&lo=&ty=&ts=1458721731523&ao=&h=1&CoNo=3292b85181511c0a&dT=1&er=0&si=p-Autocult_FRA_SERP_2%3A600x796
<?php
include_once 'simple_html_dom.php';
$html = file_get_html('iframe.html');
// find iframe from within html doc
foreach($html->find('iframe') as $iframe)
{
$src = $iframe->getAttribute('src'); // src extracted
$ch = curl_init(); // Initialise a cURL handle
// Set any other cURL options that are required
curl_setopt($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, TRUE);
curl_setopt($ch, CURLOPT_COOKIESESSION, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36');
curl_setopt($ch, CURLOPT_URL,$src);
$results = curl_exec($ch); // Execute a cURL request
//echo curl_error($ch);
curl_close($ch); // Closing the curl
$bool = TRUE; $match = array(); $int = 0;
while(preg_match('/<a[^>]+href=([\'"])(.+?)\1[^>]*>/i', $results, $matches))
{
if($bool)
{
// print captured group that's actually the url your searching for
echo $matches[2].'<br>'.'<br>'.'<br>'.'<br>';
$bool = false;
}
}
}
You can use proxy.
$ip = '100.100.100.100:234' //example $ip
curl_setopt($ch, CURLOPT_PROXY,$ip);
without curl:
$aContext = array(
'http' => array(
'proxy' => 'tcp://'.$ip,
'request_fulluri' => true,
),
);
$cxContext = stream_context_create($aContext);
$sFile = file_get_contents("http://www.google.com", False, $cxContext);
If you lookin' for proxies, there's some adresses easy to scrape:
'http://proxylist.hidemyass.com/',
'http://ipaddress.com/proxy-list/',
'http://nntime.com/proxy-ip-'.$i.'.htm',
'http://www.proxylisty.com/ip-proxylist-'.$i
over 2000 ips

Can anyone give me an example for PHP's CURLFile class?

I had a very simple PHP code to upload a file to a remote server; the way I was doing it (as has been suggested here in some other solutions) is to use cUrl to upload the file.
Here's my code:
$ch = curl_init("http://www.remotesite.com/upload.php");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, array('fileupload' => '#'.$_FILES['Filedata']['tmp_name']));
echo curl_exec($ch);
The server is running PHP 5.5.0 and it appears that #filename has been deprecated in PHP >= 5.5.0 as stated here under the CURLOPT_POSTFIELDS description, and therefore, I'm getting this error:
Deprecated: curl_setopt(): The usage of the #filename API for file uploading is deprecated. Please use the CURLFile class instead in ...
Interestingly, there is absolutely nothing about this Class on php.net aside from a basic class overview. No examples, no description of methods or properties. It's basically blank here. I understand that is a brand new class with little to no documentation and very little real-world use which is why practically nothing relevant is coming up in searches on Google or here on Stackoverflow on this class.
I'm wondering if there's anyone who has used this CURLFile class and can possibly help me or give me an example as to using it in place of #filename in my code.
Edit:
I wanted to add my "upload.php" code as well; this code would work with the traditional #filename method but is no longer working with the CURLFile class code:
$folder = "try/";
$path = $folder . basename( $_FILES['file']['tmp_name']);
if(move_uploaded_file($_FILES['file']['tmp_name'], $path)) {
echo "The file ". basename( $_FILES['file']['tmp_name']). " has been uploaded";
}
Final Edit:
Wanted to add Final / Working code for others looking for similar working example of the scarcely-documented CURLFile class ...
curl.php (local server)
<form action="<?php echo $_SERVER['PHP_SELF']; ?>" method="post" enctype="multipart/form-data">
<label for="file">Filename:</label> <input type="file" name="Filedata" id="Filedata" />
<br />
<input type="submit" name="submit" value="Submit" />
</form>
<?php
if ($_POST['submit']) {
$uploadDir = "/uploads/";
$RealTitleID = $_FILES['Filedata']['name'];
$ch = curl_init("http://www.remotesite.com/upload.php");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$args['file'] = new CurlFile($_FILES['Filedata']['tmp_name'],'file/exgpd',$RealTitleID);
curl_setopt($ch, CURLOPT_POSTFIELDS, $args);
$result = curl_exec($ch);
}
?>
upload.php (remote server)
$folder = "try/";
$path = $folder . $_FILES['file']['name'];
if(move_uploaded_file($_FILES['file']['tmp_name'], $path)) {
echo "The file ". basename( $_FILES['file']['name']). " has been uploaded";
}
There is a snippet on the RFC for the code: https://wiki.php.net/rfc/curl-file-upload
curl_setopt($curl_handle, CURLOPT_POST, 1);
$args['file'] = new CurlFile('filename.png', 'image/png', 'filename.png');
curl_setopt($curl_handle, CURLOPT_POSTFIELDS, $args);
You can also use the seemingly pointless function curl_file_create( string $filename [, string $mimetype [, string $postname ]] ) if you have a phobia of creating objects.
curl_setopt($curl_handle, CURLOPT_POST, 1);
$args['file'] = curl_file_create('filename.png', 'image/png', 'filename.png');
curl_setopt($curl_handle, CURLOPT_POSTFIELDS, $args);
Thanks for your help, using your working code I was able to solve my problem with php 5.5 and Facebook SDK. I was getting this error from code in the sdk class.
I don't thinks this count as a response, but I'm sure there are people searching for this error like me related to facebook SDK and php 5.5
In case someone has the same problem, the solution for me was to change a little code from base_facebook.php to use the CurlFile Class instead of the #filename.
Since I'm calling the sdk from several places, I've just modified a few lines of the sdk:
In the method called "makeRequest" I made this change:
In this part of the code:
if ($this->getFileUploadSupport()){
$opts[CURLOPT_POSTFIELDS] = $params;
} else {
$opts[CURLOPT_POSTFIELDS] = http_build_query($params, null, '&');
}
Change the first part (with file upload enabled) to:
if ($this->getFileUploadSupport()){
if(!empty($params['source'])){
$nameArr = explode('/', $params['source']);
$name = $nameArr[count($nameArr)-1];
$source = str_replace('#', '', $params['source']);
$size = getimagesize($source);
$mime = $size['mime'];
$params['source'] = new CurlFile($source,$mime,$name);
}
if(!empty($params['image'])){
$nameArr = explode('/', $params['image']);
$name = $nameArr[count($nameArr)-1];
$image = str_replace('#', '', $params['image']);
$size = getimagesize($image);
$mime = $size['mime'];
$params['image'] = new CurlFile($image,$mime,$name);
}
$opts[CURLOPT_POSTFIELDS] = $params;
} else {
$opts[CURLOPT_POSTFIELDS] = http_build_query($params, null, '&');
}
Maybe this can be improved parsing every $param and looking for '#' in the value.. but I did it just for source and image because was what I needed.
FOR curl_setopt(): The usage of the #filename API for file uploading is deprecated. Please usethe CURLFile class instead
$img='image.jpg';
$data_array = array(
'board' => $board_id,
'note' => $note,
'image' => new CurlFile($img)
);
$curinit = curl_init($url);
curl_setopt($curinit, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curinit, CURLOPT_POST, true);
curl_setopt($curinit, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($curinit, CURLOPT_POSTFIELDS, $data_array);
curl_setopt($curinit, CURLOPT_SAFE_UPLOAD, false);
$json = curl_exec($curinit);
$phpObj = json_decode($json, TRUE);
return $phpObj;
CURLFile has been explained well above, but for simple one file transfers where you don't want to send a multipart message (not needed for one file, and some APIs don't support multipart), then the following works.
$ch = curl_init('https://example.com');
$verbose = fopen('/tmp/curloutput.log', 'w+'); // Not for production, but useful for debugging curl issues.
$filetocurl = fopen(realpath($filename), 'r');
// Input the filetocurl via fopen, because CURLOPT_POSTFIELDS created multipart which some apis do not accept.
// Change the options as needed.
$options = array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_POST => true,
CURLOPT_HTTPHEADER => array(
'Content-type: application/whatever_you_need_here',
'Authorization: Basic ' . $username . ":" . $password) // Use this if you need password login
),
CURLOPT_NOPROGRESS => false,
CURLOPT_UPLOAD => 1,
CURLOPT_TIMEOUT => 3600,
CURLOPT_INFILE => $filetocurl,
CURLOPT_INFILESIZE => filesize($filename),
CURLOPT_VERBOSE => true,
CURLOPT_STDERR => $verbose // Remove this for production
);
if (curl_setopt_array($ch, $options) !== false) {
$result = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
} else {
// A Curl option could not be set. Set exception here
}
Note the above code has some extra debug - remove them once it is working.
Php POST request send multiple files with curl function:
<?php
$file1 = realpath('ads/ads0.jpg');
$file2 = realpath('ads/ads1.jpg');
// Old method
// Single file
// $data = array('name' => 'Alexia', 'address' => 'Usa', 'age' => 21, 'file' => '#'.$file1);
// $data = array('name' => 'Alexia', 'address' => 'Usa', 'age' => 21, 'file[0]' => '#'.$file1, 'file[1]' => '#'.$file2);
// CurlFile method
$f1 = new CurlFile($file1, mime_content_type($file1), basename($file1));
$f2 = new CurlFile($file2, mime_content_type($file2), basename($file2));
$data = array('name' => 'Alexia', 'address' => 'Usa', 'age' => 21, 'file[1]' => $f1, 'file[2]' => $f2);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://url.x/upload.php');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_SAFE_UPLOAD, false); // !!!! required as of PHP 5.6.0 for files !!!
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2) Gecko/20100115 Firefox/3.6 (.NET CLR 3.5.30729)");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); // 1, 2
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
// curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$res2 = curl_exec($ch);
echo $res2;
?>
<?php
// upload.php
$json = json_decode(file_get_contents('php://input'), true);
if(!empty($json)){ print_r($json); }
if(!empty($_GET)){ print_r($_GET); }
if(!empty($_POST)){ print_r($_POST); }
if(!empty($_FILES)){ print_r($_FILES); }
?>

Logging in to Channel Advisor

I'm trying to login to channel advisor but it output an error of: HTTP/1.1 302 Moved Temporarily.
But last week it runs perfectly that I login and retrieve my data and now I run again it has an error.
Here is my code:
$pages = array('home' =>
'https://login.channeladvisor.com/?gotourl=https%3a%2f%2fcomplete.channeladvisor.com%2f',
'login' =>
'https://login.channeladvisor.com/?gotourl=https%3a%2f%2fcomplete.channeladvisor.com%2f',
'data' =>
'https://merchant.channeladvisor.com/AM/MyInventory/View_Inventory.aspx?apid=32001263');
$ch = curl_init();
//Set options for curl session
$options = array(CURLOPT_USERAGENT => 'Mozilla/12.0 (compatible; MSIE 6.0; Windows NT 5.1)',
CURLOPT_SSL_VERIFYPEER => FALSE,
CURLOPT_SSL_VERIFYHOST => 2,
CURLOPT_HEADER => TRUE,
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_COOKIEFILE => 'cookies.txt',
CURLOPT_COOKIEJAR => 'cookies.txt');
//Hit home page for session cookie
$options[CURLOPT_URL] = $pages['home'];
curl_setopt_array($ch, $options);
//curl_exec($ch);
//Login
$options[CURLOPT_URL] = $pages['login'];
$options[CURLOPT_POST] = TRUE;
$options[CURLOPT_POSTFIELDS] = 'username=xxxxx#gmail.com&password=xxxxxxx';
$options[CURLOPT_FOLLOWLOCATION] = false;
curl_setopt_array($ch, $options);
curl_exec($ch);
//Hit data page
$options[CURLOPT_URL] = $pages['data'];
curl_setopt_array($ch, $options);
$data = curl_exec($ch);
//Output data
echo $data;
//Close curl session
curl_close($ch);
If you are looking for data from 'https://merchant.channeladvisor.com/AM/MyInventory/View_Inventory.aspx' Why not use the API instead?
http://developer.channeladvisor.com/display/cadn/Inventory+Service
If exporting your inventory information is all you're trying to do, ChannelAdvisor has its own UI-available inventory export service. At very least, you can automate your code to kick that off and download the exported csv or tab delim file: http://ssc.channeladvisor.com/howto/exporting-inventory
I think you should handle such update from Channel Advisor and so, follow the redirect.
Since I don't think this is a common way to login to Channel Advisor using curl, you will always need to update your code on each Channel Advisor update. It remember me when we can only use curl to retrieve Google Analytics data: every time they update the login system, you have to rewrite your own curl login method - boring.
You can read this answer about following a Header: Location with curl. Basically:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$a = curl_exec($ch);
if(preg_match('#Location: (.*)#', $a, $r))
$l = trim($r[1]);

How to get redirecting url link with php from bit.ly

I'm trying to get url links to those bit.ly redirects. I've tried to open bit.ly links with file_get_contents but it already gets content from redirected site, but how to get its url?
I was unaware of the bit.ly API, here is the raw way to do it:
$context = array
(
'http' => array
(
'method' => 'GET',
'max_redirects' => 1,
),
);
#file_get_contents('http://bit.ly/cmUTtb', null, stream_context_create($context));
echo 'Redirect to: ' . str_replace('Location: ', '', $http_response_header[6]);
You can query bit.ly's API (documentation) for the long URL. You will need your username and API key (which can be found on your account page).
$endpoint = 'http://api.bit.ly/v3/expand?';
$params = array(
'shortUrl' => 'http://bit.ly/aUmUDq',
'login' => 'your_bitly_username',
'apiKey' => 'your_api_key',
'format' => 'txt'
);
$api_url = $endpoint . http_build_query($params);
echo file_get_contents($api_url);
Use curl, which will not follow redirects by default.
see https://stackoverflow.com/a/41680608/7426396
I implemented to get a each line of a plain text file, with one shortened url per line, the according redirect url:
<?php
// input: textfile with one bitly shortened url per line
$plain_urls = file_get_contents('in.txt');
$bitly_urls = explode("\r\n", $plain_urls);
// output: where should we write
$w_out = fopen("out.csv", "a+") or die("Unable to open file!");
foreach($bitly_urls as $bitly_url) {
$c = curl_init($bitly_url);
curl_setopt($c, CURLOPT_USERAGENT, 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36');
curl_setopt($c, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($c, CURLOPT_HEADER, 1);
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_CONNECTTIMEOUT, 20);
// curl_setopt($c, CURLOPT_PROXY, 'localhost:9150');
// curl_setopt($c, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);
$r = curl_exec($c);
// get the redirect url:
$redirect_url = curl_getinfo($c)['redirect_url'];
// write output as csv
$out = '"'.$bitly_url.'";"'.$redirect_url.'"'."\n";
fwrite($w_out, $out);
}
fclose($w_out);
Have fun and enjoy!
pw

Trying to login to site with PHP & cURL?

I've never done something like this before...I'm trying to log into swagbucks.com and get retrieve some information, but it's not working. Can someone tell me what's wrong with my script?
<?php
$pages = array('home' =>
'http://swagbucks.com/?cmd=home',
'login' =>
'http://swagbucks.com/?cmd=sb-login&from=/?cmd=home',
'schedule' =>
'http://swagbucks.com/?cmd=sb-acct-account&display=2');
$ch = curl_init();
//Set options for curl session
$options = array(CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; `rv:1.9.2) Gecko/20100115 Firefox/3.6',`
CURLOPT_HEADER => TRUE,
//CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_COOKIEFILE => 'cookie.txt',
CURLOPT_COOKIEJAR => 'cookies.txt');
//Hit home page for session cookie
$options[CURLOPT_URL] = $pages['home'];
curl_setopt_array($ch, $options);
curl_exec($ch);
//Login
$options[CURLOPT_URL] = $pages['login'];
$options[CURLOPT_POST] = TRUE;
$options[CURLOPT_POSTFIELDS] = 'emailAddress=lala#yahoo.com&pswd=jblake&persist=on';
$options[CURLOPT_FOLLOWLOCATION] = FALSE;
curl_setopt_array($ch, $options);
curl_exec($ch);
//Hit schedule page
$options[CURLOPT_URL] = $pages['schedule'];
curl_setopt_array($ch, $options);
$schedule = curl_exec($ch);
//Output schedule
echo $schedule;
//Close curl session
curl_close($ch);
?>
But it still doesn't log me in. What's wrong?
try to echo each request to see if something went wrong.
(enabling CURLOPT_RETURNTRANSFER)
I suggest you to use
curl_setopt($ch, CURLOPT_COOKIEFILE, '/dev/null');
This way cookies are stored internally in-memory without the need of a separated file.
It works for me with "persist=1" , not "persist=on" :
$options[CURLOPT_POSTFIELDS] = 'emailAddress=lala#yahoo.com&pswd=jblake&persist=on'; // doesn't work
$options[CURLOPT_POSTFIELDS] = 'emailAddress=lala#yahoo.com&pswd=jblake&persist=1'; // works
$options[CURLOPT_POSTFIELDS] = 'emailAddress=lala#yahoo.com&pswd=jblake'; // also works

Categories