how to search data from other website using curl - php

Hi how can i search data from other website using curl and php. i want to search imei number from this website https://www.example.com/xxx
this is what i have tried so far
$imei = '013887009861498';
$cookie_file_path = "cookies/cookiejar.txt";
$fp = fopen("$cookie_file_path","w") or die("<BR><B>Unable to open cookie file $mycookiefile for write!<BR>");
fclose($fp);
$url="https://example.com/xxx";
$agent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$imei);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
$result = curl_exec ($ch);
echo $result ;

(this is not a full answer, but too long to be a comment. i can't be arsed to figure out all the small details for you)
there are several different problems here, the first is how to do a POST request with php/curl, of which you can find an example here.
another problem, is how to parse HTML in PHP, of which there are several options listed here. (i highly recommend the DOMDocument & DOMXPath combo)
another problem, is how to get past CAPTCHA challenges in PHP, 1 solution is to use the deathbycaptcha API (which is a paid service, by the way), you can find an example of that here.
another problem is that they're using 3 different CSRF-like tokens, called __VIEWSTATE, __EVENTVALIDATION, and hdnCaptchaInstance, all of which must be parsed out and submitted with the captcha answer. also you need to handle cookies, as the CSRF tokens and captcha is tied to your cookie session (luckily you can let curl handle cookies automatically with CURLOPT_COOKIEFILE )

Related

Logging in to a webpage through CURL, differently

There's a webpage that I need to log in to. I used CURL with post to login, but it's not enough. When you log in from the website the post also includes a string that is always changing. Is threre a way to get over that?
I use this:
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; he-IL; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6");
curl_setopt ($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_POST, TRUE);
$post = "username=$username&password=$password";
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
$result = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
It's like I need the code to actually go to the webpage and fill the form regularly.
I looked everywhere but all I could find was using post data.
Thanks!
To pass that you need to visit the page with the form, grab the field and then use it in POST request when you submit the form.
I suggest you visit the form page not only for that, but also for the following reasons (some of which can be used to figure people using automatic requests):
You recieve cookies
You don't fake referrer, you actually visited the page
You might want to check form fields to see if there's any new ones added since you wrote the script. That could be the case if form setup changes and you might want to adapt to that, if you don't then your script might stop working one day

PHP cURL cookies blocking

<?php
$ebay_user_id = "id"; // Please set your Ebay ID
$ebay_user_password = "password"; // Please set your Ebay Password
$cookie_file_path = dirname(__FILE__).'/cookie.txt'; // Please set your Cookie File path
$LOGINURL = "http://signin.ebay.com/aw-cgi/eBayISAPI.dll?SignIn";
$agent = "Mozilla/4.0 (compatible;)";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$LOGINURL);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
$result = curl_exec ($ch);
curl_close ($ch);
$LOGINURL = "http://signin.ebay.com/aw-cgi/eBayISAPI.dll";
$POSTFIELDS = 'MfcISAPICommand=SignInWelcome&siteid=0&co_partnerId=2&UsingSSL=0&ru=&pp=&pa1=&pa2=&pa3=&i1=-1&pageType=-1&userid='. $ebay_user_id .'&pass='. $ebay_user_password;
$reffer = "http://signin.ebay.com/aw-cgi/eBayISAPI.dll?SignIn";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$LOGINURL);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$POSTFIELDS);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_REFERER, $reffer);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
$result = curl_exec ($ch);
curl_close ($ch);
print $result; ?>
I'm really new player on cURL...
I have this code now using in login into ebay.
The problem for now is the cookies it told me that it was blocked by something.
The message it shows: Your web browser settings are blocking cookies.
I use firefox for test and tried other browser also got the same issues.
I have confirmed that my browser setting are accepted for the cookies access.
Also, I have checked there has conntent inside the cookies.txt file, so that mean the cookies.txt can be access correctly.
So....What is the problem for this issue? The code I used are correct?
Thanks everyone for help.
Try modifying the agent to something similar;
'Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20100101 Firefox/15.0.1'
Edit: actually I believe the problem is you need to query the signin page first,
first visit "http://signin.ebay.com/aw-cgi/eBayISAPI.dll?SignIn"
this will set the cookies, then sign in as you have.
you can try it in a browser, navigate to the eBay sign in page,
clear your cookies and then signin.
You will get the browser not supporting cookies error.
You need to understand something and that is that doing a HTTP request with curl through php has nothing to do with your browser. The website you are accessing doesn't care what browser you use to run the php script. The actual request is done by your server, not by your browser.
On the other hand, if eBay engineers are smart they'd block this, you probably aren's supposed to do things like this, that's what the Ebay API's are for.
And a little tip, use a HTTP Client library, doing things like this in plain cURL is a pita and gives some very bad and unreadable code.
Check https://github.com/guzzle/guzzle for example.

Automatic login to Facebook with cron/cURL

I've seen so many solutions for this but haven't been able to implement any of them sucessfully. I have created an App in Facebook and can successfully use FQL to retrieve data. I will be pulling in event information from pages and groups that I am part of. I will only parse the event information of those who explicitly register with my App/website.
I'm using the Facebook PHP SDK. The issue is that I want to create a cron task to retrieve event information periodically, but I don't know how to allow the cron task to log in automatically.
I've seen that there are real-time updates, but as far as I know, they don't show events.
I tried to use cURL but it just brought up a blank screen. The code was:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://login.facebook.com/login.php?');
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'facebook_cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'facebook_cookies.txt');
curl_setopt($ch, CURLOPT_USERAGENT,
"Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20100101 Firefox/12.0");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$email = 'xxxxxxxxxxx';
$pass = 'xxxxxxxxxxx';
curl_setopt($ch, CURLOPT_POSTFIELDS, 'email='.urlencode($email).'&
pass='.urlencode($pass).'&login=Login');
$result = curl_exec($ch);
echo $result;
curl_close($ch);
That was taken from another SO question: Remote login to facebook account
Maybe you don't need to log, app access_token can't solve your problem?

curl simple form, why session false

i tried many tutorials but all failed, i know for an experienced user it might be obvious, thx anyway.
there is the simple form:
https://www.shab.ch/shabforms/COMMON/application/applicationGrid.jsp?template=1&view=2&page=/COMMON/search/searchForm.jsp?MODE=SHAB
here is my script which returns only the empty form instead of my POST search:
(i used tamper-data to get the Post-variables, i also use https)
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://www.shab.ch/shabforms/COMMON/application/applicationGrid.jsp?template=1&view=2&page=/COMMON/search/searchForm.jsp?MODE=SHAB');
curl_setopt($ch, CURLOPT_POSTFIELDS,'KEYWORDS=&NOTICE_NR=&TIMESPAN=TODAY&STAT_TM_1=&STAT_TM_2=&SELTYPE=HR&TYPE_CD_AW=&TYPE_CD_AN=&TYPE_CD_BL=&TYPE_CD_VM=&TYPE_CD_HR=HR01&LEGAL_FORM_NR_HR=&FIRM_ID_HR=&HR_CANTON_AG=ON&HR_CANTON_BE=ON&TYPE_CD_IS=&TYPE_CD_KK=&YN_KK=&TYPE_CD_IP=&TYPE_CD_NA=&YN_NA=&TYPE_CD_SB=&YN_SB=&TYPE_CD_SR=&FIRM_NAME_TX_UP=&FIRM_CITY_TX_UP=&command=Recherchieren');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_REFERER,"https://www.shab.ch/shabforms/COMMON/application/applicationGrid.jsp?template=1&view=2&page=/COMMON/search/searchForm.jsp%3Fcategory%3DHR");
curl_setopt($ch, CURLOPT_COOKIEJAR, "my_cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "my_cookies.txt");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3");
$result = curl_exec($ch);
echo $result;
This has strangely been written into my_cookies.txt
www.shab.ch FALSE /shabforms FALSE 0 JSESSIONID E884A3B4187C68253CEEBCD58E7E934E
www.shab.ch FALSE / FALSE 1287673522 BC_HA_C30B29681466613B 131BDF
What is wrong? :)
UPDATE:
Ok, i got the error. it was related to the post-url. the script on the website seems to do the process by ajax,... without changing the url to send (i could not even find the correct url in tamper data!!).
Fortunately i could figure that out, its "shabforms/servlet/web/DocumentSearch".
Now it works, thx
I just ran this script and got a German website saved in $result.
Maybe your curl setup needs tweeking? Have you got it working with another site?
it was related to the post-url. the script on the website seems to do the process by ajax,... without changing the url to send (i could not even find the correct url in tamper data!!). Fortunately i could figure that out, its "shabforms/servlet/web/DocumentSearch".
Now it works, thx

php: Get url content (json) with cURL

I want to access https://graph.facebook.com/19165649929?fields=name (obviously it's also accessable with "http") with cURL to get the file's content, more specific: I need the "name" (it's json).
Since allow_url_fopen is disabled on my webserver, I can't use get_file_contents! So I tried it this way:
<?php
$page = 'http://graph.facebook.com/19165649929?fields=name';
$ch = curl_init();
//$useragent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1";
//curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_URL, $page);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
curl_close($ch);
?>
With that code I get a blank page! When I use another page, like http://www.google.com it works like a charm (I get the page's content). I guess facebook is checking something I don't know... What can it be? How can I make the code work? Thanks!
did you double post this here?
php: Get html source code with cURL
however in the thread above we found your problem beeing unable to resolve the host and this was the solution:
//$url = "https://graph.facebook.com/19165649929?fields=name";
$url = "https://66.220.146.224/19165649929?fields=name";
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: graph.facebook.com'));
$output = curl_exec($ch);
curl_close($ch);
Note that the Facebook Graph API requires authentication before you can view any of these pages.
You basically got two options for this. Either you login as an application (you've registered before) or as a user. See the api documentation to find out how this works.
My recommendation for you is to use the official PHP-SDK. You'll find it here. It does all the session and cURL magic for you and is very easy to use. Take the examples which are included in the package and start to experiment.
Good luck.

Categories