I have the following code to login into an external site application (asp.net app) from a local site login form (written in php):
<?php
$curl_connection = curl_init('www.external.com/login.aspx');
curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl_connection, CURLOPT_USERAGENT,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
// Post data array
$post_data['LoginControl$UserName'] = 'ExampleUName';
$post_data['LoginControl$Password'] = 'ExamplePWord';
// Add form fields into an array to get ready to post
foreach ($post_data as $key => $value)
{
$post_items[] = $key . '=' . $value;
}
$post_string = implode ('&', $post_items);
// Tell cURL which string to post
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $post_string);
// Execute and post
$result = curl_exec($curl_connection);
?>
I get directed to the login form of the external site instead of being directed to the application logged in. I think the problem is that I need to pass the viewstate values through, but i'm not sure how to go about doing that?
I don't have control over the external application. But we want users to be able to login to the application through our website, to maintain branding etc.
I've posted a couple of other threads recently about the use of php cURL, but I'm at the stage now where I think the viewstate is the problem ...
Thanks, Mark.
This seems to be a real problem when trying to scrape the asp.net pages.
The pages contain a hidden field named "__VIEWSTATE" which contains a base64 encoded set of va;ues containing some or all of the page state when the page was sent. It usually also contains the SHA1 of the viewstate.
What this means is that your post must contain everything in the _VIEWSTATE or it will fail.
I have been able to post a simple login page that has only 2 fields but not a more complex page in which the author has chosen to put the entire page state in the viewstate.
As yet I have not been able to come up with a solution.
Change:
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
To:
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, false);
You also need to set up a cookie file, take a look at CURLOPT_COOKIEFILE
CURLOPT_COOKIEFILE:
The name of the file containing the cookie data. The cookie file can be in Netscape format, or just plain HTTP-style headers dumped into a file.
CURLOPT_COOKIE:
The contents of the "Cookie: " header to be used in the HTTP request. Note that multiple cookies are separated with a semicolon followed by a space (e.g., "fruit=apple; colour=red")
CURLOPT_COOKIEJAR:
he name of a file to save all internal cookies to when the connection closes.
#see http://www.php.net/manual/en/function.curl-setopt.php
curl_setopt($curl_connection, CURLOPT_COOKIEFILE, 'cookiefile.txt');
curl_setopt($curl_connection, CURLOPT_COOKIEJAR, 'cookiefile.txt');
Don't expect it to work without encoding the __VIEWSTATE string in php using
rawurlencode($viewstate);
I've encountered the same problem recently, so I just leave my way to go about it here, in case someone else stumbles on this thread looking for an answer too.
I solved this by preceding every POST request with a GET request to the same url, and scraping all the input fields into an array of key-value pairs out of the response from that GET. Then I replaced some values in that array (login field values, for example), and sent the whole thing back in the subsequent POST. This way my POST request contained all the valid __VIEWSTATE, __EVENTVALIDATOR and yada-yada data generated for that particular url too.
This way the site allowed me to log in and visit subdomains normally.
Related
I`m using cuRL to get some data from remote server... The response is in JSON format..
This is my code:
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER , 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
curl_setopt($ch, CURLOPT_URL, 'http://www.myaddress.com/mypage.php');
curl_setopt($ch, CURLOPT_POSTFIELDS, array("id" => $id));
$return = curl_exec($ch);
curl_close($ch);
If I access the link in the browser the page load OK, but if I access through the cuRL return a 404 error...
I can guess a few things that it can be checked from the server side, to show the error.
1) As it is stated in other answers, be sure to set all the necessary headers, you can check them e.g. by firebug, as it is shown in here,
or you can get the headers by php get_headers function.
to set it use
curl_setopt($ch, CURLOPT_HTTPHEADER, array("HeaderName: HeaderValue"));
2) When you open a page in the browser(excluding form submit with post method) it makes a get request, instead of post, so if in the server side it is checked $_GET, then your post request will not be considered.
3) If you sure that it should be a post request(say, it is a form submit), then the following can be a problem: some forms can have hidden fields, that again are being checked in the server, and if they are not set, error can be returned. So, you should look at the source code of the form and add them(if there are any) to your post parameters.
4) if you are submitting a form, be sure to set the submit button with its name and value as well, because similar to hidden fields, this can be checked as well.
5) Cookies can be a problem as well, because by default browser has it , and curl does not. To to able to set and read cookies use this code
// set cookie
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
// use cookie
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
here, $cookie_file path to the cookies file. Do not know in linux or mac, but in windows be sure to use absolute path to the cookie file.
6) Also, you can set the referer by
curl_setopt($ch, CURLOPT_REFERER, 'http://www.myaddress.com/mypage.php');
EDIT: In case of ajax request you might want to add a header X-Requested-With with value as XMLHttpRequest
It's possible the server check the HTTP Header, it's the case in the majority of case.
So add the same HTTP Header of your browser, verify with Firebug :
curl_setopt($ch, CURLOPT_HTTPHEADER, array('SomeName: SomeValue'));
Probably there is something else the browser is sending your cURL code is not. You can use any of the tools other folks have suggested, Firebug, Wireshark, Fiddler, etc, etc.
What you need to do is add missing pieces to your request to match the browser as closely as possible in the cURL request until the remote page responds with a 200.
I notice you're doing a POST. In many cases what happens with your browser is you visit a page with a GET request. A session is initialized on the remote site and a cookie is saved in your browser with the session id.
This cookie then needs to be supplied along with subsequent POST requests. PHP cURL has many options to support cookies. There may be other requirements such as CSRF tokens and so forth.
Again, reverse-engineering is the key.
I want to send some value to a asp.net form for sending sms by PHP but it doesnt send any value , i think the problem is from asp.net form element name for example a name of text field is "ctl00$ContentPlaceHolder1$txtnum"
my code :
<?
$text=$_GET['nome'];
$strCookie = "ASP.NET_SessionId=uwf4xv55la1ojtjsm0sbnynb;vipcard=210927C5B78543C4A0645F15A561D171F5D3FEADF320B173E380799EB6B5086F7472F9910C39496F84C6EF7C86356E5E1BB8E22F9E93C59C347FC7301D954CA669F22E6D4A3655136BD0929FE4AE1A36416DD2FAC65A6CF4EBC3DC5D7907B455C02D1762D11FFEA07C4DA8260371EA4dfgdfA6CEC59A7";
// set URL and other appropriate options
$url = 'http://www.sms.com/seller/HomePage.aspx';
//set POST variables
$fields = array(
'ctl00$ContentPlaceHolder1$txtnum' => '092983940***',
'ctl00$ContentPlaceHolder1$dds1' => '30001607' ,
'ctl00$ContentPlaceHolder1$t3' => "hello how are u",
'ctl00$ContentPlaceHolder1$Button2' => 'submit'
);
// set user agent
$useragent = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.5) Gecko/20060719 Firefox/1.5.0.5';
//open connection
$ch = curl_init();
//set the url, POST data, UserAgent, Timeout, etc.
curl_setopt( $ch, CURLOPT_COOKIE, $strCookie );
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_POSTFIELDS,$fields);
//curl_setopt($ch, CURLOPT_CONNECTTIMEOUT_MS, 500000); //time out of 0.5 seconds.
//execute post
curl_exec($ch);
?>
is this possible to send value to asp.net form ?
is there any other idea ?
Thanks
It is possible to send data directly to an asp.net web page, however it may ignore the request if it isn't designed to process direct posts.
In reference to the form names you are seeing asp.net builds control names based on their location in the page hierarchy.
The name ctl00$ContentPlaceHolder1$txtnum implies that the field named txtnum is placed inside a contentplaceholder.
On the server side the asp.net form collection will contain a form post variable named txtnum, and not the longer version which is there to ensure uniqueness within the page.
All this being said if the page is not setup to process direct post submissions then it will ignore the data being sent.
ASP.net pages typically will work using post back events which means the processing of the data will be placed inside an onclick event for that button control, the event is raised and handled within the page lifecycle.
If you need to post directly to the ASP.net page I would check with the company hosting the page if they have a page or service that you can post directly to.
I would like to crab data from a website that uses an ajax request to load new data from the server into a DIV.
When I click on the button of the website, that will load new data into the website, I can see that the browser does only 1 POST request with the following post string:
AJAXREQUEST=_viewRoot&j_id376=j_id376&javax.faces.ViewState=j_id3&j_id376%3Aj_id382=j_id376%3Aj_id382&valueChanged=false&AJAX%3AEVENTS_COUNT=1&
When I do the above post request using php curl I don't get any useful data.
Does someone know how to crab data for this kind of request?
UPDATE1:
This is what I use in php:
$ch = curl_init ('http://www.website.com');
$post_string = 'AJAXREQUEST=_viewRoot&j_id376=j_id376&javax.faces.ViewState=j_id3&j_id376%3Aj_id382=j_id376%3Aj_id382&valueChanged=false&AJAX%3AEVENTS_COUNT=1&';
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_string);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt ($ch, CURLOPT_COOKIEFILE, $ckfile);
$output = curl_exec ($ch);
I don't get any results, also no errors or messages.
Your problem probably isn't with your PHP code, its more likely with what you are actually sending to the server. I'm assuming you listed website.com as a place holder for whatever service you are trying to interact with, but since you haven't listed any of the information as to where your sending the request or what your getting back I'm assuming that what your posting to the server is simply being ignored because what your sending is invalid, or incomplete, or requires further POST/GET requests. Another possibility is that your attempting to POST to a service that requires an authenticated session (the POST variables you listed could include some sort of token to identify the session) which you have not established.
I would recommend that you first test your code on a simpler "controlled test case". Setup a basic web form that returns true or something when you POST a value to it. Test your code with the simpler case first to make sure your POST code works.
Then using a debugging tool such as LiveHTTPHeaders or Firebug record the entire POST/GET request interaction with the server. It might be a good idea to first try to "replay" this interaction with a debugging tool to prove that your methodology works. Then once you know exactly what you need to do from a high level, repeat this process in your PHP code.
There is not much other advice anyone can give you with the information you have given us.
Im writing a form to post data off to paypal, and this works fine, i create the form with hidden fields and then have a submit button that submits everything to paypal.
However, when the user clicks that button, there is more that i want to do, for example change their cart status in the database. So, i want to be able to execute some code when they click submit and then post the data to paypal.
I dont want to use javascript for this.
My method at the moment is using cURL, the form posts back to the current page, i then check for $_POST data, do my other commands like updating the status of the cart, and then create a curl command, and post the form data to paypal. Now, it gets posted successfully, but the browser doesnt go off to paypal.
At first i was just retirieving the result in a string and then echoing it out and i could see that the post was successful, then i set CURLOPT_FOLLOWLOCATION to 1 assuming this would let the browser go off to paypal but it doesnt, what it seems to do is grab some stuff from paypal and put it on my page.
Is using cURL the right thing for this? and if so, how do i get round this problem? I want to stay away from javascript for this as only users with javascript enabled would have their cart updated.
The code im using for curl is below, the post_data is an array i created and then read key-value pairs into post_string.
//Set cURL Options
curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl_connection, CURLOPT_USERAGENT,
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, false);
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl_connection, CURLOPT_MAXREDIRS, 20);
//Set Data to be Posted
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $post_string);
//Execute Request
curl_exec($curl_connection);
Perhaps try to combine CURL with a header statement
<?php
header ("Location: https://www.paypal.com/");
This might work depending if the browser need to pass any state (session) info to paypal. This session info are gathered in your CURL call, but there is no way to pass that on to the browser unfortunately. So if you need to maintain the state, then this will not work.
Curl operates on the server side. It won't affect your browser. CURLOPT_FOLLOWLOCATION simply instructs CURL to obey when for instance paypal sends it a header like the one mentioned above. The only way to instruct the browser to do something is to respond with an HTTP header like above or with some kind of meta tag in your HTML response e.g.
<META HTTP-EQUIV="refresh" CONTENT="10;URL=https://www.paypal.com/">
As an alternative, does paypal not provide a redirect function that will, when the transaction is finished, redirect back to your server. If so then just change the card status during this callback and the skip the whole curl process?
After receiving the form and doing what you need to do you should present another form with only a button (all other data in hidden fields) for the user to click and finally head off to paypal.
I'm trying to write a simple PHP script which automatically sets up new etherpads (see http://etherpad.com/).
They don't have an API (yet) for creating new pads so I'm trying to figure if I can do things another way.
After playing around some, I found that if you append a random string to etherpad.com to a not-yet-created pad, it'll come back with a form asking if you want to create a new etherpad at that address. If you submit that form, a new pad will be created at that URL.
My thought then was I could just create a PHP script using CURL that would duplicate that form and trick etherpad into creating a new pad at whatever URL I give it. I wrote the script but so far I can't get it working. Can someone tell me what I'm doing wrong?
First, here's the HTML form on the etherpad creation page:
`
<p><tt id="padurl">http://etherpad.com/lsdjfsljfa-fdj-lsdf</tt></p>
<br/>
<p>There is no EtherPad document here. Would you like to create one?</p>
<input type="hidden" value="lsdjfsljfa-fdj-lsdf" name="padId"/>
<input type="submit" value="Create Pad" id="createPad"/>
`
Then here's my code which tries to submit the form using CURL
$ch = curl_init();
//set POST variables
$url = "http://etherpad.com/ep/pad/create?padId=ldjfal-djfa-ldkfjal";
$fields = array(
'padId'=>urlencode("ldjfal-djfa-ldkfjal"),
);
$useragent="Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)";
// set user agent
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
//url-ify the data for the POST
foreach($fields as $key=>$value) { $fields_string .= $key.'='.$value; }
print_r($fields_string);
//open connection
$ch = curl_init();
//set the url, number of POST vars, POST data
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST,count($fields));
curl_setopt($ch,CURLOPT_POSTFIELDS,$fields_string);
//execute post
$result = curl_exec($ch);
print_r($result);
//close connection
curl_close($ch);
When I run the script, PHP reports back that everything executed correctly but etherpad doesn't create my pad. Any clues what's going on?
I have not investigated this specific site but I guess there are some important headers which are missing. Here is a very general approach that is applicable for nearly any website:
Use a network sniffer such as Wireshark to capture all connectons. Then compare the sent POST fields with yours.
An even easier way is to use Netcat. Just save the page to disk, change the form-URL to http://localhost:3333/ and run
$ nc -l -p 3333
Now open the local HTML file and fill in the fields appropriately. Immediately you will see all headers that would have been transmitted to the host.
(There are also extensions for Mozilla Firefox but in general they just slow down the browser without providing much benefit.)
Also read what I have posted on To auto fill a text area using php curl as it might help you with your realization in PHP.
By the way, you are sending the parameter "padId" via GET and POST. That is not necessary. Check what the Etherpad-form actually uses and stick with it.
My guess is that you're missing the cookies and/or the referrer. It may be checking the referrer to ensure people aren't creating pads without confirmation.
Wireshark will help, but add that to your curl and see if it works.
Here's the answer a friend helped me come up with:
They're apparently doing some cookie
validation, that's why your script
isn't working. You can find this out
by loading the new pad creation prompt
page, clearing your cookies, and then
reloading the page. It won't work.
Tricky, but effective for most casual
bots.
Here's a script that gets around the
limitation. Just insert your desired
$padId and away you go.
<?php
$padId = 'asdfjklsjfgadslkjflskj';
$ch = curl_init();
# for debugging
curl_setopt($ch, CURLOPT_HEADER, true);
# parse cookies and follow all redirects
curl_setopt($ch, CURLOPT_COOKIEFILE, '/dev/null');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
# first, post to get a cookie
curl_setopt($ch, CURLOPT_URL, 'http://etherpad.com/' . urlencode
($padId));
$result = curl_exec($ch);
echo $result;
# next, post to actually create the etherpad
curl_setopt($ch, CURLOPT_URL, 'http://etherpad.com/ep/pad/create');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, 'padId=' . urlencode($padId));
$result = curl_exec($ch);
echo $result;
curl_close($ch);
To create a file directly from HTML or TEXT
Use the setText or setHTML API endpoint. http://etherpad.org/doc/v1.5.0/#index_sethtml_padid_html
To easily do this use the Etherpad PHP Client https://github.com/TomNomNom/etherpad-lite-client
To post from a file
This feature is provided by an Etherpad plugin. To enable it...
Install the Etherpad ep_post_data plugin by typing npm install ep_post_data on your Etherpad instance.
At your client machine CLI type: curl -X POST -d #yourfile.here http://youretherpad/post
Replace yourfile.here with your file
Replace the url with the Etherpad instance you want to work to.
Source: http://blog.etherpad.org/2014/12/17/post-to-etherpad-with-this-simple-plugin/