I've a tricky question on how to deal with consecutive curls in PHP. I have this incredibile data flow:
example.com/one.php post data via curl to another.com/two.php
another.com/two.php post data via curl to another.com/three.php
another.com/three.php responds me with XML (or JSON) back to another.com/two.php
another.php/two.php transforms the XML into a php array and then into a query string that i post via curl back to the origin example.com/one.php
It works. If you are wondering why i have this insane data flow it's due to the fact that another.com/three.php is an obfuscated file with Ioncube. I can't edit it but i have to add some checks before i can send data to him. Don't waste time trying to figure out how i can make it in an alternative way because there isn't one (trust me).
On example.com/one.php there's a form where users fill in data. When they press "Submit" they remain on this page while "silently" i make 1->2->3->4 to get their response (the $_POST of step 4) then can save it into example.com/log.txt. Again it works.
Now my question is: how can i display the $_POST response (which is the same i get in log.txt file) in example.com/one.php? I mean this is what i have in step 4.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, example.com/one.php);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $query_string);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$xml = curl_exec($ch);
$_POST arrives on example.com/one.php but of course users and this curl are on two different level. I tryed playing with file_get_contents(), sleep() and CURLOPT_RETURNTRANSFER with no success. What's the answer this this question? I would go for sessions or ob_start() but i'm not sure of it.
Related
So I am trying to retrieve only headers using cURL with the following:
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, true); // we want headers
curl_setopt ($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
$output = curl_exec($ch);
The problem is that while trying to get headers of large file the script uses all the memory. I would like to avoid getting also the body and I have tried to use:
curl_setopt($ch, CURLOPT_NOBODY, true);
The problem is that this issues a HEAD request instead of GET. And some website retrun an error when you request with HEAD.
Is there any way with curl to retrieve only header without doing HEAD request?
First, don't use CURLOPT_RETURNTRANSFER as that's the option that makes it keep the entire response in memory.
Then, two options:
A) use a write callback and make that abort the transfer as soon as the first byte of the body is returned. There's a write callback example in the PHP.net docs.
B) use CURLOPT_RANGE and ask for only the first byte to be retrieved, 0-0. This avoids the write callback but has the downside that not all HTTP servers and URLs will acknowledge it.
If you dont have to use cURL, you could use get_headers(). By default get_headers uses a GET request to fetch the headers. And you could also modify that request by using stream_context_set_default()
$headers = get_headers('http://example.com');
More Info: PHP: get_headers
i need to post raw data (XML) to an external site, the XML data is good to go, and is working with the below example...
I am using PHP / cURL
normally i do this by cURL, but in this instance, i need to redirect the client to the URL with the post data...
It works using the below code, but it won't redirect the user to the $URL, this method just prints the data while still being in the script on the same server.
eg:
Post from site1.com to site2.com, the below method keeps the client on site1, but needs to redirect the user to site2.com with the raw post data...
$ch = curl_init($URL);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_xml);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_AUTOREFERER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
$val = curl_exec($ch);
curl_close($ch);
i have searched all-over but cannot find a method where it functions as a "regular" form submit where the user is redirected to the URL which the data is posted to... but with RAW post data.
I am trying to use php curl function to log in to a https webpage "https://portal.opalonline.co.uk/Home/PortalCore/SignIn/SignIn.aspx"
but I have run out of ideas how I can post values to this particular page (username, password) and 'press 'sign in'.
$postfields = array('ctl00_MasterContentContentPane_Signin1_userID_txt'=>'email#address.com',
'ctl00_MasterContentContentPane_Signin1_password_txt'=>'somepassword123');
/* LOG IN TO TalkTalk ACCOUNT */
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://portal.opalonline.co.uk/Home/PortalCore/SignIn/SignIn.aspx?");
curl_setopt($ch, CURLOPT_HEADER, false);
// curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
// curl_setopt($ch, CURLOPT_COOKIEJAR, COOKIE_FILE);
// curl_setopt($ch, CURLOPT_COOKIE, COOKIE_FILE);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);
curl_setopt($ch, CURLOPT_POST, 1);
var_dump($ch);
$string_exec = curl_exec($ch);
var_dump($string_exec);
I can not even display the page with var_dump :( . Ideas / suggestions much appreciated
First, I don't think you can do the 'array' thing like that as that will make PHP/CURL create multipart formpost instead, and this is not such a form. Provide the data in "name=value&name2=value2" style.
Then, make sure you also submit all the hidden fields in the form. There are at least four of them. One of them is set by the HTML to a long value that you need to extract and set, and there is also some javascript magic that sets some of the others. You probably need to use your browser's networking tool to snoop on what exactly your browser sends to be able to mimic that perfectly.
The login page sets cookies and you probably need to pass those cookies on when you submit the login form. So you need to first fetch (GET) the login form page to get the cookies, then file the login POST.
With that fixed, you should be closer. If that isn't all that takes, then continue comparing the browser's request with what your request is sending and make sure they are as similar as possible.
Open the website in google chrome, open the console, to go the network tab.
Login to the website. You should see the request in the network tab. Do a right click on it, select "copy as cURL". It will give you a command line, that will help you understand what you need.
I would like to crab data from a website that uses an ajax request to load new data from the server into a DIV.
When I click on the button of the website, that will load new data into the website, I can see that the browser does only 1 POST request with the following post string:
AJAXREQUEST=_viewRoot&j_id376=j_id376&javax.faces.ViewState=j_id3&j_id376%3Aj_id382=j_id376%3Aj_id382&valueChanged=false&AJAX%3AEVENTS_COUNT=1&
When I do the above post request using php curl I don't get any useful data.
Does someone know how to crab data for this kind of request?
UPDATE1:
This is what I use in php:
$ch = curl_init ('http://www.website.com');
$post_string = 'AJAXREQUEST=_viewRoot&j_id376=j_id376&javax.faces.ViewState=j_id3&j_id376%3Aj_id382=j_id376%3Aj_id382&valueChanged=false&AJAX%3AEVENTS_COUNT=1&';
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_string);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt ($ch, CURLOPT_COOKIEFILE, $ckfile);
$output = curl_exec ($ch);
I don't get any results, also no errors or messages.
Your problem probably isn't with your PHP code, its more likely with what you are actually sending to the server. I'm assuming you listed website.com as a place holder for whatever service you are trying to interact with, but since you haven't listed any of the information as to where your sending the request or what your getting back I'm assuming that what your posting to the server is simply being ignored because what your sending is invalid, or incomplete, or requires further POST/GET requests. Another possibility is that your attempting to POST to a service that requires an authenticated session (the POST variables you listed could include some sort of token to identify the session) which you have not established.
I would recommend that you first test your code on a simpler "controlled test case". Setup a basic web form that returns true or something when you POST a value to it. Test your code with the simpler case first to make sure your POST code works.
Then using a debugging tool such as LiveHTTPHeaders or Firebug record the entire POST/GET request interaction with the server. It might be a good idea to first try to "replay" this interaction with a debugging tool to prove that your methodology works. Then once you know exactly what you need to do from a high level, repeat this process in your PHP code.
There is not much other advice anyone can give you with the information you have given us.
I'm using a web-service from a provider who is being a little too helpful in anticipating my needs. They have given me a HTML snippet to paste on my website, for users to click on to trigger their services. I'd prefer to script this process, so I've got a php script which posts a cURL request to the same url, as appropriate. However, this provider is keeping tabs on my session, and interprets each new request as an update of the first one, rather than each being a unique request.
I've contacted the provider regarding my issue, and they've gone so far as to inform me that their system is working as intended, and that it's impossible for me to avoid using the same ASP.NET session for each subsequent cURL request. While my favored option would be to switch to a different vendor, that doesn't appear to be an option right now. Is there a reliable way to get a new ASP.NET session with each cURL request?
I've tried the following set of CURLOPT's, to no avail:
//initialize curl
$ch = curl_init($url);
//build a string out of the post_vars
$post_str = http_build_query($post_vars);
//set the necessary curl options
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FAILONERROR, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_str);
curl_setopt($ch, CURLOPT_FAILONERROR, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIESESSION, 1);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, 1);
curl_setopt($ch, CURLOPT_FORBID_REUSE, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "UZ_".uniqid());
curl_setopt($ch, CURLOPT_REFERER, CURRENT_SITE_URL."index.php?newsession=".uniqid());
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Pragma: no-cache", "Cache-Control: no-cache"));
//execute the call to the backend script, retrieve the results
$xmlstr = curl_exec($ch);
If cURL isn't helping much, why not try other methods to call the services from your script, like php's file() function, or file_get_contents().
If you see do not see any difference at all, then the service provider might be using your ip to track your requests. Try using some proxy for a test.
Normal Asp.net session is tracked by a cookie called ASP.NET_SessionId. This cookie is sent within the response to your first request. So as long as your curl requests don't send back this asp.net cookie, each of your requests will have no connection to each other. Use the curl -c option to see what cookies are flying in-between you and them. Overriding this cookie with a cookie file should work if you confirm that it is normal asp.net session being used here.
It is quite poor for a service to use session (http has much cleaner ways of maintaining state which ReST exploits) so I wouldn't completely rule out the vendor switch option.
Well given the options you are using, it seems you have covered your basics. Can you find out how their sessions are setup?
If you know how they setup a session, IE what they use (if it is IP or what not) and then you can figure out a work around. Another option is trying to set the cookies in a different cookie file:
CURLOPT_COOKIEFILE - The name of the file containing the cookie data. The cookie file can be in Netscape format, or just plain HTTP-style headers dumped into a file.
But if all they do is check cookies your current code should work. If you can figure out what the cookie's name is, you can pass a custom cookie that is blank with the request to see if that works. But if you can get information out of them on how their session's work, that would be best.
use these two line to handle the session:
curl_setopt($ch, CURLOPT_COOKIEJAR, "path/to/cookies.txt"); // cookies.txt should be writable
curl_setopt($ch, CURLOPT_COOKIEFILE, "path/to/cookies.txt");