I try to program a webboot using PHP/CURL, but I face a problem in handling a specific page that it's loading some contents dynamically !! .. to explain more :
when I try to download the page using PHP/CURL, I do not get some contents ! then I discovered that this contents are loaded after page is loaded. and this is why CURL does not handle these missed contents.
can any one help me !
my sample code is :
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_REFERER, $reffer);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, $redirect);
curl_setopt($ch, CURLOPT_COOKIEFILE, ABSOLUTE_PATH."Cookies/cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEJAR, ABSOLUTE_PATH."Cookies/cookies.txt");
$result = curl_exec($ch);
What URL are you trying to load? It could be that the page you're requesting has one or more AJAX requests that load content in after the fact. I don't think that cURL can accomodate runtime-loaded information via AJAX or other XHR request.
You might want to look at something like PhantomJS, which is a headless WebKit browser which will execute the page fully and return the dynamically assembled DOM.
Because the page uses javascript to load the content, you are not going to be able to do this via cURL. Check out this page for more information on the problem: http://googlewebmastercentral.blogspot.com/2007/11/spiders-view-of-web-20.html
Related
Currently I have page say page1.php. Now in a certain event, I just want to run another link say http://example.com without actually refreshing this page. The link is a kind of script which updates my database. I tried using shell_exec('php '.$url); where $url='http://example.com' however it showed me an error that could not open file so I suppose shell_exec works only for internal files present on the server. Is there a way to do this directly or I have to go with AJAX? Thanks in advance
Try using curl to send the request to the server with php.
$url = 'http://example.com';
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_NOBODY, TRUE);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_exec($ch);
curl_close($ch);
Alternatively you could try file_get_contents
file_get_contents('http://example.com');
I would do this front-end and I would use JSONP: much clean and safer IMHO.
I am trying to use php curl function to log in to a https webpage "https://portal.opalonline.co.uk/Home/PortalCore/SignIn/SignIn.aspx"
but I have run out of ideas how I can post values to this particular page (username, password) and 'press 'sign in'.
$postfields = array('ctl00_MasterContentContentPane_Signin1_userID_txt'=>'email#address.com',
'ctl00_MasterContentContentPane_Signin1_password_txt'=>'somepassword123');
/* LOG IN TO TalkTalk ACCOUNT */
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://portal.opalonline.co.uk/Home/PortalCore/SignIn/SignIn.aspx?");
curl_setopt($ch, CURLOPT_HEADER, false);
// curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
// curl_setopt($ch, CURLOPT_COOKIEJAR, COOKIE_FILE);
// curl_setopt($ch, CURLOPT_COOKIE, COOKIE_FILE);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postfields);
curl_setopt($ch, CURLOPT_POST, 1);
var_dump($ch);
$string_exec = curl_exec($ch);
var_dump($string_exec);
I can not even display the page with var_dump :( . Ideas / suggestions much appreciated
First, I don't think you can do the 'array' thing like that as that will make PHP/CURL create multipart formpost instead, and this is not such a form. Provide the data in "name=value&name2=value2" style.
Then, make sure you also submit all the hidden fields in the form. There are at least four of them. One of them is set by the HTML to a long value that you need to extract and set, and there is also some javascript magic that sets some of the others. You probably need to use your browser's networking tool to snoop on what exactly your browser sends to be able to mimic that perfectly.
The login page sets cookies and you probably need to pass those cookies on when you submit the login form. So you need to first fetch (GET) the login form page to get the cookies, then file the login POST.
With that fixed, you should be closer. If that isn't all that takes, then continue comparing the browser's request with what your request is sending and make sure they are as similar as possible.
Open the website in google chrome, open the console, to go the network tab.
Login to the website. You should see the request in the network tab. Do a right click on it, select "copy as cURL". It will give you a command line, that will help you understand what you need.
How does one download a file from a web page without a direct path to the file. For example a URL with GET information instead of the path. The code below seems to be downloading the actual page html instead of the file...
Not sure what I'm doing wrong. I also would like to augment this to also perform on sites that require logins but I think I would just have to add
curl_setopt($ch, CURLOPT_USERPWD, "$username:$password")
to the code?
$output_filename = "advanced.exe";
$host = "http://download.cnet.com/Advanced-SystemCare-Free/3001-2086_4-10407614.html?hlndr=1";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $host);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_AUTOREFERER, false);
curl_setopt($ch, CURLOPT_REFERER, "http://download.cnet.com");
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_setopt($ch, CURLOPT_HEADER, 0);
$result = curl_exec($ch);
curl_close($ch);
$fp = fopen($output_filename, 'w');
fwrite($fp, $result);
fclose($fp);
The link you have there isn't the actual link to the file, only the page that initiates the download. By the looks of it, the page uses JavaScript to trigger the download, so you would want to dig through their code to find out exactly how they do it. Then you can find the real URL to the file.
A simple way, if you are doing this only for one file, would be to download the file in your browser, and then access the URL it used from the browser's download manager. (In Firefox, for example, right click the file and choose "Copy Download Link")
I also would like to augment this to also perform on sites that require logins but I think I would just have to add ...
That would work only for HTTP based authentication. If the site uses a traditional login form, this will not work. You'd have to submit several, sequential HTTP requests via CURL, using cookies to store the session state.
I have an application that includes a file (FileX.php) which under certain conditions, will echo an iFrame to the screen which loads a tracking URL (FileY.php). In a production environment where I directly include FileX.php into the main page (FileA.php), the iFrame gets echoed to the screen and FileY.php is successfully called.
In testing though, I need to call multiple versions of FileA.php which each include FileX.php which outputs the iFrame to call FileY.php. I am automating this large number of requests using cURL requests.
When loading FileA.php through a cURL request, it successfully does the include() of FileX.php but because it is happening through cURL, the iFrame never loads it's destination (FileY.php).
The cURL request for fileA looks something like this:
TestFile.php
// URL
$url = "http://www.example.com/FileA.php";
// New Cookie file
$ckfile = tempnam("/tmp", "CURLCOOKIE");
// New Connection
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt ($ch, CURLOPT_COOKIEFILE, $ckfile);
curl_setopt($ch, CURLOPT_COOKIESESSION, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_exec($ch);
curl_close($ch);
FileA.php:
include_once('FileX.php');
FileX.php:
echo("<iframe src='http://www.example.com/FileY.php' width='0' height='0'></iframe>");
FileY.php
// Contains logging stuff to log the fact that FileY.php was called.
Like I said, if I call FileA.php directly in my browser, FileX.php is included and FileY.php is loaded in the iFrame successfully. When I call FileA.php via cURL the iFrame doesn't load and FileY.php is never called.
I've tried wrapping the echo() in FileX.php with ob_start() and ob_end_flush() to force the output but that didn't work. I've tried adding a sleep(1) in case maybe the request was happening too fast, no luck.
Is there a cURL option I can change to allow this to occur? I can't figure out why it won't load the iFrame src.
Ah, so it turns out I was using an option incorrectly.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
This should actually be false or 0. When using true or 1, all output is caught and returned via the cURL request rather than being output. So it should be:
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
I have a codeigniter application that has a checkout system through authorize.net. The authorize.net library that I uses preforms a curl to make the payment, but after it is done I cannot redirect because
headers already sent by (output started at /Users/phil/Sites/Medbridge/httpdocs/application/libraries/AuthorizeCimLib.php:1
That is what the log says. If I comment out the payment thing it will redirect fine. I don't know if I am not understand a curl and that is why it is doing something or if I need to change some curl settings.
Thank you
EDIT
Here is the link to the library I am using, it is big and didn't want to repost the whole code
http://www.communitymx.com/content/article.cfm?page=4&cid=FDB14
Here is the curl part and maybe someone could see if this is doing the output to the header
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $this->_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, Array("Content-Type: text/xml"));
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLINFO_HEADER_OUT, FALSE);
curl_setopt($ch, CURLOPT_POSTFIELDS, $this->_xml);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$this->_response = curl_exec($ch);
Thank you
This is because the server has output something to your browser, and can't redirect through php after that.
As mentioned in the comments, you're most likely including some whitespace somewhere in your code. For practice and less troubleshooting, you don't need to include the ending ?> at the bottom of your php files. This is simply not required and sometimes there might end up a space after that ?> which causes an echo during the execution of your code.
Are you includeing any file in your code with might have a at the end of the file?
Another "solution" if you may, is to echo
<script type="text/javascript">
top.location = '<?=$str_redirect_url?>';
</script>
EDIT:
If you're using Codeigniter, you should check your model, helper and library files for whitespaces.
Perhaps you can use AJAX to issue the payment call, get the response and redirect using Javascript?