This question already has answers here:
how to get the cookies from a php curl into a variable
(8 answers)
Closed 8 years ago.
When I get a response from a page, it gives a response data but if I want to get cookie of the session which is set by page, how can I get it with PHP cURL?
There are two ways(may be more) you can do this.
Using the cookie file:
$cookie_file = 'e:/demo/cookies.txt';
curl_setopt($ch,CURLOPT_COOKIEJAR, $cookie_file);
curl_setopt($ch,CURLOPT_COOKIEFILE, $cookie_file);
Using from the header that is responded back with html source from curl.
curl_setopt($curl_connection, CURLOPT_HEADER, true);
// this is returning the http response header along with html
You'll find the cookies there under the Set-Cookie: header for second example.
By the way, I assume you know how to handle curl. If you don't here are few helps.
Related
This question already has answers here:
Scrape web page data generated by javascript
(2 answers)
Web-scraping JavaScript page with Python
(18 answers)
Closed 3 years ago.
I am trying to create a a basic web crawler the specifically looks for links from adverts.
I have managed to find a script that uses cURL to get the contents of the target webpage
I also found one that uses DOM
<?php
$ch = curl_init("http://www.nbcnews.com");
$fp = fopen("source_code.txt", "w");
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
curl_close($ch);
fclose($fp);
?>
These are great and I certainly feel like I'm heading in the right direction except quite a few adverts are displayed using JS and as it's client side, it obviously isn't processed and I only see the JS code and not the ads.
Basically, is there any way of getting the JS to execute before I start trying to extract the links?
Thanks
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Fetching data from another website
I want to create a webpage, that would display another webpage on it at user's request. User enters URL and sees webpage he wants on my website. Request to another page has to come from my server, not from user. Otherwise I could just use iframe.
I'm willing to write it on php because I know some of it. Can anyone tell me what subjects one must know to do this ?
You need some kind of "PHP Proxy" for this, that means get the website contents via curl or file_get_contents(). Have a look at this here: http://davidwalsh.name/curl-download
Your proxy script that may look like this:
function get_data($url) {
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
echo get_data($_GET["url"]);
Please note that you may have to pay attention to headers for images etc. and there may also be some security flaws, but that is the basic idea.
Now you have to parse the contents of the initial website you just got and change all links from this format:
http://example.com/thecss.css
to
http://yoursite.com/proxy.php?url=http://example.com/thecss.css
Some regexes or PHP HTML parser may work here.
You could just use
echo file_get_contents('http://google.com')
But why not just download a php webproxy package like http://sourceforge.net/projects/poxy/
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
How to read a web page in PHP
I'm sure there is some simple way to do this. I need to pass get variables through to my cart software to record a conversion, but not redirect the user, I just want the server to send GET variables to a URL. I'd rather not turn on allow_url_fopen in php.ini.
Anyone know the best way to do this? Thanks in advance.
Server side, your best option is probably to use cURL - see the documentation for details, it's not too difficult to use.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.example.com/script.php");
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec($ch);
curl_close($ch);
if ($response === false) {
// Failed to connect or some error occurred
} else {
// Everything was fine, check the response here if you need to
}
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
How to send HTTP request and retrieve response in PHP (with fine-tuning of headers)?
8.1.4.1 Sample ping request
HTTP request:
POST /api/ra/v1/ping HTTP/1.0
Host: app.test.net
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=
Content-Type: application/json
"Are you there?"
can someone please help me with some php raw example?
try curl
http://www.php.net/manual/en/curl.examples-basic.php
$ch = curl_init("http://www.example.com/");
$fp = fopen("example_homepage.txt", "w");
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
curl_close($ch);
fclose($fp);
I made the same thing for a game site, where I would have to constantly log in to use my turns. So instead of that I used curl to sign in, get authorization token and using it, send another request with curl to do certain actions. Then I ran that script with the windows task scheduler.
I have the code somewhere, I'll look into it when I get home if you still need help with this.
I have, of course, read several questions with exactly this asked, but I have to say it didn't work for me at all. What I am about to accomplish is
sending 'X-Requested-With: XMLHttpRequest' header via PHP and curl
sending other http request headers via PHP and curl
provided solutions didn't work for me.
How do I know I'm not sending right http request headers?
Simply by
(1)comparing real headers generated by XMLHttpRequest(triggering JQuery click) and those simulated by PHP and curl in Firefox add-on Live HTTP headers
(2)Print_r() -ing $_SERVER variable in target script
What do I get that is incorrect/below my expectations?
First and most important:
Firefox Live HTTP headers does not capture my headers (just like they don't exists).
Second, by print_r($_SERVER):
if I get anything of simulated headers at all, I get [HTTP_X_REQUESTED_WITH] => XMLHttpRequest - not the: [X_REQUESTED_WITH] => XMLHttpRequest.
That problem persists almost for any header I send via curl_setopt($ch, CURLOPT_HTTPHEADER, $curl_header) - any of these is being prefixed with 'HTTP' ('Header1: value1' - I get 'HTTP_HEADER1').
I'm using XAMPP with PHP version 5.4.7, CURL 7.24.0 .
Before I ask if what I'm trying to accomplish is possible or maybe not and say thanks in advance for responses, it's not bad idea to provide my code - one of many code solutions that I've tried.
$curl_header = array('X-Requested-With: XMLHttpRequest');
$data = "name=miloshio"; // just to be sure I'm doing the POST request
$ch = curl_init('http://example.com/test.php');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $curl_header);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_VERBOSE, true);
$result = curl_exec($ch);
echo $result;
Sum of my questions:
Is it possible to send exactly 'X-Requested-With: XMLHttpRequest'
header via PHP and curl?
Is it possible to avoid attaching 'HTTP_' prefix to custom headers
send by PHP and curl?
Are there well-known limitations in matter of using PHP and curl?
Firefox Live HTTP headers won't show your headers as they're sent by the server to another server and not to the client(browser).
Curl send the headers correctly, using CURLOPT_PROXY You can try to put curl traffic through a debuging proxy like Fiddler if You're using windows for development, I'm sure there are linux alternatives
If you try to get the headers from $SERVER variable, they will be prefixed with HTTP, you can use apache_request_headers to get the headers without HTTP_ prefix.