I have 0 experince with curl, and i want to run a request that will return a json body inside php.
using firefox firbug i can copy curl order and it looks like this
curl 'http://*****.com/index.php/home/search/keyword/c815f2fec5?value=keyworkd' -H
'Host: *****.com' -H
'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0' -H
'Accept: */*' -H
'Accept-Language: en-US,en;q=0.5' -H
'Accept-Encoding: gzip, deflate' -H
'X-Requested-With: XMLHttpRequest' -H
'Referer: http://****.com/' -H
'Cookie: _ga=GA1.2.373422434.1399222050; PHPSESSID=l8sf036kjaijt6cjvcqnu992l4'
can i simulate such request with php ?
Of course it is, without going much into unnecessary things and based on your limited info:
<?php
$c = curl_init('http://*****.com/index.php/home/search/keyword/c815f2fec5?value=keyworkd');
curl_setopt($c, CURLOPT_USERAGENT, "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0");
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($c, CURLOPT_COOKIE,
'_ga=GA1.2.373422434.1399222050; PHPSESSID=l8sf036kjaijt6cjvcqnu992l4');
curl_setopt($c, CURLOPT_REFERER, 'http://****.com/');
$z = curl_getinfo($c);
$s = curl_exec($c);
curl_close($c);
?>
Edit: Damn though it is POST not COOKIE, repaired.
Edit2: With cookie file.
<?php
$cookie_file = "cookie.txt"; //remember to check if it exists
$c = curl_init('http://*****.com/index.php/home/search/keyword/c815f2fec5?value=keyworkd');
curl_setopt($c, CURLOPT_USERAGENT, "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0");
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($c, CURLOPT_COOKIEJAR, $cookie_file);
curl_setopt($c, CURLOPT_COOKIEFILE, $cookie_file);
curl_setopt($c, CURLOPT_REFERER, 'http://****.com/');
$z = curl_getinfo($c);
$s = curl_exec($c);
curl_close($c);
?>
Related
include_once('simple_html_dom.php');
$usuario = "username";
$password = "password";
$url = 'https://www.instagram.com/';
$url_login = 'https://www.instagram.com/accounts/login/ajax/';
$user_agent = array("Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 ",
"(KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36");
$ch = curl_init();
$headers = [
'Accept-Encoding: gzip, deflate',
'Accept-Language: en-US;q=0.6,en;q=0.4',
'Connection: keep-alive',
'Content-Length: 0',
'Host: www.instagram.com',
'Origin: https://www.instagram.com',
'Referer: https://www.instagram.com/',
'User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36',
'X-Instagram-AJAX: 1',
'X-Requested-With: XMLHttpRequest'
];
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_COOKIEFILE, "/tmp/cookie/pruebalogininsta2.txt");
curl_setopt($ch, CURLOPT_REFERER, $sTarget);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
$html = curl_exec($ch);
preg_match_all('/^Set-Cookie:\s*([^;]*)/mi', $html, $matches);
$cookies = array();
foreach($matches[1] as $item) {
parse_str($item, $cookie);
$cookies = array_merge($cookies, $cookie);
}
$headers = [
'Accept-Encoding: gzip, deflate',
//'Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4',
'Accept-Language: en-US;q=0.6,en;q=0.4',
'Connection: keep-alive',
'Content-Length: 0',
'Host: www.instagram.com',
'Origin: https://www.instagram.com',
'Referer: https://www.instagram.com/',
'User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36',
'X-Instagram-AJAX: 1',
'X-Requested-With: XMLHttpRequest'
];
$cadena_agregar_vector = 'X-CSRFToken:'. $cookies["csrftoken"];
$headers[] = $cadena_agregar_vector ;
$sPost = "username=".$usuario . "&password=". $password ;
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_POSTFIELDS, $sPost);
curl_setopt($ch, CURLOPT_URL, $url_login);
$html2 = curl_exec($ch);
curl_setopt($ch, CURLOPT_URL, "http://www.instagram.com/");
$html4 = curl_exec($ch);
echo $html4;
this is what I get
the problem is the way you hardcode Accept-Encoding: gzip, deflate, this makes curl send the encoding header indeed, but it does not turn on the decoding feature of curl, thus you get the raw data, without curl decoding it for you.
remove 'Accept-Encoding: gzip, deflate', and add curl_setopt($ch, CURLOPT_ENCODING, 'gzip, deflate'); , and curl will decode it for you (provided that curl is compiled with gzip & deflate support) - or better yet, just do curl_setopt($ch, CURLOPT_ENCODING, ''); , and curl will automatically list all supported encodings, so you dont run into the encoding problem where curl isn't compiled with gzip support.
on an unrelated note, you probably want to use CURLOPT_USERAGENT, not set the user-agent header manually. else, the UA-string will just be sent with this 1 request, and be reset on the next request, while CURLOPT_USERAGENT is kept until curl_close($ch)
edit: on my first revision of this post, i wrote CURLOPT_POSTFIELDS instead of CURLOPT_ENCODING, sorry, fixed that
edit 2: on another unrelated note, you're encoding the username/password wrong. instead of $sPost = "username=".$usuario . "&password=". $password ;, do
$sPost=http_build_query(array('username'=>$usuario,'password'=>$password));, else accounts with & or = or NULLs in the password or username wont work properly
The answer posted by #hanshenrik should really be accepted. But if you just want an easy solution that works and is not incorrect, remove the 'Accept-Encoding: gzip, deflate' from your headers array.
I think Firefox automatic proxy configuration is TSL 1.2:
$headers = array(
'host:ip-adress.eu',
'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language: en-US,en;q=0.5',
'Accept-Encoding: gzip, deflate, br',
'Proxy-Authorization: Basic b2xkLmV1cm9tYWtlckB5YWhvby5jb206WEpzV0Uwc0ZqT1pOK2MydDZWZWc4WWFreklaUVVDSUcxbDVrWE1yK0xKVT0=',
'Connection: keep-alive'
);
$proxy = "GQ2S4MZSFYZTMLRRHAYSGMJUG4ZTQMJRGIYDA.cd-n.net:143";
$url = 'ip-adress.eu';
$curl = curl_init();
curl_setopt($curl, CURLOPT_SSLVERSION, 6);
curl_setopt($curl, CURLOPT_HTTPPROXYTUNNEL, true); // OK
curl_setopt($curl, CURLOPT_PROXY, $proxy); // OK
curl_setopt($curl, CURLOPT_URL, trim($url)); // OK
curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0'); // OK
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
curl_setopt($curl, CURLOPT_TIMEOUT, 30);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_VERBOSE, false);
$httpresult = curl_exec($curl);
$httpstatus = curl_getinfo($curl, CURLINFO_HTTP_CODE);
print curl_errno ($curl);
curl_close($curl);
This code returns CURL Error - curl: (56) Recv failure: Connection reset by peer
PHP's curl is surprisingly undebugable and obscure. I have some problem downloading a JSON API data with cURL. I want to see what is exactly cURL sending to the remote HTTP server.
Currently the only debug option I have is to temporarily send request to some simple HTTP server that writes input to stdout. I would need to write that server just to debug curl!
What I do:
function get_data($url) {
$ch = curl_init();
echo "Download: $url.\n";
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
// I hoped to get some debug info
// but this setting has no effect
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_HEADER, array(
'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0',
'X-Purpose: Counting downloads.'
));
echo "Sending: \n".curl_getinfo($ch, CURLINFO_HEADER_OUT);
$data = curl_exec($ch);
var_dump($data);
echo curl_error($ch)." ".curl_errno($ch);
curl_close($ch);
return $data;
}
How can I get the data that is sent by cURL as a text?
If you want to define the headers you should use CURLOPT_HTTPHEADER and not CURLOPT_HEADER, i.e.:
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0',
'X-Purpose: Counting downloads.'
));
To get the the content curl is sending use:
curl_setopt($handle, CURLOPT_VERBOSE, true);
curl_setopt($handle, CURLOPT_STDERR,$f = fopen($verbosePath, "w+"));
function get_data($url) {
$verbosePath = __DIR__.DIRECTORY_SEPARATOR.'verbose.txt';
echo "Saving verbose to: $verbosePath\n";
$handle=curl_init('http://www.google.com/');
curl_setopt($handle, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($handle, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($handle, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($handle, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($handle, CURLOPT_HTTPHEADER, array(
'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0',
'X-Purpose: Counting downloads.'
));
curl_setopt($handle, CURLOPT_VERBOSE, true);
curl_setopt($handle, CURLOPT_STDERR,$f = fopen($verbosePath, "w+"));
$data = curl_exec($handle);
curl_close($handle);
fclose($f);
return $data;
}
get_data("https://www.google.com");
verbose.txt
* About to connect() to www.google.com port 80
* Trying 172.217.0.100... * connected
* Connected to www.google.com (172.217.0.100) port 80
> GET / HTTP/1.1
Host: www.google.com
Accept: */*
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0
X-Purpose: Counting downloads.
I am unable to set the host in curl. It still shows as localhost if i use the following code
function wget($url)
{
$agent= 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1';
$curlHeaders = array (
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Encoding: gzip, deflate',
'Accept-Language: en-US,en;q=0.5',
'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1',
'Connection: Keep-Alive',
'Pragma: no-cache',
'Referer: http://example.com/',
'Host: hostname',
'Cache-Control: no-cache',
'Cookie: visid_incap_185989=9v1q8Ar0ToSOja48BRmb8nn1GFUAAAAAQUIPAAAAAABCRWagbDIfmlN9NTrcvrct; incap_ses_108_185989=Z1orY6Bd0z3nGYE2lbJ/AXn1GFUAAAAAmb41m+jMLFCJB1rTIF28Mg==; _ga=GA1.3.637468927.1427699070; _gat=1; frontend=rqg7g9hp2ht788l309m7gk8qi7; _gat_UA-1279175-12=1; __utma=233911437.637468927.1427699070.1427699078.1427699078.1; __utmb=233911437.2.10.1427699078; __utmc=233911437; __utmz=233911437.1427699078.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmt_UA-1279175-1=1; _cb_ls=1; _chartbeat2=S0WVXDwMWnCFBgQp.1427699081322.1427699232786.1; PRUM_EPISODES=s=1427699568560&r=http%3A//example.com/'
);
$ch = curl_init();
curl_setopt ($ch, CURLOPT_HTTPHEADER, $curlHeaders);
curl_setopt ($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_URL,$url);
$result=curl_exec($ch);
return $result;
}
I use fiddler to track the network requests. where I found the host is still as localhost
If I load this same Link in browser i get as following in fiddler
I need my specified domain to be accessed. How can I achieve this?
Note: I am aware that host name should not contain the protocol.
Alternatively
Also i would like to know is it possible to get the source code of a website the could be seen in browser through terminal?
Assuming we are not trying spoof the Host header, omit the Host header altogether and let curl sort it out. In this case, just remove 'Host: hostname', because you already get curl to automatically set this with your code near the bottom with curl_setopt($ch, CURLOPT_URL, $url);.
If you really want to set the Host header yourself, then just replace
'Host: hostname',
with
"Host: ". parse_url($url, PHP_URL_HOST),
(Note: This function doesn't work with relative URLs.)
try like this,
curl_init('XXX.XXX.XXX.XXX');
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: subdomain.hostname.com'));
If you are using windows and xampp then try to use virtual host rather than localhost, then it will start working, I did the same.
According to HTTP quick specification read, I assume your problems are happening because of improper Host header being send. I was able to download some websites with following code:
function wget($url, $follow = true) {
$host = parse_url($url);
$agent = 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1';
$curlHeaders = array(
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Encoding: gzip, deflate',
'Accept-Language: en-US,en;q=0.5',
'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1',
'Connection: Keep-Alive',
'Pragma: no-cache',
'Referer: http://example.com/',
'Host: ' . $host['host'] . (isset($host['port']) ? ':' . $host['port'] : null), // building host header
'Cache-Control: no-cache',
'Cookie: visid_incap_185989=9v1q8Ar0ToSOja48BRmb8nn1GFUAAAAAQUIPAAAAAABCRWagbDIfmlN9NTrcvrct; incap_ses_108_185989=Z1orY6Bd0z3nGYE2lbJ/AXn1GFUAAAAAmb41m+jMLFCJB1rTIF28Mg==; _ga=GA1.3.637468927.1427699070; _gat=1; frontend=rqg7g9hp2ht788l309m7gk8qi7; _gat_UA-1279175-12=1; __utma=233911437.637468927.1427699070.1427699078.1427699078.1; __utmb=233911437.2.10.1427699078; __utmc=233911437; __utmz=233911437.1427699078.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmt_UA-1279175-1=1; _cb_ls=1; _chartbeat2=S0WVXDwMWnCFBgQp.1427699081322.1427699232786.1; PRUM_EPISODES=s=1427699568560&r=http%3A//example.com/'
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_HTTPHEADER, $curlHeaders);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, $follow); // following redirects or not
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
return $result;
}
echo(wget('http://example.com'));
Anyway this function is not an universal build. Personally I would add saving cookies between redirection requests etc. Essential change is within 'Host' header line. I'm building there proper Host header based on full $url provided to function.
Set the full URL into CURLOPT_URL.
Well, I'm trying to get some results from a form in a Webpage, I've already logged in, having the PHPSESSID a the responses are OK in the previous steps. If I execute from command line like this:
curl "http://domain/"%"27/ajaxform" -H "Cookie: PHPSESSID=l2pgk9rqihvo5n5gf3uq79c5e6" -H "Origin: http://domain" -H "Accept-Encoding: gzip,deflate" -H "Accept-Language: es,en-US;q=0.8,en;q=0.6,es-419;q=0.4" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36" -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" -H "Accept: */*" -H "Referer: http://domain/inicio" -H "X-Requested-With: XMLHttpRequest" -H "Connection: keep-alive" --data "formName=option1&other=attributes" --compressed > resp.txt
I can get the results in the resp.txt file.
But, running it in PHP all I get is http_code 500, I'm using the following code in PHP
function check_cdd($optionname, $sessionid){
$url="http://domain/".urlencode("'")."/ajaxform";
$cookie="cookie.txt";
$postvars = 'formName='.$optionname.'&other=attributes';
$ch = curl_init();
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_ENCODING , "");
curl_setopt ($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $postvars);
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt ($ch, CURLINFO_HEADER_OUT, false);
curl_setopt ($ch, CURLOPT_HEADER, false);
curl_setopt($ch,CURLOPT_HTTPHEADER,array(
'Cookie: PHPSESSID='.$sessionid,
'Origin: http://domain',
'Accept-Encoding:gzip,deflate',
'Accept-Language: es,en-US;q=0.8,en;q=0.6,es-419;q=0.4',
'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36',
'Content-Type: application/x-www-form-urlencoded; charset=UTF-8',
'Accept: */*',
'Referer: http://domain/inicio',
'X-Requested-With: XMLHttpRequest',
'Connection: keep-alive'));
$result = curl_exec ($ch);
$header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
$header = substr($result, 0, $header_size);
print("<pre>");
print_r(curl_error($ch));
print_r(curl_getinfo($ch));
echo $result;
print("</pre>");
curl_close($ch);}.
I've tryed differents params and options in PHP side, but always get 500 code in response, don't know what else to try. Thanks in advance for your answers
500 means internal server error...so help me to debug your code, by removing $postvars from your curl_setopt() function, it does not exist from your command line, and i suspect that it can be the origin of server error