I wish to mimic, using CURL with PHP, the operation of a website that retrieves data using an AJAX POST.
Normally when I'm viewing POST requests using Firebug you will see variable/value pairs, but in this case all you see is a single JSON string. E.g.
{"refId":"14536"}
Is there a way to mimic this request using CURL? I've looked at CURL but as far as I can see the CURLOPT_POSTFIELDS parameter has to be a query string made up of one or more name/value.
Here is my test code with a normal POST request using a single name/value pair. I'd like to modify it to do the above.
$curlOptions = array(
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // don't return headers
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_USERAGENT => "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3",
CURLOPT_CONNECTTIMEOUT => 600, // timeout on connect
CURLOPT_TIMEOUT => 600, // timeout on response
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => 'var1=113',
CURLOPT_URL => "http://localhost/t4.php"
);
$curlCh = curl_init();
curl_setopt_array( $curlCh, $curlOptions );
$fileContents = curl_exec( $curlCh );
$curlErr = curl_errno( $curlCh );
$curlErrmsg = curl_error( $curlCh );
if( $curlErr ) echo "CURL ERROR:</b> $curlErr $curlErrmsg";
echo $fileContents; //check worked
curl_close( $curlCh );
How about something like:
$postData = json_encode(array('refId' => '14536'));
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
Related
how can i getting a web page content with curl for this site:
divar . ir
I want try to get a page from a website with curl,But not work
I wrote the following code page 404 is displayed.
function get_web_page( $url )
{
$options = array(
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // don't return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_USERAGENT => "spider", // who am i
CURLOPT_AUTOREFERER => true, // set referer on redirect
CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect
CURLOPT_TIMEOUT => 120, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
CURLOPT_SSL_VERIFYPEER => false // Disabled SSL Cert checks
);
$ch = curl_init( $url );
curl_setopt_array( $ch, $options );
$content = curl_exec( $ch );
$err = curl_errno( $ch );
$errmsg = curl_error( $ch );
$header = curl_getinfo( $ch );
curl_close( $ch );
$header['errno'] = $err;
$header['errmsg'] = $errmsg;
$header['content'] = $content;
return $header;
}
$link="https://divar.ir/";
$res =get_web_page($link);
echo $res['content'];
Change CURLOPT_USERAGENT
CURLOPT_USERAGENT => "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0",
I am using this code to get the contents of a post request url using php curl
Code looks as below:
// Get cURL resource
$curl = curl_init();
// Set some options - we are passing in a useragent too here
curl_setopt_array($curl, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'http://www1.ptt.gov.tr/tr/interaktif/sonuc-yd.php',
CURLOPT_USERAGENT => 'Codular Sample cURL Request',
CURLOPT_POST => 1,
CURLOPT_POSTFIELDS => array(
'barcode' => 'CP021325078TR',
'security_code' => $capcha2
)
));
// Send the request & save response to $resp
$resp = curl_exec($curl);
// Close request to clear up some resources
curl_close($curl);
echo "<pre>";
var_dump($resp);
echo "</pre>";
The result doesn’t seem to return anything at all.
What is wrong with this code?
Try this:
$url = 'http://www1.ptt.gov.tr/tr/interaktif/sonuc-yd.php';
$postvals = array(
'barcode' => 'CP021325078TR',
'security_code' => $capcha2
);
$resp = Request($url,$postvals);
echo "<pre>"; var_dump($resp); exit;
...
function Request($url,$params=array()){
$ch = curl_init();
$curlOpts = array(
CURLOPT_URL => $url,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0',
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true
);
if(!empty($params)){
$curlOpts[CURLOPT_POST] = true;
$curlOpts[CURLOPT_POSTFIELDS] = $params;
}
curl_setopt_array($ch,$curlOpts);
$answer = curl_exec($ch);
if (curl_error($ch)) {
echo curl_error($ch); exit;
}
curl_close($ch);
return $answer;
}
EDIT:
I tested this and got:
Could not resolve host: www1.ptt.gov.tr
So make sure you're calling the right endpoint.
Actually you need to set this variable
$captcha2
To use it here -
'security_code' => $capcha2
How i can redirect user to another web site with cookie?
Im using this code
<?php
$fields_string = 'client_login=jadro&client_pass=jadro&client_remember=on&action=client_login';
$options = array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_USERAGENT => "Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 FirePHP/0.3",
CURLOPT_AUTOREFERER => false,
CURLOPT_CONNECTTIMEOUT => 120,
CURLOPT_TIMEOUT => 120,
CURLOPT_MAXREDIRS => 10,
);
$ch = curl_init();
curl_setopt_array( $ch, $options );
curl_setopt($ch,CURLOPT_URL,'http://orion10.ru');
//curl_setopt($ch,CURLOPT_POST,count(explode('&',$fields)));
curl_setopt($ch,CURLOPT_POSTFIELDS,$fields_string);
//curl_setopt($ch,CURLOPT_COOKIEJAR, 'cooc.txt');
//curl_setopt($ch,CURLOPT_COOKIEFILE, 'cooc.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, getcwd()."/cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, getcwd()."/cookies.txt");
$result = curl_exec($ch);
echo $result;
//header("Location: http://orion10.ru".session_name().'='.session_id());
header('Refresh: 15; URL='.$url['http://orion10.ru']);
exit();
?>
I need authorize user to another site.
cURL is being executed on your server. Therefore, the website in question thinks that your server is the user. i.e. When you redirect the actual user to the website in question, it won't recognize them. Read this.
I am trying to scrape a website using PHP, CURL and POST method in order to submit a form before web scraping the page. The problem I am experiencing is that there is connected with POST method: no data is submitted to the server, so the scraped webpage doesn't contain what I am looking for.
I quit sure the problem is connected with the form type: enctype="multipart/form-data".
How can I manage this POST request, considering that the form is multipart/form-data?
Do I have to encode the post_string in a special way?
Here's the code I'm using:
function curl($url) {
//POST string
$post_string="XXXX";
$options = Array(
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_FOLLOWLOCATION => TRUE,
CURLOPT_AUTOREFERER => TRUE,
CURLOPT_CONNECTTIMEOUT => 120,
CURLOPT_TIMEOUT => 120,
CURLOPT_MAXREDIRS => 10,
CURLOPT_USERAGENT => "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1a2pre) Gecko/2008073000 Shredder/3.0a2pre ThunderBrowse/3.2.1.8",
CURLOPT_URL => $url,
CURLOPT_CAINFO => dirname(__FILE__)."/cacert.pem",
CURLOPT_POSTFIELDS => $post_string,
);
$ch = curl_init();
curl_setopt_array($ch, $options);
$data = curl_exec($ch);
curl_error($ch);
curl_close($ch);
return $data;
}
$scraped_page = curl("XXXURLXXX");
echo $scraped_page;
Thank you!
Set the CURLOPT_POST to true:
CURLOPT_POST = true
Then fill your post fields like this 'setup':
$postfields = array();
$postfields['field1'] = 'value1';
$postfields['field2'] = 'value2';
CURLOPT_POSTFIELDS => $postfields
If value is an array, the Content-Type header will be set to multipart/form-data.
The PHP manual
Yes, $post_string needs to be an array.
Also set CURLOPT_POST to true.
I feel each step i take this one 5 more walls come up its fun but also frustrating.
So I'm trying to login to a website through cURL and after a lot of head scratching and the use of firefox developer tools I saw that my browser was responding in JSON. I honestly dont know anything about JSON but i feel like if i get through this I'll be in the clear.
So the question is how do i emulate these JSON responses with cURL? do i just copy and paste what they are in an array? Do i include this array in each http requests or do i do them all seperately.
Here is the JSON code if you guys want to see it.
__jsonp1__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
__jsonp2__([{"id":"3","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"id":"2","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/subscribe","successful":true,"subscription":"/user/11585628"}]);
__jsonp3__([{"id":"4","clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","channel":"/meta/connect","successful":true,"advice":{"reconnect":"retry","interval":0,"timeout":600000}},{"channel":"/user/11585628","data":{"type":"subscribe"},"clientId":"adag3o01k7uyb0ub4s2v1h4r7fs1m3zfvp0","id":"5","authenticated":true}]);
_jsonp4__([{"id":"1","channel":"/meta/handshake","successful":true,"version":"1.0","supportedConnectionTypes":["long-polling","cross-origin-long-polling","callback-polling","websocket","eventsource","in-process"],"clientId":"uiqqkp0vf66rl0mlc8281ufknaw1qkcriu1","advice":{"reconnect":"retry","interval":0,"timeout":600000}}]);
here is my code I was trying somethings so thats why it make look a little weird
<?php
$ckfile =' __utma=173730677.1410450142.1370766442.1370882903.1370893342.8; __utmz=173730677.1370893342.8.6.utmcsr=web.groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/groups; __utma=64278953.892306882..1370882931.1370893339.9; __utmz=64278953.1370882931.8.4.utmcsr=groupme.com|utmccn=(referral)|utmcmd=referral|utmcct=/signin; _g=%3D%3D--772097f0c6a077ac0f904c981ba5523ddffef3d5; __utmc=64278953; __utmc=173730677; __utmb=64278953.1.10.1370893339; __utmb=173730677.2.10.1370893342';
$postfields = '{"username":"#gmail.com","password":"somepass","app_id":"groupme.com","grant_type":"password"}';
$postfields2 ='{"group":{"name":"test","memberships":[]}}';
$custom = 'X-Access-Token: CEbhaIkkKTc9dtVMpxyc2IZOfnzEoh5w4UTzsVSb';
$ch2 = curl_init();
$ch3 = curl_init();
$ch = curl_init();
curl_setopt_array(
$ch,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate')
)
);
curl_setopt_array(
$ch2,
array(
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => 'https://web.groupme.com/#access_token=some token',
CURLOPT_COOKIE=> $ckfile,
CURLOPT_USERAGENT =>'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0',
CURLOPT_REFERER => 'https://groupme.com/signin',
CURLOPT_RETURNTRANSFER => true,
//CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER => array('Host: web.groupme.com','Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language: en-US,en;q=0.5','Accept-Encoding: gzip, deflate','Content-Type=text/html;charset=utf-8','Server=thin 1.3.1 codename Triple Espresso','Strict-Transport-Security=max-age=31536000','X-Frame-Options=sameorigin','x-xss-protection=1; mode=block','Content-Length=24275')
)
);
$response = curl_exec($ch2);
//curl_close($ch);
echo '<pre>';
print_r($response);
echo '</pre>'
?>