I would like to send request to server. I am using this code for it
$data=[..];
$header=[
"Accept-language: en-US,en;q=0.5",
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Connection: keep-alive",
"Host: www.rvvi.cz",
"Referer: https://www.rvvi.cz/riv?s=rozsirene-vyhledavani",
"Upgrade-Insecure_Requests:1",
"User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/2010",
"Cookie: PHPSESSID=4ck9tc3vm4prgfubnjvutilgd2",
"Content-type: application/x-www-form-urlencoded"
];
$url = "https://www.rvvi.cz/riv";
$query = http_build_query($data);
$ch=curl_init();
curl_setopt($ch,CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_POST,strlen($query));
curl_setopt($ch,CURLOPT_POSTFIELDS,$query);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
$page=curl_exec($ch);
echo($page);
but problem is with PHPSESSID in header. To access that, i have to manually go to this server with my browser ( this that probably start my SESSION ), copy PHPSESSID and paste it into my script. But, i would like to make this script automatic. I need to to active them from my server without using browser or copying something.
Is there any way how can i start this session just with my php code?
$curl = curl_init( 'https://httpbin.org/post' );
curl_setopt( $curl, CURLOPT_POST, true );
curl_setopt( $curl, CURLOPT_POSTFIELDS, array( 'field1' => 'some data', 'field2' => 'some more data' ) );
curl_setopt( $curl, CURLOPT_RETURNTRANSFER, true );
$response = curl_exec( $curl );
curl_close( $curl );
Related
I have the method below that is to get the response from webserver using cURL.
function login (string $_login, string $_password) : string {
$url = "https://acweb.net.br/api/orcamentos/login";
$fields = [
"login" => $_login,
"password" => $_password
];
$headers = [
"Try" => "Trying"
];
$ch = curl_init();
curl_setopt( $ch, CURLOPT_URL, $url);
curl_setopt( $ch, CURLOPT_POST, true);
curl_setopt( $ch, CURLOPT_POSTFIELDS, $fields);
curl_setopt( $ch, CURLOPT_SSL_VERIFYPEER, false );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
return curl_exec( $ch );
}
It works fine!
i can get the value of $_POST with
print_r ($_POST)
But i can't get the value of CURLOPT_HTTPHEADER.
EDIT:
I did try so:
print_r ($_SERVER)
but it wasn't there.
How can i get the value of CURLOPT_HTTPHEADER?
all HTTP_headers in $_SERVER:
[HTTP_HOST] => ctemcasb.com.br
[HTTP_CONNECTION] => keep-alive
[HTTP_UPGRADE_INSECURE_REQUESTS] => 1
[HTTP_USER_AGENT] => Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36
[HTTP_SEC_FETCH_USER] => ?1
[HTTP_ACCEPT] => text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
[HTTP_SEC_FETCH_SITE] => none
[HTTP_SEC_FETCH_MODE] => navigate
[HTTP_ACCEPT_ENCODING] => gzip, deflate, br
[HTTP_ACCEPT_LANGUAGE] => pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7
[HTTP_COOKIE] => PHPSESSID=j4cqqdc83fia68nk0gsglqk1bv
HTTP_TRY, not exists.
And now?
I did this in the server:
print_r($_SERVER)
and
print_r ($_SERVER["HTTP_TRY]);
The headers shouldn't be an associative array, it should be an indexed array of strings.
$headers = [
'Try: Trying',
'Content-Type: text/html',
...
];
Then you should be able to access the header with: $_SERVER['HTTP_TRY'] since custom headers are prefixed with HTTP_
I want to extract data from this URL with Php Curl
https://www.tiket.com/ajax/pingFlightSearch?d=CGK&a=DPS&date=2017-08-18&adult=1&child=0&infant=0&airlines=%5B%22LION%22%5D&subsidy=true&page_view=roundseperate
But i got an empty page.
I check to that link and inspect with mozilla network->xhr and there is an content in response tab.
How I can extract the data from that link with php curl?
Here is my code
$url = 'https://www.tiket.com/ajax/pingFlightSearch?d=CGK&a=DPS&date=2017-08-18&adult=1&child=0&infant=0&airlines=%5B%22LION%22%5D&subsidy=true&page_view=roundseperate';
$cURL = curl_init();
curl_setopt($cURL, CURLOPT_URL, $url);
curl_setopt($cURL, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17');
curl_setopt($cURL, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($cURL, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($cURL, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($cURL, CURLOPT_AUTOREFERER, true);
curl_setopt($cURL, CURLINFO_HEADER_OUT, true);
curl_setopt($cURL, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($cURL, CURLOPT_VERBOSE,true);
curl_setopt($cURL, CURLOPT_HTTPHEADER, array(
"Host" => "www.tiket.com",
"User-Agent" => "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:55.0) Gecko/20100101 Firefox/55.0",
"Accept" => "application/json, text/javascript, */*; q=0.01",
"Accept-Language" => "en-us,en;q=0.5",
"Accept-Encoding" => "gzip, deflate, br",
"Content-Type" => "application/cap+xml;charset=utf-8",
"X-NewRelic-ID" => "UQIGUlJXGwACUFZaAAM=",
"X-Requested-With" => "XMLHttpRequest",
"Connection" => "keep-alive",
"Pragma" => "no-cache",
"Cache-Control" => "no-cache"
));
$result = curl_exec($cURL);
print_r($result);
curl_close($cURL);
I must add 'X-Requested-With: XMLHttpRequest' at header of that link then response will come out.
edit header
Then the response will come out
[response result][2]
You've done a mistake with CURLOPT_HTTPHEADER. You set params as Key => Value but it's incorrect. You should set params as values with ":" as the delimiter.
curl_setopt($cURL, CURLOPT_HTTPHEADER, array(
"Host: www.tiket.com",
"Accept: application/json, text/javascript, */*; q=0.01",
"Accept-Language: en-us,en;q=0.5",
"Content-Type: application/cap+xml;charset=utf-8",
"X-NewRelic-ID: UQIGUlJXGwACUFZaAAM=",
"X-Requested-With: XMLHttpRequest",
"Connection: keep-alive",
"Pragma: no-cache",
"Cache-Control: no-cache"
));
Also I have removed this string to get uncompressed data
"Accept-Encoding: gzip, deflate, br",
Mate are u sure that this link work? because I inspected the code with mozilla and chrome and the response is empty, and I tried to scrape with a simple php class that i have with curl and the result is empty cuz the link dont have response
I'm struggling using cURL within PHP. I'm not sure what I need to do to "translate" this:
curl -X POST -u "{username}:{password}" --header "Content-Type: application/json" --data-binary #profile.json "https://gateway.watsonplatform.net/personality-insights/api/v3/profile?version=2016-10-20&consumption_preferences=true&raw_scores=true"
into PHP to execute it there.
This is all I've got so far, but I'm feeling like I'm not even close:
$url2 = 'https://watson-api-explorer.mybluemix.net/personality-insights/api/v3/profile?raw_scores=false&csv_headers=false&consumption_preferences=true&version=2017-02-01';
$request_headers = array();
$request_headers[] = 'Accept: application/json';
$request_headers[] = 'Content-Type: text/plain';
$request_headers[] = 'Content-Language: en';
$request_headers[] = 'Accept-Language: en';
$ch2 = curl_init( $url2 );
curl_setopt( $ch2, CURLOPT_POST, 1);
curl_setopt( $ch2, CURLOPT_POSTFIELDS, $myvars2);
curl_setopt( $ch2, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt( $ch2, CURLOPT_HEADER, $request_headers);
curl_setopt( $ch2, CURLOPT_RETURNTRANSFER, 1);
$response2 = curl_exec( $ch2 );
var_dump($response2);
It looks like you are just missing the authentication piece:
curl_setopt( $ch2, CURLOPT_USERPWD, "yourUsername:yourPassword");
Check out the manual. Also, you can do it this way, which can be a little easier:
curl_setopt_array( $ch2, array(
CURLOPT_POST => 1,
CURLOPT_POSTFIELDS => $myvars2,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_HEADER => $request_headers,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_USERPWD => 'yourUsername:yourPassword'
);
I'm curling a URL with the following code at the moment, which works fine with either the get attached to the end of the URL or the POST data. But not with the get and the post.
However when I use the advanced rest client (add on for google chrome) it works just fine. Annoyingly though, I can't see the request that it sends to mimic it.
Heres the call i'm making with it.
$fields = array(
'searchPaginationResultsPerPage'=>500 );
foreach($fields as $key=>$value) { $fields_string .= $key.'='.$value.'&'; }
$fields_string = rtrim($fields_string,'&');
$curl = curl_init();
curl_setopt( $curl, CURLOPT_URL, 'http://www.microgenerationcertification.org/mcs-consumer/installer-search.php?searchPaginationPage=1' );
curl_setopt( $curl, CURLOPT_FOLLOWLOCATION, true );
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl,CURLOPT_POST,count($fields));
curl_setopt($curl,CURLOPT_POSTFIELDS,$fields_string);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 80);
$str = curl_exec($curl);
curl_close($curl);
Just using this as a bit of a test more than anything else, but can't seem to get it working. I can get the first 500 results all the time, but not the next 500.
This works
$fields = array (
'searchPaginationResultsPerPage' => 500,
'searchPaginationPage' => 1
);
$headers = array (
"Connection: keep-alive",
"User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.162 Safari/535.19",
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Encoding: gzip,deflate,sdch",
"Accept-Language: en-US,en;q=0.8",
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3"
);
$fields_string = http_build_query ( $fields );
$cookie = 'cf6c650fc5361e46b4e6b7d5918692cd=49d369a493e3088837720400c8dba3fa; __utma=148531883.862638000.1335434431.1335434431.1335434431.1; __utmc=148531883; __utmz=148531883.1335434431.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); mcs=698afe33a415257006ed24d33c7d467d; style=default';
$ch = curl_init ();
curl_setopt ( $ch, CURLOPT_URL, 'http://www.microgenerationcertification.org/mcs-consumer/installer-search.php?searchPaginationPage=1&searchPaginationResultsPerPage=500' );
curl_setopt ( $ch, CURLOPT_FOLLOWLOCATION, true );
curl_setopt ( $ch, CURLOPT_RETURNTRANSFER, 1 );
curl_setopt ( $ch, CURLOPT_CONNECTTIMEOUT, 80 );
curl_setopt ( $ch, CURLOPT_COOKIE, $cookie );
curl_setopt ( $ch, CURLOPT_HTTPHEADER, $headers );
$str = curl_exec ( $ch );
curl_close ( $ch );
echo $str;
You needed cookie information and make sure curl is using GET not POST
See Demo : http://codepad.viper-7.com/gTThxX (I hope the cokkies is not expired before you view it )
Not sure why that fails, looks fine.. What happens when you skip CURL and go for the PHP stream method:
$postdata = http_build_query(
array(
'searchPaginationResultsPerPage' => 500
)
);
$opts = array('http' =>
array(
'method' => 'POST',
'header' => 'Content-type: application/x-www-form-urlencoded',
'content' => $postdata
)
);
$context = stream_context_create($opts);
$result = file_get_contents('http://www.microgenerationcertification.org/mcs-consumer/installer-search.php?searchPaginationPage=1', false, $context);
I had a look at the page you are scraping and noticed the following:
When you change the results per page it posts your search again
They appear to be using the session to store your search parameters
You are not preserving the session ID when using CURL (and doing so is probably a bit more complex than you'd like) so this will not behave the same as on the website.
I did notice however that if you append the searchPaginationResultsPerPage parameter to the URL it works fine. Like this:
http://www.microgenerationcertification.org/mcs-consumer/installer-search.php?searchPaginationPage=0&searchPaginationResultsPerPage=500
That means you could actually use file_get_contents and not worry about the CURL stuff.
Start page
Frame page
End page
I can't figure out what I'm missing. I've tried mimicking the firefox request headers, but it doesn't work.
Also, the frame page reaches the end page using a javascript ajax request. It posts the data to $post_to_link (see code below) then navigates to the expected result (not my current result), where the megaupload link is located.
Expected output:
/membersonly/components/com_iceplayer/GMorBMlet.php?url=http%3A%2F%2Fwww.megaupload.com%2F%3Fd%3DVNICBFWL&
Current output:
file_get_contents outputs 3
curl outputs error 403 forbidden access
Here is my code:
// call it like so...
echo GetHosterLink( 1148, 252636, '', '37fn8Oklq', 15, -75 );
// $s is incremented every second you are 'visiting' the referer page
// $m decreases below zero when you move your mouse `down` on the start page
function GetHosterLink( $id, $link_id, $cap, $sec, $s, $m )
{
$link_page = str_replace( '[ID]', $id, 'http://www.icefilms.info/membersonly/components/com_iceplayer/video.php?vid=[ID]' );
$post = "id={$link_id}&s={$s}&iqs=&url=&m={$m}&cap=&sec={$sec}&t={$id}";
$header = implode( "\r\n", array(
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7",
"Accept-Encoding: gzip, deflate",
"Accept-Language: en-us,en;q=0.5",
"Cache-Control: no-cache",
"Connection: keep-alive",
"Content-Length: " . strlen( $post ),
"Content-Type: application/x-www-form-urlencoded; charset=UTF-8",
"Host: www.icefilms.info",
"Pragma: no-cache",
"Referer: http://www.icefilms.info/membersonly/components/com_iceplayer/video.php?h=374&w=631&vid={$id}&img=",
"User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0"
));
$opts = array('http' =>
array(
'method' => 'POST',
'header' => implode("\r\n",
array(
'Content-type: application/x-www-form-urlencoded',
'Content-length: ' . strlen( $post ),
'Referer: ' . $link_page . '&h=374&w=631',
'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0',
'Host: www.icefilms.info'
)
),
'content' => http_build_query(
array(
'id' => $link_id,
's' => $s,
'sec' => $sec,
't' => $id,
'm' => $m,
'iqs' => '',
'url' => '',
'cap' => ''
)
)
)
);
$context = stream_context_create($opts);
$post_to_link = 'http://www.icefilms.info/membersonly/components/com_iceplayer/video.phpAjaxResp.php';
$get_result = file_get_contents( $post_to_link, false, $context );
$f_result = cURL::DoRequest( $post_to_link, $post, '',
array( array( CURLOPT_HTTPHEADER, $header ) ) );
$f_r = array(
'result' => $f_result,
'get_result' => $get_result,
'get_opts' => $opts,
'get_response' => $http_response_header,
'req_post' => $post,
'req_href' => $post_to_link,
'req_header' => $header
);
return ( $f_r );
}
Here is my curl.php file:
class cURL
{
public static function DoRequest( $url, $post = '',
$cookie_file = '', $variables = array() )
{
$curl = curl_init();
#session_start();
$cookie = ( 'PHPSESSID=' . session_id() . '; path=/' );
#session_write_close();
curl_setopt( $curl, CURLOPT_URL, $url );
curl_setopt( $curl, CURLOPT_RETURNTRANSFER, 1 );
curl_setopt( $curl, CURLOPT_COOKIE, $cookie );
if ( !empty( $cookie_file ) )
{
curl_setopt( $curl, CURLOPT_COOKIEFILE, $cookie_file );
curl_setopt( $curl, CURLOPT_COOKIEJAR, $cookie_file );
}
if ( !empty( $post ) )
{
//curl_setopt( $curl, CURLOPT_POST, 1 );
curl_setopt( $curl, CURLOPT_POSTFIELDS, $post );
}
foreach ( $variables as $var )
curl_setopt( $curl, $var[0], $var[1] );
$result = curl_exec( $curl );
curl_close( $curl );
return ( $result );
}
}
Try visiting initial page (the one's url you have in $link_page) with curl and make sure the file you point to in:
curl_setopt( $curl, CURLOPT_COOKIE, $cookie );
exists and is writeable.
Then request the $post_to_link url with same curl resource.
By visiting the initial page you are getting cookies and making sure there is a valid session for your next request. That also secures the referer you then provide in headers. There are many ways to figure "automatic" requests, and things like checking cookie and if you actually visited "referer"-ed link are quite common.