I need to get response from server using curl, but I can't.
The site: https://www.investing.com/holiday-calendar/
I can get that calendar using get request, but I need a list with custom dates. That mean I should use that datepicker. So when I press "apply" it sent post request with data I needed to get. (see the screenshots)
The DatePicker:
A post request with JSON response:
Code:
ini_set('error_reporting', E_ALL);
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
define('DIR', __DIR__);
$user_agent = isset($_SERVER['HTTP_USER_AGENT']) ? $_SERVER['HTTP_USER_AGENT'] : 'Traider';
$cookie = dirname(__FILE__).DIRECTORY_SEPARATOR.'init_cookie.txt';
$f = fopen('init_deb.txt', 'w');
$ch = curl_init();
$getUrl = 'https://www.investing.com/holiday-calendar/';
$postUrl = 'https://www.investing.com/holiday-calendar/Service/getCalendarFilteredData';
$dateFrom='2017-01-14';
$dateTo='2017-12-31';
$limit_from = 0;
$params = [
'dateFrom' => $dateFrom,
'dateTo' => $dateTo,
'county' => '',
'limit_from' => $limit_from
];
curl_setopt($ch, CURLOPT_URL, $postUrl);
curl_setopt($ch, CURLOPT_POSTFIELDS, $params);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_STDERR, $f);
$response = curl_exec($ch);
fclose($f);
curl_close($ch);
echo $response;
Step 1. I sent a get request, save cookies.
Step 2. I sent a post request changing $getUrl -> $postUrl. I always get the main page. Why I can't get JSON response?
after a bit of testing, the big secret is that they refuse requests that dont have the X-Requested-With:XMLHttpRequest header attached. attach that (using CURLOPT_HTTPHEADER), and you dont even need a cookie session. i guess its part of some XSS protection scheme.
working example code using hhb_curl from https://github.com/divinity76/hhb_.inc.php/blob/master/hhb_.inc.php :
<?php
declare(strict_types=1);
require_once('hhb_.inc.php');
$hc=new hhb_curl();
$hc->_setComfortableOptions();
$hc->setopt_array(array(
CURLOPT_POST=>true,
CURLOPT_POSTFIELDS=>http_build_query(array(
'dateFrom'=>'2017-01-28',
'dateTo'=>'2017-01-28',
'country'=>'',
'limit_from'=>'0'
)),
CURLOPT_HTTPHEADER=>array(
'X-Requested-With:XMLHttpRequest'
)
));
$hc->exec('https://www.investing.com/holiday-calendar/Service/getCalendarFilteredData');
hhb_var_dump($hc->getResponseBody());
Related
I’m building a login system in php where I need to use credentials from another website, I’m using an API to login to another server and I’m doing it using cURL. The server where the login credentials are stored does create a cookie with a unique tolken after the user has logged in correctly and this cookie is important to view other webpages and interrogate this pages using other APIs.
This is what I’ve done so far and it seems to work fine, in the login controller php file I’ve got this code
$km_username = filter_var($_POST['userName'], FILTER_SANITIZE_STRING);
$km_user_password = $_POST['userPassword'];
$cookieFile = "cookies.txt";
if(!file_exists($cookieFile)) {
$fh = fopen($cookieFile, "w");
fwrite($fh, "");
fclose($fh);
}
$url = 'https://www.apiwebsite.com/api/login.jsp?';
$fields = array(
'userid' => $km_username,
'password' => $km_user_password
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($fields));
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookieFile); // Cookie aware
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookieFile); // Cookie aware
$content = curl_exec($ch);
curl_close($ch);
In the page where I want to interrogate the server to get other datas I’ve got
$dates = array(
'd_inizio' => '01/01/2017',
'd_fine' => '31/12/2017'
);
$url = "https://www.apiwebsite.com/api/ricevute.jsp?";
$cookie = "../../km-controllers/cookies.txt";
$ch = curl_init ($url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($dates));
curl_setopt ($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec ($ch);
So basically after user has logged into the website cURL saves a cookie txt file into my server and this allows me to use that file any other time i want to make another call using for example a different api
Now the question is: what happen if I’ve got more than one user logging into the system? Do I need to create x number of cookies according on how many users log into the system? Would it not be simpler to save the cookie into the user’s browser?
Yes, you should use a different file for each client. What you can do is use tempnam() to generate a unique filename for the client, and save this in a session variable, then use it as the cookie jar. The login controller can do this:
session_start();
if (!isset($_SESSION['cookiefile'])) {
$cookiefile = tempnam(".", "cookie");
$_SESSION['cookiefile'] = basename($cookiefile);
}
And then the later page can use
$cookie = "../../km-controllers/" . $_SESSION['cookiefile'];
When the user logs out, you should delete this file and remove the session variable.
There's nothing that will automatically pass the cookies through from the curl session to the client browser or vice versa. If you don't want to save the cookies in a file, you can use curl_getinfo($ch, CURLINFO_COOKIELIST) to get the cookies, but you'll then have to parse the info yourself, and later use CURLOPT_COOKIE to set each cookie when making future calls. The cookie file automates all this.
Full code for login controller.
session_start(); // This needs to be at the very beginning of your script, before anything that produces output
//...
$km_username = filter_var($_POST['userName'], FILTER_SANITIZE_STRING);
$km_user_password = $_POST['userPassword'];
if (!isset($_SESSION['cookiefile'])) {
$cookieFile = tempnam(".", "cookie");
$_SESSION['cookiefile'] = basename($cookiefile);
file_put_contents($cookieFile, "");
}
$cookieFile = $_SESSION['cookiefile'];
$url = 'https://www.apiwebsite.com/api/login.jsp?';
$fields = array(
'userid' => $km_username,
'password' => $km_user_password
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($fields));
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookieFile); // Cookie aware
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookieFile); // Cookie aware
$content = curl_exec($ch);
curl_close($ch);
And the later interrogator:
session_start(); // at very beginning
// ...
$dates = array(
'd_inizio' => '01/01/2017',
'd_fine' => '31/12/2017'
);
$url = "https://www.apiwebsite.com/api/ricevute.jsp?";
if (!isset($_SESSION['cookiefile'])) {
die("no cookie file");
}
$cookie = "../../km-controllers/" . $_SESSION['cookiefile'];
$ch = curl_init ($url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($dates));
curl_setopt ($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec ($ch);
I'm trying, without success, to get some results using Bing's Image search API without HTTP/Request2.php component (as used in the official examples).
I understand that the only two required parameters to make a very primitive call are q which is the query string and the subscription key. The key must be sent using headers. After looking around I found this very simple example to send headers with PHP:
$sURL = "https://api.cognitive.microsoft.com/bing/v5.0/images/search?q=cats";
$aHTTP = array(
'Ocp-Apim-Subscription-Key' => 'xxxxxxx',
);
$context = stream_context_create($aHTTP);
$contents = file_get_contents($sURL, false, $context);
echo $contents;
But it does not output anything. Would you kindly help me with a very basic example of use of Bing's API?
SOLVED
Thanks to Vadim's hint I changed the way headers are sent and now output is a Json encoded result. (Remember to add your API subscription key.)
$sURL = "https://api.cognitive.microsoft.com/bing/v5.0/images/search?q=cats";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $sURL);
curl_setopt($ch, CURLOPT_TIMEOUT, '1');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: multipart/form-data',
'Ocp-Apim-Subscription-Key: xxxxx'
));
$content = curl_exec($ch);
echo $content;
Just another tip. The syntax of query filters and other parameters change form version to version. For example the following work correctly in version 5.0:
To search only for JPEG images of cats and get 30 results use:
q=cats&encodingFormat='jpeg'&count=30
To search only for 'portrait' aspect images with size between 200x200 and 500x500 use:
q=cats&aspect=Tall&size=Medium
Try using cURL
$sURL = "https://api.cognitive.microsoft.com/bing/v5.0/images/search?q=cats";
$key = "xxxxxxx";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $sURL);
curl_setopt($ch, CURLOPT_TIMEOUT, '1');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 'ocp-apim-subscription-key:$key');
$content = curl_exec($ch);
echo $content;
Here is my working code..
Replace ******** with your bing subscription key.
$sURL = "https://api.cognitive.microsoft.com/bing/v5.0/images/search?q=microsoft-surface&count=6&mkt=en-US";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $sURL);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: multipart/form-data',
'Ocp-Apim-Subscription-Key: *******************'
));
$contents = curl_exec($ch);
$myContents = json_decode($contents);
if(count($myContents->value) > 0) {
foreach ($myContents->value as $imageContent) {
echo '<pre/>';
print_r($imageContent);
}
}
This code always returns user doesn't exist from the API:
$data2 = array('user'=>$vars['mcusername'],
'pwd'=>$vars['mcpassword'],
'group'=>$postfields['group'],
'action'=>'Save');
// Connect to dvb API
$configWebAddress = "http://192.168.0.12:4040/dvbapi.html?part=userconfig&";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $configWebAddress);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data2);
$data = curl_exec($ch);
curl_close($ch);
The string that works in a browser is this:
dvbapi.html?part=userconfig&user=PeterTest&pwd=obfuscated&group=1,2&disabled=0&action=Save
When you access the URL in the browser, you're performing a GET. In your cURL attempt, you're attempting to POST. This is likely the issue; the script may only accept GET.
Try using this cURL code instead:
// Gather up all the values to send to the script
$data2 = array('part' => 'userconfig',
'user' => $vars['mcusername'],
'pwd' => $vars['mcpassword'],
'group' => $postfields['group'],
'action' => 'Save');
// Generate the request URL
$configWebAddress = "http://192.168.0.12:4040/dvbapi.html?".http_build_query($data2);
// cURL the URL for a responce
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $configWebAddress);
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$data = curl_exec($ch);
curl_close($ch);
// Show the responce
var_dump($data);
You can use http_build_query() to turn your array into a URL-encoded string to make the GET request.
I use cURL but untill now I used it for requesting data from servers. But now I want ot write API and data will be requested with cURL. But I don't know how Server reads data from cURL request.
This is my "client server" side request:
function sendRequest($site_name,$send_xml,$header_type=array('Content-Type: text/xml'))
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$site_name);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$send_xml);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST,1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER,0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_HTTPHEADER,$header_type);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 120);
$result = curl_exec($ch);
return $result;
}
$xml = "<request>
<session>
<user>exampleuser</user>
<pass>examplepass</pass>
</session>
</request>";
$sendreq = sendRequest("http://sitename.com/example.php",$xml);
echo $sendreq;
How do I need to write "main server" side script so I can read what user and pass from request are???
Thank you a lot.
To just be able to read it try this
curl_setopt($ch, CURLOPT_POSTFIELDS,array('data'=>$send_xml));
Then
print_r($_POST['data'])
Alternatively skip the XML and try something like this:
$data = array(
'request' => array(
'session' => array(
'user'=>'exampleuser',
'pass'=>'examplepass')
)
);
$sendreq = sendRequest("http://sitename.com/example.php",$data);
In example.php
print_r($_POST)
I have a Affiliate URL Like http://track.abc.com/?affid=1234
open this link will go to http://www.abc.com
now i want to execute the http://track.abc.com/?affid=1234 Using CURL
and now how i can Get http://www.abc.com
with Curl ?
If you want cURL to follow redirect headers from the responses it receives, you need to set that option with:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
You may also want to limit the number of redirects it follows using:
curl_setopt($ch, CURLOPT_MAXREDIRS, 3);
So you'd using something similar to this:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://track.abc.com/?affid=1234");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_MAXREDIRS, 3);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$data = curl_exec($ch);
Edit: Question wasn't exactly clear but from the comment below, if you want to get the redirect location, you need to get the headers from cURL and parse them for the Location header:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://track.abc.com/?affid=1234");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, true);
$data = curl_exec($ch);
This will give you the headers returned by the server in $data, simply parse through them to get the location header and you'll get your result. This question shows you how to do that.
I wrote a function that will extract any header from a cURL header response.
function getHeader($headerString, $key) {
preg_match('#\s\b' . $key . '\b:\s.*\s#', $headerString, $header);
return substr($header[0], strlen($key) + 3, -2);
}
In this case, you're looking for the value of the header Location. I tested the function by retrieving headers from a TinyURL, that redirects to http://google.se, using cURL.
$url = "http://tinyurl.com/dtrkv";
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($ch);
curl_close($ch);
$location = getHeader($data, 'Location');
var_dump($location);
Output from the var_dump.
string(16) "http://google.se"