When i am Decoding using commented "$jsonString" String it is working very well.
But after using curl it is not working, showing Null.
Please Help Me in this.
if (isset($_POST['dkno'])) {
$dcktNo = $_POST['dkno'];
$url = 'http://ExampleStatus.php?dkno=' . $dcktNo;
$myvars = '';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $myvars);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$jsonString = curl_exec($ch);
// $jsonString = '[{"branchname":"BHUBNESHWAR","consignee":"ICICI BANK LTD","currentstatus":"Delivered by : BHUBNESHWAR On - 25/07/2015 01:00","dlyflag":"Y","PODuploaded":"Not Uploaded"}]';
if ($jsonString != '') {
$json = str_replace(array('[', ']'), '', $jsonString);
echo $json;
$obj = json_decode($json);
if (is_null($obj)) {
die("<br/>Invalid JSON, don't need to keep on working on it");
} else {
$podStatus = $obj->PODuploaded;
}
}
}
}
After curl I used following concept to get only JSON data from HTML Page.
1) fetchData.php
$url = 'http://DocketStatusApp.aspx?dkno=' . $dcktNo;
$myvars = '';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $myvars);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$jsonString = curl_exec($ch);
// now get only value
$dom = new DOMDocument();
$dom->loadHTML($jsonString);
$thediv = $dom->getElementById('Label1');
echo $thediv->textContent;
2) JSONprocess.php
if (isset($_POST['dkno'])) {
$dcktNo = $_POST['dkno'];
ob_start(); // begin collecting output
include_once 'fetchData.php';
$result = ob_get_clean(); // Completed collecting output
// Now it will show & take only JSON Data from Div Tag
$json = str_replace(array('[', ']'), '', $result);
$obj = json_decode($json);
if (is_null($obj)) {
die("<br/>Invalid JSON, don't need to keep on working on it");
} else {
$podStatus = $obj->PODuploaded;
}
}
Related
I have a very strange problem. I am trying to return an array from a JSON GET request using the Skyscanner API.
The Skyscanner URL I am generating and using in the cURL is: http://partners.api.skyscanner.net/apiservices/browseroutes/v1.0/DE/EUR/de-DE/HAM/PT/2016-12?apiKey=API_KEY
It returns 3 quotes but when I am using a JSON editor like http://www.jsoneditoronline.org/ it gives me 9 quotes!!!
Why does it show less quotes in my solution? And how do I get all quotes???
<?php
$cities = array("HAM");
$countries = array("PT");
$url_api = "http://partners.api.skyscanner.net/apiservices/browseroutes/v1.0/DE/EUR/de-DE/";
$api_key = "API_KEY_HERE";
$next_month = date('m')+1;
if ($next_month=="13") {
$this_year = date('Y')+1;
$next_month = "01";
} else {
$this_year = date('Y');
}
foreach($cities as $city){
foreach($countries as $country){
$url = $url_api.$city."/".$country."/".$this_year."-".$next_month."?apiKey=".$api_key;
$ip = "218.255.245.210";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_POST, false);
curl_setopt($ch, CURLOPT_ENCODING, 'gzip');
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Accept: application/json', 'X_FORWARDED_FOR: '.$ip));
curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 );
$data = curl_exec($ch);
$response = json_decode($data);
echo $url."<br/>";
$quotes = $response->Quotes;
foreach($quotes as $quote){
echo $quote->QuoteId.". ";
echo $quote->Direct."<br/>";
}
}
}
?>
I'm having trouble scraping an ASPX.NET website. Even after searching through stackoverflow I haven't been able to find a fix.
First off, this is a public accessible website that I'm trying to scrape so that I can email myself a copy occasionally in order to not bother having to go there with a browser.
My code goes so far as to capture the __VIEWSTATE and __EVENTVALIDATION but when I submit the form I get the "Validation of viewstate MAC failed" error.
Any ideas?
<?php
require_once ("simple_html_dom.php");
ini_set('display_errors', 'On');
error_reporting(E_ALL);
// Create curl connection
$url = 'http://www.ariautodirect.com/ui/index.aspx';
$cookieFile = 'cookie.txt';
$ch = curl_init();
// We must request the login page and get the ViewState and EventValidation hidden values
// and pass those along in the post request.
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setOpt($ch, CURLOPT_REFERER, 'http://www.ariautodirect.com/ui/index.aspx');
curl_setopt($ch, CURLOPT_HTTPHEADER,array('Origin: http://www.ariautodirect.com', 'Host: www.ariautodirect.com'));
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookieFile);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookieFile);
$curl_scraped_page = curl_exec($ch);
// Grab ViewState and EventValidation data
$html = str_get_html($curl_scraped_page);
$viewState = $html->find("#__VIEWSTATE", 0);
$eventValidation = $html->find("#__EVENTVALIDATION", 0);
$previousPage = $html->find("#__PREVIOUSPAGE", 0);
//create array of data to be posted
// This matches exactly what I am seeing being posted when looking at Fiddler
$post_data['__EVENTTARGET'] = '';
$post_data['__EVENTARGUMENT'] = '';
$post_data['__VIEWSTATE'] = $viewState->value;
$post_data['__EVENTVALIDATION'] = $eventValidation->value;
$post_data['__PREVIOUSPAGE'] = 'Kn9nmdEGkbDIIX-v7Z_vg0t3njAt1lzkUm-uXH6djRcAGjPo6-w6RWKF3BBQz1ijEoJZGJVqOvHTqB5ghEU40C8xhnw1';//hard-coded because $previousPage->value throughs error
$post_data['ctl00$HeaderCtl1$LoginControl1$txtUserName'] = '';
$post_data['ctl00$HeaderCtl1$LoginControl1$txtPassword'] = '';
$post_data['ctl00$HeaderCtl1$LoginControl1$txtClientCode'] = '';
$post_data['ctl00$SearchCriteria1$drpDownMiles'] = '50';
$post_data['ctl00$SearchCriteria1$txtZipCode'] = '30301';
$post_data['ctl00$SearchCriteria1$ddlMake'] = 'All Makes ';
$post_data['ctl00$SearchCriteria1$CollapsiblePanelExtender1_ClientState'] = 'true';
$post_data['ctl00$SearchCriteria1$CollapsiblePanelExtender2_ClientState'] = 'true';
$post_data['ctl00$SearchCriteria1$CollapsiblePanelExtender3_ClientState'] = 'true';
$post_data['ctl00$SearchCriteria1$CollapsiblePanelExtender4_ClientState'] = 'true';
$post_data['ctl00$SearchCriteria1$CollapsiblePanelExtender5_ClientState'] = 'true';
$post_data['ctl00$SearchCriteria1$CollapsiblePanelExtender6_ClientState'] = 'true';
$post_data['ctl00$SearchCriteria1$CollapsiblePanelExtender7_ClientState'] = 'true';
$post_data['ctl00$SearchCriteria1$CollapsiblePanelExtender8_ClientState'] = 'true';
$post_data['ctl00$ContentPlaceHolder1$DataGridResults1$drpDownMiles'] = '50';
$post_data['ctl00$ContentPlaceHolder1$DataGridResults1$txtZipCode'] = '30301';
$post_data['ctl00$ContentPlaceHolder1$DataGridResults1$btnSearch'] = 'Search';
$post_data['ctl00$ContentPlaceHolder1$DataGridResults1$drpDownPageList'] = '100';
$post_data['ctl00$ContentPlaceHolder1$DataGridResults1$txtVinSearch'] = '';
$post_data['ctl00$ContentPlaceHolder1$DataGridResults1$SrtxtClientId'] = '';
$post_data['ctl00$ContentPlaceHolder1$DataGridResults1$SrtxtVehicleNo'] = '';
$post_data['ctl00$ContentPlaceHolder1$DataGridResults1$hdnSortYear'] = '';
$post_data['ctl00$ContentPlaceHolder1$DataGridResults1$hdnSortModel'] = '';
$post_data['ctl00$ContentPlaceHolder1$DataGridResults1$hdnSortMileage'] = '';
$post_data['ctl00$ContentPlaceHolder1$DataGridResults1$hdnSortPrice'] = '';
//traverse array and prepare data for posting (key1=value1)
foreach ( $post_data as $key => $value) {
$post_items[] = rawurlencode($key) . '=' . rawurlencode($value);
}
//create the final string to be posted using implode()
$post_string = implode ('&', $post_items);
$post_string = http_build_query($post_data);
//Set options for post
$urlAcctSummary = "http://www.ariautodirect.com/ui/DisplayResults.aspx";
curl_setopt($ch, CURLOPT_URL, $urlAcctSummary);
curl_setOpt($ch, CURLOPT_POST, TRUE);
curl_setopt($ch, CURLOPT_HTTPHEADER,array('Origin: http://www.ariautodirect.com', 'Host: www.ariautodirect.com', 'Content-Type: application/x-www-form-urlencoded'));
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_string);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setOpt($ch, CURLOPT_REFERER, 'http://www.ariautodirect.com/ui/index.aspx');
// Perform our post request
$curl_scraped_page = curl_exec($ch);
echo $curl_scraped_page;
curl_close($ch);
?>
I wanted to make an inline bot! and when i do this:
function sendResponse($url, $data){
$ch = curl_init();
//curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: multipart/form-data'));
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, array('inline_query_id' => $data['inline_query_id'], 'results' => json_encode($data['results'])));
$output = curl_exec($ch);
return $output;
}
It wont work, the error (with or without the header): {"ok":false,"error_code":400,"description":"[Error]: Bad request: Field \"message_text\" must be of type String"}
but when I do it like this:
function sendResponse($url, $data){
$ch = curl_init();
//curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: multipart/form-data'));
curl_setopt($ch, CURLOPT_URL, $url.'?inline_query_id='.rawurlencode($data['inline_query_id']).'&results='.rawurlencode(json_encode($data['results'])));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
//curl_setopt($ch, CURLOPT_POST, 1);
//curl_setopt($ch, CURLOPT_POSTFIELDS, $q);
$output = curl_exec($ch);
return $output;
}
It works ! the problem is the second method request URI will be too large so I cannot use it!
Any way I can send these data is okay with me! thanks!
and the code for making $data is here:
$result = connectWebsite(SITE_SEARCH_URL, urlencode($update['inline_query']['query']));
$result = json_decode($result);
$output = array();
$output['inline_query_id'] = $update['inline_query']['id'];
$i = 0;
foreach($result as $post){
$data = array();
$data['type'] = 'article';
$data['id'] = strval($post->ID);
$data['title'] = '('.$post->atypes.') '.$post->title;
if(strlen($post->content) > 2100)
$tmp = substr($post->content, 0, 2096).'...';
$data['message_text'] = '<b>'.$post->title.'</b>'.ucwords($post->genre, ',').$tmp;
$data['parse_mode'] = 'HTML';
if(strlen($post->content) > 200)
$tmp = substr($post->content, 0, 196).'...';
//$data['description'] = ucwords($post->genre, ',').' | '.$tmp;
$output['results'][$i] = $data;
$i++;
if($i == MAX_RESULTS)
break;
}
sendResponse(API_URL.'answerInlineQuery', $output);
It might help someone so I'll answer it myself.
the problem was the UTF-8 encoding
I replaced substr with mb_substr
besides at the first line I'v added this: mb_internal_encoding("UTF-8")
and ... the problem was solved. now I can send my inline query results (or any other command) without the URL length problem
Thanks everyone for your help
This is my cURL POST function:
public function curlPost($url, $data)
{
$fields = '';
foreach($data as $key => $value) {
$fields .= $key . '=' . $value . '&';
}
rtrim($fields, '&');
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, count($data));
curl_setopt($ch, CURLOPT_POSTFIELDS, $fields);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
}
$this->curlPost('remoteServer', array(data));
How do I read the POST on the remote server?
The remote server is using PHP... but what var in $_POST[] should I read
for e.g:- $_POST['fields'] or $_POST['result']
You code works but i'll advice you to add 2 other things
A. CURLOPT_FOLLOWLOCATION because of HTTP 302
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
B. return in case you need to output the result
return $result ;
Example
function curlPost($url, $data) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
$result = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
return $result;
}
print(curlPost("http://yahoo.com", array()));
Another Example
print(curlPost("http://your_SITE", array("greeting"=>"Hello World")));
To read your post you can use
print($_REQUEST['greeting']);
or
print($_POST['greeting']);
as a normal POST request ... all data posted can be found in $_POST ... except files of course :) add an &action=request1 for example to URL
if ($_GET['action'] == 'request1') {
print_r ($_POST);
}
EDIT: To see the POST vars use the folowing in your POST handler file
if ($_GET['action'] == 'request1') {
ob_start();
print_r($_POST);
$contents = ob_get_contents();
ob_end_clean();
error_log($contents, 3, 'log.txt' );
}
hi guys i have this script but obviously im not using foreach right i was wondering is there anyway i could combine these two requests to works as one
$url = "http://api.website.com/1.0/country?source=ballsche&programid=5380&campaignid=100000";
$ch = curl_init();
$timeout = 0;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$rawdata = curl_exec($ch);
curl_close($ch);
$array = json_decode($rawdata,true);
foreach($array as $obj) {
$country = $obj['country'];
$countrycode = $obj['countryCode'];
}
foreach($countrycode as $did) {
$wgurl = "http://api.website.com/1.0/city?source=ballsche&programid=5380&campaignid=100000&country-code=$did";
$wgch = curl_init();
$wgtimeout = 0;
curl_setopt($wgch, CURLOPT_URL, $wgurl);
curl_setopt($wgch, CURLOPT_HEADER, 0);
curl_setopt($wgch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($wgch, CURLOPT_CONNECTTIMEOUT, $wgtimeout);
$wgrawdata = curl_exec($wgch);
curl_close($wgch);
$wgarray = json_decode($wgrawdata,true);
}
foreach($wgarray as $wgobj) {
$city = $wgobj['city'];
$citycode = $wgobj['cityCode'];
if($city){
$sql = "INSERT INTO `pt_city` (city, citycode) VALUES ('$city', '$citycode')";
database_queryModify($sql,$result);
}else{
echo "dint work";
}
}
there must be an easy way to do this im guesing creating another array from the data but i cant quite get it right i keep getting errors ive tried this and a few other things my problem is i need to cycle trough county codes and make requests from the code there a 150 requests i need to cycle trough to get all the city information and for each request i need to decode the json thats coming back and insert it into my city table
just put foreach inside each other like this
$url = "http://api.website.com/1.0/country?source=ballsche&programid=5380&campaignid=100000";
$ch = curl_init();
$timeout = 0;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$rawdata = curl_exec($ch);
curl_close($ch);
$array = json_decode($rawdata,true);
foreach($array as $obj) {
$country = $obj['country'];
$countrycode = $obj['countryCode'];
foreach($countrycode as $did) {
$wgurl = "http://api.website.com/1.0/city?source=ballsche&programid=5380&campaignid=100000&country-code=$did";
$wgch = curl_init();
$wgtimeout = 0;
curl_setopt($wgch, CURLOPT_URL, $wgurl);
curl_setopt($wgch, CURLOPT_HEADER, 0);
curl_setopt($wgch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($wgch, CURLOPT_CONNECTTIMEOUT, $wgtimeout);
$wgrawdata = curl_exec($wgch);
curl_close($wgch);
$wgarray = json_decode($wgrawdata,true);
foreach($wgarray as $wgobj) {
$city = $wgobj['city'];
$citycode = $wgobj['cityCode'];
if($city){
$sql = "INSERT INTO `pt_city` (city, citycode) VALUES ('$city', '$citycode')";
database_queryModify($sql,$result);
} else {
echo "dint work";
}
}
}
}