I am using cURL via PHP to test service connections, and I'm getting some inconsistent results. When I run the test via PHP & cURL this is my result:
{"response":"\n\n\n\n \n \n
When I put that same URL in my browser I get this:
{"response":"\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\n<html xmlns=\"http://www.w3.org/1999/xhtml\">\n<head>\n <link href=\"/images/global/global.css\...and so on
The response in my browser is cut short, but you get the idea.
With my PHP, I read in a JSON file, parse out the URL I need and the use cURL to send a GET request. Here is the code that I am using to test the service via PHP:
<?php
include ("serviceURLs.php");
class callService {
function testService($url){
$ch = curl_init($url);
curl_exec($ch);
$info = curl_getinfo($ch);
if ($info['http_code'] == 200){
echo("Test has passed </br>");
}else{
echo("Test Failed.</br> ");
}
var_dump($info);
curl_close($ch);
}
function readFile(){
$myFile = "./service/catalog-adaptation.json";
$fr = fopen($myFile, 'r');
$fileData = fread($fr, filesize($myFile));
$json_a = json_decode($fileData, TRUE);
$prodServer = $json_a['serverRoots']['%SERVER_ROOT']['PROD'];
$demoServer = $json_a['serverRoots']['%SERVER_ROOT']['DEMO'];
$testServer = $json_a['serverRoots']['%SERVER_ROOT']['TEST'];
$testUrls = $json_a['commands'];
foreach($testUrls as $tURL){
$mURL = $tURL['URL'];
if(stripos($mURL, "%")===0){
$testTestService = str_replace("%SERVER_ROOT", $testServer, $mURL);
$testDemoService = str_replace("%SERVER_ROOT", $demoServer, $mURL);
$testProdService = str_replace("%SERVER_ROOT", $prodServer, $mURL);
echo ("Production test: ");
$this->testService($testProdService);
echo ("Demo test: ");
$this->testService($testDemoService);
echo ("Test test: ");
$this->testService($testTestService);
}
}
}
}
$newServiceTest = new callService;
$newServiceTest->readFile();
?>
Can anyone tell my why I am getting different results and how I can fix my code so I can get consistent results?
You need to set below option for return the transfer as a string of the return value of curl_exec() instead of outputting it out directly.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
Related
I'm new to JSON Code. I want to learn about the update function. Currently, I successfully can update data to the database. Below is the code.
<?php
require_once "../config/configPDO.php";
$photo_after = 'kk haha';
$report_id = 1;
$url = "http://172.20.0.45/TGWebService/TGWebService.asmx/ot_maintainReport?taskname=&reportStatus=&photoBefore=&photoAfter=". urlencode($photo_after) . "&reportID=$report_id";
$data = file_get_contents($url);
$json = json_decode($data);
$query = $json->otReportList;
if($query){
echo "Data Save!";
}else{
echo "Error!! Not Saved";
}
?>
the problem is, if the value of $photo_after is base64 string, which is too large string, it will give the error:
1) PHP Warning: file_get_contents.....
2) PHP Notice: Trying to get property 'otReportList' of non-object in C:
BUT
when I change the code to this,
<?php
require_once "../config/configPDO.php";
$photo_after = 'mama kk';
$report_id = 1;
$sql = "UPDATE ot_report SET photo_after ='$photo_after', time_photo_after = GETDATE(), ot_end = '20:30:00' WHERE report_id = '$report_id'";
$query = $conn->prepare($sql);
$query->execute();
if($query){
echo "Data Save!";
}else{
echo "Error!! Not Saved";
}
?>
The data will updated including when the value of $photo_after is in base 64 string.
Can I know what is the problem? Any solution to allow the base64 string update thru json link?
Thanks
// ...
// It's likely that the following line failed
$data = file_get_contents($url);
// ...
If the length of $url is more than 2048 bytes, that could cause file_get_contents($url) to fail. See What is the maximum length of a URL in different browsers?.
Consequent to such failure, you end up with a value of $json which is not an object. Ultimately, the property otReportList would not exist in $json hence the error: ...trying to get property 'otReportList' of non-object in C....
To surmount the URL length limitation, it would be best to embed the value of $photo_after in the request body. As requests made with GET method should not have a body, using POST method would be appropriate.
Below is a conceptual adjustment of your code to send the data with a POST method:
<?php
require_once "../config/configPDO.php";
# You must adapt backend behind this URL to be able to service the
# POST request
$url = "http://172.20.0.45/TGWebService/TGWebService.asmx/ot_maintainReport";
$report_id = 1;
$photo_after = 'very-long-base64-encoding-of-an-image';
$request_content = <<<CONTENT
{
"taskname": $taskname,
"report_id": $report_id,
"photoBefore": $photoBefore,
"photo_after": $photo_after,
"reportStatus": $reportStatus
}
CONTENT;
$request_content_length = strlen($request_content);
# Depending on your server configuration, you may need to set
# $request_headers as an associative array instead of a string.
$request_headers = <<<HEADERS
Content-type: application/json
Content-Length: $request_content_length
HEADERS;
$request_options = array(
'http' => array(
'method' => "POST",
'header' => $request_headers,
'content' => $request_content
)
);
$request_context = stream_context_create($request_options);
$data = file_get_contents($url, false, $request_context);
# The request may fail for whatever reason, you should handle that case.
if (!$data) {
throw new Exception('Request failed, data is invalid');
}
$json = json_decode($data);
$query = $json->otReportList;
if ($query) {
echo "Data Save!";
} else {
echo "Error!! Not Saved";
}
?>
sending a long GET URL is not a good practice. You need to use POST method with cURL. And your webservice should receive the data using post method.
Here's example sending post using PHP:
//
// A very simple PHP example that sends a HTTP POST to a remote site
//
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"http://www.example.com/tester.phtml");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,
"postvar1=value1&postvar2=value2&postvar3=value3");
// In real life you should use something like:
// curl_setopt($ch, CURLOPT_POSTFIELDS,
// http_build_query(array('postvar1' => 'value1')));
// Receive server response ...
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$server_output = curl_exec($ch);
curl_close ($ch);
// Further processing ...
if ($server_output == "OK") { ... } else { ... }
Sample code from: PHP + curl, HTTP POST sample code?
And all output from the webservice will put in the curl_exec() method and from there you can decode the replied json string.
I previously had a Google geocoding script working to extract longitude and latitude using local addresses in a database.
In the last 6 months I've switched hosts, and apparently Google has implemented a new forward geocoder. Now it just returns the url not loading error from the xml script call.
I've tried everything to get my code working. Even sample coding from other websites won't work on my server. What am I missing? Is there possibly a server side setting that is blocking this from executing properly?
Attempt # 1:
$request_url = "http://maps.googleapis.com/maps/api/geocode/xml?new_forward_geocoder=true&address=1600+Amphitheatre+Parkway,+Mountain+View,+CA";
echo $request_url;
$xml = simplexml_load_file($request_url) or die("url not loading");
$status = $xml->status;
return $status;
Simply returns url not loading. I have tried with and without the new_forwad_geocoder. I have also tried with and without https.
The $request_url string DOES return proper results if you simply copy and paste it to a browser.
Also tried this just to see if I could get a file to return. Attempt 2:
$request_url = "http://maps.googleapis.com/maps/api/geocode/json?new_forward_geocoder=true&address=1600+Amphitheatre+Parkway,+Mountain+View,+CA";//&sensor=true
echo $request_url."<br>";
$tmp = file_get_contents($request_url);
echo $tmp;
Any idea what could be causing the connection failure?
I wasn't ever able to get this working with XML again and the file_get_contents call was the culprit I'm almost positive.
I've posted what I did get to work with JSON/Curl (below) in case anyone has similar issues.
Ultimately I think the problems I ran into had to do with an upgrade to our Apache version on the server; and some of the default settings related to file_get_contents and fopen being more restrictive. I haven't confirmed this though.
This code does work though:
class geocoder{
static private $url = "http://maps.google.com/maps/api/geocode/json?sensor=false&address=";
static public function getLocation($address){
$url = self::$url.$address;
$resp_json = self::curl_file_get_contents($url);
$resp = json_decode($resp_json, true);
//var_dump($resp);
if($resp['status']='OK'){
//var_dump($resp['results'][0]['geometry']['location']);
//echo "<br>";
//var_dump($resp['results'][0]['geometry']['location_type']);
//echo "<br>";
//var_dump($resp['results'][0]['place_id']);
return array ($resp['results'][0]['geometry']['location'], $resp['results'][0]['geometry']['location_type'], $resp['results'][0]['place_id']);
}else{
return false;
}
}
static private function curl_file_get_contents($URL){
$c = curl_init();
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_URL, $URL);
$contents = curl_exec($c);
curl_close($c);
if ($contents) return $contents;
else return FALSE;
}
}
$Address = "1600 Amphitheatre Parkway, Mountain View, CA";
$Address = urlencode(trim($Address));
list ($loc, $type, $place_id) = geocoder::getLocation($Address);
//var_dump($loc);
$lat = $loc["lat"];
$lng = $loc["lng"];
echo "<br><br> Address: ".$Address;
echo "<br>Lat: ".$lat;
echo "<br>Lon: ".$lng;
echo "<br>Location: ".$type;
echo "<br>Place ID: ".$place_id;
I want to scrap some information of a webpage .It uses a table layout structure.
I want to extract the third table inside the nested table layout which contains a series of nested tables .Each publishing a result .But the code is not working
include('simple_html_dom.php');
$url = 'http://exams.keralauniversity.ac.in/Login/index.php?reslt=1';
$html = file_get_contents($url);
$result =$html->find("table", 2);
echo $result;
I Used Curl to extract website but the problem is its tags is in out of order so it cannot be extracted using simple dom element .
function curl($url) {
$ch = curl_init(); // Initialising cURL
curl_setopt($ch, CURLOPT_URL,$url); // Setting cURL's URL option with the $url variable passed into the function
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE); // Setting cURL's option to return the webpage data
$data = curl_exec($ch); // Executing the cURL request and assigning the returned data to the $data variable
curl_close($ch); // Closing cURL
return $data; // Returning the data from the function
}
function scrape_between($data, $start, $end){
$data = stristr($data, $start); // Stripping all data from before $start
$data = substr($data, strlen($start)); // Stripping $start
$stop = stripos($data, $end); // Getting the position of the $end of the data to scrape
$data = substr($data, 0, $stop); // Stripping all data from after and including the $end of the data to scrape
return $data; // Returning the scraped data from the function
}
$scraped_page = curl($url); // Executing our curl function to scrape the webpage http://www.example.com and return the results into the $scraped_website variable
$scraped_data = scrape_between($scraped_page, ' </html>', '</table></td><td></td></tr>
</table>');
echo $scraped_data;
$myfile = fopen("newfile.html", "w") or die("Unable to open file!");
fwrite($myfile, $scraped_data);
fclose($myfile);
How to scrape the result and save the pdf
Simple HTML Dom can't handle that html. So first switch to this library,
Then do:
require_once('advanced_html_dom.php');
$dom = file_get_html('http://exams.keralauniversity.ac.in/Login/index.php?reslt=1');
$rows = array();
foreach($dom->find('tr.Function_Text_Normal:has(td[3])') as $tr){
$row['num'] = $tr->find('td[2]', 0)->text;
$row['text'] = $tr->find('td[3]', 0)->text;
$row['pdf'] = $tr->find('td[3] a', 0)->href;
if(preg_match_all('/\d+/', $tr->parent->find('u', 0)->text, $m)){
list($row['day'], $row['month'], $row['year']) = $m[0];
}
// uncomment next 2 lines to save the pdf
// $filename = preg_replace('/.*\//', '', $row['pdf']);
// file_put_contents($filename, file_get_contents($row['pdf']));
$rows[] = $row;
}
var_dump($rows);
Find a sample code
?php
// Defining the basic cURL function
function curl($url) {
$ch = curl_init(); // Initialising cURL
curl_setopt($ch, CURLOPT_URL, $url); // Setting cURL's URL option with the $url variable passed into the function
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE); // Setting cURL's option to return the webpage data
$data = curl_exec($ch); // Executing the cURL request and assigning the returned data to the $data variable
curl_close($ch); // Closing cURL
return $data; // Returning the data from the function
}
?>
<?php
$scraped_website = curl("http://www.example.com"); // Executing our curl function to scrape the webpage http://www.example.com and return the results into the $scraped_website variable
$result =$substring($scraped_website ,11,7); //change values 11,7 for table
echo $result;
?>
(I'm scraping this stuff with the permission of the website in question, by the way).
Pretty simple web scraper, was working fine when I was loading all the links by hand, but when I've tried to load them in via JSON and variables (so I can do lots of scraping with the one script and make the process more modular by just adding more links to JSON) it runs on an infinite loop.
(Page has been loading for about 15 minutes now)
Here is my JSON. Only one store is in there for testing purposes but there is going to be about 15 more.
[
{
"store":"Incu Men",
"cat":"Accessories",
"general_cat":"Accessories",
"spec_cat":"accessories",
"url":"http://www.incuclothing.com/shop-men/accessories/",
"baseurl":"http://www.incuclothing.com",
"next_select":"a.next",
"prod_name_select":".infobox .fn",
"label_name_select":".infobox .brand",
"desc_select":".infobox .description",
"price_select":"#price",
"mainImg_select":"",
"more_imgs":".product-images",
"product_url":".hproduct .photo-link"
}
]
Here is the PHP scraper code:
<?php
//Set infinite time limit
set_time_limit (0);
// Include simple html dom
include('simple_html_dom.php');
// Defining the basic cURL function
function curl($url) {
$ch = curl_init();
// Initialising cURL
curl_setopt($ch, CURLOPT_URL, $url);
// Setting cURL's URL option with the $url variable passed into the function
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
// Setting cURL's option to return the webpage data
$data = curl_exec($ch);
// Executing the cURL request and assigning the returned data to the $data variable
curl_close($ch);
// Closing cURL
return $data;
// Returning the data from the function
}
function getLinks($catURL, $prodURL, $baseURL, $next_select) {
$urls = array();
while($catURL) {
echo "Indexing: $url" . PHP_EOL;
$html = str_get_html(curl($catURL));
foreach ($html->find($prodURL) as $el) {
$urls[] = $baseURL . $el->href;
}
$next = $html->find($next_select, 0);
$url = $next ? $baseURL . $next->href : null;
echo "Results: $next" . PHP_EOL;
}
return $urls;
}
$string = file_get_contents("jsonWorkers/incuMens.json");
$json_array = json_decode($string,true);
foreach ($json_array as $value){
$baseURL = $value['baseurl'];
$catURL = $value['url'];
$store = $value['store'];
$general_cat = $value['general_cat'];
$spec_cat = $value['spec_cat'];
$next_select = $value['next_select'];
$prod_name = $value['prod_name_select'];
$label_name = $value['label_name_select'];
$description = $value['desc_select'];
$price = $value['price_select'];
$prodURL = $value['product_url'];
if (!is_null($value['mainImg_select'])){
$mainImg = $value['mainImg_select'];
}
$more_imgs = $value['more_imgs'];
$allLinks = getLinks($catURL, $prodURL, $baseURL, $next_select);
}
?>
Any ideas why the script would be running infinitely and not returning anything/stopping/printing anything to screen? I'm just gonna let it run until it stops. When I was doing this by hand it would only take a minute or so, sometimes less, so I'm sure it's a problem with my variables/json but I can't for the life of me see what the issues lie.
Can anyone take a quick look and point me in the right direction?
There is a problem with your while($catURL) loop. What do you want to do ?
Moreover, you can force to display information on your browser with the flush() command.
I'm using CURL to post to a script hosted on a remote server.
I'm sending a multidimensional array using this:
$urlserver = "myserver";
$arraytag = array('tags'=>$taggenerici,'tagesplosi'=>$tagesplosi,'matrice'=>$matricefin,'id' =>$identificativo);
$postfields = http_build_query($arraytag);
//open connection
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL,$urlserver);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch,CURLOPT_POST,sizeof($postfields));
curl_setopt($ch, CURLOPT_POSTFIELDS,$postfields);
curl_setopt($ch, CURLOPT_RETURNTRANSFER ,1); // RETURN THE CONTENTS OF THE CALL
//execute request sending post
$result = curl_exec($ch);
echo $result;
//close connection
curl_close($ch);
The problem is about the result: infact if i try to execute my script i get a randomic result. I'd like to view an array of 20rows X 43columns but it stops at row10 and column28. But if i refresh my page after some try i get my full array.
I'd like to say that i have tried to get the array before sending it to the remote server and it works fine cos i get my array entirely without any kind of cutting.
script being called (minus unused mysql connection):
<?php
$taggenerici = $_POST['tags'];
$matrice = $_POST['matrice'];
$identificativo = $_POST['id'];
$tagesplosi = $_POST['tagesplosi'];
//Here i create the array with "a" and "?"
for($dom=0;$dom<sizeof($identificativo);$dom++) {
for ($tag=0;$tag<sizeof($taggenerici);$tag++) {
$matrice[$dom][$tag] = "a, ";
}
$tagAdd=sizeof($taggenerici)+1;
$matrice[$dom][$tagAdd] ="?";
}
//Here i set "p".
for($dom=0;$dom<sizeof($identificativo);$dom++) {
for ($tag=0;$tag<sizeof($taggenerici);$tag++) {
for ($tagarray=0;$tagarray<sizeof($tagesplosi[$dom]);$tagarray++) {
if ($taggenerici[$tag] == $tagesplosi[$dom][$tagarray]) {
$matrice[$dom][$tag] = "p, ";
}
}
}
}
//this is the $result which I call on the client. (echo $valore);
foreach ($matrice as $kappa => $vu) {
echo "<br>";
foreach ($vu as $kii => $valore)
echo $valore;
}
}