I have a curl script that fetches json data from a server. When I save data into a file, it works fine and retrieves data only when new values are available.
But if I insert an INSERT query to save values into database, it starts fetching redundant values in a loop. Like new data is available each 2 minutes and when I save data into file, it fetches each 2 minute new data only. But I replace query with the file data insert code, it starts looping and fetching redundant data something like each 5 seconds.
How can I correct the script so that it fetches data when newer is available. Like it's doing when I save data into file.
Here is my code:
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="refresh" content="300">
<title>Weather Data</title>
<meta charset="utf-8">
</head>
<body>
<h2>Connect.php</h2>
<div>
<?php
require("Connection.php");
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://cors.io/?http://api.xxxxx.com/live/xxxxxxxxxx=m/s",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
//CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"Cache-Control: no-cache"
),
));
$response = curl_exec($curl);
$Wdata = json_decode($response, true);
$tem = $Wdata["temperature"];
$tem2 = $Wdata["2nd_temp"];
$hum = $Wdata["humidity"];
$tim = $Wdata["dateTime"];
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
echo "<br>";
echo "temp1:".$tem;
echo "<br>";
echo "humidity:".$hum;
echo "<br>";
echo "time:".$tim;
echo "<br>";
if ( $fl = fopen('weatherData.json','a'))
{ fwrite($fl,"\"Weather Data\": { \"Time\" : \"". $tim . "\" ,"."\"Humidity\" : \"". $hum . "\"}\n" );
//echo $_id.';'.$_time;
fclose($fl); }
// Here code to save $response into database; When I un-comment it, the script starts fetching redundant data in a loop
/*
try{
$sql = "INSERT INTO weatherdata (datetime, temperature, humidity)
VALUES ('".$tim."','".$tem."','".$hum."')";
echo "<meta http-equiv='refresh' content='0'>";
($conn->query($sql));
//$conn = null;
}
catch(PDOException $e)
{
echo $e->getMessage();
}
*/
}
?>
</div>
</body>
</html>
Related
I want to grab this number (28/28) with PHP from a tracking software (https://aircrasher.livefpv.com/live/scoring/) for drone races. But I can't. I think it's a websocket, but I don't know how to retrieve information from it. I tried curl, I tried the "ultimate-web-scraper" from GitHub (https://github.com/cubiclesoft/ultimate-web-scraper). I tried a code example from another post from here (PHP simple web socket client) but I'm not able to get this number. Is it imposible to get this number (with php)?
This is what I tried so far with the help of "ultimate-web-scraper" (https://kleesit.de/beta/livetime/webscraper/test.php):
<?php
require_once "support/web_browser.php";
require_once "support/tag_filter.php";
// Retrieve the standard HTML parsing array for later use.
$htmloptions = TagFilter::GetHTMLOptions();
// Retrieve a URL (emulating Firefox by default).
$url = "https://aircrasher.livefpv.com/live/scoring/";
$web = new WebBrowser();
$result = $web->Process($url);
// Check for connectivity and response errors.
if (!$result["success"])
{
echo "Error retrieving URL. " . $result["error"] . "\n";
exit();
}
if ($result["response"]["code"] != 200)
{
echo "Error retrieving URL. Server returned: " . $result["response"]["code"] . " " . $result["response"]["meaning"] . "\n";
exit();
}
// Get the final URL after redirects.
$baseurl = $result["url"];
// Use TagFilter to parse the content.
$html = TagFilter::Explode($result["body"], $htmloptions);
// Find all anchor tags inside a div with a specific class.
// A useful CSS selector cheat sheet: https://gist.github.com/magicznyleszek/809a69dd05e1d5f12d01
echo "All the URLs:\n";
$result2 = $html->Find("div.someclass a[href]");
if (!$result2["success"])
{
echo "Error parsing/finding URLs. " . $result2["error"] . "\n";
exit();
}
foreach ($result2["ids"] as $id)
{
// Faster direct access.
echo "\t" . $html->nodes[$id]["attrs"]["href"] . "\n";
echo "\t" . HTTP::ConvertRelativeToAbsoluteURL($baseurl, $html->nodes[$id]["attrs"]["href"]) . "\n";
}
// Find all table rows that have 'th' tags.
// The 'tr' tag IDs are returned.
$result2 = $html->Filter($html->Find("tr"), "th");
if (!$result2["success"])
{
echo "Error parsing/finding table rows. " . $result2["error"] . "\n";
exit();
}
foreach ($result2["ids"] as $id)
{
echo "\t" . $html->GetOuterHTML($id) . "\n\n";
}
?>
This is what I tried with CURL and the help of a stackoverflow post (https://kleesit.de/beta/livetime/test2.php):
<?php
// random 7 chars digit/alphabet for "t" param
$random = substr(md5(mt_rand()), 0, 7);
$socket_server='https://aircrasher.livefpv.com/live/scoring/';
// create curl resource
$ch = curl_init();
//N°1 GET request
curl_setopt_array(
$ch,
[
CURLOPT_URL => $socket_server,
CURLOPT_RETURNTRANSFER => TRUE,
// since it is TRUE by default we should disable it
// on our localhost, but from real https server it will work with TRUE
CURLOPT_SSL_VERIFYPEER => FALSE
]);
// $output contains the output string
$output = curl_exec($ch);
$output=substr($output, 1);
echo $socket_server;
// server response in my case was looking like that:
// '{"sid":"4liJK2jWEwmTykwjAAAR","upgrades":["websocket"],"pingInterval":25000,"pingTimeout":20000,"maxPayload":1000000}'
var_dump($output);
$decod=json_decode($output);
// setting ping Interval accordingly with server response
$pingInterval = $decod->pingInterval;
//N°2 POST request
$socket_server_with_sid = $socket_server.'&sid='.$decod->sid;
curl_setopt_array(
$ch,
[
CURLOPT_URL => $socket_server_with_sid,
CURLOPT_POST => TRUE,
CURLOPT_TIMEOUT_MS => $pingInterval,
// 4 => Engine.IO "message" packet type
// 0 => Socket.IO "CONNECT" packet type
CURLOPT_POSTFIELDS => '40'
]);
$output = curl_exec($ch);
// ok
var_dump($output);
// Pervious 2 requests are called "hand shake" now we can send a message
// N°3 socket.emit
if ($output==='ok') {
curl_setopt_array(
$ch,
[
CURLOPT_URL => $socket_server_with_sid,
CURLOPT_TIMEOUT_MS => $pingInterval,
CURLOPT_POST => TRUE,
// 4 => Engine.IO "message" packet type
// 2 => Engine.IO "EVENT" packet type
CURLOPT_POSTFIELDS => '42["chat message","0devhost message 10"]'
]);
$output = curl_exec($ch);
// ok
echo $output.'<br/>';
echo $socket_server_with_sid;
}
// close curl resource to free up system resources
curl_close($ch);
?>
Why I cannot display an image inside an HTML tag ?.
<?php
session_start();
$ch = curl_init( 'https://mighty-inlet-78383.herokuapp.com/api/hotels/imagedata');
curl_setopt_array($ch, array(
CURLOPT_RETURNTRANSFER => TRUE
));
// Send the request
$response = curl_exec($ch);
// Check for errors
if($response === FALSE){
die(curl_error($ch));
echo 'No responce';
}
// Decode the response
$responseData = json_decode($response, true);
// get data from api to json
foreach($responseData AS $response) {
$hotelImage = $response['hotelImage'];
$imgdecord = base64_decode($hotelImage). "\n";
if(empty($email)){
echo "No responce from server";
}
else{
echo '<br>';
echo'
<html>
<head></head>
<body>
<img src="data:image/png;base64,' . $imgdecord . '" />
</body>
</html>
';
}
}
?>
It shows below result in the browser
���z�C�ZM�]y�eb��3K�� �� �<~_^%�HW � t}��-l��)Z/�2H����.r�M���g�q��kޭ��q�і��/�P���W��.�b�$/�a������O,U�S����5��\mr��8�ӓPs|�C|� u��W~l��t�l-��������9����%͓��E��ַ�5�]z�� �O ���`��Tl���GO��oQ�E�����y�o�l�,児
My API returns values like this
[
{
"_id": "5e8f2e81c15e30001791379e",
"name": "harlyRox",
"email": "hishansjc#gmail.com",
"phone_number": 713677319,
"hotelImage": "iVBORw0KGgoAAAANSUhEUgAABVYAAAMACAYAAADPPjzCAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAP+6SURBVHhe7J0FYB3Htf6P0GLJkm1JlkySGWLG2I7DDG3StEnhFd//vcJrX5k5hddX5vaVU06TtkmaNE2TNszMTHYSc8ys
},..
]
it works when I remove this line
$imgdecord = base64_decode($hotelImage). "\n";
and change this HTML tag to
<img src="data:image/png;base64,' . $hotelImage . '" />
as #LawrenceCherone mentioned above
I am trying to fetch JSON data from a remote server using curl and save it in the database. The data on the remote server is updated each minute.
The curl script is running fine with fetching the data. But it is fetching data around every 10 seconds. What I want is it fetches the data once in a minute.
I searched SO and found that cronjob and windows task scheduler is the solutions provided for the scheduled jobs using curl. My problem is I need not run the web page containing curl so it may start fetching data from the server but I have to read data once in a minute.
Because the data is updated each minute's 5th second like 9:10:05 -> 9:11:05 ->9:12:05 ->... One option is to use cronjob OR windows scheduler to run the web page at e.g. 3rd second and close at e.g. 7th second so each minute one value is read. But this doesn't seem a good or reliable solution for me or is it?
My question is how do I read the data from the remote server once in a minute using curl. Is there some programming solution or using cronjob/windows task scheduler is the choice?
I am doing development on Windows workstation but I need to deploy it on server running Ubuntu.
EDIT
I have 2 more questions about my code:
1) As I stated in OP above, the curl is fetching data constantly from the source like it is an infinite loop running. Is this default behavior of curl or I need to do some configuration to fetch the data e.g. once through it ?
2) I need to keep the browser window opened so the curl start fetching the data. It doesn't work if the webpage is not opened. Is it default behavior of curl or it can also run as backend job so the webpage containing it should not be kept opened all the time ?
Here is my code:
Index.PhP
HTML + PHP + JS for data input and display
Connect.PhP
Database connection
RequestData.PhP
<!DOCTYPE html>
<html lang="en">
<head>
<title>Weather Data</title>
<meta charset="utf-8">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
</head>
<body>
<h2>Connect.php</h2>
<div>
<?php
require("Connection.php");
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://cors.io/?http://api.holfuy.com/live/?s=759&pw=h1u5l4kka&m=JSON&tu=C&su=m/s",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"Cache-Control: no-cache"
),
));
$response = curl_exec($curl);
$Wdata = json_decode($response, true);
$tem = $Wdata["temperature"];
$hum = $Wdata["humidity"];
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
// Here code to save $response into database;
echo $response;
echo "<br>";
echo $tem;
echo "<br>";
echo $hum;
}
try{
$sql = "INSERT INTO weatherdata (humidity, temprature)
VALUES ('".$tem."','".$hum."')";
echo "<meta http-equiv='refresh' content='0'>";
($conn->query($sql));
//$conn = null;
}
catch(PDOException $e)
{
}
?>
</div>
</body>
</html>
I would say using cronjob is the best way to go about doing this! (Assuming you are using a linux distro)
In cron you select the interval, but if you would like you could use a sleep function to delay the command being run several seconds:
*/1 * * * * sleep 5; mycommand
^^ ^^
every minute five seconds
EDIT: Your post doesn't seem really clear on what kind of Operating System you are running, I suggested using cronjob in my post but this is only available to linux systems. Windows task scheduler could be used if you are running it from a windows system although I am not sure how this one works so I wont be able to help you set it up.
If you don't want to use cron job and if you want to use this page directly by browser then use this code .
use meta tag for refresh after five minute using find content = 300
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="refresh" content="300">
<title>Weather Data</title>
<meta charset="utf-8">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js">
</script>
</head>
<body>
<h2>Connect.php</h2>
<div>
<?php
require("Connection.php");
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://cors.io/?http://api.holfuy.com/live/?s=759&pw=h1u5l4kka&m=JSON&tu=C&su=m/s",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"Cache-Control: no-cache"
),
));
$response = curl_exec($curl);
$Wdata = json_decode($response, true);
$tem = $Wdata["temperature"];
$hum = $Wdata["humidity"];
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
// Here code to save $response into database;
echo $response;
echo "<br>";
echo $tem;
echo "<br>";
echo $hum;
}
try{
$sql = "INSERT INTO weatherdata (humidity, temprature)
VALUES ('".$tem."','".$hum."')";
echo "<meta http-equiv='refresh' content='0'>";
($conn->query($sql));
//$conn = null;
}
catch(PDOException $e)
{
}
?>
</div>
</body>
</html>
when i execute the below code to access koinex ticker api , I am getting an array but how can i display only certain values :
echo $response[0][0][1];
echo $response['prices']['BTC'][0];
code :
<?php
$getCurrency = "inr";
$displayArrayOutput = true;
// Get cURL resource
$curl = curl_init();
// Set some options - we are passing in a useragent too here
curl_setopt_array($curl, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'https://koinex.in/api/ticker',
CURLOPT_USERAGENT => 'Something here'
));
// Send the request & save response to $resp
$response = curl_exec($curl);
// Close request to clear up some resources
$err = curl_error($curl);
curl_close($curl);
if ($err) {
} else {
if($displayArrayOutput){
$response = json_decode($response, true);
print_r($response);
}
else{
header("Content-type:application/json");
echo 'touine';
}
}
echo $response[0][0][1];
echo $response['prices']['BTC'][0];
?>
sorry a little new to php , open to any other kind of approach for decoding json
I have a json output below that am trying to get its contents
{"content":"people",
"nextLink":"https://example.com/people?$skip=3",
"value":[
{
"id":"100","displayName":"Room Rainier",
"Addresses":[{"location":"12 orlando street","Spore":8.0}],
"phones":[{"type":"home","number":"10000000000"}],
"personType":{"class":"new","subclass":"Room1"}
},
{
"id":"102","displayName":"Tony Blur",
"Addresses":[{"location":"19 saco street","Spore":4.0}],
"phones":[{"type":"business","number":"1080000000"}],
"personType":{"class":"Other","subclass":"Room2"}
},
{
"id":"103","displayName":"veronica huges",
"Addresses":[{"location":"6 nano street","Spore":7.0}],
"phones":[{"type":"business","number":"111000000"}],
"personType":{"class":"old","subclass":"Room5"}
}]}
Below is my working Json Curl in PHP
<?php
session_start();
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://example.com/people",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
//CURLOPT_CUSTOMREQUEST => "GET",
//CURLOPT_POSTFIELDS => "$data",
CURLOPT_HTTPHEADER => array(
"authorization: Bearer --my bearer goes here--"
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
$res=json_decode($response);
$res1=json_decode($response, true);
$js = json_decode($response, true);
echo '<pre>' . print_r($response, true) . '</pre>';
$re = $js['value'];
foreach ($re as $value1) {
echo $value1['id'];
echo '<br>';
echo $value1['Addresses']['location'];
echo '<br>';
echo $value1['phones']['type'];
echo '<br>';
}
if ($err) {
echo "cURL Error #:" . $err;
} else {
//echo $response;
}
?>
my success
I was able to get all values for id **(echo $value1['id'])** in for each loops.
my Problem
1.) I cannot get values for Addresses Locations **(echo $value1['Addresses']['location'])** and phones Types **(echo $value1['phones']['type'])** in for each loop respectively as it shows nothing.
2.) The json file has a next link options hence
"nextLink":"https://example.com/people?$skip=3"
how can I display more users/peoples data based on the link. Thanks
The Addresses and Phones are Arrays so if you want to get the first you can use index 0 or you can get all with loop in case there is more than 1 address or phone.
foreach ($re as $value1) {
/**
* To get the first address and phone use index 0
*/
echo $value1['Addresses'][0]['location'];
echo $value1['phones'][0]['type'];
/**
* Loop to get all addresses and phones
*/
foreach($value1['Addresses'] as $address) {
echo $address['location'];
}
foreach($value1['phones'] as $phone) {
echo $phone['type'];
}
}
If there is a next link you should request the nextLink just use curl again with the next link.
It will be easier if you create a function for this and just call the function again with the next link.