I am trying to fetch JSON data from a remote server using curl and save it in the database. The data on the remote server is updated each minute.
The curl script is running fine with fetching the data. But it is fetching data around every 10 seconds. What I want is it fetches the data once in a minute.
I searched SO and found that cronjob and windows task scheduler is the solutions provided for the scheduled jobs using curl. My problem is I need not run the web page containing curl so it may start fetching data from the server but I have to read data once in a minute.
Because the data is updated each minute's 5th second like 9:10:05 -> 9:11:05 ->9:12:05 ->... One option is to use cronjob OR windows scheduler to run the web page at e.g. 3rd second and close at e.g. 7th second so each minute one value is read. But this doesn't seem a good or reliable solution for me or is it?
My question is how do I read the data from the remote server once in a minute using curl. Is there some programming solution or using cronjob/windows task scheduler is the choice?
I am doing development on Windows workstation but I need to deploy it on server running Ubuntu.
EDIT
I have 2 more questions about my code:
1) As I stated in OP above, the curl is fetching data constantly from the source like it is an infinite loop running. Is this default behavior of curl or I need to do some configuration to fetch the data e.g. once through it ?
2) I need to keep the browser window opened so the curl start fetching the data. It doesn't work if the webpage is not opened. Is it default behavior of curl or it can also run as backend job so the webpage containing it should not be kept opened all the time ?
Here is my code:
Index.PhP
HTML + PHP + JS for data input and display
Connect.PhP
Database connection
RequestData.PhP
<!DOCTYPE html>
<html lang="en">
<head>
<title>Weather Data</title>
<meta charset="utf-8">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
</head>
<body>
<h2>Connect.php</h2>
<div>
<?php
require("Connection.php");
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://cors.io/?http://api.holfuy.com/live/?s=759&pw=h1u5l4kka&m=JSON&tu=C&su=m/s",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"Cache-Control: no-cache"
),
));
$response = curl_exec($curl);
$Wdata = json_decode($response, true);
$tem = $Wdata["temperature"];
$hum = $Wdata["humidity"];
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
// Here code to save $response into database;
echo $response;
echo "<br>";
echo $tem;
echo "<br>";
echo $hum;
}
try{
$sql = "INSERT INTO weatherdata (humidity, temprature)
VALUES ('".$tem."','".$hum."')";
echo "<meta http-equiv='refresh' content='0'>";
($conn->query($sql));
//$conn = null;
}
catch(PDOException $e)
{
}
?>
</div>
</body>
</html>
I would say using cronjob is the best way to go about doing this! (Assuming you are using a linux distro)
In cron you select the interval, but if you would like you could use a sleep function to delay the command being run several seconds:
*/1 * * * * sleep 5; mycommand
^^ ^^
every minute five seconds
EDIT: Your post doesn't seem really clear on what kind of Operating System you are running, I suggested using cronjob in my post but this is only available to linux systems. Windows task scheduler could be used if you are running it from a windows system although I am not sure how this one works so I wont be able to help you set it up.
If you don't want to use cron job and if you want to use this page directly by browser then use this code .
use meta tag for refresh after five minute using find content = 300
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="refresh" content="300">
<title>Weather Data</title>
<meta charset="utf-8">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js">
</script>
</head>
<body>
<h2>Connect.php</h2>
<div>
<?php
require("Connection.php");
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://cors.io/?http://api.holfuy.com/live/?s=759&pw=h1u5l4kka&m=JSON&tu=C&su=m/s",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"Cache-Control: no-cache"
),
));
$response = curl_exec($curl);
$Wdata = json_decode($response, true);
$tem = $Wdata["temperature"];
$hum = $Wdata["humidity"];
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
// Here code to save $response into database;
echo $response;
echo "<br>";
echo $tem;
echo "<br>";
echo $hum;
}
try{
$sql = "INSERT INTO weatherdata (humidity, temprature)
VALUES ('".$tem."','".$hum."')";
echo "<meta http-equiv='refresh' content='0'>";
($conn->query($sql));
//$conn = null;
}
catch(PDOException $e)
{
}
?>
</div>
</body>
</html>
Related
I am trying to send curl request from source to destination in loop. Loop runs for 2 times. First request lasts for 32 seconds and second one for 50 seconds. Finally times out. Controlling timeout is not in my control as it is shared hosting.
Source section below is being run in the browser. the below error message shows after using up 120 seconds
Error Details: Fatal error: Maximum execution time of 120 seconds
exceeded
Question
I am assuming that the request should not timeout, since both requests are submitted separately through their own curl request. Still, it seems like it is getting consolidated to form total one request.
In case I run the loop for one time, then everything works as it takes 30 seconds.
Am I missing anything?
Source
for($i = 0; $i <= 200; $i+= 100) {
$postData = array(
'start' => $i,
'end' => $i + 100
);
$ch = curl_init('Server url');
curl_setopt_array($ch, array(
CURLOPT_POST => TRUE,
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_HTTPHEADER => array(
'Content-Type: application/json'
),
CURLOPT_POSTFIELDS => json_encode($postData)
));
$response = curl_exec($ch);
$responseData = json_decode($response, TRUE);
curl_close($ch);
echo $response;
}
Destination
public function methodname()
{
$json = json_decode(file_get_contents('php://input'), true);
// .
// .
// Logic that runs for 32 seconds
// .
// .
header('Content-type: application/json');
echo json_encode("message");
}
Try to add a sleep(1) function inner your loop. It could be that the server which you are requested dont like multiple POST request in a short time.
try using cURl's CURLOPT_TIMEOUT or similar configurations. More information https://www.php.net/manual/en/function.curl-setopt.php here
Answer: read the documentation
LE:
You could also use set_time_limit(0); // or value > 120 to increase your script execution timeout
eBay is making changes to their APIs, shutting down the finding API at the end of the year. I'm using a very simple application on my WordPress site that displays products based on a specific query I hard code. I am able to replicate my page using the Browse API, but I'm really struggling with the Oauth part of things. From what I understand, the Browse get requests only require an application access token (not a user access token). I'm just still struggling to get my head around all of this. I'm trying to add a function that generates an auth token, but it's not working and I don't think I'm calling the function correctly...so looking for help on a few things here. Most importantly--can i do it this way? And am i entering the variables correctly and how exactly do i call the Oauth function?
<?php
/* Template Name: XXXX */
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);
function getOAuthCreds() {
$endpoint = 'https://api.ebay.com/identity/v1/oauth2/token';
$request = "grant_type=client_credentials";
$request .= "scope=https://api.ebay.com/oauth/api_scope";
$session = curl_init($endpoint);
curl_setopt($session, CURLOPT_POST, true);
curl_setopt($session, CURLOPT_POSTFIELDS, $request);
curl_setopt($session, CURLOPT_RETURNTRANSFER, true);
$headers = [
'Content-Type: application/json',
'Authorization = Bearer CODE HERE// I'm using the auth code generated from the application access token (not sure if thats' right?
];
curl_setopt($session, CURLOPT_HTTPHEADER, $headers);
$response = curl_exec($session);
curl_close($session);
return $response;
}
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://api.ebay.com/buy/browse/v1/item_summary/search?q=iphone&sort=-",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"Content-Type:application/json",
"Authorization:" getOAuthCreds(),///am i calling this correctly?
"X-EBAY-C-MARKETPLACE-ID:EBAY_US",
"X-EBAY-C-ENDUSERCTX:affiliateCampaignId=xx,affiliateReferenceId=xx",
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
//print_r($response);
if ($err) {
if ($debug) echo "cURL Error #:" . $err;
else echo "Oops, something went wrong. Please try again later.";
} else {
//Create an array of objects from the JSON returned by the API
$jsondata = json_decode($response);
$resp = $jsondata->itemSummaries;
//Create a simple grid style for the listings
$pageCSS = "<style>
.netflix-wrapper{
display:grid;
grid-template-columns: 200px 200px 200px;
}
.show-wrapper{padding:10px;}
</style>";
//Create the WordPress page content HTML
$pageHTML="<h2>test</h2>";
$pageHTML.="<div class='test-wrapper'>";
//Loop through the API results
foreach($resp as $item) {
//Put each show into an html structure
// Note: if your theme uses bootstrap use responsive classes here
$pageHTML.="<div class='show-wrapper'>";
//Not all items have a 'poster', so in that case use the img field
$pic = $item->image->imageUrl;
$itemID = $item->legacyItemId;
$link = 'https://www.ebay.com/itm/'.$itemID.'?mkrid=ss-0&siteid=0&mkcid=1&campid=ss&toolid=ss&mkevt=1&customId=ss';
$title = $item->title;
$price = $item->price->value;
$bids = $item->bidCount;
if(empty($bids)){
$bids = 0;
}
// For each SearchResultItem node, build a link and append it to $results
$results .= "<div class=\"item\"><div class=\"ui small image\"><img height=\"200px\" width=\"130px\" src=\"$pic\"></div><div class=\"content\"><div class=\"header\">$title</div><div class=\"meta\" style=\"margin-top:1.1em\"><span class=\"price\"><button class=\"ui teal button\">watch watchers</button></span><div class=\"extra\"><button class=\"ui button\"><b>Current Bids:</b> $bids </button></div><div class=\"extra\"><button class=\"ui orange button\">Current Price: $$price</button></div><div class=\"description\"></div></div></div></div>";
//Show the image first to keep the top edge of the grid level
$pageHTML.="<img style='max-width:166px;float:left;' src='".$pic."' />";
$pageHTML.="<h3>".$item->title."</h3>";
// $pageHTML.="<span>added to netflix ".$showObj->titledate."</span>";
// $pageHTML.="<div style='float:left;'>".$showObj->synopsis."</div>";
$pageHTML.="</div>";
}
$pageHTML.="</div>";
}
?>
<?php get_header(); ?>
<!-- Build the HTML page with values from the call response -->
<html>
<head>
<div class="wrp cnt"><div class="spr"></div>
<section class="bSe fullWidth">
<article><div class="awr lnd">
<title>Most Watched <?php echo $query; ?> on eBay</title>
<style type="text/css">body { font-family: arial,sans-serif;} </style>
<link
rel="stylesheet"
href="//cdn.jsdelivr.net/npm/semantic-ui#2.4.2/dist/semantic.min.css"
/>
</head>
<body>
For reference:
https://developer.ebay.com/api-docs/static/oauth-client-credentials-grant.html
I do not know much about php but gathered from your code there are a few things that are not right.
1. The headers section
In function getOAuthCreds()
Headers should be:
Content-Type = application/x-www-form-urlencoded
Authorization = Basic [B64-encoded_oauth_credentials]
Content-Type = application/x-www-form-urlencoded header. eBay wants you to POST to its server so you will need this. Not sure if curl would automatically add to the header.
Authorization header. From eBay reference above, the header will use Basic authentication scheme. Then followed by the Base64 encoding of your "client_id":"client_secret".
[B64-encoded_oauth_credentials] = Base64Encode(client_id:client_secret)
Sample:
Authorization = Basic Y2xpZW50X2lkOmNsaWVudF9zZWNyZXQ=
In case you are not sure where to get "client_id" and "client_secret". Visit this page.
https://developer.ebay.com/api-docs/static/creating-edp-account.html
If everything is correct, you will receive the following response from eBay
Sample:
{
"access_token": "v^1.1#i^1#p^1#r^0#I^3#f^0#t^H4s ... wu67e3xAhskz4DAAA",
"expires_in": 7200,
"token_type": "Application Access Token"
}
The access_token from the response will be valid for 7200 seconds. Cache and reuse it as the previous comment mentioned. When it is expired, get a new access_token.
2. When calling Browse API
You will use the access_token in the Authorization header (in Browse API call) as follow.
Sample:
Authorization: Bearer v^1.1#i^1#p^1#r^0#I^3#f^0#t^H4s ... wu67e3xAhskz4DAAA
I have a curl script that fetches json data from a server. When I save data into a file, it works fine and retrieves data only when new values are available.
But if I insert an INSERT query to save values into database, it starts fetching redundant values in a loop. Like new data is available each 2 minutes and when I save data into file, it fetches each 2 minute new data only. But I replace query with the file data insert code, it starts looping and fetching redundant data something like each 5 seconds.
How can I correct the script so that it fetches data when newer is available. Like it's doing when I save data into file.
Here is my code:
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="refresh" content="300">
<title>Weather Data</title>
<meta charset="utf-8">
</head>
<body>
<h2>Connect.php</h2>
<div>
<?php
require("Connection.php");
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://cors.io/?http://api.xxxxx.com/live/xxxxxxxxxx=m/s",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
//CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_HTTPHEADER => array(
"Cache-Control: no-cache"
),
));
$response = curl_exec($curl);
$Wdata = json_decode($response, true);
$tem = $Wdata["temperature"];
$tem2 = $Wdata["2nd_temp"];
$hum = $Wdata["humidity"];
$tim = $Wdata["dateTime"];
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
echo "<br>";
echo "temp1:".$tem;
echo "<br>";
echo "humidity:".$hum;
echo "<br>";
echo "time:".$tim;
echo "<br>";
if ( $fl = fopen('weatherData.json','a'))
{ fwrite($fl,"\"Weather Data\": { \"Time\" : \"". $tim . "\" ,"."\"Humidity\" : \"". $hum . "\"}\n" );
//echo $_id.';'.$_time;
fclose($fl); }
// Here code to save $response into database; When I un-comment it, the script starts fetching redundant data in a loop
/*
try{
$sql = "INSERT INTO weatherdata (datetime, temperature, humidity)
VALUES ('".$tim."','".$tem."','".$hum."')";
echo "<meta http-equiv='refresh' content='0'>";
($conn->query($sql));
//$conn = null;
}
catch(PDOException $e)
{
echo $e->getMessage();
}
*/
}
?>
</div>
</body>
</html>
My Controller.
public function showMonthlyReport($site_id, $report_id){
$reports = Report::where('report_id', $report_id)->firstOrFail();
$uptime = ???
return view('records', compact('site_id', 'report_id', 'reports', 'uptime'));
}
And my UptimeRobot.php reference https://uptimerobot.com/api getMonitors()method
<?php
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://api.uptimerobot.com/v2/getMonitors",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => "Your Api Key",
CURLOPT_HTTPHEADER => array(
"cache-control: no-cache",
"content-type: application/x-www-form-urlencoded"
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
$data = json_decode($response);
$custom_uptime = ($data->monitors[0]->custom_uptime_ratio);
$uptime = explode("-",$custom_uptime);
}
?>
ApiCommand.php
public function handle()
{
include(app_path() . '/Includes/DeepCrawl.php');
include(app_path() . '/Includes/Uptime.php');
include(app_path() . '/Includes/HelloAnalytics.php');
$stringData = ApiCommand::drawGraph($uptime, $dates, $users, $otherResultsRows, $array_issues, $array_pages_breakdown, $array_uncrawled_url, $array_non_200_pages, $array_orphaned_pages, $array_non_indexable_pages, $array_crawl_source_gap, $array_https_http);
Storage::disk('local')->put('hello.txt', $stringData);
}
Currently building a laravel web application.
I am just wondering how can i able to gather data from uptimerobot. I'm going to use my controller so I can pass it to my view but I don't know how. I have code below with the curl php type above. Really confused what am I doing new programmer here. Can someone explain if I'm at the right path or is it possible to do in controller. Thanks in advance.
I can suggest slightly different solution:
Extract your curl code in a separate console command and run this command each minute (for example, as a cron job).
The result of the command save to database/file/memory.
In your showMonthlyReport() refer to existing result.
Benefits:
In this way you would not have to wait for your curl result on each showMonthlyReport(). All code will run asynchronously
All errors processing will be in one place
Command is testable
I'm facing a strange problem using cURL with PHP on a Windows server. I have a very basic code:
private function curlConnection($method, $url, $timeout, $charset, array $data = null)
{
if (strtoupper($method) === 'POST') {
$postFields = ($data ? http_build_query($data, '', '&') : "");
$contentLength = "Content-length: " . strlen($postFields);
$methodOptions = array(
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => $postFields,
);
} else {
$contentLength = null;
$methodOptions = array(
CURLOPT_HTTPGET => true
);
}
$options = array(
CURLOPT_HTTPHEADER => array(
"Content-Type: application/x-www-form-urlencoded; charset=" . $charset,
$contentLength,
'lib-description: php:' . PagSeguroLibrary::getVersion(),
'language-engine-description: php:' . PagSeguroLibrary::getPHPVersion()
),
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_CONNECTTIMEOUT => $timeout
);
$options = ($options + $methodOptions);
$curl = curl_init();
curl_setopt_array($curl, $options);
$resp = curl_exec($curl);
$info = curl_getinfo($curl);
$error = curl_errno($curl);
$errorMessage = curl_error($curl);
curl_close($curl);
$this->setStatus((int) $info['http_code']);
$this->setResponse((String) $resp);
if ($error) {
throw new Exception("CURL can't connect: $errorMessage");
} else {
return true;
}
}
The problem is that the first time this script is called, the response is always this: string(22) "SSL connection timeout".
Subsequent calls to the script output the desired result, but, if I wait a couple of minutes before calling the script again, the timeout issue happens again.
So, steps to reproduce the "error":
Call the script -> SSL connection timeout
Call the script again -> works fine
Call the script one more time -> works fine
Call the script n more times -> works fine
Wait 10 minutes
Call the script -> SSL connection timeout
Call the script n more times again -> works fine
If I call any other script the response is immediate, even after a period of inactivity, so this behaviour only happen when cURL is involved.
PHP - 5.2.17
CURL - libcurl/7.16.0 OpenSSL/0.9.8q zlib/1.2.3
The server is running Windows 2012 with IIS 8, latest upgrades, running PHP on FastCGI.
Does anyone have any idea on how I can solve this?
Thanks.
Try using the ca-bundle.crt bundle available here https://raw.githubusercontent.com/bagder/ca-bundle/master/ca-bundle.crt
Upload file
curl_setopt($ch, CURLOPT_CAINFO, "path");
Reference: http://richardwarrender.com/2007/05/the-secret-to-curl-in-php-on-windows
Hope this helps.
Try to check the state of the 'Server' Windows service, if stopped - could be the reason. Have no idea how it is related, but it helped for me with the same issue.