I wrote a function that uses YQL to pull stock info as follows:
function quote_func() {
$BASE_URL = "https://query.yahooapis.com/v1/public/yql";
// Form YQL query and build URI to YQL Web service
$yql_query = "select * from yahoo.finance.quotes where symbol in ('AAPL')";
$yql_query_url = $BASE_URL . "?q=" . urlencode($yql_query) . "&format=json&env=http://datatables.org/alltables.env&callback=";
// Make call with cURL
$session = curl_init($yql_query_url);
curl_setopt($session, CURLOPT_RETURNTRANSFER,true);
$json = curl_exec($session);
// Convert JSON to PHP object
$phpObj = json_decode($json);
// Confirm that results were returned before parsing
if(!is_null($phpObj->query->results)){
$quote = $phpObj->query->results->quote;
return $quote;
}
}
I then display the information I want using:
<?php echo quote_func()->DATA-I-WANT; ?>
I know that this is bad because each time I use the function I'm executing a GET request.
Can someone point me in the right direction at least?
I'd suggest that you cache the result. You could store the data in a database but the best place is probably a proper cache like memcached, APC or even just the filesystem.
Here's a dead simple filesystem implementation (untested and just for illustration).
$cacheFile = '/tmp/myCache.txt';
$expirePeriod = 1800; // in seconds
if (file_exists($cacheFile) && time() - filemtime() < $expirePeriod) {
$quoteData = file_get_contents($cacheFile);
} else {
$quoteData = quote_func();
file_put_contents($cacheFile, $quoteData);
}
Related
I previously had a Google geocoding script working to extract longitude and latitude using local addresses in a database.
In the last 6 months I've switched hosts, and apparently Google has implemented a new forward geocoder. Now it just returns the url not loading error from the xml script call.
I've tried everything to get my code working. Even sample coding from other websites won't work on my server. What am I missing? Is there possibly a server side setting that is blocking this from executing properly?
Attempt # 1:
$request_url = "http://maps.googleapis.com/maps/api/geocode/xml?new_forward_geocoder=true&address=1600+Amphitheatre+Parkway,+Mountain+View,+CA";
echo $request_url;
$xml = simplexml_load_file($request_url) or die("url not loading");
$status = $xml->status;
return $status;
Simply returns url not loading. I have tried with and without the new_forwad_geocoder. I have also tried with and without https.
The $request_url string DOES return proper results if you simply copy and paste it to a browser.
Also tried this just to see if I could get a file to return. Attempt 2:
$request_url = "http://maps.googleapis.com/maps/api/geocode/json?new_forward_geocoder=true&address=1600+Amphitheatre+Parkway,+Mountain+View,+CA";//&sensor=true
echo $request_url."<br>";
$tmp = file_get_contents($request_url);
echo $tmp;
Any idea what could be causing the connection failure?
I wasn't ever able to get this working with XML again and the file_get_contents call was the culprit I'm almost positive.
I've posted what I did get to work with JSON/Curl (below) in case anyone has similar issues.
Ultimately I think the problems I ran into had to do with an upgrade to our Apache version on the server; and some of the default settings related to file_get_contents and fopen being more restrictive. I haven't confirmed this though.
This code does work though:
class geocoder{
static private $url = "http://maps.google.com/maps/api/geocode/json?sensor=false&address=";
static public function getLocation($address){
$url = self::$url.$address;
$resp_json = self::curl_file_get_contents($url);
$resp = json_decode($resp_json, true);
//var_dump($resp);
if($resp['status']='OK'){
//var_dump($resp['results'][0]['geometry']['location']);
//echo "<br>";
//var_dump($resp['results'][0]['geometry']['location_type']);
//echo "<br>";
//var_dump($resp['results'][0]['place_id']);
return array ($resp['results'][0]['geometry']['location'], $resp['results'][0]['geometry']['location_type'], $resp['results'][0]['place_id']);
}else{
return false;
}
}
static private function curl_file_get_contents($URL){
$c = curl_init();
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_URL, $URL);
$contents = curl_exec($c);
curl_close($c);
if ($contents) return $contents;
else return FALSE;
}
}
$Address = "1600 Amphitheatre Parkway, Mountain View, CA";
$Address = urlencode(trim($Address));
list ($loc, $type, $place_id) = geocoder::getLocation($Address);
//var_dump($loc);
$lat = $loc["lat"];
$lng = $loc["lng"];
echo "<br><br> Address: ".$Address;
echo "<br>Lat: ".$lat;
echo "<br>Lon: ".$lng;
echo "<br>Location: ".$type;
echo "<br>Place ID: ".$place_id;
I am trying to import yahoo contacts with help of YQL but stuck with a problem.
I am using approach on this link https://developer.yahoo.com/yql/guide/yql-code-examples.html#yql_php
But I got this error:
{"error":{"lang":"en-US","description":"Authentication Error. The table social.contacts requires a higher security level than is provided, you provided ANY but at least USER is expected"}}
Do I am missing something
I have run this query on yql console its showing result but nt getting results directly hitting url or with curl.
My code is :
$BASE_URL = "https://query.yahooapis.com/v1/public/yql";
// Form YQL query and build URI to YQL Web service
$yql_query = "select * from social.contacts(0, 500) where guid=me";
$yql_query_url = $BASE_URL . "?q=" . urlencode($yql_query) . "&format=json";
// Make call with cURL
$session = curl_init($yql_query_url);
curl_setopt($session, CURLOPT_RETURNTRANSFER,true);
$json = curl_exec($session);
// Convert JSON to PHP object
$phpObj = json_decode($json);
// Confirm that results were returned before parsing
if(!is_null($phpObj->query->results)){
// Parse results and extract data to display
foreach($phpObj->query->results->event as $event){
$events .= "<div><h2>" . $event->name . "</h2><p>";
$events .= html_entity_decode(wordwrap($event->description, 80, "<br/>"));
$events .="</p><br/>$event->venue_name<br/>$event->venue_address<br/>";
$events .="$event->venue_city, $event->venue_state_name";
$events .="<p><a href=$event->ticket_url>Buy Tickets</a></p></div>";
}
}
// No results were returned
if(empty($events)){
$events = "Sorry, no events matching $query in $location";
}
// Display results and unset the global array $_GET
echo $events;
unset($_GET);
New to PHP, so bear with me...
I'm trying to send/make available the output variables from this simpleXml parser script to this other PHP file, which is supposed to send data to Brightcove's Media API.
Sending Script:
<?php
session_name("FeedParse");
session_start();
$_SESSION['bcName'] = $title;
$_SESSION['shortDescription'] = $description;
$_SESSION['remoteUrl'] = $videoFile;
$html = "";
$url = "http://feeds.nascar.com/feeds/video?command=search_videos&media_delivery=http&custom_fields=adtitle%2cfranchise&page_size=100&sort_by=PUBLISH_DATE:DESC&token=217e0d96-bd4a-4451-88ec-404debfaf425&any=franchise:%20Preview%20Show&any=franchise:%20Weekend%20Top%205&any=franchise:Up%20to%20Speed&any=franchise:Press%20Pass&any=franchise:Sprint%20Cup%20Practice%20Clips&any=franchise:Sprint%20Cup%20Highlights&any=franchise:Sprint%20Cup%20Final%20Laps&any=franchise:Sprint%20Cup%20Victory%20Lane&any=franchise:Sprint%20Cup%20Post%20Race%20Reactions&any=franchise:All%20Access&any=franchise:Nationwide%20Series%20Qualifying%20Clips&any=franchise:Nationwide%20Series%20Highlights&any=franchise:Nationwide%20Series%20Final%20Laps&any=franchise:Nationwide%20Series%20Victory%20Lane&any=franchise:Nationwide%20Series%20Post%20Race%20Reactions&any=franchise:Truck%20Series%20Qualifying%20Clips&any=franchise:Truck%20Series%20Highlights&any=franchise:Truck%20Series%20Final%20Laps&any=franchise:Truck%20Series%20Victory%20Lane&any=franchise:Truck%20Series%20Post%20Race%20Reactions&output=mrss";
$xml = simplexml_load_file($url);
$namespaces = $xml->getNamespaces(true); // get namespaces
for($i = 0; $i < 50; $i++){ // will return the 50 most recent videos
$title = $xml->channel->item[$i]->video;
$link = $xml->channel->item[$i]->link;
$title = $xml->channel->item[$i]->title;
$pubDate = $xml->channel->item[$i]->pubDate;
$description = $xml->channel->item[$i]->description;
$titleid = $xml->channel->item[$i]->children($namespaces['bc'])->titleid;
$m_attrs = $xml->channel->item[$i]->children($namespaces['media'])->content[0]->attributes();
$videoFile = $m_attrs["url"];
$html .= //"<h3>$title</h3>$description<p>$pubDate<p>$url<p>Video ID: $titleid<p>
print $title;
print $description;
print $videoFile;
// echo $html;/* tutorial for this script is here https://www.youtube.com/watch?v=4ZLZkdiKGE0 */
}
//http://support.brightcove.com/en/video-cloud/docs/media-write-api-php-example-upload-video
?>
Receiving Script:
<?php
session_start();
$title = $_SESSION['bcName'];
$description = $_SESSION['shortDescription'];
$videoFile = $_SESSION['remoteUrl'];
// Instantiate the Brightcove class
$bc = new Brightcove(
'//readtoken//', //Read Token BC
'//writetoken//' //Write Token BC
);
// Set the data for the new video DTO using the form values
$metaData = array(
'$title' => $_POST['bcName'],
'$description' => $_POST['bcShortDescription'],
);
//changed all the code below to what i think works for remoteUrl and URLs as opposed to actual video files
// Rename the file to its original file name (instead of temp names like "a445ertd3")
$url = $_URL['remoteUrl'];
//rename($url['tmp_name'], '/tmp/' . $url['name']);
//$url = '/tmp/' . $url['name'];
// Send the file to Brightcove
//Actually, this has been changed to send URL to BC, not file
echo $bc->createVideo($url,$metaData);
class Brightcove {
public $token_read = 'UmILcDyAFKzjtWO90HNzc67X-wLZK_OUEZliwd9b3lZPWosBPgm1AQ..'; //Read Token from USA Today Sports BC
public $token_write = 'svP0oJ8lx3zVkIrMROb6gEkMW6wlX_CK1MoJxTbIajxdn_ElL8MZVg..'; //Write Token from USA Today Sports BC
public $read_url = 'http://api.brightcove.com/services/library?';
public $write_url = 'http://api.brightcove.com/services/post';
public function __construct($token_read, $token_write = NULL ) {
$this->token_read = $token_read;
$this->token_write = $token_write;
}
public function createVideo($url = NULL, $meta) {
$request = array();
$post = array();
$params = array();
$video = array();
foreach($meta as $key => $value) {
$video[$key] = $value;
}
$params['token'] = $this->token_write;
$params['video'] = $video;
$post['method'] = 'create_video';
$post['params'] = $params;
$request['json'] = json_encode($post);
if($file) {
$request['file'] = '#' . $file;
}
// Utilize CURL library to handle HTTP request
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $this->write_url);
curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($curl, CURLOPT_POSTFIELDS, $request);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_VERBOSE, TRUE );
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 300);
curl_setopt($curl, CURLOPT_TIMEOUT, 300);
$response = curl_exec($curl);
curl_close($curl);
// Responses are transfered in JSON, decode into PHP object
$json = json_decode($response);
// Check request error code and re-call createVideo if request
// returned a 213 error. A 213 error occurs when you have
// exceeded your allowed number of concurrent write requests
if(isset($json->error)) {
if($json->error->code == 213) {
return $this->createVideo($url, $meta);
} else {
return FALSE;
}
} else {
return $response;
}
}
}
?>
Did I set up sessions to work correctly here? Any ideas on why the receiving PHP script isn't picking up the data/variables outputted by the PHP feed parser script?
In your sending script, it looks like you're setting the session variables at the beginning of the script and you're expecting them to get updated whenever the local variable changes. This won't happen with the way you've written it.
You could make this happen by assigning the variables by reference, by putting an ampersand (&) before the local variable name, but this can get kinda tricky in some scenarios, and it might be best to skip that headache and instead just update the session variable directly.
However, another issue is that you're attempting to store multiple values (50, as indicated by the code comments) into a scalar session variable. So every time your loop iterates, it would overwrite the previous value. Perhaps what would be better would be to use an array structure:
<?php
session_name("FeedParse");
session_start();
$_SESSION['videos'] = array(); // initialize a session variable 'videos' to be an array
$url = "http://feeds.nascar.com/feeds/video?command=search_videos&media_delivery=http&custom_fields=adtitle%2cfranchise&page_size=100&sort_by=PUBLISH_DATE:DESC&token=217e0d96-bd4a-4451-88ec-404debfaf425&any=franchise:%20Preview%20Show&any=franchise:%20Weekend%20Top%205&any=franchise:Up%20to%20Speed&any=franchise:Press%20Pass&any=franchise:Sprint%20Cup%20Practice%20Clips&any=franchise:Sprint%20Cup%20Highlights&any=franchise:Sprint%20Cup%20Final%20Laps&any=franchise:Sprint%20Cup%20Victory%20Lane&any=franchise:Sprint%20Cup%20Post%20Race%20Reactions&any=franchise:All%20Access&any=franchise:Nationwide%20Series%20Qualifying%20Clips&any=franchise:Nationwide%20Series%20Highlights&any=franchise:Nationwide%20Series%20Final%20Laps&any=franchise:Nationwide%20Series%20Victory%20Lane&any=franchise:Nationwide%20Series%20Post%20Race%20Reactions&any=franchise:Truck%20Series%20Qualifying%20Clips&any=franchise:Truck%20Series%20Highlights&any=franchise:Truck%20Series%20Final%20Laps&any=franchise:Truck%20Series%20Victory%20Lane&any=franchise:Truck%20Series%20Post%20Race%20Reactions&output=mrss";
$xml = simplexml_load_file($url);
$namespaces = $xml->getNamespaces(true); // get namespaces
for($i = 0; $i < 50; $i++){ // will return the 50 most recent videos
$m_attrs = $xml->channel->item[$i]->children($namespaces['media'])->content[0]->attributes();
// on each loop iteration, create a new array structure with the video info in it and
// push it onto the 'video' array session variable
$video = array(
'bcName' => $xml->channel->item[$i]->video,
'shortDescription' => $xml->channel->item[$i]->description,
'remoteUrl' => $m_attrs["url"],
);
$_SESSION['videos'][] = $video;
}
Then, on your receiving script, you'll loop through $_SESSION['videos']:
<?php
session_start();
// Instantiate the Brightcove class
$bc = new Brightcove(
'UmILcDyAFKzjtWO90HNzc67X-wLZK_OUEZliwd9b3lZPWosBPgm1AQ..', //Read Token from USA Today Sports BC
'svP0oJ8lx3zVkIrMROb6gEkMW6wlX_CK1MoJxTbIajxdn_ElL8MZVg..' //Write Token from USA Today Sports BC
foreach ((array)$_SESSION['videos'] as $video) {
$title = $video['bcName'];
$description = $video['shortDescription'];
$videoFile = $video['remoteUrl'];
// The code below this line may need to be adjusted. It does not seem quite right.
// Are you actually sending anything to this script via POST? Or should those just
// be the values we set above?
// What is $_URL? Should that just be the $videoFile value?
// Set the data for the new video DTO using the form values
$metaData = array(
'$title' => $_POST['bcName'],
'$description' => $_POST['bcShortDescription'],
);
//changed all the code below to what i think works for remoteUrl and URLs as opposed to actual video files
// Rename the file to its original file name (instead of temp names like "a445ertd3")
$url = $_URL['remoteUrl'];
//rename($url['tmp_name'], '/tmp/' . $url['name']);
//$url = '/tmp/' . $url['name'];
// Send the file to Brightcove
//Actually, this has been changed to send URL to BC, not file
echo $bc->createVideo($url,$metaData);
}
IMPORTANT NOTE:
Keep in mind that this will call the API once for each video in the session (sounds like up to 50 each time). So you'll be making 50 cURL requests on each run of this script. That seems a bit heavy, but perhaps that's expected. It would be worth investigating if their API allows you to compile the data into one call and send it all up at once, as opposed to connecting, sending the data, parsing the response, and disconnecting, 50 times.
I have been looking around for a solution to cache multiple JSON Objects into a single file on my server (without a database). The reason for this is because I am developing a site which has multiple JSON requests from users YouTube channels and the page takes a while to load because of this.
A short section of my PHP File which makes the JSON Requests from YouTube (there are a lot more JSON requests in this json-yt.php file:
// VNM Jar
$realUserName = 'TheEnijar';
$data = file_get_contents('http://gdata.youtube.com/feeds/api/users/' . $realUserName . '?v=2&alt=json');
$data = json_decode($data, true);
$TheEnijar = $data['entry']['yt$statistics'];
// VNM Jinxed
$realUserName = 'OhhJinxed';
$data = file_get_contents('http://gdata.youtube.com/feeds/api/users/' . $realUserName . '?v=2&alt=json');
$data = json_decode($data, true);
$OhhJinxed = $data['entry']['yt$statistics'];
// VNM Pin
$realUserName = 'ImGreenii';
$data = file_get_contents('http://gdata.youtube.com/feeds/api/users/' . $realUserName . '?v=2&alt=json');
$data = json_decode($data, true);
$ImGreenii = $data['entry']['yt$statistics'];
// VNM Zq
$realUserName = 'Zqonalized';
$data = file_get_contents('http://gdata.youtube.com/feeds/api/users/' . $realUserName . '?v=2&alt=json');
$data = json_decode($data, true);
$Zqonalized = $data['entry']['yt$statistics'];
Is this even possible, and if so, can anybody point me in the right direction or provide me with a solution to storing JSON requests prior to the user loading the page and JSON making the request to all the different YouTube channels.
To be honest, there is a LOT of different ways to do this. The simplest is to make an assoc array of your data, serialize it and write to the file.
$dataArray = array();
// ...
$data = file_get_contents('http://gdata.youtube.com/feeds/api/users/' . $realUserName . '?v=2&alt=json');
$dataArray['jinxed'] = $data;
$data = serialize($data);
file_put_contents('cache.txt', $data);
Than pull it back:
$data = unserialize(file_get_contents('cache.txt'));
But, as I said, there is a lot of another ways to cache requests. And chosing one of them depends on what else you need from your caching system.
UPD: Don't forget that cached info will become outdated sometime, so you must to update your cache every certain period. To not saving any additional values (like cached time) use filemtime() function for your cache file.
you can do something like:
<?php
function getCachableContent($url){
$hash = md5($url);
$cacheFile = "/tmp/foo-cache/$hash";
if ( file_exists($cacheFile) and filemtime($cacheFile) < time() - 300 ) {
$data = file_get_contents($cacheFile);
} else {
$data = file_get_contents($url);
file_put_contents($cacheFile, $data);
}
return json_decode($data);
}
$data = getCachableContent('http://gdata.youtube.com/feeds/api/users/' . $realUserName . '?v=2&alt=json');
but I think it's better to use solutions like redis that provides expire time. also you can create a ramdisk for temp directory (if you're using linux) which makes it faster.
(I'm scraping this stuff with the permission of the website in question, by the way).
Pretty simple web scraper, was working fine when I was loading all the links by hand, but when I've tried to load them in via JSON and variables (so I can do lots of scraping with the one script and make the process more modular by just adding more links to JSON) it runs on an infinite loop.
(Page has been loading for about 15 minutes now)
Here is my JSON. Only one store is in there for testing purposes but there is going to be about 15 more.
[
{
"store":"Incu Men",
"cat":"Accessories",
"general_cat":"Accessories",
"spec_cat":"accessories",
"url":"http://www.incuclothing.com/shop-men/accessories/",
"baseurl":"http://www.incuclothing.com",
"next_select":"a.next",
"prod_name_select":".infobox .fn",
"label_name_select":".infobox .brand",
"desc_select":".infobox .description",
"price_select":"#price",
"mainImg_select":"",
"more_imgs":".product-images",
"product_url":".hproduct .photo-link"
}
]
Here is the PHP scraper code:
<?php
//Set infinite time limit
set_time_limit (0);
// Include simple html dom
include('simple_html_dom.php');
// Defining the basic cURL function
function curl($url) {
$ch = curl_init();
// Initialising cURL
curl_setopt($ch, CURLOPT_URL, $url);
// Setting cURL's URL option with the $url variable passed into the function
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
// Setting cURL's option to return the webpage data
$data = curl_exec($ch);
// Executing the cURL request and assigning the returned data to the $data variable
curl_close($ch);
// Closing cURL
return $data;
// Returning the data from the function
}
function getLinks($catURL, $prodURL, $baseURL, $next_select) {
$urls = array();
while($catURL) {
echo "Indexing: $url" . PHP_EOL;
$html = str_get_html(curl($catURL));
foreach ($html->find($prodURL) as $el) {
$urls[] = $baseURL . $el->href;
}
$next = $html->find($next_select, 0);
$url = $next ? $baseURL . $next->href : null;
echo "Results: $next" . PHP_EOL;
}
return $urls;
}
$string = file_get_contents("jsonWorkers/incuMens.json");
$json_array = json_decode($string,true);
foreach ($json_array as $value){
$baseURL = $value['baseurl'];
$catURL = $value['url'];
$store = $value['store'];
$general_cat = $value['general_cat'];
$spec_cat = $value['spec_cat'];
$next_select = $value['next_select'];
$prod_name = $value['prod_name_select'];
$label_name = $value['label_name_select'];
$description = $value['desc_select'];
$price = $value['price_select'];
$prodURL = $value['product_url'];
if (!is_null($value['mainImg_select'])){
$mainImg = $value['mainImg_select'];
}
$more_imgs = $value['more_imgs'];
$allLinks = getLinks($catURL, $prodURL, $baseURL, $next_select);
}
?>
Any ideas why the script would be running infinitely and not returning anything/stopping/printing anything to screen? I'm just gonna let it run until it stops. When I was doing this by hand it would only take a minute or so, sometimes less, so I'm sure it's a problem with my variables/json but I can't for the life of me see what the issues lie.
Can anyone take a quick look and point me in the right direction?
There is a problem with your while($catURL) loop. What do you want to do ?
Moreover, you can force to display information on your browser with the flush() command.