I'm hoping someone can help me here. I'm building a Wordpress plugin that will pull data from an XML feed and store it in a database table. The data includes image, so it is also downloading all of the images into Wordpress's "uploads" folder so that when the data is displayed on the front end of the site it doesn't have to make remote calls to display those images.
When pulling in about 100 or so items from the XML feed, it's ok. But in some cases there may be 500 - 1000 items that need to be pulled in and that is taking a huge amount of time, resulting of course in a 504 gateway timeout error.
Here is the function I'm running:
function add_items_to_database(){
global $wpdb;
$items_table = $wpdb->prefix . "stored_items";
// CREATE ARRAY FROM SUBMITTED ITEM ID'S //
$items_ids = explode(',',$_POST['item_ids']);
// GET EACH ITEMS FROM XML FEED AND STORE THEM IN THE WEBSITE DATABASE //
$allItems = array();
foreach($item_ids as $item_id){
$i=0;
do{
$request_url = 'https://example.com/Item/Rss?ouid='.$item_id.'&pageindex=' . $i . '&searchresultsperpage=thirty';
$results = getItems($request_url);
$xml = simplexml_load_string($results, "SimpleXMLElement", LIBXML_NOCDATA);
$json = json_encode($xml);
$array = json_decode($json,TRUE);
$itemsCount = $array['items']['#attributes']['totalcount'];
foreach($array['entry'] as $item){
$allItems[] = $item['item']['#attributes']['itemid'];
}
$i++;
} while (count($allItems) < $itemsCount);
foreach($allItems as $item_id){
$item = get_item($item_id);
$fields = get_item_fields($item);
$wpdb->insert($items_table,$fields);
}
}
}
So here, "getItems" is another function which is just grabbing an array of all item_ids from the XML feed. Then for each of those id's, "get_item" then grabs all the XML data for that item. "get_item_fields" then assigns each bit of data from the XML feed to a php array variable called $fields and also downloads all of the images and stores their new local URL's in that $fields variable as well. The contents of the $fields variable is then saved into the database.
Now, of course, it appears that it's the downloading of all the image files which is hanging the system up the longest and causing the gateway timeout issue.
After a bit of Googling, there appears to be the suggestion that I might be able to fix this issue by using "curl_multi" to run all of these processes in parallel. I'm really not familiar with curl at all and I'm having a bit of trouble getting my head around it. I'm hoping someone might be able to shed some light on how I might be able to alter the above code to correctly implement the use of "curl_multi" and whether or not that really is the way to go?
Thanks in advance.
The user creates a XML-file and stores it in the database as BLOB. Every XML-file gets its own row.
My script should query the Database and retrieve the XML-file of every row.
Every XML-file should be in an own php object and stored in an array. After that I need to access the DOM of every XML to display the data in an html template.
This is what I have got so far:
$stmt = $pdo->query('SELECT xml FROM table');
foreach ($stmt as $row)
{
echo $row['xml'] . "\n";
}
It will output the information without the xml-tags. I wonder why. The problem is, I somehow need to access the DOM.
I read about
simplexml_load_file
but it needs a filepath, which I don't have because the files are stored as BLOB in the DB.
Thank you!
You can use simplexml_load_string instead...
$stmt = $pdo->query('SELECT xml FROM table');
foreach ($stmt as $row)
{
$data = simplexml_load_string($row['xml']);
echo "<pre>".$data->asXML()."</pre>";
}
The reason why you probably don't see the tags is that you are echoing them out to a HTML page, which tries to interpret them as HTML tags.
Not sure about how your doing your foreach as it would more commonly be written as
while ($row = $stmt->fetch(\PDO::FETCH_ASSOC)) // Add correct retrieval method your after
{
$data = simplexml_load_string($row['xml']);
echo "<pre>".$data->asXML()."</pre>";
}
This depends on the API and if your using your own class methods though.
My goal is to use a playlist ID to get the video ID of each video within that playlist.
I'm having trouble figuring out YouTube's API (I'm a relatively new programmer).
Basically, I want to:
Send an XML request to http://gdata.youtube.com/feeds/api/playlists/ID_OF_PLAYLIST
Using the response (which is the part I don't understand how to navigate), append all the video IDs to an array.
Seems pretty straightforward. I've been grabbing the information from YouTube using simplexml_load_file($url); in case that is important.
Thanks for the help!
Here you go, I made an example:
<?php
$data = file_get_contents('http://gdata.youtube.com/feeds/api/playlists/PL4BC045240D2FB11B/?v=2&alt=json&feature=plcp');
if (!$data)
throw new Exception("Data retrieval failed.");
$dataAsArray = json_decode($data);
$feed = $dataAsArray->feed->entry;
$videoID_array = array();
if(count($feed))
{
foreach($feed as $item)
array_push($videoID_array, $item->{'media$group'}->{'yt$videoid'}->{'$t'});
}
print_r($videoID_array);
?>
Where PL4BC045240D2FB11B of course is the Playlist ID!
Example PHPFiddle
I'm trying to build a little script that would let me do this:
http://example.com/appicons.php?id=284417350
and then display this in plain text
http://a3.mzstatic.com/us/r1000/005/Purple/2c/a0/b7/mzl.msucaqmg.png
This is the API query to get that information (artworkUrl512):
http://ax.itunes.apple.com/WebObjects/MZStoreServices.woa/wa/wsLookup?id=284417350
Any help and example code would be much appreciated!
I am not sure why you have jQuery in your tags, unless you want to make the request dynamically without a page refresh. However you can do this simply in PHP using the following example:
$request = array (
"app_id" => #$_GET["id"]
);
// parse the requests.
if (empty($request["app_id"])) {
// redirects back / displays error
}
else {
$app_uri = "http://ax.itunes.apple.com/WebObjects/MZStoreServices.woa/wa/wsLookup?id=" . $request["app_id"];
$data = file_get_contents ($app_uri);
$json = json_decode (trim($data));
print($json->results[0]->artworkUrl100);
}
$request = file_get_contents($itms_url);
$json = json_decode(trim($request));
echo $json[0]->artworkUrl512;
should work in PHP. Unless of course there is more than one hit to the search. A solution using jQuery is probably not very much more difficult.
I'm trying to create an iOS application which upon loading, will initially connect via HTTP back to a PHP web service which will output data as JSON from a MySQL database. I would then like it to import this data into a local SQLite database within the iOS app. I've already downloaded the JSON-Framework for Objective-C.
My question is two fold.
1) What is the best way to output the JSON from PHP so that I can send multiple database tables in the same JSON file? I have 4 tables of data that I'm trying to send (user, building, room, device).
Here is how I am currently outputting the JSON data:
// Users
$query = "SELECT * from user";
$result = mysql_query($query,$conn) or die('Errant query: '.$query);
$users = array();
if(mysql_num_rows($result)) {
while($user = mysql_fetch_assoc($result)) {
$users[] = array('user'=>$user);
}
}
// Buildings
$query = "SELECT * from building";
$result = mysql_query($query,$conn) or die('Errant query: '.$query);
$buildings = array();
if(mysql_num_rows($result)) {
while($building = mysql_fetch_assoc($result)) {
$buildings[] = array('building'=>$building);
}
}
// Rooms
$query = "SELECT * from room";
$result = mysql_query($query,$conn) or die('Errant query: '.$query);
$rooms = array();
if(mysql_num_rows($result)) {
while($room = mysql_fetch_assoc($result)) {
$rooms[] = array('room'=>$room);
}
}
// Devices
$query = "SELECT * from device";
$result = mysql_query($query,$conn) or die('Errant query: '.$query);
$devices = array();
if(mysql_num_rows($result)) {
while($device = mysql_fetch_assoc($result)) {
$devices[] = array('device'=>$device);
}
}
header('Content-type: application/json');
echo json_encode(array('users'=>$users));
echo json_encode(array('buildings'=>$buildings));
echo json_encode(array('rooms'=>$rooms));
echo json_encode(array('devices'=>$devices));
I fear that this method isn't the right way to send multiple objects.
2) In the iOS app, how can I automatically take this JSON data and insert it into the corresponding local database tables in SQLite?
Thanks for any help.
On 1. Instead of JSOn you could use binary Property lists they are natively implemented on the iPhone and there is a library to turn PHP into binary Plist https://github.com/rodneyrehm/CFPropertyList
There are many benefits to using binary property lists, they are probably 1/5 of the size of JSON, you don't need a external library to parse them, therefore all code is much simpler, etc.
On 2. There is no easy way to take the JSON/Plist structure and insert it to a SQL database, because JSON/Plist allow much more flexibility then SQL tables. So you would have to first create the right tables in your SQLite DB and then use normal INSERT to insert the data one by one into the database exactly like you would do with PHP.
Yeah Luke's recommendation is good but you will be fine with the way you are exporting your tables. You may just have to dig "deeper" into the structure to get what you want - i.e. your output with return a "dictionary of dictionaries of arrays" which will then contain the data for each table.
As for downloading them first:
1) NSURLConnection and its delegate methods - you can send asynchronous request to your webserver to get this file and get notified when the data has been downloaded so the user interface is never blocked in your app.
Here's the documentation with some good examples from Apple: http://developer.apple.com/library/mac/#documentation/Cocoa/Reference/Foundation/Classes/NSURLConnection_Class/Reference/Reference.html
At the end of the download, you will have an NSData object which can then be converted back to a string using NSString *jsonContents = [[NSString alloc] initWithData:jsonData encoding:NSUTF8StringEncoding.
You can then use a JSON parser library - I recommend SBJSON https://github.com/stig/json-framework - which will parse the data and return it as a dictionary or array depending on your structure.
From there you can access your tables and value with valueForKey in dictionaries or objectAtIndex: in arrays and then map it into your chosen local storage, for which I recommend Coredata (or you could use sqlite if you are familiar with it too).
Hope it helps.
Rog
I can't speak to 2), but for 1), I would recommend combining your JSON into a single array. One of the nice things about JSON (and arrays) is the ability to nest elements as deeply as you like.
echo json_encode(array(
'users'=>$users,
'buildings'=>$buildings,
'rooms'=>$rooms,
'devices'=>$devices,
));
What about preparing the SQLite database on the webserver and downloading that to the iOS application? This way, you do the heavy lifting on the PHP side. If the data is relatively static, you can even setup a scheduled task to generate the SQLite database on a regular interval.
We've done this for one of our apps and it worked very well.
One thing to keep in mind then is that you should enable gzip compression on the webserver to minimize the data transfer. Remember that you have to do some extra stuff to use gzip compression with NSURLConnection:
http://www.bigevilempire.com/codelog/entry/nsurlconnection-with-gzip/
you can use REST server and RESTKit.
If you would like a more full-featured solution that what is offered by a standalone parsing library, you may want to take a look at RestKit: http://restkit.org/
The framework wraps the operations of fetching, parsing, and mapping JSON payloads into objects. It can handle deeply nested structures and can map directly back to Core Data for persistence.
At a high level, here's what your fetch & post operations would feel like in RestKit:
- (void)loadObjects {
[[RKObjectManager sharedManager] loadObjectsAtResourcePath:[#"/path/to/stuff.json" delegate:self];
}
- (void)objectLoader:(RKObjectLoader*)loader didLoadObjects:(NSArray*)objects {
NSLog(#"These are my JSON decoded, mapped objects: %#", objects);
// Mutate and PUT the changes back to the server
MyObject* anObject = [objects objectAtIndex:0];
anObject.name = #"This is the new name!";
[[RKObjectManager sharedManager] putObject:anObject delegate:self];
}
The framework takes care of the JSON parsing/encoding on a background thread and let's you declare how attributes in the JSON map to properties on your object. A number of people in the community are working with PHP backends + RestKit with great success.