I am trying to parse a large amount of JSON data generated from a remote web service. The output produced is paginated across 500 URIs and each URI contains 100 JSON objects. I need to match a property in each JSON object, it's DOI (a digital object identifier), against a corresponding field fetched from a local database and then update the record.
The issue I am having is controlling my looping constructs to seek out the matching JSON DOI while making sure that all the data has been parsed.
As you can see I have tried to use a combination of break and continue statements but I am not able to 'move' beyond the first URI.
I later introduced a flag variable to help control the loops without effect.
while($obj = $result->fetch_object()){
for($i=1;$i<=$outputs_json['meta']['response']['total-pages'];$i++){
$url = 'xxxxxxxxxxxxxxx&page%5Bnumber%5D='."$i".'&page%5Bsize%5D=100';
if($outputs = json_decode(file_get_contents($url),true)===false){
}
else{
try{
$outputs = json_decode(file_get_contents($url),true);
$j=0;
do{
$flag = false;
$doi = trim($outputs['data'][$j]['attributes']['identifiers']['dois'][0], '"');
if(!utf8_encode($obj->doi)===$doi) continue;
}else{
$flag = true;
$j++;
}
}while($j!==101);
if($flag===true) break;
} catch(Exception $e) {
}
}
}
}
}
What is the optimal approach that guarantees each JSON object at all URIs is parsed and that CRUD operations are only performed on my database when a fetched record's DOI field matches the DOI property of the incoming JSON data?
I'm not 100% sure I understand every aspect of your question but for me it would make sense to change the order of execution
fetch page from external service
decode json and iterate through all 100 objects
get one DOI
fetch corresponding record from database
change db record
when all json-objects are progressed - fetch next url
repeat until all 100 urls are fetched
I think it's not a good idea to fetch one record from local DB and try to find it in 100 different remote calls - instead it's better to base your workflow/loops on fetched remote data and try to find the corresponding elements in your local DB
If you think that approach will fit your task - I can of course help you with the code :)
Related
As part of a search routine for a specific crop, I'm making two calls to the same database table, merging the results and sending them down the line. The first call looks for data relating to the searched-for crop (e.g. "beans"). The second call looks for data relating to the crop group of that crop (e.g. legumes).
Data returned from the first call will be more relevant/focused than that from the second call. I want to add an identifier to the respective data sets that reflects this so that I can subsequently sort/present the data on the basis of relevance in my Vue component.
The following code extracts the crop-specific information from the database; how can I add/insert/append a new variable (e.g. "relevance" = 1) to each row in $factsheets before I "array_merge" it with the data returned from the crop-group sql call?
(For sake of simplicitly, I've not included the code that determines the crops.id value from the name of the crop entered by the user.)
public function getFactsheets($cropId){
$factsheets = Factsheet::whereIn('crop_id',$cropId)
->join("crop_factsheet as cf","factsheets.id","=","cf.factsheet_id")
->join("crops as crops","crops.id","=","cf.crop_id")
->select('crops.name','title', 'factsheets.id', 'shortdesc', 'shortimg', 'factsheets.slug')
->orderBy('crops.name')
->get()->toArray();
return $factsheets;
}
Thanks, Tom.
If you give & to value so whatever changes will happen to value will directly saved on its address.
foreach($factsheets as $key => &$factsheet){
$factsheet['relevance'] = 1;
}
Working demo.
Here is concise explanation of references in official doc.
you can simply do a foreach
foreach($factsheets as $key => $factsheet){
$factsheet['relevance'] = 1;
}
I'm revamping the packaging screen in our inventory system. The user opens a package and inserts/removes the items in it, and when he presses Save, a json array of all the parts that are in the package is sent to a PHP endpoint which takes care of saving the package in the database.
So far everything is good, I have created my functions to send data between PHP and javascript. However, even though the json array that javascript sends to PHP contains all the info on the products (as javascript needed to retrieve it anyway to fill in the grid in the gui), I still have to validate everything in PHP because I can't be sure that the user didn't tampered with the data from the console before trying to save.
With that said, in PHP, I receive the json array, which I use the IDs to load a list of proper objects into an array. I'm doing this:
$in = str_repeat('?,', count($itemIDs) - 1) . '?';
$sql = "SELECT * FROM tbProduct WHERE nID IN ($in)";
$sttmt = $db->prepare($sql);
$sttmt->execute($itemIDs);
$res = $sttmt->fetchAll(PDO::FETCH_ASSOC);
foreach ($res as $key => $productInfo) {
$prod = new tbProduct($db);
$prod->loadFromArray($productInfo);
}
so far so good, I now have an array of tbProduct, which is my class for the products.
What I now have to do is run a validation on this array of objects to make sure all these objets are in the proper status to be packaged together. These validations include making sure all the products have the same status, that none of the products are assembly parts, that all the products have the same owner, etc. This way, even if the array was tampered with in the browser's console, I'll be sure to use the information from the DB anyway.
So I need a way to validate this. I could just do a foreach, store everything that needs to be checked from the very first item in variables, and just compare each subsequent item to these variables, but I'm sure there are better ways to do this. I need something that will be as efficient as possible. My packages can(and will) contain several hundreds of products, so the solution needs to be fast.
What would be the best way of doing this?
Thank you!
You can do it during the loop that's processing all the data returned by the query. Set a variable to the first object, then you can compare other elements to it.
$res = $sttmt->fetchAll(PDO::FETCH_ASSOC);
$firstprod = new tbProduct($db);
$firstprod->loadFromArray(array_shift($res));
foreach ($res as $key => $productInfo) {
$prod = new tbProduct($db);
$prod->loadFromArray($productInfo);
if ($prod->status != $firstprod->status) {
// report inconsistent status
} elseif ($prod->owner != $firstprod->owner) {
// report inconsistent owner
} elseif ($prod->type != "assembly") {
// report that it must ba an assembly part
} ...
}
I'm developing a tool for a website and I came up with an odd problem, or better, an odd situation.
I'm using the code bellow to retrieve data from the TeamSpeak server. I use this info to build a profile on a user.
$ts3 = TeamSpeak3::factory("serverquery://dadada:dadada#dadada:1234/");
// Get the clients list
$a=$ts3->clientList();
// Get the groups list
$b=$ts3->ServerGroupList();
// Get the channels list
$c=$ts3->channelList();
Now, the odd situation is that the output of this code block:
// Get the clients list
$a=$ts3->clientList();
// Get the groups list
$b=$ts3->ServerGroupList();
// Get the channels list
$c=$ts3->channelList();
echo "<pre>";print_r($a);die();
(Notice the print_r)
Is totally different from the output of this code block:
// Get the clients list
$a=$ts3->clientList();
// Get the groups list
#$b=$ts3->ServerGroupList();
// Get the channels list
#$c=$ts3->channelList();
echo "<pre>";print_r($a);die();
What I mean is, the functions I call after clientList() (which output I store in the variable $a) are changing that variable's contents. This is, they're kind of appending their output to the variable.
I've never learned PHP professionally, I'm just trying it out... Am I missing something about this language that justifies this behavior? If I am, what can I do to stop it?
Thank you all.
You're seeing parts of the "Object" in Object Oriented Programming
$ts3 represents an Object containing all the information needed, along with some methods (or functions) that let you get data from the object. Some of these methods will do different things to the object itself, in order to retrieve additional data needed for a particular method call.
Consider the following simple Object:
Bike
color
gears
function __construct($color, $gears)
this.color = $color; this.gears = $gears
function upgrade()
this.headlight = true; this.gears = 10;
Now, when you first create it, it only has two properties:
$myBike = new Bike('red',5);
// $myBike.color = 'red';
// $myBike.gears = 5;
...but once you upgrade, properties have changed, and new ones are added.
$myBike->upgrade();
// $myBike.color = 'red';
// $myBike.gears = 10;
// $myBike.headlight = true;
Objects usually pass references rather than copying data, in order to save memory.
...but if you want to make sure that you're getting a copy that won't change (i.e. does not use data references to the $ts3 object), clone the variable.
$a = clone($ts3->clientList());
Be warned, this will effectively double the memory and processor usage for that variable.
I am new to php development.
How to display multiple json objects fetched from database? So far I am getting single data. I need to display all inserted data in database in json format.
When I insert first data, I get a response like this. If I insert second data, it should display first and second data and so on in upload_details object but it displaying last inserted data only.
{"code":200,
"message":"The file FileUpload1444329638_li.jpg has been uploaded.",
"upload_details": {"desc":"hi",
"file_name":"abc.com\/FileUpload1444329637_li.jpg"}}
When I insert second data:
{"code":200,
"message":"The file FileUpload1444329638_li.jpg has been uploaded.",
"upload_details": {"desc":"h2",
"file_name":"abc.com\/FileUpload1444329638_li.jpg"}}
Here is my code:
<?php
include 'db_config.php'; //echo "hi";exit;
if($_POST['api_name']=="upload_file"){
if(!empty($_FILES["profile_pic"]["name"])){
$fileName = time().'_'.$_FILES["profile_pic"]["name"];
if (move_uploaded_file($_FILES["profile_pic"]["tmp_name"], "uploads/".$fileName)) {
$sql = "Insert into file_upload(`desc`,`file_name`) values ('".$_POST['desc']."','".$fileName."');";
if($conn->query($sql)){
$response= array('code'=>200,'message'=>"The file ". basename( $_FILES["profile_pic"]["name"]). " has been uploaded.",'upload_details'=>array("desc"=>$_POST['desc'],"file_name"=>$_SERVER['SERVER_NAME'].dirname($_SERVER['SCRIPT_NAME']).$fileName));
//print_r($response);exit;
}else{
$response= array('code'=>500,'message'=>"Error in uploading file");
}
} else {
$response= array('code'=>500,'message'=>"Error in uploading file");
}
}else{
$response= array('code'=>500,'message'=>"Error in uploading file");
}
}elseif ($_POST['api_name']=="get_files"){
$response['code']=200;
$response['file_lists'] = array();
$res = $conn->query("select * from file_upload");
while($row = $res->fetch_object()){
array_push($response['file_lists'],array('desc'=>$row- >desc,'file_path'=>$_SERVER['SERVER_NAME'].dirname($_SERVER['SCRIPT_NAME']).$row ->file_name));
}
}
echo json_encode($response);exit;
I expect to get response as data from all which is inserted in table, like:
{"code":200,
"message":"The file has been uploaded.",
"upload_details": {"desc":"hi",
" file_name":"abc.com\/FileUpload1444329637_li.jpg"},
{"desc":"hi2",
"file_name":"abc.com\/FileUpload1444329638_li.jpg"}
}
First, your script currently only has access to the data being inserted for that POST. Your code does nothing at all to query all records that currently exist in file_upload table. Without this, I don't know how you expect these other records to magically be returned in the response.
Second, it would be VERY atypical to tie a single insert operation to a full listing of all records in the table into which the insert was made. This seems like a problematic approach in that you slow down the process of returning a successful insert message to the caller while you try to query the full table. Normally, if one wanted a full listing of records, one might expect to make a GET request against an API specifically designed to do this rather than make a POST (insert) to get this information. Additionally, it seems very odd to intentionally design an API that will get slower and slower over time as you add more records and have a larger payload to return to the caller. The problem is compounded by the fact you are trying to json_encode the data structure. This means that your script will actually continue to take more and more memory to execute with each call to the insert API (as you have to hold the entire data structure for all records in memory to encode it). This means that, at a server level, calls to the API will continue to take a greater percentage of system resources with each passing request, perhaps causing you to need to scale hardware just to meet this use case. You should REALLY, REALLY, REALLY (is that enough REALLY's?) reconsider this requirement.
Third, You have significant SQL injection vulnerability. You should look into using prepared statements and/or sanitizing/validating the user input before making the insert.
Finally, the response format you propose is not valid JSON. I would think that, if you decided you REALLY want to return all records for each successful insert, you would want a format like:
{
"code":200,
"message":"The file has been uploaded.",
"upload_details": [
{
"desc":"hi",
"file_name":"abc.com\/FileUpload1444329637_li.jpg"
},
{
"desc":"hi2",
"file_name":"abc.com\/FileUpload1444329638_li.jpg"}
}
]
}
Note the array wrapper around the two returned records.
Good morning.
I'm currently trying to build a very basic caching system for one of my scripts. The cache is JSON data and contains only 1 key and it's value, but many individual fields, something like this;
{"Item1":"Item1 Description"}
{"Item2":"Item2 Description"}
{"Item3":"Item3 Description"}
What I'm intending to do is;
First check if a cache file is available
Then check if an item exists in the cache
Then add the new item along with it's description if it's not already in the cache...
...or return the item description if it's not there.
All data being stored is strings. The cache file doesn't store any other type of data.
I've put together a basic function but I'm having trouble getting it functioning;
function ItemIsInCache($CacheFile, $ItemId) {
if(file_exists($CacheFile)) {
$json = json_decode(file_get_contents($CacheFile, true));
if(in_array($ItemId, $json)) { // <<
$itemname = array_search($ItemId, $json);
return itemname;
} else {
$item[$itemId] = GrabItemName($ItemId);
$itemname = array_search($ItemId, $json); // <<
return $itemname;
}
} else {
$item[$ItemId] = GrabItemName($ItemId);
$ejson = json_encode($item);
file_put_contents($CacheFile, $ejson);
return $item[$ItemId];
}
}
Notes
GrabItemName is a different function that returns the description data based on the $ItemId.
The warnings I'm getting are Wrong datatype for second argument in both array_search() and in_array(), on lines 4 and lines 9 respectively (those are the line numbers in the above code - due to the nature of my script these numbers are later on) -- for simplicity, I've marked the problem lines with // <<.
The function is running in a loop which I've no problems with. The problems lie within this function.
What currently happens
Right now, if the cache doesn't exist, it creates it and adds the first item from the loop to the cache file in it's respective JSON format (that fires since the cache file doesn't exist, so after the final else statement).
However, items from the loop after that don't get added, presumably because the file exists and there's something wrong with the code.
The last part of the function works exactly as I want it to but the first part does not.
Expected behaviour with fixed code
Check cache > Return description if item exists ELSE add new item to cache.
The items and their associated descriptions will NOT change, but I'm pulling them from a rate limited API, and I need to ensure I cache whatever I can for everyones benefit.
So, any ideas what I'm doing wrong with the function? I'm sure it's something incredibly simple that I'm overlooking.
Your file is not JSON for an erray. The correct JSON for an array is
[
{"Item1":"Item1 Description"},
{"Item2":"Item2 Description"},
{"Item3":"Item3 Description"}
]
You're missing the brackets around the array, so you just get a single object.
When creating the initial file, you need to do:
$ejson = json_encode(array($item));
so that it's initialized as an array of one item, not just an item.