update a JSON file using PHP/MYSQL - limit size issue - php

The code below works great until i have around 25000-30000 objects in the Json array stored. Then the file isn't update anymore.
Does someone know what could be the issue?
foreach ($serials as $key => $serial_list) {
foreach ($serial_list as $key => $serial) {
$url = 'json/voltage/voltage_' . $serial . '.json';
$file = file_get_contents($url, true);
if ($file != true) {
$sql = "SELECT * FROM realtimedata_V WHERE device_serial ='{$serial}' ORDER by timeStamp ASC limit 1200 ";
$result = $db->mysqli->query($sql);
while ($row = $result->fetch_assoc()) {
$voltage_tmp[$serial][] = array(
(int) strtotime($row['timeStamp']) * 1000,
(int) $row['Va'],
(int) $row['Vb'],
(int) $row['Vc']
);
}
echo '<hr>';
echo 'new_datas_' . $serial . '= ' . json_encode($voltage_tmp[$serial]);
echo '<hr>';
file_put_contents($url, json_encode($voltage_tmp[$serial]));
} else {
$data = json_decode($file, true);
if ($data === NULL)
die('Unable to decode');
unset($file);
$AllKeys = array_keys($data);
echo '<hr> Last_Key= <hr>';
echo $last_index = end($AllKeys);
echo '<hr>';
$key = $data[$AllKeys[$last_index]];
$key_unix = $key[0]/1000;
$limit_sql = $last_index +2000;
echo '<hr>';
echo 'Last_date ' . $serial . '= ' . $last_date = date('Y-m-d H:i:s', $key_unix);
echo '<hr>';
echo $sql = "SELECT * FROM realtimedata_V WHERE device_serial ='{$serial}' AND timeStamp > '{$last_date}' ORDER by timeStamp ASC limit {$limit_sql} ";
$result = $db->mysqli->query($sql);
$row_cnt = $result->num_rows;
if ($row_cnt === 0) {
echo 'rows = ' . $row_cnt;
echo '<hr>Any new datas for the serial N°: ' . $serial . '<hr>';
continue;
} else {
while ($row = $result->fetch_assoc()) {
$voltage[$serial][] = array(
(int) strtotime($row['timeStamp']) * 1000,
(int) $row['Va'],
(int) $row['Vb'],
(int) $row['Vc']
);
}
$result = $data + $voltage[$serial];
echo '<hr>';
echo 'new_datas_' . $serial . '= ' . $new_data = json_encode($result);
echo '<hr>';
if (file_put_contents($url, $new_data) === false) {
die('unable to write file');
}
unset($result);
file_put_contents($url, $new_data, LOCK_EX);
//$result->free();
}
}
}
}
I changed the configuration of the php.ini in case of memory limits. But nothing...
I tried in an another server but same issue.
Does someone have an idea?
Thanks

Related

Using loops by reading data from a database

I have a form with three fields that I want to read separately from the database
$_POST['wkNumer1'];
$_POST['wkNumer2'];
$_POST['wkNumer3'];
How can I read this data without repeating the same code 3 times? In this code, only the variable $wkNumer value will be change.
<?php
if (isset($_POST['show_diagram'])) {
$goodname = $_POST['htDriver'];
$wkNumer = $_POST['wkNumer1'];
// Table with data
$sql = "SELECT WorkingDay, OrderNo, NameFinish, Type FROM `status` where WEEK(WorkingDay) = :wknumer AND NameFinish = :nameFinish"; // SQL with parameters
$stmt = $conn->prepare($sql);
$stmt->bindParam("wknumer", $wkNumer);
$stmt->bindParam("nameFinish", $goodname);
$stmt->execute();
$OCSdatas = $stmt->fetchAll(PDO::FETCH_ASSOC);
$count = $stmt->rowCount();
$countWithoutE = 0;
$countE = 0;
foreach ($OCSdatas as $data) {
if ($data['Type'] != 'E') {
$countWithoutE = $countWithoutE + 1 ;
}
if ($data['Type'] == 'E') {
$countE = $countE + 1 ;
}
}
echo $goodname . '<br />';
echo $wkNumer . '<br />';
echo $countWithoutE . '<br>';
echo $countE . '<br>';
$countE = $countE/2;
$countTotal = $countWithoutE + $countE;
echo $countTotal/5 . '<br>';
echo $count/5;
}
?>
You can put the numbers inside an array and iterate through that array to execute the same code.
$goodname = $_POST['htDriver'];
// Add the numbers to an array which we can iterate
$numbers = [
$_POST['wkNumber1'],
$_POST['wkNumber2'],
$_POST['wkNumber3'],
];
// Prepare the statement only once before the loop and reuse it
$sql = "SELECT WorkingDay, OrderNo, NameFinish, Type FROM `status` where WEEK(WorkingDay) = :wknumer AND NameFinish = :nameFinish";
$stmt = $conn->prepare($sql);
// Loop through the numbers
foreach ($numbers as $number) {
// Add the current number
$stmt->bindParam("wknumer", $number);
$stmt->bindParam("nameFinish", $goodname);
$stmt->execute();
// Now have everything as you had before
$OCSdatas = $stmt->fetchAll(PDO::FETCH_ASSOC);
$count = $stmt->rowCount();
$countWithoutE = 0;
$countE = 0;
foreach ($OCSdatas as $data) {
if ($data['Type'] != 'E') {
$countWithoutE = $countWithoutE + 1 ;
}
if ($data['Type'] == 'E') {
$countE = $countE + 1 ;
}
}
echo $goodname . '<br />';
echo $wkNumer . '<br />';
echo $countWithoutE . '<br>';
echo $countE . '<br>';
$countE = $countE/2;
$countTotal = $countWithoutE + $countE;
echo $countTotal/5 . '<br>';
echo $count/5;
}
How does this work for you?
$workNumbers = array($_POST['wkNumer1'],$_POST['wkNumer2'],$_POST['wkNumer3']);
foreach($workNumbers as $wkNumer){
//Your Code block here
}

Why XMLReader repeat mysql insert function inside while and can't stop?

I am using XMLReader to import huge xml file by elements to MySQL database. Xml contains 1 547 772 tags (element) named 'RECORD'.
XML example
<?xml version="1.0" encoding="utf-8"?>
<RECORD><NAME>ДОШКІЛЬНИЙ НАВЧАЛЬНИЙ ЗАКЛАД №1 ЗАГАЛЬНОГО РОЗВИТКУ УЖГОРОДСЬКОЇ МІСЬКОЇ РАДИ ЗАКАРПАТСЬКОЇ ОБЛАСТІ</NAME><SHORT_NAME>ДНЗ №1</SHORT_NAME><EDRPOU>34888585</EDRPOU><ADDRESS>88000, Закарпатська обл., місто Ужгород, ВУЛИЦЯ М.ВОВЧКА, будинок 47, "А"</ADDRESS><BOSS>НАКОНЕЧНА ОЛЕНА АНАТОЛІЇВНА</BOSS><KVED>85.10 Дошкільна освіта</KVED><STAN>зареєстровано</STAN><FOUNDERS><FOUNDER>УПРАВЛІННЯ ОСВІТИ УЖГОРОДСЬКОЇ МІСЬКОЇ РАДИ, розмір внеску до статутного фонду - 0.00 грн.</FOUNDER>...</FOUNDERS></RECORD>...
For MySQL connection use
function connectBase(){
include __DIR__ . '/../../settings/sql.set.php';
$mysql = mysqli_connect($_sqlhost, $_sqluser, $_sqlpass, $_sqldb);
mysqli_query($mysql, "Set charset utf8");
mysqli_query($mysql, "Set character_set_client = utf8");
mysqli_query($mysql, "Set character_set_connection = utf8");
mysqli_query($mysql, "Set character_set_results = utf8");
mysqli_query($mysql, "Set collation_connection = utf8_general_ci");
return $mysql;
}
Main function for parsing
function XMLReaderToDB($setting = false)
{
$mysql = connectBase();
$dir = __DIR__ . '/../../tmp/';
$xmlURL = $dir . $setting['file'];
$xml = new XMLReader();
$xml->open($xmlURL);
$start_time = time();
$start = $setting['start'];
$limit = $setting['limit'];
$stop = $start + $limit;
$i = 0;
$count = 0;
$result = 1;
while($xml->read())
{
if ($xml->nodeType == XMLReader::ELEMENT && $xml->name == $setting['tag']) {
$item[] = "('items', '" . mysqli_real_escape_string($mysql, $xml->readOuterXML()) . "')";
}
if ($xml->nodeType == XMLReader::END_ELEMENT && $xml->name == $setting['tag']) {
$i++;
$count++;
if ($count >= 500) {
insertXMLtoDB($mysql, $item);
$item = array();
$count = 0;
}
}
if($i == $stop){
break;
}
}
$xml->close();
insertXMLtoDB($mysql, $item);
$mysql->close();
$end_time = time();
$time_elapsed_secs = $end_time - $start_time;
echo '<br/>Items: ' . $i . '<br/>';
echo 'Start: ' . date('H:i:s', $start_time) . '<br/>';
echo 'End: ' . date('H:i:s', $end_time) . '<br/>';
echo $time_elapsed_secs . ' sec. (' . ($time_elapsed_secs /60) . ' min.)';
die;
}
And for MySQL insert
function insertXMLtoDB($mysql, $data = false){
mysqli_query($mysql,"INSERT INTO _parse_tmp (parse_key, parse_value) VALUES " . implode(", ", $data));
$data = array();
// echo 'Success';
}
But, MySQL requests don't stop after 1 547 772 inserts and "while" continues to run. I notice if uncommenting echo 'Success'; in insertXMLtoDB function, "while" is stopping at 1 547 772 inserts and finishing correctly.
What is wrong in my functions?
I've never personally used XMLReader however, I'd assume you could try to add another condition to your statement such as while($xml->read() && $i != $stop) opposed to having it within your loop directly. Ah also just noticed after your usage of the function insertXMLtoDB you re-declare $data although you'll notice you set it to NULL every time your call the function, therefore setting it to an array is pointless.
I can't understand why the script doesn't work fine. That's why I decided to change it.
function XMLReaderToDB($setting = false)
{
$mysql = connectBase();
$dir = __DIR__ . '/../../tmp/';
$xmlURL = $dir . $setting['file'];
$xml = new XMLReader();
$xml->open($xmlURL);
$start_time = time();
$start = $setting['start'];
$limit = $setting['limit'];
$stop = $start + $limit;
$i = 0;
$count = 0;
while (($valid = $xml->read()) && $xml->name !== $setting['tag']) ;
while ($valid) {
$i++;
$count++;
$item[] = "('" . $setting['type'] . "', '" . mysqli_real_escape_string($mysql, $xml->readOuterXML()) . "')";
if ($count == 500) {
insertXMLtoDB($mysql, $item);
$item = array();
$count = 0;
}
$valid = $xml->next($setting['tag']);
}
$xml->close();
insertXMLtoDB($mysql, $item);
$mysql->close();
$end_time = time();
$time_elapsed_secs = $end_time - $start_time;
echo '<br/>Items: ' . $i . '<br/>';
echo 'Start: ' . date('H:i:s', $start_time) . '<br/>';
echo 'End: ' . date('H:i:s', $end_time) . '<br/>';
echo $time_elapsed_secs . ' sec. (' . ($time_elapsed_secs /60) . ' min.)';
return;
}

Returning JSON from a PHP function

I want to transform this PHP function.. that should return JSON data.
<?php
$query = 'SELECT * FROM `' . mix_player::table() . '` a';
if (isset($_GET['cat']) || isset($_GET['order']))
if (isset($_GET['cat'])) {
$query .= ' INNER JOIN `' . mix_player::table_cat_rel() . '` b '
. "ON (a.`id` = b.`idtrack`) WHERE `idcat` = '" . $wpdb->escape($_GET['cat']) . "'";
$random = $wpdb->get_var('SELECT `random`, `order` FROM `' . mix_player::table_categories() . "` WHERE `id` = '"
. $wpdb->escape($_GET['cat']) . "'");
if (!$random)
$order = $wpdb->get_var(NULL, 1);
}
if (isset($_GET['order']))
$order = $_GET['order'];
if ($order != '') {
if (isset($_GET['cat']))
$query .= ' AND ';
else
$query .= ' WHERE ';
$tracks = mix_player::order_list($query, $order);
}
} else {
$random = '0';
}
$query .= ' ORDER BY `id` ASC';
if (isset($tracks) || ($tracks = $wpdb->get_results($query, ARRAY_A))) {
// option "shuffle = true" not always working into mix. Do it our own way...
if ($random == 1) { // shuffle tracks?
list($usec, $sec) = explode(' ', microtime());
mt_srand((float) $sec + ((float) $usec * 100000));
$nrows = count($tracks);
for ($i = 0; $i < $nrows; $i++) {
$j = mt_rand(0, $nrows - 1); // pick j at random
$row = $tracks[$i]; // swap i, j
$tracks[$i] = $tracks[$j];
$tracks[$j] = $row;
}
}
foreach ($tracks as $row) {
$artist = (mix_player::entities($row['artist']));
echo ($artist);
$title = (mix_player::entities($row['title']));
echo ($title);
$url =(xspf_player::entities($row['url']));
echo ($url);
}
}
?>
to display like this json file :
{"title":"title", "artist":"artist","media":"url media.mp3","color":"#56B0E8" },
Can you help me?
Thanks in advance.
You can simply create an array and populate it with your desired values, then return it as JSON:
function tracks2json( $tracks )
{
$retval = array();
foreach( $tracks as $row )
{
$array = array();
$array['artist'] = mix_player::entities($row['artist']);
$array['title'] = mix_player::entities($row['title']);
$array['media'] = 'url '.xspf_player::entities($row['url']);
$array['color'] = '#56B0E8';
$retval[] = $array;
}
return json_encode( $retval );
}
if( isset($tracks) || ($tracks = $wpdb->get_results($query, ARRAY_A)) )
{
// Your MySQL routine here
$json = tracks2json( $tracks );
}
echo json_encode(array("title"=>$title,"artist"=>$artist,"url"=>$url));

How To Change Numbers Based On Results

I have a follow up question on something I got help with here the other day (No Table Three Column Category Layout).
The script is as follows:
$res = mysql_query($query);
$system->check_mysql($res, $query, __LINE__, __FILE__);
$parent_node = mysql_fetch_assoc($res);
$id = (isset($parent_node['cat_id'])) ? $parent_node['cat_id'] : $id;
$catalist = '';
if ($parent_node['left_id'] != 1)
{
$children = $catscontrol->get_children_list($parent_node['left_id'], $parent_node['right_id']);
$childarray = array($id);
foreach ($children as $k => $v)
{
$childarray[] = $v['cat_id'];
}
$catalist = '(';
$catalist .= implode(',', $childarray);
$catalist .= ')';
$all_items = false;
}
$NOW = time();
/*
specified category number
look into table - and if we don't have such category - redirect to full list
*/
$query = "SELECT * FROM " . $DBPrefix . "categories WHERE cat_id = " . $id;
$result = mysql_query($query);
$system->check_mysql($result, $query, __LINE__, __FILE__);
$category = mysql_fetch_assoc($result);
if (mysql_num_rows($result) == 0)
{
// redirect to global categories list
header ('location: browse.php?id=0');
exit;
}
else
{
// Retrieve the translated category name
$par_id = $category['parent_id'];
$TPL_categories_string = '';
$crumbs = $catscontrol->get_bread_crumbs($category['left_id'], $category['right_id']);
for ($i = 0; $i < count($crumbs); $i++)
{
if ($crumbs[$i]['cat_id'] > 0)
{
if ($i > 0)
{
$TPL_categories_string .= ' > ';
}
$TPL_categories_string .= '' . $category_names[$crumbs[$i]['cat_id']] . '';
}
}
// get list of subcategories of this category
$subcat_count = 0;
$query = "SELECT * FROM " . $DBPrefix . "categories WHERE parent_id = " . $id . " ORDER BY cat_name";
$result = mysql_query($query);
$system->check_mysql($result, $query, __LINE__, __FILE__);
$need_to_continue = 1;
$cycle = 1;
$column = 1;
$TPL_main_value = '';
while ($row = mysql_fetch_array($result))
{
++$subcat_count;
if ($cycle == 1)
{
$TPL_main_value .= '<div class="col'.$column.'"><ul>' . "\n";
}
$sub_counter = $row['sub_counter'];
$cat_counter = $row['counter'];
if ($sub_counter != 0)
{
$count_string = ' (' . $sub_counter . ')';
}
else
{
if ($cat_counter != 0)
{
$count_string = ' (' . $cat_counter . ')';
}
else
{
$count_string = '';
}
}
if ($row['cat_colour'] != '')
{
$BG = 'bgcolor=' . $row['cat_colour'];
}
else
{
$BG = '';
}
// Retrieve the translated category name
$row['cat_name'] = $category_names[$row['cat_id']];
$catimage = (!empty($row['cat_image'])) ? '<img src="' . $row['cat_image'] . '" border=0>' : '';
$TPL_main_value .= "\t" . '<li>' . $catimage . '' . $row['cat_name'] . $count_string . '</li>' . "\n";
++$cycle;
if ($cycle == 7) // <---- here
{
$cycle = 1;
$TPL_main_value .= '</ul></div>' . "\n";
++$column;
}
}
if ($cycle >= 2 && $cycle <= 6) // <---- here minus 1
{
while ($cycle < 7) // <---- and here
{
$TPL_main_value .= ' <p> </p>' . "\n";
++$cycle;
}
$TPL_main_value .= '</ul></div>'.$number.'
' . "\n";
}
I was needing to divide the resulting links into three columns to fit my html layout.
We accomplished this by changing the numbers in the code marked with "// <---- here".
Because the amount of links returned could be different each time, I am trying to figure out how to change those numbers on the fly. I tried using
$number_a = mysql_num_rows($result);
$number_b = $number_a / 3;
$number_b = ceil($number_b);
$number_c = $number_b - 1;
and then replacing the numbers with $number_b or $number_c but that doesn't work. Any ideas?
As mentioned before, you can use the mod (%) function to do that.
Basically what it does is to get the remainder after division. So, if you say 11 % 3, you will get 2 since that is the remainder after division. You can then make use of this to check when a number is divisible by 3 (the remainder will be zero), and insert an end </div> in your code.
Here is a simplified example on how to use it to insert a newline after every 3 columns:
$cycle = 1;
$arr = range (1, 20);
$len = sizeof ($arr);
for ( ; $cycle <= $len; $cycle++)
{
echo "{$arr[$cycle - 1]} ";
if ($cycle % 3 == 0)
{
echo "\n";
}
}
echo "\n\n";

foreach loop does not work with include

Would someone of you know why I'm not able to use a (long)piece of code within a foreach loop?
The code in the foreach loop is only executed once.
This code at topictweets.php works fine on its own but I want to repeat it for each forum.
The foreach loop works fine without the include. I also tried to have the code from topic tweets.php plainly in the foreach loop, this didn't work either of course.
The code it includes is used to get topics of a forum from the database and find related tweets, and save those in the database.
Is there some other way to do this?
foreach ($forumlist as $x => $fID) {
echo 'id:'.$fID.'<br>';
include 'topictweets.php';
/////////
////////
}
online version: http://oudhollandsedrop.nl/webendata/FeedForum/fetchtweets.php
bunch of code in topic tweets.php
<?php
//?/ VVVV ---- SELECT TOPICS FOR CURRENT FORUM ----- VVVV ////
echo $fID;
$sql = "SELECT Topics_TopicID
FROM Topics_crosstable
WHERE Forums_ForumID = '$fID'";
$result = mysql_query($sql);
if (!$result) {
//echo 'The topiclist could not be displayed, please try again later.';
} else {
if (mysql_num_rows($result) == 0) {
// echo 'This topic doesn′t exist.';
} else {
while ($row = mysql_fetch_assoc($result)) {
//display post data
// echo $row['Topics_TopicID'];
// echo': ';
$topic = "SELECT Name
FROM Topics
WHERE TopicID = " . mysql_real_escape_string($row['Topics_TopicID']);
$topicname = mysql_query($topic);
if (!$topicname) {
// echo 'The topic could not be displayed, please try again later.';
} else {
if (mysql_num_rows($topicname) == 0) {
// echo 'This topic doesn′t exist.';
} else {
while ($row = mysql_fetch_assoc($topicname)) {
//display post data
// echo $row['Name'];
// echo'<br>';
$topiclist[] = $row['Name'];
}
}
}
}
}
}
foreach ($topiclist as $key => $value) {
$terms .= "" . $value . ",";
}
//echo'<p>';
//echo rtrim($terms, ",");
//echo'<p>';
//echo'<p>';
//echo $terms;
//$terms="vintage";
//Twitter account information
$username = "Username";
$password = "Password";
while (true) {
//$terms="vintage";
//echo "search terms: " . substr_replace($terms, "", -1) . "\n";
$url = "https://stream.twitter.com/1/statuses/filter.json";
$cred = sprintf('Authorization: Basic %s', base64_encode("$username:$password"));
$param = "track=" . urlencode(substr_replace($terms, "", -1));
$opts = array(
'http' => array(
'method' => 'POST',
'header' => $cred,
'content' => $param,
'Content-type' => 'application/x-www-form-urlencoded'),
'ssl' => array('verify_peer' => false)
);
$ctx = stream_context_create($opts);
$handle = fopen($url, 'r', false, $ctx);
//var_dump($handle);
$content = "";
$flag = true;
while ($flag) {
$buffer = fread($handle, 100);
//$buffer = stream_get_line($handle, 1024, "\n");
$a = explode("\n", $buffer, 2);
$content = $content . $a[0];
#var_dump($a);
if (count($a) > 1) {
#echo $content;
#echo "\n";
$r = json_decode($content, true);
#var_dump($r);
// echo '<p>';
// echo "text: " . $r["text"];
// echo '<br>';
// echo "\nrceated_at: " . $r["created_at"];
// echo '<br>';
// echo "\nuser screen name: " . $r["user"]["screen_name"];
// echo '<br>';
// echo "\nuser id: " . $r["user"]["id"];
// echo '<br>';
// echo "\nid : " . $r["id"];
// echo '<br>';
// echo "\nin_reply_to_status_id: " . $r["in_reply_to_status_id"];
// echo '<p>';
// echo "\n\n";
$created_at = $r["created_at"];
$created_at = strtotime($created_at);
$mysqldate = date('Y-m-d H:i:s', $created_at);
//
// echo'<p>';
foreach ($topiclist as $key => $value) {
// echo'getshere!';
//$whichterm = $r["text"];
$whichterm = '"' . $r["text"] . '"';
//echo $whichterm;
if (stripos($whichterm, $value) !== false) {
// echo 'true:' . $value . '';
//find topicid
$whattopic = "SELECT TopicID
FROM Topics
WHERE Name = '$value'";
//var_dump($whattopic);
$tID = mysql_query($whattopic);
//var_dump($tID);
if (!$tID) {
// echo 'topic id not found.';
} else {
if (mysql_num_rows($tID) == 0) {
// echo 'This topic doesn′t exist.';
} else {
while ($rec = mysql_fetch_assoc($tID)) {
$inserttweets = "INSERT INTO
Tweets(Topics_TopicID, AddDate, Tweetcontent)
VALUES('" . mysql_real_escape_string($rec['TopicID']) . "',
'" . mysql_real_escape_string($mysqldate) . "',
'" . mysql_real_escape_string($r["text"]) . "')";
//WHERE TopicID = " . mysql_real_escape_string($row['Topics_TopicID'])
}
}
$addtweet = mysql_query($inserttweets);
if (!$addtweet) {
//something went wrong, display the error
//echo 'Something went wrong while adding tweet.';
//echo mysql_error(); //debugging purposes, uncomment when needed
} else {
echo 'Succesfully added tweet';
}
}
}
}
die();
$content = $a[1];
}
}
fclose($handle);
}
?>
"Pasting" a bunch of code inside a loop isn't a great practice. In fact, what you're looking for is a function or the use of a defined class. So, if you can, define a function in your topictweets.php that will contain your code and use it in your loop:
include 'topictweets.php';
foreach ($forumlist as $x => $fID) {
echo 'id:'.$fID.'<br>';
processYourForums($fID);
/////////
////////
}
try include_once()
however, why not have a loop within topictweets.php?
you can do the query, etc.. in this page, but then loop through it in the include
This should work fine:
include 'topictweets.php';
foreach ($forumlist as $x => $fID) {
echo 'id:'.$fID.'<br>';
}
You only need to include once.

Categories