php xmlreader not returning complete data - php

Hello I am trying to get all the data from XML File , so i have used xmltoassoc function. It is working for 5 mb file but not for more than 9mb file.
Here is my code:
I have modified this function to get json code,
function xml2assoc($xml, $name)
{
$tree = null;
while($xml->read())
{
if($xml->nodeType == XMLReader::END_ELEMENT)
{
return $tree;
}
else if($xml->nodeType == XMLReader::ELEMENT)
{
$node = array();
if($xml->hasAttributes)
{
$attributes = array();
while($xml->moveToNextAttribute())
{
$attributes[$xml->name] = $xml->value;
}
}
if(!$xml->isEmptyElement)
{
$childs = xml2assoc($xml, $node['tag']);
if(isset($childs['text']))
{
$tree = $childs;
} else {
$tree['text'] = $childs[0];
}
}
}
else if($xml->nodeType == XMLReader::TEXT)
{
if(isset($xmlArr['text']))
{
$tree = $xmlArr;
} else {
$tree['text'] = $xmlArr[0];
}
}
}
return $tree;
}
I have used this function to return JSON by passing URL.
function PARSE_XML_JSON($url)
{
$text = "";
$xml = new XMLReader();
$xml->open($url);
$assoc = xml2assoc($xml, "root");
$xml->close();
if(isset($assoc['text']))
{
$text = $assoc['text'];
}
//StoreInTxtFile($text);
return $text;
}
I have also tried to save data in files by doing this:
function StoreInTxtFile($data)
{
$myFile = 'jsonfile-'.time().'.txt';
$fh = fopen($myFile, 'w') or die("can't open file");
fwrite($fh, $data);
fclose($fh);
}
Please tell me what i'm missing.
Thanks

use LIBXML_PARSEHUGE
$xml = new XMLReader();
$xml->open($url, NULL, LIBXML_PARSEHUGE);

Related

handle multiple request accessing a json file

I have json file in text having a sample data
{"msisdn":"xxxxxxxxxx","productID":"YYYYYYYY","subdate":"2018-09-28 16:30:35","Status":"1"}
{"msisdn":"xxxxxxxxxx","productID":"YYYYYYYY","subdate":"2018-09-28 16:30:35","Status":"1"}
and I have a php code that check the json file for existing msisdn
class JSONObject implements JsonSerializable
{
public function __construct($json = false)
{
if ($json)
$this->set(json_decode($json, true));
}
public function set($data)
{
foreach ($data AS $key => $value) {
if (is_array($value)) {
$sub = new JSONObject;
$sub->set($value);
$value = $sub;
}
$this->{$key} = $value;
}
}
public function jsonSerialize()
{
return (object) get_object_vars($this);
}
}
function checkmsisdnallreadyexists($file,$msisdn)
{
if (is_file($file)) {
if (($handle = fopen($file, 'r'))) {
while (!feof($handle)) {
$line = trim(fgets($handle));
$jsonString = json_encode(json_decode($line));
// Here's the sweetness.
//$class = new JSONObject($jsonString);
$class = new JSONObject($jsonString);
if($class->msisdn == $msisdn)
{
$date1=date_create($class->subdate);
$date2=date_create(date('Y-m-d H:i:s'));
$diff=date_diff($date1,$date2);
if($diff->format('%a') < 31)
{
fclose($handle);
return true;
}
}
}
fclose($handle);
}
}
return false;
}
Everything is working fine initially but when my json file has more than 30 000 records, we have a read timeout. as we have a huge request on my server approx 200k request per hour resulting in efficiency of the whole process.
Can any one provide a solution or a alternate method?
Note: I can't use database here
You can use file() instead of fopen() and fclose()
function checkmsisdnallreadyexists($msisdn){
$file_array = file('give file path',FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
foreach($file_array as $arr){
$msisdn_array = json_decode($arr,true);
$msisdn_value = $msisdn_array['msisdn'];
if($msisdn_value == $msisdn) {
$date1=date_create($msisdn_array['subdate']);
$date2=date_create(date('Y-m-d H:i:s'));
$diff=date_diff($date1,$date2);
if($diff->format('%a') < 31) {
return true;
}
}
}
}

Generator function not executed PHP

In the following code, I don't get 'handled' on my output. I see the filehandle a resource, the file gets opened en the contructor of FqReader is called, I checked all that. But with execution of FqReader::getread() I don't see output and the returned array is empty. The first while loop also does not get exectuted when I put while(1) instead of the logic test as in the code now.
<?php
class FastqFile {
function __construct($filename) {
if (substr($filename, -3, 3) == '.gz') {
$this->handle = gzopen($filename, 'r');
return $this->handle;
}
else
$this->handle = fopen($filename, 'r');
return $this->handle;
}
}
class FqReader {
function __construct($file_handle) {
$this->handle = $file_handle;
}
function getread() {
while ($header = fgets($this->handle) !== false) {
echo "handled";
$bases = fgets($this->handle);
$plus = fgets($this->handle);
$scores = fgets($this->handle);
yield array($header, $plus, $scores);
}
}
}
$filename = $argv[1];
$file_handle = new FastqFile($filename);
var_dump($file_handle);
$reader = new FqReader($file_handle);
var_dump($reader->getread());
It outputs:
object(FastqFile)#1 (1) {
["handle"]=>
resource(5) of type (stream)
}
object(Generator)#3 (0) {
}
$file_handle is a FastqFileinstance. Then you pass that object to fgets(), but you need to pass that object's handle to fgets(). For instance:
class FqReader {
function __construct($file_handle) {
$this->handle = $file_handle->handle;
}
function getread() {
while ($header = fgets($this->handle) !== false) {
echo "handled";
$bases = fgets($this->handle);
$plus = fgets($this->handle);
$scores = fgets($this->handle);
yield array($header, $plus, $scores);
}
}
}
The usage of yield was not showing you that error.
Exactly, this works like a charm:
(using a function to open file, not a class)
function openfq($filename)
{
if (substr($filename, -3, 3) == '.gz') {
$handle = gzopen($filename, 'r');
return $handle;
}
else
$handle = fopen($filename, 'r');
return $handle;
}
class FqReader {
function __construct($file_handle) {
$this->handle = $file_handle;
}
function getread() {
while (($header = fgets($this->handle)) !== false) {
echo "handled";
$bases = fgets($this->handle);
$plus = fgets($this->handle);
$scores = fgets($this->handle);
yield array($header, $bases, $scores);
}
}
}
$filename = $argv[1];
$file_handle = openfq($filename);
var_dump($file_handle);
$reader = new FqReader($file_handle);
var_dump($reader->getread());
foreach($reader->getread() as $read) {
var_dump($read);
}

How to save the changes made with a script

With this script, change the order of the lines in a text file, but I'd like to save the file as a new file. How can I do that?
<?php
$file = "menu2.txt";
$righe = file($file);
$numrighe = count($righe);
$portieri = array();
$difensori = array();
$aladestra = array();
$alasinistra = array();
$attaccante = array();
for($i=0; $i < $numrighe;$i++) {
$riga = $righe[$i];
list($giocatore, $ruolo) = preg_split("[>]", $riga);
if(strcmp($ruolo,"Portiere")) { array_push($portieri, $giocatore); }
else if(strcmp($ruolo,"Difensore")) { array_push($difensori, $giocatore); }
else if(strcmp($ruolo,"Ala destra")) { array_push($aladestra, $giocatore); }
else if(strcmp($ruolo,"Ala sinistra")) { array_push($alasinistra, $giocatore); }
else if(strcmp($ruolo,"Attaccante")) { array_push($attaccante, $giocatore); }
}
?>
Excuse me but probably because of google translator or because they are prevented, but you probably will not understand it, I did it:
<?php
$file = "menu2.txt";
$righe = file($file);
$numrighe = count($righe);
$portieri = array();
$difensori = array();
$aladestra = array();
$alasinistra = array();
$attaccante = array();
for($i=0; $i < $numrighe;$i++) {
$riga = $righe[$i];
list($giocatore, $ruolo) = preg_split("[>]", $riga);
if(strcmp($ruolo,"Portiere")) { array_push($portieri, $giocatore); }
else if(strcmp($ruolo,"Difensore")) { array_push($difensori, $giocatore); }
else if(strcmp($ruolo,"Ala destra")) { array_push($aladestra, $giocatore); }
else if(strcmp($ruolo,"Ala sinistra")) { array_push($alasinistra, $giocatore); }
else if(strcmp($ruolo,"Attaccante")) { array_push($attaccante, $giocatore); }
}
// open your new file
$newFile = fopen('newmenu2.txt', 'w');
// write to your file using $newFile file handler,
// with the imploded data you want to write into your file
fwrite($newFile, implode("\n", $data));
// close your file handler
fclose($newFile);
?>
Notice: Undefined variable: data on line 29
Warning: implode(): Invalid arguments passed on line 29

Upload csv files to database

HI Guys I have A problem I want to upload csv files and store to database
But doesn't work..
I tried to debug it several times but not working hope you help me guys.
I Have a csv file containing..
firstname lastname middlename gender
test test test male
Then when I upload this csv file doesn't work.
This is my code..
<?php
session_start();
include ("config.php");
$extension = end(explode(".",basename($_FILES['file']['name'])));
if (isset($_FILES['file']) && $_FILES['file']['size'] < 10485760 && $extension== 'csv')
{
$file = $_FILES['file']['tmp_name'];
$handle = fopen($file, "r");
try
{
$connection = new pdo("mysql:host=$hostname;dbname=upload",$username,$password);
if
$upload = $connection->prepare("INSERT INTO tbl_upload(firstname,lastname,middlename,gender)
VALUES (?,?,?,?)");
if($handle !== false)
{
fgets($handle);
while (($data = fgetcsv($handle, 10000, ',') !== false))
{
$upload->execute($data);
}
fclose($handle);
$connection = null;
echo "<p class='bg-success'>Upload Success</p>";
header ("location: index.php");
}
}
catch(pdoExecption $e)
{
die($e->getmessage());
}
}
else
{
header("location:config.php");
}
?>
thanks for the help..
Your method is not viable.
To import CSV you can either use the mysqlimport utility or you have to split the csv records in separate fields to match your INSERT statement. You cannot just feed CSV to the insert statement and hope that it sorts out things for itself.
I can give you a little CsvImport class I wrote some time ago make importing CSV a little bit easier
<?php
final class CsvImport {
private $file = "";
private $fields = array(); //array("field1", "field2", "field3"); ...
private $data = array(); //array([1] => array("value", "value", "value") ...
private $delimiter = "";
private $fieldCount = 0;
private $rowCount = 0;
private $internalCounter = 0;
private $loaded = false;
public function __construct($_file, $_delimiter = "") {
$this->file = $_file;
if(is_file($this->file) == true) {
if(($handle = fopen($this->file, "r")) !== false) {
//If the delimiter is not set try to suggest it
if(strlen($_delimiter) == 0) {
$this->delimiter = $this->suggestDelimiter();
} else {
$this->delimiter = $_delimiter;
}
if(strlen($this->delimiter) > 0) {
$row = 0;
while(($data = fgetcsv($handle, 0, $this->delimiter)) !== false) {
if($row == 0) {
$this->fieldCount = count($data);
}
if($this->fieldCount > 0) {
for($c = 0; $c < $this->fieldCount; $c++) {
if($row == 0) {
$this->fieldCount = count($data);
$this->fields[] = $data[$c];
} else {
$this->data[$row][$this->fields[$c]] = utf8_encode($data[$c]);
}
}
}
$row++;
}
$this->rowCount = $row;
if($this->fieldCount > 0) {
$this->loaded = true;
}
}
}
}
}
public function getNextRow() {
$retVal = false;
if($this->loaded == true) {
if($this->internalCounter < $this->rowCount) {
$this->internalCounter++;
$retVal = true;
} else {
$this->internalCounter = 0;
}
}
return $retVal;
}
public function readField($field) {
$retVal = false;
if($this->isLoaded() == true) {
if(isset($this->data[$this->internalCounter][$field]) == true) {
$retVal = $this->data[$this->internalCounter][$field];
}
}
return $retVal;
}
public function resetInternalCounter() {
$this->internalCounter = 0;
}
public function getFieldCount() {
return $this->fieldCount;
}
public function getRowCount() {
return $this->rowCount;
}
public function getFieldList() {
return $this->fields;
}
public function getDelimiter() {
return $this->delimiter;
}
public function isLoaded() {
return $this->loaded;
}
private function suggestDelimiter() {
$retVal = "";
$file = fopen($this->file, 'r');
$content = fgets($file);
fclose($file);
if(strlen($content) > 0) {
$list = array(
"," => substr_count($content, ","),
"." => substr_count($content, "."),
"&" => substr_count($content, "&"),
"%" => substr_count($content, "%"),
"-" => substr_count($content, "-"),
";" => substr_count($content, ";"),
"'" => substr_count($content, "'"),
"\"" => substr_count($content, "\""),
);
$maxCount = 0;
foreach($list as $key => $value) {
if($value > 0) {
if($value > $maxCount) {
$retVal = $key;
$maxCount = $value;
}
}
}
}
return $retVal;
}
private function __clone() { }
}
?>
The usage is as simple as this:
$file = "/path/to/file.csv";
$import = new CsvImport($file);
if($import->isLoaded() == true) {
while($import->getNextRow()) {
foreach($import->getFieldList() as $fieldName) {
$value = $import->readField($fieldName);
echo $fieldName . " => " . $value . "<br />";
}
}
}
I am sure you can build your queries with all field names in the foreach loop and fire each query in the while loop.

ChildNodes of DOMDocument lost order

I have a html string. I want to traverse it and extract some information. My code is as following:
$str = '<p>aaa</p><img src="http://stackoverflow.com/questions/ask"/><p>sss</p><img src="http://stackoverflow.com/"/>';
function parseContent($str) {
$contents = array();
$dom = new DOMDocument('1.0', 'UTF-8');
if (!$dom->loadHTML($str)) {
return $contents;
}
$stack = array($dom);
while (count($stack) > 0) {
$node = array_shift($stack);
foreach ($node->childNodes as $node) {
if ($node->hasChildNodes()) {
$stack[] = $node;
} else {
switch ($node->nodeType) {
case XML_ELEMENT_NODE:
if ('img' == $node->tagName) {
$contents[] = $node->attributes->getNamedItem('src')->nodeValue;
}
break;
case XML_TEXT_NODE:
$contents[] = $node->textContent;
break;
}
}
}
}
return $contents;
}
The problem is: When I dumped the return value of this function, it was something like this:
array(
'http://stackoverflow.com/questions/ask',
'http://stackoverflow.com/',
'aaa',
'sss',
)
Could someone point it out why the order was lost?
Extending from comment:
That's because each <p> also has child node (a text node), so they go into the first if ($node->hasChildNodes()) statement and are stacked one more time.
To avoid this, one way is to add one more condition:
/* ... */
if ($node->hasChildNodes()) {
if ($node->childNodes->length==1 && $node->childNodes->item(0)->nodeType==XML_TEXT_NODE) {
$contents[] = $node->childNodes->item(0)->textContent;
} else {
$stack[] = $node;
}
} else {
/* ... */

Categories