I am working on a e-commerce site.
I want to read data from file (CSV or TXT). First time I read data from 1 to 1000 then the process is accidentally stopped. So second time read process should start from 1001.
Anyone please help me!
Option 1
Insert elements one by one to table like:
// take all data
$prods = json_decode(file_get_contents('products.php'));
while (!empty($prods)) {
// remove first element from array. Origin array is changed
$product = array_shift($prods);
// do your insert logic
$this->insert($product);
// for some reasons you don't insert all products
if ($this->break) {
break;
}
}
// write to file what is left, if any
file_put_contents('products.php', json_encode($prods));
Option 2
Sort products by ID and empty-loop while you don't reach your element:
foreach ($products as $product) {
if ($product->id < $lastId) {
continue;
}
$this->insert($product);
}
Related
As the title sais, I'm trying to get the next and previous file from the same directory. So I did some this like this. Is there any better way of doing it? (This is from next auto index file.php code about related files, I have change it for my needs.)
db screenshot if you want to look - ibb.co/wzkDxd3
$title = $file->name; //get the current file name
$in_dir=$file->indir; //current dir id
$r_file = $db->select("SELECT * FROM `". MAI_PREFIX ."files` WHERE `indir`='$in_dir'"); //all of the file from the current dir
$rcount=count($r_file);
$related='';
if($rcount > 2){
$i = 0; // temp variable
foreach($r_file as $key => $r){ //foreach the array to get the key
if($r->name == $title){ //Trying to get the current file key number
$next =$key+1; //Getting next and prev file key number
$prv =$key-1;
foreach($r_file as $keyy => $e){ //getting the file list again to get the prev file
if($prv == $keyy){
$related .=$e->name;
}
}
foreach($r_file as $keyy => $e){ // same for the next file
if($next == $keyy){
$related .=$e->name;
}
}
}
}
Without knowing your DB background and use case, there still should be the possibility to use something like $r_file[$key], $r_file[$next] and $r_file[$prev] to directly access the specific elements. So at least two of your foreach loops could be avoided.
Please note, that nesting loops is extremely inefficient. E. g., if your $r_file contains 100 elements, this would mean 10.000 iterations (100 times 100) with your original code!
Also, you should leave a loop as soon as possible once its task is done. You can use break to do this.
Example, based on the relevant part of your code and how I understand it is supposed to work:
foreach($r_file as $key => $r){ //foreach the array to get the key
if($r->name == $title) { //Trying to get the current file key number
$next =$key+1; //Getting next and prev file key number
$prv =$key-1;
$related .= $r_file[$prv]->name; //Directly accessing the previous file
$related .= $r_file[$next]->name; //Directly accessing the next file
break; // Don't go on with the rest of the elements, if we're already done
}
}
Possibly, looping through all the elements to compare $r->name == $title could also be avoided by using some numbering mechanisms, but without knowing your system better, I can't tell anything more about that.
Background
I'm trying to complete a code challenge where I need to refactor a simple PHP application that accepts a JSON file of people, sorts them by registration date, and outputs them to a CSV file. The provided program is already functioning and works fine with a small input but intentionally fails with a large input. In order to complete the challenge, the program should be modified to be able to parse and sort a 100,000 record, 90MB file without running out of memory, like it does now.
In it's current state, the program uses file_get_contents(), followed by json_decode(), and then usort() to sort the items. This works fine with the small sample data file, however not with the large sample data file - it runs out of memory.
The input file
The file is in JSON format and contains 100,000 objects. Each object has a registered attribute (example value 2017-12-25 04:55:33) and this is how the records in the CSV file should be sorted, in ascending order.
My attempted solution
Currently, I've used the halaxa/json-machine package, and I'm able to iterate over each object in the file. For example
$people = \JsonMachine\JsonMachine::fromFile($fileName);
foreach ($people as $person) {
// do something
}
Reading the whole file into memory as a PHP array is not an option, as it takes up too much memory, so the only solution I've been able to come up with so far has been iterating over each object in the file, finding the person with the earliest registration date and printing that. Then, iterating over the whole file again, finding the next person with the earliest registration date and printing that etc.
The big issue with that is that the nested loops: a loop which runs 100,000 times containing a loop that runs 100,000 times. It's not a viable solution, and that's the furthest I've made it.
How can I parse, sort, and print to CSV, a JSON file with 100,000 records? Usage of packages / services is allowed.
I ended up importing into MongoDB in chunks and then retrieving in the correct order to print
Example import:
$collection = (new Client($uri))->collection->people;
$collection->drop();
$people = JsonMachine::fromFile($fileName);
$chunk = [];
$chunkSize = 5000;
$personNumber = 0;
foreach ($people as $person) {
$personNumber += 1;
$chunk[] = $person;
if ($personNumber % $chunkSize == 0) { // Chunk is full
$this->collection->insertMany($chunk);
$chunk = [];
}
}
// The very last chunk was not filled to the max, but we still need to import it
if(count($chunk)) {
$this->collection->insertMany($chunk);
}
// Create an index for quicker sorting
$this->collection->createIndex([ 'registered' => 1 ]);
Example retrieve:
$results = $this->collection->find([],
[
'sort' => ['registered' => 1],
]
);
// For every person...
foreach ($results as $person) {
// For every attribute...
foreach ($person as $key => $value) {
if($key != '_id') { // No need to include the new MongoDB ID
echo some_csv_encode_function($value) . ',';
}
}
echo PHP_EOL;
}
thanks for reading!
I have an app that allows people to add, edit and delete items in a CSV. I've encountered a bug where if there are non-unique IDs and you try to edit or delete them, it will edit or delete all of them, as the system parses through the spreadsheet to find the ID - which also corresponds to the object's order when using it so the user must be able to change the ID
The solution I've come up with is quite simple, should the user edit an object and change its ID to one that already exists, then the system will take all of the objects with an ID bigger than or equal to the new ID and increment them all by one.
The following code is my if statement that checks whether the ID already exists
if($exists == "true") //does the $newImageID already exist in the gallery?
{
$table = fopen($fullURL,'r'); //$fullURL is the location of the CSV tested and works
$temp_table_two = fopen($tempURL,'w');
while (!feof($temp_table_two) ) {
$getid = fgetcsv($temp_table_two, 1024);
if($getid[0] >= $newImageID)
{
// $getid[0]++; //increase id in temp_table_two by 1 if it is > $newImageID
echo $getid[0];
}
}
fclose($table);
fclose($temp_table);
rename($tempURL,$fullURL);
}
This code takes place after fopen and before fclose. In context, $exists is either "true" or "false" (will change to boolean later on), the while loop parses through my $temp_table (a fopen) and if the first column object (the ID) is equal to or bigger than the one in the new ID then it is incremented. This means that the new object gets "slotted in" so to speak and pushes the rest down
Strangely my request is timing out after a long spinner after I execute this code and I have no idea what the problem is
Thanks for all your help in advance
EDIT: I have found the source of the problem is the while loop itself, should I comment everything out as such:
while (!feof($temp_table_two) ) {
$getid = fgetcsv($temp_table_two, 1024);
// if($getid[0] >= $newImageID)
// {
// // $getid[0]++; //increase id in temp_table_two by 1 if it is > $newImageID
// echo $getid[0];
// }
}
The code still doesn't work yet the only thing left to run is the loop that doesn't do anything
EDIT 2:
Following an answer, I did away with the temp table and just work from the table itself, this if statement is executed BEFORE adding the new data with its ID
if($exists == "true") //does the $newImageID already exist in the gallery?
{
$table = fopen($fullURL,'r+');
while (!feof($table) ) {
$getid = fgetcsv($table, 1024);
if($getid[0] >= $newImageID)
{
echo $getid[0];
$getid[0]++; //increase id in temp_table_two by 1 if it is > $newImageID
}
}
fclose($table);
}
The code no longer times out, but the items inside $getid[0] are not incremented. I have echoed them and it does echo all of the ID's equal to or bigger than my $newImageID but the $getid[0]++; doesn't seem to be affecting the CSV at all
You are testing if you reach the end of the temp file and that's wrong. You need to check the origin file and also read from it!
while (!feof($table) ) {
$getid = fgetcsv($table, 1024);
Try this:
if ($csv = fopen($temp_table_two, 'r+')) do {
$getid = fgetcsv($csv, 1024);
if($getid[0] >= $newImageID)
{
echo $getid[0]; // $getid[0]++;
}
} while (!feof($csv));
That will prevent your while loop from timing out due to being stuck in an infinite if there is a problem opening the file. feof will return true only if it reaches EOF, it will return false otherwise which will cause it to never be able to break out.
For actually writing your data back to the CSV file, your current code won't work as fgetcsv just gives you an array representation of a CSV line in the file. Writing to that array just changes the array, not back to the file.
For that, see this similar answer: Append data to middle line/row of a csv instead of the last line or row
or
http://php.net/manual/en/function.fputcsv.php
I searched all day on this forum but couldn't find any answer.
I use following php code to display the next line from myfile.txt on each pageload (looping when gone through entire file):
<?php
session_start();
$item = file("myfile.txt");
$itemCount = count($item);
if ($_SESSION['sess_row'] === NULL) {
$_SESSION['sess_row'] = 0;
} else {
$_SESSION['sess_row'] = ($_SESSION['sess_row'] + 1) % $itemCount;
}
echo $item[$_SESSION['sess_row']];
?>
Now I want to shuffle the lines in myfile.txt before each session.
For example, if myfile.txt contains 5 lines, it now displays lines in same order every session: 123451234512345...
With shuffle it should display one session: 325413254132541..., another session: 413254132541325..., another session: 142351423514235..., and so on.
How can above code be changed to shuffle myfile.txt before each session?
Use shuffle to shuffle your array each time.
$item = file("myfile.txt");
shuffle($item);
If you need to have different number for each user, you should have a database or something (it could be just a single file containing one number) to keep track of how many of your items are already assigned to users. Each time, read that number (call it cursor), assign the number associated with that cursor, increment the cursor by one and write it back to the file (or database ...)
If you need unique data, you can use array_unique:
$item = file("myfile.txt");
$item = array_unique($items);
shuffle($item);
I have an issue with a processing script. I would like to allow 2 duplicate ip addresses maximum in a csv file, to prevent some spamming and to take into consideration that the user could make a mistake in form fill. I cant seem to reference the $ip variable correctly in the script, or there might be something I am missing altogether. Here is the code snippet thus far:
<?php
#VARIABLE DECLARATIONS (filename and post vars) GO HERE
$counter = 0;
if (file_exists($filename))
{
$file = fopen($filename, "a");
while($data = fgetcsv($filename)){
if(isset($data[$ip])){
$counter++;
continue;
if((isset($data[$ip])){
$counter++;
if($counter == 2){
echo "";
}
}
}
}
##file write goes here
}
?>
Any help on this would be appreciated,
Jim
You will have to read all the elements in an array first and only after you have the number of occurrences of each IP address ready, should go ahead with writing the file (a separate file may be?).
You can first prepare an array with IP indexes and all the rows corresponding to an IP as value attributes to that key.
This could be done -
$csvArray = str_getcsv(file_get_contents('myCSVFile.csv'));
foreach($csvArray as $eachRow)
{
//prepare IP indexed array with every detail row as an attribute of the corresponding IP key.
$properlyFormattedArray[$eachRow[5]][] = $eachRow;
}
You get an array like this -
Array(['92.27.21.171'] =>
[0] => Array("Mr","Test","davis","07972889989","01159174767","92.27.21.171"),
[1] => Array("Mr","bob","jones","07998998008","01159174767","92.27.21.171"),
...
['92.27.21.172'] => ...
)
Once you have this array, just loop over it, and write only at max 2 rows for every IP.
foreach($properlyFormattedArray as $ip => $details)
{
$count = 0;
foreach($details as $eachDetail)
{
if($count<2)
{
//write $eachDetail into file
$count++;
}
}
}
But, in this case, the order of data (compared with your input file) will be changed, and the duplicates will be written in consecutive rows in the csv file (not sure if you would be okay with it).