I have an issue with a processing script. I would like to allow 2 duplicate ip addresses maximum in a csv file, to prevent some spamming and to take into consideration that the user could make a mistake in form fill. I cant seem to reference the $ip variable correctly in the script, or there might be something I am missing altogether. Here is the code snippet thus far:
<?php
#VARIABLE DECLARATIONS (filename and post vars) GO HERE
$counter = 0;
if (file_exists($filename))
{
$file = fopen($filename, "a");
while($data = fgetcsv($filename)){
if(isset($data[$ip])){
$counter++;
continue;
if((isset($data[$ip])){
$counter++;
if($counter == 2){
echo "";
}
}
}
}
##file write goes here
}
?>
Any help on this would be appreciated,
Jim
You will have to read all the elements in an array first and only after you have the number of occurrences of each IP address ready, should go ahead with writing the file (a separate file may be?).
You can first prepare an array with IP indexes and all the rows corresponding to an IP as value attributes to that key.
This could be done -
$csvArray = str_getcsv(file_get_contents('myCSVFile.csv'));
foreach($csvArray as $eachRow)
{
//prepare IP indexed array with every detail row as an attribute of the corresponding IP key.
$properlyFormattedArray[$eachRow[5]][] = $eachRow;
}
You get an array like this -
Array(['92.27.21.171'] =>
[0] => Array("Mr","Test","davis","07972889989","01159174767","92.27.21.171"),
[1] => Array("Mr","bob","jones","07998998008","01159174767","92.27.21.171"),
...
['92.27.21.172'] => ...
)
Once you have this array, just loop over it, and write only at max 2 rows for every IP.
foreach($properlyFormattedArray as $ip => $details)
{
$count = 0;
foreach($details as $eachDetail)
{
if($count<2)
{
//write $eachDetail into file
$count++;
}
}
}
But, in this case, the order of data (compared with your input file) will be changed, and the duplicates will be written in consecutive rows in the csv file (not sure if you would be okay with it).
Related
As the title sais, I'm trying to get the next and previous file from the same directory. So I did some this like this. Is there any better way of doing it? (This is from next auto index file.php code about related files, I have change it for my needs.)
db screenshot if you want to look - ibb.co/wzkDxd3
$title = $file->name; //get the current file name
$in_dir=$file->indir; //current dir id
$r_file = $db->select("SELECT * FROM `". MAI_PREFIX ."files` WHERE `indir`='$in_dir'"); //all of the file from the current dir
$rcount=count($r_file);
$related='';
if($rcount > 2){
$i = 0; // temp variable
foreach($r_file as $key => $r){ //foreach the array to get the key
if($r->name == $title){ //Trying to get the current file key number
$next =$key+1; //Getting next and prev file key number
$prv =$key-1;
foreach($r_file as $keyy => $e){ //getting the file list again to get the prev file
if($prv == $keyy){
$related .=$e->name;
}
}
foreach($r_file as $keyy => $e){ // same for the next file
if($next == $keyy){
$related .=$e->name;
}
}
}
}
Without knowing your DB background and use case, there still should be the possibility to use something like $r_file[$key], $r_file[$next] and $r_file[$prev] to directly access the specific elements. So at least two of your foreach loops could be avoided.
Please note, that nesting loops is extremely inefficient. E. g., if your $r_file contains 100 elements, this would mean 10.000 iterations (100 times 100) with your original code!
Also, you should leave a loop as soon as possible once its task is done. You can use break to do this.
Example, based on the relevant part of your code and how I understand it is supposed to work:
foreach($r_file as $key => $r){ //foreach the array to get the key
if($r->name == $title) { //Trying to get the current file key number
$next =$key+1; //Getting next and prev file key number
$prv =$key-1;
$related .= $r_file[$prv]->name; //Directly accessing the previous file
$related .= $r_file[$next]->name; //Directly accessing the next file
break; // Don't go on with the rest of the elements, if we're already done
}
}
Possibly, looping through all the elements to compare $r->name == $title could also be avoided by using some numbering mechanisms, but without knowing your system better, I can't tell anything more about that.
Background
I'm trying to complete a code challenge where I need to refactor a simple PHP application that accepts a JSON file of people, sorts them by registration date, and outputs them to a CSV file. The provided program is already functioning and works fine with a small input but intentionally fails with a large input. In order to complete the challenge, the program should be modified to be able to parse and sort a 100,000 record, 90MB file without running out of memory, like it does now.
In it's current state, the program uses file_get_contents(), followed by json_decode(), and then usort() to sort the items. This works fine with the small sample data file, however not with the large sample data file - it runs out of memory.
The input file
The file is in JSON format and contains 100,000 objects. Each object has a registered attribute (example value 2017-12-25 04:55:33) and this is how the records in the CSV file should be sorted, in ascending order.
My attempted solution
Currently, I've used the halaxa/json-machine package, and I'm able to iterate over each object in the file. For example
$people = \JsonMachine\JsonMachine::fromFile($fileName);
foreach ($people as $person) {
// do something
}
Reading the whole file into memory as a PHP array is not an option, as it takes up too much memory, so the only solution I've been able to come up with so far has been iterating over each object in the file, finding the person with the earliest registration date and printing that. Then, iterating over the whole file again, finding the next person with the earliest registration date and printing that etc.
The big issue with that is that the nested loops: a loop which runs 100,000 times containing a loop that runs 100,000 times. It's not a viable solution, and that's the furthest I've made it.
How can I parse, sort, and print to CSV, a JSON file with 100,000 records? Usage of packages / services is allowed.
I ended up importing into MongoDB in chunks and then retrieving in the correct order to print
Example import:
$collection = (new Client($uri))->collection->people;
$collection->drop();
$people = JsonMachine::fromFile($fileName);
$chunk = [];
$chunkSize = 5000;
$personNumber = 0;
foreach ($people as $person) {
$personNumber += 1;
$chunk[] = $person;
if ($personNumber % $chunkSize == 0) { // Chunk is full
$this->collection->insertMany($chunk);
$chunk = [];
}
}
// The very last chunk was not filled to the max, but we still need to import it
if(count($chunk)) {
$this->collection->insertMany($chunk);
}
// Create an index for quicker sorting
$this->collection->createIndex([ 'registered' => 1 ]);
Example retrieve:
$results = $this->collection->find([],
[
'sort' => ['registered' => 1],
]
);
// For every person...
foreach ($results as $person) {
// For every attribute...
foreach ($person as $key => $value) {
if($key != '_id') { // No need to include the new MongoDB ID
echo some_csv_encode_function($value) . ',';
}
}
echo PHP_EOL;
}
thanks for reading!
I have an app that allows people to add, edit and delete items in a CSV. I've encountered a bug where if there are non-unique IDs and you try to edit or delete them, it will edit or delete all of them, as the system parses through the spreadsheet to find the ID - which also corresponds to the object's order when using it so the user must be able to change the ID
The solution I've come up with is quite simple, should the user edit an object and change its ID to one that already exists, then the system will take all of the objects with an ID bigger than or equal to the new ID and increment them all by one.
The following code is my if statement that checks whether the ID already exists
if($exists == "true") //does the $newImageID already exist in the gallery?
{
$table = fopen($fullURL,'r'); //$fullURL is the location of the CSV tested and works
$temp_table_two = fopen($tempURL,'w');
while (!feof($temp_table_two) ) {
$getid = fgetcsv($temp_table_two, 1024);
if($getid[0] >= $newImageID)
{
// $getid[0]++; //increase id in temp_table_two by 1 if it is > $newImageID
echo $getid[0];
}
}
fclose($table);
fclose($temp_table);
rename($tempURL,$fullURL);
}
This code takes place after fopen and before fclose. In context, $exists is either "true" or "false" (will change to boolean later on), the while loop parses through my $temp_table (a fopen) and if the first column object (the ID) is equal to or bigger than the one in the new ID then it is incremented. This means that the new object gets "slotted in" so to speak and pushes the rest down
Strangely my request is timing out after a long spinner after I execute this code and I have no idea what the problem is
Thanks for all your help in advance
EDIT: I have found the source of the problem is the while loop itself, should I comment everything out as such:
while (!feof($temp_table_two) ) {
$getid = fgetcsv($temp_table_two, 1024);
// if($getid[0] >= $newImageID)
// {
// // $getid[0]++; //increase id in temp_table_two by 1 if it is > $newImageID
// echo $getid[0];
// }
}
The code still doesn't work yet the only thing left to run is the loop that doesn't do anything
EDIT 2:
Following an answer, I did away with the temp table and just work from the table itself, this if statement is executed BEFORE adding the new data with its ID
if($exists == "true") //does the $newImageID already exist in the gallery?
{
$table = fopen($fullURL,'r+');
while (!feof($table) ) {
$getid = fgetcsv($table, 1024);
if($getid[0] >= $newImageID)
{
echo $getid[0];
$getid[0]++; //increase id in temp_table_two by 1 if it is > $newImageID
}
}
fclose($table);
}
The code no longer times out, but the items inside $getid[0] are not incremented. I have echoed them and it does echo all of the ID's equal to or bigger than my $newImageID but the $getid[0]++; doesn't seem to be affecting the CSV at all
You are testing if you reach the end of the temp file and that's wrong. You need to check the origin file and also read from it!
while (!feof($table) ) {
$getid = fgetcsv($table, 1024);
Try this:
if ($csv = fopen($temp_table_two, 'r+')) do {
$getid = fgetcsv($csv, 1024);
if($getid[0] >= $newImageID)
{
echo $getid[0]; // $getid[0]++;
}
} while (!feof($csv));
That will prevent your while loop from timing out due to being stuck in an infinite if there is a problem opening the file. feof will return true only if it reaches EOF, it will return false otherwise which will cause it to never be able to break out.
For actually writing your data back to the CSV file, your current code won't work as fgetcsv just gives you an array representation of a CSV line in the file. Writing to that array just changes the array, not back to the file.
For that, see this similar answer: Append data to middle line/row of a csv instead of the last line or row
or
http://php.net/manual/en/function.fputcsv.php
I'm trying to display only the rows that contain a specific word in a specific column. Basically I would like to show only the rows that have "yes" in the Display column.
First_Name, Last_Name, Display
Kevin, Smith, yes
Jack, White, yes
Joe, Schmo, no
I've been trying various things with fgetcsv & str_getcsv from other answers and from php.net but nothing is working so far.
It doesn't do anything but this is my current code:
$csv = fopen('file.csv', 'r');
$array = fgetcsv($csv);
foreach ($array as $result) {
if ($array[2] == "yes") {
print ($result);
}
}
Let's have a look at the documentation for fgetcsv():
Gets line from file pointer and parse for CSV fields
fgetcsv reads a single line, not the whole file. You can keep reading lines until you reach the end of the file by putting it in a while loop, e.g.
<?php
$csv = fopen('file.csv', 'r');
// Keep looping as long as we get a new $row
while ($row = fgetcsv($csv)) {
if ($row[2] == "yes") {
// We can't just echo $row because it's an array
//
// Instead, let's join the fields with a comma
echo implode(',', $row);
echo "\n";
}
}
// Don't forget to close the file!
fclose($csv);
You should use data tables.
https://datatables.net/examples/basic_init/zero_configuration.html
That's how I deal with my textfiles. But be carefull, with a large amount of Data (> 10000 rows) you should have a loog at the deferRender option.
https://datatables.net/reference/option/deferRender <-- JSON DATA required.
I have a huge issue, I cant find any way to sort array entries. My code:
<?php
error_reporting(0);
$lines=array();
$fp=fopen('file.txt, 'r');
$i=0;
while (!feof($fp))
{
$line=fgets($fp);
$line=trim($line);
$lines[]=$line;
$oneline = explode("|", $line);
if($i>30){
$fz=fopen('users.txt', 'r');
while (!feof($fz))
{
$linez=fgets($fz);
$linez=trim($linez);
$lineza[]=$linez;
$onematch = explode(",", $linez);
if (strpos($oneline[1], $onematch[1])){
echo $onematch[0],$oneline[4],'<br>';
}
else{
}
rewind($onematch);
}
}
$i++;
}
fclose($fp);
?>
The thing is, I want to sort items that are being echo'ed by $oneline[4]. I tried several other posts from stackoverflow - But was not been able to find a solution.
The anser to your question is that in order to sort $oneline[4], which seems to contain a string value, you need to apply the following steps:
split the string into an array ($oneline[4] = explode(',',
$oneline[4]))
sort the resulting array (sort($oneline[4]))
combine the array into a string ($oneline[4] = implode(',',
$oneline[4]))
As I got the impression variable naming is low on the list of priorities I'm re-using the $oneline[4] variable. Mostly to clarify which part of the code I am referring to.
That being said, there are other improvements you should be making, if you want to be on speaking terms with your future self (in case you need to work on this code in a couple of months)
Choose a single coding style and stick to it, the original code looked like it was copy/pasted from at least 4 different sources (mostly inconsistent quote-marks and curly braces)
Try to limit repeating costly operations, such as opening files whenever you can (to be fair, the agents.data could contain 31 lines and the users.txt would be opened only once resulting in me looking like a fool)
I have updated your code sample to try to show what I mean by the points above.
<?php
error_reporting(0);
$lines = array();
$users = false;
$fp = fopen('http://20.19.202.221/exports/agents.data', 'r');
while ($fp && !feof($fp)) {
$line = trim(fgets($fp));
$lines[] = $line;
$oneline = explode('|', $line);
// if we have $users (starts as false, is turned into an array
// inside this if-block) or if we have collected 30 or more
// lines (this condition is only checked while $users = false)
if ($users || count($lines) > 30) {
// your code sample implies the users.txt to be small enough
// to process several times consider using some form of
// caching like this
if (!$users) {
// always initialize what you intend to use
$users = [];
$fz = fopen('users.txt', 'r');
while ($fz && !feof($fz)) {
$users[] = explode(',', trim(fgets($fz)));
}
// always close whatever you open.
fclose($fz);
}
// walk through $users, which contains the exploded contents
// of each line in users.txt
foreach ($users as $onematch) {
if (strpos($oneline[1], $onematch[1])) {
// now, the actual question: how to sort $oneline[4]
// as the requested example was not available at the
// time of writing, I assume
// it to be a string like: 'b,d,c,a'
// first, explode it into an array
$oneline[4] = explode(',', $oneline[4]);
// now sort it using the sort function of your liking
sort($oneline[4]);
// and implode the sorted array back into a string
$oneline[4] = implode(',', $oneline[4]);
echo $onematch[0], $oneline[4], '<br>';
}
}
}
}
fclose($fp);
I hope this doesn't offend you too much, just trying to help and not just providing the solution to the question at hand.