I have multiple big CSV files with more than 1 million lines. I can't change that.
I need in a loop, get a random line of a random CSV file and output this line.
For now I have :
for ($i=0; $i < 50; $i++) {
$randomFile = $files[array_rand($files)];
if (($handle = fopen($randomFile, "r")) !== FALSE) {
//HERE I NEED TO GET A RANDOM LINE OF $handle FILE
}
}
How can I get a random line of my CSV $handle, without loop like all solution I see everywhere ?
I already tried loop solution but my CSv files are too big and my script freeze each time.
Thanks !
Is that solve the problem ?
<?php
for ($i=0; $i < 50; $i++) {
$randomFile = $files[array_rand($files)];
if (($handle = fopen($randomFile, "r")) !== FALSE) {
$randomLineIndex = array_rand($data = fgetcsv($handle, filesize($randomFile), ","));
print_r($data[$randomLineIndex]);
exit;
}
}
Updated Answer :
According To #Robbie's answer below in the comment section would be with better performance.
<?php
for ($i=0; $i < 50; $i++) {
$randomFile = $files[array_rand($files)];
$randomLineIndex = array_rand(file($randomFile));
print_r($data[$randomLineIndex]);
}
It's not possible. You can read file to array and get a random cell from array (it requires more computer resources, not good for big file) or you can generate random number and go to this line using loop and break after reading needed line.
Related
After opening CSV files with fopen(), I'm currently skipping the three first lines of these files like this:
fgetcsv($file, 0, ';');
fgetcsv($file, 0, ';');
fgetcsv($file, 0, ';');
Is there a nicer method to skip the n first lines ?
If you need to skip more than a handful of lines, simply add one instance of the "throw-away read" code you already have inside a for loop. Number of loops equals lines skipped (e.g. 8 lines):
for ($i = 0; $i < 8; $i++) {
fgetcsv($file, 0, ';');
}
Do this before you begin your main while loop getting the CSV lines you care about. You could turn the loop into a utility function -- if you find yourself doing this often at different skip lengths.
function fskipcsv_lines($handle, int $lines) {
for ($i = 0; $i < $lines; $i++) {
fgetcsv($handle, 0, ';');
}
}
Simple enough. Same construct applies for the "dumb" repetition of any other function you don't need to get a a return value from, that simply needs to be called N times.
P.S. Don't place the "skip line" check routine inside your main CSV iteration loop. Why? Because the check (e.g. $row < 3; or in_array($row, $skips)) would happen at each loop iteration. Suppose you need to read 100,000+ lines, the overhead will start adding up, unnecessarily burdening each loop iteration for the first (now past) few lines' sake.
Your are doing right, but this code may help if you are going to skip many. :
$skippTheseLines = array(1,2,3);
$i = 0;
$totlLines= 10000;
while (($emapData = fgetcsv($file, $totalLines, ";")) !== FALSE) {
if(in_array($i, $skippTheseLines)) {
continue;
}else{
// rest of your code
}
$i = $i + 1;
}
Check out this code. Similar questions: skip first line of fgetcsv method in php
$file = fopen('example.csv', 'r'); // Here example is a CSV name
$row = 1;
$number_to_skip = 3
while (($line = fgetcsv($file,0, ",")) !== FALSE) {
// $line is an array of the csv elements
if($row < $number_to_skip)
{
$row++; continue; // continue is used for skip row 0,1,2
}
print_r($line);
}
So my csv file format would be like this
Site：
Site ID：
Owner:
Interval:
Location:
Last Record:
---Data From User File---：
Date, Content
2019-01-10-12:15, 1
2019-01-10-12:15, 1.5
2019-01-10-12:15, 0.8
2019-01-10-12:15, 1.4
2019-01-10-12:15, 1.9
2019-01-10-12:15, 1.4
2019-01-10-12:15, 1.6
2019-01-10-12:15, 1.7
2019-01-10-12:15, 1.1
And I need do start to read csv file from line 10 (2019-01-10-12:15, 1) to end of the file by using php. I have check most of methods online and it did not help me to solve the problem, my code is provided below.
and it turned out to be reading the previous 8 lines(which is opposite of what I want).
<?php
$index = 8;
$i = 1;
$file = fopen("data.csv", "r");
$data = fgetcsv($file);
while (! feof($file)) {
if ($index >= $i) {
print_r($data = fgetcsv($file));
}
$i++;
}
fclose($file);
?>
Need to read from line 10
<?php
$skip_lines = 8;
$i = 0;
$file = fopen('/path/to.csv', 'r');
while (($row = fgetcsv($file)) !== FALSE) {
if($i++ >= $skip_lines) {
print_r($row);
}
}
fclose($file);
There can be several solutions to process your data.
As your other lines does not start with a number, what I would do is check first character of new line is numeric or not (get first character by using PHP's substr() function and then check that variable it with is_numeric() function). If you get a numeric value, that is your first useful line.
I have a file with ~ 10.000 lines inside. I want every time user access my website, it auto pick 10 lines randomly among them.
Code I currently used:
$filelog = 'items.txt';
$random_lines = (file_exists($filelog))? file($filelog) : array();
$random_count = count($random_lines);
$random_file_html = '';
if ($random_count > 10)
{
$random_file_html = '<div><ul>';
for ($i = 0; $i < 10; $i++)
{
$random_number = rand(0, $random_count - 1); // Duplicate are accepted
$random_file_html .= '<li>'.$random_lines[$random_number]."</li>\r\n";
}
$random_file_html .= '</ul>
</div>';
}
When I have < 1000 lines, every things is ok. But now, with 1000 lines. It make my website slow dow significantly.
That I'm thinking to other methods, like:
Divide file to 50 files, select randomly them, then select 10 lines randoms inside the selected file.
-- or --
I knew total lines (items). Make 10 numbers randomly, then read file use
$file = new SplFileObject('items.txt');
$file->seek($ranđom_number);
echo $file->current();
(My server does not support any type of SQL)
Maybe you have other methods that best suit for me. What is best method for my problem? Thank you very much!
The fastest way would be apparently not to pick 10 lines randomly out of a file with ~ 10.000 lines inside on every user's request.
It's impossible to answer more as we know no details of this "XY problem".
If it is possible to adjust the contents of the file then simply pad each of the lines so they have a common length. Then you can access the lines in the file using random access.
$lineLength = 50; // this is the assumed length of each line
$total = filesize($filename);
$numLines = $total/$lineLength;
// get ten random numbers
$fp = fopen($filename, "r");
for ($x = 0; $x < 10; $x++){
fseek($fp, (rand(1, $numLines)-1)*$lineLength, SEEK_SET);
echo fgets($fp, 50);
}
fclose($fp);
try:
$lines = file('YOUR_TXT_FILE.txt');
$rand = array_rand($lines);
echo $lines[$rand];
for 10 of them just put it in a loop:
$lines = file('YOUR_TXT_FILE.txt');
for ($i = 0; $i < 10; $i++) {
$rand = array_rand($lines);
echo $lines[$rand];
}
NOTE: ** the above code does not guarantee that **2 same lines wont be picked. In order to guarantee uniqueness you need to add extra while loop and an array that holds all randomly generated indexes, so next time it generates it and it already exists in an array, generate another one until its not in the array.
The above solution might not be fastest but might fulfill your needs. Since your server does not support any type of SQL, maybe switch to a different server? Because I am wondering how you are storing User Data? Are those stored in files also?
Thank you for taking the time to read this and I will appreciate every single response no mater the quality of content. :)
Using PHP, I'm trying to get the last 15 lines of a text document (.txt) and store that data into a php variable. I understand that this is possible, however when I do get the last 15 lines, is it possible to retain the order? For example:
text document:
A
B
C
When I grab the text document from the last 15 characters, I don't want the echo to end up like:
C
B
A
All assistance is appreciated and I look forward to your replies; thank you. :) If I didn't explain anything clearly and/or you'd like me to explain in more detail, please reply. :)
Thank you.
Try using array_slice, which will return a part of an array. In this case you want it to return the last 15 lines of the array, so:
$filearray = file("filename");
$lastfifteenlines = array_slice($filearray,-15);
If you don't mind loading the entire file into memory:
$lines = array_slice(file('test.txt'), -15);
print_r($lines );
If the file is too large to fit into memory you can use a circular method:
// Read the last $num lines from stream $fp
function read_last_lines($fp, $num)
{
$idx = 0;
$lines = array();
while(($line = fgets($fp)))
{
$lines[$idx] = $line;
$idx = ($idx + 1) % $num;
}
$p1 = array_slice($lines, $idx);
$p2 = array_slice($lines, 0, $idx);
$ordered_lines = array_merge($p1, $p2);
return $ordered_lines;
}
// Open the file and read the last 15 lines
$fp = fopen('test.txt', 'r');
$lines = read_last_lines($fp, 15);
fclose($fp);
// Output array
print_r($lines);
This method will also work if the file has less than 15 lines- returning an array with however many lines are in the file.
You can use fseek with a negative position to seek backwards through the file, counting newlines as you go.
I'm too tired to write up copy/past-able code, but there are some examples in the comments to the manual page for fseek that are very close to what you want.
If the file isn't bigger than available memory you can do this:
$fArray = file("filename");
$len = sizeof($fArray);
for($i=$len -15;$i<$len ;$i++)
{
echo $fArray[$i];
}
If you have a file that is hundreds of megabytes :
$rc = fopen("file","r");
for ($i=0; $line = fgets($file) ;$i++)
{
if ($i%15 == 0)
{
$last15 = array();
}
$last15[] = $line;
}
echo join("\n",$last15);
the longer array solution:
array_slice(explode("\n",file_get_contents($file)),-15);
the shorter array solution:
array_slice(file($file),-15);
This code will open the file, show the total lines, show the header of file and show the last lines of file defined in $limit.
<?php
// open the file in read mode
$file = new SplFileObject('file.csv', 'r');
// get the total lines
$file->seek(PHP_INT_MAX);
$last_line = $file->key();
echo $last_line;
echo "<br>";
// Rewind to first line to get header
$file->rewind();
// Output first line if you need use the header to make something
echo $file->current();
echo "<br>";
// selecting the limit
$limit = 6;
// selecting the last lines using the $limit
$lines = new LimitIterator($file, $last_line - $limit, $last_line);
//print all the last 6 lines array
//print_r(iterator_to_array($lines));
//echo "<br>";
// Loop over whole file to use a single line
foreach ($lines as $line) {
print_r($line);
echo "<br>";
}
I'm trying to process a for loop within a for loop, and just a little wary of the syntax... Will this work? Essentially, I want to run code for every 1,000 records while the count is equal to or less than the $count... Will the syntax below work, or is there a better way?
for($x = 0; $x <= 700000; $x++) {
for($i = 0; $i <= 1000; $i++) {
//run the code
}
}
The syntax you have will work, but I don't think it's going to do exactly what you want. Right now, it's going to do the outer loop 700,001 times, and for every single one of those 700,001 times, it's going to do the inner loop.
That means, in total, the inner loop is going to run 700,001 x 1001 = about 700.7 million times.
If this isn't what you want, can you give a bit more information? I can't really work out what "I want to run code for every 1,000 records while the count is equal to or less than the $count" means. I don't see any variable named $count at all.
Well, essentially, I'm reading in a text file and inserting each of the lines into a db. I did originally try while(!feof($f)) [where $f = filename], but it keeps complaining of a broken pipe. I thought this would be another way to go
$f should be file-handle returned by fopen(), not a filename.
$file_handle = fopen($filename, 'r');
while(!feof($file_handle)) {
$line = fgets($file_handle);
$line = trim($line); // remove space chars at beginning and end
if(!$line) continue; // we don't need empty lines
mysql_query('INSERT INTO table (column) '
.'VALUES ("'.mysql_real_escape_string($line).'")');
}
Read through the documentation at php.net for fopen(), fgets(). You might also need explode() if you need to split your string.
If your file isn't big, you might want to read it into an array at once like this:
$filelines = file($filename, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
foreach($filelines as $line) {
do_stuff_with($line);
}
Hmm. Ok... Well, essentially, I'm reading in a text file and inserting each of the lines into a db. I did originally try while(!feof($f)) [where $f = filename], but it keeps complaining of a broken pipe. I thought this would be another way to go..
To read a text file line by line I usually:
$file = file("path to file")
foreach($file as $line){
//insert $line into db
}
Strictly answering the question, you'd want something more like this:
// $x would be 0, then 1000, then 2000, then 3000
for($x = 0; $x < 700000; $x += 1000) {
// $i would be $x through $x + 999
for($i = $x; $i < $x + 1000; $i++) {
//run the code
}
}
However, you should really consider one of the other methods for importing files to a database.