working with data in TXT file - php

I have a text file that contain data like this (EURUSD quotes)
19710104,000000,0.53690,0.53690,0.53690,0.53690,1
19710105,000000,0.53660,0.53660,0.53660,0.53660,1
19710106,000000,0.53650,0.53650,0.53650,0.53650,1
19710107,000000,0.53680,0.53680,0.53680,0.53680,1
19710108,000000,0.53710,0.53710,0.53710,0.53710,1
19710111,000000,0.53710,0.53710,0.53710,0.53710,1
19710112,000000,0.53710,0.53710,0.53710,0.53710,1
I want to move some data to another file like
0.53690,0.53690,0.53690,0.53690
and add some difrent calculated number to each line like (Moving average and RSI, Stoch ...) so the file can be trained by Neural Network, final file must be like this
OPEN, HIGH, LOW, CLOSE, VOL, MA50, MA20, RSI14, StochMain, StochSignal,
so I need some hints

You should use the PHP functions fgetcsv and fputcsv. See working example below that you can tweak to your needs.
It assumes that your input values given are in the format OPEN, CLOSE, HIGH, LOW, VOL. Introduce RSI and Stochastic etc in the same way that the Moving Average works.
<?php
// Prepare variables
$row = 1;
$output = array();
// Attempt to open the file quotes.txt
if (($handle = fopen("quotes.txt", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$row++;
// This part of the while loop will be hit for each line in the quotes.txt
// $data is an array that contains the values of this line from the csv
// $output is a new array that you are generating
// Create a new sub-array in the output array (add a new line to the main output)
$output[$row] = array();
// Add the third value of the input array to the start of the new array (the opening price)
$output[$row][] = $data[2];
// Add the fourth value of the input array to the new array (the closing price)
$output[$row][] = $data[3];
// Add the fifth value of the input array to the new array (the high price)
$output[$row][] = $data[4];
// Add the sixth value of the input array to the new array (the low price)
$output[$row][] = $data[5];
// Add the seventh value of the input array to the new array (the volume)
$output[$row][] = $data[6];
// Add moving average to the new array
$output[$row][] = calculate_ma($output, $row);
}
fclose($handle);
}
// Create new file or open existing to save the output to
$handle = fopen('output.csv', 'w');
// Flatten the arrays and save the csv data
foreach ($output as $file) {
$result = [];
array_walk_recursive($file, function($item) use (&$result) {
$result[] = $item;
});
fputcsv($handle, $result);
}
/**
* Calculate the value for the MA using the values in $output.
*/
function calculate_ma($output, $row) {
// For this example we will just say that the MA is equal to the closing price of this period
// and the previous four periods, divided by 5.
$ma = $output[$row][1] + $output[$row-1][1] + $output[$row-2][1] + $output[$row-3][1] + $output[$row-4][1];
$ma = $ma / 5;
return $ma;
}
?>
The output of the above code, using the same input as you have pasted in your question, will be:
0.53690,0.53690,0.53690,0.53690,1,0.10738
0.53660,0.53660,0.53660,0.53660,1,0.2147
0.53650,0.53650,0.53650,0.53650,1,0.322
0.53680,0.53680,0.53680,0.53680,1,0.42936
0.53710,0.53710,0.53710,0.53710,1,0.53678
0.53710,0.53710,0.53710,0.53710,1,0.53682
0.53710,0.53710,0.53710,0.53710,1,0.53692
Bear in mind that the first four moving averages will be incorrect, as they do not have 5 periods of data from which to calculate the 5-period MA.
To calculate a larger MA (50-period) without a huge bunch of code, replace the MA function with:
function calculate_ma($output, $row) {
$period = 50;
for ($x = 0 ; $x < $period ; $x++){
$ma = $ma + $output[$row-$x][1];
}
$ma = $ma / $period;
return $ma;
}

I strongly believe that you should open file, read line by line, explode it with ',' and then store every line to some kind of map, do some calculations and finally save it to another file.
http://php.net/manual/en/function.explode.php
How to read a file line by line in php

Related

PHP - Count Distinct Value in a CSV file

I have a csv file with a very large number of item (5000 lines) in this format
storeId,bookId,nb
124,48361,0
124,48363,6
125,48362,8
125,48363,2
126,28933,4
142,55433,6
142,55434,10
171,55871,7
171,55872,6
I need to count the number of stores in the file, so for exemple with the line above the result should be 5. But I need to doo it with 5000 lines so I can't just loop.
How can I achieve that?
I also need too return the max quantity, so 10
I began by converting the file into an array:
if (file_exists($file)) {
$csv = array_map('str_getcsv', file($file));
#Stores
$storeIds = array_column($csv, 0);
$eachStoreNb = array_count_values($storeIds);
$storeCount = count($eachStoreNb);
}
print_r($storeCount);
Is there a better way to do it? Faster ? Maybe without using the array
Faster here would come in the context of micro-optimization, however you can see an improvement in memory usage.
You could just read the file line by line instead of collecting all store IDs in an array and then doing an array_count_values() saving you an extra loop and unnecessary linear storage of all duplicate values.
Store IDs would just be made as a key for an associative array.
For max NB, you can just keep a max variable keeping the track of max value using max() function. Rest is self-explanatory.
Snippet:
<?php
$file = 'test.csv';
if (file_exists($file)) {
$fp = fopen($file ,'r');
$max_nb = 0;
$store_set = [];
fgetcsv($fp); // ignoring headers
while(!feof($fp)){
$row = fgetcsv($fp);
$store_set[$row[0]] = true;
$max_nb = max($max_nb,end($row));
}
fclose($fp);
echo "Num Stores : ",count($store_set),"<br/>";
echo "Max NB : ",$max_nb;
}else{
echo "No such CSV file found.";
}
Note: For profiling, I suggest you to try both scripts using xdebug
What if you looped through the file line by line?
I mean ...
$datas = [];
$handle = fopen("filename.csv", "r");
$flagFirstLine = true;
while(!feof($handle)){
//dont read first line
if($flagFirstLine) continue;
$flagFirstLine = false;
$csvLine = fgetcsv($handle);
$storeID = $csvLine[0];
$datas[] = $storeID;
}
echo "all row: " . count($datas);
echo "\nnum store: " . count(array_unique($datas));
What 'nice_dev' says, but a little more compact.
$fp = fopen('<your_file>', 'r');
fseek($fp, strpos($content, "\n") + 1); // skip first line
$stores = [];
while($row = fgetcsv($fp)) {
$stores[$row[0]] = max([($stores[$row[0]] ?? 0), $row[2]]);
}
Working example.
An answer with awk would be:
awk -F, 'BEGIN {getline}
{ a[$1]++; m=$3>m?$3:m }
END{ for (i in a){ print i, a[i] };
print "Number of stores",length(a), "max:",m}' testfile
getline to skip the first line
increment the element with the value of the first column $1 in array a with one, and keep the max value in m
loop over the array a and print all counts (optional)
print the total 'Number of stores', and the max value.
output:
124 52
125 52
126 26
142 52
171 52
Number of stores 5 max: 10
Solution in AWK, to compare the difference. This includes the count of each store as well. AWK should be able to process millions in less than 1 second. I use the same to filter duplicates from a file.
BEGIN{ # Set some variables initially
FS="," # field separator for INPUT
mymax=0 # init variable mymax
}
NR>1 { # skip the header line, this matches line 2 onwards
mycount[$1]++ # increase associative array at that position
if ($3>mymax){ # compare with max
mymax=$3
}
}
END{ # finally print results
for (i in mycount){
if (length(i)>0){
print "value " i " has " mycount[i]
}
}
print "Maximum value is " mymax
}

array_keys returning higher number than end($array)

I have a CSV file that contains around 8500 lines but I'm getting a really weird "bug".
I'm validating the data inside the CSV to make sure the data is cool to import into the database. I currently just log the data errors to a log file, but when I open it I see error reports for rows upto 8800 (give or take).
I did some basic debugging to see what's what and did this to begin with:
foreach ($csv as $key => $row)
{
if ($key > 8500) {
echo '<pre>';
print_r($row);
echo '</pre>';
}
}
and that only returned about 50/60 more which is fine as the total rows is around that number.
I then tried doing this to get the end array result:
$last = end($csv);
print_r($last);
and that showed an array with data as expected. However when I do this:
var_dump(array_keys($csv));
then it shows 8800 (give or take) values. Doing count($csv) returns the same number.
I've tried going into the actual CSV and highlighting everything below the last row and hitting clear but it still has the same affect..
Here's how I build my $csv array:
$skus = $csv = [];
if (($handle = fopen($fileTmp, 'r')) !== false) {
set_time_limit(0);
$i = 0;
while (($csvData = fgetcsv($handle, 1000, ',')) !== false)
{
$colCount = count($csvData);
$csv[$i]['sku'] = $csvData[0];
$csv[$i]['desc'] = $csvData[1];
$csv[$i]['ean'] = $csvData[2];
$csv[$i]['rrp_less_vat'] = $csvData[3];
$csv[$i]['rrp_inc_vat'] = $csvData[4];
$csv[$i]['stock'] = $csvData[5];
$csv[$i]['est_delivery'] = $csvData[6];
$csv[$i]['img_name'] = $csvData[7];
$csv[$i]['vatable'] = $csvData[8];
$csv[$i]['obsolete'] = $csvData[9];
$csv[$i]['dead'] = $csvData[10];
$csv[$i]['replacement_product'] = $csvData[11];
$csv[$i]['brand'] = $csvData[12];
$csv[$i]['ext_desc'] = $csvData[13];
$i++;
}
fclose($handle);
}
Am I doing something wrong that I can't see in building the array or is this unexpected behaviour?
PHP version: 7.1
OS: Linux Mint
You have lines that are longer than the $length argument you are passing to fgetcsv(). From the documentation, emphasis mine:
Must be greater than the longest line (in characters) to be found in the CSV file (allowing for trailing line-end characters). Otherwise the line is split in chunks of length characters, unless the split would occur inside an enclosure.
The easiest fix is to stop limiting the length of the line to 1000:
while (($csvData = fgetcsv($handle)) !== false)

adding string after every 'n' number of lines in a file using php

I have a .lst(playlist) file with around 1800 lines of data. Each line contains a URL to an audio file that is being played on my radio station.
The thing is I need to add URLs to some Advertisements after every 'n' number of lines in that file.
There are 10 URLs of advertisements from which 1 URL needs to be added after every 5 lines.
Example: URLs of Advertisements are in this format: file1.mp3, file2.mp3 ... file10.mp3
They will be added like this: file1.mp3 on line 5, file2.mp3 on line 10, file3.mp3 on line 15 and so on. After file10.mp3 has been inserted, it will again start from file1.mp3. I hope you understand.
So far I have managed to cook up the following code, but it only takes up one string to be added and have to manually tell the line number on which the string will be added. Unable to figure out the looping logic to do the aforementioned work.
$url = "/home/station/content/file1.mp3";
$line_number = 5; //add above URL on line number 5
$contents = file('playlist.lst', FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
if($line_number > sizeof($contents)) {
$line_number = sizeof($contents) + 1;
}
array_splice($contents, $line_number-1, 0, array($url));
$contents = implode("\n", $contents);
file_put_contents('playlist.lst', $contents);
How can I achieve this ?
You can use array_chunk to split your array into $line_number. Then, use array_map() to add your advertisements to each group. Finally, you could reduce to a linear array. You can format the $url using sprintf().
$url = "/home/station/content/file%d.mp3"; // use %d to use using sprintf()
$line_number = 5; //add above URL on line number 5
$counter = 1;
$contents = file('playlist.lst', FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
// split in group of $line_number
$groups = array_chunk($contents, $line_number);
// for each group:
$groups = array_map(function($arr) use ($url, &$counter) { // pass $counter as reference
// format the link
$adv = sprintf($url, $counter++) ;
// restart to 1 if greater than 10
if ($counter > 10) $counter = 1;
// append to group
$arr[] = $adv;
return $arr ;
},$groups);
// transform to linear array
$contents = array_reduce($groups, 'array_merge', array());
// save new file
file_put_contents('playlist.lst', implode("\n", $contents));
You could do it this way, with a simple loop:
//changing it to a "model" string, we are going to add the correct file number later
$url = "/home/station/content/file";
$contents = file('playlist.lst', FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
$count = 0;
$AddCount = 1;
//Loop until there is nothing left in our radio list
while ($count < sizeof($contents)) {
//if we have a multiple of 5, we are inserting an ad
if (($count % 5) == 0) {
// to know wich ad to add we use our AddCounter
$tmp = $url . strval($AddCount) . ".mp3";
//As suggested by Justinas, don't forget that each time you had a line you need to also increase the index for the next one using $count%5 to know how many lines you already added
array_splice($contents, $count - 1 + ($count % 5) , 0, array($tmp));
$AddCount += 1;
if ($AddCount > 10)
$AddCount = 1;
}
$count += 1;
}
$contents = implode("\n", $contents);
file_put_contents('playlist.lst', $contents);
This way, you don't even have to handle the advertisements file selection yourself as long as they are all formated like you said.
You should do a loop in such way.
$line_number = 5;
$line_count = 0;
for($i=0; $i < sizeof($contents); $i++)
{
$line_count = $line_count +1; // start with Line 1 that +1
if($line_count == $line_number)
{
// do your magic code of adding a line
$line_count = 0; // refresh counting from the beginning
}
}
You don't need to handle each line in the file one at a time.
Leave the file contents as its original string, inject placeholders with regex, then replace those placeholders with your ad array strings.
Code: (Basic Demo)
$filename = 'playlist.lst';
file_put_contents(
$filename,
vsprintf(
preg_replace(
'/(?:.+(\R)){5}\K/', // match (and forget) 5 lines, capture the newline character
'%s\1', // inject the placeholder following the the newline character
file_get_contents($filename),
count($ads) // make the exact number of needed placeholders
),
$ads // replace the placeholders with the ad strings
)
);

How to sum column text file

Hello everyone and I immediately apologize, as
I have seen various threads on the site, but unfortunately my knowledge is still insufficient to complete my project.
I have a text file and I have to do the sum of each column (just need the total):
1003|name1|1208.00|2.00 |96.00 |0.00|0.00|0.00|0.00|98.00 |90.95 |7.05 |8516.40
1011|name2|1450.00|2.00 |49.00 |0.00|0.00|0.00|0.00|51.00 |44.62 |6.38 |9243.7
1004|name3|1450.00|25.00|170.00|0.00|0.00|0.00|0.00|195.00|175.75|19.25|27912.5 <br>
1002|name4|765.00 |1.00 |17.00 |0.00|0.00|0.00|0.00|18.00 |15.13 |2.87 |2193.26
I need to get this(I have this file on linux then we can use Bash, PHP, Mysql... ):
1003|name1|1208.00|2.00 |96.00 |0.00|0.00|0.00|0.00|98.00 |90.95 |7.05 |8516.40
1011|name2|1450.00|2.00 |49.00 |0.00|0.00|0.00|0.00|51.00 |44.62 |6.38 |9243.7
1004|name3|1450.00|25.00|170.00|0.00|0.00|0.00|0.00|195.00|175.75|19.25|27912.5 <br>
1002|name4|765.00 |1.00 |17.00 |0.00|0.00|0.00|0.00|18.00 |15.13 |2.87 |2193.26 <br>
xxxx|Total |4873.00|30.00|332.00|0.00|0.00|0.00|0.00|362.00 |326.45|35.55|47865.86
Where xxxx is the Id number (No sum here).
I've been trying to do this in PHP and MySQL -- No luck so far.
try something like:
$file = '/path/to/your_file.txt';
if ( ($file = fopen($file, "r")) !== FALSE) {
$total = 0;
$row_1 = 0;
while (($line = fgetcsv($file, 1000, "|")) !== FALSE) {
// brutal dirt sanitization
foreach ( $line as $k => $v ) {
$line[$k] = (float) preg_replace('#[^0-9\.]#','', $v);
}
$total = $total + array_sum(array_slice($line, 2));
$row_1 = $row_1 + array_sum(array_slice($line, 2, 1));
//...
}
echo $total.' | '.$row_1; //...
}
else echo 'error ...';
also, you can sanitize each row by replacing array_sum() by array_map() wih a callback function
Psuedocode:
open source file for reading
open destination file for writing
initialise totaling array to zero values
while not EOF
read in line from file
explode line into working array
for x=2 ; x<14; x++
add totalling array with floatval( working array )
write line out to destination file
close read file
write out totals array to destination file
close destingation file
Try to get the text file data into an excel spreadsheet and then add up the columns.
You can use VB to get the text into excel and then continue adding up the values of each column.
1) replace all | chars with , using str_replace
2) Use str_getcsv to create array out of the above resulting csv string
3) use foreach and loop through each row and calculate total
some PHP code
$str = file_get_contents('myfile.txt');
$str = str_replace('|', ',', $str);
$csv = str_getcsv($str);
$totals = array(0,0,0,0);
foreach ($csv as $row) {
$totals[0] += trim($row[0]);
$totals[1] += trim($row[2]);
$totals[2] += trim($row[3]);
$totals[3] += trim($row[4]);
}
the $totals array contains all totals!

PHP Grab last 15 lines in txt file

Thank you for taking the time to read this and I will appreciate every single response no mater the quality of content. :)
Using PHP, I'm trying to get the last 15 lines of a text document (.txt) and store that data into a php variable. I understand that this is possible, however when I do get the last 15 lines, is it possible to retain the order? For example:
text document:
A
B
C
When I grab the text document from the last 15 characters, I don't want the echo to end up like:
C
B
A
All assistance is appreciated and I look forward to your replies; thank you. :) If I didn't explain anything clearly and/or you'd like me to explain in more detail, please reply. :)
Thank you.
Try using array_slice, which will return a part of an array. In this case you want it to return the last 15 lines of the array, so:
$filearray = file("filename");
$lastfifteenlines = array_slice($filearray,-15);
If you don't mind loading the entire file into memory:
$lines = array_slice(file('test.txt'), -15);
print_r($lines );
If the file is too large to fit into memory you can use a circular method:
// Read the last $num lines from stream $fp
function read_last_lines($fp, $num)
{
$idx = 0;
$lines = array();
while(($line = fgets($fp)))
{
$lines[$idx] = $line;
$idx = ($idx + 1) % $num;
}
$p1 = array_slice($lines, $idx);
$p2 = array_slice($lines, 0, $idx);
$ordered_lines = array_merge($p1, $p2);
return $ordered_lines;
}
// Open the file and read the last 15 lines
$fp = fopen('test.txt', 'r');
$lines = read_last_lines($fp, 15);
fclose($fp);
// Output array
print_r($lines);
This method will also work if the file has less than 15 lines- returning an array with however many lines are in the file.
You can use fseek with a negative position to seek backwards through the file, counting newlines as you go.
I'm too tired to write up copy/past-able code, but there are some examples in the comments to the manual page for fseek that are very close to what you want.
If the file isn't bigger than available memory you can do this:
$fArray = file("filename");
$len = sizeof($fArray);
for($i=$len -15;$i<$len ;$i++)
{
echo $fArray[$i];
}
If you have a file that is hundreds of megabytes :
$rc = fopen("file","r");
for ($i=0; $line = fgets($file) ;$i++)
{
if ($i%15 == 0)
{
$last15 = array();
}
$last15[] = $line;
}
echo join("\n",$last15);
the longer array solution:
array_slice(explode("\n",file_get_contents($file)),-15);
the shorter array solution:
array_slice(file($file),-15);
This code will open the file, show the total lines, show the header of file and show the last lines of file defined in $limit.
<?php
// open the file in read mode
$file = new SplFileObject('file.csv', 'r');
// get the total lines
$file->seek(PHP_INT_MAX);
$last_line = $file->key();
echo $last_line;
echo "<br>";
// Rewind to first line to get header
$file->rewind();
// Output first line if you need use the header to make something
echo $file->current();
echo "<br>";
// selecting the limit
$limit = 6;
// selecting the last lines using the $limit
$lines = new LimitIterator($file, $last_line - $limit, $last_line);
//print all the last 6 lines array
//print_r(iterator_to_array($lines));
//echo "<br>";
// Loop over whole file to use a single line
foreach ($lines as $line) {
print_r($line);
echo "<br>";
}

Categories