Trying to get multiple pages of an API returning JSON into one CSV file. Currently it will be sufficient to set the maximum amount of pages it will loop through to 120.
This is the first request without any loop which creates a working CSV file (output.txt).
<?php
$pageNo = "1";
$jsonString = file_get_contents("http://api.domain.com/apikey/?
name1=value1&name2=value2&pageNo=$pageNo");
$jsonDecoded = json_decode($jsonString, true);
$csvHeader=array();
$csvData=array();
$csvFileName = 'output.txt';
$fp = fopen($csvFileName, 'w');
$counter=0;
foreach($jsonDecoded["result"]["items"] as $key => $value)
{
jsontocsv($value);
if($counter==0)
{
fputcsv($fp, $csvHeader, ';');
$counter++;
}
fputcsv($fp, $csvData, ';');
$csvData=array();
}
fclose($fp);
function jsontocsv($data)
{
global $csvData,$csvHeader;
foreach($data as $key => $value)
{
if(!is_array($value))
{
$csvData[]=$value;
$csvHeader[]=$key;
}
else
{
jsontocsv($value);
}
}
}
?>
I'm guessing this should be in a while loop, with something like ++$pageNo at the end, updating the value. However all previous attempts have resulted in the file being overwritten each time and only the last page being written into the file. How can I add stuff to the CSV each time it loops? Ideally, though, each time the entire script runs (once per day), the entire CSV file will be replaced with the updated values.
Thanks
you are opening the file in write mode, which places the pointer to the beginning of the file (fopen($csvFileName, 'w')), this is why it gets overridden.. try using append mode, it should place the pointer to the end.. also, if you want to create the file if it does not exist, use the plus sign like this: fopen($csvFileName, 'a+')
Related
This question already has answers here:
Reading very large files in PHP
(8 answers)
Closed 1 year ago.
I have a file with around 100 records for now.
The file has users in json format per line.
Eg
{"user_id" : 1,"user_name": "Alex"}
{"user_id" : 2,"user_name": "Bob"}
{"user_id" : 3,"user_name": "Mark"}
Note : This is a just very simple example, I have more complex json values per line in the file.
I am reading the file line by line and store that in an array which obviously will be big if there are a lot of items in the file.
public function read(string $file) : array
{
//Open the file in "reading only" mode.
$fileHandle = fopen($file, "r");
//If we failed to get a file handle, throw an Exception.
if ($fileHandle === false) {
throw new Exception('Could not get file handle for: ' . $file);
}
$lines = [];
//While we haven't reach the end of the file.
while (!feof($fileHandle)) {
//Read the current line in.
$lines[] = json_decode(fgets($fileHandle));
}
//Finally, close the file handle.
fclose($fileHandle);
return $lines;
}
Next, Ill process this array and only take the parameters I need (some parameters might be further processed) and then Ill export this array to csv.
public function processInput($users){
$data = [];
foreach ($users as $key => $user)
{
$data[$key]['user_id'] = $user->user_id;
$data[$key]['user_name'] = strtoupper($user->user_name);
}
// Call export to csv $data.
}
What should be the best way to read the file (incase we have a big file)?
I know file_get_contents is not optimized way and instead fgets is a better approach.
Is there a much better way considering big file read and then put it to csv.
You need to modify your reader to make it more "lazy" in some sense. For example consider this:
public function read(string $file, callable $rowProcessor) : void
{
//Open the file in "reading only" mode.
$fileHandle = fopen($file, "r");
//If we failed to get a file handle, throw an Exception.
if ($fileHandle === false) {
throw new Exception('Could not get file handle for: ' . $file);
}
//While we haven't reach the end of the file.
while (!feof($fileHandle)) {
//Read the current line in.
$line = json_decode(fgets($fileHandle));
$rowProcessor($line);
}
//Finally, close the file handle.
fclose($fileHandle);
return $lines;
}
Then your will need different code that works with this:
function processAndWriteJson($filename) { //Names are hard
$writer = fopen('output.csv', 'w');
read($filename, function ($row) use ($writer) {
// Do processing of the single row here
fputcsv($writer, $processedRow);
});
}
If you want to get the same result as before with your read method you can do:
$lines = [];
read($filename, function ($row) use ($writer) {
$lines[] = $row;
});
It does provide some more flexibility. Unfortunately it does mean you can only process one line at a time and scanning up and down the file is harder
I have no luck when the subject is reading text files. I have a small script to read a log file (real time updated) but I want to send some data to DB.
And the problem is, if I don't stat reading from the end of the files, I will get duplicated entries in database. Wich can't happen!
// Keep alive
for (;;)
{
$handle = fopen("data.log", "r");
if (!$handle) die("Open error - data.log");
while (!feof($handle))
{
$line = fgets($handle, 4096);
// If match with, I output the result
if (strpos($line, ':gshop_trade:') > 0)
{
if (!preg_match('/([\d-: ]+)\s*.*\sformatlog:gshop_trade:userid=(\d+):(.*)item_id=(\d+):expire=(\d+):item_count=(\d+):cash_need=(\d+):cash_left=(\d+).*$/', $line, $data))
{
echo "Parsing error on line: {$line}";
}
// show the data
}
}
sleep(5);
}
This script is working, but as I mentioned above, I need to send the data to BD. But also, I need to leave script running, with this current code, the script match the wanted string, and instead of wait for new entries on data.log he starting reading the whole file again.
I see this question here and I tested but doesn't work. I'll start the script when I start the service that generates "data.log" but to prevent duplicate entries in database, I need to read the last lines.
How can I do that?
Keep a track of the file offset from the previous reading using ftell() and keeping that result in a variable, and jump to that offset in the file when you re-open it for the next reading using fseek()
$lastPos = 0;
for (;;)
{
$handle = fopen("data.log", "r");
if (!$handle) die("Open error - data.log");
fseek($handle, $lastPos); // <--- jump to last read position
while (!feof($handle))
{
$line = fgets($handle, 4096);
$lastPos = ftell($handle); // <--- maintain last read position
// If match with, I output the result
if (strpos($line, ':gshop_trade:') > 0)
{
if (!preg_match('/([\d-: ]+)\s*.*\sformatlog:gshop_trade:userid=(\d+):(.*)item_id=(\d+):expire=(\d+):item_count=(\d+):cash_need=(\d+):cash_left=(\d+).*$/', $line, $data))
{
echo "Parsing error on line: {$line}";
}
// show the data
}
}
sleep(5);
}
Maybe you can use file_get_contents, explode and read the array backwards?
$arr = explode(PHP_EOL, file_get_contents("data.log")); // or file("data.log");
$arr = array_reverse($arr);
foreach($arr as $line){
// do stuff here in reverse order
}
From comments above I suggest this method to only use the new data in your code.
It will read your log and a text file with what has been read last time.
Remove what was read last time and use the new data in the code.
$logfile = file_get_contents("data.log");
$ReadData = file_get_contents("readdata.txt");
$newdata = str_replace($ReadData, "", $logfile); // this is what is new since last run.
file_put_contents("readdata.txt", $logfile); // save what has been read.
$arr = explode(PHP_EOL, $newdata);
foreach($arr as $line){
// do your stuff here with the new data.
}
?>
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="refresh" content="5"> <!-- This will run the page every five seconds.
</head>
</html>
I am having this strange issue and can't figure it out.
On some websites I have this script works perfect... same code, same server settings...
With php, there is a simple page view hit counter that stores locally in a txt file.
Then I echo out the value on the footer copyright area of my websites to give the client a quick statistic... its pretty cool how fast it grows.
Anyway.. i have a client corner grill ny . com (seo purposes I added spaces )
On that website.. its been working great for years.
Now another website and a bunch more.. for example... savianos . com
This breaks.. and the text value is blank.
This is the counter.php code
<?php
session_start();
$counter_name = "counter/hits.txt";
//Check if a text file exists. If not create one and initialize it to zero.
if (!file_exists($counter_name)) {
$f = fopen($counter_name, "w");
fwrite($f,"0");
fclose($f);
}
// Read the current value of our counter file
$f = fopen($counter_name,"r");
$counterVal = fread($f, filesize($counter_name));
fclose($f);
// Has visitor been counted in this session?
// If not, increase counter value by one
if(!isset($_SESSION['hasVisited'])){
$_SESSION['hasVisited']="yes";
$counterVal++;
$f = fopen($counter_name, "w");
fwrite($f, $counterVal);
fclose($f);
}
?>
Now, if I add a value in the txt file.. like 1040... and go to the website it starts to work... then after a week or so I check it .. its blank again.
Any ideas?
I am thinking that this may be happening because the website might get a TON of views during dinner time friday night.. and the simple script can't handle it so.. while its trying to write a added a number it just breaks and go to blank.. and never starts back up again.
The structure is this.
/counter/ folder has
counter.php and a hits.txt file
Every page of the website the very first thing is
<?php include ('counter/counter.php'); ?>
and in the footer of the website we have
<?php echo $counterVal; ?>
Your code looks perfect, but let's understand the situation. You have a file which can be accessed concurrently for many users, because page visit can be done by multiple users on same time. This does't seem right you have to lock the file manipulation for another user while someone is modifying it, right?. Please have a look
Visits counter without database with PHP
It is most likely because you have two concurrent scripts that tried to open the file at one and one of them fail. You have to use flock() when there are multiple instances of the script that could operate at the same time. Counter are some of the heaviest things if you going to use file reading and writing. I wrote this wrapper to easily implement file locking.
If you want to check out one of my counters that in operation try http://ozlu.org. That dynamic counter image was self-built. The fileReadAll will read the entire file in one shot. The file writer only has two modes, write or append. You can pass the fileWriter an array or a string and it will write it to the file. The function will not add any \n to format your text so you would have to add that. The default mode for the fileWriteAll is w if you do not set the third argument.
function fileWriteAll($file, $content, $mode = "w"){
$mode = $mode === "w" || $mode === "a"? $mode : "w";
$FILE = fopen($file, $mode);
while (!flock($FILE, LOCK_EX)) { usleep(1); }
if( is_array($content) ){
for ($i = 0; $i < count($content); $i++){
fwrite($FILE, $content[$i]);
}
} else {
fwrite($FILE, $content);
}
flock($FILE, LOCK_UN);
fclose($FILE);
}
function fileReadAll($file){
$FILE = fopen($file, 'r');
while (!flock($FILE, LOCK_SH)) { usleep(1); }
$content = fread($FILE, filesize($file));
flock($FILE, LOCK_UN);
fclose($FILE);
return $content;
}
Your modified code:
session_start();
$counterName = './views.txt';
if (!file_exists($counterName)) {
$file = fopen($counterName, 'w');
fwrite($file, '0');
fclose($file);
}
$file = fopen($counterName, 'r');
$value = fread($file, filesize($counterName));
fclose($file);
if (! isset($_SESSION['visited'])) {
$_SESSION['visited'] = 'yes';
$value++;
$file = fopen($counterName, 'w');
fwrite($file, $value);
fclose($file);
}
session_unset();
echo $value;
I'm trying to execute the code below and I keep on getting a message reading killed when I load the file in my terminal. I'm aware that I'm using lots of memory, so I set the memory limit to the maximum amount allowed on apache. I have a text file called codes.txt that contains a list of numbers from 0 to 1000000. I need to randomize the occurrence of these numbers and then write the new order of them to a new text file. Then, I need to store the new occurrence of them in an array.
ini_set('memory_limit', '2048M');
// Get all of the values from the .txt file
// and store them in an array
$file = fopen("codes.txt", "r");
$codes = array();
while(!feof($file)) {
$codes[] = trim(fgets($file));
}
fclose($file);
// Randomize the elements in the array
shuffle($codes);
// Write each element in the shuffled array to a new .txt file
$new_file = fopen("new_codes.txt", "w");
for($i=0;$i<1000000;$i++) {
fwrite($new_file, $codes[$i].PHP_EOL);
}
fclose($new_file);
// Put all of the new elements into a new array
$new_file = fopen("new_codes.txt", "r");
$code = array();
while(!feof($new_file)) {
$code[] = trim(fgets($new_file));
}
print_r($code);
Don't worry about a new array, $codes already has them. If you need to close, reopen the file and read them into a new array, and memory is the issue, then kill the old array first by using unset($codes) before opening the file.
ini_set('memory_limit', '2048M');
// Get all of the values from the .txt file
// and store them in an array
$file = fopen("codes.txt", "r");
$codes = array();
while (!feof($file)) {
$codes[] = trim(fgets($file));
}
fclose($file);
// Randomize the elements in the array
shuffle($codes);
// Write each element in the shuffled array to a new .txt file
$new_file = fopen("new_codes.txt", "w");
foreach($codes as $k => $v){
fwrite($new_file, $v.PHP_EOL);
}
fclose($new_file);
I am writing a program in php to check ip's , now I know that are easier ways to do so, but i want to do it my way. This is what i have written so far
<?php
if($_POST) {
$file=fopen("names.txt","a") or exit("Unable to open file!");
$ipadres=fopen("ip.txt","a") or exit("Unable to open file!");
$name = $_POST['username'];
$file_content = $name. "|";
$ipadres_content = $_SERVER["REMOTE_ADDR"] . "|";
$iparray = array();
$i=0;
fputs($file,$file_content);
fputs($ipadres,$ipadres_content);
while(!feof($ipadres))
{
$iparray = explode("|", fgets($file));
}
fclose($file);
fclose($ipadres);
}
?>
As you can see i tried using a while loop to put the ip-adresses in to an array to check. but when I try to run it it just keeps running until it finally crashes in to this error= Fatal error: Maximum execution time of 30 seconds exceeded.Oh and yes i tried to put the max crash limit up a bit but still no sign of succes.
Your while loop is faulty:
while(!feof($ipadres))
{
$iparray = explode("|", fgets($file));
}
You're checking for feof($ipadres) and using fgets($file)
i.e. you keep checking end of file with file pointer $ipadres but reading from file pointer $file which will cause infinite loop and program will crash eventually.
Probably you meant:
while(!feof($ipadres)) {
$iparray = explode("|", fgets($ipadres));
}
OR else use file function which returns all the lines of a file in an array.