I use PHP app in Google appengine,
I'm trying to read a input file and write it to an output file in Storage bucket like below.
$input_file = fopen('gs://mybucket/input.csv','r');
$output_file = fopen('gs://mybucket/output.csv', 'w');
And trying to write some date like
while(!feof($input_file)) {
$csv = fgetcsv($input_file,1024);
if(!$csv[0]){
fclose($output_file); //Close the connection when the loop ends
fclose($input_file);
exit(0);
}
fwrite($output_file, $csv[0]."\r\n");
}
It works perfectly, When i try to upload some data in to input file and it successfully write in to output.csv as well. but if i try more than 5 or 6th time it starts to throw an error in appengine logs like below. Any help to troubleshoot this issue will be highly appreciated!
2015-04-08 21:31:29.006 PHP Fatal error: Undefined class constant 'WRITE_SCOPE' in /base/data/home/runtimes/php/sdk/google/appengine/ext/cloud_storage_streams/CloudStorageWriteClient.php on line 214
Update:
I think this is because of opening 2 file streams at same time,
Did some work around and solved this!
$input_file = fopen('gs://mybucket/input.csv','r');
$array_acc = array();
while(!feof($input_file)) {
$csv = fgetcsv($input_file, 1024);
if($csv[0]) array_push($array_acc, $csv[0]);
}
fclose($input_file); //close the file
$acc_count = count($array_acc);
$output_file = fopen('gs://tool-synclio/output.csv','w'); // Open the output file now
while($acc_count > 0){
fwrite($output_file,$array_acc[$acc_count]."\r\n");
$acc_count --;
}
fclose($output_file);
But, I'm still waiting for some one to give better solution.
You have 2 dollar signs in a variable:
fwrite($output_file, $$csv[0]."\r\n");
Related
I have a piece of code, the issue is the file "data" is over 8GB. This is very memory intensive. I want to reduce the usage of RAM and saw the f_load would be ideal, however, how could i explode this data?
This is my current code:
$data = file_get_contents("data");
$data = explode("|", $data);
foreach ($data as $d) { // rest of code
theoretically, i need to open a pipe, stream and close a pipe How would i go about this?
I've tried using f_open rather than file_get_contents but errors started popping up so i'm doing something wrong and would really like to learn.
You can use stream_get_line to read your data block per block (with | as the delimiter character) : PHP stream_get_line()
$fh = fopen('data', 'r'); // open the file in read-only mode
while( $d = stream_get_line($fh, 1000, '|') ) // read the file until the next |
{
echo $d . PHP_EOL ; // display one block of data
}
fclose($fh); // close the file
In this code today I am trying to update data in bulk. User uploads a CSV file, the system checks if the file type is correct, then fetches the data and updates the data accordingly with the help of a WHILE loop. Have a look at the code below.
if(isset($_POST['upload'])) {
if($_FILES['theFile']['name']) {
$filename = explode(".", $_FILES['theFile']['name']);
if(end($filename) == "csv") {
$handle = fopen($_FILES['theFile']['tmp_name'], "r");
while($file_data = fgetcsv($handle)) {
$id = $DBcon->real_escape_string($file_data[0]);
$newdate = $DBcon->real_escape_string($file_data[1]);
$newdate = strtotime($newdate);
$newdate = date('Y-m-d', $newdate);
$query = "UPDATE IGNORE products SET expirydate='".$newdate."' WHERE id='".$vc."'";
if ($DBcon->query($query)) {
$serror = "Customer Updatation Successful";
} else {
$error = "Something Went Wrong - Contact Support";
}
}
fclose($handle);
$serror = "Successfully Fetched Details";
} else {
$error = "Please select a CSV file";
}
} else {
$error = "Please select a CSV file";
}
}
The thing is everything works fine until we follow the rules. Let's have a look into the CSV file that works just fine.
123,11-06-2019
124,11-06-2019
125,11-06-2019
126,11-06-2019
127,11-06-2019
But when the CSV output does not includes a " , " between the columns it throws unwanted error.
Here's the CSV file example
123 11-06-2019
124 11-06-2019
125 11-06-2019
126 11-06-2019
127 11-06-2019
Getting this error:
Okay
Notice
: Undefined offset: 1 in
D:\server\filepath\htdocs\filename.php
on line
21
Screenshot:
http://prntscr.com/o4r9vi
The line 21 defines the next column in this case.
$newdate = $DBcon->real_escape_string($file_data[1]);
What can be done in this case to avoid/fix the error? I was thinking to throw an error when the file content does not meets certain requirements, such as when the file does not contains certain number of commas defining all the required/expected columns. But how can I do that exactly? I am pretty new to this part of PHP.
Thank you for your time.
You can start simply by doing a count on your $file_data variable to ensure that the number of fields you expect exist. You could also go further and check that the type of data you expect is valid. There is a library to do this called CSVUtils. Effectively the best all round approach is to first lint the CSV to ensure it conforms to RFC 4180. It should also be noted that there are issues with respect to PHPs fputcsv so if the CSVs are being generated by PHP in the first place linting could be an issue in some instances. To add to this if you wanted you could even check the mime type of the file first and also turn your notices into an exception. The solution to this can be found here.
Example of count.
if (count($file_data) < 2) {
// do something (e.g. fill error array or throw exeption)
}
I've got a php problem.
I've got a php file that reads data from a .txt file.
This works, with this code:
$filename= "deadlines.txt";
$fp = fopen($filename,"r");
$content = fread($fp, filesize($filename));
$rawArray = setRawArray($content);
$epochAndTitleArray = toEpoch($rawArray);
Now, I want to make it so that this stuff is executed every second, not just once at the start.
So, I tried to fit it into a function, like this:
$filename= "deadlines.txt";
$fp = 0;
$content = 0;
$rawArray = 0;
$epochAndTitleArray = 0;
function readFile(){
$GLOBALS['fp'] = fopen($GLOBALS['filename'], "r");
$GLOBALS['content'] = fread($GLOBALS['fp'], filesize($GLOBALS['filename']));
$GLOBALS['rawArray'] = setRawArray($GLOBALS['content']);
$GLOBALS['epochAndTitleArray'] = toEpoch($GLOBALS['rawArray']);
}
In this case I'm working with globals, before, I did it without, and also left out the lines before the function itself. This was incorrect I think, so I added the globals.
Now, this doesn't work.
It gives me the following error:
Fatal error: Cannot redeclare readFile() in .....on line 28,
this line 28 is the line of the closing } at the end of the function.
Can you guys help me in completing this task?
Thanks already!
readfile is a defined function in php , you cannot redeclare it or redeclare any function using the same name .
for more reference about how to declare valid functions in php
PHP does not support function overloading, nor is it possible to
undefine or redefine previously-declared functions.
Rename 'readFile' to another, readfile() is predefined function 'http://php.net/manual/kr/function.readfile.php'
I tried to write a counter for visits of my page, which only counts each IP once per day.
I transcribed a found code to understand how it works and to implement some other features later.
Unfortunately my new code doesn't work.
Code I made from it:
<?php
$timeReset = 86400; //24h
$ipPath = "ipsafe.txt";
$countPath = "counts.txt";
$timePath = "time.txt";
$ip = $REMOTE_ADDR;
$time = time();
$resetTime = $time+$timeReset;
$timeFile = fopen($timePath, "r+");
$timeData = fread($timeFile, filesize($timePath));
//if resetTime is passed -> Reset
if($time >= $timeData) {
//clear ipSafe
$ipFile1 = fopen($ipPath, "w+");
fclose($ipFile);
//set new resetTime
rewind($timeData);
fwrite ($timeData, $resetTime);
}
fclose($timeFile);
//creat IP-List
$ipData = file($ipPath);
while (list ($line_num, $line) = each ($ipData)) {
$digits = explode(",",$line);
}
//if IP was not saved in last timeframe
if(!in_array($ip, $digits))
{
//write new IP
$ipFile2=fopen($ipPath, "a+");
#flock($ipFile1,2);
fwrite($ipFile2, "$ip".",");
#flock($ipFile1,3);
fclose($ipFile2);
//increment counter
$countFile = fopen($countPath,"r+");
$countData = fread($countFile, filesize($countPath);
rewind($countFile);
$countData++;
fwrite($countFile, $countData);
fclose($countFile);
}
?>
with the following questions:
what's wrong with my code?
flock is used to manage the access to the files, but why shall I use different names for the same file?
Thanks for your suggestions.
EDIT:
Sorry for being so unspecific in explaining the problem. After integrating a debugger, I changed "REMOTE_ADDR" to "$_SERVER['REMOTE_ADDR']" so I fixed 1 error. Now i got the following problems:
Warning: fopen(time.txt): failed to open stream: No such file or directory in .../counter.php on line 15
But the file is in the same directory as counter.php - of course the fread and filesize failed too
Warning: fopen(ipsafe.txt): failed to open stream: Permission denied in .../counter.php on line 20
The file doesn't exist yet, but I thought "fopen($ipPath, "w+");" creats it if it doesn't exist.
Fatal error: Call to undefined function fb() in .../counter.php on line 26
Doesn't "fb($timeFile);" send the value to firePHP?
Not doing a full code review (which would result in an almost full rewrite of your code), here are the things I would recommend changing to fix your listed issues:
The following line uses $REMOTE_ADDR, which is undefined.
$ip = $REMOTE_ADDR;
You most likely meant to use $_SERVER['REMOTE_ADDR'] (which you confirmed with your edit), so the line should read:
$ip = $_SERVER['REMOTE_ADDR'];
You state you're receiving an error attempting to open the time.txt file: "No such file or directory". You also say that the file exists in the same directory as counter.php - and yet you're still receiving this error. This can mean one of two things; the first, the file doesn't exist and you're mistaken; the second, your counter.php file is in a different directory than whatever script is executing it - this would mean that the time.txt file would need to be in the directory of the executing script. I would recommend two things for this.
The first, if the file actually exists, use an absolute path for it:
$timePath = '/path/to/your/files/time.txt';
The second, verify if the file exists (in code) and if not, create it. fopen() with r/r+ flags does not create a file if it doesn't exist, so this needs to be manual. Try changing the code to the following:
if (!file_exists($timePath)) touch($timePath);
$timeFile = fopen($timePath, 'r+');
if ($timeFile) {
// the rest of your code
}
You say you're receiving an error when opening the ipsafe.txt file: "Permission denied". This is a fun one; Though the line numbers don't match with your sample code, I believe these are the lines in question:
//clear ipSafe
$ipFile1 = fopen($ipPath, "w+");
fclose($ipFile);
The first issue is that you never use $ipFile1 in your code (except erroneously locking/unlocking it; see below for that info). The second is that you call fclose($ipFile);. There is no handle named $ipFile! You can remove these lines of code!
You attempt to rewind the file pointer in your time.txt file to write the new time, however, you're using the value you read-in from the file $timeData - not the file itself $timeFile. Update the lines to the following:
rewind($timeFile);
fwrite($timeFile, $resetTime);
You read in the file ipsafe.txt, again, to look through the full list of IP addresses - if the current user's IP isn't in it, you add it. This may be a logic issue, but you're only checking the IP addresses of the last line of the file - but you still iterate through every line of the item. Per the comment in the code: //if IP was not saved in last timeframe, I'm going to assume that this is what you're intending to do - but just in case - I would recommend moving the full if-statement block that appears below the loop to inside the loop.
Here's the fun block:
//write new IP
$ipFile2=fopen($ipPath, "a+");
#flock($ipFile1,2);
fwrite($ipFile2, "$ip".",");
#flock($ipFile1,3);
fclose($ipFile2);
You open the file into $ipFile2 (2), then you lock a completely different file $ipFile1 (1), write to 2, unlock 1, and then close 2. I think that you're intentions were to lock/unlock 2, which would make perfect sense here, so I'd recommend changing $ipFile1 to $ipFile2 in those lines.
That should about do it (potentially). After everything, the code should resemble something similar to:
<?php
$timeReset = 86400; //24h
$ipPath = "ipsafe.txt";
$countPath = "counts.txt";
$timePath = "time.txt";
$ip = $_SERVER['REMOTE_ADDR'];
$time = time();
$resetTime = $time+$timeReset;
if (!file_exists($timePath)) touch($timePath);
$timeFile = fopen($timePath, "r+");
if ($timeFile) {
$timeData = fread($timeFile, filesize($timePath));
//if resetTime is passed -> Reset
if($time >= $timeData) {
//set new resetTime
rewind($timeFile);
fwrite ($timeFile, $resetTime);
}
fclose($timeFile);
//creat IP-List
$ipData = file($ipPath);
while (list ($line_num, $line) = each ($ipData)) {
$digits = explode(",",$line);
}
//if IP was not saved in last timeframe
if(!in_array($ip, $digits)) {
//write new IP
$ipFile2=fopen($ipPath, "a+");
#flock($ipFile2, LOCK_EX);
fwrite($ipFile2, "$ip".",");
#flock($ipFile2, LOCK_UN);
fclose($ipFile2);
//increment counter
if (!file_exists($countPath)) touch($countPath);
$countFile = fopen($countPath,"r+");
if ($countFile) {
$countData = fread($countFile, filesize($countPath);
if (empty($countData)) $countData = 0; // initialize the counter to '0' if it hasn't been set
rewind($countFile);
$countData++;
fwrite($countFile, $countData);
fclose($countFile);
}
}
}
?>
Now, I also have one additional logic issue regarding your $resetTime variable. Ideally, incrementing by a single day is to add 24 hours to a given timestamp (24 hours = 86400 seconds). However, daylight savings time will break this. Instead of doing the following:
$resetTime = time() + 86400;
Try going a different route (if you're worried about this, that is):
$resetTime = strtotime('+1 day');
Hope you help me... I've been at this for the past 2 days and have to admit that I'm stumped.
The OS I'm on is Ubuntu 9.10 Karmic.
I successfully installed and tested Mapserver. For my class project, I have a php script that I am using to create a layer see below....
The error I get when run the script on a cmd line prompt:
Warning: [MapServer Error]: msProcessProjection(): no system list, errno: 2
in /var/www/mapserverdocs/ms4w/apps/world/mapscripts/staticwms.php on line 16
Warning: Failed to open map file static.map in /var/www/mapserverdocs/ms4w/apps/world/mapscripts/staticwms.php on line 16
Fatal error: Call to a member function owsdispatch() on a non-object in /var/www/mapserverdocs/ms4w/apps/world/mapscripts/staticwms.php on line 18
PHP SCRIPT:
<?php
if (!extension_loaded("MapScript")) dl("php_mapscript");
$request = ms_newowsrequestobj();
foreach ($_GET as $k=>$v) {
$request->setParameter($k, $v);
}
$request->setParameter("VeRsIoN","1.0.0");
ms_ioinstallstdouttobuffer();
$oMap = ms_newMapobj("static.map");
$oMap->owsdispatch($request);
$contenttype = ms_iostripstdoutbuffercontenttype();
if ($contenttype == 'image/png') {
header('Content-type: image/png');
ms_iogetStdoutBufferBytes();
} else {
$buffer = ms_iogetstdoutbufferstring();
echo $buffer;
}
ms_ioresethandlers();
?>
I made the directory and files world wide rwx just to make sure it was not a permissions issue
Any help would be greatly appreciated!!
Thanks
Chris
As meagar said, the issue is probably that this line:
$oMap = ms_newMapobj("static.map");
is unable to find "static.map". The current working directory of PHP is very often not what you'd expect it to be. Try making the path be relative to the current script. If static.map is in the same directory as static.map, try this code:
$mapPath = dirname(__FILE__).'/static.map';
$oMap = ms_newMapobj($mapPath);
$oMap->owsdispatch($request);
if static.map is at, let's say, /var/www/mapserverdocs/ms4w/apps/world/mapfiles/static.map, then try:
$mapPath = dirname(__FILE__).'/../static.map';
$oMap = ms_newMapobj($mapPath);
$oMap->owsdispatch($request);
Notice the */../*static.map. dirname(__FILE__) will return the name of the directory of the PHP file you place that code in.