Opening a CSV file from SFTP server in PHP - php

We currently open a csv from our server and then do some stuff with the data like this:
$CSVfile = fopen('filename.csv', "r");
if($CSVfile !== FALSE) {
$count = 0;
while(! feof($CSVfile)) {
$data = fgetcsv($CSVfile, 5000, ",");
if ($count > 0 && !empty($data)) {
// Do something
}
}
}
We need to change the system as the file will now be hosted on an external server so we need to SFTP into it to retrieve the file. I've installed phpseclib and got the connection working but it just keeps echoing the file contents on the screen. I have it set up like this:
include 'vendor/autoload.php';
$sftp = new \phpseclib\Net\SFTP('SERVER');
if (!$sftp->login('USERNAME', 'PASSWORD')) {
exit('Login Failed');
} else {
$file = $sftp->fetch('FILE_LOCATION');
}
$CSVfile = fopen($file, "r");
if($CSVfile !== FALSE) {
$count = 0;
while(! feof($CSVfile)) {
$data = fgetcsv($CSVfile, 5000, ",");
if ($count > 0 && !empty($data)) {
// Do Something
}
}
}
How do I get the new system to read the file contents and do something with it rather than just showing all the file contents?

As #verjas commented already, there's no fetch method in phpseclib.
If you want to download a file to a string, use SFTP::get:
$contents = $sftp->get("/remote/path/file.csv");
You can then use str_getcsv to parse the contents. According to a contributed note at the function documentation, this should do:
$data = str_getcsv($contents, "\n"); // parse the rows
foreach ($data as $line)
{
$row_data = str_getcsv($line); // parse the items in rows
}

Related

Automating CSV conversion from iso-8859-2 to utf-8

I have quite a few CSV files that are unfortunately encoded with iso-8859-2 (according to Brackets). I would like to iterate over these files with PHP and convert them.
I found https://csv.thephpleague.com/9.0/converter/charset/ but the way I can use the conversion function is uncertain to me.
Their example code
use League\Csv\CharsetConverter;
$csv = new SplFileObject('/path/to/french.csv', 'r');
$csv->setFlags(SplFileObject::READ_CSV | SplFileObject::SKIP_EMPTY);
$encoder = (new CharsetConverter())->inputEncoding('iso-8859-15');
$records = $encoder->convert($csv);
This is my code so far that is part of a form to upload one file and save the contents to the database for testing. It of course saves the text in the incorrect format.
$db = ConnectDB::getConnection('address_dtb');
$sql = " ... ";
$stmt = $db->prepare($sql);
$rowCount = 0;
$temp_name = $_FILES['adresscsv']['tmp_name'];
$file_handle = fopen($temp_name, 'r');
while (($items = fgetcsv($file_handle, 1000, ';')) !== FALSE) {
if($flag) { $flag = false; continue; }
$stmt->execute($items);
$rowCount++;
}
fclose($file_handle);
ConnectDB::closeConnection($db);
What is the correct way to use the PHP CSV library above to iterate over locally saved files in a for loop to automate the process?
I ended up using iconv as hinted.
$files = glob('address/*.csv');
foreach ($files as $csv) {
$file_data = file_get_contents($csv);
$utf8_file_data = iconv('Windows-1250', 'UTF-8', $file_data);
file_put_contents($csv, $utf8_file_data);
}
You do not have to use a library. There is a function in PHP that can do that iconv

Loading CSV file with 2.5 million records

I have a large CSV file with every postcode in the UK, it comes to 2,558,797 records and I need to import it, manipulate the data by sorting it into a multi-dimensional array before saving the data in the multi-dimensional array in the database.
The problem is if I try and access the whole file I get an allowed memory exceeded exception. I can access about 128,000 records in any one go. Is there a way I can split the task down so that I can process the whole file? I've tried looking at fseek but that uses the number of bytes, not the number of rows, and I don't know how many bytes 128,000 rows is.
How can I process the entire file without hitting the memory limit? I've been trying to get this working for the last 6 hours and I've not had any joy.
This is my code so far:
// This script takes a long time to run
ini_set('max_execution_time', 300);
// First we need to verify the files that have been uploaded.
$file = Validation::factory($_FILES);
$file->rule('import_update_file', 'Upload::not_empty');
$file->rule('import_update_file', 'Upload::valid');
$file->rule('import_update_file', 'Upload::size', array(':value', '8M'));
$file->rule('import_update_file', 'Upload::type', array(':value', array('zip')));
if (Request::current()->method() == Request::POST && $file->check())
{
$file_name = date('Y-m-d-').'update.zip';
$dir = Upload::save($file['import_update_file'], $file_name);
if ($dir === false)
{
throw new Kohana_Exception('Unable to save uploaded file!', NULL, 1);
}
$zip = new ZipArchive;
if ($zip->open($dir) !== TRUE)
{
throw new Kohana_Exception('Unable to open uploaded zip file! Error: '.$res, NULL, 1);
}
$zip->extractTo(realpath(Upload::$default_directory), array('localauthority.csv', 'postcode.csv'));
$zip->close();
if( ! file_exists(realpath(Upload::$default_directory).DIRECTORY_SEPARATOR.'localauthority.csv') OR
! file_exists(realpath(Upload::$default_directory).DIRECTORY_SEPARATOR.'postcode.csv'))
{
throw new Kohana_Exception('Missing file from uploaded zip archive! Expected localauthority.csv and postcode.csv', NULL, 1);
}
$local_authorities = Request::factory('local_authority/read')->execute();
// We start by combining the files, sorting the postcodes and local authority names under the local authority codes.
$update = array();
if (($fp = fopen(realpath(Upload::$default_directory).DIRECTORY_SEPARATOR.'localauthority.csv', 'r')) === FALSE)
{
throw new Kohana_Exception('Unable to open localauthority.csv file.', NULL, 1);
}
while (($line = fgetcsv($fp)) !== FALSE)
{
// Column 0 = Local Authority Code
// Column 1 = Local Authority Name
$update[$line[0]] = array(
'name' => $line[1],
'postcodes' => array()
);
}
fclose($fp);
unlink(realpath(Upload::$default_directory).DIRECTORY_SEPARATOR.'localauthority.csv');
if (($fp = fopen(realpath(Upload::$default_directory).DIRECTORY_SEPARATOR.'postcode.csv', 'r')) === FALSE)
{
throw new Kohana_Exception('Unable to open postcode.csv file.', NULL, 1);
}
$i = 1;
while (($line = fgetcsv($fp)) !== FALSE && $i <= 128000)
{
$postcode = trim(substr($line[0], 0, 4));
echo "Line ".sprintf("%03d", $i++) . ": Postcode: ".$line[0]."; Shortened Postcode: ".$postcode."; LAC: ".$line[1]."<br>";
// Column 0 = Postcode
// Column 1 = Local Authority Code
if ( ! array_key_exists($line[1], $update))
{
echo $line[1]." not in array<br>";
continue;
}
if ( ! in_array($postcode, $update[$line[1]]['postcodes']))
{
$update[$line[1]]['postcodes'][] = $postcode;
}
}
fclose($fp);
unlink(realpath(Upload::$default_directory).DIRECTORY_SEPARATOR.'postcode.csv');
echo '<pre>'; var_dump($update); echo '</pre>';
}
else
{
throw new Kohana_Exception('Invalid file uploaded!', NULL, 1);
}

Modify a csv file line by line

I have a big file that I want to modify every line in it.
I want to use PHP to do it quickly :
My file is CSV file ;
20010103,02,00,00,0.9496
20010103,03,00,00,0.9504
20010103,04,00,00,0.9499
I want to make it like this to be able to use it late with Highchart:
[Date.UTC(2001,01,03,02,00,00),0.9496],
[Date.UTC(2001,01,03,03,00,00),0.9504],
[Date.UTC(2001,01,03,04,00,00),0.9499],
How canI loop every line and make this modification ?
See the fgetcsv and fputcsv PHP functions. It will basically be something like:
if (($handle1 = fopen("input.csv", "r")) !== FALSE) {
if (($handle2 = fopen("output.csv", "w")) !== FALSE) {
while (($data = fgetcsv($handle1, 1000, ",")) !== FALSE) {
// Alter your data
$data[0] = '...';
// Write back to CSV format
fputcsv($handle2, $data);
}
fclose($handle2);
}
fclose($handle1);
}
Try this code:
<?php
$filename = 'info.csv';
$contents = file($filename);
foreach($contents as $line) {
$data = explode(",",$line);
$val = "[Date.UTC(".substr($data[0],0,4).",".(substr($data[0],4,2)).",".substr($data[0],6,2).",".$data[1].",".$data[2].",".$data[3]."),".$data[4]."],";
}
?>

How to generically open a file in php

I've looked for questions on this topic, but failed to get what I'm looking for. This is for C++, I need similar for PHP. This is for including php files, I just want to read a CSV file.
I have this:
if(file_exists("data.csv")){
echo "CSV file found";
$csv_data = file_get_contents("data.csv");
$lines = explode("\n", trim($csv_data));
$array = array();
foreach ($lines as $line){
$array[] = str_getcsv($line);
}else {echo "File not found";}
But I want to NOT specify the file name - i.e. generically load/read/open the file.
Is there any simple why of doing that? Doesn't make sense, but I was told to not have anything hard coded in my PHP script.
Thanks in advance.
use fgetcsv
if(file_exists("data.csv")){
echo "CSV file found";
$handle = fopen("data.csv", "r");
if(!$handle) die("Could not open file!");
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
$row++;
for ($c=0; $c < $num; $c++) {
echo $data[$c] . "<br />\n";
}
}
fclose($handle);
}else {echo "File not found";}
If you may not have anything hard coded in your script, you need to put those hardcoded things into some sort of external config file. You will have to hardcode the name of that config file into your bootstrap or whatever comes first in your application. Once the config is loaded, make the configuration data available in the places where it is needed. Not hardcoding configuration data into your code will allow you to create more reusable components and code, e.g. CSV Reader that can read any CSV file instead of a CSV Reader that can only read that one particular CSV file hardcoded into it.
Example:
// config.php
<?php
return array(
'csvFile' => '/path/to/file.csv',
…
);
// bootstrap.php
<?php
$config = include '/path/to/config.php';
…
// someFile.php
<?php
include '/path/to/bootstrap.php';
$file = new SplFileObject($config['csvFile']);
$file->setFlags(SplFileObject::READ_CSV);
foreach ($file as $row) {
// Do something with values
}
Put your code into a function...
function open_file($file_name)
{
if (!file_exists($file_name))
{
return false;
}
$csv_data = file_get_contents($file_name);
$lines = explode("\n", trim($csv_data));
$array = array();
foreach ($lines as $line)
{
$array[] = str_getcsv($line);
}
return $array;
}

Need PHP script to decompress and loop through zipped file

I am using a fairly straight-forward script to open and parse several xml files that are gzipped. I also need to do the same basic operation with a ZIP file. It seems like it should be simple, but I haven't been able to find what looked like equivalent code anywhere.
Here is the simple version of what I am already doing:
$import_file = "source.gz";
$sfp = gzopen($import_file, "rb"); ///// OPEN GZIPPED data
while ($string = gzread($sfp, 4096)) { //Loop through the data
/// Parse Output And Do Stuff with $string
}
gzclose($sfp);
What would do the same thing for a zipped file?
If you have PHP 5 >= 5.2.0, PECL zip >= 1.5.0 then you may use the ZipArchive libraries:
$zip = new ZipArchive;
if ($zip->open('source.zip') === TRUE)
{
for($i = 0; $i < $zip->numFiles; $i++)
{
$fp = $zip->getStream($zip->getNameIndex($i));
if(!$fp) exit("failed\n");
while (!feof($fp)) {
$contents = fread($fp, 8192);
// do some stuff
}
fclose($fp);
}
}
else
{
echo 'Error reading zip-archive!';
}
There is a smart way to do it with ZipArchive. You can use a for loop with ZipArchive::statIndex() to get all the information you need. You can access the files by their index (ZipArchive::getFromIndex()) or name (ZipArchive::getFromName()).
For example:
function processZip(string $zipFile): bool
{
$zip = new ZipArchive();
if ($zip->open($zipFile) !== true) {
echo '<p>Can\'t open zip archive!</p>';
return false;
}
// As long as statIndex() does not return false keep iterating
for ($idx = 0; $zipFile = $zip->statIndex($idx); $idx++) {
$directory = \dirname($zipFile['name']);
if (!\is_dir($zipFile['name'])) {
// file contents
$contents = $zip->getFromIndex($idx);
}
}
$zip->close();
}

Categories