I am trying to import an XLS file into PHP, where I can then edit the information and import it into mySQL. I have never done anything related to this, so I am having a hard time grasping how to approach it.
I have looked at a few open source projects:
PHP Excel Reader
ExcelRead
PHPExcel
None of these options perfectly fit what I want to do or maybe I just haven't gone deep enough into the documentation.
There are some things that needed to be taken into consideration. The XLS file cannot be converted into any other file format. This is being made for ease-of-access for nontechnical users. The XLS file is a report generated on another website that will have the same format (columns) every time.
For example, every XLS file with have the same amount of columns (this would be A1):
*ID |Email |First Name |Last Name |Paid |Active |State |Country|*
But, there are more columns in the XLS file than what is going to be imported into the DB.
For example, the rows that are being imported (this would be A1):
*ID |Email |First Name |Last Name |Country*
I know one of two ways to do edit the data would be A. Use something like PHPExcel to read in the data, edit it, then send it to the DB or B. Use something like PHPExcel to convert the XLS to CSV, do a raw import into a temp table, edit the data, and insert it into the old table.
I have read a lot of the PHPExcel documentation but, it doesn't have anything on importing into a database and I don't really even know where to start with editing the XLS before or after importing.
I have googled a lot of keywords and mostly found results on how to read/write/preview XLS. I am looking for advice on the best way of doing all of these things in the least and simplest steps.
See this article on using PHP-ExcelReader, in particular the short section titled "Turning the Tables".
Any solution you have will end up looking like this:
Read a row from the XLS (requires an XLS reader)
Modify the data from the row as needed for your database.
Insert modified data into the database.
You seem to have this fixation on "Editing the data". This is just PHP--you get a value from the XLS reader, modify it with PHP code, then insert into the database. There's no intermediate file, you don't modify the XLS--it's just PHP.
This is a super-simple, untested example of the inner loop of the program you need to write. This is just to illustrate the general pattern.
$colsYouWant = array(1,2,3,4,8);
$sql = 'INSERT INTO data (id, email, fname, lname, country) VALUES (?,?,?,?,?)';
$stmt = $pdo->prepare($sql);
$sheet = $excel->sheets[0];
// the excel reader seems to index by 1 instead of 0: be careful!
for ($rowindex=2; $rowindex <= $sheet['numRows']; $rowindex++) {
$xlsRow = $sheet['cells'][$rowindex];
$row = array();
foreach ($colsYouWant as $colindex) {
$row[] = $xlsRow[$colindex];
}
// now let's "edit the row"
// trim all strings
$row = array_map('trim', $row);
// convert id to an integer
$row[0] = (int) $row[0];
// capitalize first and last name
// (use mb_* functions if non-ascii--I don't know spreadsheet's charset)
$row[2] = ucfirst(strtolower($row[2]));
$row[3] = ucfirst(strtolower($row[3]));
// do whatever other normalization you want to $row
// Insert into db:
$stmt->execute($row);
}
Related
Let I have created a table named after pb00k via phpMyAdmin which SQL is as bellow:
CREATE TABLE `pb00k` (
`k3y` int(255) NOT NULL AUTO_INCREMENT,
`n4m3` text NOT NULL,
`numb3r` text NOT NULL,
PRIMARY KEY (`k3y`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
So it generates a pb00k.frm file in mysql/data/ which contains this structure. Now I want to get this structure from this file using PHP.
Is it possible?
Yes it's possible to recover at least part of information. (For the benefit of other readers the poser of the question is already aware that there are easier ways to get the column metadata).
The challenge is that .frm files are not so well documented because any need to decipher them by the general community is pretty rare. Also the format of the files may vary with the operating system.
However, by viewing the files with hexdump or a similar utility you can see partly what is going on. Then you better informed to read the files in a PHP program and decode the raw binary data.
I did this as an exercise some time back, and I was able to recover number of columns, column names and column types.
Below is a sample to show how to extract column names. My .frm was for a table names "stops", but you can substitute your own .frm.
<?php
$fileName = "stops.frm";
// read file into an array of char
//---------------------------------
$handle = fopen($fileName, "rb");
$contents = fread($handle, filesize($fileName));
fclose($handle);
$fileSize=strlen($contents); // save the filesize fot later printing
// locate the column data near the end of the file
//-------------------------------------------------
$index = 6; // location of io_size
$io_size_lo = ord($contents[$index]);
$io_size_hi = ord($contents[$index+1]);
$io_size = $io_size_hi *0x100 + $io_size_lo; // read IO_SIZE
$index = 10; // location of record length
$rec_len_lo = ord($contents[$index]);
$rec_len_hi = ord($contents[$index+1]);
$rec_len = $rec_len_hi * 0x100 + $rec_len_lo; // read rec_length
// this formula uses io_size and rec_length to get to column data
$colIndex = ( ( (($io_size + $rec_len)/$io_size) + 1) * $io_size ) + 258;
$colIndex -= 0x3000; // this is not documented but seems to work!
// find number of columns in the table
//-------------------------------------------------
echo PHP_EOL."Col data at 0x".dechex($colIndex).PHP_EOL;
$numCols = ord($contents[$colIndex]);
//Extract the column names
//--------------------------------------
$colNameIndex = $colIndex+0x50; //0X50 by inspection
echo "Col names at 0x".dechex($colNameIndex).PHP_EOL;
$cols=array();
for ($col = 0; $col < $numCols; $col++){
$nameLen = ord($contents[$colNameIndex++]); // name length is at ist posn
$cols[]['ColumnName']= substr($contents,$colNameIndex,$nameLen-1); // read the name
$colNameIndex+=$nameLen+2; // skip ahead to next name (2 byte gap after \0)
}
print_r($cols);
This should get you started. I will add to this when I have time in the coming days if you think is heading in the right direction.
EDIT. I updated the code so it should work for any .frm file (from Table). For sure there is a free tool to recover mySQL (based on innoDB engine) available at https://github.com/twindb/undrop-for-innodb. Having read through the code and the associated blogs, they are not using the .FRM files for recovery. The same table information is also stored in the innoDB dictionary and they are using this to recover table formats etc.
There is also a way to read the .FRM files content. This described here https://twindb.com/how-to-recover-table-structure-from-frm-files-online/. However, they are using mySQL to read the .frm files and recreating tables from there.
There is also a utility a package of utilities found here https://www.mysql.com/why-mysql/presentations/mysql-utilities/ that contains a .frm reader. This was made by Oracle, who are the only people who know the format of the .frm files! The utility is free so you can download it.
Oracle publish some information on the format of .frm files https://dev.mysql.com/doc/internals/en/frm-file-format.html, but it is both incomplete and incorrect! See this previous Stack question.
https://dba.stackexchange.com/questions/208198/mysql-frm-file-format-how-to-extract-column-info
Now after all that if you still want to try to parse the .frm files yourself for fun or for learning, then you need to be patient and spend time unravelling quite a complicated structure. If you want to keep trying that is OK but send me your .FRM file ( to sand_groper80#hotmail.com) so I can check it out and I will send you some PHP code in a few days that will extract some additional information like datatype and display sizes.
I have spent the last days trying to solve this issue. I have a spreadsheet with a couple tabs (sheets). The last tab has the values input in all other tabs. Those values are populated automatically by using formulas like:
=$'otherSheetName'.A1
If I leave calculateFormulas = true those values are read as null.
If I set calculateFormulas = false the values are read as the literal formula.
I was firstly using LaravelExcel but I gave up from finding a solution for that, so I moved to PHPExcel. Solutions for Laravel Excel will be appreciated too though.
Thanks
Update:
How I am reading values in PHPExcel:
$inputFileType = PHPExcel_IOFactory::identify($file->getRealPath());
$objReader = PHPExcel_IOFactory::createReader($inputFileType);
$objReader->setLoadSheetsOnly("MySheet");
$objPHPExcel = $objReader->load($file->getRealPath());
$rows = $objPHPExcel->getActiveSheet()->toArray(null,true,true,true);
Then, I iterate over the array $rows and extract the values that I want.
You need to load all the dependency sheets that are used in those formulae, though you don't necessarily need to load every worksheet, but can pass a list of sheet names to load as an array
$objReader->setLoadSheetsOnly(["MySheet", "OtherSheet1", "OtherSheet2"]);
You don't need to iterate over any other worksheets, but can set the active worksheet to whichever sheet you want
$objReader->setActiveSheetIndexByName("MySheet");
and then any calls to the active sheet ($objPHPExcel->getActiveSheet()->...) to access cells will be a reference to that worksheet.
How can i import UTF-8 data form Movielens to MySql.
I get the data from http://grouplens.org/datasets/movielens/ and for my recommender system Thesis purpose, i just want the 100K and Tag Gnome data only.
I've been searching on google and in this forum and i don't find anything about importing these files to MySQl. Myself, currently using PhpMyAdmin for managing MySQL, so if anybody know how to easily import those files to MySQL.
I'm fine if you guys recommend me to iterate it one by one using php, but please explain to me the code.
You'll need to write some custom code to import all of their data into MySQL. Dumbest answer on Stack Overflow ever, right?
So they provide a set of flat files, each described in the README.
README
allbut.pl
mku.sh
u.data
u.genre
u.info
u.item
u.occupation
u.user
u1.base
u1.test
u2.base
u2.test
u3.base
u3.test
u4.base
u4.test
u5.base
u5.test
ua.base
ua.test
ub.base
ub.test
In a nutshell:
Make your own database and tables in MySQL.
Programatically open a file and parse each line to SQL.
Import the SQL into MySQL.
???
Profit!
Yeah, I know I still haven't really told you anything, let's do one and you can hopefully do the others.
I'll do u.genre, because I'm lazy and it is easy.
Make a new table, I'll assume you know how to make tables and such.
u.genre has two things: a genre and an id.
unknown|0
Action|1
...etc...
So your table should have two fields.
You'll use two data types: https://dev.mysql.com/doc/refman/5.7/en/data-types.html
id - unsigned TINYINT
TINYINT unsigned is 0 to 255
genre - VARCHAR(20)
VARCHAR 20 is up to 20 characters, their longest is "Documentary" so that'll give you a bit of extra room if they add a new one.
Open the file get the contents: https://secure.php.net/manual/en/function.file-get-contents.php
$filecontents = file_get_contents("u.genre");
Now let's split up the file by line: https://secure.php.net/manual/en/function.explode.php
$genres = explode("\n", $filecontents);
Now we'll loop through the $genres using foreach and explode again: https://secure.php.net/manual/en/control-structures.foreach.php
foreach ($genres as &$row) {
list($genre,$id) = explode("|",$row);
# more here later
}
Now let's just output SQL, skipping if either of the fields are empty.
if ($genre!="" && $id!=="") {
print "INSERT INTO genre (genre,id) VALUES ($genre,$id);\n";
}
Put it all together...
<?php
$filecontents = file_get_contents("u.genre");
$genres = explode("\n", $filecontents);
foreach ($genres as &$row) {
list($genre,$id) = explode("|",$row);
if ($genre!="" && $id!=="") {
$sql = "INSERT INTO genre (genre,id) VALUES ($genre,$id);\n";
print $sql;
# Insert each into your DB here.
}
}
?>
Save it and run it from the commandline or put it in a browser for no good reason.
There are too many resources out there showing how to insert data into MySQL, so I'll leave it at this. Everyone's database setup is a bit different, so writing it up for my particular setup won't help you.
I have a rough and complete working CSV transformer. The way my current system works is it imports the CSV file into an SQL database table with static column names, and exports only specific (needed) columns. This system works great but is only specific to one type of CSV file (because the column names are pre-defined.) I'm wondering how I can make this universal. Instead of having it insert column1, column2, column3. I want to insert Spreadsheet Column1, Spreadsheet Column2, Spreadsheet Column3, etc. How would I go about pulling the column names from the CSV file, and creating a new table in the database with the column names being those from the first row of the CSV file.
The current system:
Client uploads CSV file.
A table is created with predefined column names (column 1, column 2, column 3)
Using LOAD DATA INFILE -> PHP scripts will insert the information from the CSV file into the recently created table.
The next query that is ran is simply something along the lines of taking only specific columns out of the table and exporting it to a final CSV file.
The system that would be ideal:
Client uploads CSV file.
PHP scripts read the CSV file and takes only the first row (column names), after taking these column names, it'll create a new table based on the column names.
PHP scripts now use LOAD DATA INFILE.
The rest is the same as current system.
Current code:
import.php
include("/inc/database.php");
include("/inc/functions.php");
include("/inc/data.php");
if($_SERVER['REQUEST_METHOD'] == 'POST'){
$string = random_string(7);
$new_file_name = 'report_'. $string .'.csv';
$themove = move_uploaded_file($_FILES['csv']['tmp_name'], 'C:/xampp/htdocs/uploads/'.$new_file_name);
mysql_query("CREATE TABLE report_". $string ."(". $colNames .")") or die(mysql_error());
$sql = "LOAD DATA INFILE '/xampp/htdocs/uploads/report_". $string .".csv'
INTO TABLE report_". $string ."
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(". $insertColNames .")";
$query = mysql_query($sql) or die(mysql_error());
header('Location: download.php?dlname='.$string.'');
}
data.php (shortened most of this. In reality there are about 200 columns going in, twenty-thirty coming out)
<?php
$colNames = "Web_Site_Member_ID text,
Master_Member_ID text,
API_GUID text,
Constituent_ID text";
$insertColNames = "Web_Site_Member_ID,
Master_Member_ID,
API_GUID,
Constituent_ID";
$exportNames = "Web_Site_Member_ID, Date_Membership_Expires, Membership, Member_Type_Code";
?>
functions.php just includes the block of code for generating a random string/file name.
For CSV file reading please look at the fgetcsv() function. You should easily be able to extract a row of data and access each individual field in the resulting array for your column header definitions.
I am trying to export my database to .csv the exporting done ,but it put all the fields
in one column (without separating)
the code :
$sql = "SELECT ARP_name ,Student_name ,institute ,id ,Major from istyle ";
$results=mysql_query($sql);
$filename = "uploaded/".time().".csv";
$handle = fopen($filename, 'w+');
fputcsv($handle, array_keys("ARP_name","Student_name","institute"));
while($row = mysql_fetch_assoc($results))
{
fputcsv($handle, array($row["ARP_name"], $row["Student_name"],$row["institute"]));
}
the result is :
Since your csv export looks perfectly fine on first glance I assume you mean that the spreadsheet you are trying to import the data into puts everything into one single column?
There are various settings you can adjust to describe the csv format details when importing data into spreadsheet applications. Check the applications preferences dialog. OpenOffice and LibreOffice Calc applications come with a great wizard for this.
For separate columns use ' \t ' for separate rows use ' \n '