I have this code which I use in order to upload a CSV file.
$handle = fopen($_FILES['filename']['tmp_name'], "r");
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$import="INSERT into matching(date, time, location, epos_id, rbpos_id, type_of_payment, basket_information, user_id, points) values('$data[0]', '$data[1]', '$data[2]', '$data[3]', '$data[4]', '$data[5]', '$data[6]', '$data[7]', '$data[8]')";
mysql_query($import) or die(mysql_error());
$query = 'SELECT * FROM users WHERE ID="'.$data[7].'"';
$result = mysql_query($query) or die(mysql_error());
while($row = mysql_fetch_array($result)){
$id_user = $row['user_id'];
$phone = $row['phone'];
echo $name = $row['first_name'];
}
mysql_query($import) or die(mysql_error());
}
fclose($handle);
Now, I need to iterate each data on the column $data[7], so I echo the first name of each person who has that user_id, but apparently it's wrong because nothing is printed.
PS. please note that I'm the only one uploading the data, I'm not concerned about security stuff or whatever.
Done plenty PHP database stuff but not much with mysql. Would have thought the way you've written the select statement with double quotes surrounding the ID string should be the other way round? I.e. use double quotes for the query string enclosing single quotes for the ID value:
$query = "SELECT * FROM users WHERE ID='".$data[7]."'";
You might want to try some debugging - like putting print_r($row); into your while loop.
I am submitting sample code to read data from the csv file. I have assumed that you have written the precise code for uploading the csv file by using the move_uploaded_file command. So my sample code continues from that point.
// Set path to CSV file
$csvFile = 'test.csv';
$file_handle = fopen($csvFile, 'r');
//fgetcsv($file_handle);//skips the first line while reading the csv file, uncomment this line if you want to skip the first line in csv file
while (!feof($file_handle) )
{
$csv_data[] = fgetcsv($file_handle, 1024);
}
fclose($file_handle);
/*echo '<pre>';
print_r($csv_data);
echo '</pre>';*/
foreach($csv_data as $data)
{
echo "<br>column 1 data = ".$data[0];
//Your insert will go something like this
$import="INSERT into matching(date, time, ..) values('".mysql_real_escape_string($data[0])."', '".mysql_real_escape_string($data[1])."',....)";
mysql_query($import) or die(mysql_error());
}
For demo only I have only printed the value of the first column only. If you want to know the complete structure of the data array you can do it by uncommenting the print_r block. The function mysql_real_escape_string will safely insert the csv data into the database table, if you don't use this function there is high possibility that your mysql query can fail if your csv data contains single or double quotes which breaks the query.
Related
I have a .csv file that is about 5mb (~45,000 rows). What I need to do is run through each row of the file and check if the ID in each line is already in a table in my database. If it is, I can delete that row from the file.
I did a good amount of research on the most memory efficient way to do this, so I've been using a method of writing lines that don't need to get deleted to a temporary file and then renaming that file as the original. Code below:
$file= fopen($filename, 'r');
$temp = fopen($tempFilename, 'w');
while(($row = fgetcsv($file)) != FALSE){
// id is the 7th value in the row
$id = $row[6];
// check table to see if id exists
$sql = "SELECT id FROM table WHERE id = $id";
$result = mysqli_query($conn, $sql);
// if id is in the database, skip to next row
if(mysqli_num_rows($result) > 0){
continue;
}
// else write line to temp file
fputcsv($temp, $row);
}
fclose($file);
fclose($temp);
// overwrite original file
rename($tempFilename, $filename);
Problem is, I'm running into a timeout while executing this bit of code. Anything I can do to make the code more efficient?
You fire a database query per line, aka 45.000 queries... that takes too much time.
Better you do a query before the loop and read the existing id into a lookup array, then only check this array in the loop.
Pseudo code:
$st = query('SELECT id FROM table');
while ($row = $st->fetch()) {
$lookup[ $row['id'] ] = $row['id'];
}
// now read CSV
while($row = fgetcsv($h)) {
$id = $row[6];
if (isset($lookup[ $id ])) {
// exist...
continue;
}
// write the non-existing id to different file...
}
edit:
Assume memory isn't sufficient to hold 1 million integer from the database. How can it still be done efficiently?
Collect ids from CSV into an array. Write a single query to find all those ids in the database and collect (it can be maximal so many as in the CSV). Now array_diff() the ids from file with the ids from database - those ids remaining exist in CSV but not in database.
Pseudo code:
$ids_csv = [];
while($row = fgetcsv($h)) {
$id = row[6];
$ids_csv[] = intval($id);
}
$sql = sprintf('SELECT id FROM table WHERE id IN(%s)', implode(',', $ids_csv));
$ids_db = [];
$st = query($sql);
while ($row = $st->fetch()) {
$ids_db[] = $row['id'];
}
$missing_in_db = array_diff($ids_csv, $ids_db);
I would use LOAD DATA INFILE: https://dev.mysql.com/doc/refman/8.0/en/load-data.html
Your database user needs to have FILE priveleges on the database to use.
to read the csv file into a separate table.
Then you can run one query to delete id's already exist (delete from join ...)
And export the rows that were left intact.
Other option is use your loop to insert your csv file into a seperate table, and then proceed with step 2.
Update: I use LOAD DATA INFILE with csv files up to 2 million rows (at the moment) and do some bulk data manipulation with big queries, it's blazingly fast and I would recommend this route for files containing > 100k lines.
I am in need of a way to export my MYSQL Database to CSV via PHP, but I need to select the column names as well. So Far I have the following which does everything I need except get the column names.
echo "Export Starting \n";
$SQL = ("SELECT *
FROM INF_TimeEntries
WHERE Exported IS NULL");
$result = mysqli_query($db_conn, $SQL) or die("Selection Error " . mysqli_error($db_conn));
echo "Export Data Selected \n";
$fp = fopen('../updateDatabase/timesheetExport/TimeEntries.csv', 'w');
echo "Starting Write to CSV \n";
while($row = mysqli_fetch_assoc($result)){
fputcsv($fp, $row);
$RowID = $row['ID'];
$exportTime = date("Y-m-d H:i:s");
$sql = ("UPDATE INF_TimeEntries
SET Exported = '$exportTime'
WHERE ID = '$RowID'");
if ($mysqli_app->query($sql) === TRUE) {
}
else {
echo date("Y-m-d H:i:s")."\n";
echo "An Error Occured please contact the administrator ". $mysqli_app->error."\n";
}
}
echo "Export Completed \n";
fclose($fp);
mysqli_close($mysqli_app);
mysqli_close($db_conn);
I am not sure how I would go about Achieving this. I do not simply need to get column names but Column names and the data contained in each of these columns. I have not found any information on this in the other question suggested.
Since you're using mysqli_fetch_assoc, the name of the columns are the keys of the $row array in each iteration. You can put that in the file in the first iteration:
echo "Starting Write to CSV \n";
$first = true;
while($row = mysqli_fetch_assoc($result)){
if ($first) {
fputcsv($fp, array_keys($row));
$first = false;
}
fputcsv($fp, $row);
// ..
}
Once you have your $result set from your mysqli_query() method, you can use mysqli_fetch_fields() to return an array of descriptions of the columns in the result set.
Each element of that array is a an object with several properties. One property is name -- which you can use as a header for your csv file. You also get properties like max_length, length, and table. The linked documentation shows an example of using this metadata.
This metadata is especially useful if you have a query more complex than SELECT * FROM table: if you assign aliases to the columns in your query, they show up in the name properties of the metadata array elements.
This works even if the result set has no rows in it.
Sounds pretty simple, as long as everything else is already working for you. You can create an array with the column names, and fputcsv($fp, $array_of_column_names) right before your while loop.
I wrote the following to retrieve values from 2 columns WHEN 'PART_EPOCH' value is the lowest.
$next_event = "SELECT
PART_ID
PART_EPOCH
FROM crud_mysqli
WHERE PART_EPOCH = (SELECT MIN(PART_EPOCH) FROM crud_mysqli)
";
$query = mysqli_query($conn, $next_event);
The expected result should return PART_ID and PART_EPOCH values.
How could i send these 2 values into a TXT file with PHP, comma separated and \n at the end of the line ?
I tried to use array(); with no success, TXT file stays empty and chmod does not changed either.
$path = getcwd() . '/test.txt';
$file = fopen($path, "w"); // this will open the file and erase all datas
$array = array();
$array = mysqli_fetch_array($query);
print_r(array_values($array)); // show result as debug purposes
fwrite($file, $array);
fclose($file);
chmod($file, 0777);
Try using file_put_contents() php function.
$filename ="something.csv";
file_put_contents($filename, serialize($array));
I am in need of a way to export my MYSQL Database to CSV via PHP, but I need to select the column names as well. So Far I have the following which does everything I need except get the column names.
echo "Export Starting \n";
$SQL = ("SELECT *
FROM INF_TimeEntries
WHERE Exported IS NULL");
$result = mysqli_query($db_conn, $SQL) or die("Selection Error " . mysqli_error($db_conn));
echo "Export Data Selected \n";
$fp = fopen('../updateDatabase/timesheetExport/TimeEntries.csv', 'w');
echo "Starting Write to CSV \n";
while($row = mysqli_fetch_assoc($result)){
fputcsv($fp, $row);
$RowID = $row['ID'];
$exportTime = date("Y-m-d H:i:s");
$sql = ("UPDATE INF_TimeEntries
SET Exported = '$exportTime'
WHERE ID = '$RowID'");
if ($mysqli_app->query($sql) === TRUE) {
}
else {
echo date("Y-m-d H:i:s")."\n";
echo "An Error Occured please contact the administrator ". $mysqli_app->error."\n";
}
}
echo "Export Completed \n";
fclose($fp);
mysqli_close($mysqli_app);
mysqli_close($db_conn);
I am not sure how I would go about Achieving this. I do not simply need to get column names but Column names and the data contained in each of these columns. I have not found any information on this in the other question suggested.
Since you're using mysqli_fetch_assoc, the name of the columns are the keys of the $row array in each iteration. You can put that in the file in the first iteration:
echo "Starting Write to CSV \n";
$first = true;
while($row = mysqli_fetch_assoc($result)){
if ($first) {
fputcsv($fp, array_keys($row));
$first = false;
}
fputcsv($fp, $row);
// ..
}
Once you have your $result set from your mysqli_query() method, you can use mysqli_fetch_fields() to return an array of descriptions of the columns in the result set.
Each element of that array is a an object with several properties. One property is name -- which you can use as a header for your csv file. You also get properties like max_length, length, and table. The linked documentation shows an example of using this metadata.
This metadata is especially useful if you have a query more complex than SELECT * FROM table: if you assign aliases to the columns in your query, they show up in the name properties of the metadata array elements.
This works even if the result set has no rows in it.
Sounds pretty simple, as long as everything else is already working for you. You can create an array with the column names, and fputcsv($fp, $array_of_column_names) right before your while loop.
I am trying to write a Database Query to a file, But am just wondering how I get the SQL Data to be input into the file.
It is currently outputting nothing to the .txt file at all. I suspect its something to do with the while loop and am questioning whether it needs to be there or not.
My code is :
function backup_tables($host,$user,$pass,$name,$tables = '*')
{
$link = mysql_connect($host,$user,$pass);
mysql_select_db($name,$link);
$query = "SELECT gamename, username, MAX(thescore)
FROM game_scores
GROUP BY gamename, username
ORDER BY gamename, thescore DESC";
$result = mysql_query($query);
while($row = mysql_fetch_array($result)) {
$scoredata = $row;
}
//save file
$handle = fopen('scores/games-backup'.'.txt','w+');
fwrite($handle,$scoredate);
fclose($handle);
echo "Success";
}
Any help on writing the SQL Results to this text file would be much appreciated.
$row is an array, you will need to make it a string before you can write it to a file, you can use implode().
$scoredata is also being overwritten in each loop, maybe use $scoredata[] instead
while($row = mysql_fetch_array($result)) {
$scoredata[] = implode("; ", $row);
}
This will make $scoredata an array also, so you will need to convert that to a string too!
$handle = fopen('scores/games-backup'.'.txt','w+');
fwrite($handle,implode("\r\n", $scoredata));
fclose($handle);
This should print each database row on a new line in the file.
EDIT: This will just be a text file, formatted as text, not that great for a backup file. You will need to structure the text into an SQL format to make it useful..
Try something like :
$handle = fopen('scores/games-backup'.'.txt','w+');
while($row = mysql_fetch_array($result)) {
fputs($handle, join(';', $row)."\n");
}
fclose($handle);
$row is an array, you must join it (or access the elements )
$row is not a string, you have to make it a string before you can put it into a file.
And you have to change $scoredata to $scoredata[] because, it is now being overwritten continiously.
You can use print_r function with second parameter set true.
$str = print_r($row, true);
fwrite($handle,implode("\r\n", $str));
Or you can serialize the array
$str = serialize($row);
fwrite($handle,implode("\r\n", $str));