Storing images as a blob in mysql database - php

So I have searched high and low for a relevant answer, and I have yet to find one, so I'm just going to ask the question myself. In my MySQL tables I have 5 columns of id, customer__id, file_data, file_name, mime_typethe relevant one here is file_data which is a type of LONGBLOB. Now from what I understand the size of files that a LONGBLOB can handle is pretty substantial but when I try to upload a file that is around 978 KB it fails, could it be that the dimensions of the image are too large (2048 x 1536)?
Here is the code for my uploader. It works very elegantly for things like excel sheets, pdf, word documents, and other stuff, but when it comes to images it fails:
<?php
require_once '../../inc/config.php';
$response = array();
$response['errors'] = false;
$id = $_REQUEST['id'];
if(!empty($_FILES)){
//set default data arrays
$names = array(); //stores file names
$files = array(); //stores the file data
$mime_types = array(); //store the file type as a mime type
//force each file name to the names array
foreach($_FILES['file']['name'] as $name){
array_push($names, $name);
}
//force the file data into its own array spot in the files array
foreach($_FILES['file']['tmp_name'] as $temp){
array_push($files, prepareImageDBString($temp));
}
//force the mimetypes into the mime_types array
foreach($_FILES['file']['type'] as $type){
array_push($mime_types, $type);
}
//process all three of the file arrays simultaneously so that no data is left out
for($i = 0; $i < count($names); $i++){
$file_name = $names[$i];
$file_data = $files[$i];
$mime_type = $mime_types[$i];
//set the query for the data to go into the note_file table in the database
$q = "INSERT INTO brb.files (customer__id, file_name, file_data, mime_type)
VALUES('$id', '$file_name', '$file_data', '$mime_type')";
//run the query
if($stmt = $CONN->prepare($q)){
//process any errors that may occur
if(!$stmt->execute()){
printf("Error Message: %s\n", $CONN->error);
}
}
}
}
echo json_encode($response);
function prepareImageDBString($filepath){
$out = 'null';
$handle = fopen($filepath, 'r');
if($handle){
$content = fread($handle, filesize($filepath));
$content = bin2hex($content);
fclose($handle);
$out = $content;
}
return $out;
}
?>
If someone could point me in the direction that would be fantastic. Please do not provide be an answer on why the practice is bad, I'm aware it's bad practice and know what proper practice is, this is a learning tool for myself and nothing more, just bear with me on it for a minute.
Thank you to everyone who provides help!

Related

Automating CSV conversion from iso-8859-2 to utf-8

I have quite a few CSV files that are unfortunately encoded with iso-8859-2 (according to Brackets). I would like to iterate over these files with PHP and convert them.
I found https://csv.thephpleague.com/9.0/converter/charset/ but the way I can use the conversion function is uncertain to me.
Their example code
use League\Csv\CharsetConverter;
$csv = new SplFileObject('/path/to/french.csv', 'r');
$csv->setFlags(SplFileObject::READ_CSV | SplFileObject::SKIP_EMPTY);
$encoder = (new CharsetConverter())->inputEncoding('iso-8859-15');
$records = $encoder->convert($csv);
This is my code so far that is part of a form to upload one file and save the contents to the database for testing. It of course saves the text in the incorrect format.
$db = ConnectDB::getConnection('address_dtb');
$sql = " ... ";
$stmt = $db->prepare($sql);
$rowCount = 0;
$temp_name = $_FILES['adresscsv']['tmp_name'];
$file_handle = fopen($temp_name, 'r');
while (($items = fgetcsv($file_handle, 1000, ';')) !== FALSE) {
if($flag) { $flag = false; continue; }
$stmt->execute($items);
$rowCount++;
}
fclose($file_handle);
ConnectDB::closeConnection($db);
What is the correct way to use the PHP CSV library above to iterate over locally saved files in a for loop to automate the process?
I ended up using iconv as hinted.
$files = glob('address/*.csv');
foreach ($files as $csv) {
$file_data = file_get_contents($csv);
$utf8_file_data = iconv('Windows-1250', 'UTF-8', $file_data);
file_put_contents($csv, $utf8_file_data);
}
You do not have to use a library. There is a function in PHP that can do that iconv

Storing image file to SQL Server using php

I have a SQL Server table with a column called personal image with image data type.
I want to insert or update this field using PHP sqlsrv extension.
I have that image file for update query and i try this but the image not inserted correctly.
function prepareImageDBString($filepath)
{
$out = 'null';
$handle = #fopen($filepath, 'rb');
if ($handle)
{
$content = #fread($handle, filesize($filepath));
$content = bin2hex($content);
#fclose($handle);
$out = "0x".$content;
}
return $out;
}
$out = prepareImageDBString('871190915.jpg');
$update = "UPDATE Person SET PersonelImage=(?) WHERE no=871190915";
$image1=array($out);
sqlsrv_query($conn, $update, $image1);
Can anyone help me?

PDF Files from database keep getting corrupted

So I am storing my files in a database. Don't ask why, just know that I am not in control of this. Next, I am able to successfully store them as a hexidecimal representation and then spit them back for display with no problem, but then I attach them to an email using PHPMailer and they get sent properly with the right name and all, but they are corrupted. I will walk you through step by step below so that you know exactly how it is being stored, and this may help me debug my issue. (Please note that all code is paraphrased to save space and only show what is needed)
STEP 1
File is grabbed and then processed
$name = $_FILES['file_data']['name'];
$file = prepareImageDBString($_FILES['file_data']['tmp_name']);
$mime_type = $_FILES['file_data']['type'];
name, file, and mime_type are stored
here is the function prepareImageDBString()
function prepareImageDBString($filepath){
$out = 'null';
$handle = #fopen($filepath, 'r');
if($handle){
$content = #fread($handle, filesize($filepath));
$content = bin2hex($content);
#fclose($handle);
$out = $content;
}
return $out;
}
STEP 2
When the file is being viewed I show it as an embedded object. This file is small so I just posted the whole code. Do note that the file shows up with no problems here.
$q = "SELECT lease_doc_file_data FROM lease_doc_file WHERE lease_doc__id ='".$_GET['id']."'";
$file = "";
foreach($CONN->query($q) as $row){
$file = $row['lease_doc_file_data'];
}
if(!empty($file)){
header("Content-type: application/pdf");
ob_clean();
flush();
echo hextobin($file);
}
Here is the function hextobin()
function hextobin($hexstr){
$n = strlen($hexstr);
$sbin = "";
$i = 0;
while($i < $n){
$a = substr($hexstr,$i,2);
$c = pack("H*", $a);
if ( $i == 0 ){ $sbin = $c; }
else { $sbin .= $c;}
$i += 2;
}
return $sbin;
}
STEP 3
Finally the part where I go to send it as a mailer.
$q = "SELECT lease_doc_file_data, lease_doc_file_name, lease_doc_file_type FROM lease_doc_file WHERE lease_doc__id ='$id'";
$file_data = "";
$file_name = "";
$file_type = "";
foreach($CONN->query($q) as $row){
$file_data = $row['lease_doc_file_data'];
$file_name = $row['lease_doc_file_name'];
$file_type = $row['lease_doc_file_type'];
}
$file_data = hextobin($file_data);
$mail->AddStringAttachment($file_data, $file_name, 'binary', $file_type);
So this is the three step process and I"m not sure where the error is coming from. Hopefully someone can help! Thank you for all help in advance!

How can i upload a csv file in mysql database using multithreads? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I have a csv file, containing millions of email addresses which I want to upload fast into a mysql database with PHP.
Right now I'm using a single threaded program which takes too much time to upload.
//get the csv file
$file = $_FILES['csv']['tmp_name'];
$handle = fopen($file,"r");
//loop through the csv file and insert into database
do {
if ($data[0]) {
$expression = "/^[_a-z0-9-]+(\.[_a-z0-9-]+)*#[a-z0-9-]+(\.[a-z0-9-]+)*(\.[a-z]{2,3})$/";
if (preg_match($expression, $data[0])) {
$query=mysql_query("SELECT * FROM `postfix`.`recipient_access` where recipient='".$data[0]."'");
mysql_query("SET NAMES utf8");
$fetch=mysql_fetch_array($query);
if($fetch['recipient']!=$data[0]){
$query=mysql_query("INSERT INTO `postfix`.`recipient_access`(`recipient`, `note`) VALUES('".addslashes($data[0])."','".$_POST['note']."')");
}
}
}
} while ($data = fgetcsv($handle,1000,",","'"));
First of all, I can't stress enough; fix your indentation - it will make life easier for everyone.
Secondly, the answer depends a lot on the actual bottlenecks you are encountering:
Regular expressions are very slow, especially when they're in a loop.
Databases tend to either work well for WRITES or for READS but not BOTH: try decreasing the amount of queries beforehand.
It stands to reason that the less PHP code in your loop, the faster it will work. Consider decreasing conditions (for instance).
For the record, your code is not safe against mysql injection: filter $_POST before hand [*]
[*] speaking of which, it's faster to access a variable than the index of an array, like $_POST.
You can simulate multithreading by having your main program split the huge CSV file into a smaller one and run each file into a different process.
common.php
class FileLineFinder {
protected $handle, $length, $curpos;
public function __construct($file){
$handle = fopen($file, 'r');
$length = strlen(PHP_EOL);
}
public function next_line(){
while(!feof($this->handle)){
$b = fread($this->handle, $this->length);
$this->curpos += $this->length;
if ($b == PHP_EOL) return $this->curpos;
}
return false;
}
public function skip_lines($count){
for($i = 0; $i < $count; $i++)
$this->next_line();
}
public function __destruct(){
fclose($this->handle);
}
}
function exec_async($cmd, $outfile, $pidfile){
exec(sprintf("%s > %s 2>&1 & echo $! >> %s", $cmd, $outfile, $pidfile));
}
main.php
require('common.php');
$maxlines = 200; // maximum lines subtask will be processing at a time
$note = $_POST['note'];
$file = $_FILES['csv']['tmp_name'];
$outdir = dirname(__FILE__) . DIRECTORY_SEPARATOR . 'out' . DIRECTORY_SEPARATOR;
//make sure our output directory exists
if(!is_dir($outdir))
if(!mkdir($outdir, 0755, true))
die('Cannot create output directory: '.$outdir);
// run a task for each chunk of lines in the csv file
$i = 0; $pos = 0;
$l = new FileLineFinder($file);
do {
$i++;
exec_async(
'php -f sub.php -- '.$pos.' '.$maxlines.' '.escapeshellarg($file).' '.escapeshellarg($note),
$outdir.'proc'.$i.'.log',
$outdir.'proc'.$i.'.pid'
);
$l->skip_lines($maxlines);
} while($pos = $l->next_line());
// wait for each task to finish
do {
$tasks = count(glob($outdir.'proc*.pid'));
echo 'Remaining Tasks: '.$tasks.PHP_EOL;
} while ($tasks > 0);
echo 'Finished!'.PHP_EOL;
sub.php
require('common.php');
$start = (int)$argv[1];
$count = (int)$argv[2];
$file = $argv[3];
$note = mysql_real_escape_string($argv[4]);
$lines = 0;
$handle = fopen($file, 'r');
fseek($handle, $start, SEEK_SET);
$expression = "/^[_a-z0-9-]+(\.[_a-z0-9-]+)*#[a-z0-9-]+(\.[a-z0-9-]+)*(\.[a-z]{2,3})$/";
mysql_query('SET NAMES utf8');
//loop through the csv file and insert into database
do {
$lines++;
if ($data[0]) {
if (preg_match($expression, $data[0])) {
$query = mysql_query('SELECT * FROM `postfix`.`recipient_access` where recipient="'.$data[0].'"');
$fetch = mysql_fetch_array($query);
if($fetch['recipient'] != $data[0]){
$query = mysql_query('INSERT INTO `postfix`.`recipient_access`(`recipient`, `note`) VALUES("'.$data[0].'","'.$note.'")');
}
}
}
} while (($data = fgetcsv($handle, 1000, ',', '\'')) && ($lines < $count));
Credits
https://stackoverflow.com/a/2162528/314056
https://stackoverflow.com/a/45966/314056
The most pressing thing to do is to make sure your database is properly indexed so the lookup query you do for every row is as fast as possible.
Other than that, there simply isn't that much you can do. For a multithreaded solution, you'll have to go outside PHP.
You could also just import the CSV file in mySQL, and then weed out the superfluous data using your PHP script - that is likely to be the fastest way.
Just a general suggestion: The key to speed up any program is to know which part take most of the time.
And then figure out how to reduce it. Sometimes you will be very surprised by the actual result.
btw, I don't think multithreading would solve your your problem.
Put the whole loop inside an SQL transaction. That will speed things up by an order of magnitude.

CSV into MySQL using PHP

I'm trying to import data from my students.csv file into mysql using php. The entries in the csv file is in such a way that column (student_number, fname, lname, level) will be inserted into biodata table..
I'm also uploading the student.csv file from my computer.
When I run the page I dont get anything out on the screen.
session_start();
require('includes/dbconnect.php');
require 'includes/header.inc.php';
//check for file upload
if (isset($_FILES['csv_file']) && is_uploaded_file($_FILES['csv_file']['tmp_name'])) {
//upload directory
$upload_dir = "C:\Users\DOTMAN\Documents\students.csv";
//create file name
$file_path = $upload_dir . $_FILES['csv_file']['name'];
//move uploaded file to upload dir
if (!move_uploaded_file($_FILES['csv_file']['tmp_name'], $file_path)) {
//error moving upload file
echo "Error moving file upload";
}
//open the csv file for reading
$handle = fopen($file_path, 'r');
//turn off autocommit and deletethe bio data
mysql_query("SET AUTOCOMMIT=0");
mysql_query("BEGIN");
mysql_query("TRUNCATE TABLE biodata") or die(mysql_error());
while (($data = fgetcsv($handle, 1000, ',')) !== FALSE) {
//Access field data in $data array ex.
$student_number = $data[0];
$fname = $data[1];
$lname = $data[2];
$level = $data[3];
//Use data to insert into db
$query = "INSERT INTO biodata (student_number, fname, lname, level)
VALUES ('$student_number', '$fname', '$lname', '$level')";
mysql_query($query) or die (mysql_error());
}
}
I'd suggest you to upload CSV-file with LOAD DATA INFILE command. This is fast method.
if you only need to do this once, i would consider using something like: http://csv2sql.com/
One immediate issue I can see is here:
$upload_dir = "C:\Users\DOTMAN\Documents\students.csv";
//create file name
$file_path = $upload_dir . $_FILES['csv_file']['name'];
You are already assigning the entire path, including the file name, to the $upload_dir variable - and then you're appending the uploaded file name again.
If you think there are errors in your code, start by adding
ini_set('display_errors', 1);
error_reporting(E_ALL);
to the beginning of your PHP code and fix any warnings/errors displayed. You can then turn off printing error messages by changing the second parameter to 0 in the first call.
Have u debug the $_FILES:
print_r($_FILES);
before doing any thing
Solution using PHP
$file = 'path/to.csv';
$lines = file($file);
$firstLine = $lines[0];
foreach ($lines as $line_num => $line) {
if($line_num==0) { continue; } //escape the header column
$arr = explode(",",$line);
$column1= $arr[0];
$column2= $arr[1];
echo $column1.$column2."<br />";
//put the mysql insert statement here
}

Categories