I know this is an ugly idea, but how can I bypass this maximum execution error. I don't want to configure any .ini that I have read in some solutions.
I have this code to upload csv to sql database but this keeps me giving an error when uploading large files. Please help me revise my code guys
<?php
if(isset($_POST['submit'])) {
$host = 'localhost';
$user = 'root';
$password = '';
$db = 'jeremy_db';
$con = mysqli_connect($host,$user,$password) or die('Could not' .mysqli_error($con));
mysqli_select_db($con, $db) or die ('Could not' .mysqli_error($con));
$file = $_FILES['file']['tmp_name'];
$handle = fopen($file, "r");
$c = 0;
while(($csvdata = fgetcsv($handle,10000,","))!== FALSE){
$sha1 = $csvdata[0];
$vsdt = $csvdata[1];
$trendx = $csvdata[2];
$sql = "INSERT INTO jeremy_table_trend (sha1,vsdt,trendx) VALUES ('$sha1','$vsdt','$trendx')";
$query = mysqli_query($con , $sql);
$c = $c+1;
}
if($query){
echo "success";
}
else {
echo "error";
}
}
?>
try this
add this line to your code
ini_set('max_execution_time', 300); //300 seconds = 5 minutes //whatever you want
Taken from How to increase maximum execution time in php
Executing insert statements in the loop is expensive. You can try dumping the csv file directly into MySQL, that will take relatively lesser execution time.
Refer to this question
Related
This question already has answers here:
Can I mix MySQL APIs in PHP?
(4 answers)
Closed 5 years ago.
I want to insert data to phymyadmin table using CSV file from front-end of website, which must UPDATE all the record previously in my table.
This code isn't working at all. Please locate the error.
<?php
$servername = "localhost";
$username = "";
$password = "";
$dbname = "";
$conn = mysql_connect($servername, $username , $password );
if(! $conn ) {
die('Could not Find Some Error Occured: ' . mysql_error());
}
if(isset($_POST["submit"]))
{
if($_FILES['file']['name'])
{
$filename = explode(".", $_FILES['file']['name']);
if($filename[1] == 'csv')
{
$handle = fopen($_FILES['file']['tmp_name'], "r");
while($data = fgetcsv($handle))
{
$uname = mysqli_real_escape_string($connect, $data[0]);
$pass = mysqli_real_escape_string($connect, $data[1]);
mysql_select_db('desiresq_record');
$query = "INSERT into login (username, password)
VALUES ('$uname','$pass')";
mysqli_query($connect, $query);
}
fclose($handle);
echo "<script>alert('Import done');</script>";
}
}
}
?>
You're using both MySQL and the MySQLi extensions in your script.
I'd suggest just sticking to MySQLi, seeing as MySQL is long deprecated and shouldn't be used due to security issues.
Best of luck!
I'm trying to insert values extracted from a csv file to a mysql table. It runs but the table is not populated. I've tried to debug for the last XXXX but just can't see my error. Echo-ing out the values give me the correct SQL but when it comes to the INSERT - no dice.
Thanks very much for your help.
<?php
$host = 'localhost';
$user = 'fulltime_admin';
$pass = 'secret';
$database = 'fulltime_db';
$db = mysql_connect($host, $user, $pass);
mysql_query($database, $db);
//////////////////////////////// EDIT ////////////////////////////////////
$redirect_num = 500; // Select how many rows to insert each time before refresh.
// More rows = faster insertion. However cannot be too high otherwise it will timeout.
$filename = "ps4_emails.csv"; // The file we are going to get the data from...
$table = "`ps4_emails`";
////////////////////////////// END EDIT //////////////////////////////////
$file = file($filename);
$lines = count($file);
// Have we just redirected?
$nextline = $_GET['nextline'];
if (!isset($nextline)){
$nextline = 0;
}
$query = "INSERT INTO ".$table." (email) VALUES ('".$final_line[0]."')";
for ($line=$nextline; $line<=$lines; $line++){
$final_line = explode(",", $file[$line]);
if ($line!=$lines){
mysql_query($query,$db);
}
if ($line % $redirect_num){
// something needs to go here
} else {
$nextline = $line+1;
exit ('<meta http-equiv="refresh" content="0;url=texttomysqlemails.php?nextline='.$nextline.'" />');
}
echo ( $line==$lines ) ? "Done" : "";
}
?>
Put your query inside loop in order use it with variable $final_line.
Try this :
$final_line = explode(",", $file[$line]);
if ($line!=$lines){
$query = "INSERT INTO ".$table." (email) VALUES ('".$final_line[0]."')";
mysql_query($query,$db);
}
Don't use mysql_*. It's deprecated and removed from PHP 7. Use mysqli_* or PDO.
This seems like a perfect script to run from the command line PHP CLI and therefore you can forget about all the refresh complexity.
If the file is huge, like your comment suggest, loading all the file into memory may also bring you up against the PHP memory limits, so it might be better to read a line at a time rather than the whole file using fgetcsv() which is intended for reading csv files.
<?php
$host = 'localhost';
$user = 'fulltime_admin';
$pass = 'secret';
$database = 'fulltime_db';
$db = mysql_connect($host, $user, $pass);
mysql_query($database, $db);
$filename = "ps4_emails.csv";
$table = "";
$handle = fopen('ps4_emails.csv', 'r');
if ( ! $handle ) {
echo 'File does not exists in this location';
exit;
}
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$query = "INSERT INTO `ps4_emails` (email) VALUES( '{$data[0]}')";
mysql_query($query,$db);
}
?>
You now just run this script from the command line/terminal like
>php script.php
And it can run for minutes/hours/days with no likleyhood of blowing any limits.
I have to mention this or someone will nag me for not saying it
Please dont use the mysql_ database extension, it is deprecated (gone for ever in PHP7)
Especially if you are just learning PHP, spend your energies learning the PDO or mysqli_ database extensions,
and here is some help to decide which to use
When you need to upload the real file, it would also be a good idea to add a restart mechanism, so you can restart the process from whereever a problem happened or someone shut the database down for a backup or some other unforseen hiccup.
<?php
$host = 'localhost';
$user = 'fulltime_admin';
$pass = 'secret';
$database = 'fulltime_db';
$restart_from = 0;
$db = mysql_connect($host, $user, $pass);
mysql_query($database, $db);
$filename = "ps4_emails.csv";
$table = "";
$handle = fopen('ps4_emails.csv', 'r');
if ( ! $handle ) {
echo 'File does not exists in this location';
exit;
}
// is it a restart?
if ( file_exists('restart.txt') ) {
// its a restart
$restart_from = file_get_contents('restart.txt');
// read up the file to the last good row inserted
for ( $i=0; $i<=$restart_from; $i++ ) {
$data = fget($handle, 1000);
}
}
$upd_cnt = restart_from;
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$query = "INSERT INTO `ps4_emails` (email) VALUES( '{$data[0]}')";
mysql_query($query,$db);
$upd_cnt++;
file_put_contents('restart.txt', $upd_cnt);
}
?>
The above restart code is not tested, but I have used something very like this in the past very successfully. So you will have to check I have not made any silly mistakes, but it should give you an idea of how to do a restart from the last row successfully updated before a crash.
You can use LOAD DATA INFILE to insert from file.
refer http://dev.mysql.com/doc/refman/5.7/en/load-data.html
insert csv file data into mysql
I am working on some project, in which i want to read and Parse 100000+ files with 6GB of size.
My Problem:
1> Read and Parse One XML File (sizes between 5kb-500kb) in few seconds.
so the complete set of XML Files (100000+ files with 6GB size) read & Parse in 3-5 Hours.
2> Fastest way to do this
Currently One XML files (5KB-500KB) take a minute of less to read and Parse.
Regards,
Mian
P.S. Please also review Code:
<HTML>
<HEAD>
<META HTTP-EQUIV="CACHE-CONTROL" CONTENT="NO-CACHE">
<META HTTP-EQUIV="EXPIRES" CONTENT="0">
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"><style type="text/css">
<!--
body,td,th {
color: #CCCCCC;
}
body {
background-color: #000066;
}
-->
</style></HEAD>
</BODY>
<script>
<!--
/*
Auto Refresh Page with Time script By JavaScript Kit (javascriptkit.com) Over 200+ free scripts here!
*/
//enter refresh time in "minutes:seconds" Minutes should range from 0 to inifinity. Seconds should range from 0 to 59
var limit="00:10"
if (document.images){
var parselimit=limit.split(":")
parselimit=parselimit[0]*60+parselimit[1]*1
}
function beginrefresh(){
if (!document.images)
return
if (parselimit==1)
window.location.reload()
else{
parselimit-=1
curmin=Math.floor(parselimit/60)
cursec=parselimit%60
if (curmin!=0)
curtime=curmin+" minutes and "+cursec+" seconds left until page refresh!"
else
curtime=cursec+" seconds left until page refresh!"
window.status=curtime
setTimeout("beginrefresh()",1000)
}
}
window.onload=beginrefresh
//-->
</script>
</HEAD>
<BODY>
<?php
require("MagicParser.php");
//header("Content-Type: text/plain");
$dbServer = "127.0.0.1";
$dbUser = "root";
$dbPass = "";
$dbName = "GDatabase";
$text = '';
$c = mysql_connect($dbServer, $dbUser, $dbPass) or die("Couldn't connect to database");
$d = mysql_select_db($dbName) or die("Couldn't select database");
//mysql_query("SET NAMES utf8;");
//mysql_query("SET CHARACTER_SET utf8;");
$sql = "select
id, file_name
from
tableP_files
where status = '' limit 1";
$result = mysql_query($sql,$c);
while($row = mysql_fetch_array($result))
{
$id = $row['id'];
$file_name = $row['file_name'];
$url = 'http://localhost/GDatabase/XML/' . $file_name;
}
$formatString = MagicParser_getFormat($url);
$update_query = "update tableP_files set format_string = '$formatString' where id = $id";
if(!mysql_query($update_query,$c))
{
echo 'ERROR';
}
print "Format String: ".$formatString."\n\n";
// MagicParser_parse($url,"myRecordHandler",$formatString);
// MagicParser_parse($url,"myRecordHandler","xml|ARTICLE/FLOATS-WRAP/TABLE-WRAP/TABLE/TBODY/TR/TD/");
MagicParser_parse($url,"myRecordHandler","xml|ARTICLE/");
function myRecordHandler($record)
{
$dbServer = "127.0.0.1";
$dbUser = "root";
$dbPass = "";
$dbName = "GDatabase";
$c = mysql_connect($dbServer, $dbUser, $dbPass) or die("Couldn't connect to database");
$d = mysql_select_db($dbName) or die("Couldn't select database");
mysql_query("SET NAMES utf8;");
mysql_query("SET CHARACTER_SET utf8;");
$sql = "select
id, file_name
from
tableP_files
where status = '' limit 1";
$result = mysql_query($sql,$c);
while($row = mysql_fetch_array($result))
{
$id = $row['id'];
$file_name = $row['file_name'];
$file_name = 'http://localhost/GDatabase/test/' . $file_name;
}
foreach($record as $key => $value)
{
$tag = addslashes($key);
$value = addslashes($value);
$insert_query = "insert into tableP_xml set file_id = '$id', file_name = '$file_name', tag = '$tag', value = '$value', status = ''";
if(!mysql_query($insert_query,$c))
{
echo 'ERROR';
}
}
$update_query = "update tableP_files set status = 'done' where id = $id";
if(!mysql_query($update_query,$c))
{
echo 'ERROR';
}
echo "Done: " . $id . " - " . $file_name;
return TRUE;
}
?>
</BODY>
</HTML>
I just created 100000 xml files of size 60kb each and in a php tried to just read them with file_get_contents and it took 87.5 seconds. Mind you! I am on an ssd, have plenty of ram and a powerful i5 4th gen processor. It took ~90 seconds to just load it in memory.
So, how do you make this faster? concurrency.
I split the task into 4 chunks of 25000xml files and the time to load the files to memory (sequentially) dropped to ~30 seconds. Again, this is just the time to load the xml into memory. So if you are going to do more processing on the xml, you need more processing power or time.
Now, how do you scale this? Enter gearman. Gearman lets you process parallel tasks by handing out jobs to workers via a central server. You can even have a bunch of workers on different servers register for doing your tasks. I don't think you need a super computer at all. You just need to define all your jobs once and let the workers do the jobs(asynchronously?).
Right now I am inserting blob files into a database. I have read up on the update syntax for mysql I can not figure out how to modify my code to update a row with the BLOB instead of inserting a new row with the BLOB. Could someone help me with this?
Here is my code:
<?php
// Create MySQL login values and
// set them to your login information.
$username = "root";
$password = "";
$host = "localhost";
$database = "test";
$tbl_name="members";
// Make the connect to MySQL or die
// and display an error.
$link = mysql_connect($host, $username, $password);
if (!$link) {
die('Could not connect: ' . mysql_error());
}
// Select your database
mysql_select_db ($database);
// Make sure the user actually
// selected and uploaded a file
if (isset($_FILES['image']) && $_FILES['image']['size'] > 0) {
// Temporary file name stored on the server
$tmpName = $_FILES['image']['tmp_name'];
// Read the file
$fp = fopen($tmpName, 'r');
$data = fread($fp, filesize($tmpName));
$data = addslashes($data);
fclose($fp);
// Create the query and insert
// into our database.
$query = "INSERT INTO members ";
$query .= "(image) VALUES ('$data')";
$results = mysql_query($query, $link);
// Print results
print "Thank you, your file has been uploaded.";
}
else {
print "No image selected/uploaded";
}
// Close our MySQL Link
mysql_close($link);
?>
1° You need to pass a referecence for what Data you are trying to update, like the Primary Key Id From Table.
2° Update SQL should be like it
$image = mysql_real_escape_string($unsafe_image);
$id = mysql_real_escape_string($unsafe_id);
$query = "UPDATE members SET image = '$data' WHERE id_image = $id";
$results = mysql_query($query, $link);
Using Johnboy's tutorial on importing .csv files to MySQL, I tried to make a script that would take exchange rate data from Yahoo finance, and write it to the MySQL database.
<?php
//connect to the database
$host = "****"
$user = "****"
$password = "****"
$database = "****"
$connect = mysql_connect($host,$user,$password);
mysql_select_db($database,$connect);
//select the table
if ($_FILES[csv][size] > 0) {
//get the csv file
$symbol = "ZARINR"
$tag = "l1"
$file = "http://finance.yahoo.com/d/quotes.csv?e=.csv&f=$tag&s='$symbol'=x";
$handle = fopen($file,"r");
//loop through the csv file and insert into database
do {
if ($data[0]) {
mysql_query("INSERT INTO ZAR_to_INR(exchange_rate) VALUES
(
'".addslashes($data[0])."',
)
");
echo "done";
}
} while ($data = fgetcsv($handle,1000,",","'"));
//redirect
header('Location: import.php?success=1'); die;
}
else{
echo "nope";
}
?>
I added the echos in the hope that they'd tell me whether or not the script worked. It doesn't work at all. There are no error messages or anything. When I run the script by opening it in my webhost, it simply does not run.
I'd appreciate any advice on how to make this script work (or even an alternate way of solving the problem).
try using mysql debugs :
mysql_select_db($database) or die('Cant connect to database');
$result = mysql_query($query) or die('query fail : ' . mysql_error());
$connect = mysql_connect($host,$user,$password)
or die('Cant connect to server: ' . mysql_error());
to find this outputs you need to check your php error_log : where-does-php-store-the-error-log